[Perf][Nemotron] Skip redundant latent-MoE all-reduce at TP>1 (~13% decode win) (#52301)
Signed-off-by: Junhao Shen <junshen@nvidia.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Misha Goin <mgoin64@gmail.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
This commit is contained in:
co-authored by
Claude Opus 5
Misha Goin
mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
parent
a2e8f1ea94
commit
4bac767695
@@ -457,9 +457,18 @@ class MoERunner(MoERunnerInterface):
|
||||
Latent MoE output transforms may contain non-linear ops, e.g. RMSNorm.
|
||||
TP partial routed outputs must be summed in latent space before such
|
||||
transforms are applied.
|
||||
|
||||
A transform that commutes with the TP sum is exempt: if
|
||||
``sum_r T(x_r) == T(sum_r x_r)``, applying the transform to the local
|
||||
partial output and letting the existing late all-reduce sum the
|
||||
combined result is equivalent, and costs one collective instead of two.
|
||||
Such a transform opts out by setting ``reduce_commutative = True``.
|
||||
The default is False, so transforms that do not declare themselves
|
||||
keep being reduced early.
|
||||
"""
|
||||
if (
|
||||
self.routed_output_transform is not None
|
||||
and not getattr(self.routed_output_transform, "reduce_commutative", False)
|
||||
and not self.moe_config.is_sequence_parallel
|
||||
and (self.moe_config.tp_size > 1 or self.moe_config.ep_size > 1)
|
||||
and not fused_output_is_reduced
|
||||
|
||||
Reference in New Issue
Block a user