[Feature] Support sequence parallel without the need for DP, 1.9%~5.0% E2E Throughput Improvement (#47070)

Signed-off-by: yewentao256 <zhyanwentao@126.com>
Signed-off-by: gcanlin <canlinguosdu@gmail.com>
Co-authored-by: Canlin Guo <canlinguosdu@gmail.com>
This commit is contained in:
Wentao Ye
2026-07-05 05:41:30 -07:00
committed by GitHub
co-authored by Canlin Guo
parent fa4321de3d
commit b6cc46ec3b
7 changed files with 33 additions and 22 deletions
+2 -3
View File
@@ -632,8 +632,8 @@ class ParallelConfig:
# The all_reduce at the end of attention (during o_proj) means that
# inputs are replicated across each rank of the tensor parallel group.
# If using expert-parallelism with DeepEP All2All ops, replicated
# tokens results in useless duplicate computation and communication.
# If using expert-parallelism, replicated tokens results in useless
# duplicate computation and communication.
#
# In this case, ensure the input to the experts is sequence parallel
# to avoid the excess work.
@@ -652,7 +652,6 @@ class ParallelConfig:
)
and self.enable_expert_parallel
and self.tensor_parallel_size > 1
and self.data_parallel_size > 1
)
@property