[Perf][DSpark] Collapse DeepSeek-V4.1 draft states before SP all-gather (#56903)

Signed-off-by: Juntian Liu <Juntianl777@gmail.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai>
This commit is contained in:
Juntian Liu
2026-09-15 02:58:57 -07:00
committed by GitHub
co-authored by Codex Woosuk Kwon
parent abb4cb9fc5
commit 073f883b56
2 changed files with 23 additions and 3 deletions
+2 -3
View File
@@ -217,15 +217,14 @@ class DSparkDeepseekV4Model(nn.Module):
residual,
)
hidden_states = mhc_post_tilelang(hidden_states, residual, post_mix, res_mix)
if self.use_sequence_parallel:
hidden_states = sp_all_gather(hidden_states)[:full_num_tokens]
pre_mix = sp_all_gather(pre_mix)[:full_num_tokens]
# Collapse the hc copies with the pre-mix from the last layer's FFN
# mixes — the mix the reference forward_head applies (hc_pre with the
# last block's ffn pre-mix). Return the PRE-norm head hidden;
# compute_logits applies self.norm.
assert pre_mix is not None
hidden_states = hc_collapse_triton(hidden_states, pre_mix)
if self.use_sequence_parallel:
hidden_states = sp_all_gather(hidden_states)[:full_num_tokens]
return hidden_states