[ROCm][Perf] Enable CSA multi-stream overlap for DeepSeek-V4 (#51794)

Signed-off-by: shen-shanshan <467638484@qq.com>
Signed-off-by: Shanshan Shen <87969357+shen-shanshan@users.noreply.github.com>
This commit is contained in:
Shanshan Shen
2026-09-14 11:21:30 +00:00
committed by GitHub
parent e82794f48f
commit a6c5d6d0fc
4 changed files with 214 additions and 25 deletions
+2 -1
View File
@@ -198,11 +198,12 @@ class DeepseekV4CPUIndexer(DeepseekV4Indexer):
self,
hidden_states: torch.Tensor,
qr: torch.Tensor,
compressed_kv_score: torch.Tensor,
compressed_kv_score: torch.Tensor | None,
indexer_weights: torch.Tensor,
positions: torch.Tensor,
rotary_emb: nn.Module,
qr_scale: torch.Tensor | None = None,
skip_compressor: bool = False,
) -> tuple[torch.Tensor | None, torch.Tensor | None, torch.Tensor | None]:
"""CPU override: no aux streams, so wq_b_and_q_quant and the
compressor run straight-line instead of through