[Attention] Skip sparse indexer scoring for dense short prefills (#48407)

Signed-off-by: Yiliu Dong <91178480+qianlihuang@users.noreply.github.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
This commit is contained in:
Yiliu Dong
2026-07-28 16:17:22 +00:00
committed by GitHub
co-authored by OpenAI Codex
parent 6453fc0b8c
commit ba702e978e
7 changed files with 230 additions and 12 deletions
@@ -1178,6 +1178,9 @@ class DeepseekV2MLAAttention(nn.Module):
# the V1 proposer. A frozen True would leave the draft reading a
# never-written topk buffer.
skip_topk=_skip_topk and not is_mtp_layer,
# Do not skip scoring for MTP layers: their top-k buffer may be
# reused by later draft iterations through index sharing.
allow_short_prefill_indexer_scoring_skip=not is_mtp_layer,
)
def forward(