[Attention] Skip sparse indexer scoring for dense short prefills (#48407)
Signed-off-by: Yiliu Dong <91178480+qianlihuang@users.noreply.github.com> Co-authored-by: OpenAI Codex <codex@openai.com>
This commit is contained in:
co-authored by
OpenAI Codex
parent
6453fc0b8c
commit
ba702e978e
@@ -1178,6 +1178,9 @@ class DeepseekV2MLAAttention(nn.Module):
|
||||
# the V1 proposer. A frozen True would leave the draft reading a
|
||||
# never-written topk buffer.
|
||||
skip_topk=_skip_topk and not is_mtp_layer,
|
||||
# Do not skip scoring for MTP layers: their top-k buffer may be
|
||||
# reused by later draft iterations through index sharing.
|
||||
allow_short_prefill_indexer_scoring_skip=not is_mtp_layer,
|
||||
)
|
||||
|
||||
def forward(
|
||||
|
||||
Reference in New Issue
Block a user