[DSA] Wire DeepGEMM sparse MQA logits into the DeepSeek V4.1 indexer (#56254)
Signed-off-by: Jared Wen <w13431838023@gmail.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5.1
parent
f2aad6aa70
commit
1257512305
@@ -86,6 +86,15 @@ class AttentionConfig:
|
||||
indexer). Quantized formats (fp8, mxfp4, nvfp4) require indexer kernel
|
||||
support in the backend."""
|
||||
|
||||
indexer_sparse_logits: bool = False
|
||||
"""DeepSeek V4.1 two-level indexer: score only the candidate blocks with
|
||||
DeepGEMM's sparse MQA-logits kernels instead of computing dense logits over
|
||||
the whole context and masking them. Requires `indexer_kv_dtype="mxfp4"`,
|
||||
an SM100-class GPU, DeepGEMM >= 2.8 and the DeepSelect top-k extension
|
||||
(the top-k runs on the kernels' bf16 logits). The sparse path costs
|
||||
O(candidate blocks) per query regardless of context length, so it pays off
|
||||
for long contexts (roughly 32K tokens and beyond) and is slower below."""
|
||||
|
||||
hisparse_config: HiSparseConfig | None = None
|
||||
"""HiSparse host-resident KV configuration. Setting this enables experimental
|
||||
Model Runner V2-only HiSparse sparse-MLA decode hot-buffering. It is inferred
|
||||
|
||||
Reference in New Issue
Block a user