[Model][DSv4.1] FlashMLA mega attention and the NVFP4 compressed KV cache (#56935)
Signed-off-by: Yongye Zhu <yongye@inferact.ai> Signed-off-by: 云挚 <ningyunxiao.nyx@antgroup.com> Signed-off-by: Yongye Zhu <zyy1102000@gmail.com> Co-authored-by: Yongye Zhu <yongye@inferact.ai> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Yunxiao Ning <73625538+foraxe@users.noreply.github.com> Co-authored-by: AI Assistant <noreply@openai.com>
This commit is contained in:
co-authored by
Yongye Zhu
Claude Opus 5
Yunxiao Ning
AI Assistant
parent
ceb87de065
commit
d6a1677d55
@@ -425,6 +425,12 @@ STABLE_TORCH_LIBRARY_FRAGMENT(_C, ops) {
|
||||
"bool is_neox, Tensor position_ids, "
|
||||
"int forced_token_heads_per_warp=-1) -> ()");
|
||||
|
||||
// q_head_padded is the padded Q head count of the returned tensor, or 0 to
|
||||
// do the KV insert alone and return an empty tensor. The Q knobs are
|
||||
// independent: apply_q_norm and apply_q_rope each drop that step for Q
|
||||
// alone (KV is always rotated), and is_q_interleaved reads and writes Q in
|
||||
// FlashMLA's mega-attention chunk-interleaved layout, which moves the
|
||||
// padding heads to the tail of every head-dim chunk.
|
||||
ops.def(
|
||||
"fused_deepseek_v4_kv_rope_insert("
|
||||
"Tensor kv, Tensor! k_cache, Tensor slot_mapping, Tensor position_ids, "
|
||||
@@ -436,7 +442,8 @@ STABLE_TORCH_LIBRARY_FRAGMENT(_C, ops) {
|
||||
"Tensor q_in, Tensor kv, Tensor! k_cache, "
|
||||
"Tensor slot_mapping, Tensor position_ids, Tensor cos_sin_cache, "
|
||||
"int q_head_padded, float eps, int cache_block_size, "
|
||||
"bool apply_q_norm=True, bool kv_mxfp8=False) -> Tensor");
|
||||
"bool apply_q_norm=True, bool kv_mxfp8=False, bool apply_q_rope=True, "
|
||||
"bool is_q_interleaved=False) -> Tensor");
|
||||
|
||||
// FlashInfer V4 full-cache variants: write Q in place (bf16) or to a separate
|
||||
// FP8 tensor, and KV into a contiguous 512-wide token-strided cache.
|
||||
|
||||
Reference in New Issue
Block a user