[Model][DSv4.1] FlashMLA mega attention and the NVFP4 compressed KV cache (#56935)

Signed-off-by: Yongye Zhu <yongye@inferact.ai>
Signed-off-by: 云挚 <ningyunxiao.nyx@antgroup.com>
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com>
Co-authored-by: Yongye Zhu <yongye@inferact.ai>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Yunxiao Ning <73625538+foraxe@users.noreply.github.com>
Co-authored-by: AI Assistant <noreply@openai.com>
This commit is contained in:
Yongye Zhu
2026-09-16 03:07:50 -07:00
committed by GitHub
co-authored by Yongye Zhu Claude Opus 5 Yunxiao Ning AI Assistant
parent ceb87de065
commit d6a1677d55
20 changed files with 1811 additions and 156 deletions
+8 -1
View File
@@ -425,6 +425,12 @@ STABLE_TORCH_LIBRARY_FRAGMENT(_C, ops) {
"bool is_neox, Tensor position_ids, "
"int forced_token_heads_per_warp=-1) -> ()");
// q_head_padded is the padded Q head count of the returned tensor, or 0 to
// do the KV insert alone and return an empty tensor. The Q knobs are
// independent: apply_q_norm and apply_q_rope each drop that step for Q
// alone (KV is always rotated), and is_q_interleaved reads and writes Q in
// FlashMLA's mega-attention chunk-interleaved layout, which moves the
// padding heads to the tail of every head-dim chunk.
ops.def(
"fused_deepseek_v4_kv_rope_insert("
"Tensor kv, Tensor! k_cache, Tensor slot_mapping, Tensor position_ids, "
@@ -436,7 +442,8 @@ STABLE_TORCH_LIBRARY_FRAGMENT(_C, ops) {
"Tensor q_in, Tensor kv, Tensor! k_cache, "
"Tensor slot_mapping, Tensor position_ids, Tensor cos_sin_cache, "
"int q_head_padded, float eps, int cache_block_size, "
"bool apply_q_norm=True, bool kv_mxfp8=False) -> Tensor");
"bool apply_q_norm=True, bool kv_mxfp8=False, bool apply_q_rope=True, "
"bool is_q_interleaved=False) -> Tensor");
// FlashInfer V4 full-cache variants: write Q in place (bf16) or to a separate
// FP8 tensor, and KV into a contiguous 512-wide token-strided cache.