[Model][DSv4.1] Store the whole KV in MXFP8 (FlashMLA V4.1 record) (#56893)

Signed-off-by: Yongye Zhu <yongye@inferact.ai>
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com>
Co-authored-by: Yongye Zhu <yongye@inferact.ai>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Yongye Zhu
2026-09-14 20:12:55 -07:00
committed by GitHub
co-authored by Yongye Zhu Claude Opus 5
parent a529c1a748
commit e6eb0d120c
14 changed files with 709 additions and 110 deletions
@@ -91,6 +91,17 @@ Bytes, structured as:
The first `ue8m0` is the scale for the first 64 `float8_e4m3` values,
the second for the next 64, and so on.
For DeepSeek V4.1, each token's KV cache is 528 Bytes, structured as:
- **First 512 bytes:** all 512 dims as `float8_e4m3`. Unlike V4 the 64 RoPE
dims are quantized too, so there is no `bfloat16` part.
- **Last 16 bytes:** Scale factors, containing 16 `ue8m0` values, one per 32
consecutive `float8_e4m3` values (i.e. MXFP8).
The V4 and V4.1 records are not laid out token-by-token within a page: a page
holds all its data rows first and all its scale rows after, so a page is
`block_size * bytes_per_token` bytes rounded up to the decode kernel's TMA
stride (576 B for V4, 512 B for V4.1).
In the "nvfp4_ds_mla" format (SM100 only, DeepSeek V3.2 geometry), each
token's KV cache is 352 Bytes, structured as:
- **First 256 bytes:** 512 `e2m1` NoPE values packed 2/byte (low nibble =