[Model][DSv4.1] Store the whole KV in MXFP8 (FlashMLA V4.1 record) (#56893)
Signed-off-by: Yongye Zhu <yongye@inferact.ai> Signed-off-by: Yongye Zhu <zyy1102000@gmail.com> Co-authored-by: Yongye Zhu <yongye@inferact.ai> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Yongye Zhu
Claude Opus 5
parent
a529c1a748
commit
e6eb0d120c
@@ -91,6 +91,17 @@ Bytes, structured as:
|
||||
The first `ue8m0` is the scale for the first 64 `float8_e4m3` values,
|
||||
the second for the next 64, and so on.
|
||||
|
||||
For DeepSeek V4.1, each token's KV cache is 528 Bytes, structured as:
|
||||
- **First 512 bytes:** all 512 dims as `float8_e4m3`. Unlike V4 the 64 RoPE
|
||||
dims are quantized too, so there is no `bfloat16` part.
|
||||
- **Last 16 bytes:** Scale factors, containing 16 `ue8m0` values, one per 32
|
||||
consecutive `float8_e4m3` values (i.e. MXFP8).
|
||||
|
||||
The V4 and V4.1 records are not laid out token-by-token within a page: a page
|
||||
holds all its data rows first and all its scale rows after, so a page is
|
||||
`block_size * bytes_per_token` bytes rounded up to the decode kernel's TMA
|
||||
stride (576 B for V4, 512 B for V4.1).
|
||||
|
||||
In the "nvfp4_ds_mla" format (SM100 only, DeepSeek V3.2 geometry), each
|
||||
token's KV cache is 352 Bytes, structured as:
|
||||
- **First 256 bytes:** 512 `e2m1` NoPE values packed 2/byte (low nibble =
|
||||
|
||||
Reference in New Issue
Block a user