[Model][DSv4.1] Store the whole KV in MXFP8 (FlashMLA V4.1 record) (#56893)
Signed-off-by: Yongye Zhu <yongye@inferact.ai> Signed-off-by: Yongye Zhu <zyy1102000@gmail.com> Co-authored-by: Yongye Zhu <yongye@inferact.ai> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Yongye Zhu
Claude Opus 5
parent
a529c1a748
commit
e6eb0d120c
@@ -259,6 +259,8 @@ def _insert_context_kv(
|
||||
attn.padded_heads,
|
||||
attn.eps,
|
||||
block_size,
|
||||
True, # apply_q_norm (unused: the query is a discarded dummy)
|
||||
attn.kv_mxfp8,
|
||||
)
|
||||
elif cache_dtype == torch.bfloat16:
|
||||
swa_3d = swa_cache.view(-1, block_size, attn.head_dim)
|
||||
|
||||
Reference in New Issue
Block a user