[KV Cache] GLM-5.3-Flash: use the generic packed KV layout; CircularBufferSpec tail with speculative ring slots (#57169)
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai> Signed-off-by: Dakai An <dakaian108@gmail.com> Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com> Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai> Co-authored-by: OpenAI Codex <noreply@openai.com> Co-authored-by: OpenAI Codex <codex@openai.com> Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
This commit is contained in:
co-authored by
Yifan Qiao
OpenAI Codex
OpenAI Codex
Lucas Wilkinson
Claude Opus 5.5
Lucas Wilkinson
parent
3766a67a32
commit
02e55d2af3
@@ -29,10 +29,11 @@ _DEEPGEMM_BLACKWELL_EXCLUDED_MODEL_TYPES: set[str] = {
|
||||
"qwen3_5_moe_text",
|
||||
}
|
||||
|
||||
# KV page sizes (in cache entries) supported by the paged-MQA logits kernels
|
||||
# (fp8_fp4_paged_mqa_logits / get_paged_mqa_logits_metadata). Larger storage
|
||||
# blocks must be virtually split into one of these page sizes.
|
||||
PAGED_MQA_PAGE_SIZES = (32, 64)
|
||||
|
||||
def get_paged_mqa_page_sizes() -> tuple[int, ...]:
|
||||
"""KV page sizes (in cache entries) the paged-MQA logits kernels take (only
|
||||
64 on SM120); larger storage blocks are split into one of these."""
|
||||
return (64,) if current_platform.is_device_capability_family(120) else (32, 64)
|
||||
|
||||
|
||||
def should_auto_disable_deep_gemm(model_type: str | None) -> bool:
|
||||
|
||||
Reference in New Issue
Block a user