[KV Cache] GLM-5.3-Flash: use the generic packed KV layout; CircularBufferSpec tail with speculative ring slots (#57169)
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai> Signed-off-by: Dakai An <dakaian108@gmail.com> Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com> Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai> Co-authored-by: OpenAI Codex <noreply@openai.com> Co-authored-by: OpenAI Codex <codex@openai.com> Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
This commit is contained in:
co-authored by
Yifan Qiao
OpenAI Codex
OpenAI Codex
Lucas Wilkinson
Claude Opus 5.5
Lucas Wilkinson
parent
3766a67a32
commit
02e55d2af3
@@ -335,8 +335,8 @@ class EngineCore:
|
||||
vllm_config.cache_config.num_gpu_blocks = scheduler_kv_cache_config.num_blocks
|
||||
kv_cache_groups = scheduler_kv_cache_config.kv_cache_groups
|
||||
if kv_cache_groups:
|
||||
# Exclude groups that opt out of prefix caching (e.g. GLM-5.3-Flash
|
||||
# kpool tail, a 1-block/req scratch buffer with block_size=kpool):
|
||||
# Exclude groups that opt out of prefix caching (e.g. a circular
|
||||
# buffer, a 1-block/req scratch buffer with a small ring block):
|
||||
# their small block_size would otherwise drag the global block_size
|
||||
# below the real allocator block size and desync it from mamba.
|
||||
participating = [
|
||||
|
||||
Reference in New Issue
Block a user