[KV Cache] GLM-5.3-Flash: use the generic packed KV layout; CircularBufferSpec tail with speculative ring slots (#57169)
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai> Signed-off-by: Dakai An <dakaian108@gmail.com> Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com> Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai> Co-authored-by: OpenAI Codex <noreply@openai.com> Co-authored-by: OpenAI Codex <codex@openai.com> Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
This commit is contained in:
co-authored by
Yifan Qiao
OpenAI Codex
OpenAI Codex
Lucas Wilkinson
Claude Opus 5.5
Lucas Wilkinson
parent
3766a67a32
commit
02e55d2af3
@@ -432,7 +432,7 @@ class DFlashSpeculator(DraftModelSpeculator):
|
||||
temperature,
|
||||
seeds,
|
||||
self.block_tables.input_block_tables[gid],
|
||||
self.block_tables.kernel_block_sizes[gid],
|
||||
self.block_tables.block_sizes[gid],
|
||||
self.block_tables.cp_rank,
|
||||
self.dcp_size,
|
||||
self.block_tables.cp_interleave,
|
||||
|
||||
@@ -577,6 +577,11 @@ class TargetDependentARSpeculator(DraftModelSpeculator):
|
||||
idx_mapping = self.idx_mapping[:num_reqs]
|
||||
|
||||
if batch_desc.cg_mode == CUDAGraphMode.FULL:
|
||||
for groups in self.attn_groups:
|
||||
for group in groups:
|
||||
group.map_to_kernel_block_table(
|
||||
self.block_tables.input_block_tables[group.kv_cache_group_id]
|
||||
)
|
||||
assert self.decode_cudagraph_manager is not None
|
||||
self.decode_cudagraph_manager.run_fullgraph(batch_desc)
|
||||
return
|
||||
|
||||
Reference in New Issue
Block a user