[3/N][KV-Cache Layout Refactor] Standardize Mamba cache; drop get_transfer_cache_regions (#44456)

Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
This commit is contained in:
Lucas Wilkinson
2026-07-21 09:16:15 +00:00
committed by GitHub
co-authored by Claude OpenAI Codex
parent eb44b3aaa4
commit 6700813f86
10 changed files with 115 additions and 225 deletions
+5 -2
View File
@@ -533,9 +533,12 @@ def bind_kv_cache(
for layer_name in layer_names:
runner_kv_caches.append(kv_caches[layer_name])
# Bind kv_caches to forward context
# Bind kv_caches to forward context. Each layer's bind_kv_cache unpacks
# its raw allocation into the per-layer view(s) it needs (e.g. Mamba
# splits conv/ssm), so the kv_caches dict can hold a single tensor per
# layer for the KV connector to register.
for layer_name, kv_cache in kv_caches.items():
forward_context[layer_name].kv_cache = kv_cache
forward_context[layer_name].bind_kv_cache(kv_cache)
def copy_kv_cache_blocks_inplace(