[3/N][KV-Cache Layout Refactor] Standardize Mamba cache; drop get_transfer_cache_regions (#44456)
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: OpenAI Codex <codex@openai.com>
This commit is contained in:
co-authored by
Claude
OpenAI Codex
parent
eb44b3aaa4
commit
6700813f86
@@ -533,9 +533,12 @@ def bind_kv_cache(
|
||||
for layer_name in layer_names:
|
||||
runner_kv_caches.append(kv_caches[layer_name])
|
||||
|
||||
# Bind kv_caches to forward context
|
||||
# Bind kv_caches to forward context. Each layer's bind_kv_cache unpacks
|
||||
# its raw allocation into the per-layer view(s) it needs (e.g. Mamba
|
||||
# splits conv/ssm), so the kv_caches dict can hold a single tensor per
|
||||
# layer for the KV connector to register.
|
||||
for layer_name, kv_cache in kv_caches.items():
|
||||
forward_context[layer_name].kv_cache = kv_cache
|
||||
forward_context[layer_name].bind_kv_cache(kv_cache)
|
||||
|
||||
|
||||
def copy_kv_cache_blocks_inplace(
|
||||
|
||||
Reference in New Issue
Block a user