[Bugfix][Prefix caching] Fix inconsistent hybrid Mamba prefix cache boundaries (#60533)
Signed-off-by: Wei Zhao <weizha@aws-cmh-slurm-1-login-01.cm.cluster> Signed-off-by: zjy0516 <riverclouds.zhu@qq.com> Co-authored-by: Wei Zhao <weizha@aws-cmh-slurm-1-login-01.cm.cluster> Co-authored-by: Codex <noreply@openai.com> Co-authored-by: zjy0516 <riverclouds.zhu@qq.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
This commit is contained in:
co-authored by
Wei Zhao
Codex
zjy0516
Claude Opus 5.5
mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
parent
1664c4f444
commit
d8fa463e2b
@@ -821,10 +821,15 @@ class Worker(WorkerBase):
|
||||
# so that it's available to the warmup stage.
|
||||
self.cache_config.num_gpu_blocks = kv_cache_config.num_blocks
|
||||
|
||||
# Adopt the engine core's layout; workers spawned after resolution
|
||||
# (e.g. elastic EP scale-up) only see it through the config.
|
||||
# Adopt the engine core's layout and prefix-cache granularity; workers
|
||||
# spawned after resolution (e.g. elastic EP scale-up) only see them
|
||||
# through the config.
|
||||
if kv_cache_config.kv_cache_layout is not None:
|
||||
record_kv_cache_layout(self.cache_config, kv_cache_config.kv_cache_layout)
|
||||
self.cache_config.hash_block_size = kv_cache_config.hash_block_size
|
||||
self.cache_config.cache_hit_alignment_tokens = (
|
||||
kv_cache_config.cache_hit_alignment_tokens
|
||||
)
|
||||
|
||||
# Init kv cache connector here, because it requires
|
||||
# `kv_cache_config`.
|
||||
|
||||
Reference in New Issue
Block a user