[Bugfix][Prefix caching] Fix inconsistent hybrid Mamba prefix cache boundaries (#60533)
Signed-off-by: Wei Zhao <weizha@aws-cmh-slurm-1-login-01.cm.cluster> Signed-off-by: zjy0516 <riverclouds.zhu@qq.com> Co-authored-by: Wei Zhao <weizha@aws-cmh-slurm-1-login-01.cm.cluster> Co-authored-by: Codex <noreply@openai.com> Co-authored-by: zjy0516 <riverclouds.zhu@qq.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
This commit is contained in:
co-authored by
Wei Zhao
Codex
zjy0516
Claude Opus 5.5
mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
parent
1664c4f444
commit
d8fa463e2b
@@ -479,8 +479,8 @@ class BlockPool:
|
||||
kv_cache_group_id: KV cache group that owns the partial entry.
|
||||
block_size: Cache block size for the owning group. The partial
|
||||
entry hash itself is always the prefix-chain hash at
|
||||
``num_tokens``; ``block_size`` is used to assert that the
|
||||
entry is partial within the owning cache block.
|
||||
``num_tokens``; ``block_size`` must be a multiple of the hash
|
||||
block size.
|
||||
replace_existing_hashes: Whether the block contents were replaced
|
||||
and all existing cache entries must be removed before the new
|
||||
entry is registered.
|
||||
@@ -494,9 +494,6 @@ class BlockPool:
|
||||
return None
|
||||
|
||||
assert block_size % self.hash_block_size == 0
|
||||
assert replace_existing_hashes or (
|
||||
block_size > self.hash_block_size and num_tokens % block_size != 0
|
||||
)
|
||||
block_hash = self._get_partial_block_hash(request, num_tokens)
|
||||
num_hash_blocks = num_tokens // self.hash_block_size
|
||||
block_hash_with_group_id = make_block_hash_with_group_id(
|
||||
|
||||
Reference in New Issue
Block a user