[Core][Bugfix]: fix prefix caching for blockv2 (#5364)

Signed-off-by: Lei Wen <wenlei03@qiyi.com>
Co-authored-by: Lei Wen <wenlei03@qiyi.com>
This commit is contained in:
leiwen83
2024-06-14 17:23:56 -07:00
committed by GitHub
co-authored by Lei Wen
parent bd7efe95d0
commit 1b8a0d71cf
2 changed files with 72 additions and 2 deletions
+5 -2
View File
@@ -176,14 +176,17 @@ class PrefixCachingBlockAllocator(BlockAllocator):
self._refcounter.incr(block_id)
# the block comes from evictor already contain computed result
# Now this block is pop from evictor and ready to write
# with new content which most probably different with
# original content. So need to tell worker to recompute
# its kvcache
block = self._create_block(
prev_block=prev_block,
token_ids=[],
block_size=self._block_size,
allocator=self,
block_id=block_id,
computed=True,
computed=False,
)
assert block.content_hash is None