[Mamba] Key the prompt tail when it lands on a block end
With a hash unit finer than the block, the prompt's tail key lands exactly on a block end once every block/unit turns. The tail key is the position the next turn resumes at: the last unit boundary, one unit lower under the eagle drop. It is then a full-block key, so the partial-tail registration does nothing, and under retention interval 0 the mask kept only the replay boundary one block below it. That state was never hashed. The next turn therefore resumed one block lower after a cold turn, or one turn lower (at the held source) after a partial-tail resume, re-prefilling the previous turn's extension. MambaManager.cache_blocks now adds that block-aligned tail to the reachable boundaries, so the block ending there is hashed like any boundary state, retired into the held checkpoint and freed last. At the block unit the tail is the replay boundary itself and nothing changes. When the hash unit is below the Mamba block, MambaManager logs one startup line saying the block-aligned tail key is hashed. Tests: a cold and a partial-tail-resume case whose tail lands on a block end, under a concurrent burst, with the no-leak check. Both fail on the previous commit (the tail state is not keyed and the next turn resumes lower). A block-unit case asserts the hashed set is exactly the replay boundary. Co-Authored-By: Hinata <hinata@idihore.id> Signed-off-by: MOVZX <movzx@yahoo.com>
This commit is contained in:
@@ -1722,6 +1722,17 @@ class MambaManager(SingleTypeKVCacheManager):
|
||||
self.block_pool.hash_block_size,
|
||||
self.block_size,
|
||||
)
|
||||
if (
|
||||
self.mamba_cache_mode == "align"
|
||||
and self.enable_caching
|
||||
and self.block_size > self.block_pool.hash_block_size
|
||||
):
|
||||
logger.info_once(
|
||||
"Mamba align: a prompt tail key that lands on a block end is "
|
||||
"hashed too (hash unit %d, block %d)",
|
||||
self.block_pool.hash_block_size,
|
||||
self.block_size,
|
||||
)
|
||||
|
||||
def _retire_state_blocks(
|
||||
self, request_id: str, retired: list[tuple[int, KVCacheBlock]]
|
||||
@@ -2384,6 +2395,14 @@ class MambaManager(SingleTypeKVCacheManager):
|
||||
replay_boundaries: Sequence[int],
|
||||
) -> None:
|
||||
num_cached_blocks_before = self.num_cached_block.get(request.request_id, 0)
|
||||
tail = self._block_aligned_prompt_tail(request)
|
||||
if tail:
|
||||
# With a hash unit finer than the block, the prompt's tail
|
||||
# key (where the next turn resumes; `_cache_partial_tail_block`)
|
||||
# can land exactly on a block end. It is then a full-block key,
|
||||
# not a partial one, and the replay-boundary mask alone (one block
|
||||
# below it under the eagle drop) would never hash that state.
|
||||
replay_boundaries = (*replay_boundaries, tail)
|
||||
super().cache_blocks(
|
||||
request,
|
||||
num_tokens,
|
||||
@@ -2425,6 +2444,22 @@ class MambaManager(SingleTypeKVCacheManager):
|
||||
def new_step_starts(self) -> None:
|
||||
self.cached_blocks_this_step.clear()
|
||||
|
||||
def _block_aligned_prompt_tail(self, request: Request) -> int:
|
||||
"""The prompt's tail key when it falls on a block end, else 0.
|
||||
|
||||
Only with a hash unit finer than the block (at the block unit the
|
||||
tail IS the replay boundary, which the mask already keeps). The key
|
||||
is placed exactly as `_cache_partial_tail_block` places a sub-block
|
||||
tail, eagle shift included.
|
||||
"""
|
||||
hash_block_size = self.block_pool.hash_block_size
|
||||
if self.mamba_cache_mode != "align" or self.block_size <= hash_block_size:
|
||||
return 0
|
||||
tail = request.num_prompt_tokens // hash_block_size * hash_block_size
|
||||
if self.drop_eagle_checkpoint_block:
|
||||
tail = max(tail - hash_block_size, 0)
|
||||
return tail if tail % self.block_size == 0 else 0
|
||||
|
||||
def _cache_partial_tail_block(
|
||||
self,
|
||||
request: Request,
|
||||
|
||||
Reference in New Issue
Block a user