[Mamba] Key the prompt tail when it lands on a block end

With a hash unit finer than the block, the prompt's tail key lands
exactly on a block end once every block/unit turns. The tail key is the
position the next turn resumes at: the last unit boundary, one unit
lower under the eagle drop. It is then a full-block key, so the
partial-tail registration does nothing, and under retention interval 0
the mask kept only the replay boundary one block below it. That state
was never hashed. The next turn therefore resumed one block lower after
a cold turn, or one turn lower (at the held source) after a partial-tail
resume, re-prefilling the previous turn's extension.

MambaManager.cache_blocks now adds that block-aligned tail to the
reachable boundaries, so the block ending there is hashed like any
boundary state, retired into the held checkpoint and freed last. At the
block unit the tail is the replay boundary itself and nothing changes.
When the hash unit is below the Mamba block, MambaManager logs one
startup line saying the block-aligned tail key is hashed.

Tests: a cold and a partial-tail-resume case whose tail lands on a block
end, under a concurrent burst, with the no-leak check. Both fail on the
previous commit (the tail state is not keyed and the next turn resumes
lower). A block-unit case asserts the hashed set is exactly the replay
boundary.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
This commit is contained in:
2026-10-09 20:16:37 +07:00
co-authored by Hinata
parent 944007d9e5
commit 95ba99ed29
@@ -1722,6 +1722,17 @@ class MambaManager(SingleTypeKVCacheManager):
self.block_pool.hash_block_size,
self.block_size,
)
if (
self.mamba_cache_mode == "align"
and self.enable_caching
and self.block_size > self.block_pool.hash_block_size
):
logger.info_once(
"Mamba align: a prompt tail key that lands on a block end is "
"hashed too (hash unit %d, block %d)",
self.block_pool.hash_block_size,
self.block_size,
)
def _retire_state_blocks(
self, request_id: str, retired: list[tuple[int, KVCacheBlock]]
@@ -2384,6 +2395,14 @@ class MambaManager(SingleTypeKVCacheManager):
replay_boundaries: Sequence[int],
) -> None:
num_cached_blocks_before = self.num_cached_block.get(request.request_id, 0)
tail = self._block_aligned_prompt_tail(request)
if tail:
# With a hash unit finer than the block, the prompt's tail
# key (where the next turn resumes; `_cache_partial_tail_block`)
# can land exactly on a block end. It is then a full-block key,
# not a partial one, and the replay-boundary mask alone (one block
# below it under the eagle drop) would never hash that state.
replay_boundaries = (*replay_boundaries, tail)
super().cache_blocks(
request,
num_tokens,
@@ -2425,6 +2444,22 @@ class MambaManager(SingleTypeKVCacheManager):
def new_step_starts(self) -> None:
self.cached_blocks_this_step.clear()
def _block_aligned_prompt_tail(self, request: Request) -> int:
"""The prompt's tail key when it falls on a block end, else 0.
Only with a hash unit finer than the block (at the block unit the
tail IS the replay boundary, which the mask already keeps). The key
is placed exactly as `_cache_partial_tail_block` places a sub-block
tail, eagle shift included.
"""
hash_block_size = self.block_pool.hash_block_size
if self.mamba_cache_mode != "align" or self.block_size <= hash_block_size:
return 0
tail = request.num_prompt_tokens // hash_block_size * hash_block_size
if self.drop_eagle_checkpoint_block:
tail = max(tail - hash_block_size, 0)
return tail if tail % self.block_size == 0 else 0
def _cache_partial_tail_block(
self,
request: Request,