[Mamba] Hold the partial-tail CoW checkpoint until the request is freed

With --prefix-match-unit below the block size, a follow-up turn that
resumes at a partial tail (mid-block) never hashes a block-boundary GDN
state. Under retention interval 0 the only boundary kept is the replay
boundary, which lies below the resume point. The request's own tail is
published on its running block and moved into a CoW block by the next
chunk, and the scheduler releases both copy endpoints once the copy has
run. That block then sat at ref 0 in the free queue for the rest of the
request, ahead of its attention blocks, and concurrent prefills popped
it. The hybrid hit is the shortest group's, so the next turn resumed at
0. Under a long-context multi-turn agent load, 7 of 61 follow-ups at
unit 64 (6 at 256/512/1024) came back with cached_tokens 0 on 80-100K
prompts whose whole attention prefix was still cached. Every one
followed a partial-tail resume, none followed a block-aligned one, and
unit 4096 never missed.

The existing retention (VLLM_E2_MAMBA_RETAIN_CHECKPOINT) only held
hashed blocks retired from the table, so it never covered that copy.
Now, under the same switch:

- The running-request CoW branch holds the moved tail block in
  _retained_checkpoints with a ref of its own; the copy retention pair
  is unchanged. The new-request branch holds the hit source after
  _apply_cow for the same reason: a request that stops before its own
  tail still leaves the state it resumed from.
- _retained_checkpoints is keyed by block_hash_num_tokens instead of
  block index, so a sub-block tail key orders correctly against full
  block states; _retire_state_blocks uses the same key. The newest
  position supersedes and releases the older one, so a request still
  holds at most one block per group.
- When a producer moves its tail hashes into its CoW block, another
  request holding that source (a sibling that resumed at the tail while
  it was the producer's live block) follows the hashes to the copy.
- Admission reserves one extra block per Mamba group for a partial-tail
  hit: the held source and the moved tail coexist at the tail step.

The hold is released on every free path (finish, abort, preemption,
deferred pop_blocks_for_free) by the coordinator, last, after the
request's attention blocks, as before. When the hold is active (switch
on, hash unit below the Mamba block) MambaManager logs one startup line
saying so, with the hash unit and block size.

Tests (CPU): a replay of the miss that fails on v2.5.0 with the
retention switch on (next-turn hit 0 instead of the tail) and passes
here; the structural variant; admission accounting; no-leak checks for
a request freed with its copy still queued, in both release orders; and
the moved-hash follow. The retention suite's one block-index assertion
now reads the token position.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
This commit is contained in:
2026-10-09 20:16:36 +07:00
co-authored by Hinata
parent db2851e54a
commit 944007d9e5
2 changed files with 104 additions and 12 deletions
+4 -1
View File
@@ -253,7 +253,10 @@ class KVCacheCoordinator(ABC):
# the free queue that check reads.
num_blocks_to_allocate += (
manager.get_num_retained_checkpoint_blocks_to_reserve(
request_id, num_tokens_main_model, new_computed_blocks[i]
request_id,
num_tokens_main_model,
new_computed_blocks[i],
num_local_computed_tokens,
)
)
return num_blocks_to_allocate
+100 -11
View File
@@ -622,6 +622,7 @@ class SingleTypeKVCacheManager(ABC):
request_id: str,
num_tokens: int,
new_computed_blocks: Sequence[KVCacheBlock] = (),
num_local_computed_tokens: int = 0,
) -> int:
"""Blocks the admission gate must add for checkpoints this request
will hold outside its block table (0 for every manager but align-mode
@@ -1685,7 +1686,24 @@ class MambaManager(SingleTypeKVCacheManager):
and self.enable_caching
and envs.VLLM_E2_MAMBA_RETAIN_CHECKPOINT
)
# {request_id: (logical block index, block)}
# {request_id: (hashed token position, block)}: the newest hashed
# state the request resumed from or produced, held with one ref of
# its own (ordered by ``block_hash_num_tokens`` rather than
# block index so a sub-block partial-tail key sorts correctly against
# full-block boundary states).
#
# With ``--prefix-match-unit`` below the block size a request
# that resumed at a partial tail (mid-block) never hashes a boundary
# state: the replay boundary lies below its resume point. Its only
# reusable GDN state is its own partial-tail checkpoint, which the
# next chunk moves into a CoW block in no block table; the scheduler
# drops the copy's retention refs once the copy has run, so that
# block sat at ref 0 in the free queue for the rest of the request,
# ahead of its attention blocks, and concurrent prefills evicted it
# (measured: 7/61 follow-ups with cached_tokens 0 at unit 64, every
# one after a partial-tail resume; none at unit 4096). The CoW paths
# therefore hold their hashed endpoint here too: the moved tail for
# a running request, and the hit source a new request copies from.
self._retained_checkpoints: dict[str, tuple[int, KVCacheBlock]] = {}
if self.mamba_cache_mode == "align" and self.enable_caching:
# The A/B evidence line: boot logs must show which arm ran.
@@ -1694,6 +1712,16 @@ class MambaManager(SingleTypeKVCacheManager):
"(VLLM_E2_MAMBA_RETAIN_CHECKPOINT)",
"on" if self._retain_checkpoint else "off",
)
if (
self._retain_checkpoint
and self.block_size > self.block_pool.hash_block_size
):
logger.info_once(
"Mamba align: each request holds its partial-tail CoW checkpoint "
"until it is freed (hash unit %d, block %d)",
self.block_pool.hash_block_size,
self.block_size,
)
def _retire_state_blocks(
self, request_id: str, retired: list[tuple[int, KVCacheBlock]]
@@ -1708,13 +1736,16 @@ class MambaManager(SingleTypeKVCacheManager):
"""
if self._retain_checkpoint:
newest: tuple[int, KVCacheBlock] | None = None
for idx, block in retired:
if block.block_hash is not None and (
newest is None or idx > newest[0]
):
newest = (idx, block)
for _, block in retired:
if block.block_hash is None:
continue
position = block.block_hash_num_tokens
assert position is not None
if newest is None or position > newest[0]:
newest = (position, block)
held = self._retained_checkpoints.get(request_id)
if newest is not None and (held is None or newest[0] > held[0]):
# The retired block's table ref becomes the hold's ref.
self._retained_checkpoints[request_id] = newest
retired = [entry for entry in retired if entry[1] is not newest[1]]
if held is not None:
@@ -1723,6 +1754,43 @@ class MambaManager(SingleTypeKVCacheManager):
if retired:
self.block_pool.free_blocks([block for _, block in retired])
def _hold_checkpoint(self, request_id: str, block: KVCacheBlock) -> bool:
"""Hold ``block`` (a hashed state outside the request's table) as the
request's prefix checkpoint, taking one extra ref for the hold.
Only the newest position is held: an older one is left alone (it ages
from wherever it is), a superseded one is released here. The hold's
ref is returned by ``pop_retained_checkpoints`` -> ``free_blocks``.
Returns whether the block is now held.
"""
if not self._retain_checkpoint or block.is_null or block.block_hash is None:
return False
position = block.block_hash_num_tokens
assert position is not None
held = self._retained_checkpoints.get(request_id)
if held is not None and (held[1] is block or held[0] >= position):
return False
self.block_pool.touch([block])
self._retained_checkpoints[request_id] = (position, block)
if held is not None:
self.block_pool.free_blocks([held[1]])
return True
def _follow_moved_hash(
self, owner_id: str, source_block: KVCacheBlock, cow_block: KVCacheBlock
) -> None:
"""``owner_id`` just moved ``source_block``'s hashes to ``cow_block``
and keeps writing ``source_block``. A request that resumed at that
tail while it was still the owner's live block holds the source for
those hashes; move its hold to the copy, so it pins the hashed state
and not another request's running block.
"""
for request_id, (position, block) in list(self._retained_checkpoints.items()):
if request_id != owner_id and block is source_block:
self.block_pool.touch([cow_block])
self._retained_checkpoints[request_id] = (position, cow_block)
self.block_pool.free_blocks([source_block])
def pop_retained_checkpoints(self, request_id: str) -> list[KVCacheBlock]:
if self.mamba_cache_mode != "align":
return []
@@ -1734,16 +1802,25 @@ class MambaManager(SingleTypeKVCacheManager):
request_id: str,
num_tokens: int,
new_computed_blocks: Sequence[KVCacheBlock] = (),
num_local_computed_tokens: int = 0,
) -> int:
# A request that already holds a checkpoint only ever swaps it. One
# that does not will hold one once a hashed state retires, which needs
# at least one state block behind the running one.
if (
not self._retain_checkpoint
or request_id in self._retained_checkpoints
or num_tokens <= self.block_size
):
if not self._retain_checkpoint or request_id in self._retained_checkpoints:
return 0
# A sub-block (partial-tail) hit holds the hit source only
# until the request's own tail is moved into a CoW block that is held
# in its place; at that step both are referenced, so the request's
# lifetime peak is one block above what the hit brought (whatever the
# prompt length: the tail CoW happens inside the first block too).
if self._has_partial_local_hit(new_computed_blocks, num_local_computed_tokens):
return 1
if num_tokens <= self.block_size:
# No state block retires behind the running one, but a prompt
# whose tail is keyed inside the block still moves that tail into
# a held CoW block when the hash unit is finer than the block.
return int(self.block_pool.hash_block_size < self.block_size)
# A prefix hit already brings the block that becomes the held
# checkpoint -- the hit state, counted by get_num_blocks_to_allocate as
# an evictable computed block (or already referenced elsewhere) -- and
@@ -2195,6 +2272,7 @@ class MambaManager(SingleTypeKVCacheManager):
# overwrites source_block.
assert req_blocks[block_idx] is source_block
self.block_pool.move_block_hashes(source_block, cow_block)
self._follow_moved_hash(request_id, source_block, cow_block)
self._pending_cow_copies.append((source_block, cow_block))
source_block.ref_cnt += 1
producer_tail = self._producer_partial_tail_reqs.pop(
@@ -2218,9 +2296,20 @@ class MambaManager(SingleTypeKVCacheManager):
# The moved entry is only filled by this step's
# copy, so defer same-step hits on it.
self.cached_blocks_this_step.add(cow_block.block_hash)
# The moved tail is the request's newest
# reusable state and lives in no block table; hold it
# past the copy's retention so the request's next
# turn can resume at it (see __init__).
self._hold_checkpoint(request_id, cow_block)
else:
self._apply_cow(request_id, block_idx, source_block, cow_block)
returned_blocks = [cow_block] + returned_blocks
# The hit source keeps the hash while the
# request advances its private copy; hold it until
# the request's own tail supersedes it, so a request
# that stops early (preempted, aborted, block-aligned
# prompt) still leaves the state it resumed from.
self._hold_checkpoint(request_id, source_block)
req_blocks.extend(new_blocks)
self._allocated_block_reqs.add(request_id)
self._partial_hit_reqs.pop(request_id, None)