[Scheduler] Release a waiter's pin on the deferred free path too
The scheduler takes pop_blocks_for_free when a step may still be in flight, for example when a preempted request is aborted before its fence drained. A waiting request's pin is never written by any step, so it is now released there at once. Before, it stayed until the next yield or reset, keeping its blocks pinned and a waiter slot taken. pin_waiting_hit touches each block in the pool it belongs to; private pool groups route like free_blocks. Its docstring states the known gap: a pin is a snapshot of the hit at pin time. It is not refreshed if the hit grows while the request waits, or if a producer moves a pinned live block's hash into its copy. The admission lookup still sees whatever is cached, so the cost is a smaller hit, never a wrong one. MAX_PINNED_WAITERS goes from 8 to 16. Test: a pinned waiter popped through the deferred path leaves no pin, no waiter slot and the pool at its baseline. Co-Authored-By: Hinata <hinata@idihore.id> Signed-off-by: MOVZX <movzx@yahoo.com>
This commit is contained in:
@@ -649,8 +649,9 @@ class KVCacheManager:
|
||||
|
||||
# Waiting requests whose prefix-cache hit is pinned at once (arrival
|
||||
# order). Pins never block anyone (they yield before a running request
|
||||
# fails to allocate), so the bound only caps the lookup work per step.
|
||||
MAX_PINNED_WAITERS = 8
|
||||
# fails to allocate), so the bound only caps the lookup work per step;
|
||||
# twice the deployed max_num_seqs covers every waiter seen under load.
|
||||
MAX_PINNED_WAITERS = 16
|
||||
|
||||
@property
|
||||
def num_pinned_waiters(self) -> int:
|
||||
@@ -666,7 +667,15 @@ class KVCacheManager:
|
||||
prefix up to the key and the GDN state it resumes from) takes a ref
|
||||
and leaves the free queue. Idempotent. Returns the number of blocks
|
||||
pinned now (0 if nothing hit, already pinned, caching is off, or the
|
||||
waiter bound is reached)."""
|
||||
waiter bound is reached).
|
||||
|
||||
Known gap: a pin is a snapshot of the hit at pin time. It is not
|
||||
refreshed if the hit grows while the request waits (a sibling
|
||||
computing more of the same prefix) or if a producer moves the hash of
|
||||
a pinned live block into its CoW copy (a resume at a running
|
||||
producer's tail, unreachable under the EAGLE drop); the admission
|
||||
lookup still finds whatever is cached then, so the only cost is a
|
||||
smaller hit, never a wrong one."""
|
||||
request_id = request.request_id
|
||||
if (
|
||||
request_id in self._pinned_hits
|
||||
@@ -699,7 +708,10 @@ class KVCacheManager:
|
||||
if cached and cached[0].block_id not in seen:
|
||||
seen.add(cached[0].block_id)
|
||||
blocks.append(cached[0])
|
||||
self.block_pool.touch(blocks)
|
||||
# Blocks of a private-pool group (hisparse host tier) are touched in
|
||||
# their own pool, as free_blocks routes them.
|
||||
for block in blocks:
|
||||
(block.pool or self.block_pool).touch([block])
|
||||
self._pinned_hits[request_id] = blocks
|
||||
return len(blocks)
|
||||
|
||||
@@ -772,6 +784,9 @@ class KVCacheManager:
|
||||
reversed, then concatenated group by group).
|
||||
|
||||
"""
|
||||
# A pin is never written by an in-flight step (the waiter has no
|
||||
# table), so it is released now, not with the deferred blocks.
|
||||
self.release_pinned_hit(request.request_id)
|
||||
return self.coordinator.pop_blocks_for_free(request.request_id)
|
||||
|
||||
def evict_blocks(self, block_ids: set[int]) -> None:
|
||||
|
||||
Reference in New Issue
Block a user