[Bugfix][KV Offload] Stop offloading the final sampled token's KV slot (#54288)

Signed-off-by: almogtavor <almogtavor@gmail.com>
Signed-off-by: Almog Tavor <70065337+almogtavor@users.noreply.github.com>
Co-authored-by: Or Ozeri <or@ozery.com>
This commit is contained in:
Almog Tavor
2026-09-06 15:22:19 +03:00
committed by GitHub
co-authored by Or Ozeri
parent 569adb5a97
commit dc02934a07
2 changed files with 40 additions and 31 deletions
@@ -1384,7 +1384,10 @@ class OffloadingConnectorScheduler:
if req.status is RequestStatus.FINISHED_ABORTED:
num_tokens_after_batch = req.num_computed_tokens
elif req.is_finished():
num_tokens_after_batch = req.num_tokens
# Clamp to the GPU prefix cache's commit point. The final sampled
# token's slot is never committed (under spec decode it holds a
# rejected draft's KV), so a block ending there must not be stored.
num_tokens_after_batch = max(req.num_prompt_tokens, req.num_tokens - 1)
else:
num_scheduled_tokens = scheduler_output.num_scheduled_tokens[req_id]
num_tokens_after_batch = req.num_computed_tokens + num_scheduled_tokens