[BugFix] Fix num_output_placeholders preemption underflow (#48245)

Signed-off-by: Chris Eastwood <chris.eastwood@pwn4g3.dev>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: Chris Eastwood <chris.eastwood@pwn4g3.dev>
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai>
This commit is contained in:
Nick Hill
2026-07-29 06:41:54 -07:00
committed by GitHub
co-authored by Chris Eastwood Yifan Qiao
parent 43eaefba5a
commit a0c092ee72
5 changed files with 385 additions and 77 deletions
+6 -1
View File
@@ -149,7 +149,12 @@ class Request:
# Used in async scheduling.
self.num_output_placeholders = 0
self.async_tokens_to_discard = 0
# Tokens of output in flight when the request was preempted: delivered
# on return, but must not mutate the reset counters.
self.num_stale_output_tokens = 0
# Drop the stale output instead, for same-step preempt + resume
# (reset_prefix_cache).
self.drop_stale_output = False
# Tokens of steps whose output is not yet processed (async scheduling
# and PP run ahead of the GPU); `num_computed_tokens` counts them