[Bugfix] Defer block freeing until in-flight steps finish under async scheduling + PD KV consumer (#45357)
Signed-off-by: llx-08 <2596671364@qq.com> Signed-off-by: Nick Hill <nickhill123@gmail.com> Co-authored-by: Nick Hill <nickhill123@gmail.com> Co-authored-by: Jiangyun Zhu <riverclouds.zhu@qq.com>
This commit is contained in:
co-authored by
Nick Hill
Jiangyun Zhu
parent
76a373eff4
commit
d467a2a7f2
@@ -145,6 +145,10 @@ class Request:
|
||||
# so the worker's broadcast slot ring stays consistent.
|
||||
self.next_decode_eligible_step = 0
|
||||
|
||||
# Seq of the most recent step this request was scheduled in; fences
|
||||
# deferred block freeing (see Scheduler._free_request_blocks).
|
||||
self.last_sched_seq = 0
|
||||
|
||||
self.spec_token_ids: list[int] = []
|
||||
self.num_computed_tokens = 0
|
||||
self.cache_salt: str | None = cache_salt
|
||||
|
||||
Reference in New Issue
Block a user