[Bugfix][PD] Pad resumed speculative decode requests (#55126)
Signed-off-by: Ziming Huang <48115868+ZeldaHuang@users.noreply.github.com> Co-authored-by: OpenAI Codex <codex@openai.com>
This commit is contained in:
co-authored by
OpenAI Codex
parent
25268f0c9c
commit
a26b71d868
@@ -973,7 +973,8 @@ class Scheduler(SchedulerInterface):
|
||||
(self.num_spec_tokens > 0 and self.dynamic_sd_lookup is None)
|
||||
and self.num_sampled_tokens_per_step > 0
|
||||
and num_new_tokens == 1
|
||||
and (scheduled_running_reqs and not prefill_scheduled)
|
||||
and not prefill_scheduled
|
||||
and (scheduled_running_reqs or num_computed_tokens > 0)
|
||||
):
|
||||
padded_num_tokens = 1 + self.num_spec_tokens
|
||||
# Pad only when there is room for the sampled token(s).
|
||||
|
||||
Reference in New Issue
Block a user