[Bugfix] Don't split a multimodal item that a prefill chunk starts exactly on (#52412)
Signed-off-by: Taksh <takshkothari09@gmail.com> Signed-off-by: Nick Hill <nickhill123@gmail.com>
This commit is contained in:
@@ -1868,7 +1868,10 @@ class Scheduler(SchedulerInterface):
|
||||
# only cover part of the mm input, roll back to before the mm item.
|
||||
if (
|
||||
self.scheduler_config.disable_chunked_mm_input
|
||||
and num_computed_tokens < start_pos
|
||||
and not self.is_encoder_decoder
|
||||
# `<=`, not `<`: a chunk that starts exactly at the item still
|
||||
# splits it when it cannot cover the whole item.
|
||||
and num_computed_tokens <= start_pos
|
||||
and (num_computed_tokens + num_new_tokens)
|
||||
< (start_pos + num_encoder_tokens)
|
||||
):
|
||||
|
||||
Reference in New Issue
Block a user