[Perf][PP] Skip sampled-token broadcasts whose requests leave the engine (#58542)
Signed-off-by: LostFox11 <wangziyue17@huawei.com> Signed-off-by: Nick Hill <nickhill123@gmail.com> Co-authored-by: LostFox11 <wangziyue17@huawei.com> Co-authored-by: Nick Hill <nickhill123@gmail.com>
This commit is contained in:
co-authored by
LostFox11
Nick Hill
parent
34c458d27f
commit
7314360c9e
@@ -194,6 +194,7 @@ def _slice_input_batch(
|
||||
if input_batch.prompt_lens is None
|
||||
else input_batch.prompt_lens[req_start:req_stop]
|
||||
),
|
||||
max_seq_len_np=None,
|
||||
fast_prefill=None,
|
||||
)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user