[Perf][MRV2] Allow FULL decode graphs for one-token prompt tails (#58400)
Signed-off-by: Nick Hill <nickhill123@gmail.com> Co-authored-by: Nicolò Lucchesi <nicolo.lucchesi@mistral.ai> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Yongye Zhu <zyy1102000@gmail.com>
This commit is contained in:
co-authored by
Nicolò Lucchesi
Claude Opus 5.5
mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Yongye Zhu
parent
ef42093a36
commit
e7c903609f
@@ -181,6 +181,11 @@ def _slice_input_batch(
|
||||
req_start:req_stop
|
||||
],
|
||||
is_prefilling_np=input_batch.is_prefilling_np[req_start:req_stop],
|
||||
prefill_runs_as_decode_np=(
|
||||
None
|
||||
if input_batch.prefill_runs_as_decode_np is None
|
||||
else input_batch.prefill_runs_as_decode_np[req_start:req_stop]
|
||||
),
|
||||
input_ids=input_batch.input_ids[tok_start:tok_stop],
|
||||
positions=input_batch.positions[tok_start:tok_stop],
|
||||
is_padding=input_batch.is_padding[tok_start:tok_stop],
|
||||
|
||||
Reference in New Issue
Block a user