[Perf][MRV2] Allow FULL decode graphs for one-token prompt tails (#58400)

Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Nicolò Lucchesi <nicolo.lucchesi@mistral.ai>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com>
This commit is contained in:
Nick Hill
2026-09-28 12:48:52 -07:00
committed by GitHub
co-authored by Nicolò Lucchesi Claude Opus 5.5 mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Yongye Zhu
parent ef42093a36
commit e7c903609f
12 changed files with 144 additions and 42 deletions
+5
View File
@@ -181,6 +181,11 @@ def _slice_input_batch(
req_start:req_stop
],
is_prefilling_np=input_batch.is_prefilling_np[req_start:req_stop],
prefill_runs_as_decode_np=(
None
if input_batch.prefill_runs_as_decode_np is None
else input_batch.prefill_runs_as_decode_np[req_start:req_stop]
),
input_ids=input_batch.input_ids[tok_start:tok_stop],
positions=input_batch.positions[tok_start:tok_stop],
is_padding=input_batch.is_padding[tok_start:tok_stop],