[Perf] Async Scheduling + Speculative Decoding + Structured Outputs (#29821)

Signed-off-by: Benjamin Chislett <bchislett@nvidia.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
This commit is contained in:
Benjamin Chislett
2026-01-06 18:50:37 +00:00
committed by GitHub
co-authored by Nick Hill
parent 4e67a8f616
commit f7008ce1c4
8 changed files with 185 additions and 55 deletions
+1 -2
View File
@@ -158,12 +158,11 @@ class InputProcessor:
or params.presence_penalty != 0.0
or params.repetition_penalty != 1.0
or params.bad_words_token_ids
or params.structured_outputs
)
):
raise ValueError(
"async scheduling with spec decoding doesn't yet support "
"penalties, bad words or structured outputs in sampling parameters."
"penalties or bad words in sampling parameters."
)
def _validate_params(