[Bugfix] Treat generation_config max_tokens as default not ceiling (#34063)

Signed-off-by: almogtavor <almogtavor@gmail.com>
This commit is contained in:
Almog Tavor
2026-02-16 07:58:24 -08:00
committed by GitHub
parent a3205beffb
commit 72d5951d02
7 changed files with 157 additions and 17 deletions
@@ -1174,6 +1174,7 @@ class OpenAIServing:
context.request.max_output_tokens,
self._extract_prompt_len(engine_prompt),
self.default_sampling_params, # type: ignore
self.override_max_tokens, # type: ignore
)
# OPTIMIZATION