From 0b53bec60b9b41c9e16cefa2db367afb6a60628d Mon Sep 17 00:00:00 2001 From: Vincent Gimenes <147169146+VincentG1234@users.noreply.github.com> Date: Tue, 27 Jan 2026 04:05:02 +0100 Subject: [PATCH] [DOC]: Add warning about max_num_batched_tokens and max_model_len when chunked prefill is disabled (#33109) Signed-off-by: Vincent Gimenes <147169146+VincentG1234@users.noreply.github.com> --- docs/configuration/optimization.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/configuration/optimization.md b/docs/configuration/optimization.md index 556d9f8b94..80b12ae335 100644 --- a/docs/configuration/optimization.md +++ b/docs/configuration/optimization.md @@ -47,6 +47,10 @@ You can tune the performance by adjusting `max_num_batched_tokens`: - For optimal throughput, we recommend setting `max_num_batched_tokens > 8192` especially for smaller models on large GPUs. - If `max_num_batched_tokens` is the same as `max_model_len`, that's almost the equivalent to the V0 default scheduling policy (except that it still prioritizes decodes). +!!! warning + When chunked prefill is disabled, `max_num_batched_tokens` must be greater than `max_model_len`. + In that case, if `max_num_batched_tokens < max_model_len`, vLLM may crash at server start‑up. + ```python from vllm import LLM