[Docs] Clarify admission control limits apply server-wide, not per DP rank (#55124)

Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>
This commit is contained in:
Nicolò Lucchesi
2026-09-07 14:08:48 +00:00
committed by GitHub
parent f2d45f26bd
commit cd64c2dea9
2 changed files with 14 additions and 1 deletions
+12 -1
View File
@@ -76,7 +76,13 @@ class SchedulerConfig:
at the same time, or None for no limit. When the limit is reached, new
requests are rejected with HTTP 503 so the client can retry on another
instance. This bounds vLLM's otherwise unbounded request queue and is
primarily a coarse capacity valve."""
primarily a coarse capacity valve.
Unlike ``max_num_seqs``, which applies per data-parallel rank, this
limit is enforced in the API server process and counts in-flight
requests across all DP ranks it routes to. Size it as roughly
``data_parallel_size * max_num_seqs`` plus the desired queue depth if
it should not bind before per-rank admission does."""
max_num_queued_tokens: int | None = Field(default=None, ge=0)
"""Maximum total prompt tokens of requests currently in the prefill
@@ -89,6 +95,11 @@ class SchedulerConfig:
prefill-decode setup this maps directly to the prefill pool's
capacity.
Like ``max_num_queued_reqs``, this limit is enforced in the API
server process and covers the prefill backlog across all DP ranks it
routes to, so ``prefill_throughput`` in the formula above is the
aggregate throughput of the deployment.
Note: the count is conservative. A partially prefilled request
still contributes its full ``prompt_len`` until it transitions out
of the prefill phase, because the scheduler's per-iteration