Merge branch 'main' into patches

Signed-off-by: MOVZX <movzx@yahoo.com>
This commit is contained in:
2026-10-15 21:16:39 +07:00
1693 changed files with 99919 additions and 32440 deletions
+1
View File
@@ -26,6 +26,7 @@ nav:
- models/supported_models.md
- models/generative_models.md
- Pooling Models: models/pooling_models
- models/structured_decisions_models.md
- models/extensions
- Hardware Supported Models:
- models/hardware_supported_models/*
+30 -2
View File
@@ -9,7 +9,7 @@ vllm --help
Available Commands:
```bash
vllm {chat,complete,serve,launch,bench,collect-env,run-batch,preload}
vllm {chat,complete,serve,launch,bench,collect-env,download-kernels,run-batch,preload}
```
## serve
@@ -131,9 +131,22 @@ To use benchmark commands, please install with extra dependencies using `pip ins
Available Commands:
```bash
vllm bench {latency, serve, throughput}
vllm bench {forward, latency, serve, throughput}
```
### forward
Benchmark the model forward of exact batches: each case lists its requests as `[N*]Q[@K][c][:D]` (query length Q, KV length K including the query, D draft tokens). A prefill's prefix is a prefix-cache hit unless `c` marks a chunked prefill; they differ only on models with SWA bounded replay, which recompute the last window of a hit. Each request's prefix length varies by ±50% by default (`--kv-len-jitter`), identically on every DP rank. Each forward is captured as a one-shot CUDA graph; sampling and CPU overhead aren't measured. Use `torchrun`, one process per GPU, for DP/TP/EP. CUDA only for now.
```bash
vllm bench forward \
--model meta-llama/Llama-3.2-1B-Instruct \
--case decode=32*1@4096 \
--case mix=2048,32*1@4096
```
See [vllm bench forward](./bench/forward.md) for the full reference of all available arguments.
### latency
Benchmark the latency of a single batch of requests.
@@ -188,6 +201,21 @@ Start collecting environment information.
vllm collect-env
```
## download-kernels
Install the precompiled kernels that vLLM's dependencies would otherwise download
or compile at startup. Run it after installing or upgrading vLLM. It currently
installs FlashInfer's kernels and does nothing when FlashInfer is not installed.
```bash
vllm download-kernels
# Print the install commands without running them
vllm download-kernels --dry-run
```
See [vllm download-kernels](./download-kernels.md) for the full reference of all available arguments.
## run-batch
Run batch prompts and write results to file.
+12 -6
View File
@@ -41,6 +41,7 @@ If you are only developing vLLM's Python code, install vLLM using:
```bash
VLLM_USE_PRECOMPILED=1 uv pip install -e .
vllm download-kernels
```
To rebuild only the Rust frontend binary:
@@ -328,9 +329,15 @@ review process:
in the upstream repository. Your branch must contain every commit currently
on that branch: **zero commits behind**. This also applies to PRs targeting
release branches or another PR's branch.
- If the PR is behind, the bot reports the count without creating a new build.
Merge or rebase onto the latest target branch, then rerun the command. If the
target branch advances in the meantime, update your branch and try again.
- If the PR is behind, the bot merges the target branch into your PR branch
(like the **Update branch** button) and runs CI on the resulting merge commit.
Pull the branch before pushing more changes. This needs **Allow edits by
maintainers** enabled on fork PRs. If the merge conflicts or is not allowed,
the bot reports the count without creating a new build; merge or rebase onto
the latest target branch yourself, then rerun the command.
- PR authors without write access cannot start another full run (including
`--allow-stale`) on a commit that already has a CI build. Push new changes,
use `/ci retry` to rerun failed jobs, or ask a reviewer with write access.
- To run CI on an outdated branch at your own risk, append `--allow-stale` to
any run command, for example `/ci run --allow-stale` or
`/amd-ci run all --allow-stale`. The same authorization requirements apply.
@@ -338,9 +345,8 @@ review process:
failures. Before merging, merge or rebase onto the latest target branch and
rerun CI without `--allow-stale` on the latest PR commit.
- Retrying or cancelling builds does not check branch freshness.
- These commands do not modify your branch or enforce merge requirements.
Direct Buildkite launches are outside this workflow. No additional token is
needed.
- Apart from that merge, these commands do not modify your branch or enforce
merge requirements. Direct Buildkite launches are outside this workflow.
Maintainers must configure required Buildkite status checks on the latest PR
commit and enable GitHub's
+8 -8
View File
@@ -8,11 +8,11 @@ It is a [CNCF Sandbox project](https://www.cncf.io/blog/2026/03/24/welcome-llm-d
A single vLLM server is fast, but at scale the picture changes: across many replicas, cache locality breaks under round-robin load balancing, long prompts inflate time-to-first-token, and accelerators sit underused. llm-d adds the cluster-level layer that vLLM does not aim to provide on its own:
- **[Prefix-aware routing](https://llm-d.ai/docs/guides/precise-prefix-cache-aware).** Instead of round-robin, llm-d reads vLLM's KV-cache events and routes each request to the replica that already holds its prefix, reusing cache instead of recomputing it.
- **[Distributed KV-cache management](https://llm-d.ai/docs/guides#advanced-kv-cache-management).** A global index tracks which token blocks live on which replica, and [tiered offloading](https://llm-d.ai/docs/guides/tiered-prefix-cache) spills cache to CPU memory or local SSD, extending the working set beyond accelerator HBM.
- **[Prefill/decode disaggregation](https://llm-d.ai/docs/guides/pd-disaggregation).** Prompt processing and token generation run on separate vLLM workers, with KV-cache moved over the vLLM [NIXL connector](https://docs.vllm.ai/en/latest/features/nixl_connector_usage/), lowering TTFT and steadying per-token latency on long prompts.
- **[Wide expert-parallelism](https://llm-d.ai/docs/guides/wide-expert-parallelism).** Serve large Mixture-of-Experts models such as DeepSeek-R1 and GPT-OSS across nodes with combined data and expert parallelism, for more KV-cache capacity and throughput.
- **SLO-aware [autoscaling](https://llm-d.ai/docs/guides/workload-autoscaling) and [flow control](https://llm-d.ai/docs/guides/flow-control).** Scale vLLM pools on real inference signals (queue depth, true demand) rather than raw GPU utilization, with multi-tenant fairness and priority dispatch.
- **[Prefix-aware routing](https://llm-d.ai/docs/well-lit-paths/foundations/precise-prefix-cache-routing).** Instead of round-robin, llm-d reads vLLM's KV-cache events and routes each request to the replica that already holds its prefix, reusing cache instead of recomputing it.
- **[Distributed KV-cache management](https://llm-d.ai/docs/architecture/advanced/kv-management).** A global index tracks which token blocks live on which replica, and [tiered offloading](https://llm-d.ai/docs/well-lit-paths/foundations/tiered-prefix-cache) spills cache to CPU memory or local SSD, extending the working set beyond accelerator HBM.
- **[Prefill/decode disaggregation](https://llm-d.ai/docs/well-lit-paths/foundations/pd-disaggregation).** Prompt processing and token generation run on separate vLLM workers, with KV-cache moved over the vLLM [NIXL connector](https://docs.vllm.ai/en/latest/features/nixl_connector_usage/), lowering TTFT and steadying per-token latency on long prompts.
- **[Wide expert-parallelism](https://llm-d.ai/docs/well-lit-paths/foundations/wide-expert-parallelism).** Serve large Mixture-of-Experts models such as DeepSeek-R1 and GPT-OSS across nodes with combined data and expert parallelism, for more KV-cache capacity and throughput.
- **SLO-aware [autoscaling](https://llm-d.ai/docs/well-lit-paths/foundations/workload-autoscaling) and [flow control](https://llm-d.ai/docs/well-lit-paths/foundations/flow-control).** Scale vLLM pools on real inference signals (queue depth, true demand) rather than raw GPU utilization, with multi-tenant fairness and priority dispatch.
These are composable. Most teams start by adding prefix-aware routing over an existing vLLM pool, then layer in the rest as specific bottlenecks appear.
@@ -28,9 +28,9 @@ See the [full list](https://github.com/llm-d/llm-d#performance-highlights) and r
## Get started
1. Deploy the [Optimized Baseline](https://llm-d.ai/docs/guides/optimized-baseline) with the [Quickstart](https://llm-d.ai/docs/getting-started/quickstart). It stands up an intelligent router over a vLLM pool on Kubernetes in a tested configuration.
2. Browse the [well-lit path guides](https://llm-d.ai/docs/guides), each a tested recipe for one of the capabilities above, and add the optimization that fits your workload.
3. Read the [Introduction](https://llm-d.ai/docs/getting-started) and [Architecture overview](https://llm-d.ai/docs/architecture) to see how the pieces wrap your vLLM deployment.
1. Deploy the [Optimized Baseline](https://llm-d.ai/docs/well-lit-paths/foundations/optimized-baseline) with the [Quickstart](https://llm-d.ai/docs/getting-started/quickstart). It stands up an intelligent router over a vLLM pool on Kubernetes in a tested configuration.
2. Browse the [well-lit path guides](https://llm-d.ai/docs/well-lit-paths), each a tested recipe for one of the capabilities above, and add the optimization that fits your workload.
3. Read the [Introduction](https://llm-d.ai/docs) and [Architecture overview](https://llm-d.ai/docs/architecture) to see how the pieces wrap your vLLM deployment.
You can also deploy vLLM with llm-d via [KServe's LLMInferenceService](https://kserve.github.io/website/docs/model-serving/generative-inference/llmisvc/llmisvc-overview).
+1 -1
View File
@@ -147,7 +147,7 @@ Priority is **1 = highest** (tried first).
--8<-- "gen:priority-mla"
> **\*** For sparse MLA, FP8 KV cache always prefers `FLASHINFER_MLA_SPARSE`. With BF16 KV cache, `FLASHINFER_MLA_SPARSE` is preferred for low query-head counts (<= 16), while `FLASHMLA_SPARSE` is preferred otherwise.
> **\*** For sparse MLA, FP8 KV cache always prefers `FLASHINFER_MLA_SPARSE`. With BF16 KV cache, `FLASH_ATTN_MLA_SPARSE_FA4` (then `FLASHINFER_MLA_SPARSE`) is preferred for low query-head counts (<= 16) on SM 10.x, while `FLASHMLA_SPARSE` is preferred otherwise.
>
> **Note:** ROCm and CPU platforms have their own selection logic. See the platform-specific documentation for details.
+1 -5
View File
@@ -52,8 +52,6 @@ While `NONE` , `PIECEWISE`, and `FULL` are single-mode configurations and simply
!!! note
Here, the single-modes `NONE`, `PIECEWISE`, and `FULL` are treated as the runtime modes for CUDA Graphs dispatching. If using a dual-mode, the dispatcher will always dispatch to one of its member modes (plus a potential `NONE` if no suitable CUDA Graph available), depending on the batch composition.
While cascade attention is not cudagraph compatible, it is now compatible with all possible cudagraph mode configurations. If a batch uses cascade attention, it always gets dispatched to `PIECEWISE` mode if available (otherwise `NONE`).
!!! note
Not all CUDA Graph modes are compatible with every attention backend. We automatically "downgrade" modes to the closest supported mode. For example, if a backend only supports CUDA Graphs for pure decode/uniform batches, we convert `FULL` to `FULL_AND_PIECEWISE` if piecewise compilation is enabled, and `FULL_DECODE_ONLY` otherwise.
@@ -154,9 +152,7 @@ To signal the CUDA Graphs compatibility of the attention backends, we introduce
```python
class AttentionCGSupport(enum.Enum):
""" Constants for the CUDA Graphs support of the attention backend
Here we do not consider the cascade attention, as currently
it is never CUDA Graphs supported."""
"""Constants for the CUDA Graphs support of the attention backend"""
ALWAYS = 3
"""CUDA Graphs always supported; supports mixed-prefill-decode"""
+3 -1
View File
@@ -126,7 +126,9 @@ Do not expect a single endpoint plugin to also mutate engine/worker state. If yo
## Path-prefix convention
There is currently no route conflict enforcement (tracked as a follow-up to RFC [#46565](https://github.com/vllm-project/vllm/issues/46565)). A plugin's `attach_router` can register a path that collides with a core route and routes attached later win. To avoid surprising operators:
Plugin hooks run after the core routers have been registered. HTTP routes added by plugins replace earlier routes with the same path template and HTTP method, including their OpenAPI description. Methods not overridden by the plugin remain available. When multiple plugins register the same operation, the last registered route takes precedence. Different path templates are not treated as equivalent.
There is currently no rejection of route conflicts (tracked as a follow-up to RFC [#46565](https://github.com/vllm-project/vllm/issues/46565)). To avoid surprising operators:
- Namespace your routes under a distinct prefix, e.g. `/plugins/<plugin-name>/...`, rather than reusing `/v1/...` or other core prefixes
- Only register routes under a core prefix (like the worked example's `/v1/admin/scheduler_config`) if you specifically intend to override or extend existing behavior and document that clearly for operators allowlisting your plugin
+121
View File
@@ -0,0 +1,121 @@
# Expert substitution (MoNE)
[Mixture of Novices and Experts (MoNE)](https://arxiv.org/abs/2507.00390)
prunes MoE layers by replacing redundant experts with "novices" that return a
constant vector. vLLM loads the retained experts into compact weights and adds
the novices' output separately, so pruned experts use no MLP weight memory and
no MoE backend needs to know about them.
## Checkpoint format
The checkpoint keeps the original architecture. `config.json` lists the
substituted experts of each decoder layer:
```json
{
"architectures": ["DeepseekV2ForCausalLM"],
"n_routed_experts": 64,
"approximate_experts": {
"1": [1, 6, 8, 12],
"2": [0, 3, 17, 40]
}
}
```
For VLMs, `approximate_experts` may live on the text config. Retained experts
keep their logical checkpoint names. Each substituted expert stores one
`[hidden_size]` tensor instead of its MLP:
```text
model.layers.1.mlp.experts.0.gate_proj.weight
model.layers.1.mlp.experts.1.approx_value
```
Alternatively, `compression_config.transform_config.expert_substitution`
declares the same information with explicitly named constant tensors:
```json
{
"compression_config": {
"transform_config": {
"expert_substitution": {
"version": 1,
"router_semantics": {
"preserve_logical_expert_ids": true,
"preserve_router_weights": true,
"renormalize_after_substitution": false
},
"targets": {
"model.layers.1.mlp.experts": {
"num_logical_experts": 64,
"weight_layout": "compact_retained_experts",
"replacements": {
"1": {
"format": "constant-v1",
"tensors": {"value": "model.layers.1.mlp.expert_replacements.1.value"}
}
}
}
}
}
}
}
}
```
Only the values shown for `version`, `router_semantics`, `weight_layout`, and
`format` are supported. Each target path must contain exactly one layer index.
This form is not yet accepted alongside compressed-tensors quantization, whose
`transform_config` schema does not include `expert_substitution`.
## Semantics
For a substituted expert `j` with constant `v_j` and router weight `w_j(x)`,
the expert contributes `w_j(x) v_j`. Routing is unchanged: router weights are
neither renormalized nor redistributed to retained experts.
## Execution
`FusedMoEFactory` matches each MoE layer to `approximate_experts` by the layer
index in its prefix and, for a match, builds `SubstitutedRoutedExperts`, which:
1. allocates weights only for retained experts, in ascending logical order;
2. after routing, sums `w_j(x) v_j` over substituted routes in FP32;
3. rewrites substituted routes as zero-weight routes to physical expert 0 and
retained routes as compact physical IDs, preserving the `[num_tokens, top_k]`
contract of every decomposed backend;
4. adds the constant output on one tensor-parallel rank, before the partial
outputs are all-reduced.
Zero-weight routes may schedule some unnecessary GEMM work; the memory savings
are unaffected. Monolithic backends route internally and are not selected.
## Weight loading
Retained experts are loaded through the standard expert mapping by logical ID
and placed in their compact row. Substituted IDs map to no local row, the same
way expert parallelism skips non-local experts.
Constant tensors are consumed at the model's `load_weights` boundary, so
initial loading and direct weight updates share one path. `approx_value` tensors of
layers that the model does not build (other pipeline stages, MTP layers) are
passed through to the model's loader, which skips them like any other weight
it does not own. Explicitly named constants are always consumed, and dropped
when their layer is built elsewhere. Loading fails if a local substituted
expert has no constant.
Direct `load_weights` updates may change individual constants. Layerwise
reloads (`reload_weights` in checkpoint format) are rejected, and the model
keeps its previous weights.
## Supported configurations
- Models whose MoE layers use `FusedMoEFactory` with the default
`RoutedExperts`, and whose MoE prefix contains exactly one layer index.
- Unquantized experts.
- Tensor and pipeline parallelism, including MTP draft models.
- Any decomposed MoE backend.
Expert and data parallelism, EPLB, MoE LoRA, fused shared experts, quantized
experts, and routed input/output transforms are rejected at initialization.
Deferred MoE finalize and layerwise weight reloads are rejected when used.
+2 -2
View File
@@ -232,7 +232,7 @@ Supported quantization scheme/hardware combinations:
- Pass: [`vllm/compilation/passes/fusion/rms_quant_fusion.py`](https://github.com/vllm-project/vllm/blob/main/vllm/compilation/passes/fusion/rms_quant_fusion.py)
- ROCm AITER pass: [`vllm/compilation/passes/fusion/rocm_aiter_fusion.py`](https://github.com/vllm-project/vllm/blob/main/vllm/compilation/passes/fusion/rocm_aiter_fusion.py)
- CUDA/HIP kernels: [`csrc/libtorch_stable/layernorm_quant_kernels.cu`](https://github.com/vllm-project/vllm/blob/main/csrc/libtorch_stable/layernorm_quant_kernels.cu)
- CUDA/HIP kernels: [`csrc/layernorm_quant_kernels.cu`](https://github.com/vllm-project/vllm/blob/main/csrc/layernorm_quant_kernels.cu)
### SiLU+Mul + Quantization (`fuse_act_quant`)
@@ -258,7 +258,7 @@ Supported quantization scheme/hardware combinations:
- Pass: [`vllm/compilation/passes/fusion/act_quant_fusion.py`](https://github.com/vllm-project/vllm/blob/main/vllm/compilation/passes/fusion/act_quant_fusion.py)
- ROCm AITER pass: [`vllm/compilation/passes/fusion/rocm_aiter_fusion.py`](https://github.com/vllm-project/vllm/blob/main/vllm/compilation/passes/fusion/rocm_aiter_fusion.py)
- CUDA/HIP kernels: [`csrc/quantization/`](https://github.com/vllm-project/vllm/blob/main/csrc/quantization/)
- Fused SiLU+Mul+BlockQuant kernel: [`csrc/libtorch_stable/quantization/fused_kernels/fused_silu_mul_block_quant.cu`](https://github.com/vllm-project/vllm/blob/main/csrc/libtorch_stable/quantization/fused_kernels/fused_silu_mul_block_quant.cu)
- Fused SiLU+Mul+BlockQuant kernel: [`csrc/quantization/fused_kernels/fused_silu_mul_block_quant.cu`](https://github.com/vllm-project/vllm/blob/main/csrc/quantization/fused_kernels/fused_silu_mul_block_quant.cu)
### RMSNorm + Padding (`fuse_act_padding`)
+1 -1
View File
@@ -204,7 +204,7 @@ The second question is the cache eviction policy. For now, we use one LRU queue
### Case 3: mamba models
The prefix caching support of the mamba model is work in progress. Once implemented, models with mamba layer + full attention layer can be supported via the full attention + X algorithm in case 2.
Models with mamba layers + full attention layers support prefix caching via the full attention + X algorithm in case 2, with the Mamba state cached at block boundaries. See [Automatic Prefix Caching](../features/automatic_prefix_caching.md).
## Implementation
+1
View File
@@ -30,6 +30,7 @@ In v1, an extensive set of metrics are exposed via a Prometheus-compatible `/met
- `vllm:prompt_tokens_total` (Counter) - Total number of prompt tokens processed.
- `vllm:generation_tokens_total` (Counter) - Total number of generated tokens.
- `vllm:request_success_total` (Counter) - Number of finished requests (by finish reason).
- `vllm:admission_rejections_total` (Counter) - Admission attempts rejected by a queue limit, by reason.
- `vllm:request_prompt_tokens` (Histogram) - Histogram of input prompt token counts.
- `vllm:request_generation_tokens` (Histogram) - Histogram of generation token counts.
- `vllm:time_to_first_token_seconds` (Histogram) - Time to first token (TTFT).
+1
View File
@@ -90,6 +90,7 @@ GPU-side fusion is controlled by `multimodal_config.mm_device_do_normalize`:
| `llama-nemotron-vl-embed` | `LlamaNemotronVLForEmbedding` | `nvidia/llama-nemotron-embed-vl-1b-v2` |
| `llama-nemotron-vl-rerank` | `LlamaNemotronVLForSequenceClassification` | `nvidia/llama-nemotron-rerank-vl-1b-v2` |
| `mistral3` | `Mistral3ForConditionalGeneration` | `mistralai/Mistral-Small-3.2-24B-Instruct-2506`, etc. |
| `nemotron-parse` | `NemotronParseForConditionalGeneration` | `nvidia/NVIDIA-Nemotron-Parse-2.0`, etc. |
| `qwen2-vl` | `Qwen2VLForConditionalGeneration` | `Qwen/Qwen2-VL-2B-Instruct`, etc. |
| `qwen2.5-vl` | `Qwen2_5_VLForConditionalGeneration` | `Qwen/Qwen2.5-VL-3B-Instruct`, etc. |
| `qwen3-vl` | `Qwen3VLForConditionalGeneration` | `Qwen/Qwen3-VL-4B-Instruct`, etc. |
+1 -1
View File
@@ -91,7 +91,7 @@ To be used with a particular `FusedMoEPrepareAndFinalizeModular` subclass, MoE k
| marlin | standard,</br>batched | <sup>3</sup> / N/A | <sup>3</sup> / N/A | silu,</br>swigluoai | Y | Y | [`fused_marlin_moe`][vllm.model_executor.layers.fused_moe.experts.marlin_moe.fused_marlin_moe],</br>[`MarlinExperts`][vllm.model_executor.layers.fused_moe.experts.marlin_moe.MarlinExperts],</br>[`BatchedMarlinExperts`][vllm.model_executor.layers.fused_moe.experts.marlin_moe.BatchedMarlinExperts] |
| trtllm | standard | mxfp4,</br>nvfp4 | G(16),G(32) | <sup>5</sup> | N | Y | [`TrtLlmMxfp4ExpertsMonolithic`][vllm.model_executor.layers.fused_moe.experts.trtllm_mxfp4_moe.TrtLlmMxfp4ExpertsMonolithic],</br>[`TrtLlmMxfp4ExpertsModular`][vllm.model_executor.layers.fused_moe.experts.trtllm_mxfp4_moe.TrtLlmMxfp4ExpertsModular],</br>[`TrtLlmNvFp4ExpertsMonolithic`][vllm.model_executor.layers.fused_moe.experts.trtllm_nvfp4_moe.TrtLlmNvFp4ExpertsMonolithic],</br>[`TrtLlmNvfp4ExpertsModular`][vllm.model_executor.layers.fused_moe.experts.trtllm_nvfp4_moe.TrtLlmNvFp4ExpertsModular] |
| hpc | standard | fp8 | G(128),T | silu | Y | Y | [`HPCExperts`][vllm.model_executor.layers.fused_moe.hpc_moe.HPCExperts] |
| rocm aiter moe | standard | mxfp4,</br>fp8 | G(32),G(128),A,T | silu, gelu,</br>swigluoai | Y | N | `rocm_aiter_fused_experts`,</br>`AiterExperts` |
| rocm aiter moe | standard | none,</br>mxfp4,</br>fp8 | G(32),G(128),A,T | silu, gelu,</br>swigluoai,</br>situ | Y | Y | [`rocm_aiter_fused_experts`][vllm.model_executor.layers.fused_moe.experts.rocm_aiter_moe.rocm_aiter_fused_experts],</br>[`AiterUnquantizedExperts`][vllm.model_executor.layers.fused_moe.experts.rocm_aiter_moe.AiterUnquantizedExperts],</br>[`AiterFp8BlockExperts`][vllm.model_executor.layers.fused_moe.experts.rocm_aiter_moe.AiterFp8BlockExperts],</br>[`AiterFp8TensorExperts`][vllm.model_executor.layers.fused_moe.experts.rocm_aiter_moe.AiterFp8TensorExperts],</br>[`AiterFp8ChannelExperts`][vllm.model_executor.layers.fused_moe.experts.rocm_aiter_moe.AiterFp8ChannelExperts],</br>[`AiterMxfp4Experts`][vllm.model_executor.layers.fused_moe.experts.rocm_aiter_moe.AiterMxfp4Experts] |
| cpu_moe | standard | N/A | N/A | silu, gelu,</br>gelu_tanh,</br>swigluoai | Y | N | [`X86CPUUnquantizedExperts`][vllm.model_executor.layers.fused_moe.experts.cpu_moe.X86CPUUnquantizedExperts],</br>[`ArmCPUUnquantizedExperts`][vllm.model_executor.layers.fused_moe.experts.cpu_moe.ArmCPUUnquantizedExperts],</br>[`CPUUnquantizedExperts`][vllm.model_executor.layers.fused_moe.experts.cpu_moe.CPUUnquantizedExperts] |
| naive batched<sup>4</sup> | batched | int8,</br>fp8 | G,A,T | silu, gelu | <sup>6</sup> | Y | [`NaiveBatchedExperts`][vllm.model_executor.layers.fused_moe.experts.fused_batched_moe.NaiveBatchedExperts] |
+302
View File
@@ -0,0 +1,302 @@
# Zero-Copy Shared-Memory Tensor Arena for `MessageQueue`
This document describes an optimization to vLLM's engine→worker IPC path for
multimodal serving under tensor parallelism, implemented in
`vllm/distributed/device_communicators/shm_tensor_arena.py` and wired into
`shm_broadcast.py`'s `MessageQueue`. It **builds on the existing out-of-band
tensor channel** (`_reduce_tensor` / `_rebuild_tensor`, still in
`shm_broadcast.py`), which already keeps CPU-tensor bytes out of the pickle
stream.
> **Status: experimental, opt-in.** This lands as an increment over the
> existing out-of-band channel, not a replacement for it. vLLM also has an
> existing multimodal-tensor shm caching mechanism; reconciling the two is
> intentionally left as follow-up work rather than blocking this PR.
## TL;DR
`MessageQueue.enqueue` already avoids copying CPU tensor bytes into the pickle
stream: `_reduce_tensor` emits each tensor as a protocol-5 out-of-band
`PickleBuffer`, and large payloads are published to the local readers over a
ZMQ socket. That removed the dominant cost of the older *in-band* path — the
per-tensor `pickle` serialize (see [History](#history-the-in-band-pickle-era)).
Two costs remain for a large multimodal `pixel_values` tensor on a TP=N worker:
1. **N transport copies.** On the overflow (large-payload) path the out-of-band
buffer is published to each of the N node-local readers via `send_multipart`,
so the tensor bytes are moved once **per reader** by the transport — ~N×200 MB
of host data motion for a payload the model needs exactly once.
2. **A pageable H2D.** Each reader zero-copies a view over its received ZMQ
frame, but that frame is ordinary pageable memory, so the reader's subsequent
`cudaMemcpyAsync` to the GPU pays pageable-copy staging plus first-touch
faults.
This change adds a slotted shared-memory **arena**: the writer does **one**
memcpy of the tensor into a free slot; every reader takes a **zero-copy view**
of the same slot (no per-reader transport copy); and the mapping is
`cudaHostRegister`-pinned so the H2D is a true DMA. Net for a 200 MB image on
TP=4: **1 copy + a pinned DMA** instead of **4 transport copies + a pageable
H2D**. Small tensors, non-contiguous tensors, and arena exhaustion all fall
through to the existing `_reduce_tensor` path unchanged.
## Scope of the win (and what still needs measuring)
The arena is an **incremental** improvement layered on the out-of-band channel,
not a from-scratch fix — the out-of-band channel already eliminated the single
largest cost (the serialize copy). The benefit that remains is:
- **N→1 copies**, which grows with TP degree (no gain at TP=1), and
- the **pinned vs. pageable H2D**, which shows up as reader-side H2D latency.
> **The benchmarks in [§4](#4-validation) were measured against the *in-band
> pickle* baseline that predates `_reduce_tensor`**, where the dominant cost was
> the serialize copy. They are retained as motivation and as an upper bound. The
> incremental improvement of the arena *over the current out-of-band path on
> `main`* is smaller and has **not yet been re-measured**; that A/B (arena vs.
> oob, same seeded workload) is the outstanding validation item before this is
> merge-ready.
## 1. Where it bites
Any multimodal model served with `--tensor-parallel-size > 1` through the
multiproc executor. The `rpc_broadcast_mq` `MessageQueue`
(`v1/executor/multiproc_executor.py`) carries scheduler output — including the
raw preprocessed `pixel_values` — from the EngineCore to every TP worker each
step. Images at native resolution reach tens to hundreds of MB (a ~4k×4k image
is ~200 MB as bf16 patches). The out-of-band channel keeps those bytes out of
the pickle stream, but on the overflow path they are still published **once per
node-local reader**, and each reader's H2D is from **pageable** memory. The
per-reader copy cost scales with TP degree; the pinning cost is a fixed per-image
tax on the reader H2D. Latency-sensitive (interactive) serving makes the residual
visible in TTFT and, via head-of-line blocking, in the end-to-end latency of
*other* requests.
### History: the in-band-pickle era
Before `_reduce_tensor`, `MessageQueue.enqueue` pickled the whole payload —
including tensor bytes — because `torch.Tensor.__reduce_ex__` does not emit
protocol-5 out-of-band buffers. A ~200 MB tensor was byte-copied into the pickle
stream on the engine (`THPStorage_writeFileRaw`, ~6.5 ms/MB → ~1.3 s), *inside*
the EngineCore step loop, so all GPUs on the worker went idle for **0.9–1.3 s per
large image** (the vision-encoder compute for the same image is 50–90 ms →
transport dominated compute ~15:1); each reader then paid a full `pickle.loads`
byte-copy. Per 200 MB image on TP=4 that was ~1.8 GB of host data motion.
The `_reduce_tensor` out-of-band channel removed the serialize copy (and the
per-reader `pickle.loads` copy on the zero-copy ZMQ-frame path). Note that
merely raising `VLLM_MQ_MAX_CHUNK_BYTES_MB` (ring vs. socket path) did **not**
help in the in-band era: the serialize ran before the ring-vs-socket branch, so
it was common to both — a negative result that pinned the real cost on the
byte-copies, not the transport choice.
## 2. The arena
The arena itself (`ShmTensorArena`, `_ArenaPickler`, `_rebuild_arena_tensor`)
lives in `shm_tensor_arena.py`; `shm_broadcast.py`'s `MessageQueue` creates
and wires it in but otherwise treats it as an internal implementation detail.
### 2.1 `ShmTensorArena`
A second shared-memory region created alongside the existing `ShmRingBuffer`:
- **N slots × slot_bytes** (default 8 × 256 MB, internal constants -- see
§3), plus per-slot metadata `[written_flag, reader0_done … readerN_done]`.
- Concurrency uses the **same lock-free single-writer/N-reader protocol as
`ShmRingBuffer`** (memory fences, per-reader done flags), so the model is one
the codebase already trusts.
- Created by the queue **writer** in `MessageQueue.__init__` when all readers are
node-local; readers attach via a new `tensor_arena_handle` field on the queue
`Handle`. Queues with remote readers get no arena and keep today's behavior.
- Pages are allocated lazily by the kernel on first write, so arenas on queues
that never carry big tensors (e.g. worker→engine response queues) cost
approximately nothing beyond the (virtual) reservation.
### 2.2 Writer path — arena diversion, composing with `_reduce_tensor`
`enqueue` builds one `dispatch_table` that routes CPU tensors through
`_reduce_tensor` (out-of-band `PickleBuffer`). When a node-local arena exists,
the pickler is an `_ArenaPickler` whose `reducer_override` *additionally* diverts
**large contiguous** CPU tensors into the arena:
```python
class _ArenaPickler(pickle.Pickler):
def reducer_override(self, obj):
rebuild_fn = _ARENA_REBUILD_FNS.get(type(obj)) # {Tensor: ..., Parameter: ...}
if (rebuild_fn is not None and obj.device.type == "cpu"
and obj.layout is torch.strided and obj.is_contiguous()
and not obj.requires_grad
and obj.numel() * obj.element_size() >= MIN_BYTES):
idx = self.arena.write_tensor(obj) # ONE memcpy into a free slot
if idx is not None:
return (rebuild_fn,
(self.arena.shared_memory.name, idx, nbytes, dtype_str, shape))
return NotImplemented # fall through to dispatch_table → _reduce_tensor
```
Note the exact-type dict lookup (not `isinstance`), matching how
`dispatch_table` itself dispatches: an unrecognized `Tensor` subclass must
decline here too, or it would come back as a plain `Tensor`, silently
losing its subclass identity. `torch.nn.Parameter` is the one subclass this
*does* divert, via its own rebuild function (§2.3).
`reducer_override` is consulted before an object's normal reduction, and
returning `NotImplemented` falls through to the `dispatch_table` — so a diverted
tensor becomes a ~100-byte rebuild stub, and **everything the arena declines**
(too small, non-contiguous, or arena full) is handled by `_reduce_tensor`
exactly as on `main`. The single arena copy —
`torch.frombuffer(slot).copy_(t.view(torch.uint8))` — is a multithreaded memcpy
at memory bandwidth (~50–70 ms for 200 MB), and, unlike the oob path, the *same*
slot is then read zero-copy by all N readers rather than transported to each.
**The writer never blocks.** If no slot is free or the tensor exceeds the slot
size, `write_tensor` returns `None`; the pickler falls through to `_reduce_tensor`
— the worst case is exactly the current out-of-band behavior, and deadlock is
structurally impossible.
### 2.3 Reader path — zero copies
The stub unpickles through a module-level rebuild function:
```python
def _rebuild_arena_tensor(arena_name, slot_idx, nbytes, dtype_str, shape):
arena = _TENSOR_ARENAS[arena_name] # this process's registry of attached arenas
# get_tensor: a view over the mapped slot — zero bytes copied
return arena.get_tensor(slot_idx, nbytes, getattr(torch, dtype_str), shape)
```
`_rebuild_arena_parameter` wraps the same tensor in
`torch.nn.Parameter(t, requires_grad=False)`, which shares its storage.
`_ARENA_REBUILD_FNS = {torch.Tensor: _rebuild_arena_tensor, torch.nn.
Parameter: _rebuild_arena_parameter}` is the full set of types the arena
diverts; anything else (any other `Tensor` subclass) declines in
`reducer_override` and takes the out-of-band path.
The registry is keyed by arena shm name (`_TENSOR_ARENAS:
weakref.WeakValueDictionary[str, ShmTensorArena]`) rather than a single
per-process slot, since nothing guarantees a process only ever attaches to
one arena-bearing queue. It holds only a *weak* reference — a strong one
would keep the arena (and its pinned mapping) alive for the process
lifetime — and needs no explicit removal on `MessageQueue.shutdown()`
(unsafe anyway, since shutdown can run on a different thread than an
in-flight `dequeue()`); an entry just disappears once nothing else
references that arena.
The rebuilt tensor *is* the shared memory — no transport copy and no deserialize
on any rank.
**Slot lifecycle.** The rebuilt tensor is the *source* of an async H2D while
the worker executes that step, so the reader must not release the slot at
unpickle time. Release is tied to **the lifetime of the slot's tensor
storage**, not a fixed "next dequeue" schedule — needed for callers like
`prompt_embeds` that retain the tensor across many `dequeue` calls during
chunked prefill, where a fixed schedule would let the writer reclaim (and
corrupt) the slot while the worker was still reading it. `get_tensor` builds
the tensor via `torch.from_numpy` over a numpy array wrapping the slot; the
tensor's storage holds the only reference to that array, so a
`weakref.finalize` on it fires only once every tensor, view/slice and wrapper
(e.g. `Parameter`) sharing the storage has been freed. Once queued, the
pinned fast path (§2.4) adds a second gate: the H2D is a true async DMA that
can outlive `execute_model`, so "the storage was freed" alone isn't
sufficient.
`flush_releases` records a CUDA event on the compute stream after each
step's H2D and only marks a slot done once that event completes
(non-blocking `event.query()`; not-yet-done just waits one more dequeue).
Unpinned, `cudaMemcpyAsync` from pageable memory already stages
synchronously, so the slot releases immediately. Either way, the writer
requires every reader's done flag before reusing a slot.
> **Note.** Assumes the multimodal H2D is issued on the worker's
> current/default compute stream (true today: mm inputs are copied eagerly, outside the decode CUDA
> graph). If a future vLLM issues that copy on a dedicated side stream, the event
> must be recorded at the copy site rather than at `flush_releases`.
### 2.4 Pinning — the zero-copy trap
A zero-copy view over shared memory the reader process has never touched makes
the subsequent `cudaMemcpyAsync` pay **first-touch page faults on ~50k pages plus
pageable-copy staging** — the same tax the current oob path pays on its pageable
ZMQ frame. Fix: each reader lazily `cudaHostRegister`s the whole arena mapping
once (~1 s, first use), after which every H2D from the arena is a **pinned-memory
DMA** (~10 ms for 192 MB). Processes without a CUDA context skip registration
silently. (Because the arena mapping is stable and reused, this one-time pin
amortizes across all future images — an option the transient per-message ZMQ
frame does not have.)
## 3. Configuration
A single CLI flag toggles the arena; the slot count/size and divert threshold
are internal constants (no env vars).
| Flag (`ParallelConfig` field) | Default | Meaning |
| --- | --- | --- |
| `--enable-shm-tensor-arena` / `--disable-shm-tensor-arena` (`enable_shm_tensor_arena`) | **off** | Opt in to route large CPU tensors through the arena (reserves slots in `/dev/shm`). Off = the out-of-band `_reduce_tensor` path only, identical to stock behavior. |
Internal constants in `shm_tensor_arena.py`: **8 slots × 256 MB**; tensors larger
than a slot, or smaller than the **8 MB** divert threshold, take the out-of-band
`_reduce_tensor` path.
## 4. Validation
> **Baseline caveat.** The A/B numbers below were collected against the *in-band
> pickle* path that predates `_reduce_tensor`. They quantify the arena vs. that
> older baseline (an upper bound), **not** the arena vs. the current out-of-band
> path. The current-`main` oob baseline would land between the "stock" and
> "arena + pinning" rows; measuring exactly where is the open item.
1. **Unit tests** (`tests/distributed/test_shm_broadcast.py`): byte-exact
zero-copy round-trips (incl. bf16/fp8), slot lifecycle (no reuse until every
reader releases; exhaustion / oversize / non-contiguous fall back to the
out-of-band path), event-gated release on the pinned path, `_ArenaPickler` ⊕
`_reduce_tensor` composition, and an end-to-end `MessageQueue` broadcast
through forked processes. (Historical microbenchmark: enqueue of a 199 MB
payload took **66.7 ms** vs ~1275 ms for the old in-band serialize.)
2. **Same-seed A/B** (baseline = in-band pickle, pre-`_reduce_tensor`), interactive
multimodal workload, uncapped input images, TP=4 ×2 workers on one 8-GPU node,
low qps so individual images decompose cleanly (~1.2k aligned requests):
| TTFT (ms) | p50 | p90 | p99 | max | >1 s | >1.5 s |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| in-band pickle (old baseline) | 93 | 401 | 1321 | 2139 | 30 | 9 |
| + bigger MQ chunk (config lever) | 91 | 368 | 1287 | 2027 | 22 | 5 |
| **arena + pinning** | **89** | **241** | **862** | **1375** | **7** | **0** |
p50 unchanged → `reducer_override` adds no measurable overhead when no large
tensor is present.
3. **Per-image stall** (CUDA-level all-GPU idle gap around the pixel H2D), vs the
in-band baseline: 192 MB image 1318 ms → 681 ms (arena only) → **351 ms class**
(arena + pinning); vision-encoder compute for the same image is ~80 ms.
4. **Outstanding — arena vs. out-of-band on `main`.** Re-run the same seeded A/B
with the baseline set to current `main` (i.e. `_reduce_tensor` on, arena off
via `--disable-shm-tensor-arena`) vs. arena on, to quantify the incremental N→1
copy + pinning gain at TP=4 (and ideally TP=2/8 to show the copy-count scaling).
## 5. Limitations and future work
- **Value is TP- and size-dependent.** The arena helps only when a queue has
≥2 node-local readers *and* carries tensors ≥ `MIN_MB`; at TP=1, or for small
payloads, it adds nothing and everything takes the out-of-band path.
- **Shared-memory reservation.** The arena reserves `slots × slot_bytes` of
`/dev/shm` (default 2 GB, lazily paged). Like `ShmRingBuffer`, creation is
guarded by a free-space check (`check_shm_free_space`), so an undersized
`/dev/shm` fails fast with a clear error instead of a `SIGBUS` the first
time a tensor is copied into pages beyond tmpfs capacity.
- **Relationship to `_reduce_tensor`.** This is strictly additive: the arena is
an opt-in fast path for the large-multimodal-tensor case; declining it (or
disabling via env) reverts to the merged out-of-band behavior.
- **Relationship to existing mm tensor shm caching.** vLLM already has a
separate shared-memory caching path for multimodal tensors; this arena is
not yet unified with it (see the experimental note in §1). Reconciling the
two is follow-up work.
- **Slot release granularity**: releases are tied to the slot's tensor
storage being freed and, on the pinned path, further gated on a per-slot
CUDA event recorded after the consuming H2D (§2.3), which closes both the
multi-step-retention and the async-DMA reuse windows. Bursts deeper than
the slot count safely fall back to the out-of-band path when the arena is
exhausted.
- **Fallback observability**: arena exhaustion (no free slot) is a
rate-limited log line today; a counter metric would be better. An
oversize tensor (bigger than a slot) falls back silently, with no log at
all -- worth adding if that path turns out to matter in practice.
- **Scope**: the arena activates only when every queue reader is node-local.
Remote readers (multi-node PP/TP) keep the existing socket path.
+3 -4
View File
@@ -15,23 +15,22 @@ Set `enable_prefix_caching=True` in vLLM engine to enable APC. Here is an exampl
## Hybrid Mamba models
Under `--mamba-cache-mode align`, Mamba state is stored only on the Mamba block grid, so a prefix-cache hit can resume only at a block boundary. `--enable-mamba-shared-prefix-checkpoint` also stores a checkpoint at the shared-prefix junction, the point where an earlier request with the same prefix stopped. Requests whose shared prefix ends inside a block can then reuse it.
With prefix caching enabled, Mamba state is stored only on the Mamba block grid, so a prefix-cache hit can resume only at a block boundary. `--enable-mamba-shared-prefix-checkpoint` also stores a checkpoint at the shared-prefix junction, the point where an earlier request with the same prefix stopped. Requests whose shared prefix ends inside a block can then reuse it.
This helps when many requests share a long system prompt and then diverge. It is off by default, and takes effect only when all of the following hold:
- `--mamba-cache-mode align`
- prefix caching enabled
- EAGLE/MTP speculative decoding on the Mamba group
- `--prefix-match-unit` smaller than the Mamba block size
- the model does not use multi-module MTP
```bash
vllm serve <hybrid-model> \
--mamba-cache-mode align \
--prefix-match-unit 64 \
--enable-mamba-shared-prefix-checkpoint
```
`--prefix-match-unit` is required. It sets the granularity at which prefix-cache keys are computed. When unset it defaults to the greatest common divisor of the prefix-cacheable KV cache group block sizes. Under `align` that is the block size itself, so no sub-block boundary exists and the flag has no effect.
`--prefix-match-unit` is required. It sets the granularity at which prefix-cache keys are computed. When unset it defaults to the greatest common divisor of the prefix-cacheable KV cache group block sizes. For hybrid Mamba models that is the block size itself, so no sub-block boundary exists and the flag has no effect.
Choose a value that divides the block size of every prefix-cacheable KV cache group, and that is a multiple of the per-state compression ratio for models that use one, such as sparse MLA. vLLM validates both at startup and names the offending sizes in the error. Read the served block size from the startup log. 64 is a reasonable starting point.
+1
View File
@@ -20,6 +20,7 @@ Batch invariance is crucial for several use cases:
Batch invariance is supported on the following platforms:
- NVIDIA GPUs with compute capability 8.0 or higher.
- AMD Instinct GPUs (gfx90a, gfx942, gfx950).
- Intel XPUs with Triton support.
### Attention Backend Selection for XPU
+1 -1
View File
@@ -96,7 +96,7 @@ EC transfer is configured via `--ec-transfer-config` (CLI) or the `ec_transfer_c
| --- | --- | --- | --- |
| `ec_enable_nixl` | `bool` | No (default `false`) | Enables NIXL P2P transfer in addition to local CPU offload. Omitted or `false` imports no NIXL/ZMQ. Extra config is not type coerced, so a string value is parsed: `"true"`, `"1"`, `"yes"` enable it, anything else does not. |
| `consumer_ack_timeout_s` | `float` | No (default `2.0`) | How long a consumer waits for an `XferAck` before giving up on a read. The producer answers `XferReq`s from its own scheduler step, so its reply latency scales with the encoder's `--max-num-batched-tokens`: a loaded encoder whose steps run longer than this makes consumers abandon reads the producer is about to grant. Raise it for large encoder batches. |
| `ec_cpu_bytes` | `int` | Yes | Total size, in bytes, of the shared CPU mmap region. `ECCPUConnector` raises `ValueError` if unset. Block count = `ec_cpu_bytes // block_size_bytes`, where `block_size_bytes = hidden_dim * dtype.element_size()` (`hidden_dim` accounts for Qwen3-VL deepstack: `out_hidden_size * (1 + num_deepstack_layers)`). |
| `ec_cpu_bytes` | `int` | Yes | Total size, in bytes, of the shared CPU mmap region. `ECCPUConnector` raises `ValueError` if unset. Block count = `ec_cpu_bytes // block_size_bytes`, where each block holds one LLM input-embedding vector. Entries allocate enough blocks for their modality-specific output shape, including visual DeepStack features. |
### Environment Variables
+31
View File
@@ -108,10 +108,28 @@ vllm serve <model> \
| `store_threshold` | no | `0` | single-tier | Min lookups before a block is offloaded. Values ≥ 2 are rejected by `TieringOffloadingSpec`. |
| `max_tracker_size` | no | `64000` | single-tier | Max entries in the lookup tracker. |
| `secondary_tiers` | no | `[]` | multi-tier | List of secondary tier configs (see below). |
| `tier_poll_interval_s` | no | `0.001` | multi-tier | Pause between tier-poll rounds for tiers that must be serviced between engine steps (currently `p2p`). See [Threading model](#threading-model). Set to `0` to disable the thread and service those tiers once per engine step instead. |
| `offload_prompt_only` | no | `true` | both | If `true`, only prompt (prefill) blocks are offloaded; decode blocks are skipped. |
| `self_describing_kv_events` | no | `false` | both | Opt-in. When `true` *and* KV cache events are enabled (`--kv-events-config` with `enable_kv_cache_events`), the connector emits self-describing block-granular `BlockStored`/`BlockRemoved` payloads (constituent block hashes, whole-chunk `token_ids`, per-block `block_size`, parent hash, LoRA + group/cache-spec metadata) instead of the placeholder fallback, so external KV-event consumers can index offloaded blocks. Inert unless events are enabled. With `TieringOffloadingSpec`, a CPU promotion is self-describing when a local request observes its primary-tier `HIT` before event translation; otherwise its stored event may retain the placeholder, while a later `HIT` can backfill metadata for removal. Pending-removal/re-promotion races and externally initiated promotions may also produce placeholders, and consumers must ignore removals for unknown hashes. Partial recurrent tails emit the hash-aligned portion from the physical block start through the tail boundary. Other sliding-window/SSM chunks keep the placeholder fallback. In chunk mode (`block_size` > GPU block size, or `blocks_per_chunk` > 1), overlapping chunks re-announce shared per-block hashes, so consumers must reference-count (deduplicate) repeated store/remove announcements. |
| `spec_module_path` | no | — | both | Python import path for a custom `OffloadingSpec` not in the built-in registry. Required only when `spec_name` is not built-in (advanced). |
## Info Metric Labels
The connector publishes one Prometheus gauge, `vllm:kv_offload_config_info`, and always sets it to 1. The labels hold static facts of the offloading configuration, so a user reads the resolved numbers instead of the launch flags. The gauge appears from the first scheduler step, so an idle engine exposes no series. Each engine reports its own configuration, and not the instance total.
The gauge holds one series for each configuration that the manager reports. A manager that holds one configuration gives one series. A manager that holds several adds a label of its own, which tells the series apart.
| Label | Declared by | Meaning | Notes |
| --- | --- | --- | --- |
| `tier` | `TieringOffloadingSpec` | The tier that this series reports. | `<index>:<type>`. Index `0` is the CPU primary tier, and a secondary tier index starts at 1. `<type>` is the `type` key of the tier config, such as `1:fs`. |
| `cpu_num_chunks` | `CPUOffloadingSpec` or `TieringOffloadingSpec` | Chunk slots in the CPU tier. | Chunks, not GPU blocks. `cpu_bytes_to_use` divided by the bytes of one chunk. |
The spec declares the label names in the API server process, and the manager fills the values in the engine process. The names of every series bind once, at the declaration. A label that one series owns reads empty on every other series.
`CPUOffloadingSpec` publishes `cpu_num_chunks` on its one series. `TieringOffloadingSpec` publishes it on the `0:primary` series. For a worked example of a secondary tier, the `example` tier publishes one label, `example_info`.
A manager adds a label with `config_info_keys()` of its spec and `config_info()` of its own class. A name that a manager fills and the spec does not declare is dropped, and the engine log then holds one warning line.
## Custom Eviction Policies
`eviction_policy` resolves through `CachePolicyFactory` (`vllm/v1/kv_offload/cpu/policies/factory.py`), which pre-registers the built-in `lru` and `arc` policies.
@@ -150,6 +168,19 @@ The filesystem and object-store tiers can publish hash-only `BlockStored` KV eve
Set the optional `locality` tier field to `LOCAL` or `REMOTE` to describe the tier's storage location relative to the publishing vLLM instance. `LOCAL` marks storage local to that instance, while `REMOTE` marks storage that is not local to it. When the setting is omitted, locality is unspecified. vLLM does not infer it from the tier type, so an `obj` tier is not implicitly `REMOTE`. A KV event includes `locality` only when the tier explicitly configures it. This metadata describes the tier property without implying that a consumer can already route requests to its blocks.
### Threading model
Tier methods run in the scheduler process, under the tiering manager's lock, so they are never entered concurrently. They are not always called from the same thread, though.
Most tiers only ever see the scheduler thread, which holds the lock for the length of one engine step. A tier whose counterpart is not driven by the engine — the `p2p` tier, answering a remote peer — needs servicing while the engine is busy running the model, because otherwise a peer's lookup or fetch waits for the next step boundary. On a saturated rank that wait can be seconds, dwarfing the transfer it gates. Such tiers set `tolerates_off_thread_polling`, and the tiering manager runs one tier-poll thread that, once per `tier_poll_interval_s`, polls them for finished jobs and lets them serve external requests. A tier that implements `serve_external_requests()` without setting the flag is served once per engine step only.
Two consequences worth knowing:
- Any tier can be reached from that thread indirectly, because serving a peer looks keys up through the tiering manager, which fans out to the other tiers. Tier state must therefore not assume a particular thread, only that the manager lock is held.
- Work the thread deliberately leaves alone is batched per step: a promotion it initiates is submitted at the next `on_schedule_end()`, not immediately.
Setting `tier_poll_interval_s` to `0` disables the thread; opted-in tiers then get serviced once per engine step, as they were before it existed.
### Filesystem (FS)
The filesystem tier (`type: "fs"`) writes blocks to a filesystem directory.
+2 -2
View File
@@ -52,7 +52,7 @@ CUDA_VISIBLE_DEVICES=0 vllm serve Qwen/Qwen2.5-VL-3B-Instruct \
--port 19534 \
--gpu-memory-utilization 0.35 \
--enable-request-id-headers \
--no-enable-prefix-caching \
--disable-prefix-caching \
--max-num-batched-tokens 16384 \
--allowed-local-media-path /path/to/media \
--ec-transfer-config '{
@@ -72,7 +72,7 @@ CUDA_VISIBLE_DEVICES=1 vllm serve Qwen/Qwen2.5-VL-3B-Instruct \
--gpu-memory-utilization 0.75 \
--enable-mm-embeds \
--enable-request-id-headers \
--no-enable-prefix-caching \
--disable-prefix-caching \
--allowed-local-media-path /path/to/media \
--ec-transfer-config '{
"ec_connector": "ECMooncakeConnector",
+1 -1
View File
@@ -398,7 +398,7 @@ The `kv_load_failure_policy` setting controls how the system handles failures wh
### For NVIDIA GB-series GPUs
GB-series GPUs support multi-node NVLink. NIXL supports this capability, but KVCache must be registered as VMM during KVCache registration. To enable this feature, you need to set `--enable-cumem-allocator` or `--enable-sleep-mode` flags, and set `UCX_CUDA_IPC_ENABLE_MNNVL: 'y'` env. Otherwise, NIXL can only use RDMA/TCP for cross-node KVCache transfers.
GB-series GPUs support multi-node NVLink. NIXL supports this capability, but KVCache must be registered as VMM during KVCache registration. To enable this feature, you need to set the `--enable-cumem-allocator` flag, and set `UCX_CUDA_IPC_ENABLE_MNNVL: 'y'` env. Otherwise, NIXL can only use RDMA/TCP for cross-node KVCache transfers.
## Experimental Feature
+28 -2
View File
@@ -50,6 +50,26 @@ vllm serve meta-llama/Llama-3.1-8B-Instruct \
The daemon itself must load from disk; passing `--load-format ipc_cache` to
`vllm preload` is an error.
## Preloading the FlashInfer autotune cache
FlashInfer has several implementations of each operation and chooses between
them by benchmarking. vLLM runs that pass during kernel warmup and persists the
result per rank in the on-disk FlashInfer autotune cache (under
`VLLM_CACHE_ROOT`, or `VLLM_FLASHINFER_AUTOTUNE_CACHE_DIR`), so a later engine
with the same configuration loads the tuned configs instead of profiling again.
Normally the first serving engine pays for the pass, and on a large MoE model it
dominates what is left of the startup time once the weights come from the
daemon.
Unless FlashInfer autotune is disabled (`--disable-flashinfer-autotune`),
`vllm preload` pays it at preload time: each daemon runs the autotune pass in
place — profiling dummy runs against the shard it already holds — after
loading and before binding its socket, then writes the cache:
```bash
vllm preload --model /path/to/model --tensor-parallel-size 4
```
## How it works
1. `vllm preload` spawns one daemon process per GPU. Each daemon loads its
@@ -164,8 +184,14 @@ Socket paths are derived from the GPU UUID, so they are stable regardless of
- **Platform**: CUDA and ROCm only; other platforms raise
`UnsupportedPlatformForIPCError` even when `fallback` is enabled, since it
is a permanent misconfiguration rather than a transient daemon outage.
- **Parallelism**: tensor, expert and data parallelism are supported;
launching the daemon with pipeline parallelism is rejected.
- **Parallelism**: tensor, expert, pipeline and data parallelism are
supported, including across nodes.
- **Autotune preloading**: tuning runs across the daemon's world group,
including across nodes and DP ranks. With model-based speculative decoding
both daemon groups skip it (the target cannot run the drafter's dummy
passes, and a table tuned on the draft model would key on the draft's
config, which no engine looks up), so those deployments' first engine
tunes and fills the cache instead.
- **Quantization**: every quantization method in the model must declare
support for pre-processed weights (the daemon transfers weights *after*
quantization post-processing). Unsupported methods raise
+31 -6
View File
@@ -17,7 +17,7 @@ vllm serve <model> \
--moe-backend b12x
```
Only pass `--moe-backend b12x` for a compatible NVFP4 or MXFP4 MoE model. The
Only pass `--moe-backend b12x` for a compatible packed-weight MoE model. The
linear and MoE backends can be selected independently.
b12x uses MXFP8 activations by default for MXFP4 MoE and the checkpoint's
@@ -29,9 +29,34 @@ force BF16 activations for either FP4 weight format.
| Backend | Supported configurations |
| ------- | ------------------------ |
| Linear | Per-tensor FP8, 128x128 block FP8, MXFP8, NVFP4, and MXFP4 |
| MoE | Tensor-parallel MXFP4 weights with BF16 or MXFP8 activations; NVFP4 weights with BF16, NVFP4, or MXFP8 activations |
| Linear | Per-tensor FP8, 128x128 block FP8, MXFP8, NVFP4 (including W4A16), MXFP4, IQ2_XS, IQ2_XXS, and Q8_0 |
| MoE | Tensor-parallel MXFP4 weights with BF16 or MXFP8 activations; NVFP4 weights with BF16, NVFP4, or MXFP8 activations; IQ2_XS, IQ2_XXS, and Q8_0 weights with BF16 activations |
| Embedding | Packed Q8_0 lookup with BF16 output |
Dense W4A16 layers are not handled by b12x and continue to use another
compatible backend such as Marlin. The b12x MoE backend does not support expert
parallelism, expert maps, EXL3, or NF3.
The b12x MoE backend does not support expert parallelism, expert maps, EXL3, or NF3.
## ModelOpt block codecs
ModelOpt mixed-precision checkpoints select block codecs through the
`quantization_config.quantized_layers` entries in `config.json`. Each entry
declares `quant_algo`, `group_size`, `block_payload_bytes`, and `packing="ggml"`:
| `quant_algo` | `group_size` | `block_payload_bytes` |
| --- | --- | --- |
| `IQ2_XS` | 256 | 74 |
| `IQ2_XXS` | 256 | 66 |
| `Q8_0` | 32 | 34 |
Serialized matrices have uint8 shape `[N, K/group_size, block_payload_bytes]`.
Dense preparation rearranges the encoded blocks without expanding the matrix
to BF16. Routed experts use bias-free SiLU or non-gated ReLU²; tensor-parallel
shards must preserve whole blocks and satisfy the kernel's alignment constraints.
Only Q8_0 supports packed embedding lookup.
Quantized normalization weights and other packed vectors are not supported.
Store normalization weights as ordinary floating-point tensors and omit their
entries from `quantization_config.quantized_layers`.
Dense, embedding, and expert matrices remain packed. Their execution uses BF16
activations without activation quantization. Prepare the kernels and workspaces
before CUDA graph capture.
+14 -3
View File
@@ -111,7 +111,10 @@ Currently supported combinations are:
| Checkpoint layer | Online target | Requirements |
| ---------------- | ------------- | ------------ |
| Serialized ModelOpt MXFP8 linear | any | Target-specific hardware requirements |
| Serialized ModelOpt MXFP8 linear | any online linear method | Target-specific hardware requirements |
| Serialized compressed-tensors NVFP4 MoE | any online MoE method | Target-specific hardware requirements |
| Serialized ModelOpt NVFP4 MoE | any online MoE method | Target-specific hardware requirements |
| Serialized Quark NVFP4 MoE | any online MoE method | Target-specific hardware requirements |
For example, serialized ModelOpt MXFP8 linears can be converted to
per-channel-weight/per-token-activation FP8 at load time:
@@ -122,10 +125,18 @@ vllm serve MiniMaxAI/MiniMax-M3-MXFP8 \
--quantization-config.linear fp8_per_channel
```
Similarly, for example, NVFP4 MoE checkpoints can be re-quantized to any online MoE method,
for example compressed-tensors NVFP4 experts to online MXFP4:
```bash
vllm serve RedHatAI/Qwen3-30B-A3B-NVFP4 \
--quantization-config '{"targets":{"*mlp.experts*":"mxfp4"}}'
```
!!! info
This is only available for linear layers at the moment.
Re-quantization is available for both linear and MoE layers.
Checkpoint-exclusion policy is not enforced by online quantization/re-quantization. Only exclusion through `--quantization-config.targets`/`--quantization-config.ignore` is enforced.
### Online quantization on unquantized layers from partially-quantized checkpoints
+36 -3
View File
@@ -77,8 +77,15 @@ llm.wake_up(tags=["kv_cache"])
#### Retaining frozen weights during RLHF updates
Set `sleep_preserve_parameter_names` (CLI: `--sleep-preserve-parameter-names`)
to runtime parameter-name glob
patterns for weights that stay frozen during training. Each pattern must match
to runtime parameter-name glob patterns for weights that stay frozen during
training. It covers two use cases:
- **Frozen Engram tables (original use case):** retain table weights and scales
omitted by the trainer, without retransmitting them after every sleep.
- **Frozen draft weights:** retain draft-private parameters when the trainer
updates only the target model.
Target patterns match
`model.named_parameters()`; checkpoint names and `requires_grad` are not used.
Level-2 sleep backs up selected GPU parameters to pageable CPU memory; weights
@@ -87,7 +94,33 @@ copy. Allow enough host memory for backups. Level-1 behavior is unchanged.
The trainer must omit these parameters from updates, and reload/post-processing
must preserve their values and storage (including avoiding replacement with meta
tensors). This option does not filter updates and applies only to the target model.
tensors). This option does not filter updates.
For frozen Engram tables omitted by the trainer, select the target table weights
and scales. For example, DeepSeek V4.1 uses these runtime parameter names:
```bash
VLLM_SERVER_DEV_MODE=1 vllm serve deepseek-ai/DeepSeek-V4.1-Flash --enable-sleep-mode \
--sleep-preserve-parameter-names '*.engram.embed_tokens.weight*'
```
CPU-offloaded tables remain on CPU without an extra backup copy. Engram's
[inference-time CPU offload](engram.md#cpu-offload) is configured separately;
this option preserves the frozen tables across level-2 sleep and weight updates.
To retain a frozen drafter while updating only target weights, use `draft.*`:
```bash
VLLM_SERVER_DEV_MODE=1 vllm serve Qwen/Qwen3.8-27B --enable-sleep-mode \
--speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
--sleep-preserve-parameter-names 'draft.*'
```
vLLM selects all draft-private parameters internally; you do not need to list
them individually. Use `draft.<name>` to select specific draft parameters.
Parameters shared with the target are excluded from the draft selection, so
target updates remain authoritative. An empty `draft.*` selection is valid.
The Engram and draft patterns can be supplied together.
#### Release only KV cache memory
+1 -3
View File
@@ -6,14 +6,13 @@ To train your own draft models for optimized speculative decoding, see [vllm-pro
## vLLM Speculation Methods
vLLM supports a variety of methods of speculative decoding. Model-based methods such as EAGLE, MTP, draft models, PARD and MLP provide the best latency reduction, while simpler methods such as n-gram and suffix decoding provide modest speedups without increasing workload during peak traffic.
vLLM supports a variety of methods of speculative decoding. Model-based methods such as EAGLE, MTP, draft models and PARD provide the best latency reduction, while simpler methods such as n-gram and suffix decoding provide modest speedups without increasing workload during peak traffic.
- [EAGLE](eagle.md)
- [LiLiCorr](lilicorr.md)
- [Multi-Token Prediction (MTP)](mtp.md)
- [Draft Model](draft_model.md)
- [Parallel Draft Model (PARD)](parallel_draft_model.md)
- [Multi-Layer Perceptron](mlp.md)
- [N-Gram](n_gram.md)
- [Suffix Decoding](suffix.md)
- [Hidden State Extraction](extract_hidden_states.md)
@@ -33,7 +32,6 @@ depend on your model family, traffic pattern, hardware, and sampling settings.
| MTP | High gain | Medium to high gain | Best when the target model has native MTP support. |
| Draft model | High gain | Medium gain | Needs a separate draft model. |
| Parallel Draft Model | High gain | Medium to high gain | Low draft model latency. |
| MLP speculator | Medium to high gain | Medium gain | Good when compatible MLP speculators are available. |
| N-gram | Low to medium gain | Medium gain | Lightweight and easy to enable. |
| Suffix decoding | Low to medium gain | Medium gain | No extra draft model; dynamic speculation depth. |
| Custom Proposer | Varies | Varies | Bring your own proposer class (experimental). |
@@ -4,7 +4,7 @@ Speculative decoding buys fewer decode steps with more compute. At batch size 1
That matters because per-position acceptance decays fast. While the GPU is memory-bound that slot is effectively free and worth the gamble; once it saturates the gamble has a real throughput cost. The crossover moves with load and with workload-dependent acceptance rates, so no static `num_speculative_tokens` is right across concurrencies.
Adaptive verification decides per step how much of the draft to verify instead. Every (request, position) draft slot is scored by its *survival probability*, the running product of that request's per-position confidences, and the highest-scoring slots are admitted until a global budget is spent. Slots compete across requests: position 5 of a confident request can outrank position 1 of a doubtful one, so one request keeps its full block while another could be trimmed after a token or two.
Adaptive verification decides per step how much of the draft to verify instead. Every (request, position) draft slot is scored by its *survival probability*, the running product of that request's per-position acceptance estimates, and the highest-scoring slots are admitted until a global budget is spent. Slots compete across requests: position 5 of a confident request can outrank position 1 of a doubtful one, so one request keeps its full block while another could be trimmed after a token or two.
The budget itself comes from a cost model profiled at startup. vLLM measures what a step costs at each shape, then picks the token count that maximizes expected accepted tokens per second.
@@ -12,11 +12,14 @@ The practical effect is that one configuration holds up across the whole load ra
## Support
Adaptive verification needs per-position acceptance estimates, so today it is only supported for DSpark with a **confidence head**.
Adaptive verification is supported for model-based drafters (EAGLE, MTP, DFlash, DSpark and standalone draft models). The per-position acceptance estimates come from one of two sources:
- **Confidence head**: DSpark checkpoints that ship a trained confidence head use its predictions directly.
- **Online acceptance estimator**: every other drafter predicts acceptance from its draft logits with a small logistic model, recalibrated during serving against the target's verification results.
## Usage
It is off by default. Enable it in the speculative config:
It is off by default. Enable it in the speculative config with `enable_adaptive_verification`. For example, with DSpark:
```bash
vllm serve deepseek-ai/DeepSeek-V4-Flash-DSpark \
@@ -30,6 +33,21 @@ vllm serve deepseek-ai/DeepSeek-V4-Flash-DSpark \
}'
```
Or with DFlash on MiMo-V2.5-Pro, whose drafter ships in the checkpoint's `dflash/` subdirectory:
```bash
hf download XiaomiMiMo/MiMo-V2.5-Pro-FP4-DFlash --local-dir MiMo-V2.5-Pro-FP4-DFlash
vllm serve MiMo-V2.5-Pro-FP4-DFlash \
--tensor-parallel-size 4 --trust-remote-code --max-model-len auto \
--speculative-config '{
"method": "dflash",
"model": "MiMo-V2.5-Pro-FP4-DFlash/dflash",
"num_speculative_tokens": 8,
"enable_adaptive_verification": true
}'
```
Set `enable_adaptive_verification: false` to verify the full block for every request.
## Requirements and limitations
@@ -37,6 +55,7 @@ Set `enable_adaptive_verification: false` to verify the full block for every req
- The attention backend must tolerate device-decided query lengths, since the CPU lengths only bound them from above. Backends that plan off the CPU lengths are excluded by the attention selector, and rejected at startup for models that hard-wire their backend.
- Full cudagraphs are required: step costs are profiled from captured graphs, so `--enforce-eager` is rejected at startup.
- Not supported with LoRA (the per-token LoRA mapping is built from CPU-side boundaries) or pipeline parallelism (cost curves and confidences exist only on the last rank).
- Without a confidence head, `use_local_argmax_reduction` is rejected. The estimator needs the full draft logits, which local argmax reduction never materializes.
## Tuning the cost profile
-48
View File
@@ -1,48 +0,0 @@
# MLP Draft Models
The following code configures vLLM to use speculative decoding where proposals are generated by draft models that condition draft predictions on both context vectors and sampled tokens. For more information, see [The Hitchhiker's Guide to Speculative Decoding](https://pytorch.org/blog/hitchhikers-guide-speculative-decoding/) and [IBM Research's Technical Report](https://arxiv.org/abs/2404.19124).
## MLP Drafter Example
```python
from vllm import LLM, SamplingParams
prompts = ["The future of AI is"]
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
llm = LLM(
model="meta-llama/Meta-Llama-3.1-8B-Instruct",
tensor_parallel_size=1,
speculative_config={
"model": "ibm-ai-platform/llama3-8b-accelerator",
"draft_tensor_parallel_size": 1,
"method": "mlp_speculator",
},
)
outputs = llm.generate(prompts, sampling_params)
for output in outputs:
prompt = output.prompt
generated_text = output.outputs[0].text
print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
```
!!! warning "Known issue"
`ibm-ai-platform/llama3-70b-accelerator` can fail with:
`AttributeError: 'MLPSpeculatorConfig' object has no attribute 'num_attention_heads'`.
Track status in [#34106](https://github.com/vllm-project/vllm/issues/34106)
and [#34163](https://github.com/vllm-project/vllm/pull/34163).
## Pre-Trained MLP Drafter Models
A variety of speculative models of this type are available on HF hub:
- [llama-13b-accelerator](https://huggingface.co/ibm-ai-platform/llama-13b-accelerator)
- [llama3-8b-accelerator](https://huggingface.co/ibm-ai-platform/llama3-8b-accelerator)
- [codellama-34b-accelerator](https://huggingface.co/ibm-ai-platform/codellama-34b-accelerator)
- [llama2-70b-accelerator](https://huggingface.co/ibm-ai-platform/llama2-70b-accelerator)
- [llama3-70b-accelerator](https://huggingface.co/ibm-ai-platform/llama3-70b-accelerator)
- [granite-3b-code-instruct-accelerator](https://huggingface.co/ibm-granite/granite-3b-code-instruct-accelerator)
- [granite-8b-code-instruct-accelerator](https://huggingface.co/ibm-granite/granite-8b-code-instruct-accelerator)
- [granite-7b-instruct-accelerator](https://huggingface.co/ibm-granite/granite-7b-instruct-accelerator)
- [granite-20b-code-instruct-accelerator](https://huggingface.co/ibm-granite/granite-20b-code-instruct-accelerator)
@@ -12,6 +12,17 @@ Speculators provides the following key features:
- **Standardized, extensible format**: Provides a Hugging Face-compatible format for defining speculative models, with tools to convert from external research repositories into a standard speculators format for easy adoption.
- **Seamless vLLM Integration**: Built for direct deployment into vLLM, enabling low-latency, production-grade inference with minimal overhead.
## Verifier model references
When serving a Speculators checkpoint directly, `speculators_config.verifier.name_or_path` must be a remote model repository ID. References to local files or directories, including local paths that look like repository IDs, are rejected.
To use a local verifier, select it explicitly and configure the Speculators checkpoint as the draft model:
```bash
vllm serve /path/to/verifier --speculative-config \
'{"model": "/path/to/speculator", "method": "eagle3", "num_speculative_tokens": 1}'
```
## Why use Speculators?
Large language models generate text one token at a time, which creates a fundamental bottleneck: each token requires a full forward pass through the model, leaving GPU compute underutilized while waiting for memory-bound operations.
+3 -1
View File
@@ -146,7 +146,7 @@ Most OpenAI-compatible clients and agent frameworks never set `strict` on their
| Value | Behavior |
| --- | --- |
| `auto` (default) | Follow the request's tool choice and per-tool strictness. Required/named choices activate structural tags; `tool_choice="auto"` activates them when at least one tool sets `strict: true`. |
| `auto` | Follow the request's tool choice and per-tool strictness. Required/named choices activate structural tags; `tool_choice="auto"` activates them when at least one tool sets `strict: true`. |
| `function` | Constrain the tool-call envelope (markup and the function name) for every request with tools, leaving argument contents free unless the client marked the tool `strict: true`. |
| `parameter` | Additionally pin argument schemas for every tool, as if every tool had `strict: true`. |
@@ -154,6 +154,8 @@ Most OpenAI-compatible clients and agent frameworks never set `strict` on their
vllm serve ... --tool-strict-level function
```
When the option is unset, the tool parser's default level applies: `function` for `glm47`, `auto` for the other parsers. Passing `--tool-strict-level auto` overrides a parser default.
The floor never relaxes a constraint the request would already receive: tools the client marked `strict: true` keep their schemas at every level. With `tool_choice="auto"`, the grammar does not force a tool call; a plain text response stays valid. `VLLM_ENFORCE_STRICT_TOOL_CALLING=false` disables structural tags entirely and takes precedence over this option.
## Automatic Function Calling
+28
View File
@@ -56,6 +56,34 @@ watermarking algorithm that does not support it natively, at the cost of
weaker detectability. See
[Speculative decoding](#speculative-decoding).
## Monitoring
When `watermark_config` is set, the Prometheus counter
`vllm:watermark_requests_total` counts finished requests by `status`:
- `watermarked`: the request was sampled with the watermark switched on.
- `skipped`: the request asked for the watermark (omitted or `true`) but could
not use it, such as `temperature=0` and trace replay requests. A beam search
request that asks for the watermark runs without it and logs a warning. Its
beam steps count as `not_watermarked`, not `skipped`.
- `not_watermarked`: all other finished requests. This includes explicit
opt-outs, pooling requests, and the internal requests of beam search (one
per beam step), generative scoring and language detection.
Each finished request has exactly one status, so the sum over `status` equals
`vllm:request_success_total` on the same instance. Requests that the client
aborts are in neither counter. The counter does not exist without
`watermark_config`. With `n > 1`, each sequence counts once. In a
disaggregated prefill/decode deployment, both instances count a request: sum
over decode instances only. For streaming input, the first input chunk sets
the status.
The status is per request, not per token. A `watermarked` request can contain
tokens without the watermark: accepted draft tokens with
`allow_target_only_watermarking`, tokens whose context repeats
(`deduplicate_contexts`), the first `context_width` generated tokens with
`deduplicate_contexts: "all"`, and tokens that a grammar forces.
## Architecture
`WatermarkConfig` selects an algorithm and PRF. Model Runner V2 constructs the
+1 -1
View File
@@ -159,7 +159,7 @@ VLLM_USE_PRECOMPILED=1 VLLM_PRECOMPILED_WHEEL_VARIANT=cpu VLLM_TARGET_DEVICE=cpu
### Which `dtype` should be used?
- Currently, vLLM CPU uses model default settings as `dtype`. However, due to unstable float16 support in torch CPU, it is recommended to explicitly set `dtype=bfloat16` if there are any performance or accuracy problem.
- On AMD Zen CPUs (`ZenCpuPlatform`), `float16` is **not** supported. Only `bfloat16` and `float32` are accepted; models declared with `float16` are auto-downcast to `bfloat16` at model load time. See [AMD Zen optimizations](#amd-zen-optimizations).
- On AMD Zen CPUs (`ZenCpuPlatform`), `float16` requires AVX512-FP16, available from Venice (6th Gen EPYC) onwards. On Turin (5th Gen EPYC) and earlier only `bfloat16` and `float32` are accepted, and models declared with `float16` are auto-downcast to `bfloat16` at model load time. See [AMD Zen optimizations](#amd-zen-optimizations).
### How to launch a vLLM service on CPU?
@@ -273,7 +273,9 @@ Otherwise, vLLM falls back to the default `CpuPlatform` (oneDNN / sgl-kernel pat
### Supported dtypes
`float16` is **not** supported on `ZenCpuPlatform`. `ZenCpuPlatform.supported_dtypes` advertises only `bfloat16` and `float32`, so models declared with `torch_dtype=float16` are auto-downcast to `bfloat16` at load time with the standard `"Your device 'cpu' doesn't support torch.float16. Falling back to torch.bfloat16 for compatibility."` warning emitted from `vllm/config/model.py`.
`ZenCpuPlatform.supported_dtypes` always advertises `bfloat16` and `float32`, and adds `float16` when `torch.ops.zentorch.zentorch_is_fp16_supported()` reports AVX512-FP16, which AMD ships from Venice (6th Gen EPYC) onwards. On Turin (5th Gen EPYC) and earlier, models declared with `torch_dtype=float16` are auto-downcast to `bfloat16` at load time with the standard `"Your device 'cpu' doesn't support torch.float16. Falling back to torch.bfloat16 for compatibility."` warning emitted from `vllm/config/model.py`.
`bfloat16` is listed first either way, so `--dtype auto` keeps resolving to it.
### Environment variables
@@ -23,14 +23,28 @@ Therefore, it is recommended to install vLLM with a **fresh new** environment. I
```bash
uv pip install vllm --torch-backend=auto
vllm download-kernels
```
??? console "pip"
```bash
# Install vLLM with CUDA 12.9.
pip install vllm --extra-index-url https://download.pytorch.org/whl/cu129
vllm download-kernels
```
`vllm download-kernels` installs [FlashInfer](https://docs.flashinfer.ai/)'s
precompiled kernels for the installed FlashInfer and CUDA versions. Without them,
vLLM downloads and compiles kernels at startup, which can add several minutes on
Hopper and newer GPUs, **especially Blackwell**, and logs a warning. Run it in the
same Python environment after every vLLM install or upgrade, and in custom
container images. The kernels are tied to the FlashInfer version, and FlashInfer
refuses to import when they do not match, so an upgrade needs them refreshed.
With a PyTorch built for CUDA older than 12.9, only `flashinfer-cubin` is
installed, because FlashInfer does not publish `flashinfer-jit-cache` for it.
For CUDA overrides and nightly kernels, use the
[FlashInfer CLI](https://docs.flashinfer.ai/cli.html#download-kernels) directly.
We recommend leveraging `uv` to [automatically select the appropriate PyTorch index at runtime](https://docs.astral.sh/uv/guides/integration/pytorch/#automatic-backend-selection) by inspecting the installed CUDA driver version via `--torch-backend=auto` (or `UV_TORCH_BACKEND=auto`). To select a specific backend (e.g., `cu130`), set `--torch-backend=cu130` (or `UV_TORCH_BACKEND=cu130`). If this doesn't work, try running `uv self update` to update `uv` first.
!!! note
@@ -94,9 +108,10 @@ If you only need to change Python code, you can build and install vLLM without c
git clone https://github.com/vllm-project/vllm.git
cd vllm
VLLM_USE_PRECOMPILED=1 uv pip install --editable . --torch-backend=auto
vllm download-kernels
```
This command will do the following:
The install command will do the following:
1. Look for the current branch in your vLLM clone.
1. Identify the corresponding base commit in the main branch.
+5 -2
View File
@@ -25,8 +25,11 @@ This guide will help you quickly get started with vLLM to perform:
uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install vllm --torch-backend=auto
vllm download-kernels
```
`vllm download-kernels` installs precompiled kernels that vLLM otherwise downloads and compiles at startup, which can add several minutes on Hopper and newer GPUs. See the [installation guide](installation/gpu.md) for details.
`uv` can [automatically select the appropriate PyTorch index at runtime](https://docs.astral.sh/uv/guides/integration/pytorch/#automatic-backend-selection) by inspecting the installed CUDA driver version via `--torch-backend=auto` (or `UV_TORCH_BACKEND=auto`). To select a specific backend (e.g., `cu126`), set `--torch-backend=cu126` (or `UV_TORCH_BACKEND=cu126`).
Another delightful way is to use `uv run` with `--with [dependency]` option, which allows you to run commands such as `vllm serve` without creating any permanent environment:
@@ -329,5 +332,5 @@ Some of the available backend options include:
- On AMD ROCm: `TRITON_ATTN`, `ROCM_ATTN`, `ROCM_AITER_FA`, `ROCM_AITER_UNIFIED_ATTN`, `TRITON_MLA`, `ROCM_AITER_MLA` or `ROCM_AITER_TRITON_MLA`.
- On Intel XPU: `FLASH_ATTN`, `TRITON_ATTN`, `TRITON_MLA`, `XPU_MLA_SPARSE`, `TORCH_SDPA` or `TURBOQUANT`.
!!! warning
There are no pre-built vllm wheels containing Flash Infer, so you must install it in your environment first. Refer to the [Flash Infer official docs](https://docs.flashinfer.ai/) or see [docker/Dockerfile](../../docker/Dockerfile) for instructions on how to install it.
!!! tip
FlashInfer is installed with vLLM, but its precompiled kernels are not. Run `vllm download-kernels` after installing or upgrading vLLM so they are not downloaded and compiled at startup.
@@ -145,6 +145,7 @@ def auto_mock(module_name: str, attr: str, max_mocks: int = 100):
)
bench_forward = auto_mock("vllm.benchmarks.forward", "main")
bench_latency = auto_mock("vllm.benchmarks", "latency")
bench_serve = auto_mock("vllm.benchmarks", "serve")
bench_startup = auto_mock("vllm.benchmarks", "startup")
@@ -178,6 +179,9 @@ LaunchSubcommandBase = auto_mock("vllm.entrypoints.cli.launch", "LaunchSubcomman
launch_description = auto_mock("vllm.entrypoints.cli.launch", "DESCRIPTION")
RenderSubcommand = auto_mock("vllm.entrypoints.cli.launch", "RenderSubcommand")
PreloadSubcommand = auto_mock("vllm.entrypoints.cli.preload", "PreloadSubcommand")
DownloadKernelsSubcommand = auto_mock(
"vllm.entrypoints.cli.download_kernels", "DownloadKernelsSubcommand"
)
sweep_subcommands = auto_mock("vllm.benchmarks.sweep.cli", "SUBCOMMANDS")
openai_cli_args = auto_mock("vllm.entrypoints.launchers", "cli_args")
run_batch = auto_mock("vllm.entrypoints.launchers", "run_batch")
@@ -332,7 +336,12 @@ pages = {
"cli/complete.md": (create_parser(CompleteCommand.add_cli_args), False),
"cli/run-batch.md": (create_parser(run_batch.make_arg_parser), True),
"cli/preload.md": (create_parser(PreloadSubcommand.add_cli_args), True),
"cli/download-kernels.md": (
create_parser(DownloadKernelsSubcommand.add_cli_args),
False,
),
"cli/launch/render.md": (create_parser(RenderSubcommand.add_cli_args), True),
"cli/bench/forward.md": (create_parser(bench_forward.add_cli_args), True),
"cli/bench/latency.md": (create_parser(bench_latency.add_cli_args), True),
# URL kept as `mm_processor` for back-compat; command name is `mm-processor`
"cli/bench/mm_processor.md": (
@@ -38,7 +38,6 @@ def title(text: str) -> str:
"openai": "OpenAI",
"lmcache": "LMCache",
"multilora": "MultiLoRA",
"mlpspeculator": "MLPSpeculator",
r"fp\d+": lambda x: x.group(0).upper(), # e.g. fp16, fp32
r"int\d+": lambda x: x.group(0).upper(), # e.g. int8, int16
}
@@ -51,6 +51,20 @@ vllm serve s3://core-llm/Llama-3-8b \
--load-format runai_streamer
```
## Model-specific checkpoint selection
For models exposing `is_unused_checkpoint_weight`, `runai_streamer` skips
safetensors files containing only unused weights before streaming. It uses the
same model-owned rule as the default loader, including MTP draft models with
this hook; it does not assume a particular MTP tensor prefix.
Object-storage checkpoints use `model.safetensors.index.json` and verify excluded
shards against their safetensors headers before streaming. Mixed shards remain
included. Missing or inconsistent metadata retains the original file list.
Models without a selection rule keep all files. Distributed streaming with
pipeline parallelism also retains the full file list. This optimization does
not change `runai_streamer_sharded`.
## Tunable parameters
You can tune parameters using `--model-loader-extra-config`:
+1 -1
View File
@@ -127,7 +127,7 @@ If the model doesn't have a chat template or you want to specify another one,
you can explicitly pass a chat template:
```python
from vllm.entrypoints.chat_utils import load_chat_template
from vllm.renderers.chat_utils import load_chat_template
# You can find a list of existing chat templates under `examples/`
custom_template = load_chat_template(chat_template="<path_to_template>")
+3 -3
View File
@@ -37,7 +37,7 @@ You can compute pairwise similarity scores to build a similarity matrix using th
| Architecture | Models | Example HF Models | [LoRA](../../features/lora.md) | [PP](../../serving/parallelism_scaling.md) |
| ------------ | ------ | ----------------- | ------------------------------ | ------------------------------------------ |
| `BertModel` | BERT-based | `BAAI/bge-base-en-v1.5`, `Snowflake/snowflake-arctic-embed-xs`, etc. | | |
| `BertModel` | BERT-based | `BAAI/bge-base-en-v1.5`, `Snowflake/snowflake-arctic-embed-xs`, etc. | ✅︎ | |
| `BertSpladeSparseEmbeddingModel` | SPLADE | `naver/splade-v3` | | |
| `BgeM3EmbeddingModel` | BGE-M3 | `BAAI/bge-m3` | | |
| `Gemma2Model`<sup>C</sup> | Gemma 2-based | `BAAI/bge-multilingual-gemma2`, etc. | ✅︎ | ✅︎ |
@@ -51,9 +51,9 @@ You can compute pairwise similarity scores to build a similarity matrix using th
| `NomicBertModel` | Nomic BERT | `nomic-ai/nomic-embed-text-v1`, `nomic-ai/nomic-embed-text-v2-moe`, `Snowflake/snowflake-arctic-embed-m-long`, etc. | | |
| `Qwen2Model`<sup>C</sup>, `Qwen2ForCausalLM`<sup>C</sup> | Qwen2-based | `ssmits/Qwen2-7B-Instruct-embed-base` (see note), `Alibaba-NLP/gte-Qwen2-7B-instruct` (see note), etc. | ✅︎ | ✅︎ |
| `Qwen3Model`<sup>C</sup>, `Qwen3ForCausalLM`<sup>C</sup> | Qwen3-based | `Qwen/Qwen3-Embedding-0.6B`, `ai-sage/Giga-Embeddings-instruct-480M-0826`<sup>^</sup>, etc. | ✅︎ | ✅︎ |
| `RobertaModel`, `RobertaForMaskedLM` | RoBERTa-based | `sentence-transformers/all-roberta-large-v1`, etc. | | |
| `RobertaModel`, `RobertaForMaskedLM` | RoBERTa-based | `sentence-transformers/all-roberta-large-v1`, etc. | ✅︎ | |
| `VoyageQwen3BidirectionalEmbedModel`<sup>C</sup> | Voyage Qwen3-based with bidirectional attention | `voyageai/voyage-4-nano`, etc. | ✅︎ | ✅︎ |
| `XLMRobertaModel` | XLMRobertaModel-based | `BAAI/bge-m3` (see note), `intfloat/multilingual-e5-base`, `jinaai/jina-embeddings-v3` (see note), etc. | | |
| `XLMRobertaModel` | XLMRobertaModel-based | `BAAI/bge-m3` (see note), `intfloat/multilingual-e5-base`, `jinaai/jina-embeddings-v3` (see note), etc. | ✅︎ | |
| `*Model`<sup>C</sup>, `*ForCausalLM`<sup>C</sup>, etc. | Generative models | N/A | \* | \* |
!!! note
+34
View File
@@ -90,6 +90,7 @@ The following [pooling parameters][vllm.PoolingParams] are supported.
```python
--8<-- "vllm/pooling_params.py:common-pooling-params"
--8<-- "vllm/pooling_params.py:embed-pooling-params"
--8<-- "vllm/pooling_params.py:token-embed-pooling-params"
```
### `LLM.encode`
@@ -108,6 +109,39 @@ data = output.outputs.data
print(f"Data: {data!r}")
```
### Fixed-length late chunking
Late chunking encodes the full input before averaging contextual token representations within each chunk. The chunk means pass through the model's embedding head, including its projection and optional normalization.
This mode is currently available through the offline `LLM.encode` API for plain-text input. With an instance configured for `token_embed`, enable it per request:
```python
from vllm import PoolingParams
from vllm.pooling_params import LateChunkingParams
outputs = llm.encode(
[document],
pooling_task="token_embed",
pooling_params=PoolingParams(
late_chunking_params=LateChunkingParams(chunk_size=256)
),
)
vectors = outputs[0].outputs.data # [num_chunks, embedding_dimension]
chunks = outputs[0].late_chunking.chunks
```
For `N` model-input tokens and chunk size `C`, the output contains `ceil(N / C)` vectors. The last chunk uses its actual length. Special tokens and instruction prefixes count toward both chunk boundaries and means. Omitting `late_chunking_params` preserves ordinary token embeddings.
Each vector row corresponds to one entry in `chunks`. Its `token_range` and `char_range` are zero-based, half-open ranges. Character ranges index the submitted Python string; use `document[start:end]` to retrieve the source text. A chunk containing only special tokens has `char_range=None`.
Use nonempty text within the model's context window and a fast tokenizer that provides aligned source offsets. Input truncation, explicit padding and renderer text normalization are not supported. The custom document/query outputs of `JinaForRanking` are also unsupported. Late chunking does not extend the model's context window or reduce its attention computation.
The [offline example](../../../examples/pooling/token_embed/late_chunking_offline.py) loads an E5 model and prints the vector shape, token ranges and source excerpts for two documents:
```bash
python examples/pooling/token_embed/late_chunking_offline.py --chunk-size 16
```
### `LLM.score`
The [score][vllm.entrypoints.pooling.offline.PoolingOfflineMixin.score] method outputs similarity scores between sentence pairs.
+358
View File
@@ -0,0 +1,358 @@
# Structured Decisions Models
Structured Decisions Models (aka JEV Models) map inputs directly to predefined
options or categories and return probabilities and confidence scores, commonly
used for classification and routing.
## Supported Models
### Text-only Models
| Architecture | ReadStrategy | Task | Head | Example HF Models |
|---------------------------|---------------------|------------------|-------------------|-------------------------|
| `ClefFlashForJointSchema` | `ClefFlashStrategy` | `token_classify` | `JointSchemaHead` | `Cloudflare/clef-flash` |
| `Qwen3ForCausalLM` | `NextTokenStrategy` | `generate` | N/A | `Qwen/Qwen3-0.6B`, etc. |
### Multimodal Models
!!! note
For more information about multimodal models inputs, see [this page](./supported_models.md#list-of-multimodal-language-models).
| Architecture | Inputs | ReadStrategy | Task | Head | Example HF Models |
|--------------------------------------|-------------------------------------|--------------------------------|------------------|-------------------|------------------------------------------|
| `ClefOmniForJointSchema` | T + I + V | `JointSchemaStrategy` | `token_classify` | `JointSchemaHead` | `Cloudflare/clef-omni` |
| `D3ForSequenceClassification` | T + I + V | `D3Strategy` | `classify` | Linear (`score`) | `vllm-sr/d3`, `vllm-sr/d3-mini`, etc. |
| `Lfm2VlForConditionalGeneration` | T + I<sup>+</sup> | `LiquidStrategy` | `generate` | N/A | `LiquidAI/d1-3B` |
| `Qwen3_5ForConditionalGeneration` | T + I<sup>E+</sup> + V<sup>E+</sup> | `NextTokenStrategy` | `generate` | N/A | `Qwen/Qwen3.5-0.8B`, etc. |
| `Qwen3_5MoeForConditionalGeneration` | T + I<sup>E+</sup> + V<sup>E+</sup> | `NextTokenStrategy` | `generate` | N/A | `Qwen/Qwen3.5-35B-A3B`, etc. |
| `DiffusionGemmaForBlockDiffusion` | T + I<sup>+</sup> + V<sup>+</sup> | `DiffusionGemmaCanvasStrategy` | `generate` | N/A | `google/diffusiongemma-26B-A4B-it`, etc. |
## Online Serving
### TypeSafe System One Endpoint
The `/v1/systemone` endpoint answers a set of typed questions about a state and
returns a probability for every allowed answer.
The endpoint is available by default on generation models. Models without a
supported read strategy return 501. Supported architectures require
`--logprobs-mode raw_logprobs` or `processed_logprobs`, since `label_mass` needs
log probabilities.
#### How it works
1. Each question is one read. The user message is the state, followed by the
question and its options with the type's labels, and an instruction to
answer with one label.
2. The read is the reply's first token. Each label must be one token there,
or the request returns 400.
3. The read returns the logprobs of that question's label tokens. Softmax over
the labels gives the answer's probabilities.
Thinking is off unless `chat_template_kwargs` turns it on.
Probabilities can differ between identical requests. They depend on which
requests share a batch and on whether a read hits the prefix cache. Set
`VLLM_BATCH_INVARIANT=1` for reads that repeat exactly, at a cost in
throughput.
Every read of a request starts with the state, so with
`--enable-prefix-caching` the state is prefilled once and each further
question prefills only its own text, then reads one token.
#### Question types
| type | criteria | answer |
| --- | --- | --- |
| `choice` | map of option name to a description or `null` | `choice`, `probabilities` by option name, `confidence` |
| `noul` | `null`, or an object with optional `true` and `false` descriptions | `noul` (probability of yes), `probabilities`, `confidence` |
| `score` | ordered list of level names | `score` (expected 0-indexed level), `legend`, `probabilities` by level index, `confidence` |
A choice has at least one option, a score at least two levels. A question id
is a non-empty string.
#### Example
```bash
curl -s http://localhost:8000/v1/systemone \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen3-0.6B",
"state": {"ticket": "My card was charged twice for one order."},
"questions": {
"team": {
"type": "choice",
"instructions": "Which team should handle this ticket?",
"criteria": {
"billing": "payments and refunds",
"shipping": "deliveries",
"security": "account access"
}
},
"urgent": {
"type": "noul",
"instructions": "Does the customer need a reply within the hour?",
"criteria": {"true": "the account is locked or money is moving"}
},
"tone": {
"type": "score",
"instructions": "How upset is the customer?",
"criteria": ["calm", "annoyed", "furious"]
}
},
"chat_template_kwargs": {"enable_thinking": false}
}'
```
Response, with probabilities that depend on the model:
```json
{
"id": "decision-...",
"object": "structured_decision",
"created": 1790000000,
"model": "Qwen/Qwen3-0.6B",
"answers": {
"team": {
"type": "choice",
"choice": "billing",
"probabilities": {"billing": 0.97, "shipping": 0.01, "security": 0.02},
"confidence": 0.90
},
"urgent": {
"type": "noul",
"noul": 0.02,
"probabilities": {"yes": 0.02, "no": 0.98},
"confidence": 0.95
},
"tone": {
"type": "score",
"score": 0.6,
"legend": {"0": "calm", "1": "annoyed", "2": "furious"},
"probabilities": {"0": 0.7, "1": 0.2, "2": 0.1},
"confidence": 0.65
}
},
"usage": {"input_tokens": 142, "output_tokens": 3},
"diagnostics": {
"team": {"label_mass": 0.93, "argmax_is_label": true},
"urgent": {"label_mass": 0.98, "argmax_is_label": true},
"tone": {"label_mass": 0.91, "argmax_is_label": true}
}
}
```
`diagnostics.label_mass` is the probability the model put on the labels across
its whole vocabulary. A low value means most of the probability went to tokens
that are not labels. `confidence` is the chosen option's probability over the
whole vocabulary: its share of the labels times `label_mass`.
#### Labels
Labels follow the type: a `choice` labels its options `A` to `Z` in the order
the request lists them, a `noul` labels its options `yes` and `no`, and a
`score` labels its levels `0` to `9`, so a level's label is its score.
#### Limits
| limit | value |
| --- | --- |
| questions per request | 64 |
| options per `choice` | 26 |
| levels per `score` | 10 |
#### Request fields
| field | meaning |
| --- | --- |
| `state` | what the questions are about: a string, or JSON that is sent as its JSON text |
| `questions` | question id to `{type, instructions, criteria}`, asked in this order |
| `instructions` | optional context placed ahead of the questions |
| `chat_template_kwargs` | passed to the chat template, for example `{"enable_thinking": false}` |
| `cache_salt` | optional [prefix-cache salt](../usage/security.md#cache-salting) applied to every question |
| `images`, `audio`, `videos` | optional media URLs or data URLs, for models that take them ([Clef-Flash](#clef-flash), images/video; [Clef-Omni](#clef-omni); [Decision 3.0](#decision-30), images/video; [Liquid d1](#liquid-d1), images only) |
A question with any field other than `type`, `instructions` and `criteria` is
rejected with a 400.
When [tracing](../../examples/observability/opentelemetry/README.md) is enabled
with `--otlp-traces-endpoint`, the HTTP `traceparent` and `tracestate` headers are
forwarded to every question's generation request.
### OpenAI Decisions API
vLLM provides a text-only implementation of OpenAI's
[`POST /v1/decisions`](https://developers.openai.com/api/docs/guides/decisions)
API. Start a supported model:
```bash
vllm serve Qwen/Qwen3-0.6B
```
```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
decision = client.decisions.create(
model="Qwen/Qwen3-0.6B",
input="The package arrived with a broken screen.",
questions=[
{
"type": "predicate",
"name": "damaged",
"instructions": "Does the customer report a damaged item?",
},
{
"type": "choice",
"name": "department",
"instructions": "Which department should handle this complaint?",
"choices": [{"value": "returns"}, {"value": "billing"}],
},
{
"type": "score",
"instructions": "How severe is the damage?",
"levels": [{"label": "cosmetic"}, {"label": "broken"}],
},
],
)
print(decision.answers)
```
Use OpenAI Python SDK 3.26.0 or newer for `client.decisions.create`.
`input` accepts a string or an array of user messages with string content or
`input_text` parts. Answers retain question order and echo `name`, including
`null` for unnamed questions. Choice values preserve strings and booleans.
Scores are probability-weighted averages of zero-based level indices.
The request contract follows the OpenAI OpenAPI schemas retrieved on
October 7, 2026. This MVP supports 1–64 questions, 2–26 choices, and 2–10 score
levels. Upstream permits 200 questions and 255 choices; this backend returns
a 400 when either model limit is exceeded.
The optional `safety_identifier` is caller metadata, available in request
body logging with `--enable-log-requests` and DEBUG logging; it does not
authenticate callers.
#### Implementation and limits
This MVP uses the same label-reading backend as
[structured decisions](structured_decisions_models.md). It supports generation
architectures with a registered read strategy, including Qwen, DiffusionGemma,
and Liquid d1; unsupported models return 501. Each question reads a model's
label tokens. Usage reports the actual prompt and output tokens across these
reads, including prefix-cache counters.
Probabilities are normalized over the supplied options. `confidence` is the
winning label's probability over the full vocabulary; it is not calibrated
to OpenAI's models. Thinking is disabled for these reads.
Image inputs, other message roles, tools, streaming, and per-question refusal
scoring are outside this MVP. Unsupported request fields and input types are
rejected. The `/v1/systemone` endpoint retains its existing request format and
is available by default.
## Examples
### Clef-Flash
[Clef-Flash](https://huggingface.co/Cloudflare/clef-flash) uses a Qwen3.5-9B
backbone and a joint schema head to score all questions in one forward pass.
vLLM detects the head and serves the checkpoint as a pooling model.
```bash
vllm serve Cloudflare/clef-flash --max-model-len 16384
```
Send requests to `/v1/systemone` using `state` and `questions`, as in the
examples above. `choice`, `noul` and `score` questions are supported.
- Text and JSON states use the checkpoint's fixed prompt. JSON is compact
with sorted keys. `instructions` returns 400; `chat_template_kwargs`
has no effect.
- The state is truncated to fit 16,384 tokens, or `max_model_len` if smaller.
A schema that exceeds this bound returns 400.
- `images` and `videos` are placed before the state. Videos are sampled at
2 fps and processed by the Qwen3.5 vision processor. Audio inputs return
400, and video soundtracks are ignored.
- Each request performs one pooling read. `usage.input_tokens` counts the
prompt once, `output_tokens` is 0, and `label_mass` is 1.
### Clef-Omni
[Clef-Omni](https://huggingface.co/Cloudflare/clef-omni) is a decision model:
a joint schema head on the Qwen3-Omni thinker scores every option of every
question in one forward pass. vLLM serves it as a pooling model.
```bash
vllm serve Cloudflare/clef-omni
```
- A request is one read. The prompt is the one the checkpoint's
`joint_schema_model.py` builds: the state, then every question with its
options. A JSON state is written as compact JSON with sorted keys.
- A state too long for the model is truncated to fit, as the reference does.
A request whose questions alone exceed 64,000 tokens, or `max_model_len`,
returns 400.
- `instructions` returns 400, since the prompt is fixed. `chat_template_kwargs`
has no effect.
- There are no labels, so `label_mass` is 1 and `confidence` is the chosen
option's probability. `usage.input_tokens` counts the prompt once, and
`output_tokens` is 0.
- `images`, `audio` and `videos` are read as the reference does: placed
before the state, videos sampled at 2 fps, and a video's soundtrack heard
with it when every video has one. Audio clips cannot be sent together with
videos that have soundtracks.
### Decision 3.0
The Decision 3.0 models ([`vllm-sr/d3`](https://huggingface.co/vllm-sr/d3) and
its smaller variants such as `vllm-sr/d3-mini` and `vllm-sr/d3-edge`) are a
Qwen3.5 backbone with an answer-code readout: one forward pass per question,
and the readout scores the question's options at the last token. vLLM detects
the checkpoint's `decision_config.json` and serves it as a classification
model.
```bash
vllm serve vllm-sr/d3
```
- Each question is one read of the prompt the checkpoint's `d3_format.py`
writes: the state, the question and one line per option under an answer
code. A question's probabilities are the softmax of its options' code
logits at the checkpoint's temperature, so picks and probabilities match
`model.system_one()`. `confidence` and `usage` follow the definitions on
this page instead.
- A choice can have up to 255 options. A JSON state is written as JSON text,
and an empty state as `(empty)`.
- The full attention layers read the whole prompt in both directions, as the
checkpoint declares, so chunked prefill and prefix caching are off.
- `instructions` returns 400, since the prompt is fixed.
`chat_template_kwargs` has no effect.
- `images` and `videos` are placed before the text, images read at up to
1.6 megapixels and videos at 2 fps, at most 32 frames of up to
0.2 megapixels each, as the checkpoint's runtime does. The media are read
once and shared by every question of the request.
- `/classify` and `/pooling` return the raw logits of all 255 codes; only the
first codes, one per option, belong to a question.
### Liquid d1
[`LiquidAI/d1-3B`](https://huggingface.co/LiquidAI/d1-3B) is LFM2.5-VL-3B
post-trained to answer decisions as the first token of its reply. vLLM serves
it as a generative model.
```bash
vllm serve LiquidAI/d1-3B
```
- The server writes the prompt and reads the label tokens the way the
`prompt.py` shipped with the checkpoint does, so picks and probabilities
match `model.system_one()`. `confidence` and `usage` follow the definitions
on this page instead.
- A JSON state is written as indented JSON. A `null` state leaves the images,
or the question alone, as the prompt.
- `images` are placed before the state and shrunk to at most 1024 x 1024
pixels, as the reference does.
- A `choice` whose option names are all single letters is labeled with those
letters.
- When each request carries one question over a state that changes from
request to request, `--disable-prefix-caching` lowers latency.
+4 -2
View File
@@ -372,6 +372,7 @@ th {
| Architecture | Models | Example HF Models | [LoRA](../features/lora.md) | [PP](../serving/parallelism_scaling.md) |
| ------------ | ------ | ----------------- | -------------------- | ------------------------- |
| `AfmoeForCausalLM` | Afmoe | TBA | ✅︎ | ✅︎ |
| `AliceAIForCausalLM` | AliceAI | `yandex/AliceAI-Foundation-80B-A3B-Base` | ✅︎ | |
| `ApertusForCausalLM` | Apertus | `swiss-ai/Apertus-8B-2509`, `swiss-ai/Apertus-70B-Instruct-2509`, etc. | ✅︎ | ✅︎ |
| `AXK1ForCausalLM` | A.X-K1 | `skt/A.X-K1`, etc. | | ✅︎ |
| `BailingMoeForCausalLM` | Ling | `inclusionAI/Ling-lite-1.5`, `inclusionAI/Ling-plus`, etc. | ✅︎ | ✅︎ |
@@ -486,6 +487,7 @@ Some models are supported only via the [Transformers modeling backend](#transfor
| `HunYuanDenseV1ForCausalLM` | Hunyuan Dense | `tencent/Hunyuan-7B-Instruct` | ✅︎ | ✅︎ |
| `HunYuanMoEV1ForCausalLM` | Hunyuan-A13B | `tencent/Hunyuan-A13B-Instruct`, `tencent/Hunyuan-A13B-Pretrain`, `tencent/Hunyuan-A13B-Instruct-FP8`, etc. | ✅︎ | ✅︎ |
| `Jais2ForCausalLM` | Jais2 | `inceptionai/Jais-2-8B-Chat`, `inceptionai/Jais-2-70B-Chat`, etc. | ✅︎ | ✅︎ |
| `Kolibri1ForCausalLM` | Kolibri 1 | `Aleph-Alpha/Kolibri-1`, `Aleph-Alpha/Kolibri-1-BF16`, etc. | ✅︎ | ✅︎ |
| `MellumForCausalLM` | Mellum 2 | `JetBrains/Mellum2-12B-A2.5B-Base`, etc. | ✅︎ | ✅︎ |
| `NanbeigeForCausalLM` | Nanbeige4.2 | `Nanbeige/Nanbeige4.2-3B`, etc. | ✅︎ | ✅︎ |
| `OlmoForCausalLM` | OLMo | `allenai/OLMo-1B-hf`, `allenai/OLMo-7B-hf`, etc. | ✅︎ | ✅︎ |
@@ -580,7 +582,7 @@ These models primarily accept the [`LLM.generate`](./generative_models.md#llmgen
| `KimiK25ForConditionalGeneration` | Kimi-K2.5 | T + I<sup>+</sup> | `moonshotai/Kimi-K2.5` | | ✅︎ |
| `KimiK3ForConditionalGeneration` | Kimi-K3 | T + I<sup>+</sup> | `moonshotai/Kimi-K3` | | ✅︎ |
| `KimiVLForConditionalGeneration` | Kimi-VL-A3B-Instruct, Kimi-VL-A3B-Thinking | T + I<sup>+</sup> | `moonshotai/Kimi-VL-A3B-Instruct`, `moonshotai/Kimi-VL-A3B-Thinking` | | ✅︎ |
| `LightOnOCRForConditionalGeneration` | LightOnOCR-1B | T + I<sup>+</sup> | `lightonai/LightOnOCR-1B`, etc | ✅︎ | ✅︎ |
| `LightOnOCRForConditionalGeneration` | LightOnOCR-1B | T + I<sup>E+</sup> | `lightonai/LightOnOCR-1B`, etc | ✅︎ | ✅︎ |
| `Lfm2VlForConditionalGeneration` | LFM2-VL | T + I<sup>+</sup> | `LiquidAI/LFM2-VL-450M`, `LiquidAI/LFM2-VL-3B`, `LiquidAI/LFM2-VL-8B-A1B`, etc. | ✅︎ | ✅︎ |
| `Llama4ForConditionalGeneration` | Llama 4 | T + I<sup>+</sup> | `meta-llama/Llama-4-Scout-17B-16E-Instruct`, `meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8`, `meta-llama/Llama-4-Maverick-17B-128E-Instruct`, etc. | ✅︎ | ✅︎ |
| `Llama_Nemotron_Nano_VL` | Llama Nemotron Nano VL | T + I<sup>E+</sup> | `nvidia/Llama-3.1-Nemotron-Nano-VL-8B-V1` | ✅︎ | ✅︎ |
@@ -592,7 +594,7 @@ These models primarily accept the [`LLM.generate`](./generative_models.md#llmgen
| `MiDashengLMModel` | MiDashengLM | T + A<sup>+</sup> | `mispeech/midashenglm-7b` | | ✅︎ |
| `MiMoV2OmniForCausalLM` | MiMo-V2.5-Omni | T + I<sup>E+</sup> + V<sup>E+</sup> + A<sup>+</sup> | `XiaomiMiMo/MiMo-V2.5-Omni` | | ✅︎ |
| `MiniCPMO` | MiniCPM-O | T + I<sup>E+</sup> + V<sup>E+</sup> + A<sup>E+</sup> | `openbmb/MiniCPM-o-2_6`, etc. | ✅︎ | ✅︎ |
| `MiniCPMV` | MiniCPM-V | T + I<sup>E+</sup> + V<sup>E+</sup> | `openbmb/MiniCPM-V-2` (see note), `openbmb/MiniCPM-Llama3-V-2_5`, `openbmb/MiniCPM-V-2_6`, `openbmb/MiniCPM-V-4`, `openbmb/MiniCPM-V-4_5`, `openbmb/MiniCPM-V-4_6`, etc. | ✅︎ | |
| `MiniCPMV` | MiniCPM-V | T + I<sup>E+</sup> + V<sup>E+</sup> | `openbmb/MiniCPM-V-2` (see note), `openbmb/MiniCPM-Llama3-V-2_5`, `openbmb/MiniCPM-V-2_6`, `openbmb/MiniCPM-V-4`, `openbmb/MiniCPM-V-4_5`, `openbmb/MiniCPM-V-4_6`, `openbmb/MiniCPM-V-4_7`, etc. | ✅︎ | |
| `MiniMaxM3SparseForConditionalGeneration` | MiniMax-M3 | T + I<sup>+</sup> + V<sup>+</sup> | `MiniMaxAI/MiniMax-M3`, `MiniMaxAI/MiniMax-M3-MXFP8`, etc. | | ✅︎ |
| `MiniMaxVL01ForConditionalGeneration` | MiniMax-VL | T + I<sup>E+</sup> | `MiniMaxAI/MiniMax-VL-01`, etc. | | ✅︎ |
| `Mistral3ForConditionalGeneration` | Mistral3 (HF Transformers) | T + I<sup>+</sup> | `mistralai/Mistral-Small-3.1-24B-Instruct-2503`, etc. | ✅︎ | ✅︎ |
+3 -1
View File
@@ -16,6 +16,8 @@ We currently support the following OpenAI APIs:
- [Chat Completions batch API](./openai_compatible_server.md#chat-api) (`/v1/chat/completions/batch`)
- [Responses API](./openai_compatible_server.md#responses-api) (`/v1/responses`, `/v1/responses/{response_id}`, `/v1/responses/{response_id}/cancel`)
- Only applicable to [text generation models](../../models/generative_models.md).
- [Decisions API](../../models/structured_decisions_models.md#openai-decisions-api) (`/v1/decisions`)
- Supports text-only predicate, choice, and score questions on supported generation models.
- [Embeddings API](../../models/pooling_models/embed.md#openai-compatible-embeddings-api) (`/v1/embeddings`)
- Only applicable to [embedding models](../../models/pooling_models/embed.md).
- [Transcriptions API](./speech_to_text.md#transcriptions-api) (`/v1/audio/transcriptions`)
@@ -76,7 +78,7 @@ For further details on speech to text, please refer to [this page](speech_to_tex
- [Generative Scoring API](generative_scoring.md) (`/generative_scoring`)
- Applicable to [CausalLM models](../../models/generative_models.md) (task `"generate"`).
- Computes next-token probabilities for specified `label_token_ids`.
- [Structured Decisions API](structured_decisions.md) (`/v1/systemone`)
- [Structured Decisions API](../../models/structured_decisions_models.md#typesafe-system-one-endpoint) (`/v1/systemone`)
- Applicable to [text generation models](../../models/generative_models.md) with a [chat template](#chat-template).
- Answers typed questions about a state with a probability for every allowed answer.
+6 -2
View File
@@ -154,11 +154,15 @@ should split those fields:
continue to work.
Callers that need only the token layout and item hashes, such as cache-aware
routers, can set `"return_mm_kwargs": false` on the render request. The response
routers, can set `"mm_payload": "none"` on the render request. The response
then keeps `mm_hashes` and `mm_placeholders` and sets `kwargs_data` and
`mm_metadata` to null, so the processed tensors are neither serialized nor sent.
Do not forward such a response to `/inference/v1/generate`, which reads a null
`kwargs_data` as every item being cached.
`kwargs_data` as every item being cached. The default, `"tensors"`, returns
both fields.
`"return_mm_kwargs": false` is a deprecated alias for `"mm_payload": "none"`
and is ignored when `mm_payload` is set.
## Example
@@ -1,145 +0,0 @@
# Structured Decisions
The `/v1/systemone` endpoint answers a set of typed questions about a state and
returns a probability for every allowed answer.
The endpoint is available by default on generation models. Models without a
supported read strategy return 501. Supported architectures require
`--logprobs-mode raw_logprobs` or `processed_logprobs`, since `label_mass` needs
log probabilities.
## How it works
1. Each question is one read. The user message is the state, followed by the
question and its options with the type's labels, and an instruction to
answer with one label.
2. The read is the reply's first token. Each label must be one token there,
or the request returns 400.
3. The read returns the logprobs of that question's label tokens. Softmax over
the labels gives the answer's probabilities.
Thinking is off unless `chat_template_kwargs` turns it on.
This endpoint does not serve diffusion models yet.
Every read of a request starts with the state, so with
`--enable-prefix-caching` the state is prefilled once and each further
question prefills only its own text, then reads one token.
## Question types
| type | criteria | answer |
| --- | --- | --- |
| `choice` | map of option name to a description or `null` | `choice`, `probabilities` by option name, `confidence` |
| `noul` | `null`, or an object with optional `true` and `false` descriptions | `noul` (probability of yes), `probabilities`, `confidence` |
| `score` | ordered list of level names | `score` (expected 0-indexed level), `legend`, `probabilities` by level index, `confidence` |
A choice has at least one option, a score at least two levels. A question id
is a non-empty string.
## Example
```bash
curl -s http://localhost:8000/v1/systemone \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen3-0.6B",
"state": {"ticket": "My card was charged twice for one order."},
"questions": {
"team": {
"type": "choice",
"instructions": "Which team should handle this ticket?",
"criteria": {
"billing": "payments and refunds",
"shipping": "deliveries",
"security": "account access"
}
},
"urgent": {
"type": "noul",
"instructions": "Does the customer need a reply within the hour?",
"criteria": {"true": "the account is locked or money is moving"}
},
"tone": {
"type": "score",
"instructions": "How upset is the customer?",
"criteria": ["calm", "annoyed", "furious"]
}
},
"chat_template_kwargs": {"enable_thinking": false}
}'
```
Response, with probabilities that depend on the model:
```json
{
"id": "decision-...",
"object": "structured_decision",
"created": 1790000000,
"model": "Qwen/Qwen3-0.6B",
"answers": {
"team": {
"type": "choice",
"choice": "billing",
"probabilities": {"billing": 0.97, "shipping": 0.01, "security": 0.02},
"confidence": 0.90
},
"urgent": {
"type": "noul",
"noul": 0.02,
"probabilities": {"yes": 0.02, "no": 0.98},
"confidence": 0.95
},
"tone": {
"type": "score",
"score": 0.6,
"legend": {"0": "calm", "1": "annoyed", "2": "furious"},
"probabilities": {"0": 0.7, "1": 0.2, "2": 0.1},
"confidence": 0.65
}
},
"usage": {"input_tokens": 142, "output_tokens": 3},
"diagnostics": {
"team": {"label_mass": 0.93, "argmax_is_label": true},
"urgent": {"label_mass": 0.98, "argmax_is_label": true},
"tone": {"label_mass": 0.91, "argmax_is_label": true}
}
}
```
`diagnostics.label_mass` is the probability the model put on the labels across
its whole vocabulary. A low value means most of the probability went to tokens
that are not labels. `confidence` is the chosen option's probability over the
whole vocabulary: its share of the labels times `label_mass`.
## Labels
Labels follow the type: a `choice` labels its options `A` to `Z` in the order
the request lists them, a `noul` labels its options `yes` and `no`, and a
`score` labels its levels `0` to `9`, so a level's label is its score.
## Limits
| limit | value |
| --- | --- |
| questions per request | 64 |
| options per `choice` | 26 |
| levels per `score` | 10 |
## Request fields
| field | meaning |
| --- | --- |
| `state` | what the questions are about: a string, or JSON that is sent as its JSON text |
| `questions` | question id to `{type, instructions, criteria}`, asked in this order |
| `instructions` | optional context placed ahead of the questions |
| `chat_template_kwargs` | passed to the chat template, for example `{"enable_thinking": false}` |
| `cache_salt` | optional [prefix-cache salt](../../usage/security.md#cache-salting) applied to every question |
A question with any field other than `type`, `instructions` and `criteria` is
rejected with a 400.
When [tracing](../../../examples/observability/opentelemetry/README.md) is enabled
with `--otlp-traces-endpoint`, the HTTP `traceparent` and `tracestate` headers are
forwarded to every question's generation request.
@@ -55,6 +55,21 @@ The server returns a 400 for:
For using `output_mode` with separate prefill and decode pools, see [Disaggregated Prefilling](../../features/disagg_prefill.md#generate-api-output-modes).
## Weight versions
The final output of every choice carries `weight_versions`, the label of the weights that sampled each range of its output tokens. Ranges are half-open over output-token indexes (prompt excluded), contiguous and in generation order, so a request that crossed one weight update gets two:
```json
"weight_versions": [
{"version": "v7", "start": 0, "end": 57},
{"version": "v8", "start": 57, "end": 128}
]
```
- With `stream: true`, only the last chunk of each choice carries the list, including the last chunk after an abort. Its offsets cover the whole output. Earlier chunks carry `null` or omit the field.
- An output with no tokens, such as a request aborted before its first token, gets `[]`.
- The labels are the ones you set with `/update_weight_version` or `/finish_weight_update`, see [Async Reinforcement Learning](../../training/async_rl.md#weight-versions-in-outputs). vLLM doesn't order or check them, and the label is `"default"` until you set one.
## Aborting requests
`POST /inference/v1/abort_requests` aborts in-flight requests. It is registered wherever `/inference/v1/generate` is and requires the API key when `--api-key` is set, like `/inference/v1/generate`.
+1 -1
View File
@@ -71,7 +71,7 @@ Ray also offers high-level APIs for large-scale [offline batch inference](https:
Ray is an optional dependency. Install it explicitly before using Ray-based execution, for example:
```bash
pip install "ray[cgraph]"
pip install ray
```
For details, see the [Ray documentation](https://docs.ray.io/en/latest/index.html).
+18 -2
View File
@@ -43,10 +43,11 @@ With `VLLM_SERVER_DEV_MODE=1`, the vLLM HTTP server exposes the same functionali
- `POST /pause?mode=keep` - Pause generation
- `POST /resume` - Resume generation
- `POST /abort_requests` - Abort in-flight requests without pausing the scheduler (send `{}` to abort all, or `{"request_ids": [...]}`)
- `POST /update_weight_version` - Set the `weight_version` label without changing weights (send `{"new_version": "v8"}`)
- `GET /weight_info` - Return the latest committed `weight_version`
!!! note "Data Parallelism"
When using data parallelism with vLLM's **internal load balancer** (i.e. `data_parallel_backend="ray"`), pause and resume are handled automatically across all DP ranks -- a single call is sufficient. When using an **external load balancer** (i.e. multiple independent vLLM instances behind a proxy), you must send pause and resume requests to **every** engine instance individually before and after the weight update.
When using data parallelism with vLLM's **internal load balancer** (i.e. `data_parallel_backend="ray"`), pause and resume are handled automatically across all DP ranks -- a single call is sufficient. When using an **external load balancer** (i.e. multiple independent vLLM instances behind a proxy), you must send pause and resume requests to **every** engine instance individually before and after the weight update, and set the weight version on every instance too.
## Typical Async RL Flow
@@ -54,12 +55,27 @@ A typical async RL loop with weight syncing looks like this:
1. Start generating rollouts from the current policy
2. Once trainer has new weights to update to, pause generation with `mode="keep"`
3. Sync the updated weights from the trainer to the inference engine (see [Weight Transfer](weight_transfer/README.md))
3. Sync the updated weights from the trainer to the inference engine (see [Weight Transfer](weight_transfer/README.md)) and set their weight version
4. Resume generation -- in-flight requests continue with the new weights
5. Repeat
The key insight is that requests paused with `mode="keep"` will produce tokens from the **old** weights before the pause and tokens from the **new** weights after resume. The `clear_cache` parameter controls whether the KV cache is invalidated during the pause. When `clear_cache=True`, previously cached key-value entries are discarded, so all tokens generated after resume will be computed entirely with the new weights. When `clear_cache=False`, existing KV cache entries are retained, meaning some tokens in context may still reflect the old weights (stale KV cache).
## Weight Versions in Outputs
To tell which tokens came from which weights, set a weight-version label while generation is paused, after the new weights are loaded: pass `weight_version` to `finish_weight_update`, or call `update_weight_version` (`POST /update_weight_version` over HTTP). Labels are strings you choose, such as the trainer step; vLLM doesn't order or check them.
The final output of every sequence then carries `weight_versions`, one span per label over its output-token indexes. A request paused with `mode="keep"` across one update gets two spans:
```python
completion = final_output.outputs[0]
for span in completion.weight_versions:
# The tokens in [span.start, span.end) were sampled by weights `span.version`.
print(span.version, completion.token_ids[span.start : span.end])
```
The offsets cover the whole output, so with streamed delta outputs, collect the tokens before slicing. `/inference/v1/generate` returns the same list on each choice; see [Tokens In <> Tokens Out API](../serving/online_serving/token_in_token_out.md#weight-versions).
## Example
The [async RLHF example](../../examples/rl/rlhf_async_new_apis.py) demonstrates this pattern with `vllm.AsyncLLMEngine`, NCCL weight transfer, and mid-flight pause/resume with validation.
+4 -1
View File
@@ -50,7 +50,10 @@ The following metrics are exposed:
These metrics are exposed when the `HiSparseConnector` KV connector is
configured. The host-pool gauges are reported by the scheduler once per engine
step, for the host pool shared by all tensor-parallel ranks; evictable cached
blocks count as free.
blocks count as free. With `--kv-cache-metrics`, the
`vllm:hisparse_host_block_*_seconds` histograms sample host-pool block
residency, like the device `vllm:kv_block_*_seconds` histograms, which cover
only the GPU pool.
--8<-- "gen:metrics-hisparse"
+27 -1
View File
@@ -94,7 +94,9 @@ denial of service:
| Environment Variable | Default | Description |
| --- | --- | --- |
| `VLLM_MAX_MEDIA_DOWNLOAD_SIZE_MB` | `256` | Maximum size in MB for a single remote media response. Oversized responses are rejected while streaming before the full body is materialized in memory. |
| `VLLM_MAX_MEDIA_REQUEST_DOWNLOAD_SIZE_MB` | `256` | Maximum total HTTP media bytes downloaded for one request, including concurrent downloads. Enforced while streaming. |
| `VLLM_MAX_IMAGE_PIXELS` | `178956970` (~179M pixels) | Maximum decoded image size in pixels. Images exceeding this are rejected before raster memory is allocated. Default matches PIL's built-in 2x decompression-bomb threshold (~680 MB for RGB). |
| `VLLM_MAX_MEDIA_REQUEST_DECODE_BYTES` | `715827880` (~683 MiB) | Maximum aggregate decoded image, video-frame, and audio bytes in one request. Images are charged at four bytes per pixel before decoding; audio is limited before decoding; video files are charged after frame extraction. |
| `VLLM_MAX_AUDIO_CLIP_FILESIZE_MB` | `25` | Maximum compressed filesize in MB for a single audio file. Enforced on all audio inputs (multimodal chat URLs, speech-to-text uploads, data: URLs, and local file paths) before decoding begins. |
| `VLLM_MAX_AUDIO_DECODE_DURATION_S` | `600` | Maximum decoded audio duration in seconds. Prevents compressed audio from expanding into gigabytes of float32 PCM. |
| `VLLM_MAX_AUDIO_DECODE_BYTES` | `268435456` (256 MiB) | Maximum float32 PCM bytes that audio decoding may allocate. Guards against sample-rate forgery where an inflated header sample rate bypasses the duration guard while the actual frame count causes a multi-GiB allocation. |
@@ -173,12 +175,13 @@ When `--api-key` is configured, the following endpoints require Bearer token aut
- `/v1/responses/render` - Render a self-contained response request (available on `vllm serve` only when `--enable-scale-out` is set, or on `vllm launch render` unless explicitly disabled)
- `/v1/responses/{response_id}` - Retrieve a response
- `/v1/responses/{response_id}/cancel` - Cancel a response
- `/v1/decisions` - Text decisions API
- `/v1/score` - Scoring API
- `/v1/rerank` - Reranking API
- `/v1/systemone` - Structured decisions API
- `/v1/load_lora_adapter` - Load a LoRA adapter (can alter model behavior; only available when `--enable-lora` is set and `VLLM_ALLOW_RUNTIME_LORA_UPDATING=True`)
- `/v1/unload_lora_adapter` - Unload a LoRA adapter (can alter model behavior; only available when `--enable-lora` is set and `VLLM_ALLOW_RUNTIME_LORA_UPDATING=True`)
- `/inference/v1/generate` - Generate completions (available when `--enable-scale-out` is set, or with `--tokens-only`)
- `/inference/v1/generate` - Generate completions (available when `--enable-scale-out` is set, or with `--tokens-only`; otherwise returns HTTP 403 with instructions to enable it)
- `/cohere/v2/chat/render` - Render Cohere Chat v2 requests (requires `VLLM_ENABLE_COHERE_API=1` and either `--enable-scale-out` or `--tokens-only`)
- `/v2/embed` - Cohere Embed API
- `/v2/rerank` - Cohere Rerank API
@@ -208,9 +211,13 @@ The following endpoints **do not require authentication** even when `--api-key`
- `/update_weights` - Update model weights (can alter model behavior)
- `/get_world_size` - Get distributed world size
- `/abort_requests` - Abort in-flight requests (available with `--tokens-only`. Use the authenticated `/inference/v1/abort_requests` otherwise)
- `/fault_tolerance/apply` - Trigger fault-recovery operations across ranks (available when fault tolerance is enabled)
- `/fault_tolerance/status` - Query fault-recovery status (available when fault tolerance is enabled)
**Utility endpoints:**
- `/metrics` - Prometheus metrics, including model, throughput, queue, and KV-cache statistics
- `/kv_event_sources` - KV-event publisher configuration for each data-parallel rank, which may reveal internal transport endpoints (when KV events are configured)
- `/tokenize` - Tokenize text (not gated by `--enable-scale-out`)
- `/detokenize` - Detokenize tokens
- `/health` - Health check
@@ -303,6 +310,25 @@ Only deployments whose API clients are trusted should start the server with
`--trust-request-mm-kwargs` to restore per-request overrides. Do not enable
this option on an endpoint exposed to untrusted clients.
### P2P KV-Offload Peer Session Limits
Deployments that enable the P2P secondary tier (`OffloadingConnector` with `TieringOffloadingSpec` and a `p2p` secondary tier) accept peer addresses from `kv_transfer_params` in API requests. Without limits, an attacker with API access could submit many unique non-listening peer addresses, each consuming ZeroMQ sockets until the context quota is exhausted, crashing the EngineCore.
vLLM enforces several configurable limits to prevent this:
| Environment Variable | Default | Description |
| --- | --- | --- |
| `VLLM_P2P_MAX_PEERS` | `128` | Maximum concurrent P2P peer sessions. New peers (both outbound and inbound) are rejected when this limit is reached. |
| `VLLM_P2P_HANDSHAKE_TIMEOUT_S` | `30` | Seconds to wait for a newly connected peer to complete the P2P handshake. Sessions that remain connected but not ready after this deadline are torn down. |
| `VLLM_P2P_IDLE_TIMEOUT_S` | `300` | Seconds a P2P session may remain idle (no pending work, no messages) before it is eligible for eviction. Idle eviction runs only when remaining peer slots are within 8 of `VLLM_P2P_MAX_PEERS`. Set to `0` to disable idle eviction. |
Connections that fail (ZMQ socket errors, handshake failures) are handled gracefully — the request falls back to local prefill instead of crashing the engine.
**Recommendations:**
- **Public-facing deployments with P2P enabled:** Lower `VLLM_P2P_MAX_PEERS` to the expected number of legitimate peers (e.g., `16` or `32`).
- **Sensitive deployments:** Consider deploying behind a reverse proxy that strips or validates `kv_transfer_params` from API requests to prevent client-supplied peer addresses entirely.
## Tool Server and MCP Security
vLLM supports connecting to external tool servers via the `--tool-server` argument. This enables models to call tools through the Responses API (`/v1/responses`). Tool server support works with all models — it is not limited to specific model architectures.