Merge branch 'main' into patches
Signed-off-by: MOVZX <movzx@yahoo.com>
This commit is contained in:
@@ -26,6 +26,7 @@ nav:
|
||||
- models/supported_models.md
|
||||
- models/generative_models.md
|
||||
- Pooling Models: models/pooling_models
|
||||
- models/structured_decisions_models.md
|
||||
- models/extensions
|
||||
- Hardware Supported Models:
|
||||
- models/hardware_supported_models/*
|
||||
|
||||
+30
-2
@@ -9,7 +9,7 @@ vllm --help
|
||||
Available Commands:
|
||||
|
||||
```bash
|
||||
vllm {chat,complete,serve,launch,bench,collect-env,run-batch,preload}
|
||||
vllm {chat,complete,serve,launch,bench,collect-env,download-kernels,run-batch,preload}
|
||||
```
|
||||
|
||||
## serve
|
||||
@@ -131,9 +131,22 @@ To use benchmark commands, please install with extra dependencies using `pip ins
|
||||
Available Commands:
|
||||
|
||||
```bash
|
||||
vllm bench {latency, serve, throughput}
|
||||
vllm bench {forward, latency, serve, throughput}
|
||||
```
|
||||
|
||||
### forward
|
||||
|
||||
Benchmark the model forward of exact batches: each case lists its requests as `[N*]Q[@K][c][:D]` (query length Q, KV length K including the query, D draft tokens). A prefill's prefix is a prefix-cache hit unless `c` marks a chunked prefill; they differ only on models with SWA bounded replay, which recompute the last window of a hit. Each request's prefix length varies by ±50% by default (`--kv-len-jitter`), identically on every DP rank. Each forward is captured as a one-shot CUDA graph; sampling and CPU overhead aren't measured. Use `torchrun`, one process per GPU, for DP/TP/EP. CUDA only for now.
|
||||
|
||||
```bash
|
||||
vllm bench forward \
|
||||
--model meta-llama/Llama-3.2-1B-Instruct \
|
||||
--case decode=32*1@4096 \
|
||||
--case mix=2048,32*1@4096
|
||||
```
|
||||
|
||||
See [vllm bench forward](./bench/forward.md) for the full reference of all available arguments.
|
||||
|
||||
### latency
|
||||
|
||||
Benchmark the latency of a single batch of requests.
|
||||
@@ -188,6 +201,21 @@ Start collecting environment information.
|
||||
vllm collect-env
|
||||
```
|
||||
|
||||
## download-kernels
|
||||
|
||||
Install the precompiled kernels that vLLM's dependencies would otherwise download
|
||||
or compile at startup. Run it after installing or upgrading vLLM. It currently
|
||||
installs FlashInfer's kernels and does nothing when FlashInfer is not installed.
|
||||
|
||||
```bash
|
||||
vllm download-kernels
|
||||
|
||||
# Print the install commands without running them
|
||||
vllm download-kernels --dry-run
|
||||
```
|
||||
|
||||
See [vllm download-kernels](./download-kernels.md) for the full reference of all available arguments.
|
||||
|
||||
## run-batch
|
||||
|
||||
Run batch prompts and write results to file.
|
||||
|
||||
@@ -41,6 +41,7 @@ If you are only developing vLLM's Python code, install vLLM using:
|
||||
|
||||
```bash
|
||||
VLLM_USE_PRECOMPILED=1 uv pip install -e .
|
||||
vllm download-kernels
|
||||
```
|
||||
|
||||
To rebuild only the Rust frontend binary:
|
||||
@@ -328,9 +329,15 @@ review process:
|
||||
in the upstream repository. Your branch must contain every commit currently
|
||||
on that branch: **zero commits behind**. This also applies to PRs targeting
|
||||
release branches or another PR's branch.
|
||||
- If the PR is behind, the bot reports the count without creating a new build.
|
||||
Merge or rebase onto the latest target branch, then rerun the command. If the
|
||||
target branch advances in the meantime, update your branch and try again.
|
||||
- If the PR is behind, the bot merges the target branch into your PR branch
|
||||
(like the **Update branch** button) and runs CI on the resulting merge commit.
|
||||
Pull the branch before pushing more changes. This needs **Allow edits by
|
||||
maintainers** enabled on fork PRs. If the merge conflicts or is not allowed,
|
||||
the bot reports the count without creating a new build; merge or rebase onto
|
||||
the latest target branch yourself, then rerun the command.
|
||||
- PR authors without write access cannot start another full run (including
|
||||
`--allow-stale`) on a commit that already has a CI build. Push new changes,
|
||||
use `/ci retry` to rerun failed jobs, or ask a reviewer with write access.
|
||||
- To run CI on an outdated branch at your own risk, append `--allow-stale` to
|
||||
any run command, for example `/ci run --allow-stale` or
|
||||
`/amd-ci run all --allow-stale`. The same authorization requirements apply.
|
||||
@@ -338,9 +345,8 @@ review process:
|
||||
failures. Before merging, merge or rebase onto the latest target branch and
|
||||
rerun CI without `--allow-stale` on the latest PR commit.
|
||||
- Retrying or cancelling builds does not check branch freshness.
|
||||
- These commands do not modify your branch or enforce merge requirements.
|
||||
Direct Buildkite launches are outside this workflow. No additional token is
|
||||
needed.
|
||||
- Apart from that merge, these commands do not modify your branch or enforce
|
||||
merge requirements. Direct Buildkite launches are outside this workflow.
|
||||
|
||||
Maintainers must configure required Buildkite status checks on the latest PR
|
||||
commit and enable GitHub's
|
||||
|
||||
@@ -8,11 +8,11 @@ It is a [CNCF Sandbox project](https://www.cncf.io/blog/2026/03/24/welcome-llm-d
|
||||
|
||||
A single vLLM server is fast, but at scale the picture changes: across many replicas, cache locality breaks under round-robin load balancing, long prompts inflate time-to-first-token, and accelerators sit underused. llm-d adds the cluster-level layer that vLLM does not aim to provide on its own:
|
||||
|
||||
- **[Prefix-aware routing](https://llm-d.ai/docs/guides/precise-prefix-cache-aware).** Instead of round-robin, llm-d reads vLLM's KV-cache events and routes each request to the replica that already holds its prefix, reusing cache instead of recomputing it.
|
||||
- **[Distributed KV-cache management](https://llm-d.ai/docs/guides#advanced-kv-cache-management).** A global index tracks which token blocks live on which replica, and [tiered offloading](https://llm-d.ai/docs/guides/tiered-prefix-cache) spills cache to CPU memory or local SSD, extending the working set beyond accelerator HBM.
|
||||
- **[Prefill/decode disaggregation](https://llm-d.ai/docs/guides/pd-disaggregation).** Prompt processing and token generation run on separate vLLM workers, with KV-cache moved over the vLLM [NIXL connector](https://docs.vllm.ai/en/latest/features/nixl_connector_usage/), lowering TTFT and steadying per-token latency on long prompts.
|
||||
- **[Wide expert-parallelism](https://llm-d.ai/docs/guides/wide-expert-parallelism).** Serve large Mixture-of-Experts models such as DeepSeek-R1 and GPT-OSS across nodes with combined data and expert parallelism, for more KV-cache capacity and throughput.
|
||||
- **SLO-aware [autoscaling](https://llm-d.ai/docs/guides/workload-autoscaling) and [flow control](https://llm-d.ai/docs/guides/flow-control).** Scale vLLM pools on real inference signals (queue depth, true demand) rather than raw GPU utilization, with multi-tenant fairness and priority dispatch.
|
||||
- **[Prefix-aware routing](https://llm-d.ai/docs/well-lit-paths/foundations/precise-prefix-cache-routing).** Instead of round-robin, llm-d reads vLLM's KV-cache events and routes each request to the replica that already holds its prefix, reusing cache instead of recomputing it.
|
||||
- **[Distributed KV-cache management](https://llm-d.ai/docs/architecture/advanced/kv-management).** A global index tracks which token blocks live on which replica, and [tiered offloading](https://llm-d.ai/docs/well-lit-paths/foundations/tiered-prefix-cache) spills cache to CPU memory or local SSD, extending the working set beyond accelerator HBM.
|
||||
- **[Prefill/decode disaggregation](https://llm-d.ai/docs/well-lit-paths/foundations/pd-disaggregation).** Prompt processing and token generation run on separate vLLM workers, with KV-cache moved over the vLLM [NIXL connector](https://docs.vllm.ai/en/latest/features/nixl_connector_usage/), lowering TTFT and steadying per-token latency on long prompts.
|
||||
- **[Wide expert-parallelism](https://llm-d.ai/docs/well-lit-paths/foundations/wide-expert-parallelism).** Serve large Mixture-of-Experts models such as DeepSeek-R1 and GPT-OSS across nodes with combined data and expert parallelism, for more KV-cache capacity and throughput.
|
||||
- **SLO-aware [autoscaling](https://llm-d.ai/docs/well-lit-paths/foundations/workload-autoscaling) and [flow control](https://llm-d.ai/docs/well-lit-paths/foundations/flow-control).** Scale vLLM pools on real inference signals (queue depth, true demand) rather than raw GPU utilization, with multi-tenant fairness and priority dispatch.
|
||||
|
||||
These are composable. Most teams start by adding prefix-aware routing over an existing vLLM pool, then layer in the rest as specific bottlenecks appear.
|
||||
|
||||
@@ -28,9 +28,9 @@ See the [full list](https://github.com/llm-d/llm-d#performance-highlights) and r
|
||||
|
||||
## Get started
|
||||
|
||||
1. Deploy the [Optimized Baseline](https://llm-d.ai/docs/guides/optimized-baseline) with the [Quickstart](https://llm-d.ai/docs/getting-started/quickstart). It stands up an intelligent router over a vLLM pool on Kubernetes in a tested configuration.
|
||||
2. Browse the [well-lit path guides](https://llm-d.ai/docs/guides), each a tested recipe for one of the capabilities above, and add the optimization that fits your workload.
|
||||
3. Read the [Introduction](https://llm-d.ai/docs/getting-started) and [Architecture overview](https://llm-d.ai/docs/architecture) to see how the pieces wrap your vLLM deployment.
|
||||
1. Deploy the [Optimized Baseline](https://llm-d.ai/docs/well-lit-paths/foundations/optimized-baseline) with the [Quickstart](https://llm-d.ai/docs/getting-started/quickstart). It stands up an intelligent router over a vLLM pool on Kubernetes in a tested configuration.
|
||||
2. Browse the [well-lit path guides](https://llm-d.ai/docs/well-lit-paths), each a tested recipe for one of the capabilities above, and add the optimization that fits your workload.
|
||||
3. Read the [Introduction](https://llm-d.ai/docs) and [Architecture overview](https://llm-d.ai/docs/architecture) to see how the pieces wrap your vLLM deployment.
|
||||
|
||||
You can also deploy vLLM with llm-d via [KServe's LLMInferenceService](https://kserve.github.io/website/docs/model-serving/generative-inference/llmisvc/llmisvc-overview).
|
||||
|
||||
|
||||
@@ -147,7 +147,7 @@ Priority is **1 = highest** (tried first).
|
||||
|
||||
--8<-- "gen:priority-mla"
|
||||
|
||||
> **\*** For sparse MLA, FP8 KV cache always prefers `FLASHINFER_MLA_SPARSE`. With BF16 KV cache, `FLASHINFER_MLA_SPARSE` is preferred for low query-head counts (<= 16), while `FLASHMLA_SPARSE` is preferred otherwise.
|
||||
> **\*** For sparse MLA, FP8 KV cache always prefers `FLASHINFER_MLA_SPARSE`. With BF16 KV cache, `FLASH_ATTN_MLA_SPARSE_FA4` (then `FLASHINFER_MLA_SPARSE`) is preferred for low query-head counts (<= 16) on SM 10.x, while `FLASHMLA_SPARSE` is preferred otherwise.
|
||||
>
|
||||
> **Note:** ROCm and CPU platforms have their own selection logic. See the platform-specific documentation for details.
|
||||
|
||||
|
||||
@@ -52,8 +52,6 @@ While `NONE` , `PIECEWISE`, and `FULL` are single-mode configurations and simply
|
||||
!!! note
|
||||
Here, the single-modes `NONE`, `PIECEWISE`, and `FULL` are treated as the runtime modes for CUDA Graphs dispatching. If using a dual-mode, the dispatcher will always dispatch to one of its member modes (plus a potential `NONE` if no suitable CUDA Graph available), depending on the batch composition.
|
||||
|
||||
While cascade attention is not cudagraph compatible, it is now compatible with all possible cudagraph mode configurations. If a batch uses cascade attention, it always gets dispatched to `PIECEWISE` mode if available (otherwise `NONE`).
|
||||
|
||||
!!! note
|
||||
Not all CUDA Graph modes are compatible with every attention backend. We automatically "downgrade" modes to the closest supported mode. For example, if a backend only supports CUDA Graphs for pure decode/uniform batches, we convert `FULL` to `FULL_AND_PIECEWISE` if piecewise compilation is enabled, and `FULL_DECODE_ONLY` otherwise.
|
||||
|
||||
@@ -154,9 +152,7 @@ To signal the CUDA Graphs compatibility of the attention backends, we introduce
|
||||
|
||||
```python
|
||||
class AttentionCGSupport(enum.Enum):
|
||||
""" Constants for the CUDA Graphs support of the attention backend
|
||||
Here we do not consider the cascade attention, as currently
|
||||
it is never CUDA Graphs supported."""
|
||||
"""Constants for the CUDA Graphs support of the attention backend"""
|
||||
|
||||
ALWAYS = 3
|
||||
"""CUDA Graphs always supported; supports mixed-prefill-decode"""
|
||||
|
||||
@@ -126,7 +126,9 @@ Do not expect a single endpoint plugin to also mutate engine/worker state. If yo
|
||||
|
||||
## Path-prefix convention
|
||||
|
||||
There is currently no route conflict enforcement (tracked as a follow-up to RFC [#46565](https://github.com/vllm-project/vllm/issues/46565)). A plugin's `attach_router` can register a path that collides with a core route and routes attached later win. To avoid surprising operators:
|
||||
Plugin hooks run after the core routers have been registered. HTTP routes added by plugins replace earlier routes with the same path template and HTTP method, including their OpenAPI description. Methods not overridden by the plugin remain available. When multiple plugins register the same operation, the last registered route takes precedence. Different path templates are not treated as equivalent.
|
||||
|
||||
There is currently no rejection of route conflicts (tracked as a follow-up to RFC [#46565](https://github.com/vllm-project/vllm/issues/46565)). To avoid surprising operators:
|
||||
|
||||
- Namespace your routes under a distinct prefix, e.g. `/plugins/<plugin-name>/...`, rather than reusing `/v1/...` or other core prefixes
|
||||
- Only register routes under a core prefix (like the worked example's `/v1/admin/scheduler_config`) if you specifically intend to override or extend existing behavior and document that clearly for operators allowlisting your plugin
|
||||
|
||||
@@ -0,0 +1,121 @@
|
||||
# Expert substitution (MoNE)
|
||||
|
||||
[Mixture of Novices and Experts (MoNE)](https://arxiv.org/abs/2507.00390)
|
||||
prunes MoE layers by replacing redundant experts with "novices" that return a
|
||||
constant vector. vLLM loads the retained experts into compact weights and adds
|
||||
the novices' output separately, so pruned experts use no MLP weight memory and
|
||||
no MoE backend needs to know about them.
|
||||
|
||||
## Checkpoint format
|
||||
|
||||
The checkpoint keeps the original architecture. `config.json` lists the
|
||||
substituted experts of each decoder layer:
|
||||
|
||||
```json
|
||||
{
|
||||
"architectures": ["DeepseekV2ForCausalLM"],
|
||||
"n_routed_experts": 64,
|
||||
"approximate_experts": {
|
||||
"1": [1, 6, 8, 12],
|
||||
"2": [0, 3, 17, 40]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
For VLMs, `approximate_experts` may live on the text config. Retained experts
|
||||
keep their logical checkpoint names. Each substituted expert stores one
|
||||
`[hidden_size]` tensor instead of its MLP:
|
||||
|
||||
```text
|
||||
model.layers.1.mlp.experts.0.gate_proj.weight
|
||||
model.layers.1.mlp.experts.1.approx_value
|
||||
```
|
||||
|
||||
Alternatively, `compression_config.transform_config.expert_substitution`
|
||||
declares the same information with explicitly named constant tensors:
|
||||
|
||||
```json
|
||||
{
|
||||
"compression_config": {
|
||||
"transform_config": {
|
||||
"expert_substitution": {
|
||||
"version": 1,
|
||||
"router_semantics": {
|
||||
"preserve_logical_expert_ids": true,
|
||||
"preserve_router_weights": true,
|
||||
"renormalize_after_substitution": false
|
||||
},
|
||||
"targets": {
|
||||
"model.layers.1.mlp.experts": {
|
||||
"num_logical_experts": 64,
|
||||
"weight_layout": "compact_retained_experts",
|
||||
"replacements": {
|
||||
"1": {
|
||||
"format": "constant-v1",
|
||||
"tensors": {"value": "model.layers.1.mlp.expert_replacements.1.value"}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Only the values shown for `version`, `router_semantics`, `weight_layout`, and
|
||||
`format` are supported. Each target path must contain exactly one layer index.
|
||||
This form is not yet accepted alongside compressed-tensors quantization, whose
|
||||
`transform_config` schema does not include `expert_substitution`.
|
||||
|
||||
## Semantics
|
||||
|
||||
For a substituted expert `j` with constant `v_j` and router weight `w_j(x)`,
|
||||
the expert contributes `w_j(x) v_j`. Routing is unchanged: router weights are
|
||||
neither renormalized nor redistributed to retained experts.
|
||||
|
||||
## Execution
|
||||
|
||||
`FusedMoEFactory` matches each MoE layer to `approximate_experts` by the layer
|
||||
index in its prefix and, for a match, builds `SubstitutedRoutedExperts`, which:
|
||||
|
||||
1. allocates weights only for retained experts, in ascending logical order;
|
||||
2. after routing, sums `w_j(x) v_j` over substituted routes in FP32;
|
||||
3. rewrites substituted routes as zero-weight routes to physical expert 0 and
|
||||
retained routes as compact physical IDs, preserving the `[num_tokens, top_k]`
|
||||
contract of every decomposed backend;
|
||||
4. adds the constant output on one tensor-parallel rank, before the partial
|
||||
outputs are all-reduced.
|
||||
|
||||
Zero-weight routes may schedule some unnecessary GEMM work; the memory savings
|
||||
are unaffected. Monolithic backends route internally and are not selected.
|
||||
|
||||
## Weight loading
|
||||
|
||||
Retained experts are loaded through the standard expert mapping by logical ID
|
||||
and placed in their compact row. Substituted IDs map to no local row, the same
|
||||
way expert parallelism skips non-local experts.
|
||||
|
||||
Constant tensors are consumed at the model's `load_weights` boundary, so
|
||||
initial loading and direct weight updates share one path. `approx_value` tensors of
|
||||
layers that the model does not build (other pipeline stages, MTP layers) are
|
||||
passed through to the model's loader, which skips them like any other weight
|
||||
it does not own. Explicitly named constants are always consumed, and dropped
|
||||
when their layer is built elsewhere. Loading fails if a local substituted
|
||||
expert has no constant.
|
||||
|
||||
Direct `load_weights` updates may change individual constants. Layerwise
|
||||
reloads (`reload_weights` in checkpoint format) are rejected, and the model
|
||||
keeps its previous weights.
|
||||
|
||||
## Supported configurations
|
||||
|
||||
- Models whose MoE layers use `FusedMoEFactory` with the default
|
||||
`RoutedExperts`, and whose MoE prefix contains exactly one layer index.
|
||||
- Unquantized experts.
|
||||
- Tensor and pipeline parallelism, including MTP draft models.
|
||||
- Any decomposed MoE backend.
|
||||
|
||||
Expert and data parallelism, EPLB, MoE LoRA, fused shared experts, quantized
|
||||
experts, and routed input/output transforms are rejected at initialization.
|
||||
Deferred MoE finalize and layerwise weight reloads are rejected when used.
|
||||
@@ -232,7 +232,7 @@ Supported quantization scheme/hardware combinations:
|
||||
|
||||
- Pass: [`vllm/compilation/passes/fusion/rms_quant_fusion.py`](https://github.com/vllm-project/vllm/blob/main/vllm/compilation/passes/fusion/rms_quant_fusion.py)
|
||||
- ROCm AITER pass: [`vllm/compilation/passes/fusion/rocm_aiter_fusion.py`](https://github.com/vllm-project/vllm/blob/main/vllm/compilation/passes/fusion/rocm_aiter_fusion.py)
|
||||
- CUDA/HIP kernels: [`csrc/libtorch_stable/layernorm_quant_kernels.cu`](https://github.com/vllm-project/vllm/blob/main/csrc/libtorch_stable/layernorm_quant_kernels.cu)
|
||||
- CUDA/HIP kernels: [`csrc/layernorm_quant_kernels.cu`](https://github.com/vllm-project/vllm/blob/main/csrc/layernorm_quant_kernels.cu)
|
||||
|
||||
### SiLU+Mul + Quantization (`fuse_act_quant`)
|
||||
|
||||
@@ -258,7 +258,7 @@ Supported quantization scheme/hardware combinations:
|
||||
- Pass: [`vllm/compilation/passes/fusion/act_quant_fusion.py`](https://github.com/vllm-project/vllm/blob/main/vllm/compilation/passes/fusion/act_quant_fusion.py)
|
||||
- ROCm AITER pass: [`vllm/compilation/passes/fusion/rocm_aiter_fusion.py`](https://github.com/vllm-project/vllm/blob/main/vllm/compilation/passes/fusion/rocm_aiter_fusion.py)
|
||||
- CUDA/HIP kernels: [`csrc/quantization/`](https://github.com/vllm-project/vllm/blob/main/csrc/quantization/)
|
||||
- Fused SiLU+Mul+BlockQuant kernel: [`csrc/libtorch_stable/quantization/fused_kernels/fused_silu_mul_block_quant.cu`](https://github.com/vllm-project/vllm/blob/main/csrc/libtorch_stable/quantization/fused_kernels/fused_silu_mul_block_quant.cu)
|
||||
- Fused SiLU+Mul+BlockQuant kernel: [`csrc/quantization/fused_kernels/fused_silu_mul_block_quant.cu`](https://github.com/vllm-project/vllm/blob/main/csrc/quantization/fused_kernels/fused_silu_mul_block_quant.cu)
|
||||
|
||||
### RMSNorm + Padding (`fuse_act_padding`)
|
||||
|
||||
|
||||
@@ -204,7 +204,7 @@ The second question is the cache eviction policy. For now, we use one LRU queue
|
||||
|
||||
### Case 3: mamba models
|
||||
|
||||
The prefix caching support of the mamba model is work in progress. Once implemented, models with mamba layer + full attention layer can be supported via the full attention + X algorithm in case 2.
|
||||
Models with mamba layers + full attention layers support prefix caching via the full attention + X algorithm in case 2, with the Mamba state cached at block boundaries. See [Automatic Prefix Caching](../features/automatic_prefix_caching.md).
|
||||
|
||||
## Implementation
|
||||
|
||||
|
||||
@@ -30,6 +30,7 @@ In v1, an extensive set of metrics are exposed via a Prometheus-compatible `/met
|
||||
- `vllm:prompt_tokens_total` (Counter) - Total number of prompt tokens processed.
|
||||
- `vllm:generation_tokens_total` (Counter) - Total number of generated tokens.
|
||||
- `vllm:request_success_total` (Counter) - Number of finished requests (by finish reason).
|
||||
- `vllm:admission_rejections_total` (Counter) - Admission attempts rejected by a queue limit, by reason.
|
||||
- `vllm:request_prompt_tokens` (Histogram) - Histogram of input prompt token counts.
|
||||
- `vllm:request_generation_tokens` (Histogram) - Histogram of generation token counts.
|
||||
- `vllm:time_to_first_token_seconds` (Histogram) - Time to first token (TTFT).
|
||||
|
||||
@@ -90,6 +90,7 @@ GPU-side fusion is controlled by `multimodal_config.mm_device_do_normalize`:
|
||||
| `llama-nemotron-vl-embed` | `LlamaNemotronVLForEmbedding` | `nvidia/llama-nemotron-embed-vl-1b-v2` |
|
||||
| `llama-nemotron-vl-rerank` | `LlamaNemotronVLForSequenceClassification` | `nvidia/llama-nemotron-rerank-vl-1b-v2` |
|
||||
| `mistral3` | `Mistral3ForConditionalGeneration` | `mistralai/Mistral-Small-3.2-24B-Instruct-2506`, etc. |
|
||||
| `nemotron-parse` | `NemotronParseForConditionalGeneration` | `nvidia/NVIDIA-Nemotron-Parse-2.0`, etc. |
|
||||
| `qwen2-vl` | `Qwen2VLForConditionalGeneration` | `Qwen/Qwen2-VL-2B-Instruct`, etc. |
|
||||
| `qwen2.5-vl` | `Qwen2_5_VLForConditionalGeneration` | `Qwen/Qwen2.5-VL-3B-Instruct`, etc. |
|
||||
| `qwen3-vl` | `Qwen3VLForConditionalGeneration` | `Qwen/Qwen3-VL-4B-Instruct`, etc. |
|
||||
|
||||
@@ -91,7 +91,7 @@ To be used with a particular `FusedMoEPrepareAndFinalizeModular` subclass, MoE k
|
||||
| marlin | standard,</br>batched | <sup>3</sup> / N/A | <sup>3</sup> / N/A | silu,</br>swigluoai | Y | Y | [`fused_marlin_moe`][vllm.model_executor.layers.fused_moe.experts.marlin_moe.fused_marlin_moe],</br>[`MarlinExperts`][vllm.model_executor.layers.fused_moe.experts.marlin_moe.MarlinExperts],</br>[`BatchedMarlinExperts`][vllm.model_executor.layers.fused_moe.experts.marlin_moe.BatchedMarlinExperts] |
|
||||
| trtllm | standard | mxfp4,</br>nvfp4 | G(16),G(32) | <sup>5</sup> | N | Y | [`TrtLlmMxfp4ExpertsMonolithic`][vllm.model_executor.layers.fused_moe.experts.trtllm_mxfp4_moe.TrtLlmMxfp4ExpertsMonolithic],</br>[`TrtLlmMxfp4ExpertsModular`][vllm.model_executor.layers.fused_moe.experts.trtllm_mxfp4_moe.TrtLlmMxfp4ExpertsModular],</br>[`TrtLlmNvFp4ExpertsMonolithic`][vllm.model_executor.layers.fused_moe.experts.trtllm_nvfp4_moe.TrtLlmNvFp4ExpertsMonolithic],</br>[`TrtLlmNvfp4ExpertsModular`][vllm.model_executor.layers.fused_moe.experts.trtllm_nvfp4_moe.TrtLlmNvFp4ExpertsModular] |
|
||||
| hpc | standard | fp8 | G(128),T | silu | Y | Y | [`HPCExperts`][vllm.model_executor.layers.fused_moe.hpc_moe.HPCExperts] |
|
||||
| rocm aiter moe | standard | mxfp4,</br>fp8 | G(32),G(128),A,T | silu, gelu,</br>swigluoai | Y | N | `rocm_aiter_fused_experts`,</br>`AiterExperts` |
|
||||
| rocm aiter moe | standard | none,</br>mxfp4,</br>fp8 | G(32),G(128),A,T | silu, gelu,</br>swigluoai,</br>situ | Y | Y | [`rocm_aiter_fused_experts`][vllm.model_executor.layers.fused_moe.experts.rocm_aiter_moe.rocm_aiter_fused_experts],</br>[`AiterUnquantizedExperts`][vllm.model_executor.layers.fused_moe.experts.rocm_aiter_moe.AiterUnquantizedExperts],</br>[`AiterFp8BlockExperts`][vllm.model_executor.layers.fused_moe.experts.rocm_aiter_moe.AiterFp8BlockExperts],</br>[`AiterFp8TensorExperts`][vllm.model_executor.layers.fused_moe.experts.rocm_aiter_moe.AiterFp8TensorExperts],</br>[`AiterFp8ChannelExperts`][vllm.model_executor.layers.fused_moe.experts.rocm_aiter_moe.AiterFp8ChannelExperts],</br>[`AiterMxfp4Experts`][vllm.model_executor.layers.fused_moe.experts.rocm_aiter_moe.AiterMxfp4Experts] |
|
||||
| cpu_moe | standard | N/A | N/A | silu, gelu,</br>gelu_tanh,</br>swigluoai | Y | N | [`X86CPUUnquantizedExperts`][vllm.model_executor.layers.fused_moe.experts.cpu_moe.X86CPUUnquantizedExperts],</br>[`ArmCPUUnquantizedExperts`][vllm.model_executor.layers.fused_moe.experts.cpu_moe.ArmCPUUnquantizedExperts],</br>[`CPUUnquantizedExperts`][vllm.model_executor.layers.fused_moe.experts.cpu_moe.CPUUnquantizedExperts] |
|
||||
| naive batched<sup>4</sup> | batched | int8,</br>fp8 | G,A,T | silu, gelu | <sup>6</sup> | Y | [`NaiveBatchedExperts`][vllm.model_executor.layers.fused_moe.experts.fused_batched_moe.NaiveBatchedExperts] |
|
||||
|
||||
|
||||
@@ -0,0 +1,302 @@
|
||||
# Zero-Copy Shared-Memory Tensor Arena for `MessageQueue`
|
||||
|
||||
This document describes an optimization to vLLM's engine→worker IPC path for
|
||||
multimodal serving under tensor parallelism, implemented in
|
||||
`vllm/distributed/device_communicators/shm_tensor_arena.py` and wired into
|
||||
`shm_broadcast.py`'s `MessageQueue`. It **builds on the existing out-of-band
|
||||
tensor channel** (`_reduce_tensor` / `_rebuild_tensor`, still in
|
||||
`shm_broadcast.py`), which already keeps CPU-tensor bytes out of the pickle
|
||||
stream.
|
||||
|
||||
> **Status: experimental, opt-in.** This lands as an increment over the
|
||||
> existing out-of-band channel, not a replacement for it. vLLM also has an
|
||||
> existing multimodal-tensor shm caching mechanism; reconciling the two is
|
||||
> intentionally left as follow-up work rather than blocking this PR.
|
||||
|
||||
## TL;DR
|
||||
|
||||
`MessageQueue.enqueue` already avoids copying CPU tensor bytes into the pickle
|
||||
stream: `_reduce_tensor` emits each tensor as a protocol-5 out-of-band
|
||||
`PickleBuffer`, and large payloads are published to the local readers over a
|
||||
ZMQ socket. That removed the dominant cost of the older *in-band* path — the
|
||||
per-tensor `pickle` serialize (see [History](#history-the-in-band-pickle-era)).
|
||||
|
||||
Two costs remain for a large multimodal `pixel_values` tensor on a TP=N worker:
|
||||
|
||||
1. **N transport copies.** On the overflow (large-payload) path the out-of-band
|
||||
buffer is published to each of the N node-local readers via `send_multipart`,
|
||||
so the tensor bytes are moved once **per reader** by the transport — ~N×200 MB
|
||||
of host data motion for a payload the model needs exactly once.
|
||||
2. **A pageable H2D.** Each reader zero-copies a view over its received ZMQ
|
||||
frame, but that frame is ordinary pageable memory, so the reader's subsequent
|
||||
`cudaMemcpyAsync` to the GPU pays pageable-copy staging plus first-touch
|
||||
faults.
|
||||
|
||||
This change adds a slotted shared-memory **arena**: the writer does **one**
|
||||
memcpy of the tensor into a free slot; every reader takes a **zero-copy view**
|
||||
of the same slot (no per-reader transport copy); and the mapping is
|
||||
`cudaHostRegister`-pinned so the H2D is a true DMA. Net for a 200 MB image on
|
||||
TP=4: **1 copy + a pinned DMA** instead of **4 transport copies + a pageable
|
||||
H2D**. Small tensors, non-contiguous tensors, and arena exhaustion all fall
|
||||
through to the existing `_reduce_tensor` path unchanged.
|
||||
|
||||
## Scope of the win (and what still needs measuring)
|
||||
|
||||
The arena is an **incremental** improvement layered on the out-of-band channel,
|
||||
not a from-scratch fix — the out-of-band channel already eliminated the single
|
||||
largest cost (the serialize copy). The benefit that remains is:
|
||||
|
||||
- **N→1 copies**, which grows with TP degree (no gain at TP=1), and
|
||||
- the **pinned vs. pageable H2D**, which shows up as reader-side H2D latency.
|
||||
|
||||
> **The benchmarks in [§4](#4-validation) were measured against the *in-band
|
||||
> pickle* baseline that predates `_reduce_tensor`**, where the dominant cost was
|
||||
> the serialize copy. They are retained as motivation and as an upper bound. The
|
||||
> incremental improvement of the arena *over the current out-of-band path on
|
||||
> `main`* is smaller and has **not yet been re-measured**; that A/B (arena vs.
|
||||
> oob, same seeded workload) is the outstanding validation item before this is
|
||||
> merge-ready.
|
||||
|
||||
## 1. Where it bites
|
||||
|
||||
Any multimodal model served with `--tensor-parallel-size > 1` through the
|
||||
multiproc executor. The `rpc_broadcast_mq` `MessageQueue`
|
||||
(`v1/executor/multiproc_executor.py`) carries scheduler output — including the
|
||||
raw preprocessed `pixel_values` — from the EngineCore to every TP worker each
|
||||
step. Images at native resolution reach tens to hundreds of MB (a ~4k×4k image
|
||||
is ~200 MB as bf16 patches). The out-of-band channel keeps those bytes out of
|
||||
the pickle stream, but on the overflow path they are still published **once per
|
||||
node-local reader**, and each reader's H2D is from **pageable** memory. The
|
||||
per-reader copy cost scales with TP degree; the pinning cost is a fixed per-image
|
||||
tax on the reader H2D. Latency-sensitive (interactive) serving makes the residual
|
||||
visible in TTFT and, via head-of-line blocking, in the end-to-end latency of
|
||||
*other* requests.
|
||||
|
||||
### History: the in-band-pickle era
|
||||
|
||||
Before `_reduce_tensor`, `MessageQueue.enqueue` pickled the whole payload —
|
||||
including tensor bytes — because `torch.Tensor.__reduce_ex__` does not emit
|
||||
protocol-5 out-of-band buffers. A ~200 MB tensor was byte-copied into the pickle
|
||||
stream on the engine (`THPStorage_writeFileRaw`, ~6.5 ms/MB → ~1.3 s), *inside*
|
||||
the EngineCore step loop, so all GPUs on the worker went idle for **0.9–1.3 s per
|
||||
large image** (the vision-encoder compute for the same image is 50–90 ms →
|
||||
transport dominated compute ~15:1); each reader then paid a full `pickle.loads`
|
||||
byte-copy. Per 200 MB image on TP=4 that was ~1.8 GB of host data motion.
|
||||
|
||||
The `_reduce_tensor` out-of-band channel removed the serialize copy (and the
|
||||
per-reader `pickle.loads` copy on the zero-copy ZMQ-frame path). Note that
|
||||
merely raising `VLLM_MQ_MAX_CHUNK_BYTES_MB` (ring vs. socket path) did **not**
|
||||
help in the in-band era: the serialize ran before the ring-vs-socket branch, so
|
||||
it was common to both — a negative result that pinned the real cost on the
|
||||
byte-copies, not the transport choice.
|
||||
|
||||
## 2. The arena
|
||||
|
||||
The arena itself (`ShmTensorArena`, `_ArenaPickler`, `_rebuild_arena_tensor`)
|
||||
lives in `shm_tensor_arena.py`; `shm_broadcast.py`'s `MessageQueue` creates
|
||||
and wires it in but otherwise treats it as an internal implementation detail.
|
||||
|
||||
### 2.1 `ShmTensorArena`
|
||||
|
||||
A second shared-memory region created alongside the existing `ShmRingBuffer`:
|
||||
|
||||
- **N slots × slot_bytes** (default 8 × 256 MB, internal constants -- see
|
||||
§3), plus per-slot metadata `[written_flag, reader0_done … readerN_done]`.
|
||||
- Concurrency uses the **same lock-free single-writer/N-reader protocol as
|
||||
`ShmRingBuffer`** (memory fences, per-reader done flags), so the model is one
|
||||
the codebase already trusts.
|
||||
- Created by the queue **writer** in `MessageQueue.__init__` when all readers are
|
||||
node-local; readers attach via a new `tensor_arena_handle` field on the queue
|
||||
`Handle`. Queues with remote readers get no arena and keep today's behavior.
|
||||
- Pages are allocated lazily by the kernel on first write, so arenas on queues
|
||||
that never carry big tensors (e.g. worker→engine response queues) cost
|
||||
approximately nothing beyond the (virtual) reservation.
|
||||
|
||||
### 2.2 Writer path — arena diversion, composing with `_reduce_tensor`
|
||||
|
||||
`enqueue` builds one `dispatch_table` that routes CPU tensors through
|
||||
`_reduce_tensor` (out-of-band `PickleBuffer`). When a node-local arena exists,
|
||||
the pickler is an `_ArenaPickler` whose `reducer_override` *additionally* diverts
|
||||
**large contiguous** CPU tensors into the arena:
|
||||
|
||||
```python
|
||||
class _ArenaPickler(pickle.Pickler):
|
||||
def reducer_override(self, obj):
|
||||
rebuild_fn = _ARENA_REBUILD_FNS.get(type(obj)) # {Tensor: ..., Parameter: ...}
|
||||
if (rebuild_fn is not None and obj.device.type == "cpu"
|
||||
and obj.layout is torch.strided and obj.is_contiguous()
|
||||
and not obj.requires_grad
|
||||
and obj.numel() * obj.element_size() >= MIN_BYTES):
|
||||
idx = self.arena.write_tensor(obj) # ONE memcpy into a free slot
|
||||
if idx is not None:
|
||||
return (rebuild_fn,
|
||||
(self.arena.shared_memory.name, idx, nbytes, dtype_str, shape))
|
||||
return NotImplemented # fall through to dispatch_table → _reduce_tensor
|
||||
```
|
||||
|
||||
Note the exact-type dict lookup (not `isinstance`), matching how
|
||||
`dispatch_table` itself dispatches: an unrecognized `Tensor` subclass must
|
||||
decline here too, or it would come back as a plain `Tensor`, silently
|
||||
losing its subclass identity. `torch.nn.Parameter` is the one subclass this
|
||||
*does* divert, via its own rebuild function (§2.3).
|
||||
|
||||
`reducer_override` is consulted before an object's normal reduction, and
|
||||
returning `NotImplemented` falls through to the `dispatch_table` — so a diverted
|
||||
tensor becomes a ~100-byte rebuild stub, and **everything the arena declines**
|
||||
(too small, non-contiguous, or arena full) is handled by `_reduce_tensor`
|
||||
exactly as on `main`. The single arena copy —
|
||||
`torch.frombuffer(slot).copy_(t.view(torch.uint8))` — is a multithreaded memcpy
|
||||
at memory bandwidth (~50–70 ms for 200 MB), and, unlike the oob path, the *same*
|
||||
slot is then read zero-copy by all N readers rather than transported to each.
|
||||
|
||||
**The writer never blocks.** If no slot is free or the tensor exceeds the slot
|
||||
size, `write_tensor` returns `None`; the pickler falls through to `_reduce_tensor`
|
||||
— the worst case is exactly the current out-of-band behavior, and deadlock is
|
||||
structurally impossible.
|
||||
|
||||
### 2.3 Reader path — zero copies
|
||||
|
||||
The stub unpickles through a module-level rebuild function:
|
||||
|
||||
```python
|
||||
def _rebuild_arena_tensor(arena_name, slot_idx, nbytes, dtype_str, shape):
|
||||
arena = _TENSOR_ARENAS[arena_name] # this process's registry of attached arenas
|
||||
# get_tensor: a view over the mapped slot — zero bytes copied
|
||||
return arena.get_tensor(slot_idx, nbytes, getattr(torch, dtype_str), shape)
|
||||
```
|
||||
|
||||
`_rebuild_arena_parameter` wraps the same tensor in
|
||||
`torch.nn.Parameter(t, requires_grad=False)`, which shares its storage.
|
||||
`_ARENA_REBUILD_FNS = {torch.Tensor: _rebuild_arena_tensor, torch.nn.
|
||||
Parameter: _rebuild_arena_parameter}` is the full set of types the arena
|
||||
diverts; anything else (any other `Tensor` subclass) declines in
|
||||
`reducer_override` and takes the out-of-band path.
|
||||
|
||||
The registry is keyed by arena shm name (`_TENSOR_ARENAS:
|
||||
weakref.WeakValueDictionary[str, ShmTensorArena]`) rather than a single
|
||||
per-process slot, since nothing guarantees a process only ever attaches to
|
||||
one arena-bearing queue. It holds only a *weak* reference — a strong one
|
||||
would keep the arena (and its pinned mapping) alive for the process
|
||||
lifetime — and needs no explicit removal on `MessageQueue.shutdown()`
|
||||
(unsafe anyway, since shutdown can run on a different thread than an
|
||||
in-flight `dequeue()`); an entry just disappears once nothing else
|
||||
references that arena.
|
||||
|
||||
The rebuilt tensor *is* the shared memory — no transport copy and no deserialize
|
||||
on any rank.
|
||||
|
||||
**Slot lifecycle.** The rebuilt tensor is the *source* of an async H2D while
|
||||
the worker executes that step, so the reader must not release the slot at
|
||||
unpickle time. Release is tied to **the lifetime of the slot's tensor
|
||||
storage**, not a fixed "next dequeue" schedule — needed for callers like
|
||||
`prompt_embeds` that retain the tensor across many `dequeue` calls during
|
||||
chunked prefill, where a fixed schedule would let the writer reclaim (and
|
||||
corrupt) the slot while the worker was still reading it. `get_tensor` builds
|
||||
the tensor via `torch.from_numpy` over a numpy array wrapping the slot; the
|
||||
tensor's storage holds the only reference to that array, so a
|
||||
`weakref.finalize` on it fires only once every tensor, view/slice and wrapper
|
||||
(e.g. `Parameter`) sharing the storage has been freed. Once queued, the
|
||||
pinned fast path (§2.4) adds a second gate: the H2D is a true async DMA that
|
||||
can outlive `execute_model`, so "the storage was freed" alone isn't
|
||||
sufficient.
|
||||
`flush_releases` records a CUDA event on the compute stream after each
|
||||
step's H2D and only marks a slot done once that event completes
|
||||
(non-blocking `event.query()`; not-yet-done just waits one more dequeue).
|
||||
Unpinned, `cudaMemcpyAsync` from pageable memory already stages
|
||||
synchronously, so the slot releases immediately. Either way, the writer
|
||||
requires every reader's done flag before reusing a slot.
|
||||
|
||||
> **Note.** Assumes the multimodal H2D is issued on the worker's
|
||||
> current/default compute stream (true today: mm inputs are copied eagerly, outside the decode CUDA
|
||||
> graph). If a future vLLM issues that copy on a dedicated side stream, the event
|
||||
> must be recorded at the copy site rather than at `flush_releases`.
|
||||
|
||||
### 2.4 Pinning — the zero-copy trap
|
||||
|
||||
A zero-copy view over shared memory the reader process has never touched makes
|
||||
the subsequent `cudaMemcpyAsync` pay **first-touch page faults on ~50k pages plus
|
||||
pageable-copy staging** — the same tax the current oob path pays on its pageable
|
||||
ZMQ frame. Fix: each reader lazily `cudaHostRegister`s the whole arena mapping
|
||||
once (~1 s, first use), after which every H2D from the arena is a **pinned-memory
|
||||
DMA** (~10 ms for 192 MB). Processes without a CUDA context skip registration
|
||||
silently. (Because the arena mapping is stable and reused, this one-time pin
|
||||
amortizes across all future images — an option the transient per-message ZMQ
|
||||
frame does not have.)
|
||||
|
||||
## 3. Configuration
|
||||
|
||||
A single CLI flag toggles the arena; the slot count/size and divert threshold
|
||||
are internal constants (no env vars).
|
||||
|
||||
| Flag (`ParallelConfig` field) | Default | Meaning |
|
||||
| --- | --- | --- |
|
||||
| `--enable-shm-tensor-arena` / `--disable-shm-tensor-arena` (`enable_shm_tensor_arena`) | **off** | Opt in to route large CPU tensors through the arena (reserves slots in `/dev/shm`). Off = the out-of-band `_reduce_tensor` path only, identical to stock behavior. |
|
||||
|
||||
Internal constants in `shm_tensor_arena.py`: **8 slots × 256 MB**; tensors larger
|
||||
than a slot, or smaller than the **8 MB** divert threshold, take the out-of-band
|
||||
`_reduce_tensor` path.
|
||||
|
||||
## 4. Validation
|
||||
|
||||
> **Baseline caveat.** The A/B numbers below were collected against the *in-band
|
||||
> pickle* path that predates `_reduce_tensor`. They quantify the arena vs. that
|
||||
> older baseline (an upper bound), **not** the arena vs. the current out-of-band
|
||||
> path. The current-`main` oob baseline would land between the "stock" and
|
||||
> "arena + pinning" rows; measuring exactly where is the open item.
|
||||
|
||||
1. **Unit tests** (`tests/distributed/test_shm_broadcast.py`): byte-exact
|
||||
zero-copy round-trips (incl. bf16/fp8), slot lifecycle (no reuse until every
|
||||
reader releases; exhaustion / oversize / non-contiguous fall back to the
|
||||
out-of-band path), event-gated release on the pinned path, `_ArenaPickler` ⊕
|
||||
`_reduce_tensor` composition, and an end-to-end `MessageQueue` broadcast
|
||||
through forked processes. (Historical microbenchmark: enqueue of a 199 MB
|
||||
payload took **66.7 ms** vs ~1275 ms for the old in-band serialize.)
|
||||
2. **Same-seed A/B** (baseline = in-band pickle, pre-`_reduce_tensor`), interactive
|
||||
multimodal workload, uncapped input images, TP=4 ×2 workers on one 8-GPU node,
|
||||
low qps so individual images decompose cleanly (~1.2k aligned requests):
|
||||
|
||||
| TTFT (ms) | p50 | p90 | p99 | max | >1 s | >1.5 s |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
|
||||
| in-band pickle (old baseline) | 93 | 401 | 1321 | 2139 | 30 | 9 |
|
||||
| + bigger MQ chunk (config lever) | 91 | 368 | 1287 | 2027 | 22 | 5 |
|
||||
| **arena + pinning** | **89** | **241** | **862** | **1375** | **7** | **0** |
|
||||
|
||||
p50 unchanged → `reducer_override` adds no measurable overhead when no large
|
||||
tensor is present.
|
||||
3. **Per-image stall** (CUDA-level all-GPU idle gap around the pixel H2D), vs the
|
||||
in-band baseline: 192 MB image 1318 ms → 681 ms (arena only) → **351 ms class**
|
||||
(arena + pinning); vision-encoder compute for the same image is ~80 ms.
|
||||
4. **Outstanding — arena vs. out-of-band on `main`.** Re-run the same seeded A/B
|
||||
with the baseline set to current `main` (i.e. `_reduce_tensor` on, arena off
|
||||
via `--disable-shm-tensor-arena`) vs. arena on, to quantify the incremental N→1
|
||||
copy + pinning gain at TP=4 (and ideally TP=2/8 to show the copy-count scaling).
|
||||
|
||||
## 5. Limitations and future work
|
||||
|
||||
- **Value is TP- and size-dependent.** The arena helps only when a queue has
|
||||
≥2 node-local readers *and* carries tensors ≥ `MIN_MB`; at TP=1, or for small
|
||||
payloads, it adds nothing and everything takes the out-of-band path.
|
||||
- **Shared-memory reservation.** The arena reserves `slots × slot_bytes` of
|
||||
`/dev/shm` (default 2 GB, lazily paged). Like `ShmRingBuffer`, creation is
|
||||
guarded by a free-space check (`check_shm_free_space`), so an undersized
|
||||
`/dev/shm` fails fast with a clear error instead of a `SIGBUS` the first
|
||||
time a tensor is copied into pages beyond tmpfs capacity.
|
||||
- **Relationship to `_reduce_tensor`.** This is strictly additive: the arena is
|
||||
an opt-in fast path for the large-multimodal-tensor case; declining it (or
|
||||
disabling via env) reverts to the merged out-of-band behavior.
|
||||
- **Relationship to existing mm tensor shm caching.** vLLM already has a
|
||||
separate shared-memory caching path for multimodal tensors; this arena is
|
||||
not yet unified with it (see the experimental note in §1). Reconciling the
|
||||
two is follow-up work.
|
||||
- **Slot release granularity**: releases are tied to the slot's tensor
|
||||
storage being freed and, on the pinned path, further gated on a per-slot
|
||||
CUDA event recorded after the consuming H2D (§2.3), which closes both the
|
||||
multi-step-retention and the async-DMA reuse windows. Bursts deeper than
|
||||
the slot count safely fall back to the out-of-band path when the arena is
|
||||
exhausted.
|
||||
- **Fallback observability**: arena exhaustion (no free slot) is a
|
||||
rate-limited log line today; a counter metric would be better. An
|
||||
oversize tensor (bigger than a slot) falls back silently, with no log at
|
||||
all -- worth adding if that path turns out to matter in practice.
|
||||
- **Scope**: the arena activates only when every queue reader is node-local.
|
||||
Remote readers (multi-node PP/TP) keep the existing socket path.
|
||||
@@ -15,23 +15,22 @@ Set `enable_prefix_caching=True` in vLLM engine to enable APC. Here is an exampl
|
||||
|
||||
## Hybrid Mamba models
|
||||
|
||||
Under `--mamba-cache-mode align`, Mamba state is stored only on the Mamba block grid, so a prefix-cache hit can resume only at a block boundary. `--enable-mamba-shared-prefix-checkpoint` also stores a checkpoint at the shared-prefix junction, the point where an earlier request with the same prefix stopped. Requests whose shared prefix ends inside a block can then reuse it.
|
||||
With prefix caching enabled, Mamba state is stored only on the Mamba block grid, so a prefix-cache hit can resume only at a block boundary. `--enable-mamba-shared-prefix-checkpoint` also stores a checkpoint at the shared-prefix junction, the point where an earlier request with the same prefix stopped. Requests whose shared prefix ends inside a block can then reuse it.
|
||||
|
||||
This helps when many requests share a long system prompt and then diverge. It is off by default, and takes effect only when all of the following hold:
|
||||
|
||||
- `--mamba-cache-mode align`
|
||||
- prefix caching enabled
|
||||
- EAGLE/MTP speculative decoding on the Mamba group
|
||||
- `--prefix-match-unit` smaller than the Mamba block size
|
||||
- the model does not use multi-module MTP
|
||||
|
||||
```bash
|
||||
vllm serve <hybrid-model> \
|
||||
--mamba-cache-mode align \
|
||||
--prefix-match-unit 64 \
|
||||
--enable-mamba-shared-prefix-checkpoint
|
||||
```
|
||||
|
||||
`--prefix-match-unit` is required. It sets the granularity at which prefix-cache keys are computed. When unset it defaults to the greatest common divisor of the prefix-cacheable KV cache group block sizes. Under `align` that is the block size itself, so no sub-block boundary exists and the flag has no effect.
|
||||
`--prefix-match-unit` is required. It sets the granularity at which prefix-cache keys are computed. When unset it defaults to the greatest common divisor of the prefix-cacheable KV cache group block sizes. For hybrid Mamba models that is the block size itself, so no sub-block boundary exists and the flag has no effect.
|
||||
|
||||
Choose a value that divides the block size of every prefix-cacheable KV cache group, and that is a multiple of the per-state compression ratio for models that use one, such as sparse MLA. vLLM validates both at startup and names the offending sizes in the error. Read the served block size from the startup log. 64 is a reasonable starting point.
|
||||
|
||||
|
||||
@@ -20,6 +20,7 @@ Batch invariance is crucial for several use cases:
|
||||
Batch invariance is supported on the following platforms:
|
||||
|
||||
- NVIDIA GPUs with compute capability 8.0 or higher.
|
||||
- AMD Instinct GPUs (gfx90a, gfx942, gfx950).
|
||||
- Intel XPUs with Triton support.
|
||||
|
||||
### Attention Backend Selection for XPU
|
||||
|
||||
@@ -96,7 +96,7 @@ EC transfer is configured via `--ec-transfer-config` (CLI) or the `ec_transfer_c
|
||||
| --- | --- | --- | --- |
|
||||
| `ec_enable_nixl` | `bool` | No (default `false`) | Enables NIXL P2P transfer in addition to local CPU offload. Omitted or `false` imports no NIXL/ZMQ. Extra config is not type coerced, so a string value is parsed: `"true"`, `"1"`, `"yes"` enable it, anything else does not. |
|
||||
| `consumer_ack_timeout_s` | `float` | No (default `2.0`) | How long a consumer waits for an `XferAck` before giving up on a read. The producer answers `XferReq`s from its own scheduler step, so its reply latency scales with the encoder's `--max-num-batched-tokens`: a loaded encoder whose steps run longer than this makes consumers abandon reads the producer is about to grant. Raise it for large encoder batches. |
|
||||
| `ec_cpu_bytes` | `int` | Yes | Total size, in bytes, of the shared CPU mmap region. `ECCPUConnector` raises `ValueError` if unset. Block count = `ec_cpu_bytes // block_size_bytes`, where `block_size_bytes = hidden_dim * dtype.element_size()` (`hidden_dim` accounts for Qwen3-VL deepstack: `out_hidden_size * (1 + num_deepstack_layers)`). |
|
||||
| `ec_cpu_bytes` | `int` | Yes | Total size, in bytes, of the shared CPU mmap region. `ECCPUConnector` raises `ValueError` if unset. Block count = `ec_cpu_bytes // block_size_bytes`, where each block holds one LLM input-embedding vector. Entries allocate enough blocks for their modality-specific output shape, including visual DeepStack features. |
|
||||
|
||||
### Environment Variables
|
||||
|
||||
|
||||
@@ -108,10 +108,28 @@ vllm serve <model> \
|
||||
| `store_threshold` | no | `0` | single-tier | Min lookups before a block is offloaded. Values ≥ 2 are rejected by `TieringOffloadingSpec`. |
|
||||
| `max_tracker_size` | no | `64000` | single-tier | Max entries in the lookup tracker. |
|
||||
| `secondary_tiers` | no | `[]` | multi-tier | List of secondary tier configs (see below). |
|
||||
| `tier_poll_interval_s` | no | `0.001` | multi-tier | Pause between tier-poll rounds for tiers that must be serviced between engine steps (currently `p2p`). See [Threading model](#threading-model). Set to `0` to disable the thread and service those tiers once per engine step instead. |
|
||||
| `offload_prompt_only` | no | `true` | both | If `true`, only prompt (prefill) blocks are offloaded; decode blocks are skipped. |
|
||||
| `self_describing_kv_events` | no | `false` | both | Opt-in. When `true` *and* KV cache events are enabled (`--kv-events-config` with `enable_kv_cache_events`), the connector emits self-describing block-granular `BlockStored`/`BlockRemoved` payloads (constituent block hashes, whole-chunk `token_ids`, per-block `block_size`, parent hash, LoRA + group/cache-spec metadata) instead of the placeholder fallback, so external KV-event consumers can index offloaded blocks. Inert unless events are enabled. With `TieringOffloadingSpec`, a CPU promotion is self-describing when a local request observes its primary-tier `HIT` before event translation; otherwise its stored event may retain the placeholder, while a later `HIT` can backfill metadata for removal. Pending-removal/re-promotion races and externally initiated promotions may also produce placeholders, and consumers must ignore removals for unknown hashes. Partial recurrent tails emit the hash-aligned portion from the physical block start through the tail boundary. Other sliding-window/SSM chunks keep the placeholder fallback. In chunk mode (`block_size` > GPU block size, or `blocks_per_chunk` > 1), overlapping chunks re-announce shared per-block hashes, so consumers must reference-count (deduplicate) repeated store/remove announcements. |
|
||||
| `spec_module_path` | no | — | both | Python import path for a custom `OffloadingSpec` not in the built-in registry. Required only when `spec_name` is not built-in (advanced). |
|
||||
|
||||
## Info Metric Labels
|
||||
|
||||
The connector publishes one Prometheus gauge, `vllm:kv_offload_config_info`, and always sets it to 1. The labels hold static facts of the offloading configuration, so a user reads the resolved numbers instead of the launch flags. The gauge appears from the first scheduler step, so an idle engine exposes no series. Each engine reports its own configuration, and not the instance total.
|
||||
|
||||
The gauge holds one series for each configuration that the manager reports. A manager that holds one configuration gives one series. A manager that holds several adds a label of its own, which tells the series apart.
|
||||
|
||||
| Label | Declared by | Meaning | Notes |
|
||||
| --- | --- | --- | --- |
|
||||
| `tier` | `TieringOffloadingSpec` | The tier that this series reports. | `<index>:<type>`. Index `0` is the CPU primary tier, and a secondary tier index starts at 1. `<type>` is the `type` key of the tier config, such as `1:fs`. |
|
||||
| `cpu_num_chunks` | `CPUOffloadingSpec` or `TieringOffloadingSpec` | Chunk slots in the CPU tier. | Chunks, not GPU blocks. `cpu_bytes_to_use` divided by the bytes of one chunk. |
|
||||
|
||||
The spec declares the label names in the API server process, and the manager fills the values in the engine process. The names of every series bind once, at the declaration. A label that one series owns reads empty on every other series.
|
||||
|
||||
`CPUOffloadingSpec` publishes `cpu_num_chunks` on its one series. `TieringOffloadingSpec` publishes it on the `0:primary` series. For a worked example of a secondary tier, the `example` tier publishes one label, `example_info`.
|
||||
|
||||
A manager adds a label with `config_info_keys()` of its spec and `config_info()` of its own class. A name that a manager fills and the spec does not declare is dropped, and the engine log then holds one warning line.
|
||||
|
||||
## Custom Eviction Policies
|
||||
|
||||
`eviction_policy` resolves through `CachePolicyFactory` (`vllm/v1/kv_offload/cpu/policies/factory.py`), which pre-registers the built-in `lru` and `arc` policies.
|
||||
@@ -150,6 +168,19 @@ The filesystem and object-store tiers can publish hash-only `BlockStored` KV eve
|
||||
|
||||
Set the optional `locality` tier field to `LOCAL` or `REMOTE` to describe the tier's storage location relative to the publishing vLLM instance. `LOCAL` marks storage local to that instance, while `REMOTE` marks storage that is not local to it. When the setting is omitted, locality is unspecified. vLLM does not infer it from the tier type, so an `obj` tier is not implicitly `REMOTE`. A KV event includes `locality` only when the tier explicitly configures it. This metadata describes the tier property without implying that a consumer can already route requests to its blocks.
|
||||
|
||||
### Threading model
|
||||
|
||||
Tier methods run in the scheduler process, under the tiering manager's lock, so they are never entered concurrently. They are not always called from the same thread, though.
|
||||
|
||||
Most tiers only ever see the scheduler thread, which holds the lock for the length of one engine step. A tier whose counterpart is not driven by the engine — the `p2p` tier, answering a remote peer — needs servicing while the engine is busy running the model, because otherwise a peer's lookup or fetch waits for the next step boundary. On a saturated rank that wait can be seconds, dwarfing the transfer it gates. Such tiers set `tolerates_off_thread_polling`, and the tiering manager runs one tier-poll thread that, once per `tier_poll_interval_s`, polls them for finished jobs and lets them serve external requests. A tier that implements `serve_external_requests()` without setting the flag is served once per engine step only.
|
||||
|
||||
Two consequences worth knowing:
|
||||
|
||||
- Any tier can be reached from that thread indirectly, because serving a peer looks keys up through the tiering manager, which fans out to the other tiers. Tier state must therefore not assume a particular thread, only that the manager lock is held.
|
||||
- Work the thread deliberately leaves alone is batched per step: a promotion it initiates is submitted at the next `on_schedule_end()`, not immediately.
|
||||
|
||||
Setting `tier_poll_interval_s` to `0` disables the thread; opted-in tiers then get serviced once per engine step, as they were before it existed.
|
||||
|
||||
### Filesystem (FS)
|
||||
|
||||
The filesystem tier (`type: "fs"`) writes blocks to a filesystem directory.
|
||||
|
||||
@@ -52,7 +52,7 @@ CUDA_VISIBLE_DEVICES=0 vllm serve Qwen/Qwen2.5-VL-3B-Instruct \
|
||||
--port 19534 \
|
||||
--gpu-memory-utilization 0.35 \
|
||||
--enable-request-id-headers \
|
||||
--no-enable-prefix-caching \
|
||||
--disable-prefix-caching \
|
||||
--max-num-batched-tokens 16384 \
|
||||
--allowed-local-media-path /path/to/media \
|
||||
--ec-transfer-config '{
|
||||
@@ -72,7 +72,7 @@ CUDA_VISIBLE_DEVICES=1 vllm serve Qwen/Qwen2.5-VL-3B-Instruct \
|
||||
--gpu-memory-utilization 0.75 \
|
||||
--enable-mm-embeds \
|
||||
--enable-request-id-headers \
|
||||
--no-enable-prefix-caching \
|
||||
--disable-prefix-caching \
|
||||
--allowed-local-media-path /path/to/media \
|
||||
--ec-transfer-config '{
|
||||
"ec_connector": "ECMooncakeConnector",
|
||||
|
||||
@@ -398,7 +398,7 @@ The `kv_load_failure_policy` setting controls how the system handles failures wh
|
||||
|
||||
### For NVIDIA GB-series GPUs
|
||||
|
||||
GB-series GPUs support multi-node NVLink. NIXL supports this capability, but KVCache must be registered as VMM during KVCache registration. To enable this feature, you need to set `--enable-cumem-allocator` or `--enable-sleep-mode` flags, and set `UCX_CUDA_IPC_ENABLE_MNNVL: 'y'` env. Otherwise, NIXL can only use RDMA/TCP for cross-node KVCache transfers.
|
||||
GB-series GPUs support multi-node NVLink. NIXL supports this capability, but KVCache must be registered as VMM during KVCache registration. To enable this feature, you need to set the `--enable-cumem-allocator` flag, and set `UCX_CUDA_IPC_ENABLE_MNNVL: 'y'` env. Otherwise, NIXL can only use RDMA/TCP for cross-node KVCache transfers.
|
||||
|
||||
## Experimental Feature
|
||||
|
||||
|
||||
@@ -50,6 +50,26 @@ vllm serve meta-llama/Llama-3.1-8B-Instruct \
|
||||
The daemon itself must load from disk; passing `--load-format ipc_cache` to
|
||||
`vllm preload` is an error.
|
||||
|
||||
## Preloading the FlashInfer autotune cache
|
||||
|
||||
FlashInfer has several implementations of each operation and chooses between
|
||||
them by benchmarking. vLLM runs that pass during kernel warmup and persists the
|
||||
result per rank in the on-disk FlashInfer autotune cache (under
|
||||
`VLLM_CACHE_ROOT`, or `VLLM_FLASHINFER_AUTOTUNE_CACHE_DIR`), so a later engine
|
||||
with the same configuration loads the tuned configs instead of profiling again.
|
||||
Normally the first serving engine pays for the pass, and on a large MoE model it
|
||||
dominates what is left of the startup time once the weights come from the
|
||||
daemon.
|
||||
|
||||
Unless FlashInfer autotune is disabled (`--disable-flashinfer-autotune`),
|
||||
`vllm preload` pays it at preload time: each daemon runs the autotune pass in
|
||||
place — profiling dummy runs against the shard it already holds — after
|
||||
loading and before binding its socket, then writes the cache:
|
||||
|
||||
```bash
|
||||
vllm preload --model /path/to/model --tensor-parallel-size 4
|
||||
```
|
||||
|
||||
## How it works
|
||||
|
||||
1. `vllm preload` spawns one daemon process per GPU. Each daemon loads its
|
||||
@@ -164,8 +184,14 @@ Socket paths are derived from the GPU UUID, so they are stable regardless of
|
||||
- **Platform**: CUDA and ROCm only; other platforms raise
|
||||
`UnsupportedPlatformForIPCError` even when `fallback` is enabled, since it
|
||||
is a permanent misconfiguration rather than a transient daemon outage.
|
||||
- **Parallelism**: tensor, expert and data parallelism are supported;
|
||||
launching the daemon with pipeline parallelism is rejected.
|
||||
- **Parallelism**: tensor, expert, pipeline and data parallelism are
|
||||
supported, including across nodes.
|
||||
- **Autotune preloading**: tuning runs across the daemon's world group,
|
||||
including across nodes and DP ranks. With model-based speculative decoding
|
||||
both daemon groups skip it (the target cannot run the drafter's dummy
|
||||
passes, and a table tuned on the draft model would key on the draft's
|
||||
config, which no engine looks up), so those deployments' first engine
|
||||
tunes and fills the cache instead.
|
||||
- **Quantization**: every quantization method in the model must declare
|
||||
support for pre-processed weights (the daemon transfers weights *after*
|
||||
quantization post-processing). Unsupported methods raise
|
||||
|
||||
@@ -17,7 +17,7 @@ vllm serve <model> \
|
||||
--moe-backend b12x
|
||||
```
|
||||
|
||||
Only pass `--moe-backend b12x` for a compatible NVFP4 or MXFP4 MoE model. The
|
||||
Only pass `--moe-backend b12x` for a compatible packed-weight MoE model. The
|
||||
linear and MoE backends can be selected independently.
|
||||
|
||||
b12x uses MXFP8 activations by default for MXFP4 MoE and the checkpoint's
|
||||
@@ -29,9 +29,34 @@ force BF16 activations for either FP4 weight format.
|
||||
|
||||
| Backend | Supported configurations |
|
||||
| ------- | ------------------------ |
|
||||
| Linear | Per-tensor FP8, 128x128 block FP8, MXFP8, NVFP4, and MXFP4 |
|
||||
| MoE | Tensor-parallel MXFP4 weights with BF16 or MXFP8 activations; NVFP4 weights with BF16, NVFP4, or MXFP8 activations |
|
||||
| Linear | Per-tensor FP8, 128x128 block FP8, MXFP8, NVFP4 (including W4A16), MXFP4, IQ2_XS, IQ2_XXS, and Q8_0 |
|
||||
| MoE | Tensor-parallel MXFP4 weights with BF16 or MXFP8 activations; NVFP4 weights with BF16, NVFP4, or MXFP8 activations; IQ2_XS, IQ2_XXS, and Q8_0 weights with BF16 activations |
|
||||
| Embedding | Packed Q8_0 lookup with BF16 output |
|
||||
|
||||
Dense W4A16 layers are not handled by b12x and continue to use another
|
||||
compatible backend such as Marlin. The b12x MoE backend does not support expert
|
||||
parallelism, expert maps, EXL3, or NF3.
|
||||
The b12x MoE backend does not support expert parallelism, expert maps, EXL3, or NF3.
|
||||
|
||||
## ModelOpt block codecs
|
||||
|
||||
ModelOpt mixed-precision checkpoints select block codecs through the
|
||||
`quantization_config.quantized_layers` entries in `config.json`. Each entry
|
||||
declares `quant_algo`, `group_size`, `block_payload_bytes`, and `packing="ggml"`:
|
||||
|
||||
| `quant_algo` | `group_size` | `block_payload_bytes` |
|
||||
| --- | --- | --- |
|
||||
| `IQ2_XS` | 256 | 74 |
|
||||
| `IQ2_XXS` | 256 | 66 |
|
||||
| `Q8_0` | 32 | 34 |
|
||||
|
||||
Serialized matrices have uint8 shape `[N, K/group_size, block_payload_bytes]`.
|
||||
Dense preparation rearranges the encoded blocks without expanding the matrix
|
||||
to BF16. Routed experts use bias-free SiLU or non-gated ReLU²; tensor-parallel
|
||||
shards must preserve whole blocks and satisfy the kernel's alignment constraints.
|
||||
Only Q8_0 supports packed embedding lookup.
|
||||
|
||||
Quantized normalization weights and other packed vectors are not supported.
|
||||
Store normalization weights as ordinary floating-point tensors and omit their
|
||||
entries from `quantization_config.quantized_layers`.
|
||||
|
||||
Dense, embedding, and expert matrices remain packed. Their execution uses BF16
|
||||
activations without activation quantization. Prepare the kernels and workspaces
|
||||
before CUDA graph capture.
|
||||
|
||||
@@ -111,7 +111,10 @@ Currently supported combinations are:
|
||||
|
||||
| Checkpoint layer | Online target | Requirements |
|
||||
| ---------------- | ------------- | ------------ |
|
||||
| Serialized ModelOpt MXFP8 linear | any | Target-specific hardware requirements |
|
||||
| Serialized ModelOpt MXFP8 linear | any online linear method | Target-specific hardware requirements |
|
||||
| Serialized compressed-tensors NVFP4 MoE | any online MoE method | Target-specific hardware requirements |
|
||||
| Serialized ModelOpt NVFP4 MoE | any online MoE method | Target-specific hardware requirements |
|
||||
| Serialized Quark NVFP4 MoE | any online MoE method | Target-specific hardware requirements |
|
||||
|
||||
For example, serialized ModelOpt MXFP8 linears can be converted to
|
||||
per-channel-weight/per-token-activation FP8 at load time:
|
||||
@@ -122,10 +125,18 @@ vllm serve MiniMaxAI/MiniMax-M3-MXFP8 \
|
||||
--quantization-config.linear fp8_per_channel
|
||||
```
|
||||
|
||||
Similarly, for example, NVFP4 MoE checkpoints can be re-quantized to any online MoE method,
|
||||
for example compressed-tensors NVFP4 experts to online MXFP4:
|
||||
|
||||
```bash
|
||||
vllm serve RedHatAI/Qwen3-30B-A3B-NVFP4 \
|
||||
--quantization-config '{"targets":{"*mlp.experts*":"mxfp4"}}'
|
||||
```
|
||||
|
||||
!!! info
|
||||
|
||||
This is only available for linear layers at the moment.
|
||||
|
||||
Re-quantization is available for both linear and MoE layers.
|
||||
|
||||
Checkpoint-exclusion policy is not enforced by online quantization/re-quantization. Only exclusion through `--quantization-config.targets`/`--quantization-config.ignore` is enforced.
|
||||
|
||||
### Online quantization on unquantized layers from partially-quantized checkpoints
|
||||
|
||||
@@ -77,8 +77,15 @@ llm.wake_up(tags=["kv_cache"])
|
||||
#### Retaining frozen weights during RLHF updates
|
||||
|
||||
Set `sleep_preserve_parameter_names` (CLI: `--sleep-preserve-parameter-names`)
|
||||
to runtime parameter-name glob
|
||||
patterns for weights that stay frozen during training. Each pattern must match
|
||||
to runtime parameter-name glob patterns for weights that stay frozen during
|
||||
training. It covers two use cases:
|
||||
|
||||
- **Frozen Engram tables (original use case):** retain table weights and scales
|
||||
omitted by the trainer, without retransmitting them after every sleep.
|
||||
- **Frozen draft weights:** retain draft-private parameters when the trainer
|
||||
updates only the target model.
|
||||
|
||||
Target patterns match
|
||||
`model.named_parameters()`; checkpoint names and `requires_grad` are not used.
|
||||
|
||||
Level-2 sleep backs up selected GPU parameters to pageable CPU memory; weights
|
||||
@@ -87,7 +94,33 @@ copy. Allow enough host memory for backups. Level-1 behavior is unchanged.
|
||||
|
||||
The trainer must omit these parameters from updates, and reload/post-processing
|
||||
must preserve their values and storage (including avoiding replacement with meta
|
||||
tensors). This option does not filter updates and applies only to the target model.
|
||||
tensors). This option does not filter updates.
|
||||
|
||||
For frozen Engram tables omitted by the trainer, select the target table weights
|
||||
and scales. For example, DeepSeek V4.1 uses these runtime parameter names:
|
||||
|
||||
```bash
|
||||
VLLM_SERVER_DEV_MODE=1 vllm serve deepseek-ai/DeepSeek-V4.1-Flash --enable-sleep-mode \
|
||||
--sleep-preserve-parameter-names '*.engram.embed_tokens.weight*'
|
||||
```
|
||||
|
||||
CPU-offloaded tables remain on CPU without an extra backup copy. Engram's
|
||||
[inference-time CPU offload](engram.md#cpu-offload) is configured separately;
|
||||
this option preserves the frozen tables across level-2 sleep and weight updates.
|
||||
|
||||
To retain a frozen drafter while updating only target weights, use `draft.*`:
|
||||
|
||||
```bash
|
||||
VLLM_SERVER_DEV_MODE=1 vllm serve Qwen/Qwen3.8-27B --enable-sleep-mode \
|
||||
--speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
|
||||
--sleep-preserve-parameter-names 'draft.*'
|
||||
```
|
||||
|
||||
vLLM selects all draft-private parameters internally; you do not need to list
|
||||
them individually. Use `draft.<name>` to select specific draft parameters.
|
||||
Parameters shared with the target are excluded from the draft selection, so
|
||||
target updates remain authoritative. An empty `draft.*` selection is valid.
|
||||
The Engram and draft patterns can be supplied together.
|
||||
|
||||
#### Release only KV cache memory
|
||||
|
||||
|
||||
@@ -6,14 +6,13 @@ To train your own draft models for optimized speculative decoding, see [vllm-pro
|
||||
|
||||
## vLLM Speculation Methods
|
||||
|
||||
vLLM supports a variety of methods of speculative decoding. Model-based methods such as EAGLE, MTP, draft models, PARD and MLP provide the best latency reduction, while simpler methods such as n-gram and suffix decoding provide modest speedups without increasing workload during peak traffic.
|
||||
vLLM supports a variety of methods of speculative decoding. Model-based methods such as EAGLE, MTP, draft models and PARD provide the best latency reduction, while simpler methods such as n-gram and suffix decoding provide modest speedups without increasing workload during peak traffic.
|
||||
|
||||
- [EAGLE](eagle.md)
|
||||
- [LiLiCorr](lilicorr.md)
|
||||
- [Multi-Token Prediction (MTP)](mtp.md)
|
||||
- [Draft Model](draft_model.md)
|
||||
- [Parallel Draft Model (PARD)](parallel_draft_model.md)
|
||||
- [Multi-Layer Perceptron](mlp.md)
|
||||
- [N-Gram](n_gram.md)
|
||||
- [Suffix Decoding](suffix.md)
|
||||
- [Hidden State Extraction](extract_hidden_states.md)
|
||||
@@ -33,7 +32,6 @@ depend on your model family, traffic pattern, hardware, and sampling settings.
|
||||
| MTP | High gain | Medium to high gain | Best when the target model has native MTP support. |
|
||||
| Draft model | High gain | Medium gain | Needs a separate draft model. |
|
||||
| Parallel Draft Model | High gain | Medium to high gain | Low draft model latency. |
|
||||
| MLP speculator | Medium to high gain | Medium gain | Good when compatible MLP speculators are available. |
|
||||
| N-gram | Low to medium gain | Medium gain | Lightweight and easy to enable. |
|
||||
| Suffix decoding | Low to medium gain | Medium gain | No extra draft model; dynamic speculation depth. |
|
||||
| Custom Proposer | Varies | Varies | Bring your own proposer class (experimental). |
|
||||
|
||||
@@ -4,7 +4,7 @@ Speculative decoding buys fewer decode steps with more compute. At batch size 1
|
||||
|
||||
That matters because per-position acceptance decays fast. While the GPU is memory-bound that slot is effectively free and worth the gamble; once it saturates the gamble has a real throughput cost. The crossover moves with load and with workload-dependent acceptance rates, so no static `num_speculative_tokens` is right across concurrencies.
|
||||
|
||||
Adaptive verification decides per step how much of the draft to verify instead. Every (request, position) draft slot is scored by its *survival probability*, the running product of that request's per-position confidences, and the highest-scoring slots are admitted until a global budget is spent. Slots compete across requests: position 5 of a confident request can outrank position 1 of a doubtful one, so one request keeps its full block while another could be trimmed after a token or two.
|
||||
Adaptive verification decides per step how much of the draft to verify instead. Every (request, position) draft slot is scored by its *survival probability*, the running product of that request's per-position acceptance estimates, and the highest-scoring slots are admitted until a global budget is spent. Slots compete across requests: position 5 of a confident request can outrank position 1 of a doubtful one, so one request keeps its full block while another could be trimmed after a token or two.
|
||||
|
||||
The budget itself comes from a cost model profiled at startup. vLLM measures what a step costs at each shape, then picks the token count that maximizes expected accepted tokens per second.
|
||||
|
||||
@@ -12,11 +12,14 @@ The practical effect is that one configuration holds up across the whole load ra
|
||||
|
||||
## Support
|
||||
|
||||
Adaptive verification needs per-position acceptance estimates, so today it is only supported for DSpark with a **confidence head**.
|
||||
Adaptive verification is supported for model-based drafters (EAGLE, MTP, DFlash, DSpark and standalone draft models). The per-position acceptance estimates come from one of two sources:
|
||||
|
||||
- **Confidence head**: DSpark checkpoints that ship a trained confidence head use its predictions directly.
|
||||
- **Online acceptance estimator**: every other drafter predicts acceptance from its draft logits with a small logistic model, recalibrated during serving against the target's verification results.
|
||||
|
||||
## Usage
|
||||
|
||||
It is off by default. Enable it in the speculative config:
|
||||
It is off by default. Enable it in the speculative config with `enable_adaptive_verification`. For example, with DSpark:
|
||||
|
||||
```bash
|
||||
vllm serve deepseek-ai/DeepSeek-V4-Flash-DSpark \
|
||||
@@ -30,6 +33,21 @@ vllm serve deepseek-ai/DeepSeek-V4-Flash-DSpark \
|
||||
}'
|
||||
```
|
||||
|
||||
Or with DFlash on MiMo-V2.5-Pro, whose drafter ships in the checkpoint's `dflash/` subdirectory:
|
||||
|
||||
```bash
|
||||
hf download XiaomiMiMo/MiMo-V2.5-Pro-FP4-DFlash --local-dir MiMo-V2.5-Pro-FP4-DFlash
|
||||
|
||||
vllm serve MiMo-V2.5-Pro-FP4-DFlash \
|
||||
--tensor-parallel-size 4 --trust-remote-code --max-model-len auto \
|
||||
--speculative-config '{
|
||||
"method": "dflash",
|
||||
"model": "MiMo-V2.5-Pro-FP4-DFlash/dflash",
|
||||
"num_speculative_tokens": 8,
|
||||
"enable_adaptive_verification": true
|
||||
}'
|
||||
```
|
||||
|
||||
Set `enable_adaptive_verification: false` to verify the full block for every request.
|
||||
|
||||
## Requirements and limitations
|
||||
@@ -37,6 +55,7 @@ Set `enable_adaptive_verification: false` to verify the full block for every req
|
||||
- The attention backend must tolerate device-decided query lengths, since the CPU lengths only bound them from above. Backends that plan off the CPU lengths are excluded by the attention selector, and rejected at startup for models that hard-wire their backend.
|
||||
- Full cudagraphs are required: step costs are profiled from captured graphs, so `--enforce-eager` is rejected at startup.
|
||||
- Not supported with LoRA (the per-token LoRA mapping is built from CPU-side boundaries) or pipeline parallelism (cost curves and confidences exist only on the last rank).
|
||||
- Without a confidence head, `use_local_argmax_reduction` is rejected. The estimator needs the full draft logits, which local argmax reduction never materializes.
|
||||
|
||||
## Tuning the cost profile
|
||||
|
||||
|
||||
@@ -1,48 +0,0 @@
|
||||
# MLP Draft Models
|
||||
|
||||
The following code configures vLLM to use speculative decoding where proposals are generated by draft models that condition draft predictions on both context vectors and sampled tokens. For more information, see [The Hitchhiker's Guide to Speculative Decoding](https://pytorch.org/blog/hitchhikers-guide-speculative-decoding/) and [IBM Research's Technical Report](https://arxiv.org/abs/2404.19124).
|
||||
|
||||
## MLP Drafter Example
|
||||
|
||||
```python
|
||||
from vllm import LLM, SamplingParams
|
||||
|
||||
prompts = ["The future of AI is"]
|
||||
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
|
||||
|
||||
llm = LLM(
|
||||
model="meta-llama/Meta-Llama-3.1-8B-Instruct",
|
||||
tensor_parallel_size=1,
|
||||
speculative_config={
|
||||
"model": "ibm-ai-platform/llama3-8b-accelerator",
|
||||
"draft_tensor_parallel_size": 1,
|
||||
"method": "mlp_speculator",
|
||||
},
|
||||
)
|
||||
outputs = llm.generate(prompts, sampling_params)
|
||||
|
||||
for output in outputs:
|
||||
prompt = output.prompt
|
||||
generated_text = output.outputs[0].text
|
||||
print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
|
||||
```
|
||||
|
||||
!!! warning "Known issue"
|
||||
`ibm-ai-platform/llama3-70b-accelerator` can fail with:
|
||||
`AttributeError: 'MLPSpeculatorConfig' object has no attribute 'num_attention_heads'`.
|
||||
Track status in [#34106](https://github.com/vllm-project/vllm/issues/34106)
|
||||
and [#34163](https://github.com/vllm-project/vllm/pull/34163).
|
||||
|
||||
## Pre-Trained MLP Drafter Models
|
||||
|
||||
A variety of speculative models of this type are available on HF hub:
|
||||
|
||||
- [llama-13b-accelerator](https://huggingface.co/ibm-ai-platform/llama-13b-accelerator)
|
||||
- [llama3-8b-accelerator](https://huggingface.co/ibm-ai-platform/llama3-8b-accelerator)
|
||||
- [codellama-34b-accelerator](https://huggingface.co/ibm-ai-platform/codellama-34b-accelerator)
|
||||
- [llama2-70b-accelerator](https://huggingface.co/ibm-ai-platform/llama2-70b-accelerator)
|
||||
- [llama3-70b-accelerator](https://huggingface.co/ibm-ai-platform/llama3-70b-accelerator)
|
||||
- [granite-3b-code-instruct-accelerator](https://huggingface.co/ibm-granite/granite-3b-code-instruct-accelerator)
|
||||
- [granite-8b-code-instruct-accelerator](https://huggingface.co/ibm-granite/granite-8b-code-instruct-accelerator)
|
||||
- [granite-7b-instruct-accelerator](https://huggingface.co/ibm-granite/granite-7b-instruct-accelerator)
|
||||
- [granite-20b-code-instruct-accelerator](https://huggingface.co/ibm-granite/granite-20b-code-instruct-accelerator)
|
||||
@@ -12,6 +12,17 @@ Speculators provides the following key features:
|
||||
- **Standardized, extensible format**: Provides a Hugging Face-compatible format for defining speculative models, with tools to convert from external research repositories into a standard speculators format for easy adoption.
|
||||
- **Seamless vLLM Integration**: Built for direct deployment into vLLM, enabling low-latency, production-grade inference with minimal overhead.
|
||||
|
||||
## Verifier model references
|
||||
|
||||
When serving a Speculators checkpoint directly, `speculators_config.verifier.name_or_path` must be a remote model repository ID. References to local files or directories, including local paths that look like repository IDs, are rejected.
|
||||
|
||||
To use a local verifier, select it explicitly and configure the Speculators checkpoint as the draft model:
|
||||
|
||||
```bash
|
||||
vllm serve /path/to/verifier --speculative-config \
|
||||
'{"model": "/path/to/speculator", "method": "eagle3", "num_speculative_tokens": 1}'
|
||||
```
|
||||
|
||||
## Why use Speculators?
|
||||
|
||||
Large language models generate text one token at a time, which creates a fundamental bottleneck: each token requires a full forward pass through the model, leaving GPU compute underutilized while waiting for memory-bound operations.
|
||||
|
||||
@@ -146,7 +146,7 @@ Most OpenAI-compatible clients and agent frameworks never set `strict` on their
|
||||
|
||||
| Value | Behavior |
|
||||
| --- | --- |
|
||||
| `auto` (default) | Follow the request's tool choice and per-tool strictness. Required/named choices activate structural tags; `tool_choice="auto"` activates them when at least one tool sets `strict: true`. |
|
||||
| `auto` | Follow the request's tool choice and per-tool strictness. Required/named choices activate structural tags; `tool_choice="auto"` activates them when at least one tool sets `strict: true`. |
|
||||
| `function` | Constrain the tool-call envelope (markup and the function name) for every request with tools, leaving argument contents free unless the client marked the tool `strict: true`. |
|
||||
| `parameter` | Additionally pin argument schemas for every tool, as if every tool had `strict: true`. |
|
||||
|
||||
@@ -154,6 +154,8 @@ Most OpenAI-compatible clients and agent frameworks never set `strict` on their
|
||||
vllm serve ... --tool-strict-level function
|
||||
```
|
||||
|
||||
When the option is unset, the tool parser's default level applies: `function` for `glm47`, `auto` for the other parsers. Passing `--tool-strict-level auto` overrides a parser default.
|
||||
|
||||
The floor never relaxes a constraint the request would already receive: tools the client marked `strict: true` keep their schemas at every level. With `tool_choice="auto"`, the grammar does not force a tool call; a plain text response stays valid. `VLLM_ENFORCE_STRICT_TOOL_CALLING=false` disables structural tags entirely and takes precedence over this option.
|
||||
|
||||
## Automatic Function Calling
|
||||
|
||||
@@ -56,6 +56,34 @@ watermarking algorithm that does not support it natively, at the cost of
|
||||
weaker detectability. See
|
||||
[Speculative decoding](#speculative-decoding).
|
||||
|
||||
## Monitoring
|
||||
|
||||
When `watermark_config` is set, the Prometheus counter
|
||||
`vllm:watermark_requests_total` counts finished requests by `status`:
|
||||
|
||||
- `watermarked`: the request was sampled with the watermark switched on.
|
||||
- `skipped`: the request asked for the watermark (omitted or `true`) but could
|
||||
not use it, such as `temperature=0` and trace replay requests. A beam search
|
||||
request that asks for the watermark runs without it and logs a warning. Its
|
||||
beam steps count as `not_watermarked`, not `skipped`.
|
||||
- `not_watermarked`: all other finished requests. This includes explicit
|
||||
opt-outs, pooling requests, and the internal requests of beam search (one
|
||||
per beam step), generative scoring and language detection.
|
||||
|
||||
Each finished request has exactly one status, so the sum over `status` equals
|
||||
`vllm:request_success_total` on the same instance. Requests that the client
|
||||
aborts are in neither counter. The counter does not exist without
|
||||
`watermark_config`. With `n > 1`, each sequence counts once. In a
|
||||
disaggregated prefill/decode deployment, both instances count a request: sum
|
||||
over decode instances only. For streaming input, the first input chunk sets
|
||||
the status.
|
||||
|
||||
The status is per request, not per token. A `watermarked` request can contain
|
||||
tokens without the watermark: accepted draft tokens with
|
||||
`allow_target_only_watermarking`, tokens whose context repeats
|
||||
(`deduplicate_contexts`), the first `context_width` generated tokens with
|
||||
`deduplicate_contexts: "all"`, and tokens that a grammar forces.
|
||||
|
||||
## Architecture
|
||||
|
||||
`WatermarkConfig` selects an algorithm and PRF. Model Runner V2 constructs the
|
||||
|
||||
@@ -159,7 +159,7 @@ VLLM_USE_PRECOMPILED=1 VLLM_PRECOMPILED_WHEEL_VARIANT=cpu VLLM_TARGET_DEVICE=cpu
|
||||
### Which `dtype` should be used?
|
||||
|
||||
- Currently, vLLM CPU uses model default settings as `dtype`. However, due to unstable float16 support in torch CPU, it is recommended to explicitly set `dtype=bfloat16` if there are any performance or accuracy problem.
|
||||
- On AMD Zen CPUs (`ZenCpuPlatform`), `float16` is **not** supported. Only `bfloat16` and `float32` are accepted; models declared with `float16` are auto-downcast to `bfloat16` at model load time. See [AMD Zen optimizations](#amd-zen-optimizations).
|
||||
- On AMD Zen CPUs (`ZenCpuPlatform`), `float16` requires AVX512-FP16, available from Venice (6th Gen EPYC) onwards. On Turin (5th Gen EPYC) and earlier only `bfloat16` and `float32` are accepted, and models declared with `float16` are auto-downcast to `bfloat16` at model load time. See [AMD Zen optimizations](#amd-zen-optimizations).
|
||||
|
||||
### How to launch a vLLM service on CPU?
|
||||
|
||||
|
||||
@@ -273,7 +273,9 @@ Otherwise, vLLM falls back to the default `CpuPlatform` (oneDNN / sgl-kernel pat
|
||||
|
||||
### Supported dtypes
|
||||
|
||||
`float16` is **not** supported on `ZenCpuPlatform`. `ZenCpuPlatform.supported_dtypes` advertises only `bfloat16` and `float32`, so models declared with `torch_dtype=float16` are auto-downcast to `bfloat16` at load time with the standard `"Your device 'cpu' doesn't support torch.float16. Falling back to torch.bfloat16 for compatibility."` warning emitted from `vllm/config/model.py`.
|
||||
`ZenCpuPlatform.supported_dtypes` always advertises `bfloat16` and `float32`, and adds `float16` when `torch.ops.zentorch.zentorch_is_fp16_supported()` reports AVX512-FP16, which AMD ships from Venice (6th Gen EPYC) onwards. On Turin (5th Gen EPYC) and earlier, models declared with `torch_dtype=float16` are auto-downcast to `bfloat16` at load time with the standard `"Your device 'cpu' doesn't support torch.float16. Falling back to torch.bfloat16 for compatibility."` warning emitted from `vllm/config/model.py`.
|
||||
|
||||
`bfloat16` is listed first either way, so `--dtype auto` keeps resolving to it.
|
||||
|
||||
### Environment variables
|
||||
|
||||
|
||||
@@ -23,14 +23,28 @@ Therefore, it is recommended to install vLLM with a **fresh new** environment. I
|
||||
|
||||
```bash
|
||||
uv pip install vllm --torch-backend=auto
|
||||
vllm download-kernels
|
||||
```
|
||||
|
||||
??? console "pip"
|
||||
```bash
|
||||
# Install vLLM with CUDA 12.9.
|
||||
pip install vllm --extra-index-url https://download.pytorch.org/whl/cu129
|
||||
vllm download-kernels
|
||||
```
|
||||
|
||||
`vllm download-kernels` installs [FlashInfer](https://docs.flashinfer.ai/)'s
|
||||
precompiled kernels for the installed FlashInfer and CUDA versions. Without them,
|
||||
vLLM downloads and compiles kernels at startup, which can add several minutes on
|
||||
Hopper and newer GPUs, **especially Blackwell**, and logs a warning. Run it in the
|
||||
same Python environment after every vLLM install or upgrade, and in custom
|
||||
container images. The kernels are tied to the FlashInfer version, and FlashInfer
|
||||
refuses to import when they do not match, so an upgrade needs them refreshed.
|
||||
With a PyTorch built for CUDA older than 12.9, only `flashinfer-cubin` is
|
||||
installed, because FlashInfer does not publish `flashinfer-jit-cache` for it.
|
||||
For CUDA overrides and nightly kernels, use the
|
||||
[FlashInfer CLI](https://docs.flashinfer.ai/cli.html#download-kernels) directly.
|
||||
|
||||
We recommend leveraging `uv` to [automatically select the appropriate PyTorch index at runtime](https://docs.astral.sh/uv/guides/integration/pytorch/#automatic-backend-selection) by inspecting the installed CUDA driver version via `--torch-backend=auto` (or `UV_TORCH_BACKEND=auto`). To select a specific backend (e.g., `cu130`), set `--torch-backend=cu130` (or `UV_TORCH_BACKEND=cu130`). If this doesn't work, try running `uv self update` to update `uv` first.
|
||||
|
||||
!!! note
|
||||
@@ -94,9 +108,10 @@ If you only need to change Python code, you can build and install vLLM without c
|
||||
git clone https://github.com/vllm-project/vllm.git
|
||||
cd vllm
|
||||
VLLM_USE_PRECOMPILED=1 uv pip install --editable . --torch-backend=auto
|
||||
vllm download-kernels
|
||||
```
|
||||
|
||||
This command will do the following:
|
||||
The install command will do the following:
|
||||
|
||||
1. Look for the current branch in your vLLM clone.
|
||||
1. Identify the corresponding base commit in the main branch.
|
||||
|
||||
@@ -25,8 +25,11 @@ This guide will help you quickly get started with vLLM to perform:
|
||||
uv venv --python 3.12 --seed
|
||||
source .venv/bin/activate
|
||||
uv pip install vllm --torch-backend=auto
|
||||
vllm download-kernels
|
||||
```
|
||||
|
||||
`vllm download-kernels` installs precompiled kernels that vLLM otherwise downloads and compiles at startup, which can add several minutes on Hopper and newer GPUs. See the [installation guide](installation/gpu.md) for details.
|
||||
|
||||
`uv` can [automatically select the appropriate PyTorch index at runtime](https://docs.astral.sh/uv/guides/integration/pytorch/#automatic-backend-selection) by inspecting the installed CUDA driver version via `--torch-backend=auto` (or `UV_TORCH_BACKEND=auto`). To select a specific backend (e.g., `cu126`), set `--torch-backend=cu126` (or `UV_TORCH_BACKEND=cu126`).
|
||||
|
||||
Another delightful way is to use `uv run` with `--with [dependency]` option, which allows you to run commands such as `vllm serve` without creating any permanent environment:
|
||||
@@ -329,5 +332,5 @@ Some of the available backend options include:
|
||||
- On AMD ROCm: `TRITON_ATTN`, `ROCM_ATTN`, `ROCM_AITER_FA`, `ROCM_AITER_UNIFIED_ATTN`, `TRITON_MLA`, `ROCM_AITER_MLA` or `ROCM_AITER_TRITON_MLA`.
|
||||
- On Intel XPU: `FLASH_ATTN`, `TRITON_ATTN`, `TRITON_MLA`, `XPU_MLA_SPARSE`, `TORCH_SDPA` or `TURBOQUANT`.
|
||||
|
||||
!!! warning
|
||||
There are no pre-built vllm wheels containing Flash Infer, so you must install it in your environment first. Refer to the [Flash Infer official docs](https://docs.flashinfer.ai/) or see [docker/Dockerfile](../../docker/Dockerfile) for instructions on how to install it.
|
||||
!!! tip
|
||||
FlashInfer is installed with vLLM, but its precompiled kernels are not. Run `vllm download-kernels` after installing or upgrading vLLM so they are not downloaded and compiled at startup.
|
||||
|
||||
@@ -145,6 +145,7 @@ def auto_mock(module_name: str, attr: str, max_mocks: int = 100):
|
||||
)
|
||||
|
||||
|
||||
bench_forward = auto_mock("vllm.benchmarks.forward", "main")
|
||||
bench_latency = auto_mock("vllm.benchmarks", "latency")
|
||||
bench_serve = auto_mock("vllm.benchmarks", "serve")
|
||||
bench_startup = auto_mock("vllm.benchmarks", "startup")
|
||||
@@ -178,6 +179,9 @@ LaunchSubcommandBase = auto_mock("vllm.entrypoints.cli.launch", "LaunchSubcomman
|
||||
launch_description = auto_mock("vllm.entrypoints.cli.launch", "DESCRIPTION")
|
||||
RenderSubcommand = auto_mock("vllm.entrypoints.cli.launch", "RenderSubcommand")
|
||||
PreloadSubcommand = auto_mock("vllm.entrypoints.cli.preload", "PreloadSubcommand")
|
||||
DownloadKernelsSubcommand = auto_mock(
|
||||
"vllm.entrypoints.cli.download_kernels", "DownloadKernelsSubcommand"
|
||||
)
|
||||
sweep_subcommands = auto_mock("vllm.benchmarks.sweep.cli", "SUBCOMMANDS")
|
||||
openai_cli_args = auto_mock("vllm.entrypoints.launchers", "cli_args")
|
||||
run_batch = auto_mock("vllm.entrypoints.launchers", "run_batch")
|
||||
@@ -332,7 +336,12 @@ pages = {
|
||||
"cli/complete.md": (create_parser(CompleteCommand.add_cli_args), False),
|
||||
"cli/run-batch.md": (create_parser(run_batch.make_arg_parser), True),
|
||||
"cli/preload.md": (create_parser(PreloadSubcommand.add_cli_args), True),
|
||||
"cli/download-kernels.md": (
|
||||
create_parser(DownloadKernelsSubcommand.add_cli_args),
|
||||
False,
|
||||
),
|
||||
"cli/launch/render.md": (create_parser(RenderSubcommand.add_cli_args), True),
|
||||
"cli/bench/forward.md": (create_parser(bench_forward.add_cli_args), True),
|
||||
"cli/bench/latency.md": (create_parser(bench_latency.add_cli_args), True),
|
||||
# URL kept as `mm_processor` for back-compat; command name is `mm-processor`
|
||||
"cli/bench/mm_processor.md": (
|
||||
|
||||
@@ -38,7 +38,6 @@ def title(text: str) -> str:
|
||||
"openai": "OpenAI",
|
||||
"lmcache": "LMCache",
|
||||
"multilora": "MultiLoRA",
|
||||
"mlpspeculator": "MLPSpeculator",
|
||||
r"fp\d+": lambda x: x.group(0).upper(), # e.g. fp16, fp32
|
||||
r"int\d+": lambda x: x.group(0).upper(), # e.g. int8, int16
|
||||
}
|
||||
|
||||
@@ -51,6 +51,20 @@ vllm serve s3://core-llm/Llama-3-8b \
|
||||
--load-format runai_streamer
|
||||
```
|
||||
|
||||
## Model-specific checkpoint selection
|
||||
|
||||
For models exposing `is_unused_checkpoint_weight`, `runai_streamer` skips
|
||||
safetensors files containing only unused weights before streaming. It uses the
|
||||
same model-owned rule as the default loader, including MTP draft models with
|
||||
this hook; it does not assume a particular MTP tensor prefix.
|
||||
|
||||
Object-storage checkpoints use `model.safetensors.index.json` and verify excluded
|
||||
shards against their safetensors headers before streaming. Mixed shards remain
|
||||
included. Missing or inconsistent metadata retains the original file list.
|
||||
Models without a selection rule keep all files. Distributed streaming with
|
||||
pipeline parallelism also retains the full file list. This optimization does
|
||||
not change `runai_streamer_sharded`.
|
||||
|
||||
## Tunable parameters
|
||||
|
||||
You can tune parameters using `--model-loader-extra-config`:
|
||||
|
||||
@@ -127,7 +127,7 @@ If the model doesn't have a chat template or you want to specify another one,
|
||||
you can explicitly pass a chat template:
|
||||
|
||||
```python
|
||||
from vllm.entrypoints.chat_utils import load_chat_template
|
||||
from vllm.renderers.chat_utils import load_chat_template
|
||||
|
||||
# You can find a list of existing chat templates under `examples/`
|
||||
custom_template = load_chat_template(chat_template="<path_to_template>")
|
||||
|
||||
@@ -37,7 +37,7 @@ You can compute pairwise similarity scores to build a similarity matrix using th
|
||||
|
||||
| Architecture | Models | Example HF Models | [LoRA](../../features/lora.md) | [PP](../../serving/parallelism_scaling.md) |
|
||||
| ------------ | ------ | ----------------- | ------------------------------ | ------------------------------------------ |
|
||||
| `BertModel` | BERT-based | `BAAI/bge-base-en-v1.5`, `Snowflake/snowflake-arctic-embed-xs`, etc. | | |
|
||||
| `BertModel` | BERT-based | `BAAI/bge-base-en-v1.5`, `Snowflake/snowflake-arctic-embed-xs`, etc. | ✅︎ | |
|
||||
| `BertSpladeSparseEmbeddingModel` | SPLADE | `naver/splade-v3` | | |
|
||||
| `BgeM3EmbeddingModel` | BGE-M3 | `BAAI/bge-m3` | | |
|
||||
| `Gemma2Model`<sup>C</sup> | Gemma 2-based | `BAAI/bge-multilingual-gemma2`, etc. | ✅︎ | ✅︎ |
|
||||
@@ -51,9 +51,9 @@ You can compute pairwise similarity scores to build a similarity matrix using th
|
||||
| `NomicBertModel` | Nomic BERT | `nomic-ai/nomic-embed-text-v1`, `nomic-ai/nomic-embed-text-v2-moe`, `Snowflake/snowflake-arctic-embed-m-long`, etc. | | |
|
||||
| `Qwen2Model`<sup>C</sup>, `Qwen2ForCausalLM`<sup>C</sup> | Qwen2-based | `ssmits/Qwen2-7B-Instruct-embed-base` (see note), `Alibaba-NLP/gte-Qwen2-7B-instruct` (see note), etc. | ✅︎ | ✅︎ |
|
||||
| `Qwen3Model`<sup>C</sup>, `Qwen3ForCausalLM`<sup>C</sup> | Qwen3-based | `Qwen/Qwen3-Embedding-0.6B`, `ai-sage/Giga-Embeddings-instruct-480M-0826`<sup>^</sup>, etc. | ✅︎ | ✅︎ |
|
||||
| `RobertaModel`, `RobertaForMaskedLM` | RoBERTa-based | `sentence-transformers/all-roberta-large-v1`, etc. | | |
|
||||
| `RobertaModel`, `RobertaForMaskedLM` | RoBERTa-based | `sentence-transformers/all-roberta-large-v1`, etc. | ✅︎ | |
|
||||
| `VoyageQwen3BidirectionalEmbedModel`<sup>C</sup> | Voyage Qwen3-based with bidirectional attention | `voyageai/voyage-4-nano`, etc. | ✅︎ | ✅︎ |
|
||||
| `XLMRobertaModel` | XLMRobertaModel-based | `BAAI/bge-m3` (see note), `intfloat/multilingual-e5-base`, `jinaai/jina-embeddings-v3` (see note), etc. | | |
|
||||
| `XLMRobertaModel` | XLMRobertaModel-based | `BAAI/bge-m3` (see note), `intfloat/multilingual-e5-base`, `jinaai/jina-embeddings-v3` (see note), etc. | ✅︎ | |
|
||||
| `*Model`<sup>C</sup>, `*ForCausalLM`<sup>C</sup>, etc. | Generative models | N/A | \* | \* |
|
||||
|
||||
!!! note
|
||||
|
||||
@@ -90,6 +90,7 @@ The following [pooling parameters][vllm.PoolingParams] are supported.
|
||||
```python
|
||||
--8<-- "vllm/pooling_params.py:common-pooling-params"
|
||||
--8<-- "vllm/pooling_params.py:embed-pooling-params"
|
||||
--8<-- "vllm/pooling_params.py:token-embed-pooling-params"
|
||||
```
|
||||
|
||||
### `LLM.encode`
|
||||
@@ -108,6 +109,39 @@ data = output.outputs.data
|
||||
print(f"Data: {data!r}")
|
||||
```
|
||||
|
||||
### Fixed-length late chunking
|
||||
|
||||
Late chunking encodes the full input before averaging contextual token representations within each chunk. The chunk means pass through the model's embedding head, including its projection and optional normalization.
|
||||
|
||||
This mode is currently available through the offline `LLM.encode` API for plain-text input. With an instance configured for `token_embed`, enable it per request:
|
||||
|
||||
```python
|
||||
from vllm import PoolingParams
|
||||
from vllm.pooling_params import LateChunkingParams
|
||||
|
||||
outputs = llm.encode(
|
||||
[document],
|
||||
pooling_task="token_embed",
|
||||
pooling_params=PoolingParams(
|
||||
late_chunking_params=LateChunkingParams(chunk_size=256)
|
||||
),
|
||||
)
|
||||
vectors = outputs[0].outputs.data # [num_chunks, embedding_dimension]
|
||||
chunks = outputs[0].late_chunking.chunks
|
||||
```
|
||||
|
||||
For `N` model-input tokens and chunk size `C`, the output contains `ceil(N / C)` vectors. The last chunk uses its actual length. Special tokens and instruction prefixes count toward both chunk boundaries and means. Omitting `late_chunking_params` preserves ordinary token embeddings.
|
||||
|
||||
Each vector row corresponds to one entry in `chunks`. Its `token_range` and `char_range` are zero-based, half-open ranges. Character ranges index the submitted Python string; use `document[start:end]` to retrieve the source text. A chunk containing only special tokens has `char_range=None`.
|
||||
|
||||
Use nonempty text within the model's context window and a fast tokenizer that provides aligned source offsets. Input truncation, explicit padding and renderer text normalization are not supported. The custom document/query outputs of `JinaForRanking` are also unsupported. Late chunking does not extend the model's context window or reduce its attention computation.
|
||||
|
||||
The [offline example](../../../examples/pooling/token_embed/late_chunking_offline.py) loads an E5 model and prints the vector shape, token ranges and source excerpts for two documents:
|
||||
|
||||
```bash
|
||||
python examples/pooling/token_embed/late_chunking_offline.py --chunk-size 16
|
||||
```
|
||||
|
||||
### `LLM.score`
|
||||
|
||||
The [score][vllm.entrypoints.pooling.offline.PoolingOfflineMixin.score] method outputs similarity scores between sentence pairs.
|
||||
|
||||
@@ -0,0 +1,358 @@
|
||||
# Structured Decisions Models
|
||||
|
||||
Structured Decisions Models (aka JEV Models) map inputs directly to predefined
|
||||
options or categories and return probabilities and confidence scores, commonly
|
||||
used for classification and routing.
|
||||
|
||||
## Supported Models
|
||||
|
||||
### Text-only Models
|
||||
|
||||
| Architecture | ReadStrategy | Task | Head | Example HF Models |
|
||||
|---------------------------|---------------------|------------------|-------------------|-------------------------|
|
||||
| `ClefFlashForJointSchema` | `ClefFlashStrategy` | `token_classify` | `JointSchemaHead` | `Cloudflare/clef-flash` |
|
||||
| `Qwen3ForCausalLM` | `NextTokenStrategy` | `generate` | N/A | `Qwen/Qwen3-0.6B`, etc. |
|
||||
|
||||
### Multimodal Models
|
||||
|
||||
!!! note
|
||||
For more information about multimodal models inputs, see [this page](./supported_models.md#list-of-multimodal-language-models).
|
||||
|
||||
| Architecture | Inputs | ReadStrategy | Task | Head | Example HF Models |
|
||||
|--------------------------------------|-------------------------------------|--------------------------------|------------------|-------------------|------------------------------------------|
|
||||
| `ClefOmniForJointSchema` | T + I + V | `JointSchemaStrategy` | `token_classify` | `JointSchemaHead` | `Cloudflare/clef-omni` |
|
||||
| `D3ForSequenceClassification` | T + I + V | `D3Strategy` | `classify` | Linear (`score`) | `vllm-sr/d3`, `vllm-sr/d3-mini`, etc. |
|
||||
| `Lfm2VlForConditionalGeneration` | T + I<sup>+</sup> | `LiquidStrategy` | `generate` | N/A | `LiquidAI/d1-3B` |
|
||||
| `Qwen3_5ForConditionalGeneration` | T + I<sup>E+</sup> + V<sup>E+</sup> | `NextTokenStrategy` | `generate` | N/A | `Qwen/Qwen3.5-0.8B`, etc. |
|
||||
| `Qwen3_5MoeForConditionalGeneration` | T + I<sup>E+</sup> + V<sup>E+</sup> | `NextTokenStrategy` | `generate` | N/A | `Qwen/Qwen3.5-35B-A3B`, etc. |
|
||||
| `DiffusionGemmaForBlockDiffusion` | T + I<sup>+</sup> + V<sup>+</sup> | `DiffusionGemmaCanvasStrategy` | `generate` | N/A | `google/diffusiongemma-26B-A4B-it`, etc. |
|
||||
|
||||
## Online Serving
|
||||
|
||||
### TypeSafe System One Endpoint
|
||||
|
||||
The `/v1/systemone` endpoint answers a set of typed questions about a state and
|
||||
returns a probability for every allowed answer.
|
||||
|
||||
The endpoint is available by default on generation models. Models without a
|
||||
supported read strategy return 501. Supported architectures require
|
||||
`--logprobs-mode raw_logprobs` or `processed_logprobs`, since `label_mass` needs
|
||||
log probabilities.
|
||||
|
||||
#### How it works
|
||||
|
||||
1. Each question is one read. The user message is the state, followed by the
|
||||
question and its options with the type's labels, and an instruction to
|
||||
answer with one label.
|
||||
2. The read is the reply's first token. Each label must be one token there,
|
||||
or the request returns 400.
|
||||
3. The read returns the logprobs of that question's label tokens. Softmax over
|
||||
the labels gives the answer's probabilities.
|
||||
|
||||
Thinking is off unless `chat_template_kwargs` turns it on.
|
||||
|
||||
Probabilities can differ between identical requests. They depend on which
|
||||
requests share a batch and on whether a read hits the prefix cache. Set
|
||||
`VLLM_BATCH_INVARIANT=1` for reads that repeat exactly, at a cost in
|
||||
throughput.
|
||||
|
||||
Every read of a request starts with the state, so with
|
||||
`--enable-prefix-caching` the state is prefilled once and each further
|
||||
question prefills only its own text, then reads one token.
|
||||
|
||||
#### Question types
|
||||
|
||||
| type | criteria | answer |
|
||||
| --- | --- | --- |
|
||||
| `choice` | map of option name to a description or `null` | `choice`, `probabilities` by option name, `confidence` |
|
||||
| `noul` | `null`, or an object with optional `true` and `false` descriptions | `noul` (probability of yes), `probabilities`, `confidence` |
|
||||
| `score` | ordered list of level names | `score` (expected 0-indexed level), `legend`, `probabilities` by level index, `confidence` |
|
||||
|
||||
A choice has at least one option, a score at least two levels. A question id
|
||||
is a non-empty string.
|
||||
|
||||
#### Example
|
||||
|
||||
```bash
|
||||
curl -s http://localhost:8000/v1/systemone \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "Qwen/Qwen3-0.6B",
|
||||
"state": {"ticket": "My card was charged twice for one order."},
|
||||
"questions": {
|
||||
"team": {
|
||||
"type": "choice",
|
||||
"instructions": "Which team should handle this ticket?",
|
||||
"criteria": {
|
||||
"billing": "payments and refunds",
|
||||
"shipping": "deliveries",
|
||||
"security": "account access"
|
||||
}
|
||||
},
|
||||
"urgent": {
|
||||
"type": "noul",
|
||||
"instructions": "Does the customer need a reply within the hour?",
|
||||
"criteria": {"true": "the account is locked or money is moving"}
|
||||
},
|
||||
"tone": {
|
||||
"type": "score",
|
||||
"instructions": "How upset is the customer?",
|
||||
"criteria": ["calm", "annoyed", "furious"]
|
||||
}
|
||||
},
|
||||
"chat_template_kwargs": {"enable_thinking": false}
|
||||
}'
|
||||
```
|
||||
|
||||
Response, with probabilities that depend on the model:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "decision-...",
|
||||
"object": "structured_decision",
|
||||
"created": 1790000000,
|
||||
"model": "Qwen/Qwen3-0.6B",
|
||||
"answers": {
|
||||
"team": {
|
||||
"type": "choice",
|
||||
"choice": "billing",
|
||||
"probabilities": {"billing": 0.97, "shipping": 0.01, "security": 0.02},
|
||||
"confidence": 0.90
|
||||
},
|
||||
"urgent": {
|
||||
"type": "noul",
|
||||
"noul": 0.02,
|
||||
"probabilities": {"yes": 0.02, "no": 0.98},
|
||||
"confidence": 0.95
|
||||
},
|
||||
"tone": {
|
||||
"type": "score",
|
||||
"score": 0.6,
|
||||
"legend": {"0": "calm", "1": "annoyed", "2": "furious"},
|
||||
"probabilities": {"0": 0.7, "1": 0.2, "2": 0.1},
|
||||
"confidence": 0.65
|
||||
}
|
||||
},
|
||||
"usage": {"input_tokens": 142, "output_tokens": 3},
|
||||
"diagnostics": {
|
||||
"team": {"label_mass": 0.93, "argmax_is_label": true},
|
||||
"urgent": {"label_mass": 0.98, "argmax_is_label": true},
|
||||
"tone": {"label_mass": 0.91, "argmax_is_label": true}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`diagnostics.label_mass` is the probability the model put on the labels across
|
||||
its whole vocabulary. A low value means most of the probability went to tokens
|
||||
that are not labels. `confidence` is the chosen option's probability over the
|
||||
whole vocabulary: its share of the labels times `label_mass`.
|
||||
|
||||
#### Labels
|
||||
|
||||
Labels follow the type: a `choice` labels its options `A` to `Z` in the order
|
||||
the request lists them, a `noul` labels its options `yes` and `no`, and a
|
||||
`score` labels its levels `0` to `9`, so a level's label is its score.
|
||||
|
||||
#### Limits
|
||||
|
||||
| limit | value |
|
||||
| --- | --- |
|
||||
| questions per request | 64 |
|
||||
| options per `choice` | 26 |
|
||||
| levels per `score` | 10 |
|
||||
|
||||
#### Request fields
|
||||
|
||||
| field | meaning |
|
||||
| --- | --- |
|
||||
| `state` | what the questions are about: a string, or JSON that is sent as its JSON text |
|
||||
| `questions` | question id to `{type, instructions, criteria}`, asked in this order |
|
||||
| `instructions` | optional context placed ahead of the questions |
|
||||
| `chat_template_kwargs` | passed to the chat template, for example `{"enable_thinking": false}` |
|
||||
| `cache_salt` | optional [prefix-cache salt](../usage/security.md#cache-salting) applied to every question |
|
||||
| `images`, `audio`, `videos` | optional media URLs or data URLs, for models that take them ([Clef-Flash](#clef-flash), images/video; [Clef-Omni](#clef-omni); [Decision 3.0](#decision-30), images/video; [Liquid d1](#liquid-d1), images only) |
|
||||
|
||||
A question with any field other than `type`, `instructions` and `criteria` is
|
||||
rejected with a 400.
|
||||
|
||||
When [tracing](../../examples/observability/opentelemetry/README.md) is enabled
|
||||
with `--otlp-traces-endpoint`, the HTTP `traceparent` and `tracestate` headers are
|
||||
forwarded to every question's generation request.
|
||||
|
||||
### OpenAI Decisions API
|
||||
|
||||
vLLM provides a text-only implementation of OpenAI's
|
||||
[`POST /v1/decisions`](https://developers.openai.com/api/docs/guides/decisions)
|
||||
API. Start a supported model:
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3-0.6B
|
||||
```
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
|
||||
decision = client.decisions.create(
|
||||
model="Qwen/Qwen3-0.6B",
|
||||
input="The package arrived with a broken screen.",
|
||||
questions=[
|
||||
{
|
||||
"type": "predicate",
|
||||
"name": "damaged",
|
||||
"instructions": "Does the customer report a damaged item?",
|
||||
},
|
||||
{
|
||||
"type": "choice",
|
||||
"name": "department",
|
||||
"instructions": "Which department should handle this complaint?",
|
||||
"choices": [{"value": "returns"}, {"value": "billing"}],
|
||||
},
|
||||
{
|
||||
"type": "score",
|
||||
"instructions": "How severe is the damage?",
|
||||
"levels": [{"label": "cosmetic"}, {"label": "broken"}],
|
||||
},
|
||||
],
|
||||
)
|
||||
print(decision.answers)
|
||||
```
|
||||
|
||||
Use OpenAI Python SDK 3.26.0 or newer for `client.decisions.create`.
|
||||
`input` accepts a string or an array of user messages with string content or
|
||||
`input_text` parts. Answers retain question order and echo `name`, including
|
||||
`null` for unnamed questions. Choice values preserve strings and booleans.
|
||||
Scores are probability-weighted averages of zero-based level indices.
|
||||
|
||||
The request contract follows the OpenAI OpenAPI schemas retrieved on
|
||||
October 7, 2026. This MVP supports 1–64 questions, 2–26 choices, and 2–10 score
|
||||
levels. Upstream permits 200 questions and 255 choices; this backend returns
|
||||
a 400 when either model limit is exceeded.
|
||||
The optional `safety_identifier` is caller metadata, available in request
|
||||
body logging with `--enable-log-requests` and DEBUG logging; it does not
|
||||
authenticate callers.
|
||||
|
||||
#### Implementation and limits
|
||||
|
||||
This MVP uses the same label-reading backend as
|
||||
[structured decisions](structured_decisions_models.md). It supports generation
|
||||
architectures with a registered read strategy, including Qwen, DiffusionGemma,
|
||||
and Liquid d1; unsupported models return 501. Each question reads a model's
|
||||
label tokens. Usage reports the actual prompt and output tokens across these
|
||||
reads, including prefix-cache counters.
|
||||
|
||||
Probabilities are normalized over the supplied options. `confidence` is the
|
||||
winning label's probability over the full vocabulary; it is not calibrated
|
||||
to OpenAI's models. Thinking is disabled for these reads.
|
||||
|
||||
Image inputs, other message roles, tools, streaming, and per-question refusal
|
||||
scoring are outside this MVP. Unsupported request fields and input types are
|
||||
rejected. The `/v1/systemone` endpoint retains its existing request format and
|
||||
is available by default.
|
||||
|
||||
## Examples
|
||||
|
||||
### Clef-Flash
|
||||
|
||||
[Clef-Flash](https://huggingface.co/Cloudflare/clef-flash) uses a Qwen3.5-9B
|
||||
backbone and a joint schema head to score all questions in one forward pass.
|
||||
vLLM detects the head and serves the checkpoint as a pooling model.
|
||||
|
||||
```bash
|
||||
vllm serve Cloudflare/clef-flash --max-model-len 16384
|
||||
```
|
||||
|
||||
Send requests to `/v1/systemone` using `state` and `questions`, as in the
|
||||
examples above. `choice`, `noul` and `score` questions are supported.
|
||||
|
||||
- Text and JSON states use the checkpoint's fixed prompt. JSON is compact
|
||||
with sorted keys. `instructions` returns 400; `chat_template_kwargs`
|
||||
has no effect.
|
||||
- The state is truncated to fit 16,384 tokens, or `max_model_len` if smaller.
|
||||
A schema that exceeds this bound returns 400.
|
||||
- `images` and `videos` are placed before the state. Videos are sampled at
|
||||
2 fps and processed by the Qwen3.5 vision processor. Audio inputs return
|
||||
400, and video soundtracks are ignored.
|
||||
- Each request performs one pooling read. `usage.input_tokens` counts the
|
||||
prompt once, `output_tokens` is 0, and `label_mass` is 1.
|
||||
|
||||
### Clef-Omni
|
||||
|
||||
[Clef-Omni](https://huggingface.co/Cloudflare/clef-omni) is a decision model:
|
||||
a joint schema head on the Qwen3-Omni thinker scores every option of every
|
||||
question in one forward pass. vLLM serves it as a pooling model.
|
||||
|
||||
```bash
|
||||
vllm serve Cloudflare/clef-omni
|
||||
```
|
||||
|
||||
- A request is one read. The prompt is the one the checkpoint's
|
||||
`joint_schema_model.py` builds: the state, then every question with its
|
||||
options. A JSON state is written as compact JSON with sorted keys.
|
||||
- A state too long for the model is truncated to fit, as the reference does.
|
||||
A request whose questions alone exceed 64,000 tokens, or `max_model_len`,
|
||||
returns 400.
|
||||
- `instructions` returns 400, since the prompt is fixed. `chat_template_kwargs`
|
||||
has no effect.
|
||||
- There are no labels, so `label_mass` is 1 and `confidence` is the chosen
|
||||
option's probability. `usage.input_tokens` counts the prompt once, and
|
||||
`output_tokens` is 0.
|
||||
- `images`, `audio` and `videos` are read as the reference does: placed
|
||||
before the state, videos sampled at 2 fps, and a video's soundtrack heard
|
||||
with it when every video has one. Audio clips cannot be sent together with
|
||||
videos that have soundtracks.
|
||||
|
||||
### Decision 3.0
|
||||
|
||||
The Decision 3.0 models ([`vllm-sr/d3`](https://huggingface.co/vllm-sr/d3) and
|
||||
its smaller variants such as `vllm-sr/d3-mini` and `vllm-sr/d3-edge`) are a
|
||||
Qwen3.5 backbone with an answer-code readout: one forward pass per question,
|
||||
and the readout scores the question's options at the last token. vLLM detects
|
||||
the checkpoint's `decision_config.json` and serves it as a classification
|
||||
model.
|
||||
|
||||
```bash
|
||||
vllm serve vllm-sr/d3
|
||||
```
|
||||
|
||||
- Each question is one read of the prompt the checkpoint's `d3_format.py`
|
||||
writes: the state, the question and one line per option under an answer
|
||||
code. A question's probabilities are the softmax of its options' code
|
||||
logits at the checkpoint's temperature, so picks and probabilities match
|
||||
`model.system_one()`. `confidence` and `usage` follow the definitions on
|
||||
this page instead.
|
||||
- A choice can have up to 255 options. A JSON state is written as JSON text,
|
||||
and an empty state as `(empty)`.
|
||||
- The full attention layers read the whole prompt in both directions, as the
|
||||
checkpoint declares, so chunked prefill and prefix caching are off.
|
||||
- `instructions` returns 400, since the prompt is fixed.
|
||||
`chat_template_kwargs` has no effect.
|
||||
- `images` and `videos` are placed before the text, images read at up to
|
||||
1.6 megapixels and videos at 2 fps, at most 32 frames of up to
|
||||
0.2 megapixels each, as the checkpoint's runtime does. The media are read
|
||||
once and shared by every question of the request.
|
||||
- `/classify` and `/pooling` return the raw logits of all 255 codes; only the
|
||||
first codes, one per option, belong to a question.
|
||||
|
||||
### Liquid d1
|
||||
|
||||
[`LiquidAI/d1-3B`](https://huggingface.co/LiquidAI/d1-3B) is LFM2.5-VL-3B
|
||||
post-trained to answer decisions as the first token of its reply. vLLM serves
|
||||
it as a generative model.
|
||||
|
||||
```bash
|
||||
vllm serve LiquidAI/d1-3B
|
||||
```
|
||||
|
||||
- The server writes the prompt and reads the label tokens the way the
|
||||
`prompt.py` shipped with the checkpoint does, so picks and probabilities
|
||||
match `model.system_one()`. `confidence` and `usage` follow the definitions
|
||||
on this page instead.
|
||||
- A JSON state is written as indented JSON. A `null` state leaves the images,
|
||||
or the question alone, as the prompt.
|
||||
- `images` are placed before the state and shrunk to at most 1024 x 1024
|
||||
pixels, as the reference does.
|
||||
- A `choice` whose option names are all single letters is labeled with those
|
||||
letters.
|
||||
- When each request carries one question over a state that changes from
|
||||
request to request, `--disable-prefix-caching` lowers latency.
|
||||
@@ -372,6 +372,7 @@ th {
|
||||
| Architecture | Models | Example HF Models | [LoRA](../features/lora.md) | [PP](../serving/parallelism_scaling.md) |
|
||||
| ------------ | ------ | ----------------- | -------------------- | ------------------------- |
|
||||
| `AfmoeForCausalLM` | Afmoe | TBA | ✅︎ | ✅︎ |
|
||||
| `AliceAIForCausalLM` | AliceAI | `yandex/AliceAI-Foundation-80B-A3B-Base` | ✅︎ | |
|
||||
| `ApertusForCausalLM` | Apertus | `swiss-ai/Apertus-8B-2509`, `swiss-ai/Apertus-70B-Instruct-2509`, etc. | ✅︎ | ✅︎ |
|
||||
| `AXK1ForCausalLM` | A.X-K1 | `skt/A.X-K1`, etc. | | ✅︎ |
|
||||
| `BailingMoeForCausalLM` | Ling | `inclusionAI/Ling-lite-1.5`, `inclusionAI/Ling-plus`, etc. | ✅︎ | ✅︎ |
|
||||
@@ -486,6 +487,7 @@ Some models are supported only via the [Transformers modeling backend](#transfor
|
||||
| `HunYuanDenseV1ForCausalLM` | Hunyuan Dense | `tencent/Hunyuan-7B-Instruct` | ✅︎ | ✅︎ |
|
||||
| `HunYuanMoEV1ForCausalLM` | Hunyuan-A13B | `tencent/Hunyuan-A13B-Instruct`, `tencent/Hunyuan-A13B-Pretrain`, `tencent/Hunyuan-A13B-Instruct-FP8`, etc. | ✅︎ | ✅︎ |
|
||||
| `Jais2ForCausalLM` | Jais2 | `inceptionai/Jais-2-8B-Chat`, `inceptionai/Jais-2-70B-Chat`, etc. | ✅︎ | ✅︎ |
|
||||
| `Kolibri1ForCausalLM` | Kolibri 1 | `Aleph-Alpha/Kolibri-1`, `Aleph-Alpha/Kolibri-1-BF16`, etc. | ✅︎ | ✅︎ |
|
||||
| `MellumForCausalLM` | Mellum 2 | `JetBrains/Mellum2-12B-A2.5B-Base`, etc. | ✅︎ | ✅︎ |
|
||||
| `NanbeigeForCausalLM` | Nanbeige4.2 | `Nanbeige/Nanbeige4.2-3B`, etc. | ✅︎ | ✅︎ |
|
||||
| `OlmoForCausalLM` | OLMo | `allenai/OLMo-1B-hf`, `allenai/OLMo-7B-hf`, etc. | ✅︎ | ✅︎ |
|
||||
@@ -580,7 +582,7 @@ These models primarily accept the [`LLM.generate`](./generative_models.md#llmgen
|
||||
| `KimiK25ForConditionalGeneration` | Kimi-K2.5 | T + I<sup>+</sup> | `moonshotai/Kimi-K2.5` | | ✅︎ |
|
||||
| `KimiK3ForConditionalGeneration` | Kimi-K3 | T + I<sup>+</sup> | `moonshotai/Kimi-K3` | | ✅︎ |
|
||||
| `KimiVLForConditionalGeneration` | Kimi-VL-A3B-Instruct, Kimi-VL-A3B-Thinking | T + I<sup>+</sup> | `moonshotai/Kimi-VL-A3B-Instruct`, `moonshotai/Kimi-VL-A3B-Thinking` | | ✅︎ |
|
||||
| `LightOnOCRForConditionalGeneration` | LightOnOCR-1B | T + I<sup>+</sup> | `lightonai/LightOnOCR-1B`, etc | ✅︎ | ✅︎ |
|
||||
| `LightOnOCRForConditionalGeneration` | LightOnOCR-1B | T + I<sup>E+</sup> | `lightonai/LightOnOCR-1B`, etc | ✅︎ | ✅︎ |
|
||||
| `Lfm2VlForConditionalGeneration` | LFM2-VL | T + I<sup>+</sup> | `LiquidAI/LFM2-VL-450M`, `LiquidAI/LFM2-VL-3B`, `LiquidAI/LFM2-VL-8B-A1B`, etc. | ✅︎ | ✅︎ |
|
||||
| `Llama4ForConditionalGeneration` | Llama 4 | T + I<sup>+</sup> | `meta-llama/Llama-4-Scout-17B-16E-Instruct`, `meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8`, `meta-llama/Llama-4-Maverick-17B-128E-Instruct`, etc. | ✅︎ | ✅︎ |
|
||||
| `Llama_Nemotron_Nano_VL` | Llama Nemotron Nano VL | T + I<sup>E+</sup> | `nvidia/Llama-3.1-Nemotron-Nano-VL-8B-V1` | ✅︎ | ✅︎ |
|
||||
@@ -592,7 +594,7 @@ These models primarily accept the [`LLM.generate`](./generative_models.md#llmgen
|
||||
| `MiDashengLMModel` | MiDashengLM | T + A<sup>+</sup> | `mispeech/midashenglm-7b` | | ✅︎ |
|
||||
| `MiMoV2OmniForCausalLM` | MiMo-V2.5-Omni | T + I<sup>E+</sup> + V<sup>E+</sup> + A<sup>+</sup> | `XiaomiMiMo/MiMo-V2.5-Omni` | | ✅︎ |
|
||||
| `MiniCPMO` | MiniCPM-O | T + I<sup>E+</sup> + V<sup>E+</sup> + A<sup>E+</sup> | `openbmb/MiniCPM-o-2_6`, etc. | ✅︎ | ✅︎ |
|
||||
| `MiniCPMV` | MiniCPM-V | T + I<sup>E+</sup> + V<sup>E+</sup> | `openbmb/MiniCPM-V-2` (see note), `openbmb/MiniCPM-Llama3-V-2_5`, `openbmb/MiniCPM-V-2_6`, `openbmb/MiniCPM-V-4`, `openbmb/MiniCPM-V-4_5`, `openbmb/MiniCPM-V-4_6`, etc. | ✅︎ | |
|
||||
| `MiniCPMV` | MiniCPM-V | T + I<sup>E+</sup> + V<sup>E+</sup> | `openbmb/MiniCPM-V-2` (see note), `openbmb/MiniCPM-Llama3-V-2_5`, `openbmb/MiniCPM-V-2_6`, `openbmb/MiniCPM-V-4`, `openbmb/MiniCPM-V-4_5`, `openbmb/MiniCPM-V-4_6`, `openbmb/MiniCPM-V-4_7`, etc. | ✅︎ | |
|
||||
| `MiniMaxM3SparseForConditionalGeneration` | MiniMax-M3 | T + I<sup>+</sup> + V<sup>+</sup> | `MiniMaxAI/MiniMax-M3`, `MiniMaxAI/MiniMax-M3-MXFP8`, etc. | | ✅︎ |
|
||||
| `MiniMaxVL01ForConditionalGeneration` | MiniMax-VL | T + I<sup>E+</sup> | `MiniMaxAI/MiniMax-VL-01`, etc. | | ✅︎ |
|
||||
| `Mistral3ForConditionalGeneration` | Mistral3 (HF Transformers) | T + I<sup>+</sup> | `mistralai/Mistral-Small-3.1-24B-Instruct-2503`, etc. | ✅︎ | ✅︎ |
|
||||
|
||||
@@ -16,6 +16,8 @@ We currently support the following OpenAI APIs:
|
||||
- [Chat Completions batch API](./openai_compatible_server.md#chat-api) (`/v1/chat/completions/batch`)
|
||||
- [Responses API](./openai_compatible_server.md#responses-api) (`/v1/responses`, `/v1/responses/{response_id}`, `/v1/responses/{response_id}/cancel`)
|
||||
- Only applicable to [text generation models](../../models/generative_models.md).
|
||||
- [Decisions API](../../models/structured_decisions_models.md#openai-decisions-api) (`/v1/decisions`)
|
||||
- Supports text-only predicate, choice, and score questions on supported generation models.
|
||||
- [Embeddings API](../../models/pooling_models/embed.md#openai-compatible-embeddings-api) (`/v1/embeddings`)
|
||||
- Only applicable to [embedding models](../../models/pooling_models/embed.md).
|
||||
- [Transcriptions API](./speech_to_text.md#transcriptions-api) (`/v1/audio/transcriptions`)
|
||||
@@ -76,7 +78,7 @@ For further details on speech to text, please refer to [this page](speech_to_tex
|
||||
- [Generative Scoring API](generative_scoring.md) (`/generative_scoring`)
|
||||
- Applicable to [CausalLM models](../../models/generative_models.md) (task `"generate"`).
|
||||
- Computes next-token probabilities for specified `label_token_ids`.
|
||||
- [Structured Decisions API](structured_decisions.md) (`/v1/systemone`)
|
||||
- [Structured Decisions API](../../models/structured_decisions_models.md#typesafe-system-one-endpoint) (`/v1/systemone`)
|
||||
- Applicable to [text generation models](../../models/generative_models.md) with a [chat template](#chat-template).
|
||||
- Answers typed questions about a state with a probability for every allowed answer.
|
||||
|
||||
|
||||
@@ -154,11 +154,15 @@ should split those fields:
|
||||
continue to work.
|
||||
|
||||
Callers that need only the token layout and item hashes, such as cache-aware
|
||||
routers, can set `"return_mm_kwargs": false` on the render request. The response
|
||||
routers, can set `"mm_payload": "none"` on the render request. The response
|
||||
then keeps `mm_hashes` and `mm_placeholders` and sets `kwargs_data` and
|
||||
`mm_metadata` to null, so the processed tensors are neither serialized nor sent.
|
||||
Do not forward such a response to `/inference/v1/generate`, which reads a null
|
||||
`kwargs_data` as every item being cached.
|
||||
`kwargs_data` as every item being cached. The default, `"tensors"`, returns
|
||||
both fields.
|
||||
|
||||
`"return_mm_kwargs": false` is a deprecated alias for `"mm_payload": "none"`
|
||||
and is ignored when `mm_payload` is set.
|
||||
|
||||
## Example
|
||||
|
||||
|
||||
@@ -1,145 +0,0 @@
|
||||
# Structured Decisions
|
||||
|
||||
The `/v1/systemone` endpoint answers a set of typed questions about a state and
|
||||
returns a probability for every allowed answer.
|
||||
|
||||
The endpoint is available by default on generation models. Models without a
|
||||
supported read strategy return 501. Supported architectures require
|
||||
`--logprobs-mode raw_logprobs` or `processed_logprobs`, since `label_mass` needs
|
||||
log probabilities.
|
||||
|
||||
## How it works
|
||||
|
||||
1. Each question is one read. The user message is the state, followed by the
|
||||
question and its options with the type's labels, and an instruction to
|
||||
answer with one label.
|
||||
2. The read is the reply's first token. Each label must be one token there,
|
||||
or the request returns 400.
|
||||
3. The read returns the logprobs of that question's label tokens. Softmax over
|
||||
the labels gives the answer's probabilities.
|
||||
|
||||
Thinking is off unless `chat_template_kwargs` turns it on.
|
||||
|
||||
This endpoint does not serve diffusion models yet.
|
||||
|
||||
Every read of a request starts with the state, so with
|
||||
`--enable-prefix-caching` the state is prefilled once and each further
|
||||
question prefills only its own text, then reads one token.
|
||||
|
||||
## Question types
|
||||
|
||||
| type | criteria | answer |
|
||||
| --- | --- | --- |
|
||||
| `choice` | map of option name to a description or `null` | `choice`, `probabilities` by option name, `confidence` |
|
||||
| `noul` | `null`, or an object with optional `true` and `false` descriptions | `noul` (probability of yes), `probabilities`, `confidence` |
|
||||
| `score` | ordered list of level names | `score` (expected 0-indexed level), `legend`, `probabilities` by level index, `confidence` |
|
||||
|
||||
A choice has at least one option, a score at least two levels. A question id
|
||||
is a non-empty string.
|
||||
|
||||
## Example
|
||||
|
||||
```bash
|
||||
curl -s http://localhost:8000/v1/systemone \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "Qwen/Qwen3-0.6B",
|
||||
"state": {"ticket": "My card was charged twice for one order."},
|
||||
"questions": {
|
||||
"team": {
|
||||
"type": "choice",
|
||||
"instructions": "Which team should handle this ticket?",
|
||||
"criteria": {
|
||||
"billing": "payments and refunds",
|
||||
"shipping": "deliveries",
|
||||
"security": "account access"
|
||||
}
|
||||
},
|
||||
"urgent": {
|
||||
"type": "noul",
|
||||
"instructions": "Does the customer need a reply within the hour?",
|
||||
"criteria": {"true": "the account is locked or money is moving"}
|
||||
},
|
||||
"tone": {
|
||||
"type": "score",
|
||||
"instructions": "How upset is the customer?",
|
||||
"criteria": ["calm", "annoyed", "furious"]
|
||||
}
|
||||
},
|
||||
"chat_template_kwargs": {"enable_thinking": false}
|
||||
}'
|
||||
```
|
||||
|
||||
Response, with probabilities that depend on the model:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "decision-...",
|
||||
"object": "structured_decision",
|
||||
"created": 1790000000,
|
||||
"model": "Qwen/Qwen3-0.6B",
|
||||
"answers": {
|
||||
"team": {
|
||||
"type": "choice",
|
||||
"choice": "billing",
|
||||
"probabilities": {"billing": 0.97, "shipping": 0.01, "security": 0.02},
|
||||
"confidence": 0.90
|
||||
},
|
||||
"urgent": {
|
||||
"type": "noul",
|
||||
"noul": 0.02,
|
||||
"probabilities": {"yes": 0.02, "no": 0.98},
|
||||
"confidence": 0.95
|
||||
},
|
||||
"tone": {
|
||||
"type": "score",
|
||||
"score": 0.6,
|
||||
"legend": {"0": "calm", "1": "annoyed", "2": "furious"},
|
||||
"probabilities": {"0": 0.7, "1": 0.2, "2": 0.1},
|
||||
"confidence": 0.65
|
||||
}
|
||||
},
|
||||
"usage": {"input_tokens": 142, "output_tokens": 3},
|
||||
"diagnostics": {
|
||||
"team": {"label_mass": 0.93, "argmax_is_label": true},
|
||||
"urgent": {"label_mass": 0.98, "argmax_is_label": true},
|
||||
"tone": {"label_mass": 0.91, "argmax_is_label": true}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`diagnostics.label_mass` is the probability the model put on the labels across
|
||||
its whole vocabulary. A low value means most of the probability went to tokens
|
||||
that are not labels. `confidence` is the chosen option's probability over the
|
||||
whole vocabulary: its share of the labels times `label_mass`.
|
||||
|
||||
## Labels
|
||||
|
||||
Labels follow the type: a `choice` labels its options `A` to `Z` in the order
|
||||
the request lists them, a `noul` labels its options `yes` and `no`, and a
|
||||
`score` labels its levels `0` to `9`, so a level's label is its score.
|
||||
|
||||
## Limits
|
||||
|
||||
| limit | value |
|
||||
| --- | --- |
|
||||
| questions per request | 64 |
|
||||
| options per `choice` | 26 |
|
||||
| levels per `score` | 10 |
|
||||
|
||||
## Request fields
|
||||
|
||||
| field | meaning |
|
||||
| --- | --- |
|
||||
| `state` | what the questions are about: a string, or JSON that is sent as its JSON text |
|
||||
| `questions` | question id to `{type, instructions, criteria}`, asked in this order |
|
||||
| `instructions` | optional context placed ahead of the questions |
|
||||
| `chat_template_kwargs` | passed to the chat template, for example `{"enable_thinking": false}` |
|
||||
| `cache_salt` | optional [prefix-cache salt](../../usage/security.md#cache-salting) applied to every question |
|
||||
|
||||
A question with any field other than `type`, `instructions` and `criteria` is
|
||||
rejected with a 400.
|
||||
|
||||
When [tracing](../../../examples/observability/opentelemetry/README.md) is enabled
|
||||
with `--otlp-traces-endpoint`, the HTTP `traceparent` and `tracestate` headers are
|
||||
forwarded to every question's generation request.
|
||||
@@ -55,6 +55,21 @@ The server returns a 400 for:
|
||||
|
||||
For using `output_mode` with separate prefill and decode pools, see [Disaggregated Prefilling](../../features/disagg_prefill.md#generate-api-output-modes).
|
||||
|
||||
## Weight versions
|
||||
|
||||
The final output of every choice carries `weight_versions`, the label of the weights that sampled each range of its output tokens. Ranges are half-open over output-token indexes (prompt excluded), contiguous and in generation order, so a request that crossed one weight update gets two:
|
||||
|
||||
```json
|
||||
"weight_versions": [
|
||||
{"version": "v7", "start": 0, "end": 57},
|
||||
{"version": "v8", "start": 57, "end": 128}
|
||||
]
|
||||
```
|
||||
|
||||
- With `stream: true`, only the last chunk of each choice carries the list, including the last chunk after an abort. Its offsets cover the whole output. Earlier chunks carry `null` or omit the field.
|
||||
- An output with no tokens, such as a request aborted before its first token, gets `[]`.
|
||||
- The labels are the ones you set with `/update_weight_version` or `/finish_weight_update`, see [Async Reinforcement Learning](../../training/async_rl.md#weight-versions-in-outputs). vLLM doesn't order or check them, and the label is `"default"` until you set one.
|
||||
|
||||
## Aborting requests
|
||||
|
||||
`POST /inference/v1/abort_requests` aborts in-flight requests. It is registered wherever `/inference/v1/generate` is and requires the API key when `--api-key` is set, like `/inference/v1/generate`.
|
||||
|
||||
@@ -71,7 +71,7 @@ Ray also offers high-level APIs for large-scale [offline batch inference](https:
|
||||
Ray is an optional dependency. Install it explicitly before using Ray-based execution, for example:
|
||||
|
||||
```bash
|
||||
pip install "ray[cgraph]"
|
||||
pip install ray
|
||||
```
|
||||
|
||||
For details, see the [Ray documentation](https://docs.ray.io/en/latest/index.html).
|
||||
|
||||
@@ -43,10 +43,11 @@ With `VLLM_SERVER_DEV_MODE=1`, the vLLM HTTP server exposes the same functionali
|
||||
- `POST /pause?mode=keep` - Pause generation
|
||||
- `POST /resume` - Resume generation
|
||||
- `POST /abort_requests` - Abort in-flight requests without pausing the scheduler (send `{}` to abort all, or `{"request_ids": [...]}`)
|
||||
- `POST /update_weight_version` - Set the `weight_version` label without changing weights (send `{"new_version": "v8"}`)
|
||||
- `GET /weight_info` - Return the latest committed `weight_version`
|
||||
|
||||
!!! note "Data Parallelism"
|
||||
When using data parallelism with vLLM's **internal load balancer** (i.e. `data_parallel_backend="ray"`), pause and resume are handled automatically across all DP ranks -- a single call is sufficient. When using an **external load balancer** (i.e. multiple independent vLLM instances behind a proxy), you must send pause and resume requests to **every** engine instance individually before and after the weight update.
|
||||
When using data parallelism with vLLM's **internal load balancer** (i.e. `data_parallel_backend="ray"`), pause and resume are handled automatically across all DP ranks -- a single call is sufficient. When using an **external load balancer** (i.e. multiple independent vLLM instances behind a proxy), you must send pause and resume requests to **every** engine instance individually before and after the weight update, and set the weight version on every instance too.
|
||||
|
||||
## Typical Async RL Flow
|
||||
|
||||
@@ -54,12 +55,27 @@ A typical async RL loop with weight syncing looks like this:
|
||||
|
||||
1. Start generating rollouts from the current policy
|
||||
2. Once trainer has new weights to update to, pause generation with `mode="keep"`
|
||||
3. Sync the updated weights from the trainer to the inference engine (see [Weight Transfer](weight_transfer/README.md))
|
||||
3. Sync the updated weights from the trainer to the inference engine (see [Weight Transfer](weight_transfer/README.md)) and set their weight version
|
||||
4. Resume generation -- in-flight requests continue with the new weights
|
||||
5. Repeat
|
||||
|
||||
The key insight is that requests paused with `mode="keep"` will produce tokens from the **old** weights before the pause and tokens from the **new** weights after resume. The `clear_cache` parameter controls whether the KV cache is invalidated during the pause. When `clear_cache=True`, previously cached key-value entries are discarded, so all tokens generated after resume will be computed entirely with the new weights. When `clear_cache=False`, existing KV cache entries are retained, meaning some tokens in context may still reflect the old weights (stale KV cache).
|
||||
|
||||
## Weight Versions in Outputs
|
||||
|
||||
To tell which tokens came from which weights, set a weight-version label while generation is paused, after the new weights are loaded: pass `weight_version` to `finish_weight_update`, or call `update_weight_version` (`POST /update_weight_version` over HTTP). Labels are strings you choose, such as the trainer step; vLLM doesn't order or check them.
|
||||
|
||||
The final output of every sequence then carries `weight_versions`, one span per label over its output-token indexes. A request paused with `mode="keep"` across one update gets two spans:
|
||||
|
||||
```python
|
||||
completion = final_output.outputs[0]
|
||||
for span in completion.weight_versions:
|
||||
# The tokens in [span.start, span.end) were sampled by weights `span.version`.
|
||||
print(span.version, completion.token_ids[span.start : span.end])
|
||||
```
|
||||
|
||||
The offsets cover the whole output, so with streamed delta outputs, collect the tokens before slicing. `/inference/v1/generate` returns the same list on each choice; see [Tokens In <> Tokens Out API](../serving/online_serving/token_in_token_out.md#weight-versions).
|
||||
|
||||
## Example
|
||||
|
||||
The [async RLHF example](../../examples/rl/rlhf_async_new_apis.py) demonstrates this pattern with `vllm.AsyncLLMEngine`, NCCL weight transfer, and mid-flight pause/resume with validation.
|
||||
|
||||
@@ -50,7 +50,10 @@ The following metrics are exposed:
|
||||
These metrics are exposed when the `HiSparseConnector` KV connector is
|
||||
configured. The host-pool gauges are reported by the scheduler once per engine
|
||||
step, for the host pool shared by all tensor-parallel ranks; evictable cached
|
||||
blocks count as free.
|
||||
blocks count as free. With `--kv-cache-metrics`, the
|
||||
`vllm:hisparse_host_block_*_seconds` histograms sample host-pool block
|
||||
residency, like the device `vllm:kv_block_*_seconds` histograms, which cover
|
||||
only the GPU pool.
|
||||
|
||||
--8<-- "gen:metrics-hisparse"
|
||||
|
||||
|
||||
+27
-1
@@ -94,7 +94,9 @@ denial of service:
|
||||
| Environment Variable | Default | Description |
|
||||
| --- | --- | --- |
|
||||
| `VLLM_MAX_MEDIA_DOWNLOAD_SIZE_MB` | `256` | Maximum size in MB for a single remote media response. Oversized responses are rejected while streaming before the full body is materialized in memory. |
|
||||
| `VLLM_MAX_MEDIA_REQUEST_DOWNLOAD_SIZE_MB` | `256` | Maximum total HTTP media bytes downloaded for one request, including concurrent downloads. Enforced while streaming. |
|
||||
| `VLLM_MAX_IMAGE_PIXELS` | `178956970` (~179M pixels) | Maximum decoded image size in pixels. Images exceeding this are rejected before raster memory is allocated. Default matches PIL's built-in 2x decompression-bomb threshold (~680 MB for RGB). |
|
||||
| `VLLM_MAX_MEDIA_REQUEST_DECODE_BYTES` | `715827880` (~683 MiB) | Maximum aggregate decoded image, video-frame, and audio bytes in one request. Images are charged at four bytes per pixel before decoding; audio is limited before decoding; video files are charged after frame extraction. |
|
||||
| `VLLM_MAX_AUDIO_CLIP_FILESIZE_MB` | `25` | Maximum compressed filesize in MB for a single audio file. Enforced on all audio inputs (multimodal chat URLs, speech-to-text uploads, data: URLs, and local file paths) before decoding begins. |
|
||||
| `VLLM_MAX_AUDIO_DECODE_DURATION_S` | `600` | Maximum decoded audio duration in seconds. Prevents compressed audio from expanding into gigabytes of float32 PCM. |
|
||||
| `VLLM_MAX_AUDIO_DECODE_BYTES` | `268435456` (256 MiB) | Maximum float32 PCM bytes that audio decoding may allocate. Guards against sample-rate forgery where an inflated header sample rate bypasses the duration guard while the actual frame count causes a multi-GiB allocation. |
|
||||
@@ -173,12 +175,13 @@ When `--api-key` is configured, the following endpoints require Bearer token aut
|
||||
- `/v1/responses/render` - Render a self-contained response request (available on `vllm serve` only when `--enable-scale-out` is set, or on `vllm launch render` unless explicitly disabled)
|
||||
- `/v1/responses/{response_id}` - Retrieve a response
|
||||
- `/v1/responses/{response_id}/cancel` - Cancel a response
|
||||
- `/v1/decisions` - Text decisions API
|
||||
- `/v1/score` - Scoring API
|
||||
- `/v1/rerank` - Reranking API
|
||||
- `/v1/systemone` - Structured decisions API
|
||||
- `/v1/load_lora_adapter` - Load a LoRA adapter (can alter model behavior; only available when `--enable-lora` is set and `VLLM_ALLOW_RUNTIME_LORA_UPDATING=True`)
|
||||
- `/v1/unload_lora_adapter` - Unload a LoRA adapter (can alter model behavior; only available when `--enable-lora` is set and `VLLM_ALLOW_RUNTIME_LORA_UPDATING=True`)
|
||||
- `/inference/v1/generate` - Generate completions (available when `--enable-scale-out` is set, or with `--tokens-only`)
|
||||
- `/inference/v1/generate` - Generate completions (available when `--enable-scale-out` is set, or with `--tokens-only`; otherwise returns HTTP 403 with instructions to enable it)
|
||||
- `/cohere/v2/chat/render` - Render Cohere Chat v2 requests (requires `VLLM_ENABLE_COHERE_API=1` and either `--enable-scale-out` or `--tokens-only`)
|
||||
- `/v2/embed` - Cohere Embed API
|
||||
- `/v2/rerank` - Cohere Rerank API
|
||||
@@ -208,9 +211,13 @@ The following endpoints **do not require authentication** even when `--api-key`
|
||||
- `/update_weights` - Update model weights (can alter model behavior)
|
||||
- `/get_world_size` - Get distributed world size
|
||||
- `/abort_requests` - Abort in-flight requests (available with `--tokens-only`. Use the authenticated `/inference/v1/abort_requests` otherwise)
|
||||
- `/fault_tolerance/apply` - Trigger fault-recovery operations across ranks (available when fault tolerance is enabled)
|
||||
- `/fault_tolerance/status` - Query fault-recovery status (available when fault tolerance is enabled)
|
||||
|
||||
**Utility endpoints:**
|
||||
|
||||
- `/metrics` - Prometheus metrics, including model, throughput, queue, and KV-cache statistics
|
||||
- `/kv_event_sources` - KV-event publisher configuration for each data-parallel rank, which may reveal internal transport endpoints (when KV events are configured)
|
||||
- `/tokenize` - Tokenize text (not gated by `--enable-scale-out`)
|
||||
- `/detokenize` - Detokenize tokens
|
||||
- `/health` - Health check
|
||||
@@ -303,6 +310,25 @@ Only deployments whose API clients are trusted should start the server with
|
||||
`--trust-request-mm-kwargs` to restore per-request overrides. Do not enable
|
||||
this option on an endpoint exposed to untrusted clients.
|
||||
|
||||
### P2P KV-Offload Peer Session Limits
|
||||
|
||||
Deployments that enable the P2P secondary tier (`OffloadingConnector` with `TieringOffloadingSpec` and a `p2p` secondary tier) accept peer addresses from `kv_transfer_params` in API requests. Without limits, an attacker with API access could submit many unique non-listening peer addresses, each consuming ZeroMQ sockets until the context quota is exhausted, crashing the EngineCore.
|
||||
|
||||
vLLM enforces several configurable limits to prevent this:
|
||||
|
||||
| Environment Variable | Default | Description |
|
||||
| --- | --- | --- |
|
||||
| `VLLM_P2P_MAX_PEERS` | `128` | Maximum concurrent P2P peer sessions. New peers (both outbound and inbound) are rejected when this limit is reached. |
|
||||
| `VLLM_P2P_HANDSHAKE_TIMEOUT_S` | `30` | Seconds to wait for a newly connected peer to complete the P2P handshake. Sessions that remain connected but not ready after this deadline are torn down. |
|
||||
| `VLLM_P2P_IDLE_TIMEOUT_S` | `300` | Seconds a P2P session may remain idle (no pending work, no messages) before it is eligible for eviction. Idle eviction runs only when remaining peer slots are within 8 of `VLLM_P2P_MAX_PEERS`. Set to `0` to disable idle eviction. |
|
||||
|
||||
Connections that fail (ZMQ socket errors, handshake failures) are handled gracefully — the request falls back to local prefill instead of crashing the engine.
|
||||
|
||||
**Recommendations:**
|
||||
|
||||
- **Public-facing deployments with P2P enabled:** Lower `VLLM_P2P_MAX_PEERS` to the expected number of legitimate peers (e.g., `16` or `32`).
|
||||
- **Sensitive deployments:** Consider deploying behind a reverse proxy that strips or validates `kv_transfer_params` from API requests to prevent client-supplied peer addresses entirely.
|
||||
|
||||
## Tool Server and MCP Security
|
||||
|
||||
vLLM supports connecting to external tool servers via the `--tool-server` argument. This enables models to call tools through the Responses API (`/v1/responses`). Tool server support works with all models — it is not limited to specific model architectures.
|
||||
|
||||
Reference in New Issue
Block a user