Merge remote-tracking branch 'upstream/main' into patches

Signed-off-by: MOVZX <movzx@yahoo.com>
This commit is contained in:
2026-10-03 21:59:03 +07:00
1012 changed files with 61873 additions and 14499 deletions
+6
View File
@@ -88,6 +88,12 @@ vllm bench sweep serve \
By default, each parameter combination is benchmarked 3 times to make the results more reliable. You can adjust the number of runs by setting `--num-runs`.
You can optionally run an unmeasured warmup before each parameter combination by setting `--warmup-num-prompts`. The warmup result is saved as `warmup.json` for inspection, but is excluded from the measured summary.
You can use `--continue-on-error` to record a failed warmup, benchmark run, or server configuration and continue processing the remaining sweep combinations. Failure details are saved in `.failure.json` files.
Both options are also supported by `vllm bench sweep serve_workload`.
!!! important
If both `--serve-params` and `--bench-params` are passed, the script will iterate over the Cartesian product between them.
You can use `--dry-run` to preview the commands to be run.
+3
View File
@@ -135,6 +135,9 @@ API_AUTONAV_EXCLUDE=vllm mkdocs serve # API ref off (~15 seconds)
Once you see a `Serving on http://<address>:<port>/` line in the logs, the live preview is ready!
Open that address in your browser to see it — `http://127.0.0.1:8000/` by default, or whichever host/port you passed to `-a`.
On pull requests, Read the Docs builds a documentation preview only if the PR changes files that affect the docs (e.g. `docs/`, `examples/` or `mkdocs.yaml`) and the `pre-commit` checks have not failed.
Changes to `vllm/` alone do not trigger a build, so if your PR changes the API or CLI reference, add the `build-docs` label to force one.
For additional features and advanced configurations, refer to the:
- [MkDocs documentation](https://www.mkdocs.org/)
+2
View File
@@ -85,6 +85,8 @@ def forward(
For reference, check out our [Llama implementation](../../../vllm/model_executor/models/llama.py). vLLM already supports a large number of models. It is recommended to find a model similar to yours and adapt it to your model's architecture. Check out [vllm/model_executor/models](../../../vllm/model_executor/models) for more examples.
Adapting model-specific structure or boilerplate can keep architectures independent. Reuse shared layer implementations, such as Attention and MoE, where their contracts fit the model.
## 3. (Optional) Implement tensor parallelism and quantization support
If your model is too large to fit into a single GPU, you can use tensor parallelism to manage it.
+7 -4
View File
@@ -130,10 +130,13 @@ path. The resolver consumes the existing graph-stable request mapping from
attention metadata; neither the worker nor individual cache handles keep a
duplicate mapping.
Speculative decoding resolves and consumes each verification step in order.
Each step receives distinct replayable plan rows while sharing the request's
hot-cache state, so a later step cannot reuse a hot row before an earlier step
has consumed it.
Speculative decoding resolves all verification rows of a request in one pass:
one block resolves the union of the rows' top-k against the request's hot-cache
state, so rows that select the same host row share its hot row and no row
evicts a hot row another row of the step still reads. Draft layers write their
rows after the target forward, so their host mirror runs at the start of the
next step, after the drafter, and the step's page transfers are submitted
behind it.
## P/D import target
+17 -11
View File
@@ -78,17 +78,23 @@ GPU-side fusion is controlled by `multimodal_config.mm_device_do_normalize`:
- The flag is **enabled by default** for all models that support it.
- Currently, it’s on by default for these architectures:
| name | Architecture | Example HF Models |
|--------------------|------------------------------------------|----------------------------------------|
| `intern-s1-pro` | `InternS1ProForConditionalGeneration` | `internlm/Intern-S1-Pro` |
| `intern-s2-mobius` | `InternS2MobiusForConditionalGeneration` | `internlm/Intern-S2-Mobius` |
| `qwen2-vl` | `Qwen2VLForConditionalGeneration` | `Qwen/Qwen2-VL-2B-Instruct`, etc. |
| `qwen2.5-vl` | `Qwen2_5_VLForConditionalGeneration` | `Qwen/Qwen2.5-VL-3B-Instruct`, etc. |
| `qwen3-vl` | `Qwen3VLForConditionalGeneration` | `Qwen/Qwen3-VL-4B-Instruct`, etc. |
| `qwen3-vl-moe` | `Qwen3VLMoeForConditionalGeneration` | `Qwen/Qwen3-VL-30B-A3B-Instruct`, etc. |
| `qwen3.5` | `Qwen3_5ForConditionalGeneration` | `Qwen/Qwen3.5-0.8B`, etc. |
| `qwen3.5-moe` | `Qwen3_5MoeForConditionalGeneration` | `Qwen/Qwen3.5-35B-A3B`, etc. |
| `qwen4-exp` | `Qwen4ExpForConditionalGeneration` | `Qwen/Qwen3.8-Flash-Next` |
| name | Architecture | Example HF Models |
| -------------------------- | ------------------------------------------ | ----------------------------------------------------- |
| `glm-4v` | `Glm4vForConditionalGeneration` | `zai-org/GLM-4.1V-9B-Thinking`, etc. |
| `glm-5-next` | `Glm5NextForConditionalGeneration` | `zai-org/GLM-5.3-Flash` |
| `glm-ocr` | `GlmOcrForConditionalGeneration` | `zai-org/GLM-OCR` |
| `intern-s1-pro` | `InternS1ProForConditionalGeneration` | `internlm/Intern-S1-Pro` |
| `intern-s2-mobius` | `InternS2MobiusForConditionalGeneration` | `internlm/Intern-S2-Mobius` |
| `llama-nemotron-vl-embed` | `LlamaNemotronVLForEmbedding` | `nvidia/llama-nemotron-embed-vl-1b-v2` |
| `llama-nemotron-vl-rerank` | `LlamaNemotronVLForSequenceClassification` | `nvidia/llama-nemotron-rerank-vl-1b-v2` |
| `mistral3` | `Mistral3ForConditionalGeneration` | `mistralai/Mistral-Small-3.2-24B-Instruct-2506`, etc. |
| `qwen2-vl` | `Qwen2VLForConditionalGeneration` | `Qwen/Qwen2-VL-2B-Instruct`, etc. |
| `qwen2.5-vl` | `Qwen2_5_VLForConditionalGeneration` | `Qwen/Qwen2.5-VL-3B-Instruct`, etc. |
| `qwen3-vl` | `Qwen3VLForConditionalGeneration` | `Qwen/Qwen3-VL-4B-Instruct`, etc. |
| `qwen3-vl-moe` | `Qwen3VLMoeForConditionalGeneration` | `Qwen/Qwen3-VL-30B-A3B-Instruct`, etc. |
| `qwen3.5` | `Qwen3_5ForConditionalGeneration` | `Qwen/Qwen3.5-0.8B`, etc. |
| `qwen3.5-moe` | `Qwen3_5MoeForConditionalGeneration` | `Qwen/Qwen3.5-35B-A3B`, etc. |
| `qwen4-exp` | `Qwen4ExpForConditionalGeneration` | `Qwen/Qwen3.8-Flash-Next` |
#### Key Properties and Gains
+9 -3
View File
@@ -127,7 +127,7 @@ Batch invariance has been tested and verified on the following models:
- **Qwen3-VL (Vision-Language)**: `Qwen/Qwen3-VL-2B-Instruct`, `Qwen/Qwen3-VL-4B-Instruct` (single image and video inputs)
- **Qwen3 (MoE)**: `Qwen/Qwen3-30B-A3B`, `Qwen/Qwen3-Next-80B-A3B-Instruct`, `Qwen/Qwen3-30B-A3B-Thinking-2507-FP8`
- **Qwen2.5**: `Qwen/Qwen2.5-0.5B-Instruct`, `Qwen/Qwen2.5-1.5B-Instruct`, `Qwen/Qwen2.5-3B-Instruct`, `Qwen/Qwen2.5-7B-Instruct`, `Qwen/Qwen2.5-14B-Instruct`, `Qwen/Qwen2.5-32B-Instruct`
- **Llama 3**: Llama3.1 and 3.2 series, `meta-llama/Llama-3.2-3B-Instruct` for example
- **Llama 3**: Llama3.1 and 3.2 series, `meta-llama/Llama-3.2-1B-Instruct`, `meta-llama/Llama-3.2-3B-Instruct` for example
- **GPT-OSS**: `openai/gpt-oss-20b`, `openai/gpt-oss-120b`
- **Mistral**: `mistralai/Mistral-7B-v0.3`
- **Phi series**: `microsoft/Phi-3.5-mini-instruct`
@@ -135,6 +135,8 @@ Batch invariance has been tested and verified on the following models:
- **Granite 3.1 (Dense)**: `ibm-granite/granite-3.1-2b-instruct`, `ibm-granite/granite-3.1-8b-instruct`
- **EXAONE 4.0 series**: `LGAI-EXAONE/EXAONE-4.0-1.2B`, `LGAI-EXAONE/EXAONE-4.0.1-32B`, `LGAI-EXAONE/EXAONE-4.0-32B`
- **OLMo 2**: `allenai/OLMo-2-0425-1B-Instruct`
- **SmolLM2**: `HuggingFaceTB/SmolLM2-1.7B-Instruct`
- **PLaMo3**: `pfnet/plamo-3-nict-2b-base`
Other models may also work, but these have been explicitly validated. If you encounter issues with a specific model, please report them on the [GitHub issue tracker](https://github.com/vllm-project/vllm/issues/new/choose).
@@ -144,11 +146,15 @@ When batch invariance is enabled, vLLM:
1. Uses deterministic kernel implementations for attention and other operations
2. Ensures consistent numerical behavior across different batch sizes
3. Disables certain optimizations that may introduce non-determinism (such as custom all-reduce operations in tensor parallel mode, and sequence parallelism / async TP, whose reduce-scatter path is not batch-invariant)
4. On CUDA devices with tuned matmul table entries for the model's bf16 unquantized forward linear layers (Ada, Hopper, Blackwell),
3. Disables certain optimizations that may introduce non-determinism (such as sequence parallelism / async TP, whose reduce-scatter path is not batch-invariant)
4. Under tensor parallelism, keeps custom all-reduce on with a fixed reduction order (the 1-stage kernel is pinned, and large inputs are reduced in fixed-size chunks), and disables FlashInfer, AITER and QuickReduce all-reduce
5. On CUDA devices with tuned matmul table entries for the model's bf16 unquantized forward linear layers (Ada, Hopper, Blackwell),
runs without `torch.compile` using breakable CUDA graphs so tile configs follow the runtime batch size; set `VLLM_USE_BREAKABLE_CUDAGRAPH=0` to opt out
(not applied when sequence parallelism / async TP are enabled, since those are `torch.compile` passes).
!!! warning
Batch invariance under tensor parallelism is not yet supported for all-reduces whose size isn't a multiple of 16 bytes, for example a hidden size that isn't a multiple of 8 in fp16/bf16. Such tensors can switch between NCCL and custom all-reduce depending on batch size. All validated models meet this requirement.
!!! note
Enabling batch invariance may impact performance compared to the default non-deterministic mode. This trade-off is intentional to guarantee reproducibility.
+4 -3
View File
@@ -45,9 +45,10 @@ The following scripts demonstrate the workflow with **ExampleConnector**:
### ECMooncakeConnector
**ECMooncakeConnector** transfers encoder outputs using the Mooncake TransferEngine.
See the [Mooncake integration example](../../tests/v1/ec_connector/integration/run_epd_mooncake_ec_full_pipeline.sh)
for a complete 1 Encoder instance + 1 PD instance setup, including the producer and
consumer `--ec-transfer-config` settings and proxy configuration.
See the [ECMooncakeConnector Usage Guide](mooncake_ec_connector_usage.md) for
producer/consumer `--ec-transfer-config`, proxy `--ec-consumer-zmq-addrs`, and
transport options. The [Mooncake integration example](../../tests/v1/ec_connector/integration/run_epd_mooncake_ec_full_pipeline.sh)
is a complete 1 Encoder instance + 1 PD instance setup.
With vLLM and Mooncake installed, run the example from the repository root:
+1 -1
View File
@@ -27,7 +27,7 @@ Now supports 9 types of connectors:
--kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_both", "kv_buffer_device":"cuda", "kv_connector_extra_config":{"backends":["UCX", "GDS"]}}'
```
- **MooncakeConnector**: refer to [examples/disaggregated/mooncake_connector/run_mooncake_connector.sh](../../examples/disaggregated/mooncake_connector/run_mooncake_connector.sh) for the example usage of MooncakeConnector disaggregated prefilling. For detailed usage guide, see [MooncakeConnector Usage Guide](mooncake_connector_usage.md).
- **MooncakeConnector**: refer to [examples/disaggregated/mooncake_connector/run_mooncake_connector.sh](../../examples/disaggregated/mooncake_connector/run_mooncake_connector.sh) for the example usage of MooncakeConnector disaggregated prefilling. For detailed usage guide, see [MooncakeConnector Usage Guide](mooncake_connector_usage.md). For multimodal encoder-cache transfer, see [ECMooncakeConnector Usage Guide](mooncake_ec_connector_usage.md).
- **MoRIIOConnector** (ROCm only): see [MoRI-IO Usage Guide](moriio_connector_usage.md) for example usage and detailed documentation.
- **MultiConnector**: take advantage of the kv_connector_extra_config: dict[str, Any] already present in KVTransferConfig to stash all the connectors we want in an ordered list of kwargs.such as:
+4 -4
View File
@@ -42,10 +42,10 @@ Snapshots currently require:
and CRIU still reopens it by path, so replacing that file permanently
invalidates the artifact without an early error.
The official `vllm/vllm-openai` Linux x86-64 image includes the snapshot
runtime. It still requires a compatible host driver, kernel, and privileges.
Arm64 images omit it. Source installs must set `CRIU_CUDA_PLUGIN_DIR` to the
directory containing `cuda_plugin.so`.
The official CUDA 13 `vllm/vllm-openai` Linux x86-64 images include the
snapshot runtime. CUDA 12.x and Arm64 images omit it. A compatible host
driver, kernel, and privileges are still required. Source installs must set
`CRIU_CUDA_PLUGIN_DIR` to the directory containing `cuda_plugin.so`.
Run snapshot commands with `docker exec` inside a long-lived container. Restore
hands the API server off as a detached process, so a one-shot container would
@@ -38,6 +38,9 @@ python examples/disaggregated/mooncake_connector/mooncake_connector_proxy.py --p
Now you can send requests to the proxy server through port 8000.
For multimodal Encode-Prefill-Decode (encoder embeddings rather than KV
cache), see [ECMooncakeConnector Usage Guide](mooncake_ec_connector_usage.md).
## Environment Variables
- `VLLM_MOONCAKE_BOOTSTRAP_PORT`: Port for Mooncake bootstrap server
@@ -76,6 +79,7 @@ Now you can send requests to the proxy server through port 8000.
- **num_workers**: Size of thread pool for one prefiller worker to transfer KV caches by mooncake. (default 10)
- **mooncake_protocol**: Mooncake connector protocol. (default "rdma")
- **device_name**: Comma-separated whitelist of RDMA devices (e.g. `"mlx5_0,mlx5_1"`) to restrict topology discovery to. Empty discovers every device. Useful on hosts exposing a mix of InfiniBand and RoCE ports, where both peers must settle on the same link layer.
- **bootstrap_server_address**: Optional `"host:port"` of an already running Mooncake bootstrap server. All prefiller ranks register with this endpoint instead of launching an embedded server. The caller owns the server lifetime and must configure the proxy to use the same endpoint. The example proxy assumes the bootstrap server is on the prefiller API host; a different host requires a proxy that supplies that address in `remote_bootstrap_addr`. Omit this option to use the embedded server and `VLLM_MOONCAKE_BOOTSTRAP_PORT`.
## Example Scripts/Code
@@ -0,0 +1,158 @@
# ECMooncakeConnector Usage Guide
`ECMooncakeConnector` moves multimodal **encoder embeddings** (encoder cache)
between a disaggregated encoder instance and a language-model instance over
[Mooncake Transfer Engine](https://github.com/kvcache-ai/Mooncake). Use it
when the vision encoder should scale independently of prefill and decode.
For KV-cache transfer between prefiller and decoder, see
[MooncakeConnector Usage Guide](mooncake_connector_usage.md). That path uses
`MooncakeConnector`, not `ECMooncakeConnector`.
Added in [PR #41567](https://github.com/vllm-project/vllm/pull/41567).
Overview of disaggregated encoding:
[Disaggregated Encoder](disagg_encoder.md).
!!! important
Static proxy flags (`--ec-consumer-zmq-addrs`) support Mooncake EC only in
**1E + 1PD** mode (`--prefill-servers-urls disable`). For 1E + 1P + 1D,
register `ec_zmq_addrs` on the **prefill** consumer through the proxy's
dynamic registration API instead of the static CLI flag. See
`examples/disaggregated/disaggregated_encoder/README.md`.
## Prerequisites
Install Mooncake Transfer Engine:
```bash
# CUDA 13 (vLLM default)
uv pip install mooncake-transfer-engine-cuda13
# CUDA 12
uv pip install mooncake-transfer-engine
```
The two wheels are the same release built against different CUDA majors.
Importing the wrong one fails with
`libcudart.so.<major>: cannot open shared object file`.
## 1E + 1PD
One encoder producer and one combined prefill/decode consumer. The consumer
listens on `ec_ip` + `ec_port` (ZMQ control). The proxy must advertise that
address with `--ec-consumer-zmq-addrs`.
### Encoder (producer)
The producer must run with `tensor_parallel_size=1`,
`pipeline_parallel_size=1`, and `data_parallel_size=1`.
```bash
CUDA_VISIBLE_DEVICES=0 vllm serve Qwen/Qwen2.5-VL-3B-Instruct \
--port 19534 \
--gpu-memory-utilization 0.35 \
--enable-request-id-headers \
--no-enable-prefix-caching \
--max-num-batched-tokens 16384 \
--allowed-local-media-path /path/to/media \
--ec-transfer-config '{
"ec_connector": "ECMooncakeConnector",
"ec_role": "ec_producer",
"ec_connector_extra_config": {
"mooncake_protocol": "rdma"
}
}'
```
### Prefill+Decode (consumer)
```bash
CUDA_VISIBLE_DEVICES=1 vllm serve Qwen/Qwen2.5-VL-3B-Instruct \
--port 19537 \
--gpu-memory-utilization 0.75 \
--enable-mm-embeds \
--enable-request-id-headers \
--no-enable-prefix-caching \
--allowed-local-media-path /path/to/media \
--ec-transfer-config '{
"ec_connector": "ECMooncakeConnector",
"ec_role": "ec_consumer",
"ec_ip": "127.0.0.1",
"ec_port": 19019,
"ec_connector_extra_config": {
"mooncake_protocol": "rdma"
}
}'
```
`--enable-mm-embeds` lets the PD instance inject transferred embeddings
instead of re-running the vision encoder.
### Proxy
```bash
python examples/disaggregated/disaggregated_encoder/disagg_epd_proxy.py \
--host 0.0.0.0 \
--port 10002 \
--encode-servers-urls http://127.0.0.1:19534 \
--prefill-servers-urls disable \
--decode-servers-urls http://127.0.0.1:19537 \
--ec-consumer-zmq-addrs tcp://127.0.0.1:19019
```
`--ec-consumer-zmq-addrs` must align with `--decode-servers-urls` (the
combined PD instances). For data-parallel consumers, list each replica
consecutively and set `--ec-consumer-dp-size`.
### Correctness script
```bash
# TCP (no RDMA device required; CI default)
MOONCAKE_EC_PROTOCOL=tcp \
bash tests/v1/ec_connector/integration/run_epd_mooncake_ec_full_pipeline.sh
# RDMA
MOONCAKE_EC_PROTOCOL=rdma \
bash tests/v1/ec_connector/integration/run_epd_mooncake_ec_full_pipeline.sh
```
The TCP path sets `MC_FORCE_TCP=1` so Transfer Engine does not auto-select
RDMA when an HCA is present.
## Configuration
Producer and consumer must use the same `mooncake_protocol`.
| Setting | Where | Default | Notes |
| --------- | ------- | --------- | ------- |
| `mooncake_protocol` | `ec_connector_extra_config` | `rdma` | Set `tcp` when no verbs device is available. With TCP, also export `MC_FORCE_TCP=1`. |
| `ec_ip` / `ec_port` | consumer `ECTransferConfig` | `127.0.0.1` / `14579` | First ZMQ control port. Tensor-parallel rank `r` listens on `ec_port + dp_index * tp_size + r`. |
| `ec_buffer_size` | `ECTransferConfig` | `1e9` | Registered receive arena on the consumer, in bytes. |
| `control_timeout_s` | `ec_connector_extra_config` | `30` | Reservation RPC timeout. |
| `push_wait_timeout_s` | `ec_connector_extra_config` | `60` | How long the consumer waits for a matching encoder push. |
| `WITH_NVIDIA_PEERMEM` | process environment | `1` | Read by Mooncake. Set `0` on hosts without `nvidia-peermem` (DMA-BUF path). |
| `--ec-consumer-zmq-addrs` | proxy | empty | `tcp://<ec_ip>:<ec_port>` for each PD replica. Required for Mooncake EC. |
## 1E + 1P + 1D
The filesystem example is `examples/disaggregated/disaggregated_encoder/disagg_1e1p1d_example.sh`
(`ECExampleConnector` for encoder cache, `NixlConnector` for KV). P→D KV can
use `MooncakeConnector` instead; see
[MooncakeConnector Usage Guide](mooncake_connector_usage.md).
Do **not** pass `--ec-consumer-zmq-addrs` together with independent prefill
URLs: the static proxy flag is E+PD only. For Mooncake EC with a dedicated
prefill, use dynamic registration and set `ec_zmq_addrs` on the prefill
consumer as described in
`examples/disaggregated/disaggregated_encoder/README.md`.
## Troubleshooting
| Symptom | Likely cause |
| --------- | ---------------- |
| `Install mooncake-transfer-engine` / `libcudart.so.*` | Wrong or missing Mooncake wheel for this CUDA major. |
| `Failed to register memory ... Bad address [14]` | GPUDirect registration failed. Set `WITH_NVIDIA_PEERMEM=0`. |
| Proxy startup error about E+PD only | `--ec-consumer-zmq-addrs` used with a real `--prefill-servers-urls`. |
| Consumer waits then errors; encoder never finishes the push | `--ec-consumer-zmq-addrs` does not match consumer `ec_ip`/`ec_port`, or producer/consumer protocols differ. |
| TCP transfers stall on a host that also has RDMA | Export `MC_FORCE_TCP=1`. |
| Producer rejected at startup | Encoder used tensor, pipeline, or data parallelism. Keep the producer unsharded. |
+10 -9
View File
@@ -1210,7 +1210,7 @@ vLLM decodes audio bytes into waveforms using a selectable decoding backend:
| Backend | Description |
| --- | --- |
| `auto` (default) | soundfile, falling back to torchcodec, then PyAV |
| `auto` (default) | torchcodec, falling back to soundfile, then PyAV |
| `soundfile` | libsndfile only, no fallback |
| `pyav` | PyAV (FFmpeg) only, no fallback |
| `torchcodec` | TorchCodec (PyTorch-native) only, no fallback |
@@ -1226,17 +1226,18 @@ vllm serve mistralai/Voxtral-Mini-3B-2507 \
`pyav` drives FFmpeg through a per-frame Python generator, so under
concurrency the Python/C crossings contend on the GIL. `torchcodec`
decodes each stream in a single call that releases the GIL for its whole
duration. Select it explicitly for concurrent decoding workloads.
`auto` prefers soundfile for supported formats to preserve their existing
decoding behavior, including encoder padding. Audio extracted from formats
that soundfile cannot read, such as video containers, can use torchcodec
through the fallback chain.
duration, which is why `auto` prefers it when many requests decode audio
concurrently, such as when audio tracks are extracted from video.
vLLM normalizes codec padding on this path (e.g. it trims trailing Vorbis
padding that older system FFmpeg versions (< 5.0) fail to trim), so the
decoded waveform matches the soundfile reference length.
!!! note
`torchcodec` ships as a requirement on CUDA, CPU and XPU builds. On other
platforms (e.g. ROCm, TPU), install it manually to enable that backend.
torchcodec also links against a system FFmpeg installation. If the package
or FFmpeg is unavailable, `auto` uses the soundfile → PyAV chain.
platforms (e.g. ROCm, TPU) `auto` falls back to the soundfile → PyAV
chain unless you install it manually. torchcodec also links against a
system FFmpeg installation; if FFmpeg is missing, `auto` falls back the
same way.
### Embedding Inputs
+1
View File
@@ -454,6 +454,7 @@ exported when NixlConnector is active:
| `vllm:nixl_num_failed_transfers` | Counter | Cumulative count of failed NIXL KV-block transfers, including handshake and notification (`send_notif`) failures. These are grouped because all are sporadic lower-transport-layer events. |
| `vllm:nixl_num_failed_notifications` | Counter | Cumulative count of failed notifications (`send_notif`), retained for compatibility. These failures are also included in `vllm:nixl_num_failed_transfers`; do not sum the two counters. |
| `vllm:nixl_num_kv_expired_reqs` | Counter | Requests whose KV blocks expired on the prefiller before the decoder read them (tracked on the P instance). Kept separate from the failure counters above: KV expiry is an autoscaler/lease-tuning signal, not a transport issue. |
| `vllm:nixl_num_notifications_after_expiry` | Counter | Completion notifications that arrive for a request that is no longer tracked, usually because its KV lease expired first. Pull connector: counted on P, where it means the decoder may have read blocks that were already reused; each matches a "Potentially invalid KV blocks" error log. Push connector: counted on whichever instance tracks the request as neither a send nor a receive (P after the lease expired, or D after the receive completed or was abandoned); each matches an "Unrecognized request" error log. Counted per notification, not per request: each remote rank notifies separately, so with TP > 1 or heterogeneous TP one late request counts several times, and the value is not comparable to `vllm:nixl_num_kv_expired_reqs`. |
!!! tip
High `vllm:nixl_num_kv_expired_reqs` indicates that the prefiller's lease
+1
View File
@@ -29,6 +29,7 @@ vLLM currently supports the following reasoning models:
| [MiniMax-M2](https://huggingface.co/MiniMaxAI/MiniMax-M2) | `minimax_m2_append_think` | `json`, `regex` | ✅ |
| [Qwen3 series](https://huggingface.co/collections/Qwen/qwen3-67dd247413f0e2e4f653967f) | `qwen3` | `json`, `regex` | ✅ |
| [QwQ-32B](https://huggingface.co/Qwen/QwQ-32B) | `deepseek_r1` | `json`, `regex` | ✅ |
| [Step-3.5 / Step-3.7 Flash](https://huggingface.co/stepfun-ai/Step-3.5-Flash) | `step3p5` | `json`, `regex` | ✅ |
!!! note
IBM Granite 3.2 and DeepSeek-V3.1 reasoning is disabled by default; to enable it, you must also pass `thinking=True` in your `chat_template_kwargs`.
+46
View File
@@ -171,6 +171,29 @@ from HuggingFace; and you can find an example of this in a `tokenizer_config.jso
If your favorite tool-calling model is not supported, please feel free to contribute a parser & tool use chat template!
### Checkpoint Response Templates (`hf`)
Checkpoints can define a `response_template` in `tokenizer_config.json` to describe their reasoning, content, and tool-call wire format. Select the
`hf` parser to parse output from this metadata instead of a model-specific parser. Like `--tokenizer-mode hf`, it relies on the checkpoint's
Hugging Face metadata rather than model-specific code:
```bash
vllm serve <model> \
--enable-auto-tool-choice \
--tool-call-parser hf \
--reasoning-parser hf
```
Either parser can be selected on its own. Startup fails if the tokenizer has no `response_template`, or if it lacks the `thinking` or `tool_calls`
field the selected parser needs. The `hf` parser cannot be combined with other reasoning or tool call parsers.
When streaming, each tool call is emitted whole once its region parses. A call cut off before its closer, for example by a closer that is also a
stop token, is parsed from the text generated so far; a call that does not parse is dropped.
Response templates currently provide parsing, not format-specific constrained decoding. The parser therefore rejects requests for strict tools,
required or named tool choice, or `parallel_tool_calls=false`. A custom server or request chat template is allowed, but the parser still expects the
checkpoint's output format.
!!! note
With `tool_choice="auto"`, structural-tag constraints require both `VLLM_ENFORCE_STRICT_TOOL_CALLING=true` (the default) and at least one tool with `strict: true`, or a server-side floor set via `--tool-strict-level`. When these conditions are met and the selected parser supports structural tags, vLLM constrains the tool-call envelope and pins the argument schema of each tool that sets `strict: true` (or of every tool under `--tool-strict-level parameter`). Otherwise, vLLM extracts tool calls from raw text, so arguments may occasionally be malformed or violate the function's parameter schema.
@@ -458,6 +481,29 @@ Supported models:
Flags: `--tool-call-parser qwen3_xml`
### MiMo-V2.6 Models (`mimo`)
Supported models:
* `XiaomiMiMo/MiMo-V2.6-Pro-RL`
* `XiaomiMiMo/MiMo-V2.6-Flash-RL`
Use the checkpoint's chat template with `--tool-call-parser mimo --reasoning-parser mimo --enable-auto-tool-choice`.
For schema-constrained tool arguments, set `strict: true` inside each tool's `function`, or use `--tool-strict-level parameter`.
The parser supports `auto`, `required`, named tools, and `parallel_tool_calls=False`, preserving leading/trailing whitespace in string arguments.
### Step-3.5 and Step-3.7 Models (`step3p5`)
Supported models:
* `stepfun-ai/Step-3.5-Flash`
* `stepfun-ai/Step-3.5-Flash-FP8`
* `stepfun-ai/Step-3.7-Flash`
Flags: `--tool-call-parser step3p5 --reasoning-parser step3p5 --enable-auto-tool-choice`
The parser supports `auto`, `required`, and named tools.
### Olmo 3 Models (`olmo3`)
Olmo 3 models output tool calls in a format that is very similar to the one expected by the `pythonic` parser (see below), with a few differences. Each tool call is a pythonic string, but the parallel tool calls are newline-delimited, and the calls are wrapped within XML tags as `<function_calls>..</function_calls>`. In addition, the parser also allows JSON boolean and null literals (`true`, `false`, and `null`) in addition to the pythonic ones (`True`, `False`, and `None`).
+14 -15
View File
@@ -77,30 +77,17 @@ watermarked. The watermark signal is diluted in proportion to the share of
output tokens supplied by accepted drafts; rejected drafts do not dilute it
because their recovery tokens are watermarked.
Speculative-decoding token paths do not currently support generation-side
context deduplication. The configured `deduplicate_contexts` policy is not
applied to accepted drafts, rejection-recovery tokens, or bonus tokens.
For `dual_key_gumbel`, `alpha` has no effect under speculative decoding. The
speculative protocol selects the key for each token instead.
## Algorithms
### Gumbel-max
Gumbel-max derives a deterministic pseudorandom value from the key, prior token
context, and every candidate token, then uses the resulting Gumbel noise for
categorical sampling. See
[Aaronson's original presentation](https://simons.berkeley.edu/sites/default/files/2024-10/LLM24-2%20Slides%20-%20Scott%20Aaronson.pdf).
Gumbel-max requires stochastic sampling. Greedy requests (`temperature=0`)
bypass watermarking and emit a warning once per worker.
## Context deduplication
When a token context is repeated, generation can use ordinary sampling for that
occurrence. Reusing the repeated context leads to a bias over the sequence, as
certain token choices would be correlated. Ordinary sampling at these positions
allows for single-sequence non-distortion (see section G.3 of the
[SynthID-Text supplementary materials](https://media.springernature.com/original/springer-static/esm/art%3A10.1038%2Fs41586-024-08025-4/MediaObjects/41586_2024_8025_MOESM1_ESM.pdf)).
This also applies to drafter watermarking during speculative decoding.
The detector independently deduplicates contexts so repeated keyed random
vectors are not treated as independent evidence, meaning that context
deduplication at generation time does not reduce the watermarking signal, unless
@@ -143,6 +130,18 @@ vllm serve MODEL \
'{"algorithm":"gumbel","key":42,"deduplicate_contexts":"all"}'
```
## Algorithms
### Gumbel-max
Gumbel-max derives a deterministic pseudorandom value from the key, prior token
context, and every candidate token, then uses the resulting Gumbel noise for
categorical sampling. See
[Aaronson's original presentation](https://simons.berkeley.edu/sites/default/files/2024-10/LLM24-2%20Slides%20-%20Scott%20Aaronson.pdf).
Gumbel-max requires stochastic sampling. Greedy requests (`temperature=0`)
bypass watermarking and emit a warning once per worker.
### Dual-key Gumbel-max
Dual-key Gumbel-max derives independent keys A and B from one configured master
+2 -2
View File
@@ -312,8 +312,8 @@ vLLM CPU supports data parallel (DP), tensor parallel (TP) and pipeline parallel
### Which quantization configs does vLLM CPU support?
- vLLM CPU supports quantizations:
- AWQ (x86, s390x)
- GPTQ (x86, s390x)
- AWQ (x86, s390x, Power)
- GPTQ (x86, s390x, Power)
- compressed-tensor INT8 W8A8 (x86, s390x only)
### Why do I see `get_mempolicy: Operation not permitted` when running in Docker?
+5 -2
View File
@@ -22,6 +22,7 @@ Sorted alphabetically by GitHub handle:
- [@DarkLight1337](https://github.com/DarkLight1337): Multimodality, API server
- [@dllehr-amd](https://github.com/dllehr-amd): AMD integration
- [@esmeetu](https://github.com/esmeetu): developer marketing, community
- [@gau-nernst](https://github.com/gau-nernst): Kernels and performance
- [@gshtras](https://github.com/gshtras): ROCm / AMD GPU integration
- [@Harry-Chen](https://github.com/Harry-Chen): Build, CI, CUDA architecture
- [@heheda12345](https://github.com/heheda12345): Hybrid memory allocator
@@ -54,6 +55,7 @@ Sorted alphabetically by GitHub handle:
- [@tomeras91](https://github.com/tomeras91): MoE, Mamba, Ray
- [@vadiklyutiy](https://github.com/vadiklyutiy): CI, performance, kernels
- [@WoosukKwon](https://github.com/WoosukKwon): Project lead, engine core
- [@wzhao18](https://github.com/wzhao18): Kernels and performance, KV offloading
- [@xinli-sw](https://github.com/xinli-sw): Quantization and model support
- [@xuechendi](https://github.com/xuechendi): Intel CPU/XPU integration, KV connector
- [@yaochengji](https://github.com/yaochengji): TPU integration
@@ -173,9 +175,10 @@ If you have PRs touching the area, please feel free to ping the area owner for r
### External Kernels Integration
- FlashAttention: @LucasWilkinson, @MatthewBonanni
- FlashInfer: @LucasWilkinson, @mgoin, @WoosukKwon, @MatthewBonanni
- FlashInfer: @LucasWilkinson, @mgoin, @WoosukKwon, @MatthewBonanni, @wzhao18
- Blackwell Kernels: @mgoin, @yewentao256
- DeepEP/DeepGEMM: @mgoin, @yewentao256
- FlashKDA: @gau-nernst
### Integrations
@@ -186,7 +189,7 @@ If you have PRs touching the area, please feel free to ping the area owner for r
### Hardware
- Plugin Interface: @youkaichao, @Yikun
- NVIDIA GPU: @pavanimajety
- NVIDIA GPU: @pavanimajety, @wzhao18
- AMD GPU: @gshtras, @tjtanaa, @hongxiayang, @shen-shanshan, @AndreasKaratzas
- Intel CPU/GPU: @jikunshang, @bigPYJ1151, @xuechendi
- Google TPU: @yaochengji
+34
View File
@@ -0,0 +1,34 @@
<!-- Material nav.html (https://github.com/squidfunk/mkdocs-material/blob/9.7.7/material/templates/partials/nav.html) with navigation.prune applied to the API reference only -->
{% import "partials/nav-item.html" as item with context %}
{% set features = features + ["navigation.prune"] %}
{% import "partials/nav-item.html" as pruned_item with context %}
{% set class = "md-nav md-nav--primary" %}
{% if "navigation.tabs" in features %}
{% set class = class ~ " md-nav--lifted" %}
{% endif %}
{% if "toc.integrate" in features %}
{% set class = class ~ " md-nav--integrated" %}
{% endif %}
<nav class="{{ class }}" aria-label="{{ lang.t('nav') }}" data-md-level="0">
<label class="md-nav__title" for="__drawer">
<a href="{{ config.extra.homepage | d(nav.homepage.url, true) | url }}" title="{{ config.site_name | e }}" class="md-nav__button md-logo" aria-label="{{ config.site_name }}" data-md-component="logo">
{% include "partials/logo.html" %}
</a>
{{ config.site_name }}
</label>
{% if config.repo_url %}
<div class="md-nav__source">
{% include "partials/source.html" %}
</div>
{% endif %}
<ul class="md-nav__list" data-md-scrollfix>
{% for nav_item in nav.items %}
{% set path = "__nav_" ~ loop.index %}
{% if nav_item.children and (nav_item.children | first).url == "api/" %}
{{ pruned_item.render(nav_item, path, 1) }}
{% else %}
{{ item.render(nav_item, path, 1) }}
{% endif %}
{% endfor %}
</ul>
</nav>
+4 -4
View File
@@ -374,7 +374,6 @@ th {
| ------------ | ------ | ----------------- | -------------------- | ------------------------- |
| `AfmoeForCausalLM` | Afmoe | TBA | ✅︎ | ✅︎ |
| `ApertusForCausalLM` | Apertus | `swiss-ai/Apertus-8B-2509`, `swiss-ai/Apertus-70B-Instruct-2509`, etc. | ✅︎ | ✅︎ |
| `ArceeForCausalLM` | Arcee (AFM) | `arcee-ai/AFM-4.5B-Base`, etc. | ✅︎ | ✅︎ |
| `AXK1ForCausalLM` | A.X-K1 | `skt/A.X-K1`, etc. | | ✅︎ |
| `BailingMoeForCausalLM` | Ling | `inclusionAI/Ling-lite-1.5`, `inclusionAI/Ling-plus`, etc. | ✅︎ | ✅︎ |
| `BailingMoeV2ForCausalLM` | Ling | `inclusionAI/Ling-mini-2.0`, etc. | ✅︎ | ✅︎ |
@@ -384,7 +383,6 @@ th {
| `ChatGLMModel`, `ChatGLMForConditionalGeneration` | ChatGLM | `zai-org/chatglm2-6b`, `zai-org/chatglm3-6b`, `thu-coai/ShieldLM-6B-chatglm3`, etc. | ✅︎ | ✅︎ |
| `CohereForCausalLM`, `Cohere2ForCausalLM` | Command-R, Command-A | `CohereLabs/c4ai-command-r-v01`, `CohereLabs/c4ai-command-r7b-12-2024`, `CohereLabs/c4ai-command-a-03-2025`, `CohereLabs/command-a-reasoning-08-2025`, etc. | ✅︎ | ✅︎ |
| `Cohere2MoeForCausalLM` | North-Mini-Code | `CohereLabs/North-Mini-Code`, etc. | ✅︎ | ✅︎ |
| `CwmForCausalLM` | CWM | `facebook/cwm`, etc. | ✅︎ | ✅︎ |
| `DbrxForCausalLM` | DBRX | `databricks/dbrx-base`, `databricks/dbrx-instruct`, etc. | | ✅︎ |
| `DeciLMForCausalLM` | DeciLM | `nvidia/Llama-3_3-Nemotron-Super-49B-v1`, etc. | ✅︎ | ✅︎ |
| `DeepseekForCausalLM` | DeepSeek | `deepseek-ai/deepseek-llm-67b-base`, `deepseek-ai/deepseek-llm-7b-chat`, etc. | ✅︎ | ✅︎ |
@@ -406,7 +404,6 @@ th {
| `Gemma3ForCausalLM` | Gemma 3 | `google/gemma-3-1b-it`, etc. | ✅︎ | ✅︎ |
| `Gemma3nForCausalLM` | Gemma 3n | `google/gemma-3n-E2B-it`, `google/gemma-3n-E4B-it`, etc. | | |
| `Gemma4ForCausalLM` | Gemma 4 | `google/gemma-4-E2B-it`, etc. | ✅︎ | ✅︎ |
| `GlmForCausalLM` | GLM-4 | `zai-org/glm-4-9b-chat-hf`, etc. | ✅︎ | ✅︎ |
| `Glm4ForCausalLM` | GLM-4-0414 | `zai-org/GLM-4-32B-0414`, etc. | ✅︎ | ✅︎ |
| `Glm4MoeForCausalLM` | GLM-4.5, GLM-4.6, GLM-4.7 | `zai-org/GLM-4.5`, etc. | ✅︎ | ✅︎ |
| `Glm4MoeLiteForCausalLM` | GLM-4.7-Flash | `zai-org/GLM-4.7-Flash`, etc. | ✅︎ | ✅︎ |
@@ -439,7 +436,6 @@ th {
| `LongcatFlashForCausalLM` | LongCat-Flash | `meituan-longcat/LongCat-Flash-Chat`, `meituan-longcat/LongCat-Flash-Chat-FP8` | ✅︎ | ✅︎ |
| `MambaForCausalLM` | Mamba | `state-spaces/mamba-130m-hf`, `state-spaces/mamba-790m-hf`, `state-spaces/mamba-2.8b-hf`, etc. | | ✅︎ |
| `Mamba2ForCausalLM` | Mamba2 | `mistralai/Mamba-Codestral-7B-v0.1`, etc. | | ✅︎ |
| `MellumForCausalLM` | Mellum 2 | `JetBrains/Mellum2-12B-A2.5B-Base`, etc. | | ✅︎ |
| `MiMoForCausalLM` | MiMo | `XiaomiMiMo/MiMo-7B-RL`, etc. | ✅︎ | ✅︎ |
| `MiMoV2FlashForCausalLM` | MiMoV2Flash | `XiaomiMiMo/MiMo-V2-Flash`, etc. | | ✅︎ |
| `MiMoV2ForCausalLM` | MiMoV2Pro | `XiaomiMiMo/MiMo-V2.5-Pro`, etc. | | ✅︎ |
@@ -486,10 +482,14 @@ Some models are supported only via the [Transformers modeling backend](#transfor
| Architecture | Models | Example HF Models | [LoRA](../features/lora.md) | [PP](../serving/parallelism_scaling.md) |
| ------------ | ------ | ----------------- | -------------------- | ------------------------- |
| `ArceeForCausalLM` | Arcee (AFM) | `arcee-ai/AFM-4.5B-Base`, etc. | ✅︎ | ✅︎ |
| `CwmForCausalLM` | CWM | `facebook/cwm`, etc. | ✅︎ | ✅︎ |
| `FlexOlmoForCausalLM` | FlexOlmo | `allenai/FlexOlmo-7x7B-1T`, `allenai/FlexOlmo-7x7B-1T-RT`, etc. | | ✅︎ |
| `GlmForCausalLM` | GLM-4 | `zai-org/glm-4-9b-chat-hf`, etc. | ✅︎ | ✅︎ |
| `GPTBigCodeForCausalLM` | StarCoder, SantaCoder, WizardCoder | `bigcode/starcoder`, `bigcode/gpt_bigcode-santacoder`, `WizardLM/WizardCoder-15B-V1.0`, etc. | ✅︎ | |
| `HunYuanDenseV1ForCausalLM` | Hunyuan Dense | `tencent/Hunyuan-7B-Instruct` | ✅︎ | ✅︎ |
| `HunYuanMoEV1ForCausalLM` | Hunyuan-A13B | `tencent/Hunyuan-A13B-Instruct`, `tencent/Hunyuan-A13B-Pretrain`, `tencent/Hunyuan-A13B-Instruct-FP8`, etc. | ✅︎ | ✅︎ |
| `MellumForCausalLM` | Mellum 2 | `JetBrains/Mellum2-12B-A2.5B-Base`, etc. | | ✅︎ |
| `NanbeigeForCausalLM` | Nanbeige4.2 | `Nanbeige/Nanbeige4.2-3B`, etc. | ✅︎ | ✅︎ |
| `OlmoForCausalLM` | OLMo | `allenai/OLMo-1B-hf`, `allenai/OLMo-7B-hf`, etc. | ✅︎ | ✅︎ |
| `Olmo2ForCausalLM` | OLMo2 | `allenai/OLMo-2-0425-1B`, etc. | ✅︎ | ✅︎ |
+3 -9
View File
@@ -1,4 +1,3 @@
# shellcheck disable=SC2317 # code after the exit/exit-183 gate below is intentionally dead pending re-enablement (see line comment)
if [ "$READTHEDOCS_VERSION_TYPE" != "external" ]; then
echo "Not a PR build (version type=$READTHEDOCS_VERSION_TYPE); skipping pre-run-check gate."
exit 0
@@ -11,24 +10,19 @@ if [ -n "$GITHUB_TOKEN" ]; then
CURL_AUTH=(-H "Authorization: Bearer $GITHUB_TOKEN")
fi
# Docs builds are now manually enabled via the 'build-docs' label.
# The 'build-docs' label forces a docs build, bypassing the checks below.
echo "Checking for the 'build-docs' label on PR #${READTHEDOCS_VERSION_NAME}..."
LABELS=$(curl -sS "${CURL_AUTH[@]}" "https://api.github.com/repos/vllm-project/vllm/issues/${READTHEDOCS_VERSION_NAME}/labels" | python3 -c "import sys, json; print('\n'.join(l.get('name', '') for l in json.load(sys.stdin)))")
if printf '%s\n' "$LABELS" | grep -qx "build-docs"; then
echo "PR has the 'build-docs' label; continuing build."
echo "PR has the 'build-docs' label; forcing build."
exit 0
else
echo "PR does not have the 'build-docs' label; cancelling build."
# See https://docs.readthedocs.com/platform/latest/guides/build/skip-build.html for info on exit code
exit 183
fi
# Everything below this line is effectively disabled as a temporary measure.
echo "Checking for changes to docs-affecting files vs origin/main..."
DOCS_PATHS=(
docs/ # Actual docs content
examples/ # Examples are rendered in docs
vllm/ # API & CLI reference
# vllm/ # API & CLI reference (too broad, use 'build-docs' label)
requirements/test/cuda.txt # CLI reference (see docs/mkdocs/gen_files/generate_argparse.py)
mkdocs.yaml # Affects build process
.readthedocs.yaml # Affects build process
+19 -4
View File
@@ -23,6 +23,10 @@ endpoints.
The derender step needs more than the engine's `token_ids`. It also consumes the original `chat_request`/`completion_request` and `prompt_tokens` carried over from the render step (see [Request format](#request-format)) so the tool and reasoning parsers have the context they need. The chat streaming path additionally needs `prompt_token_ids` (see [Streaming cost](#streaming-cost)) when a parser is configured.
All derender endpoints also accept an optional `prompt_token_ids` (the `token_ids` of the `GenerateRequest` returned by `/render`). It's used to seed detokenization from the end of the prompt, the same way the engine does. Without it, the first output token is decoded as if it started a new sequence. On SentencePiece tokenizers (Llama-2, Phi-3) that drops its leading space compared to `/v1/chat/completions` and `/v1/completions`. If it's omitted, derender falls back to the generate response's `prompt_token_ids` (set when the `GenerateRequest` has `return_token_ids`), then to decoding without prompt context.
Seeding only reads the last few IDs (7 today), so a suffix of the prompt is enough and a long prompt doesn't need to travel back to derender. The exception is streaming chat with a tool or reasoning parser configured. There the parser also reads `prompt_token_ids` to decide whether the prompt left reasoning open and it can look back to any point in the prompt. In this case, send the full list (see [Streaming cost](#streaming-cost)).
## API Reference
- Chat Completions Derender API (`/v1/chat/completions/derender`)
@@ -96,9 +100,10 @@ The server keeps no state between calls. Everything the next call needs is in `s
- **`n > 1` is N streams.** Generate interleaves chunks for different choices and tags each with its `index`. Keep one `stream_state` per index and send each chunk with the state for its index. Each index gets its own `role` delta on its first chunk.
- **Send every chunk through, including the last ones.** The finish reason usually arrives with the final tokens. A chunk with a `finish_reason` and no tokens is accepted too and flushes any buffered tool call arguments. With `stream_options: {"include_usage": true}`, generate ends with a usage only chunk (`choices: []`). Send it through derender like any other chunk to get the usage chunk for the response. `usage.prompt_tokens` is the request's `prompt_tokens` if set, otherwise the generate chunk's.
- **Send the same context on every call.** `chat_request` and `prompt_token_ids` aren't kept between calls, so they go with every chunk including the usage chunk. Both are required when a tool or reasoning parser is configured. `prompt_token_ids` is the `token_ids` of the `GenerateRequest` returned by `/render`.
- **Send `prompt_token_ids` on the first chunk, even without a parser.** It seeds detokenization so the first token's leading space matches the coupled endpoint (see [Request format](#request-format)). Later chunks already carry that context in `stream_state`.
- **Don't forward `[DONE]`.** It marks the end of the generate stream and isn't a chunk.
Streaming chunks don't carry logprobs yet, so `logprobs` on a generate chunk are dropped. The non-streaming endpoints do resolve them, including `token_id:N` placeholders.
Streaming derender resolves `logprobs` on the plain detokenization path, including `token_id:N` placeholders, same as the non-streaming endpoints (see [Streaming state and logprobs](#streaming-state-and-logprobs)). The parser path doesn't resolve them yet and drops them.
## Streaming cost
@@ -110,11 +115,20 @@ When a tool or reasoning parser is configured, parser internal state (buffered m
- **Transport**: `output_token_ids` and `output_chunk_lens` round-trip in full in both directions on every call. `output_chunk_lens` has one entry per chunk, which is one per token without speculative decoding. This means O(n) bytes per chunk, O(n²) bytes over a full generation. Bounded by `max_model_len`. `prompt_token_ids` is sent in full on every call too and it isn't trimmed as `output_token_ids` grows. This means that for most of a stream it dominates the per chunk payload. A 100k token prompt with 1k tokens of output means `prompt_token_ids` is ~99% of the request body on every chunk.
- **Compute**: replay is O(n) `parse_delta` calls per chunk (O(n²) per generation). `parse_delta` itself is O(n) for parsers that re-scan accumulated text (e.g. Hermes tool-call JSON, DeepSeek-R1 reasoning). The per-generation cost is O(n³) character work, not O(n²). This is a deliberately minimal first implementation with no caching layer.
- The parser path also requires `prompt_token_ids` so `parse_delta` can settle whether the prompt left reasoning open or not. Since parser state can't be carried across calls, it re-scans the full prompt once per chunk.
- The parser path also requires `prompt_token_ids` so `parse_delta` can settle whether the prompt left reasoning open or not. Since parser state can't be carried across calls, it re-scans the full prompt once per chunk. Send the full list and not a suffix because the last reasoning marker can be anywhere in the prompt.
- Replay runs off the event loop on the renderer's executor (`renderer_num_workers`, default `1`). Size it for the expected number of concurrent parser configured streams.
`output_token_ids` and `prompt_token_ids` are both bounded by `max_model_len` but callers streaming long reasoning traces through a parser configured model should expect materially more state transport and CPU cost than the plain detokenization path.
## Streaming state and logprobs
Streaming derender is stateless on the server side: all mutable state lives in the client-carried `stream_state` (`DerenderStreamState`), passed back on every per-chunk call. Besides the bounded incremental detokenization window (`prev_tokens`, `prefix_offset`, `read_offset`), `role_sent`, and the parser path's replay fields (see [Streaming cost](#streaming-cost)), the state carries two fields for logprob handling:
- `logprob_context_token_ids`: the trailing sampled token IDs (at most 4) from previous chunks, used to seed byte-fallback (U+FFFD) correction so multi-byte characters whose tokens split across chunk boundaries still resolve to real strings
- `logprob_text_offset`: the cumulative emitted text length, so `text_offset` in completion streaming logprobs stays absolute across chunks instead of restarting at 0
When a streamed `GenerateResponseStreamChoice` carries `logprobs`, the `token_id:N` placeholders are resolved per chunk and the resolved logprobs are attached to the corresponding streamed choice — for chat as `ChatCompletionLogProbs`, for completions converted to the flat `CompletionLogProbs` lists. Chunks without `logprobs` produce choices with `logprobs: null`.
## Example
The example below drives the full `render → generate → derender` round trip for a chat request against a GPU less render server (`/render`, `/derender`) and a token-in / token-out engine (`/inference/v1/generate`).
@@ -144,7 +158,7 @@ with httpx.Client(timeout=60.0) as client:
generate_request = client.post(
f"{RENDER}/v1/chat/completions/render", json=chat_request
).json()
prompt_tokens = len(generate_request["token_ids"])
prompt_token_ids = generate_request["token_ids"]
# 2. Generate: token IDs -> token IDs (token-in / token-out engine)
generate_response = client.post(
@@ -157,7 +171,8 @@ with httpx.Client(timeout=60.0) as client:
json={
"model": MODEL,
"generate_response": generate_response,
"prompt_tokens": prompt_tokens,
"prompt_tokens": len(prompt_token_ids),
"prompt_token_ids": prompt_token_ids,
"chat_request": chat_request,
},
).json()
+1 -2
View File
@@ -51,8 +51,7 @@ and provision transport limits and memory accordingly.
For example, start a standard inference server with scale-out endpoints enabled:
```bash
VLLM_ENABLE_SCALE_OUT_ENDPOINTS=1 \
vllm serve meta-llama/Llama-3.1-8B-Instruct
vllm serve meta-llama/Llama-3.1-8B-Instruct --enable-scale-out
```
Send a Responses request and use `jq` to extract the `token_ids` field:
+1 -1
View File
@@ -16,7 +16,7 @@ The following open-source RL libraries use vLLM for fast rollouts (sorted alphab
- [Unsloth](https://github.com/unslothai/unsloth)
- [verl](https://github.com/volcengine/verl)
For weight synchronization between training and inference, see the [Weight Transfer](weight_transfer/README.md) documentation, which covers the pluggable backend system with [NCCL](weight_transfer/nccl.md) (multi-GPU) and [IPC](weight_transfer/ipc.md) (same-GPU) engines.
For weight synchronization between training and inference, see the [Weight Transfer](weight_transfer/README.md) documentation, which covers the pluggable backend system with [NCCL](weight_transfer/nccl.md) (multi-GPU) and [IPC](weight_transfer/ipc.md) (same-GPU) engines. To verify that an update landed on every rank, see the [Weight Checker](weight_checker.md).
For pipelining generation and training to improve GPU utilization and throughput, see the [Async Reinforcement Learning](async_rl.md) guide, which covers the pause/resume API for safely updating weights mid-flight.
+39 -3
View File
@@ -53,6 +53,8 @@ The mask is also available via the `/inference/v1/generate` HTTP endpoint:
}
```
With `stream=true`, each chunk carries the masks of the tokens it emits.
## Requirements
| Requirement | Reason |
@@ -63,9 +65,12 @@ The mask is also available via the `/inference/v1/generate` HTTP endpoint:
| `top_k > 0` | Bounds mask size; pure top-p can produce vocab-sized masks |
| Model Runner V2 | Required by the async D2H copy pipeline |
Speculative decoding is supported; each emitted token gets the support of the
target distribution at its position.
The engine rejects unsupported combinations at startup or request time:
- Speculative decoding
- Speculative decoding with adaptive verification
- Diffusion models
- Custom logits processors (engine-level `--logits-processors`)
@@ -104,10 +109,41 @@ log_prob = log_softmax(masked_logits)[sampled_token_id]
Both sides normalize over the same token set, so the importance ratio is
consistent.
## Score centering with top-k logprobs
[Score centering](https://arxiv.org/abs/2609.20807) needs the sampler's top-k
token IDs and logprobs plus the sampled token's logprob; the paper uses
`k=128` and does not need the exact sampling support. vLLM already returns
this through the ordinary `logprobs` parameter:
```bash
vllm serve <model> --logprobs-mode processed_logprobs --max-logprobs 128
```
```json
{"sampling_params": {"temperature": 1.0, "top_p": 0.9, "logprobs": 128}}
```
Each position then carries the sampled token's logprob and the top-128
`(token_id, logprob)` pairs, normalized over the post-truncation support:
```json
{"token": "token_id:11", "logprob": -0.757,
"top_logprobs": [{"token": "token_id:11", "logprob": -0.757},
{"token": "token_id:13", "logprob": -0.632},
{"token": "token_id:0", "logprob": -9999.0}, ...]}
```
- `--max-logprobs` defaults to 20; raise it to the requested `k`.
- Tokens outside the support are returned as `-9999.0` with arbitrary IDs;
drop them. The finite entries are the sampling mask when it has at most `k`
tokens.
- The sampled token is always the first entry and may repeat inside the top-k.
- `/inference/v1/generate` encodes IDs as `token_id:<id>`; the OpenAI
endpoints need `return_tokens_as_token_ids`.
## Limitations
- **Engine-level flag:** `--return-sampling-mask` globally disables the
FlashInfer fused sampler. All requests pay the cost of the PyTorch sampling
path, even if they don't need the mask.
- **No streaming support:** The mask is returned only in the final response,
not in intermediate streaming chunks.
+60
View File
@@ -0,0 +1,60 @@
# Weight Checker
The Weight Checker verifies that an RL weight update landed on every rank. It
hashes each model parameter with SHA-256, so a caller can save the original
digests, reset the weights, transfer them back, and confirm they match.
Enable it with `VLLM_SERVER_DEV_MODE=1`. All operations use
`POST /weight_checker`:
| Request | Response |
| --- | --- |
| `{"action": "checksum"}` | `{"checksums": {key: sha256_hex}}` |
| `{"action": "reset"}` | `{"status": "reset"}` |
| `{"action": "compare", "baseline": {key: sha256_hex}}` | `{"match": bool, "mismatches": [key]}` |
Keys have the form `dp{dp}:pp{pp}:pcp{pcp}:tp{tp}:{tensor_name}`, so each
shard is checked separately. All actions cover the target model's parameters,
which is what weight loading writes; `reset` zeroes them. The endpoint keeps no
state, so the caller holds the baseline. Invalid requests return HTTP 400.
## RL weight-update check
```bash
URL=http://localhost:8000
# 1. Stop serving, then save the original digests.
curl -X POST "$URL/pause?mode=abort"
curl -X POST $URL/weight_checker -H 'Content-Type: application/json' \
-d '{"action":"checksum"}' > baseline.json
# 2. Zero the weights.
curl -X POST $URL/weight_checker -H 'Content-Type: application/json' \
-d '{"action":"reset"}'
# 3. Transfer the original weights with start/update/finish_weight_update.
# 4. Compare against the baseline; expect {"match": true, "mismatches": []}.
curl -X POST $URL/weight_checker -H 'Content-Type: application/json' \
-d "{\"action\":\"compare\",\"baseline\":$(jq .checksums baseline.json)}"
curl -X POST $URL/resume
```
## Limitations
- Stay paused from `checksum` to `compare`: the weights are invalid between
`reset` and the transfer, and EPLB moves experts while serving.
- A request covers the engines managed by the API server it reaches. With
several API servers (for example `--data-parallel-external-lb`), send it to
each one.
- Buffers, draft models and LoRA adapters are not checked.
- CPU backends that repack linear weights hide them from the checker.
- compressed-tensors online transforms are not supported: their Hadamard
weights are normalized once per storage, so restored copies differ.
- MoE weights padded beyond the checkpoint shape (DeepEP hidden size, MXFP4)
keep uninitialized padding, so `compare` reports them as mismatches.
- The engine must be awake: sleep levels 1 and 2 release the weight memory.
- `--offload-backend prefetch` is not supported: parameters point at staging
buffers, not the offloaded weights.
- Hashing copies every weight to CPU, so keep it off latency-sensitive paths.
+1
View File
@@ -36,6 +36,7 @@ engine drives on your behalf:
| [sparse_nccl](nccl.md#sparse-nccl) | NCCL broadcast | Checkpoint-coordinate sparse weight patches |
| [sharded_rdt](sharded_rdt.md) | NIXL / Ray Direct Transport (pull-based) | Very large models where each worker needs only its own slice (MoE with expert parallelism) |
| [nccl_m2n](m2n.md) | NCCL M2N reshard | Trainer and inference use different sharding layouts |
| [ModelExpress](modelexpress.md) | ModelExpress version-based transfer | Install immutable published weight versions, including object-storage checkpoints and deltas |
## Quickstart
@@ -0,0 +1,156 @@
# ModelExpress Engine
The `modelexpress` weight transfer engine installs immutable weight versions
published through [ModelExpress](https://github.com/ai-dynamo/modelexpress).
Each update identifies the exact version to install, while ModelExpress handles
source selection, transfer, and installation.
## When to Use ModelExpress
- Your workflow manages immutable weight versions through the ModelExpress
Refit API.
- Inference workers need to pull a specific published version through
ModelExpress, using GPU peer transfer or object storage.
- You want to load full checkpoints or replay checkpoint deltas from S3.
## How It Works
1. A publisher creates a weight-version resource through the ModelExpress Refit
API, publishes its payload, and marks the version READY.
2. The orchestrator pauses generation and calls `start_weight_update` on the
inference engine.
3. The orchestrator calls `update_weights` with the target `version_id`.
Each worker's `ModelExpressGeneratorClient` resolves and stages that version,
then applies it to the live vLLM model.
4. After a successful update, `finish_weight_update` releases the staged handle.
The orchestrator can then resume inference.
Initialize the backend before starting an update session, and send exactly one
version per session. Invalid transitions must raise; an orchestrator must keep
inference paused after an update error and retry with a new session only after
cleanup. An error does not roll back weights that were already applied.
`ModelExpressTrainerClient` is an optional publisher. Custom integrations can
manage weight-version resources directly through the ModelExpress server's
Refit API. For S3, create a STAGING version with the correct model, payload
format, lineage, and index URI; upload the index and referenced shards in the
format expected by `ModelExpressGeneratorClient`; then mark the version READY.
See the [S3 Delta Weight Refit guide](https://github.com/ai-dynamo/modelexpress/blob/main/docs/S3_DELTA_WEIGHT_REFIT.md)
for the publication sequence and generator artifact requirements.
vLLM's `modelexpress_engine.py` re-exports the engine and its init/update
dataclasses from
`modelexpress_rl.inference.engines.vllm.weight_transfer_engine`.
The implementation and configuration schema live in ModelExpress. Upgrading
the installed ModelExpress package and restarting workers picks up compatible
changes without a vLLM code change. The package must continue to implement
vLLM's `WeightTransferEngine` interface.
## Inference Side
!!! warning "Development version"
This backend currently uses unreleased ModelExpress features from `main`,
which may introduce breaking changes. These features will be included in
the official ModelExpress v0.8.0 release.
Install ModelExpress from the `main` branch on every worker:
```bash
uv pip install "git+https://github.com/ai-dynamo/modelexpress@main#subdirectory=modelexpress_client/python"
```
The backend is registered natively and loaded only when selected;
`VLLM_PLUGINS=modelexpress` is not required for weight transfer. If the plugin
is enabled, it preserves the native registration and uses the same engine
implementation.
```python
from vllm import LLM
from vllm.config import WeightTransferConfig
llm = LLM(model="my-model", weight_transfer_config=WeightTransferConfig(backend="modelexpress"))
```
```bash
vllm serve /models/launch \
--load-format safetensors \
--weight-transfer-config '{"backend":"modelexpress"}'
```
### Initialize the engine
Initialize once by passing `init_info` to `init_weight_transfer_engine`.
For HTTP administration, set `VLLM_SERVER_DEV_MODE=1` before starting vLLM and
restrict these endpoints to trusted orchestrators.
For S3 updates, send the following body to `POST /init_weight_transfer_engine`:
```json
{
"init_info": {
"model_name": "policy",
"server_url": "modelexpress:8001",
"object_storage_type": "S3",
"initial_base_version_id": "policy-v0",
"seed_checkpoint_path": "/models/launch",
"refit_checkpoint_dir": "/mxdelta/receiver",
"refit_checkpoint_max_size_gb": 200,
"object_storage_region_name": "us-west-2"
}
}
```
### `ModelExpressWeightTransferInitInfo`
These fields belong inside `init_info`, rather than
`--weight-transfer-config`. The installed ModelExpress package defines the
schema and defaults.
| Field | Default | Description |
| ----- | ------- | ----------- |
| `model_name` | vLLM model name | Logical model identity used for published ModelExpress versions. Set this when it differs from the vLLM model path or name. |
| `server_url` | ModelExpress configuration | ModelExpress server address, such as `modelexpress:8001`. If omitted, uses ModelExpress's environment settings, then `localhost:8001`. |
| `initial_serving_version_id` | `None` | Version already loaded by the inference worker. If omitted, ModelExpress uses the initial version supplied by its runtime, when available. |
| `registration_ttl_seconds` | Three MX heartbeat intervals | Worker registration lifetime in seconds, renewed by ModelExpress. |
| `lease_ttl_seconds` | Registration lifetime | Weight-version lease lifetime in seconds, renewed while held. |
| `max_transfer_attempts` | `3` | Maximum source-discovery and transfer attempts for one staged update. |
| `max_replay_chain_length` | `64` | Maximum number of payload revisions replayed to reach the requested version. |
| `rpc_timeout_seconds` | `30.0` | Deadline in seconds for each control-plane or manifest RPC. |
### Object storage settings
To enable object storage, provide `object_storage_type`,
`initial_base_version_id`, `seed_checkpoint_path`, and `refit_checkpoint_dir`
inside `init_info`. Endpoint and region settings also require these four fields.
| Field | Default | Description |
| ----- | ------- | ----------- |
| `object_storage_type` | `None` | Object storage provider. Use `"S3"`; the generator currently supports S3. |
| `initial_base_version_id` | `None` | Registered base version matching the seed checkpoint. Required for object storage. |
| `seed_checkpoint_path` | `None` | Local seed checkpoint path accessible to each worker. Required for object storage. |
| `refit_checkpoint_dir` | `None` | Local directory for cached checkpoints and delta replay. Required for object storage. |
| `refit_checkpoint_max_size_gb` | ModelExpress default | Checkpoint cache quota in decimal GB. The example sets an explicit 200 GB quota; it does not reserve disk space. |
| `object_storage_endpoint_url` | `None` | Optional endpoint override for an S3-compatible service. |
| `object_storage_region_name` | `None` | Optional S3 region, such as `us-west-2`. |
The seed checkpoint must match `initial_base_version_id`. Credentials use
ModelExpress's normal credential chain. Omit the object storage settings when
using only GPU peer transfer.
### Apply a weight update
After the target version is READY, pause generation and call
`POST /start_weight_update`. Then send the exact immutable version ID to
`POST /update_weights`:
```json
{"update_info": {"version_id": "policy-v1"}}
```
Call `POST /finish_weight_update` only after the update succeeds, then resume
inference. The staged handle is retained until finish or shutdown. Staging and
installation failures propagate; do not resume a worker with uncertain weights.
See the ModelExpress documentation for publication, source selection, object
storage, and deployment requirements.
+42
View File
@@ -75,6 +75,48 @@ These metrics are available via `--enable-mfu-metrics`:
--8<-- "gen:metrics-mfu"
## Custom Histogram Buckets
The core engine histograms ship with default bucket boundaries tuned for
typical serving workloads. The `--custom-histogram-buckets` option replaces
the boundaries of one or more *bucket families* — exactly the histograms
listed in the table below — with your own list; histograms owned by other
subsystems (for example, the NIXL connector metrics) are not affected. Use it,
for example, to track sub-300ms latency SLOs with the request-phase
histograms, whose smallest default boundary is 0.3s:
```bash
vllm serve Qwen/Qwen3-0.6B \
--custom-histogram-buckets '{"request_latency": [0.01, 0.05, 0.1, 0.25, 0.5, 1.0, 5.0, 30.0]}'
```
Each family key overrides a group of related histograms:
| Family key | Histograms |
| --- | --- |
| `request_latency` | `vllm:e2e_request_latency_seconds`, `vllm:request_queue_time_seconds`, `vllm:request_inference_time_seconds`, `vllm:request_prefill_time_seconds`, `vllm:request_decode_time_seconds` |
| `time_to_first_token` | `vllm:time_to_first_token_seconds` |
| `inter_token_latency` | `vllm:inter_token_latency_seconds`, `vllm:request_time_per_output_token_seconds` |
| `iteration_tokens` | `vllm:iteration_tokens_total` |
| `request_params_n` | `vllm:request_params_n` |
| `request_num_preemptions` | `vllm:request_num_preemptions` |
| `request_tokens` | `vllm:request_prompt_tokens`, `vllm:request_generation_tokens`, `vllm:request_max_num_generation_tokens`, `vllm:request_params_max_tokens`, `vllm:request_prefill_kv_computed_tokens` |
| `kv_cache_residency` | `vllm:kv_block_lifetime_seconds`, `vllm:kv_block_idle_before_evict_seconds`, `vllm:kv_block_reuse_gap_seconds` |
Bucket values must be positive, finite, and strictly increasing; unknown
family keys are rejected at startup. Families you do not list keep their
default boundaries. The `request_tokens` defaults normally scale with
`--max-model-len`; an override replaces that computed list. The
`kv_cache_residency` family only takes effect when `--kv-cache-metrics` is
enabled.
!!! warning "Bucket cardinality"
Every bucket boundary creates one extra time series per metric and per
label combination (model and engine index, multiplied under data-parallel
deployments). Long bucket lists inflate Prometheus storage, scrape sizes,
and query costs. Keep custom lists short, and only override the families
you actively monitor.
## Deprecation Policy
Note: when metrics are deprecated in version `X.Y`, they are hidden in version `X.Y+1`
+1 -1
View File
@@ -375,7 +375,7 @@ An attacker who can reach the gRPC port can:
1. **Run arbitrary inference** via the `Generate` and `GenerateStream` RPCs without any credentials
2. **Mutate engine state** by pausing generation, sleeping the engine, or initiating configured RL weight updates through the `Control` service
3. **Consume GPU and compute resources** by submitting unbounded generation requests
4. **Cause Denial of Service** by exploiting bugs in the gRPC interface that can crash vLLM.
4. **Stop a managed engine** through `Control.Shutdown`, or cause denial of service by exploiting bugs in the gRPC interface.
### Recommendations