Merge remote-tracking branch 'upstream/main' into patches
Signed-off-by: MOVZX <movzx@yahoo.com>
This commit is contained in:
@@ -127,7 +127,7 @@ Batch invariance has been tested and verified on the following models:
|
||||
- **Qwen3-VL (Vision-Language)**: `Qwen/Qwen3-VL-2B-Instruct`, `Qwen/Qwen3-VL-4B-Instruct` (single image and video inputs)
|
||||
- **Qwen3 (MoE)**: `Qwen/Qwen3-30B-A3B`, `Qwen/Qwen3-Next-80B-A3B-Instruct`, `Qwen/Qwen3-30B-A3B-Thinking-2507-FP8`
|
||||
- **Qwen2.5**: `Qwen/Qwen2.5-0.5B-Instruct`, `Qwen/Qwen2.5-1.5B-Instruct`, `Qwen/Qwen2.5-3B-Instruct`, `Qwen/Qwen2.5-7B-Instruct`, `Qwen/Qwen2.5-14B-Instruct`, `Qwen/Qwen2.5-32B-Instruct`
|
||||
- **Llama 3**: Llama3.1 and 3.2 series, `meta-llama/Llama-3.2-3B-Instruct` for example
|
||||
- **Llama 3**: Llama3.1 and 3.2 series, `meta-llama/Llama-3.2-1B-Instruct`, `meta-llama/Llama-3.2-3B-Instruct` for example
|
||||
- **GPT-OSS**: `openai/gpt-oss-20b`, `openai/gpt-oss-120b`
|
||||
- **Mistral**: `mistralai/Mistral-7B-v0.3`
|
||||
- **Phi series**: `microsoft/Phi-3.5-mini-instruct`
|
||||
@@ -135,6 +135,8 @@ Batch invariance has been tested and verified on the following models:
|
||||
- **Granite 3.1 (Dense)**: `ibm-granite/granite-3.1-2b-instruct`, `ibm-granite/granite-3.1-8b-instruct`
|
||||
- **EXAONE 4.0 series**: `LGAI-EXAONE/EXAONE-4.0-1.2B`, `LGAI-EXAONE/EXAONE-4.0.1-32B`, `LGAI-EXAONE/EXAONE-4.0-32B`
|
||||
- **OLMo 2**: `allenai/OLMo-2-0425-1B-Instruct`
|
||||
- **SmolLM2**: `HuggingFaceTB/SmolLM2-1.7B-Instruct`
|
||||
- **PLaMo3**: `pfnet/plamo-3-nict-2b-base`
|
||||
|
||||
Other models may also work, but these have been explicitly validated. If you encounter issues with a specific model, please report them on the [GitHub issue tracker](https://github.com/vllm-project/vllm/issues/new/choose).
|
||||
|
||||
@@ -144,11 +146,15 @@ When batch invariance is enabled, vLLM:
|
||||
|
||||
1. Uses deterministic kernel implementations for attention and other operations
|
||||
2. Ensures consistent numerical behavior across different batch sizes
|
||||
3. Disables certain optimizations that may introduce non-determinism (such as custom all-reduce operations in tensor parallel mode, and sequence parallelism / async TP, whose reduce-scatter path is not batch-invariant)
|
||||
4. On CUDA devices with tuned matmul table entries for the model's bf16 unquantized forward linear layers (Ada, Hopper, Blackwell),
|
||||
3. Disables certain optimizations that may introduce non-determinism (such as sequence parallelism / async TP, whose reduce-scatter path is not batch-invariant)
|
||||
4. Under tensor parallelism, keeps custom all-reduce on with a fixed reduction order (the 1-stage kernel is pinned, and large inputs are reduced in fixed-size chunks), and disables FlashInfer, AITER and QuickReduce all-reduce
|
||||
5. On CUDA devices with tuned matmul table entries for the model's bf16 unquantized forward linear layers (Ada, Hopper, Blackwell),
|
||||
runs without `torch.compile` using breakable CUDA graphs so tile configs follow the runtime batch size; set `VLLM_USE_BREAKABLE_CUDAGRAPH=0` to opt out
|
||||
(not applied when sequence parallelism / async TP are enabled, since those are `torch.compile` passes).
|
||||
|
||||
!!! warning
|
||||
Batch invariance under tensor parallelism is not yet supported for all-reduces whose size isn't a multiple of 16 bytes, for example a hidden size that isn't a multiple of 8 in fp16/bf16. Such tensors can switch between NCCL and custom all-reduce depending on batch size. All validated models meet this requirement.
|
||||
|
||||
!!! note
|
||||
Enabling batch invariance may impact performance compared to the default non-deterministic mode. This trade-off is intentional to guarantee reproducibility.
|
||||
|
||||
|
||||
@@ -45,9 +45,10 @@ The following scripts demonstrate the workflow with **ExampleConnector**:
|
||||
### ECMooncakeConnector
|
||||
|
||||
**ECMooncakeConnector** transfers encoder outputs using the Mooncake TransferEngine.
|
||||
See the [Mooncake integration example](../../tests/v1/ec_connector/integration/run_epd_mooncake_ec_full_pipeline.sh)
|
||||
for a complete 1 Encoder instance + 1 PD instance setup, including the producer and
|
||||
consumer `--ec-transfer-config` settings and proxy configuration.
|
||||
See the [ECMooncakeConnector Usage Guide](mooncake_ec_connector_usage.md) for
|
||||
producer/consumer `--ec-transfer-config`, proxy `--ec-consumer-zmq-addrs`, and
|
||||
transport options. The [Mooncake integration example](../../tests/v1/ec_connector/integration/run_epd_mooncake_ec_full_pipeline.sh)
|
||||
is a complete 1 Encoder instance + 1 PD instance setup.
|
||||
|
||||
With vLLM and Mooncake installed, run the example from the repository root:
|
||||
|
||||
|
||||
@@ -27,7 +27,7 @@ Now supports 9 types of connectors:
|
||||
--kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_both", "kv_buffer_device":"cuda", "kv_connector_extra_config":{"backends":["UCX", "GDS"]}}'
|
||||
```
|
||||
|
||||
- **MooncakeConnector**: refer to [examples/disaggregated/mooncake_connector/run_mooncake_connector.sh](../../examples/disaggregated/mooncake_connector/run_mooncake_connector.sh) for the example usage of MooncakeConnector disaggregated prefilling. For detailed usage guide, see [MooncakeConnector Usage Guide](mooncake_connector_usage.md).
|
||||
- **MooncakeConnector**: refer to [examples/disaggregated/mooncake_connector/run_mooncake_connector.sh](../../examples/disaggregated/mooncake_connector/run_mooncake_connector.sh) for the example usage of MooncakeConnector disaggregated prefilling. For detailed usage guide, see [MooncakeConnector Usage Guide](mooncake_connector_usage.md). For multimodal encoder-cache transfer, see [ECMooncakeConnector Usage Guide](mooncake_ec_connector_usage.md).
|
||||
- **MoRIIOConnector** (ROCm only): see [MoRI-IO Usage Guide](moriio_connector_usage.md) for example usage and detailed documentation.
|
||||
- **MultiConnector**: take advantage of the kv_connector_extra_config: dict[str, Any] already present in KVTransferConfig to stash all the connectors we want in an ordered list of kwargs.such as:
|
||||
|
||||
|
||||
@@ -42,10 +42,10 @@ Snapshots currently require:
|
||||
and CRIU still reopens it by path, so replacing that file permanently
|
||||
invalidates the artifact without an early error.
|
||||
|
||||
The official `vllm/vllm-openai` Linux x86-64 image includes the snapshot
|
||||
runtime. It still requires a compatible host driver, kernel, and privileges.
|
||||
Arm64 images omit it. Source installs must set `CRIU_CUDA_PLUGIN_DIR` to the
|
||||
directory containing `cuda_plugin.so`.
|
||||
The official CUDA 13 `vllm/vllm-openai` Linux x86-64 images include the
|
||||
snapshot runtime. CUDA 12.x and Arm64 images omit it. A compatible host
|
||||
driver, kernel, and privileges are still required. Source installs must set
|
||||
`CRIU_CUDA_PLUGIN_DIR` to the directory containing `cuda_plugin.so`.
|
||||
|
||||
Run snapshot commands with `docker exec` inside a long-lived container. Restore
|
||||
hands the API server off as a detached process, so a one-shot container would
|
||||
|
||||
@@ -38,6 +38,9 @@ python examples/disaggregated/mooncake_connector/mooncake_connector_proxy.py --p
|
||||
|
||||
Now you can send requests to the proxy server through port 8000.
|
||||
|
||||
For multimodal Encode-Prefill-Decode (encoder embeddings rather than KV
|
||||
cache), see [ECMooncakeConnector Usage Guide](mooncake_ec_connector_usage.md).
|
||||
|
||||
## Environment Variables
|
||||
|
||||
- `VLLM_MOONCAKE_BOOTSTRAP_PORT`: Port for Mooncake bootstrap server
|
||||
@@ -76,6 +79,7 @@ Now you can send requests to the proxy server through port 8000.
|
||||
- **num_workers**: Size of thread pool for one prefiller worker to transfer KV caches by mooncake. (default 10)
|
||||
- **mooncake_protocol**: Mooncake connector protocol. (default "rdma")
|
||||
- **device_name**: Comma-separated whitelist of RDMA devices (e.g. `"mlx5_0,mlx5_1"`) to restrict topology discovery to. Empty discovers every device. Useful on hosts exposing a mix of InfiniBand and RoCE ports, where both peers must settle on the same link layer.
|
||||
- **bootstrap_server_address**: Optional `"host:port"` of an already running Mooncake bootstrap server. All prefiller ranks register with this endpoint instead of launching an embedded server. The caller owns the server lifetime and must configure the proxy to use the same endpoint. The example proxy assumes the bootstrap server is on the prefiller API host; a different host requires a proxy that supplies that address in `remote_bootstrap_addr`. Omit this option to use the embedded server and `VLLM_MOONCAKE_BOOTSTRAP_PORT`.
|
||||
|
||||
## Example Scripts/Code
|
||||
|
||||
|
||||
@@ -0,0 +1,158 @@
|
||||
# ECMooncakeConnector Usage Guide
|
||||
|
||||
`ECMooncakeConnector` moves multimodal **encoder embeddings** (encoder cache)
|
||||
between a disaggregated encoder instance and a language-model instance over
|
||||
[Mooncake Transfer Engine](https://github.com/kvcache-ai/Mooncake). Use it
|
||||
when the vision encoder should scale independently of prefill and decode.
|
||||
|
||||
For KV-cache transfer between prefiller and decoder, see
|
||||
[MooncakeConnector Usage Guide](mooncake_connector_usage.md). That path uses
|
||||
`MooncakeConnector`, not `ECMooncakeConnector`.
|
||||
|
||||
Added in [PR #41567](https://github.com/vllm-project/vllm/pull/41567).
|
||||
Overview of disaggregated encoding:
|
||||
[Disaggregated Encoder](disagg_encoder.md).
|
||||
|
||||
!!! important
|
||||
Static proxy flags (`--ec-consumer-zmq-addrs`) support Mooncake EC only in
|
||||
**1E + 1PD** mode (`--prefill-servers-urls disable`). For 1E + 1P + 1D,
|
||||
register `ec_zmq_addrs` on the **prefill** consumer through the proxy's
|
||||
dynamic registration API instead of the static CLI flag. See
|
||||
`examples/disaggregated/disaggregated_encoder/README.md`.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Install Mooncake Transfer Engine:
|
||||
|
||||
```bash
|
||||
# CUDA 13 (vLLM default)
|
||||
uv pip install mooncake-transfer-engine-cuda13
|
||||
|
||||
# CUDA 12
|
||||
uv pip install mooncake-transfer-engine
|
||||
```
|
||||
|
||||
The two wheels are the same release built against different CUDA majors.
|
||||
Importing the wrong one fails with
|
||||
`libcudart.so.<major>: cannot open shared object file`.
|
||||
|
||||
## 1E + 1PD
|
||||
|
||||
One encoder producer and one combined prefill/decode consumer. The consumer
|
||||
listens on `ec_ip` + `ec_port` (ZMQ control). The proxy must advertise that
|
||||
address with `--ec-consumer-zmq-addrs`.
|
||||
|
||||
### Encoder (producer)
|
||||
|
||||
The producer must run with `tensor_parallel_size=1`,
|
||||
`pipeline_parallel_size=1`, and `data_parallel_size=1`.
|
||||
|
||||
```bash
|
||||
CUDA_VISIBLE_DEVICES=0 vllm serve Qwen/Qwen2.5-VL-3B-Instruct \
|
||||
--port 19534 \
|
||||
--gpu-memory-utilization 0.35 \
|
||||
--enable-request-id-headers \
|
||||
--no-enable-prefix-caching \
|
||||
--max-num-batched-tokens 16384 \
|
||||
--allowed-local-media-path /path/to/media \
|
||||
--ec-transfer-config '{
|
||||
"ec_connector": "ECMooncakeConnector",
|
||||
"ec_role": "ec_producer",
|
||||
"ec_connector_extra_config": {
|
||||
"mooncake_protocol": "rdma"
|
||||
}
|
||||
}'
|
||||
```
|
||||
|
||||
### Prefill+Decode (consumer)
|
||||
|
||||
```bash
|
||||
CUDA_VISIBLE_DEVICES=1 vllm serve Qwen/Qwen2.5-VL-3B-Instruct \
|
||||
--port 19537 \
|
||||
--gpu-memory-utilization 0.75 \
|
||||
--enable-mm-embeds \
|
||||
--enable-request-id-headers \
|
||||
--no-enable-prefix-caching \
|
||||
--allowed-local-media-path /path/to/media \
|
||||
--ec-transfer-config '{
|
||||
"ec_connector": "ECMooncakeConnector",
|
||||
"ec_role": "ec_consumer",
|
||||
"ec_ip": "127.0.0.1",
|
||||
"ec_port": 19019,
|
||||
"ec_connector_extra_config": {
|
||||
"mooncake_protocol": "rdma"
|
||||
}
|
||||
}'
|
||||
```
|
||||
|
||||
`--enable-mm-embeds` lets the PD instance inject transferred embeddings
|
||||
instead of re-running the vision encoder.
|
||||
|
||||
### Proxy
|
||||
|
||||
```bash
|
||||
python examples/disaggregated/disaggregated_encoder/disagg_epd_proxy.py \
|
||||
--host 0.0.0.0 \
|
||||
--port 10002 \
|
||||
--encode-servers-urls http://127.0.0.1:19534 \
|
||||
--prefill-servers-urls disable \
|
||||
--decode-servers-urls http://127.0.0.1:19537 \
|
||||
--ec-consumer-zmq-addrs tcp://127.0.0.1:19019
|
||||
```
|
||||
|
||||
`--ec-consumer-zmq-addrs` must align with `--decode-servers-urls` (the
|
||||
combined PD instances). For data-parallel consumers, list each replica
|
||||
consecutively and set `--ec-consumer-dp-size`.
|
||||
|
||||
### Correctness script
|
||||
|
||||
```bash
|
||||
# TCP (no RDMA device required; CI default)
|
||||
MOONCAKE_EC_PROTOCOL=tcp \
|
||||
bash tests/v1/ec_connector/integration/run_epd_mooncake_ec_full_pipeline.sh
|
||||
|
||||
# RDMA
|
||||
MOONCAKE_EC_PROTOCOL=rdma \
|
||||
bash tests/v1/ec_connector/integration/run_epd_mooncake_ec_full_pipeline.sh
|
||||
```
|
||||
|
||||
The TCP path sets `MC_FORCE_TCP=1` so Transfer Engine does not auto-select
|
||||
RDMA when an HCA is present.
|
||||
|
||||
## Configuration
|
||||
|
||||
Producer and consumer must use the same `mooncake_protocol`.
|
||||
|
||||
| Setting | Where | Default | Notes |
|
||||
| --------- | ------- | --------- | ------- |
|
||||
| `mooncake_protocol` | `ec_connector_extra_config` | `rdma` | Set `tcp` when no verbs device is available. With TCP, also export `MC_FORCE_TCP=1`. |
|
||||
| `ec_ip` / `ec_port` | consumer `ECTransferConfig` | `127.0.0.1` / `14579` | First ZMQ control port. Tensor-parallel rank `r` listens on `ec_port + dp_index * tp_size + r`. |
|
||||
| `ec_buffer_size` | `ECTransferConfig` | `1e9` | Registered receive arena on the consumer, in bytes. |
|
||||
| `control_timeout_s` | `ec_connector_extra_config` | `30` | Reservation RPC timeout. |
|
||||
| `push_wait_timeout_s` | `ec_connector_extra_config` | `60` | How long the consumer waits for a matching encoder push. |
|
||||
| `WITH_NVIDIA_PEERMEM` | process environment | `1` | Read by Mooncake. Set `0` on hosts without `nvidia-peermem` (DMA-BUF path). |
|
||||
| `--ec-consumer-zmq-addrs` | proxy | empty | `tcp://<ec_ip>:<ec_port>` for each PD replica. Required for Mooncake EC. |
|
||||
|
||||
## 1E + 1P + 1D
|
||||
|
||||
The filesystem example is `examples/disaggregated/disaggregated_encoder/disagg_1e1p1d_example.sh`
|
||||
(`ECExampleConnector` for encoder cache, `NixlConnector` for KV). P→D KV can
|
||||
use `MooncakeConnector` instead; see
|
||||
[MooncakeConnector Usage Guide](mooncake_connector_usage.md).
|
||||
|
||||
Do **not** pass `--ec-consumer-zmq-addrs` together with independent prefill
|
||||
URLs: the static proxy flag is E+PD only. For Mooncake EC with a dedicated
|
||||
prefill, use dynamic registration and set `ec_zmq_addrs` on the prefill
|
||||
consumer as described in
|
||||
`examples/disaggregated/disaggregated_encoder/README.md`.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Symptom | Likely cause |
|
||||
| --------- | ---------------- |
|
||||
| `Install mooncake-transfer-engine` / `libcudart.so.*` | Wrong or missing Mooncake wheel for this CUDA major. |
|
||||
| `Failed to register memory ... Bad address [14]` | GPUDirect registration failed. Set `WITH_NVIDIA_PEERMEM=0`. |
|
||||
| Proxy startup error about E+PD only | `--ec-consumer-zmq-addrs` used with a real `--prefill-servers-urls`. |
|
||||
| Consumer waits then errors; encoder never finishes the push | `--ec-consumer-zmq-addrs` does not match consumer `ec_ip`/`ec_port`, or producer/consumer protocols differ. |
|
||||
| TCP transfers stall on a host that also has RDMA | Export `MC_FORCE_TCP=1`. |
|
||||
| Producer rejected at startup | Encoder used tensor, pipeline, or data parallelism. Keep the producer unsharded. |
|
||||
@@ -1210,7 +1210,7 @@ vLLM decodes audio bytes into waveforms using a selectable decoding backend:
|
||||
|
||||
| Backend | Description |
|
||||
| --- | --- |
|
||||
| `auto` (default) | soundfile, falling back to torchcodec, then PyAV |
|
||||
| `auto` (default) | torchcodec, falling back to soundfile, then PyAV |
|
||||
| `soundfile` | libsndfile only, no fallback |
|
||||
| `pyav` | PyAV (FFmpeg) only, no fallback |
|
||||
| `torchcodec` | TorchCodec (PyTorch-native) only, no fallback |
|
||||
@@ -1226,17 +1226,18 @@ vllm serve mistralai/Voxtral-Mini-3B-2507 \
|
||||
`pyav` drives FFmpeg through a per-frame Python generator, so under
|
||||
concurrency the Python/C crossings contend on the GIL. `torchcodec`
|
||||
decodes each stream in a single call that releases the GIL for its whole
|
||||
duration. Select it explicitly for concurrent decoding workloads.
|
||||
`auto` prefers soundfile for supported formats to preserve their existing
|
||||
decoding behavior, including encoder padding. Audio extracted from formats
|
||||
that soundfile cannot read, such as video containers, can use torchcodec
|
||||
through the fallback chain.
|
||||
duration, which is why `auto` prefers it when many requests decode audio
|
||||
concurrently, such as when audio tracks are extracted from video.
|
||||
vLLM normalizes codec padding on this path (e.g. it trims trailing Vorbis
|
||||
padding that older system FFmpeg versions (< 5.0) fail to trim), so the
|
||||
decoded waveform matches the soundfile reference length.
|
||||
|
||||
!!! note
|
||||
`torchcodec` ships as a requirement on CUDA, CPU and XPU builds. On other
|
||||
platforms (e.g. ROCm, TPU), install it manually to enable that backend.
|
||||
torchcodec also links against a system FFmpeg installation. If the package
|
||||
or FFmpeg is unavailable, `auto` uses the soundfile → PyAV chain.
|
||||
platforms (e.g. ROCm, TPU) `auto` falls back to the soundfile → PyAV
|
||||
chain unless you install it manually. torchcodec also links against a
|
||||
system FFmpeg installation; if FFmpeg is missing, `auto` falls back the
|
||||
same way.
|
||||
|
||||
### Embedding Inputs
|
||||
|
||||
|
||||
@@ -454,6 +454,7 @@ exported when NixlConnector is active:
|
||||
| `vllm:nixl_num_failed_transfers` | Counter | Cumulative count of failed NIXL KV-block transfers, including handshake and notification (`send_notif`) failures. These are grouped because all are sporadic lower-transport-layer events. |
|
||||
| `vllm:nixl_num_failed_notifications` | Counter | Cumulative count of failed notifications (`send_notif`), retained for compatibility. These failures are also included in `vllm:nixl_num_failed_transfers`; do not sum the two counters. |
|
||||
| `vllm:nixl_num_kv_expired_reqs` | Counter | Requests whose KV blocks expired on the prefiller before the decoder read them (tracked on the P instance). Kept separate from the failure counters above: KV expiry is an autoscaler/lease-tuning signal, not a transport issue. |
|
||||
| `vllm:nixl_num_notifications_after_expiry` | Counter | Completion notifications that arrive for a request that is no longer tracked, usually because its KV lease expired first. Pull connector: counted on P, where it means the decoder may have read blocks that were already reused; each matches a "Potentially invalid KV blocks" error log. Push connector: counted on whichever instance tracks the request as neither a send nor a receive (P after the lease expired, or D after the receive completed or was abandoned); each matches an "Unrecognized request" error log. Counted per notification, not per request: each remote rank notifies separately, so with TP > 1 or heterogeneous TP one late request counts several times, and the value is not comparable to `vllm:nixl_num_kv_expired_reqs`. |
|
||||
|
||||
!!! tip
|
||||
High `vllm:nixl_num_kv_expired_reqs` indicates that the prefiller's lease
|
||||
|
||||
@@ -29,6 +29,7 @@ vLLM currently supports the following reasoning models:
|
||||
| [MiniMax-M2](https://huggingface.co/MiniMaxAI/MiniMax-M2) | `minimax_m2_append_think` | `json`, `regex` | ✅ |
|
||||
| [Qwen3 series](https://huggingface.co/collections/Qwen/qwen3-67dd247413f0e2e4f653967f) | `qwen3` | `json`, `regex` | ✅ |
|
||||
| [QwQ-32B](https://huggingface.co/Qwen/QwQ-32B) | `deepseek_r1` | `json`, `regex` | ✅ |
|
||||
| [Step-3.5 / Step-3.7 Flash](https://huggingface.co/stepfun-ai/Step-3.5-Flash) | `step3p5` | `json`, `regex` | ✅ |
|
||||
|
||||
!!! note
|
||||
IBM Granite 3.2 and DeepSeek-V3.1 reasoning is disabled by default; to enable it, you must also pass `thinking=True` in your `chat_template_kwargs`.
|
||||
|
||||
@@ -171,6 +171,29 @@ from HuggingFace; and you can find an example of this in a `tokenizer_config.jso
|
||||
|
||||
If your favorite tool-calling model is not supported, please feel free to contribute a parser & tool use chat template!
|
||||
|
||||
### Checkpoint Response Templates (`hf`)
|
||||
|
||||
Checkpoints can define a `response_template` in `tokenizer_config.json` to describe their reasoning, content, and tool-call wire format. Select the
|
||||
`hf` parser to parse output from this metadata instead of a model-specific parser. Like `--tokenizer-mode hf`, it relies on the checkpoint's
|
||||
Hugging Face metadata rather than model-specific code:
|
||||
|
||||
```bash
|
||||
vllm serve <model> \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser hf \
|
||||
--reasoning-parser hf
|
||||
```
|
||||
|
||||
Either parser can be selected on its own. Startup fails if the tokenizer has no `response_template`, or if it lacks the `thinking` or `tool_calls`
|
||||
field the selected parser needs. The `hf` parser cannot be combined with other reasoning or tool call parsers.
|
||||
|
||||
When streaming, each tool call is emitted whole once its region parses. A call cut off before its closer, for example by a closer that is also a
|
||||
stop token, is parsed from the text generated so far; a call that does not parse is dropped.
|
||||
|
||||
Response templates currently provide parsing, not format-specific constrained decoding. The parser therefore rejects requests for strict tools,
|
||||
required or named tool choice, or `parallel_tool_calls=false`. A custom server or request chat template is allowed, but the parser still expects the
|
||||
checkpoint's output format.
|
||||
|
||||
!!! note
|
||||
With `tool_choice="auto"`, structural-tag constraints require both `VLLM_ENFORCE_STRICT_TOOL_CALLING=true` (the default) and at least one tool with `strict: true`, or a server-side floor set via `--tool-strict-level`. When these conditions are met and the selected parser supports structural tags, vLLM constrains the tool-call envelope and pins the argument schema of each tool that sets `strict: true` (or of every tool under `--tool-strict-level parameter`). Otherwise, vLLM extracts tool calls from raw text, so arguments may occasionally be malformed or violate the function's parameter schema.
|
||||
|
||||
@@ -458,6 +481,29 @@ Supported models:
|
||||
|
||||
Flags: `--tool-call-parser qwen3_xml`
|
||||
|
||||
### MiMo-V2.6 Models (`mimo`)
|
||||
|
||||
Supported models:
|
||||
|
||||
* `XiaomiMiMo/MiMo-V2.6-Pro-RL`
|
||||
* `XiaomiMiMo/MiMo-V2.6-Flash-RL`
|
||||
|
||||
Use the checkpoint's chat template with `--tool-call-parser mimo --reasoning-parser mimo --enable-auto-tool-choice`.
|
||||
For schema-constrained tool arguments, set `strict: true` inside each tool's `function`, or use `--tool-strict-level parameter`.
|
||||
The parser supports `auto`, `required`, named tools, and `parallel_tool_calls=False`, preserving leading/trailing whitespace in string arguments.
|
||||
|
||||
### Step-3.5 and Step-3.7 Models (`step3p5`)
|
||||
|
||||
Supported models:
|
||||
|
||||
* `stepfun-ai/Step-3.5-Flash`
|
||||
* `stepfun-ai/Step-3.5-Flash-FP8`
|
||||
* `stepfun-ai/Step-3.7-Flash`
|
||||
|
||||
Flags: `--tool-call-parser step3p5 --reasoning-parser step3p5 --enable-auto-tool-choice`
|
||||
|
||||
The parser supports `auto`, `required`, and named tools.
|
||||
|
||||
### Olmo 3 Models (`olmo3`)
|
||||
|
||||
Olmo 3 models output tool calls in a format that is very similar to the one expected by the `pythonic` parser (see below), with a few differences. Each tool call is a pythonic string, but the parallel tool calls are newline-delimited, and the calls are wrapped within XML tags as `<function_calls>..</function_calls>`. In addition, the parser also allows JSON boolean and null literals (`true`, `false`, and `null`) in addition to the pythonic ones (`True`, `False`, and `None`).
|
||||
|
||||
@@ -77,30 +77,17 @@ watermarked. The watermark signal is diluted in proportion to the share of
|
||||
output tokens supplied by accepted drafts; rejected drafts do not dilute it
|
||||
because their recovery tokens are watermarked.
|
||||
|
||||
Speculative-decoding token paths do not currently support generation-side
|
||||
context deduplication. The configured `deduplicate_contexts` policy is not
|
||||
applied to accepted drafts, rejection-recovery tokens, or bonus tokens.
|
||||
|
||||
For `dual_key_gumbel`, `alpha` has no effect under speculative decoding. The
|
||||
speculative protocol selects the key for each token instead.
|
||||
|
||||
## Algorithms
|
||||
|
||||
### Gumbel-max
|
||||
|
||||
Gumbel-max derives a deterministic pseudorandom value from the key, prior token
|
||||
context, and every candidate token, then uses the resulting Gumbel noise for
|
||||
categorical sampling. See
|
||||
[Aaronson's original presentation](https://simons.berkeley.edu/sites/default/files/2024-10/LLM24-2%20Slides%20-%20Scott%20Aaronson.pdf).
|
||||
|
||||
Gumbel-max requires stochastic sampling. Greedy requests (`temperature=0`)
|
||||
bypass watermarking and emit a warning once per worker.
|
||||
## Context deduplication
|
||||
|
||||
When a token context is repeated, generation can use ordinary sampling for that
|
||||
occurrence. Reusing the repeated context leads to a bias over the sequence, as
|
||||
certain token choices would be correlated. Ordinary sampling at these positions
|
||||
allows for single-sequence non-distortion (see section G.3 of the
|
||||
[SynthID-Text supplementary materials](https://media.springernature.com/original/springer-static/esm/art%3A10.1038%2Fs41586-024-08025-4/MediaObjects/41586_2024_8025_MOESM1_ESM.pdf)).
|
||||
This also applies to drafter watermarking during speculative decoding.
|
||||
The detector independently deduplicates contexts so repeated keyed random
|
||||
vectors are not treated as independent evidence, meaning that context
|
||||
deduplication at generation time does not reduce the watermarking signal, unless
|
||||
@@ -143,6 +130,18 @@ vllm serve MODEL \
|
||||
'{"algorithm":"gumbel","key":42,"deduplicate_contexts":"all"}'
|
||||
```
|
||||
|
||||
## Algorithms
|
||||
|
||||
### Gumbel-max
|
||||
|
||||
Gumbel-max derives a deterministic pseudorandom value from the key, prior token
|
||||
context, and every candidate token, then uses the resulting Gumbel noise for
|
||||
categorical sampling. See
|
||||
[Aaronson's original presentation](https://simons.berkeley.edu/sites/default/files/2024-10/LLM24-2%20Slides%20-%20Scott%20Aaronson.pdf).
|
||||
|
||||
Gumbel-max requires stochastic sampling. Greedy requests (`temperature=0`)
|
||||
bypass watermarking and emit a warning once per worker.
|
||||
|
||||
### Dual-key Gumbel-max
|
||||
|
||||
Dual-key Gumbel-max derives independent keys A and B from one configured master
|
||||
|
||||
Reference in New Issue
Block a user