Merge remote-tracking branch 'upstream/main' into patches

Signed-off-by: MOVZX <movzx@yahoo.com>
This commit is contained in:
2026-10-03 21:59:03 +07:00
1012 changed files with 61873 additions and 14499 deletions
+9 -3
View File
@@ -127,7 +127,7 @@ Batch invariance has been tested and verified on the following models:
- **Qwen3-VL (Vision-Language)**: `Qwen/Qwen3-VL-2B-Instruct`, `Qwen/Qwen3-VL-4B-Instruct` (single image and video inputs)
- **Qwen3 (MoE)**: `Qwen/Qwen3-30B-A3B`, `Qwen/Qwen3-Next-80B-A3B-Instruct`, `Qwen/Qwen3-30B-A3B-Thinking-2507-FP8`
- **Qwen2.5**: `Qwen/Qwen2.5-0.5B-Instruct`, `Qwen/Qwen2.5-1.5B-Instruct`, `Qwen/Qwen2.5-3B-Instruct`, `Qwen/Qwen2.5-7B-Instruct`, `Qwen/Qwen2.5-14B-Instruct`, `Qwen/Qwen2.5-32B-Instruct`
- **Llama 3**: Llama3.1 and 3.2 series, `meta-llama/Llama-3.2-3B-Instruct` for example
- **Llama 3**: Llama3.1 and 3.2 series, `meta-llama/Llama-3.2-1B-Instruct`, `meta-llama/Llama-3.2-3B-Instruct` for example
- **GPT-OSS**: `openai/gpt-oss-20b`, `openai/gpt-oss-120b`
- **Mistral**: `mistralai/Mistral-7B-v0.3`
- **Phi series**: `microsoft/Phi-3.5-mini-instruct`
@@ -135,6 +135,8 @@ Batch invariance has been tested and verified on the following models:
- **Granite 3.1 (Dense)**: `ibm-granite/granite-3.1-2b-instruct`, `ibm-granite/granite-3.1-8b-instruct`
- **EXAONE 4.0 series**: `LGAI-EXAONE/EXAONE-4.0-1.2B`, `LGAI-EXAONE/EXAONE-4.0.1-32B`, `LGAI-EXAONE/EXAONE-4.0-32B`
- **OLMo 2**: `allenai/OLMo-2-0425-1B-Instruct`
- **SmolLM2**: `HuggingFaceTB/SmolLM2-1.7B-Instruct`
- **PLaMo3**: `pfnet/plamo-3-nict-2b-base`
Other models may also work, but these have been explicitly validated. If you encounter issues with a specific model, please report them on the [GitHub issue tracker](https://github.com/vllm-project/vllm/issues/new/choose).
@@ -144,11 +146,15 @@ When batch invariance is enabled, vLLM:
1. Uses deterministic kernel implementations for attention and other operations
2. Ensures consistent numerical behavior across different batch sizes
3. Disables certain optimizations that may introduce non-determinism (such as custom all-reduce operations in tensor parallel mode, and sequence parallelism / async TP, whose reduce-scatter path is not batch-invariant)
4. On CUDA devices with tuned matmul table entries for the model's bf16 unquantized forward linear layers (Ada, Hopper, Blackwell),
3. Disables certain optimizations that may introduce non-determinism (such as sequence parallelism / async TP, whose reduce-scatter path is not batch-invariant)
4. Under tensor parallelism, keeps custom all-reduce on with a fixed reduction order (the 1-stage kernel is pinned, and large inputs are reduced in fixed-size chunks), and disables FlashInfer, AITER and QuickReduce all-reduce
5. On CUDA devices with tuned matmul table entries for the model's bf16 unquantized forward linear layers (Ada, Hopper, Blackwell),
runs without `torch.compile` using breakable CUDA graphs so tile configs follow the runtime batch size; set `VLLM_USE_BREAKABLE_CUDAGRAPH=0` to opt out
(not applied when sequence parallelism / async TP are enabled, since those are `torch.compile` passes).
!!! warning
Batch invariance under tensor parallelism is not yet supported for all-reduces whose size isn't a multiple of 16 bytes, for example a hidden size that isn't a multiple of 8 in fp16/bf16. Such tensors can switch between NCCL and custom all-reduce depending on batch size. All validated models meet this requirement.
!!! note
Enabling batch invariance may impact performance compared to the default non-deterministic mode. This trade-off is intentional to guarantee reproducibility.
+4 -3
View File
@@ -45,9 +45,10 @@ The following scripts demonstrate the workflow with **ExampleConnector**:
### ECMooncakeConnector
**ECMooncakeConnector** transfers encoder outputs using the Mooncake TransferEngine.
See the [Mooncake integration example](../../tests/v1/ec_connector/integration/run_epd_mooncake_ec_full_pipeline.sh)
for a complete 1 Encoder instance + 1 PD instance setup, including the producer and
consumer `--ec-transfer-config` settings and proxy configuration.
See the [ECMooncakeConnector Usage Guide](mooncake_ec_connector_usage.md) for
producer/consumer `--ec-transfer-config`, proxy `--ec-consumer-zmq-addrs`, and
transport options. The [Mooncake integration example](../../tests/v1/ec_connector/integration/run_epd_mooncake_ec_full_pipeline.sh)
is a complete 1 Encoder instance + 1 PD instance setup.
With vLLM and Mooncake installed, run the example from the repository root:
+1 -1
View File
@@ -27,7 +27,7 @@ Now supports 9 types of connectors:
--kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_both", "kv_buffer_device":"cuda", "kv_connector_extra_config":{"backends":["UCX", "GDS"]}}'
```
- **MooncakeConnector**: refer to [examples/disaggregated/mooncake_connector/run_mooncake_connector.sh](../../examples/disaggregated/mooncake_connector/run_mooncake_connector.sh) for the example usage of MooncakeConnector disaggregated prefilling. For detailed usage guide, see [MooncakeConnector Usage Guide](mooncake_connector_usage.md).
- **MooncakeConnector**: refer to [examples/disaggregated/mooncake_connector/run_mooncake_connector.sh](../../examples/disaggregated/mooncake_connector/run_mooncake_connector.sh) for the example usage of MooncakeConnector disaggregated prefilling. For detailed usage guide, see [MooncakeConnector Usage Guide](mooncake_connector_usage.md). For multimodal encoder-cache transfer, see [ECMooncakeConnector Usage Guide](mooncake_ec_connector_usage.md).
- **MoRIIOConnector** (ROCm only): see [MoRI-IO Usage Guide](moriio_connector_usage.md) for example usage and detailed documentation.
- **MultiConnector**: take advantage of the kv_connector_extra_config: dict[str, Any] already present in KVTransferConfig to stash all the connectors we want in an ordered list of kwargs.such as:
+4 -4
View File
@@ -42,10 +42,10 @@ Snapshots currently require:
and CRIU still reopens it by path, so replacing that file permanently
invalidates the artifact without an early error.
The official `vllm/vllm-openai` Linux x86-64 image includes the snapshot
runtime. It still requires a compatible host driver, kernel, and privileges.
Arm64 images omit it. Source installs must set `CRIU_CUDA_PLUGIN_DIR` to the
directory containing `cuda_plugin.so`.
The official CUDA 13 `vllm/vllm-openai` Linux x86-64 images include the
snapshot runtime. CUDA 12.x and Arm64 images omit it. A compatible host
driver, kernel, and privileges are still required. Source installs must set
`CRIU_CUDA_PLUGIN_DIR` to the directory containing `cuda_plugin.so`.
Run snapshot commands with `docker exec` inside a long-lived container. Restore
hands the API server off as a detached process, so a one-shot container would
@@ -38,6 +38,9 @@ python examples/disaggregated/mooncake_connector/mooncake_connector_proxy.py --p
Now you can send requests to the proxy server through port 8000.
For multimodal Encode-Prefill-Decode (encoder embeddings rather than KV
cache), see [ECMooncakeConnector Usage Guide](mooncake_ec_connector_usage.md).
## Environment Variables
- `VLLM_MOONCAKE_BOOTSTRAP_PORT`: Port for Mooncake bootstrap server
@@ -76,6 +79,7 @@ Now you can send requests to the proxy server through port 8000.
- **num_workers**: Size of thread pool for one prefiller worker to transfer KV caches by mooncake. (default 10)
- **mooncake_protocol**: Mooncake connector protocol. (default "rdma")
- **device_name**: Comma-separated whitelist of RDMA devices (e.g. `"mlx5_0,mlx5_1"`) to restrict topology discovery to. Empty discovers every device. Useful on hosts exposing a mix of InfiniBand and RoCE ports, where both peers must settle on the same link layer.
- **bootstrap_server_address**: Optional `"host:port"` of an already running Mooncake bootstrap server. All prefiller ranks register with this endpoint instead of launching an embedded server. The caller owns the server lifetime and must configure the proxy to use the same endpoint. The example proxy assumes the bootstrap server is on the prefiller API host; a different host requires a proxy that supplies that address in `remote_bootstrap_addr`. Omit this option to use the embedded server and `VLLM_MOONCAKE_BOOTSTRAP_PORT`.
## Example Scripts/Code
@@ -0,0 +1,158 @@
# ECMooncakeConnector Usage Guide
`ECMooncakeConnector` moves multimodal **encoder embeddings** (encoder cache)
between a disaggregated encoder instance and a language-model instance over
[Mooncake Transfer Engine](https://github.com/kvcache-ai/Mooncake). Use it
when the vision encoder should scale independently of prefill and decode.
For KV-cache transfer between prefiller and decoder, see
[MooncakeConnector Usage Guide](mooncake_connector_usage.md). That path uses
`MooncakeConnector`, not `ECMooncakeConnector`.
Added in [PR #41567](https://github.com/vllm-project/vllm/pull/41567).
Overview of disaggregated encoding:
[Disaggregated Encoder](disagg_encoder.md).
!!! important
Static proxy flags (`--ec-consumer-zmq-addrs`) support Mooncake EC only in
**1E + 1PD** mode (`--prefill-servers-urls disable`). For 1E + 1P + 1D,
register `ec_zmq_addrs` on the **prefill** consumer through the proxy's
dynamic registration API instead of the static CLI flag. See
`examples/disaggregated/disaggregated_encoder/README.md`.
## Prerequisites
Install Mooncake Transfer Engine:
```bash
# CUDA 13 (vLLM default)
uv pip install mooncake-transfer-engine-cuda13
# CUDA 12
uv pip install mooncake-transfer-engine
```
The two wheels are the same release built against different CUDA majors.
Importing the wrong one fails with
`libcudart.so.<major>: cannot open shared object file`.
## 1E + 1PD
One encoder producer and one combined prefill/decode consumer. The consumer
listens on `ec_ip` + `ec_port` (ZMQ control). The proxy must advertise that
address with `--ec-consumer-zmq-addrs`.
### Encoder (producer)
The producer must run with `tensor_parallel_size=1`,
`pipeline_parallel_size=1`, and `data_parallel_size=1`.
```bash
CUDA_VISIBLE_DEVICES=0 vllm serve Qwen/Qwen2.5-VL-3B-Instruct \
--port 19534 \
--gpu-memory-utilization 0.35 \
--enable-request-id-headers \
--no-enable-prefix-caching \
--max-num-batched-tokens 16384 \
--allowed-local-media-path /path/to/media \
--ec-transfer-config '{
"ec_connector": "ECMooncakeConnector",
"ec_role": "ec_producer",
"ec_connector_extra_config": {
"mooncake_protocol": "rdma"
}
}'
```
### Prefill+Decode (consumer)
```bash
CUDA_VISIBLE_DEVICES=1 vllm serve Qwen/Qwen2.5-VL-3B-Instruct \
--port 19537 \
--gpu-memory-utilization 0.75 \
--enable-mm-embeds \
--enable-request-id-headers \
--no-enable-prefix-caching \
--allowed-local-media-path /path/to/media \
--ec-transfer-config '{
"ec_connector": "ECMooncakeConnector",
"ec_role": "ec_consumer",
"ec_ip": "127.0.0.1",
"ec_port": 19019,
"ec_connector_extra_config": {
"mooncake_protocol": "rdma"
}
}'
```
`--enable-mm-embeds` lets the PD instance inject transferred embeddings
instead of re-running the vision encoder.
### Proxy
```bash
python examples/disaggregated/disaggregated_encoder/disagg_epd_proxy.py \
--host 0.0.0.0 \
--port 10002 \
--encode-servers-urls http://127.0.0.1:19534 \
--prefill-servers-urls disable \
--decode-servers-urls http://127.0.0.1:19537 \
--ec-consumer-zmq-addrs tcp://127.0.0.1:19019
```
`--ec-consumer-zmq-addrs` must align with `--decode-servers-urls` (the
combined PD instances). For data-parallel consumers, list each replica
consecutively and set `--ec-consumer-dp-size`.
### Correctness script
```bash
# TCP (no RDMA device required; CI default)
MOONCAKE_EC_PROTOCOL=tcp \
bash tests/v1/ec_connector/integration/run_epd_mooncake_ec_full_pipeline.sh
# RDMA
MOONCAKE_EC_PROTOCOL=rdma \
bash tests/v1/ec_connector/integration/run_epd_mooncake_ec_full_pipeline.sh
```
The TCP path sets `MC_FORCE_TCP=1` so Transfer Engine does not auto-select
RDMA when an HCA is present.
## Configuration
Producer and consumer must use the same `mooncake_protocol`.
| Setting | Where | Default | Notes |
| --------- | ------- | --------- | ------- |
| `mooncake_protocol` | `ec_connector_extra_config` | `rdma` | Set `tcp` when no verbs device is available. With TCP, also export `MC_FORCE_TCP=1`. |
| `ec_ip` / `ec_port` | consumer `ECTransferConfig` | `127.0.0.1` / `14579` | First ZMQ control port. Tensor-parallel rank `r` listens on `ec_port + dp_index * tp_size + r`. |
| `ec_buffer_size` | `ECTransferConfig` | `1e9` | Registered receive arena on the consumer, in bytes. |
| `control_timeout_s` | `ec_connector_extra_config` | `30` | Reservation RPC timeout. |
| `push_wait_timeout_s` | `ec_connector_extra_config` | `60` | How long the consumer waits for a matching encoder push. |
| `WITH_NVIDIA_PEERMEM` | process environment | `1` | Read by Mooncake. Set `0` on hosts without `nvidia-peermem` (DMA-BUF path). |
| `--ec-consumer-zmq-addrs` | proxy | empty | `tcp://<ec_ip>:<ec_port>` for each PD replica. Required for Mooncake EC. |
## 1E + 1P + 1D
The filesystem example is `examples/disaggregated/disaggregated_encoder/disagg_1e1p1d_example.sh`
(`ECExampleConnector` for encoder cache, `NixlConnector` for KV). P→D KV can
use `MooncakeConnector` instead; see
[MooncakeConnector Usage Guide](mooncake_connector_usage.md).
Do **not** pass `--ec-consumer-zmq-addrs` together with independent prefill
URLs: the static proxy flag is E+PD only. For Mooncake EC with a dedicated
prefill, use dynamic registration and set `ec_zmq_addrs` on the prefill
consumer as described in
`examples/disaggregated/disaggregated_encoder/README.md`.
## Troubleshooting
| Symptom | Likely cause |
| --------- | ---------------- |
| `Install mooncake-transfer-engine` / `libcudart.so.*` | Wrong or missing Mooncake wheel for this CUDA major. |
| `Failed to register memory ... Bad address [14]` | GPUDirect registration failed. Set `WITH_NVIDIA_PEERMEM=0`. |
| Proxy startup error about E+PD only | `--ec-consumer-zmq-addrs` used with a real `--prefill-servers-urls`. |
| Consumer waits then errors; encoder never finishes the push | `--ec-consumer-zmq-addrs` does not match consumer `ec_ip`/`ec_port`, or producer/consumer protocols differ. |
| TCP transfers stall on a host that also has RDMA | Export `MC_FORCE_TCP=1`. |
| Producer rejected at startup | Encoder used tensor, pipeline, or data parallelism. Keep the producer unsharded. |
+10 -9
View File
@@ -1210,7 +1210,7 @@ vLLM decodes audio bytes into waveforms using a selectable decoding backend:
| Backend | Description |
| --- | --- |
| `auto` (default) | soundfile, falling back to torchcodec, then PyAV |
| `auto` (default) | torchcodec, falling back to soundfile, then PyAV |
| `soundfile` | libsndfile only, no fallback |
| `pyav` | PyAV (FFmpeg) only, no fallback |
| `torchcodec` | TorchCodec (PyTorch-native) only, no fallback |
@@ -1226,17 +1226,18 @@ vllm serve mistralai/Voxtral-Mini-3B-2507 \
`pyav` drives FFmpeg through a per-frame Python generator, so under
concurrency the Python/C crossings contend on the GIL. `torchcodec`
decodes each stream in a single call that releases the GIL for its whole
duration. Select it explicitly for concurrent decoding workloads.
`auto` prefers soundfile for supported formats to preserve their existing
decoding behavior, including encoder padding. Audio extracted from formats
that soundfile cannot read, such as video containers, can use torchcodec
through the fallback chain.
duration, which is why `auto` prefers it when many requests decode audio
concurrently, such as when audio tracks are extracted from video.
vLLM normalizes codec padding on this path (e.g. it trims trailing Vorbis
padding that older system FFmpeg versions (< 5.0) fail to trim), so the
decoded waveform matches the soundfile reference length.
!!! note
`torchcodec` ships as a requirement on CUDA, CPU and XPU builds. On other
platforms (e.g. ROCm, TPU), install it manually to enable that backend.
torchcodec also links against a system FFmpeg installation. If the package
or FFmpeg is unavailable, `auto` uses the soundfile → PyAV chain.
platforms (e.g. ROCm, TPU) `auto` falls back to the soundfile → PyAV
chain unless you install it manually. torchcodec also links against a
system FFmpeg installation; if FFmpeg is missing, `auto` falls back the
same way.
### Embedding Inputs
+1
View File
@@ -454,6 +454,7 @@ exported when NixlConnector is active:
| `vllm:nixl_num_failed_transfers` | Counter | Cumulative count of failed NIXL KV-block transfers, including handshake and notification (`send_notif`) failures. These are grouped because all are sporadic lower-transport-layer events. |
| `vllm:nixl_num_failed_notifications` | Counter | Cumulative count of failed notifications (`send_notif`), retained for compatibility. These failures are also included in `vllm:nixl_num_failed_transfers`; do not sum the two counters. |
| `vllm:nixl_num_kv_expired_reqs` | Counter | Requests whose KV blocks expired on the prefiller before the decoder read them (tracked on the P instance). Kept separate from the failure counters above: KV expiry is an autoscaler/lease-tuning signal, not a transport issue. |
| `vllm:nixl_num_notifications_after_expiry` | Counter | Completion notifications that arrive for a request that is no longer tracked, usually because its KV lease expired first. Pull connector: counted on P, where it means the decoder may have read blocks that were already reused; each matches a "Potentially invalid KV blocks" error log. Push connector: counted on whichever instance tracks the request as neither a send nor a receive (P after the lease expired, or D after the receive completed or was abandoned); each matches an "Unrecognized request" error log. Counted per notification, not per request: each remote rank notifies separately, so with TP > 1 or heterogeneous TP one late request counts several times, and the value is not comparable to `vllm:nixl_num_kv_expired_reqs`. |
!!! tip
High `vllm:nixl_num_kv_expired_reqs` indicates that the prefiller's lease
+1
View File
@@ -29,6 +29,7 @@ vLLM currently supports the following reasoning models:
| [MiniMax-M2](https://huggingface.co/MiniMaxAI/MiniMax-M2) | `minimax_m2_append_think` | `json`, `regex` | ✅ |
| [Qwen3 series](https://huggingface.co/collections/Qwen/qwen3-67dd247413f0e2e4f653967f) | `qwen3` | `json`, `regex` | ✅ |
| [QwQ-32B](https://huggingface.co/Qwen/QwQ-32B) | `deepseek_r1` | `json`, `regex` | ✅ |
| [Step-3.5 / Step-3.7 Flash](https://huggingface.co/stepfun-ai/Step-3.5-Flash) | `step3p5` | `json`, `regex` | ✅ |
!!! note
IBM Granite 3.2 and DeepSeek-V3.1 reasoning is disabled by default; to enable it, you must also pass `thinking=True` in your `chat_template_kwargs`.
+46
View File
@@ -171,6 +171,29 @@ from HuggingFace; and you can find an example of this in a `tokenizer_config.jso
If your favorite tool-calling model is not supported, please feel free to contribute a parser & tool use chat template!
### Checkpoint Response Templates (`hf`)
Checkpoints can define a `response_template` in `tokenizer_config.json` to describe their reasoning, content, and tool-call wire format. Select the
`hf` parser to parse output from this metadata instead of a model-specific parser. Like `--tokenizer-mode hf`, it relies on the checkpoint's
Hugging Face metadata rather than model-specific code:
```bash
vllm serve <model> \
--enable-auto-tool-choice \
--tool-call-parser hf \
--reasoning-parser hf
```
Either parser can be selected on its own. Startup fails if the tokenizer has no `response_template`, or if it lacks the `thinking` or `tool_calls`
field the selected parser needs. The `hf` parser cannot be combined with other reasoning or tool call parsers.
When streaming, each tool call is emitted whole once its region parses. A call cut off before its closer, for example by a closer that is also a
stop token, is parsed from the text generated so far; a call that does not parse is dropped.
Response templates currently provide parsing, not format-specific constrained decoding. The parser therefore rejects requests for strict tools,
required or named tool choice, or `parallel_tool_calls=false`. A custom server or request chat template is allowed, but the parser still expects the
checkpoint's output format.
!!! note
With `tool_choice="auto"`, structural-tag constraints require both `VLLM_ENFORCE_STRICT_TOOL_CALLING=true` (the default) and at least one tool with `strict: true`, or a server-side floor set via `--tool-strict-level`. When these conditions are met and the selected parser supports structural tags, vLLM constrains the tool-call envelope and pins the argument schema of each tool that sets `strict: true` (or of every tool under `--tool-strict-level parameter`). Otherwise, vLLM extracts tool calls from raw text, so arguments may occasionally be malformed or violate the function's parameter schema.
@@ -458,6 +481,29 @@ Supported models:
Flags: `--tool-call-parser qwen3_xml`
### MiMo-V2.6 Models (`mimo`)
Supported models:
* `XiaomiMiMo/MiMo-V2.6-Pro-RL`
* `XiaomiMiMo/MiMo-V2.6-Flash-RL`
Use the checkpoint's chat template with `--tool-call-parser mimo --reasoning-parser mimo --enable-auto-tool-choice`.
For schema-constrained tool arguments, set `strict: true` inside each tool's `function`, or use `--tool-strict-level parameter`.
The parser supports `auto`, `required`, named tools, and `parallel_tool_calls=False`, preserving leading/trailing whitespace in string arguments.
### Step-3.5 and Step-3.7 Models (`step3p5`)
Supported models:
* `stepfun-ai/Step-3.5-Flash`
* `stepfun-ai/Step-3.5-Flash-FP8`
* `stepfun-ai/Step-3.7-Flash`
Flags: `--tool-call-parser step3p5 --reasoning-parser step3p5 --enable-auto-tool-choice`
The parser supports `auto`, `required`, and named tools.
### Olmo 3 Models (`olmo3`)
Olmo 3 models output tool calls in a format that is very similar to the one expected by the `pythonic` parser (see below), with a few differences. Each tool call is a pythonic string, but the parallel tool calls are newline-delimited, and the calls are wrapped within XML tags as `<function_calls>..</function_calls>`. In addition, the parser also allows JSON boolean and null literals (`true`, `false`, and `null`) in addition to the pythonic ones (`True`, `False`, and `None`).
+14 -15
View File
@@ -77,30 +77,17 @@ watermarked. The watermark signal is diluted in proportion to the share of
output tokens supplied by accepted drafts; rejected drafts do not dilute it
because their recovery tokens are watermarked.
Speculative-decoding token paths do not currently support generation-side
context deduplication. The configured `deduplicate_contexts` policy is not
applied to accepted drafts, rejection-recovery tokens, or bonus tokens.
For `dual_key_gumbel`, `alpha` has no effect under speculative decoding. The
speculative protocol selects the key for each token instead.
## Algorithms
### Gumbel-max
Gumbel-max derives a deterministic pseudorandom value from the key, prior token
context, and every candidate token, then uses the resulting Gumbel noise for
categorical sampling. See
[Aaronson's original presentation](https://simons.berkeley.edu/sites/default/files/2024-10/LLM24-2%20Slides%20-%20Scott%20Aaronson.pdf).
Gumbel-max requires stochastic sampling. Greedy requests (`temperature=0`)
bypass watermarking and emit a warning once per worker.
## Context deduplication
When a token context is repeated, generation can use ordinary sampling for that
occurrence. Reusing the repeated context leads to a bias over the sequence, as
certain token choices would be correlated. Ordinary sampling at these positions
allows for single-sequence non-distortion (see section G.3 of the
[SynthID-Text supplementary materials](https://media.springernature.com/original/springer-static/esm/art%3A10.1038%2Fs41586-024-08025-4/MediaObjects/41586_2024_8025_MOESM1_ESM.pdf)).
This also applies to drafter watermarking during speculative decoding.
The detector independently deduplicates contexts so repeated keyed random
vectors are not treated as independent evidence, meaning that context
deduplication at generation time does not reduce the watermarking signal, unless
@@ -143,6 +130,18 @@ vllm serve MODEL \
'{"algorithm":"gumbel","key":42,"deduplicate_contexts":"all"}'
```
## Algorithms
### Gumbel-max
Gumbel-max derives a deterministic pseudorandom value from the key, prior token
context, and every candidate token, then uses the resulting Gumbel noise for
categorical sampling. See
[Aaronson's original presentation](https://simons.berkeley.edu/sites/default/files/2024-10/LLM24-2%20Slides%20-%20Scott%20Aaronson.pdf).
Gumbel-max requires stochastic sampling. Greedy requests (`temperature=0`)
bypass watermarking and emit a warning once per worker.
### Dual-key Gumbel-max
Dual-key Gumbel-max derives independent keys A and B from one configured master