Merge branch 'main' into patches

Signed-off-by: MOVZX <movzx@yahoo.com>
This commit is contained in:
2026-10-09 17:41:13 +00:00
1122 changed files with 60114 additions and 20956 deletions
+3 -4
View File
@@ -126,7 +126,7 @@ Batch invariance has been tested and verified on the following models:
- **Qwen3 (Dense)**: `Qwen/Qwen3-1.7B`, `Qwen/Qwen3-8B`, `Qwen/Qwen3-4B-AWQ`, `Qwen/Qwen3-8B-AWQ`
- **Qwen3-VL (Vision-Language)**: `Qwen/Qwen3-VL-2B-Instruct`, `Qwen/Qwen3-VL-4B-Instruct` (single image and video inputs)
- **Qwen3 (MoE)**: `Qwen/Qwen3-30B-A3B`, `Qwen/Qwen3-Next-80B-A3B-Instruct`, `Qwen/Qwen3-30B-A3B-Thinking-2507-FP8`
- **Qwen2.5**: `Qwen/Qwen2.5-0.5B-Instruct`, `Qwen/Qwen2.5-1.5B-Instruct`, `Qwen/Qwen2.5-3B-Instruct`, `Qwen/Qwen2.5-7B-Instruct`, `Qwen/Qwen2.5-14B-Instruct`, `Qwen/Qwen2.5-32B-Instruct`
- **Qwen2.5**: `Qwen/Qwen2.5-0.5B-Instruct`, `Qwen/Qwen2.5-1.5B-Instruct`, `Qwen/Qwen2.5-3B-Instruct`, `Qwen/Qwen2.5-7B-Instruct`, `Qwen/Qwen2.5-14B-Instruct`, `Qwen/Qwen2.5-32B-Instruct`, `Qwen/Qwen2.5-Coder-7B-Instruct`
- **Llama 3**: Llama3.1 and 3.2 series, `meta-llama/Llama-3.2-1B-Instruct`, `meta-llama/Llama-3.2-3B-Instruct` for example
- **GPT-OSS**: `openai/gpt-oss-20b`, `openai/gpt-oss-120b`
- **Mistral**: `mistralai/Mistral-7B-v0.3`
@@ -148,11 +148,10 @@ When batch invariance is enabled, vLLM:
1. Uses deterministic kernel implementations for attention and other operations
2. Ensures consistent numerical behavior across different batch sizes
3. Disables certain optimizations that may introduce non-determinism (such as sequence parallelism / async TP, whose reduce-scatter path is not batch-invariant)
3. Disables certain optimizations that may introduce non-determinism
4. Under tensor parallelism, keeps custom all-reduce on with a fixed reduction order (the 1-stage kernel is pinned, and large inputs are reduced in fixed-size chunks), and disables FlashInfer, AITER and QuickReduce all-reduce
5. On CUDA devices with tuned matmul table entries for the model's bf16 unquantized forward linear layers (Ada, Hopper, Blackwell),
runs without `torch.compile` using breakable CUDA graphs so tile configs follow the runtime batch size; set `VLLM_USE_BREAKABLE_CUDAGRAPH=0` to opt out
(not applied when sequence parallelism / async TP are enabled, since those are `torch.compile` passes).
runs without `torch.compile` using breakable CUDA graphs so tile configs follow the runtime batch size; set `VLLM_USE_BREAKABLE_CUDAGRAPH=0` to opt out.
6. Splits the LoRA shrink reduction into 8 fixed K chunks that are summed in a fixed order instead of with atomics
!!! warning
+9 -4
View File
@@ -280,10 +280,10 @@ curl -X POST http://localhost:8000/v1/load_lora_adapter \
!!! warning "You must know your adapter's layout"
Under `--enable-mixed-moe-lora-format`, vLLM trusts whatever
`is_3d_lora_weight` the caller declares — it does **not** inspect the
checkpoint to verify. A wrong declaration will load weights into the
wrong stacked buffers and silently produce garbage outputs, with no
error at load time. Confirm the layout before serving:
`is_3d_lora_weight` the caller declares. Loading checks reject fused 3D
expert weights routed to a 2D wrapper without the required flags, but
do not validate every layout mismatch. A wrong declaration can still
cause loading failures or incorrect outputs. Confirm the layout before serving:
- **2D (per-expert, megatron-style)** → set `is_3d_lora_weight: false`.
Adapter keys look like `...experts.{idx}.gate_proj.lora_A.weight`,
@@ -299,6 +299,11 @@ is ignored: vLLM picks the wrapper from the base model's
`is_3d_moe_weight` and the adapter is required to match. The field is
also ignored for non-MoE models.
For a model using a 2D MoE wrapper (such as Qwen3-MoE), loading fused 3D
expert adapter weights requires both `enable_mixed_moe_lora_format=True`
on the engine and `is_3d_lora_weight=True` on the request. Otherwise, adapter
loading raises a `ValueError` naming these settings before activation.
## LoRA model lineage in model card
The new format of `--lora-modules` is mainly to support the display of parent model information in the model card. Here's an explanation of how your current response supports this:
+3 -2
View File
@@ -49,12 +49,13 @@ cache), see [ECMooncakeConnector Usage Guide](mooncake_ec_connector_usage.md).
- For headless instances, must be the same as the master instance
- Each instance needs a unique port on its host; using the same port number across different hosts is fine
- `VLLM_MOONCAKE_CONNECTOR_TIMEOUT`: Per-request timeout (in seconds) for MooncakeConnector HTTP calls to the bootstrap server, including prefiller worker registration at startup. (Optional)
- `VLLM_MOONCAKE_CONNECTOR_TIMEOUT`: Per-request timeout (in seconds) for MooncakeConnector HTTP calls to the bootstrap server, including prefiller worker registration at startup and metadata queries. (Optional)
- Default: 30.0
- Global rank 0 hosts the bootstrap server in the same process that mounts the Mooncake transfer engine's host segment, so a large `MOONCAKE_GLOBAL_SEGMENT_SIZE` can delay responses for several seconds during startup
- Raise this if bootstrap server timeout warnings appear on hosts with very large host-memory segments
- Bootstrap registration uses up to 3 attempts internally, with exponential backoff capped at 10 seconds; connection errors and timeouts are both retried
- Once the attempts are exhausted the caller raises instead of blocking, so failures surface rather than hanging
- Once registration attempts are exhausted the caller raises instead of blocking, so failures surface rather than hanging
- Metadata queries reuse the same timeout and 3-attempt limit, with a fixed 100ms delay between attempts. If all attempts fail, the error is logged and waiting queries are notified
- `WITH_NVIDIA_PEERMEM`: Selects how mooncake registers GPU memory for RDMA. Read by mooncake, not vLLM.
- Default: 1, which uses `ibv_reg_mr()` and requires the `nvidia-peermem` kernel module to be loaded
+8 -1
View File
@@ -1020,6 +1020,13 @@ vllm serve Qwen/Qwen3-VL-30B-A3B-Instruct \
When you extract video frames on the client side and send them as `video/jpeg` (base64-concatenated JPEG frames), you can preserve the original video metadata by using `media_io_kwargs` in your request. This enables more accurate video understanding by preserving temporal information that would otherwise be lost during client-side frame extraction.
!!! warning
Per-request `media_io_kwargs` (and `mm_processor_kwargs`) are rejected by
default because they let a client change media loading and preprocessing
resource usage. Start the server with `--trust-request-mm-kwargs` only when
your API clients are trusted; see
[Per-request multimodal arguments](../usage/security.md#per-request-multimodal-arguments).
**Supported Parameters:**
| Parameter | Type | Description |
@@ -1042,7 +1049,7 @@ When you extract video frames on the client side and send them as `video/jpeg` (
frames_b64 = ",".join([encode_image(f) for f in frames])
video_url = f"data:video/jpeg;base64,{frames_b64}"
# Pass video metadata via media_io_kwargs
# Pass video metadata via media_io_kwargs (server started with --trust-request-mm-kwargs)
response = client.chat.completions.create(
model="your-multimodal-model",
messages=[{
+29
View File
@@ -99,6 +99,35 @@ vllm serve openai/gpt-oss-20b --quantization-config.moe.activation mxfp8
Combine with `--moe-backend` to pin a specific kernel family.
### Re-quantizing layers to a different precision
An online target can re-quantize a checkpoint-quantized (prequantized) layer at load time.
vLLM uses the checkpoint method to load and dequantize the checkpoint weight, and then applies the online quantization method. This requires the checkpoint
method to implement weight dequantization, unsupported source and target
combinations raise an error.
Currently supported combinations are:
| Checkpoint layer | Online target | Requirements |
| ---------------- | ------------- | ------------ |
| Serialized ModelOpt MXFP8 linear | any | Target-specific hardware requirements |
For example, serialized ModelOpt MXFP8 linears can be converted to
per-channel-weight/per-token-activation FP8 at load time:
```bash
vllm serve MiniMaxAI/MiniMax-M3-MXFP8 \
--linear-backend auto \
--quantization-config.linear fp8_per_channel
```
!!! info
This is only available for linear layers at the moment.
Checkpoint-exclusion policy is not enforced by online quantization/re-quantization. Only exclusion through `--quantization-config.targets`/`--quantization-config.ignore` is enforced.
### Online quantization on unquantized layers from partially-quantized checkpoints
Online quantization can be used on already quantized checkpoints independently of their original `quant_method` (`modelopt`, `compressed-tensors`, `quark`, etc.), for layers that are left unquantized in the original checkpoint.
+1 -1
View File
@@ -107,7 +107,7 @@ By default, CUDA graph memory stays on the GPU while asleep. With
`sleep_mode_offload_cudagraph=True` (off by default), CUDA graphs are captured
into a cuMem pool that sleep backs up to CPU memory at both levels and any wake
restores in place, so graphs are replayed, not recaptured. It needs the default
`cumem` backend, CUDA and CUDA graphs; otherwise it has no effect.
`cumem` backend, CUDA or ROCm, and CUDA graphs; otherwise it has no effect.
```python
llm = LLM("Qwen/Qwen3-8B", enable_sleep_mode=True, sleep_mode_offload_cudagraph=True)
+2
View File
@@ -94,6 +94,8 @@ For best results, we recommend ensuring that the expected output format / schema
To use a named function, you need to define the functions in the `tools` parameter of the chat completion request, and
specify the `name` of one of the tools in the `tool_choice` parameter of the chat completion request.
In the Responses API, `tool_choice={"type": "function", "name": "..."}` can select a function inside a namespace by its full `namespace__function` name or by its local name. Names are matched literally first, so a plain function named `other__add` never stands in for `add`. A local name defined in more than one namespace must be selected by its full name.
## Required Function Calling
vLLM supports the `tool_choice='required'` option in the chat completion API. Similar to the named function calling, it also uses structured outputs, so this is enabled by default and will work with any supported model. However, support for alternative decoding backends are on the [roadmap](../usage/v1_guide.md#features) for the V1 engine.
+42 -11
View File
@@ -18,8 +18,16 @@ vllm serve MODEL \
```
Watermarking is disabled when `--watermark-config` is omitted. Gumbel-max (see
`gumbel`) is the default algorithm within an enabled `WatermarkConfig`. When
watermarking is configured, it is enabled for requests by default.
`gumbel`) is the default algorithm within an enabled `WatermarkConfig`.
The request-level `watermarking` option has two effective states:
- Omitted (`None` in Python) or `true` requests watermarking.
- `false` explicitly opts out.
When a request asks for watermarking but the engine or generation mode cannot
apply it, vLLM emits a server-side warning once and runs the request without a
watermark.
Requests can opt out without changing the engine-level algorithm or key:
@@ -30,14 +38,12 @@ sampling_params = SamplingParams(watermarking=False)
```
The Python OpenAI-compatible APIs and the Rust chat/completions APIs accept the
same `watermarking: false` request field. The Rust token API accepts it in
`sampling_params`, and the Rust gRPC `GenerateRequest` accepts
the optional `watermarking` field. Omitting the field keeps watermarking enabled
when the engine has a watermark configuration.
same optional `watermarking` request field. The Rust token API accepts it in
`sampling_params`, and the Rust gRPC `GenerateRequest` accepts it directly.
Omitting the field requests watermarking by default.
Deployments that require watermarking must restrict this field to trusted
callers, or strip and validate it at the ingress boundary, so untrusted clients
cannot opt out.
Deployments that require watermarking must restrict explicit opt-out to trusted
callers, or strip and validate the field at the ingress boundary.
`context_width` controls how many prior tokens seed each watermark decision
and defaults to 4. Larger values make the watermark less robust to
@@ -63,6 +69,31 @@ the reference algorithms. `WatermarkDetector` consumes token IDs, so callers
remain responsible for using the tokenizer and watermark profile that match
generation.
## Interactions with generation features
- Greedy decoding (`temperature=0`) cannot apply a Gumbel watermark. The first
such request emits a server-side warning; it and subsequent greedy requests
use ordinary greedy sampling without a watermark. This includes
transcription, translation, and realtime transcription, which default to
greedy decoding. Transcription and translation can use a nonzero temperature
to apply the configured watermark.
- Beam search cannot apply a watermark. The first watermarked beam request emits
a server-side warning and runs without watermarking.
- Trace replay replaces the sampled token and therefore cannot apply a
watermark. The first watermarked trace replay request emits a server-side
warning and runs without watermarking.
- Parallel sampling (`n > 1`) is supported, but plain Gumbel candidates can be
identical while their contexts remain identical.
- Structured outputs and tool grammars are applied before watermark sampling.
Restrictive constraints can reduce the statistical evidence available to a
detector.
- Automatic language detection and generative scoring use unwatermarked
auxiliary generations. Language-detection tokens are not part of the returned
transcription, and generative scoring returns scores rather than generated
text.
- Speculative decoding has additional requirements and partial-watermarking
modes described in [Speculative decoding](#speculative-decoding).
## Speculative decoding
Watermarking requires speculative decoding to use probabilistic draft sampling,
@@ -245,7 +276,7 @@ watermarked output or to modify watermarked text so it is no longer detected.
- Watermarking is currently available only with Model Runner V2.
- Not all watermarking algorithms have native speculative-decoding support.
- Beam search expands candidates from model log probabilities and does not apply
Gumbel-max watermarking.
- Models that replace the vLLM sampler with a custom sampler cannot use
configured watermarking.
- Global custom logits processors are unavailable because Model Runner V2 does
not support them.