Merge branch 'main' into patches
Signed-off-by: MOVZX <movzx@yahoo.com>
This commit is contained in:
@@ -126,7 +126,7 @@ Batch invariance has been tested and verified on the following models:
|
||||
- **Qwen3 (Dense)**: `Qwen/Qwen3-1.7B`, `Qwen/Qwen3-8B`, `Qwen/Qwen3-4B-AWQ`, `Qwen/Qwen3-8B-AWQ`
|
||||
- **Qwen3-VL (Vision-Language)**: `Qwen/Qwen3-VL-2B-Instruct`, `Qwen/Qwen3-VL-4B-Instruct` (single image and video inputs)
|
||||
- **Qwen3 (MoE)**: `Qwen/Qwen3-30B-A3B`, `Qwen/Qwen3-Next-80B-A3B-Instruct`, `Qwen/Qwen3-30B-A3B-Thinking-2507-FP8`
|
||||
- **Qwen2.5**: `Qwen/Qwen2.5-0.5B-Instruct`, `Qwen/Qwen2.5-1.5B-Instruct`, `Qwen/Qwen2.5-3B-Instruct`, `Qwen/Qwen2.5-7B-Instruct`, `Qwen/Qwen2.5-14B-Instruct`, `Qwen/Qwen2.5-32B-Instruct`
|
||||
- **Qwen2.5**: `Qwen/Qwen2.5-0.5B-Instruct`, `Qwen/Qwen2.5-1.5B-Instruct`, `Qwen/Qwen2.5-3B-Instruct`, `Qwen/Qwen2.5-7B-Instruct`, `Qwen/Qwen2.5-14B-Instruct`, `Qwen/Qwen2.5-32B-Instruct`, `Qwen/Qwen2.5-Coder-7B-Instruct`
|
||||
- **Llama 3**: Llama3.1 and 3.2 series, `meta-llama/Llama-3.2-1B-Instruct`, `meta-llama/Llama-3.2-3B-Instruct` for example
|
||||
- **GPT-OSS**: `openai/gpt-oss-20b`, `openai/gpt-oss-120b`
|
||||
- **Mistral**: `mistralai/Mistral-7B-v0.3`
|
||||
@@ -148,11 +148,10 @@ When batch invariance is enabled, vLLM:
|
||||
|
||||
1. Uses deterministic kernel implementations for attention and other operations
|
||||
2. Ensures consistent numerical behavior across different batch sizes
|
||||
3. Disables certain optimizations that may introduce non-determinism (such as sequence parallelism / async TP, whose reduce-scatter path is not batch-invariant)
|
||||
3. Disables certain optimizations that may introduce non-determinism
|
||||
4. Under tensor parallelism, keeps custom all-reduce on with a fixed reduction order (the 1-stage kernel is pinned, and large inputs are reduced in fixed-size chunks), and disables FlashInfer, AITER and QuickReduce all-reduce
|
||||
5. On CUDA devices with tuned matmul table entries for the model's bf16 unquantized forward linear layers (Ada, Hopper, Blackwell),
|
||||
runs without `torch.compile` using breakable CUDA graphs so tile configs follow the runtime batch size; set `VLLM_USE_BREAKABLE_CUDAGRAPH=0` to opt out
|
||||
(not applied when sequence parallelism / async TP are enabled, since those are `torch.compile` passes).
|
||||
runs without `torch.compile` using breakable CUDA graphs so tile configs follow the runtime batch size; set `VLLM_USE_BREAKABLE_CUDAGRAPH=0` to opt out.
|
||||
6. Splits the LoRA shrink reduction into 8 fixed K chunks that are summed in a fixed order instead of with atomics
|
||||
|
||||
!!! warning
|
||||
|
||||
@@ -280,10 +280,10 @@ curl -X POST http://localhost:8000/v1/load_lora_adapter \
|
||||
|
||||
!!! warning "You must know your adapter's layout"
|
||||
Under `--enable-mixed-moe-lora-format`, vLLM trusts whatever
|
||||
`is_3d_lora_weight` the caller declares — it does **not** inspect the
|
||||
checkpoint to verify. A wrong declaration will load weights into the
|
||||
wrong stacked buffers and silently produce garbage outputs, with no
|
||||
error at load time. Confirm the layout before serving:
|
||||
`is_3d_lora_weight` the caller declares. Loading checks reject fused 3D
|
||||
expert weights routed to a 2D wrapper without the required flags, but
|
||||
do not validate every layout mismatch. A wrong declaration can still
|
||||
cause loading failures or incorrect outputs. Confirm the layout before serving:
|
||||
|
||||
- **2D (per-expert, megatron-style)** → set `is_3d_lora_weight: false`.
|
||||
Adapter keys look like `...experts.{idx}.gate_proj.lora_A.weight`,
|
||||
@@ -299,6 +299,11 @@ is ignored: vLLM picks the wrapper from the base model's
|
||||
`is_3d_moe_weight` and the adapter is required to match. The field is
|
||||
also ignored for non-MoE models.
|
||||
|
||||
For a model using a 2D MoE wrapper (such as Qwen3-MoE), loading fused 3D
|
||||
expert adapter weights requires both `enable_mixed_moe_lora_format=True`
|
||||
on the engine and `is_3d_lora_weight=True` on the request. Otherwise, adapter
|
||||
loading raises a `ValueError` naming these settings before activation.
|
||||
|
||||
## LoRA model lineage in model card
|
||||
|
||||
The new format of `--lora-modules` is mainly to support the display of parent model information in the model card. Here's an explanation of how your current response supports this:
|
||||
|
||||
@@ -49,12 +49,13 @@ cache), see [ECMooncakeConnector Usage Guide](mooncake_ec_connector_usage.md).
|
||||
- For headless instances, must be the same as the master instance
|
||||
- Each instance needs a unique port on its host; using the same port number across different hosts is fine
|
||||
|
||||
- `VLLM_MOONCAKE_CONNECTOR_TIMEOUT`: Per-request timeout (in seconds) for MooncakeConnector HTTP calls to the bootstrap server, including prefiller worker registration at startup. (Optional)
|
||||
- `VLLM_MOONCAKE_CONNECTOR_TIMEOUT`: Per-request timeout (in seconds) for MooncakeConnector HTTP calls to the bootstrap server, including prefiller worker registration at startup and metadata queries. (Optional)
|
||||
- Default: 30.0
|
||||
- Global rank 0 hosts the bootstrap server in the same process that mounts the Mooncake transfer engine's host segment, so a large `MOONCAKE_GLOBAL_SEGMENT_SIZE` can delay responses for several seconds during startup
|
||||
- Raise this if bootstrap server timeout warnings appear on hosts with very large host-memory segments
|
||||
- Bootstrap registration uses up to 3 attempts internally, with exponential backoff capped at 10 seconds; connection errors and timeouts are both retried
|
||||
- Once the attempts are exhausted the caller raises instead of blocking, so failures surface rather than hanging
|
||||
- Once registration attempts are exhausted the caller raises instead of blocking, so failures surface rather than hanging
|
||||
- Metadata queries reuse the same timeout and 3-attempt limit, with a fixed 100ms delay between attempts. If all attempts fail, the error is logged and waiting queries are notified
|
||||
|
||||
- `WITH_NVIDIA_PEERMEM`: Selects how mooncake registers GPU memory for RDMA. Read by mooncake, not vLLM.
|
||||
- Default: 1, which uses `ibv_reg_mr()` and requires the `nvidia-peermem` kernel module to be loaded
|
||||
|
||||
@@ -1020,6 +1020,13 @@ vllm serve Qwen/Qwen3-VL-30B-A3B-Instruct \
|
||||
|
||||
When you extract video frames on the client side and send them as `video/jpeg` (base64-concatenated JPEG frames), you can preserve the original video metadata by using `media_io_kwargs` in your request. This enables more accurate video understanding by preserving temporal information that would otherwise be lost during client-side frame extraction.
|
||||
|
||||
!!! warning
|
||||
Per-request `media_io_kwargs` (and `mm_processor_kwargs`) are rejected by
|
||||
default because they let a client change media loading and preprocessing
|
||||
resource usage. Start the server with `--trust-request-mm-kwargs` only when
|
||||
your API clients are trusted; see
|
||||
[Per-request multimodal arguments](../usage/security.md#per-request-multimodal-arguments).
|
||||
|
||||
**Supported Parameters:**
|
||||
|
||||
| Parameter | Type | Description |
|
||||
@@ -1042,7 +1049,7 @@ When you extract video frames on the client side and send them as `video/jpeg` (
|
||||
frames_b64 = ",".join([encode_image(f) for f in frames])
|
||||
video_url = f"data:video/jpeg;base64,{frames_b64}"
|
||||
|
||||
# Pass video metadata via media_io_kwargs
|
||||
# Pass video metadata via media_io_kwargs (server started with --trust-request-mm-kwargs)
|
||||
response = client.chat.completions.create(
|
||||
model="your-multimodal-model",
|
||||
messages=[{
|
||||
|
||||
@@ -99,6 +99,35 @@ vllm serve openai/gpt-oss-20b --quantization-config.moe.activation mxfp8
|
||||
|
||||
Combine with `--moe-backend` to pin a specific kernel family.
|
||||
|
||||
### Re-quantizing layers to a different precision
|
||||
|
||||
An online target can re-quantize a checkpoint-quantized (prequantized) layer at load time.
|
||||
|
||||
vLLM uses the checkpoint method to load and dequantize the checkpoint weight, and then applies the online quantization method. This requires the checkpoint
|
||||
method to implement weight dequantization, unsupported source and target
|
||||
combinations raise an error.
|
||||
|
||||
Currently supported combinations are:
|
||||
|
||||
| Checkpoint layer | Online target | Requirements |
|
||||
| ---------------- | ------------- | ------------ |
|
||||
| Serialized ModelOpt MXFP8 linear | any | Target-specific hardware requirements |
|
||||
|
||||
For example, serialized ModelOpt MXFP8 linears can be converted to
|
||||
per-channel-weight/per-token-activation FP8 at load time:
|
||||
|
||||
```bash
|
||||
vllm serve MiniMaxAI/MiniMax-M3-MXFP8 \
|
||||
--linear-backend auto \
|
||||
--quantization-config.linear fp8_per_channel
|
||||
```
|
||||
|
||||
!!! info
|
||||
|
||||
This is only available for linear layers at the moment.
|
||||
|
||||
Checkpoint-exclusion policy is not enforced by online quantization/re-quantization. Only exclusion through `--quantization-config.targets`/`--quantization-config.ignore` is enforced.
|
||||
|
||||
### Online quantization on unquantized layers from partially-quantized checkpoints
|
||||
|
||||
Online quantization can be used on already quantized checkpoints independently of their original `quant_method` (`modelopt`, `compressed-tensors`, `quark`, etc.), for layers that are left unquantized in the original checkpoint.
|
||||
|
||||
@@ -107,7 +107,7 @@ By default, CUDA graph memory stays on the GPU while asleep. With
|
||||
`sleep_mode_offload_cudagraph=True` (off by default), CUDA graphs are captured
|
||||
into a cuMem pool that sleep backs up to CPU memory at both levels and any wake
|
||||
restores in place, so graphs are replayed, not recaptured. It needs the default
|
||||
`cumem` backend, CUDA and CUDA graphs; otherwise it has no effect.
|
||||
`cumem` backend, CUDA or ROCm, and CUDA graphs; otherwise it has no effect.
|
||||
|
||||
```python
|
||||
llm = LLM("Qwen/Qwen3-8B", enable_sleep_mode=True, sleep_mode_offload_cudagraph=True)
|
||||
|
||||
@@ -94,6 +94,8 @@ For best results, we recommend ensuring that the expected output format / schema
|
||||
To use a named function, you need to define the functions in the `tools` parameter of the chat completion request, and
|
||||
specify the `name` of one of the tools in the `tool_choice` parameter of the chat completion request.
|
||||
|
||||
In the Responses API, `tool_choice={"type": "function", "name": "..."}` can select a function inside a namespace by its full `namespace__function` name or by its local name. Names are matched literally first, so a plain function named `other__add` never stands in for `add`. A local name defined in more than one namespace must be selected by its full name.
|
||||
|
||||
## Required Function Calling
|
||||
|
||||
vLLM supports the `tool_choice='required'` option in the chat completion API. Similar to the named function calling, it also uses structured outputs, so this is enabled by default and will work with any supported model. However, support for alternative decoding backends are on the [roadmap](../usage/v1_guide.md#features) for the V1 engine.
|
||||
|
||||
@@ -18,8 +18,16 @@ vllm serve MODEL \
|
||||
```
|
||||
|
||||
Watermarking is disabled when `--watermark-config` is omitted. Gumbel-max (see
|
||||
`gumbel`) is the default algorithm within an enabled `WatermarkConfig`. When
|
||||
watermarking is configured, it is enabled for requests by default.
|
||||
`gumbel`) is the default algorithm within an enabled `WatermarkConfig`.
|
||||
|
||||
The request-level `watermarking` option has two effective states:
|
||||
|
||||
- Omitted (`None` in Python) or `true` requests watermarking.
|
||||
- `false` explicitly opts out.
|
||||
|
||||
When a request asks for watermarking but the engine or generation mode cannot
|
||||
apply it, vLLM emits a server-side warning once and runs the request without a
|
||||
watermark.
|
||||
|
||||
Requests can opt out without changing the engine-level algorithm or key:
|
||||
|
||||
@@ -30,14 +38,12 @@ sampling_params = SamplingParams(watermarking=False)
|
||||
```
|
||||
|
||||
The Python OpenAI-compatible APIs and the Rust chat/completions APIs accept the
|
||||
same `watermarking: false` request field. The Rust token API accepts it in
|
||||
`sampling_params`, and the Rust gRPC `GenerateRequest` accepts
|
||||
the optional `watermarking` field. Omitting the field keeps watermarking enabled
|
||||
when the engine has a watermark configuration.
|
||||
same optional `watermarking` request field. The Rust token API accepts it in
|
||||
`sampling_params`, and the Rust gRPC `GenerateRequest` accepts it directly.
|
||||
Omitting the field requests watermarking by default.
|
||||
|
||||
Deployments that require watermarking must restrict this field to trusted
|
||||
callers, or strip and validate it at the ingress boundary, so untrusted clients
|
||||
cannot opt out.
|
||||
Deployments that require watermarking must restrict explicit opt-out to trusted
|
||||
callers, or strip and validate the field at the ingress boundary.
|
||||
|
||||
`context_width` controls how many prior tokens seed each watermark decision
|
||||
and defaults to 4. Larger values make the watermark less robust to
|
||||
@@ -63,6 +69,31 @@ the reference algorithms. `WatermarkDetector` consumes token IDs, so callers
|
||||
remain responsible for using the tokenizer and watermark profile that match
|
||||
generation.
|
||||
|
||||
## Interactions with generation features
|
||||
|
||||
- Greedy decoding (`temperature=0`) cannot apply a Gumbel watermark. The first
|
||||
such request emits a server-side warning; it and subsequent greedy requests
|
||||
use ordinary greedy sampling without a watermark. This includes
|
||||
transcription, translation, and realtime transcription, which default to
|
||||
greedy decoding. Transcription and translation can use a nonzero temperature
|
||||
to apply the configured watermark.
|
||||
- Beam search cannot apply a watermark. The first watermarked beam request emits
|
||||
a server-side warning and runs without watermarking.
|
||||
- Trace replay replaces the sampled token and therefore cannot apply a
|
||||
watermark. The first watermarked trace replay request emits a server-side
|
||||
warning and runs without watermarking.
|
||||
- Parallel sampling (`n > 1`) is supported, but plain Gumbel candidates can be
|
||||
identical while their contexts remain identical.
|
||||
- Structured outputs and tool grammars are applied before watermark sampling.
|
||||
Restrictive constraints can reduce the statistical evidence available to a
|
||||
detector.
|
||||
- Automatic language detection and generative scoring use unwatermarked
|
||||
auxiliary generations. Language-detection tokens are not part of the returned
|
||||
transcription, and generative scoring returns scores rather than generated
|
||||
text.
|
||||
- Speculative decoding has additional requirements and partial-watermarking
|
||||
modes described in [Speculative decoding](#speculative-decoding).
|
||||
|
||||
## Speculative decoding
|
||||
|
||||
Watermarking requires speculative decoding to use probabilistic draft sampling,
|
||||
@@ -245,7 +276,7 @@ watermarked output or to modify watermarked text so it is no longer detected.
|
||||
|
||||
- Watermarking is currently available only with Model Runner V2.
|
||||
- Not all watermarking algorithms have native speculative-decoding support.
|
||||
- Beam search expands candidates from model log probabilities and does not apply
|
||||
Gumbel-max watermarking.
|
||||
- Models that replace the vLLM sampler with a custom sampler cannot use
|
||||
configured watermarking.
|
||||
- Global custom logits processors are unavailable because Model Runner V2 does
|
||||
not support them.
|
||||
|
||||
Reference in New Issue
Block a user