Merge branch 'main' into patches

Signed-off-by: MOVZX <movzx@yahoo.com>
This commit is contained in:
2026-10-15 21:16:39 +07:00
1693 changed files with 99919 additions and 32440 deletions
+3 -4
View File
@@ -15,23 +15,22 @@ Set `enable_prefix_caching=True` in vLLM engine to enable APC. Here is an exampl
## Hybrid Mamba models
Under `--mamba-cache-mode align`, Mamba state is stored only on the Mamba block grid, so a prefix-cache hit can resume only at a block boundary. `--enable-mamba-shared-prefix-checkpoint` also stores a checkpoint at the shared-prefix junction, the point where an earlier request with the same prefix stopped. Requests whose shared prefix ends inside a block can then reuse it.
With prefix caching enabled, Mamba state is stored only on the Mamba block grid, so a prefix-cache hit can resume only at a block boundary. `--enable-mamba-shared-prefix-checkpoint` also stores a checkpoint at the shared-prefix junction, the point where an earlier request with the same prefix stopped. Requests whose shared prefix ends inside a block can then reuse it.
This helps when many requests share a long system prompt and then diverge. It is off by default, and takes effect only when all of the following hold:
- `--mamba-cache-mode align`
- prefix caching enabled
- EAGLE/MTP speculative decoding on the Mamba group
- `--prefix-match-unit` smaller than the Mamba block size
- the model does not use multi-module MTP
```bash
vllm serve <hybrid-model> \
--mamba-cache-mode align \
--prefix-match-unit 64 \
--enable-mamba-shared-prefix-checkpoint
```
`--prefix-match-unit` is required. It sets the granularity at which prefix-cache keys are computed. When unset it defaults to the greatest common divisor of the prefix-cacheable KV cache group block sizes. Under `align` that is the block size itself, so no sub-block boundary exists and the flag has no effect.
`--prefix-match-unit` is required. It sets the granularity at which prefix-cache keys are computed. When unset it defaults to the greatest common divisor of the prefix-cacheable KV cache group block sizes. For hybrid Mamba models that is the block size itself, so no sub-block boundary exists and the flag has no effect.
Choose a value that divides the block size of every prefix-cacheable KV cache group, and that is a multiple of the per-state compression ratio for models that use one, such as sparse MLA. vLLM validates both at startup and names the offending sizes in the error. Read the served block size from the startup log. 64 is a reasonable starting point.
+1
View File
@@ -20,6 +20,7 @@ Batch invariance is crucial for several use cases:
Batch invariance is supported on the following platforms:
- NVIDIA GPUs with compute capability 8.0 or higher.
- AMD Instinct GPUs (gfx90a, gfx942, gfx950).
- Intel XPUs with Triton support.
### Attention Backend Selection for XPU
+1 -1
View File
@@ -96,7 +96,7 @@ EC transfer is configured via `--ec-transfer-config` (CLI) or the `ec_transfer_c
| --- | --- | --- | --- |
| `ec_enable_nixl` | `bool` | No (default `false`) | Enables NIXL P2P transfer in addition to local CPU offload. Omitted or `false` imports no NIXL/ZMQ. Extra config is not type coerced, so a string value is parsed: `"true"`, `"1"`, `"yes"` enable it, anything else does not. |
| `consumer_ack_timeout_s` | `float` | No (default `2.0`) | How long a consumer waits for an `XferAck` before giving up on a read. The producer answers `XferReq`s from its own scheduler step, so its reply latency scales with the encoder's `--max-num-batched-tokens`: a loaded encoder whose steps run longer than this makes consumers abandon reads the producer is about to grant. Raise it for large encoder batches. |
| `ec_cpu_bytes` | `int` | Yes | Total size, in bytes, of the shared CPU mmap region. `ECCPUConnector` raises `ValueError` if unset. Block count = `ec_cpu_bytes // block_size_bytes`, where `block_size_bytes = hidden_dim * dtype.element_size()` (`hidden_dim` accounts for Qwen3-VL deepstack: `out_hidden_size * (1 + num_deepstack_layers)`). |
| `ec_cpu_bytes` | `int` | Yes | Total size, in bytes, of the shared CPU mmap region. `ECCPUConnector` raises `ValueError` if unset. Block count = `ec_cpu_bytes // block_size_bytes`, where each block holds one LLM input-embedding vector. Entries allocate enough blocks for their modality-specific output shape, including visual DeepStack features. |
### Environment Variables
+31
View File
@@ -108,10 +108,28 @@ vllm serve <model> \
| `store_threshold` | no | `0` | single-tier | Min lookups before a block is offloaded. Values ≥ 2 are rejected by `TieringOffloadingSpec`. |
| `max_tracker_size` | no | `64000` | single-tier | Max entries in the lookup tracker. |
| `secondary_tiers` | no | `[]` | multi-tier | List of secondary tier configs (see below). |
| `tier_poll_interval_s` | no | `0.001` | multi-tier | Pause between tier-poll rounds for tiers that must be serviced between engine steps (currently `p2p`). See [Threading model](#threading-model). Set to `0` to disable the thread and service those tiers once per engine step instead. |
| `offload_prompt_only` | no | `true` | both | If `true`, only prompt (prefill) blocks are offloaded; decode blocks are skipped. |
| `self_describing_kv_events` | no | `false` | both | Opt-in. When `true` *and* KV cache events are enabled (`--kv-events-config` with `enable_kv_cache_events`), the connector emits self-describing block-granular `BlockStored`/`BlockRemoved` payloads (constituent block hashes, whole-chunk `token_ids`, per-block `block_size`, parent hash, LoRA + group/cache-spec metadata) instead of the placeholder fallback, so external KV-event consumers can index offloaded blocks. Inert unless events are enabled. With `TieringOffloadingSpec`, a CPU promotion is self-describing when a local request observes its primary-tier `HIT` before event translation; otherwise its stored event may retain the placeholder, while a later `HIT` can backfill metadata for removal. Pending-removal/re-promotion races and externally initiated promotions may also produce placeholders, and consumers must ignore removals for unknown hashes. Partial recurrent tails emit the hash-aligned portion from the physical block start through the tail boundary. Other sliding-window/SSM chunks keep the placeholder fallback. In chunk mode (`block_size` > GPU block size, or `blocks_per_chunk` > 1), overlapping chunks re-announce shared per-block hashes, so consumers must reference-count (deduplicate) repeated store/remove announcements. |
| `spec_module_path` | no | — | both | Python import path for a custom `OffloadingSpec` not in the built-in registry. Required only when `spec_name` is not built-in (advanced). |
## Info Metric Labels
The connector publishes one Prometheus gauge, `vllm:kv_offload_config_info`, and always sets it to 1. The labels hold static facts of the offloading configuration, so a user reads the resolved numbers instead of the launch flags. The gauge appears from the first scheduler step, so an idle engine exposes no series. Each engine reports its own configuration, and not the instance total.
The gauge holds one series for each configuration that the manager reports. A manager that holds one configuration gives one series. A manager that holds several adds a label of its own, which tells the series apart.
| Label | Declared by | Meaning | Notes |
| --- | --- | --- | --- |
| `tier` | `TieringOffloadingSpec` | The tier that this series reports. | `<index>:<type>`. Index `0` is the CPU primary tier, and a secondary tier index starts at 1. `<type>` is the `type` key of the tier config, such as `1:fs`. |
| `cpu_num_chunks` | `CPUOffloadingSpec` or `TieringOffloadingSpec` | Chunk slots in the CPU tier. | Chunks, not GPU blocks. `cpu_bytes_to_use` divided by the bytes of one chunk. |
The spec declares the label names in the API server process, and the manager fills the values in the engine process. The names of every series bind once, at the declaration. A label that one series owns reads empty on every other series.
`CPUOffloadingSpec` publishes `cpu_num_chunks` on its one series. `TieringOffloadingSpec` publishes it on the `0:primary` series. For a worked example of a secondary tier, the `example` tier publishes one label, `example_info`.
A manager adds a label with `config_info_keys()` of its spec and `config_info()` of its own class. A name that a manager fills and the spec does not declare is dropped, and the engine log then holds one warning line.
## Custom Eviction Policies
`eviction_policy` resolves through `CachePolicyFactory` (`vllm/v1/kv_offload/cpu/policies/factory.py`), which pre-registers the built-in `lru` and `arc` policies.
@@ -150,6 +168,19 @@ The filesystem and object-store tiers can publish hash-only `BlockStored` KV eve
Set the optional `locality` tier field to `LOCAL` or `REMOTE` to describe the tier's storage location relative to the publishing vLLM instance. `LOCAL` marks storage local to that instance, while `REMOTE` marks storage that is not local to it. When the setting is omitted, locality is unspecified. vLLM does not infer it from the tier type, so an `obj` tier is not implicitly `REMOTE`. A KV event includes `locality` only when the tier explicitly configures it. This metadata describes the tier property without implying that a consumer can already route requests to its blocks.
### Threading model
Tier methods run in the scheduler process, under the tiering manager's lock, so they are never entered concurrently. They are not always called from the same thread, though.
Most tiers only ever see the scheduler thread, which holds the lock for the length of one engine step. A tier whose counterpart is not driven by the engine — the `p2p` tier, answering a remote peer — needs servicing while the engine is busy running the model, because otherwise a peer's lookup or fetch waits for the next step boundary. On a saturated rank that wait can be seconds, dwarfing the transfer it gates. Such tiers set `tolerates_off_thread_polling`, and the tiering manager runs one tier-poll thread that, once per `tier_poll_interval_s`, polls them for finished jobs and lets them serve external requests. A tier that implements `serve_external_requests()` without setting the flag is served once per engine step only.
Two consequences worth knowing:
- Any tier can be reached from that thread indirectly, because serving a peer looks keys up through the tiering manager, which fans out to the other tiers. Tier state must therefore not assume a particular thread, only that the manager lock is held.
- Work the thread deliberately leaves alone is batched per step: a promotion it initiates is submitted at the next `on_schedule_end()`, not immediately.
Setting `tier_poll_interval_s` to `0` disables the thread; opted-in tiers then get serviced once per engine step, as they were before it existed.
### Filesystem (FS)
The filesystem tier (`type: "fs"`) writes blocks to a filesystem directory.
+2 -2
View File
@@ -52,7 +52,7 @@ CUDA_VISIBLE_DEVICES=0 vllm serve Qwen/Qwen2.5-VL-3B-Instruct \
--port 19534 \
--gpu-memory-utilization 0.35 \
--enable-request-id-headers \
--no-enable-prefix-caching \
--disable-prefix-caching \
--max-num-batched-tokens 16384 \
--allowed-local-media-path /path/to/media \
--ec-transfer-config '{
@@ -72,7 +72,7 @@ CUDA_VISIBLE_DEVICES=1 vllm serve Qwen/Qwen2.5-VL-3B-Instruct \
--gpu-memory-utilization 0.75 \
--enable-mm-embeds \
--enable-request-id-headers \
--no-enable-prefix-caching \
--disable-prefix-caching \
--allowed-local-media-path /path/to/media \
--ec-transfer-config '{
"ec_connector": "ECMooncakeConnector",
+1 -1
View File
@@ -398,7 +398,7 @@ The `kv_load_failure_policy` setting controls how the system handles failures wh
### For NVIDIA GB-series GPUs
GB-series GPUs support multi-node NVLink. NIXL supports this capability, but KVCache must be registered as VMM during KVCache registration. To enable this feature, you need to set `--enable-cumem-allocator` or `--enable-sleep-mode` flags, and set `UCX_CUDA_IPC_ENABLE_MNNVL: 'y'` env. Otherwise, NIXL can only use RDMA/TCP for cross-node KVCache transfers.
GB-series GPUs support multi-node NVLink. NIXL supports this capability, but KVCache must be registered as VMM during KVCache registration. To enable this feature, you need to set the `--enable-cumem-allocator` flag, and set `UCX_CUDA_IPC_ENABLE_MNNVL: 'y'` env. Otherwise, NIXL can only use RDMA/TCP for cross-node KVCache transfers.
## Experimental Feature
+28 -2
View File
@@ -50,6 +50,26 @@ vllm serve meta-llama/Llama-3.1-8B-Instruct \
The daemon itself must load from disk; passing `--load-format ipc_cache` to
`vllm preload` is an error.
## Preloading the FlashInfer autotune cache
FlashInfer has several implementations of each operation and chooses between
them by benchmarking. vLLM runs that pass during kernel warmup and persists the
result per rank in the on-disk FlashInfer autotune cache (under
`VLLM_CACHE_ROOT`, or `VLLM_FLASHINFER_AUTOTUNE_CACHE_DIR`), so a later engine
with the same configuration loads the tuned configs instead of profiling again.
Normally the first serving engine pays for the pass, and on a large MoE model it
dominates what is left of the startup time once the weights come from the
daemon.
Unless FlashInfer autotune is disabled (`--disable-flashinfer-autotune`),
`vllm preload` pays it at preload time: each daemon runs the autotune pass in
place — profiling dummy runs against the shard it already holds — after
loading and before binding its socket, then writes the cache:
```bash
vllm preload --model /path/to/model --tensor-parallel-size 4
```
## How it works
1. `vllm preload` spawns one daemon process per GPU. Each daemon loads its
@@ -164,8 +184,14 @@ Socket paths are derived from the GPU UUID, so they are stable regardless of
- **Platform**: CUDA and ROCm only; other platforms raise
`UnsupportedPlatformForIPCError` even when `fallback` is enabled, since it
is a permanent misconfiguration rather than a transient daemon outage.
- **Parallelism**: tensor, expert and data parallelism are supported;
launching the daemon with pipeline parallelism is rejected.
- **Parallelism**: tensor, expert, pipeline and data parallelism are
supported, including across nodes.
- **Autotune preloading**: tuning runs across the daemon's world group,
including across nodes and DP ranks. With model-based speculative decoding
both daemon groups skip it (the target cannot run the drafter's dummy
passes, and a table tuned on the draft model would key on the draft's
config, which no engine looks up), so those deployments' first engine
tunes and fills the cache instead.
- **Quantization**: every quantization method in the model must declare
support for pre-processed weights (the daemon transfers weights *after*
quantization post-processing). Unsupported methods raise
+31 -6
View File
@@ -17,7 +17,7 @@ vllm serve <model> \
--moe-backend b12x
```
Only pass `--moe-backend b12x` for a compatible NVFP4 or MXFP4 MoE model. The
Only pass `--moe-backend b12x` for a compatible packed-weight MoE model. The
linear and MoE backends can be selected independently.
b12x uses MXFP8 activations by default for MXFP4 MoE and the checkpoint's
@@ -29,9 +29,34 @@ force BF16 activations for either FP4 weight format.
| Backend | Supported configurations |
| ------- | ------------------------ |
| Linear | Per-tensor FP8, 128x128 block FP8, MXFP8, NVFP4, and MXFP4 |
| MoE | Tensor-parallel MXFP4 weights with BF16 or MXFP8 activations; NVFP4 weights with BF16, NVFP4, or MXFP8 activations |
| Linear | Per-tensor FP8, 128x128 block FP8, MXFP8, NVFP4 (including W4A16), MXFP4, IQ2_XS, IQ2_XXS, and Q8_0 |
| MoE | Tensor-parallel MXFP4 weights with BF16 or MXFP8 activations; NVFP4 weights with BF16, NVFP4, or MXFP8 activations; IQ2_XS, IQ2_XXS, and Q8_0 weights with BF16 activations |
| Embedding | Packed Q8_0 lookup with BF16 output |
Dense W4A16 layers are not handled by b12x and continue to use another
compatible backend such as Marlin. The b12x MoE backend does not support expert
parallelism, expert maps, EXL3, or NF3.
The b12x MoE backend does not support expert parallelism, expert maps, EXL3, or NF3.
## ModelOpt block codecs
ModelOpt mixed-precision checkpoints select block codecs through the
`quantization_config.quantized_layers` entries in `config.json`. Each entry
declares `quant_algo`, `group_size`, `block_payload_bytes`, and `packing="ggml"`:
| `quant_algo` | `group_size` | `block_payload_bytes` |
| --- | --- | --- |
| `IQ2_XS` | 256 | 74 |
| `IQ2_XXS` | 256 | 66 |
| `Q8_0` | 32 | 34 |
Serialized matrices have uint8 shape `[N, K/group_size, block_payload_bytes]`.
Dense preparation rearranges the encoded blocks without expanding the matrix
to BF16. Routed experts use bias-free SiLU or non-gated ReLU²; tensor-parallel
shards must preserve whole blocks and satisfy the kernel's alignment constraints.
Only Q8_0 supports packed embedding lookup.
Quantized normalization weights and other packed vectors are not supported.
Store normalization weights as ordinary floating-point tensors and omit their
entries from `quantization_config.quantized_layers`.
Dense, embedding, and expert matrices remain packed. Their execution uses BF16
activations without activation quantization. Prepare the kernels and workspaces
before CUDA graph capture.
+14 -3
View File
@@ -111,7 +111,10 @@ Currently supported combinations are:
| Checkpoint layer | Online target | Requirements |
| ---------------- | ------------- | ------------ |
| Serialized ModelOpt MXFP8 linear | any | Target-specific hardware requirements |
| Serialized ModelOpt MXFP8 linear | any online linear method | Target-specific hardware requirements |
| Serialized compressed-tensors NVFP4 MoE | any online MoE method | Target-specific hardware requirements |
| Serialized ModelOpt NVFP4 MoE | any online MoE method | Target-specific hardware requirements |
| Serialized Quark NVFP4 MoE | any online MoE method | Target-specific hardware requirements |
For example, serialized ModelOpt MXFP8 linears can be converted to
per-channel-weight/per-token-activation FP8 at load time:
@@ -122,10 +125,18 @@ vllm serve MiniMaxAI/MiniMax-M3-MXFP8 \
--quantization-config.linear fp8_per_channel
```
Similarly, for example, NVFP4 MoE checkpoints can be re-quantized to any online MoE method,
for example compressed-tensors NVFP4 experts to online MXFP4:
```bash
vllm serve RedHatAI/Qwen3-30B-A3B-NVFP4 \
--quantization-config '{"targets":{"*mlp.experts*":"mxfp4"}}'
```
!!! info
This is only available for linear layers at the moment.
Re-quantization is available for both linear and MoE layers.
Checkpoint-exclusion policy is not enforced by online quantization/re-quantization. Only exclusion through `--quantization-config.targets`/`--quantization-config.ignore` is enforced.
### Online quantization on unquantized layers from partially-quantized checkpoints
+36 -3
View File
@@ -77,8 +77,15 @@ llm.wake_up(tags=["kv_cache"])
#### Retaining frozen weights during RLHF updates
Set `sleep_preserve_parameter_names` (CLI: `--sleep-preserve-parameter-names`)
to runtime parameter-name glob
patterns for weights that stay frozen during training. Each pattern must match
to runtime parameter-name glob patterns for weights that stay frozen during
training. It covers two use cases:
- **Frozen Engram tables (original use case):** retain table weights and scales
omitted by the trainer, without retransmitting them after every sleep.
- **Frozen draft weights:** retain draft-private parameters when the trainer
updates only the target model.
Target patterns match
`model.named_parameters()`; checkpoint names and `requires_grad` are not used.
Level-2 sleep backs up selected GPU parameters to pageable CPU memory; weights
@@ -87,7 +94,33 @@ copy. Allow enough host memory for backups. Level-1 behavior is unchanged.
The trainer must omit these parameters from updates, and reload/post-processing
must preserve their values and storage (including avoiding replacement with meta
tensors). This option does not filter updates and applies only to the target model.
tensors). This option does not filter updates.
For frozen Engram tables omitted by the trainer, select the target table weights
and scales. For example, DeepSeek V4.1 uses these runtime parameter names:
```bash
VLLM_SERVER_DEV_MODE=1 vllm serve deepseek-ai/DeepSeek-V4.1-Flash --enable-sleep-mode \
--sleep-preserve-parameter-names '*.engram.embed_tokens.weight*'
```
CPU-offloaded tables remain on CPU without an extra backup copy. Engram's
[inference-time CPU offload](engram.md#cpu-offload) is configured separately;
this option preserves the frozen tables across level-2 sleep and weight updates.
To retain a frozen drafter while updating only target weights, use `draft.*`:
```bash
VLLM_SERVER_DEV_MODE=1 vllm serve Qwen/Qwen3.8-27B --enable-sleep-mode \
--speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
--sleep-preserve-parameter-names 'draft.*'
```
vLLM selects all draft-private parameters internally; you do not need to list
them individually. Use `draft.<name>` to select specific draft parameters.
Parameters shared with the target are excluded from the draft selection, so
target updates remain authoritative. An empty `draft.*` selection is valid.
The Engram and draft patterns can be supplied together.
#### Release only KV cache memory
+1 -3
View File
@@ -6,14 +6,13 @@ To train your own draft models for optimized speculative decoding, see [vllm-pro
## vLLM Speculation Methods
vLLM supports a variety of methods of speculative decoding. Model-based methods such as EAGLE, MTP, draft models, PARD and MLP provide the best latency reduction, while simpler methods such as n-gram and suffix decoding provide modest speedups without increasing workload during peak traffic.
vLLM supports a variety of methods of speculative decoding. Model-based methods such as EAGLE, MTP, draft models and PARD provide the best latency reduction, while simpler methods such as n-gram and suffix decoding provide modest speedups without increasing workload during peak traffic.
- [EAGLE](eagle.md)
- [LiLiCorr](lilicorr.md)
- [Multi-Token Prediction (MTP)](mtp.md)
- [Draft Model](draft_model.md)
- [Parallel Draft Model (PARD)](parallel_draft_model.md)
- [Multi-Layer Perceptron](mlp.md)
- [N-Gram](n_gram.md)
- [Suffix Decoding](suffix.md)
- [Hidden State Extraction](extract_hidden_states.md)
@@ -33,7 +32,6 @@ depend on your model family, traffic pattern, hardware, and sampling settings.
| MTP | High gain | Medium to high gain | Best when the target model has native MTP support. |
| Draft model | High gain | Medium gain | Needs a separate draft model. |
| Parallel Draft Model | High gain | Medium to high gain | Low draft model latency. |
| MLP speculator | Medium to high gain | Medium gain | Good when compatible MLP speculators are available. |
| N-gram | Low to medium gain | Medium gain | Lightweight and easy to enable. |
| Suffix decoding | Low to medium gain | Medium gain | No extra draft model; dynamic speculation depth. |
| Custom Proposer | Varies | Varies | Bring your own proposer class (experimental). |
@@ -4,7 +4,7 @@ Speculative decoding buys fewer decode steps with more compute. At batch size 1
That matters because per-position acceptance decays fast. While the GPU is memory-bound that slot is effectively free and worth the gamble; once it saturates the gamble has a real throughput cost. The crossover moves with load and with workload-dependent acceptance rates, so no static `num_speculative_tokens` is right across concurrencies.
Adaptive verification decides per step how much of the draft to verify instead. Every (request, position) draft slot is scored by its *survival probability*, the running product of that request's per-position confidences, and the highest-scoring slots are admitted until a global budget is spent. Slots compete across requests: position 5 of a confident request can outrank position 1 of a doubtful one, so one request keeps its full block while another could be trimmed after a token or two.
Adaptive verification decides per step how much of the draft to verify instead. Every (request, position) draft slot is scored by its *survival probability*, the running product of that request's per-position acceptance estimates, and the highest-scoring slots are admitted until a global budget is spent. Slots compete across requests: position 5 of a confident request can outrank position 1 of a doubtful one, so one request keeps its full block while another could be trimmed after a token or two.
The budget itself comes from a cost model profiled at startup. vLLM measures what a step costs at each shape, then picks the token count that maximizes expected accepted tokens per second.
@@ -12,11 +12,14 @@ The practical effect is that one configuration holds up across the whole load ra
## Support
Adaptive verification needs per-position acceptance estimates, so today it is only supported for DSpark with a **confidence head**.
Adaptive verification is supported for model-based drafters (EAGLE, MTP, DFlash, DSpark and standalone draft models). The per-position acceptance estimates come from one of two sources:
- **Confidence head**: DSpark checkpoints that ship a trained confidence head use its predictions directly.
- **Online acceptance estimator**: every other drafter predicts acceptance from its draft logits with a small logistic model, recalibrated during serving against the target's verification results.
## Usage
It is off by default. Enable it in the speculative config:
It is off by default. Enable it in the speculative config with `enable_adaptive_verification`. For example, with DSpark:
```bash
vllm serve deepseek-ai/DeepSeek-V4-Flash-DSpark \
@@ -30,6 +33,21 @@ vllm serve deepseek-ai/DeepSeek-V4-Flash-DSpark \
}'
```
Or with DFlash on MiMo-V2.5-Pro, whose drafter ships in the checkpoint's `dflash/` subdirectory:
```bash
hf download XiaomiMiMo/MiMo-V2.5-Pro-FP4-DFlash --local-dir MiMo-V2.5-Pro-FP4-DFlash
vllm serve MiMo-V2.5-Pro-FP4-DFlash \
--tensor-parallel-size 4 --trust-remote-code --max-model-len auto \
--speculative-config '{
"method": "dflash",
"model": "MiMo-V2.5-Pro-FP4-DFlash/dflash",
"num_speculative_tokens": 8,
"enable_adaptive_verification": true
}'
```
Set `enable_adaptive_verification: false` to verify the full block for every request.
## Requirements and limitations
@@ -37,6 +55,7 @@ Set `enable_adaptive_verification: false` to verify the full block for every req
- The attention backend must tolerate device-decided query lengths, since the CPU lengths only bound them from above. Backends that plan off the CPU lengths are excluded by the attention selector, and rejected at startup for models that hard-wire their backend.
- Full cudagraphs are required: step costs are profiled from captured graphs, so `--enforce-eager` is rejected at startup.
- Not supported with LoRA (the per-token LoRA mapping is built from CPU-side boundaries) or pipeline parallelism (cost curves and confidences exist only on the last rank).
- Without a confidence head, `use_local_argmax_reduction` is rejected. The estimator needs the full draft logits, which local argmax reduction never materializes.
## Tuning the cost profile
-48
View File
@@ -1,48 +0,0 @@
# MLP Draft Models
The following code configures vLLM to use speculative decoding where proposals are generated by draft models that condition draft predictions on both context vectors and sampled tokens. For more information, see [The Hitchhiker's Guide to Speculative Decoding](https://pytorch.org/blog/hitchhikers-guide-speculative-decoding/) and [IBM Research's Technical Report](https://arxiv.org/abs/2404.19124).
## MLP Drafter Example
```python
from vllm import LLM, SamplingParams
prompts = ["The future of AI is"]
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
llm = LLM(
model="meta-llama/Meta-Llama-3.1-8B-Instruct",
tensor_parallel_size=1,
speculative_config={
"model": "ibm-ai-platform/llama3-8b-accelerator",
"draft_tensor_parallel_size": 1,
"method": "mlp_speculator",
},
)
outputs = llm.generate(prompts, sampling_params)
for output in outputs:
prompt = output.prompt
generated_text = output.outputs[0].text
print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
```
!!! warning "Known issue"
`ibm-ai-platform/llama3-70b-accelerator` can fail with:
`AttributeError: 'MLPSpeculatorConfig' object has no attribute 'num_attention_heads'`.
Track status in [#34106](https://github.com/vllm-project/vllm/issues/34106)
and [#34163](https://github.com/vllm-project/vllm/pull/34163).
## Pre-Trained MLP Drafter Models
A variety of speculative models of this type are available on HF hub:
- [llama-13b-accelerator](https://huggingface.co/ibm-ai-platform/llama-13b-accelerator)
- [llama3-8b-accelerator](https://huggingface.co/ibm-ai-platform/llama3-8b-accelerator)
- [codellama-34b-accelerator](https://huggingface.co/ibm-ai-platform/codellama-34b-accelerator)
- [llama2-70b-accelerator](https://huggingface.co/ibm-ai-platform/llama2-70b-accelerator)
- [llama3-70b-accelerator](https://huggingface.co/ibm-ai-platform/llama3-70b-accelerator)
- [granite-3b-code-instruct-accelerator](https://huggingface.co/ibm-granite/granite-3b-code-instruct-accelerator)
- [granite-8b-code-instruct-accelerator](https://huggingface.co/ibm-granite/granite-8b-code-instruct-accelerator)
- [granite-7b-instruct-accelerator](https://huggingface.co/ibm-granite/granite-7b-instruct-accelerator)
- [granite-20b-code-instruct-accelerator](https://huggingface.co/ibm-granite/granite-20b-code-instruct-accelerator)
@@ -12,6 +12,17 @@ Speculators provides the following key features:
- **Standardized, extensible format**: Provides a Hugging Face-compatible format for defining speculative models, with tools to convert from external research repositories into a standard speculators format for easy adoption.
- **Seamless vLLM Integration**: Built for direct deployment into vLLM, enabling low-latency, production-grade inference with minimal overhead.
## Verifier model references
When serving a Speculators checkpoint directly, `speculators_config.verifier.name_or_path` must be a remote model repository ID. References to local files or directories, including local paths that look like repository IDs, are rejected.
To use a local verifier, select it explicitly and configure the Speculators checkpoint as the draft model:
```bash
vllm serve /path/to/verifier --speculative-config \
'{"model": "/path/to/speculator", "method": "eagle3", "num_speculative_tokens": 1}'
```
## Why use Speculators?
Large language models generate text one token at a time, which creates a fundamental bottleneck: each token requires a full forward pass through the model, leaving GPU compute underutilized while waiting for memory-bound operations.
+3 -1
View File
@@ -146,7 +146,7 @@ Most OpenAI-compatible clients and agent frameworks never set `strict` on their
| Value | Behavior |
| --- | --- |
| `auto` (default) | Follow the request's tool choice and per-tool strictness. Required/named choices activate structural tags; `tool_choice="auto"` activates them when at least one tool sets `strict: true`. |
| `auto` | Follow the request's tool choice and per-tool strictness. Required/named choices activate structural tags; `tool_choice="auto"` activates them when at least one tool sets `strict: true`. |
| `function` | Constrain the tool-call envelope (markup and the function name) for every request with tools, leaving argument contents free unless the client marked the tool `strict: true`. |
| `parameter` | Additionally pin argument schemas for every tool, as if every tool had `strict: true`. |
@@ -154,6 +154,8 @@ Most OpenAI-compatible clients and agent frameworks never set `strict` on their
vllm serve ... --tool-strict-level function
```
When the option is unset, the tool parser's default level applies: `function` for `glm47`, `auto` for the other parsers. Passing `--tool-strict-level auto` overrides a parser default.
The floor never relaxes a constraint the request would already receive: tools the client marked `strict: true` keep their schemas at every level. With `tool_choice="auto"`, the grammar does not force a tool call; a plain text response stays valid. `VLLM_ENFORCE_STRICT_TOOL_CALLING=false` disables structural tags entirely and takes precedence over this option.
## Automatic Function Calling
+28
View File
@@ -56,6 +56,34 @@ watermarking algorithm that does not support it natively, at the cost of
weaker detectability. See
[Speculative decoding](#speculative-decoding).
## Monitoring
When `watermark_config` is set, the Prometheus counter
`vllm:watermark_requests_total` counts finished requests by `status`:
- `watermarked`: the request was sampled with the watermark switched on.
- `skipped`: the request asked for the watermark (omitted or `true`) but could
not use it, such as `temperature=0` and trace replay requests. A beam search
request that asks for the watermark runs without it and logs a warning. Its
beam steps count as `not_watermarked`, not `skipped`.
- `not_watermarked`: all other finished requests. This includes explicit
opt-outs, pooling requests, and the internal requests of beam search (one
per beam step), generative scoring and language detection.
Each finished request has exactly one status, so the sum over `status` equals
`vllm:request_success_total` on the same instance. Requests that the client
aborts are in neither counter. The counter does not exist without
`watermark_config`. With `n > 1`, each sequence counts once. In a
disaggregated prefill/decode deployment, both instances count a request: sum
over decode instances only. For streaming input, the first input chunk sets
the status.
The status is per request, not per token. A `watermarked` request can contain
tokens without the watermark: accepted draft tokens with
`allow_target_only_watermarking`, tokens whose context repeats
(`deduplicate_contexts`), the first `context_width` generated tokens with
`deduplicate_contexts: "all"`, and tokens that a grammar forces.
## Architecture
`WatermarkConfig` selects an algorithm and PRF. Model Runner V2 constructs the