Merge remote-tracking branch 'upstream/main' into patches

Signed-off-by: MOVZX <movzx@yahoo.com>
This commit is contained in:
2026-10-03 22:06:10 +07:00
327 changed files with 13704 additions and 5433 deletions
+1
View File
@@ -135,6 +135,7 @@ Batch invariance has been tested and verified on the following models:
- **Granite 3.1 (Dense)**: `ibm-granite/granite-3.1-2b-instruct`, `ibm-granite/granite-3.1-8b-instruct`
- **EXAONE 4.0 series**: `LGAI-EXAONE/EXAONE-4.0-1.2B`, `LGAI-EXAONE/EXAONE-4.0.1-32B`, `LGAI-EXAONE/EXAONE-4.0-32B`
- **OLMo 2**: `allenai/OLMo-2-0425-1B-Instruct`
- **ERNIE 4.5**: `baidu/ERNIE-4.5-0.3B-PT`
- **SmolLM2**: `HuggingFaceTB/SmolLM2-1.7B-Instruct`
- **PLaMo3**: `pfnet/plamo-3-nict-2b-base`
+6
View File
@@ -79,6 +79,12 @@ decode = client.chat.completions.create(
If `messages` has non-text content or `echo` is set, the ids are ignored and `messages` is rendered instead, so it must match the prefill request. Otherwise `kv_transfer_params["prompt_token_ids"]` must be a non-empty list of non-negative integers, or the request fails with HTTP 400, as it always does on `/v1/chat/completions/batch`.
## Generate API output modes
When the prefill and decode stages use the [Generate API](../serving/online_serving/token_in_token_out.md) (`/inference/v1/generate`), only the decode response reaches the client, so set `output_mode` on the decode request only. A proxy that reuses the client's request body for the prefill request must reset `output_mode` to `tokens` there. A prefill instance started with `--tokens-only` has no tokenizer and rejects `output_mode: "text"` with a 400.
A decode instance that returns text needs a tokenizer, so start it with `--enable-scale-out` and without `--tokens-only`. Prefill instances can keep `--tokens-only`.
## Development
We implement disaggregated prefilling by running 2 vLLM instances. One for prefill (we call it prefill instance) and one for decode (we call it decode instance), and then use a connector to transfer the prefill KV caches and results from prefill instance to decode instance.
+11 -2
View File
@@ -31,6 +31,11 @@ Snapshots currently require:
Current Hugging Face hub clients hold their connections for the process
lifetime, so download the model in a separate step and run create with
`HF_HUB_OFFLINE=1`, as the quickstart below does.
- Snapshot creation defaults `NCCL_IB_DISABLE=1` for its singleton donor and
inherited workers because CRIU cannot capture live InfiniBand/RDMA state.
An explicit caller value is retained, but creation rejects an open
`/dev/infiniband/` descriptor before CRIU. Close non-NCCL RDMA clients before
capture.
- A remote model ID and an immutable 40-character `--revision`. Local model
directories and mutable revisions are not supported.
- Enough disk for the artifact, with the same installed vLLM package, model
@@ -138,9 +143,13 @@ snapshot or external CRIU operation may use a shared `/dev/shm` mount at a time.
and directories. Recreate the artifact after an unclean host shutdown.
- Restore currently requires the same host, GPU, driver, kernel, Python,
PyTorch, installed vLLM version, model revision, engine arguments,
selected environment variables, and CRIU plugin binaries.
selected effective environment variables, and CRIU plugin binaries. An unset
`NCCL_IB_DISABLE` therefore matches the snapshot donor default of `1`, while
an explicit different value does not.
- Only dense float16 TP1 has been validated. Other model formats depend on their
existing sleep level 2 reload support; NCCL state is not restored.
existing sleep level 2 reload support; distributed and RDMA snapshots are not
supported. The snapshot-only NCCL default does not change ordinary serve or
multi-GPU defaults.
- CRIU support varies by kernel and driver. Preserve package, library, model,
and generated-cache paths for the artifact lifetime.
- A snapshot can include application secrets or request state present in the
+1 -3
View File
@@ -33,9 +33,7 @@ following `quantization.quant_algo` values:
`flashinfer_cutlass`, `flashinfer_cutedsl`, `flashinfer_trtllm`,
`flashinfer_cudnn`, and `marlin`; see `KernelConfig` on the
[Engine Arguments](../../configuration/engine_args.md) page and shown by
`vllm serve --help=KernelConfig`. For `W4A16_NVFP4`, `auto` currently
selects Marlin. Models with BF16 activations can explicitly select the
FlashInfer CuTe-DSL backend with `--linear-backend flashinfer_cutedsl`.
`vllm serve --help=KernelConfig`.
!!! note
For models quantized to MXFP8 with BF16 activations on SM100-family GPUs,