Merge remote-tracking branch 'upstream/main' into patches
Signed-off-by: MOVZX <movzx@yahoo.com>
This commit is contained in:
@@ -135,6 +135,7 @@ Batch invariance has been tested and verified on the following models:
|
||||
- **Granite 3.1 (Dense)**: `ibm-granite/granite-3.1-2b-instruct`, `ibm-granite/granite-3.1-8b-instruct`
|
||||
- **EXAONE 4.0 series**: `LGAI-EXAONE/EXAONE-4.0-1.2B`, `LGAI-EXAONE/EXAONE-4.0.1-32B`, `LGAI-EXAONE/EXAONE-4.0-32B`
|
||||
- **OLMo 2**: `allenai/OLMo-2-0425-1B-Instruct`
|
||||
- **ERNIE 4.5**: `baidu/ERNIE-4.5-0.3B-PT`
|
||||
- **SmolLM2**: `HuggingFaceTB/SmolLM2-1.7B-Instruct`
|
||||
- **PLaMo3**: `pfnet/plamo-3-nict-2b-base`
|
||||
|
||||
|
||||
@@ -79,6 +79,12 @@ decode = client.chat.completions.create(
|
||||
|
||||
If `messages` has non-text content or `echo` is set, the ids are ignored and `messages` is rendered instead, so it must match the prefill request. Otherwise `kv_transfer_params["prompt_token_ids"]` must be a non-empty list of non-negative integers, or the request fails with HTTP 400, as it always does on `/v1/chat/completions/batch`.
|
||||
|
||||
## Generate API output modes
|
||||
|
||||
When the prefill and decode stages use the [Generate API](../serving/online_serving/token_in_token_out.md) (`/inference/v1/generate`), only the decode response reaches the client, so set `output_mode` on the decode request only. A proxy that reuses the client's request body for the prefill request must reset `output_mode` to `tokens` there. A prefill instance started with `--tokens-only` has no tokenizer and rejects `output_mode: "text"` with a 400.
|
||||
|
||||
A decode instance that returns text needs a tokenizer, so start it with `--enable-scale-out` and without `--tokens-only`. Prefill instances can keep `--tokens-only`.
|
||||
|
||||
## Development
|
||||
|
||||
We implement disaggregated prefilling by running 2 vLLM instances. One for prefill (we call it prefill instance) and one for decode (we call it decode instance), and then use a connector to transfer the prefill KV caches and results from prefill instance to decode instance.
|
||||
|
||||
@@ -31,6 +31,11 @@ Snapshots currently require:
|
||||
Current Hugging Face hub clients hold their connections for the process
|
||||
lifetime, so download the model in a separate step and run create with
|
||||
`HF_HUB_OFFLINE=1`, as the quickstart below does.
|
||||
- Snapshot creation defaults `NCCL_IB_DISABLE=1` for its singleton donor and
|
||||
inherited workers because CRIU cannot capture live InfiniBand/RDMA state.
|
||||
An explicit caller value is retained, but creation rejects an open
|
||||
`/dev/infiniband/` descriptor before CRIU. Close non-NCCL RDMA clients before
|
||||
capture.
|
||||
- A remote model ID and an immutable 40-character `--revision`. Local model
|
||||
directories and mutable revisions are not supported.
|
||||
- Enough disk for the artifact, with the same installed vLLM package, model
|
||||
@@ -138,9 +143,13 @@ snapshot or external CRIU operation may use a shared `/dev/shm` mount at a time.
|
||||
and directories. Recreate the artifact after an unclean host shutdown.
|
||||
- Restore currently requires the same host, GPU, driver, kernel, Python,
|
||||
PyTorch, installed vLLM version, model revision, engine arguments,
|
||||
selected environment variables, and CRIU plugin binaries.
|
||||
selected effective environment variables, and CRIU plugin binaries. An unset
|
||||
`NCCL_IB_DISABLE` therefore matches the snapshot donor default of `1`, while
|
||||
an explicit different value does not.
|
||||
- Only dense float16 TP1 has been validated. Other model formats depend on their
|
||||
existing sleep level 2 reload support; NCCL state is not restored.
|
||||
existing sleep level 2 reload support; distributed and RDMA snapshots are not
|
||||
supported. The snapshot-only NCCL default does not change ordinary serve or
|
||||
multi-GPU defaults.
|
||||
- CRIU support varies by kernel and driver. Preserve package, library, model,
|
||||
and generated-cache paths for the artifact lifetime.
|
||||
- A snapshot can include application secrets or request state present in the
|
||||
|
||||
@@ -33,9 +33,7 @@ following `quantization.quant_algo` values:
|
||||
`flashinfer_cutlass`, `flashinfer_cutedsl`, `flashinfer_trtllm`,
|
||||
`flashinfer_cudnn`, and `marlin`; see `KernelConfig` on the
|
||||
[Engine Arguments](../../configuration/engine_args.md) page and shown by
|
||||
`vllm serve --help=KernelConfig`. For `W4A16_NVFP4`, `auto` currently
|
||||
selects Marlin. Models with BF16 activations can explicitly select the
|
||||
FlashInfer CuTe-DSL backend with `--linear-backend flashinfer_cutedsl`.
|
||||
`vllm serve --help=KernelConfig`.
|
||||
|
||||
!!! note
|
||||
For models quantized to MXFP8 with BF16 activations on SM100-family GPUs,
|
||||
|
||||
Reference in New Issue
Block a user