Merge remote-tracking branch 'upstream/main' into patches
Signed-off-by: MOVZX <movzx@yahoo.com>
This commit is contained in:
@@ -0,0 +1,55 @@
|
||||
# Reviewers
|
||||
|
||||
Reviewers are trusted community members who can perform official reviews, helping contributors get their pull requests ready to merge alongside committers, who give final approval. Authors are expected to respond to reviewer feedback, either by making changes or by explaining why a change isn't needed. The role was introduced in [RFC #58253](https://github.com/vllm-project/vllm/issues/58253).
|
||||
|
||||
This page lists the active reviewers and describes the role. See the [committers](../governance/committers.md) page for the committers and area owners who hold final acceptance and merge authority.
|
||||
|
||||
## Active Reviewers
|
||||
|
||||
Community members can reach out to the reviewers below for help with PRs in their areas, for example in the `#pr-reviews` Slack channel. Reviewers will add their areas of expertise to this list.
|
||||
Sorted alphabetically by GitHub handle:
|
||||
|
||||
- [@andylolu2](https://github.com/andylolu2)
|
||||
- [@bnellnm](https://github.com/bnellnm) MoE, Quantization layers
|
||||
- [@BowenBao](https://github.com/BowenBao)
|
||||
- [@cjackal](https://github.com/cjackal): Multimodality; KV Connector and offload
|
||||
- [@divakar-amd](https://github.com/divakar-amd)
|
||||
- [@etelis](https://github.com/etelis): KV cache offloading (CPU, filesystem, and P2P connectors; KV transfer and cache layouts); MoE serving and expert parallelism (developing review focus)
|
||||
- [@Fangzhou-Ai](https://github.com/Fangzhou-Ai)
|
||||
- [@fxmarty-amd](https://github.com/fxmarty-amd): Quantization, linear/MOE oracles and modeling definition
|
||||
- [@GirasoleY](https://github.com/GirasoleY): Kernels and performance
|
||||
- [@itayalroy](https://github.com/itayalroy): MoE serving (kernels, EP, All2All, EPLB, elasticity/fault tolerance); KV connectors and NIXL integrations
|
||||
- [@JartX](https://github.com/JartX): ROCm / HIP on Radeon (RDNA3), quantization (W4A16/GPTQ, MXFP4), KV cache quantization
|
||||
- [@kliuae](https://github.com/kliuae): ROCm feature enablement, model performance, fused kernels
|
||||
- [@lengrongfu](https://github.com/lengrongfu)
|
||||
- [@linitra24](https://github.com/linitra24)
|
||||
- [@LopezCastroRoberto](https://github.com/LopezCastroRoberto): Kernels and quantization
|
||||
- [@mawong-amd](https://github.com/mawong-amd)
|
||||
- [@micah-wil](https://github.com/micah-wil): ROCm / AMD GPU integration, CI
|
||||
- [@netanel-haber](https://github.com/netanel-haber): Nemotron/NVIDIA models, Mamba/Linear attention hybrids, VLMs
|
||||
- [@Rohan138](https://github.com/Rohan138): ROCm performance, CI, torch.compile fusions
|
||||
- [@simondanielsson](https://github.com/simondanielsson): ROCm performance (CDNA), MoRI-IO.
|
||||
- [@taneem-ibrahim](https://github.com/taneem-ibrahim): Pooling models
|
||||
- [@TheEpicDolphin](https://github.com/TheEpicDolphin)
|
||||
- [@varun-sundar-rabindranath](https://github.com/varun-sundar-rabindranath): KV cache offloading, LoRA
|
||||
- [@vllmellm](https://github.com/vllmellm)
|
||||
- [@wangxiyuan](https://github.com/wangxiyuan): Platform, KV cache, Mooncake
|
||||
|
||||
## The Reviewer Role
|
||||
|
||||
Reviewers assess the problem and proposed solution, guide validation, resolve concerns with authors, and determine when a contribution is ready for a committer. A reviewer's approval means a PR is ready for a committer's final look.
|
||||
|
||||
Reviewers can:
|
||||
|
||||
- Approve PRs or request changes.
|
||||
- Trigger CI when a contribution is ready for evaluation.
|
||||
- Triage PRs and issues, apply labels, and flag important PRs for expedited committer attention.
|
||||
- Close duplicate PRs and issues.
|
||||
- Escalate design questions to the appropriate committer or SIG.
|
||||
- Flag PRs as ready for committer review.
|
||||
|
||||
### Becoming a Reviewer
|
||||
|
||||
Candidates should demonstrate a sustained record of useful reviews, sound engineering judgment, constructive communication, and familiarity with vLLM's contribution standards. Deep ownership of a subsystem is not required.
|
||||
|
||||
A reviewer is nominated by two committers from different organizations, who post the nomination in the committers Slack channel. As a rule of thumb, not a hard requirement, candidates have either 15 merged PRs and 30 reviewed PRs, or 45 reviewed PRs. A reviewed PR is one where the candidate gave a substantive review and the PR reached a resolution (merged or closed).
|
||||
@@ -307,15 +307,23 @@ review process:
|
||||
|
||||
- After the PR is submitted, the PR will be assigned to a reviewer. Every
|
||||
reviewer will pick up the PRs based on their expertise and availability.
|
||||
Official reviews can be performed by any of the
|
||||
[reviewers](../community/reviewers.md) or
|
||||
[committers](../governance/committers.md); committers perform the final
|
||||
merge. To find someone to review your PR, see the reviewers' and
|
||||
committers' areas of expertise or ask in the `#pr-reviews` channel on
|
||||
[Slack](https://slack.vllm.ai).
|
||||
- After the PR is assigned, the reviewer will provide status updates every 2-3
|
||||
days. If the PR is not reviewed within 7 days, please feel free to ping the
|
||||
reviewer or the vLLM team.
|
||||
- After the review, the reviewer will put an `action-required` label on the PR
|
||||
if there are changes required. The contributor should address the comments and
|
||||
ping the reviewer to re-review the PR.
|
||||
- Please respond to all comments within a reasonable time frame. If a comment
|
||||
- Please respond to all comments within a reasonable time frame. Address each
|
||||
piece of reviewer feedback with a change or an explanation. If a comment
|
||||
isn't clear or you disagree with a suggestion, feel free to ask for
|
||||
clarification or discuss the suggestion.
|
||||
clarification or discuss the suggestion. If you and a reviewer can't reach
|
||||
agreement, a committer can help decide.
|
||||
- Note that not all CI checks will be executed due to limited computational
|
||||
resources. Reviewers with write access and configured trusted contributors
|
||||
can comment `/ci run` for upstream CI or `/amd-ci run` for AMD CI only when
|
||||
|
||||
@@ -44,6 +44,19 @@ cross-file context, and fixes for repeated mistakes.
|
||||
Each entry should be short, specific, and actionable — e.g., which files to
|
||||
touch, what order to change them in, and which tests to run.
|
||||
|
||||
## Skills
|
||||
|
||||
Skills live in `.agents/skills/<name>/` and follow the
|
||||
[Agent Skills specification](https://agentskills.io/specification).
|
||||
|
||||
- Claude Code does not read `.agents/`, so symlink each skill:
|
||||
`ln -s ../../.agents/skills/<name> .claude/skills/<name>`.
|
||||
- Frontmatter: `name` matches the directory. `description` says what the
|
||||
skill does and when to use it; quote it if it contains a colon.
|
||||
- Keep `SKILL.md` under 500 lines and move detail into files it links to.
|
||||
- `tools/pre_commit/check_agent_files.py` runs in pre-commit to check the
|
||||
skill and `AGENTS.md` rules
|
||||
|
||||
## Keeping Docs Lean
|
||||
|
||||
- Every addition should trigger review of surrounding content for stale or redundant items.
|
||||
|
||||
@@ -37,6 +37,7 @@ th {
|
||||
| deepep_low_latency | batched | fp8 | G(128),A,T<sup>3</sup> | Y | Y | [`DeepEPLLPrepareAndFinalize`][vllm.model_executor.layers.fused_moe.prepare_finalize.deepep_ll.DeepEPLLPrepareAndFinalize] |
|
||||
| flashinfer_nvlink_two_sided | standard | nvfp4,fp8 | G,A,T | N | N | [`FlashInferNVLinkTwoSidedPrepareAndFinalize`][vllm.model_executor.layers.fused_moe.prepare_finalize.flashinfer_nvlink_two_sided.FlashInferNVLinkTwoSidedPrepareAndFinalize] |
|
||||
| flashinfer_nvlink_one_sided | standard | nvfp4,bf16,mxfp8,fp8 | G,A,T | N | N | [`FlashInferNVLinkOneSidedPrepareAndFinalize`][vllm.model_executor.layers.fused_moe.prepare_finalize.flashinfer_nvlink_one_sided.FlashInferNVLinkOneSidedPrepareAndFinalize] |
|
||||
| passthrough | standard | nvfp4,mxfp4 | G | N | Y | [`PassThroughPrepareAndFinalize`][vllm.model_executor.layers.fused_moe.prepare_finalize.passthrough.PassThroughPrepareAndFinalize] |
|
||||
|
||||
!!! info "Table key"
|
||||
1. All types: mxfp4, nvfp4, int4, int8, fp8
|
||||
|
||||
@@ -135,6 +135,7 @@ Batch invariance has been tested and verified on the following models:
|
||||
- **Granite 3.1 (Dense)**: `ibm-granite/granite-3.1-2b-instruct`, `ibm-granite/granite-3.1-8b-instruct`
|
||||
- **EXAONE 4.0 series**: `LGAI-EXAONE/EXAONE-4.0-1.2B`, `LGAI-EXAONE/EXAONE-4.0.1-32B`, `LGAI-EXAONE/EXAONE-4.0-32B`
|
||||
- **OLMo 2**: `allenai/OLMo-2-0425-1B-Instruct`
|
||||
- **ERNIE 4.5**: `baidu/ERNIE-4.5-0.3B-PT`
|
||||
- **SmolLM2**: `HuggingFaceTB/SmolLM2-1.7B-Instruct`
|
||||
- **PLaMo3**: `pfnet/plamo-3-nict-2b-base`
|
||||
|
||||
|
||||
@@ -79,6 +79,12 @@ decode = client.chat.completions.create(
|
||||
|
||||
If `messages` has non-text content or `echo` is set, the ids are ignored and `messages` is rendered instead, so it must match the prefill request. Otherwise `kv_transfer_params["prompt_token_ids"]` must be a non-empty list of non-negative integers, or the request fails with HTTP 400, as it always does on `/v1/chat/completions/batch`.
|
||||
|
||||
## Generate API output modes
|
||||
|
||||
When the prefill and decode stages use the [Generate API](../serving/online_serving/token_in_token_out.md) (`/inference/v1/generate`), only the decode response reaches the client, so set `output_mode` on the decode request only. A proxy that reuses the client's request body for the prefill request must reset `output_mode` to `tokens` there. A prefill instance started with `--tokens-only` has no tokenizer and rejects `output_mode: "text"` with a 400.
|
||||
|
||||
A decode instance that returns text needs a tokenizer, so start it with `--enable-scale-out` and without `--tokens-only`. Prefill instances can keep `--tokens-only`.
|
||||
|
||||
## Development
|
||||
|
||||
We implement disaggregated prefilling by running 2 vLLM instances. One for prefill (we call it prefill instance) and one for decode (we call it decode instance), and then use a connector to transfer the prefill KV caches and results from prefill instance to decode instance.
|
||||
|
||||
@@ -31,6 +31,11 @@ Snapshots currently require:
|
||||
Current Hugging Face hub clients hold their connections for the process
|
||||
lifetime, so download the model in a separate step and run create with
|
||||
`HF_HUB_OFFLINE=1`, as the quickstart below does.
|
||||
- Snapshot creation defaults `NCCL_IB_DISABLE=1` for its singleton donor and
|
||||
inherited workers because CRIU cannot capture live InfiniBand/RDMA state.
|
||||
An explicit caller value is retained, but creation rejects an open
|
||||
`/dev/infiniband/` descriptor before CRIU. Close non-NCCL RDMA clients before
|
||||
capture.
|
||||
- A remote model ID and an immutable 40-character `--revision`. Local model
|
||||
directories and mutable revisions are not supported.
|
||||
- Enough disk for the artifact, with the same installed vLLM package, model
|
||||
@@ -138,9 +143,13 @@ snapshot or external CRIU operation may use a shared `/dev/shm` mount at a time.
|
||||
and directories. Recreate the artifact after an unclean host shutdown.
|
||||
- Restore currently requires the same host, GPU, driver, kernel, Python,
|
||||
PyTorch, installed vLLM version, model revision, engine arguments,
|
||||
selected environment variables, and CRIU plugin binaries.
|
||||
selected effective environment variables, and CRIU plugin binaries. An unset
|
||||
`NCCL_IB_DISABLE` therefore matches the snapshot donor default of `1`, while
|
||||
an explicit different value does not.
|
||||
- Only dense float16 TP1 has been validated. Other model formats depend on their
|
||||
existing sleep level 2 reload support; NCCL state is not restored.
|
||||
existing sleep level 2 reload support; distributed and RDMA snapshots are not
|
||||
supported. The snapshot-only NCCL default does not change ordinary serve or
|
||||
multi-GPU defaults.
|
||||
- CRIU support varies by kernel and driver. Preserve package, library, model,
|
||||
and generated-cache paths for the artifact lifetime.
|
||||
- A snapshot can include application secrets or request state present in the
|
||||
|
||||
@@ -33,9 +33,7 @@ following `quantization.quant_algo` values:
|
||||
`flashinfer_cutlass`, `flashinfer_cutedsl`, `flashinfer_trtllm`,
|
||||
`flashinfer_cudnn`, and `marlin`; see `KernelConfig` on the
|
||||
[Engine Arguments](../../configuration/engine_args.md) page and shown by
|
||||
`vllm serve --help=KernelConfig`. For `W4A16_NVFP4`, `auto` currently
|
||||
selects Marlin. Models with BF16 activations can explicitly select the
|
||||
FlashInfer CuTe-DSL backend with `--linear-backend flashinfer_cutedsl`.
|
||||
`vllm serve --help=KernelConfig`.
|
||||
|
||||
!!! note
|
||||
For models quantized to MXFP8 with BF16 activations on SM100-family GPUs,
|
||||
|
||||
@@ -440,7 +440,8 @@ ref changes. Use `--no-cache-filter extensions-build` to refresh an empty,
|
||||
branch, or tag revision.
|
||||
|
||||
For `FINAL_BASE_IMAGE`, use the public, multi-arch
|
||||
`nvcr.io/nvidia/cuda-dl-base:26.08-cuda13.4-devel-ubuntu24.04` image.
|
||||
`nvidia/cuda:13.4.1-base-ubuntu24.04` image. Set `NCCL_VERSION` to 2.32.3 or
|
||||
newer, the first NCCL release with Rubin (SM107) support.
|
||||
For `BUILD_BASE_IMAGE`, use:
|
||||
|
||||
- `pytorch/manylinux2_28-builder:cuda13.4` for x86_64 CPUs.
|
||||
@@ -462,7 +463,8 @@ For `BUILD_BASE_IMAGE`, use:
|
||||
--build-arg TRITON_INSTALL_FROM_SOURCE_REVISION=3f6e41132b5edf639bfb872ad73d4688765e08b8 \
|
||||
--build-arg CUDA_VERSION=13.4 \
|
||||
--build-arg BUILD_BASE_IMAGE="pytorch/manylinuxaarch64-builder:cuda13.4" \
|
||||
--build-arg FINAL_BASE_IMAGE="nvcr.io/nvidia/cuda-dl-base:26.08-cuda13.4-devel-ubuntu24.04" \
|
||||
--build-arg FINAL_BASE_IMAGE="nvidia/cuda:13.4.1-base-ubuntu24.04" \
|
||||
--build-arg NCCL_VERSION=2.32.3 \
|
||||
.
|
||||
```
|
||||
|
||||
@@ -482,7 +484,8 @@ For `BUILD_BASE_IMAGE`, use:
|
||||
--build-arg TRITON_INSTALL_FROM_SOURCE_REVISION=3f6e41132b5edf639bfb872ad73d4688765e08b8 \
|
||||
--build-arg CUDA_VERSION=13.4 \
|
||||
--build-arg BUILD_BASE_IMAGE="pytorch/manylinux2_28-builder:cuda13.4" \
|
||||
--build-arg FINAL_BASE_IMAGE="nvcr.io/nvidia/cuda-dl-base:26.08-cuda13.4-devel-ubuntu24.04" \
|
||||
--build-arg FINAL_BASE_IMAGE="nvidia/cuda:13.4.1-base-ubuntu24.04" \
|
||||
--build-arg NCCL_VERSION=2.32.3 \
|
||||
.
|
||||
```
|
||||
|
||||
@@ -490,9 +493,6 @@ For `BUILD_BASE_IMAGE`, use:
|
||||
Keep the default explicit `torch_cuda_arch_list`. GPU-less BuildKit builds
|
||||
cannot inspect the host GPU. R100 and VR200 report compute capability 10.7,
|
||||
for which the generic `10.0` target provides family-compatible kernels.
|
||||
The Ubuntu `devel` final image is also required: the corresponding `base`
|
||||
image lacks the CUDA runtime/JIT package closure used by vLLM and the
|
||||
prerelease PyTorch wheel.
|
||||
|
||||
`RUN_WHEEL_CHECK=false` disables only the PyPI publication-size guard for
|
||||
this private staging image.
|
||||
|
||||
@@ -410,7 +410,6 @@ th {
|
||||
| `GlmMoeDsaForCausalLM` | GLM-5, GLM-5.1, GLM-5.2 | `zai-org/GLM-5`, etc. | ✅︎ | ✅︎ |
|
||||
| `GPT2LMHeadModel` | GPT-2 | `openai-community/gpt2`, `openai-community/gpt2-xl`, etc. | | ✅︎ |
|
||||
| `GPTJForCausalLM` | GPT-J | `EleutherAI/gpt-j-6b`, `nomic-ai/gpt4all-j`, etc. | | ✅︎ |
|
||||
| `GPTNeoXForCausalLM` | GPT-NeoX, Pythia, OpenAssistant, Dolly V2, StableLM | `EleutherAI/gpt-neox-20b`, `EleutherAI/pythia-12b`, `OpenAssistant/oasst-sft-4-pythia-12b-epoch-3.5`, `databricks/dolly-v2-12b`, `stabilityai/stablelm-tuned-alpha-7b`, etc. | | ✅︎ |
|
||||
| `GptOssForCausalLM` | GPT-OSS | `openai/gpt-oss-120b`, `openai/gpt-oss-20b` | ✅︎ | ✅︎ |
|
||||
| `GraniteForCausalLM` | Granite 3.0, Granite 3.1, PowerLM | `ibm-granite/granite-3.0-2b-base`, `ibm-granite/granite-3.1-8b-instruct`, `ibm/PowerLM-3b`, etc. | ✅︎ | ✅︎ |
|
||||
| `GraniteMoeForCausalLM` | Granite 3.0 MoE, PowerMoE | `ibm-granite/granite-3.0-1b-a400m-base`, `ibm-granite/granite-3.0-3b-a800m-instruct`, `ibm/PowerMoE-3b`, etc. | ✅︎ | ✅︎ |
|
||||
@@ -426,7 +425,6 @@ th {
|
||||
| `InternLM3ForCausalLM` | InternLM3 | `internlm/internlm3-8b-instruct`, etc. | ✅︎ | ✅︎ |
|
||||
| `IQuestCoderForCausalLM` | IQuestCoderV1 | `IQuestLab/IQuest-Coder-V1-40B-Instruct`, etc. | | |
|
||||
| `IQuestLoopCoderForCausalLM` | IQuestLoopCoderV1 | `IQuestLab/IQuest-Coder-V1-40B-Loop-Instruct`, etc. | | |
|
||||
| `Jais2ForCausalLM` | Jais2 | `inceptionai/Jais-2-8B-Chat`, `inceptionai/Jais-2-70B-Chat`, etc. | | ✅︎ |
|
||||
| `JambaForCausalLM` | Jamba | `ai21labs/AI21-Jamba-1.5-Large`, `ai21labs/AI21-Jamba-1.5-Mini`, `ai21labs/Jamba-v0.1`, etc. | ✅︎ | ✅︎ |
|
||||
| `K2HorizonForCausalLM` | K2-Horizon | `IFM/K2-Horizon-7B`, `IFM/K2-Horizon-375B` | | ✅︎ |
|
||||
| `KimiLinearForCausalLM` | Kimi-Linear-48B-A3B-Base, Kimi-Linear-48B-A3B-Instruct | `moonshotai/Kimi-Linear-48B-A3B-Base`, `moonshotai/Kimi-Linear-48B-A3B-Instruct` | | ✅︎ |
|
||||
@@ -456,7 +454,6 @@ th {
|
||||
| `PanguProMoEV2ForCausalLM` | openpangu-pro-moe-v2 | | ✅︎ | ✅︎ |
|
||||
| `PanguUltraMoEForCausalLM` | openpangu-ultra-moe-718b-model | `FreedomIntelligence/openPangu-Ultra-MoE-718B-V1.1` | ✅︎ | ✅︎ |
|
||||
| `Param2MoEForCausalLM` | param2moe | `bharatgenai/Param2-17B-A2.4B-Thinking`, etc. | ✅︎ | ✅︎ |
|
||||
| `PhiForCausalLM` | Phi | `microsoft/phi-1_5`, `microsoft/phi-2`, etc. | ✅︎ | ✅︎ |
|
||||
| `Phi3ForCausalLM` | Phi-4, Phi-3 | `microsoft/Phi-4-mini-instruct`, `microsoft/Phi-4`, `microsoft/Phi-3-mini-4k-instruct`, `microsoft/Phi-3-mini-128k-instruct`, `microsoft/Phi-3-medium-128k-instruct`, etc. | ✅︎ | ✅︎ |
|
||||
| `PhiMoEForCausalLM` | Phi-3.5-MoE | `microsoft/Phi-3.5-MoE-instruct`, etc. | ✅︎ | ✅︎ |
|
||||
| `Plamo3ForCausalLM` | PLaMo3 | `pfnet/plamo-3-nict-2b-base`, `pfnet/plamo-3-nict-8b-base`, etc. | ✅ | ✅︎ |
|
||||
@@ -468,7 +465,6 @@ th {
|
||||
| `Rnj1ForCausalLM` | Rnj1 | `EssentialAI/rnj-1-instruct`, etc. | | |
|
||||
| `SarvamMoEForCausalLM` | Sarvam 2 | `sarvamai/sarvam2-30b-a3b`, etc. | ✅︎ | ✅︎ |
|
||||
| `SarvamMLAForCausalLM` | Sarvam 2 | `sarvamai/sarvam2-105b-a9b`, etc. | | ✅︎ |
|
||||
| `SeedOssForCausalLM` | SeedOss | `ByteDance-Seed/Seed-OSS-36B-Instruct`, etc. | ✅︎ | ✅︎ |
|
||||
| `SolarForCausalLM` | Solar Pro | `upstage/solar-pro-preview-instruct`, etc. | ✅︎ | ✅︎ |
|
||||
| `StableLmForCausalLM` | StableLM | `stabilityai/stablelm-3b-4e1t`, `stabilityai/stablelm-base-alpha-7b-v2`, etc. | | |
|
||||
| `Step1ForCausalLM` | Step-Audio | `stepfun-ai/Step-Audio-EditX`, etc. | ✅︎ | ✅︎ |
|
||||
@@ -484,16 +480,20 @@ Some models are supported only via the [Transformers modeling backend](#transfor
|
||||
| ------------ | ------ | ----------------- | -------------------- | ------------------------- |
|
||||
| `ArceeForCausalLM` | Arcee (AFM) | `arcee-ai/AFM-4.5B-Base`, etc. | ✅︎ | ✅︎ |
|
||||
| `CwmForCausalLM` | CWM | `facebook/cwm`, etc. | ✅︎ | ✅︎ |
|
||||
| `FlexOlmoForCausalLM` | FlexOlmo | `allenai/FlexOlmo-7x7B-1T`, `allenai/FlexOlmo-7x7B-1T-RT`, etc. | | ✅︎ |
|
||||
| `FlexOlmoForCausalLM` | FlexOlmo | `allenai/FlexOlmo-7x7B-1T`, `allenai/FlexOlmo-7x7B-1T-RT`, etc. | ✅︎ | ✅︎ |
|
||||
| `GlmForCausalLM` | GLM-4 | `zai-org/glm-4-9b-chat-hf`, etc. | ✅︎ | ✅︎ |
|
||||
| `GPTBigCodeForCausalLM` | StarCoder, SantaCoder, WizardCoder | `bigcode/starcoder`, `bigcode/gpt_bigcode-santacoder`, `WizardLM/WizardCoder-15B-V1.0`, etc. | ✅︎ | |
|
||||
| `GPTNeoXForCausalLM` | GPT-NeoX, Pythia, OpenAssistant, Dolly V2, StableLM | `EleutherAI/gpt-neox-20b`, `EleutherAI/pythia-12b`, `OpenAssistant/oasst-sft-4-pythia-12b-epoch-3.5`, `databricks/dolly-v2-12b`, `stabilityai/stablelm-tuned-alpha-7b`, etc. | | ✅︎ |
|
||||
| `HunYuanDenseV1ForCausalLM` | Hunyuan Dense | `tencent/Hunyuan-7B-Instruct` | ✅︎ | ✅︎ |
|
||||
| `HunYuanMoEV1ForCausalLM` | Hunyuan-A13B | `tencent/Hunyuan-A13B-Instruct`, `tencent/Hunyuan-A13B-Pretrain`, `tencent/Hunyuan-A13B-Instruct-FP8`, etc. | ✅︎ | ✅︎ |
|
||||
| `MellumForCausalLM` | Mellum 2 | `JetBrains/Mellum2-12B-A2.5B-Base`, etc. | | ✅︎ |
|
||||
| `Jais2ForCausalLM` | Jais2 | `inceptionai/Jais-2-8B-Chat`, `inceptionai/Jais-2-70B-Chat`, etc. | ✅︎ | ✅︎ |
|
||||
| `MellumForCausalLM` | Mellum 2 | `JetBrains/Mellum2-12B-A2.5B-Base`, etc. | ✅︎ | ✅︎ |
|
||||
| `NanbeigeForCausalLM` | Nanbeige4.2 | `Nanbeige/Nanbeige4.2-3B`, etc. | ✅︎ | ✅︎ |
|
||||
| `OlmoForCausalLM` | OLMo | `allenai/OLMo-1B-hf`, `allenai/OLMo-7B-hf`, etc. | ✅︎ | ✅︎ |
|
||||
| `Olmo2ForCausalLM` | OLMo2 | `allenai/OLMo-2-0425-1B`, etc. | ✅︎ | ✅︎ |
|
||||
| `Olmo3ForCausalLM` | OLMo3 | `allenai/Olmo-3-7B-Instruct`, `allenai/Olmo-3-32B-Think`, etc. | ✅︎ | ✅︎ |
|
||||
| `PhiForCausalLM` | Phi | `microsoft/phi-1_5`, `microsoft/phi-2`, etc. | ✅︎ | ✅︎ |
|
||||
| `SeedOssForCausalLM` | SeedOss | `ByteDance-Seed/Seed-OSS-36B-Instruct`, etc. | ✅︎ | ✅︎ |
|
||||
| `SmolLM3ForCausalLM` | SmolLM3 | `HuggingFaceTB/SmolLM3-3B` | ✅︎ | ✅︎ |
|
||||
| `Starcoder2ForCausalLM` | Starcoder2 | `bigcode/starcoder2-3b`, `bigcode/starcoder2-7b`, `bigcode/starcoder2-15b`, etc. | ✅︎ | ✅︎ |
|
||||
| `VaultGemmaForCausalLM` | VaultGemma | `google/vaultgemma-1b` | ✅︎ | ✅︎ |
|
||||
|
||||
@@ -27,6 +27,7 @@ vLLM provides multiple communication backends for EP. Use `--all2all-backend` to
|
||||
| `deepep_high_throughput` | Multi-node prefill | Grouped GEMM with continuous layout, optimized for prefill | Prefill-dominated workloads, high-throughput scenarios |
|
||||
| `deepep_low_latency` | Multi-node decode | CUDA graph support, masked layout, optimized for decode | Decode-dominated workloads, low-latency scenarios |
|
||||
| `flashinfer_nvlink_one_sided` | MNNVL systems | FlashInfer's one-sided A2A strategy for multi-node NVLink | High-throughput workloads |
|
||||
| `passthrough` | Any | Internal, not accepted on the CLI: bound automatically for MoE backends that dispatch and combine themselves, so no all-to-all runs in the framework | `--moe-backend flashinfer_moe_ep_*` (FlashInfer MoE-EP megakernels) |
|
||||
| `flashinfer_nvlink_two_sided` | MNNVL systems | FlashInfer's two-sided A2A strategy for multi-node NVLink | Systems with NVLink across nodes |
|
||||
|
||||
## Single Node Deployment
|
||||
|
||||
@@ -125,8 +125,10 @@ Scale-out APIs are disabled by default on `vllm serve`. Set `--enable-scale-out`
|
||||
|
||||
### Tokens IN <> Tokens OUT APIs
|
||||
|
||||
- `/inference/v1/generate` - Generate completions
|
||||
- `/abort_requests` - Abort in-flight requests (only when `--tokens-only` is also set)
|
||||
- [Generate API](token_in_token_out.md) (`/inference/v1/generate`)
|
||||
- Generate completions from token IDs, optionally with detokenized text (`output_mode`)
|
||||
- `/inference/v1/abort_requests` - Abort in-flight requests (registered wherever `/inference/v1/generate` is)
|
||||
- `/abort_requests` - Unauthenticated alias of `/inference/v1/abort_requests` (`--tokens-only` only)
|
||||
|
||||
### Renderer APIs
|
||||
|
||||
|
||||
@@ -40,6 +40,8 @@ Seeding only reads the last few IDs (7 today), so a suffix of the prompt is enou
|
||||
|
||||
Each request wraps the engine's `GenerateResponse`(s) together with the caller metadata needed to reconstruct the final response without a GPU.
|
||||
|
||||
Only `output_mode: "tokens"` responses are accepted which is the default. A response from a request with `output_mode: "text"` gets a 400 because its text is already detokenized with stop strings removed. Responses without `output_mode`, from older servers, are treated as `tokens`.
|
||||
|
||||
`/v1/chat/completions/derender`:
|
||||
|
||||
??? code
|
||||
@@ -127,7 +129,7 @@ Streaming derender is stateless on the server side: all mutable state lives in t
|
||||
- `logprob_context_token_ids`: the trailing sampled token IDs (at most 4) from previous chunks, used to seed byte-fallback (U+FFFD) correction so multi-byte characters whose tokens split across chunk boundaries still resolve to real strings
|
||||
- `logprob_text_offset`: the cumulative emitted text length, so `text_offset` in completion streaming logprobs stays absolute across chunks instead of restarting at 0
|
||||
|
||||
When a streamed `GenerateResponseStreamChoice` carries `logprobs`, the `token_id:N` placeholders are resolved per chunk and the resolved logprobs are attached to the corresponding streamed choice — for chat as `ChatCompletionLogProbs`, for completions converted to the flat `CompletionLogProbs` lists. Chunks without `logprobs` produce choices with `logprobs: null`.
|
||||
When a streamed `GenerateTokensStreamChoice` carries `logprobs`, the `token_id:N` placeholders are resolved per chunk and the resolved logprobs are attached to the corresponding streamed choice. For chat it is as `ChatCompletionLogProbs` and for completions it is converted to the flat `CompletionLogProbs` lists. Chunks without `logprobs` produce choices with `logprobs: null`.
|
||||
|
||||
## Example
|
||||
|
||||
|
||||
@@ -0,0 +1,71 @@
|
||||
# Tokens In <> Tokens Out API
|
||||
|
||||
`/inference/v1/generate` takes prompt token IDs and returns generated token IDs. It is the generation step of the `render → generate → derender` pipeline used in disaggregated serving (see the [Renderer APIs](renderer.md) and [Derenderer APIs](derenderer.md)).
|
||||
|
||||
The endpoint is registered on `vllm serve` with `--enable-scale-out`, and always with `--tokens-only`. `--tokens-only` also skips loading the tokenizer, so the server can't turn token IDs back into text.
|
||||
|
||||
## Output modes
|
||||
|
||||
The request field `output_mode` selects how much of the postprocessing the server does for the caller.
|
||||
|
||||
| `output_mode` | Response adds | Server needs |
|
||||
| --- | --- | --- |
|
||||
| `tokens` (default) | nothing, token IDs only | nothing |
|
||||
| `text` | `text` on every choice and decoded tokens with `bytes` in `logprobs` | a tokenizer |
|
||||
|
||||
`text` returns text and token IDs in a single call, without a separate [derender](derenderer.md) hop per chunk. This suits latency sensitive streaming, where the derender hop lands on every output token.
|
||||
|
||||
Every response and every stream chunk echoes `output_mode`. Servers that predate the field ignore it and return token IDs only with a 200, so check that the `output_mode` in the response matches the one you sent. Older servers don't return it.
|
||||
|
||||
```python
|
||||
import httpx
|
||||
|
||||
response = httpx.post(
|
||||
"http://localhost:8000/inference/v1/generate",
|
||||
json={
|
||||
"token_ids": [151644, 872, 198],
|
||||
"sampling_params": {"max_tokens": 32},
|
||||
"output_mode": "text",
|
||||
},
|
||||
).json()
|
||||
|
||||
assert response["output_mode"] == "text"
|
||||
print(response["choices"][0]["text"])
|
||||
print(response["choices"][0]["token_ids"])
|
||||
```
|
||||
|
||||
### Text
|
||||
|
||||
- `text` comes from the request's own detokenizer, so `skip_special_tokens`, `spaces_between_special_tokens`, `stop` and `include_stop_str_in_output` behave as they do on `/v1/completions`.
|
||||
- A matched stop string is cut from `text` unless `include_stop_str_in_output` is set. `token_ids` keeps every generated token, so decoding `token_ids` yourself brings the stop string back.
|
||||
- `/derender` only accepts `output_mode: "tokens"` responses and returns a 400 for text responses which are already detokenized.
|
||||
- With `stream: true`, each choice's `text` is the delta since the previous chunk. A chunk is sent whenever the engine output carries new text or a `finish_reason`, even with no new token IDs. That covers text held back for stop string matching and the final output after an abort.
|
||||
|
||||
### Logprobs
|
||||
|
||||
With `output_mode: "tokens"`, logprob entries carry `token_id:N` placeholders. With `output_mode: "text"`, they carry the decoded token strings and their UTF-8 `bytes` for the sampled token and every entry in `top_logprobs`. A server started with `--return-tokens-as-token-ids` always returns the placeholders, as `/v1/completions` does.
|
||||
|
||||
### Errors
|
||||
|
||||
The server returns a 400 for:
|
||||
|
||||
- `output_mode: "text"` on a server without a tokenizer (`--tokens-only` or `--skip-tokenizer-init`).
|
||||
- `output_mode: "text"` with `sampling_params.detokenize: false`.
|
||||
- An unsupported `output_mode` value.
|
||||
|
||||
For using `output_mode` with separate prefill and decode pools, see [Disaggregated Prefilling](../../features/disagg_prefill.md#generate-api-output-modes).
|
||||
|
||||
## Aborting requests
|
||||
|
||||
`POST /inference/v1/abort_requests` aborts in-flight requests. It is registered wherever `/inference/v1/generate` is and requires the API key when `--api-key` is set, like `/inference/v1/generate`.
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:8000/inference/v1/abort_requests \
|
||||
-H "Authorization: Bearer $VLLM_API_KEY" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"request_ids": ["generate-tokens-42"]}'
|
||||
```
|
||||
|
||||
The request ID is the `request_id` the server returned: the `X-Request-Id` header or body `request_id` you sent, prefixed with `generate-tokens-`. A missing `request_ids` returns a 400. The response is empty and the abort finishes in the background.
|
||||
|
||||
With `--tokens-only`, the same handler is also served at `POST /abort_requests` for existing deployments. That path doesn't require the API key, even when `--api-key` is set. See [Security](../../usage/security.md#unprotected-endpoints-no-api-key-required).
|
||||
@@ -2,14 +2,14 @@
|
||||
|
||||
In on-policy distillation, the student records its top-K candidate tokens at
|
||||
every response position, and the teacher provides the log probabilities of
|
||||
exactly those candidates as the distillation target. The caller passes the
|
||||
candidates as `prompt_logprob_token_ids`; the teacher runs a single prefill
|
||||
over the prompt plus response and gathers their log probabilities on the GPU at
|
||||
every scored row, starting from `prompt_logprob_start` (typically the prompt
|
||||
length minus one, so only response positions are scored). The result is a
|
||||
`[rows, K]` matrix whose row `i` scores prompt token
|
||||
`prompt_logprob_start + i + 1` and whose columns follow the candidate order, so
|
||||
it lines up position by position with the student's top-K.
|
||||
exactly those candidates as the distillation target. The caller passes one
|
||||
candidate list per scored row as `prompt_logprob_token_ids`; the teacher runs a
|
||||
single prefill over the prompt plus response and gathers their log
|
||||
probabilities on the GPU, starting from `prompt_logprob_start` (typically the
|
||||
prompt length minus one, so only response positions are scored). The result is
|
||||
a `[rows, K]` matrix whose row `i` scores prompt token
|
||||
`prompt_logprob_start + i + 1` and whose columns follow that row's candidate
|
||||
order, so it lines up position by position with the student's top-K.
|
||||
|
||||
## Quick start
|
||||
|
||||
@@ -22,13 +22,13 @@ output = llm.generate(
|
||||
TokensPrompt(prompt_token_ids=prompt_ids + response_ids),
|
||||
SamplingParams(
|
||||
max_tokens=1,
|
||||
prompt_logprob_token_ids=candidate_ids,
|
||||
prompt_logprob_token_ids=student_topk_ids, # [len(response_ids), K]
|
||||
prompt_logprob_start=len(prompt_ids) - 1,
|
||||
),
|
||||
)
|
||||
scores = output[0].prompt_token_id_logprobs
|
||||
# scores.shape == (len(response_ids), len(candidate_ids))
|
||||
# scores[i, j] = log p(candidate_ids[j] | prompt_ids + response_ids[:i])
|
||||
# scores.shape == (len(response_ids), K)
|
||||
# scores[i, j] = log p(student_topk_ids[i][j] | prompt_ids + response_ids[:i])
|
||||
```
|
||||
|
||||
The same parameters are accepted by the `/inference/v1/generate` HTTP endpoint
|
||||
@@ -42,13 +42,17 @@ import numpy as np
|
||||
scores = np.load(io.BytesIO(base64.b64decode(response["prompt_token_id_logprobs"])))
|
||||
```
|
||||
|
||||
The matrix has `max(prompt_len - 1 - prompt_logprob_start, 0)` rows. With a
|
||||
logits `--logprobs-mode`, it holds logits instead of log probabilities.
|
||||
`prompt_logprob_token_ids` is an integer array or a list of lists with exactly
|
||||
`prompt_len - 1 - prompt_logprob_start` rows; shorter rows are padded with `-1`,
|
||||
and every `-1` entry scores `-inf`. Pass an `int32` NumPy array, such as the
|
||||
student's top-K IDs, which is sent to the engine without conversion; nested
|
||||
lists of Python ints are much slower to copy and serialize. With a logits
|
||||
`--logprobs-mode`, the matrix holds logits instead of log probabilities.
|
||||
|
||||
## Requirements
|
||||
|
||||
- The V2 model runner (`VLLM_USE_V2_MODEL_RUNNER=1`).
|
||||
- `len(prompt_logprob_token_ids)` at most `--max-logprobs`.
|
||||
- The longest row at most `--max-logprobs` candidates.
|
||||
- No `--kv-sharing-fast-prefill`.
|
||||
|
||||
The request skips reading the prefix cache (local and KV connector), since
|
||||
@@ -56,8 +60,6 @@ cached rows have no logits to score.
|
||||
|
||||
## Limitations
|
||||
|
||||
- One candidate set per request, shared by every scored row; per-position
|
||||
candidates are planned ([#56860](https://github.com/vllm-project/vllm/issues/56860)).
|
||||
- Non-streaming only; `stream=true` requests are rejected.
|
||||
- Not exposed through the OpenAI-compatible endpoints.
|
||||
- If a prefill starts past the first scored row (e.g. `skip_reading_prefix_cache=False`
|
||||
|
||||
@@ -206,7 +206,7 @@ The following endpoints **do not require authentication** even when `--api-key`
|
||||
- `/init_weight_transfer_engine` - Initialize weight transfer engine for RLHF
|
||||
- `/update_weights` - Update model weights (can alter model behavior)
|
||||
- `/get_world_size` - Get distributed world size
|
||||
- `/abort_requests` - Abort in-flight requests (available with `--tokens-only`)
|
||||
- `/abort_requests` - Abort in-flight requests (available with `--tokens-only`. Use the authenticated `/inference/v1/abort_requests` otherwise)
|
||||
|
||||
**Utility endpoints:**
|
||||
|
||||
|
||||
Reference in New Issue
Block a user