Merge remote-tracking branch 'upstream/main' into patches

Signed-off-by: MOVZX <movzx@yahoo.com>
This commit is contained in:
2026-10-03 21:59:03 +07:00
1012 changed files with 61873 additions and 14499 deletions
+112
View File
@@ -0,0 +1,112 @@
---
name: check-api-compat
description: Determine the correct behavior of vLLM's OpenAI-compatible or Anthropic-compatible API by checking the official upstream spec, instead of guessing from vLLM's own code. Use for any bug, issue, PR, review, or question touching /v1/chat/completions, /v1/completions, /v1/responses, /v1/embeddings, /v1/messages, or their request/response models — including "is this a bug or intended", "what should this field default to", "does upstream allow this", and any change that adds, removes, or renames a field on those models. Do not use for vLLM-internal APIs (engine, scheduler, LLM class) or for endpoints with no upstream counterpart.
---
# Checking vLLM API compatibility against the upstream spec
vLLM's OpenAI and Anthropic server surfaces are *reimplementations* of someone
else's spec. vLLM's own code, tests, and docstrings are not evidence of correct
behavior — they are the thing under test. Establish what upstream says first,
then compare.
Never answer "what should this field do" from memory. Field names, types, and
especially defaults drift between spec versions.
## 1. Pin down the question
Name three things before looking anything up:
- **Which surface** — OpenAI-compatible or Anthropic-compatible. They are separate
code paths in vLLM and separate specs.
- **Which endpoint**, by path.
- **Which field or behavior** — a request field, a response field, a streaming
event shape, an error code, or a status code.
## 2. Read vLLM's implementation
In a vLLM checkout, the request/response models live beside each endpoint:
| surface | path |
| --- | --- |
| OpenAI chat | `vllm/entrypoints/openai/chat_completion/` |
| OpenAI completions | `vllm/entrypoints/openai/completion/` |
| OpenAI responses | `vllm/entrypoints/openai/responses/` |
| OpenAI embeddings | `vllm/entrypoints/pooling/embed/` |
| OpenAI shared base models, errors, usage | `vllm/entrypoints/serve/engine/protocol.py` |
| Anthropic messages | `vllm/entrypoints/anthropic/` |
| audio | `vllm/entrypoints/speech_to_text/*/protocol.py` |
Each directory holds `protocol.py` (the Pydantic models), `serving.py` (behavior),
and `api_router.py` (routing and status codes). Read the field declaration *and*
the code that consumes it — a field can be declared faithfully and then ignored,
which is still an incompatibility.
## 3. Fetch the upstream spec
Prefer a source that can actually be read as text, in this order.
### OpenAI
1. The OpenAPI spec — authoritative, machine-readable, and fetches cleanly:
`https://raw.githubusercontent.com/openai/openai-openapi/master/openapi.yaml`
2. The installed `openai` Python SDK's typed params, which track the spec closely.
Locate them with `python -c "import openai,os;print(os.path.dirname(openai.__file__))"`,
then read e.g. `types/chat/completion_create_params.py`,
`types/chat/chat_completion.py`, `types/chat/chat_completion_chunk.py`,
`types/responses/`. Note the SDK version — an old SDK is an old spec.
3. The prose reference at `https://platform.openai.com/docs/api-reference/...`
for semantics the types do not carry. It is script-heavy and often fetches
poorly; treat a failed fetch as a fetch failure, not as absence of a field.
### Anthropic
1. `https://docs.claude.com/en/api/messages` and
`https://docs.claude.com/en/api/messages-streaming` — these render as clean
text and are the primary source for both fields and streaming event order.
2. The installed `anthropic` SDK's `types/message_create_params.py`,
`types/message.py`, `types/raw_message_stream_event.py`.
Say which source and which version answered the question, so the finding can be
re-checked later.
## 4. Compare on all four axes
A field is compatible only when **name**, **type**, **default**, and **semantics**
all match. Most real bugs hide in the last two: a field that exists with the right
type but the wrong default, or one that is accepted and silently dropped.
Also check, when relevant:
- **Streaming** — chunk shape, the order of events, and the terminal event
(`data: [DONE]` for Chat Completions and Completions; `response.completed`,
`response.failed`, or `response.incomplete` with no `[DONE]` for Responses;
`message_stop` for Anthropic).
- **Errors** — HTTP status *and* the error body's `type`/`code` strings.
- **Omitted vs null** — upstream often distinguishes an absent field from an
explicit `null`. Pydantic defaults erase that distinction unless the model is
written to preserve it.
## 5. Classify before calling it a bug
Three outcomes, and they are not the same:
- **Matches upstream** — no action.
- **vLLM extension** — a field upstream has no name for at all (`vllm_xargs`,
`chat_template_kwargs`, sampling knobs like `min_p`). These are intentional and
allowed. Do not report them as violations. Check per surface: `top_k` is an
extension on OpenAI but a real upstream field on Anthropic Messages.
- **Incompatibility** — a field that *shares a name with upstream* but differs in
type, default, or behavior, or an upstream field that is missing, ignored, or
malformed. This is the reportable case.
An unsupported-but-declared field is its own subcase: decide whether vLLM rejects
it explicitly or accepts and ignores it, and report which, since silently ignoring
a documented field is the more damaging failure.
## 6. Report
State: the endpoint and field, what upstream specifies (with the source), what
vLLM does (with `file:line`), which of the three outcomes it is, and the smallest
fix. When behavior turns on a spec detail that is easy to disbelieve, quote the
upstream line rather than paraphrasing it.
@@ -0,0 +1,4 @@
interface:
display_name: "Check API Compat"
short_description: "Check vLLM's OpenAI/Anthropic-compatible API against the upstream spec."
default_prompt: "Use the vLLM API compatibility skill to check this endpoint or field against the upstream OpenAI or Anthropic spec."
+190
View File
@@ -0,0 +1,190 @@
---
name: pr-checklist
description: Prepare a vLLM change for human PR review or re-review, or analyze an open pull request. Use to check design fit, behavioral coverage, performance evidence, diff quality, and closure of previous feedback before requesting maintainer attention.
---
# PR Checklist
This SKILL is designed to help systematically enforce vLLM's code quality and contribution standards during the pull request process,
and to reduce the burden of manual code review on maintainers.
## Usage
The following checklist can be used to polish a change in preparation for a vLLM pull request, analyze an open pull request (either as the author or as a reviewer), or ensure that all contribution standards are met before requesting maintainer attention. To use this checklist effectively, go through each item and verify that it has been addressed in the diff / pull request.
This is intended to be used with a human in the loop: when running programmatically, prepare notes on every section to ensure a thorough report, with a clear header summarizing the findings and areas needing attention. When running interactively, present this report to the user with a clear list of actionable follow-up tasks and design decisions to make, then iterate.
### Re-Review
To update a PR following a review, read all comments and threads on the PR. For each concrete concern, work with your human operator to decide on a plan of action to address each either by a direct reply or by updating the change and acknowledging with a comment. Ensure the PR description remains fresh and all relevant documentation (including comments) in the change is up-to-date.
`gh pr view --comments` does not include inline review comments. Set `pr` to the PR number, then fetch all pages of conversation comments, review submissions, and inline comments (including replies):
```bash
gh api --paginate "repos/vllm-project/vllm/issues/$pr/comments"
gh api --paginate "repos/vllm-project/vllm/pulls/$pr/reviews"
gh api --paginate "repos/vllm-project/vllm/pulls/$pr/comments"
```
Use `in_reply_to_id` to group inline replies with their parent comments. Use GraphQL `reviewThreads` when resolved or outdated thread status is needed.
## Sections
### 1. Design Fit
Does the change align with the overall design and architecture of vLLM? Does it introduce any design inconsistencies or anti-patterns? Are there any potential long-term maintenance concerns related to this change?
#### 1.1: Impact on Core Components
vLLM's core components are shared across many different deployments. Changes to these files have high risk and changes should be structured in order to be easily maintainable and minimally invasive.
Such components include:
- The Model Runner (`vllm/v1/worker/*/model_runner.py`)
- The Scheduler (`vllm/v1/core/sched/*`)
- Core infrastructure (`vllm/v1/core/*`, including KV cache managers and utilities)
etc.
Changes should be structured to minimize complexity added to these files. Consider refactoring or isolating new functionality to reduce the impact on core components, both in terms of code maintainability and potential for introducing bugs.
Nontrivial logic associated with non-standard/non-default functionality should not be added to core/common code paths and files.
#### 1.2: Replication
Some changes may duplicate code that already exists elsewhere in the codebase. Does the change introduce unnecessary replication, or could existing functionality be reused instead? Consider refactoring to reduce duplication and improve maintainability.
Consult the affected area's agent instructions and contribution guides for its reuse conventions. See the [model implementation guide](../../../docs/contributing/model/basic.md) for model code and the [MRV2 development guidance](../../../docs/design/model_runner_v2.md#development-philosophy). Justify any duplication needed to isolate functionality and limit its impact on existing workflows.
#### 1.3: Complexity
Changes which increase the complexity of the codebase should be carefully scrutinized. Specifically those which add new abstractions, dependencies, or branching cases to existing components should be evaluated for their necessity and potential impact on maintainability.
Consider whether the added complexity could be mitigated through refactoring, modularization, or other design improvements. Aim to keep the codebase as simple and understandable as possible while still achieving the desired functionality.
As a rule of thumb, the complexity of a change can be estimated by the number of new lines of code added to the main codebase:
- Low complexity: < 50 lines. Generally easy to review and unlikely to introduce significant complexity.
- Medium complexity: 50-250 lines. Requires careful review and consideration of potential impact on maintainability.
- High complexity: > 250 lines. May introduce significant complexity and requires thorough review and justification.
You may exclude test cases and documentation updates from this line count, as they typically do not contribute significantly to the complexity of the main change. Note that line count is only a coarse indicator, meant to guide and flag possible overly-complex contributions. Use judgement when assessing the true semantic complexity and implied review burden.
Bug fixes should aim to remain below the "High complexity" bar. Those which exceed this threshold often warrant additional scrutiny and justification. Redesigning core components, modifying core interfaces, or otherwise introducing significant complexity in order to patch a bug is very often discouraged and should be carefully justified.
A PR that adds significant complexity or a substantial amount of new code must be well justified. A new feature should be relevant to, or requested by, enough users to warrant its size and complexity. A performance optimization should deliver end-to-end improvements on workloads that matter, and those gains should be large enough to outweigh the complexity it adds.
#### 1.4: Correctness and Compatibility
Ensure that all changes to vLLM are sound and do not violate existing contracts. Specifically, verify that all affected deployments which share an execution pathway with the changed code will remain correct. Trace related implementations, alternative backends sharing the same flow, and affected call sites of modified functions.
When adding features or changing contracts, ensure that relevant feature compatibility is assessed. Identify any potential incompatibilities or breaking changes and justify them explicitly in the PR notes.
When adding functionality, consider parallels with existing designs and identify opportunities to simplify and unify them rather than adding a separate implementation alongside them.
#### 1.5: Tradeoffs
When making a design change, adding a feature, or contributing a performance improvement, assess the tradeoffs and potential downsides. When changing a default backend to improve performance as measured on one configuration, could this cause a decrease in performance for another configuration? Could a new architecture change accelerate one workload but cause a feature incompatibility or performance regression in another? Ensure tradeoffs are clearly noted and validation is sufficient to catch potential lurking regressions or drawbacks.
Check that the change does not introduce CPU/GPU synchronization on the main model execution thread during steady-state inference. Inspect explicit synchronization, host reads of GPU tensors, boolean tensors as indices, and transfers that can block the host. Run the affected configuration with `VLLM_GPU_SYNC_CHECK=error` to confirm a concern; passing CI is insufficient when CI does not exercise that configuration. See the [MRV2 async execution design](../../../docs/design/model_runner_v2.md#2-async-first) for examples.
#### 1.6: Model Runner V1 Deprecation
Account for Model Runner V1 (MRV1) deprecation when reviewing model-runner changes. Target new GPU model-runner features at Model Runner V2 (MRV2) and follow its [design and development guidance](../../../docs/design/model_runner_v2.md#development-philosophy). For fixes needed by deployments still using MRV1, explain why the legacy path needs the change and check whether MRV2 is also affected.
### 2. Testing and Validation
All serving-engine changes contributed to vLLM must be _validated_: bug fixes must be tested on a reproducer, and new features or performance optimizations must be run locally and/or benchmarked to ensure correctness and effectiveness. It is unacceptable, for example, to contribute a bugfix patch without running vLLM end-to-end and verifying that it resolves the issue as intended.
Beyond _validation_, many (but not all) changes to vLLM should be _tested_: this includes writing new tests, updating existing tests, and ensuring that the test suite continues to provide reliable coverage of the affected functionality.
#### 2.1: Test Coverage and Effectiveness
When adding or modifying tests, consider both the coverage and the effectiveness of the test suite. Coverage refers to the extent to which the codebase is exercised by tests, while effectiveness refers to the ability of the tests to catch regressions and ensure correctness.
Assess whether the included test cases are meaningful, concise, and maintainable. Testing should not be exhaustive: local validation is typically sufficient to ensure a baseline correctness of the change. Contributed tests should focus exclusively on areas that are deemed prone to errors, regressions, or complex interactions within the codebase. Recommend deletion of redundant, trivial, or otherwise unnecessary tests: this is one of the most common complaints raised during code reviews.
Verify that testing actually exercises the new path, rather than passing through an unchanged path or fallback. Ensure added speculative decoding correctness tests assert nonzero acceptance rate, selection tests identify the desired backend, etc.
Aim to write tests that are meaningful, targeted, and maintainable. Avoid redundant or trivial tests that do not contribute to the overall reliability of the test suite. Avoid using mock objects, especially when highly coupled with internal logic.
#### 2.2: Test Reliability and Robustness
Some tests are considered "flaky": they may pass or fail intermittently without any changes to the codebase. Flaky tests can undermine confidence in the test suite and block successful CI runs. Analyze the added tests for potential flakiness and ensure they are reliable and robust.
Frequent sources of flaky tests include:
- Assertions requiring bit-exact output correctness when minor floating-point variation is expected. For example, a change in default kernel selection priority may cause exact-floating-point-correctness assertions to fail.
- Tests that depend on the order of execution or the state of shared resources, which can lead to intermittent failures when tests are run in parallel or in different environments.
#### 2.3: Test Automation and Integration
vLLM's CI runs in Buildkite, with the launch configuration under `.buildkite/`. Ensure new tests run in an appropriate CI workflow when the required resources are available; add or update a workflow where needed. If CI cannot exercise the changed path because the required hardware or checkpoint is unavailable, document the missing coverage, the resource constraint, and validation performed outside CI. A CI coverage gap does not waive validation: the human submitter must confirm the required validation before publishing the PR.
### 3. Code Quality and Style
Code quality is another of the primary areas that maintainer effort is spent to address. Carefully audit the changes for adherence to coding standards, readability, maintainability, and overall design principles.
#### 3.1: Comments
Keep comments concise and use them sparingly. Consider whether the same information can be conveyed in fewer lines, and prefer clear code with meaningful, concise variable names, and fewer comments overall. Use comments to explain the reasoning behind complex logic, non-obvious decisions, or important context that is not immediately apparent from the code itself. Context regarding specific user decisions made during the design phase are often not needed and can instead be included as notes in the PR description instead.
When writing new functions, consider when a docstring may be useful. Most short helper functions do not need a docstring, but more complex functions or those forming part of the public API should include one. In particular, docstrings are useful when adding custom kernels or other performance-optimized implementations of otherwise-straightforward logic, as they make clear the input and output shapes/dtypes/assumptions, and can shed light on hard-to-follow implementations.
For bug fixes, there is no need to include details of the original bug and explanation of why the pre-changed version of the code would be incorrect. Those details should be in the PR description instead.
#### 3.2: Documentation and Examples
Ensure that any new features, changes, or public APIs are properly documented. This includes updating relevant documentation files under `docs/` and providing usage examples where applicable. Refer to `supported_models.md` for new models and `examples/` for usage examples.
#### 3.3: Helper Functions
Identify and inline trivially-simple helper functions which have only one call site.
#### 3.4: Formatting
Avoid formatting or whitespace changes that are unrelated to the PR, as long as pre-commit still passes.
### 4. Pull Request Contents
This section pertains to the contents of the pull request itself, rather than the changes.
When acting as the author, prepare or update these contents. When acting as a reviewer, assess them and report missing context or evidence.
Start from `.github/PULL_REQUEST_TEMPLATE.md`, retaining its sections and checklist. When creating a PR interactively, use `gh pr create --template .github/PULL_REQUEST_TEMPLATE.md`. For noninteractive creation or editing, fill out a copy of that template and pass the completed file with `--body-file`.
#### 4.1: Description and Context
Ensure the PR description has a short 1-2 line summary of the changes to facilitate quick understanding by reviewers. This section should link relevant issues, predecessor/blocking PRs, and provide any necessary context for the changes.
Keep Claims to a few concrete bullets and explain motivation briefly. Keep explanations in Validation and Details concise too, while retaining the evidence, root cause, tradeoffs, and limitations needed to assess the change. Link full logs or use collapsible HTML details for lengthy supporting material rather than polluting the main narrative.
#### 4.2: Claims and Supporting Evidence
Ensure the PR makes clear, concise claims about the accomplishment:
- For bug fixes, what specific issue is being resolved and how can it be verified?
- For new features, what functionality is being added and how can it be utilized?
- For refactoring, what was the motivation and what improvements are expected?
- For performance optimizations, what specific model/hardware/serving configurations & workloads are being targeted and how can the performance gains be measured?
For the stated claims, ensure the PR provides sufficient evidence, examples, or references to support them, making it easy for reviewers to verify the correctness and impact of the changes.
Summarize model evaluations and benchmarks in compact tables with the configuration, baseline, changed result, and reproduction commands. For repeated benchmarks, report the run count and variability. For kernel performance changes, include before/after profile visualizations/timelines when they help explain the measured gains. Link relevant CI failures or regressing PRs when they motivate the change.
- For bug fixes, ensure the reproducer is clearly provided and was validated to confirm the issue and the resolution.
- For new features, ensure that usage examples are included and validated to confirm that the feature works as intended.
- For refactoring, ensure that changes are sufficiently validated at appropriate test levels (unit is sufficient for minor refactors, E2E serving is required for significant refactoring).
- For performance optimizations, ensure that benchmarks or tests are included to validate the claimed performance improvements. Ideally, an end-to-end evaluation should demonstrate speedup on a concrete serving example. Microbenchmark-only validation is often insufficient: make a note if end-to-end evaluation is not included (e.g. due to lack of access to sufficient hardware to fully validate).
#### 4.3: Root-Cause Analysis
When fixing bugs or issues, provide a clear root-cause analysis. Explain what caused the problem, how it was identified, and why the chosen solution effectively addresses it. A patch which resolves the issue without a proper root-cause analysis may lead to recurring problems and increased maintenance overhead.
#### 4.4: Implementation Details
Provide a detailed explanation of the implementation, including design decisions, trade-offs, and any relevant technical details. Highlight any non-obvious choices, tradeoffs, potential limitations, and areas where maintainer attention is required.
#### 4.5: Contributing Guide
Ensure that the PR follows the [contributing guide](https://docs.vllm.ai/en/latest/contributing), including adhering to coding standards, commit message conventions, and any other project-specific guidelines. This helps maintain consistency and quality across the codebase.
+4
View File
@@ -24,6 +24,9 @@ steps:
- tests/kernels/test_onednn.py
- tests/kernels/test_awq_int4_to_int8.py
- tests/kernels/quantization/test_cpu_fp8_scaled_mm.py
- tests/kernels/quantization/test_zen_da8w4.py
- vllm/model_executor/kernels/linear/mixed_precision/zentorch.py
- vllm/model_executor/layers/fused_moe/experts/zentorch_moe.py
- tests/kernels/mamba/cpu/test_cpu_gdn_ops.py
- tests/kernels/mamba/test_cpu_short_conv.py
- tests/kernels/mamba/test_causal_conv1d.py
@@ -46,6 +49,7 @@ steps:
tests/kernels/test_onednn.py \
tests/kernels/test_awq_int4_to_int8.py \
tests/kernels/quantization/test_cpu_fp8_scaled_mm.py \
tests/kernels/quantization/test_zen_da8w4.py \
tests/kernels/mamba/cpu/test_cpu_gdn_ops.py \
tests/kernels/mamba/test_causal_conv1d.py \
tests/kernels/mamba/test_mamba_ssm.py \
+1 -1
View File
@@ -62,7 +62,7 @@ steps:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests &&
pytest -v -s v1/engine --ignore v1/engine/test_preprocess_error_handling.py &&
pytest -v -s v1/engine --ignore v1/engine/test_preprocess_error_handling.py --ignore v1/engine/test_core_engine_actor_manager.py &&
pytest -v -s test_config.py'
- label: V1 e2e (2 GPUs)
@@ -77,6 +77,7 @@ steps:
'export VLLM_WORKER_MULTIPROC_METHOD=spawn &&
cd tests &&
pytest -v -s entrypoints/serve --ignore=entrypoints/serve/dev/rpc
--ignore=entrypoints/serve/dev/rlhf/test_weight_operation_metrics.py
--deselect="tests/entrypoints/serve/dev/rlhf/state_transitions/test_pause_resume.py::TestPauseResume::test_clear_cache_preserves_output_and_controls_prefix_cache[MRV1]" &&
PYTHONPATH=/workspace/vllm pytest -v -s entrypoints/serve/dev/rpc'
+2 -2
View File
@@ -99,14 +99,14 @@ steps:
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests && pip install dpctl &&
pytest -v -s v1/core --ignore=v1/core/test_reset_prefix_cache_e2e.py --ignore=v1/core/test_scheduler_e2e.py &&
pytest -v -s v1/engine --ignore=v1/engine/test_output_processor.py &&
pytest -v -s v1/engine --ignore=v1/engine/test_output_processor.py --ignore=v1/engine/test_core_engine_actor_manager.py &&
pytest -v -s v1/sample --ignore=v1/sample/test_logprobs.py --ignore=v1/sample/test_logprobs_e2e.py -k "not test_topk_only and not test_topp_only and not test_topk_and_topp" &&
pytest -v -s v1/worker --ignore=v1/worker/test_gpu_model_runner.py --ignore=v1/worker/test_worker_memory_snapshot.py &&
pytest -v -s v1/structured_output &&
pytest -v -s v1/test_serial_utils.py &&
pytest -v -s v1/e2e/general/test_correctness_sliding_window.py &&
pytest -v -s v1/spec_decode --ignore=v1/spec_decode/test_max_len.py --ignore=v1/spec_decode/test_speculators_eagle3.py --ignore=v1/spec_decode/test_acceptance_length.py --ignore=v1/spec_decode/test_speculators_correctness.py --deselect=tests/v1/spec_decode/test_mtp.py::test_glm_mtp_defers_lm_head &&
pytest -v -s v1/kv_connector/unit --ignore=v1/kv_connector/unit/test_lmcache_integration.py --ignore=v1/kv_connector/unit/test_hf3fs_client.py --ignore=v1/kv_connector/unit/test_hf3fs_connector.py --ignore=v1/kv_connector/unit/test_hf3fs_metadata_server.py --ignore=v1/kv_connector/unit/test_offloading_connector.py --deselect "v1/kv_connector/unit/test_nixl_connector.py::test_abort_timeout_on_prefiller[ray]" --ignore=v1/kv_connector/unit/test_nixl_connector_hma.py'
pytest -v -s v1/kv_connector/unit --ignore=v1/kv_connector/unit/test_lmcache_integration.py --ignore=v1/kv_connector/unit/test_hf3fs_client.py --ignore=v1/kv_connector/unit/test_hf3fs_connector.py --ignore=v1/kv_connector/unit/test_hf3fs_metadata_server.py --ignore=v1/kv_connector/unit/test_offloading_connector.py --ignore=v1/kv_connector/unit/test_nixl_connector_hma.py --deselect=tests/v1/kv_connector/unit/test_nixl_connector.py::test_abort_timeout_on_prefiller[ray]'
parallelism: 2
- label: "XPU server test"
key: xpu-server-test
+16 -3
View File
@@ -71,9 +71,11 @@ if [[ -z "${BK_TOKEN}" ]]; then
fi
AUDIENCE="pytorch-cross-repo-ci-relay"
# Pre-flight only; the token minted here is discarded. Checking now means an
# unusable OIDC setup costs seconds rather than being discovered after hours of
# polling. The token that is actually sent is minted after the wait, below.
# OIDC redaction also requires the unavailable Job API socket.
OIDC_TOKEN="$(buildkite-agent oidc request-token --skip-redaction --audience "${AUDIENCE}" 2>/dev/null)"
if [[ -z "${OIDC_TOKEN}" ]]; then
if [[ -z "$(buildkite-agent oidc request-token --skip-redaction --audience "${AUDIENCE}" 2>/dev/null)" ]]; then
echo "could not mint a Buildkite OIDC token -- skipping report"
exit 0
fi
@@ -142,7 +144,7 @@ POLL_INTERVAL_S="${CRCR_POLL_INTERVAL_S:-900}"
# meaningless whenever the step was itself delayed.
#
# Resolved after the first fetch, from the build's own created_at.
MAX_BUILD_AGE_S="${CRCR_MAX_BUILD_AGE_S:-25200}"
MAX_BUILD_AGE_S="${CRCR_MAX_BUILD_AGE_S:-14400}"
WAIT_DEADLINE=""
# Used until created_at is known. Without it, an API outage on the very first
# request leaves the deadline unresolved and the loop spins to the step timeout.
@@ -220,6 +222,17 @@ done
# Resolved from this script's location: the pipeline runs it from
# /vllm-workspace/tests, so a repo-relative path would not resolve.
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# Minted here, not before the poll loop: a Buildkite OIDC token defaults to a
# five-minute lifetime and the loop above can run to the build-age deadline.
# Build 91827 waited 6h48m and then took a 401 on all 360 callbacks from a
# token that had expired hours earlier.
OIDC_TOKEN="$(buildkite-agent oidc request-token --skip-redaction --audience "${AUDIENCE}" 2>/dev/null)"
if [[ -z "${OIDC_TOKEN}" ]]; then
echo "could not mint a Buildkite OIDC token after the wait -- skipping report"
exit 0
fi
python3 "${SCRIPT_DIR}/crcr_report.py" \
--build-json "${BUILD_JSON}" \
--callback-url "${CALLBACK_URL}" \
@@ -37,14 +37,14 @@ case "${test_suite}" in
cd tests
pytest -v -s v1/core --ignore=v1/core/test_reset_prefix_cache_e2e.py --ignore=v1/core/test_scheduler_e2e.py
pytest -v -s v1/engine --ignore=v1/engine/test_output_processor.py
pytest -v -s v1/engine --ignore=v1/engine/test_output_processor.py --ignore=v1/engine/test_core_engine_actor_manager.py
pytest -v -s v1/sample --ignore=v1/sample/test_logprobs.py --ignore=v1/sample/test_logprobs_e2e.py -k "not test_topk_only and not test_topp_only and not test_topk_and_topp"
pytest -v -s v1/worker --ignore=v1/worker/test_gpu_model_runner.py --ignore=v1/worker/test_worker_memory_snapshot.py
pytest -v -s v1/structured_output
pytest -v -s v1/test_serial_utils.py
pytest -v -s v1/e2e/general/test_correctness_sliding_window.py
pytest -v -s v1/spec_decode --ignore=v1/spec_decode/test_max_len.py --ignore=v1/spec_decode/test_speculators_eagle3.py --ignore=v1/spec_decode/test_acceptance_length.py --ignore=v1/spec_decode/test_speculators_correctness.py --deselect=tests/v1/spec_decode/test_mtp.py::test_glm_mtp_defers_lm_head
pytest -v -s v1/kv_connector/unit --ignore=v1/kv_connector/unit/test_multi_connector.py --ignore=v1/kv_connector/unit/test_example_connector.py --ignore=v1/kv_connector/unit/test_lmcache_integration.py --ignore=v1/kv_connector/unit/test_hf3fs_client.py --ignore=v1/kv_connector/unit/test_hf3fs_connector.py --ignore=v1/kv_connector/unit/test_hf3fs_metadata_server.py --ignore=v1/kv_connector/unit/test_offloading_connector.py
pytest -v -s v1/kv_connector/unit --ignore=v1/kv_connector/unit/test_multi_connector.py --ignore=v1/kv_connector/unit/test_example_connector.py --ignore=v1/kv_connector/unit/test_lmcache_integration.py --ignore=v1/kv_connector/unit/test_hf3fs_client.py --ignore=v1/kv_connector/unit/test_hf3fs_connector.py --ignore=v1/kv_connector/unit/test_hf3fs_metadata_server.py --ignore=v1/kv_connector/unit/test_offloading_connector.py --ignore=v1/kv_connector/unit/test_nixl_connector_hma.py --deselect=tests/v1/kv_connector/unit/test_nixl_connector.py::test_abort_timeout_on_prefiller[ray]
;;
server)
pip install av
@@ -57,7 +57,7 @@ case "${test_suite}" in
cd tests
pytest -v -s quantization/test_auto_round.py
pytest -v -s quantization/test_online.py
pytest -v -s quantization/test_online.py --deselect=tests/quantization/test_online.py::test_online_quantization_loads_real_weights
;;
compressed-tensors-fp8)
cd tests
+570 -567
View File
File diff suppressed because it is too large Load Diff
+2 -2
View File
@@ -50,9 +50,9 @@ steps:
- pytest -v -s basic_correctness/memory/sleep_mode
mirror:
amd:
label: ":amd: (MI355 DPX) Basic Correctness Sleep Mode"
label: ":amd: (MI300) Basic Correctness Sleep Mode"
dind: false
device: mi355_dpx
device: mi300_1
timeout_in_minutes: 60
depends_on:
- image-build-amd
+129
View File
@@ -12,6 +12,30 @@ steps:
commands:
- export VLLM_TEST_CLEAN_GPU_MEMORY=1
- pytest -v -s tests/compile/correctness_e2e/test_sequence_parallel.py
mirror:
amd:
label: ":amd: (MI355) Sequence Parallel Correctness"
dind: false
device: mi355_2
num_devices: 2
timeout_in_minutes: 180
depends_on:
- image-build-amd
source_file_dependencies:
- .buildkite/test_areas/compile.yaml
- vllm/compilation/
- vllm/config/
- vllm/distributed/
- vllm/model_executor/kernels/linear/
- vllm/model_executor/layers/
- vllm/model_executor/models/llama.py
- vllm/v1/attention/
- vllm/v1/worker/
- vllm/v1/cudagraph_dispatcher.py
- vllm/platforms/rocm.py
- tests/compile/correctness_e2e/test_sequence_parallel.py
- tests/models/registry.py
- tests/utils.py
- label: ":nvidia: (B200) AsyncTP Correctness"
key: asynctp-correctness-tests-b200
@@ -23,6 +47,38 @@ steps:
commands:
- export VLLM_TEST_CLEAN_GPU_MEMORY=1
- pytest -v -s tests/compile/correctness_e2e/test_async_tp.py
mirror:
amd:
label: ":amd: (MI355) AsyncTP Correctness"
dind: false
device: mi355_2
num_devices: 2
timeout_in_minutes: 60
working_dir: "/vllm-workspace/"
depends_on:
- image-build-amd
source_file_dependencies:
- .buildkite/test_areas/compile.yaml
- vllm/compilation/
- vllm/config/
- vllm/distributed/
- vllm/model_executor/layers/
- vllm/model_executor/kernels/linear/
- vllm/model_executor/models/llama.py
- vllm/model_executor/models/qwen3.py
- vllm/v1/attention/
- vllm/_aiter_ops.py
- vllm/platforms/rocm.py
- vllm/transformers_utils/repo_utils.py
- tests/compile/correctness_e2e/test_async_tp.py
- tests/compile/fusions_e2e/common.py
- tests/models/registry.py
- tests/utils.py
commands:
- export VLLM_TEST_CLEAN_GPU_MEMORY=1
- pytest -v -s tests/compile/correctness_e2e/test_async_tp.py::test_rocm_async_tp_bf16_output_correctness
- pytest -v -s tests/compile/correctness_e2e/test_async_tp.py::test_async_tp_pass_correctness -k RedHatAI
- pytest -v -s tests/compile/correctness_e2e/test_async_tp.py::test_async_tp_pass_nvfp4_correctness
- label: ":nvidia: (H100) Distributed Compile"
key: distributed-compile-unit-tests-2xh100
@@ -188,6 +244,27 @@ steps:
# Qwen/Deepseek requires +quant_fp8 as -quant_fp8 rms+quant fusion is not supported
# Run just llama3 (fp8 & fp4) for all config combinations (only inductor partition)
- pytest -v -s tests/compile/fusions_e2e/test_tp1_quant.py -k "inductor_partition and (FLASHINFER and not +rms_norm and (not +quant_fp8 or +quant_fp8 and (qwen3 or deepseek)) or llama-3)"
mirror:
amd:
label: ":amd: (MI355 DPX) Fusion E2E FP8 Config Sweep"
dind: false
device: mi355_dpx
timeout_in_minutes: 45
working_dir: "/vllm-workspace/"
depends_on:
- image-build-amd
source_file_dependencies:
- .buildkite/test_areas/compile.yaml
- csrc/quantization/
- vllm/compilation/
- vllm/model_executor/
- vllm/v1/attention/
- vllm/_aiter_ops.py
- vllm/platforms/rocm.py
- tests/compile/fusions_e2e/
commands:
- rocm-smi
- pytest -v -s tests/compile/fusions_e2e/test_tp1_quant.py -k "test_tp1_fp8_fusions and inductor_partition and not +rms_norm and not True and ((Llama-3 and TRITON_ATTN) or (Qwen3-30B and +quant_fp8 and TRITON_ATTN) or (DeepSeek-Coder and +quant_fp8 and TRITON_MLA))"
- label: ":nvidia: (H100) Fusion E2E TP2 Quick"
key: fusion-e2e-tp2-quick-h100
@@ -267,6 +344,31 @@ steps:
- nvidia-smi
# Run just llama3 (fp8 & bf16) for all config combinations
- pytest -v -s tests/compile/fusions_e2e/test_tp2_ar_rms.py -k "llama-3"
mirror:
amd:
label: ":amd: (MI355) Fusion E2E TP2 AR-RMS FP8 Config Sweep"
dind: false
device: mi355_2
num_devices: 2
timeout_in_minutes: 45
working_dir: "/vllm-workspace/"
depends_on:
- image-build-amd
source_file_dependencies:
- vllm/compilation/
- vllm/model_executor/
- vllm/distributed/
- vllm/v1/attention/
- vllm/_aiter_ops.py
- vllm/platforms/rocm.py
- tests/compile/fusions_e2e/
- tests/utils.py
- tests/conftest.py
- csrc/quantization/
commands:
# Existing MI300 jobs own BF16; TP2 Quick owns the excluded Llama cell.
- amd-smi
- VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_CUSTOM_AR=1 pytest -v -s tests/compile/fusions_e2e/test_tp2_ar_rms.py -k 'test_tp2_ar_rms_fp8_fusions and Llama-3 and ROCM_AITER_UNIFIED_ATTN and (dynamo_partition or +quant_fp8 or +rms_norm)'
- label: ":nvidia: (H100) Fusion E2E TP2 AsyncTP Config Sweep"
key: fusion-e2e-tp2-asynctp-config-sweep-h100
@@ -334,3 +436,30 @@ steps:
# for ar-rms-quant-fp4, also sweep llama3
- pytest -v -s tests/compile/fusions_e2e/test_tp2_ar_rms.py -k "(FLASHINFER and inductor_partition and not +rms_norm and (not +quant_fp8 or +quant_fp8 and (qwen3 or deepseek))) or Llama-3.1-8B-Instruct-FP4"
- pytest -v -s tests/compile/fusions_e2e/test_tp2_async_tp.py -k "FLASHINFER and inductor_partition and not +rms_norm and (not +quant_fp8 or +quant_fp8 and (qwen3 or deepseek))"
mirror:
amd:
label: ":amd: (MI355) Fusion E2E TP2 FP8 BF16 rewrites"
dind: false
device: mi355_2
num_devices: 2
timeout_in_minutes: 45
working_dir: "/vllm-workspace/"
depends_on:
- image-build-amd
source_file_dependencies:
- vllm/compilation/
- vllm/model_executor/
- vllm/distributed/
- vllm/v1/attention/
- vllm/_aiter_ops.py
- vllm/platforms/rocm.py
- tests/compile/fusions_e2e/
- tests/utils.py
- tests/conftest.py
- csrc/quantization/
commands:
# Dummy-weight rewrite coverage; trained FP8 AR/RMS equivalence is unresolved.
# TODO: FP4 output-quant and trained MXFP4 AR/RMS fusion remain unsupported.
- amd-smi
- VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_CUSTOM_AR=1 pytest -v -s tests/compile/fusions_e2e/test_tp2_ar_rms.py -k 'test_tp2_ar_rms_fp8_fusions and qwen3 and ROCM_AITER_UNIFIED_ATTN and +quant_fp8 and (dynamo_partition or not +rms_norm)'
- pytest -v -s tests/compile/fusions_e2e/test_tp2_rocm_moe.py
+49
View File
@@ -439,6 +439,27 @@ steps:
- pytest -v -s tests/distributed/test_nccl_symm_mem.py
- pytest -v -s tests/v1/distributed/test_dbo.py
- pytest -v -s tests/distributed/test_mnnvl_alltoall.py
mirror:
amd:
label: ":amd: (MI355) Distributed AgRs All2All"
dind: false
device: mi355_2
num_devices: 2
timeout_in_minutes: 30
working_dir: "/vllm-workspace/"
depends_on:
- image-build-amd
source_file_dependencies:
- .buildkite/test_areas/distributed.yaml
- vllm/distributed/
- vllm/config/parallel.py
- vllm/config/vllm.py
- vllm/forward_context.py
- vllm/platforms/rocm.py
- tests/distributed/test_mnnvl_alltoall.py
- tests/utils.py
commands:
- pytest -v -s tests/distributed/test_mnnvl_alltoall.py::test_args_dispatch_combine
- label: ":nvidia: (B200) GEMM-RS/AR"
key: gemm-rs-ar-2xb200
@@ -452,6 +473,34 @@ steps:
- tests/kernels/test_gemm_rs_ar.py
commands:
- pytest -v -s tests/kernels/test_gemm_rs_ar.py
mirror:
amd:
label: ":amd: (MI355) GEMM-RS-AR"
dind: false
device: mi355_4
num_devices: 4
timeout_in_minutes: 20
working_dir: /vllm-workspace/
depends_on:
- image-build-amd
source_file_dependencies:
- .buildkite/test_areas/distributed.yaml
- tests/kernels/test_gemm_rs_ar.py
- tests/kernels/test_rocm_gemm_rs_ar.py
- tests/conftest.py
- vllm/models/common/ops/sequence_parallel.py
- vllm/model_executor/layers/linear.py
- vllm/model_executor/layers/utils.py
- vllm/model_executor/layers/quantization/online/
- vllm/model_executor/layers/quantization/utils/mxfp8_utils.py
- vllm/model_executor/kernels/linear/
- vllm/distributed/
- vllm/_aiter_ops.py
- vllm/_custom_ops.py
- csrc/rocm/skinny_gemms.cu
- vllm/platforms/rocm.py
commands:
- pytest -v -s tests/kernels/test_rocm_gemm_rs_ar.py
- label: ":nvidia: (L4) Distributed 2-Node"
key: 2-node-test-4-gpus
+48 -2
View File
@@ -59,6 +59,11 @@ steps:
- image-build-amd
source_file_dependencies:
- vllm/platforms/rocm.py
commands:
- pytest -v -s v1/engine/test_preprocess_error_handling.py
- pytest -v -s v1/engine --ignore v1/engine/test_preprocess_error_handling.py
- pytest -v -s v1/test_tensor_ipc_queue.py
- pytest -v -s config/test_mp_reducer.py
- label: ":nvidia: (H200 MIG 18GB) E2E Scheduling"
key: e2e-scheduling-1-gpu
@@ -80,6 +85,26 @@ steps:
source_file_dependencies:
- vllm/platforms/rocm.py
- label: ":nvidia: (H200 MIG 18GB) E2E Scheduling Accuracy"
key: e2e-scheduling-accuracy-1-gpu
timeout_in_minutes: 53
device: h200_18gb
source_file_dependencies:
- vllm/v1/
- tests/v1/e2e/general/
commands:
- pytest -v -s v1/e2e/general/test_async_scheduling_accuracy.py
mirror:
amd:
label: ":amd: (MI250) E2E Scheduling Accuracy"
dind: false
device: mi250_1
timeout_in_minutes: 55
depends_on:
- image-build-amd
source_file_dependencies:
- vllm/platforms/rocm.py
- label: ":nvidia: (H200 MIG 18GB) E2E Core"
device: h200_18gb
key: e2e-core-1-gpu
@@ -88,13 +113,13 @@ steps:
- vllm/v1/
- tests/v1/e2e/general/
commands:
- pytest -v -s v1/e2e/general --ignore v1/e2e/general/test_async_scheduling.py --ignore v1/e2e/general/test_kv_sharing_fast_prefill.py -k "not test_mamba_prefix_cache_mrv2"
- pytest -v -s v1/e2e/general --ignore v1/e2e/general/test_async_scheduling.py --ignore v1/e2e/general/test_async_scheduling_accuracy.py --ignore v1/e2e/general/test_kv_sharing_fast_prefill.py -k "not test_mamba_prefix_cache_mrv2"
mirror:
amd:
label: ":amd: (MI250) E2E Core"
dind: false
commands:
- pytest -v -s v1/e2e/general --ignore v1/e2e/general/test_async_scheduling.py --ignore v1/e2e/general/test_kv_sharing_fast_prefill.py -k "not test_mamba_prefix_cache_mrv2"
- pytest -v -s v1/e2e/general --ignore v1/e2e/general/test_async_scheduling.py --ignore v1/e2e/general/test_async_scheduling_accuracy.py --ignore v1/e2e/general/test_kv_sharing_fast_prefill.py -k "not test_mamba_prefix_cache_mrv2"
device: mi250_1
timeout_in_minutes: 50
depends_on:
@@ -133,6 +158,27 @@ steps:
- pytest -v -s v1/e2e/general/test_kv_sharing_fast_prefill.py
- pytest -v -s v1/e2e/general/test_mamba_prefix_cache.py::test_mamba_prefix_cache_mrv2 v1/e2e/general/test_mamba_prefix_cache.py::test_mamba_prefix_cache_mrv2_async
- label: ":nvidia: (H100) ReplaySSM E2E"
device: h100
num_devices: 2
key: replayssm-e2e
timeout_in_minutes: 90
working_dir: "/vllm-workspace/tests"
source_file_dependencies:
- tests/v1/e2e/test_replayssm_decode.py
- vllm/config/cache.py
- vllm/config/vllm.py
- vllm/model_executor/layers/mamba/
- vllm/model_executor/models/nemotron_h.py
- vllm/model_executor/warmup/replayssm_warmup.py
- vllm/v1/attention/backends/mamba_attn.py
- vllm/v1/spec_decode/
- vllm/v1/worker/gpu/
- vllm/v1/worker/gpu_model_runner.py
- vllm/v1/worker/gpu_worker.py
commands:
- pytest -v -s v1/e2e/test_replayssm_decode.py
- label: ":nvidia: (L4) V1 E2E"
key: v1-e2e-2-gpus
timeout_in_minutes: 25 # TODO: Fix timeout after we have more confidence in the test stability
+2 -2
View File
@@ -23,9 +23,9 @@ steps:
mirror:
amd:
# Focused MRV2 regression; the broader model/backend matrix is pending.
label: ':amd: (MI355) MRV2 Sampler JIT Warmup'
label: ':amd: (MI355 DPX) MRV2 Sampler JIT Warmup'
dind: false
device: mi355_1
device: mi355_dpx
num_devices: 1
timeout_in_minutes: 10
working_dir: /vllm-workspace/
+101 -28
View File
@@ -184,6 +184,20 @@ steps:
- tests/kernels/
commands:
- pytest -v -s kernels/mhc
mirror:
amd:
label: ":amd: (MI355) mHC Kernels"
dind: false
device: mi355_1
timeout_in_minutes: 50
working_dir: "/vllm-workspace/tests"
depends_on:
- image-build-amd
source_file_dependencies:
- csrc/
- vllm/
- tests/kernels/
- vllm/platforms/rocm.py
- label: ":nvidia: (B200) FLA Ops Kernels"
key: kernels-fla-ops-test-b200
@@ -196,6 +210,20 @@ steps:
- tests/kernels/
commands:
- pytest -v -s kernels/fla
mirror:
amd:
label: ":amd: (MI355) FLA Ops Kernels"
dind: false
device: mi355_1
timeout_in_minutes: 50
working_dir: "/vllm-workspace/tests"
depends_on:
- image-build-amd
source_file_dependencies:
- csrc/
- vllm/
- tests/kernels/
- vllm/platforms/rocm.py
# Catch-all for test files at the tests/kernels root. This job collects
# the whole root so new files are wired by default.
@@ -231,6 +259,50 @@ steps:
--ignore=kernels/test_shuffle_rows.py
# BROKEN on main, pending kernel fixes (B200):
# test_shuffle_rows.py (1: test_shuffle_rows_edge_cases)
mirror:
amd:
label: ":amd: (MI355) Miscellaneous Kernels"
dind: false
device: mi355_1
timeout_in_minutes: 55
working_dir: "/vllm-workspace/tests"
depends_on:
- image-build-amd
source_file_dependencies:
- csrc/
- vllm/
- tests/kernels/
- vllm/platforms/rocm.py
# Keep the ROCm exclusions from #50519: incompatible FlyDSL TurboQuant
# and a CUDA source check whose files are absent from the AMD test image.
commands:
- pytest -v -s
kernels/test_apply_repetition_penalties.py
kernels/test_awq_int4_to_int8.py
kernels/test_bf16_skinny_gemm.py
kernels/test_cache_kernels.py
kernels/test_compressor_kv_cache.py
kernels/test_cp_gather_fp8.py
kernels/test_deepseek_v4_cpu_kernels.py
kernels/test_dsv41_mega_attn_layouts.py
kernels/test_engram.py
kernels/test_flashinfer_bf16_gemm.py
kernels/test_flex_attention.py
kernels/test_fp32_router_gemm.py
kernels/test_fused_deepseek_v32_norm_rope.py
kernels/test_fused_indexer_q_rope_quant.py
kernels/test_fused_inv_rope_fp8_quant.py
kernels/test_fused_minimax_m3_qknorm_rope_kv_insert.py
kernels/test_fused_quant_activation.py
kernels/test_gate_linear_rocm_dispatch.py
kernels/test_gemm_rs_ar.py
kernels/test_kpool_decode_update_batched.py
kernels/test_minimax_m3_amd_ops.py
kernels/test_minimax_m3_sparse_attn_fp8_scale.py
kernels/test_onednn.py
kernels/test_relu2_fp8_quant.py
kernels/test_rocm_fp32_router_gemm.py
--deselect=tests/kernels/test_bf16_skinny_gemm.py::test_every_dsv3_routed_shape_is_instantiated
- label: ":nvidia: (L4) Attention Kernels Shard %N"
key: kernels-attention-test
@@ -725,6 +797,35 @@ steps:
- pytest -v -s kernels/moe/test_flashinfer_moe.py
- pytest -v -s kernels/moe/test_nvfp4_moe.py
- pytest -v -s kernels/moe/test_ocp_mx_moe.py
mirror:
amd:
label: ":amd: (MI355 DPX) FP4 and INT4 MoE Kernels"
dind: false
device: mi355_dpx
num_devices: 1
timeout_in_minutes: 40
optional: false
working_dir: /vllm-workspace/tests
depends_on:
- image-build-amd
source_file_dependencies:
- csrc/moe/
- csrc/rocm/
- vllm/model_executor/layers/fused_moe/
- vllm/model_executor/layers/quantization/
- vllm/_aiter_ops.py
- vllm/platforms/rocm.py
- vllm/envs.py
- vllm/config/
- tests/kernels/moe/test_rocm_aiter_moe.py
- tests/kernels/moe/test_ocp_mx_moe.py
- tests/kernels/moe/test_flydsl_moe.py
- tests/kernels/moe/utils.py
- tests/kernels/utils.py
commands:
- pytest -v -s kernels/moe/test_rocm_aiter_moe.py -k 'mi350_mxfp4 or quant_scheme_support_matches_gfx950'
- pytest -v -s kernels/moe/test_ocp_mx_moe.py::test_rocm_mxfp4_moe_oracle -k AITER_MXFP4
- pytest -v -s kernels/moe/test_flydsl_moe.py
- label: ":nvidia: (H100) FusedMoE Layer Kernels"
@@ -847,34 +948,6 @@ steps:
- pytest -v -s kernels/quantization/test_rocm_mxfp8_linear.py
- pytest -v -s kernels/quantization/test_rocm_skinny_gemms.py -k wvsplitkrc
- label: ":amd: (MI355 DPX) FP4 and INT4 MoE Kernels"
key: amd-fp4-moe-kernels-mi355
dind: false
device: mi355_dpx
num_devices: 1
timeout_in_minutes: 40
working_dir: /vllm-workspace/tests
depends_on:
- image-build-amd
source_file_dependencies:
- csrc/moe/
- csrc/rocm/
- vllm/model_executor/layers/fused_moe/
- vllm/model_executor/layers/quantization/
- vllm/_aiter_ops.py
- vllm/platforms/rocm.py
- vllm/envs.py
- vllm/config/
- tests/kernels/moe/test_rocm_aiter_moe.py
- tests/kernels/moe/test_ocp_mx_moe.py
- tests/kernels/moe/test_flydsl_moe.py
- tests/kernels/moe/utils.py
- tests/kernels/utils.py
commands:
- pytest -v -s kernels/moe/test_rocm_aiter_moe.py -k 'mi350_mxfp4 or quant_scheme_support_matches_gfx950'
- pytest -v -s kernels/moe/test_ocp_mx_moe.py::test_rocm_mxfp4_moe_oracle -k AITER_MXFP4
- pytest -v -s kernels/moe/test_flydsl_moe.py
- label: ":amd: (MI355 DPX) FP8 MoE Kernels"
key: amd-fp8-moe-kernels-mi355
dind: false
+233 -9
View File
@@ -151,15 +151,18 @@ steps:
label: ":amd: (MI355 DPX) LM Eval Small Models"
dind: false
device: mi355_dpx
num_devices: 1
timeout_in_minutes: 45
working_dir: /vllm-workspace/tests
depends_on:
- image-build-amd
source_file_dependencies:
- .buildkite/test_areas/lm_eval.yaml
- csrc/
- tests/evals/gsm8k/
- vllm/model_executor/layers/quantization
- vllm/model_executor/kernels/linear/
- vllm/model_executor/layers/fused_moe/
- vllm/model_executor/models/
- vllm/model_executor/model_loader/
- vllm/v1/attention/backends/
@@ -383,6 +386,49 @@ steps:
- vllm/model_executor/layers/quantization
commands:
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-blackwell-ep.txt
mirror:
amd:
label: ":amd: (MI355) LM Eval Large Models EP"
dind: false
device: mi355_2
num_devices: 2
timeout_in_minutes: 120
working_dir: "/vllm-workspace/tests"
depends_on:
- image-build-amd
source_file_dependencies:
- .buildkite/test_areas/lm_eval.yaml
- tests/evals/gsm8k/
- tests/utils.py
- tests/conftest.py
- csrc/
- vllm/config/
- vllm/distributed/
- vllm/model_executor/models/config.py
- vllm/model_executor/models/registry.py
- vllm/model_executor/models/utils.py
- vllm/model_executor/models/qwen3_next.py
- vllm/model_executor/models/qwen3_next_mtp.py
- vllm/model_executor/models/nemotron_h.py
- vllm/model_executor/models/nemotron_h_mtp.py
- vllm/transformers_utils/configs/qwen3_next.py
- vllm/transformers_utils/configs/nemotron_h.py
- vllm/model_executor/model_loader/
- vllm/model_executor/layers/quantization/
- vllm/model_executor/layers/fused_moe/
- vllm/model_executor/layers/mamba/
- vllm/model_executor/layers/layernorm.py
- vllm/model_executor/layers/linear.py
- vllm/model_executor/kernels/linear/
- vllm/third_party/flash_linear_attention/ops/
- vllm/v1/attention/
- vllm/v1/spec_decode/
- vllm/v1/worker/
- vllm/_aiter_ops.py
- vllm/platforms/rocm.py
commands:
- export VLLM_WORKER_MULTIPROC_METHOD=spawn
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-blackwell-ep-mi355.txt
- label: ":nvidia: (B200) LM Eval Qwen3.5 Models"
key: lm-eval-qwen3-5-models-2xb200
@@ -405,7 +451,7 @@ steps:
label: ":amd: (MI355) LM Eval Qwen3.5 Models"
dind: false
device: mi355_2
timeout_in_minutes: 40
timeout_in_minutes: 75
depends_on:
- image-build-amd
source_file_dependencies:
@@ -418,10 +464,18 @@ steps:
- vllm/model_executor/models/qwen3_next.py
- vllm/model_executor/models/qwen3_next_mtp.py
- vllm/third_party/flash_linear_attention/ops/
- vllm/distributed/
- vllm/model_executor/layers/fused_moe/
- vllm/model_executor/layers/quantization/modelopt.py
- vllm/model_executor/kernels/linear/nvfp4/
- vllm/v1/attention/
- vllm/v1/worker/
- tests/evals/gsm8k/configs/models-qwen35-mi355.txt
- tests/evals/gsm8k/configs/Qwen3.5-35B-A3B-DEP2.yaml
- tests/evals/gsm8k/configs/Qwen3.5-35B-A3B-FP8-DEP2.yaml
- tests/evals/gsm8k/configs/Qwen3.5-35B-A3B-MXFP4-AITER-TP2.yaml
- tests/evals/gsm8k/configs/Qwen3.5-35B-A3B-MXFP4-EMU-TP2.yaml
- tests/evals/gsm8k/configs/Qwen3.5-397B-A17B-NVFP4-DEP2.yaml
- tests/evals/gsm8k/test_gsm8k_correctness.py
- vllm/_aiter_ops.py
- vllm/platforms/rocm.py
@@ -436,6 +490,40 @@ steps:
num_devices: 8
commands:
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-h200-deepseek-r1-tp.txt
mirror:
amd:
label: ":amd: (MI355) LM Eval DeepSeek R1 TP"
dind: false
device: mi355_4
num_devices: 4
timeout_in_minutes: 65
working_dir: /vllm-workspace/tests
depends_on:
- image-build-amd
source_file_dependencies:
- .buildkite/test_areas/lm_eval.yaml
- tests/evals/gsm8k/
- tests/utils.py
- tests/conftest.py
- vllm/config/
- vllm/distributed/
- vllm/model_executor/models/
- vllm/models/
- vllm/model_executor/model_loader/
- vllm/model_executor/layers/quantization/
- vllm/model_executor/layers/fused_moe/
- vllm/v1/attention/
- vllm/v1/spec_decode/
- vllm/v1/worker/gpu/spec_decode/
- vllm/v1/core/kv_cache_utils.py
- vllm/model_executor/layers/layernorm.py
- csrc/
- vllm/_aiter_ops.py
- vllm/platforms/rocm.py
- vllm/model_executor/layers/sparse_attn_indexer.py
commands:
- export VLLM_WORKER_MULTIPROC_METHOD=spawn
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-mi355-deepseek-r1-tp.txt
- label: ":nvidia: (H200) LM Eval DeepSeek R1 DP"
key: lm-eval-deepseek-r1-dp-8xh200
@@ -445,6 +533,40 @@ steps:
num_devices: 8
commands:
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-h200-deepseek-r1-dp.txt
mirror:
amd:
label: ":amd: (MI355) LM Eval DeepSeek R1 DP"
dind: false
device: mi355_4
num_devices: 4
timeout_in_minutes: 65
working_dir: /vllm-workspace/tests
depends_on:
- image-build-amd
source_file_dependencies:
- .buildkite/test_areas/lm_eval.yaml
- tests/evals/gsm8k/
- tests/utils.py
- tests/conftest.py
- vllm/config/
- vllm/distributed/
- vllm/model_executor/models/
- vllm/models/
- vllm/model_executor/model_loader/
- vllm/model_executor/layers/quantization/
- vllm/model_executor/layers/fused_moe/
- vllm/v1/attention/
- vllm/v1/spec_decode/
- vllm/v1/worker/gpu/spec_decode/
- vllm/v1/core/kv_cache_utils.py
- vllm/model_executor/layers/layernorm.py
- csrc/
- vllm/_aiter_ops.py
- vllm/platforms/rocm.py
- vllm/model_executor/layers/sparse_attn_indexer.py
commands:
- export VLLM_WORKER_MULTIPROC_METHOD=spawn
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-mi355-deepseek-r1-dp.txt
- label: ":nvidia: (H200) LM Eval DeepSeek V3.2 TP"
key: lm-eval-deepseek-v32-tp-8xh200
@@ -454,6 +576,40 @@ steps:
num_devices: 8
commands:
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-h200-deepseek-v32-tp.txt
mirror:
amd:
label: ":amd: (MI355) LM Eval DeepSeek V3-2 TP"
dind: false
device: mi355_4
num_devices: 4
timeout_in_minutes: 65
working_dir: /vllm-workspace/tests
depends_on:
- image-build-amd
source_file_dependencies:
- .buildkite/test_areas/lm_eval.yaml
- tests/evals/gsm8k/
- tests/utils.py
- tests/conftest.py
- vllm/config/
- vllm/distributed/
- vllm/model_executor/models/
- vllm/models/
- vllm/model_executor/model_loader/
- vllm/model_executor/layers/quantization/
- vllm/model_executor/layers/fused_moe/
- vllm/v1/attention/
- vllm/v1/spec_decode/
- vllm/v1/worker/gpu/spec_decode/
- vllm/v1/core/kv_cache_utils.py
- vllm/model_executor/layers/layernorm.py
- csrc/
- vllm/_aiter_ops.py
- vllm/platforms/rocm.py
- vllm/model_executor/layers/sparse_attn_indexer.py
commands:
- export VLLM_WORKER_MULTIPROC_METHOD=spawn
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-mi355-deepseek-v32-tp.txt
- label: ":nvidia: (H200) LM Eval DeepSeek V3.2 DP"
key: lm-eval-deepseek-v32-dp-8xh200
@@ -463,6 +619,40 @@ steps:
num_devices: 8
commands:
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-h200-deepseek-v32-dp.txt
mirror:
amd:
label: ":amd: (MI355) LM Eval DeepSeek V3-2 DP"
dind: false
device: mi355_4
num_devices: 4
timeout_in_minutes: 65
working_dir: /vllm-workspace/tests
depends_on:
- image-build-amd
source_file_dependencies:
- .buildkite/test_areas/lm_eval.yaml
- tests/evals/gsm8k/
- tests/utils.py
- tests/conftest.py
- vllm/config/
- vllm/distributed/
- vllm/model_executor/models/
- vllm/models/
- vllm/model_executor/model_loader/
- vllm/model_executor/layers/quantization/
- vllm/model_executor/layers/fused_moe/
- vllm/v1/attention/
- vllm/v1/spec_decode/
- vllm/v1/worker/gpu/spec_decode/
- vllm/v1/core/kv_cache_utils.py
- vllm/model_executor/layers/layernorm.py
- csrc/
- vllm/_aiter_ops.py
- vllm/platforms/rocm.py
- vllm/model_executor/layers/sparse_attn_indexer.py
commands:
- export VLLM_WORKER_MULTIPROC_METHOD=spawn
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-mi355-deepseek-v32-dp.txt
- label: ":nvidia: (H200) LM Eval Nemotron 3 Super"
key: lm-eval-nemotron-3-super-8xh200
@@ -472,6 +662,40 @@ steps:
num_devices: 8
commands:
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-h200-nemotron-3-super.txt
mirror:
amd:
label: ":amd: (MI355) LM Eval Nemotron 3 Super"
dind: false
device: mi355_4
num_devices: 4
timeout_in_minutes: 65
working_dir: /vllm-workspace/tests
depends_on:
- image-build-amd
source_file_dependencies:
- .buildkite/test_areas/lm_eval.yaml
- tests/evals/gsm8k/
- tests/utils.py
- tests/conftest.py
- vllm/config/
- vllm/distributed/
- vllm/model_executor/models/
- vllm/models/
- vllm/model_executor/model_loader/
- vllm/model_executor/layers/quantization/
- vllm/model_executor/layers/fused_moe/
- vllm/v1/attention/
- vllm/v1/spec_decode/
- vllm/v1/worker/gpu/spec_decode/
- vllm/v1/core/kv_cache_utils.py
- vllm/model_executor/layers/layernorm.py
- csrc/
- vllm/_aiter_ops.py
- vllm/platforms/rocm.py
- vllm/model_executor/layers/mamba/
commands:
- export VLLM_WORKER_MULTIPROC_METHOD=spawn
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-mi355-nemotron-3-super.txt
- label: ":amd: (MI300) LM Eval Large Models"
key: amd-lm-eval-large-models
@@ -685,9 +909,9 @@ steps:
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=evals/gsm8k/configs/models-turboquant-k8v4.txt
mirror:
amd:
label: ":amd: (MI355) LM Eval TurboQuant k8v4"
label: ":amd: (MI355 DPX) LM Eval TurboQuant k8v4"
dind: false
device: mi355_1
device: mi355_dpx
num_devices: 1
timeout_in_minutes: 30
working_dir: /vllm-workspace/tests
@@ -725,9 +949,9 @@ steps:
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=evals/gsm8k/configs/models-turboquant-t4nc.txt
mirror:
amd:
label: ":amd: (MI355) LM Eval TurboQuant t4nc"
label: ":amd: (MI355 DPX) LM Eval TurboQuant t4nc"
dind: false
device: mi355_1
device: mi355_dpx
num_devices: 1
timeout_in_minutes: 30
working_dir: /vllm-workspace/tests
@@ -765,9 +989,9 @@ steps:
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=evals/gsm8k/configs/models-turboquant-k3v4nc.txt
mirror:
amd:
label: ":amd: (MI355) LM Eval TurboQuant k3v4nc"
label: ":amd: (MI355 DPX) LM Eval TurboQuant k3v4nc"
dind: false
device: mi355_1
device: mi355_dpx
num_devices: 1
timeout_in_minutes: 30
working_dir: /vllm-workspace/tests
@@ -805,9 +1029,9 @@ steps:
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=evals/gsm8k/configs/models-turboquant-t3nc.txt
mirror:
amd:
label: ":amd: (MI355) LM Eval TurboQuant t3nc"
label: ":amd: (MI355 DPX) LM Eval TurboQuant t3nc"
dind: false
device: mi355_1
device: mi355_dpx
num_devices: 1
timeout_in_minutes: 30
working_dir: /vllm-workspace/tests
+2 -2
View File
@@ -530,7 +530,7 @@ steps:
- vllm/v1/
- vllm/version.py
- cmake/utils.cmake
- tests/test_access_log_filter.py
- tests/test_uvicorn_logging.py
- tests/test_attention_backend_registry.py
- tests/test_audio_media_size_precheck.py
- tests/test_cmake_utils.py
@@ -571,7 +571,7 @@ steps:
- pip install cmake # test_cmake_utils.py shells out to cmake; the CPU image has none
- python3 standalone_tests/lazy_imports.py
- pytest -v -s device_allocator
- pytest -v -s test_access_log_filter.py
- pytest -v -s test_uvicorn_logging.py
- pytest -v -s test_attention_backend_registry.py
- pytest -v -s test_audio_media_size_precheck.py
- pytest -v -s test_cmake_utils.py
+6 -14
View File
@@ -66,13 +66,15 @@ steps:
- tests/models/transformers/test_backend.py
- tests/models/test_registry.py
- tests/models/test_deepseek_v4_vl_rocm.py
- tests/models/test_deepseek_v41_decoder_replay_layers.py
- tests/models/test_deepseek_v41_replay_batch.py
commands:
- pytest -v -s models/test_terratorch.py models/transformers/test_backend.py models/test_registry.py models/test_deepseek_v4_vl_rocm.py
- pytest -v -s models/test_terratorch.py models/transformers/test_backend.py models/test_registry.py models/test_deepseek_v4_vl_rocm.py models/test_deepseek_v41_decoder_replay_layers.py models/test_deepseek_v41_replay_batch.py
mirror:
amd:
label: ":amd: (MI355 DPX) Basic Models (Other)"
label: ":amd: (MI355) Basic Models (Other)"
dind: false
device: mi355_dpx
device: mi355_1
timeout_in_minutes: 65
depends_on:
- image-build-amd
@@ -176,7 +178,7 @@ steps:
- vllm/v1/attention/backends/short_conv_attn.py
- tests/models/qwen4_exp/
commands:
- pytest -v -s models/qwen4_exp/test_hc_ops.py models/qwen4_exp/test_qsa_pre_indexer.py models/qwen4_exp/test_qsa_reference.py
- pytest -v -s models/qwen4_exp/test_hc_ops.py models/qwen4_exp/test_qsa_prepare.py models/qwen4_exp/test_qsa_reference.py
mirror:
amd:
label: ":amd: (MI355 DPX) Qwen4 Exp"
@@ -221,13 +223,3 @@ steps:
device: cpu-small
commands:
- pytest -v -s models/test_utils.py models/test_vision.py models/test_adapters.py models/test_qwen3_5_mtp_config.py models/transformers/fusers/ model_executor/layers/test_activation.py
mirror:
amd:
label: ":amd: (MI250) Basic Models Other"
dind: false
device: mi250_1
no_gpu: true
working_dir: "/vllm-workspace/tests"
timeout_in_minutes: 35
depends_on:
- image-build-amd
@@ -26,8 +26,7 @@ steps:
pytest models/language -v -s -m 'distributed(num_gpus=2)'
;;
2)
pytest models/multimodal/generation/test_phi4siglip.py -v -s -m 'distributed(num_gpus=2)'
pytest models/multimodal -v -s -m 'distributed(num_gpus=2)' --ignore models/multimodal/generation/test_whisper.py --ignore models/multimodal/generation/test_phi4siglip.py
pytest models/multimodal -v -s -m 'distributed(num_gpus=2)' --ignore models/multimodal/generation/test_whisper.py
VLLM_WORKER_MULTIPROC_METHOD=spawn pytest models/multimodal/generation/test_whisper.py -v -s -m 'distributed(num_gpus=2)'
;;
*) exit 2 ;;
@@ -72,8 +71,7 @@ steps:
VLLM_ROCM_USE_AITER=1 pytest -v -s models/test_hyv4_rocm.py -m 'distributed(num_gpus=2)'
;;
2)
pytest models/multimodal -v -s -m 'distributed(num_gpus=2)' --ignore models/multimodal/generation/test_whisper.py --ignore models/multimodal/generation/test_phi4siglip.py &&
pytest models/multimodal/generation/test_phi4siglip.py -v -s -m 'distributed(num_gpus=2)' &&
pytest models/multimodal -v -s -m 'distributed(num_gpus=2)' --ignore models/multimodal/generation/test_whisper.py &&
VLLM_WORKER_MULTIPROC_METHOD=spawn pytest models/multimodal/generation/test_whisper.py -v -s -m 'distributed(num_gpus=2)'
;;
*) exit 2 ;;
+1 -12
View File
@@ -99,17 +99,6 @@ steps:
commands:
- pytest -v -s models/multimodal/processing --ignore models/multimodal/processing/test_tensor_schema.py --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT --shard-id=$$BUILDKITE_PARALLEL_JOB
parallelism: 4
mirror:
amd:
label: ":amd: (MI250) Multimodal Processor Shard %N"
dind: false
device: mi250_1
no_gpu: true
working_dir: "/vllm-workspace/tests"
timeout_in_minutes: 90
parallelism: 6
depends_on:
- image-build-amd
- label: ":nvidia: (H200 MIG 18GB) Multimodal Processor Shard %N"
key: multi-modal-processor
@@ -176,7 +165,7 @@ steps:
label: ":amd: (MI300) Multimodal Models (Extended Generation 1)"
dind: false
device: mi300_1
timeout_in_minutes: 90
timeout_in_minutes: 100
depends_on:
- image-build-amd
+2
View File
@@ -102,6 +102,7 @@ steps:
depends_on:
- image-build-amd
source_file_dependencies:
- tests/quantization/test_gfx950_moe.py
- tests/quantization/test_rocm_moe.py
- tests/utils.py
- vllm/model_executor/models/deepseek_v2.py
@@ -118,6 +119,7 @@ steps:
- vllm/envs.py
commands:
- pytest -v -s tests/quantization/test_rocm_moe.py
- pytest -v -s tests/quantization/test_gfx950_moe.py
- label: ":nvidia: (H200 MIG 35GB) Quantized Models"
device: h200_35gb
+26
View File
@@ -423,6 +423,32 @@ steps:
- tests/v1/e2e/spec_decode/
commands:
- pytest -v -s v1/e2e/spec_decode/acceptance_rates/dflash/ -k "dflash2"
mirror:
amd:
label: ":amd: (MI355) Spec Decode AL DFlash2 Nightly"
dind: false
device: mi355_1
num_devices: 1
timeout_in_minutes: 60
depends_on:
- image-build-amd
source_file_dependencies:
- .buildkite/test_areas/spec_decode.yaml
- tests/v1/e2e/spec_decode/
- tests/evals/gsm8k/
- tests/utils.py
- vllm/v1/spec_decode/
- vllm/v1/worker/gpu/spec_decode/
- vllm/v1/worker/gpu/sample/
- vllm/v1/worker/gpu/model_runner.py
- vllm/v1/attention/
- vllm/model_executor/models/qwen3_dflash.py
- vllm/model_executor/models/qwen3_dflash2.py
- vllm/model_executor/models/qwen3_5.py
- vllm/model_executor/model_loader/
- vllm/model_executor/layers/quantization/
- vllm/model_executor/kernels/linear/
- vllm/platforms/rocm.py
- label: ":nvidia: (H200 MIG 35GB) Spec Decode AL DSpark Nightly"
key: spec-decode-dspark-nightly
+1
View File
@@ -0,0 +1 @@
../../.agents/skills/check-api-compat
+1
View File
@@ -0,0 +1 @@
../../.agents/skills/pr-checklist
+12 -4
View File
@@ -3,10 +3,10 @@
# This lists cover the "core" components of vLLM that require careful review
/vllm/compilation @zou3519 @youkaichao @ProExpertProg @BoyuanFeng
/vllm/distributed/kv_transfer @NickLucche @ApostaC @orozery @xuechendi @ivanium
/vllm/distributed/kv_transfer @NickLucche @ApostaC @orozery @xuechendi @ivanium @wzhao18
/vllm/lora @jeejeelee
/vllm/model_executor/layers/attention @LucasWilkinson @MatthewBonanni
/vllm/model_executor/layers/fused_moe @mgoin @pavanimajety @zyongye
/vllm/model_executor/layers/fused_moe @mgoin @pavanimajety @zyongye @wzhao18
/vllm/model_executor/layers/quantization @mgoin @robertgshaw2-redhat @tlrmchlsmth @yewentao256 @pavanimajety @zyongye
/vllm/model_executor/layers/mamba @tdoublep @tomeras91
/vllm/model_executor/layers/mamba/gdn/qwen_gdn_linear_attn.py @tdoublep @ZJY0516 @vadiklyutiy
@@ -67,7 +67,7 @@
/vllm/v1/attention @LucasWilkinson @MatthewBonanni
/vllm/v1/attention/backend.py @WoosukKwon @zhuohan123 @youkaichao @alexm-redhat @njhill
/vllm/v1/attention/backends/mla @pavanimajety
/vllm/v1/attention/backends/flashinfer.py @mgoin @pavanimajety @vadiklyutiy
/vllm/v1/attention/backends/flashinfer.py @mgoin @pavanimajety @vadiklyutiy @wzhao18
/vllm/v1/attention/backends/triton_attn.py @tdoublep
/vllm/v1/attention/backends/gdn_attn.py @ZJY0516 @vadiklyutiy
/vllm/v1/core @WoosukKwon @robertgshaw2-redhat @njhill @ywang96 @alexm-redhat @heheda12345 @ApostaC @orozery @ivanium
@@ -113,7 +113,7 @@
/tests/lora @jeejeelee
/tests/models/language/generation/test_hybrid.py @tdoublep @tomeras91
/tests/v1/kv_connector/nixl_integration @NickLucche
/tests/v1/kv_connector @ApostaC @orozery @ivanium
/tests/v1/kv_connector @ApostaC @orozery @ivanium @wzhao18
/tests/v1/kv_offload @ApostaC @orozery
/tests/v1/simple_kv_offload @ivanium
/tests/v1/determinism @yewentao256
@@ -158,6 +158,7 @@ mkdocs.yaml @hmellor
# Qwen-specific files
/vllm/model_executor/models/qwen* @sighingnow @vadiklyutiy
/vllm/transformers_utils/configs/qwen* @sighingnow @vadiklyutiy
/vllm/models/qwen4_exp @gau-nernst
# MTP-specific files
/vllm/model_executor/models/deepseek_mtp.py @luccafong
@@ -166,6 +167,9 @@ mkdocs.yaml @hmellor
/vllm/models/deepseek_v4 @zyongye
/vllm/model_executor/layers/sparse_attn_indexer.py @zyongye
# Kimi-K3-specific files
/vllm/models/kimi_k3/nvidia/kda.py @gau-nernst
# Mistral-specific files
/vllm/model_executor/models/mistral*.py @patrickvonplaten
/vllm/model_executor/models/mixtral*.py @patrickvonplaten
@@ -178,6 +182,10 @@ mkdocs.yaml @hmellor
/vllm/v1/attention/ops/chunked_prefill_paged_decode.py @tdoublep
/vllm/v1/attention/ops/triton_unified_attention.py @tdoublep
/vllm/third_party/flash_linear_attention @ZJY0516 @vadiklyutiy
/vllm/model_executor/layers/mamba/ops/gdn_chunk_cutedsl @gau-nernst
/vllm/model_executor/layers/fused_moe/router/bf16x3_router_gemm_cutedsl.py @gau-nernst
/vllm/model_executor/kernels/linear/cute_dsl/gemm_rs_ar.py @gau-nernst
/vllm/cute_utils @gau-nernst
# ROCm related: specify owner with write access to notify AMD folks for careful code review
/vllm/**/*rocm* @tjtanaa @dllehr-amd @shen-shanshan
+23 -10
View File
@@ -1,20 +1,33 @@
<!-- markdownlint-disable -->
PLEASE FILL IN THE PR DESCRIPTION HERE ENSURING ALL CHECKLIST ITEMS (AT THE BOTTOM) HAVE BEEN CONSIDERED.
## Purpose
## Overview
## Test Plan
<!-- Summarize the change in 1-2 lines. Link relevant issues and PRs. -->
## Test Result
## Claims
<!-- State the accomplishments in a few concrete bullets: speedups, feature support, bug fixes, etc. -->
## Validation
<!-- Show evidence that the claims are met by this change, and steps to reproduce validations. -->
## Details
<!-- Explain motivation, design decisions, root cause, tradeoffs, and limitations concisely. Link or collapse lengthy supporting material. -->
---
<details>
<summary> Essential Elements of an Effective PR Description Checklist </summary>
- [ ] The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
- [ ] The test plan, such as providing test command.
- [ ] The test results, such as pasting the results comparison before and after, or e2e results
- [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
<details>
<summary> Pull Request Checklist </summary>
- [ ] I used vLLM's `/pr-checklist` skill. (Mandatory for agents, optional for humans).
- [ ] AI assistance was used during the creation of this PR.
- [ ] **Design Fit:** Minimizes impact on core components, reuses existing functionality, and justifies added complexity.
- [ ] **Testing and Validation:** Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.
- [ ] **Code Quality and Style:** Keeps code and comments clear and concise, and updates relevant documentation and examples.
- [ ] **Pull Request Contents:** Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.
</details>
**BEFORE SUBMITTING, PLEASE READ <https://docs.vllm.ai/en/latest/contributing>** (anything written below this line will be removed by GitHub Actions)
+41 -2
View File
@@ -5,7 +5,6 @@ pull_request_rules:
- label != stale
- -closed
- or:
- files~=^[^/]+\.md$
- files~=^docs/
- files~=^examples/
actions:
@@ -572,7 +571,7 @@ pull_request_rules:
- base = main
- label=ready
- "#approved-reviews-by >= 1"
- "#commits-behind >= 50"
- "#commits-behind >= 100"
- "#check-failure = 0"
- -closed
- -draft
@@ -661,6 +660,30 @@ pull_request_rules:
add:
- kv-connector
- name: label-mooncake
description: Automatically apply mooncake label
conditions:
- label != stale
- or:
- files~=^vllm/distributed/kv_transfer/kv_connector/v1/mooncake/
# Covers the mooncake/ connector dir, mooncake_store_embedding/ and
# mooncake_ec_connector.py.
- files~=^vllm/distributed/ec_transfer/ec_connector/mooncake
- files=vllm/distributed/mooncake_store.py
- files~=^tests/v1/kv_connector/.*mooncake
- files~=^tests/v1/ec_connector/.*mooncake
- files~=^docs/features/mooncake.*\.md
- files=.buildkite/test_areas/disaggregated_mooncake.yaml
- title~=(?i)mooncake
actions:
label:
add:
- mooncake
assign:
users:
- "stmatengss"
- "LCAIZJ"
- name: label-scheduler
description: Automatically apply scheduler label
conditions:
@@ -725,3 +748,19 @@ pull_request_rules:
label:
add:
- vllm-ir
- name: label-agents
description: Automatically apply agents label
conditions:
- label != stale
- or:
- files~=(?:^|/)AGENTS\.md$
- files~=^\.agents/
- files~=^\.claude/
# The [Agents] title tag only: a bare "agents" also matches CI, K8s and
# NIXL agents.
- title~=(?i)\[agents\]
actions:
label:
add:
- agents
+1
View File
@@ -31,6 +31,7 @@ Do not open one-off PRs for tiny edits (single typo, isolated style change, one
- Test commands run and results.
- Model evaluation results when the change affects output, accuracy, or serving.
- Clear statement that AI assistance was used.
- Use vLLM's [`/pr-checklist`](.agents/skills/pr-checklist/SKILL.md) skill before submitting a PR to ensure quality and adherence to the contribution guidelines.
### Fail-closed behavior
+21
View File
@@ -590,9 +590,30 @@ if(VLLM_GPU_LANG STREQUAL "CUDA" OR VLLM_GPU_LANG STREQUAL "HIP")
endif()
endif()
# The fused MiniMax-M3 KV insert quantizes NVFP4 with SM100-family-only
# cvt instructions, so its SM10x/11x code is built for the family targets.
set(MINIMAX_M3_FUSED_SRC
"csrc/libtorch_stable/fused_minimax_m3_qknorm_rope_kv_insert_kernel.cu")
list(REMOVE_ITEM VLLM_STABLE_EXT_SRC "${MINIMAX_M3_FUSED_SRC}")
set_gencode_flags_for_srcs(
SRCS "${VLLM_STABLE_EXT_SRC}"
CUDA_ARCHS "${CUDA_ARCHS}")
if(${CMAKE_CUDA_COMPILER_VERSION} VERSION_GREATER_EQUAL 13.0)
cuda_archs_loose_intersection(MINIMAX_M3_FUSED_ARCHS
"10.0f;10.7f;11.0f" "${CUDA_ARCHS}")
else()
cuda_archs_loose_intersection(MINIMAX_M3_FUSED_ARCHS
"10.0a;10.1a;10.3a" "${CUDA_ARCHS}")
endif()
foreach(_ARCH ${CUDA_ARCHS})
if(_ARCH VERSION_LESS 10.0 OR _ARCH VERSION_GREATER_EQUAL 12.0)
list(APPEND MINIMAX_M3_FUSED_ARCHS "${_ARCH}")
endif()
endforeach()
set_gencode_flags_for_srcs(
SRCS "${MINIMAX_M3_FUSED_SRC}"
CUDA_ARCHS "${MINIMAX_M3_FUSED_ARCHS}")
list(APPEND VLLM_STABLE_EXT_SRC "${MINIMAX_M3_FUSED_SRC}")
if(COOPERATIVE_TOPK_ARCHS)
list(APPEND VLLM_STABLE_EXT_SRC
+217
View File
@@ -0,0 +1,217 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""Micro-benchmark for CPU sampling kernels.
Compares fused Gumbel-max / greedy argmax against the baseline
(softmax → exponential → div → argmax) across vocab and batch sizes.
Usage:
.venv/bin/python benchmarks/kernels/bench_cpu_sampling.py
.venv/bin/python benchmarks/kernels/bench_cpu_sampling.py --profile
.venv/bin/python benchmarks/kernels/bench_cpu_sampling.py --vocab 128256 --batch 16
"""
import argparse
import time
import torch
import vllm._C # noqa: F401
from torch.profiler import ProfilerActivity, profile, record_function
def baseline_random_sample(logits: torch.Tensor) -> torch.Tensor:
probs = logits.softmax(dim=-1, dtype=torch.float32)
q = torch.empty_like(probs)
q.exponential_()
return probs.div(q).argmax(dim=-1).view(-1)
def baseline_greedy_sample(logits: torch.Tensor) -> torch.Tensor:
return logits.argmax(dim=-1).view(-1)
def bench_latency(fn, args, n_warmup=20, n_iters=500):
for _ in range(n_warmup):
fn(*args)
t0 = time.perf_counter()
for _ in range(n_iters):
fn(*args)
elapsed = time.perf_counter() - t0
return elapsed / n_iters * 1e6 # µs
def run_profile(logits, seeds, n_iters=50):
"""Run torch.profiler and print comparison tables."""
# Profile baseline random sampling
with profile(
activities=[ProfilerActivity.CPU],
record_shapes=True,
) as prof_base_rand:
for _ in range(n_iters):
with record_function("baseline_random"):
baseline_random_sample(logits)
# Profile fused Gumbel-max
with profile(
activities=[ProfilerActivity.CPU],
record_shapes=True,
) as prof_fused:
for _ in range(n_iters):
with record_function("fused_gumbel_argmax"):
torch.ops._C.fused_gumbel_argmax(logits, seeds)
# Profile baseline greedy
with profile(
activities=[ProfilerActivity.CPU],
record_shapes=True,
) as prof_base_grdy:
for _ in range(n_iters):
with record_function("baseline_greedy"):
baseline_greedy_sample(logits)
# Profile custom greedy
with profile(
activities=[ProfilerActivity.CPU],
record_shapes=True,
) as prof_cust_grdy:
for _ in range(n_iters):
with record_function("custom_greedy_argmax"):
torch.ops._C.greedy_argmax(logits)
B, V = logits.shape
print(f"\n{'=' * 80}")
print(f"torch.profiler breakdown (batch={B}, vocab={V}, iters={n_iters})")
print(f"{'=' * 80}")
print("\n--- Baseline Random (softmax → exp → div → argmax) ---")
print(prof_base_rand.key_averages().table(sort_by="cpu_time_total", row_limit=15))
print("\n--- Fused Gumbel-max (table lookup + add + argmax) ---")
print(prof_fused.key_averages().table(sort_by="cpu_time_total", row_limit=15))
print("\n--- Baseline Greedy (torch.argmax) ---")
print(prof_base_grdy.key_averages().table(sort_by="cpu_time_total", row_limit=15))
print("\n--- Custom Greedy (vec_op argmax) ---")
print(prof_cust_grdy.key_averages().table(sort_by="cpu_time_total", row_limit=15))
# Summary comparison
def avg_us(prof, label):
for e in prof.key_averages():
if e.key == label:
return e.cpu_time_total / e.count
return 0.0
t_br = avg_us(prof_base_rand, "baseline_random")
t_fg = avg_us(prof_fused, "fused_gumbel_argmax")
t_bg = avg_us(prof_base_grdy, "baseline_greedy")
t_cg = avg_us(prof_cust_grdy, "custom_greedy_argmax")
print(f"\n{'=' * 60}")
print(f" Summary (batch={B}, vocab={V})")
print(f"{'=' * 60}")
print(f" {'Kernel':<30} {'Avg (µs)':>10} {'Speedup':>10}")
print(f" {'-' * 50}")
print(f" {'baseline random':<30} {t_br:>10.1f} {'—':>10}")
print(
f" {'fused gumbel-max':<30} {t_fg:>10.1f} "
f"{t_br / t_fg if t_fg > 0 else 0:>9.2f}x"
)
print(f" {'baseline greedy':<30} {t_bg:>10.1f} {'—':>10}")
print(
f" {'custom greedy':<30} {t_cg:>10.1f} {t_bg / t_cg if t_cg > 0 else 0:>9.2f}x"
)
print(f"{'=' * 60}")
def main():
parser = argparse.ArgumentParser(description="Benchmark CPU sampling kernels")
parser.add_argument(
"--vocab",
type=int,
nargs="+",
default=[32000, 49152, 128256],
help="Vocab sizes to benchmark",
)
parser.add_argument(
"--batch",
type=int,
nargs="+",
default=[1, 4, 16],
help="Batch sizes to benchmark",
)
parser.add_argument(
"--profile",
action="store_true",
help="Run torch.profiler and export chrome trace",
)
parser.add_argument(
"--iters", type=int, default=500, help="Iterations per measurement"
)
args = parser.parse_args()
header = (
f"{'batch':>5} {'vocab':>7} "
f"{'base_rand':>10} {'fused_rand':>10} {'rand_spdup':>10} "
f"{'base_grdy':>10} {'cust_grdy':>10} {'grdy_spdup':>10}"
)
units = (
f"{'':>5} {'':>7} "
f"{'(µs)':>10} {'(µs)':>10} {'':>10} "
f"{'(µs)':>10} {'(µs)':>10} {'':>10}"
)
print("\n" + "=" * len(header))
print("CPU Sampling Kernel Benchmark")
print("=" * len(header))
print(header)
print(units)
print("-" * len(header))
for V in args.vocab:
for B in args.batch:
logits = torch.randn(B, V, dtype=torch.float32)
seeds = torch.arange(B, dtype=torch.long)
t_base_rand = bench_latency(
baseline_random_sample, (logits,), n_iters=args.iters
)
t_fused = bench_latency(
torch.ops._C.fused_gumbel_argmax, (logits, seeds), n_iters=args.iters
)
t_base_grdy = bench_latency(
baseline_greedy_sample, (logits,), n_iters=args.iters
)
t_cust_grdy = bench_latency(
torch.ops._C.greedy_argmax, (logits,), n_iters=args.iters
)
rand_speedup = t_base_rand / t_fused if t_fused > 0 else 0
grdy_speedup = t_base_grdy / t_cust_grdy if t_cust_grdy > 0 else 0
print(
f"{B:>5} {V:>7} "
f"{t_base_rand:>10.1f} {t_fused:>10.1f} "
f"{rand_speedup:>9.2f}x "
f"{t_base_grdy:>10.1f} {t_cust_grdy:>10.1f} "
f"{grdy_speedup:>9.2f}x"
)
print("-" * len(header))
if args.profile:
print("\nRunning torch.profiler (batch=16, vocab=128256) ...")
logits = torch.randn(16, 128256, dtype=torch.float32)
seeds = torch.arange(16, dtype=torch.long)
# warmup
for _ in range(10):
baseline_random_sample(logits)
torch.ops._C.fused_gumbel_argmax(logits, seeds)
run_profile(logits, seeds)
if __name__ == "__main__":
main()
@@ -0,0 +1,256 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""Benchmark the NIXL host-buffer KV copies across KV cache groups.
`NixlBaseConnectorWorker.sync_recved_kv_to_device` / `save_kv_to_host` move a
request's blocks between the CPU transfer buffer and the device KV cache. Block
ids are unique across KV cache groups, so the per-group copies can be issued as
one. This measures what that coalescing saves: the copied bytes are identical,
only the number of launches changes.
"""
import time
import torch
from tabulate import tabulate
from vllm.distributed.kv_transfer.kv_connector.utils import copy_kv_blocks
from vllm.logger import init_logger
from vllm.utils.argparse_utils import FlexibleArgumentParser
from vllm.utils.torch_utils import STR_DTYPE_TO_TORCH_DTYPE
logger = init_logger(__name__)
def _make_caches(
num_layers: int,
num_blocks: int,
block_size: int,
num_heads: int,
head_size: int,
dtype: torch.dtype,
device: str,
) -> tuple[dict[str, torch.Tensor], dict[str, torch.Tensor]]:
"""Host transfer buffers and device KV caches, shaped as the worker builds them."""
shape = (num_blocks, block_size, num_heads, head_size)
host = {
f"layer.{i}": torch.randn(shape, dtype=dtype, device="cpu")
for i in range(num_layers)
}
device_caches = {
f"layer.{i}": torch.zeros(shape, dtype=dtype, device=device)
for i in range(num_layers)
}
return host, device_caches
def _split_into_groups(block_ids: list[int], num_groups: int) -> list[list[int]]:
"""Partition block ids into groups, mirroring the global BlockPool id space."""
per_group = len(block_ids) // num_groups
return [block_ids[i * per_group : (i + 1) * per_group] for i in range(num_groups)]
def _percentile(sorted_values: list[float], q: float) -> float:
return sorted_values[min(len(sorted_values) - 1, int(len(sorted_values) * q))]
@torch.inference_mode()
def _time_paired(
host: dict[str, torch.Tensor],
device_caches: dict[str, torch.Tensor],
group_block_ids: list[list[int]],
direction: str,
num_iters: int,
) -> tuple[list[float], list[float]]:
"""Per-group and coalesced latencies, measured alternately.
Interleaving the two variants keeps drift in machine state from landing on
one of them: each iteration times both, back to back.
"""
src, dst = (host, device_caches) if direction == "h2d" else (device_caches, host)
merged = [[b for group in group_block_ids for b in group]]
def _run(batches: list[list[int]]) -> float:
start = time.perf_counter()
for ids in batches:
copy_kv_blocks(src, dst, ids, ids, direction)
torch.accelerator.synchronize()
return time.perf_counter() - start
for _ in range(5): # warmup
_run(group_block_ids)
_run(merged)
per_group: list[float] = []
coalesced: list[float] = []
for i in range(num_iters):
# Alternate which variant goes first so ordering cannot favour either.
if i % 2:
per_group.append(_run(group_block_ids))
coalesced.append(_run(merged))
else:
coalesced.append(_run(merged))
per_group.append(_run(group_block_ids))
return sorted(per_group), sorted(coalesced)
@torch.inference_mode()
def _count_device_ops(
host: dict[str, torch.Tensor],
device_caches: dict[str, torch.Tensor],
group_block_ids: list[list[int]],
direction: str,
coalesced: bool,
) -> dict[str, int]:
"""Device-op counts for one request-sync, via the profiler.
Counts do not depend on how busy the machine is, so this is the part of
the comparison that reproduces exactly.
"""
from torch.profiler import ProfilerActivity, profile
src, dst = (host, device_caches) if direction == "h2d" else (device_caches, host)
batches = (
[[b for group in group_block_ids for b in group]]
if coalesced
else group_block_ids
)
for ids in batches: # warm up allocator/autograd caches
copy_kv_blocks(src, dst, ids, ids, direction)
torch.accelerator.synchronize()
with profile(activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA]) as prof:
for ids in batches:
copy_kv_blocks(src, dst, ids, ids, direction)
torch.accelerator.synchronize()
counts: dict[str, int] = {}
for evt in prof.key_averages():
if evt.count and (evt.self_device_time_total or evt.device_time_total):
counts[evt.key] = counts.get(evt.key, 0) + evt.count
return counts
def _report_op_counts(host, device_caches, block_ids, groups):
print("\nDevice-op counts per request-sync (profiler; independent of load):")
rows = []
for num_groups in groups:
group_block_ids = _split_into_groups(block_ids, num_groups)
for direction in ("h2d", "d2h"):
before = _count_device_ops(
host, device_caches, group_block_ids, direction, coalesced=False
)
after = _count_device_ops(
host, device_caches, group_block_ids, direction, coalesced=True
)
for key in sorted(set(before) | set(after)):
b, a = before.get(key, 0), after.get(key, 0)
if b != a:
rows.append([num_groups, direction, key, b, a])
print(
tabulate(
rows,
headers=["groups", "direction", "op", "per-group", "coalesced"],
)
)
def main(args):
dtype = STR_DTYPE_TO_TORCH_DTYPE[args.dtype]
torch.manual_seed(args.seed)
host, device_caches = _make_caches(
num_layers=args.num_layers,
num_blocks=args.num_blocks,
block_size=args.block_size,
num_heads=args.num_heads,
head_size=args.head_size,
dtype=dtype,
device="cuda",
)
block_ids = list(range(args.blocks_per_request))
rows = []
for num_groups in args.groups:
group_block_ids = _split_into_groups(block_ids, num_groups)
for direction in ("h2d", "d2h"):
per_group, coalesced = _time_paired(
host,
device_caches,
group_block_ids,
direction,
num_iters=args.iters,
)
# Interference only ever adds time, so the minimum is the most
# stable estimator here; the p10 spread above it says how noisy
# the machine was while sampling.
best_before = per_group[0]
best_after = coalesced[0]
spread = (_percentile(coalesced, 0.5) - best_after) / best_after
rows.append(
[
num_groups,
direction,
best_before * 1e6,
best_after * 1e6,
(best_before - best_after) / best_before * 100.0,
spread * 100.0,
]
)
print(
f"layers={args.num_layers} num_blocks={args.num_blocks} "
f"block_size={args.block_size} heads={args.num_heads} "
f"head_size={args.head_size} dtype={args.dtype} "
f"blocks/request={args.blocks_per_request} iters={args.iters}"
)
print(
tabulate(
rows,
headers=[
"groups",
"direction",
"per-group min (µs)",
"coalesced min (µs)",
"saved (%)",
"p50 over min (%)",
],
floatfmt=".3f",
)
)
if args.op_counts:
_report_op_counts(host, device_caches, block_ids, args.groups)
if __name__ == "__main__":
parser = FlexibleArgumentParser()
parser.add_argument("--num-layers", type=int, default=48)
parser.add_argument("--num-blocks", type=int, default=2048)
parser.add_argument("--block-size", type=int, choices=[16, 32], default=16)
parser.add_argument("--num-heads", type=int, default=8)
parser.add_argument("--head-size", type=int, default=128)
parser.add_argument("--blocks-per-request", type=int, default=60)
parser.add_argument(
"--groups",
type=int,
nargs="+",
default=[1, 2, 3, 6],
help="KV cache group counts to sweep (1 = non-hybrid model).",
)
parser.add_argument(
"--dtype",
type=str,
choices=["half", "bfloat16", "float"],
default="bfloat16",
)
parser.add_argument("--iters", type=int, default=100)
parser.add_argument(
"--op-counts",
action="store_true",
help="Also report profiler op counts, which do not vary with load.",
)
parser.add_argument("--seed", type=int, default=0)
main(parser.parse_args())
+6 -1
View File
@@ -501,6 +501,7 @@ set(VLLM_EXT_SRC
"csrc/cpu/cpu_fused_moe.cpp"
"csrc/cpu/cpu_attn.cpp"
"csrc/cpu/cpu_isa.cpp"
"csrc/cpu/sampling_kernels.cpp"
"csrc/cpu/torch_bindings.cpp")
if (CMAKE_SYSTEM_PROCESSOR MATCHES "riscv64" AND VLLM_RVV_VLEN AND
@@ -523,14 +524,16 @@ if (ASIMD_FOUND AND NOT APPLE_SILICON_FOUND)
"csrc/cpu/cpu_tanhf_neon.hpp"
${VLLM_EXT_SRC})
if (ARM_BF16_FOUND)
set(VLLM_EXT_SRC "csrc/cpu/sgl-kernels/conv.cpp" ${VLLM_EXT_SRC})
if (ARM_I8MM_FOUND)
set(VLLM_EXT_SRC "csrc/cpu/cpu_fused_moe_int8.cpp" ${VLLM_EXT_SRC})
endif()
endif()
endif()
if (POWER9_FOUND OR POWER10_FOUND OR POWER11_FOUND)
if (POWER9_FOUND OR POWER10_FOUND OR POWER11_FOUND)
set(VLLM_EXT_SRC
"csrc/cpu/cpu_wna16.cpp"
"csrc/cpu/shm.cpp"
${VLLM_EXT_SRC})
endif()
@@ -589,6 +592,7 @@ if (ENABLE_X86_ISA)
"csrc/cpu/cpu_isa.cpp"
"csrc/cpu/dnnl_kernels.cpp"
"csrc/cpu/mamba_cpu.cpp"
"csrc/cpu/sampling_kernels.cpp"
"csrc/cpu/torch_bindings.cpp"
# TODO: Remove these files
"csrc/cpu/activation.cpp"
@@ -606,6 +610,7 @@ if (ENABLE_X86_ISA)
"csrc/cpu/cpu_isa.cpp"
"csrc/cpu/mamba_cpu.cpp"
"csrc/cpu/dnnl_kernels.cpp"
"csrc/cpu/sampling_kernels.cpp"
"csrc/cpu/torch_bindings.cpp"
# TODO: Remove these files
"csrc/cpu/activation.cpp"
+15 -1
View File
@@ -17,7 +17,7 @@ else()
FetchContent_Declare(
fmha_sm100
GIT_REPOSITORY https://github.com/vllm-project/MSA.git
GIT_TAG f355c37eb4e1413f21ee2ad8bbad25079e6bef9d
GIT_TAG be4a52aea420b6bef73be9af1a649859964790cb
GIT_PROGRESS TRUE
CONFIGURE_COMMAND ""
BUILD_COMMAND ""
@@ -39,6 +39,8 @@ install(FILES
"${FMHA_SM100_PY_ROOT}/api.py"
"${FMHA_SM100_PY_ROOT}/bench_utils.py"
"${FMHA_SM100_PY_ROOT}/jit.py"
"${FMHA_SM100_PY_ROOT}/nvfp4_kv.py"
"${FMHA_SM100_PY_ROOT}/q8kv4_decode_adapter.py"
"${FMHA_SM100_PY_ROOT}/sparse.py"
"${FMHA_SM100_PY_ROOT}/sparse_fmha_adapter.py"
DESTINATION vllm/third_party/fmha_sm100
@@ -51,6 +53,18 @@ install(DIRECTORY "${FMHA_SM100_PY_ROOT}/csrc/"
PATTERN "*.pyc" EXCLUDE
PATTERN ".git*" EXCLUDE)
install(DIRECTORY "${FMHA_SM100_PY_ROOT}/decode_q8kv4/"
DESTINATION vllm/third_party/fmha_sm100/decode_q8kv4
COMPONENT fmha_sm100
PATTERN "__pycache__" EXCLUDE
PATTERN "*.pyc" EXCLUDE)
install(DIRECTORY "${FMHA_SM100_PY_ROOT}/kvouter/"
DESTINATION vllm/third_party/fmha_sm100/kvouter
COMPONENT fmha_sm100
PATTERN "__pycache__" EXCLUDE
PATTERN "*.pyc" EXCLUDE)
install(DIRECTORY "${FMHA_SM100_PY_ROOT}/cute/"
DESTINATION vllm/third_party/fmha_sm100/cute
COMPONENT fmha_sm100
+45 -1
View File
@@ -417,7 +417,7 @@ struct FP32Vec8 : public Vec<FP32Vec8> {
tmp.val[1] = two_x;
FP32Vec8 temp_vec(tmp);
FP32Vec8 exp_vec = temp_vec.exp();
vector float e = temp_vec.exp().reg.val[0];
vector float e = exp_vec.reg.val[0];
vector float num = vec_sub(e, one);
vector float den = vec_add(e, one);
@@ -624,6 +624,24 @@ struct FP32Vec16 : public Vec<FP32Vec16> {
explicit FP32Vec16(const BF16Vec8& v) : FP32Vec16(FP32Vec8(v)) {}
// De-pack 16 x 4-bit nibbles from a 64-bit value and convert with offset.
// Used by WNA16 (AWQ/GPTQ) dequantization.
// Fully vectorized using VSX / AltiVec intrinsics with zero stack stores.
FORCE_INLINE explicit FP32Vec16(int64_t value, const FP32Vec16& lut) {
uint64_t u = static_cast<uint64_t>(value);
float f[16];
#pragma GCC unroll 16
for (int i = 0; i < 16; i++) {
f[i] = static_cast<float>((u >> (i * 4)) & 0xF);
}
// Extract base offset from LUT[0] (0.0f for AWQ, -8.0f for GPTQ)
__vector float offset = vec_splats(vec_extract(lut.reg.val[0], 0));
reg.val[0] = vec_add(vec_xl(0, f), offset);
reg.val[1] = vec_add(vec_xl(16, f), offset);
reg.val[2] = vec_add(vec_xl(32, f), offset);
reg.val[3] = vec_add(vec_xl(48, f), offset);
}
// FP8 stub: dead code on PowerPC (fp8 KV cache is x86-only), needed for
// load_b_pair_vec template to compile on all platforms.
explicit FP32Vec16(const BF16Vec32&, int) : reg{} {}
@@ -1102,6 +1120,32 @@ inline void prefetch(const void* addr) {
__asm__ __volatile__("dcbt 0, %0" : : "r"(addr) : "memory");
}
static void interleave_save(const BF16Vec16& vec0, const BF16Vec16& vec1,
void* ptr) {
__vector signed short h0 = vec_mergeh(vec0.reg.val[0], vec1.reg.val[0]);
__vector signed short l0 = vec_mergel(vec0.reg.val[0], vec1.reg.val[0]);
__vector signed short h1 = vec_mergeh(vec0.reg.val[1], vec1.reg.val[1]);
__vector signed short l1 = vec_mergel(vec0.reg.val[1], vec1.reg.val[1]);
vec_xst(h0, 0, reinterpret_cast<signed short*>(ptr));
vec_xst(l0, 16, reinterpret_cast<signed short*>(ptr));
vec_xst(h1, 32, reinterpret_cast<signed short*>(ptr));
vec_xst(l1, 48, reinterpret_cast<signed short*>(ptr));
}
static void interleave_save(const FP16Vec16& vec0, const FP16Vec16& vec1,
void* ptr) {
__vector signed short h0 = vec_mergeh(vec0.reg.val[0], vec1.reg.val[0]);
__vector signed short l0 = vec_mergel(vec0.reg.val[0], vec1.reg.val[0]);
__vector signed short h1 = vec_mergeh(vec0.reg.val[1], vec1.reg.val[1]);
__vector signed short l1 = vec_mergel(vec0.reg.val[1], vec1.reg.val[1]);
vec_xst(h0, 0, reinterpret_cast<signed short*>(ptr));
vec_xst(l0, 16, reinterpret_cast<signed short*>(ptr));
vec_xst(h1, 32, reinterpret_cast<signed short*>(ptr));
vec_xst(l1, 48, reinterpret_cast<signed short*>(ptr));
}
struct INT8Vec64 {
__vector signed char data[4];
INT8Vec64() = default;
+31 -2
View File
@@ -7,6 +7,9 @@
#if defined(__riscv_v)
#include "cpu/micro_gemm/cpu_micro_gemm_rvv.hpp"
#endif
#if defined(__powerpc__)
#include "cpu/micro_gemm/cpu_micro_gemm_vsx.hpp"
#endif
#include "cpu/micro_gemm/cpu_micro_gemm_vec.hpp"
#define VLLM_DISPATCH_CASE_16B_TYPES(...) \
@@ -104,8 +107,8 @@ class Dequantizer4b {
scalar_vec_t output_vec_0(wb_0);
scalar_vec_t output_vec_1(wb_1);
// AMX needs to interleave K elements to pack as 32 bits
if constexpr (isa == ISA::AMX) {
// AMX and VSX need to interleave K elements to pack as 32 bits
if constexpr (isa == ISA::AMX || isa == ISA::VSX) {
vec_op::interleave_save(output_vec_0, output_vec_1, curr_weight);
} else {
output_vec_0.save(curr_weight);
@@ -303,6 +306,8 @@ void cpu_gemm_wna16(
return ISA::VEC;
} else if (isa_hint == "rvv") {
return ISA::RVV;
} else if (isa_hint == "vsx") {
return ISA::VSX;
} else {
TORCH_CHECK(false, "unsupported isa hint: " + isa_hint);
}
@@ -378,6 +383,30 @@ void cpu_gemm_wna16(
scales_group_stride, zeros_group_stride, group_size, pack_factor);
return;
}
} else if (isa == ISA::VSX) {
TORCH_CHECK(input.scalar_type() == at::ScalarType::BFloat16,
"VSX WNA16 GEMM only supports BFloat16");
using gemm_t = cpu_micro_gemm::MicroGemm<ISA::VSX, scalar_t>;
if (has_zp) {
using dequantizer_t = Dequantizer4b<scalar_t, ISA::VSX, true>;
cpu_gemm_wna16_impl<scalar_t, dequantizer_t, gemm_t>(
input.data_ptr<scalar_t>(), q_weight.data_ptr<int32_t>(),
output.data_ptr<scalar_t>(), scales.data_ptr<scalar_t>(), zeros_ptr,
bias.has_value() ? bias->data_ptr<scalar_t>() : nullptr, a_m_size,
b_n_size, a_k_size, a_m_stride, output_m_stride,
scales_group_stride, zeros_group_stride, group_size, pack_factor);
return;
}
{
using dequantizer_t = Dequantizer4b<scalar_t, ISA::VSX, false>;
cpu_gemm_wna16_impl<scalar_t, dequantizer_t, gemm_t>(
input.data_ptr<scalar_t>(), q_weight.data_ptr<int32_t>(),
output.data_ptr<scalar_t>(), scales.data_ptr<scalar_t>(), zeros_ptr,
bias.has_value() ? bias->data_ptr<scalar_t>() : nullptr, a_m_size,
b_n_size, a_k_size, a_m_stride, output_m_stride,
scales_group_stride, zeros_group_stride, group_size, pack_factor);
return;
}
}
});
}
+151
View File
@@ -0,0 +1,151 @@
#include "cpu_types.hpp"
#include <torch/library.h>
#include <cmath>
#include <cstdint>
#include <limits>
#include <mutex>
#include <random>
namespace {
constexpr int GUMBEL_TABLE_SIZE = 1 << 20;
constexpr int GUMBEL_TABLE_MASK = GUMBEL_TABLE_SIZE - 1;
static float g_gumbel_table[GUMBEL_TABLE_SIZE];
static std::once_flag g_gumbel_init_flag;
static void init_gumbel_table() {
std::mt19937 rng(42);
std::uniform_real_distribution<float> uniform(1.0e-7f, 1.0f - 1.0e-7f);
for (int i = 0; i < GUMBEL_TABLE_SIZE; ++i) {
float u = uniform(rng);
g_gumbel_table[i] = -std::log(-std::log(u));
}
}
static inline void ensure_gumbel_table() {
std::call_once(g_gumbel_init_flag, init_gumbel_table);
}
// Fused Gumbel-max kernel
static void fused_gumbel_argmax_kernel(int64_t* __restrict__ output,
const float* __restrict__ logits,
const int64_t* __restrict__ seeds,
const int64_t batch_size,
const int64_t vocab_size) {
ensure_gumbel_table();
constexpr int VEC_ELEM_NUM = vec_op::FP32Vec8::VEC_ELEM_NUM;
const int64_t vec_end = vocab_size - (vocab_size % VEC_ELEM_NUM);
#pragma omp parallel for schedule(static)
for (int64_t b = 0; b < batch_size; ++b) {
const float* row = logits + b * vocab_size;
const uint32_t seed = static_cast<uint32_t>(seeds[b]);
float best_score = -std::numeric_limits<float>::infinity();
int64_t best_idx = 0;
for (int64_t i = 0; i < vec_end; i += VEC_ELEM_NUM) {
vec_op::FP32Vec8 logit_vec(row + i);
float gumbel_buf[VEC_ELEM_NUM];
for (int k = 0; k < VEC_ELEM_NUM; ++k)
gumbel_buf[k] = g_gumbel_table[(seed + i + k) & GUMBEL_TABLE_MASK];
vec_op::FP32Vec8 gumbel_vec(gumbel_buf);
vec_op::FP32Vec8 score = logit_vec + gumbel_vec;
float score_buf[VEC_ELEM_NUM];
score.save(score_buf);
for (int k = 0; k < VEC_ELEM_NUM; ++k) {
if (score_buf[k] > best_score) {
best_score = score_buf[k];
best_idx = i + k;
}
}
}
for (int64_t i = vec_end; i < vocab_size; ++i) {
float s = row[i] + g_gumbel_table[(seed + i) & GUMBEL_TABLE_MASK];
if (s > best_score) {
best_score = s;
best_idx = i;
}
}
output[b] = best_idx;
}
}
static void greedy_argmax_kernel(int64_t* __restrict__ output,
const float* __restrict__ logits,
const int64_t batch_size,
const int64_t vocab_size) {
constexpr int VEC_ELEM_NUM = vec_op::FP32Vec16::VEC_ELEM_NUM;
const int64_t vec_end = vocab_size - (vocab_size % VEC_ELEM_NUM);
#pragma omp parallel for schedule(static)
for (int64_t b = 0; b < batch_size; ++b) {
const float* row = logits + b * vocab_size;
vec_op::FP32Vec16 vmax(-std::numeric_limits<float>::infinity());
for (int64_t i = 0; i < vec_end; i += VEC_ELEM_NUM) {
vmax = vmax.max(vec_op::FP32Vec16(row + i));
}
float best_val = vmax.reduce_max();
for (int64_t i = vec_end; i < vocab_size; ++i) {
if (row[i] > best_val) {
best_val = row[i];
}
}
int64_t best_idx = 0;
for (int64_t i = 0; i < vocab_size; ++i) {
if (row[i] == best_val) {
best_idx = i;
break;
}
}
output[b] = best_idx;
}
}
} // namespace
torch::Tensor fused_gumbel_argmax(const torch::Tensor& logits,
const torch::Tensor& seeds) {
TORCH_CHECK(logits.dim() == 2, "logits must be 2-D [batch, vocab]");
TORCH_CHECK(logits.scalar_type() == torch::kFloat32,
"logits must be float32");
TORCH_CHECK(seeds.dim() == 1 && seeds.size(0) == logits.size(0),
"seeds must be 1-D with batch_size elements");
auto logits_contig = logits.contiguous();
auto output = torch::empty({logits_contig.size(0)}, torch::kInt64);
fused_gumbel_argmax_kernel(
output.data_ptr<int64_t>(), logits_contig.data_ptr<float>(),
seeds.data_ptr<int64_t>(), logits_contig.size(0), logits_contig.size(1));
return output;
}
torch::Tensor greedy_argmax(const torch::Tensor& logits) {
TORCH_CHECK(logits.dim() == 2, "logits must be 2-D [batch, vocab]");
TORCH_CHECK(logits.scalar_type() == torch::kFloat32,
"logits must be float32");
// A single row is faster in ATen: OpenMP fork/join dominates the scan
// (~1 ms vs ~50 µs on s390x at vocab=32k).
if (logits.size(0) <= 1) {
return logits.argmax(/*dim=*/-1);
}
auto logits_contig = logits.contiguous();
auto output = torch::empty({logits_contig.size(0)}, torch::kInt64);
greedy_argmax_kernel(output.data_ptr<int64_t>(),
logits_contig.data_ptr<float>(), logits_contig.size(0),
logits_contig.size(1));
return output;
}
+19 -7
View File
@@ -66,6 +66,10 @@ struct tinygemm_kernel {
}
};
#if defined(ARM_BF16_SUPPORT)
#include "conv_arm.h"
#endif
#if defined(CPU_CAPABILITY_AVX512)
template <int K, int BLOCK_N, bool has_bias, bool has_silu>
struct tinygemm_kernel<at::BFloat16, K, BLOCK_N, has_bias, has_silu> {
@@ -543,16 +547,24 @@ at::Tensor causal_conv1d_weight_pack(const at::Tensor& weight) {
int64_t dim = weight.size(0);
int64_t width = weight.size(1);
constexpr int64_t BLOCK_N = block_size_n();
constexpr int64_t BLOCK_N =
#if defined(ARM_BF16_SUPPORT)
8;// 8 channel blocks for microtile
#else
block_size_n();
#endif
TORCH_CHECK(width == 4, "causal_conv1d_weight_pack: support only width of 4");
TORCH_CHECK(dim % BLOCK_N == 0, "causal_conv1d_weight_pack: invalid dim size ", dim);
TORCH_CHECK(
dim % block_size_n() == 0,
"causal_conv1d_weight_pack: invalid dim size ",
dim);
const int64_t N = dim, K2 = width >> 1;
const int64_t NB = div_up(N, BLOCK_N);
auto packed_weight = at::empty_like(weight);
AT_DISPATCH_REDUCED_FLOATING_TYPES(weight.scalar_type(), "causal_conv1d_fwd_kernel_impl", [&] {
// cast to float32 as vnni size is 2
// Copy each pair as an opaque 32-bit word. This is not a bf16->fp32 conversion.
const float* w_data = reinterpret_cast<float*>(weight.data_ptr<scalar_t>());
float* packed_data = reinterpret_cast<float*>(packed_weight.data_ptr<scalar_t>());
@@ -642,9 +654,9 @@ at::Tensor causal_conv1d_fwd_cpu(
const std::optional<at::Tensor>& has_initial_state,
bool silu_activation,
int64_t pad_slot_id,
bool is_vnni) {
bool is_weight_packed) {
CHECK_CONTIGUOUS(weight);
auto packed_w = is_vnni ? weight : causal_conv1d_weight_pack(weight);
auto packed_w = is_weight_packed ? weight : causal_conv1d_weight_pack(weight);
const bool is_var_seqlen = query_start_loc.has_value();
const int64_t input_ndim = is_var_seqlen ? 2 : 3;
@@ -755,10 +767,10 @@ at::Tensor causal_conv1d_update_cpu(
const std::optional<at::Tensor>& num_accepted_tokens,
const std::optional<at::Tensor>& conv_state_indices,
int64_t pad_slot_id,
bool is_vnni) {
bool is_weight_packed) {
CHECK_CONTIGUOUS(x);
CHECK_CONTIGUOUS(weight);
auto packed_w = is_vnni ? weight : causal_conv1d_weight_pack(weight);
auto packed_w = is_weight_packed ? weight : causal_conv1d_weight_pack(weight);
TORCH_CHECK(
x.dim() == 2 || x.dim() == 3,
+126
View File
@@ -0,0 +1,126 @@
// SPDX-License-Identifier: Apache-2.0
// SPDX-FileCopyrightText: Copyright contributors to the vLLM project
#pragma once
#if defined(ARM_BF16_SUPPORT)
template <int K, int BLOCK_N, bool has_bias, bool has_silu>
struct tinygemm_kernel<at::BFloat16, K, BLOCK_N, has_bias, has_silu> {
static inline bfloat16x8_t load_bf16x8(const at::BFloat16* ptr) {
return vld1q_bf16(reinterpret_cast<const bfloat16_t*>(ptr));
}
// Input layout: A[m][channel]
// Weight layout: B[N/MICRO_N, K/2, MICRO_N, 2]
// The packed weights are reused across all M positions.
static inline void tinygemm_neon_8(
const at::BFloat16* __restrict__ A, const at::BFloat16* __restrict__ B,
at::BFloat16* __restrict__ C, const at::BFloat16* __restrict__ bias,
const at::BFloat16* __restrict__ conv_states, bool has_initial_state,
int64_t M, int64_t lda, bool is_first_token) {
bfloat16x8_t va[4];
// Load 4 pairs of kernel weights into 4 registers
const bfloat16x8_t vb0 = load_bf16x8(B); // channels 0-3, taps 0-1
const bfloat16x8_t vb1 = load_bf16x8(B + 8); // channels 4-7, taps 0-1
const bfloat16x8_t vb2 = load_bf16x8(B + 16); // channels 0-3, taps 2-3
const bfloat16x8_t vb3 = load_bf16x8(B + 24); // channels 4-7, taps 2-3
// Load the input data. A single register holds the values for a
// single token across all 8 channels for this microtile.
auto load_a = [&](int64_t m) {
if (m < 0 && is_first_token) {
if (!has_initial_state) {
return vreinterpretq_bf16_u16(vdupq_n_u16(0));
}
return load_bf16x8(conv_states + (m + K - 1) * lda);
}
return load_bf16x8(A + m * lda);
};
// Load the previous inputs from the cached state if available.
va[1] = load_a(-3);
va[2] = load_a(-2);
va[3] = load_a(-1);
// Iterate over the sequence block, updating the previous conv inputs
// and reading the next token's input data (8 channels).
for (int64_t m = 0; m < M; ++m) {
va[0] = va[1];
va[1] = va[2];
va[2] = va[3];
va[3] = load_a(m);
float32x4_t vc0;
float32x4_t vc1;
if constexpr (has_bias) {
const bfloat16x8_t bias_vec = load_bf16x8(bias);
vc0 = vcvtq_low_f32_bf16(bias_vec);
vc1 = vcvtq_high_f32_bf16(bias_vec);
} else {
vc0 = vdupq_n_f32(0.0f);
vc1 = vdupq_n_f32(0.0f);
}
// Use the zip operation to match the input layout to the weights layout
bfloat16x8_t va_pair =
vreinterpretq_bf16_u16( // channels 0-3, inputs for taps 0-1
vzip1q_u16(vreinterpretq_u16_bf16(va[0]),
vreinterpretq_u16_bf16(va[1])));
vc0 = vbfdotq_f32(vc0, va_pair,
vb0); // 4 independent length-two dot products
va_pair = vreinterpretq_bf16_u16( // channels 4-7, inputs for taps 0-1
vzip2q_u16(vreinterpretq_u16_bf16(va[0]),
vreinterpretq_u16_bf16(va[1])));
vc1 = vbfdotq_f32(vc1, va_pair, vb1);
va_pair = vreinterpretq_bf16_u16( // channels 0-3, inputs for taps 2-3
vzip1q_u16(vreinterpretq_u16_bf16(va[2]),
vreinterpretq_u16_bf16(va[3])));
// Add to the previous result for the same channels
vc0 = vbfdotq_f32(vc0, va_pair, vb2);
va_pair = vreinterpretq_bf16_u16( // channels 4-7, inputs for taps 2-3
vzip2q_u16(vreinterpretq_u16_bf16(va[2]),
vreinterpretq_u16_bf16(va[3])));
vc1 = vbfdotq_f32(vc1, va_pair, vb3);
using fVec = at::vec::Vectorized<float>;
fVec out0(vc0);
fVec out1(vc1);
if constexpr (has_silu) {
const fVec one(1.0f);
out0 = out0 / (one + out0.neg().fexp_u20());
out1 = out1 / (one + out1.neg().fexp_u20());
}
// Convert the 2 x 4 fp32 results to 1 x 8 bf16
const bfloat16x8_t out =
vcombine_bf16(vcvt_bf16_f32(static_cast<float32x4_t>(out0)),
vcvt_bf16_f32(static_cast<float32x4_t>(out1)));
vst1q_bf16(reinterpret_cast<bfloat16_t*>(C + m * lda), out);
}
}
static inline void apply(const at::BFloat16* __restrict__ A,
const at::BFloat16* __restrict__ B,
at::BFloat16* __restrict__ C,
const at::BFloat16* __restrict__ bias,
const at::BFloat16* __restrict__ conv_states,
bool has_initial_state, int64_t M, int64_t lda,
bool is_first_token) {
static_assert(K == 4);
constexpr int64_t MICRO_N = 8;
static_assert(BLOCK_N % MICRO_N == 0);
// BLOCK_N is always 32 or 64. M is the block sequence length.
// Input layout: A[m][channel]
// Weight layout: B[N/MICRO_N, K/2, MICRO_N, 2]
// Use an 8 channel microtile and iterate over the 8 channel blocks.
for (int64_t n = 0; n < BLOCK_N; n += MICRO_N) {
tinygemm_neon_8(A + n, B + n * K, C + n, has_bias ? bias + n : nullptr,
conv_states ? conv_states + n : nullptr,
has_initial_state, M, lda, is_first_token);
}
}
};
#endif
+21 -7
View File
@@ -2306,21 +2306,35 @@ std::tuple<at::Tensor, at::Tensor> chunk_gated_delta_rule_cpu(
chunk_gated_delta_rule_fwd_intra<CHUNK_SIZE>(key_, value, g_, beta, cu_seqlens, chunk_indices);
// fused `chunk_gated_delta_rule_fwd_h` + `chunk_fwd_o`
auto launch_inter = [&](auto state_tag) {
auto launch_inter = [local_query = query_,
local_key = key_,
local_w = w,
local_u = u,
local_decay_mask = decay_mask,
local_chunk_offsets = chunk_offsets,
g_,
initial_state,
output_final_state,
cu_seqlens,
initial_state_indices](auto state_tag) {
using state_t = decltype(state_tag);
return chunk_gated_delta_rule_fwd_inter<state_t, CHUNK_SIZE>(
query_,
key_,
w,
u,
local_query,
local_key,
local_w,
local_u,
g_,
decay_mask,
local_decay_mask,
initial_state,
output_final_state,
cu_seqlens,
chunk_offsets,
local_chunk_offsets,
initial_state_indices);
};
auto output_and_state =
initial_state.scalar_type() == at::kFloat
? launch_inter(float{})
+43 -21
View File
@@ -281,14 +281,14 @@ at::Tensor causal_conv1d_fwd_cpu(
const std::optional<at::Tensor>& query_start_loc,
const std::optional<at::Tensor>& cache_indices,
const std::optional<at::Tensor>& has_initial_state, bool silu_activation,
int64_t pad_slot_id, bool is_vnni);
int64_t pad_slot_id, bool is_weight_packed);
at::Tensor causal_conv1d_update_cpu(
const at::Tensor& x, const at::Tensor& conv_states,
const at::Tensor& weight, const std::optional<at::Tensor>& bias,
bool silu_activation, const std::optional<at::Tensor>& num_accepted_tokens,
const std::optional<at::Tensor>& conv_state_indices, int64_t pad_slot_id,
bool is_vnni);
bool is_weight_packed);
void activation_lut_bf16(torch::Tensor& out, torch::Tensor& input,
const std::string& activation);
@@ -393,6 +393,11 @@ void mamba_chunk_scan_fwd_cpu_impl(at::Tensor& out, at::Tensor& final_states,
const c10::optional<at::Tensor>& z,
const at::Tensor& cu_seqlens);
torch::Tensor fused_gumbel_argmax(const torch::Tensor& logits,
const torch::Tensor& seeds);
torch::Tensor greedy_argmax(const torch::Tensor& logits);
void init_cpu_memory_env(std::vector<int64_t> node_ids);
namespace cpu_utils {
@@ -512,6 +517,33 @@ TORCH_LIBRARY_EXPAND(TORCH_EXTENSION_NAME, ops) {
#endif // (defined(__aarch64__) && !defined(__APPLE__))
#if (defined(__aarch64__) && defined(ARM_BF16_SUPPORT) && \
!defined(__APPLE__)) || \
(defined(__AVX512BF16__) && defined(__AVX512F__) && \
defined(__AVX512VNNI__))
// Adapted from sglang: casual_conv1d kernels
ops.def("causal_conv1d_weight_pack(Tensor weight) -> Tensor");
ops.impl("causal_conv1d_weight_pack", torch::kCPU,
&causal_conv1d_weight_pack);
ops.def(
"causal_conv1d_fwd_cpu(Tensor x, Tensor weight, Tensor? bias, Tensor? "
"conv_states, Tensor? query_start_loc,"
"Tensor? cache_indices, Tensor? has_initial_state, bool silu_activation, "
"int pad_slot_id, bool is_weight_packed) -> "
"Tensor");
ops.impl("causal_conv1d_fwd_cpu", torch::kCPU, &causal_conv1d_fwd_cpu);
ops.def(
"causal_conv1d_update_cpu(Tensor x, Tensor(a!) conv_states, Tensor "
"weight, Tensor? bias, bool silu_activation,"
"Tensor? num_accepted_tokens, Tensor? conv_state_indices, int "
"pad_slot_id, "
"bool is_weight_packed) -> Tensor");
ops.impl("causal_conv1d_update_cpu", torch::kCPU, &causal_conv1d_update_cpu);
#endif
// Layernorm
// Apply Root Mean Square (RMS) Normalization to the input tensor.
ops.def(
@@ -660,24 +692,6 @@ TORCH_LIBRARY_EXPAND(TORCH_EXTENSION_NAME, ops) {
"-> (Tensor, Tensor)");
ops.impl("quantize_fp8e4m3_vec", torch::kCPU, &quantize_fp8e4m3_vec);
ops.def("causal_conv1d_weight_pack(Tensor weight) -> Tensor");
ops.impl("causal_conv1d_weight_pack", torch::kCPU,
&causal_conv1d_weight_pack);
ops.def(
"causal_conv1d_fwd_cpu(Tensor x, Tensor weight, Tensor? bias, Tensor? "
"conv_states, Tensor? query_start_loc,"
"Tensor? cache_indices, Tensor? has_initial_state, bool silu_activation, "
"int pad_slot_id, bool is_vnni) -> "
"Tensor");
ops.impl("causal_conv1d_fwd_cpu", torch::kCPU, &causal_conv1d_fwd_cpu);
ops.def(
"causal_conv1d_update_cpu(Tensor x, Tensor(a!) conv_states, Tensor "
"weight, Tensor? bias, bool silu_activation,"
"Tensor? num_accepted_tokens, Tensor? conv_state_indices, int "
"pad_slot_id, "
"bool is_vnni) -> Tensor");
ops.impl("causal_conv1d_update_cpu", torch::kCPU, &causal_conv1d_update_cpu);
// Adapted from sglang: MLA CPU kernels (AMX-only, DeepSeek V2/V3/R1)
ops.def(
"decode_attention_cpu(Tensor query, Tensor k_buffer, Tensor v_buffer, "
@@ -897,7 +911,8 @@ TORCH_LIBRARY_EXPAND(TORCH_EXTENSION_NAME, ops) {
ops.def("dynamic_per_token_scaled_fp8_quant() -> ()", placeholder_op);
// WNA16
#if defined(__AVX512F__) || defined(__riscv_v) || defined(__s390x__)
#if defined(__AVX512F__) || defined(__riscv_v) || defined(__s390x__) || \
defined(__powerpc__)
ops.def(
"cpu_gemm_wna16(Tensor input, Tensor q_weight, Tensor(a2!) output, "
"Tensor scales, Tensor? zeros, Tensor? bias, SymInt "
@@ -969,6 +984,13 @@ TORCH_LIBRARY_EXPAND(TORCH_EXTENSION_NAME, ops) {
ops.def("init_cpu_memory_env(SymInt[] node_ids) -> ()", &init_cpu_memory_env);
// Fused sampling kernels
ops.def("fused_gumbel_argmax(Tensor logits, Tensor seeds) -> Tensor");
ops.impl("fused_gumbel_argmax", torch::kCPU, &fused_gumbel_argmax);
ops.def("greedy_argmax(Tensor logits) -> Tensor");
ops.impl("greedy_argmax", torch::kCPU, &greedy_argmax);
// Speculative decoding kernels
ops.def(
"eagle_prepare_inputs_padded_kernel_impl(Tensor cu_num_draft_tokens, "
@@ -285,6 +285,262 @@ __device__ __forceinline__ void storeScaledQElemsFp8(
storeElemsFp8(dst, scaled);
}
// index_q: E4M3(scalar_t(v)) with unit scale, i.e. q is rounded to the model
// dtype before the fp8 cast. On CUDA each element pair is rounded to scalar_t
// and then converted to E4M3, with one packed cvt each.
template <typename scalar_t>
__device__ __forceinline__ void storeIndexQElemsFp8(
uint8_t* __restrict__ dst, float const (&elems)[kElemsPerLane]) {
#ifndef USE_ROCM
using Converter = vllm::_typeConvert<scalar_t>;
__nv_fp8x2_storage_t out2[kElemsPerLane / 2];
#pragma unroll
for (int i = 0; i < kElemsPerLane / 2; i++) {
auto const v =
Converter::convert(make_float2(elems[2 * i], elems[2 * i + 1]));
if constexpr (std::is_same_v<scalar_t, c10::BFloat16>) {
out2[i] = __nv_cvt_bfloat16raw2_to_fp8x2(v, __NV_SATFINITE, __NV_E4M3);
} else {
out2[i] = __nv_cvt_halfraw2_to_fp8x2(v, __NV_SATFINITE, __NV_E4M3);
}
}
*reinterpret_cast<uint32_t*>(dst) = *reinterpret_cast<uint32_t const*>(out2);
#else
storeScaledQElemsFp8<scalar_t>(dst, elems, 1.0f);
#endif
}
// ────────────────────────────────────────────────────────────────────────────
// NVFP4 main K/V cache store
// ────────────────────────────────────────────────────────────────────────────
// The block encoding of reshape_and_cache_nvfp4_kernel
// (nvfp4_kv_cache_kernels.cu) in the MSA readers' head-slot page layout. The
// quantizer differs from that kernel's reciprocal-multiply math: per 16-value
// block, scale = min(448, max(E4M3(max(amax, 1e-12) / 6), 1/512)) and
// code = sign | RNE-E2M1(min(|x / scale|, 6)), with zero stored as +0, all on
// x / the global scale.
// An HND page is 2*nkv slots of [bs, 72] bytes; slot 2*head + side (K = 0,
// V = 1) holds that head's packed E2M1 data [bs, 64] followed by its E4M3 block
// scales [bs, 8]. Every head is one contiguous 2*bs*72-byte run, so a page
// split across TP ranks is a contiguous byte slice per rank. K scales are
// linear, V scales use the SM100 trtllm-gen 4x4 token-quad swizzle. One
// 16-element scale group is four lanes. SM100-family builds of this file
// (CMake gives it family-specific SM10x/11x targets) encode E2M1 with
// cvt.rn.satfinite.e2m1x2; other targets use a bit-exact software encoder.
#if defined(__CUDA_ARCH__) && __CUDA_ARCH__ >= 1000 && \
defined(__CUDA_ARCH_FAMILY_SPECIFIC__)
#define MINIMAX_M3_NVFP4_NATIVE_CVT 1
#else
#define MINIMAX_M3_NVFP4_NATIVE_CVT 0
#endif
#ifndef USE_ROCM
constexpr int kNvfp4DataBytes = kHeadDim / 2; // 64
constexpr int kNvfp4ScaleBytes = kHeadDim / 16; // 8
constexpr int kNvfp4LanesPerGroup = 16 / kElemsPerLane; // 4
// Round-to-nearest-even, saturating E2M1 code of ``x`` (matches
// cvt.rn.satfinite.e2m1x2.f32). Below 1 the grid is the 0.5-step subnormals;
// from 1 up it is fp32 rounded to one mantissa bit, whose (exponent, mantissa)
// bits are the code offset by 252 once the exponent is rebiased.
__device__ __forceinline__ uint32_t e2m1Code(float x) {
float const ax = fabsf(x);
uint32_t mag;
if (ax < 1.0f) {
mag = static_cast<uint32_t>(__float2int_rn(2.0f * ax));
} else {
uint32_t const bits = __float_as_uint(ax);
uint32_t const rounded = bits + 0x1FFFFFu + ((bits >> 22) & 1u);
mag = min((rounded >> 22) - 252u, 7u);
}
return mag | ((__float_as_uint(x) >> 28) & 0x8u);
}
__device__ __forceinline__ float rcpApproxFtz(float a) {
float b;
asm("rcp.approx.ftz.f32 %0, %1;" : "=f"(b) : "f"(a));
return b;
}
// a / b rounded to nearest even from r = RN(1 / b): Markstein's correction of
// a * r, which is __fdiv_rn's fast path without its range-check branch. Exact
// unless a * r overflows or the residual a - q * b underflows (a within a few
// binades of the subnormals).
__device__ __forceinline__ float divRnByRcp(float a, float b, float r) {
float const q = __fmul_rn(a, r);
return __fmaf_rn(__fmaf_rn(-q, b, a), r, q);
}
// Block scale E4M3(max(amax, 1e-12) / 6) clamped to [1/512, 448], with
// sf_scale applied to it as to the values, and its reciprocal RN(1 / sf_value).
__device__ __forceinline__ void nvfp4GroupScale(float sf_scale, float group_max,
uint8_t& sf_byte,
float& sf_value,
float& sf_rcp) {
constexpr float kRcp6 = 0x1.555556p-3f; // RN(1 / 6)
float const amax = fmaxf(group_max, 1e-12f);
// amax / 6 (RNE). amax * kRcp6 bounds it from above, and fminf picks it for
// amax = inf, where the correction is NaN.
float const amax_6 =
fminf(divRnByRcp(amax, 6.0f, kRcp6), __fmul_rn(amax, kRcp6));
// Flooring before the rounding gives the 1/512 minimum; satfinite caps it
// at 448.
__nv_fp8_e4m3 const sf(fmaxf(sf_scale * amax_6, 0x1p-9f));
sf_byte = sf.__x;
sf_value = float(sf);
// rcp.approx is correctly rounded on every positive E4M3 value.
sf_rcp = rcpApproxFtz(sf_value);
}
// nvfp4GroupScale for sf_scale = 1 and a bf16/fp16 group_max. Uncorrected,
// amax * RN(1 / 6) still rounds to the E4M3 byte of amax / 6 (checked for every
// such amax: amax / 6 is too far from an E4M3 rounding boundary to cross it,
// and exact ties round back), and the 1e-12 floor lies below the 1/512 one.
__device__ __forceinline__ void nvfp4UnitGroupScale(float group_max,
uint8_t& sf_byte,
float& sf_rcp) {
constexpr float kRcp6 = 0x1.555556p-3f; // RN(1 / 6)
__nv_fp8_e4m3 const sf(fmaxf(__fmul_rn(group_max, kRcp6), 0x1p-9f));
sf_byte = sf.__x;
sf_rcp = rcpApproxFtz(float(sf));
}
// (x * sf_scale) / sf_value with the E2M1 code of the IEEE quotient. The
// quotient itself is the IEEE one except below 1e-35 (code 0 either way) and
// for magnitudes over 4096, which are clamped (NaN kept) so that the division
// stays finite: 4096 / 448 > 5, so their code stays the saturated 6.
__device__ __forceinline__ float nvfp4Quotient(float x, float sf_scale,
float sf_value, float sf_rcp) {
constexpr float kMaxMag = 4096.0f;
float n = __fmul_rn(x, sf_scale);
n = fabsf(n) > kMaxMag ? copysignf(kMaxMag, n) : n;
return divRnByRcp(n, sf_value, sf_rcp);
}
// Zero is stored as +0: clear the sign of every nibble whose magnitude is 0.
// Adding 7 to a 3-bit magnitude carries into the sign bit exactly when it is
// nonzero.
__device__ __forceinline__ uint16_t clearNegZeroNibbles(uint16_t packed) {
return packed & (((packed & 0x7777u) + 0x7777u) | 0x7777u);
}
// Quantize one K or V head row of one token into its NVFP4 page, where
// ``scale`` is the dequantization (global) scale. Every lane of the warp must
// call this (the scale-group reductions shuffle).
template <typename scalar_t>
__device__ __forceinline__ void storeNvfp4CacheElems(
uint8_t* __restrict__ page, int const kv, int const head, int const token,
int const block_size, int const laneId, float const (&elems)[kElemsPerLane],
float const scale) {
// Quantize the model-dtype value that the unfused path reads back from qkv.
using Converter = vllm::_typeConvert<scalar_t>;
float x[kElemsPerLane];
float group_max = 0.0f;
#pragma unroll
for (int i = 0; i < kElemsPerLane; i++) {
x[i] = static_cast<float>(Converter::convert(elems[i]));
group_max = fmaxf(group_max, fabsf(x[i]));
}
group_max = fmaxf(group_max, __shfl_xor_sync(FINAL_MASK, group_max, 1, 32));
group_max = fmaxf(group_max, __shfl_xor_sync(FINAL_MASK, group_max, 2, 32));
uint8_t sf_byte;
float sf_rcp;
float q[kElemsPerLane]; // (x / scale) / sf_value, E2M1-encoded below
if (scale == 1.0f) {
static_assert(sizeof(scalar_t) == 2, "the unit-scale path needs bf16/fp16");
nvfp4UnitGroupScale(group_max, sf_byte, sf_rcp);
// x / sf_value truncated from x * RN(1 / sf_value). RN(1 / s) >= 1 / s for
// every E4M3 s, so a bf16/fp16 x on an E2M1 midpoint lands exactly on it
// (and the cvt rounds it to even), while any other bf16/fp16 x is too far
// from a midpoint to reach one. No clamp either: a finite product never
// truncates to inf, and inf saturates in the cvt.
#if MINIMAX_M3_NVFP4_NATIVE_CVT
float2 const r2 = make_float2(sf_rcp, sf_rcp);
#pragma unroll
for (int i = 0; i < kElemsPerLane; i += 2) {
float2 const p = __fmul2_rz(make_float2(x[i], x[i + 1]), r2);
q[i] = p.x;
q[i + 1] = p.y;
}
#else
#pragma unroll
for (int i = 0; i < kElemsPerLane; i++) q[i] = __fmul_rz(x[i], sf_rcp);
#endif
} else {
float const sf_scale = __frcp_rn(scale); // exactly 1.0f / scale
float sf_value;
nvfp4GroupScale(sf_scale, group_max, sf_byte, sf_value, sf_rcp);
// A correctly rounded division, not a reciprocal multiply, so that exact
// E2M1 midpoints round to even.
#if MINIMAX_M3_NVFP4_NATIVE_CVT
// nvfp4Quotient on element pairs, with SM100's packed FP32 instructions.
float2 const s2 = make_float2(sf_scale, sf_scale);
float2 const neg_v2 = make_float2(-sf_value, -sf_value);
float2 const r2 = make_float2(sf_rcp, sf_rcp);
#pragma unroll
for (int i = 0; i < kElemsPerLane; i += 2) {
float2 n = __fmul2_rn(make_float2(x[i], x[i + 1]), s2);
// |n| <= 4096 as in nvfp4Quotient, NaN kept.
asm("max.NaN.f32 %0, %0, 0fC5800000;\n\t"
"min.NaN.f32 %0, %0, 0f45800000;\n\t"
"max.NaN.f32 %1, %1, 0fC5800000;\n\t"
"min.NaN.f32 %1, %1, 0f45800000;"
: "+f"(n.x), "+f"(n.y));
float2 const q0 = __fmul2_rn(n, r2);
float2 const p = __ffma2_rn(__ffma2_rn(q0, neg_v2, n), r2, q0);
q[i] = p.x;
q[i + 1] = p.y;
}
#else
#pragma unroll
for (int i = 0; i < kElemsPerLane; i++) {
q[i] = nvfp4Quotient(x[i], sf_scale, sf_value, sf_rcp);
}
#endif
}
uint16_t packed;
#if MINIMAX_M3_NVFP4_NATIVE_CVT
// Low nibble = even element; cvt packs its first source in the high nibble.
asm("{\n"
" .reg .b8 lo, hi;\n"
" cvt.rn.satfinite.e2m1x2.f32 lo, %2, %1;\n"
" cvt.rn.satfinite.e2m1x2.f32 hi, %4, %3;\n"
" mov.b16 %0, {lo, hi};\n"
"}"
: "=h"(packed)
: "f"(q[0]), "f"(q[1]), "f"(q[2]), "f"(q[3]));
#else
uint32_t codes = 0;
#pragma unroll
for (int i = 0; i < kElemsPerLane; i++) codes |= e2m1Code(q[i]) << (4 * i);
packed = static_cast<uint16_t>(codes);
#endif
packed = clearNegZeroNibbles(packed);
uint8_t* const slot = page + static_cast<int64_t>(2 * head + kv) *
block_size *
(kNvfp4DataBytes + kNvfp4ScaleBytes);
uint8_t* const data =
slot + token * kNvfp4DataBytes + laneId * (kElemsPerLane / 2);
*reinterpret_cast<uint16_t*>(data) = packed;
if (laneId % kNvfp4LanesPerGroup == 0) {
int const group = laneId / kNvfp4LanesPerGroup;
int offset;
if (kv == 0) {
offset = token * kNvfp4ScaleBytes + group;
} else {
constexpr int kGroupsPerQuad = kNvfp4ScaleBytes / 4;
int const swizzled_t = (token / 4) * 4 + group / kGroupsPerQuad;
int const swizzled_s = (group % kGroupsPerQuad) * 4 + token % 4;
offset = swizzled_t * kNvfp4ScaleBytes + swizzled_s;
}
slot[block_size * kNvfp4DataBytes + offset] = sf_byte;
}
}
#endif // USE_ROCM
// ────────────────────────────────────────────────────────────────────────────
// Kernel
// ────────────────────────────────────────────────────────────────────────────
@@ -307,9 +563,8 @@ __device__ __forceinline__ void storeScaledQElemsFp8(
// branch and writes index_q/index_k outputs. Skip-index-topk reuse layers keep
// kHasIndex=true but set kProcessIndex=false.
template <typename scalar_t, typename cache_t, Fp8KVCacheDataType kv_dt,
typename out_idx_t, bool kHasIndex, bool kInsertKV,
bool kProcessIndex,
bool kFp8Idx>
bool kNvfp4, typename out_idx_t, bool kHasIndex, bool kInsertKV,
bool kProcessIndex, bool kFp8Idx>
__global__ void fusedMiniMaxM3QNormRopeKVInsertKernel(
scalar_t* __restrict__ qkv, // [N, qkv_row] in/out (packs index if sparse)
scalar_t* __restrict__ q_out, // [N, nq*128] contiguous, or nullptr
@@ -332,7 +587,10 @@ __global__ void fusedMiniMaxM3QNormRopeKVInsertKernel(
// The content (last) dim is always innermost-contiguous (stride 1), so the
// NHD/HND layout choice is captured by the head/token strides.
int64_t const kv_s_block, int64_t const kv_s_head, int64_t const kv_s_token,
int64_t const kv_s_dim) {
int64_t const kv_s_dim,
// NVFP4 main cache only: per-layer K/V dequantization (global) scales.
float const* __restrict__ kv_k_scale,
float const* __restrict__ kv_v_scale) {
#if (!defined(__CUDA_ARCH__) || __CUDA_ARCH__ < 800) && !defined(USE_ROCM)
// _typeConvert<BFloat16> is unavailable on pre-Ampere; the M3 kernel only
// runs with bf16/fp16 inputs in practice. Discard the bf16 body there.
@@ -352,8 +610,11 @@ __global__ void fusedMiniMaxM3QNormRopeKVInsertKernel(
int const idx_slots = kProcessIndex ? niq + 1 : 0;
int const slots_per_token = nq + nkv + v_slots + idx_slots;
int const tokenIdx = globalWarpIdx / slots_per_token;
int const slot = globalWarpIdx % slots_per_token;
// Unsigned: both are non-negative, and the signed division's sign handling
// would cost every warp.
unsigned const warp_u = globalWarpIdx, slots_u = slots_per_token;
int const tokenIdx = warp_u / slots_u;
int const slot = warp_u % slots_u;
if (tokenIdx >= num_tokens) return;
// Slot boundaries.
@@ -420,10 +681,13 @@ __global__ void fusedMiniMaxM3QNormRopeKVInsertKernel(
// flat TMA descriptor can address them as [tokens*heads, head_dim]; this
// folds the de-interleaving into the store the kernel already does, instead
// of a separate q.contiguous() copy. Everything else stays in place.
// Given q_fp8_out but no q_out, q is emitted only in fp8 (null store_ptr).
scalar_t* store_ptr = row_ptr;
if (isQ && q_out != nullptr) {
store_ptr = q_out + static_cast<int64_t>(tokenIdx) * nq * kHeadDim +
slot * kHeadDim;
} else if (isQ && q_fp8_out != nullptr) {
store_ptr = nullptr;
} else if (isIQ && index_q_out != nullptr) {
// bf16 index_q_out: gather here. fp8: written by the explicit fp8 store.
if constexpr (!kFp8Idx) {
@@ -433,6 +697,13 @@ __global__ void fusedMiniMaxM3QNormRopeKVInsertKernel(
}
}
// NVFP4: the per-layer global scale is not produced by the predecessor,
// so load it before the PDL wait.
float nvfp4_scale = 0.0f;
if constexpr (kNvfp4) {
if (isK || isV) nvfp4_scale = isK ? *kv_k_scale : *kv_v_scale;
}
// PDL: wait for the predecessor kernel (the qkv-projection GEMM that
// produces ``qkv``) to finish before touching any global memory. No-op
// when PDL is not enabled on the launch. The CUDA runtime wrapper emits
@@ -451,16 +722,17 @@ __global__ void fusedMiniMaxM3QNormRopeKVInsertKernel(
normAndRope<scalar_t>(elems, laneId, eps, norm_w, do_rope, rotary_dim,
cos_ptr, /*apply_norm=*/norm_w != nullptr);
if constexpr (kFp8Idx) {
// index_q is e4m3 bytes; Q/K (and in-place index_k) stay scalar_t.
// index_q is E4M3(scalar_t(q)) bytes; Q/K (and in-place index_k) stay
// scalar_t.
if (isIQ && index_q_out != nullptr) {
storeElemsFp8(index_q_out +
static_cast<int64_t>(tokenIdx) * niq * kHeadDim +
(slot - iq_begin) * kHeadDim + dim_base,
elems);
} else {
storeIndexQElemsFp8<scalar_t>(
index_q_out + static_cast<int64_t>(tokenIdx) * niq * kHeadDim +
(slot - iq_begin) * kHeadDim + dim_base,
elems);
} else if (store_ptr != nullptr) {
storeElems<scalar_t>(store_ptr + dim_base, elems);
}
} else {
} else if (store_ptr != nullptr) {
storeElems<scalar_t>(store_ptr + dim_base, elems);
}
if (isQ && q_fp8_out != nullptr) {
@@ -487,6 +759,16 @@ __global__ void fusedMiniMaxM3QNormRopeKVInsertKernel(
} else {
storeElems<scalar_t>(index_cache + sm * kHeadDim + dim_base, elems);
}
} else if constexpr (kNvfp4) {
#ifndef USE_ROCM
// kv_cache is [num_blocks, 2*nkv, block_size, 72] bytes (HND), slot
// 2*head + side; kv_s_block is its page stride.
storeNvfp4CacheElems<scalar_t>(
reinterpret_cast<uint8_t*>(kv_cache) +
(sm / block_size) * kv_s_block,
isK ? 0 : 1, head, static_cast<int>(sm % block_size), block_size,
laneId, elems, nvfp4_scale);
#endif
} else if (isK || isV) {
// kv_cache logical shape [num_blocks, nkv, block_size, 2*head_dim].
// Paging is logical (block = sm/block_size, token = sm%block_size);
@@ -515,7 +797,8 @@ __global__ void fusedMiniMaxM3QNormRopeKVInsertKernel(
// ────────────────────────────────────────────────────────────────────────────
// Launch wrapper
// ────────────────────────────────────────────────────────────────────────────
template <typename scalar_t, typename cache_t, Fp8KVCacheDataType kv_dt>
template <typename scalar_t, typename cache_t, Fp8KVCacheDataType kv_dt,
bool kNvfp4 = false>
void launchFusedMiniMaxM3(
scalar_t* qkv, scalar_t* q_out, uint8_t* q_fp8_out, void* index_q_out,
scalar_t const* q_norm_w, scalar_t const* k_norm_w,
@@ -527,7 +810,8 @@ void launchFusedMiniMaxM3(
int const nq, int const nkv, int const niq, int const block_size,
int64_t const kv_s_block, int64_t const kv_s_head, int64_t const kv_s_token,
int64_t const kv_s_dim, bool const has_index, bool const insert_kv,
bool const process_index, bool const fp8_idx, cudaStream_t stream) {
bool const process_index, bool const fp8_idx, float const* kv_k_scale,
float const* kv_v_scale, cudaStream_t stream) {
// Index outputs are scalar_t (bf16) or e4m3 bytes (uint8_t); reinterpret the
// void* pointers per instantiation in the LAUNCH macro.
// Slot count must match the kernel's compile-time gating.
@@ -559,35 +843,44 @@ void launchFusedMiniMaxM3(
config.attrs = attrs;
config.numAttrs = (sm_version >= 90) ? 1 : 0;
#define LAUNCH(HAS_INDEX, INSERT, PROCESS_INDEX, FP8, OUT_T) \
cudaLaunchKernelEx( \
&config, \
fusedMiniMaxM3QNormRopeKVInsertKernel<scalar_t, cache_t, kv_dt, OUT_T, \
HAS_INDEX, INSERT, \
PROCESS_INDEX, FP8>, \
qkv, q_out, q_fp8_out, reinterpret_cast<OUT_T*>(index_q_out), \
q_norm_w, k_norm_w, iq_norm_w, ik_norm_w, cos_sin_cache, positions, \
slot_mapping, index_slot_mapping, kv_cache, \
reinterpret_cast<OUT_T*>(index_cache), eps, q_fp8_inv_scale, \
rotary_dim, num_tokens, nq, nkv, niq, block_size, kv_s_block, \
kv_s_head, kv_s_token, kv_s_dim)
#else
// ROCm: standard kernel launch syntax (no PDL/stream serialization).
// clang-format off
#define LAUNCH(HAS_INDEX, INSERT, PROCESS_INDEX, FP8, OUT_T) \
fusedMiniMaxM3QNormRopeKVInsertKernel< \
scalar_t, cache_t, kv_dt, OUT_T, HAS_INDEX, INSERT, PROCESS_INDEX, \
FP8><<<grid, kBlockSize, 0, stream>>>( \
cudaLaunchKernelEx( \
&config, \
fusedMiniMaxM3QNormRopeKVInsertKernel<scalar_t, cache_t, kv_dt, \
kNvfp4, OUT_T, HAS_INDEX, \
INSERT, PROCESS_INDEX, FP8>, \
qkv, q_out, q_fp8_out, reinterpret_cast<OUT_T*>(index_q_out), \
q_norm_w, k_norm_w, iq_norm_w, ik_norm_w, cos_sin_cache, positions, \
slot_mapping, index_slot_mapping, kv_cache, \
reinterpret_cast<OUT_T*>(index_cache), eps, q_fp8_inv_scale, \
rotary_dim, num_tokens, nq, nkv, niq, block_size, kv_s_block, \
kv_s_head, kv_s_token, kv_s_dim)
kv_s_head, kv_s_token, kv_s_dim, kv_k_scale, kv_v_scale)
#else
// ROCm: standard kernel launch syntax (no PDL/stream serialization).
// clang-format off
#define LAUNCH(HAS_INDEX, INSERT, PROCESS_INDEX, FP8, OUT_T) \
fusedMiniMaxM3QNormRopeKVInsertKernel< \
scalar_t, cache_t, kv_dt, kNvfp4, OUT_T, HAS_INDEX, INSERT, \
PROCESS_INDEX, FP8><<<grid, kBlockSize, 0, stream>>>( \
qkv, q_out, q_fp8_out, reinterpret_cast<OUT_T*>(index_q_out), \
q_norm_w, k_norm_w, iq_norm_w, ik_norm_w, cos_sin_cache, positions, \
slot_mapping, index_slot_mapping, kv_cache, \
reinterpret_cast<OUT_T*>(index_cache), eps, q_fp8_inv_scale, \
rotary_dim, num_tokens, nq, nkv, niq, block_size, kv_s_block, \
kv_s_head, kv_s_token, kv_s_dim, kv_k_scale, kv_v_scale)
// clang-format on
#endif
if (has_index) {
if constexpr (kNvfp4) {
// NVFP4 main cache: sparse serving only (the host checks insert_kv).
if (!process_index) {
LAUNCH(true, true, false, false, scalar_t);
} else if (fp8_idx) {
LAUNCH(true, true, true, true, uint8_t);
} else {
LAUNCH(true, true, true, false, scalar_t);
}
} else if (has_index) {
if (!process_index) {
if (insert_kv) {
LAUNCH(true, true, false, false, scalar_t);
@@ -621,8 +914,9 @@ void launchFusedMiniMaxM3(
} // namespace vllm
// clang-format off
#define CALL_FUSED_MINIMAX_M3(_RAW_T, CACHE_T, KV_DTYPE) \
vllm::minimax_m3_fused_ops::launchFusedMiniMaxM3<st, CACHE_T, KV_DTYPE>( \
#define CALL_FUSED_MINIMAX_M3_IMPL(CACHE_T, KV_DTYPE, NVFP4) \
vllm::minimax_m3_fused_ops::launchFusedMiniMaxM3<st, CACHE_T, KV_DTYPE, \
NVFP4>( \
reinterpret_cast<st*>(qkv.data_ptr()), \
q_out.has_value() ? reinterpret_cast<st*>(q_out->data_ptr()) : nullptr, \
q_fp8_out.has_value() \
@@ -654,7 +948,10 @@ void launchFusedMiniMaxM3(
static_cast<float>(eps), 1.0f / static_cast<float>(q_fp8_scale), \
static_cast<int>(rotary_dim), num_tokens, nq, nkv, niq, \
static_cast<int>(block_size), kv_s_block, kv_s_head, kv_s_token, \
kv_s_dim, has_index, insert_kv, process_index, fp8_idx, stream)
kv_s_dim, has_index, insert_kv, process_index, fp8_idx, kv_k_scale_ptr, \
kv_v_scale_ptr, stream)
#define CALL_FUSED_MINIMAX_M3(_RAW_T, CACHE_T, KV_DTYPE) \
CALL_FUSED_MINIMAX_M3_IMPL(CACHE_T, KV_DTYPE, false)
// clang-format on
// ────────────────────────────────────────────────────────────────────────────
@@ -681,7 +978,9 @@ void fused_minimax_m3_qknorm_rope_kv_insert(
const std::string& kv_cache_dtype, bool skip_index_branch,
std::optional<torch::stable::Tensor>
q_fp8_out, // [N, nq*128] contiguous E4M3
double q_fp8_scale) {
double q_fp8_scale,
std::optional<torch::stable::Tensor> kv_k_scale, // [] f32, NVFP4 only
std::optional<torch::stable::Tensor> kv_v_scale) {
STD_TORCH_CHECK(qkv.is_cuda() && qkv.is_contiguous(),
"qkv must be contiguous CUDA");
STD_TORCH_CHECK(
@@ -721,8 +1020,10 @@ void fused_minimax_m3_qknorm_rope_kv_insert(
bool const has_index = niq > 0;
bool const insert_kv = kv_cache.has_value();
bool const process_index = has_index && !skip_index_branch;
bool const nvfp4_kv = kv_cache_dtype == "nvfp4";
vllm::Fp8KVCacheDataType const kv_dt =
vllm::get_fp8_kv_cache_data_type(kv_cache_dtype);
nvfp4_kv ? vllm::Fp8KVCacheDataType::kAuto
: vllm::get_fp8_kv_cache_data_type(kv_cache_dtype);
int const kHeadDim = vllm::minimax_m3_fused_ops::kHeadDim;
int const expected_row =
(nq + 2 * nkv + (has_index ? niq + 1 : 0)) * kHeadDim;
@@ -769,8 +1070,13 @@ void fused_minimax_m3_qknorm_rope_kv_insert(
index_slot_mapping->numel() == slot_mapping->numel()),
"index_slot_mapping must be int64 CUDA with slot_mapping length");
}
// Main attention KV cache: auto matches qkv, fp8 uses uint8 storage.
if (kv_dt == vllm::Fp8KVCacheDataType::kAuto) {
// Main attention KV cache: auto matches qkv, fp8 and nvfp4 use uint8
// storage.
if (nvfp4_kv) {
STD_TORCH_CHECK(
kv_cache->scalar_type() == torch::headeronly::ScalarType::Byte,
"nvfp4 kv_cache must use uint8 storage");
} else if (kv_dt == vllm::Fp8KVCacheDataType::kAuto) {
STD_TORCH_CHECK(kv_cache->scalar_type() == qkv.scalar_type(),
"auto kv_cache dtype must match qkv");
} else {
@@ -790,6 +1096,27 @@ void fused_minimax_m3_qknorm_rope_kv_insert(
STD_TORCH_CHECK(kv_cache->dim() == 4 && kv_cache->stride(3) == 1,
"kv_cache must be [nb,nkv,bs,2*head_dim] with contiguous "
"content dim (stride(3)==1)");
if (nvfp4_kv) {
// Packed HND pages [nb, 2*nkv, bs, 72]: slot 2*head + side is one
// contiguous [bs, 72] block, which the store addresses directly.
int64_t const full_dim = kHeadDim / 2 + kHeadDim / 16;
STD_TORCH_CHECK(
kv_cache->size(1) == 2 * nkv && kv_cache->size(2) == block_size &&
kv_cache->size(3) == full_dim &&
kv_cache->stride(2) == full_dim &&
kv_cache->stride(1) == block_size * full_dim &&
block_size % 4 == 0,
"nvfp4 kv_cache must be HND [nb, 2*num_kv_heads, block_size, 72] "
"with contiguous pages and block_size % 4 == 0");
for (auto const* scale : {&kv_k_scale, &kv_v_scale}) {
STD_TORCH_CHECK(
scale->has_value() && (*scale)->is_cuda() &&
(*scale)->scalar_type() ==
torch::headeronly::ScalarType::Float &&
(*scale)->numel() >= 1,
"nvfp4 kv_cache requires float32 CUDA kv_k_scale and kv_v_scale");
}
}
kv_s_block = kv_cache->stride(0);
kv_s_head = kv_cache->stride(1);
kv_s_token = kv_cache->stride(2);
@@ -854,10 +1181,31 @@ void fused_minimax_m3_qknorm_rope_kv_insert(
"fp8 index path: index_q_out must be fp8 e4m3");
}
STD_TORCH_CHECK(!nvfp4_kv || insert_kv,
"nvfp4 kv_cache_dtype requires kv_cache (insert mode)");
float const* kv_k_scale_ptr =
nvfp4_kv ? kv_k_scale->const_data_ptr<float>() : nullptr;
float const* kv_v_scale_ptr =
nvfp4_kv ? kv_v_scale->const_data_ptr<float>() : nullptr;
const torch::stable::accelerator::DeviceGuard device_guard(
qkv.get_device_index());
auto stream = get_current_cuda_stream(qkv.get_device_index());
if (nvfp4_kv) {
#ifndef USE_ROCM
VLLM_STABLE_DISPATCH_HALF_TYPES(
qkv.scalar_type(), "fused_minimax_m3_qknorm_rope_kv_insert_nvfp4", [&] {
using st = scalar_t;
CALL_FUSED_MINIMAX_M3_IMPL(uint8_t, vllm::Fp8KVCacheDataType::kAuto,
true);
});
#else
STD_TORCH_CHECK(false, "nvfp4 kv_cache is not supported on ROCm");
#endif
return;
}
VLLM_STABLE_DISPATCH_HALF_TYPES(
qkv.scalar_type(), "fused_minimax_m3_qknorm_rope_kv_insert", [&] {
using st = scalar_t;
@@ -867,3 +1215,4 @@ void fused_minimax_m3_qknorm_rope_kv_insert(
}
#undef CALL_FUSED_MINIMAX_M3
#undef CALL_FUSED_MINIMAX_M3_IMPL
+406 -211
View File
@@ -30,9 +30,7 @@
namespace {
constexpr int kWarpSize = 32;
// Sentinel in the shared top-k scratch: entry already resolved (hit /
// newest / invalid), no miss handling needed.
constexpr int32_t kTokenDone = -1;
// Empty key in the shared open-addressing table of host rows.
constexpr int32_t kHashEmpty = -1;
bool is_pinned_cpu_tensor(const torch::stable::Tensor& tensor) {
@@ -46,10 +44,15 @@ bool is_pinned_cpu_tensor(const torch::stable::Tensor& tensor) {
return attributes.type == cudaMemoryTypeHost;
}
__device__ __forceinline__ int32_t hash_slot(int32_t key, int32_t hash_size) {
// Knuth multiplicative hash for the open-addressing table.
return static_cast<int32_t>((static_cast<uint32_t>(key) * 2654435761u) %
static_cast<uint32_t>(hash_size));
__device__ __forceinline__ int32_t hash_slot(int32_t key, int size) {
// Knuth multiplicative hash scaled onto [0, size) by a multiply-high, so
// the table can take any size without a division.
return static_cast<int32_t>(
__umulhi(static_cast<uint32_t>(key) * 2654435761u, size));
}
__device__ __forceinline__ int next_slot(int h, int size) {
return h + 1 == size ? 0 : h + 1;
}
// Copy one row of `row_bytes` bytes with a single warp. MLA rows take the
@@ -176,16 +179,117 @@ __device__ __forceinline__ void store_hot_index(
}
}
// One block per batch row.
// A union entry's value holds its first (row, top-k) position until phase 2
// or 3 resolves it to a hot slot tagged with how it was resolved.
constexpr int32_t kEntryHit = 1 << 30;
constexpr int32_t kEntryMiss = 1 << 29;
constexpr int32_t kEntryInvalid = 1 << 28;
constexpr int32_t kEntrySlotMask = 0xFFFF;
constexpr int kResidencyCounters = 5;
// Per-request table entries per referenced top-k position (load <= 1/4
// until the hash_size cap).
constexpr int kHashLoadInverse = 4;
// Translate a request-relative top-k position to its resident GPU row (-1 when
// the page is not resident) and returns its host row (-1 when not host-backed).
__device__ __forceinline__ int32_t translate_topk_entry(
const int32_t token_index, const int32_t request_row,
const int32_t* __restrict__ source_block_table,
const int32_t* __restrict__ resident_block_table, const int64_t host_rows,
const int64_t source_bt_stride, const int32_t source_num_reqs,
const int32_t source_num_blocks, const int32_t source_block_size,
const int64_t resident_bt_stride, const int32_t resident_num_reqs,
const int32_t resident_num_blocks, const int32_t resident_block_size,
const int32_t resident_null_block, int32_t& resident_row) {
int32_t g = token_index;
resident_row = -1;
if (source_block_table != nullptr) {
const int32_t source_block =
token_index >= 0 ? token_index / source_block_size : -1;
if (request_row >= 0 && request_row < source_num_reqs &&
source_block >= 0 && source_block < source_num_blocks) {
const int32_t physical_block =
source_block_table[static_cast<int64_t>(request_row) *
source_bt_stride +
source_block];
g = physical_block > 0 ? physical_block * source_block_size +
token_index % source_block_size
: -1;
} else {
g = -1;
}
if (resident_block_table != nullptr) {
const int32_t resident_block =
token_index >= 0 ? token_index / resident_block_size : -1;
if (request_row >= 0 && request_row < resident_num_reqs &&
resident_block >= 0 && resident_block < resident_num_blocks) {
const int32_t physical_block =
resident_block_table[static_cast<int64_t>(request_row) *
resident_bt_stride +
resident_block];
if (physical_block != resident_null_block && physical_block >= 0) {
resident_row = physical_block * resident_block_size +
token_index % resident_block_size;
}
}
}
}
return g >= host_rows ? -1 : g;
}
// Write the hot index of one reference to a resolved union entry and tally it.
__device__ __forceinline__ void write_entry_index(
const int32_t entry, const int row, const int i, const int32_t top_k,
int32_t* __restrict__ hot_indices, int32_t* __restrict__ attention_indices,
const int64_t attention_row_stride, int32_t* __restrict__ miss_mask,
const int32_t* __restrict__ hot_block_table, const int request_row,
const int64_t hot_table_stride, const int32_t hot_block_size,
const int64_t attention_block_stride, const bool first_reference, int& hits,
int& misses) {
int32_t* row_out = hot_indices + static_cast<int64_t>(row) * top_k;
int32_t* row_attention =
attention_indices != nullptr
? attention_indices + static_cast<int64_t>(row) * attention_row_stride
: nullptr;
if (entry & kEntryInvalid) {
store_hot_index(row_out, row_attention, i, -1, hot_block_size,
attention_block_stride);
return;
}
store_hot_index(row_out, row_attention, i,
static_cast<int32_t>(get_physical_hot_row(
hot_block_table, request_row, hot_table_stride,
hot_block_size, entry & kEntrySlotMask)),
hot_block_size, attention_block_stride);
// Only an entry's first reference loads it; later references reuse the
// slot like a hit.
if ((entry & kEntryMiss) && first_reference) {
if (miss_mask != nullptr) {
miss_mask[static_cast<int64_t>(row) * top_k + i] = 1;
}
++misses;
} else {
++hits;
}
}
// One block per request, launched per batch row. A request's rows (several
// with speculative decoding) are contiguous and share its hot-buffer state, so
// the block of its first row resolves the union of their host misses once and
// maps every row onto it; the other blocks exit. The union is bounded by the
// caller's max_union_rows (every decode query's top-k), which
// device_buffer_size covers; a larger union traps. Misses take eviction slots
// in (row, top-k) order of first reference, so the plan is deterministic.
//
// Shared memory layout (int32 region followed by int16 region):
// s_topk[top_k] top-k global ids; reused as miss scratch
// s_chunk_off[nbc + 1] prefix sums for hit (then miss) compaction
// s_hash_keys[hash_size] open addressing: union of the rows' host ids
// s_hash_vals[hash_size] first position, then slot | kEntry* tag
// s_chunk_off[nbc + 1] prefix sums for hit, then miss, compaction
// s_evict_off[nbc + 1] prefix sums for evictable compaction
// s_hash_keys[hash_size] open addressing: global id -> top-k index
// s_counters[3] hits, phase-1 resolved, valid count
// s_counters[5] distinct hits, union size, per-row valid count,
// misses, per-entry hits
// s_done[nbc] positions phase 3 resolved, one word per chunk
// s_lru_out[hot_size] int16, compacted slots: [hits fwd | evict bwd]
// s_hash_vals[hash_size] int16 hash values (top-k index)
// Valid global ids must be unique within each row.
__global__ __launch_bounds__(1024) void hisparse_resolve_residency_kernel(
const int32_t* __restrict__ hot_block_table, // [max_rows, hot_blocks]
@@ -218,11 +322,25 @@ __global__ __launch_bounds__(1024) void hisparse_resolve_residency_kernel(
const int64_t attention_row_stride, const int64_t valid_count_stride) {
const int NUM_WARPS = blockDim.x / kWarpSize;
const int num_buffer_chunks = (hot_size + kWarpSize - 1) / kWarpSize;
const int num_token_chunks = (top_k + kWarpSize - 1) / kWarpSize;
const int batch_row = blockIdx.x;
// The first row of each contiguous run of a request's rows leads it. FULL
// CUDA-graph padding rows map to request 0 after the real rows, forming a
// second run, but carry no valid top-k: that run returns before phase 2
// touches the request's shared state.
const int first_row = blockIdx.x;
const int request_row =
request_ids != nullptr ? request_ids[batch_row] : batch_row;
request_ids != nullptr ? request_ids[first_row] : first_row;
if (request_ids != nullptr && first_row > 0 &&
request_ids[first_row - 1] == request_row) {
return;
}
int end_row = first_row + 1;
if (request_ids != nullptr) {
while (end_row < static_cast<int>(gridDim.x) &&
request_ids[end_row] == request_row) {
++end_row;
}
}
const int state_row =
request_state_indices != nullptr && request_row >= 0 &&
request_row < request_state_count
@@ -230,23 +348,24 @@ __global__ __launch_bounds__(1024) void hisparse_resolve_residency_kernel(
: (request_state_indices == nullptr ? request_row : -1);
// V2 publishes -1 for CUDA-graph padding rows.
if (state_row < 0) {
for (int i = threadIdx.x; i < top_k; i += blockDim.x) {
const int64_t index = static_cast<int64_t>(batch_row) * top_k + i;
hot_indices[index] = -1;
if (attention_indices != nullptr) {
attention_indices[static_cast<int64_t>(batch_row) *
attention_row_stride +
i] = -1;
for (int row = first_row; row < end_row; ++row) {
for (int i = threadIdx.x; i < top_k; i += blockDim.x) {
const int64_t index = static_cast<int64_t>(row) * top_k + i;
hot_indices[index] = -1;
if (attention_indices != nullptr) {
attention_indices[static_cast<int64_t>(row) * attention_row_stride +
i] = -1;
}
if (resolved_global_indices != nullptr) {
resolved_global_indices[index] = -1;
}
}
if (resolved_global_indices != nullptr) {
resolved_global_indices[index] = -1;
if (valid_counts != nullptr && threadIdx.x == 0) {
valid_counts[static_cast<int64_t>(row) * valid_count_stride] = 0;
}
if (swap_counts != nullptr && threadIdx.x == 0) {
swap_counts[row] = 0;
}
}
if (valid_counts != nullptr && threadIdx.x == 0) {
valid_counts[static_cast<int64_t>(batch_row) * valid_count_stride] = 0;
}
if (swap_counts != nullptr && threadIdx.x == 0) {
swap_counts[batch_row] = 0;
}
return;
}
@@ -255,136 +374,130 @@ __global__ __launch_bounds__(1024) void hisparse_resolve_residency_kernel(
const int lane_id = tid % kWarpSize;
const unsigned int lanes_before = ((unsigned int)1 << lane_id) - 1;
const int32_t* row_topk =
global_indices + static_cast<int64_t>(batch_row) * input_row_stride;
int32_t* row_out = hot_indices + static_cast<int64_t>(batch_row) * top_k;
int32_t* row_attention =
attention_indices != nullptr
? attention_indices +
static_cast<int64_t>(batch_row) * attention_row_stride
: nullptr;
int32_t* row_miss = (miss_mask != nullptr)
? miss_mask + static_cast<int64_t>(batch_row) * top_k
: nullptr;
int32_t* row_dgi =
device_global_indices + static_cast<int64_t>(state_row) * region_stride;
int16_t* row_lru = lru_slots + static_cast<int64_t>(state_row) * hot_size;
// A union within max_union_rows (hash_size - 1) and within the rows' top-k
// leaves an empty slot, so every probe sequence ends. Use a slice sized to
// this request's rows so single-row requests initialize a small table.
const int table_size =
min(hash_size, kHashLoadInverse * (end_row - first_row) * top_k);
extern __shared__ char smem_raw[];
int32_t* s_topk = reinterpret_cast<int32_t*>(smem_raw);
int32_t* s_chunk_off = s_topk + top_k;
int32_t* s_hash_keys = reinterpret_cast<int32_t*>(smem_raw);
int32_t* s_hash_vals = s_hash_keys + hash_size;
int32_t* s_chunk_off = s_hash_vals + hash_size;
int32_t* s_evict_off = s_chunk_off + (num_buffer_chunks + 1);
int32_t* s_hash_keys = s_evict_off + (num_buffer_chunks + 1);
int32_t* s_counters = s_hash_keys + hash_size;
int16_t* s_lru_out = reinterpret_cast<int16_t*>(s_counters + 3);
int16_t* s_hash_vals = s_lru_out + hot_size;
int32_t* s_counters = s_evict_off + (num_buffer_chunks + 1);
unsigned int* s_done =
reinterpret_cast<unsigned int*>(s_counters + kResidencyCounters);
int16_t* s_lru_out = reinterpret_cast<int16_t*>(s_done + num_buffer_chunks);
if (tid < 3) {
s_counters[tid] = 0;
}
__syncthreads();
// Phase 1: translate request-relative positions and resolve resident rows.
for (int i = tid; i < top_k; i += blockDim.x) {
const int32_t token_index = row_topk[i];
int32_t g = token_index;
int32_t resident_row = -1;
if (source_block_table != nullptr) {
const int32_t request_id = request_ids[batch_row];
const int32_t source_block =
token_index >= 0 ? token_index / source_block_size : -1;
if (request_id >= 0 && request_id < source_num_reqs &&
source_block >= 0 && source_block < source_num_blocks) {
const int32_t physical_block =
source_block_table[static_cast<int64_t>(request_id) *
source_bt_stride +
source_block];
g = physical_block > 0 ? physical_block * source_block_size +
token_index % source_block_size
: -1;
} else {
g = -1;
}
if (resident_block_table != nullptr) {
const int32_t resident_block =
token_index >= 0 ? token_index / resident_block_size : -1;
if (request_id >= 0 && request_id < resident_num_reqs &&
resident_block >= 0 && resident_block < resident_num_blocks) {
const int32_t physical_block =
resident_block_table[static_cast<int64_t>(request_id) *
resident_bt_stride +
resident_block];
if (physical_block != resident_null_block && physical_block >= 0) {
resident_row = physical_block * resident_block_size +
token_index % resident_block_size;
}
}
}
}
if (g >= host_rows) {
g = -1;
}
if (resolved_global_indices != nullptr) {
resolved_global_indices[static_cast<int64_t>(batch_row) * top_k + i] = g;
}
if (resident_row >= 0 || g >= 0) atomicAdd(&s_counters[2], 1);
if (row_miss != nullptr) row_miss[i] = 0;
if (resident_row >= 0) {
store_hot_index(row_out, row_attention, i, resident_row, hot_block_size,
attention_block_stride);
s_topk[i] = kTokenDone;
atomicAdd(&s_counters[1], 1);
} else if (g < 0) {
store_hot_index(row_out, row_attention, i, -1, hot_block_size,
attention_block_stride);
s_topk[i] = kTokenDone;
atomicAdd(&s_counters[1], 1);
} else {
s_topk[i] = g;
}
}
__syncthreads();
if (valid_counts != nullptr && tid == 0) {
valid_counts[static_cast<int64_t>(batch_row) * valid_count_stride] =
s_counters[2];
}
// Fully resident rows need only request-relative page translation. Avoid
// scanning or rewriting the hot LRU when no selected row can consult it.
if (s_counters[1] == top_k) {
if (tid == 0 && swap_counts != nullptr) {
swap_counts[batch_row] = 0;
}
return;
}
for (int i = tid; i < hash_size; i += blockDim.x) {
for (int i = tid; i < table_size; i += blockDim.x) {
s_hash_keys[i] = kHashEmpty;
s_hash_vals[i] = INT32_MAX;
}
for (int i = tid; i < num_buffer_chunks + 1; i += blockDim.x) {
s_chunk_off[i] = 0;
s_evict_off[i] = 0;
}
__syncthreads();
for (int i = tid; i < top_k; i += blockDim.x) {
const int32_t g = s_topk[i];
if (g == kTokenDone) {
continue;
}
int slot = hash_slot(g, hash_size);
while (true) {
const int32_t old = atomicCAS(&s_hash_keys[slot], kHashEmpty, g);
if (old == kHashEmpty || old == g) {
s_hash_vals[slot] = static_cast<int16_t>(i);
break;
}
slot = (slot + 1) % hash_size;
}
if (tid < kResidencyCounters) {
s_counters[tid] = 0;
}
__syncthreads();
// Phase 1: translate every row's request-relative positions, resolve
// resident rows directly, and collect the union of host rows.
for (int row = first_row; row < end_row; ++row) {
const int32_t* row_topk =
global_indices + static_cast<int64_t>(row) * input_row_stride;
int32_t* row_out = hot_indices + static_cast<int64_t>(row) * top_k;
int32_t* row_attention =
attention_indices != nullptr
? attention_indices +
static_cast<int64_t>(row) * attention_row_stride
: nullptr;
int32_t* row_miss = miss_mask != nullptr
? miss_mask + static_cast<int64_t>(row) * top_k
: nullptr;
for (int i = tid; i < top_k; i += blockDim.x) {
int32_t resident_row;
const int32_t g = translate_topk_entry(
row_topk[i], request_row, source_block_table, resident_block_table,
host_rows, source_bt_stride, source_num_reqs, source_num_blocks,
source_block_size, resident_bt_stride, resident_num_reqs,
resident_num_blocks, resident_block_size, resident_null_block,
resident_row);
if (resolved_global_indices != nullptr) {
resolved_global_indices[static_cast<int64_t>(row) * top_k + i] = g;
}
if (resident_row >= 0 || g >= 0) atomicAdd(&s_counters[2], 1);
if (row_miss != nullptr) row_miss[i] = 0;
if (resident_row >= 0) {
store_hot_index(row_out, row_attention, i, resident_row, hot_block_size,
attention_block_stride);
} else if (g < 0) {
store_hot_index(row_out, row_attention, i, -1, hot_block_size,
attention_block_stride);
} else {
int h = hash_slot(g, table_size);
bool inserted = false;
for (int probe = 0; probe < table_size; ++probe) {
const int32_t old = atomicCAS(&s_hash_keys[h], kHashEmpty, g);
if (old == kHashEmpty) {
atomicAdd(&s_counters[1], 1);
inserted = true;
break;
}
if (old == g) {
inserted = true;
break;
}
h = next_slot(h, table_size);
}
if (inserted) {
atomicMin(&s_hash_vals[h], (row - first_row) * top_k + i);
} else {
// A full table: push the union count past the cap so the check
// below traps.
atomicAdd(&s_counters[1], hash_size);
}
}
}
__syncthreads();
if (tid == 0) {
if (valid_counts != nullptr) {
valid_counts[static_cast<int64_t>(row) * valid_count_stride] =
s_counters[2];
}
s_counters[2] = 0;
}
__syncthreads();
}
// The table and eviction slots are sized for max_union_rows (hash_size - 1);
// a larger union would never reach an empty probe slot or run past s_lru_out.
if (s_counters[1] >= hash_size) {
if (tid == 0) {
printf(
"HiSparse residency: request %d resolves %d host rows, above the "
"configured bound of %d\n",
request_row, s_counters[1], hash_size - 1);
}
__trap();
}
// Fully resident rows need only request-relative page translation. Avoid
// scanning or rewriting the hot LRU when no selected row can consult it.
if (s_counters[1] == 0) {
for (int row = first_row + tid; row < end_row; row += blockDim.x) {
if (swap_counts != nullptr) swap_counts[row] = 0;
}
return;
}
// Phase 2: walk hot slots in LRU order, classify hit / evictable, and
// compact them (hits forward, evictables backward) into s_lru_out.
// compact them (hits forward, evictables backward) into s_lru_out. A hit
// records its slot on the union entry.
const int iters_buffer = (num_buffer_chunks + NUM_WARPS - 1) / NUM_WARPS;
int total_hit_count = 0;
int total_evict_count = 0;
@@ -397,36 +510,28 @@ __global__ __launch_bounds__(1024) void hisparse_resolve_residency_kernel(
const int16_t slot = has_valid_pos ? row_lru[pos] : int16_t(-1);
// Corruption tripwire: lru/dgi are long-lived device state; if some
// external writer (e.g. stray RDMA into reused VRAM) corrupts a slot
// out of [0, region_stride), treat it as no-hit so it degrades to a re-miss
// instead of an unbounded dgi read (phases 3/5 bound the write side).
// out of [0, region_stride), treat it as no-hit so it degrades to a
// re-miss instead of an unbounded dgi read (phase 3 bounds the write
// side).
const int32_t cached_g =
(slot >= 0 && slot < region_stride) ? row_dgi[slot] : -1;
int found_topk_idx = -1;
bool is_hit = false;
if (cached_g >= 0) {
int h = hash_slot(cached_g, hash_size);
int h = hash_slot(cached_g, table_size);
while (true) {
const int32_t k = s_hash_keys[h];
if (k == cached_g) {
found_topk_idx = static_cast<int32_t>(s_hash_vals[h]);
s_hash_vals[h] = kEntryHit | slot;
is_hit = true;
break;
}
if (k == kHashEmpty) break;
h = (h + 1) % hash_size;
h = next_slot(h, table_size);
}
}
const bool is_hit = found_topk_idx >= 0;
const bool is_evictable = has_valid_pos && !is_hit;
if (is_hit) {
s_topk[found_topk_idx] = kTokenDone;
store_hot_index(row_out, row_attention, found_topk_idx,
static_cast<int32_t>(get_physical_hot_row(
hot_block_table, request_row, hot_table_stride,
hot_block_size, slot)),
hot_block_size, attention_block_stride);
}
int local_hit_off = 0;
int local_evict_off = 0;
if (has_valid_chunk) {
@@ -465,39 +570,59 @@ __global__ __launch_bounds__(1024) void hisparse_resolve_residency_kernel(
}
__syncthreads();
// Reset prefix sums for the miss compaction (token chunks <= buffer
// chunks because hot_size >= top_k).
for (int i = tid; i < num_token_chunks + 1; i += blockDim.x) {
for (int i = tid; i < num_buffer_chunks + 1; i += blockDim.x) {
s_chunk_off[i] = 0;
}
__syncthreads();
// Phase 3: compact misses, assign them eviction slots (oldest first) and
// record the new ownership in device_global_indices.
const int iters_token = (num_token_chunks + NUM_WARPS - 1) / NUM_WARPS;
// Phase 3: walk the rows' positions in (row, top-k) order; the first
// reference to a union entry no slot holds is a miss. Assign misses
// eviction slots (oldest first) and record the new ownership in
// device_global_indices. Swaps fill the rows' swap segments in order; the
// union never exceeds their combined capacity, and position chunks never
// exceed buffer chunks because hot_size covers every row's top-k.
const int num_positions = (end_row - first_row) * top_k;
const int num_position_chunks = (num_positions + kWarpSize - 1) / kWarpSize;
const int iters_position = (num_position_chunks + NUM_WARPS - 1) / NUM_WARPS;
int miss_running_total = 0;
for (int iter = 0; iter < iters_token; iter++) {
int entry_hits = 0;
int entry_misses = 0;
for (int iter = 0; iter < iters_position; iter++) {
const int chunk_idx = warp_id + iter * NUM_WARPS;
const bool has_valid_chunk = chunk_idx < num_token_chunks;
const bool has_valid_chunk = chunk_idx < num_position_chunks;
const int i = chunk_idx * kWarpSize + lane_id;
const bool has_valid_token = has_valid_chunk && (i < top_k);
int32_t g = 0;
const int position = chunk_idx * kWarpSize + lane_id;
int32_t g = -1;
int h = -1;
int32_t entry = 0;
bool is_miss = false;
if (has_valid_token) {
is_miss = s_topk[i] != kTokenDone;
if (is_miss) {
g = s_topk[i];
if (has_valid_chunk && position < num_positions) {
const int row = first_row + position / top_k;
int32_t resident_row;
g = translate_topk_entry(
global_indices[static_cast<int64_t>(row) * input_row_stride +
position % top_k],
request_row, source_block_table, resident_block_table, host_rows,
source_bt_stride, source_num_reqs, source_num_blocks,
source_block_size, resident_bt_stride, resident_num_reqs,
resident_num_blocks, resident_block_size, resident_null_block,
resident_row);
if (resident_row < 0 && g >= 0) {
h = hash_slot(g, table_size);
while (s_hash_keys[h] != g) {
h = next_slot(h, table_size);
}
entry = s_hash_vals[h];
is_miss = entry == position;
}
}
int local_miss_off = 0;
if (has_valid_chunk) {
const unsigned int miss_mask = __ballot_sync(0xFFFFFFFF, is_miss);
local_miss_off = __popc(miss_mask & lanes_before);
const unsigned int miss_mask_bits = __ballot_sync(0xFFFFFFFF, is_miss);
local_miss_off = __popc(miss_mask_bits & lanes_before);
if (lane_id == 0) {
s_chunk_off[chunk_idx + 1] = __popc(miss_mask);
s_chunk_off[chunk_idx + 1] = __popc(miss_mask_bits);
}
}
__syncthreads();
@@ -505,60 +630,112 @@ __global__ __launch_bounds__(1024) void hisparse_resolve_residency_kernel(
if (warp_id == 0) {
miss_running_total =
warp_inclusive_scan(s_chunk_off, lane_id, chunk_idx + 1,
num_token_chunks + 1, miss_running_total);
num_position_chunks + 1, miss_running_total);
if (tid == 0) {
s_counters[3] = miss_running_total;
}
}
__syncthreads();
if (is_miss) {
const int m = s_chunk_off[chunk_idx] + local_miss_off;
const int16_t evict_slot = s_lru_out[hot_size - 1 - m];
const int64_t compact_index = static_cast<int64_t>(first_row) * top_k + m;
if (evict_slot < 0 || evict_slot >= region_stride) {
// Corruption tripwire: an out-of-range slot (corrupted lru state,
// see phase 2) must not become a hot-cache/dgi write. Resolve the
// entry as invalid (-1, masked by attention, not in the miss set);
// it re-misses on a later step. Phase 5 re-checks the same
// s_lru_out value, so its copy is skipped consistently.
store_hot_index(row_out, row_attention, i, -1, hot_block_size,
attention_block_stride);
// entry as invalid (-1, masked by attention); it re-misses on a later
// step, and its copy is skipped.
s_hash_vals[h] = kEntryInvalid;
if (swap_host_physical_rows != nullptr) {
const int64_t compact_index =
static_cast<int64_t>(batch_row) * top_k + m;
swap_host_physical_rows[compact_index] = g;
swap_device_physical_rows[compact_index] = -1;
}
} else {
// Reuse s_topk as compacted miss scratch: m < i always holds (done
// entries are skipped), so writes never overrun pending reads.
s_topk[m] = g;
const int32_t physical_row = static_cast<int32_t>(
get_physical_hot_row(hot_block_table, request_row, hot_table_stride,
hot_block_size, evict_slot));
store_hot_index(row_out, row_attention, i, physical_row, hot_block_size,
attention_block_stride);
s_hash_vals[h] = kEntryMiss | evict_slot;
if (swap_host_physical_rows != nullptr) {
const int64_t compact_index =
static_cast<int64_t>(batch_row) * top_k + m;
swap_host_physical_rows[compact_index] = g;
swap_device_physical_rows[compact_index] = physical_row;
swap_device_physical_rows[compact_index] =
static_cast<int32_t>(get_physical_hot_row(
hot_block_table, request_row, hot_table_stride,
hot_block_size, evict_slot));
}
if (row_miss != nullptr) row_miss[i] = 1;
row_dgi[evict_slot] = g;
}
entry = s_hash_vals[h];
}
// Hits, first references and references to entries resolved in an
// earlier chunk write their hot index now; later references to an entry
// resolved in this same chunk pass (only across rows) are left to phase 4.
const bool resolved =
h >= 0 && (entry & (kEntryHit | kEntryMiss | kEntryInvalid));
if (resolved) {
write_entry_index(
entry, first_row + position / top_k, position % top_k, top_k,
hot_indices, attention_indices, attention_row_stride, miss_mask,
hot_block_table, request_row, hot_table_stride, hot_block_size,
attention_block_stride, is_miss, entry_hits, entry_misses);
}
if (has_valid_chunk) {
const unsigned int done_bits = __ballot_sync(0xFFFFFFFF, resolved);
if (lane_id == 0) {
s_done[chunk_idx] = done_bits;
}
}
}
__syncthreads();
const int total_hits = s_counters[0];
const int total_misses = top_k - total_hits - s_counters[1];
if (swap_counts != nullptr && tid == 0) {
swap_counts[batch_row] = total_misses;
}
if (stats != nullptr && tid == 0) {
atomicAdd(&stats[0], static_cast<unsigned long long>(total_hits));
atomicAdd(&stats[1], static_cast<unsigned long long>(total_misses));
const int total_misses = s_counters[3];
if (swap_counts != nullptr) {
for (int row = first_row + tid; row < end_row; row += blockDim.x) {
const int remaining = total_misses - (row - first_row) * top_k;
swap_counts[row] = max(0, min(top_k, remaining));
}
}
// Phase 4: write back the LRU order: stale evictables at the front,
// Phase 4: references phase 3 left unresolved (later references across rows
// to entries loaded in the same chunk pass) map onto the union's slots.
// Single-row requests resolve everything in phase 3.
if (end_row - first_row > 1) {
for (int position = tid; position < num_positions; position += blockDim.x) {
if (s_done[position / kWarpSize] & (1u << (position % kWarpSize))) {
continue;
}
const int row = first_row + position / top_k;
int32_t resident_row;
const int32_t g = translate_topk_entry(
global_indices[static_cast<int64_t>(row) * input_row_stride +
position % top_k],
request_row, source_block_table, resident_block_table, host_rows,
source_bt_stride, source_num_reqs, source_num_blocks,
source_block_size, resident_bt_stride, resident_num_reqs,
resident_num_blocks, resident_block_size, resident_null_block,
resident_row);
if (resident_row >= 0 || g < 0) continue;
int h = hash_slot(g, table_size);
while (s_hash_keys[h] != g) {
h = next_slot(h, table_size);
}
write_entry_index(
s_hash_vals[h], row, position % top_k, top_k, hot_indices,
attention_indices, attention_row_stride, miss_mask, hot_block_table,
request_row, hot_table_stride, hot_block_size, attention_block_stride,
false, entry_hits, entry_misses);
}
}
if (stats != nullptr) {
atomicAdd(&s_counters[4], entry_hits);
atomicAdd(&s_counters[2], entry_misses);
__syncthreads();
if (tid == 0) {
atomicAdd(&stats[0], static_cast<unsigned long long>(s_counters[4]));
atomicAdd(&stats[1], static_cast<unsigned long long>(s_counters[2]));
}
}
// Phase 5: write back the LRU order: stale evictables at the front,
// freshly loaded misses next, then hits at MRU.
const int total_evictable = hot_size - total_hits;
const int remaining_evictable = total_evictable - total_misses;
@@ -783,7 +960,7 @@ void hisparse_resolve_residency(
torch::stable::Tensor& device_global_indices,
torch::stable::Tensor& lru_slots,
std::optional<torch::stable::Tensor> const& request_state_indices,
int64_t region_stride,
int64_t region_stride, int64_t max_union_rows,
std::optional<torch::stable::Tensor> const& miss_mask,
std::optional<torch::stable::Tensor> const& stats,
std::optional<torch::stable::Tensor> const& attention_indices,
@@ -1034,14 +1211,32 @@ void hisparse_resolve_residency(
}
constexpr int kBlockSize = 1024;
const int hash_size = 2 * top_k;
STD_TORCH_CHECK(max_union_rows > 0 && max_union_rows <= hot_size,
"max_union_rows must be in [1, hot_size], got ",
max_union_rows);
// One spare entry above the union bound keeps every probe sequence finite.
const int hash_size = static_cast<int>(max_union_rows) + 1;
const int num_buffer_chunks = (hot_size + kWarpSize - 1) / kWarpSize;
const size_t smem_bytes =
sizeof(int32_t) * (top_k + 2 * (num_buffer_chunks + 1) + hash_size + 3) +
sizeof(int16_t) * (hot_size + hash_size);
sizeof(int32_t) * (2 * hash_size + 2 * (num_buffer_chunks + 1) +
kResidencyCounters + num_buffer_chunks) +
sizeof(int16_t) * hot_size;
const torch::stable::accelerator::DeviceGuard device_guard(
hot_cache.get_device_index());
int max_smem_bytes = 0;
STD_TORCH_CHECK(cudaDeviceGetAttribute(
&max_smem_bytes, cudaDevAttrMaxSharedMemoryPerBlockOptin,
hot_cache.get_device_index()) == cudaSuccess,
"failed to query shared memory per block");
STD_TORCH_CHECK(smem_bytes <= static_cast<size_t>(max_smem_bytes),
"HiSparse device_buffer_size=", hot_size, " needs ",
smem_bytes,
" bytes of shared memory to resolve residency, "
"but the device allows ",
max_smem_bytes,
"; lower device_buffer_size or the number of speculative "
"tokens.");
const cudaStream_t stream = get_current_cuda_stream();
const int64_t attention_row_stride =
attention_indices.has_value() ? attention_indices.value().stride(0) : 0;
+4 -2
View File
@@ -381,7 +381,9 @@ void fused_minimax_m3_qknorm_rope_kv_insert(
std::optional<torch::stable::Tensor> q_out,
std::optional<torch::stable::Tensor> index_q_out,
const std::string& kv_cache_dtype, bool skip_index_branch,
std::optional<torch::stable::Tensor> q_fp8_out, double q_fp8_scale);
std::optional<torch::stable::Tensor> q_fp8_out, double q_fp8_scale,
std::optional<torch::stable::Tensor> kv_k_scale,
std::optional<torch::stable::Tensor> kv_v_scale);
#ifdef VLLM_ENABLE_FUSED_KDA_DECODE
void fused_kda_decode(
@@ -680,7 +682,7 @@ void hisparse_resolve_residency(
torch::stable::Tensor& device_global_indices,
torch::stable::Tensor& lru_slots,
std::optional<torch::stable::Tensor> const& request_state_indices,
int64_t region_stride,
int64_t region_stride, int64_t max_union_rows,
std::optional<torch::stable::Tensor> const& miss_mask,
std::optional<torch::stable::Tensor> const& stats,
std::optional<torch::stable::Tensor> const& attention_indices,
+3 -1
View File
@@ -513,7 +513,8 @@ STABLE_TORCH_LIBRARY_FRAGMENT(_C, ops) {
"Tensor!? kv_cache, Tensor!? index_cache, "
"int block_size, Tensor!? q_out, Tensor!? index_q_out, "
"str kv_cache_dtype, bool skip_index_branch=False, "
"Tensor!? q_fp8_out=None, float q_fp8_scale=1.0) -> ()");
"Tensor!? q_fp8_out=None, float q_fp8_scale=1.0, "
"Tensor? kv_k_scale=None, Tensor? kv_v_scale=None) -> ()");
#ifdef VLLM_ENABLE_FUSED_KDA_DECODE
ops.def(
@@ -995,6 +996,7 @@ STABLE_TORCH_LIBRARY_FRAGMENT(_C_cache_ops, ops) {
" Tensor! lru_slots,"
" Tensor? request_state_indices,"
" int region_stride,"
" int max_union_rows,"
" Tensor(a!)? miss_mask=None,"
" Tensor(b!)? stats=None,"
" Tensor(c!)? attention_indices=None,"
+2 -1
View File
@@ -906,6 +906,7 @@ ENV UV_INDEX_STRATEGY="unsafe-best-match"
ENV UV_LINK_MODE=copy
ENV UV_PYTHON_INSTALL_DIR=/opt/uv/python
ENV UV_CACHE_DIR=/opt/uv/cache
ENV UV_COMPILE_BYTECODE=1
RUN mkdir -p "${UV_PYTHON_INSTALL_DIR}" "${UV_CACHE_DIR}" \
&& chgrp -R 0 /opt/uv \
&& chmod -R g+rwX,a+rX /opt/uv
@@ -1004,7 +1005,7 @@ RUN --mount=type=cache,target=/opt/uv/cache \
# Install FlashInfer JIT cache (requires CUDA-version-specific index URL)
# https://docs.flashinfer.ai/installation.html
# From versions.json: .flashinfer.version
ARG FLASHINFER_VERSION=0.7.0
ARG FLASHINFER_VERSION=0.7.0.post1
RUN --mount=type=cache,target=/opt/uv/cache \
if [ "${INSTALL_RUBIN_PRERELEASE}" != "true" ]; then \
uv pip install --system flashinfer-jit-cache==${FLASHINFER_VERSION} \
+2
View File
@@ -138,6 +138,8 @@ ENV TARGETARCH=${TARGETARCH}
######################### COMMON BASE IMAGE + PYTHON DEPS #########################
FROM base-common AS base-common-pydeps
ENV UV_COMPILE_BYTECODE=1
# PyTorch provides its own indexes for standard and nightly builds.
# PYTORCH_NIGHTLY=1 builds the CPU image against torch nightly (parity with the
# GPU torch-nightly lane); default 0 keeps the pinned release CPU build.
+1
View File
@@ -22,6 +22,7 @@ ENV HOME=/root \
RUSTUP_HOME=/root/.rustup \
UV_CACHE_DIR=$HOME/.cache/uv \
PATH=/root/.cargo/bin:/root/.rustup/bin:${VIRTUAL_ENV}/bin:$PATH
ENV UV_COMPILE_BYTECODE=1
RUN echo "DEBUG: VLLM_VERSION=${VLLM_VERSION}"
RUN --mount=type=cache,target=/var/cache/dnf \
+1
View File
@@ -898,6 +898,7 @@ COPY --from=test_smoke /vllm-smoke-ok /
# -----------------------
# Final vLLM image
FROM mori_base AS final_common
ENV UV_COMPILE_BYTECODE=1
RUN python3 -m pip install --upgrade pip && rm -rf /var/lib/apt/lists/*
+6 -5
View File
@@ -3,14 +3,14 @@ ARG PYTHON_VERSION=3.12
ARG TORCH_VERSION=2.12.0+rocm10.0.0
ARG TORCHVISION_VERSION=0.27.0+rocm10.0.0
ARG TORCHAUDIO_VERSION=2.11.0+rocm10.0.0
ARG TRITON_BRANCH="c0a142ffcbb42f9a67a56bf791cad472be1d6425" # release/internal/3.8.x as of 9/21/2026
ARG TRITON_VERSION=3.8.0+gitc0a142ff
ARG TRITON_BRANCH="669b31ac" # release/internal/3.8.x as of 09/29/2026
ARG TRITON_VERSION=3.8.0+git669b31ac
ARG TRITON_REPO="https://github.com/ROCm/triton.git"
ARG FA_BRANCH="0e60e394"
ARG FA_REPO="https://github.com/Dao-AILab/flash-attention.git"
ARG AITER_BRANCH="v0.1.23"
ARG AITER_REPO="https://github.com/ROCm/aiter.git"
ARG MORI_BRANCH="f95cd5e0ca7b730f92f9ffdfb358484d6965d9f7" # TheRock compatible v1.1.0
ARG MORI_BRANCH="v1.2.3.post1"
ARG MORI_REPO="https://github.com/ROCm/mori.git"
ARG ROCM_SYSTEMS_REPO="https://github.com/ROCm/rocm-systems.git"
ARG ROCM_RUNTIME_COMMIT="25e14349f2606a6b95c4ccd45eb92a1835646cf2"
@@ -207,8 +207,9 @@ RUN cd mori \
&& git checkout ${MORI_BRANCH} \
&& git submodule update --init --recursive \
&& ln -sfn ${ROCM_PATH} /opt/rocm \
&& mkdir -p /usr/lib64 && ln -sf /usr/lib/x86_64-linux-gnu/libc.so /usr/lib64/libc.so \
&& python3 setup.py bdist_wheel --dist-dir=dist && ls /app/mori/dist/*.whl
&& pip install "setuptools_scm>=6.2" \
&& CMAKE_PREFIX_PATH="${ROCM_PATH}:${ROCM_PATH}/lib/rocm_sysdeps" \
python3 setup.py bdist_wheel --dist-dir=dist && ls /app/mori/dist/*.whl
RUN mkdir -p /app/install && cp /app/mori/dist/*.whl /app/install
###
+1
View File
@@ -920,6 +920,7 @@ COPY --from=test_smoke /vllm-smoke-ok /
# -----------------------
# Final vLLM image
FROM mori_base AS final_common
ENV UV_COMPILE_BYTECODE=1
RUN python3 -m pip install --upgrade pip && rm -rf /var/lib/apt/lists/*
+1 -1
View File
@@ -11,7 +11,7 @@ ARG FA_BRANCH="0e60e394"
ARG FA_REPO="https://github.com/Dao-AILab/flash-attention.git"
ARG AITER_BRANCH="v0.1.23"
ARG AITER_REPO="https://github.com/ROCm/aiter.git"
ARG MORI_BRANCH="v1.1.0"
ARG MORI_BRANCH="v1.2.3.post1"
ARG MORI_REPO="https://github.com/ROCm/mori.git"
ARG ROCM_SYSTEMS_REPO="https://github.com/ROCm/rocm-systems.git"
ARG ROCPROFILER_SDK_COMMIT="2b22ab0195cc1461cd9abf3b969e9dd7c10af350"
+1
View File
@@ -655,6 +655,7 @@ RUN mkdir src && mv vllm src/vllm
# -----------------------
# Final vLLM image
FROM mori_base AS final
ENV UV_COMPILE_BYTECODE=1
RUN python3 -m pip install --upgrade pip && rm -rf /var/lib/apt/lists/*
+1
View File
@@ -196,6 +196,7 @@ ARG PIP_EXTRA_INDEX_URL="https://download.pytorch.org/whl/cpu"
ENV LD_LIBRARY_PATH="/opt/vllm/lib64/python${PYTHON_VERSION}/site-packages/torch/lib:/usr/local/lib:/opt/rh/gcc-toolset-14/root/usr/lib64:$LD_LIBRARY_PATH"
ENV C_INCLUDE_PATH="/usr/local/include:$C_INCLUDE_PATH"
ENV UV_LINK_MODE=copy
ENV UV_COMPILE_BYTECODE=1
ENV CARGO_HOME=/root/.cargo
ENV RUSTUP_HOME=/root/.rustup
ENV GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1
+1
View File
@@ -205,6 +205,7 @@ RUN --mount=type=cache,target=/root/.cache/uv \
/tmp/ucx_source /tmp/nixl_source
FROM vllm-base AS vllm-openai
ENV UV_COMPILE_BYTECODE=1
# Must match the ucx-nixl-build stage above: ARG does not cross stages, so a stale
# value here installs a meta package that disagrees with the wheel built there.
+1 -1
View File
@@ -71,7 +71,7 @@
"default": "true"
},
"FLASHINFER_VERSION": {
"default": "0.7.0"
"default": "0.7.0.post1"
},
"GDRCOPY_CUDA_VERSION": {
"default": "12.8"
+6
View File
@@ -88,6 +88,12 @@ vllm bench sweep serve \
By default, each parameter combination is benchmarked 3 times to make the results more reliable. You can adjust the number of runs by setting `--num-runs`.
You can optionally run an unmeasured warmup before each parameter combination by setting `--warmup-num-prompts`. The warmup result is saved as `warmup.json` for inspection, but is excluded from the measured summary.
You can use `--continue-on-error` to record a failed warmup, benchmark run, or server configuration and continue processing the remaining sweep combinations. Failure details are saved in `.failure.json` files.
Both options are also supported by `vllm bench sweep serve_workload`.
!!! important
If both `--serve-params` and `--bench-params` are passed, the script will iterate over the Cartesian product between them.
You can use `--dry-run` to preview the commands to be run.
+3
View File
@@ -135,6 +135,9 @@ API_AUTONAV_EXCLUDE=vllm mkdocs serve # API ref off (~15 seconds)
Once you see a `Serving on http://<address>:<port>/` line in the logs, the live preview is ready!
Open that address in your browser to see it — `http://127.0.0.1:8000/` by default, or whichever host/port you passed to `-a`.
On pull requests, Read the Docs builds a documentation preview only if the PR changes files that affect the docs (e.g. `docs/`, `examples/` or `mkdocs.yaml`) and the `pre-commit` checks have not failed.
Changes to `vllm/` alone do not trigger a build, so if your PR changes the API or CLI reference, add the `build-docs` label to force one.
For additional features and advanced configurations, refer to the:
- [MkDocs documentation](https://www.mkdocs.org/)
+2
View File
@@ -85,6 +85,8 @@ def forward(
For reference, check out our [Llama implementation](../../../vllm/model_executor/models/llama.py). vLLM already supports a large number of models. It is recommended to find a model similar to yours and adapt it to your model's architecture. Check out [vllm/model_executor/models](../../../vllm/model_executor/models) for more examples.
Adapting model-specific structure or boilerplate can keep architectures independent. Reuse shared layer implementations, such as Attention and MoE, where their contracts fit the model.
## 3. (Optional) Implement tensor parallelism and quantization support
If your model is too large to fit into a single GPU, you can use tensor parallelism to manage it.
+7 -4
View File
@@ -130,10 +130,13 @@ path. The resolver consumes the existing graph-stable request mapping from
attention metadata; neither the worker nor individual cache handles keep a
duplicate mapping.
Speculative decoding resolves and consumes each verification step in order.
Each step receives distinct replayable plan rows while sharing the request's
hot-cache state, so a later step cannot reuse a hot row before an earlier step
has consumed it.
Speculative decoding resolves all verification rows of a request in one pass:
one block resolves the union of the rows' top-k against the request's hot-cache
state, so rows that select the same host row share its hot row and no row
evicts a hot row another row of the step still reads. Draft layers write their
rows after the target forward, so their host mirror runs at the start of the
next step, after the drafter, and the step's page transfers are submitted
behind it.
## P/D import target
+17 -11
View File
@@ -78,17 +78,23 @@ GPU-side fusion is controlled by `multimodal_config.mm_device_do_normalize`:
- The flag is **enabled by default** for all models that support it.
- Currently, it’s on by default for these architectures:
| name | Architecture | Example HF Models |
|--------------------|------------------------------------------|----------------------------------------|
| `intern-s1-pro` | `InternS1ProForConditionalGeneration` | `internlm/Intern-S1-Pro` |
| `intern-s2-mobius` | `InternS2MobiusForConditionalGeneration` | `internlm/Intern-S2-Mobius` |
| `qwen2-vl` | `Qwen2VLForConditionalGeneration` | `Qwen/Qwen2-VL-2B-Instruct`, etc. |
| `qwen2.5-vl` | `Qwen2_5_VLForConditionalGeneration` | `Qwen/Qwen2.5-VL-3B-Instruct`, etc. |
| `qwen3-vl` | `Qwen3VLForConditionalGeneration` | `Qwen/Qwen3-VL-4B-Instruct`, etc. |
| `qwen3-vl-moe` | `Qwen3VLMoeForConditionalGeneration` | `Qwen/Qwen3-VL-30B-A3B-Instruct`, etc. |
| `qwen3.5` | `Qwen3_5ForConditionalGeneration` | `Qwen/Qwen3.5-0.8B`, etc. |
| `qwen3.5-moe` | `Qwen3_5MoeForConditionalGeneration` | `Qwen/Qwen3.5-35B-A3B`, etc. |
| `qwen4-exp` | `Qwen4ExpForConditionalGeneration` | `Qwen/Qwen3.8-Flash-Next` |
| name | Architecture | Example HF Models |
| -------------------------- | ------------------------------------------ | ----------------------------------------------------- |
| `glm-4v` | `Glm4vForConditionalGeneration` | `zai-org/GLM-4.1V-9B-Thinking`, etc. |
| `glm-5-next` | `Glm5NextForConditionalGeneration` | `zai-org/GLM-5.3-Flash` |
| `glm-ocr` | `GlmOcrForConditionalGeneration` | `zai-org/GLM-OCR` |
| `intern-s1-pro` | `InternS1ProForConditionalGeneration` | `internlm/Intern-S1-Pro` |
| `intern-s2-mobius` | `InternS2MobiusForConditionalGeneration` | `internlm/Intern-S2-Mobius` |
| `llama-nemotron-vl-embed` | `LlamaNemotronVLForEmbedding` | `nvidia/llama-nemotron-embed-vl-1b-v2` |
| `llama-nemotron-vl-rerank` | `LlamaNemotronVLForSequenceClassification` | `nvidia/llama-nemotron-rerank-vl-1b-v2` |
| `mistral3` | `Mistral3ForConditionalGeneration` | `mistralai/Mistral-Small-3.2-24B-Instruct-2506`, etc. |
| `qwen2-vl` | `Qwen2VLForConditionalGeneration` | `Qwen/Qwen2-VL-2B-Instruct`, etc. |
| `qwen2.5-vl` | `Qwen2_5_VLForConditionalGeneration` | `Qwen/Qwen2.5-VL-3B-Instruct`, etc. |
| `qwen3-vl` | `Qwen3VLForConditionalGeneration` | `Qwen/Qwen3-VL-4B-Instruct`, etc. |
| `qwen3-vl-moe` | `Qwen3VLMoeForConditionalGeneration` | `Qwen/Qwen3-VL-30B-A3B-Instruct`, etc. |
| `qwen3.5` | `Qwen3_5ForConditionalGeneration` | `Qwen/Qwen3.5-0.8B`, etc. |
| `qwen3.5-moe` | `Qwen3_5MoeForConditionalGeneration` | `Qwen/Qwen3.5-35B-A3B`, etc. |
| `qwen4-exp` | `Qwen4ExpForConditionalGeneration` | `Qwen/Qwen3.8-Flash-Next` |
#### Key Properties and Gains
+9 -3
View File
@@ -127,7 +127,7 @@ Batch invariance has been tested and verified on the following models:
- **Qwen3-VL (Vision-Language)**: `Qwen/Qwen3-VL-2B-Instruct`, `Qwen/Qwen3-VL-4B-Instruct` (single image and video inputs)
- **Qwen3 (MoE)**: `Qwen/Qwen3-30B-A3B`, `Qwen/Qwen3-Next-80B-A3B-Instruct`, `Qwen/Qwen3-30B-A3B-Thinking-2507-FP8`
- **Qwen2.5**: `Qwen/Qwen2.5-0.5B-Instruct`, `Qwen/Qwen2.5-1.5B-Instruct`, `Qwen/Qwen2.5-3B-Instruct`, `Qwen/Qwen2.5-7B-Instruct`, `Qwen/Qwen2.5-14B-Instruct`, `Qwen/Qwen2.5-32B-Instruct`
- **Llama 3**: Llama3.1 and 3.2 series, `meta-llama/Llama-3.2-3B-Instruct` for example
- **Llama 3**: Llama3.1 and 3.2 series, `meta-llama/Llama-3.2-1B-Instruct`, `meta-llama/Llama-3.2-3B-Instruct` for example
- **GPT-OSS**: `openai/gpt-oss-20b`, `openai/gpt-oss-120b`
- **Mistral**: `mistralai/Mistral-7B-v0.3`
- **Phi series**: `microsoft/Phi-3.5-mini-instruct`
@@ -135,6 +135,8 @@ Batch invariance has been tested and verified on the following models:
- **Granite 3.1 (Dense)**: `ibm-granite/granite-3.1-2b-instruct`, `ibm-granite/granite-3.1-8b-instruct`
- **EXAONE 4.0 series**: `LGAI-EXAONE/EXAONE-4.0-1.2B`, `LGAI-EXAONE/EXAONE-4.0.1-32B`, `LGAI-EXAONE/EXAONE-4.0-32B`
- **OLMo 2**: `allenai/OLMo-2-0425-1B-Instruct`
- **SmolLM2**: `HuggingFaceTB/SmolLM2-1.7B-Instruct`
- **PLaMo3**: `pfnet/plamo-3-nict-2b-base`
Other models may also work, but these have been explicitly validated. If you encounter issues with a specific model, please report them on the [GitHub issue tracker](https://github.com/vllm-project/vllm/issues/new/choose).
@@ -144,11 +146,15 @@ When batch invariance is enabled, vLLM:
1. Uses deterministic kernel implementations for attention and other operations
2. Ensures consistent numerical behavior across different batch sizes
3. Disables certain optimizations that may introduce non-determinism (such as custom all-reduce operations in tensor parallel mode, and sequence parallelism / async TP, whose reduce-scatter path is not batch-invariant)
4. On CUDA devices with tuned matmul table entries for the model's bf16 unquantized forward linear layers (Ada, Hopper, Blackwell),
3. Disables certain optimizations that may introduce non-determinism (such as sequence parallelism / async TP, whose reduce-scatter path is not batch-invariant)
4. Under tensor parallelism, keeps custom all-reduce on with a fixed reduction order (the 1-stage kernel is pinned, and large inputs are reduced in fixed-size chunks), and disables FlashInfer, AITER and QuickReduce all-reduce
5. On CUDA devices with tuned matmul table entries for the model's bf16 unquantized forward linear layers (Ada, Hopper, Blackwell),
runs without `torch.compile` using breakable CUDA graphs so tile configs follow the runtime batch size; set `VLLM_USE_BREAKABLE_CUDAGRAPH=0` to opt out
(not applied when sequence parallelism / async TP are enabled, since those are `torch.compile` passes).
!!! warning
Batch invariance under tensor parallelism is not yet supported for all-reduces whose size isn't a multiple of 16 bytes, for example a hidden size that isn't a multiple of 8 in fp16/bf16. Such tensors can switch between NCCL and custom all-reduce depending on batch size. All validated models meet this requirement.
!!! note
Enabling batch invariance may impact performance compared to the default non-deterministic mode. This trade-off is intentional to guarantee reproducibility.
+4 -3
View File
@@ -45,9 +45,10 @@ The following scripts demonstrate the workflow with **ExampleConnector**:
### ECMooncakeConnector
**ECMooncakeConnector** transfers encoder outputs using the Mooncake TransferEngine.
See the [Mooncake integration example](../../tests/v1/ec_connector/integration/run_epd_mooncake_ec_full_pipeline.sh)
for a complete 1 Encoder instance + 1 PD instance setup, including the producer and
consumer `--ec-transfer-config` settings and proxy configuration.
See the [ECMooncakeConnector Usage Guide](mooncake_ec_connector_usage.md) for
producer/consumer `--ec-transfer-config`, proxy `--ec-consumer-zmq-addrs`, and
transport options. The [Mooncake integration example](../../tests/v1/ec_connector/integration/run_epd_mooncake_ec_full_pipeline.sh)
is a complete 1 Encoder instance + 1 PD instance setup.
With vLLM and Mooncake installed, run the example from the repository root:
+1 -1
View File
@@ -27,7 +27,7 @@ Now supports 9 types of connectors:
--kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_both", "kv_buffer_device":"cuda", "kv_connector_extra_config":{"backends":["UCX", "GDS"]}}'
```
- **MooncakeConnector**: refer to [examples/disaggregated/mooncake_connector/run_mooncake_connector.sh](../../examples/disaggregated/mooncake_connector/run_mooncake_connector.sh) for the example usage of MooncakeConnector disaggregated prefilling. For detailed usage guide, see [MooncakeConnector Usage Guide](mooncake_connector_usage.md).
- **MooncakeConnector**: refer to [examples/disaggregated/mooncake_connector/run_mooncake_connector.sh](../../examples/disaggregated/mooncake_connector/run_mooncake_connector.sh) for the example usage of MooncakeConnector disaggregated prefilling. For detailed usage guide, see [MooncakeConnector Usage Guide](mooncake_connector_usage.md). For multimodal encoder-cache transfer, see [ECMooncakeConnector Usage Guide](mooncake_ec_connector_usage.md).
- **MoRIIOConnector** (ROCm only): see [MoRI-IO Usage Guide](moriio_connector_usage.md) for example usage and detailed documentation.
- **MultiConnector**: take advantage of the kv_connector_extra_config: dict[str, Any] already present in KVTransferConfig to stash all the connectors we want in an ordered list of kwargs.such as:
+4 -4
View File
@@ -42,10 +42,10 @@ Snapshots currently require:
and CRIU still reopens it by path, so replacing that file permanently
invalidates the artifact without an early error.
The official `vllm/vllm-openai` Linux x86-64 image includes the snapshot
runtime. It still requires a compatible host driver, kernel, and privileges.
Arm64 images omit it. Source installs must set `CRIU_CUDA_PLUGIN_DIR` to the
directory containing `cuda_plugin.so`.
The official CUDA 13 `vllm/vllm-openai` Linux x86-64 images include the
snapshot runtime. CUDA 12.x and Arm64 images omit it. A compatible host
driver, kernel, and privileges are still required. Source installs must set
`CRIU_CUDA_PLUGIN_DIR` to the directory containing `cuda_plugin.so`.
Run snapshot commands with `docker exec` inside a long-lived container. Restore
hands the API server off as a detached process, so a one-shot container would
@@ -38,6 +38,9 @@ python examples/disaggregated/mooncake_connector/mooncake_connector_proxy.py --p
Now you can send requests to the proxy server through port 8000.
For multimodal Encode-Prefill-Decode (encoder embeddings rather than KV
cache), see [ECMooncakeConnector Usage Guide](mooncake_ec_connector_usage.md).
## Environment Variables
- `VLLM_MOONCAKE_BOOTSTRAP_PORT`: Port for Mooncake bootstrap server
@@ -76,6 +79,7 @@ Now you can send requests to the proxy server through port 8000.
- **num_workers**: Size of thread pool for one prefiller worker to transfer KV caches by mooncake. (default 10)
- **mooncake_protocol**: Mooncake connector protocol. (default "rdma")
- **device_name**: Comma-separated whitelist of RDMA devices (e.g. `"mlx5_0,mlx5_1"`) to restrict topology discovery to. Empty discovers every device. Useful on hosts exposing a mix of InfiniBand and RoCE ports, where both peers must settle on the same link layer.
- **bootstrap_server_address**: Optional `"host:port"` of an already running Mooncake bootstrap server. All prefiller ranks register with this endpoint instead of launching an embedded server. The caller owns the server lifetime and must configure the proxy to use the same endpoint. The example proxy assumes the bootstrap server is on the prefiller API host; a different host requires a proxy that supplies that address in `remote_bootstrap_addr`. Omit this option to use the embedded server and `VLLM_MOONCAKE_BOOTSTRAP_PORT`.
## Example Scripts/Code
@@ -0,0 +1,158 @@
# ECMooncakeConnector Usage Guide
`ECMooncakeConnector` moves multimodal **encoder embeddings** (encoder cache)
between a disaggregated encoder instance and a language-model instance over
[Mooncake Transfer Engine](https://github.com/kvcache-ai/Mooncake). Use it
when the vision encoder should scale independently of prefill and decode.
For KV-cache transfer between prefiller and decoder, see
[MooncakeConnector Usage Guide](mooncake_connector_usage.md). That path uses
`MooncakeConnector`, not `ECMooncakeConnector`.
Added in [PR #41567](https://github.com/vllm-project/vllm/pull/41567).
Overview of disaggregated encoding:
[Disaggregated Encoder](disagg_encoder.md).
!!! important
Static proxy flags (`--ec-consumer-zmq-addrs`) support Mooncake EC only in
**1E + 1PD** mode (`--prefill-servers-urls disable`). For 1E + 1P + 1D,
register `ec_zmq_addrs` on the **prefill** consumer through the proxy's
dynamic registration API instead of the static CLI flag. See
`examples/disaggregated/disaggregated_encoder/README.md`.
## Prerequisites
Install Mooncake Transfer Engine:
```bash
# CUDA 13 (vLLM default)
uv pip install mooncake-transfer-engine-cuda13
# CUDA 12
uv pip install mooncake-transfer-engine
```
The two wheels are the same release built against different CUDA majors.
Importing the wrong one fails with
`libcudart.so.<major>: cannot open shared object file`.
## 1E + 1PD
One encoder producer and one combined prefill/decode consumer. The consumer
listens on `ec_ip` + `ec_port` (ZMQ control). The proxy must advertise that
address with `--ec-consumer-zmq-addrs`.
### Encoder (producer)
The producer must run with `tensor_parallel_size=1`,
`pipeline_parallel_size=1`, and `data_parallel_size=1`.
```bash
CUDA_VISIBLE_DEVICES=0 vllm serve Qwen/Qwen2.5-VL-3B-Instruct \
--port 19534 \
--gpu-memory-utilization 0.35 \
--enable-request-id-headers \
--no-enable-prefix-caching \
--max-num-batched-tokens 16384 \
--allowed-local-media-path /path/to/media \
--ec-transfer-config '{
"ec_connector": "ECMooncakeConnector",
"ec_role": "ec_producer",
"ec_connector_extra_config": {
"mooncake_protocol": "rdma"
}
}'
```
### Prefill+Decode (consumer)
```bash
CUDA_VISIBLE_DEVICES=1 vllm serve Qwen/Qwen2.5-VL-3B-Instruct \
--port 19537 \
--gpu-memory-utilization 0.75 \
--enable-mm-embeds \
--enable-request-id-headers \
--no-enable-prefix-caching \
--allowed-local-media-path /path/to/media \
--ec-transfer-config '{
"ec_connector": "ECMooncakeConnector",
"ec_role": "ec_consumer",
"ec_ip": "127.0.0.1",
"ec_port": 19019,
"ec_connector_extra_config": {
"mooncake_protocol": "rdma"
}
}'
```
`--enable-mm-embeds` lets the PD instance inject transferred embeddings
instead of re-running the vision encoder.
### Proxy
```bash
python examples/disaggregated/disaggregated_encoder/disagg_epd_proxy.py \
--host 0.0.0.0 \
--port 10002 \
--encode-servers-urls http://127.0.0.1:19534 \
--prefill-servers-urls disable \
--decode-servers-urls http://127.0.0.1:19537 \
--ec-consumer-zmq-addrs tcp://127.0.0.1:19019
```
`--ec-consumer-zmq-addrs` must align with `--decode-servers-urls` (the
combined PD instances). For data-parallel consumers, list each replica
consecutively and set `--ec-consumer-dp-size`.
### Correctness script
```bash
# TCP (no RDMA device required; CI default)
MOONCAKE_EC_PROTOCOL=tcp \
bash tests/v1/ec_connector/integration/run_epd_mooncake_ec_full_pipeline.sh
# RDMA
MOONCAKE_EC_PROTOCOL=rdma \
bash tests/v1/ec_connector/integration/run_epd_mooncake_ec_full_pipeline.sh
```
The TCP path sets `MC_FORCE_TCP=1` so Transfer Engine does not auto-select
RDMA when an HCA is present.
## Configuration
Producer and consumer must use the same `mooncake_protocol`.
| Setting | Where | Default | Notes |
| --------- | ------- | --------- | ------- |
| `mooncake_protocol` | `ec_connector_extra_config` | `rdma` | Set `tcp` when no verbs device is available. With TCP, also export `MC_FORCE_TCP=1`. |
| `ec_ip` / `ec_port` | consumer `ECTransferConfig` | `127.0.0.1` / `14579` | First ZMQ control port. Tensor-parallel rank `r` listens on `ec_port + dp_index * tp_size + r`. |
| `ec_buffer_size` | `ECTransferConfig` | `1e9` | Registered receive arena on the consumer, in bytes. |
| `control_timeout_s` | `ec_connector_extra_config` | `30` | Reservation RPC timeout. |
| `push_wait_timeout_s` | `ec_connector_extra_config` | `60` | How long the consumer waits for a matching encoder push. |
| `WITH_NVIDIA_PEERMEM` | process environment | `1` | Read by Mooncake. Set `0` on hosts without `nvidia-peermem` (DMA-BUF path). |
| `--ec-consumer-zmq-addrs` | proxy | empty | `tcp://<ec_ip>:<ec_port>` for each PD replica. Required for Mooncake EC. |
## 1E + 1P + 1D
The filesystem example is `examples/disaggregated/disaggregated_encoder/disagg_1e1p1d_example.sh`
(`ECExampleConnector` for encoder cache, `NixlConnector` for KV). P→D KV can
use `MooncakeConnector` instead; see
[MooncakeConnector Usage Guide](mooncake_connector_usage.md).
Do **not** pass `--ec-consumer-zmq-addrs` together with independent prefill
URLs: the static proxy flag is E+PD only. For Mooncake EC with a dedicated
prefill, use dynamic registration and set `ec_zmq_addrs` on the prefill
consumer as described in
`examples/disaggregated/disaggregated_encoder/README.md`.
## Troubleshooting
| Symptom | Likely cause |
| --------- | ---------------- |
| `Install mooncake-transfer-engine` / `libcudart.so.*` | Wrong or missing Mooncake wheel for this CUDA major. |
| `Failed to register memory ... Bad address [14]` | GPUDirect registration failed. Set `WITH_NVIDIA_PEERMEM=0`. |
| Proxy startup error about E+PD only | `--ec-consumer-zmq-addrs` used with a real `--prefill-servers-urls`. |
| Consumer waits then errors; encoder never finishes the push | `--ec-consumer-zmq-addrs` does not match consumer `ec_ip`/`ec_port`, or producer/consumer protocols differ. |
| TCP transfers stall on a host that also has RDMA | Export `MC_FORCE_TCP=1`. |
| Producer rejected at startup | Encoder used tensor, pipeline, or data parallelism. Keep the producer unsharded. |
+10 -9
View File
@@ -1210,7 +1210,7 @@ vLLM decodes audio bytes into waveforms using a selectable decoding backend:
| Backend | Description |
| --- | --- |
| `auto` (default) | soundfile, falling back to torchcodec, then PyAV |
| `auto` (default) | torchcodec, falling back to soundfile, then PyAV |
| `soundfile` | libsndfile only, no fallback |
| `pyav` | PyAV (FFmpeg) only, no fallback |
| `torchcodec` | TorchCodec (PyTorch-native) only, no fallback |
@@ -1226,17 +1226,18 @@ vllm serve mistralai/Voxtral-Mini-3B-2507 \
`pyav` drives FFmpeg through a per-frame Python generator, so under
concurrency the Python/C crossings contend on the GIL. `torchcodec`
decodes each stream in a single call that releases the GIL for its whole
duration. Select it explicitly for concurrent decoding workloads.
`auto` prefers soundfile for supported formats to preserve their existing
decoding behavior, including encoder padding. Audio extracted from formats
that soundfile cannot read, such as video containers, can use torchcodec
through the fallback chain.
duration, which is why `auto` prefers it when many requests decode audio
concurrently, such as when audio tracks are extracted from video.
vLLM normalizes codec padding on this path (e.g. it trims trailing Vorbis
padding that older system FFmpeg versions (< 5.0) fail to trim), so the
decoded waveform matches the soundfile reference length.
!!! note
`torchcodec` ships as a requirement on CUDA, CPU and XPU builds. On other
platforms (e.g. ROCm, TPU), install it manually to enable that backend.
torchcodec also links against a system FFmpeg installation. If the package
or FFmpeg is unavailable, `auto` uses the soundfile → PyAV chain.
platforms (e.g. ROCm, TPU) `auto` falls back to the soundfile → PyAV
chain unless you install it manually. torchcodec also links against a
system FFmpeg installation; if FFmpeg is missing, `auto` falls back the
same way.
### Embedding Inputs
+1
View File
@@ -454,6 +454,7 @@ exported when NixlConnector is active:
| `vllm:nixl_num_failed_transfers` | Counter | Cumulative count of failed NIXL KV-block transfers, including handshake and notification (`send_notif`) failures. These are grouped because all are sporadic lower-transport-layer events. |
| `vllm:nixl_num_failed_notifications` | Counter | Cumulative count of failed notifications (`send_notif`), retained for compatibility. These failures are also included in `vllm:nixl_num_failed_transfers`; do not sum the two counters. |
| `vllm:nixl_num_kv_expired_reqs` | Counter | Requests whose KV blocks expired on the prefiller before the decoder read them (tracked on the P instance). Kept separate from the failure counters above: KV expiry is an autoscaler/lease-tuning signal, not a transport issue. |
| `vllm:nixl_num_notifications_after_expiry` | Counter | Completion notifications that arrive for a request that is no longer tracked, usually because its KV lease expired first. Pull connector: counted on P, where it means the decoder may have read blocks that were already reused; each matches a "Potentially invalid KV blocks" error log. Push connector: counted on whichever instance tracks the request as neither a send nor a receive (P after the lease expired, or D after the receive completed or was abandoned); each matches an "Unrecognized request" error log. Counted per notification, not per request: each remote rank notifies separately, so with TP > 1 or heterogeneous TP one late request counts several times, and the value is not comparable to `vllm:nixl_num_kv_expired_reqs`. |
!!! tip
High `vllm:nixl_num_kv_expired_reqs` indicates that the prefiller's lease
+1
View File
@@ -29,6 +29,7 @@ vLLM currently supports the following reasoning models:
| [MiniMax-M2](https://huggingface.co/MiniMaxAI/MiniMax-M2) | `minimax_m2_append_think` | `json`, `regex` | ✅ |
| [Qwen3 series](https://huggingface.co/collections/Qwen/qwen3-67dd247413f0e2e4f653967f) | `qwen3` | `json`, `regex` | ✅ |
| [QwQ-32B](https://huggingface.co/Qwen/QwQ-32B) | `deepseek_r1` | `json`, `regex` | ✅ |
| [Step-3.5 / Step-3.7 Flash](https://huggingface.co/stepfun-ai/Step-3.5-Flash) | `step3p5` | `json`, `regex` | ✅ |
!!! note
IBM Granite 3.2 and DeepSeek-V3.1 reasoning is disabled by default; to enable it, you must also pass `thinking=True` in your `chat_template_kwargs`.
+46
View File
@@ -171,6 +171,29 @@ from HuggingFace; and you can find an example of this in a `tokenizer_config.jso
If your favorite tool-calling model is not supported, please feel free to contribute a parser & tool use chat template!
### Checkpoint Response Templates (`hf`)
Checkpoints can define a `response_template` in `tokenizer_config.json` to describe their reasoning, content, and tool-call wire format. Select the
`hf` parser to parse output from this metadata instead of a model-specific parser. Like `--tokenizer-mode hf`, it relies on the checkpoint's
Hugging Face metadata rather than model-specific code:
```bash
vllm serve <model> \
--enable-auto-tool-choice \
--tool-call-parser hf \
--reasoning-parser hf
```
Either parser can be selected on its own. Startup fails if the tokenizer has no `response_template`, or if it lacks the `thinking` or `tool_calls`
field the selected parser needs. The `hf` parser cannot be combined with other reasoning or tool call parsers.
When streaming, each tool call is emitted whole once its region parses. A call cut off before its closer, for example by a closer that is also a
stop token, is parsed from the text generated so far; a call that does not parse is dropped.
Response templates currently provide parsing, not format-specific constrained decoding. The parser therefore rejects requests for strict tools,
required or named tool choice, or `parallel_tool_calls=false`. A custom server or request chat template is allowed, but the parser still expects the
checkpoint's output format.
!!! note
With `tool_choice="auto"`, structural-tag constraints require both `VLLM_ENFORCE_STRICT_TOOL_CALLING=true` (the default) and at least one tool with `strict: true`, or a server-side floor set via `--tool-strict-level`. When these conditions are met and the selected parser supports structural tags, vLLM constrains the tool-call envelope and pins the argument schema of each tool that sets `strict: true` (or of every tool under `--tool-strict-level parameter`). Otherwise, vLLM extracts tool calls from raw text, so arguments may occasionally be malformed or violate the function's parameter schema.
@@ -458,6 +481,29 @@ Supported models:
Flags: `--tool-call-parser qwen3_xml`
### MiMo-V2.6 Models (`mimo`)
Supported models:
* `XiaomiMiMo/MiMo-V2.6-Pro-RL`
* `XiaomiMiMo/MiMo-V2.6-Flash-RL`
Use the checkpoint's chat template with `--tool-call-parser mimo --reasoning-parser mimo --enable-auto-tool-choice`.
For schema-constrained tool arguments, set `strict: true` inside each tool's `function`, or use `--tool-strict-level parameter`.
The parser supports `auto`, `required`, named tools, and `parallel_tool_calls=False`, preserving leading/trailing whitespace in string arguments.
### Step-3.5 and Step-3.7 Models (`step3p5`)
Supported models:
* `stepfun-ai/Step-3.5-Flash`
* `stepfun-ai/Step-3.5-Flash-FP8`
* `stepfun-ai/Step-3.7-Flash`
Flags: `--tool-call-parser step3p5 --reasoning-parser step3p5 --enable-auto-tool-choice`
The parser supports `auto`, `required`, and named tools.
### Olmo 3 Models (`olmo3`)
Olmo 3 models output tool calls in a format that is very similar to the one expected by the `pythonic` parser (see below), with a few differences. Each tool call is a pythonic string, but the parallel tool calls are newline-delimited, and the calls are wrapped within XML tags as `<function_calls>..</function_calls>`. In addition, the parser also allows JSON boolean and null literals (`true`, `false`, and `null`) in addition to the pythonic ones (`True`, `False`, and `None`).
+14 -15
View File
@@ -77,30 +77,17 @@ watermarked. The watermark signal is diluted in proportion to the share of
output tokens supplied by accepted drafts; rejected drafts do not dilute it
because their recovery tokens are watermarked.
Speculative-decoding token paths do not currently support generation-side
context deduplication. The configured `deduplicate_contexts` policy is not
applied to accepted drafts, rejection-recovery tokens, or bonus tokens.
For `dual_key_gumbel`, `alpha` has no effect under speculative decoding. The
speculative protocol selects the key for each token instead.
## Algorithms
### Gumbel-max
Gumbel-max derives a deterministic pseudorandom value from the key, prior token
context, and every candidate token, then uses the resulting Gumbel noise for
categorical sampling. See
[Aaronson's original presentation](https://simons.berkeley.edu/sites/default/files/2024-10/LLM24-2%20Slides%20-%20Scott%20Aaronson.pdf).
Gumbel-max requires stochastic sampling. Greedy requests (`temperature=0`)
bypass watermarking and emit a warning once per worker.
## Context deduplication
When a token context is repeated, generation can use ordinary sampling for that
occurrence. Reusing the repeated context leads to a bias over the sequence, as
certain token choices would be correlated. Ordinary sampling at these positions
allows for single-sequence non-distortion (see section G.3 of the
[SynthID-Text supplementary materials](https://media.springernature.com/original/springer-static/esm/art%3A10.1038%2Fs41586-024-08025-4/MediaObjects/41586_2024_8025_MOESM1_ESM.pdf)).
This also applies to drafter watermarking during speculative decoding.
The detector independently deduplicates contexts so repeated keyed random
vectors are not treated as independent evidence, meaning that context
deduplication at generation time does not reduce the watermarking signal, unless
@@ -143,6 +130,18 @@ vllm serve MODEL \
'{"algorithm":"gumbel","key":42,"deduplicate_contexts":"all"}'
```
## Algorithms
### Gumbel-max
Gumbel-max derives a deterministic pseudorandom value from the key, prior token
context, and every candidate token, then uses the resulting Gumbel noise for
categorical sampling. See
[Aaronson's original presentation](https://simons.berkeley.edu/sites/default/files/2024-10/LLM24-2%20Slides%20-%20Scott%20Aaronson.pdf).
Gumbel-max requires stochastic sampling. Greedy requests (`temperature=0`)
bypass watermarking and emit a warning once per worker.
### Dual-key Gumbel-max
Dual-key Gumbel-max derives independent keys A and B from one configured master
+2 -2
View File
@@ -312,8 +312,8 @@ vLLM CPU supports data parallel (DP), tensor parallel (TP) and pipeline parallel
### Which quantization configs does vLLM CPU support?
- vLLM CPU supports quantizations:
- AWQ (x86, s390x)
- GPTQ (x86, s390x)
- AWQ (x86, s390x, Power)
- GPTQ (x86, s390x, Power)
- compressed-tensor INT8 W8A8 (x86, s390x only)
### Why do I see `get_mempolicy: Operation not permitted` when running in Docker?
+5 -2
View File
@@ -22,6 +22,7 @@ Sorted alphabetically by GitHub handle:
- [@DarkLight1337](https://github.com/DarkLight1337): Multimodality, API server
- [@dllehr-amd](https://github.com/dllehr-amd): AMD integration
- [@esmeetu](https://github.com/esmeetu): developer marketing, community
- [@gau-nernst](https://github.com/gau-nernst): Kernels and performance
- [@gshtras](https://github.com/gshtras): ROCm / AMD GPU integration
- [@Harry-Chen](https://github.com/Harry-Chen): Build, CI, CUDA architecture
- [@heheda12345](https://github.com/heheda12345): Hybrid memory allocator
@@ -54,6 +55,7 @@ Sorted alphabetically by GitHub handle:
- [@tomeras91](https://github.com/tomeras91): MoE, Mamba, Ray
- [@vadiklyutiy](https://github.com/vadiklyutiy): CI, performance, kernels
- [@WoosukKwon](https://github.com/WoosukKwon): Project lead, engine core
- [@wzhao18](https://github.com/wzhao18): Kernels and performance, KV offloading
- [@xinli-sw](https://github.com/xinli-sw): Quantization and model support
- [@xuechendi](https://github.com/xuechendi): Intel CPU/XPU integration, KV connector
- [@yaochengji](https://github.com/yaochengji): TPU integration
@@ -173,9 +175,10 @@ If you have PRs touching the area, please feel free to ping the area owner for r
### External Kernels Integration
- FlashAttention: @LucasWilkinson, @MatthewBonanni
- FlashInfer: @LucasWilkinson, @mgoin, @WoosukKwon, @MatthewBonanni
- FlashInfer: @LucasWilkinson, @mgoin, @WoosukKwon, @MatthewBonanni, @wzhao18
- Blackwell Kernels: @mgoin, @yewentao256
- DeepEP/DeepGEMM: @mgoin, @yewentao256
- FlashKDA: @gau-nernst
### Integrations
@@ -186,7 +189,7 @@ If you have PRs touching the area, please feel free to ping the area owner for r
### Hardware
- Plugin Interface: @youkaichao, @Yikun
- NVIDIA GPU: @pavanimajety
- NVIDIA GPU: @pavanimajety, @wzhao18
- AMD GPU: @gshtras, @tjtanaa, @hongxiayang, @shen-shanshan, @AndreasKaratzas
- Intel CPU/GPU: @jikunshang, @bigPYJ1151, @xuechendi
- Google TPU: @yaochengji
+34
View File
@@ -0,0 +1,34 @@
<!-- Material nav.html (https://github.com/squidfunk/mkdocs-material/blob/9.7.7/material/templates/partials/nav.html) with navigation.prune applied to the API reference only -->
{% import "partials/nav-item.html" as item with context %}
{% set features = features + ["navigation.prune"] %}
{% import "partials/nav-item.html" as pruned_item with context %}
{% set class = "md-nav md-nav--primary" %}
{% if "navigation.tabs" in features %}
{% set class = class ~ " md-nav--lifted" %}
{% endif %}
{% if "toc.integrate" in features %}
{% set class = class ~ " md-nav--integrated" %}
{% endif %}
<nav class="{{ class }}" aria-label="{{ lang.t('nav') }}" data-md-level="0">
<label class="md-nav__title" for="__drawer">
<a href="{{ config.extra.homepage | d(nav.homepage.url, true) | url }}" title="{{ config.site_name | e }}" class="md-nav__button md-logo" aria-label="{{ config.site_name }}" data-md-component="logo">
{% include "partials/logo.html" %}
</a>
{{ config.site_name }}
</label>
{% if config.repo_url %}
<div class="md-nav__source">
{% include "partials/source.html" %}
</div>
{% endif %}
<ul class="md-nav__list" data-md-scrollfix>
{% for nav_item in nav.items %}
{% set path = "__nav_" ~ loop.index %}
{% if nav_item.children and (nav_item.children | first).url == "api/" %}
{{ pruned_item.render(nav_item, path, 1) }}
{% else %}
{{ item.render(nav_item, path, 1) }}
{% endif %}
{% endfor %}
</ul>
</nav>
+4 -4
View File
@@ -374,7 +374,6 @@ th {
| ------------ | ------ | ----------------- | -------------------- | ------------------------- |
| `AfmoeForCausalLM` | Afmoe | TBA | ✅︎ | ✅︎ |
| `ApertusForCausalLM` | Apertus | `swiss-ai/Apertus-8B-2509`, `swiss-ai/Apertus-70B-Instruct-2509`, etc. | ✅︎ | ✅︎ |
| `ArceeForCausalLM` | Arcee (AFM) | `arcee-ai/AFM-4.5B-Base`, etc. | ✅︎ | ✅︎ |
| `AXK1ForCausalLM` | A.X-K1 | `skt/A.X-K1`, etc. | | ✅︎ |
| `BailingMoeForCausalLM` | Ling | `inclusionAI/Ling-lite-1.5`, `inclusionAI/Ling-plus`, etc. | ✅︎ | ✅︎ |
| `BailingMoeV2ForCausalLM` | Ling | `inclusionAI/Ling-mini-2.0`, etc. | ✅︎ | ✅︎ |
@@ -384,7 +383,6 @@ th {
| `ChatGLMModel`, `ChatGLMForConditionalGeneration` | ChatGLM | `zai-org/chatglm2-6b`, `zai-org/chatglm3-6b`, `thu-coai/ShieldLM-6B-chatglm3`, etc. | ✅︎ | ✅︎ |
| `CohereForCausalLM`, `Cohere2ForCausalLM` | Command-R, Command-A | `CohereLabs/c4ai-command-r-v01`, `CohereLabs/c4ai-command-r7b-12-2024`, `CohereLabs/c4ai-command-a-03-2025`, `CohereLabs/command-a-reasoning-08-2025`, etc. | ✅︎ | ✅︎ |
| `Cohere2MoeForCausalLM` | North-Mini-Code | `CohereLabs/North-Mini-Code`, etc. | ✅︎ | ✅︎ |
| `CwmForCausalLM` | CWM | `facebook/cwm`, etc. | ✅︎ | ✅︎ |
| `DbrxForCausalLM` | DBRX | `databricks/dbrx-base`, `databricks/dbrx-instruct`, etc. | | ✅︎ |
| `DeciLMForCausalLM` | DeciLM | `nvidia/Llama-3_3-Nemotron-Super-49B-v1`, etc. | ✅︎ | ✅︎ |
| `DeepseekForCausalLM` | DeepSeek | `deepseek-ai/deepseek-llm-67b-base`, `deepseek-ai/deepseek-llm-7b-chat`, etc. | ✅︎ | ✅︎ |
@@ -406,7 +404,6 @@ th {
| `Gemma3ForCausalLM` | Gemma 3 | `google/gemma-3-1b-it`, etc. | ✅︎ | ✅︎ |
| `Gemma3nForCausalLM` | Gemma 3n | `google/gemma-3n-E2B-it`, `google/gemma-3n-E4B-it`, etc. | | |
| `Gemma4ForCausalLM` | Gemma 4 | `google/gemma-4-E2B-it`, etc. | ✅︎ | ✅︎ |
| `GlmForCausalLM` | GLM-4 | `zai-org/glm-4-9b-chat-hf`, etc. | ✅︎ | ✅︎ |
| `Glm4ForCausalLM` | GLM-4-0414 | `zai-org/GLM-4-32B-0414`, etc. | ✅︎ | ✅︎ |
| `Glm4MoeForCausalLM` | GLM-4.5, GLM-4.6, GLM-4.7 | `zai-org/GLM-4.5`, etc. | ✅︎ | ✅︎ |
| `Glm4MoeLiteForCausalLM` | GLM-4.7-Flash | `zai-org/GLM-4.7-Flash`, etc. | ✅︎ | ✅︎ |
@@ -439,7 +436,6 @@ th {
| `LongcatFlashForCausalLM` | LongCat-Flash | `meituan-longcat/LongCat-Flash-Chat`, `meituan-longcat/LongCat-Flash-Chat-FP8` | ✅︎ | ✅︎ |
| `MambaForCausalLM` | Mamba | `state-spaces/mamba-130m-hf`, `state-spaces/mamba-790m-hf`, `state-spaces/mamba-2.8b-hf`, etc. | | ✅︎ |
| `Mamba2ForCausalLM` | Mamba2 | `mistralai/Mamba-Codestral-7B-v0.1`, etc. | | ✅︎ |
| `MellumForCausalLM` | Mellum 2 | `JetBrains/Mellum2-12B-A2.5B-Base`, etc. | | ✅︎ |
| `MiMoForCausalLM` | MiMo | `XiaomiMiMo/MiMo-7B-RL`, etc. | ✅︎ | ✅︎ |
| `MiMoV2FlashForCausalLM` | MiMoV2Flash | `XiaomiMiMo/MiMo-V2-Flash`, etc. | | ✅︎ |
| `MiMoV2ForCausalLM` | MiMoV2Pro | `XiaomiMiMo/MiMo-V2.5-Pro`, etc. | | ✅︎ |
@@ -486,10 +482,14 @@ Some models are supported only via the [Transformers modeling backend](#transfor
| Architecture | Models | Example HF Models | [LoRA](../features/lora.md) | [PP](../serving/parallelism_scaling.md) |
| ------------ | ------ | ----------------- | -------------------- | ------------------------- |
| `ArceeForCausalLM` | Arcee (AFM) | `arcee-ai/AFM-4.5B-Base`, etc. | ✅︎ | ✅︎ |
| `CwmForCausalLM` | CWM | `facebook/cwm`, etc. | ✅︎ | ✅︎ |
| `FlexOlmoForCausalLM` | FlexOlmo | `allenai/FlexOlmo-7x7B-1T`, `allenai/FlexOlmo-7x7B-1T-RT`, etc. | | ✅︎ |
| `GlmForCausalLM` | GLM-4 | `zai-org/glm-4-9b-chat-hf`, etc. | ✅︎ | ✅︎ |
| `GPTBigCodeForCausalLM` | StarCoder, SantaCoder, WizardCoder | `bigcode/starcoder`, `bigcode/gpt_bigcode-santacoder`, `WizardLM/WizardCoder-15B-V1.0`, etc. | ✅︎ | |
| `HunYuanDenseV1ForCausalLM` | Hunyuan Dense | `tencent/Hunyuan-7B-Instruct` | ✅︎ | ✅︎ |
| `HunYuanMoEV1ForCausalLM` | Hunyuan-A13B | `tencent/Hunyuan-A13B-Instruct`, `tencent/Hunyuan-A13B-Pretrain`, `tencent/Hunyuan-A13B-Instruct-FP8`, etc. | ✅︎ | ✅︎ |
| `MellumForCausalLM` | Mellum 2 | `JetBrains/Mellum2-12B-A2.5B-Base`, etc. | | ✅︎ |
| `NanbeigeForCausalLM` | Nanbeige4.2 | `Nanbeige/Nanbeige4.2-3B`, etc. | ✅︎ | ✅︎ |
| `OlmoForCausalLM` | OLMo | `allenai/OLMo-1B-hf`, `allenai/OLMo-7B-hf`, etc. | ✅︎ | ✅︎ |
| `Olmo2ForCausalLM` | OLMo2 | `allenai/OLMo-2-0425-1B`, etc. | ✅︎ | ✅︎ |
+3 -9
View File
@@ -1,4 +1,3 @@
# shellcheck disable=SC2317 # code after the exit/exit-183 gate below is intentionally dead pending re-enablement (see line comment)
if [ "$READTHEDOCS_VERSION_TYPE" != "external" ]; then
echo "Not a PR build (version type=$READTHEDOCS_VERSION_TYPE); skipping pre-run-check gate."
exit 0
@@ -11,24 +10,19 @@ if [ -n "$GITHUB_TOKEN" ]; then
CURL_AUTH=(-H "Authorization: Bearer $GITHUB_TOKEN")
fi
# Docs builds are now manually enabled via the 'build-docs' label.
# The 'build-docs' label forces a docs build, bypassing the checks below.
echo "Checking for the 'build-docs' label on PR #${READTHEDOCS_VERSION_NAME}..."
LABELS=$(curl -sS "${CURL_AUTH[@]}" "https://api.github.com/repos/vllm-project/vllm/issues/${READTHEDOCS_VERSION_NAME}/labels" | python3 -c "import sys, json; print('\n'.join(l.get('name', '') for l in json.load(sys.stdin)))")
if printf '%s\n' "$LABELS" | grep -qx "build-docs"; then
echo "PR has the 'build-docs' label; continuing build."
echo "PR has the 'build-docs' label; forcing build."
exit 0
else
echo "PR does not have the 'build-docs' label; cancelling build."
# See https://docs.readthedocs.com/platform/latest/guides/build/skip-build.html for info on exit code
exit 183
fi
# Everything below this line is effectively disabled as a temporary measure.
echo "Checking for changes to docs-affecting files vs origin/main..."
DOCS_PATHS=(
docs/ # Actual docs content
examples/ # Examples are rendered in docs
vllm/ # API & CLI reference
# vllm/ # API & CLI reference (too broad, use 'build-docs' label)
requirements/test/cuda.txt # CLI reference (see docs/mkdocs/gen_files/generate_argparse.py)
mkdocs.yaml # Affects build process
.readthedocs.yaml # Affects build process
+19 -4
View File
@@ -23,6 +23,10 @@ endpoints.
The derender step needs more than the engine's `token_ids`. It also consumes the original `chat_request`/`completion_request` and `prompt_tokens` carried over from the render step (see [Request format](#request-format)) so the tool and reasoning parsers have the context they need. The chat streaming path additionally needs `prompt_token_ids` (see [Streaming cost](#streaming-cost)) when a parser is configured.
All derender endpoints also accept an optional `prompt_token_ids` (the `token_ids` of the `GenerateRequest` returned by `/render`). It's used to seed detokenization from the end of the prompt, the same way the engine does. Without it, the first output token is decoded as if it started a new sequence. On SentencePiece tokenizers (Llama-2, Phi-3) that drops its leading space compared to `/v1/chat/completions` and `/v1/completions`. If it's omitted, derender falls back to the generate response's `prompt_token_ids` (set when the `GenerateRequest` has `return_token_ids`), then to decoding without prompt context.
Seeding only reads the last few IDs (7 today), so a suffix of the prompt is enough and a long prompt doesn't need to travel back to derender. The exception is streaming chat with a tool or reasoning parser configured. There the parser also reads `prompt_token_ids` to decide whether the prompt left reasoning open and it can look back to any point in the prompt. In this case, send the full list (see [Streaming cost](#streaming-cost)).
## API Reference
- Chat Completions Derender API (`/v1/chat/completions/derender`)
@@ -96,9 +100,10 @@ The server keeps no state between calls. Everything the next call needs is in `s
- **`n > 1` is N streams.** Generate interleaves chunks for different choices and tags each with its `index`. Keep one `stream_state` per index and send each chunk with the state for its index. Each index gets its own `role` delta on its first chunk.
- **Send every chunk through, including the last ones.** The finish reason usually arrives with the final tokens. A chunk with a `finish_reason` and no tokens is accepted too and flushes any buffered tool call arguments. With `stream_options: {"include_usage": true}`, generate ends with a usage only chunk (`choices: []`). Send it through derender like any other chunk to get the usage chunk for the response. `usage.prompt_tokens` is the request's `prompt_tokens` if set, otherwise the generate chunk's.
- **Send the same context on every call.** `chat_request` and `prompt_token_ids` aren't kept between calls, so they go with every chunk including the usage chunk. Both are required when a tool or reasoning parser is configured. `prompt_token_ids` is the `token_ids` of the `GenerateRequest` returned by `/render`.
- **Send `prompt_token_ids` on the first chunk, even without a parser.** It seeds detokenization so the first token's leading space matches the coupled endpoint (see [Request format](#request-format)). Later chunks already carry that context in `stream_state`.
- **Don't forward `[DONE]`.** It marks the end of the generate stream and isn't a chunk.
Streaming chunks don't carry logprobs yet, so `logprobs` on a generate chunk are dropped. The non-streaming endpoints do resolve them, including `token_id:N` placeholders.
Streaming derender resolves `logprobs` on the plain detokenization path, including `token_id:N` placeholders, same as the non-streaming endpoints (see [Streaming state and logprobs](#streaming-state-and-logprobs)). The parser path doesn't resolve them yet and drops them.
## Streaming cost
@@ -110,11 +115,20 @@ When a tool or reasoning parser is configured, parser internal state (buffered m
- **Transport**: `output_token_ids` and `output_chunk_lens` round-trip in full in both directions on every call. `output_chunk_lens` has one entry per chunk, which is one per token without speculative decoding. This means O(n) bytes per chunk, O(n²) bytes over a full generation. Bounded by `max_model_len`. `prompt_token_ids` is sent in full on every call too and it isn't trimmed as `output_token_ids` grows. This means that for most of a stream it dominates the per chunk payload. A 100k token prompt with 1k tokens of output means `prompt_token_ids` is ~99% of the request body on every chunk.
- **Compute**: replay is O(n) `parse_delta` calls per chunk (O(n²) per generation). `parse_delta` itself is O(n) for parsers that re-scan accumulated text (e.g. Hermes tool-call JSON, DeepSeek-R1 reasoning). The per-generation cost is O(n³) character work, not O(n²). This is a deliberately minimal first implementation with no caching layer.
- The parser path also requires `prompt_token_ids` so `parse_delta` can settle whether the prompt left reasoning open or not. Since parser state can't be carried across calls, it re-scans the full prompt once per chunk.
- The parser path also requires `prompt_token_ids` so `parse_delta` can settle whether the prompt left reasoning open or not. Since parser state can't be carried across calls, it re-scans the full prompt once per chunk. Send the full list and not a suffix because the last reasoning marker can be anywhere in the prompt.
- Replay runs off the event loop on the renderer's executor (`renderer_num_workers`, default `1`). Size it for the expected number of concurrent parser configured streams.
`output_token_ids` and `prompt_token_ids` are both bounded by `max_model_len` but callers streaming long reasoning traces through a parser configured model should expect materially more state transport and CPU cost than the plain detokenization path.
## Streaming state and logprobs
Streaming derender is stateless on the server side: all mutable state lives in the client-carried `stream_state` (`DerenderStreamState`), passed back on every per-chunk call. Besides the bounded incremental detokenization window (`prev_tokens`, `prefix_offset`, `read_offset`), `role_sent`, and the parser path's replay fields (see [Streaming cost](#streaming-cost)), the state carries two fields for logprob handling:
- `logprob_context_token_ids`: the trailing sampled token IDs (at most 4) from previous chunks, used to seed byte-fallback (U+FFFD) correction so multi-byte characters whose tokens split across chunk boundaries still resolve to real strings
- `logprob_text_offset`: the cumulative emitted text length, so `text_offset` in completion streaming logprobs stays absolute across chunks instead of restarting at 0
When a streamed `GenerateResponseStreamChoice` carries `logprobs`, the `token_id:N` placeholders are resolved per chunk and the resolved logprobs are attached to the corresponding streamed choice — for chat as `ChatCompletionLogProbs`, for completions converted to the flat `CompletionLogProbs` lists. Chunks without `logprobs` produce choices with `logprobs: null`.
## Example
The example below drives the full `render → generate → derender` round trip for a chat request against a GPU less render server (`/render`, `/derender`) and a token-in / token-out engine (`/inference/v1/generate`).
@@ -144,7 +158,7 @@ with httpx.Client(timeout=60.0) as client:
generate_request = client.post(
f"{RENDER}/v1/chat/completions/render", json=chat_request
).json()
prompt_tokens = len(generate_request["token_ids"])
prompt_token_ids = generate_request["token_ids"]
# 2. Generate: token IDs -> token IDs (token-in / token-out engine)
generate_response = client.post(
@@ -157,7 +171,8 @@ with httpx.Client(timeout=60.0) as client:
json={
"model": MODEL,
"generate_response": generate_response,
"prompt_tokens": prompt_tokens,
"prompt_tokens": len(prompt_token_ids),
"prompt_token_ids": prompt_token_ids,
"chat_request": chat_request,
},
).json()
+1 -2
View File
@@ -51,8 +51,7 @@ and provision transport limits and memory accordingly.
For example, start a standard inference server with scale-out endpoints enabled:
```bash
VLLM_ENABLE_SCALE_OUT_ENDPOINTS=1 \
vllm serve meta-llama/Llama-3.1-8B-Instruct
vllm serve meta-llama/Llama-3.1-8B-Instruct --enable-scale-out
```
Send a Responses request and use `jq` to extract the `token_ids` field:
+1 -1
View File
@@ -16,7 +16,7 @@ The following open-source RL libraries use vLLM for fast rollouts (sorted alphab
- [Unsloth](https://github.com/unslothai/unsloth)
- [verl](https://github.com/volcengine/verl)
For weight synchronization between training and inference, see the [Weight Transfer](weight_transfer/README.md) documentation, which covers the pluggable backend system with [NCCL](weight_transfer/nccl.md) (multi-GPU) and [IPC](weight_transfer/ipc.md) (same-GPU) engines.
For weight synchronization between training and inference, see the [Weight Transfer](weight_transfer/README.md) documentation, which covers the pluggable backend system with [NCCL](weight_transfer/nccl.md) (multi-GPU) and [IPC](weight_transfer/ipc.md) (same-GPU) engines. To verify that an update landed on every rank, see the [Weight Checker](weight_checker.md).
For pipelining generation and training to improve GPU utilization and throughput, see the [Async Reinforcement Learning](async_rl.md) guide, which covers the pause/resume API for safely updating weights mid-flight.
+39 -3
View File
@@ -53,6 +53,8 @@ The mask is also available via the `/inference/v1/generate` HTTP endpoint:
}
```
With `stream=true`, each chunk carries the masks of the tokens it emits.
## Requirements
| Requirement | Reason |
@@ -63,9 +65,12 @@ The mask is also available via the `/inference/v1/generate` HTTP endpoint:
| `top_k > 0` | Bounds mask size; pure top-p can produce vocab-sized masks |
| Model Runner V2 | Required by the async D2H copy pipeline |
Speculative decoding is supported; each emitted token gets the support of the
target distribution at its position.
The engine rejects unsupported combinations at startup or request time:
- Speculative decoding
- Speculative decoding with adaptive verification
- Diffusion models
- Custom logits processors (engine-level `--logits-processors`)
@@ -104,10 +109,41 @@ log_prob = log_softmax(masked_logits)[sampled_token_id]
Both sides normalize over the same token set, so the importance ratio is
consistent.
## Score centering with top-k logprobs
[Score centering](https://arxiv.org/abs/2609.20807) needs the sampler's top-k
token IDs and logprobs plus the sampled token's logprob; the paper uses
`k=128` and does not need the exact sampling support. vLLM already returns
this through the ordinary `logprobs` parameter:
```bash
vllm serve <model> --logprobs-mode processed_logprobs --max-logprobs 128
```
```json
{"sampling_params": {"temperature": 1.0, "top_p": 0.9, "logprobs": 128}}
```
Each position then carries the sampled token's logprob and the top-128
`(token_id, logprob)` pairs, normalized over the post-truncation support:
```json
{"token": "token_id:11", "logprob": -0.757,
"top_logprobs": [{"token": "token_id:11", "logprob": -0.757},
{"token": "token_id:13", "logprob": -0.632},
{"token": "token_id:0", "logprob": -9999.0}, ...]}
```
- `--max-logprobs` defaults to 20; raise it to the requested `k`.
- Tokens outside the support are returned as `-9999.0` with arbitrary IDs;
drop them. The finite entries are the sampling mask when it has at most `k`
tokens.
- The sampled token is always the first entry and may repeat inside the top-k.
- `/inference/v1/generate` encodes IDs as `token_id:<id>`; the OpenAI
endpoints need `return_tokens_as_token_ids`.
## Limitations
- **Engine-level flag:** `--return-sampling-mask` globally disables the
FlashInfer fused sampler. All requests pay the cost of the PyTorch sampling
path, even if they don't need the mask.
- **No streaming support:** The mask is returned only in the final response,
not in intermediate streaming chunks.
+60
View File
@@ -0,0 +1,60 @@
# Weight Checker
The Weight Checker verifies that an RL weight update landed on every rank. It
hashes each model parameter with SHA-256, so a caller can save the original
digests, reset the weights, transfer them back, and confirm they match.
Enable it with `VLLM_SERVER_DEV_MODE=1`. All operations use
`POST /weight_checker`:
| Request | Response |
| --- | --- |
| `{"action": "checksum"}` | `{"checksums": {key: sha256_hex}}` |
| `{"action": "reset"}` | `{"status": "reset"}` |
| `{"action": "compare", "baseline": {key: sha256_hex}}` | `{"match": bool, "mismatches": [key]}` |
Keys have the form `dp{dp}:pp{pp}:pcp{pcp}:tp{tp}:{tensor_name}`, so each
shard is checked separately. All actions cover the target model's parameters,
which is what weight loading writes; `reset` zeroes them. The endpoint keeps no
state, so the caller holds the baseline. Invalid requests return HTTP 400.
## RL weight-update check
```bash
URL=http://localhost:8000
# 1. Stop serving, then save the original digests.
curl -X POST "$URL/pause?mode=abort"
curl -X POST $URL/weight_checker -H 'Content-Type: application/json' \
-d '{"action":"checksum"}' > baseline.json
# 2. Zero the weights.
curl -X POST $URL/weight_checker -H 'Content-Type: application/json' \
-d '{"action":"reset"}'
# 3. Transfer the original weights with start/update/finish_weight_update.
# 4. Compare against the baseline; expect {"match": true, "mismatches": []}.
curl -X POST $URL/weight_checker -H 'Content-Type: application/json' \
-d "{\"action\":\"compare\",\"baseline\":$(jq .checksums baseline.json)}"
curl -X POST $URL/resume
```
## Limitations
- Stay paused from `checksum` to `compare`: the weights are invalid between
`reset` and the transfer, and EPLB moves experts while serving.
- A request covers the engines managed by the API server it reaches. With
several API servers (for example `--data-parallel-external-lb`), send it to
each one.
- Buffers, draft models and LoRA adapters are not checked.
- CPU backends that repack linear weights hide them from the checker.
- compressed-tensors online transforms are not supported: their Hadamard
weights are normalized once per storage, so restored copies differ.
- MoE weights padded beyond the checkpoint shape (DeepEP hidden size, MXFP4)
keep uninitialized padding, so `compare` reports them as mismatches.
- The engine must be awake: sleep levels 1 and 2 release the weight memory.
- `--offload-backend prefetch` is not supported: parameters point at staging
buffers, not the offloaded weights.
- Hashing copies every weight to CPU, so keep it off latency-sensitive paths.
+1
View File
@@ -36,6 +36,7 @@ engine drives on your behalf:
| [sparse_nccl](nccl.md#sparse-nccl) | NCCL broadcast | Checkpoint-coordinate sparse weight patches |
| [sharded_rdt](sharded_rdt.md) | NIXL / Ray Direct Transport (pull-based) | Very large models where each worker needs only its own slice (MoE with expert parallelism) |
| [nccl_m2n](m2n.md) | NCCL M2N reshard | Trainer and inference use different sharding layouts |
| [ModelExpress](modelexpress.md) | ModelExpress version-based transfer | Install immutable published weight versions, including object-storage checkpoints and deltas |
## Quickstart
@@ -0,0 +1,156 @@
# ModelExpress Engine
The `modelexpress` weight transfer engine installs immutable weight versions
published through [ModelExpress](https://github.com/ai-dynamo/modelexpress).
Each update identifies the exact version to install, while ModelExpress handles
source selection, transfer, and installation.
## When to Use ModelExpress
- Your workflow manages immutable weight versions through the ModelExpress
Refit API.
- Inference workers need to pull a specific published version through
ModelExpress, using GPU peer transfer or object storage.
- You want to load full checkpoints or replay checkpoint deltas from S3.
## How It Works
1. A publisher creates a weight-version resource through the ModelExpress Refit
API, publishes its payload, and marks the version READY.
2. The orchestrator pauses generation and calls `start_weight_update` on the
inference engine.
3. The orchestrator calls `update_weights` with the target `version_id`.
Each worker's `ModelExpressGeneratorClient` resolves and stages that version,
then applies it to the live vLLM model.
4. After a successful update, `finish_weight_update` releases the staged handle.
The orchestrator can then resume inference.
Initialize the backend before starting an update session, and send exactly one
version per session. Invalid transitions must raise; an orchestrator must keep
inference paused after an update error and retry with a new session only after
cleanup. An error does not roll back weights that were already applied.
`ModelExpressTrainerClient` is an optional publisher. Custom integrations can
manage weight-version resources directly through the ModelExpress server's
Refit API. For S3, create a STAGING version with the correct model, payload
format, lineage, and index URI; upload the index and referenced shards in the
format expected by `ModelExpressGeneratorClient`; then mark the version READY.
See the [S3 Delta Weight Refit guide](https://github.com/ai-dynamo/modelexpress/blob/main/docs/S3_DELTA_WEIGHT_REFIT.md)
for the publication sequence and generator artifact requirements.
vLLM's `modelexpress_engine.py` re-exports the engine and its init/update
dataclasses from
`modelexpress_rl.inference.engines.vllm.weight_transfer_engine`.
The implementation and configuration schema live in ModelExpress. Upgrading
the installed ModelExpress package and restarting workers picks up compatible
changes without a vLLM code change. The package must continue to implement
vLLM's `WeightTransferEngine` interface.
## Inference Side
!!! warning "Development version"
This backend currently uses unreleased ModelExpress features from `main`,
which may introduce breaking changes. These features will be included in
the official ModelExpress v0.8.0 release.
Install ModelExpress from the `main` branch on every worker:
```bash
uv pip install "git+https://github.com/ai-dynamo/modelexpress@main#subdirectory=modelexpress_client/python"
```
The backend is registered natively and loaded only when selected;
`VLLM_PLUGINS=modelexpress` is not required for weight transfer. If the plugin
is enabled, it preserves the native registration and uses the same engine
implementation.
```python
from vllm import LLM
from vllm.config import WeightTransferConfig
llm = LLM(model="my-model", weight_transfer_config=WeightTransferConfig(backend="modelexpress"))
```
```bash
vllm serve /models/launch \
--load-format safetensors \
--weight-transfer-config '{"backend":"modelexpress"}'
```
### Initialize the engine
Initialize once by passing `init_info` to `init_weight_transfer_engine`.
For HTTP administration, set `VLLM_SERVER_DEV_MODE=1` before starting vLLM and
restrict these endpoints to trusted orchestrators.
For S3 updates, send the following body to `POST /init_weight_transfer_engine`:
```json
{
"init_info": {
"model_name": "policy",
"server_url": "modelexpress:8001",
"object_storage_type": "S3",
"initial_base_version_id": "policy-v0",
"seed_checkpoint_path": "/models/launch",
"refit_checkpoint_dir": "/mxdelta/receiver",
"refit_checkpoint_max_size_gb": 200,
"object_storage_region_name": "us-west-2"
}
}
```
### `ModelExpressWeightTransferInitInfo`
These fields belong inside `init_info`, rather than
`--weight-transfer-config`. The installed ModelExpress package defines the
schema and defaults.
| Field | Default | Description |
| ----- | ------- | ----------- |
| `model_name` | vLLM model name | Logical model identity used for published ModelExpress versions. Set this when it differs from the vLLM model path or name. |
| `server_url` | ModelExpress configuration | ModelExpress server address, such as `modelexpress:8001`. If omitted, uses ModelExpress's environment settings, then `localhost:8001`. |
| `initial_serving_version_id` | `None` | Version already loaded by the inference worker. If omitted, ModelExpress uses the initial version supplied by its runtime, when available. |
| `registration_ttl_seconds` | Three MX heartbeat intervals | Worker registration lifetime in seconds, renewed by ModelExpress. |
| `lease_ttl_seconds` | Registration lifetime | Weight-version lease lifetime in seconds, renewed while held. |
| `max_transfer_attempts` | `3` | Maximum source-discovery and transfer attempts for one staged update. |
| `max_replay_chain_length` | `64` | Maximum number of payload revisions replayed to reach the requested version. |
| `rpc_timeout_seconds` | `30.0` | Deadline in seconds for each control-plane or manifest RPC. |
### Object storage settings
To enable object storage, provide `object_storage_type`,
`initial_base_version_id`, `seed_checkpoint_path`, and `refit_checkpoint_dir`
inside `init_info`. Endpoint and region settings also require these four fields.
| Field | Default | Description |
| ----- | ------- | ----------- |
| `object_storage_type` | `None` | Object storage provider. Use `"S3"`; the generator currently supports S3. |
| `initial_base_version_id` | `None` | Registered base version matching the seed checkpoint. Required for object storage. |
| `seed_checkpoint_path` | `None` | Local seed checkpoint path accessible to each worker. Required for object storage. |
| `refit_checkpoint_dir` | `None` | Local directory for cached checkpoints and delta replay. Required for object storage. |
| `refit_checkpoint_max_size_gb` | ModelExpress default | Checkpoint cache quota in decimal GB. The example sets an explicit 200 GB quota; it does not reserve disk space. |
| `object_storage_endpoint_url` | `None` | Optional endpoint override for an S3-compatible service. |
| `object_storage_region_name` | `None` | Optional S3 region, such as `us-west-2`. |
The seed checkpoint must match `initial_base_version_id`. Credentials use
ModelExpress's normal credential chain. Omit the object storage settings when
using only GPU peer transfer.
### Apply a weight update
After the target version is READY, pause generation and call
`POST /start_weight_update`. Then send the exact immutable version ID to
`POST /update_weights`:
```json
{"update_info": {"version_id": "policy-v1"}}
```
Call `POST /finish_weight_update` only after the update succeeds, then resume
inference. The staged handle is retained until finish or shutdown. Staging and
installation failures propagate; do not resume a worker with uncertain weights.
See the ModelExpress documentation for publication, source selection, object
storage, and deployment requirements.
+42
View File
@@ -75,6 +75,48 @@ These metrics are available via `--enable-mfu-metrics`:
--8<-- "gen:metrics-mfu"
## Custom Histogram Buckets
The core engine histograms ship with default bucket boundaries tuned for
typical serving workloads. The `--custom-histogram-buckets` option replaces
the boundaries of one or more *bucket families* — exactly the histograms
listed in the table below — with your own list; histograms owned by other
subsystems (for example, the NIXL connector metrics) are not affected. Use it,
for example, to track sub-300ms latency SLOs with the request-phase
histograms, whose smallest default boundary is 0.3s:
```bash
vllm serve Qwen/Qwen3-0.6B \
--custom-histogram-buckets '{"request_latency": [0.01, 0.05, 0.1, 0.25, 0.5, 1.0, 5.0, 30.0]}'
```
Each family key overrides a group of related histograms:
| Family key | Histograms |
| --- | --- |
| `request_latency` | `vllm:e2e_request_latency_seconds`, `vllm:request_queue_time_seconds`, `vllm:request_inference_time_seconds`, `vllm:request_prefill_time_seconds`, `vllm:request_decode_time_seconds` |
| `time_to_first_token` | `vllm:time_to_first_token_seconds` |
| `inter_token_latency` | `vllm:inter_token_latency_seconds`, `vllm:request_time_per_output_token_seconds` |
| `iteration_tokens` | `vllm:iteration_tokens_total` |
| `request_params_n` | `vllm:request_params_n` |
| `request_num_preemptions` | `vllm:request_num_preemptions` |
| `request_tokens` | `vllm:request_prompt_tokens`, `vllm:request_generation_tokens`, `vllm:request_max_num_generation_tokens`, `vllm:request_params_max_tokens`, `vllm:request_prefill_kv_computed_tokens` |
| `kv_cache_residency` | `vllm:kv_block_lifetime_seconds`, `vllm:kv_block_idle_before_evict_seconds`, `vllm:kv_block_reuse_gap_seconds` |
Bucket values must be positive, finite, and strictly increasing; unknown
family keys are rejected at startup. Families you do not list keep their
default boundaries. The `request_tokens` defaults normally scale with
`--max-model-len`; an override replaces that computed list. The
`kv_cache_residency` family only takes effect when `--kv-cache-metrics` is
enabled.
!!! warning "Bucket cardinality"
Every bucket boundary creates one extra time series per metric and per
label combination (model and engine index, multiplied under data-parallel
deployments). Long bucket lists inflate Prometheus storage, scrape sizes,
and query costs. Keep custom lists short, and only override the families
you actively monitor.
## Deprecation Policy
Note: when metrics are deprecated in version `X.Y`, they are hidden in version `X.Y+1`
+1 -1
View File
@@ -375,7 +375,7 @@ An attacker who can reach the gRPC port can:
1. **Run arbitrary inference** via the `Generate` and `GenerateStream` RPCs without any credentials
2. **Mutate engine state** by pausing generation, sleeping the engine, or initiating configured RL weight updates through the `Control` service
3. **Consume GPU and compute resources** by submitting unbounded generation requests
4. **Cause Denial of Service** by exploiting bugs in the gRPC interface that can crash vLLM.
4. **Stop a managed engine** through `Control.Shutdown`, or cause denial of service by exploiting bugs in the gRPC interface.
### Recommendations
@@ -57,7 +57,12 @@ The vllm instances and `disagg_encoder_proxy` supports local URIs with ```{"url"
## EC connector and KV transfer
The `ECExampleonnector` is used to store the encoder cache on local disk and facilitate transfer. To enable the encoder disaggregation feature, add the following configuration:
`ECExampleConnector` stores encoder cache on local disk. For RDMA/TCP transfer
without a shared filesystem, use `ECMooncakeConnector` (1E + 1PD only through
this proxy). See
[ECMooncakeConnector Usage Guide](../../../docs/features/mooncake_ec_connector_usage.md).
To enable filesystem encoder disaggregation, add:
```bash
# Add to encoder instance:
@@ -104,6 +109,8 @@ If you enable prefill instance (`--prefill-servers-urls` not disabled), you will
| `--encode-servers-urls` | Comma-separated list of encoder endpoints. Every multimodal item extracted from the request is fanned out to one of these URLs in a round-robin fashion. |
| `--prefill-servers-urls` | Comma-separated list of prefill endpoints. Set to `disable`, `none`, or `""` to skip the dedicated prefill phase and run E+PD (encoder + combined prefill/decode). |
| `--decode-servers-urls` | Comma-separated list of decode endpoints. Non-stream and stream paths both round-robin over this list. |
| `--ec-consumer-zmq-addrs` | Mooncake EC consumer control addresses, aligned with `--decode-servers-urls`. Required for `ECMooncakeConnector`. Supported only in E+PD mode (`--prefill-servers-urls disable`). |
| `--ec-consumer-dp-size` | Data-parallel replicas per EC consumer (default 1). |
| `--host`, `--port` | Bind address for the proxy itself (defaults: `0.0.0.0:8000`). |
### Dynamic registration
+53 -17
View File
@@ -6,6 +6,7 @@ robust and flexible configuration of the various loggers used by vLLM.
For `vllm serve`, configure logging with CLI arguments:
- Use the built-in configuration, optionally with `--log-level`.
- Emit structured JSON with `--logging-config.formatter json`.
- Use a custom Python logging configuration file with
`--logging-config.pylogging_config_file`.
- Disable vLLM logging configuration with
@@ -14,7 +15,7 @@ For `vllm serve`, configure logging with CLI arguments:
## CLI logging configuration
`--logging-config` accepts a JSON object. Its fields are `log_level`,
`configure_logging`, and `pylogging_config_file`. For example:
`formatter`, `configure_logging`, and `pylogging_config_file`. For example:
```bash
vllm serve mistralai/Mistral-7B-v0.1 \
@@ -28,6 +29,25 @@ vllm serve mistralai/Mistral-7B-v0.1 \
--logging-config.log_level DEBUG
```
The built-in `text` formatter is selected by default. To emit one JSON object
per vLLM and Uvicorn log record, select the built-in `json` formatter:
```bash
vllm serve mistralai/Mistral-7B-v0.1 \
--logging-config.formatter json
```
The built-in JSON output includes `asctime`, `levelname`, `name`,
`processName`, `process`, `message`, and `vllm_process_name`. `processName` is
Python's process name, `vllm_process_name` identifies vLLM's logical process
(including worker ranks where applicable), and `process` is the
operating-system PID.
The built-in profile formats records emitted through vLLM's Python loggers and
Uvicorn's server and access loggers when using `vllm serve`. It does not convert
direct writes to `stdout` or `stderr`. Use a custom configuration to format
other loggers.
`--log-level` is a shortcut for `--logging-config.log_level` and takes
precedence if both are supplied. It sets the level of vLLM's built-in logging
configuration. `--log-config-file` is deprecated and will be removed in
@@ -41,11 +61,11 @@ The custom logging configuration file must be JSON following Python's [logging
configuration dictionary
schema](https://docs.python.org/3/library/logging.config.html#dictionary-schema-details).
!!! note "Custom configurations override `--log-level`"
!!! note "Custom configurations override built-in formatter settings"
When `pylogging_config_file` is set, vLLM loads that JSON file and replaces
its built-in `dictConfig`; it does not merge the two. Therefore,
`--log-level` applies only when no custom configuration file is provided.
Set logger and handler levels in the custom file itself. vLLM applies the
its built-in `dictConfig`; it does not merge the two. Therefore, `log_level`
and `formatter` apply only when no custom configuration file is provided.
Set levels and formatters in the custom file itself. vLLM applies the
resolved configuration in its child processes as well.
## Environment variables
@@ -62,10 +82,11 @@ for those settings.
### Example 1: Customize vLLM root logger
For this example, we will customize the vLLM root logger to use
For more control over fields and handlers, configure the vLLM root logger and
Uvicorn's server and access loggers with
[`python-json-logger`](https://github.com/nhairs/python-json-logger)
(which is part of the container image) to log to
STDOUT of the console in JSON format with a log level of `INFO`.
(which is part of the container image) to log to STDOUT in JSON format at
`INFO` level.
To begin, first, create an appropriate JSON logging configuration file:
@@ -92,8 +113,24 @@ To begin, first, create an appropriate JSON logging configuration file:
"handlers": ["console"],
"level": "INFO",
"propagate": false
},
"uvicorn": {
"handlers": ["console"],
"level": "INFO",
"propagate": false
},
"uvicorn.error": {
"handlers": ["console"],
"level": "INFO",
"propagate": false
},
"uvicorn.access": {
"handlers": ["console"],
"level": "INFO",
"propagate": false
}
},
"disable_existing_loggers": false,
"version": 1
}
```
@@ -105,21 +142,20 @@ vllm serve mistralai/Mistral-7B-v0.1 --max-model-len 2048 \
--logging-config.pylogging_config_file /path/to/logging_config.json
```
Each vLLM log record is one JSON object and includes `vllm_process_name`, which
identifies vLLM's logical process (including worker ranks where applicable).
The standard `process` record attribute contains the operating-system PID.
vLLM avoids altering `stdout` or `stderr`, so no text is prepended to JSON
log records.
With this custom configuration, each vLLM and Uvicorn log record is one JSON
object. The example selects `vllm_process_name` and `process` explicitly; both
fields are also present in the built-in JSON output described above. vLLM
avoids altering `stdout` or `stderr`, so no text is prepended to JSON log
records.
This applies to records emitted through the configured Python loggers. A JSON
formatter cannot convert unrelated output into JSON, such as a third-party
library writing directly to `stdout` or `stderr`; configure or route such
output separately when a consumer requires every collected line to be JSON.
When serving, `VLLM_LOGGING_CONFIG_PATH` is also used as Uvicorn's logging
configuration. This example configures only `vllm`; configure `uvicorn`,
`uvicorn.error`, and `uvicorn.access` with a JSON handler when those server
logs must also be structured.
When serving, vLLM also passes this file to Uvicorn as its logging
configuration. The `uvicorn`, `uvicorn.error`, and `uvicorn.access` entries
format its server and access logs as JSON.
### Example 2: Silence a particular vLLM logger
@@ -28,6 +28,14 @@ A special case of wrapping a request-level logits processor where the processor
python examples/features/logits_processor/custom_req_init.py
```
### `dry.py` — DRY repetition penalty (Model Runner V2)
Implements the DRY (Don't Repeat Yourself) repetition penalty, ported from llama.cpp, as `DryState`, a custom logits processor for the Model Runner V2 interface. Requests turn it on through `SamplingParams.extra_args`, for example `{"dry_multiplier": 0.8}`. The example sends one prompt twice in one batch, once without DRY and once with it, and prints both outputs. `tests/v1/sample/test_dry.py` imports the processor from this file.
```bash
python examples/features/logits_processor/dry.py
```
## Key Concepts
- **Batch-level vs. request-level**: vLLM processes logits at the batch level for efficiency. If you have a per-request processor, you need to wrap it using the patterns shown in `custom_req.py` and `custom_req_init.py`.
+827
View File
@@ -0,0 +1,827 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""This example implements the DRY (Don't Repeat Yourself) repetition penalty as
a Model Runner V2 custom logits processor, ``DryState``.
A token that would extend a repeat of at least ``allowed_length`` tokens loses
``multiplier * base ** (repeat_length - allowed_length)`` from its logit, with the
exponent clamped as in llama.cpp.
Parameter names and matching semantics follow llama.cpp's
``llama_sampler_dry_apply``. llama.cpp's version is itself ported from pi6am's
koboldcpp implementation of p-e-w's scheme for text-generation-webui.
Pass the class to ``LLM`` and configure it per request through ``extra_args``::
LLM(..., logits_processors=[DryState])
SamplingParams(extra_args={"dry_multiplier": 0.8})
The recognized keys are ``dry_multiplier`` (0.0, off), ``dry_base`` (1.75),
``dry_allowed_length`` (2), ``dry_penalty_last_n`` (-1, the whole context)
and ``dry_sequence_breakers``. This processor claims the ``dry_*`` namespace,
so any other ``dry_*`` key fails the request. Speculative decoding is not
supported and is refused at construction.
Run the example with::
python examples/features/logits_processor/dry.py
It sends one prompt twice in one batch, once without DRY and once with it, and
prints both outputs, yielding an output similar to that shown below:
Generated Outputs:
------------------------------------------------------------
Prompt: 'The future of AI is', without DRY
Output: ' in the hands of the people.\n\nThe future of AI is in the hands of
the people.\n\nThe future of AI is in the hands of the
people.\n\nThe future of AI is in the hands of the people.\n\nThe
future of AI is in the hands of the people.\n'
------------------------------------------------------------
Prompt: 'The future of AI is', with DRY
Output: ' in the hands of the people.\n\nThe future of AI is the future of
the human race.\n\nThe future of AI is a future of the human race,
and the future of humanity.\n\nThe future of AI is an AI that is
capable of making decisions that are not based on human
intelligence.'
------------------------------------------------------------
"""
import math
import weakref
from typing import TYPE_CHECKING, Any, NamedTuple
import numpy as np
import torch
from vllm.logger import init_logger
from vllm.sampling_params import SamplingParams
from vllm.utils.gpu_sync_debug import gpu_sync_allowed
from vllm.utils.torch_utils import async_tensor_h2d
from vllm.v1.worker.gpu.sample.logits_processor import (
LogitsContext,
LogitsProcessor,
LogitsProcRequestState,
)
if TYPE_CHECKING:
from vllm.config import VllmConfig
logger = init_logger(__name__)
_J_BUDGET = 2048
# Run lengths feed an int16 sum, so they must stay below 2**15.
assert _J_BUDGET < 2**15, "_J_BUDGET must fit the int16 run-length sum"
# Byte budget for the per-chunk transients of the match scan. The gather
# W32[:, idx1] materializes [R, chunk, J] int32 (4 B/elem) before the
# comparison reduces it to bool; with the masks, the int8 cumprod and
# the int16 row sums, the marginal transient cost is ~8 B/elem. 24 keeps
# headroom for the fixed R*vocab penalty accumulator floor and allocator slack.
_CHUNK_BYTE_BUDGET = 256 * 1024 * 1024
_CHUNK_PEAK_BYTES_PER_ELEM = 24
# llama.cpp's FLOAT_MAX_LOG (src/llama-sampler.cpp): ln(float32 max).
_FLOAT_MAX_LOG = 88.7228391
# Breakers are truncated to this many code points. llama.cpp cuts at this many
# bytes (llama_sampler_init_dry), which keeps fewer characters of a non-ASCII
# breaker.
_MAX_BREAKER_CHAR_LEN = 40
# Upper bound on distinct breaker sets cached per tokenizer.
_MAX_CACHED_BREAKER_SETS = 64
STR_SPEC_DEC_REJECTS_DRY = (
"The DRY logits processor is not supported when speculative decoding is enabled."
)
DEFAULT_DRY_SEQUENCE_BREAKERS = ("\n", ":", '"', "*")
"""llama.cpp's default DRY sequence breakers."""
MAX_DRY_SEQUENCE_BREAKERS = 64
"""Upper bound on the dry_sequence_breakers list length. Each multi-character
breaker in an uncached set scans the vocabulary."""
_DRY_INT_MAX = 2**31 - 1
"""Upper bound on the integral DRY parameters, matching llama-server's
INT32_MAX cap. The state below stores them in an int64 numpy array, where
anything at or above 2**63 raises OverflowError inside execute_model and
takes the engine down."""
_MAX_CACHED_BREAKER_MASKS = 64
"""Upper bound on distinct breaker sets holding a device mask, mirroring
_MAX_CACHED_BREAKER_SETS on the host side."""
_WARMUP_WINDOW = 8
"""Window length of the dry_core call in DryState.__init__, long enough to
hold a match at the default dry_allowed_length."""
_DRY_DEFAULTS: dict[str, Any] = {
"dry_multiplier": 0.0,
"dry_base": 1.75,
"dry_allowed_length": 2,
"dry_penalty_last_n": -1, # whole context; llama.cpp's default is 64
"dry_sequence_breakers": DEFAULT_DRY_SEQUENCE_BREAKERS,
}
"""Recognized extra_args keys and their defaults."""
# ---------------------------------------------------------------------------
# The match computation
# ---------------------------------------------------------------------------
# ``_dry_penalties`` is a sequential port of llama.cpp's Z-algorithm, used as the
# reference. ``dry_core`` is the vectorized form: the comparisons that determine
# the match length ending at position ``i`` lie along one diagonal, so a
# cumprod-sum along ``j`` of an ``[R, K, J]`` tensor gives every match length
# without a sequential scan. Capping ``J`` at ``allowed_length + max_exponent``
# is exact, because the exponent clamp maps every longer match to the same
# penalty. Requests whose cap is unusable (``max_exponent == 0``, i.e.
# ``base <= 1.000001``, or a cap beyond ``_J_BUDGET``) use the sequential form.
def max_exponent(base: float) -> int:
"""Exponent clamp, mirroring llama.cpp bit-for-bit.
llama.cpp computes ``FLOAT_MAX_LOG / std::log(dry_base)`` entirely in
float32. Computing the quotient in float64 lands on the wrong side of
the integer truncation for some bases: at ``base=2.0`` float64 gives
127.99999998 -> 127 while float32 gives exactly 128.0 -> 128.
"""
if base <= 1.000001:
return 0
return int(np.float32(_FLOAT_MAX_LOG) / np.log(np.float32(base)))
def _dry_penalties(
window: list[int],
breakers: frozenset[int],
multiplier: float,
base: float,
allowed_length: int,
max_exp: int,
) -> dict[int, float]:
"""Compute DRY penalties for one request.
Port of the scan in ``llama_sampler_dry_apply``: a reverse-direction
Z-algorithm finds, for each position, the length of the match between
the window suffix and the sequence ending at that position; each
match's follower token is charged the penalty for the longest repeat
it would extend.
Returns a dict mapping token id -> penalty to subtract from its logit.
"""
m = len(window)
def rat(i: int) -> int: # reverse access: rat(0) == last token
return window[m - 1 - i]
# Step 1: the nearest breaker from the end caps the match length.
rep_limit = m
for i in range(m):
if rat(i) in breakers:
rep_limit = i
break
if rep_limit < allowed_length:
return {}
# Step 2: reverse-direction Z-algorithm -> per-position repeat counts
# (forward-indexed into ``window`` via cnt[last - k]).
cnt = [0] * m
last = m - 1
lt = rt = 0
for k in range(1, m):
if k > rt:
# Outside the current Z-box: extend naively.
z = 0
while z + k < m and rat(z) == rat(z + k):
z += 1
cnt[last - k] = min(z, rep_limit)
if z > 0:
lt, rt = k, k + z - 1
else:
p = k - lt
right_part_len = rt - k + 1
if cnt[last - p] < right_part_len:
# Fully inside the Z-box: copy.
cnt[last - k] = min(cnt[last - p], rep_limit)
else:
# Touches the right edge: extend past it.
j = rt + 1
while j < m and rat(j) == rat(j - k):
j += 1
cnt[last - k] = min(j - k, rep_limit)
lt, rt = k, j - 1
# Step 3: map each repeat's follower token to the longest repeat that
# it would extend.
max_token_repeat: dict[int, int] = {}
for i in range(m - 1):
repeat_len = cnt[i]
if repeat_len >= allowed_length:
tok = window[i + 1]
if max_token_repeat.get(tok, -1) < repeat_len:
max_token_repeat[tok] = repeat_len
# Step 4: exponential penalty, exponent clamped for float32 safety.
# Breaker tokens are never penalized. A value overflowing float32
# saturates the logit to -inf downstream, as in llama.cpp.
penalties: dict[int, float] = {}
for tok, repeat_len in max_token_repeat.items():
if tok in breakers:
continue
exponent = repeat_len - allowed_length
if max_exp and exponent > max_exp:
exponent = max_exp
penalties[tok] = multiplier * (base**exponent)
return penalties
def dry_core(
logits: torch.Tensor,
row_idx: torch.Tensor,
W: torch.Tensor,
n_r: torch.Tensor,
allowed: torch.Tensor,
max_exp: torch.Tensor,
mult: torch.Tensor,
base: torch.Tensor,
breaker_masks: list[torch.Tensor | None],
j_budget: int,
) -> torch.Tensor:
"""Apply DRY penalties in place given batched window tensors.
The tensor bounds below are not checked, to avoid a sync.
Args:
logits: [B, vocab] float tensor, modified in place.
row_idx: [R] int64, row of ``logits`` for each DRY request.
W: [R, N] int64 windows, right-aligned (window tokens occupy the
trailing ``n_r`` columns; leading columns are ignored via ``n_r``
masks, whatever they hold).
n_r: [R] int64 window lengths.
allowed: [R] int64 per-request allowed length.
max_exp: [R] int64 per-request exponent ceiling, > 0.
mult: [R] float32 per-request multiplier, >= 0, already rounded
through float32, as llama.cpp stores it.
base: [R] float32 per-request base, >= 1, rounded the same way.
breaker_masks: per-request [vocab] bool masks (or None for no
breakers). ``vocab`` must equal ``logits.shape[-1]``.
j_budget: max(allowed + max_exp), <= _J_BUDGET, computed by the caller on
the host.
"""
device = logits.device
vocab = logits.shape[-1]
R, N = W.shape
# j_budget arrives as an unchecked host int; this check fails loudly rather
# than wrapping an int16 run length. The routing predicate in DryState.apply
# is what actually bounds it.
if j_budget > _J_BUDGET:
raise ValueError(f"j_budget {j_budget} over _J_BUDGET {_J_BUDGET}")
# base >= 1 and mult >= 0 are preconditions too (see the amax comment below).
# They live in device tensors and reading them back would sync; use_dry()
# and validate_params enforce them instead.
# rep_limit: distance from the end of the nearest breaker (llama.cpp step 1).
# A breaker at column c is j = N-1-c tokens from the end.
rep_limit = n_r.clone()
# bool(bm.any()) would be a per-request host sync; the None check says the same.
any_breakers = any(bm is not None for bm in breaker_masks)
bmask = None
if any_breakers:
# Stacked once, used twice: nearest-breaker search here, penalty zeroing
# after the scan. The masks are cached and resident (DryState._breaker_masks),
# so the stack costs one byte per (row, vocab) entry.
bmask = torch.stack(
[
bm
if bm is not None
else torch.zeros(vocab, dtype=torch.bool, device=device)
for bm in breaker_masks
]
)
Bwin = bmask.gather(1, W.clamp(min=0))
valid_cols = torch.arange(N, device=device)[None, :] >= (N - n_r)[:, None]
Bwin &= valid_cols
has_breaker = Bwin.any(dim=1)
# max breaker column -> nearest to the end.
max_col = torch.where(
has_breaker,
(Bwin * torch.arange(1, N + 1, device=device)[None, :]).max(dim=1).values
- 1,
torch.zeros_like(n_r),
)
rep_limit = torch.where(has_breaker, (N - 1) - max_col, n_r)
# llama.cpp: if rep_limit < allowed_length, the request produces nothing.
active = rep_limit >= allowed
# Token ids fit int32, and gathering int32 halves the dominant per-chunk transient.
W32 = W.to(torch.int32)
# J is a tensor shape, so it must be known host-side; the caller computes it
# from numpy (see DryState.apply) to avoid a per-step device readback.
J = max(1, min(j_budget, N))
idx2 = torch.arange(N - 1, N - 1 - J, -1, device=device) # [J]
suffix = W32.gather(1, idx2.expand(R, J)) # [R, J]
K = N - 1 # offsets 1..N-1
chunk = max(1, _CHUNK_BYTE_BUDGET // (_CHUNK_PEAK_BYTES_PER_ELEM * max(1, R * J)))
# amax keeps one penalty per token, for its longest match, however the offsets
# are chunked, as long as the penalty does not fall as the match grows
# (base >= 1, mult >= 0).
pen = torch.zeros(R * vocab + 1, dtype=torch.float32, device=device)
for k0 in range(1, K + 1, chunk):
k1 = min(k0 + chunk, K + 1)
ks = torch.arange(k0, k1, device=device) # [C]
C = ks.shape[0]
# idx1[c, j] = N-1-j-k ; invalid (out of window) entries masked.
idx1 = idx2[None, :] - ks[:, None] # [C, J]
invalid = idx1 < (N - n_r)[:, None, None] # [R, C, J]
eq = W32[:, idx1.clamp(min=0)] == suffix[:, None, :] # [R, C, J]
eq &= ~invalid
del invalid
# Run length of leading True along j = the match length. Explicit dtypes avoid
# int64 intermediate copies; values are 0/1 and runs are <= J <= _J_BUDGET,
# so int8/int16 are exact.
L = eq.cumprod(dim=2, dtype=torch.int8).sum(dim=2, dtype=torch.int16)
del eq
# Do not narrow rep_limit to L's int16; window lengths can exceed it.
L = torch.minimum(L, rep_limit[:, None])
# Follower token of offset k lives at column N-k; the offset counts only
# while position i = N-1-k is inside the window (k <= n_r - 1).
valid_k = ks[None, :] <= (n_r - 1)[:, None] # [R, C]
charge = (allowed[:, None] <= L) & valid_k & active[:, None]
# No `if charge.any()` and no boolean indexing: both sync the host. Uncharged
# entries scatter to a trash slot one past the accumulator end, never read.
followers = W.gather(1, (N - ks).clamp(max=N - 1).expand(R, C))
rows = torch.arange(R, device=device)[:, None].expand(R, C) * vocab
flat = rows + followers.clamp(min=0)
# A Python int, not a device tensor: torch.tensor(x, device=...) here would be a
# pageable host-to-device copy, which is itself a synchronization.
flat = torch.where(charge, flat, R * vocab)
# L takes rep_limit's dtype from the minimum above. The cast keeps the
# exponent int64 whatever dtype n_r, allowed and max_exp arrive in.
exponent = torch.minimum(L.to(torch.int64) - allowed[:, None], max_exp[:, None])
# float64 pow, as llama.cpp's std::pow(float, int); a float32 pow overflows
# early (0.8 * 2**128 is finite in float32).
p_chunk = mult[:, None].double() * torch.pow(
base[:, None].double(), exponent.to(torch.float64)
)
# No where() on p_chunk: uncharged entries went to the trash slot and are
# never read.
pen.scatter_reduce_(
0,
flat.reshape(-1),
p_chunk.to(torch.float32).reshape(-1),
reduce="amax",
include_self=True,
)
# Of the penalty terms, only the penalty itself is held at [R, vocab] (4 B/entry);
# the exponent, its float64 cast, the pow result and the product stay at [R, C]
# inside the loop.
# Breakers zero the penalty rather than the logit, so a breaker keeps its value.
# The narrowing to logits.dtype below is a no-op under vLLM's sampler, which
# always passes float32 logits.
pen2 = pen[:-1].view(R, vocab)
if bmask is not None:
pen2.masked_fill_(bmask, 0.0)
# index_add_ avoids the extra [R, vocab] gather+copy of `logits[row_idx] -= pen2`.
pen2.neg_()
logits.index_add_(
0, row_idx, pen2 if logits.dtype == pen2.dtype else pen2.to(logits.dtype)
)
return logits
# ---------------------------------------------------------------------------
# Breaker resolution
# ---------------------------------------------------------------------------
class _VocabIndex(NamedTuple):
"""A tokenizer's decoded vocabulary, plus a character index into it.
``char_ids`` maps every character in ``texts`` to the ids of the tokens
whose text contains it.
"""
texts: list[str]
char_ids: dict[str, list[int]]
# Per-tokenizer caches, weakly keyed so tokenizers can be collected:
# tokenizer -> its decoded vocabulary and character index, and
# tokenizer -> {breaker string tuple -> resolved breaker ids}.
_BreakerIdsPerTokenizer = dict[tuple[str, ...], list[int]]
_vocab_index_cache: "weakref.WeakKeyDictionary[Any, _VocabIndex]" = (
weakref.WeakKeyDictionary()
)
_breaker_ids_cache: "weakref.WeakKeyDictionary[Any, _BreakerIdsPerTokenizer]" = (
weakref.WeakKeyDictionary()
)
def _vocab_index(tokenizer: Any) -> _VocabIndex:
"""Decode every token of ``tokenizer`` once and index its characters."""
index = _vocab_index_cache.get(tokenizer)
if index is None:
# Include added tokens above vocab_size, which llama.cpp's scan also covers.
n_ids = getattr(tokenizer, "max_token_id", tokenizer.vocab_size - 1) + 1
texts = tokenizer.batch_decode([[i] for i in range(n_ids)])
char_ids: dict[str, list[int]] = {}
for i, text in enumerate(texts):
for char in set(text):
char_ids.setdefault(char, []).append(i)
index = _VocabIndex(texts, char_ids)
_vocab_index_cache[tokenizer] = index
return index
def resolve_dry_breakers(tokenizer: Any, breaker_strs: tuple[str, ...]) -> list[int]:
"""Resolve breaker strings to the ids of every token whose text contains one.
This is llama.cpp's ``get_overlapping_token_sequences`` rule, without its
multi-token restart sequences. Resolution runs in ``add_request``, inside
``execute_model``, so a single-character breaker (llama.cpp's defaults and
most custom sets) resolves by lookup in the character index instead of a
vocabulary scan. Results are cached per (tokenizer, breaker set).
"""
breaker_strs = tuple(s[:_MAX_BREAKER_CHAR_LEN] for s in breaker_strs if s)
if not breaker_strs:
return []
per_tok = _breaker_ids_cache.setdefault(tokenizer, {})
cached = per_tok.get(breaker_strs)
if cached is not None:
return list(cached)
index = _vocab_index(tokenizer)
ids: set[int] = set()
for s in breaker_strs:
if len(s) == 1:
ids.update(index.char_ids.get(s, ()))
else:
ids.update(i for i, text in enumerate(index.texts) if s in text)
result = sorted(ids)
if len(per_tok) >= _MAX_CACHED_BREAKER_SETS:
# Evict the oldest entry (dict preserves insertion order).
per_tok.pop(next(iter(per_tok)))
per_tok[breaker_strs] = result
return list(result)
# ---------------------------------------------------------------------------
# The logits processor
# ---------------------------------------------------------------------------
def _dry_args(sampling_params: SamplingParams) -> dict[str, Any]:
"""Read the request's DRY arguments, filling in the defaults.
A key present with value None counts as unset, matching how
``SamplingParams`` coerces its own None-valued arguments.
"""
extra = sampling_params.extra_args or {}
return {
name: default if extra.get(name) is None else extra[name]
for name, default in _DRY_DEFAULTS.items()
}
def use_dry(multiplier: float, base: float, penalty_last_n: int) -> bool:
"""Whether DRY applies, by llama.cpp's gate in llama_sampler_dry_apply."""
return bool(multiplier) and base >= 1.0 and penalty_last_n != 0
class DryState(LogitsProcessor):
def __init__(self, vllm_config: "VllmConfig", req_states: LogitsProcRequestState):
# Refuse speculative decoding here, since admission cannot see its config.
if vllm_config.speculative_config is not None:
raise ValueError(STR_SPEC_DEC_REJECTS_DRY)
self.req_states = req_states
max_num_reqs = req_states.max_num_reqs
self.vocab_size = req_states.vocab_size
self.device = req_states.device
# float32, to round multiplier and base as llama.cpp's float members do.
self.multiplier = np.zeros(max_num_reqs, dtype=np.float32)
self.base = np.zeros(max_num_reqs, dtype=np.float32)
self.allowed_length = np.zeros(max_num_reqs, dtype=np.int64)
self.penalty_last_n = np.zeros(max_num_reqs, dtype=np.int64)
self.max_exponent = np.zeros(max_num_reqs, dtype=np.int64)
self.use_dry = np.zeros(max_num_reqs, dtype=bool)
# req_idx -> its breaker set, and breaker set -> [vocab] bool device
# mask shared by every request that asked for the same breakers.
self.breaker_ids: dict[int, frozenset[int]] = {}
self._breaker_masks: dict[frozenset[int], torch.Tensor] = {}
self._warned_unresolved = False
# Deferred import: the tokenizer registry pulls in transformers, and
# the frontend imports this module only to validate params.
from vllm.tokenizers import cached_tokenizer_from_config
# None under skip_tokenizer_init.
self._tokenizer = cached_tokenizer_from_config(vllm_config.model_config)
# Resolve the default set here to keep the vocabulary decode out of
# execute_model, where add_request runs.
self._default_breaker_ids = self._resolve_breakers(
DEFAULT_DRY_SEQUENCE_BREAKERS
)
self._warm_up_dry_core()
def _warm_up_dry_core(self) -> None:
"""Load dry_core's CUDA kernels before a request needs them.
The operands match the dtypes and ranks apply passes, and include a breaker
mask so that the breaker path's kernels load too.
"""
vocab = self.vocab_size
# The breaker takes the last id, which must not be the window's token 0.
if vocab < 2:
return
device = self.device
allowed = _DRY_DEFAULTS["dry_allowed_length"]
base = _DRY_DEFAULTS["dry_base"]
max_exp = max_exponent(base)
def col(value: float, dtype: torch.dtype) -> torch.Tensor:
return torch.full((1,), value, dtype=dtype, device=device)
breakers = torch.zeros(vocab, dtype=torch.bool, device=device)
breakers.narrow(0, vocab - 1, 1).fill_(True)
dry_core(
torch.zeros(1, vocab, dtype=torch.float32, device=device),
row_idx=col(0, torch.int64),
W=torch.zeros(1, _WARMUP_WINDOW, dtype=torch.int64, device=device),
n_r=col(_WARMUP_WINDOW, torch.int64),
allowed=col(allowed, torch.int64),
max_exp=col(max_exp, torch.int64),
mult=col(1.0, torch.float32),
base=col(base, torch.float32),
breaker_masks=[breakers],
j_budget=allowed + max_exp,
)
@classmethod
def validate_params(cls, sampling_params: SamplingParams) -> None:
"""Check the ``dry_*`` keys of ``extra_args`` at request admission.
Raises:
ValueError: on an unknown or out-of-range DRY argument.
"""
extra = sampling_params.extra_args or {}
unknown = sorted(
key for key in extra if key.startswith("dry_") and key not in _DRY_DEFAULTS
)
if unknown:
# A misspelled key would otherwise be ignored silently.
raise ValueError(
f"Unknown dry_* extra_args: {', '.join(unknown)}. "
f"Supported keys: {', '.join(_DRY_DEFAULTS)}."
)
args = _dry_args(sampling_params)
for name in (
"dry_multiplier",
"dry_base",
"dry_allowed_length",
"dry_penalty_last_n",
):
value = args[name]
# JSON true/false are bools, which pass isinstance(value, int).
if isinstance(value, bool) or not isinstance(value, (int, float)):
raise ValueError(
f"{name} must be a number, got {type(value).__name__}."
)
multiplier = args["dry_multiplier"]
base = args["dry_base"]
allowed_length = args["dry_allowed_length"]
penalty_last_n = args["dry_penalty_last_n"]
breakers = args["dry_sequence_breakers"]
if not math.isfinite(multiplier) or multiplier < 0.0:
raise ValueError(
f"dry_multiplier must be non-negative and finite, got {multiplier}."
)
if not math.isfinite(base) or base < 0.0:
raise ValueError(f"dry_base must be non-negative and finite, got {base}.")
if (
not isinstance(allowed_length, int)
or allowed_length < 0
or allowed_length > _DRY_INT_MAX
):
raise ValueError(
f"dry_allowed_length must be an integer in [0, {_DRY_INT_MAX}], "
f"got {allowed_length}."
)
if (
not isinstance(penalty_last_n, int)
or penalty_last_n < -1
or penalty_last_n > _DRY_INT_MAX
):
raise ValueError(
"dry_penalty_last_n must be an integer: -1 (whole context), "
f"0 (disable), or in [1, {_DRY_INT_MAX}], got {penalty_last_n}."
)
if not isinstance(breakers, (list, tuple)) or any(
not isinstance(s, str) for s in breakers
):
raise ValueError(
f"dry_sequence_breakers must be a list of strings, got {breakers!r}."
)
if len(breakers) > MAX_DRY_SEQUENCE_BREAKERS:
raise ValueError(
f"dry_sequence_breakers supports at most "
f"{MAX_DRY_SEQUENCE_BREAKERS} entries, got {len(breakers)}."
)
if multiplier and 0.0 <= base < 1.0:
# libllama has the same gate. llama-server resets such a base to its
# default instead.
logger.warning(
"dry_base=%s is below 1.0, which disables DRY entirely "
"(llama.cpp semantics), even though dry_multiplier=%s was "
"set. No repetition penalty will be applied.",
base,
multiplier,
)
def _resolve_breakers(self, breakers: tuple[str, ...]) -> frozenset[int]:
if self._tokenizer is None or not breakers:
return frozenset()
return frozenset(resolve_dry_breakers(self._tokenizer, breakers))
def add_request(self, req_idx: int, sampling_params: SamplingParams) -> bool:
args = _dry_args(sampling_params)
multiplier = args["dry_multiplier"]
base = args["dry_base"]
penalty_last_n = args["dry_penalty_last_n"]
enabled = use_dry(multiplier, base, penalty_last_n)
self.use_dry[req_idx] = enabled
self.breaker_ids.pop(req_idx, None)
if not enabled:
return False
self.multiplier[req_idx] = multiplier
self.base[req_idx] = base
self.allowed_length[req_idx] = args["dry_allowed_length"]
self.penalty_last_n[req_idx] = penalty_last_n
self.max_exponent[req_idx] = max_exponent(float(self.base[req_idx]))
breakers = tuple(args["dry_sequence_breakers"])
ids = (
self._default_breaker_ids
if breakers == DEFAULT_DRY_SEQUENCE_BREAKERS
else self._resolve_breakers(breakers)
)
if ids:
self.breaker_ids[req_idx] = ids
elif breakers and self._tokenizer is None and not self._warned_unresolved:
logger.warning(
"DRY sequence breakers were not resolved to token ids: this "
"engine has no tokenizer. Proceeding without breakers."
)
self._warned_unresolved = True
return True
def _breaker_mask(self, req_idx: int) -> torch.Tensor | None:
ids = self.breaker_ids.get(req_idx)
if not ids:
return None
mask = self._breaker_masks.get(ids)
if mask is None:
# Built on the host to avoid a sync; once per breaker set.
ids_np = np.fromiter(ids, dtype=np.int64, count=len(ids))
ids_np = ids_np[ids_np < self.vocab_size]
m_np = np.zeros(self.vocab_size, dtype=bool)
m_np[ids_np] = True
mask = async_tensor_h2d(m_np, self.device)
if len(self._breaker_masks) >= _MAX_CACHED_BREAKER_MASKS:
# Evict the oldest set. A client can send a new set with every request.
self._breaker_masks.pop(next(iter(self._breaker_masks)))
self._breaker_masks[ids] = mask
return mask
def apply(self, logits: torch.Tensor, ctx: LogitsContext) -> torch.Tensor:
req_indices = ctx.idx_mapping_np
active_rows = np.flatnonzero(self.use_dry[req_indices])
if active_rows.size == 0:
return logits
if logits.shape[0] != req_indices.shape[0]:
raise RuntimeError("DRY received draft-expanded logits")
# host-side bound; reading positions off GPU would sync per step
cur_len = ctx.seq_lens_upper_bound_np[active_rows].astype(np.int64)
reqs = req_indices[active_rows]
last_n = self.penalty_last_n[reqs]
window_len = np.where(last_n == -1, cur_len, np.minimum(cur_len, last_n))
allowed = self.allowed_length[reqs]
keep = window_len > allowed
if not np.any(keep):
return logits
active_rows = active_rows[keep]
reqs = reqs[keep]
cur_len = cur_len[keep]
window_len = window_len[keep]
allowed = allowed[keep]
max_exp = self.max_exponent[reqs]
# Route degenerate-clamp requests (base <= 1.000001 or oversized
# cap) through the sequential reference implementation.
fast = (max_exp > 0) & (allowed + max_exp <= _J_BUDGET)
all_tokens = self.req_states.all_token_ids.gpu
if np.any(fast):
f_rows = active_rows[fast]
f_reqs = reqs[fast]
f_len = window_len[fast]
N = int(f_len.max())
reqs_t = async_tensor_h2d(f_reqs, self.device)
cur_t = async_tensor_h2d(cur_len[fast], self.device)
j = torch.arange(N, device=self.device)
# Right-aligned gather: column j holds token (cur_len - N + j);
# out-of-window columns are masked inside dry_core via n_r.
gather_idx = (cur_t[:, None] - N + j[None, :]).clamp(min=0)
W = all_tokens[reqs_t[:, None], gather_idx].long()
dry_core(
logits,
row_idx=async_tensor_h2d(f_rows, self.device),
W=W,
n_r=async_tensor_h2d(f_len, self.device),
allowed=async_tensor_h2d(allowed[fast], self.device),
max_exp=async_tensor_h2d(max_exp[fast], self.device),
mult=async_tensor_h2d(self.multiplier[f_reqs], self.device),
base=async_tensor_h2d(self.base[f_reqs], self.device),
breaker_masks=[self._breaker_mask(r) for r in f_reqs],
j_budget=int((allowed[fast] + max_exp[fast]).max()),
)
# The sequential fallback copies each window to the host, an expected sync.
slow = ~fast
if np.any(slow):
with gpu_sync_allowed():
rows_list = []
cols_list = []
vals_list = []
for row, req, w_len, cur in zip(
active_rows[slow], reqs[slow], window_len[slow], cur_len[slow]
):
window = (
all_tokens[int(req), int(cur) - int(w_len) : int(cur)]
.cpu()
.tolist()
)
penalties = _dry_penalties(
window,
self.breaker_ids.get(int(req), frozenset()),
float(self.multiplier[req]),
float(self.base[req]),
int(self.allowed_length[req]),
int(self.max_exponent[req]),
)
for tok, val in penalties.items():
rows_list.append(int(row))
cols_list.append(tok)
vals_list.append(val)
if rows_list:
logits[
torch.tensor(rows_list, dtype=torch.int64, device=self.device),
torch.tensor(cols_list, dtype=torch.int64, device=self.device),
] -= torch.tensor(
vals_list, dtype=torch.float32, device=self.device
)
return logits
# ---------------------------------------------------------------------------
# The demo
# ---------------------------------------------------------------------------
def main():
# Imported here so that loading DryState from this module does not import LLM.
from vllm import LLM
prompts = ["The future of AI is"] * 2
sampling_params_list = [
SamplingParams(temperature=0.0, max_tokens=64),
SamplingParams(
temperature=0.0, max_tokens=64, extra_args={"dry_multiplier": 0.8}
),
]
llm = LLM(model="facebook/opt-125m", logits_processors=[DryState])
outputs = llm.generate(prompts, sampling_params_list)
print("\nGenerated Outputs:\n" + "-" * 60)
for label, output in zip(("without DRY", "with DRY"), outputs):
print(f"Prompt: {output.prompt!r}, {label}")
print(f"Output: {output.outputs[0].text!r}")
print("-" * 60)
if __name__ == "__main__":
main()
@@ -36,10 +36,9 @@ layer when available, otherwise Triton. FlashInfer cannot serve this model (a
batch mixes causal prefill with bidirectional denoising), and
`--attention-backend FLASHINFER` is rejected.
Per-request canvas widths smaller than the served canvas require async
scheduling. The diffusion async scheduler is selected automatically; no
`--scheduler-cls` argument is needed. Synchronous execution supports full-width
canvases only.
Per-request canvas widths may be smaller than the served canvas with either
synchronous or asynchronous scheduling. Omit `diffusion_canvas_length` to use
the served canvas width.
Question types: `noul` (yes/no), `choice` with `options`, `score` with
ordered `levels`. Each label must be a single token in the answer template,
-10
View File
@@ -65,13 +65,6 @@ plugins:
- awesome-nav
- glightbox
- git-revision-date-localized
- minify:
minify_html: true
minify_js: true
minify_css: true
cache_safe: true
js_files: [docs/mkdocs/javascript/*.js]
css_files: [docs/mkdocs/stylesheets/*.css]
# For API reference generation
- api-autonav:
modules: ["vllm"]
@@ -98,9 +91,6 @@ plugins:
summary: true
inventories:
- https://docs.python.org/3/objects.inv
- https://typing-extensions.readthedocs.io/en/latest/objects.inv
- https://docs.aiohttp.org/en/stable/objects.inv
- https://pillow.readthedocs.io/en/stable/objects.inv
- https://numpy.org/doc/stable/objects.inv
- https://pytorch.org/docs/stable/objects.inv
- redirects:
+5
View File
@@ -117,6 +117,11 @@ ignore_missing_imports = true
check_untyped_defs = true
follow_imports = "silent"
[[tool.mypy.overrides]]
module = "tests.*"
disable_error_code = ["arg-type", "assignment"]
enable_error_code = ["method-assign"]
[tool.pytest.ini_options]
markers = [
"slow_test",
+1 -1
View File
@@ -1,7 +1,7 @@
cmake>=3.26.1
ninja
packaging>=24.2
setuptools==77.0.3 # this version can reuse CMake build dir
setuptools==78.1.1 # this version can reuse CMake build dir; >=78.1.1 for GHSA-5rjg-fvgr-3xxf
setuptools-scm>=8
setuptools-rust>=1.9.0
torch==2.13.0+cpu; platform_machine == "x86_64" or platform_machine == "s390x" or platform_machine == "aarch64"
+1 -1
View File
@@ -2,7 +2,7 @@
cmake>=3.26.1
ninja
packaging>=24.2
setuptools>=77.0.3,<81.0.0
setuptools>=78.1.1,<81.0.0
setuptools-scm>=8
setuptools-rust>=1.9.0
torch==2.13.0
+1 -1
View File
@@ -9,7 +9,7 @@ torchaudio==2.11.0+rocm7.14.0
triton==3.7.1+git0263a6a6.rocm7.14.0
cmake>=3.26.1,<4
packaging>=26.2
setuptools>=77.0.3,<80.0.0
setuptools>=78.1.1,<80.0.0
setuptools-scm>=8
setuptools-rust>=1.9.0
wheel
+1 -1
View File
@@ -9,7 +9,7 @@ torchaudio==2.11.0+rocm7.14.0
triton==3.7.1+git0263a6a6
cmake>=3.26.1,<4
packaging>=26.2
setuptools>=77.0.3,<80.0.0
setuptools>=78.1.1,<80.0.0
setuptools-scm>=8
setuptools-rust>=1.9.0
wheel
+1 -1
View File
@@ -1,5 +1,5 @@
# Dependencies for building Rust artifacts through setuptools-rust.
setuptools>=77.0.3,<81.0.0
setuptools>=78.1.1,<81.0.0
setuptools-scm>=9.2.0
setuptools-rust>=1.9.0
wheel
+1 -1
View File
@@ -1,7 +1,7 @@
--extra-index-url https://download.pytorch.org/whl/cpu
cmake>=3.26.1
ninja
setuptools>=77.0.3,<81.0.0
setuptools>=78.1.1,<81.0.0
setuptools-scm>=8
setuptools-rust>=1.9.0
torch==2.13.0+cpu
+8 -8
View File
@@ -3,32 +3,32 @@ cachetools
psutil
sentencepiece # Required for LLaMA tokenizer.
numpy
requests >= 2.26.0
requests >= 2.33.0
tqdm
blake3
py-cpuinfo
transformers >= 5.10.4
transformers >= 5.16.1, < 5.18.0
huggingface_hub >= 1.31.0 # Required for the `huggingface_hub.utils.httpx` re-export
tokenizers >= 0.21.1 # Required for fast incremental detokenization.
tokenizers >= 0.22.0 # Required for fast incremental detokenization.
safetensors >= 0.6.2 # MXFP4/MXFP6 dtype support (F8_E8M0, F4) added in 0.6.0: https://github.com/huggingface/safetensors/pull/611
protobuf >= 5.29.6, !=6.30.*, !=6.31.*, !=6.32.*, !=6.33.0.*, !=6.33.1.*, !=6.33.2.*, !=6.33.3.*, !=6.33.4.*; platform_machine != "s390x"
protobuf == 7.36.1; platform_machine == "s390x"
fastapi[standard] >= 0.133.0, < 0.137.0 # First version supporting Starlette 1.0; < 0.137.0 avoids route-tree change that breaks model-hosting-container-standards handler overrides.
starlette >= 1.0.1 # CVE-2026-48710: Host header injection in < 1.0.1
aiohttp >= 3.13.3
aiohttp >= 3.14.3
openai >= 2.25.0 # For Responses API with namespace tools
pydantic >= 2.12.0
prometheus_client >= 0.18.0
pillow # Required for image processing
pillow >= 12.3.0 # Required for image processing
prometheus-fastapi-instrumentator >= 8.0.0 # v8 unblocks starlette >= 1.0
tiktoken >= 0.6.0 # Required for DBRX tokenizer
lm-format-enforcer == 0.11.3
llguidance >= 1.7.0, < 1.8.0; platform_machine == "x86_64" or platform_machine == "arm64" or platform_machine == "aarch64" or platform_machine == "ppc64le" or platform_machine == "s390x"
outlines_core == 0.2.14
lark == 1.2.2
xgrammar == 0.2.7; platform_machine == "x86_64" or platform_machine == "aarch64" or platform_machine == "arm64" or platform_machine == "s390x" or platform_machine == "ppc64le"
xgrammar == 0.2.8; platform_machine == "x86_64" or platform_machine == "aarch64" or platform_machine == "arm64" or platform_machine == "s390x" or platform_machine == "ppc64le"
typing_extensions >= 4.10
filelock >= 3.16.1 # need to contain https://github.com/tox-dev/filelock/pull/317
filelock >= 3.20.3 # need to contain https://github.com/tox-dev/filelock/pull/317; GHSA floor
partial-json-parser # used for parsing partial JSON outputs
jsonschema >= 4.23.0 # required for MiniMax M3 tool schema validation
pyzmq >= 25.0.0
@@ -37,7 +37,7 @@ mistral_common[image] >= 1.11.6
opencv-python-headless >= 4.13.0 # required for video IO
pyyaml
six>=1.16.0; python_version > '3.11' # transitive dependency of pandas that needs to be the latest version for python 3.12
setuptools>=77.0.3,<81.0.0; python_version > '3.11' # Setuptools is used by triton, we need to ensure a modern version is installed for 3.12+ so that it does not try to import distutils, which was removed in 3.12
setuptools>=78.1.1,<81.0.0; python_version > '3.11' # used by triton; >=78.1.1 for GHSA-5rjg-fvgr-3xxf (83.x needed for GHSA-h35f-9h28-mq5c deferred)
einops # Required for Qwen2-VL.
compressed-tensors == 0.17.0 # required for compressed-tensors
depyf==0.20.0 # required for profiling and debugging with compilation config

Some files were not shown because too many files have changed in this diff Show More