Merge remote-tracking branch 'upstream/main' into patches
Signed-off-by: MOVZX <movzx@yahoo.com>
This commit is contained in:
@@ -0,0 +1,112 @@
|
||||
---
|
||||
name: check-api-compat
|
||||
description: Determine the correct behavior of vLLM's OpenAI-compatible or Anthropic-compatible API by checking the official upstream spec, instead of guessing from vLLM's own code. Use for any bug, issue, PR, review, or question touching /v1/chat/completions, /v1/completions, /v1/responses, /v1/embeddings, /v1/messages, or their request/response models — including "is this a bug or intended", "what should this field default to", "does upstream allow this", and any change that adds, removes, or renames a field on those models. Do not use for vLLM-internal APIs (engine, scheduler, LLM class) or for endpoints with no upstream counterpart.
|
||||
---
|
||||
|
||||
# Checking vLLM API compatibility against the upstream spec
|
||||
|
||||
vLLM's OpenAI and Anthropic server surfaces are *reimplementations* of someone
|
||||
else's spec. vLLM's own code, tests, and docstrings are not evidence of correct
|
||||
behavior — they are the thing under test. Establish what upstream says first,
|
||||
then compare.
|
||||
|
||||
Never answer "what should this field do" from memory. Field names, types, and
|
||||
especially defaults drift between spec versions.
|
||||
|
||||
## 1. Pin down the question
|
||||
|
||||
Name three things before looking anything up:
|
||||
|
||||
- **Which surface** — OpenAI-compatible or Anthropic-compatible. They are separate
|
||||
code paths in vLLM and separate specs.
|
||||
- **Which endpoint**, by path.
|
||||
- **Which field or behavior** — a request field, a response field, a streaming
|
||||
event shape, an error code, or a status code.
|
||||
|
||||
## 2. Read vLLM's implementation
|
||||
|
||||
In a vLLM checkout, the request/response models live beside each endpoint:
|
||||
|
||||
| surface | path |
|
||||
| --- | --- |
|
||||
| OpenAI chat | `vllm/entrypoints/openai/chat_completion/` |
|
||||
| OpenAI completions | `vllm/entrypoints/openai/completion/` |
|
||||
| OpenAI responses | `vllm/entrypoints/openai/responses/` |
|
||||
| OpenAI embeddings | `vllm/entrypoints/pooling/embed/` |
|
||||
| OpenAI shared base models, errors, usage | `vllm/entrypoints/serve/engine/protocol.py` |
|
||||
| Anthropic messages | `vllm/entrypoints/anthropic/` |
|
||||
| audio | `vllm/entrypoints/speech_to_text/*/protocol.py` |
|
||||
|
||||
Each directory holds `protocol.py` (the Pydantic models), `serving.py` (behavior),
|
||||
and `api_router.py` (routing and status codes). Read the field declaration *and*
|
||||
the code that consumes it — a field can be declared faithfully and then ignored,
|
||||
which is still an incompatibility.
|
||||
|
||||
## 3. Fetch the upstream spec
|
||||
|
||||
Prefer a source that can actually be read as text, in this order.
|
||||
|
||||
### OpenAI
|
||||
|
||||
1. The OpenAPI spec — authoritative, machine-readable, and fetches cleanly:
|
||||
`https://raw.githubusercontent.com/openai/openai-openapi/master/openapi.yaml`
|
||||
2. The installed `openai` Python SDK's typed params, which track the spec closely.
|
||||
Locate them with `python -c "import openai,os;print(os.path.dirname(openai.__file__))"`,
|
||||
then read e.g. `types/chat/completion_create_params.py`,
|
||||
`types/chat/chat_completion.py`, `types/chat/chat_completion_chunk.py`,
|
||||
`types/responses/`. Note the SDK version — an old SDK is an old spec.
|
||||
3. The prose reference at `https://platform.openai.com/docs/api-reference/...`
|
||||
for semantics the types do not carry. It is script-heavy and often fetches
|
||||
poorly; treat a failed fetch as a fetch failure, not as absence of a field.
|
||||
|
||||
### Anthropic
|
||||
|
||||
1. `https://docs.claude.com/en/api/messages` and
|
||||
`https://docs.claude.com/en/api/messages-streaming` — these render as clean
|
||||
text and are the primary source for both fields and streaming event order.
|
||||
2. The installed `anthropic` SDK's `types/message_create_params.py`,
|
||||
`types/message.py`, `types/raw_message_stream_event.py`.
|
||||
|
||||
Say which source and which version answered the question, so the finding can be
|
||||
re-checked later.
|
||||
|
||||
## 4. Compare on all four axes
|
||||
|
||||
A field is compatible only when **name**, **type**, **default**, and **semantics**
|
||||
all match. Most real bugs hide in the last two: a field that exists with the right
|
||||
type but the wrong default, or one that is accepted and silently dropped.
|
||||
|
||||
Also check, when relevant:
|
||||
|
||||
- **Streaming** — chunk shape, the order of events, and the terminal event
|
||||
(`data: [DONE]` for Chat Completions and Completions; `response.completed`,
|
||||
`response.failed`, or `response.incomplete` with no `[DONE]` for Responses;
|
||||
`message_stop` for Anthropic).
|
||||
- **Errors** — HTTP status *and* the error body's `type`/`code` strings.
|
||||
- **Omitted vs null** — upstream often distinguishes an absent field from an
|
||||
explicit `null`. Pydantic defaults erase that distinction unless the model is
|
||||
written to preserve it.
|
||||
|
||||
## 5. Classify before calling it a bug
|
||||
|
||||
Three outcomes, and they are not the same:
|
||||
|
||||
- **Matches upstream** — no action.
|
||||
- **vLLM extension** — a field upstream has no name for at all (`vllm_xargs`,
|
||||
`chat_template_kwargs`, sampling knobs like `min_p`). These are intentional and
|
||||
allowed. Do not report them as violations. Check per surface: `top_k` is an
|
||||
extension on OpenAI but a real upstream field on Anthropic Messages.
|
||||
- **Incompatibility** — a field that *shares a name with upstream* but differs in
|
||||
type, default, or behavior, or an upstream field that is missing, ignored, or
|
||||
malformed. This is the reportable case.
|
||||
|
||||
An unsupported-but-declared field is its own subcase: decide whether vLLM rejects
|
||||
it explicitly or accepts and ignores it, and report which, since silently ignoring
|
||||
a documented field is the more damaging failure.
|
||||
|
||||
## 6. Report
|
||||
|
||||
State: the endpoint and field, what upstream specifies (with the source), what
|
||||
vLLM does (with `file:line`), which of the three outcomes it is, and the smallest
|
||||
fix. When behavior turns on a spec detail that is easy to disbelieve, quote the
|
||||
upstream line rather than paraphrasing it.
|
||||
@@ -0,0 +1,4 @@
|
||||
interface:
|
||||
display_name: "Check API Compat"
|
||||
short_description: "Check vLLM's OpenAI/Anthropic-compatible API against the upstream spec."
|
||||
default_prompt: "Use the vLLM API compatibility skill to check this endpoint or field against the upstream OpenAI or Anthropic spec."
|
||||
@@ -0,0 +1,190 @@
|
||||
---
|
||||
name: pr-checklist
|
||||
description: Prepare a vLLM change for human PR review or re-review, or analyze an open pull request. Use to check design fit, behavioral coverage, performance evidence, diff quality, and closure of previous feedback before requesting maintainer attention.
|
||||
---
|
||||
|
||||
# PR Checklist
|
||||
|
||||
This SKILL is designed to help systematically enforce vLLM's code quality and contribution standards during the pull request process,
|
||||
and to reduce the burden of manual code review on maintainers.
|
||||
|
||||
## Usage
|
||||
|
||||
The following checklist can be used to polish a change in preparation for a vLLM pull request, analyze an open pull request (either as the author or as a reviewer), or ensure that all contribution standards are met before requesting maintainer attention. To use this checklist effectively, go through each item and verify that it has been addressed in the diff / pull request.
|
||||
|
||||
This is intended to be used with a human in the loop: when running programmatically, prepare notes on every section to ensure a thorough report, with a clear header summarizing the findings and areas needing attention. When running interactively, present this report to the user with a clear list of actionable follow-up tasks and design decisions to make, then iterate.
|
||||
|
||||
### Re-Review
|
||||
|
||||
To update a PR following a review, read all comments and threads on the PR. For each concrete concern, work with your human operator to decide on a plan of action to address each either by a direct reply or by updating the change and acknowledging with a comment. Ensure the PR description remains fresh and all relevant documentation (including comments) in the change is up-to-date.
|
||||
|
||||
`gh pr view --comments` does not include inline review comments. Set `pr` to the PR number, then fetch all pages of conversation comments, review submissions, and inline comments (including replies):
|
||||
|
||||
```bash
|
||||
gh api --paginate "repos/vllm-project/vllm/issues/$pr/comments"
|
||||
gh api --paginate "repos/vllm-project/vllm/pulls/$pr/reviews"
|
||||
gh api --paginate "repos/vllm-project/vllm/pulls/$pr/comments"
|
||||
```
|
||||
|
||||
Use `in_reply_to_id` to group inline replies with their parent comments. Use GraphQL `reviewThreads` when resolved or outdated thread status is needed.
|
||||
|
||||
## Sections
|
||||
|
||||
### 1. Design Fit
|
||||
|
||||
Does the change align with the overall design and architecture of vLLM? Does it introduce any design inconsistencies or anti-patterns? Are there any potential long-term maintenance concerns related to this change?
|
||||
|
||||
#### 1.1: Impact on Core Components
|
||||
|
||||
vLLM's core components are shared across many different deployments. Changes to these files have high risk and changes should be structured in order to be easily maintainable and minimally invasive.
|
||||
|
||||
Such components include:
|
||||
|
||||
- The Model Runner (`vllm/v1/worker/*/model_runner.py`)
|
||||
- The Scheduler (`vllm/v1/core/sched/*`)
|
||||
- Core infrastructure (`vllm/v1/core/*`, including KV cache managers and utilities)
|
||||
|
||||
etc.
|
||||
|
||||
Changes should be structured to minimize complexity added to these files. Consider refactoring or isolating new functionality to reduce the impact on core components, both in terms of code maintainability and potential for introducing bugs.
|
||||
|
||||
Nontrivial logic associated with non-standard/non-default functionality should not be added to core/common code paths and files.
|
||||
|
||||
#### 1.2: Replication
|
||||
|
||||
Some changes may duplicate code that already exists elsewhere in the codebase. Does the change introduce unnecessary replication, or could existing functionality be reused instead? Consider refactoring to reduce duplication and improve maintainability.
|
||||
|
||||
Consult the affected area's agent instructions and contribution guides for its reuse conventions. See the [model implementation guide](../../../docs/contributing/model/basic.md) for model code and the [MRV2 development guidance](../../../docs/design/model_runner_v2.md#development-philosophy). Justify any duplication needed to isolate functionality and limit its impact on existing workflows.
|
||||
|
||||
#### 1.3: Complexity
|
||||
|
||||
Changes which increase the complexity of the codebase should be carefully scrutinized. Specifically those which add new abstractions, dependencies, or branching cases to existing components should be evaluated for their necessity and potential impact on maintainability.
|
||||
|
||||
Consider whether the added complexity could be mitigated through refactoring, modularization, or other design improvements. Aim to keep the codebase as simple and understandable as possible while still achieving the desired functionality.
|
||||
|
||||
As a rule of thumb, the complexity of a change can be estimated by the number of new lines of code added to the main codebase:
|
||||
|
||||
- Low complexity: < 50 lines. Generally easy to review and unlikely to introduce significant complexity.
|
||||
- Medium complexity: 50-250 lines. Requires careful review and consideration of potential impact on maintainability.
|
||||
- High complexity: > 250 lines. May introduce significant complexity and requires thorough review and justification.
|
||||
|
||||
You may exclude test cases and documentation updates from this line count, as they typically do not contribute significantly to the complexity of the main change. Note that line count is only a coarse indicator, meant to guide and flag possible overly-complex contributions. Use judgement when assessing the true semantic complexity and implied review burden.
|
||||
|
||||
Bug fixes should aim to remain below the "High complexity" bar. Those which exceed this threshold often warrant additional scrutiny and justification. Redesigning core components, modifying core interfaces, or otherwise introducing significant complexity in order to patch a bug is very often discouraged and should be carefully justified.
|
||||
|
||||
A PR that adds significant complexity or a substantial amount of new code must be well justified. A new feature should be relevant to, or requested by, enough users to warrant its size and complexity. A performance optimization should deliver end-to-end improvements on workloads that matter, and those gains should be large enough to outweigh the complexity it adds.
|
||||
|
||||
#### 1.4: Correctness and Compatibility
|
||||
|
||||
Ensure that all changes to vLLM are sound and do not violate existing contracts. Specifically, verify that all affected deployments which share an execution pathway with the changed code will remain correct. Trace related implementations, alternative backends sharing the same flow, and affected call sites of modified functions.
|
||||
|
||||
When adding features or changing contracts, ensure that relevant feature compatibility is assessed. Identify any potential incompatibilities or breaking changes and justify them explicitly in the PR notes.
|
||||
|
||||
When adding functionality, consider parallels with existing designs and identify opportunities to simplify and unify them rather than adding a separate implementation alongside them.
|
||||
|
||||
#### 1.5: Tradeoffs
|
||||
|
||||
When making a design change, adding a feature, or contributing a performance improvement, assess the tradeoffs and potential downsides. When changing a default backend to improve performance as measured on one configuration, could this cause a decrease in performance for another configuration? Could a new architecture change accelerate one workload but cause a feature incompatibility or performance regression in another? Ensure tradeoffs are clearly noted and validation is sufficient to catch potential lurking regressions or drawbacks.
|
||||
|
||||
Check that the change does not introduce CPU/GPU synchronization on the main model execution thread during steady-state inference. Inspect explicit synchronization, host reads of GPU tensors, boolean tensors as indices, and transfers that can block the host. Run the affected configuration with `VLLM_GPU_SYNC_CHECK=error` to confirm a concern; passing CI is insufficient when CI does not exercise that configuration. See the [MRV2 async execution design](../../../docs/design/model_runner_v2.md#2-async-first) for examples.
|
||||
|
||||
#### 1.6: Model Runner V1 Deprecation
|
||||
|
||||
Account for Model Runner V1 (MRV1) deprecation when reviewing model-runner changes. Target new GPU model-runner features at Model Runner V2 (MRV2) and follow its [design and development guidance](../../../docs/design/model_runner_v2.md#development-philosophy). For fixes needed by deployments still using MRV1, explain why the legacy path needs the change and check whether MRV2 is also affected.
|
||||
|
||||
### 2. Testing and Validation
|
||||
|
||||
All serving-engine changes contributed to vLLM must be _validated_: bug fixes must be tested on a reproducer, and new features or performance optimizations must be run locally and/or benchmarked to ensure correctness and effectiveness. It is unacceptable, for example, to contribute a bugfix patch without running vLLM end-to-end and verifying that it resolves the issue as intended.
|
||||
|
||||
Beyond _validation_, many (but not all) changes to vLLM should be _tested_: this includes writing new tests, updating existing tests, and ensuring that the test suite continues to provide reliable coverage of the affected functionality.
|
||||
|
||||
#### 2.1: Test Coverage and Effectiveness
|
||||
|
||||
When adding or modifying tests, consider both the coverage and the effectiveness of the test suite. Coverage refers to the extent to which the codebase is exercised by tests, while effectiveness refers to the ability of the tests to catch regressions and ensure correctness.
|
||||
|
||||
Assess whether the included test cases are meaningful, concise, and maintainable. Testing should not be exhaustive: local validation is typically sufficient to ensure a baseline correctness of the change. Contributed tests should focus exclusively on areas that are deemed prone to errors, regressions, or complex interactions within the codebase. Recommend deletion of redundant, trivial, or otherwise unnecessary tests: this is one of the most common complaints raised during code reviews.
|
||||
|
||||
Verify that testing actually exercises the new path, rather than passing through an unchanged path or fallback. Ensure added speculative decoding correctness tests assert nonzero acceptance rate, selection tests identify the desired backend, etc.
|
||||
|
||||
Aim to write tests that are meaningful, targeted, and maintainable. Avoid redundant or trivial tests that do not contribute to the overall reliability of the test suite. Avoid using mock objects, especially when highly coupled with internal logic.
|
||||
|
||||
#### 2.2: Test Reliability and Robustness
|
||||
|
||||
Some tests are considered "flaky": they may pass or fail intermittently without any changes to the codebase. Flaky tests can undermine confidence in the test suite and block successful CI runs. Analyze the added tests for potential flakiness and ensure they are reliable and robust.
|
||||
|
||||
Frequent sources of flaky tests include:
|
||||
|
||||
- Assertions requiring bit-exact output correctness when minor floating-point variation is expected. For example, a change in default kernel selection priority may cause exact-floating-point-correctness assertions to fail.
|
||||
- Tests that depend on the order of execution or the state of shared resources, which can lead to intermittent failures when tests are run in parallel or in different environments.
|
||||
|
||||
#### 2.3: Test Automation and Integration
|
||||
|
||||
vLLM's CI runs in Buildkite, with the launch configuration under `.buildkite/`. Ensure new tests run in an appropriate CI workflow when the required resources are available; add or update a workflow where needed. If CI cannot exercise the changed path because the required hardware or checkpoint is unavailable, document the missing coverage, the resource constraint, and validation performed outside CI. A CI coverage gap does not waive validation: the human submitter must confirm the required validation before publishing the PR.
|
||||
|
||||
### 3. Code Quality and Style
|
||||
|
||||
Code quality is another of the primary areas that maintainer effort is spent to address. Carefully audit the changes for adherence to coding standards, readability, maintainability, and overall design principles.
|
||||
|
||||
#### 3.1: Comments
|
||||
|
||||
Keep comments concise and use them sparingly. Consider whether the same information can be conveyed in fewer lines, and prefer clear code with meaningful, concise variable names, and fewer comments overall. Use comments to explain the reasoning behind complex logic, non-obvious decisions, or important context that is not immediately apparent from the code itself. Context regarding specific user decisions made during the design phase are often not needed and can instead be included as notes in the PR description instead.
|
||||
|
||||
When writing new functions, consider when a docstring may be useful. Most short helper functions do not need a docstring, but more complex functions or those forming part of the public API should include one. In particular, docstrings are useful when adding custom kernels or other performance-optimized implementations of otherwise-straightforward logic, as they make clear the input and output shapes/dtypes/assumptions, and can shed light on hard-to-follow implementations.
|
||||
|
||||
For bug fixes, there is no need to include details of the original bug and explanation of why the pre-changed version of the code would be incorrect. Those details should be in the PR description instead.
|
||||
|
||||
#### 3.2: Documentation and Examples
|
||||
|
||||
Ensure that any new features, changes, or public APIs are properly documented. This includes updating relevant documentation files under `docs/` and providing usage examples where applicable. Refer to `supported_models.md` for new models and `examples/` for usage examples.
|
||||
|
||||
#### 3.3: Helper Functions
|
||||
|
||||
Identify and inline trivially-simple helper functions which have only one call site.
|
||||
|
||||
#### 3.4: Formatting
|
||||
|
||||
Avoid formatting or whitespace changes that are unrelated to the PR, as long as pre-commit still passes.
|
||||
|
||||
### 4. Pull Request Contents
|
||||
|
||||
This section pertains to the contents of the pull request itself, rather than the changes.
|
||||
|
||||
When acting as the author, prepare or update these contents. When acting as a reviewer, assess them and report missing context or evidence.
|
||||
|
||||
Start from `.github/PULL_REQUEST_TEMPLATE.md`, retaining its sections and checklist. When creating a PR interactively, use `gh pr create --template .github/PULL_REQUEST_TEMPLATE.md`. For noninteractive creation or editing, fill out a copy of that template and pass the completed file with `--body-file`.
|
||||
|
||||
#### 4.1: Description and Context
|
||||
|
||||
Ensure the PR description has a short 1-2 line summary of the changes to facilitate quick understanding by reviewers. This section should link relevant issues, predecessor/blocking PRs, and provide any necessary context for the changes.
|
||||
|
||||
Keep Claims to a few concrete bullets and explain motivation briefly. Keep explanations in Validation and Details concise too, while retaining the evidence, root cause, tradeoffs, and limitations needed to assess the change. Link full logs or use collapsible HTML details for lengthy supporting material rather than polluting the main narrative.
|
||||
|
||||
#### 4.2: Claims and Supporting Evidence
|
||||
|
||||
Ensure the PR makes clear, concise claims about the accomplishment:
|
||||
|
||||
- For bug fixes, what specific issue is being resolved and how can it be verified?
|
||||
- For new features, what functionality is being added and how can it be utilized?
|
||||
- For refactoring, what was the motivation and what improvements are expected?
|
||||
- For performance optimizations, what specific model/hardware/serving configurations & workloads are being targeted and how can the performance gains be measured?
|
||||
|
||||
For the stated claims, ensure the PR provides sufficient evidence, examples, or references to support them, making it easy for reviewers to verify the correctness and impact of the changes.
|
||||
|
||||
Summarize model evaluations and benchmarks in compact tables with the configuration, baseline, changed result, and reproduction commands. For repeated benchmarks, report the run count and variability. For kernel performance changes, include before/after profile visualizations/timelines when they help explain the measured gains. Link relevant CI failures or regressing PRs when they motivate the change.
|
||||
|
||||
- For bug fixes, ensure the reproducer is clearly provided and was validated to confirm the issue and the resolution.
|
||||
- For new features, ensure that usage examples are included and validated to confirm that the feature works as intended.
|
||||
- For refactoring, ensure that changes are sufficiently validated at appropriate test levels (unit is sufficient for minor refactors, E2E serving is required for significant refactoring).
|
||||
- For performance optimizations, ensure that benchmarks or tests are included to validate the claimed performance improvements. Ideally, an end-to-end evaluation should demonstrate speedup on a concrete serving example. Microbenchmark-only validation is often insufficient: make a note if end-to-end evaluation is not included (e.g. due to lack of access to sufficient hardware to fully validate).
|
||||
|
||||
#### 4.3: Root-Cause Analysis
|
||||
|
||||
When fixing bugs or issues, provide a clear root-cause analysis. Explain what caused the problem, how it was identified, and why the chosen solution effectively addresses it. A patch which resolves the issue without a proper root-cause analysis may lead to recurring problems and increased maintenance overhead.
|
||||
|
||||
#### 4.4: Implementation Details
|
||||
|
||||
Provide a detailed explanation of the implementation, including design decisions, trade-offs, and any relevant technical details. Highlight any non-obvious choices, tradeoffs, potential limitations, and areas where maintainer attention is required.
|
||||
|
||||
#### 4.5: Contributing Guide
|
||||
|
||||
Ensure that the PR follows the [contributing guide](https://docs.vllm.ai/en/latest/contributing), including adhering to coding standards, commit message conventions, and any other project-specific guidelines. This helps maintain consistency and quality across the codebase.
|
||||
@@ -24,6 +24,9 @@ steps:
|
||||
- tests/kernels/test_onednn.py
|
||||
- tests/kernels/test_awq_int4_to_int8.py
|
||||
- tests/kernels/quantization/test_cpu_fp8_scaled_mm.py
|
||||
- tests/kernels/quantization/test_zen_da8w4.py
|
||||
- vllm/model_executor/kernels/linear/mixed_precision/zentorch.py
|
||||
- vllm/model_executor/layers/fused_moe/experts/zentorch_moe.py
|
||||
- tests/kernels/mamba/cpu/test_cpu_gdn_ops.py
|
||||
- tests/kernels/mamba/test_cpu_short_conv.py
|
||||
- tests/kernels/mamba/test_causal_conv1d.py
|
||||
@@ -46,6 +49,7 @@ steps:
|
||||
tests/kernels/test_onednn.py \
|
||||
tests/kernels/test_awq_int4_to_int8.py \
|
||||
tests/kernels/quantization/test_cpu_fp8_scaled_mm.py \
|
||||
tests/kernels/quantization/test_zen_da8w4.py \
|
||||
tests/kernels/mamba/cpu/test_cpu_gdn_ops.py \
|
||||
tests/kernels/mamba/test_causal_conv1d.py \
|
||||
tests/kernels/mamba/test_mamba_ssm.py \
|
||||
|
||||
@@ -62,7 +62,7 @@ steps:
|
||||
- >-
|
||||
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
|
||||
'cd tests &&
|
||||
pytest -v -s v1/engine --ignore v1/engine/test_preprocess_error_handling.py &&
|
||||
pytest -v -s v1/engine --ignore v1/engine/test_preprocess_error_handling.py --ignore v1/engine/test_core_engine_actor_manager.py &&
|
||||
pytest -v -s test_config.py'
|
||||
|
||||
- label: V1 e2e (2 GPUs)
|
||||
|
||||
@@ -77,6 +77,7 @@ steps:
|
||||
'export VLLM_WORKER_MULTIPROC_METHOD=spawn &&
|
||||
cd tests &&
|
||||
pytest -v -s entrypoints/serve --ignore=entrypoints/serve/dev/rpc
|
||||
--ignore=entrypoints/serve/dev/rlhf/test_weight_operation_metrics.py
|
||||
--deselect="tests/entrypoints/serve/dev/rlhf/state_transitions/test_pause_resume.py::TestPauseResume::test_clear_cache_preserves_output_and_controls_prefix_cache[MRV1]" &&
|
||||
PYTHONPATH=/workspace/vllm pytest -v -s entrypoints/serve/dev/rpc'
|
||||
|
||||
|
||||
@@ -99,14 +99,14 @@ steps:
|
||||
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
|
||||
'cd tests && pip install dpctl &&
|
||||
pytest -v -s v1/core --ignore=v1/core/test_reset_prefix_cache_e2e.py --ignore=v1/core/test_scheduler_e2e.py &&
|
||||
pytest -v -s v1/engine --ignore=v1/engine/test_output_processor.py &&
|
||||
pytest -v -s v1/engine --ignore=v1/engine/test_output_processor.py --ignore=v1/engine/test_core_engine_actor_manager.py &&
|
||||
pytest -v -s v1/sample --ignore=v1/sample/test_logprobs.py --ignore=v1/sample/test_logprobs_e2e.py -k "not test_topk_only and not test_topp_only and not test_topk_and_topp" &&
|
||||
pytest -v -s v1/worker --ignore=v1/worker/test_gpu_model_runner.py --ignore=v1/worker/test_worker_memory_snapshot.py &&
|
||||
pytest -v -s v1/structured_output &&
|
||||
pytest -v -s v1/test_serial_utils.py &&
|
||||
pytest -v -s v1/e2e/general/test_correctness_sliding_window.py &&
|
||||
pytest -v -s v1/spec_decode --ignore=v1/spec_decode/test_max_len.py --ignore=v1/spec_decode/test_speculators_eagle3.py --ignore=v1/spec_decode/test_acceptance_length.py --ignore=v1/spec_decode/test_speculators_correctness.py --deselect=tests/v1/spec_decode/test_mtp.py::test_glm_mtp_defers_lm_head &&
|
||||
pytest -v -s v1/kv_connector/unit --ignore=v1/kv_connector/unit/test_lmcache_integration.py --ignore=v1/kv_connector/unit/test_hf3fs_client.py --ignore=v1/kv_connector/unit/test_hf3fs_connector.py --ignore=v1/kv_connector/unit/test_hf3fs_metadata_server.py --ignore=v1/kv_connector/unit/test_offloading_connector.py --deselect "v1/kv_connector/unit/test_nixl_connector.py::test_abort_timeout_on_prefiller[ray]" --ignore=v1/kv_connector/unit/test_nixl_connector_hma.py'
|
||||
pytest -v -s v1/kv_connector/unit --ignore=v1/kv_connector/unit/test_lmcache_integration.py --ignore=v1/kv_connector/unit/test_hf3fs_client.py --ignore=v1/kv_connector/unit/test_hf3fs_connector.py --ignore=v1/kv_connector/unit/test_hf3fs_metadata_server.py --ignore=v1/kv_connector/unit/test_offloading_connector.py --ignore=v1/kv_connector/unit/test_nixl_connector_hma.py --deselect=tests/v1/kv_connector/unit/test_nixl_connector.py::test_abort_timeout_on_prefiller[ray]'
|
||||
parallelism: 2
|
||||
- label: "XPU server test"
|
||||
key: xpu-server-test
|
||||
|
||||
@@ -71,9 +71,11 @@ if [[ -z "${BK_TOKEN}" ]]; then
|
||||
fi
|
||||
|
||||
AUDIENCE="pytorch-cross-repo-ci-relay"
|
||||
# Pre-flight only; the token minted here is discarded. Checking now means an
|
||||
# unusable OIDC setup costs seconds rather than being discovered after hours of
|
||||
# polling. The token that is actually sent is minted after the wait, below.
|
||||
# OIDC redaction also requires the unavailable Job API socket.
|
||||
OIDC_TOKEN="$(buildkite-agent oidc request-token --skip-redaction --audience "${AUDIENCE}" 2>/dev/null)"
|
||||
if [[ -z "${OIDC_TOKEN}" ]]; then
|
||||
if [[ -z "$(buildkite-agent oidc request-token --skip-redaction --audience "${AUDIENCE}" 2>/dev/null)" ]]; then
|
||||
echo "could not mint a Buildkite OIDC token -- skipping report"
|
||||
exit 0
|
||||
fi
|
||||
@@ -142,7 +144,7 @@ POLL_INTERVAL_S="${CRCR_POLL_INTERVAL_S:-900}"
|
||||
# meaningless whenever the step was itself delayed.
|
||||
#
|
||||
# Resolved after the first fetch, from the build's own created_at.
|
||||
MAX_BUILD_AGE_S="${CRCR_MAX_BUILD_AGE_S:-25200}"
|
||||
MAX_BUILD_AGE_S="${CRCR_MAX_BUILD_AGE_S:-14400}"
|
||||
WAIT_DEADLINE=""
|
||||
# Used until created_at is known. Without it, an API outage on the very first
|
||||
# request leaves the deadline unresolved and the loop spins to the step timeout.
|
||||
@@ -220,6 +222,17 @@ done
|
||||
# Resolved from this script's location: the pipeline runs it from
|
||||
# /vllm-workspace/tests, so a repo-relative path would not resolve.
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
# Minted here, not before the poll loop: a Buildkite OIDC token defaults to a
|
||||
# five-minute lifetime and the loop above can run to the build-age deadline.
|
||||
# Build 91827 waited 6h48m and then took a 401 on all 360 callbacks from a
|
||||
# token that had expired hours earlier.
|
||||
OIDC_TOKEN="$(buildkite-agent oidc request-token --skip-redaction --audience "${AUDIENCE}" 2>/dev/null)"
|
||||
if [[ -z "${OIDC_TOKEN}" ]]; then
|
||||
echo "could not mint a Buildkite OIDC token after the wait -- skipping report"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
python3 "${SCRIPT_DIR}/crcr_report.py" \
|
||||
--build-json "${BUILD_JSON}" \
|
||||
--callback-url "${CALLBACK_URL}" \
|
||||
|
||||
@@ -37,14 +37,14 @@ case "${test_suite}" in
|
||||
cd tests
|
||||
|
||||
pytest -v -s v1/core --ignore=v1/core/test_reset_prefix_cache_e2e.py --ignore=v1/core/test_scheduler_e2e.py
|
||||
pytest -v -s v1/engine --ignore=v1/engine/test_output_processor.py
|
||||
pytest -v -s v1/engine --ignore=v1/engine/test_output_processor.py --ignore=v1/engine/test_core_engine_actor_manager.py
|
||||
pytest -v -s v1/sample --ignore=v1/sample/test_logprobs.py --ignore=v1/sample/test_logprobs_e2e.py -k "not test_topk_only and not test_topp_only and not test_topk_and_topp"
|
||||
pytest -v -s v1/worker --ignore=v1/worker/test_gpu_model_runner.py --ignore=v1/worker/test_worker_memory_snapshot.py
|
||||
pytest -v -s v1/structured_output
|
||||
pytest -v -s v1/test_serial_utils.py
|
||||
pytest -v -s v1/e2e/general/test_correctness_sliding_window.py
|
||||
pytest -v -s v1/spec_decode --ignore=v1/spec_decode/test_max_len.py --ignore=v1/spec_decode/test_speculators_eagle3.py --ignore=v1/spec_decode/test_acceptance_length.py --ignore=v1/spec_decode/test_speculators_correctness.py --deselect=tests/v1/spec_decode/test_mtp.py::test_glm_mtp_defers_lm_head
|
||||
pytest -v -s v1/kv_connector/unit --ignore=v1/kv_connector/unit/test_multi_connector.py --ignore=v1/kv_connector/unit/test_example_connector.py --ignore=v1/kv_connector/unit/test_lmcache_integration.py --ignore=v1/kv_connector/unit/test_hf3fs_client.py --ignore=v1/kv_connector/unit/test_hf3fs_connector.py --ignore=v1/kv_connector/unit/test_hf3fs_metadata_server.py --ignore=v1/kv_connector/unit/test_offloading_connector.py
|
||||
pytest -v -s v1/kv_connector/unit --ignore=v1/kv_connector/unit/test_multi_connector.py --ignore=v1/kv_connector/unit/test_example_connector.py --ignore=v1/kv_connector/unit/test_lmcache_integration.py --ignore=v1/kv_connector/unit/test_hf3fs_client.py --ignore=v1/kv_connector/unit/test_hf3fs_connector.py --ignore=v1/kv_connector/unit/test_hf3fs_metadata_server.py --ignore=v1/kv_connector/unit/test_offloading_connector.py --ignore=v1/kv_connector/unit/test_nixl_connector_hma.py --deselect=tests/v1/kv_connector/unit/test_nixl_connector.py::test_abort_timeout_on_prefiller[ray]
|
||||
;;
|
||||
server)
|
||||
pip install av
|
||||
@@ -57,7 +57,7 @@ case "${test_suite}" in
|
||||
cd tests
|
||||
|
||||
pytest -v -s quantization/test_auto_round.py
|
||||
pytest -v -s quantization/test_online.py
|
||||
pytest -v -s quantization/test_online.py --deselect=tests/quantization/test_online.py::test_online_quantization_loads_real_weights
|
||||
;;
|
||||
compressed-tensors-fp8)
|
||||
cd tests
|
||||
|
||||
+570
-567
File diff suppressed because it is too large
Load Diff
@@ -50,9 +50,9 @@ steps:
|
||||
- pytest -v -s basic_correctness/memory/sleep_mode
|
||||
mirror:
|
||||
amd:
|
||||
label: ":amd: (MI355 DPX) Basic Correctness Sleep Mode"
|
||||
label: ":amd: (MI300) Basic Correctness Sleep Mode"
|
||||
dind: false
|
||||
device: mi355_dpx
|
||||
device: mi300_1
|
||||
timeout_in_minutes: 60
|
||||
depends_on:
|
||||
- image-build-amd
|
||||
|
||||
@@ -12,6 +12,30 @@ steps:
|
||||
commands:
|
||||
- export VLLM_TEST_CLEAN_GPU_MEMORY=1
|
||||
- pytest -v -s tests/compile/correctness_e2e/test_sequence_parallel.py
|
||||
mirror:
|
||||
amd:
|
||||
label: ":amd: (MI355) Sequence Parallel Correctness"
|
||||
dind: false
|
||||
device: mi355_2
|
||||
num_devices: 2
|
||||
timeout_in_minutes: 180
|
||||
depends_on:
|
||||
- image-build-amd
|
||||
source_file_dependencies:
|
||||
- .buildkite/test_areas/compile.yaml
|
||||
- vllm/compilation/
|
||||
- vllm/config/
|
||||
- vllm/distributed/
|
||||
- vllm/model_executor/kernels/linear/
|
||||
- vllm/model_executor/layers/
|
||||
- vllm/model_executor/models/llama.py
|
||||
- vllm/v1/attention/
|
||||
- vllm/v1/worker/
|
||||
- vllm/v1/cudagraph_dispatcher.py
|
||||
- vllm/platforms/rocm.py
|
||||
- tests/compile/correctness_e2e/test_sequence_parallel.py
|
||||
- tests/models/registry.py
|
||||
- tests/utils.py
|
||||
|
||||
- label: ":nvidia: (B200) AsyncTP Correctness"
|
||||
key: asynctp-correctness-tests-b200
|
||||
@@ -23,6 +47,38 @@ steps:
|
||||
commands:
|
||||
- export VLLM_TEST_CLEAN_GPU_MEMORY=1
|
||||
- pytest -v -s tests/compile/correctness_e2e/test_async_tp.py
|
||||
mirror:
|
||||
amd:
|
||||
label: ":amd: (MI355) AsyncTP Correctness"
|
||||
dind: false
|
||||
device: mi355_2
|
||||
num_devices: 2
|
||||
timeout_in_minutes: 60
|
||||
working_dir: "/vllm-workspace/"
|
||||
depends_on:
|
||||
- image-build-amd
|
||||
source_file_dependencies:
|
||||
- .buildkite/test_areas/compile.yaml
|
||||
- vllm/compilation/
|
||||
- vllm/config/
|
||||
- vllm/distributed/
|
||||
- vllm/model_executor/layers/
|
||||
- vllm/model_executor/kernels/linear/
|
||||
- vllm/model_executor/models/llama.py
|
||||
- vllm/model_executor/models/qwen3.py
|
||||
- vllm/v1/attention/
|
||||
- vllm/_aiter_ops.py
|
||||
- vllm/platforms/rocm.py
|
||||
- vllm/transformers_utils/repo_utils.py
|
||||
- tests/compile/correctness_e2e/test_async_tp.py
|
||||
- tests/compile/fusions_e2e/common.py
|
||||
- tests/models/registry.py
|
||||
- tests/utils.py
|
||||
commands:
|
||||
- export VLLM_TEST_CLEAN_GPU_MEMORY=1
|
||||
- pytest -v -s tests/compile/correctness_e2e/test_async_tp.py::test_rocm_async_tp_bf16_output_correctness
|
||||
- pytest -v -s tests/compile/correctness_e2e/test_async_tp.py::test_async_tp_pass_correctness -k RedHatAI
|
||||
- pytest -v -s tests/compile/correctness_e2e/test_async_tp.py::test_async_tp_pass_nvfp4_correctness
|
||||
|
||||
- label: ":nvidia: (H100) Distributed Compile"
|
||||
key: distributed-compile-unit-tests-2xh100
|
||||
@@ -188,6 +244,27 @@ steps:
|
||||
# Qwen/Deepseek requires +quant_fp8 as -quant_fp8 rms+quant fusion is not supported
|
||||
# Run just llama3 (fp8 & fp4) for all config combinations (only inductor partition)
|
||||
- pytest -v -s tests/compile/fusions_e2e/test_tp1_quant.py -k "inductor_partition and (FLASHINFER and not +rms_norm and (not +quant_fp8 or +quant_fp8 and (qwen3 or deepseek)) or llama-3)"
|
||||
mirror:
|
||||
amd:
|
||||
label: ":amd: (MI355 DPX) Fusion E2E FP8 Config Sweep"
|
||||
dind: false
|
||||
device: mi355_dpx
|
||||
timeout_in_minutes: 45
|
||||
working_dir: "/vllm-workspace/"
|
||||
depends_on:
|
||||
- image-build-amd
|
||||
source_file_dependencies:
|
||||
- .buildkite/test_areas/compile.yaml
|
||||
- csrc/quantization/
|
||||
- vllm/compilation/
|
||||
- vllm/model_executor/
|
||||
- vllm/v1/attention/
|
||||
- vllm/_aiter_ops.py
|
||||
- vllm/platforms/rocm.py
|
||||
- tests/compile/fusions_e2e/
|
||||
commands:
|
||||
- rocm-smi
|
||||
- pytest -v -s tests/compile/fusions_e2e/test_tp1_quant.py -k "test_tp1_fp8_fusions and inductor_partition and not +rms_norm and not True and ((Llama-3 and TRITON_ATTN) or (Qwen3-30B and +quant_fp8 and TRITON_ATTN) or (DeepSeek-Coder and +quant_fp8 and TRITON_MLA))"
|
||||
|
||||
- label: ":nvidia: (H100) Fusion E2E TP2 Quick"
|
||||
key: fusion-e2e-tp2-quick-h100
|
||||
@@ -267,6 +344,31 @@ steps:
|
||||
- nvidia-smi
|
||||
# Run just llama3 (fp8 & bf16) for all config combinations
|
||||
- pytest -v -s tests/compile/fusions_e2e/test_tp2_ar_rms.py -k "llama-3"
|
||||
mirror:
|
||||
amd:
|
||||
label: ":amd: (MI355) Fusion E2E TP2 AR-RMS FP8 Config Sweep"
|
||||
dind: false
|
||||
device: mi355_2
|
||||
num_devices: 2
|
||||
timeout_in_minutes: 45
|
||||
working_dir: "/vllm-workspace/"
|
||||
depends_on:
|
||||
- image-build-amd
|
||||
source_file_dependencies:
|
||||
- vllm/compilation/
|
||||
- vllm/model_executor/
|
||||
- vllm/distributed/
|
||||
- vllm/v1/attention/
|
||||
- vllm/_aiter_ops.py
|
||||
- vllm/platforms/rocm.py
|
||||
- tests/compile/fusions_e2e/
|
||||
- tests/utils.py
|
||||
- tests/conftest.py
|
||||
- csrc/quantization/
|
||||
commands:
|
||||
# Existing MI300 jobs own BF16; TP2 Quick owns the excluded Llama cell.
|
||||
- amd-smi
|
||||
- VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_CUSTOM_AR=1 pytest -v -s tests/compile/fusions_e2e/test_tp2_ar_rms.py -k 'test_tp2_ar_rms_fp8_fusions and Llama-3 and ROCM_AITER_UNIFIED_ATTN and (dynamo_partition or +quant_fp8 or +rms_norm)'
|
||||
|
||||
- label: ":nvidia: (H100) Fusion E2E TP2 AsyncTP Config Sweep"
|
||||
key: fusion-e2e-tp2-asynctp-config-sweep-h100
|
||||
@@ -334,3 +436,30 @@ steps:
|
||||
# for ar-rms-quant-fp4, also sweep llama3
|
||||
- pytest -v -s tests/compile/fusions_e2e/test_tp2_ar_rms.py -k "(FLASHINFER and inductor_partition and not +rms_norm and (not +quant_fp8 or +quant_fp8 and (qwen3 or deepseek))) or Llama-3.1-8B-Instruct-FP4"
|
||||
- pytest -v -s tests/compile/fusions_e2e/test_tp2_async_tp.py -k "FLASHINFER and inductor_partition and not +rms_norm and (not +quant_fp8 or +quant_fp8 and (qwen3 or deepseek))"
|
||||
mirror:
|
||||
amd:
|
||||
label: ":amd: (MI355) Fusion E2E TP2 FP8 BF16 rewrites"
|
||||
dind: false
|
||||
device: mi355_2
|
||||
num_devices: 2
|
||||
timeout_in_minutes: 45
|
||||
working_dir: "/vllm-workspace/"
|
||||
depends_on:
|
||||
- image-build-amd
|
||||
source_file_dependencies:
|
||||
- vllm/compilation/
|
||||
- vllm/model_executor/
|
||||
- vllm/distributed/
|
||||
- vllm/v1/attention/
|
||||
- vllm/_aiter_ops.py
|
||||
- vllm/platforms/rocm.py
|
||||
- tests/compile/fusions_e2e/
|
||||
- tests/utils.py
|
||||
- tests/conftest.py
|
||||
- csrc/quantization/
|
||||
commands:
|
||||
# Dummy-weight rewrite coverage; trained FP8 AR/RMS equivalence is unresolved.
|
||||
# TODO: FP4 output-quant and trained MXFP4 AR/RMS fusion remain unsupported.
|
||||
- amd-smi
|
||||
- VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_CUSTOM_AR=1 pytest -v -s tests/compile/fusions_e2e/test_tp2_ar_rms.py -k 'test_tp2_ar_rms_fp8_fusions and qwen3 and ROCM_AITER_UNIFIED_ATTN and +quant_fp8 and (dynamo_partition or not +rms_norm)'
|
||||
- pytest -v -s tests/compile/fusions_e2e/test_tp2_rocm_moe.py
|
||||
|
||||
@@ -439,6 +439,27 @@ steps:
|
||||
- pytest -v -s tests/distributed/test_nccl_symm_mem.py
|
||||
- pytest -v -s tests/v1/distributed/test_dbo.py
|
||||
- pytest -v -s tests/distributed/test_mnnvl_alltoall.py
|
||||
mirror:
|
||||
amd:
|
||||
label: ":amd: (MI355) Distributed AgRs All2All"
|
||||
dind: false
|
||||
device: mi355_2
|
||||
num_devices: 2
|
||||
timeout_in_minutes: 30
|
||||
working_dir: "/vllm-workspace/"
|
||||
depends_on:
|
||||
- image-build-amd
|
||||
source_file_dependencies:
|
||||
- .buildkite/test_areas/distributed.yaml
|
||||
- vllm/distributed/
|
||||
- vllm/config/parallel.py
|
||||
- vllm/config/vllm.py
|
||||
- vllm/forward_context.py
|
||||
- vllm/platforms/rocm.py
|
||||
- tests/distributed/test_mnnvl_alltoall.py
|
||||
- tests/utils.py
|
||||
commands:
|
||||
- pytest -v -s tests/distributed/test_mnnvl_alltoall.py::test_args_dispatch_combine
|
||||
|
||||
- label: ":nvidia: (B200) GEMM-RS/AR"
|
||||
key: gemm-rs-ar-2xb200
|
||||
@@ -452,6 +473,34 @@ steps:
|
||||
- tests/kernels/test_gemm_rs_ar.py
|
||||
commands:
|
||||
- pytest -v -s tests/kernels/test_gemm_rs_ar.py
|
||||
mirror:
|
||||
amd:
|
||||
label: ":amd: (MI355) GEMM-RS-AR"
|
||||
dind: false
|
||||
device: mi355_4
|
||||
num_devices: 4
|
||||
timeout_in_minutes: 20
|
||||
working_dir: /vllm-workspace/
|
||||
depends_on:
|
||||
- image-build-amd
|
||||
source_file_dependencies:
|
||||
- .buildkite/test_areas/distributed.yaml
|
||||
- tests/kernels/test_gemm_rs_ar.py
|
||||
- tests/kernels/test_rocm_gemm_rs_ar.py
|
||||
- tests/conftest.py
|
||||
- vllm/models/common/ops/sequence_parallel.py
|
||||
- vllm/model_executor/layers/linear.py
|
||||
- vllm/model_executor/layers/utils.py
|
||||
- vllm/model_executor/layers/quantization/online/
|
||||
- vllm/model_executor/layers/quantization/utils/mxfp8_utils.py
|
||||
- vllm/model_executor/kernels/linear/
|
||||
- vllm/distributed/
|
||||
- vllm/_aiter_ops.py
|
||||
- vllm/_custom_ops.py
|
||||
- csrc/rocm/skinny_gemms.cu
|
||||
- vllm/platforms/rocm.py
|
||||
commands:
|
||||
- pytest -v -s tests/kernels/test_rocm_gemm_rs_ar.py
|
||||
|
||||
- label: ":nvidia: (L4) Distributed 2-Node"
|
||||
key: 2-node-test-4-gpus
|
||||
|
||||
@@ -59,6 +59,11 @@ steps:
|
||||
- image-build-amd
|
||||
source_file_dependencies:
|
||||
- vllm/platforms/rocm.py
|
||||
commands:
|
||||
- pytest -v -s v1/engine/test_preprocess_error_handling.py
|
||||
- pytest -v -s v1/engine --ignore v1/engine/test_preprocess_error_handling.py
|
||||
- pytest -v -s v1/test_tensor_ipc_queue.py
|
||||
- pytest -v -s config/test_mp_reducer.py
|
||||
|
||||
- label: ":nvidia: (H200 MIG 18GB) E2E Scheduling"
|
||||
key: e2e-scheduling-1-gpu
|
||||
@@ -80,6 +85,26 @@ steps:
|
||||
source_file_dependencies:
|
||||
- vllm/platforms/rocm.py
|
||||
|
||||
- label: ":nvidia: (H200 MIG 18GB) E2E Scheduling Accuracy"
|
||||
key: e2e-scheduling-accuracy-1-gpu
|
||||
timeout_in_minutes: 53
|
||||
device: h200_18gb
|
||||
source_file_dependencies:
|
||||
- vllm/v1/
|
||||
- tests/v1/e2e/general/
|
||||
commands:
|
||||
- pytest -v -s v1/e2e/general/test_async_scheduling_accuracy.py
|
||||
mirror:
|
||||
amd:
|
||||
label: ":amd: (MI250) E2E Scheduling Accuracy"
|
||||
dind: false
|
||||
device: mi250_1
|
||||
timeout_in_minutes: 55
|
||||
depends_on:
|
||||
- image-build-amd
|
||||
source_file_dependencies:
|
||||
- vllm/platforms/rocm.py
|
||||
|
||||
- label: ":nvidia: (H200 MIG 18GB) E2E Core"
|
||||
device: h200_18gb
|
||||
key: e2e-core-1-gpu
|
||||
@@ -88,13 +113,13 @@ steps:
|
||||
- vllm/v1/
|
||||
- tests/v1/e2e/general/
|
||||
commands:
|
||||
- pytest -v -s v1/e2e/general --ignore v1/e2e/general/test_async_scheduling.py --ignore v1/e2e/general/test_kv_sharing_fast_prefill.py -k "not test_mamba_prefix_cache_mrv2"
|
||||
- pytest -v -s v1/e2e/general --ignore v1/e2e/general/test_async_scheduling.py --ignore v1/e2e/general/test_async_scheduling_accuracy.py --ignore v1/e2e/general/test_kv_sharing_fast_prefill.py -k "not test_mamba_prefix_cache_mrv2"
|
||||
mirror:
|
||||
amd:
|
||||
label: ":amd: (MI250) E2E Core"
|
||||
dind: false
|
||||
commands:
|
||||
- pytest -v -s v1/e2e/general --ignore v1/e2e/general/test_async_scheduling.py --ignore v1/e2e/general/test_kv_sharing_fast_prefill.py -k "not test_mamba_prefix_cache_mrv2"
|
||||
- pytest -v -s v1/e2e/general --ignore v1/e2e/general/test_async_scheduling.py --ignore v1/e2e/general/test_async_scheduling_accuracy.py --ignore v1/e2e/general/test_kv_sharing_fast_prefill.py -k "not test_mamba_prefix_cache_mrv2"
|
||||
device: mi250_1
|
||||
timeout_in_minutes: 50
|
||||
depends_on:
|
||||
@@ -133,6 +158,27 @@ steps:
|
||||
- pytest -v -s v1/e2e/general/test_kv_sharing_fast_prefill.py
|
||||
- pytest -v -s v1/e2e/general/test_mamba_prefix_cache.py::test_mamba_prefix_cache_mrv2 v1/e2e/general/test_mamba_prefix_cache.py::test_mamba_prefix_cache_mrv2_async
|
||||
|
||||
- label: ":nvidia: (H100) ReplaySSM E2E"
|
||||
device: h100
|
||||
num_devices: 2
|
||||
key: replayssm-e2e
|
||||
timeout_in_minutes: 90
|
||||
working_dir: "/vllm-workspace/tests"
|
||||
source_file_dependencies:
|
||||
- tests/v1/e2e/test_replayssm_decode.py
|
||||
- vllm/config/cache.py
|
||||
- vllm/config/vllm.py
|
||||
- vllm/model_executor/layers/mamba/
|
||||
- vllm/model_executor/models/nemotron_h.py
|
||||
- vllm/model_executor/warmup/replayssm_warmup.py
|
||||
- vllm/v1/attention/backends/mamba_attn.py
|
||||
- vllm/v1/spec_decode/
|
||||
- vllm/v1/worker/gpu/
|
||||
- vllm/v1/worker/gpu_model_runner.py
|
||||
- vllm/v1/worker/gpu_worker.py
|
||||
commands:
|
||||
- pytest -v -s v1/e2e/test_replayssm_decode.py
|
||||
|
||||
- label: ":nvidia: (L4) V1 E2E"
|
||||
key: v1-e2e-2-gpus
|
||||
timeout_in_minutes: 25 # TODO: Fix timeout after we have more confidence in the test stability
|
||||
|
||||
@@ -23,9 +23,9 @@ steps:
|
||||
mirror:
|
||||
amd:
|
||||
# Focused MRV2 regression; the broader model/backend matrix is pending.
|
||||
label: ':amd: (MI355) MRV2 Sampler JIT Warmup'
|
||||
label: ':amd: (MI355 DPX) MRV2 Sampler JIT Warmup'
|
||||
dind: false
|
||||
device: mi355_1
|
||||
device: mi355_dpx
|
||||
num_devices: 1
|
||||
timeout_in_minutes: 10
|
||||
working_dir: /vllm-workspace/
|
||||
|
||||
@@ -184,6 +184,20 @@ steps:
|
||||
- tests/kernels/
|
||||
commands:
|
||||
- pytest -v -s kernels/mhc
|
||||
mirror:
|
||||
amd:
|
||||
label: ":amd: (MI355) mHC Kernels"
|
||||
dind: false
|
||||
device: mi355_1
|
||||
timeout_in_minutes: 50
|
||||
working_dir: "/vllm-workspace/tests"
|
||||
depends_on:
|
||||
- image-build-amd
|
||||
source_file_dependencies:
|
||||
- csrc/
|
||||
- vllm/
|
||||
- tests/kernels/
|
||||
- vllm/platforms/rocm.py
|
||||
|
||||
- label: ":nvidia: (B200) FLA Ops Kernels"
|
||||
key: kernels-fla-ops-test-b200
|
||||
@@ -196,6 +210,20 @@ steps:
|
||||
- tests/kernels/
|
||||
commands:
|
||||
- pytest -v -s kernels/fla
|
||||
mirror:
|
||||
amd:
|
||||
label: ":amd: (MI355) FLA Ops Kernels"
|
||||
dind: false
|
||||
device: mi355_1
|
||||
timeout_in_minutes: 50
|
||||
working_dir: "/vllm-workspace/tests"
|
||||
depends_on:
|
||||
- image-build-amd
|
||||
source_file_dependencies:
|
||||
- csrc/
|
||||
- vllm/
|
||||
- tests/kernels/
|
||||
- vllm/platforms/rocm.py
|
||||
|
||||
# Catch-all for test files at the tests/kernels root. This job collects
|
||||
# the whole root so new files are wired by default.
|
||||
@@ -231,6 +259,50 @@ steps:
|
||||
--ignore=kernels/test_shuffle_rows.py
|
||||
# BROKEN on main, pending kernel fixes (B200):
|
||||
# test_shuffle_rows.py (1: test_shuffle_rows_edge_cases)
|
||||
mirror:
|
||||
amd:
|
||||
label: ":amd: (MI355) Miscellaneous Kernels"
|
||||
dind: false
|
||||
device: mi355_1
|
||||
timeout_in_minutes: 55
|
||||
working_dir: "/vllm-workspace/tests"
|
||||
depends_on:
|
||||
- image-build-amd
|
||||
source_file_dependencies:
|
||||
- csrc/
|
||||
- vllm/
|
||||
- tests/kernels/
|
||||
- vllm/platforms/rocm.py
|
||||
# Keep the ROCm exclusions from #50519: incompatible FlyDSL TurboQuant
|
||||
# and a CUDA source check whose files are absent from the AMD test image.
|
||||
commands:
|
||||
- pytest -v -s
|
||||
kernels/test_apply_repetition_penalties.py
|
||||
kernels/test_awq_int4_to_int8.py
|
||||
kernels/test_bf16_skinny_gemm.py
|
||||
kernels/test_cache_kernels.py
|
||||
kernels/test_compressor_kv_cache.py
|
||||
kernels/test_cp_gather_fp8.py
|
||||
kernels/test_deepseek_v4_cpu_kernels.py
|
||||
kernels/test_dsv41_mega_attn_layouts.py
|
||||
kernels/test_engram.py
|
||||
kernels/test_flashinfer_bf16_gemm.py
|
||||
kernels/test_flex_attention.py
|
||||
kernels/test_fp32_router_gemm.py
|
||||
kernels/test_fused_deepseek_v32_norm_rope.py
|
||||
kernels/test_fused_indexer_q_rope_quant.py
|
||||
kernels/test_fused_inv_rope_fp8_quant.py
|
||||
kernels/test_fused_minimax_m3_qknorm_rope_kv_insert.py
|
||||
kernels/test_fused_quant_activation.py
|
||||
kernels/test_gate_linear_rocm_dispatch.py
|
||||
kernels/test_gemm_rs_ar.py
|
||||
kernels/test_kpool_decode_update_batched.py
|
||||
kernels/test_minimax_m3_amd_ops.py
|
||||
kernels/test_minimax_m3_sparse_attn_fp8_scale.py
|
||||
kernels/test_onednn.py
|
||||
kernels/test_relu2_fp8_quant.py
|
||||
kernels/test_rocm_fp32_router_gemm.py
|
||||
--deselect=tests/kernels/test_bf16_skinny_gemm.py::test_every_dsv3_routed_shape_is_instantiated
|
||||
|
||||
- label: ":nvidia: (L4) Attention Kernels Shard %N"
|
||||
key: kernels-attention-test
|
||||
@@ -725,6 +797,35 @@ steps:
|
||||
- pytest -v -s kernels/moe/test_flashinfer_moe.py
|
||||
- pytest -v -s kernels/moe/test_nvfp4_moe.py
|
||||
- pytest -v -s kernels/moe/test_ocp_mx_moe.py
|
||||
mirror:
|
||||
amd:
|
||||
label: ":amd: (MI355 DPX) FP4 and INT4 MoE Kernels"
|
||||
dind: false
|
||||
device: mi355_dpx
|
||||
num_devices: 1
|
||||
timeout_in_minutes: 40
|
||||
optional: false
|
||||
working_dir: /vllm-workspace/tests
|
||||
depends_on:
|
||||
- image-build-amd
|
||||
source_file_dependencies:
|
||||
- csrc/moe/
|
||||
- csrc/rocm/
|
||||
- vllm/model_executor/layers/fused_moe/
|
||||
- vllm/model_executor/layers/quantization/
|
||||
- vllm/_aiter_ops.py
|
||||
- vllm/platforms/rocm.py
|
||||
- vllm/envs.py
|
||||
- vllm/config/
|
||||
- tests/kernels/moe/test_rocm_aiter_moe.py
|
||||
- tests/kernels/moe/test_ocp_mx_moe.py
|
||||
- tests/kernels/moe/test_flydsl_moe.py
|
||||
- tests/kernels/moe/utils.py
|
||||
- tests/kernels/utils.py
|
||||
commands:
|
||||
- pytest -v -s kernels/moe/test_rocm_aiter_moe.py -k 'mi350_mxfp4 or quant_scheme_support_matches_gfx950'
|
||||
- pytest -v -s kernels/moe/test_ocp_mx_moe.py::test_rocm_mxfp4_moe_oracle -k AITER_MXFP4
|
||||
- pytest -v -s kernels/moe/test_flydsl_moe.py
|
||||
|
||||
|
||||
- label: ":nvidia: (H100) FusedMoE Layer Kernels"
|
||||
@@ -847,34 +948,6 @@ steps:
|
||||
- pytest -v -s kernels/quantization/test_rocm_mxfp8_linear.py
|
||||
- pytest -v -s kernels/quantization/test_rocm_skinny_gemms.py -k wvsplitkrc
|
||||
|
||||
- label: ":amd: (MI355 DPX) FP4 and INT4 MoE Kernels"
|
||||
key: amd-fp4-moe-kernels-mi355
|
||||
dind: false
|
||||
device: mi355_dpx
|
||||
num_devices: 1
|
||||
timeout_in_minutes: 40
|
||||
working_dir: /vllm-workspace/tests
|
||||
depends_on:
|
||||
- image-build-amd
|
||||
source_file_dependencies:
|
||||
- csrc/moe/
|
||||
- csrc/rocm/
|
||||
- vllm/model_executor/layers/fused_moe/
|
||||
- vllm/model_executor/layers/quantization/
|
||||
- vllm/_aiter_ops.py
|
||||
- vllm/platforms/rocm.py
|
||||
- vllm/envs.py
|
||||
- vllm/config/
|
||||
- tests/kernels/moe/test_rocm_aiter_moe.py
|
||||
- tests/kernels/moe/test_ocp_mx_moe.py
|
||||
- tests/kernels/moe/test_flydsl_moe.py
|
||||
- tests/kernels/moe/utils.py
|
||||
- tests/kernels/utils.py
|
||||
commands:
|
||||
- pytest -v -s kernels/moe/test_rocm_aiter_moe.py -k 'mi350_mxfp4 or quant_scheme_support_matches_gfx950'
|
||||
- pytest -v -s kernels/moe/test_ocp_mx_moe.py::test_rocm_mxfp4_moe_oracle -k AITER_MXFP4
|
||||
- pytest -v -s kernels/moe/test_flydsl_moe.py
|
||||
|
||||
- label: ":amd: (MI355 DPX) FP8 MoE Kernels"
|
||||
key: amd-fp8-moe-kernels-mi355
|
||||
dind: false
|
||||
|
||||
@@ -151,15 +151,18 @@ steps:
|
||||
label: ":amd: (MI355 DPX) LM Eval Small Models"
|
||||
dind: false
|
||||
device: mi355_dpx
|
||||
num_devices: 1
|
||||
timeout_in_minutes: 45
|
||||
working_dir: /vllm-workspace/tests
|
||||
depends_on:
|
||||
- image-build-amd
|
||||
source_file_dependencies:
|
||||
- .buildkite/test_areas/lm_eval.yaml
|
||||
- csrc/
|
||||
- tests/evals/gsm8k/
|
||||
- vllm/model_executor/layers/quantization
|
||||
- vllm/model_executor/kernels/linear/
|
||||
- vllm/model_executor/layers/fused_moe/
|
||||
- vllm/model_executor/models/
|
||||
- vllm/model_executor/model_loader/
|
||||
- vllm/v1/attention/backends/
|
||||
@@ -383,6 +386,49 @@ steps:
|
||||
- vllm/model_executor/layers/quantization
|
||||
commands:
|
||||
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-blackwell-ep.txt
|
||||
mirror:
|
||||
amd:
|
||||
label: ":amd: (MI355) LM Eval Large Models EP"
|
||||
dind: false
|
||||
device: mi355_2
|
||||
num_devices: 2
|
||||
timeout_in_minutes: 120
|
||||
working_dir: "/vllm-workspace/tests"
|
||||
depends_on:
|
||||
- image-build-amd
|
||||
source_file_dependencies:
|
||||
- .buildkite/test_areas/lm_eval.yaml
|
||||
- tests/evals/gsm8k/
|
||||
- tests/utils.py
|
||||
- tests/conftest.py
|
||||
- csrc/
|
||||
- vllm/config/
|
||||
- vllm/distributed/
|
||||
- vllm/model_executor/models/config.py
|
||||
- vllm/model_executor/models/registry.py
|
||||
- vllm/model_executor/models/utils.py
|
||||
- vllm/model_executor/models/qwen3_next.py
|
||||
- vllm/model_executor/models/qwen3_next_mtp.py
|
||||
- vllm/model_executor/models/nemotron_h.py
|
||||
- vllm/model_executor/models/nemotron_h_mtp.py
|
||||
- vllm/transformers_utils/configs/qwen3_next.py
|
||||
- vllm/transformers_utils/configs/nemotron_h.py
|
||||
- vllm/model_executor/model_loader/
|
||||
- vllm/model_executor/layers/quantization/
|
||||
- vllm/model_executor/layers/fused_moe/
|
||||
- vllm/model_executor/layers/mamba/
|
||||
- vllm/model_executor/layers/layernorm.py
|
||||
- vllm/model_executor/layers/linear.py
|
||||
- vllm/model_executor/kernels/linear/
|
||||
- vllm/third_party/flash_linear_attention/ops/
|
||||
- vllm/v1/attention/
|
||||
- vllm/v1/spec_decode/
|
||||
- vllm/v1/worker/
|
||||
- vllm/_aiter_ops.py
|
||||
- vllm/platforms/rocm.py
|
||||
commands:
|
||||
- export VLLM_WORKER_MULTIPROC_METHOD=spawn
|
||||
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-blackwell-ep-mi355.txt
|
||||
|
||||
- label: ":nvidia: (B200) LM Eval Qwen3.5 Models"
|
||||
key: lm-eval-qwen3-5-models-2xb200
|
||||
@@ -405,7 +451,7 @@ steps:
|
||||
label: ":amd: (MI355) LM Eval Qwen3.5 Models"
|
||||
dind: false
|
||||
device: mi355_2
|
||||
timeout_in_minutes: 40
|
||||
timeout_in_minutes: 75
|
||||
depends_on:
|
||||
- image-build-amd
|
||||
source_file_dependencies:
|
||||
@@ -418,10 +464,18 @@ steps:
|
||||
- vllm/model_executor/models/qwen3_next.py
|
||||
- vllm/model_executor/models/qwen3_next_mtp.py
|
||||
- vllm/third_party/flash_linear_attention/ops/
|
||||
- vllm/distributed/
|
||||
- vllm/model_executor/layers/fused_moe/
|
||||
- vllm/model_executor/layers/quantization/modelopt.py
|
||||
- vllm/model_executor/kernels/linear/nvfp4/
|
||||
- vllm/v1/attention/
|
||||
- vllm/v1/worker/
|
||||
- tests/evals/gsm8k/configs/models-qwen35-mi355.txt
|
||||
- tests/evals/gsm8k/configs/Qwen3.5-35B-A3B-DEP2.yaml
|
||||
- tests/evals/gsm8k/configs/Qwen3.5-35B-A3B-FP8-DEP2.yaml
|
||||
- tests/evals/gsm8k/configs/Qwen3.5-35B-A3B-MXFP4-AITER-TP2.yaml
|
||||
- tests/evals/gsm8k/configs/Qwen3.5-35B-A3B-MXFP4-EMU-TP2.yaml
|
||||
- tests/evals/gsm8k/configs/Qwen3.5-397B-A17B-NVFP4-DEP2.yaml
|
||||
- tests/evals/gsm8k/test_gsm8k_correctness.py
|
||||
- vllm/_aiter_ops.py
|
||||
- vllm/platforms/rocm.py
|
||||
@@ -436,6 +490,40 @@ steps:
|
||||
num_devices: 8
|
||||
commands:
|
||||
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-h200-deepseek-r1-tp.txt
|
||||
mirror:
|
||||
amd:
|
||||
label: ":amd: (MI355) LM Eval DeepSeek R1 TP"
|
||||
dind: false
|
||||
device: mi355_4
|
||||
num_devices: 4
|
||||
timeout_in_minutes: 65
|
||||
working_dir: /vllm-workspace/tests
|
||||
depends_on:
|
||||
- image-build-amd
|
||||
source_file_dependencies:
|
||||
- .buildkite/test_areas/lm_eval.yaml
|
||||
- tests/evals/gsm8k/
|
||||
- tests/utils.py
|
||||
- tests/conftest.py
|
||||
- vllm/config/
|
||||
- vllm/distributed/
|
||||
- vllm/model_executor/models/
|
||||
- vllm/models/
|
||||
- vllm/model_executor/model_loader/
|
||||
- vllm/model_executor/layers/quantization/
|
||||
- vllm/model_executor/layers/fused_moe/
|
||||
- vllm/v1/attention/
|
||||
- vllm/v1/spec_decode/
|
||||
- vllm/v1/worker/gpu/spec_decode/
|
||||
- vllm/v1/core/kv_cache_utils.py
|
||||
- vllm/model_executor/layers/layernorm.py
|
||||
- csrc/
|
||||
- vllm/_aiter_ops.py
|
||||
- vllm/platforms/rocm.py
|
||||
- vllm/model_executor/layers/sparse_attn_indexer.py
|
||||
commands:
|
||||
- export VLLM_WORKER_MULTIPROC_METHOD=spawn
|
||||
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-mi355-deepseek-r1-tp.txt
|
||||
|
||||
- label: ":nvidia: (H200) LM Eval DeepSeek R1 DP"
|
||||
key: lm-eval-deepseek-r1-dp-8xh200
|
||||
@@ -445,6 +533,40 @@ steps:
|
||||
num_devices: 8
|
||||
commands:
|
||||
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-h200-deepseek-r1-dp.txt
|
||||
mirror:
|
||||
amd:
|
||||
label: ":amd: (MI355) LM Eval DeepSeek R1 DP"
|
||||
dind: false
|
||||
device: mi355_4
|
||||
num_devices: 4
|
||||
timeout_in_minutes: 65
|
||||
working_dir: /vllm-workspace/tests
|
||||
depends_on:
|
||||
- image-build-amd
|
||||
source_file_dependencies:
|
||||
- .buildkite/test_areas/lm_eval.yaml
|
||||
- tests/evals/gsm8k/
|
||||
- tests/utils.py
|
||||
- tests/conftest.py
|
||||
- vllm/config/
|
||||
- vllm/distributed/
|
||||
- vllm/model_executor/models/
|
||||
- vllm/models/
|
||||
- vllm/model_executor/model_loader/
|
||||
- vllm/model_executor/layers/quantization/
|
||||
- vllm/model_executor/layers/fused_moe/
|
||||
- vllm/v1/attention/
|
||||
- vllm/v1/spec_decode/
|
||||
- vllm/v1/worker/gpu/spec_decode/
|
||||
- vllm/v1/core/kv_cache_utils.py
|
||||
- vllm/model_executor/layers/layernorm.py
|
||||
- csrc/
|
||||
- vllm/_aiter_ops.py
|
||||
- vllm/platforms/rocm.py
|
||||
- vllm/model_executor/layers/sparse_attn_indexer.py
|
||||
commands:
|
||||
- export VLLM_WORKER_MULTIPROC_METHOD=spawn
|
||||
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-mi355-deepseek-r1-dp.txt
|
||||
|
||||
- label: ":nvidia: (H200) LM Eval DeepSeek V3.2 TP"
|
||||
key: lm-eval-deepseek-v32-tp-8xh200
|
||||
@@ -454,6 +576,40 @@ steps:
|
||||
num_devices: 8
|
||||
commands:
|
||||
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-h200-deepseek-v32-tp.txt
|
||||
mirror:
|
||||
amd:
|
||||
label: ":amd: (MI355) LM Eval DeepSeek V3-2 TP"
|
||||
dind: false
|
||||
device: mi355_4
|
||||
num_devices: 4
|
||||
timeout_in_minutes: 65
|
||||
working_dir: /vllm-workspace/tests
|
||||
depends_on:
|
||||
- image-build-amd
|
||||
source_file_dependencies:
|
||||
- .buildkite/test_areas/lm_eval.yaml
|
||||
- tests/evals/gsm8k/
|
||||
- tests/utils.py
|
||||
- tests/conftest.py
|
||||
- vllm/config/
|
||||
- vllm/distributed/
|
||||
- vllm/model_executor/models/
|
||||
- vllm/models/
|
||||
- vllm/model_executor/model_loader/
|
||||
- vllm/model_executor/layers/quantization/
|
||||
- vllm/model_executor/layers/fused_moe/
|
||||
- vllm/v1/attention/
|
||||
- vllm/v1/spec_decode/
|
||||
- vllm/v1/worker/gpu/spec_decode/
|
||||
- vllm/v1/core/kv_cache_utils.py
|
||||
- vllm/model_executor/layers/layernorm.py
|
||||
- csrc/
|
||||
- vllm/_aiter_ops.py
|
||||
- vllm/platforms/rocm.py
|
||||
- vllm/model_executor/layers/sparse_attn_indexer.py
|
||||
commands:
|
||||
- export VLLM_WORKER_MULTIPROC_METHOD=spawn
|
||||
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-mi355-deepseek-v32-tp.txt
|
||||
|
||||
- label: ":nvidia: (H200) LM Eval DeepSeek V3.2 DP"
|
||||
key: lm-eval-deepseek-v32-dp-8xh200
|
||||
@@ -463,6 +619,40 @@ steps:
|
||||
num_devices: 8
|
||||
commands:
|
||||
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-h200-deepseek-v32-dp.txt
|
||||
mirror:
|
||||
amd:
|
||||
label: ":amd: (MI355) LM Eval DeepSeek V3-2 DP"
|
||||
dind: false
|
||||
device: mi355_4
|
||||
num_devices: 4
|
||||
timeout_in_minutes: 65
|
||||
working_dir: /vllm-workspace/tests
|
||||
depends_on:
|
||||
- image-build-amd
|
||||
source_file_dependencies:
|
||||
- .buildkite/test_areas/lm_eval.yaml
|
||||
- tests/evals/gsm8k/
|
||||
- tests/utils.py
|
||||
- tests/conftest.py
|
||||
- vllm/config/
|
||||
- vllm/distributed/
|
||||
- vllm/model_executor/models/
|
||||
- vllm/models/
|
||||
- vllm/model_executor/model_loader/
|
||||
- vllm/model_executor/layers/quantization/
|
||||
- vllm/model_executor/layers/fused_moe/
|
||||
- vllm/v1/attention/
|
||||
- vllm/v1/spec_decode/
|
||||
- vllm/v1/worker/gpu/spec_decode/
|
||||
- vllm/v1/core/kv_cache_utils.py
|
||||
- vllm/model_executor/layers/layernorm.py
|
||||
- csrc/
|
||||
- vllm/_aiter_ops.py
|
||||
- vllm/platforms/rocm.py
|
||||
- vllm/model_executor/layers/sparse_attn_indexer.py
|
||||
commands:
|
||||
- export VLLM_WORKER_MULTIPROC_METHOD=spawn
|
||||
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-mi355-deepseek-v32-dp.txt
|
||||
|
||||
- label: ":nvidia: (H200) LM Eval Nemotron 3 Super"
|
||||
key: lm-eval-nemotron-3-super-8xh200
|
||||
@@ -472,6 +662,40 @@ steps:
|
||||
num_devices: 8
|
||||
commands:
|
||||
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-h200-nemotron-3-super.txt
|
||||
mirror:
|
||||
amd:
|
||||
label: ":amd: (MI355) LM Eval Nemotron 3 Super"
|
||||
dind: false
|
||||
device: mi355_4
|
||||
num_devices: 4
|
||||
timeout_in_minutes: 65
|
||||
working_dir: /vllm-workspace/tests
|
||||
depends_on:
|
||||
- image-build-amd
|
||||
source_file_dependencies:
|
||||
- .buildkite/test_areas/lm_eval.yaml
|
||||
- tests/evals/gsm8k/
|
||||
- tests/utils.py
|
||||
- tests/conftest.py
|
||||
- vllm/config/
|
||||
- vllm/distributed/
|
||||
- vllm/model_executor/models/
|
||||
- vllm/models/
|
||||
- vllm/model_executor/model_loader/
|
||||
- vllm/model_executor/layers/quantization/
|
||||
- vllm/model_executor/layers/fused_moe/
|
||||
- vllm/v1/attention/
|
||||
- vllm/v1/spec_decode/
|
||||
- vllm/v1/worker/gpu/spec_decode/
|
||||
- vllm/v1/core/kv_cache_utils.py
|
||||
- vllm/model_executor/layers/layernorm.py
|
||||
- csrc/
|
||||
- vllm/_aiter_ops.py
|
||||
- vllm/platforms/rocm.py
|
||||
- vllm/model_executor/layers/mamba/
|
||||
commands:
|
||||
- export VLLM_WORKER_MULTIPROC_METHOD=spawn
|
||||
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-mi355-nemotron-3-super.txt
|
||||
|
||||
- label: ":amd: (MI300) LM Eval Large Models"
|
||||
key: amd-lm-eval-large-models
|
||||
@@ -685,9 +909,9 @@ steps:
|
||||
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=evals/gsm8k/configs/models-turboquant-k8v4.txt
|
||||
mirror:
|
||||
amd:
|
||||
label: ":amd: (MI355) LM Eval TurboQuant k8v4"
|
||||
label: ":amd: (MI355 DPX) LM Eval TurboQuant k8v4"
|
||||
dind: false
|
||||
device: mi355_1
|
||||
device: mi355_dpx
|
||||
num_devices: 1
|
||||
timeout_in_minutes: 30
|
||||
working_dir: /vllm-workspace/tests
|
||||
@@ -725,9 +949,9 @@ steps:
|
||||
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=evals/gsm8k/configs/models-turboquant-t4nc.txt
|
||||
mirror:
|
||||
amd:
|
||||
label: ":amd: (MI355) LM Eval TurboQuant t4nc"
|
||||
label: ":amd: (MI355 DPX) LM Eval TurboQuant t4nc"
|
||||
dind: false
|
||||
device: mi355_1
|
||||
device: mi355_dpx
|
||||
num_devices: 1
|
||||
timeout_in_minutes: 30
|
||||
working_dir: /vllm-workspace/tests
|
||||
@@ -765,9 +989,9 @@ steps:
|
||||
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=evals/gsm8k/configs/models-turboquant-k3v4nc.txt
|
||||
mirror:
|
||||
amd:
|
||||
label: ":amd: (MI355) LM Eval TurboQuant k3v4nc"
|
||||
label: ":amd: (MI355 DPX) LM Eval TurboQuant k3v4nc"
|
||||
dind: false
|
||||
device: mi355_1
|
||||
device: mi355_dpx
|
||||
num_devices: 1
|
||||
timeout_in_minutes: 30
|
||||
working_dir: /vllm-workspace/tests
|
||||
@@ -805,9 +1029,9 @@ steps:
|
||||
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=evals/gsm8k/configs/models-turboquant-t3nc.txt
|
||||
mirror:
|
||||
amd:
|
||||
label: ":amd: (MI355) LM Eval TurboQuant t3nc"
|
||||
label: ":amd: (MI355 DPX) LM Eval TurboQuant t3nc"
|
||||
dind: false
|
||||
device: mi355_1
|
||||
device: mi355_dpx
|
||||
num_devices: 1
|
||||
timeout_in_minutes: 30
|
||||
working_dir: /vllm-workspace/tests
|
||||
|
||||
@@ -530,7 +530,7 @@ steps:
|
||||
- vllm/v1/
|
||||
- vllm/version.py
|
||||
- cmake/utils.cmake
|
||||
- tests/test_access_log_filter.py
|
||||
- tests/test_uvicorn_logging.py
|
||||
- tests/test_attention_backend_registry.py
|
||||
- tests/test_audio_media_size_precheck.py
|
||||
- tests/test_cmake_utils.py
|
||||
@@ -571,7 +571,7 @@ steps:
|
||||
- pip install cmake # test_cmake_utils.py shells out to cmake; the CPU image has none
|
||||
- python3 standalone_tests/lazy_imports.py
|
||||
- pytest -v -s device_allocator
|
||||
- pytest -v -s test_access_log_filter.py
|
||||
- pytest -v -s test_uvicorn_logging.py
|
||||
- pytest -v -s test_attention_backend_registry.py
|
||||
- pytest -v -s test_audio_media_size_precheck.py
|
||||
- pytest -v -s test_cmake_utils.py
|
||||
|
||||
@@ -66,13 +66,15 @@ steps:
|
||||
- tests/models/transformers/test_backend.py
|
||||
- tests/models/test_registry.py
|
||||
- tests/models/test_deepseek_v4_vl_rocm.py
|
||||
- tests/models/test_deepseek_v41_decoder_replay_layers.py
|
||||
- tests/models/test_deepseek_v41_replay_batch.py
|
||||
commands:
|
||||
- pytest -v -s models/test_terratorch.py models/transformers/test_backend.py models/test_registry.py models/test_deepseek_v4_vl_rocm.py
|
||||
- pytest -v -s models/test_terratorch.py models/transformers/test_backend.py models/test_registry.py models/test_deepseek_v4_vl_rocm.py models/test_deepseek_v41_decoder_replay_layers.py models/test_deepseek_v41_replay_batch.py
|
||||
mirror:
|
||||
amd:
|
||||
label: ":amd: (MI355 DPX) Basic Models (Other)"
|
||||
label: ":amd: (MI355) Basic Models (Other)"
|
||||
dind: false
|
||||
device: mi355_dpx
|
||||
device: mi355_1
|
||||
timeout_in_minutes: 65
|
||||
depends_on:
|
||||
- image-build-amd
|
||||
@@ -176,7 +178,7 @@ steps:
|
||||
- vllm/v1/attention/backends/short_conv_attn.py
|
||||
- tests/models/qwen4_exp/
|
||||
commands:
|
||||
- pytest -v -s models/qwen4_exp/test_hc_ops.py models/qwen4_exp/test_qsa_pre_indexer.py models/qwen4_exp/test_qsa_reference.py
|
||||
- pytest -v -s models/qwen4_exp/test_hc_ops.py models/qwen4_exp/test_qsa_prepare.py models/qwen4_exp/test_qsa_reference.py
|
||||
mirror:
|
||||
amd:
|
||||
label: ":amd: (MI355 DPX) Qwen4 Exp"
|
||||
@@ -221,13 +223,3 @@ steps:
|
||||
device: cpu-small
|
||||
commands:
|
||||
- pytest -v -s models/test_utils.py models/test_vision.py models/test_adapters.py models/test_qwen3_5_mtp_config.py models/transformers/fusers/ model_executor/layers/test_activation.py
|
||||
mirror:
|
||||
amd:
|
||||
label: ":amd: (MI250) Basic Models Other"
|
||||
dind: false
|
||||
device: mi250_1
|
||||
no_gpu: true
|
||||
working_dir: "/vllm-workspace/tests"
|
||||
timeout_in_minutes: 35
|
||||
depends_on:
|
||||
- image-build-amd
|
||||
|
||||
@@ -26,8 +26,7 @@ steps:
|
||||
pytest models/language -v -s -m 'distributed(num_gpus=2)'
|
||||
;;
|
||||
2)
|
||||
pytest models/multimodal/generation/test_phi4siglip.py -v -s -m 'distributed(num_gpus=2)'
|
||||
pytest models/multimodal -v -s -m 'distributed(num_gpus=2)' --ignore models/multimodal/generation/test_whisper.py --ignore models/multimodal/generation/test_phi4siglip.py
|
||||
pytest models/multimodal -v -s -m 'distributed(num_gpus=2)' --ignore models/multimodal/generation/test_whisper.py
|
||||
VLLM_WORKER_MULTIPROC_METHOD=spawn pytest models/multimodal/generation/test_whisper.py -v -s -m 'distributed(num_gpus=2)'
|
||||
;;
|
||||
*) exit 2 ;;
|
||||
@@ -72,8 +71,7 @@ steps:
|
||||
VLLM_ROCM_USE_AITER=1 pytest -v -s models/test_hyv4_rocm.py -m 'distributed(num_gpus=2)'
|
||||
;;
|
||||
2)
|
||||
pytest models/multimodal -v -s -m 'distributed(num_gpus=2)' --ignore models/multimodal/generation/test_whisper.py --ignore models/multimodal/generation/test_phi4siglip.py &&
|
||||
pytest models/multimodal/generation/test_phi4siglip.py -v -s -m 'distributed(num_gpus=2)' &&
|
||||
pytest models/multimodal -v -s -m 'distributed(num_gpus=2)' --ignore models/multimodal/generation/test_whisper.py &&
|
||||
VLLM_WORKER_MULTIPROC_METHOD=spawn pytest models/multimodal/generation/test_whisper.py -v -s -m 'distributed(num_gpus=2)'
|
||||
;;
|
||||
*) exit 2 ;;
|
||||
|
||||
@@ -99,17 +99,6 @@ steps:
|
||||
commands:
|
||||
- pytest -v -s models/multimodal/processing --ignore models/multimodal/processing/test_tensor_schema.py --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT --shard-id=$$BUILDKITE_PARALLEL_JOB
|
||||
parallelism: 4
|
||||
mirror:
|
||||
amd:
|
||||
label: ":amd: (MI250) Multimodal Processor Shard %N"
|
||||
dind: false
|
||||
device: mi250_1
|
||||
no_gpu: true
|
||||
working_dir: "/vllm-workspace/tests"
|
||||
timeout_in_minutes: 90
|
||||
parallelism: 6
|
||||
depends_on:
|
||||
- image-build-amd
|
||||
|
||||
- label: ":nvidia: (H200 MIG 18GB) Multimodal Processor Shard %N"
|
||||
key: multi-modal-processor
|
||||
@@ -176,7 +165,7 @@ steps:
|
||||
label: ":amd: (MI300) Multimodal Models (Extended Generation 1)"
|
||||
dind: false
|
||||
device: mi300_1
|
||||
timeout_in_minutes: 90
|
||||
timeout_in_minutes: 100
|
||||
depends_on:
|
||||
- image-build-amd
|
||||
|
||||
|
||||
@@ -102,6 +102,7 @@ steps:
|
||||
depends_on:
|
||||
- image-build-amd
|
||||
source_file_dependencies:
|
||||
- tests/quantization/test_gfx950_moe.py
|
||||
- tests/quantization/test_rocm_moe.py
|
||||
- tests/utils.py
|
||||
- vllm/model_executor/models/deepseek_v2.py
|
||||
@@ -118,6 +119,7 @@ steps:
|
||||
- vllm/envs.py
|
||||
commands:
|
||||
- pytest -v -s tests/quantization/test_rocm_moe.py
|
||||
- pytest -v -s tests/quantization/test_gfx950_moe.py
|
||||
|
||||
- label: ":nvidia: (H200 MIG 35GB) Quantized Models"
|
||||
device: h200_35gb
|
||||
|
||||
@@ -423,6 +423,32 @@ steps:
|
||||
- tests/v1/e2e/spec_decode/
|
||||
commands:
|
||||
- pytest -v -s v1/e2e/spec_decode/acceptance_rates/dflash/ -k "dflash2"
|
||||
mirror:
|
||||
amd:
|
||||
label: ":amd: (MI355) Spec Decode AL DFlash2 Nightly"
|
||||
dind: false
|
||||
device: mi355_1
|
||||
num_devices: 1
|
||||
timeout_in_minutes: 60
|
||||
depends_on:
|
||||
- image-build-amd
|
||||
source_file_dependencies:
|
||||
- .buildkite/test_areas/spec_decode.yaml
|
||||
- tests/v1/e2e/spec_decode/
|
||||
- tests/evals/gsm8k/
|
||||
- tests/utils.py
|
||||
- vllm/v1/spec_decode/
|
||||
- vllm/v1/worker/gpu/spec_decode/
|
||||
- vllm/v1/worker/gpu/sample/
|
||||
- vllm/v1/worker/gpu/model_runner.py
|
||||
- vllm/v1/attention/
|
||||
- vllm/model_executor/models/qwen3_dflash.py
|
||||
- vllm/model_executor/models/qwen3_dflash2.py
|
||||
- vllm/model_executor/models/qwen3_5.py
|
||||
- vllm/model_executor/model_loader/
|
||||
- vllm/model_executor/layers/quantization/
|
||||
- vllm/model_executor/kernels/linear/
|
||||
- vllm/platforms/rocm.py
|
||||
|
||||
- label: ":nvidia: (H200 MIG 35GB) Spec Decode AL DSpark Nightly"
|
||||
key: spec-decode-dspark-nightly
|
||||
|
||||
Symlink
+1
@@ -0,0 +1 @@
|
||||
../../.agents/skills/check-api-compat
|
||||
Symlink
+1
@@ -0,0 +1 @@
|
||||
../../.agents/skills/pr-checklist
|
||||
+12
-4
@@ -3,10 +3,10 @@
|
||||
|
||||
# This lists cover the "core" components of vLLM that require careful review
|
||||
/vllm/compilation @zou3519 @youkaichao @ProExpertProg @BoyuanFeng
|
||||
/vllm/distributed/kv_transfer @NickLucche @ApostaC @orozery @xuechendi @ivanium
|
||||
/vllm/distributed/kv_transfer @NickLucche @ApostaC @orozery @xuechendi @ivanium @wzhao18
|
||||
/vllm/lora @jeejeelee
|
||||
/vllm/model_executor/layers/attention @LucasWilkinson @MatthewBonanni
|
||||
/vllm/model_executor/layers/fused_moe @mgoin @pavanimajety @zyongye
|
||||
/vllm/model_executor/layers/fused_moe @mgoin @pavanimajety @zyongye @wzhao18
|
||||
/vllm/model_executor/layers/quantization @mgoin @robertgshaw2-redhat @tlrmchlsmth @yewentao256 @pavanimajety @zyongye
|
||||
/vllm/model_executor/layers/mamba @tdoublep @tomeras91
|
||||
/vllm/model_executor/layers/mamba/gdn/qwen_gdn_linear_attn.py @tdoublep @ZJY0516 @vadiklyutiy
|
||||
@@ -67,7 +67,7 @@
|
||||
/vllm/v1/attention @LucasWilkinson @MatthewBonanni
|
||||
/vllm/v1/attention/backend.py @WoosukKwon @zhuohan123 @youkaichao @alexm-redhat @njhill
|
||||
/vllm/v1/attention/backends/mla @pavanimajety
|
||||
/vllm/v1/attention/backends/flashinfer.py @mgoin @pavanimajety @vadiklyutiy
|
||||
/vllm/v1/attention/backends/flashinfer.py @mgoin @pavanimajety @vadiklyutiy @wzhao18
|
||||
/vllm/v1/attention/backends/triton_attn.py @tdoublep
|
||||
/vllm/v1/attention/backends/gdn_attn.py @ZJY0516 @vadiklyutiy
|
||||
/vllm/v1/core @WoosukKwon @robertgshaw2-redhat @njhill @ywang96 @alexm-redhat @heheda12345 @ApostaC @orozery @ivanium
|
||||
@@ -113,7 +113,7 @@
|
||||
/tests/lora @jeejeelee
|
||||
/tests/models/language/generation/test_hybrid.py @tdoublep @tomeras91
|
||||
/tests/v1/kv_connector/nixl_integration @NickLucche
|
||||
/tests/v1/kv_connector @ApostaC @orozery @ivanium
|
||||
/tests/v1/kv_connector @ApostaC @orozery @ivanium @wzhao18
|
||||
/tests/v1/kv_offload @ApostaC @orozery
|
||||
/tests/v1/simple_kv_offload @ivanium
|
||||
/tests/v1/determinism @yewentao256
|
||||
@@ -158,6 +158,7 @@ mkdocs.yaml @hmellor
|
||||
# Qwen-specific files
|
||||
/vllm/model_executor/models/qwen* @sighingnow @vadiklyutiy
|
||||
/vllm/transformers_utils/configs/qwen* @sighingnow @vadiklyutiy
|
||||
/vllm/models/qwen4_exp @gau-nernst
|
||||
|
||||
# MTP-specific files
|
||||
/vllm/model_executor/models/deepseek_mtp.py @luccafong
|
||||
@@ -166,6 +167,9 @@ mkdocs.yaml @hmellor
|
||||
/vllm/models/deepseek_v4 @zyongye
|
||||
/vllm/model_executor/layers/sparse_attn_indexer.py @zyongye
|
||||
|
||||
# Kimi-K3-specific files
|
||||
/vllm/models/kimi_k3/nvidia/kda.py @gau-nernst
|
||||
|
||||
# Mistral-specific files
|
||||
/vllm/model_executor/models/mistral*.py @patrickvonplaten
|
||||
/vllm/model_executor/models/mixtral*.py @patrickvonplaten
|
||||
@@ -178,6 +182,10 @@ mkdocs.yaml @hmellor
|
||||
/vllm/v1/attention/ops/chunked_prefill_paged_decode.py @tdoublep
|
||||
/vllm/v1/attention/ops/triton_unified_attention.py @tdoublep
|
||||
/vllm/third_party/flash_linear_attention @ZJY0516 @vadiklyutiy
|
||||
/vllm/model_executor/layers/mamba/ops/gdn_chunk_cutedsl @gau-nernst
|
||||
/vllm/model_executor/layers/fused_moe/router/bf16x3_router_gemm_cutedsl.py @gau-nernst
|
||||
/vllm/model_executor/kernels/linear/cute_dsl/gemm_rs_ar.py @gau-nernst
|
||||
/vllm/cute_utils @gau-nernst
|
||||
|
||||
# ROCm related: specify owner with write access to notify AMD folks for careful code review
|
||||
/vllm/**/*rocm* @tjtanaa @dllehr-amd @shen-shanshan
|
||||
|
||||
@@ -1,20 +1,33 @@
|
||||
<!-- markdownlint-disable -->
|
||||
PLEASE FILL IN THE PR DESCRIPTION HERE ENSURING ALL CHECKLIST ITEMS (AT THE BOTTOM) HAVE BEEN CONSIDERED.
|
||||
|
||||
## Purpose
|
||||
## Overview
|
||||
|
||||
## Test Plan
|
||||
<!-- Summarize the change in 1-2 lines. Link relevant issues and PRs. -->
|
||||
|
||||
## Test Result
|
||||
## Claims
|
||||
|
||||
<!-- State the accomplishments in a few concrete bullets: speedups, feature support, bug fixes, etc. -->
|
||||
|
||||
## Validation
|
||||
|
||||
<!-- Show evidence that the claims are met by this change, and steps to reproduce validations. -->
|
||||
|
||||
## Details
|
||||
|
||||
<!-- Explain motivation, design decisions, root cause, tradeoffs, and limitations concisely. Link or collapse lengthy supporting material. -->
|
||||
|
||||
---
|
||||
<details>
|
||||
<summary> Essential Elements of an Effective PR Description Checklist </summary>
|
||||
|
||||
- [ ] The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
|
||||
- [ ] The test plan, such as providing test command.
|
||||
- [ ] The test results, such as pasting the results comparison before and after, or e2e results
|
||||
- [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
|
||||
<details>
|
||||
<summary> Pull Request Checklist </summary>
|
||||
|
||||
- [ ] I used vLLM's `/pr-checklist` skill. (Mandatory for agents, optional for humans).
|
||||
- [ ] AI assistance was used during the creation of this PR.
|
||||
|
||||
- [ ] **Design Fit:** Minimizes impact on core components, reuses existing functionality, and justifies added complexity.
|
||||
- [ ] **Testing and Validation:** Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.
|
||||
- [ ] **Code Quality and Style:** Keeps code and comments clear and concise, and updates relevant documentation and examples.
|
||||
- [ ] **Pull Request Contents:** Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.
|
||||
</details>
|
||||
|
||||
**BEFORE SUBMITTING, PLEASE READ <https://docs.vllm.ai/en/latest/contributing>** (anything written below this line will be removed by GitHub Actions)
|
||||
|
||||
+41
-2
@@ -5,7 +5,6 @@ pull_request_rules:
|
||||
- label != stale
|
||||
- -closed
|
||||
- or:
|
||||
- files~=^[^/]+\.md$
|
||||
- files~=^docs/
|
||||
- files~=^examples/
|
||||
actions:
|
||||
@@ -572,7 +571,7 @@ pull_request_rules:
|
||||
- base = main
|
||||
- label=ready
|
||||
- "#approved-reviews-by >= 1"
|
||||
- "#commits-behind >= 50"
|
||||
- "#commits-behind >= 100"
|
||||
- "#check-failure = 0"
|
||||
- -closed
|
||||
- -draft
|
||||
@@ -661,6 +660,30 @@ pull_request_rules:
|
||||
add:
|
||||
- kv-connector
|
||||
|
||||
- name: label-mooncake
|
||||
description: Automatically apply mooncake label
|
||||
conditions:
|
||||
- label != stale
|
||||
- or:
|
||||
- files~=^vllm/distributed/kv_transfer/kv_connector/v1/mooncake/
|
||||
# Covers the mooncake/ connector dir, mooncake_store_embedding/ and
|
||||
# mooncake_ec_connector.py.
|
||||
- files~=^vllm/distributed/ec_transfer/ec_connector/mooncake
|
||||
- files=vllm/distributed/mooncake_store.py
|
||||
- files~=^tests/v1/kv_connector/.*mooncake
|
||||
- files~=^tests/v1/ec_connector/.*mooncake
|
||||
- files~=^docs/features/mooncake.*\.md
|
||||
- files=.buildkite/test_areas/disaggregated_mooncake.yaml
|
||||
- title~=(?i)mooncake
|
||||
actions:
|
||||
label:
|
||||
add:
|
||||
- mooncake
|
||||
assign:
|
||||
users:
|
||||
- "stmatengss"
|
||||
- "LCAIZJ"
|
||||
|
||||
- name: label-scheduler
|
||||
description: Automatically apply scheduler label
|
||||
conditions:
|
||||
@@ -725,3 +748,19 @@ pull_request_rules:
|
||||
label:
|
||||
add:
|
||||
- vllm-ir
|
||||
|
||||
- name: label-agents
|
||||
description: Automatically apply agents label
|
||||
conditions:
|
||||
- label != stale
|
||||
- or:
|
||||
- files~=(?:^|/)AGENTS\.md$
|
||||
- files~=^\.agents/
|
||||
- files~=^\.claude/
|
||||
# The [Agents] title tag only: a bare "agents" also matches CI, K8s and
|
||||
# NIXL agents.
|
||||
- title~=(?i)\[agents\]
|
||||
actions:
|
||||
label:
|
||||
add:
|
||||
- agents
|
||||
|
||||
@@ -31,6 +31,7 @@ Do not open one-off PRs for tiny edits (single typo, isolated style change, one
|
||||
- Test commands run and results.
|
||||
- Model evaluation results when the change affects output, accuracy, or serving.
|
||||
- Clear statement that AI assistance was used.
|
||||
- Use vLLM's [`/pr-checklist`](.agents/skills/pr-checklist/SKILL.md) skill before submitting a PR to ensure quality and adherence to the contribution guidelines.
|
||||
|
||||
### Fail-closed behavior
|
||||
|
||||
|
||||
@@ -590,9 +590,30 @@ if(VLLM_GPU_LANG STREQUAL "CUDA" OR VLLM_GPU_LANG STREQUAL "HIP")
|
||||
endif()
|
||||
endif()
|
||||
|
||||
# The fused MiniMax-M3 KV insert quantizes NVFP4 with SM100-family-only
|
||||
# cvt instructions, so its SM10x/11x code is built for the family targets.
|
||||
set(MINIMAX_M3_FUSED_SRC
|
||||
"csrc/libtorch_stable/fused_minimax_m3_qknorm_rope_kv_insert_kernel.cu")
|
||||
list(REMOVE_ITEM VLLM_STABLE_EXT_SRC "${MINIMAX_M3_FUSED_SRC}")
|
||||
set_gencode_flags_for_srcs(
|
||||
SRCS "${VLLM_STABLE_EXT_SRC}"
|
||||
CUDA_ARCHS "${CUDA_ARCHS}")
|
||||
if(${CMAKE_CUDA_COMPILER_VERSION} VERSION_GREATER_EQUAL 13.0)
|
||||
cuda_archs_loose_intersection(MINIMAX_M3_FUSED_ARCHS
|
||||
"10.0f;10.7f;11.0f" "${CUDA_ARCHS}")
|
||||
else()
|
||||
cuda_archs_loose_intersection(MINIMAX_M3_FUSED_ARCHS
|
||||
"10.0a;10.1a;10.3a" "${CUDA_ARCHS}")
|
||||
endif()
|
||||
foreach(_ARCH ${CUDA_ARCHS})
|
||||
if(_ARCH VERSION_LESS 10.0 OR _ARCH VERSION_GREATER_EQUAL 12.0)
|
||||
list(APPEND MINIMAX_M3_FUSED_ARCHS "${_ARCH}")
|
||||
endif()
|
||||
endforeach()
|
||||
set_gencode_flags_for_srcs(
|
||||
SRCS "${MINIMAX_M3_FUSED_SRC}"
|
||||
CUDA_ARCHS "${MINIMAX_M3_FUSED_ARCHS}")
|
||||
list(APPEND VLLM_STABLE_EXT_SRC "${MINIMAX_M3_FUSED_SRC}")
|
||||
|
||||
if(COOPERATIVE_TOPK_ARCHS)
|
||||
list(APPEND VLLM_STABLE_EXT_SRC
|
||||
|
||||
@@ -0,0 +1,217 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
"""Micro-benchmark for CPU sampling kernels.
|
||||
|
||||
Compares fused Gumbel-max / greedy argmax against the baseline
|
||||
(softmax → exponential → div → argmax) across vocab and batch sizes.
|
||||
|
||||
Usage:
|
||||
.venv/bin/python benchmarks/kernels/bench_cpu_sampling.py
|
||||
.venv/bin/python benchmarks/kernels/bench_cpu_sampling.py --profile
|
||||
.venv/bin/python benchmarks/kernels/bench_cpu_sampling.py --vocab 128256 --batch 16
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import time
|
||||
|
||||
import torch
|
||||
import vllm._C # noqa: F401
|
||||
from torch.profiler import ProfilerActivity, profile, record_function
|
||||
|
||||
|
||||
def baseline_random_sample(logits: torch.Tensor) -> torch.Tensor:
|
||||
probs = logits.softmax(dim=-1, dtype=torch.float32)
|
||||
q = torch.empty_like(probs)
|
||||
q.exponential_()
|
||||
return probs.div(q).argmax(dim=-1).view(-1)
|
||||
|
||||
|
||||
def baseline_greedy_sample(logits: torch.Tensor) -> torch.Tensor:
|
||||
return logits.argmax(dim=-1).view(-1)
|
||||
|
||||
|
||||
def bench_latency(fn, args, n_warmup=20, n_iters=500):
|
||||
for _ in range(n_warmup):
|
||||
fn(*args)
|
||||
|
||||
t0 = time.perf_counter()
|
||||
for _ in range(n_iters):
|
||||
fn(*args)
|
||||
elapsed = time.perf_counter() - t0
|
||||
return elapsed / n_iters * 1e6 # µs
|
||||
|
||||
|
||||
def run_profile(logits, seeds, n_iters=50):
|
||||
"""Run torch.profiler and print comparison tables."""
|
||||
# Profile baseline random sampling
|
||||
with profile(
|
||||
activities=[ProfilerActivity.CPU],
|
||||
record_shapes=True,
|
||||
) as prof_base_rand:
|
||||
for _ in range(n_iters):
|
||||
with record_function("baseline_random"):
|
||||
baseline_random_sample(logits)
|
||||
|
||||
# Profile fused Gumbel-max
|
||||
with profile(
|
||||
activities=[ProfilerActivity.CPU],
|
||||
record_shapes=True,
|
||||
) as prof_fused:
|
||||
for _ in range(n_iters):
|
||||
with record_function("fused_gumbel_argmax"):
|
||||
torch.ops._C.fused_gumbel_argmax(logits, seeds)
|
||||
|
||||
# Profile baseline greedy
|
||||
with profile(
|
||||
activities=[ProfilerActivity.CPU],
|
||||
record_shapes=True,
|
||||
) as prof_base_grdy:
|
||||
for _ in range(n_iters):
|
||||
with record_function("baseline_greedy"):
|
||||
baseline_greedy_sample(logits)
|
||||
|
||||
# Profile custom greedy
|
||||
with profile(
|
||||
activities=[ProfilerActivity.CPU],
|
||||
record_shapes=True,
|
||||
) as prof_cust_grdy:
|
||||
for _ in range(n_iters):
|
||||
with record_function("custom_greedy_argmax"):
|
||||
torch.ops._C.greedy_argmax(logits)
|
||||
|
||||
B, V = logits.shape
|
||||
print(f"\n{'=' * 80}")
|
||||
print(f"torch.profiler breakdown (batch={B}, vocab={V}, iters={n_iters})")
|
||||
print(f"{'=' * 80}")
|
||||
|
||||
print("\n--- Baseline Random (softmax → exp → div → argmax) ---")
|
||||
print(prof_base_rand.key_averages().table(sort_by="cpu_time_total", row_limit=15))
|
||||
|
||||
print("\n--- Fused Gumbel-max (table lookup + add + argmax) ---")
|
||||
print(prof_fused.key_averages().table(sort_by="cpu_time_total", row_limit=15))
|
||||
|
||||
print("\n--- Baseline Greedy (torch.argmax) ---")
|
||||
print(prof_base_grdy.key_averages().table(sort_by="cpu_time_total", row_limit=15))
|
||||
|
||||
print("\n--- Custom Greedy (vec_op argmax) ---")
|
||||
print(prof_cust_grdy.key_averages().table(sort_by="cpu_time_total", row_limit=15))
|
||||
|
||||
# Summary comparison
|
||||
def avg_us(prof, label):
|
||||
for e in prof.key_averages():
|
||||
if e.key == label:
|
||||
return e.cpu_time_total / e.count
|
||||
return 0.0
|
||||
|
||||
t_br = avg_us(prof_base_rand, "baseline_random")
|
||||
t_fg = avg_us(prof_fused, "fused_gumbel_argmax")
|
||||
t_bg = avg_us(prof_base_grdy, "baseline_greedy")
|
||||
t_cg = avg_us(prof_cust_grdy, "custom_greedy_argmax")
|
||||
|
||||
print(f"\n{'=' * 60}")
|
||||
print(f" Summary (batch={B}, vocab={V})")
|
||||
print(f"{'=' * 60}")
|
||||
print(f" {'Kernel':<30} {'Avg (µs)':>10} {'Speedup':>10}")
|
||||
print(f" {'-' * 50}")
|
||||
print(f" {'baseline random':<30} {t_br:>10.1f} {'—':>10}")
|
||||
print(
|
||||
f" {'fused gumbel-max':<30} {t_fg:>10.1f} "
|
||||
f"{t_br / t_fg if t_fg > 0 else 0:>9.2f}x"
|
||||
)
|
||||
print(f" {'baseline greedy':<30} {t_bg:>10.1f} {'—':>10}")
|
||||
print(
|
||||
f" {'custom greedy':<30} {t_cg:>10.1f} {t_bg / t_cg if t_cg > 0 else 0:>9.2f}x"
|
||||
)
|
||||
print(f"{'=' * 60}")
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="Benchmark CPU sampling kernels")
|
||||
parser.add_argument(
|
||||
"--vocab",
|
||||
type=int,
|
||||
nargs="+",
|
||||
default=[32000, 49152, 128256],
|
||||
help="Vocab sizes to benchmark",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--batch",
|
||||
type=int,
|
||||
nargs="+",
|
||||
default=[1, 4, 16],
|
||||
help="Batch sizes to benchmark",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--profile",
|
||||
action="store_true",
|
||||
help="Run torch.profiler and export chrome trace",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--iters", type=int, default=500, help="Iterations per measurement"
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
header = (
|
||||
f"{'batch':>5} {'vocab':>7} "
|
||||
f"{'base_rand':>10} {'fused_rand':>10} {'rand_spdup':>10} "
|
||||
f"{'base_grdy':>10} {'cust_grdy':>10} {'grdy_spdup':>10}"
|
||||
)
|
||||
units = (
|
||||
f"{'':>5} {'':>7} "
|
||||
f"{'(µs)':>10} {'(µs)':>10} {'':>10} "
|
||||
f"{'(µs)':>10} {'(µs)':>10} {'':>10}"
|
||||
)
|
||||
print("\n" + "=" * len(header))
|
||||
print("CPU Sampling Kernel Benchmark")
|
||||
print("=" * len(header))
|
||||
print(header)
|
||||
print(units)
|
||||
print("-" * len(header))
|
||||
|
||||
for V in args.vocab:
|
||||
for B in args.batch:
|
||||
logits = torch.randn(B, V, dtype=torch.float32)
|
||||
seeds = torch.arange(B, dtype=torch.long)
|
||||
|
||||
t_base_rand = bench_latency(
|
||||
baseline_random_sample, (logits,), n_iters=args.iters
|
||||
)
|
||||
t_fused = bench_latency(
|
||||
torch.ops._C.fused_gumbel_argmax, (logits, seeds), n_iters=args.iters
|
||||
)
|
||||
|
||||
t_base_grdy = bench_latency(
|
||||
baseline_greedy_sample, (logits,), n_iters=args.iters
|
||||
)
|
||||
t_cust_grdy = bench_latency(
|
||||
torch.ops._C.greedy_argmax, (logits,), n_iters=args.iters
|
||||
)
|
||||
|
||||
rand_speedup = t_base_rand / t_fused if t_fused > 0 else 0
|
||||
grdy_speedup = t_base_grdy / t_cust_grdy if t_cust_grdy > 0 else 0
|
||||
|
||||
print(
|
||||
f"{B:>5} {V:>7} "
|
||||
f"{t_base_rand:>10.1f} {t_fused:>10.1f} "
|
||||
f"{rand_speedup:>9.2f}x "
|
||||
f"{t_base_grdy:>10.1f} {t_cust_grdy:>10.1f} "
|
||||
f"{grdy_speedup:>9.2f}x"
|
||||
)
|
||||
|
||||
print("-" * len(header))
|
||||
|
||||
if args.profile:
|
||||
print("\nRunning torch.profiler (batch=16, vocab=128256) ...")
|
||||
logits = torch.randn(16, 128256, dtype=torch.float32)
|
||||
seeds = torch.arange(16, dtype=torch.long)
|
||||
|
||||
# warmup
|
||||
for _ in range(10):
|
||||
baseline_random_sample(logits)
|
||||
torch.ops._C.fused_gumbel_argmax(logits, seeds)
|
||||
|
||||
run_profile(logits, seeds)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,256 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
"""Benchmark the NIXL host-buffer KV copies across KV cache groups.
|
||||
|
||||
`NixlBaseConnectorWorker.sync_recved_kv_to_device` / `save_kv_to_host` move a
|
||||
request's blocks between the CPU transfer buffer and the device KV cache. Block
|
||||
ids are unique across KV cache groups, so the per-group copies can be issued as
|
||||
one. This measures what that coalescing saves: the copied bytes are identical,
|
||||
only the number of launches changes.
|
||||
"""
|
||||
|
||||
import time
|
||||
|
||||
import torch
|
||||
from tabulate import tabulate
|
||||
|
||||
from vllm.distributed.kv_transfer.kv_connector.utils import copy_kv_blocks
|
||||
from vllm.logger import init_logger
|
||||
from vllm.utils.argparse_utils import FlexibleArgumentParser
|
||||
from vllm.utils.torch_utils import STR_DTYPE_TO_TORCH_DTYPE
|
||||
|
||||
logger = init_logger(__name__)
|
||||
|
||||
|
||||
def _make_caches(
|
||||
num_layers: int,
|
||||
num_blocks: int,
|
||||
block_size: int,
|
||||
num_heads: int,
|
||||
head_size: int,
|
||||
dtype: torch.dtype,
|
||||
device: str,
|
||||
) -> tuple[dict[str, torch.Tensor], dict[str, torch.Tensor]]:
|
||||
"""Host transfer buffers and device KV caches, shaped as the worker builds them."""
|
||||
shape = (num_blocks, block_size, num_heads, head_size)
|
||||
host = {
|
||||
f"layer.{i}": torch.randn(shape, dtype=dtype, device="cpu")
|
||||
for i in range(num_layers)
|
||||
}
|
||||
device_caches = {
|
||||
f"layer.{i}": torch.zeros(shape, dtype=dtype, device=device)
|
||||
for i in range(num_layers)
|
||||
}
|
||||
return host, device_caches
|
||||
|
||||
|
||||
def _split_into_groups(block_ids: list[int], num_groups: int) -> list[list[int]]:
|
||||
"""Partition block ids into groups, mirroring the global BlockPool id space."""
|
||||
per_group = len(block_ids) // num_groups
|
||||
return [block_ids[i * per_group : (i + 1) * per_group] for i in range(num_groups)]
|
||||
|
||||
|
||||
def _percentile(sorted_values: list[float], q: float) -> float:
|
||||
return sorted_values[min(len(sorted_values) - 1, int(len(sorted_values) * q))]
|
||||
|
||||
|
||||
@torch.inference_mode()
|
||||
def _time_paired(
|
||||
host: dict[str, torch.Tensor],
|
||||
device_caches: dict[str, torch.Tensor],
|
||||
group_block_ids: list[list[int]],
|
||||
direction: str,
|
||||
num_iters: int,
|
||||
) -> tuple[list[float], list[float]]:
|
||||
"""Per-group and coalesced latencies, measured alternately.
|
||||
|
||||
Interleaving the two variants keeps drift in machine state from landing on
|
||||
one of them: each iteration times both, back to back.
|
||||
"""
|
||||
src, dst = (host, device_caches) if direction == "h2d" else (device_caches, host)
|
||||
merged = [[b for group in group_block_ids for b in group]]
|
||||
|
||||
def _run(batches: list[list[int]]) -> float:
|
||||
start = time.perf_counter()
|
||||
for ids in batches:
|
||||
copy_kv_blocks(src, dst, ids, ids, direction)
|
||||
torch.accelerator.synchronize()
|
||||
return time.perf_counter() - start
|
||||
|
||||
for _ in range(5): # warmup
|
||||
_run(group_block_ids)
|
||||
_run(merged)
|
||||
|
||||
per_group: list[float] = []
|
||||
coalesced: list[float] = []
|
||||
for i in range(num_iters):
|
||||
# Alternate which variant goes first so ordering cannot favour either.
|
||||
if i % 2:
|
||||
per_group.append(_run(group_block_ids))
|
||||
coalesced.append(_run(merged))
|
||||
else:
|
||||
coalesced.append(_run(merged))
|
||||
per_group.append(_run(group_block_ids))
|
||||
return sorted(per_group), sorted(coalesced)
|
||||
|
||||
|
||||
@torch.inference_mode()
|
||||
def _count_device_ops(
|
||||
host: dict[str, torch.Tensor],
|
||||
device_caches: dict[str, torch.Tensor],
|
||||
group_block_ids: list[list[int]],
|
||||
direction: str,
|
||||
coalesced: bool,
|
||||
) -> dict[str, int]:
|
||||
"""Device-op counts for one request-sync, via the profiler.
|
||||
|
||||
Counts do not depend on how busy the machine is, so this is the part of
|
||||
the comparison that reproduces exactly.
|
||||
"""
|
||||
from torch.profiler import ProfilerActivity, profile
|
||||
|
||||
src, dst = (host, device_caches) if direction == "h2d" else (device_caches, host)
|
||||
batches = (
|
||||
[[b for group in group_block_ids for b in group]]
|
||||
if coalesced
|
||||
else group_block_ids
|
||||
)
|
||||
|
||||
for ids in batches: # warm up allocator/autograd caches
|
||||
copy_kv_blocks(src, dst, ids, ids, direction)
|
||||
torch.accelerator.synchronize()
|
||||
|
||||
with profile(activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA]) as prof:
|
||||
for ids in batches:
|
||||
copy_kv_blocks(src, dst, ids, ids, direction)
|
||||
torch.accelerator.synchronize()
|
||||
|
||||
counts: dict[str, int] = {}
|
||||
for evt in prof.key_averages():
|
||||
if evt.count and (evt.self_device_time_total or evt.device_time_total):
|
||||
counts[evt.key] = counts.get(evt.key, 0) + evt.count
|
||||
return counts
|
||||
|
||||
|
||||
def _report_op_counts(host, device_caches, block_ids, groups):
|
||||
print("\nDevice-op counts per request-sync (profiler; independent of load):")
|
||||
rows = []
|
||||
for num_groups in groups:
|
||||
group_block_ids = _split_into_groups(block_ids, num_groups)
|
||||
for direction in ("h2d", "d2h"):
|
||||
before = _count_device_ops(
|
||||
host, device_caches, group_block_ids, direction, coalesced=False
|
||||
)
|
||||
after = _count_device_ops(
|
||||
host, device_caches, group_block_ids, direction, coalesced=True
|
||||
)
|
||||
for key in sorted(set(before) | set(after)):
|
||||
b, a = before.get(key, 0), after.get(key, 0)
|
||||
if b != a:
|
||||
rows.append([num_groups, direction, key, b, a])
|
||||
print(
|
||||
tabulate(
|
||||
rows,
|
||||
headers=["groups", "direction", "op", "per-group", "coalesced"],
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def main(args):
|
||||
dtype = STR_DTYPE_TO_TORCH_DTYPE[args.dtype]
|
||||
torch.manual_seed(args.seed)
|
||||
|
||||
host, device_caches = _make_caches(
|
||||
num_layers=args.num_layers,
|
||||
num_blocks=args.num_blocks,
|
||||
block_size=args.block_size,
|
||||
num_heads=args.num_heads,
|
||||
head_size=args.head_size,
|
||||
dtype=dtype,
|
||||
device="cuda",
|
||||
)
|
||||
block_ids = list(range(args.blocks_per_request))
|
||||
|
||||
rows = []
|
||||
for num_groups in args.groups:
|
||||
group_block_ids = _split_into_groups(block_ids, num_groups)
|
||||
for direction in ("h2d", "d2h"):
|
||||
per_group, coalesced = _time_paired(
|
||||
host,
|
||||
device_caches,
|
||||
group_block_ids,
|
||||
direction,
|
||||
num_iters=args.iters,
|
||||
)
|
||||
# Interference only ever adds time, so the minimum is the most
|
||||
# stable estimator here; the p10 spread above it says how noisy
|
||||
# the machine was while sampling.
|
||||
best_before = per_group[0]
|
||||
best_after = coalesced[0]
|
||||
spread = (_percentile(coalesced, 0.5) - best_after) / best_after
|
||||
rows.append(
|
||||
[
|
||||
num_groups,
|
||||
direction,
|
||||
best_before * 1e6,
|
||||
best_after * 1e6,
|
||||
(best_before - best_after) / best_before * 100.0,
|
||||
spread * 100.0,
|
||||
]
|
||||
)
|
||||
|
||||
print(
|
||||
f"layers={args.num_layers} num_blocks={args.num_blocks} "
|
||||
f"block_size={args.block_size} heads={args.num_heads} "
|
||||
f"head_size={args.head_size} dtype={args.dtype} "
|
||||
f"blocks/request={args.blocks_per_request} iters={args.iters}"
|
||||
)
|
||||
print(
|
||||
tabulate(
|
||||
rows,
|
||||
headers=[
|
||||
"groups",
|
||||
"direction",
|
||||
"per-group min (µs)",
|
||||
"coalesced min (µs)",
|
||||
"saved (%)",
|
||||
"p50 over min (%)",
|
||||
],
|
||||
floatfmt=".3f",
|
||||
)
|
||||
)
|
||||
|
||||
if args.op_counts:
|
||||
_report_op_counts(host, device_caches, block_ids, args.groups)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
parser = FlexibleArgumentParser()
|
||||
parser.add_argument("--num-layers", type=int, default=48)
|
||||
parser.add_argument("--num-blocks", type=int, default=2048)
|
||||
parser.add_argument("--block-size", type=int, choices=[16, 32], default=16)
|
||||
parser.add_argument("--num-heads", type=int, default=8)
|
||||
parser.add_argument("--head-size", type=int, default=128)
|
||||
parser.add_argument("--blocks-per-request", type=int, default=60)
|
||||
parser.add_argument(
|
||||
"--groups",
|
||||
type=int,
|
||||
nargs="+",
|
||||
default=[1, 2, 3, 6],
|
||||
help="KV cache group counts to sweep (1 = non-hybrid model).",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--dtype",
|
||||
type=str,
|
||||
choices=["half", "bfloat16", "float"],
|
||||
default="bfloat16",
|
||||
)
|
||||
parser.add_argument("--iters", type=int, default=100)
|
||||
parser.add_argument(
|
||||
"--op-counts",
|
||||
action="store_true",
|
||||
help="Also report profiler op counts, which do not vary with load.",
|
||||
)
|
||||
parser.add_argument("--seed", type=int, default=0)
|
||||
|
||||
main(parser.parse_args())
|
||||
@@ -501,6 +501,7 @@ set(VLLM_EXT_SRC
|
||||
"csrc/cpu/cpu_fused_moe.cpp"
|
||||
"csrc/cpu/cpu_attn.cpp"
|
||||
"csrc/cpu/cpu_isa.cpp"
|
||||
"csrc/cpu/sampling_kernels.cpp"
|
||||
"csrc/cpu/torch_bindings.cpp")
|
||||
|
||||
if (CMAKE_SYSTEM_PROCESSOR MATCHES "riscv64" AND VLLM_RVV_VLEN AND
|
||||
@@ -523,14 +524,16 @@ if (ASIMD_FOUND AND NOT APPLE_SILICON_FOUND)
|
||||
"csrc/cpu/cpu_tanhf_neon.hpp"
|
||||
${VLLM_EXT_SRC})
|
||||
if (ARM_BF16_FOUND)
|
||||
set(VLLM_EXT_SRC "csrc/cpu/sgl-kernels/conv.cpp" ${VLLM_EXT_SRC})
|
||||
if (ARM_I8MM_FOUND)
|
||||
set(VLLM_EXT_SRC "csrc/cpu/cpu_fused_moe_int8.cpp" ${VLLM_EXT_SRC})
|
||||
endif()
|
||||
endif()
|
||||
endif()
|
||||
|
||||
if (POWER9_FOUND OR POWER10_FOUND OR POWER11_FOUND)
|
||||
if (POWER9_FOUND OR POWER10_FOUND OR POWER11_FOUND)
|
||||
set(VLLM_EXT_SRC
|
||||
"csrc/cpu/cpu_wna16.cpp"
|
||||
"csrc/cpu/shm.cpp"
|
||||
${VLLM_EXT_SRC})
|
||||
endif()
|
||||
@@ -589,6 +592,7 @@ if (ENABLE_X86_ISA)
|
||||
"csrc/cpu/cpu_isa.cpp"
|
||||
"csrc/cpu/dnnl_kernels.cpp"
|
||||
"csrc/cpu/mamba_cpu.cpp"
|
||||
"csrc/cpu/sampling_kernels.cpp"
|
||||
"csrc/cpu/torch_bindings.cpp"
|
||||
# TODO: Remove these files
|
||||
"csrc/cpu/activation.cpp"
|
||||
@@ -606,6 +610,7 @@ if (ENABLE_X86_ISA)
|
||||
"csrc/cpu/cpu_isa.cpp"
|
||||
"csrc/cpu/mamba_cpu.cpp"
|
||||
"csrc/cpu/dnnl_kernels.cpp"
|
||||
"csrc/cpu/sampling_kernels.cpp"
|
||||
"csrc/cpu/torch_bindings.cpp"
|
||||
# TODO: Remove these files
|
||||
"csrc/cpu/activation.cpp"
|
||||
|
||||
@@ -17,7 +17,7 @@ else()
|
||||
FetchContent_Declare(
|
||||
fmha_sm100
|
||||
GIT_REPOSITORY https://github.com/vllm-project/MSA.git
|
||||
GIT_TAG f355c37eb4e1413f21ee2ad8bbad25079e6bef9d
|
||||
GIT_TAG be4a52aea420b6bef73be9af1a649859964790cb
|
||||
GIT_PROGRESS TRUE
|
||||
CONFIGURE_COMMAND ""
|
||||
BUILD_COMMAND ""
|
||||
@@ -39,6 +39,8 @@ install(FILES
|
||||
"${FMHA_SM100_PY_ROOT}/api.py"
|
||||
"${FMHA_SM100_PY_ROOT}/bench_utils.py"
|
||||
"${FMHA_SM100_PY_ROOT}/jit.py"
|
||||
"${FMHA_SM100_PY_ROOT}/nvfp4_kv.py"
|
||||
"${FMHA_SM100_PY_ROOT}/q8kv4_decode_adapter.py"
|
||||
"${FMHA_SM100_PY_ROOT}/sparse.py"
|
||||
"${FMHA_SM100_PY_ROOT}/sparse_fmha_adapter.py"
|
||||
DESTINATION vllm/third_party/fmha_sm100
|
||||
@@ -51,6 +53,18 @@ install(DIRECTORY "${FMHA_SM100_PY_ROOT}/csrc/"
|
||||
PATTERN "*.pyc" EXCLUDE
|
||||
PATTERN ".git*" EXCLUDE)
|
||||
|
||||
install(DIRECTORY "${FMHA_SM100_PY_ROOT}/decode_q8kv4/"
|
||||
DESTINATION vllm/third_party/fmha_sm100/decode_q8kv4
|
||||
COMPONENT fmha_sm100
|
||||
PATTERN "__pycache__" EXCLUDE
|
||||
PATTERN "*.pyc" EXCLUDE)
|
||||
|
||||
install(DIRECTORY "${FMHA_SM100_PY_ROOT}/kvouter/"
|
||||
DESTINATION vllm/third_party/fmha_sm100/kvouter
|
||||
COMPONENT fmha_sm100
|
||||
PATTERN "__pycache__" EXCLUDE
|
||||
PATTERN "*.pyc" EXCLUDE)
|
||||
|
||||
install(DIRECTORY "${FMHA_SM100_PY_ROOT}/cute/"
|
||||
DESTINATION vllm/third_party/fmha_sm100/cute
|
||||
COMPONENT fmha_sm100
|
||||
|
||||
@@ -417,7 +417,7 @@ struct FP32Vec8 : public Vec<FP32Vec8> {
|
||||
tmp.val[1] = two_x;
|
||||
FP32Vec8 temp_vec(tmp);
|
||||
FP32Vec8 exp_vec = temp_vec.exp();
|
||||
vector float e = temp_vec.exp().reg.val[0];
|
||||
vector float e = exp_vec.reg.val[0];
|
||||
|
||||
vector float num = vec_sub(e, one);
|
||||
vector float den = vec_add(e, one);
|
||||
@@ -624,6 +624,24 @@ struct FP32Vec16 : public Vec<FP32Vec16> {
|
||||
|
||||
explicit FP32Vec16(const BF16Vec8& v) : FP32Vec16(FP32Vec8(v)) {}
|
||||
|
||||
// De-pack 16 x 4-bit nibbles from a 64-bit value and convert with offset.
|
||||
// Used by WNA16 (AWQ/GPTQ) dequantization.
|
||||
// Fully vectorized using VSX / AltiVec intrinsics with zero stack stores.
|
||||
FORCE_INLINE explicit FP32Vec16(int64_t value, const FP32Vec16& lut) {
|
||||
uint64_t u = static_cast<uint64_t>(value);
|
||||
float f[16];
|
||||
#pragma GCC unroll 16
|
||||
for (int i = 0; i < 16; i++) {
|
||||
f[i] = static_cast<float>((u >> (i * 4)) & 0xF);
|
||||
}
|
||||
// Extract base offset from LUT[0] (0.0f for AWQ, -8.0f for GPTQ)
|
||||
__vector float offset = vec_splats(vec_extract(lut.reg.val[0], 0));
|
||||
reg.val[0] = vec_add(vec_xl(0, f), offset);
|
||||
reg.val[1] = vec_add(vec_xl(16, f), offset);
|
||||
reg.val[2] = vec_add(vec_xl(32, f), offset);
|
||||
reg.val[3] = vec_add(vec_xl(48, f), offset);
|
||||
}
|
||||
|
||||
// FP8 stub: dead code on PowerPC (fp8 KV cache is x86-only), needed for
|
||||
// load_b_pair_vec template to compile on all platforms.
|
||||
explicit FP32Vec16(const BF16Vec32&, int) : reg{} {}
|
||||
@@ -1102,6 +1120,32 @@ inline void prefetch(const void* addr) {
|
||||
__asm__ __volatile__("dcbt 0, %0" : : "r"(addr) : "memory");
|
||||
}
|
||||
|
||||
static void interleave_save(const BF16Vec16& vec0, const BF16Vec16& vec1,
|
||||
void* ptr) {
|
||||
__vector signed short h0 = vec_mergeh(vec0.reg.val[0], vec1.reg.val[0]);
|
||||
__vector signed short l0 = vec_mergel(vec0.reg.val[0], vec1.reg.val[0]);
|
||||
__vector signed short h1 = vec_mergeh(vec0.reg.val[1], vec1.reg.val[1]);
|
||||
__vector signed short l1 = vec_mergel(vec0.reg.val[1], vec1.reg.val[1]);
|
||||
|
||||
vec_xst(h0, 0, reinterpret_cast<signed short*>(ptr));
|
||||
vec_xst(l0, 16, reinterpret_cast<signed short*>(ptr));
|
||||
vec_xst(h1, 32, reinterpret_cast<signed short*>(ptr));
|
||||
vec_xst(l1, 48, reinterpret_cast<signed short*>(ptr));
|
||||
}
|
||||
|
||||
static void interleave_save(const FP16Vec16& vec0, const FP16Vec16& vec1,
|
||||
void* ptr) {
|
||||
__vector signed short h0 = vec_mergeh(vec0.reg.val[0], vec1.reg.val[0]);
|
||||
__vector signed short l0 = vec_mergel(vec0.reg.val[0], vec1.reg.val[0]);
|
||||
__vector signed short h1 = vec_mergeh(vec0.reg.val[1], vec1.reg.val[1]);
|
||||
__vector signed short l1 = vec_mergel(vec0.reg.val[1], vec1.reg.val[1]);
|
||||
|
||||
vec_xst(h0, 0, reinterpret_cast<signed short*>(ptr));
|
||||
vec_xst(l0, 16, reinterpret_cast<signed short*>(ptr));
|
||||
vec_xst(h1, 32, reinterpret_cast<signed short*>(ptr));
|
||||
vec_xst(l1, 48, reinterpret_cast<signed short*>(ptr));
|
||||
}
|
||||
|
||||
struct INT8Vec64 {
|
||||
__vector signed char data[4];
|
||||
INT8Vec64() = default;
|
||||
|
||||
+31
-2
@@ -7,6 +7,9 @@
|
||||
#if defined(__riscv_v)
|
||||
#include "cpu/micro_gemm/cpu_micro_gemm_rvv.hpp"
|
||||
#endif
|
||||
#if defined(__powerpc__)
|
||||
#include "cpu/micro_gemm/cpu_micro_gemm_vsx.hpp"
|
||||
#endif
|
||||
#include "cpu/micro_gemm/cpu_micro_gemm_vec.hpp"
|
||||
|
||||
#define VLLM_DISPATCH_CASE_16B_TYPES(...) \
|
||||
@@ -104,8 +107,8 @@ class Dequantizer4b {
|
||||
scalar_vec_t output_vec_0(wb_0);
|
||||
scalar_vec_t output_vec_1(wb_1);
|
||||
|
||||
// AMX needs to interleave K elements to pack as 32 bits
|
||||
if constexpr (isa == ISA::AMX) {
|
||||
// AMX and VSX need to interleave K elements to pack as 32 bits
|
||||
if constexpr (isa == ISA::AMX || isa == ISA::VSX) {
|
||||
vec_op::interleave_save(output_vec_0, output_vec_1, curr_weight);
|
||||
} else {
|
||||
output_vec_0.save(curr_weight);
|
||||
@@ -303,6 +306,8 @@ void cpu_gemm_wna16(
|
||||
return ISA::VEC;
|
||||
} else if (isa_hint == "rvv") {
|
||||
return ISA::RVV;
|
||||
} else if (isa_hint == "vsx") {
|
||||
return ISA::VSX;
|
||||
} else {
|
||||
TORCH_CHECK(false, "unsupported isa hint: " + isa_hint);
|
||||
}
|
||||
@@ -378,6 +383,30 @@ void cpu_gemm_wna16(
|
||||
scales_group_stride, zeros_group_stride, group_size, pack_factor);
|
||||
return;
|
||||
}
|
||||
} else if (isa == ISA::VSX) {
|
||||
TORCH_CHECK(input.scalar_type() == at::ScalarType::BFloat16,
|
||||
"VSX WNA16 GEMM only supports BFloat16");
|
||||
using gemm_t = cpu_micro_gemm::MicroGemm<ISA::VSX, scalar_t>;
|
||||
if (has_zp) {
|
||||
using dequantizer_t = Dequantizer4b<scalar_t, ISA::VSX, true>;
|
||||
cpu_gemm_wna16_impl<scalar_t, dequantizer_t, gemm_t>(
|
||||
input.data_ptr<scalar_t>(), q_weight.data_ptr<int32_t>(),
|
||||
output.data_ptr<scalar_t>(), scales.data_ptr<scalar_t>(), zeros_ptr,
|
||||
bias.has_value() ? bias->data_ptr<scalar_t>() : nullptr, a_m_size,
|
||||
b_n_size, a_k_size, a_m_stride, output_m_stride,
|
||||
scales_group_stride, zeros_group_stride, group_size, pack_factor);
|
||||
return;
|
||||
}
|
||||
{
|
||||
using dequantizer_t = Dequantizer4b<scalar_t, ISA::VSX, false>;
|
||||
cpu_gemm_wna16_impl<scalar_t, dequantizer_t, gemm_t>(
|
||||
input.data_ptr<scalar_t>(), q_weight.data_ptr<int32_t>(),
|
||||
output.data_ptr<scalar_t>(), scales.data_ptr<scalar_t>(), zeros_ptr,
|
||||
bias.has_value() ? bias->data_ptr<scalar_t>() : nullptr, a_m_size,
|
||||
b_n_size, a_k_size, a_m_stride, output_m_stride,
|
||||
scales_group_stride, zeros_group_stride, group_size, pack_factor);
|
||||
return;
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
@@ -0,0 +1,151 @@
|
||||
#include "cpu_types.hpp"
|
||||
|
||||
#include <torch/library.h>
|
||||
|
||||
#include <cmath>
|
||||
#include <cstdint>
|
||||
#include <limits>
|
||||
#include <mutex>
|
||||
#include <random>
|
||||
|
||||
namespace {
|
||||
|
||||
constexpr int GUMBEL_TABLE_SIZE = 1 << 20;
|
||||
constexpr int GUMBEL_TABLE_MASK = GUMBEL_TABLE_SIZE - 1;
|
||||
|
||||
static float g_gumbel_table[GUMBEL_TABLE_SIZE];
|
||||
static std::once_flag g_gumbel_init_flag;
|
||||
|
||||
static void init_gumbel_table() {
|
||||
std::mt19937 rng(42);
|
||||
std::uniform_real_distribution<float> uniform(1.0e-7f, 1.0f - 1.0e-7f);
|
||||
for (int i = 0; i < GUMBEL_TABLE_SIZE; ++i) {
|
||||
float u = uniform(rng);
|
||||
g_gumbel_table[i] = -std::log(-std::log(u));
|
||||
}
|
||||
}
|
||||
|
||||
static inline void ensure_gumbel_table() {
|
||||
std::call_once(g_gumbel_init_flag, init_gumbel_table);
|
||||
}
|
||||
|
||||
// Fused Gumbel-max kernel
|
||||
|
||||
static void fused_gumbel_argmax_kernel(int64_t* __restrict__ output,
|
||||
const float* __restrict__ logits,
|
||||
const int64_t* __restrict__ seeds,
|
||||
const int64_t batch_size,
|
||||
const int64_t vocab_size) {
|
||||
ensure_gumbel_table();
|
||||
constexpr int VEC_ELEM_NUM = vec_op::FP32Vec8::VEC_ELEM_NUM;
|
||||
const int64_t vec_end = vocab_size - (vocab_size % VEC_ELEM_NUM);
|
||||
|
||||
#pragma omp parallel for schedule(static)
|
||||
for (int64_t b = 0; b < batch_size; ++b) {
|
||||
const float* row = logits + b * vocab_size;
|
||||
const uint32_t seed = static_cast<uint32_t>(seeds[b]);
|
||||
|
||||
float best_score = -std::numeric_limits<float>::infinity();
|
||||
int64_t best_idx = 0;
|
||||
|
||||
for (int64_t i = 0; i < vec_end; i += VEC_ELEM_NUM) {
|
||||
vec_op::FP32Vec8 logit_vec(row + i);
|
||||
|
||||
float gumbel_buf[VEC_ELEM_NUM];
|
||||
for (int k = 0; k < VEC_ELEM_NUM; ++k)
|
||||
gumbel_buf[k] = g_gumbel_table[(seed + i + k) & GUMBEL_TABLE_MASK];
|
||||
vec_op::FP32Vec8 gumbel_vec(gumbel_buf);
|
||||
|
||||
vec_op::FP32Vec8 score = logit_vec + gumbel_vec;
|
||||
|
||||
float score_buf[VEC_ELEM_NUM];
|
||||
score.save(score_buf);
|
||||
for (int k = 0; k < VEC_ELEM_NUM; ++k) {
|
||||
if (score_buf[k] > best_score) {
|
||||
best_score = score_buf[k];
|
||||
best_idx = i + k;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
for (int64_t i = vec_end; i < vocab_size; ++i) {
|
||||
float s = row[i] + g_gumbel_table[(seed + i) & GUMBEL_TABLE_MASK];
|
||||
if (s > best_score) {
|
||||
best_score = s;
|
||||
best_idx = i;
|
||||
}
|
||||
}
|
||||
|
||||
output[b] = best_idx;
|
||||
}
|
||||
}
|
||||
|
||||
static void greedy_argmax_kernel(int64_t* __restrict__ output,
|
||||
const float* __restrict__ logits,
|
||||
const int64_t batch_size,
|
||||
const int64_t vocab_size) {
|
||||
constexpr int VEC_ELEM_NUM = vec_op::FP32Vec16::VEC_ELEM_NUM;
|
||||
const int64_t vec_end = vocab_size - (vocab_size % VEC_ELEM_NUM);
|
||||
|
||||
#pragma omp parallel for schedule(static)
|
||||
for (int64_t b = 0; b < batch_size; ++b) {
|
||||
const float* row = logits + b * vocab_size;
|
||||
|
||||
vec_op::FP32Vec16 vmax(-std::numeric_limits<float>::infinity());
|
||||
for (int64_t i = 0; i < vec_end; i += VEC_ELEM_NUM) {
|
||||
vmax = vmax.max(vec_op::FP32Vec16(row + i));
|
||||
}
|
||||
float best_val = vmax.reduce_max();
|
||||
for (int64_t i = vec_end; i < vocab_size; ++i) {
|
||||
if (row[i] > best_val) {
|
||||
best_val = row[i];
|
||||
}
|
||||
}
|
||||
|
||||
int64_t best_idx = 0;
|
||||
for (int64_t i = 0; i < vocab_size; ++i) {
|
||||
if (row[i] == best_val) {
|
||||
best_idx = i;
|
||||
break;
|
||||
}
|
||||
}
|
||||
output[b] = best_idx;
|
||||
}
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
torch::Tensor fused_gumbel_argmax(const torch::Tensor& logits,
|
||||
const torch::Tensor& seeds) {
|
||||
TORCH_CHECK(logits.dim() == 2, "logits must be 2-D [batch, vocab]");
|
||||
TORCH_CHECK(logits.scalar_type() == torch::kFloat32,
|
||||
"logits must be float32");
|
||||
TORCH_CHECK(seeds.dim() == 1 && seeds.size(0) == logits.size(0),
|
||||
"seeds must be 1-D with batch_size elements");
|
||||
|
||||
auto logits_contig = logits.contiguous();
|
||||
auto output = torch::empty({logits_contig.size(0)}, torch::kInt64);
|
||||
fused_gumbel_argmax_kernel(
|
||||
output.data_ptr<int64_t>(), logits_contig.data_ptr<float>(),
|
||||
seeds.data_ptr<int64_t>(), logits_contig.size(0), logits_contig.size(1));
|
||||
return output;
|
||||
}
|
||||
|
||||
torch::Tensor greedy_argmax(const torch::Tensor& logits) {
|
||||
TORCH_CHECK(logits.dim() == 2, "logits must be 2-D [batch, vocab]");
|
||||
TORCH_CHECK(logits.scalar_type() == torch::kFloat32,
|
||||
"logits must be float32");
|
||||
|
||||
// A single row is faster in ATen: OpenMP fork/join dominates the scan
|
||||
// (~1 ms vs ~50 µs on s390x at vocab=32k).
|
||||
if (logits.size(0) <= 1) {
|
||||
return logits.argmax(/*dim=*/-1);
|
||||
}
|
||||
|
||||
auto logits_contig = logits.contiguous();
|
||||
auto output = torch::empty({logits_contig.size(0)}, torch::kInt64);
|
||||
greedy_argmax_kernel(output.data_ptr<int64_t>(),
|
||||
logits_contig.data_ptr<float>(), logits_contig.size(0),
|
||||
logits_contig.size(1));
|
||||
return output;
|
||||
}
|
||||
@@ -66,6 +66,10 @@ struct tinygemm_kernel {
|
||||
}
|
||||
};
|
||||
|
||||
#if defined(ARM_BF16_SUPPORT)
|
||||
#include "conv_arm.h"
|
||||
#endif
|
||||
|
||||
#if defined(CPU_CAPABILITY_AVX512)
|
||||
template <int K, int BLOCK_N, bool has_bias, bool has_silu>
|
||||
struct tinygemm_kernel<at::BFloat16, K, BLOCK_N, has_bias, has_silu> {
|
||||
@@ -543,16 +547,24 @@ at::Tensor causal_conv1d_weight_pack(const at::Tensor& weight) {
|
||||
|
||||
int64_t dim = weight.size(0);
|
||||
int64_t width = weight.size(1);
|
||||
constexpr int64_t BLOCK_N = block_size_n();
|
||||
constexpr int64_t BLOCK_N =
|
||||
#if defined(ARM_BF16_SUPPORT)
|
||||
8;// 8 channel blocks for microtile
|
||||
#else
|
||||
block_size_n();
|
||||
#endif
|
||||
TORCH_CHECK(width == 4, "causal_conv1d_weight_pack: support only width of 4");
|
||||
TORCH_CHECK(dim % BLOCK_N == 0, "causal_conv1d_weight_pack: invalid dim size ", dim);
|
||||
TORCH_CHECK(
|
||||
dim % block_size_n() == 0,
|
||||
"causal_conv1d_weight_pack: invalid dim size ",
|
||||
dim);
|
||||
|
||||
const int64_t N = dim, K2 = width >> 1;
|
||||
const int64_t NB = div_up(N, BLOCK_N);
|
||||
|
||||
auto packed_weight = at::empty_like(weight);
|
||||
AT_DISPATCH_REDUCED_FLOATING_TYPES(weight.scalar_type(), "causal_conv1d_fwd_kernel_impl", [&] {
|
||||
// cast to float32 as vnni size is 2
|
||||
// Copy each pair as an opaque 32-bit word. This is not a bf16->fp32 conversion.
|
||||
const float* w_data = reinterpret_cast<float*>(weight.data_ptr<scalar_t>());
|
||||
float* packed_data = reinterpret_cast<float*>(packed_weight.data_ptr<scalar_t>());
|
||||
|
||||
@@ -642,9 +654,9 @@ at::Tensor causal_conv1d_fwd_cpu(
|
||||
const std::optional<at::Tensor>& has_initial_state,
|
||||
bool silu_activation,
|
||||
int64_t pad_slot_id,
|
||||
bool is_vnni) {
|
||||
bool is_weight_packed) {
|
||||
CHECK_CONTIGUOUS(weight);
|
||||
auto packed_w = is_vnni ? weight : causal_conv1d_weight_pack(weight);
|
||||
auto packed_w = is_weight_packed ? weight : causal_conv1d_weight_pack(weight);
|
||||
|
||||
const bool is_var_seqlen = query_start_loc.has_value();
|
||||
const int64_t input_ndim = is_var_seqlen ? 2 : 3;
|
||||
@@ -755,10 +767,10 @@ at::Tensor causal_conv1d_update_cpu(
|
||||
const std::optional<at::Tensor>& num_accepted_tokens,
|
||||
const std::optional<at::Tensor>& conv_state_indices,
|
||||
int64_t pad_slot_id,
|
||||
bool is_vnni) {
|
||||
bool is_weight_packed) {
|
||||
CHECK_CONTIGUOUS(x);
|
||||
CHECK_CONTIGUOUS(weight);
|
||||
auto packed_w = is_vnni ? weight : causal_conv1d_weight_pack(weight);
|
||||
auto packed_w = is_weight_packed ? weight : causal_conv1d_weight_pack(weight);
|
||||
|
||||
TORCH_CHECK(
|
||||
x.dim() == 2 || x.dim() == 3,
|
||||
|
||||
@@ -0,0 +1,126 @@
|
||||
// SPDX-License-Identifier: Apache-2.0
|
||||
// SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
|
||||
#pragma once
|
||||
|
||||
#if defined(ARM_BF16_SUPPORT)
|
||||
|
||||
template <int K, int BLOCK_N, bool has_bias, bool has_silu>
|
||||
struct tinygemm_kernel<at::BFloat16, K, BLOCK_N, has_bias, has_silu> {
|
||||
static inline bfloat16x8_t load_bf16x8(const at::BFloat16* ptr) {
|
||||
return vld1q_bf16(reinterpret_cast<const bfloat16_t*>(ptr));
|
||||
}
|
||||
|
||||
// Input layout: A[m][channel]
|
||||
// Weight layout: B[N/MICRO_N, K/2, MICRO_N, 2]
|
||||
// The packed weights are reused across all M positions.
|
||||
static inline void tinygemm_neon_8(
|
||||
const at::BFloat16* __restrict__ A, const at::BFloat16* __restrict__ B,
|
||||
at::BFloat16* __restrict__ C, const at::BFloat16* __restrict__ bias,
|
||||
const at::BFloat16* __restrict__ conv_states, bool has_initial_state,
|
||||
int64_t M, int64_t lda, bool is_first_token) {
|
||||
bfloat16x8_t va[4];
|
||||
// Load 4 pairs of kernel weights into 4 registers
|
||||
const bfloat16x8_t vb0 = load_bf16x8(B); // channels 0-3, taps 0-1
|
||||
const bfloat16x8_t vb1 = load_bf16x8(B + 8); // channels 4-7, taps 0-1
|
||||
const bfloat16x8_t vb2 = load_bf16x8(B + 16); // channels 0-3, taps 2-3
|
||||
const bfloat16x8_t vb3 = load_bf16x8(B + 24); // channels 4-7, taps 2-3
|
||||
|
||||
// Load the input data. A single register holds the values for a
|
||||
// single token across all 8 channels for this microtile.
|
||||
auto load_a = [&](int64_t m) {
|
||||
if (m < 0 && is_first_token) {
|
||||
if (!has_initial_state) {
|
||||
return vreinterpretq_bf16_u16(vdupq_n_u16(0));
|
||||
}
|
||||
return load_bf16x8(conv_states + (m + K - 1) * lda);
|
||||
}
|
||||
return load_bf16x8(A + m * lda);
|
||||
};
|
||||
|
||||
// Load the previous inputs from the cached state if available.
|
||||
va[1] = load_a(-3);
|
||||
va[2] = load_a(-2);
|
||||
va[3] = load_a(-1);
|
||||
|
||||
// Iterate over the sequence block, updating the previous conv inputs
|
||||
// and reading the next token's input data (8 channels).
|
||||
for (int64_t m = 0; m < M; ++m) {
|
||||
va[0] = va[1];
|
||||
va[1] = va[2];
|
||||
va[2] = va[3];
|
||||
va[3] = load_a(m);
|
||||
|
||||
float32x4_t vc0;
|
||||
float32x4_t vc1;
|
||||
if constexpr (has_bias) {
|
||||
const bfloat16x8_t bias_vec = load_bf16x8(bias);
|
||||
vc0 = vcvtq_low_f32_bf16(bias_vec);
|
||||
vc1 = vcvtq_high_f32_bf16(bias_vec);
|
||||
} else {
|
||||
vc0 = vdupq_n_f32(0.0f);
|
||||
vc1 = vdupq_n_f32(0.0f);
|
||||
}
|
||||
|
||||
// Use the zip operation to match the input layout to the weights layout
|
||||
bfloat16x8_t va_pair =
|
||||
vreinterpretq_bf16_u16( // channels 0-3, inputs for taps 0-1
|
||||
vzip1q_u16(vreinterpretq_u16_bf16(va[0]),
|
||||
vreinterpretq_u16_bf16(va[1])));
|
||||
vc0 = vbfdotq_f32(vc0, va_pair,
|
||||
vb0); // 4 independent length-two dot products
|
||||
va_pair = vreinterpretq_bf16_u16( // channels 4-7, inputs for taps 0-1
|
||||
vzip2q_u16(vreinterpretq_u16_bf16(va[0]),
|
||||
vreinterpretq_u16_bf16(va[1])));
|
||||
vc1 = vbfdotq_f32(vc1, va_pair, vb1);
|
||||
va_pair = vreinterpretq_bf16_u16( // channels 0-3, inputs for taps 2-3
|
||||
vzip1q_u16(vreinterpretq_u16_bf16(va[2]),
|
||||
vreinterpretq_u16_bf16(va[3])));
|
||||
// Add to the previous result for the same channels
|
||||
vc0 = vbfdotq_f32(vc0, va_pair, vb2);
|
||||
va_pair = vreinterpretq_bf16_u16( // channels 4-7, inputs for taps 2-3
|
||||
vzip2q_u16(vreinterpretq_u16_bf16(va[2]),
|
||||
vreinterpretq_u16_bf16(va[3])));
|
||||
vc1 = vbfdotq_f32(vc1, va_pair, vb3);
|
||||
|
||||
using fVec = at::vec::Vectorized<float>;
|
||||
fVec out0(vc0);
|
||||
fVec out1(vc1);
|
||||
if constexpr (has_silu) {
|
||||
const fVec one(1.0f);
|
||||
out0 = out0 / (one + out0.neg().fexp_u20());
|
||||
out1 = out1 / (one + out1.neg().fexp_u20());
|
||||
}
|
||||
|
||||
// Convert the 2 x 4 fp32 results to 1 x 8 bf16
|
||||
const bfloat16x8_t out =
|
||||
vcombine_bf16(vcvt_bf16_f32(static_cast<float32x4_t>(out0)),
|
||||
vcvt_bf16_f32(static_cast<float32x4_t>(out1)));
|
||||
vst1q_bf16(reinterpret_cast<bfloat16_t*>(C + m * lda), out);
|
||||
}
|
||||
}
|
||||
|
||||
static inline void apply(const at::BFloat16* __restrict__ A,
|
||||
const at::BFloat16* __restrict__ B,
|
||||
at::BFloat16* __restrict__ C,
|
||||
const at::BFloat16* __restrict__ bias,
|
||||
const at::BFloat16* __restrict__ conv_states,
|
||||
bool has_initial_state, int64_t M, int64_t lda,
|
||||
bool is_first_token) {
|
||||
static_assert(K == 4);
|
||||
constexpr int64_t MICRO_N = 8;
|
||||
static_assert(BLOCK_N % MICRO_N == 0);
|
||||
|
||||
// BLOCK_N is always 32 or 64. M is the block sequence length.
|
||||
// Input layout: A[m][channel]
|
||||
// Weight layout: B[N/MICRO_N, K/2, MICRO_N, 2]
|
||||
// Use an 8 channel microtile and iterate over the 8 channel blocks.
|
||||
for (int64_t n = 0; n < BLOCK_N; n += MICRO_N) {
|
||||
tinygemm_neon_8(A + n, B + n * K, C + n, has_bias ? bias + n : nullptr,
|
||||
conv_states ? conv_states + n : nullptr,
|
||||
has_initial_state, M, lda, is_first_token);
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
#endif
|
||||
@@ -2306,21 +2306,35 @@ std::tuple<at::Tensor, at::Tensor> chunk_gated_delta_rule_cpu(
|
||||
chunk_gated_delta_rule_fwd_intra<CHUNK_SIZE>(key_, value, g_, beta, cu_seqlens, chunk_indices);
|
||||
|
||||
// fused `chunk_gated_delta_rule_fwd_h` + `chunk_fwd_o`
|
||||
auto launch_inter = [&](auto state_tag) {
|
||||
auto launch_inter = [local_query = query_,
|
||||
local_key = key_,
|
||||
local_w = w,
|
||||
local_u = u,
|
||||
local_decay_mask = decay_mask,
|
||||
local_chunk_offsets = chunk_offsets,
|
||||
g_,
|
||||
initial_state,
|
||||
output_final_state,
|
||||
cu_seqlens,
|
||||
initial_state_indices](auto state_tag) {
|
||||
using state_t = decltype(state_tag);
|
||||
return chunk_gated_delta_rule_fwd_inter<state_t, CHUNK_SIZE>(
|
||||
query_,
|
||||
key_,
|
||||
w,
|
||||
u,
|
||||
local_query,
|
||||
local_key,
|
||||
local_w,
|
||||
local_u,
|
||||
g_,
|
||||
decay_mask,
|
||||
local_decay_mask,
|
||||
initial_state,
|
||||
output_final_state,
|
||||
cu_seqlens,
|
||||
chunk_offsets,
|
||||
local_chunk_offsets,
|
||||
initial_state_indices);
|
||||
};
|
||||
|
||||
|
||||
|
||||
|
||||
auto output_and_state =
|
||||
initial_state.scalar_type() == at::kFloat
|
||||
? launch_inter(float{})
|
||||
|
||||
+43
-21
@@ -281,14 +281,14 @@ at::Tensor causal_conv1d_fwd_cpu(
|
||||
const std::optional<at::Tensor>& query_start_loc,
|
||||
const std::optional<at::Tensor>& cache_indices,
|
||||
const std::optional<at::Tensor>& has_initial_state, bool silu_activation,
|
||||
int64_t pad_slot_id, bool is_vnni);
|
||||
int64_t pad_slot_id, bool is_weight_packed);
|
||||
|
||||
at::Tensor causal_conv1d_update_cpu(
|
||||
const at::Tensor& x, const at::Tensor& conv_states,
|
||||
const at::Tensor& weight, const std::optional<at::Tensor>& bias,
|
||||
bool silu_activation, const std::optional<at::Tensor>& num_accepted_tokens,
|
||||
const std::optional<at::Tensor>& conv_state_indices, int64_t pad_slot_id,
|
||||
bool is_vnni);
|
||||
bool is_weight_packed);
|
||||
|
||||
void activation_lut_bf16(torch::Tensor& out, torch::Tensor& input,
|
||||
const std::string& activation);
|
||||
@@ -393,6 +393,11 @@ void mamba_chunk_scan_fwd_cpu_impl(at::Tensor& out, at::Tensor& final_states,
|
||||
const c10::optional<at::Tensor>& z,
|
||||
const at::Tensor& cu_seqlens);
|
||||
|
||||
torch::Tensor fused_gumbel_argmax(const torch::Tensor& logits,
|
||||
const torch::Tensor& seeds);
|
||||
|
||||
torch::Tensor greedy_argmax(const torch::Tensor& logits);
|
||||
|
||||
void init_cpu_memory_env(std::vector<int64_t> node_ids);
|
||||
|
||||
namespace cpu_utils {
|
||||
@@ -512,6 +517,33 @@ TORCH_LIBRARY_EXPAND(TORCH_EXTENSION_NAME, ops) {
|
||||
|
||||
#endif // (defined(__aarch64__) && !defined(__APPLE__))
|
||||
|
||||
#if (defined(__aarch64__) && defined(ARM_BF16_SUPPORT) && \
|
||||
!defined(__APPLE__)) || \
|
||||
(defined(__AVX512BF16__) && defined(__AVX512F__) && \
|
||||
defined(__AVX512VNNI__))
|
||||
|
||||
// Adapted from sglang: casual_conv1d kernels
|
||||
ops.def("causal_conv1d_weight_pack(Tensor weight) -> Tensor");
|
||||
ops.impl("causal_conv1d_weight_pack", torch::kCPU,
|
||||
&causal_conv1d_weight_pack);
|
||||
|
||||
ops.def(
|
||||
"causal_conv1d_fwd_cpu(Tensor x, Tensor weight, Tensor? bias, Tensor? "
|
||||
"conv_states, Tensor? query_start_loc,"
|
||||
"Tensor? cache_indices, Tensor? has_initial_state, bool silu_activation, "
|
||||
"int pad_slot_id, bool is_weight_packed) -> "
|
||||
"Tensor");
|
||||
ops.impl("causal_conv1d_fwd_cpu", torch::kCPU, &causal_conv1d_fwd_cpu);
|
||||
|
||||
ops.def(
|
||||
"causal_conv1d_update_cpu(Tensor x, Tensor(a!) conv_states, Tensor "
|
||||
"weight, Tensor? bias, bool silu_activation,"
|
||||
"Tensor? num_accepted_tokens, Tensor? conv_state_indices, int "
|
||||
"pad_slot_id, "
|
||||
"bool is_weight_packed) -> Tensor");
|
||||
ops.impl("causal_conv1d_update_cpu", torch::kCPU, &causal_conv1d_update_cpu);
|
||||
#endif
|
||||
|
||||
// Layernorm
|
||||
// Apply Root Mean Square (RMS) Normalization to the input tensor.
|
||||
ops.def(
|
||||
@@ -660,24 +692,6 @@ TORCH_LIBRARY_EXPAND(TORCH_EXTENSION_NAME, ops) {
|
||||
"-> (Tensor, Tensor)");
|
||||
ops.impl("quantize_fp8e4m3_vec", torch::kCPU, &quantize_fp8e4m3_vec);
|
||||
|
||||
ops.def("causal_conv1d_weight_pack(Tensor weight) -> Tensor");
|
||||
ops.impl("causal_conv1d_weight_pack", torch::kCPU,
|
||||
&causal_conv1d_weight_pack);
|
||||
ops.def(
|
||||
"causal_conv1d_fwd_cpu(Tensor x, Tensor weight, Tensor? bias, Tensor? "
|
||||
"conv_states, Tensor? query_start_loc,"
|
||||
"Tensor? cache_indices, Tensor? has_initial_state, bool silu_activation, "
|
||||
"int pad_slot_id, bool is_vnni) -> "
|
||||
"Tensor");
|
||||
ops.impl("causal_conv1d_fwd_cpu", torch::kCPU, &causal_conv1d_fwd_cpu);
|
||||
ops.def(
|
||||
"causal_conv1d_update_cpu(Tensor x, Tensor(a!) conv_states, Tensor "
|
||||
"weight, Tensor? bias, bool silu_activation,"
|
||||
"Tensor? num_accepted_tokens, Tensor? conv_state_indices, int "
|
||||
"pad_slot_id, "
|
||||
"bool is_vnni) -> Tensor");
|
||||
ops.impl("causal_conv1d_update_cpu", torch::kCPU, &causal_conv1d_update_cpu);
|
||||
|
||||
// Adapted from sglang: MLA CPU kernels (AMX-only, DeepSeek V2/V3/R1)
|
||||
ops.def(
|
||||
"decode_attention_cpu(Tensor query, Tensor k_buffer, Tensor v_buffer, "
|
||||
@@ -897,7 +911,8 @@ TORCH_LIBRARY_EXPAND(TORCH_EXTENSION_NAME, ops) {
|
||||
ops.def("dynamic_per_token_scaled_fp8_quant() -> ()", placeholder_op);
|
||||
|
||||
// WNA16
|
||||
#if defined(__AVX512F__) || defined(__riscv_v) || defined(__s390x__)
|
||||
#if defined(__AVX512F__) || defined(__riscv_v) || defined(__s390x__) || \
|
||||
defined(__powerpc__)
|
||||
ops.def(
|
||||
"cpu_gemm_wna16(Tensor input, Tensor q_weight, Tensor(a2!) output, "
|
||||
"Tensor scales, Tensor? zeros, Tensor? bias, SymInt "
|
||||
@@ -969,6 +984,13 @@ TORCH_LIBRARY_EXPAND(TORCH_EXTENSION_NAME, ops) {
|
||||
|
||||
ops.def("init_cpu_memory_env(SymInt[] node_ids) -> ()", &init_cpu_memory_env);
|
||||
|
||||
// Fused sampling kernels
|
||||
ops.def("fused_gumbel_argmax(Tensor logits, Tensor seeds) -> Tensor");
|
||||
ops.impl("fused_gumbel_argmax", torch::kCPU, &fused_gumbel_argmax);
|
||||
|
||||
ops.def("greedy_argmax(Tensor logits) -> Tensor");
|
||||
ops.impl("greedy_argmax", torch::kCPU, &greedy_argmax);
|
||||
|
||||
// Speculative decoding kernels
|
||||
ops.def(
|
||||
"eagle_prepare_inputs_padded_kernel_impl(Tensor cu_num_draft_tokens, "
|
||||
|
||||
@@ -285,6 +285,262 @@ __device__ __forceinline__ void storeScaledQElemsFp8(
|
||||
storeElemsFp8(dst, scaled);
|
||||
}
|
||||
|
||||
// index_q: E4M3(scalar_t(v)) with unit scale, i.e. q is rounded to the model
|
||||
// dtype before the fp8 cast. On CUDA each element pair is rounded to scalar_t
|
||||
// and then converted to E4M3, with one packed cvt each.
|
||||
template <typename scalar_t>
|
||||
__device__ __forceinline__ void storeIndexQElemsFp8(
|
||||
uint8_t* __restrict__ dst, float const (&elems)[kElemsPerLane]) {
|
||||
#ifndef USE_ROCM
|
||||
using Converter = vllm::_typeConvert<scalar_t>;
|
||||
__nv_fp8x2_storage_t out2[kElemsPerLane / 2];
|
||||
#pragma unroll
|
||||
for (int i = 0; i < kElemsPerLane / 2; i++) {
|
||||
auto const v =
|
||||
Converter::convert(make_float2(elems[2 * i], elems[2 * i + 1]));
|
||||
if constexpr (std::is_same_v<scalar_t, c10::BFloat16>) {
|
||||
out2[i] = __nv_cvt_bfloat16raw2_to_fp8x2(v, __NV_SATFINITE, __NV_E4M3);
|
||||
} else {
|
||||
out2[i] = __nv_cvt_halfraw2_to_fp8x2(v, __NV_SATFINITE, __NV_E4M3);
|
||||
}
|
||||
}
|
||||
*reinterpret_cast<uint32_t*>(dst) = *reinterpret_cast<uint32_t const*>(out2);
|
||||
#else
|
||||
storeScaledQElemsFp8<scalar_t>(dst, elems, 1.0f);
|
||||
#endif
|
||||
}
|
||||
|
||||
// ────────────────────────────────────────────────────────────────────────────
|
||||
// NVFP4 main K/V cache store
|
||||
// ────────────────────────────────────────────────────────────────────────────
|
||||
// The block encoding of reshape_and_cache_nvfp4_kernel
|
||||
// (nvfp4_kv_cache_kernels.cu) in the MSA readers' head-slot page layout. The
|
||||
// quantizer differs from that kernel's reciprocal-multiply math: per 16-value
|
||||
// block, scale = min(448, max(E4M3(max(amax, 1e-12) / 6), 1/512)) and
|
||||
// code = sign | RNE-E2M1(min(|x / scale|, 6)), with zero stored as +0, all on
|
||||
// x / the global scale.
|
||||
// An HND page is 2*nkv slots of [bs, 72] bytes; slot 2*head + side (K = 0,
|
||||
// V = 1) holds that head's packed E2M1 data [bs, 64] followed by its E4M3 block
|
||||
// scales [bs, 8]. Every head is one contiguous 2*bs*72-byte run, so a page
|
||||
// split across TP ranks is a contiguous byte slice per rank. K scales are
|
||||
// linear, V scales use the SM100 trtllm-gen 4x4 token-quad swizzle. One
|
||||
// 16-element scale group is four lanes. SM100-family builds of this file
|
||||
// (CMake gives it family-specific SM10x/11x targets) encode E2M1 with
|
||||
// cvt.rn.satfinite.e2m1x2; other targets use a bit-exact software encoder.
|
||||
#if defined(__CUDA_ARCH__) && __CUDA_ARCH__ >= 1000 && \
|
||||
defined(__CUDA_ARCH_FAMILY_SPECIFIC__)
|
||||
#define MINIMAX_M3_NVFP4_NATIVE_CVT 1
|
||||
#else
|
||||
#define MINIMAX_M3_NVFP4_NATIVE_CVT 0
|
||||
#endif
|
||||
|
||||
#ifndef USE_ROCM
|
||||
constexpr int kNvfp4DataBytes = kHeadDim / 2; // 64
|
||||
constexpr int kNvfp4ScaleBytes = kHeadDim / 16; // 8
|
||||
constexpr int kNvfp4LanesPerGroup = 16 / kElemsPerLane; // 4
|
||||
|
||||
// Round-to-nearest-even, saturating E2M1 code of ``x`` (matches
|
||||
// cvt.rn.satfinite.e2m1x2.f32). Below 1 the grid is the 0.5-step subnormals;
|
||||
// from 1 up it is fp32 rounded to one mantissa bit, whose (exponent, mantissa)
|
||||
// bits are the code offset by 252 once the exponent is rebiased.
|
||||
__device__ __forceinline__ uint32_t e2m1Code(float x) {
|
||||
float const ax = fabsf(x);
|
||||
uint32_t mag;
|
||||
if (ax < 1.0f) {
|
||||
mag = static_cast<uint32_t>(__float2int_rn(2.0f * ax));
|
||||
} else {
|
||||
uint32_t const bits = __float_as_uint(ax);
|
||||
uint32_t const rounded = bits + 0x1FFFFFu + ((bits >> 22) & 1u);
|
||||
mag = min((rounded >> 22) - 252u, 7u);
|
||||
}
|
||||
return mag | ((__float_as_uint(x) >> 28) & 0x8u);
|
||||
}
|
||||
|
||||
__device__ __forceinline__ float rcpApproxFtz(float a) {
|
||||
float b;
|
||||
asm("rcp.approx.ftz.f32 %0, %1;" : "=f"(b) : "f"(a));
|
||||
return b;
|
||||
}
|
||||
|
||||
// a / b rounded to nearest even from r = RN(1 / b): Markstein's correction of
|
||||
// a * r, which is __fdiv_rn's fast path without its range-check branch. Exact
|
||||
// unless a * r overflows or the residual a - q * b underflows (a within a few
|
||||
// binades of the subnormals).
|
||||
__device__ __forceinline__ float divRnByRcp(float a, float b, float r) {
|
||||
float const q = __fmul_rn(a, r);
|
||||
return __fmaf_rn(__fmaf_rn(-q, b, a), r, q);
|
||||
}
|
||||
|
||||
// Block scale E4M3(max(amax, 1e-12) / 6) clamped to [1/512, 448], with
|
||||
// sf_scale applied to it as to the values, and its reciprocal RN(1 / sf_value).
|
||||
__device__ __forceinline__ void nvfp4GroupScale(float sf_scale, float group_max,
|
||||
uint8_t& sf_byte,
|
||||
float& sf_value,
|
||||
float& sf_rcp) {
|
||||
constexpr float kRcp6 = 0x1.555556p-3f; // RN(1 / 6)
|
||||
float const amax = fmaxf(group_max, 1e-12f);
|
||||
// amax / 6 (RNE). amax * kRcp6 bounds it from above, and fminf picks it for
|
||||
// amax = inf, where the correction is NaN.
|
||||
float const amax_6 =
|
||||
fminf(divRnByRcp(amax, 6.0f, kRcp6), __fmul_rn(amax, kRcp6));
|
||||
// Flooring before the rounding gives the 1/512 minimum; satfinite caps it
|
||||
// at 448.
|
||||
__nv_fp8_e4m3 const sf(fmaxf(sf_scale * amax_6, 0x1p-9f));
|
||||
sf_byte = sf.__x;
|
||||
sf_value = float(sf);
|
||||
// rcp.approx is correctly rounded on every positive E4M3 value.
|
||||
sf_rcp = rcpApproxFtz(sf_value);
|
||||
}
|
||||
|
||||
// nvfp4GroupScale for sf_scale = 1 and a bf16/fp16 group_max. Uncorrected,
|
||||
// amax * RN(1 / 6) still rounds to the E4M3 byte of amax / 6 (checked for every
|
||||
// such amax: amax / 6 is too far from an E4M3 rounding boundary to cross it,
|
||||
// and exact ties round back), and the 1e-12 floor lies below the 1/512 one.
|
||||
__device__ __forceinline__ void nvfp4UnitGroupScale(float group_max,
|
||||
uint8_t& sf_byte,
|
||||
float& sf_rcp) {
|
||||
constexpr float kRcp6 = 0x1.555556p-3f; // RN(1 / 6)
|
||||
__nv_fp8_e4m3 const sf(fmaxf(__fmul_rn(group_max, kRcp6), 0x1p-9f));
|
||||
sf_byte = sf.__x;
|
||||
sf_rcp = rcpApproxFtz(float(sf));
|
||||
}
|
||||
|
||||
// (x * sf_scale) / sf_value with the E2M1 code of the IEEE quotient. The
|
||||
// quotient itself is the IEEE one except below 1e-35 (code 0 either way) and
|
||||
// for magnitudes over 4096, which are clamped (NaN kept) so that the division
|
||||
// stays finite: 4096 / 448 > 5, so their code stays the saturated 6.
|
||||
__device__ __forceinline__ float nvfp4Quotient(float x, float sf_scale,
|
||||
float sf_value, float sf_rcp) {
|
||||
constexpr float kMaxMag = 4096.0f;
|
||||
float n = __fmul_rn(x, sf_scale);
|
||||
n = fabsf(n) > kMaxMag ? copysignf(kMaxMag, n) : n;
|
||||
return divRnByRcp(n, sf_value, sf_rcp);
|
||||
}
|
||||
|
||||
// Zero is stored as +0: clear the sign of every nibble whose magnitude is 0.
|
||||
// Adding 7 to a 3-bit magnitude carries into the sign bit exactly when it is
|
||||
// nonzero.
|
||||
__device__ __forceinline__ uint16_t clearNegZeroNibbles(uint16_t packed) {
|
||||
return packed & (((packed & 0x7777u) + 0x7777u) | 0x7777u);
|
||||
}
|
||||
|
||||
// Quantize one K or V head row of one token into its NVFP4 page, where
|
||||
// ``scale`` is the dequantization (global) scale. Every lane of the warp must
|
||||
// call this (the scale-group reductions shuffle).
|
||||
template <typename scalar_t>
|
||||
__device__ __forceinline__ void storeNvfp4CacheElems(
|
||||
uint8_t* __restrict__ page, int const kv, int const head, int const token,
|
||||
int const block_size, int const laneId, float const (&elems)[kElemsPerLane],
|
||||
float const scale) {
|
||||
// Quantize the model-dtype value that the unfused path reads back from qkv.
|
||||
using Converter = vllm::_typeConvert<scalar_t>;
|
||||
float x[kElemsPerLane];
|
||||
float group_max = 0.0f;
|
||||
#pragma unroll
|
||||
for (int i = 0; i < kElemsPerLane; i++) {
|
||||
x[i] = static_cast<float>(Converter::convert(elems[i]));
|
||||
group_max = fmaxf(group_max, fabsf(x[i]));
|
||||
}
|
||||
group_max = fmaxf(group_max, __shfl_xor_sync(FINAL_MASK, group_max, 1, 32));
|
||||
group_max = fmaxf(group_max, __shfl_xor_sync(FINAL_MASK, group_max, 2, 32));
|
||||
|
||||
uint8_t sf_byte;
|
||||
float sf_rcp;
|
||||
float q[kElemsPerLane]; // (x / scale) / sf_value, E2M1-encoded below
|
||||
if (scale == 1.0f) {
|
||||
static_assert(sizeof(scalar_t) == 2, "the unit-scale path needs bf16/fp16");
|
||||
nvfp4UnitGroupScale(group_max, sf_byte, sf_rcp);
|
||||
// x / sf_value truncated from x * RN(1 / sf_value). RN(1 / s) >= 1 / s for
|
||||
// every E4M3 s, so a bf16/fp16 x on an E2M1 midpoint lands exactly on it
|
||||
// (and the cvt rounds it to even), while any other bf16/fp16 x is too far
|
||||
// from a midpoint to reach one. No clamp either: a finite product never
|
||||
// truncates to inf, and inf saturates in the cvt.
|
||||
#if MINIMAX_M3_NVFP4_NATIVE_CVT
|
||||
float2 const r2 = make_float2(sf_rcp, sf_rcp);
|
||||
#pragma unroll
|
||||
for (int i = 0; i < kElemsPerLane; i += 2) {
|
||||
float2 const p = __fmul2_rz(make_float2(x[i], x[i + 1]), r2);
|
||||
q[i] = p.x;
|
||||
q[i + 1] = p.y;
|
||||
}
|
||||
#else
|
||||
#pragma unroll
|
||||
for (int i = 0; i < kElemsPerLane; i++) q[i] = __fmul_rz(x[i], sf_rcp);
|
||||
#endif
|
||||
} else {
|
||||
float const sf_scale = __frcp_rn(scale); // exactly 1.0f / scale
|
||||
float sf_value;
|
||||
nvfp4GroupScale(sf_scale, group_max, sf_byte, sf_value, sf_rcp);
|
||||
// A correctly rounded division, not a reciprocal multiply, so that exact
|
||||
// E2M1 midpoints round to even.
|
||||
#if MINIMAX_M3_NVFP4_NATIVE_CVT
|
||||
// nvfp4Quotient on element pairs, with SM100's packed FP32 instructions.
|
||||
float2 const s2 = make_float2(sf_scale, sf_scale);
|
||||
float2 const neg_v2 = make_float2(-sf_value, -sf_value);
|
||||
float2 const r2 = make_float2(sf_rcp, sf_rcp);
|
||||
#pragma unroll
|
||||
for (int i = 0; i < kElemsPerLane; i += 2) {
|
||||
float2 n = __fmul2_rn(make_float2(x[i], x[i + 1]), s2);
|
||||
// |n| <= 4096 as in nvfp4Quotient, NaN kept.
|
||||
asm("max.NaN.f32 %0, %0, 0fC5800000;\n\t"
|
||||
"min.NaN.f32 %0, %0, 0f45800000;\n\t"
|
||||
"max.NaN.f32 %1, %1, 0fC5800000;\n\t"
|
||||
"min.NaN.f32 %1, %1, 0f45800000;"
|
||||
: "+f"(n.x), "+f"(n.y));
|
||||
float2 const q0 = __fmul2_rn(n, r2);
|
||||
float2 const p = __ffma2_rn(__ffma2_rn(q0, neg_v2, n), r2, q0);
|
||||
q[i] = p.x;
|
||||
q[i + 1] = p.y;
|
||||
}
|
||||
#else
|
||||
#pragma unroll
|
||||
for (int i = 0; i < kElemsPerLane; i++) {
|
||||
q[i] = nvfp4Quotient(x[i], sf_scale, sf_value, sf_rcp);
|
||||
}
|
||||
#endif
|
||||
}
|
||||
uint16_t packed;
|
||||
#if MINIMAX_M3_NVFP4_NATIVE_CVT
|
||||
// Low nibble = even element; cvt packs its first source in the high nibble.
|
||||
asm("{\n"
|
||||
" .reg .b8 lo, hi;\n"
|
||||
" cvt.rn.satfinite.e2m1x2.f32 lo, %2, %1;\n"
|
||||
" cvt.rn.satfinite.e2m1x2.f32 hi, %4, %3;\n"
|
||||
" mov.b16 %0, {lo, hi};\n"
|
||||
"}"
|
||||
: "=h"(packed)
|
||||
: "f"(q[0]), "f"(q[1]), "f"(q[2]), "f"(q[3]));
|
||||
#else
|
||||
uint32_t codes = 0;
|
||||
#pragma unroll
|
||||
for (int i = 0; i < kElemsPerLane; i++) codes |= e2m1Code(q[i]) << (4 * i);
|
||||
packed = static_cast<uint16_t>(codes);
|
||||
#endif
|
||||
packed = clearNegZeroNibbles(packed);
|
||||
|
||||
uint8_t* const slot = page + static_cast<int64_t>(2 * head + kv) *
|
||||
block_size *
|
||||
(kNvfp4DataBytes + kNvfp4ScaleBytes);
|
||||
uint8_t* const data =
|
||||
slot + token * kNvfp4DataBytes + laneId * (kElemsPerLane / 2);
|
||||
*reinterpret_cast<uint16_t*>(data) = packed;
|
||||
|
||||
if (laneId % kNvfp4LanesPerGroup == 0) {
|
||||
int const group = laneId / kNvfp4LanesPerGroup;
|
||||
int offset;
|
||||
if (kv == 0) {
|
||||
offset = token * kNvfp4ScaleBytes + group;
|
||||
} else {
|
||||
constexpr int kGroupsPerQuad = kNvfp4ScaleBytes / 4;
|
||||
int const swizzled_t = (token / 4) * 4 + group / kGroupsPerQuad;
|
||||
int const swizzled_s = (group % kGroupsPerQuad) * 4 + token % 4;
|
||||
offset = swizzled_t * kNvfp4ScaleBytes + swizzled_s;
|
||||
}
|
||||
slot[block_size * kNvfp4DataBytes + offset] = sf_byte;
|
||||
}
|
||||
}
|
||||
#endif // USE_ROCM
|
||||
|
||||
// ────────────────────────────────────────────────────────────────────────────
|
||||
// Kernel
|
||||
// ────────────────────────────────────────────────────────────────────────────
|
||||
@@ -307,9 +563,8 @@ __device__ __forceinline__ void storeScaledQElemsFp8(
|
||||
// branch and writes index_q/index_k outputs. Skip-index-topk reuse layers keep
|
||||
// kHasIndex=true but set kProcessIndex=false.
|
||||
template <typename scalar_t, typename cache_t, Fp8KVCacheDataType kv_dt,
|
||||
typename out_idx_t, bool kHasIndex, bool kInsertKV,
|
||||
bool kProcessIndex,
|
||||
bool kFp8Idx>
|
||||
bool kNvfp4, typename out_idx_t, bool kHasIndex, bool kInsertKV,
|
||||
bool kProcessIndex, bool kFp8Idx>
|
||||
__global__ void fusedMiniMaxM3QNormRopeKVInsertKernel(
|
||||
scalar_t* __restrict__ qkv, // [N, qkv_row] in/out (packs index if sparse)
|
||||
scalar_t* __restrict__ q_out, // [N, nq*128] contiguous, or nullptr
|
||||
@@ -332,7 +587,10 @@ __global__ void fusedMiniMaxM3QNormRopeKVInsertKernel(
|
||||
// The content (last) dim is always innermost-contiguous (stride 1), so the
|
||||
// NHD/HND layout choice is captured by the head/token strides.
|
||||
int64_t const kv_s_block, int64_t const kv_s_head, int64_t const kv_s_token,
|
||||
int64_t const kv_s_dim) {
|
||||
int64_t const kv_s_dim,
|
||||
// NVFP4 main cache only: per-layer K/V dequantization (global) scales.
|
||||
float const* __restrict__ kv_k_scale,
|
||||
float const* __restrict__ kv_v_scale) {
|
||||
#if (!defined(__CUDA_ARCH__) || __CUDA_ARCH__ < 800) && !defined(USE_ROCM)
|
||||
// _typeConvert<BFloat16> is unavailable on pre-Ampere; the M3 kernel only
|
||||
// runs with bf16/fp16 inputs in practice. Discard the bf16 body there.
|
||||
@@ -352,8 +610,11 @@ __global__ void fusedMiniMaxM3QNormRopeKVInsertKernel(
|
||||
int const idx_slots = kProcessIndex ? niq + 1 : 0;
|
||||
int const slots_per_token = nq + nkv + v_slots + idx_slots;
|
||||
|
||||
int const tokenIdx = globalWarpIdx / slots_per_token;
|
||||
int const slot = globalWarpIdx % slots_per_token;
|
||||
// Unsigned: both are non-negative, and the signed division's sign handling
|
||||
// would cost every warp.
|
||||
unsigned const warp_u = globalWarpIdx, slots_u = slots_per_token;
|
||||
int const tokenIdx = warp_u / slots_u;
|
||||
int const slot = warp_u % slots_u;
|
||||
if (tokenIdx >= num_tokens) return;
|
||||
|
||||
// Slot boundaries.
|
||||
@@ -420,10 +681,13 @@ __global__ void fusedMiniMaxM3QNormRopeKVInsertKernel(
|
||||
// flat TMA descriptor can address them as [tokens*heads, head_dim]; this
|
||||
// folds the de-interleaving into the store the kernel already does, instead
|
||||
// of a separate q.contiguous() copy. Everything else stays in place.
|
||||
// Given q_fp8_out but no q_out, q is emitted only in fp8 (null store_ptr).
|
||||
scalar_t* store_ptr = row_ptr;
|
||||
if (isQ && q_out != nullptr) {
|
||||
store_ptr = q_out + static_cast<int64_t>(tokenIdx) * nq * kHeadDim +
|
||||
slot * kHeadDim;
|
||||
} else if (isQ && q_fp8_out != nullptr) {
|
||||
store_ptr = nullptr;
|
||||
} else if (isIQ && index_q_out != nullptr) {
|
||||
// bf16 index_q_out: gather here. fp8: written by the explicit fp8 store.
|
||||
if constexpr (!kFp8Idx) {
|
||||
@@ -433,6 +697,13 @@ __global__ void fusedMiniMaxM3QNormRopeKVInsertKernel(
|
||||
}
|
||||
}
|
||||
|
||||
// NVFP4: the per-layer global scale is not produced by the predecessor,
|
||||
// so load it before the PDL wait.
|
||||
float nvfp4_scale = 0.0f;
|
||||
if constexpr (kNvfp4) {
|
||||
if (isK || isV) nvfp4_scale = isK ? *kv_k_scale : *kv_v_scale;
|
||||
}
|
||||
|
||||
// PDL: wait for the predecessor kernel (the qkv-projection GEMM that
|
||||
// produces ``qkv``) to finish before touching any global memory. No-op
|
||||
// when PDL is not enabled on the launch. The CUDA runtime wrapper emits
|
||||
@@ -451,16 +722,17 @@ __global__ void fusedMiniMaxM3QNormRopeKVInsertKernel(
|
||||
normAndRope<scalar_t>(elems, laneId, eps, norm_w, do_rope, rotary_dim,
|
||||
cos_ptr, /*apply_norm=*/norm_w != nullptr);
|
||||
if constexpr (kFp8Idx) {
|
||||
// index_q is e4m3 bytes; Q/K (and in-place index_k) stay scalar_t.
|
||||
// index_q is E4M3(scalar_t(q)) bytes; Q/K (and in-place index_k) stay
|
||||
// scalar_t.
|
||||
if (isIQ && index_q_out != nullptr) {
|
||||
storeElemsFp8(index_q_out +
|
||||
static_cast<int64_t>(tokenIdx) * niq * kHeadDim +
|
||||
(slot - iq_begin) * kHeadDim + dim_base,
|
||||
elems);
|
||||
} else {
|
||||
storeIndexQElemsFp8<scalar_t>(
|
||||
index_q_out + static_cast<int64_t>(tokenIdx) * niq * kHeadDim +
|
||||
(slot - iq_begin) * kHeadDim + dim_base,
|
||||
elems);
|
||||
} else if (store_ptr != nullptr) {
|
||||
storeElems<scalar_t>(store_ptr + dim_base, elems);
|
||||
}
|
||||
} else {
|
||||
} else if (store_ptr != nullptr) {
|
||||
storeElems<scalar_t>(store_ptr + dim_base, elems);
|
||||
}
|
||||
if (isQ && q_fp8_out != nullptr) {
|
||||
@@ -487,6 +759,16 @@ __global__ void fusedMiniMaxM3QNormRopeKVInsertKernel(
|
||||
} else {
|
||||
storeElems<scalar_t>(index_cache + sm * kHeadDim + dim_base, elems);
|
||||
}
|
||||
} else if constexpr (kNvfp4) {
|
||||
#ifndef USE_ROCM
|
||||
// kv_cache is [num_blocks, 2*nkv, block_size, 72] bytes (HND), slot
|
||||
// 2*head + side; kv_s_block is its page stride.
|
||||
storeNvfp4CacheElems<scalar_t>(
|
||||
reinterpret_cast<uint8_t*>(kv_cache) +
|
||||
(sm / block_size) * kv_s_block,
|
||||
isK ? 0 : 1, head, static_cast<int>(sm % block_size), block_size,
|
||||
laneId, elems, nvfp4_scale);
|
||||
#endif
|
||||
} else if (isK || isV) {
|
||||
// kv_cache logical shape [num_blocks, nkv, block_size, 2*head_dim].
|
||||
// Paging is logical (block = sm/block_size, token = sm%block_size);
|
||||
@@ -515,7 +797,8 @@ __global__ void fusedMiniMaxM3QNormRopeKVInsertKernel(
|
||||
// ────────────────────────────────────────────────────────────────────────────
|
||||
// Launch wrapper
|
||||
// ────────────────────────────────────────────────────────────────────────────
|
||||
template <typename scalar_t, typename cache_t, Fp8KVCacheDataType kv_dt>
|
||||
template <typename scalar_t, typename cache_t, Fp8KVCacheDataType kv_dt,
|
||||
bool kNvfp4 = false>
|
||||
void launchFusedMiniMaxM3(
|
||||
scalar_t* qkv, scalar_t* q_out, uint8_t* q_fp8_out, void* index_q_out,
|
||||
scalar_t const* q_norm_w, scalar_t const* k_norm_w,
|
||||
@@ -527,7 +810,8 @@ void launchFusedMiniMaxM3(
|
||||
int const nq, int const nkv, int const niq, int const block_size,
|
||||
int64_t const kv_s_block, int64_t const kv_s_head, int64_t const kv_s_token,
|
||||
int64_t const kv_s_dim, bool const has_index, bool const insert_kv,
|
||||
bool const process_index, bool const fp8_idx, cudaStream_t stream) {
|
||||
bool const process_index, bool const fp8_idx, float const* kv_k_scale,
|
||||
float const* kv_v_scale, cudaStream_t stream) {
|
||||
// Index outputs are scalar_t (bf16) or e4m3 bytes (uint8_t); reinterpret the
|
||||
// void* pointers per instantiation in the LAUNCH macro.
|
||||
// Slot count must match the kernel's compile-time gating.
|
||||
@@ -559,35 +843,44 @@ void launchFusedMiniMaxM3(
|
||||
config.attrs = attrs;
|
||||
config.numAttrs = (sm_version >= 90) ? 1 : 0;
|
||||
|
||||
#define LAUNCH(HAS_INDEX, INSERT, PROCESS_INDEX, FP8, OUT_T) \
|
||||
cudaLaunchKernelEx( \
|
||||
&config, \
|
||||
fusedMiniMaxM3QNormRopeKVInsertKernel<scalar_t, cache_t, kv_dt, OUT_T, \
|
||||
HAS_INDEX, INSERT, \
|
||||
PROCESS_INDEX, FP8>, \
|
||||
qkv, q_out, q_fp8_out, reinterpret_cast<OUT_T*>(index_q_out), \
|
||||
q_norm_w, k_norm_w, iq_norm_w, ik_norm_w, cos_sin_cache, positions, \
|
||||
slot_mapping, index_slot_mapping, kv_cache, \
|
||||
reinterpret_cast<OUT_T*>(index_cache), eps, q_fp8_inv_scale, \
|
||||
rotary_dim, num_tokens, nq, nkv, niq, block_size, kv_s_block, \
|
||||
kv_s_head, kv_s_token, kv_s_dim)
|
||||
#else
|
||||
// ROCm: standard kernel launch syntax (no PDL/stream serialization).
|
||||
// clang-format off
|
||||
#define LAUNCH(HAS_INDEX, INSERT, PROCESS_INDEX, FP8, OUT_T) \
|
||||
fusedMiniMaxM3QNormRopeKVInsertKernel< \
|
||||
scalar_t, cache_t, kv_dt, OUT_T, HAS_INDEX, INSERT, PROCESS_INDEX, \
|
||||
FP8><<<grid, kBlockSize, 0, stream>>>( \
|
||||
cudaLaunchKernelEx( \
|
||||
&config, \
|
||||
fusedMiniMaxM3QNormRopeKVInsertKernel<scalar_t, cache_t, kv_dt, \
|
||||
kNvfp4, OUT_T, HAS_INDEX, \
|
||||
INSERT, PROCESS_INDEX, FP8>, \
|
||||
qkv, q_out, q_fp8_out, reinterpret_cast<OUT_T*>(index_q_out), \
|
||||
q_norm_w, k_norm_w, iq_norm_w, ik_norm_w, cos_sin_cache, positions, \
|
||||
slot_mapping, index_slot_mapping, kv_cache, \
|
||||
reinterpret_cast<OUT_T*>(index_cache), eps, q_fp8_inv_scale, \
|
||||
rotary_dim, num_tokens, nq, nkv, niq, block_size, kv_s_block, \
|
||||
kv_s_head, kv_s_token, kv_s_dim)
|
||||
kv_s_head, kv_s_token, kv_s_dim, kv_k_scale, kv_v_scale)
|
||||
#else
|
||||
// ROCm: standard kernel launch syntax (no PDL/stream serialization).
|
||||
// clang-format off
|
||||
#define LAUNCH(HAS_INDEX, INSERT, PROCESS_INDEX, FP8, OUT_T) \
|
||||
fusedMiniMaxM3QNormRopeKVInsertKernel< \
|
||||
scalar_t, cache_t, kv_dt, kNvfp4, OUT_T, HAS_INDEX, INSERT, \
|
||||
PROCESS_INDEX, FP8><<<grid, kBlockSize, 0, stream>>>( \
|
||||
qkv, q_out, q_fp8_out, reinterpret_cast<OUT_T*>(index_q_out), \
|
||||
q_norm_w, k_norm_w, iq_norm_w, ik_norm_w, cos_sin_cache, positions, \
|
||||
slot_mapping, index_slot_mapping, kv_cache, \
|
||||
reinterpret_cast<OUT_T*>(index_cache), eps, q_fp8_inv_scale, \
|
||||
rotary_dim, num_tokens, nq, nkv, niq, block_size, kv_s_block, \
|
||||
kv_s_head, kv_s_token, kv_s_dim, kv_k_scale, kv_v_scale)
|
||||
// clang-format on
|
||||
#endif
|
||||
|
||||
if (has_index) {
|
||||
if constexpr (kNvfp4) {
|
||||
// NVFP4 main cache: sparse serving only (the host checks insert_kv).
|
||||
if (!process_index) {
|
||||
LAUNCH(true, true, false, false, scalar_t);
|
||||
} else if (fp8_idx) {
|
||||
LAUNCH(true, true, true, true, uint8_t);
|
||||
} else {
|
||||
LAUNCH(true, true, true, false, scalar_t);
|
||||
}
|
||||
} else if (has_index) {
|
||||
if (!process_index) {
|
||||
if (insert_kv) {
|
||||
LAUNCH(true, true, false, false, scalar_t);
|
||||
@@ -621,8 +914,9 @@ void launchFusedMiniMaxM3(
|
||||
} // namespace vllm
|
||||
|
||||
// clang-format off
|
||||
#define CALL_FUSED_MINIMAX_M3(_RAW_T, CACHE_T, KV_DTYPE) \
|
||||
vllm::minimax_m3_fused_ops::launchFusedMiniMaxM3<st, CACHE_T, KV_DTYPE>( \
|
||||
#define CALL_FUSED_MINIMAX_M3_IMPL(CACHE_T, KV_DTYPE, NVFP4) \
|
||||
vllm::minimax_m3_fused_ops::launchFusedMiniMaxM3<st, CACHE_T, KV_DTYPE, \
|
||||
NVFP4>( \
|
||||
reinterpret_cast<st*>(qkv.data_ptr()), \
|
||||
q_out.has_value() ? reinterpret_cast<st*>(q_out->data_ptr()) : nullptr, \
|
||||
q_fp8_out.has_value() \
|
||||
@@ -654,7 +948,10 @@ void launchFusedMiniMaxM3(
|
||||
static_cast<float>(eps), 1.0f / static_cast<float>(q_fp8_scale), \
|
||||
static_cast<int>(rotary_dim), num_tokens, nq, nkv, niq, \
|
||||
static_cast<int>(block_size), kv_s_block, kv_s_head, kv_s_token, \
|
||||
kv_s_dim, has_index, insert_kv, process_index, fp8_idx, stream)
|
||||
kv_s_dim, has_index, insert_kv, process_index, fp8_idx, kv_k_scale_ptr, \
|
||||
kv_v_scale_ptr, stream)
|
||||
#define CALL_FUSED_MINIMAX_M3(_RAW_T, CACHE_T, KV_DTYPE) \
|
||||
CALL_FUSED_MINIMAX_M3_IMPL(CACHE_T, KV_DTYPE, false)
|
||||
// clang-format on
|
||||
|
||||
// ────────────────────────────────────────────────────────────────────────────
|
||||
@@ -681,7 +978,9 @@ void fused_minimax_m3_qknorm_rope_kv_insert(
|
||||
const std::string& kv_cache_dtype, bool skip_index_branch,
|
||||
std::optional<torch::stable::Tensor>
|
||||
q_fp8_out, // [N, nq*128] contiguous E4M3
|
||||
double q_fp8_scale) {
|
||||
double q_fp8_scale,
|
||||
std::optional<torch::stable::Tensor> kv_k_scale, // [] f32, NVFP4 only
|
||||
std::optional<torch::stable::Tensor> kv_v_scale) {
|
||||
STD_TORCH_CHECK(qkv.is_cuda() && qkv.is_contiguous(),
|
||||
"qkv must be contiguous CUDA");
|
||||
STD_TORCH_CHECK(
|
||||
@@ -721,8 +1020,10 @@ void fused_minimax_m3_qknorm_rope_kv_insert(
|
||||
bool const has_index = niq > 0;
|
||||
bool const insert_kv = kv_cache.has_value();
|
||||
bool const process_index = has_index && !skip_index_branch;
|
||||
bool const nvfp4_kv = kv_cache_dtype == "nvfp4";
|
||||
vllm::Fp8KVCacheDataType const kv_dt =
|
||||
vllm::get_fp8_kv_cache_data_type(kv_cache_dtype);
|
||||
nvfp4_kv ? vllm::Fp8KVCacheDataType::kAuto
|
||||
: vllm::get_fp8_kv_cache_data_type(kv_cache_dtype);
|
||||
int const kHeadDim = vllm::minimax_m3_fused_ops::kHeadDim;
|
||||
int const expected_row =
|
||||
(nq + 2 * nkv + (has_index ? niq + 1 : 0)) * kHeadDim;
|
||||
@@ -769,8 +1070,13 @@ void fused_minimax_m3_qknorm_rope_kv_insert(
|
||||
index_slot_mapping->numel() == slot_mapping->numel()),
|
||||
"index_slot_mapping must be int64 CUDA with slot_mapping length");
|
||||
}
|
||||
// Main attention KV cache: auto matches qkv, fp8 uses uint8 storage.
|
||||
if (kv_dt == vllm::Fp8KVCacheDataType::kAuto) {
|
||||
// Main attention KV cache: auto matches qkv, fp8 and nvfp4 use uint8
|
||||
// storage.
|
||||
if (nvfp4_kv) {
|
||||
STD_TORCH_CHECK(
|
||||
kv_cache->scalar_type() == torch::headeronly::ScalarType::Byte,
|
||||
"nvfp4 kv_cache must use uint8 storage");
|
||||
} else if (kv_dt == vllm::Fp8KVCacheDataType::kAuto) {
|
||||
STD_TORCH_CHECK(kv_cache->scalar_type() == qkv.scalar_type(),
|
||||
"auto kv_cache dtype must match qkv");
|
||||
} else {
|
||||
@@ -790,6 +1096,27 @@ void fused_minimax_m3_qknorm_rope_kv_insert(
|
||||
STD_TORCH_CHECK(kv_cache->dim() == 4 && kv_cache->stride(3) == 1,
|
||||
"kv_cache must be [nb,nkv,bs,2*head_dim] with contiguous "
|
||||
"content dim (stride(3)==1)");
|
||||
if (nvfp4_kv) {
|
||||
// Packed HND pages [nb, 2*nkv, bs, 72]: slot 2*head + side is one
|
||||
// contiguous [bs, 72] block, which the store addresses directly.
|
||||
int64_t const full_dim = kHeadDim / 2 + kHeadDim / 16;
|
||||
STD_TORCH_CHECK(
|
||||
kv_cache->size(1) == 2 * nkv && kv_cache->size(2) == block_size &&
|
||||
kv_cache->size(3) == full_dim &&
|
||||
kv_cache->stride(2) == full_dim &&
|
||||
kv_cache->stride(1) == block_size * full_dim &&
|
||||
block_size % 4 == 0,
|
||||
"nvfp4 kv_cache must be HND [nb, 2*num_kv_heads, block_size, 72] "
|
||||
"with contiguous pages and block_size % 4 == 0");
|
||||
for (auto const* scale : {&kv_k_scale, &kv_v_scale}) {
|
||||
STD_TORCH_CHECK(
|
||||
scale->has_value() && (*scale)->is_cuda() &&
|
||||
(*scale)->scalar_type() ==
|
||||
torch::headeronly::ScalarType::Float &&
|
||||
(*scale)->numel() >= 1,
|
||||
"nvfp4 kv_cache requires float32 CUDA kv_k_scale and kv_v_scale");
|
||||
}
|
||||
}
|
||||
kv_s_block = kv_cache->stride(0);
|
||||
kv_s_head = kv_cache->stride(1);
|
||||
kv_s_token = kv_cache->stride(2);
|
||||
@@ -854,10 +1181,31 @@ void fused_minimax_m3_qknorm_rope_kv_insert(
|
||||
"fp8 index path: index_q_out must be fp8 e4m3");
|
||||
}
|
||||
|
||||
STD_TORCH_CHECK(!nvfp4_kv || insert_kv,
|
||||
"nvfp4 kv_cache_dtype requires kv_cache (insert mode)");
|
||||
float const* kv_k_scale_ptr =
|
||||
nvfp4_kv ? kv_k_scale->const_data_ptr<float>() : nullptr;
|
||||
float const* kv_v_scale_ptr =
|
||||
nvfp4_kv ? kv_v_scale->const_data_ptr<float>() : nullptr;
|
||||
|
||||
const torch::stable::accelerator::DeviceGuard device_guard(
|
||||
qkv.get_device_index());
|
||||
auto stream = get_current_cuda_stream(qkv.get_device_index());
|
||||
|
||||
if (nvfp4_kv) {
|
||||
#ifndef USE_ROCM
|
||||
VLLM_STABLE_DISPATCH_HALF_TYPES(
|
||||
qkv.scalar_type(), "fused_minimax_m3_qknorm_rope_kv_insert_nvfp4", [&] {
|
||||
using st = scalar_t;
|
||||
CALL_FUSED_MINIMAX_M3_IMPL(uint8_t, vllm::Fp8KVCacheDataType::kAuto,
|
||||
true);
|
||||
});
|
||||
#else
|
||||
STD_TORCH_CHECK(false, "nvfp4 kv_cache is not supported on ROCm");
|
||||
#endif
|
||||
return;
|
||||
}
|
||||
|
||||
VLLM_STABLE_DISPATCH_HALF_TYPES(
|
||||
qkv.scalar_type(), "fused_minimax_m3_qknorm_rope_kv_insert", [&] {
|
||||
using st = scalar_t;
|
||||
@@ -867,3 +1215,4 @@ void fused_minimax_m3_qknorm_rope_kv_insert(
|
||||
}
|
||||
|
||||
#undef CALL_FUSED_MINIMAX_M3
|
||||
#undef CALL_FUSED_MINIMAX_M3_IMPL
|
||||
|
||||
@@ -30,9 +30,7 @@
|
||||
namespace {
|
||||
|
||||
constexpr int kWarpSize = 32;
|
||||
// Sentinel in the shared top-k scratch: entry already resolved (hit /
|
||||
// newest / invalid), no miss handling needed.
|
||||
constexpr int32_t kTokenDone = -1;
|
||||
// Empty key in the shared open-addressing table of host rows.
|
||||
constexpr int32_t kHashEmpty = -1;
|
||||
|
||||
bool is_pinned_cpu_tensor(const torch::stable::Tensor& tensor) {
|
||||
@@ -46,10 +44,15 @@ bool is_pinned_cpu_tensor(const torch::stable::Tensor& tensor) {
|
||||
return attributes.type == cudaMemoryTypeHost;
|
||||
}
|
||||
|
||||
__device__ __forceinline__ int32_t hash_slot(int32_t key, int32_t hash_size) {
|
||||
// Knuth multiplicative hash for the open-addressing table.
|
||||
return static_cast<int32_t>((static_cast<uint32_t>(key) * 2654435761u) %
|
||||
static_cast<uint32_t>(hash_size));
|
||||
__device__ __forceinline__ int32_t hash_slot(int32_t key, int size) {
|
||||
// Knuth multiplicative hash scaled onto [0, size) by a multiply-high, so
|
||||
// the table can take any size without a division.
|
||||
return static_cast<int32_t>(
|
||||
__umulhi(static_cast<uint32_t>(key) * 2654435761u, size));
|
||||
}
|
||||
|
||||
__device__ __forceinline__ int next_slot(int h, int size) {
|
||||
return h + 1 == size ? 0 : h + 1;
|
||||
}
|
||||
|
||||
// Copy one row of `row_bytes` bytes with a single warp. MLA rows take the
|
||||
@@ -176,16 +179,117 @@ __device__ __forceinline__ void store_hot_index(
|
||||
}
|
||||
}
|
||||
|
||||
// One block per batch row.
|
||||
// A union entry's value holds its first (row, top-k) position until phase 2
|
||||
// or 3 resolves it to a hot slot tagged with how it was resolved.
|
||||
constexpr int32_t kEntryHit = 1 << 30;
|
||||
constexpr int32_t kEntryMiss = 1 << 29;
|
||||
constexpr int32_t kEntryInvalid = 1 << 28;
|
||||
constexpr int32_t kEntrySlotMask = 0xFFFF;
|
||||
constexpr int kResidencyCounters = 5;
|
||||
// Per-request table entries per referenced top-k position (load <= 1/4
|
||||
// until the hash_size cap).
|
||||
constexpr int kHashLoadInverse = 4;
|
||||
|
||||
// Translate a request-relative top-k position to its resident GPU row (-1 when
|
||||
// the page is not resident) and returns its host row (-1 when not host-backed).
|
||||
__device__ __forceinline__ int32_t translate_topk_entry(
|
||||
const int32_t token_index, const int32_t request_row,
|
||||
const int32_t* __restrict__ source_block_table,
|
||||
const int32_t* __restrict__ resident_block_table, const int64_t host_rows,
|
||||
const int64_t source_bt_stride, const int32_t source_num_reqs,
|
||||
const int32_t source_num_blocks, const int32_t source_block_size,
|
||||
const int64_t resident_bt_stride, const int32_t resident_num_reqs,
|
||||
const int32_t resident_num_blocks, const int32_t resident_block_size,
|
||||
const int32_t resident_null_block, int32_t& resident_row) {
|
||||
int32_t g = token_index;
|
||||
resident_row = -1;
|
||||
if (source_block_table != nullptr) {
|
||||
const int32_t source_block =
|
||||
token_index >= 0 ? token_index / source_block_size : -1;
|
||||
if (request_row >= 0 && request_row < source_num_reqs &&
|
||||
source_block >= 0 && source_block < source_num_blocks) {
|
||||
const int32_t physical_block =
|
||||
source_block_table[static_cast<int64_t>(request_row) *
|
||||
source_bt_stride +
|
||||
source_block];
|
||||
g = physical_block > 0 ? physical_block * source_block_size +
|
||||
token_index % source_block_size
|
||||
: -1;
|
||||
} else {
|
||||
g = -1;
|
||||
}
|
||||
if (resident_block_table != nullptr) {
|
||||
const int32_t resident_block =
|
||||
token_index >= 0 ? token_index / resident_block_size : -1;
|
||||
if (request_row >= 0 && request_row < resident_num_reqs &&
|
||||
resident_block >= 0 && resident_block < resident_num_blocks) {
|
||||
const int32_t physical_block =
|
||||
resident_block_table[static_cast<int64_t>(request_row) *
|
||||
resident_bt_stride +
|
||||
resident_block];
|
||||
if (physical_block != resident_null_block && physical_block >= 0) {
|
||||
resident_row = physical_block * resident_block_size +
|
||||
token_index % resident_block_size;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return g >= host_rows ? -1 : g;
|
||||
}
|
||||
|
||||
// Write the hot index of one reference to a resolved union entry and tally it.
|
||||
__device__ __forceinline__ void write_entry_index(
|
||||
const int32_t entry, const int row, const int i, const int32_t top_k,
|
||||
int32_t* __restrict__ hot_indices, int32_t* __restrict__ attention_indices,
|
||||
const int64_t attention_row_stride, int32_t* __restrict__ miss_mask,
|
||||
const int32_t* __restrict__ hot_block_table, const int request_row,
|
||||
const int64_t hot_table_stride, const int32_t hot_block_size,
|
||||
const int64_t attention_block_stride, const bool first_reference, int& hits,
|
||||
int& misses) {
|
||||
int32_t* row_out = hot_indices + static_cast<int64_t>(row) * top_k;
|
||||
int32_t* row_attention =
|
||||
attention_indices != nullptr
|
||||
? attention_indices + static_cast<int64_t>(row) * attention_row_stride
|
||||
: nullptr;
|
||||
if (entry & kEntryInvalid) {
|
||||
store_hot_index(row_out, row_attention, i, -1, hot_block_size,
|
||||
attention_block_stride);
|
||||
return;
|
||||
}
|
||||
store_hot_index(row_out, row_attention, i,
|
||||
static_cast<int32_t>(get_physical_hot_row(
|
||||
hot_block_table, request_row, hot_table_stride,
|
||||
hot_block_size, entry & kEntrySlotMask)),
|
||||
hot_block_size, attention_block_stride);
|
||||
// Only an entry's first reference loads it; later references reuse the
|
||||
// slot like a hit.
|
||||
if ((entry & kEntryMiss) && first_reference) {
|
||||
if (miss_mask != nullptr) {
|
||||
miss_mask[static_cast<int64_t>(row) * top_k + i] = 1;
|
||||
}
|
||||
++misses;
|
||||
} else {
|
||||
++hits;
|
||||
}
|
||||
}
|
||||
|
||||
// One block per request, launched per batch row. A request's rows (several
|
||||
// with speculative decoding) are contiguous and share its hot-buffer state, so
|
||||
// the block of its first row resolves the union of their host misses once and
|
||||
// maps every row onto it; the other blocks exit. The union is bounded by the
|
||||
// caller's max_union_rows (every decode query's top-k), which
|
||||
// device_buffer_size covers; a larger union traps. Misses take eviction slots
|
||||
// in (row, top-k) order of first reference, so the plan is deterministic.
|
||||
//
|
||||
// Shared memory layout (int32 region followed by int16 region):
|
||||
// s_topk[top_k] top-k global ids; reused as miss scratch
|
||||
// s_chunk_off[nbc + 1] prefix sums for hit (then miss) compaction
|
||||
// s_hash_keys[hash_size] open addressing: union of the rows' host ids
|
||||
// s_hash_vals[hash_size] first position, then slot | kEntry* tag
|
||||
// s_chunk_off[nbc + 1] prefix sums for hit, then miss, compaction
|
||||
// s_evict_off[nbc + 1] prefix sums for evictable compaction
|
||||
// s_hash_keys[hash_size] open addressing: global id -> top-k index
|
||||
// s_counters[3] hits, phase-1 resolved, valid count
|
||||
// s_counters[5] distinct hits, union size, per-row valid count,
|
||||
// misses, per-entry hits
|
||||
// s_done[nbc] positions phase 3 resolved, one word per chunk
|
||||
// s_lru_out[hot_size] int16, compacted slots: [hits fwd | evict bwd]
|
||||
// s_hash_vals[hash_size] int16 hash values (top-k index)
|
||||
// Valid global ids must be unique within each row.
|
||||
__global__ __launch_bounds__(1024) void hisparse_resolve_residency_kernel(
|
||||
const int32_t* __restrict__ hot_block_table, // [max_rows, hot_blocks]
|
||||
@@ -218,11 +322,25 @@ __global__ __launch_bounds__(1024) void hisparse_resolve_residency_kernel(
|
||||
const int64_t attention_row_stride, const int64_t valid_count_stride) {
|
||||
const int NUM_WARPS = blockDim.x / kWarpSize;
|
||||
const int num_buffer_chunks = (hot_size + kWarpSize - 1) / kWarpSize;
|
||||
const int num_token_chunks = (top_k + kWarpSize - 1) / kWarpSize;
|
||||
|
||||
const int batch_row = blockIdx.x;
|
||||
// The first row of each contiguous run of a request's rows leads it. FULL
|
||||
// CUDA-graph padding rows map to request 0 after the real rows, forming a
|
||||
// second run, but carry no valid top-k: that run returns before phase 2
|
||||
// touches the request's shared state.
|
||||
const int first_row = blockIdx.x;
|
||||
const int request_row =
|
||||
request_ids != nullptr ? request_ids[batch_row] : batch_row;
|
||||
request_ids != nullptr ? request_ids[first_row] : first_row;
|
||||
if (request_ids != nullptr && first_row > 0 &&
|
||||
request_ids[first_row - 1] == request_row) {
|
||||
return;
|
||||
}
|
||||
int end_row = first_row + 1;
|
||||
if (request_ids != nullptr) {
|
||||
while (end_row < static_cast<int>(gridDim.x) &&
|
||||
request_ids[end_row] == request_row) {
|
||||
++end_row;
|
||||
}
|
||||
}
|
||||
const int state_row =
|
||||
request_state_indices != nullptr && request_row >= 0 &&
|
||||
request_row < request_state_count
|
||||
@@ -230,23 +348,24 @@ __global__ __launch_bounds__(1024) void hisparse_resolve_residency_kernel(
|
||||
: (request_state_indices == nullptr ? request_row : -1);
|
||||
// V2 publishes -1 for CUDA-graph padding rows.
|
||||
if (state_row < 0) {
|
||||
for (int i = threadIdx.x; i < top_k; i += blockDim.x) {
|
||||
const int64_t index = static_cast<int64_t>(batch_row) * top_k + i;
|
||||
hot_indices[index] = -1;
|
||||
if (attention_indices != nullptr) {
|
||||
attention_indices[static_cast<int64_t>(batch_row) *
|
||||
attention_row_stride +
|
||||
i] = -1;
|
||||
for (int row = first_row; row < end_row; ++row) {
|
||||
for (int i = threadIdx.x; i < top_k; i += blockDim.x) {
|
||||
const int64_t index = static_cast<int64_t>(row) * top_k + i;
|
||||
hot_indices[index] = -1;
|
||||
if (attention_indices != nullptr) {
|
||||
attention_indices[static_cast<int64_t>(row) * attention_row_stride +
|
||||
i] = -1;
|
||||
}
|
||||
if (resolved_global_indices != nullptr) {
|
||||
resolved_global_indices[index] = -1;
|
||||
}
|
||||
}
|
||||
if (resolved_global_indices != nullptr) {
|
||||
resolved_global_indices[index] = -1;
|
||||
if (valid_counts != nullptr && threadIdx.x == 0) {
|
||||
valid_counts[static_cast<int64_t>(row) * valid_count_stride] = 0;
|
||||
}
|
||||
if (swap_counts != nullptr && threadIdx.x == 0) {
|
||||
swap_counts[row] = 0;
|
||||
}
|
||||
}
|
||||
if (valid_counts != nullptr && threadIdx.x == 0) {
|
||||
valid_counts[static_cast<int64_t>(batch_row) * valid_count_stride] = 0;
|
||||
}
|
||||
if (swap_counts != nullptr && threadIdx.x == 0) {
|
||||
swap_counts[batch_row] = 0;
|
||||
}
|
||||
return;
|
||||
}
|
||||
@@ -255,136 +374,130 @@ __global__ __launch_bounds__(1024) void hisparse_resolve_residency_kernel(
|
||||
const int lane_id = tid % kWarpSize;
|
||||
const unsigned int lanes_before = ((unsigned int)1 << lane_id) - 1;
|
||||
|
||||
const int32_t* row_topk =
|
||||
global_indices + static_cast<int64_t>(batch_row) * input_row_stride;
|
||||
int32_t* row_out = hot_indices + static_cast<int64_t>(batch_row) * top_k;
|
||||
int32_t* row_attention =
|
||||
attention_indices != nullptr
|
||||
? attention_indices +
|
||||
static_cast<int64_t>(batch_row) * attention_row_stride
|
||||
: nullptr;
|
||||
int32_t* row_miss = (miss_mask != nullptr)
|
||||
? miss_mask + static_cast<int64_t>(batch_row) * top_k
|
||||
: nullptr;
|
||||
int32_t* row_dgi =
|
||||
device_global_indices + static_cast<int64_t>(state_row) * region_stride;
|
||||
int16_t* row_lru = lru_slots + static_cast<int64_t>(state_row) * hot_size;
|
||||
|
||||
// A union within max_union_rows (hash_size - 1) and within the rows' top-k
|
||||
// leaves an empty slot, so every probe sequence ends. Use a slice sized to
|
||||
// this request's rows so single-row requests initialize a small table.
|
||||
const int table_size =
|
||||
min(hash_size, kHashLoadInverse * (end_row - first_row) * top_k);
|
||||
|
||||
extern __shared__ char smem_raw[];
|
||||
int32_t* s_topk = reinterpret_cast<int32_t*>(smem_raw);
|
||||
int32_t* s_chunk_off = s_topk + top_k;
|
||||
int32_t* s_hash_keys = reinterpret_cast<int32_t*>(smem_raw);
|
||||
int32_t* s_hash_vals = s_hash_keys + hash_size;
|
||||
int32_t* s_chunk_off = s_hash_vals + hash_size;
|
||||
int32_t* s_evict_off = s_chunk_off + (num_buffer_chunks + 1);
|
||||
int32_t* s_hash_keys = s_evict_off + (num_buffer_chunks + 1);
|
||||
int32_t* s_counters = s_hash_keys + hash_size;
|
||||
int16_t* s_lru_out = reinterpret_cast<int16_t*>(s_counters + 3);
|
||||
int16_t* s_hash_vals = s_lru_out + hot_size;
|
||||
int32_t* s_counters = s_evict_off + (num_buffer_chunks + 1);
|
||||
unsigned int* s_done =
|
||||
reinterpret_cast<unsigned int*>(s_counters + kResidencyCounters);
|
||||
int16_t* s_lru_out = reinterpret_cast<int16_t*>(s_done + num_buffer_chunks);
|
||||
|
||||
if (tid < 3) {
|
||||
s_counters[tid] = 0;
|
||||
}
|
||||
__syncthreads();
|
||||
|
||||
// Phase 1: translate request-relative positions and resolve resident rows.
|
||||
for (int i = tid; i < top_k; i += blockDim.x) {
|
||||
const int32_t token_index = row_topk[i];
|
||||
int32_t g = token_index;
|
||||
int32_t resident_row = -1;
|
||||
if (source_block_table != nullptr) {
|
||||
const int32_t request_id = request_ids[batch_row];
|
||||
const int32_t source_block =
|
||||
token_index >= 0 ? token_index / source_block_size : -1;
|
||||
if (request_id >= 0 && request_id < source_num_reqs &&
|
||||
source_block >= 0 && source_block < source_num_blocks) {
|
||||
const int32_t physical_block =
|
||||
source_block_table[static_cast<int64_t>(request_id) *
|
||||
source_bt_stride +
|
||||
source_block];
|
||||
g = physical_block > 0 ? physical_block * source_block_size +
|
||||
token_index % source_block_size
|
||||
: -1;
|
||||
} else {
|
||||
g = -1;
|
||||
}
|
||||
if (resident_block_table != nullptr) {
|
||||
const int32_t resident_block =
|
||||
token_index >= 0 ? token_index / resident_block_size : -1;
|
||||
if (request_id >= 0 && request_id < resident_num_reqs &&
|
||||
resident_block >= 0 && resident_block < resident_num_blocks) {
|
||||
const int32_t physical_block =
|
||||
resident_block_table[static_cast<int64_t>(request_id) *
|
||||
resident_bt_stride +
|
||||
resident_block];
|
||||
if (physical_block != resident_null_block && physical_block >= 0) {
|
||||
resident_row = physical_block * resident_block_size +
|
||||
token_index % resident_block_size;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if (g >= host_rows) {
|
||||
g = -1;
|
||||
}
|
||||
if (resolved_global_indices != nullptr) {
|
||||
resolved_global_indices[static_cast<int64_t>(batch_row) * top_k + i] = g;
|
||||
}
|
||||
if (resident_row >= 0 || g >= 0) atomicAdd(&s_counters[2], 1);
|
||||
if (row_miss != nullptr) row_miss[i] = 0;
|
||||
if (resident_row >= 0) {
|
||||
store_hot_index(row_out, row_attention, i, resident_row, hot_block_size,
|
||||
attention_block_stride);
|
||||
s_topk[i] = kTokenDone;
|
||||
atomicAdd(&s_counters[1], 1);
|
||||
} else if (g < 0) {
|
||||
store_hot_index(row_out, row_attention, i, -1, hot_block_size,
|
||||
attention_block_stride);
|
||||
s_topk[i] = kTokenDone;
|
||||
atomicAdd(&s_counters[1], 1);
|
||||
} else {
|
||||
s_topk[i] = g;
|
||||
}
|
||||
}
|
||||
__syncthreads();
|
||||
if (valid_counts != nullptr && tid == 0) {
|
||||
valid_counts[static_cast<int64_t>(batch_row) * valid_count_stride] =
|
||||
s_counters[2];
|
||||
}
|
||||
// Fully resident rows need only request-relative page translation. Avoid
|
||||
// scanning or rewriting the hot LRU when no selected row can consult it.
|
||||
if (s_counters[1] == top_k) {
|
||||
if (tid == 0 && swap_counts != nullptr) {
|
||||
swap_counts[batch_row] = 0;
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
for (int i = tid; i < hash_size; i += blockDim.x) {
|
||||
for (int i = tid; i < table_size; i += blockDim.x) {
|
||||
s_hash_keys[i] = kHashEmpty;
|
||||
s_hash_vals[i] = INT32_MAX;
|
||||
}
|
||||
for (int i = tid; i < num_buffer_chunks + 1; i += blockDim.x) {
|
||||
s_chunk_off[i] = 0;
|
||||
s_evict_off[i] = 0;
|
||||
}
|
||||
__syncthreads();
|
||||
|
||||
for (int i = tid; i < top_k; i += blockDim.x) {
|
||||
const int32_t g = s_topk[i];
|
||||
if (g == kTokenDone) {
|
||||
continue;
|
||||
}
|
||||
int slot = hash_slot(g, hash_size);
|
||||
while (true) {
|
||||
const int32_t old = atomicCAS(&s_hash_keys[slot], kHashEmpty, g);
|
||||
if (old == kHashEmpty || old == g) {
|
||||
s_hash_vals[slot] = static_cast<int16_t>(i);
|
||||
break;
|
||||
}
|
||||
slot = (slot + 1) % hash_size;
|
||||
}
|
||||
if (tid < kResidencyCounters) {
|
||||
s_counters[tid] = 0;
|
||||
}
|
||||
__syncthreads();
|
||||
|
||||
// Phase 1: translate every row's request-relative positions, resolve
|
||||
// resident rows directly, and collect the union of host rows.
|
||||
for (int row = first_row; row < end_row; ++row) {
|
||||
const int32_t* row_topk =
|
||||
global_indices + static_cast<int64_t>(row) * input_row_stride;
|
||||
int32_t* row_out = hot_indices + static_cast<int64_t>(row) * top_k;
|
||||
int32_t* row_attention =
|
||||
attention_indices != nullptr
|
||||
? attention_indices +
|
||||
static_cast<int64_t>(row) * attention_row_stride
|
||||
: nullptr;
|
||||
int32_t* row_miss = miss_mask != nullptr
|
||||
? miss_mask + static_cast<int64_t>(row) * top_k
|
||||
: nullptr;
|
||||
for (int i = tid; i < top_k; i += blockDim.x) {
|
||||
int32_t resident_row;
|
||||
const int32_t g = translate_topk_entry(
|
||||
row_topk[i], request_row, source_block_table, resident_block_table,
|
||||
host_rows, source_bt_stride, source_num_reqs, source_num_blocks,
|
||||
source_block_size, resident_bt_stride, resident_num_reqs,
|
||||
resident_num_blocks, resident_block_size, resident_null_block,
|
||||
resident_row);
|
||||
if (resolved_global_indices != nullptr) {
|
||||
resolved_global_indices[static_cast<int64_t>(row) * top_k + i] = g;
|
||||
}
|
||||
if (resident_row >= 0 || g >= 0) atomicAdd(&s_counters[2], 1);
|
||||
if (row_miss != nullptr) row_miss[i] = 0;
|
||||
if (resident_row >= 0) {
|
||||
store_hot_index(row_out, row_attention, i, resident_row, hot_block_size,
|
||||
attention_block_stride);
|
||||
} else if (g < 0) {
|
||||
store_hot_index(row_out, row_attention, i, -1, hot_block_size,
|
||||
attention_block_stride);
|
||||
} else {
|
||||
int h = hash_slot(g, table_size);
|
||||
bool inserted = false;
|
||||
for (int probe = 0; probe < table_size; ++probe) {
|
||||
const int32_t old = atomicCAS(&s_hash_keys[h], kHashEmpty, g);
|
||||
if (old == kHashEmpty) {
|
||||
atomicAdd(&s_counters[1], 1);
|
||||
inserted = true;
|
||||
break;
|
||||
}
|
||||
if (old == g) {
|
||||
inserted = true;
|
||||
break;
|
||||
}
|
||||
h = next_slot(h, table_size);
|
||||
}
|
||||
if (inserted) {
|
||||
atomicMin(&s_hash_vals[h], (row - first_row) * top_k + i);
|
||||
} else {
|
||||
// A full table: push the union count past the cap so the check
|
||||
// below traps.
|
||||
atomicAdd(&s_counters[1], hash_size);
|
||||
}
|
||||
}
|
||||
}
|
||||
__syncthreads();
|
||||
if (tid == 0) {
|
||||
if (valid_counts != nullptr) {
|
||||
valid_counts[static_cast<int64_t>(row) * valid_count_stride] =
|
||||
s_counters[2];
|
||||
}
|
||||
s_counters[2] = 0;
|
||||
}
|
||||
__syncthreads();
|
||||
}
|
||||
// The table and eviction slots are sized for max_union_rows (hash_size - 1);
|
||||
// a larger union would never reach an empty probe slot or run past s_lru_out.
|
||||
if (s_counters[1] >= hash_size) {
|
||||
if (tid == 0) {
|
||||
printf(
|
||||
"HiSparse residency: request %d resolves %d host rows, above the "
|
||||
"configured bound of %d\n",
|
||||
request_row, s_counters[1], hash_size - 1);
|
||||
}
|
||||
__trap();
|
||||
}
|
||||
// Fully resident rows need only request-relative page translation. Avoid
|
||||
// scanning or rewriting the hot LRU when no selected row can consult it.
|
||||
if (s_counters[1] == 0) {
|
||||
for (int row = first_row + tid; row < end_row; row += blockDim.x) {
|
||||
if (swap_counts != nullptr) swap_counts[row] = 0;
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
// Phase 2: walk hot slots in LRU order, classify hit / evictable, and
|
||||
// compact them (hits forward, evictables backward) into s_lru_out.
|
||||
// compact them (hits forward, evictables backward) into s_lru_out. A hit
|
||||
// records its slot on the union entry.
|
||||
const int iters_buffer = (num_buffer_chunks + NUM_WARPS - 1) / NUM_WARPS;
|
||||
int total_hit_count = 0;
|
||||
int total_evict_count = 0;
|
||||
@@ -397,36 +510,28 @@ __global__ __launch_bounds__(1024) void hisparse_resolve_residency_kernel(
|
||||
const int16_t slot = has_valid_pos ? row_lru[pos] : int16_t(-1);
|
||||
// Corruption tripwire: lru/dgi are long-lived device state; if some
|
||||
// external writer (e.g. stray RDMA into reused VRAM) corrupts a slot
|
||||
// out of [0, region_stride), treat it as no-hit so it degrades to a re-miss
|
||||
// instead of an unbounded dgi read (phases 3/5 bound the write side).
|
||||
// out of [0, region_stride), treat it as no-hit so it degrades to a
|
||||
// re-miss instead of an unbounded dgi read (phase 3 bounds the write
|
||||
// side).
|
||||
const int32_t cached_g =
|
||||
(slot >= 0 && slot < region_stride) ? row_dgi[slot] : -1;
|
||||
|
||||
int found_topk_idx = -1;
|
||||
bool is_hit = false;
|
||||
if (cached_g >= 0) {
|
||||
int h = hash_slot(cached_g, hash_size);
|
||||
int h = hash_slot(cached_g, table_size);
|
||||
while (true) {
|
||||
const int32_t k = s_hash_keys[h];
|
||||
if (k == cached_g) {
|
||||
found_topk_idx = static_cast<int32_t>(s_hash_vals[h]);
|
||||
s_hash_vals[h] = kEntryHit | slot;
|
||||
is_hit = true;
|
||||
break;
|
||||
}
|
||||
if (k == kHashEmpty) break;
|
||||
h = (h + 1) % hash_size;
|
||||
h = next_slot(h, table_size);
|
||||
}
|
||||
}
|
||||
const bool is_hit = found_topk_idx >= 0;
|
||||
const bool is_evictable = has_valid_pos && !is_hit;
|
||||
|
||||
if (is_hit) {
|
||||
s_topk[found_topk_idx] = kTokenDone;
|
||||
store_hot_index(row_out, row_attention, found_topk_idx,
|
||||
static_cast<int32_t>(get_physical_hot_row(
|
||||
hot_block_table, request_row, hot_table_stride,
|
||||
hot_block_size, slot)),
|
||||
hot_block_size, attention_block_stride);
|
||||
}
|
||||
|
||||
int local_hit_off = 0;
|
||||
int local_evict_off = 0;
|
||||
if (has_valid_chunk) {
|
||||
@@ -465,39 +570,59 @@ __global__ __launch_bounds__(1024) void hisparse_resolve_residency_kernel(
|
||||
}
|
||||
__syncthreads();
|
||||
|
||||
// Reset prefix sums for the miss compaction (token chunks <= buffer
|
||||
// chunks because hot_size >= top_k).
|
||||
for (int i = tid; i < num_token_chunks + 1; i += blockDim.x) {
|
||||
for (int i = tid; i < num_buffer_chunks + 1; i += blockDim.x) {
|
||||
s_chunk_off[i] = 0;
|
||||
}
|
||||
__syncthreads();
|
||||
|
||||
// Phase 3: compact misses, assign them eviction slots (oldest first) and
|
||||
// record the new ownership in device_global_indices.
|
||||
const int iters_token = (num_token_chunks + NUM_WARPS - 1) / NUM_WARPS;
|
||||
// Phase 3: walk the rows' positions in (row, top-k) order; the first
|
||||
// reference to a union entry no slot holds is a miss. Assign misses
|
||||
// eviction slots (oldest first) and record the new ownership in
|
||||
// device_global_indices. Swaps fill the rows' swap segments in order; the
|
||||
// union never exceeds their combined capacity, and position chunks never
|
||||
// exceed buffer chunks because hot_size covers every row's top-k.
|
||||
const int num_positions = (end_row - first_row) * top_k;
|
||||
const int num_position_chunks = (num_positions + kWarpSize - 1) / kWarpSize;
|
||||
const int iters_position = (num_position_chunks + NUM_WARPS - 1) / NUM_WARPS;
|
||||
int miss_running_total = 0;
|
||||
for (int iter = 0; iter < iters_token; iter++) {
|
||||
int entry_hits = 0;
|
||||
int entry_misses = 0;
|
||||
for (int iter = 0; iter < iters_position; iter++) {
|
||||
const int chunk_idx = warp_id + iter * NUM_WARPS;
|
||||
const bool has_valid_chunk = chunk_idx < num_token_chunks;
|
||||
const bool has_valid_chunk = chunk_idx < num_position_chunks;
|
||||
|
||||
const int i = chunk_idx * kWarpSize + lane_id;
|
||||
const bool has_valid_token = has_valid_chunk && (i < top_k);
|
||||
|
||||
int32_t g = 0;
|
||||
const int position = chunk_idx * kWarpSize + lane_id;
|
||||
int32_t g = -1;
|
||||
int h = -1;
|
||||
int32_t entry = 0;
|
||||
bool is_miss = false;
|
||||
if (has_valid_token) {
|
||||
is_miss = s_topk[i] != kTokenDone;
|
||||
if (is_miss) {
|
||||
g = s_topk[i];
|
||||
if (has_valid_chunk && position < num_positions) {
|
||||
const int row = first_row + position / top_k;
|
||||
int32_t resident_row;
|
||||
g = translate_topk_entry(
|
||||
global_indices[static_cast<int64_t>(row) * input_row_stride +
|
||||
position % top_k],
|
||||
request_row, source_block_table, resident_block_table, host_rows,
|
||||
source_bt_stride, source_num_reqs, source_num_blocks,
|
||||
source_block_size, resident_bt_stride, resident_num_reqs,
|
||||
resident_num_blocks, resident_block_size, resident_null_block,
|
||||
resident_row);
|
||||
if (resident_row < 0 && g >= 0) {
|
||||
h = hash_slot(g, table_size);
|
||||
while (s_hash_keys[h] != g) {
|
||||
h = next_slot(h, table_size);
|
||||
}
|
||||
entry = s_hash_vals[h];
|
||||
is_miss = entry == position;
|
||||
}
|
||||
}
|
||||
|
||||
int local_miss_off = 0;
|
||||
if (has_valid_chunk) {
|
||||
const unsigned int miss_mask = __ballot_sync(0xFFFFFFFF, is_miss);
|
||||
local_miss_off = __popc(miss_mask & lanes_before);
|
||||
const unsigned int miss_mask_bits = __ballot_sync(0xFFFFFFFF, is_miss);
|
||||
local_miss_off = __popc(miss_mask_bits & lanes_before);
|
||||
if (lane_id == 0) {
|
||||
s_chunk_off[chunk_idx + 1] = __popc(miss_mask);
|
||||
s_chunk_off[chunk_idx + 1] = __popc(miss_mask_bits);
|
||||
}
|
||||
}
|
||||
__syncthreads();
|
||||
@@ -505,60 +630,112 @@ __global__ __launch_bounds__(1024) void hisparse_resolve_residency_kernel(
|
||||
if (warp_id == 0) {
|
||||
miss_running_total =
|
||||
warp_inclusive_scan(s_chunk_off, lane_id, chunk_idx + 1,
|
||||
num_token_chunks + 1, miss_running_total);
|
||||
num_position_chunks + 1, miss_running_total);
|
||||
if (tid == 0) {
|
||||
s_counters[3] = miss_running_total;
|
||||
}
|
||||
}
|
||||
__syncthreads();
|
||||
|
||||
if (is_miss) {
|
||||
const int m = s_chunk_off[chunk_idx] + local_miss_off;
|
||||
const int16_t evict_slot = s_lru_out[hot_size - 1 - m];
|
||||
const int64_t compact_index = static_cast<int64_t>(first_row) * top_k + m;
|
||||
if (evict_slot < 0 || evict_slot >= region_stride) {
|
||||
// Corruption tripwire: an out-of-range slot (corrupted lru state,
|
||||
// see phase 2) must not become a hot-cache/dgi write. Resolve the
|
||||
// entry as invalid (-1, masked by attention, not in the miss set);
|
||||
// it re-misses on a later step. Phase 5 re-checks the same
|
||||
// s_lru_out value, so its copy is skipped consistently.
|
||||
store_hot_index(row_out, row_attention, i, -1, hot_block_size,
|
||||
attention_block_stride);
|
||||
// entry as invalid (-1, masked by attention); it re-misses on a later
|
||||
// step, and its copy is skipped.
|
||||
s_hash_vals[h] = kEntryInvalid;
|
||||
if (swap_host_physical_rows != nullptr) {
|
||||
const int64_t compact_index =
|
||||
static_cast<int64_t>(batch_row) * top_k + m;
|
||||
swap_host_physical_rows[compact_index] = g;
|
||||
swap_device_physical_rows[compact_index] = -1;
|
||||
}
|
||||
} else {
|
||||
// Reuse s_topk as compacted miss scratch: m < i always holds (done
|
||||
// entries are skipped), so writes never overrun pending reads.
|
||||
s_topk[m] = g;
|
||||
const int32_t physical_row = static_cast<int32_t>(
|
||||
get_physical_hot_row(hot_block_table, request_row, hot_table_stride,
|
||||
hot_block_size, evict_slot));
|
||||
store_hot_index(row_out, row_attention, i, physical_row, hot_block_size,
|
||||
attention_block_stride);
|
||||
s_hash_vals[h] = kEntryMiss | evict_slot;
|
||||
if (swap_host_physical_rows != nullptr) {
|
||||
const int64_t compact_index =
|
||||
static_cast<int64_t>(batch_row) * top_k + m;
|
||||
swap_host_physical_rows[compact_index] = g;
|
||||
swap_device_physical_rows[compact_index] = physical_row;
|
||||
swap_device_physical_rows[compact_index] =
|
||||
static_cast<int32_t>(get_physical_hot_row(
|
||||
hot_block_table, request_row, hot_table_stride,
|
||||
hot_block_size, evict_slot));
|
||||
}
|
||||
if (row_miss != nullptr) row_miss[i] = 1;
|
||||
row_dgi[evict_slot] = g;
|
||||
}
|
||||
entry = s_hash_vals[h];
|
||||
}
|
||||
|
||||
// Hits, first references and references to entries resolved in an
|
||||
// earlier chunk write their hot index now; later references to an entry
|
||||
// resolved in this same chunk pass (only across rows) are left to phase 4.
|
||||
const bool resolved =
|
||||
h >= 0 && (entry & (kEntryHit | kEntryMiss | kEntryInvalid));
|
||||
if (resolved) {
|
||||
write_entry_index(
|
||||
entry, first_row + position / top_k, position % top_k, top_k,
|
||||
hot_indices, attention_indices, attention_row_stride, miss_mask,
|
||||
hot_block_table, request_row, hot_table_stride, hot_block_size,
|
||||
attention_block_stride, is_miss, entry_hits, entry_misses);
|
||||
}
|
||||
if (has_valid_chunk) {
|
||||
const unsigned int done_bits = __ballot_sync(0xFFFFFFFF, resolved);
|
||||
if (lane_id == 0) {
|
||||
s_done[chunk_idx] = done_bits;
|
||||
}
|
||||
}
|
||||
}
|
||||
__syncthreads();
|
||||
|
||||
const int total_hits = s_counters[0];
|
||||
const int total_misses = top_k - total_hits - s_counters[1];
|
||||
if (swap_counts != nullptr && tid == 0) {
|
||||
swap_counts[batch_row] = total_misses;
|
||||
}
|
||||
if (stats != nullptr && tid == 0) {
|
||||
atomicAdd(&stats[0], static_cast<unsigned long long>(total_hits));
|
||||
atomicAdd(&stats[1], static_cast<unsigned long long>(total_misses));
|
||||
const int total_misses = s_counters[3];
|
||||
if (swap_counts != nullptr) {
|
||||
for (int row = first_row + tid; row < end_row; row += blockDim.x) {
|
||||
const int remaining = total_misses - (row - first_row) * top_k;
|
||||
swap_counts[row] = max(0, min(top_k, remaining));
|
||||
}
|
||||
}
|
||||
|
||||
// Phase 4: write back the LRU order: stale evictables at the front,
|
||||
// Phase 4: references phase 3 left unresolved (later references across rows
|
||||
// to entries loaded in the same chunk pass) map onto the union's slots.
|
||||
// Single-row requests resolve everything in phase 3.
|
||||
if (end_row - first_row > 1) {
|
||||
for (int position = tid; position < num_positions; position += blockDim.x) {
|
||||
if (s_done[position / kWarpSize] & (1u << (position % kWarpSize))) {
|
||||
continue;
|
||||
}
|
||||
const int row = first_row + position / top_k;
|
||||
int32_t resident_row;
|
||||
const int32_t g = translate_topk_entry(
|
||||
global_indices[static_cast<int64_t>(row) * input_row_stride +
|
||||
position % top_k],
|
||||
request_row, source_block_table, resident_block_table, host_rows,
|
||||
source_bt_stride, source_num_reqs, source_num_blocks,
|
||||
source_block_size, resident_bt_stride, resident_num_reqs,
|
||||
resident_num_blocks, resident_block_size, resident_null_block,
|
||||
resident_row);
|
||||
if (resident_row >= 0 || g < 0) continue;
|
||||
int h = hash_slot(g, table_size);
|
||||
while (s_hash_keys[h] != g) {
|
||||
h = next_slot(h, table_size);
|
||||
}
|
||||
write_entry_index(
|
||||
s_hash_vals[h], row, position % top_k, top_k, hot_indices,
|
||||
attention_indices, attention_row_stride, miss_mask, hot_block_table,
|
||||
request_row, hot_table_stride, hot_block_size, attention_block_stride,
|
||||
false, entry_hits, entry_misses);
|
||||
}
|
||||
}
|
||||
if (stats != nullptr) {
|
||||
atomicAdd(&s_counters[4], entry_hits);
|
||||
atomicAdd(&s_counters[2], entry_misses);
|
||||
__syncthreads();
|
||||
if (tid == 0) {
|
||||
atomicAdd(&stats[0], static_cast<unsigned long long>(s_counters[4]));
|
||||
atomicAdd(&stats[1], static_cast<unsigned long long>(s_counters[2]));
|
||||
}
|
||||
}
|
||||
|
||||
// Phase 5: write back the LRU order: stale evictables at the front,
|
||||
// freshly loaded misses next, then hits at MRU.
|
||||
const int total_evictable = hot_size - total_hits;
|
||||
const int remaining_evictable = total_evictable - total_misses;
|
||||
@@ -783,7 +960,7 @@ void hisparse_resolve_residency(
|
||||
torch::stable::Tensor& device_global_indices,
|
||||
torch::stable::Tensor& lru_slots,
|
||||
std::optional<torch::stable::Tensor> const& request_state_indices,
|
||||
int64_t region_stride,
|
||||
int64_t region_stride, int64_t max_union_rows,
|
||||
std::optional<torch::stable::Tensor> const& miss_mask,
|
||||
std::optional<torch::stable::Tensor> const& stats,
|
||||
std::optional<torch::stable::Tensor> const& attention_indices,
|
||||
@@ -1034,14 +1211,32 @@ void hisparse_resolve_residency(
|
||||
}
|
||||
|
||||
constexpr int kBlockSize = 1024;
|
||||
const int hash_size = 2 * top_k;
|
||||
STD_TORCH_CHECK(max_union_rows > 0 && max_union_rows <= hot_size,
|
||||
"max_union_rows must be in [1, hot_size], got ",
|
||||
max_union_rows);
|
||||
// One spare entry above the union bound keeps every probe sequence finite.
|
||||
const int hash_size = static_cast<int>(max_union_rows) + 1;
|
||||
const int num_buffer_chunks = (hot_size + kWarpSize - 1) / kWarpSize;
|
||||
const size_t smem_bytes =
|
||||
sizeof(int32_t) * (top_k + 2 * (num_buffer_chunks + 1) + hash_size + 3) +
|
||||
sizeof(int16_t) * (hot_size + hash_size);
|
||||
sizeof(int32_t) * (2 * hash_size + 2 * (num_buffer_chunks + 1) +
|
||||
kResidencyCounters + num_buffer_chunks) +
|
||||
sizeof(int16_t) * hot_size;
|
||||
|
||||
const torch::stable::accelerator::DeviceGuard device_guard(
|
||||
hot_cache.get_device_index());
|
||||
int max_smem_bytes = 0;
|
||||
STD_TORCH_CHECK(cudaDeviceGetAttribute(
|
||||
&max_smem_bytes, cudaDevAttrMaxSharedMemoryPerBlockOptin,
|
||||
hot_cache.get_device_index()) == cudaSuccess,
|
||||
"failed to query shared memory per block");
|
||||
STD_TORCH_CHECK(smem_bytes <= static_cast<size_t>(max_smem_bytes),
|
||||
"HiSparse device_buffer_size=", hot_size, " needs ",
|
||||
smem_bytes,
|
||||
" bytes of shared memory to resolve residency, "
|
||||
"but the device allows ",
|
||||
max_smem_bytes,
|
||||
"; lower device_buffer_size or the number of speculative "
|
||||
"tokens.");
|
||||
const cudaStream_t stream = get_current_cuda_stream();
|
||||
const int64_t attention_row_stride =
|
||||
attention_indices.has_value() ? attention_indices.value().stride(0) : 0;
|
||||
|
||||
@@ -381,7 +381,9 @@ void fused_minimax_m3_qknorm_rope_kv_insert(
|
||||
std::optional<torch::stable::Tensor> q_out,
|
||||
std::optional<torch::stable::Tensor> index_q_out,
|
||||
const std::string& kv_cache_dtype, bool skip_index_branch,
|
||||
std::optional<torch::stable::Tensor> q_fp8_out, double q_fp8_scale);
|
||||
std::optional<torch::stable::Tensor> q_fp8_out, double q_fp8_scale,
|
||||
std::optional<torch::stable::Tensor> kv_k_scale,
|
||||
std::optional<torch::stable::Tensor> kv_v_scale);
|
||||
|
||||
#ifdef VLLM_ENABLE_FUSED_KDA_DECODE
|
||||
void fused_kda_decode(
|
||||
@@ -680,7 +682,7 @@ void hisparse_resolve_residency(
|
||||
torch::stable::Tensor& device_global_indices,
|
||||
torch::stable::Tensor& lru_slots,
|
||||
std::optional<torch::stable::Tensor> const& request_state_indices,
|
||||
int64_t region_stride,
|
||||
int64_t region_stride, int64_t max_union_rows,
|
||||
std::optional<torch::stable::Tensor> const& miss_mask,
|
||||
std::optional<torch::stable::Tensor> const& stats,
|
||||
std::optional<torch::stable::Tensor> const& attention_indices,
|
||||
|
||||
@@ -513,7 +513,8 @@ STABLE_TORCH_LIBRARY_FRAGMENT(_C, ops) {
|
||||
"Tensor!? kv_cache, Tensor!? index_cache, "
|
||||
"int block_size, Tensor!? q_out, Tensor!? index_q_out, "
|
||||
"str kv_cache_dtype, bool skip_index_branch=False, "
|
||||
"Tensor!? q_fp8_out=None, float q_fp8_scale=1.0) -> ()");
|
||||
"Tensor!? q_fp8_out=None, float q_fp8_scale=1.0, "
|
||||
"Tensor? kv_k_scale=None, Tensor? kv_v_scale=None) -> ()");
|
||||
|
||||
#ifdef VLLM_ENABLE_FUSED_KDA_DECODE
|
||||
ops.def(
|
||||
@@ -995,6 +996,7 @@ STABLE_TORCH_LIBRARY_FRAGMENT(_C_cache_ops, ops) {
|
||||
" Tensor! lru_slots,"
|
||||
" Tensor? request_state_indices,"
|
||||
" int region_stride,"
|
||||
" int max_union_rows,"
|
||||
" Tensor(a!)? miss_mask=None,"
|
||||
" Tensor(b!)? stats=None,"
|
||||
" Tensor(c!)? attention_indices=None,"
|
||||
|
||||
+2
-1
@@ -906,6 +906,7 @@ ENV UV_INDEX_STRATEGY="unsafe-best-match"
|
||||
ENV UV_LINK_MODE=copy
|
||||
ENV UV_PYTHON_INSTALL_DIR=/opt/uv/python
|
||||
ENV UV_CACHE_DIR=/opt/uv/cache
|
||||
ENV UV_COMPILE_BYTECODE=1
|
||||
RUN mkdir -p "${UV_PYTHON_INSTALL_DIR}" "${UV_CACHE_DIR}" \
|
||||
&& chgrp -R 0 /opt/uv \
|
||||
&& chmod -R g+rwX,a+rX /opt/uv
|
||||
@@ -1004,7 +1005,7 @@ RUN --mount=type=cache,target=/opt/uv/cache \
|
||||
# Install FlashInfer JIT cache (requires CUDA-version-specific index URL)
|
||||
# https://docs.flashinfer.ai/installation.html
|
||||
# From versions.json: .flashinfer.version
|
||||
ARG FLASHINFER_VERSION=0.7.0
|
||||
ARG FLASHINFER_VERSION=0.7.0.post1
|
||||
RUN --mount=type=cache,target=/opt/uv/cache \
|
||||
if [ "${INSTALL_RUBIN_PRERELEASE}" != "true" ]; then \
|
||||
uv pip install --system flashinfer-jit-cache==${FLASHINFER_VERSION} \
|
||||
|
||||
@@ -138,6 +138,8 @@ ENV TARGETARCH=${TARGETARCH}
|
||||
######################### COMMON BASE IMAGE + PYTHON DEPS #########################
|
||||
FROM base-common AS base-common-pydeps
|
||||
|
||||
ENV UV_COMPILE_BYTECODE=1
|
||||
|
||||
# PyTorch provides its own indexes for standard and nightly builds.
|
||||
# PYTORCH_NIGHTLY=1 builds the CPU image against torch nightly (parity with the
|
||||
# GPU torch-nightly lane); default 0 keeps the pinned release CPU build.
|
||||
|
||||
@@ -22,6 +22,7 @@ ENV HOME=/root \
|
||||
RUSTUP_HOME=/root/.rustup \
|
||||
UV_CACHE_DIR=$HOME/.cache/uv \
|
||||
PATH=/root/.cargo/bin:/root/.rustup/bin:${VIRTUAL_ENV}/bin:$PATH
|
||||
ENV UV_COMPILE_BYTECODE=1
|
||||
|
||||
RUN echo "DEBUG: VLLM_VERSION=${VLLM_VERSION}"
|
||||
RUN --mount=type=cache,target=/var/cache/dnf \
|
||||
|
||||
@@ -898,6 +898,7 @@ COPY --from=test_smoke /vllm-smoke-ok /
|
||||
# -----------------------
|
||||
# Final vLLM image
|
||||
FROM mori_base AS final_common
|
||||
ENV UV_COMPILE_BYTECODE=1
|
||||
|
||||
RUN python3 -m pip install --upgrade pip && rm -rf /var/lib/apt/lists/*
|
||||
|
||||
|
||||
@@ -3,14 +3,14 @@ ARG PYTHON_VERSION=3.12
|
||||
ARG TORCH_VERSION=2.12.0+rocm10.0.0
|
||||
ARG TORCHVISION_VERSION=0.27.0+rocm10.0.0
|
||||
ARG TORCHAUDIO_VERSION=2.11.0+rocm10.0.0
|
||||
ARG TRITON_BRANCH="c0a142ffcbb42f9a67a56bf791cad472be1d6425" # release/internal/3.8.x as of 9/21/2026
|
||||
ARG TRITON_VERSION=3.8.0+gitc0a142ff
|
||||
ARG TRITON_BRANCH="669b31ac" # release/internal/3.8.x as of 09/29/2026
|
||||
ARG TRITON_VERSION=3.8.0+git669b31ac
|
||||
ARG TRITON_REPO="https://github.com/ROCm/triton.git"
|
||||
ARG FA_BRANCH="0e60e394"
|
||||
ARG FA_REPO="https://github.com/Dao-AILab/flash-attention.git"
|
||||
ARG AITER_BRANCH="v0.1.23"
|
||||
ARG AITER_REPO="https://github.com/ROCm/aiter.git"
|
||||
ARG MORI_BRANCH="f95cd5e0ca7b730f92f9ffdfb358484d6965d9f7" # TheRock compatible v1.1.0
|
||||
ARG MORI_BRANCH="v1.2.3.post1"
|
||||
ARG MORI_REPO="https://github.com/ROCm/mori.git"
|
||||
ARG ROCM_SYSTEMS_REPO="https://github.com/ROCm/rocm-systems.git"
|
||||
ARG ROCM_RUNTIME_COMMIT="25e14349f2606a6b95c4ccd45eb92a1835646cf2"
|
||||
@@ -207,8 +207,9 @@ RUN cd mori \
|
||||
&& git checkout ${MORI_BRANCH} \
|
||||
&& git submodule update --init --recursive \
|
||||
&& ln -sfn ${ROCM_PATH} /opt/rocm \
|
||||
&& mkdir -p /usr/lib64 && ln -sf /usr/lib/x86_64-linux-gnu/libc.so /usr/lib64/libc.so \
|
||||
&& python3 setup.py bdist_wheel --dist-dir=dist && ls /app/mori/dist/*.whl
|
||||
&& pip install "setuptools_scm>=6.2" \
|
||||
&& CMAKE_PREFIX_PATH="${ROCM_PATH}:${ROCM_PATH}/lib/rocm_sysdeps" \
|
||||
python3 setup.py bdist_wheel --dist-dir=dist && ls /app/mori/dist/*.whl
|
||||
RUN mkdir -p /app/install && cp /app/mori/dist/*.whl /app/install
|
||||
|
||||
###
|
||||
|
||||
@@ -920,6 +920,7 @@ COPY --from=test_smoke /vllm-smoke-ok /
|
||||
# -----------------------
|
||||
# Final vLLM image
|
||||
FROM mori_base AS final_common
|
||||
ENV UV_COMPILE_BYTECODE=1
|
||||
|
||||
RUN python3 -m pip install --upgrade pip && rm -rf /var/lib/apt/lists/*
|
||||
|
||||
|
||||
@@ -11,7 +11,7 @@ ARG FA_BRANCH="0e60e394"
|
||||
ARG FA_REPO="https://github.com/Dao-AILab/flash-attention.git"
|
||||
ARG AITER_BRANCH="v0.1.23"
|
||||
ARG AITER_REPO="https://github.com/ROCm/aiter.git"
|
||||
ARG MORI_BRANCH="v1.1.0"
|
||||
ARG MORI_BRANCH="v1.2.3.post1"
|
||||
ARG MORI_REPO="https://github.com/ROCm/mori.git"
|
||||
ARG ROCM_SYSTEMS_REPO="https://github.com/ROCm/rocm-systems.git"
|
||||
ARG ROCPROFILER_SDK_COMMIT="2b22ab0195cc1461cd9abf3b969e9dd7c10af350"
|
||||
|
||||
@@ -655,6 +655,7 @@ RUN mkdir src && mv vllm src/vllm
|
||||
# -----------------------
|
||||
# Final vLLM image
|
||||
FROM mori_base AS final
|
||||
ENV UV_COMPILE_BYTECODE=1
|
||||
|
||||
RUN python3 -m pip install --upgrade pip && rm -rf /var/lib/apt/lists/*
|
||||
|
||||
|
||||
@@ -196,6 +196,7 @@ ARG PIP_EXTRA_INDEX_URL="https://download.pytorch.org/whl/cpu"
|
||||
ENV LD_LIBRARY_PATH="/opt/vllm/lib64/python${PYTHON_VERSION}/site-packages/torch/lib:/usr/local/lib:/opt/rh/gcc-toolset-14/root/usr/lib64:$LD_LIBRARY_PATH"
|
||||
ENV C_INCLUDE_PATH="/usr/local/include:$C_INCLUDE_PATH"
|
||||
ENV UV_LINK_MODE=copy
|
||||
ENV UV_COMPILE_BYTECODE=1
|
||||
ENV CARGO_HOME=/root/.cargo
|
||||
ENV RUSTUP_HOME=/root/.rustup
|
||||
ENV GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1
|
||||
|
||||
@@ -205,6 +205,7 @@ RUN --mount=type=cache,target=/root/.cache/uv \
|
||||
/tmp/ucx_source /tmp/nixl_source
|
||||
|
||||
FROM vllm-base AS vllm-openai
|
||||
ENV UV_COMPILE_BYTECODE=1
|
||||
|
||||
# Must match the ucx-nixl-build stage above: ARG does not cross stages, so a stale
|
||||
# value here installs a meta package that disagrees with the wheel built there.
|
||||
|
||||
@@ -71,7 +71,7 @@
|
||||
"default": "true"
|
||||
},
|
||||
"FLASHINFER_VERSION": {
|
||||
"default": "0.7.0"
|
||||
"default": "0.7.0.post1"
|
||||
},
|
||||
"GDRCOPY_CUDA_VERSION": {
|
||||
"default": "12.8"
|
||||
|
||||
@@ -88,6 +88,12 @@ vllm bench sweep serve \
|
||||
|
||||
By default, each parameter combination is benchmarked 3 times to make the results more reliable. You can adjust the number of runs by setting `--num-runs`.
|
||||
|
||||
You can optionally run an unmeasured warmup before each parameter combination by setting `--warmup-num-prompts`. The warmup result is saved as `warmup.json` for inspection, but is excluded from the measured summary.
|
||||
|
||||
You can use `--continue-on-error` to record a failed warmup, benchmark run, or server configuration and continue processing the remaining sweep combinations. Failure details are saved in `.failure.json` files.
|
||||
|
||||
Both options are also supported by `vllm bench sweep serve_workload`.
|
||||
|
||||
!!! important
|
||||
If both `--serve-params` and `--bench-params` are passed, the script will iterate over the Cartesian product between them.
|
||||
You can use `--dry-run` to preview the commands to be run.
|
||||
|
||||
@@ -135,6 +135,9 @@ API_AUTONAV_EXCLUDE=vllm mkdocs serve # API ref off (~15 seconds)
|
||||
Once you see a `Serving on http://<address>:<port>/` line in the logs, the live preview is ready!
|
||||
Open that address in your browser to see it — `http://127.0.0.1:8000/` by default, or whichever host/port you passed to `-a`.
|
||||
|
||||
On pull requests, Read the Docs builds a documentation preview only if the PR changes files that affect the docs (e.g. `docs/`, `examples/` or `mkdocs.yaml`) and the `pre-commit` checks have not failed.
|
||||
Changes to `vllm/` alone do not trigger a build, so if your PR changes the API or CLI reference, add the `build-docs` label to force one.
|
||||
|
||||
For additional features and advanced configurations, refer to the:
|
||||
|
||||
- [MkDocs documentation](https://www.mkdocs.org/)
|
||||
|
||||
@@ -85,6 +85,8 @@ def forward(
|
||||
|
||||
For reference, check out our [Llama implementation](../../../vllm/model_executor/models/llama.py). vLLM already supports a large number of models. It is recommended to find a model similar to yours and adapt it to your model's architecture. Check out [vllm/model_executor/models](../../../vllm/model_executor/models) for more examples.
|
||||
|
||||
Adapting model-specific structure or boilerplate can keep architectures independent. Reuse shared layer implementations, such as Attention and MoE, where their contracts fit the model.
|
||||
|
||||
## 3. (Optional) Implement tensor parallelism and quantization support
|
||||
|
||||
If your model is too large to fit into a single GPU, you can use tensor parallelism to manage it.
|
||||
|
||||
@@ -130,10 +130,13 @@ path. The resolver consumes the existing graph-stable request mapping from
|
||||
attention metadata; neither the worker nor individual cache handles keep a
|
||||
duplicate mapping.
|
||||
|
||||
Speculative decoding resolves and consumes each verification step in order.
|
||||
Each step receives distinct replayable plan rows while sharing the request's
|
||||
hot-cache state, so a later step cannot reuse a hot row before an earlier step
|
||||
has consumed it.
|
||||
Speculative decoding resolves all verification rows of a request in one pass:
|
||||
one block resolves the union of the rows' top-k against the request's hot-cache
|
||||
state, so rows that select the same host row share its hot row and no row
|
||||
evicts a hot row another row of the step still reads. Draft layers write their
|
||||
rows after the target forward, so their host mirror runs at the start of the
|
||||
next step, after the drafter, and the step's page transfers are submitted
|
||||
behind it.
|
||||
|
||||
## P/D import target
|
||||
|
||||
|
||||
@@ -78,17 +78,23 @@ GPU-side fusion is controlled by `multimodal_config.mm_device_do_normalize`:
|
||||
- The flag is **enabled by default** for all models that support it.
|
||||
- Currently, it’s on by default for these architectures:
|
||||
|
||||
| name | Architecture | Example HF Models |
|
||||
|--------------------|------------------------------------------|----------------------------------------|
|
||||
| `intern-s1-pro` | `InternS1ProForConditionalGeneration` | `internlm/Intern-S1-Pro` |
|
||||
| `intern-s2-mobius` | `InternS2MobiusForConditionalGeneration` | `internlm/Intern-S2-Mobius` |
|
||||
| `qwen2-vl` | `Qwen2VLForConditionalGeneration` | `Qwen/Qwen2-VL-2B-Instruct`, etc. |
|
||||
| `qwen2.5-vl` | `Qwen2_5_VLForConditionalGeneration` | `Qwen/Qwen2.5-VL-3B-Instruct`, etc. |
|
||||
| `qwen3-vl` | `Qwen3VLForConditionalGeneration` | `Qwen/Qwen3-VL-4B-Instruct`, etc. |
|
||||
| `qwen3-vl-moe` | `Qwen3VLMoeForConditionalGeneration` | `Qwen/Qwen3-VL-30B-A3B-Instruct`, etc. |
|
||||
| `qwen3.5` | `Qwen3_5ForConditionalGeneration` | `Qwen/Qwen3.5-0.8B`, etc. |
|
||||
| `qwen3.5-moe` | `Qwen3_5MoeForConditionalGeneration` | `Qwen/Qwen3.5-35B-A3B`, etc. |
|
||||
| `qwen4-exp` | `Qwen4ExpForConditionalGeneration` | `Qwen/Qwen3.8-Flash-Next` |
|
||||
| name | Architecture | Example HF Models |
|
||||
| -------------------------- | ------------------------------------------ | ----------------------------------------------------- |
|
||||
| `glm-4v` | `Glm4vForConditionalGeneration` | `zai-org/GLM-4.1V-9B-Thinking`, etc. |
|
||||
| `glm-5-next` | `Glm5NextForConditionalGeneration` | `zai-org/GLM-5.3-Flash` |
|
||||
| `glm-ocr` | `GlmOcrForConditionalGeneration` | `zai-org/GLM-OCR` |
|
||||
| `intern-s1-pro` | `InternS1ProForConditionalGeneration` | `internlm/Intern-S1-Pro` |
|
||||
| `intern-s2-mobius` | `InternS2MobiusForConditionalGeneration` | `internlm/Intern-S2-Mobius` |
|
||||
| `llama-nemotron-vl-embed` | `LlamaNemotronVLForEmbedding` | `nvidia/llama-nemotron-embed-vl-1b-v2` |
|
||||
| `llama-nemotron-vl-rerank` | `LlamaNemotronVLForSequenceClassification` | `nvidia/llama-nemotron-rerank-vl-1b-v2` |
|
||||
| `mistral3` | `Mistral3ForConditionalGeneration` | `mistralai/Mistral-Small-3.2-24B-Instruct-2506`, etc. |
|
||||
| `qwen2-vl` | `Qwen2VLForConditionalGeneration` | `Qwen/Qwen2-VL-2B-Instruct`, etc. |
|
||||
| `qwen2.5-vl` | `Qwen2_5_VLForConditionalGeneration` | `Qwen/Qwen2.5-VL-3B-Instruct`, etc. |
|
||||
| `qwen3-vl` | `Qwen3VLForConditionalGeneration` | `Qwen/Qwen3-VL-4B-Instruct`, etc. |
|
||||
| `qwen3-vl-moe` | `Qwen3VLMoeForConditionalGeneration` | `Qwen/Qwen3-VL-30B-A3B-Instruct`, etc. |
|
||||
| `qwen3.5` | `Qwen3_5ForConditionalGeneration` | `Qwen/Qwen3.5-0.8B`, etc. |
|
||||
| `qwen3.5-moe` | `Qwen3_5MoeForConditionalGeneration` | `Qwen/Qwen3.5-35B-A3B`, etc. |
|
||||
| `qwen4-exp` | `Qwen4ExpForConditionalGeneration` | `Qwen/Qwen3.8-Flash-Next` |
|
||||
|
||||
#### Key Properties and Gains
|
||||
|
||||
|
||||
@@ -127,7 +127,7 @@ Batch invariance has been tested and verified on the following models:
|
||||
- **Qwen3-VL (Vision-Language)**: `Qwen/Qwen3-VL-2B-Instruct`, `Qwen/Qwen3-VL-4B-Instruct` (single image and video inputs)
|
||||
- **Qwen3 (MoE)**: `Qwen/Qwen3-30B-A3B`, `Qwen/Qwen3-Next-80B-A3B-Instruct`, `Qwen/Qwen3-30B-A3B-Thinking-2507-FP8`
|
||||
- **Qwen2.5**: `Qwen/Qwen2.5-0.5B-Instruct`, `Qwen/Qwen2.5-1.5B-Instruct`, `Qwen/Qwen2.5-3B-Instruct`, `Qwen/Qwen2.5-7B-Instruct`, `Qwen/Qwen2.5-14B-Instruct`, `Qwen/Qwen2.5-32B-Instruct`
|
||||
- **Llama 3**: Llama3.1 and 3.2 series, `meta-llama/Llama-3.2-3B-Instruct` for example
|
||||
- **Llama 3**: Llama3.1 and 3.2 series, `meta-llama/Llama-3.2-1B-Instruct`, `meta-llama/Llama-3.2-3B-Instruct` for example
|
||||
- **GPT-OSS**: `openai/gpt-oss-20b`, `openai/gpt-oss-120b`
|
||||
- **Mistral**: `mistralai/Mistral-7B-v0.3`
|
||||
- **Phi series**: `microsoft/Phi-3.5-mini-instruct`
|
||||
@@ -135,6 +135,8 @@ Batch invariance has been tested and verified on the following models:
|
||||
- **Granite 3.1 (Dense)**: `ibm-granite/granite-3.1-2b-instruct`, `ibm-granite/granite-3.1-8b-instruct`
|
||||
- **EXAONE 4.0 series**: `LGAI-EXAONE/EXAONE-4.0-1.2B`, `LGAI-EXAONE/EXAONE-4.0.1-32B`, `LGAI-EXAONE/EXAONE-4.0-32B`
|
||||
- **OLMo 2**: `allenai/OLMo-2-0425-1B-Instruct`
|
||||
- **SmolLM2**: `HuggingFaceTB/SmolLM2-1.7B-Instruct`
|
||||
- **PLaMo3**: `pfnet/plamo-3-nict-2b-base`
|
||||
|
||||
Other models may also work, but these have been explicitly validated. If you encounter issues with a specific model, please report them on the [GitHub issue tracker](https://github.com/vllm-project/vllm/issues/new/choose).
|
||||
|
||||
@@ -144,11 +146,15 @@ When batch invariance is enabled, vLLM:
|
||||
|
||||
1. Uses deterministic kernel implementations for attention and other operations
|
||||
2. Ensures consistent numerical behavior across different batch sizes
|
||||
3. Disables certain optimizations that may introduce non-determinism (such as custom all-reduce operations in tensor parallel mode, and sequence parallelism / async TP, whose reduce-scatter path is not batch-invariant)
|
||||
4. On CUDA devices with tuned matmul table entries for the model's bf16 unquantized forward linear layers (Ada, Hopper, Blackwell),
|
||||
3. Disables certain optimizations that may introduce non-determinism (such as sequence parallelism / async TP, whose reduce-scatter path is not batch-invariant)
|
||||
4. Under tensor parallelism, keeps custom all-reduce on with a fixed reduction order (the 1-stage kernel is pinned, and large inputs are reduced in fixed-size chunks), and disables FlashInfer, AITER and QuickReduce all-reduce
|
||||
5. On CUDA devices with tuned matmul table entries for the model's bf16 unquantized forward linear layers (Ada, Hopper, Blackwell),
|
||||
runs without `torch.compile` using breakable CUDA graphs so tile configs follow the runtime batch size; set `VLLM_USE_BREAKABLE_CUDAGRAPH=0` to opt out
|
||||
(not applied when sequence parallelism / async TP are enabled, since those are `torch.compile` passes).
|
||||
|
||||
!!! warning
|
||||
Batch invariance under tensor parallelism is not yet supported for all-reduces whose size isn't a multiple of 16 bytes, for example a hidden size that isn't a multiple of 8 in fp16/bf16. Such tensors can switch between NCCL and custom all-reduce depending on batch size. All validated models meet this requirement.
|
||||
|
||||
!!! note
|
||||
Enabling batch invariance may impact performance compared to the default non-deterministic mode. This trade-off is intentional to guarantee reproducibility.
|
||||
|
||||
|
||||
@@ -45,9 +45,10 @@ The following scripts demonstrate the workflow with **ExampleConnector**:
|
||||
### ECMooncakeConnector
|
||||
|
||||
**ECMooncakeConnector** transfers encoder outputs using the Mooncake TransferEngine.
|
||||
See the [Mooncake integration example](../../tests/v1/ec_connector/integration/run_epd_mooncake_ec_full_pipeline.sh)
|
||||
for a complete 1 Encoder instance + 1 PD instance setup, including the producer and
|
||||
consumer `--ec-transfer-config` settings and proxy configuration.
|
||||
See the [ECMooncakeConnector Usage Guide](mooncake_ec_connector_usage.md) for
|
||||
producer/consumer `--ec-transfer-config`, proxy `--ec-consumer-zmq-addrs`, and
|
||||
transport options. The [Mooncake integration example](../../tests/v1/ec_connector/integration/run_epd_mooncake_ec_full_pipeline.sh)
|
||||
is a complete 1 Encoder instance + 1 PD instance setup.
|
||||
|
||||
With vLLM and Mooncake installed, run the example from the repository root:
|
||||
|
||||
|
||||
@@ -27,7 +27,7 @@ Now supports 9 types of connectors:
|
||||
--kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_both", "kv_buffer_device":"cuda", "kv_connector_extra_config":{"backends":["UCX", "GDS"]}}'
|
||||
```
|
||||
|
||||
- **MooncakeConnector**: refer to [examples/disaggregated/mooncake_connector/run_mooncake_connector.sh](../../examples/disaggregated/mooncake_connector/run_mooncake_connector.sh) for the example usage of MooncakeConnector disaggregated prefilling. For detailed usage guide, see [MooncakeConnector Usage Guide](mooncake_connector_usage.md).
|
||||
- **MooncakeConnector**: refer to [examples/disaggregated/mooncake_connector/run_mooncake_connector.sh](../../examples/disaggregated/mooncake_connector/run_mooncake_connector.sh) for the example usage of MooncakeConnector disaggregated prefilling. For detailed usage guide, see [MooncakeConnector Usage Guide](mooncake_connector_usage.md). For multimodal encoder-cache transfer, see [ECMooncakeConnector Usage Guide](mooncake_ec_connector_usage.md).
|
||||
- **MoRIIOConnector** (ROCm only): see [MoRI-IO Usage Guide](moriio_connector_usage.md) for example usage and detailed documentation.
|
||||
- **MultiConnector**: take advantage of the kv_connector_extra_config: dict[str, Any] already present in KVTransferConfig to stash all the connectors we want in an ordered list of kwargs.such as:
|
||||
|
||||
|
||||
@@ -42,10 +42,10 @@ Snapshots currently require:
|
||||
and CRIU still reopens it by path, so replacing that file permanently
|
||||
invalidates the artifact without an early error.
|
||||
|
||||
The official `vllm/vllm-openai` Linux x86-64 image includes the snapshot
|
||||
runtime. It still requires a compatible host driver, kernel, and privileges.
|
||||
Arm64 images omit it. Source installs must set `CRIU_CUDA_PLUGIN_DIR` to the
|
||||
directory containing `cuda_plugin.so`.
|
||||
The official CUDA 13 `vllm/vllm-openai` Linux x86-64 images include the
|
||||
snapshot runtime. CUDA 12.x and Arm64 images omit it. A compatible host
|
||||
driver, kernel, and privileges are still required. Source installs must set
|
||||
`CRIU_CUDA_PLUGIN_DIR` to the directory containing `cuda_plugin.so`.
|
||||
|
||||
Run snapshot commands with `docker exec` inside a long-lived container. Restore
|
||||
hands the API server off as a detached process, so a one-shot container would
|
||||
|
||||
@@ -38,6 +38,9 @@ python examples/disaggregated/mooncake_connector/mooncake_connector_proxy.py --p
|
||||
|
||||
Now you can send requests to the proxy server through port 8000.
|
||||
|
||||
For multimodal Encode-Prefill-Decode (encoder embeddings rather than KV
|
||||
cache), see [ECMooncakeConnector Usage Guide](mooncake_ec_connector_usage.md).
|
||||
|
||||
## Environment Variables
|
||||
|
||||
- `VLLM_MOONCAKE_BOOTSTRAP_PORT`: Port for Mooncake bootstrap server
|
||||
@@ -76,6 +79,7 @@ Now you can send requests to the proxy server through port 8000.
|
||||
- **num_workers**: Size of thread pool for one prefiller worker to transfer KV caches by mooncake. (default 10)
|
||||
- **mooncake_protocol**: Mooncake connector protocol. (default "rdma")
|
||||
- **device_name**: Comma-separated whitelist of RDMA devices (e.g. `"mlx5_0,mlx5_1"`) to restrict topology discovery to. Empty discovers every device. Useful on hosts exposing a mix of InfiniBand and RoCE ports, where both peers must settle on the same link layer.
|
||||
- **bootstrap_server_address**: Optional `"host:port"` of an already running Mooncake bootstrap server. All prefiller ranks register with this endpoint instead of launching an embedded server. The caller owns the server lifetime and must configure the proxy to use the same endpoint. The example proxy assumes the bootstrap server is on the prefiller API host; a different host requires a proxy that supplies that address in `remote_bootstrap_addr`. Omit this option to use the embedded server and `VLLM_MOONCAKE_BOOTSTRAP_PORT`.
|
||||
|
||||
## Example Scripts/Code
|
||||
|
||||
|
||||
@@ -0,0 +1,158 @@
|
||||
# ECMooncakeConnector Usage Guide
|
||||
|
||||
`ECMooncakeConnector` moves multimodal **encoder embeddings** (encoder cache)
|
||||
between a disaggregated encoder instance and a language-model instance over
|
||||
[Mooncake Transfer Engine](https://github.com/kvcache-ai/Mooncake). Use it
|
||||
when the vision encoder should scale independently of prefill and decode.
|
||||
|
||||
For KV-cache transfer between prefiller and decoder, see
|
||||
[MooncakeConnector Usage Guide](mooncake_connector_usage.md). That path uses
|
||||
`MooncakeConnector`, not `ECMooncakeConnector`.
|
||||
|
||||
Added in [PR #41567](https://github.com/vllm-project/vllm/pull/41567).
|
||||
Overview of disaggregated encoding:
|
||||
[Disaggregated Encoder](disagg_encoder.md).
|
||||
|
||||
!!! important
|
||||
Static proxy flags (`--ec-consumer-zmq-addrs`) support Mooncake EC only in
|
||||
**1E + 1PD** mode (`--prefill-servers-urls disable`). For 1E + 1P + 1D,
|
||||
register `ec_zmq_addrs` on the **prefill** consumer through the proxy's
|
||||
dynamic registration API instead of the static CLI flag. See
|
||||
`examples/disaggregated/disaggregated_encoder/README.md`.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Install Mooncake Transfer Engine:
|
||||
|
||||
```bash
|
||||
# CUDA 13 (vLLM default)
|
||||
uv pip install mooncake-transfer-engine-cuda13
|
||||
|
||||
# CUDA 12
|
||||
uv pip install mooncake-transfer-engine
|
||||
```
|
||||
|
||||
The two wheels are the same release built against different CUDA majors.
|
||||
Importing the wrong one fails with
|
||||
`libcudart.so.<major>: cannot open shared object file`.
|
||||
|
||||
## 1E + 1PD
|
||||
|
||||
One encoder producer and one combined prefill/decode consumer. The consumer
|
||||
listens on `ec_ip` + `ec_port` (ZMQ control). The proxy must advertise that
|
||||
address with `--ec-consumer-zmq-addrs`.
|
||||
|
||||
### Encoder (producer)
|
||||
|
||||
The producer must run with `tensor_parallel_size=1`,
|
||||
`pipeline_parallel_size=1`, and `data_parallel_size=1`.
|
||||
|
||||
```bash
|
||||
CUDA_VISIBLE_DEVICES=0 vllm serve Qwen/Qwen2.5-VL-3B-Instruct \
|
||||
--port 19534 \
|
||||
--gpu-memory-utilization 0.35 \
|
||||
--enable-request-id-headers \
|
||||
--no-enable-prefix-caching \
|
||||
--max-num-batched-tokens 16384 \
|
||||
--allowed-local-media-path /path/to/media \
|
||||
--ec-transfer-config '{
|
||||
"ec_connector": "ECMooncakeConnector",
|
||||
"ec_role": "ec_producer",
|
||||
"ec_connector_extra_config": {
|
||||
"mooncake_protocol": "rdma"
|
||||
}
|
||||
}'
|
||||
```
|
||||
|
||||
### Prefill+Decode (consumer)
|
||||
|
||||
```bash
|
||||
CUDA_VISIBLE_DEVICES=1 vllm serve Qwen/Qwen2.5-VL-3B-Instruct \
|
||||
--port 19537 \
|
||||
--gpu-memory-utilization 0.75 \
|
||||
--enable-mm-embeds \
|
||||
--enable-request-id-headers \
|
||||
--no-enable-prefix-caching \
|
||||
--allowed-local-media-path /path/to/media \
|
||||
--ec-transfer-config '{
|
||||
"ec_connector": "ECMooncakeConnector",
|
||||
"ec_role": "ec_consumer",
|
||||
"ec_ip": "127.0.0.1",
|
||||
"ec_port": 19019,
|
||||
"ec_connector_extra_config": {
|
||||
"mooncake_protocol": "rdma"
|
||||
}
|
||||
}'
|
||||
```
|
||||
|
||||
`--enable-mm-embeds` lets the PD instance inject transferred embeddings
|
||||
instead of re-running the vision encoder.
|
||||
|
||||
### Proxy
|
||||
|
||||
```bash
|
||||
python examples/disaggregated/disaggregated_encoder/disagg_epd_proxy.py \
|
||||
--host 0.0.0.0 \
|
||||
--port 10002 \
|
||||
--encode-servers-urls http://127.0.0.1:19534 \
|
||||
--prefill-servers-urls disable \
|
||||
--decode-servers-urls http://127.0.0.1:19537 \
|
||||
--ec-consumer-zmq-addrs tcp://127.0.0.1:19019
|
||||
```
|
||||
|
||||
`--ec-consumer-zmq-addrs` must align with `--decode-servers-urls` (the
|
||||
combined PD instances). For data-parallel consumers, list each replica
|
||||
consecutively and set `--ec-consumer-dp-size`.
|
||||
|
||||
### Correctness script
|
||||
|
||||
```bash
|
||||
# TCP (no RDMA device required; CI default)
|
||||
MOONCAKE_EC_PROTOCOL=tcp \
|
||||
bash tests/v1/ec_connector/integration/run_epd_mooncake_ec_full_pipeline.sh
|
||||
|
||||
# RDMA
|
||||
MOONCAKE_EC_PROTOCOL=rdma \
|
||||
bash tests/v1/ec_connector/integration/run_epd_mooncake_ec_full_pipeline.sh
|
||||
```
|
||||
|
||||
The TCP path sets `MC_FORCE_TCP=1` so Transfer Engine does not auto-select
|
||||
RDMA when an HCA is present.
|
||||
|
||||
## Configuration
|
||||
|
||||
Producer and consumer must use the same `mooncake_protocol`.
|
||||
|
||||
| Setting | Where | Default | Notes |
|
||||
| --------- | ------- | --------- | ------- |
|
||||
| `mooncake_protocol` | `ec_connector_extra_config` | `rdma` | Set `tcp` when no verbs device is available. With TCP, also export `MC_FORCE_TCP=1`. |
|
||||
| `ec_ip` / `ec_port` | consumer `ECTransferConfig` | `127.0.0.1` / `14579` | First ZMQ control port. Tensor-parallel rank `r` listens on `ec_port + dp_index * tp_size + r`. |
|
||||
| `ec_buffer_size` | `ECTransferConfig` | `1e9` | Registered receive arena on the consumer, in bytes. |
|
||||
| `control_timeout_s` | `ec_connector_extra_config` | `30` | Reservation RPC timeout. |
|
||||
| `push_wait_timeout_s` | `ec_connector_extra_config` | `60` | How long the consumer waits for a matching encoder push. |
|
||||
| `WITH_NVIDIA_PEERMEM` | process environment | `1` | Read by Mooncake. Set `0` on hosts without `nvidia-peermem` (DMA-BUF path). |
|
||||
| `--ec-consumer-zmq-addrs` | proxy | empty | `tcp://<ec_ip>:<ec_port>` for each PD replica. Required for Mooncake EC. |
|
||||
|
||||
## 1E + 1P + 1D
|
||||
|
||||
The filesystem example is `examples/disaggregated/disaggregated_encoder/disagg_1e1p1d_example.sh`
|
||||
(`ECExampleConnector` for encoder cache, `NixlConnector` for KV). P→D KV can
|
||||
use `MooncakeConnector` instead; see
|
||||
[MooncakeConnector Usage Guide](mooncake_connector_usage.md).
|
||||
|
||||
Do **not** pass `--ec-consumer-zmq-addrs` together with independent prefill
|
||||
URLs: the static proxy flag is E+PD only. For Mooncake EC with a dedicated
|
||||
prefill, use dynamic registration and set `ec_zmq_addrs` on the prefill
|
||||
consumer as described in
|
||||
`examples/disaggregated/disaggregated_encoder/README.md`.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Symptom | Likely cause |
|
||||
| --------- | ---------------- |
|
||||
| `Install mooncake-transfer-engine` / `libcudart.so.*` | Wrong or missing Mooncake wheel for this CUDA major. |
|
||||
| `Failed to register memory ... Bad address [14]` | GPUDirect registration failed. Set `WITH_NVIDIA_PEERMEM=0`. |
|
||||
| Proxy startup error about E+PD only | `--ec-consumer-zmq-addrs` used with a real `--prefill-servers-urls`. |
|
||||
| Consumer waits then errors; encoder never finishes the push | `--ec-consumer-zmq-addrs` does not match consumer `ec_ip`/`ec_port`, or producer/consumer protocols differ. |
|
||||
| TCP transfers stall on a host that also has RDMA | Export `MC_FORCE_TCP=1`. |
|
||||
| Producer rejected at startup | Encoder used tensor, pipeline, or data parallelism. Keep the producer unsharded. |
|
||||
@@ -1210,7 +1210,7 @@ vLLM decodes audio bytes into waveforms using a selectable decoding backend:
|
||||
|
||||
| Backend | Description |
|
||||
| --- | --- |
|
||||
| `auto` (default) | soundfile, falling back to torchcodec, then PyAV |
|
||||
| `auto` (default) | torchcodec, falling back to soundfile, then PyAV |
|
||||
| `soundfile` | libsndfile only, no fallback |
|
||||
| `pyav` | PyAV (FFmpeg) only, no fallback |
|
||||
| `torchcodec` | TorchCodec (PyTorch-native) only, no fallback |
|
||||
@@ -1226,17 +1226,18 @@ vllm serve mistralai/Voxtral-Mini-3B-2507 \
|
||||
`pyav` drives FFmpeg through a per-frame Python generator, so under
|
||||
concurrency the Python/C crossings contend on the GIL. `torchcodec`
|
||||
decodes each stream in a single call that releases the GIL for its whole
|
||||
duration. Select it explicitly for concurrent decoding workloads.
|
||||
`auto` prefers soundfile for supported formats to preserve their existing
|
||||
decoding behavior, including encoder padding. Audio extracted from formats
|
||||
that soundfile cannot read, such as video containers, can use torchcodec
|
||||
through the fallback chain.
|
||||
duration, which is why `auto` prefers it when many requests decode audio
|
||||
concurrently, such as when audio tracks are extracted from video.
|
||||
vLLM normalizes codec padding on this path (e.g. it trims trailing Vorbis
|
||||
padding that older system FFmpeg versions (< 5.0) fail to trim), so the
|
||||
decoded waveform matches the soundfile reference length.
|
||||
|
||||
!!! note
|
||||
`torchcodec` ships as a requirement on CUDA, CPU and XPU builds. On other
|
||||
platforms (e.g. ROCm, TPU), install it manually to enable that backend.
|
||||
torchcodec also links against a system FFmpeg installation. If the package
|
||||
or FFmpeg is unavailable, `auto` uses the soundfile → PyAV chain.
|
||||
platforms (e.g. ROCm, TPU) `auto` falls back to the soundfile → PyAV
|
||||
chain unless you install it manually. torchcodec also links against a
|
||||
system FFmpeg installation; if FFmpeg is missing, `auto` falls back the
|
||||
same way.
|
||||
|
||||
### Embedding Inputs
|
||||
|
||||
|
||||
@@ -454,6 +454,7 @@ exported when NixlConnector is active:
|
||||
| `vllm:nixl_num_failed_transfers` | Counter | Cumulative count of failed NIXL KV-block transfers, including handshake and notification (`send_notif`) failures. These are grouped because all are sporadic lower-transport-layer events. |
|
||||
| `vllm:nixl_num_failed_notifications` | Counter | Cumulative count of failed notifications (`send_notif`), retained for compatibility. These failures are also included in `vllm:nixl_num_failed_transfers`; do not sum the two counters. |
|
||||
| `vllm:nixl_num_kv_expired_reqs` | Counter | Requests whose KV blocks expired on the prefiller before the decoder read them (tracked on the P instance). Kept separate from the failure counters above: KV expiry is an autoscaler/lease-tuning signal, not a transport issue. |
|
||||
| `vllm:nixl_num_notifications_after_expiry` | Counter | Completion notifications that arrive for a request that is no longer tracked, usually because its KV lease expired first. Pull connector: counted on P, where it means the decoder may have read blocks that were already reused; each matches a "Potentially invalid KV blocks" error log. Push connector: counted on whichever instance tracks the request as neither a send nor a receive (P after the lease expired, or D after the receive completed or was abandoned); each matches an "Unrecognized request" error log. Counted per notification, not per request: each remote rank notifies separately, so with TP > 1 or heterogeneous TP one late request counts several times, and the value is not comparable to `vllm:nixl_num_kv_expired_reqs`. |
|
||||
|
||||
!!! tip
|
||||
High `vllm:nixl_num_kv_expired_reqs` indicates that the prefiller's lease
|
||||
|
||||
@@ -29,6 +29,7 @@ vLLM currently supports the following reasoning models:
|
||||
| [MiniMax-M2](https://huggingface.co/MiniMaxAI/MiniMax-M2) | `minimax_m2_append_think` | `json`, `regex` | ✅ |
|
||||
| [Qwen3 series](https://huggingface.co/collections/Qwen/qwen3-67dd247413f0e2e4f653967f) | `qwen3` | `json`, `regex` | ✅ |
|
||||
| [QwQ-32B](https://huggingface.co/Qwen/QwQ-32B) | `deepseek_r1` | `json`, `regex` | ✅ |
|
||||
| [Step-3.5 / Step-3.7 Flash](https://huggingface.co/stepfun-ai/Step-3.5-Flash) | `step3p5` | `json`, `regex` | ✅ |
|
||||
|
||||
!!! note
|
||||
IBM Granite 3.2 and DeepSeek-V3.1 reasoning is disabled by default; to enable it, you must also pass `thinking=True` in your `chat_template_kwargs`.
|
||||
|
||||
@@ -171,6 +171,29 @@ from HuggingFace; and you can find an example of this in a `tokenizer_config.jso
|
||||
|
||||
If your favorite tool-calling model is not supported, please feel free to contribute a parser & tool use chat template!
|
||||
|
||||
### Checkpoint Response Templates (`hf`)
|
||||
|
||||
Checkpoints can define a `response_template` in `tokenizer_config.json` to describe their reasoning, content, and tool-call wire format. Select the
|
||||
`hf` parser to parse output from this metadata instead of a model-specific parser. Like `--tokenizer-mode hf`, it relies on the checkpoint's
|
||||
Hugging Face metadata rather than model-specific code:
|
||||
|
||||
```bash
|
||||
vllm serve <model> \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser hf \
|
||||
--reasoning-parser hf
|
||||
```
|
||||
|
||||
Either parser can be selected on its own. Startup fails if the tokenizer has no `response_template`, or if it lacks the `thinking` or `tool_calls`
|
||||
field the selected parser needs. The `hf` parser cannot be combined with other reasoning or tool call parsers.
|
||||
|
||||
When streaming, each tool call is emitted whole once its region parses. A call cut off before its closer, for example by a closer that is also a
|
||||
stop token, is parsed from the text generated so far; a call that does not parse is dropped.
|
||||
|
||||
Response templates currently provide parsing, not format-specific constrained decoding. The parser therefore rejects requests for strict tools,
|
||||
required or named tool choice, or `parallel_tool_calls=false`. A custom server or request chat template is allowed, but the parser still expects the
|
||||
checkpoint's output format.
|
||||
|
||||
!!! note
|
||||
With `tool_choice="auto"`, structural-tag constraints require both `VLLM_ENFORCE_STRICT_TOOL_CALLING=true` (the default) and at least one tool with `strict: true`, or a server-side floor set via `--tool-strict-level`. When these conditions are met and the selected parser supports structural tags, vLLM constrains the tool-call envelope and pins the argument schema of each tool that sets `strict: true` (or of every tool under `--tool-strict-level parameter`). Otherwise, vLLM extracts tool calls from raw text, so arguments may occasionally be malformed or violate the function's parameter schema.
|
||||
|
||||
@@ -458,6 +481,29 @@ Supported models:
|
||||
|
||||
Flags: `--tool-call-parser qwen3_xml`
|
||||
|
||||
### MiMo-V2.6 Models (`mimo`)
|
||||
|
||||
Supported models:
|
||||
|
||||
* `XiaomiMiMo/MiMo-V2.6-Pro-RL`
|
||||
* `XiaomiMiMo/MiMo-V2.6-Flash-RL`
|
||||
|
||||
Use the checkpoint's chat template with `--tool-call-parser mimo --reasoning-parser mimo --enable-auto-tool-choice`.
|
||||
For schema-constrained tool arguments, set `strict: true` inside each tool's `function`, or use `--tool-strict-level parameter`.
|
||||
The parser supports `auto`, `required`, named tools, and `parallel_tool_calls=False`, preserving leading/trailing whitespace in string arguments.
|
||||
|
||||
### Step-3.5 and Step-3.7 Models (`step3p5`)
|
||||
|
||||
Supported models:
|
||||
|
||||
* `stepfun-ai/Step-3.5-Flash`
|
||||
* `stepfun-ai/Step-3.5-Flash-FP8`
|
||||
* `stepfun-ai/Step-3.7-Flash`
|
||||
|
||||
Flags: `--tool-call-parser step3p5 --reasoning-parser step3p5 --enable-auto-tool-choice`
|
||||
|
||||
The parser supports `auto`, `required`, and named tools.
|
||||
|
||||
### Olmo 3 Models (`olmo3`)
|
||||
|
||||
Olmo 3 models output tool calls in a format that is very similar to the one expected by the `pythonic` parser (see below), with a few differences. Each tool call is a pythonic string, but the parallel tool calls are newline-delimited, and the calls are wrapped within XML tags as `<function_calls>..</function_calls>`. In addition, the parser also allows JSON boolean and null literals (`true`, `false`, and `null`) in addition to the pythonic ones (`True`, `False`, and `None`).
|
||||
|
||||
@@ -77,30 +77,17 @@ watermarked. The watermark signal is diluted in proportion to the share of
|
||||
output tokens supplied by accepted drafts; rejected drafts do not dilute it
|
||||
because their recovery tokens are watermarked.
|
||||
|
||||
Speculative-decoding token paths do not currently support generation-side
|
||||
context deduplication. The configured `deduplicate_contexts` policy is not
|
||||
applied to accepted drafts, rejection-recovery tokens, or bonus tokens.
|
||||
|
||||
For `dual_key_gumbel`, `alpha` has no effect under speculative decoding. The
|
||||
speculative protocol selects the key for each token instead.
|
||||
|
||||
## Algorithms
|
||||
|
||||
### Gumbel-max
|
||||
|
||||
Gumbel-max derives a deterministic pseudorandom value from the key, prior token
|
||||
context, and every candidate token, then uses the resulting Gumbel noise for
|
||||
categorical sampling. See
|
||||
[Aaronson's original presentation](https://simons.berkeley.edu/sites/default/files/2024-10/LLM24-2%20Slides%20-%20Scott%20Aaronson.pdf).
|
||||
|
||||
Gumbel-max requires stochastic sampling. Greedy requests (`temperature=0`)
|
||||
bypass watermarking and emit a warning once per worker.
|
||||
## Context deduplication
|
||||
|
||||
When a token context is repeated, generation can use ordinary sampling for that
|
||||
occurrence. Reusing the repeated context leads to a bias over the sequence, as
|
||||
certain token choices would be correlated. Ordinary sampling at these positions
|
||||
allows for single-sequence non-distortion (see section G.3 of the
|
||||
[SynthID-Text supplementary materials](https://media.springernature.com/original/springer-static/esm/art%3A10.1038%2Fs41586-024-08025-4/MediaObjects/41586_2024_8025_MOESM1_ESM.pdf)).
|
||||
This also applies to drafter watermarking during speculative decoding.
|
||||
The detector independently deduplicates contexts so repeated keyed random
|
||||
vectors are not treated as independent evidence, meaning that context
|
||||
deduplication at generation time does not reduce the watermarking signal, unless
|
||||
@@ -143,6 +130,18 @@ vllm serve MODEL \
|
||||
'{"algorithm":"gumbel","key":42,"deduplicate_contexts":"all"}'
|
||||
```
|
||||
|
||||
## Algorithms
|
||||
|
||||
### Gumbel-max
|
||||
|
||||
Gumbel-max derives a deterministic pseudorandom value from the key, prior token
|
||||
context, and every candidate token, then uses the resulting Gumbel noise for
|
||||
categorical sampling. See
|
||||
[Aaronson's original presentation](https://simons.berkeley.edu/sites/default/files/2024-10/LLM24-2%20Slides%20-%20Scott%20Aaronson.pdf).
|
||||
|
||||
Gumbel-max requires stochastic sampling. Greedy requests (`temperature=0`)
|
||||
bypass watermarking and emit a warning once per worker.
|
||||
|
||||
### Dual-key Gumbel-max
|
||||
|
||||
Dual-key Gumbel-max derives independent keys A and B from one configured master
|
||||
|
||||
@@ -312,8 +312,8 @@ vLLM CPU supports data parallel (DP), tensor parallel (TP) and pipeline parallel
|
||||
### Which quantization configs does vLLM CPU support?
|
||||
|
||||
- vLLM CPU supports quantizations:
|
||||
- AWQ (x86, s390x)
|
||||
- GPTQ (x86, s390x)
|
||||
- AWQ (x86, s390x, Power)
|
||||
- GPTQ (x86, s390x, Power)
|
||||
- compressed-tensor INT8 W8A8 (x86, s390x only)
|
||||
|
||||
### Why do I see `get_mempolicy: Operation not permitted` when running in Docker?
|
||||
|
||||
@@ -22,6 +22,7 @@ Sorted alphabetically by GitHub handle:
|
||||
- [@DarkLight1337](https://github.com/DarkLight1337): Multimodality, API server
|
||||
- [@dllehr-amd](https://github.com/dllehr-amd): AMD integration
|
||||
- [@esmeetu](https://github.com/esmeetu): developer marketing, community
|
||||
- [@gau-nernst](https://github.com/gau-nernst): Kernels and performance
|
||||
- [@gshtras](https://github.com/gshtras): ROCm / AMD GPU integration
|
||||
- [@Harry-Chen](https://github.com/Harry-Chen): Build, CI, CUDA architecture
|
||||
- [@heheda12345](https://github.com/heheda12345): Hybrid memory allocator
|
||||
@@ -54,6 +55,7 @@ Sorted alphabetically by GitHub handle:
|
||||
- [@tomeras91](https://github.com/tomeras91): MoE, Mamba, Ray
|
||||
- [@vadiklyutiy](https://github.com/vadiklyutiy): CI, performance, kernels
|
||||
- [@WoosukKwon](https://github.com/WoosukKwon): Project lead, engine core
|
||||
- [@wzhao18](https://github.com/wzhao18): Kernels and performance, KV offloading
|
||||
- [@xinli-sw](https://github.com/xinli-sw): Quantization and model support
|
||||
- [@xuechendi](https://github.com/xuechendi): Intel CPU/XPU integration, KV connector
|
||||
- [@yaochengji](https://github.com/yaochengji): TPU integration
|
||||
@@ -173,9 +175,10 @@ If you have PRs touching the area, please feel free to ping the area owner for r
|
||||
### External Kernels Integration
|
||||
|
||||
- FlashAttention: @LucasWilkinson, @MatthewBonanni
|
||||
- FlashInfer: @LucasWilkinson, @mgoin, @WoosukKwon, @MatthewBonanni
|
||||
- FlashInfer: @LucasWilkinson, @mgoin, @WoosukKwon, @MatthewBonanni, @wzhao18
|
||||
- Blackwell Kernels: @mgoin, @yewentao256
|
||||
- DeepEP/DeepGEMM: @mgoin, @yewentao256
|
||||
- FlashKDA: @gau-nernst
|
||||
|
||||
### Integrations
|
||||
|
||||
@@ -186,7 +189,7 @@ If you have PRs touching the area, please feel free to ping the area owner for r
|
||||
### Hardware
|
||||
|
||||
- Plugin Interface: @youkaichao, @Yikun
|
||||
- NVIDIA GPU: @pavanimajety
|
||||
- NVIDIA GPU: @pavanimajety, @wzhao18
|
||||
- AMD GPU: @gshtras, @tjtanaa, @hongxiayang, @shen-shanshan, @AndreasKaratzas
|
||||
- Intel CPU/GPU: @jikunshang, @bigPYJ1151, @xuechendi
|
||||
- Google TPU: @yaochengji
|
||||
|
||||
@@ -0,0 +1,34 @@
|
||||
<!-- Material nav.html (https://github.com/squidfunk/mkdocs-material/blob/9.7.7/material/templates/partials/nav.html) with navigation.prune applied to the API reference only -->
|
||||
{% import "partials/nav-item.html" as item with context %}
|
||||
{% set features = features + ["navigation.prune"] %}
|
||||
{% import "partials/nav-item.html" as pruned_item with context %}
|
||||
{% set class = "md-nav md-nav--primary" %}
|
||||
{% if "navigation.tabs" in features %}
|
||||
{% set class = class ~ " md-nav--lifted" %}
|
||||
{% endif %}
|
||||
{% if "toc.integrate" in features %}
|
||||
{% set class = class ~ " md-nav--integrated" %}
|
||||
{% endif %}
|
||||
<nav class="{{ class }}" aria-label="{{ lang.t('nav') }}" data-md-level="0">
|
||||
<label class="md-nav__title" for="__drawer">
|
||||
<a href="{{ config.extra.homepage | d(nav.homepage.url, true) | url }}" title="{{ config.site_name | e }}" class="md-nav__button md-logo" aria-label="{{ config.site_name }}" data-md-component="logo">
|
||||
{% include "partials/logo.html" %}
|
||||
</a>
|
||||
{{ config.site_name }}
|
||||
</label>
|
||||
{% if config.repo_url %}
|
||||
<div class="md-nav__source">
|
||||
{% include "partials/source.html" %}
|
||||
</div>
|
||||
{% endif %}
|
||||
<ul class="md-nav__list" data-md-scrollfix>
|
||||
{% for nav_item in nav.items %}
|
||||
{% set path = "__nav_" ~ loop.index %}
|
||||
{% if nav_item.children and (nav_item.children | first).url == "api/" %}
|
||||
{{ pruned_item.render(nav_item, path, 1) }}
|
||||
{% else %}
|
||||
{{ item.render(nav_item, path, 1) }}
|
||||
{% endif %}
|
||||
{% endfor %}
|
||||
</ul>
|
||||
</nav>
|
||||
@@ -374,7 +374,6 @@ th {
|
||||
| ------------ | ------ | ----------------- | -------------------- | ------------------------- |
|
||||
| `AfmoeForCausalLM` | Afmoe | TBA | ✅︎ | ✅︎ |
|
||||
| `ApertusForCausalLM` | Apertus | `swiss-ai/Apertus-8B-2509`, `swiss-ai/Apertus-70B-Instruct-2509`, etc. | ✅︎ | ✅︎ |
|
||||
| `ArceeForCausalLM` | Arcee (AFM) | `arcee-ai/AFM-4.5B-Base`, etc. | ✅︎ | ✅︎ |
|
||||
| `AXK1ForCausalLM` | A.X-K1 | `skt/A.X-K1`, etc. | | ✅︎ |
|
||||
| `BailingMoeForCausalLM` | Ling | `inclusionAI/Ling-lite-1.5`, `inclusionAI/Ling-plus`, etc. | ✅︎ | ✅︎ |
|
||||
| `BailingMoeV2ForCausalLM` | Ling | `inclusionAI/Ling-mini-2.0`, etc. | ✅︎ | ✅︎ |
|
||||
@@ -384,7 +383,6 @@ th {
|
||||
| `ChatGLMModel`, `ChatGLMForConditionalGeneration` | ChatGLM | `zai-org/chatglm2-6b`, `zai-org/chatglm3-6b`, `thu-coai/ShieldLM-6B-chatglm3`, etc. | ✅︎ | ✅︎ |
|
||||
| `CohereForCausalLM`, `Cohere2ForCausalLM` | Command-R, Command-A | `CohereLabs/c4ai-command-r-v01`, `CohereLabs/c4ai-command-r7b-12-2024`, `CohereLabs/c4ai-command-a-03-2025`, `CohereLabs/command-a-reasoning-08-2025`, etc. | ✅︎ | ✅︎ |
|
||||
| `Cohere2MoeForCausalLM` | North-Mini-Code | `CohereLabs/North-Mini-Code`, etc. | ✅︎ | ✅︎ |
|
||||
| `CwmForCausalLM` | CWM | `facebook/cwm`, etc. | ✅︎ | ✅︎ |
|
||||
| `DbrxForCausalLM` | DBRX | `databricks/dbrx-base`, `databricks/dbrx-instruct`, etc. | | ✅︎ |
|
||||
| `DeciLMForCausalLM` | DeciLM | `nvidia/Llama-3_3-Nemotron-Super-49B-v1`, etc. | ✅︎ | ✅︎ |
|
||||
| `DeepseekForCausalLM` | DeepSeek | `deepseek-ai/deepseek-llm-67b-base`, `deepseek-ai/deepseek-llm-7b-chat`, etc. | ✅︎ | ✅︎ |
|
||||
@@ -406,7 +404,6 @@ th {
|
||||
| `Gemma3ForCausalLM` | Gemma 3 | `google/gemma-3-1b-it`, etc. | ✅︎ | ✅︎ |
|
||||
| `Gemma3nForCausalLM` | Gemma 3n | `google/gemma-3n-E2B-it`, `google/gemma-3n-E4B-it`, etc. | | |
|
||||
| `Gemma4ForCausalLM` | Gemma 4 | `google/gemma-4-E2B-it`, etc. | ✅︎ | ✅︎ |
|
||||
| `GlmForCausalLM` | GLM-4 | `zai-org/glm-4-9b-chat-hf`, etc. | ✅︎ | ✅︎ |
|
||||
| `Glm4ForCausalLM` | GLM-4-0414 | `zai-org/GLM-4-32B-0414`, etc. | ✅︎ | ✅︎ |
|
||||
| `Glm4MoeForCausalLM` | GLM-4.5, GLM-4.6, GLM-4.7 | `zai-org/GLM-4.5`, etc. | ✅︎ | ✅︎ |
|
||||
| `Glm4MoeLiteForCausalLM` | GLM-4.7-Flash | `zai-org/GLM-4.7-Flash`, etc. | ✅︎ | ✅︎ |
|
||||
@@ -439,7 +436,6 @@ th {
|
||||
| `LongcatFlashForCausalLM` | LongCat-Flash | `meituan-longcat/LongCat-Flash-Chat`, `meituan-longcat/LongCat-Flash-Chat-FP8` | ✅︎ | ✅︎ |
|
||||
| `MambaForCausalLM` | Mamba | `state-spaces/mamba-130m-hf`, `state-spaces/mamba-790m-hf`, `state-spaces/mamba-2.8b-hf`, etc. | | ✅︎ |
|
||||
| `Mamba2ForCausalLM` | Mamba2 | `mistralai/Mamba-Codestral-7B-v0.1`, etc. | | ✅︎ |
|
||||
| `MellumForCausalLM` | Mellum 2 | `JetBrains/Mellum2-12B-A2.5B-Base`, etc. | | ✅︎ |
|
||||
| `MiMoForCausalLM` | MiMo | `XiaomiMiMo/MiMo-7B-RL`, etc. | ✅︎ | ✅︎ |
|
||||
| `MiMoV2FlashForCausalLM` | MiMoV2Flash | `XiaomiMiMo/MiMo-V2-Flash`, etc. | | ✅︎ |
|
||||
| `MiMoV2ForCausalLM` | MiMoV2Pro | `XiaomiMiMo/MiMo-V2.5-Pro`, etc. | | ✅︎ |
|
||||
@@ -486,10 +482,14 @@ Some models are supported only via the [Transformers modeling backend](#transfor
|
||||
|
||||
| Architecture | Models | Example HF Models | [LoRA](../features/lora.md) | [PP](../serving/parallelism_scaling.md) |
|
||||
| ------------ | ------ | ----------------- | -------------------- | ------------------------- |
|
||||
| `ArceeForCausalLM` | Arcee (AFM) | `arcee-ai/AFM-4.5B-Base`, etc. | ✅︎ | ✅︎ |
|
||||
| `CwmForCausalLM` | CWM | `facebook/cwm`, etc. | ✅︎ | ✅︎ |
|
||||
| `FlexOlmoForCausalLM` | FlexOlmo | `allenai/FlexOlmo-7x7B-1T`, `allenai/FlexOlmo-7x7B-1T-RT`, etc. | | ✅︎ |
|
||||
| `GlmForCausalLM` | GLM-4 | `zai-org/glm-4-9b-chat-hf`, etc. | ✅︎ | ✅︎ |
|
||||
| `GPTBigCodeForCausalLM` | StarCoder, SantaCoder, WizardCoder | `bigcode/starcoder`, `bigcode/gpt_bigcode-santacoder`, `WizardLM/WizardCoder-15B-V1.0`, etc. | ✅︎ | |
|
||||
| `HunYuanDenseV1ForCausalLM` | Hunyuan Dense | `tencent/Hunyuan-7B-Instruct` | ✅︎ | ✅︎ |
|
||||
| `HunYuanMoEV1ForCausalLM` | Hunyuan-A13B | `tencent/Hunyuan-A13B-Instruct`, `tencent/Hunyuan-A13B-Pretrain`, `tencent/Hunyuan-A13B-Instruct-FP8`, etc. | ✅︎ | ✅︎ |
|
||||
| `MellumForCausalLM` | Mellum 2 | `JetBrains/Mellum2-12B-A2.5B-Base`, etc. | | ✅︎ |
|
||||
| `NanbeigeForCausalLM` | Nanbeige4.2 | `Nanbeige/Nanbeige4.2-3B`, etc. | ✅︎ | ✅︎ |
|
||||
| `OlmoForCausalLM` | OLMo | `allenai/OLMo-1B-hf`, `allenai/OLMo-7B-hf`, etc. | ✅︎ | ✅︎ |
|
||||
| `Olmo2ForCausalLM` | OLMo2 | `allenai/OLMo-2-0425-1B`, etc. | ✅︎ | ✅︎ |
|
||||
|
||||
@@ -1,4 +1,3 @@
|
||||
# shellcheck disable=SC2317 # code after the exit/exit-183 gate below is intentionally dead pending re-enablement (see line comment)
|
||||
if [ "$READTHEDOCS_VERSION_TYPE" != "external" ]; then
|
||||
echo "Not a PR build (version type=$READTHEDOCS_VERSION_TYPE); skipping pre-run-check gate."
|
||||
exit 0
|
||||
@@ -11,24 +10,19 @@ if [ -n "$GITHUB_TOKEN" ]; then
|
||||
CURL_AUTH=(-H "Authorization: Bearer $GITHUB_TOKEN")
|
||||
fi
|
||||
|
||||
# Docs builds are now manually enabled via the 'build-docs' label.
|
||||
# The 'build-docs' label forces a docs build, bypassing the checks below.
|
||||
echo "Checking for the 'build-docs' label on PR #${READTHEDOCS_VERSION_NAME}..."
|
||||
LABELS=$(curl -sS "${CURL_AUTH[@]}" "https://api.github.com/repos/vllm-project/vllm/issues/${READTHEDOCS_VERSION_NAME}/labels" | python3 -c "import sys, json; print('\n'.join(l.get('name', '') for l in json.load(sys.stdin)))")
|
||||
if printf '%s\n' "$LABELS" | grep -qx "build-docs"; then
|
||||
echo "PR has the 'build-docs' label; continuing build."
|
||||
echo "PR has the 'build-docs' label; forcing build."
|
||||
exit 0
|
||||
else
|
||||
echo "PR does not have the 'build-docs' label; cancelling build."
|
||||
# See https://docs.readthedocs.com/platform/latest/guides/build/skip-build.html for info on exit code
|
||||
exit 183
|
||||
fi
|
||||
|
||||
# Everything below this line is effectively disabled as a temporary measure.
|
||||
echo "Checking for changes to docs-affecting files vs origin/main..."
|
||||
DOCS_PATHS=(
|
||||
docs/ # Actual docs content
|
||||
examples/ # Examples are rendered in docs
|
||||
vllm/ # API & CLI reference
|
||||
# vllm/ # API & CLI reference (too broad, use 'build-docs' label)
|
||||
requirements/test/cuda.txt # CLI reference (see docs/mkdocs/gen_files/generate_argparse.py)
|
||||
mkdocs.yaml # Affects build process
|
||||
.readthedocs.yaml # Affects build process
|
||||
|
||||
@@ -23,6 +23,10 @@ endpoints.
|
||||
|
||||
The derender step needs more than the engine's `token_ids`. It also consumes the original `chat_request`/`completion_request` and `prompt_tokens` carried over from the render step (see [Request format](#request-format)) so the tool and reasoning parsers have the context they need. The chat streaming path additionally needs `prompt_token_ids` (see [Streaming cost](#streaming-cost)) when a parser is configured.
|
||||
|
||||
All derender endpoints also accept an optional `prompt_token_ids` (the `token_ids` of the `GenerateRequest` returned by `/render`). It's used to seed detokenization from the end of the prompt, the same way the engine does. Without it, the first output token is decoded as if it started a new sequence. On SentencePiece tokenizers (Llama-2, Phi-3) that drops its leading space compared to `/v1/chat/completions` and `/v1/completions`. If it's omitted, derender falls back to the generate response's `prompt_token_ids` (set when the `GenerateRequest` has `return_token_ids`), then to decoding without prompt context.
|
||||
|
||||
Seeding only reads the last few IDs (7 today), so a suffix of the prompt is enough and a long prompt doesn't need to travel back to derender. The exception is streaming chat with a tool or reasoning parser configured. There the parser also reads `prompt_token_ids` to decide whether the prompt left reasoning open and it can look back to any point in the prompt. In this case, send the full list (see [Streaming cost](#streaming-cost)).
|
||||
|
||||
## API Reference
|
||||
|
||||
- Chat Completions Derender API (`/v1/chat/completions/derender`)
|
||||
@@ -96,9 +100,10 @@ The server keeps no state between calls. Everything the next call needs is in `s
|
||||
- **`n > 1` is N streams.** Generate interleaves chunks for different choices and tags each with its `index`. Keep one `stream_state` per index and send each chunk with the state for its index. Each index gets its own `role` delta on its first chunk.
|
||||
- **Send every chunk through, including the last ones.** The finish reason usually arrives with the final tokens. A chunk with a `finish_reason` and no tokens is accepted too and flushes any buffered tool call arguments. With `stream_options: {"include_usage": true}`, generate ends with a usage only chunk (`choices: []`). Send it through derender like any other chunk to get the usage chunk for the response. `usage.prompt_tokens` is the request's `prompt_tokens` if set, otherwise the generate chunk's.
|
||||
- **Send the same context on every call.** `chat_request` and `prompt_token_ids` aren't kept between calls, so they go with every chunk including the usage chunk. Both are required when a tool or reasoning parser is configured. `prompt_token_ids` is the `token_ids` of the `GenerateRequest` returned by `/render`.
|
||||
- **Send `prompt_token_ids` on the first chunk, even without a parser.** It seeds detokenization so the first token's leading space matches the coupled endpoint (see [Request format](#request-format)). Later chunks already carry that context in `stream_state`.
|
||||
- **Don't forward `[DONE]`.** It marks the end of the generate stream and isn't a chunk.
|
||||
|
||||
Streaming chunks don't carry logprobs yet, so `logprobs` on a generate chunk are dropped. The non-streaming endpoints do resolve them, including `token_id:N` placeholders.
|
||||
Streaming derender resolves `logprobs` on the plain detokenization path, including `token_id:N` placeholders, same as the non-streaming endpoints (see [Streaming state and logprobs](#streaming-state-and-logprobs)). The parser path doesn't resolve them yet and drops them.
|
||||
|
||||
## Streaming cost
|
||||
|
||||
@@ -110,11 +115,20 @@ When a tool or reasoning parser is configured, parser internal state (buffered m
|
||||
|
||||
- **Transport**: `output_token_ids` and `output_chunk_lens` round-trip in full in both directions on every call. `output_chunk_lens` has one entry per chunk, which is one per token without speculative decoding. This means O(n) bytes per chunk, O(n²) bytes over a full generation. Bounded by `max_model_len`. `prompt_token_ids` is sent in full on every call too and it isn't trimmed as `output_token_ids` grows. This means that for most of a stream it dominates the per chunk payload. A 100k token prompt with 1k tokens of output means `prompt_token_ids` is ~99% of the request body on every chunk.
|
||||
- **Compute**: replay is O(n) `parse_delta` calls per chunk (O(n²) per generation). `parse_delta` itself is O(n) for parsers that re-scan accumulated text (e.g. Hermes tool-call JSON, DeepSeek-R1 reasoning). The per-generation cost is O(n³) character work, not O(n²). This is a deliberately minimal first implementation with no caching layer.
|
||||
- The parser path also requires `prompt_token_ids` so `parse_delta` can settle whether the prompt left reasoning open or not. Since parser state can't be carried across calls, it re-scans the full prompt once per chunk.
|
||||
- The parser path also requires `prompt_token_ids` so `parse_delta` can settle whether the prompt left reasoning open or not. Since parser state can't be carried across calls, it re-scans the full prompt once per chunk. Send the full list and not a suffix because the last reasoning marker can be anywhere in the prompt.
|
||||
- Replay runs off the event loop on the renderer's executor (`renderer_num_workers`, default `1`). Size it for the expected number of concurrent parser configured streams.
|
||||
|
||||
`output_token_ids` and `prompt_token_ids` are both bounded by `max_model_len` but callers streaming long reasoning traces through a parser configured model should expect materially more state transport and CPU cost than the plain detokenization path.
|
||||
|
||||
## Streaming state and logprobs
|
||||
|
||||
Streaming derender is stateless on the server side: all mutable state lives in the client-carried `stream_state` (`DerenderStreamState`), passed back on every per-chunk call. Besides the bounded incremental detokenization window (`prev_tokens`, `prefix_offset`, `read_offset`), `role_sent`, and the parser path's replay fields (see [Streaming cost](#streaming-cost)), the state carries two fields for logprob handling:
|
||||
|
||||
- `logprob_context_token_ids`: the trailing sampled token IDs (at most 4) from previous chunks, used to seed byte-fallback (U+FFFD) correction so multi-byte characters whose tokens split across chunk boundaries still resolve to real strings
|
||||
- `logprob_text_offset`: the cumulative emitted text length, so `text_offset` in completion streaming logprobs stays absolute across chunks instead of restarting at 0
|
||||
|
||||
When a streamed `GenerateResponseStreamChoice` carries `logprobs`, the `token_id:N` placeholders are resolved per chunk and the resolved logprobs are attached to the corresponding streamed choice — for chat as `ChatCompletionLogProbs`, for completions converted to the flat `CompletionLogProbs` lists. Chunks without `logprobs` produce choices with `logprobs: null`.
|
||||
|
||||
## Example
|
||||
|
||||
The example below drives the full `render → generate → derender` round trip for a chat request against a GPU less render server (`/render`, `/derender`) and a token-in / token-out engine (`/inference/v1/generate`).
|
||||
@@ -144,7 +158,7 @@ with httpx.Client(timeout=60.0) as client:
|
||||
generate_request = client.post(
|
||||
f"{RENDER}/v1/chat/completions/render", json=chat_request
|
||||
).json()
|
||||
prompt_tokens = len(generate_request["token_ids"])
|
||||
prompt_token_ids = generate_request["token_ids"]
|
||||
|
||||
# 2. Generate: token IDs -> token IDs (token-in / token-out engine)
|
||||
generate_response = client.post(
|
||||
@@ -157,7 +171,8 @@ with httpx.Client(timeout=60.0) as client:
|
||||
json={
|
||||
"model": MODEL,
|
||||
"generate_response": generate_response,
|
||||
"prompt_tokens": prompt_tokens,
|
||||
"prompt_tokens": len(prompt_token_ids),
|
||||
"prompt_token_ids": prompt_token_ids,
|
||||
"chat_request": chat_request,
|
||||
},
|
||||
).json()
|
||||
|
||||
@@ -51,8 +51,7 @@ and provision transport limits and memory accordingly.
|
||||
For example, start a standard inference server with scale-out endpoints enabled:
|
||||
|
||||
```bash
|
||||
VLLM_ENABLE_SCALE_OUT_ENDPOINTS=1 \
|
||||
vllm serve meta-llama/Llama-3.1-8B-Instruct
|
||||
vllm serve meta-llama/Llama-3.1-8B-Instruct --enable-scale-out
|
||||
```
|
||||
|
||||
Send a Responses request and use `jq` to extract the `token_ids` field:
|
||||
|
||||
@@ -16,7 +16,7 @@ The following open-source RL libraries use vLLM for fast rollouts (sorted alphab
|
||||
- [Unsloth](https://github.com/unslothai/unsloth)
|
||||
- [verl](https://github.com/volcengine/verl)
|
||||
|
||||
For weight synchronization between training and inference, see the [Weight Transfer](weight_transfer/README.md) documentation, which covers the pluggable backend system with [NCCL](weight_transfer/nccl.md) (multi-GPU) and [IPC](weight_transfer/ipc.md) (same-GPU) engines.
|
||||
For weight synchronization between training and inference, see the [Weight Transfer](weight_transfer/README.md) documentation, which covers the pluggable backend system with [NCCL](weight_transfer/nccl.md) (multi-GPU) and [IPC](weight_transfer/ipc.md) (same-GPU) engines. To verify that an update landed on every rank, see the [Weight Checker](weight_checker.md).
|
||||
|
||||
For pipelining generation and training to improve GPU utilization and throughput, see the [Async Reinforcement Learning](async_rl.md) guide, which covers the pause/resume API for safely updating weights mid-flight.
|
||||
|
||||
|
||||
@@ -53,6 +53,8 @@ The mask is also available via the `/inference/v1/generate` HTTP endpoint:
|
||||
}
|
||||
```
|
||||
|
||||
With `stream=true`, each chunk carries the masks of the tokens it emits.
|
||||
|
||||
## Requirements
|
||||
|
||||
| Requirement | Reason |
|
||||
@@ -63,9 +65,12 @@ The mask is also available via the `/inference/v1/generate` HTTP endpoint:
|
||||
| `top_k > 0` | Bounds mask size; pure top-p can produce vocab-sized masks |
|
||||
| Model Runner V2 | Required by the async D2H copy pipeline |
|
||||
|
||||
Speculative decoding is supported; each emitted token gets the support of the
|
||||
target distribution at its position.
|
||||
|
||||
The engine rejects unsupported combinations at startup or request time:
|
||||
|
||||
- Speculative decoding
|
||||
- Speculative decoding with adaptive verification
|
||||
- Diffusion models
|
||||
- Custom logits processors (engine-level `--logits-processors`)
|
||||
|
||||
@@ -104,10 +109,41 @@ log_prob = log_softmax(masked_logits)[sampled_token_id]
|
||||
Both sides normalize over the same token set, so the importance ratio is
|
||||
consistent.
|
||||
|
||||
## Score centering with top-k logprobs
|
||||
|
||||
[Score centering](https://arxiv.org/abs/2609.20807) needs the sampler's top-k
|
||||
token IDs and logprobs plus the sampled token's logprob; the paper uses
|
||||
`k=128` and does not need the exact sampling support. vLLM already returns
|
||||
this through the ordinary `logprobs` parameter:
|
||||
|
||||
```bash
|
||||
vllm serve <model> --logprobs-mode processed_logprobs --max-logprobs 128
|
||||
```
|
||||
|
||||
```json
|
||||
{"sampling_params": {"temperature": 1.0, "top_p": 0.9, "logprobs": 128}}
|
||||
```
|
||||
|
||||
Each position then carries the sampled token's logprob and the top-128
|
||||
`(token_id, logprob)` pairs, normalized over the post-truncation support:
|
||||
|
||||
```json
|
||||
{"token": "token_id:11", "logprob": -0.757,
|
||||
"top_logprobs": [{"token": "token_id:11", "logprob": -0.757},
|
||||
{"token": "token_id:13", "logprob": -0.632},
|
||||
{"token": "token_id:0", "logprob": -9999.0}, ...]}
|
||||
```
|
||||
|
||||
- `--max-logprobs` defaults to 20; raise it to the requested `k`.
|
||||
- Tokens outside the support are returned as `-9999.0` with arbitrary IDs;
|
||||
drop them. The finite entries are the sampling mask when it has at most `k`
|
||||
tokens.
|
||||
- The sampled token is always the first entry and may repeat inside the top-k.
|
||||
- `/inference/v1/generate` encodes IDs as `token_id:<id>`; the OpenAI
|
||||
endpoints need `return_tokens_as_token_ids`.
|
||||
|
||||
## Limitations
|
||||
|
||||
- **Engine-level flag:** `--return-sampling-mask` globally disables the
|
||||
FlashInfer fused sampler. All requests pay the cost of the PyTorch sampling
|
||||
path, even if they don't need the mask.
|
||||
- **No streaming support:** The mask is returned only in the final response,
|
||||
not in intermediate streaming chunks.
|
||||
|
||||
@@ -0,0 +1,60 @@
|
||||
# Weight Checker
|
||||
|
||||
The Weight Checker verifies that an RL weight update landed on every rank. It
|
||||
hashes each model parameter with SHA-256, so a caller can save the original
|
||||
digests, reset the weights, transfer them back, and confirm they match.
|
||||
|
||||
Enable it with `VLLM_SERVER_DEV_MODE=1`. All operations use
|
||||
`POST /weight_checker`:
|
||||
|
||||
| Request | Response |
|
||||
| --- | --- |
|
||||
| `{"action": "checksum"}` | `{"checksums": {key: sha256_hex}}` |
|
||||
| `{"action": "reset"}` | `{"status": "reset"}` |
|
||||
| `{"action": "compare", "baseline": {key: sha256_hex}}` | `{"match": bool, "mismatches": [key]}` |
|
||||
|
||||
Keys have the form `dp{dp}:pp{pp}:pcp{pcp}:tp{tp}:{tensor_name}`, so each
|
||||
shard is checked separately. All actions cover the target model's parameters,
|
||||
which is what weight loading writes; `reset` zeroes them. The endpoint keeps no
|
||||
state, so the caller holds the baseline. Invalid requests return HTTP 400.
|
||||
|
||||
## RL weight-update check
|
||||
|
||||
```bash
|
||||
URL=http://localhost:8000
|
||||
|
||||
# 1. Stop serving, then save the original digests.
|
||||
curl -X POST "$URL/pause?mode=abort"
|
||||
curl -X POST $URL/weight_checker -H 'Content-Type: application/json' \
|
||||
-d '{"action":"checksum"}' > baseline.json
|
||||
|
||||
# 2. Zero the weights.
|
||||
curl -X POST $URL/weight_checker -H 'Content-Type: application/json' \
|
||||
-d '{"action":"reset"}'
|
||||
|
||||
# 3. Transfer the original weights with start/update/finish_weight_update.
|
||||
|
||||
# 4. Compare against the baseline; expect {"match": true, "mismatches": []}.
|
||||
curl -X POST $URL/weight_checker -H 'Content-Type: application/json' \
|
||||
-d "{\"action\":\"compare\",\"baseline\":$(jq .checksums baseline.json)}"
|
||||
|
||||
curl -X POST $URL/resume
|
||||
```
|
||||
|
||||
## Limitations
|
||||
|
||||
- Stay paused from `checksum` to `compare`: the weights are invalid between
|
||||
`reset` and the transfer, and EPLB moves experts while serving.
|
||||
- A request covers the engines managed by the API server it reaches. With
|
||||
several API servers (for example `--data-parallel-external-lb`), send it to
|
||||
each one.
|
||||
- Buffers, draft models and LoRA adapters are not checked.
|
||||
- CPU backends that repack linear weights hide them from the checker.
|
||||
- compressed-tensors online transforms are not supported: their Hadamard
|
||||
weights are normalized once per storage, so restored copies differ.
|
||||
- MoE weights padded beyond the checkpoint shape (DeepEP hidden size, MXFP4)
|
||||
keep uninitialized padding, so `compare` reports them as mismatches.
|
||||
- The engine must be awake: sleep levels 1 and 2 release the weight memory.
|
||||
- `--offload-backend prefetch` is not supported: parameters point at staging
|
||||
buffers, not the offloaded weights.
|
||||
- Hashing copies every weight to CPU, so keep it off latency-sensitive paths.
|
||||
@@ -36,6 +36,7 @@ engine drives on your behalf:
|
||||
| [sparse_nccl](nccl.md#sparse-nccl) | NCCL broadcast | Checkpoint-coordinate sparse weight patches |
|
||||
| [sharded_rdt](sharded_rdt.md) | NIXL / Ray Direct Transport (pull-based) | Very large models where each worker needs only its own slice (MoE with expert parallelism) |
|
||||
| [nccl_m2n](m2n.md) | NCCL M2N reshard | Trainer and inference use different sharding layouts |
|
||||
| [ModelExpress](modelexpress.md) | ModelExpress version-based transfer | Install immutable published weight versions, including object-storage checkpoints and deltas |
|
||||
|
||||
## Quickstart
|
||||
|
||||
|
||||
@@ -0,0 +1,156 @@
|
||||
# ModelExpress Engine
|
||||
|
||||
The `modelexpress` weight transfer engine installs immutable weight versions
|
||||
published through [ModelExpress](https://github.com/ai-dynamo/modelexpress).
|
||||
Each update identifies the exact version to install, while ModelExpress handles
|
||||
source selection, transfer, and installation.
|
||||
|
||||
## When to Use ModelExpress
|
||||
|
||||
- Your workflow manages immutable weight versions through the ModelExpress
|
||||
Refit API.
|
||||
- Inference workers need to pull a specific published version through
|
||||
ModelExpress, using GPU peer transfer or object storage.
|
||||
- You want to load full checkpoints or replay checkpoint deltas from S3.
|
||||
|
||||
## How It Works
|
||||
|
||||
1. A publisher creates a weight-version resource through the ModelExpress Refit
|
||||
API, publishes its payload, and marks the version READY.
|
||||
2. The orchestrator pauses generation and calls `start_weight_update` on the
|
||||
inference engine.
|
||||
3. The orchestrator calls `update_weights` with the target `version_id`.
|
||||
Each worker's `ModelExpressGeneratorClient` resolves and stages that version,
|
||||
then applies it to the live vLLM model.
|
||||
4. After a successful update, `finish_weight_update` releases the staged handle.
|
||||
The orchestrator can then resume inference.
|
||||
|
||||
Initialize the backend before starting an update session, and send exactly one
|
||||
version per session. Invalid transitions must raise; an orchestrator must keep
|
||||
inference paused after an update error and retry with a new session only after
|
||||
cleanup. An error does not roll back weights that were already applied.
|
||||
|
||||
`ModelExpressTrainerClient` is an optional publisher. Custom integrations can
|
||||
manage weight-version resources directly through the ModelExpress server's
|
||||
Refit API. For S3, create a STAGING version with the correct model, payload
|
||||
format, lineage, and index URI; upload the index and referenced shards in the
|
||||
format expected by `ModelExpressGeneratorClient`; then mark the version READY.
|
||||
See the [S3 Delta Weight Refit guide](https://github.com/ai-dynamo/modelexpress/blob/main/docs/S3_DELTA_WEIGHT_REFIT.md)
|
||||
for the publication sequence and generator artifact requirements.
|
||||
|
||||
vLLM's `modelexpress_engine.py` re-exports the engine and its init/update
|
||||
dataclasses from
|
||||
`modelexpress_rl.inference.engines.vllm.weight_transfer_engine`.
|
||||
The implementation and configuration schema live in ModelExpress. Upgrading
|
||||
the installed ModelExpress package and restarting workers picks up compatible
|
||||
changes without a vLLM code change. The package must continue to implement
|
||||
vLLM's `WeightTransferEngine` interface.
|
||||
|
||||
## Inference Side
|
||||
|
||||
!!! warning "Development version"
|
||||
|
||||
This backend currently uses unreleased ModelExpress features from `main`,
|
||||
which may introduce breaking changes. These features will be included in
|
||||
the official ModelExpress v0.8.0 release.
|
||||
|
||||
Install ModelExpress from the `main` branch on every worker:
|
||||
|
||||
```bash
|
||||
uv pip install "git+https://github.com/ai-dynamo/modelexpress@main#subdirectory=modelexpress_client/python"
|
||||
```
|
||||
|
||||
The backend is registered natively and loaded only when selected;
|
||||
`VLLM_PLUGINS=modelexpress` is not required for weight transfer. If the plugin
|
||||
is enabled, it preserves the native registration and uses the same engine
|
||||
implementation.
|
||||
|
||||
```python
|
||||
from vllm import LLM
|
||||
from vllm.config import WeightTransferConfig
|
||||
|
||||
llm = LLM(model="my-model", weight_transfer_config=WeightTransferConfig(backend="modelexpress"))
|
||||
```
|
||||
|
||||
```bash
|
||||
vllm serve /models/launch \
|
||||
--load-format safetensors \
|
||||
--weight-transfer-config '{"backend":"modelexpress"}'
|
||||
```
|
||||
|
||||
### Initialize the engine
|
||||
|
||||
Initialize once by passing `init_info` to `init_weight_transfer_engine`.
|
||||
For HTTP administration, set `VLLM_SERVER_DEV_MODE=1` before starting vLLM and
|
||||
restrict these endpoints to trusted orchestrators.
|
||||
|
||||
For S3 updates, send the following body to `POST /init_weight_transfer_engine`:
|
||||
|
||||
```json
|
||||
{
|
||||
"init_info": {
|
||||
"model_name": "policy",
|
||||
"server_url": "modelexpress:8001",
|
||||
"object_storage_type": "S3",
|
||||
"initial_base_version_id": "policy-v0",
|
||||
"seed_checkpoint_path": "/models/launch",
|
||||
"refit_checkpoint_dir": "/mxdelta/receiver",
|
||||
"refit_checkpoint_max_size_gb": 200,
|
||||
"object_storage_region_name": "us-west-2"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### `ModelExpressWeightTransferInitInfo`
|
||||
|
||||
These fields belong inside `init_info`, rather than
|
||||
`--weight-transfer-config`. The installed ModelExpress package defines the
|
||||
schema and defaults.
|
||||
|
||||
| Field | Default | Description |
|
||||
| ----- | ------- | ----------- |
|
||||
| `model_name` | vLLM model name | Logical model identity used for published ModelExpress versions. Set this when it differs from the vLLM model path or name. |
|
||||
| `server_url` | ModelExpress configuration | ModelExpress server address, such as `modelexpress:8001`. If omitted, uses ModelExpress's environment settings, then `localhost:8001`. |
|
||||
| `initial_serving_version_id` | `None` | Version already loaded by the inference worker. If omitted, ModelExpress uses the initial version supplied by its runtime, when available. |
|
||||
| `registration_ttl_seconds` | Three MX heartbeat intervals | Worker registration lifetime in seconds, renewed by ModelExpress. |
|
||||
| `lease_ttl_seconds` | Registration lifetime | Weight-version lease lifetime in seconds, renewed while held. |
|
||||
| `max_transfer_attempts` | `3` | Maximum source-discovery and transfer attempts for one staged update. |
|
||||
| `max_replay_chain_length` | `64` | Maximum number of payload revisions replayed to reach the requested version. |
|
||||
| `rpc_timeout_seconds` | `30.0` | Deadline in seconds for each control-plane or manifest RPC. |
|
||||
|
||||
### Object storage settings
|
||||
|
||||
To enable object storage, provide `object_storage_type`,
|
||||
`initial_base_version_id`, `seed_checkpoint_path`, and `refit_checkpoint_dir`
|
||||
inside `init_info`. Endpoint and region settings also require these four fields.
|
||||
|
||||
| Field | Default | Description |
|
||||
| ----- | ------- | ----------- |
|
||||
| `object_storage_type` | `None` | Object storage provider. Use `"S3"`; the generator currently supports S3. |
|
||||
| `initial_base_version_id` | `None` | Registered base version matching the seed checkpoint. Required for object storage. |
|
||||
| `seed_checkpoint_path` | `None` | Local seed checkpoint path accessible to each worker. Required for object storage. |
|
||||
| `refit_checkpoint_dir` | `None` | Local directory for cached checkpoints and delta replay. Required for object storage. |
|
||||
| `refit_checkpoint_max_size_gb` | ModelExpress default | Checkpoint cache quota in decimal GB. The example sets an explicit 200 GB quota; it does not reserve disk space. |
|
||||
| `object_storage_endpoint_url` | `None` | Optional endpoint override for an S3-compatible service. |
|
||||
| `object_storage_region_name` | `None` | Optional S3 region, such as `us-west-2`. |
|
||||
|
||||
The seed checkpoint must match `initial_base_version_id`. Credentials use
|
||||
ModelExpress's normal credential chain. Omit the object storage settings when
|
||||
using only GPU peer transfer.
|
||||
|
||||
### Apply a weight update
|
||||
|
||||
After the target version is READY, pause generation and call
|
||||
`POST /start_weight_update`. Then send the exact immutable version ID to
|
||||
`POST /update_weights`:
|
||||
|
||||
```json
|
||||
{"update_info": {"version_id": "policy-v1"}}
|
||||
```
|
||||
|
||||
Call `POST /finish_weight_update` only after the update succeeds, then resume
|
||||
inference. The staged handle is retained until finish or shutdown. Staging and
|
||||
installation failures propagate; do not resume a worker with uncertain weights.
|
||||
|
||||
See the ModelExpress documentation for publication, source selection, object
|
||||
storage, and deployment requirements.
|
||||
@@ -75,6 +75,48 @@ These metrics are available via `--enable-mfu-metrics`:
|
||||
|
||||
--8<-- "gen:metrics-mfu"
|
||||
|
||||
## Custom Histogram Buckets
|
||||
|
||||
The core engine histograms ship with default bucket boundaries tuned for
|
||||
typical serving workloads. The `--custom-histogram-buckets` option replaces
|
||||
the boundaries of one or more *bucket families* — exactly the histograms
|
||||
listed in the table below — with your own list; histograms owned by other
|
||||
subsystems (for example, the NIXL connector metrics) are not affected. Use it,
|
||||
for example, to track sub-300ms latency SLOs with the request-phase
|
||||
histograms, whose smallest default boundary is 0.3s:
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3-0.6B \
|
||||
--custom-histogram-buckets '{"request_latency": [0.01, 0.05, 0.1, 0.25, 0.5, 1.0, 5.0, 30.0]}'
|
||||
```
|
||||
|
||||
Each family key overrides a group of related histograms:
|
||||
|
||||
| Family key | Histograms |
|
||||
| --- | --- |
|
||||
| `request_latency` | `vllm:e2e_request_latency_seconds`, `vllm:request_queue_time_seconds`, `vllm:request_inference_time_seconds`, `vllm:request_prefill_time_seconds`, `vllm:request_decode_time_seconds` |
|
||||
| `time_to_first_token` | `vllm:time_to_first_token_seconds` |
|
||||
| `inter_token_latency` | `vllm:inter_token_latency_seconds`, `vllm:request_time_per_output_token_seconds` |
|
||||
| `iteration_tokens` | `vllm:iteration_tokens_total` |
|
||||
| `request_params_n` | `vllm:request_params_n` |
|
||||
| `request_num_preemptions` | `vllm:request_num_preemptions` |
|
||||
| `request_tokens` | `vllm:request_prompt_tokens`, `vllm:request_generation_tokens`, `vllm:request_max_num_generation_tokens`, `vllm:request_params_max_tokens`, `vllm:request_prefill_kv_computed_tokens` |
|
||||
| `kv_cache_residency` | `vllm:kv_block_lifetime_seconds`, `vllm:kv_block_idle_before_evict_seconds`, `vllm:kv_block_reuse_gap_seconds` |
|
||||
|
||||
Bucket values must be positive, finite, and strictly increasing; unknown
|
||||
family keys are rejected at startup. Families you do not list keep their
|
||||
default boundaries. The `request_tokens` defaults normally scale with
|
||||
`--max-model-len`; an override replaces that computed list. The
|
||||
`kv_cache_residency` family only takes effect when `--kv-cache-metrics` is
|
||||
enabled.
|
||||
|
||||
!!! warning "Bucket cardinality"
|
||||
Every bucket boundary creates one extra time series per metric and per
|
||||
label combination (model and engine index, multiplied under data-parallel
|
||||
deployments). Long bucket lists inflate Prometheus storage, scrape sizes,
|
||||
and query costs. Keep custom lists short, and only override the families
|
||||
you actively monitor.
|
||||
|
||||
## Deprecation Policy
|
||||
|
||||
Note: when metrics are deprecated in version `X.Y`, they are hidden in version `X.Y+1`
|
||||
|
||||
@@ -375,7 +375,7 @@ An attacker who can reach the gRPC port can:
|
||||
1. **Run arbitrary inference** via the `Generate` and `GenerateStream` RPCs without any credentials
|
||||
2. **Mutate engine state** by pausing generation, sleeping the engine, or initiating configured RL weight updates through the `Control` service
|
||||
3. **Consume GPU and compute resources** by submitting unbounded generation requests
|
||||
4. **Cause Denial of Service** by exploiting bugs in the gRPC interface that can crash vLLM.
|
||||
4. **Stop a managed engine** through `Control.Shutdown`, or cause denial of service by exploiting bugs in the gRPC interface.
|
||||
|
||||
### Recommendations
|
||||
|
||||
|
||||
@@ -57,7 +57,12 @@ The vllm instances and `disagg_encoder_proxy` supports local URIs with ```{"url"
|
||||
|
||||
## EC connector and KV transfer
|
||||
|
||||
The `ECExampleonnector` is used to store the encoder cache on local disk and facilitate transfer. To enable the encoder disaggregation feature, add the following configuration:
|
||||
`ECExampleConnector` stores encoder cache on local disk. For RDMA/TCP transfer
|
||||
without a shared filesystem, use `ECMooncakeConnector` (1E + 1PD only through
|
||||
this proxy). See
|
||||
[ECMooncakeConnector Usage Guide](../../../docs/features/mooncake_ec_connector_usage.md).
|
||||
|
||||
To enable filesystem encoder disaggregation, add:
|
||||
|
||||
```bash
|
||||
# Add to encoder instance:
|
||||
@@ -104,6 +109,8 @@ If you enable prefill instance (`--prefill-servers-urls` not disabled), you will
|
||||
| `--encode-servers-urls` | Comma-separated list of encoder endpoints. Every multimodal item extracted from the request is fanned out to one of these URLs in a round-robin fashion. |
|
||||
| `--prefill-servers-urls` | Comma-separated list of prefill endpoints. Set to `disable`, `none`, or `""` to skip the dedicated prefill phase and run E+PD (encoder + combined prefill/decode). |
|
||||
| `--decode-servers-urls` | Comma-separated list of decode endpoints. Non-stream and stream paths both round-robin over this list. |
|
||||
| `--ec-consumer-zmq-addrs` | Mooncake EC consumer control addresses, aligned with `--decode-servers-urls`. Required for `ECMooncakeConnector`. Supported only in E+PD mode (`--prefill-servers-urls disable`). |
|
||||
| `--ec-consumer-dp-size` | Data-parallel replicas per EC consumer (default 1). |
|
||||
| `--host`, `--port` | Bind address for the proxy itself (defaults: `0.0.0.0:8000`). |
|
||||
|
||||
### Dynamic registration
|
||||
|
||||
@@ -6,6 +6,7 @@ robust and flexible configuration of the various loggers used by vLLM.
|
||||
For `vllm serve`, configure logging with CLI arguments:
|
||||
|
||||
- Use the built-in configuration, optionally with `--log-level`.
|
||||
- Emit structured JSON with `--logging-config.formatter json`.
|
||||
- Use a custom Python logging configuration file with
|
||||
`--logging-config.pylogging_config_file`.
|
||||
- Disable vLLM logging configuration with
|
||||
@@ -14,7 +15,7 @@ For `vllm serve`, configure logging with CLI arguments:
|
||||
## CLI logging configuration
|
||||
|
||||
`--logging-config` accepts a JSON object. Its fields are `log_level`,
|
||||
`configure_logging`, and `pylogging_config_file`. For example:
|
||||
`formatter`, `configure_logging`, and `pylogging_config_file`. For example:
|
||||
|
||||
```bash
|
||||
vllm serve mistralai/Mistral-7B-v0.1 \
|
||||
@@ -28,6 +29,25 @@ vllm serve mistralai/Mistral-7B-v0.1 \
|
||||
--logging-config.log_level DEBUG
|
||||
```
|
||||
|
||||
The built-in `text` formatter is selected by default. To emit one JSON object
|
||||
per vLLM and Uvicorn log record, select the built-in `json` formatter:
|
||||
|
||||
```bash
|
||||
vllm serve mistralai/Mistral-7B-v0.1 \
|
||||
--logging-config.formatter json
|
||||
```
|
||||
|
||||
The built-in JSON output includes `asctime`, `levelname`, `name`,
|
||||
`processName`, `process`, `message`, and `vllm_process_name`. `processName` is
|
||||
Python's process name, `vllm_process_name` identifies vLLM's logical process
|
||||
(including worker ranks where applicable), and `process` is the
|
||||
operating-system PID.
|
||||
|
||||
The built-in profile formats records emitted through vLLM's Python loggers and
|
||||
Uvicorn's server and access loggers when using `vllm serve`. It does not convert
|
||||
direct writes to `stdout` or `stderr`. Use a custom configuration to format
|
||||
other loggers.
|
||||
|
||||
`--log-level` is a shortcut for `--logging-config.log_level` and takes
|
||||
precedence if both are supplied. It sets the level of vLLM's built-in logging
|
||||
configuration. `--log-config-file` is deprecated and will be removed in
|
||||
@@ -41,11 +61,11 @@ The custom logging configuration file must be JSON following Python's [logging
|
||||
configuration dictionary
|
||||
schema](https://docs.python.org/3/library/logging.config.html#dictionary-schema-details).
|
||||
|
||||
!!! note "Custom configurations override `--log-level`"
|
||||
!!! note "Custom configurations override built-in formatter settings"
|
||||
When `pylogging_config_file` is set, vLLM loads that JSON file and replaces
|
||||
its built-in `dictConfig`; it does not merge the two. Therefore,
|
||||
`--log-level` applies only when no custom configuration file is provided.
|
||||
Set logger and handler levels in the custom file itself. vLLM applies the
|
||||
its built-in `dictConfig`; it does not merge the two. Therefore, `log_level`
|
||||
and `formatter` apply only when no custom configuration file is provided.
|
||||
Set levels and formatters in the custom file itself. vLLM applies the
|
||||
resolved configuration in its child processes as well.
|
||||
|
||||
## Environment variables
|
||||
@@ -62,10 +82,11 @@ for those settings.
|
||||
|
||||
### Example 1: Customize vLLM root logger
|
||||
|
||||
For this example, we will customize the vLLM root logger to use
|
||||
For more control over fields and handlers, configure the vLLM root logger and
|
||||
Uvicorn's server and access loggers with
|
||||
[`python-json-logger`](https://github.com/nhairs/python-json-logger)
|
||||
(which is part of the container image) to log to
|
||||
STDOUT of the console in JSON format with a log level of `INFO`.
|
||||
(which is part of the container image) to log to STDOUT in JSON format at
|
||||
`INFO` level.
|
||||
|
||||
To begin, first, create an appropriate JSON logging configuration file:
|
||||
|
||||
@@ -92,8 +113,24 @@ To begin, first, create an appropriate JSON logging configuration file:
|
||||
"handlers": ["console"],
|
||||
"level": "INFO",
|
||||
"propagate": false
|
||||
},
|
||||
"uvicorn": {
|
||||
"handlers": ["console"],
|
||||
"level": "INFO",
|
||||
"propagate": false
|
||||
},
|
||||
"uvicorn.error": {
|
||||
"handlers": ["console"],
|
||||
"level": "INFO",
|
||||
"propagate": false
|
||||
},
|
||||
"uvicorn.access": {
|
||||
"handlers": ["console"],
|
||||
"level": "INFO",
|
||||
"propagate": false
|
||||
}
|
||||
},
|
||||
"disable_existing_loggers": false,
|
||||
"version": 1
|
||||
}
|
||||
```
|
||||
@@ -105,21 +142,20 @@ vllm serve mistralai/Mistral-7B-v0.1 --max-model-len 2048 \
|
||||
--logging-config.pylogging_config_file /path/to/logging_config.json
|
||||
```
|
||||
|
||||
Each vLLM log record is one JSON object and includes `vllm_process_name`, which
|
||||
identifies vLLM's logical process (including worker ranks where applicable).
|
||||
The standard `process` record attribute contains the operating-system PID.
|
||||
vLLM avoids altering `stdout` or `stderr`, so no text is prepended to JSON
|
||||
log records.
|
||||
With this custom configuration, each vLLM and Uvicorn log record is one JSON
|
||||
object. The example selects `vllm_process_name` and `process` explicitly; both
|
||||
fields are also present in the built-in JSON output described above. vLLM
|
||||
avoids altering `stdout` or `stderr`, so no text is prepended to JSON log
|
||||
records.
|
||||
|
||||
This applies to records emitted through the configured Python loggers. A JSON
|
||||
formatter cannot convert unrelated output into JSON, such as a third-party
|
||||
library writing directly to `stdout` or `stderr`; configure or route such
|
||||
output separately when a consumer requires every collected line to be JSON.
|
||||
|
||||
When serving, `VLLM_LOGGING_CONFIG_PATH` is also used as Uvicorn's logging
|
||||
configuration. This example configures only `vllm`; configure `uvicorn`,
|
||||
`uvicorn.error`, and `uvicorn.access` with a JSON handler when those server
|
||||
logs must also be structured.
|
||||
When serving, vLLM also passes this file to Uvicorn as its logging
|
||||
configuration. The `uvicorn`, `uvicorn.error`, and `uvicorn.access` entries
|
||||
format its server and access logs as JSON.
|
||||
|
||||
### Example 2: Silence a particular vLLM logger
|
||||
|
||||
|
||||
@@ -28,6 +28,14 @@ A special case of wrapping a request-level logits processor where the processor
|
||||
python examples/features/logits_processor/custom_req_init.py
|
||||
```
|
||||
|
||||
### `dry.py` — DRY repetition penalty (Model Runner V2)
|
||||
|
||||
Implements the DRY (Don't Repeat Yourself) repetition penalty, ported from llama.cpp, as `DryState`, a custom logits processor for the Model Runner V2 interface. Requests turn it on through `SamplingParams.extra_args`, for example `{"dry_multiplier": 0.8}`. The example sends one prompt twice in one batch, once without DRY and once with it, and prints both outputs. `tests/v1/sample/test_dry.py` imports the processor from this file.
|
||||
|
||||
```bash
|
||||
python examples/features/logits_processor/dry.py
|
||||
```
|
||||
|
||||
## Key Concepts
|
||||
|
||||
- **Batch-level vs. request-level**: vLLM processes logits at the batch level for efficiency. If you have a per-request processor, you need to wrap it using the patterns shown in `custom_req.py` and `custom_req_init.py`.
|
||||
|
||||
@@ -0,0 +1,827 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
"""This example implements the DRY (Don't Repeat Yourself) repetition penalty as
|
||||
a Model Runner V2 custom logits processor, ``DryState``.
|
||||
|
||||
A token that would extend a repeat of at least ``allowed_length`` tokens loses
|
||||
``multiplier * base ** (repeat_length - allowed_length)`` from its logit, with the
|
||||
exponent clamped as in llama.cpp.
|
||||
Parameter names and matching semantics follow llama.cpp's
|
||||
``llama_sampler_dry_apply``. llama.cpp's version is itself ported from pi6am's
|
||||
koboldcpp implementation of p-e-w's scheme for text-generation-webui.
|
||||
|
||||
Pass the class to ``LLM`` and configure it per request through ``extra_args``::
|
||||
|
||||
LLM(..., logits_processors=[DryState])
|
||||
SamplingParams(extra_args={"dry_multiplier": 0.8})
|
||||
|
||||
The recognized keys are ``dry_multiplier`` (0.0, off), ``dry_base`` (1.75),
|
||||
``dry_allowed_length`` (2), ``dry_penalty_last_n`` (-1, the whole context)
|
||||
and ``dry_sequence_breakers``. This processor claims the ``dry_*`` namespace,
|
||||
so any other ``dry_*`` key fails the request. Speculative decoding is not
|
||||
supported and is refused at construction.
|
||||
|
||||
Run the example with::
|
||||
|
||||
python examples/features/logits_processor/dry.py
|
||||
|
||||
It sends one prompt twice in one batch, once without DRY and once with it, and
|
||||
prints both outputs, yielding an output similar to that shown below:
|
||||
|
||||
Generated Outputs:
|
||||
------------------------------------------------------------
|
||||
Prompt: 'The future of AI is', without DRY
|
||||
Output: ' in the hands of the people.\n\nThe future of AI is in the hands of
|
||||
the people.\n\nThe future of AI is in the hands of the
|
||||
people.\n\nThe future of AI is in the hands of the people.\n\nThe
|
||||
future of AI is in the hands of the people.\n'
|
||||
------------------------------------------------------------
|
||||
Prompt: 'The future of AI is', with DRY
|
||||
Output: ' in the hands of the people.\n\nThe future of AI is the future of
|
||||
the human race.\n\nThe future of AI is a future of the human race,
|
||||
and the future of humanity.\n\nThe future of AI is an AI that is
|
||||
capable of making decisions that are not based on human
|
||||
intelligence.'
|
||||
------------------------------------------------------------
|
||||
"""
|
||||
|
||||
import math
|
||||
import weakref
|
||||
from typing import TYPE_CHECKING, Any, NamedTuple
|
||||
|
||||
import numpy as np
|
||||
import torch
|
||||
|
||||
from vllm.logger import init_logger
|
||||
from vllm.sampling_params import SamplingParams
|
||||
from vllm.utils.gpu_sync_debug import gpu_sync_allowed
|
||||
from vllm.utils.torch_utils import async_tensor_h2d
|
||||
from vllm.v1.worker.gpu.sample.logits_processor import (
|
||||
LogitsContext,
|
||||
LogitsProcessor,
|
||||
LogitsProcRequestState,
|
||||
)
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from vllm.config import VllmConfig
|
||||
|
||||
logger = init_logger(__name__)
|
||||
|
||||
_J_BUDGET = 2048
|
||||
# Run lengths feed an int16 sum, so they must stay below 2**15.
|
||||
assert _J_BUDGET < 2**15, "_J_BUDGET must fit the int16 run-length sum"
|
||||
|
||||
# Byte budget for the per-chunk transients of the match scan. The gather
|
||||
# W32[:, idx1] materializes [R, chunk, J] int32 (4 B/elem) before the
|
||||
# comparison reduces it to bool; with the masks, the int8 cumprod and
|
||||
# the int16 row sums, the marginal transient cost is ~8 B/elem. 24 keeps
|
||||
# headroom for the fixed R*vocab penalty accumulator floor and allocator slack.
|
||||
_CHUNK_BYTE_BUDGET = 256 * 1024 * 1024
|
||||
_CHUNK_PEAK_BYTES_PER_ELEM = 24
|
||||
|
||||
# llama.cpp's FLOAT_MAX_LOG (src/llama-sampler.cpp): ln(float32 max).
|
||||
_FLOAT_MAX_LOG = 88.7228391
|
||||
|
||||
# Breakers are truncated to this many code points. llama.cpp cuts at this many
|
||||
# bytes (llama_sampler_init_dry), which keeps fewer characters of a non-ASCII
|
||||
# breaker.
|
||||
_MAX_BREAKER_CHAR_LEN = 40
|
||||
|
||||
# Upper bound on distinct breaker sets cached per tokenizer.
|
||||
_MAX_CACHED_BREAKER_SETS = 64
|
||||
|
||||
STR_SPEC_DEC_REJECTS_DRY = (
|
||||
"The DRY logits processor is not supported when speculative decoding is enabled."
|
||||
)
|
||||
|
||||
DEFAULT_DRY_SEQUENCE_BREAKERS = ("\n", ":", '"', "*")
|
||||
"""llama.cpp's default DRY sequence breakers."""
|
||||
|
||||
MAX_DRY_SEQUENCE_BREAKERS = 64
|
||||
"""Upper bound on the dry_sequence_breakers list length. Each multi-character
|
||||
breaker in an uncached set scans the vocabulary."""
|
||||
|
||||
_DRY_INT_MAX = 2**31 - 1
|
||||
"""Upper bound on the integral DRY parameters, matching llama-server's
|
||||
INT32_MAX cap. The state below stores them in an int64 numpy array, where
|
||||
anything at or above 2**63 raises OverflowError inside execute_model and
|
||||
takes the engine down."""
|
||||
|
||||
_MAX_CACHED_BREAKER_MASKS = 64
|
||||
"""Upper bound on distinct breaker sets holding a device mask, mirroring
|
||||
_MAX_CACHED_BREAKER_SETS on the host side."""
|
||||
|
||||
_WARMUP_WINDOW = 8
|
||||
"""Window length of the dry_core call in DryState.__init__, long enough to
|
||||
hold a match at the default dry_allowed_length."""
|
||||
|
||||
_DRY_DEFAULTS: dict[str, Any] = {
|
||||
"dry_multiplier": 0.0,
|
||||
"dry_base": 1.75,
|
||||
"dry_allowed_length": 2,
|
||||
"dry_penalty_last_n": -1, # whole context; llama.cpp's default is 64
|
||||
"dry_sequence_breakers": DEFAULT_DRY_SEQUENCE_BREAKERS,
|
||||
}
|
||||
"""Recognized extra_args keys and their defaults."""
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# The match computation
|
||||
# ---------------------------------------------------------------------------
|
||||
# ``_dry_penalties`` is a sequential port of llama.cpp's Z-algorithm, used as the
|
||||
# reference. ``dry_core`` is the vectorized form: the comparisons that determine
|
||||
# the match length ending at position ``i`` lie along one diagonal, so a
|
||||
# cumprod-sum along ``j`` of an ``[R, K, J]`` tensor gives every match length
|
||||
# without a sequential scan. Capping ``J`` at ``allowed_length + max_exponent``
|
||||
# is exact, because the exponent clamp maps every longer match to the same
|
||||
# penalty. Requests whose cap is unusable (``max_exponent == 0``, i.e.
|
||||
# ``base <= 1.000001``, or a cap beyond ``_J_BUDGET``) use the sequential form.
|
||||
|
||||
|
||||
def max_exponent(base: float) -> int:
|
||||
"""Exponent clamp, mirroring llama.cpp bit-for-bit.
|
||||
|
||||
llama.cpp computes ``FLOAT_MAX_LOG / std::log(dry_base)`` entirely in
|
||||
float32. Computing the quotient in float64 lands on the wrong side of
|
||||
the integer truncation for some bases: at ``base=2.0`` float64 gives
|
||||
127.99999998 -> 127 while float32 gives exactly 128.0 -> 128.
|
||||
"""
|
||||
if base <= 1.000001:
|
||||
return 0
|
||||
return int(np.float32(_FLOAT_MAX_LOG) / np.log(np.float32(base)))
|
||||
|
||||
|
||||
def _dry_penalties(
|
||||
window: list[int],
|
||||
breakers: frozenset[int],
|
||||
multiplier: float,
|
||||
base: float,
|
||||
allowed_length: int,
|
||||
max_exp: int,
|
||||
) -> dict[int, float]:
|
||||
"""Compute DRY penalties for one request.
|
||||
|
||||
Port of the scan in ``llama_sampler_dry_apply``: a reverse-direction
|
||||
Z-algorithm finds, for each position, the length of the match between
|
||||
the window suffix and the sequence ending at that position; each
|
||||
match's follower token is charged the penalty for the longest repeat
|
||||
it would extend.
|
||||
|
||||
Returns a dict mapping token id -> penalty to subtract from its logit.
|
||||
"""
|
||||
m = len(window)
|
||||
|
||||
def rat(i: int) -> int: # reverse access: rat(0) == last token
|
||||
return window[m - 1 - i]
|
||||
|
||||
# Step 1: the nearest breaker from the end caps the match length.
|
||||
rep_limit = m
|
||||
for i in range(m):
|
||||
if rat(i) in breakers:
|
||||
rep_limit = i
|
||||
break
|
||||
if rep_limit < allowed_length:
|
||||
return {}
|
||||
|
||||
# Step 2: reverse-direction Z-algorithm -> per-position repeat counts
|
||||
# (forward-indexed into ``window`` via cnt[last - k]).
|
||||
cnt = [0] * m
|
||||
last = m - 1
|
||||
lt = rt = 0
|
||||
for k in range(1, m):
|
||||
if k > rt:
|
||||
# Outside the current Z-box: extend naively.
|
||||
z = 0
|
||||
while z + k < m and rat(z) == rat(z + k):
|
||||
z += 1
|
||||
cnt[last - k] = min(z, rep_limit)
|
||||
if z > 0:
|
||||
lt, rt = k, k + z - 1
|
||||
else:
|
||||
p = k - lt
|
||||
right_part_len = rt - k + 1
|
||||
if cnt[last - p] < right_part_len:
|
||||
# Fully inside the Z-box: copy.
|
||||
cnt[last - k] = min(cnt[last - p], rep_limit)
|
||||
else:
|
||||
# Touches the right edge: extend past it.
|
||||
j = rt + 1
|
||||
while j < m and rat(j) == rat(j - k):
|
||||
j += 1
|
||||
cnt[last - k] = min(j - k, rep_limit)
|
||||
lt, rt = k, j - 1
|
||||
|
||||
# Step 3: map each repeat's follower token to the longest repeat that
|
||||
# it would extend.
|
||||
max_token_repeat: dict[int, int] = {}
|
||||
for i in range(m - 1):
|
||||
repeat_len = cnt[i]
|
||||
if repeat_len >= allowed_length:
|
||||
tok = window[i + 1]
|
||||
if max_token_repeat.get(tok, -1) < repeat_len:
|
||||
max_token_repeat[tok] = repeat_len
|
||||
|
||||
# Step 4: exponential penalty, exponent clamped for float32 safety.
|
||||
# Breaker tokens are never penalized. A value overflowing float32
|
||||
# saturates the logit to -inf downstream, as in llama.cpp.
|
||||
penalties: dict[int, float] = {}
|
||||
for tok, repeat_len in max_token_repeat.items():
|
||||
if tok in breakers:
|
||||
continue
|
||||
exponent = repeat_len - allowed_length
|
||||
if max_exp and exponent > max_exp:
|
||||
exponent = max_exp
|
||||
penalties[tok] = multiplier * (base**exponent)
|
||||
return penalties
|
||||
|
||||
|
||||
def dry_core(
|
||||
logits: torch.Tensor,
|
||||
row_idx: torch.Tensor,
|
||||
W: torch.Tensor,
|
||||
n_r: torch.Tensor,
|
||||
allowed: torch.Tensor,
|
||||
max_exp: torch.Tensor,
|
||||
mult: torch.Tensor,
|
||||
base: torch.Tensor,
|
||||
breaker_masks: list[torch.Tensor | None],
|
||||
j_budget: int,
|
||||
) -> torch.Tensor:
|
||||
"""Apply DRY penalties in place given batched window tensors.
|
||||
|
||||
The tensor bounds below are not checked, to avoid a sync.
|
||||
|
||||
Args:
|
||||
logits: [B, vocab] float tensor, modified in place.
|
||||
row_idx: [R] int64, row of ``logits`` for each DRY request.
|
||||
W: [R, N] int64 windows, right-aligned (window tokens occupy the
|
||||
trailing ``n_r`` columns; leading columns are ignored via ``n_r``
|
||||
masks, whatever they hold).
|
||||
n_r: [R] int64 window lengths.
|
||||
allowed: [R] int64 per-request allowed length.
|
||||
max_exp: [R] int64 per-request exponent ceiling, > 0.
|
||||
mult: [R] float32 per-request multiplier, >= 0, already rounded
|
||||
through float32, as llama.cpp stores it.
|
||||
base: [R] float32 per-request base, >= 1, rounded the same way.
|
||||
breaker_masks: per-request [vocab] bool masks (or None for no
|
||||
breakers). ``vocab`` must equal ``logits.shape[-1]``.
|
||||
j_budget: max(allowed + max_exp), <= _J_BUDGET, computed by the caller on
|
||||
the host.
|
||||
|
||||
"""
|
||||
device = logits.device
|
||||
vocab = logits.shape[-1]
|
||||
R, N = W.shape
|
||||
|
||||
# j_budget arrives as an unchecked host int; this check fails loudly rather
|
||||
# than wrapping an int16 run length. The routing predicate in DryState.apply
|
||||
# is what actually bounds it.
|
||||
if j_budget > _J_BUDGET:
|
||||
raise ValueError(f"j_budget {j_budget} over _J_BUDGET {_J_BUDGET}")
|
||||
# base >= 1 and mult >= 0 are preconditions too (see the amax comment below).
|
||||
# They live in device tensors and reading them back would sync; use_dry()
|
||||
# and validate_params enforce them instead.
|
||||
|
||||
# rep_limit: distance from the end of the nearest breaker (llama.cpp step 1).
|
||||
# A breaker at column c is j = N-1-c tokens from the end.
|
||||
rep_limit = n_r.clone()
|
||||
# bool(bm.any()) would be a per-request host sync; the None check says the same.
|
||||
any_breakers = any(bm is not None for bm in breaker_masks)
|
||||
bmask = None
|
||||
if any_breakers:
|
||||
# Stacked once, used twice: nearest-breaker search here, penalty zeroing
|
||||
# after the scan. The masks are cached and resident (DryState._breaker_masks),
|
||||
# so the stack costs one byte per (row, vocab) entry.
|
||||
bmask = torch.stack(
|
||||
[
|
||||
bm
|
||||
if bm is not None
|
||||
else torch.zeros(vocab, dtype=torch.bool, device=device)
|
||||
for bm in breaker_masks
|
||||
]
|
||||
)
|
||||
Bwin = bmask.gather(1, W.clamp(min=0))
|
||||
valid_cols = torch.arange(N, device=device)[None, :] >= (N - n_r)[:, None]
|
||||
Bwin &= valid_cols
|
||||
has_breaker = Bwin.any(dim=1)
|
||||
# max breaker column -> nearest to the end.
|
||||
max_col = torch.where(
|
||||
has_breaker,
|
||||
(Bwin * torch.arange(1, N + 1, device=device)[None, :]).max(dim=1).values
|
||||
- 1,
|
||||
torch.zeros_like(n_r),
|
||||
)
|
||||
rep_limit = torch.where(has_breaker, (N - 1) - max_col, n_r)
|
||||
|
||||
# llama.cpp: if rep_limit < allowed_length, the request produces nothing.
|
||||
active = rep_limit >= allowed
|
||||
|
||||
# Token ids fit int32, and gathering int32 halves the dominant per-chunk transient.
|
||||
W32 = W.to(torch.int32)
|
||||
# J is a tensor shape, so it must be known host-side; the caller computes it
|
||||
# from numpy (see DryState.apply) to avoid a per-step device readback.
|
||||
J = max(1, min(j_budget, N))
|
||||
idx2 = torch.arange(N - 1, N - 1 - J, -1, device=device) # [J]
|
||||
suffix = W32.gather(1, idx2.expand(R, J)) # [R, J]
|
||||
K = N - 1 # offsets 1..N-1
|
||||
chunk = max(1, _CHUNK_BYTE_BUDGET // (_CHUNK_PEAK_BYTES_PER_ELEM * max(1, R * J)))
|
||||
|
||||
# amax keeps one penalty per token, for its longest match, however the offsets
|
||||
# are chunked, as long as the penalty does not fall as the match grows
|
||||
# (base >= 1, mult >= 0).
|
||||
pen = torch.zeros(R * vocab + 1, dtype=torch.float32, device=device)
|
||||
|
||||
for k0 in range(1, K + 1, chunk):
|
||||
k1 = min(k0 + chunk, K + 1)
|
||||
ks = torch.arange(k0, k1, device=device) # [C]
|
||||
C = ks.shape[0]
|
||||
# idx1[c, j] = N-1-j-k ; invalid (out of window) entries masked.
|
||||
idx1 = idx2[None, :] - ks[:, None] # [C, J]
|
||||
invalid = idx1 < (N - n_r)[:, None, None] # [R, C, J]
|
||||
eq = W32[:, idx1.clamp(min=0)] == suffix[:, None, :] # [R, C, J]
|
||||
eq &= ~invalid
|
||||
del invalid
|
||||
# Run length of leading True along j = the match length. Explicit dtypes avoid
|
||||
# int64 intermediate copies; values are 0/1 and runs are <= J <= _J_BUDGET,
|
||||
# so int8/int16 are exact.
|
||||
L = eq.cumprod(dim=2, dtype=torch.int8).sum(dim=2, dtype=torch.int16)
|
||||
del eq
|
||||
# Do not narrow rep_limit to L's int16; window lengths can exceed it.
|
||||
L = torch.minimum(L, rep_limit[:, None])
|
||||
|
||||
# Follower token of offset k lives at column N-k; the offset counts only
|
||||
# while position i = N-1-k is inside the window (k <= n_r - 1).
|
||||
valid_k = ks[None, :] <= (n_r - 1)[:, None] # [R, C]
|
||||
charge = (allowed[:, None] <= L) & valid_k & active[:, None]
|
||||
# No `if charge.any()` and no boolean indexing: both sync the host. Uncharged
|
||||
# entries scatter to a trash slot one past the accumulator end, never read.
|
||||
followers = W.gather(1, (N - ks).clamp(max=N - 1).expand(R, C))
|
||||
rows = torch.arange(R, device=device)[:, None].expand(R, C) * vocab
|
||||
flat = rows + followers.clamp(min=0)
|
||||
# A Python int, not a device tensor: torch.tensor(x, device=...) here would be a
|
||||
# pageable host-to-device copy, which is itself a synchronization.
|
||||
flat = torch.where(charge, flat, R * vocab)
|
||||
# L takes rep_limit's dtype from the minimum above. The cast keeps the
|
||||
# exponent int64 whatever dtype n_r, allowed and max_exp arrive in.
|
||||
exponent = torch.minimum(L.to(torch.int64) - allowed[:, None], max_exp[:, None])
|
||||
# float64 pow, as llama.cpp's std::pow(float, int); a float32 pow overflows
|
||||
# early (0.8 * 2**128 is finite in float32).
|
||||
p_chunk = mult[:, None].double() * torch.pow(
|
||||
base[:, None].double(), exponent.to(torch.float64)
|
||||
)
|
||||
# No where() on p_chunk: uncharged entries went to the trash slot and are
|
||||
# never read.
|
||||
pen.scatter_reduce_(
|
||||
0,
|
||||
flat.reshape(-1),
|
||||
p_chunk.to(torch.float32).reshape(-1),
|
||||
reduce="amax",
|
||||
include_self=True,
|
||||
)
|
||||
|
||||
# Of the penalty terms, only the penalty itself is held at [R, vocab] (4 B/entry);
|
||||
# the exponent, its float64 cast, the pow result and the product stay at [R, C]
|
||||
# inside the loop.
|
||||
# Breakers zero the penalty rather than the logit, so a breaker keeps its value.
|
||||
# The narrowing to logits.dtype below is a no-op under vLLM's sampler, which
|
||||
# always passes float32 logits.
|
||||
pen2 = pen[:-1].view(R, vocab)
|
||||
if bmask is not None:
|
||||
pen2.masked_fill_(bmask, 0.0)
|
||||
# index_add_ avoids the extra [R, vocab] gather+copy of `logits[row_idx] -= pen2`.
|
||||
pen2.neg_()
|
||||
logits.index_add_(
|
||||
0, row_idx, pen2 if logits.dtype == pen2.dtype else pen2.to(logits.dtype)
|
||||
)
|
||||
return logits
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Breaker resolution
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class _VocabIndex(NamedTuple):
|
||||
"""A tokenizer's decoded vocabulary, plus a character index into it.
|
||||
|
||||
``char_ids`` maps every character in ``texts`` to the ids of the tokens
|
||||
whose text contains it.
|
||||
"""
|
||||
|
||||
texts: list[str]
|
||||
char_ids: dict[str, list[int]]
|
||||
|
||||
|
||||
# Per-tokenizer caches, weakly keyed so tokenizers can be collected:
|
||||
# tokenizer -> its decoded vocabulary and character index, and
|
||||
# tokenizer -> {breaker string tuple -> resolved breaker ids}.
|
||||
_BreakerIdsPerTokenizer = dict[tuple[str, ...], list[int]]
|
||||
_vocab_index_cache: "weakref.WeakKeyDictionary[Any, _VocabIndex]" = (
|
||||
weakref.WeakKeyDictionary()
|
||||
)
|
||||
_breaker_ids_cache: "weakref.WeakKeyDictionary[Any, _BreakerIdsPerTokenizer]" = (
|
||||
weakref.WeakKeyDictionary()
|
||||
)
|
||||
|
||||
|
||||
def _vocab_index(tokenizer: Any) -> _VocabIndex:
|
||||
"""Decode every token of ``tokenizer`` once and index its characters."""
|
||||
index = _vocab_index_cache.get(tokenizer)
|
||||
if index is None:
|
||||
# Include added tokens above vocab_size, which llama.cpp's scan also covers.
|
||||
n_ids = getattr(tokenizer, "max_token_id", tokenizer.vocab_size - 1) + 1
|
||||
texts = tokenizer.batch_decode([[i] for i in range(n_ids)])
|
||||
char_ids: dict[str, list[int]] = {}
|
||||
for i, text in enumerate(texts):
|
||||
for char in set(text):
|
||||
char_ids.setdefault(char, []).append(i)
|
||||
index = _VocabIndex(texts, char_ids)
|
||||
_vocab_index_cache[tokenizer] = index
|
||||
return index
|
||||
|
||||
|
||||
def resolve_dry_breakers(tokenizer: Any, breaker_strs: tuple[str, ...]) -> list[int]:
|
||||
"""Resolve breaker strings to the ids of every token whose text contains one.
|
||||
|
||||
This is llama.cpp's ``get_overlapping_token_sequences`` rule, without its
|
||||
multi-token restart sequences. Resolution runs in ``add_request``, inside
|
||||
``execute_model``, so a single-character breaker (llama.cpp's defaults and
|
||||
most custom sets) resolves by lookup in the character index instead of a
|
||||
vocabulary scan. Results are cached per (tokenizer, breaker set).
|
||||
"""
|
||||
breaker_strs = tuple(s[:_MAX_BREAKER_CHAR_LEN] for s in breaker_strs if s)
|
||||
if not breaker_strs:
|
||||
return []
|
||||
per_tok = _breaker_ids_cache.setdefault(tokenizer, {})
|
||||
cached = per_tok.get(breaker_strs)
|
||||
if cached is not None:
|
||||
return list(cached)
|
||||
|
||||
index = _vocab_index(tokenizer)
|
||||
ids: set[int] = set()
|
||||
for s in breaker_strs:
|
||||
if len(s) == 1:
|
||||
ids.update(index.char_ids.get(s, ()))
|
||||
else:
|
||||
ids.update(i for i, text in enumerate(index.texts) if s in text)
|
||||
result = sorted(ids)
|
||||
if len(per_tok) >= _MAX_CACHED_BREAKER_SETS:
|
||||
# Evict the oldest entry (dict preserves insertion order).
|
||||
per_tok.pop(next(iter(per_tok)))
|
||||
per_tok[breaker_strs] = result
|
||||
return list(result)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# The logits processor
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _dry_args(sampling_params: SamplingParams) -> dict[str, Any]:
|
||||
"""Read the request's DRY arguments, filling in the defaults.
|
||||
|
||||
A key present with value None counts as unset, matching how
|
||||
``SamplingParams`` coerces its own None-valued arguments.
|
||||
"""
|
||||
extra = sampling_params.extra_args or {}
|
||||
return {
|
||||
name: default if extra.get(name) is None else extra[name]
|
||||
for name, default in _DRY_DEFAULTS.items()
|
||||
}
|
||||
|
||||
|
||||
def use_dry(multiplier: float, base: float, penalty_last_n: int) -> bool:
|
||||
"""Whether DRY applies, by llama.cpp's gate in llama_sampler_dry_apply."""
|
||||
return bool(multiplier) and base >= 1.0 and penalty_last_n != 0
|
||||
|
||||
|
||||
class DryState(LogitsProcessor):
|
||||
def __init__(self, vllm_config: "VllmConfig", req_states: LogitsProcRequestState):
|
||||
# Refuse speculative decoding here, since admission cannot see its config.
|
||||
if vllm_config.speculative_config is not None:
|
||||
raise ValueError(STR_SPEC_DEC_REJECTS_DRY)
|
||||
self.req_states = req_states
|
||||
max_num_reqs = req_states.max_num_reqs
|
||||
self.vocab_size = req_states.vocab_size
|
||||
self.device = req_states.device
|
||||
|
||||
# float32, to round multiplier and base as llama.cpp's float members do.
|
||||
self.multiplier = np.zeros(max_num_reqs, dtype=np.float32)
|
||||
self.base = np.zeros(max_num_reqs, dtype=np.float32)
|
||||
self.allowed_length = np.zeros(max_num_reqs, dtype=np.int64)
|
||||
self.penalty_last_n = np.zeros(max_num_reqs, dtype=np.int64)
|
||||
self.max_exponent = np.zeros(max_num_reqs, dtype=np.int64)
|
||||
self.use_dry = np.zeros(max_num_reqs, dtype=bool)
|
||||
|
||||
# req_idx -> its breaker set, and breaker set -> [vocab] bool device
|
||||
# mask shared by every request that asked for the same breakers.
|
||||
self.breaker_ids: dict[int, frozenset[int]] = {}
|
||||
self._breaker_masks: dict[frozenset[int], torch.Tensor] = {}
|
||||
|
||||
self._warned_unresolved = False
|
||||
|
||||
# Deferred import: the tokenizer registry pulls in transformers, and
|
||||
# the frontend imports this module only to validate params.
|
||||
from vllm.tokenizers import cached_tokenizer_from_config
|
||||
|
||||
# None under skip_tokenizer_init.
|
||||
self._tokenizer = cached_tokenizer_from_config(vllm_config.model_config)
|
||||
# Resolve the default set here to keep the vocabulary decode out of
|
||||
# execute_model, where add_request runs.
|
||||
self._default_breaker_ids = self._resolve_breakers(
|
||||
DEFAULT_DRY_SEQUENCE_BREAKERS
|
||||
)
|
||||
self._warm_up_dry_core()
|
||||
|
||||
def _warm_up_dry_core(self) -> None:
|
||||
"""Load dry_core's CUDA kernels before a request needs them.
|
||||
|
||||
The operands match the dtypes and ranks apply passes, and include a breaker
|
||||
mask so that the breaker path's kernels load too.
|
||||
"""
|
||||
vocab = self.vocab_size
|
||||
# The breaker takes the last id, which must not be the window's token 0.
|
||||
if vocab < 2:
|
||||
return
|
||||
device = self.device
|
||||
allowed = _DRY_DEFAULTS["dry_allowed_length"]
|
||||
base = _DRY_DEFAULTS["dry_base"]
|
||||
max_exp = max_exponent(base)
|
||||
|
||||
def col(value: float, dtype: torch.dtype) -> torch.Tensor:
|
||||
return torch.full((1,), value, dtype=dtype, device=device)
|
||||
|
||||
breakers = torch.zeros(vocab, dtype=torch.bool, device=device)
|
||||
breakers.narrow(0, vocab - 1, 1).fill_(True)
|
||||
dry_core(
|
||||
torch.zeros(1, vocab, dtype=torch.float32, device=device),
|
||||
row_idx=col(0, torch.int64),
|
||||
W=torch.zeros(1, _WARMUP_WINDOW, dtype=torch.int64, device=device),
|
||||
n_r=col(_WARMUP_WINDOW, torch.int64),
|
||||
allowed=col(allowed, torch.int64),
|
||||
max_exp=col(max_exp, torch.int64),
|
||||
mult=col(1.0, torch.float32),
|
||||
base=col(base, torch.float32),
|
||||
breaker_masks=[breakers],
|
||||
j_budget=allowed + max_exp,
|
||||
)
|
||||
|
||||
@classmethod
|
||||
def validate_params(cls, sampling_params: SamplingParams) -> None:
|
||||
"""Check the ``dry_*`` keys of ``extra_args`` at request admission.
|
||||
|
||||
Raises:
|
||||
ValueError: on an unknown or out-of-range DRY argument.
|
||||
|
||||
"""
|
||||
extra = sampling_params.extra_args or {}
|
||||
unknown = sorted(
|
||||
key for key in extra if key.startswith("dry_") and key not in _DRY_DEFAULTS
|
||||
)
|
||||
if unknown:
|
||||
# A misspelled key would otherwise be ignored silently.
|
||||
raise ValueError(
|
||||
f"Unknown dry_* extra_args: {', '.join(unknown)}. "
|
||||
f"Supported keys: {', '.join(_DRY_DEFAULTS)}."
|
||||
)
|
||||
|
||||
args = _dry_args(sampling_params)
|
||||
for name in (
|
||||
"dry_multiplier",
|
||||
"dry_base",
|
||||
"dry_allowed_length",
|
||||
"dry_penalty_last_n",
|
||||
):
|
||||
value = args[name]
|
||||
# JSON true/false are bools, which pass isinstance(value, int).
|
||||
if isinstance(value, bool) or not isinstance(value, (int, float)):
|
||||
raise ValueError(
|
||||
f"{name} must be a number, got {type(value).__name__}."
|
||||
)
|
||||
|
||||
multiplier = args["dry_multiplier"]
|
||||
base = args["dry_base"]
|
||||
allowed_length = args["dry_allowed_length"]
|
||||
penalty_last_n = args["dry_penalty_last_n"]
|
||||
breakers = args["dry_sequence_breakers"]
|
||||
|
||||
if not math.isfinite(multiplier) or multiplier < 0.0:
|
||||
raise ValueError(
|
||||
f"dry_multiplier must be non-negative and finite, got {multiplier}."
|
||||
)
|
||||
if not math.isfinite(base) or base < 0.0:
|
||||
raise ValueError(f"dry_base must be non-negative and finite, got {base}.")
|
||||
if (
|
||||
not isinstance(allowed_length, int)
|
||||
or allowed_length < 0
|
||||
or allowed_length > _DRY_INT_MAX
|
||||
):
|
||||
raise ValueError(
|
||||
f"dry_allowed_length must be an integer in [0, {_DRY_INT_MAX}], "
|
||||
f"got {allowed_length}."
|
||||
)
|
||||
if (
|
||||
not isinstance(penalty_last_n, int)
|
||||
or penalty_last_n < -1
|
||||
or penalty_last_n > _DRY_INT_MAX
|
||||
):
|
||||
raise ValueError(
|
||||
"dry_penalty_last_n must be an integer: -1 (whole context), "
|
||||
f"0 (disable), or in [1, {_DRY_INT_MAX}], got {penalty_last_n}."
|
||||
)
|
||||
if not isinstance(breakers, (list, tuple)) or any(
|
||||
not isinstance(s, str) for s in breakers
|
||||
):
|
||||
raise ValueError(
|
||||
f"dry_sequence_breakers must be a list of strings, got {breakers!r}."
|
||||
)
|
||||
if len(breakers) > MAX_DRY_SEQUENCE_BREAKERS:
|
||||
raise ValueError(
|
||||
f"dry_sequence_breakers supports at most "
|
||||
f"{MAX_DRY_SEQUENCE_BREAKERS} entries, got {len(breakers)}."
|
||||
)
|
||||
if multiplier and 0.0 <= base < 1.0:
|
||||
# libllama has the same gate. llama-server resets such a base to its
|
||||
# default instead.
|
||||
logger.warning(
|
||||
"dry_base=%s is below 1.0, which disables DRY entirely "
|
||||
"(llama.cpp semantics), even though dry_multiplier=%s was "
|
||||
"set. No repetition penalty will be applied.",
|
||||
base,
|
||||
multiplier,
|
||||
)
|
||||
|
||||
def _resolve_breakers(self, breakers: tuple[str, ...]) -> frozenset[int]:
|
||||
if self._tokenizer is None or not breakers:
|
||||
return frozenset()
|
||||
return frozenset(resolve_dry_breakers(self._tokenizer, breakers))
|
||||
|
||||
def add_request(self, req_idx: int, sampling_params: SamplingParams) -> bool:
|
||||
args = _dry_args(sampling_params)
|
||||
multiplier = args["dry_multiplier"]
|
||||
base = args["dry_base"]
|
||||
penalty_last_n = args["dry_penalty_last_n"]
|
||||
enabled = use_dry(multiplier, base, penalty_last_n)
|
||||
self.use_dry[req_idx] = enabled
|
||||
self.breaker_ids.pop(req_idx, None)
|
||||
if not enabled:
|
||||
return False
|
||||
self.multiplier[req_idx] = multiplier
|
||||
self.base[req_idx] = base
|
||||
self.allowed_length[req_idx] = args["dry_allowed_length"]
|
||||
self.penalty_last_n[req_idx] = penalty_last_n
|
||||
self.max_exponent[req_idx] = max_exponent(float(self.base[req_idx]))
|
||||
|
||||
breakers = tuple(args["dry_sequence_breakers"])
|
||||
ids = (
|
||||
self._default_breaker_ids
|
||||
if breakers == DEFAULT_DRY_SEQUENCE_BREAKERS
|
||||
else self._resolve_breakers(breakers)
|
||||
)
|
||||
if ids:
|
||||
self.breaker_ids[req_idx] = ids
|
||||
elif breakers and self._tokenizer is None and not self._warned_unresolved:
|
||||
logger.warning(
|
||||
"DRY sequence breakers were not resolved to token ids: this "
|
||||
"engine has no tokenizer. Proceeding without breakers."
|
||||
)
|
||||
self._warned_unresolved = True
|
||||
return True
|
||||
|
||||
def _breaker_mask(self, req_idx: int) -> torch.Tensor | None:
|
||||
ids = self.breaker_ids.get(req_idx)
|
||||
if not ids:
|
||||
return None
|
||||
mask = self._breaker_masks.get(ids)
|
||||
if mask is None:
|
||||
# Built on the host to avoid a sync; once per breaker set.
|
||||
ids_np = np.fromiter(ids, dtype=np.int64, count=len(ids))
|
||||
ids_np = ids_np[ids_np < self.vocab_size]
|
||||
m_np = np.zeros(self.vocab_size, dtype=bool)
|
||||
m_np[ids_np] = True
|
||||
mask = async_tensor_h2d(m_np, self.device)
|
||||
if len(self._breaker_masks) >= _MAX_CACHED_BREAKER_MASKS:
|
||||
# Evict the oldest set. A client can send a new set with every request.
|
||||
self._breaker_masks.pop(next(iter(self._breaker_masks)))
|
||||
self._breaker_masks[ids] = mask
|
||||
return mask
|
||||
|
||||
def apply(self, logits: torch.Tensor, ctx: LogitsContext) -> torch.Tensor:
|
||||
req_indices = ctx.idx_mapping_np
|
||||
active_rows = np.flatnonzero(self.use_dry[req_indices])
|
||||
if active_rows.size == 0:
|
||||
return logits
|
||||
if logits.shape[0] != req_indices.shape[0]:
|
||||
raise RuntimeError("DRY received draft-expanded logits")
|
||||
|
||||
# host-side bound; reading positions off GPU would sync per step
|
||||
cur_len = ctx.seq_lens_upper_bound_np[active_rows].astype(np.int64)
|
||||
|
||||
reqs = req_indices[active_rows]
|
||||
last_n = self.penalty_last_n[reqs]
|
||||
window_len = np.where(last_n == -1, cur_len, np.minimum(cur_len, last_n))
|
||||
allowed = self.allowed_length[reqs]
|
||||
keep = window_len > allowed
|
||||
if not np.any(keep):
|
||||
return logits
|
||||
active_rows = active_rows[keep]
|
||||
reqs = reqs[keep]
|
||||
cur_len = cur_len[keep]
|
||||
window_len = window_len[keep]
|
||||
allowed = allowed[keep]
|
||||
max_exp = self.max_exponent[reqs]
|
||||
|
||||
# Route degenerate-clamp requests (base <= 1.000001 or oversized
|
||||
# cap) through the sequential reference implementation.
|
||||
fast = (max_exp > 0) & (allowed + max_exp <= _J_BUDGET)
|
||||
all_tokens = self.req_states.all_token_ids.gpu
|
||||
|
||||
if np.any(fast):
|
||||
f_rows = active_rows[fast]
|
||||
f_reqs = reqs[fast]
|
||||
f_len = window_len[fast]
|
||||
N = int(f_len.max())
|
||||
reqs_t = async_tensor_h2d(f_reqs, self.device)
|
||||
cur_t = async_tensor_h2d(cur_len[fast], self.device)
|
||||
j = torch.arange(N, device=self.device)
|
||||
# Right-aligned gather: column j holds token (cur_len - N + j);
|
||||
# out-of-window columns are masked inside dry_core via n_r.
|
||||
gather_idx = (cur_t[:, None] - N + j[None, :]).clamp(min=0)
|
||||
W = all_tokens[reqs_t[:, None], gather_idx].long()
|
||||
dry_core(
|
||||
logits,
|
||||
row_idx=async_tensor_h2d(f_rows, self.device),
|
||||
W=W,
|
||||
n_r=async_tensor_h2d(f_len, self.device),
|
||||
allowed=async_tensor_h2d(allowed[fast], self.device),
|
||||
max_exp=async_tensor_h2d(max_exp[fast], self.device),
|
||||
mult=async_tensor_h2d(self.multiplier[f_reqs], self.device),
|
||||
base=async_tensor_h2d(self.base[f_reqs], self.device),
|
||||
breaker_masks=[self._breaker_mask(r) for r in f_reqs],
|
||||
j_budget=int((allowed[fast] + max_exp[fast]).max()),
|
||||
)
|
||||
|
||||
# The sequential fallback copies each window to the host, an expected sync.
|
||||
slow = ~fast
|
||||
if np.any(slow):
|
||||
with gpu_sync_allowed():
|
||||
rows_list = []
|
||||
cols_list = []
|
||||
vals_list = []
|
||||
for row, req, w_len, cur in zip(
|
||||
active_rows[slow], reqs[slow], window_len[slow], cur_len[slow]
|
||||
):
|
||||
window = (
|
||||
all_tokens[int(req), int(cur) - int(w_len) : int(cur)]
|
||||
.cpu()
|
||||
.tolist()
|
||||
)
|
||||
penalties = _dry_penalties(
|
||||
window,
|
||||
self.breaker_ids.get(int(req), frozenset()),
|
||||
float(self.multiplier[req]),
|
||||
float(self.base[req]),
|
||||
int(self.allowed_length[req]),
|
||||
int(self.max_exponent[req]),
|
||||
)
|
||||
for tok, val in penalties.items():
|
||||
rows_list.append(int(row))
|
||||
cols_list.append(tok)
|
||||
vals_list.append(val)
|
||||
if rows_list:
|
||||
logits[
|
||||
torch.tensor(rows_list, dtype=torch.int64, device=self.device),
|
||||
torch.tensor(cols_list, dtype=torch.int64, device=self.device),
|
||||
] -= torch.tensor(
|
||||
vals_list, dtype=torch.float32, device=self.device
|
||||
)
|
||||
return logits
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# The demo
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def main():
|
||||
# Imported here so that loading DryState from this module does not import LLM.
|
||||
from vllm import LLM
|
||||
|
||||
prompts = ["The future of AI is"] * 2
|
||||
sampling_params_list = [
|
||||
SamplingParams(temperature=0.0, max_tokens=64),
|
||||
SamplingParams(
|
||||
temperature=0.0, max_tokens=64, extra_args={"dry_multiplier": 0.8}
|
||||
),
|
||||
]
|
||||
llm = LLM(model="facebook/opt-125m", logits_processors=[DryState])
|
||||
outputs = llm.generate(prompts, sampling_params_list)
|
||||
print("\nGenerated Outputs:\n" + "-" * 60)
|
||||
for label, output in zip(("without DRY", "with DRY"), outputs):
|
||||
print(f"Prompt: {output.prompt!r}, {label}")
|
||||
print(f"Output: {output.outputs[0].text!r}")
|
||||
print("-" * 60)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -36,10 +36,9 @@ layer when available, otherwise Triton. FlashInfer cannot serve this model (a
|
||||
batch mixes causal prefill with bidirectional denoising), and
|
||||
`--attention-backend FLASHINFER` is rejected.
|
||||
|
||||
Per-request canvas widths smaller than the served canvas require async
|
||||
scheduling. The diffusion async scheduler is selected automatically; no
|
||||
`--scheduler-cls` argument is needed. Synchronous execution supports full-width
|
||||
canvases only.
|
||||
Per-request canvas widths may be smaller than the served canvas with either
|
||||
synchronous or asynchronous scheduling. Omit `diffusion_canvas_length` to use
|
||||
the served canvas width.
|
||||
|
||||
Question types: `noul` (yes/no), `choice` with `options`, `score` with
|
||||
ordered `levels`. Each label must be a single token in the answer template,
|
||||
|
||||
-10
@@ -65,13 +65,6 @@ plugins:
|
||||
- awesome-nav
|
||||
- glightbox
|
||||
- git-revision-date-localized
|
||||
- minify:
|
||||
minify_html: true
|
||||
minify_js: true
|
||||
minify_css: true
|
||||
cache_safe: true
|
||||
js_files: [docs/mkdocs/javascript/*.js]
|
||||
css_files: [docs/mkdocs/stylesheets/*.css]
|
||||
# For API reference generation
|
||||
- api-autonav:
|
||||
modules: ["vllm"]
|
||||
@@ -98,9 +91,6 @@ plugins:
|
||||
summary: true
|
||||
inventories:
|
||||
- https://docs.python.org/3/objects.inv
|
||||
- https://typing-extensions.readthedocs.io/en/latest/objects.inv
|
||||
- https://docs.aiohttp.org/en/stable/objects.inv
|
||||
- https://pillow.readthedocs.io/en/stable/objects.inv
|
||||
- https://numpy.org/doc/stable/objects.inv
|
||||
- https://pytorch.org/docs/stable/objects.inv
|
||||
- redirects:
|
||||
|
||||
@@ -117,6 +117,11 @@ ignore_missing_imports = true
|
||||
check_untyped_defs = true
|
||||
follow_imports = "silent"
|
||||
|
||||
[[tool.mypy.overrides]]
|
||||
module = "tests.*"
|
||||
disable_error_code = ["arg-type", "assignment"]
|
||||
enable_error_code = ["method-assign"]
|
||||
|
||||
[tool.pytest.ini_options]
|
||||
markers = [
|
||||
"slow_test",
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
cmake>=3.26.1
|
||||
ninja
|
||||
packaging>=24.2
|
||||
setuptools==77.0.3 # this version can reuse CMake build dir
|
||||
setuptools==78.1.1 # this version can reuse CMake build dir; >=78.1.1 for GHSA-5rjg-fvgr-3xxf
|
||||
setuptools-scm>=8
|
||||
setuptools-rust>=1.9.0
|
||||
torch==2.13.0+cpu; platform_machine == "x86_64" or platform_machine == "s390x" or platform_machine == "aarch64"
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
cmake>=3.26.1
|
||||
ninja
|
||||
packaging>=24.2
|
||||
setuptools>=77.0.3,<81.0.0
|
||||
setuptools>=78.1.1,<81.0.0
|
||||
setuptools-scm>=8
|
||||
setuptools-rust>=1.9.0
|
||||
torch==2.13.0
|
||||
|
||||
@@ -9,7 +9,7 @@ torchaudio==2.11.0+rocm7.14.0
|
||||
triton==3.7.1+git0263a6a6.rocm7.14.0
|
||||
cmake>=3.26.1,<4
|
||||
packaging>=26.2
|
||||
setuptools>=77.0.3,<80.0.0
|
||||
setuptools>=78.1.1,<80.0.0
|
||||
setuptools-scm>=8
|
||||
setuptools-rust>=1.9.0
|
||||
wheel
|
||||
|
||||
@@ -9,7 +9,7 @@ torchaudio==2.11.0+rocm7.14.0
|
||||
triton==3.7.1+git0263a6a6
|
||||
cmake>=3.26.1,<4
|
||||
packaging>=26.2
|
||||
setuptools>=77.0.3,<80.0.0
|
||||
setuptools>=78.1.1,<80.0.0
|
||||
setuptools-scm>=8
|
||||
setuptools-rust>=1.9.0
|
||||
wheel
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
# Dependencies for building Rust artifacts through setuptools-rust.
|
||||
setuptools>=77.0.3,<81.0.0
|
||||
setuptools>=78.1.1,<81.0.0
|
||||
setuptools-scm>=9.2.0
|
||||
setuptools-rust>=1.9.0
|
||||
wheel
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
--extra-index-url https://download.pytorch.org/whl/cpu
|
||||
cmake>=3.26.1
|
||||
ninja
|
||||
setuptools>=77.0.3,<81.0.0
|
||||
setuptools>=78.1.1,<81.0.0
|
||||
setuptools-scm>=8
|
||||
setuptools-rust>=1.9.0
|
||||
torch==2.13.0+cpu
|
||||
|
||||
@@ -3,32 +3,32 @@ cachetools
|
||||
psutil
|
||||
sentencepiece # Required for LLaMA tokenizer.
|
||||
numpy
|
||||
requests >= 2.26.0
|
||||
requests >= 2.33.0
|
||||
tqdm
|
||||
blake3
|
||||
py-cpuinfo
|
||||
transformers >= 5.10.4
|
||||
transformers >= 5.16.1, < 5.18.0
|
||||
huggingface_hub >= 1.31.0 # Required for the `huggingface_hub.utils.httpx` re-export
|
||||
tokenizers >= 0.21.1 # Required for fast incremental detokenization.
|
||||
tokenizers >= 0.22.0 # Required for fast incremental detokenization.
|
||||
safetensors >= 0.6.2 # MXFP4/MXFP6 dtype support (F8_E8M0, F4) added in 0.6.0: https://github.com/huggingface/safetensors/pull/611
|
||||
protobuf >= 5.29.6, !=6.30.*, !=6.31.*, !=6.32.*, !=6.33.0.*, !=6.33.1.*, !=6.33.2.*, !=6.33.3.*, !=6.33.4.*; platform_machine != "s390x"
|
||||
protobuf == 7.36.1; platform_machine == "s390x"
|
||||
fastapi[standard] >= 0.133.0, < 0.137.0 # First version supporting Starlette 1.0; < 0.137.0 avoids route-tree change that breaks model-hosting-container-standards handler overrides.
|
||||
starlette >= 1.0.1 # CVE-2026-48710: Host header injection in < 1.0.1
|
||||
aiohttp >= 3.13.3
|
||||
aiohttp >= 3.14.3
|
||||
openai >= 2.25.0 # For Responses API with namespace tools
|
||||
pydantic >= 2.12.0
|
||||
prometheus_client >= 0.18.0
|
||||
pillow # Required for image processing
|
||||
pillow >= 12.3.0 # Required for image processing
|
||||
prometheus-fastapi-instrumentator >= 8.0.0 # v8 unblocks starlette >= 1.0
|
||||
tiktoken >= 0.6.0 # Required for DBRX tokenizer
|
||||
lm-format-enforcer == 0.11.3
|
||||
llguidance >= 1.7.0, < 1.8.0; platform_machine == "x86_64" or platform_machine == "arm64" or platform_machine == "aarch64" or platform_machine == "ppc64le" or platform_machine == "s390x"
|
||||
outlines_core == 0.2.14
|
||||
lark == 1.2.2
|
||||
xgrammar == 0.2.7; platform_machine == "x86_64" or platform_machine == "aarch64" or platform_machine == "arm64" or platform_machine == "s390x" or platform_machine == "ppc64le"
|
||||
xgrammar == 0.2.8; platform_machine == "x86_64" or platform_machine == "aarch64" or platform_machine == "arm64" or platform_machine == "s390x" or platform_machine == "ppc64le"
|
||||
typing_extensions >= 4.10
|
||||
filelock >= 3.16.1 # need to contain https://github.com/tox-dev/filelock/pull/317
|
||||
filelock >= 3.20.3 # need to contain https://github.com/tox-dev/filelock/pull/317; GHSA floor
|
||||
partial-json-parser # used for parsing partial JSON outputs
|
||||
jsonschema >= 4.23.0 # required for MiniMax M3 tool schema validation
|
||||
pyzmq >= 25.0.0
|
||||
@@ -37,7 +37,7 @@ mistral_common[image] >= 1.11.6
|
||||
opencv-python-headless >= 4.13.0 # required for video IO
|
||||
pyyaml
|
||||
six>=1.16.0; python_version > '3.11' # transitive dependency of pandas that needs to be the latest version for python 3.12
|
||||
setuptools>=77.0.3,<81.0.0; python_version > '3.11' # Setuptools is used by triton, we need to ensure a modern version is installed for 3.12+ so that it does not try to import distutils, which was removed in 3.12
|
||||
setuptools>=78.1.1,<81.0.0; python_version > '3.11' # used by triton; >=78.1.1 for GHSA-5rjg-fvgr-3xxf (83.x needed for GHSA-h35f-9h28-mq5c deferred)
|
||||
einops # Required for Qwen2-VL.
|
||||
compressed-tensors == 0.17.0 # required for compressed-tensors
|
||||
depyf==0.20.0 # required for profiling and debugging with compilation config
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user