Merge branch 'main' into patches
Signed-off-by: MOVZX <movzx@yahoo.com>
This commit is contained in:
@@ -0,0 +1,151 @@
|
||||
---
|
||||
name: model-recipes
|
||||
description: Find vLLM deployment recipes and configs for LLMs and Hugging Face models by model and hardware. Use for recipe lookup, creation, and updates.
|
||||
---
|
||||
|
||||
# Model Recipes
|
||||
|
||||
Model serving recipes are maintained in the
|
||||
[models/ directory of vllm-project/recipes](https://github.com/vllm-project/recipes/tree/main/models).
|
||||
The published catalog is at
|
||||
[recipes.vllm.ai](https://recipes.vllm.ai).
|
||||
|
||||
## Usage notes
|
||||
|
||||
1. Recipe lookup returns commands and deployment guidance without launching
|
||||
`vllm serve`.
|
||||
2. Use published configurations or the recipes repository's command generator
|
||||
for supported selections. Return commands as published, including their
|
||||
flags and parallelism settings.
|
||||
3. Recipes are starting points for deployment and may not be tuned for your
|
||||
workload. Check each recipe for its verified configurations, and use
|
||||
benchmarks of your workload to guide further tuning.
|
||||
4. To create or update a recipe, follow the
|
||||
[authoring workflow below](#create-or-update-a-recipe).
|
||||
|
||||
## Find a recipe
|
||||
|
||||
Read recipes from the published JSON on recipes.vllm.ai. It is generated from
|
||||
the source YAML with variant, hardware, and strategy settings already
|
||||
resolved; the YAML contains comments and conditional overrides, so do not use
|
||||
it for lookup. When presenting results, link the website page. Read the source
|
||||
YAML only when creating or updating a recipe.
|
||||
|
||||
Each recipe has a page at `https://recipes.vllm.ai/<org>/<repo>` and JSON at
|
||||
the same path plus `.json`. Paths use the full Hugging Face ID, preserving
|
||||
case; do not substitute `meta.slug` or a display title. For example,
|
||||
`Qwen/Qwen3.8-27B` maps to
|
||||
[recipes.vllm.ai/Qwen/Qwen3.8-27B](https://recipes.vllm.ai/Qwen/Qwen3.8-27B).
|
||||
|
||||
Recipe JSON is often over 20 KB, mostly the `guide`. Instead of printing it,
|
||||
read only what the request needs: each step below returns the paths for the
|
||||
next, so stop once the request is answered. Commands need `curl` and `jq`.
|
||||
[scripts/read_recipe.sh](scripts/read_recipe.sh) is in this skill's directory;
|
||||
the examples use its path from the vLLM repository root.
|
||||
|
||||
### Search the catalog
|
||||
|
||||
Skip this step if the exact HF ID is known. Otherwise search by model ID,
|
||||
family, or provider using case-insensitive keywords. Each
|
||||
whitespace-separated word must occur in the ID, title, or provider:
|
||||
|
||||
```bash
|
||||
curl -fsSL https://recipes.vllm.ai/models.json |
|
||||
jq --arg q 'qwen 27b' '
|
||||
($q | ascii_downcase | [scan("\\S+")]) as $words | .[] |
|
||||
([.hf_id, .title, .provider] | join(" ") | ascii_downcase) as $text |
|
||||
select(all($words[]; . as $word | $text | contains($word))) |
|
||||
{hf_id, json, url, derived_from}'
|
||||
```
|
||||
|
||||
`json` is the recipe data path; `url` is its page path. Prefix both with
|
||||
`https://recipes.vllm.ai`. A `derived_from` entry is a variant of a parent
|
||||
recipe: fetch its own `json` to get that checkpoint's command. Its page may
|
||||
belong to the parent, so link the returned `url` rather than assuming every
|
||||
checkpoint has a separate page.
|
||||
|
||||
If there is no exact match, search the family and inspect `variants` before
|
||||
concluding the recipe is missing.
|
||||
|
||||
### Summarize the recipe
|
||||
|
||||
Save the recipe JSON to a file and summarize it:
|
||||
|
||||
```bash
|
||||
read_recipe=.agents/skills/model-recipes/scripts/read_recipe.sh
|
||||
recipe=$(mktemp)
|
||||
curl -fsSL https://recipes.vllm.ai/Qwen/Qwen3.8-27B.json -o "$recipe"
|
||||
"$read_recipe" "$recipe"
|
||||
```
|
||||
|
||||
`default` is the recommended hardware, strategy, and variant; `hardware` maps
|
||||
each hardware ID to its command path; `recipe_verified_hardware` lists hardware
|
||||
where the recipe as a whole was verified end to end. For capability
|
||||
requirements, such as tool calling or text-to-video, check `features` and
|
||||
`omni_tasks`, then confirm in the guide.
|
||||
Some recipes, including Omni, lack `hardware`; skip hardware selection and
|
||||
use the guide's launch and task-specific sections.
|
||||
|
||||
### Select hardware and strategy
|
||||
|
||||
Match a hardware ID to the user's GPU and fetch its path from `hardware`,
|
||||
prefixed with `https://recipes.vllm.ai`, rather than constructing a URL. That
|
||||
response is the command object itself: single-node objects carry `command`,
|
||||
and multi-node objects carry `head_command` and `worker_commands` instead, so
|
||||
list its keys before selecting fields.
|
||||
Check `hardware_profile` to confirm that the selected configuration matches
|
||||
the user's GPU model and specifications. Its `gpu_count` describes the profile,
|
||||
not necessarily the number of GPUs used by the command. Check the command's
|
||||
parallelism settings, `node_count`, and all `worker_commands` against the
|
||||
user's available topology, and compare the variant's `vram_minimum_gb`, an
|
||||
estimate, with the total memory of the GPUs the command uses; a configuration
|
||||
below that minimum is not a working option unless the guide says it was
|
||||
tested. Ask when the user's hardware is unclear. If no
|
||||
published command or guide command fits, report the mismatch and return the
|
||||
closest one unchanged; list any adjustments the user's setup would need
|
||||
separately and mark them unverified instead of editing the command.
|
||||
|
||||
For another strategy, follow `alternatives` in the same response. These are
|
||||
static files; query parameters do not change their configuration. For other
|
||||
feature/topology selections, use the recipes repository's command synthesis.
|
||||
|
||||
### Read deployment guidance
|
||||
|
||||
Read the guide sections that apply to the selection, such as prerequisites,
|
||||
launch notes for the selected hardware or variant, troubleshooting, and known
|
||||
limitations. Pass entries from `guide_sections` or prefixes of their text.
|
||||
The script prints every matching section with its subsections and fails when
|
||||
a section matches nothing:
|
||||
|
||||
```bash
|
||||
"$read_recipe" "$recipe" prerequisites troubleshooting
|
||||
```
|
||||
|
||||
Query other fields with `jq` on the saved file, such as `features.<name>`,
|
||||
`hardware_overrides`, or `omni.tasks` for request examples.
|
||||
|
||||
### Return the result
|
||||
|
||||
Return the website link, selected checkpoint and hardware, the requested
|
||||
command, and relevant prerequisites and caveats from the guide. Pick one
|
||||
command and copy it, including its Docker form, exactly as the command object
|
||||
or guide section gives it; do not combine commands from different sections or
|
||||
add, drop, or reorder flags. List optional changes separately.
|
||||
|
||||
For each command, state its source, either a generated hardware or strategy
|
||||
configuration or a named guide section, and whether that source says it was
|
||||
tested. A `recipe_verified_hardware` entry covers the recipe on that
|
||||
hardware, not every variant, strategy, or guide command, so it does not make a
|
||||
command verified. Generated configurations are not automatically verified.
|
||||
|
||||
## Create or update a recipe
|
||||
|
||||
Recipes cover official or widely used checkpoints, with configurations verified
|
||||
end to end on real hardware. Do not add recipes for arbitrary community
|
||||
uploads, untested hardware claims, or non-standard flags that only work for one
|
||||
setup. Before opening a pull request, confirm that the user ran the
|
||||
configuration and can share the results.
|
||||
|
||||
Follow the recipes repository's
|
||||
[add-recipe skill](https://github.com/vllm-project/recipes/blob/main/.agents/skills/add-recipe/SKILL.md)
|
||||
in a checkout of `vllm-project/recipes` or the user's intended fork.
|
||||
@@ -0,0 +1,4 @@
|
||||
interface:
|
||||
display_name: "Model Recipes"
|
||||
short_description: "Find vLLM deployment recipes and configs for LLMs and Hugging Face models by model and hardware."
|
||||
default_prompt: "Use $model-recipes to find a recipe for my model and hardware, and return its commands and deployment caveats."
|
||||
+66
@@ -0,0 +1,66 @@
|
||||
#!/bin/bash
|
||||
# Usage: read_recipe.sh <recipe.json> [guide-section...]
|
||||
# Without sections, print a compact summary of a recipes.vllm.ai recipe JSON.
|
||||
# With sections, print each guide section whose heading starts with one of
|
||||
# them (case-insensitive), including its subsections.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
if [ $# -lt 1 ]; then
|
||||
echo "Usage: $0 <recipe.json> [guide-section...]" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Print "<level>\t<heading text>\t<line>" for guide lines, marking headings
|
||||
# outside code fences with their level and other lines with 0.
|
||||
guide_lines() {
|
||||
jq -r '.guide // ""' "$1" | awk '
|
||||
/^[[:space:]]*(```|~~~)/ { fence = !fence }
|
||||
!fence && match($0, /^#+ /) {
|
||||
print RLENGTH - 1 "\t" substr($0, RLENGTH + 1) "\t" $0
|
||||
next
|
||||
}
|
||||
{ print "0\t\t" $0 }'
|
||||
}
|
||||
|
||||
if [ $# -gt 1 ]; then
|
||||
recipe=$1
|
||||
shift
|
||||
status=0
|
||||
for section in "$@"; do
|
||||
guide_lines "$recipe" | awk -F '\t' -v h="$section" '
|
||||
BEGIN { h = tolower(h); sub(/^#+[[:space:]]*/, "", h) }
|
||||
$1 > 0 && lvl && $1 <= lvl { lvl = 0 }
|
||||
$1 > 1 && !lvl && index(tolower($2), h) == 1 { lvl = $1; found = 1 }
|
||||
lvl { sub(/^[^\t]*\t[^\t]*\t/, ""); print }
|
||||
END {
|
||||
if (!found) {
|
||||
print "No guide section starts with: " h > "/dev/stderr"
|
||||
exit 1
|
||||
}
|
||||
}' || status=1
|
||||
done
|
||||
exit $status
|
||||
fi
|
||||
|
||||
sections=$(guide_lines "$1" | awk -F '\t' '$1 == 2 || $1 == 3 { print $3 }' |
|
||||
jq -R . | jq -s .)
|
||||
|
||||
jq --argjson sections "$sections" '{
|
||||
hf_id,
|
||||
title: .meta.title,
|
||||
tasks: .meta.tasks,
|
||||
recipe_verified_hardware: .meta.hardware,
|
||||
omni_tasks: (.omni.tasks // null |
|
||||
if . then map(objects |= {id, label, endpoint, model_id, description})
|
||||
else null end),
|
||||
variants: (.variants // {} | map_values(
|
||||
{model_id, precision, vram_minimum_gb, supported_hardware} |
|
||||
with_entries(select(.value != null)))),
|
||||
features: (.features // {} | keys),
|
||||
opt_in_features,
|
||||
default: (.recommended_command // null |
|
||||
if . then {hardware, strategy, variant} else null end),
|
||||
hardware: .recommended_command.by_hardware,
|
||||
guide_sections: $sections
|
||||
} | with_entries(select(.value != null))' "$1"
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
name: pr-checklist
|
||||
description: Prepare a vLLM change for human PR review or re-review, or analyze an open pull request. Use to check design fit, behavioral coverage, performance evidence, diff quality, and closure of previous feedback before requesting maintainer attention.
|
||||
description: Prepare a vLLM change for human PR review or re-review, or analyze an open pull request. Use before opening a PR, drafts included (e.g. `gh pr create`), or pushing review fixes, to check design fit, behavioral coverage, performance evidence, diff quality, and closure of previous feedback.
|
||||
---
|
||||
|
||||
# PR Checklist
|
||||
@@ -181,6 +181,8 @@ Summarize model evaluations and benchmarks in compact tables with the configurat
|
||||
|
||||
When fixing bugs or issues, provide a clear root-cause analysis. Explain what caused the problem, how it was identified, and why the chosen solution effectively addresses it. A patch which resolves the issue without a proper root-cause analysis may lead to recurring problems and increased maintenance overhead.
|
||||
|
||||
For fixes that involve disaggregated prefill/decode (PD) logic, such as the NIXL connector (`vllm/distributed/kv_transfer/kv_connector/v1/nixl/`), make a best effort to include a minimal example of the flow being changed and/or the initial flow that caused the failure. A short numbered sequence (or diagram) of a request's path across the P and D instances, e.g. scheduler admission, prefill execution, KV transfer registration and write, decode-side read, token generation, lets reviewers trace the root cause and the fix without reconstructing the distributed flow from the diff. Keep the example minimal: only the steps and components involved in the change. See the [disaggregated prefill](../../../docs/features/disagg_prefill.md), [NIXL KV push](../../../docs/design/nixl_kv_push_connector.md), and [NIXL KV cache lease](../../../docs/design/nixl_kv_cache_lease.md) docs for the flows involved.
|
||||
|
||||
#### 4.4: Implementation Details
|
||||
|
||||
Provide a detailed explanation of the implementation, including design decisions, trade-offs, and any relevant technical details. Highlight any non-obvious choices, tradeoffs, potential limitations, and areas where maintainer attention is required.
|
||||
|
||||
@@ -0,0 +1,178 @@
|
||||
---
|
||||
name: transformers-upgrade
|
||||
description: "Playbook for changing vLLM's Transformers version bounds. Use when bumping the pinned or maximum Transformers version, raising the minimum Transformers version, or deleting vendored configs, processors, or compat shims that upstream Transformers now provides."
|
||||
---
|
||||
|
||||
# Upgrading Transformers
|
||||
|
||||
The supported range lives in `requirements/common.txt`. CI pins one exact
|
||||
version in `requirements/test/*.in` and the compiled `requirements/test/*.txt`.
|
||||
A version change is one of two jobs, and they usually come as a pair:
|
||||
|
||||
- **Bump the pin**: a new Transformers release becomes the CI version and
|
||||
the upper bound. vLLM grows compat code for it.
|
||||
- **Raise the floor**: the minimum moves up. That compat code, and every
|
||||
vendored config or processor upstream now ships, becomes dead weight.
|
||||
Every floor raise ends with an **unvendoring pass**.
|
||||
|
||||
## Bump the pin
|
||||
|
||||
1. Edit the pin line in each `requirements/test/*.in` and `*.txt` and the
|
||||
upper bound in `requirements/common.txt`. Past bumps edit only the pin
|
||||
line in the compiled files; see `git log -- requirements/common.txt`.
|
||||
2. Read the release notes for config validation changes, renamed auto-mapping
|
||||
formats, and removed attributes; these are what break vLLM.
|
||||
3. Search `tests/` for `min_transformers_version` and skip markers naming the
|
||||
new version, and un-gate them.
|
||||
4. Run the suites in [Testing](#testing) and fix what breaks. Mark every
|
||||
shim with a `# Transformers >= X` comment so the next floor raise finds it.
|
||||
|
||||
Done when the [Testing](#testing) suites pass on the new pin and every test
|
||||
waiting on this version is un-gated.
|
||||
|
||||
## Raise the floor (unvendoring pass)
|
||||
|
||||
### 1. Inventory
|
||||
|
||||
Inventory candidates with the floor version installed (no vLLM install needed):
|
||||
|
||||
```bash
|
||||
uv run --no-project --with transformers==<floor> \
|
||||
python .agents/skills/transformers-upgrade/scripts/find_upstreamed.py
|
||||
```
|
||||
|
||||
It lists:
|
||||
|
||||
- vendored configs whose `model_type` upstream now owns;
|
||||
- `_CONFIG_REGISTRY` overrides of upstream types;
|
||||
- registered processors that collide with upstream class names or models;
|
||||
- version-gated code at or below the floor;
|
||||
- code that reads the installed Transformers version, whatever its threshold;
|
||||
- patches citing a Transformers issue or PR (with whether a referenced PR is
|
||||
in the floor release) or a future minimum version.
|
||||
|
||||
It finds candidates by name only. Every hit still needs a verdict.
|
||||
|
||||
### 2. Port each candidate to upstream
|
||||
|
||||
Every hit is deleted by default. A vendored copy is a liability: it drifts
|
||||
from upstream, hides upstream fixes, and shadows the native class. Port the
|
||||
vLLM model code to the upstream schema instead, renaming attribute reads and
|
||||
deriving values from upstream fields.
|
||||
|
||||
Fan candidates out to parallel subagents. Each one compares the vendored and
|
||||
upstream classes, then greps every attribute the vLLM model code reads.
|
||||
[Upstreaming hazards](#upstreaming-hazards) lists the usual fixes.
|
||||
|
||||
Keep a vendored class only when upstream cannot load a checkpoint vLLM must
|
||||
keep serving, and record why in the PR description. A different upstream
|
||||
schema, a model-code rewrite, or a needed eval is work for the PR, not a
|
||||
reason to keep.
|
||||
|
||||
### 3. Prove it on real checkpoints
|
||||
|
||||
For every checkpoint in `tests/models/registry.py` that uses a ported
|
||||
config, dump what vLLM builds before and after the change, then diff the two.
|
||||
Run both dumps; each catches what the other misses:
|
||||
|
||||
- `dump_configs.py`: the resolved config, attribute by attribute.
|
||||
- `dump_model.py`: the model built on the meta device with no GPU or weights.
|
||||
It records ModelConfig sizes, every parameter shape, and every module's
|
||||
scalar attributes (`head_dim`, `num_kv_heads`, `scaling`, ...). Missing
|
||||
fields that are read with defaults only show up here.
|
||||
|
||||
```bash
|
||||
git worktree add --detach <scratch>/vllm-base upstream/main
|
||||
S=.agents/skills/transformers-upgrade/scripts
|
||||
for dump in dump_configs dump_model; do
|
||||
PYTHONPATH=<scratch>/vllm-base .venv/bin/python $S/$dump.py before.jsonl <repos>
|
||||
PYTHONPATH=$PWD .venv/bin/python $S/$dump.py after.jsonl <repos>
|
||||
.venv/bin/python $S/diff_dumps.py before.jsonl after.jsonl
|
||||
done
|
||||
```
|
||||
|
||||
Add `--trust-remote-code` for checkpoints whose `config.json` has an
|
||||
`auto_map`, and dump both ways.
|
||||
|
||||
How to read the diffs:
|
||||
|
||||
- **A `<MISSING>` config field the model reads:** this crashes or silently
|
||||
falls back to a default. Port the model code to the upstream field.
|
||||
- **New upstream-only defaults** (`attention_bias=False`,
|
||||
`pad_token_id=None`): usually benign. A model diff of 0 confirms it.
|
||||
- **Any model diff:** this is a numerics change, and the PR needs a model
|
||||
eval. For example, A.X-K1 gained the checkpoint's YaRN rope scaling, which
|
||||
the vendored class had been dropping; it shows up as
|
||||
`layers.*.self_attn.scaling (x61)`.
|
||||
|
||||
### 4. Delete
|
||||
|
||||
For each deleted config:
|
||||
|
||||
- Delete the file.
|
||||
- Remove its entries from `_CLASS_TO_MODULE` and `__all__` in
|
||||
`vllm/transformers_utils/configs/__init__.py`.
|
||||
- Remove its entries from `_CONFIG_REGISTRY` in
|
||||
`vllm/transformers_utils/config.py`.
|
||||
- Switch every importer, including tests, to `from transformers import ...`.
|
||||
- Remove the `source_file_dependencies` lines naming the file from
|
||||
`.buildkite/`.
|
||||
|
||||
Deleted processors follow the same steps in
|
||||
`vllm/transformers_utils/processors/__init__.py`.
|
||||
|
||||
Done when every line of the inventory output is deleted, or kept with a
|
||||
reason that meets the bar in step 2. The PR also carries the config diff
|
||||
results and evals for any changed values.
|
||||
|
||||
## Upstreaming hazards
|
||||
|
||||
Upstream configs are `@strict` dataclasses. Checkpoint keys upstream does not
|
||||
declare survive as plain attributes. Defaults that only vLLM sets, and values
|
||||
vLLM derives, vanish.
|
||||
|
||||
| Hazard | Signal | Fix |
|
||||
| --- | --- | --- |
|
||||
| Upstream consumes legacy keys | `__post_init__` pops keys the model reads (`deepseek_v4` `compress_ratios`) | Read the upstream replacement (`layer_types`, `mlp_layer_types`, per-layer `rope_parameters`) |
|
||||
| Renamed or dropped fields | Diff shows `<MISSING>` for a field the model reads | Read the upstream name; derive dropped flags from what implies them (`enable_moe_block` from `num_experts`) |
|
||||
| Type check on the config | `ctx.get_hf_config(SomeConfig)` relies on the vendored base class | Accept the upstream class too, e.g. a `Cosmos3ProcessingInfo` overriding `get_hf_config` |
|
||||
| Remote code takes over | Checkpoint `config.json` has `auto_map`; with `--trust-remote-code`, the hub class replaces the native one once vLLM stops registering its own | Check the remote class loads; if not, register the upstream class for that `model_type` |
|
||||
| Read-only upstream fields | Properties or derived fields break subclasses and `hf_overrides` (`nemotron_h` `hybrid_override_pattern`) | Override the source field (`layers_block_type`) instead |
|
||||
| Behavioural rewrite | Upstream derives fields differently (`diffusion_gemma` `sliding_window`) | Upstream matches the reference implementation; verify with an eval |
|
||||
| Original checkpoint format | Upstream targets `-hf` conversions; vLLM serves the originals (`qwen3_asr`, `qianfan_ocr`) | Keep only if upstream cannot load the original at all |
|
||||
| Processor output contract | Model code consumes keys upstream does not emit | Port the model to upstream's keys |
|
||||
| Name collision | A registered vendored processor shares an upstream class name and hijacks `get_processor` (`InternVLProcessor`) | Drop the registry entry |
|
||||
|
||||
Version gates from the inventory need a decision:
|
||||
|
||||
- **Code** only reachable below the floor is deleted. This covers:
|
||||
- comparisons against `TRANSFORMERS_VERSION`;
|
||||
- branches labelled `# Transformers < X`;
|
||||
- fallbacks for fields that upstream configs now always set, e.g.
|
||||
`mlp_layer_types`, `per_layer_config` or legacy `layer_types` names.
|
||||
|
||||
A `skipif` whose condition is now always true becomes an unconditional
|
||||
`skip` with the same reason.
|
||||
- **`min_transformers_version`** at or below the floor is removed.
|
||||
- **`max_transformers_version`** below the floor means CI never tests that
|
||||
model; report it rather than deleting it silently.
|
||||
- **Patches citing a Transformers issue or PR:**
|
||||
- Delete the patch when its fix is in the floor release.
|
||||
- For an issue, find the PR that closed it and check that PR. A closed
|
||||
issue is not a fixed one: #43329 was closed while the bug was still in
|
||||
5.16.1.
|
||||
- Re-run the patched behaviour before deleting. A fix in the release does
|
||||
not always cover vLLM's path: #47924 made Emu3's processor add BOS, but
|
||||
vLLM tokenizes without the processor.
|
||||
- Links that only explain a design choice (e.g. a float32 RoPE note) are
|
||||
not patches; leave them.
|
||||
|
||||
## Testing
|
||||
|
||||
```bash
|
||||
.venv/bin/python -m pytest tests/transformers_utils tests/config tests/tokenizers_
|
||||
.venv/bin/python -m pytest tests/models/multimodal/processing/test_common.py -k "<models>"
|
||||
```
|
||||
|
||||
Add `tests/models/test_initialization.py -k "<archs>"` where a GPU is
|
||||
available, and a model eval for any model whose resolved config changed.
|
||||
@@ -0,0 +1,4 @@
|
||||
interface:
|
||||
display_name: "Transformers Upgrade"
|
||||
short_description: "Bump vLLM's Transformers version and delete vendored code upstream now provides."
|
||||
default_prompt: "Use $transformers-upgrade to raise the minimum Transformers version and remove vendored configs, processors, and shims it makes redundant."
|
||||
@@ -0,0 +1,51 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
"""Diff two dump_configs.py or dump_model.py outputs, per repo.
|
||||
|
||||
python diff_dumps.py before.jsonl after.jsonl
|
||||
"""
|
||||
|
||||
import json
|
||||
import sys
|
||||
from collections import Counter
|
||||
|
||||
import regex as re
|
||||
|
||||
IGNORED = {"transformers_version", "_name_or_path", "architectures", "auto_map"}
|
||||
|
||||
|
||||
def load(path: str) -> dict[str, dict]:
|
||||
with open(path) as f:
|
||||
return {record["repo"]: record for record in map(json.loads, f)}
|
||||
|
||||
|
||||
def diff(before: dict, after: dict, prefix: str = ""):
|
||||
for key in sorted(set(before) | set(after)):
|
||||
if key in IGNORED:
|
||||
continue
|
||||
old, new = before.get(key, "<MISSING>"), after.get(key, "<MISSING>")
|
||||
if isinstance(old, dict) and isinstance(new, dict) and key != "rope_parameters":
|
||||
yield from diff(old, new, f"{prefix}{key}.")
|
||||
elif old != new:
|
||||
yield f"{prefix}{key}", json.dumps(old), json.dumps(new)
|
||||
|
||||
|
||||
def collapse(diffs) -> Counter:
|
||||
"""Merge diffs that differ only by layer index, e.g. `layers.*.scaling`."""
|
||||
return Counter(
|
||||
(re.sub(r"\.\d+\.", ".*.", path), old, new) for path, old, new in diffs
|
||||
)
|
||||
|
||||
|
||||
before, after = load(sys.argv[1]), load(sys.argv[2])
|
||||
for repo, old in before.items():
|
||||
new = after.get(repo, {"error": "missing from after"})
|
||||
if "error" in old or "error" in new:
|
||||
print(f"== {repo}\n before: {old.get('error', 'ok')}")
|
||||
print(f" after: {new.get('error', 'ok')}")
|
||||
continue
|
||||
diffs = collapse(diff(old["cfg"], new["cfg"]))
|
||||
print(f"== {repo}: {old['cls']} -> {new['cls']} ({sum(diffs.values())} diffs)")
|
||||
for (path, old_value, new_value), count in diffs.items():
|
||||
times = f" (x{count})" if count > 1 else ""
|
||||
print(f" {path}{times}: before={old_value} after={new_value}")
|
||||
@@ -0,0 +1,30 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
"""Dump the config vLLM resolves for each checkpoint, one JSON line per repo.
|
||||
|
||||
Run once per vLLM tree (before and after a change), then compare the two
|
||||
outputs with diff_dumps.py:
|
||||
|
||||
PYTHONPATH=<vllm tree> .venv/bin/python dump_configs.py out.jsonl <repo>...
|
||||
|
||||
Pass --trust-remote-code to resolve checkpoints the way users who set it do.
|
||||
"""
|
||||
|
||||
import json
|
||||
import sys
|
||||
|
||||
from vllm.transformers_utils.config import get_config
|
||||
|
||||
out, *args = sys.argv[1:]
|
||||
trust_remote_code = "--trust-remote-code" in args
|
||||
with open(out, "w") as f:
|
||||
for repo in (arg for arg in args if arg != "--trust-remote-code"):
|
||||
try:
|
||||
config = get_config(repo, trust_remote_code=trust_remote_code)
|
||||
cls = f"{type(config).__module__}.{type(config).__name__}"
|
||||
record = {"repo": repo, "cls": cls, "cfg": config.to_dict()}
|
||||
except Exception as e:
|
||||
error = f"{type(e).__name__}: {e}".splitlines()[0]
|
||||
record = {"repo": repo, "error": error}
|
||||
f.write(json.dumps(record, default=str, sort_keys=True) + "\n")
|
||||
f.flush()
|
||||
@@ -0,0 +1,114 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
"""Build each model on the meta device and dump what vLLM constructs.
|
||||
|
||||
Records ModelConfig-derived sizes, every parameter shape, and scalar
|
||||
attributes of every module (head_dim, num_kv_heads, sliding_window, ...).
|
||||
Needs no GPU and no weights. Run once per vLLM tree, then compare the two
|
||||
outputs with diff_dumps.py:
|
||||
|
||||
PYTHONPATH=<vllm tree> .venv/bin/python dump_model.py out.jsonl <repo>...
|
||||
|
||||
Pass --trust-remote-code to resolve checkpoints the way users who set it do.
|
||||
"""
|
||||
|
||||
import json
|
||||
import socket
|
||||
import sys
|
||||
|
||||
import torch
|
||||
|
||||
from vllm.config import set_current_vllm_config
|
||||
from vllm.config.vllm import VllmConfig
|
||||
from vllm.distributed import (
|
||||
init_distributed_environment,
|
||||
initialize_model_parallel,
|
||||
model_parallel_is_initialized,
|
||||
)
|
||||
from vllm.engine.arg_utils import EngineArgs
|
||||
from vllm.model_executor.model_loader.utils import initialize_model
|
||||
from vllm.utils.torch_utils import set_default_torch_dtype
|
||||
|
||||
# Construction only: skip model runner checks that need a GPU or Triton
|
||||
VllmConfig._get_v1_model_runner_unsupported_features = lambda self: [] # type: ignore[method-assign]
|
||||
|
||||
# Model code that moves submodules to a device must leave them on meta
|
||||
_module_to = torch.nn.Module.to
|
||||
|
||||
|
||||
def _meta_safe_to(self, *args, **kwargs):
|
||||
kwargs.pop("device", None)
|
||||
args = tuple(a for a in args if not isinstance(a, (torch.device, str, int)))
|
||||
return _module_to(self, *args, **kwargs)
|
||||
|
||||
|
||||
torch.nn.Module.to = _meta_safe_to # type: ignore[method-assign]
|
||||
|
||||
SCALARS = (bool, int, float, str)
|
||||
|
||||
|
||||
def module_attrs(model: torch.nn.Module) -> dict[str, dict]:
|
||||
attrs = {}
|
||||
for name, module in model.named_modules():
|
||||
scalars = {
|
||||
key: value
|
||||
for key, value in vars(module).items()
|
||||
if not key.startswith("_") and isinstance(value, SCALARS)
|
||||
}
|
||||
if scalars:
|
||||
attrs[name or "<root>"] = scalars
|
||||
return attrs
|
||||
|
||||
|
||||
def dump(repo: str, trust_remote_code: bool) -> dict:
|
||||
args = EngineArgs(
|
||||
model=repo,
|
||||
load_format="dummy",
|
||||
enforce_eager=True,
|
||||
max_model_len=4096,
|
||||
trust_remote_code=trust_remote_code,
|
||||
)
|
||||
vllm_config = args.create_engine_config()
|
||||
model_config = vllm_config.model_config
|
||||
with set_current_vllm_config(vllm_config):
|
||||
if not model_parallel_is_initialized():
|
||||
with socket.socket() as sock:
|
||||
sock.bind(("", 0))
|
||||
port = sock.getsockname()[1]
|
||||
init_distributed_environment(1, 0, f"tcp://127.0.0.1:{port}", 0, "gloo")
|
||||
initialize_model_parallel(1, 1)
|
||||
with set_default_torch_dtype(model_config.dtype), torch.device("meta"):
|
||||
model = initialize_model(vllm_config=vllm_config)
|
||||
return {
|
||||
"repo": repo,
|
||||
"cls": type(model).__name__,
|
||||
"cfg": {
|
||||
"model_config": {
|
||||
"head_size": model_config.get_head_size(),
|
||||
"total_num_kv_heads": model_config.get_total_num_kv_heads(),
|
||||
"sliding_window": model_config.get_sliding_window(),
|
||||
"num_layers": model_config.get_num_layers(vllm_config.parallel_config),
|
||||
},
|
||||
"params": {n: list(p.shape) for n, p in model.named_parameters()},
|
||||
"modules": module_attrs(model),
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def main() -> None:
|
||||
out, *args = sys.argv[1:]
|
||||
trust_remote_code = "--trust-remote-code" in args
|
||||
repos = [arg for arg in args if arg != "--trust-remote-code"]
|
||||
with open(out, "w") as f:
|
||||
for repo in repos:
|
||||
try:
|
||||
record = dump(repo, trust_remote_code)
|
||||
except Exception as e:
|
||||
error = f"{type(e).__name__}: {e}".splitlines()[0]
|
||||
record = {"repo": repo, "error": error}
|
||||
f.write(json.dumps(record, default=str, sort_keys=True) + "\n")
|
||||
f.flush()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,147 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
"""List vendored vLLM code that the installed Transformers may make redundant.
|
||||
|
||||
Run with the Transformers version under test, from the vLLM repo root:
|
||||
|
||||
uv run --no-project --with transformers==<version> \
|
||||
python .agents/skills/transformers-upgrade/scripts/find_upstreamed.py
|
||||
|
||||
Imports nothing from vLLM, so no vLLM install is needed. Uses the `gh` CLI, if
|
||||
available, to check whether referenced Transformers PRs are in this release.
|
||||
"""
|
||||
|
||||
import json
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import regex as re
|
||||
import transformers
|
||||
from packaging.version import Version
|
||||
from transformers.models.auto.configuration_auto import CONFIG_MAPPING_NAMES
|
||||
from transformers.models.auto.processing_auto import PROCESSOR_MAPPING_NAMES
|
||||
|
||||
ROOT = Path(sys.argv[1] if len(sys.argv) > 1 else ".")
|
||||
FLOOR = Version(transformers.__version__)
|
||||
UPSTREAM_NAMES = set(dir(transformers))
|
||||
|
||||
MODEL_TYPE = re.compile(r'model_type\s*(?::\s*str\s*)?=\s*"([^"]+)"')
|
||||
REGISTRY_KEY = re.compile(r'^\s+(?:\*\*\{)?"?([\w-]+)"?(?:=|": )"(\w+)"', re.M)
|
||||
CLASS_KEY = re.compile(r'^\s+"(\w+)": ', re.M)
|
||||
UPSTREAM_REF = re.compile(
|
||||
r"huggingface/transformers(?:/(?:issues|pull)/|#)(\d+)"
|
||||
r"|(?:once|until|when) [Tt]ransformers [\d.]+ is the minimum"
|
||||
)
|
||||
VERSIONED = re.compile(
|
||||
r"(?:min_transformers_version|max_transformers_version|check_version|Version)"
|
||||
r'\(?\s*=?\s*"([45]\.\d+(?:\.\d+)?)[^"]*"'
|
||||
r"|[Tt]ransformers\s*(?:v|[<>]=?\s*)([45]\.\d+(?:\.\d+)?|4)\b"
|
||||
)
|
||||
READS_VERSION = re.compile(
|
||||
r"transformers import __version__|transformers\.__version__"
|
||||
r"|version\([\"']transformers[\"']\)|TRANSFORMERS_VERSION|check_version\("
|
||||
)
|
||||
|
||||
|
||||
def section(title: str) -> None:
|
||||
print(f"\n## {title}")
|
||||
|
||||
|
||||
def vendored_configs() -> None:
|
||||
section(f"Vendored configs whose model_type is upstream in {FLOOR}")
|
||||
files = sorted(ROOT.glob("vllm/transformers_utils/configs/*.py"))
|
||||
files += sorted(ROOT.glob("vllm/models/*/config*.py"))
|
||||
files += sorted(ROOT.glob("vllm/models/*/configs/*.py"))
|
||||
for path in files:
|
||||
types = sorted(set(MODEL_TYPE.findall(path.read_text())))
|
||||
hits = [t for t in types if t in CONFIG_MAPPING_NAMES]
|
||||
if hits:
|
||||
upstream = ", ".join(f"{t} -> {CONFIG_MAPPING_NAMES[t]}" for t in hits)
|
||||
print(f"- {path.relative_to(ROOT)}: {upstream}")
|
||||
|
||||
registry = (ROOT / "vllm/transformers_utils/config.py").read_text()
|
||||
start = registry.index("_CONFIG_REGISTRY")
|
||||
body = registry[start : registry.index("\n)\n", start)]
|
||||
section("_CONFIG_REGISTRY entries that override an upstream model_type")
|
||||
for model_type, cls in REGISTRY_KEY.findall(body):
|
||||
if model_type in CONFIG_MAPPING_NAMES:
|
||||
print(f"- {model_type}: vLLM {cls} over {CONFIG_MAPPING_NAMES[model_type]}")
|
||||
|
||||
|
||||
def vendored_processors() -> None:
|
||||
init = (ROOT / "vllm/transformers_utils/processors/__init__.py").read_text()
|
||||
names = sorted(set(CLASS_KEY.findall(init)))
|
||||
section("Registered processor names that collide with upstream classes")
|
||||
for name in names:
|
||||
if name in UPSTREAM_NAMES:
|
||||
print(f"- {name}")
|
||||
section("Processor files named after a model_type with an upstream processor")
|
||||
for path in sorted(ROOT.glob("vllm/transformers_utils/processors/*.py")):
|
||||
if path.stem in PROCESSOR_MAPPING_NAMES:
|
||||
upstream = PROCESSOR_MAPPING_NAMES[path.stem]
|
||||
print(f"- {path.relative_to(ROOT)} -> {upstream}")
|
||||
|
||||
|
||||
def version_gates() -> None:
|
||||
files = sorted([*ROOT.glob("vllm/**/*.py"), *ROOT.glob("tests/**/*.py")])
|
||||
section(f"Version-gated code at or below {FLOOR} (review each)")
|
||||
for path in files:
|
||||
for lineno, line in enumerate(path.read_text().splitlines(), 1):
|
||||
for match in VERSIONED.finditer(line):
|
||||
version = Version(match.group(1) or match.group(2))
|
||||
if version <= FLOOR:
|
||||
print(f"- {path.relative_to(ROOT)}:{lineno}: {line.strip()}")
|
||||
break
|
||||
section("Code that reads the installed Transformers version (review each)")
|
||||
for path in files:
|
||||
for lineno, line in enumerate(path.read_text().splitlines(), 1):
|
||||
if READS_VERSION.search(line):
|
||||
print(f"- {path.relative_to(ROOT)}:{lineno}: {line.strip()}")
|
||||
|
||||
|
||||
def upstream_status(number: str) -> str:
|
||||
"""Whether Transformers issue/PR `number` is resolved in this release."""
|
||||
|
||||
def gh(path: str) -> dict | None:
|
||||
try:
|
||||
out = subprocess.run(["gh", "api", path], capture_output=True, text=True)
|
||||
except FileNotFoundError:
|
||||
return None
|
||||
return json.loads(out.stdout) if out.returncode == 0 else None
|
||||
|
||||
repo = "repos/huggingface/transformers"
|
||||
if (issue := gh(f"{repo}/issues/{number}")) is None:
|
||||
return "status unknown"
|
||||
if "pull_request" not in issue:
|
||||
return f"issue {issue['state']}, check which PR fixed it"
|
||||
pr = gh(f"{repo}/pulls/{number}")
|
||||
if pr is None or not pr["merged_at"]:
|
||||
return "PR not merged"
|
||||
compare = gh(f"{repo}/compare/v{FLOOR}...{pr['merge_commit_sha']}")
|
||||
included = compare and compare["status"] in ("behind", "identical")
|
||||
return f"PR merged, {'in' if included else 'not in'} v{FLOOR}"
|
||||
|
||||
|
||||
def upstream_refs() -> None:
|
||||
section("Patches referencing Transformers issues/PRs or a future minimum")
|
||||
files = sorted([*ROOT.glob("vllm/**/*.py"), *ROOT.glob("tests/**/*.py")])
|
||||
statuses: dict[str, str] = {}
|
||||
for path in files:
|
||||
for lineno, line in enumerate(path.read_text().splitlines(), 1):
|
||||
if (match := UPSTREAM_REF.search(line)) is None:
|
||||
continue
|
||||
note = ""
|
||||
if number := match.group(1):
|
||||
if number not in statuses:
|
||||
statuses[number] = upstream_status(number)
|
||||
note = f" [#{number}: {statuses[number]}]"
|
||||
print(f"- {path.relative_to(ROOT)}:{lineno}:{note} {line.strip()}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
print(f"# Transformers {FLOOR}")
|
||||
vendored_configs()
|
||||
vendored_processors()
|
||||
version_gates()
|
||||
upstream_refs()
|
||||
@@ -20,8 +20,10 @@ steps:
|
||||
- tests/kernels/moe/test_cpu_fused_moe.py
|
||||
- tests/kernels/moe/test_cpu_quant_fused_moe.py
|
||||
- tests/kernels/moe/test_zen_cpu_int8_moe.py
|
||||
- tests/test_zentorch_version_pin.py
|
||||
- vllm/model_executor/layers/fused_moe/experts/cpu_moe.py
|
||||
- tests/kernels/test_onednn.py
|
||||
- tests/kernels/test_cpu_fused_sampling.py
|
||||
- tests/kernels/test_awq_int4_to_int8.py
|
||||
- tests/kernels/quantization/test_cpu_fp8_scaled_mm.py
|
||||
- tests/kernels/quantization/test_zen_da8w4.py
|
||||
@@ -45,8 +47,10 @@ steps:
|
||||
tests/kernels/moe/test_cpu_fused_moe.py \
|
||||
tests/kernels/moe/test_cpu_quant_fused_moe.py \
|
||||
tests/kernels/moe/test_zen_cpu_int8_moe.py \
|
||||
tests/test_zentorch_version_pin.py \
|
||||
tests/kernels/mamba/test_cpu_short_conv.py \
|
||||
tests/kernels/test_onednn.py \
|
||||
tests/kernels/test_cpu_fused_sampling.py \
|
||||
tests/kernels/test_awq_int4_to_int8.py \
|
||||
tests/kernels/quantization/test_cpu_fp8_scaled_mm.py \
|
||||
tests/kernels/quantization/test_zen_da8w4.py \
|
||||
@@ -85,7 +89,7 @@ steps:
|
||||
- |
|
||||
bash .buildkite/scripts/hardware_ci/run-cpu-test.sh 30m "
|
||||
pytest -x -v -s \
|
||||
tests/v1/worker/test_gpu_autoregressive_speculator.py \
|
||||
tests/v1/worker/test_gpu_target_dependent_ar_speculator.py \
|
||||
tests/v1/worker/test_gpu_extract_hidden_states_speculator.py \
|
||||
tests/v1/worker/test_gpu_batch_ordering.py \
|
||||
tests/v1/worker/test_gpu_rejection_sampler_chunking.py \
|
||||
@@ -228,3 +232,57 @@ steps:
|
||||
"$$BUILDKITE_PARALLEL_JOB"
|
||||
"$REGISTRY/$REPO:$BUILDKITE_COMMIT-arm64-cpu"
|
||||
parallelism: 3
|
||||
|
||||
- label: ":docker: Build Zen CPU image"
|
||||
key: zen-cpu-image-build
|
||||
# Reuse the shared `<repo>:<commit>-cpu` image that image-build-cpu publishes
|
||||
# to public ECR, so Dockerfile.zen just layers on top instead of recompiling
|
||||
# vLLM from source. If the pull fails, the build script still falls back to
|
||||
# building the CPU base from source.
|
||||
depends_on:
|
||||
- image-build-cpu
|
||||
soft_fail: true
|
||||
agents:
|
||||
queue: amd-cpu
|
||||
source_file_dependencies:
|
||||
- .buildkite/hardware_tests/cpu.yaml
|
||||
- .buildkite/image_build/image_build_zen_cpu.sh
|
||||
- docker/Dockerfile.zen
|
||||
- docker/Dockerfile.cpu
|
||||
- requirements/cpu.txt
|
||||
- requirements/build/cpu.txt
|
||||
- requirements/test/cpu.txt
|
||||
- cmake/cpu_extension.cmake
|
||||
- vllm/model_executor/layers/utils.py
|
||||
- vllm/model_executor/kernels/linear/zentorch_utils.py
|
||||
- vllm/model_executor/kernels/linear/scaled_mm/zentorch.py
|
||||
- vllm/model_executor/kernels/linear/mixed_precision/zentorch.py
|
||||
- vllm/platforms/cpu.py
|
||||
- vllm/platforms/zen_cpu.py
|
||||
- vllm/platforms/__init__.py
|
||||
- tests/model_executor/test_cpu_unquantized_gemm_dispatch.py
|
||||
- tests/test_zen_cpu_platform_detection.py
|
||||
- tests/kernels/moe/test_zen_cpu_fused_moe.py
|
||||
- tests/kernels/moe/test_zen_cpu_int8_moe.py
|
||||
- vllm/v1/attention/backends/zentorch_sdpa.py
|
||||
- tests/test_zentorch_version_pin.py
|
||||
- vllm/v1/attention/backends/mla/prefill/zen_cpu_sdpa.py
|
||||
- vllm/model_executor/layers/fused_moe/experts/zentorch_moe.py
|
||||
- tests/kernels/quantization/test_zen_da8w4.py
|
||||
commands:
|
||||
- .buildkite/image_build/image_build_zen_cpu.sh "$REGISTRY" "$REPO" "$BUILDKITE_COMMIT"
|
||||
env:
|
||||
DOCKER_BUILDKIT: "1"
|
||||
# Install zentorch directly (if-branch in Dockerfile.zen) instead of
|
||||
# `uv pip install "vllm[zen]"`, which forces uv to re-resolve vLLM's full
|
||||
# dependency closure and chokes on its transitive triton-cpu URL dependency.
|
||||
# Keep this pin in sync with the `zen` extra in setup.py.
|
||||
ZENTORCH_VERSION: "2.13.0.1"
|
||||
retry:
|
||||
automatic:
|
||||
- exit_status: 1 # Transient Docker/BuildKit failure
|
||||
limit: 1
|
||||
- exit_status: -1 # Agent was lost
|
||||
limit: 2
|
||||
- exit_status: -10 # Agent was lost
|
||||
limit: 2
|
||||
|
||||
@@ -2,10 +2,15 @@ group: Hardware
|
||||
steps:
|
||||
- label: "GH200 Test"
|
||||
key: gh200-test
|
||||
timeout_in_minutes: 30
|
||||
soft_fail: true
|
||||
device: gh200
|
||||
no_plugin: true
|
||||
optional: true
|
||||
commands:
|
||||
- nvidia-smi
|
||||
- bash .buildkite/scripts/hardware_ci/run-gh200-test.sh
|
||||
working_dir: "/vllm-workspace"
|
||||
depends_on:
|
||||
- arm64-image-build
|
||||
env:
|
||||
VLLM_WORKER_MULTIPROC_METHOD: "spawn"
|
||||
commands:
|
||||
- nvidia-smi
|
||||
- python3 examples/basic/offline_inference/generate.py --model meta-llama/Llama-3.2-1B
|
||||
|
||||
@@ -61,6 +61,8 @@ NIGHTLY_UBUNTU_VERSION="24.04"
|
||||
NIGHTLY_FINAL_BASE_IMAGE="nvidia/cuda:${NIGHTLY_CUDA_VERSION}-base-ubuntu${NIGHTLY_UBUNTU_VERSION}"
|
||||
|
||||
echo "--- :docker: Building torch nightly image (CUDA ${NIGHTLY_CUDA_VERSION})"
|
||||
# Provenance labels match the bake `_labels` target in docker/docker-bake.hcl;
|
||||
# the initialized-snapshot E2E refuses images whose commit labels differ.
|
||||
docker buildx build --file docker/Dockerfile \
|
||||
--build-arg max_jobs=16 \
|
||||
--build-arg buildkite_commit="$BUILDKITE_COMMIT" \
|
||||
@@ -70,6 +72,10 @@ docker buildx build --file docker/Dockerfile \
|
||||
--build-arg BUILD_BASE_IMAGE="${NIGHTLY_BUILD_BASE_IMAGE}" \
|
||||
--build-arg FINAL_BASE_IMAGE="${NIGHTLY_FINAL_BASE_IMAGE}" \
|
||||
--build-arg torch_cuda_arch_list="8.0 8.9 9.0 10.0 12.0" \
|
||||
--label "org.opencontainers.image.revision=${BUILDKITE_COMMIT}" \
|
||||
--label "ai.vllm.build.commit=${BUILDKITE_COMMIT}" \
|
||||
--label "ai.vllm.build.pipeline=${BUILDKITE_PIPELINE_SLUG:-local}" \
|
||||
--label "ai.vllm.build.url=${BUILDKITE_BUILD_URL:-}" \
|
||||
--tag "$IMAGE_TAG" \
|
||||
--push \
|
||||
--target test \
|
||||
|
||||
+75
@@ -0,0 +1,75 @@
|
||||
#!/bin/bash
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
#
|
||||
# Build the AMD Zen CPU image (vLLM + zentorch) by layering docker/Dockerfile.zen
|
||||
# on top of a CPU base image that already has vLLM installed.
|
||||
#
|
||||
# To avoid recompiling vLLM from source, prefer reusing the shared CPU image that
|
||||
# the image-build-cpu step already builds and publishes for this commit
|
||||
# (`<repo>:<commit>-cpu`). It lives in public ECR, which allows anonymous pulls,
|
||||
# so no registry credentials are required. If the pull fails (e.g. the image
|
||||
# isn't published yet), fall back to building the base from docker/Dockerfile.cpu.
|
||||
#
|
||||
# The zen image is not pushed to a registry.
|
||||
#
|
||||
# See docker/Dockerfile.zen for the build workflow this mirrors.
|
||||
set -euo pipefail
|
||||
|
||||
if [[ $# -lt 3 ]]; then
|
||||
echo "Usage: $0 <registry> <repo> <commit>"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
REGISTRY=$1
|
||||
REPO=$2
|
||||
BUILDKITE_COMMIT=$3
|
||||
|
||||
image_repo() {
|
||||
if [[ -n "${REGISTRY}" ]]; then
|
||||
printf '%s/%s' "${REGISTRY}" "${REPO}"
|
||||
else
|
||||
printf '%s' "${REPO}"
|
||||
fi
|
||||
}
|
||||
|
||||
IMAGE_REPO="$(image_repo)"
|
||||
|
||||
# The shared CPU image published by image-build-cpu (public ECR, anonymous pull).
|
||||
SHARED_CPU_IMAGE=""
|
||||
if [[ -n "${REGISTRY}" ]]; then
|
||||
SHARED_CPU_IMAGE="$REGISTRY/$REPO:$BUILDKITE_COMMIT-cpu"
|
||||
fi
|
||||
FALLBACK_BASE_IMAGE="zen-cpu-base:$BUILDKITE_COMMIT"
|
||||
IMAGE="$IMAGE_REPO:$BUILDKITE_COMMIT-zen-cpu"
|
||||
|
||||
# ZENTORCH_VERSION is optional; when unset the Dockerfile falls back to
|
||||
# installing zentorch via `vllm[zen]`.
|
||||
ZENTORCH_VERSION=${ZENTORCH_VERSION:-}
|
||||
|
||||
# Step 1: obtain the CPU base image that Dockerfile.zen layers on. Prefer pulling
|
||||
# the published `-cpu` image; fall back to building it from source if unavailable.
|
||||
if [[ -n "${SHARED_CPU_IMAGE}" ]] && docker pull "$SHARED_CPU_IMAGE"; then
|
||||
echo "--- :docker: Using published CPU image as base: $SHARED_CPU_IMAGE"
|
||||
BASE_IMAGE="$SHARED_CPU_IMAGE"
|
||||
else
|
||||
echo "--- :docker: Published CPU image unavailable; building base from source"
|
||||
docker build --file docker/Dockerfile.cpu \
|
||||
--platform linux/amd64 \
|
||||
--build-arg buildkite_commit="$BUILDKITE_COMMIT" \
|
||||
--build-arg VLLM_CPU_X86=true \
|
||||
--tag "$FALLBACK_BASE_IMAGE" \
|
||||
--target vllm-openai \
|
||||
--progress plain .
|
||||
BASE_IMAGE="$FALLBACK_BASE_IMAGE"
|
||||
fi
|
||||
|
||||
# Step 2: build the zen test image on top of the CPU base.
|
||||
echo "--- :docker: Building Zen test image"
|
||||
docker build --file docker/Dockerfile.zen \
|
||||
--platform linux/amd64 \
|
||||
--build-arg BASE_IMAGE="$BASE_IMAGE" \
|
||||
${ZENTORCH_VERSION:+--build-arg ZENTORCH_VERSION="$ZENTORCH_VERSION"} \
|
||||
--tag "$IMAGE" \
|
||||
--target vllm-zen-test \
|
||||
--progress plain .
|
||||
@@ -19,14 +19,14 @@ steps:
|
||||
source_file_dependencies:
|
||||
- vllm/entrypoints
|
||||
- tests/entrypoints/unit_tests
|
||||
- tests/entrypoints/weight_transfer
|
||||
- tests/entrypoints/rl/weight_transfer
|
||||
- tests/entrypoints/launchers
|
||||
commands:
|
||||
- >-
|
||||
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
|
||||
'cd tests &&
|
||||
pytest -v -s entrypoints/unit_tests &&
|
||||
pytest -v -s entrypoints/weight_transfer'
|
||||
pytest -v -s entrypoints/rl/weight_transfer'
|
||||
|
||||
- label: ":intel: (BMG) Entrypoints Integration (LLM)"
|
||||
key: entrypoints-integration-llm-intel
|
||||
@@ -71,14 +71,17 @@ steps:
|
||||
- vllm/
|
||||
- "!vllm/distributed/kv_transfer/"
|
||||
- tests/entrypoints/serve
|
||||
- tests/entrypoints/rl
|
||||
commands:
|
||||
- >-
|
||||
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
|
||||
'export VLLM_WORKER_MULTIPROC_METHOD=spawn &&
|
||||
cd tests &&
|
||||
pytest -v -s entrypoints/serve --ignore=entrypoints/serve/dev/rpc
|
||||
--ignore=entrypoints/serve/dev/rlhf/test_weight_operation_metrics.py
|
||||
--deselect="tests/entrypoints/serve/dev/rlhf/state_transitions/test_pause_resume.py::TestPauseResume::test_clear_cache_preserves_output_and_controls_prefix_cache[MRV1]" &&
|
||||
pytest -v -s entrypoints/serve --ignore=entrypoints/serve/dev/rpc &&
|
||||
pytest -v -s entrypoints/rl --ignore=entrypoints/rl/weight_transfer
|
||||
--ignore=entrypoints/rl/return_routed_experts/test_openai_api.py
|
||||
--ignore=entrypoints/rl/weight_transfer_metrics
|
||||
--deselect="tests/entrypoints/rl/state_transitions/test_pause_resume.py::TestPauseResume::test_clear_cache_preserves_output_and_controls_prefix_cache[MRV1]" &&
|
||||
PYTHONPATH=/workspace/vllm pytest -v -s entrypoints/serve/dev/rpc'
|
||||
|
||||
- label: ":intel: (BMG) Entrypoints Integration (API Server Scale Out)"
|
||||
|
||||
@@ -9,7 +9,7 @@ steps:
|
||||
agent_tags:
|
||||
label: production
|
||||
gpu: 1+
|
||||
mem: 24+
|
||||
mem: 16+
|
||||
no_plugin: true
|
||||
working_dir: "."
|
||||
env:
|
||||
@@ -33,7 +33,6 @@ steps:
|
||||
pytest -v -s lora/test_peft_helper.py &&
|
||||
pytest -v -s lora/test_resolver.py &&
|
||||
pytest -v -s lora/test_utils.py &&
|
||||
pytest -v -s lora/test_add_lora.py &&
|
||||
pytest -v -s lora/test_worker.py'
|
||||
|
||||
- label: LoRA Fused/MoE Kernels
|
||||
@@ -120,7 +119,7 @@ steps:
|
||||
agent_tags:
|
||||
label: production
|
||||
gpu: 2+
|
||||
mem: 24+
|
||||
mem: 16+
|
||||
no_plugin: true
|
||||
working_dir: "."
|
||||
env:
|
||||
@@ -138,8 +137,7 @@ steps:
|
||||
pytest -v -s lora/test_quant_model.py --deselect="tests/lora/test_quant_model.py::test_quant_model_lora[model0]" --deselect="tests/lora/test_quant_model.py::test_quant_model_lora[model1]" --deselect="tests/lora/test_quant_model.py::test_quant_model_tp_equality[model0]" &&
|
||||
pytest -v -s lora/test_transformers_model.py &&
|
||||
pytest -v -s lora/test_chatglm3_tp.py &&
|
||||
pytest -v -s lora/test_llama_tp.py::test_llama_lora &&
|
||||
pytest -s -v lora/test_minicpmv_tp.py'
|
||||
pytest -v -s lora/test_llama_tp.py::test_llama_lora'
|
||||
|
||||
- label: LoRA Multimodal
|
||||
key: lora-multimodal
|
||||
@@ -148,6 +146,30 @@ steps:
|
||||
agent_tags:
|
||||
label: production
|
||||
gpu: 1+
|
||||
mem: 16+
|
||||
no_plugin: true
|
||||
working_dir: "."
|
||||
env:
|
||||
REGISTRY: "public.ecr.aws/q9t5s3a7"
|
||||
REPO: "vllm-ci-test-repo"
|
||||
VLLM_TEST_DEVICE: "xpu"
|
||||
source_file_dependencies:
|
||||
- vllm/lora
|
||||
- tests/lora
|
||||
commands:
|
||||
- >-
|
||||
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
|
||||
'cd tests &&
|
||||
export VLLM_WORKER_MULTIPROC_METHOD=spawn &&
|
||||
pytest -v -s lora/test_whisper.py'
|
||||
|
||||
- label: LoRA Add + Multimodal
|
||||
key: lora-add-multimodal
|
||||
timeout_in_minutes: 45
|
||||
device: intel_gpu
|
||||
agent_tags:
|
||||
label: production
|
||||
gpu: 2+
|
||||
mem: 24+
|
||||
no_plugin: true
|
||||
working_dir: "."
|
||||
@@ -163,5 +185,6 @@ steps:
|
||||
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
|
||||
'cd tests &&
|
||||
export VLLM_WORKER_MULTIPROC_METHOD=spawn &&
|
||||
pytest -v -s lora/test_default_mm_loras.py &&
|
||||
pytest -v -s lora/test_whisper.py'
|
||||
pytest -v -s lora/test_add_lora.py &&
|
||||
pytest -v -s lora/test_default_mm_loras.py &&
|
||||
pytest -s -v lora/test_minicpmv_tp.py'
|
||||
|
||||
@@ -10,7 +10,7 @@ steps:
|
||||
agent_tags:
|
||||
label: production
|
||||
gpu: 2+
|
||||
mem: 24+
|
||||
mem: 16+
|
||||
no_plugin: true
|
||||
working_dir: "."
|
||||
env:
|
||||
@@ -34,6 +34,40 @@ steps:
|
||||
key: v1-core-kv-metrics-intel
|
||||
timeout_in_minutes: 45
|
||||
device: intel_gpu
|
||||
agent_tags:
|
||||
label: production
|
||||
gpu: 1+
|
||||
mem: 16+
|
||||
no_plugin: true
|
||||
working_dir: "."
|
||||
env:
|
||||
REGISTRY: "public.ecr.aws/q9t5s3a7"
|
||||
REPO: "vllm-ci-test-repo"
|
||||
VLLM_TEST_DEVICE: "xpu"
|
||||
source_file_dependencies:
|
||||
- vllm/
|
||||
- tests/v1/core
|
||||
- tests/v1/executor
|
||||
- tests/v1/kv_offload
|
||||
- tests/v1/worker
|
||||
- tests/v1/kv_connector/unit
|
||||
- tests/v1/ec_connector/unit
|
||||
- tests/v1/metrics
|
||||
- tests/entrypoints/openai/correctness/test_lmeval.py
|
||||
commands:
|
||||
- >-
|
||||
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
|
||||
'pip install -r requirements/kv_connectors.txt &&
|
||||
export VLLM_WORKER_MULTIPROC_METHOD=spawn &&
|
||||
cd tests &&
|
||||
pytest -v -s v1/executor &&
|
||||
VLLM_BATCH_INVARIANT=1 pytest -v -s -m "not cpu_test" v1/ec_connector/unit
|
||||
--ignore=v1/ec_connector/unit/test_ec_cpu_connector.py'
|
||||
|
||||
- label: EC CPU Connector (XPU)
|
||||
key: ec-cpu-connector-xpu
|
||||
timeout_in_minutes: 45
|
||||
device: intel_gpu
|
||||
agent_tags:
|
||||
label: production
|
||||
gpu: 1+
|
||||
@@ -60,8 +94,7 @@ steps:
|
||||
'pip install -r requirements/kv_connectors.txt &&
|
||||
export VLLM_WORKER_MULTIPROC_METHOD=spawn &&
|
||||
cd tests &&
|
||||
pytest -v -s v1/executor &&
|
||||
VLLM_BATCH_INVARIANT=1 pytest -v -s -m "not cpu_test" v1/ec_connector/unit'
|
||||
pytest -v -s v1/ec_connector/unit/test_ec_cpu_connector.py'
|
||||
|
||||
- label: V1 Sample + Logits
|
||||
key: v1-sample-logits-intel
|
||||
@@ -70,7 +103,7 @@ steps:
|
||||
agent_tags:
|
||||
label: production
|
||||
gpu: 1+
|
||||
mem: 24+
|
||||
mem: 16+
|
||||
no_plugin: true
|
||||
working_dir: "."
|
||||
env:
|
||||
@@ -265,7 +298,7 @@ steps:
|
||||
agent_tags:
|
||||
label: production
|
||||
gpu: 1+
|
||||
mem: 24+
|
||||
mem: 16+
|
||||
no_plugin: true
|
||||
working_dir: "."
|
||||
env:
|
||||
@@ -293,10 +326,47 @@ steps:
|
||||
- >-
|
||||
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
|
||||
'cd tests &&
|
||||
pytest -v -s detokenizer &&
|
||||
pytest -v -s detokenizer --ignore=detokenizer/test_stop_strings.py &&
|
||||
pytest -v -s -m "not cpu_test" ./multimodal &&
|
||||
pytest -v -s utils_ --ignore=utils_/test_mem_utils.py'
|
||||
|
||||
- label: Detokenizer Stop Strings (XPU)
|
||||
key: detokenizer-stop-strings-xpu
|
||||
timeout_in_minutes: 45
|
||||
device: intel_gpu
|
||||
agent_tags:
|
||||
label: production
|
||||
gpu: 1+
|
||||
mem: 24+
|
||||
no_plugin: true
|
||||
working_dir: "."
|
||||
env:
|
||||
REGISTRY: "public.ecr.aws/q9t5s3a7"
|
||||
REPO: "vllm-ci-test-repo"
|
||||
VLLM_TEST_DEVICE: "xpu"
|
||||
source_file_dependencies:
|
||||
- vllm/assets/
|
||||
- vllm/config/
|
||||
- vllm/distributed/
|
||||
- vllm/engine/
|
||||
- vllm/inputs/
|
||||
- vllm/model_executor/
|
||||
- vllm/multimodal/
|
||||
- vllm/platforms/
|
||||
- vllm/sampling_params.py
|
||||
- vllm/tokenizers/
|
||||
- vllm/transformers_utils/
|
||||
- vllm/utils/
|
||||
- vllm/v1/
|
||||
- tests/detokenizer
|
||||
- tests/multimodal
|
||||
- tests/utils_
|
||||
commands:
|
||||
- >-
|
||||
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
|
||||
'cd tests &&
|
||||
pytest -v -s detokenizer/test_stop_strings.py'
|
||||
|
||||
- label: Fusion Unit Tests
|
||||
key: fusion-unit-tests
|
||||
timeout_in_minutes: 30
|
||||
|
||||
@@ -6,6 +6,34 @@ steps:
|
||||
key: model-executor-intel
|
||||
timeout_in_minutes: 45
|
||||
device: intel_gpu
|
||||
agent_tags:
|
||||
label: production
|
||||
gpu: 1+
|
||||
mem: 16+
|
||||
no_plugin: true
|
||||
working_dir: "."
|
||||
env:
|
||||
REGISTRY: "public.ecr.aws/q9t5s3a7"
|
||||
REPO: "vllm-ci-test-repo"
|
||||
VLLM_TEST_DEVICE: "xpu"
|
||||
source_file_dependencies:
|
||||
- vllm/engine/arg_utils.py
|
||||
- vllm/config/model.py
|
||||
- vllm/model_executor
|
||||
- tests/model_executor
|
||||
commands:
|
||||
- >-
|
||||
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
|
||||
'apt-get update && apt-get install -y curl libsodium23 &&
|
||||
export VLLM_WORKER_MULTIPROC_METHOD=spawn &&
|
||||
export PYTHONFAULTHANDLER=1 &&
|
||||
cd tests &&
|
||||
pytest -v -s model_executor -m "not slow_test" --ignore="model_executor/layers/test_rocm_unquantized_gemm.py" --ignore="model_executor/model_loader/tensorizer_loader/test_tensorizer.py" --deselect="tests/model_executor/model_loader/test_reload.py::test_kv_scale_reload"'
|
||||
|
||||
- label: Tensorizer Model Loader Tests (XPU)
|
||||
key: tensorizer-model-loader-tests-xpu
|
||||
timeout_in_minutes: 45
|
||||
device: intel_gpu
|
||||
agent_tags:
|
||||
label: production
|
||||
gpu: 1+
|
||||
@@ -28,13 +56,4 @@ steps:
|
||||
export VLLM_WORKER_MULTIPROC_METHOD=spawn &&
|
||||
export PYTHONFAULTHANDLER=1 &&
|
||||
cd tests &&
|
||||
pytest -v -s model_executor -m "not slow_test" --ignore="model_executor/layers/test_rocm_unquantized_gemm.py" --deselect="tests/model_executor/model_loader/test_reload.py::test_kv_scale_reload"
|
||||
--deselect="tests/model_executor/layers/test_mm_input_norm.py::TestFusedMMInputNormModule::test_matches_reference_above_cudnn_grid_limit"
|
||||
--deselect="tests/model_executor/layers/test_mm_input_norm.py::TestFusedMMInputNormDtypes::test_dtype_combinations[in_dtype0-out_dtype0]"
|
||||
--deselect="tests/model_executor/layers/test_mm_input_norm.py::TestFusedMMInputNormDtypes::test_dtype_combinations[in_dtype1-out_dtype1]"
|
||||
--deselect="tests/model_executor/layers/test_mm_input_norm.py::TestFusedMMInputNormDtypes::test_dtype_combinations[in_dtype2-out_dtype2]"
|
||||
--deselect="tests/model_executor/layers/test_mm_input_norm.py::TestFusedMMInputNormDtypes::test_dtype_combinations[in_dtype3-out_dtype3]"
|
||||
--deselect="tests/model_executor/layers/test_mm_input_norm.py::TestFusedMMInputNormDtypes::test_dtype_combinations[in_dtype4-out_dtype4]"
|
||||
--deselect="tests/model_executor/layers/test_mm_input_norm.py::TestFusedMMInputNormDtypes::test_dtype_combinations[in_dtype5-out_dtype5]"
|
||||
--deselect="tests/model_executor/layers/test_mm_input_norm.py::TestFusedMMInputNormShapes"
|
||||
--deselect="tests/model_executor/layers/test_mm_input_norm.py::TestFusedMMInputNormInputHandling"'
|
||||
pytest -v -s model_executor/model_loader/tensorizer_loader/test_tensorizer.py -m "not slow_test"'
|
||||
|
||||
@@ -9,7 +9,7 @@ steps:
|
||||
agent_tags:
|
||||
label: production
|
||||
gpu: 1+
|
||||
mem: 24+
|
||||
mem: 16+
|
||||
no_plugin: true
|
||||
working_dir: "."
|
||||
env:
|
||||
@@ -33,7 +33,7 @@ steps:
|
||||
agent_tags:
|
||||
label: production
|
||||
gpu: 1+
|
||||
mem: 24+
|
||||
mem: 16+
|
||||
no_plugin: true
|
||||
working_dir: "."
|
||||
env:
|
||||
@@ -72,7 +72,9 @@ steps:
|
||||
- >-
|
||||
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
|
||||
'cd tests &&
|
||||
pytest -v -s models/multimodal/generation/test_common.py -m core_model -k "not qwen2 and not qwen3 and not gemma"'
|
||||
pytest -v -s models/multimodal/generation/test_common.py -m core_model -k "not qwen2 and not qwen3 and not gemma" &&
|
||||
pytest -v -s models/multimodal/generation/test_gemma_ragged_audio.py -m core_model
|
||||
pytest -v -s models/multimodal/generation/test_qwen2_vl.py -m core_model'
|
||||
|
||||
- label: "Multi-Modal Models (Standard) 4: other + whisper"
|
||||
key: multi-modal-models-standard-4-other-whisper
|
||||
@@ -81,7 +83,7 @@ steps:
|
||||
agent_tags:
|
||||
label: production
|
||||
gpu: 1+
|
||||
mem: 24+
|
||||
mem: 16+
|
||||
no_plugin: true
|
||||
working_dir: "."
|
||||
env:
|
||||
@@ -95,7 +97,7 @@ steps:
|
||||
- >-
|
||||
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
|
||||
'cd tests &&
|
||||
pytest -v -s models/multimodal -m core_model --ignore models/multimodal/generation/test_common.py --ignore models/multimodal/generation/test_ultravox.py --ignore models/multimodal/generation/test_qwen2_5_vl.py --ignore models/multimodal/generation/test_qwen2_vl.py --ignore models/multimodal/generation/test_whisper.py --ignore models/multimodal/generation/test_memory_leak.py --ignore models/multimodal/processing'
|
||||
pytest -v -s models/multimodal -m core_model --ignore models/multimodal/generation/test_common.py --ignore models/multimodal/generation/test_ultravox.py --ignore models/multimodal/generation/test_qwen2_5_vl.py --ignore models/multimodal/generation/test_qwen2_vl.py --ignore models/multimodal/generation/test_whisper.py --ignore models/multimodal/generation/test_gemma_ragged_audio.py --ignore models/multimodal/generation/test_memory_leak.py --ignore models/multimodal/processing'
|
||||
|
||||
- label: Multi-Modal Processor # 44min
|
||||
key: multi-modal-processor
|
||||
@@ -131,7 +133,7 @@ steps:
|
||||
agent_tags:
|
||||
label: production
|
||||
gpu: 1+
|
||||
mem: 24+
|
||||
mem: 16+
|
||||
no_plugin: true
|
||||
working_dir: "."
|
||||
env:
|
||||
|
||||
@@ -25,5 +25,7 @@ steps:
|
||||
- >-
|
||||
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
|
||||
'VLLM_TEST_FORCE_LOAD_FORMAT=auto pytest -v -s tests/quantization/test_per_token_kv_cache.py --deselect="tests/quantization/test_per_token_kv_cache.py::test_triton_unified_attention_per_token_head_scale[int4-16-128-num_heads0-seq_lens1]"
|
||||
&& VLLM_TEST_FORCE_LOAD_FORMAT=auto pytest -v -s tests/quantization/test_auto_awq.py'
|
||||
&& VLLM_TEST_FORCE_LOAD_FORMAT=auto pytest -v -s tests/quantization/test_auto_awq.py
|
||||
&& VLLM_TEST_FORCE_LOAD_FORMAT=auto pytest -v -s tests/quantization/test_fp8_per_channel.py
|
||||
&& VLLM_TEST_FORCE_LOAD_FORMAT=auto pytest -v -s tests/models/quantization/test_fp8_per_channel.py --deselect="tests/models/quantization/test_fp8_per_channel.py::test_fp8_per_channel_logprobs[moe]"'
|
||||
|
||||
|
||||
@@ -6,6 +6,33 @@ steps:
|
||||
key: samplers-test-xpu-kernel-intel
|
||||
timeout_in_minutes: 40
|
||||
device: intel_gpu
|
||||
agent_tags:
|
||||
label: production
|
||||
gpu: 1+
|
||||
mem: 16+
|
||||
no_plugin: true
|
||||
working_dir: "."
|
||||
env:
|
||||
REGISTRY: "public.ecr.aws/q9t5s3a7"
|
||||
REPO: "vllm-ci-test-repo"
|
||||
VLLM_TEST_DEVICE: "xpu"
|
||||
source_file_dependencies:
|
||||
- vllm/model_executor/layers
|
||||
- vllm/sampling_metadata.py
|
||||
- tests/samplers
|
||||
- tests/conftest.py
|
||||
- vllm/entrypoints/generate/beam_search
|
||||
commands:
|
||||
- >-
|
||||
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
|
||||
'cd tests &&
|
||||
VLLM_XPU_USE_SAMPLER_KERNEL=1 pytest -v -s samplers
|
||||
--ignore=samplers/test_beam_search.py'
|
||||
|
||||
- label: Samplers Beam Search Test (XPU Kernel)
|
||||
key: samplers-beam-search-test-xpu-kernel-intel
|
||||
timeout_in_minutes: 40
|
||||
device: intel_gpu
|
||||
agent_tags:
|
||||
label: production
|
||||
gpu: 1+
|
||||
@@ -26,4 +53,4 @@ steps:
|
||||
- >-
|
||||
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
|
||||
'cd tests &&
|
||||
VLLM_XPU_USE_SAMPLER_KERNEL=1 pytest -v -s samplers'
|
||||
VLLM_XPU_USE_SAMPLER_KERNEL=1 pytest -v -s samplers/test_beam_search.py'
|
||||
|
||||
@@ -23,7 +23,7 @@ steps:
|
||||
agent_tags:
|
||||
label: production
|
||||
gpu: 2+
|
||||
mem: 24+
|
||||
mem: 16+
|
||||
no_plugin: true
|
||||
env:
|
||||
REGISTRY: "public.ecr.aws/q9t5s3a7"
|
||||
@@ -46,7 +46,6 @@ steps:
|
||||
python3 examples/basic/offline_inference/generate.py --model ibm-research/PowerMoE-3b --block-size 64 --enforce-eager -tp 2 &&
|
||||
python3 examples/basic/offline_inference/generate.py --model ibm-research/PowerMoE-3b --block-size 64 --enforce-eager -tp 2 --enable-expert-parallel &&
|
||||
python3 examples/basic/offline_inference/generate.py --model superjob/Qwen3-4B-Instruct-2507-GPTQ-Int4 --max-model-len 8192 &&
|
||||
VLLM_XPU_FUSED_MOE_USE_REF=1 python3 examples/basic/offline_inference/generate.py --model Qwen/Qwen3-30B-A3B-Instruct-2507-FP8 --enforce-eager -tp 2 --max-model-len 8192 &&
|
||||
python3 examples/basic/offline_inference/generate.py --model INCModel/Qwen3-30B-A3B-Instruct-2507-MXFP4-LLMC -tp 2 --max-model-len 8192
|
||||
'
|
||||
- label: "XPU W8A8 FP8 Linear Examples"
|
||||
@@ -58,7 +57,7 @@ steps:
|
||||
agent_tags:
|
||||
label: production
|
||||
gpu: 1+
|
||||
mem: 24+
|
||||
mem: 16+
|
||||
no_plugin: true
|
||||
env:
|
||||
REGISTRY: "public.ecr.aws/q9t5s3a7"
|
||||
@@ -86,7 +85,7 @@ steps:
|
||||
agent_tags:
|
||||
label: production
|
||||
gpu: 1+
|
||||
mem: 24+
|
||||
mem: 16+
|
||||
no_plugin: true
|
||||
env:
|
||||
REGISTRY: "public.ecr.aws/q9t5s3a7"
|
||||
@@ -106,8 +105,34 @@ steps:
|
||||
pytest -v -s v1/test_serial_utils.py &&
|
||||
pytest -v -s v1/e2e/general/test_correctness_sliding_window.py &&
|
||||
pytest -v -s v1/spec_decode --ignore=v1/spec_decode/test_max_len.py --ignore=v1/spec_decode/test_speculators_eagle3.py --ignore=v1/spec_decode/test_acceptance_length.py --ignore=v1/spec_decode/test_speculators_correctness.py --deselect=tests/v1/spec_decode/test_mtp.py::test_glm_mtp_defers_lm_head &&
|
||||
pytest -v -s v1/kv_connector/unit --ignore=v1/kv_connector/unit/test_lmcache_integration.py --ignore=v1/kv_connector/unit/test_hf3fs_client.py --ignore=v1/kv_connector/unit/test_hf3fs_connector.py --ignore=v1/kv_connector/unit/test_hf3fs_metadata_server.py --ignore=v1/kv_connector/unit/test_offloading_connector.py --ignore=v1/kv_connector/unit/test_nixl_connector_hma.py --deselect=tests/v1/kv_connector/unit/test_nixl_connector.py::test_abort_timeout_on_prefiller[ray]'
|
||||
pytest -v -s v1/kv_connector/unit --ignore=v1/kv_connector/unit/test_lmcache_integration.py --ignore=v1/kv_connector/unit/test_hf3fs_client.py --ignore=v1/kv_connector/unit/test_hf3fs_connector.py --ignore=v1/kv_connector/unit/test_hf3fs_metadata_server.py --ignore=v1/kv_connector/unit/test_offloading_connector.py --ignore=v1/kv_connector/unit/test_nixl_connector_hma.py --ignore=v1/kv_connector/unit/test_multi_connector.py --ignore=v1/kv_connector/unit/test_example_connector.py --deselect=tests/v1/kv_connector/unit/test_nixl_connector.py::test_abort_timeout_on_prefiller[ray]'
|
||||
parallelism: 2
|
||||
- label: "XPU Qwen3-30B + V1 KV Connector"
|
||||
key: xpu-qwen3-30b-v1-kv-connector
|
||||
depends_on:
|
||||
- image-build-xpu
|
||||
timeout_in_minutes: 90
|
||||
device: intel_gpu
|
||||
agent_tags:
|
||||
label: production
|
||||
gpu: 2+
|
||||
mem: 24+
|
||||
no_plugin: true
|
||||
env:
|
||||
REGISTRY: "public.ecr.aws/q9t5s3a7"
|
||||
REPO: "vllm-ci-test-repo"
|
||||
source_file_dependencies:
|
||||
- vllm/
|
||||
- .buildkite/intel_jobs/test-intel.yaml
|
||||
commands:
|
||||
- >-
|
||||
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
|
||||
'cd tests &&
|
||||
pip install dpctl &&
|
||||
pytest -v -s v1/kv_connector/unit/test_multi_connector.py &&
|
||||
pytest -v -s v1/kv_connector/unit/test_example_connector.py &&
|
||||
cd .. &&
|
||||
VLLM_XPU_FUSED_MOE_USE_REF=1 python3 examples/basic/offline_inference/generate.py --model Qwen/Qwen3-30B-A3B-Instruct-2507-FP8 --enforce-eager -tp 2 --max-model-len 8192'
|
||||
- label: "XPU server test"
|
||||
key: xpu-server-test
|
||||
depends_on:
|
||||
@@ -153,7 +178,7 @@ steps:
|
||||
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
|
||||
'cd tests &&
|
||||
pytest -v -s quantization/test_auto_round.py &&
|
||||
pytest -v -s quantization/test_online.py --deselect=tests/quantization/test_online.py::test_online_quantization_loads_real_weights'
|
||||
pytest -v -s quantization/test_online.py --deselect=tests/quantization/test_online.py::test_online_quantization_loads_real_weights --deselect="tests/quantization/test_online.py::test_online_quantization[False-requantization_mxfp8_ptcp_fp8]"'
|
||||
- label: "XPU compressed tensors FP8 test"
|
||||
key: xpu-compressed-tensors-fp8-test
|
||||
depends_on:
|
||||
|
||||
@@ -8,16 +8,28 @@ pytest -s -v test_lm_eval_correctness.py \
|
||||
--tp-size=1
|
||||
"""
|
||||
|
||||
import gc
|
||||
import os
|
||||
import time
|
||||
from contextlib import contextmanager
|
||||
|
||||
import lm_eval
|
||||
import pytest
|
||||
import torch
|
||||
import yaml
|
||||
from lm_eval.api.registry import get_model
|
||||
from lm_eval.utils import simple_parse_args_string
|
||||
|
||||
from vllm import LLM
|
||||
from vllm.distributed import cleanup_dist_env_and_memory
|
||||
from vllm.logger import init_logger
|
||||
from vllm.platforms import current_platform
|
||||
|
||||
logger = init_logger(__name__)
|
||||
|
||||
DEFAULT_RTOL = 0.08
|
||||
ROCM_ENGINE_SHUTDOWN_TIMEOUT_S = 60.0
|
||||
MEMORY_RELEASE_TIMEOUT_S = 240.0
|
||||
|
||||
|
||||
@contextmanager
|
||||
@@ -46,6 +58,48 @@ def scoped_env_vars(new_env: dict[str, str]):
|
||||
os.environ.pop(key, None)
|
||||
|
||||
|
||||
def _wait_for_memory_release(gpu_memory_utilization: float) -> None:
|
||||
devices = range(torch.accelerator.device_count())
|
||||
deadline = time.monotonic() + MEMORY_RELEASE_TIMEOUT_S
|
||||
while True:
|
||||
memory = [torch.accelerator.get_memory_info(d) for d in devices]
|
||||
if all(free >= total * gpu_memory_utilization for free, total in memory):
|
||||
return
|
||||
if time.monotonic() >= deadline:
|
||||
usage = ", ".join(
|
||||
f"{d}: {free / 2**30:.2f}/{total / 2**30:.2f} GiB free"
|
||||
for d, (free, total) in zip(devices, memory)
|
||||
)
|
||||
logger.warning(
|
||||
"GPU memory not released after %ss (%s); the next model needs %.2f",
|
||||
MEMORY_RELEASE_TIMEOUT_S,
|
||||
usage,
|
||||
gpu_memory_utilization,
|
||||
)
|
||||
return
|
||||
time.sleep(1)
|
||||
|
||||
|
||||
def _shutdown_lm(lm) -> None:
|
||||
if not current_platform.is_rocm():
|
||||
return
|
||||
llm = getattr(lm, "model", None)
|
||||
if not isinstance(llm, LLM):
|
||||
return
|
||||
gpu_memory_utilization = (
|
||||
llm.llm_engine.vllm_config.cache_config.gpu_memory_utilization
|
||||
)
|
||||
try:
|
||||
llm.llm_engine.engine_core.shutdown(timeout=ROCM_ENGINE_SHUTDOWN_TIMEOUT_S)
|
||||
except Exception:
|
||||
logger.exception("Engine core shutdown raised; GPU memory may leak")
|
||||
del llm
|
||||
lm.model = None
|
||||
gc.collect()
|
||||
cleanup_dist_env_and_memory()
|
||||
_wait_for_memory_release(gpu_memory_utilization)
|
||||
|
||||
|
||||
def launch_lm_eval(eval_config, tp_size):
|
||||
trust_remote_code = eval_config.get("trust_remote_code", False)
|
||||
max_model_len = eval_config.get("max_model_len", 4096)
|
||||
@@ -82,9 +136,17 @@ def launch_lm_eval(eval_config, tp_size):
|
||||
|
||||
env_vars = eval_config.get("env_vars", None)
|
||||
with scoped_env_vars(env_vars):
|
||||
if current_platform.is_rocm():
|
||||
model = get_model(backend).create_from_arg_string(
|
||||
model_args,
|
||||
{"batch_size": batch_size, "max_batch_size": None, "device": None},
|
||||
)
|
||||
model_kwargs = {"metadata": simple_parse_args_string(model_args)}
|
||||
else:
|
||||
model = backend
|
||||
model_kwargs = {"model_args": model_args}
|
||||
results = lm_eval.simple_evaluate(
|
||||
model=backend,
|
||||
model_args=model_args,
|
||||
model=model,
|
||||
tasks=[task["name"] for task in eval_config["tasks"]],
|
||||
num_fewshot=eval_config["num_fewshot"],
|
||||
limit=eval_config["limit"],
|
||||
@@ -98,7 +160,9 @@ def launch_lm_eval(eval_config, tp_size):
|
||||
# Forward decoding and early-stop controls (e.g., max_gen_toks, until=...)
|
||||
gen_kwargs=eval_config.get("gen_kwargs"),
|
||||
batch_size=batch_size,
|
||||
**model_kwargs,
|
||||
)
|
||||
_shutdown_lm(model)
|
||||
return results
|
||||
|
||||
|
||||
|
||||
@@ -355,6 +355,10 @@ steps:
|
||||
queue: arm64_cpu_queue_release
|
||||
commands:
|
||||
- "aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7"
|
||||
# TODO(arpera):
|
||||
# Remove BUILD_NIXL=true once NIXL has a wheel built against torch 2.15
|
||||
# For details: https://github.com/vllm-project/vllm/pull/60090
|
||||
# Tracking issue report: https://github.com/ai-dynamo/nixl/issues/2297
|
||||
- |
|
||||
DOCKER_BUILDKIT=1 docker buildx build --load \
|
||||
$(bash .buildkite/scripts/docker-build-metadata-args.sh cu134) \
|
||||
@@ -366,6 +370,7 @@ steps:
|
||||
--build-arg torch_cuda_arch_list="${CUDA_ARCH_AARCH64}" \
|
||||
--build-arg INSTALL_KV_CONNECTORS=true \
|
||||
--build-arg INSTALL_RUBIN_PRERELEASE=true \
|
||||
--build-arg BUILD_NIXL=true \
|
||||
--build-arg TRITON_INSTALL_FROM_SOURCE_REPO=https://github.com/triton-lang/triton.git \
|
||||
--build-arg TRITON_INSTALL_FROM_SOURCE_REVISION=3f6e41132b5edf639bfb872ad73d4688765e08b8 \
|
||||
--build-arg RUN_WHEEL_CHECK=false \
|
||||
@@ -1077,6 +1082,38 @@ steps:
|
||||
DOCKER_BUILDKIT: "1"
|
||||
DOCKERHUB_USERNAME: "vllmbot"
|
||||
|
||||
- trigger: "perf-eval"
|
||||
label: "Trigger perf-eval (CUDA)"
|
||||
key: trigger-perf-eval-cuda
|
||||
if: build.env("NIGHTLY") == "1" || build.env("RUN_PERF_EVAL") == "1"
|
||||
depends_on:
|
||||
- build-release-image-x86
|
||||
async: true
|
||||
soft_fail: true
|
||||
build:
|
||||
branch: "main"
|
||||
message: "(NIGHTLY=${NIGHTLY:-0}) Evaluate vLLM $BUILDKITE_COMMIT (CUDA)"
|
||||
env:
|
||||
NIGHTLY: 1 # run all perf-eval workloads
|
||||
VLLM_COMMIT: "$BUILDKITE_COMMIT"
|
||||
VLLM_IMAGE_CUDA: "public.ecr.aws/q9t5s3a7/vllm-release-repo:$BUILDKITE_COMMIT-x86_64"
|
||||
|
||||
- trigger: "perf-eval"
|
||||
label: "Trigger perf-eval (ROCm)"
|
||||
key: trigger-perf-eval-rocm
|
||||
if: build.env("NIGHTLY") == "1" || build.env("RUN_PERF_EVAL") == "1"
|
||||
depends_on:
|
||||
- build-rocm-release-image
|
||||
async: true
|
||||
soft_fail: true
|
||||
build:
|
||||
branch: "main"
|
||||
message: "(NIGHTLY=${NIGHTLY:-0}) Evaluate vLLM $BUILDKITE_COMMIT (ROCm)"
|
||||
env:
|
||||
NIGHTLY: 1 # run all perf-eval workloads
|
||||
VLLM_COMMIT: "$BUILDKITE_COMMIT"
|
||||
VLLM_IMAGE_ROCM: "public.ecr.aws/q9t5s3a7/vllm-release-repo:$BUILDKITE_COMMIT-rocm"
|
||||
|
||||
# =============================================================================
|
||||
# ROCk Release Pipeline (x86_64 only)
|
||||
# =============================================================================
|
||||
|
||||
@@ -25,6 +25,7 @@ export CMAKE_BUILD_PARALLEL_LEVEL="${CMAKE_BUILD_PARALLEL_LEVEL:-4}"
|
||||
|
||||
uv venv --python 3.12
|
||||
uv pip install -r requirements/build/cpu.txt --index-strategy unsafe-best-match
|
||||
uv build --wheel --no-build-isolation -o artifacts/dist
|
||||
uv build --wheel --no-build-isolation -o artifacts/dist \
|
||||
--config-setting=--build-option=--py-limited-api=cp38
|
||||
|
||||
ls -l artifacts/dist/*.whl
|
||||
|
||||
@@ -43,6 +43,7 @@ kernel_tests() {
|
||||
docker exec "$CONTAINER_NAME" bash -c "
|
||||
set -e
|
||||
pytest -x -v -s tests/kernels/test_onednn.py
|
||||
pytest -x -v -s tests/kernels/test_cpu_fused_sampling.py
|
||||
pytest -x -v -s tests/kernels/attention/test_cpu_attn.py
|
||||
pytest -x -v -s tests/kernels/core/test_cpu_activation.py
|
||||
pytest -x -v -s tests/kernels/moe/test_cpu_fused_moe.py
|
||||
|
||||
@@ -19,10 +19,9 @@ DISK_USAGE_THRESHOLD=${DISK_USAGE_THRESHOLD:-80}
|
||||
CACHE_MAX_AGE=${CACHE_MAX_AGE:-24h}
|
||||
|
||||
# Reclaim disk only when the host is under pressure, aging out anything unused
|
||||
# for less than CACHE_MAX_AGE so hot layers survive -- same `--filter
|
||||
# until=<N>h` pattern the TPU CI scripts already rely on. `docker buildx
|
||||
# prune --max-used-space` is a no-op on this host's `docker` driver (BuildKit
|
||||
# embedded in dockerd never enforces the size cap), so we don't use it.
|
||||
# for less than CACHE_MAX_AGE so hot layers survive (`--filter until=<N>h`).
|
||||
# `docker buildx prune --max-used-space` is a no-op on this host's `docker`
|
||||
# driver (BuildKit embedded in dockerd never enforces the size cap), so we don't use it.
|
||||
prune_if_disk_pressure() {
|
||||
local docker_root disk_usage
|
||||
docker_root=$(docker info -f '{{.DockerRootDir}}' 2>/dev/null || true)
|
||||
|
||||
@@ -1,29 +0,0 @@
|
||||
#!/bin/bash
|
||||
|
||||
# This script build the GH200 docker image and run the offline inference inside the container.
|
||||
# It serves a sanity check for compilation and basic model usage.
|
||||
set -ex
|
||||
|
||||
# Skip the new torch installation during build since we are using the specified version for arm64 in the Dockerfile
|
||||
python3 tools/use_existing_torch.py
|
||||
|
||||
# Try building the docker image
|
||||
DOCKER_BUILDKIT=1 docker build . \
|
||||
--file docker/Dockerfile \
|
||||
--target vllm-openai \
|
||||
--platform "linux/arm64" \
|
||||
-t gh200-test \
|
||||
--build-arg max_jobs=66 \
|
||||
--build-arg nvcc_threads=2 \
|
||||
--build-arg BUILD_BASE_IMAGE=pytorch/manylinuxaarch64-builder:cuda13.0-b8b5f17a7d9ccfc25bbc5cf17b3fcea12964a042 \
|
||||
--build-arg torch_cuda_arch_list="9.0+PTX"
|
||||
|
||||
# Setup cleanup
|
||||
remove_docker_container() { docker rm -f gh200-test || true; }
|
||||
trap remove_docker_container EXIT
|
||||
remove_docker_container
|
||||
|
||||
# Run the image and test offline inference
|
||||
docker run -e HF_TOKEN -e VLLM_WORKER_MULTIPROC_METHOD=spawn -v /root/.cache/huggingface:/root/.cache/huggingface --name gh200-test --gpus=all --entrypoint="" gh200-test bash -c '
|
||||
python3 examples/basic/offline_inference/generate.py --model meta-llama/Llama-3.2-1B
|
||||
'
|
||||
@@ -57,7 +57,7 @@ case "${test_suite}" in
|
||||
cd tests
|
||||
|
||||
pytest -v -s quantization/test_auto_round.py
|
||||
pytest -v -s quantization/test_online.py --deselect=tests/quantization/test_online.py::test_online_quantization_loads_real_weights
|
||||
pytest -v -s quantization/test_online.py --deselect=tests/quantization/test_online.py::test_online_quantization_loads_real_weights --deselect="tests/quantization/test_online.py::test_online_quantization[False-requantization_mxfp8_ptcp_fp8]"
|
||||
;;
|
||||
compressed-tensors-fp8)
|
||||
cd tests
|
||||
|
||||
@@ -1,166 +0,0 @@
|
||||
#!/bin/bash
|
||||
|
||||
set -xu
|
||||
|
||||
|
||||
remove_docker_container() {
|
||||
docker rm -f tpu-test || true;
|
||||
}
|
||||
|
||||
trap remove_docker_container EXIT
|
||||
|
||||
# Remove the container that might not be cleaned up in the previous run.
|
||||
remove_docker_container
|
||||
|
||||
# Build the docker image.
|
||||
docker build -f docker/Dockerfile.tpu -t vllm-tpu .
|
||||
|
||||
# Set up cleanup.
|
||||
cleanup_docker() {
|
||||
# Get Docker's root directory
|
||||
docker_root=$(docker info -f '{{.DockerRootDir}}')
|
||||
if [ -z "$docker_root" ]; then
|
||||
echo "Failed to determine Docker root directory."
|
||||
exit 1
|
||||
fi
|
||||
echo "Docker root directory: $docker_root"
|
||||
# Check disk usage of the filesystem where Docker's root directory is located
|
||||
disk_usage=$(df "$docker_root" | tail -1 | awk '{print $5}' | sed 's/%//')
|
||||
# Define the threshold
|
||||
threshold=70
|
||||
if [ "$disk_usage" -gt "$threshold" ]; then
|
||||
echo "Disk usage is above $threshold%. Cleaning up Docker images and volumes..."
|
||||
# Remove dangling images (those that are not tagged and not used by any container)
|
||||
docker image prune -f
|
||||
# Remove unused volumes / force the system prune for old images as well.
|
||||
docker volume prune -f && docker system prune --force --filter "until=72h" --all
|
||||
echo "Docker images and volumes cleanup completed."
|
||||
else
|
||||
echo "Disk usage is below $threshold%. No cleanup needed."
|
||||
fi
|
||||
}
|
||||
cleanup_docker
|
||||
|
||||
# For HF_TOKEN.
|
||||
source /etc/environment
|
||||
|
||||
docker run --privileged --net host --shm-size=16G -it \
|
||||
-e "HF_TOKEN=$HF_TOKEN" --name tpu-test \
|
||||
vllm-tpu /bin/bash -c '
|
||||
set -e # Exit immediately if a command exits with a non-zero status.
|
||||
set -u # Treat unset variables as an error.
|
||||
|
||||
echo "--- Starting script inside Docker container ---"
|
||||
|
||||
# Create results directory
|
||||
RESULTS_DIR=$(mktemp -d)
|
||||
# If mktemp fails, set -e will cause the script to exit.
|
||||
echo "Results will be stored in: $RESULTS_DIR"
|
||||
|
||||
# Install dependencies
|
||||
echo "--- Installing Python dependencies ---"
|
||||
python3 -m pip install --progress-bar off git+https://github.com/thuml/depyf.git \
|
||||
&& python3 -m pip install --progress-bar off pytest pytest-asyncio tpu-info \
|
||||
&& python3 -m pip install --progress-bar off "lm-eval[api]>=0.4.12" \
|
||||
&& python3 -m pip install --progress-bar off hf-transfer tblib==3.1.0
|
||||
echo "--- Python dependencies installed ---"
|
||||
|
||||
export VLLM_XLA_CHECK_RECOMPILATION=1
|
||||
export VLLM_XLA_CACHE_PATH=
|
||||
|
||||
echo "--- Hardware Information ---"
|
||||
# tpu-info
|
||||
echo "--- Starting Tests ---"
|
||||
set +e
|
||||
overall_script_exit_code=0
|
||||
|
||||
# --- Test Definitions ---
|
||||
# If a test fails, this function will print logs and will not cause the main script to exit.
|
||||
run_test() {
|
||||
local test_num=$1
|
||||
local test_name=$2
|
||||
local test_command=$3
|
||||
local log_file="$RESULTS_DIR/test_${test_num}.log"
|
||||
local actual_exit_code
|
||||
|
||||
echo "--- TEST_$test_num: Running $test_name ---"
|
||||
|
||||
# Execute the test command.
|
||||
eval "$test_command" > >(tee -a "$log_file") 2> >(tee -a "$log_file" >&2)
|
||||
actual_exit_code=$?
|
||||
|
||||
echo "TEST_${test_num}_COMMAND_EXIT_CODE: $actual_exit_code" # This goes to main log
|
||||
echo "TEST_${test_num}_COMMAND_EXIT_CODE: $actual_exit_code" >> "$log_file" # Also to per-test log
|
||||
|
||||
if [ "$actual_exit_code" -ne 0 ]; then
|
||||
echo "TEST_$test_num ($test_name) FAILED with exit code $actual_exit_code." >&2
|
||||
echo "--- Log for failed TEST_$test_num ($test_name) ---" >&2
|
||||
if [ -f "$log_file" ]; then
|
||||
cat "$log_file" >&2
|
||||
else
|
||||
echo "Log file $log_file not found for TEST_$test_num ($test_name)." >&2
|
||||
fi
|
||||
echo "--- End of log for TEST_$test_num ($test_name) ---" >&2
|
||||
return "$actual_exit_code" # Return the failure code
|
||||
else
|
||||
echo "TEST_$test_num ($test_name) PASSED."
|
||||
return 0 # Return success
|
||||
fi
|
||||
}
|
||||
|
||||
# Helper function to call run_test and update the overall script exit code
|
||||
run_and_track_test() {
|
||||
local test_num_arg="$1"
|
||||
local test_name_arg="$2"
|
||||
local test_command_arg="$3"
|
||||
|
||||
# Run the test
|
||||
run_test "$test_num_arg" "$test_name_arg" "$test_command_arg"
|
||||
local test_specific_exit_code=$?
|
||||
|
||||
# If the test failed, set the overall script exit code to 1
|
||||
if [ "$test_specific_exit_code" -ne 0 ]; then
|
||||
# No need for extra echo here, run_test already logged the failure.
|
||||
overall_script_exit_code=1
|
||||
fi
|
||||
}
|
||||
|
||||
# --- Actual Test Execution ---
|
||||
run_and_track_test 1 "test_struct_output_generate.py" \
|
||||
"python3 -m pytest -s -v /workspace/vllm/tests/entrypoints/llm/test_struct_output_generate.py -k \"not test_structured_output_with_reasoning_matrices\""
|
||||
run_and_track_test 2 "test_moe_pallas.py" \
|
||||
"python3 -m pytest -s -v /workspace/vllm/tests/tpu/test_moe_pallas.py"
|
||||
run_and_track_test 3 "test_lora.py" \
|
||||
"VLLM_XLA_CHECK_RECOMPILATION=0 python3 -m pytest -s -v /workspace/vllm/tests/tpu/lora/test_lora.py"
|
||||
run_and_track_test 4 "test_tpu_qkv_linear.py" \
|
||||
"python3 -m pytest -s -v /workspace/vllm/tests/v1/tpu/test_tpu_qkv_linear.py"
|
||||
run_and_track_test 5 "test_spmd_model_weight_loading.py" \
|
||||
"python3 -m pytest -s -v /workspace/vllm/tests/v1/tpu/test_spmd_model_weight_loading.py"
|
||||
run_and_track_test 6 "test_kv_cache_update_kernel.py" \
|
||||
"python3 -m pytest -s -v /workspace/vllm/tests/v1/tpu/test_kv_cache_update_kernel.py"
|
||||
run_and_track_test 7 "test_tpu_int8.py" \
|
||||
"python3 -m pytest -s -v /workspace/vllm/tests/v1/tpu/test_tpu_int8.py"
|
||||
|
||||
# After all tests have been attempted, exit with the overall status.
|
||||
if [ "$overall_script_exit_code" -ne 0 ]; then
|
||||
echo "--- One or more tests FAILED. Overall script exiting with failure code 1. ---"
|
||||
else
|
||||
echo "--- All tests have completed and PASSED. Overall script exiting with success code 0. ---"
|
||||
fi
|
||||
exit "$overall_script_exit_code"
|
||||
' # IMPORTANT: This is the closing single quote for the bash -c "..." command. Ensure it is present and correct.
|
||||
|
||||
# Capture the exit code of the docker run command
|
||||
DOCKER_RUN_EXIT_CODE=$?
|
||||
|
||||
# The trap will run for cleanup.
|
||||
# Exit the main script with the Docker run command's exit code.
|
||||
if [ "$DOCKER_RUN_EXIT_CODE" -ne 0 ]; then
|
||||
echo "Docker run command failed with exit code $DOCKER_RUN_EXIT_CODE."
|
||||
exit "$DOCKER_RUN_EXIT_CODE"
|
||||
else
|
||||
echo "Docker run command completed successfully."
|
||||
exit 0
|
||||
fi
|
||||
# TODO: This test fails because it uses RANDOM_SEED sampling
|
||||
# pytest -v -s /workspace/vllm/tests/tpu/test_custom_dispatcher.py \
|
||||
@@ -1,172 +0,0 @@
|
||||
#!/bin/bash
|
||||
|
||||
set -xu
|
||||
|
||||
|
||||
remove_docker_container() {
|
||||
docker rm -f tpu-test || true;
|
||||
}
|
||||
|
||||
trap remove_docker_container EXIT
|
||||
|
||||
# Remove the container that might not be cleaned up in the previous run.
|
||||
remove_docker_container
|
||||
|
||||
# Build the docker image.
|
||||
docker build -f docker/Dockerfile.tpu -t vllm-tpu .
|
||||
|
||||
# Set up cleanup.
|
||||
cleanup_docker() {
|
||||
# Get Docker's root directory
|
||||
docker_root=$(docker info -f '{{.DockerRootDir}}')
|
||||
if [ -z "$docker_root" ]; then
|
||||
echo "Failed to determine Docker root directory."
|
||||
exit 1
|
||||
fi
|
||||
echo "Docker root directory: $docker_root"
|
||||
# Check disk usage of the filesystem where Docker's root directory is located
|
||||
disk_usage=$(df "$docker_root" | tail -1 | awk '{print $5}' | sed 's/%//')
|
||||
# Define the threshold
|
||||
threshold=70
|
||||
if [ "$disk_usage" -gt "$threshold" ]; then
|
||||
echo "Disk usage is above $threshold%. Cleaning up Docker images and volumes..."
|
||||
# Remove dangling images (those that are not tagged and not used by any container)
|
||||
docker image prune -f
|
||||
# Remove unused volumes / force the system prune for old images as well.
|
||||
docker volume prune -f && docker system prune --force --filter "until=72h" --all
|
||||
echo "Docker images and volumes cleanup completed."
|
||||
else
|
||||
echo "Disk usage is below $threshold%. No cleanup needed."
|
||||
fi
|
||||
}
|
||||
cleanup_docker
|
||||
|
||||
# For HF_TOKEN.
|
||||
source /etc/environment
|
||||
|
||||
docker run --privileged --net host --shm-size=16G -it \
|
||||
-e "HF_TOKEN=$HF_TOKEN" --name tpu-test \
|
||||
vllm-tpu /bin/bash -c '
|
||||
set -e # Exit immediately if a command exits with a non-zero status.
|
||||
set -u # Treat unset variables as an error.
|
||||
|
||||
echo "--- Starting script inside Docker container ---"
|
||||
|
||||
# Create results directory
|
||||
RESULTS_DIR=$(mktemp -d)
|
||||
# If mktemp fails, set -e will cause the script to exit.
|
||||
echo "Results will be stored in: $RESULTS_DIR"
|
||||
|
||||
# Install dependencies
|
||||
echo "--- Installing Python dependencies ---"
|
||||
python3 -m pip install --progress-bar off git+https://github.com/thuml/depyf.git \
|
||||
&& python3 -m pip install --progress-bar off pytest pytest-asyncio tpu-info \
|
||||
&& python3 -m pip install --progress-bar off "lm-eval[api]>=0.4.12" \
|
||||
&& python3 -m pip install --progress-bar off hf-transfer tblib==3.1.0
|
||||
echo "--- Python dependencies installed ---"
|
||||
|
||||
export VLLM_XLA_CHECK_RECOMPILATION=1
|
||||
export VLLM_XLA_CACHE_PATH=
|
||||
|
||||
echo "--- Hardware Information ---"
|
||||
# tpu-info
|
||||
echo "--- Starting Tests ---"
|
||||
set +e
|
||||
overall_script_exit_code=0
|
||||
|
||||
# --- Test Definitions ---
|
||||
# If a test fails, this function will print logs and will not cause the main script to exit.
|
||||
run_test() {
|
||||
local test_num=$1
|
||||
local test_name=$2
|
||||
local test_command=$3
|
||||
local log_file="$RESULTS_DIR/test_${test_num}.log"
|
||||
local actual_exit_code
|
||||
|
||||
echo "--- TEST_$test_num: Running $test_name ---"
|
||||
|
||||
# Execute the test command.
|
||||
eval "$test_command" > >(tee -a "$log_file") 2> >(tee -a "$log_file" >&2)
|
||||
actual_exit_code=$?
|
||||
|
||||
echo "TEST_${test_num}_COMMAND_EXIT_CODE: $actual_exit_code" # This goes to main log
|
||||
echo "TEST_${test_num}_COMMAND_EXIT_CODE: $actual_exit_code" >> "$log_file" # Also to per-test log
|
||||
|
||||
if [ "$actual_exit_code" -ne 0 ]; then
|
||||
echo "TEST_$test_num ($test_name) FAILED with exit code $actual_exit_code." >&2
|
||||
echo "--- Log for failed TEST_$test_num ($test_name) ---" >&2
|
||||
if [ -f "$log_file" ]; then
|
||||
cat "$log_file" >&2
|
||||
else
|
||||
echo "Log file $log_file not found for TEST_$test_num ($test_name)." >&2
|
||||
fi
|
||||
echo "--- End of log for TEST_$test_num ($test_name) ---" >&2
|
||||
return "$actual_exit_code" # Return the failure code
|
||||
else
|
||||
echo "TEST_$test_num ($test_name) PASSED."
|
||||
return 0 # Return success
|
||||
fi
|
||||
}
|
||||
|
||||
# Helper function to call run_test and update the overall script exit code
|
||||
run_and_track_test() {
|
||||
local test_num_arg="$1"
|
||||
local test_name_arg="$2"
|
||||
local test_command_arg="$3"
|
||||
|
||||
# Run the test
|
||||
run_test "$test_num_arg" "$test_name_arg" "$test_command_arg"
|
||||
local test_specific_exit_code=$?
|
||||
|
||||
# If the test failed, set the overall script exit code to 1
|
||||
if [ "$test_specific_exit_code" -ne 0 ]; then
|
||||
# No need for extra echo here, run_test already logged the failure.
|
||||
overall_script_exit_code=1
|
||||
fi
|
||||
}
|
||||
|
||||
# --- Actual Test Execution ---
|
||||
run_and_track_test 0 "test_perf.py" \
|
||||
"python3 -m pytest -s -v /workspace/vllm/tests/v1/tpu/test_perf.py"
|
||||
run_and_track_test 1 "test_compilation.py" \
|
||||
"python3 -m pytest -s -v /workspace/vllm/tests/tpu/test_compilation.py"
|
||||
run_and_track_test 2 "test_basic.py" \
|
||||
"python3 -m pytest -s -v /workspace/vllm/tests/v1/tpu/test_basic.py"
|
||||
run_and_track_test 3 "test_accuracy.py::test_lm_eval_accuracy_v1_engine" \
|
||||
"python3 -m pytest -s -v /workspace/vllm/tests/entrypoints/llm/test_accuracy.py::test_lm_eval_accuracy_v1_engine"
|
||||
run_and_track_test 4 "test_quantization_accuracy.py" \
|
||||
"python3 -m pytest -s -v /workspace/vllm/tests/tpu/test_quantization_accuracy.py"
|
||||
run_and_track_test 6 "test_tpu_model_runner.py" \
|
||||
"python3 -m pytest -s -v /workspace/vllm/tests/v1/tpu/worker/test_tpu_model_runner.py"
|
||||
run_and_track_test 7 "test_sampler.py" \
|
||||
"python3 -m pytest -s -v /workspace/vllm/tests/v1/tpu/test_sampler.py"
|
||||
run_and_track_test 8 "test_topk_topp_sampler.py" \
|
||||
"python3 -m pytest -s -v /workspace/vllm/tests/v1/tpu/test_topk_topp_sampler.py"
|
||||
run_and_track_test 9 "test_multimodal.py" \
|
||||
"python3 -m pytest -s -v /workspace/vllm/tests/v1/tpu/test_multimodal.py"
|
||||
run_and_track_test 10 "test_pallas.py" \
|
||||
"python3 -m pytest -s -v /workspace/vllm/tests/v1/tpu/test_pallas.py"
|
||||
|
||||
# After all tests have been attempted, exit with the overall status.
|
||||
if [ "$overall_script_exit_code" -ne 0 ]; then
|
||||
echo "--- One or more tests FAILED. Overall script exiting with failure code 1. ---"
|
||||
else
|
||||
echo "--- All tests have completed and PASSED. Overall script exiting with success code 0. ---"
|
||||
fi
|
||||
exit "$overall_script_exit_code"
|
||||
' # IMPORTANT: This is the closing single quote for the bash -c "..." command. Ensure it is present and correct.
|
||||
|
||||
# Capture the exit code of the docker run command
|
||||
DOCKER_RUN_EXIT_CODE=$?
|
||||
|
||||
# The trap will run for cleanup.
|
||||
# Exit the main script with the Docker run command's exit code.
|
||||
if [ "$DOCKER_RUN_EXIT_CODE" -ne 0 ]; then
|
||||
echo "Docker run command failed with exit code $DOCKER_RUN_EXIT_CODE."
|
||||
exit "$DOCKER_RUN_EXIT_CODE"
|
||||
else
|
||||
echo "Docker run command completed successfully."
|
||||
exit 0
|
||||
fi
|
||||
# TODO: This test fails because it uses RANDOM_SEED sampling
|
||||
# pytest -v -s /workspace/vllm/tests/tpu/test_custom_dispatcher.py \
|
||||
@@ -10,7 +10,7 @@ fi
|
||||
|
||||
# TODO: Use the PyPI wheel once it is published.
|
||||
uv pip install --system \
|
||||
"kvcr @ git+https://github.com/ai-dynamo/kvcr.git@v0.1.0"
|
||||
"kvcr @ git+https://github.com/ai-dynamo/kvcr.git@v0.1.1"
|
||||
|
||||
# Keep only the NIXL wheel matching the CI image's CUDA runtime.
|
||||
NIXL_VERSION=$(uv pip show --system nixl | sed -n 's/^Version: //p')
|
||||
|
||||
@@ -23,7 +23,7 @@ export PATH="$CARGO_HOME/bin:$PATH"
|
||||
|
||||
CARGO_BINSTALL_VERSION="${CARGO_BINSTALL_VERSION:-1.20.1}"
|
||||
UV_VERSION="${UV_VERSION:-0.11.28}"
|
||||
PYO3_PYTHON_VERSION="${PYO3_PYTHON_VERSION:-3.12}"
|
||||
UV_SCRIPT_PYTHON_VERSION="${UV_SCRIPT_PYTHON_VERSION:-3.12}"
|
||||
|
||||
CARGO_SORT_VERSION_REQ="${CARGO_SORT_VERSION_REQ:-2}"
|
||||
CARGO_DENY_VERSION_REQ="${CARGO_DENY_VERSION_REQ:-0.20}"
|
||||
@@ -60,12 +60,12 @@ install_cargo_binstall() {
|
||||
|
||||
install_cargo_sort() {
|
||||
log_section "Installing cargo-sort ${CARGO_SORT_VERSION_REQ}"
|
||||
cargo binstall --no-confirm --force "cargo-sort@${CARGO_SORT_VERSION_REQ}"
|
||||
cargo binstall --no-confirm --force --locked "cargo-sort@${CARGO_SORT_VERSION_REQ}"
|
||||
}
|
||||
|
||||
install_cargo_deny() {
|
||||
log_section "Installing cargo-deny ${CARGO_DENY_VERSION_REQ}"
|
||||
cargo binstall --no-confirm --force "cargo-deny@${CARGO_DENY_VERSION_REQ}"
|
||||
cargo binstall --no-confirm --force --locked "cargo-deny@${CARGO_DENY_VERSION_REQ}"
|
||||
}
|
||||
|
||||
install_cargo_nextest() {
|
||||
@@ -74,6 +74,7 @@ install_cargo_nextest() {
|
||||
--no-confirm \
|
||||
--force \
|
||||
--secure \
|
||||
--locked \
|
||||
"cargo-nextest@${CARGO_NEXTEST_VERSION_REQ}"
|
||||
}
|
||||
|
||||
@@ -83,34 +84,7 @@ install_uv() {
|
||||
"https://github.com/astral-sh/uv/releases/download/${UV_VERSION}/uv-installer.sh" \
|
||||
| env UV_INSTALL_DIR="$CARGO_HOME/bin" sh
|
||||
uv --version
|
||||
}
|
||||
|
||||
setup_pyo3_python() {
|
||||
log_section "Installing Python ${PYO3_PYTHON_VERSION} for PyO3 tests"
|
||||
uv python install "$PYO3_PYTHON_VERSION"
|
||||
PYO3_PYTHON="$(uv python find \
|
||||
--managed-python \
|
||||
--no-project \
|
||||
--resolve-links \
|
||||
"$PYO3_PYTHON_VERSION")"
|
||||
export PYO3_PYTHON
|
||||
|
||||
local python_libdir
|
||||
python_libdir="$("$PYO3_PYTHON" - <<'PY'
|
||||
import pathlib
|
||||
import sysconfig
|
||||
|
||||
libdir = pathlib.Path(sysconfig.get_config_var("LIBDIR"))
|
||||
ldlibrary = sysconfig.get_config_var("LDLIBRARY")
|
||||
assert sysconfig.get_config_var("Py_ENABLE_SHARED") == 1
|
||||
assert ldlibrary
|
||||
assert (libdir / ldlibrary).exists(), libdir / ldlibrary
|
||||
print(libdir)
|
||||
PY
|
||||
)"
|
||||
|
||||
export LD_LIBRARY_PATH="${python_libdir}:${LD_LIBRARY_PATH:-}"
|
||||
export LIBRARY_PATH="${python_libdir}:${LIBRARY_PATH:-}"
|
||||
uv python install "$UV_SCRIPT_PYTHON_VERSION"
|
||||
}
|
||||
|
||||
run_style_clippy() {
|
||||
@@ -144,7 +118,6 @@ run_style_clippy() {
|
||||
|
||||
run_tests() {
|
||||
install_uv
|
||||
setup_pyo3_python
|
||||
install_cargo_binstall
|
||||
install_cargo_nextest
|
||||
|
||||
|
||||
@@ -1,24 +0,0 @@
|
||||
#!/bin/bash
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
docker_root=$(docker info -f '{{.DockerRootDir}}')
|
||||
if [ -z "$docker_root" ]; then
|
||||
echo "Failed to determine Docker root directory."
|
||||
exit 1
|
||||
fi
|
||||
echo "Docker root directory: $docker_root"
|
||||
# Check disk usage of the filesystem where Docker's root directory is located
|
||||
disk_usage=$(df "$docker_root" | tail -1 | awk '{print $5}' | sed 's/%//')
|
||||
# Define the threshold
|
||||
threshold=70
|
||||
if [ "$disk_usage" -gt "$threshold" ]; then
|
||||
echo "Disk usage is above $threshold%. Cleaning up Docker images and volumes..."
|
||||
# Remove dangling images (those that are not tagged and not used by any container)
|
||||
docker image prune -f
|
||||
# Remove unused volumes / force the system prune for old images as well.
|
||||
docker volume prune -f && docker system prune --force --filter "until=24h" --all
|
||||
echo "Docker images and volumes cleanup completed."
|
||||
else
|
||||
echo "Disk usage is below $threshold%. No cleanup needed."
|
||||
fi
|
||||
@@ -1,14 +0,0 @@
|
||||
# Environment config
|
||||
TEST_NAME=llama8b
|
||||
CONTAINER_NAME=tpu-test
|
||||
|
||||
# vllm config
|
||||
MODEL=meta-llama/Llama-3.1-8B-Instruct
|
||||
MAX_NUM_SEQS=256
|
||||
MAX_NUM_BATCHED_TOKENS=1024
|
||||
TENSOR_PARALLEL_SIZE=1
|
||||
MAX_MODEL_LEN=2048
|
||||
DOWNLOAD_DIR=/mnt/disks/persist
|
||||
EXPECTED_THROUGHPUT=8.0
|
||||
INPUT_LEN=1800
|
||||
OUTPUT_LEN=128
|
||||
@@ -1,91 +0,0 @@
|
||||
#!/bin/bash
|
||||
|
||||
if [ ! -f "$1" ]; then
|
||||
echo "Error: The env file '$1' does not exist."
|
||||
exit 1 # Exit the script with a non-zero status to indicate an error
|
||||
fi
|
||||
|
||||
ENV_FILE=$1
|
||||
|
||||
# For testing on local vm, use `set -a` to export all variables
|
||||
source /etc/environment
|
||||
# shellcheck source=/dev/null
|
||||
source "$ENV_FILE"
|
||||
|
||||
remove_docker_container() {
|
||||
docker rm -f "$CONTAINER_NAME" || true;
|
||||
}
|
||||
|
||||
trap remove_docker_container EXIT
|
||||
|
||||
# Remove the container that might not be cleaned up in the previous run.
|
||||
remove_docker_container
|
||||
|
||||
LOG_ROOT=$(mktemp -d)
|
||||
# If mktemp fails, set -e will cause the script to exit.
|
||||
echo "Results will be stored in: $LOG_ROOT"
|
||||
|
||||
if [ -z "$HF_TOKEN" ]; then
|
||||
echo "Error: HF_TOKEN is not set or is empty."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Make sure mounted disk or dir exists
|
||||
if [ ! -d "$DOWNLOAD_DIR" ]; then
|
||||
echo "Error: Folder $DOWNLOAD_DIR does not exist. This is useually a mounted drive. If no mounted drive, just create a folder."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "Run model $MODEL"
|
||||
echo
|
||||
|
||||
echo "starting docker...$CONTAINER_NAME"
|
||||
echo
|
||||
docker run \
|
||||
-v "$DOWNLOAD_DIR":"$DOWNLOAD_DIR" \
|
||||
--env-file "$ENV_FILE" \
|
||||
-e HF_TOKEN="$HF_TOKEN" \
|
||||
-e TARGET_COMMIT="$BUILDKITE_COMMIT" \
|
||||
-e MODEL="$MODEL" \
|
||||
-e WORKSPACE=/workspace \
|
||||
--name "$CONTAINER_NAME" \
|
||||
-d \
|
||||
--privileged \
|
||||
--network host \
|
||||
-v /dev/shm:/dev/shm \
|
||||
vllm/vllm-tpu-bm tail -f /dev/null
|
||||
|
||||
echo "run script..."
|
||||
echo
|
||||
docker exec "$CONTAINER_NAME" /bin/bash -c ".buildkite/scripts/tpu/run_bm.sh"
|
||||
|
||||
echo "copy result back..."
|
||||
VLLM_LOG="$LOG_ROOT/$TEST_NAME"_vllm_log.txt
|
||||
BM_LOG="$LOG_ROOT/$TEST_NAME"_bm_log.txt
|
||||
docker cp "$CONTAINER_NAME:/workspace/vllm_log.txt" "$VLLM_LOG"
|
||||
docker cp "$CONTAINER_NAME:/workspace/bm_log.txt" "$BM_LOG"
|
||||
|
||||
throughput=$(grep "Request throughput (req/s):" "$BM_LOG" | sed 's/[^0-9.]//g')
|
||||
echo "throughput for $TEST_NAME at $BUILDKITE_COMMIT: $throughput"
|
||||
|
||||
if [ "$BUILDKITE" = "true" ]; then
|
||||
echo "Running inside Buildkite"
|
||||
buildkite-agent artifact upload "$VLLM_LOG"
|
||||
buildkite-agent artifact upload "$BM_LOG"
|
||||
else
|
||||
echo "Not running inside Buildkite"
|
||||
fi
|
||||
|
||||
#
|
||||
# compare the throughput with EXPECTED_THROUGHPUT
|
||||
# and assert meeting the expectation
|
||||
#
|
||||
if [[ -z "$throughput" || ! "$throughput" =~ ^[0-9]+([.][0-9]+)?$ ]]; then
|
||||
echo "Failed to get the throughput"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if (( $(echo "$throughput < $EXPECTED_THROUGHPUT" | bc -l) )); then
|
||||
echo "Error: throughput($throughput) is less than expected($EXPECTED_THROUGHPUT)"
|
||||
exit 1
|
||||
fi
|
||||
@@ -1,14 +0,0 @@
|
||||
# Environment config
|
||||
TEST_NAME=llama8bw8a8
|
||||
CONTAINER_NAME=tpu-test
|
||||
|
||||
# vllm config
|
||||
MODEL=RedHatAI/Meta-Llama-3.1-8B-Instruct-quantized.w8a8
|
||||
MAX_NUM_SEQS=128
|
||||
MAX_NUM_BATCHED_TOKENS=1024
|
||||
TENSOR_PARALLEL_SIZE=1
|
||||
MAX_MODEL_LEN=2048
|
||||
DOWNLOAD_DIR=/mnt/disks/persist
|
||||
EXPECTED_THROUGHPUT=8.7
|
||||
INPUT_LEN=1800
|
||||
OUTPUT_LEN=128
|
||||
@@ -1,93 +0,0 @@
|
||||
#!/bin/bash
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
VLLM_LOG="$WORKSPACE/vllm_log.txt"
|
||||
BM_LOG="$WORKSPACE/bm_log.txt"
|
||||
|
||||
if [ -n "$TARGET_COMMIT" ]; then
|
||||
head_hash=$(git rev-parse HEAD)
|
||||
if [ "$TARGET_COMMIT" != "$head_hash" ]; then
|
||||
echo "Error: target commit $TARGET_COMMIT does not match HEAD: $head_hash"
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
echo "model: $MODEL"
|
||||
echo
|
||||
|
||||
#
|
||||
# create a log folder
|
||||
#
|
||||
mkdir "$WORKSPACE/log"
|
||||
|
||||
# TODO: Move to image building.
|
||||
pip install pandas
|
||||
pip install datasets
|
||||
|
||||
#
|
||||
# create sonnet_4x
|
||||
#
|
||||
echo "Create sonnet_4x.txt"
|
||||
echo "" > benchmarks/sonnet_4x.txt
|
||||
for _ in {1..4}
|
||||
do
|
||||
cat benchmarks/sonnet.txt >> benchmarks/sonnet_4x.txt
|
||||
done
|
||||
|
||||
#
|
||||
# start vllm service in backend
|
||||
#
|
||||
echo "lanching vllm..."
|
||||
echo "logging to $VLLM_LOG"
|
||||
echo
|
||||
|
||||
vllm serve "$MODEL" \
|
||||
--seed 42 \
|
||||
--max-num-seqs "$MAX_NUM_SEQS" \
|
||||
--max-num-batched-tokens "$MAX_NUM_BATCHED_TOKENS" \
|
||||
--tensor-parallel-size "$TENSOR_PARALLEL_SIZE" \
|
||||
--no-enable-prefix-caching \
|
||||
--download_dir "$DOWNLOAD_DIR" \
|
||||
--max-model-len "$MAX_MODEL_LEN" > "$VLLM_LOG" 2>&1 &
|
||||
|
||||
|
||||
echo "wait for 20 minutes.."
|
||||
echo
|
||||
# sleep 1200
|
||||
# wait for 10 minutes...
|
||||
for _ in {1..120}; do
|
||||
# TODO: detect other type of errors.
|
||||
if grep -Fq "raise RuntimeError" "$VLLM_LOG"; then
|
||||
echo "Detected RuntimeError, exiting."
|
||||
exit 1
|
||||
elif grep -Fq "Application startup complete" "$VLLM_LOG"; then
|
||||
echo "Application started"
|
||||
break
|
||||
else
|
||||
echo "wait for 10 seconds..."
|
||||
sleep 10
|
||||
fi
|
||||
done
|
||||
|
||||
#
|
||||
# run test
|
||||
#
|
||||
echo "run benchmark test..."
|
||||
echo "logging to $BM_LOG"
|
||||
echo
|
||||
vllm bench serve \
|
||||
--backend vllm \
|
||||
--model "$MODEL" \
|
||||
--dataset-name sonnet \
|
||||
--dataset-path benchmarks/sonnet_4x.txt \
|
||||
--sonnet-input-len "$INPUT_LEN" \
|
||||
--sonnet-output-len "$OUTPUT_LEN" \
|
||||
--ignore-eos > "$BM_LOG"
|
||||
|
||||
echo "completed..."
|
||||
echo
|
||||
|
||||
throughput=$(grep "Request throughput (req/s):" "$BM_LOG" | sed 's/[^0-9.]//g')
|
||||
echo "throughput: $throughput"
|
||||
echo
|
||||
+6
-152
@@ -111,38 +111,6 @@ steps:
|
||||
# #
|
||||
#########################################################################################################################################
|
||||
|
||||
- label: ":computer: (CPU) Rust Frontend Cargo Style + Clippy" # TBD
|
||||
timeout_in_minutes: 35
|
||||
mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]
|
||||
dind: false
|
||||
agent_pool: mi250_1
|
||||
no_gpu: true
|
||||
optional: true
|
||||
working_dir: "/vllm-workspace"
|
||||
source_file_dependencies:
|
||||
- rust/
|
||||
- rust-toolchain.toml
|
||||
- .buildkite/test_areas/rust_frontend_cargo.yaml
|
||||
- .buildkite/scripts/run-rust-frontend-cargo-ci.sh
|
||||
commands:
|
||||
- .buildkite/scripts/run-rust-frontend-cargo-ci.sh style-clippy
|
||||
|
||||
- label: ":computer: (CPU) Rust Frontend Cargo" # TBD
|
||||
timeout_in_minutes: 35
|
||||
mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]
|
||||
dind: false
|
||||
agent_pool: mi250_1
|
||||
no_gpu: true
|
||||
optional: true
|
||||
working_dir: "/vllm-workspace"
|
||||
source_file_dependencies:
|
||||
- rust/
|
||||
- rust-toolchain.toml
|
||||
- .buildkite/test_areas/rust_frontend_cargo.yaml
|
||||
- .buildkite/scripts/run-rust-frontend-cargo-ci.sh
|
||||
commands:
|
||||
- .buildkite/scripts/run-rust-frontend-cargo-ci.sh test
|
||||
|
||||
- label: ":computer: (CPU) Docker Build Metadata" # TBD
|
||||
timeout_in_minutes: 35
|
||||
mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]
|
||||
@@ -269,8 +237,6 @@ steps:
|
||||
- tests/models/language/generation_ppl_test
|
||||
- vllm/model_executor/models/qwen3_5.py
|
||||
- vllm/model_executor/models/qwen3_5_mtp.py
|
||||
- vllm/transformers_utils/configs/qwen3_5.py
|
||||
- vllm/transformers_utils/configs/qwen3_5_moe.py
|
||||
- vllm/model_executor/models/qwen2.py
|
||||
- vllm/model_executor/models/qwen3.py
|
||||
- vllm/model_executor/models/qwen3_next.py
|
||||
@@ -725,28 +691,8 @@ steps:
|
||||
- vllm/platforms/rocm.py
|
||||
commands:
|
||||
- export VLLM_TEST_CLEAN_GPU_MEMORY=1
|
||||
- VLLM_TEST_CLEAN_GPU_MEMORY=1 pytest -v -s tests/compile/passes/distributed/test_async_tp.py
|
||||
- pytest -v -s tests/compile/fusions_e2e/test_tp2_ar_rms.py::test_tp2_ar_rms_fusions -k "dynamo_partition"
|
||||
|
||||
# - label: ":amd: (MI300) Sequence Parallel Correctness" # TBD
|
||||
# timeout_in_minutes: 180
|
||||
# mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
|
||||
# dind: false
|
||||
# agent_pool: mi300_2
|
||||
# num_gpus: 2
|
||||
# optional: true
|
||||
# working_dir: "/vllm-workspace/"
|
||||
# source_file_dependencies:
|
||||
# - vllm/model_executor/layers/
|
||||
# - vllm/compilation/
|
||||
# - vllm/v1/worker/
|
||||
# - vllm/v1/cudagraph_dispatcher.py
|
||||
# - tests/compile/correctness_e2e/test_sequence_parallel.py
|
||||
# - vllm/platforms/rocm.py
|
||||
# commands:
|
||||
# - export VLLM_TEST_CLEAN_GPU_MEMORY=1
|
||||
# - pytest -v -s tests/compile/correctness_e2e/test_sequence_parallel.py
|
||||
|
||||
- label: ":amd: (MI300) Distributed Compile + RPC" # TBD
|
||||
timeout_in_minutes: 65
|
||||
mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
|
||||
@@ -1076,12 +1022,12 @@ steps:
|
||||
source_file_dependencies:
|
||||
- vllm/entrypoints
|
||||
- tests/entrypoints/unit_tests
|
||||
- tests/entrypoints/weight_transfer
|
||||
- tests/entrypoints/rl/weight_transfer
|
||||
- tests/entrypoints/launchers
|
||||
- vllm/platforms/rocm.py
|
||||
commands:
|
||||
- pytest -v -s entrypoints/unit_tests
|
||||
- pytest -v -s entrypoints/weight_transfer
|
||||
- pytest -v -s entrypoints/rl/weight_transfer
|
||||
- pytest -v -s entrypoints/launchers
|
||||
|
||||
- label: ":amd: (MI300) Entrypoints Integration (Responses API)" # TBD
|
||||
@@ -2657,7 +2603,6 @@ steps:
|
||||
- vllm/model_executor/models/qwen3_next.py
|
||||
- vllm/model_executor/models/registry.py
|
||||
- vllm/transformers_utils/configs/extract_hidden_states.py
|
||||
- vllm/transformers_utils/configs/qwen3_5.py
|
||||
- vllm/v1/attention/backends/
|
||||
- vllm/v1/attention/selector.py
|
||||
- vllm/v1/kv_cache_interface.py
|
||||
@@ -3212,7 +3157,6 @@ steps:
|
||||
# TODO: FP4 output-quant and trained MXFP4 AR/RMS fusion remain unsupported.
|
||||
- amd-smi
|
||||
- VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_CUSTOM_AR=1 pytest -v -s tests/compile/fusions_e2e/test_tp2_ar_rms.py -k 'test_tp2_ar_rms_fp8_fusions and qwen3 and ROCM_AITER_UNIFIED_ATTN and +quant_fp8 and (dynamo_partition or not +rms_norm)'
|
||||
- pytest -v -s tests/compile/fusions_e2e/test_tp2_rocm_moe.py
|
||||
|
||||
- label: ":amd: (MI355) Fusion and Compile FP8"
|
||||
timeout_in_minutes: 45
|
||||
@@ -3237,64 +3181,6 @@ steps:
|
||||
- pytest -v -s tests/compile/passes/test_fusion_attn.py tests/compile/passes/test_silu_mul_quant_fusion.py tests/compile/passes/test_silu_mul_quant_manual_fusion.py
|
||||
- pytest -v -s tests/compile/correctness_e2e/test_attn_quant.py
|
||||
|
||||
- label: ":amd: (MI355) Sequence Parallel Correctness"
|
||||
timeout_in_minutes: 180
|
||||
mirror_hardwares: [amdexperimental, amdproduction, amdgfx950nightly, amdmi355]
|
||||
dind: false
|
||||
agent_pool: mi355_2
|
||||
num_gpus: 2
|
||||
optional: true
|
||||
working_dir: "/vllm-workspace/"
|
||||
source_file_dependencies:
|
||||
- .buildkite/test_areas/compile.yaml
|
||||
- vllm/compilation/
|
||||
- vllm/config/
|
||||
- vllm/distributed/
|
||||
- vllm/model_executor/kernels/linear/
|
||||
- vllm/model_executor/layers/
|
||||
- vllm/model_executor/models/llama.py
|
||||
- vllm/v1/attention/
|
||||
- vllm/v1/worker/
|
||||
- vllm/v1/cudagraph_dispatcher.py
|
||||
- vllm/platforms/rocm.py
|
||||
- tests/compile/correctness_e2e/test_sequence_parallel.py
|
||||
- tests/models/registry.py
|
||||
- tests/utils.py
|
||||
commands:
|
||||
- export VLLM_TEST_CLEAN_GPU_MEMORY=1
|
||||
- pytest -v -s tests/compile/correctness_e2e/test_sequence_parallel.py
|
||||
|
||||
- label: ":amd: (MI355) AsyncTP Correctness"
|
||||
timeout_in_minutes: 60
|
||||
mirror_hardwares: [amdexperimental, amdproduction, amdgfx950nightly, amdmi355]
|
||||
dind: false
|
||||
agent_pool: mi355_2
|
||||
num_gpus: 2
|
||||
optional: true
|
||||
working_dir: "/vllm-workspace/"
|
||||
source_file_dependencies:
|
||||
- .buildkite/test_areas/compile.yaml
|
||||
- vllm/compilation/
|
||||
- vllm/config/
|
||||
- vllm/distributed/
|
||||
- vllm/model_executor/layers/
|
||||
- vllm/model_executor/kernels/linear/
|
||||
- vllm/model_executor/models/llama.py
|
||||
- vllm/model_executor/models/qwen3.py
|
||||
- vllm/v1/attention/
|
||||
- vllm/_aiter_ops.py
|
||||
- vllm/platforms/rocm.py
|
||||
- vllm/transformers_utils/repo_utils.py
|
||||
- tests/compile/correctness_e2e/test_async_tp.py
|
||||
- tests/compile/fusions_e2e/common.py
|
||||
- tests/models/registry.py
|
||||
- tests/utils.py
|
||||
commands:
|
||||
- export VLLM_TEST_CLEAN_GPU_MEMORY=1
|
||||
- pytest -v -s tests/compile/correctness_e2e/test_async_tp.py::test_rocm_async_tp_bf16_output_correctness
|
||||
- pytest -v -s tests/compile/correctness_e2e/test_async_tp.py::test_async_tp_pass_correctness -k RedHatAI
|
||||
- pytest -v -s tests/compile/correctness_e2e/test_async_tp.py::test_async_tp_pass_nvfp4_correctness
|
||||
|
||||
- label: ":amd: (MI355 DPX) Fusion E2E FP8 Config Sweep"
|
||||
timeout_in_minutes: 45
|
||||
mirror_hardwares: [amdexperimental, amdproduction, amdgfx950nightly, amdmi355]
|
||||
@@ -3778,11 +3664,13 @@ steps:
|
||||
- vllm/
|
||||
- tests/entrypoints/serve
|
||||
- tests/entrypoints/scale_out
|
||||
- tests/entrypoints/rl
|
||||
commands:
|
||||
- export VLLM_WORKER_MULTIPROC_METHOD=spawn
|
||||
- pytest -v -s entrypoints/serve --ignore=entrypoints/serve/dev/rpc --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT --shard-id=$$BUILDKITE_PARALLEL_JOB
|
||||
- if [ "$$BUILDKITE_PARALLEL_JOB" = "1" ]; then PYTHONPATH=/vllm-workspace pytest -v -s entrypoints/serve/dev/rpc; fi
|
||||
- pytest -v -s entrypoints/scale_out --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT --shard-id=$$BUILDKITE_PARALLEL_JOB
|
||||
- pytest -v -s entrypoints/rl --ignore=entrypoints/rl/weight_transfer --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT --shard-id=$$BUILDKITE_PARALLEL_JOB
|
||||
|
||||
- label: ":amd: (MI355 DPX) Entrypoints Integration (OpenAI API completion)" # TBD
|
||||
timeout_in_minutes: 65
|
||||
@@ -4023,8 +3911,6 @@ steps:
|
||||
source_file_dependencies:
|
||||
- vllm/model_executor/models/qwen3_5.py
|
||||
- vllm/model_executor/models/qwen3_5_mtp.py
|
||||
- vllm/transformers_utils/configs/qwen3_5.py
|
||||
- vllm/transformers_utils/configs/qwen3_5_moe.py
|
||||
- vllm/model_executor/models/qwen2.py
|
||||
- vllm/model_executor/models/qwen3.py
|
||||
- vllm/model_executor/models/qwen3_next.py
|
||||
@@ -4066,8 +3952,6 @@ steps:
|
||||
- vllm/model_executor/models/qwen3_next_mtp.py
|
||||
- vllm/model_executor/models/nemotron_h.py
|
||||
- vllm/model_executor/models/nemotron_h_mtp.py
|
||||
- vllm/transformers_utils/configs/qwen3_next.py
|
||||
- vllm/transformers_utils/configs/nemotron_h.py
|
||||
- vllm/model_executor/model_loader/
|
||||
- vllm/model_executor/layers/quantization/
|
||||
- vllm/model_executor/layers/fused_moe/
|
||||
@@ -4265,7 +4149,6 @@ steps:
|
||||
- tests/evals/gsm8k/
|
||||
- tests/utils.py
|
||||
- vllm/distributed/
|
||||
- vllm/transformers_utils/configs/diffusion_gemma.py
|
||||
- vllm/v1/attention/ops/
|
||||
- tests/evals/gsm8k/configs/models-small-tp-mi355.txt
|
||||
commands:
|
||||
@@ -5157,8 +5040,6 @@ steps:
|
||||
source_file_dependencies:
|
||||
- vllm/model_executor/models/qwen3_5.py
|
||||
- vllm/model_executor/models/qwen3_5_mtp.py
|
||||
- vllm/transformers_utils/configs/qwen3_5.py
|
||||
- vllm/transformers_utils/configs/qwen3_5_moe.py
|
||||
- vllm/model_executor/models/qwen2.py
|
||||
- vllm/model_executor/models/qwen3.py
|
||||
- vllm/model_executor/models/qwen3_next.py
|
||||
@@ -5888,6 +5769,7 @@ steps:
|
||||
timeout_in_minutes: 20
|
||||
source_file_dependencies:
|
||||
- tests/models/test_deepseek_v4_vl_rocm.py
|
||||
- tests/models/test_deepseek_v4_aiter_mega_moe.py
|
||||
- vllm/models/deepseek_v4/amd/
|
||||
- vllm/models/deepseek_v4/common/
|
||||
- vllm/model_executor/layers/fused_moe/
|
||||
@@ -5901,6 +5783,7 @@ steps:
|
||||
- vllm/envs.py
|
||||
commands:
|
||||
- VLLM_ROCM_USE_AITER_MOE=0 pytest -v -s models/test_deepseek_v4_vl_rocm.py
|
||||
- pytest -v -s models/test_deepseek_v4_aiter_mega_moe.py
|
||||
dind: false
|
||||
working_dir: /vllm-workspace/tests
|
||||
agent_pool: mi355_1
|
||||
@@ -5944,35 +5827,6 @@ steps:
|
||||
- amdmi355
|
||||
optional: true
|
||||
|
||||
- label: ':amd: (MI355) Fusion E2E TP2 AsyncTP BF16 Config Sweep'
|
||||
timeout_in_minutes: 30
|
||||
working_dir: /vllm-workspace/
|
||||
source_file_dependencies:
|
||||
- vllm/compilation/
|
||||
- vllm/model_executor/layers/linear.py
|
||||
- vllm/model_executor/layers/utils.py
|
||||
- vllm/distributed/
|
||||
- vllm/v1/attention/backends/triton_attn.py
|
||||
- tests/compile/fusions_e2e/
|
||||
- tests/compile/passes/distributed/test_async_tp.py
|
||||
- tests/compile/rocm/test_async_tp.py
|
||||
- tests/compile/correctness_e2e/test_async_tp.py
|
||||
- vllm/model_executor/kernels/linear/
|
||||
- vllm/model_executor/models/qwen3.py
|
||||
commands:
|
||||
- rocm-smi
|
||||
- pytest -v -s tests/compile/rocm/test_async_tp.py
|
||||
- pytest -v -s tests/compile/fusions_e2e/test_tp2_rocm_async_tp.py
|
||||
- pytest -v -s tests/compile/correctness_e2e/test_async_tp.py::test_rocm_async_tp_bf16_output_correctness
|
||||
dind: false
|
||||
agent_pool: mi355_2
|
||||
num_gpus: 2
|
||||
mirror_hardwares:
|
||||
- amdexperimental
|
||||
- amdproduction
|
||||
- amdgfx950nightly
|
||||
- amdmi355
|
||||
optional: true
|
||||
|
||||
- label: ":amd: (MI355 DPX) MRV2 Sampler JIT Warmup"
|
||||
timeout_in_minutes: 10
|
||||
|
||||
@@ -4,7 +4,7 @@ depends_on:
|
||||
steps:
|
||||
- label: ":nvidia: (H200 MIG 18GB) Benchmarks CLI"
|
||||
key: benchmarks-cli-test
|
||||
timeout_in_minutes: 45
|
||||
timeout_in_minutes: 55
|
||||
device: h200_18gb
|
||||
source_file_dependencies:
|
||||
- vllm/
|
||||
|
||||
@@ -2,84 +2,6 @@ group: Compile
|
||||
depends_on:
|
||||
- image-build
|
||||
steps:
|
||||
- label: ":nvidia: (B200) Sequence Parallel Correctness"
|
||||
key: sequence-parallel-correctness-tests-2xb200
|
||||
timeout_in_minutes: 45
|
||||
working_dir: "/vllm-workspace/"
|
||||
device: b200-k8s
|
||||
optional: true
|
||||
num_devices: 2
|
||||
commands:
|
||||
- export VLLM_TEST_CLEAN_GPU_MEMORY=1
|
||||
- pytest -v -s tests/compile/correctness_e2e/test_sequence_parallel.py
|
||||
mirror:
|
||||
amd:
|
||||
label: ":amd: (MI355) Sequence Parallel Correctness"
|
||||
dind: false
|
||||
device: mi355_2
|
||||
num_devices: 2
|
||||
timeout_in_minutes: 180
|
||||
depends_on:
|
||||
- image-build-amd
|
||||
source_file_dependencies:
|
||||
- .buildkite/test_areas/compile.yaml
|
||||
- vllm/compilation/
|
||||
- vllm/config/
|
||||
- vllm/distributed/
|
||||
- vllm/model_executor/kernels/linear/
|
||||
- vllm/model_executor/layers/
|
||||
- vllm/model_executor/models/llama.py
|
||||
- vllm/v1/attention/
|
||||
- vllm/v1/worker/
|
||||
- vllm/v1/cudagraph_dispatcher.py
|
||||
- vllm/platforms/rocm.py
|
||||
- tests/compile/correctness_e2e/test_sequence_parallel.py
|
||||
- tests/models/registry.py
|
||||
- tests/utils.py
|
||||
|
||||
- label: ":nvidia: (B200) AsyncTP Correctness"
|
||||
key: asynctp-correctness-tests-b200
|
||||
timeout_in_minutes: 30
|
||||
working_dir: "/vllm-workspace/"
|
||||
device: b200-k8s
|
||||
optional: true
|
||||
num_devices: 2
|
||||
commands:
|
||||
- export VLLM_TEST_CLEAN_GPU_MEMORY=1
|
||||
- pytest -v -s tests/compile/correctness_e2e/test_async_tp.py
|
||||
mirror:
|
||||
amd:
|
||||
label: ":amd: (MI355) AsyncTP Correctness"
|
||||
dind: false
|
||||
device: mi355_2
|
||||
num_devices: 2
|
||||
timeout_in_minutes: 60
|
||||
working_dir: "/vllm-workspace/"
|
||||
depends_on:
|
||||
- image-build-amd
|
||||
source_file_dependencies:
|
||||
- .buildkite/test_areas/compile.yaml
|
||||
- vllm/compilation/
|
||||
- vllm/config/
|
||||
- vllm/distributed/
|
||||
- vllm/model_executor/layers/
|
||||
- vllm/model_executor/kernels/linear/
|
||||
- vllm/model_executor/models/llama.py
|
||||
- vllm/model_executor/models/qwen3.py
|
||||
- vllm/v1/attention/
|
||||
- vllm/_aiter_ops.py
|
||||
- vllm/platforms/rocm.py
|
||||
- vllm/transformers_utils/repo_utils.py
|
||||
- tests/compile/correctness_e2e/test_async_tp.py
|
||||
- tests/compile/fusions_e2e/common.py
|
||||
- tests/models/registry.py
|
||||
- tests/utils.py
|
||||
commands:
|
||||
- export VLLM_TEST_CLEAN_GPU_MEMORY=1
|
||||
- pytest -v -s tests/compile/correctness_e2e/test_async_tp.py::test_rocm_async_tp_bf16_output_correctness
|
||||
- pytest -v -s tests/compile/correctness_e2e/test_async_tp.py::test_async_tp_pass_correctness -k RedHatAI
|
||||
- pytest -v -s tests/compile/correctness_e2e/test_async_tp.py::test_async_tp_pass_nvfp4_correctness
|
||||
|
||||
- label: ":nvidia: (H100) Distributed Compile"
|
||||
key: distributed-compile-unit-tests-2xh100
|
||||
timeout_in_minutes: 45
|
||||
@@ -300,7 +222,6 @@ steps:
|
||||
- nvidia-smi
|
||||
# Run all models and attn backends but only Inductor partition and native custom ops
|
||||
- pytest -v -s tests/compile/fusions_e2e/test_tp2_ar_rms.py -k "inductor_partition and not +rms_norm and (not +quant_fp8 or +quant_fp8 and (qwen3 or deepseek))"
|
||||
- pytest -v -s tests/compile/fusions_e2e/test_tp2_async_tp.py -k "inductor_partition and not +rms_norm and (not +quant_fp8 or +quant_fp8 and (qwen3 or deepseek))"
|
||||
mirror:
|
||||
amd:
|
||||
label: ":amd: (MI355) Fusion E2E TP2 Quick"
|
||||
@@ -388,53 +309,6 @@ steps:
|
||||
- amd-smi
|
||||
- VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_CUSTOM_AR=1 pytest -v -s tests/compile/fusions_e2e/test_tp2_ar_rms.py -k 'test_tp2_ar_rms_fp8_fusions and Llama-3 and ROCM_AITER_UNIFIED_ATTN and (dynamo_partition or +quant_fp8 or +rms_norm)'
|
||||
|
||||
- label: ":nvidia: (H100) Fusion E2E TP2 AsyncTP Config Sweep"
|
||||
key: fusion-e2e-tp2-asynctp-config-sweep-h100
|
||||
timeout_in_minutes: 40
|
||||
working_dir: "/vllm-workspace/"
|
||||
device: h100
|
||||
num_devices: 2
|
||||
source_file_dependencies:
|
||||
- csrc/quantization/
|
||||
- vllm/compilation/
|
||||
# can affect pattern matching
|
||||
- vllm/model_executor/layers/layernorm.py
|
||||
- vllm/model_executor/layers/activation.py
|
||||
- vllm/model_executor/layers/attention/attention.py
|
||||
- vllm/model_executor/layers/quantization/input_quant_fp8.py
|
||||
- tests/compile/fusions_e2e/
|
||||
commands:
|
||||
- nvidia-smi
|
||||
# Run just llama3 (fp8 & bf16) for all config combinations
|
||||
- pytest -v -s tests/compile/fusions_e2e/test_tp2_async_tp.py -k "llama-3"
|
||||
mirror:
|
||||
amd:
|
||||
label: ':amd: (MI355) Fusion E2E TP2 AsyncTP BF16 Config Sweep'
|
||||
dind: false
|
||||
device: mi355_2
|
||||
num_devices: 2
|
||||
working_dir: /vllm-workspace/
|
||||
timeout_in_minutes: 30
|
||||
depends_on:
|
||||
- image-build-amd
|
||||
source_file_dependencies:
|
||||
- vllm/compilation/
|
||||
- vllm/model_executor/layers/linear.py
|
||||
- vllm/model_executor/layers/utils.py
|
||||
- vllm/distributed/
|
||||
- vllm/v1/attention/backends/triton_attn.py
|
||||
- tests/compile/fusions_e2e/
|
||||
- tests/compile/passes/distributed/test_async_tp.py
|
||||
- tests/compile/rocm/test_async_tp.py
|
||||
- tests/compile/correctness_e2e/test_async_tp.py
|
||||
- vllm/model_executor/kernels/linear/
|
||||
- vllm/model_executor/models/qwen3.py
|
||||
commands:
|
||||
- rocm-smi
|
||||
- pytest -v -s tests/compile/rocm/test_async_tp.py
|
||||
- pytest -v -s tests/compile/fusions_e2e/test_tp2_rocm_async_tp.py
|
||||
- pytest -v -s tests/compile/correctness_e2e/test_async_tp.py::test_rocm_async_tp_bf16_output_correctness
|
||||
|
||||
- label: ":nvidia: (B200) Fusion E2E TP2"
|
||||
key: fusion-e2e-tp2-b200
|
||||
timeout_in_minutes: 45
|
||||
@@ -453,7 +327,6 @@ steps:
|
||||
# include qwen/deepseek with +quant_fp8 as -quant_fp8 rms+quant fusion is not supported
|
||||
# for ar-rms-quant-fp4, also sweep llama3
|
||||
- pytest -v -s tests/compile/fusions_e2e/test_tp2_ar_rms.py -k "(FLASHINFER and inductor_partition and not +rms_norm and (not +quant_fp8 or +quant_fp8 and (qwen3 or deepseek))) or Llama-3.1-8B-Instruct-FP4"
|
||||
- pytest -v -s tests/compile/fusions_e2e/test_tp2_async_tp.py -k "FLASHINFER and inductor_partition and not +rms_norm and (not +quant_fp8 or +quant_fp8 and (qwen3 or deepseek))"
|
||||
mirror:
|
||||
amd:
|
||||
label: ":amd: (MI355) Fusion E2E TP2 FP8 BF16 rewrites"
|
||||
@@ -480,7 +353,6 @@ steps:
|
||||
# TODO: FP4 output-quant and trained MXFP4 AR/RMS fusion remain unsupported.
|
||||
- amd-smi
|
||||
- VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_CUSTOM_AR=1 pytest -v -s tests/compile/fusions_e2e/test_tp2_ar_rms.py -k 'test_tp2_ar_rms_fp8_fusions and qwen3 and ROCM_AITER_UNIFIED_ATTN and +quant_fp8 and (dynamo_partition or not +rms_norm)'
|
||||
- pytest -v -s tests/compile/fusions_e2e/test_tp2_rocm_moe.py
|
||||
|
||||
- label: ':amd: (MI355) Fusion E2E TP2 AR-RMS (Inductor Partition)'
|
||||
key: fusion-e2e-tp2-ar-rms-inductor-partition-amd
|
||||
|
||||
@@ -33,14 +33,20 @@ steps:
|
||||
- label: ":nvidia: (H200 MIG 18GB) CUDAGraph"
|
||||
device: h200_18gb
|
||||
key: cudagraph
|
||||
timeout_in_minutes: 30
|
||||
automatic_shard: true
|
||||
timeout_in_minutes: 45
|
||||
source_file_dependencies:
|
||||
- tests/v1/cudagraph
|
||||
- vllm/v1/cudagraph_dispatcher.py
|
||||
- vllm/config/compilation.py
|
||||
- vllm/config/vllm.py
|
||||
- vllm/compilation
|
||||
- vllm/v1/worker/gpu/cudagraph_utils.py
|
||||
- vllm/v1/worker/gpu/dp_utils.py
|
||||
- vllm/v1/worker/encoder_cudagraph.py
|
||||
- vllm/v1/worker/encoder_cudagraph_defs.py
|
||||
- vllm/v1/worker/gpu/model_runner.py
|
||||
- vllm/v1/worker/gpu/ubatch_utils.py
|
||||
commands:
|
||||
- pytest -v -s v1/cudagraph/test_cudagraph_dispatch.py
|
||||
- pytest -v -s v1/cudagraph/test_cudagraph_mode.py
|
||||
|
||||
@@ -397,6 +397,7 @@ steps:
|
||||
- label: ":nvidia: (H100) Distributed DP + EP"
|
||||
key: distributed-tests-8xh100
|
||||
timeout_in_minutes: 20
|
||||
optional: true
|
||||
device: h100
|
||||
num_devices: 8
|
||||
working_dir: "/vllm-workspace/tests"
|
||||
@@ -419,6 +420,7 @@ steps:
|
||||
amd:
|
||||
label: ":amd: (MI355) Distributed DP + EP"
|
||||
dind: false
|
||||
optional: false
|
||||
device: mi355_8
|
||||
timeout_in_minutes: 50
|
||||
depends_on:
|
||||
|
||||
@@ -38,6 +38,7 @@ steps:
|
||||
|
||||
- label: ":nvidia: (L4) V1 Engine"
|
||||
key: engine-1-gpu
|
||||
automatic_shard: true
|
||||
device: l4
|
||||
num_devices: 1
|
||||
timeout_in_minutes: 45
|
||||
@@ -108,6 +109,7 @@ steps:
|
||||
- label: ":nvidia: (H200 MIG 18GB) E2E Core"
|
||||
device: h200_18gb
|
||||
key: e2e-core-1-gpu
|
||||
automatic_shard: true
|
||||
timeout_in_minutes: 65
|
||||
source_file_dependencies:
|
||||
- vllm/v1/
|
||||
|
||||
@@ -26,16 +26,15 @@ steps:
|
||||
- label: ":nvidia: (H200 MIG 35GB) Entrypoints Unit"
|
||||
device: h200_35gb
|
||||
key: entrypoints-unit-tests
|
||||
automatic_shard: true
|
||||
timeout_in_minutes: 40
|
||||
working_dir: "/vllm-workspace/tests"
|
||||
source_file_dependencies:
|
||||
- vllm/entrypoints
|
||||
- tests/entrypoints/unit_tests
|
||||
- tests/entrypoints/weight_transfer
|
||||
- tests/entrypoints/launchers
|
||||
commands:
|
||||
- pytest -v -s entrypoints/unit_tests
|
||||
- pytest -v -s entrypoints/weight_transfer
|
||||
- pytest -v -s entrypoints/launchers
|
||||
mirror:
|
||||
amd:
|
||||
@@ -170,6 +169,7 @@ steps:
|
||||
- label: ":nvidia: (H200 MIG 35GB) Entrypoints Integration (Responses API)"
|
||||
device: h200_35gb
|
||||
key: entrypoints-integration-responses-api
|
||||
automatic_shard: true
|
||||
timeout_in_minutes: 50
|
||||
working_dir: "/vllm-workspace/tests"
|
||||
source_file_dependencies:
|
||||
@@ -190,6 +190,7 @@ steps:
|
||||
- label: ":nvidia: (H200 MIG 35GB) Entrypoints Integration (Speech to Text)"
|
||||
device: h200_35gb
|
||||
key: entrypoints-integration-speech_to_text
|
||||
automatic_shard: true
|
||||
timeout_in_minutes: 45
|
||||
working_dir: "/vllm-workspace/tests"
|
||||
source_file_dependencies:
|
||||
@@ -233,6 +234,7 @@ steps:
|
||||
- label: ":nvidia: (H200 MIG 35GB) Entrypoints Integration (Pooling)"
|
||||
device: h200_35gb
|
||||
key: entrypoints-integration-pooling
|
||||
automatic_shard: true
|
||||
timeout_in_minutes: 75
|
||||
working_dir: "/vllm-workspace/tests"
|
||||
source_file_dependencies:
|
||||
|
||||
@@ -155,6 +155,7 @@ steps:
|
||||
- image-build-amd
|
||||
source_file_dependencies:
|
||||
- tests/models/test_deepseek_v4_vl_rocm.py
|
||||
- tests/models/test_deepseek_v4_aiter_mega_moe.py
|
||||
- vllm/models/deepseek_v4/amd/
|
||||
- vllm/models/deepseek_v4/common/
|
||||
- vllm/model_executor/layers/fused_moe/
|
||||
@@ -170,6 +171,7 @@ steps:
|
||||
# Common qnorm/RoPE/cache kernels already run in the H100 AMD mirror.
|
||||
# Pin the tested BF16 Triton routing and shared-expert path before imports.
|
||||
- VLLM_ROCM_USE_AITER_MOE=0 pytest -v -s models/test_deepseek_v4_vl_rocm.py
|
||||
- pytest -v -s models/test_deepseek_v4_aiter_mega_moe.py
|
||||
|
||||
# Own job so mHC never shares a process with FlashInfer's import-time
|
||||
# ptxas override.
|
||||
@@ -254,7 +256,9 @@ steps:
|
||||
--ignore=kernels/test_fused_qk_norm_rope_gate.py
|
||||
--ignore=kernels/test_fused_deepseek_v4_qnorm_rope_kv_insert.py
|
||||
--ignore=kernels/test_top_k_per_row.py
|
||||
--ignore=kernels/test_cpu_fused_sampling.py
|
||||
--ignore=kernels/test_bf16x3_router_gemm_cutedsl.py
|
||||
--ignore=kernels/test_bf16x3_router_gemm_rocm.py
|
||||
--ignore=kernels/test_ll_bf16_gemm.py
|
||||
--ignore=kernels/test_shuffle_rows.py
|
||||
# BROKEN on main, pending kernel fixes (B200):
|
||||
@@ -280,6 +284,7 @@ steps:
|
||||
kernels/test_apply_repetition_penalties.py
|
||||
kernels/test_awq_int4_to_int8.py
|
||||
kernels/test_bf16_skinny_gemm.py
|
||||
kernels/test_bf16x3_router_gemm_rocm.py
|
||||
kernels/test_cache_kernels.py
|
||||
kernels/test_compressor_kv_cache.py
|
||||
kernels/test_cp_gather_fp8.py
|
||||
@@ -499,6 +504,7 @@ steps:
|
||||
- label: ":nvidia: (H200 MIG 35GB) Mamba Kernels"
|
||||
device: h200_35gb
|
||||
key: kernels-mamba-test
|
||||
automatic_shard: true
|
||||
timeout_in_minutes: 60
|
||||
source_file_dependencies:
|
||||
- csrc/mamba/
|
||||
@@ -546,6 +552,21 @@ steps:
|
||||
- pytest -v -s kernels/attention/test_deepgemm_attention.py
|
||||
- pytest -v -s quantization/test_cutlass_w4a16.py
|
||||
|
||||
- label: ":nvidia: (B200) MiniMax-M3 Attention Kernels"
|
||||
key: kernels-minimax-m3-attention-b200
|
||||
timeout_in_minutes: 30
|
||||
working_dir: "/vllm-workspace/"
|
||||
device: b200-k8s
|
||||
num_devices: 1
|
||||
source_file_dependencies:
|
||||
- vllm/models/minimax_m3/
|
||||
- vllm/config/cache.py
|
||||
- vllm/v1/kv_cache_interface.py
|
||||
- tests/v1/attention/utils.py
|
||||
- tests/kernels/attention/test_minimax_m3.py
|
||||
commands:
|
||||
- pytest -v -s tests/kernels/attention/test_minimax_m3.py
|
||||
|
||||
- label: ":nvidia: (B200) Kernels Shard %N"
|
||||
key: kernels-b200
|
||||
timeout_in_minutes: 30
|
||||
@@ -595,7 +616,6 @@ steps:
|
||||
- tests/kernels/test_fused_minimax_m3_qknorm_rope_kv_insert.py
|
||||
- tests/kernels/mamba/test_gdn_prefill_cutedsl.py
|
||||
- tests/kernels/test_bf16x3_router_gemm_cutedsl.py
|
||||
- tests/kernels/attention/test_minimax_m3.py
|
||||
- tests/kernels/test_ll_bf16_gemm.py
|
||||
- tests/kernels/test_top_k_per_row.py
|
||||
- tests/kernels/quantization/test_silu_mul_nvfp4_quant.py
|
||||
@@ -641,7 +661,6 @@ steps:
|
||||
pytest -v -s tests/kernels/moe/test_flashinfer.py
|
||||
pytest -v -s tests/kernels/moe/test_trtllm_nvfp4_moe.py
|
||||
pytest -v -s tests/kernels/mamba/test_gdn_prefill_cutedsl.py
|
||||
pytest -v -s tests/kernels/attention/test_minimax_m3.py
|
||||
;;
|
||||
*)
|
||||
echo "unexpected BUILDKITE_PARALLEL_JOB=$$BUILDKITE_PARALLEL_JOB" >&2
|
||||
|
||||
@@ -5,6 +5,7 @@ steps:
|
||||
- label: ":nvidia: (H200 MIG 35GB) LM Eval Small Models"
|
||||
device: h200_35gb
|
||||
key: lm-eval-small-models
|
||||
automatic_shard: true
|
||||
timeout_in_minutes: 45
|
||||
source_file_dependencies:
|
||||
- csrc/
|
||||
@@ -208,7 +209,6 @@ steps:
|
||||
- tests/evals/gsm8k/
|
||||
- tests/utils.py
|
||||
- vllm/distributed/
|
||||
- vllm/transformers_utils/configs/diffusion_gemma.py
|
||||
- vllm/v1/attention/ops/
|
||||
commands:
|
||||
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-mi3xx-fp8-and-mixed.txt
|
||||
@@ -270,6 +270,9 @@ steps:
|
||||
source_file_dependencies:
|
||||
- .buildkite/test_areas/lm_eval.yaml
|
||||
- tests/evals/gsm8k/
|
||||
- tests/watermarking/test_compatibility.py
|
||||
- tests/watermarking/test_engine_compatibility.py
|
||||
- tests/watermarking/test_parallel_compatibility.py
|
||||
- vllm/config/speculative.py
|
||||
- vllm/config/vllm.py
|
||||
- vllm/config/watermarking.py
|
||||
@@ -282,6 +285,7 @@ steps:
|
||||
autorun_on_main: true
|
||||
commands:
|
||||
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-watermark-dspark-tp2.txt
|
||||
- pytest -s -v watermarking/test_compatibility.py watermarking/test_engine_compatibility.py watermarking/test_parallel_compatibility.py
|
||||
mirror:
|
||||
amd:
|
||||
label: ":amd: (MI355) LM Eval Watermark Feature Combination"
|
||||
@@ -318,9 +322,11 @@ steps:
|
||||
optional: true
|
||||
source_file_dependencies:
|
||||
- tests/evals/gsm8k/configs/DeepSeek-V4-Flash-DSpark-confidence-TEP4.yaml
|
||||
- tests/evals/gsm8k/configs/DeepSeek-V4.1-Flash-NVFP4-DSpark-TP4.yaml
|
||||
- tests/evals/gsm8k/configs/Kimi-K3-pruned75-DSpark-TP4.yaml
|
||||
- tests/evals/gsm8k/configs/models-spec-decode.txt
|
||||
- vllm/models/deepseek_v4/
|
||||
- vllm/models/deepseek_v41/
|
||||
- vllm/models/kimi_k3/
|
||||
- vllm/model_executor/models/qwen3_dspark.py
|
||||
- vllm/v1/spec_decode/metrics.py
|
||||
@@ -411,8 +417,6 @@ steps:
|
||||
- vllm/model_executor/models/qwen3_next_mtp.py
|
||||
- vllm/model_executor/models/nemotron_h.py
|
||||
- vllm/model_executor/models/nemotron_h_mtp.py
|
||||
- vllm/transformers_utils/configs/qwen3_next.py
|
||||
- vllm/transformers_utils/configs/nemotron_h.py
|
||||
- vllm/model_executor/model_loader/
|
||||
- vllm/model_executor/layers/quantization/
|
||||
- vllm/model_executor/layers/fused_moe/
|
||||
@@ -439,8 +443,6 @@ steps:
|
||||
source_file_dependencies:
|
||||
- vllm/model_executor/models/qwen3_5.py
|
||||
- vllm/model_executor/models/qwen3_5_mtp.py
|
||||
- vllm/transformers_utils/configs/qwen3_5.py
|
||||
- vllm/transformers_utils/configs/qwen3_5_moe.py
|
||||
- vllm/model_executor/models/qwen3_next.py
|
||||
- vllm/model_executor/models/qwen3_next_mtp.py
|
||||
- vllm/third_party/flash_linear_attention/ops/
|
||||
@@ -457,8 +459,6 @@ steps:
|
||||
source_file_dependencies:
|
||||
- vllm/model_executor/models/qwen3_5.py
|
||||
- vllm/model_executor/models/qwen3_5_mtp.py
|
||||
- vllm/transformers_utils/configs/qwen3_5.py
|
||||
- vllm/transformers_utils/configs/qwen3_5_moe.py
|
||||
- vllm/model_executor/models/qwen2.py
|
||||
- vllm/model_executor/models/qwen3.py
|
||||
- vllm/model_executor/models/qwen3_next.py
|
||||
|
||||
@@ -4,7 +4,8 @@ depends_on:
|
||||
steps:
|
||||
- label: ":nvidia: (H200 MIG 18GB) V1 Sample"
|
||||
key: v1-sample
|
||||
timeout_in_minutes: 35
|
||||
automatic_shard: true
|
||||
timeout_in_minutes: 45
|
||||
device: h200_18gb
|
||||
source_file_dependencies: &v1-sample-logits-deps
|
||||
- vllm/config/
|
||||
@@ -38,7 +39,7 @@ steps:
|
||||
|
||||
- label: ":nvidia: (H200 MIG 18GB) V1 Logits + Oracle"
|
||||
key: v1-logits-oracle
|
||||
timeout_in_minutes: 35
|
||||
timeout_in_minutes: 45
|
||||
device: h200_18gb
|
||||
source_file_dependencies: *v1-sample-logits-deps
|
||||
commands:
|
||||
@@ -296,7 +297,6 @@ steps:
|
||||
- vllm/model_executor/models/qwen3_next.py
|
||||
- vllm/model_executor/models/registry.py
|
||||
- vllm/transformers_utils/configs/extract_hidden_states.py
|
||||
- vllm/transformers_utils/configs/qwen3_5.py
|
||||
- vllm/v1/attention/backends/
|
||||
- vllm/v1/attention/selector.py
|
||||
- vllm/v1/kv_cache_interface.py
|
||||
@@ -556,6 +556,8 @@ steps:
|
||||
- tests/multimodal
|
||||
- tests/renderers
|
||||
- tests/standalone_tests/lazy_imports.py
|
||||
- tests/standalone_tests/test_precompiled_wheel.py
|
||||
- setup.py
|
||||
- tests/reasoning
|
||||
- tools/pre_commit/check_test_tethering.py
|
||||
- tools/pre_commit/test_tethering_allowlist.txt
|
||||
@@ -572,6 +574,7 @@ steps:
|
||||
commands:
|
||||
- pip install cmake # test_cmake_utils.py shells out to cmake; the CPU image has none
|
||||
- python3 standalone_tests/lazy_imports.py
|
||||
- python3 standalone_tests/test_precompiled_wheel.py
|
||||
- pytest -v -s device_allocator
|
||||
- pytest -v -s test_uvicorn_logging.py
|
||||
- pytest -v -s test_attention_backend_registry.py
|
||||
|
||||
@@ -68,8 +68,10 @@ steps:
|
||||
- tests/models/test_deepseek_v4_vl_rocm.py
|
||||
- tests/models/test_deepseek_v41_decoder_replay_layers.py
|
||||
- tests/models/test_deepseek_v41_replay_batch.py
|
||||
- tests/models/transformers/test_layer_registry_e2e.py
|
||||
- tests/models/test_deepseek_v41_replay_start.py
|
||||
commands:
|
||||
- pytest -v -s models/test_terratorch.py models/transformers/test_backend.py models/test_registry.py models/test_deepseek_v4_vl_rocm.py models/test_deepseek_v41_decoder_replay_layers.py models/test_deepseek_v41_replay_batch.py
|
||||
- pytest -v -s models/test_terratorch.py models/transformers/test_backend.py models/test_registry.py models/test_deepseek_v4_vl_rocm.py models/transformers/test_layer_registry_e2e.py models/test_deepseek_v41_replay_start.py models/test_deepseek_v41_decoder_replay_layers.py models/test_deepseek_v41_replay_batch.py
|
||||
mirror:
|
||||
amd:
|
||||
label: ":amd: (MI355) Basic Models (Other)"
|
||||
@@ -79,10 +81,18 @@ steps:
|
||||
depends_on:
|
||||
- image-build-amd
|
||||
source_file_dependencies:
|
||||
- vllm/models/deepseek_v4/
|
||||
- tests/models/test_deepseek_v4_vl_rocm.py
|
||||
- tests/models/test_hyv4_rocm.py
|
||||
- tests/models/transformers/test_layer_registry_e2e.py
|
||||
- tests/models/test_deepseek_v41_replay_start.py
|
||||
- tests/models/test_deepseek_v41_decoder_replay_layers.py
|
||||
- tests/models/test_deepseek_v41_replay_batch.py
|
||||
- tests/models/test_deepseek_v4_dspark_rocm.py
|
||||
- tests/models/test_deepseek_v4_rocm_wo_a.py
|
||||
- tests/models/test_deepseek_v4_rocm_compressor_gemm_fusion.py
|
||||
commands:
|
||||
- pytest -v -s models/test_terratorch.py models/transformers/test_backend.py models/test_registry.py models/test_deepseek_v4_vl_rocm.py
|
||||
- pytest -v -s models/test_terratorch.py models/transformers/test_backend.py models/test_registry.py models/test_deepseek_v4_vl_rocm.py models/transformers/test_layer_registry_e2e.py models/test_deepseek_v41_replay_start.py models/test_deepseek_v41_decoder_replay_layers.py models/test_deepseek_v41_replay_batch.py models/test_deepseek_v4_dspark_rocm.py models/test_deepseek_v4_rocm_wo_a.py models/test_deepseek_v4_rocm_compressor_gemm_fusion.py
|
||||
- VLLM_ROCM_USE_AITER=1 pytest -v -s models/test_hyv4_rocm.py -m 'not distributed'
|
||||
|
||||
- label: ":nvidia: (B200) Inkling"
|
||||
@@ -206,7 +216,7 @@ steps:
|
||||
depends_on:
|
||||
- image-build-amd
|
||||
commands:
|
||||
- pytest -v -s models/qwen4_exp/test_qsa_amd.py
|
||||
- pytest -v -s models/qwen4_exp/test_qsa_amd.py models/qwen4_exp/test_qsa_prepare_amd.py
|
||||
# PLE and QSA import conflicting HC registrations in a shared process.
|
||||
- pytest -v -s models/qwen4_exp/test_ple.py
|
||||
|
||||
@@ -229,7 +239,7 @@ steps:
|
||||
key: basic-models-test-other-cpu
|
||||
depends_on:
|
||||
- image-build-cpu
|
||||
timeout_in_minutes: 20
|
||||
timeout_in_minutes: 30
|
||||
source_file_dependencies:
|
||||
- vllm/
|
||||
- "!vllm/distributed/kv_transfer/"
|
||||
@@ -239,6 +249,11 @@ steps:
|
||||
- tests/models/test_qwen3_5_mtp_config.py
|
||||
- tests/models/transformers/fusers/
|
||||
- tests/model_executor/layers/test_activation.py
|
||||
- tests/models/test_language_model_cache_is_weak.py
|
||||
- tests/models/deepseek_v32/test_sequence_parallel.py
|
||||
- tests/models/test_deepseek_v4_fi_moe_ep.py
|
||||
- tests/models/test_dspark_mla.py
|
||||
- tests/models/transformers/test_layer_registry.py
|
||||
device: cpu-small
|
||||
commands:
|
||||
- pytest -v -s models/test_utils.py models/test_vision.py models/test_adapters.py models/test_qwen3_5_mtp_config.py models/transformers/fusers/ model_executor/layers/test_activation.py
|
||||
- pytest -v -s models/test_utils.py models/test_vision.py models/test_adapters.py models/test_qwen3_5_mtp_config.py models/transformers/fusers/ model_executor/layers/test_activation.py models/test_language_model_cache_is_weak.py models/deepseek_v32/test_sequence_parallel.py models/test_deepseek_v4_fi_moe_ep.py models/test_dspark_mla.py models/transformers/test_layer_registry.py
|
||||
|
||||
@@ -184,8 +184,6 @@ steps:
|
||||
source_file_dependencies:
|
||||
- vllm/model_executor/models/qwen3_5.py
|
||||
- vllm/model_executor/models/qwen3_5_mtp.py
|
||||
- vllm/transformers_utils/configs/qwen3_5.py
|
||||
- vllm/transformers_utils/configs/qwen3_5_moe.py
|
||||
- vllm/model_executor/models/qwen2.py
|
||||
- vllm/model_executor/models/qwen3.py
|
||||
- vllm/model_executor/models/qwen3_next.py
|
||||
@@ -222,10 +220,10 @@ steps:
|
||||
depends_on:
|
||||
- image-build-amd
|
||||
|
||||
- label: ":nvidia: (H200 MIG 18GB) Language Models (MTEB)"
|
||||
- label: ":nvidia: (H200 MIG 35GB) Language Models (MTEB)"
|
||||
key: language-models-test-mteb
|
||||
timeout_in_minutes: 68
|
||||
device: h200_18gb
|
||||
device: h200_35gb
|
||||
optional: true
|
||||
source_file_dependencies:
|
||||
- vllm/
|
||||
|
||||
@@ -123,6 +123,7 @@ steps:
|
||||
|
||||
- label: ":nvidia: (L4) PyTorch Compilation Passes"
|
||||
key: pytorch-compilation-passes-unit-tests
|
||||
automatic_shard: true
|
||||
device: l4
|
||||
num_devices: 1
|
||||
timeout_in_minutes: 45
|
||||
|
||||
@@ -12,6 +12,7 @@ steps:
|
||||
source_file_dependencies:
|
||||
- csrc/
|
||||
- vllm/model_executor/layers/quantization
|
||||
- vllm/models/deepseek_v41/quant_config.py
|
||||
- tests/quantization
|
||||
commands:
|
||||
# Pin torchao to the version compatible with the CI PyTorch and CUDA stack.
|
||||
@@ -79,6 +80,7 @@ steps:
|
||||
|
||||
- label: ":nvidia: (B200) Quantized MoE"
|
||||
key: quantized-moe-test-b200
|
||||
automatic_shard: true
|
||||
timeout_in_minutes: 120
|
||||
working_dir: "/vllm-workspace/"
|
||||
device: b200-k8s
|
||||
@@ -126,6 +128,7 @@ steps:
|
||||
- label: ":nvidia: (H200 MIG 35GB) Quantized Models"
|
||||
device: h200_35gb
|
||||
key: quantized-models-test
|
||||
automatic_shard: true
|
||||
timeout_in_minutes: 65
|
||||
env:
|
||||
VLLM_USE_V2_MODEL_RUNNER: "0"
|
||||
|
||||
@@ -0,0 +1,25 @@
|
||||
group: RL
|
||||
depends_on:
|
||||
- image-build
|
||||
steps:
|
||||
- label: ":nvidia: (H200 MIG 35GB) RL Entrypoints"
|
||||
device: h200_35gb
|
||||
key: rl-entrypoints-tests
|
||||
timeout_in_minutes: 40
|
||||
working_dir: "/vllm-workspace/tests"
|
||||
source_file_dependencies:
|
||||
- vllm/
|
||||
- "!vllm/distributed/kv_transfer/"
|
||||
- tests/entrypoints/rl
|
||||
commands:
|
||||
- pytest -v -s entrypoints/rl/weight_transfer
|
||||
- export VLLM_WORKER_MULTIPROC_METHOD=spawn
|
||||
- pytest -v -s entrypoints/rl --ignore=entrypoints/rl/weight_transfer
|
||||
mirror:
|
||||
amd:
|
||||
label: ":amd: (MI355 DPX) RL Entrypoints"
|
||||
dind: false
|
||||
device: mi355_dpx
|
||||
timeout_in_minutes: 50
|
||||
depends_on:
|
||||
- image-build-amd
|
||||
@@ -82,7 +82,6 @@ steps:
|
||||
- tests/entrypoints/scale_out/token_in_token_out/test_serving_tokens.py
|
||||
- tests/entrypoints/serve/instrumentator/test_basic.py
|
||||
- tests/entrypoints/serve/instrumentator/test_metrics.py
|
||||
# - tests/entrypoints/serve/dev/test_sleep.py
|
||||
- tests/entrypoints/serve/tokenize/test_tokenization.py
|
||||
commands:
|
||||
- export VLLM_USE_RUST_FRONTEND=1
|
||||
@@ -93,7 +92,6 @@ steps:
|
||||
# test_generate_logprobs expects Python-style top_logprobs truncation (dedup sampled + cap at max(k, 1)).
|
||||
- pytest -v -s entrypoints/scale_out/token_in_token_out/test_serving_tokens.py -k "not stream and not lora and not test_generate_logprobs and not stop_string_workflow"
|
||||
- pytest -v -s entrypoints/serve/instrumentator/test_metrics.py -k "text and not show and not run_batch and not test_metrics_counts and not test_metrics_exist"
|
||||
# - pytest -v -s entrypoints/serve/dev/test_sleep.py
|
||||
# /tokenizer_info is not implemented in the Rust frontend (the CLI flag is accepted as a no-op).
|
||||
- pytest -v -s entrypoints/serve/tokenize/test_tokenization.py -k "not tokenizer_info"
|
||||
mirror:
|
||||
|
||||
@@ -14,16 +14,6 @@ steps:
|
||||
- .buildkite/scripts/run-rust-frontend-cargo-ci.sh
|
||||
commands:
|
||||
- .buildkite/scripts/run-rust-frontend-cargo-ci.sh style-clippy
|
||||
mirror:
|
||||
amd:
|
||||
label: ":amd: (MI250) Rust Frontend Cargo Style + Clippy"
|
||||
dind: false
|
||||
device: mi250_1
|
||||
no_gpu: true
|
||||
working_dir: "/vllm-workspace"
|
||||
timeout_in_minutes: 35
|
||||
depends_on:
|
||||
- image-build-amd
|
||||
|
||||
- label: ":computer: (CPU) Rust Frontend Cargo"
|
||||
key: rust-frontend-cargo-tests
|
||||
@@ -38,13 +28,3 @@ steps:
|
||||
- .buildkite/scripts/run-rust-frontend-cargo-ci.sh
|
||||
commands:
|
||||
- .buildkite/scripts/run-rust-frontend-cargo-ci.sh test
|
||||
mirror:
|
||||
amd:
|
||||
label: ":amd: (MI250) Rust Frontend Cargo"
|
||||
dind: false
|
||||
device: mi250_1
|
||||
no_gpu: true
|
||||
working_dir: "/vllm-workspace"
|
||||
timeout_in_minutes: 35
|
||||
depends_on:
|
||||
- image-build-amd
|
||||
|
||||
@@ -5,6 +5,7 @@ steps:
|
||||
- label: ":nvidia: (H200 MIG 35GB) V1 Spec Decode"
|
||||
device: h200_35gb
|
||||
key: v1-spec-decode
|
||||
automatic_shard: true
|
||||
timeout_in_minutes: 40
|
||||
source_file_dependencies:
|
||||
- vllm/config/
|
||||
|
||||
Symlink
+1
@@ -0,0 +1 @@
|
||||
../../.agents/skills/model-recipes
|
||||
+1
@@ -0,0 +1 @@
|
||||
../../.agents/skills/transformers-upgrade
|
||||
+5
-5
@@ -4,6 +4,7 @@
|
||||
# This lists cover the "core" components of vLLM that require careful review
|
||||
/vllm/compilation @zou3519 @youkaichao @ProExpertProg @BoyuanFeng
|
||||
/vllm/distributed/kv_transfer @NickLucche @ApostaC @orozery @xuechendi @ivanium @wzhao18
|
||||
/vllm/distributed/kv_transfer/kv_connector/v1/hisparse @MatthewBonanni @LucasWilkinson
|
||||
/vllm/lora @jeejeelee
|
||||
/vllm/model_executor/layers/attention @LucasWilkinson @MatthewBonanni
|
||||
/vllm/model_executor/layers/fused_moe @mgoin @pavanimajety @zyongye @wzhao18
|
||||
@@ -44,7 +45,7 @@
|
||||
/vllm/entrypoints/serve @njhill
|
||||
/vllm/entrypoints/*.py @njhill
|
||||
/vllm/entrypoints/chat_utils.py @DarkLight1337
|
||||
/vllm/entrypoints/offline_utils.py @DarkLight1337
|
||||
/vllm/entrypoints/common @DarkLight1337
|
||||
/vllm/entrypoints/llm.py @DarkLight1337
|
||||
|
||||
# Rust Frontend
|
||||
@@ -77,6 +78,7 @@
|
||||
/vllm/v1/kv_cache_interface.py @heheda12345 @ivanium
|
||||
/vllm/v1/kv_offload @ApostaC @orozery
|
||||
/vllm/v1/simple_kv_offload @ivanium
|
||||
/vllm/v1/hisparse @MatthewBonanni @LucasWilkinson
|
||||
/vllm/v1/engine @njhill
|
||||
/vllm/v1/executor @njhill
|
||||
/vllm/v1/worker @njhill
|
||||
@@ -227,13 +229,11 @@ mkdocs.yaml @hmellor
|
||||
|
||||
# Reinforcement learning
|
||||
/vllm/distributed/weight_transfer @aoshen02
|
||||
/vllm/entrypoints/serve/dev/rlhf @njhill @aoshen02
|
||||
/vllm/entrypoints/rl @njhill @aoshen02
|
||||
/tests/distributed/test_weight_transfer.py @aoshen02
|
||||
/tests/entrypoints/weight_transfer @DarkLight1337 @robertgshaw2-redhat @aarnphm @NickLucche @AndreasKaratzas @aoshen02
|
||||
/tests/entrypoints/serve/dev/rlhf @DarkLight1337 @robertgshaw2-redhat @aarnphm @NickLucche @AndreasKaratzas @aoshen02
|
||||
/tests/entrypoints/rl @DarkLight1337 @robertgshaw2-redhat @njhill @aarnphm @NickLucche @AndreasKaratzas @aoshen02
|
||||
/tests/v1/worker/test_gpu_worker_weight_transfer.py @aoshen02
|
||||
/examples/rl @aoshen02
|
||||
/examples/features/pause_resume @aoshen02
|
||||
/docs/training/rlhf.md @aoshen02
|
||||
/docs/training/async_rl.md @aoshen02
|
||||
/docs/training/weight_transfer @aoshen02
|
||||
|
||||
@@ -549,8 +549,6 @@ pull_request_rules:
|
||||
- label != stale
|
||||
- or:
|
||||
- files~=tpu.py
|
||||
- files~=_tpu
|
||||
- files~=tpu_
|
||||
- files~=/tpu/
|
||||
actions:
|
||||
label:
|
||||
@@ -564,8 +562,6 @@ pull_request_rules:
|
||||
- label != stale
|
||||
- and:
|
||||
- -files~=tpu.py
|
||||
- -files~=_tpu
|
||||
- -files~=tpu_
|
||||
- -files~=/tpu/
|
||||
actions:
|
||||
label:
|
||||
|
||||
+6
-22
@@ -199,29 +199,13 @@ repos:
|
||||
additional_dependencies: [PyYAML, regex]
|
||||
verbose: true # full-scan `note:` lines are advisory, so the hook passes
|
||||
require_serial: true # one full scan, not one per pre-commit file batch
|
||||
- id: mypy-3.10 # TODO: Use https://github.com/pre-commit/mirrors-mypy when mypy setup is less awkward
|
||||
name: Run mypy for Python 3.10
|
||||
entry: python tools/pre_commit/mypy.py "3.10"
|
||||
<<: &mypy_common
|
||||
language: python
|
||||
types_or: [python, pyi]
|
||||
require_serial: true
|
||||
additional_dependencies: ["mypy==1.20.2", regex, types-cachetools, types-setuptools, types-PyYAML, types-requests, types-torch, pydantic]
|
||||
- id: mypy-3.11 # TODO: Use https://github.com/pre-commit/mirrors-mypy when mypy setup is less awkward
|
||||
name: Run mypy for Python 3.11
|
||||
- id: mypy # TODO: Use https://github.com/pre-commit/mirrors-mypy when mypy setup is less awkward
|
||||
name: Run mypy for the oldest supported Python version
|
||||
entry: python tools/pre_commit/mypy.py "3.11"
|
||||
<<: *mypy_common
|
||||
stages: [manual] # Only run in CI
|
||||
- id: mypy-3.12 # TODO: Use https://github.com/pre-commit/mirrors-mypy when mypy setup is less awkward
|
||||
name: Run mypy for Python 3.12
|
||||
entry: python tools/pre_commit/mypy.py "3.12"
|
||||
<<: *mypy_common
|
||||
stages: [manual] # Only run in CI
|
||||
- id: mypy-3.13 # TODO: Use https://github.com/pre-commit/mirrors-mypy when mypy setup is less awkward
|
||||
name: Run mypy for Python 3.13
|
||||
entry: python tools/pre_commit/mypy.py "3.13"
|
||||
<<: *mypy_common
|
||||
stages: [manual] # Only run in CI
|
||||
language: python
|
||||
types_or: [python, pyi]
|
||||
require_serial: true
|
||||
additional_dependencies: ["mypy==1.20.2", regex, types-cachetools, types-setuptools, types-PyYAML, types-requests, types-torch, pydantic]
|
||||
- id: png-lint
|
||||
name: Lint PNG exports from excalidraw
|
||||
entry: tools/pre_commit/png-lint.sh
|
||||
|
||||
@@ -31,7 +31,7 @@ Do not open one-off PRs for tiny edits (single typo, isolated style change, one
|
||||
- Test commands run and results.
|
||||
- Model evaluation results when the change affects output, accuracy, or serving.
|
||||
- Clear statement that AI assistance was used.
|
||||
- Use vLLM's [`/pr-checklist`](.agents/skills/pr-checklist/SKILL.md) skill before submitting a PR to ensure quality and adherence to the contribution guidelines.
|
||||
- Before opening a PR (drafts included) or requesting re-review, run the [`pr-checklist`](.agents/skills/pr-checklist/SKILL.md) skill and address its findings.
|
||||
|
||||
### Fail-closed behavior
|
||||
|
||||
@@ -114,9 +114,6 @@ pre-commit run --all-files
|
||||
|
||||
# Run a specific hook:
|
||||
pre-commit run ruff-check --all-files
|
||||
|
||||
# Run mypy as it is in CI:
|
||||
pre-commit run mypy-3.12 --all-files --hook-stage manual
|
||||
```
|
||||
|
||||
The line length limit for Python code is 88 characters. If you are not sure, use pre-commit to check.
|
||||
|
||||
+33
-27
@@ -46,7 +46,7 @@ install(CODE "set(CMAKE_INSTALL_LOCAL_ONLY TRUE)" ALL_COMPONENTS)
|
||||
# Supported python versions. These versions will be searched in order, the
|
||||
# first match will be selected. These should be kept in sync with setup.py.
|
||||
#
|
||||
set(PYTHON_SUPPORTED_VERSIONS "3.10" "3.11" "3.12" "3.13" "3.14")
|
||||
set(PYTHON_SUPPORTED_VERSIONS "3.11" "3.12" "3.13" "3.14")
|
||||
|
||||
# Supported AMD GPU architectures.
|
||||
set(HIP_SUPPORTED_ARCHS "gfx906;gfx908;gfx90a;gfx942;gfx950;gfx1250;gfx1030;gfx1100;gfx1101;gfx1102;gfx1103;gfx1150;gfx1151;gfx1152;gfx1153;gfx1200;gfx1201")
|
||||
@@ -134,40 +134,34 @@ endif()
|
||||
#
|
||||
# spinloop extension (pure CXX; must stay above the non-CUDA device branch so
|
||||
# CPU builds define the target before the early return)
|
||||
# This extension requires SABI 3.11 since it relies on Py_buffer support. Loading
|
||||
# failure is handled gracefully on vLLM side for lower Python versions.
|
||||
# Skip the target entirely on Python < 3.11 so the build doesn't break.
|
||||
# This extension requires SABI 3.11 since it relies on Py_buffer support.
|
||||
#
|
||||
if(Python_VERSION VERSION_GREATER_EQUAL "3.11")
|
||||
set(VLLM_SPINLOOP_EXT_SRC "csrc/spinloop.cpp")
|
||||
set(SPINLOOP_COMPILE_FLAGS "")
|
||||
if(CMAKE_SYSTEM_PROCESSOR MATCHES "x86_64|amd64")
|
||||
list(APPEND SPINLOOP_COMPILE_FLAGS "-mmwaitx")
|
||||
endif()
|
||||
define_extension_target(
|
||||
spinloop
|
||||
DESTINATION vllm
|
||||
LANGUAGE CXX
|
||||
SOURCES ${VLLM_SPINLOOP_EXT_SRC}
|
||||
COMPILE_FLAGS ${SPINLOOP_COMPILE_FLAGS}
|
||||
USE_SABI 3.11
|
||||
WITH_SOABI)
|
||||
set(VLLM_SPINLOOP_EXT_SRC "csrc/spinloop.cpp")
|
||||
set(SPINLOOP_COMPILE_FLAGS "")
|
||||
if(CMAKE_SYSTEM_PROCESSOR MATCHES "x86_64|amd64")
|
||||
list(APPEND SPINLOOP_COMPILE_FLAGS "-mmwaitx")
|
||||
endif()
|
||||
define_extension_target(
|
||||
spinloop
|
||||
DESTINATION vllm
|
||||
LANGUAGE CXX
|
||||
SOURCES ${VLLM_SPINLOOP_EXT_SRC}
|
||||
COMPILE_FLAGS ${SPINLOOP_COMPILE_FLAGS}
|
||||
USE_SABI 3.11
|
||||
WITH_SOABI)
|
||||
|
||||
#
|
||||
# fs_io extension (pure CXX; must stay above the non-CUDA device branch
|
||||
# so CPU builds define the target before the early return).
|
||||
# GIL-releasing filesystem helpers for FileSystemTierManager.
|
||||
#
|
||||
if(Python_VERSION VERSION_GREATER_EQUAL "3.11")
|
||||
define_extension_target(
|
||||
fs_io_C
|
||||
DESTINATION vllm
|
||||
LANGUAGE CXX
|
||||
SOURCES csrc/fs_io.cpp
|
||||
USE_SABI 3.11
|
||||
WITH_SOABI)
|
||||
endif()
|
||||
define_extension_target(
|
||||
fs_io_C
|
||||
DESTINATION vllm
|
||||
LANGUAGE CXX
|
||||
SOURCES csrc/fs_io.cpp
|
||||
USE_SABI 3.11
|
||||
WITH_SOABI)
|
||||
|
||||
#
|
||||
# Forward the non-CUDA device extensions to external CMake scripts.
|
||||
@@ -1574,6 +1568,18 @@ if(VLLM_GPU_LANG STREQUAL "CUDA" OR VLLM_GPU_LANG STREQUAL "HIP")
|
||||
include(cmake/external_projects/triton_kernels.cmake)
|
||||
endif()
|
||||
|
||||
# DeepGEMM and DeepSelect host code uses C++20 <format>, which libstdc++
|
||||
# provides only from GCC 13 onward.
|
||||
if (VLLM_GPU_LANG STREQUAL "CUDA"
|
||||
AND CMAKE_CXX_COMPILER_ID STREQUAL "GNU"
|
||||
AND CMAKE_CXX_COMPILER_VERSION VERSION_LESS "13")
|
||||
message(FATAL_ERROR
|
||||
"GCC >= 13 is required to build vLLM with CUDA (found ${CMAKE_CXX_COMPILER_VERSION}). "
|
||||
"The DeepGEMM and DeepSelect dependencies use C++20 <format> in host code, "
|
||||
"which libstdc++ does not provide before GCC 13. "
|
||||
"See: https://github.com/vllm-project/vllm/issues/58158")
|
||||
endif()
|
||||
|
||||
# For CUDA we also build and ship some external projects.
|
||||
if (VLLM_GPU_LANG STREQUAL "CUDA")
|
||||
include(cmake/external_projects/deepgemm.cmake)
|
||||
|
||||
@@ -0,0 +1,165 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
"""Eight-GPU AttnRes+FP8 projection comparison with rotated graph timing."""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import statistics
|
||||
from pathlib import Path
|
||||
|
||||
import torch
|
||||
import torch.distributed as dist
|
||||
|
||||
from vllm._aiter_ops import rocm_aiter_ops
|
||||
from vllm.config import VllmConfig, set_current_vllm_config
|
||||
from vllm.models.kimi_k3.amd.ops.attn_res import attn_res
|
||||
from vllm.platforms import current_platform
|
||||
|
||||
|
||||
def capture(fn, repeats):
|
||||
stream = torch.cuda.Stream()
|
||||
stream.wait_stream(torch.cuda.current_stream())
|
||||
with torch.cuda.stream(stream):
|
||||
for _ in range(5):
|
||||
fn()
|
||||
torch.cuda.current_stream().wait_stream(stream)
|
||||
torch.accelerator.synchronize()
|
||||
graph = torch.cuda.CUDAGraph()
|
||||
with torch.cuda.graph(graph):
|
||||
for _ in range(repeats):
|
||||
fn()
|
||||
return graph
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--output", type=Path, required=True)
|
||||
parser.add_argument(
|
||||
"--tokens", type=int, nargs="+", default=[1, 4, 16, 64, 256, 1024, 4096, 16384]
|
||||
)
|
||||
parser.add_argument("--blocks", type=int, nargs="+", default=[1, 4, 8])
|
||||
args = parser.parse_args()
|
||||
assert int(os.environ["WORLD_SIZE"]) == 8
|
||||
assert torch.accelerator.device_count() == 8
|
||||
rank = int(os.environ["LOCAL_RANK"])
|
||||
torch.accelerator.set_device_index(rank)
|
||||
torch.set_num_threads(1)
|
||||
torch.manual_seed(1729 + rank)
|
||||
dist.init_process_group("gloo")
|
||||
args.output.mkdir(parents=True, exist_ok=True)
|
||||
dtype = current_platform.fp8_dtype()
|
||||
rows = []
|
||||
with set_current_vllm_config(VllmConfig()):
|
||||
weight = torch.randn(2112, 7168, device="cuda", dtype=torch.bfloat16) * 0.02
|
||||
weight, ws = rocm_aiter_ops.per_token_quant(weight, dtype)
|
||||
weight = rocm_aiter_ops.shuffle_weight(weight)
|
||||
for m in args.tokens:
|
||||
prefix = torch.randn(m, 7168, device="cuda", dtype=torch.bfloat16)
|
||||
delta = torch.zeros_like(prefix)
|
||||
blocks = torch.randn(m, 9, 7168, device="cuda", dtype=torch.bfloat16)
|
||||
norm = torch.ones(7168, device="cuda", dtype=torch.bfloat16)
|
||||
score = torch.randn_like(norm) / 7168**0.5
|
||||
for b in args.blocks:
|
||||
|
||||
def produce(
|
||||
fused,
|
||||
prefix=prefix,
|
||||
delta=delta,
|
||||
blocks=blocks,
|
||||
norm=norm,
|
||||
score=score,
|
||||
b=b,
|
||||
):
|
||||
result = attn_res(
|
||||
prefix,
|
||||
delta,
|
||||
blocks,
|
||||
norm,
|
||||
score,
|
||||
norm,
|
||||
b,
|
||||
-1,
|
||||
1e-5,
|
||||
1e-5,
|
||||
quant_dtype=dtype if fused else None,
|
||||
)
|
||||
return (
|
||||
result
|
||||
if fused
|
||||
else rocm_aiter_ops.per_token_quant(result, dtype)
|
||||
)
|
||||
|
||||
def segment(fused, produce=produce):
|
||||
x, scale = produce(fused)
|
||||
return rocm_aiter_ops.preshuffled_per_token_w8a8_gemm(
|
||||
x, weight, scale, ws, output_dtype=torch.bfloat16
|
||||
)
|
||||
|
||||
expected, actual = segment(False), segment(True)
|
||||
relative_l2 = (
|
||||
(actual.float() - expected.float()).norm() / expected.float().norm()
|
||||
).item()
|
||||
assert relative_l2 < 0.002, relative_l2
|
||||
repeats = 20 if m < 1024 else 4
|
||||
functions = {
|
||||
"baseline_producer": lambda produce=produce: produce(False),
|
||||
"fused_producer": lambda produce=produce: produce(True),
|
||||
"baseline_segment": lambda segment=segment: segment(False),
|
||||
"fused_segment": lambda segment=segment: segment(True),
|
||||
}
|
||||
graphs = {k: capture(fn, repeats) for k, fn in functions.items()}
|
||||
samples = {k: [] for k in functions}
|
||||
names = list(functions)
|
||||
for sample in range(9):
|
||||
for name in names[sample % 4 :] + names[: sample % 4]:
|
||||
dist.barrier()
|
||||
graphs[name].replay()
|
||||
torch.accelerator.synchronize()
|
||||
start, end = [
|
||||
torch.cuda.Event(enable_timing=True) for _ in range(2)
|
||||
]
|
||||
start.record()
|
||||
graphs[name].replay()
|
||||
end.record()
|
||||
end.synchronize()
|
||||
samples[name].append(start.elapsed_time(end) * 1000 / repeats)
|
||||
row = {
|
||||
"tokens": m,
|
||||
"blocks": b,
|
||||
"relative_l2": relative_l2,
|
||||
"samples_us": samples,
|
||||
"median_us": {k: statistics.median(v) for k, v in samples.items()},
|
||||
}
|
||||
rows.append(row)
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"rank": rank,
|
||||
"tokens": m,
|
||||
"blocks": b,
|
||||
"median_us": row["median_us"],
|
||||
}
|
||||
),
|
||||
flush=True,
|
||||
)
|
||||
del graphs
|
||||
(args.output / f"rank-{rank}.json").write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"rank": rank,
|
||||
"world_size": 8,
|
||||
"torch": torch.__version__,
|
||||
"rows": rows,
|
||||
"scope": "AttnRes + FP8 quant + TP8 MLA input projection",
|
||||
},
|
||||
indent=2,
|
||||
)
|
||||
+ "\n"
|
||||
)
|
||||
dist.barrier()
|
||||
dist.destroy_process_group()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -303,9 +303,13 @@ def benchmark_config(
|
||||
run()
|
||||
torch.accelerator.synchronize()
|
||||
|
||||
# Capture 10 invocations with CUDA graph
|
||||
graph = torch.cuda.CUDAGraph()
|
||||
with torch.cuda.graph(graph):
|
||||
# Capture 10 invocations via graph(); CUDA needs a side stream
|
||||
graph = (
|
||||
torch.cuda.CUDAGraph()
|
||||
if current_platform.is_cuda_alike()
|
||||
else torch.xpu.XPUGraph()
|
||||
)
|
||||
with current_platform.graph(graph):
|
||||
for _ in range(10):
|
||||
run()
|
||||
torch.accelerator.synchronize()
|
||||
@@ -361,11 +365,33 @@ def get_rocm_tuning_space(use_fp16):
|
||||
return param_ranges
|
||||
|
||||
|
||||
def get_xpu_tuning_space(use_fp16):
|
||||
# BLOCK_SIZE_N=32 and num_warps=32 never won; grf_mode=256 beat 128 on B70.
|
||||
block_m_range = [16, 32, 64, 128, 256]
|
||||
block_n_range = [64, 128]
|
||||
block_k_range = [32, 64]
|
||||
num_warps_range = [4, 8, 16]
|
||||
group_m_range = [1, 8, 16]
|
||||
num_stage_range = [3, 4]
|
||||
|
||||
return {
|
||||
"BLOCK_SIZE_M": block_m_range,
|
||||
"BLOCK_SIZE_N": block_n_range,
|
||||
"BLOCK_SIZE_K": block_k_range,
|
||||
"GROUP_SIZE_M": group_m_range,
|
||||
"num_warps": num_warps_range,
|
||||
"num_stages": num_stage_range,
|
||||
"grf_mode": ["256"],
|
||||
}
|
||||
|
||||
|
||||
def get_configs_compute_bound(use_fp16, block_quant_shape) -> list[dict[str, int]]:
|
||||
configs: list[BenchmarkConfig] = []
|
||||
|
||||
if current_platform.is_rocm():
|
||||
param_ranges = get_rocm_tuning_space(use_fp16)
|
||||
elif current_platform.is_xpu():
|
||||
param_ranges = get_xpu_tuning_space(use_fp16)
|
||||
else:
|
||||
# Reduced search space for faster tuning.
|
||||
# TODO(woosuk): Increase the search space and use a performance model to
|
||||
@@ -408,6 +434,14 @@ def get_configs_compute_bound(use_fp16, block_quant_shape) -> list[dict[str, int
|
||||
return configs
|
||||
|
||||
|
||||
def prune_xpu_search_space(num_tokens, search_space, topk, num_experts):
|
||||
# Cap the tile by rows routed to one expert, not total rows: wider tiles
|
||||
# just pad and are the slowest to compile.
|
||||
rows_per_expert = (num_tokens * topk + num_experts - 1) // num_experts
|
||||
max_block_m = max(16, triton.next_power_of_2(rows_per_expert))
|
||||
return [c for c in search_space if c["BLOCK_SIZE_M"] <= max_block_m]
|
||||
|
||||
|
||||
def prune_rocm_search_space(
|
||||
num_tokens, shard_intermediate_size, hidden_size, search_space, is_fp16, topk
|
||||
):
|
||||
@@ -522,7 +556,7 @@ def merge_unique_dicts(list1, list2):
|
||||
@ray.remote(num_gpus=1)
|
||||
class BenchmarkWorker:
|
||||
def __init__(self, seed: int) -> None:
|
||||
torch.set_default_device("cuda")
|
||||
torch.set_default_device(current_platform.device_type)
|
||||
set_random_seed(seed)
|
||||
self.seed = seed
|
||||
# Get the device ID to allocate tensors and kernels
|
||||
@@ -619,6 +653,10 @@ class BenchmarkWorker:
|
||||
is_fp16,
|
||||
topk,
|
||||
)
|
||||
elif current_platform.is_xpu():
|
||||
search_space = prune_xpu_search_space(
|
||||
num_tokens, search_space, topk, num_experts
|
||||
)
|
||||
|
||||
need_device_guard = False
|
||||
if current_platform.is_rocm():
|
||||
@@ -690,6 +728,7 @@ def sort_config(config: BenchmarkConfig) -> BenchmarkConfig:
|
||||
else {}
|
||||
),
|
||||
**({"kpack": config["kpack"]} if "kpack" in config else {}),
|
||||
**({"grf_mode": config["grf_mode"]} if "grf_mode" in config else {}),
|
||||
**({"SPLIT_K": config["SPLIT_K"]} if "SPLIT_K" in config else {}),
|
||||
}
|
||||
|
||||
|
||||
@@ -56,13 +56,9 @@ def benchmark_permute(
|
||||
qhidden_states, input_gating, topk, False
|
||||
)
|
||||
scratch = MoEPermuteScratch(
|
||||
max_num_tokens=num_tokens,
|
||||
topk=topk,
|
||||
num_experts=num_experts,
|
||||
num_local_experts=num_experts,
|
||||
device=qhidden_states.device,
|
||||
hidden_size=hidden_size,
|
||||
hidden_dtype=qhidden_states.dtype,
|
||||
)
|
||||
|
||||
def prepare(i: int):
|
||||
@@ -135,13 +131,9 @@ def benchmark_unpermute(
|
||||
qhidden_states, input_gating, topk, False
|
||||
)
|
||||
scratch = MoEPermuteScratch(
|
||||
max_num_tokens=num_tokens,
|
||||
topk=topk,
|
||||
num_experts=num_experts,
|
||||
num_local_experts=num_experts,
|
||||
device=qhidden_states.device,
|
||||
hidden_size=hidden_size,
|
||||
hidden_dtype=qhidden_states.dtype,
|
||||
)
|
||||
|
||||
def prepare():
|
||||
|
||||
@@ -15,6 +15,9 @@ import os
|
||||
|
||||
import torch
|
||||
|
||||
from vllm.model_executor.kernels.linear.mixed_precision.rdna_hybrid_w4a16 import (
|
||||
pack_skinny_int4,
|
||||
)
|
||||
from vllm.triton_utils import triton
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -56,20 +59,25 @@ def prepare_hybrid_weights(K, N, group_size, device="cuda"):
|
||||
|
||||
Returns (w_q_skinny, w_s_skinny, w_fp16, w_zp). The triton path derives
|
||||
its int32 view from w_q_skinny, so no separate int32 buffer is returned.
|
||||
|
||||
Weights go through ``pack_skinny_int4``, the packer the layer uses, so the
|
||||
benchmark sees the production row stride rather than an always-dense one.
|
||||
"""
|
||||
num_groups = K // group_size
|
||||
|
||||
# Random packed weights — actual values don't matter for throughput
|
||||
w_q_skinny_i32 = torch.randint(
|
||||
0, 2**31, (N, K // 8), dtype=torch.int32, device=device
|
||||
)
|
||||
w_q_skinny = w_q_skinny_i32.view(torch.int8).contiguous()
|
||||
# Values don't matter for throughput, but the layout does.
|
||||
unpacked = torch.randint(0, 16, (N, K), dtype=torch.int32, device=device)
|
||||
w_q_skinny = pack_skinny_int4(unpacked)
|
||||
del unpacked
|
||||
w_s_skinny = torch.randn(N, num_groups, dtype=torch.float16, device=device) * 0.01
|
||||
|
||||
# Raw per-group zero-points for asymmetric benchmarks
|
||||
w_zp = torch.randint(0, 16, (N, num_groups), dtype=torch.int32, device=device).to(
|
||||
torch.float16
|
||||
)
|
||||
# Per-group zero-points for asymmetric benchmarks, in the packed layout the
|
||||
# kernels read: [N//8, num_groups] int32, row n's nibble at bits 4*(n%8).
|
||||
zp_raw = torch.randint(0, 16, (N, num_groups), dtype=torch.int32, device=device)
|
||||
shifts = (torch.arange(8, device=device, dtype=torch.int32) * 4)[:, None]
|
||||
w_zp = torch.sum(
|
||||
(zp_raw.view(N // 8, 8, num_groups) & 0xF) << shifts, dim=1, dtype=torch.int32
|
||||
).contiguous()
|
||||
|
||||
# FP16 baseline for F.linear
|
||||
w_fp16 = torch.randn(N, K, dtype=torch.float16, device=device) * 0.01
|
||||
@@ -97,6 +105,13 @@ PROVIDERS = ["torch-fp16", "hybrid-w4a16", "hybrid-w4a16-zp"]
|
||||
)
|
||||
)
|
||||
def benchmark(batch_size, provider, N, K, group_size, weights):
|
||||
"""Time one GEMM.
|
||||
|
||||
``do_bench`` rather than ``do_bench_cudagraph``: it zeroes a 256 MB buffer
|
||||
between reps, so the weight is read from DRAM as it would be in a model.
|
||||
With a cudagraph replay the weight stays resident in the 32 MB MALL, which
|
||||
flatters wide-BLOCK_N tiles and hides row-stride effects entirely.
|
||||
"""
|
||||
M = batch_size
|
||||
device = "cuda"
|
||||
dtype = torch.float16
|
||||
@@ -106,7 +121,7 @@ def benchmark(batch_size, provider, N, K, group_size, weights):
|
||||
|
||||
if provider == "torch-fp16":
|
||||
w_fp16 = weights["w_fp16"]
|
||||
ms, min_ms, max_ms = triton.testing.do_bench_cudagraph(
|
||||
ms, min_ms, max_ms = triton.testing.do_bench(
|
||||
lambda: torch.nn.functional.linear(a, w_fp16),
|
||||
quantiles=quantiles,
|
||||
)
|
||||
@@ -133,7 +148,7 @@ def benchmark(batch_size, provider, N, K, group_size, weights):
|
||||
group_size,
|
||||
)
|
||||
|
||||
ms, min_ms, max_ms = triton.testing.do_bench_cudagraph(
|
||||
ms, min_ms, max_ms = triton.testing.do_bench(
|
||||
run,
|
||||
quantiles=quantiles,
|
||||
)
|
||||
|
||||
@@ -0,0 +1,130 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
"""Benchmark the ROCm bf16 sparse-MLA decode split-K kernel in isolation.
|
||||
|
||||
Example usage:
|
||||
python3 benchmarks/kernels/benchmark_sparse_mla_decode_bf16.py \
|
||||
--rows 1 4 16 64 --lens 2048 --splits auto 4 16 32
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import itertools
|
||||
import json
|
||||
|
||||
import torch
|
||||
|
||||
from vllm.triton_utils import triton
|
||||
from vllm.v1.attention.ops.rocm_aiter_mla_sparse import (
|
||||
_rocm_sparse_attn_decode_ragged_bf16_triton,
|
||||
_rocm_sparse_attn_prefill_ragged_triton,
|
||||
rocm_sparse_decode_bf16_num_splits,
|
||||
)
|
||||
|
||||
BLOCK_H = 16
|
||||
|
||||
|
||||
def make_inputs(rows: int, live: int, args):
|
||||
torch.manual_seed(args.seed)
|
||||
kv = (
|
||||
torch.randn(args.capacity, args.head_dim, device="cuda", dtype=torch.bfloat16)
|
||||
* 0.125
|
||||
)
|
||||
q = (
|
||||
torch.randn(
|
||||
rows, args.heads, args.head_dim, device="cuda", dtype=torch.bfloat16
|
||||
)
|
||||
* 0.125
|
||||
)
|
||||
slots = torch.stack(
|
||||
[torch.randperm(args.capacity, device="cuda")[:live] for _ in range(rows)]
|
||||
)
|
||||
indices = slots.reshape(-1).to(torch.int32)
|
||||
indptr = torch.arange(rows + 1, device="cuda", dtype=torch.int32) * live
|
||||
sink = (
|
||||
torch.randn(args.heads, device="cuda", dtype=torch.float32)
|
||||
if args.sink
|
||||
else None
|
||||
)
|
||||
return q, kv, indices, indptr, sink
|
||||
|
||||
|
||||
def resolve_splits(spec: str, rows: int, live: int, args) -> int:
|
||||
if spec != "auto":
|
||||
return int(spec)
|
||||
return rocm_sparse_decode_bf16_num_splits(rows, args.heads, live)
|
||||
|
||||
|
||||
def benchmark(backend: str, rows: int, live: int, spec: str, args) -> dict:
|
||||
q, kv, indices, indptr, sink = make_inputs(rows, live, args)
|
||||
nope, rope = args.head_dim, 0
|
||||
splits = resolve_splits(spec, rows, live, args)
|
||||
|
||||
def ragged():
|
||||
return _rocm_sparse_attn_prefill_ragged_triton(
|
||||
q, kv, indices, indptr, args.scale, sink, nope, rope
|
||||
)
|
||||
|
||||
def splitk():
|
||||
return _rocm_sparse_attn_decode_ragged_bf16_triton(
|
||||
q, kv, indices, indptr, args.scale, sink, nope, rope, splits
|
||||
)
|
||||
|
||||
run = ragged if backend == "ragged" else splitk
|
||||
torch.testing.assert_close(run(), ragged(), atol=2e-2, rtol=2e-2)
|
||||
ms = triton.testing.do_bench(run, warmup=args.warmup, rep=args.rep)
|
||||
return dict(
|
||||
backend=backend,
|
||||
rows=rows,
|
||||
live=live,
|
||||
heads=args.heads,
|
||||
head_dim=args.head_dim,
|
||||
splits=splits if backend == "splitk" else 1,
|
||||
workgroups=rows * triton.cdiv(args.heads, BLOCK_H) * splits,
|
||||
us=ms * 1000,
|
||||
kv_gbps=rows * live * args.head_dim * 2 / (ms * 1e6),
|
||||
)
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--rows", type=int, nargs="+", default=[1, 2, 4, 8, 16, 64])
|
||||
parser.add_argument(
|
||||
"--lens",
|
||||
type=int,
|
||||
nargs="+",
|
||||
default=[2048],
|
||||
help="Selected KV slots per row; GLM-5.3-Flash uses index_topk=2048",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--splits",
|
||||
nargs="+",
|
||||
default=["auto"],
|
||||
help="Split counts to sweep, or auto for the runtime heuristic",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--backends",
|
||||
nargs="+",
|
||||
choices=["ragged", "splitk"],
|
||||
default=["ragged", "splitk"],
|
||||
)
|
||||
parser.add_argument(
|
||||
"--heads", type=int, default=8, help="Heads per rank; 64 at TP8 gives 8"
|
||||
)
|
||||
parser.add_argument("--head-dim", type=int, default=512)
|
||||
parser.add_argument("--capacity", type=int, default=65536)
|
||||
parser.add_argument("--scale", type=float, default=0.044194173824159216)
|
||||
parser.add_argument("--sink", action="store_true")
|
||||
parser.add_argument("--warmup", type=int, default=25)
|
||||
parser.add_argument("--rep", type=int, default=200)
|
||||
parser.add_argument("--seed", type=int, default=123)
|
||||
args = parser.parse_args()
|
||||
|
||||
print(json.dumps(dict(gpu=torch.cuda.get_device_name(0), args=vars(args))))
|
||||
for backend, rows, live in itertools.product(args.backends, args.rows, args.lens):
|
||||
specs = ["1"] if backend == "ragged" else args.splits
|
||||
for spec in specs:
|
||||
print(json.dumps(benchmark(backend, rows, live, spec, args)), flush=True)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,518 +0,0 @@
|
||||
FROM fairest creatures we desire increase,
|
||||
That thereby beauty's rose might never die,
|
||||
But as the riper should by time decease,
|
||||
His tender heir might bear his memory:
|
||||
But thou, contracted to thine own bright eyes,
|
||||
Feed'st thy light'st flame with self-substantial fuel,
|
||||
Making a famine where abundance lies,
|
||||
Thyself thy foe, to thy sweet self too cruel.
|
||||
Thou that art now the world's fresh ornament
|
||||
And only herald to the gaudy spring,
|
||||
Within thine own bud buriest thy content
|
||||
And, tender churl, makest waste in niggarding.
|
||||
Pity the world, or else this glutton be,
|
||||
To eat the world's due, by the grave and thee.
|
||||
When forty winters shall beseige thy brow,
|
||||
And dig deep trenches in thy beauty's field,
|
||||
Thy youth's proud livery, so gazed on now,
|
||||
Will be a tatter'd weed, of small worth held:
|
||||
Then being ask'd where all thy beauty lies,
|
||||
Where all the treasure of thy lusty days,
|
||||
To say, within thine own deep-sunken eyes,
|
||||
Were an all-eating shame and thriftless praise.
|
||||
How much more praise deserved thy beauty's use,
|
||||
If thou couldst answer 'This fair child of mine
|
||||
Shall sum my count and make my old excuse,'
|
||||
Proving his beauty by succession thine!
|
||||
This were to be new made when thou art old,
|
||||
And see thy blood warm when thou feel'st it cold.
|
||||
Look in thy glass, and tell the face thou viewest
|
||||
Now is the time that face should form another;
|
||||
Whose fresh repair if now thou not renewest,
|
||||
Thou dost beguile the world, unbless some mother.
|
||||
For where is she so fair whose unear'd womb
|
||||
Disdains the tillage of thy husbandry?
|
||||
Or who is he so fond will be the tomb
|
||||
Of his self-love, to stop posterity?
|
||||
Thou art thy mother's glass, and she in thee
|
||||
Calls back the lovely April of her prime:
|
||||
So thou through windows of thine age shall see
|
||||
Despite of wrinkles this thy golden time.
|
||||
But if thou live, remember'd not to be,
|
||||
Die single, and thine image dies with thee.
|
||||
Unthrifty loveliness, why dost thou spend
|
||||
Upon thyself thy beauty's legacy?
|
||||
Nature's bequest gives nothing but doth lend,
|
||||
And being frank she lends to those are free.
|
||||
Then, beauteous niggard, why dost thou abuse
|
||||
The bounteous largess given thee to give?
|
||||
Profitless usurer, why dost thou use
|
||||
So great a sum of sums, yet canst not live?
|
||||
For having traffic with thyself alone,
|
||||
Thou of thyself thy sweet self dost deceive.
|
||||
Then how, when nature calls thee to be gone,
|
||||
What acceptable audit canst thou leave?
|
||||
Thy unused beauty must be tomb'd with thee,
|
||||
Which, used, lives th' executor to be.
|
||||
Those hours, that with gentle work did frame
|
||||
The lovely gaze where every eye doth dwell,
|
||||
Will play the tyrants to the very same
|
||||
And that unfair which fairly doth excel:
|
||||
For never-resting time leads summer on
|
||||
To hideous winter and confounds him there;
|
||||
Sap cheque'd with frost and lusty leaves quite gone,
|
||||
Beauty o'ersnow'd and bareness every where:
|
||||
Then, were not summer's distillation left,
|
||||
A liquid prisoner pent in walls of glass,
|
||||
Beauty's effect with beauty were bereft,
|
||||
Nor it nor no remembrance what it was:
|
||||
But flowers distill'd though they with winter meet,
|
||||
Leese but their show; their substance still lives sweet.
|
||||
Then let not winter's ragged hand deface
|
||||
In thee thy summer, ere thou be distill'd:
|
||||
Make sweet some vial; treasure thou some place
|
||||
With beauty's treasure, ere it be self-kill'd.
|
||||
That use is not forbidden usury,
|
||||
Which happies those that pay the willing loan;
|
||||
That's for thyself to breed another thee,
|
||||
Or ten times happier, be it ten for one;
|
||||
Ten times thyself were happier than thou art,
|
||||
If ten of thine ten times refigured thee:
|
||||
Then what could death do, if thou shouldst depart,
|
||||
Leaving thee living in posterity?
|
||||
Be not self-will'd, for thou art much too fair
|
||||
To be death's conquest and make worms thine heir.
|
||||
Lo! in the orient when the gracious light
|
||||
Lifts up his burning head, each under eye
|
||||
Doth homage to his new-appearing sight,
|
||||
Serving with looks his sacred majesty;
|
||||
And having climb'd the steep-up heavenly hill,
|
||||
Resembling strong youth in his middle age,
|
||||
yet mortal looks adore his beauty still,
|
||||
Attending on his golden pilgrimage;
|
||||
But when from highmost pitch, with weary car,
|
||||
Like feeble age, he reeleth from the day,
|
||||
The eyes, 'fore duteous, now converted are
|
||||
From his low tract and look another way:
|
||||
So thou, thyself out-going in thy noon,
|
||||
Unlook'd on diest, unless thou get a son.
|
||||
Music to hear, why hear'st thou music sadly?
|
||||
Sweets with sweets war not, joy delights in joy.
|
||||
Why lovest thou that which thou receivest not gladly,
|
||||
Or else receivest with pleasure thine annoy?
|
||||
If the true concord of well-tuned sounds,
|
||||
By unions married, do offend thine ear,
|
||||
They do but sweetly chide thee, who confounds
|
||||
In singleness the parts that thou shouldst bear.
|
||||
Mark how one string, sweet husband to another,
|
||||
Strikes each in each by mutual ordering,
|
||||
Resembling sire and child and happy mother
|
||||
Who all in one, one pleasing note do sing:
|
||||
Whose speechless song, being many, seeming one,
|
||||
Sings this to thee: 'thou single wilt prove none.'
|
||||
Is it for fear to wet a widow's eye
|
||||
That thou consumest thyself in single life?
|
||||
Ah! if thou issueless shalt hap to die.
|
||||
The world will wail thee, like a makeless wife;
|
||||
The world will be thy widow and still weep
|
||||
That thou no form of thee hast left behind,
|
||||
When every private widow well may keep
|
||||
By children's eyes her husband's shape in mind.
|
||||
Look, what an unthrift in the world doth spend
|
||||
Shifts but his place, for still the world enjoys it;
|
||||
But beauty's waste hath in the world an end,
|
||||
And kept unused, the user so destroys it.
|
||||
No love toward others in that bosom sits
|
||||
That on himself such murderous shame commits.
|
||||
For shame! deny that thou bear'st love to any,
|
||||
Who for thyself art so unprovident.
|
||||
Grant, if thou wilt, thou art beloved of many,
|
||||
But that thou none lovest is most evident;
|
||||
For thou art so possess'd with murderous hate
|
||||
That 'gainst thyself thou stick'st not to conspire.
|
||||
Seeking that beauteous roof to ruinate
|
||||
Which to repair should be thy chief desire.
|
||||
O, change thy thought, that I may change my mind!
|
||||
Shall hate be fairer lodged than gentle love?
|
||||
Be, as thy presence is, gracious and kind,
|
||||
Or to thyself at least kind-hearted prove:
|
||||
Make thee another self, for love of me,
|
||||
That beauty still may live in thine or thee.
|
||||
As fast as thou shalt wane, so fast thou growest
|
||||
In one of thine, from that which thou departest;
|
||||
And that fresh blood which youngly thou bestowest
|
||||
Thou mayst call thine when thou from youth convertest.
|
||||
Herein lives wisdom, beauty and increase:
|
||||
Without this, folly, age and cold decay:
|
||||
If all were minded so, the times should cease
|
||||
And threescore year would make the world away.
|
||||
Let those whom Nature hath not made for store,
|
||||
Harsh featureless and rude, barrenly perish:
|
||||
Look, whom she best endow'd she gave the more;
|
||||
Which bounteous gift thou shouldst in bounty cherish:
|
||||
She carved thee for her seal, and meant thereby
|
||||
Thou shouldst print more, not let that copy die.
|
||||
When I do count the clock that tells the time,
|
||||
And see the brave day sunk in hideous night;
|
||||
When I behold the violet past prime,
|
||||
And sable curls all silver'd o'er with white;
|
||||
When lofty trees I see barren of leaves
|
||||
Which erst from heat did canopy the herd,
|
||||
And summer's green all girded up in sheaves
|
||||
Borne on the bier with white and bristly beard,
|
||||
Then of thy beauty do I question make,
|
||||
That thou among the wastes of time must go,
|
||||
Since sweets and beauties do themselves forsake
|
||||
And die as fast as they see others grow;
|
||||
And nothing 'gainst Time's scythe can make defence
|
||||
Save breed, to brave him when he takes thee hence.
|
||||
O, that you were yourself! but, love, you are
|
||||
No longer yours than you yourself here live:
|
||||
Against this coming end you should prepare,
|
||||
And your sweet semblance to some other give.
|
||||
So should that beauty which you hold in lease
|
||||
Find no determination: then you were
|
||||
Yourself again after yourself's decease,
|
||||
When your sweet issue your sweet form should bear.
|
||||
Who lets so fair a house fall to decay,
|
||||
Which husbandry in honour might uphold
|
||||
Against the stormy gusts of winter's day
|
||||
And barren rage of death's eternal cold?
|
||||
O, none but unthrifts! Dear my love, you know
|
||||
You had a father: let your son say so.
|
||||
Not from the stars do I my judgment pluck;
|
||||
And yet methinks I have astronomy,
|
||||
But not to tell of good or evil luck,
|
||||
Of plagues, of dearths, or seasons' quality;
|
||||
Nor can I fortune to brief minutes tell,
|
||||
Pointing to each his thunder, rain and wind,
|
||||
Or say with princes if it shall go well,
|
||||
By oft predict that I in heaven find:
|
||||
But from thine eyes my knowledge I derive,
|
||||
And, constant stars, in them I read such art
|
||||
As truth and beauty shall together thrive,
|
||||
If from thyself to store thou wouldst convert;
|
||||
Or else of thee this I prognosticate:
|
||||
Thy end is truth's and beauty's doom and date.
|
||||
When I consider every thing that grows
|
||||
Holds in perfection but a little moment,
|
||||
That this huge stage presenteth nought but shows
|
||||
Whereon the stars in secret influence comment;
|
||||
When I perceive that men as plants increase,
|
||||
Cheered and cheque'd even by the self-same sky,
|
||||
Vaunt in their youthful sap, at height decrease,
|
||||
And wear their brave state out of memory;
|
||||
Then the conceit of this inconstant stay
|
||||
Sets you most rich in youth before my sight,
|
||||
Where wasteful Time debateth with Decay,
|
||||
To change your day of youth to sullied night;
|
||||
And all in war with Time for love of you,
|
||||
As he takes from you, I engraft you new.
|
||||
But wherefore do not you a mightier way
|
||||
Make war upon this bloody tyrant, Time?
|
||||
And fortify yourself in your decay
|
||||
With means more blessed than my barren rhyme?
|
||||
Now stand you on the top of happy hours,
|
||||
And many maiden gardens yet unset
|
||||
With virtuous wish would bear your living flowers,
|
||||
Much liker than your painted counterfeit:
|
||||
So should the lines of life that life repair,
|
||||
Which this, Time's pencil, or my pupil pen,
|
||||
Neither in inward worth nor outward fair,
|
||||
Can make you live yourself in eyes of men.
|
||||
To give away yourself keeps yourself still,
|
||||
And you must live, drawn by your own sweet skill.
|
||||
Who will believe my verse in time to come,
|
||||
If it were fill'd with your most high deserts?
|
||||
Though yet, heaven knows, it is but as a tomb
|
||||
Which hides your life and shows not half your parts.
|
||||
If I could write the beauty of your eyes
|
||||
And in fresh numbers number all your graces,
|
||||
The age to come would say 'This poet lies:
|
||||
Such heavenly touches ne'er touch'd earthly faces.'
|
||||
So should my papers yellow'd with their age
|
||||
Be scorn'd like old men of less truth than tongue,
|
||||
And your true rights be term'd a poet's rage
|
||||
And stretched metre of an antique song:
|
||||
But were some child of yours alive that time,
|
||||
You should live twice; in it and in my rhyme.
|
||||
Shall I compare thee to a summer's day?
|
||||
Thou art more lovely and more temperate:
|
||||
Rough winds do shake the darling buds of May,
|
||||
And summer's lease hath all too short a date:
|
||||
Sometime too hot the eye of heaven shines,
|
||||
And often is his gold complexion dimm'd;
|
||||
And every fair from fair sometime declines,
|
||||
By chance or nature's changing course untrimm'd;
|
||||
But thy eternal summer shall not fade
|
||||
Nor lose possession of that fair thou owest;
|
||||
Nor shall Death brag thou wander'st in his shade,
|
||||
When in eternal lines to time thou growest:
|
||||
So long as men can breathe or eyes can see,
|
||||
So long lives this and this gives life to thee.
|
||||
Devouring Time, blunt thou the lion's paws,
|
||||
And make the earth devour her own sweet brood;
|
||||
Pluck the keen teeth from the fierce tiger's jaws,
|
||||
And burn the long-lived phoenix in her blood;
|
||||
Make glad and sorry seasons as thou fleets,
|
||||
And do whate'er thou wilt, swift-footed Time,
|
||||
To the wide world and all her fading sweets;
|
||||
But I forbid thee one most heinous crime:
|
||||
O, carve not with thy hours my love's fair brow,
|
||||
Nor draw no lines there with thine antique pen;
|
||||
Him in thy course untainted do allow
|
||||
For beauty's pattern to succeeding men.
|
||||
Yet, do thy worst, old Time: despite thy wrong,
|
||||
My love shall in my verse ever live young.
|
||||
A woman's face with Nature's own hand painted
|
||||
Hast thou, the master-mistress of my passion;
|
||||
A woman's gentle heart, but not acquainted
|
||||
With shifting change, as is false women's fashion;
|
||||
An eye more bright than theirs, less false in rolling,
|
||||
Gilding the object whereupon it gazeth;
|
||||
A man in hue, all 'hues' in his controlling,
|
||||
Much steals men's eyes and women's souls amazeth.
|
||||
And for a woman wert thou first created;
|
||||
Till Nature, as she wrought thee, fell a-doting,
|
||||
And by addition me of thee defeated,
|
||||
By adding one thing to my purpose nothing.
|
||||
But since she prick'd thee out for women's pleasure,
|
||||
Mine be thy love and thy love's use their treasure.
|
||||
So is it not with me as with that Muse
|
||||
Stirr'd by a painted beauty to his verse,
|
||||
Who heaven itself for ornament doth use
|
||||
And every fair with his fair doth rehearse
|
||||
Making a couplement of proud compare,
|
||||
With sun and moon, with earth and sea's rich gems,
|
||||
With April's first-born flowers, and all things rare
|
||||
That heaven's air in this huge rondure hems.
|
||||
O' let me, true in love, but truly write,
|
||||
And then believe me, my love is as fair
|
||||
As any mother's child, though not so bright
|
||||
As those gold candles fix'd in heaven's air:
|
||||
Let them say more than like of hearsay well;
|
||||
I will not praise that purpose not to sell.
|
||||
My glass shall not persuade me I am old,
|
||||
So long as youth and thou are of one date;
|
||||
But when in thee time's furrows I behold,
|
||||
Then look I death my days should expiate.
|
||||
For all that beauty that doth cover thee
|
||||
Is but the seemly raiment of my heart,
|
||||
Which in thy breast doth live, as thine in me:
|
||||
How can I then be elder than thou art?
|
||||
O, therefore, love, be of thyself so wary
|
||||
As I, not for myself, but for thee will;
|
||||
Bearing thy heart, which I will keep so chary
|
||||
As tender nurse her babe from faring ill.
|
||||
Presume not on thy heart when mine is slain;
|
||||
Thou gavest me thine, not to give back again.
|
||||
As an unperfect actor on the stage
|
||||
Who with his fear is put besides his part,
|
||||
Or some fierce thing replete with too much rage,
|
||||
Whose strength's abundance weakens his own heart.
|
||||
So I, for fear of trust, forget to say
|
||||
The perfect ceremony of love's rite,
|
||||
And in mine own love's strength seem to decay,
|
||||
O'ercharged with burden of mine own love's might.
|
||||
O, let my books be then the eloquence
|
||||
And dumb presagers of my speaking breast,
|
||||
Who plead for love and look for recompense
|
||||
More than that tongue that more hath more express'd.
|
||||
O, learn to read what silent love hath writ:
|
||||
To hear with eyes belongs to love's fine wit.
|
||||
Mine eye hath play'd the painter and hath stell'd
|
||||
Thy beauty's form in table of my heart;
|
||||
My body is the frame wherein 'tis held,
|
||||
And perspective it is the painter's art.
|
||||
For through the painter must you see his skill,
|
||||
To find where your true image pictured lies;
|
||||
Which in my bosom's shop is hanging still,
|
||||
That hath his windows glazed with thine eyes.
|
||||
Now see what good turns eyes for eyes have done:
|
||||
Mine eyes have drawn thy shape, and thine for me
|
||||
Are windows to my breast, where-through the sun
|
||||
Delights to peep, to gaze therein on thee;
|
||||
Yet eyes this cunning want to grace their art;
|
||||
They draw but what they see, know not the heart.
|
||||
Let those who are in favour with their stars
|
||||
Of public honour and proud titles boast,
|
||||
Whilst I, whom fortune of such triumph bars,
|
||||
Unlook'd for joy in that I honour most.
|
||||
Great princes' favourites their fair leaves spread
|
||||
But as the marigold at the sun's eye,
|
||||
And in themselves their pride lies buried,
|
||||
For at a frown they in their glory die.
|
||||
The painful warrior famoused for fight,
|
||||
After a thousand victories once foil'd,
|
||||
Is from the book of honour razed quite,
|
||||
And all the rest forgot for which he toil'd:
|
||||
Then happy I, that love and am beloved
|
||||
Where I may not remove nor be removed.
|
||||
Lord of my love, to whom in vassalage
|
||||
Thy merit hath my duty strongly knit,
|
||||
To thee I send this written embassage,
|
||||
To witness duty, not to show my wit:
|
||||
Duty so great, which wit so poor as mine
|
||||
May make seem bare, in wanting words to show it,
|
||||
But that I hope some good conceit of thine
|
||||
In thy soul's thought, all naked, will bestow it;
|
||||
Till whatsoever star that guides my moving
|
||||
Points on me graciously with fair aspect
|
||||
And puts apparel on my tatter'd loving,
|
||||
To show me worthy of thy sweet respect:
|
||||
Then may I dare to boast how I do love thee;
|
||||
Till then not show my head where thou mayst prove me.
|
||||
Weary with toil, I haste me to my bed,
|
||||
The dear repose for limbs with travel tired;
|
||||
But then begins a journey in my head,
|
||||
To work my mind, when body's work's expired:
|
||||
For then my thoughts, from far where I abide,
|
||||
Intend a zealous pilgrimage to thee,
|
||||
And keep my drooping eyelids open wide,
|
||||
Looking on darkness which the blind do see
|
||||
Save that my soul's imaginary sight
|
||||
Presents thy shadow to my sightless view,
|
||||
Which, like a jewel hung in ghastly night,
|
||||
Makes black night beauteous and her old face new.
|
||||
Lo! thus, by day my limbs, by night my mind,
|
||||
For thee and for myself no quiet find.
|
||||
How can I then return in happy plight,
|
||||
That am debarr'd the benefit of rest?
|
||||
When day's oppression is not eased by night,
|
||||
But day by night, and night by day, oppress'd?
|
||||
And each, though enemies to either's reign,
|
||||
Do in consent shake hands to torture me;
|
||||
The one by toil, the other to complain
|
||||
How far I toil, still farther off from thee.
|
||||
I tell the day, to please them thou art bright
|
||||
And dost him grace when clouds do blot the heaven:
|
||||
So flatter I the swart-complexion'd night,
|
||||
When sparkling stars twire not thou gild'st the even.
|
||||
But day doth daily draw my sorrows longer
|
||||
And night doth nightly make grief's strength seem stronger.
|
||||
When, in disgrace with fortune and men's eyes,
|
||||
I all alone beweep my outcast state
|
||||
And trouble deal heaven with my bootless cries
|
||||
And look upon myself and curse my fate,
|
||||
Wishing me like to one more rich in hope,
|
||||
Featured like him, like him with friends possess'd,
|
||||
Desiring this man's art and that man's scope,
|
||||
With what I most enjoy contented least;
|
||||
Yet in these thoughts myself almost despising,
|
||||
Haply I think on thee, and then my state,
|
||||
Like to the lark at break of day arising
|
||||
From sullen earth, sings hymns at heaven's gate;
|
||||
For thy sweet love remember'd such wealth brings
|
||||
That then I scorn to change my state with kings.
|
||||
When to the sessions of sweet silent thought
|
||||
I summon up remembrance of things past,
|
||||
I sigh the lack of many a thing I sought,
|
||||
And with old woes new wail my dear time's waste:
|
||||
Then can I drown an eye, unused to flow,
|
||||
For precious friends hid in death's dateless night,
|
||||
And weep afresh love's long since cancell'd woe,
|
||||
And moan the expense of many a vanish'd sight:
|
||||
Then can I grieve at grievances foregone,
|
||||
And heavily from woe to woe tell o'er
|
||||
The sad account of fore-bemoaned moan,
|
||||
Which I new pay as if not paid before.
|
||||
But if the while I think on thee, dear friend,
|
||||
All losses are restored and sorrows end.
|
||||
Thy bosom is endeared with all hearts,
|
||||
Which I by lacking have supposed dead,
|
||||
And there reigns love and all love's loving parts,
|
||||
And all those friends which I thought buried.
|
||||
How many a holy and obsequious tear
|
||||
Hath dear religious love stol'n from mine eye
|
||||
As interest of the dead, which now appear
|
||||
But things removed that hidden in thee lie!
|
||||
Thou art the grave where buried love doth live,
|
||||
Hung with the trophies of my lovers gone,
|
||||
Who all their parts of me to thee did give;
|
||||
That due of many now is thine alone:
|
||||
Their images I loved I view in thee,
|
||||
And thou, all they, hast all the all of me.
|
||||
If thou survive my well-contented day,
|
||||
When that churl Death my bones with dust shall cover,
|
||||
And shalt by fortune once more re-survey
|
||||
These poor rude lines of thy deceased lover,
|
||||
Compare them with the bettering of the time,
|
||||
And though they be outstripp'd by every pen,
|
||||
Reserve them for my love, not for their rhyme,
|
||||
Exceeded by the height of happier men.
|
||||
O, then vouchsafe me but this loving thought:
|
||||
'Had my friend's Muse grown with this growing age,
|
||||
A dearer birth than this his love had brought,
|
||||
To march in ranks of better equipage:
|
||||
But since he died and poets better prove,
|
||||
Theirs for their style I'll read, his for his love.'
|
||||
Full many a glorious morning have I seen
|
||||
Flatter the mountain-tops with sovereign eye,
|
||||
Kissing with golden face the meadows green,
|
||||
Gilding pale streams with heavenly alchemy;
|
||||
Anon permit the basest clouds to ride
|
||||
With ugly rack on his celestial face,
|
||||
And from the forlorn world his visage hide,
|
||||
Stealing unseen to west with this disgrace:
|
||||
Even so my sun one early morn did shine
|
||||
With all triumphant splendor on my brow;
|
||||
But out, alack! he was but one hour mine;
|
||||
The region cloud hath mask'd him from me now.
|
||||
Yet him for this my love no whit disdaineth;
|
||||
Suns of the world may stain when heaven's sun staineth.
|
||||
Why didst thou promise such a beauteous day,
|
||||
And make me travel forth without my cloak,
|
||||
To let base clouds o'ertake me in my way,
|
||||
Hiding thy bravery in their rotten smoke?
|
||||
'Tis not enough that through the cloud thou break,
|
||||
To dry the rain on my storm-beaten face,
|
||||
For no man well of such a salve can speak
|
||||
That heals the wound and cures not the disgrace:
|
||||
Nor can thy shame give physic to my grief;
|
||||
Though thou repent, yet I have still the loss:
|
||||
The offender's sorrow lends but weak relief
|
||||
To him that bears the strong offence's cross.
|
||||
Ah! but those tears are pearl which thy love sheds,
|
||||
And they are rich and ransom all ill deeds.
|
||||
No more be grieved at that which thou hast done:
|
||||
Roses have thorns, and silver fountains mud;
|
||||
Clouds and eclipses stain both moon and sun,
|
||||
And loathsome canker lives in sweetest bud.
|
||||
All men make faults, and even I in this,
|
||||
Authorizing thy trespass with compare,
|
||||
Myself corrupting, salving thy amiss,
|
||||
Excusing thy sins more than thy sins are;
|
||||
For to thy sensual fault I bring in sense--
|
||||
Thy adverse party is thy advocate--
|
||||
And 'gainst myself a lawful plea commence:
|
||||
Such civil war is in my love and hate
|
||||
That I an accessary needs must be
|
||||
To that sweet thief which sourly robs from me.
|
||||
Let me confess that we two must be twain,
|
||||
Although our undivided loves are one:
|
||||
So shall those blots that do with me remain
|
||||
Without thy help by me be borne alone.
|
||||
In our two loves there is but one respect,
|
||||
Though in our lives a separable spite,
|
||||
Which though it alter not love's sole effect,
|
||||
Yet doth it steal sweet hours from love's delight.
|
||||
I may not evermore acknowledge thee,
|
||||
Lest my bewailed guilt should do thee shame,
|
||||
Nor thou with public kindness honour me,
|
||||
Unless thou take that honour from thy name:
|
||||
But do not so; I love thee in such sort
|
||||
As, thou being mine, mine is thy good report.
|
||||
As a decrepit father takes delight
|
||||
To see his active child do deeds of youth,
|
||||
So I, made lame by fortune's dearest spite,
|
||||
Take all my comfort of thy worth and truth.
|
||||
For whether beauty, birth, or wealth, or wit,
|
||||
Or any of these all, or all, or more,
|
||||
Entitled in thy parts do crowned sit,
|
||||
I make my love engrafted to this store:
|
||||
So then I am not lame, poor, nor despised,
|
||||
Whilst that this shadow doth such substance give
|
||||
That I in thy abundance am sufficed
|
||||
And by a part of all thy glory live.
|
||||
Look, what is best, that best I wish in thee:
|
||||
This wish I have; then ten times happy me!
|
||||
@@ -33,7 +33,8 @@ else()
|
||||
# port, the SM90 paged-MQA kv_block=32/next_n=4 port, configurable SwiGLU
|
||||
# alpha/beta, and SiTU for FP8/FP4 Mega MoE. Also includes the CUDA 12.x
|
||||
# layout header fix from vllm-project/DeepGEMM#12.
|
||||
set(_DEEPGEMM_UPSTREAM_TAG "e1f418c2a4f20818221f6b0e578b4c2f634d4c3f")
|
||||
# TORCH_LIBRARY build with a CPython 3.10 stable-ABI floor.
|
||||
set(_DEEPGEMM_UPSTREAM_TAG "1e1842a833699298f7afc02eefb2ed168fff6938")
|
||||
|
||||
set(_deepgemm_fc_root "${FETCHCONTENT_BASE_DIR}")
|
||||
if(NOT _deepgemm_fc_root)
|
||||
@@ -43,7 +44,7 @@ else()
|
||||
set(_deepgemm_bin "${_deepgemm_fc_root}/deepgemm-build")
|
||||
set(_deepgemm_sub "${_deepgemm_fc_root}/deepgemm-subbuild")
|
||||
|
||||
if(EXISTS "${_deepgemm_src}/csrc/python_api.cpp")
|
||||
if(EXISTS "${_deepgemm_src}/deep_gemm/_C.py")
|
||||
set(deepgemm_SOURCE_DIR "${_deepgemm_src}")
|
||||
set(deepgemm_BINARY_DIR "${_deepgemm_bin}")
|
||||
else()
|
||||
@@ -93,33 +94,17 @@ if(DEEPGEMM_ARCHS)
|
||||
#
|
||||
# DeepGEMM integration notes
|
||||
# --------------------------
|
||||
# We vendor DeepGEMM into vllm/third_party/deep_gemm/ and bundle a
|
||||
# `_C.cpython-X.Y-*.so` for every CPython in `requires-python`. The
|
||||
# per-Python build is delegated to tools/build_deepgemm_C.py.
|
||||
# We vendor DeepGEMM into vllm/third_party/deep_gemm/ and bundle:
|
||||
# - deep_gemm/_C.py (Python shim over torch.ops.deep_gemm)
|
||||
# - deep_gemm/_C_extension.abi3.so (single limited-API extension)
|
||||
# The build is delegated to tools/build_deepgemm_C.py (setup.py build_ext).
|
||||
#
|
||||
# Why per-Python: DeepGEMM's binding uses PYBIND11_MODULE, which links
|
||||
# private CPython symbols — a single `_C.abi3.so` is not viable today
|
||||
# (see #41476 / #41512 for the failed attempt).
|
||||
#
|
||||
# TODOs (tracked in vllm-project/vllm#42431):
|
||||
# - Replace DeepGEMM's pybind11 binding with a TORCH_LIBRARY + shim
|
||||
# binding (cf. vllm-flash-attention/csrc/common/pytorch_shim.h) to
|
||||
# collapse to one `_C.abi3.so`. Needs either an upstream change or
|
||||
# a maintained binding fork in vLLM.
|
||||
# - AOT-compile DeepGEMM's CUDA kernels instead of runtime JIT to drop
|
||||
# the vendored CUTLASS/CCCL headers and the CUDA-toolkit-at-runtime
|
||||
# requirement.
|
||||
# TODO: AOT-compile DeepGEMM's CUDA kernels instead of runtime JIT to drop
|
||||
# the vendored CUTLASS/CCCL headers and the CUDA-toolkit-at-runtime
|
||||
# requirement.
|
||||
#
|
||||
|
||||
# DEEPGEMM_PYTHON_INTERPRETERS: ":"-separated target Python paths.
|
||||
# Empty/unset → fall back to the build interpreter (editable installs).
|
||||
# (Empty-but-set env vars test as DEFINED in cmake — treat as unset.)
|
||||
if(NOT "$ENV{DEEPGEMM_PYTHON_INTERPRETERS}" STREQUAL "")
|
||||
string(REPLACE ":" ";" _dg_pythons "$ENV{DEEPGEMM_PYTHON_INTERPRETERS}")
|
||||
else()
|
||||
set(_dg_pythons "${Python_EXECUTABLE}")
|
||||
endif()
|
||||
message(STATUS "DeepGEMM _C will be built for: ${_dg_pythons}")
|
||||
message(STATUS "DeepGEMM extension will be built with: ${Python_EXECUTABLE}")
|
||||
|
||||
# add_custom_command does no implicit header scanning; glob explicitly so
|
||||
# header-only edits in DeepGEMM/cutlass/deep_jit re-trigger the rebuild.
|
||||
@@ -132,40 +117,29 @@ if(DEEPGEMM_ARCHS)
|
||||
"${deepgemm_SOURCE_DIR}/third-party/deep_jit/include/*.h"
|
||||
"${deepgemm_SOURCE_DIR}/third-party/deep_jit/include/*.hpp")
|
||||
|
||||
set(_dg_markers)
|
||||
set(_dg_seen_soabis)
|
||||
foreach(_pybin IN LISTS _dg_pythons)
|
||||
execute_process(
|
||||
COMMAND "${_pybin}" -c
|
||||
"import sysconfig; print(sysconfig.get_config_var('SOABI'))"
|
||||
OUTPUT_VARIABLE _dg_soabi
|
||||
OUTPUT_STRIP_TRAILING_WHITESPACE
|
||||
COMMAND_ERROR_IS_FATAL ANY)
|
||||
# Dedup interpreters that resolve to the same CPython.
|
||||
if(_dg_soabi IN_LIST _dg_seen_soabis)
|
||||
continue()
|
||||
endif()
|
||||
list(APPEND _dg_seen_soabis "${_dg_soabi}")
|
||||
set(_dg_dir "${CMAKE_CURRENT_BINARY_DIR}/deepgemm_C_${_dg_soabi}")
|
||||
set(_dg_marker "${_dg_dir}/.built")
|
||||
add_custom_command(
|
||||
OUTPUT "${_dg_marker}"
|
||||
COMMAND "${Python_EXECUTABLE}"
|
||||
"${CMAKE_SOURCE_DIR}/tools/build_deepgemm_C.py"
|
||||
"${deepgemm_SOURCE_DIR}" "${_dg_dir}" "${_pybin}"
|
||||
COMMAND "${CMAKE_COMMAND}" -E touch "${_dg_marker}"
|
||||
DEPENDS "${CMAKE_SOURCE_DIR}/tools/build_deepgemm_C.py"
|
||||
"${deepgemm_SOURCE_DIR}/csrc/python_api.cpp"
|
||||
${_dg_headers}
|
||||
COMMENT "Building DeepGEMM _C for ${_pybin}"
|
||||
VERBATIM)
|
||||
list(APPEND _dg_markers "${_dg_marker}")
|
||||
install(DIRECTORY "${_dg_dir}/"
|
||||
DESTINATION vllm/third_party/deep_gemm
|
||||
COMPONENT _deep_gemm_C
|
||||
FILES_MATCHING PATTERN "_C.cpython-*.so")
|
||||
endforeach()
|
||||
add_custom_target(_deep_gemm_C ALL DEPENDS ${_dg_markers})
|
||||
set(_dg_dir "${CMAKE_CURRENT_BINARY_DIR}/deepgemm_C")
|
||||
set(_dg_marker "${_dg_dir}/.built")
|
||||
add_custom_command(
|
||||
OUTPUT "${_dg_marker}"
|
||||
COMMAND "${Python_EXECUTABLE}"
|
||||
"${CMAKE_SOURCE_DIR}/tools/build_deepgemm_C.py"
|
||||
"${deepgemm_SOURCE_DIR}" "${_dg_dir}"
|
||||
COMMAND "${CMAKE_COMMAND}" -E touch "${_dg_marker}"
|
||||
DEPENDS "${CMAKE_SOURCE_DIR}/tools/build_deepgemm_C.py"
|
||||
"${deepgemm_SOURCE_DIR}/csrc/python_api.cpp"
|
||||
"${deepgemm_SOURCE_DIR}/deep_gemm/_C.py"
|
||||
"${deepgemm_SOURCE_DIR}/setup.py"
|
||||
${_dg_headers}
|
||||
COMMENT "Building DeepGEMM _C_extension (abi3)"
|
||||
VERBATIM)
|
||||
add_custom_target(_deep_gemm_C ALL DEPENDS "${_dg_marker}")
|
||||
|
||||
install(DIRECTORY "${_dg_dir}/"
|
||||
DESTINATION vllm/third_party/deep_gemm
|
||||
COMPONENT _deep_gemm_C
|
||||
FILES_MATCHING
|
||||
PATTERN "_C.py"
|
||||
PATTERN "_C_extension*.so")
|
||||
|
||||
#
|
||||
# Vendor DeepGEMM Python package files
|
||||
|
||||
@@ -78,8 +78,9 @@ torch::Tensor get_scheduler_metadata(
|
||||
input.causal = causal;
|
||||
input.isa = isa;
|
||||
input.enable_kv_split = enable_kv_split;
|
||||
input.dynamic_causal =
|
||||
dynamic_causal.has_value() ? dynamic_causal->data_ptr<bool>() : nullptr;
|
||||
input.dynamic_causal = dynamic_causal.has_value()
|
||||
? dynamic_causal->data_ptr<int32_t>()
|
||||
: nullptr;
|
||||
|
||||
const int64_t kv_cache_idx =
|
||||
static_cast<int64_t>(parse_fp8_kv_dtype(kv_cache_dtype));
|
||||
@@ -256,8 +257,9 @@ void cpu_attention_with_kv_cache(
|
||||
input.s_aux = nullptr;
|
||||
input.s_aux_is_bf16 = false;
|
||||
}
|
||||
input.dynamic_causal =
|
||||
dynamic_causal.has_value() ? dynamic_causal->data_ptr<bool>() : nullptr;
|
||||
input.dynamic_causal = dynamic_causal.has_value()
|
||||
? dynamic_causal->data_ptr<int32_t>()
|
||||
: nullptr;
|
||||
input.scale = scale;
|
||||
input.causal = causal;
|
||||
input.sliding_window_size = sliding_window;
|
||||
|
||||
@@ -397,7 +397,7 @@ class AttentionScheduler {
|
||||
int32_t max_num_q_per_iter; // max Q head num can be hold in registers
|
||||
int32_t kv_block_alignment; // context length alignment requirement
|
||||
bool enable_kv_split;
|
||||
bool* dynamic_causal;
|
||||
int32_t* dynamic_causal;
|
||||
};
|
||||
|
||||
static constexpr int32_t MaxQTileIterNum = 128;
|
||||
@@ -450,7 +450,7 @@ class AttentionScheduler {
|
||||
const int32_t q_token_num =
|
||||
input.query_start_loc[req_id + 1] - input.query_start_loc[req_id];
|
||||
const bool req_causal =
|
||||
is_dynamic_causal ? input.dynamic_causal[req_id] : causal;
|
||||
is_dynamic_causal ? input.dynamic_causal[req_id] != 0 : causal;
|
||||
const int32_t q_start_pos = seq_len - q_token_num;
|
||||
const int32_t kv_start_pos = 0;
|
||||
const int32_t kv_end_pos = seq_len;
|
||||
@@ -492,7 +492,7 @@ class AttentionScheduler {
|
||||
const int32_t q_token_num =
|
||||
input.query_start_loc[req_id + 1] - input.query_start_loc[req_id];
|
||||
const bool req_causal =
|
||||
is_dynamic_causal ? input.dynamic_causal[req_id] : causal;
|
||||
is_dynamic_causal ? input.dynamic_causal[req_id] != 0 : causal;
|
||||
const int32_t q_start_pos = seq_len - q_token_num;
|
||||
const int32_t kv_start_pos = 0;
|
||||
const int32_t kv_end_pos = seq_len;
|
||||
@@ -884,7 +884,7 @@ struct AttentionInput {
|
||||
// provided as fp32 and executed in full float precision.
|
||||
const void* s_aux;
|
||||
bool s_aux_is_bf16;
|
||||
bool* dynamic_causal;
|
||||
int32_t* dynamic_causal;
|
||||
float scale;
|
||||
bool causal;
|
||||
int32_t sliding_window_size;
|
||||
@@ -1571,7 +1571,7 @@ class AttentionMainLoop {
|
||||
const float* alibi_slopes = input->alibi_slopes;
|
||||
const void* s_aux = input->s_aux;
|
||||
const bool s_aux_is_bf16 = input->s_aux_is_bf16;
|
||||
const bool* dynamic_causal = input->dynamic_causal;
|
||||
const int32_t* dynamic_causal = input->dynamic_causal;
|
||||
const bool is_dynamic_causal = dynamic_causal != nullptr;
|
||||
|
||||
const bool causal = input->causal;
|
||||
@@ -1658,7 +1658,8 @@ class AttentionMainLoop {
|
||||
|
||||
const int32_t current_group_idx = current_workitem_group->req_id;
|
||||
const int32_t current_group_causal =
|
||||
is_dynamic_causal ? dynamic_causal[current_group_idx] : causal;
|
||||
is_dynamic_causal ? dynamic_causal[current_group_idx] != 0
|
||||
: causal;
|
||||
auto [sliding_window_left, sliding_window_right] =
|
||||
AttentionScheduler::calcu_sliding_window_size(
|
||||
sliding_window_size, current_group_causal);
|
||||
|
||||
@@ -1,82 +1,73 @@
|
||||
#include "cpu_types.hpp"
|
||||
|
||||
#include <torch/library.h>
|
||||
#include <torch/all.h>
|
||||
|
||||
#include <cmath>
|
||||
#include <cstdint>
|
||||
#include <limits>
|
||||
#include <mutex>
|
||||
#include <random>
|
||||
|
||||
namespace {
|
||||
|
||||
constexpr int GUMBEL_TABLE_SIZE = 1 << 20;
|
||||
constexpr int GUMBEL_TABLE_MASK = GUMBEL_TABLE_SIZE - 1;
|
||||
// Gumbel-max: argmax_i(logit_i + g_i) with g_i ~ Gumbel(0, 1) matches
|
||||
// softmax. Noise is SplitMix64(seed, i), so tokens are independent and
|
||||
// consecutive seeds are not a 1-token table shift (vllm#59786).
|
||||
constexpr uint64_t SPLITMIX_GAMMA = 0x9E3779B97F4A7C15ULL;
|
||||
constexpr double GUMBEL_MAX = 37.5; // 53-bit u in (0,1) => -log(-log u) < 37.5
|
||||
|
||||
static float g_gumbel_table[GUMBEL_TABLE_SIZE];
|
||||
static std::once_flag g_gumbel_init_flag;
|
||||
static inline uint64_t splitmix64(uint64_t z) {
|
||||
z = (z ^ (z >> 30)) * 0xBF58476D1CE4E5B9ULL;
|
||||
z = (z ^ (z >> 27)) * 0x94D049BB133111EBULL;
|
||||
return z ^ (z >> 31);
|
||||
}
|
||||
|
||||
static void init_gumbel_table() {
|
||||
std::mt19937 rng(42);
|
||||
std::uniform_real_distribution<float> uniform(1.0e-7f, 1.0f - 1.0e-7f);
|
||||
for (int i = 0; i < GUMBEL_TABLE_SIZE; ++i) {
|
||||
float u = uniform(rng);
|
||||
g_gumbel_table[i] = -std::log(-std::log(u));
|
||||
static void fused_gumbel_argmax_row(int64_t* __restrict__ output,
|
||||
const float* __restrict__ row,
|
||||
uint64_t seed, int64_t vocab_size) {
|
||||
const uint64_t key = splitmix64(seed);
|
||||
|
||||
// Compare in (logit, noise) space: x + g > best_x + best_g is
|
||||
// (x - best_x) + g > best_g. Adding g into a 1e38 logit loses it in
|
||||
// double ULP, so ties at the float max would always keep the lower index.
|
||||
double best_logit = 0.0;
|
||||
double best_g = 0.0;
|
||||
int64_t best_idx = 0;
|
||||
bool have_best = false;
|
||||
for (int64_t i = 0; i < vocab_size; ++i) {
|
||||
const double x = row[i];
|
||||
if (x == -std::numeric_limits<double>::infinity()) continue;
|
||||
if (have_best && (x - best_logit) + GUMBEL_MAX <= best_g) continue;
|
||||
const uint64_t bits =
|
||||
splitmix64(key + uint64_t(i + 1) * SPLITMIX_GAMMA) >> 11;
|
||||
const double one_minus_u =
|
||||
(double((1ULL << 53) - 1 - bits) + 0.5) * 0x1.0p-53;
|
||||
if (have_best && one_minus_u >= std::exp((x - best_logit) - best_g))
|
||||
continue;
|
||||
const double u = (double(bits) + 0.5) * 0x1.0p-53;
|
||||
const double g = -std::log(-std::log(u));
|
||||
if (!have_best || (x - best_logit) + g > best_g) {
|
||||
have_best = true;
|
||||
best_logit = x;
|
||||
best_g = g;
|
||||
best_idx = i;
|
||||
}
|
||||
}
|
||||
*output = best_idx;
|
||||
}
|
||||
|
||||
static inline void ensure_gumbel_table() {
|
||||
std::call_once(g_gumbel_init_flag, init_gumbel_table);
|
||||
}
|
||||
|
||||
// Fused Gumbel-max kernel
|
||||
|
||||
static void fused_gumbel_argmax_kernel(int64_t* __restrict__ output,
|
||||
const float* __restrict__ logits,
|
||||
const int64_t* __restrict__ seeds,
|
||||
const int64_t batch_size,
|
||||
const int64_t vocab_size) {
|
||||
ensure_gumbel_table();
|
||||
constexpr int VEC_ELEM_NUM = vec_op::FP32Vec8::VEC_ELEM_NUM;
|
||||
const int64_t vec_end = vocab_size - (vocab_size % VEC_ELEM_NUM);
|
||||
|
||||
if (batch_size <= 0) return;
|
||||
if (batch_size == 1) {
|
||||
fused_gumbel_argmax_row(output, logits, uint64_t(seeds[0]), vocab_size);
|
||||
return;
|
||||
}
|
||||
#pragma omp parallel for schedule(static)
|
||||
for (int64_t b = 0; b < batch_size; ++b) {
|
||||
const float* row = logits + b * vocab_size;
|
||||
const uint32_t seed = static_cast<uint32_t>(seeds[b]);
|
||||
|
||||
float best_score = -std::numeric_limits<float>::infinity();
|
||||
int64_t best_idx = 0;
|
||||
|
||||
for (int64_t i = 0; i < vec_end; i += VEC_ELEM_NUM) {
|
||||
vec_op::FP32Vec8 logit_vec(row + i);
|
||||
|
||||
float gumbel_buf[VEC_ELEM_NUM];
|
||||
for (int k = 0; k < VEC_ELEM_NUM; ++k)
|
||||
gumbel_buf[k] = g_gumbel_table[(seed + i + k) & GUMBEL_TABLE_MASK];
|
||||
vec_op::FP32Vec8 gumbel_vec(gumbel_buf);
|
||||
|
||||
vec_op::FP32Vec8 score = logit_vec + gumbel_vec;
|
||||
|
||||
float score_buf[VEC_ELEM_NUM];
|
||||
score.save(score_buf);
|
||||
for (int k = 0; k < VEC_ELEM_NUM; ++k) {
|
||||
if (score_buf[k] > best_score) {
|
||||
best_score = score_buf[k];
|
||||
best_idx = i + k;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
for (int64_t i = vec_end; i < vocab_size; ++i) {
|
||||
float s = row[i] + g_gumbel_table[(seed + i) & GUMBEL_TABLE_MASK];
|
||||
if (s > best_score) {
|
||||
best_score = s;
|
||||
best_idx = i;
|
||||
}
|
||||
}
|
||||
|
||||
output[b] = best_idx;
|
||||
fused_gumbel_argmax_row(output + b, logits + b * vocab_size,
|
||||
uint64_t(seeds[b]), vocab_size);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -91,22 +82,41 @@ static void greedy_argmax_kernel(int64_t* __restrict__ output,
|
||||
for (int64_t b = 0; b < batch_size; ++b) {
|
||||
const float* row = logits + b * vocab_size;
|
||||
|
||||
// Vector max has no consistent NaN rule, so NaNs are caught via the sum:
|
||||
// any NaN (or inf + -inf, a harmless false positive) makes it NaN.
|
||||
vec_op::FP32Vec16 vmax(-std::numeric_limits<float>::infinity());
|
||||
vec_op::FP32Vec16 vsum(0.0f);
|
||||
for (int64_t i = 0; i < vec_end; i += VEC_ELEM_NUM) {
|
||||
vmax = vmax.max(vec_op::FP32Vec16(row + i));
|
||||
vec_op::FP32Vec16 v(row + i);
|
||||
vmax = vmax.max(v);
|
||||
vsum = vsum + v;
|
||||
}
|
||||
float best_val = vmax.reduce_max();
|
||||
bool maybe_nan = std::isnan(vsum.reduce_sum());
|
||||
for (int64_t i = vec_end; i < vocab_size; ++i) {
|
||||
maybe_nan |= std::isnan(row[i]);
|
||||
if (row[i] > best_val) {
|
||||
best_val = row[i];
|
||||
}
|
||||
}
|
||||
|
||||
int64_t best_idx = 0;
|
||||
for (int64_t i = 0; i < vocab_size; ++i) {
|
||||
if (row[i] == best_val) {
|
||||
best_idx = i;
|
||||
break;
|
||||
int64_t best_idx = -1;
|
||||
if (maybe_nan) {
|
||||
// torch.argmax treats NaN as the maximum: return the first NaN.
|
||||
for (int64_t i = 0; i < vocab_size; ++i) {
|
||||
if (std::isnan(row[i])) {
|
||||
best_idx = i;
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
if (best_idx < 0) {
|
||||
best_idx = 0;
|
||||
for (int64_t i = 0; i < vocab_size; ++i) {
|
||||
if (row[i] == best_val) {
|
||||
best_idx = i;
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
output[b] = best_idx;
|
||||
@@ -117,21 +127,28 @@ static void greedy_argmax_kernel(int64_t* __restrict__ output,
|
||||
|
||||
torch::Tensor fused_gumbel_argmax(const torch::Tensor& logits,
|
||||
const torch::Tensor& seeds) {
|
||||
TORCH_CHECK(logits.device().is_cpu(), "logits must be a CPU tensor");
|
||||
TORCH_CHECK(logits.dim() == 2, "logits must be 2-D [batch, vocab]");
|
||||
TORCH_CHECK(logits.scalar_type() == torch::kFloat32,
|
||||
"logits must be float32");
|
||||
TORCH_CHECK(logits.size(1) > 0, "vocab_size must be positive");
|
||||
TORCH_CHECK(seeds.device().is_cpu(), "seeds must be a CPU tensor");
|
||||
TORCH_CHECK(seeds.scalar_type() == torch::kInt64, "seeds must be int64");
|
||||
TORCH_CHECK(seeds.dim() == 1 && seeds.size(0) == logits.size(0),
|
||||
"seeds must be 1-D with batch_size elements");
|
||||
|
||||
auto logits_contig = logits.contiguous();
|
||||
auto seeds_contig = seeds.contiguous();
|
||||
auto output = torch::empty({logits_contig.size(0)}, torch::kInt64);
|
||||
fused_gumbel_argmax_kernel(
|
||||
output.data_ptr<int64_t>(), logits_contig.data_ptr<float>(),
|
||||
seeds.data_ptr<int64_t>(), logits_contig.size(0), logits_contig.size(1));
|
||||
fused_gumbel_argmax_kernel(output.data_ptr<int64_t>(),
|
||||
logits_contig.data_ptr<float>(),
|
||||
seeds_contig.data_ptr<int64_t>(),
|
||||
logits_contig.size(0), logits_contig.size(1));
|
||||
return output;
|
||||
}
|
||||
|
||||
torch::Tensor greedy_argmax(const torch::Tensor& logits) {
|
||||
TORCH_CHECK(logits.device().is_cpu(), "logits must be a CPU tensor");
|
||||
TORCH_CHECK(logits.dim() == 2, "logits must be 2-D [batch, vocab]");
|
||||
TORCH_CHECK(logits.scalar_type() == torch::kFloat32,
|
||||
"logits must be float32");
|
||||
|
||||
+22
-31
@@ -203,38 +203,29 @@ void create_and_map(unsigned long long device, ssize_t size, CUdeviceptr d_mem,
|
||||
}
|
||||
#endif
|
||||
|
||||
CUmemAccessDesc accessDesc = {};
|
||||
accessDesc.location.type = CU_MEM_LOCATION_TYPE_DEVICE;
|
||||
accessDesc.location.id = device;
|
||||
accessDesc.flags = CU_MEM_ACCESS_FLAGS_PROT_READWRITE;
|
||||
|
||||
CUDA_CHECK(cuMemSetAccess(d_mem, size, &accessDesc, 1));
|
||||
if (error_code != 0) {
|
||||
return;
|
||||
}
|
||||
|
||||
#ifdef USE_ROCM
|
||||
// ROCm-only: zero the freshly (re)mapped region.
|
||||
//
|
||||
// On some amdgpu driver versions the physical VRAM backing a buffer is not
|
||||
// scrubbed on release, so the pages handed back here (e.g. on wake_up after a
|
||||
// sleep that discarded a tag) can still contain stale data from a previously
|
||||
// freed allocation -- possibly from another process. Callers of the device
|
||||
// allocator expect newly mapped memory to be well defined; in particular
|
||||
// hybrid Mamba / gated-delta-net (GDN) models keep a persisted conv/recurrent
|
||||
// state cache, and a new sequence that reads an unzeroed slot will propagate
|
||||
// garbage/NaN, collapsing generation to token id 0 (decoded as "!").
|
||||
//
|
||||
// This is scoped to the ROCm path on purpose: the CUDA path is unchanged.
|
||||
// Use CUDA_CHECK so a failure sets the global error_code; the Python caller
|
||||
// (python_create_and_map) only inspects error_code, so returning without
|
||||
// setting it would make a failed memset look successful and still hand back
|
||||
// unzeroed memory.
|
||||
CUDA_CHECK((CUresult)hipMemset(reinterpret_cast<void*>(d_mem), 0, size));
|
||||
if (error_code != 0) {
|
||||
return;
|
||||
}
|
||||
#if defined(USE_ROCM) && HIP_VERSION >= 70200000
|
||||
// Also map for the host, as PyTorch does for its expandable segments: on
|
||||
// ROCm >= 7.2, Tensor.item() on a large-BAR device dereferences VRAM
|
||||
// directly from the host, which segfaults on a device-only mapping.
|
||||
constexpr int num_desc = 2;
|
||||
#else
|
||||
constexpr int num_desc = 1;
|
||||
#endif
|
||||
CUmemAccessDesc accessDesc[num_desc] = {};
|
||||
accessDesc[0].location.type = CU_MEM_LOCATION_TYPE_DEVICE;
|
||||
accessDesc[0].location.id = device;
|
||||
accessDesc[0].flags = CU_MEM_ACCESS_FLAGS_PROT_READWRITE;
|
||||
#if defined(USE_ROCM) && HIP_VERSION >= 70200000
|
||||
accessDesc[1].location.type = hipMemLocationTypeHost;
|
||||
accessDesc[1].location.id = 0; // ignored
|
||||
accessDesc[1].flags = CU_MEM_ACCESS_FLAGS_PROT_READWRITE;
|
||||
#endif
|
||||
|
||||
CUDA_CHECK(cuMemSetAccess(d_mem, size, accessDesc, num_desc));
|
||||
if (error_code != 0) {
|
||||
return;
|
||||
}
|
||||
|
||||
// std::cout << "create_and_map: device=" << device << ", size=" << size << ",
|
||||
// d_mem=" << d_mem << ", p_memHandle=" << p_memHandle << std::endl;
|
||||
}
|
||||
|
||||
@@ -8,6 +8,7 @@
|
||||
|
||||
#include <stdexcept>
|
||||
#include <algorithm>
|
||||
#include <utility> // std::in_range
|
||||
|
||||
#include "../cuda_compat.h"
|
||||
#include "dispatch_utils.h"
|
||||
@@ -121,6 +122,7 @@ __device__ __forceinline__ uint32_t zp_nibble(const uint32_t* zp_packed,
|
||||
// GROUP_SIZE: 0 = per-channel scale [M], >0 = per-group scale [M,
|
||||
// K/GROUP_SIZE].
|
||||
// Requires GROUP_SIZE % A_CHUNK == 0 when GROUP_SIZE > 0.
|
||||
// B_row_stride_bytes is the per-row byte stride of the packed weights.
|
||||
#if defined(__HIP__GFX1X__)
|
||||
template <typename scalar_t, int THRDS, int YTILE, int WvPrGrp, int A_CHUNK,
|
||||
int UNRL, int N, int GROUP_SIZE = 0, bool HAS_ZERO_POINTS = false>
|
||||
@@ -130,9 +132,9 @@ __global__ void __launch_bounds__(WvPrGrp* THRDS)
|
||||
const scalar_t* __restrict__ A, const scalar_t* scale,
|
||||
const uint32_t* zero_points,
|
||||
const scalar_t* __restrict__ BIAS, scalar_t* C,
|
||||
const int _WvPrGrp, const int CuCount) {
|
||||
const int _WvPrGrp, const int CuCount,
|
||||
const int B_row_stride_bytes) {
|
||||
constexpr int max_lds_len = LDS_SIZE / 2;
|
||||
const int K_packed = K / 2;
|
||||
|
||||
union bigTypeA {
|
||||
scalar_t h[A_CHUNK];
|
||||
@@ -181,9 +183,9 @@ __global__ void __launch_bounds__(WvPrGrp* THRDS)
|
||||
uint32_t k_ = k + threadIdx.x * A_CHUNK;
|
||||
if (k_ >= K) break;
|
||||
|
||||
const uint8_t* B_ = &B_packed[(m + 0) * K_packed + k_ / 2];
|
||||
const uint8_t* B_ = &B_packed[(m + 0) * B_row_stride_bytes + k_ / 2];
|
||||
for (int y = 0; y < YTILE; y++) {
|
||||
const float* src = (const float*)(&B_[y * K_packed]);
|
||||
const float* src = (const float*)(&B_[y * B_row_stride_bytes]);
|
||||
#pragma unroll
|
||||
for (int i = 0; i < A_CHUNK / 8; i++)
|
||||
bigB[y][k2].f[i] = loadnt((float*)&src[i]);
|
||||
@@ -344,14 +346,12 @@ __global__ void __launch_bounds__(WvPrGrp* THRDS)
|
||||
#else // !defined(__HIP__GFX1X__)
|
||||
template <typename scalar_t, int THRDS, int YTILE, int WvPrGrp, int A_CHUNK,
|
||||
int UNRL, int N, int GROUP_SIZE = 0, bool HAS_ZERO_POINTS = false>
|
||||
__global__ void wvSplitK_int4_hf_sml_(const int K, const int M, const int Bx,
|
||||
const int By, const uint8_t* B_packed,
|
||||
const scalar_t* __restrict__ A,
|
||||
const scalar_t* scale,
|
||||
const uint32_t* zero_points,
|
||||
const scalar_t* __restrict__ BIAS,
|
||||
scalar_t* C, const int _WvPrGrp,
|
||||
const int CuCount) {
|
||||
__global__ void wvSplitK_int4_hf_sml_(
|
||||
const int K, const int M, const int Bx, const int By,
|
||||
const uint8_t* B_packed, const scalar_t* __restrict__ A,
|
||||
const scalar_t* scale, const uint32_t* zero_points,
|
||||
const scalar_t* __restrict__ BIAS, scalar_t* C, const int _WvPrGrp,
|
||||
const int CuCount, const int B_row_stride_bytes) {
|
||||
UNREACHABLE_CODE
|
||||
}
|
||||
#endif // defined(__HIP__GFX1X__)
|
||||
@@ -367,9 +367,9 @@ __global__ void __launch_bounds__(WvPrGrp* THRDS)
|
||||
const uint8_t* B_packed, const scalar_t* __restrict__ A,
|
||||
const scalar_t* scale, const uint32_t* zero_points,
|
||||
const scalar_t* __restrict__ BIAS, scalar_t* C,
|
||||
const int _WvPrGrp, const int CuCount) {
|
||||
const int _WvPrGrp, const int CuCount,
|
||||
const int B_row_stride_bytes) {
|
||||
constexpr int max_lds_len = LDS_SIZE / 2;
|
||||
const int K_packed = K / 2;
|
||||
|
||||
union bigTypeA {
|
||||
scalar_t h[A_CHUNK];
|
||||
@@ -430,9 +430,9 @@ __global__ void __launch_bounds__(WvPrGrp* THRDS)
|
||||
uint32_t k_ = k + threadIdx.x * A_CHUNK;
|
||||
if (k_ >= K) break;
|
||||
|
||||
const uint8_t* B_ = &B_packed[(m + 0) * K_packed + k_ / 2];
|
||||
const uint8_t* B_ = &B_packed[(m + 0) * B_row_stride_bytes + k_ / 2];
|
||||
for (int y = 0; y < YTILE; y++) {
|
||||
const float* src = (const float*)(&B_[y * K_packed]);
|
||||
const float* src = (const float*)(&B_[y * B_row_stride_bytes]);
|
||||
#pragma unroll
|
||||
for (int i = 0; i < A_CHUNK / 8; i++)
|
||||
bigB[y][k2].f[i] = loadnt((float*)&src[i]);
|
||||
@@ -606,14 +606,12 @@ __global__ void __launch_bounds__(WvPrGrp* THRDS)
|
||||
#else // !defined(__HIP__GFX1X__)
|
||||
template <typename scalar_t, int THRDS, int YTILE, int WvPrGrp, int A_CHUNK,
|
||||
int UNRL, int N, int GROUP_SIZE = 0, bool HAS_ZERO_POINTS = false>
|
||||
__global__ void wvSplitK_int4_hf_(const int K, const int M, const int Bx,
|
||||
const int By, const uint8_t* B_packed,
|
||||
const scalar_t* __restrict__ A,
|
||||
const scalar_t* scale,
|
||||
const uint32_t* zero_points,
|
||||
const scalar_t* __restrict__ BIAS,
|
||||
scalar_t* C, const int _WvPrGrp,
|
||||
const int CuCount) {
|
||||
__global__ void wvSplitK_int4_hf_(
|
||||
const int K, const int M, const int Bx, const int By,
|
||||
const uint8_t* B_packed, const scalar_t* __restrict__ A,
|
||||
const scalar_t* scale, const uint32_t* zero_points,
|
||||
const scalar_t* __restrict__ BIAS, scalar_t* C, const int _WvPrGrp,
|
||||
const int CuCount, const int B_row_stride_bytes) {
|
||||
UNREACHABLE_CODE
|
||||
}
|
||||
#endif // defined(__HIP__GFX1X__)
|
||||
@@ -658,10 +656,17 @@ torch::Tensor wvSplitK_int4_g(const at::Tensor& in_a, const at::Tensor& in_b,
|
||||
? in_bias->size(0)
|
||||
: 1;
|
||||
|
||||
int64_t expected_weight_bytes = M_in * K_in / 2;
|
||||
int64_t actual_weight_bytes = in_a.numel() * in_a.element_size();
|
||||
const int64_t expected_weight_bytes = M_in * K_in / 2;
|
||||
const int64_t actual_weight_bytes = in_a.numel() * in_a.element_size();
|
||||
TORCH_CHECK(actual_weight_bytes == expected_weight_bytes,
|
||||
"Weight tensor must contain M*K/2 bytes for int4 packing");
|
||||
const int64_t b_row_stride_bytes = in_a.stride(0) * in_a.element_size();
|
||||
TORCH_CHECK(b_row_stride_bytes >= K_in / 2, "Weight row stride (",
|
||||
b_row_stride_bytes, " bytes) must hold at least K/2=", K_in / 2,
|
||||
" bytes per row");
|
||||
TORCH_CHECK(std::in_range<int>(b_row_stride_bytes), "Weight row stride (",
|
||||
b_row_stride_bytes, " bytes) exceeds int range");
|
||||
const int b_row_stride_bytes_i32 = static_cast<int>(b_row_stride_bytes);
|
||||
TORCH_CHECK(
|
||||
in_b.dtype() == torch::kFloat16 || in_b.dtype() == torch::kBFloat16,
|
||||
"Activation must be float16 or bfloat16");
|
||||
@@ -722,12 +727,12 @@ torch::Tensor wvSplitK_int4_g(const at::Tensor& in_a, const at::Tensor& in_b,
|
||||
wvSplitK_int4_hf_sml_<fptype, _THRDS, _YTILE, 16, 16, _UNRL, _N, _GS, \
|
||||
_HAS_ZP><<<grid, block, 0, stream>>>( \
|
||||
K_in, M_in, Bx_in, By_in, wptr, aptr, sptr, zpptr, biasptr, cptr, \
|
||||
__wvPrGrp, CuCount); \
|
||||
__wvPrGrp, CuCount, b_row_stride_bytes_i32); \
|
||||
else \
|
||||
wvSplitK_int4_hf_<fptype, _THRDS, _YTILE, 16, 16, _UNRL, _N, _GS, \
|
||||
_HAS_ZP><<<grid, block, 0, stream>>>( \
|
||||
K_in, M_in, Bx_in, By_in, wptr, aptr, sptr, zpptr, biasptr, cptr, \
|
||||
__wvPrGrp, CuCount); \
|
||||
__wvPrGrp, CuCount, b_row_stride_bytes_i32); \
|
||||
}
|
||||
|
||||
#define WVSPLITK_INT4G(_YTILE, _UNRL, _N, _GS, _HAS_ZP) \
|
||||
|
||||
+43
-1
@@ -92,6 +92,11 @@ ARG UV_KEYRING_PROVIDER=${PIP_KEYRING_PROVIDER}
|
||||
# Flag enables built-in KV-connector dependency libs into docker images
|
||||
ARG INSTALL_KV_CONNECTORS=false
|
||||
|
||||
# Flag builds NIXL, including its NIXL EP extension for the installed PyTorch,
|
||||
# from source and installs it in place of the nixl packages from the KV-connector
|
||||
# requirements
|
||||
ARG BUILD_NIXL=false
|
||||
|
||||
#################### BASE BUILD IMAGE ####################
|
||||
# prepare basic build environment
|
||||
FROM ${BUILD_BASE_IMAGE} AS base
|
||||
@@ -148,7 +153,7 @@ RUN if [ "$USE_SCCACHE" = "1" ]; then \
|
||||
&& export SCCACHE_DOWNLOAD_URL="${SCCACHE_DOWNLOAD_URL:-https://github.com/mozilla/sccache/releases/download/v0.8.1/sccache-v0.8.1-${SCCACHE_ARCH}-unknown-linux-musl.tar.gz}" \
|
||||
&& curl -L -o sccache.tar.gz ${SCCACHE_DOWNLOAD_URL} \
|
||||
&& tar -xzf sccache.tar.gz \
|
||||
&& sudo mv sccache-v0.8.1-${SCCACHE_ARCH}-unknown-linux-musl/sccache /usr/bin/sccache \
|
||||
&& mv sccache-v0.8.1-${SCCACHE_ARCH}-unknown-linux-musl/sccache /usr/bin/sccache \
|
||||
&& rm -rf sccache.tar.gz sccache-v0.8.1-${SCCACHE_ARCH}-unknown-linux-musl; \
|
||||
fi
|
||||
|
||||
@@ -548,6 +553,24 @@ RUN --mount=type=cache,target=/opt/uv/cache \
|
||||
(cd "${TRITON_WORKSPACE}/dist" \
|
||||
&& sha256sum build-state > wheel.sha256); \
|
||||
fi
|
||||
|
||||
# Build NIXL and its NIXL EP extension against the PyTorch installed here when
|
||||
# requested. The wheels are installed by the final image, which has no compiler.
|
||||
ARG BUILD_NIXL
|
||||
COPY tools/build_nixl_with_torch215_for_rubin.py /tmp/build_nixl_for_rubin.py
|
||||
RUN --mount=type=bind,source=requirements,target=/tmp/vllm-requirements,ro \
|
||||
set -eux; \
|
||||
NIXL_WORKSPACE=/tmp/nixl_workspace; \
|
||||
mkdir -p "${NIXL_WORKSPACE}/dist"; \
|
||||
if [ "${BUILD_NIXL}" = "true" ]; then \
|
||||
dnf install -y --setopt=install_weak_deps=False autoconf automake libtool; \
|
||||
dnf clean all; \
|
||||
rm -rf /var/cache/dnf; \
|
||||
NIXL_REQUIREMENTS_FILE=/tmp/vllm-requirements/kv_connectors.txt \
|
||||
WHEELS_CACHE_HOME="${NIXL_WORKSPACE}/dist" \
|
||||
MAX_JOBS="${max_jobs:-}" \
|
||||
python3 /tmp/build_nixl_for_rubin.py --skip-system-deps --build-only; \
|
||||
fi
|
||||
#################### EXTENSIONS BUILD IMAGE ####################
|
||||
|
||||
#################### WHEEL BUILD IMAGE ####################
|
||||
@@ -1260,6 +1283,25 @@ RUN --mount=type=cache,target=/opt/uv/cache \
|
||||
/tmp/install_triton_from_source.sh \
|
||||
/tmp/triton-source-wheel.sha256
|
||||
|
||||
# Install source-built NIXL last, replacing the nixl packages installed above, so
|
||||
# the image has a NIXL EP build for its PyTorch version. There is no GPU driver
|
||||
# at build time, so check for the extension instead of importing it.
|
||||
ARG BUILD_NIXL
|
||||
RUN --mount=type=cache,target=/opt/uv/cache \
|
||||
--mount=type=bind,from=extensions-build,source=/tmp/nixl_workspace/dist,target=/tmp/nixl-dist,ro \
|
||||
set -eu; \
|
||||
if [ "${BUILD_NIXL}" = "true" ]; then \
|
||||
ls /tmp/nixl-dist/*.whl; \
|
||||
uv pip uninstall --system nixl nixl-cu12 nixl-cu13; \
|
||||
uv pip install --system --no-deps /tmp/nixl-dist/*.whl; \
|
||||
python3 -c "import glob, importlib.util as u, os, torch; \
|
||||
spec = u.find_spec('nixl_ep_cu' + os.environ['CUDA_VERSION'].split('.')[0]); \
|
||||
mm = ''.join(torch.__version__.split('.')[:2]); \
|
||||
ext = glob.glob(os.path.join(spec.submodule_search_locations[0], 'nixl_ep_cpp_torch' + mm + '.*')) if spec else []; \
|
||||
assert ext, 'NIXL EP has no build for torch ' + torch.__version__; \
|
||||
print('NIXL EP extension:', ext[0])"; \
|
||||
fi
|
||||
|
||||
ENV VLLM_USAGE_SOURCE production-docker-image
|
||||
ENV VLLM_BUILD_COMMIT=${VLLM_BUILD_COMMIT:-unknown} \
|
||||
VLLM_BUILD_PIPELINE=${VLLM_BUILD_PIPELINE:-local} \
|
||||
|
||||
@@ -13,7 +13,7 @@
|
||||
# vllm-dev: used for development
|
||||
#
|
||||
# Build arguments:
|
||||
# PYTHON_VERSION=3.13|3.12 (default)|3.11|3.10
|
||||
# PYTHON_VERSION=3.13|3.12 (default)|3.11
|
||||
# VLLM_CPU_X86=false (default)|true (for cross-compilation)
|
||||
# VLLM_CPU_ARM_BF16=false (default)|true (for cross-compilation)
|
||||
#
|
||||
@@ -441,6 +441,11 @@ ADD ./cmake/ ./cmake/
|
||||
# (e.g. tests/tools/test_check_test_tethering.py imports
|
||||
# tools.pre_commit.check_test_tethering).
|
||||
ADD ./tools/ ./tools/
|
||||
# tests/test_zentorch_version_pin.py string-matches the zentorch extra in
|
||||
# setup.py against the torch pin in requirements/cpu.txt. The wheel does not
|
||||
# ship either file, and this stage does not inherit the vllm-src tree.
|
||||
ADD ./setup.py ./setup.py
|
||||
ADD ./requirements/cpu.txt ./requirements/cpu.txt
|
||||
|
||||
# install development dependencies (for testing)
|
||||
RUN --mount=type=cache,target=/root/.cache/uv \
|
||||
|
||||
@@ -37,6 +37,9 @@
|
||||
"INSTALL_KV_CONNECTORS": {
|
||||
"default": "false"
|
||||
},
|
||||
"BUILD_NIXL": {
|
||||
"default": "false"
|
||||
},
|
||||
"SCCACHE_BUCKET_NAME": {
|
||||
"default": "vllm-build-sccache"
|
||||
},
|
||||
|
||||
@@ -23,7 +23,6 @@ th {
|
||||
| ShareGPT4V (Image) | ✅ | ✅ | `wget https://huggingface.co/datasets/Lin-Chen/ShareGPT4V/resolve/main/sharegpt4v_instruct_gpt4-vision_cap100k.json`<br>Note that the images need to be downloaded separately. For example, to download COCO's 2017 Train images:<br>`wget http://images.cocodataset.org/zips/train2017.zip` |
|
||||
| ShareGPT4Video (Video) | ✅ | ✅ | `git clone https://huggingface.co/datasets/ShareGPT4Video/ShareGPT4Video` |
|
||||
| BurstGPT | ✅ | ✅ | `wget https://github.com/HPMLL/BurstGPT/releases/download/v1.1/BurstGPT_without_fails_2.csv` |
|
||||
| Sonnet (deprecated) | ✅ | ✅ | Local file: `benchmarks/sonnet.txt` |
|
||||
| Random | ✅ | ✅ | `synthetic` |
|
||||
| RandomMultiModal (Image/Video) | ✅ | ✅ | `synthetic` |
|
||||
| RandomForReranking | ✅ | ✅ | `synthetic` |
|
||||
@@ -773,16 +772,17 @@ Using KV cache metrics for load pattern configuration:
|
||||
```bash
|
||||
vllm bench throughput \
|
||||
--model NousResearch/Hermes-3-Llama-3.1-8B \
|
||||
--dataset-name sonnet \
|
||||
--dataset-path vllm/benchmarks/sonnet.txt \
|
||||
--dataset-name random \
|
||||
--random-input-len 500 \
|
||||
--random-output-len 150 \
|
||||
--num-prompts 10
|
||||
```
|
||||
|
||||
If successful, you will see the following output
|
||||
|
||||
```text
|
||||
Throughput: 7.15 requests/s, 4656.00 total tokens/s, 1072.15 output tokens/s
|
||||
Total num prompt tokens: 5014
|
||||
Throughput: 7.15 requests/s, 4647.50 total tokens/s, 1072.50 output tokens/s
|
||||
Total num prompt tokens: 5000
|
||||
Total num output tokens: 1500
|
||||
```
|
||||
|
||||
|
||||
@@ -9,7 +9,7 @@ This page lists the active reviewers and describes the role. See the [committers
|
||||
Community members can reach out to the reviewers below for help with PRs in their areas, for example in the `#pr-reviews` Slack channel. Reviewers will add their areas of expertise to this list.
|
||||
Sorted alphabetically by GitHub handle:
|
||||
|
||||
- [@andylolu2](https://github.com/andylolu2)
|
||||
- [@andylolu2](https://github.com/andylolu2): Mistral models, speculative decoding, quantization, scheduler, MLA
|
||||
- [@bnellnm](https://github.com/bnellnm) MoE, Quantization layers
|
||||
- [@BowenBao](https://github.com/BowenBao)
|
||||
- [@cjackal](https://github.com/cjackal): Multimodality; KV Connector and offload
|
||||
@@ -30,7 +30,7 @@ Sorted alphabetically by GitHub handle:
|
||||
- [@Rohan138](https://github.com/Rohan138): ROCm performance, CI, torch.compile fusions
|
||||
- [@simondanielsson](https://github.com/simondanielsson): ROCm performance (CDNA), MoRI-IO.
|
||||
- [@taneem-ibrahim](https://github.com/taneem-ibrahim): Pooling models
|
||||
- [@TheEpicDolphin](https://github.com/TheEpicDolphin)
|
||||
- [@TheEpicDolphin](https://github.com/TheEpicDolphin): Spec decode
|
||||
- [@varun-sundar-rabindranath](https://github.com/varun-sundar-rabindranath): KV cache offloading, LoRA
|
||||
- [@vllmellm](https://github.com/vllmellm)
|
||||
- [@wangxiyuan](https://github.com/wangxiyuan): Platform, KV cache, Mooncake
|
||||
|
||||
@@ -80,7 +80,7 @@ llm = LLM(model="meta-llama/Llama-3.1-8B-Instruct", enforce_eager=True)
|
||||
If you run out of CPU RAM, try the following options:
|
||||
|
||||
- (Multi-modal models only) you can set the size of multi-modal cache by setting `mm_processor_cache_gb` engine argument (default 4 GiB).
|
||||
- (CPU backend only) you can set the size of KV cache using `VLLM_CPU_KVCACHE_SPACE` environment variable (default 4 GiB).
|
||||
- (CPU backend only) you can set the size of KV cache using `VLLM_CPU_KVCACHE_SPACE` environment variable, in GiB (default 0, meaning vLLM doesn't override the KV cache size through this variable).
|
||||
|
||||
## Multi-modal input limits
|
||||
|
||||
|
||||
@@ -74,7 +74,7 @@ For an optimized workflow when iterating on C++/CUDA kernels, see the [Increment
|
||||
For JIT kernel warmup conventions, see [JIT Kernel Warmup](./jit_kernel_warmup.md).
|
||||
|
||||
!!! tip
|
||||
vLLM is compatible with Python versions 3.10 to 3.13. However, vLLM's default [Dockerfile](../../docker/Dockerfile) ships with Python 3.12 and tests in CI (except `mypy`) are run with Python 3.12.
|
||||
vLLM is compatible with Python versions 3.11 to 3.14. However, vLLM's default [Dockerfile](../../docker/Dockerfile) ships with Python 3.12 and tests in CI (except `mypy`) are run with Python 3.12.
|
||||
|
||||
Therefore, we recommend developing with Python 3.12 to minimise the chance of your local environment clashing with our CI environment.
|
||||
|
||||
@@ -102,14 +102,6 @@ integration, use the configuration files in `tools/pre_commit/`:
|
||||
pre-commit run -a # runs on all files (short for --all-files)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
Some `pre-commit` hooks only run in CI. If you need to, you can run them locally with:
|
||||
|
||||
```bash
|
||||
pre-commit run --hook-stage manual mypy-3.11
|
||||
```
|
||||
|
||||
### Documentation
|
||||
|
||||
MkDocs is a fast, simple and downright gorgeous static site generator that's geared towards building project documentation. Documentation source files are written in Markdown, and configured with a single YAML configuration file, [mkdocs.yaml](../../mkdocs.yaml).
|
||||
|
||||
@@ -24,6 +24,11 @@ The `extensions-build` stage can also produce an optional source-built Triton
|
||||
wheel. `vllm-openai-base` installs that wheel after its other Python
|
||||
dependencies so dependency resolution cannot restore an older Triton version.
|
||||
|
||||
With `BUILD_NIXL=true`, `extensions-build` also builds NIXL, including its NIXL
|
||||
EP extension for the installed PyTorch, with `tools/build_nixl_with_torch215_for_rubin.py`.
|
||||
`vllm-openai-base` then replaces the NIXL packages installed from the KV-connector
|
||||
requirements with these wheels.
|
||||
|
||||
> <figure markdown="span">
|
||||
> { align="center" alt="query" width="100%" }
|
||||
> </figure>
|
||||
|
||||
@@ -31,6 +31,18 @@ When using `vllm bench serve`, you can enable profiling by passing the `--profil
|
||||
|
||||
Traces can be visualized using <https://ui.perfetto.dev/>.
|
||||
|
||||
Each worker writes its trace and summary table under `torch_profiler_dir`,
|
||||
named after its rank, for example `rank0.<id>.pt.trace.json.gz` and
|
||||
`rank0.profiler_out.txt`. Depending on the parallel configuration, the rank
|
||||
name can also include the parallel ranks, such as
|
||||
`dp0_pp0_tp0_dcp0_ep0_rank0`. The AsyncLLM CPU trace uses
|
||||
`<hostname>_<pid>.async_llm` instead of the rank name. When a session sets
|
||||
`profile_prefix`, it is prepended as `<profile_prefix>_`, so summaries from
|
||||
different sessions do not overwrite each other.
|
||||
|
||||
!!! note
|
||||
The summary table was previously written to `profiler_out_<rank>.txt`.
|
||||
|
||||
!!! tip
|
||||
You can directly call bench module without installing vLLM using `python -m vllm.entrypoints.cli.main bench`.
|
||||
|
||||
@@ -88,6 +100,31 @@ curl -X POST http://localhost:8000/v1/chat/completions \
|
||||
$ curl -X POST http://localhost:8000/stop_profile
|
||||
```
|
||||
|
||||
`/start_profile` also accepts an optional JSON body with settings for that
|
||||
profiling session. The Python and Rust API frontends accept the same fields:
|
||||
|
||||
```shell
|
||||
curl -X POST http://localhost:8000/start_profile \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"profile_prefix": "sharegpt_run_1",
|
||||
"delay_iterations": 5000,
|
||||
"max_iterations": 20
|
||||
}'
|
||||
```
|
||||
|
||||
`profile_prefix` is prepended to the generated trace and summary file names. It
|
||||
must be 1-128 ASCII characters, start with a letter or number, and contain only
|
||||
letters, numbers, `.`, `_`, or `-`. `delay_iterations` skips worker iterations
|
||||
before collection starts, and `max_iterations` limits the number of collected
|
||||
worker iterations (`0` means no limit). Both must be non-negative JSON integers;
|
||||
values of the wrong type, such as `"5"` or `5.0`, are rejected rather than
|
||||
converted. Unknown fields are ignored. These iteration
|
||||
bounds do not apply to the AsyncLLM frontend CPU profiler, which records from
|
||||
`/start_profile` until `/stop_profile`. Always call `/stop_profile`, including
|
||||
after the worker reaches `max_iterations`; starting another session before
|
||||
stopping the current one returns HTTP 409.
|
||||
|
||||
## Profile with Triton Proton
|
||||
|
||||
[Proton](https://github.com/triton-lang/triton/tree/main/third_party/proton)
|
||||
|
||||
@@ -37,7 +37,7 @@ See the vLLM SkyPilot YAML for serving, [serving.yaml](https://github.com/skypil
|
||||
HF_TOKEN: <your-huggingface-token> # Change to your own huggingface token, or use --env to pass.
|
||||
|
||||
setup: |
|
||||
conda create -n vllm python=3.10 -y
|
||||
conda create -n vllm python=3.12 -y
|
||||
conda activate vllm
|
||||
|
||||
pip install vllm==0.4.0.post1
|
||||
@@ -136,7 +136,7 @@ SkyPilot can scale up the service to multiple service replicas with built-in aut
|
||||
HF_TOKEN: <your-huggingface-token> # Change to your own huggingface token, or use --env to pass.
|
||||
|
||||
setup: |
|
||||
conda create -n vllm python=3.10 -y
|
||||
conda create -n vllm python=3.12 -y
|
||||
conda activate vllm
|
||||
|
||||
pip install vllm==0.4.0.post1
|
||||
@@ -248,7 +248,7 @@ This will scale the service up to when the QPS exceeds 2 for each replica.
|
||||
HF_TOKEN: <your-huggingface-token> # Change to your own huggingface token, or use --env to pass.
|
||||
|
||||
setup: |
|
||||
conda create -n vllm python=3.10 -y
|
||||
conda create -n vllm python=3.12 -y
|
||||
conda activate vllm
|
||||
|
||||
pip install vllm==0.4.0.post1
|
||||
@@ -293,7 +293,7 @@ It is also possible to access the Llama-3 service with a separate GUI frontend,
|
||||
cpus: 2
|
||||
|
||||
setup: |
|
||||
conda create -n vllm python=3.10 -y
|
||||
conda create -n vllm python=3.12 -y
|
||||
conda activate vllm
|
||||
|
||||
# Install Gradio for web UI.
|
||||
|
||||
@@ -225,9 +225,9 @@ outputs = model.generate(
|
||||
)
|
||||
```
|
||||
|
||||
### Piecewise compilation and full graph custom passes (attention fusion, sequence parallelism)
|
||||
### Piecewise compilation and full graph custom passes (attention fusion)
|
||||
|
||||
Unfortunately, some custom compile passes have to see the whole graph to be effective and hence aren't compatible with piecewise compilation. This includes `AttnQuantFusionPass` and `SequenceParallelismPass`. As a short-term solution, we automatically disable piecewise compilation (by setting `splitting_ops=[]`) when attention fusion is enabled. We use CUDA Graph modes `FULL` or `FULL_DECODE_ONLY` (depending on backend support). However, this leads to another optimization incompatibility and confusing performance tradeoffs.
|
||||
Unfortunately, some custom compile passes have to see the whole graph to be effective and hence aren't compatible with piecewise compilation. This includes `AttnQuantFusionPass`. As a short-term solution, we automatically disable piecewise compilation (by setting `splitting_ops=[]`) when attention fusion is enabled. We use CUDA Graph modes `FULL` or `FULL_DECODE_ONLY` (depending on backend support). However, this leads to another optimization incompatibility and confusing performance tradeoffs.
|
||||
|
||||
Long term, we've added the ability to partition the graph in Inductor instead of right after Dynamo. It can be enabled with `CompilationConfig.use_inductor_graph_partition=True` but is currently experimental and only available with `torch>=2.9`. This also increases compilation time as it has to compile the whole graph and cannot reuse piecewise compilation artifacts. Once vLLM supports 2.9, we plan to make this the default approach as it will also speed up piecewise cudagraph capture.
|
||||
|
||||
|
||||
@@ -7,27 +7,20 @@ For two-tower vision encoders (e.g., DeepSeek-OCR's SAM + CLIP with dynamic tili
|
||||
!!! note
|
||||
Encoder CUDA Graphs are orthogonal to decoder CUDA Graphs — both can be enabled simultaneously. Encoder graphs capture the vision encoder execution (e.g., ViT in Qwen3-VL), while decoder graphs capture the language model execution as described in the [CUDA Graphs design document](cuda_graphs.md).
|
||||
|
||||
## Compatibility Matrix
|
||||
|
||||
!!! note
|
||||
The symbols used below have the following meanings:
|
||||
|
||||
- ✅ = Full compatibility
|
||||
- 🟠 = Partial compatibility
|
||||
- ❌ = No compatibility
|
||||
- ❔ = Unknown or TBD
|
||||
|
||||
### Model x Feature
|
||||
## Supported Models
|
||||
|
||||
| Architecture | Models | CG for Image | CG for Video | Multi-Path Graph |
|
||||
| ------------ | ------ | ------------ | ------------ | --------------- |
|
||||
| `DeepseekOCRForCausalLM` | `DeepSeek-OCR` | ✅︎ | ❌︎ | ✅︎ |
|
||||
| `DeepseekV41ForCausalLM` | `DeepSeek-V4.1-Flash` | ✅︎ | ❌︎ | ❌︎ |
|
||||
| `Ernie4_5_VLMoeForConditionalGeneration` | `ERNIE-4.5-VL` | ✅︎ | ❌︎ | ❌︎ |
|
||||
| `Gemma3ForConditionalGeneration` | `Gemma3` | ✅︎ | ❌︎ | ❌︎ |
|
||||
| `Glm4vForConditionalGeneration` | `GLM-4.1V, GLM-4.6V-Flash` | ✅︎ | ✅︎ | ❌︎ |
|
||||
| `Gemma4ForConditionalGeneration` | `Gemma-4` | ✅︎ | ✅︎ | ❌︎ |
|
||||
| `InternVLChatModel` | `InternVL3.5`, `InternVL3`, `InternVL2.5`, `InternVL2` | ✅︎ | ✅︎ | ❌︎ |
|
||||
| `KimiVLForConditionalGeneration` | `Kimi-VL` | ✅︎ | ❌︎ | ❌︎ |
|
||||
| `KimiK25ForConditionalGeneration` | `Kimi-K2.5, Kimi-K2.6` | ✅︎ | ❌︎ | ❌︎ |
|
||||
| `KimiK3ForConditionalGeneration` | `Kimi-K3` | ✅︎ | ❌︎ | ❌︎ |
|
||||
| `Llama4ForConditionalGeneration` | `Llama 4` | ✅︎ | ❌︎ | ❌︎ |
|
||||
| `Qwen2VLForConditionalGeneration` | `Qwen2-VL` | ✅︎ | ✅︎ | ❌︎ |
|
||||
| `Qwen2_5_VLForConditionalGeneration` | `Qwen2.5-VL` | ✅︎ | ✅︎ | ❌︎ |
|
||||
@@ -38,26 +31,6 @@ For two-tower vision encoders (e.g., DeepSeek-OCR's SAM + CLIP with dynamic tili
|
||||
| `MiniCPMV` | `MiniCPMV2.5` | ✅︎ | ❌︎ | ❌︎ |
|
||||
| `MiniCPMV` | `MiniCPMV2.6`, `MiniCPMV4.0` | ✅︎ | ✅︎ | ❌︎ |
|
||||
|
||||
### Model x Hardware
|
||||
|
||||
| Architecture | NV Blackwell | NV Ampere | AMD MI300X | AMD MI350X / MI355X |
|
||||
| ------------ | ---------------- | ------------- | -------------- | --------------------- |
|
||||
| `DeepseekOCRForCausalLM` | ✅︎ | ✅︎ | ❔ | ✅︎ |
|
||||
| `Ernie4_5_VLMoeForConditionalGeneration` | ✅︎ | ✅︎ | ❔ | ✅︎ |
|
||||
| `Gemma3ForConditionalGeneration` | ✅︎ | ✅︎ | ❔ | ✅︎ |
|
||||
| `Glm4vForConditionalGeneration` | ✅︎ | ✅︎ | ❔ | ✅︎ |
|
||||
| `Gemma4ForConditionalGeneration` | ✅︎ | ✅︎ | ❔ | ✅︎ |
|
||||
| `InternVLChatModel` | ✅︎ | ✅︎ | ❔ | ✅︎ |
|
||||
| `KimiVLForConditionalGeneration` | ✅︎ | ✅︎ | ❔ | ✅︎ |
|
||||
| `Llama4ForConditionalGeneration` | ✅︎ | ✅︎ | ❔ | ✅︎ |
|
||||
| `Qwen2VLForConditionalGeneration` | ✅︎ | ✅︎ | ❔ | ✅︎ |
|
||||
| `Qwen2_5_VLForConditionalGeneration` | ✅︎ | ✅︎ | ❔ | ✅︎ |
|
||||
| `Qwen3VLForConditionalGeneration` | ✅︎ | ✅︎ | ❔ | ✅︎ |
|
||||
| `Qwen3_5ForConditionalGeneration` | ✅︎ | ✅︎ | ❔ | ✅︎ |
|
||||
| `Qwen3_5MoeForConditionalGeneration` | ✅︎ | ✅︎ | ❔ | ✅︎ |
|
||||
| `Step3VLForConditionalGeneration` | ✅︎ | ✅︎ | ❔ | ✅︎ |
|
||||
| `MiniCPMV` | ✅︎ | ✅︎ | ❔ | ❔ |
|
||||
|
||||
!!! note
|
||||
Encoder CUDA Graph has currently been tested with `--mm-encoder-attn-backend=FLASH_ATTN` and `--mm-encoder-attn-backend=FLASHINFER` on Blackwell GPUs.
|
||||
For Qwen2-VL and Qwen2.5-VL only FA2 and FA3 has been tested.
|
||||
@@ -80,7 +53,7 @@ The encoder CUDA Graph system uses a **budget-based capture/replay** strategy, m
|
||||
* [EncoderItemSpec][vllm.v1.worker.encoder_cudagraph_defs.EncoderItemSpec]: describes a single encoder input item (image or video) with its input size and output token count.
|
||||
* [BudgetGraphMetadata][vllm.v1.worker.encoder_cudagraph.BudgetGraphMetadata]: holds the captured CUDA Graph and its associated I/O buffers for a single token budget level.
|
||||
|
||||
### Budget-based graph capture
|
||||
### Budget-Based Graph Capture
|
||||
|
||||
Multiple CUDA Graphs are pre-captured at different **token budget** levels (e.g., `[2048, 4096, 8192, 13824]`). Each budget defines a fixed token capacity, and all budgets share the same maximum batch size (number of images). The `BudgetGraphMetadata` for each level stores the graph along with pre-allocated input, metadata, and output buffers:
|
||||
|
||||
@@ -99,20 +72,7 @@ Budgets are auto-generated as power-of-2 levels from a model-provided range via
|
||||
|
||||
Each entry in `EncoderCudaGraphConfig.paths` defines an independently captured encoder path. A path can provide its own minimum token budget and opt into zero-token batches; the manager generates and stores a separate budget graph set for every configured path.
|
||||
|
||||
### Greedy bin-packing at runtime
|
||||
|
||||
When a batch of images arrives, the manager sorts images by output token count (smallest first) and greedily packs as many images as possible into each sub-batch while staying within the **largest** token budget and the maximum batch size. Once a sub-batch is finalized (the next image would overflow either constraint), the manager finds the **smallest** budget that fits the sub-batch's total tokens and replays the corresponding CUDA Graph. This repeats until the batch is exhausted. Images that exceed all budgets fall back to eager execution.
|
||||
|
||||
For multi-path models, the same greedy packing loop constrains every configured path simultaneously (see [Multi-Path graph capture](#multi-path-graph-capture)).
|
||||
|
||||
For each graph replay:
|
||||
|
||||
1. Call `prepare_encoder_cudagraph_replay_buffers()` to compute buffer values (including `pixel_values` and precomputed metadata) from actual batch inputs.
|
||||
2. Zero the pre-allocated `input_buffers`, then slice-copy the replay values into them.
|
||||
3. Replay the CUDA Graph.
|
||||
4. Clone outputs from `output_buffer` (cloning is necessary since the buffer is reused across replays).
|
||||
|
||||
### Multi-Path graph capture
|
||||
### Multi-Path Graph Capture
|
||||
|
||||
`EncoderCudaGraphConfig.paths` maps path names to `EncoderCudaGraphPathConfig` capture policies. For example, DeepSeek-OCR configures a **global** image path and a **local** patch path, which are captured independently under `budget_graphs["global"]` and `budget_graphs["local"]`.
|
||||
|
||||
@@ -127,7 +87,7 @@ Both lists are capped at the same `max_budget`.
|
||||
|
||||
* Sort images by total output tokens (global + local), smallest first.
|
||||
* Greedily pack images: an image is added to the current sub-batch only if every accumulated path token count is within that path's maximum budget, with the image count ≤ `max_batch_size`.
|
||||
* Once any path constraint would overflow, finalize the sub-batch and find the smallest fitting budget **independently** for each path.
|
||||
* Once any path constraint would overflow, or the budget-cliff deferral condition fires on any path, finalize the sub-batch and find the smallest fitting budget **independently** for each path.
|
||||
* Repeat until all images are packed.
|
||||
|
||||
**Partial graph fallback.** For each non-empty path, the manager replays the smallest fitting graph or runs only that path eagerly when no graph fits. Paths with zero tokens are skipped; a `0`-budget graph is never captured or replayed.
|
||||
@@ -139,11 +99,41 @@ Both lists are capped at the same `max_budget`.
|
||||
!!! note
|
||||
The dual-path design enables partial CUDA graph coverage — one path can hit while the other falls back to eager. This avoids wasted compute on zero-padded patch buffers for untiled images and avoids graph invalidation caused by variable `crop_shape` per image.
|
||||
|
||||
### Data-parallel support
|
||||
### Greedy Bin-Packing at Runtime
|
||||
|
||||
When a batch of images arrives, the manager sorts images by output token count (smallest first) and greedily packs as many images as possible into each sub-batch while staying within the **largest** token budget and the maximum batch size. Once a sub-batch is finalized (the next image would overflow either constraint, or deferring it is cheaper), the manager finds the **smallest** budget that fits the sub-batch's total tokens and replays the corresponding CUDA Graph. This repeats until the batch is exhausted. Images that exceed all budgets fall back to eager execution.
|
||||
|
||||
For multi-path models, the same greedy packing loop constrains every configured path simultaneously.
|
||||
|
||||
For each graph replay:
|
||||
|
||||
1. Call `prepare_encoder_cudagraph_replay_buffers()` to compute buffer values (including `pixel_values` and precomputed metadata) from actual batch inputs.
|
||||
2. Zero the pre-allocated `input_buffers`, then slice-copy the replay values into them.
|
||||
3. Replay the CUDA Graph.
|
||||
4. Clone outputs from `output_buffer` (cloning is necessary since the buffer is reused across replays).
|
||||
|
||||
**Budget-cliff deferral:**
|
||||
|
||||
Greedy packing can be wasteful when a large image arrives after small ones: merging it into the current sub-batch may force the whole batch into a much larger budget graph even though splitting would cost less. Because token budgets are discrete (e.g., `[256, 4096, 8192]`), one extra token can double the required graph level — a "budget cliff". Images that overflow the largest budget still finalize the sub-batch unconditionally; for images that *fit*, the packer additionally applies a deferral test: finalize the current sub-batch (replaying it at its own small budget) and start a new sub-batch with the deferred image.
|
||||
|
||||
For each candidate image, the packer compares three budgets per path: `b_cur` (the smallest budget fitting the current sub-batch), `b_alone` (the smallest budget fitting the image alone), and `b_merged` (the smallest budget fitting both). Merging costs `b_merged`; deferring costs `b_cur + b_alone` plus one extra replay. The image is deferred when, on any path, both conditions hold:
|
||||
|
||||
1. `b_cur + b_alone < b_merged` — splitting strictly saves budget. The strict inequality makes ties merge: with budgets `[256, 4096, 8192]`, two 4000-token images tie (`4096 + 4096 == 8192`), and merging wins because a single replay avoids an extra graph launch.
|
||||
2. `b_alone >= b_cur` — the image alone occupies at least as large a budget bucket as the entire current sub-batch.
|
||||
|
||||
**Condition 1 is the cost test.** For example, a 200-token image followed by a 4000-token image with budgets `[256, 4096, 8192]`: merging takes `b_merged = 8192` (4200 tokens round up to 8192), while deferring takes `256 + 4096 = 4352` — the image is deferred, saving 3840 tokens of padding compute.
|
||||
|
||||
**Condition 2 separates "whales" from "tag-alongs".** Because images are sorted smallest first, a whale arrives last, and deferring it lets the small sub-batch replay at its own small budget instead of being dragged into a large bucket for one image's sake. Without condition 2, deferring mid-size images fragments a flood of similar images into extra replays: with budgets `[256, 2048, 4096, 8192]` and six equal 1366-token images, condition 1 alone defers every third image and yields three 4096 replays (12288 budget tokens in total), while with condition 2 five images merge into one 8192 replay plus a single 2048 replay — fewer replays *and* fewer budget tokens (10240).
|
||||
|
||||
The conditions compose cleanly: when `b_alone == b_cur`, condition 1 requires `b_merged > 2 * b_cur`, i.e. deferral fires only when merging jumps at least two budget levels, so ties always merge. This also lets small images defer across a huge cliff: with budgets `[256, 4096]` and four 100-token images, adding the third image (300 tokens) would jump from 256 to 4096, so the packer defers it and packs the images as two 256-budget pairs.
|
||||
|
||||
For multi-path items, the conditions are evaluated per path and OR-ed: any path on which the image is a whale defers the whole image, while paths where the image contributes zero tokens are skipped.
|
||||
|
||||
### ViT DP Mode Support
|
||||
|
||||
When `mm_encoder_tp_mode="data"`, the manager distributes images across TP ranks using load-balanced assignment via `get_load_balance_assignment`, executes locally on each rank, then gathers results back in the original order via `tensor_model_parallel_all_gather`.
|
||||
|
||||
### Video inference support
|
||||
### Video Inference Support
|
||||
|
||||
Following <https://github.com/vllm-project/vllm/pull/35963> (ViT full CUDA graph support for image inference), <https://github.com/vllm-project/vllm/pull/38061> extends the encoder CUDA graph framework to support video inference for Qwen3-VL. Previously, the CUDA graph capture/replay path only handled image inputs (`pixel_values` + `image_grid_thw`). Video inputs use different keys (`pixel_values_videos` + `video_grid_thw`) and require larger `cu_seqlens` buffers because each video item contributes multiple frames (`T` attention sequences). This PR generalizes the protocol and manager to handle both modalities through a single shared graph manager.
|
||||
|
||||
@@ -152,7 +142,7 @@ Following <https://github.com/vllm-project/vllm/pull/35963> (ViT full CUDA graph
|
||||
|
||||
Mixed inputs (image+video) per prompt are also supported now.
|
||||
|
||||
## Model integration via `SupportsEncoderCudaGraph`
|
||||
## Model Integration via `SupportsEncoderCudaGraph`
|
||||
|
||||
Models opt-in to encoder CUDA Graphs by implementing the [SupportsEncoderCudaGraph][vllm.model_executor.models.interfaces.SupportsEncoderCudaGraph] protocol. This protocol encapsulates all model-specific logic so that the manager remains model-agnostic. The protocol defines the following methods:
|
||||
|
||||
@@ -180,9 +170,9 @@ Four fields in `CompilationConfig` control encoder CUDA Graphs:
|
||||
|
||||
Multi-path mode is configured at the model level through `EncoderCudaGraphConfig.paths`. Each `EncoderCudaGraphPathConfig` can set a path-specific minimum budget and whether zero-token batches are allowed. The manager automatically generates separate budget lists and uses the same execution loop for single- and multi-path models.
|
||||
|
||||
## Usage guide
|
||||
## Usage Guide
|
||||
|
||||
### Image inference
|
||||
### Image Inference
|
||||
|
||||
Enable encoder CUDA Graphs via `compilation_config`:
|
||||
|
||||
@@ -226,7 +216,7 @@ model = vllm.LLM(
|
||||
|
||||
The manager tracks hit/miss statistics and logs them periodically. A "hit" means an image was processed via CUDA Graph replay; a "miss" means eager fallback (image exceeded all budgets).
|
||||
|
||||
### Video inference
|
||||
### Video Inference
|
||||
|
||||
Enable encoder CUDA Graphs via `compilation_config`:
|
||||
|
||||
|
||||
@@ -251,6 +251,18 @@ that may call 1+ triton kernels. On rare (but unfortunate) occasions, it may
|
||||
produce an incorrect triton kernel. This may manifest as silent incorrectness,
|
||||
CUDA illegal memory accesses, or loud errors.
|
||||
|
||||
### Deterministic compilation
|
||||
|
||||
When vLLM selects combo-kernel defaults, it disables combo-kernel benchmarking
|
||||
if Inductor's `deterministic` setting is enabled. A `deterministic` value in
|
||||
`inductor_compile_config` takes precedence over the global Inductor setting.
|
||||
On torch 2.13, `TORCHINDUCTOR_DETERMINISTIC=1` alone is reset by the first
|
||||
compiled frame ([pytorch#198563](https://github.com/pytorch/pytorch/issues/198563)),
|
||||
so set `deterministic` in `inductor_compile_config` instead.
|
||||
Explicit `combo_kernels` or `benchmark_combo_kernel` settings are preserved.
|
||||
This avoids conflicting compilation defaults; it does not guarantee reproducible
|
||||
model outputs.
|
||||
|
||||
### Inductor runtime assertions
|
||||
|
||||
By default (on torch < 2.12), vLLM disables Inductor's runtime assertions
|
||||
|
||||
+1
-75
@@ -25,8 +25,6 @@ or just on the low or high end.
|
||||
| [MLA Attention + Quant](#attention--quantization-fuse_attn_quant) | `fuse_attn_quant` | MLA Attention output → FP8/NVFP4 quant | Off by default | TBD | Yes | Always |
|
||||
| [RoPE + KV-Cache Update](#rope--kv-cache-update-fuse_rope_kvcache) | `fuse_rope_kvcache` | Rotary embedding → KV cache write | O2 (ROCm/AITER only) | 2-4% | No | Low |
|
||||
| [QK Norm + RoPE](#qk-norm--rope-enable_qk_norm_rope_fusion) | `enable_qk_norm_rope_fusion` | Q/K RMSNorm → rotary embedding | Off by default | 2-3% | No | Low |
|
||||
| [Sequence Parallelism](#sequence-parallelism-enable_sp) | `enable_sp` | AllReduce → ReduceScatter + AllGather | Off by default | Prereq for AsyncTP | Yes | High |
|
||||
| [AsyncTP GEMM + collective](#asynctp-gemm--collective-overlap-fuse_gemm_comms) | `fuse_gemm_comms` | GEMM → reduce-scatter / all-gather → GEMM | Off by default | 7-10% | Yes | High |
|
||||
| [RMSNorm + Quant](#rmsnorm--quantization-fuse_norm_quant) | `fuse_norm_quant` | RMSNorm (+residual add) → FP8/FP4 quant | O1 (conditional) | 1-4% | No | Always |
|
||||
| [SiLU+Mul + Quant](#silumul--quantization-fuse_act_quant) | `fuse_act_quant` | SiLU+Mul activation → FP8/FP4 quant | O1 (conditional) | 1-4% | No | Always |
|
||||
| [RMSNorm + Padding](#rmsnorm--padding-fuse_act_padding) | `fuse_act_padding` | Residual add + RMSNorm → padding | O1 (ROCm/AITER only) | TBD | No | Always |
|
||||
@@ -44,9 +42,7 @@ The table below lists the quantization schemes supported by each fusion on each
|
||||
| `fuse_attn_quant`\* | FP8 static\*, NVFP4\* | FP8 static\* | FP8 static\* | — | FP8 static\* |
|
||||
| `fuse_attn_quant` (MLA)\* | FP8 static\*, FP8 per-group\*, NVFP4\* | FP8 static\*, FP8 per-group\* | FP8 static\*, FP8 per-group\* | — | FP8 static\* (untested) |
|
||||
| `fuse_rope_kvcache` | — | — | — | — | FP16/BF16 |
|
||||
| `enable_qk_norm_rope_fusion` | FP16/BF16 | FP16/BF16 | FP16/BF16† | FP16/BF16† | — |
|
||||
| `enable_sp` | FP16/BF16, FP8 static† | FP16/BF16, FP8 static | FP16/BF16† | FP16/BF16† | — |
|
||||
| `fuse_gemm_comms` | FP16/BF16, FP8 static† | FP16/BF16, FP8 static | FP16/BF16† | FP16/BF16† | — |
|
||||
| `enable_qk_norm_rope_fusion` | FP16/BF16 | FP16/BF16 | FP16/BF16 | FP16/BF16 | — |
|
||||
| `fuse_norm_quant` | FP8 static, FP8 per-token, FP8 per-group | FP8 static, FP8 per-token, FP8 per-group | FP8 static, FP8 per-token, FP8 per-group | — | FP8 static, FP8 per-token, FP8 per-group |
|
||||
| `fuse_act_quant` | FP8 static, NVFP4 | FP8 static, FP8 per-group (128/64) | FP8 static, FP8 per-group (128/64) | — | FP8 per-group |
|
||||
| `fuse_act_padding` | — | — | — | — | FP16/BF16 |
|
||||
@@ -56,10 +52,6 @@ The table below lists the quantization schemes supported by each fusion on each
|
||||
fused quantization output. See the [`fuse_attn_quant` section](#attention--quantization-fuse_attn_quant)
|
||||
for per-backend details.
|
||||
|
||||
† `enable_sp` and `fuse_gemm_comms` are only autoconfigured for SM90 today;
|
||||
other architectures support requires setting `PassConfig.sp_min_token_num` explicitly.
|
||||
SM100 support also requires setting `VLLM_DISABLED_KERNELS=FlashInferFP8ScaledMMLinearKernel`.
|
||||
|
||||
## Enabling / Disabling Fusions
|
||||
|
||||
Fusions are exposed through `PassConfig`, which is nested inside `CompilationConfig`:
|
||||
@@ -186,72 +178,6 @@ If these conditions are set, the fusion is enabled automatically for optimizatio
|
||||
|
||||
- Pass: [`vllm/compilation/passes/fusion/rope_kvcache_fusion.py`](https://github.com/vllm-project/vllm/blob/main/vllm/compilation/passes/fusion/rope_kvcache_fusion.py)
|
||||
|
||||
### Sequence Parallelism (`enable_sp`)
|
||||
|
||||
**What it fuses.** Replaces all-reduce collectives with reduce-scatter + local RMSNorm + all-gather,
|
||||
splitting the sequence dimension across TP ranks. This restructures the graph so the subsequent AsyncTP
|
||||
pass can fuse the reduce-scatter / all-gather with the surrounding GEMMs.
|
||||
|
||||
Sequence Parallelism itself does not directly improve performance; it is a prerequisite for the
|
||||
AsyncTP pass (`fuse_gemm_comms`). SP is only applied above a minimum token threshold that is
|
||||
autoconfigured based on device capability and model `hidden_size`. Currently only active on
|
||||
H100/SM90 for models with `hidden_size >= 8192`. The threshold is configurable via
|
||||
`PassConfig.sp_min_token_num`.
|
||||
|
||||
The general transformation:
|
||||
|
||||
```text
|
||||
Input → AllReduce → RMSNorm → Output
|
||||
becomes:
|
||||
Input → ReduceScatter → local RMSNorm → AllGather → Output
|
||||
```
|
||||
|
||||
Patterns covered:
|
||||
|
||||
- First block: `AllReduce → RMSNorm` → `ReduceScatter → RMSNorm → AllGather`
|
||||
- Middle blocks: `AllReduce → fused_add_RMSNorm` → `ReduceScatter → fused_add_RMSNorm → AllGather`
|
||||
- Both with optional `→ FP8 static quant` suffix
|
||||
|
||||
Requires: `use_inductor_graph_partition=True` **or** piecewise compilation with static sizes
|
||||
divisible by `tensor_parallel_size`.
|
||||
|
||||
Supported hardware: Only tested on NVIDIA CUDA, possibly works on ROCm. FP8 all-gather requires sm90+.
|
||||
|
||||
**Code locations.**
|
||||
|
||||
- Pass: [`vllm/compilation/passes/fusion/sequence_parallelism.py`](https://github.com/vllm-project/vllm/blob/main/vllm/compilation/passes/fusion/sequence_parallelism.py)
|
||||
|
||||
### AsyncTP GEMM + Collective Overlap (`fuse_gemm_comms`)
|
||||
|
||||
!!! info
|
||||
Requires `enable_sp=True` (enabled automatically). This pass is a no-op if Sequence Parallelism has not been applied.
|
||||
|
||||
**What it fuses.** After Sequence Parallelism transforms the graph, fuses GEMM kernels with the
|
||||
surrounding reduce-scatter (output projection) and all-gather (input projection) using
|
||||
`torch.ops.symm_mem` symmetric-memory primitives, overlapping communication and computation.
|
||||
This overlap is only profitable for large `num_tokens`, so the fusion (and preceding SP)
|
||||
is only performed in the higher compiled range above `PassConfig.sp_min_token_num`.
|
||||
|
||||
Patterns covered:
|
||||
|
||||
- `GEMM → reduce-scatter` → `fused_matmul_reduce_scatter`
|
||||
- `all-gather → GEMM` → `all_gather_matmul`
|
||||
- FP8 scaled variants of both patterns
|
||||
|
||||
Supported hardware: NVIDIA CUDA with symmetric-memory (`torch.distributed._symmetric_memory`) support.
|
||||
|
||||
On B200, pattern-matching fp8 FlashInfer scaled MM is not supported, so it must be disabled
|
||||
([#27893](https://github.com/vllm-project/vllm/issues/27893))
|
||||
|
||||
```shell
|
||||
VLLM_DISABLED_KERNELS=FlashInferFP8ScaledMMLinearKernel ...
|
||||
```
|
||||
|
||||
**Code locations.**
|
||||
|
||||
- Pass: [`vllm/compilation/passes/fusion/collective_fusion.py`](https://github.com/vllm-project/vllm/blob/main/vllm/compilation/passes/fusion/collective_fusion.py)
|
||||
- Sequence parallelism pass: [`vllm/compilation/passes/fusion/sequence_parallelism.py`](https://github.com/vllm-project/vllm/blob/main/vllm/compilation/passes/fusion/sequence_parallelism.py)
|
||||
|
||||
### QK Norm + RoPE (`enable_qk_norm_rope_fusion`)
|
||||
|
||||
!!! info
|
||||
|
||||
@@ -37,6 +37,14 @@ memory consumption is therefore topology- and implementation-dependent. The
|
||||
realized capacity may be slightly smaller because the budget is rounded down
|
||||
to complete host blocks.
|
||||
|
||||
Startup logs report two concurrency bounds at `max_model_len`. The generic
|
||||
`Maximum concurrency` line charges each request its full admission footprint,
|
||||
including the in-flight window of every resident group. The `HiSparse
|
||||
steady-state maximum concurrency` line charges running requests that read from
|
||||
host only their active tail pages, plus one request being admitted at its full
|
||||
footprint. Host-pool metrics are listed in
|
||||
[Metrics](../usage/metrics.md#hisparse-kv-connector-metrics).
|
||||
|
||||
## Ownership
|
||||
|
||||
| Thing | Owner | What “owner” means |
|
||||
|
||||
@@ -85,6 +85,8 @@ GPU-side fusion is controlled by `multimodal_config.mm_device_do_normalize`:
|
||||
| `glm-ocr` | `GlmOcrForConditionalGeneration` | `zai-org/GLM-OCR` |
|
||||
| `intern-s1-pro` | `InternS1ProForConditionalGeneration` | `internlm/Intern-S1-Pro` |
|
||||
| `intern-s2-mobius` | `InternS2MobiusForConditionalGeneration` | `internlm/Intern-S2-Mobius` |
|
||||
| `kimi-k2.5` | `KimiK25ForConditionalGeneration` | `moonshotai/Kimi-K2.5`, etc. |
|
||||
| `kimi-k3` | `KimiK3ForConditionalGeneration` | `moonshotai/Kimi-K3` |
|
||||
| `llama-nemotron-vl-embed` | `LlamaNemotronVLForEmbedding` | `nvidia/llama-nemotron-embed-vl-1b-v2` |
|
||||
| `llama-nemotron-vl-rerank` | `LlamaNemotronVLForSequenceClassification` | `nvidia/llama-nemotron-rerank-vl-1b-v2` |
|
||||
| `mistral3` | `Mistral3ForConditionalGeneration` | `mistralai/Mistral-Small-3.2-24B-Instruct-2506`, etc. |
|
||||
|
||||
@@ -233,19 +233,17 @@ storing this information in vLLM's `PassContext` for later use in clone eliminat
|
||||
### 3. IR Fusion and Transformation Passes
|
||||
|
||||
After functionalization, custom vLLM passes operate on the functional FX graph containing high-level IR operations.
|
||||
These passes can perform fusion, distribute operations for sequence parallelism, and other transformations:
|
||||
These passes can perform fusion and other transformations:
|
||||
|
||||
```python
|
||||
# Example: Sequence Parallelism (see SequenceParallelismPass)
|
||||
# Before SP pass
|
||||
# Example: AllReduce + RMSNorm fusion (see AllReduceFusionPass)
|
||||
# Before
|
||||
|
||||
all_reduce = torch.ops.vllm.all_reduce(x, "tp:0")
|
||||
rms_norm = torch.ops.vllm_ir.rms_norm(all_reduce, weight, 1e-5)
|
||||
|
||||
# after SP pass
|
||||
reduce_scatter = torch.ops.vllm.reduce_scatter(x, "tp:0")
|
||||
rms_norm = torch.ops.vllm_ir.rms_norm(all_reduce, weight, 1e-5)
|
||||
all_gather = torch.ops.vllm.all_gather(x, "tp:0")
|
||||
# After
|
||||
rms_norm = torch.ops.vllm.flashinfer_trtllm_fused_allreduce_norm(x, weight, ...)
|
||||
```
|
||||
|
||||
Fusion passes benefit from the high-level representation: they don't need to match against low-level PyTorch operations,
|
||||
|
||||
@@ -126,7 +126,7 @@ Batch invariance has been tested and verified on the following models:
|
||||
- **Qwen3 (Dense)**: `Qwen/Qwen3-1.7B`, `Qwen/Qwen3-8B`, `Qwen/Qwen3-4B-AWQ`, `Qwen/Qwen3-8B-AWQ`
|
||||
- **Qwen3-VL (Vision-Language)**: `Qwen/Qwen3-VL-2B-Instruct`, `Qwen/Qwen3-VL-4B-Instruct` (single image and video inputs)
|
||||
- **Qwen3 (MoE)**: `Qwen/Qwen3-30B-A3B`, `Qwen/Qwen3-Next-80B-A3B-Instruct`, `Qwen/Qwen3-30B-A3B-Thinking-2507-FP8`
|
||||
- **Qwen2.5**: `Qwen/Qwen2.5-0.5B-Instruct`, `Qwen/Qwen2.5-1.5B-Instruct`, `Qwen/Qwen2.5-3B-Instruct`, `Qwen/Qwen2.5-7B-Instruct`, `Qwen/Qwen2.5-14B-Instruct`, `Qwen/Qwen2.5-32B-Instruct`
|
||||
- **Qwen2.5**: `Qwen/Qwen2.5-0.5B-Instruct`, `Qwen/Qwen2.5-1.5B-Instruct`, `Qwen/Qwen2.5-3B-Instruct`, `Qwen/Qwen2.5-7B-Instruct`, `Qwen/Qwen2.5-14B-Instruct`, `Qwen/Qwen2.5-32B-Instruct`, `Qwen/Qwen2.5-Coder-7B-Instruct`
|
||||
- **Llama 3**: Llama3.1 and 3.2 series, `meta-llama/Llama-3.2-1B-Instruct`, `meta-llama/Llama-3.2-3B-Instruct` for example
|
||||
- **GPT-OSS**: `openai/gpt-oss-20b`, `openai/gpt-oss-120b`
|
||||
- **Mistral**: `mistralai/Mistral-7B-v0.3`
|
||||
@@ -148,11 +148,10 @@ When batch invariance is enabled, vLLM:
|
||||
|
||||
1. Uses deterministic kernel implementations for attention and other operations
|
||||
2. Ensures consistent numerical behavior across different batch sizes
|
||||
3. Disables certain optimizations that may introduce non-determinism (such as sequence parallelism / async TP, whose reduce-scatter path is not batch-invariant)
|
||||
3. Disables certain optimizations that may introduce non-determinism
|
||||
4. Under tensor parallelism, keeps custom all-reduce on with a fixed reduction order (the 1-stage kernel is pinned, and large inputs are reduced in fixed-size chunks), and disables FlashInfer, AITER and QuickReduce all-reduce
|
||||
5. On CUDA devices with tuned matmul table entries for the model's bf16 unquantized forward linear layers (Ada, Hopper, Blackwell),
|
||||
runs without `torch.compile` using breakable CUDA graphs so tile configs follow the runtime batch size; set `VLLM_USE_BREAKABLE_CUDAGRAPH=0` to opt out
|
||||
(not applied when sequence parallelism / async TP are enabled, since those are `torch.compile` passes).
|
||||
runs without `torch.compile` using breakable CUDA graphs so tile configs follow the runtime batch size; set `VLLM_USE_BREAKABLE_CUDAGRAPH=0` to opt out.
|
||||
6. Splits the LoRA shrink reduction into 8 fixed K chunks that are summed in a fixed order instead of with atomics
|
||||
|
||||
!!! warning
|
||||
|
||||
@@ -280,10 +280,10 @@ curl -X POST http://localhost:8000/v1/load_lora_adapter \
|
||||
|
||||
!!! warning "You must know your adapter's layout"
|
||||
Under `--enable-mixed-moe-lora-format`, vLLM trusts whatever
|
||||
`is_3d_lora_weight` the caller declares — it does **not** inspect the
|
||||
checkpoint to verify. A wrong declaration will load weights into the
|
||||
wrong stacked buffers and silently produce garbage outputs, with no
|
||||
error at load time. Confirm the layout before serving:
|
||||
`is_3d_lora_weight` the caller declares. Loading checks reject fused 3D
|
||||
expert weights routed to a 2D wrapper without the required flags, but
|
||||
do not validate every layout mismatch. A wrong declaration can still
|
||||
cause loading failures or incorrect outputs. Confirm the layout before serving:
|
||||
|
||||
- **2D (per-expert, megatron-style)** → set `is_3d_lora_weight: false`.
|
||||
Adapter keys look like `...experts.{idx}.gate_proj.lora_A.weight`,
|
||||
@@ -299,6 +299,11 @@ is ignored: vLLM picks the wrapper from the base model's
|
||||
`is_3d_moe_weight` and the adapter is required to match. The field is
|
||||
also ignored for non-MoE models.
|
||||
|
||||
For a model using a 2D MoE wrapper (such as Qwen3-MoE), loading fused 3D
|
||||
expert adapter weights requires both `enable_mixed_moe_lora_format=True`
|
||||
on the engine and `is_3d_lora_weight=True` on the request. Otherwise, adapter
|
||||
loading raises a `ValueError` naming these settings before activation.
|
||||
|
||||
## LoRA model lineage in model card
|
||||
|
||||
The new format of `--lora-modules` is mainly to support the display of parent model information in the model card. Here's an explanation of how your current response supports this:
|
||||
|
||||
@@ -49,12 +49,13 @@ cache), see [ECMooncakeConnector Usage Guide](mooncake_ec_connector_usage.md).
|
||||
- For headless instances, must be the same as the master instance
|
||||
- Each instance needs a unique port on its host; using the same port number across different hosts is fine
|
||||
|
||||
- `VLLM_MOONCAKE_CONNECTOR_TIMEOUT`: Per-request timeout (in seconds) for MooncakeConnector HTTP calls to the bootstrap server, including prefiller worker registration at startup. (Optional)
|
||||
- `VLLM_MOONCAKE_CONNECTOR_TIMEOUT`: Per-request timeout (in seconds) for MooncakeConnector HTTP calls to the bootstrap server, including prefiller worker registration at startup and metadata queries. (Optional)
|
||||
- Default: 30.0
|
||||
- Global rank 0 hosts the bootstrap server in the same process that mounts the Mooncake transfer engine's host segment, so a large `MOONCAKE_GLOBAL_SEGMENT_SIZE` can delay responses for several seconds during startup
|
||||
- Raise this if bootstrap server timeout warnings appear on hosts with very large host-memory segments
|
||||
- Bootstrap registration uses up to 3 attempts internally, with exponential backoff capped at 10 seconds; connection errors and timeouts are both retried
|
||||
- Once the attempts are exhausted the caller raises instead of blocking, so failures surface rather than hanging
|
||||
- Once registration attempts are exhausted the caller raises instead of blocking, so failures surface rather than hanging
|
||||
- Metadata queries reuse the same timeout and 3-attempt limit, with a fixed 100ms delay between attempts. If all attempts fail, the error is logged and waiting queries are notified
|
||||
|
||||
- `WITH_NVIDIA_PEERMEM`: Selects how mooncake registers GPU memory for RDMA. Read by mooncake, not vLLM.
|
||||
- Default: 1, which uses `ibv_reg_mr()` and requires the `nvidia-peermem` kernel module to be loaded
|
||||
|
||||
@@ -1020,6 +1020,13 @@ vllm serve Qwen/Qwen3-VL-30B-A3B-Instruct \
|
||||
|
||||
When you extract video frames on the client side and send them as `video/jpeg` (base64-concatenated JPEG frames), you can preserve the original video metadata by using `media_io_kwargs` in your request. This enables more accurate video understanding by preserving temporal information that would otherwise be lost during client-side frame extraction.
|
||||
|
||||
!!! warning
|
||||
Per-request `media_io_kwargs` (and `mm_processor_kwargs`) are rejected by
|
||||
default because they let a client change media loading and preprocessing
|
||||
resource usage. Start the server with `--trust-request-mm-kwargs` only when
|
||||
your API clients are trusted; see
|
||||
[Per-request multimodal arguments](../usage/security.md#per-request-multimodal-arguments).
|
||||
|
||||
**Supported Parameters:**
|
||||
|
||||
| Parameter | Type | Description |
|
||||
@@ -1042,7 +1049,7 @@ When you extract video frames on the client side and send them as `video/jpeg` (
|
||||
frames_b64 = ",".join([encode_image(f) for f in frames])
|
||||
video_url = f"data:video/jpeg;base64,{frames_b64}"
|
||||
|
||||
# Pass video metadata via media_io_kwargs
|
||||
# Pass video metadata via media_io_kwargs (server started with --trust-request-mm-kwargs)
|
||||
response = client.chat.completions.create(
|
||||
model="your-multimodal-model",
|
||||
messages=[{
|
||||
|
||||
@@ -99,6 +99,35 @@ vllm serve openai/gpt-oss-20b --quantization-config.moe.activation mxfp8
|
||||
|
||||
Combine with `--moe-backend` to pin a specific kernel family.
|
||||
|
||||
### Re-quantizing layers to a different precision
|
||||
|
||||
An online target can re-quantize a checkpoint-quantized (prequantized) layer at load time.
|
||||
|
||||
vLLM uses the checkpoint method to load and dequantize the checkpoint weight, and then applies the online quantization method. This requires the checkpoint
|
||||
method to implement weight dequantization, unsupported source and target
|
||||
combinations raise an error.
|
||||
|
||||
Currently supported combinations are:
|
||||
|
||||
| Checkpoint layer | Online target | Requirements |
|
||||
| ---------------- | ------------- | ------------ |
|
||||
| Serialized ModelOpt MXFP8 linear | any | Target-specific hardware requirements |
|
||||
|
||||
For example, serialized ModelOpt MXFP8 linears can be converted to
|
||||
per-channel-weight/per-token-activation FP8 at load time:
|
||||
|
||||
```bash
|
||||
vllm serve MiniMaxAI/MiniMax-M3-MXFP8 \
|
||||
--linear-backend auto \
|
||||
--quantization-config.linear fp8_per_channel
|
||||
```
|
||||
|
||||
!!! info
|
||||
|
||||
This is only available for linear layers at the moment.
|
||||
|
||||
Checkpoint-exclusion policy is not enforced by online quantization/re-quantization. Only exclusion through `--quantization-config.targets`/`--quantization-config.ignore` is enforced.
|
||||
|
||||
### Online quantization on unquantized layers from partially-quantized checkpoints
|
||||
|
||||
Online quantization can be used on already quantized checkpoints independently of their original `quant_method` (`modelopt`, `compressed-tensors`, `quark`, etc.), for layers that are left unquantized in the original checkpoint.
|
||||
|
||||
@@ -107,7 +107,7 @@ By default, CUDA graph memory stays on the GPU while asleep. With
|
||||
`sleep_mode_offload_cudagraph=True` (off by default), CUDA graphs are captured
|
||||
into a cuMem pool that sleep backs up to CPU memory at both levels and any wake
|
||||
restores in place, so graphs are replayed, not recaptured. It needs the default
|
||||
`cumem` backend, CUDA and CUDA graphs; otherwise it has no effect.
|
||||
`cumem` backend, CUDA or ROCm, and CUDA graphs; otherwise it has no effect.
|
||||
|
||||
```python
|
||||
llm = LLM("Qwen/Qwen3-8B", enable_sleep_mode=True, sleep_mode_offload_cudagraph=True)
|
||||
|
||||
@@ -94,6 +94,8 @@ For best results, we recommend ensuring that the expected output format / schema
|
||||
To use a named function, you need to define the functions in the `tools` parameter of the chat completion request, and
|
||||
specify the `name` of one of the tools in the `tool_choice` parameter of the chat completion request.
|
||||
|
||||
In the Responses API, `tool_choice={"type": "function", "name": "..."}` can select a function inside a namespace by its full `namespace__function` name or by its local name. Names are matched literally first, so a plain function named `other__add` never stands in for `add`. A local name defined in more than one namespace must be selected by its full name.
|
||||
|
||||
## Required Function Calling
|
||||
|
||||
vLLM supports the `tool_choice='required'` option in the chat completion API. Similar to the named function calling, it also uses structured outputs, so this is enabled by default and will work with any supported model. However, support for alternative decoding backends are on the [roadmap](../usage/v1_guide.md#features) for the V1 engine.
|
||||
|
||||
@@ -18,8 +18,16 @@ vllm serve MODEL \
|
||||
```
|
||||
|
||||
Watermarking is disabled when `--watermark-config` is omitted. Gumbel-max (see
|
||||
`gumbel`) is the default algorithm within an enabled `WatermarkConfig`. When
|
||||
watermarking is configured, it is enabled for requests by default.
|
||||
`gumbel`) is the default algorithm within an enabled `WatermarkConfig`.
|
||||
|
||||
The request-level `watermarking` option has two effective states:
|
||||
|
||||
- Omitted (`None` in Python) or `true` requests watermarking.
|
||||
- `false` explicitly opts out.
|
||||
|
||||
When a request asks for watermarking but the engine or generation mode cannot
|
||||
apply it, vLLM emits a server-side warning once and runs the request without a
|
||||
watermark.
|
||||
|
||||
Requests can opt out without changing the engine-level algorithm or key:
|
||||
|
||||
@@ -30,14 +38,12 @@ sampling_params = SamplingParams(watermarking=False)
|
||||
```
|
||||
|
||||
The Python OpenAI-compatible APIs and the Rust chat/completions APIs accept the
|
||||
same `watermarking: false` request field. The Rust token API accepts it in
|
||||
`sampling_params`, and the Rust gRPC `GenerateRequest` accepts
|
||||
the optional `watermarking` field. Omitting the field keeps watermarking enabled
|
||||
when the engine has a watermark configuration.
|
||||
same optional `watermarking` request field. The Rust token API accepts it in
|
||||
`sampling_params`, and the Rust gRPC `GenerateRequest` accepts it directly.
|
||||
Omitting the field requests watermarking by default.
|
||||
|
||||
Deployments that require watermarking must restrict this field to trusted
|
||||
callers, or strip and validate it at the ingress boundary, so untrusted clients
|
||||
cannot opt out.
|
||||
Deployments that require watermarking must restrict explicit opt-out to trusted
|
||||
callers, or strip and validate the field at the ingress boundary.
|
||||
|
||||
`context_width` controls how many prior tokens seed each watermark decision
|
||||
and defaults to 4. Larger values make the watermark less robust to
|
||||
@@ -63,6 +69,31 @@ the reference algorithms. `WatermarkDetector` consumes token IDs, so callers
|
||||
remain responsible for using the tokenizer and watermark profile that match
|
||||
generation.
|
||||
|
||||
## Interactions with generation features
|
||||
|
||||
- Greedy decoding (`temperature=0`) cannot apply a Gumbel watermark. The first
|
||||
such request emits a server-side warning; it and subsequent greedy requests
|
||||
use ordinary greedy sampling without a watermark. This includes
|
||||
transcription, translation, and realtime transcription, which default to
|
||||
greedy decoding. Transcription and translation can use a nonzero temperature
|
||||
to apply the configured watermark.
|
||||
- Beam search cannot apply a watermark. The first watermarked beam request emits
|
||||
a server-side warning and runs without watermarking.
|
||||
- Trace replay replaces the sampled token and therefore cannot apply a
|
||||
watermark. The first watermarked trace replay request emits a server-side
|
||||
warning and runs without watermarking.
|
||||
- Parallel sampling (`n > 1`) is supported, but plain Gumbel candidates can be
|
||||
identical while their contexts remain identical.
|
||||
- Structured outputs and tool grammars are applied before watermark sampling.
|
||||
Restrictive constraints can reduce the statistical evidence available to a
|
||||
detector.
|
||||
- Automatic language detection and generative scoring use unwatermarked
|
||||
auxiliary generations. Language-detection tokens are not part of the returned
|
||||
transcription, and generative scoring returns scores rather than generated
|
||||
text.
|
||||
- Speculative decoding has additional requirements and partial-watermarking
|
||||
modes described in [Speculative decoding](#speculative-decoding).
|
||||
|
||||
## Speculative decoding
|
||||
|
||||
Watermarking requires speculative decoding to use probabilistic draft sampling,
|
||||
@@ -245,7 +276,7 @@ watermarked output or to modify watermarked text so it is no longer detected.
|
||||
|
||||
- Watermarking is currently available only with Model Runner V2.
|
||||
- Not all watermarking algorithms have native speculative-decoding support.
|
||||
- Beam search expands candidates from model log probabilities and does not apply
|
||||
Gumbel-max watermarking.
|
||||
- Models that replace the vLLM sampler with a custom sampler cannot use
|
||||
configured watermarking.
|
||||
- Global custom logits processors are unavailable because Model Runner V2 does
|
||||
not support them.
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user