diff --git a/.agents/skills/model-recipes/SKILL.md b/.agents/skills/model-recipes/SKILL.md new file mode 100644 index 0000000000..c462b85a8a --- /dev/null +++ b/.agents/skills/model-recipes/SKILL.md @@ -0,0 +1,151 @@ +--- +name: model-recipes +description: Find vLLM deployment recipes and configs for LLMs and Hugging Face models by model and hardware. Use for recipe lookup, creation, and updates. +--- + +# Model Recipes + +Model serving recipes are maintained in the +[models/ directory of vllm-project/recipes](https://github.com/vllm-project/recipes/tree/main/models). +The published catalog is at +[recipes.vllm.ai](https://recipes.vllm.ai). + +## Usage notes + +1. Recipe lookup returns commands and deployment guidance without launching + `vllm serve`. +2. Use published configurations or the recipes repository's command generator + for supported selections. Return commands as published, including their + flags and parallelism settings. +3. Recipes are starting points for deployment and may not be tuned for your + workload. Check each recipe for its verified configurations, and use + benchmarks of your workload to guide further tuning. +4. To create or update a recipe, follow the + [authoring workflow below](#create-or-update-a-recipe). + +## Find a recipe + +Read recipes from the published JSON on recipes.vllm.ai. It is generated from +the source YAML with variant, hardware, and strategy settings already +resolved; the YAML contains comments and conditional overrides, so do not use +it for lookup. When presenting results, link the website page. Read the source +YAML only when creating or updating a recipe. + +Each recipe has a page at `https://recipes.vllm.ai//` and JSON at +the same path plus `.json`. Paths use the full Hugging Face ID, preserving +case; do not substitute `meta.slug` or a display title. For example, +`Qwen/Qwen3.8-27B` maps to +[recipes.vllm.ai/Qwen/Qwen3.8-27B](https://recipes.vllm.ai/Qwen/Qwen3.8-27B). + +Recipe JSON is often over 20 KB, mostly the `guide`. Instead of printing it, +read only what the request needs: each step below returns the paths for the +next, so stop once the request is answered. Commands need `curl` and `jq`. +[scripts/read_recipe.sh](scripts/read_recipe.sh) is in this skill's directory; +the examples use its path from the vLLM repository root. + +### Search the catalog + +Skip this step if the exact HF ID is known. Otherwise search by model ID, +family, or provider using case-insensitive keywords. Each +whitespace-separated word must occur in the ID, title, or provider: + +```bash +curl -fsSL https://recipes.vllm.ai/models.json | + jq --arg q 'qwen 27b' ' + ($q | ascii_downcase | [scan("\\S+")]) as $words | .[] | + ([.hf_id, .title, .provider] | join(" ") | ascii_downcase) as $text | + select(all($words[]; . as $word | $text | contains($word))) | + {hf_id, json, url, derived_from}' +``` + +`json` is the recipe data path; `url` is its page path. Prefix both with +`https://recipes.vllm.ai`. A `derived_from` entry is a variant of a parent +recipe: fetch its own `json` to get that checkpoint's command. Its page may +belong to the parent, so link the returned `url` rather than assuming every +checkpoint has a separate page. + +If there is no exact match, search the family and inspect `variants` before +concluding the recipe is missing. + +### Summarize the recipe + +Save the recipe JSON to a file and summarize it: + +```bash +read_recipe=.agents/skills/model-recipes/scripts/read_recipe.sh +recipe=$(mktemp) +curl -fsSL https://recipes.vllm.ai/Qwen/Qwen3.8-27B.json -o "$recipe" +"$read_recipe" "$recipe" +``` + +`default` is the recommended hardware, strategy, and variant; `hardware` maps +each hardware ID to its command path; `recipe_verified_hardware` lists hardware +where the recipe as a whole was verified end to end. For capability +requirements, such as tool calling or text-to-video, check `features` and +`omni_tasks`, then confirm in the guide. +Some recipes, including Omni, lack `hardware`; skip hardware selection and +use the guide's launch and task-specific sections. + +### Select hardware and strategy + +Match a hardware ID to the user's GPU and fetch its path from `hardware`, +prefixed with `https://recipes.vllm.ai`, rather than constructing a URL. That +response is the command object itself: single-node objects carry `command`, +and multi-node objects carry `head_command` and `worker_commands` instead, so +list its keys before selecting fields. +Check `hardware_profile` to confirm that the selected configuration matches +the user's GPU model and specifications. Its `gpu_count` describes the profile, +not necessarily the number of GPUs used by the command. Check the command's +parallelism settings, `node_count`, and all `worker_commands` against the +user's available topology, and compare the variant's `vram_minimum_gb`, an +estimate, with the total memory of the GPUs the command uses; a configuration +below that minimum is not a working option unless the guide says it was +tested. Ask when the user's hardware is unclear. If no +published command or guide command fits, report the mismatch and return the +closest one unchanged; list any adjustments the user's setup would need +separately and mark them unverified instead of editing the command. + +For another strategy, follow `alternatives` in the same response. These are +static files; query parameters do not change their configuration. For other +feature/topology selections, use the recipes repository's command synthesis. + +### Read deployment guidance + +Read the guide sections that apply to the selection, such as prerequisites, +launch notes for the selected hardware or variant, troubleshooting, and known +limitations. Pass entries from `guide_sections` or prefixes of their text. +The script prints every matching section with its subsections and fails when +a section matches nothing: + +```bash +"$read_recipe" "$recipe" prerequisites troubleshooting +``` + +Query other fields with `jq` on the saved file, such as `features.`, +`hardware_overrides`, or `omni.tasks` for request examples. + +### Return the result + +Return the website link, selected checkpoint and hardware, the requested +command, and relevant prerequisites and caveats from the guide. Pick one +command and copy it, including its Docker form, exactly as the command object +or guide section gives it; do not combine commands from different sections or +add, drop, or reorder flags. List optional changes separately. + +For each command, state its source, either a generated hardware or strategy +configuration or a named guide section, and whether that source says it was +tested. A `recipe_verified_hardware` entry covers the recipe on that +hardware, not every variant, strategy, or guide command, so it does not make a +command verified. Generated configurations are not automatically verified. + +## Create or update a recipe + +Recipes cover official or widely used checkpoints, with configurations verified +end to end on real hardware. Do not add recipes for arbitrary community +uploads, untested hardware claims, or non-standard flags that only work for one +setup. Before opening a pull request, confirm that the user ran the +configuration and can share the results. + +Follow the recipes repository's +[add-recipe skill](https://github.com/vllm-project/recipes/blob/main/.agents/skills/add-recipe/SKILL.md) +in a checkout of `vllm-project/recipes` or the user's intended fork. diff --git a/.agents/skills/model-recipes/agents/openai.yaml b/.agents/skills/model-recipes/agents/openai.yaml new file mode 100644 index 0000000000..6ad0f4d4db --- /dev/null +++ b/.agents/skills/model-recipes/agents/openai.yaml @@ -0,0 +1,4 @@ +interface: + display_name: "Model Recipes" + short_description: "Find vLLM deployment recipes and configs for LLMs and Hugging Face models by model and hardware." + default_prompt: "Use $model-recipes to find a recipe for my model and hardware, and return its commands and deployment caveats." diff --git a/.agents/skills/model-recipes/scripts/read_recipe.sh b/.agents/skills/model-recipes/scripts/read_recipe.sh new file mode 100755 index 0000000000..28e880537e --- /dev/null +++ b/.agents/skills/model-recipes/scripts/read_recipe.sh @@ -0,0 +1,66 @@ +#!/bin/bash +# Usage: read_recipe.sh [guide-section...] +# Without sections, print a compact summary of a recipes.vllm.ai recipe JSON. +# With sections, print each guide section whose heading starts with one of +# them (case-insensitive), including its subsections. + +set -euo pipefail + +if [ $# -lt 1 ]; then + echo "Usage: $0 [guide-section...]" >&2 + exit 1 +fi + +# Print "\t\t" for guide lines, marking headings +# outside code fences with their level and other lines with 0. +guide_lines() { + jq -r '.guide // ""' "$1" | awk ' + /^[[:space:]]*(```|~~~)/ { fence = !fence } + !fence && match($0, /^#+ /) { + print RLENGTH - 1 "\t" substr($0, RLENGTH + 1) "\t" $0 + next + } + { print "0\t\t" $0 }' +} + +if [ $# -gt 1 ]; then + recipe=$1 + shift + status=0 + for section in "$@"; do + guide_lines "$recipe" | awk -F '\t' -v h="$section" ' + BEGIN { h = tolower(h); sub(/^#+[[:space:]]*/, "", h) } + $1 > 0 && lvl && $1 <= lvl { lvl = 0 } + $1 > 1 && !lvl && index(tolower($2), h) == 1 { lvl = $1; found = 1 } + lvl { sub(/^[^\t]*\t[^\t]*\t/, ""); print } + END { + if (!found) { + print "No guide section starts with: " h > "/dev/stderr" + exit 1 + } + }' || status=1 + done + exit $status +fi + +sections=$(guide_lines "$1" | awk -F '\t' '$1 == 2 || $1 == 3 { print $3 }' | + jq -R . | jq -s .) + +jq --argjson sections "$sections" '{ + hf_id, + title: .meta.title, + tasks: .meta.tasks, + recipe_verified_hardware: .meta.hardware, + omni_tasks: (.omni.tasks // null | + if . then map(objects |= {id, label, endpoint, model_id, description}) + else null end), + variants: (.variants // {} | map_values( + {model_id, precision, vram_minimum_gb, supported_hardware} | + with_entries(select(.value != null)))), + features: (.features // {} | keys), + opt_in_features, + default: (.recommended_command // null | + if . then {hardware, strategy, variant} else null end), + hardware: .recommended_command.by_hardware, + guide_sections: $sections +} | with_entries(select(.value != null))' "$1" diff --git a/.agents/skills/pr-checklist/SKILL.md b/.agents/skills/pr-checklist/SKILL.md index 886e6211a7..ae3099b6a0 100644 --- a/.agents/skills/pr-checklist/SKILL.md +++ b/.agents/skills/pr-checklist/SKILL.md @@ -1,6 +1,6 @@ --- name: pr-checklist -description: Prepare a vLLM change for human PR review or re-review, or analyze an open pull request. Use to check design fit, behavioral coverage, performance evidence, diff quality, and closure of previous feedback before requesting maintainer attention. +description: Prepare a vLLM change for human PR review or re-review, or analyze an open pull request. Use before opening a PR, drafts included (e.g. `gh pr create`), or pushing review fixes, to check design fit, behavioral coverage, performance evidence, diff quality, and closure of previous feedback. --- # PR Checklist @@ -181,6 +181,8 @@ Summarize model evaluations and benchmarks in compact tables with the configurat When fixing bugs or issues, provide a clear root-cause analysis. Explain what caused the problem, how it was identified, and why the chosen solution effectively addresses it. A patch which resolves the issue without a proper root-cause analysis may lead to recurring problems and increased maintenance overhead. +For fixes that involve disaggregated prefill/decode (PD) logic, such as the NIXL connector (`vllm/distributed/kv_transfer/kv_connector/v1/nixl/`), make a best effort to include a minimal example of the flow being changed and/or the initial flow that caused the failure. A short numbered sequence (or diagram) of a request's path across the P and D instances, e.g. scheduler admission, prefill execution, KV transfer registration and write, decode-side read, token generation, lets reviewers trace the root cause and the fix without reconstructing the distributed flow from the diff. Keep the example minimal: only the steps and components involved in the change. See the [disaggregated prefill](../../../docs/features/disagg_prefill.md), [NIXL KV push](../../../docs/design/nixl_kv_push_connector.md), and [NIXL KV cache lease](../../../docs/design/nixl_kv_cache_lease.md) docs for the flows involved. + #### 4.4: Implementation Details Provide a detailed explanation of the implementation, including design decisions, trade-offs, and any relevant technical details. Highlight any non-obvious choices, tradeoffs, potential limitations, and areas where maintainer attention is required. diff --git a/.agents/skills/transformers-upgrade/SKILL.md b/.agents/skills/transformers-upgrade/SKILL.md new file mode 100644 index 0000000000..de9065b6fa --- /dev/null +++ b/.agents/skills/transformers-upgrade/SKILL.md @@ -0,0 +1,178 @@ +--- +name: transformers-upgrade +description: "Playbook for changing vLLM's Transformers version bounds. Use when bumping the pinned or maximum Transformers version, raising the minimum Transformers version, or deleting vendored configs, processors, or compat shims that upstream Transformers now provides." +--- + +# Upgrading Transformers + +The supported range lives in `requirements/common.txt`. CI pins one exact +version in `requirements/test/*.in` and the compiled `requirements/test/*.txt`. +A version change is one of two jobs, and they usually come as a pair: + +- **Bump the pin**: a new Transformers release becomes the CI version and + the upper bound. vLLM grows compat code for it. +- **Raise the floor**: the minimum moves up. That compat code, and every + vendored config or processor upstream now ships, becomes dead weight. + Every floor raise ends with an **unvendoring pass**. + +## Bump the pin + +1. Edit the pin line in each `requirements/test/*.in` and `*.txt` and the + upper bound in `requirements/common.txt`. Past bumps edit only the pin + line in the compiled files; see `git log -- requirements/common.txt`. +2. Read the release notes for config validation changes, renamed auto-mapping + formats, and removed attributes; these are what break vLLM. +3. Search `tests/` for `min_transformers_version` and skip markers naming the + new version, and un-gate them. +4. Run the suites in [Testing](#testing) and fix what breaks. Mark every + shim with a `# Transformers >= X` comment so the next floor raise finds it. + +Done when the [Testing](#testing) suites pass on the new pin and every test +waiting on this version is un-gated. + +## Raise the floor (unvendoring pass) + +### 1. Inventory + +Inventory candidates with the floor version installed (no vLLM install needed): + +```bash +uv run --no-project --with transformers== \ + python .agents/skills/transformers-upgrade/scripts/find_upstreamed.py +``` + +It lists: + +- vendored configs whose `model_type` upstream now owns; +- `_CONFIG_REGISTRY` overrides of upstream types; +- registered processors that collide with upstream class names or models; +- version-gated code at or below the floor; +- code that reads the installed Transformers version, whatever its threshold; +- patches citing a Transformers issue or PR (with whether a referenced PR is + in the floor release) or a future minimum version. + +It finds candidates by name only. Every hit still needs a verdict. + +### 2. Port each candidate to upstream + +Every hit is deleted by default. A vendored copy is a liability: it drifts +from upstream, hides upstream fixes, and shadows the native class. Port the +vLLM model code to the upstream schema instead, renaming attribute reads and +deriving values from upstream fields. + +Fan candidates out to parallel subagents. Each one compares the vendored and +upstream classes, then greps every attribute the vLLM model code reads. +[Upstreaming hazards](#upstreaming-hazards) lists the usual fixes. + +Keep a vendored class only when upstream cannot load a checkpoint vLLM must +keep serving, and record why in the PR description. A different upstream +schema, a model-code rewrite, or a needed eval is work for the PR, not a +reason to keep. + +### 3. Prove it on real checkpoints + +For every checkpoint in `tests/models/registry.py` that uses a ported +config, dump what vLLM builds before and after the change, then diff the two. +Run both dumps; each catches what the other misses: + +- `dump_configs.py`: the resolved config, attribute by attribute. +- `dump_model.py`: the model built on the meta device with no GPU or weights. + It records ModelConfig sizes, every parameter shape, and every module's + scalar attributes (`head_dim`, `num_kv_heads`, `scaling`, ...). Missing + fields that are read with defaults only show up here. + +```bash +git worktree add --detach /vllm-base upstream/main +S=.agents/skills/transformers-upgrade/scripts +for dump in dump_configs dump_model; do + PYTHONPATH=/vllm-base .venv/bin/python $S/$dump.py before.jsonl + PYTHONPATH=$PWD .venv/bin/python $S/$dump.py after.jsonl + .venv/bin/python $S/diff_dumps.py before.jsonl after.jsonl +done +``` + +Add `--trust-remote-code` for checkpoints whose `config.json` has an +`auto_map`, and dump both ways. + +How to read the diffs: + +- **A `` config field the model reads:** this crashes or silently + falls back to a default. Port the model code to the upstream field. +- **New upstream-only defaults** (`attention_bias=False`, + `pad_token_id=None`): usually benign. A model diff of 0 confirms it. +- **Any model diff:** this is a numerics change, and the PR needs a model + eval. For example, A.X-K1 gained the checkpoint's YaRN rope scaling, which + the vendored class had been dropping; it shows up as + `layers.*.self_attn.scaling (x61)`. + +### 4. Delete + +For each deleted config: + +- Delete the file. +- Remove its entries from `_CLASS_TO_MODULE` and `__all__` in + `vllm/transformers_utils/configs/__init__.py`. +- Remove its entries from `_CONFIG_REGISTRY` in + `vllm/transformers_utils/config.py`. +- Switch every importer, including tests, to `from transformers import ...`. +- Remove the `source_file_dependencies` lines naming the file from + `.buildkite/`. + +Deleted processors follow the same steps in +`vllm/transformers_utils/processors/__init__.py`. + +Done when every line of the inventory output is deleted, or kept with a +reason that meets the bar in step 2. The PR also carries the config diff +results and evals for any changed values. + +## Upstreaming hazards + +Upstream configs are `@strict` dataclasses. Checkpoint keys upstream does not +declare survive as plain attributes. Defaults that only vLLM sets, and values +vLLM derives, vanish. + +| Hazard | Signal | Fix | +| --- | --- | --- | +| Upstream consumes legacy keys | `__post_init__` pops keys the model reads (`deepseek_v4` `compress_ratios`) | Read the upstream replacement (`layer_types`, `mlp_layer_types`, per-layer `rope_parameters`) | +| Renamed or dropped fields | Diff shows `` for a field the model reads | Read the upstream name; derive dropped flags from what implies them (`enable_moe_block` from `num_experts`) | +| Type check on the config | `ctx.get_hf_config(SomeConfig)` relies on the vendored base class | Accept the upstream class too, e.g. a `Cosmos3ProcessingInfo` overriding `get_hf_config` | +| Remote code takes over | Checkpoint `config.json` has `auto_map`; with `--trust-remote-code`, the hub class replaces the native one once vLLM stops registering its own | Check the remote class loads; if not, register the upstream class for that `model_type` | +| Read-only upstream fields | Properties or derived fields break subclasses and `hf_overrides` (`nemotron_h` `hybrid_override_pattern`) | Override the source field (`layers_block_type`) instead | +| Behavioural rewrite | Upstream derives fields differently (`diffusion_gemma` `sliding_window`) | Upstream matches the reference implementation; verify with an eval | +| Original checkpoint format | Upstream targets `-hf` conversions; vLLM serves the originals (`qwen3_asr`, `qianfan_ocr`) | Keep only if upstream cannot load the original at all | +| Processor output contract | Model code consumes keys upstream does not emit | Port the model to upstream's keys | +| Name collision | A registered vendored processor shares an upstream class name and hijacks `get_processor` (`InternVLProcessor`) | Drop the registry entry | + +Version gates from the inventory need a decision: + +- **Code** only reachable below the floor is deleted. This covers: + - comparisons against `TRANSFORMERS_VERSION`; + - branches labelled `# Transformers < X`; + - fallbacks for fields that upstream configs now always set, e.g. + `mlp_layer_types`, `per_layer_config` or legacy `layer_types` names. + + A `skipif` whose condition is now always true becomes an unconditional + `skip` with the same reason. +- **`min_transformers_version`** at or below the floor is removed. +- **`max_transformers_version`** below the floor means CI never tests that + model; report it rather than deleting it silently. +- **Patches citing a Transformers issue or PR:** + - Delete the patch when its fix is in the floor release. + - For an issue, find the PR that closed it and check that PR. A closed + issue is not a fixed one: #43329 was closed while the bug was still in + 5.16.1. + - Re-run the patched behaviour before deleting. A fix in the release does + not always cover vLLM's path: #47924 made Emu3's processor add BOS, but + vLLM tokenizes without the processor. + - Links that only explain a design choice (e.g. a float32 RoPE note) are + not patches; leave them. + +## Testing + +```bash +.venv/bin/python -m pytest tests/transformers_utils tests/config tests/tokenizers_ +.venv/bin/python -m pytest tests/models/multimodal/processing/test_common.py -k "" +``` + +Add `tests/models/test_initialization.py -k ""` where a GPU is +available, and a model eval for any model whose resolved config changed. diff --git a/.agents/skills/transformers-upgrade/agents/openai.yaml b/.agents/skills/transformers-upgrade/agents/openai.yaml new file mode 100644 index 0000000000..b881c5d7ea --- /dev/null +++ b/.agents/skills/transformers-upgrade/agents/openai.yaml @@ -0,0 +1,4 @@ +interface: + display_name: "Transformers Upgrade" + short_description: "Bump vLLM's Transformers version and delete vendored code upstream now provides." + default_prompt: "Use $transformers-upgrade to raise the minimum Transformers version and remove vendored configs, processors, and shims it makes redundant." diff --git a/.agents/skills/transformers-upgrade/scripts/diff_dumps.py b/.agents/skills/transformers-upgrade/scripts/diff_dumps.py new file mode 100644 index 0000000000..76fb2038a1 --- /dev/null +++ b/.agents/skills/transformers-upgrade/scripts/diff_dumps.py @@ -0,0 +1,51 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Diff two dump_configs.py or dump_model.py outputs, per repo. + +python diff_dumps.py before.jsonl after.jsonl +""" + +import json +import sys +from collections import Counter + +import regex as re + +IGNORED = {"transformers_version", "_name_or_path", "architectures", "auto_map"} + + +def load(path: str) -> dict[str, dict]: + with open(path) as f: + return {record["repo"]: record for record in map(json.loads, f)} + + +def diff(before: dict, after: dict, prefix: str = ""): + for key in sorted(set(before) | set(after)): + if key in IGNORED: + continue + old, new = before.get(key, ""), after.get(key, "") + if isinstance(old, dict) and isinstance(new, dict) and key != "rope_parameters": + yield from diff(old, new, f"{prefix}{key}.") + elif old != new: + yield f"{prefix}{key}", json.dumps(old), json.dumps(new) + + +def collapse(diffs) -> Counter: + """Merge diffs that differ only by layer index, e.g. `layers.*.scaling`.""" + return Counter( + (re.sub(r"\.\d+\.", ".*.", path), old, new) for path, old, new in diffs + ) + + +before, after = load(sys.argv[1]), load(sys.argv[2]) +for repo, old in before.items(): + new = after.get(repo, {"error": "missing from after"}) + if "error" in old or "error" in new: + print(f"== {repo}\n before: {old.get('error', 'ok')}") + print(f" after: {new.get('error', 'ok')}") + continue + diffs = collapse(diff(old["cfg"], new["cfg"])) + print(f"== {repo}: {old['cls']} -> {new['cls']} ({sum(diffs.values())} diffs)") + for (path, old_value, new_value), count in diffs.items(): + times = f" (x{count})" if count > 1 else "" + print(f" {path}{times}: before={old_value} after={new_value}") diff --git a/.agents/skills/transformers-upgrade/scripts/dump_configs.py b/.agents/skills/transformers-upgrade/scripts/dump_configs.py new file mode 100644 index 0000000000..6d6913fb51 --- /dev/null +++ b/.agents/skills/transformers-upgrade/scripts/dump_configs.py @@ -0,0 +1,30 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Dump the config vLLM resolves for each checkpoint, one JSON line per repo. + +Run once per vLLM tree (before and after a change), then compare the two +outputs with diff_dumps.py: + + PYTHONPATH= .venv/bin/python dump_configs.py out.jsonl ... + +Pass --trust-remote-code to resolve checkpoints the way users who set it do. +""" + +import json +import sys + +from vllm.transformers_utils.config import get_config + +out, *args = sys.argv[1:] +trust_remote_code = "--trust-remote-code" in args +with open(out, "w") as f: + for repo in (arg for arg in args if arg != "--trust-remote-code"): + try: + config = get_config(repo, trust_remote_code=trust_remote_code) + cls = f"{type(config).__module__}.{type(config).__name__}" + record = {"repo": repo, "cls": cls, "cfg": config.to_dict()} + except Exception as e: + error = f"{type(e).__name__}: {e}".splitlines()[0] + record = {"repo": repo, "error": error} + f.write(json.dumps(record, default=str, sort_keys=True) + "\n") + f.flush() diff --git a/.agents/skills/transformers-upgrade/scripts/dump_model.py b/.agents/skills/transformers-upgrade/scripts/dump_model.py new file mode 100644 index 0000000000..47efbe873f --- /dev/null +++ b/.agents/skills/transformers-upgrade/scripts/dump_model.py @@ -0,0 +1,114 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Build each model on the meta device and dump what vLLM constructs. + +Records ModelConfig-derived sizes, every parameter shape, and scalar +attributes of every module (head_dim, num_kv_heads, sliding_window, ...). +Needs no GPU and no weights. Run once per vLLM tree, then compare the two +outputs with diff_dumps.py: + + PYTHONPATH= .venv/bin/python dump_model.py out.jsonl ... + +Pass --trust-remote-code to resolve checkpoints the way users who set it do. +""" + +import json +import socket +import sys + +import torch + +from vllm.config import set_current_vllm_config +from vllm.config.vllm import VllmConfig +from vllm.distributed import ( + init_distributed_environment, + initialize_model_parallel, + model_parallel_is_initialized, +) +from vllm.engine.arg_utils import EngineArgs +from vllm.model_executor.model_loader.utils import initialize_model +from vllm.utils.torch_utils import set_default_torch_dtype + +# Construction only: skip model runner checks that need a GPU or Triton +VllmConfig._get_v1_model_runner_unsupported_features = lambda self: [] # type: ignore[method-assign] + +# Model code that moves submodules to a device must leave them on meta +_module_to = torch.nn.Module.to + + +def _meta_safe_to(self, *args, **kwargs): + kwargs.pop("device", None) + args = tuple(a for a in args if not isinstance(a, (torch.device, str, int))) + return _module_to(self, *args, **kwargs) + + +torch.nn.Module.to = _meta_safe_to # type: ignore[method-assign] + +SCALARS = (bool, int, float, str) + + +def module_attrs(model: torch.nn.Module) -> dict[str, dict]: + attrs = {} + for name, module in model.named_modules(): + scalars = { + key: value + for key, value in vars(module).items() + if not key.startswith("_") and isinstance(value, SCALARS) + } + if scalars: + attrs[name or ""] = scalars + return attrs + + +def dump(repo: str, trust_remote_code: bool) -> dict: + args = EngineArgs( + model=repo, + load_format="dummy", + enforce_eager=True, + max_model_len=4096, + trust_remote_code=trust_remote_code, + ) + vllm_config = args.create_engine_config() + model_config = vllm_config.model_config + with set_current_vllm_config(vllm_config): + if not model_parallel_is_initialized(): + with socket.socket() as sock: + sock.bind(("", 0)) + port = sock.getsockname()[1] + init_distributed_environment(1, 0, f"tcp://127.0.0.1:{port}", 0, "gloo") + initialize_model_parallel(1, 1) + with set_default_torch_dtype(model_config.dtype), torch.device("meta"): + model = initialize_model(vllm_config=vllm_config) + return { + "repo": repo, + "cls": type(model).__name__, + "cfg": { + "model_config": { + "head_size": model_config.get_head_size(), + "total_num_kv_heads": model_config.get_total_num_kv_heads(), + "sliding_window": model_config.get_sliding_window(), + "num_layers": model_config.get_num_layers(vllm_config.parallel_config), + }, + "params": {n: list(p.shape) for n, p in model.named_parameters()}, + "modules": module_attrs(model), + }, + } + + +def main() -> None: + out, *args = sys.argv[1:] + trust_remote_code = "--trust-remote-code" in args + repos = [arg for arg in args if arg != "--trust-remote-code"] + with open(out, "w") as f: + for repo in repos: + try: + record = dump(repo, trust_remote_code) + except Exception as e: + error = f"{type(e).__name__}: {e}".splitlines()[0] + record = {"repo": repo, "error": error} + f.write(json.dumps(record, default=str, sort_keys=True) + "\n") + f.flush() + + +if __name__ == "__main__": + main() diff --git a/.agents/skills/transformers-upgrade/scripts/find_upstreamed.py b/.agents/skills/transformers-upgrade/scripts/find_upstreamed.py new file mode 100644 index 0000000000..4cd73c29cb --- /dev/null +++ b/.agents/skills/transformers-upgrade/scripts/find_upstreamed.py @@ -0,0 +1,147 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""List vendored vLLM code that the installed Transformers may make redundant. + +Run with the Transformers version under test, from the vLLM repo root: + + uv run --no-project --with transformers== \ + python .agents/skills/transformers-upgrade/scripts/find_upstreamed.py + +Imports nothing from vLLM, so no vLLM install is needed. Uses the `gh` CLI, if +available, to check whether referenced Transformers PRs are in this release. +""" + +import json +import subprocess +import sys +from pathlib import Path + +import regex as re +import transformers +from packaging.version import Version +from transformers.models.auto.configuration_auto import CONFIG_MAPPING_NAMES +from transformers.models.auto.processing_auto import PROCESSOR_MAPPING_NAMES + +ROOT = Path(sys.argv[1] if len(sys.argv) > 1 else ".") +FLOOR = Version(transformers.__version__) +UPSTREAM_NAMES = set(dir(transformers)) + +MODEL_TYPE = re.compile(r'model_type\s*(?::\s*str\s*)?=\s*"([^"]+)"') +REGISTRY_KEY = re.compile(r'^\s+(?:\*\*\{)?"?([\w-]+)"?(?:=|": )"(\w+)"', re.M) +CLASS_KEY = re.compile(r'^\s+"(\w+)": ', re.M) +UPSTREAM_REF = re.compile( + r"huggingface/transformers(?:/(?:issues|pull)/|#)(\d+)" + r"|(?:once|until|when) [Tt]ransformers [\d.]+ is the minimum" +) +VERSIONED = re.compile( + r"(?:min_transformers_version|max_transformers_version|check_version|Version)" + r'\(?\s*=?\s*"([45]\.\d+(?:\.\d+)?)[^"]*"' + r"|[Tt]ransformers\s*(?:v|[<>]=?\s*)([45]\.\d+(?:\.\d+)?|4)\b" +) +READS_VERSION = re.compile( + r"transformers import __version__|transformers\.__version__" + r"|version\([\"']transformers[\"']\)|TRANSFORMERS_VERSION|check_version\(" +) + + +def section(title: str) -> None: + print(f"\n## {title}") + + +def vendored_configs() -> None: + section(f"Vendored configs whose model_type is upstream in {FLOOR}") + files = sorted(ROOT.glob("vllm/transformers_utils/configs/*.py")) + files += sorted(ROOT.glob("vllm/models/*/config*.py")) + files += sorted(ROOT.glob("vllm/models/*/configs/*.py")) + for path in files: + types = sorted(set(MODEL_TYPE.findall(path.read_text()))) + hits = [t for t in types if t in CONFIG_MAPPING_NAMES] + if hits: + upstream = ", ".join(f"{t} -> {CONFIG_MAPPING_NAMES[t]}" for t in hits) + print(f"- {path.relative_to(ROOT)}: {upstream}") + + registry = (ROOT / "vllm/transformers_utils/config.py").read_text() + start = registry.index("_CONFIG_REGISTRY") + body = registry[start : registry.index("\n)\n", start)] + section("_CONFIG_REGISTRY entries that override an upstream model_type") + for model_type, cls in REGISTRY_KEY.findall(body): + if model_type in CONFIG_MAPPING_NAMES: + print(f"- {model_type}: vLLM {cls} over {CONFIG_MAPPING_NAMES[model_type]}") + + +def vendored_processors() -> None: + init = (ROOT / "vllm/transformers_utils/processors/__init__.py").read_text() + names = sorted(set(CLASS_KEY.findall(init))) + section("Registered processor names that collide with upstream classes") + for name in names: + if name in UPSTREAM_NAMES: + print(f"- {name}") + section("Processor files named after a model_type with an upstream processor") + for path in sorted(ROOT.glob("vllm/transformers_utils/processors/*.py")): + if path.stem in PROCESSOR_MAPPING_NAMES: + upstream = PROCESSOR_MAPPING_NAMES[path.stem] + print(f"- {path.relative_to(ROOT)} -> {upstream}") + + +def version_gates() -> None: + files = sorted([*ROOT.glob("vllm/**/*.py"), *ROOT.glob("tests/**/*.py")]) + section(f"Version-gated code at or below {FLOOR} (review each)") + for path in files: + for lineno, line in enumerate(path.read_text().splitlines(), 1): + for match in VERSIONED.finditer(line): + version = Version(match.group(1) or match.group(2)) + if version <= FLOOR: + print(f"- {path.relative_to(ROOT)}:{lineno}: {line.strip()}") + break + section("Code that reads the installed Transformers version (review each)") + for path in files: + for lineno, line in enumerate(path.read_text().splitlines(), 1): + if READS_VERSION.search(line): + print(f"- {path.relative_to(ROOT)}:{lineno}: {line.strip()}") + + +def upstream_status(number: str) -> str: + """Whether Transformers issue/PR `number` is resolved in this release.""" + + def gh(path: str) -> dict | None: + try: + out = subprocess.run(["gh", "api", path], capture_output=True, text=True) + except FileNotFoundError: + return None + return json.loads(out.stdout) if out.returncode == 0 else None + + repo = "repos/huggingface/transformers" + if (issue := gh(f"{repo}/issues/{number}")) is None: + return "status unknown" + if "pull_request" not in issue: + return f"issue {issue['state']}, check which PR fixed it" + pr = gh(f"{repo}/pulls/{number}") + if pr is None or not pr["merged_at"]: + return "PR not merged" + compare = gh(f"{repo}/compare/v{FLOOR}...{pr['merge_commit_sha']}") + included = compare and compare["status"] in ("behind", "identical") + return f"PR merged, {'in' if included else 'not in'} v{FLOOR}" + + +def upstream_refs() -> None: + section("Patches referencing Transformers issues/PRs or a future minimum") + files = sorted([*ROOT.glob("vllm/**/*.py"), *ROOT.glob("tests/**/*.py")]) + statuses: dict[str, str] = {} + for path in files: + for lineno, line in enumerate(path.read_text().splitlines(), 1): + if (match := UPSTREAM_REF.search(line)) is None: + continue + note = "" + if number := match.group(1): + if number not in statuses: + statuses[number] = upstream_status(number) + note = f" [#{number}: {statuses[number]}]" + print(f"- {path.relative_to(ROOT)}:{lineno}:{note} {line.strip()}") + + +if __name__ == "__main__": + print(f"# Transformers {FLOOR}") + vendored_configs() + vendored_processors() + version_gates() + upstream_refs() diff --git a/.buildkite/hardware_tests/cpu.yaml b/.buildkite/hardware_tests/cpu.yaml index d54197f836..a29b44ed3c 100644 --- a/.buildkite/hardware_tests/cpu.yaml +++ b/.buildkite/hardware_tests/cpu.yaml @@ -20,8 +20,10 @@ steps: - tests/kernels/moe/test_cpu_fused_moe.py - tests/kernels/moe/test_cpu_quant_fused_moe.py - tests/kernels/moe/test_zen_cpu_int8_moe.py + - tests/test_zentorch_version_pin.py - vllm/model_executor/layers/fused_moe/experts/cpu_moe.py - tests/kernels/test_onednn.py + - tests/kernels/test_cpu_fused_sampling.py - tests/kernels/test_awq_int4_to_int8.py - tests/kernels/quantization/test_cpu_fp8_scaled_mm.py - tests/kernels/quantization/test_zen_da8w4.py @@ -45,8 +47,10 @@ steps: tests/kernels/moe/test_cpu_fused_moe.py \ tests/kernels/moe/test_cpu_quant_fused_moe.py \ tests/kernels/moe/test_zen_cpu_int8_moe.py \ + tests/test_zentorch_version_pin.py \ tests/kernels/mamba/test_cpu_short_conv.py \ tests/kernels/test_onednn.py \ + tests/kernels/test_cpu_fused_sampling.py \ tests/kernels/test_awq_int4_to_int8.py \ tests/kernels/quantization/test_cpu_fp8_scaled_mm.py \ tests/kernels/quantization/test_zen_da8w4.py \ @@ -85,7 +89,7 @@ steps: - | bash .buildkite/scripts/hardware_ci/run-cpu-test.sh 30m " pytest -x -v -s \ - tests/v1/worker/test_gpu_autoregressive_speculator.py \ + tests/v1/worker/test_gpu_target_dependent_ar_speculator.py \ tests/v1/worker/test_gpu_extract_hidden_states_speculator.py \ tests/v1/worker/test_gpu_batch_ordering.py \ tests/v1/worker/test_gpu_rejection_sampler_chunking.py \ @@ -228,3 +232,57 @@ steps: "$$BUILDKITE_PARALLEL_JOB" "$REGISTRY/$REPO:$BUILDKITE_COMMIT-arm64-cpu" parallelism: 3 + +- label: ":docker: Build Zen CPU image" + key: zen-cpu-image-build + # Reuse the shared `:-cpu` image that image-build-cpu publishes + # to public ECR, so Dockerfile.zen just layers on top instead of recompiling + # vLLM from source. If the pull fails, the build script still falls back to + # building the CPU base from source. + depends_on: + - image-build-cpu + soft_fail: true + agents: + queue: amd-cpu + source_file_dependencies: + - .buildkite/hardware_tests/cpu.yaml + - .buildkite/image_build/image_build_zen_cpu.sh + - docker/Dockerfile.zen + - docker/Dockerfile.cpu + - requirements/cpu.txt + - requirements/build/cpu.txt + - requirements/test/cpu.txt + - cmake/cpu_extension.cmake + - vllm/model_executor/layers/utils.py + - vllm/model_executor/kernels/linear/zentorch_utils.py + - vllm/model_executor/kernels/linear/scaled_mm/zentorch.py + - vllm/model_executor/kernels/linear/mixed_precision/zentorch.py + - vllm/platforms/cpu.py + - vllm/platforms/zen_cpu.py + - vllm/platforms/__init__.py + - tests/model_executor/test_cpu_unquantized_gemm_dispatch.py + - tests/test_zen_cpu_platform_detection.py + - tests/kernels/moe/test_zen_cpu_fused_moe.py + - tests/kernels/moe/test_zen_cpu_int8_moe.py + - vllm/v1/attention/backends/zentorch_sdpa.py + - tests/test_zentorch_version_pin.py + - vllm/v1/attention/backends/mla/prefill/zen_cpu_sdpa.py + - vllm/model_executor/layers/fused_moe/experts/zentorch_moe.py + - tests/kernels/quantization/test_zen_da8w4.py + commands: + - .buildkite/image_build/image_build_zen_cpu.sh "$REGISTRY" "$REPO" "$BUILDKITE_COMMIT" + env: + DOCKER_BUILDKIT: "1" + # Install zentorch directly (if-branch in Dockerfile.zen) instead of + # `uv pip install "vllm[zen]"`, which forces uv to re-resolve vLLM's full + # dependency closure and chokes on its transitive triton-cpu URL dependency. + # Keep this pin in sync with the `zen` extra in setup.py. + ZENTORCH_VERSION: "2.13.0.1" + retry: + automatic: + - exit_status: 1 # Transient Docker/BuildKit failure + limit: 1 + - exit_status: -1 # Agent was lost + limit: 2 + - exit_status: -10 # Agent was lost + limit: 2 diff --git a/.buildkite/hardware_tests/gh200.yaml b/.buildkite/hardware_tests/gh200.yaml index 2d42aa4541..54f2e27632 100644 --- a/.buildkite/hardware_tests/gh200.yaml +++ b/.buildkite/hardware_tests/gh200.yaml @@ -2,10 +2,15 @@ group: Hardware steps: - label: "GH200 Test" key: gh200-test + timeout_in_minutes: 30 soft_fail: true device: gh200 - no_plugin: true optional: true - commands: - - nvidia-smi - - bash .buildkite/scripts/hardware_ci/run-gh200-test.sh + working_dir: "/vllm-workspace" + depends_on: + - arm64-image-build + env: + VLLM_WORKER_MULTIPROC_METHOD: "spawn" + commands: + - nvidia-smi + - python3 examples/basic/offline_inference/generate.py --model meta-llama/Llama-3.2-1B diff --git a/.buildkite/image_build/image_build_torch_nightly.sh b/.buildkite/image_build/image_build_torch_nightly.sh index 56c7116e14..4d978b5299 100755 --- a/.buildkite/image_build/image_build_torch_nightly.sh +++ b/.buildkite/image_build/image_build_torch_nightly.sh @@ -61,6 +61,8 @@ NIGHTLY_UBUNTU_VERSION="24.04" NIGHTLY_FINAL_BASE_IMAGE="nvidia/cuda:${NIGHTLY_CUDA_VERSION}-base-ubuntu${NIGHTLY_UBUNTU_VERSION}" echo "--- :docker: Building torch nightly image (CUDA ${NIGHTLY_CUDA_VERSION})" +# Provenance labels match the bake `_labels` target in docker/docker-bake.hcl; +# the initialized-snapshot E2E refuses images whose commit labels differ. docker buildx build --file docker/Dockerfile \ --build-arg max_jobs=16 \ --build-arg buildkite_commit="$BUILDKITE_COMMIT" \ @@ -70,6 +72,10 @@ docker buildx build --file docker/Dockerfile \ --build-arg BUILD_BASE_IMAGE="${NIGHTLY_BUILD_BASE_IMAGE}" \ --build-arg FINAL_BASE_IMAGE="${NIGHTLY_FINAL_BASE_IMAGE}" \ --build-arg torch_cuda_arch_list="8.0 8.9 9.0 10.0 12.0" \ + --label "org.opencontainers.image.revision=${BUILDKITE_COMMIT}" \ + --label "ai.vllm.build.commit=${BUILDKITE_COMMIT}" \ + --label "ai.vllm.build.pipeline=${BUILDKITE_PIPELINE_SLUG:-local}" \ + --label "ai.vllm.build.url=${BUILDKITE_BUILD_URL:-}" \ --tag "$IMAGE_TAG" \ --push \ --target test \ diff --git a/.buildkite/image_build/image_build_zen_cpu.sh b/.buildkite/image_build/image_build_zen_cpu.sh new file mode 100755 index 0000000000..3e42e38a8d --- /dev/null +++ b/.buildkite/image_build/image_build_zen_cpu.sh @@ -0,0 +1,75 @@ +#!/bin/bash +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +# +# Build the AMD Zen CPU image (vLLM + zentorch) by layering docker/Dockerfile.zen +# on top of a CPU base image that already has vLLM installed. +# +# To avoid recompiling vLLM from source, prefer reusing the shared CPU image that +# the image-build-cpu step already builds and publishes for this commit +# (`:-cpu`). It lives in public ECR, which allows anonymous pulls, +# so no registry credentials are required. If the pull fails (e.g. the image +# isn't published yet), fall back to building the base from docker/Dockerfile.cpu. +# +# The zen image is not pushed to a registry. +# +# See docker/Dockerfile.zen for the build workflow this mirrors. +set -euo pipefail + +if [[ $# -lt 3 ]]; then + echo "Usage: $0 " + exit 1 +fi + +REGISTRY=$1 +REPO=$2 +BUILDKITE_COMMIT=$3 + +image_repo() { + if [[ -n "${REGISTRY}" ]]; then + printf '%s/%s' "${REGISTRY}" "${REPO}" + else + printf '%s' "${REPO}" + fi +} + +IMAGE_REPO="$(image_repo)" + +# The shared CPU image published by image-build-cpu (public ECR, anonymous pull). +SHARED_CPU_IMAGE="" +if [[ -n "${REGISTRY}" ]]; then + SHARED_CPU_IMAGE="$REGISTRY/$REPO:$BUILDKITE_COMMIT-cpu" +fi +FALLBACK_BASE_IMAGE="zen-cpu-base:$BUILDKITE_COMMIT" +IMAGE="$IMAGE_REPO:$BUILDKITE_COMMIT-zen-cpu" + +# ZENTORCH_VERSION is optional; when unset the Dockerfile falls back to +# installing zentorch via `vllm[zen]`. +ZENTORCH_VERSION=${ZENTORCH_VERSION:-} + +# Step 1: obtain the CPU base image that Dockerfile.zen layers on. Prefer pulling +# the published `-cpu` image; fall back to building it from source if unavailable. +if [[ -n "${SHARED_CPU_IMAGE}" ]] && docker pull "$SHARED_CPU_IMAGE"; then + echo "--- :docker: Using published CPU image as base: $SHARED_CPU_IMAGE" + BASE_IMAGE="$SHARED_CPU_IMAGE" +else + echo "--- :docker: Published CPU image unavailable; building base from source" + docker build --file docker/Dockerfile.cpu \ + --platform linux/amd64 \ + --build-arg buildkite_commit="$BUILDKITE_COMMIT" \ + --build-arg VLLM_CPU_X86=true \ + --tag "$FALLBACK_BASE_IMAGE" \ + --target vllm-openai \ + --progress plain . + BASE_IMAGE="$FALLBACK_BASE_IMAGE" +fi + +# Step 2: build the zen test image on top of the CPU base. +echo "--- :docker: Building Zen test image" +docker build --file docker/Dockerfile.zen \ + --platform linux/amd64 \ + --build-arg BASE_IMAGE="$BASE_IMAGE" \ + ${ZENTORCH_VERSION:+--build-arg ZENTORCH_VERSION="$ZENTORCH_VERSION"} \ + --tag "$IMAGE" \ + --target vllm-zen-test \ + --progress plain . diff --git a/.buildkite/intel_jobs/entrypoints_intel.yaml b/.buildkite/intel_jobs/entrypoints_intel.yaml index 7e823e0245..96cbd2002a 100644 --- a/.buildkite/intel_jobs/entrypoints_intel.yaml +++ b/.buildkite/intel_jobs/entrypoints_intel.yaml @@ -19,14 +19,14 @@ steps: source_file_dependencies: - vllm/entrypoints - tests/entrypoints/unit_tests - - tests/entrypoints/weight_transfer + - tests/entrypoints/rl/weight_transfer - tests/entrypoints/launchers commands: - >- bash .buildkite/scripts/hardware_ci/run-intel-test.sh 'cd tests && pytest -v -s entrypoints/unit_tests && - pytest -v -s entrypoints/weight_transfer' + pytest -v -s entrypoints/rl/weight_transfer' - label: ":intel: (BMG) Entrypoints Integration (LLM)" key: entrypoints-integration-llm-intel @@ -71,14 +71,17 @@ steps: - vllm/ - "!vllm/distributed/kv_transfer/" - tests/entrypoints/serve + - tests/entrypoints/rl commands: - >- bash .buildkite/scripts/hardware_ci/run-intel-test.sh 'export VLLM_WORKER_MULTIPROC_METHOD=spawn && cd tests && - pytest -v -s entrypoints/serve --ignore=entrypoints/serve/dev/rpc - --ignore=entrypoints/serve/dev/rlhf/test_weight_operation_metrics.py - --deselect="tests/entrypoints/serve/dev/rlhf/state_transitions/test_pause_resume.py::TestPauseResume::test_clear_cache_preserves_output_and_controls_prefix_cache[MRV1]" && + pytest -v -s entrypoints/serve --ignore=entrypoints/serve/dev/rpc && + pytest -v -s entrypoints/rl --ignore=entrypoints/rl/weight_transfer + --ignore=entrypoints/rl/return_routed_experts/test_openai_api.py + --ignore=entrypoints/rl/weight_transfer_metrics + --deselect="tests/entrypoints/rl/state_transitions/test_pause_resume.py::TestPauseResume::test_clear_cache_preserves_output_and_controls_prefix_cache[MRV1]" && PYTHONPATH=/workspace/vllm pytest -v -s entrypoints/serve/dev/rpc' - label: ":intel: (BMG) Entrypoints Integration (API Server Scale Out)" diff --git a/.buildkite/intel_jobs/lora_intel.yaml b/.buildkite/intel_jobs/lora_intel.yaml index 2cecbf18f7..885e4b5466 100644 --- a/.buildkite/intel_jobs/lora_intel.yaml +++ b/.buildkite/intel_jobs/lora_intel.yaml @@ -9,7 +9,7 @@ steps: agent_tags: label: production gpu: 1+ - mem: 24+ + mem: 16+ no_plugin: true working_dir: "." env: @@ -33,7 +33,6 @@ steps: pytest -v -s lora/test_peft_helper.py && pytest -v -s lora/test_resolver.py && pytest -v -s lora/test_utils.py && - pytest -v -s lora/test_add_lora.py && pytest -v -s lora/test_worker.py' - label: LoRA Fused/MoE Kernels @@ -120,7 +119,7 @@ steps: agent_tags: label: production gpu: 2+ - mem: 24+ + mem: 16+ no_plugin: true working_dir: "." env: @@ -138,8 +137,7 @@ steps: pytest -v -s lora/test_quant_model.py --deselect="tests/lora/test_quant_model.py::test_quant_model_lora[model0]" --deselect="tests/lora/test_quant_model.py::test_quant_model_lora[model1]" --deselect="tests/lora/test_quant_model.py::test_quant_model_tp_equality[model0]" && pytest -v -s lora/test_transformers_model.py && pytest -v -s lora/test_chatglm3_tp.py && - pytest -v -s lora/test_llama_tp.py::test_llama_lora && - pytest -s -v lora/test_minicpmv_tp.py' + pytest -v -s lora/test_llama_tp.py::test_llama_lora' - label: LoRA Multimodal key: lora-multimodal @@ -148,6 +146,30 @@ steps: agent_tags: label: production gpu: 1+ + mem: 16+ + no_plugin: true + working_dir: "." + env: + REGISTRY: "public.ecr.aws/q9t5s3a7" + REPO: "vllm-ci-test-repo" + VLLM_TEST_DEVICE: "xpu" + source_file_dependencies: + - vllm/lora + - tests/lora + commands: + - >- + bash .buildkite/scripts/hardware_ci/run-intel-test.sh + 'cd tests && + export VLLM_WORKER_MULTIPROC_METHOD=spawn && + pytest -v -s lora/test_whisper.py' + +- label: LoRA Add + Multimodal + key: lora-add-multimodal + timeout_in_minutes: 45 + device: intel_gpu + agent_tags: + label: production + gpu: 2+ mem: 24+ no_plugin: true working_dir: "." @@ -163,5 +185,6 @@ steps: bash .buildkite/scripts/hardware_ci/run-intel-test.sh 'cd tests && export VLLM_WORKER_MULTIPROC_METHOD=spawn && - pytest -v -s lora/test_default_mm_loras.py && - pytest -v -s lora/test_whisper.py' + pytest -v -s lora/test_add_lora.py && + pytest -v -s lora/test_default_mm_loras.py && + pytest -s -v lora/test_minicpmv_tp.py' diff --git a/.buildkite/intel_jobs/misc_intel.yaml b/.buildkite/intel_jobs/misc_intel.yaml index 5fa4d833d7..b7ae9f9078 100644 --- a/.buildkite/intel_jobs/misc_intel.yaml +++ b/.buildkite/intel_jobs/misc_intel.yaml @@ -10,7 +10,7 @@ steps: agent_tags: label: production gpu: 2+ - mem: 24+ + mem: 16+ no_plugin: true working_dir: "." env: @@ -34,6 +34,40 @@ steps: key: v1-core-kv-metrics-intel timeout_in_minutes: 45 device: intel_gpu + agent_tags: + label: production + gpu: 1+ + mem: 16+ + no_plugin: true + working_dir: "." + env: + REGISTRY: "public.ecr.aws/q9t5s3a7" + REPO: "vllm-ci-test-repo" + VLLM_TEST_DEVICE: "xpu" + source_file_dependencies: + - vllm/ + - tests/v1/core + - tests/v1/executor + - tests/v1/kv_offload + - tests/v1/worker + - tests/v1/kv_connector/unit + - tests/v1/ec_connector/unit + - tests/v1/metrics + - tests/entrypoints/openai/correctness/test_lmeval.py + commands: + - >- + bash .buildkite/scripts/hardware_ci/run-intel-test.sh + 'pip install -r requirements/kv_connectors.txt && + export VLLM_WORKER_MULTIPROC_METHOD=spawn && + cd tests && + pytest -v -s v1/executor && + VLLM_BATCH_INVARIANT=1 pytest -v -s -m "not cpu_test" v1/ec_connector/unit + --ignore=v1/ec_connector/unit/test_ec_cpu_connector.py' + +- label: EC CPU Connector (XPU) + key: ec-cpu-connector-xpu + timeout_in_minutes: 45 + device: intel_gpu agent_tags: label: production gpu: 1+ @@ -60,8 +94,7 @@ steps: 'pip install -r requirements/kv_connectors.txt && export VLLM_WORKER_MULTIPROC_METHOD=spawn && cd tests && - pytest -v -s v1/executor && - VLLM_BATCH_INVARIANT=1 pytest -v -s -m "not cpu_test" v1/ec_connector/unit' + pytest -v -s v1/ec_connector/unit/test_ec_cpu_connector.py' - label: V1 Sample + Logits key: v1-sample-logits-intel @@ -70,7 +103,7 @@ steps: agent_tags: label: production gpu: 1+ - mem: 24+ + mem: 16+ no_plugin: true working_dir: "." env: @@ -265,7 +298,7 @@ steps: agent_tags: label: production gpu: 1+ - mem: 24+ + mem: 16+ no_plugin: true working_dir: "." env: @@ -293,10 +326,47 @@ steps: - >- bash .buildkite/scripts/hardware_ci/run-intel-test.sh 'cd tests && - pytest -v -s detokenizer && + pytest -v -s detokenizer --ignore=detokenizer/test_stop_strings.py && pytest -v -s -m "not cpu_test" ./multimodal && pytest -v -s utils_ --ignore=utils_/test_mem_utils.py' +- label: Detokenizer Stop Strings (XPU) + key: detokenizer-stop-strings-xpu + timeout_in_minutes: 45 + device: intel_gpu + agent_tags: + label: production + gpu: 1+ + mem: 24+ + no_plugin: true + working_dir: "." + env: + REGISTRY: "public.ecr.aws/q9t5s3a7" + REPO: "vllm-ci-test-repo" + VLLM_TEST_DEVICE: "xpu" + source_file_dependencies: + - vllm/assets/ + - vllm/config/ + - vllm/distributed/ + - vllm/engine/ + - vllm/inputs/ + - vllm/model_executor/ + - vllm/multimodal/ + - vllm/platforms/ + - vllm/sampling_params.py + - vllm/tokenizers/ + - vllm/transformers_utils/ + - vllm/utils/ + - vllm/v1/ + - tests/detokenizer + - tests/multimodal + - tests/utils_ + commands: + - >- + bash .buildkite/scripts/hardware_ci/run-intel-test.sh + 'cd tests && + pytest -v -s detokenizer/test_stop_strings.py' + - label: Fusion Unit Tests key: fusion-unit-tests timeout_in_minutes: 30 diff --git a/.buildkite/intel_jobs/model_executor_intel.yaml b/.buildkite/intel_jobs/model_executor_intel.yaml index 98e4b0ff45..d641c93e5a 100644 --- a/.buildkite/intel_jobs/model_executor_intel.yaml +++ b/.buildkite/intel_jobs/model_executor_intel.yaml @@ -6,6 +6,34 @@ steps: key: model-executor-intel timeout_in_minutes: 45 device: intel_gpu + agent_tags: + label: production + gpu: 1+ + mem: 16+ + no_plugin: true + working_dir: "." + env: + REGISTRY: "public.ecr.aws/q9t5s3a7" + REPO: "vllm-ci-test-repo" + VLLM_TEST_DEVICE: "xpu" + source_file_dependencies: + - vllm/engine/arg_utils.py + - vllm/config/model.py + - vllm/model_executor + - tests/model_executor + commands: + - >- + bash .buildkite/scripts/hardware_ci/run-intel-test.sh + 'apt-get update && apt-get install -y curl libsodium23 && + export VLLM_WORKER_MULTIPROC_METHOD=spawn && + export PYTHONFAULTHANDLER=1 && + cd tests && + pytest -v -s model_executor -m "not slow_test" --ignore="model_executor/layers/test_rocm_unquantized_gemm.py" --ignore="model_executor/model_loader/tensorizer_loader/test_tensorizer.py" --deselect="tests/model_executor/model_loader/test_reload.py::test_kv_scale_reload"' + +- label: Tensorizer Model Loader Tests (XPU) + key: tensorizer-model-loader-tests-xpu + timeout_in_minutes: 45 + device: intel_gpu agent_tags: label: production gpu: 1+ @@ -28,13 +56,4 @@ steps: export VLLM_WORKER_MULTIPROC_METHOD=spawn && export PYTHONFAULTHANDLER=1 && cd tests && - pytest -v -s model_executor -m "not slow_test" --ignore="model_executor/layers/test_rocm_unquantized_gemm.py" --deselect="tests/model_executor/model_loader/test_reload.py::test_kv_scale_reload" - --deselect="tests/model_executor/layers/test_mm_input_norm.py::TestFusedMMInputNormModule::test_matches_reference_above_cudnn_grid_limit" - --deselect="tests/model_executor/layers/test_mm_input_norm.py::TestFusedMMInputNormDtypes::test_dtype_combinations[in_dtype0-out_dtype0]" - --deselect="tests/model_executor/layers/test_mm_input_norm.py::TestFusedMMInputNormDtypes::test_dtype_combinations[in_dtype1-out_dtype1]" - --deselect="tests/model_executor/layers/test_mm_input_norm.py::TestFusedMMInputNormDtypes::test_dtype_combinations[in_dtype2-out_dtype2]" - --deselect="tests/model_executor/layers/test_mm_input_norm.py::TestFusedMMInputNormDtypes::test_dtype_combinations[in_dtype3-out_dtype3]" - --deselect="tests/model_executor/layers/test_mm_input_norm.py::TestFusedMMInputNormDtypes::test_dtype_combinations[in_dtype4-out_dtype4]" - --deselect="tests/model_executor/layers/test_mm_input_norm.py::TestFusedMMInputNormDtypes::test_dtype_combinations[in_dtype5-out_dtype5]" - --deselect="tests/model_executor/layers/test_mm_input_norm.py::TestFusedMMInputNormShapes" - --deselect="tests/model_executor/layers/test_mm_input_norm.py::TestFusedMMInputNormInputHandling"' + pytest -v -s model_executor/model_loader/tensorizer_loader/test_tensorizer.py -m "not slow_test"' diff --git a/.buildkite/intel_jobs/models_multimodal_intel.yaml b/.buildkite/intel_jobs/models_multimodal_intel.yaml index eb36e0c02e..89654ec3ae 100644 --- a/.buildkite/intel_jobs/models_multimodal_intel.yaml +++ b/.buildkite/intel_jobs/models_multimodal_intel.yaml @@ -9,7 +9,7 @@ steps: agent_tags: label: production gpu: 1+ - mem: 24+ + mem: 16+ no_plugin: true working_dir: "." env: @@ -33,7 +33,7 @@ steps: agent_tags: label: production gpu: 1+ - mem: 24+ + mem: 16+ no_plugin: true working_dir: "." env: @@ -72,7 +72,9 @@ steps: - >- bash .buildkite/scripts/hardware_ci/run-intel-test.sh 'cd tests && - pytest -v -s models/multimodal/generation/test_common.py -m core_model -k "not qwen2 and not qwen3 and not gemma"' + pytest -v -s models/multimodal/generation/test_common.py -m core_model -k "not qwen2 and not qwen3 and not gemma" && + pytest -v -s models/multimodal/generation/test_gemma_ragged_audio.py -m core_model + pytest -v -s models/multimodal/generation/test_qwen2_vl.py -m core_model' - label: "Multi-Modal Models (Standard) 4: other + whisper" key: multi-modal-models-standard-4-other-whisper @@ -81,7 +83,7 @@ steps: agent_tags: label: production gpu: 1+ - mem: 24+ + mem: 16+ no_plugin: true working_dir: "." env: @@ -95,7 +97,7 @@ steps: - >- bash .buildkite/scripts/hardware_ci/run-intel-test.sh 'cd tests && - pytest -v -s models/multimodal -m core_model --ignore models/multimodal/generation/test_common.py --ignore models/multimodal/generation/test_ultravox.py --ignore models/multimodal/generation/test_qwen2_5_vl.py --ignore models/multimodal/generation/test_qwen2_vl.py --ignore models/multimodal/generation/test_whisper.py --ignore models/multimodal/generation/test_memory_leak.py --ignore models/multimodal/processing' + pytest -v -s models/multimodal -m core_model --ignore models/multimodal/generation/test_common.py --ignore models/multimodal/generation/test_ultravox.py --ignore models/multimodal/generation/test_qwen2_5_vl.py --ignore models/multimodal/generation/test_qwen2_vl.py --ignore models/multimodal/generation/test_whisper.py --ignore models/multimodal/generation/test_gemma_ragged_audio.py --ignore models/multimodal/generation/test_memory_leak.py --ignore models/multimodal/processing' - label: Multi-Modal Processor # 44min key: multi-modal-processor @@ -131,7 +133,7 @@ steps: agent_tags: label: production gpu: 1+ - mem: 24+ + mem: 16+ no_plugin: true working_dir: "." env: diff --git a/.buildkite/intel_jobs/quantization.yaml b/.buildkite/intel_jobs/quantization.yaml index 8f1dfebe50..af1b23b8a0 100644 --- a/.buildkite/intel_jobs/quantization.yaml +++ b/.buildkite/intel_jobs/quantization.yaml @@ -25,5 +25,7 @@ steps: - >- bash .buildkite/scripts/hardware_ci/run-intel-test.sh 'VLLM_TEST_FORCE_LOAD_FORMAT=auto pytest -v -s tests/quantization/test_per_token_kv_cache.py --deselect="tests/quantization/test_per_token_kv_cache.py::test_triton_unified_attention_per_token_head_scale[int4-16-128-num_heads0-seq_lens1]" - && VLLM_TEST_FORCE_LOAD_FORMAT=auto pytest -v -s tests/quantization/test_auto_awq.py' + && VLLM_TEST_FORCE_LOAD_FORMAT=auto pytest -v -s tests/quantization/test_auto_awq.py + && VLLM_TEST_FORCE_LOAD_FORMAT=auto pytest -v -s tests/quantization/test_fp8_per_channel.py + && VLLM_TEST_FORCE_LOAD_FORMAT=auto pytest -v -s tests/models/quantization/test_fp8_per_channel.py --deselect="tests/models/quantization/test_fp8_per_channel.py::test_fp8_per_channel_logprobs[moe]"' diff --git a/.buildkite/intel_jobs/samplers_intel.yaml b/.buildkite/intel_jobs/samplers_intel.yaml index 41ec3c4df7..027237225d 100644 --- a/.buildkite/intel_jobs/samplers_intel.yaml +++ b/.buildkite/intel_jobs/samplers_intel.yaml @@ -6,6 +6,33 @@ steps: key: samplers-test-xpu-kernel-intel timeout_in_minutes: 40 device: intel_gpu + agent_tags: + label: production + gpu: 1+ + mem: 16+ + no_plugin: true + working_dir: "." + env: + REGISTRY: "public.ecr.aws/q9t5s3a7" + REPO: "vllm-ci-test-repo" + VLLM_TEST_DEVICE: "xpu" + source_file_dependencies: + - vllm/model_executor/layers + - vllm/sampling_metadata.py + - tests/samplers + - tests/conftest.py + - vllm/entrypoints/generate/beam_search + commands: + - >- + bash .buildkite/scripts/hardware_ci/run-intel-test.sh + 'cd tests && + VLLM_XPU_USE_SAMPLER_KERNEL=1 pytest -v -s samplers + --ignore=samplers/test_beam_search.py' + +- label: Samplers Beam Search Test (XPU Kernel) + key: samplers-beam-search-test-xpu-kernel-intel + timeout_in_minutes: 40 + device: intel_gpu agent_tags: label: production gpu: 1+ @@ -26,4 +53,4 @@ steps: - >- bash .buildkite/scripts/hardware_ci/run-intel-test.sh 'cd tests && - VLLM_XPU_USE_SAMPLER_KERNEL=1 pytest -v -s samplers' + VLLM_XPU_USE_SAMPLER_KERNEL=1 pytest -v -s samplers/test_beam_search.py' diff --git a/.buildkite/intel_jobs/test-intel.yaml b/.buildkite/intel_jobs/test-intel.yaml index 08f8dcdeab..a2a0e14be1 100644 --- a/.buildkite/intel_jobs/test-intel.yaml +++ b/.buildkite/intel_jobs/test-intel.yaml @@ -23,7 +23,7 @@ steps: agent_tags: label: production gpu: 2+ - mem: 24+ + mem: 16+ no_plugin: true env: REGISTRY: "public.ecr.aws/q9t5s3a7" @@ -46,7 +46,6 @@ steps: python3 examples/basic/offline_inference/generate.py --model ibm-research/PowerMoE-3b --block-size 64 --enforce-eager -tp 2 && python3 examples/basic/offline_inference/generate.py --model ibm-research/PowerMoE-3b --block-size 64 --enforce-eager -tp 2 --enable-expert-parallel && python3 examples/basic/offline_inference/generate.py --model superjob/Qwen3-4B-Instruct-2507-GPTQ-Int4 --max-model-len 8192 && - VLLM_XPU_FUSED_MOE_USE_REF=1 python3 examples/basic/offline_inference/generate.py --model Qwen/Qwen3-30B-A3B-Instruct-2507-FP8 --enforce-eager -tp 2 --max-model-len 8192 && python3 examples/basic/offline_inference/generate.py --model INCModel/Qwen3-30B-A3B-Instruct-2507-MXFP4-LLMC -tp 2 --max-model-len 8192 ' - label: "XPU W8A8 FP8 Linear Examples" @@ -58,7 +57,7 @@ steps: agent_tags: label: production gpu: 1+ - mem: 24+ + mem: 16+ no_plugin: true env: REGISTRY: "public.ecr.aws/q9t5s3a7" @@ -86,7 +85,7 @@ steps: agent_tags: label: production gpu: 1+ - mem: 24+ + mem: 16+ no_plugin: true env: REGISTRY: "public.ecr.aws/q9t5s3a7" @@ -106,8 +105,34 @@ steps: pytest -v -s v1/test_serial_utils.py && pytest -v -s v1/e2e/general/test_correctness_sliding_window.py && pytest -v -s v1/spec_decode --ignore=v1/spec_decode/test_max_len.py --ignore=v1/spec_decode/test_speculators_eagle3.py --ignore=v1/spec_decode/test_acceptance_length.py --ignore=v1/spec_decode/test_speculators_correctness.py --deselect=tests/v1/spec_decode/test_mtp.py::test_glm_mtp_defers_lm_head && - pytest -v -s v1/kv_connector/unit --ignore=v1/kv_connector/unit/test_lmcache_integration.py --ignore=v1/kv_connector/unit/test_hf3fs_client.py --ignore=v1/kv_connector/unit/test_hf3fs_connector.py --ignore=v1/kv_connector/unit/test_hf3fs_metadata_server.py --ignore=v1/kv_connector/unit/test_offloading_connector.py --ignore=v1/kv_connector/unit/test_nixl_connector_hma.py --deselect=tests/v1/kv_connector/unit/test_nixl_connector.py::test_abort_timeout_on_prefiller[ray]' + pytest -v -s v1/kv_connector/unit --ignore=v1/kv_connector/unit/test_lmcache_integration.py --ignore=v1/kv_connector/unit/test_hf3fs_client.py --ignore=v1/kv_connector/unit/test_hf3fs_connector.py --ignore=v1/kv_connector/unit/test_hf3fs_metadata_server.py --ignore=v1/kv_connector/unit/test_offloading_connector.py --ignore=v1/kv_connector/unit/test_nixl_connector_hma.py --ignore=v1/kv_connector/unit/test_multi_connector.py --ignore=v1/kv_connector/unit/test_example_connector.py --deselect=tests/v1/kv_connector/unit/test_nixl_connector.py::test_abort_timeout_on_prefiller[ray]' parallelism: 2 + - label: "XPU Qwen3-30B + V1 KV Connector" + key: xpu-qwen3-30b-v1-kv-connector + depends_on: + - image-build-xpu + timeout_in_minutes: 90 + device: intel_gpu + agent_tags: + label: production + gpu: 2+ + mem: 24+ + no_plugin: true + env: + REGISTRY: "public.ecr.aws/q9t5s3a7" + REPO: "vllm-ci-test-repo" + source_file_dependencies: + - vllm/ + - .buildkite/intel_jobs/test-intel.yaml + commands: + - >- + bash .buildkite/scripts/hardware_ci/run-intel-test.sh + 'cd tests && + pip install dpctl && + pytest -v -s v1/kv_connector/unit/test_multi_connector.py && + pytest -v -s v1/kv_connector/unit/test_example_connector.py && + cd .. && + VLLM_XPU_FUSED_MOE_USE_REF=1 python3 examples/basic/offline_inference/generate.py --model Qwen/Qwen3-30B-A3B-Instruct-2507-FP8 --enforce-eager -tp 2 --max-model-len 8192' - label: "XPU server test" key: xpu-server-test depends_on: @@ -153,7 +178,7 @@ steps: bash .buildkite/scripts/hardware_ci/run-intel-test.sh 'cd tests && pytest -v -s quantization/test_auto_round.py && - pytest -v -s quantization/test_online.py --deselect=tests/quantization/test_online.py::test_online_quantization_loads_real_weights' + pytest -v -s quantization/test_online.py --deselect=tests/quantization/test_online.py::test_online_quantization_loads_real_weights --deselect="tests/quantization/test_online.py::test_online_quantization[False-requantization_mxfp8_ptcp_fp8]"' - label: "XPU compressed tensors FP8 test" key: xpu-compressed-tensors-fp8-test depends_on: diff --git a/.buildkite/lm-eval-harness/test_lm_eval_correctness.py b/.buildkite/lm-eval-harness/test_lm_eval_correctness.py index 5576c84b05..e4c99adb7b 100644 --- a/.buildkite/lm-eval-harness/test_lm_eval_correctness.py +++ b/.buildkite/lm-eval-harness/test_lm_eval_correctness.py @@ -8,16 +8,28 @@ pytest -s -v test_lm_eval_correctness.py \ --tp-size=1 """ +import gc import os +import time from contextlib import contextmanager import lm_eval import pytest +import torch import yaml +from lm_eval.api.registry import get_model +from lm_eval.utils import simple_parse_args_string +from vllm import LLM +from vllm.distributed import cleanup_dist_env_and_memory +from vllm.logger import init_logger from vllm.platforms import current_platform +logger = init_logger(__name__) + DEFAULT_RTOL = 0.08 +ROCM_ENGINE_SHUTDOWN_TIMEOUT_S = 60.0 +MEMORY_RELEASE_TIMEOUT_S = 240.0 @contextmanager @@ -46,6 +58,48 @@ def scoped_env_vars(new_env: dict[str, str]): os.environ.pop(key, None) +def _wait_for_memory_release(gpu_memory_utilization: float) -> None: + devices = range(torch.accelerator.device_count()) + deadline = time.monotonic() + MEMORY_RELEASE_TIMEOUT_S + while True: + memory = [torch.accelerator.get_memory_info(d) for d in devices] + if all(free >= total * gpu_memory_utilization for free, total in memory): + return + if time.monotonic() >= deadline: + usage = ", ".join( + f"{d}: {free / 2**30:.2f}/{total / 2**30:.2f} GiB free" + for d, (free, total) in zip(devices, memory) + ) + logger.warning( + "GPU memory not released after %ss (%s); the next model needs %.2f", + MEMORY_RELEASE_TIMEOUT_S, + usage, + gpu_memory_utilization, + ) + return + time.sleep(1) + + +def _shutdown_lm(lm) -> None: + if not current_platform.is_rocm(): + return + llm = getattr(lm, "model", None) + if not isinstance(llm, LLM): + return + gpu_memory_utilization = ( + llm.llm_engine.vllm_config.cache_config.gpu_memory_utilization + ) + try: + llm.llm_engine.engine_core.shutdown(timeout=ROCM_ENGINE_SHUTDOWN_TIMEOUT_S) + except Exception: + logger.exception("Engine core shutdown raised; GPU memory may leak") + del llm + lm.model = None + gc.collect() + cleanup_dist_env_and_memory() + _wait_for_memory_release(gpu_memory_utilization) + + def launch_lm_eval(eval_config, tp_size): trust_remote_code = eval_config.get("trust_remote_code", False) max_model_len = eval_config.get("max_model_len", 4096) @@ -82,9 +136,17 @@ def launch_lm_eval(eval_config, tp_size): env_vars = eval_config.get("env_vars", None) with scoped_env_vars(env_vars): + if current_platform.is_rocm(): + model = get_model(backend).create_from_arg_string( + model_args, + {"batch_size": batch_size, "max_batch_size": None, "device": None}, + ) + model_kwargs = {"metadata": simple_parse_args_string(model_args)} + else: + model = backend + model_kwargs = {"model_args": model_args} results = lm_eval.simple_evaluate( - model=backend, - model_args=model_args, + model=model, tasks=[task["name"] for task in eval_config["tasks"]], num_fewshot=eval_config["num_fewshot"], limit=eval_config["limit"], @@ -98,7 +160,9 @@ def launch_lm_eval(eval_config, tp_size): # Forward decoding and early-stop controls (e.g., max_gen_toks, until=...) gen_kwargs=eval_config.get("gen_kwargs"), batch_size=batch_size, + **model_kwargs, ) + _shutdown_lm(model) return results diff --git a/.buildkite/release-pipeline.yaml b/.buildkite/release-pipeline.yaml index d6e47715fa..6064f93104 100644 --- a/.buildkite/release-pipeline.yaml +++ b/.buildkite/release-pipeline.yaml @@ -355,6 +355,10 @@ steps: queue: arm64_cpu_queue_release commands: - "aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7" + # TODO(arpera): + # Remove BUILD_NIXL=true once NIXL has a wheel built against torch 2.15 + # For details: https://github.com/vllm-project/vllm/pull/60090 + # Tracking issue report: https://github.com/ai-dynamo/nixl/issues/2297 - | DOCKER_BUILDKIT=1 docker buildx build --load \ $(bash .buildkite/scripts/docker-build-metadata-args.sh cu134) \ @@ -366,6 +370,7 @@ steps: --build-arg torch_cuda_arch_list="${CUDA_ARCH_AARCH64}" \ --build-arg INSTALL_KV_CONNECTORS=true \ --build-arg INSTALL_RUBIN_PRERELEASE=true \ + --build-arg BUILD_NIXL=true \ --build-arg TRITON_INSTALL_FROM_SOURCE_REPO=https://github.com/triton-lang/triton.git \ --build-arg TRITON_INSTALL_FROM_SOURCE_REVISION=3f6e41132b5edf639bfb872ad73d4688765e08b8 \ --build-arg RUN_WHEEL_CHECK=false \ @@ -1077,6 +1082,38 @@ steps: DOCKER_BUILDKIT: "1" DOCKERHUB_USERNAME: "vllmbot" + - trigger: "perf-eval" + label: "Trigger perf-eval (CUDA)" + key: trigger-perf-eval-cuda + if: build.env("NIGHTLY") == "1" || build.env("RUN_PERF_EVAL") == "1" + depends_on: + - build-release-image-x86 + async: true + soft_fail: true + build: + branch: "main" + message: "(NIGHTLY=${NIGHTLY:-0}) Evaluate vLLM $BUILDKITE_COMMIT (CUDA)" + env: + NIGHTLY: 1 # run all perf-eval workloads + VLLM_COMMIT: "$BUILDKITE_COMMIT" + VLLM_IMAGE_CUDA: "public.ecr.aws/q9t5s3a7/vllm-release-repo:$BUILDKITE_COMMIT-x86_64" + + - trigger: "perf-eval" + label: "Trigger perf-eval (ROCm)" + key: trigger-perf-eval-rocm + if: build.env("NIGHTLY") == "1" || build.env("RUN_PERF_EVAL") == "1" + depends_on: + - build-rocm-release-image + async: true + soft_fail: true + build: + branch: "main" + message: "(NIGHTLY=${NIGHTLY:-0}) Evaluate vLLM $BUILDKITE_COMMIT (ROCm)" + env: + NIGHTLY: 1 # run all perf-eval workloads + VLLM_COMMIT: "$BUILDKITE_COMMIT" + VLLM_IMAGE_ROCM: "public.ecr.aws/q9t5s3a7/vllm-release-repo:$BUILDKITE_COMMIT-rocm" + # ============================================================================= # ROCk Release Pipeline (x86_64 only) # ============================================================================= diff --git a/.buildkite/scripts/build-macos-wheel.sh b/.buildkite/scripts/build-macos-wheel.sh index cb95d92ff1..d956f71038 100755 --- a/.buildkite/scripts/build-macos-wheel.sh +++ b/.buildkite/scripts/build-macos-wheel.sh @@ -25,6 +25,7 @@ export CMAKE_BUILD_PARALLEL_LEVEL="${CMAKE_BUILD_PARALLEL_LEVEL:-4}" uv venv --python 3.12 uv pip install -r requirements/build/cpu.txt --index-strategy unsafe-best-match -uv build --wheel --no-build-isolation -o artifacts/dist +uv build --wheel --no-build-isolation -o artifacts/dist \ + --config-setting=--build-option=--py-limited-api=cp38 ls -l artifacts/dist/*.whl diff --git a/.buildkite/scripts/hardware_ci/run-cpu-test-arm.sh b/.buildkite/scripts/hardware_ci/run-cpu-test-arm.sh index 689ca92d6a..1f598b277b 100755 --- a/.buildkite/scripts/hardware_ci/run-cpu-test-arm.sh +++ b/.buildkite/scripts/hardware_ci/run-cpu-test-arm.sh @@ -43,6 +43,7 @@ kernel_tests() { docker exec "$CONTAINER_NAME" bash -c " set -e pytest -x -v -s tests/kernels/test_onednn.py + pytest -x -v -s tests/kernels/test_cpu_fused_sampling.py pytest -x -v -s tests/kernels/attention/test_cpu_attn.py pytest -x -v -s tests/kernels/core/test_cpu_activation.py pytest -x -v -s tests/kernels/moe/test_cpu_fused_moe.py diff --git a/.buildkite/scripts/hardware_ci/run-cpu-test.sh b/.buildkite/scripts/hardware_ci/run-cpu-test.sh index eecf70ea2b..5f9355f9e9 100644 --- a/.buildkite/scripts/hardware_ci/run-cpu-test.sh +++ b/.buildkite/scripts/hardware_ci/run-cpu-test.sh @@ -19,10 +19,9 @@ DISK_USAGE_THRESHOLD=${DISK_USAGE_THRESHOLD:-80} CACHE_MAX_AGE=${CACHE_MAX_AGE:-24h} # Reclaim disk only when the host is under pressure, aging out anything unused -# for less than CACHE_MAX_AGE so hot layers survive -- same `--filter -# until=h` pattern the TPU CI scripts already rely on. `docker buildx -# prune --max-used-space` is a no-op on this host's `docker` driver (BuildKit -# embedded in dockerd never enforces the size cap), so we don't use it. +# for less than CACHE_MAX_AGE so hot layers survive (`--filter until=h`). +# `docker buildx prune --max-used-space` is a no-op on this host's `docker` +# driver (BuildKit embedded in dockerd never enforces the size cap), so we don't use it. prune_if_disk_pressure() { local docker_root disk_usage docker_root=$(docker info -f '{{.DockerRootDir}}' 2>/dev/null || true) diff --git a/.buildkite/scripts/hardware_ci/run-gh200-test.sh b/.buildkite/scripts/hardware_ci/run-gh200-test.sh deleted file mode 100644 index 20295fc25e..0000000000 --- a/.buildkite/scripts/hardware_ci/run-gh200-test.sh +++ /dev/null @@ -1,29 +0,0 @@ -#!/bin/bash - -# This script build the GH200 docker image and run the offline inference inside the container. -# It serves a sanity check for compilation and basic model usage. -set -ex - -# Skip the new torch installation during build since we are using the specified version for arm64 in the Dockerfile -python3 tools/use_existing_torch.py - -# Try building the docker image -DOCKER_BUILDKIT=1 docker build . \ - --file docker/Dockerfile \ - --target vllm-openai \ - --platform "linux/arm64" \ - -t gh200-test \ - --build-arg max_jobs=66 \ - --build-arg nvcc_threads=2 \ - --build-arg BUILD_BASE_IMAGE=pytorch/manylinuxaarch64-builder:cuda13.0-b8b5f17a7d9ccfc25bbc5cf17b3fcea12964a042 \ - --build-arg torch_cuda_arch_list="9.0+PTX" - -# Setup cleanup -remove_docker_container() { docker rm -f gh200-test || true; } -trap remove_docker_container EXIT -remove_docker_container - -# Run the image and test offline inference -docker run -e HF_TOKEN -e VLLM_WORKER_MULTIPROC_METHOD=spawn -v /root/.cache/huggingface:/root/.cache/huggingface --name gh200-test --gpus=all --entrypoint="" gh200-test bash -c ' - python3 examples/basic/offline_inference/generate.py --model meta-llama/Llama-3.2-1B -' diff --git a/.buildkite/scripts/hardware_ci/run-intel-ci-test.sh b/.buildkite/scripts/hardware_ci/run-intel-ci-test.sh index c679d0aff2..44f0d9944a 100644 --- a/.buildkite/scripts/hardware_ci/run-intel-ci-test.sh +++ b/.buildkite/scripts/hardware_ci/run-intel-ci-test.sh @@ -57,7 +57,7 @@ case "${test_suite}" in cd tests pytest -v -s quantization/test_auto_round.py - pytest -v -s quantization/test_online.py --deselect=tests/quantization/test_online.py::test_online_quantization_loads_real_weights + pytest -v -s quantization/test_online.py --deselect=tests/quantization/test_online.py::test_online_quantization_loads_real_weights --deselect="tests/quantization/test_online.py::test_online_quantization[False-requantization_mxfp8_ptcp_fp8]" ;; compressed-tensors-fp8) cd tests diff --git a/.buildkite/scripts/hardware_ci/run-tpu-v1-test-part2.sh b/.buildkite/scripts/hardware_ci/run-tpu-v1-test-part2.sh deleted file mode 100755 index f300b7f809..0000000000 --- a/.buildkite/scripts/hardware_ci/run-tpu-v1-test-part2.sh +++ /dev/null @@ -1,166 +0,0 @@ -#!/bin/bash - -set -xu - - -remove_docker_container() { - docker rm -f tpu-test || true; -} - -trap remove_docker_container EXIT - -# Remove the container that might not be cleaned up in the previous run. -remove_docker_container - -# Build the docker image. -docker build -f docker/Dockerfile.tpu -t vllm-tpu . - -# Set up cleanup. -cleanup_docker() { - # Get Docker's root directory - docker_root=$(docker info -f '{{.DockerRootDir}}') - if [ -z "$docker_root" ]; then - echo "Failed to determine Docker root directory." - exit 1 - fi - echo "Docker root directory: $docker_root" - # Check disk usage of the filesystem where Docker's root directory is located - disk_usage=$(df "$docker_root" | tail -1 | awk '{print $5}' | sed 's/%//') - # Define the threshold - threshold=70 - if [ "$disk_usage" -gt "$threshold" ]; then - echo "Disk usage is above $threshold%. Cleaning up Docker images and volumes..." - # Remove dangling images (those that are not tagged and not used by any container) - docker image prune -f - # Remove unused volumes / force the system prune for old images as well. - docker volume prune -f && docker system prune --force --filter "until=72h" --all - echo "Docker images and volumes cleanup completed." - else - echo "Disk usage is below $threshold%. No cleanup needed." - fi -} -cleanup_docker - -# For HF_TOKEN. -source /etc/environment - -docker run --privileged --net host --shm-size=16G -it \ - -e "HF_TOKEN=$HF_TOKEN" --name tpu-test \ - vllm-tpu /bin/bash -c ' -set -e # Exit immediately if a command exits with a non-zero status. -set -u # Treat unset variables as an error. - -echo "--- Starting script inside Docker container ---" - -# Create results directory -RESULTS_DIR=$(mktemp -d) -# If mktemp fails, set -e will cause the script to exit. -echo "Results will be stored in: $RESULTS_DIR" - -# Install dependencies -echo "--- Installing Python dependencies ---" -python3 -m pip install --progress-bar off git+https://github.com/thuml/depyf.git \ - && python3 -m pip install --progress-bar off pytest pytest-asyncio tpu-info \ - && python3 -m pip install --progress-bar off "lm-eval[api]>=0.4.12" \ - && python3 -m pip install --progress-bar off hf-transfer tblib==3.1.0 -echo "--- Python dependencies installed ---" - -export VLLM_XLA_CHECK_RECOMPILATION=1 -export VLLM_XLA_CACHE_PATH= - -echo "--- Hardware Information ---" -# tpu-info -echo "--- Starting Tests ---" -set +e -overall_script_exit_code=0 - -# --- Test Definitions --- -# If a test fails, this function will print logs and will not cause the main script to exit. -run_test() { - local test_num=$1 - local test_name=$2 - local test_command=$3 - local log_file="$RESULTS_DIR/test_${test_num}.log" - local actual_exit_code - - echo "--- TEST_$test_num: Running $test_name ---" - - # Execute the test command. - eval "$test_command" > >(tee -a "$log_file") 2> >(tee -a "$log_file" >&2) - actual_exit_code=$? - - echo "TEST_${test_num}_COMMAND_EXIT_CODE: $actual_exit_code" # This goes to main log - echo "TEST_${test_num}_COMMAND_EXIT_CODE: $actual_exit_code" >> "$log_file" # Also to per-test log - - if [ "$actual_exit_code" -ne 0 ]; then - echo "TEST_$test_num ($test_name) FAILED with exit code $actual_exit_code." >&2 - echo "--- Log for failed TEST_$test_num ($test_name) ---" >&2 - if [ -f "$log_file" ]; then - cat "$log_file" >&2 - else - echo "Log file $log_file not found for TEST_$test_num ($test_name)." >&2 - fi - echo "--- End of log for TEST_$test_num ($test_name) ---" >&2 - return "$actual_exit_code" # Return the failure code - else - echo "TEST_$test_num ($test_name) PASSED." - return 0 # Return success - fi -} - -# Helper function to call run_test and update the overall script exit code -run_and_track_test() { - local test_num_arg="$1" - local test_name_arg="$2" - local test_command_arg="$3" - - # Run the test - run_test "$test_num_arg" "$test_name_arg" "$test_command_arg" - local test_specific_exit_code=$? - - # If the test failed, set the overall script exit code to 1 - if [ "$test_specific_exit_code" -ne 0 ]; then - # No need for extra echo here, run_test already logged the failure. - overall_script_exit_code=1 - fi -} - -# --- Actual Test Execution --- -run_and_track_test 1 "test_struct_output_generate.py" \ - "python3 -m pytest -s -v /workspace/vllm/tests/entrypoints/llm/test_struct_output_generate.py -k \"not test_structured_output_with_reasoning_matrices\"" -run_and_track_test 2 "test_moe_pallas.py" \ - "python3 -m pytest -s -v /workspace/vllm/tests/tpu/test_moe_pallas.py" -run_and_track_test 3 "test_lora.py" \ - "VLLM_XLA_CHECK_RECOMPILATION=0 python3 -m pytest -s -v /workspace/vllm/tests/tpu/lora/test_lora.py" -run_and_track_test 4 "test_tpu_qkv_linear.py" \ - "python3 -m pytest -s -v /workspace/vllm/tests/v1/tpu/test_tpu_qkv_linear.py" -run_and_track_test 5 "test_spmd_model_weight_loading.py" \ - "python3 -m pytest -s -v /workspace/vllm/tests/v1/tpu/test_spmd_model_weight_loading.py" -run_and_track_test 6 "test_kv_cache_update_kernel.py" \ - "python3 -m pytest -s -v /workspace/vllm/tests/v1/tpu/test_kv_cache_update_kernel.py" -run_and_track_test 7 "test_tpu_int8.py" \ - "python3 -m pytest -s -v /workspace/vllm/tests/v1/tpu/test_tpu_int8.py" - -# After all tests have been attempted, exit with the overall status. -if [ "$overall_script_exit_code" -ne 0 ]; then - echo "--- One or more tests FAILED. Overall script exiting with failure code 1. ---" -else - echo "--- All tests have completed and PASSED. Overall script exiting with success code 0. ---" -fi -exit "$overall_script_exit_code" -' # IMPORTANT: This is the closing single quote for the bash -c "..." command. Ensure it is present and correct. - -# Capture the exit code of the docker run command -DOCKER_RUN_EXIT_CODE=$? - -# The trap will run for cleanup. -# Exit the main script with the Docker run command's exit code. -if [ "$DOCKER_RUN_EXIT_CODE" -ne 0 ]; then - echo "Docker run command failed with exit code $DOCKER_RUN_EXIT_CODE." - exit "$DOCKER_RUN_EXIT_CODE" -else - echo "Docker run command completed successfully." - exit 0 -fi -# TODO: This test fails because it uses RANDOM_SEED sampling -# pytest -v -s /workspace/vllm/tests/tpu/test_custom_dispatcher.py \ diff --git a/.buildkite/scripts/hardware_ci/run-tpu-v1-test.sh b/.buildkite/scripts/hardware_ci/run-tpu-v1-test.sh deleted file mode 100755 index 6f4e0d37db..0000000000 --- a/.buildkite/scripts/hardware_ci/run-tpu-v1-test.sh +++ /dev/null @@ -1,172 +0,0 @@ -#!/bin/bash - -set -xu - - -remove_docker_container() { - docker rm -f tpu-test || true; -} - -trap remove_docker_container EXIT - -# Remove the container that might not be cleaned up in the previous run. -remove_docker_container - -# Build the docker image. -docker build -f docker/Dockerfile.tpu -t vllm-tpu . - -# Set up cleanup. -cleanup_docker() { - # Get Docker's root directory - docker_root=$(docker info -f '{{.DockerRootDir}}') - if [ -z "$docker_root" ]; then - echo "Failed to determine Docker root directory." - exit 1 - fi - echo "Docker root directory: $docker_root" - # Check disk usage of the filesystem where Docker's root directory is located - disk_usage=$(df "$docker_root" | tail -1 | awk '{print $5}' | sed 's/%//') - # Define the threshold - threshold=70 - if [ "$disk_usage" -gt "$threshold" ]; then - echo "Disk usage is above $threshold%. Cleaning up Docker images and volumes..." - # Remove dangling images (those that are not tagged and not used by any container) - docker image prune -f - # Remove unused volumes / force the system prune for old images as well. - docker volume prune -f && docker system prune --force --filter "until=72h" --all - echo "Docker images and volumes cleanup completed." - else - echo "Disk usage is below $threshold%. No cleanup needed." - fi -} -cleanup_docker - -# For HF_TOKEN. -source /etc/environment - -docker run --privileged --net host --shm-size=16G -it \ - -e "HF_TOKEN=$HF_TOKEN" --name tpu-test \ - vllm-tpu /bin/bash -c ' -set -e # Exit immediately if a command exits with a non-zero status. -set -u # Treat unset variables as an error. - -echo "--- Starting script inside Docker container ---" - -# Create results directory -RESULTS_DIR=$(mktemp -d) -# If mktemp fails, set -e will cause the script to exit. -echo "Results will be stored in: $RESULTS_DIR" - -# Install dependencies -echo "--- Installing Python dependencies ---" -python3 -m pip install --progress-bar off git+https://github.com/thuml/depyf.git \ - && python3 -m pip install --progress-bar off pytest pytest-asyncio tpu-info \ - && python3 -m pip install --progress-bar off "lm-eval[api]>=0.4.12" \ - && python3 -m pip install --progress-bar off hf-transfer tblib==3.1.0 -echo "--- Python dependencies installed ---" - -export VLLM_XLA_CHECK_RECOMPILATION=1 -export VLLM_XLA_CACHE_PATH= - -echo "--- Hardware Information ---" -# tpu-info -echo "--- Starting Tests ---" -set +e -overall_script_exit_code=0 - -# --- Test Definitions --- -# If a test fails, this function will print logs and will not cause the main script to exit. -run_test() { - local test_num=$1 - local test_name=$2 - local test_command=$3 - local log_file="$RESULTS_DIR/test_${test_num}.log" - local actual_exit_code - - echo "--- TEST_$test_num: Running $test_name ---" - - # Execute the test command. - eval "$test_command" > >(tee -a "$log_file") 2> >(tee -a "$log_file" >&2) - actual_exit_code=$? - - echo "TEST_${test_num}_COMMAND_EXIT_CODE: $actual_exit_code" # This goes to main log - echo "TEST_${test_num}_COMMAND_EXIT_CODE: $actual_exit_code" >> "$log_file" # Also to per-test log - - if [ "$actual_exit_code" -ne 0 ]; then - echo "TEST_$test_num ($test_name) FAILED with exit code $actual_exit_code." >&2 - echo "--- Log for failed TEST_$test_num ($test_name) ---" >&2 - if [ -f "$log_file" ]; then - cat "$log_file" >&2 - else - echo "Log file $log_file not found for TEST_$test_num ($test_name)." >&2 - fi - echo "--- End of log for TEST_$test_num ($test_name) ---" >&2 - return "$actual_exit_code" # Return the failure code - else - echo "TEST_$test_num ($test_name) PASSED." - return 0 # Return success - fi -} - -# Helper function to call run_test and update the overall script exit code -run_and_track_test() { - local test_num_arg="$1" - local test_name_arg="$2" - local test_command_arg="$3" - - # Run the test - run_test "$test_num_arg" "$test_name_arg" "$test_command_arg" - local test_specific_exit_code=$? - - # If the test failed, set the overall script exit code to 1 - if [ "$test_specific_exit_code" -ne 0 ]; then - # No need for extra echo here, run_test already logged the failure. - overall_script_exit_code=1 - fi -} - -# --- Actual Test Execution --- -run_and_track_test 0 "test_perf.py" \ - "python3 -m pytest -s -v /workspace/vllm/tests/v1/tpu/test_perf.py" -run_and_track_test 1 "test_compilation.py" \ - "python3 -m pytest -s -v /workspace/vllm/tests/tpu/test_compilation.py" -run_and_track_test 2 "test_basic.py" \ - "python3 -m pytest -s -v /workspace/vllm/tests/v1/tpu/test_basic.py" -run_and_track_test 3 "test_accuracy.py::test_lm_eval_accuracy_v1_engine" \ - "python3 -m pytest -s -v /workspace/vllm/tests/entrypoints/llm/test_accuracy.py::test_lm_eval_accuracy_v1_engine" -run_and_track_test 4 "test_quantization_accuracy.py" \ - "python3 -m pytest -s -v /workspace/vllm/tests/tpu/test_quantization_accuracy.py" -run_and_track_test 6 "test_tpu_model_runner.py" \ - "python3 -m pytest -s -v /workspace/vllm/tests/v1/tpu/worker/test_tpu_model_runner.py" -run_and_track_test 7 "test_sampler.py" \ - "python3 -m pytest -s -v /workspace/vllm/tests/v1/tpu/test_sampler.py" -run_and_track_test 8 "test_topk_topp_sampler.py" \ - "python3 -m pytest -s -v /workspace/vllm/tests/v1/tpu/test_topk_topp_sampler.py" -run_and_track_test 9 "test_multimodal.py" \ - "python3 -m pytest -s -v /workspace/vllm/tests/v1/tpu/test_multimodal.py" -run_and_track_test 10 "test_pallas.py" \ - "python3 -m pytest -s -v /workspace/vllm/tests/v1/tpu/test_pallas.py" - -# After all tests have been attempted, exit with the overall status. -if [ "$overall_script_exit_code" -ne 0 ]; then - echo "--- One or more tests FAILED. Overall script exiting with failure code 1. ---" -else - echo "--- All tests have completed and PASSED. Overall script exiting with success code 0. ---" -fi -exit "$overall_script_exit_code" -' # IMPORTANT: This is the closing single quote for the bash -c "..." command. Ensure it is present and correct. - -# Capture the exit code of the docker run command -DOCKER_RUN_EXIT_CODE=$? - -# The trap will run for cleanup. -# Exit the main script with the Docker run command's exit code. -if [ "$DOCKER_RUN_EXIT_CODE" -ne 0 ]; then - echo "Docker run command failed with exit code $DOCKER_RUN_EXIT_CODE." - exit "$DOCKER_RUN_EXIT_CODE" -else - echo "Docker run command completed successfully." - exit 0 -fi -# TODO: This test fails because it uses RANDOM_SEED sampling -# pytest -v -s /workspace/vllm/tests/tpu/test_custom_dispatcher.py \ diff --git a/.buildkite/scripts/install-kv-offload.sh b/.buildkite/scripts/install-kv-offload.sh index 46d756b621..eb8bab9e39 100644 --- a/.buildkite/scripts/install-kv-offload.sh +++ b/.buildkite/scripts/install-kv-offload.sh @@ -10,7 +10,7 @@ fi # TODO: Use the PyPI wheel once it is published. uv pip install --system \ - "kvcr @ git+https://github.com/ai-dynamo/kvcr.git@v0.1.0" + "kvcr @ git+https://github.com/ai-dynamo/kvcr.git@v0.1.1" # Keep only the NIXL wheel matching the CI image's CUDA runtime. NIXL_VERSION=$(uv pip show --system nixl | sed -n 's/^Version: //p') diff --git a/.buildkite/scripts/run-rust-frontend-cargo-ci.sh b/.buildkite/scripts/run-rust-frontend-cargo-ci.sh index 62a70fd664..aa16302da8 100755 --- a/.buildkite/scripts/run-rust-frontend-cargo-ci.sh +++ b/.buildkite/scripts/run-rust-frontend-cargo-ci.sh @@ -23,7 +23,7 @@ export PATH="$CARGO_HOME/bin:$PATH" CARGO_BINSTALL_VERSION="${CARGO_BINSTALL_VERSION:-1.20.1}" UV_VERSION="${UV_VERSION:-0.11.28}" -PYO3_PYTHON_VERSION="${PYO3_PYTHON_VERSION:-3.12}" +UV_SCRIPT_PYTHON_VERSION="${UV_SCRIPT_PYTHON_VERSION:-3.12}" CARGO_SORT_VERSION_REQ="${CARGO_SORT_VERSION_REQ:-2}" CARGO_DENY_VERSION_REQ="${CARGO_DENY_VERSION_REQ:-0.20}" @@ -60,12 +60,12 @@ install_cargo_binstall() { install_cargo_sort() { log_section "Installing cargo-sort ${CARGO_SORT_VERSION_REQ}" - cargo binstall --no-confirm --force "cargo-sort@${CARGO_SORT_VERSION_REQ}" + cargo binstall --no-confirm --force --locked "cargo-sort@${CARGO_SORT_VERSION_REQ}" } install_cargo_deny() { log_section "Installing cargo-deny ${CARGO_DENY_VERSION_REQ}" - cargo binstall --no-confirm --force "cargo-deny@${CARGO_DENY_VERSION_REQ}" + cargo binstall --no-confirm --force --locked "cargo-deny@${CARGO_DENY_VERSION_REQ}" } install_cargo_nextest() { @@ -74,6 +74,7 @@ install_cargo_nextest() { --no-confirm \ --force \ --secure \ + --locked \ "cargo-nextest@${CARGO_NEXTEST_VERSION_REQ}" } @@ -83,34 +84,7 @@ install_uv() { "https://github.com/astral-sh/uv/releases/download/${UV_VERSION}/uv-installer.sh" \ | env UV_INSTALL_DIR="$CARGO_HOME/bin" sh uv --version -} - -setup_pyo3_python() { - log_section "Installing Python ${PYO3_PYTHON_VERSION} for PyO3 tests" - uv python install "$PYO3_PYTHON_VERSION" - PYO3_PYTHON="$(uv python find \ - --managed-python \ - --no-project \ - --resolve-links \ - "$PYO3_PYTHON_VERSION")" - export PYO3_PYTHON - - local python_libdir - python_libdir="$("$PYO3_PYTHON" - <<'PY' -import pathlib -import sysconfig - -libdir = pathlib.Path(sysconfig.get_config_var("LIBDIR")) -ldlibrary = sysconfig.get_config_var("LDLIBRARY") -assert sysconfig.get_config_var("Py_ENABLE_SHARED") == 1 -assert ldlibrary -assert (libdir / ldlibrary).exists(), libdir / ldlibrary -print(libdir) -PY -)" - - export LD_LIBRARY_PATH="${python_libdir}:${LD_LIBRARY_PATH:-}" - export LIBRARY_PATH="${python_libdir}:${LIBRARY_PATH:-}" + uv python install "$UV_SCRIPT_PYTHON_VERSION" } run_style_clippy() { @@ -144,7 +118,6 @@ run_style_clippy() { run_tests() { install_uv - setup_pyo3_python install_cargo_binstall install_cargo_nextest diff --git a/.buildkite/scripts/tpu/cleanup_docker.sh b/.buildkite/scripts/tpu/cleanup_docker.sh deleted file mode 100755 index 740d81fb39..0000000000 --- a/.buildkite/scripts/tpu/cleanup_docker.sh +++ /dev/null @@ -1,24 +0,0 @@ -#!/bin/bash - -set -euo pipefail - -docker_root=$(docker info -f '{{.DockerRootDir}}') -if [ -z "$docker_root" ]; then - echo "Failed to determine Docker root directory." - exit 1 -fi -echo "Docker root directory: $docker_root" -# Check disk usage of the filesystem where Docker's root directory is located -disk_usage=$(df "$docker_root" | tail -1 | awk '{print $5}' | sed 's/%//') -# Define the threshold -threshold=70 -if [ "$disk_usage" -gt "$threshold" ]; then - echo "Disk usage is above $threshold%. Cleaning up Docker images and volumes..." - # Remove dangling images (those that are not tagged and not used by any container) - docker image prune -f - # Remove unused volumes / force the system prune for old images as well. - docker volume prune -f && docker system prune --force --filter "until=24h" --all - echo "Docker images and volumes cleanup completed." -else - echo "Disk usage is below $threshold%. No cleanup needed." -fi diff --git a/.buildkite/scripts/tpu/config_v6e_1.env b/.buildkite/scripts/tpu/config_v6e_1.env deleted file mode 100644 index c9e3c26571..0000000000 --- a/.buildkite/scripts/tpu/config_v6e_1.env +++ /dev/null @@ -1,14 +0,0 @@ -# Environment config -TEST_NAME=llama8b -CONTAINER_NAME=tpu-test - -# vllm config -MODEL=meta-llama/Llama-3.1-8B-Instruct -MAX_NUM_SEQS=256 -MAX_NUM_BATCHED_TOKENS=1024 -TENSOR_PARALLEL_SIZE=1 -MAX_MODEL_LEN=2048 -DOWNLOAD_DIR=/mnt/disks/persist -EXPECTED_THROUGHPUT=8.0 -INPUT_LEN=1800 -OUTPUT_LEN=128 diff --git a/.buildkite/scripts/tpu/docker_run_bm.sh b/.buildkite/scripts/tpu/docker_run_bm.sh deleted file mode 100755 index efb632e0a8..0000000000 --- a/.buildkite/scripts/tpu/docker_run_bm.sh +++ /dev/null @@ -1,91 +0,0 @@ -#!/bin/bash - -if [ ! -f "$1" ]; then - echo "Error: The env file '$1' does not exist." - exit 1 # Exit the script with a non-zero status to indicate an error -fi - -ENV_FILE=$1 - -# For testing on local vm, use `set -a` to export all variables -source /etc/environment -# shellcheck source=/dev/null -source "$ENV_FILE" - -remove_docker_container() { - docker rm -f "$CONTAINER_NAME" || true; -} - -trap remove_docker_container EXIT - -# Remove the container that might not be cleaned up in the previous run. -remove_docker_container - -LOG_ROOT=$(mktemp -d) -# If mktemp fails, set -e will cause the script to exit. -echo "Results will be stored in: $LOG_ROOT" - -if [ -z "$HF_TOKEN" ]; then - echo "Error: HF_TOKEN is not set or is empty." - exit 1 -fi - -# Make sure mounted disk or dir exists -if [ ! -d "$DOWNLOAD_DIR" ]; then - echo "Error: Folder $DOWNLOAD_DIR does not exist. This is useually a mounted drive. If no mounted drive, just create a folder." - exit 1 -fi - -echo "Run model $MODEL" -echo - -echo "starting docker...$CONTAINER_NAME" -echo -docker run \ - -v "$DOWNLOAD_DIR":"$DOWNLOAD_DIR" \ - --env-file "$ENV_FILE" \ - -e HF_TOKEN="$HF_TOKEN" \ - -e TARGET_COMMIT="$BUILDKITE_COMMIT" \ - -e MODEL="$MODEL" \ - -e WORKSPACE=/workspace \ - --name "$CONTAINER_NAME" \ - -d \ - --privileged \ - --network host \ - -v /dev/shm:/dev/shm \ - vllm/vllm-tpu-bm tail -f /dev/null - -echo "run script..." -echo -docker exec "$CONTAINER_NAME" /bin/bash -c ".buildkite/scripts/tpu/run_bm.sh" - -echo "copy result back..." -VLLM_LOG="$LOG_ROOT/$TEST_NAME"_vllm_log.txt -BM_LOG="$LOG_ROOT/$TEST_NAME"_bm_log.txt -docker cp "$CONTAINER_NAME:/workspace/vllm_log.txt" "$VLLM_LOG" -docker cp "$CONTAINER_NAME:/workspace/bm_log.txt" "$BM_LOG" - -throughput=$(grep "Request throughput (req/s):" "$BM_LOG" | sed 's/[^0-9.]//g') -echo "throughput for $TEST_NAME at $BUILDKITE_COMMIT: $throughput" - -if [ "$BUILDKITE" = "true" ]; then - echo "Running inside Buildkite" - buildkite-agent artifact upload "$VLLM_LOG" - buildkite-agent artifact upload "$BM_LOG" -else - echo "Not running inside Buildkite" -fi - -# -# compare the throughput with EXPECTED_THROUGHPUT -# and assert meeting the expectation -# -if [[ -z "$throughput" || ! "$throughput" =~ ^[0-9]+([.][0-9]+)?$ ]]; then - echo "Failed to get the throughput" - exit 1 -fi - -if (( $(echo "$throughput < $EXPECTED_THROUGHPUT" | bc -l) )); then - echo "Error: throughput($throughput) is less than expected($EXPECTED_THROUGHPUT)" - exit 1 -fi diff --git a/.buildkite/scripts/tpu/quantized_v6e_1.env b/.buildkite/scripts/tpu/quantized_v6e_1.env deleted file mode 100644 index ecb98d4516..0000000000 --- a/.buildkite/scripts/tpu/quantized_v6e_1.env +++ /dev/null @@ -1,14 +0,0 @@ -# Environment config -TEST_NAME=llama8bw8a8 -CONTAINER_NAME=tpu-test - -# vllm config -MODEL=RedHatAI/Meta-Llama-3.1-8B-Instruct-quantized.w8a8 -MAX_NUM_SEQS=128 -MAX_NUM_BATCHED_TOKENS=1024 -TENSOR_PARALLEL_SIZE=1 -MAX_MODEL_LEN=2048 -DOWNLOAD_DIR=/mnt/disks/persist -EXPECTED_THROUGHPUT=8.7 -INPUT_LEN=1800 -OUTPUT_LEN=128 diff --git a/.buildkite/scripts/tpu/run_bm.sh b/.buildkite/scripts/tpu/run_bm.sh deleted file mode 100755 index b5d001bea0..0000000000 --- a/.buildkite/scripts/tpu/run_bm.sh +++ /dev/null @@ -1,93 +0,0 @@ -#!/bin/bash - -set -euo pipefail - -VLLM_LOG="$WORKSPACE/vllm_log.txt" -BM_LOG="$WORKSPACE/bm_log.txt" - -if [ -n "$TARGET_COMMIT" ]; then - head_hash=$(git rev-parse HEAD) - if [ "$TARGET_COMMIT" != "$head_hash" ]; then - echo "Error: target commit $TARGET_COMMIT does not match HEAD: $head_hash" - exit 1 - fi -fi - -echo "model: $MODEL" -echo - -# -# create a log folder -# -mkdir "$WORKSPACE/log" - -# TODO: Move to image building. -pip install pandas -pip install datasets - -# -# create sonnet_4x -# -echo "Create sonnet_4x.txt" -echo "" > benchmarks/sonnet_4x.txt -for _ in {1..4} - do - cat benchmarks/sonnet.txt >> benchmarks/sonnet_4x.txt -done - -# -# start vllm service in backend -# -echo "lanching vllm..." -echo "logging to $VLLM_LOG" -echo - -vllm serve "$MODEL" \ - --seed 42 \ - --max-num-seqs "$MAX_NUM_SEQS" \ - --max-num-batched-tokens "$MAX_NUM_BATCHED_TOKENS" \ - --tensor-parallel-size "$TENSOR_PARALLEL_SIZE" \ - --no-enable-prefix-caching \ - --download_dir "$DOWNLOAD_DIR" \ - --max-model-len "$MAX_MODEL_LEN" > "$VLLM_LOG" 2>&1 & - - -echo "wait for 20 minutes.." -echo -# sleep 1200 -# wait for 10 minutes... -for _ in {1..120}; do - # TODO: detect other type of errors. - if grep -Fq "raise RuntimeError" "$VLLM_LOG"; then - echo "Detected RuntimeError, exiting." - exit 1 - elif grep -Fq "Application startup complete" "$VLLM_LOG"; then - echo "Application started" - break - else - echo "wait for 10 seconds..." - sleep 10 - fi -done - -# -# run test -# -echo "run benchmark test..." -echo "logging to $BM_LOG" -echo -vllm bench serve \ - --backend vllm \ - --model "$MODEL" \ - --dataset-name sonnet \ - --dataset-path benchmarks/sonnet_4x.txt \ - --sonnet-input-len "$INPUT_LEN" \ - --sonnet-output-len "$OUTPUT_LEN" \ - --ignore-eos > "$BM_LOG" - -echo "completed..." -echo - -throughput=$(grep "Request throughput (req/s):" "$BM_LOG" | sed 's/[^0-9.]//g') -echo "throughput: $throughput" -echo diff --git a/.buildkite/test-amd.yaml b/.buildkite/test-amd.yaml index 0a050b6979..a3f7dbd838 100644 --- a/.buildkite/test-amd.yaml +++ b/.buildkite/test-amd.yaml @@ -111,38 +111,6 @@ steps: # # ######################################################################################################################################### -- label: ":computer: (CPU) Rust Frontend Cargo Style + Clippy" # TBD - timeout_in_minutes: 35 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250] - dind: false - agent_pool: mi250_1 - no_gpu: true - optional: true - working_dir: "/vllm-workspace" - source_file_dependencies: - - rust/ - - rust-toolchain.toml - - .buildkite/test_areas/rust_frontend_cargo.yaml - - .buildkite/scripts/run-rust-frontend-cargo-ci.sh - commands: - - .buildkite/scripts/run-rust-frontend-cargo-ci.sh style-clippy - -- label: ":computer: (CPU) Rust Frontend Cargo" # TBD - timeout_in_minutes: 35 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250] - dind: false - agent_pool: mi250_1 - no_gpu: true - optional: true - working_dir: "/vllm-workspace" - source_file_dependencies: - - rust/ - - rust-toolchain.toml - - .buildkite/test_areas/rust_frontend_cargo.yaml - - .buildkite/scripts/run-rust-frontend-cargo-ci.sh - commands: - - .buildkite/scripts/run-rust-frontend-cargo-ci.sh test - - label: ":computer: (CPU) Docker Build Metadata" # TBD timeout_in_minutes: 35 mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250] @@ -269,8 +237,6 @@ steps: - tests/models/language/generation_ppl_test - vllm/model_executor/models/qwen3_5.py - vllm/model_executor/models/qwen3_5_mtp.py - - vllm/transformers_utils/configs/qwen3_5.py - - vllm/transformers_utils/configs/qwen3_5_moe.py - vllm/model_executor/models/qwen2.py - vllm/model_executor/models/qwen3.py - vllm/model_executor/models/qwen3_next.py @@ -725,28 +691,8 @@ steps: - vllm/platforms/rocm.py commands: - export VLLM_TEST_CLEAN_GPU_MEMORY=1 - - VLLM_TEST_CLEAN_GPU_MEMORY=1 pytest -v -s tests/compile/passes/distributed/test_async_tp.py - pytest -v -s tests/compile/fusions_e2e/test_tp2_ar_rms.py::test_tp2_ar_rms_fusions -k "dynamo_partition" -# - label: ":amd: (MI300) Sequence Parallel Correctness" # TBD -# timeout_in_minutes: 180 -# mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300] -# dind: false -# agent_pool: mi300_2 -# num_gpus: 2 -# optional: true -# working_dir: "/vllm-workspace/" -# source_file_dependencies: -# - vllm/model_executor/layers/ -# - vllm/compilation/ -# - vllm/v1/worker/ -# - vllm/v1/cudagraph_dispatcher.py -# - tests/compile/correctness_e2e/test_sequence_parallel.py -# - vllm/platforms/rocm.py -# commands: -# - export VLLM_TEST_CLEAN_GPU_MEMORY=1 -# - pytest -v -s tests/compile/correctness_e2e/test_sequence_parallel.py - - label: ":amd: (MI300) Distributed Compile + RPC" # TBD timeout_in_minutes: 65 mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300] @@ -1076,12 +1022,12 @@ steps: source_file_dependencies: - vllm/entrypoints - tests/entrypoints/unit_tests - - tests/entrypoints/weight_transfer + - tests/entrypoints/rl/weight_transfer - tests/entrypoints/launchers - vllm/platforms/rocm.py commands: - pytest -v -s entrypoints/unit_tests - - pytest -v -s entrypoints/weight_transfer + - pytest -v -s entrypoints/rl/weight_transfer - pytest -v -s entrypoints/launchers - label: ":amd: (MI300) Entrypoints Integration (Responses API)" # TBD @@ -2657,7 +2603,6 @@ steps: - vllm/model_executor/models/qwen3_next.py - vllm/model_executor/models/registry.py - vllm/transformers_utils/configs/extract_hidden_states.py - - vllm/transformers_utils/configs/qwen3_5.py - vllm/v1/attention/backends/ - vllm/v1/attention/selector.py - vllm/v1/kv_cache_interface.py @@ -3212,7 +3157,6 @@ steps: # TODO: FP4 output-quant and trained MXFP4 AR/RMS fusion remain unsupported. - amd-smi - VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_CUSTOM_AR=1 pytest -v -s tests/compile/fusions_e2e/test_tp2_ar_rms.py -k 'test_tp2_ar_rms_fp8_fusions and qwen3 and ROCM_AITER_UNIFIED_ATTN and +quant_fp8 and (dynamo_partition or not +rms_norm)' - - pytest -v -s tests/compile/fusions_e2e/test_tp2_rocm_moe.py - label: ":amd: (MI355) Fusion and Compile FP8" timeout_in_minutes: 45 @@ -3237,64 +3181,6 @@ steps: - pytest -v -s tests/compile/passes/test_fusion_attn.py tests/compile/passes/test_silu_mul_quant_fusion.py tests/compile/passes/test_silu_mul_quant_manual_fusion.py - pytest -v -s tests/compile/correctness_e2e/test_attn_quant.py -- label: ":amd: (MI355) Sequence Parallel Correctness" - timeout_in_minutes: 180 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx950nightly, amdmi355] - dind: false - agent_pool: mi355_2 - num_gpus: 2 - optional: true - working_dir: "/vllm-workspace/" - source_file_dependencies: - - .buildkite/test_areas/compile.yaml - - vllm/compilation/ - - vllm/config/ - - vllm/distributed/ - - vllm/model_executor/kernels/linear/ - - vllm/model_executor/layers/ - - vllm/model_executor/models/llama.py - - vllm/v1/attention/ - - vllm/v1/worker/ - - vllm/v1/cudagraph_dispatcher.py - - vllm/platforms/rocm.py - - tests/compile/correctness_e2e/test_sequence_parallel.py - - tests/models/registry.py - - tests/utils.py - commands: - - export VLLM_TEST_CLEAN_GPU_MEMORY=1 - - pytest -v -s tests/compile/correctness_e2e/test_sequence_parallel.py - -- label: ":amd: (MI355) AsyncTP Correctness" - timeout_in_minutes: 60 - mirror_hardwares: [amdexperimental, amdproduction, amdgfx950nightly, amdmi355] - dind: false - agent_pool: mi355_2 - num_gpus: 2 - optional: true - working_dir: "/vllm-workspace/" - source_file_dependencies: - - .buildkite/test_areas/compile.yaml - - vllm/compilation/ - - vllm/config/ - - vllm/distributed/ - - vllm/model_executor/layers/ - - vllm/model_executor/kernels/linear/ - - vllm/model_executor/models/llama.py - - vllm/model_executor/models/qwen3.py - - vllm/v1/attention/ - - vllm/_aiter_ops.py - - vllm/platforms/rocm.py - - vllm/transformers_utils/repo_utils.py - - tests/compile/correctness_e2e/test_async_tp.py - - tests/compile/fusions_e2e/common.py - - tests/models/registry.py - - tests/utils.py - commands: - - export VLLM_TEST_CLEAN_GPU_MEMORY=1 - - pytest -v -s tests/compile/correctness_e2e/test_async_tp.py::test_rocm_async_tp_bf16_output_correctness - - pytest -v -s tests/compile/correctness_e2e/test_async_tp.py::test_async_tp_pass_correctness -k RedHatAI - - pytest -v -s tests/compile/correctness_e2e/test_async_tp.py::test_async_tp_pass_nvfp4_correctness - - label: ":amd: (MI355 DPX) Fusion E2E FP8 Config Sweep" timeout_in_minutes: 45 mirror_hardwares: [amdexperimental, amdproduction, amdgfx950nightly, amdmi355] @@ -3778,11 +3664,13 @@ steps: - vllm/ - tests/entrypoints/serve - tests/entrypoints/scale_out + - tests/entrypoints/rl commands: - export VLLM_WORKER_MULTIPROC_METHOD=spawn - pytest -v -s entrypoints/serve --ignore=entrypoints/serve/dev/rpc --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT --shard-id=$$BUILDKITE_PARALLEL_JOB - if [ "$$BUILDKITE_PARALLEL_JOB" = "1" ]; then PYTHONPATH=/vllm-workspace pytest -v -s entrypoints/serve/dev/rpc; fi - pytest -v -s entrypoints/scale_out --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT --shard-id=$$BUILDKITE_PARALLEL_JOB + - pytest -v -s entrypoints/rl --ignore=entrypoints/rl/weight_transfer --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT --shard-id=$$BUILDKITE_PARALLEL_JOB - label: ":amd: (MI355 DPX) Entrypoints Integration (OpenAI API completion)" # TBD timeout_in_minutes: 65 @@ -4023,8 +3911,6 @@ steps: source_file_dependencies: - vllm/model_executor/models/qwen3_5.py - vllm/model_executor/models/qwen3_5_mtp.py - - vllm/transformers_utils/configs/qwen3_5.py - - vllm/transformers_utils/configs/qwen3_5_moe.py - vllm/model_executor/models/qwen2.py - vllm/model_executor/models/qwen3.py - vllm/model_executor/models/qwen3_next.py @@ -4066,8 +3952,6 @@ steps: - vllm/model_executor/models/qwen3_next_mtp.py - vllm/model_executor/models/nemotron_h.py - vllm/model_executor/models/nemotron_h_mtp.py - - vllm/transformers_utils/configs/qwen3_next.py - - vllm/transformers_utils/configs/nemotron_h.py - vllm/model_executor/model_loader/ - vllm/model_executor/layers/quantization/ - vllm/model_executor/layers/fused_moe/ @@ -4265,7 +4149,6 @@ steps: - tests/evals/gsm8k/ - tests/utils.py - vllm/distributed/ - - vllm/transformers_utils/configs/diffusion_gemma.py - vllm/v1/attention/ops/ - tests/evals/gsm8k/configs/models-small-tp-mi355.txt commands: @@ -5157,8 +5040,6 @@ steps: source_file_dependencies: - vllm/model_executor/models/qwen3_5.py - vllm/model_executor/models/qwen3_5_mtp.py - - vllm/transformers_utils/configs/qwen3_5.py - - vllm/transformers_utils/configs/qwen3_5_moe.py - vllm/model_executor/models/qwen2.py - vllm/model_executor/models/qwen3.py - vllm/model_executor/models/qwen3_next.py @@ -5888,6 +5769,7 @@ steps: timeout_in_minutes: 20 source_file_dependencies: - tests/models/test_deepseek_v4_vl_rocm.py + - tests/models/test_deepseek_v4_aiter_mega_moe.py - vllm/models/deepseek_v4/amd/ - vllm/models/deepseek_v4/common/ - vllm/model_executor/layers/fused_moe/ @@ -5901,6 +5783,7 @@ steps: - vllm/envs.py commands: - VLLM_ROCM_USE_AITER_MOE=0 pytest -v -s models/test_deepseek_v4_vl_rocm.py + - pytest -v -s models/test_deepseek_v4_aiter_mega_moe.py dind: false working_dir: /vllm-workspace/tests agent_pool: mi355_1 @@ -5944,35 +5827,6 @@ steps: - amdmi355 optional: true -- label: ':amd: (MI355) Fusion E2E TP2 AsyncTP BF16 Config Sweep' - timeout_in_minutes: 30 - working_dir: /vllm-workspace/ - source_file_dependencies: - - vllm/compilation/ - - vllm/model_executor/layers/linear.py - - vllm/model_executor/layers/utils.py - - vllm/distributed/ - - vllm/v1/attention/backends/triton_attn.py - - tests/compile/fusions_e2e/ - - tests/compile/passes/distributed/test_async_tp.py - - tests/compile/rocm/test_async_tp.py - - tests/compile/correctness_e2e/test_async_tp.py - - vllm/model_executor/kernels/linear/ - - vllm/model_executor/models/qwen3.py - commands: - - rocm-smi - - pytest -v -s tests/compile/rocm/test_async_tp.py - - pytest -v -s tests/compile/fusions_e2e/test_tp2_rocm_async_tp.py - - pytest -v -s tests/compile/correctness_e2e/test_async_tp.py::test_rocm_async_tp_bf16_output_correctness - dind: false - agent_pool: mi355_2 - num_gpus: 2 - mirror_hardwares: - - amdexperimental - - amdproduction - - amdgfx950nightly - - amdmi355 - optional: true - label: ":amd: (MI355 DPX) MRV2 Sampler JIT Warmup" timeout_in_minutes: 10 diff --git a/.buildkite/test_areas/benchmarks.yaml b/.buildkite/test_areas/benchmarks.yaml index 4b681cd589..40f8f97963 100644 --- a/.buildkite/test_areas/benchmarks.yaml +++ b/.buildkite/test_areas/benchmarks.yaml @@ -4,7 +4,7 @@ depends_on: steps: - label: ":nvidia: (H200 MIG 18GB) Benchmarks CLI" key: benchmarks-cli-test - timeout_in_minutes: 45 + timeout_in_minutes: 55 device: h200_18gb source_file_dependencies: - vllm/ diff --git a/.buildkite/test_areas/compile.yaml b/.buildkite/test_areas/compile.yaml index acc1bafa7f..4a5a4f1a3c 100644 --- a/.buildkite/test_areas/compile.yaml +++ b/.buildkite/test_areas/compile.yaml @@ -2,84 +2,6 @@ group: Compile depends_on: - image-build steps: -- label: ":nvidia: (B200) Sequence Parallel Correctness" - key: sequence-parallel-correctness-tests-2xb200 - timeout_in_minutes: 45 - working_dir: "/vllm-workspace/" - device: b200-k8s - optional: true - num_devices: 2 - commands: - - export VLLM_TEST_CLEAN_GPU_MEMORY=1 - - pytest -v -s tests/compile/correctness_e2e/test_sequence_parallel.py - mirror: - amd: - label: ":amd: (MI355) Sequence Parallel Correctness" - dind: false - device: mi355_2 - num_devices: 2 - timeout_in_minutes: 180 - depends_on: - - image-build-amd - source_file_dependencies: - - .buildkite/test_areas/compile.yaml - - vllm/compilation/ - - vllm/config/ - - vllm/distributed/ - - vllm/model_executor/kernels/linear/ - - vllm/model_executor/layers/ - - vllm/model_executor/models/llama.py - - vllm/v1/attention/ - - vllm/v1/worker/ - - vllm/v1/cudagraph_dispatcher.py - - vllm/platforms/rocm.py - - tests/compile/correctness_e2e/test_sequence_parallel.py - - tests/models/registry.py - - tests/utils.py - -- label: ":nvidia: (B200) AsyncTP Correctness" - key: asynctp-correctness-tests-b200 - timeout_in_minutes: 30 - working_dir: "/vllm-workspace/" - device: b200-k8s - optional: true - num_devices: 2 - commands: - - export VLLM_TEST_CLEAN_GPU_MEMORY=1 - - pytest -v -s tests/compile/correctness_e2e/test_async_tp.py - mirror: - amd: - label: ":amd: (MI355) AsyncTP Correctness" - dind: false - device: mi355_2 - num_devices: 2 - timeout_in_minutes: 60 - working_dir: "/vllm-workspace/" - depends_on: - - image-build-amd - source_file_dependencies: - - .buildkite/test_areas/compile.yaml - - vllm/compilation/ - - vllm/config/ - - vllm/distributed/ - - vllm/model_executor/layers/ - - vllm/model_executor/kernels/linear/ - - vllm/model_executor/models/llama.py - - vllm/model_executor/models/qwen3.py - - vllm/v1/attention/ - - vllm/_aiter_ops.py - - vllm/platforms/rocm.py - - vllm/transformers_utils/repo_utils.py - - tests/compile/correctness_e2e/test_async_tp.py - - tests/compile/fusions_e2e/common.py - - tests/models/registry.py - - tests/utils.py - commands: - - export VLLM_TEST_CLEAN_GPU_MEMORY=1 - - pytest -v -s tests/compile/correctness_e2e/test_async_tp.py::test_rocm_async_tp_bf16_output_correctness - - pytest -v -s tests/compile/correctness_e2e/test_async_tp.py::test_async_tp_pass_correctness -k RedHatAI - - pytest -v -s tests/compile/correctness_e2e/test_async_tp.py::test_async_tp_pass_nvfp4_correctness - - label: ":nvidia: (H100) Distributed Compile" key: distributed-compile-unit-tests-2xh100 timeout_in_minutes: 45 @@ -300,7 +222,6 @@ steps: - nvidia-smi # Run all models and attn backends but only Inductor partition and native custom ops - pytest -v -s tests/compile/fusions_e2e/test_tp2_ar_rms.py -k "inductor_partition and not +rms_norm and (not +quant_fp8 or +quant_fp8 and (qwen3 or deepseek))" - - pytest -v -s tests/compile/fusions_e2e/test_tp2_async_tp.py -k "inductor_partition and not +rms_norm and (not +quant_fp8 or +quant_fp8 and (qwen3 or deepseek))" mirror: amd: label: ":amd: (MI355) Fusion E2E TP2 Quick" @@ -388,53 +309,6 @@ steps: - amd-smi - VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_CUSTOM_AR=1 pytest -v -s tests/compile/fusions_e2e/test_tp2_ar_rms.py -k 'test_tp2_ar_rms_fp8_fusions and Llama-3 and ROCM_AITER_UNIFIED_ATTN and (dynamo_partition or +quant_fp8 or +rms_norm)' -- label: ":nvidia: (H100) Fusion E2E TP2 AsyncTP Config Sweep" - key: fusion-e2e-tp2-asynctp-config-sweep-h100 - timeout_in_minutes: 40 - working_dir: "/vllm-workspace/" - device: h100 - num_devices: 2 - source_file_dependencies: - - csrc/quantization/ - - vllm/compilation/ - # can affect pattern matching - - vllm/model_executor/layers/layernorm.py - - vllm/model_executor/layers/activation.py - - vllm/model_executor/layers/attention/attention.py - - vllm/model_executor/layers/quantization/input_quant_fp8.py - - tests/compile/fusions_e2e/ - commands: - - nvidia-smi - # Run just llama3 (fp8 & bf16) for all config combinations - - pytest -v -s tests/compile/fusions_e2e/test_tp2_async_tp.py -k "llama-3" - mirror: - amd: - label: ':amd: (MI355) Fusion E2E TP2 AsyncTP BF16 Config Sweep' - dind: false - device: mi355_2 - num_devices: 2 - working_dir: /vllm-workspace/ - timeout_in_minutes: 30 - depends_on: - - image-build-amd - source_file_dependencies: - - vllm/compilation/ - - vllm/model_executor/layers/linear.py - - vllm/model_executor/layers/utils.py - - vllm/distributed/ - - vllm/v1/attention/backends/triton_attn.py - - tests/compile/fusions_e2e/ - - tests/compile/passes/distributed/test_async_tp.py - - tests/compile/rocm/test_async_tp.py - - tests/compile/correctness_e2e/test_async_tp.py - - vllm/model_executor/kernels/linear/ - - vllm/model_executor/models/qwen3.py - commands: - - rocm-smi - - pytest -v -s tests/compile/rocm/test_async_tp.py - - pytest -v -s tests/compile/fusions_e2e/test_tp2_rocm_async_tp.py - - pytest -v -s tests/compile/correctness_e2e/test_async_tp.py::test_rocm_async_tp_bf16_output_correctness - - label: ":nvidia: (B200) Fusion E2E TP2" key: fusion-e2e-tp2-b200 timeout_in_minutes: 45 @@ -453,7 +327,6 @@ steps: # include qwen/deepseek with +quant_fp8 as -quant_fp8 rms+quant fusion is not supported # for ar-rms-quant-fp4, also sweep llama3 - pytest -v -s tests/compile/fusions_e2e/test_tp2_ar_rms.py -k "(FLASHINFER and inductor_partition and not +rms_norm and (not +quant_fp8 or +quant_fp8 and (qwen3 or deepseek))) or Llama-3.1-8B-Instruct-FP4" - - pytest -v -s tests/compile/fusions_e2e/test_tp2_async_tp.py -k "FLASHINFER and inductor_partition and not +rms_norm and (not +quant_fp8 or +quant_fp8 and (qwen3 or deepseek))" mirror: amd: label: ":amd: (MI355) Fusion E2E TP2 FP8 BF16 rewrites" @@ -480,7 +353,6 @@ steps: # TODO: FP4 output-quant and trained MXFP4 AR/RMS fusion remain unsupported. - amd-smi - VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_CUSTOM_AR=1 pytest -v -s tests/compile/fusions_e2e/test_tp2_ar_rms.py -k 'test_tp2_ar_rms_fp8_fusions and qwen3 and ROCM_AITER_UNIFIED_ATTN and +quant_fp8 and (dynamo_partition or not +rms_norm)' - - pytest -v -s tests/compile/fusions_e2e/test_tp2_rocm_moe.py - label: ':amd: (MI355) Fusion E2E TP2 AR-RMS (Inductor Partition)' key: fusion-e2e-tp2-ar-rms-inductor-partition-amd diff --git a/.buildkite/test_areas/cuda.yaml b/.buildkite/test_areas/cuda.yaml index 02ac11879f..b3bfa9f097 100644 --- a/.buildkite/test_areas/cuda.yaml +++ b/.buildkite/test_areas/cuda.yaml @@ -33,14 +33,20 @@ steps: - label: ":nvidia: (H200 MIG 18GB) CUDAGraph" device: h200_18gb key: cudagraph - timeout_in_minutes: 30 + automatic_shard: true + timeout_in_minutes: 45 source_file_dependencies: - tests/v1/cudagraph - vllm/v1/cudagraph_dispatcher.py - vllm/config/compilation.py + - vllm/config/vllm.py - vllm/compilation + - vllm/v1/worker/gpu/cudagraph_utils.py + - vllm/v1/worker/gpu/dp_utils.py - vllm/v1/worker/encoder_cudagraph.py - vllm/v1/worker/encoder_cudagraph_defs.py + - vllm/v1/worker/gpu/model_runner.py + - vllm/v1/worker/gpu/ubatch_utils.py commands: - pytest -v -s v1/cudagraph/test_cudagraph_dispatch.py - pytest -v -s v1/cudagraph/test_cudagraph_mode.py diff --git a/.buildkite/test_areas/distributed.yaml b/.buildkite/test_areas/distributed.yaml index 3710f80bfd..78a6af2e0c 100644 --- a/.buildkite/test_areas/distributed.yaml +++ b/.buildkite/test_areas/distributed.yaml @@ -397,6 +397,7 @@ steps: - label: ":nvidia: (H100) Distributed DP + EP" key: distributed-tests-8xh100 timeout_in_minutes: 20 + optional: true device: h100 num_devices: 8 working_dir: "/vllm-workspace/tests" @@ -419,6 +420,7 @@ steps: amd: label: ":amd: (MI355) Distributed DP + EP" dind: false + optional: false device: mi355_8 timeout_in_minutes: 50 depends_on: diff --git a/.buildkite/test_areas/engine.yaml b/.buildkite/test_areas/engine.yaml index 53499d4a98..26485e0a17 100644 --- a/.buildkite/test_areas/engine.yaml +++ b/.buildkite/test_areas/engine.yaml @@ -38,6 +38,7 @@ steps: - label: ":nvidia: (L4) V1 Engine" key: engine-1-gpu + automatic_shard: true device: l4 num_devices: 1 timeout_in_minutes: 45 @@ -108,6 +109,7 @@ steps: - label: ":nvidia: (H200 MIG 18GB) E2E Core" device: h200_18gb key: e2e-core-1-gpu + automatic_shard: true timeout_in_minutes: 65 source_file_dependencies: - vllm/v1/ diff --git a/.buildkite/test_areas/entrypoints.yaml b/.buildkite/test_areas/entrypoints.yaml index 5f8e1d1c35..d31c66a8cb 100644 --- a/.buildkite/test_areas/entrypoints.yaml +++ b/.buildkite/test_areas/entrypoints.yaml @@ -26,16 +26,15 @@ steps: - label: ":nvidia: (H200 MIG 35GB) Entrypoints Unit" device: h200_35gb key: entrypoints-unit-tests + automatic_shard: true timeout_in_minutes: 40 working_dir: "/vllm-workspace/tests" source_file_dependencies: - vllm/entrypoints - tests/entrypoints/unit_tests - - tests/entrypoints/weight_transfer - tests/entrypoints/launchers commands: - pytest -v -s entrypoints/unit_tests - - pytest -v -s entrypoints/weight_transfer - pytest -v -s entrypoints/launchers mirror: amd: @@ -170,6 +169,7 @@ steps: - label: ":nvidia: (H200 MIG 35GB) Entrypoints Integration (Responses API)" device: h200_35gb key: entrypoints-integration-responses-api + automatic_shard: true timeout_in_minutes: 50 working_dir: "/vllm-workspace/tests" source_file_dependencies: @@ -190,6 +190,7 @@ steps: - label: ":nvidia: (H200 MIG 35GB) Entrypoints Integration (Speech to Text)" device: h200_35gb key: entrypoints-integration-speech_to_text + automatic_shard: true timeout_in_minutes: 45 working_dir: "/vllm-workspace/tests" source_file_dependencies: @@ -233,6 +234,7 @@ steps: - label: ":nvidia: (H200 MIG 35GB) Entrypoints Integration (Pooling)" device: h200_35gb key: entrypoints-integration-pooling + automatic_shard: true timeout_in_minutes: 75 working_dir: "/vllm-workspace/tests" source_file_dependencies: diff --git a/.buildkite/test_areas/kernels.yaml b/.buildkite/test_areas/kernels.yaml index 515c3966f5..aa0971b4c5 100644 --- a/.buildkite/test_areas/kernels.yaml +++ b/.buildkite/test_areas/kernels.yaml @@ -155,6 +155,7 @@ steps: - image-build-amd source_file_dependencies: - tests/models/test_deepseek_v4_vl_rocm.py + - tests/models/test_deepseek_v4_aiter_mega_moe.py - vllm/models/deepseek_v4/amd/ - vllm/models/deepseek_v4/common/ - vllm/model_executor/layers/fused_moe/ @@ -170,6 +171,7 @@ steps: # Common qnorm/RoPE/cache kernels already run in the H100 AMD mirror. # Pin the tested BF16 Triton routing and shared-expert path before imports. - VLLM_ROCM_USE_AITER_MOE=0 pytest -v -s models/test_deepseek_v4_vl_rocm.py + - pytest -v -s models/test_deepseek_v4_aiter_mega_moe.py # Own job so mHC never shares a process with FlashInfer's import-time # ptxas override. @@ -254,7 +256,9 @@ steps: --ignore=kernels/test_fused_qk_norm_rope_gate.py --ignore=kernels/test_fused_deepseek_v4_qnorm_rope_kv_insert.py --ignore=kernels/test_top_k_per_row.py + --ignore=kernels/test_cpu_fused_sampling.py --ignore=kernels/test_bf16x3_router_gemm_cutedsl.py + --ignore=kernels/test_bf16x3_router_gemm_rocm.py --ignore=kernels/test_ll_bf16_gemm.py --ignore=kernels/test_shuffle_rows.py # BROKEN on main, pending kernel fixes (B200): @@ -280,6 +284,7 @@ steps: kernels/test_apply_repetition_penalties.py kernels/test_awq_int4_to_int8.py kernels/test_bf16_skinny_gemm.py + kernels/test_bf16x3_router_gemm_rocm.py kernels/test_cache_kernels.py kernels/test_compressor_kv_cache.py kernels/test_cp_gather_fp8.py @@ -499,6 +504,7 @@ steps: - label: ":nvidia: (H200 MIG 35GB) Mamba Kernels" device: h200_35gb key: kernels-mamba-test + automatic_shard: true timeout_in_minutes: 60 source_file_dependencies: - csrc/mamba/ @@ -546,6 +552,21 @@ steps: - pytest -v -s kernels/attention/test_deepgemm_attention.py - pytest -v -s quantization/test_cutlass_w4a16.py +- label: ":nvidia: (B200) MiniMax-M3 Attention Kernels" + key: kernels-minimax-m3-attention-b200 + timeout_in_minutes: 30 + working_dir: "/vllm-workspace/" + device: b200-k8s + num_devices: 1 + source_file_dependencies: + - vllm/models/minimax_m3/ + - vllm/config/cache.py + - vllm/v1/kv_cache_interface.py + - tests/v1/attention/utils.py + - tests/kernels/attention/test_minimax_m3.py + commands: + - pytest -v -s tests/kernels/attention/test_minimax_m3.py + - label: ":nvidia: (B200) Kernels Shard %N" key: kernels-b200 timeout_in_minutes: 30 @@ -595,7 +616,6 @@ steps: - tests/kernels/test_fused_minimax_m3_qknorm_rope_kv_insert.py - tests/kernels/mamba/test_gdn_prefill_cutedsl.py - tests/kernels/test_bf16x3_router_gemm_cutedsl.py - - tests/kernels/attention/test_minimax_m3.py - tests/kernels/test_ll_bf16_gemm.py - tests/kernels/test_top_k_per_row.py - tests/kernels/quantization/test_silu_mul_nvfp4_quant.py @@ -641,7 +661,6 @@ steps: pytest -v -s tests/kernels/moe/test_flashinfer.py pytest -v -s tests/kernels/moe/test_trtllm_nvfp4_moe.py pytest -v -s tests/kernels/mamba/test_gdn_prefill_cutedsl.py - pytest -v -s tests/kernels/attention/test_minimax_m3.py ;; *) echo "unexpected BUILDKITE_PARALLEL_JOB=$$BUILDKITE_PARALLEL_JOB" >&2 diff --git a/.buildkite/test_areas/lm_eval.yaml b/.buildkite/test_areas/lm_eval.yaml index 04337d03d2..0660795105 100644 --- a/.buildkite/test_areas/lm_eval.yaml +++ b/.buildkite/test_areas/lm_eval.yaml @@ -5,6 +5,7 @@ steps: - label: ":nvidia: (H200 MIG 35GB) LM Eval Small Models" device: h200_35gb key: lm-eval-small-models + automatic_shard: true timeout_in_minutes: 45 source_file_dependencies: - csrc/ @@ -208,7 +209,6 @@ steps: - tests/evals/gsm8k/ - tests/utils.py - vllm/distributed/ - - vllm/transformers_utils/configs/diffusion_gemma.py - vllm/v1/attention/ops/ commands: - pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-mi3xx-fp8-and-mixed.txt @@ -270,6 +270,9 @@ steps: source_file_dependencies: - .buildkite/test_areas/lm_eval.yaml - tests/evals/gsm8k/ + - tests/watermarking/test_compatibility.py + - tests/watermarking/test_engine_compatibility.py + - tests/watermarking/test_parallel_compatibility.py - vllm/config/speculative.py - vllm/config/vllm.py - vllm/config/watermarking.py @@ -282,6 +285,7 @@ steps: autorun_on_main: true commands: - pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-watermark-dspark-tp2.txt + - pytest -s -v watermarking/test_compatibility.py watermarking/test_engine_compatibility.py watermarking/test_parallel_compatibility.py mirror: amd: label: ":amd: (MI355) LM Eval Watermark Feature Combination" @@ -318,9 +322,11 @@ steps: optional: true source_file_dependencies: - tests/evals/gsm8k/configs/DeepSeek-V4-Flash-DSpark-confidence-TEP4.yaml + - tests/evals/gsm8k/configs/DeepSeek-V4.1-Flash-NVFP4-DSpark-TP4.yaml - tests/evals/gsm8k/configs/Kimi-K3-pruned75-DSpark-TP4.yaml - tests/evals/gsm8k/configs/models-spec-decode.txt - vllm/models/deepseek_v4/ + - vllm/models/deepseek_v41/ - vllm/models/kimi_k3/ - vllm/model_executor/models/qwen3_dspark.py - vllm/v1/spec_decode/metrics.py @@ -411,8 +417,6 @@ steps: - vllm/model_executor/models/qwen3_next_mtp.py - vllm/model_executor/models/nemotron_h.py - vllm/model_executor/models/nemotron_h_mtp.py - - vllm/transformers_utils/configs/qwen3_next.py - - vllm/transformers_utils/configs/nemotron_h.py - vllm/model_executor/model_loader/ - vllm/model_executor/layers/quantization/ - vllm/model_executor/layers/fused_moe/ @@ -439,8 +443,6 @@ steps: source_file_dependencies: - vllm/model_executor/models/qwen3_5.py - vllm/model_executor/models/qwen3_5_mtp.py - - vllm/transformers_utils/configs/qwen3_5.py - - vllm/transformers_utils/configs/qwen3_5_moe.py - vllm/model_executor/models/qwen3_next.py - vllm/model_executor/models/qwen3_next_mtp.py - vllm/third_party/flash_linear_attention/ops/ @@ -457,8 +459,6 @@ steps: source_file_dependencies: - vllm/model_executor/models/qwen3_5.py - vllm/model_executor/models/qwen3_5_mtp.py - - vllm/transformers_utils/configs/qwen3_5.py - - vllm/transformers_utils/configs/qwen3_5_moe.py - vllm/model_executor/models/qwen2.py - vllm/model_executor/models/qwen3.py - vllm/model_executor/models/qwen3_next.py diff --git a/.buildkite/test_areas/misc.yaml b/.buildkite/test_areas/misc.yaml index cbcfaeaea7..93f59b50e5 100644 --- a/.buildkite/test_areas/misc.yaml +++ b/.buildkite/test_areas/misc.yaml @@ -4,7 +4,8 @@ depends_on: steps: - label: ":nvidia: (H200 MIG 18GB) V1 Sample" key: v1-sample - timeout_in_minutes: 35 + automatic_shard: true + timeout_in_minutes: 45 device: h200_18gb source_file_dependencies: &v1-sample-logits-deps - vllm/config/ @@ -38,7 +39,7 @@ steps: - label: ":nvidia: (H200 MIG 18GB) V1 Logits + Oracle" key: v1-logits-oracle - timeout_in_minutes: 35 + timeout_in_minutes: 45 device: h200_18gb source_file_dependencies: *v1-sample-logits-deps commands: @@ -296,7 +297,6 @@ steps: - vllm/model_executor/models/qwen3_next.py - vllm/model_executor/models/registry.py - vllm/transformers_utils/configs/extract_hidden_states.py - - vllm/transformers_utils/configs/qwen3_5.py - vllm/v1/attention/backends/ - vllm/v1/attention/selector.py - vllm/v1/kv_cache_interface.py @@ -556,6 +556,8 @@ steps: - tests/multimodal - tests/renderers - tests/standalone_tests/lazy_imports.py + - tests/standalone_tests/test_precompiled_wheel.py + - setup.py - tests/reasoning - tools/pre_commit/check_test_tethering.py - tools/pre_commit/test_tethering_allowlist.txt @@ -572,6 +574,7 @@ steps: commands: - pip install cmake # test_cmake_utils.py shells out to cmake; the CPU image has none - python3 standalone_tests/lazy_imports.py + - python3 standalone_tests/test_precompiled_wheel.py - pytest -v -s device_allocator - pytest -v -s test_uvicorn_logging.py - pytest -v -s test_attention_backend_registry.py diff --git a/.buildkite/test_areas/models_basic.yaml b/.buildkite/test_areas/models_basic.yaml index c2b2b054a3..fd21324a7e 100644 --- a/.buildkite/test_areas/models_basic.yaml +++ b/.buildkite/test_areas/models_basic.yaml @@ -68,8 +68,10 @@ steps: - tests/models/test_deepseek_v4_vl_rocm.py - tests/models/test_deepseek_v41_decoder_replay_layers.py - tests/models/test_deepseek_v41_replay_batch.py + - tests/models/transformers/test_layer_registry_e2e.py + - tests/models/test_deepseek_v41_replay_start.py commands: - - pytest -v -s models/test_terratorch.py models/transformers/test_backend.py models/test_registry.py models/test_deepseek_v4_vl_rocm.py models/test_deepseek_v41_decoder_replay_layers.py models/test_deepseek_v41_replay_batch.py + - pytest -v -s models/test_terratorch.py models/transformers/test_backend.py models/test_registry.py models/test_deepseek_v4_vl_rocm.py models/transformers/test_layer_registry_e2e.py models/test_deepseek_v41_replay_start.py models/test_deepseek_v41_decoder_replay_layers.py models/test_deepseek_v41_replay_batch.py mirror: amd: label: ":amd: (MI355) Basic Models (Other)" @@ -79,10 +81,18 @@ steps: depends_on: - image-build-amd source_file_dependencies: + - vllm/models/deepseek_v4/ - tests/models/test_deepseek_v4_vl_rocm.py - tests/models/test_hyv4_rocm.py + - tests/models/transformers/test_layer_registry_e2e.py + - tests/models/test_deepseek_v41_replay_start.py + - tests/models/test_deepseek_v41_decoder_replay_layers.py + - tests/models/test_deepseek_v41_replay_batch.py + - tests/models/test_deepseek_v4_dspark_rocm.py + - tests/models/test_deepseek_v4_rocm_wo_a.py + - tests/models/test_deepseek_v4_rocm_compressor_gemm_fusion.py commands: - - pytest -v -s models/test_terratorch.py models/transformers/test_backend.py models/test_registry.py models/test_deepseek_v4_vl_rocm.py + - pytest -v -s models/test_terratorch.py models/transformers/test_backend.py models/test_registry.py models/test_deepseek_v4_vl_rocm.py models/transformers/test_layer_registry_e2e.py models/test_deepseek_v41_replay_start.py models/test_deepseek_v41_decoder_replay_layers.py models/test_deepseek_v41_replay_batch.py models/test_deepseek_v4_dspark_rocm.py models/test_deepseek_v4_rocm_wo_a.py models/test_deepseek_v4_rocm_compressor_gemm_fusion.py - VLLM_ROCM_USE_AITER=1 pytest -v -s models/test_hyv4_rocm.py -m 'not distributed' - label: ":nvidia: (B200) Inkling" @@ -206,7 +216,7 @@ steps: depends_on: - image-build-amd commands: - - pytest -v -s models/qwen4_exp/test_qsa_amd.py + - pytest -v -s models/qwen4_exp/test_qsa_amd.py models/qwen4_exp/test_qsa_prepare_amd.py # PLE and QSA import conflicting HC registrations in a shared process. - pytest -v -s models/qwen4_exp/test_ple.py @@ -229,7 +239,7 @@ steps: key: basic-models-test-other-cpu depends_on: - image-build-cpu - timeout_in_minutes: 20 + timeout_in_minutes: 30 source_file_dependencies: - vllm/ - "!vllm/distributed/kv_transfer/" @@ -239,6 +249,11 @@ steps: - tests/models/test_qwen3_5_mtp_config.py - tests/models/transformers/fusers/ - tests/model_executor/layers/test_activation.py + - tests/models/test_language_model_cache_is_weak.py + - tests/models/deepseek_v32/test_sequence_parallel.py + - tests/models/test_deepseek_v4_fi_moe_ep.py + - tests/models/test_dspark_mla.py + - tests/models/transformers/test_layer_registry.py device: cpu-small commands: - - pytest -v -s models/test_utils.py models/test_vision.py models/test_adapters.py models/test_qwen3_5_mtp_config.py models/transformers/fusers/ model_executor/layers/test_activation.py + - pytest -v -s models/test_utils.py models/test_vision.py models/test_adapters.py models/test_qwen3_5_mtp_config.py models/transformers/fusers/ model_executor/layers/test_activation.py models/test_language_model_cache_is_weak.py models/deepseek_v32/test_sequence_parallel.py models/test_deepseek_v4_fi_moe_ep.py models/test_dspark_mla.py models/transformers/test_layer_registry.py diff --git a/.buildkite/test_areas/models_language.yaml b/.buildkite/test_areas/models_language.yaml index fa8013ec54..002fad8ac6 100644 --- a/.buildkite/test_areas/models_language.yaml +++ b/.buildkite/test_areas/models_language.yaml @@ -184,8 +184,6 @@ steps: source_file_dependencies: - vllm/model_executor/models/qwen3_5.py - vllm/model_executor/models/qwen3_5_mtp.py - - vllm/transformers_utils/configs/qwen3_5.py - - vllm/transformers_utils/configs/qwen3_5_moe.py - vllm/model_executor/models/qwen2.py - vllm/model_executor/models/qwen3.py - vllm/model_executor/models/qwen3_next.py @@ -222,10 +220,10 @@ steps: depends_on: - image-build-amd -- label: ":nvidia: (H200 MIG 18GB) Language Models (MTEB)" +- label: ":nvidia: (H200 MIG 35GB) Language Models (MTEB)" key: language-models-test-mteb timeout_in_minutes: 68 - device: h200_18gb + device: h200_35gb optional: true source_file_dependencies: - vllm/ diff --git a/.buildkite/test_areas/pytorch.yaml b/.buildkite/test_areas/pytorch.yaml index 0ba608e725..60061af414 100644 --- a/.buildkite/test_areas/pytorch.yaml +++ b/.buildkite/test_areas/pytorch.yaml @@ -123,6 +123,7 @@ steps: - label: ":nvidia: (L4) PyTorch Compilation Passes" key: pytorch-compilation-passes-unit-tests + automatic_shard: true device: l4 num_devices: 1 timeout_in_minutes: 45 diff --git a/.buildkite/test_areas/quantization.yaml b/.buildkite/test_areas/quantization.yaml index e73054a4bc..55cef9fbd3 100644 --- a/.buildkite/test_areas/quantization.yaml +++ b/.buildkite/test_areas/quantization.yaml @@ -12,6 +12,7 @@ steps: source_file_dependencies: - csrc/ - vllm/model_executor/layers/quantization + - vllm/models/deepseek_v41/quant_config.py - tests/quantization commands: # Pin torchao to the version compatible with the CI PyTorch and CUDA stack. @@ -79,6 +80,7 @@ steps: - label: ":nvidia: (B200) Quantized MoE" key: quantized-moe-test-b200 + automatic_shard: true timeout_in_minutes: 120 working_dir: "/vllm-workspace/" device: b200-k8s @@ -126,6 +128,7 @@ steps: - label: ":nvidia: (H200 MIG 35GB) Quantized Models" device: h200_35gb key: quantized-models-test + automatic_shard: true timeout_in_minutes: 65 env: VLLM_USE_V2_MODEL_RUNNER: "0" diff --git a/.buildkite/test_areas/rl.yaml b/.buildkite/test_areas/rl.yaml new file mode 100644 index 0000000000..7141a81bde --- /dev/null +++ b/.buildkite/test_areas/rl.yaml @@ -0,0 +1,25 @@ +group: RL +depends_on: + - image-build +steps: +- label: ":nvidia: (H200 MIG 35GB) RL Entrypoints" + device: h200_35gb + key: rl-entrypoints-tests + timeout_in_minutes: 40 + working_dir: "/vllm-workspace/tests" + source_file_dependencies: + - vllm/ + - "!vllm/distributed/kv_transfer/" + - tests/entrypoints/rl + commands: + - pytest -v -s entrypoints/rl/weight_transfer + - export VLLM_WORKER_MULTIPROC_METHOD=spawn + - pytest -v -s entrypoints/rl --ignore=entrypoints/rl/weight_transfer + mirror: + amd: + label: ":amd: (MI355 DPX) RL Entrypoints" + dind: false + device: mi355_dpx + timeout_in_minutes: 50 + depends_on: + - image-build-amd diff --git a/.buildkite/test_areas/rust_frontend.yaml b/.buildkite/test_areas/rust_frontend.yaml index c5899a9c59..59c78525c5 100644 --- a/.buildkite/test_areas/rust_frontend.yaml +++ b/.buildkite/test_areas/rust_frontend.yaml @@ -82,7 +82,6 @@ steps: - tests/entrypoints/scale_out/token_in_token_out/test_serving_tokens.py - tests/entrypoints/serve/instrumentator/test_basic.py - tests/entrypoints/serve/instrumentator/test_metrics.py - # - tests/entrypoints/serve/dev/test_sleep.py - tests/entrypoints/serve/tokenize/test_tokenization.py commands: - export VLLM_USE_RUST_FRONTEND=1 @@ -93,7 +92,6 @@ steps: # test_generate_logprobs expects Python-style top_logprobs truncation (dedup sampled + cap at max(k, 1)). - pytest -v -s entrypoints/scale_out/token_in_token_out/test_serving_tokens.py -k "not stream and not lora and not test_generate_logprobs and not stop_string_workflow" - pytest -v -s entrypoints/serve/instrumentator/test_metrics.py -k "text and not show and not run_batch and not test_metrics_counts and not test_metrics_exist" - # - pytest -v -s entrypoints/serve/dev/test_sleep.py # /tokenizer_info is not implemented in the Rust frontend (the CLI flag is accepted as a no-op). - pytest -v -s entrypoints/serve/tokenize/test_tokenization.py -k "not tokenizer_info" mirror: diff --git a/.buildkite/test_areas/rust_frontend_cargo.yaml b/.buildkite/test_areas/rust_frontend_cargo.yaml index 53d1da7edf..11bca62526 100644 --- a/.buildkite/test_areas/rust_frontend_cargo.yaml +++ b/.buildkite/test_areas/rust_frontend_cargo.yaml @@ -14,16 +14,6 @@ steps: - .buildkite/scripts/run-rust-frontend-cargo-ci.sh commands: - .buildkite/scripts/run-rust-frontend-cargo-ci.sh style-clippy - mirror: - amd: - label: ":amd: (MI250) Rust Frontend Cargo Style + Clippy" - dind: false - device: mi250_1 - no_gpu: true - working_dir: "/vllm-workspace" - timeout_in_minutes: 35 - depends_on: - - image-build-amd - label: ":computer: (CPU) Rust Frontend Cargo" key: rust-frontend-cargo-tests @@ -38,13 +28,3 @@ steps: - .buildkite/scripts/run-rust-frontend-cargo-ci.sh commands: - .buildkite/scripts/run-rust-frontend-cargo-ci.sh test - mirror: - amd: - label: ":amd: (MI250) Rust Frontend Cargo" - dind: false - device: mi250_1 - no_gpu: true - working_dir: "/vllm-workspace" - timeout_in_minutes: 35 - depends_on: - - image-build-amd diff --git a/.buildkite/test_areas/spec_decode.yaml b/.buildkite/test_areas/spec_decode.yaml index 6882c24b37..d031d4fdea 100644 --- a/.buildkite/test_areas/spec_decode.yaml +++ b/.buildkite/test_areas/spec_decode.yaml @@ -5,6 +5,7 @@ steps: - label: ":nvidia: (H200 MIG 35GB) V1 Spec Decode" device: h200_35gb key: v1-spec-decode + automatic_shard: true timeout_in_minutes: 40 source_file_dependencies: - vllm/config/ diff --git a/.claude/skills/model-recipes b/.claude/skills/model-recipes new file mode 120000 index 0000000000..5231457b9c --- /dev/null +++ b/.claude/skills/model-recipes @@ -0,0 +1 @@ +../../.agents/skills/model-recipes \ No newline at end of file diff --git a/.claude/skills/transformers-upgrade b/.claude/skills/transformers-upgrade new file mode 120000 index 0000000000..d71bb14724 --- /dev/null +++ b/.claude/skills/transformers-upgrade @@ -0,0 +1 @@ +../../.agents/skills/transformers-upgrade \ No newline at end of file diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index b483e1f8b6..65d9454b28 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -4,6 +4,7 @@ # This lists cover the "core" components of vLLM that require careful review /vllm/compilation @zou3519 @youkaichao @ProExpertProg @BoyuanFeng /vllm/distributed/kv_transfer @NickLucche @ApostaC @orozery @xuechendi @ivanium @wzhao18 +/vllm/distributed/kv_transfer/kv_connector/v1/hisparse @MatthewBonanni @LucasWilkinson /vllm/lora @jeejeelee /vllm/model_executor/layers/attention @LucasWilkinson @MatthewBonanni /vllm/model_executor/layers/fused_moe @mgoin @pavanimajety @zyongye @wzhao18 @@ -44,7 +45,7 @@ /vllm/entrypoints/serve @njhill /vllm/entrypoints/*.py @njhill /vllm/entrypoints/chat_utils.py @DarkLight1337 -/vllm/entrypoints/offline_utils.py @DarkLight1337 +/vllm/entrypoints/common @DarkLight1337 /vllm/entrypoints/llm.py @DarkLight1337 # Rust Frontend @@ -77,6 +78,7 @@ /vllm/v1/kv_cache_interface.py @heheda12345 @ivanium /vllm/v1/kv_offload @ApostaC @orozery /vllm/v1/simple_kv_offload @ivanium +/vllm/v1/hisparse @MatthewBonanni @LucasWilkinson /vllm/v1/engine @njhill /vllm/v1/executor @njhill /vllm/v1/worker @njhill @@ -227,13 +229,11 @@ mkdocs.yaml @hmellor # Reinforcement learning /vllm/distributed/weight_transfer @aoshen02 -/vllm/entrypoints/serve/dev/rlhf @njhill @aoshen02 +/vllm/entrypoints/rl @njhill @aoshen02 /tests/distributed/test_weight_transfer.py @aoshen02 -/tests/entrypoints/weight_transfer @DarkLight1337 @robertgshaw2-redhat @aarnphm @NickLucche @AndreasKaratzas @aoshen02 -/tests/entrypoints/serve/dev/rlhf @DarkLight1337 @robertgshaw2-redhat @aarnphm @NickLucche @AndreasKaratzas @aoshen02 +/tests/entrypoints/rl @DarkLight1337 @robertgshaw2-redhat @njhill @aarnphm @NickLucche @AndreasKaratzas @aoshen02 /tests/v1/worker/test_gpu_worker_weight_transfer.py @aoshen02 /examples/rl @aoshen02 -/examples/features/pause_resume @aoshen02 /docs/training/rlhf.md @aoshen02 /docs/training/async_rl.md @aoshen02 /docs/training/weight_transfer @aoshen02 diff --git a/.github/mergify.yml b/.github/mergify.yml index 0798654d34..98c7d0327a 100644 --- a/.github/mergify.yml +++ b/.github/mergify.yml @@ -549,8 +549,6 @@ pull_request_rules: - label != stale - or: - files~=tpu.py - - files~=_tpu - - files~=tpu_ - files~=/tpu/ actions: label: @@ -564,8 +562,6 @@ pull_request_rules: - label != stale - and: - -files~=tpu.py - - -files~=_tpu - - -files~=tpu_ - -files~=/tpu/ actions: label: diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 3e418e005e..92963cf69a 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -199,29 +199,13 @@ repos: additional_dependencies: [PyYAML, regex] verbose: true # full-scan `note:` lines are advisory, so the hook passes require_serial: true # one full scan, not one per pre-commit file batch - - id: mypy-3.10 # TODO: Use https://github.com/pre-commit/mirrors-mypy when mypy setup is less awkward - name: Run mypy for Python 3.10 - entry: python tools/pre_commit/mypy.py "3.10" - <<: &mypy_common - language: python - types_or: [python, pyi] - require_serial: true - additional_dependencies: ["mypy==1.20.2", regex, types-cachetools, types-setuptools, types-PyYAML, types-requests, types-torch, pydantic] - - id: mypy-3.11 # TODO: Use https://github.com/pre-commit/mirrors-mypy when mypy setup is less awkward - name: Run mypy for Python 3.11 + - id: mypy # TODO: Use https://github.com/pre-commit/mirrors-mypy when mypy setup is less awkward + name: Run mypy for the oldest supported Python version entry: python tools/pre_commit/mypy.py "3.11" - <<: *mypy_common - stages: [manual] # Only run in CI - - id: mypy-3.12 # TODO: Use https://github.com/pre-commit/mirrors-mypy when mypy setup is less awkward - name: Run mypy for Python 3.12 - entry: python tools/pre_commit/mypy.py "3.12" - <<: *mypy_common - stages: [manual] # Only run in CI - - id: mypy-3.13 # TODO: Use https://github.com/pre-commit/mirrors-mypy when mypy setup is less awkward - name: Run mypy for Python 3.13 - entry: python tools/pre_commit/mypy.py "3.13" - <<: *mypy_common - stages: [manual] # Only run in CI + language: python + types_or: [python, pyi] + require_serial: true + additional_dependencies: ["mypy==1.20.2", regex, types-cachetools, types-setuptools, types-PyYAML, types-requests, types-torch, pydantic] - id: png-lint name: Lint PNG exports from excalidraw entry: tools/pre_commit/png-lint.sh diff --git a/AGENTS.md b/AGENTS.md index b3ce260383..a31d156b4b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -31,7 +31,7 @@ Do not open one-off PRs for tiny edits (single typo, isolated style change, one - Test commands run and results. - Model evaluation results when the change affects output, accuracy, or serving. - Clear statement that AI assistance was used. -- Use vLLM's [`/pr-checklist`](.agents/skills/pr-checklist/SKILL.md) skill before submitting a PR to ensure quality and adherence to the contribution guidelines. +- Before opening a PR (drafts included) or requesting re-review, run the [`pr-checklist`](.agents/skills/pr-checklist/SKILL.md) skill and address its findings. ### Fail-closed behavior @@ -114,9 +114,6 @@ pre-commit run --all-files # Run a specific hook: pre-commit run ruff-check --all-files - -# Run mypy as it is in CI: -pre-commit run mypy-3.12 --all-files --hook-stage manual ``` The line length limit for Python code is 88 characters. If you are not sure, use pre-commit to check. diff --git a/CMakeLists.txt b/CMakeLists.txt index 0a5414b4bf..b4dd9ada5e 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -46,7 +46,7 @@ install(CODE "set(CMAKE_INSTALL_LOCAL_ONLY TRUE)" ALL_COMPONENTS) # Supported python versions. These versions will be searched in order, the # first match will be selected. These should be kept in sync with setup.py. # -set(PYTHON_SUPPORTED_VERSIONS "3.10" "3.11" "3.12" "3.13" "3.14") +set(PYTHON_SUPPORTED_VERSIONS "3.11" "3.12" "3.13" "3.14") # Supported AMD GPU architectures. set(HIP_SUPPORTED_ARCHS "gfx906;gfx908;gfx90a;gfx942;gfx950;gfx1250;gfx1030;gfx1100;gfx1101;gfx1102;gfx1103;gfx1150;gfx1151;gfx1152;gfx1153;gfx1200;gfx1201") @@ -134,40 +134,34 @@ endif() # # spinloop extension (pure CXX; must stay above the non-CUDA device branch so # CPU builds define the target before the early return) -# This extension requires SABI 3.11 since it relies on Py_buffer support. Loading -# failure is handled gracefully on vLLM side for lower Python versions. -# Skip the target entirely on Python < 3.11 so the build doesn't break. +# This extension requires SABI 3.11 since it relies on Py_buffer support. # -if(Python_VERSION VERSION_GREATER_EQUAL "3.11") - set(VLLM_SPINLOOP_EXT_SRC "csrc/spinloop.cpp") - set(SPINLOOP_COMPILE_FLAGS "") - if(CMAKE_SYSTEM_PROCESSOR MATCHES "x86_64|amd64") - list(APPEND SPINLOOP_COMPILE_FLAGS "-mmwaitx") - endif() - define_extension_target( - spinloop - DESTINATION vllm - LANGUAGE CXX - SOURCES ${VLLM_SPINLOOP_EXT_SRC} - COMPILE_FLAGS ${SPINLOOP_COMPILE_FLAGS} - USE_SABI 3.11 - WITH_SOABI) +set(VLLM_SPINLOOP_EXT_SRC "csrc/spinloop.cpp") +set(SPINLOOP_COMPILE_FLAGS "") +if(CMAKE_SYSTEM_PROCESSOR MATCHES "x86_64|amd64") + list(APPEND SPINLOOP_COMPILE_FLAGS "-mmwaitx") endif() +define_extension_target( + spinloop + DESTINATION vllm + LANGUAGE CXX + SOURCES ${VLLM_SPINLOOP_EXT_SRC} + COMPILE_FLAGS ${SPINLOOP_COMPILE_FLAGS} + USE_SABI 3.11 + WITH_SOABI) # # fs_io extension (pure CXX; must stay above the non-CUDA device branch # so CPU builds define the target before the early return). # GIL-releasing filesystem helpers for FileSystemTierManager. # -if(Python_VERSION VERSION_GREATER_EQUAL "3.11") - define_extension_target( - fs_io_C - DESTINATION vllm - LANGUAGE CXX - SOURCES csrc/fs_io.cpp - USE_SABI 3.11 - WITH_SOABI) -endif() +define_extension_target( + fs_io_C + DESTINATION vllm + LANGUAGE CXX + SOURCES csrc/fs_io.cpp + USE_SABI 3.11 + WITH_SOABI) # # Forward the non-CUDA device extensions to external CMake scripts. @@ -1574,6 +1568,18 @@ if(VLLM_GPU_LANG STREQUAL "CUDA" OR VLLM_GPU_LANG STREQUAL "HIP") include(cmake/external_projects/triton_kernels.cmake) endif() +# DeepGEMM and DeepSelect host code uses C++20 , which libstdc++ +# provides only from GCC 13 onward. +if (VLLM_GPU_LANG STREQUAL "CUDA" + AND CMAKE_CXX_COMPILER_ID STREQUAL "GNU" + AND CMAKE_CXX_COMPILER_VERSION VERSION_LESS "13") + message(FATAL_ERROR + "GCC >= 13 is required to build vLLM with CUDA (found ${CMAKE_CXX_COMPILER_VERSION}). " + "The DeepGEMM and DeepSelect dependencies use C++20 in host code, " + "which libstdc++ does not provide before GCC 13. " + "See: https://github.com/vllm-project/vllm/issues/58158") +endif() + # For CUDA we also build and ship some external projects. if (VLLM_GPU_LANG STREQUAL "CUDA") include(cmake/external_projects/deepgemm.cmake) diff --git a/benchmarks/kernels/benchmark_kimi_k3_amd_fp8_fusions.py b/benchmarks/kernels/benchmark_kimi_k3_amd_fp8_fusions.py new file mode 100644 index 0000000000..dd210ae55d --- /dev/null +++ b/benchmarks/kernels/benchmark_kimi_k3_amd_fp8_fusions.py @@ -0,0 +1,165 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Eight-GPU AttnRes+FP8 projection comparison with rotated graph timing.""" + +import argparse +import json +import os +import statistics +from pathlib import Path + +import torch +import torch.distributed as dist + +from vllm._aiter_ops import rocm_aiter_ops +from vllm.config import VllmConfig, set_current_vllm_config +from vllm.models.kimi_k3.amd.ops.attn_res import attn_res +from vllm.platforms import current_platform + + +def capture(fn, repeats): + stream = torch.cuda.Stream() + stream.wait_stream(torch.cuda.current_stream()) + with torch.cuda.stream(stream): + for _ in range(5): + fn() + torch.cuda.current_stream().wait_stream(stream) + torch.accelerator.synchronize() + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + for _ in range(repeats): + fn() + return graph + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--output", type=Path, required=True) + parser.add_argument( + "--tokens", type=int, nargs="+", default=[1, 4, 16, 64, 256, 1024, 4096, 16384] + ) + parser.add_argument("--blocks", type=int, nargs="+", default=[1, 4, 8]) + args = parser.parse_args() + assert int(os.environ["WORLD_SIZE"]) == 8 + assert torch.accelerator.device_count() == 8 + rank = int(os.environ["LOCAL_RANK"]) + torch.accelerator.set_device_index(rank) + torch.set_num_threads(1) + torch.manual_seed(1729 + rank) + dist.init_process_group("gloo") + args.output.mkdir(parents=True, exist_ok=True) + dtype = current_platform.fp8_dtype() + rows = [] + with set_current_vllm_config(VllmConfig()): + weight = torch.randn(2112, 7168, device="cuda", dtype=torch.bfloat16) * 0.02 + weight, ws = rocm_aiter_ops.per_token_quant(weight, dtype) + weight = rocm_aiter_ops.shuffle_weight(weight) + for m in args.tokens: + prefix = torch.randn(m, 7168, device="cuda", dtype=torch.bfloat16) + delta = torch.zeros_like(prefix) + blocks = torch.randn(m, 9, 7168, device="cuda", dtype=torch.bfloat16) + norm = torch.ones(7168, device="cuda", dtype=torch.bfloat16) + score = torch.randn_like(norm) / 7168**0.5 + for b in args.blocks: + + def produce( + fused, + prefix=prefix, + delta=delta, + blocks=blocks, + norm=norm, + score=score, + b=b, + ): + result = attn_res( + prefix, + delta, + blocks, + norm, + score, + norm, + b, + -1, + 1e-5, + 1e-5, + quant_dtype=dtype if fused else None, + ) + return ( + result + if fused + else rocm_aiter_ops.per_token_quant(result, dtype) + ) + + def segment(fused, produce=produce): + x, scale = produce(fused) + return rocm_aiter_ops.preshuffled_per_token_w8a8_gemm( + x, weight, scale, ws, output_dtype=torch.bfloat16 + ) + + expected, actual = segment(False), segment(True) + relative_l2 = ( + (actual.float() - expected.float()).norm() / expected.float().norm() + ).item() + assert relative_l2 < 0.002, relative_l2 + repeats = 20 if m < 1024 else 4 + functions = { + "baseline_producer": lambda produce=produce: produce(False), + "fused_producer": lambda produce=produce: produce(True), + "baseline_segment": lambda segment=segment: segment(False), + "fused_segment": lambda segment=segment: segment(True), + } + graphs = {k: capture(fn, repeats) for k, fn in functions.items()} + samples = {k: [] for k in functions} + names = list(functions) + for sample in range(9): + for name in names[sample % 4 :] + names[: sample % 4]: + dist.barrier() + graphs[name].replay() + torch.accelerator.synchronize() + start, end = [ + torch.cuda.Event(enable_timing=True) for _ in range(2) + ] + start.record() + graphs[name].replay() + end.record() + end.synchronize() + samples[name].append(start.elapsed_time(end) * 1000 / repeats) + row = { + "tokens": m, + "blocks": b, + "relative_l2": relative_l2, + "samples_us": samples, + "median_us": {k: statistics.median(v) for k, v in samples.items()}, + } + rows.append(row) + print( + json.dumps( + { + "rank": rank, + "tokens": m, + "blocks": b, + "median_us": row["median_us"], + } + ), + flush=True, + ) + del graphs + (args.output / f"rank-{rank}.json").write_text( + json.dumps( + { + "rank": rank, + "world_size": 8, + "torch": torch.__version__, + "rows": rows, + "scope": "AttnRes + FP8 quant + TP8 MLA input projection", + }, + indent=2, + ) + + "\n" + ) + dist.barrier() + dist.destroy_process_group() + + +if __name__ == "__main__": + main() diff --git a/benchmarks/kernels/benchmark_moe.py b/benchmarks/kernels/benchmark_moe.py index 1fcc7ddde7..f746e9aab3 100644 --- a/benchmarks/kernels/benchmark_moe.py +++ b/benchmarks/kernels/benchmark_moe.py @@ -303,9 +303,13 @@ def benchmark_config( run() torch.accelerator.synchronize() - # Capture 10 invocations with CUDA graph - graph = torch.cuda.CUDAGraph() - with torch.cuda.graph(graph): + # Capture 10 invocations via graph(); CUDA needs a side stream + graph = ( + torch.cuda.CUDAGraph() + if current_platform.is_cuda_alike() + else torch.xpu.XPUGraph() + ) + with current_platform.graph(graph): for _ in range(10): run() torch.accelerator.synchronize() @@ -361,11 +365,33 @@ def get_rocm_tuning_space(use_fp16): return param_ranges +def get_xpu_tuning_space(use_fp16): + # BLOCK_SIZE_N=32 and num_warps=32 never won; grf_mode=256 beat 128 on B70. + block_m_range = [16, 32, 64, 128, 256] + block_n_range = [64, 128] + block_k_range = [32, 64] + num_warps_range = [4, 8, 16] + group_m_range = [1, 8, 16] + num_stage_range = [3, 4] + + return { + "BLOCK_SIZE_M": block_m_range, + "BLOCK_SIZE_N": block_n_range, + "BLOCK_SIZE_K": block_k_range, + "GROUP_SIZE_M": group_m_range, + "num_warps": num_warps_range, + "num_stages": num_stage_range, + "grf_mode": ["256"], + } + + def get_configs_compute_bound(use_fp16, block_quant_shape) -> list[dict[str, int]]: configs: list[BenchmarkConfig] = [] if current_platform.is_rocm(): param_ranges = get_rocm_tuning_space(use_fp16) + elif current_platform.is_xpu(): + param_ranges = get_xpu_tuning_space(use_fp16) else: # Reduced search space for faster tuning. # TODO(woosuk): Increase the search space and use a performance model to @@ -408,6 +434,14 @@ def get_configs_compute_bound(use_fp16, block_quant_shape) -> list[dict[str, int return configs +def prune_xpu_search_space(num_tokens, search_space, topk, num_experts): + # Cap the tile by rows routed to one expert, not total rows: wider tiles + # just pad and are the slowest to compile. + rows_per_expert = (num_tokens * topk + num_experts - 1) // num_experts + max_block_m = max(16, triton.next_power_of_2(rows_per_expert)) + return [c for c in search_space if c["BLOCK_SIZE_M"] <= max_block_m] + + def prune_rocm_search_space( num_tokens, shard_intermediate_size, hidden_size, search_space, is_fp16, topk ): @@ -522,7 +556,7 @@ def merge_unique_dicts(list1, list2): @ray.remote(num_gpus=1) class BenchmarkWorker: def __init__(self, seed: int) -> None: - torch.set_default_device("cuda") + torch.set_default_device(current_platform.device_type) set_random_seed(seed) self.seed = seed # Get the device ID to allocate tensors and kernels @@ -619,6 +653,10 @@ class BenchmarkWorker: is_fp16, topk, ) + elif current_platform.is_xpu(): + search_space = prune_xpu_search_space( + num_tokens, search_space, topk, num_experts + ) need_device_guard = False if current_platform.is_rocm(): @@ -690,6 +728,7 @@ def sort_config(config: BenchmarkConfig) -> BenchmarkConfig: else {} ), **({"kpack": config["kpack"]} if "kpack" in config else {}), + **({"grf_mode": config["grf_mode"]} if "grf_mode" in config else {}), **({"SPLIT_K": config["SPLIT_K"]} if "SPLIT_K" in config else {}), } diff --git a/benchmarks/kernels/benchmark_moe_permute_unpermute.py b/benchmarks/kernels/benchmark_moe_permute_unpermute.py index 6867274ebe..f5aa21fb4b 100644 --- a/benchmarks/kernels/benchmark_moe_permute_unpermute.py +++ b/benchmarks/kernels/benchmark_moe_permute_unpermute.py @@ -56,13 +56,9 @@ def benchmark_permute( qhidden_states, input_gating, topk, False ) scratch = MoEPermuteScratch( - max_num_tokens=num_tokens, - topk=topk, num_experts=num_experts, num_local_experts=num_experts, device=qhidden_states.device, - hidden_size=hidden_size, - hidden_dtype=qhidden_states.dtype, ) def prepare(i: int): @@ -135,13 +131,9 @@ def benchmark_unpermute( qhidden_states, input_gating, topk, False ) scratch = MoEPermuteScratch( - max_num_tokens=num_tokens, - topk=topk, num_experts=num_experts, num_local_experts=num_experts, device=qhidden_states.device, - hidden_size=hidden_size, - hidden_dtype=qhidden_states.dtype, ) def prepare(): diff --git a/benchmarks/kernels/benchmark_rdna_hybrid_w4a16_gemm.py b/benchmarks/kernels/benchmark_rdna_hybrid_w4a16_gemm.py index 2d5ec80c6a..7ce49e2e11 100644 --- a/benchmarks/kernels/benchmark_rdna_hybrid_w4a16_gemm.py +++ b/benchmarks/kernels/benchmark_rdna_hybrid_w4a16_gemm.py @@ -15,6 +15,9 @@ import os import torch +from vllm.model_executor.kernels.linear.mixed_precision.rdna_hybrid_w4a16 import ( + pack_skinny_int4, +) from vllm.triton_utils import triton # --------------------------------------------------------------------------- @@ -56,20 +59,25 @@ def prepare_hybrid_weights(K, N, group_size, device="cuda"): Returns (w_q_skinny, w_s_skinny, w_fp16, w_zp). The triton path derives its int32 view from w_q_skinny, so no separate int32 buffer is returned. + + Weights go through ``pack_skinny_int4``, the packer the layer uses, so the + benchmark sees the production row stride rather than an always-dense one. """ num_groups = K // group_size - # Random packed weights — actual values don't matter for throughput - w_q_skinny_i32 = torch.randint( - 0, 2**31, (N, K // 8), dtype=torch.int32, device=device - ) - w_q_skinny = w_q_skinny_i32.view(torch.int8).contiguous() + # Values don't matter for throughput, but the layout does. + unpacked = torch.randint(0, 16, (N, K), dtype=torch.int32, device=device) + w_q_skinny = pack_skinny_int4(unpacked) + del unpacked w_s_skinny = torch.randn(N, num_groups, dtype=torch.float16, device=device) * 0.01 - # Raw per-group zero-points for asymmetric benchmarks - w_zp = torch.randint(0, 16, (N, num_groups), dtype=torch.int32, device=device).to( - torch.float16 - ) + # Per-group zero-points for asymmetric benchmarks, in the packed layout the + # kernels read: [N//8, num_groups] int32, row n's nibble at bits 4*(n%8). + zp_raw = torch.randint(0, 16, (N, num_groups), dtype=torch.int32, device=device) + shifts = (torch.arange(8, device=device, dtype=torch.int32) * 4)[:, None] + w_zp = torch.sum( + (zp_raw.view(N // 8, 8, num_groups) & 0xF) << shifts, dim=1, dtype=torch.int32 + ).contiguous() # FP16 baseline for F.linear w_fp16 = torch.randn(N, K, dtype=torch.float16, device=device) * 0.01 @@ -97,6 +105,13 @@ PROVIDERS = ["torch-fp16", "hybrid-w4a16", "hybrid-w4a16-zp"] ) ) def benchmark(batch_size, provider, N, K, group_size, weights): + """Time one GEMM. + + ``do_bench`` rather than ``do_bench_cudagraph``: it zeroes a 256 MB buffer + between reps, so the weight is read from DRAM as it would be in a model. + With a cudagraph replay the weight stays resident in the 32 MB MALL, which + flatters wide-BLOCK_N tiles and hides row-stride effects entirely. + """ M = batch_size device = "cuda" dtype = torch.float16 @@ -106,7 +121,7 @@ def benchmark(batch_size, provider, N, K, group_size, weights): if provider == "torch-fp16": w_fp16 = weights["w_fp16"] - ms, min_ms, max_ms = triton.testing.do_bench_cudagraph( + ms, min_ms, max_ms = triton.testing.do_bench( lambda: torch.nn.functional.linear(a, w_fp16), quantiles=quantiles, ) @@ -133,7 +148,7 @@ def benchmark(batch_size, provider, N, K, group_size, weights): group_size, ) - ms, min_ms, max_ms = triton.testing.do_bench_cudagraph( + ms, min_ms, max_ms = triton.testing.do_bench( run, quantiles=quantiles, ) diff --git a/benchmarks/kernels/benchmark_sparse_mla_decode_bf16.py b/benchmarks/kernels/benchmark_sparse_mla_decode_bf16.py new file mode 100644 index 0000000000..c32c8519aa --- /dev/null +++ b/benchmarks/kernels/benchmark_sparse_mla_decode_bf16.py @@ -0,0 +1,130 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Benchmark the ROCm bf16 sparse-MLA decode split-K kernel in isolation. + +Example usage: +python3 benchmarks/kernels/benchmark_sparse_mla_decode_bf16.py \ + --rows 1 4 16 64 --lens 2048 --splits auto 4 16 32 +""" + +import argparse +import itertools +import json + +import torch + +from vllm.triton_utils import triton +from vllm.v1.attention.ops.rocm_aiter_mla_sparse import ( + _rocm_sparse_attn_decode_ragged_bf16_triton, + _rocm_sparse_attn_prefill_ragged_triton, + rocm_sparse_decode_bf16_num_splits, +) + +BLOCK_H = 16 + + +def make_inputs(rows: int, live: int, args): + torch.manual_seed(args.seed) + kv = ( + torch.randn(args.capacity, args.head_dim, device="cuda", dtype=torch.bfloat16) + * 0.125 + ) + q = ( + torch.randn( + rows, args.heads, args.head_dim, device="cuda", dtype=torch.bfloat16 + ) + * 0.125 + ) + slots = torch.stack( + [torch.randperm(args.capacity, device="cuda")[:live] for _ in range(rows)] + ) + indices = slots.reshape(-1).to(torch.int32) + indptr = torch.arange(rows + 1, device="cuda", dtype=torch.int32) * live + sink = ( + torch.randn(args.heads, device="cuda", dtype=torch.float32) + if args.sink + else None + ) + return q, kv, indices, indptr, sink + + +def resolve_splits(spec: str, rows: int, live: int, args) -> int: + if spec != "auto": + return int(spec) + return rocm_sparse_decode_bf16_num_splits(rows, args.heads, live) + + +def benchmark(backend: str, rows: int, live: int, spec: str, args) -> dict: + q, kv, indices, indptr, sink = make_inputs(rows, live, args) + nope, rope = args.head_dim, 0 + splits = resolve_splits(spec, rows, live, args) + + def ragged(): + return _rocm_sparse_attn_prefill_ragged_triton( + q, kv, indices, indptr, args.scale, sink, nope, rope + ) + + def splitk(): + return _rocm_sparse_attn_decode_ragged_bf16_triton( + q, kv, indices, indptr, args.scale, sink, nope, rope, splits + ) + + run = ragged if backend == "ragged" else splitk + torch.testing.assert_close(run(), ragged(), atol=2e-2, rtol=2e-2) + ms = triton.testing.do_bench(run, warmup=args.warmup, rep=args.rep) + return dict( + backend=backend, + rows=rows, + live=live, + heads=args.heads, + head_dim=args.head_dim, + splits=splits if backend == "splitk" else 1, + workgroups=rows * triton.cdiv(args.heads, BLOCK_H) * splits, + us=ms * 1000, + kv_gbps=rows * live * args.head_dim * 2 / (ms * 1e6), + ) + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--rows", type=int, nargs="+", default=[1, 2, 4, 8, 16, 64]) + parser.add_argument( + "--lens", + type=int, + nargs="+", + default=[2048], + help="Selected KV slots per row; GLM-5.3-Flash uses index_topk=2048", + ) + parser.add_argument( + "--splits", + nargs="+", + default=["auto"], + help="Split counts to sweep, or auto for the runtime heuristic", + ) + parser.add_argument( + "--backends", + nargs="+", + choices=["ragged", "splitk"], + default=["ragged", "splitk"], + ) + parser.add_argument( + "--heads", type=int, default=8, help="Heads per rank; 64 at TP8 gives 8" + ) + parser.add_argument("--head-dim", type=int, default=512) + parser.add_argument("--capacity", type=int, default=65536) + parser.add_argument("--scale", type=float, default=0.044194173824159216) + parser.add_argument("--sink", action="store_true") + parser.add_argument("--warmup", type=int, default=25) + parser.add_argument("--rep", type=int, default=200) + parser.add_argument("--seed", type=int, default=123) + args = parser.parse_args() + + print(json.dumps(dict(gpu=torch.cuda.get_device_name(0), args=vars(args)))) + for backend, rows, live in itertools.product(args.backends, args.rows, args.lens): + specs = ["1"] if backend == "ragged" else args.splits + for spec in specs: + print(json.dumps(benchmark(backend, rows, live, spec, args)), flush=True) + + +if __name__ == "__main__": + main() diff --git a/benchmarks/sonnet.txt b/benchmarks/sonnet.txt deleted file mode 100644 index 34c444e8ce..0000000000 --- a/benchmarks/sonnet.txt +++ /dev/null @@ -1,518 +0,0 @@ -FROM fairest creatures we desire increase, -That thereby beauty's rose might never die, -But as the riper should by time decease, -His tender heir might bear his memory: -But thou, contracted to thine own bright eyes, -Feed'st thy light'st flame with self-substantial fuel, -Making a famine where abundance lies, -Thyself thy foe, to thy sweet self too cruel. -Thou that art now the world's fresh ornament -And only herald to the gaudy spring, -Within thine own bud buriest thy content -And, tender churl, makest waste in niggarding. -Pity the world, or else this glutton be, -To eat the world's due, by the grave and thee. -When forty winters shall beseige thy brow, -And dig deep trenches in thy beauty's field, -Thy youth's proud livery, so gazed on now, -Will be a tatter'd weed, of small worth held: -Then being ask'd where all thy beauty lies, -Where all the treasure of thy lusty days, -To say, within thine own deep-sunken eyes, -Were an all-eating shame and thriftless praise. -How much more praise deserved thy beauty's use, -If thou couldst answer 'This fair child of mine -Shall sum my count and make my old excuse,' -Proving his beauty by succession thine! -This were to be new made when thou art old, -And see thy blood warm when thou feel'st it cold. -Look in thy glass, and tell the face thou viewest -Now is the time that face should form another; -Whose fresh repair if now thou not renewest, -Thou dost beguile the world, unbless some mother. -For where is she so fair whose unear'd womb -Disdains the tillage of thy husbandry? -Or who is he so fond will be the tomb -Of his self-love, to stop posterity? -Thou art thy mother's glass, and she in thee -Calls back the lovely April of her prime: -So thou through windows of thine age shall see -Despite of wrinkles this thy golden time. -But if thou live, remember'd not to be, -Die single, and thine image dies with thee. -Unthrifty loveliness, why dost thou spend -Upon thyself thy beauty's legacy? -Nature's bequest gives nothing but doth lend, -And being frank she lends to those are free. -Then, beauteous niggard, why dost thou abuse -The bounteous largess given thee to give? -Profitless usurer, why dost thou use -So great a sum of sums, yet canst not live? -For having traffic with thyself alone, -Thou of thyself thy sweet self dost deceive. -Then how, when nature calls thee to be gone, -What acceptable audit canst thou leave? -Thy unused beauty must be tomb'd with thee, -Which, used, lives th' executor to be. -Those hours, that with gentle work did frame -The lovely gaze where every eye doth dwell, -Will play the tyrants to the very same -And that unfair which fairly doth excel: -For never-resting time leads summer on -To hideous winter and confounds him there; -Sap cheque'd with frost and lusty leaves quite gone, -Beauty o'ersnow'd and bareness every where: -Then, were not summer's distillation left, -A liquid prisoner pent in walls of glass, -Beauty's effect with beauty were bereft, -Nor it nor no remembrance what it was: -But flowers distill'd though they with winter meet, -Leese but their show; their substance still lives sweet. -Then let not winter's ragged hand deface -In thee thy summer, ere thou be distill'd: -Make sweet some vial; treasure thou some place -With beauty's treasure, ere it be self-kill'd. -That use is not forbidden usury, -Which happies those that pay the willing loan; -That's for thyself to breed another thee, -Or ten times happier, be it ten for one; -Ten times thyself were happier than thou art, -If ten of thine ten times refigured thee: -Then what could death do, if thou shouldst depart, -Leaving thee living in posterity? -Be not self-will'd, for thou art much too fair -To be death's conquest and make worms thine heir. -Lo! in the orient when the gracious light -Lifts up his burning head, each under eye -Doth homage to his new-appearing sight, -Serving with looks his sacred majesty; -And having climb'd the steep-up heavenly hill, -Resembling strong youth in his middle age, -yet mortal looks adore his beauty still, -Attending on his golden pilgrimage; -But when from highmost pitch, with weary car, -Like feeble age, he reeleth from the day, -The eyes, 'fore duteous, now converted are -From his low tract and look another way: -So thou, thyself out-going in thy noon, -Unlook'd on diest, unless thou get a son. -Music to hear, why hear'st thou music sadly? -Sweets with sweets war not, joy delights in joy. -Why lovest thou that which thou receivest not gladly, -Or else receivest with pleasure thine annoy? -If the true concord of well-tuned sounds, -By unions married, do offend thine ear, -They do but sweetly chide thee, who confounds -In singleness the parts that thou shouldst bear. -Mark how one string, sweet husband to another, -Strikes each in each by mutual ordering, -Resembling sire and child and happy mother -Who all in one, one pleasing note do sing: -Whose speechless song, being many, seeming one, -Sings this to thee: 'thou single wilt prove none.' -Is it for fear to wet a widow's eye -That thou consumest thyself in single life? -Ah! if thou issueless shalt hap to die. -The world will wail thee, like a makeless wife; -The world will be thy widow and still weep -That thou no form of thee hast left behind, -When every private widow well may keep -By children's eyes her husband's shape in mind. -Look, what an unthrift in the world doth spend -Shifts but his place, for still the world enjoys it; -But beauty's waste hath in the world an end, -And kept unused, the user so destroys it. -No love toward others in that bosom sits -That on himself such murderous shame commits. -For shame! deny that thou bear'st love to any, -Who for thyself art so unprovident. -Grant, if thou wilt, thou art beloved of many, -But that thou none lovest is most evident; -For thou art so possess'd with murderous hate -That 'gainst thyself thou stick'st not to conspire. -Seeking that beauteous roof to ruinate -Which to repair should be thy chief desire. -O, change thy thought, that I may change my mind! -Shall hate be fairer lodged than gentle love? -Be, as thy presence is, gracious and kind, -Or to thyself at least kind-hearted prove: -Make thee another self, for love of me, -That beauty still may live in thine or thee. -As fast as thou shalt wane, so fast thou growest -In one of thine, from that which thou departest; -And that fresh blood which youngly thou bestowest -Thou mayst call thine when thou from youth convertest. -Herein lives wisdom, beauty and increase: -Without this, folly, age and cold decay: -If all were minded so, the times should cease -And threescore year would make the world away. -Let those whom Nature hath not made for store, -Harsh featureless and rude, barrenly perish: -Look, whom she best endow'd she gave the more; -Which bounteous gift thou shouldst in bounty cherish: -She carved thee for her seal, and meant thereby -Thou shouldst print more, not let that copy die. -When I do count the clock that tells the time, -And see the brave day sunk in hideous night; -When I behold the violet past prime, -And sable curls all silver'd o'er with white; -When lofty trees I see barren of leaves -Which erst from heat did canopy the herd, -And summer's green all girded up in sheaves -Borne on the bier with white and bristly beard, -Then of thy beauty do I question make, -That thou among the wastes of time must go, -Since sweets and beauties do themselves forsake -And die as fast as they see others grow; -And nothing 'gainst Time's scythe can make defence -Save breed, to brave him when he takes thee hence. -O, that you were yourself! but, love, you are -No longer yours than you yourself here live: -Against this coming end you should prepare, -And your sweet semblance to some other give. -So should that beauty which you hold in lease -Find no determination: then you were -Yourself again after yourself's decease, -When your sweet issue your sweet form should bear. -Who lets so fair a house fall to decay, -Which husbandry in honour might uphold -Against the stormy gusts of winter's day -And barren rage of death's eternal cold? -O, none but unthrifts! Dear my love, you know -You had a father: let your son say so. -Not from the stars do I my judgment pluck; -And yet methinks I have astronomy, -But not to tell of good or evil luck, -Of plagues, of dearths, or seasons' quality; -Nor can I fortune to brief minutes tell, -Pointing to each his thunder, rain and wind, -Or say with princes if it shall go well, -By oft predict that I in heaven find: -But from thine eyes my knowledge I derive, -And, constant stars, in them I read such art -As truth and beauty shall together thrive, -If from thyself to store thou wouldst convert; -Or else of thee this I prognosticate: -Thy end is truth's and beauty's doom and date. -When I consider every thing that grows -Holds in perfection but a little moment, -That this huge stage presenteth nought but shows -Whereon the stars in secret influence comment; -When I perceive that men as plants increase, -Cheered and cheque'd even by the self-same sky, -Vaunt in their youthful sap, at height decrease, -And wear their brave state out of memory; -Then the conceit of this inconstant stay -Sets you most rich in youth before my sight, -Where wasteful Time debateth with Decay, -To change your day of youth to sullied night; -And all in war with Time for love of you, -As he takes from you, I engraft you new. -But wherefore do not you a mightier way -Make war upon this bloody tyrant, Time? -And fortify yourself in your decay -With means more blessed than my barren rhyme? -Now stand you on the top of happy hours, -And many maiden gardens yet unset -With virtuous wish would bear your living flowers, -Much liker than your painted counterfeit: -So should the lines of life that life repair, -Which this, Time's pencil, or my pupil pen, -Neither in inward worth nor outward fair, -Can make you live yourself in eyes of men. -To give away yourself keeps yourself still, -And you must live, drawn by your own sweet skill. -Who will believe my verse in time to come, -If it were fill'd with your most high deserts? -Though yet, heaven knows, it is but as a tomb -Which hides your life and shows not half your parts. -If I could write the beauty of your eyes -And in fresh numbers number all your graces, -The age to come would say 'This poet lies: -Such heavenly touches ne'er touch'd earthly faces.' -So should my papers yellow'd with their age -Be scorn'd like old men of less truth than tongue, -And your true rights be term'd a poet's rage -And stretched metre of an antique song: -But were some child of yours alive that time, -You should live twice; in it and in my rhyme. -Shall I compare thee to a summer's day? -Thou art more lovely and more temperate: -Rough winds do shake the darling buds of May, -And summer's lease hath all too short a date: -Sometime too hot the eye of heaven shines, -And often is his gold complexion dimm'd; -And every fair from fair sometime declines, -By chance or nature's changing course untrimm'd; -But thy eternal summer shall not fade -Nor lose possession of that fair thou owest; -Nor shall Death brag thou wander'st in his shade, -When in eternal lines to time thou growest: -So long as men can breathe or eyes can see, -So long lives this and this gives life to thee. -Devouring Time, blunt thou the lion's paws, -And make the earth devour her own sweet brood; -Pluck the keen teeth from the fierce tiger's jaws, -And burn the long-lived phoenix in her blood; -Make glad and sorry seasons as thou fleets, -And do whate'er thou wilt, swift-footed Time, -To the wide world and all her fading sweets; -But I forbid thee one most heinous crime: -O, carve not with thy hours my love's fair brow, -Nor draw no lines there with thine antique pen; -Him in thy course untainted do allow -For beauty's pattern to succeeding men. -Yet, do thy worst, old Time: despite thy wrong, -My love shall in my verse ever live young. -A woman's face with Nature's own hand painted -Hast thou, the master-mistress of my passion; -A woman's gentle heart, but not acquainted -With shifting change, as is false women's fashion; -An eye more bright than theirs, less false in rolling, -Gilding the object whereupon it gazeth; -A man in hue, all 'hues' in his controlling, -Much steals men's eyes and women's souls amazeth. -And for a woman wert thou first created; -Till Nature, as she wrought thee, fell a-doting, -And by addition me of thee defeated, -By adding one thing to my purpose nothing. -But since she prick'd thee out for women's pleasure, -Mine be thy love and thy love's use their treasure. -So is it not with me as with that Muse -Stirr'd by a painted beauty to his verse, -Who heaven itself for ornament doth use -And every fair with his fair doth rehearse -Making a couplement of proud compare, -With sun and moon, with earth and sea's rich gems, -With April's first-born flowers, and all things rare -That heaven's air in this huge rondure hems. -O' let me, true in love, but truly write, -And then believe me, my love is as fair -As any mother's child, though not so bright -As those gold candles fix'd in heaven's air: -Let them say more than like of hearsay well; -I will not praise that purpose not to sell. -My glass shall not persuade me I am old, -So long as youth and thou are of one date; -But when in thee time's furrows I behold, -Then look I death my days should expiate. -For all that beauty that doth cover thee -Is but the seemly raiment of my heart, -Which in thy breast doth live, as thine in me: -How can I then be elder than thou art? -O, therefore, love, be of thyself so wary -As I, not for myself, but for thee will; -Bearing thy heart, which I will keep so chary -As tender nurse her babe from faring ill. -Presume not on thy heart when mine is slain; -Thou gavest me thine, not to give back again. -As an unperfect actor on the stage -Who with his fear is put besides his part, -Or some fierce thing replete with too much rage, -Whose strength's abundance weakens his own heart. -So I, for fear of trust, forget to say -The perfect ceremony of love's rite, -And in mine own love's strength seem to decay, -O'ercharged with burden of mine own love's might. -O, let my books be then the eloquence -And dumb presagers of my speaking breast, -Who plead for love and look for recompense -More than that tongue that more hath more express'd. -O, learn to read what silent love hath writ: -To hear with eyes belongs to love's fine wit. -Mine eye hath play'd the painter and hath stell'd -Thy beauty's form in table of my heart; -My body is the frame wherein 'tis held, -And perspective it is the painter's art. -For through the painter must you see his skill, -To find where your true image pictured lies; -Which in my bosom's shop is hanging still, -That hath his windows glazed with thine eyes. -Now see what good turns eyes for eyes have done: -Mine eyes have drawn thy shape, and thine for me -Are windows to my breast, where-through the sun -Delights to peep, to gaze therein on thee; -Yet eyes this cunning want to grace their art; -They draw but what they see, know not the heart. -Let those who are in favour with their stars -Of public honour and proud titles boast, -Whilst I, whom fortune of such triumph bars, -Unlook'd for joy in that I honour most. -Great princes' favourites their fair leaves spread -But as the marigold at the sun's eye, -And in themselves their pride lies buried, -For at a frown they in their glory die. -The painful warrior famoused for fight, -After a thousand victories once foil'd, -Is from the book of honour razed quite, -And all the rest forgot for which he toil'd: -Then happy I, that love and am beloved -Where I may not remove nor be removed. -Lord of my love, to whom in vassalage -Thy merit hath my duty strongly knit, -To thee I send this written embassage, -To witness duty, not to show my wit: -Duty so great, which wit so poor as mine -May make seem bare, in wanting words to show it, -But that I hope some good conceit of thine -In thy soul's thought, all naked, will bestow it; -Till whatsoever star that guides my moving -Points on me graciously with fair aspect -And puts apparel on my tatter'd loving, -To show me worthy of thy sweet respect: -Then may I dare to boast how I do love thee; -Till then not show my head where thou mayst prove me. -Weary with toil, I haste me to my bed, -The dear repose for limbs with travel tired; -But then begins a journey in my head, -To work my mind, when body's work's expired: -For then my thoughts, from far where I abide, -Intend a zealous pilgrimage to thee, -And keep my drooping eyelids open wide, -Looking on darkness which the blind do see -Save that my soul's imaginary sight -Presents thy shadow to my sightless view, -Which, like a jewel hung in ghastly night, -Makes black night beauteous and her old face new. -Lo! thus, by day my limbs, by night my mind, -For thee and for myself no quiet find. -How can I then return in happy plight, -That am debarr'd the benefit of rest? -When day's oppression is not eased by night, -But day by night, and night by day, oppress'd? -And each, though enemies to either's reign, -Do in consent shake hands to torture me; -The one by toil, the other to complain -How far I toil, still farther off from thee. -I tell the day, to please them thou art bright -And dost him grace when clouds do blot the heaven: -So flatter I the swart-complexion'd night, -When sparkling stars twire not thou gild'st the even. -But day doth daily draw my sorrows longer -And night doth nightly make grief's strength seem stronger. -When, in disgrace with fortune and men's eyes, -I all alone beweep my outcast state -And trouble deal heaven with my bootless cries -And look upon myself and curse my fate, -Wishing me like to one more rich in hope, -Featured like him, like him with friends possess'd, -Desiring this man's art and that man's scope, -With what I most enjoy contented least; -Yet in these thoughts myself almost despising, -Haply I think on thee, and then my state, -Like to the lark at break of day arising -From sullen earth, sings hymns at heaven's gate; -For thy sweet love remember'd such wealth brings -That then I scorn to change my state with kings. -When to the sessions of sweet silent thought -I summon up remembrance of things past, -I sigh the lack of many a thing I sought, -And with old woes new wail my dear time's waste: -Then can I drown an eye, unused to flow, -For precious friends hid in death's dateless night, -And weep afresh love's long since cancell'd woe, -And moan the expense of many a vanish'd sight: -Then can I grieve at grievances foregone, -And heavily from woe to woe tell o'er -The sad account of fore-bemoaned moan, -Which I new pay as if not paid before. -But if the while I think on thee, dear friend, -All losses are restored and sorrows end. -Thy bosom is endeared with all hearts, -Which I by lacking have supposed dead, -And there reigns love and all love's loving parts, -And all those friends which I thought buried. -How many a holy and obsequious tear -Hath dear religious love stol'n from mine eye -As interest of the dead, which now appear -But things removed that hidden in thee lie! -Thou art the grave where buried love doth live, -Hung with the trophies of my lovers gone, -Who all their parts of me to thee did give; -That due of many now is thine alone: -Their images I loved I view in thee, -And thou, all they, hast all the all of me. -If thou survive my well-contented day, -When that churl Death my bones with dust shall cover, -And shalt by fortune once more re-survey -These poor rude lines of thy deceased lover, -Compare them with the bettering of the time, -And though they be outstripp'd by every pen, -Reserve them for my love, not for their rhyme, -Exceeded by the height of happier men. -O, then vouchsafe me but this loving thought: -'Had my friend's Muse grown with this growing age, -A dearer birth than this his love had brought, -To march in ranks of better equipage: -But since he died and poets better prove, -Theirs for their style I'll read, his for his love.' -Full many a glorious morning have I seen -Flatter the mountain-tops with sovereign eye, -Kissing with golden face the meadows green, -Gilding pale streams with heavenly alchemy; -Anon permit the basest clouds to ride -With ugly rack on his celestial face, -And from the forlorn world his visage hide, -Stealing unseen to west with this disgrace: -Even so my sun one early morn did shine -With all triumphant splendor on my brow; -But out, alack! he was but one hour mine; -The region cloud hath mask'd him from me now. -Yet him for this my love no whit disdaineth; -Suns of the world may stain when heaven's sun staineth. -Why didst thou promise such a beauteous day, -And make me travel forth without my cloak, -To let base clouds o'ertake me in my way, -Hiding thy bravery in their rotten smoke? -'Tis not enough that through the cloud thou break, -To dry the rain on my storm-beaten face, -For no man well of such a salve can speak -That heals the wound and cures not the disgrace: -Nor can thy shame give physic to my grief; -Though thou repent, yet I have still the loss: -The offender's sorrow lends but weak relief -To him that bears the strong offence's cross. -Ah! but those tears are pearl which thy love sheds, -And they are rich and ransom all ill deeds. -No more be grieved at that which thou hast done: -Roses have thorns, and silver fountains mud; -Clouds and eclipses stain both moon and sun, -And loathsome canker lives in sweetest bud. -All men make faults, and even I in this, -Authorizing thy trespass with compare, -Myself corrupting, salving thy amiss, -Excusing thy sins more than thy sins are; -For to thy sensual fault I bring in sense-- -Thy adverse party is thy advocate-- -And 'gainst myself a lawful plea commence: -Such civil war is in my love and hate -That I an accessary needs must be -To that sweet thief which sourly robs from me. -Let me confess that we two must be twain, -Although our undivided loves are one: -So shall those blots that do with me remain -Without thy help by me be borne alone. -In our two loves there is but one respect, -Though in our lives a separable spite, -Which though it alter not love's sole effect, -Yet doth it steal sweet hours from love's delight. -I may not evermore acknowledge thee, -Lest my bewailed guilt should do thee shame, -Nor thou with public kindness honour me, -Unless thou take that honour from thy name: -But do not so; I love thee in such sort -As, thou being mine, mine is thy good report. -As a decrepit father takes delight -To see his active child do deeds of youth, -So I, made lame by fortune's dearest spite, -Take all my comfort of thy worth and truth. -For whether beauty, birth, or wealth, or wit, -Or any of these all, or all, or more, -Entitled in thy parts do crowned sit, -I make my love engrafted to this store: -So then I am not lame, poor, nor despised, -Whilst that this shadow doth such substance give -That I in thy abundance am sufficed -And by a part of all thy glory live. -Look, what is best, that best I wish in thee: -This wish I have; then ten times happy me! \ No newline at end of file diff --git a/cmake/external_projects/deepgemm.cmake b/cmake/external_projects/deepgemm.cmake index a17031cc5e..1dcee345f8 100644 --- a/cmake/external_projects/deepgemm.cmake +++ b/cmake/external_projects/deepgemm.cmake @@ -33,7 +33,8 @@ else() # port, the SM90 paged-MQA kv_block=32/next_n=4 port, configurable SwiGLU # alpha/beta, and SiTU for FP8/FP4 Mega MoE. Also includes the CUDA 12.x # layout header fix from vllm-project/DeepGEMM#12. - set(_DEEPGEMM_UPSTREAM_TAG "e1f418c2a4f20818221f6b0e578b4c2f634d4c3f") + # TORCH_LIBRARY build with a CPython 3.10 stable-ABI floor. + set(_DEEPGEMM_UPSTREAM_TAG "1e1842a833699298f7afc02eefb2ed168fff6938") set(_deepgemm_fc_root "${FETCHCONTENT_BASE_DIR}") if(NOT _deepgemm_fc_root) @@ -43,7 +44,7 @@ else() set(_deepgemm_bin "${_deepgemm_fc_root}/deepgemm-build") set(_deepgemm_sub "${_deepgemm_fc_root}/deepgemm-subbuild") - if(EXISTS "${_deepgemm_src}/csrc/python_api.cpp") + if(EXISTS "${_deepgemm_src}/deep_gemm/_C.py") set(deepgemm_SOURCE_DIR "${_deepgemm_src}") set(deepgemm_BINARY_DIR "${_deepgemm_bin}") else() @@ -93,33 +94,17 @@ if(DEEPGEMM_ARCHS) # # DeepGEMM integration notes # -------------------------- - # We vendor DeepGEMM into vllm/third_party/deep_gemm/ and bundle a - # `_C.cpython-X.Y-*.so` for every CPython in `requires-python`. The - # per-Python build is delegated to tools/build_deepgemm_C.py. + # We vendor DeepGEMM into vllm/third_party/deep_gemm/ and bundle: + # - deep_gemm/_C.py (Python shim over torch.ops.deep_gemm) + # - deep_gemm/_C_extension.abi3.so (single limited-API extension) + # The build is delegated to tools/build_deepgemm_C.py (setup.py build_ext). # - # Why per-Python: DeepGEMM's binding uses PYBIND11_MODULE, which links - # private CPython symbols — a single `_C.abi3.so` is not viable today - # (see #41476 / #41512 for the failed attempt). - # - # TODOs (tracked in vllm-project/vllm#42431): - # - Replace DeepGEMM's pybind11 binding with a TORCH_LIBRARY + shim - # binding (cf. vllm-flash-attention/csrc/common/pytorch_shim.h) to - # collapse to one `_C.abi3.so`. Needs either an upstream change or - # a maintained binding fork in vLLM. - # - AOT-compile DeepGEMM's CUDA kernels instead of runtime JIT to drop - # the vendored CUTLASS/CCCL headers and the CUDA-toolkit-at-runtime - # requirement. + # TODO: AOT-compile DeepGEMM's CUDA kernels instead of runtime JIT to drop + # the vendored CUTLASS/CCCL headers and the CUDA-toolkit-at-runtime + # requirement. # - # DEEPGEMM_PYTHON_INTERPRETERS: ":"-separated target Python paths. - # Empty/unset → fall back to the build interpreter (editable installs). - # (Empty-but-set env vars test as DEFINED in cmake — treat as unset.) - if(NOT "$ENV{DEEPGEMM_PYTHON_INTERPRETERS}" STREQUAL "") - string(REPLACE ":" ";" _dg_pythons "$ENV{DEEPGEMM_PYTHON_INTERPRETERS}") - else() - set(_dg_pythons "${Python_EXECUTABLE}") - endif() - message(STATUS "DeepGEMM _C will be built for: ${_dg_pythons}") + message(STATUS "DeepGEMM extension will be built with: ${Python_EXECUTABLE}") # add_custom_command does no implicit header scanning; glob explicitly so # header-only edits in DeepGEMM/cutlass/deep_jit re-trigger the rebuild. @@ -132,40 +117,29 @@ if(DEEPGEMM_ARCHS) "${deepgemm_SOURCE_DIR}/third-party/deep_jit/include/*.h" "${deepgemm_SOURCE_DIR}/third-party/deep_jit/include/*.hpp") - set(_dg_markers) - set(_dg_seen_soabis) - foreach(_pybin IN LISTS _dg_pythons) - execute_process( - COMMAND "${_pybin}" -c - "import sysconfig; print(sysconfig.get_config_var('SOABI'))" - OUTPUT_VARIABLE _dg_soabi - OUTPUT_STRIP_TRAILING_WHITESPACE - COMMAND_ERROR_IS_FATAL ANY) - # Dedup interpreters that resolve to the same CPython. - if(_dg_soabi IN_LIST _dg_seen_soabis) - continue() - endif() - list(APPEND _dg_seen_soabis "${_dg_soabi}") - set(_dg_dir "${CMAKE_CURRENT_BINARY_DIR}/deepgemm_C_${_dg_soabi}") - set(_dg_marker "${_dg_dir}/.built") - add_custom_command( - OUTPUT "${_dg_marker}" - COMMAND "${Python_EXECUTABLE}" - "${CMAKE_SOURCE_DIR}/tools/build_deepgemm_C.py" - "${deepgemm_SOURCE_DIR}" "${_dg_dir}" "${_pybin}" - COMMAND "${CMAKE_COMMAND}" -E touch "${_dg_marker}" - DEPENDS "${CMAKE_SOURCE_DIR}/tools/build_deepgemm_C.py" - "${deepgemm_SOURCE_DIR}/csrc/python_api.cpp" - ${_dg_headers} - COMMENT "Building DeepGEMM _C for ${_pybin}" - VERBATIM) - list(APPEND _dg_markers "${_dg_marker}") - install(DIRECTORY "${_dg_dir}/" - DESTINATION vllm/third_party/deep_gemm - COMPONENT _deep_gemm_C - FILES_MATCHING PATTERN "_C.cpython-*.so") - endforeach() - add_custom_target(_deep_gemm_C ALL DEPENDS ${_dg_markers}) + set(_dg_dir "${CMAKE_CURRENT_BINARY_DIR}/deepgemm_C") + set(_dg_marker "${_dg_dir}/.built") + add_custom_command( + OUTPUT "${_dg_marker}" + COMMAND "${Python_EXECUTABLE}" + "${CMAKE_SOURCE_DIR}/tools/build_deepgemm_C.py" + "${deepgemm_SOURCE_DIR}" "${_dg_dir}" + COMMAND "${CMAKE_COMMAND}" -E touch "${_dg_marker}" + DEPENDS "${CMAKE_SOURCE_DIR}/tools/build_deepgemm_C.py" + "${deepgemm_SOURCE_DIR}/csrc/python_api.cpp" + "${deepgemm_SOURCE_DIR}/deep_gemm/_C.py" + "${deepgemm_SOURCE_DIR}/setup.py" + ${_dg_headers} + COMMENT "Building DeepGEMM _C_extension (abi3)" + VERBATIM) + add_custom_target(_deep_gemm_C ALL DEPENDS "${_dg_marker}") + + install(DIRECTORY "${_dg_dir}/" + DESTINATION vllm/third_party/deep_gemm + COMPONENT _deep_gemm_C + FILES_MATCHING + PATTERN "_C.py" + PATTERN "_C_extension*.so") # # Vendor DeepGEMM Python package files diff --git a/csrc/cpu/cpu_attn.cpp b/csrc/cpu/cpu_attn.cpp index 13e430fb41..afdc0e0eca 100644 --- a/csrc/cpu/cpu_attn.cpp +++ b/csrc/cpu/cpu_attn.cpp @@ -78,8 +78,9 @@ torch::Tensor get_scheduler_metadata( input.causal = causal; input.isa = isa; input.enable_kv_split = enable_kv_split; - input.dynamic_causal = - dynamic_causal.has_value() ? dynamic_causal->data_ptr() : nullptr; + input.dynamic_causal = dynamic_causal.has_value() + ? dynamic_causal->data_ptr() + : nullptr; const int64_t kv_cache_idx = static_cast(parse_fp8_kv_dtype(kv_cache_dtype)); @@ -256,8 +257,9 @@ void cpu_attention_with_kv_cache( input.s_aux = nullptr; input.s_aux_is_bf16 = false; } - input.dynamic_causal = - dynamic_causal.has_value() ? dynamic_causal->data_ptr() : nullptr; + input.dynamic_causal = dynamic_causal.has_value() + ? dynamic_causal->data_ptr() + : nullptr; input.scale = scale; input.causal = causal; input.sliding_window_size = sliding_window; diff --git a/csrc/cpu/cpu_attn_impl.hpp b/csrc/cpu/cpu_attn_impl.hpp index d914e15063..0666ee413f 100644 --- a/csrc/cpu/cpu_attn_impl.hpp +++ b/csrc/cpu/cpu_attn_impl.hpp @@ -397,7 +397,7 @@ class AttentionScheduler { int32_t max_num_q_per_iter; // max Q head num can be hold in registers int32_t kv_block_alignment; // context length alignment requirement bool enable_kv_split; - bool* dynamic_causal; + int32_t* dynamic_causal; }; static constexpr int32_t MaxQTileIterNum = 128; @@ -450,7 +450,7 @@ class AttentionScheduler { const int32_t q_token_num = input.query_start_loc[req_id + 1] - input.query_start_loc[req_id]; const bool req_causal = - is_dynamic_causal ? input.dynamic_causal[req_id] : causal; + is_dynamic_causal ? input.dynamic_causal[req_id] != 0 : causal; const int32_t q_start_pos = seq_len - q_token_num; const int32_t kv_start_pos = 0; const int32_t kv_end_pos = seq_len; @@ -492,7 +492,7 @@ class AttentionScheduler { const int32_t q_token_num = input.query_start_loc[req_id + 1] - input.query_start_loc[req_id]; const bool req_causal = - is_dynamic_causal ? input.dynamic_causal[req_id] : causal; + is_dynamic_causal ? input.dynamic_causal[req_id] != 0 : causal; const int32_t q_start_pos = seq_len - q_token_num; const int32_t kv_start_pos = 0; const int32_t kv_end_pos = seq_len; @@ -884,7 +884,7 @@ struct AttentionInput { // provided as fp32 and executed in full float precision. const void* s_aux; bool s_aux_is_bf16; - bool* dynamic_causal; + int32_t* dynamic_causal; float scale; bool causal; int32_t sliding_window_size; @@ -1571,7 +1571,7 @@ class AttentionMainLoop { const float* alibi_slopes = input->alibi_slopes; const void* s_aux = input->s_aux; const bool s_aux_is_bf16 = input->s_aux_is_bf16; - const bool* dynamic_causal = input->dynamic_causal; + const int32_t* dynamic_causal = input->dynamic_causal; const bool is_dynamic_causal = dynamic_causal != nullptr; const bool causal = input->causal; @@ -1658,7 +1658,8 @@ class AttentionMainLoop { const int32_t current_group_idx = current_workitem_group->req_id; const int32_t current_group_causal = - is_dynamic_causal ? dynamic_causal[current_group_idx] : causal; + is_dynamic_causal ? dynamic_causal[current_group_idx] != 0 + : causal; auto [sliding_window_left, sliding_window_right] = AttentionScheduler::calcu_sliding_window_size( sliding_window_size, current_group_causal); diff --git a/csrc/cpu/sampling_kernels.cpp b/csrc/cpu/sampling_kernels.cpp index 3e2f00a0ba..1d30b92a8d 100644 --- a/csrc/cpu/sampling_kernels.cpp +++ b/csrc/cpu/sampling_kernels.cpp @@ -1,82 +1,73 @@ #include "cpu_types.hpp" -#include +#include #include #include #include -#include -#include namespace { -constexpr int GUMBEL_TABLE_SIZE = 1 << 20; -constexpr int GUMBEL_TABLE_MASK = GUMBEL_TABLE_SIZE - 1; +// Gumbel-max: argmax_i(logit_i + g_i) with g_i ~ Gumbel(0, 1) matches +// softmax. Noise is SplitMix64(seed, i), so tokens are independent and +// consecutive seeds are not a 1-token table shift (vllm#59786). +constexpr uint64_t SPLITMIX_GAMMA = 0x9E3779B97F4A7C15ULL; +constexpr double GUMBEL_MAX = 37.5; // 53-bit u in (0,1) => -log(-log u) < 37.5 -static float g_gumbel_table[GUMBEL_TABLE_SIZE]; -static std::once_flag g_gumbel_init_flag; +static inline uint64_t splitmix64(uint64_t z) { + z = (z ^ (z >> 30)) * 0xBF58476D1CE4E5B9ULL; + z = (z ^ (z >> 27)) * 0x94D049BB133111EBULL; + return z ^ (z >> 31); +} -static void init_gumbel_table() { - std::mt19937 rng(42); - std::uniform_real_distribution uniform(1.0e-7f, 1.0f - 1.0e-7f); - for (int i = 0; i < GUMBEL_TABLE_SIZE; ++i) { - float u = uniform(rng); - g_gumbel_table[i] = -std::log(-std::log(u)); +static void fused_gumbel_argmax_row(int64_t* __restrict__ output, + const float* __restrict__ row, + uint64_t seed, int64_t vocab_size) { + const uint64_t key = splitmix64(seed); + + // Compare in (logit, noise) space: x + g > best_x + best_g is + // (x - best_x) + g > best_g. Adding g into a 1e38 logit loses it in + // double ULP, so ties at the float max would always keep the lower index. + double best_logit = 0.0; + double best_g = 0.0; + int64_t best_idx = 0; + bool have_best = false; + for (int64_t i = 0; i < vocab_size; ++i) { + const double x = row[i]; + if (x == -std::numeric_limits::infinity()) continue; + if (have_best && (x - best_logit) + GUMBEL_MAX <= best_g) continue; + const uint64_t bits = + splitmix64(key + uint64_t(i + 1) * SPLITMIX_GAMMA) >> 11; + const double one_minus_u = + (double((1ULL << 53) - 1 - bits) + 0.5) * 0x1.0p-53; + if (have_best && one_minus_u >= std::exp((x - best_logit) - best_g)) + continue; + const double u = (double(bits) + 0.5) * 0x1.0p-53; + const double g = -std::log(-std::log(u)); + if (!have_best || (x - best_logit) + g > best_g) { + have_best = true; + best_logit = x; + best_g = g; + best_idx = i; + } } + *output = best_idx; } -static inline void ensure_gumbel_table() { - std::call_once(g_gumbel_init_flag, init_gumbel_table); -} - -// Fused Gumbel-max kernel - static void fused_gumbel_argmax_kernel(int64_t* __restrict__ output, const float* __restrict__ logits, const int64_t* __restrict__ seeds, const int64_t batch_size, const int64_t vocab_size) { - ensure_gumbel_table(); - constexpr int VEC_ELEM_NUM = vec_op::FP32Vec8::VEC_ELEM_NUM; - const int64_t vec_end = vocab_size - (vocab_size % VEC_ELEM_NUM); - + if (batch_size <= 0) return; + if (batch_size == 1) { + fused_gumbel_argmax_row(output, logits, uint64_t(seeds[0]), vocab_size); + return; + } #pragma omp parallel for schedule(static) for (int64_t b = 0; b < batch_size; ++b) { - const float* row = logits + b * vocab_size; - const uint32_t seed = static_cast(seeds[b]); - - float best_score = -std::numeric_limits::infinity(); - int64_t best_idx = 0; - - for (int64_t i = 0; i < vec_end; i += VEC_ELEM_NUM) { - vec_op::FP32Vec8 logit_vec(row + i); - - float gumbel_buf[VEC_ELEM_NUM]; - for (int k = 0; k < VEC_ELEM_NUM; ++k) - gumbel_buf[k] = g_gumbel_table[(seed + i + k) & GUMBEL_TABLE_MASK]; - vec_op::FP32Vec8 gumbel_vec(gumbel_buf); - - vec_op::FP32Vec8 score = logit_vec + gumbel_vec; - - float score_buf[VEC_ELEM_NUM]; - score.save(score_buf); - for (int k = 0; k < VEC_ELEM_NUM; ++k) { - if (score_buf[k] > best_score) { - best_score = score_buf[k]; - best_idx = i + k; - } - } - } - - for (int64_t i = vec_end; i < vocab_size; ++i) { - float s = row[i] + g_gumbel_table[(seed + i) & GUMBEL_TABLE_MASK]; - if (s > best_score) { - best_score = s; - best_idx = i; - } - } - - output[b] = best_idx; + fused_gumbel_argmax_row(output + b, logits + b * vocab_size, + uint64_t(seeds[b]), vocab_size); } } @@ -91,22 +82,41 @@ static void greedy_argmax_kernel(int64_t* __restrict__ output, for (int64_t b = 0; b < batch_size; ++b) { const float* row = logits + b * vocab_size; + // Vector max has no consistent NaN rule, so NaNs are caught via the sum: + // any NaN (or inf + -inf, a harmless false positive) makes it NaN. vec_op::FP32Vec16 vmax(-std::numeric_limits::infinity()); + vec_op::FP32Vec16 vsum(0.0f); for (int64_t i = 0; i < vec_end; i += VEC_ELEM_NUM) { - vmax = vmax.max(vec_op::FP32Vec16(row + i)); + vec_op::FP32Vec16 v(row + i); + vmax = vmax.max(v); + vsum = vsum + v; } float best_val = vmax.reduce_max(); + bool maybe_nan = std::isnan(vsum.reduce_sum()); for (int64_t i = vec_end; i < vocab_size; ++i) { + maybe_nan |= std::isnan(row[i]); if (row[i] > best_val) { best_val = row[i]; } } - int64_t best_idx = 0; - for (int64_t i = 0; i < vocab_size; ++i) { - if (row[i] == best_val) { - best_idx = i; - break; + int64_t best_idx = -1; + if (maybe_nan) { + // torch.argmax treats NaN as the maximum: return the first NaN. + for (int64_t i = 0; i < vocab_size; ++i) { + if (std::isnan(row[i])) { + best_idx = i; + break; + } + } + } + if (best_idx < 0) { + best_idx = 0; + for (int64_t i = 0; i < vocab_size; ++i) { + if (row[i] == best_val) { + best_idx = i; + break; + } } } output[b] = best_idx; @@ -117,21 +127,28 @@ static void greedy_argmax_kernel(int64_t* __restrict__ output, torch::Tensor fused_gumbel_argmax(const torch::Tensor& logits, const torch::Tensor& seeds) { + TORCH_CHECK(logits.device().is_cpu(), "logits must be a CPU tensor"); TORCH_CHECK(logits.dim() == 2, "logits must be 2-D [batch, vocab]"); TORCH_CHECK(logits.scalar_type() == torch::kFloat32, "logits must be float32"); + TORCH_CHECK(logits.size(1) > 0, "vocab_size must be positive"); + TORCH_CHECK(seeds.device().is_cpu(), "seeds must be a CPU tensor"); + TORCH_CHECK(seeds.scalar_type() == torch::kInt64, "seeds must be int64"); TORCH_CHECK(seeds.dim() == 1 && seeds.size(0) == logits.size(0), "seeds must be 1-D with batch_size elements"); auto logits_contig = logits.contiguous(); + auto seeds_contig = seeds.contiguous(); auto output = torch::empty({logits_contig.size(0)}, torch::kInt64); - fused_gumbel_argmax_kernel( - output.data_ptr(), logits_contig.data_ptr(), - seeds.data_ptr(), logits_contig.size(0), logits_contig.size(1)); + fused_gumbel_argmax_kernel(output.data_ptr(), + logits_contig.data_ptr(), + seeds_contig.data_ptr(), + logits_contig.size(0), logits_contig.size(1)); return output; } torch::Tensor greedy_argmax(const torch::Tensor& logits) { + TORCH_CHECK(logits.device().is_cpu(), "logits must be a CPU tensor"); TORCH_CHECK(logits.dim() == 2, "logits must be 2-D [batch, vocab]"); TORCH_CHECK(logits.scalar_type() == torch::kFloat32, "logits must be float32"); diff --git a/csrc/cumem_allocator.cpp b/csrc/cumem_allocator.cpp index 720c990450..efb1f879d2 100644 --- a/csrc/cumem_allocator.cpp +++ b/csrc/cumem_allocator.cpp @@ -203,38 +203,29 @@ void create_and_map(unsigned long long device, ssize_t size, CUdeviceptr d_mem, } #endif - CUmemAccessDesc accessDesc = {}; - accessDesc.location.type = CU_MEM_LOCATION_TYPE_DEVICE; - accessDesc.location.id = device; - accessDesc.flags = CU_MEM_ACCESS_FLAGS_PROT_READWRITE; - - CUDA_CHECK(cuMemSetAccess(d_mem, size, &accessDesc, 1)); - if (error_code != 0) { - return; - } - -#ifdef USE_ROCM - // ROCm-only: zero the freshly (re)mapped region. - // - // On some amdgpu driver versions the physical VRAM backing a buffer is not - // scrubbed on release, so the pages handed back here (e.g. on wake_up after a - // sleep that discarded a tag) can still contain stale data from a previously - // freed allocation -- possibly from another process. Callers of the device - // allocator expect newly mapped memory to be well defined; in particular - // hybrid Mamba / gated-delta-net (GDN) models keep a persisted conv/recurrent - // state cache, and a new sequence that reads an unzeroed slot will propagate - // garbage/NaN, collapsing generation to token id 0 (decoded as "!"). - // - // This is scoped to the ROCm path on purpose: the CUDA path is unchanged. - // Use CUDA_CHECK so a failure sets the global error_code; the Python caller - // (python_create_and_map) only inspects error_code, so returning without - // setting it would make a failed memset look successful and still hand back - // unzeroed memory. - CUDA_CHECK((CUresult)hipMemset(reinterpret_cast(d_mem), 0, size)); - if (error_code != 0) { - return; - } +#if defined(USE_ROCM) && HIP_VERSION >= 70200000 + // Also map for the host, as PyTorch does for its expandable segments: on + // ROCm >= 7.2, Tensor.item() on a large-BAR device dereferences VRAM + // directly from the host, which segfaults on a device-only mapping. + constexpr int num_desc = 2; +#else + constexpr int num_desc = 1; #endif + CUmemAccessDesc accessDesc[num_desc] = {}; + accessDesc[0].location.type = CU_MEM_LOCATION_TYPE_DEVICE; + accessDesc[0].location.id = device; + accessDesc[0].flags = CU_MEM_ACCESS_FLAGS_PROT_READWRITE; +#if defined(USE_ROCM) && HIP_VERSION >= 70200000 + accessDesc[1].location.type = hipMemLocationTypeHost; + accessDesc[1].location.id = 0; // ignored + accessDesc[1].flags = CU_MEM_ACCESS_FLAGS_PROT_READWRITE; +#endif + + CUDA_CHECK(cuMemSetAccess(d_mem, size, accessDesc, num_desc)); + if (error_code != 0) { + return; + } + // std::cout << "create_and_map: device=" << device << ", size=" << size << ", // d_mem=" << d_mem << ", p_memHandle=" << p_memHandle << std::endl; } diff --git a/csrc/rocm/skinny_gemms_int4.cu b/csrc/rocm/skinny_gemms_int4.cu index a20ca41d41..4588b216fe 100644 --- a/csrc/rocm/skinny_gemms_int4.cu +++ b/csrc/rocm/skinny_gemms_int4.cu @@ -8,6 +8,7 @@ #include #include +#include // std::in_range #include "../cuda_compat.h" #include "dispatch_utils.h" @@ -121,6 +122,7 @@ __device__ __forceinline__ uint32_t zp_nibble(const uint32_t* zp_packed, // GROUP_SIZE: 0 = per-channel scale [M], >0 = per-group scale [M, // K/GROUP_SIZE]. // Requires GROUP_SIZE % A_CHUNK == 0 when GROUP_SIZE > 0. +// B_row_stride_bytes is the per-row byte stride of the packed weights. #if defined(__HIP__GFX1X__) template @@ -130,9 +132,9 @@ __global__ void __launch_bounds__(WvPrGrp* THRDS) const scalar_t* __restrict__ A, const scalar_t* scale, const uint32_t* zero_points, const scalar_t* __restrict__ BIAS, scalar_t* C, - const int _WvPrGrp, const int CuCount) { + const int _WvPrGrp, const int CuCount, + const int B_row_stride_bytes) { constexpr int max_lds_len = LDS_SIZE / 2; - const int K_packed = K / 2; union bigTypeA { scalar_t h[A_CHUNK]; @@ -181,9 +183,9 @@ __global__ void __launch_bounds__(WvPrGrp* THRDS) uint32_t k_ = k + threadIdx.x * A_CHUNK; if (k_ >= K) break; - const uint8_t* B_ = &B_packed[(m + 0) * K_packed + k_ / 2]; + const uint8_t* B_ = &B_packed[(m + 0) * B_row_stride_bytes + k_ / 2]; for (int y = 0; y < YTILE; y++) { - const float* src = (const float*)(&B_[y * K_packed]); + const float* src = (const float*)(&B_[y * B_row_stride_bytes]); #pragma unroll for (int i = 0; i < A_CHUNK / 8; i++) bigB[y][k2].f[i] = loadnt((float*)&src[i]); @@ -344,14 +346,12 @@ __global__ void __launch_bounds__(WvPrGrp* THRDS) #else // !defined(__HIP__GFX1X__) template -__global__ void wvSplitK_int4_hf_sml_(const int K, const int M, const int Bx, - const int By, const uint8_t* B_packed, - const scalar_t* __restrict__ A, - const scalar_t* scale, - const uint32_t* zero_points, - const scalar_t* __restrict__ BIAS, - scalar_t* C, const int _WvPrGrp, - const int CuCount) { +__global__ void wvSplitK_int4_hf_sml_( + const int K, const int M, const int Bx, const int By, + const uint8_t* B_packed, const scalar_t* __restrict__ A, + const scalar_t* scale, const uint32_t* zero_points, + const scalar_t* __restrict__ BIAS, scalar_t* C, const int _WvPrGrp, + const int CuCount, const int B_row_stride_bytes) { UNREACHABLE_CODE } #endif // defined(__HIP__GFX1X__) @@ -367,9 +367,9 @@ __global__ void __launch_bounds__(WvPrGrp* THRDS) const uint8_t* B_packed, const scalar_t* __restrict__ A, const scalar_t* scale, const uint32_t* zero_points, const scalar_t* __restrict__ BIAS, scalar_t* C, - const int _WvPrGrp, const int CuCount) { + const int _WvPrGrp, const int CuCount, + const int B_row_stride_bytes) { constexpr int max_lds_len = LDS_SIZE / 2; - const int K_packed = K / 2; union bigTypeA { scalar_t h[A_CHUNK]; @@ -430,9 +430,9 @@ __global__ void __launch_bounds__(WvPrGrp* THRDS) uint32_t k_ = k + threadIdx.x * A_CHUNK; if (k_ >= K) break; - const uint8_t* B_ = &B_packed[(m + 0) * K_packed + k_ / 2]; + const uint8_t* B_ = &B_packed[(m + 0) * B_row_stride_bytes + k_ / 2]; for (int y = 0; y < YTILE; y++) { - const float* src = (const float*)(&B_[y * K_packed]); + const float* src = (const float*)(&B_[y * B_row_stride_bytes]); #pragma unroll for (int i = 0; i < A_CHUNK / 8; i++) bigB[y][k2].f[i] = loadnt((float*)&src[i]); @@ -606,14 +606,12 @@ __global__ void __launch_bounds__(WvPrGrp* THRDS) #else // !defined(__HIP__GFX1X__) template -__global__ void wvSplitK_int4_hf_(const int K, const int M, const int Bx, - const int By, const uint8_t* B_packed, - const scalar_t* __restrict__ A, - const scalar_t* scale, - const uint32_t* zero_points, - const scalar_t* __restrict__ BIAS, - scalar_t* C, const int _WvPrGrp, - const int CuCount) { +__global__ void wvSplitK_int4_hf_( + const int K, const int M, const int Bx, const int By, + const uint8_t* B_packed, const scalar_t* __restrict__ A, + const scalar_t* scale, const uint32_t* zero_points, + const scalar_t* __restrict__ BIAS, scalar_t* C, const int _WvPrGrp, + const int CuCount, const int B_row_stride_bytes) { UNREACHABLE_CODE } #endif // defined(__HIP__GFX1X__) @@ -658,10 +656,17 @@ torch::Tensor wvSplitK_int4_g(const at::Tensor& in_a, const at::Tensor& in_b, ? in_bias->size(0) : 1; - int64_t expected_weight_bytes = M_in * K_in / 2; - int64_t actual_weight_bytes = in_a.numel() * in_a.element_size(); + const int64_t expected_weight_bytes = M_in * K_in / 2; + const int64_t actual_weight_bytes = in_a.numel() * in_a.element_size(); TORCH_CHECK(actual_weight_bytes == expected_weight_bytes, "Weight tensor must contain M*K/2 bytes for int4 packing"); + const int64_t b_row_stride_bytes = in_a.stride(0) * in_a.element_size(); + TORCH_CHECK(b_row_stride_bytes >= K_in / 2, "Weight row stride (", + b_row_stride_bytes, " bytes) must hold at least K/2=", K_in / 2, + " bytes per row"); + TORCH_CHECK(std::in_range(b_row_stride_bytes), "Weight row stride (", + b_row_stride_bytes, " bytes) exceeds int range"); + const int b_row_stride_bytes_i32 = static_cast(b_row_stride_bytes); TORCH_CHECK( in_b.dtype() == torch::kFloat16 || in_b.dtype() == torch::kBFloat16, "Activation must be float16 or bfloat16"); @@ -722,12 +727,12 @@ torch::Tensor wvSplitK_int4_g(const at::Tensor& in_a, const at::Tensor& in_b, wvSplitK_int4_hf_sml_<<>>( \ K_in, M_in, Bx_in, By_in, wptr, aptr, sptr, zpptr, biasptr, cptr, \ - __wvPrGrp, CuCount); \ + __wvPrGrp, CuCount, b_row_stride_bytes_i32); \ else \ wvSplitK_int4_hf_<<>>( \ K_in, M_in, Bx_in, By_in, wptr, aptr, sptr, zpptr, biasptr, cptr, \ - __wvPrGrp, CuCount); \ + __wvPrGrp, CuCount, b_row_stride_bytes_i32); \ } #define WVSPLITK_INT4G(_YTILE, _UNRL, _N, _GS, _HAS_ZP) \ diff --git a/docker/Dockerfile b/docker/Dockerfile index d8a1258a74..d524d0fddd 100644 --- a/docker/Dockerfile +++ b/docker/Dockerfile @@ -92,6 +92,11 @@ ARG UV_KEYRING_PROVIDER=${PIP_KEYRING_PROVIDER} # Flag enables built-in KV-connector dependency libs into docker images ARG INSTALL_KV_CONNECTORS=false +# Flag builds NIXL, including its NIXL EP extension for the installed PyTorch, +# from source and installs it in place of the nixl packages from the KV-connector +# requirements +ARG BUILD_NIXL=false + #################### BASE BUILD IMAGE #################### # prepare basic build environment FROM ${BUILD_BASE_IMAGE} AS base @@ -148,7 +153,7 @@ RUN if [ "$USE_SCCACHE" = "1" ]; then \ && export SCCACHE_DOWNLOAD_URL="${SCCACHE_DOWNLOAD_URL:-https://github.com/mozilla/sccache/releases/download/v0.8.1/sccache-v0.8.1-${SCCACHE_ARCH}-unknown-linux-musl.tar.gz}" \ && curl -L -o sccache.tar.gz ${SCCACHE_DOWNLOAD_URL} \ && tar -xzf sccache.tar.gz \ - && sudo mv sccache-v0.8.1-${SCCACHE_ARCH}-unknown-linux-musl/sccache /usr/bin/sccache \ + && mv sccache-v0.8.1-${SCCACHE_ARCH}-unknown-linux-musl/sccache /usr/bin/sccache \ && rm -rf sccache.tar.gz sccache-v0.8.1-${SCCACHE_ARCH}-unknown-linux-musl; \ fi @@ -548,6 +553,24 @@ RUN --mount=type=cache,target=/opt/uv/cache \ (cd "${TRITON_WORKSPACE}/dist" \ && sha256sum build-state > wheel.sha256); \ fi + +# Build NIXL and its NIXL EP extension against the PyTorch installed here when +# requested. The wheels are installed by the final image, which has no compiler. +ARG BUILD_NIXL +COPY tools/build_nixl_with_torch215_for_rubin.py /tmp/build_nixl_for_rubin.py +RUN --mount=type=bind,source=requirements,target=/tmp/vllm-requirements,ro \ + set -eux; \ + NIXL_WORKSPACE=/tmp/nixl_workspace; \ + mkdir -p "${NIXL_WORKSPACE}/dist"; \ + if [ "${BUILD_NIXL}" = "true" ]; then \ + dnf install -y --setopt=install_weak_deps=False autoconf automake libtool; \ + dnf clean all; \ + rm -rf /var/cache/dnf; \ + NIXL_REQUIREMENTS_FILE=/tmp/vllm-requirements/kv_connectors.txt \ + WHEELS_CACHE_HOME="${NIXL_WORKSPACE}/dist" \ + MAX_JOBS="${max_jobs:-}" \ + python3 /tmp/build_nixl_for_rubin.py --skip-system-deps --build-only; \ + fi #################### EXTENSIONS BUILD IMAGE #################### #################### WHEEL BUILD IMAGE #################### @@ -1260,6 +1283,25 @@ RUN --mount=type=cache,target=/opt/uv/cache \ /tmp/install_triton_from_source.sh \ /tmp/triton-source-wheel.sha256 +# Install source-built NIXL last, replacing the nixl packages installed above, so +# the image has a NIXL EP build for its PyTorch version. There is no GPU driver +# at build time, so check for the extension instead of importing it. +ARG BUILD_NIXL +RUN --mount=type=cache,target=/opt/uv/cache \ + --mount=type=bind,from=extensions-build,source=/tmp/nixl_workspace/dist,target=/tmp/nixl-dist,ro \ + set -eu; \ + if [ "${BUILD_NIXL}" = "true" ]; then \ + ls /tmp/nixl-dist/*.whl; \ + uv pip uninstall --system nixl nixl-cu12 nixl-cu13; \ + uv pip install --system --no-deps /tmp/nixl-dist/*.whl; \ + python3 -c "import glob, importlib.util as u, os, torch; \ + spec = u.find_spec('nixl_ep_cu' + os.environ['CUDA_VERSION'].split('.')[0]); \ + mm = ''.join(torch.__version__.split('.')[:2]); \ + ext = glob.glob(os.path.join(spec.submodule_search_locations[0], 'nixl_ep_cpp_torch' + mm + '.*')) if spec else []; \ + assert ext, 'NIXL EP has no build for torch ' + torch.__version__; \ + print('NIXL EP extension:', ext[0])"; \ + fi + ENV VLLM_USAGE_SOURCE production-docker-image ENV VLLM_BUILD_COMMIT=${VLLM_BUILD_COMMIT:-unknown} \ VLLM_BUILD_PIPELINE=${VLLM_BUILD_PIPELINE:-local} \ diff --git a/docker/Dockerfile.cpu b/docker/Dockerfile.cpu index f61752475d..b3943a1225 100644 --- a/docker/Dockerfile.cpu +++ b/docker/Dockerfile.cpu @@ -13,7 +13,7 @@ # vllm-dev: used for development # # Build arguments: -# PYTHON_VERSION=3.13|3.12 (default)|3.11|3.10 +# PYTHON_VERSION=3.13|3.12 (default)|3.11 # VLLM_CPU_X86=false (default)|true (for cross-compilation) # VLLM_CPU_ARM_BF16=false (default)|true (for cross-compilation) # @@ -441,6 +441,11 @@ ADD ./cmake/ ./cmake/ # (e.g. tests/tools/test_check_test_tethering.py imports # tools.pre_commit.check_test_tethering). ADD ./tools/ ./tools/ +# tests/test_zentorch_version_pin.py string-matches the zentorch extra in +# setup.py against the torch pin in requirements/cpu.txt. The wheel does not +# ship either file, and this stage does not inherit the vllm-src tree. +ADD ./setup.py ./setup.py +ADD ./requirements/cpu.txt ./requirements/cpu.txt # install development dependencies (for testing) RUN --mount=type=cache,target=/root/.cache/uv \ diff --git a/docker/versions.json b/docker/versions.json index bcec27fd55..68f426d0c7 100644 --- a/docker/versions.json +++ b/docker/versions.json @@ -37,6 +37,9 @@ "INSTALL_KV_CONNECTORS": { "default": "false" }, + "BUILD_NIXL": { + "default": "false" + }, "SCCACHE_BUCKET_NAME": { "default": "vllm-build-sccache" }, diff --git a/docs/benchmarking/cli.md b/docs/benchmarking/cli.md index d05f0c233e..a1001149aa 100644 --- a/docs/benchmarking/cli.md +++ b/docs/benchmarking/cli.md @@ -23,7 +23,6 @@ th { | ShareGPT4V (Image) | ✅ | ✅ | `wget https://huggingface.co/datasets/Lin-Chen/ShareGPT4V/resolve/main/sharegpt4v_instruct_gpt4-vision_cap100k.json`
Note that the images need to be downloaded separately. For example, to download COCO's 2017 Train images:
`wget http://images.cocodataset.org/zips/train2017.zip` | | ShareGPT4Video (Video) | ✅ | ✅ | `git clone https://huggingface.co/datasets/ShareGPT4Video/ShareGPT4Video` | | BurstGPT | ✅ | ✅ | `wget https://github.com/HPMLL/BurstGPT/releases/download/v1.1/BurstGPT_without_fails_2.csv` | -| Sonnet (deprecated) | ✅ | ✅ | Local file: `benchmarks/sonnet.txt` | | Random | ✅ | ✅ | `synthetic` | | RandomMultiModal (Image/Video) | ✅ | ✅ | `synthetic` | | RandomForReranking | ✅ | ✅ | `synthetic` | @@ -773,16 +772,17 @@ Using KV cache metrics for load pattern configuration: ```bash vllm bench throughput \ --model NousResearch/Hermes-3-Llama-3.1-8B \ - --dataset-name sonnet \ - --dataset-path vllm/benchmarks/sonnet.txt \ + --dataset-name random \ + --random-input-len 500 \ + --random-output-len 150 \ --num-prompts 10 ``` If successful, you will see the following output ```text -Throughput: 7.15 requests/s, 4656.00 total tokens/s, 1072.15 output tokens/s -Total num prompt tokens: 5014 +Throughput: 7.15 requests/s, 4647.50 total tokens/s, 1072.50 output tokens/s +Total num prompt tokens: 5000 Total num output tokens: 1500 ``` diff --git a/docs/community/reviewers.md b/docs/community/reviewers.md index fe85e1af49..57840ffd8d 100644 --- a/docs/community/reviewers.md +++ b/docs/community/reviewers.md @@ -9,7 +9,7 @@ This page lists the active reviewers and describes the role. See the [committers Community members can reach out to the reviewers below for help with PRs in their areas, for example in the `#pr-reviews` Slack channel. Reviewers will add their areas of expertise to this list. Sorted alphabetically by GitHub handle: -- [@andylolu2](https://github.com/andylolu2) +- [@andylolu2](https://github.com/andylolu2): Mistral models, speculative decoding, quantization, scheduler, MLA - [@bnellnm](https://github.com/bnellnm) MoE, Quantization layers - [@BowenBao](https://github.com/BowenBao) - [@cjackal](https://github.com/cjackal): Multimodality; KV Connector and offload @@ -30,7 +30,7 @@ Sorted alphabetically by GitHub handle: - [@Rohan138](https://github.com/Rohan138): ROCm performance, CI, torch.compile fusions - [@simondanielsson](https://github.com/simondanielsson): ROCm performance (CDNA), MoRI-IO. - [@taneem-ibrahim](https://github.com/taneem-ibrahim): Pooling models -- [@TheEpicDolphin](https://github.com/TheEpicDolphin) +- [@TheEpicDolphin](https://github.com/TheEpicDolphin): Spec decode - [@varun-sundar-rabindranath](https://github.com/varun-sundar-rabindranath): KV cache offloading, LoRA - [@vllmellm](https://github.com/vllmellm) - [@wangxiyuan](https://github.com/wangxiyuan): Platform, KV cache, Mooncake diff --git a/docs/configuration/conserving_memory.md b/docs/configuration/conserving_memory.md index 96a903bc31..b397283a62 100644 --- a/docs/configuration/conserving_memory.md +++ b/docs/configuration/conserving_memory.md @@ -80,7 +80,7 @@ llm = LLM(model="meta-llama/Llama-3.1-8B-Instruct", enforce_eager=True) If you run out of CPU RAM, try the following options: - (Multi-modal models only) you can set the size of multi-modal cache by setting `mm_processor_cache_gb` engine argument (default 4 GiB). -- (CPU backend only) you can set the size of KV cache using `VLLM_CPU_KVCACHE_SPACE` environment variable (default 4 GiB). +- (CPU backend only) you can set the size of KV cache using `VLLM_CPU_KVCACHE_SPACE` environment variable, in GiB (default 0, meaning vLLM doesn't override the KV cache size through this variable). ## Multi-modal input limits diff --git a/docs/contributing/README.md b/docs/contributing/README.md index f8e9cfcbb1..e7a6448ca6 100644 --- a/docs/contributing/README.md +++ b/docs/contributing/README.md @@ -74,7 +74,7 @@ For an optimized workflow when iterating on C++/CUDA kernels, see the [Increment For JIT kernel warmup conventions, see [JIT Kernel Warmup](./jit_kernel_warmup.md). !!! tip - vLLM is compatible with Python versions 3.10 to 3.13. However, vLLM's default [Dockerfile](../../docker/Dockerfile) ships with Python 3.12 and tests in CI (except `mypy`) are run with Python 3.12. + vLLM is compatible with Python versions 3.11 to 3.14. However, vLLM's default [Dockerfile](../../docker/Dockerfile) ships with Python 3.12 and tests in CI (except `mypy`) are run with Python 3.12. Therefore, we recommend developing with Python 3.12 to minimise the chance of your local environment clashing with our CI environment. @@ -102,14 +102,6 @@ integration, use the configuration files in `tools/pre_commit/`: pre-commit run -a # runs on all files (short for --all-files) ``` - --- - - Some `pre-commit` hooks only run in CI. If you need to, you can run them locally with: - - ```bash - pre-commit run --hook-stage manual mypy-3.11 - ``` - ### Documentation MkDocs is a fast, simple and downright gorgeous static site generator that's geared towards building project documentation. Documentation source files are written in Markdown, and configured with a single YAML configuration file, [mkdocs.yaml](../../mkdocs.yaml). diff --git a/docs/contributing/dockerfile/dockerfile.md b/docs/contributing/dockerfile/dockerfile.md index db46540f9f..279c8f0c6a 100644 --- a/docs/contributing/dockerfile/dockerfile.md +++ b/docs/contributing/dockerfile/dockerfile.md @@ -24,6 +24,11 @@ The `extensions-build` stage can also produce an optional source-built Triton wheel. `vllm-openai-base` installs that wheel after its other Python dependencies so dependency resolution cannot restore an older Triton version. +With `BUILD_NIXL=true`, `extensions-build` also builds NIXL, including its NIXL +EP extension for the installed PyTorch, with `tools/build_nixl_with_torch215_for_rubin.py`. +`vllm-openai-base` then replaces the NIXL packages installed from the KV-connector +requirements with these wheels. + >
> ![](../../assets/contributing/dockerfile-stages-dependency.png){ align="center" alt="query" width="100%" } >
diff --git a/docs/contributing/profiling.md b/docs/contributing/profiling.md index 86bd68acef..5e5e270263 100644 --- a/docs/contributing/profiling.md +++ b/docs/contributing/profiling.md @@ -31,6 +31,18 @@ When using `vllm bench serve`, you can enable profiling by passing the `--profil Traces can be visualized using . +Each worker writes its trace and summary table under `torch_profiler_dir`, +named after its rank, for example `rank0..pt.trace.json.gz` and +`rank0.profiler_out.txt`. Depending on the parallel configuration, the rank +name can also include the parallel ranks, such as +`dp0_pp0_tp0_dcp0_ep0_rank0`. The AsyncLLM CPU trace uses +`_.async_llm` instead of the rank name. When a session sets +`profile_prefix`, it is prepended as `_`, so summaries from +different sessions do not overwrite each other. + +!!! note + The summary table was previously written to `profiler_out_.txt`. + !!! tip You can directly call bench module without installing vLLM using `python -m vllm.entrypoints.cli.main bench`. @@ -88,6 +100,31 @@ curl -X POST http://localhost:8000/v1/chat/completions \ $ curl -X POST http://localhost:8000/stop_profile ``` +`/start_profile` also accepts an optional JSON body with settings for that +profiling session. The Python and Rust API frontends accept the same fields: + +```shell +curl -X POST http://localhost:8000/start_profile \ + -H "Content-Type: application/json" \ + -d '{ + "profile_prefix": "sharegpt_run_1", + "delay_iterations": 5000, + "max_iterations": 20 + }' +``` + +`profile_prefix` is prepended to the generated trace and summary file names. It +must be 1-128 ASCII characters, start with a letter or number, and contain only +letters, numbers, `.`, `_`, or `-`. `delay_iterations` skips worker iterations +before collection starts, and `max_iterations` limits the number of collected +worker iterations (`0` means no limit). Both must be non-negative JSON integers; +values of the wrong type, such as `"5"` or `5.0`, are rejected rather than +converted. Unknown fields are ignored. These iteration +bounds do not apply to the AsyncLLM frontend CPU profiler, which records from +`/start_profile` until `/stop_profile`. Always call `/stop_profile`, including +after the worker reaches `max_iterations`; starting another session before +stopping the current one returns HTTP 409. + ## Profile with Triton Proton [Proton](https://github.com/triton-lang/triton/tree/main/third_party/proton) diff --git a/docs/deployment/frameworks/skypilot.md b/docs/deployment/frameworks/skypilot.md index e032769f15..309367068d 100644 --- a/docs/deployment/frameworks/skypilot.md +++ b/docs/deployment/frameworks/skypilot.md @@ -37,7 +37,7 @@ See the vLLM SkyPilot YAML for serving, [serving.yaml](https://github.com/skypil HF_TOKEN: # Change to your own huggingface token, or use --env to pass. setup: | - conda create -n vllm python=3.10 -y + conda create -n vllm python=3.12 -y conda activate vllm pip install vllm==0.4.0.post1 @@ -136,7 +136,7 @@ SkyPilot can scale up the service to multiple service replicas with built-in aut HF_TOKEN: # Change to your own huggingface token, or use --env to pass. setup: | - conda create -n vllm python=3.10 -y + conda create -n vllm python=3.12 -y conda activate vllm pip install vllm==0.4.0.post1 @@ -248,7 +248,7 @@ This will scale the service up to when the QPS exceeds 2 for each replica. HF_TOKEN: # Change to your own huggingface token, or use --env to pass. setup: | - conda create -n vllm python=3.10 -y + conda create -n vllm python=3.12 -y conda activate vllm pip install vllm==0.4.0.post1 @@ -293,7 +293,7 @@ It is also possible to access the Llama-3 service with a separate GUI frontend, cpus: 2 setup: | - conda create -n vllm python=3.10 -y + conda create -n vllm python=3.12 -y conda activate vllm # Install Gradio for web UI. diff --git a/docs/design/cuda_graphs.md b/docs/design/cuda_graphs.md index abebdd54ae..9d53a98f65 100644 --- a/docs/design/cuda_graphs.md +++ b/docs/design/cuda_graphs.md @@ -225,9 +225,9 @@ outputs = model.generate( ) ``` -### Piecewise compilation and full graph custom passes (attention fusion, sequence parallelism) +### Piecewise compilation and full graph custom passes (attention fusion) -Unfortunately, some custom compile passes have to see the whole graph to be effective and hence aren't compatible with piecewise compilation. This includes `AttnQuantFusionPass` and `SequenceParallelismPass`. As a short-term solution, we automatically disable piecewise compilation (by setting `splitting_ops=[]`) when attention fusion is enabled. We use CUDA Graph modes `FULL` or `FULL_DECODE_ONLY` (depending on backend support). However, this leads to another optimization incompatibility and confusing performance tradeoffs. +Unfortunately, some custom compile passes have to see the whole graph to be effective and hence aren't compatible with piecewise compilation. This includes `AttnQuantFusionPass`. As a short-term solution, we automatically disable piecewise compilation (by setting `splitting_ops=[]`) when attention fusion is enabled. We use CUDA Graph modes `FULL` or `FULL_DECODE_ONLY` (depending on backend support). However, this leads to another optimization incompatibility and confusing performance tradeoffs. Long term, we've added the ability to partition the graph in Inductor instead of right after Dynamo. It can be enabled with `CompilationConfig.use_inductor_graph_partition=True` but is currently experimental and only available with `torch>=2.9`. This also increases compilation time as it has to compile the whole graph and cannot reuse piecewise compilation artifacts. Once vLLM supports 2.9, we plan to make this the default approach as it will also speed up piecewise cudagraph capture. diff --git a/docs/design/cuda_graphs_multimodal.md b/docs/design/cuda_graphs_multimodal.md index dca200e2de..0a8a831f01 100644 --- a/docs/design/cuda_graphs_multimodal.md +++ b/docs/design/cuda_graphs_multimodal.md @@ -7,27 +7,20 @@ For two-tower vision encoders (e.g., DeepSeek-OCR's SAM + CLIP with dynamic tili !!! note Encoder CUDA Graphs are orthogonal to decoder CUDA Graphs — both can be enabled simultaneously. Encoder graphs capture the vision encoder execution (e.g., ViT in Qwen3-VL), while decoder graphs capture the language model execution as described in the [CUDA Graphs design document](cuda_graphs.md). -## Compatibility Matrix - -!!! note - The symbols used below have the following meanings: - - - ✅ = Full compatibility - - 🟠 = Partial compatibility - - ❌ = No compatibility - - ❔ = Unknown or TBD - -### Model x Feature +## Supported Models | Architecture | Models | CG for Image | CG for Video | Multi-Path Graph | | ------------ | ------ | ------------ | ------------ | --------------- | | `DeepseekOCRForCausalLM` | `DeepSeek-OCR` | ✅︎ | ❌︎ | ✅︎ | +| `DeepseekV41ForCausalLM` | `DeepSeek-V4.1-Flash` | ✅︎ | ❌︎ | ❌︎ | | `Ernie4_5_VLMoeForConditionalGeneration` | `ERNIE-4.5-VL` | ✅︎ | ❌︎ | ❌︎ | | `Gemma3ForConditionalGeneration` | `Gemma3` | ✅︎ | ❌︎ | ❌︎ | | `Glm4vForConditionalGeneration` | `GLM-4.1V, GLM-4.6V-Flash` | ✅︎ | ✅︎ | ❌︎ | | `Gemma4ForConditionalGeneration` | `Gemma-4` | ✅︎ | ✅︎ | ❌︎ | | `InternVLChatModel` | `InternVL3.5`, `InternVL3`, `InternVL2.5`, `InternVL2` | ✅︎ | ✅︎ | ❌︎ | | `KimiVLForConditionalGeneration` | `Kimi-VL` | ✅︎ | ❌︎ | ❌︎ | +| `KimiK25ForConditionalGeneration` | `Kimi-K2.5, Kimi-K2.6` | ✅︎ | ❌︎ | ❌︎ | +| `KimiK3ForConditionalGeneration` | `Kimi-K3` | ✅︎ | ❌︎ | ❌︎ | | `Llama4ForConditionalGeneration` | `Llama 4` | ✅︎ | ❌︎ | ❌︎ | | `Qwen2VLForConditionalGeneration` | `Qwen2-VL` | ✅︎ | ✅︎ | ❌︎ | | `Qwen2_5_VLForConditionalGeneration` | `Qwen2.5-VL` | ✅︎ | ✅︎ | ❌︎ | @@ -38,26 +31,6 @@ For two-tower vision encoders (e.g., DeepSeek-OCR's SAM + CLIP with dynamic tili | `MiniCPMV` | `MiniCPMV2.5` | ✅︎ | ❌︎ | ❌︎ | | `MiniCPMV` | `MiniCPMV2.6`, `MiniCPMV4.0` | ✅︎ | ✅︎ | ❌︎ | -### Model x Hardware - -| Architecture | NV Blackwell | NV Ampere | AMD MI300X | AMD MI350X / MI355X | -| ------------ | ---------------- | ------------- | -------------- | --------------------- | -| `DeepseekOCRForCausalLM` | ✅︎ | ✅︎ | ❔ | ✅︎ | -| `Ernie4_5_VLMoeForConditionalGeneration` | ✅︎ | ✅︎ | ❔ | ✅︎ | -| `Gemma3ForConditionalGeneration` | ✅︎ | ✅︎ | ❔ | ✅︎ | -| `Glm4vForConditionalGeneration` | ✅︎ | ✅︎ | ❔ | ✅︎ | -| `Gemma4ForConditionalGeneration` | ✅︎ | ✅︎ | ❔ | ✅︎ | -| `InternVLChatModel` | ✅︎ | ✅︎ | ❔ | ✅︎ | -| `KimiVLForConditionalGeneration` | ✅︎ | ✅︎ | ❔ | ✅︎ | -| `Llama4ForConditionalGeneration` | ✅︎ | ✅︎ | ❔ | ✅︎ | -| `Qwen2VLForConditionalGeneration` | ✅︎ | ✅︎ | ❔ | ✅︎ | -| `Qwen2_5_VLForConditionalGeneration` | ✅︎ | ✅︎ | ❔ | ✅︎ | -| `Qwen3VLForConditionalGeneration` | ✅︎ | ✅︎ | ❔ | ✅︎ | -| `Qwen3_5ForConditionalGeneration` | ✅︎ | ✅︎ | ❔ | ✅︎ | -| `Qwen3_5MoeForConditionalGeneration` | ✅︎ | ✅︎ | ❔ | ✅︎ | -| `Step3VLForConditionalGeneration` | ✅︎ | ✅︎ | ❔ | ✅︎ | -| `MiniCPMV` | ✅︎ | ✅︎ | ❔ | ❔ | - !!! note Encoder CUDA Graph has currently been tested with `--mm-encoder-attn-backend=FLASH_ATTN` and `--mm-encoder-attn-backend=FLASHINFER` on Blackwell GPUs. For Qwen2-VL and Qwen2.5-VL only FA2 and FA3 has been tested. @@ -80,7 +53,7 @@ The encoder CUDA Graph system uses a **budget-based capture/replay** strategy, m * [EncoderItemSpec][vllm.v1.worker.encoder_cudagraph_defs.EncoderItemSpec]: describes a single encoder input item (image or video) with its input size and output token count. * [BudgetGraphMetadata][vllm.v1.worker.encoder_cudagraph.BudgetGraphMetadata]: holds the captured CUDA Graph and its associated I/O buffers for a single token budget level. -### Budget-based graph capture +### Budget-Based Graph Capture Multiple CUDA Graphs are pre-captured at different **token budget** levels (e.g., `[2048, 4096, 8192, 13824]`). Each budget defines a fixed token capacity, and all budgets share the same maximum batch size (number of images). The `BudgetGraphMetadata` for each level stores the graph along with pre-allocated input, metadata, and output buffers: @@ -99,20 +72,7 @@ Budgets are auto-generated as power-of-2 levels from a model-provided range via Each entry in `EncoderCudaGraphConfig.paths` defines an independently captured encoder path. A path can provide its own minimum token budget and opt into zero-token batches; the manager generates and stores a separate budget graph set for every configured path. -### Greedy bin-packing at runtime - -When a batch of images arrives, the manager sorts images by output token count (smallest first) and greedily packs as many images as possible into each sub-batch while staying within the **largest** token budget and the maximum batch size. Once a sub-batch is finalized (the next image would overflow either constraint), the manager finds the **smallest** budget that fits the sub-batch's total tokens and replays the corresponding CUDA Graph. This repeats until the batch is exhausted. Images that exceed all budgets fall back to eager execution. - -For multi-path models, the same greedy packing loop constrains every configured path simultaneously (see [Multi-Path graph capture](#multi-path-graph-capture)). - -For each graph replay: - -1. Call `prepare_encoder_cudagraph_replay_buffers()` to compute buffer values (including `pixel_values` and precomputed metadata) from actual batch inputs. -2. Zero the pre-allocated `input_buffers`, then slice-copy the replay values into them. -3. Replay the CUDA Graph. -4. Clone outputs from `output_buffer` (cloning is necessary since the buffer is reused across replays). - -### Multi-Path graph capture +### Multi-Path Graph Capture `EncoderCudaGraphConfig.paths` maps path names to `EncoderCudaGraphPathConfig` capture policies. For example, DeepSeek-OCR configures a **global** image path and a **local** patch path, which are captured independently under `budget_graphs["global"]` and `budget_graphs["local"]`. @@ -127,7 +87,7 @@ Both lists are capped at the same `max_budget`. * Sort images by total output tokens (global + local), smallest first. * Greedily pack images: an image is added to the current sub-batch only if every accumulated path token count is within that path's maximum budget, with the image count ≤ `max_batch_size`. -* Once any path constraint would overflow, finalize the sub-batch and find the smallest fitting budget **independently** for each path. +* Once any path constraint would overflow, or the budget-cliff deferral condition fires on any path, finalize the sub-batch and find the smallest fitting budget **independently** for each path. * Repeat until all images are packed. **Partial graph fallback.** For each non-empty path, the manager replays the smallest fitting graph or runs only that path eagerly when no graph fits. Paths with zero tokens are skipped; a `0`-budget graph is never captured or replayed. @@ -139,11 +99,41 @@ Both lists are capped at the same `max_budget`. !!! note The dual-path design enables partial CUDA graph coverage — one path can hit while the other falls back to eager. This avoids wasted compute on zero-padded patch buffers for untiled images and avoids graph invalidation caused by variable `crop_shape` per image. -### Data-parallel support +### Greedy Bin-Packing at Runtime + +When a batch of images arrives, the manager sorts images by output token count (smallest first) and greedily packs as many images as possible into each sub-batch while staying within the **largest** token budget and the maximum batch size. Once a sub-batch is finalized (the next image would overflow either constraint, or deferring it is cheaper), the manager finds the **smallest** budget that fits the sub-batch's total tokens and replays the corresponding CUDA Graph. This repeats until the batch is exhausted. Images that exceed all budgets fall back to eager execution. + +For multi-path models, the same greedy packing loop constrains every configured path simultaneously. + +For each graph replay: + +1. Call `prepare_encoder_cudagraph_replay_buffers()` to compute buffer values (including `pixel_values` and precomputed metadata) from actual batch inputs. +2. Zero the pre-allocated `input_buffers`, then slice-copy the replay values into them. +3. Replay the CUDA Graph. +4. Clone outputs from `output_buffer` (cloning is necessary since the buffer is reused across replays). + +**Budget-cliff deferral:** + +Greedy packing can be wasteful when a large image arrives after small ones: merging it into the current sub-batch may force the whole batch into a much larger budget graph even though splitting would cost less. Because token budgets are discrete (e.g., `[256, 4096, 8192]`), one extra token can double the required graph level — a "budget cliff". Images that overflow the largest budget still finalize the sub-batch unconditionally; for images that *fit*, the packer additionally applies a deferral test: finalize the current sub-batch (replaying it at its own small budget) and start a new sub-batch with the deferred image. + +For each candidate image, the packer compares three budgets per path: `b_cur` (the smallest budget fitting the current sub-batch), `b_alone` (the smallest budget fitting the image alone), and `b_merged` (the smallest budget fitting both). Merging costs `b_merged`; deferring costs `b_cur + b_alone` plus one extra replay. The image is deferred when, on any path, both conditions hold: + +1. `b_cur + b_alone < b_merged` — splitting strictly saves budget. The strict inequality makes ties merge: with budgets `[256, 4096, 8192]`, two 4000-token images tie (`4096 + 4096 == 8192`), and merging wins because a single replay avoids an extra graph launch. +2. `b_alone >= b_cur` — the image alone occupies at least as large a budget bucket as the entire current sub-batch. + +**Condition 1 is the cost test.** For example, a 200-token image followed by a 4000-token image with budgets `[256, 4096, 8192]`: merging takes `b_merged = 8192` (4200 tokens round up to 8192), while deferring takes `256 + 4096 = 4352` — the image is deferred, saving 3840 tokens of padding compute. + +**Condition 2 separates "whales" from "tag-alongs".** Because images are sorted smallest first, a whale arrives last, and deferring it lets the small sub-batch replay at its own small budget instead of being dragged into a large bucket for one image's sake. Without condition 2, deferring mid-size images fragments a flood of similar images into extra replays: with budgets `[256, 2048, 4096, 8192]` and six equal 1366-token images, condition 1 alone defers every third image and yields three 4096 replays (12288 budget tokens in total), while with condition 2 five images merge into one 8192 replay plus a single 2048 replay — fewer replays *and* fewer budget tokens (10240). + +The conditions compose cleanly: when `b_alone == b_cur`, condition 1 requires `b_merged > 2 * b_cur`, i.e. deferral fires only when merging jumps at least two budget levels, so ties always merge. This also lets small images defer across a huge cliff: with budgets `[256, 4096]` and four 100-token images, adding the third image (300 tokens) would jump from 256 to 4096, so the packer defers it and packs the images as two 256-budget pairs. + +For multi-path items, the conditions are evaluated per path and OR-ed: any path on which the image is a whale defers the whole image, while paths where the image contributes zero tokens are skipped. + +### ViT DP Mode Support When `mm_encoder_tp_mode="data"`, the manager distributes images across TP ranks using load-balanced assignment via `get_load_balance_assignment`, executes locally on each rank, then gathers results back in the original order via `tensor_model_parallel_all_gather`. -### Video inference support +### Video Inference Support Following (ViT full CUDA graph support for image inference), extends the encoder CUDA graph framework to support video inference for Qwen3-VL. Previously, the CUDA graph capture/replay path only handled image inputs (`pixel_values` + `image_grid_thw`). Video inputs use different keys (`pixel_values_videos` + `video_grid_thw`) and require larger `cu_seqlens` buffers because each video item contributes multiple frames (`T` attention sequences). This PR generalizes the protocol and manager to handle both modalities through a single shared graph manager. @@ -152,7 +142,7 @@ Following (ViT full CUDA graph Mixed inputs (image+video) per prompt are also supported now. -## Model integration via `SupportsEncoderCudaGraph` +## Model Integration via `SupportsEncoderCudaGraph` Models opt-in to encoder CUDA Graphs by implementing the [SupportsEncoderCudaGraph][vllm.model_executor.models.interfaces.SupportsEncoderCudaGraph] protocol. This protocol encapsulates all model-specific logic so that the manager remains model-agnostic. The protocol defines the following methods: @@ -180,9 +170,9 @@ Four fields in `CompilationConfig` control encoder CUDA Graphs: Multi-path mode is configured at the model level through `EncoderCudaGraphConfig.paths`. Each `EncoderCudaGraphPathConfig` can set a path-specific minimum budget and whether zero-token batches are allowed. The manager automatically generates separate budget lists and uses the same execution loop for single- and multi-path models. -## Usage guide +## Usage Guide -### Image inference +### Image Inference Enable encoder CUDA Graphs via `compilation_config`: @@ -226,7 +216,7 @@ model = vllm.LLM( The manager tracks hit/miss statistics and logs them periodically. A "hit" means an image was processed via CUDA Graph replay; a "miss" means eager fallback (image exceeded all budgets). -### Video inference +### Video Inference Enable encoder CUDA Graphs via `compilation_config`: diff --git a/docs/design/debug_vllm_compile.md b/docs/design/debug_vllm_compile.md index 7edda6fa64..af9b3f530d 100644 --- a/docs/design/debug_vllm_compile.md +++ b/docs/design/debug_vllm_compile.md @@ -251,6 +251,18 @@ that may call 1+ triton kernels. On rare (but unfortunate) occasions, it may produce an incorrect triton kernel. This may manifest as silent incorrectness, CUDA illegal memory accesses, or loud errors. +### Deterministic compilation + +When vLLM selects combo-kernel defaults, it disables combo-kernel benchmarking +if Inductor's `deterministic` setting is enabled. A `deterministic` value in +`inductor_compile_config` takes precedence over the global Inductor setting. +On torch 2.13, `TORCHINDUCTOR_DETERMINISTIC=1` alone is reset by the first +compiled frame ([pytorch#198563](https://github.com/pytorch/pytorch/issues/198563)), +so set `deterministic` in `inductor_compile_config` instead. +Explicit `combo_kernels` or `benchmark_combo_kernel` settings are preserved. +This avoids conflicting compilation defaults; it does not guarantee reproducible +model outputs. + ### Inductor runtime assertions By default (on torch < 2.12), vLLM disables Inductor's runtime assertions diff --git a/docs/design/fusions.md b/docs/design/fusions.md index 002984c113..4095c41a7e 100644 --- a/docs/design/fusions.md +++ b/docs/design/fusions.md @@ -25,8 +25,6 @@ or just on the low or high end. | [MLA Attention + Quant](#attention--quantization-fuse_attn_quant) | `fuse_attn_quant` | MLA Attention output → FP8/NVFP4 quant | Off by default | TBD | Yes | Always | | [RoPE + KV-Cache Update](#rope--kv-cache-update-fuse_rope_kvcache) | `fuse_rope_kvcache` | Rotary embedding → KV cache write | O2 (ROCm/AITER only) | 2-4% | No | Low | | [QK Norm + RoPE](#qk-norm--rope-enable_qk_norm_rope_fusion) | `enable_qk_norm_rope_fusion` | Q/K RMSNorm → rotary embedding | Off by default | 2-3% | No | Low | -| [Sequence Parallelism](#sequence-parallelism-enable_sp) | `enable_sp` | AllReduce → ReduceScatter + AllGather | Off by default | Prereq for AsyncTP | Yes | High | -| [AsyncTP GEMM + collective](#asynctp-gemm--collective-overlap-fuse_gemm_comms) | `fuse_gemm_comms` | GEMM → reduce-scatter / all-gather → GEMM | Off by default | 7-10% | Yes | High | | [RMSNorm + Quant](#rmsnorm--quantization-fuse_norm_quant) | `fuse_norm_quant` | RMSNorm (+residual add) → FP8/FP4 quant | O1 (conditional) | 1-4% | No | Always | | [SiLU+Mul + Quant](#silumul--quantization-fuse_act_quant) | `fuse_act_quant` | SiLU+Mul activation → FP8/FP4 quant | O1 (conditional) | 1-4% | No | Always | | [RMSNorm + Padding](#rmsnorm--padding-fuse_act_padding) | `fuse_act_padding` | Residual add + RMSNorm → padding | O1 (ROCm/AITER only) | TBD | No | Always | @@ -44,9 +42,7 @@ The table below lists the quantization schemes supported by each fusion on each | `fuse_attn_quant`\* | FP8 static\*, NVFP4\* | FP8 static\* | FP8 static\* | — | FP8 static\* | | `fuse_attn_quant` (MLA)\* | FP8 static\*, FP8 per-group\*, NVFP4\* | FP8 static\*, FP8 per-group\* | FP8 static\*, FP8 per-group\* | — | FP8 static\* (untested) | | `fuse_rope_kvcache` | — | — | — | — | FP16/BF16 | -| `enable_qk_norm_rope_fusion` | FP16/BF16 | FP16/BF16 | FP16/BF16† | FP16/BF16† | — | -| `enable_sp` | FP16/BF16, FP8 static† | FP16/BF16, FP8 static | FP16/BF16† | FP16/BF16† | — | -| `fuse_gemm_comms` | FP16/BF16, FP8 static† | FP16/BF16, FP8 static | FP16/BF16† | FP16/BF16† | — | +| `enable_qk_norm_rope_fusion` | FP16/BF16 | FP16/BF16 | FP16/BF16 | FP16/BF16 | — | | `fuse_norm_quant` | FP8 static, FP8 per-token, FP8 per-group | FP8 static, FP8 per-token, FP8 per-group | FP8 static, FP8 per-token, FP8 per-group | — | FP8 static, FP8 per-token, FP8 per-group | | `fuse_act_quant` | FP8 static, NVFP4 | FP8 static, FP8 per-group (128/64) | FP8 static, FP8 per-group (128/64) | — | FP8 per-group | | `fuse_act_padding` | — | — | — | — | FP16/BF16 | @@ -56,10 +52,6 @@ The table below lists the quantization schemes supported by each fusion on each fused quantization output. See the [`fuse_attn_quant` section](#attention--quantization-fuse_attn_quant) for per-backend details. -† `enable_sp` and `fuse_gemm_comms` are only autoconfigured for SM90 today; -other architectures support requires setting `PassConfig.sp_min_token_num` explicitly. -SM100 support also requires setting `VLLM_DISABLED_KERNELS=FlashInferFP8ScaledMMLinearKernel`. - ## Enabling / Disabling Fusions Fusions are exposed through `PassConfig`, which is nested inside `CompilationConfig`: @@ -186,72 +178,6 @@ If these conditions are set, the fusion is enabled automatically for optimizatio - Pass: [`vllm/compilation/passes/fusion/rope_kvcache_fusion.py`](https://github.com/vllm-project/vllm/blob/main/vllm/compilation/passes/fusion/rope_kvcache_fusion.py) -### Sequence Parallelism (`enable_sp`) - -**What it fuses.** Replaces all-reduce collectives with reduce-scatter + local RMSNorm + all-gather, -splitting the sequence dimension across TP ranks. This restructures the graph so the subsequent AsyncTP -pass can fuse the reduce-scatter / all-gather with the surrounding GEMMs. - -Sequence Parallelism itself does not directly improve performance; it is a prerequisite for the -AsyncTP pass (`fuse_gemm_comms`). SP is only applied above a minimum token threshold that is -autoconfigured based on device capability and model `hidden_size`. Currently only active on -H100/SM90 for models with `hidden_size >= 8192`. The threshold is configurable via -`PassConfig.sp_min_token_num`. - -The general transformation: - -```text -Input → AllReduce → RMSNorm → Output -becomes: -Input → ReduceScatter → local RMSNorm → AllGather → Output -``` - -Patterns covered: - -- First block: `AllReduce → RMSNorm` → `ReduceScatter → RMSNorm → AllGather` -- Middle blocks: `AllReduce → fused_add_RMSNorm` → `ReduceScatter → fused_add_RMSNorm → AllGather` -- Both with optional `→ FP8 static quant` suffix - -Requires: `use_inductor_graph_partition=True` **or** piecewise compilation with static sizes -divisible by `tensor_parallel_size`. - -Supported hardware: Only tested on NVIDIA CUDA, possibly works on ROCm. FP8 all-gather requires sm90+. - -**Code locations.** - -- Pass: [`vllm/compilation/passes/fusion/sequence_parallelism.py`](https://github.com/vllm-project/vllm/blob/main/vllm/compilation/passes/fusion/sequence_parallelism.py) - -### AsyncTP GEMM + Collective Overlap (`fuse_gemm_comms`) - -!!! info - Requires `enable_sp=True` (enabled automatically). This pass is a no-op if Sequence Parallelism has not been applied. - -**What it fuses.** After Sequence Parallelism transforms the graph, fuses GEMM kernels with the -surrounding reduce-scatter (output projection) and all-gather (input projection) using -`torch.ops.symm_mem` symmetric-memory primitives, overlapping communication and computation. -This overlap is only profitable for large `num_tokens`, so the fusion (and preceding SP) -is only performed in the higher compiled range above `PassConfig.sp_min_token_num`. - -Patterns covered: - -- `GEMM → reduce-scatter` → `fused_matmul_reduce_scatter` -- `all-gather → GEMM` → `all_gather_matmul` -- FP8 scaled variants of both patterns - -Supported hardware: NVIDIA CUDA with symmetric-memory (`torch.distributed._symmetric_memory`) support. - -On B200, pattern-matching fp8 FlashInfer scaled MM is not supported, so it must be disabled -([#27893](https://github.com/vllm-project/vllm/issues/27893)) - -```shell -VLLM_DISABLED_KERNELS=FlashInferFP8ScaledMMLinearKernel ... -``` - -**Code locations.** - -- Pass: [`vllm/compilation/passes/fusion/collective_fusion.py`](https://github.com/vllm-project/vllm/blob/main/vllm/compilation/passes/fusion/collective_fusion.py) -- Sequence parallelism pass: [`vllm/compilation/passes/fusion/sequence_parallelism.py`](https://github.com/vllm-project/vllm/blob/main/vllm/compilation/passes/fusion/sequence_parallelism.py) - ### QK Norm + RoPE (`enable_qk_norm_rope_fusion`) !!! info diff --git a/docs/design/hisparse.md b/docs/design/hisparse.md index 8651b3a06d..f5fab3c0f5 100644 --- a/docs/design/hisparse.md +++ b/docs/design/hisparse.md @@ -37,6 +37,14 @@ memory consumption is therefore topology- and implementation-dependent. The realized capacity may be slightly smaller because the budget is rounded down to complete host blocks. +Startup logs report two concurrency bounds at `max_model_len`. The generic +`Maximum concurrency` line charges each request its full admission footprint, +including the in-flight window of every resident group. The `HiSparse +steady-state maximum concurrency` line charges running requests that read from +host only their active tail pages, plus one request being admitted at its full +footprint. Host-pool metrics are listed in +[Metrics](../usage/metrics.md#hisparse-kv-connector-metrics). + ## Ownership | Thing | Owner | What “owner” means | diff --git a/docs/design/mm_processing.md b/docs/design/mm_processing.md index 67b1a7742b..a267aee38e 100644 --- a/docs/design/mm_processing.md +++ b/docs/design/mm_processing.md @@ -85,6 +85,8 @@ GPU-side fusion is controlled by `multimodal_config.mm_device_do_normalize`: | `glm-ocr` | `GlmOcrForConditionalGeneration` | `zai-org/GLM-OCR` | | `intern-s1-pro` | `InternS1ProForConditionalGeneration` | `internlm/Intern-S1-Pro` | | `intern-s2-mobius` | `InternS2MobiusForConditionalGeneration` | `internlm/Intern-S2-Mobius` | +| `kimi-k2.5` | `KimiK25ForConditionalGeneration` | `moonshotai/Kimi-K2.5`, etc. | +| `kimi-k3` | `KimiK3ForConditionalGeneration` | `moonshotai/Kimi-K3` | | `llama-nemotron-vl-embed` | `LlamaNemotronVLForEmbedding` | `nvidia/llama-nemotron-embed-vl-1b-v2` | | `llama-nemotron-vl-rerank` | `LlamaNemotronVLForSequenceClassification` | `nvidia/llama-nemotron-rerank-vl-1b-v2` | | `mistral3` | `Mistral3ForConditionalGeneration` | `mistralai/Mistral-Small-3.2-24B-Instruct-2506`, etc. | diff --git a/docs/design/vllm_ir.md b/docs/design/vllm_ir.md index 82628f3762..e324ad25d0 100644 --- a/docs/design/vllm_ir.md +++ b/docs/design/vllm_ir.md @@ -233,19 +233,17 @@ storing this information in vLLM's `PassContext` for later use in clone eliminat ### 3. IR Fusion and Transformation Passes After functionalization, custom vLLM passes operate on the functional FX graph containing high-level IR operations. -These passes can perform fusion, distribute operations for sequence parallelism, and other transformations: +These passes can perform fusion and other transformations: ```python -# Example: Sequence Parallelism (see SequenceParallelismPass) -# Before SP pass +# Example: AllReduce + RMSNorm fusion (see AllReduceFusionPass) +# Before all_reduce = torch.ops.vllm.all_reduce(x, "tp:0") rms_norm = torch.ops.vllm_ir.rms_norm(all_reduce, weight, 1e-5) -# after SP pass -reduce_scatter = torch.ops.vllm.reduce_scatter(x, "tp:0") -rms_norm = torch.ops.vllm_ir.rms_norm(all_reduce, weight, 1e-5) -all_gather = torch.ops.vllm.all_gather(x, "tp:0") +# After +rms_norm = torch.ops.vllm.flashinfer_trtllm_fused_allreduce_norm(x, weight, ...) ``` Fusion passes benefit from the high-level representation: they don't need to match against low-level PyTorch operations, diff --git a/docs/features/batch_invariance.md b/docs/features/batch_invariance.md index e82336d241..03d07896cc 100644 --- a/docs/features/batch_invariance.md +++ b/docs/features/batch_invariance.md @@ -126,7 +126,7 @@ Batch invariance has been tested and verified on the following models: - **Qwen3 (Dense)**: `Qwen/Qwen3-1.7B`, `Qwen/Qwen3-8B`, `Qwen/Qwen3-4B-AWQ`, `Qwen/Qwen3-8B-AWQ` - **Qwen3-VL (Vision-Language)**: `Qwen/Qwen3-VL-2B-Instruct`, `Qwen/Qwen3-VL-4B-Instruct` (single image and video inputs) - **Qwen3 (MoE)**: `Qwen/Qwen3-30B-A3B`, `Qwen/Qwen3-Next-80B-A3B-Instruct`, `Qwen/Qwen3-30B-A3B-Thinking-2507-FP8` -- **Qwen2.5**: `Qwen/Qwen2.5-0.5B-Instruct`, `Qwen/Qwen2.5-1.5B-Instruct`, `Qwen/Qwen2.5-3B-Instruct`, `Qwen/Qwen2.5-7B-Instruct`, `Qwen/Qwen2.5-14B-Instruct`, `Qwen/Qwen2.5-32B-Instruct` +- **Qwen2.5**: `Qwen/Qwen2.5-0.5B-Instruct`, `Qwen/Qwen2.5-1.5B-Instruct`, `Qwen/Qwen2.5-3B-Instruct`, `Qwen/Qwen2.5-7B-Instruct`, `Qwen/Qwen2.5-14B-Instruct`, `Qwen/Qwen2.5-32B-Instruct`, `Qwen/Qwen2.5-Coder-7B-Instruct` - **Llama 3**: Llama3.1 and 3.2 series, `meta-llama/Llama-3.2-1B-Instruct`, `meta-llama/Llama-3.2-3B-Instruct` for example - **GPT-OSS**: `openai/gpt-oss-20b`, `openai/gpt-oss-120b` - **Mistral**: `mistralai/Mistral-7B-v0.3` @@ -148,11 +148,10 @@ When batch invariance is enabled, vLLM: 1. Uses deterministic kernel implementations for attention and other operations 2. Ensures consistent numerical behavior across different batch sizes -3. Disables certain optimizations that may introduce non-determinism (such as sequence parallelism / async TP, whose reduce-scatter path is not batch-invariant) +3. Disables certain optimizations that may introduce non-determinism 4. Under tensor parallelism, keeps custom all-reduce on with a fixed reduction order (the 1-stage kernel is pinned, and large inputs are reduced in fixed-size chunks), and disables FlashInfer, AITER and QuickReduce all-reduce 5. On CUDA devices with tuned matmul table entries for the model's bf16 unquantized forward linear layers (Ada, Hopper, Blackwell), - runs without `torch.compile` using breakable CUDA graphs so tile configs follow the runtime batch size; set `VLLM_USE_BREAKABLE_CUDAGRAPH=0` to opt out - (not applied when sequence parallelism / async TP are enabled, since those are `torch.compile` passes). + runs without `torch.compile` using breakable CUDA graphs so tile configs follow the runtime batch size; set `VLLM_USE_BREAKABLE_CUDAGRAPH=0` to opt out. 6. Splits the LoRA shrink reduction into 8 fixed K chunks that are summed in a fixed order instead of with atomics !!! warning diff --git a/docs/features/lora.md b/docs/features/lora.md index 4c95bde8cd..c05bc8c0fa 100644 --- a/docs/features/lora.md +++ b/docs/features/lora.md @@ -280,10 +280,10 @@ curl -X POST http://localhost:8000/v1/load_lora_adapter \ !!! warning "You must know your adapter's layout" Under `--enable-mixed-moe-lora-format`, vLLM trusts whatever - `is_3d_lora_weight` the caller declares — it does **not** inspect the - checkpoint to verify. A wrong declaration will load weights into the - wrong stacked buffers and silently produce garbage outputs, with no - error at load time. Confirm the layout before serving: + `is_3d_lora_weight` the caller declares. Loading checks reject fused 3D + expert weights routed to a 2D wrapper without the required flags, but + do not validate every layout mismatch. A wrong declaration can still + cause loading failures or incorrect outputs. Confirm the layout before serving: - **2D (per-expert, megatron-style)** → set `is_3d_lora_weight: false`. Adapter keys look like `...experts.{idx}.gate_proj.lora_A.weight`, @@ -299,6 +299,11 @@ is ignored: vLLM picks the wrapper from the base model's `is_3d_moe_weight` and the adapter is required to match. The field is also ignored for non-MoE models. +For a model using a 2D MoE wrapper (such as Qwen3-MoE), loading fused 3D +expert adapter weights requires both `enable_mixed_moe_lora_format=True` +on the engine and `is_3d_lora_weight=True` on the request. Otherwise, adapter +loading raises a `ValueError` naming these settings before activation. + ## LoRA model lineage in model card The new format of `--lora-modules` is mainly to support the display of parent model information in the model card. Here's an explanation of how your current response supports this: diff --git a/docs/features/mooncake_connector_usage.md b/docs/features/mooncake_connector_usage.md index 866d6d6bd4..00bfdeabc1 100644 --- a/docs/features/mooncake_connector_usage.md +++ b/docs/features/mooncake_connector_usage.md @@ -49,12 +49,13 @@ cache), see [ECMooncakeConnector Usage Guide](mooncake_ec_connector_usage.md). - For headless instances, must be the same as the master instance - Each instance needs a unique port on its host; using the same port number across different hosts is fine -- `VLLM_MOONCAKE_CONNECTOR_TIMEOUT`: Per-request timeout (in seconds) for MooncakeConnector HTTP calls to the bootstrap server, including prefiller worker registration at startup. (Optional) +- `VLLM_MOONCAKE_CONNECTOR_TIMEOUT`: Per-request timeout (in seconds) for MooncakeConnector HTTP calls to the bootstrap server, including prefiller worker registration at startup and metadata queries. (Optional) - Default: 30.0 - Global rank 0 hosts the bootstrap server in the same process that mounts the Mooncake transfer engine's host segment, so a large `MOONCAKE_GLOBAL_SEGMENT_SIZE` can delay responses for several seconds during startup - Raise this if bootstrap server timeout warnings appear on hosts with very large host-memory segments - Bootstrap registration uses up to 3 attempts internally, with exponential backoff capped at 10 seconds; connection errors and timeouts are both retried - - Once the attempts are exhausted the caller raises instead of blocking, so failures surface rather than hanging + - Once registration attempts are exhausted the caller raises instead of blocking, so failures surface rather than hanging + - Metadata queries reuse the same timeout and 3-attempt limit, with a fixed 100ms delay between attempts. If all attempts fail, the error is logged and waiting queries are notified - `WITH_NVIDIA_PEERMEM`: Selects how mooncake registers GPU memory for RDMA. Read by mooncake, not vLLM. - Default: 1, which uses `ibv_reg_mr()` and requires the `nvidia-peermem` kernel module to be loaded diff --git a/docs/features/multimodal_inputs.md b/docs/features/multimodal_inputs.md index 40ceb8158b..cb091c1c79 100644 --- a/docs/features/multimodal_inputs.md +++ b/docs/features/multimodal_inputs.md @@ -1020,6 +1020,13 @@ vllm serve Qwen/Qwen3-VL-30B-A3B-Instruct \ When you extract video frames on the client side and send them as `video/jpeg` (base64-concatenated JPEG frames), you can preserve the original video metadata by using `media_io_kwargs` in your request. This enables more accurate video understanding by preserving temporal information that would otherwise be lost during client-side frame extraction. +!!! warning + Per-request `media_io_kwargs` (and `mm_processor_kwargs`) are rejected by + default because they let a client change media loading and preprocessing + resource usage. Start the server with `--trust-request-mm-kwargs` only when + your API clients are trusted; see + [Per-request multimodal arguments](../usage/security.md#per-request-multimodal-arguments). + **Supported Parameters:** | Parameter | Type | Description | @@ -1042,7 +1049,7 @@ When you extract video frames on the client side and send them as `video/jpeg` ( frames_b64 = ",".join([encode_image(f) for f in frames]) video_url = f"data:video/jpeg;base64,{frames_b64}" - # Pass video metadata via media_io_kwargs + # Pass video metadata via media_io_kwargs (server started with --trust-request-mm-kwargs) response = client.chat.completions.create( model="your-multimodal-model", messages=[{ diff --git a/docs/features/quantization/online.md b/docs/features/quantization/online.md index d3a4edd271..a81ea219f7 100644 --- a/docs/features/quantization/online.md +++ b/docs/features/quantization/online.md @@ -99,6 +99,35 @@ vllm serve openai/gpt-oss-20b --quantization-config.moe.activation mxfp8 Combine with `--moe-backend` to pin a specific kernel family. +### Re-quantizing layers to a different precision + +An online target can re-quantize a checkpoint-quantized (prequantized) layer at load time. + +vLLM uses the checkpoint method to load and dequantize the checkpoint weight, and then applies the online quantization method. This requires the checkpoint +method to implement weight dequantization, unsupported source and target +combinations raise an error. + +Currently supported combinations are: + +| Checkpoint layer | Online target | Requirements | +| ---------------- | ------------- | ------------ | +| Serialized ModelOpt MXFP8 linear | any | Target-specific hardware requirements | + +For example, serialized ModelOpt MXFP8 linears can be converted to +per-channel-weight/per-token-activation FP8 at load time: + +```bash +vllm serve MiniMaxAI/MiniMax-M3-MXFP8 \ + --linear-backend auto \ + --quantization-config.linear fp8_per_channel +``` + +!!! info + + This is only available for linear layers at the moment. + + Checkpoint-exclusion policy is not enforced by online quantization/re-quantization. Only exclusion through `--quantization-config.targets`/`--quantization-config.ignore` is enforced. + ### Online quantization on unquantized layers from partially-quantized checkpoints Online quantization can be used on already quantized checkpoints independently of their original `quant_method` (`modelopt`, `compressed-tensors`, `quark`, etc.), for layers that are left unquantized in the original checkpoint. diff --git a/docs/features/sleep_mode.md b/docs/features/sleep_mode.md index a8aa15af7b..0db82de9d3 100644 --- a/docs/features/sleep_mode.md +++ b/docs/features/sleep_mode.md @@ -107,7 +107,7 @@ By default, CUDA graph memory stays on the GPU while asleep. With `sleep_mode_offload_cudagraph=True` (off by default), CUDA graphs are captured into a cuMem pool that sleep backs up to CPU memory at both levels and any wake restores in place, so graphs are replayed, not recaptured. It needs the default -`cumem` backend, CUDA and CUDA graphs; otherwise it has no effect. +`cumem` backend, CUDA or ROCm, and CUDA graphs; otherwise it has no effect. ```python llm = LLM("Qwen/Qwen3-8B", enable_sleep_mode=True, sleep_mode_offload_cudagraph=True) diff --git a/docs/features/tool_calling.md b/docs/features/tool_calling.md index 8dccd3a8b8..3081047dce 100644 --- a/docs/features/tool_calling.md +++ b/docs/features/tool_calling.md @@ -94,6 +94,8 @@ For best results, we recommend ensuring that the expected output format / schema To use a named function, you need to define the functions in the `tools` parameter of the chat completion request, and specify the `name` of one of the tools in the `tool_choice` parameter of the chat completion request. +In the Responses API, `tool_choice={"type": "function", "name": "..."}` can select a function inside a namespace by its full `namespace__function` name or by its local name. Names are matched literally first, so a plain function named `other__add` never stands in for `add`. A local name defined in more than one namespace must be selected by its full name. + ## Required Function Calling vLLM supports the `tool_choice='required'` option in the chat completion API. Similar to the named function calling, it also uses structured outputs, so this is enabled by default and will work with any supported model. However, support for alternative decoding backends are on the [roadmap](../usage/v1_guide.md#features) for the V1 engine. diff --git a/docs/features/watermarking.md b/docs/features/watermarking.md index 7ab02d3363..66b6465eb8 100644 --- a/docs/features/watermarking.md +++ b/docs/features/watermarking.md @@ -18,8 +18,16 @@ vllm serve MODEL \ ``` Watermarking is disabled when `--watermark-config` is omitted. Gumbel-max (see -`gumbel`) is the default algorithm within an enabled `WatermarkConfig`. When -watermarking is configured, it is enabled for requests by default. +`gumbel`) is the default algorithm within an enabled `WatermarkConfig`. + +The request-level `watermarking` option has two effective states: + +- Omitted (`None` in Python) or `true` requests watermarking. +- `false` explicitly opts out. + +When a request asks for watermarking but the engine or generation mode cannot +apply it, vLLM emits a server-side warning once and runs the request without a +watermark. Requests can opt out without changing the engine-level algorithm or key: @@ -30,14 +38,12 @@ sampling_params = SamplingParams(watermarking=False) ``` The Python OpenAI-compatible APIs and the Rust chat/completions APIs accept the -same `watermarking: false` request field. The Rust token API accepts it in -`sampling_params`, and the Rust gRPC `GenerateRequest` accepts -the optional `watermarking` field. Omitting the field keeps watermarking enabled -when the engine has a watermark configuration. +same optional `watermarking` request field. The Rust token API accepts it in +`sampling_params`, and the Rust gRPC `GenerateRequest` accepts it directly. +Omitting the field requests watermarking by default. -Deployments that require watermarking must restrict this field to trusted -callers, or strip and validate it at the ingress boundary, so untrusted clients -cannot opt out. +Deployments that require watermarking must restrict explicit opt-out to trusted +callers, or strip and validate the field at the ingress boundary. `context_width` controls how many prior tokens seed each watermark decision and defaults to 4. Larger values make the watermark less robust to @@ -63,6 +69,31 @@ the reference algorithms. `WatermarkDetector` consumes token IDs, so callers remain responsible for using the tokenizer and watermark profile that match generation. +## Interactions with generation features + +- Greedy decoding (`temperature=0`) cannot apply a Gumbel watermark. The first + such request emits a server-side warning; it and subsequent greedy requests + use ordinary greedy sampling without a watermark. This includes + transcription, translation, and realtime transcription, which default to + greedy decoding. Transcription and translation can use a nonzero temperature + to apply the configured watermark. +- Beam search cannot apply a watermark. The first watermarked beam request emits + a server-side warning and runs without watermarking. +- Trace replay replaces the sampled token and therefore cannot apply a + watermark. The first watermarked trace replay request emits a server-side + warning and runs without watermarking. +- Parallel sampling (`n > 1`) is supported, but plain Gumbel candidates can be + identical while their contexts remain identical. +- Structured outputs and tool grammars are applied before watermark sampling. + Restrictive constraints can reduce the statistical evidence available to a + detector. +- Automatic language detection and generative scoring use unwatermarked + auxiliary generations. Language-detection tokens are not part of the returned + transcription, and generative scoring returns scores rather than generated + text. +- Speculative decoding has additional requirements and partial-watermarking + modes described in [Speculative decoding](#speculative-decoding). + ## Speculative decoding Watermarking requires speculative decoding to use probabilistic draft sampling, @@ -245,7 +276,7 @@ watermarked output or to modify watermarked text so it is no longer detected. - Watermarking is currently available only with Model Runner V2. - Not all watermarking algorithms have native speculative-decoding support. -- Beam search expands candidates from model log probabilities and does not apply - Gumbel-max watermarking. - Models that replace the vLLM sampler with a custom sampler cannot use configured watermarking. +- Global custom logits processors are unavailable because Model Runner V2 does + not support them. diff --git a/docs/getting_started/installation/cpu.md b/docs/getting_started/installation/cpu.md index 20834529b1..1e58a68fcf 100644 --- a/docs/getting_started/installation/cpu.md +++ b/docs/getting_started/installation/cpu.md @@ -30,7 +30,7 @@ When open a Github issue about the CPU backend, please add `[CPU Backend]` in th ## Requirements -- Python: 3.10 -- 3.13 +- Python: 3.11 -- 3.14 === "Intel/AMD x86" @@ -241,6 +241,9 @@ export VLLM_VERSION=$(curl -s https://api.github.com/repos/vllm-project/vllm/rel uv pip install "vllm[zen]" --extra-index-url https://wheels.vllm.ai/${VLLM_VERSION}/cpu --index-strategy first-index --torch-backend cpu ``` +!!! warning + `zentorch` is not supported with vLLM 0.27.0 and 0.27.1. + vLLM auto-detects the platform and routes linear layers through ZenDNN-optimized kernels - no flag needed. To verify it is engaged, look for the platform-selection line in the server's startup logs: ```bash @@ -290,7 +293,7 @@ See [AMD Zen optimizations](#amd-zen-optimizations) for detection rules, support ### How to decide `VLLM_CPU_KVCACHE_SPACE`? -This value is 4GB by default. Larger space can support more concurrent requests, longer context length. However, users should take care of memory capacity of each NUMA node. The memory usage of each TP rank is the sum of `weight shard size` and `VLLM_CPU_KVCACHE_SPACE`, if it exceeds the capacity of a single NUMA node, the TP worker will be killed with `exitcode 9` due to out-of-memory. +This value is 0 by default, which means vLLM doesn't override the KV cache size through this variable. When set, the value is in GiB (e.g, `VLLM_CPU_KVCACHE_SPACE`=40 means 40 GiB). Larger space can support more concurrent requests, longer context length. However, users should take care of memory capacity of each NUMA node. The memory usage of each TP rank is the sum of `weight shard size` and `VLLM_CPU_KVCACHE_SPACE`, if it exceeds the capacity of a single NUMA node, the TP worker will be killed with `exitcode 9` due to out-of-memory. This is a legacy setting; use `--kv-cache-memory-bytes` instead. ### How to do performance tuning for vLLM CPU? diff --git a/docs/getting_started/installation/gpu.cuda.inc.md b/docs/getting_started/installation/gpu.cuda.inc.md index 0d0d0c0c85..f125a189fc 100644 --- a/docs/getting_started/installation/gpu.cuda.inc.md +++ b/docs/getting_started/installation/gpu.cuda.inc.md @@ -117,18 +117,21 @@ If you need to recompile the `vllm-rs` Rust frontend binary, you can rebuild and This will install the required Rust toolchain if needed, build the binary, and place it in `vllm/vllm-rs`. -In case you see an error about wheel not found when running the above command, it might be because the commit you based on in the `main` branch was just merged and its precompiled wheel is not available yet. You can wait around an hour and retry, or set `VLLM_PRECOMPILED_WHEEL_COMMIT=nightly` to automatically select the most recent already-built commit on `main`. +Wheels are published an hour or two after a commit lands on `main`. If the wheel for +your merge-base with upstream `main` is not available yet, the installer uses the +nearest older commit that has one, searching up to 20 commits back. It stops with an +error if compiled sources or build configuration (`csrc/`, `cmake/`, `CMakeLists.txt`, +`pyproject.toml`, `vllm/_custom_ops.py`) changed between that commit and your +merge-base, and warns if `rust/` or `setup.py` did. This is a source-level check, not +a guarantee of binary compatibility. -```bash -export VLLM_PRECOMPILED_WHEEL_COMMIT=nightly -export VLLM_USE_PRECOMPILED=1 -uv pip install --editable . -``` +Setting `VLLM_PRECOMPILED_WHEEL_LOCATION` or a full SHA in +`VLLM_PRECOMPILED_WHEEL_COMMIT` skips this search and check. There are more environment variables to control the behavior of Python-only build: - `VLLM_PRECOMPILED_WHEEL_LOCATION`: specify the exact wheel URL or local file path of a pre-compiled wheel to use. All other logic to find the wheel will be skipped. -- `VLLM_PRECOMPILED_WHEEL_COMMIT`: override the commit hash to download the pre-compiled wheel. It can be `nightly` to use the last **already built** commit on the main branch. +- `VLLM_PRECOMPILED_WHEEL_COMMIT`: override the commit to download the pre-compiled wheel from. Must be a full 40-character SHA. - `VLLM_PRECOMPILED_WHEEL_VARIANT`: specify the variant subdirectory to use on the nightly index, e.g., `cu129`, `cu130`, `cpu`. If not specified, the variant is auto-detected based on your system's CUDA version (from PyTorch or nvidia-smi). You can also set `VLLM_MAIN_CUDA_VERSION` to override auto-detection. You can find more information about vLLM's wheels in [Install the latest code](#install-the-latest-code). @@ -140,13 +143,8 @@ You can find more information about vLLM's wheels in [Install the latest code](# #### Full build (with compilation) {#full-build} !!! note "Compiler requirement" - Building from source requires GCC/G++ ≥ 11.3. PyTorch's C++20 headers are - not compatible with GCC 10 or GCC < 11.3. On Ubuntu 22.04: - ```bash - sudo apt-get install -y gcc-11 g++-11 - sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-11 110 \ - --slave /usr/bin/g++ g++ /usr/bin/g++-11 - ``` + Building from source requires GCC/G++ ≥ 13 ([#58158](https://github.com/vllm-project/vllm/issues/58158)). + Ubuntu 24.04, the base of the default vLLM image, ships GCC 13 by default. If you want to modify C++ or CUDA code, you'll need to build vLLM from source. This can take several minutes: @@ -439,6 +437,11 @@ BuildKit does not automatically invalidate cached layers when a mutable Git ref changes. Use `--no-cache-filter extensions-build` to refresh an empty, branch, or tag revision. +Set `BUILD_NIXL=true` to build NIXL from source. NIXL's release wheels do not +include the NIXL EP extension for the PyTorch nightly used by the Rubin build. +The build uses the NIXL version pinned in `requirements/kv_connectors.txt` and +replaces the NIXL packages installed from the KV-connector requirements. + For `FINAL_BASE_IMAGE`, use the public, multi-arch `nvidia/cuda:13.4.1-base-ubuntu24.04` image. Set `NCCL_VERSION` to 2.32.3 or newer, the first NCCL release with Rubin (SM107) support. diff --git a/docs/getting_started/installation/gpu.md b/docs/getting_started/installation/gpu.md index 91d933dd4e..fc8d9148c5 100644 --- a/docs/getting_started/installation/gpu.md +++ b/docs/getting_started/installation/gpu.md @@ -25,7 +25,7 @@ vLLM is a Python library that supports the following GPU variants. Select your G ## Requirements - OS: Linux -- Python: 3.10 -- 3.13 +- Python: 3.11 -- 3.14 !!! note vLLM does not support Windows natively. To run vLLM on Windows, you can use the Windows Subsystem for Linux (WSL) with a compatible Linux distribution, or use some community-maintained forks, e.g. [https://github.com/SystemPanic/vllm-windows](https://github.com/SystemPanic/vllm-windows). diff --git a/docs/getting_started/quickstart.md b/docs/getting_started/quickstart.md index 6f17094a9f..afa9713341 100644 --- a/docs/getting_started/quickstart.md +++ b/docs/getting_started/quickstart.md @@ -8,7 +8,7 @@ This guide will help you quickly get started with vLLM to perform: ## Prerequisites - OS: Linux -- Python: 3.10 -- 3.13 +- Python: 3.11 -- 3.14 !!! note vLLM also works on macOS with [vLLM-Metal](https://github.com/vllm-project/vllm-metal) for Apple Silicon GPU acceleration. See the [GPU installation guide](installation/gpu.md) and select the "Apple Silicon" tab. diff --git a/docs/mkdocs/gen_files/generate_attention_backends.py b/docs/mkdocs/gen_files/generate_attention_backends.py index 725d7072ca..135dcf3a35 100644 --- a/docs/mkdocs/gen_files/generate_attention_backends.py +++ b/docs/mkdocs/gen_files/generate_attention_backends.py @@ -51,7 +51,7 @@ BACKEND_KV_DTYPE_EXCLUDES: dict[str, set[str]] = { MLA_PREFILL_DIR = BACKENDS_DIR / "mla" / "prefill" MLA_PREFILL_REGISTRY_FILE = MLA_PREFILL_DIR / "registry.py" -MLA_PREFILL_SELECTOR_FILE = MLA_PREFILL_DIR / "selector.py" +PLATFORM_INTERFACE_FILE = REPO_ROOT / "vllm" / "platforms" / "interface.py" # --------------------------------------------------------------------------- @@ -320,67 +320,83 @@ def parse_mla_prefill_registry() -> dict[str, str]: def parse_mla_prefill_priorities() -> dict[str, list[str]]: - """Parse MLA prefill backend priorities from selector.py. + """Parse MLA prefill backend priorities from platforms/interface.py. Returns: A dict with keys like 'blackwell' and 'default' containing lists of backend enum names in priority order. """ - if not MLA_PREFILL_SELECTOR_FILE.exists(): + if not PLATFORM_INTERFACE_FILE.exists(): return {} try: - tree = ast.parse(MLA_PREFILL_SELECTOR_FILE.read_text()) + tree = ast.parse(PLATFORM_INTERFACE_FILE.read_text()) except Exception: return {} + def extract_backends(stmt: ast.AST) -> list[str] | None: + """Read a backend enum list from a return/assignment statement.""" + if isinstance(stmt, ast.Return) or ( + isinstance(stmt, ast.Assign) + and len(stmt.targets) == 1 + and isinstance(stmt.targets[0], ast.Name) + ): + value = stmt.value + else: + return None + if not isinstance(value, ast.List): + return None + return [elt.attr for elt in value.elts if isinstance(elt, ast.Attribute)] + + def extract_last_list(nodes: list[ast.stmt], recurse: bool) -> list[str] | None: + """Return the last backend list found among the given statements.""" + result = None + for stmt in nodes: + backends = extract_backends(stmt) + if backends is not None: + result = backends + elif recurse and isinstance(stmt, ast.If): + nested = extract_last_list([*stmt.body, *stmt.orelse], True) + if nested is not None: + result = nested + return result + + def is_blackwell_check(stmt: ast.AST) -> bool: + return ( + isinstance(stmt, ast.If) + and isinstance(stmt.test, ast.Compare) + and isinstance(stmt.test.left, ast.Attribute) + and stmt.test.left.attr == "major" + and stmt.test.comparators + and isinstance(stmt.test.comparators[0], ast.Constant) + and stmt.test.comparators[0].value == 10 + ) + priorities: dict[str, list[str]] = {} for node in ast.walk(tree): - if not isinstance(node, ast.FunctionDef): + if not isinstance(node, ast.ClassDef) or node.name != "Platform": continue - if node.name != "_get_mla_prefill_backend_priorities": + + method = find_method(node, "get_mla_prefill_backend_cls") + if method is None: continue # Look for if statements checking device_capability.major - for stmt in ast.walk(node): + for stmt in ast.walk(method): if not isinstance(stmt, ast.If): continue - # Check if it's a capability.major == 10 check (Blackwell) - is_blackwell = ( - isinstance(stmt.test, ast.Compare) - and isinstance(stmt.test.left, ast.Attribute) - and stmt.test.left.attr == "major" - and stmt.test.comparators - and isinstance(stmt.test.comparators[0], ast.Constant) - and stmt.test.comparators[0].value == 10 - ) - - # Extract backends from return statements - for body_stmt in stmt.body: - if isinstance(body_stmt, ast.Return) and isinstance( - body_stmt.value, ast.List - ): - backends = [] - for elt in body_stmt.value.elts: - if isinstance(elt, ast.Attribute): - backends.append(elt.attr) - if is_blackwell: - priorities["blackwell"] = backends - else: - priorities["default"] = backends - - # Extract from else branch - for else_stmt in stmt.orelse: - if isinstance(else_stmt, ast.Return) and isinstance( - else_stmt.value, ast.List - ): - backends = [] - for elt in else_stmt.value.elts: - if isinstance(elt, ast.Attribute): - backends.append(elt.attr) + if is_blackwell_check(stmt): + # The Blackwell lists live in the nested if/else inside the + # major == 10 branch. + backends = extract_last_list(stmt.body, True) + if backends is not None: + priorities["blackwell"] = backends + else: + backends = extract_last_list([*stmt.body, *stmt.orelse], False) + if backends is not None: priorities["default"] = backends return priorities diff --git a/docs/mkdocs/gen_files/generate_metrics.py b/docs/mkdocs/gen_files/generate_metrics.py index 4d4fa93cc5..32bbefa88c 100644 --- a/docs/mkdocs/gen_files/generate_metrics.py +++ b/docs/mkdocs/gen_files/generate_metrics.py @@ -23,6 +23,10 @@ METRIC_SOURCE_FILES = [ "path": "vllm/distributed/kv_transfer/kv_connector/v1/nixl/stats.py", "key": "metrics-nixl", }, + { + "path": "vllm/distributed/kv_transfer/kv_connector/v1/hisparse/stats.py", + "key": "metrics-hisparse", + }, { "path": "vllm/v1/simple_kv_offload/metrics.py", "key": "metrics-simple-kv-offload", diff --git a/docs/models/extensions/tensorizer.md b/docs/models/extensions/tensorizer.md index 0f3ee1ede4..1e0d5da81e 100644 --- a/docs/models/extensions/tensorizer.md +++ b/docs/models/extensions/tensorizer.md @@ -32,25 +32,15 @@ python examples/features/tensorize_vllm_model.py \ --suffix v1 ``` -This saves the model tensors at `s3://my-bucket/vllm/facebook/opt-125m/v1`. If you intend on applying a LoRA adapter to your tensorized model, you can pass the HF id of the LoRA adapter in the above command, and the artifacts will be saved there too: - -```bash -python examples/features/tensorize_vllm_model.py \ - --model facebook/opt-125m \ - --lora-path \ - serialize \ - --serialized-directory s3://my-bucket \ - --suffix v1 -``` +This saves the model tensors at `s3://my-bucket/vllm/facebook/opt-125m/v1`. ## Serving the model using Tensorizer -Once the model is serialized where you want it, you can load the model using `vllm serve` or the `LLM` entrypoint. You can pass the directory where you saved the model to the `model` argument for `LLM()` and `vllm serve`. For example, to serve the tensorized model saved previously with the LoRA adapter, you'd do: +Once the model is serialized where you want it, you can load the model using `vllm serve` or the `LLM` entrypoint. You can pass the directory where you saved the model to the `model` argument for `LLM()` and `vllm serve`. For example, to serve the tensorized model saved previously, you'd do: ```bash vllm serve s3://my-bucket/vllm/facebook/opt-125m/v1 \ - --load-format tensorizer \ - --enable-lora + --load-format tensorizer ``` Or, with `LLM()`: @@ -60,7 +50,6 @@ from vllm import LLM llm = LLM( "s3://my-bucket/vllm/facebook/opt-125m/v1", load_format="tensorizer", - enable_lora=True, ) ``` @@ -73,7 +62,6 @@ As an example, CPU concurrency can be limited when serializing with `tensorizer` ```bash python examples/features/tensorize_vllm_model.py \ --model facebook/opt-125m \ - --lora-path \ serialize \ --serialized-directory s3://my-bucket \ --serialization-kwargs '{"limit_cpu_concurrency": 2}' \ @@ -85,7 +73,6 @@ As an example when customizing the loading process via `TensorDeserializer`, you ```bash vllm serve s3://my-bucket/vllm/facebook/opt-125m/v1 \ --load-format tensorizer \ - --enable-lora \ --model-loader-extra-config '{"deserialization_kwargs": {"num_readers": 2}}' ``` @@ -96,7 +83,6 @@ from vllm import LLM llm = LLM( "s3://my-bucket/vllm/facebook/opt-125m/v1", load_format="tensorizer", - enable_lora=True, model_loader_extra_config={"deserialization_kwargs": {"num_readers": 2}}, ) ``` diff --git a/docs/models/pooling_models/embed.md b/docs/models/pooling_models/embed.md index 13a2e0d381..4c33126977 100644 --- a/docs/models/pooling_models/embed.md +++ b/docs/models/pooling_models/embed.md @@ -50,7 +50,7 @@ You can compute pairwise similarity scores to build a similarity matrix using th | `ModernBertModel` | ModernBERT-based | `Alibaba-NLP/gte-modernbert-base`, etc. | ✅︎ | | | `NomicBertModel` | Nomic BERT | `nomic-ai/nomic-embed-text-v1`, `nomic-ai/nomic-embed-text-v2-moe`, `Snowflake/snowflake-arctic-embed-m-long`, etc. | | | | `Qwen2Model`C, `Qwen2ForCausalLM`C | Qwen2-based | `ssmits/Qwen2-7B-Instruct-embed-base` (see note), `Alibaba-NLP/gte-Qwen2-7B-instruct` (see note), etc. | ✅︎ | ✅︎ | -| `Qwen3Model`C, `Qwen3ForCausalLM`C | Qwen3-based | `Qwen/Qwen3-Embedding-0.6B`, etc. | ✅︎ | ✅︎ | +| `Qwen3Model`C, `Qwen3ForCausalLM`C | Qwen3-based | `Qwen/Qwen3-Embedding-0.6B`, `ai-sage/Giga-Embeddings-instruct-480M-0826`^, etc. | ✅︎ | ✅︎ | | `RobertaModel`, `RobertaForMaskedLM` | RoBERTa-based | `sentence-transformers/all-roberta-large-v1`, etc. | | | | `VoyageQwen3BidirectionalEmbedModel`C | Voyage Qwen3-based with bidirectional attention | `voyageai/voyage-4-nano`, etc. | ✅︎ | ✅︎ | | `XLMRobertaModel` | XLMRobertaModel-based | `BAAI/bge-m3` (see note), `intfloat/multilingual-e5-base`, `jinaai/jina-embeddings-v3` (see note), etc. | | | @@ -97,6 +97,7 @@ You can compute pairwise similarity scores to build a similarity matrix using th | `*ForConditionalGeneration`C, `*ForCausalLM`C, etc. | Generative models | \* | N/A | \* | \* | C Automatically converted into an embedding model via `--convert embed`. ([details](./README.md#model-conversion)) +^ You need to set the architecture name via `--hf-overrides` to match the one in vLLM. \* Feature support is the same as that of the original model. If your model is not in the above list, we will try to automatically convert the model using diff --git a/docs/models/pooling_models/scoring.md b/docs/models/pooling_models/scoring.md index 1c33c6852d..0198810cbf 100644 --- a/docs/models/pooling_models/scoring.md +++ b/docs/models/pooling_models/scoring.md @@ -49,7 +49,7 @@ The three supported scoring functions are as illustrated in the figure below. | `LlamaBidirectionalForSequenceClassification`C | Llama-based with bidirectional attention | `nvidia/llama-nemotron-rerank-1b-v2`, etc. | [nemotron-rerank.jinja](../../../examples/pooling/score/template/nemotron-rerank.jinja) | ✅︎ | ✅︎ | | `ModernBertForSequenceClassification` | ModernBERT-based | `Alibaba-NLP/gte-reranker-modernbert-base`, etc. | N/A | | | | `Qwen2ForSequenceClassification`C | Qwen2-based | `mixedbread-ai/mxbai-rerank-base-v2`(see note), etc. | [mxbai_rerank_v2.jinja](../../../examples/pooling/score/template/mxbai_rerank_v2.jinja) | ✅︎ | ✅︎ | -| `Qwen3ForSequenceClassification`C | Qwen3-based | `tomaarsen/Qwen3-Reranker-0.6B-seq-cls`, `Qwen/Qwen3-Reranker-0.6B`(see note), etc. | [qwen3_reranker.jinja](../../../examples/pooling/score/template/qwen3_reranker.jinja) | ✅︎ | ✅︎ | +| `Qwen3ForSequenceClassification`C | Qwen3-based | `tomaarsen/Qwen3-Reranker-0.6B-seq-cls`, `Qwen/Qwen3-Reranker-0.6B`(see note), `polaria-tech/zerank-2-reranker-vllm`(see note), etc. | [qwen3_reranker.jinja](../../../examples/pooling/score/template/qwen3_reranker.jinja), [zerank2.jinja](../../../examples/pooling/score/template/zerank2.jinja) | ✅︎ | ✅︎ | | `RobertaForSequenceClassification` | RoBERTa-based | `cross-encoder/quora-roberta-base`, etc. | N/A | ✅︎ | | | `XLMRobertaForSequenceClassification` | XLM-RoBERTa-based | `BAAI/bge-reranker-v2-m3`, etc. | N/A | ✅︎ | | | `*Model`C, `*ForCausalLM`C, etc. | Generative models | N/A | N/A | \* | \* | @@ -88,6 +88,13 @@ The three supported scoring functions are as illustrated in the figure below. vllm serve Qwen/Qwen3-Reranker-0.6B --hf_overrides '{"architectures": ["Qwen3ForSequenceClassification"],"classifier_from_token": ["no", "yes"],"is_original_qwen3_reranker": true}' ``` +!!! note + `polaria-tech/zerank-2-reranker-vllm` is [zeroentropy/zerank-2-reranker](https://huggingface.co/zeroentropy/zerank-2-reranker) converted to `Qwen3ForSequenceClassification`; it returns zerank-2's calibrated score `sigmoid(logit / 5)`. Serve it with [zerank2.jinja](../../../examples/pooling/score/template/zerank2.jinja), which also supports the `instruction` request field. + + ```bash + vllm serve polaria-tech/zerank-2-reranker-vllm --chat-template examples/pooling/score/template/zerank2.jinja + ``` + #### Multimodal Models !!! note diff --git a/docs/models/supported_models.md b/docs/models/supported_models.md index a55559b0d6..221ab877ea 100644 --- a/docs/models/supported_models.md +++ b/docs/models/supported_models.md @@ -199,6 +199,7 @@ Some model architectures are supported via vLLM plugins. These plugins extend vL | ------------ | ------ | ----------------- | | `BartForConditionalGeneration` | BART | [bart-plugin](https://github.com/vllm-project/bart-plugin) | | `Florence2ForConditionalGeneration` | Florence-2 | [bart-plugin](https://github.com/vllm-project/bart-plugin) | +| `Nemotron3_5AsrForRNNT` | NVIDIA Nemotron 3.5 ASR | [vllm-nemotron-asr-plugin](https://github.com/Sohaib-Ahmed21/vllm-nemotron-asr-plugin) | For other model architectures not natively supported, in particular for Encoder-Decoder models, we recommend following a similar pattern by implementing support through the plugin system. diff --git a/docs/serving/online_serving/README.md b/docs/serving/online_serving/README.md index 8e1a4f3398..fc46a482c1 100644 --- a/docs/serving/online_serving/README.md +++ b/docs/serving/online_serving/README.md @@ -76,6 +76,9 @@ For further details on speech to text, please refer to [this page](speech_to_tex - [Generative Scoring API](generative_scoring.md) (`/generative_scoring`) - Applicable to [CausalLM models](../../models/generative_models.md) (task `"generate"`). - Computes next-token probabilities for specified `label_token_ids`. +- [Structured Decisions API](structured_decisions.md) (`/v1/systemone`) + - Applicable to [text generation models](../../models/generative_models.md) with a [chat template](#chat-template). + - Answers typed questions about a state with a probability for every allowed answer. ## Instrumentator APIs diff --git a/docs/serving/online_serving/derenderer.md b/docs/serving/online_serving/derenderer.md index 9585031312..f85433a963 100644 --- a/docs/serving/online_serving/derenderer.md +++ b/docs/serving/online_serving/derenderer.md @@ -105,7 +105,7 @@ The server keeps no state between calls. Everything the next call needs is in `s - **Send `prompt_token_ids` on the first chunk, even without a parser.** It seeds detokenization so the first token's leading space matches the coupled endpoint (see [Request format](#request-format)). Later chunks already carry that context in `stream_state`. - **Don't forward `[DONE]`.** It marks the end of the generate stream and isn't a chunk. -Streaming derender resolves `logprobs` on the plain detokenization path, including `token_id:N` placeholders, same as the non-streaming endpoints (see [Streaming state and logprobs](#streaming-state-and-logprobs)). The parser path doesn't resolve them yet and drops them. +`/inference/v1/generate` returns `GenerateLogProbs` with integer `token_id`s (the generate server has no tokenizer), and derender turns those into `ChatCompletionLogProbs` / `CompletionLogProbs`, filling `token` and `bytes` from the tokenizer with the usual U+FFFD byte-fallback correction. Streaming derender does the same per chunk on the plain detokenization path (see [Streaming state and logprobs](#streaming-state-and-logprobs)). The parser path doesn't resolve them yet and drops them. ## Streaming cost @@ -129,7 +129,7 @@ Streaming derender is stateless on the server side: all mutable state lives in t - `logprob_context_token_ids`: the trailing sampled token IDs (at most 4) from previous chunks, used to seed byte-fallback (U+FFFD) correction so multi-byte characters whose tokens split across chunk boundaries still resolve to real strings - `logprob_text_offset`: the cumulative emitted text length, so `text_offset` in completion streaming logprobs stays absolute across chunks instead of restarting at 0 -When a streamed `GenerateTokensStreamChoice` carries `logprobs`, the `token_id:N` placeholders are resolved per chunk and the resolved logprobs are attached to the corresponding streamed choice. For chat it is as `ChatCompletionLogProbs` and for completions it is converted to the flat `CompletionLogProbs` lists. Chunks without `logprobs` produce choices with `logprobs: null`. +When a streamed `GenerateTokensStreamChoice` carries `logprobs`, the integer `token_id`s are decoded per chunk and the resolved logprobs are attached to the corresponding streamed choice. For chat it is as `ChatCompletionLogProbs` and for completions it is converted to the flat `CompletionLogProbs` lists. Chunks without `logprobs` produce choices with `logprobs: null`. ## Example diff --git a/docs/serving/online_serving/generative_scoring.md b/docs/serving/online_serving/generative_scoring.md index d0c578f210..eebe1f965e 100644 --- a/docs/serving/online_serving/generative_scoring.md +++ b/docs/serving/online_serving/generative_scoring.md @@ -62,3 +62,10 @@ Here, each item is appended to the query to form prompts like `"Is this city the "usage": {"prompt_tokens": 45, "total_tokens": 48, "completion_tokens": 3} } ``` + +## Cache salting + +The optional `cache_salt` request field is applied to every item's prompt. With +prefix caching enabled, requests with different salts do not share cached prefix +blocks. See [Cache Salting](../../usage/security.md#cache-salting) for how to choose +and manage salts. diff --git a/docs/serving/online_serving/openai_compatible_server.md b/docs/serving/online_serving/openai_compatible_server.md index c50910040d..7b4e84a17f 100644 --- a/docs/serving/online_serving/openai_compatible_server.md +++ b/docs/serving/online_serving/openai_compatible_server.md @@ -204,6 +204,12 @@ To get prompt token IDs before generation, use a self-contained Responses request without inference and returns the prompt token IDs, so a separate `/tokenize` call is not needed. +#### Sampling parameters + +In addition to the OpenAI request fields, the Responses API forwards vLLM +[sampling parameters](../../api/README.md#inference-parameters) such as `top_k`, +`min_p`, `repetition_penalty`, `seed`, `stop`, and `ignore_eos` to the engine. + #### Extra parameters The following extra parameters in the request object are supported: diff --git a/docs/serving/online_serving/renderer.md b/docs/serving/online_serving/renderer.md index fc6579c258..93ecf34f4f 100644 --- a/docs/serving/online_serving/renderer.md +++ b/docs/serving/online_serving/renderer.md @@ -79,6 +79,60 @@ for the existing authentication boundaries. For the post processing counterpart that turns generated token IDs back into OpenAI compatible responses, see the [Derenderer APIs](derenderer.md). +## Generate Output Logprobs + +`/inference/v1/generate` is a token in / token out API, so its output logprobs +identify tokens by integer ID rather than by the OpenAI string token. With +`sampling_params.logprobs` set and `output_mode: "tokens"` (the default), each +choice carries a `GenerateLogProbs`; `output_mode: "text"` returns decoded +`ChatCompletionLogProbs` instead (see [token in, token out](token_in_token_out.md)): + +```json +{ + "logprobs": { + "content": [ + { + "token_id": 262, + "logprob": -0.10, + "rank": 1, + "top_logprobs": [ + {"token_id": 262, "logprob": -0.10, "rank": 1}, + {"token_id": 257, "logprob": -1.20, "rank": 2} + ] + } + ] + } +} +``` + +- `content` has one entry per generated token, in generation order. +- `top_logprobs` is a list, not a dict: JSON turns dict keys into strings and + the ordering would be implicit. It follows the engine's order: the sampled + token first, then the remaining candidates in rank order. With non-greedy + sampling the sampled token can sit outside the top k (for example ranks + `[5, 1, 2]` at `logprobs=2`); it then takes one of the `logprobs` slots and + the rank-k candidate is left out, as on the OpenAI endpoints. +- `rank` is the token's rank in the vocabulary distribution (1 = most likely) + on every entry, the sampled one included; a top-k candidate's rank is its + top-k position. The list is not sorted by it, so sort by `rank` if you need + rank order. It is `null` when the engine could not rank the token (a NaN + logprob, which is sent as `-9999.0`). +- There is no `token` or `bytes` field. The generate server has no tokenizer; + [derender](derenderer.md) fills those in when it converts the response to the + OpenAI shapes. +- `prompt_logprobs` on the same response is unchanged + (`list[dict[int, Logprob] | None]`). + +!!! warning "Changed in this release" + Output logprobs used to be `ChatCompletionLogProbs` with every token written + as a `"token_id:N"` placeholder string, and `bytes` set to the UTF-8 bytes of + that placeholder by the Rust frontend but left unset by the Python one. + Clients that read only `content[i].logprob` are unaffected. Clients that + parsed the placeholder should read `content[i].token_id` instead. + `return_tokens_as_token_ids` on `/v1/chat/completions` and `/v1/completions` + is unchanged: it is a user-facing OpenAI option and still uses the + `token_id:N` format. + ## Multimodal Render Features Multimodal render responses include a `features` object with per-modality @@ -99,6 +153,13 @@ should split those fields: - Legacy clients that ignore `mm_metadata` and keep sending `kwargs_data` continue to work. +Callers that need only the token layout and item hashes, such as cache-aware +routers, can set `"return_mm_kwargs": false` on the render request. The response +then keeps `mm_hashes` and `mm_placeholders` and sets `kwargs_data` and +`mm_metadata` to null, so the processed tensors are neither serialized nor sent. +Do not forward such a response to `/inference/v1/generate`, which reads a null +`kwargs_data` as every item being cached. + ## Example The example below shows how a disaggregated encode / prefill coordinator can diff --git a/docs/serving/online_serving/structured_decisions.md b/docs/serving/online_serving/structured_decisions.md new file mode 100644 index 0000000000..a1f58a3769 --- /dev/null +++ b/docs/serving/online_serving/structured_decisions.md @@ -0,0 +1,145 @@ +# Structured Decisions + +The `/v1/systemone` endpoint answers a set of typed questions about a state and +returns a probability for every allowed answer. + +The endpoint is available by default on generation models. Models without a +supported read strategy return 501. Supported architectures require +`--logprobs-mode raw_logprobs` or `processed_logprobs`, since `label_mass` needs +log probabilities. + +## How it works + +1. Each question is one read. The user message is the state, followed by the + question and its options with the type's labels, and an instruction to + answer with one label. +2. The read is the reply's first token. Each label must be one token there, + or the request returns 400. +3. The read returns the logprobs of that question's label tokens. Softmax over + the labels gives the answer's probabilities. + +Thinking is off unless `chat_template_kwargs` turns it on. + +This endpoint does not serve diffusion models yet. + +Every read of a request starts with the state, so with +`--enable-prefix-caching` the state is prefilled once and each further +question prefills only its own text, then reads one token. + +## Question types + +| type | criteria | answer | +| --- | --- | --- | +| `choice` | map of option name to a description or `null` | `choice`, `probabilities` by option name, `confidence` | +| `noul` | `null`, or an object with optional `true` and `false` descriptions | `noul` (probability of yes), `probabilities`, `confidence` | +| `score` | ordered list of level names | `score` (expected 0-indexed level), `legend`, `probabilities` by level index, `confidence` | + +A choice has at least one option, a score at least two levels. A question id +is a non-empty string. + +## Example + +```bash +curl -s http://localhost:8000/v1/systemone \ + -H "Content-Type: application/json" \ + -d '{ + "model": "Qwen/Qwen3-0.6B", + "state": {"ticket": "My card was charged twice for one order."}, + "questions": { + "team": { + "type": "choice", + "instructions": "Which team should handle this ticket?", + "criteria": { + "billing": "payments and refunds", + "shipping": "deliveries", + "security": "account access" + } + }, + "urgent": { + "type": "noul", + "instructions": "Does the customer need a reply within the hour?", + "criteria": {"true": "the account is locked or money is moving"} + }, + "tone": { + "type": "score", + "instructions": "How upset is the customer?", + "criteria": ["calm", "annoyed", "furious"] + } + }, + "chat_template_kwargs": {"enable_thinking": false} + }' +``` + +Response, with probabilities that depend on the model: + +```json +{ + "id": "decision-...", + "object": "structured_decision", + "created": 1790000000, + "model": "Qwen/Qwen3-0.6B", + "answers": { + "team": { + "type": "choice", + "choice": "billing", + "probabilities": {"billing": 0.97, "shipping": 0.01, "security": 0.02}, + "confidence": 0.90 + }, + "urgent": { + "type": "noul", + "noul": 0.02, + "probabilities": {"yes": 0.02, "no": 0.98}, + "confidence": 0.95 + }, + "tone": { + "type": "score", + "score": 0.6, + "legend": {"0": "calm", "1": "annoyed", "2": "furious"}, + "probabilities": {"0": 0.7, "1": 0.2, "2": 0.1}, + "confidence": 0.65 + } + }, + "usage": {"input_tokens": 142, "output_tokens": 3}, + "diagnostics": { + "team": {"label_mass": 0.93, "argmax_is_label": true}, + "urgent": {"label_mass": 0.98, "argmax_is_label": true}, + "tone": {"label_mass": 0.91, "argmax_is_label": true} + } +} +``` + +`diagnostics.label_mass` is the probability the model put on the labels across +its whole vocabulary. A low value means most of the probability went to tokens +that are not labels. `confidence` is the chosen option's probability over the +whole vocabulary: its share of the labels times `label_mass`. + +## Labels + +Labels follow the type: a `choice` labels its options `A` to `Z` in the order +the request lists them, a `noul` labels its options `yes` and `no`, and a +`score` labels its levels `0` to `9`, so a level's label is its score. + +## Limits + +| limit | value | +| --- | --- | +| questions per request | 64 | +| options per `choice` | 26 | +| levels per `score` | 10 | + +## Request fields + +| field | meaning | +| --- | --- | +| `state` | what the questions are about: a string, or JSON that is sent as its JSON text | +| `questions` | question id to `{type, instructions, criteria}`, asked in this order | +| `instructions` | optional context placed ahead of the questions | +| `chat_template_kwargs` | passed to the chat template, for example `{"enable_thinking": false}` | +| `cache_salt` | optional [prefix-cache salt](../../usage/security.md#cache-salting) applied to every question | + +A question with any field other than `type`, `instructions` and `criteria` is +rejected with a 400. + +When [tracing](../../../examples/observability/opentelemetry/README.md) is enabled +with `--otlp-traces-endpoint`, the HTTP `traceparent` and `tracestate` headers are +forwarded to every question's generation request. diff --git a/docs/serving/online_serving/token_in_token_out.md b/docs/serving/online_serving/token_in_token_out.md index b4e0d465bd..e22d490615 100644 --- a/docs/serving/online_serving/token_in_token_out.md +++ b/docs/serving/online_serving/token_in_token_out.md @@ -43,7 +43,7 @@ print(response["choices"][0]["token_ids"]) ### Logprobs -With `output_mode: "tokens"`, logprob entries carry `token_id:N` placeholders. With `output_mode: "text"`, they carry the decoded token strings and their UTF-8 `bytes` for the sampled token and every entry in `top_logprobs`. A server started with `--return-tokens-as-token-ids` always returns the placeholders, as `/v1/completions` does. +With `output_mode: "tokens"`, logprobs are `GenerateLogProbs` with an integer `token_id` and `rank` per entry and no `token` or `bytes` (see [Generate Output Logprobs](renderer.md#generate-output-logprobs)). With `output_mode: "text"`, they are `ChatCompletionLogProbs` carrying the decoded token strings and their UTF-8 `bytes` for the sampled token and every entry in `top_logprobs`; a server started with `--return-tokens-as-token-ids` returns `token_id:N` placeholders there instead, as `/v1/completions` does. ### Errors diff --git a/docs/usage/metrics.md b/docs/usage/metrics.md index 5cb5eebc55..b87ff1c28f 100644 --- a/docs/usage/metrics.md +++ b/docs/usage/metrics.md @@ -45,6 +45,15 @@ The following metrics are exposed: --8<-- "gen:metrics-nixl" +## HiSparse KV Connector Metrics + +These metrics are exposed when the `HiSparseConnector` KV connector is +configured. The host-pool gauges are reported by the scheduler once per engine +step, for the host pool shared by all tensor-parallel ranks; evictable cached +blocks count as free. + +--8<-- "gen:metrics-hisparse" + ## Simple CPU Offload Connector Metrics These metrics are exposed when the `SimpleCPUOffloadConnector` KV connector diff --git a/docs/usage/reproducibility.md b/docs/usage/reproducibility.md index 680791bbe2..ef34855ef4 100644 --- a/docs/usage/reproducibility.md +++ b/docs/usage/reproducibility.md @@ -7,7 +7,7 @@ reproducible results: or enable [batch invariance](../features/batch_invariance.md) to make the outputs insensitive to scheduling. - In online mode, you can only enable [batch invariance](../features/batch_invariance.md). -Example: [examples/features/batch_invariance/reproducibility_offline.py](../../examples/features/batch_invariance/reproducibility_offline.py) +Example: [examples/rl/batch_invariance/reproducibility_offline.py](../../examples/rl/batch_invariance/reproducibility_offline.py) !!! warning diff --git a/docs/usage/security.md b/docs/usage/security.md index d350b6bdfd..ab5aa96bec 100644 --- a/docs/usage/security.md +++ b/docs/usage/security.md @@ -175,6 +175,7 @@ When `--api-key` is configured, the following endpoints require Bearer token aut - `/v1/responses/{response_id}/cancel` - Cancel a response - `/v1/score` - Scoring API - `/v1/rerank` - Reranking API +- `/v1/systemone` - Structured decisions API - `/v1/load_lora_adapter` - Load a LoRA adapter (can alter model behavior; only available when `--enable-lora` is set and `VLLM_ALLOW_RUNTIME_LORA_UPDATING=True`) - `/v1/unload_lora_adapter` - Unload a LoRA adapter (can alter model behavior; only available when `--enable-lora` is set and `VLLM_ALLOW_RUNTIME_LORA_UPDATING=True`) - `/inference/v1/generate` - Generate completions (available when `--enable-scale-out` is set, or with `--tokens-only`) @@ -550,6 +551,15 @@ ensure that only trusted principals can submit work to the cluster: - Place the Ray cluster on an isolated network segment. - Do not expose the Ray client port or dashboard to untrusted networks. +## Multi-Tenant Deployments + +vLLM does not provide isolation between tenants that share the same server process. Requests from different callers are scheduled together and share the same caches. Options such as `cache_salt` reduce specific cross-tenant risks, but they are not an isolation boundary. + +If tenants must be isolated from each other, enforce it in the deployment architecture: + +- Run a dedicated vLLM instance per tenant. +- Place a gateway in front of vLLM that authenticates callers and scopes client-supplied cache identifiers, such as `cache_salt` and multimodal `uuid` values, per tenant. + ## Prefix Cache Timing Side-Channel Mitigation (Cache Salting) ### Background @@ -562,7 +572,7 @@ An attacker sharing the same backend can measure differences in Time to First To vLLM accepts an optional `cache_salt` parameter on requests. The salt is mixed into the hash of the first KV cache block, so only requests carrying the same salt can share cached prefix blocks. See [Automatic Prefix Caching](../design/prefix_caching.md) for the implementation details. -`cache_salt` is accepted by the OpenAI-compatible chat completions, completions, responses, and pooling (embeddings, classification, scoring) endpoints, and by the Anthropic `/v1/messages` endpoint. +`cache_salt` is accepted by the OpenAI-compatible chat completions, completions, responses, and pooling (embeddings, classification, scoring) endpoints, the Anthropic `/v1/messages` endpoint, the structured decisions `/v1/systemone` endpoint, and the generative scoring `/generative_scoring` endpoint. #### Usage with the OpenAI Python client @@ -633,7 +643,7 @@ For additional cross-tenant isolation, set `cache_salt` on each request (see [Pr - **Multi-tenant deployments**: Always generate cryptographically random UUIDs per media item. Additionally, set `cache_salt` to a per-tenant secret for defense in depth. - **Single-tenant deployments**: Ensure UUIDs are unique per distinct media content. `cache_salt` is unnecessary when there is no cross-tenant threat. -- **Default behavior**: Omitting `uuid` entirely preserves the default content-hash-based identity, which is safe against this class of collision but requires hashing the media bytes on every request. +- **Default behavior**: Omitting `uuid` uses a hash of the media bytes as the cache identity, which requires hashing the media on every request. This avoids accidental collisions between callers, but it does not isolate tenants from each other. See [Multi-Tenant Deployments](#multi-tenant-deployments). ## Reporting Security Vulnerabilities diff --git a/docs/usage/v1_guide.md b/docs/usage/v1_guide.md index 4ef2b0b5b6..d16db46895 100644 --- a/docs/usage/v1_guide.md +++ b/docs/usage/v1_guide.md @@ -138,6 +138,7 @@ Whisper is supported natively. Other encoder-decoder models are supported via th - **BART**: `BartForConditionalGeneration` is supported via the official [bart-plugin](https://github.com/vllm-project/bart-plugin). - **Florence-2**: `Florence2ForConditionalGeneration` is supported via the official [bart-plugin](https://github.com/vllm-project/bart-plugin). +- **Nemotron 3.5 ASR**: `Nemotron3_5AsrForRNNT` is supported via [vllm-nemotron-asr-plugin](https://github.com/Sohaib-Ahmed21/vllm-nemotron-asr-plugin). For other encoder-decoder models (e.g., `MllamaForConditionalGeneration`), we recommend following a similar pattern by implementing support through the [plugin system](../design/plugin_system.md). diff --git a/examples/features/structured_outputs/pyproject.toml b/examples/features/structured_outputs/pyproject.toml index 5e366ab0a0..d5ffe3c8d4 100644 --- a/examples/features/structured_outputs/pyproject.toml +++ b/examples/features/structured_outputs/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "examples-online-structured-outputs" -requires-python = ">=3.10, <3.14" +requires-python = ">=3.11, <3.15" dependencies = ["openai==1.78.1", "pydantic==2.11.4"] version = "0.0.0" diff --git a/examples/features/tensorize_vllm_model.py b/examples/features/tensorize_vllm_model.py index a89b178126..4f4a40233c 100644 --- a/examples/features/tensorize_vllm_model.py +++ b/examples/features/tensorize_vllm_model.py @@ -6,13 +6,11 @@ import logging import os import uuid -from vllm import LLM, SamplingParams +from vllm import LLM from vllm.engine.arg_utils import EngineArgs -from vllm.lora.request import LoRARequest from vllm.model_executor.model_loader.tensorizer import ( TensorizerArgs, TensorizerConfig, - tensorize_lora_adapter, tensorize_vllm_model, tensorizer_kwargs_arg, ) @@ -105,21 +103,6 @@ loading with tensorizer that are given to `TensorizerConfig`, run: under the `tensorizer options` section. These can also be used for deserialization in this example script, although `--tensorizer-uri` and `--path-to-tensors` are functionally the same in this case. - -Tensorizer can also be used to save and load LoRA adapters. A LoRA adapter -can be serialized directly with the path to the LoRA adapter on HF Hub and -a TensorizerConfig object. In this script, passing a HF id to a LoRA adapter -will serialize the LoRA adapter artifacts to `--serialized-directory`. - -You can then use the LoRA adapter with `vllm serve`, for instance, by ensuring -the LoRA artifacts are in your model artifacts directory and specifying -`--enable-lora`. For instance: - -``` -vllm serve s3://my-bucket/vllm/facebook/opt-125m/v1 \ - --load-format tensorizer \ - --enable-lora -``` """ @@ -134,18 +117,6 @@ def get_parser(): ) parser = EngineArgs.add_cli_args(parser) - parser.add_argument( - "--lora-path", - type=str, - required=False, - help="Path to a LoRA adapter to " - "serialize along with model tensors. This can then be deserialized " - "along with the model by instantiating a TensorizerConfig object, " - "creating a dict from it with TensorizerConfig.to_serializable(), " - "and passing it to LoRARequest's initializer with the kwarg " - "tensorizer_config_dict.", - ) - subparsers = parser.add_subparsers(dest="command", required=True) serialize_parser = subparsers.add_parser( @@ -263,42 +234,12 @@ def merge_extra_config_with_tensorizer_config(extra_cfg: dict, cfg: TensorizerCo def deserialize(args, tensorizer_config): - if args.lora_path: - tensorizer_config.lora_dir = tensorizer_config.tensorizer_dir - llm = LLM( - model=args.model, - load_format="tensorizer", - tensor_parallel_size=args.tensor_parallel_size, - model_loader_extra_config=tensorizer_config, - enable_lora=True, - ) - sampling_params = SamplingParams( - temperature=0, max_tokens=256, stop=["[/assistant]"] - ) - - # Truncating this as the extra text isn't necessary - prompts = ["[user] Write a SQL query to answer the question based on ..."] - - # Test LoRA load - print( - llm.generate( - prompts, - sampling_params, - lora_request=LoRARequest( - "sql-lora", - 1, - args.lora_path, - tensorizer_config_dict=tensorizer_config.to_serializable(), - ), - ) - ) - else: - llm = LLM( - model=args.model, - load_format="tensorizer", - tensor_parallel_size=args.tensor_parallel_size, - model_loader_extra_config=tensorizer_config, - ) + llm = LLM( + model=args.model, + load_format="tensorizer", + tensor_parallel_size=args.tensor_parallel_size, + model_loader_extra_config=tensorizer_config, + ) return llm @@ -366,10 +307,6 @@ def main(): **credentials, ) - if args.lora_path: - tensorizer_config.lora_dir = tensorizer_config.tensorizer_dir - tensorize_lora_adapter(args.lora_path, tensorizer_config) - merge_extra_config_with_tensorizer_config(extra_config, tensorizer_config) tensorize_vllm_model(engine_args, tensorizer_config) diff --git a/examples/pooling/score/template/zerank2.jinja b/examples/pooling/score/template/zerank2.jinja new file mode 100644 index 0000000000..8bd16e4301 --- /dev/null +++ b/examples/pooling/score/template/zerank2.jinja @@ -0,0 +1,13 @@ +{#- zerank-2 (zeroentropy/zerank-2-reranker): query in the system turn, document in the user turn. + The assistant suffix is emitted unconditionally, since score templates are rendered + without add_generation_prompt. An optional `instruction` is wrapped as recommended by + ZeroEntropy; without it, the prompt is the official zerank-2 prompt. -#} +{%- set query = messages | selectattr("role", "eq", "query") | map(attribute="content") | list -%} +{%- set document = messages | selectattr("role", "eq", "document") | map(attribute="content") | list -%} +{%- set query = query[0] if query else '' -%} +{%- if instruction is defined and instruction is string and instruction | trim -%} +{%- set query = '' + query + '\n' + instruction + '' -%} +{%- endif -%} +{{- '<|im_start|>system\n' + query + '<|im_end|>\n' -}} +{{- '<|im_start|>user\n' + (document[0] if document else '') + '<|im_end|>\n' -}} +{{- '<|im_start|>assistant\n' -}} diff --git a/examples/pooling/score/using_template_offline.py b/examples/pooling/score/using_template_offline.py index abe8157d4c..1b7fe9f7c0 100644 --- a/examples/pooling/score/using_template_offline.py +++ b/examples/pooling/score/using_template_offline.py @@ -53,6 +53,7 @@ def get_chat_template(model: str) -> str: "mixedbread-ai/mxbai-rerank-base-v2": "mxbai_rerank_v2.jinja", "mixedbread-ai/mxbai-rerank-large-v2": "mxbai_rerank_v2.jinja", "nvidia/llama-nemotron-rerank-1b-v2": "nemotron-rerank.jinja", + "polaria-tech/zerank-2-reranker-vllm": "zerank2.jinja", } # Get the template filename for the specified model @@ -110,6 +111,7 @@ def get_hf_overrides(model: str) -> dict[str, Any]: "method": "from_2_way_softmax", }, "nvidia/llama-nemotron-rerank-1b-v2": {}, + "polaria-tech/zerank-2-reranker-vllm": {}, } hf_overrides = model_name_to_hf_overrides_map.get(model) diff --git a/examples/pooling/score/using_template_online.py b/examples/pooling/score/using_template_online.py index f7dc109dd8..9cf0703ca4 100644 --- a/examples/pooling/score/using_template_online.py +++ b/examples/pooling/score/using_template_online.py @@ -19,6 +19,7 @@ run: vllm serve tomaarsen/Qwen3-Reranker-0.6B-seq-cls --chat-template examples/pooling/score/template/qwen3_reranker.jinja vllm serve mixedbread-ai/mxbai-rerank-base-v2 --hf_overrides '{"architectures": ["Qwen2ForSequenceClassification"],"classifier_from_token": ["0", "1"], "method": "from_2_way_softmax"}' --chat-template examples/pooling/score/template/mxbai_rerank_v2.jinja vllm serve nvidia/llama-nemotron-rerank-1b-v2 --runner pooling --trust-remote-code --chat-template examples/pooling/score/template/nemotron-rerank.jinja + vllm serve polaria-tech/zerank-2-reranker-vllm --runner pooling --chat-template examples/pooling/score/template/zerank2.jinja vllm serve Qwen/Qwen3-Reranker-0.6B --runner pooling --hf_overrides '{"architectures": ["Qwen3ForSequenceClassification"],"classifier_from_token": ["no", "yes"],"is_original_qwen3_reranker": true}' --chat-template examples/pooling/score/template/qwen3_reranker.jinja """ diff --git a/examples/features/batch_invariance/reproducibility_offline.py b/examples/rl/batch_invariance/reproducibility_offline.py similarity index 100% rename from examples/features/batch_invariance/reproducibility_offline.py rename to examples/rl/batch_invariance/reproducibility_offline.py diff --git a/examples/features/pause_resume/data_parallel_pause_resume.py b/examples/rl/pause_resume/data_parallel_pause_resume.py similarity index 100% rename from examples/features/pause_resume/data_parallel_pause_resume.py rename to examples/rl/pause_resume/data_parallel_pause_resume.py diff --git a/examples/features/pause_resume/pause_resume_offline.py b/examples/rl/pause_resume/pause_resume_offline.py similarity index 100% rename from examples/features/pause_resume/pause_resume_offline.py rename to examples/rl/pause_resume/pause_resume_offline.py diff --git a/mkdocs.yaml b/mkdocs.yaml index 8c712257ee..c272e6a687 100644 --- a/mkdocs.yaml +++ b/mkdocs.yaml @@ -102,6 +102,8 @@ plugins: features/quantization/int8.md: features/quantization/llm_compressor/int8_w8a8.md serving/openai_compatible_server.md: serving/online_serving/README.md examples/others/lmcache.md: examples/disaggregated/lmcache.md + examples/features/batch_invariance.md: examples/rl/batch_invariance.md + examples/features/pause_resume.md: examples/rl/pause_resume.md markdown_extensions: - attr_list diff --git a/pyproject.toml b/pyproject.toml index 890dd6bdb7..bc924440fa 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -21,7 +21,6 @@ license-files = ["LICENSE"] readme = "README.md" description = "A high-throughput and memory-efficient inference and serving engine for LLMs" classifiers = [ - "Programming Language :: Python :: 3.10", "Programming Language :: Python :: 3.11", "Programming Language :: Python :: 3.12", "Programming Language :: Python :: 3.13", @@ -32,7 +31,7 @@ classifiers = [ "Topic :: Scientific/Engineering :: Artificial Intelligence", "Topic :: Scientific/Engineering :: Information Analysis", ] -requires-python = ">=3.10,<3.15" +requires-python = ">=3.11,<3.15" dynamic = [ "version", "dependencies", "optional-dependencies"] [project.urls] @@ -103,6 +102,8 @@ ignore = [ "B007", # f-string format "UP032", + # `StrEnum` changes `str()` and `format()` output of members + "UP042", ] [tool.ruff.lint.pydocstyle] @@ -192,7 +193,6 @@ python = "./.venv" [tool.typos.files] # these files may be written in non english words extend-exclude = ["tests/models/fixtures/*", "tests/prompts/*", "tests/tokenizers_/*", - "benchmarks/sonnet.txt", "rust/src/bench/src/datasets/sonnet.txt", "tests/lora/data/*", "build/*", "examples/pooling/token_embed/*", "tests/models/language/pooling/*", "vllm/third_party/*", "vllm/entrypoints/serve/instrumentator/static/*", diff --git a/requirements/common.txt b/requirements/common.txt index 433494056d..ed62ca42f8 100644 --- a/requirements/common.txt +++ b/requirements/common.txt @@ -7,7 +7,7 @@ requests >= 2.33.0 tqdm blake3 py-cpuinfo -transformers >= 5.16.1, < 5.19.0 +transformers >= 5.16.1, < 5.20.0 huggingface_hub >= 1.31.0 # Required for the `huggingface_hub.utils.httpx` re-export tokenizers >= 0.23.2 # Required for fast incremental detokenization. safetensors >= 0.6.2 # MXFP4/MXFP6 dtype support (F8_E8M0, F4) added in 0.6.0: https://github.com/huggingface/safetensors/pull/611 diff --git a/requirements/cpu.txt b/requirements/cpu.txt index 191ed26ae0..0570517e74 100644 --- a/requirements/cpu.txt +++ b/requirements/cpu.txt @@ -22,7 +22,6 @@ torchcodec >= 0.14; platform_machine != "riscv64" and platform_machine != "ppc64 intel-openmp==2024.2.1; platform_machine == "x86_64" # triton-cpu, only for x86_64 CPUs -triton @ https://github.com/triton-lang/triton-cpu/releases/download/v3.8.0/triton-3.8.0+cpu-cp310-cp310-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl ; platform_machine == "x86_64" and python_version == "3.10" triton @ https://github.com/triton-lang/triton-cpu/releases/download/v3.8.0/triton-3.8.0+cpu-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl ; platform_machine == "x86_64" and python_version == "3.11" triton @ https://github.com/triton-lang/triton-cpu/releases/download/v3.8.0/triton-3.8.0+cpu-cp312-abi3-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl ; platform_machine == "x86_64" and python_version >= "3.12" diff --git a/requirements/kv_connectors.txt b/requirements/kv_connectors.txt index 9d9d127881..f6cacd200b 100644 --- a/requirements/kv_connectors.txt +++ b/requirements/kv_connectors.txt @@ -2,7 +2,7 @@ lmcache >= 0.3.9 # CuPy 14.1.0 imports pytest from cupy.testing._random, which breaks runtime # images. 14.1.1 fixes the root cause, so only 14.1.0 is excluded. cupy-cuda13x != 14.1.0 -nixl == 1.4.1 +nixl == 1.5.0 # CUDA 12 build. On the CUDA 13 image install-kv-connectors.sh swaps this for # the mooncake-transfer-engine-cuda13 variant of the same version. mooncake-transfer-engine >= 0.3.12 diff --git a/requirements/test/cpu.txt b/requirements/test/cpu.txt index 45725d903b..7d1f13970f 100644 --- a/requirements/test/cpu.txt +++ b/requirements/test/cpu.txt @@ -1228,7 +1228,7 @@ traitlets==5.16.1 # via # jupyter-client # jupyter-core -transformers==5.18.0 +transformers==5.19.0 # via # -r requirements/test/../common.txt # -r requirements/test/cuda.in diff --git a/requirements/test/cuda.in b/requirements/test/cuda.in index b40d80413e..4a32471652 100644 --- a/requirements/test/cuda.in +++ b/requirements/test/cuda.in @@ -40,7 +40,7 @@ datamodel_code_generator>=0.64.0 # required for minicpm3 test nltk>=3.10.3 # pin transitive rouge-score/lm-eval dep for known GHSAs lm-eval[api]>=0.4.12 # required for model evaluation test mteb[bm25s]>=2, <3 # required for mteb test -transformers==5.18.0 +transformers==5.19.0 schemathesis>=4.0.0 # Required for openai schema test. # quantization buildkite-test-collector==0.1.9 diff --git a/requirements/test/cuda.txt b/requirements/test/cuda.txt index 1f64120e31..29553d78c7 100644 --- a/requirements/test/cuda.txt +++ b/requirements/test/cuda.txt @@ -1335,7 +1335,7 @@ traitlets==5.16.1 # via # jupyter-client # jupyter-core -transformers==5.18.0 +transformers==5.19.0 # via # -c requirements/common.txt # -r requirements/test/../common.txt diff --git a/requirements/test/nightly-torch.txt b/requirements/test/nightly-torch.txt index a229116859..66704d50d3 100644 --- a/requirements/test/nightly-torch.txt +++ b/requirements/test/nightly-torch.txt @@ -29,7 +29,7 @@ opencv-python-headless >= 4.13.0 # required for video test datamodel_code_generator>=0.64.0 # required for minicpm3 test lm-eval[api]>=0.4.12 # required for model evaluation test mteb[bm25s]>=2, <3 # required for mteb test -transformers==5.18.0 +transformers==5.19.0 schemathesis>=4.0.0 # Required for openai schema test. # quantization buildkite-test-collector==0.1.9 diff --git a/requirements/test/rocm.in b/requirements/test/rocm.in index 7c0e4848e9..0ad0b25511 100644 --- a/requirements/test/rocm.in +++ b/requirements/test/rocm.in @@ -36,7 +36,7 @@ datamodel_code_generator>=0.64.0 # required for minicpm3 test nltk>=3.10.3 # pin transitive rouge-score/lm-eval dep for known GHSAs lm-eval[api]>=0.4.12 # required for model evaluation test mteb[bm25s]>=2, <3 # required for mteb test -transformers==5.18.0 +transformers==5.19.0 schemathesis>=4.0.0 # Required for openai schema test # quantization bitsandbytes==0.50.0 diff --git a/requirements/test/rocm.txt b/requirements/test/rocm.txt index b2919a6e83..a0ca0fd833 100644 --- a/requirements/test/rocm.txt +++ b/requirements/test/rocm.txt @@ -1282,7 +1282,7 @@ traitlets==5.16.1 # via # jupyter-client # jupyter-core -transformers==5.18.0 +transformers==5.19.0 # via # -c requirements/common.txt # -r requirements/test/../common.txt diff --git a/requirements/test/xpu.in b/requirements/test/xpu.in index 4b3d68ddfc..c5d08edde9 100644 --- a/requirements/test/xpu.in +++ b/requirements/test/xpu.in @@ -20,7 +20,7 @@ nltk>=3.10.3 # pin transitive rouge-score/lm-eval dep for known GHSAs modelscope<1.38 runai-model-streamer[s3,gcs,azure]>=0.15.7 tensorizer==2.10.1 -transformers==5.18.0 +transformers==5.19.0 # --- Audio Processing --- av diff --git a/requirements/test/xpu.txt b/requirements/test/xpu.txt index 19cd271017..96fda8331f 100644 --- a/requirements/test/xpu.txt +++ b/requirements/test/xpu.txt @@ -1134,7 +1134,7 @@ traitlets==5.16.1 # via # jupyter-client # jupyter-core -transformers==5.18.0 +transformers==5.19.0 # via # -c requirements/common.txt # -r requirements/test/../common.txt diff --git a/requirements/tpu.txt b/requirements/tpu.txt index d8906d8fc1..a16281664c 100644 --- a/requirements/tpu.txt +++ b/requirements/tpu.txt @@ -9,7 +9,6 @@ wheel jinja2>=3.1.6 ray[default] ray[data] -setuptools==78.1.0 setuptools-rust>=1.9.0 nixl==0.3.0 -tpu-inference==0.30.0 +tpu-inference==0.31.0 diff --git a/rust/Cargo.lock b/rust/Cargo.lock index 2ba9832b6b..c7b3d66e4c 100644 --- a/rust/Cargo.lock +++ b/rust/Cargo.lock @@ -593,6 +593,17 @@ version = "1.0.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" +[[package]] +name = "chacha20" +version = "0.10.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "65c35e4b699c7e15ccbe7ee35c005e4fc0a278d22238a2857e6ce2dadeda1b06" +dependencies = [ + "cfg-if", + "cpufeatures 0.3.0", + "rand_core 0.10.1", +] + [[package]] name = "chrono" version = "0.4.44" @@ -1614,6 +1625,7 @@ dependencies = [ "cfg-if", "libc", "r-efi 6.0.0", + "rand_core 0.10.1", "wasip2", "wasip3", ] @@ -3403,74 +3415,6 @@ version = "0.1.29" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e0c5ccf5294c6ccd63a74f1565028353830a9c2f5eb0c682c355c471726a6e3f" -[[package]] -name = "pyo3" -version = "0.29.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4688ddedf473e32662b9b067670129a8afb8c18e351482c70d62ba4a88171e8b" -dependencies = [ - "libc", - "once_cell", - "portable-atomic", - "pyo3-build-config", - "pyo3-ffi", - "pyo3-macros", -] - -[[package]] -name = "pyo3-build-config" -version = "0.29.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f41027e41b4bd03f6e60f9f417fe24a6341a6bb744edd62b6f709f2a52ea30e9" -dependencies = [ - "target-lexicon", -] - -[[package]] -name = "pyo3-ffi" -version = "0.29.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e591a95526fead067432c3b3a33fc74770b87b1e04e73671090d9c2055a2b327" -dependencies = [ - "libc", - "pyo3-build-config", -] - -[[package]] -name = "pyo3-macros" -version = "0.29.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "73225868fc1cd84eef2c3c230ddb91273bf1de46aeb8a4248da76d32a0924a1c" -dependencies = [ - "proc-macro2", - "pyo3-macros-backend", - "quote", - "syn 2.0.117", -] - -[[package]] -name = "pyo3-macros-backend" -version = "0.29.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "571575aa3749fa6216757dd47d2a3e7ef360f329a40f0666a9fbd14889024952" -dependencies = [ - "heck", - "proc-macro2", - "quote", - "syn 2.0.117", -] - -[[package]] -name = "pythonize" -version = "0.29.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6ec376e1216e0c929a74964ce2020012a1a39f32d80e78aa688721219ea7fb89" -dependencies = [ - "pyo3", - "serde", - "serde_json", -] - [[package]] name = "quick-error" version = "2.0.1" @@ -3519,6 +3463,17 @@ dependencies = [ "rand_core 0.9.5", ] +[[package]] +name = "rand" +version = "0.10.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "65c9fb96cbc91e3478eaae79a69fcd3f1ae4ad052e471fe6732fff548984b4af" +dependencies = [ + "chacha20", + "getrandom 0.4.2", + "rand_core 0.10.1", +] + [[package]] name = "rand_chacha" version = "0.3.1" @@ -3557,6 +3512,12 @@ dependencies = [ "getrandom 0.3.4", ] +[[package]] +name = "rand_core" +version = "0.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "63b8176103e19a2643978565ca18b50549f6101881c443590420e4dc998a3c69" + [[package]] name = "rand_distr" version = "0.5.1" @@ -4004,9 +3965,9 @@ checksum = "9774ba4a74de5f7b1c1451ed6cd5285a32eddb5cccb8cc655a4e50009e06477f" [[package]] name = "saa" -version = "5.5.0" +version = "5.6.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "16c7f49c9d5caa3bf4b3106900484b447b9253fe99670ceb81cb6cb5027855e1" +checksum = "bd85992a680de788217d9c70b74aa7685b108cf1aba02f728fb27f1b27199ad1" [[package]] name = "same-file" @@ -4028,12 +3989,12 @@ dependencies = [ [[package]] name = "scc" -version = "3.6.9" +version = "3.8.8" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "45bb5ce9efd4a6e7b0f86c2697fe4c1d78d1f4e6d988c54b752d577cafe22fe8" +checksum = "5d5f8ca67c4f6208caf824037a6a9df0211b4e0937fccc4f543441f294d04511" dependencies = [ "saa", - "sdd 4.7.3", + "sdd 4.8.11", ] [[package]] @@ -4107,9 +4068,12 @@ checksum = "490dcfcbfef26be6800d11870ff2df8774fa6e86d047e3e8c8a76b25655e41ca" [[package]] name = "sdd" -version = "4.7.3" +version = "4.8.11" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b21a75f5913ab130e4b369fb8693be25f29b983e2ecad4279df9bfa5dd8aaf3e" +checksum = "a31877eee03686b905845993b5d9809a220c9dd5a10e9d6021d7ca67384c9070" +dependencies = [ + "saa", +] [[package]] name = "secrecy" @@ -4792,12 +4756,6 @@ dependencies = [ "libc", ] -[[package]] -name = "target-lexicon" -version = "0.13.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "adb6935a6f5c20170eeceb1a3835a49e12e19d792f6dd344ccc76a985ca5a6ca" - [[package]] name = "task-local" version = "0.1.1" @@ -5065,7 +5023,6 @@ dependencies = [ "bytes", "libc", "mio", - "parking_lot", "pin-project-lite", "signal-hook-registry", "socket2", @@ -5873,6 +5830,7 @@ dependencies = [ "half", "hex", "itertools 0.14.0", + "libc", "parking_lot", "rmp-serde", "rmpv", @@ -5882,6 +5840,7 @@ dependencies = [ "serde_repr", "serde_tuple", "serde_with", + "socket2", "task-local", "tempfile", "thiserror 2.0.18", @@ -5988,7 +5947,7 @@ dependencies = [ [[package]] name = "vllm-proto" -version = "0.4.0" +version = "0.5.0" dependencies = [ "prost", "prost-types", @@ -6113,17 +6072,6 @@ dependencies = [ "tracing", ] -[[package]] -name = "vllm-tool-parser-py" -version = "0.1.0" -dependencies = [ - "pyo3", - "pythonize", - "serde_json", - "thiserror-ext", - "vllm-parser", -] - [[package]] name = "vllm-tracing" version = "0.1.0" @@ -6728,22 +6676,21 @@ checksum = "b97154e67e32c85465826e8bcc1c59429aaaf107c1e4a9e53c8d8ccd5eff88d0" [[package]] name = "zeromq" version = "0.6.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "efb2c254fd8f366755335c9e43b865f8484fe3bd717d65ffe7c3f28852863030" +source = "git+https://github.com/zeromq/zmq.rs?rev=67dcba97559ebe8e8206dab753710d8b9cb07591#67dcba97559ebe8e8206dab753710d8b9cb07591" dependencies = [ "async-trait", "asynchronous-codec", "bytes", "crossbeam-queue", "futures", + "futures-channel", "log", - "num-traits", - "once_cell", "parking_lot", - "rand 0.9.5", + "rand 0.10.3", "regex", - "scc 3.6.9", - "thiserror 1.0.69", + "scc 3.8.8", + "socket2", + "thiserror 2.0.18", "tokio", "tokio-util", "uuid", diff --git a/rust/Cargo.toml b/rust/Cargo.toml index e9935d9e28..57f680cc14 100644 --- a/rust/Cargo.toml +++ b/rust/Cargo.toml @@ -11,7 +11,6 @@ members = [ "src/metrics", "src/mock-engine", "src/parser", - "src/parser/python", "src/server", "src/text", "src/tokenizer", @@ -79,8 +78,6 @@ prometheus-client-derive-encode = "0.5.0" prost = "0.14.3" prost-types = "0.14.3" protox = "0.9.1" -pyo3 = "0.29.0" -pythonize = "0.29.0" rand = "0.9.3" rand_distr = "0.5.1" rayon = "1.11.0" @@ -158,7 +155,7 @@ vllm-tokenizer = { path = "src/tokenizer" } vllm-tracing = { path = "src/tracing" } winnow = { version = "1.0.2", features = ["simd"] } xgrammar-structural-tag = "0.4.1" -zeromq = { version = "0.6.0", default-features = false, features = [ +zeromq = { git = "https://github.com/zeromq/zmq.rs", rev = "67dcba97559ebe8e8206dab753710d8b9cb07591", default-features = false, features = [ "tokio-runtime", "all-transport", ] } diff --git a/rust/proto/Cargo.toml b/rust/proto/Cargo.toml index 492cb5a06d..c2e32d43a7 100644 --- a/rust/proto/Cargo.toml +++ b/rust/proto/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "vllm-proto" -version = "0.4.0" +version = "0.5.0" edition.workspace = true license.workspace = true description = "Protobuf messages and gRPC bindings for the vLLM Rust frontend" diff --git a/rust/proto/README.md b/rust/proto/README.md index d312e27aca..3a60c05dfd 100644 --- a/rust/proto/README.md +++ b/rust/proto/README.md @@ -16,7 +16,7 @@ the schemas on pull requests, including on forks. ```toml [dependencies] -vllm-proto = "0.4" +vllm-proto = "0.5" ``` For example, import `vllm_proto::inference_client::InferenceClient` or diff --git a/rust/proto/inference.proto b/rust/proto/inference.proto index 2bc35bcb64..2daacbfc35 100644 --- a/rust/proto/inference.proto +++ b/rust/proto/inference.proto @@ -58,6 +58,25 @@ message GenerateRequest { optional bool watermarking = 16; KVHintsEnvelope kv_hints = 17; + + // Transitional, to be deprecated: only consulted when the engine runs its own + // reasoning parser and the structured-output constraint covers only the + // output after reasoning. Prefer a constraint that covers the full output + // from the first generated token. + EngineReasoningGate engine_reasoning_gate = 18; +} + +// Inputs to the engine's reasoning gate, which keeps a structured-output +// constraint inactive until the engine's reasoning parser detects the end of +// reasoning. +message EngineReasoningGate { + // Omitted: the engine infers the initial state from the prompt. + // true: apply the constraint from the first generated token. + // false: generation starts inside reasoning. + optional bool reasoning_ended = 1; + // Chat template kwargs used to render the prompt, so the engine's reasoning + // parser observes the same thinking mode. + google.protobuf.Struct chat_template_kwargs = 2; } message KVHintsEnvelope { @@ -75,9 +94,9 @@ message KVHintAction { message RandomSampling { uint32 num_sequences = 1; // "n", default (0) means 1 - uint32 top_k = 2; // 0 means default - float top_p = 3; // 0 means default - float min_p = 4; // 0 means default + optional uint32 top_k = 2; // Omitted means default; 0 disables top-k. + optional float top_p = 3; // Omitted means default; must be in (0, 1]. + optional float min_p = 4; // Omitted means default; 0 disables min-p. optional int64 seed = 5; } @@ -102,6 +121,9 @@ message DecodingParameters { bool json_object = 10; string structural_tag = 11; } + + // Token sequences that must not be generated. Each sequence must be nonempty. + repeated TokenIds bad_words_token_ids = 12; } message StoppingCriteria { @@ -115,6 +137,9 @@ message StoppingCriteria { bool include_stop_strings = 5; bool ignore_eos = 6; + + // Omitted or -1 disables the budget; zero immediately closes reasoning. + optional int64 thinking_token_budget = 7; } message ResponseOptions { @@ -210,6 +235,7 @@ message FinishInfo { google.protobuf.Struct kv_transfer_params = 6; //uint64 seed = 7; google.protobuf.Struct ec_transfer_params = 8; + optional uint32 num_cached_tokens = 9; } // Info for candidate tokens other than the input/sampled diff --git a/rust/src/bench/AGENTS.md b/rust/src/bench/AGENTS.md index d9a17f63f2..62fd963bbf 100644 --- a/rust/src/bench/AGENTS.md +++ b/rust/src/bench/AGENTS.md @@ -47,16 +47,16 @@ cargo test -p vllm-bench -- --ignored - `src/datasets/random.rs` — Random dataset generation with rayon parallelism - `src/datasets/random_mm.rs` — Random multimodal dataset (synthetic JPEG images, bucket config sampling, pre-serialized JSON fragments); `--enable-multimodal-chat` pre-builds the chat `messages` array at dataset time (mirrors Python's `apply_multimodal_chat_transformation`) - `src/datasets/sharegpt.rs` — ShareGPT JSON loader + HuggingFace Hub auto-download with caching -- `src/datasets/sonnet.rs` and `src/datasets/sonnet.txt` — Sonnet dataset (built-in Shakespeare sonnets via `include_str!("sonnet.txt")`; controllable token length + shared prefix; mirrors Python `SonnetDataset`) - `src/datasets/speed_bench.rs` — NVIDIA SPEED-Bench loader (HF datasets-server API, 6 configs, 11 categories, local cache) - `src/datasets/hf_dataset.rs` — Generic HuggingFace dataset loader (parquet shards via hf-hub into the standard HF hub cache; datasets-server for config/split discovery; native main-branch parquet/JSON fallback for private datasets; column auto-detection) - `src/datasets/custom.rs` — Custom JSONL dataset (`{"prompt": ..., "output_tokens": ...}` per line; `--custom-output-len -1` uses per-line output_tokens; prompts always sent raw — no client-side chat template) +- `src/datasets/timed_trace.rs` — Mooncake-style replay trace (`{timestamp, input_length, output_length, hash_ids}` JSONL; `hash_ids` expand to memoized token chunks so prefix sharing reproduces; `--self-timed` fires each request at its recorded offset instead of `--request-rate`; mirrors Python `TimedTrace`) - `src/datasets/prefix_repetition.rs` — Prefix repetition dataset (N shared prefixes × fresh random suffixes, standard prefix-cache stress; mirrors Python `PrefixRepetitionRandomDataset`) - `src/datasets/random_rerank.rs` — Random rerank dataset (one query + batched documents per request for `vllm-rerank`; `--no-reranker` for embedding-based scoring; mirrors Python `RandomDatasetForReranking`) - `src/datasets/multi_turn.rs` — Multi-turn synthetic generator + ShareGPT multi-turn loader (3-tier prefix sharing: global/conversation/unique-suffix; `per_turn_input_len`) - `src/metrics/mod.rs` — `BenchmarkMetrics` and `MultiTurnMetrics` structs - `src/metrics/calculator.rs` — TTFT/TPOT/ITL/E2EL/throughput stats, goodput SLO checking, peak concurrency, `calculate_multi_turn_metrics` -- `src/metrics/steady_state.rs` — Steady-state window detection (in-flight concurrency plateau via two-pointer start/end merge) + plateau throughput/TTFT/TPOT; gated on `--max-concurrency` set + `--request-rate inf` (closed-loop) +- `src/metrics/steady_state.rs` — Steady-state window detection (in-flight concurrency plateau via two-pointer start/end merge) + plateau throughput/TTFT/TPOT; gated on `--max-concurrency` set + `--request-rate inf` and off under `--self-timed` (closed-loop only) - `src/output/console.rs` — Terminal output matching Python format + multi-turn per-turn breakdown - `src/output/json.rs` — JSON result file (compatible with Python schema) + multi-turn JSON with `per_turn_metrics` diff --git a/rust/src/bench/README.md b/rust/src/bench/README.md index fb1e5b529d..043605d9c8 100644 --- a/rust/src/bench/README.md +++ b/rust/src/bench/README.md @@ -13,7 +13,7 @@ vllm-bench --backend vllm --base-url http://127.0.0.1:8000 \ - **Fast** — ~7 ms startup, single ~7 MB static binary, no Python imports. - **Scales** — `Arc` prompt sharing + mimalloc keep memory <100 MB at 1400+ concurrency. -- **Many datasets** — `random`, `random-mm` (VLM), `sharegpt`, `sonnet`, `speed-bench`, and any HuggingFace dataset. +- **Many datasets** — `random`, `random-mm` (VLM), `sharegpt`, `speed-bench`, and any HuggingFace dataset. - **Many backends** — completions, chat, embeddings, pooling, and rerank. - **Beyond a single run** — concurrency/rate **sweeps**, **multi-run** stats, **multi-turn** conversations, **LoRA** multi-adapter, and result **comparison**. - **Steady-state metrics** — throughput/latency measured over the saturated plateau, excluding ramp-up and drain. @@ -127,7 +127,7 @@ vllm-bench \
-Datasets (ShareGPT / Sonnet / HuggingFace / SPEED-Bench) +Datasets (ShareGPT / HuggingFace / SPEED-Bench) ```bash # ShareGPT (auto-downloads from HuggingFace on first run, cached afterwards) @@ -141,14 +141,6 @@ vllm-bench \ --dataset-name sharegpt --dataset-path /path/to/ShareGPT_V3.json \ --num-prompts 500 --save-result -# Sonnet — built-in Shakespeare sonnets, no dataset file needed. -# Generates prompts of a controllable token length with a shared prefix. -vllm-bench \ - --backend openai-chat --base-url http://127.0.0.1:8000 --model \ - --dataset-name sonnet \ - --sonnet-input-len 550 --sonnet-output-len 150 --sonnet-prefix-len 200 \ - --num-prompts 500 - # Any HuggingFace dataset (downloads parquet shards into the standard HF hub # cache — shared with `hf download` and Python tooling — auto-detects columns) vllm-bench \ @@ -449,7 +441,7 @@ to measure a concurrent serving-style workload. | `vllm-pooling` | `/pooling` | vLLM native pooling endpoint | | `vllm-rerank` | `/v1/rerank` | vLLM reranking (query from prompt, documents via `--extra-body`) | -Pooling backends are non-streaming and report E2EL (end-to-end latency) only. Use `--dataset-name sharegpt`, `sonnet`, or `hf` for text-based embedding/rerank benchmarks, or `random` for token-ID-based embedding benchmarks. +Pooling backends are non-streaming and report E2EL (end-to-end latency) only. Use `--dataset-name sharegpt` or `hf` for text-based embedding/rerank benchmarks, or `random` for token-ID-based embedding benchmarks. ## Supported Datasets @@ -458,7 +450,6 @@ Pooling backends are non-streaming and report E2EL (end-to-end latency) only. Us | `random` | Synthetic prompts with exact token-length matching (default) | | `random-mm` | Synthetic multimodal prompts with random JPEG images for VLM benchmarking (requires `openai-chat`) | | `sharegpt` | Real conversations from ShareGPT (auto-downloads from HuggingFace, or use `--dataset-path`) | -| `sonnet` | Built-in Shakespeare sonnets; controllable token length + shared prefix, no dataset file needed | | `speed-bench` | NVIDIA SPEED-Bench for speculative decoding evaluation (auto-downloads, 11 categories) | | `hf` | Any HuggingFace dataset with parquet data (downloads parquet shards via hf-hub into the standard HF hub cache, auto-detects chat/text columns) | @@ -523,8 +514,8 @@ Run `vllm-bench --help` for the authoritative list. Grouped reference below. | Flag | Default | Description | | ------ | --------- | ------------- | -| `--dataset-name` | `random` | Dataset type (`random`, `random-mm`, `sharegpt`, `sonnet`, `speed-bench`, `hf`) | -| `--dataset-path` | — | Path to dataset file (optional for `sharegpt`/`sonnet`, which auto-source) | +| `--dataset-name` | `random` | Dataset type (`random`, `random-mm`, `sharegpt`, `speed-bench`, `hf`) | +| `--dataset-path` | — | Path to dataset file (`sharegpt` auto-downloads when omitted) | | `--num-prompts` | `1000` | Number of prompts to generate (conversations in multi-turn mode) | | `--max-model-len` | — | Filter out requests where `prompt_len + output_len` exceeds this context length | | `--input-len` | — | Override input length (general) | @@ -545,10 +536,6 @@ Run `vllm-bench --help` for the authoritative list. Grouped reference below. | `--random-mm-bucket-config` | `{(256,256,1): 0.5, (720,1280,1): 0.5}` | `(height,width,frames)` → probability (Python tuple syntax; frames=1 = image) | | **ShareGPT** | | | | `--sharegpt-output-len` | — | Override output length | -| **Sonnet** | | | -| `--sonnet-input-len` | `550` | Input tokens per request | -| `--sonnet-output-len` | `150` | Output tokens per request | -| `--sonnet-prefix-len` | `200` | Prefix tokens shared across requests | | **SPEED-Bench** | | | | `--speed-bench-config` | `qualitative` | Split (`qualitative`, `throughput_1k`/`2k`/`8k`/`16k`/`32k`); alias: `--speed-bench-dataset-subset` (Python name) | | `--speed-bench-output-len` | `4096` | Output tokens per request (matches Python default) | @@ -817,7 +804,6 @@ src/ │ ├── random_mm.rs # Random multimodal dataset (JPEG generation, bucket sampling) │ ├── multi_turn.rs # Multi-turn synthetic + ShareGPT conversation generators │ ├── sharegpt.rs # ShareGPT JSON dataset loader -│ ├── sonnet.rs # Sonnet dataset (built-in Shakespeare sonnets) │ ├── speed_bench.rs # NVIDIA SPEED-Bench loader (auto-download + cache) │ └── hf_dataset.rs # Generic HuggingFace dataset (auto-download, column detection) ├── metrics/ diff --git a/rust/src/bench/src/benchmark.rs b/rust/src/bench/src/benchmark.rs index 54b6e8701f..5833a3bdce 100644 --- a/rust/src/bench/src/benchmark.rs +++ b/rust/src/bench/src/benchmark.rs @@ -17,9 +17,13 @@ use crate::metrics::calculator::{calculate_embedding_metrics, calculate_metrics} use crate::metrics::steady_state; use crate::output::console::print_results; use crate::output::json::{append_result, build_result_json, compute_result_filename, save_result}; -use crate::rate_control::compute_schedule; +use crate::rate_control::{compute_schedule, trace_schedule}; use crate::ready_checker::{get_first_model, wait_for_endpoint}; +/// Past this first-arrival offset a self-timed run is almost certainly a +/// wrongly scaled or epoch-anchored trace rather than a real idle head. +const TRACE_START_WARN_SECONDS: f64 = 3600.0; + /// Pre-resolve the hostname in `base_url` and pin all resolved IPs on the /// client builder via [`reqwest::ClientBuilder::resolve_to_addrs`]. This /// avoids repeated DNS lookups under high concurrency which can cause @@ -370,14 +374,6 @@ pub async fn run_benchmark(config: &BenchConfig) -> Result { config.num_prompts, config.dataset_path.as_deref().unwrap_or("auto-download") ), - DatasetName::Sonnet => format!( - "{} prompts from Sonnet ({}, isl={}, osl={}, prefix={})", - config.num_prompts, - config.dataset_path.as_deref().unwrap_or("built-in"), - config.sonnet_input_len, - config.sonnet_output_len, - config.sonnet_prefix_len, - ), DatasetName::SpeedBench => { let truncate_info = config .speed_bench_max_input_len @@ -412,6 +408,12 @@ pub async fn run_benchmark(config: &BenchConfig) -> Result { "{} random rerank requests (batch={}, reranker={})", config.num_prompts, config.random_batch_size, config.is_reranker, ), + DatasetName::TimedTrace => format!( + "{} requests from timed trace ({}, chunk_hash_size={})", + config.num_prompts, + config.dataset_path.as_deref().unwrap_or("unknown"), + config.timed_trace_chunk_hash_size, + ), }; tracing::info!( dataset = ?config.dataset_name, @@ -484,21 +486,6 @@ pub async fn run_benchmark(config: &BenchConfig) -> Result { config.disable_shuffle, )? } - DatasetName::Sonnet => { - let tok = tokenizer - .as_ref() - .ok_or_else(|| BenchError::Config("Sonnet dataset requires a tokenizer".into()))?; - crate::datasets::sonnet::load_sonnet_dataset( - tok, - config.dataset_path.as_deref(), - config.num_prompts, - config.sonnet_input_len, - config.sonnet_output_len, - config.sonnet_prefix_len, - config.seed, - &config.request_id_prefix, - )? - } DatasetName::SpeedBench => { let tok = tokenizer.as_ref().ok_or_else(|| { BenchError::Config("SPEED-Bench dataset requires a tokenizer".into()) @@ -604,6 +591,30 @@ pub async fn run_benchmark(config: &BenchConfig) -> Result { config.is_reranker, )? } + DatasetName::TimedTrace => { + let tok = tokenizer.as_ref().ok_or_else(|| { + BenchError::Config("Timed trace dataset requires a tokenizer".into()) + })?; + let path = config.dataset_path.as_deref().ok_or_else(|| { + BenchError::Config( + "--dataset-path is required for --dataset-name timed_trace".into(), + ) + })?; + crate::datasets::timed_trace::load_timed_trace_dataset( + tok, + path, + config.num_prompts, + &config.request_id_prefix, + &crate::datasets::timed_trace::TimedTraceOptions { + chunk_size: config.timed_trace_chunk_hash_size, + sec_multiplier: config.timed_trace_sec_multiplier, + label_timestamp: &config.timed_trace_label_timestamp, + label_input_length: &config.timed_trace_label_input_length, + label_output_length: &config.timed_trace_label_output_length, + label_hash_ids: &config.timed_trace_label_hash_ids, + }, + )? + } }; let gen_elapsed = gen_start.elapsed(); @@ -823,20 +834,30 @@ pub async fn run_benchmark(config: &BenchConfig) -> Result { tracing::info!("detected speculative decoding; collecting metrics"); } - // Main benchmark - let distribution = if config.burstiness == 1.0 { - "Poisson process" + // Main benchmark. Self-timed runs ignore --request-rate/--burstiness + // entirely (config.rs rejects them), so don't report a rate that isn't used. + if config.self_timed { + tracing::info!( + timing = "trace", + max_concurrency = config.max_concurrency.unwrap_or(config.num_prompts), + prompts = config.num_prompts, + "starting main benchmark run; firing requests at their trace timestamps" + ); } else { - "Gamma distribution" - }; - tracing::info!( - request_rate = config.request_rate, - burstiness = config.burstiness, - distribution, - max_concurrency = config.max_concurrency.unwrap_or(config.num_prompts), - prompts = config.num_prompts, - "starting main benchmark run" - ); + let distribution = if config.burstiness == 1.0 { + "Poisson process" + } else { + "Gamma distribution" + }; + tracing::info!( + request_rate = config.request_rate, + burstiness = config.burstiness, + distribution, + max_concurrency = config.max_concurrency.unwrap_or(config.num_prompts), + prompts = config.num_prompts, + "starting main benchmark run" + ); + } // Pre-assign LoRA adapters to each request (None when --lora-modules not set). let lora_assignments = assign_lora_modules( @@ -855,14 +876,40 @@ pub async fn run_benchmark(config: &BenchConfig) -> Result { ); } - // Compute request schedule - let schedule = compute_schedule( - input_requests.len(), - config.request_rate, - config.burstiness, - config.seed, - config.ramp_up.as_ref(), - ); + // Compute request schedule: the trace's recorded arrival offsets when + // self-timed, synthetic gamma inter-arrivals otherwise. Either way the + // dispatch loop below sleeps to benchmark_start + delay, an absolute + // deadline, so trace replay does not accumulate drift. + let schedule = if config.self_timed { + let schedule = trace_schedule(input_requests.iter().map(|r| r.timestamp)); + let first = schedule.delays.first().copied().unwrap_or(0.0); + let last = schedule.delays.iter().copied().fold(0.0, f64::max); + tracing::info!( + first_arrival_seconds = first, + last_arrival_seconds = last, + "replaying trace arrival schedule" + ); + // Timestamps are absolute offsets from now, never normalized (Python + // does the same). An unscaled millisecond trace or an epoch-anchored + // one parks every request past any plausible run length, so say so + // rather than sitting at 0/N. + if first > TRACE_START_WARN_SECONDS { + tracing::warn!( + first_arrival_seconds = first, + "first trace arrival is far in the future; check \ + --timed-trace-sec-multiplier or re-anchor the trace timestamps" + ); + } + schedule + } else { + compute_schedule( + input_requests.len(), + config.request_rate, + config.burstiness, + config.seed, + config.ramp_up.as_ref(), + ) + }; // Progress bar let pb = if config.disable_tqdm { @@ -1081,7 +1128,10 @@ pub async fn run_benchmark(config: &BenchConfig) -> Result { }; // Attach steady-state metrics when the closed-loop scope gate passes. + // Trace replay is open-loop: an in-flight plateau there reflects a burst + // in the trace, not a saturated server, so it is out of scope. let scope_ok = !config.no_steady_state + && !config.self_timed && config.max_concurrency.is_some() && config.request_rate.is_infinite(); if scope_ok { diff --git a/rust/src/bench/src/cli.rs b/rust/src/bench/src/cli.rs index 302dbb4792..93aa0d9c05 100644 --- a/rust/src/bench/src/cli.rs +++ b/rust/src/bench/src/cli.rs @@ -73,8 +73,6 @@ pub enum DatasetName { RandomMm, #[value(name = "sharegpt")] ShareGpt, - #[value(name = "sonnet")] - Sonnet, #[value(name = "speed-bench", alias = "speed_bench")] SpeedBench, #[value(name = "hf")] @@ -85,6 +83,8 @@ pub enum DatasetName { PrefixRepetition, #[value(name = "random-rerank")] RandomRerank, + #[value(name = "timed_trace", alias = "timed-trace")] + TimedTrace, } /// Ramp-up strategy for request rate. @@ -323,6 +323,33 @@ pub struct BenchServeArgs { #[arg(long, default_value_t = 128)] pub prefix_repetition_output_len: usize, + // --- Timed-trace dataset --- + /// How many tokens each prefix hash in the trace represents + /// (e.g. 512 for Moonshot traces, 16 for Qwen/Alibaba). + #[arg(long, default_value_t = 16)] + pub timed_trace_chunk_hash_size: usize, + + /// Multiplier converting trace timestamps to seconds + /// (e.g. 0.001 if timestamps are in milliseconds). + #[arg(long, default_value_t = 1.0)] + pub timed_trace_sec_multiplier: f64, + + /// JSON key of the timestamp field in the trace. + #[arg(long, default_value = "timestamp")] + pub timed_trace_label_timestamp: String, + + /// JSON key of the input length field in the trace. + #[arg(long, default_value = "input_length")] + pub timed_trace_label_input_length: String, + + /// JSON key of the output length field in the trace. + #[arg(long, default_value = "output_length")] + pub timed_trace_label_output_length: String, + + /// JSON key of the hash ids field in the trace. + #[arg(long, default_value = "hash_ids")] + pub timed_trace_label_hash_ids: String, + /// Number of prompts to generate. #[arg(long, default_value_t = 1000)] pub num_prompts: usize, @@ -335,6 +362,16 @@ pub struct BenchServeArgs { #[arg(long, default_value_t = 1.0)] pub burstiness: f64, + /// Schedule requests at the timestamps recorded in the trace instead of + /// --request-rate. Defaults to on for --dataset-name timed_trace (the only + /// dataset that carries timestamps); an error for any other dataset. + #[arg(long, overrides_with = "no_self_timed")] + pub self_timed: bool, + + /// Force trace-driven timing off for timed_trace and use --request-rate. + #[arg(long, overrides_with = "self_timed")] + pub no_self_timed: bool, + /// Maximum number of concurrent requests. #[arg(long)] pub max_concurrency: Option, @@ -521,19 +558,6 @@ pub struct BenchServeArgs { #[arg(long, default_value_t = false)] pub disable_shuffle: bool, - // --- Sonnet dataset --- - /// Number of input tokens per request (sonnet dataset). - #[arg(long, default_value_t = crate::datasets::sonnet::DEFAULT_INPUT_LEN)] - pub sonnet_input_len: usize, - - /// Number of output tokens per request (sonnet dataset). - #[arg(long, default_value_t = crate::datasets::sonnet::DEFAULT_OUTPUT_LEN)] - pub sonnet_output_len: usize, - - /// Number of prefix tokens shared across requests (sonnet dataset). - #[arg(long, default_value_t = crate::datasets::sonnet::DEFAULT_PREFIX_LEN)] - pub sonnet_prefix_len: usize, - /// SPEED-Bench config/split (qualitative, throughput_1k, throughput_2k, throughput_8k, /// throughput_16k, throughput_32k). #[arg( diff --git a/rust/src/bench/src/config.rs b/rust/src/bench/src/config.rs index 84f55ad427..3e180cfb30 100644 --- a/rust/src/bench/src/config.rs +++ b/rust/src/bench/src/config.rs @@ -145,9 +145,6 @@ pub struct BenchConfig { pub prefix_repetition_num_prefixes: usize, pub prefix_repetition_output_len: usize, pub sharegpt_output_len: Option, - pub sonnet_input_len: usize, - pub sonnet_output_len: usize, - pub sonnet_prefix_len: usize, pub no_oversample: bool, pub disable_shuffle: bool, pub num_prompts: usize, @@ -201,6 +198,15 @@ pub struct BenchConfig { pub hf_subset: Option, pub hf_output_len: Option, pub hf_text_column: Option, + /// Fire each request at its trace-recorded timestamp instead of a + /// synthetic --request-rate schedule. Only timed_trace can set this. + pub self_timed: bool, + pub timed_trace_chunk_hash_size: usize, + pub timed_trace_sec_multiplier: f64, + pub timed_trace_label_timestamp: String, + pub timed_trace_label_input_length: String, + pub timed_trace_label_output_length: String, + pub timed_trace_label_hash_ids: String, pub reset_prefix_cache: bool, pub prompt_token_ids: bool, // --- Random multimodal dataset --- @@ -379,6 +385,79 @@ impl BenchConfig { let mut multi_turn_min_turns = args.multi_turn_num_turns; let mut multi_turn_max_turns = args.multi_turn_num_turns; + let self_timed = if args.dataset_name == DatasetName::TimedTrace { + if !matches!(args.backend, BackendKind::Vllm | BackendKind::Openai) { + return Err(BenchError::Config( + "timed_trace passes pre-tokenized prompts (list[int]) and requires \ + a completions backend ('vllm' or 'openai')" + .into(), + )); + } + if args.dataset_path.is_none() { + return Err(BenchError::Config( + "--dataset-path is required for --dataset-name timed_trace".into(), + )); + } + // A zero, negative or non-finite scale turns trace timestamps into + // offsets the scheduler cannot sleep on. + if !(args.timed_trace_sec_multiplier.is_finite() + && args.timed_trace_sec_multiplier > 0.0) + { + return Err(BenchError::Config(format!( + "--timed-trace-sec-multiplier must be finite and positive, got {}", + args.timed_trace_sec_multiplier + ))); + } + let self_timed = !args.no_self_timed; + if self_timed { + // Flags that shape a synthetic schedule are unreachable once + // the trace supplies arrival times. Accepting them silently + // would still stamp the result JSON and its filename with a + // request rate that never ran. + let mut unused: Vec<&str> = Vec::new(); + if args.request_rate.is_finite() { + unused.push("--request-rate"); + } + if args.burstiness != 1.0 { + unused.push("--burstiness"); + } + if args.ramp_up_strategy.is_some() { + unused.push("--ramp-up-strategy"); + } + if args.sweep_request_rate.is_some() { + unused.push("--sweep-request-rate"); + } + // A per-point num_prompts truncates the trace to a different + // number of rows, so each sweep point replays a different span. + if args.sweep_max_concurrency.is_some() && args.sweep_num_prompts_factor.is_some() { + return Err(BenchError::Config( + "--sweep-num-prompts-factor truncates the trace to a different \ + number of rows per sweep point, so each point replays a \ + different span; drop it to sweep concurrency over the whole trace" + .into(), + )); + } + if !unused.is_empty() { + return Err(BenchError::Config(format!( + "{} {} no effect under trace-driven timing; pass --no-self-timed \ + to schedule timed_trace by --request-rate instead", + unused.join(" / "), + if unused.len() == 1 { "has" } else { "have" }, + ))); + } + } + self_timed + } else { + if args.self_timed || args.no_self_timed { + return Err(BenchError::Config( + "--self-timed/--no-self-timed is only supported with \ + --dataset-name timed_trace" + .into(), + )); + } + false + }; + // For random datasets with openai-compatible backends, default to ignore_eos. // Exception: multi-turn mode, where ignore_eos causes unbounded context growth // across turns. Multi-turn uses min_tokens instead for output length control. @@ -387,6 +466,8 @@ impl BenchConfig { false } else { args.ignore_eos + // timed_trace: generation must run to the trace's output_length + || args.dataset_name == DatasetName::TimedTrace || ((args.dataset_name == DatasetName::Random || args.dataset_name == DatasetName::RandomMm) && args.backend.is_openai_compatible() @@ -447,8 +528,7 @@ impl BenchConfig { (MmLimitPerPrompt::default(), Vec::new()) }; - // Note: --dataset-path is optional for sharegpt (auto-downloads) and - // sonnet (uses built-in Shakespeare's sonnets). + // --dataset-path is optional for sharegpt (auto-downloads). // Range ratio (Python semantics: [len*(1-r), len*(1+r)], each r in [0,1)) let random_range_ratio = RangeRatio::parse(&args.random_range_ratio)?; @@ -689,9 +769,6 @@ impl BenchConfig { random_cache_hit_fraction: args.random_cache_hit_fraction, random_cache_ratio: args.random_cache_ratio, sharegpt_output_len: args.sharegpt_output_len, - sonnet_input_len: args.sonnet_input_len, - sonnet_output_len: args.sonnet_output_len, - sonnet_prefix_len: args.sonnet_prefix_len, no_oversample: args.no_oversample, disable_shuffle: args.disable_shuffle, num_prompts: args.num_prompts, @@ -752,6 +829,13 @@ impl BenchConfig { hf_subset: args.hf_subset.clone(), hf_output_len: args.hf_output_len, hf_text_column: args.hf_text_column.clone(), + self_timed, + timed_trace_chunk_hash_size: args.timed_trace_chunk_hash_size, + timed_trace_sec_multiplier: args.timed_trace_sec_multiplier, + timed_trace_label_timestamp: args.timed_trace_label_timestamp.clone(), + timed_trace_label_input_length: args.timed_trace_label_input_length.clone(), + timed_trace_label_output_length: args.timed_trace_label_output_length.clone(), + timed_trace_label_hash_ids: args.timed_trace_label_hash_ids.clone(), reset_prefix_cache: args.reset_prefix_cache, prompt_token_ids: args.prompt_token_ids, random_mm_base_items_per_request: args.random_mm_base_items_per_request, @@ -906,6 +990,98 @@ mod tests { TestCli::parse_from(args).args } + fn base_timed_trace_args() -> Vec<&'static str> { + vec![ + "vllm-bench", + "--model", + "test-model", + "--dataset-name", + "timed_trace", + "--dataset-path", + "trace.jsonl", + ] + } + + #[test] + fn test_timed_trace_defaults_to_self_timed() { + let config = BenchConfig::from_args(&parse_args(base_timed_trace_args())).unwrap(); + assert!(config.self_timed); + assert!(config.ignore_eos); + } + + #[test] + fn test_no_self_timed_restores_rate_scheduling() { + let mut args = base_timed_trace_args(); + args.extend(["--no-self-timed", "--request-rate", "10"]); + let config = BenchConfig::from_args(&parse_args(args)).unwrap(); + assert!(!config.self_timed); + assert_eq!(config.request_rate, 10.0); + } + + /// Rate/burstiness/ramp-up shape a synthetic schedule the trace replaces; + /// accepting them would mislabel the result JSON with a rate that never ran. + #[test] + fn test_self_timed_rejects_synthetic_schedule_flags() { + for flag in [ + vec!["--request-rate", "10"], + vec!["--burstiness", "0.5"], + vec!["--sweep-request-rate", "1,5,10"], + ] { + let mut args = base_timed_trace_args(); + args.extend(flag.iter().copied()); + let msg = BenchConfig::from_args(&parse_args(args)) + .expect_err("self-timed replay should reject synthetic schedule flags") + .to_string(); + assert!( + msg.contains(flag[0]) && msg.contains("--no-self-timed"), + "{flag:?}: {msg}" + ); + } + } + + /// A per-point num_prompts truncates the trace differently at every sweep + /// point, so the replayed span stops being comparable across them. + #[test] + fn test_self_timed_rejects_sweep_num_prompts_factor() { + let mut args = base_timed_trace_args(); + args.extend([ + "--sweep-max-concurrency", + "8,16", + "--sweep-num-prompts-factor", + "4", + ]); + let err = BenchConfig::from_args(&parse_args(args)).expect_err("should reject the factor"); + let msg = err.to_string(); + assert!( + msg.contains("--sweep-num-prompts-factor") && msg.contains("different span"), + "{msg}" + ); + } + + #[test] + fn test_self_timed_allows_concurrency_sweep_over_full_trace() { + let mut args = base_timed_trace_args(); + args.extend(["--sweep-max-concurrency", "8,16"]); + assert!(BenchConfig::from_args(&parse_args(args)).is_ok()); + } + + #[test] + fn test_non_positive_sec_multiplier_fails() { + // `=` form: clap reads a bare "-0.001" as a flag. + for value in [ + "--timed-trace-sec-multiplier=0", + "--timed-trace-sec-multiplier=-0.001", + "--timed-trace-sec-multiplier=inf", + "--timed-trace-sec-multiplier=nan", + ] { + let mut args = base_timed_trace_args(); + args.push(value); + let err = BenchConfig::from_args(&parse_args(args)) + .expect_err("should reject a multiplier that cannot scale timestamps"); + assert!(err.to_string().contains("--timed-trace-sec-multiplier")); + } + } + fn base_multi_turn_args() -> Vec<&'static str> { vec![ "vllm-bench", diff --git a/rust/src/bench/src/datasets/custom.rs b/rust/src/bench/src/datasets/custom.rs index 11ddee02b1..a9fcba3354 100644 --- a/rust/src/bench/src/datasets/custom.rs +++ b/rust/src/bench/src/datasets/custom.rs @@ -43,29 +43,10 @@ pub fn load_custom_dataset( no_oversample: bool, disable_shuffle: bool, ) -> Result> { - let content = std::fs::read_to_string(path) - .map_err(|e| BenchError::Config(format!("Failed to read custom dataset '{path}': {e}")))?; - - let mut lines: Vec = Vec::new(); - for (lineno, line) in content.lines().enumerate() { - let line = line.trim(); - if line.is_empty() { - continue; - } - let parsed: CustomLine = serde_json::from_str(line).map_err(|e| { - BenchError::Config(format!( - "Invalid JSONL at {path}:{}: {e} (each line must be an object \ - with a 'prompt' field)", - lineno + 1 - )) - })?; - lines.push(parsed); - } - if lines.is_empty() { - return Err(BenchError::Config(format!( - "Custom dataset '{path}' contains no entries" - ))); - } + let mut lines: Vec = super::read_jsonl(path, "custom dataset", None)? + .into_iter() + .map(|(_, line)| line) + .collect(); // Python shuffles the loaded data (seeded) before taking num_requests. if !disable_shuffle { @@ -118,26 +99,13 @@ pub fn load_custom_dataset( #[cfg(test)] mod tests { + use super::super::test_support::{test_tokenizer, write_temp_jsonl}; use super::*; - fn write_temp_jsonl(name: &str, content: &str) -> String { - let path = std::env::temp_dir().join(format!("vllm-bench-custom-{name}.jsonl")); - std::fs::write(&path, content).unwrap(); - path.to_string_lossy().into_owned() - } - - /// gpt2 via built-in tiktoken encoding — loads without network access. - fn test_tokenizer() -> TokenizerKind { - TokenizerKind::Tiktoken( - crate::tiktoken::load_builtin_tiktoken("gpt2") - .expect("gpt2 built-in tiktoken should always load without network"), - ) - } - #[test] fn test_load_custom_dataset_basic() { let path = write_temp_jsonl( - "basic", + "custom-basic", r#"{"prompt": "hello world", "output_tokens": 10} {"prompt": "foo bar baz", "output_tokens": 20} "#, @@ -154,7 +122,7 @@ mod tests { #[test] fn test_load_custom_dataset_per_line_output_tokens() { let path = write_temp_jsonl( - "perline", + "custom-perline", r#"{"prompt": "hello", "output_tokens": 10} {"prompt": "world", "output_tokens": 20} "#, @@ -167,7 +135,7 @@ mod tests { #[test] fn test_load_custom_dataset_missing_output_tokens_errors() { - let path = write_temp_jsonl("missing", r#"{"prompt": "hello"}"#); + let path = write_temp_jsonl("custom-missing", r#"{"prompt": "hello"}"#); let err = load_custom_dataset(&test_tokenizer(), &path, 1, -1, 0, "t-", true, true) .expect_err("should fail without output_tokens"); assert!(err.to_string().contains("output_tokens")); @@ -175,7 +143,7 @@ mod tests { #[test] fn test_load_custom_dataset_missing_prompt_errors() { - let path = write_temp_jsonl("noprompt", r#"{"text": "hello"}"#); + let path = write_temp_jsonl("custom-noprompt", r#"{"text": "hello"}"#); assert!( load_custom_dataset(&test_tokenizer(), &path, 1, 256, 0, "t-", true, true).is_err() ); diff --git a/rust/src/bench/src/datasets/mod.rs b/rust/src/bench/src/datasets/mod.rs index 8f81f8d3c7..bfdc4006a6 100644 --- a/rust/src/bench/src/datasets/mod.rs +++ b/rust/src/bench/src/datasets/mod.rs @@ -10,8 +10,8 @@ pub mod random; pub mod random_mm; pub mod random_rerank; pub mod sharegpt; -pub mod sonnet; pub mod speed_bench; +pub mod timed_trace; use std::sync::Arc; @@ -55,6 +55,10 @@ pub struct SampleRequest { /// rerank sends `[0]` as the query and `[1..]` as documents (random-rerank). /// Mirrors Python's list-valued `SampleRequest.prompt`. pub prompt_list: Option]>>, + /// Arrival time in seconds relative to benchmark start (timed_trace only). + /// With --self-timed the scheduler fires the request at this offset + /// instead of a synthetic --request-rate schedule. + pub timestamp: Option, } impl Default for SampleRequest { @@ -70,6 +74,7 @@ impl Default for SampleRequest { multi_modal_content: None, chat_messages_json: None, prompt_list: None, + timestamp: None, } } } @@ -138,6 +143,64 @@ pub fn batch_requests( .collect() } +/// Read a JSONL file into `T` values, skipping blank lines. +/// +/// `what` names the dataset in error messages ("custom dataset", "timed +/// trace"); `max_entries` stops the read early. Returns `(line_number, value)` +/// pairs with 0-based line numbers so callers can index or report by row. +pub fn read_jsonl( + path: &str, + what: &str, + max_entries: Option, +) -> crate::error::Result> { + use crate::error::BenchError; + + let content = std::fs::read_to_string(path) + .map_err(|e| BenchError::Config(format!("Failed to read {what} '{path}': {e}")))?; + + let mut entries: Vec<(usize, T)> = Vec::new(); + for (lineno, line) in content.lines().enumerate() { + if max_entries.is_some_and(|max| entries.len() >= max) { + break; + } + let line = line.trim(); + if line.is_empty() { + continue; + } + let value = serde_json::from_str(line).map_err(|e| { + BenchError::Config(format!("Invalid JSONL at {path}:{}: {e}", lineno + 1)) + })?; + entries.push((lineno, value)); + } + + if entries.is_empty() { + return Err(BenchError::Config(format!("No entries in {what} '{path}'"))); + } + Ok(entries) +} + +/// Fixtures shared by the dataset test modules. +#[cfg(test)] +pub(crate) mod test_support { + use crate::tokenizer::TokenizerKind; + + /// gpt2 via built-in tiktoken encoding — loads without network access. + pub(crate) fn test_tokenizer() -> TokenizerKind { + TokenizerKind::Tiktoken( + crate::tiktoken::load_builtin_tiktoken("gpt2") + .expect("gpt2 built-in tiktoken should always load without network"), + ) + } + + /// Write a JSONL fixture to the temp dir. `name` must be unique across the + /// whole test suite — all fixtures share one directory. + pub(crate) fn write_temp_jsonl(name: &str, content: &str) -> String { + let path = std::env::temp_dir().join(format!("vllm-bench-{name}.jsonl")); + std::fs::write(&path, content).unwrap(); + path.to_string_lossy().into_owned() + } +} + #[cfg(test)] mod tests { use super::*; diff --git a/rust/src/bench/src/datasets/prefix_repetition.rs b/rust/src/bench/src/datasets/prefix_repetition.rs index e4dfcd36a3..7eb5fac415 100644 --- a/rust/src/bench/src/datasets/prefix_repetition.rs +++ b/rust/src/bench/src/datasets/prefix_repetition.rs @@ -107,16 +107,9 @@ pub fn generate_prefix_repetition_dataset( #[cfg(test)] mod tests { + use super::super::test_support::test_tokenizer; use super::*; - /// gpt2 via built-in tiktoken encoding — loads without network access. - fn test_tokenizer() -> TokenizerKind { - TokenizerKind::Tiktoken( - crate::tiktoken::load_builtin_tiktoken("gpt2") - .expect("gpt2 built-in tiktoken should always load without network"), - ) - } - #[test] fn test_prefix_repetition_structure() { let tok = test_tokenizer(); diff --git a/rust/src/bench/src/datasets/random_rerank.rs b/rust/src/bench/src/datasets/random_rerank.rs index ad3686cf3f..76a7eb3427 100644 --- a/rust/src/bench/src/datasets/random_rerank.rs +++ b/rust/src/bench/src/datasets/random_rerank.rs @@ -135,16 +135,9 @@ pub fn generate_random_rerank_dataset( #[cfg(test)] mod tests { + use super::super::test_support::test_tokenizer; use super::*; - /// gpt2 via built-in tiktoken encoding — loads without network access. - fn test_tokenizer() -> TokenizerKind { - TokenizerKind::Tiktoken( - crate::tiktoken::load_builtin_tiktoken("gpt2") - .expect("gpt2 built-in tiktoken should always load without network"), - ) - } - fn fixed_ratio() -> RangeRatio { RangeRatio::parse("0.0").unwrap() } diff --git a/rust/src/bench/src/datasets/sonnet.rs b/rust/src/bench/src/datasets/sonnet.rs deleted file mode 100644 index 6fc7a75c0b..0000000000 --- a/rust/src/bench/src/datasets/sonnet.rs +++ /dev/null @@ -1,131 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright contributors to the vLLM project - -use std::sync::Arc; - -use rand::SeedableRng; -use rand::rngs::StdRng; -use rand::seq::IndexedRandom; - -use super::SampleRequest; -use crate::error::{BenchError, Result}; -use crate::tokenizer::TokenizerKind; - -/// Default values mirror Python's SonnetDataset defaults (datasets.py). -pub const DEFAULT_PREFIX_LEN: usize = 200; -pub const DEFAULT_INPUT_LEN: usize = 550; -pub const DEFAULT_OUTPUT_LEN: usize = 150; - -const BASE_PROMPT: &str = "Pick as many lines as you can from these poem lines:\n"; - -/// Shakespeare's sonnets, public domain. Bundled so `--dataset-name sonnet` works -/// out of the box without `--dataset-path`. Source: -/// https://raw.githubusercontent.com/vllm-project/vllm/main/benchmarks/sonnet.txt -const BUILTIN_SONNET: &str = include_str!("sonnet.txt"); - -/// Load the sonnet dataset and generate `num_requests` prompts targeting `input_len` -/// total prompt tokens. -/// -/// Mirrors Python's `SonnetDataset.sample()` from `vllm/benchmarks/datasets/datasets.py`. -/// The Rust port skips `apply_chat_template` (no Jinja runtime here): `base_offset` -/// and `prompt_len` are computed from the raw text. The resulting prompts are slightly -/// shorter than Python's chat-template-formatted version (off by the chat scaffolding -/// tokens, typically <20). -pub fn load_sonnet_dataset( - tokenizer: &TokenizerKind, - dataset_path: Option<&str>, - num_requests: usize, - input_len: usize, - output_len: usize, - prefix_len: usize, - seed: u64, - request_id_prefix: &str, -) -> Result> { - let content = match dataset_path { - Some(path) => std::fs::read_to_string(path) - .map_err(|e| BenchError::Config(format!("Failed to read sonnet file '{path}': {e}")))?, - None => BUILTIN_SONNET.to_string(), - }; - - // Match Python's f.readlines(): keep trailing newlines so that joining lines - // reconstructs the original text without inserting extra separators. - let lines: Vec = content.split_inclusive('\n').map(|s| s.to_string()).collect(); - - if lines.is_empty() { - let src = dataset_path.unwrap_or(""); - return Err(BenchError::Config(format!("Sonnet file '{src}' is empty"))); - } - - // Average tokens per line (used to estimate how many lines to draw). - let mut total_tokens: usize = 0; - for line in &lines { - let ids = tokenizer.encode(line, false)?; - total_tokens += ids.len(); - } - let avg_len = total_tokens as f64 / lines.len() as f64; - if avg_len <= 0.0 { - return Err(BenchError::Config( - "Sonnet lines tokenized to zero tokens on average".into(), - )); - } - - let base_ids = tokenizer.encode(BASE_PROMPT, false)?; - let base_offset = base_ids.len(); - if input_len <= base_offset { - return Err(BenchError::Config(format!( - "--sonnet-input-len ({input_len}) must be larger than the base prompt length ({base_offset})" - ))); - } - - let num_input_lines = ((input_len - base_offset) as f64 / avg_len).round() as i64; - let num_prefix_lines = - (((prefix_len as i64 - base_offset as i64) as f64) / avg_len).round().max(0.0) as i64; - let num_input_lines = num_input_lines.max(0) as usize; - let num_prefix_lines = (num_prefix_lines as usize).min(lines.len()); - let num_input_lines = num_input_lines.max(num_prefix_lines); - - let prefix_lines: &[String] = &lines[..num_prefix_lines]; - let extras_per_request = num_input_lines - num_prefix_lines; - - let mut rng = StdRng::seed_from_u64(seed); - let mut samples = Vec::with_capacity(num_requests); - let mut ind = 0usize; - let mut attempts = 0usize; - let max_attempts = num_requests.saturating_mul(20).max(1000); - - while samples.len() < num_requests { - if attempts >= max_attempts { - return Err(BenchError::Config(format!( - "Could not assemble {num_requests} sonnet prompts under input_len={input_len} \ - after {attempts} attempts. Try increasing --sonnet-input-len." - ))); - } - attempts += 1; - - let mut prompt = String::with_capacity(BASE_PROMPT.len() + 256 * num_input_lines); - prompt.push_str(BASE_PROMPT); - for line in prefix_lines { - prompt.push_str(line); - } - for _ in 0..extras_per_request { - // random.choices with replacement — duplicates are allowed. - let line = lines.choose(&mut rng).unwrap(); - prompt.push_str(line); - } - - let prompt_ids = tokenizer.encode(&prompt, false)?; - let prompt_len = prompt_ids.len(); - if prompt_len <= input_len { - samples.push(SampleRequest { - prompt: Arc::from(prompt), - prompt_len, - expected_output_len: output_len, - request_id: Some(format!("{request_id_prefix}{ind}")), - ..Default::default() - }); - ind += 1; - } - } - - Ok(samples) -} diff --git a/rust/src/bench/src/datasets/sonnet.txt b/rust/src/bench/src/datasets/sonnet.txt deleted file mode 100644 index 34c444e8ce..0000000000 --- a/rust/src/bench/src/datasets/sonnet.txt +++ /dev/null @@ -1,518 +0,0 @@ -FROM fairest creatures we desire increase, -That thereby beauty's rose might never die, -But as the riper should by time decease, -His tender heir might bear his memory: -But thou, contracted to thine own bright eyes, -Feed'st thy light'st flame with self-substantial fuel, -Making a famine where abundance lies, -Thyself thy foe, to thy sweet self too cruel. -Thou that art now the world's fresh ornament -And only herald to the gaudy spring, -Within thine own bud buriest thy content -And, tender churl, makest waste in niggarding. -Pity the world, or else this glutton be, -To eat the world's due, by the grave and thee. -When forty winters shall beseige thy brow, -And dig deep trenches in thy beauty's field, -Thy youth's proud livery, so gazed on now, -Will be a tatter'd weed, of small worth held: -Then being ask'd where all thy beauty lies, -Where all the treasure of thy lusty days, -To say, within thine own deep-sunken eyes, -Were an all-eating shame and thriftless praise. -How much more praise deserved thy beauty's use, -If thou couldst answer 'This fair child of mine -Shall sum my count and make my old excuse,' -Proving his beauty by succession thine! -This were to be new made when thou art old, -And see thy blood warm when thou feel'st it cold. -Look in thy glass, and tell the face thou viewest -Now is the time that face should form another; -Whose fresh repair if now thou not renewest, -Thou dost beguile the world, unbless some mother. -For where is she so fair whose unear'd womb -Disdains the tillage of thy husbandry? -Or who is he so fond will be the tomb -Of his self-love, to stop posterity? -Thou art thy mother's glass, and she in thee -Calls back the lovely April of her prime: -So thou through windows of thine age shall see -Despite of wrinkles this thy golden time. -But if thou live, remember'd not to be, -Die single, and thine image dies with thee. -Unthrifty loveliness, why dost thou spend -Upon thyself thy beauty's legacy? -Nature's bequest gives nothing but doth lend, -And being frank she lends to those are free. -Then, beauteous niggard, why dost thou abuse -The bounteous largess given thee to give? -Profitless usurer, why dost thou use -So great a sum of sums, yet canst not live? -For having traffic with thyself alone, -Thou of thyself thy sweet self dost deceive. -Then how, when nature calls thee to be gone, -What acceptable audit canst thou leave? -Thy unused beauty must be tomb'd with thee, -Which, used, lives th' executor to be. -Those hours, that with gentle work did frame -The lovely gaze where every eye doth dwell, -Will play the tyrants to the very same -And that unfair which fairly doth excel: -For never-resting time leads summer on -To hideous winter and confounds him there; -Sap cheque'd with frost and lusty leaves quite gone, -Beauty o'ersnow'd and bareness every where: -Then, were not summer's distillation left, -A liquid prisoner pent in walls of glass, -Beauty's effect with beauty were bereft, -Nor it nor no remembrance what it was: -But flowers distill'd though they with winter meet, -Leese but their show; their substance still lives sweet. -Then let not winter's ragged hand deface -In thee thy summer, ere thou be distill'd: -Make sweet some vial; treasure thou some place -With beauty's treasure, ere it be self-kill'd. -That use is not forbidden usury, -Which happies those that pay the willing loan; -That's for thyself to breed another thee, -Or ten times happier, be it ten for one; -Ten times thyself were happier than thou art, -If ten of thine ten times refigured thee: -Then what could death do, if thou shouldst depart, -Leaving thee living in posterity? -Be not self-will'd, for thou art much too fair -To be death's conquest and make worms thine heir. -Lo! in the orient when the gracious light -Lifts up his burning head, each under eye -Doth homage to his new-appearing sight, -Serving with looks his sacred majesty; -And having climb'd the steep-up heavenly hill, -Resembling strong youth in his middle age, -yet mortal looks adore his beauty still, -Attending on his golden pilgrimage; -But when from highmost pitch, with weary car, -Like feeble age, he reeleth from the day, -The eyes, 'fore duteous, now converted are -From his low tract and look another way: -So thou, thyself out-going in thy noon, -Unlook'd on diest, unless thou get a son. -Music to hear, why hear'st thou music sadly? -Sweets with sweets war not, joy delights in joy. -Why lovest thou that which thou receivest not gladly, -Or else receivest with pleasure thine annoy? -If the true concord of well-tuned sounds, -By unions married, do offend thine ear, -They do but sweetly chide thee, who confounds -In singleness the parts that thou shouldst bear. -Mark how one string, sweet husband to another, -Strikes each in each by mutual ordering, -Resembling sire and child and happy mother -Who all in one, one pleasing note do sing: -Whose speechless song, being many, seeming one, -Sings this to thee: 'thou single wilt prove none.' -Is it for fear to wet a widow's eye -That thou consumest thyself in single life? -Ah! if thou issueless shalt hap to die. -The world will wail thee, like a makeless wife; -The world will be thy widow and still weep -That thou no form of thee hast left behind, -When every private widow well may keep -By children's eyes her husband's shape in mind. -Look, what an unthrift in the world doth spend -Shifts but his place, for still the world enjoys it; -But beauty's waste hath in the world an end, -And kept unused, the user so destroys it. -No love toward others in that bosom sits -That on himself such murderous shame commits. -For shame! deny that thou bear'st love to any, -Who for thyself art so unprovident. -Grant, if thou wilt, thou art beloved of many, -But that thou none lovest is most evident; -For thou art so possess'd with murderous hate -That 'gainst thyself thou stick'st not to conspire. -Seeking that beauteous roof to ruinate -Which to repair should be thy chief desire. -O, change thy thought, that I may change my mind! -Shall hate be fairer lodged than gentle love? -Be, as thy presence is, gracious and kind, -Or to thyself at least kind-hearted prove: -Make thee another self, for love of me, -That beauty still may live in thine or thee. -As fast as thou shalt wane, so fast thou growest -In one of thine, from that which thou departest; -And that fresh blood which youngly thou bestowest -Thou mayst call thine when thou from youth convertest. -Herein lives wisdom, beauty and increase: -Without this, folly, age and cold decay: -If all were minded so, the times should cease -And threescore year would make the world away. -Let those whom Nature hath not made for store, -Harsh featureless and rude, barrenly perish: -Look, whom she best endow'd she gave the more; -Which bounteous gift thou shouldst in bounty cherish: -She carved thee for her seal, and meant thereby -Thou shouldst print more, not let that copy die. -When I do count the clock that tells the time, -And see the brave day sunk in hideous night; -When I behold the violet past prime, -And sable curls all silver'd o'er with white; -When lofty trees I see barren of leaves -Which erst from heat did canopy the herd, -And summer's green all girded up in sheaves -Borne on the bier with white and bristly beard, -Then of thy beauty do I question make, -That thou among the wastes of time must go, -Since sweets and beauties do themselves forsake -And die as fast as they see others grow; -And nothing 'gainst Time's scythe can make defence -Save breed, to brave him when he takes thee hence. -O, that you were yourself! but, love, you are -No longer yours than you yourself here live: -Against this coming end you should prepare, -And your sweet semblance to some other give. -So should that beauty which you hold in lease -Find no determination: then you were -Yourself again after yourself's decease, -When your sweet issue your sweet form should bear. -Who lets so fair a house fall to decay, -Which husbandry in honour might uphold -Against the stormy gusts of winter's day -And barren rage of death's eternal cold? -O, none but unthrifts! Dear my love, you know -You had a father: let your son say so. -Not from the stars do I my judgment pluck; -And yet methinks I have astronomy, -But not to tell of good or evil luck, -Of plagues, of dearths, or seasons' quality; -Nor can I fortune to brief minutes tell, -Pointing to each his thunder, rain and wind, -Or say with princes if it shall go well, -By oft predict that I in heaven find: -But from thine eyes my knowledge I derive, -And, constant stars, in them I read such art -As truth and beauty shall together thrive, -If from thyself to store thou wouldst convert; -Or else of thee this I prognosticate: -Thy end is truth's and beauty's doom and date. -When I consider every thing that grows -Holds in perfection but a little moment, -That this huge stage presenteth nought but shows -Whereon the stars in secret influence comment; -When I perceive that men as plants increase, -Cheered and cheque'd even by the self-same sky, -Vaunt in their youthful sap, at height decrease, -And wear their brave state out of memory; -Then the conceit of this inconstant stay -Sets you most rich in youth before my sight, -Where wasteful Time debateth with Decay, -To change your day of youth to sullied night; -And all in war with Time for love of you, -As he takes from you, I engraft you new. -But wherefore do not you a mightier way -Make war upon this bloody tyrant, Time? -And fortify yourself in your decay -With means more blessed than my barren rhyme? -Now stand you on the top of happy hours, -And many maiden gardens yet unset -With virtuous wish would bear your living flowers, -Much liker than your painted counterfeit: -So should the lines of life that life repair, -Which this, Time's pencil, or my pupil pen, -Neither in inward worth nor outward fair, -Can make you live yourself in eyes of men. -To give away yourself keeps yourself still, -And you must live, drawn by your own sweet skill. -Who will believe my verse in time to come, -If it were fill'd with your most high deserts? -Though yet, heaven knows, it is but as a tomb -Which hides your life and shows not half your parts. -If I could write the beauty of your eyes -And in fresh numbers number all your graces, -The age to come would say 'This poet lies: -Such heavenly touches ne'er touch'd earthly faces.' -So should my papers yellow'd with their age -Be scorn'd like old men of less truth than tongue, -And your true rights be term'd a poet's rage -And stretched metre of an antique song: -But were some child of yours alive that time, -You should live twice; in it and in my rhyme. -Shall I compare thee to a summer's day? -Thou art more lovely and more temperate: -Rough winds do shake the darling buds of May, -And summer's lease hath all too short a date: -Sometime too hot the eye of heaven shines, -And often is his gold complexion dimm'd; -And every fair from fair sometime declines, -By chance or nature's changing course untrimm'd; -But thy eternal summer shall not fade -Nor lose possession of that fair thou owest; -Nor shall Death brag thou wander'st in his shade, -When in eternal lines to time thou growest: -So long as men can breathe or eyes can see, -So long lives this and this gives life to thee. -Devouring Time, blunt thou the lion's paws, -And make the earth devour her own sweet brood; -Pluck the keen teeth from the fierce tiger's jaws, -And burn the long-lived phoenix in her blood; -Make glad and sorry seasons as thou fleets, -And do whate'er thou wilt, swift-footed Time, -To the wide world and all her fading sweets; -But I forbid thee one most heinous crime: -O, carve not with thy hours my love's fair brow, -Nor draw no lines there with thine antique pen; -Him in thy course untainted do allow -For beauty's pattern to succeeding men. -Yet, do thy worst, old Time: despite thy wrong, -My love shall in my verse ever live young. -A woman's face with Nature's own hand painted -Hast thou, the master-mistress of my passion; -A woman's gentle heart, but not acquainted -With shifting change, as is false women's fashion; -An eye more bright than theirs, less false in rolling, -Gilding the object whereupon it gazeth; -A man in hue, all 'hues' in his controlling, -Much steals men's eyes and women's souls amazeth. -And for a woman wert thou first created; -Till Nature, as she wrought thee, fell a-doting, -And by addition me of thee defeated, -By adding one thing to my purpose nothing. -But since she prick'd thee out for women's pleasure, -Mine be thy love and thy love's use their treasure. -So is it not with me as with that Muse -Stirr'd by a painted beauty to his verse, -Who heaven itself for ornament doth use -And every fair with his fair doth rehearse -Making a couplement of proud compare, -With sun and moon, with earth and sea's rich gems, -With April's first-born flowers, and all things rare -That heaven's air in this huge rondure hems. -O' let me, true in love, but truly write, -And then believe me, my love is as fair -As any mother's child, though not so bright -As those gold candles fix'd in heaven's air: -Let them say more than like of hearsay well; -I will not praise that purpose not to sell. -My glass shall not persuade me I am old, -So long as youth and thou are of one date; -But when in thee time's furrows I behold, -Then look I death my days should expiate. -For all that beauty that doth cover thee -Is but the seemly raiment of my heart, -Which in thy breast doth live, as thine in me: -How can I then be elder than thou art? -O, therefore, love, be of thyself so wary -As I, not for myself, but for thee will; -Bearing thy heart, which I will keep so chary -As tender nurse her babe from faring ill. -Presume not on thy heart when mine is slain; -Thou gavest me thine, not to give back again. -As an unperfect actor on the stage -Who with his fear is put besides his part, -Or some fierce thing replete with too much rage, -Whose strength's abundance weakens his own heart. -So I, for fear of trust, forget to say -The perfect ceremony of love's rite, -And in mine own love's strength seem to decay, -O'ercharged with burden of mine own love's might. -O, let my books be then the eloquence -And dumb presagers of my speaking breast, -Who plead for love and look for recompense -More than that tongue that more hath more express'd. -O, learn to read what silent love hath writ: -To hear with eyes belongs to love's fine wit. -Mine eye hath play'd the painter and hath stell'd -Thy beauty's form in table of my heart; -My body is the frame wherein 'tis held, -And perspective it is the painter's art. -For through the painter must you see his skill, -To find where your true image pictured lies; -Which in my bosom's shop is hanging still, -That hath his windows glazed with thine eyes. -Now see what good turns eyes for eyes have done: -Mine eyes have drawn thy shape, and thine for me -Are windows to my breast, where-through the sun -Delights to peep, to gaze therein on thee; -Yet eyes this cunning want to grace their art; -They draw but what they see, know not the heart. -Let those who are in favour with their stars -Of public honour and proud titles boast, -Whilst I, whom fortune of such triumph bars, -Unlook'd for joy in that I honour most. -Great princes' favourites their fair leaves spread -But as the marigold at the sun's eye, -And in themselves their pride lies buried, -For at a frown they in their glory die. -The painful warrior famoused for fight, -After a thousand victories once foil'd, -Is from the book of honour razed quite, -And all the rest forgot for which he toil'd: -Then happy I, that love and am beloved -Where I may not remove nor be removed. -Lord of my love, to whom in vassalage -Thy merit hath my duty strongly knit, -To thee I send this written embassage, -To witness duty, not to show my wit: -Duty so great, which wit so poor as mine -May make seem bare, in wanting words to show it, -But that I hope some good conceit of thine -In thy soul's thought, all naked, will bestow it; -Till whatsoever star that guides my moving -Points on me graciously with fair aspect -And puts apparel on my tatter'd loving, -To show me worthy of thy sweet respect: -Then may I dare to boast how I do love thee; -Till then not show my head where thou mayst prove me. -Weary with toil, I haste me to my bed, -The dear repose for limbs with travel tired; -But then begins a journey in my head, -To work my mind, when body's work's expired: -For then my thoughts, from far where I abide, -Intend a zealous pilgrimage to thee, -And keep my drooping eyelids open wide, -Looking on darkness which the blind do see -Save that my soul's imaginary sight -Presents thy shadow to my sightless view, -Which, like a jewel hung in ghastly night, -Makes black night beauteous and her old face new. -Lo! thus, by day my limbs, by night my mind, -For thee and for myself no quiet find. -How can I then return in happy plight, -That am debarr'd the benefit of rest? -When day's oppression is not eased by night, -But day by night, and night by day, oppress'd? -And each, though enemies to either's reign, -Do in consent shake hands to torture me; -The one by toil, the other to complain -How far I toil, still farther off from thee. -I tell the day, to please them thou art bright -And dost him grace when clouds do blot the heaven: -So flatter I the swart-complexion'd night, -When sparkling stars twire not thou gild'st the even. -But day doth daily draw my sorrows longer -And night doth nightly make grief's strength seem stronger. -When, in disgrace with fortune and men's eyes, -I all alone beweep my outcast state -And trouble deal heaven with my bootless cries -And look upon myself and curse my fate, -Wishing me like to one more rich in hope, -Featured like him, like him with friends possess'd, -Desiring this man's art and that man's scope, -With what I most enjoy contented least; -Yet in these thoughts myself almost despising, -Haply I think on thee, and then my state, -Like to the lark at break of day arising -From sullen earth, sings hymns at heaven's gate; -For thy sweet love remember'd such wealth brings -That then I scorn to change my state with kings. -When to the sessions of sweet silent thought -I summon up remembrance of things past, -I sigh the lack of many a thing I sought, -And with old woes new wail my dear time's waste: -Then can I drown an eye, unused to flow, -For precious friends hid in death's dateless night, -And weep afresh love's long since cancell'd woe, -And moan the expense of many a vanish'd sight: -Then can I grieve at grievances foregone, -And heavily from woe to woe tell o'er -The sad account of fore-bemoaned moan, -Which I new pay as if not paid before. -But if the while I think on thee, dear friend, -All losses are restored and sorrows end. -Thy bosom is endeared with all hearts, -Which I by lacking have supposed dead, -And there reigns love and all love's loving parts, -And all those friends which I thought buried. -How many a holy and obsequious tear -Hath dear religious love stol'n from mine eye -As interest of the dead, which now appear -But things removed that hidden in thee lie! -Thou art the grave where buried love doth live, -Hung with the trophies of my lovers gone, -Who all their parts of me to thee did give; -That due of many now is thine alone: -Their images I loved I view in thee, -And thou, all they, hast all the all of me. -If thou survive my well-contented day, -When that churl Death my bones with dust shall cover, -And shalt by fortune once more re-survey -These poor rude lines of thy deceased lover, -Compare them with the bettering of the time, -And though they be outstripp'd by every pen, -Reserve them for my love, not for their rhyme, -Exceeded by the height of happier men. -O, then vouchsafe me but this loving thought: -'Had my friend's Muse grown with this growing age, -A dearer birth than this his love had brought, -To march in ranks of better equipage: -But since he died and poets better prove, -Theirs for their style I'll read, his for his love.' -Full many a glorious morning have I seen -Flatter the mountain-tops with sovereign eye, -Kissing with golden face the meadows green, -Gilding pale streams with heavenly alchemy; -Anon permit the basest clouds to ride -With ugly rack on his celestial face, -And from the forlorn world his visage hide, -Stealing unseen to west with this disgrace: -Even so my sun one early morn did shine -With all triumphant splendor on my brow; -But out, alack! he was but one hour mine; -The region cloud hath mask'd him from me now. -Yet him for this my love no whit disdaineth; -Suns of the world may stain when heaven's sun staineth. -Why didst thou promise such a beauteous day, -And make me travel forth without my cloak, -To let base clouds o'ertake me in my way, -Hiding thy bravery in their rotten smoke? -'Tis not enough that through the cloud thou break, -To dry the rain on my storm-beaten face, -For no man well of such a salve can speak -That heals the wound and cures not the disgrace: -Nor can thy shame give physic to my grief; -Though thou repent, yet I have still the loss: -The offender's sorrow lends but weak relief -To him that bears the strong offence's cross. -Ah! but those tears are pearl which thy love sheds, -And they are rich and ransom all ill deeds. -No more be grieved at that which thou hast done: -Roses have thorns, and silver fountains mud; -Clouds and eclipses stain both moon and sun, -And loathsome canker lives in sweetest bud. -All men make faults, and even I in this, -Authorizing thy trespass with compare, -Myself corrupting, salving thy amiss, -Excusing thy sins more than thy sins are; -For to thy sensual fault I bring in sense-- -Thy adverse party is thy advocate-- -And 'gainst myself a lawful plea commence: -Such civil war is in my love and hate -That I an accessary needs must be -To that sweet thief which sourly robs from me. -Let me confess that we two must be twain, -Although our undivided loves are one: -So shall those blots that do with me remain -Without thy help by me be borne alone. -In our two loves there is but one respect, -Though in our lives a separable spite, -Which though it alter not love's sole effect, -Yet doth it steal sweet hours from love's delight. -I may not evermore acknowledge thee, -Lest my bewailed guilt should do thee shame, -Nor thou with public kindness honour me, -Unless thou take that honour from thy name: -But do not so; I love thee in such sort -As, thou being mine, mine is thy good report. -As a decrepit father takes delight -To see his active child do deeds of youth, -So I, made lame by fortune's dearest spite, -Take all my comfort of thy worth and truth. -For whether beauty, birth, or wealth, or wit, -Or any of these all, or all, or more, -Entitled in thy parts do crowned sit, -I make my love engrafted to this store: -So then I am not lame, poor, nor despised, -Whilst that this shadow doth such substance give -That I in thy abundance am sufficed -And by a part of all thy glory live. -Look, what is best, that best I wish in thee: -This wish I have; then ten times happy me! \ No newline at end of file diff --git a/rust/src/bench/src/datasets/timed_trace.rs b/rust/src/bench/src/datasets/timed_trace.rs new file mode 100644 index 0000000000..f90a94450a --- /dev/null +++ b/rust/src/bench/src/datasets/timed_trace.rs @@ -0,0 +1,376 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +//! Timed-trace dataset: JSONL replay trace with per-request arrival timestamps. +//! +//! ```jsonl +//! {"timestamp": 12.4, "input_length": 42, "output_length": 8, "hash_ids": [981, 4412, 7]} +//! ``` +//! +//! Mirrors Python's `TimedTrace`. Prompts carry no text: each `hash_id` expands +//! to a pseudo-random token chunk of `chunk_size` tokens (the last chunk may be +//! partial so the prompt length equals `input_length` exactly). Chunks are +//! memoized per (hash_id, size), so requests sharing a hash prefix get +//! byte-identical token prefixes and prefix-cache hits reproduce. Field names +//! are remappable via `--timed-trace-label-*`; timestamps are scaled by +//! `--timed-trace-sec-multiplier`. +//! +//! Deviation from Python: chunk contents are seeded from (hash_id, size) with a +//! fixed mixing function, so they are bit-reproducible across runs. Python +//! seeds via `hash(f"{h}:{size}")`, which is randomized per process unless +//! PYTHONHASHSEED is pinned. + +use std::collections::HashMap; +use std::sync::Arc; + +use rand::rngs::StdRng; +use rand::{Rng, SeedableRng}; +use serde_json::Value; + +use super::SampleRequest; +use crate::error::{BenchError, Result}; +use crate::tokenizer::TokenizerKind; + +pub struct TimedTraceOptions<'a> { + pub chunk_size: usize, + pub sec_multiplier: f64, + pub label_timestamp: &'a str, + pub label_input_length: &'a str, + pub label_output_length: &'a str, + pub label_hash_ids: &'a str, +} + +/// Load a timed-trace JSONL dataset. Row order (and thus the timestamp +/// schedule) is preserved: no shuffle, no oversampling. +pub fn load_timed_trace_dataset( + tokenizer: &TokenizerKind, + path: &str, + num_requests: usize, + request_id_prefix: &str, + opts: &TimedTraceOptions<'_>, +) -> Result> { + let allowed_tokens = tokenizer.get_allowed_tokens(); + if allowed_tokens.is_empty() { + return Err(BenchError::Tokenizer("No allowed tokens found".into())); + } + + let entries: Vec<(usize, Value)> = super::read_jsonl(path, "timed trace", Some(num_requests))?; + + let mut chunk_cache: HashMap<(u64, usize), Arc<[u32]>> = HashMap::new(); + let mut requests: Vec = Vec::with_capacity(entries.len()); + + for (lineno, entry) in &entries { + let input_length = require_u64( + entry, + opts.label_input_length, + "--timed-trace-label-input-length", + )? as usize; + let output_length = require_u64( + entry, + opts.label_output_length, + "--timed-trace-label-output-length", + )? as usize; + let timestamp = require_f64(entry, opts.label_timestamp, "--timed-trace-label-timestamp")? + * opts.sec_multiplier; + // A non-finite offset panics in `Duration::from_secs_f64` once the + // scheduler consumes it; reject here, naming the offending row. + if !timestamp.is_finite() { + return Err(BenchError::Config(format!( + "Field '{}' at {path}:{} scales to a non-finite offset ({timestamp}); \ + check --timed-trace-sec-multiplier", + opts.label_timestamp, + lineno + 1, + ))); + } + let hash_ids = parse_hash_ids(entry, opts.label_hash_ids)?; + + let prompt_ids = expand_prompt( + &hash_ids, + input_length, + opts.chunk_size, + &allowed_tokens, + &mut chunk_cache, + ); + + requests.push(SampleRequest { + prompt_len: prompt_ids.len(), + prompt_token_ids: Some(Arc::from(prompt_ids)), + expected_output_len: output_length, + request_id: Some(format!("{request_id_prefix}{lineno}")), + timestamp: Some(timestamp), + ..Default::default() + }); + } + + Ok(requests) +} + +/// Expand prefix-hash ids into token ids, consuming `chunk_size` tokens per +/// hash until `target_len` is reached (the final chunk may be partial). A +/// repeated (hash_id, size) yields the identical chunk via the cache. If the +/// trace supplies fewer hashes than `target_len / chunk_size`, the prompt +/// comes out shorter — same as Python. +fn expand_prompt( + hash_ids: &[u64], + target_len: usize, + chunk_size: usize, + allowed_tokens: &[u32], + cache: &mut HashMap<(u64, usize), Arc<[u32]>>, +) -> Vec { + // `target_len` is raw trace input, so reserve only what the hashes can + // actually produce: a corrupt row, or one in the wrong unit, would + // otherwise ask the allocator for terabytes before any length filter runs. + let capacity = target_len.min(hash_ids.len().saturating_mul(chunk_size)); + let mut prompt = Vec::with_capacity(capacity); + let mut remaining = target_len; + for &h in hash_ids { + let size = remaining.min(chunk_size); + let chunk = cache.entry((h, size)).or_insert_with(|| { + let mut rng = StdRng::seed_from_u64(chunk_seed(h, size)); + (0..size) + .map(|_| allowed_tokens[rng.random_range(0..allowed_tokens.len())]) + .collect() + }); + prompt.extend_from_slice(chunk); + remaining -= size; + if remaining == 0 { + break; + } + } + prompt +} + +/// Stable seed for a chunk. The golden-ratio multiply is bijective on u64, so +/// distinct `hash_id`s never collide at a fixed `size`. The trailing xor is not +/// injective across sizes: two unrelated hashes seen at different sizes can +/// alias and share a token prefix, at ~2^-64 odds per pair. +fn chunk_seed(hash_id: u64, size: usize) -> u64 { + hash_id.wrapping_mul(0x9E37_79B9_7F4A_7C15) ^ (size as u64) +} + +fn require_field<'v>(entry: &'v Value, label: &str, flag: &str) -> Result<&'v Value> { + entry.get(label).ok_or_else(|| { + let available: Vec<&str> = entry + .as_object() + .map(|o| o.keys().map(String::as_str).collect()) + .unwrap_or_default(); + BenchError::Config(format!( + "Field '{label}' not found in trace entry. Available fields: \ + {available:?}. Use {flag} to specify the correct field name." + )) + }) +} + +/// Float-encoded integers (`512.0`) are accepted, matching Python's `int()`. +fn require_u64(entry: &Value, label: &str, flag: &str) -> Result { + let v = require_field(entry, label, flag)?; + v.as_u64() + .or_else(|| { + v.as_f64() + .filter(|f| f.fract() == 0.0 && (0.0..u64::MAX as f64).contains(f)) + .map(|f| f as u64) + }) + .ok_or_else(|| { + BenchError::Config(format!( + "Field '{label}' must be a non-negative integer, got: {v}" + )) + }) +} + +fn require_f64(entry: &Value, label: &str, flag: &str) -> Result { + let v = require_field(entry, label, flag)?; + v.as_f64() + .ok_or_else(|| BenchError::Config(format!("Field '{label}' must be a number, got: {v}"))) +} + +/// Missing hash_ids field means an empty prompt (mirrors Python's +/// `entry.get(label, [])`); a present field must be an array of integers. +/// +/// Ids are kept as `u64` because traces carry raw unsigned 64-bit block hashes; +/// negative ids keep the two's-complement value `chunk_seed` used before. +fn parse_hash_ids(entry: &Value, label: &str) -> Result> { + let Some(v) = entry.get(label) else { + return Ok(Vec::new()); + }; + let arr = v + .as_array() + .ok_or_else(|| BenchError::Config(format!("Field '{label}' must be an array, got: {v}")))?; + arr.iter() + .map(|x| { + x.as_u64().or_else(|| x.as_i64().map(|i| i as u64)).ok_or_else(|| { + BenchError::Config(format!( + "Field '{label}' must contain only integers, got: {x}" + )) + }) + }) + .collect() +} + +#[cfg(test)] +mod tests { + use super::super::test_support::{test_tokenizer, write_temp_jsonl}; + use super::*; + + fn opts(chunk_size: usize, sec_multiplier: f64) -> TimedTraceOptions<'static> { + TimedTraceOptions { + chunk_size, + sec_multiplier, + label_timestamp: "timestamp", + label_input_length: "input_length", + label_output_length: "output_length", + label_hash_ids: "hash_ids", + } + } + + #[test] + fn test_prompt_len_matches_input_length_and_fields_map() { + let path = write_temp_jsonl( + "timed-trace-basic", + r#"{"timestamp": 1000, "input_length": 40, "output_length": 7, "hash_ids": [1, 2, 3]} +{"timestamp": 2500, "input_length": 32, "output_length": 9, "hash_ids": [1, 4]} +"#, + ); + let reqs = load_timed_trace_dataset(&test_tokenizer(), &path, 10, "t-", &opts(16, 0.001)) + .expect("load should succeed"); + assert_eq!(reqs.len(), 2); + // 40 = 16 + 16 + partial 8 from three hashes + assert_eq!(reqs[0].prompt_len, 40); + assert_eq!(reqs[0].prompt_token_ids.as_ref().unwrap().len(), 40); + assert_eq!(reqs[0].expected_output_len, 7); + assert_eq!(reqs[0].timestamp, Some(1.0)); + assert_eq!(reqs[1].timestamp, Some(2.5)); + assert_eq!(reqs[0].request_id.as_deref(), Some("t-0")); + } + + #[test] + fn test_shared_hash_prefix_gives_identical_tokens() { + let path = write_temp_jsonl( + "timed-trace-prefix", + r#"{"timestamp": 0, "input_length": 32, "output_length": 1, "hash_ids": [7, 8]} +{"timestamp": 1, "input_length": 32, "output_length": 1, "hash_ids": [7, 9]} +"#, + ); + let reqs = load_timed_trace_dataset(&test_tokenizer(), &path, 10, "t-", &opts(16, 1.0)) + .expect("load should succeed"); + let a = reqs[0].prompt_token_ids.as_ref().unwrap(); + let b = reqs[1].prompt_token_ids.as_ref().unwrap(); + assert_eq!(a[..16], b[..16], "shared hash 7 must expand identically"); + assert_ne!(a[16..], b[16..], "distinct hashes 8/9 must differ"); + } + + #[test] + fn test_reload_is_bit_identical() { + let path = write_temp_jsonl( + "timed-trace-determinism", + r#"{"timestamp": 0, "input_length": 40, "output_length": 1, "hash_ids": [11, 12, 13]}"#, + ); + let load = || { + load_timed_trace_dataset(&test_tokenizer(), &path, 10, "t-", &opts(16, 1.0)) + .expect("load should succeed") + }; + assert_eq!(load()[0].prompt_token_ids, load()[0].prompt_token_ids); + } + + #[test] + fn test_too_few_hashes_yields_short_prompt() { + let path = write_temp_jsonl( + "timed-trace-short", + r#"{"timestamp": 0, "input_length": 100, "output_length": 1, "hash_ids": [1]}"#, + ); + let reqs = load_timed_trace_dataset(&test_tokenizer(), &path, 10, "t-", &opts(16, 1.0)) + .expect("load should succeed"); + assert_eq!(reqs[0].prompt_len, 16, "one hash covers only one chunk"); + } + + /// `input_length` is untrusted: the reservation must follow what the hashes + /// can produce, not the raw field, or the allocator aborts the process. + #[test] + fn test_absurd_input_length_does_not_overallocate() { + let path = write_temp_jsonl( + "timed-trace-absurd-len", + r#"{"timestamp": 0, "input_length": 10000000000000, "output_length": 1, "hash_ids": [1]}"#, + ); + let reqs = load_timed_trace_dataset(&test_tokenizer(), &path, 10, "t-", &opts(16, 1.0)) + .expect("load should succeed"); + assert_eq!(reqs[0].prompt_len, 16); + } + + #[test] + fn test_num_requests_caps_rows() { + let path = write_temp_jsonl( + "timed-trace-cap", + r#"{"timestamp": 0, "input_length": 8, "output_length": 1, "hash_ids": [1]} +{"timestamp": 1, "input_length": 8, "output_length": 1, "hash_ids": [2]} +{"timestamp": 2, "input_length": 8, "output_length": 1, "hash_ids": [3]} +"#, + ); + let reqs = load_timed_trace_dataset(&test_tokenizer(), &path, 2, "t-", &opts(16, 1.0)) + .expect("load should succeed"); + assert_eq!(reqs.len(), 2); + } + + #[test] + fn test_missing_field_error_names_label_and_flag() { + let path = write_temp_jsonl( + "timed-trace-missing", + r#"{"timestamp": 0, "input_length": 8, "output_length": 1, "hash_ids": [1]}"#, + ); + let mut o = opts(16, 1.0); + o.label_input_length = "input_len"; + let err = load_timed_trace_dataset(&test_tokenizer(), &path, 1, "t-", &o) + .expect_err("should fail on missing field"); + let msg = err.to_string(); + assert!(msg.contains("input_len") && msg.contains("--timed-trace-label-input-length")); + } + + /// Hashes above `i64::MAX` are ordinary 64-bit block hashes, and Python + /// takes them as unbounded ints. + #[test] + fn test_accepts_unsigned_64bit_hash_ids() { + let path = write_temp_jsonl( + "timed-trace-u64-hash", + r#"{"timestamp": 0, "input_length": 16, "output_length": 1, "hash_ids": [18446744073709551615]}"#, + ); + let reqs = load_timed_trace_dataset(&test_tokenizer(), &path, 1, "t-", &opts(16, 1.0)) + .expect("load should succeed"); + assert_eq!(reqs[0].prompt_len, 16); + } + + /// Exporters that round-trip lengths through floats emit `512.0`; Python's + /// `int()` accepts it. + #[test] + fn test_accepts_float_encoded_lengths() { + let path = write_temp_jsonl( + "timed-trace-float-len", + r#"{"timestamp": 0, "input_length": 16.0, "output_length": 4.0, "hash_ids": [1]}"#, + ); + let reqs = load_timed_trace_dataset(&test_tokenizer(), &path, 1, "t-", &opts(16, 1.0)) + .expect("load should succeed"); + assert_eq!(reqs[0].prompt_len, 16); + assert_eq!(reqs[0].expected_output_len, 4); + } + + #[test] + fn test_non_finite_scaled_timestamp_is_rejected() { + let path = write_temp_jsonl( + "timed-trace-overflow-ts", + r#"{"timestamp": 1e308, "input_length": 16, "output_length": 1, "hash_ids": [1]}"#, + ); + let err = load_timed_trace_dataset(&test_tokenizer(), &path, 1, "t-", &opts(16, 1e10)) + .expect_err("should reject an offset that overflows to inf"); + assert!(err.to_string().contains("non-finite offset")); + } + + /// Negative offsets load unchanged; the scheduler clamps them to fire + /// immediately (Python's `get_request` does the same). + #[test] + fn test_negative_timestamp_loads() { + let path = write_temp_jsonl( + "timed-trace-negative-ts", + r#"{"timestamp": -5.0, "input_length": 16, "output_length": 1, "hash_ids": [1]}"#, + ); + let reqs = load_timed_trace_dataset(&test_tokenizer(), &path, 1, "t-", &opts(16, 1.0)) + .expect("load should succeed"); + assert_eq!(reqs[0].timestamp, Some(-5.0)); + } +} diff --git a/rust/src/bench/src/multi_turn.rs b/rust/src/bench/src/multi_turn.rs index 366fd4f6fb..085684d5de 100644 --- a/rust/src/bench/src/multi_turn.rs +++ b/rust/src/bench/src/multi_turn.rs @@ -170,17 +170,15 @@ pub async fn run_multi_turn_benchmark(config: &BenchConfig) -> Result { - return Err(BenchError::Config( - "Sonnet multi-turn is not yet supported. Use 'random' or 'sharegpt' with --multi-turn.".into(), - )); - } DatasetName::SpeedBench => { return Err(BenchError::Config( "SPEED-Bench multi-turn is not yet supported. Use 'random' or 'sharegpt' with --multi-turn.".into(), )); } - DatasetName::Custom | DatasetName::PrefixRepetition | DatasetName::RandomRerank => { + DatasetName::Custom + | DatasetName::PrefixRepetition + | DatasetName::RandomRerank + | DatasetName::TimedTrace => { return Err(BenchError::Config( "This dataset does not support multi-turn. Use 'random' or 'sharegpt' with --multi-turn.".into(), )); diff --git a/rust/src/bench/src/rate_control.rs b/rust/src/bench/src/rate_control.rs index aa05f8f8a8..1311cb9354 100644 --- a/rust/src/bench/src/rate_control.rs +++ b/rust/src/bench/src/rate_control.rs @@ -104,10 +104,30 @@ pub fn compute_schedule( } } +/// Schedule built from trace-recorded arrival offsets (`--self-timed`). +/// +/// Offsets are clamped at zero so a row anchored before the trace start fires +/// immediately, as Python's `get_request` does; `Duration::from_secs_f64` in +/// the dispatch loop would otherwise panic. `rates` carries no meaning here. +pub fn trace_schedule(timestamps: impl ExactSizeIterator>) -> RequestSchedule { + let rates = vec![0.0; timestamps.len()]; + RequestSchedule { + delays: timestamps.map(|t| t.unwrap_or(0.0).max(0.0)).collect(), + rates, + } +} + #[cfg(test)] mod tests { use super::*; + #[test] + fn test_trace_schedule_uses_offsets_and_clamps_negatives() { + let sched = trace_schedule([Some(-3.0), Some(0.0), Some(2.5), None].into_iter()); + assert_eq!(sched.delays, vec![0.0, 0.0, 2.5, 0.0]); + assert_eq!(sched.rates.len(), 4); + } + #[test] fn test_infinite_rate_all_zero_delays() { let sched = compute_schedule(100, f64::INFINITY, 1.0, 42, None); diff --git a/rust/src/bench/tests/cli_parity.rs b/rust/src/bench/tests/cli_parity.rs index b0fd09ab9e..b0bb707aa1 100644 --- a/rust/src/bench/tests/cli_parity.rs +++ b/rust/src/bench/tests/cli_parity.rs @@ -36,13 +36,6 @@ const PYTHON_ONLY: &[&str] = &[ // spec_bench dataset "--spec-bench-category", "--spec-bench-output-len", - // timed-trace dataset - "--timed-trace-chunk-hash-size", - "--timed-trace-label-hash-ids", - "--timed-trace-label-input-length", - "--timed-trace-label-output-length", - "--timed-trace-label-timestamp", - "--timed-trace-sec-multiplier", // client-side chat templating / request shaping "--chat-template-kwargs", "--custom-ensure-client-side-data", @@ -53,8 +46,6 @@ const PYTHON_ONLY: &[&str] = &[ "--timeline-itl-thresholds", // misc python-side controls "--hf-name", - "--no-self-timed", - "--self-timed", "--no-stream", "--probe-request-rate", ]; diff --git a/rust/src/bench/tests/python_serve_flags.txt b/rust/src/bench/tests/python_serve_flags.txt index d200e5bf24..3206196178 100644 --- a/rust/src/bench/tests/python_serve_flags.txt +++ b/rust/src/bench/tests/python_serve_flags.txt @@ -86,9 +86,6 @@ --sharegpt-output-len --skip-chat-template --skip-tokenizer-init ---sonnet-input-len ---sonnet-output-len ---sonnet-prefix-len --spec-bench-category --spec-bench-output-len --speed-bench-category diff --git a/rust/src/chat/src/output/default/mod.rs b/rust/src/chat/src/output/default/mod.rs index 3f75aed63d..b31c5e9e4c 100644 --- a/rust/src/chat/src/output/default/mod.rs +++ b/rust/src/chat/src/output/default/mod.rs @@ -311,6 +311,7 @@ mod tests { "required": ["query"] }), strict, + defer_loading: None, }]; let mut request = ChatRequest { tool_context: ResolvedToolContext::new(&[], tools, Some(choice), true).unwrap(), @@ -346,6 +347,7 @@ mod tests { description: None, parameters: serde_json::json!({"type": "object"}), strict: Some(true), + defer_loading: None, }]; let mut request = ChatRequest { tool_context: ResolvedToolContext::new( diff --git a/rust/src/chat/src/output/default/unified.rs b/rust/src/chat/src/output/default/unified.rs index b4a2dcbe5d..9f872d9794 100644 --- a/rust/src/chat/src/output/default/unified.rs +++ b/rust/src/chat/src/output/default/unified.rs @@ -884,6 +884,7 @@ mod tests { description: None, parameters: serde_json::json!({ "type": "object" }), strict: None, + defer_loading: None, }]; let parser = Gemma4UnifiedParser::new(&tools, Arc::new(tokenizer)).unwrap(); let events = vec![ diff --git a/rust/src/chat/src/output/harmony/tests.rs b/rust/src/chat/src/output/harmony/tests.rs index 035aef3cd4..5b89e53bb9 100644 --- a/rust/src/chat/src/output/harmony/tests.rs +++ b/rust/src/chat/src/output/harmony/tests.rs @@ -99,6 +99,7 @@ fn request_with_tools() -> ChatRequest { "required": ["city"] }), strict: None, + defer_loading: None, }]; ChatRequest { tool_context: ResolvedToolContext::new(&[], tools, Some(ChatToolChoice::Auto), true) diff --git a/rust/src/chat/src/renderer/deepseek_v32/tests.rs b/rust/src/chat/src/renderer/deepseek_v32/tests.rs index dd9ec13ddc..804e4c2bc9 100644 --- a/rust/src/chat/src/renderer/deepseek_v32/tests.rs +++ b/rust/src/chat/src/renderer/deepseek_v32/tests.rs @@ -152,6 +152,7 @@ fn request_level_tools_are_lowered_as_synthetic_leading_system_message() { "required": ["query"] }), strict: None, + defer_loading: None, }]; let mut request = ChatRequest { request_id: "deepseek-v32-tools".to_string(), diff --git a/rust/src/chat/src/renderer/deepseek_v4/tests.rs b/rust/src/chat/src/renderer/deepseek_v4/tests.rs index 8702e8d73c..78e9044a3b 100644 --- a/rust/src/chat/src/renderer/deepseek_v4/tests.rs +++ b/rust/src/chat/src/renderer/deepseek_v4/tests.rs @@ -183,6 +183,7 @@ fn tools_keep_historical_developer() { description: None, parameters: Value::Object(Default::default()), strict: None, + defer_loading: None, }; let request = thinking_request(vec![ ChatMessage::developer("old instruction", Some(vec![tool])), diff --git a/rust/src/chat/src/renderer/hf/mod.rs b/rust/src/chat/src/renderer/hf/mod.rs index 5cb0262191..06b58e27ee 100644 --- a/rust/src/chat/src/renderer/hf/mod.rs +++ b/rust/src/chat/src/renderer/hf/mod.rs @@ -324,6 +324,8 @@ struct TemplateToolDefinition { parameters: JsonValue, #[serde(skip_serializing_if = "Option::is_none")] strict: Option, + #[serde(skip_serializing_if = "Option::is_none")] + defer_loading: Option, } /// Convert chat messages into the JSON shape expected by Jinja chat templates. @@ -604,6 +606,7 @@ fn to_template_tools(tools: &[ChatTool]) -> Vec { description: tool.description.clone(), parameters: tool.parameters.clone(), strict: tool.strict, + defer_loading: tool.defer_loading, }, }) .collect() @@ -649,6 +652,7 @@ mod tests { description: None, parameters: serde_json::json!({"type": "object"}), strict: None, + defer_loading: None, }]), ), ChatMessage::user("hello"), @@ -1020,6 +1024,7 @@ mod tests { "required": ["city"], }), strict: Some(true), + defer_loading: None, }]), )]); @@ -1419,6 +1424,7 @@ mod tests { "required": ["city"], }), strict: None, + defer_loading: None, }]; request.tool_context = crate::request::ResolvedToolContext::new( &request.messages, @@ -1445,6 +1451,7 @@ mod tests { description: Some("Get weather".to_string()), parameters: serde_json::json!({"type": "object"}), strict: None, + defer_loading: None, }]; request.tool_context = crate::request::ResolvedToolContext::new( &request.messages, @@ -1472,12 +1479,21 @@ mod tests { description: None, parameters: Value::Null, strict: None, + defer_loading: None, }, ChatTool { name: "with_strict".to_string(), description: Some("description".to_string()), parameters: serde_json::json!({"type": "object"}), strict: Some(false), + defer_loading: None, + }, + ChatTool { + name: "deferred".to_string(), + description: None, + parameters: Value::Null, + strict: None, + defer_loading: Some(true), }, ]; request.tool_context = crate::request::ResolvedToolContext::new( @@ -1498,7 +1514,7 @@ mod tests { assert_eq!( rendered, - "name=\"without_strict\"|description=null|parameters=null|;name=\"with_strict\"|description=\"description\"|parameters={\"type\": \"object\"}|strict=false|;" + "name=\"without_strict\"|description=null|parameters=null|;name=\"with_strict\"|description=\"description\"|parameters={\"type\": \"object\"}|strict=false|;name=\"deferred\"|description=null|parameters=null|defer_loading=true|;" ); } diff --git a/rust/src/chat/src/renderer/kimi_k3/tests.rs b/rust/src/chat/src/renderer/kimi_k3/tests.rs index 5aadb3d403..29a6ecb473 100644 --- a/rust/src/chat/src/renderer/kimi_k3/tests.rs +++ b/rust/src/chat/src/renderer/kimi_k3/tests.rs @@ -178,6 +178,7 @@ fn tool_declare_omits_absent_fields_and_preserves_parameter_nulls() { } }), strict: None, + defer_loading: None, }]; request.tool_context = crate::request::ResolvedToolContext::new(&request.messages, tools, None, true) @@ -200,6 +201,7 @@ fn tool_declare_preserves_present_optional_fields() { description: Some("Look up a record".to_string()), parameters: json!({"type": "object"}), strict: Some(false), + defer_loading: None, }]; request.tool_context = crate::request::ResolvedToolContext::new(&request.messages, tools, None, true) diff --git a/rust/src/chat/src/renderer/test_utils.rs b/rust/src/chat/src/renderer/test_utils.rs index 2c59ed7a22..e42dba76a5 100644 --- a/rust/src/chat/src/renderer/test_utils.rs +++ b/rust/src/chat/src/renderer/test_utils.rs @@ -280,6 +280,7 @@ fn to_chat_tools(tools: &[FixtureTool]) -> Vec { description: tool.function.description.clone(), parameters: tool.function.parameters.clone(), strict: tool.function.strict, + defer_loading: None, }) .collect() } diff --git a/rust/src/chat/src/request.rs b/rust/src/chat/src/request.rs index f7ddd9b128..b52e9fbc0b 100644 --- a/rust/src/chat/src/request.rs +++ b/rust/src/chat/src/request.rs @@ -801,6 +801,7 @@ mod tests { "properties": {"city": {"type": "string"}}, }), strict: Some(true), + defer_loading: None, }]), ); @@ -815,6 +816,7 @@ mod tests { description: None, parameters: json!({"type": "object"}), strict: None, + defer_loading: None, } } diff --git a/rust/src/chat/tests/chat.rs b/rust/src/chat/tests/chat.rs index ab366b8827..b920b5f0ed 100644 --- a/rust/src/chat/tests/chat.rs +++ b/rust/src/chat/tests/chat.rs @@ -297,6 +297,7 @@ fn sample_tool_request(request_id: &str) -> ChatRequest { "required": ["city"], }), strict: None, + defer_loading: None, }]; request.tool_context = vllm_chat::ResolvedToolContext::new( &request.messages, diff --git a/rust/src/chat/tests/roundtrip.rs b/rust/src/chat/tests/roundtrip.rs index b1a6505cf8..079ce3e1a7 100644 --- a/rust/src/chat/tests/roundtrip.rs +++ b/rust/src/chat/tests/roundtrip.rs @@ -1289,6 +1289,7 @@ fn test_tools() -> Vec { "required": ["location"] }), strict: None, + defer_loading: None, }, ChatTool { name: "add".to_string(), @@ -1306,6 +1307,7 @@ fn test_tools() -> Vec { "required": ["y", "x", "items"] }), strict: None, + defer_loading: None, }, ] } diff --git a/rust/src/cmd/src/cli.rs b/rust/src/cmd/src/cli.rs index 1269e0b8af..d3e369bbe3 100644 --- a/rust/src/cmd/src/cli.rs +++ b/rust/src/cmd/src/cli.rs @@ -501,8 +501,8 @@ impl SharedRuntimeArgs { self, listen_fd: i32, grpc_listen_fd: Option, - input_address: String, - output_address: String, + input_listener_fd: i32, + output_listener_fd: i32, coordinator_address: Option, engine_start_index: u32, engine_count: usize, @@ -518,8 +518,8 @@ impl SharedRuntimeArgs { Config { transport_mode: TransportMode::Bootstrapped { - input_address, - output_address, + input_listener_fd, + output_listener_fd, engine_start_index, engine_count, data_parallel_size, @@ -697,14 +697,12 @@ pub struct FrontendArgs { /// supervisor. When not set, no gRPC server is started. #[arg(long)] pub grpc_listen_fd: Option, - /// Frontend input ROUTER socket address that the Python engines will - /// connect to. + /// Inherited frontend input ROUTER listener file descriptor. #[arg(long)] - pub input_address: String, - /// Frontend output PULL socket address that the Python engines will push - /// responses to. + pub input_listener_fd: i32, + /// Inherited frontend output PULL listener file descriptor. #[arg(long)] - pub output_address: String, + pub output_listener_fd: i32, /// Optional Python-owned frontend-side DP coordinator socket address for /// external coordinator mode in the bootstrapped frontend path, i.e., /// `stats_update_address`. @@ -733,8 +731,8 @@ impl FrontendArgs { self.runtime.into_bootstrapped_config( self.listen_fd, self.grpc_listen_fd, - self.input_address, - self.output_address, + self.input_listener_fd, + self.output_listener_fd, self.coordinator_address, self.engine_start_index, self.engine_count, diff --git a/rust/src/cmd/src/cli/tests.rs b/rust/src/cmd/src/cli/tests.rs index 944febc36c..aa164bc618 100644 --- a/rust/src/cmd/src/cli/tests.rs +++ b/rust/src/cmd/src/cli/tests.rs @@ -668,10 +668,10 @@ fn frontend_args_json_passes_tls_into_config() { "frontend", "--listen-fd", "3", - "--input-address", - "ipc:///tmp/input.sock", - "--output-address", - "ipc:///tmp/output.sock", + "--input-listener-fd", + "999999", + "--output-listener-fd", + "999998", "--args-json", r#"{"model_tag":"Qwen/Qwen3-0.6B","ssl_certfile":"/tmp/cert.pem","ssl_keyfile":"/tmp/key.pem"}"#, ]) @@ -693,10 +693,10 @@ fn frontend_args_json_rejects_out_of_range_cert_reqs() { "frontend", "--listen-fd", "3", - "--input-address", - "ipc:///tmp/input.sock", - "--output-address", - "ipc:///tmp/output.sock", + "--input-listener-fd", + "999999", + "--output-listener-fd", + "999998", "--args-json", r#"{"model_tag":"Qwen/Qwen3-0.6B","ssl_certfile":"/tmp/cert.pem","ssl_cert_reqs":5}"#, ]) @@ -718,10 +718,10 @@ fn frontend_args_json_passes_enable_request_id_headers_into_config() { "frontend", "--listen-fd", "3", - "--input-address", - "ipc:///tmp/input.sock", - "--output-address", - "ipc:///tmp/output.sock", + "--input-listener-fd", + "999999", + "--output-listener-fd", + "999998", "--args-json", r#"{"model_tag":"Qwen/Qwen3-0.6B","enable_request_id_headers":true}"#, ]) @@ -807,10 +807,10 @@ fn frontend_args_json_passes_lora_modules_into_config() { "frontend", "--listen-fd", "3", - "--input-address", - "ipc:///tmp/input.sock", - "--output-address", - "ipc:///tmp/output.sock", + "--input-listener-fd", + "999999", + "--output-listener-fd", + "999998", "--args-json", r#"{"model_tag":"Qwen/Qwen3-0.6B","lora_modules":[{"name":"alice","path":"org/alice","base_model_name":null,"is_3d_lora_weight":true}]}"#, ]) @@ -862,10 +862,10 @@ fn frontend_args_json_accepts_api_key_string() { "frontend", "--listen-fd", "3", - "--input-address", - "ipc:///tmp/input.sock", - "--output-address", - "ipc:///tmp/output.sock", + "--input-listener-fd", + "999999", + "--output-listener-fd", + "999998", "--args-json", r#"{"model_tag":"Qwen/Qwen3-0.6B","api_key":"secret"}"#, ]) @@ -885,10 +885,10 @@ fn frontend_args_json_accepts_api_key_list() { "frontend", "--listen-fd", "3", - "--input-address", - "ipc:///tmp/input.sock", - "--output-address", - "ipc:///tmp/output.sock", + "--input-listener-fd", + "999999", + "--output-listener-fd", + "999998", "--args-json", r#"{"model_tag":"Qwen/Qwen3-0.6B","api_key":["secret-a","secret-b"]}"#, ]) @@ -991,10 +991,10 @@ fn frontend_args_accept_json() { "frontend", "--listen-fd", "3", - "--input-address", - "ipc:///tmp/input.sock", - "--output-address", - "ipc:///tmp/output.sock", + "--input-listener-fd", + "999999", + "--output-listener-fd", + "999998", "--coordinator-address", "tcp://127.0.0.1:7000", "--args-json", @@ -1008,8 +1008,8 @@ fn frontend_args_accept_json() { FrontendArgs { listen_fd: 3, grpc_listen_fd: None, - input_address: "ipc:///tmp/input.sock", - output_address: "ipc:///tmp/output.sock", + input_listener_fd: 999999, + output_listener_fd: 999998, coordinator_address: Some( "tcp://127.0.0.1:7000", ), @@ -1083,10 +1083,10 @@ fn frontend_args_json_applies_defaults() { "frontend", "--listen-fd", "3", - "--input-address", - "ipc:///tmp/input.sock", - "--output-address", - "ipc:///tmp/output.sock", + "--input-listener-fd", + "999999", + "--output-listener-fd", + "999998", "--engine-count", "4", "--args-json", @@ -1115,10 +1115,10 @@ fn frontend_args_json_ignores_engine_owned_max_model_len() { "frontend", "--listen-fd", "3", - "--input-address", - "ipc:///tmp/input.sock", - "--output-address", - "ipc:///tmp/output.sock", + "--input-listener-fd", + "999999", + "--output-listener-fd", + "999998", "--args-json", r#"{"model_tag":"Qwen/Qwen3-0.6B","max_model_len":-1}"#, ]) @@ -1165,10 +1165,10 @@ fn hf_overrides_preserve_merge_patch_in_cli_and_python_bootstrap() { "frontend", "--listen-fd", "3", - "--input-address", - "ipc:///tmp/input.sock", - "--output-address", - "ipc:///tmp/output.sock", + "--input-listener-fd", + "999999", + "--output-listener-fd", + "999998", "--args-json", &payload.to_string(), ]) @@ -1204,10 +1204,10 @@ fn frontend_args_json_accepts_supported_non_default_fields() { "frontend", "--listen-fd", "3", - "--input-address", - "ipc:///tmp/input.sock", - "--output-address", - "ipc:///tmp/output.sock", + "--input-listener-fd", + "999999", + "--output-listener-fd", + "999998", "--args-json", r#"{"model_tag":"Qwen/Qwen3-0.6B","generation_config":"vllm","revision":"release","engine_ready_timeout_secs":42,"tool_call_parser":"hermes","reasoning_parser":"qwen3_thinking","tokenizer_mode":"deepseek_v32","language_model_only":true,"max_logprobs":-1,"shutdown_timeout":3}"#, ]) @@ -1286,10 +1286,10 @@ fn frontend_args_json_ignores_unknown_fields() { "frontend", "--listen-fd", "3", - "--input-address", - "ipc:///tmp/input.sock", - "--output-address", - "ipc:///tmp/output.sock", + "--input-listener-fd", + "999999", + "--output-listener-fd", + "999998", "--args-json", r#"{"model_tag":"Qwen/Qwen3-0.6B","uds":"/tmp/vllm.sock","nested_unknown":{"x":1}}"#, ]) @@ -1308,10 +1308,10 @@ fn frontend_args_json_sets_prompt_tokens_details_flag() { "frontend", "--listen-fd", "3", - "--input-address", - "ipc:///tmp/input.sock", - "--output-address", - "ipc:///tmp/output.sock", + "--input-listener-fd", + "999999", + "--output-listener-fd", + "999998", "--args-json", r#"{"model_tag":"Qwen/Qwen3-0.6B","api_server_count":2,"enable_prompt_tokens_details":true}"#, ]) @@ -1371,10 +1371,10 @@ fn frontend_args_json_parses_cors_fields() { "frontend", "--listen-fd", "3", - "--input-address", - "ipc:///tmp/input.sock", - "--output-address", - "ipc:///tmp/output.sock", + "--input-listener-fd", + "999999", + "--output-listener-fd", + "999998", "--args-json", r#"{"model_tag":"Qwen/Qwen3-0.6B","allowed_origins":["http://a.com"],"allow_credentials":true}"#, ]) @@ -1396,10 +1396,10 @@ fn frontend_args_json_rejects_unsupported_fields() { "frontend", "--listen-fd", "3", - "--input-address", - "ipc:///tmp/input.sock", - "--output-address", - "ipc:///tmp/output.sock", + "--input-listener-fd", + "999999", + "--output-listener-fd", + "999998", "--args-json", r#"{"model_tag":"Qwen/Qwen3-0.6B","root_path":"/prefix"}"#, ]) @@ -1423,10 +1423,10 @@ fn frontend_args_json_aggregates_multiple_unsupported_fields() { "frontend", "--listen-fd", "3", - "--input-address", - "ipc:///tmp/input.sock", - "--output-address", - "ipc:///tmp/output.sock", + "--input-listener-fd", + "999999", + "--output-listener-fd", + "999998", "--args-json", r#"{"model_tag":"Qwen/Qwen3-0.6B","response_role":"assistant","root_path":"/prefix"}"#, ]) @@ -1452,10 +1452,10 @@ fn frontend_args_json_rejects_malformed_json() { "frontend", "--listen-fd", "3", - "--input-address", - "ipc:///tmp/input.sock", - "--output-address", - "ipc:///tmp/output.sock", + "--input-listener-fd", + "999999", + "--output-listener-fd", + "999998", "--args-json", r#"{"model_tag":"Qwen/Qwen3-0.6B""#, ]) @@ -1999,10 +1999,10 @@ fn frontend_args_reject_legacy_handshake_flags() { "frontend", "--listen-fd", "3", - "--input-address", - "ipc:///tmp/input.sock", - "--output-address", - "ipc:///tmp/output.sock", + "--input-listener-fd", + "999999", + "--output-listener-fd", + "999998", "--args-json", r#"{"model_tag":"Qwen/Qwen3-0.6B"}"#, "--handshake-address", @@ -2043,10 +2043,10 @@ fn frontend_config_uses_external_coordinator_when_coordinator_address_is_present "frontend", "--listen-fd", "3", - "--input-address", - "ipc:///tmp/input.sock", - "--output-address", - "ipc:///tmp/output.sock", + "--input-listener-fd", + "999999", + "--output-listener-fd", + "999998", "--coordinator-address", "tcp://127.0.0.1:7000", "--engine-start-index", @@ -2068,8 +2068,8 @@ fn frontend_config_uses_external_coordinator_when_coordinator_address_is_present expect![[r#" Config { transport_mode: Bootstrapped { - input_address: "ipc:///tmp/input.sock", - output_address: "ipc:///tmp/output.sock", + input_listener_fd: 999999, + output_listener_fd: 999998, engine_start_index: 3, engine_count: 1, data_parallel_size: 4, @@ -2200,10 +2200,10 @@ fn frontend_config_adopts_inherited_grpc_listener() { "3", "--grpc-listen-fd", "4", - "--input-address", - "ipc:///tmp/input.sock", - "--output-address", - "ipc:///tmp/output.sock", + "--input-listener-fd", + "999999", + "--output-listener-fd", + "999998", "--args-json", r#"{"model_tag":"Qwen/Qwen3-0.6B","grpc_port":50051}"#, ]) @@ -2227,10 +2227,10 @@ fn frontend_args_json_enables_profiling_when_profiler_config_set() { "frontend", "--listen-fd", "3", - "--input-address", - "ipc:///tmp/input.sock", - "--output-address", - "ipc:///tmp/output.sock", + "--input-listener-fd", + "999999", + "--output-listener-fd", + "999998", "--args-json", r#"{"model_tag":"Qwen/Qwen3-0.6B","profiler_config":{"profiler":"torch","torch_profiler_dir":"/tmp/profile"}}"#, ]) @@ -2251,10 +2251,10 @@ fn frontend_args_json_disables_profiling_when_profiler_config_absent() { "frontend", "--listen-fd", "3", - "--input-address", - "ipc:///tmp/input.sock", - "--output-address", - "ipc:///tmp/output.sock", + "--input-listener-fd", + "999999", + "--output-listener-fd", + "999998", "--args-json", r#"{"model_tag":"Qwen/Qwen3-0.6B"}"#, ]) @@ -2275,10 +2275,10 @@ fn frontend_args_json_disables_profiling_when_profiler_type_is_null() { "frontend", "--listen-fd", "3", - "--input-address", - "ipc:///tmp/input.sock", - "--output-address", - "ipc:///tmp/output.sock", + "--input-listener-fd", + "999999", + "--output-listener-fd", + "999998", "--args-json", r#"{"model_tag":"Qwen/Qwen3-0.6B","profiler_config":{"profiler":null}}"#, ]) diff --git a/rust/src/engine-core-client/Cargo.toml b/rust/src/engine-core-client/Cargo.toml index 14b9fe1423..86ffe4c4cd 100644 --- a/rust/src/engine-core-client/Cargo.toml +++ b/rust/src/engine-core-client/Cargo.toml @@ -18,6 +18,7 @@ futures.workspace = true half.workspace = true hex.workspace = true itertools.workspace = true +libc.workspace = true parking_lot.workspace = true rmp-serde.workspace = true rmpv.workspace = true @@ -27,6 +28,7 @@ serde_json.workspace = true serde_repr.workspace = true serde_tuple.workspace = true serde_with.workspace = true +socket2.workspace = true task-local.workspace = true tempfile = { workspace = true, optional = true } thiserror.workspace = true diff --git a/rust/src/engine-core-client/src/client.rs b/rust/src/engine-core-client/src/client.rs index 0514292b90..b3516936fe 100644 --- a/rust/src/engine-core-client/src/client.rs +++ b/rust/src/engine-core-client/src/client.rs @@ -9,7 +9,7 @@ use futures::future::join_all; use itertools::Itertools; use serde::Serialize; use serde_json::Value as JsonValue; -use tokio::sync::mpsc; +use tokio::sync::{Mutex, mpsc}; use tokio_util::task::AbortOnDropHandle; use tracing::{debug, info, trace}; @@ -53,16 +53,15 @@ pub enum TransportMode { local_output_address: Option, }, - /// The Python supervisor has already chosen the frontend transport - /// addresses, and the Rust process only needs to bind them and wait for - /// engine registration frames. + /// The Python supervisor has already bound the frontend transport + /// listeners. Rust adopts them and waits for engine registration frames. + /// `EngineCoreClient::connect` consumes each descriptor exactly once; + /// copied integer values do not create additional descriptor ownership. Bootstrapped { - /// Input ROUTER socket address that engines will connect to for - /// requests. - input_address: String, - /// Output PULL socket address that engines will connect to for - /// responses. - output_address: String, + /// Raw input ROUTER listener descriptor inherited from the supervisor. + input_listener_fd: i32, + /// Raw output PULL listener descriptor inherited from the supervisor. + output_listener_fd: i32, /// First data-parallel engine rank expected to register on this /// transport. engine_start_index: u32, @@ -283,6 +282,9 @@ pub struct EngineCoreClient { inner: Arc, coordinator: Option, abort_tx: mpsc::UnboundedSender, + /// Whether a profiling session is awaiting an explicit stop. Held across each + /// start/stop utility call so a stale stop reply cannot clear a newer session. + profile_active: Mutex, /// Runtime used to send messages to the engine and drive all background tasks. runtime: BackgroundShutdownRuntime, @@ -333,8 +335,8 @@ impl EngineCoreClient { } TransportMode::Bootstrapped { - input_address, - output_address, + input_listener_fd, + output_listener_fd, engine_start_index, engine_count, ready_timeout, @@ -345,8 +347,8 @@ impl EngineCoreClient { } transport::connect_bootstrapped( - input_address, - output_address, + *input_listener_fd, + *output_listener_fd, *engine_start_index, *engine_count, *ready_timeout, @@ -431,6 +433,7 @@ impl EngineCoreClient { inner, coordinator, abort_tx, + profile_active: Mutex::new(false), runtime, output_task, dispatcher_task, @@ -963,14 +966,38 @@ impl EngineCoreClient { } /// Start profiling the engine. - pub async fn start_profile(&self, profile_prefix: Option<&str>) -> Result<()> { - self.call_utility::<(), _>("profile", (true, profile_prefix)).await?; + pub async fn start_profile( + &self, + profile_prefix: Option<&str>, + delay_iterations: Option, + max_iterations: Option, + ) -> Result<()> { + let mut active = self.profile_active.lock().await; + if *active { + return Err(Error::ProfileAlreadyActive); + } + + // A cancelled start may still reach the engine, so only a reported + // failure clears the session; otherwise a stop is needed. + *active = true; + if let Err(error) = self + .call_utility::<(), _>( + "profile", + (true, profile_prefix, delay_iterations, max_iterations), + ) + .await + { + *active = false; + return Err(error); + } Ok(()) } /// Stop profiling the engine. pub async fn stop_profile(&self, profile_prefix: Option<&str>) -> Result<()> { + let mut active = self.profile_active.lock().await; self.call_utility::<(), _>("profile", (false, profile_prefix)).await?; + *active = false; Ok(()) } diff --git a/rust/src/engine-core-client/src/error.rs b/rust/src/engine-core-client/src/error.rs index d43952f023..d55f2cbbcb 100644 --- a/rust/src/engine-core-client/src/error.rs +++ b/rust/src/engine-core-client/src/error.rs @@ -83,6 +83,8 @@ pub enum Error { ClientClosed { message: String }, #[error("request output stream for `{request_id}` closed unexpectedly")] RequestStreamClosed { request_id: String }, + #[error("profiling session is already active")] + ProfileAlreadyActive, #[error("utility call `{method}` failed (call_id={call_id}): {message}")] UtilityCallFailed { method: String, diff --git a/rust/src/engine-core-client/src/metrics.rs b/rust/src/engine-core-client/src/metrics.rs index b392d52601..e8b0741066 100644 --- a/rust/src/engine-core-client/src/metrics.rs +++ b/rust/src/engine-core-client/src/metrics.rs @@ -6,14 +6,15 @@ use std::collections::BTreeSet; use std::time::{SystemTime, UNIX_EPOCH}; use vllm_metrics::{ - EngineLabels, EnginePositionLabels, F64Gauge, Family, HistogramMetric, LoraAdapterNames, - LoraInfoLabels, MooncakeOperationCounterFamily, MooncakeOperationHistogramFamily, - MooncakeOperationLabels, RequestMetrics, SchedulerLogStatsAccumulator, SchedulerMetrics, - U64Counter, U64Gauge, WaitingReasonLabels, + CacheHitSourceLabels, EngineLabels, EnginePositionLabels, F64Gauge, Family, HistogramMetric, + LoraAdapterNames, LoraInfoLabels, MooncakeOperationCounterFamily, + MooncakeOperationHistogramFamily, MooncakeOperationLabels, RequestMetrics, + SchedulerLogStatsAccumulator, SchedulerMetrics, U64Counter, U64Gauge, WaitingReasonLabels, }; use crate::protocol::stats::{ - KvConnectorStats, MooncakeStats, MultiConnectorStats, NixlStats, SchedulerStats, + CachedTokensBySource, KvConnectorStats, MooncakeStats, MultiConnectorStats, NixlStats, + SchedulerStats, }; use crate::transport::ConnectedEngine; @@ -60,6 +61,8 @@ struct SchedulerStatsHandles { prefix_cache_hits: U64Counter, external_prefix_cache_queries: U64Counter, external_prefix_cache_hits: U64Counter, + /// One handle per `CachedTokensBySource::SOURCES` entry. + prompt_tokens_cached_by_source: [U64Counter; 5], // Speculative decoding counters. spec_decode_num_drafts: U64Counter, @@ -165,6 +168,15 @@ fn resolve_scheduler_stats_handles( .external_prefix_cache_queries .get_or_create_owned(&labels), external_prefix_cache_hits: metrics.external_prefix_cache_hits.get_or_create_owned(&labels), + prompt_tokens_cached_by_source: CachedTokensBySource::SOURCES.map(|source| { + metrics + .prompt_tokens_cached_by_source + .get_or_create_owned(&CacheHitSourceLabels { + model_name: model_name.to_string(), + engine, + source, + }) + }), spec_decode_num_drafts: metrics.spec_decode_num_drafts.get_or_create_owned(&labels), spec_decode_num_draft_tokens: metrics .spec_decode_num_draft_tokens @@ -220,6 +232,8 @@ fn record_scheduler_stats_with_handles(handles: &SchedulerStatsHandles, stats: & // Prefix-cache counters, including the connector-backed external cache path. handles.prefix_cache_queries.inc_by(stats.prefix_cache_stats.base.queries); handles.prefix_cache_hits.inc_by(stats.prefix_cache_stats.base.hits); + // Local prefix-cache hits are the `device` tier. + handles.prompt_tokens_cached_by_source[0].inc_by(stats.prefix_cache_stats.base.hits); if let Some(connector_prefix_cache_stats) = &stats.connector_prefix_cache_stats { handles @@ -228,6 +242,10 @@ fn record_scheduler_stats_with_handles(handles: &SchedulerStatsHandles, stats: & handles .external_prefix_cache_hits .inc_by(connector_prefix_cache_stats.base.hits); + let counts = connector_prefix_cache_stats.hits_by_source.counts(); + for (handle, count) in handles.prompt_tokens_cached_by_source.iter().zip(counts) { + handle.inc_by(count); + } } // Speculative decoding counters. diff --git a/rust/src/engine-core-client/src/protocol/stats.rs b/rust/src/engine-core-client/src/protocol/stats.rs index aa1b9c69ab..7797ecf6e4 100644 --- a/rust/src/engine-core-client/src/protocol/stats.rs +++ b/rust/src/engine-core-client/src/protocol/stats.rs @@ -21,6 +21,34 @@ pub struct BaseCacheStats { pub hits: u64, } +/// Cache-hit tokens per source tier. Python sends a sparse +/// `dict[CacheHitSource, int]`, so absent tiers default to zero. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, Serialize, Deserialize)] +#[serde(default)] +pub struct CachedTokensBySource { + pub device: u64, + pub host: u64, + pub p2p: u64, + pub disk: u64, + pub external_unspecified: u64, +} + +impl CachedTokensBySource { + /// Label values, in the same order as [`Self::counts`]. + pub const SOURCES: [&'static str; 5] = + ["device", "host", "p2p", "disk", "external_unspecified"]; + + pub fn counts(&self) -> [u64; 5] { + [ + self.device, + self.host, + self.p2p, + self.disk, + self.external_unspecified, + ] + } +} + /// Stores prefix cache hit statistics. /// - `reset`: Whether `reset_prefix_cache` was invoked. /// - `queries`: Refers to the number of tokens that were queried. @@ -38,6 +66,9 @@ pub struct PrefixCacheStats { pub preempted_queries: u64, /// The `hits` number for preempted requests. pub preempted_hits: u64, + /// `hits` split by the cache tier that supplied them (connector stats only). + #[serde(default)] + pub hits_by_source: CachedTokensBySource, } /// Single KV cache block eviction sample. @@ -304,3 +335,31 @@ pub struct SchedulerStats { /// Estimated MFU/performance stats, when enabled. pub perf_stats: Option, } + +#[cfg(test)] +mod tests { + use super::{CachedTokensBySource, PrefixCacheStats}; + + #[test] + fn prefix_cache_stats_hits_by_source_defaults_and_decodes() { + // Engines without per-source attribution omit the field entirely. + let legacy = serde_json::json!({ + "reset": false, "requests": 1, "queries": 8, "hits": 4, + "preempted_requests": 0, "preempted_queries": 0, "preempted_hits": 0 + }); + let wire = rmp_serde::to_vec_named(&legacy).unwrap(); + let stats: PrefixCacheStats = rmp_serde::from_slice(&wire).unwrap(); + assert_eq!(stats.base.hits, 4); + assert_eq!(stats.hits_by_source, CachedTokensBySource::default()); + + let payload = serde_json::json!({ + "reset": false, "requests": 1, "queries": 16, "hits": 12, + "preempted_requests": 0, "preempted_queries": 0, "preempted_hits": 0, + "hits_by_source": {"host": 8, "p2p": 4} + }); + let wire = rmp_serde::to_vec_named(&payload).unwrap(); + let stats: PrefixCacheStats = rmp_serde::from_slice(&wire).unwrap(); + assert_eq!(stats.hits_by_source.counts(), [0, 8, 4, 0, 0]); + assert_eq!(CachedTokensBySource::SOURCES[1], "host"); + } +} diff --git a/rust/src/engine-core-client/src/tests/client.rs b/rust/src/engine-core-client/src/tests/client.rs index 0ac81764a9..dbd3fa69ce 100644 --- a/rust/src/engine-core-client/src/tests/client.rs +++ b/rust/src/engine-core-client/src/tests/client.rs @@ -4,6 +4,8 @@ use std::collections::{BTreeMap, BTreeSet}; use std::convert::TryFrom; use std::io::Cursor; +use std::os::fd::IntoRawFd; +use std::os::unix::net::UnixListener; use std::path::PathBuf; use std::process::Command; use std::sync::Once; @@ -311,10 +313,17 @@ fn bootstrapped_test_config( client_index: u32, coordinator_mode: Option, ) -> EngineCoreClientConfig { + fn listener(address: &str) -> i32 { + let path = address.strip_prefix("ipc://").expect("bootstrapped tests use IPC listeners"); + let _ = std::fs::remove_file(path); + let listener = UnixListener::bind(path).expect("bind inherited test listener"); + listener.into_raw_fd() + } + EngineCoreClientConfig { transport_mode: TransportMode::Bootstrapped { - input_address, - output_address, + input_listener_fd: listener(&input_address), + output_listener_fd: listener(&output_address), engine_start_index: 0, engine_count, data_parallel_size: engine_count, @@ -362,15 +371,22 @@ fn bootstrapped_test_config_with_start_index( #[test] fn client_config_validates_bootstrapped_dp_range() { - let mut config = bootstrapped_test_config_with_start_index( - "ipc://unused-input".to_string(), - "ipc://unused-output".to_string(), - 1, - 2, - Duration::from_secs(1), - 0, - None, - ); + // Validation never consumes the descriptors, so placeholders avoid opening + // listeners that nothing would close. + let mut config = EngineCoreClientConfig { + transport_mode: TransportMode::Bootstrapped { + input_listener_fd: -1, + output_listener_fd: -1, + engine_start_index: 1, + engine_count: 2, + data_parallel_size: 3, + ready_timeout: Duration::from_secs(1), + }, + coordinator_mode: None, + model_name: "test-model".to_string(), + client_index: 0, + engine_stats_enabled: true, + }; config.validate().expect("frontend may own a subset of global DP ranks"); let TransportMode::Bootstrapped { diff --git a/rust/src/engine-core-client/src/transport.rs b/rust/src/engine-core-client/src/transport.rs index 7aac8774ca..e453cafdc8 100644 --- a/rust/src/engine-core-client/src/transport.rs +++ b/rust/src/engine-core-client/src/transport.rs @@ -4,6 +4,8 @@ use std::collections::{BTreeMap, BTreeSet}; use std::fmt::Debug; use std::ops::Deref; +use std::os::fd::{FromRawFd, OwnedFd, RawFd}; +use std::os::unix::net::UnixListener as StdUnixListener; use std::time::Duration; use bytes::Bytes; @@ -327,8 +329,8 @@ pub async fn connect_handshake( /// identities are synthesized from contiguous rank order instead of being /// discovered through a Rust-owned handshake. pub async fn connect_bootstrapped( - input_address: &str, - output_address: &str, + input_listener_fd: RawFd, + output_listener_fd: RawFd, engine_start_index: u32, engine_count: usize, ready_timeout: Duration, @@ -354,10 +356,16 @@ pub async fn connect_bootstrapped( } let mut input_socket = RouterSocket::new(); - let input_address = input_socket.bind(input_address).await?.to_string(); + let input_address = input_socket + .bind_listener(adopt_listener(input_listener_fd)?) + .await? + .to_string(); let mut output_socket = PullSocket::new(); - let output_address = output_socket.bind(output_address).await?.to_string(); + let output_address = output_socket + .bind_listener(adopt_listener(output_listener_fd)?) + .await? + .to_string(); let engines = wait_for_input_registrations( &mut input_socket, @@ -385,6 +393,34 @@ pub async fn connect_bootstrapped( }) } +fn adopt_listener(fd: RawFd) -> Result { + if fd < 0 || unsafe { libc::fcntl(fd, libc::F_GETFD) } < 0 { + return Err(Error::InvalidClientConfig { + message: format!("inherited ZMQ listener fd {fd} is not open"), + }); + } + + // SAFETY: Python's Popen(pass_fds) gives the frontend its own descriptor. + // The bootstrapped startup path calls this function once per descriptor, + // transferring ownership to the listener constructed below. + let fd = unsafe { OwnedFd::from_raw_fd(fd) }; + let socket: socket2::Socket = fd.into(); + let domain = socket.domain()?; + + if domain == socket2::Domain::IPV4 || domain == socket2::Domain::IPV6 { + let listener: std::net::TcpListener = socket.into(); + return Ok(listener.into()); + } + if domain == socket2::Domain::UNIX { + let listener: StdUnixListener = socket.into(); + return Ok(listener.into()); + } + + Err(Error::InvalidClientConfig { + message: format!("unsupported inherited listener domain {domain:?}"), + }) +} + /// Bind new input and output sockets. async fn bind_local_sockets( local_host: &str, @@ -603,7 +639,12 @@ pub async fn run_output_loop( #[cfg(test)] mod tests { - use super::bind_local_sockets; + use std::os::fd::IntoRawFd; + + use zeromq::RouterSocket; + use zeromq::prelude::Socket; + + use super::{adopt_listener, bind_local_sockets}; #[tokio::test] async fn bind_local_sockets_resolves_zero_port_bindings() { @@ -614,4 +655,21 @@ mod tests { assert!(output_address.starts_with("tcp://127.0.0.1:")); assert_ne!(input_address, output_address); } + + #[tokio::test] + async fn inherited_tcp_listener_is_adopted_without_rebinding() { + let listener = std::net::TcpListener::bind(("127.0.0.1", 0)).expect("bind listener"); + let expected = listener.local_addr().expect("listener address"); + let mut router = RouterSocket::new(); + + let endpoint = router + .bind_listener( + adopt_listener(listener.into_raw_fd()).expect("adopt inherited listener fd"), + ) + .await + .expect("bind inherited listener"); + + assert_eq!(endpoint.to_string(), format!("tcp://{expected}")); + assert!(router.close().await.is_empty()); + } } diff --git a/rust/src/metrics/src/scheduler.rs b/rust/src/metrics/src/scheduler.rs index cdc25a244d..64ca465418 100644 --- a/rust/src/metrics/src/scheduler.rs +++ b/rust/src/metrics/src/scheduler.rs @@ -86,6 +86,13 @@ pub struct EngineLabels { pub engine: u32, } +#[derive(Clone, Debug, Hash, PartialEq, Eq, EncodeLabelSet)] +pub struct CacheHitSourceLabels { + pub model_name: String, + pub engine: u32, + pub source: &'static str, +} + #[derive(Clone, Debug, Hash, PartialEq, Eq, EncodeLabelSet)] pub struct EnginePositionLabels { pub model_name: String, @@ -236,6 +243,7 @@ pub struct SchedulerMetrics { pub prefix_cache_hits: Family, pub external_prefix_cache_queries: Family, pub external_prefix_cache_hits: Family, + pub prompt_tokens_cached_by_source: Family, // Speculative decoding counters. pub spec_decode_num_drafts: Family, @@ -339,6 +347,13 @@ impl SchedulerMetrics { ); let external_prefix_cache_hits = Family::default(); + + let prompt_tokens_cached_by_source = Family::default(); + registry.register( + "vllm:prompt_tokens_cached_by_source", + "Prefix-cache hit tokens by the cache tier that supplied them: device (local HBM), host (offloaded to DRAM), disk, p2p (transferred from another vLLM instance) or external_unspecified. Counted at admission; sums to vllm:prefix_cache_hits + vllm:external_prefix_cache_hits.", + prompt_tokens_cached_by_source.clone(), + ); registry.register( "vllm:external_prefix_cache_hits", "External prefix cache hits from KV connector cross-instance cache sharing, in terms of number of cached tokens.", @@ -528,6 +543,7 @@ impl SchedulerMetrics { prefix_cache_hits, external_prefix_cache_queries, external_prefix_cache_hits, + prompt_tokens_cached_by_source, spec_decode_num_drafts, spec_decode_num_draft_tokens, spec_decode_num_accepted_tokens, diff --git a/rust/src/parser/python/Cargo.toml b/rust/src/parser/python/Cargo.toml deleted file mode 100644 index aadae5638f..0000000000 --- a/rust/src/parser/python/Cargo.toml +++ /dev/null @@ -1,19 +0,0 @@ -[package] -name = "vllm-tool-parser-py" -version.workspace = true -edition.workspace = true -license.workspace = true - -[lib] -name = "_rust_tool_parser" -crate-type = ["cdylib", "rlib"] - -[dependencies] -pyo3.workspace = true -pythonize = { workspace = true, features = ["serde_json"] } -serde_json.workspace = true -thiserror-ext.workspace = true -vllm-parser.workspace = true - -[lints] -workspace = true diff --git a/rust/src/parser/python/src/lib.rs b/rust/src/parser/python/src/lib.rs deleted file mode 100644 index 38b5f1d461..0000000000 --- a/rust/src/parser/python/src/lib.rs +++ /dev/null @@ -1,398 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright contributors to the vLLM project - -//! Thin PyO3 bindings for `vllm_parser::tool`. -//! -//! This crate exposes the Rust tool parser trait and data shapes to Python -//! while keeping parser state, grammar, and schema-aware argument conversion in -//! Rust. Python callers should use this module as a typed bridge and keep any -//! vLLM protocol adaptation outside the binding. - -use pyo3::exceptions::PyValueError; -use pyo3::prelude::*; -use pyo3::types::{PyAny, PyModule}; -use pythonize::{depythonize, pythonize}; -use serde_json::Value; -use thiserror_ext::AsReport as _; -use vllm_parser::tool::{Tool, ToolCallDelta, ToolParser, ToolParserOutput}; - -macro_rules! tool_parser_factory { - ($($parser:ident),+ $(,)?) => { - fn create_tool_parser( - name: &str, - tools: &[Tool], - ) -> PyResult> { - match name { - $( - stringify!($parser) => { - ::create(tools) - } - )+ - _ => { - return Err(PyValueError::new_err(format!( - "unsupported tool parser `{name}`" - ))); - } - } - .map_err(|error| PyValueError::new_err(error.to_report_string())) - } - }; -} - -// Export a tool parser to Python by registering it here. -tool_parser_factory! { - MinimaxM3ToolParser, - - // Below are the parsers just for testing purposes on Python side. - DeepSeekV4ToolParser, - KimiK2ToolParser, -} - -#[pyclass(name = "Tool", module = "vllm._rust_tool_parser", skip_from_py_object)] -#[derive(Clone)] -struct PyTool(Tool); - -#[pymethods] -impl PyTool { - #[new] - #[pyo3(signature = (name, description, parameters, strict=None))] - fn new( - name: String, - description: Option, - parameters: &Bound<'_, PyAny>, - strict: Option, - ) -> PyResult { - let parameters = depythonize::(parameters).map_err(|error| { - PyValueError::new_err(format!( - "failed to convert tool parameters from Python to JSON: {error}" - )) - })?; - Ok(Self(Tool { - name, - description, - parameters, - strict, - })) - } - - #[getter] - fn name(&self) -> &str { - &self.0.name - } - - #[getter] - fn description(&self) -> Option<&str> { - self.0.description.as_deref() - } - - #[getter] - fn parameters(&self, py: Python<'_>) -> PyResult> { - pythonize(py, &self.0.parameters).map(Bound::unbind).map_err(|error| { - PyValueError::new_err(format!( - "failed to convert tool parameters from JSON to Python: {error}" - )) - }) - } - - #[getter] - fn strict(&self) -> Option { - self.0.strict - } -} - -#[pyclass( - name = "ToolCallDelta", - module = "vllm._rust_tool_parser", - skip_from_py_object -)] -#[derive(Clone)] -struct PyToolCallDelta(ToolCallDelta); - -#[pymethods] -impl PyToolCallDelta { - #[new] - #[pyo3(signature = (tool_index, name, arguments))] - fn new(tool_index: usize, name: Option, arguments: String) -> Self { - Self(ToolCallDelta { - tool_index, - name, - arguments, - }) - } - - #[getter] - fn tool_index(&self) -> usize { - self.0.tool_index - } - - #[getter] - fn name(&self) -> Option<&str> { - self.0.name.as_deref() - } - - #[getter] - fn arguments(&self) -> &str { - &self.0.arguments - } -} - -#[pyclass( - name = "ToolParserOutput", - module = "vllm._rust_tool_parser", - skip_from_py_object -)] -#[derive(Clone)] -struct PyToolParserOutput(ToolParserOutput); - -#[pymethods] -impl PyToolParserOutput { - #[new] - #[pyo3(signature = (normal_text="", calls=None))] - fn new(py: Python<'_>, normal_text: &str, calls: Option>>) -> Self { - let mut output = ToolParserOutput::default(); - output.push_text(normal_text); - for call in calls.unwrap_or_default() { - output.push_call(call.borrow(py).0.clone()); - } - Self(output) - } - - #[getter] - fn normal_text(&self) -> String { - self.0.normal_text() - } - - #[getter] - fn calls(&self) -> Vec { - self.0.calls().into_iter().cloned().map(PyToolCallDelta).collect() - } - - fn append(&mut self, other: PyRef<'_, PyToolParserOutput>) { - self.0.append(other.0.clone()); - } - - fn coalesce(&self) -> Self { - Self(self.0.clone().coalesce()) - } -} - -#[pyclass(name = "ToolParser", module = "vllm._rust_tool_parser", unsendable)] -struct PyToolParser(Box); - -impl PyToolParser { - fn parse_into_output(&mut self, chunk: &str, output: &mut PyToolParserOutput) -> PyResult<()> { - self.0 - .parse_into(chunk, &mut output.0) - .map_err(|error| PyValueError::new_err(error.to_report_string())) - } -} - -#[pymethods] -impl PyToolParser { - #[new] - fn new(py: Python<'_>, parser_name: &str, tools: Vec>) -> PyResult { - let tools = tools.iter().map(|tool| tool.borrow(py).0.clone()).collect::>(); - create_tool_parser(parser_name, &tools).map(Self) - } - - fn parse_into( - &mut self, - chunk: &str, - mut output: PyRefMut<'_, PyToolParserOutput>, - ) -> PyResult<()> { - self.parse_into_output(chunk, &mut output) - } - - fn finish(&mut self) -> PyResult { - self.0 - .finish() - .map(PyToolParserOutput) - .map_err(|error| PyValueError::new_err(error.to_report_string())) - } - - fn reset(&mut self) -> String { - self.0.reset() - } - - fn preserve_special_tokens(&self) -> bool { - self.0.preserve_special_tokens() - } - - fn tool_call_id(&self, tool_index: usize) -> Option<&str> { - self.0.tool_call_id(tool_index) - } -} - -#[pymodule] -fn _rust_tool_parser(m: &Bound<'_, PyModule>) -> PyResult<()> { - m.add_class::()?; - m.add_class::()?; - m.add_class::()?; - m.add_class::()?; - Ok(()) -} - -#[cfg(test)] -mod tests { - use serde_json::json; - - use super::*; - - fn with_python(f: impl for<'py> FnOnce(Python<'py>) -> R) -> R { - Python::initialize(); - Python::attach(f) - } - - fn tool_schema() -> Value { - json!({ - "type": "object", - "properties": { - "user_id": {"type": "integer"}, - "shipping": { - "type": "object", - "properties": { - "city": {"type": "string"}, - "zip": {"type": "integer"} - } - } - } - }) - } - - fn build_call() -> String { - r#"<|DSML|tool_calls> -<|DSML|invoke name="create_order"> -<|DSML|parameter name="user_id" string="false">42 -<|DSML|parameter name="shipping" string="false">{"city":"Singapore","zip":18956} - -"# - .to_owned() - } - - fn make_py_tool(py: Python<'_>) -> PyResult> { - let parameters = pythonize(py, &tool_schema()).map_err(|error| { - PyValueError::new_err(format!( - "failed to convert test schema from JSON to Python: {error}" - )) - })?; - Py::new( - py, - PyTool::new( - "create_order".to_owned(), - Some("Create an order".to_owned()), - ¶meters, - None, - )?, - ) - } - - #[test] - fn tool_round_trips_typed_fields() { - with_python(|py| { - let tool = make_py_tool(py)?; - let borrowed = tool.borrow(py); - assert_eq!(borrowed.name(), "create_order"); - assert_eq!(borrowed.description(), Some("Create an order")); - assert_eq!(borrowed.strict(), None); - - let parameters = borrowed.parameters(py)?; - let parameters = depythonize::(parameters.bind(py))?; - assert_eq!(parameters, tool_schema()); - PyResult::Ok(()) - }) - .unwrap(); - } - - #[test] - fn output_append_and_coalesce() { - with_python(|py| { - let first = Py::new( - py, - PyToolCallDelta::new(0, Some("create_order".to_owned()), "{\"a\"".to_owned()), - )?; - let second = Py::new(py, PyToolCallDelta::new(0, None, ":1}".to_owned()))?; - let mut output = PyToolParserOutput::new(py, "text", Some(vec![first])); - let other = Py::new(py, PyToolParserOutput::new(py, "", Some(vec![second])))?; - output.append(other.borrow(py)); - - let coalesced = output.coalesce(); - assert_eq!(coalesced.normal_text(), "text"); - let calls = coalesced.calls(); - assert_eq!(calls.len(), 1); - assert_eq!(calls[0].tool_index(), 0); - assert_eq!(calls[0].name(), Some("create_order")); - assert_eq!(calls[0].arguments(), "{\"a\":1}"); - PyResult::Ok(()) - }) - .unwrap(); - } - - #[test] - fn parser_parse_finish_and_preserve_special_tokens() { - with_python(|py| { - let tool = make_py_tool(py)?; - let mut parser = PyToolParser::new(py, "DeepSeekV4ToolParser", vec![tool])?; - assert!(parser.preserve_special_tokens()); - - let mut output = PyToolParserOutput::new(py, "", None); - parser.parse_into_output(&build_call(), &mut output)?; - let finish = Py::new(py, parser.finish()?)?; - output.append(finish.borrow(py)); - let output = output.coalesce(); - - assert_eq!(output.normal_text(), ""); - let calls = output.calls(); - assert_eq!(calls.len(), 1); - assert_eq!(calls[0].name(), Some("create_order")); - assert_eq!( - serde_json::from_str::(calls[0].arguments()).unwrap(), - json!({ - "user_id": 42, - "shipping": { - "city": "Singapore", - "zip": 18956 - } - }) - ); - - assert_eq!(parser.reset(), ""); - PyResult::Ok(()) - }) - .unwrap(); - } - - #[test] - fn parser_exposes_model_emitted_tool_call_ids() { - with_python(|py| { - let tool = make_py_tool(py)?; - let mut parser = PyToolParser::new(py, "KimiK2ToolParser", vec![tool])?; - - let input = "<|tool_calls_section_begin|>\ - <|tool_call_begin|>functions.create_order:0<|tool_call_argument_begin|>\ - {\"user_id\":42}<|tool_call_end|>\ - <|tool_calls_section_end|>"; - let mut output = PyToolParserOutput::new(py, "", None); - parser.parse_into_output(input, &mut output)?; - - assert_eq!(parser.tool_call_id(0), Some("functions.create_order:0")); - assert_eq!(parser.tool_call_id(1), None); - PyResult::Ok(()) - }) - .unwrap(); - } - - #[test] - fn parser_errors_for_unknown_name() { - with_python(|py| { - let tool = make_py_tool(py)?; - let error = match PyToolParser::new(py, "missing", vec![tool]) { - Ok(_) => panic!("missing parser name unexpectedly succeeded"), - Err(error) => error, - }; - let message = format!("{error}"); - assert!(message.contains("unsupported tool parser `missing`")); - PyResult::Ok(()) - }) - .unwrap(); - } -} diff --git a/rust/src/parser/src/lib.rs b/rust/src/parser/src/lib.rs index fd4aa0aa7f..26256d3321 100644 --- a/rust/src/parser/src/lib.rs +++ b/rust/src/parser/src/lib.rs @@ -5,6 +5,7 @@ pub mod output_grammar; pub mod reasoning; +pub(crate) mod schema; pub mod tool; pub mod unified; pub(crate) mod utils; diff --git a/rust/src/parser/src/output_grammar.rs b/rust/src/parser/src/output_grammar.rs index 9058b4a58b..53c52538ee 100644 --- a/rust/src/parser/src/output_grammar.rs +++ b/rust/src/parser/src/output_grammar.rs @@ -255,6 +255,7 @@ mod tests { "required": ["query"] }), strict, + defer_loading: None, } } diff --git a/rust/src/parser/src/output_grammar/arguments.rs b/rust/src/parser/src/output_grammar/arguments.rs index a7d9fb3078..0f621789a3 100644 --- a/rust/src/parser/src/output_grammar/arguments.rs +++ b/rust/src/parser/src/output_grammar/arguments.rs @@ -25,14 +25,11 @@ //! - `minProperties` and `maxProperties` are not enforced. //! - `allOf` with more than one schema accepts any value. -use std::collections::HashSet; - -use serde_json::{Map, Value, json}; +use serde_json::{Value, json}; use xgrammar_structural_tag::format::{Format, JsonSchemaFormat}; -/// Bound on `$ref` and combinator nesting within one value, which keeps a long -/// chain of references from overflowing the stack. -const MAX_SCHEMA_DEPTH: usize = 32; +pub use crate::schema::JsonType; +use crate::schema::{self, OptionSource, SchemaOption, SchemaRoot}; /// The schema accepting any value. static ANY_SCHEMA: Value = Value::Bool(true); @@ -77,95 +74,22 @@ impl ParameterKey<'_> { } } -/// JSON type of a value option. -#[derive(Debug, Clone, Copy, PartialEq, Eq)] -pub enum JsonType { - String, - Number, - Boolean, - Null, - Object, - Array, -} - -impl JsonType { - const ALL: [Self; 6] = [ - Self::String, - Self::Number, - Self::Boolean, - Self::Null, - Self::Object, - Self::Array, - ]; - - /// The type's JSON name. Integers are numbers. - pub fn name(self) -> &'static str { - match self { - Self::String => "string", - Self::Number => "number", - Self::Boolean => "boolean", - Self::Null => "null", - Self::Object => "object", - Self::Array => "array", - } - } - - fn of_schema_type(name: &str) -> Option { - Some(match name { - "string" => Self::String, - "integer" | "number" => Self::Number, - "boolean" => Self::Boolean, - "null" => Self::Null, - "object" => Self::Object, - "array" => Self::Array, - _ => return None, - }) - } - - fn of_value(value: &Value) -> Self { - match value { - Value::String(_) => Self::String, - Value::Number(_) => Self::Number, - Value::Bool(_) => Self::Boolean, - Value::Null => Self::Null, - Value::Object(_) => Self::Object, - Value::Array(_) => Self::Array, - } - } - - /// Schema accepting every JSON value of this type. - fn any_schema(self) -> Value { - match self { - Self::Object => json!({ "type": "object", "additionalProperties": true }), - Self::Array => json!({ "type": "array", "items": true }), - _ => json!({ "type": self.name() }), - } - } -} - /// One option of a parameter value: a single JSON type under a narrowed /// schema, a constant, or any value of the type. pub struct ValueOption<'a> { - /// The option's JSON type. + /// The option's JSON Schema type. pub ty: JsonType, - source: OptionSource, + source: OptionSource<'a>, cx: &'a ArgumentContext<'a>, } -enum OptionSource { - /// A schema narrowed to [`ValueOption::ty`]. - Schema(Map), - /// A `const` or `enum` value. - Literal(Value), - /// Any value of [`ValueOption::ty`]. - Any, -} - impl ValueOption<'_> { /// The option as JSON text. pub fn json(&self) -> Format { match &self.source { - OptionSource::Schema(schema) => self.cx.json(Value::Object(schema.clone())), + OptionSource::Schema(schema) => { + self.cx.json(Value::Object(schema::narrow(schema, self.ty).into_owned())) + } OptionSource::Literal(value) => self.cx.json(json!({ "const": value })), OptionSource::Any => self.cx.json(self.ty.any_schema()), } @@ -216,26 +140,18 @@ pub struct ArgumentOptions { pub excludes: Vec, } -/// Reference targets expanded while collecting one value's options. -#[derive(Default)] -struct Expansion { - expanded: HashSet<*const Value>, - /// Whether a reference was skipped because its target already expanded. - skipped: bool, -} - /// Request state shared by the value options of one call. struct ArgumentContext<'a> { options: &'a ArgumentOptions, - /// The parameters schema, which local `$ref`s resolve against. - root: &'a Value, + root: SchemaRoot<'a>, } impl ArgumentContext<'_> { /// A JSON value under `schema`, which keeps the root's definitions so its /// local `$ref`s still resolve. fn json(&self, mut schema: Value) -> Format { - if let (Some(schema), Some(root)) = (schema.as_object_mut(), self.root.as_object()) { + if let (Some(schema), Some(root)) = (schema.as_object_mut(), self.root.schema().as_object()) + { for key in ["$defs", "definitions"] { if let Some(definitions) = root.get(key) { schema.entry(key).or_insert_with(|| definitions.clone()); @@ -250,122 +166,14 @@ impl ArgumentContext<'_> { }) } - fn resolve_ref(&self, reference: &str) -> Option<&Value> { - self.root.pointer(reference.strip_prefix('#')?) - } - - /// Follow `$ref`s and single-schema `allOf`s at the root. - fn resolve<'s>(&'s self, schema: &'s Value, depth: usize) -> &'s Value { - if depth > MAX_SCHEMA_DEPTH { - return &ANY_SCHEMA; - } - if let Some(reference) = schema.get("$ref").and_then(Value::as_str) { - return match self.resolve_ref(reference) { - Some(target) => self.resolve(target, depth + 1), - None => &ANY_SCHEMA, - }; - } - match schema.get("allOf").and_then(Value::as_array).map(Vec::as_slice) { - Some([schema]) => self.resolve(schema, depth + 1), - _ => schema, - } - } - /// Every option of a value under `schema`. - /// - /// Each reference target expands at most once. A union gains no values - /// from a repeated alternative, and a definition shared by several - /// alternatives would otherwise expand once per path to it, exponentially - /// in the nesting. A schema whose references only lead back into it - /// accepts any value. - fn value_options(&self, schema: &Value) -> Vec> { - let mut expansion = Expansion::default(); - let options = self.value_options_at(schema, 0, &mut expansion); - if options.is_empty() && expansion.skipped { - return self.any_value(); - } - options - } - - fn value_options_at( - &self, - schema: &Value, - depth: usize, - expansion: &mut Expansion, - ) -> Vec> { - let schema = match schema { - _ if depth > MAX_SCHEMA_DEPTH => return self.any_value(), - Value::Bool(false) => return vec![], - Value::Object(schema) => schema, - _ => return self.any_value(), - }; - if let Some(reference) = schema.get("$ref").and_then(Value::as_str) { - let Some(target) = self.resolve_ref(reference) else { - return self.any_value(); - }; - if !expansion.expanded.insert(std::ptr::from_ref(target)) { - expansion.skipped = true; - return vec![]; - } - return self.value_options_at(target, depth + 1, expansion); - } - if let Some(value) = schema.get("const") { - return vec![self.literal(value)]; - } - if let Some(values) = schema.get("enum").and_then(Value::as_array) { - return values.iter().map(|value| self.literal(value)).collect(); - } - if let Some(options) = - schema.get("anyOf").or_else(|| schema.get("oneOf")).and_then(Value::as_array) - { - return options - .iter() - .flat_map(|option| self.value_options_at(option, depth + 1, expansion)) - .collect(); - } - if let Some(schemas) = schema.get("allOf").and_then(Value::as_array) { - return match schemas.as_slice() { - [schema] => self.value_options_at(schema, depth + 1, expansion), - _ => self.any_value(), - }; - } - match schema.get("type") { - Some(Value::String(name)) => self.typed(schema.clone(), name).into_iter().collect(), - Some(Value::Array(names)) => names - .iter() - .filter_map(Value::as_str) - .filter_map(|name| { - let mut schema = schema.clone(); - schema.insert("type".to_string(), Value::String(name.to_string())); - self.typed(schema, name) - }) - .collect(), - _ => self.any_value(), - } - } - - fn typed(&self, schema: Map, type_name: &str) -> Option> { - Some(ValueOption { - ty: JsonType::of_schema_type(type_name)?, - source: OptionSource::Schema(schema), - cx: self, - }) - } - - fn literal(&self, value: &Value) -> ValueOption<'_> { - ValueOption { - ty: JsonType::of_value(value), - source: OptionSource::Literal(value.clone()), - cx: self, - } - } - - fn any_value(&self) -> Vec> { - JsonType::ALL + fn value_options<'s>(&'s self, schema: &'s Value) -> Vec> { + let root: SchemaRoot<'s> = self.root; + root.options(schema) .into_iter() - .map(|ty| ValueOption { + .map(|SchemaOption { ty, source }| ValueOption { ty, - source: OptionSource::Any, + source, cx: self, }) .collect() @@ -377,12 +185,12 @@ impl ArgumentContext<'_> { pub fn arguments(schema: &Value, syntax: &dyn ArgumentSyntax, options: &ArgumentOptions) -> Format { let cx = ArgumentContext { options, - root: schema, + root: SchemaRoot::new(schema), }; let parameter = |key: ParameterKey<'_>, schema: &Value| syntax.parameter(key, &cx.value_options(schema)); - let schema = cx.resolve(schema, 0); + let schema = cx.root.resolved(); let (properties, additional) = match schema { Value::Object(schema) => { let additional = match schema.get("additionalProperties") { @@ -492,6 +300,8 @@ pub fn group_by<'o, 'a, K: PartialEq>( #[cfg(test)] mod tests { + use serde_json::Map; + use super::*; use crate::output_grammar::test_utils::outline; use expect_test::{Expect, expect}; @@ -617,7 +427,7 @@ mod tests { or `celsius` `fahrenheit` - tag `` json(integer(minimum=1)) any_order `` + tag `` json(integer(minimum=1)) any_order `` "#]], ); } @@ -639,13 +449,13 @@ mod tests { sequence optional or - tag `` json(integer) `` + tag `` json(integer) `` tag `` text excluding [``] `` tag `` json(null) `` optional or tag `` `fast` `` - tag `` json(1) `` + tag `` json(1) `` or tag `` /[a-z]+/ `` tag `` json(string[]) `` diff --git a/rust/src/parser/src/schema.rs b/rust/src/parser/src/schema.rs new file mode 100644 index 0000000000..c27427de2e --- /dev/null +++ b/rust/src/parser/src/schema.rs @@ -0,0 +1,499 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +//! Views of tool parameter JSON schemas. +//! +//! Tool parsers coerce generated parameter values by their schema, and output +//! grammars constrain the same values by it. Both read a parameter's schema +//! through [`SchemaRoot`], so they agree on which JSON types the value may take. + +use std::borrow::Cow; +use std::collections::HashSet; + +use serde_json::{Map, Value, json}; + +/// Bound on `$ref` and combinator nesting within one value, which keeps a long +/// chain of references from overflowing the stack. +const MAX_SCHEMA_DEPTH: usize = 32; + +/// The schema accepting any value. +static ANY_SCHEMA: Value = Value::Bool(true); + +/// A tool's parameters schema, which local `$ref`s resolve against. +#[derive(Debug, Clone, Copy)] +pub(crate) struct SchemaRoot<'a> { + schema: &'a Value, +} + +/// Reference targets expanded while collecting one value's options. +#[derive(Default)] +struct Expansion { + expanded: HashSet<*const Value>, + /// Whether a reference was skipped because its target already expanded. + skipped: bool, +} + +/// Roots are equal when they are the same schema, which a pointer compare +/// decides without walking it. +impl PartialEq for SchemaRoot<'_> { + fn eq(&self, other: &Self) -> bool { + std::ptr::eq(self.schema, other.schema) + } +} + +impl Eq for SchemaRoot<'_> {} + +/// One option of a value: a single JSON type under a narrowed schema, a +/// constant, or any value of the type. +#[derive(Debug, Clone)] +pub(crate) struct SchemaOption<'a> { + pub(crate) ty: JsonType, + pub(crate) source: OptionSource<'a>, +} + +/// Where a [`SchemaOption`] comes from. +#[derive(Debug, Clone)] +pub(crate) enum OptionSource<'a> { + /// A schema, of which the option takes the values of its type; [`narrow`] + /// sets that type on it. + Schema(&'a Map), + /// A `const` or `enum` value. + Literal(&'a Value), + /// Any value of the option's type. + Any, +} + +/// JSON Schema type of a value option. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum JsonType { + String, + Integer, + Number, + Boolean, + Null, + Object, + Array, +} + +impl<'a> SchemaRoot<'a> { + pub(crate) fn new(schema: &'a Value) -> Self { + Self { schema } + } + + /// The parameters schema itself. + pub(crate) fn schema(&self) -> &'a Value { + self.schema + } + + /// The parameters schema after following `$ref`s and single-schema + /// `allOf`s. + pub(crate) fn resolved(&self) -> &'a Value { + self.resolve(self.schema) + } + + /// Follow `$ref`s and single-schema `allOf`s. A reference that does not + /// resolve, or a chain deeper than [`MAX_SCHEMA_DEPTH`], accepts any value. + fn resolve(&self, schema: &'a Value) -> &'a Value { + let mut schema = schema; + for _ in 0..=MAX_SCHEMA_DEPTH { + if let Some(reference) = schema.get("$ref").and_then(Value::as_str) { + match self.resolve_ref(reference) { + Some(target) => schema = target, + None => return &ANY_SCHEMA, + } + continue; + } + match schema.get("allOf").and_then(Value::as_array).map(Vec::as_slice) { + Some([inner]) => schema = inner, + _ => return schema, + } + } + &ANY_SCHEMA + } + + /// Every option of a value under `schema`. + /// + /// `const` and `enum` values become literal options, and `anyOf`, `oneOf`, + /// and type arrays one option per alternative, in schema order. An object + /// or array schema without `type` is typed by its `properties`, + /// `additionalProperties`, or `items`. `"nullable": true`, the OpenAPI + /// spelling of an optional value, adds a `null` option. Type names accept + /// the aliases that real tool schemas use, such as `int` or `dict`. + /// + /// Each reference target expands at most once. A union gains no values + /// from a repeated alternative, and a definition shared by several + /// alternatives would otherwise expand once per path to it, exponentially + /// in the nesting. A schema that constrains nothing, or whose references + /// only lead back into it, yields [`Self::any`]. + pub(crate) fn options(&self, schema: &'a Value) -> Vec> { + let mut expansion = Expansion::default(); + let mut options = self.options_at(schema, 0, &mut expansion); + if options.is_empty() && expansion.skipped { + options = Self::any(); + } + let nullable = schema.get("nullable").and_then(Value::as_bool) == Some(true); + if nullable && !options.iter().any(|option| option.ty == JsonType::Null) { + options.push(SchemaOption { + ty: JsonType::Null, + source: OptionSource::Any, + }); + } + options + } + + /// One option per JSON value type, each accepting any value of the type. + pub(crate) fn any() -> Vec> { + JsonType::VALUE_TYPES + .into_iter() + .map(|ty| SchemaOption { + ty, + source: OptionSource::Any, + }) + .collect() + } + + fn resolve_ref(&self, reference: &str) -> Option<&'a Value> { + self.schema.pointer(reference.strip_prefix('#')?) + } + + fn options_at( + &self, + schema: &'a Value, + depth: usize, + expansion: &mut Expansion, + ) -> Vec> { + let schema = match schema { + _ if depth > MAX_SCHEMA_DEPTH => return Self::any(), + Value::Bool(false) => return vec![], + Value::Object(schema) => schema, + _ => return Self::any(), + }; + if let Some(reference) = schema.get("$ref").and_then(Value::as_str) { + let Some(target) = self.resolve_ref(reference) else { + return Self::any(); + }; + if !expansion.expanded.insert(std::ptr::from_ref(target)) { + expansion.skipped = true; + return vec![]; + } + return self.options_at(target, depth + 1, expansion); + } + if let Some(value) = schema.get("const") { + return vec![SchemaOption::literal(value)]; + } + if let Some(values) = schema.get("enum").and_then(Value::as_array) { + return values.iter().map(SchemaOption::literal).collect(); + } + if let Some(options) = + schema.get("anyOf").or_else(|| schema.get("oneOf")).and_then(Value::as_array) + { + return options + .iter() + .flat_map(|option| self.options_at(option, depth + 1, expansion)) + .collect(); + } + if let Some(schemas) = schema.get("allOf").and_then(Value::as_array) { + return match schemas.as_slice() { + [schema] => self.options_at(schema, depth + 1, expansion), + _ => Self::any(), + }; + } + match schema.get("type") { + Some(Value::String(name)) => JsonType::parse(name) + .map(|ty| SchemaOption::typed(schema, ty)) + .into_iter() + .collect(), + Some(Value::Array(names)) => names + .iter() + .filter_map(Value::as_str) + .filter_map(JsonType::parse) + .map(|ty| SchemaOption::typed(schema, ty)) + .collect(), + Some(_) => Self::any(), + None if schema.contains_key("properties") + || schema.contains_key("additionalProperties") => + { + vec![SchemaOption::typed(schema, JsonType::Object)] + } + None if schema.contains_key("items") => { + vec![SchemaOption::typed(schema, JsonType::Array)] + } + None => Self::any(), + } + } +} + +impl<'a> SchemaOption<'a> { + /// The schema of a [`OptionSource::Schema`] option. + pub(crate) fn schema(&self) -> Option<&'a Map> { + match self.source { + OptionSource::Schema(schema) => Some(schema), + OptionSource::Literal(_) | OptionSource::Any => None, + } + } + + /// The option of `schema` that takes its values of type `ty`. + fn typed(schema: &'a Map, ty: JsonType) -> Self { + Self { + ty, + source: OptionSource::Schema(schema), + } + } + + fn literal(value: &'a Value) -> Self { + Self { + ty: JsonType::of_value(value), + source: OptionSource::Literal(value), + } + } +} + +/// `schema` with its `type` set to the canonical name of `ty`, copied only +/// when that changes it. +pub(crate) fn narrow(schema: &Map, ty: JsonType) -> Cow<'_, Map> { + if schema.get("type").and_then(Value::as_str) == Some(ty.name()) { + Cow::Borrowed(schema) + } else { + let mut schema = schema.clone(); + schema.insert("type".to_string(), Value::String(ty.name().to_string())); + Cow::Owned(schema) + } +} + +impl JsonType { + /// The types of JSON values, which do not distinguish integers. + const VALUE_TYPES: [Self; 6] = [ + Self::String, + Self::Number, + Self::Boolean, + Self::Null, + Self::Object, + Self::Array, + ]; + + /// The JSON Schema type name. + pub fn name(self) -> &'static str { + match self { + Self::String => "string", + Self::Integer => "integer", + Self::Number => "number", + Self::Boolean => "boolean", + Self::Null => "null", + Self::Object => "object", + Self::Array => "array", + } + } + + /// Schema accepting every JSON value of this type. + pub(crate) fn any_schema(self) -> Value { + match self { + Self::Object => json!({ "type": "object", "additionalProperties": true }), + Self::Array => json!({ "type": "array", "items": true }), + ty => json!({ "type": ty.name() }), + } + } + + /// Parse a schema type name, including aliases used by real tool schemas. + fn parse(name: &str) -> Option { + let name = name.trim().to_ascii_lowercase(); + Some(match name.as_str() { + "string" | "str" | "text" | "varchar" | "char" | "enum" => Self::String, + "integer" | "int" => Self::Integer, + "number" | "float" | "double" => Self::Number, + "boolean" | "bool" | "binary" => Self::Boolean, + "object" | "dict" | "map" => Self::Object, + "array" | "arr" | "list" | "sequence" => Self::Array, + "null" => Self::Null, + _ if name.starts_with("int") + || name.starts_with("uint") + || name.starts_with("long") + || name.starts_with("short") + || name.starts_with("unsigned") => + { + Self::Integer + } + _ if name.starts_with("num") || name.starts_with("float") => Self::Number, + _ if name.starts_with("dict") => Self::Object, + _ if name.starts_with("list") => Self::Array, + _ => return None, + }) + } + + fn of_value(value: &Value) -> Self { + match value { + Value::String(_) => Self::String, + Value::Number(number) if number.is_f64() => Self::Number, + Value::Number(_) => Self::Integer, + Value::Bool(_) => Self::Boolean, + Value::Null => Self::Null, + Value::Object(_) => Self::Object, + Value::Array(_) => Self::Array, + } + } +} + +#[cfg(test)] +mod tests { + use expect_test::{Expect, expect}; + use serde_json::json; + + use super::*; + + /// A Pydantic v2 `model_json_schema()` for a nested model, an enum, and an + /// optional nested model. + fn pydantic_schema() -> Value { + json!({ + "$defs": { + "Place": { + "properties": { "city": { "title": "City", "type": "string" } }, + "required": ["city"], + "title": "Place", + "type": "object" + }, + "Unit": { "enum": ["celsius", "fahrenheit"], "title": "Unit", "type": "string" } + }, + "properties": { + "place": { "$ref": "#/$defs/Place" }, + "unit": { "$ref": "#/$defs/Unit", "default": "celsius" }, + "backup": { "anyOf": [{ "$ref": "#/$defs/Place" }, { "type": "null" }], "default": null } + }, + "required": ["place"], + "title": "Args", + "type": "object" + }) + } + + fn check_options(root: &Value, schema: &Value, expected: Expect) { + let options = SchemaRoot::new(root) + .options(schema) + .into_iter() + .map(|option| match option.source { + OptionSource::Schema(schema) => format!( + "{} {}", + option.ty.name(), + Value::Object(narrow(schema, option.ty).into_owned()) + ), + OptionSource::Literal(value) => format!("{} = {value}", option.ty.name()), + OptionSource::Any => format!("{} any", option.ty.name()), + }) + .collect::>(); + expected.assert_debug_eq(&options); + } + + #[test] + fn references_resolve_to_their_definitions() { + let root = pydantic_schema(); + let properties = &root["properties"]; + check_options( + &root, + &properties["place"], + expect![[r#" + [ + "object {\"properties\":{\"city\":{\"title\":\"City\",\"type\":\"string\"}},\"required\":[\"city\"],\"title\":\"Place\",\"type\":\"object\"}", + ] + "#]], + ); + check_options( + &root, + &properties["unit"], + expect![[r#" + [ + "string = \"celsius\"", + "string = \"fahrenheit\"", + ] + "#]], + ); + check_options( + &root, + &properties["backup"], + expect![[r#" + [ + "object {\"properties\":{\"city\":{\"title\":\"City\",\"type\":\"string\"}},\"required\":[\"city\"],\"title\":\"Place\",\"type\":\"object\"}", + "null {\"type\":\"null\"}", + ] + "#]], + ); + } + + #[test] + fn unresolvable_and_cyclic_references_accept_any_value() { + let root = json!({ + "$defs": { "a": { "$ref": "#/$defs/b" }, "b": { "$ref": "#/$defs/a" } }, + "properties": { + "missing": { "$ref": "#/$defs/missing" }, + "cycle": { "$ref": "#/$defs/a" }, + "remote": { "$ref": "https://example.com/schema.json" } + } + }); + let schema_root = SchemaRoot::new(&root); + for key in ["missing", "cycle", "remote"] { + assert_eq!( + schema_root.resolve(&root["properties"][key]), + &ANY_SCHEMA, + "{key}" + ); + assert!( + schema_root + .options(&root["properties"][key]) + .iter() + .all(|option| matches!(option.source, OptionSource::Any)), + "{key}" + ); + } + } + + #[test] + fn options_normalize_common_schema_spellings() { + let root = json!({}); + check_options( + &root, + &json!({ "type": ["int", "null"], "minimum": 0 }), + expect![[r#" + [ + "integer {\"type\":\"integer\",\"minimum\":0}", + "null {\"type\":\"null\",\"minimum\":0}", + ] + "#]], + ); + check_options( + &root, + &json!({ "type": "string", "nullable": true }), + expect![[r#" + [ + "string {\"type\":\"string\",\"nullable\":true}", + "null any", + ] + "#]], + ); + check_options( + &root, + &json!({ "enum": ["a", 1, 1.5, null] }), + expect![[r#" + [ + "string = \"a\"", + "integer = 1", + "number = 1.5", + "null = null", + ] + "#]], + ); + check_options( + &root, + &json!({ "allOf": [{ "type": "boolean" }] }), + expect![[r#" + [ + "boolean {\"type\":\"boolean\"}", + ] + "#]], + ); + check_options( + &root, + &json!({ "items": { "type": "string" } }), + expect![[r#" + [ + "array {\"items\":{\"type\":\"string\"},\"type\":\"array\"}", + ] + "#]], + ); + } +} diff --git a/rust/src/parser/src/tool/deepseek_dsml/deepseek_v41/tests.rs b/rust/src/parser/src/tool/deepseek_dsml/deepseek_v41/tests.rs index 208d88ea33..58043cc95c 100644 --- a/rust/src/parser/src/tool/deepseek_dsml/deepseek_v41/tests.rs +++ b/rust/src/parser/src/tool/deepseek_dsml/deepseek_v41/tests.rs @@ -28,6 +28,7 @@ fn tools() -> Vec { } }), strict: None, + defer_loading: None, }] } diff --git a/rust/src/parser/src/tool/json/mod.rs b/rust/src/parser/src/tool/json/mod.rs index 1bc5058873..487be423ae 100644 --- a/rust/src/parser/src/tool/json/mod.rs +++ b/rust/src/parser/src/tool/json/mod.rs @@ -378,9 +378,16 @@ fn safe_text_event( input: &mut JsonToolInput<'_>, config: JsonToolCallConfig, ) -> ModalResult { + // Stop only where a tool call can start: with exact whitespace, a bare + // start marker without it is text. + let with_whitespace = |marker: &str| match config.marker_whitespace { + JsonToolCallWhitespace::Optional => marker.to_string(), + JsonToolCallWhitespace::Exact(whitespace) => format!("{marker}{whitespace}"), + }; + let start_marker = with_whitespace(config.start_marker); match config.framed_start_marker { - Some(marker) => safe_text_len_mul(input, &[marker, config.start_marker]), - None => safe_text_len(input, config.start_marker), + Some(marker) => safe_text_len_mul(input, &[with_whitespace(marker), start_marker]), + None => safe_text_len(input, start_marker), } .map(|len| JsonToolCallEvent::Text { len }) } diff --git a/rust/src/parser/src/tool/json/qwen.rs b/rust/src/parser/src/tool/json/qwen.rs index ca7c5919cf..c4db8c2fbc 100644 --- a/rust/src/parser/src/tool/json/qwen.rs +++ b/rust/src/parser/src/tool/json/qwen.rs @@ -205,6 +205,23 @@ mod tests { assert!(output.calls().is_empty()); } + #[test] + fn qwen_xml_streams_start_marker_without_newline_as_text() { + for chunk in [ + "{\"name\":\"get_weather\"", + "hi\n{\"name\":\"get_weather\"", + ] { + let mut parser = Qwen3XmlToolParser::new(&test_tools()); + let mut output = ToolParserOutput::default(); + + parser.parse_into(chunk, &mut output).unwrap(); + + // Not a tool-call start: released without waiting for the end of the stream. + assert_eq!(output.normal_text(), chunk); + assert!(output.calls().is_empty()); + } + } + #[test] fn qwen_xml_requires_newline_before_tool_call_end() { let mut parser = Qwen3XmlToolParser::new(&test_tools()); diff --git a/rust/src/parser/src/tool/mimo.rs b/rust/src/parser/src/tool/mimo.rs index f6491e9caa..f7ba06fb06 100644 --- a/rust/src/parser/src/tool/mimo.rs +++ b/rust/src/parser/src/tool/mimo.rs @@ -65,6 +65,7 @@ mod tests { name: "convert".into(), description: None, strict: None, + defer_loading: None, parameters: json!({"type": "object", "properties": { "flag": {"type": "boolean"}, "empty": {"type": ["string", "null"]}, "text": {"type": "string"}, "literal": {"type": "string"}, "payload": {"type": "object"} diff --git a/rust/src/parser/src/tool/minimax_m3.rs b/rust/src/parser/src/tool/minimax_m3.rs index b0643dd1a1..b6e179917e 100644 --- a/rust/src/parser/src/tool/minimax_m3.rs +++ b/rust/src/parser/src/tool/minimax_m3.rs @@ -459,6 +459,7 @@ mod tests { } }), strict: None, + defer_loading: None, }); tools } diff --git a/rust/src/parser/src/tool/mod.rs b/rust/src/parser/src/tool/mod.rs index 7789752626..e4afbda3e9 100644 --- a/rust/src/parser/src/tool/mod.rs +++ b/rust/src/parser/src/tool/mod.rs @@ -52,6 +52,9 @@ pub struct Tool { pub description: Option, pub parameters: Value, pub strict: Option, + /// Whether the tool's schema is loaded on demand (tool search). Only chat + /// templates consume it; the tool stays callable for parsing and grammars. + pub defer_loading: Option, } /// One tool-call update emitted while parsing assistant text. diff --git a/rust/src/parser/src/tool/parameters.rs b/rust/src/parser/src/tool/parameters.rs index 19d700830b..391ca90f10 100644 --- a/rust/src/parser/src/tool/parameters.rs +++ b/rust/src/parser/src/tool/parameters.rs @@ -5,6 +5,7 @@ use std::collections::BTreeMap; use serde_json::{Map, Number, Value}; +use crate::schema::{JsonType, OptionSource, SchemaRoot}; use crate::tool::Tool; /// Normalized parameter schemas for all tools in one request. @@ -21,7 +22,9 @@ pub(crate) struct ToolSchemas { /// downstream tool call execution. #[derive(Debug, Clone, Default, PartialEq, Eq)] pub(super) struct ToolSchema { - params: BTreeMap, + /// The tool's parameters schema, read one level at a time as values are + /// converted, so recursive schemas cost only as deep as the value. + parameters: Value, } /// Parameter input for schema-aware conversion. @@ -47,22 +50,38 @@ pub(crate) struct ParamElement { pub value: ParamInput, } -/// Normalized JSON parameter type used for raw string coercion. +/// One level of a parameter's normalized type, used for raw string coercion. +/// +/// Object and array types normalize their children only when a structured +/// value reaches them. #[derive(Debug, Clone, PartialEq, Eq)] -pub(super) enum JsonParamType { +enum JsonParamType<'a> { String, Integer, Number, Boolean, - Object { - properties: BTreeMap, - additional_properties: Option>, - }, - Array { - items: Option>, - }, + Object(ObjectType<'a>), + Array(ArrayType<'a>), Null, - OneOf(Vec), + /// A non-string `const` or `enum` value: text spelling it as JSON decodes + /// to it. + Literal(&'a Value), + OneOf(Vec>), +} + +/// An object type whose field types are normalized on access. +#[derive(Debug, Clone, PartialEq, Eq)] +struct ObjectType<'a> { + root: SchemaRoot<'a>, + properties: Option<&'a Map>, + additional_properties: Option<&'a Value>, +} + +/// An array type whose item type is normalized on access. +#[derive(Debug, Clone, PartialEq, Eq)] +struct ArrayType<'a> { + root: SchemaRoot<'a>, + items: Option<&'a Value>, } impl ToolSchemas { @@ -70,7 +89,12 @@ impl ToolSchemas { pub(crate) fn from_tools(tools: &[Tool]) -> Self { let tools = tools .iter() - .map(|tool| (tool.name.clone(), ToolSchema::from_schema(&tool.parameters))) + .map(|tool| { + ( + tool.name.clone(), + ToolSchema::from_schema(tool.parameters.clone()), + ) + }) .collect(); Self { tools } @@ -117,25 +141,14 @@ impl ToolSchema { /// parameters to be treated as strings. const fn empty() -> &'static Self { static EMPTY: ToolSchema = ToolSchema { - params: BTreeMap::new(), + parameters: Value::Null, }; &EMPTY } - /// Normalize an OpenAI-style tool parameters JSON schema. - fn from_schema(parameters: &Value) -> Self { - let Some(properties) = parameters.get("properties").and_then(Value::as_object) else { - return Self::default(); - }; - - let params = properties - .iter() - .filter_map(|(name, schema)| { - JsonParamType::from_schema(schema).map(|param_type| (name.clone(), param_type)) - }) - .collect(); - - Self { params } + /// Keep an OpenAI-style tool parameters JSON schema. + fn from_schema(parameters: Value) -> Self { + Self { parameters } } /// Convert one parameter value using its normalized schema type. @@ -144,169 +157,86 @@ impl ToolSchema { /// the value fails to convert, this falls back to returning the raw /// string as a JSON string value, or object-like JSON for structured input. fn convert(&self, name: &str, input: ParamInput) -> Value { - convert_with_optional_schema(self.params.get(name), &input) + let root = SchemaRoot::new(&self.parameters); + let param_type = root + .resolved() + .get("properties") + .and_then(|properties| properties.get(name)) + .and_then(|schema| JsonParamType::from_schema(&root, schema)); + convert_with_optional_schema(param_type.as_ref(), &input) } } -impl JsonParamType { - /// Normalize one parameter property schema. - /// - /// `"nullable": true` (the OpenAPI spelling of an optional parameter) adds - /// `null` to the accepted types. - fn from_schema(schema: &Value) -> Option { - let param_type = Self::from_schema_ignoring_nullable(schema)?; - let nullable = schema.get("nullable").and_then(Value::as_bool) == Some(true); - Some(match param_type { - _ if !nullable || param_type.accepts_null() => param_type, - Self::OneOf(mut types) => { - types.push(Self::Null); - Self::OneOf(types) - } - param_type => Self::OneOf(vec![param_type, Self::Null]), - }) - } - - /// Return whether `null` is one of the accepted types. - fn accepts_null(&self) -> bool { - match self { - Self::Null => true, - Self::OneOf(types) => types.iter().any(Self::accepts_null), - _ => false, +impl<'a> JsonParamType<'a> { + /// Normalize one parameter or nested value schema, or `None` when it does + /// not constrain the value's type. + fn from_schema(root: &SchemaRoot<'a>, schema: &'a Value) -> Option { + let options = root.options(schema); + if options.iter().all(|option| matches!(option.source, OptionSource::Any)) { + return None; } - } - - /// Normalize one parameter property schema, without `nullable`. - fn from_schema_ignoring_nullable(schema: &Value) -> Option { - let schema = schema.as_object()?; - - if let Some(type_value) = schema.get("type") { - return Self::from_type_value(type_value, schema); - } - - if let Some(composite) = schema.get("anyOf").or_else(|| schema.get("oneOf")) { - let param_type = composite - .as_array() - .map(|schemas| schemas.iter().filter_map(Self::from_schema).collect::>()) - .filter(|types| !types.is_empty()) - .map(Self::one_of) - .unwrap_or_else(|| Self::object_from_schema(Some(schema))); - return Some(param_type); - } - - // Typically, these types are already handled by checking the "type" field, but - // we can also infer them from their characteristic fields if "type" is missing. - if let Some(values) = schema.get("enum").and_then(Value::as_array) { - // Enum values are treated as strings, except that a `null` member - // makes the parameter nullable (mirrors Python's enum type - // inference), so a literal "null" coerces to JSON null. - if values.iter().any(Value::is_null) { - return Some(Self::one_of(vec![Self::String, Self::Null])); - } - return Some(Self::String); - } - if schema.contains_key("items") { - return Some(Self::array_from_schema(Some(schema))); - } - if schema.contains_key("properties") || schema.contains_key("additionalProperties") { - return Some(Self::object_from_schema(Some(schema))); - } - - None - } - - /// Normalize a JSON schema `type` value. - fn from_type_value(type_value: &Value, schema: &Map) -> Option { - match type_value { - Value::String(kind) => Self::from_type_name(kind, Some(schema)), - Value::Array(kinds) => { - let types = kinds - .iter() - .filter_map(Value::as_str) - .filter_map(|kind| Self::from_type_name(kind, Some(schema))) - .collect::>(); - if types.is_empty() { - None - } else { - Some(Self::one_of(types)) - } - } - _ => None, - } - } - - /// Normalize one JSON schema type name. - fn from_type_name(kind: &str, schema: Option<&Map>) -> Option { - let kind = kind.trim().to_ascii_lowercase(); - match kind.as_str() { - "string" | "str" | "text" | "varchar" | "char" | "enum" => Some(Self::String), - "integer" | "int" => Some(Self::Integer), - "number" | "float" | "double" => Some(Self::Number), - "boolean" | "bool" | "binary" => Some(Self::Boolean), - "object" | "dict" | "map" => Some(Self::object_from_schema(schema)), - "array" | "arr" | "list" | "sequence" => Some(Self::array_from_schema(schema)), - "null" => Some(Self::Null), - _ if kind.starts_with("int") - || kind.starts_with("uint") - || kind.starts_with("long") - || kind.starts_with("short") - || kind.starts_with("unsigned") => + let mut types = Vec::new(); + for option in &options { + // String literals stay with the string type, which keeps text + // verbatim instead of unquoting text that spells one as JSON. + if let OptionSource::Literal(value) = option.source + && !value.is_string() + && !types.contains(&Self::Literal(value)) { - Some(Self::Integer) + types.push(Self::Literal(value)); } - _ if kind.starts_with("num") || kind.starts_with("float") => Some(Self::Number), - _ if kind.starts_with("dict") => Some(Self::object_from_schema(schema)), - _ if kind.starts_with("list") => Some(Self::array_from_schema(schema)), - _ => None, + let schema = option.schema(); + let param_type = match option.ty { + JsonType::String => Self::String, + JsonType::Integer => Self::Integer, + JsonType::Number => Self::Number, + JsonType::Boolean => Self::Boolean, + JsonType::Null => Self::Null, + JsonType::Object => Self::Object(ObjectType { + root: *root, + properties: schema + .and_then(|schema| schema.get("properties")) + .and_then(Value::as_object), + additional_properties: schema + .and_then(|schema| schema.get("additionalProperties")) + .filter(|schema| schema.is_object()), + }), + JsonType::Array => Self::Array(ArrayType { + root: *root, + items: schema.and_then(|schema| schema.get("items")), + }), + }; + if !types.contains(¶m_type) { + types.push(param_type); + } + } + // Text spelling a literal decodes to it before any type, a string in + // particular, accepts the same text. + types.sort_by_key(|param_type| !matches!(param_type, Self::Literal(_))); + match types.len() { + 0 => None, + 1 => types.pop(), + _ => Some(Self::OneOf(types)), } } +} - /// Normalize object schema fields. - fn object_from_schema(schema: Option<&Map>) -> Self { - let properties = schema - .and_then(|schema| schema.get("properties")) - .and_then(Value::as_object) - .map(|properties| { - properties - .iter() - .filter_map(|(name, schema)| { - Self::from_schema(schema).map(|param_type| (name.clone(), param_type)) - }) - .collect() - }) - .unwrap_or_default(); - - let additional_properties = - schema.and_then(|schema| schema.get("additionalProperties")).and_then(|schema| { - if schema.is_object() { - Self::from_schema(schema).map(Box::new) - } else { - None - } - }); - - Self::Object { - properties, - additional_properties, - } +impl<'a> ObjectType<'a> { + /// The type of the field named `name`, from its declared schema or the + /// additional properties' schema. + fn field(&self, name: &str) -> Option> { + let schema = self + .properties + .and_then(|properties| properties.get(name)) + .or(self.additional_properties)?; + JsonParamType::from_schema(&self.root, schema) } +} - /// Normalize array schema fields. - fn array_from_schema(schema: Option<&Map>) -> Self { - let items = schema - .and_then(|schema| schema.get("items")) - .and_then(Self::from_schema) - .map(Box::new); - - Self::Array { items } - } - - /// Collapse a candidate type list into one normalized type. - fn one_of(mut types: Vec) -> Self { - if types.len() == 1 { - types.remove(0) - } else { - Self::OneOf(types) - } +impl<'a> ArrayType<'a> { + /// The type of the array's items. + fn items(&self) -> Option> { + JsonParamType::from_schema(&self.root, self.items?) } } @@ -318,7 +248,10 @@ fn is_null_literal(value: &str) -> bool { } /// Convert one parameter input to a normalized JSON value. -fn convert_with_optional_schema(param_type: Option<&JsonParamType>, input: &ParamInput) -> Value { +fn convert_with_optional_schema( + param_type: Option<&JsonParamType<'_>>, + input: &ParamInput, +) -> Value { // Coerce the literal text `null` to JSON null, except for `string`-typed // params, where it must stay the string "null": a model emitting the literal // text "null" for a string field means the string, not a missing value. @@ -341,13 +274,13 @@ fn convert_with_optional_schema(param_type: Option<&JsonParamType>, input: &Para ParamInput::Text(value) => Value::String(value.clone()), ParamInput::Elements(elements) => { // Convert structured input to object without a schema. - Value::Object(convert_elements_to_object(elements, &BTreeMap::new(), None)) + Value::Object(convert_elements_to_object(elements, None)) } } } /// Convert one parameter input to a normalized JSON type. -fn try_convert_value(param_type: &JsonParamType, input: &ParamInput) -> Option { +fn try_convert_value(param_type: &JsonParamType<'_>, input: &ParamInput) -> Option { match input { ParamInput::Text(value) => try_convert_text_value(param_type, value), ParamInput::Elements(elements) => try_convert_elements_value(param_type, elements), @@ -358,18 +291,21 @@ fn try_convert_value(param_type: &JsonParamType, input: &ParamInput) -> Option Option { +fn try_convert_text_value(param_type: &JsonParamType<'_>, value: &str) -> Option { match param_type { JsonParamType::String => Some(Value::String(value.to_string())), JsonParamType::Integer => convert_integer_text(value), JsonParamType::Number => convert_number_text(value), JsonParamType::Boolean => try_convert_boolean(value), - JsonParamType::Object { .. } if value.trim().is_empty() => Some(Value::Object(Map::new())), - JsonParamType::Array { .. } if value.trim().is_empty() => Some(Value::Array(Vec::new())), + JsonParamType::Object(_) if value.trim().is_empty() => Some(Value::Object(Map::new())), + JsonParamType::Array(_) if value.trim().is_empty() => Some(Value::Array(Vec::new())), // For composite types with string input, interpret the string as JSON of that type. - JsonParamType::Object { .. } => serde_json::from_str(value).ok().filter(Value::is_object), - JsonParamType::Array { .. } => serde_json::from_str(value).ok().filter(Value::is_array), + JsonParamType::Object(_) => serde_json::from_str(value).ok().filter(Value::is_object), + JsonParamType::Array(_) => serde_json::from_str(value).ok().filter(Value::is_array), JsonParamType::Null => is_null_literal(value).then_some(Value::Null), + JsonParamType::Literal(literal) => serde_json::from_str::(value.trim()) + .ok() + .filter(|value| value == *literal), JsonParamType::OneOf(types) => { types.iter().find_map(|param_type| try_convert_text_value(param_type, value)) } @@ -378,25 +314,24 @@ fn try_convert_text_value(param_type: &JsonParamType, value: &str) -> Option, elements: &[ParamElement], ) -> Option { match param_type { - JsonParamType::Object { - properties, - additional_properties, - } => Some(Value::Object(convert_elements_to_object( + JsonParamType::Object(object) => Some(Value::Object(convert_elements_to_object( elements, - properties, - additional_properties.as_deref(), + Some(object), ))), - JsonParamType::Array { items } => Some(Value::Array( - // Collect all child elements into an array, regardless of their names. - elements - .iter() - .map(|element| convert_with_optional_schema(items.as_deref(), &element.value)) - .collect(), - )), + JsonParamType::Array(array) => { + let items = array.items(); + Some(Value::Array( + // Collect all child elements into an array, regardless of their names. + elements + .iter() + .map(|element| convert_with_optional_schema(items.as_ref(), &element.value)) + .collect(), + )) + } JsonParamType::OneOf(types) => types .iter() .find_map(|param_type| try_convert_elements_value(param_type, elements)), @@ -406,20 +341,20 @@ fn try_convert_elements_value( | JsonParamType::Integer | JsonParamType::Number | JsonParamType::Boolean - | JsonParamType::Null => None, + | JsonParamType::Null + | JsonParamType::Literal(_) => None, } } /// Convert structured elements to an object, using field schemas when present. fn convert_elements_to_object( elements: &[ParamElement], - properties: &BTreeMap, - additional_properties: Option<&JsonParamType>, + object_type: Option<&ObjectType<'_>>, ) -> Map { let mut object = Map::with_capacity(elements.len()); for element in elements { - let param_type = properties.get(&element.name).or(additional_properties); - let value = convert_with_optional_schema(param_type, &element.value); + let param_type = object_type.and_then(|object_type| object_type.field(&element.name)); + let value = convert_with_optional_schema(param_type.as_ref(), &element.value); insert_object_value(&mut object, element.name.clone(), value); } object @@ -502,12 +437,13 @@ mod tests { description: None, parameters, strict: None, + defer_loading: None, } } #[test] fn invalid_schema_converts_everything_as_string() { - let params = ToolSchema::from_schema(&json!({ "type": "object" })); + let params = ToolSchema::from_schema(json!({ "type": "object" })); assert_eq!(params.convert("count", text("42")), json!("42")); assert_eq!(params.convert("count", text("null")), json!(null)); @@ -515,7 +451,7 @@ mod tests { #[test] fn skips_unknown_property_schema_and_unknown_type() { - let params = ToolSchema::from_schema(&json!({ + let params = ToolSchema::from_schema(json!({ "type": "object", "properties": { "unknown_schema": true, @@ -531,7 +467,7 @@ mod tests { #[test] fn converts_supported_types() { - let params = ToolSchema::from_schema(&json!({ + let params = ToolSchema::from_schema(json!({ "type": "object", "properties": { "text": { "type": "string" }, @@ -570,7 +506,7 @@ mod tests { #[test] fn number_conversion_preserves_json_number_spelling_with_legacy_fallback() { - let params = ToolSchema::from_schema(&json!({ + let params = ToolSchema::from_schema(json!({ "type": "object", "properties": { "value": { "type": "number" } @@ -617,7 +553,7 @@ mod tests { #[test] fn non_string_values_ignore_surrounding_whitespace() { - let params = ToolSchema::from_schema(&json!({ + let params = ToolSchema::from_schema(json!({ "type": "object", "properties": { "text": { "type": "string" }, @@ -640,7 +576,7 @@ mod tests { #[test] fn numbers_accept_digit_separators() { - let params = ToolSchema::from_schema(&json!({ + let params = ToolSchema::from_schema(json!({ "type": "object", "properties": { "count": { "type": "integer" }, @@ -657,7 +593,7 @@ mod tests { #[test] fn nullable_schemas_accept_null() { - let params = ToolSchema::from_schema(&json!({ + let params = ToolSchema::from_schema(json!({ "type": "object", "properties": { "text": { "type": "string", "nullable": true }, @@ -674,7 +610,7 @@ mod tests { #[test] fn composite_values_must_decode_to_their_type() { - let params = ToolSchema::from_schema(&json!({ + let params = ToolSchema::from_schema(json!({ "type": "object", "properties": { "payload": { "type": "object" }, @@ -694,7 +630,7 @@ mod tests { #[test] fn converts_upstream_aliases() { - let params = ToolSchema::from_schema(&json!({ + let params = ToolSchema::from_schema(json!({ "type": "object", "properties": { "s": { "type": "varchar" }, @@ -716,13 +652,13 @@ mod tests { #[test] fn preserves_union_type_order() { - let integer_first = ToolSchema::from_schema(&json!({ + let integer_first = ToolSchema::from_schema(json!({ "type": "object", "properties": { "value": { "type": ["integer", "string"] } } })); - let string_first = ToolSchema::from_schema(&json!({ + let string_first = ToolSchema::from_schema(json!({ "type": "object", "properties": { "value": { "type": ["string", "integer"] } @@ -735,7 +671,7 @@ mod tests { #[test] fn converts_composite_schemas() { - let params = ToolSchema::from_schema(&json!({ + let params = ToolSchema::from_schema(json!({ "type": "object", "properties": { "choice": { @@ -744,7 +680,7 @@ mod tests { { "type": "string" } ] }, - "fallback_object": { + "unknown_alternatives": { "oneOf": [ { "type": "mystery" } ] @@ -753,15 +689,139 @@ mod tests { })); assert_eq!(params.convert("choice", text("42")), json!(42)); + // Alternatives of unknown types leave the value unconstrained, like an + // unknown `type`. assert_eq!( - params.convert("fallback_object", text(r#"{"x":1}"#)), - json!({ "x": 1 }) + params.convert("unknown_alternatives", text(r#"{"x":1}"#)), + json!(r#"{"x":1}"#) ); } + #[test] + fn resolves_references_in_pydantic_schemas() { + let params = ToolSchema::from_schema(json!({ + "$defs": { + "Place": { + "properties": { "city": { "type": "string" }, "days": { "type": "integer" } }, + "type": "object" + }, + "Count": { "type": "integer" } + }, + "type": "object", + "properties": { + "place": { "$ref": "#/$defs/Place" }, + "backup": { "anyOf": [{ "$ref": "#/$defs/Place" }, { "type": "null" }] }, + "count": { "$ref": "#/$defs/Count" } + } + })); + + assert_eq!( + params.convert("place", text(r#"{"city": "Paris", "days": 3}"#)), + json!({ "city": "Paris", "days": 3 }) + ); + assert_eq!( + params.convert("backup", text(r#"{"city": "Paris"}"#)), + json!({ "city": "Paris" }) + ); + assert_eq!(params.convert("backup", text("null")), Value::Null); + assert_eq!(params.convert("count", text("3")), json!(3)); + } + + #[test] + fn recursive_references_convert_without_unbounded_expansion() { + let params = ToolSchema::from_schema(json!({ + "$defs": { + "Node": { + "type": "object", + "properties": { + "id": { "type": "integer" }, + "children": { "type": "array", "items": { "$ref": "#/$defs/Node" } } + } + } + }, + "type": "object", + "properties": { "tree": { "$ref": "#/$defs/Node" } } + })); + + assert_eq!( + params.convert( + "tree", + text(r#"{"id": 1, "children": [{"id": 2, "children": []}]}"#) + ), + json!({ "id": 1, "children": [{ "id": 2, "children": [] }] }) + ); + } + + #[test] + fn branching_recursive_references_convert_structured_values_by_level() { + // Two recursive children per node expand to 2^depth types if the schema + // is walked ahead of the value. + let params = ToolSchema::from_schema(json!({ + "$defs": { + "Tree": { + "type": "object", + "properties": { + "value": { "type": "integer" }, + "left": { "$ref": "#/$defs/Tree" }, + "right": { "$ref": "#/$defs/Tree" } + } + } + }, + "type": "object", + "properties": { "tree": { "$ref": "#/$defs/Tree" } } + })); + + let leaf = |value: &str| ParamInput::Elements(vec![elem("value", text(value))]); + let tree = ParamInput::Elements(vec![ + elem("value", text("1")), + elem( + "left", + ParamInput::Elements(vec![elem("value", text("2")), elem("right", leaf("3"))]), + ), + elem("right", leaf("4")), + ]); + assert_eq!( + params.convert("tree", tree), + json!({ + "value": 1, + "left": { "value": 2, "right": { "value": 3 } }, + "right": { "value": 4 } + }) + ); + } + + #[test] + fn mixed_enums_decode_text_spelling_a_non_string_literal_to_it() { + let params = ToolSchema::from_schema(json!({ + "type": "object", + "properties": { + "mode": { "enum": ["fast", 1] }, + "flag": { "enum": ["auto", true] } + } + })); + + assert_eq!(params.convert("mode", text("1")), json!(1)); + assert_eq!(params.convert("mode", text("fast")), json!("fast")); + // Text matching no literal keeps the schema order, which tries the + // string first. + assert_eq!(params.convert("mode", text("2")), json!("2")); + assert_eq!(params.convert("flag", text("true")), json!(true)); + assert_eq!(params.convert("flag", text("auto")), json!("auto")); + } + + #[test] + fn integer_enums_convert_to_numbers() { + let params = ToolSchema::from_schema(json!({ + "type": "object", + "properties": { "level": { "enum": [1, 2, 3] } } + })); + + assert_eq!(params.convert("level", text("2")), json!(2)); + } + #[test] fn infers_type_from_schema_shape_without_type() { - let params = ToolSchema::from_schema(&json!({ + let params = ToolSchema::from_schema(json!({ "type": "object", "properties": { "choice": { "enum": ["a", "b"] }, @@ -844,7 +904,7 @@ mod tests { // A `string`-typed param whose value is the literal text "null"/"NULL" // must stay a string (the original case is preserved), rather than being // coerced to JSON null. Non-string types keep coercing "null" to null. - let params = ToolSchema::from_schema(&json!({ + let params = ToolSchema::from_schema(json!({ "type": "object", "properties": { "name": { "type": "string" }, @@ -867,7 +927,7 @@ mod tests { // must coerce to JSON null (matching Python's `extract_types_from_schema`, // which infers `null` from the enum values), while a non-null enum keeps // "null" as a string. - let params = ToolSchema::from_schema(&json!({ + let params = ToolSchema::from_schema(json!({ "type": "object", "properties": { "mode": { "enum": [null, "auto"] }, diff --git a/rust/src/parser/src/tool/qwen_coder.rs b/rust/src/parser/src/tool/qwen_coder.rs index e233f4720c..e804ce0bb2 100644 --- a/rust/src/parser/src/tool/qwen_coder.rs +++ b/rust/src/parser/src/tool/qwen_coder.rs @@ -306,7 +306,7 @@ mod tests { use serde_json::{Value, json}; use thiserror_ext::AsReport; - use super::{Qwen3CoderToolParser, ToolParser}; + use super::{Qwen3CoderToolParser, Tool, ToolParser}; use crate::tool::test_utils::{collect_stream, split_by_chars, test_tools}; use crate::tool::tests::assert_tool_framing_preserves_body_whitespace; use crate::tool::{ToolParserOutput, ToolParserTestExt as _}; @@ -545,6 +545,39 @@ mod tests { ); } + #[test] + fn qwen_coder_parse_complete_resolves_schema_references() { + let tools = [Tool { + name: "plan_trip".to_string(), + description: None, + parameters: json!({ + "$defs": { + "Place": { "type": "object", "properties": { "city": { "type": "string" } } }, + "Days": { "type": "integer" } + }, + "type": "object", + "properties": { + "place": { "$ref": "#/$defs/Place" }, + "days": { "$ref": "#/$defs/Days" } + } + }), + strict: None, + defer_loading: None, + }]; + let mut parser = Qwen3CoderToolParser::new(&tools); + let output = parser + .parse_complete(&build_tool_call( + "plan_trip", + &[("place", "\n{\"city\": \"Paris\"}\n"), ("days", "\n3\n")], + )) + .unwrap(); + + assert_eq!( + serde_json::from_str::(&output.calls()[0].arguments).unwrap(), + json!({ "place": { "city": "Paris" }, "days": 3 }) + ); + } + #[test] fn qwen_coder_streaming_extracts_single_tool_call() { let mut parser = Qwen3CoderToolParser::new(&test_tools()); diff --git a/rust/src/parser/src/tool/test_utils.rs b/rust/src/parser/src/tool/test_utils.rs index ef3143183f..ec45a31ec1 100644 --- a/rust/src/parser/src/tool/test_utils.rs +++ b/rust/src/parser/src/tool/test_utils.rs @@ -24,6 +24,7 @@ pub fn test_tools() -> Vec { } }), strict: None, + defer_loading: None, }, Tool { name: "add".to_string(), @@ -36,6 +37,7 @@ pub fn test_tools() -> Vec { } }), strict: None, + defer_loading: None, }, Tool { name: "convert".to_string(), @@ -51,6 +53,7 @@ pub fn test_tools() -> Vec { } }), strict: None, + defer_loading: None, }, Tool { name: "calculate_area".to_string(), @@ -64,6 +67,7 @@ pub fn test_tools() -> Vec { } }), strict: None, + defer_loading: None, }, Tool { name: "update_record".to_string(), @@ -80,6 +84,7 @@ pub fn test_tools() -> Vec { } }), strict: None, + defer_loading: None, }, ] } diff --git a/rust/src/parser/src/unified/combined.rs b/rust/src/parser/src/unified/combined.rs index 6f6849ec95..ba44e7fb96 100644 --- a/rust/src/parser/src/unified/combined.rs +++ b/rust/src/parser/src/unified/combined.rs @@ -197,6 +197,7 @@ mod tests { }, }), strict: None, + defer_loading: None, }] } diff --git a/rust/src/parser/src/unified/gemma4.rs b/rust/src/parser/src/unified/gemma4.rs index 2f90555b26..1c65a7171c 100644 --- a/rust/src/parser/src/unified/gemma4.rs +++ b/rust/src/parser/src/unified/gemma4.rs @@ -617,48 +617,56 @@ mod tests { description: None, parameters: json!({ "type": "object" }), strict: None, + defer_loading: None, }, Tool { name: "get_time".to_string(), description: None, parameters: json!({ "type": "object" }), strict: None, + defer_loading: None, }, Tool { name: "write_file".to_string(), description: None, parameters: json!({ "type": "object" }), strict: None, + defer_loading: None, }, Tool { name: "Edit".to_string(), description: None, parameters: json!({ "type": "object" }), strict: None, + defer_loading: None, }, Tool { name: "search".to_string(), description: None, parameters: json!({ "type": "object" }), strict: None, + defer_loading: None, }, Tool { name: "set".to_string(), description: None, parameters: json!({ "type": "object" }), strict: None, + defer_loading: None, }, Tool { name: "get_status".to_string(), description: None, parameters: json!({ "type": "object" }), strict: None, + defer_loading: None, }, Tool { name: "todowrite".to_string(), description: None, parameters: json!({ "type": "object" }), strict: None, + defer_loading: None, }, ] } diff --git a/rust/src/parser/src/unified/hf/parser.rs b/rust/src/parser/src/unified/hf/parser.rs index f523c0f31a..5d5065f109 100644 --- a/rust/src/parser/src/unified/hf/parser.rs +++ b/rust/src/parser/src/unified/hf/parser.rs @@ -599,7 +599,7 @@ fn parse_next_event( match safe_text_len_mul(input, &markers) { Ok(_) => return Ok(HfEvent::Text), // A marker, complete or partial, starts here (or the input is empty). - Err(ErrMode::Incomplete(_)) => {} + Err(ErrMode::Backtrack(_) | ErrMode::Incomplete(_)) => {} Err(error) => return Err(error), } diff --git a/rust/src/parser/src/unified/hf/tests.rs b/rust/src/parser/src/unified/hf/tests.rs index 7e27a42805..ed2efa6f39 100644 --- a/rust/src/parser/src/unified/hf/tests.rs +++ b/rust/src/parser/src/unified/hf/tests.rs @@ -686,6 +686,7 @@ fn tools_convert_string_arguments() { }, }), strict: None, + defer_loading: None, }]; let text = "\n\n\n7\n\n\ntrue\n\n\nwake up\n\n12\n\n"; assert_eq!( @@ -888,6 +889,7 @@ fn differential_against_transformers() { description: None, parameters: tool["function"]["parameters"].clone(), strict: None, + defer_loading: None, }) .collect(); let expected = match DIVERGENCES.iter().find(|(divergent, _)| *divergent == text) { diff --git a/rust/src/parser/src/unified/hy/hy_v3.rs b/rust/src/parser/src/unified/hy/hy_v3.rs index 30e021b5bc..3599ce2cb9 100644 --- a/rust/src/parser/src/unified/hy/hy_v3.rs +++ b/rust/src/parser/src/unified/hy/hy_v3.rs @@ -114,6 +114,7 @@ mod tests { "properties": { "city": { "type": "string" } }, }), strict: None, + defer_loading: None, }] } diff --git a/rust/src/parser/src/unified/hy/hy_v4.rs b/rust/src/parser/src/unified/hy/hy_v4.rs index 777688bcaa..a5d81e0f2a 100644 --- a/rust/src/parser/src/unified/hy/hy_v4.rs +++ b/rust/src/parser/src/unified/hy/hy_v4.rs @@ -114,12 +114,14 @@ mod tests { "properties": { "city": { "type": "string" } }, }), strict: None, + defer_loading: None, }, Tool { name: "get_current_date".to_string(), description: None, parameters: json!({ "type": "object", "properties": {} }), strict: None, + defer_loading: None, }, ] } diff --git a/rust/src/parser/src/unified/inkling.rs b/rust/src/parser/src/unified/inkling.rs index 4372cafb24..7d819b0640 100644 --- a/rust/src/parser/src/unified/inkling.rs +++ b/rust/src/parser/src/unified/inkling.rs @@ -557,6 +557,7 @@ mod tests { }, }), strict: None, + defer_loading: None, }] } diff --git a/rust/src/parser/src/unified/kimi_k3/structural_tag.rs b/rust/src/parser/src/unified/kimi_k3/structural_tag.rs index 8f0f098f06..4754726c1b 100644 --- a/rust/src/parser/src/unified/kimi_k3/structural_tag.rs +++ b/rust/src/parser/src/unified/kimi_k3/structural_tag.rs @@ -46,12 +46,12 @@ impl ArgumentSyntax for XtmlArguments { } ParameterKey::Free => ParameterKey::Free, }; - let arguments = group_by(options, |option| option.ty) + let arguments = group_by(options, |option| xtml_type(option.ty)) .into_iter() - .filter_map(|(ty, options)| { + .filter_map(|(type_name, options)| { let values = options .into_iter() - .filter_map(|option| match ty { + .filter_map(|option| match option.ty { // The parser ends the call at its close marker, so a // string value may contain neither close marker. JsonType::String => option.raw_string(&[ARG_CLOSE, CALL_CLOSE]), @@ -59,7 +59,7 @@ impl ArgumentSyntax for XtmlArguments { }) .collect::>(); (!values.is_empty()).then(|| { - let suffix = format!("\" type=\"{}\"{SEP}", ty.name()); + let suffix = format!("\" type=\"{type_name}\"{SEP}"); key.tag( &format!("{ARG_OPEN} key=\""), &suffix, @@ -73,6 +73,20 @@ impl ArgumentSyntax for XtmlArguments { } } +/// The `type` attribute K3 renders for a value of type `ty`, as +/// `_xtml_type` in the checkpoint's `encoding_k3.py`: integers and floats are +/// both `number`. +fn xtml_type(ty: JsonType) -> &'static str { + match ty { + JsonType::String => "string", + JsonType::Integer | JsonType::Number => "number", + JsonType::Boolean => "boolean", + JsonType::Null => "null", + JsonType::Object => "object", + JsonType::Array => "array", + } +} + /// Kimi K3 XTML structural-tag builder. #[derive(Debug, Clone, Copy, Default)] pub struct KimiK3StructuralTagBuilder; diff --git a/rust/src/parser/src/utils.rs b/rust/src/parser/src/utils.rs index 1324e297fb..cb7bdce284 100644 --- a/rust/src/parser/src/utils.rs +++ b/rust/src/parser/src/utils.rs @@ -118,7 +118,10 @@ pub fn safe_text_len<'i, I: MarkerStream<'i>, M: MarkerLike>( /// the end of the input for plain markers, and only a genuine special token's /// incomplete marker for guarded ones. /// -/// Returns the text length in bytes, and advances the input. +/// Returns the text length in bytes, and advances the input. Fails with +/// `Backtrack` when a complete marker starts the input, so another branch can +/// consume it, and with `Incomplete` when the input is empty or starts with a +/// marker that needs more input to decide. pub fn safe_text_len_mul<'i, I: MarkerStream<'i>, M: MarkerLike>( input: &mut I, markers: &[M], @@ -128,12 +131,18 @@ pub fn safe_text_len_mul<'i, I: MarkerStream<'i>, M: MarkerLike>( } let start = input.offset(); - let stop = match scan_markers(input, markers, start)? { - Scan::Found(at) | Scan::Pending(at) | Scan::Clear(at) => at, + let (stop, found) = match scan_markers(input, markers, start)? { + Scan::Found(at) => (at, true), + Scan::Pending(at) | Scan::Clear(at) => (at, false), }; let emit_len = stop - start; if emit_len == 0 { - return incomplete(); + // A complete marker starts here: there is no text, whatever input follows. + return if found { + Err(ErrMode::Backtrack(ContextError::new())) + } else { + incomplete() + }; } input.next_slice(emit_len); @@ -529,6 +538,17 @@ mod tests { assert!(matches!(error, ErrMode::Incomplete(_))); } + #[test] + fn safe_text_len_backtracks_at_complete_marker() { + let mut input = Partial::new("body"); + let checkpoint = input.checkpoint(); + + let error = safe_text_len(&mut input, "").unwrap_err(); + + assert!(matches!(error, ErrMode::Backtrack(_))); + assert_eq!(input.offset_from(&checkpoint), 0); + } + #[test] fn safe_text_len_mul_stops_before_earliest_marker() { let mut input = Partial::new("hello<|tool_call>"); diff --git a/rust/src/server/src/error.rs b/rust/src/server/src/error.rs index 5f7adc8fbb..77f7295b56 100644 --- a/rust/src/server/src/error.rs +++ b/rust/src/server/src/error.rs @@ -22,6 +22,8 @@ pub enum ApiError { ModelNotFound { model: String }, /// The request body could not be parsed as valid JSON. JsonParseError { message: String }, + /// The requested operation conflicts with current server state. + Conflict { message: String }, /// An unexpected internal failure happened before streaming started. ServerError { message: String }, } @@ -32,6 +34,7 @@ impl ApiError { match self { Self::InvalidRequest { .. } => StatusCode::BAD_REQUEST, Self::ModelNotFound { .. } => StatusCode::NOT_FOUND, + Self::Conflict { .. } => StatusCode::CONFLICT, Self::ServerError { .. } => StatusCode::INTERNAL_SERVER_ERROR, Self::JsonParseError { .. } => StatusCode::BAD_REQUEST, } @@ -65,6 +68,12 @@ impl ApiError { param: None, code: Some("json_parse_error".to_string()), }, + Self::Conflict { message } => ErrorDetail { + message: message.clone(), + error_type: "conflict_error".to_string(), + param: None, + code: Some("conflict_error".to_string()), + }, }; ErrorResponse { error } diff --git a/rust/src/server/src/grpc/convert.rs b/rust/src/server/src/grpc/convert.rs index d5dd1ae64d..d495fea485 100644 --- a/rust/src/server/src/grpc/convert.rs +++ b/rust/src/server/src/grpc/convert.rs @@ -9,6 +9,7 @@ use tonic::Status; use uuid::Uuid; use vllm_engine_core_client::protocol::kv_hints::{KvHintAction, KvHintsEnvelope}; use vllm_engine_core_client::protocol::output::StopReason; +use vllm_engine_core_client::protocol::request::ReasoningParserKwargs; use vllm_engine_core_client::protocol::structured_outputs::StructuredOutputsParams; use vllm_text::{ DecodedLogprobs, DecodedPromptLogprobs, FinishReason, Finished, Prompt, PromptTruncation, @@ -84,6 +85,7 @@ pub fn to_text_request( }; let session_id = req.session_id.filter(|s| !s.is_empty()); let kv_hints = req.kv_hints.map(kv_hints_from_proto); + let reasoning_gate = req.engine_reasoning_gate.unwrap_or_default(); let sampling = req.sampling.as_ref(); let decoding = req.decoding.as_ref(); @@ -137,8 +139,15 @@ pub fn to_text_request( data_parallel_rank: None, session_id, kv_hints, - reasoning_parser_kwargs: Default::default(), - reasoning_ended: None, + reasoning_parser_kwargs: ReasoningParserKwargs { + chat_template_kwargs: reasoning_gate + .chat_template_kwargs + .map(|kwargs| { + kwargs.fields.iter().map(|(k, v)| (k.clone(), proto_value_to_json(v))).collect() + }) + .unwrap_or_default(), + }, + reasoning_ended: reasoning_gate.reasoning_ended, lora_request: None, arrival_time: None, }) @@ -161,10 +170,7 @@ fn build_sampling_params( ..SamplingParams::default() }; - // RandomSampling: for every remaining sampling field the protobuf default (`0`) - // is treated as "unset" and leaves the resolved value to the lowering - // stage, which falls back to the model-provided default or a - // neutral/disabled value otherwise. + // Preserve explicit values; omitted fields inherit model defaults during lowering. if let Some(s) = sampling { // num_sequences (n > 1) is not supported yet by the TextLlm layer; the response // path also hardcodes SequenceOutput.index = 0, so accepting >1 would silently @@ -174,15 +180,9 @@ fn build_sampling_params( "num_sequences > 1 is not supported", )); } - if s.top_k != 0 { - params.top_k = Some(s.top_k); - } - if s.top_p != 0.0 { - params.top_p = Some(s.top_p); - } - if s.min_p != 0.0 { - params.min_p = Some(s.min_p); - } + params.top_k = s.top_k; + params.top_p = s.top_p; + params.min_p = s.min_p; params.seed = s.seed; } @@ -203,6 +203,10 @@ fn build_sampling_params( if !d.allowed_token_ids.is_empty() { params.allowed_token_ids = Some(d.allowed_token_ids.clone()); } + if !d.bad_words_token_ids.is_empty() { + params.bad_words_token_ids = + Some(d.bad_words_token_ids.iter().map(|sequence| sequence.ids.clone()).collect()); + } params.structured_outputs = convert_structured_output(d)?; } @@ -218,6 +222,7 @@ fn build_sampling_params( params.stop_token_ids = Some(s.stop_token_ids.clone()); } params.ignore_eos = s.ignore_eos; + params.thinking_token_budget = s.thinking_token_budget; } // ResponseOptions → logprobs @@ -403,6 +408,7 @@ fn to_finish_info(finished: &Finished, token_ids: &[u32]) -> Result= 0).map(|v| v as u64), + ); + let request = pb::GenerateRequest { + engine_reasoning_gate: gate, + stopping: Some(pb::StoppingCriteria { + thinking_token_budget: budget, + ..Default::default() + }), + ..base_request() + }; + let encoded = request.encode_to_vec(); + for stream in [false, true] { + let request = pb::GenerateRequest::decode(encoded.as_slice()).unwrap(); + let text = to_text_request(request, stream, &["test-model".to_string()]) + .expect("reasoning controls accepted"); + let engine = vllm_text::lower_text_request( + text, + vec![1], + SamplingHints::default(), + SamplingLimits { + max_model_len: 256, + max_logprobs: 20, + model_vocab_size: 512, + tokenizer_vocab_size: 512, + }, + &TestTokenizer::new(), + ) + .expect("reasoning controls lower to engine") + .generate_request; + assert_eq!( + ( + serde_json::to_value(engine.reasoning_parser_kwargs).unwrap(), + engine.reasoning_ended, + engine.sampling_params.thinking_token_budget, + ), + expected, + ); + } + } + } + #[test] fn watermarking_defaults_and_opt_out_survive_protobuf_conversion() { for watermarking in [None, Some(true), Some(false)] { @@ -750,6 +862,52 @@ mod tests { } } + #[test] + fn grpc_sampling_preserves_explicit_values_and_model_defaults_through_lowering() { + for ((top_k, top_p, min_p), expected) in [ + ((None, None, None), (8, 0.9, 0.2)), + ((Some(0), Some(1.0), Some(0.0)), (0, 1.0, 0.0)), + ((Some(50), Some(0.8), Some(0.1)), (50, 0.8, 0.1)), + ] { + let request = pb::GenerateRequest { + temperature: Some(0.7), + sampling: Some(pb::RandomSampling { + top_k, + top_p, + min_p, + ..Default::default() + }), + ..base_request() + }; + let encoded = request.encode_to_vec(); + for stream in [false, true] { + let decoded = pb::GenerateRequest::decode(encoded.as_slice()).unwrap(); + let text = to_text_request(decoded, stream, &["test-model".to_string()]).unwrap(); + let engine = vllm_text::lower_text_request( + text, + vec![1], + SamplingHints { + default_top_k: Some(8), + default_top_p: Some(0.9), + default_min_p: Some(0.2), + ..Default::default() + }, + SamplingLimits { + max_model_len: 256, + max_logprobs: 20, + model_vocab_size: 512, + tokenizer_vocab_size: 512, + }, + &TestTokenizer::new(), + ) + .unwrap() + .generate_request; + let params = engine.sampling_params; + assert_eq!((params.top_k, params.top_p, params.min_p), expected); + } + } + } + #[test] fn bypass_prefix_cache_maps_to_skip_reading_prefix_cache() { let req = pb::GenerateRequest { @@ -859,7 +1017,9 @@ mod tests { #[test] fn to_sequence_output_threads_token_ids_into_eos_id() { - let fin = finished(FinishReason::Stop(None)); + let mut fin = finished(FinishReason::Stop(None)); + fin.usage.prompt_token_count = 8; + fin.usage.cached_token_count = 5; let opts = ResponseOpts { output_text: true, output_token_ids: true, @@ -868,10 +1028,12 @@ mod tests { let out = to_sequence_output("hello", &[10, 20, 30], None, Some(&fin), &opts) .expect("sequence output"); + let out = pb::SequenceOutput::decode(out.encode_to_vec().as_slice()).unwrap(); let finish = out.finish_info.expect("finish_info should be present"); assert_eq!(finish.finish_reason, PbFinishReason::Stop as i32); assert_eq!(finish.stop_reason, Some(PbStopReason::EosTokenId(30))); + assert_eq!(finish.num_cached_tokens, Some(5)); } #[test] diff --git a/rust/src/server/src/grpc/tests.rs b/rust/src/server/src/grpc/tests.rs index 9f01886062..0ddbfb096e 100644 --- a/rust/src/server/src/grpc/tests.rs +++ b/rust/src/server/src/grpc/tests.rs @@ -5,6 +5,8 @@ use std::collections::BTreeMap; use std::fs; use std::future::Future; use std::io; +use std::os::fd::IntoRawFd; +use std::os::unix::net::UnixListener; use std::pin::Pin; use std::sync::Arc; use std::task::{Context, Poll}; @@ -690,6 +692,7 @@ async fn unary_generate_returns_collected_text() { pb::finish_info::FinishReason::Stop as i32 ); assert_eq!(finish.num_output_tokens, 3); + assert_eq!(finish.num_cached_tokens, Some(0)); let prompt = response.prompt_info.expect("prompt_info present"); assert_eq!(prompt.num_prompt_tokens, 5); // "hello" = 5 bytes @@ -1363,22 +1366,24 @@ async fn unary_generate_invalid_sampling_params_returns_invalid_argument() { ) .await; - let status = client - .generate(pb::GenerateRequest { - request_id: "test-invalid-sampling".to_string(), - model: "test-model".to_string(), - prompt: Some(pb::generate_request::Prompt::Text("hi".to_string())), - sampling: Some(pb::RandomSampling { - top_p: 2.0, + for top_p in [0.0, 2.0] { + let status = client + .generate(pb::GenerateRequest { + request_id: "test-invalid-sampling".to_string(), + model: "test-model".to_string(), + prompt: Some(pb::generate_request::Prompt::Text("hi".to_string())), + sampling: Some(pb::RandomSampling { + top_p: Some(top_p), + ..Default::default() + }), ..Default::default() - }), - ..Default::default() - }) - .await - .expect_err("should fail when top_p is out of range"); + }) + .await + .expect_err("should fail when top_p is out of range"); - assert_eq!(status.code(), tonic::Code::InvalidArgument); - assert!(status.message().contains("top_p")); + assert_eq!(status.code(), tonic::Code::InvalidArgument); + assert!(status.message().contains("top_p")); + } server_task.abort(); } @@ -1438,6 +1443,7 @@ async fn streaming_generate_yields_incremental_responses() { .find_map(|r| r.outputs.as_ref()) .expect("at least one output"); let finish = last_output.finish_info.as_ref().expect("finish_info on last output"); + assert_eq!(finish.num_cached_tokens, Some(0)); assert_eq!( finish.finish_reason, pb::finish_info::FinishReason::Stop as i32 @@ -1515,8 +1521,8 @@ async fn unary_generate_with_sampling_params() { prompt: Some(pb::generate_request::Prompt::Text("test".to_string())), temperature: Some(0.7), sampling: Some(pb::RandomSampling { - top_k: 50, - top_p: 0.9, + top_k: Some(50), + top_p: Some(0.9), seed: Some(42), ..Default::default() }), @@ -2228,10 +2234,14 @@ async fn control_aggregates_multi_engine_capacity() { ready_1.world_size = 12; ready_1.data_parallel_rank = start_rank + 1; + let listener_fd = |address: &str| { + let path = address.strip_prefix("ipc://").expect("IPC test endpoint"); + UnixListener::bind(path).expect("bind inherited test listener").into_raw_fd() + }; let client_config = EngineCoreClientConfig { transport_mode: TransportMode::Bootstrapped { - input_address: input_address.clone(), - output_address: output_address.clone(), + input_listener_fd: listener_fd(&input_address), + output_listener_fd: listener_fd(&output_address), engine_start_index: start_rank, engine_count: 2, data_parallel_size: global_size, diff --git a/rust/src/server/src/routes/inference/generate.rs b/rust/src/server/src/routes/inference/generate.rs index 0e682f7c45..99d7aec1b7 100644 --- a/rust/src/server/src/routes/inference/generate.rs +++ b/rust/src/server/src/routes/inference/generate.rs @@ -5,7 +5,7 @@ mod convert; mod types; mod validate; -use std::collections::HashMap; +use std::collections::{HashMap, HashSet}; use std::convert::Infallible; use std::result::Result; use std::sync::Arc; @@ -31,16 +31,17 @@ use vllm_llm::{ use self::convert::{ResponseOptions, prepare_generate_request}; use self::types::{ - GenerateLogprob, GenerateResponse, GenerateResponseChoice, GenerateResponseStreamChoice, - GenerateStreamResponse, MultiModalPlaceholders, PerRequestMetrics, PlaceholderRangeInfo, - SpeculativeDecodingMetrics, StreamingSpeculativeDecodingMetrics, + GenerateLogProb, GenerateLogProbs, GenerateLogProbsContent, GenerateLogprob, GenerateResponse, + GenerateResponseChoice, GenerateResponseStreamChoice, GenerateStreamResponse, + MultiModalPlaceholders, PerRequestMetrics, PlaceholderRangeInfo, SpeculativeDecodingMetrics, + StreamingSpeculativeDecodingMetrics, }; pub(crate) use self::types::{GenerateRequest, GenerateSamplingParams}; pub(crate) use self::validate::validate_request_compat; use crate::config::ApiServerOptions; use crate::error::{ApiError, bail_server_error, server_error, text_submit_error}; use crate::routes::openai::utils::logprobs::clamp_logprob; -use crate::routes::openai::utils::types::{ChatLogProbs, ChatLogProbsContent, TopLogProb, Usage}; +use crate::routes::openai::utils::types::Usage; use crate::routes::openai::utils::validated_json::ValidatedJson; use crate::state::AppState; use crate::utils::{resolve_request_context, sse_response}; @@ -157,7 +158,7 @@ async fn generate_chunk_stream( ResponseOptions { include_usage, include_continuous_usage, - include_logprobs, + logprobs: output_logprobs, // Ignored: raw generate streaming has no prompt-logprobs wire shape. include_prompt_logprobs: _, return_token_ids, @@ -212,13 +213,15 @@ async fn generate_chunk_stream( continue; } - let logprobs = if include_logprobs && !token_ids.is_empty() { + let logprobs = if let Some(requested) = output_logprobs + && !token_ids.is_empty() + { let logprobs = output.logprobs.as_ref().ok_or_else(|| { server_error!( "raw generate stream requested logprobs but generation returned none" ) })?; - Some(raw_logprobs_to_openai_chat(logprobs)?) + Some(raw_logprobs_to_generate(logprobs, requested)?) } else { None }; @@ -282,19 +285,19 @@ fn collect_generate( include_usage: _, // Ignored: continuous usage is a streaming-only option. include_continuous_usage: _, - include_logprobs, + logprobs: output_logprobs, include_prompt_logprobs, return_token_ids, }: ResponseOptions, mm_placeholders: Option, ) -> Result { - let logprobs = if include_logprobs { + let logprobs = if let Some(requested) = output_logprobs { let logprobs = collected.logprobs.as_ref().ok_or_else(|| { ApiError::server_error( "raw generate response requested logprobs but generation returned none".to_string(), ) })?; - Some(raw_logprobs_to_openai_chat(logprobs)?) + Some(raw_logprobs_to_generate(logprobs, requested)?) } else { None }; @@ -371,14 +374,17 @@ fn extract_mm_placeholders(features: Option<&[MmFeatureSpec]>) -> Option Result { +fn raw_logprobs_to_generate( + logprobs: &Logprobs, + requested: i32, +) -> Result { let content = logprobs .positions .iter() - .map(position_to_chat_logprobs_content) + .map(|position| position_to_generate_logprobs_content(position, requested)) .collect::, _>>()?; - Ok(ChatLogProbs { + Ok(GenerateLogProbs { content: Some(content), }) } @@ -396,35 +402,47 @@ fn raw_prompt_logprobs_to_maps( .collect() } -fn position_to_chat_logprobs_content( +/// `top_logprobs` matches the Python generate server: the engine's entries (the +/// sampled token first, then ranks 1..k) with repeated token ids dropped, cut to +/// `max(requested, 1)` entries, or all of them for `-1`. +fn position_to_generate_logprobs_content( position: &PositionLogprobs, -) -> Result { + requested: i32, +) -> Result { let chosen = position.entries.first().ok_or_else(|| { ApiError::server_error( "raw generate logprobs position unexpectedly had no token candidates".to_string(), ) })?; - let token = format_token_id(chosen.token_id); - Ok(ChatLogProbsContent { - token: token.clone(), + // One pass: the first entry of each token id wins, as in Python's dict. The + // sampled token is the only one the engine can list twice. + let mut seen = HashSet::with_capacity(position.entries.len()); + Ok(GenerateLogProbsContent { + token_id: chosen.token_id, logprob: clamp_logprob(chosen.logprob), - bytes: Some(token.as_bytes().to_vec()), + rank: wire_rank(chosen.rank), top_logprobs: position .entries .iter() - .map(|entry| { - let token = format_token_id(entry.token_id); - TopLogProb { - token: token.clone(), - logprob: clamp_logprob(entry.logprob), - bytes: Some(token.into_bytes()), - } + .filter(|entry| seen.insert(entry.token_id)) + .take(usize::try_from(requested).map_or(usize::MAX, |n| n.max(1))) + .map(|entry| GenerateLogProb { + token_id: entry.token_id, + logprob: clamp_logprob(entry.logprob), + rank: wire_rank(entry.rank), }) .collect(), }) } +/// The engine reports rank 0 for a sampled token whose logprob is NaN (no value +/// compares >= NaN). That is not a rank, so it goes on the wire as `None`, the +/// way `clamp_logprob` handles the logprob itself. +fn wire_rank(rank: u32) -> Option { + (rank > 0).then_some(rank) +} + fn position_to_logprob_map(position: &PositionLogprobs) -> HashMap { position .entries @@ -523,6 +541,80 @@ mod tests { use super::*; + fn position(entries: &[(u32, f32, u32)]) -> PositionLogprobs { + PositionLogprobs { + entries: entries + .iter() + .map(|&(token_id, logprob, rank)| { + vllm_engine_core_client::protocol::logprobs::TokenLogprob { + token_id, + logprob, + rank, + } + }) + .collect(), + } + } + + fn top(position: &PositionLogprobs, requested: i32) -> Vec<(u32, Option)> { + position_to_generate_logprobs_content(position, requested) + .unwrap() + .top_logprobs + .iter() + .map(|entry| (entry.token_id, entry.rank)) + .collect() + } + + #[test] + fn generate_top_logprobs_drop_the_repeated_sampled_token() { + // Sampled token 7 is also top-1: the engine row is [7, 7, 8]. + let pos = position(&[(7, -0.1, 1), (7, -0.1, 1), (8, -2.0, 2)]); + assert_eq!(top(&pos, 2), vec![(7, Some(1)), (8, Some(2))]); + } + + #[test] + fn generate_top_logprobs_keep_sampled_first_when_outside_top_k() { + // Sampled token 50 has vocab rank 5; it takes a slot and rank 2 drops + // out, as on the Python generate server and the OpenAI endpoints. + let pos = position(&[(50, -3.0, 5), (10, -0.2, 1), (20, -1.0, 2)]); + assert_eq!(top(&pos, 2), vec![(50, Some(5)), (10, Some(1))]); + } + + #[test] + fn generate_top_logprobs_respect_zero_and_all() { + let pos = position(&[(7, -0.1, 1), (7, -0.1, 1), (8, -2.0, 2), (9, -3.0, 3)]); + assert_eq!(top(&pos, 0), vec![(7, Some(1))]); + assert_eq!( + top(&pos, -1), + vec![(7, Some(1)), (8, Some(2)), (9, Some(3))] + ); + } + + #[test] + fn generate_top_logprobs_full_vocab_is_linear() { + // logprobs = -1 returns the whole vocabulary; deduping it must not be + // quadratic in the number of entries. + const VOCAB: u32 = 200_000; + let mut entries = vec![(5, -0.5, 1)]; + entries.extend((0..VOCAB).map(|t| (t, -1.0 - t as f32, t + 1))); + let pos = position(&entries); + let start = std::time::Instant::now(); + let got = top(&pos, -1); + assert!(start.elapsed() < std::time::Duration::from_secs(5)); + assert_eq!(got.len(), VOCAB as usize); + assert_eq!(got[0], (5, Some(1))); + } + + #[test] + fn generate_rank_zero_from_a_nan_logprob_is_none() { + // The engine reports rank 0 for a sampled token whose logprob is NaN. + let pos = position(&[(7, f32::NAN, 0), (8, -0.2, 1)]); + let content = position_to_generate_logprobs_content(&pos, 1).unwrap(); + assert_eq!(content.rank, None); + assert_eq!(content.logprob, -9999.0); + assert_eq!(top(&pos, 1), vec![(7, None)]); + } + #[tokio::test] async fn generate_chunk_stream_captures_late_prompt_info() { let stream = stream::iter(vec![ diff --git a/rust/src/server/src/routes/inference/generate/convert.rs b/rust/src/server/src/routes/inference/generate/convert.rs index 4c54619ed4..36a497f6dd 100644 --- a/rust/src/server/src/routes/inference/generate/convert.rs +++ b/rust/src/server/src/routes/inference/generate/convert.rs @@ -26,8 +26,10 @@ pub(super) struct ResponseOptions { pub include_usage: bool, /// Whether the caller asked for usage on every streamed chunk. pub include_continuous_usage: bool, - /// Whether the caller requested output logprobs on generate choices. - pub include_logprobs: bool, + /// Requested output logprobs (`sampling_params.logprobs`): `None` omits + /// them, `Some(n)` returns `max(n, 1)` candidates per position, `Some(-1)` + /// all of them. + pub logprobs: Option, /// Whether the caller requested top-level prompt logprobs. pub include_prompt_logprobs: bool, /// Whether the caller requested final prompt token metadata. @@ -56,7 +58,7 @@ pub(super) fn prepare_generate_request( .as_ref() .and_then(|options| options.continuous_usage_stats) .unwrap_or(false); - let include_logprobs = request.sampling_params.inner.logprobs.is_some(); + let logprobs = request.sampling_params.inner.logprobs; let include_prompt_logprobs = request.sampling_params.inner.prompt_logprobs.is_some(); let return_token_ids = request.return_token_ids.unwrap_or(false); let mut sampling_params = request.sampling_params.inner; @@ -96,7 +98,7 @@ pub(super) fn prepare_generate_request( options: ResponseOptions { include_usage, include_continuous_usage, - include_logprobs, + logprobs, include_prompt_logprobs, return_token_ids, }, diff --git a/rust/src/server/src/routes/inference/generate/types.rs b/rust/src/server/src/routes/inference/generate/types.rs index b0b93d7080..bd277ef34a 100644 --- a/rust/src/server/src/routes/inference/generate/types.rs +++ b/rust/src/server/src/routes/inference/generate/types.rs @@ -10,7 +10,7 @@ use validator::Validate; use vllm_engine_core_client::protocol::output::RequestSpecDecodeMetrics; use vllm_text::SamplingParams; -use crate::routes::openai::utils::types::{ChatLogProbs, Normalizable, StreamOptions, Usage}; +use crate::routes::openai::utils::types::{Normalizable, StreamOptions, Usage}; /// Sampling parameters for the token-in/token-out generate API. /// @@ -61,7 +61,7 @@ impl Normalizable for GenerateRequest {} #[derive(Debug, Clone, Serialize)] pub(super) struct GenerateResponseChoice { pub index: u32, - pub logprobs: Option, + pub logprobs: Option, pub finish_reason: Option, pub token_ids: Vec, pub sampling_mask: Option>>, @@ -72,7 +72,7 @@ pub(super) struct GenerateResponseChoice { #[derive(Debug, Clone, Serialize)] pub(super) struct GenerateResponseStreamChoice { pub index: u32, - pub logprobs: Option, + pub logprobs: Option, pub finish_reason: Option, pub token_ids: Vec, pub sampling_mask: Option>>, @@ -197,6 +197,32 @@ pub(super) struct PlaceholderRangeInfo { pub length: usize, } +/// Mirrors the Python vLLM `GenerateLogProbs` class: output logprobs for one +/// choice, carrying integer token ids rather than the OpenAI string token. +#[derive(Debug, Clone, Serialize)] +pub(super) struct GenerateLogProbs { + pub content: Option>, +} + +/// Mirrors the Python vLLM `GenerateLogProbsContent` class: the sampled token +/// at one position plus its top-k candidates, in the engine's order: the +/// sampled token first, then the remaining candidates in rank order. +#[derive(Debug, Clone, Serialize)] +pub(super) struct GenerateLogProbsContent { + pub token_id: u32, + pub logprob: f32, + pub rank: Option, + pub top_logprobs: Vec, +} + +/// Mirrors the Python vLLM `GenerateLogProb` class: one candidate token. +#[derive(Debug, Clone, Serialize)] +pub(super) struct GenerateLogProb { + pub token_id: u32, + pub logprob: f32, + pub rank: Option, +} + /// Mirrors the Python vLLM `Logprob` class used in prompt-logprobs payloads. #[derive(Debug, Clone, Serialize)] pub(super) struct GenerateLogprob { diff --git a/rust/src/server/src/routes/openai/chat_completions/convert.rs b/rust/src/server/src/routes/openai/chat_completions/convert.rs index 418ab23ce8..4f80dda6c7 100644 --- a/rust/src/server/src/routes/openai/chat_completions/convert.rs +++ b/rust/src/server/src/routes/openai/chat_completions/convert.rs @@ -171,6 +171,7 @@ pub(super) fn prepare_chat_request( logit_bias: convert_logit_bias(request.logit_bias)?, allowed_token_ids: request.allowed_token_ids, bad_words: request.bad_words, + bad_words_token_ids: None, logprob_token_ids: None, structured_outputs, skip_reading_prefix_cache: None, @@ -409,6 +410,7 @@ pub(crate) fn convert_tools(tools: Option>) -> Result, A description: tool.function.description, parameters: tool.function.parameters, strict: tool.function.strict, + defer_loading: tool.function.defer_loading.or(tool.defer_loading), }) }) .collect() @@ -712,10 +714,50 @@ mod tests { description: None, parameters: serde_json::Value::Null, strict: None, + defer_loading: None, }] ); } + #[test] + fn prepare_chat_request_maps_tool_defer_loading() { + let request: ChatCompletionRequest = serde_json::from_value(json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "messages": [{"role": "user", "content": "hello"}], + "tools": [ + {"function": {"name": "tool_level"}, "defer_loading": true}, + {"function": {"name": "function_level", "defer_loading": true}}, + { + "function": {"name": "function_wins", "defer_loading": false}, + "defer_loading": true, + }, + ], + })) + .expect("parse defer_loading"); + + let prepared = prepare_chat_request( + request, + &served(&["Qwen/Qwen1.5-0.5B-Chat"]), + ResolvedRequestContext::default(), + ) + .expect("prepare defer_loading"); + + let defer_loading = prepared + .chat_request + .tools() + .iter() + .map(|tool| (tool.name.as_str(), tool.defer_loading)) + .collect::>(); + assert_eq!( + defer_loading, + [ + ("tool_level", Some(true)), + ("function_level", Some(true)), + ("function_wins", Some(false)), + ] + ); + } + #[test] fn chat_http_request_rejects_empty_cache_salt() { let request: ChatCompletionRequest = serde_json::from_value(json!({ @@ -1022,7 +1064,9 @@ mod tests { "properties": {"city": {"type": "string"}}, }), strict: Some(true), + defer_loading: None, }, + defer_loading: None, }]), name: None, }], @@ -1048,6 +1092,7 @@ mod tests { "properties": {"city": {"type": "string"}}, }), strict: Some(true), + defer_loading: None, }]), )] ); @@ -1378,7 +1423,9 @@ mod tests { "properties": {"city": {"type": "string"}}, }), strict: None, + defer_loading: None, }, + defer_loading: None, }]), tool_choice: Some(ToolChoice::Value(ToolChoiceValue::None)), ..base_request() @@ -1413,6 +1460,7 @@ mod tests { "properties": {"city": {"type": "string"}}, }), strict: None, + defer_loading: None, }] ); assert_eq!(prepared.chat_request.tool_choice(), &ChatToolChoice::None); @@ -1431,7 +1479,9 @@ mod tests { "properties": {"city": {"type": "string"}}, }), strict: None, + defer_loading: None, }, + defer_loading: None, }]), tool_choice: Some(ToolChoice::Value(ToolChoiceValue::Required)), ..base_request() @@ -1464,7 +1514,9 @@ mod tests { "properties": {"city": {"type": "string"}}, }), strict: None, + defer_loading: None, }, + defer_loading: None, }]), tool_choice: Some(ToolChoice::Function { tool_type: "function".to_string(), diff --git a/rust/src/server/src/routes/openai/chat_completions/validate.rs b/rust/src/server/src/routes/openai/chat_completions/validate.rs index 1bfc1d9a09..579b815ba1 100644 --- a/rust/src/server/src/routes/openai/chat_completions/validate.rs +++ b/rust/src/server/src/routes/openai/chat_completions/validate.rs @@ -195,7 +195,9 @@ mod tests { description: None, parameters: json!({}), strict: None, + defer_loading: None, }, + defer_loading: None, }]), ..base_request() }; @@ -212,7 +214,9 @@ mod tests { description: None, parameters: json!({}), strict: None, + defer_loading: None, }, + defer_loading: None, }]), name: None, }], @@ -234,7 +238,9 @@ mod tests { description: None, parameters: json!({}), strict: None, + defer_loading: None, }, + defer_loading: None, }]), name: None, }], diff --git a/rust/src/server/src/routes/openai/completions.rs b/rust/src/server/src/routes/openai/completions.rs index 7074b8c629..646a67d1fb 100644 --- a/rust/src/server/src/routes/openai/completions.rs +++ b/rust/src/server/src/routes/openai/completions.rs @@ -271,6 +271,13 @@ async fn completion_chunk_stream( macro_rules! yield_chunk { ($chunk:expr) => {{ let mut chunk = $chunk; + if return_token_ids { + for choice in &mut chunk.choices { + if choice.token_ids.is_none() { + choice.token_ids = Some(Vec::new()); + } + } + } if include_continuous_usage { chunk.usage = Some(continuous_usage.to_usage()); } @@ -787,6 +794,70 @@ mod tests { } } + #[tokio::test] + async fn completion_stream_serializes_requested_token_ids_on_every_choice() { + let stream = stream::iter(vec![ + Ok(DecodedTextEvent::Start { + prompt_token_ids: vec![1, 2].into(), + prompt_logprobs: None, + }), + Ok(decoded_delta( + "ok", + vec![7, 8], + None, + Some(Finished { + usage: vllm_llm::TokenUsage { + prompt_token_count: 2, + output_token_count: 2, + cached_token_count: 0, + }, + finish_reason: FinishReason::Length, + kv_transfer_params: None, + ec_transfer_params: None, + sampling_mask: None, + }), + )), + ]); + + let chunks = completion_chunk_stream( + stream, + "cmpl-1".to_string(), + "model".to_string(), + 1, + ApiServerOptions::default(), + ResponseOptions { + return_token_ids: true, + ..Default::default() + }, + ) + .collect::>() + .await; + + let chunks: Vec<_> = chunks.into_iter().try_collect().expect("stream should succeed"); + let payloads: Vec<_> = chunks + .iter() + .map(|chunk| serde_json::to_value(chunk).expect("chunk should serialize")) + .collect(); + + assert_eq!(payloads.len(), 3); + assert_eq!( + payloads[0]["choices"][0]["prompt_token_ids"], + serde_json::json!([1, 2]) + ); + assert_eq!( + payloads[0]["choices"][0]["token_ids"], + serde_json::json!([]) + ); + assert_eq!( + payloads[1]["choices"][0]["token_ids"], + serde_json::json!([7, 8]) + ); + assert_eq!( + payloads[2]["choices"][0]["token_ids"], + serde_json::json!([]) + ); + } + #[tokio::test] async fn collect_completion_hides_internal_prompt_only_token() { let stream = stream::iter(vec![ diff --git a/rust/src/server/src/routes/openai/completions/convert.rs b/rust/src/server/src/routes/openai/completions/convert.rs index d05cfffc89..0b37f84057 100644 --- a/rust/src/server/src/routes/openai/completions/convert.rs +++ b/rust/src/server/src/routes/openai/completions/convert.rs @@ -136,6 +136,7 @@ pub(super) fn prepare_completion_request( logit_bias: convert_logit_bias(request.logit_bias)?, allowed_token_ids: request.allowed_token_ids, bad_words: None, + bad_words_token_ids: None, logprob_token_ids: None, structured_outputs, skip_reading_prefix_cache: None, diff --git a/rust/src/server/src/routes/openai/utils/types.rs b/rust/src/server/src/routes/openai/utils/types.rs index 85a891341c..e2187bae67 100644 --- a/rust/src/server/src/routes/openai/utils/types.rs +++ b/rust/src/server/src/routes/openai/utils/types.rs @@ -271,6 +271,10 @@ pub struct Tool { #[serde(default = "default_function_tool_type")] pub tool_type: String, pub function: Function, + /// Tool-level form of [`Function::defer_loading`]; the function-level value + /// takes precedence. + #[serde(skip_serializing_if = "Option::is_none")] + pub defer_loading: Option, } fn default_function_tool_type() -> String { @@ -286,6 +290,9 @@ pub struct Function { pub parameters: Value, /// Whether to enable strict schema adherence (OpenAI structured outputs). pub strict: Option, + /// Whether the schema is loaded on demand via tool search (vLLM extension, + /// mirroring the OpenAI Responses and Anthropic Messages field). + pub defer_loading: Option, } #[derive(Debug, Clone, Deserialize, Serialize)] diff --git a/rust/src/server/src/routes/profile.rs b/rust/src/server/src/routes/profile.rs index de793a7b3b..763a4f8c1b 100644 --- a/rust/src/server/src/routes/profile.rs +++ b/rust/src/server/src/routes/profile.rs @@ -3,22 +3,75 @@ use std::sync::Arc; +use axum::Json; +use axum::body::Bytes; use axum::extract::State; use axum::http::StatusCode; +use serde::Deserialize; use tracing::info; +use vllm_engine_core_client::Error as EngineCoreClientError; use crate::error::ApiError; use crate::state::AppState; use crate::utils::utility_call_error; +#[derive(Debug, Default, Deserialize)] +pub(crate) struct StartProfileRequest { + profile_prefix: Option, + delay_iterations: Option, + max_iterations: Option, +} + +fn valid_profile_prefix(prefix: &str) -> bool { + let bytes = prefix.as_bytes(); + (1..=128).contains(&bytes.len()) + && bytes[0].is_ascii_alphanumeric() + && bytes[1..] + .iter() + .all(|byte| byte.is_ascii_alphanumeric() || matches!(byte, b'.' | b'_' | b'-')) +} + /// Start profiling the engine. -pub async fn start_profile(State(state): State>) -> Result { +pub async fn start_profile( + State(state): State>, + body: Bytes, +) -> Result { + let body = if body.is_empty() { + StartProfileRequest::default() + } else { + Json::::from_bytes(&body)?.0 + }; + if body + .profile_prefix + .as_deref() + .is_some_and(|prefix| !valid_profile_prefix(prefix)) + { + return Err(ApiError::invalid_request( + "profile_prefix must be 1-128 ASCII characters, start with a letter or \ + number, and contain only letters, numbers, '.', '_', or '-'" + .to_string(), + Some("profile_prefix"), + )); + } info!("starting profiler"); - state + if let Err(error) = state .engine_core_client() - .start_profile(None) + .start_profile( + body.profile_prefix.as_deref(), + body.delay_iterations, + body.max_iterations, + ) .await - .map_err(|error| utility_call_error("start_profile", error))?; + { + if matches!(error, EngineCoreClientError::ProfileAlreadyActive) { + return Err(ApiError::conflict( + "A profiling session is already active. Call /stop_profile before \ + starting another session." + .to_string(), + )); + } + return Err(utility_call_error("start_profile", error)); + } info!("profiler started"); Ok(StatusCode::OK) } @@ -34,3 +87,19 @@ pub async fn stop_profile(State(state): State>) -> Result) -> u64 { + let utility = recv_engine_message(dealer).await; + assert_eq!(utility[0].as_ref(), &[0x03]); + let payload = decode_value(&utility[1]).expect("decode utility payload"); + let array = payload.as_array().expect("utility payload array"); + assert_eq!(array[2], Value::from("profile")); + assert_eq!(array[3], Value::Array(expected_args)); + array[1].as_u64().expect("call id") +} + +/// A start issued while a stop is in flight waits for that stop instead of +/// racing it, so a stale stop reply cannot clear the new session. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn start_profile_waits_for_inflight_stop() { + let (stop_received_tx, stop_received_rx) = tokio::sync::oneshot::channel(); + let (release_stop_tx, release_stop_rx) = tokio::sync::oneshot::channel::<()>(); + let (app, engine_task) = test_profiling_app_with_engine_script(move |dealer, push| { + boxed_test_future(async move { + let start = || vec![Value::from(true), Value::Nil, Value::Nil, Value::Nil]; + + let call_id = recv_profile_call(dealer, start()).await; + send_outputs(push, utility_outputs(call_id, utility_none_result())).await; + + let call_id = recv_profile_call(dealer, vec![Value::from(false), Value::Nil]).await; + stop_received_tx.send(()).expect("notify stop received"); + release_stop_rx.await.expect("release stop"); + send_outputs(push, utility_outputs(call_id, utility_none_result())).await; + + let call_id = recv_profile_call(dealer, start()).await; + send_outputs(push, utility_outputs(call_id, utility_none_result())).await; + }) + }) + .await; + + let post = |uri: &'static str| { + let mut app = app.clone(); + async move { + app.call( + Request::builder() + .method("POST") + .uri(uri) + .body(Body::empty()) + .expect("build request"), + ) + .await + .expect("call app") + .status() + } + }; + + let timeout = Duration::from_secs(2); + assert_eq!(post("/start_profile").await, StatusCode::OK); + let stop = tokio::spawn(post("/stop_profile")); + tokio::time::timeout(timeout, stop_received_rx) + .await + .expect("stop dispatch timeout") + .expect("stop received"); + + // Nothing before the profile lock awaits, so one poll reaches it; an + // unserialized client would instead reject against the stale session. + let mut restart = std::pin::pin!(post("/start_profile")); + assert!(futures::poll!(restart.as_mut()).is_pending()); + + release_stop_tx.send(()).expect("release stop"); + let stop = tokio::time::timeout(timeout, stop).await.expect("stop timeout"); + assert_eq!(stop.expect("stop task"), StatusCode::OK); + let restart = tokio::time::timeout(timeout, restart).await.expect("restart timeout"); + assert_eq!(restart, StatusCode::OK); + tokio::time::timeout(timeout, engine_task) + .await + .expect("mock engine timeout") + .expect("mock engine task"); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn start_profile_route_rejects_invalid_prefix() { + let (app, engine_task) = + test_profiling_app_with_engine_script(|_, _| boxed_test_future(async move {})).await; + + let response = app + .clone() + .call( + Request::builder() + .method("POST") + .uri("/start_profile") + .header("content-type", "application/json") + .body(Body::from( + json!({"profile_prefix": "../trace"}).to_string(), + )) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::BAD_REQUEST); engine_task.await.expect("mock engine task"); } @@ -7846,16 +8110,8 @@ async fn start_profile_route_sends_expected_utility_call() { async fn stop_profile_route_sends_expected_utility_call() { let (app, engine_task) = test_profiling_app_with_engine_script(|dealer, push| { boxed_test_future(async move { - let utility = recv_engine_message(dealer).await; - assert_eq!(utility[0].as_ref(), &[0x03]); - - let payload = decode_value(&utility[1]).expect("decode utility payload"); - let array = payload.as_array().expect("utility payload array"); - let call_id = array[1].as_u64().expect("call id"); - - assert_eq!(array[2], Value::from("profile")); - assert_eq!(array[3], Value::Array(vec![Value::from(false), Value::Nil])); - + let stop = vec![Value::from(false), Value::Nil]; + let call_id = recv_profile_call(dealer, stop).await; send_outputs(push, utility_outputs(call_id, utility_none_result())).await; }) }) diff --git a/rust/src/server/src/routes/tokenize/types.rs b/rust/src/server/src/routes/tokenize/types.rs index d907073c87..ed3fd11d1d 100644 --- a/rust/src/server/src/routes/tokenize/types.rs +++ b/rust/src/server/src/routes/tokenize/types.rs @@ -215,6 +215,7 @@ mod tests { "properties": {"city": {"type": "string"}}, }), strict: None, + defer_loading: None, }] ); } diff --git a/rust/src/text/src/lower.rs b/rust/src/text/src/lower.rs index cf53704374..2b09070049 100644 --- a/rust/src/text/src/lower.rs +++ b/rust/src/text/src/lower.rs @@ -119,6 +119,7 @@ pub fn lower_sampling_params( logit_bias, allowed_token_ids, bad_words, + bad_words_token_ids, logprob_token_ids, structured_outputs, skip_reading_prefix_cache, @@ -173,6 +174,11 @@ pub fn lower_sampling_params( merge_unique_token_ids(&mut stop_token_ids, extra_eos_token_ids.iter().copied()); } + let mut bad_words_token_ids = bad_words_token_ids.unwrap_or_default(); + if let Some(tokenized) = tokenize_bad_words(bad_words.as_deref(), tokenizer)? { + bad_words_token_ids.extend(tokenized); + } + let params = EngineCoreSamplingParams { temperature, watermarking, @@ -201,7 +207,7 @@ pub fn lower_sampling_params( all_stop_token_ids, logit_bias, allowed_token_ids, - bad_words_token_ids: tokenize_bad_words(bad_words.as_deref(), tokenizer)?, + bad_words_token_ids: (!bad_words_token_ids.is_empty()).then_some(bad_words_token_ids), // TODO: Validate structured-output schemas and regexes before submitting requests to engine-core. structured_outputs, logprob_token_ids, @@ -1255,25 +1261,72 @@ mod tests { #[test] fn lower_sampling_params_rejects_out_of_vocab_bad_words() { let tokenizer = TestTokenizer::new().with_regular_token("blocked", 2000); - let error = lower_sampling_params( + for sampling_params in [ SamplingParams { bad_words: Some(vec!["blocked".to_string()]), ..Default::default() }, - SamplingHints::default(), + SamplingParams { + bad_words_token_ids: Some(vec![vec![1999, 2000]]), + ..Default::default() + }, + ] { + let error = lower_sampling_params( + sampling_params, + SamplingHints::default(), + sample_sampling_limits(), + 3, + &tokenizer, + ) + .unwrap_err(); + + assert!(matches!( + error, + Error::TokenIds(TokenIdsError::OutOfVocab { + parameter: "bad_words", + token_ids, + vocab_size: 2000, + }) if token_ids == vec![2000] + )); + } + } + + #[test] + fn lower_sampling_params_preserves_raw_bad_words_alongside_strings() { + let tokenizer = TestTokenizer::new() + .with_regular_token("blocked", 300) + .with_regular_token(" blocked", 301); + for raw in [None, Some(vec![]), Some(vec![vec![7, 11]])] { + let params = lower_sampling_params( + SamplingParams { + bad_words: Some(vec!["blocked".to_string()]), + bad_words_token_ids: raw.clone(), + ..Default::default() + }, + SamplingHints::default(), + sample_sampling_limits(), + 3, + &tokenizer, + ) + .unwrap(); + let expected = if raw.as_ref().is_some_and(|ids| !ids.is_empty()) { + vec![vec![7, 11], vec![300], vec![301]] + } else { + vec![vec![300], vec![301]] + }; + assert_eq!(params.bad_words_token_ids, Some(expected)); + } + let error = lower_sampling_params_with_limits( + SamplingParams { + bad_words_token_ids: Some(vec![vec![]]), + ..Default::default() + }, sample_sampling_limits(), - 3, - &tokenizer, ) .unwrap_err(); - assert!(matches!( error, - Error::TokenIds(TokenIdsError::OutOfVocab { - parameter: "bad_words", - token_ids, - vocab_size: 2000, - }) if token_ids == vec![2000] + Error::TokenIds(TokenIdsError::EmptyBadWordSequence) )); } diff --git a/rust/src/text/src/lower/token_ids.rs b/rust/src/text/src/lower/token_ids.rs index 8512a830e6..51329f9290 100644 --- a/rust/src/text/src/lower/token_ids.rs +++ b/rust/src/text/src/lower/token_ids.rs @@ -12,6 +12,8 @@ use crate::SamplingLimits; pub enum TokenIdsError { #[error("allowed_token_ids should not be empty")] EmptyAllowedTokenIds, + #[error("bad_words_token_ids must not contain empty sequences")] + EmptyBadWordSequence, #[error( "token_id(s) {token_ids:?} in {parameter} are out of vocabulary. \ Vocabulary size: {vocab_size}" @@ -96,6 +98,9 @@ pub(crate) fn validate_vocab_range( } if let Some(bad_words_token_ids) = params.bad_words_token_ids.as_deref() { + if bad_words_token_ids.iter().any(Vec::is_empty) { + return Err(TokenIdsError::EmptyBadWordSequence); + } validate_param( "bad_words", bad_words_token_ids.iter().flatten().copied(), diff --git a/rust/src/text/src/request.rs b/rust/src/text/src/request.rs index c37e21060e..08758748ac 100644 --- a/rust/src/text/src/request.rs +++ b/rust/src/text/src/request.rs @@ -129,6 +129,8 @@ pub struct SamplingParams { pub allowed_token_ids: Option>, /// Words to avoid during generation (tokenized to IDs during lowering). pub bad_words: Option>, + /// Pre-tokenized sequences to prohibit, in addition to `bad_words`. + pub bad_words_token_ids: Option>>, /// Specific token IDs for which log probabilities should be returned at /// each position. /// @@ -173,6 +175,7 @@ impl Default for SamplingParams { logit_bias: None, allowed_token_ids: None, bad_words: None, + bad_words_token_ids: None, logprob_token_ids: None, structured_outputs: None, skip_reading_prefix_cache: None, diff --git a/setup.py b/setup.py index 67f192cdd3..b173b9dca8 100644 --- a/setup.py +++ b/setup.py @@ -87,6 +87,8 @@ def get_missing_precompiled_rust_extension_modules() -> list[str]: def has_precompiled_rust_extensions() -> bool: + if not rust_build.rust_py_extension_module_names(): + return False return not get_missing_precompiled_rust_extension_modules() @@ -266,8 +268,8 @@ class cmake_build_ext(build_ext): cfg = envs.CMAKE_BUILD_TYPE or default_cfg cmake_args = [ - "-DCMAKE_BUILD_TYPE={}".format(cfg), - "-DVLLM_TARGET_DEVICE={}".format(VLLM_TARGET_DEVICE), + f"-DCMAKE_BUILD_TYPE={cfg}", + f"-DVLLM_TARGET_DEVICE={VLLM_TARGET_DEVICE}", ] verbose = envs.VERBOSE @@ -291,7 +293,7 @@ class cmake_build_ext(build_ext): # Pass the python executable to cmake so it can find an exact # match. - cmake_args += ["-DVLLM_PYTHON_EXECUTABLE={}".format(sys.executable)] + cmake_args += [f"-DVLLM_PYTHON_EXECUTABLE={sys.executable}"] # Pass the python path to cmake so it can reuse the build dependencies # on subsequent calls to python. @@ -303,7 +305,7 @@ class cmake_build_ext(build_ext): # To override this, set the FETCHCONTENT_BASE_DIR environment variable. fc_base_dir = os.path.join(ROOT_DIR, ".deps") fc_base_dir = os.environ.get("FETCHCONTENT_BASE_DIR", fc_base_dir) - cmake_args += ["-DFETCHCONTENT_BASE_DIR={}".format(fc_base_dir)] + cmake_args += [f"-DFETCHCONTENT_BASE_DIR={fc_base_dir}"] # # Setup parallelism and build tool @@ -311,13 +313,13 @@ class cmake_build_ext(build_ext): num_jobs, nvcc_threads = self.compute_num_jobs() if nvcc_threads: - cmake_args += ["-DNVCC_THREADS={}".format(nvcc_threads)] + cmake_args += [f"-DNVCC_THREADS={nvcc_threads}"] if is_ninja_available(): build_tool = ["-G", "Ninja"] cmake_args += [ "-DCMAKE_JOB_POOL_COMPILE:STRING=compile", - "-DCMAKE_JOB_POOLS:STRING=compile={}".format(num_jobs), + f"-DCMAKE_JOB_POOLS:STRING=compile={num_jobs}", ] else: # Default build tool to whatever cmake picks. @@ -894,6 +896,93 @@ class precompiled_wheel_utils: print(f"Using ROCm precompiled wheel: {wheel_url}") return wheel_url, download_filename + # Publication lags main by roughly 5-10 commits; anything much older is + # unlikely to still match the checkout's compiled sources. + WHEEL_SEARCH_DEPTH = 20 + # Changes here make an older wheel's extensions unusable. + WHEEL_BLOCKING_PATHS = ( + "csrc", + "cmake", + "CMakeLists.txt", + "pyproject.toml", + "vllm/_custom_ops.py", + ) + # Changes here are usually harmless for an older wheel. + WHEEL_WARNING_PATHS = ("rust", "setup.py") + + @staticmethod + def find_published_wheel_commit( + base_commit: str, variant: str | None, arch: str + ) -> str: + """Find the nearest ancestor of base_commit that has a published wheel. + + Raises: + ValueError: If no wheel is published within WHEEL_SEARCH_DEPTH + commits, or compiled sources changed since the nearest one. + + """ + from urllib.error import HTTPError + + utils = precompiled_wheel_utils + candidates = subprocess.check_output( + [ + "git", + "rev-list", + "--first-parent", + f"--max-count={utils.WHEEL_SEARCH_DEPTH}", + base_commit, + ], + text=True, + ).splitlines() + for distance, commit in enumerate(candidates): + try: + wheels, _ = utils.fetch_metadata_for_variant(commit, variant) + except HTTPError as err: + if err.code != 404: + raise + continue + if not any( + wheel.get("package_name") == "vllm" + and arch in wheel.get("platform_tag", "") + for wheel in wheels + ): + continue + if distance == 0: + return commit + + diff = ["git", "diff", "--name-only", commit, base_commit, "--"] + blocking, warning = ( + subprocess.check_output([*diff, *paths], text=True).strip() + for paths in (utils.WHEEL_BLOCKING_PATHS, utils.WHEEL_WARNING_PATHS) + ) + if blocking: + raise ValueError( + f"No precompiled wheel is published yet for {base_commit}, and " + f"the nearest one ({commit}, {distance} commits older) was " + f"built from different compiled sources:\n{blocking}\n" + "Wait for wheel publication, rebase onto a commit with a " + "published wheel, build from source (unset " + "VLLM_USE_PRECOMPILED), or set VLLM_PRECOMPILED_WHEEL_COMMIT " + "to a full commit SHA to skip this check." + ) + if warning: + logger.warning( + "Precompiled wheel %s predates changes to:\n%s\n" + "The Rust frontend and packaged files may be stale.", + commit, + warning, + ) + print( + f"No precompiled wheel is published yet for {base_commit}; using " + f"{commit} ({distance} commits older)" + ) + return commit + raise ValueError( + f"No published precompiled wheel for variant {variant} and " + f"architecture {arch} within {utils.WHEEL_SEARCH_DEPTH} commits of " + f"{base_commit}. Wait for wheel publication or build vLLM from source." + ) + @staticmethod def determine_wheel_url() -> tuple[str, str | None]: """Try to determine the precompiled wheel URL or path to use. @@ -904,8 +993,8 @@ class precompiled_wheel_utils: or CUDA variant selected from VLLM_MAIN_CUDA_VERSION, torch, or nvidia-smi If downloading from the nightly repo, the commit can be specified via - VLLM_PRECOMPILED_WHEEL_COMMIT; otherwise, the head commit in the main branch - is used. + VLLM_PRECOMPILED_WHEEL_COMMIT; otherwise, local checkouts search backwards + from their merge-base with upstream main for a compatible published wheel. """ wheel_location = os.getenv("VLLM_PRECOMPILED_WHEEL_LOCATION", None) if wheel_location is not None: @@ -931,6 +1020,14 @@ class precompiled_wheel_utils: ", trying to fetch base commit in main branch" ) commit = precompiled_wheel_utils.get_base_commit_in_main_branch() + if ( + envs.VLLM_USE_PRECOMPILED + and not envs.VLLM_DOCKER_BUILD_CONTEXT + and commit != "nightly" + ): + commit = precompiled_wheel_utils.find_published_wheel_commit( + commit, variant, arch + ) print(f"Using precompiled wheel commit {commit} with variant {variant}") download_filename = None try: @@ -1147,17 +1244,9 @@ class precompiled_wheel_utils: ["git", "fetch", "https://github.com/vllm-project/vllm", "main"] ) - # Then get the commit hash of the current branch that is the same as - # the upstream main commit. - current_branch = ( - subprocess.check_output(["git", "branch", "--show-current"]) - .decode("utf-8") - .strip() - ) - base_commit = ( subprocess.check_output( - ["git", "merge-base", f"{upstream_main_commit}", current_branch] + ["git", "merge-base", upstream_main_commit, "HEAD"] ) .decode("utf-8") .strip() @@ -1309,6 +1398,74 @@ def get_vllm_version() -> str: return version +def _check_requirements_preinstalled(requirements: list[str]) -> None: + """`python setup.py develop` satisfies missing `install_requires` via + setuptools' legacy easy_install path instead of pip's wheel-aware + resolver. That path has repeatedly tried (and failed) to compile old + sdists of fast-moving C-extension packages such as aiohttp from source + against the running Python's headers, e.g. + https://github.com/vllm-project/vllm/issues/34073. Fail fast with an + actionable message instead of a cryptic compiler error. + + Skipped under `--no-deps`, which makes setuptools skip `install_requires` + processing entirely, so the easy_install path this guards against is + unreachable anyway. + """ + from importlib.metadata import PackageNotFoundError + from importlib.metadata import requires as installed_requires + from importlib.metadata import version as installed_version + + from packaging.requirements import Requirement + + missing = [] + checked = set() + + def _check(req_str: str, extra: str | None = None) -> None: + # Strip inline "# comment" suffixes (as pip does when reading + # requirements files); packaging.Requirement cannot parse them. + req_str = re.split(r"\s+#", req_str, maxsplit=1)[0].strip() + try: + req = Requirement(req_str) + except Exception: + return + # `extra` is only set when recursing into an extra's own deps + # below; it selects which `; extra == "..."` markers apply. + if req.marker is not None and not req.marker.evaluate( + {"extra": extra} if extra else None + ): + return + if (req.name, extra) in checked: + return + checked.add((req.name, extra)) + try: + installed = installed_version(req.name) + except PackageNotFoundError: + missing.append(req_str) + return + if req.specifier and not req.specifier.contains(installed, prereleases=True): + missing.append(f"{req_str} (found {req.name}=={installed})") + return + # A requested extra (e.g. "fastapi[standard]") pulls in packages + # that easy_install resolves too, so check those as well. + for extra_name in req.extras: + for dep in installed_requires(req.name) or []: + _check(dep, extra=extra_name) + + for req_str in requirements: + _check(req_str) + + if missing: + raise RuntimeError( + "The following dependencies are missing or outdated:\n " + + "\n ".join(missing) + + "\n\n`python setup.py develop` cannot reliably install these " + "itself (see https://github.com/vllm-project/vllm/issues/34073). " + "Install them with pip first, e.g.:\n" + " pip install -r requirements/rocm.txt\n" + "then re-run `python setup.py develop`." + ) + + def get_requirements() -> list[str]: """Get Python package dependencies from requirements.txt.""" requirements_dir = ROOT_DIR / "requirements" @@ -1339,10 +1496,10 @@ def get_requirements() -> list[str]: # vllm-flash-attn is built only for CUDA 12.x. # Skip for other versions. continue - if "flashinfer-cubin" in req: - # Not on PyPI since 0.6.14 (only https://flashinfer.ai/whl), so - # it cannot be a wheel dependency; flashinfer falls back to - # fetching cubins at runtime when the package is absent. + if "flashinfer-cubin" in req or "flashinfer-jit-cache" in req: + # Not on PyPI (only https://flashinfer.ai/whl), so they + # cannot be wheel dependencies; flashinfer handles the + # absence of pre-compiled cubins/jit-cache at runtime. continue if "nvidia-cutlass-dsl[cu13]" in req and cuda_major == "12": # [cu13] extra is the default; strip it on CUDA 12 builds. @@ -1353,6 +1510,8 @@ def get_requirements() -> list[str]: requirements = modified_requirements elif _is_hip(): requirements = _read_requirements("rocm.txt") + if "develop" in sys.argv[1:] and "--no-deps" not in sys.argv[1:]: + _check_requirements_preinstalled(requirements) elif _is_tpu(): requirements = _read_requirements("tpu.txt") elif _is_cpu(): @@ -1372,7 +1531,7 @@ if _is_cuda() or _is_hip(): # copying the relevant .py files from the source repository. ext_modules.append(CMakeExtension(name="vllm.triton_kernels", optional=True)) -if not _is_xpu() and sys.version_info >= (3, 11): +if not _is_xpu(): ext_modules.append(CMakeExtension(name="vllm.spinloop")) ext_modules.append(CMakeExtension(name="vllm.fs_io_C")) diff --git a/tests/basic_correctness/memory/cumem/test_cumem.py b/tests/basic_correctness/memory/cumem/test_cumem.py index df8c604882..c7ebfd2609 100644 --- a/tests/basic_correctness/memory/cumem/test_cumem.py +++ b/tests/basic_correctness/memory/cumem/test_cumem.py @@ -359,7 +359,7 @@ def test_runtime_state_survives_sleep(level): @create_new_process_for_each_test("fork" if current_platform.is_cuda() else "spawn") @pytest.mark.skipif(current_platform.is_xpu(), reason="Uses the CuMem allocator") def test_level2_discards_ordinary_tensor_with_weights_tag(): - """Reproduce the level-2 variant for an ordinary tensor in weights.""" + """Discarded weights-tag memory is remapped; ROCm zeroes it over stale pages.""" allocator = get_mem_allocator_instance() with allocator.use_memory_pool("weights"): @@ -374,8 +374,9 @@ def test_level2_discards_ordinary_tensor_with_weights_tag(): torch.accelerator.synchronize() assert (fake_weight.data_ptr(), ordinary_tensor.data_ptr()) == pointers - assert torch.all(fake_weight == 0xA5) - assert torch.all(ordinary_tensor == 0xA5) + expected = 0 if current_platform.is_rocm() else 0xA5 + assert torch.all(fake_weight == expected) + assert torch.all(ordinary_tensor == expected) @create_new_process_for_each_test("fork" if current_platform.is_cuda() else "spawn") @@ -427,7 +428,9 @@ def test_cumem_with_cudagraph(): @pytest.mark.parametrize("level", [1, 2], ids=["sleep-1", "sleep-2"]) @create_new_process_for_each_test("fork" if current_platform.is_cuda() else "spawn") -@pytest.mark.skipif(not current_platform.is_cuda(), reason="cuMem CUDA graph pool") +@pytest.mark.skipif( + not current_platform.is_cuda_alike(), reason="cuMem CUDA graph pool" +) def test_cudagraph_pool_sleep(level): """Routing, and the graph pool backed up at both sleep levels and restored in place by any wake.""" diff --git a/tests/basic_correctness/memory/sleep_mode/test_sleep_mode.py b/tests/basic_correctness/memory/sleep_mode/test_sleep_mode.py index 73af6e3910..d897d71001 100644 --- a/tests/basic_correctness/memory/sleep_mode/test_sleep_mode.py +++ b/tests/basic_correctness/memory/sleep_mode/test_sleep_mode.py @@ -209,8 +209,20 @@ def _cudagraph_bytes(worker) -> int: def _custom_ar_active(worker) -> bool: from vllm.distributed.parallel_state import get_tp_group - ca_comm = get_tp_group().device_communicator.ca_comm - return ca_comm is not None and not ca_comm.disabled + comm = get_tp_group().device_communicator + return any( + c is not None and not c.disabled for c in (comm.ca_comm, comm.aiter_ar_comm) + ) + + +def _capture_registers_buffers(worker) -> bool: + from vllm.distributed.parallel_state import get_tp_group + + comm = get_tp_group().device_communicator + ca, aiter = comm.ca_comm, comm.aiter_ar_comm + return (ca is not None and ca._capture_registered) or ( + aiter is not None and aiter.aiter_ca.enable_register_for_capturing + ) @pytest.mark.parametrize( @@ -227,7 +239,9 @@ def _custom_ar_active(worker) -> bool: ids=["full", "piecewise", "breakable", "custom-ar-tp2", "off-by-default"], ) @create_new_process_for_each_test() -@pytest.mark.skipif(not current_platform.is_cuda(), reason="cuMem CUDA graph pool") +@pytest.mark.skipif( + not current_platform.is_cuda_alike(), reason="cuMem CUDA graph pool" +) def test_sleep_cudagraph_pool(monkeypatch, mode, breakable, tp, offload): """Each capture site uses the pool, exact across sleeps; TP=2 needs unregistered custom AR; off changes nothing.""" @@ -255,8 +269,10 @@ def test_sleep_cudagraph_pool(monkeypatch, mode, breakable, tp, offload): assert all(graph_bytes) if offload else not any(graph_bytes) if not offload: return - if tp > 1 and not all(llm.collective_rpc(_custom_ar_active)): - pytest.skip("Custom allreduce is unavailable on these GPUs") + if tp > 1: + if not all(llm.collective_rpc(_custom_ar_active)): + pytest.skip("Custom allreduce is unavailable on these GPUs") + assert not any(llm.collective_rpc(_capture_registers_buffers)) prompt, params = "How are you?", SamplingParams(temperature=0, max_tokens=10) expected = llm.generate(prompt, params)[0].outputs[0].text for level in (1, 2): @@ -268,6 +284,42 @@ def test_sleep_cudagraph_pool(monkeypatch, mode, breakable, tp, offload): assert llm.generate(prompt, params)[0].outputs[0].text == expected +@multi_gpu_test(num_gpus=2) +@create_new_process_for_each_test() +@pytest.mark.skipif(not current_platform.is_rocm(), reason="AITER fused allreduce") +def test_sleep_cudagraph_pool_aiter_fused_ar_group_quant(monkeypatch): + """The fused AITER AR + RMSNorm + per-group FP8 quant captures into the + cuMem pool without IPC-registering its input, across sleeps.""" + for name, value in [ + ("VLLM_USE_V2_MODEL_RUNNER", "1"), + ("VLLM_ROCM_USE_AITER", "1"), + ("VLLM_ALLOW_INSECURE_SERIALIZATION", "1"), + ]: + monkeypatch.setenv(name, value) + monkeypatch.delenv("NCCL_GRAPH_REGISTER", raising=False) + llm = LLM( + "Qwen/Qwen3-0.6B-FP8", + enable_sleep_mode=True, + sleep_mode_offload_cudagraph=True, + tensor_parallel_size=2, + max_model_len=1024, + compilation_config={"pass_config": {"fuse_allreduce_rms": True}}, + ) + if not all(llm.collective_rpc(_custom_ar_active)): + pytest.skip("AITER custom allreduce is unavailable on these GPUs") + assert not any(llm.collective_rpc(_capture_registers_buffers)) + assert all(llm.collective_rpc(_cudagraph_bytes)) + prompt, params = "How are you?", SamplingParams(temperature=0, max_tokens=10) + assert llm.generate(prompt, params)[0].outputs[0].text + for level in (1, 2): + llm.sleep(level=level) + llm.wake_up(tags=["weights"]) + if level == 2: + llm.collective_rpc("reload_weights") + llm.wake_up(tags=["kv_cache"]) + assert llm.generate(prompt, params)[0].outputs[0].text + + @create_new_process_for_each_test() def test_deep_sleep_async(gpu_memory_cleared): async def test(): @@ -323,7 +375,6 @@ def _attn_kv_scales(model) -> dict[str, list[float]]: } -@pytest.mark.skipif(not current_platform.is_cuda(), reason="Segfaults on ROCm.") @requires_fp8 @create_new_process_for_each_test() def test_deep_sleep_compressed_tensors_kv_scales( diff --git a/tests/benchmarks/test_audio_dataset.py b/tests/benchmarks/test_audio_dataset.py index e81c0cd173..54a165dc61 100644 --- a/tests/benchmarks/test_audio_dataset.py +++ b/tests/benchmarks/test_audio_dataset.py @@ -98,6 +98,17 @@ class _FakeSession: return _FakeResponse() +class _FakeProfileSession: + def __init__(self) -> None: + self.url: str | None = None + self.headers: dict[str, str] | None = None + + def post(self, *, url: str, headers: dict[str, str]): + self.url = url + self.headers = headers + return _FakeResponse() + + def test_asr_dataset_sample_handles_local_audio_paths(tmp_path: Path) -> None: audio_path = tmp_path / "earnings.wav" _write_wav(audio_path, duration_s=0.1) @@ -222,6 +233,42 @@ def test_async_request_openai_audio_handles_decoded_audio_arrays( assert output.generated_text == "hello" +def test_audio_request_rejects_profile_endpoint() -> None: + request_input = RequestFuncInput( + prompt="", + api_url="http://localhost:8000/start_profile", + prompt_len=1, + output_len=32, + model="openai/whisper-large-v3", + ) + + with pytest.raises(ValueError, match="OpenAI Audio API"): + asyncio.run( + request_func_module.async_request_openai_audio( + request_input, _FakeProfileSession() + ) + ) + + +@pytest.mark.parametrize("endpoint", ["start_profile", "stop_profile"]) +def test_audio_benchmark_profile_request_has_no_inference_payload( + endpoint: str, +) -> None: + session = _FakeProfileSession() + + output = asyncio.run( + request_func_module.async_request_profile( + f"http://localhost:8000/{endpoint}", + session, + extra_headers={"X-Test-Header": "profile"}, + ) + ) + + assert output.success is True + assert session.url == f"http://localhost:8000/{endpoint}" + assert session.headers == {"X-Test-Header": "profile"} + + _COHERE_ASR_PROMPT = ( "<|startofcontext|><|startoftranscript|>" "<|emo:undefined|><|en|><|en|><|pnc|><|noitn|>" diff --git a/tests/benchmarks/test_responses_request_func.py b/tests/benchmarks/test_responses_request_func.py index 38f1bdc507..1ecf6115ff 100644 --- a/tests/benchmarks/test_responses_request_func.py +++ b/tests/benchmarks/test_responses_request_func.py @@ -183,6 +183,7 @@ def test_request_payload_is_accepted_by_the_server_schema(): "temperature": 0.7, "top_p": 0.9, "top_k": 20, + "min_p": 0.05, "frequency_penalty": 0.2, "presence_penalty": 0.1, "repetition_penalty": 1.05, @@ -217,9 +218,6 @@ def test_request_payload_is_accepted_by_the_server_schema(): assert set(seen["payload"]) <= set(ResponsesRequest.model_fields) ResponsesRequest.model_validate(seen["payload"]) - # `vllm bench serve` rejects --min-p for this backend for the same reason. - assert "min_p" not in ResponsesRequest.model_fields - def test_latency_accounting_matches_the_other_endpoints(): """`latency - ttft` must equal `sum(itl)`. diff --git a/tests/compile/correctness_e2e/test_async_tp.py b/tests/compile/correctness_e2e/test_async_tp.py deleted file mode 100644 index 307f39039b..0000000000 --- a/tests/compile/correctness_e2e/test_async_tp.py +++ /dev/null @@ -1,270 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# SPDX-FileCopyrightText: Copyright contributors to the vLLM project - -import json -import logging -from contextlib import nullcontext - -import pytest - -from tests.compile.fusions_e2e.common import FUSION_LOG_PATTERNS -from tests.models.registry import HF_EXAMPLE_MODELS -from tests.utils import ( - compare_two_settings, - create_new_process_for_each_test, - multi_gpu_test, -) -from vllm.config import ( - CompilationMode, -) -from vllm.platforms import current_platform -from vllm.transformers_utils.repo_utils import hf_api -from vllm.utils.flashinfer import has_flashinfer - -NVFP4_MODEL_ID = "nvidia/Llama-3.1-8B-Instruct-NVFP4" -NVFP4_HF_OVERRIDES = { - "num_hidden_layers": 4, - "hidden_size": 512, - "intermediate_size": 800, - "num_attention_heads": 4, - "num_key_value_heads": 1, -} - - -@multi_gpu_test(num_gpus=2) -@pytest.mark.skipif(not current_platform.is_rocm(), reason="ROCm AsyncTP contract") -def test_rocm_async_tp_bf16_output_correctness(caplog_mp_spawn): - """Trained-model outputs must agree with unfused TP, including padded decode.""" - # A snapshot path pins both servers and the comparison helper's tokenizer. - model = hf_api().snapshot_download( - "Qwen/Qwen3-0.6B", revision="c1899de289a04d12100db370d81485cdf75e47ca" - ) - common_args = [ - "--dtype", - "bfloat16", - "--load-format", - "auto", - "--tensor-parallel-size", - "2", - "--distributed-executor-backend", - "mp", - "--max-model-len", - "512", - "--max-num-batched-tokens", - "512", - "--max-num-seqs", - "8", - "--kv-cache-memory-bytes", - str(64 * 1024**2), - "--gpu-memory-utilization", - "0.01", - "--attention-backend", - "TRITON_ATTN", - "--disable-custom-all-reduce", - ] - - def settings(enabled): - return [ - *common_args, - "--compilation-config", - json.dumps( - { - "mode": CompilationMode.VLLM_COMPILE, - "cudagraph_mode": "NONE", - "splitting_ops": [], - "inductor_compile_config": {"force_disable_caches": True}, - "pass_config": { - "enable_sp": enabled, - "fuse_gemm_comms": enabled, - "fuse_allreduce_rms": False, - "sp_min_token_num": 1, - }, - } - ), - ] - - env = {"VLLM_ROCM_USE_AITER": "0", "VLLM_DISABLE_COMPILE_CACHE": "1"} - with caplog_mp_spawn(logging.DEBUG) as logs: - compare_two_settings( - model, - settings(False), - settings(True), - env1=env, - env2=env, - method="generate", - force_v1_runner=True, - ) - matches = [ - int(count) for count in FUSION_LOG_PATTERNS["async_tp"].findall(logs.text) - ] - print(f"AsyncTP rewrite counts: {matches}") - assert len(matches) >= 2 and all(count > 0 for count in matches), ( - f"Expected positive AsyncTP rewrites on both TP workers, found {matches}" - ) - - -@create_new_process_for_each_test() -@pytest.mark.parametrize( - "model_id", - ["meta-llama/Llama-3.2-1B-Instruct", "RedHatAI/Llama-3.2-1B-Instruct-FP8"], -) -@pytest.mark.parametrize("tp_size", [2]) -@pytest.mark.parametrize("async_tp_enabled", [True]) -@pytest.mark.parametrize("distributed_backend", ["mp"]) -def test_async_tp_pass_correctness( - model_id: str, - tp_size: int, - async_tp_enabled: bool, - distributed_backend: str, - num_gpus_available: int, - monkeypatch, -): - # Disable FlashInfer FP8 scaled_mm kernel as it is incompatible with - # async TP patterns. No-op on H100 (kernel requires CC >= 100). - monkeypatch.setenv("VLLM_DISABLED_KERNELS", "FlashInferFP8ScaledMMLinearKernel") - - model_info = HF_EXAMPLE_MODELS.find_hf_info(model_id) - model_info.check_transformers_version(on_fail="skip") - model_info.check_available_online(on_fail="skip") - - pp_size = 1 - if num_gpus_available < tp_size: - pytest.skip(f"Need at least {tp_size} x {pp_size} GPUs") - - common_args = [ - "--dtype", - "bfloat16", - "--max-model-len", - "2048", - "--max-num-seqs", - "8", - ] - - compilation_config = { - "mode": CompilationMode.VLLM_COMPILE, - "compile_sizes": [2, 4, 8], - "splitting_ops": [], - "pass_config": {"fuse_gemm_comms": async_tp_enabled}, - } - - async_tp_args = [ - *common_args, - "--tensor-parallel-size", - str(tp_size), - "--distributed-executor-backend", - distributed_backend, - "--compilation_config", - json.dumps(compilation_config), - ] - - tp_args = [ - *common_args, - "--tensor-parallel-size", - str(tp_size), - "--distributed-executor-backend", - "mp", - ] - - compare_two_settings( - model_id, - async_tp_args, - tp_args, - method="generate", - force_v1_runner=True, - ) - - -@create_new_process_for_each_test() -def test_async_tp_pass_nvfp4_correctness(num_gpus_available: int, caplog_mp_spawn): - is_blackwell = current_platform.is_cuda() and ( - current_platform.is_device_capability_family(100) - ) - is_rocm_cdna = False - if current_platform.is_rocm(): - from vllm.platforms.rocm import on_cdna - - is_rocm_cdna = on_cdna() - if not (is_blackwell or is_rocm_cdna): - pytest.skip("NVFP4 AsyncTP requires Blackwell or ROCm CDNA") - if is_blackwell and not has_flashinfer(): - pytest.skip("FlashInfer is required for the NVFP4 AsyncTP path") - - tp_size = 2 - if num_gpus_available < tp_size: - pytest.skip(f"Need at least {tp_size} GPUs") - - common_args = [ - "--dtype", - "bfloat16", - "--max-model-len", - "2048", - "--max-num-seqs", - "8", - "--load-format", - "dummy", - *(["--linear-backend", "flashinfer_cutlass"] if is_blackwell else []), - "--hf-overrides", - json.dumps(NVFP4_HF_OVERRIDES), - ] - - compilation_config = { - "mode": CompilationMode.VLLM_COMPILE, - "compile_sizes": [2, 4, 8], - "splitting_ops": [], - "pass_config": { - "enable_sp": True, - "fuse_gemm_comms": True, - "fuse_allreduce_rms": False, - "sp_min_token_num": 1, - }, - } - if is_rocm_cdna: - compilation_config["inductor_compile_config"] = {"force_disable_caches": True} - - async_tp_args = [ - *common_args, - "--tensor-parallel-size", - str(tp_size), - "--distributed-executor-backend", - "mp", - "--compilation_config", - json.dumps(compilation_config), - ] - - tp_args = [ - *common_args, - "--tensor-parallel-size", - str(tp_size), - "--distributed-executor-backend", - "mp", - ] - - env = {"VLLM_DISABLE_COMPILE_CACHE": "1"} if is_rocm_cdna else None - log_context = caplog_mp_spawn(logging.DEBUG) if is_rocm_cdna else nullcontext() - with log_context as logs: - compare_two_settings( - NVFP4_MODEL_ID, - async_tp_args, - tp_args, - env1=env, - env2=env, - method="generate", - force_v1_runner=True, - ) - if is_rocm_cdna: - assert logs is not None - matches_by_rank = { - rank: [ - int(count) - for line in logs.text.splitlines() - if line.startswith(f"(Worker_TP{rank} pid=") - for count in FUSION_LOG_PATTERNS["async_tp"].findall(line) - ] - for rank in range(tp_size) - } - assert all( - any(count > 0 for count in matches) for matches in matches_by_rank.values() - ), ( - "Expected positive NVFP4 AsyncTP rewrites on both TP workers, " - f"found {matches_by_rank}" - ) diff --git a/tests/compile/correctness_e2e/test_sequence_parallel.py b/tests/compile/correctness_e2e/test_sequence_parallel.py deleted file mode 100644 index 10d25320c0..0000000000 --- a/tests/compile/correctness_e2e/test_sequence_parallel.py +++ /dev/null @@ -1,414 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# SPDX-FileCopyrightText: Copyright contributors to the vLLM project -"""WARNING: This test runs in both single-node (4 GPUs) and multi-node -(2 node with 2 GPUs each) modes. If the test only uses 2 GPUs, it is -important to set the distributed backend to "mp" to avoid Ray scheduling -all workers in a node other than the head node, which can cause the test -to fail. -""" - -import json -import os -from dataclasses import dataclass -from typing import Literal, NamedTuple - -import pytest - -from vllm.config.compilation import CompilationMode -from vllm.config.model import RunnerOption -from vllm.logger import init_logger -from vllm.platforms import current_platform -from vllm.utils.torch_utils import is_torch_equal_or_newer - -from ...models.registry import HF_EXAMPLE_MODELS, _HfExamplesInfo -from ...utils import compare_all_settings, create_new_process_for_each_test - -logger = init_logger("test_sequence_parallel") - -VLLM_MULTI_NODE = os.getenv("VLLM_MULTI_NODE", "0") == "1" -NVFP4_MODEL_ID = "nvidia/Llama-3.1-8B-Instruct-NVFP4" -NVFP4_MODEL_INFO = _HfExamplesInfo(NVFP4_MODEL_ID) - - -class ParallelSetup(NamedTuple): - tp_size: int - pp_size: int - fuse_norm_quant: bool - fuse_act_quant: bool - eager_mode: bool - - -class SPTestOptions(NamedTuple): - multi_node_only: bool - load_format: str | None = None - model_info: _HfExamplesInfo | None = None - - -@dataclass -class SPTestSettings: - parallel_setups: list[ParallelSetup] - distributed_backends: list[str] - runner: RunnerOption - test_options: SPTestOptions - - @staticmethod - def fast( - *, - tp_base: int = 2, - pp_base: int = 1, - runner: RunnerOption = "auto", - multi_node_only: bool = False, - load_format: str | None = None, - ): - parallel_setups = [ - ParallelSetup( - tp_size=tp_base, - pp_size=pp_multiplier * pp_base, - fuse_norm_quant=False, - fuse_act_quant=False, - eager_mode=False, - ) - for pp_multiplier in [1, 2] - ] - return SPTestSettings( - parallel_setups=parallel_setups, - distributed_backends=["mp", "ray"], - runner=runner, - test_options=SPTestOptions( - multi_node_only=multi_node_only, load_format=load_format - ), - ) - - @staticmethod - def fp8_quant( - *, - tp_base: int = 2, - pp_base: int = 1, - runner: RunnerOption = "auto", - multi_node_only: bool = False, - load_format: str | None = None, - ): - parallel_setups = [] - for fusion_val in [False, True]: - parallel_setups.append( - ParallelSetup( - tp_size=tp_base, - pp_size=pp_base, - fuse_norm_quant=fusion_val, - fuse_act_quant=fusion_val, - eager_mode=True, - ) - ) - return SPTestSettings( - parallel_setups=parallel_setups, - distributed_backends=["mp", "ray"], - runner=runner, - test_options=SPTestOptions( - multi_node_only=multi_node_only, load_format=load_format - ), - ) - - def iter_params(self, model_id: str): - opts = self.test_options - - for parallel_setup in self.parallel_setups: - for backend in self.distributed_backends: - yield ( - model_id, - parallel_setup, - backend, - self.runner, - opts, - ) - - -def _build_sp_args( - model_id: str, - parallel_setup: ParallelSetup, - distributed_backend: str, - runner: RunnerOption, - test_options: SPTestOptions, - num_gpus_available: int, - use_inductor_graph_partition: bool, - fuse_gemm_comms: bool, - enable_prompt_embeds: bool, - *, - is_multimodal: bool, -) -> tuple[list[str], list[str]] | None: - ( - tp_size, - pp_size, - fuse_norm_quant, - fuse_act_quant, - eager_mode, - ) = parallel_setup - - multi_node_only = test_options.multi_node_only - load_format = test_options.load_format - - model_info = test_options.model_info or HF_EXAMPLE_MODELS.find_hf_info(model_id) - model_info.check_transformers_version(on_fail="skip") - - trust_remote_code = model_info.trust_remote_code - tokenizer_mode = model_info.tokenizer_mode - hf_overrides = dict(model_info.hf_overrides) - require_embed_inputs = model_info.require_embed_inputs - - if load_format == "dummy": - # Avoid OOM - text_overrides = { - "num_hidden_layers": 4, - "hidden_size": 512, - "intermediate_size": 800, - "num_attention_heads": 4, - "num_key_value_heads": 1, - } - - if is_multimodal: - hf_overrides.update({"text_config": text_overrides}) - else: - hf_overrides.update(text_overrides) - else: - model_info.check_available_online(on_fail="skip") - - if num_gpus_available < tp_size * pp_size: - return None - if VLLM_MULTI_NODE and distributed_backend == "mp": - return None - if multi_node_only and not VLLM_MULTI_NODE: - return None - - common_args = [ - "--dtype", - "bfloat16", - "--max-model-len", - "2048", - "--max-num-seqs", - "8", - ] - if eager_mode: - common_args.append("-cc.cudagraph_mode=none") - if runner != "auto": - common_args.extend(["--runner", runner]) - if trust_remote_code: - common_args.append("--trust-remote-code") - if tokenizer_mode: - common_args.extend(["--tokenizer-mode", tokenizer_mode]) - if load_format: - common_args.extend(["--load-format", load_format]) - if hf_overrides: - common_args.extend(["--hf-overrides", json.dumps(hf_overrides)]) - if require_embed_inputs: - common_args.extend( - [ - "--skip-tokenizer-init", - "--enable-prompt-embeds", - "--enable-mm-embeds", - ] - ) - elif enable_prompt_embeds: - common_args.append("--enable-prompt-embeds") - - compilation_config = { - "mode": CompilationMode.VLLM_COMPILE, - "compile_sizes": [4, 8], - "pass_config": { - "enable_sp": True, - "fuse_gemm_comms": fuse_gemm_comms, - "fuse_norm_quant": fuse_norm_quant, - "fuse_act_quant": fuse_act_quant, - "fuse_allreduce_rms": False, - "eliminate_noops": True, - "sp_min_token_num": 0, - }, - "use_inductor_graph_partition": use_inductor_graph_partition, - } - if not use_inductor_graph_partition: - compilation_config["splitting_ops"] = [] - - tp_sp_args = [ - *common_args, - "--tensor-parallel-size", - str(tp_size), - "--pipeline-parallel-size", - str(pp_size), - "--distributed-executor-backend", - distributed_backend, - "--compilation_config", - json.dumps(compilation_config), - ] - - tp_args = [ - *common_args, - "--tensor-parallel-size", - str(tp_size), - "--distributed-executor-backend", - "mp", - ] - - return tp_args, tp_sp_args - - -def _compare_sp_settings( - model_id: str, - settings: list[tuple[list[str], list[str]]], - *, - method: Literal["generate", "encode"], -) -> None: - if not settings: - pytest.skip("No supported sequence-parallel configurations") - - settings_by_tp_args: dict[tuple[str, ...], list[list[str]]] = {} - for tp_args, tp_sp_args in settings: - settings_by_tp_args.setdefault(tuple(tp_args), []).append(tp_sp_args) - - for tp_args_key, grouped_tp_sp_args in settings_by_tp_args.items(): - all_args = [list(tp_args_key), *grouped_tp_sp_args] - compare_all_settings( - model_id, - all_args, - [None] * len(all_args), - method=method, - force_v1_runner=True, - ) - - -SP_TEXT_GENERATION_MODELS = { - # [Decoder-only] - "hmellor/tiny-random-LlamaForCausalLM": SPTestSettings.fast(), - "RedHatAI/Llama-3.2-1B-Instruct-FP8": SPTestSettings.fp8_quant(), -} - - -@pytest.mark.parametrize( - ("model_id", "settings"), - [ - pytest.param(model_id, settings, id=model_id) - for model_id, settings in SP_TEXT_GENERATION_MODELS.items() - ], -) -@create_new_process_for_each_test() -def test_tp_sp_generation( - model_id: str, - settings: SPTestSettings, - num_gpus_available, -): - # Skip FP8 SP-only test on sm89 (compute capability 8.9). - # An unknown capability (None) is left to fail in the test body rather than - # being silently skipped here. - capability = current_platform.get_device_capability() - if "fp8" in model_id.lower() and capability is not None and capability < (9, 0): - pytest.skip("FP8 reduction support begins with sm90 capable devices.") - - graph_partition_options = [False] - if is_torch_equal_or_newer("2.9.0.dev"): - graph_partition_options.append(True) - - comparisons = [] - for _, parallel_setup, backend, runner, test_options in settings.iter_params( - model_id - ): - for use_inductor_graph_partition in graph_partition_options: - comparison = _build_sp_args( - model_id, - parallel_setup, - backend, - runner, - test_options, - num_gpus_available, - use_inductor_graph_partition, - fuse_gemm_comms=False, # TODO: enable async TP - enable_prompt_embeds=False, - is_multimodal=False, - ) - if comparison is not None: - comparisons.append(comparison) - - _compare_sp_settings(model_id, comparisons, method="generate") - - -# Focused regression test for the SP + prompt_embeds graph-rewrite path. -# Covers pp_size=1 (SP only) and pp_size=2 (SP + PP); kept small on purpose so -# we don't double the matrix of `test_tp_sp_generation` above. -SP_PROMPT_EMBEDS_PARALLEL_SETUPS = [ - ParallelSetup( - tp_size=2, - pp_size=pp_size, - fuse_norm_quant=False, - fuse_act_quant=False, - eager_mode=False, - ) - for pp_size in [1, 2] -] - - -@create_new_process_for_each_test() -def test_tp_sp_generation_prompt_embeds( - num_gpus_available, -): - model_id = "hmellor/tiny-random-LlamaForCausalLM" - graph_partition_options = [False] - if is_torch_equal_or_newer("2.9.0.dev"): - graph_partition_options.append(True) - - comparisons = [] - for parallel_setup in SP_PROMPT_EMBEDS_PARALLEL_SETUPS: - for use_inductor_graph_partition in graph_partition_options: - comparison = _build_sp_args( - model_id, - parallel_setup, - distributed_backend="mp", - runner="auto", - test_options=SPTestOptions(multi_node_only=False, load_format=None), - num_gpus_available=num_gpus_available, - use_inductor_graph_partition=use_inductor_graph_partition, - fuse_gemm_comms=False, - enable_prompt_embeds=True, - is_multimodal=False, - ) - if comparison is not None: - comparisons.append(comparison) - - _compare_sp_settings(model_id, comparisons, method="generate") - - -@create_new_process_for_each_test() -def test_tp_sp_nvfp4_generation(num_gpus_available: int): - is_blackwell = current_platform.is_cuda() and ( - current_platform.is_device_capability_family(100) - ) - is_rocm_cdna = False - if current_platform.is_rocm(): - from vllm.platforms.rocm import on_cdna - - is_rocm_cdna = on_cdna() - if not (is_blackwell or is_rocm_cdna): - pytest.skip("NVFP4 sequence parallelism requires Blackwell or ROCm CDNA") - - comparison = _build_sp_args( - NVFP4_MODEL_ID, - ParallelSetup( - tp_size=2, - pp_size=1, - fuse_norm_quant=True, - fuse_act_quant=True, - eager_mode=True, - ), - "mp", - "auto", - SPTestOptions( - multi_node_only=False, - load_format="dummy", - model_info=NVFP4_MODEL_INFO, - ), - num_gpus_available, - use_inductor_graph_partition=False, - fuse_gemm_comms=False, - enable_prompt_embeds=False, - is_multimodal=False, - ) - _compare_sp_settings( - NVFP4_MODEL_ID, - [] if comparison is None else [comparison], - method="generate", - ) diff --git a/tests/compile/fullgraph/test_full_graph.py b/tests/compile/fullgraph/test_full_graph.py index 4ce7d6767f..fceed6346d 100644 --- a/tests/compile/fullgraph/test_full_graph.py +++ b/tests/compile/fullgraph/test_full_graph.py @@ -10,11 +10,17 @@ import torch from tests.quantization.utils import is_quant_method_supported from vllm import LLM, SamplingParams -from vllm.config import CompilationConfig, CompilationMode, CUDAGraphMode, PassConfig +from vllm.config import ( + CacheConfig, + CompilationConfig, + CompilationMode, + CUDAGraphMode, + PassConfig, +) from vllm.platforms import current_platform from vllm.utils.torch_utils import is_torch_equal_or_newer -from ...utils import create_new_process_for_each_test +from ...utils import create_new_process_for_each_test, wait_for_memory_to_settle def models_list(*, all: bool = True, keywords: list[str] | None = None): @@ -173,6 +179,14 @@ def run_model( if compilation_config.cudagraph_mode is None: compilation_config.cudagraph_mode = CUDAGraphMode.NONE + # V1 startup requires free_memory >= total * gpu_memory_utilization, so a + # device that is not free yet fails engine init rather than waiting for it. + # VllmRunner guards this; building LLM directly does not. No-op off ROCm. + wait_for_memory_to_settle( + threshold_ratio=1.0 + - model_kwargs.get("gpu_memory_utilization", CacheConfig.gpu_memory_utilization) + ) + llm = LLM( model=model, compilation_config=compilation_config, diff --git a/tests/compile/fusions_e2e/common.py b/tests/compile/fusions_e2e/common.py index 1f92ff5a19..a4d996b48d 100644 --- a/tests/compile/fusions_e2e/common.py +++ b/tests/compile/fusions_e2e/common.py @@ -21,8 +21,6 @@ class Matches(NamedTuple): # distributed ar_rms_fusion: int = 0 aiter_ar_rms_fusion: int = 0 - sequence_parallel: int = 0 - async_tp: int = 0 class ModelFusionInfo(NamedTuple): @@ -143,8 +141,4 @@ FUSION_LOG_PATTERNS: dict[str, re.Pattern] = { "aiter_ar_rms_fusion": re.compile( r"RocmAiterAllReduceFusionPass Replaced (\d+) patterns" ), - "sequence_parallel": re.compile( - r"sequence_parallelism.py:\d+] Replaced (\d+) patterns" - ), - "async_tp": re.compile(r"collective_fusion.py:\d+] Replaced (\d+) patterns"), } diff --git a/tests/compile/fusions_e2e/conftest.py b/tests/compile/fusions_e2e/conftest.py index db7ba05a57..8da7aa2a5a 100644 --- a/tests/compile/fusions_e2e/conftest.py +++ b/tests/compile/fusions_e2e/conftest.py @@ -170,26 +170,8 @@ def run_e2e_fusion_test(monkeypatch, caplog_mp_spawn, vllm_runner): for match_name in matches_check: log_matches = list(int(ms) for ms in log_matches_dict[match_name]) - # AR+RMS skips the largest range; SP skips the smallest. - # When both are enabled, AR+RMS activation count is - # model-dependent (hidden_size affects threshold), so derive - # from log data. - if ( - match_name == "ar_rms_fusion" - and "sequence_parallel" in matches_check - and num_compile_ranges >= 2 - ): - assert ( - len(log_matches) >= tp_size and len(log_matches) % tp_size == 0 - ), ( - f"Expected multiple of {tp_size} ar_rms log entries, " - f"found {len(log_matches)}" - ) - num_ranges_activated = len(log_matches) // tp_size - elif ( - match_name in ("ar_rms_fusion", "sequence_parallel") - and num_compile_ranges >= 2 - ): + # AR+RMS skips the largest range. + if match_name == "ar_rms_fusion" and num_compile_ranges >= 2: num_ranges_activated = num_compile_ranges - 1 else: num_ranges_activated = num_compile_ranges @@ -233,44 +215,6 @@ def run_e2e_fusion_test(monkeypatch, caplog_mp_spawn, vllm_runner): f"Expecting at least {expected_matches - matches.ar_rms_fusion} " f"where ar+rms+quant was activated" ) - elif ( - match_name == "async_tp" - and "sequence_parallel" in matches_check - and num_compile_ranges >= 2 - ): - # AsyncTP only finds patterns on ranges where SP ran. - n_sp_ranges = num_compile_ranges - 1 - assert ( - sum(m == expected_matches for m in log_matches) - == tp_size * n_sp_ranges - ), ( - f"Expecting {expected_matches} async_tp on " - f"{tp_size * n_sp_ranges} SP-range entries, " - f"found: {log_matches}" - ) - assert sum(m == 0 for m in log_matches) == tp_size, ( - f"Expecting 0 async_tp on {tp_size} small-range entries " - f"(no SP), found: {log_matches}" - ) - elif ( - match_name == "ar_rms_fusion" - and "sequence_parallel" in matches_check - and num_compile_ranges >= 2 - ): - # SP consumes allreduce patterns first, so AR+RMS finds - # full matches only on the smallest range (no SP). - assert sum(m == expected_matches for m in log_matches) == tp_size, ( - f"Expecting {expected_matches} ar_rms on " - f"{tp_size} small-range entries, found: {log_matches}" - ) - assert sum(m == 0 for m in log_matches) == tp_size * ( - num_ranges_activated - 1 - ), ( - f"Expecting 0 ar_rms on " - f"{tp_size * (num_ranges_activated - 1)} large-range " - f"entries (SP took precedence), found: {log_matches}" - ) - elif match_name == "act_quant_fusion": actual_match = match_table.get("activation_quant_fusion_pass", 0) assert actual_match == expected_matches * n_expected, ( @@ -305,17 +249,4 @@ def run_e2e_fusion_test(monkeypatch, caplog_mp_spawn, vllm_runner): f"(found {len(log_matches)}) in:\n {log_holder.text}" ) - if match_name == "sequence_parallel" and num_compile_ranges >= 2: - log_matches = re.findall( - r"pass_manager.py:\d+] Skipping " - r".*SequenceParallelismPass.* with compile range", - log_holder.text, - ) - - n_expected = tp_size * (num_compile_ranges - num_ranges_activated) - assert len(log_matches) == n_expected, ( - f'Could not find {n_expected} "Skipping SequenceParallelismPass" ' - f"(found {len(log_matches)}) in:\n {log_holder.text}" - ) - return run diff --git a/tests/compile/fusions_e2e/models.py b/tests/compile/fusions_e2e/models.py index 082ab0bf52..d80d171984 100644 --- a/tests/compile/fusions_e2e/models.py +++ b/tests/compile/fusions_e2e/models.py @@ -73,8 +73,6 @@ llama3_8b = ModelFusionInfo( matches=lambda n_layers: Matches( ar_rms_fusion=n_layers * 2 + 1, aiter_ar_rms_fusion=n_layers * 2 + 1, - sequence_parallel=n_layers * 2 + 1, - async_tp=n_layers * 4, ), ) @@ -85,8 +83,6 @@ llama3_8b_fp8 = ModelFusionInfo( act_quant_fusion=n_layers, attn_quant_fusion=n_layers, ar_rms_fusion=n_layers * 2 + 1, - sequence_parallel=n_layers * 2 + 1, - async_tp=n_layers * 4, ), ) @@ -96,8 +92,6 @@ llama3_8b_fp4 = ModelFusionInfo( act_quant_fusion=n_layers, attn_quant_fusion=n_layers, ar_rms_fusion=n_layers * 2 + 1, - sequence_parallel=n_layers * 2 + 1, - async_tp=n_layers * 4, ), ) @@ -127,8 +121,6 @@ llama4_scout_fp8 = ModelFusionInfo( rms_quant_fusion=n_layers, attn_quant_fusion=n_layers, ar_rms_fusion=n_layers * 2, - sequence_parallel=n_layers * 2, - async_tp=n_layers * 2 - 1, ), ) @@ -138,8 +130,6 @@ llama4_scout_fp4 = ModelFusionInfo( matches=lambda n_layers: Matches( attn_quant_fusion=n_layers, ar_rms_fusion=n_layers * 2, - sequence_parallel=n_layers * 2, - async_tp=n_layers * 2 - 1, ), ) @@ -149,8 +139,6 @@ qwen3_a3b = ModelFusionInfo( norm_rope_fusion=n_layers, ar_rms_fusion=n_layers * 2 + 1, aiter_ar_rms_fusion=n_layers * 2 + 1, - sequence_parallel=n_layers * 2 + 1, - async_tp=n_layers * 2, ), ) @@ -161,8 +149,6 @@ qwen3_a3b_fp8 = ModelFusionInfo( norm_rope_fusion=n_layers, attn_quant_fusion=0, # attn + group quant not supported ar_rms_fusion=n_layers * 2 + 1, - sequence_parallel=n_layers * 2 + 1, - async_tp=n_layers * 2, ), ) @@ -192,9 +178,6 @@ deepseek_v3_fp8 = ModelFusionInfo( # MLA attn + per-group FP8 quant attn_quant_fusion=n_layers, ar_rms_fusion=n_layers * 2 + 1, - # TODO - # sequence_parallel= n_layers * 2 + 1, - # async_tp=n_layers * 2, ), ) @@ -213,8 +196,6 @@ gpt_oss_20b = ModelFusionInfo( matches=lambda n_layers: Matches( ar_rms_fusion=n_layers * 2 + 1, aiter_ar_rms_fusion=n_layers + 1, - sequence_parallel=n_layers * 2 + 1, - async_tp=n_layers * 2, ), model_kwargs=( {"quantization_config": {"moe": {"activation": "mxfp8"}}} diff --git a/tests/compile/fusions_e2e/test_tp2_async_tp.py b/tests/compile/fusions_e2e/test_tp2_async_tp.py deleted file mode 100644 index b0d32dad1c..0000000000 --- a/tests/compile/fusions_e2e/test_tp2_async_tp.py +++ /dev/null @@ -1,339 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# SPDX-FileCopyrightText: Copyright contributors to the vLLM project -from collections.abc import Callable - -import pytest - -from vllm.config import PassConfig -from vllm.platforms import current_platform - -from ...utils import multi_gpu_test -from .common import ( - INDUCTOR_GRAPH_PARTITION, - AttentionBackendCase, - Matches, - custom_ops_combos, - is_blackwell, - nvfp4_kernel_exposes_input_quant_key, -) -from .models import ( - FLASHINFER_ATTN, - TRITON_ATTN, - llama3_8b, - llama3_8b_fp4, - llama3_8b_fp8, - llama4_scout_fp8, - qwen3_a3b, -) - -pytestmark = pytest.mark.skipif(not current_platform.is_cuda(), reason="Only test CUDA") - - -@multi_gpu_test(num_gpus=2) -@pytest.mark.parametrize( - "model_name, matches_fn, model_kwargs, hf_overrides", - [llama3_8b_fp8, llama4_scout_fp8], -) -@pytest.mark.parametrize("attn_backend", [TRITON_ATTN, FLASHINFER_ATTN]) -@pytest.mark.parametrize("n_layers", [4]) -@pytest.mark.parametrize("custom_ops", custom_ops_combos("quant_fp8", "rms_norm")) -@pytest.mark.parametrize("inductor_graph_partition", INDUCTOR_GRAPH_PARTITION) -def test_tp2_async_tp_fp8_fusions( - model_name: str, - matches_fn: Callable[[int], Matches], - model_kwargs: dict, - hf_overrides: Callable[[int], dict], - attn_backend: AttentionBackendCase, - n_layers: int, - custom_ops: str, - inductor_graph_partition: bool, - run_e2e_fusion_test, -): - matches = matches_fn(n_layers) - - # Reduce size of model and skip weight loading time - model_kwargs["hf_overrides"] = hf_overrides(n_layers) - model_kwargs["load_format"] = "dummy" - model_kwargs["max_model_len"] = 1024 - model_kwargs["kernel_config"] = {"enable_flashinfer_autotune": False} - - compilation_config = dict( - use_inductor_graph_partition=inductor_graph_partition, - custom_ops=custom_ops.split(","), - pass_config=PassConfig( - fuse_norm_quant=True, - fuse_act_quant=False, - fuse_attn_quant=True, - enable_qk_norm_rope_fusion=True, - enable_sp=True, - fuse_gemm_comms=True, - fuse_allreduce_rms=False, - # Override threshold for testing (models have small hidden_size) - sp_min_token_num=512, - ), - ) - - matches_check = [ - "rms_quant_fusion", - "norm_rope_fusion", - "attn_quant_fusion", - "sequence_parallel", - "async_tp", - ] - - run_e2e_fusion_test( - model_name, - matches, - model_kwargs, - attn_backend, - compilation_config, - matches_check, - tp_size=2, - ) - - -@multi_gpu_test(num_gpus=2) -@pytest.mark.parametrize( - "model_name, matches_fn, model_kwargs, hf_overrides", - [llama3_8b_fp4], -) -@pytest.mark.parametrize("attn_backend", [FLASHINFER_ATTN]) -@pytest.mark.parametrize("n_layers", [4]) -@pytest.mark.parametrize("custom_ops", custom_ops_combos("rms_norm")) -@pytest.mark.parametrize("inductor_graph_partition", INDUCTOR_GRAPH_PARTITION) -@pytest.mark.skipif(not is_blackwell(), reason="Blackwell required for fp4") -@pytest.mark.skipif(not current_platform.is_cuda(), reason="Only test CUDA") -def test_tp2_async_tp_nvfp4_fusions( - model_name: str, - matches_fn: Callable[[int], Matches], - model_kwargs: dict, - hf_overrides: Callable[[int], dict], - attn_backend: AttentionBackendCase, - n_layers: int, - custom_ops: str, - inductor_graph_partition: bool, - run_e2e_fusion_test, -): - if nvfp4_kernel_exposes_input_quant_key(): - pytest.skip( - "NVFP4 kernel exposes input_quant_key; manual fusion fires " - "instead of compiler pass-based fusion" - ) - - # NVFP4 currently wires the all-gather + GEMM path only. - matches = matches_fn(n_layers)._replace(async_tp=n_layers * 2) - - # Reduce size of model and skip weight loading time - model_kwargs["hf_overrides"] = hf_overrides(n_layers) - model_kwargs["load_format"] = "dummy" - model_kwargs["max_model_len"] = 1024 - model_kwargs["kernel_config"] = {"enable_flashinfer_autotune": False} - - compilation_config = dict( - use_inductor_graph_partition=inductor_graph_partition, - custom_ops=custom_ops.split(","), - pass_config=PassConfig( - fuse_act_quant=False, - fuse_attn_quant=True, - enable_sp=True, - fuse_gemm_comms=True, - fuse_allreduce_rms=False, - # Override threshold for testing (models have small hidden_size) - sp_min_token_num=512, - ), - ) - - matches_check = [ - "attn_quant_fusion", - "sequence_parallel", - "async_tp", - ] - - run_e2e_fusion_test( - model_name, - matches, - model_kwargs, - attn_backend, - compilation_config, - matches_check, - tp_size=2, - ) - - -@multi_gpu_test(num_gpus=2) -@pytest.mark.parametrize( - "model_name, matches_fn, model_kwargs, hf_overrides", - [llama3_8b, qwen3_a3b], -) -@pytest.mark.parametrize("attn_backend", [TRITON_ATTN]) -@pytest.mark.parametrize("n_layers", [4]) -@pytest.mark.parametrize("custom_ops", custom_ops_combos("rms_norm")) -@pytest.mark.parametrize("inductor_graph_partition", INDUCTOR_GRAPH_PARTITION) -def test_tp2_async_tp_fusions( - model_name: str, - matches_fn: Callable[[int], Matches], - model_kwargs: dict, - hf_overrides: Callable[[int], dict], - attn_backend: AttentionBackendCase, - n_layers: int, - custom_ops: str, - inductor_graph_partition: bool, - run_e2e_fusion_test, -): - matches = matches_fn(n_layers) - - # Reduce size of model and skip weight loading time - model_kwargs["hf_overrides"] = hf_overrides(n_layers) - model_kwargs["load_format"] = "dummy" - model_kwargs["max_model_len"] = 1024 - model_kwargs["kernel_config"] = {"enable_flashinfer_autotune": False} - - compilation_config = dict( - use_inductor_graph_partition=inductor_graph_partition, - custom_ops=custom_ops.split(","), - pass_config=PassConfig( - enable_qk_norm_rope_fusion=True, - enable_sp=True, - fuse_gemm_comms=True, - fuse_allreduce_rms=False, - # Override threshold for testing (models have small hidden_size) - sp_min_token_num=512, - ), - ) - - matches_check = [ - "norm_rope_fusion", - "sequence_parallel", - "async_tp", - ] - - run_e2e_fusion_test( - model_name, - matches, - model_kwargs, - attn_backend, - compilation_config, - matches_check, - tp_size=2, - ) - - -@multi_gpu_test(num_gpus=2) -@pytest.mark.parametrize( - "model_name, matches_fn, model_kwargs, hf_overrides", - [llama3_8b_fp8, llama4_scout_fp8], -) -@pytest.mark.parametrize("attn_backend", [TRITON_ATTN, FLASHINFER_ATTN]) -@pytest.mark.parametrize("n_layers", [4]) -@pytest.mark.parametrize("custom_ops", custom_ops_combos("quant_fp8", "rms_norm")) -@pytest.mark.parametrize("inductor_graph_partition", INDUCTOR_GRAPH_PARTITION) -def test_tp2_sp_ar_rms_fp8_fusions( - model_name: str, - matches_fn: Callable[[int], Matches], - model_kwargs: dict, - hf_overrides: Callable[[int], dict], - attn_backend: AttentionBackendCase, - n_layers: int, - custom_ops: str, - inductor_graph_partition: bool, - run_e2e_fusion_test, -): - matches = matches_fn(n_layers) - - # Reduce size of model and skip weight loading time - model_kwargs["hf_overrides"] = hf_overrides(n_layers) - model_kwargs["load_format"] = "dummy" - model_kwargs["max_model_len"] = 1024 - - compilation_config = dict( - use_inductor_graph_partition=inductor_graph_partition, - custom_ops=custom_ops.split(","), - pass_config=PassConfig( - fuse_norm_quant=True, - fuse_act_quant=False, - fuse_attn_quant=True, - enable_qk_norm_rope_fusion=True, - enable_sp=True, - fuse_gemm_comms=True, - fuse_allreduce_rms=True, - # Override threshold for testing (models have small hidden_size) - sp_min_token_num=512, - ), - ) - - matches_check = [ - "rms_quant_fusion", - "norm_rope_fusion", - "attn_quant_fusion", - "ar_rms_fusion", - "sequence_parallel", - "async_tp", - ] - - run_e2e_fusion_test( - model_name, - matches, - model_kwargs, - attn_backend, - compilation_config, - matches_check, - tp_size=2, - ) - - -@multi_gpu_test(num_gpus=2) -@pytest.mark.parametrize( - "model_name, matches_fn, model_kwargs, hf_overrides", - [llama3_8b, qwen3_a3b], -) -@pytest.mark.parametrize("attn_backend", [TRITON_ATTN]) -@pytest.mark.parametrize("n_layers", [4]) -@pytest.mark.parametrize("custom_ops", custom_ops_combos("rms_norm")) -@pytest.mark.parametrize("inductor_graph_partition", INDUCTOR_GRAPH_PARTITION) -def test_tp2_sp_ar_rms_fusions( - model_name: str, - matches_fn: Callable[[int], Matches], - model_kwargs: dict, - hf_overrides: Callable[[int], dict], - attn_backend: AttentionBackendCase, - n_layers: int, - custom_ops: str, - inductor_graph_partition: bool, - run_e2e_fusion_test, -): - matches = matches_fn(n_layers) - - # Reduce size of model and skip weight loading time - model_kwargs["hf_overrides"] = hf_overrides(n_layers) - model_kwargs["load_format"] = "dummy" - model_kwargs["max_model_len"] = 1024 - - compilation_config = dict( - use_inductor_graph_partition=inductor_graph_partition, - custom_ops=custom_ops.split(","), - pass_config=PassConfig( - enable_qk_norm_rope_fusion=True, - enable_sp=True, - fuse_gemm_comms=True, - fuse_allreduce_rms=True, - # Override threshold for testing (models have small hidden_size) - sp_min_token_num=512, - ), - ) - - matches_check = [ - "norm_rope_fusion", - "ar_rms_fusion", - "sequence_parallel", - "async_tp", - ] - - run_e2e_fusion_test( - model_name, - matches, - model_kwargs, - attn_backend, - compilation_config, - matches_check, - tp_size=2, - ) diff --git a/tests/compile/fusions_e2e/test_tp2_rocm_async_tp.py b/tests/compile/fusions_e2e/test_tp2_rocm_async_tp.py deleted file mode 100644 index 3143143bec..0000000000 --- a/tests/compile/fusions_e2e/test_tp2_rocm_async_tp.py +++ /dev/null @@ -1,58 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# SPDX-FileCopyrightText: Copyright contributors to the vLLM project -"""Bounded ROCm model coverage for actual sequence-parallel AsyncTP rewrites.""" - -import pytest - -from tests.utils import multi_gpu_test -from vllm.config import PassConfig -from vllm.platforms import current_platform - -from .common import INDUCTOR_GRAPH_PARTITION, custom_ops_combos -from .models import TRITON_ATTN, llama3_8b - - -@multi_gpu_test(num_gpus=2) -@pytest.mark.skipif(not current_platform.is_rocm(), reason="ROCm AsyncTP contract") -@pytest.mark.parametrize("inductor_graph_partition", INDUCTOR_GRAPH_PARTITION) -@pytest.mark.parametrize("custom_ops", tuple(custom_ops_combos("rms_norm"))) -@pytest.mark.parametrize("attn_backend", [TRITON_ATTN]) -def test_tp2_rocm_async_tp_bf16_fusions( - inductor_graph_partition, custom_ops, attn_backend, run_e2e_fusion_test -): - n_layers = 4 - model_kwargs = dict( - hf_overrides=dict( - num_hidden_layers=n_layers, - hidden_size=512, - intermediate_size=1024, - num_attention_heads=8, - num_key_value_heads=2, - head_dim=64, - ), - load_format="dummy", - max_model_len=1024, - kv_cache_memory_bytes=64 * 1024**2, - gpu_memory_utilization=0.01, - max_num_batched_tokens=1024, - max_num_seqs=32, - ) - run_e2e_fusion_test( - llama3_8b.model_name, - llama3_8b.matches(n_layers), - model_kwargs, - attn_backend, - dict( - use_inductor_graph_partition=inductor_graph_partition, - custom_ops=custom_ops.split(","), - pass_config=PassConfig( - enable_qk_norm_rope_fusion=True, - enable_sp=True, - fuse_gemm_comms=True, - fuse_allreduce_rms=False, - sp_min_token_num=512, - ), - ), - ["norm_rope_fusion", "sequence_parallel", "async_tp"], - tp_size=2, - ) diff --git a/tests/compile/fusions_e2e/test_tp2_rocm_moe.py b/tests/compile/fusions_e2e/test_tp2_rocm_moe.py deleted file mode 100644 index 9a228ca76c..0000000000 --- a/tests/compile/fusions_e2e/test_tp2_rocm_moe.py +++ /dev/null @@ -1,60 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# SPDX-FileCopyrightText: Copyright contributors to the vLLM project -"""ROCm MoE coverage for the TP2 GEMM/collective fusion contract.""" - -import pytest - -from tests.utils import multi_gpu_test -from vllm.config import PassConfig -from vllm.platforms import current_platform - -from .common import INDUCTOR_GRAPH_PARTITION -from .models import TRITON_ATTN, qwen3_a3b - - -@multi_gpu_test(num_gpus=2) -@pytest.mark.skipif(not current_platform.is_rocm(), reason="ROCm AsyncTP contract") -@pytest.mark.parametrize("inductor_graph_partition", INDUCTOR_GRAPH_PARTITION[:1]) -@pytest.mark.parametrize("attn_backend", [TRITON_ATTN]) -def test_tp2_rocm_moe_async_tp_bf16_fusions( - inductor_graph_partition, attn_backend, run_e2e_fusion_test -): - """MoE dense projections retain the Qwen SP/AsyncTP rewrite counts.""" - n_layers = 4 - run_e2e_fusion_test( - qwen3_a3b.model_name, - qwen3_a3b.matches(n_layers), - dict( - hf_overrides=dict( - num_hidden_layers=n_layers, - hidden_size=512, - intermediate_size=1024, - moe_intermediate_size=128, - num_attention_heads=8, - num_key_value_heads=2, - head_dim=64, - num_experts=8, - num_experts_per_tok=2, - ), - load_format="dummy", - max_model_len=1024, - kv_cache_memory_bytes=64 * 1024**2, - gpu_memory_utilization=0.01, - max_num_batched_tokens=1024, - max_num_seqs=32, - ), - attn_backend, - dict( - use_inductor_graph_partition=inductor_graph_partition, - custom_ops=["-rms_norm"], - pass_config=PassConfig( - enable_qk_norm_rope_fusion=True, - enable_sp=True, - fuse_gemm_comms=True, - fuse_allreduce_rms=False, - sp_min_token_num=512, - ), - ), - ["norm_rope_fusion", "sequence_parallel", "async_tp"], - tp_size=2, - ) diff --git a/tests/compile/passes/distributed/test_async_tp.py b/tests/compile/passes/distributed/test_async_tp.py deleted file mode 100644 index c0fd4c00a8..0000000000 --- a/tests/compile/passes/distributed/test_async_tp.py +++ /dev/null @@ -1,394 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# SPDX-FileCopyrightText: Copyright contributors to the vLLM project - - -import pytest -import torch - -import vllm.envs as envs -from tests.compile.backend import TestBackend -from tests.utils import ( - multi_gpu_test, -) -from vllm.compilation.passes.fusion.collective_fusion import AsyncTPPass -from vllm.config import ( - CompilationConfig, - DeviceConfig, - ModelConfig, - PassConfig, - VllmConfig, - set_current_vllm_config, -) -from vllm.config.utils import Range -from vllm.distributed import ( - tensor_model_parallel_all_gather, - tensor_model_parallel_reduce_scatter, -) -from vllm.distributed.parallel_state import ( - init_distributed_environment, - initialize_model_parallel, -) -from vllm.platforms import current_platform -from vllm.utils.network_utils import get_file_store_init_method -from vllm.utils.system_utils import update_environment_variables -from vllm.utils.torch_utils import set_random_seed - -DEVICE_TYPE = current_platform.device_type -FP8_DTYPE = current_platform.fp8_dtype() - -prompts = [ - "Hello, my name is", - "The president of the United States is", - "The capital of France is", - "The future of AI is", -] - - -class TestMMRSModel(torch.nn.Module): - def __init__(self, hidden_size=16, dtype=torch.bfloat16): - super().__init__() - self.hidden_size = hidden_size - self.dtype = dtype - self.gate_proj = torch.nn.Parameter( - torch.empty((self.hidden_size * 2, hidden_size)), requires_grad=False - ) - # Initialize weights - torch.nn.init.normal_(self.gate_proj, std=0.02) - - def forward(self, hidden_states): - """Forward pass implementing the mm + reduce scatter in the FX graph.""" - # Reshape input - view = hidden_states.reshape(-1, self.hidden_size) - - # matrix multiplication - permute = self.gate_proj.permute(1, 0) - mm = torch.mm(view, permute) - reduce_scatter = tensor_model_parallel_reduce_scatter(mm, dim=0) - return reduce_scatter - - def ops_in_model_before(self): - return [torch.ops.vllm.reduce_scatter.default] - - def ops_in_model_after(self): - return [torch.ops.symm_mem.fused_matmul_reduce_scatter.default] - - -class TestAGMMModel(torch.nn.Module): - def __init__(self, hidden_size=16, dtype=torch.bfloat16): - super().__init__() - self.hidden_size = hidden_size - self.dtype = dtype - self.weight = torch.nn.Parameter( - torch.empty((hidden_size, hidden_size)), requires_grad=False - ) - # Initialize weights - torch.nn.init.normal_(self.weight, std=0.02) - - def forward(self, hidden_states): - """Forward pass implementing the mm + all gather in the FX graph.""" - # Reshape input - view = hidden_states.reshape(-1, self.hidden_size) - all_gather = tensor_model_parallel_all_gather(view, dim=0) - permute = self.weight.permute(1, 0) - mm = torch.mm(all_gather, permute) - return mm - - def ops_in_model_before(self): - return [torch.ops.vllm.all_gather.default] - - def ops_in_model_after(self): - return [torch.ops.symm_mem.fused_all_gather_matmul.default] - - -class _BaseScaledMMModel(torch.nn.Module): - def __init__(self, hidden_size=16, dtype=torch.bfloat16): - super().__init__() - self.hidden_size = hidden_size - self.dtype = dtype - self.weight = ( - torch.empty([hidden_size, hidden_size], dtype=FP8_DTYPE) - .contiguous() - .transpose(0, 1) - ) - - # Initialize scale_b for _scaled_mm. - self.scale_b = torch.ones(1, self.hidden_size, dtype=torch.float32) - - -class TestScaledMMRSModel(_BaseScaledMMModel): - def forward(self, input: torch.Tensor): - """Forward pass implementing the scaled_mm + reduce scatter in the FX graph.""" - fp8_input = input.to(FP8_DTYPE) - scale_a = torch.ones(input.shape[0], 1, dtype=torch.float32) - scaled_mm = torch._scaled_mm( - fp8_input, - self.weight, - scale_a=scale_a, - scale_b=self.scale_b, - out_dtype=self.dtype, - ) - reduce_scatter = tensor_model_parallel_reduce_scatter(scaled_mm, dim=0) - return reduce_scatter - - def ops_in_model_before(self): - return [torch.ops.vllm.reduce_scatter.default] - - def ops_in_model_after(self): - return [torch.ops.vllm.patched_fused_scaled_matmul_reduce_scatter.default] - - -class TestAGScaledMMModel(_BaseScaledMMModel): - def forward(self, input: torch.Tensor): - """Forward pass implementing the all gather + scaled_mm in the FX graph.""" - # Reshape input - fp8_input = input.to(FP8_DTYPE) - all_gather = tensor_model_parallel_all_gather(fp8_input, dim=0) - - scale_a = torch.ones(all_gather.shape[0], 1, dtype=torch.float32) - scaled_mm = torch._scaled_mm( - all_gather, - self.weight, - scale_a=scale_a, - scale_b=self.scale_b, - out_dtype=self.dtype, - ) - return scaled_mm - - def ops_in_model_before(self): - return [torch.ops.vllm.all_gather.default] - - def ops_in_model_after(self): - return [torch.ops.symm_mem.fused_all_gather_scaled_matmul.default] - - -class TestCutlassScaledMMRSModel(_BaseScaledMMModel): - def forward(self, input: torch.Tensor): - """Forward pass implementing the cutlass_scaled_mm + reduce scatter - in the FX graph - - """ - fp8_input = input.to(FP8_DTYPE) - scale_a = torch.ones(input.shape[0], 1, dtype=torch.float32) - mm_out = torch.empty( - (fp8_input.shape[0], self.weight.shape[1]), - dtype=self.dtype, - device=input.device, - ) - torch.ops._C.cutlass_scaled_mm( - mm_out, fp8_input, self.weight, scale_a, self.scale_b, None - ) - reduce_scatter = tensor_model_parallel_reduce_scatter(mm_out, dim=0) - return reduce_scatter - - def ops_in_model_before(self): - return [torch.ops.vllm.reduce_scatter.default] - - def ops_in_model_after(self): - return [torch.ops.vllm.patched_fused_scaled_matmul_reduce_scatter.default] - - -class TestAGCutlassScaledMMModel(_BaseScaledMMModel): - def forward(self, input: torch.Tensor): - """Forward pass implementing the all gather + cutlass_scaled_mm - in the FX graph - """ - # Reshape input - fp8_input = input.to(FP8_DTYPE) - all_gather = tensor_model_parallel_all_gather(fp8_input, dim=0) - - scale_a = torch.ones(all_gather.shape[0], 1, dtype=torch.float32) - - mm_out = torch.empty( - (all_gather.shape[0], self.weight.shape[1]), - dtype=self.dtype, - device=all_gather.device, - ) - torch.ops._C.cutlass_scaled_mm( - mm_out, all_gather, self.weight, scale_a, self.scale_b, None - ) - return mm_out - - def ops_in_model_before(self): - return [torch.ops.vllm.all_gather.default] - - def ops_in_model_after(self): - return [torch.ops.symm_mem.fused_all_gather_scaled_matmul.default] - - -@multi_gpu_test(num_gpus=2) -@pytest.mark.parametrize( - "test_model", - [ - TestMMRSModel, - TestAGMMModel, - TestScaledMMRSModel, - TestAGScaledMMModel, - pytest.param( - TestCutlassScaledMMRSModel, - marks=pytest.mark.skipif( - not hasattr(torch.ops._C, "cutlass_scaled_mm"), - reason="Requires cutlass_scaled_mm", - ), - ), - pytest.param( - TestAGCutlassScaledMMModel, - marks=pytest.mark.skipif( - not hasattr(torch.ops._C, "cutlass_scaled_mm"), - reason="Requires cutlass_scaled_mm", - ), - ), - ], -) -@pytest.mark.parametrize("batch_size", [8]) -@pytest.mark.parametrize("seq_len", [16]) -@pytest.mark.parametrize("hidden_size", [16]) -@pytest.mark.parametrize("dtype", [torch.bfloat16]) -@pytest.mark.parametrize("dynamic", [True, False]) -@pytest.mark.skipif(envs.VLLM_TARGET_DEVICE not in ["cuda"], reason="Only test on CUDA") -def test_async_tp_pass_replace( - test_model: str, - batch_size: int, - seq_len: int, - hidden_size: int, - dtype: torch.dtype, - dynamic: bool, -): - num_processes = 2 - distributed_init_method = get_file_store_init_method() - - def run_torch_spawn(fn, nprocs): - # need to use torch.mp.spawn otherwise will have problems with - # torch.distributed and cuda - torch.multiprocessing.spawn( - fn, - args=( - num_processes, - test_model, - batch_size, - seq_len, - hidden_size, - dtype, - dynamic, - distributed_init_method, - ), - nprocs=nprocs, - ) - - run_torch_spawn(async_tp_pass_on_test_model, num_processes) - - -def test_async_tp_pass_requires_full_graph_compilation(): - vllm_config = VllmConfig() - vllm_config.compilation_config.use_inductor_graph_partition = False - vllm_config.compilation_config.splitting_ops = [ - "vllm::unified_attention_with_output" - ] - - async_tp_pass = object.__new__(AsyncTPPass) - async_tp_pass.compilation_config = vllm_config.compilation_config - - with pytest.raises( - AssertionError, match="AsyncTPPass requires full-graph compilation" - ): - async_tp_pass.is_applicable_for_range(Range(start=8, end=8)) - - -def async_tp_pass_on_test_model( - local_rank: int, - world_size: int, - test_model_cls: torch.nn.Module, - batch_size: int, - seq_len: int, - hidden_size: int, - dtype: torch.dtype, - dynamic: bool, - distributed_init_method: str, - check_numerics: bool = False, -): - set_random_seed(0) - - device = torch.device(f"{DEVICE_TYPE}:{local_rank}") - torch.accelerator.set_device_index(device) - torch.set_default_device(device) - torch.set_default_dtype(dtype) - - update_environment_variables( - { - "RANK": str(local_rank), - "LOCAL_RANK": str(local_rank), - "WORLD_SIZE": str(world_size), - } - ) - - # initialize distributed - init_distributed_environment( - world_size=world_size, - rank=local_rank, - distributed_init_method=distributed_init_method, - local_rank=local_rank, - ) - - # configure vllm config for SequenceParallelismPass - vllm_config = VllmConfig() - vllm_config.compilation_config = CompilationConfig( - pass_config=PassConfig( - fuse_gemm_comms=True, - ), - ) - vllm_config.device_config = DeviceConfig(device=torch.device(DEVICE_TYPE)) - - # this is a fake model name to construct the model config - # in the vllm_config, it's not really used. - model_name = "RedHatAI/Llama-3.2-1B-Instruct-FP8" - vllm_config.model_config = ModelConfig( - model=model_name, trust_remote_code=True, dtype=dtype, seed=42 - ) - - with set_current_vllm_config(vllm_config): - initialize_model_parallel(tensor_model_parallel_size=world_size) - - async_tp_pass = AsyncTPPass(vllm_config) - backend = TestBackend(async_tp_pass) - - assert ( - async_tp_pass.compilation_config.splitting_ops - == vllm_config.compilation_config.splitting_ops - ) - assert ( - async_tp_pass.compilation_config.use_inductor_graph_partition - == vllm_config.compilation_config.use_inductor_graph_partition - ) - - model = test_model_cls(hidden_size, dtype) # Pass dtype to model constructor - - hidden_states = torch.randn( - (batch_size * seq_len, hidden_size), dtype=dtype, requires_grad=False - ) - if check_numerics: - hidden_states += local_rank * 0.125 - - if dynamic: - torch._dynamo.mark_dynamic(hidden_states, 0) - - compiled_model = torch.compile(model, backend=backend) - actual = compiled_model(hidden_states) - - if check_numerics: - # These are new checks, so no existing tolerance is relaxed. - # Allow BF16 rounding vs. eager, but keep the lifetime check exact. - expected = model(hidden_states) - torch.testing.assert_close(actual, expected, rtol=1e-2, atol=1e-2) - snapshot = actual.clone() - changed = compiled_model(-hidden_states) - torch.testing.assert_close(changed, -expected, rtol=1e-2, atol=1e-2) - torch.testing.assert_close(actual, snapshot, rtol=0, atol=0) - - assert async_tp_pass.matched_count == 1 - - # In pre-nodes, all gather or reduce scatter should exist, - # fused_matmul_reduce_scatter or fused_all_gather_matmul should not - backend.check_before_ops(model.ops_in_model_before(), fully_replaced=False) - - # In post-nodes, fused_matmul_reduce_scatter or \ - # fused_all_gather_matmul should exist - backend.check_after_ops(model.ops_in_model_after()) diff --git a/tests/compile/passes/distributed/test_sequence_parallelism.py b/tests/compile/passes/distributed/test_sequence_parallelism.py deleted file mode 100644 index c40d75f675..0000000000 --- a/tests/compile/passes/distributed/test_sequence_parallelism.py +++ /dev/null @@ -1,343 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# SPDX-FileCopyrightText: Copyright contributors to the vLLM project - -import pytest -import torch - -import vllm.envs as envs -from tests.compile.backend import TestBackend -from tests.utils import TestFP8Layer, multi_gpu_test -from vllm.compilation.passes.fusion.rms_quant_fusion import RMSNormQuantFusionPass -from vllm.compilation.passes.fusion.sequence_parallelism import SequenceParallelismPass -from vllm.compilation.passes.utility.noop_elimination import NoOpEliminationPass -from vllm.compilation.passes.utility.post_cleanup import PostCleanupPass -from vllm.compilation.passes.vllm_inductor_pass import VllmInductorPass -from vllm.config import ( - CompilationConfig, - CUDAGraphMode, - DeviceConfig, - ModelConfig, - PassConfig, - VllmConfig, - get_current_vllm_config, - set_current_vllm_config, -) -from vllm.config.utils import Range -from vllm.distributed import tensor_model_parallel_all_reduce -from vllm.distributed.parallel_state import ( - init_distributed_environment, - initialize_model_parallel, -) -from vllm.model_executor.layers.layernorm import RMSNorm -from vllm.model_executor.layers.quantization.utils.quant_utils import ( - kFp8StaticTensorSym, -) -from vllm.platforms import current_platform -from vllm.utils.system_utils import update_environment_variables -from vllm.utils.torch_utils import set_random_seed - -DEVICE_TYPE = current_platform.device_type - -pytestmark = pytest.mark.skipif(not current_platform.is_cuda(), reason="Only test CUDA") - -FP8_DTYPE = current_platform.fp8_dtype() -prompts = [ - "Hello, my name is", - "The president of the United States is", - "The capital of France is", - "The future of AI is", -] - - -class TestAllReduceRMSNormModel(torch.nn.Module): - def __init__(self, hidden_size=16, eps=1e-6): - super().__init__() - self.hidden_size = hidden_size - self.eps = eps - self.norm = [RMSNorm(hidden_size, eps) for i in range(4)] - self.w = [torch.rand(hidden_size, hidden_size) for _ in range(3)] - - def forward(self, x): - z = torch.relu(x) - x = resid = tensor_model_parallel_all_reduce(z) - y = self.norm[0](x) - - z2 = torch.mm(y, self.w[0]) - x2 = tensor_model_parallel_all_reduce(z2) - - y2, resid = self.norm[1](x2, resid) - - z3 = torch.mm(y2, self.w[1]) - x3 = tensor_model_parallel_all_reduce(z3) - - y3, resid = self.norm[2](x3, resid) - - z4 = torch.mm(y3, self.w[2]) - x4 = tensor_model_parallel_all_reduce(z4) - - y4, resid = self.norm[3](x4, resid) - return y4 - - def ops_in_model_before(self): - return [torch.ops.vllm.all_reduce.default] - - def ops_in_model_after(self): - return [ - torch.ops.vllm.all_gather.default, - torch.ops.vllm.reduce_scatter.default, - ] - - def ops_in_model(self): - return [ - torch.ops.vllm_ir.rms_norm, - torch.ops.vllm_ir.fused_add_rms_norm, - ] - - -class TestAllReduceRMSNormStaticQuantFP8Model(torch.nn.Module): - quant_key = kFp8StaticTensorSym - - def __init__(self, hidden_size=16, eps=1e-6): - super().__init__() - self.vllm_config = get_current_vllm_config() - self.hidden_size = hidden_size - self.eps = eps - self.norm = [RMSNorm(hidden_size, eps) for i in range(4)] - self.fp8_linear_layers = [ - TestFP8Layer( - weight_shape=(hidden_size, hidden_size), - activation_quant_key=self.quant_key, - weight_quant_key=self.quant_key, - input_dtype=self.vllm_config.model_config.dtype, - ) - for i in range(3) - ] - - def forward(self, hidden_states): - # avoid having graph input be an arg to a pattern directly - z = torch.relu(hidden_states) - x = resid = tensor_model_parallel_all_reduce(z) - y = self.norm[0](x) - - z2 = self.fp8_linear_layers[0](y) - - x2 = tensor_model_parallel_all_reduce(z2) - y2, resid = self.norm[1](x2, resid) - - z3 = self.fp8_linear_layers[1](y2) - - x3 = tensor_model_parallel_all_reduce(z3) - y3, resid = self.norm[2](x3, resid) # use resid here - - z4 = self.fp8_linear_layers[2](y3) - x4 = tensor_model_parallel_all_reduce(z4) - y4, resid = self.norm[3](x4, resid) # use resid here - return y4 - - def ops_in_model_after(self): - return [ - torch.ops.vllm.all_gather.default, - torch.ops.vllm.reduce_scatter.default, - ] - - def ops_in_model_before(self): - return [ - torch.ops.vllm.all_reduce.default, - ] - - def ops_in_model(self): - if self.vllm_config.compilation_config.pass_config.fuse_norm_quant: - return [torch.ops._C.fused_add_rms_norm_static_fp8_quant.default] - else: - quant_ops = ( - [torch.ops._C.static_scaled_fp8_quant.default] - if any(layer.is_quant_fp8_enabled() for layer in self.fp8_linear_layers) - else [torch.ops.aten.reciprocal] - ) - return [ - torch.ops.vllm_ir.rms_norm, - torch.ops.vllm_ir.fused_add_rms_norm, - *quant_ops, - ] - - -@multi_gpu_test(num_gpus=2) -@pytest.mark.parametrize( - "test_model_cls, custom_ops", - [ - (TestAllReduceRMSNormModel, "+rms_norm"), - (TestAllReduceRMSNormModel, "-rms_norm"), - (TestAllReduceRMSNormStaticQuantFP8Model, "+rms_norm,+quant_fp8"), - (TestAllReduceRMSNormStaticQuantFP8Model, "+rms_norm,-quant_fp8"), - (TestAllReduceRMSNormStaticQuantFP8Model, "-rms_norm,+quant_fp8"), - (TestAllReduceRMSNormStaticQuantFP8Model, "-rms_norm,-quant_fp8"), - ], -) -@pytest.mark.parametrize("batch_size", [8]) -@pytest.mark.parametrize("seq_len", [16]) -@pytest.mark.parametrize("hidden_size", [16]) -@pytest.mark.parametrize("dtype", [torch.float16, torch.bfloat16]) -@pytest.mark.parametrize("fuse_norm_quant", [True, False]) -@pytest.mark.parametrize("dynamic", [False, True]) -@pytest.mark.skipif(envs.VLLM_TARGET_DEVICE not in ["cuda"], reason="Only test on CUDA") -def test_sequence_parallelism_pass( - test_model_cls: type[torch.nn.Module], - custom_ops: str, - batch_size: int, - seq_len: int, - hidden_size: int, - dtype: torch.dtype, - fuse_norm_quant: bool, - dynamic: bool, -): - num_processes = 2 - - def run_torch_spawn(fn, nprocs): - # need to use torch.mp.spawn otherwise will have problems with - # torch.distributed and cuda - torch.multiprocessing.spawn( - fn, - args=( - num_processes, - test_model_cls, - custom_ops, - batch_size, - seq_len, - hidden_size, - dtype, - fuse_norm_quant, - dynamic, - ), - nprocs=nprocs, - ) - - run_torch_spawn(sequence_parallelism_pass_on_test_model, num_processes) - - -def test_sequence_parallelism_pass_requires_full_graph_compilation(): - vllm_config = VllmConfig() - vllm_config.compilation_config.use_inductor_graph_partition = False - vllm_config.compilation_config.splitting_ops = [ - "vllm::unified_attention_with_output" - ] - - sequence_parallelism_pass = object.__new__(SequenceParallelismPass) - sequence_parallelism_pass.compilation_config = vllm_config.compilation_config - sequence_parallelism_pass.min_token_num = 1 - - with pytest.raises( - AssertionError, - match="SequenceParallelismPass requires full-graph compilation", - ): - sequence_parallelism_pass.is_applicable_for_range(Range(start=8, end=8)) - - -def sequence_parallelism_pass_on_test_model( - local_rank: int, - world_size: int, - test_model_cls: type[torch.nn.Module], - custom_ops: str, - batch_size: int, - seq_len: int, - hidden_size: int, - dtype: torch.dtype, - fuse_norm_quant: bool, - dynamic: bool, -): - set_random_seed(0) - - device = torch.device(f"{DEVICE_TYPE}:{local_rank}") - torch.accelerator.set_device_index(device) - torch.set_default_device(device) - torch.set_default_dtype(dtype) - - update_environment_variables( - { - "RANK": str(local_rank), - "LOCAL_RANK": str(local_rank), - "WORLD_SIZE": str(world_size), - "MASTER_ADDR": "localhost", - "MASTER_PORT": "12345", - } - ) - - # initialize distributed - init_distributed_environment() - - # configure vllm config for SequenceParallelismPass - custom_ops_list = custom_ops.split(",") if custom_ops else [] - compilation_config = CompilationConfig( - splitting_ops=[], # avoid automatic rms_norm enablement - cudagraph_mode=CUDAGraphMode.NONE, # avoid piecewise warnings - custom_ops=custom_ops_list, - pass_config=PassConfig( - enable_sp=True, - fuse_norm_quant=fuse_norm_quant, - eliminate_noops=True, - ), - ) # NoOp needed for fusion - device_config = DeviceConfig(device=torch.device(DEVICE_TYPE)) - - # this is a fake model name to construct the model config - # in the vllm_config, it's not really used. - model_name = "RedHatAI/Llama-3.2-1B-Instruct-FP8" - model_config = ModelConfig( - model=model_name, trust_remote_code=True, dtype=dtype, seed=42 - ) - - vllm_config = VllmConfig( - model_config=model_config, - device_config=device_config, - compilation_config=compilation_config, - ) - - with set_current_vllm_config(vllm_config): - initialize_model_parallel(tensor_model_parallel_size=world_size) - noop_pass = NoOpEliminationPass(vllm_config) - sequence_parallelism_pass = SequenceParallelismPass(vllm_config) - cleanup_pass = PostCleanupPass(vllm_config) - assert ( - sequence_parallelism_pass.compilation_config.splitting_ops - == vllm_config.compilation_config.splitting_ops - ) - assert ( - sequence_parallelism_pass.compilation_config.use_inductor_graph_partition - == vllm_config.compilation_config.use_inductor_graph_partition - ) - passes_for_backend: list[VllmInductorPass] = [ - noop_pass, - sequence_parallelism_pass, - ] - - if fuse_norm_quant: - fusion_pass = RMSNormQuantFusionPass(vllm_config) - passes_for_backend.append(fusion_pass) - - passes_for_backend.append(cleanup_pass) - - backend = TestBackend(*passes_for_backend) - - model = test_model_cls(hidden_size) - - hidden_states = torch.randn((batch_size * seq_len, hidden_size), dtype=dtype) - - if dynamic: - torch._dynamo.mark_dynamic(hidden_states, 0) - - compiled_model = torch.compile(model, backend=backend) - compiled_model(hidden_states) - - assert sequence_parallelism_pass.matched_count == 4 - - # In pre-nodes, all reduce should be there, - # reduce scatter and all gather should not - for op in model.ops_in_model_before(): - assert backend.op_count(op, before=True) == 4 - - # In post-nodes, reduce scatter and all gather should be there, - # all reduce should not - for op in model.ops_in_model_after(): - assert backend.op_count(op, before=False) == 4 - - for op in model.ops_in_model(): - assert backend.op_count(op, before=False) > 0 diff --git a/tests/compile/passes/test_fuse_mla_dual_rms_norm.py b/tests/compile/passes/test_fuse_mla_dual_rms_norm.py index 0f631b9472..75a0eee469 100644 --- a/tests/compile/passes/test_fuse_mla_dual_rms_norm.py +++ b/tests/compile/passes/test_fuse_mla_dual_rms_norm.py @@ -285,3 +285,134 @@ def test_fuse_mla_dual_rms_norm_fp8_per_token( backend.check_before_ops(model.ops_in_model_before()) backend.check_after_ops(model.ops_in_model_after()) + + +class MLADualRMSNormFp8GroupTestModel(torch.nn.Module): + """Minimal model reproducing the FP8 MLA attention path with *group* quant + (block-scaled fp8 weights, 128-wide activation groups): + + linear -> split([q_dim, kv_dim]) + +-- q_c (getitem 0) -> rocm_aiter_rmsnorm_fp8_group_quant -> dequant + +-- kv_lora (getitem 1) -> split([kv_c_dim, k_pe_dim]) + +-- kv_c (getitem 0) -> rms_norm (bf16) + +-- k_pe + """ + + GROUP_SIZE = 128 + + def __init__( + self, + hidden_size: int, + q_dim: int = Q_DIM, + kv_c_dim: int = KV_C_DIM, + k_pe_dim: int = K_PE_DIM, + eps: float = EPS, + ): + super().__init__() + self.q_dim = q_dim + self.kv_dim = kv_c_dim + k_pe_dim + self.kv_c_dim = kv_c_dim + self.k_pe_dim = k_pe_dim + self.eps = eps + self.proj = torch.nn.Linear(hidden_size, q_dim + self.kv_dim, bias=False) + self.q_weight = torch.nn.Parameter(torch.ones(q_dim)) + self.kv_norm = RMSNorm(kv_c_dim, eps=eps) + + def _dequant(self, x_fp8: torch.Tensor, scale: torch.Tensor) -> torch.Tensor: + # Group: one (M, N / 128) scale per 128-column group of the row. + m = x_fp8.shape[0] + x = x_fp8.to(torch.float32).view(m, -1, self.GROUP_SIZE) + return (x * scale.view(m, -1, 1)).view(m, -1).to(torch.bfloat16) + + def forward(self, x: torch.Tensor): + # Avoid graph input being a direct arg to a matched pattern node + x = torch.relu(x) + projected = self.proj(x) + q_c, kv_lora = projected.split([self.q_dim, self.kv_dim], dim=-1) + kv_c, k_pe = kv_lora.split([self.kv_c_dim, self.k_pe_dim], dim=-1) + q_fp8, q_scale = torch.ops.vllm.rocm_aiter_rmsnorm_fp8_group_quant( + q_c, self.q_weight, self.eps, self.GROUP_SIZE + ) + kv_normed = self.kv_norm(kv_c) + # Also return the group scales so the test can check them directly: + # equal scales pin any fp8 payload difference to rounding alone. + return self._dequant(q_fp8, q_scale), q_scale, kv_normed, k_pe + + def ops_in_model_before(self): + return [ + torch.ops.vllm.rocm_aiter_rmsnorm_fp8_group_quant.default, + torch.ops.vllm_ir.rms_norm.default, + ] + + def ops_in_model_after(self): + return [torch.ops.vllm.fused_mla_dual_rms_norm_group_quant.default] + + +@pytest.mark.parametrize("dtype", [torch.bfloat16]) +@pytest.mark.parametrize("hidden_size", [7168]) +@pytest.mark.skipif( + not is_aiter_found_and_supported(), + reason="Only test on ROCm with AITER installed and supported", +) +def test_fuse_mla_dual_rms_norm_fp8_group( + dtype: torch.dtype, + hidden_size: int, + monkeypatch: pytest.MonkeyPatch, +): + torch._dynamo.reset() + vllm_config = VllmConfig( + model_config=ModelConfig(dtype=dtype), + compilation_config=CompilationConfig( + mode=CompilationMode.VLLM_COMPILE, + custom_ops=["+rms_norm"], + pass_config=PassConfig( + fuse_mla_dual_rms_norm=True, + eliminate_noops=True, + ), + ), + ) + with vllm.config.set_current_vllm_config(vllm_config), monkeypatch.context() as m: + from vllm.compilation.passes.fusion.rocm_aiter_fusion import ( + MLADualRMSNormFusionPass, + ) + + torch.set_default_device("cuda") + torch.set_default_dtype(dtype) + torch.manual_seed(42) + m.setenv("VLLM_ROCM_USE_AITER", "1") + rocm_aiter_ops.refresh_env_variables() + + fusion_pass = MLADualRMSNormFusionPass(vllm_config) + passes = [ + NoOpEliminationPass(vllm_config), + fusion_pass, + PostCleanupPass(vllm_config), + ] + backend = TestBackend(*passes) + model = MLADualRMSNormFp8GroupTestModel(hidden_size) + + x = torch.randn(64, hidden_size) + torch._dynamo.mark_dynamic(x, 0) + + with torch.inference_mode(): + outputs_unfused = model(x) + model_fused = torch.compile(model, backend=backend) + outputs_fused = model_fused(x) + + q_deq_u, q_scale_u, kv_normed_u, k_pe_u = outputs_unfused + q_deq_f, q_scale_f, kv_normed_f, k_pe_f = outputs_fused + + torch.testing.assert_close(k_pe_u, k_pe_f, atol=0, rtol=0) + torch.testing.assert_close(kv_normed_u, kv_normed_f, atol=1e-2, rtol=1e-2) + torch.testing.assert_close(q_scale_u, q_scale_f, atol=0, rtol=1e-5) + + E4M3_STEP = 0.125 + exact_frac = (q_deq_u == q_deq_f).float().mean().item() + assert exact_frac > 0.99, f"q: only {exact_frac:.4f} of elements bit-exact" + torch.testing.assert_close(q_deq_u, q_deq_f, atol=1e-2, rtol=E4M3_STEP) + + assert fusion_pass.matched_count == 1, ( + f"Expected 1 fused pair, got {fusion_pass.matched_count}" + ) + backend.check_before_ops(model.ops_in_model_before()) + backend.check_after_ops(model.ops_in_model_after()) diff --git a/tests/compile/rocm/test_async_tp.py b/tests/compile/rocm/test_async_tp.py deleted file mode 100644 index a4462265f8..0000000000 --- a/tests/compile/rocm/test_async_tp.py +++ /dev/null @@ -1,75 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# SPDX-FileCopyrightText: Copyright contributors to the vLLM project -"""ROCm AsyncTP numerical coverage kept outside the shared distributed suite.""" - -import pytest -import torch - -from tests.compile.passes.distributed.test_async_tp import ( - TestAGMMModel, - TestMMRSModel, - async_tp_pass_on_test_model, -) -from tests.utils import multi_gpu_test -from vllm.distributed import ( - tensor_model_parallel_all_gather, - tensor_model_parallel_reduce_scatter, -) -from vllm.platforms import current_platform -from vllm.utils.network_utils import get_file_store_init_method - - -class TestRocmMMRSModel(TestMMRSModel): - def forward(self, hidden_states): - view = hidden_states.reshape(-1, self.hidden_size) - mm = torch.ops.vllm.rocm_unquantized_gemm(view, self.gate_proj, None) - return tensor_model_parallel_reduce_scatter(mm, dim=0) - - def ops_in_model_before(self): - return super().ops_in_model_before() + [ - torch.ops.vllm.rocm_unquantized_gemm.default - ] - - -class TestRocmAGMMModel(TestAGMMModel): - def __init__(self, hidden_size=16, dtype=torch.bfloat16): - super().__init__(hidden_size, dtype) - self.weight = torch.nn.Parameter( - torch.empty((hidden_size * 2, hidden_size)), requires_grad=False - ) - torch.nn.init.normal_(self.weight, std=0.02) - - def forward(self, hidden_states): - view = hidden_states.reshape(-1, self.hidden_size) - gathered = tensor_model_parallel_all_gather(view, dim=0) - return torch.ops.vllm.rocm_unquantized_gemm(gathered, self.weight, None) - - def ops_in_model_before(self): - return super().ops_in_model_before() + [ - torch.ops.vllm.rocm_unquantized_gemm.default - ] - - -@multi_gpu_test(num_gpus=2) -@pytest.mark.skipif(not current_platform.is_rocm(), reason="ROCm backend contract") -@pytest.mark.parametrize( - "test_model", [TestMMRSModel, TestAGMMModel, TestRocmMMRSModel, TestRocmAGMMModel] -) -@pytest.mark.parametrize("dynamic", [False, True]) -def test_rocm_async_tp_bf16_rewrite_and_correctness(test_model, dynamic): - """Real symmetric-memory AG/GEMM and GEMM/RS must replace the collectives.""" - torch.multiprocessing.spawn( - async_tp_pass_on_test_model, - args=( - 2, - test_model, - 8, - 16, - 128, - torch.bfloat16, - dynamic, - get_file_store_init_method(), - True, - ), - nprocs=2, - ) diff --git a/tests/compile/test_config.py b/tests/compile/test_config.py index 24cb816b5d..4e7a756f8c 100644 --- a/tests/compile/test_config.py +++ b/tests/compile/test_config.py @@ -464,49 +464,41 @@ def test_should_split(): ( "cudagraph_capture_sizes", "max_cudagraph_capture_size", - "tp_size", - "enable_sp", "max_num_batched_tokens", "cudagraph_mode", "expected_max_size", ), [ - (None, None, 1, False, 2048, CUDAGraphMode.FULL_AND_PIECEWISE, 256), - ([1, 2, 4], 4, 1, False, 2048, CUDAGraphMode.FULL_AND_PIECEWISE, 4), + (None, None, 2048, CUDAGraphMode.FULL_AND_PIECEWISE, 256), + ([1, 2, 4], 4, 2048, CUDAGraphMode.FULL_AND_PIECEWISE, 4), ( [1, 2, 4], 8, - 1, - False, 2048, CUDAGraphMode.FULL_AND_PIECEWISE, ValidationError, ), - ([1, 256], None, 1, False, 2048, CUDAGraphMode.FULL_AND_PIECEWISE, 256), - ([], None, 1, False, 2048, CUDAGraphMode.NONE, 0), - (None, 0, 1, False, 2048, CUDAGraphMode.NONE, 0), + ([1, 256], None, 2048, CUDAGraphMode.FULL_AND_PIECEWISE, 256), + ([], None, 2048, CUDAGraphMode.NONE, 0), + (None, 0, 2048, CUDAGraphMode.NONE, 0), # truncated to nearest multiple of 8 or 16 - (None, 257, 1, False, 2048, CUDAGraphMode.FULL_AND_PIECEWISE, 256), + (None, 257, 2048, CUDAGraphMode.FULL_AND_PIECEWISE, 256), # max_num_batched_tokens <= max_cudagraph_capture_size should always be # captured even if not landing on a 16-stride step - (None, 2048, 1, False, 257, CUDAGraphMode.FULL_AND_PIECEWISE, 257), + (None, 2048, 257, CUDAGraphMode.FULL_AND_PIECEWISE, 257), # max from list - ([1, 2, 4, 15], None, 1, False, 2048, CUDAGraphMode.FULL_AND_PIECEWISE, 15), - # SP forces full-graph compilation, sizes are filtered by TP - ([1, 2, 4, 15], None, 2, True, 2048, CUDAGraphMode.FULL_AND_PIECEWISE, 4), + ([1, 2, 4, 15], None, 2048, CUDAGraphMode.FULL_AND_PIECEWISE, 15), # limited by the max_tokens - ([1, 2, 4, 15], None, 1, False, 8, CUDAGraphMode.FULL_AND_PIECEWISE, 4), + ([1, 2, 4, 15], None, 8, CUDAGraphMode.FULL_AND_PIECEWISE, 4), # the list should contain at least 1 element when use cudagraph - ([], None, 1, False, 2048, CUDAGraphMode.FULL_AND_PIECEWISE, ValidationError), + ([], None, 2048, CUDAGraphMode.FULL_AND_PIECEWISE, ValidationError), # the max capturing size should be >= 1 when use cudagraph - (None, 0, 1, False, 2048, CUDAGraphMode.FULL_AND_PIECEWISE, ValidationError), + (None, 0, 2048, CUDAGraphMode.FULL_AND_PIECEWISE, ValidationError), ], ) def test_cudagraph_sizes_post_init( cudagraph_capture_sizes, max_cudagraph_capture_size, - tp_size, - enable_sp, max_num_batched_tokens, cudagraph_mode, expected_max_size, @@ -515,10 +507,7 @@ def test_cudagraph_sizes_post_init( if expected_max_size == ValidationError: ctx = pytest.raises(expected_max_size) - with ( - ctx, - patch.object(current_platform, "device_count", return_value=tp_size), - ): + with ctx: kwargs = {} if cudagraph_capture_sizes is not None: kwargs["cudagraph_capture_sizes"] = cudagraph_capture_sizes @@ -526,18 +515,15 @@ def test_cudagraph_sizes_post_init( kwargs["max_cudagraph_capture_size"] = max_cudagraph_capture_size compilation_config = CompilationConfig( pass_config=PassConfig( - enable_sp=enable_sp, fuse_norm_quant=True, fuse_act_quant=True, eliminate_noops=True, - sp_min_token_num=512 if enable_sp else None, ), cudagraph_mode=cudagraph_mode, **kwargs, ) engine_args = EngineArgs( model="facebook/opt-125m", - tensor_parallel_size=tp_size, max_num_seqs=min(max_num_batched_tokens, 128), max_num_batched_tokens=max_num_batched_tokens, compilation_config=compilation_config, @@ -728,35 +714,6 @@ def test_default_cudagraph_capture_sizes_keep_all_sizes_bounded(): ) -def test_cudagraph_capture_sizes_respect_sequence_parallelism(): - """Sequence-parallel capture sizes stay divisible by tensor parallel size.""" - compilation_config = CompilationConfig( - cudagraph_mode=CUDAGraphMode.FULL_AND_PIECEWISE - ) - compilation_config.pass_config.enable_sp = True - config = _mock_config_for_cudagraph_sizes( - max_num_seqs=32, - num_speculative_tokens=16, - max_num_batched_tokens=32768, - compilation_config=compilation_config, - ) - config.parallel_config = SimpleNamespace(tensor_parallel_size=2) - config.update_sizes_for_sequence_parallelism = lambda sizes: ( - VllmConfig.update_sizes_for_sequence_parallelism(config, sizes) - ) - - with patch.object( - current_platform, - "is_device_capability_family", - return_value=False, - ): - VllmConfig._set_cudagraph_sizes(config) - - assert all(size % 2 == 0 for size in compilation_config.cudagraph_capture_sizes) - assert 544 not in compilation_config.cudagraph_capture_sizes - assert compilation_config.max_cudagraph_capture_size == 512 - - @pytest.mark.skipif(not current_platform.is_cuda(), reason="Only test CUDA") def test_real_vllm_config_caps_widest_ngram_decode_batch(): """Real config post-init preserves the default capture ceiling.""" @@ -1046,123 +1003,6 @@ def test_blackwell_cudagraph_default(is_blackwell, expected_max_size): ) -@pytest.mark.skipif( - not current_platform.support_static_graph_mode(), - reason="Skip if not cudagraph mode supported", -) -@pytest.mark.parametrize( - ( - "cudagraph_mode", - "use_inductor_graph_partition", - "expected_enable_sp", - "expected_cudagraph_mode", - "expected_piecewise_compile", - "expected_capture_sizes", - "expected_max_size", - ), - [ - (CUDAGraphMode.PIECEWISE, False, True, CUDAGraphMode.FULL, False, [2, 4], 4), - ( - CUDAGraphMode.FULL_DECODE_ONLY, - False, - True, - CUDAGraphMode.FULL_DECODE_ONLY, - False, - [2, 4], - 4, - ), - ( - CUDAGraphMode.FULL_AND_PIECEWISE, - False, - True, - CUDAGraphMode.FULL, - False, - [2, 4], - 4, - ), - ( - CUDAGraphMode.FULL_AND_PIECEWISE, - True, - True, - CUDAGraphMode.FULL_AND_PIECEWISE, - True, - [2, 4], - 4, - ), - ], -) -def test_sequence_parallelism_requires_full_graph_compilation( - cudagraph_mode: CUDAGraphMode, - use_inductor_graph_partition: bool, - expected_enable_sp: bool, - expected_cudagraph_mode: CUDAGraphMode, - expected_piecewise_compile: bool, - expected_capture_sizes: list[int], - expected_max_size: int, -): - with patch.object(current_platform, "device_count", return_value=2): - vllm_config = VllmConfig( - parallel_config=ParallelConfig(tensor_parallel_size=2), - scheduler_config=SchedulerConfig( - max_num_seqs=128, - max_num_batched_tokens=2048, - max_model_len=2048, - is_encoder_decoder=False, - ), - ) - vllm_config.model_config = MagicMock( - dtype=torch.float16, - enforce_eager=False, - is_moe=False, - disable_cascade_attn=False, - get_hidden_size=MagicMock(return_value=4096), - ) - vllm_config.compilation_config = CompilationConfig( - mode=CompilationMode.VLLM_COMPILE, - cudagraph_capture_sizes=[1, 2, 4, 15], - max_cudagraph_capture_size=None, - compile_sizes=["cudagraph_capture_sizes"], - use_inductor_graph_partition=use_inductor_graph_partition, - pass_config=PassConfig( - enable_sp=True, - fuse_gemm_comms=True, - fuse_norm_quant=True, - fuse_act_quant=True, - eliminate_noops=True, - sp_min_token_num=512, - ), - cudagraph_mode=cudagraph_mode, - ) - vllm_config.compilation_config.set_splitting_ops_for_v1( - all2all_backend=vllm_config.parallel_config.all2all_backend, - data_parallel_size=1, - ) - vllm_config._set_compile_ranges() - vllm_config._set_cudagraph_sizes() - - assert ( - vllm_config.compilation_config.use_inductor_graph_partition - == use_inductor_graph_partition - ) - assert ( - bool(vllm_config.compilation_config.splitting_ops) == expected_piecewise_compile - ) - assert vllm_config.compilation_config.pass_config.enable_sp == expected_enable_sp - assert ( - vllm_config.compilation_config.pass_config.fuse_gemm_comms == expected_enable_sp - ) - assert vllm_config.compilation_config.cudagraph_mode == expected_cudagraph_mode - assert ( - vllm_config.compilation_config.cudagraph_capture_sizes == expected_capture_sizes - ) - assert ( - vllm_config.compilation_config.max_cudagraph_capture_size == expected_max_size - ) - assert ( - 511 in vllm_config.compilation_config.compile_ranges_endpoints - ) == expected_enable_sp - - def test_cached_compilation_config(default_vllm_config): import torch from torch._inductor.utils import run_and_get_code @@ -1366,3 +1206,70 @@ def test_inductor_asserts_user_override(monkeypatch): assert config.inductor_compile_config.get("size_asserts") is True if not _is_torch_equal_or_newer(torch.__version__, "2.12.0.dev"): assert config.inductor_compile_config.get("alignment_asserts") is False + + +@pytest.mark.parametrize("deterministic", [False, True]) +@pytest.mark.parametrize("override", [None, False, True]) +def test_combo_kernel_benchmarking_respects_deterministic(deterministic, override): + from torch._inductor import config as inductor_config + + overrides = {} if override is None else {"deterministic": override} + with ( + inductor_config.patch(deterministic=deterministic), + patch("vllm.config.compilation.current_platform.is_cpu", return_value=False), + ): + config = CompilationConfig(inductor_compile_config=overrides) + + assert config.inductor_compile_config["combo_kernels"] is True + effective_deterministic = deterministic if override is None else override + assert config.inductor_compile_config["benchmark_combo_kernel"] is ( + not effective_deterministic + ) + + +@pytest.mark.parametrize("deterministic", [False, True]) +@pytest.mark.parametrize( + "overrides", + [ + {"combo_kernels": False}, + {"combo_kernels": True}, + {"benchmark_combo_kernel": False}, + {"benchmark_combo_kernel": True}, + ], +) +def test_combo_kernel_explicit_settings_preserved(deterministic, overrides): + from torch._inductor import config as inductor_config + + with ( + inductor_config.patch(deterministic=deterministic), + patch("vllm.config.compilation.current_platform.is_cpu", return_value=False), + ): + config = CompilationConfig(inductor_compile_config=overrides.copy()) + + for key in ("combo_kernels", "benchmark_combo_kernel"): + assert config.inductor_compile_config.get(key) == overrides.get(key) + + +@pytest.mark.parametrize("is_cpu,torch_version", [(True, "2.13.0"), (False, "2.8.0")]) +def test_combo_kernel_defaults_require_supported_platform_and_torch( + is_cpu, torch_version +): + with ( + patch("vllm.config.compilation.current_platform.is_cpu", return_value=is_cpu), + patch("torch.__version__", torch_version), + ): + config = CompilationConfig() + + assert "combo_kernels" not in config.inductor_compile_config + assert "benchmark_combo_kernel" not in config.inductor_compile_config + + +def test_combo_kernel_defaults_without_inductor_deterministic_setting(): + with ( + patch("torch._inductor.config", SimpleNamespace()), + patch("vllm.config.compilation.current_platform.is_cpu", return_value=False), + ): + config = CompilationConfig() + + assert config.inductor_compile_config["combo_kernels"] is True + assert config.inductor_compile_config["benchmark_combo_kernel"] is True diff --git a/tests/compile/test_sequence_parallelism_threshold.py b/tests/compile/test_sequence_parallelism_threshold.py deleted file mode 100644 index 090b77b330..0000000000 --- a/tests/compile/test_sequence_parallelism_threshold.py +++ /dev/null @@ -1,192 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# SPDX-FileCopyrightText: Copyright contributors to the vLLM project - -import pytest - -from vllm.compilation.passes.fusion.sequence_parallelism import ( - SP_MIN_HIDDEN_SIZE, - SP_MIN_PER_GPU_SIZE_MB, - get_sequence_parallelism_threshold, -) - - -class TestGetSequenceParallelismThreshold: - """Tests for get_sequence_parallelism_threshold function.""" - - def test_non_cuda_returns_none(self, mock_cuda_platform): - """Non-CUDA platforms should return None.""" - with mock_cuda_platform(is_cuda=False): - result = get_sequence_parallelism_threshold( - hidden_size=8192, tp_size=2, element_size=2 - ) - assert result is None - - def test_unsupported_device_capability_returns_none(self, mock_cuda_platform): - """Unsupported device capabilities (e.g., sm80) should return None.""" - with mock_cuda_platform(capability=(8, 0)): - result = get_sequence_parallelism_threshold( - hidden_size=8192, tp_size=2, element_size=2 - ) - assert result is None - - def test_small_hidden_size_returns_none(self, mock_cuda_platform): - """H100 with hidden_size below threshold should return None.""" - with mock_cuda_platform(capability=(9, 0)): - result = get_sequence_parallelism_threshold( - hidden_size=4096, - tp_size=2, - element_size=2, # 4096 < 8192 - ) - assert result is None - - def test_h100_large_model_returns_threshold(self, mock_cuda_platform): - """H100 with large enough hidden_size should return calculated threshold.""" - with mock_cuda_platform(capability=(9, 0)): - hidden_size = 8192 - tp_size = 2 - element_size = 2 # float16/bfloat16 - - result = get_sequence_parallelism_threshold( - hidden_size=hidden_size, - tp_size=tp_size, - element_size=element_size, - ) - - # Verify calculation: (8 * 2 * 1024 * 1024) // (8192 * 2) = 1024 - MiB = 1024 * 1024 - expected = int( - (SP_MIN_PER_GPU_SIZE_MB[90] * tp_size * MiB) - // (hidden_size * element_size) - ) - assert result == expected - assert result == 1024 - - @pytest.mark.parametrize( - "hidden_size,tp_size,element_size,expected", - [ - # Boundary: exactly at min hidden size threshold, tp_size=1 - # (8 * 1 * 1024 * 1024) // (8192 * 2) = 512 - (8192, 1, 2, 512), - # Larger hidden size reduces token threshold - # (8 * 1 * 1024 * 1024) // (16384 * 2) = 256 - (16384, 1, 2, 256), - # Larger tp_size increases token threshold - # (8 * 4 * 1024 * 1024) // (8192 * 2) = 2048 - (8192, 4, 2, 2048), - # Larger element_size (fp32) reduces token threshold - # (8 * 2 * 1024 * 1024) // (8192 * 4) = 512 - (8192, 2, 4, 512), - ], - ) - def test_threshold_calculation_variations( - self, mock_cuda_platform, hidden_size, tp_size, element_size, expected - ): - """Test threshold calculation with various parameter combinations.""" - with mock_cuda_platform(capability=(9, 0)): - result = get_sequence_parallelism_threshold( - hidden_size=hidden_size, - tp_size=tp_size, - element_size=element_size, - ) - assert result == expected - - def test_hidden_size_boundary(self, mock_cuda_platform): - """Test behavior at the exact hidden_size boundary.""" - with mock_cuda_platform(capability=(9, 0)): - # Just below threshold - result = get_sequence_parallelism_threshold( - hidden_size=SP_MIN_HIDDEN_SIZE[90] - 1, - tp_size=2, - element_size=2, - ) - assert result is None - - # Exactly at threshold - result = get_sequence_parallelism_threshold( - hidden_size=SP_MIN_HIDDEN_SIZE[90], - tp_size=2, - element_size=2, - ) - assert result is not None - - -# XPU-specific constants (must match sequence_parallelism.py values) -_XPU_MIN_HIDDEN_SIZE = 4096 -_XPU_MIN_PER_GPU_SIZE_MB = 8.0 - - -class TestGetSequenceParallelismThresholdXPU: - """Tests for get_sequence_parallelism_threshold on XPU platform.""" - - def test_xpu_small_hidden_size_returns_none(self, mock_xpu_platform): - """XPU with hidden_size below threshold should return None.""" - with mock_xpu_platform(): - result = get_sequence_parallelism_threshold( - hidden_size=_XPU_MIN_HIDDEN_SIZE - 1, - tp_size=2, - element_size=2, - ) - assert result is None - - def test_xpu_large_model_returns_threshold(self, mock_xpu_platform): - """XPU with hidden_size >= threshold should return calculated value.""" - with mock_xpu_platform(): - hidden_size = _XPU_MIN_HIDDEN_SIZE - tp_size = 2 - element_size = 2 - result = get_sequence_parallelism_threshold( - hidden_size=hidden_size, - tp_size=tp_size, - element_size=element_size, - ) - # (8 * 2 * 1024 * 1024) // (4096 * 2) = 2048 - MiB = 1024 * 1024 - expected = int( - (_XPU_MIN_PER_GPU_SIZE_MB * tp_size * MiB) // (hidden_size * element_size) - ) - assert result == expected - assert result == 2048 - - @pytest.mark.parametrize( - "hidden_size,tp_size,element_size,expected", - [ - # (8 * 1 * 1024 * 1024) // (4096 * 2) = 1024 - (4096, 1, 2, 1024), - # (8 * 4 * 1024 * 1024) // (4096 * 2) = 4096 - (4096, 4, 2, 4096), - # (8 * 2 * 1024 * 1024) // (8192 * 2) = 1024 - (8192, 2, 2, 1024), - # (8 * 2 * 1024 * 1024) // (4096 * 4) = 1024 - (4096, 2, 4, 1024), - ], - ) - def test_xpu_threshold_calculation_variations( - self, mock_xpu_platform, hidden_size, tp_size, element_size, expected - ): - """Test XPU threshold calculation with various parameter combinations.""" - with mock_xpu_platform(): - result = get_sequence_parallelism_threshold( - hidden_size=hidden_size, - tp_size=tp_size, - element_size=element_size, - ) - assert result == expected - - def test_xpu_hidden_size_boundary(self, mock_xpu_platform): - """Test behavior at the exact XPU hidden_size boundary.""" - with mock_xpu_platform(): - # Just below threshold - result = get_sequence_parallelism_threshold( - hidden_size=_XPU_MIN_HIDDEN_SIZE - 1, - tp_size=2, - element_size=2, - ) - assert result is None - - # Exactly at threshold - result = get_sequence_parallelism_threshold( - hidden_size=_XPU_MIN_HIDDEN_SIZE, - tp_size=2, - element_size=2, - ) - assert result is not None diff --git a/tests/config/test_model_arch_config.py b/tests/config/test_model_arch_config.py index ae4c7220f2..8ec16ee7f9 100644 --- a/tests/config/test_model_arch_config.py +++ b/tests/config/test_model_arch_config.py @@ -10,13 +10,11 @@ from typing import cast from unittest.mock import Mock import pytest -from transformers import PreTrainedConfig +from transformers import DiffusionGemmaConfig, PreTrainedConfig from transformers.models.gemma4.configuration_gemma4 import Gemma4TextConfig from vllm.config import ModelConfig, ParallelConfig, SpeculativeConfig from vllm.config.model_arch import ModelArchitectureConfig -from vllm.transformers_utils.configs.diffusion_gemma import DiffusionGemmaConfig -from vllm.transformers_utils.configs.gemma4 import gemma4_layer_config from vllm.transformers_utils.model_arch_config_convertor import ( MODEL_ARCH_CONFIG_CONVERTORS, Gemma4ModelArchConfigConvertor, @@ -318,11 +316,7 @@ def _gemma4_text_config(**overrides) -> Gemma4TextConfig: def test_gemma4_head_dims_vary_by_layer_type(): - """Gemma4's full attention layers are wider than its sliding ones. - - Transformers >= 5.15.0 says so in the config; this exercises the convertor - building the same per-layer view from the flat attributes used before that. - """ + """Gemma4's full attention layers are wider than its sliding ones.""" text_config = _gemma4_text_config( num_global_key_value_heads=8, attention_k_eq_v=True ) @@ -332,9 +326,9 @@ def test_gemma4_head_dims_vary_by_layer_type(): assert (arch.head_size, arch.total_num_kv_heads) == (32, 8) assert [arch[i].head_size for i in range(6)] == [16] * 5 + [32] assert [arch[i].total_num_kv_heads for i in range(6)] == [4] * 5 + [8] - # The model files resolve each layer through the same helper, so the KV cache - # vLLM allocates and the projections the model builds cannot disagree. - assert [gemma4_layer_config(text_config, i).head_dim for i in range(6)] == [ + # The model files read the same per-layer configs, so the KV cache vLLM + # allocates and the projections the model builds cannot disagree. + assert [text_config.per_layer_config[i].head_dim for i in range(6)] == [ arch[i].head_size for i in range(6) ] diff --git a/tests/distributed/aux_output_connector/test_store.py b/tests/distributed/aux_output_connector/test_store.py index 39277f3e7e..62d4472811 100644 --- a/tests/distributed/aux_output_connector/test_store.py +++ b/tests/distributed/aux_output_connector/test_store.py @@ -96,6 +96,10 @@ class _SchedulerRequest: default_factory=lambda: SimpleNamespace(routed_experts_prompt_start=0) ) + @property + def num_prompt_tokens(self) -> int: + return self.num_tokens - self.num_output_tokens + def is_finished(self) -> bool: return self.finished @@ -247,9 +251,6 @@ def test_next_step_does_not_consume_pending_output(monkeypatch): event = Mock() monkeypatch.setattr(torch.cuda, "Event", lambda **kwargs: event) monkeypatch.setattr(async_utils, "stream", lambda *args: nullcontext()) - monkeypatch.setattr( - async_utils, "async_copy_to_np", lambda tensor: tensor.numpy().copy() - ) worker = _make_worker(1) worker.begin_step( _metadata(0, [_request_metadata("request", 0, 1, 0, [])], {}).metadata @@ -1439,6 +1440,18 @@ def test_scheduler_starts_worker_output_at_requested_prompt_token(): assert metadata.requests == {request.request_id: 2} + # A P/D prefiller cuts its prompt short of the last token, which can pass + # the requested start; the rows then start at the end of the cut prompt. + connector = _make_connector() + request = _scheduler_request( + "cut", [b"a" * 32], num_tokens=3, num_output_tokens=0, prompt_start=4 + ) + metadata = connector.build_connector_meta( + _step_output([request.request_id], [0], [3]), + {request.request_id: request}, + ) + assert metadata.requests == {request.request_id: 3} + def test_scheduler_connector_preserves_request_finish_order(): connector = _make_connector() diff --git a/tests/distributed/test_engram_dp_shard.py b/tests/distributed/test_engram_dp_shard.py index 637d179a5d..8b7d3e8cec 100644 --- a/tests/distributed/test_engram_dp_shard.py +++ b/tests/distributed/test_engram_dp_shard.py @@ -39,9 +39,16 @@ from vllm.distributed.parallel_state import ( initialize_model_parallel, ) from vllm.forward_context import set_forward_context +from vllm.model_executor.layers.quantization.utils.mxfp8_utils import ( + dequant_mxfp8_to_bf16, +) +from vllm.model_executor.layers.quantization.utils.nvfp4_emulation_utils import ( + convert_swizzled_to_linear, +) from vllm.models.deepseek_v41.common import engram as engram_ops from vllm.models.deepseek_v41.common.engram import ( Engram, + EngramBatch, EngramLayout, ParallelEngramEmbedding, engram_head_shard_rank, @@ -207,10 +214,75 @@ def _worker(rank: int, tp_size: int, port: int) -> None: n_heads, sequence_parallel, ) + if tp_size == 1: + _check_fp8_batch(vllm_config) finally: cleanup_dist_env_and_memory() +def _check_fp8_batch(vllm_config): + """MXFP8 staging matches the tables across peer/all-to-all slots, idle + replicas and graph replays.""" + from vllm.compilation.breakable_cudagraph import BreakableCUDAGraphCapture + + dp_rank = vllm_config.parallel_config.data_parallel_rank + layers = [] + for index in range(2): + weight, scale = _full_table(sum(HEAD_SIZES)) + weight = (weight.float() * (1 - 2 * index)).to(torch.float8_e4m3fn) + table = ParallelEngramEmbedding( + sum(HEAD_SIZES), DIM, HEAD_SIZES, cpu_offload=True + ) + table.weight.weight_loader(table.weight, weight) + table.weight_scale_inv.weight_loader(table.weight_scale_inv, scale) + staged = torch.empty(0, device="cuda") + reference = (weight.cuda(), scale.cuda()) + layers.append(SimpleNamespace(embed_tokens=table, staged_rows=staged)) + layers[-1].layer_hash_index, layers[-1].reference = index, reference + batch = EngramBatch(layers, 257) + + def check(ids): + tokens = ids.shape[0] + for layer in layers: + x = batch.wkv_input(layer, tokens) + sf = convert_swizzled_to_linear(x.scale, tokens, x.data.shape[1], BLOCK) + actual = dequant_mxfp8_to_bf16(x.data, sf).unflatten(-1, (-1, DIM)) + weight, scale = layer.reference + expected = _reference(weight, scale, ids[:, layer.layer_hash_index]) + torch.testing.assert_close(actual, expected, atol=0, rtol=0) + + for counts in ((7, 0, 5, 1), (127,) * 4, (128,) * 4, (257, 0, 129, 1)): + tokens = counts[dp_rank] + seed = 700 + 10 * dp_rank + ids = torch.stack([_make_ids(HEAD_SIZES, tokens, seed + i) for i in (0, 1)], 1) + with set_forward_context( + None, + vllm_config, + num_tokens=tokens, + num_tokens_across_dp=torch.tensor(counts, dtype=torch.int32), + ): + batch.prepare_embeddings(ids) + check(ids) + for capture in ("full", "breakable") if len(set(counts)) == 1 else (): + warmup = torch.cuda.Stream() + warmup.wait_stream(torch.cuda.current_stream()) + if capture == "full": + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph, stream=warmup): + batch.prepare_embeddings(ids) + else: + graph = BreakableCUDAGraphCapture() + with torch.cuda.stream(warmup), graph: + batch.prepare_embeddings(ids) + torch.cuda.current_stream().wait_stream(warmup) + for step in (2, 3): + replay_ids = _make_ids(HEAD_SIZES, tokens, seed + step) + ids.copy_(torch.stack([replay_ids] * 2, 1)) + graph.replay() + check(ids) + del graph + + def _check_shared_storage_lifetime_and_failures(failure): """One peer's failure must roll back every mapping/registration, without hangs.""" import gc @@ -516,6 +588,7 @@ def _check_dummy_hash_model_forward( model.use_native_mega_moe = False model.use_sequence_parallel = False model.fuse_mhc_all_reduce = False + model.engram_batch = None model.engram_hash = state model.engram_swa_prefix = "swa" model.engram_dp_shared_memory = dp_shared_memory diff --git a/tests/distributed/test_eplb_execute.py b/tests/distributed/test_eplb_execute.py index 2037535e3d..acdb9f39fc 100644 --- a/tests/distributed/test_eplb_execute.py +++ b/tests/distributed/test_eplb_execute.py @@ -36,7 +36,11 @@ def test_gloo_receive_staging_does_not_force_gpu_sync(monkeypatch): gsd._install_copy_checkers() monkeypatch.setattr(eplb_comm, "is_local_first_rank", lambda: False) - monkeypatch.setattr(eplb_comm, "P2POp", lambda op, tensor, peer, group: tensor) + monkeypatch.setattr( + eplb_comm, + "P2POp", + lambda op, tensor, peer=None, group=None, tag=0, group_peer=None: tensor, + ) def receive(tensors): for tensor in tensors: @@ -849,3 +853,42 @@ def test_rearrange_expert_weights_profile_mode(world_size): _test_rearrange_expert_weights_profile_mode, world_size, ) + + +def _test_eplb_communicator_pipeline_parallel(env, world_size, backend): + set_env_vars_and_device(env) + config = VllmConfig() + config.parallel_config.tensor_parallel_size = 2 + config.parallel_config.pipeline_parallel_size = world_size // 2 + + with set_current_vllm_config(config): + ensure_model_parallel_initialized( + tensor_model_parallel_size=2, + pipeline_model_parallel_size=world_size // 2, + ) + ep_group = get_tp_group() + peer_rank = 1 - ep_group.rank_in_group + rank = torch.distributed.get_rank() + send_tensor = torch.full((16,), rank, device="cuda", dtype=torch.int32) + recv_tensor = torch.empty_like(send_tensor) + communicator = create_eplb_communicator_or_raise( + group_coordinator=ep_group, + backend=backend, + expert_weights=[[send_tensor]], + expert_buffer=[recv_tensor], + ) + communicator.add_send([send_tensor], peer_rank, expert_id=0) + communicator.add_recv([recv_tensor], peer_rank, expert_id=0) + communicator.execute() + + expected = torch.full_like(recv_tensor, ep_group.ranks[peer_rank]) + torch.testing.assert_close(recv_tensor, expected) + + +@pytest.mark.parametrize("world_size", [2, 4]) +@pytest.mark.parametrize("backend", ["torch_nccl", "torch_gloo"]) +def test_eplb_communicator_pipeline_parallel(world_size, backend): + """Transfer weights within each PP stage, including nonzero global ranks.""" + if torch.accelerator.device_count() < world_size: + pytest.skip(f"Need at least {world_size} GPUs to run the test") + distributed_run(_test_eplb_communicator_pipeline_parallel, world_size, backend) diff --git a/tests/distributed/test_mnnvl_alltoall.py b/tests/distributed/test_mnnvl_alltoall.py index f7dac37940..bd7ff4df60 100644 --- a/tests/distributed/test_mnnvl_alltoall.py +++ b/tests/distributed/test_mnnvl_alltoall.py @@ -150,7 +150,9 @@ def _init_dp_environment(world_size, rank, port, dp_size, dp_port): @contextmanager -def _make_forward_context(rank, world_size, num_tokens_per_rank): +def _make_forward_context( + rank, world_size, num_tokens_per_rank, num_tokens_across_dp=None +): """Keep the vLLM config and mock-DP forward context active together. Returns a context manager suitable for ``with`` statements. @@ -175,15 +177,15 @@ def _make_forward_context(rank, world_size, num_tokens_per_rank): is_moe_model=True, data_parallel_rank=rank, ) + if num_tokens_across_dp is None: + num_tokens_across_dp = [num_tokens_per_rank] * world_size with ( set_current_vllm_config(vllm_config), set_forward_context( _AttnMeta(), vllm_config, num_tokens=num_tokens_per_rank, - num_tokens_across_dp=torch.tensor( - [num_tokens_per_rank] * world_size, dtype=torch.int - ), + num_tokens_across_dp=torch.tensor(num_tokens_across_dp, dtype=torch.int), ), ): yield @@ -516,6 +518,121 @@ def test_one_sided_manager_workspace_grow(world_size): # --------------------------------------------------------------------------- +class _FakeCombineGroup: + def __init__(self, combine_input): + self.combine_input = combine_input + self.allocation_calls = 0 + self.combined_input = None + + def allocate_combine_input(self, shape, dtype, device, **kwargs): + self.allocation_calls += 1 + assert self.combine_input is not None + assert self.combine_input.shape == shape + assert self.combine_input.dtype == dtype + assert self.combine_input.device == device + return self.combine_input + + def combine_into_output(self, hidden_states, output, **kwargs): + self.combined_input = hidden_states + output.copy_(hidden_states) + return output + + +def test_args_cpu_workspace_allocation(monkeypatch): + from types import SimpleNamespace + + import vllm.model_executor.layers.fused_moe.modular_kernel as mk + from vllm.model_executor.layers.fused_moe.activation import MoEActivation + + kernel = object.__new__(mk.FusedMoEKernelModularImpl) + kernel.fused_experts = SimpleNamespace( + workspace_dtype=lambda dtype: dtype, + workspace_shapes=lambda *args: ((0,), (0,), (2, 3)), + finalize_weight_and_reduce_impl=lambda: None, + ) + kernel.prepare_finalize = SimpleNamespace( + allocate_fused_expert_output=lambda *args: None, + ) + monkeypatch.setattr(mk.current_platform, "is_cpu", lambda: True) + + workspace13, workspace2, output, specialized = kernel._allocate_buffers( + torch.float32, + torch.device("cpu"), + 2, + 2, + 3, + 3, + 1, + 1, + 1, + None, + MoEActivation.SILU, + ) + assert workspace13.numel() == workspace2.numel() == 0 + assert output.shape == (2, 3) + assert output.device.type == "cpu" + assert not specialized + + +def test_args_finalize_noop_reuses_fused_expert_output(monkeypatch): + import vllm.model_executor.layers.fused_moe.prepare_finalize.naive_dp_ep as pf + from vllm.model_executor.layers.fused_moe.topk_weight_and_reduce import ( + TopKWeightAndReduceNoOP, + ) + + fused_expert_output = torch.arange(6, dtype=torch.float32).view(2, 3) + output = torch.empty_like(fused_expert_output) + group = _FakeCombineGroup(combine_input=None) + monkeypatch.setattr(pf, "get_ep_group", lambda: group) + + impl = pf.MoEPrepareAndFinalizeNaiveDPEPModular() + impl.finalize( + output, + fused_expert_output, + torch.ones((2, 1)), + torch.zeros((2, 1), dtype=torch.long), + False, + TopKWeightAndReduceNoOP(), + ) + + assert group.allocation_calls == 0 + assert group.combined_input is fused_expert_output + torch.testing.assert_close(output, fused_expert_output) + + +def test_args_finalize_contiguous_writes_combine_input(monkeypatch): + import vllm.model_executor.layers.fused_moe.prepare_finalize.naive_dp_ep as pf + import vllm.model_executor.layers.fused_moe.topk_weight_and_reduce as wr + + fused_expert_output = torch.arange(12, dtype=torch.float32).view(2, 2, 3) + original = fused_expert_output.clone() + topk_weights = torch.tensor([[0.25, 0.75], [0.4, 0.6]]) + combine_input = torch.empty((2, 3), dtype=torch.float32) + output = torch.empty_like(combine_input) + group = _FakeCombineGroup(combine_input) + monkeypatch.setattr(pf, "get_ep_group", lambda: group) + monkeypatch.setattr( + wr.ops, + "moe_sum", + lambda hidden_states, out: out.copy_(hidden_states.sum(dim=1)), + ) + + impl = pf.MoEPrepareAndFinalizeNaiveDPEPModular() + impl.finalize( + output, + fused_expert_output, + topk_weights, + torch.zeros((2, 2), dtype=torch.long), + False, + wr.TopKWeightAndReduceContiguous(), + ) + + expected = (original * topk_weights.unsqueeze(-1)).sum(dim=1) + assert group.allocation_calls == 1 + assert group.combined_input is combine_input + torch.testing.assert_close(output, expected) + + def _args_dispatch_combine_worker(rank, world_size): from vllm.distributed.device_communicators.all2all import AgRsAll2AllManager from vllm.forward_context import get_forward_context @@ -604,14 +721,39 @@ def _args_dispatch_combine_worker(rank, world_size): # -- combine (reduce-scatter) -- # Each token i has value i in all columns; after reduce-scatter # each rank gets its slice, summed across ranks. - expert_out = ( + expert_values = ( torch.arange(total_tokens, device=device, dtype=torch.float32) .unsqueeze(1) .expand(total_tokens, hidden_size) .contiguous() ) + expert_out = manager.allocate_combine_input( + expert_values.shape, + expert_values.dtype, + expert_values.device, + is_sequence_parallel=True, + ) + if expert_out is None: + expert_out = expert_values + else: + from vllm.distributed.device_communicators.pynccl_allocator import ( + is_symmetric_memory_tensor, + ) - combined = manager.combine(expert_out, is_sequence_parallel=True) + assert is_symmetric_memory_tensor(expert_out) + expert_out.copy_(expert_values) + + combine_output = torch.empty( + (tokens_per_rank, hidden_size), + device=device, + dtype=expert_out.dtype, + ) + combined = manager.combine_into_output( + expert_out, + combine_output, + is_sequence_parallel=True, + ) + assert combined.data_ptr() == combine_output.data_ptr() assert combined.shape == (tokens_per_rank, hidden_size) for i in range(tokens_per_rank): @@ -623,6 +765,61 @@ def _args_dispatch_combine_worker(rank, world_size): torch.distributed.barrier() + uneven_sizes = [tokens_per_rank + r for r in range(world_size)] + with _make_forward_context( + rank, + world_size, + uneven_sizes[rank], + num_tokens_across_dp=uneven_sizes, + ): + dp_metadata = get_forward_context().dp_metadata + assert dp_metadata is not None + with dp_metadata.sp_local_sizes(sequence_parallel_size=1): + assert ( + manager.allocate_combine_input( + (sum(uneven_sizes), hidden_size), + torch.float32, + device, + is_sequence_parallel=True, + ) + is None + ) + + total_uneven_tokens = sum(uneven_sizes) + uneven_input = ( + torch.arange( + total_uneven_tokens, + dtype=torch.float32, + device=device, + ) + .unsqueeze(1) + .expand(total_uneven_tokens, hidden_size) + .contiguous() + ) + uneven_input.add_(rank) + uneven_output = torch.empty( + (uneven_sizes[rank], hidden_size), + dtype=torch.float32, + device=device, + ) + combined = get_ep_group().reduce_scatterv_into_output( + uneven_input, + uneven_output, + dim=0, + sizes=uneven_sizes, + ) + assert combined.data_ptr() == uneven_output.data_ptr() + + start = sum(uneven_sizes[:rank]) + expected = torch.arange( + start, + start + uneven_sizes[rank], + dtype=torch.float32, + device=device, + ) * world_size + sum(range(world_size)) + expected = expected.unsqueeze(1).expand_as(combined) + torch.testing.assert_close(combined, expected) + @requires_multi_gpu @pytest.mark.parametrize("world_size", [2]) diff --git a/tests/distributed/test_nccl_symm_mem.py b/tests/distributed/test_nccl_symm_mem.py index bd0270fc48..d3013445cf 100644 --- a/tests/distributed/test_nccl_symm_mem.py +++ b/tests/distributed/test_nccl_symm_mem.py @@ -3,6 +3,7 @@ import random import typing +from types import SimpleNamespace import pytest import torch @@ -12,11 +13,15 @@ import torch.multiprocessing as mp import vllm.envs as envs from tests.utils import ensure_current_vllm_config from vllm.distributed import cleanup_dist_env_and_memory -from vllm.distributed.device_communicators.cuda_communicator import CudaCommunicator +from vllm.distributed.device_communicators.cuda_communicator import ( + NCCL_DIRECT_SYMM_RS_OUTPUT_MIN_VERSION, + CudaCommunicator, +) from vllm.distributed.device_communicators.pynccl import register_nccl_symmetric_ops from vllm.distributed.device_communicators.pynccl_allocator import ( get_nccl_mem_pool, is_symmetric_memory_enabled, + is_symmetric_memory_tensor, ) from vllm.distributed.parallel_state import ( get_tp_group, @@ -32,6 +37,53 @@ random.seed(44) test_size_elements = 4 * 1024 * 1024 +def test_disabled_pynccl_does_not_allocate_symmetric_buffer(): + communicator = object.__new__(CudaCommunicator) + communicator.pynccl_comm = SimpleNamespace(disabled=True) + + assert ( + communicator.get_symmetric_memory_buffer( + "test", (1,), torch.float32, torch.device("cuda") + ) + is None + ) + + +@pytest.mark.parametrize("use_aiter", [False, True]) +def test_reduce_scatterv_preserves_caller_output(monkeypatch, use_aiter): + import vllm.distributed.device_communicators.cuda_communicator as cc + + communicator = object.__new__(CudaCommunicator) + communicator.world_size = 2 + communicator.rank_in_group = 0 + input_ = torch.arange(12, dtype=torch.float32).view(4, 3) + output = torch.full((2, 3), float("nan")) + + def reduce_scatter(out, inp): + assert out is output + out.copy_(inp.view(2, 2, 3).sum(dim=0)) + + communicator.pynccl_comm = SimpleNamespace( + disabled=False, nccl_version=22902, reduce_scatter=reduce_scatter + ) + communicator._can_use_aiter_ag_rs = lambda sizes: use_aiter and sizes is None + communicator.aiter_ar_comm = SimpleNamespace( + should_custom_rs=lambda inp, dim: True, + custom_reduce_scatter=lambda inp, out, dim: reduce_scatter(out, inp), + ) + monkeypatch.setattr(cc, "should_nccl_symm_mem_ag_rs", lambda: True) + monkeypatch.setattr(cc, "is_symmetric_memory_tensor", lambda tensor: False) + monkeypatch.setenv("VLLM_BATCH_INVARIANT", "0") + + # Explicit uniform sizes must not stage an ordinary input into symmetric + # scratch, and AITER must write the supplied output after normalization. + result = communicator.reduce_scatterv_into_output( + input_, output, dim=0, sizes=[2, 2] + ) + assert result.data_ptr() == output.data_ptr() + torch.testing.assert_close(output, input_.view(2, 2, 3).sum(dim=0)) + + def nccl_symm_mem_allreduce_worker(local_rank: int, world_size: int): monkeypatch = pytest.MonkeyPatch() with monkeypatch.context() as m: @@ -193,15 +245,107 @@ def nccl_symm_mem_reduce_scatter_worker(local_rank: int, world_size: int): pytest.skip("NCCL symmetric memory is disabled.") per_rank_size = test_size_elements // world_size - input_tensor = torch.randint( + ordinary_input = torch.randint( 1, 23, (test_size_elements,), dtype=dtype, device=device ) - input_clone = input_tensor.clone() - output = cuda_communicator.reduce_scatter(input_tensor, dim=0) + ordinary_clone = ordinary_input.clone() + ordinary_output = cuda_communicator.reduce_scatter(ordinary_input, dim=0) group = get_tp_group().device_group - expected = torch.empty(per_rank_size, dtype=dtype, device=device) + ordinary_expected = torch.empty(per_rank_size, dtype=dtype, device=device) + dist.reduce_scatter_tensor(ordinary_expected, ordinary_clone, group=group) + torch.testing.assert_close( + ordinary_output, ordinary_expected, atol=2.5, rtol=0.1 + ) + assert is_symmetric_memory_tensor(ordinary_output) + + # Both calls reuse rs_out; validate the first result before overwriting it. + for sizes in (None, [per_rank_size] * world_size): + ordinary_v_output = cuda_communicator.reduce_scatterv( + ordinary_input, dim=0, sizes=sizes + ) + torch.testing.assert_close( + ordinary_v_output, ordinary_expected, atol=2.5, rtol=0.1 + ) + assert is_symmetric_memory_tensor(ordinary_v_output) + + pynccl_comm = cuda_communicator.pynccl_comm + assert pynccl_comm is not None + if pynccl_comm.nccl_version < NCCL_DIRECT_SYMM_RS_OUTPUT_MIN_VERSION: + return + + from vllm.v1.worker import ubatching + + m.setattr(ubatching, "dbo_current_ubatch_id", lambda: 0) + ubatch0 = cuda_communicator._get_symm_scratch("ag_out", (128,), dtype, device) + ubatch0.fill_(1) + m.setattr(ubatching, "dbo_current_ubatch_id", lambda: 1) + ubatch1 = cuda_communicator._get_symm_scratch("ag_out", (128,), dtype, device) + ubatch1.fill_(2) + assert ubatch0.data_ptr() != ubatch1.data_ptr() + assert (ubatch0 == 1).all() + + m.setattr(ubatching, "dbo_current_ubatch_id", lambda: 0) + smaller = cuda_communicator._get_symm_scratch("ag_out", (64,), dtype, device) + assert smaller.shape == (64,) + assert smaller.data_ptr() == ubatch0.data_ptr() + + grown = cuda_communicator._get_symm_scratch("ag_out", (129,), dtype, device) + assert grown.shape == (129,) + assert grown.data_ptr() != ubatch0.data_ptr() + reused = cuda_communicator._get_symm_scratch("ag_out", (200,), dtype, device) + assert reused.data_ptr() == grown.data_ptr() + + input_tensor = cuda_communicator._get_symm_scratch( + "test_graph_rs_input", (test_size_elements,), dtype, device + ) + input_tensor.random_(1, 23) + input_clone = input_tensor.clone() + output = torch.empty(per_rank_size, dtype=dtype, device=device) + assert is_symmetric_memory_tensor(input_tensor) + assert not is_symmetric_memory_tensor(output) + + result = cuda_communicator.reduce_scatterv_into_output( + input_tensor, + output, + dim=0, + sizes=[per_rank_size] * world_size, + ) + assert result.data_ptr() == output.data_ptr() + + expected = torch.empty_like(output) dist.reduce_scatter_tensor(expected, input_clone, group=group) + torch.testing.assert_close(result, expected, atol=2.5, rtol=0.1) + + dist.barrier(group=get_tp_group().cpu_group) + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + graph_result = cuda_communicator.reduce_scatterv_into_output( + input_tensor, + output, + dim=0, + sizes=[per_rank_size] * world_size, + ) + assert graph_result.data_ptr() == output.data_ptr() + + input_tensor.random_(24, 47) + input_clone.copy_(input_tensor) + dist.reduce_scatter_tensor(expected, input_clone, group=group) + captured_input_ptr = input_tensor.data_ptr() + del input_tensor + grown_input = cuda_communicator._get_symm_scratch( + "test_graph_rs_input", (test_size_elements + 1,), dtype, device + ) + assert grown_input.data_ptr() != captured_input_ptr + retired = cuda_communicator.__dict__["_retired_symm_scratch_bufs"] + assert any( + buf.data_ptr() == captured_input_ptr + for buffers in retired.values() + for buf in buffers + ) + output.zero_() + graph.replay() + torch.accelerator.synchronize() torch.testing.assert_close(output, expected, atol=2.5, rtol=0.1) diff --git a/tests/distributed/test_shm_broadcast.py b/tests/distributed/test_shm_broadcast.py index 7debed4cbb..d28732261a 100644 --- a/tests/distributed/test_shm_broadcast.py +++ b/tests/distributed/test_shm_broadcast.py @@ -1,11 +1,13 @@ # SPDX-License-Identifier: Apache-2.0 # SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import gc import io import pickle import random import threading import time +import weakref from types import SimpleNamespace from unittest import mock @@ -634,6 +636,26 @@ def test_acquire_read_releases_slot_when_reader_raises(): reader.shutdown() +def test_reader_in_reference_cycle_is_collected_without_blocking(): + writer = MessageQueue( + n_reader=1, + n_local_reader=1, + max_chunk_bytes=1024 * 1024, + max_chunks=1, + ) + reader = MessageQueue.create_from_handle(writer.export_handle(), rank=0) + reader.cycle = reader + reader_ref = weakref.ref(reader) + del reader + + collector = threading.Thread(target=gc.collect, daemon=True) + collector.start() + collector.join(timeout=10) + + assert not collector.is_alive() + assert reader_ref() is None + + def test_warning_logs(caplog_vllm): """Test that warning logs are emitted at VLLM_RINGBUFFER_WARNING_INTERVAL intervals when indefinite=False, and are not emitted when indefinite=True. diff --git a/tests/engine/test_arg_utils.py b/tests/engine/test_arg_utils.py index 924d2bafec..fc9f0229c4 100644 --- a/tests/engine/test_arg_utils.py +++ b/tests/engine/test_arg_utils.py @@ -857,13 +857,14 @@ def test_cloud_storage_tokenizer_skips_get_model_path(monkeypatch): class TestDeviceIds: - def test_device_ids_with_cvd_out_of_range(self, monkeypatch): + @pytest.mark.parametrize("device_ids", [[0, 2], [-1]]) + def test_device_ids_with_cvd_out_of_range(self, monkeypatch, device_ids): """--device-ids index beyond the CVD set raises ValueError.""" from vllm.platforms import current_platform key = current_platform.device_control_env_var monkeypatch.setenv(key, "4,5") - args = EngineArgs(model="m", device_ids=[0, 2]) + args = EngineArgs(model="m", device_ids=device_ids) with pytest.raises(ValueError, match="out of range"): args._resolve_device_ids() diff --git a/tests/entrypoints/anthropic/test_anthropic_messages_conversion.py b/tests/entrypoints/anthropic/test_anthropic_messages_conversion.py index fd0d0dc2a9..399c898d0b 100644 --- a/tests/entrypoints/anthropic/test_anthropic_messages_conversion.py +++ b/tests/entrypoints/anthropic/test_anthropic_messages_conversion.py @@ -1987,6 +1987,32 @@ class TestThinkingConfig: assert result.include_reasoning is True assert result.thinking_token_budget is None + @pytest.mark.parametrize( + "config", + [ + pytest.param({"output_config": {"effort": "low"}}, id="effort"), + pytest.param({"thinking": {"type": "disabled"}}, id="disabled"), + pytest.param( + {"thinking": {"type": "enabled", "budget_tokens": 2048}}, + id="enabled", + ), + ], + ) + def test_count_tokens_matches_messages(self, config): + """Effort and thinking change the rendered prompt, so count_tokens must + apply them the same way /v1/messages does.""" + messages = [{"role": "user", "content": "Hello"}] + expected = _convert( + AnthropicMessagesRequest( + model="test-model", max_tokens=4096, messages=messages, **config + ) + ) + result = _convert( + AnthropicCountTokensRequest(model="test-model", messages=messages, **config) + ) + assert result.reasoning_effort == expected.reasoning_effort + assert result.thinking_token_budget == expected.thinking_token_budget + class TestProbeDisabledThinkingEffort: """``auto`` falls back to ``low`` when ``none`` cannot turn thinking off.""" @@ -2117,3 +2143,30 @@ class TestMidConversationToolChanges: ValidationError, match='only allowed in messages with role "system"' ): request_cls(model="test-model", max_tokens=128, **fields) + + +class TestWatermarking: + def test_defaults_to_unspecified(self): + request = _make_request([{"role": "user", "content": "hi"}]) + + assert _convert(request).watermarking is None + + def test_forwards_explicit_enable(self): + request = _make_request([{"role": "user", "content": "hi"}], watermarking=True) + + assert _convert(request).watermarking is True + + def test_forwards_the_opt_out(self): + request = _make_request([{"role": "user", "content": "hi"}], watermarking=False) + + assert _convert(request).watermarking is False + + @pytest.mark.parametrize("watermarking", [None, True, False]) + def test_reaches_sampling_params(self, watermarking): + request = _make_request( + [{"role": "user", "content": "hi"}], watermarking=watermarking + ) + + params = _convert(request).to_sampling_params(16, {}) + + assert params.watermarking is watermarking diff --git a/tests/entrypoints/cohere/test_api_router.py b/tests/entrypoints/cohere/test_api_router.py index 94324dbed7..209ceb4e3e 100644 --- a/tests/entrypoints/cohere/test_api_router.py +++ b/tests/entrypoints/cohere/test_api_router.py @@ -32,8 +32,14 @@ from vllm.entrypoints.cohere.protocol import ( AssistantMessageResponse, CohereChatV2Response, ) -from vllm.entrypoints.cohere.serving import CohereServingChatV2 -from vllm.entrypoints.scale_out.token_in_token_out.protocol import GenerateRequest +from vllm.entrypoints.cohere.serving import ( + _API_SERVER_ONLY_TEMPLATE_KWARGS, + CohereServingChatV2, +) +from vllm.entrypoints.scale_out.token_in_token_out.protocol import ( + GenerateRequest, + ReasoningParserKwargs, +) from vllm.entrypoints.serve.engine.protocol import ErrorInfo, ErrorResponse from vllm.entrypoints.serve.exception_handling.handlers.http import ( http_exception_handler, @@ -44,7 +50,7 @@ from vllm.entrypoints.serve.exception_handling.handlers.validation import ( from vllm.entrypoints.serve.exception_handling.handlers.vllm_error import ( vllm_error_handler, ) -from vllm.exceptions import VLLMError +from vllm.exceptions import VLLMError, VLLMValidationError from vllm.sampling_params import SamplingParams @@ -72,7 +78,7 @@ class _Handler: * a :class:`CohereChatV2Response` (non-streaming JSON path); * an async generator yielding SSE frames (streaming path); * an :class:`ErrorResponse` (error envelope path); or - * an exception (router-level 500 path). + * an exception (router-level error path). """ def __init__(self, result): @@ -97,6 +103,12 @@ class _RenderChatHandler: def to_chat_completion_request(self, request): return CohereServingChatV2._convert_v2_to_chat_completion(request) + def _engine_chat_template_kwargs(self, chat_template_kwargs): + return CohereServingChatV2._engine_chat_template_kwargs( + self, # type: ignore[arg-type] + chat_template_kwargs, + ) + class _RenderHandler: """Minimal stand-in for :class:`ServingRender`. @@ -359,6 +371,22 @@ class TestEndpoint: body = r.json() assert body == {"message": "kaboom"} + @pytest.mark.parametrize( + "exc", + [ + VLLMValidationError("bad temperature", parameter="temperature"), + ValueError("bad temperature"), + ], + ids=["vllm_validation_error", "value_error"], + ) + def test_handler_client_error_returns_400_envelope(self, exc): + """Client-caused handler errors are 4xx, matching the OpenAI routes.""" + app = _build_app(handler=_Handler(exc)) + with TestClient(app) as client: + r = client.post("/cohere/v2/chat", json=_minimal_request_body()) + assert r.status_code == HTTPStatus.BAD_REQUEST + assert "bad temperature" in r.json()["message"] + def test_non_json_content_type_rejected(self): """The ``validate_json_request`` dependency raises ``RequestValidationError`` (HTTP 422) for non-JSON content @@ -459,6 +487,24 @@ class TestRenderEndpoint: # The conversion lower-cases safety_mode for the renderer. assert kwargs["safety_mode"] == "strict" + def test_api_server_only_kwargs_not_forwarded(self): + """As on ``/cohere/v2/chat``, the engine's reasoning parser must not + receive the template kwargs that only the API server reads.""" + result = _generate_request() + result.reasoning_parser_kwargs = ReasoningParserKwargs( + chat_template_kwargs={ + "safety_mode": "strict", + **dict.fromkeys(_API_SERVER_ONLY_TEMPLATE_KWARGS, "x"), + } + ) + app = _build_render_app(_RenderChatHandler(), _RenderHandler(result)) + with TestClient(app) as client: + r = client.post("/cohere/v2/chat/render", json=_minimal_request_body()) + assert r.status_code == HTTPStatus.OK + assert r.json()["reasoning_parser_kwargs"] == { + "chat_template_kwargs": {"safety_mode": "strict"} + } + def test_501_when_chat_handler_missing(self): app = _build_render_app(None, _RenderHandler(_generate_request())) with TestClient(app) as client: @@ -495,10 +541,10 @@ class TestRenderEndpoint: assert r.status_code == HTTPStatus.INTERNAL_SERVER_ERROR assert r.json() == {"message": "tokenizer exploded"} - def test_conversion_exception_returns_500_envelope(self): + def test_conversion_client_error_returns_400_envelope(self): """Conversion runs inside the same ``try`` as the render call, so - a bad v2 body that slips past Pydantic still yields the Cohere - error envelope rather than an unhandled 500 with vLLM's shape. + a bad v2 body that slips past Pydantic still yields a 4xx in the + Cohere error envelope rather than an unhandled 500 with vLLM's shape. """ class _Boom: @@ -508,7 +554,7 @@ class TestRenderEndpoint: app = _build_render_app(_Boom(), _RenderHandler(_generate_request())) with TestClient(app) as client: r = client.post("/cohere/v2/chat/render", json=_minimal_request_body()) - assert r.status_code == HTTPStatus.INTERNAL_SERVER_ERROR + assert r.status_code == HTTPStatus.BAD_REQUEST assert r.json() == {"message": "unconvertible message"} def test_invalid_body_returns_422(self): diff --git a/tests/entrypoints/cohere/test_serving_conversion.py b/tests/entrypoints/cohere/test_serving_conversion.py index 5cbe13df02..e6696c26b4 100644 --- a/tests/entrypoints/cohere/test_serving_conversion.py +++ b/tests/entrypoints/cohere/test_serving_conversion.py @@ -25,6 +25,7 @@ from vllm.entrypoints.cohere.cohere_chat_message import ( from vllm.entrypoints.cohere.protocol import ( CohereChatV2Request, CohereChatV2Response, + CohereLogprobItem, ) from vllm.entrypoints.cohere.serving import ( _FINISH_REASON_MAP, @@ -130,6 +131,8 @@ def _build_chat_completion_response( citations: list[Any] | None = None, usage: dict[str, Any] | None = None, kv_transfer_params: dict[str, Any] | None = None, + logprobs: list[tuple[str, float]] | None = None, + token_ids: list[int] | None = None, ) -> ChatCompletionResponse: message: dict[str, Any] = {"role": "assistant"} if content is not None: @@ -151,6 +154,12 @@ def _build_chat_completion_response( "message": message, "finish_reason": finish_reason, "stop_reason": stop_reason, + "logprobs": ( + {"content": [{"token": t, "logprob": lp} for t, lp in logprobs]} + if logprobs is not None + else None + ), + "token_ids": token_ids, } ], # ``usage`` is a required field on ChatCompletionResponse, but the @@ -689,6 +698,8 @@ class TestBuildBaseChatCompletion: assert result.top_k == 50 assert result.top_p == 0.95 assert result.logprobs is True + # Needed to fill ``token_ids`` in the Cohere logprobs response. + assert result.return_token_ids is True assert result.priority == 2 assert result.kv_transfer_params == {"x": 1} # ``chat_template_kwargs`` may be expanded by _apply_cohere_*; the @@ -1907,6 +1918,7 @@ class TestChatCompletionToV2: assert v2.message.tool_calls is None assert v2.message.tool_plan is None assert v2.usage is None + assert v2.logprobs is None def test_stop_sequence_finish_reason(self): serving = _serving() @@ -1990,6 +2002,19 @@ class TestChatCompletionToV2: v2 = serving._chat_completion_to_v2(resp, _make_request()) assert v2.kv_transfer_params == {"k": 1} + def test_logprobs_propagated(self): + serving = _serving() + resp = _build_chat_completion_response( + content="Hello", + logprobs=[("Hel", -0.25), ("lo", -0.5)], + token_ids=[101, 102], + ) + v2 = serving._chat_completion_to_v2(resp, _make_request(logprobs=True)) + assert v2.logprobs == [ + CohereLogprobItem(text="Hel", token_ids=[101], logprobs=[-0.25]), + CohereLogprobItem(text="lo", token_ids=[102], logprobs=[-0.5]), + ] + # ====================================================================== # _build_usage @@ -2510,3 +2535,24 @@ class TestCreateErrorResponse: assert err.error.message == "oops" assert err.error.code == 400 assert err.error.type == "bad_request" + + +def test_watermarking_defaults_to_unspecified(): + assert _convert(_make_request()).watermarking is None + + +def test_watermarking_explicit_enable_is_forwarded(): + assert _convert(_make_request(watermarking=True)).watermarking is True + + +def test_watermarking_opt_out_is_forwarded(): + assert _convert(_make_request(watermarking=False)).watermarking is False + + +@pytest.mark.parametrize("watermarking", [None, True, False]) +def test_watermarking_reaches_sampling_params(watermarking): + request = _make_request(watermarking=watermarking) + + params = _convert(request).to_sampling_params(16, {}) + + assert params.watermarking is watermarking diff --git a/tests/entrypoints/generate/generative_scoring/test_generative_scoring.py b/tests/entrypoints/generate/generative_scoring/test_generative_scoring.py index f413869c90..ea7c81ea26 100644 --- a/tests/entrypoints/generate/generative_scoring/test_generative_scoring.py +++ b/tests/entrypoints/generate/generative_scoring/test_generative_scoring.py @@ -12,6 +12,7 @@ Tests cover: import math from dataclasses import dataclass, field +from types import SimpleNamespace from typing import Any from unittest.mock import MagicMock @@ -138,6 +139,7 @@ class TestProtocolModels: assert req_str.apply_softmax is True # default assert req_str.item_first is False # default assert req_str.add_special_tokens is True # default + assert req_str.cache_salt is None # Test request with pre-tokenized inputs and custom options req_tok = GenerativeScoringRequest( @@ -295,28 +297,49 @@ class TestGeneration: """Tests for the full generation flow with mocked engine.""" @pytest.mark.asyncio - async def test_successful_generation(self): - """Test successful score generation returns valid response.""" + @pytest.mark.parametrize("cache_salt", [None, "tenant-a"]) + @pytest.mark.parametrize( + "query,items", + [ + ("Is this city the capital of France? ", ["Paris", "London"]), + ([100, 101], [[200, 201], [300, 301]]), + ], + ids=["text", "token_ids"], + ) + async def test_successful_generation(self, cache_salt, query, items): + """Test scoring succeeds and preserves cache_salt for every item.""" mock_engine = _create_mock_engine() serving = _create_serving(mock_engine) mock_logprobs = {1234: -0.5, 5678: -2.0, 100: -3.0} mock_output = _create_mock_request_output(mock_logprobs) + state = SimpleNamespace(sampling_params=None, prompts=[]) async def mock_generate(*args, **kwargs): + state.prompts.append(args[0]) + state.sampling_params = args[1] yield mock_output mock_engine.generate = mock_generate request = GenerativeScoringRequest( model=MODEL_NAME, - query="Is Paris the capital?", - items=["Yes", "No"], + query=query, + items=items, label_token_ids=[1234, 5678], + cache_salt=cache_salt, ) result = await serving.create_generative_scoring(request, None) assert isinstance(result, GenerativeScoringResponse) + assert state.sampling_params is not None + assert state.sampling_params.watermarking is False + assert len(state.prompts) == len(items) + for prompt in state.prompts: + if cache_salt is None: + assert "cache_salt" not in prompt + else: + assert prompt["cache_salt"] == cache_salt assert len(result.data) == 2 for item_result in result.data: assert 0.0 <= item_result.score <= 1.0 diff --git a/tests/entrypoints/serve/dev/rlhf/__init__.py b/tests/entrypoints/generate/structured_decisions/__init__.py similarity index 100% rename from tests/entrypoints/serve/dev/rlhf/__init__.py rename to tests/entrypoints/generate/structured_decisions/__init__.py diff --git a/tests/entrypoints/generate/structured_decisions/test_question_types.py b/tests/entrypoints/generate/structured_decisions/test_question_types.py new file mode 100644 index 0000000000..3b9ad7ba60 --- /dev/null +++ b/tests/entrypoints/generate/structured_decisions/test_question_types.py @@ -0,0 +1,179 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +from typing import Any + +import pytest + +from vllm.entrypoints.generate.structured_decisions.question_types import ( + QUESTION_TYPES, + Option, + Question, + QuestionType, + StructuredDecisionError, + build_question, + register_question_type, +) + + +def choice(qid="bucket", criteria=None, instructions="Which team?", max_options=128): + return build_question( + qid, + "choice", + instructions, + criteria or {"billing": "money", "outage": None, "other": None}, + max_options, + ) + + +def noul(qid="urgent", criteria=None, instructions="Reply within the hour?"): + return build_question(qid, "noul", instructions, criteria, 128) + + +def score(qid="tone", criteria=None, instructions="How angry?", max_options=128): + return build_question( + qid, + "score", + instructions, + criteria or ["calm", "annoyed", "furious"], + max_options, + ) + + +def test_choice_labels_follow_option_order(): + q = choice() + assert q.labels == ("A", "B", "C") + assert [a.name for a in q.options] == ["billing", "outage", "other"] + assert q.options[0].description == "money" + wide = choice(criteria={str(i): None for i in range(26)}) + assert wide.labels[-1] == "Z" + + +def test_choice_prompt(): + q = choice() + assert q.type.prompt(q) == ( + "Question: Which team?\n" + "A: billing - money\n" + "B: outage\n" + "C: other\n" + "Answer with the letter of one option only." + ) + + +def test_choice_answer_shape(): + q = choice() + answer = q.type.answer(q, [0.2, 0.7, 0.1], 0.5) + assert answer == { + "type": "choice", + "choice": "outage", + "probabilities": {"billing": 0.2, "outage": 0.7, "other": 0.1}, + "confidence": 0.35, + } + + +@pytest.mark.parametrize( + "qid,type_name,criteria,match", + [ + ("", "choice", {"x": None, "y": None}, "non-empty"), + ("q", "nope", {"x": None, "y": None}, "unknown question type"), + ("q", "choice", ["x", "y"], "must map option names"), + ("q", "noul", ["yes", "no"], "must be an object"), + ("q", "noul", {"maybe": "same day"}, "must be an object"), + ("q", "score", {"0": "calm", "1": "furious"}, "ordered list of levels"), + ("q", "score", ["calm"], "ordered list of levels"), + ], +) +def test_build_question_rejects(qid, type_name, criteria, match): + with pytest.raises(StructuredDecisionError, match=match): + build_question(qid, type_name, "", criteria, 128) + + +def test_one_option_choice(): + q = choice(criteria={"only": None}) + assert q.labels == ("A",) + assert q.type.answer(q, [1.0], 0.4)["confidence"] == 0.4 + + +def test_option_limit(): + with pytest.raises(StructuredDecisionError, match="at most 2 options"): + choice(criteria={"x": None, "y": None, "z": None}, max_options=2) + with pytest.raises(StructuredDecisionError, match="at most 26 options"): + choice(criteria={str(i): None for i in range(27)}) + + +def test_noul_labels_and_prompt(): + q = noul() + assert q.labels == ("yes", "no") + assert [a.name for a in q.options] == ["yes", "no"] + assert q.type.prompt(q) == ( + "Question: Reply within the hour?\nAnswer with yes or no only." + ) + described = noul(criteria={"true": "same day", "false": "later"}) + assert described.type.prompt(described) == ( + "Question: Reply within the hour?\n" + "yes: same day\n" + "no: later\n" + "Answer with yes or no only." + ) + + +def test_noul_answer_shape(): + q = noul() + answer = q.type.answer(q, [0.8, 0.2], 0.5) + assert answer == { + "type": "noul", + "noul": 0.8, + "probabilities": {"yes": 0.8, "no": 0.2}, + "confidence": 0.4, + } + + +def test_score_labels_are_levels(): + assert score().labels == ("0", "1", "2") + with pytest.raises(StructuredDecisionError, match="at most 10"): + score(criteria=[str(i) for i in range(11)]) + + +def test_score_prompt(): + q = score() + assert q.type.prompt(q) == ( + "Question: How angry?\n" + "0: calm\n" + "1: annoyed\n" + "2: furious\n" + "Answer with the number of one level only." + ) + + +def test_score_answer_shape(): + q = score() + answer = q.type.answer(q, [0.5, 0.25, 0.25], 0.4) + assert answer == { + "type": "score", + "score": 0.75, + "legend": {"0": "calm", "1": "annoyed", "2": "furious"}, + "probabilities": {"0": 0.5, "1": 0.25, "2": 0.25}, + "confidence": 0.2, + } + + +def test_registered_type_plugs_in(): + class BinaryQuestion(QuestionType): + name = "test_binary" + + def parse_options(self, qid: str, criteria: Any) -> list[Option]: + return [Option("yes"), Option("no")] + + def answer( + self, question: Question, probs: list[float], label_mass: float + ) -> dict[str, Any]: + return {"type": self.name, "yes": probs[0]} + + register_question_type(BinaryQuestion) + try: + q = build_question("ok", "test_binary", "Is it fine?", None, 128) + assert q.labels == ("A", "B") + assert q.type.answer(q, [0.9, 0.1], 1.0) == {"type": "test_binary", "yes": 0.9} + with pytest.raises(ValueError, match="already registered"): + register_question_type(BinaryQuestion) + finally: + del QUESTION_TYPES["test_binary"] diff --git a/tests/entrypoints/generate/structured_decisions/test_strategies.py b/tests/entrypoints/generate/structured_decisions/test_strategies.py new file mode 100644 index 0000000000..5dcb8edb30 --- /dev/null +++ b/tests/entrypoints/generate/structured_decisions/test_strategies.py @@ -0,0 +1,222 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +from types import SimpleNamespace +from typing import Any +from unittest.mock import MagicMock + +import pytest +from fastapi import FastAPI, Request +from fastapi.testclient import TestClient + +from vllm.config import ModelConfig +from vllm.engine.protocol import EngineClient +from vllm.entrypoints.generate.structured_decisions.api_router import ( + register_structured_decisions_api_router, +) +from vllm.entrypoints.generate.structured_decisions.protocol import ( + StructuredDecisionRequest, + StructuredDecisionResponse, +) +from vllm.entrypoints.generate.structured_decisions.question_types import LABELS +from vllm.entrypoints.generate.structured_decisions.serving import ( + ServingStructuredDecisions, + state_text, +) +from vllm.entrypoints.generate.structured_decisions.strategies import ( + NextTokenStrategy, + ReadContext, + reply_label_ids, + select_read_strategy, +) +from vllm.entrypoints.openai.models.protocol import BaseModelPath +from vllm.entrypoints.openai.models.serving import OpenAIServingModels +from vllm.entrypoints.serve.exception_handling.register import ( + init_exception_handler, +) +from vllm.logprobs import Logprob +from vllm.outputs import CompletionOutput, RequestOutput +from vllm.renderers.hf import HfRenderer +from vllm.renderers.online_renderer import OnlineRenderer + +TRACE_HEADERS = { + "traceparent": "00-0123456789abcdef0123456789abcdef-0123456789abcdef-01", + "tracestate": "vendor=value", +} + + +def model(architecture: str, logprobs_mode: str = "raw_logprobs") -> Any: + return SimpleNamespace(architecture=architecture, logprobs_mode=logprobs_mode) + + +def test_strategy_selection(): + qwen = "Qwen3ForCausalLM" + assert select_read_strategy(model(qwen)) is NextTokenStrategy + assert select_read_strategy(model(qwen, "processed_logprobs")) is NextTokenStrategy + with pytest.raises(ValueError, match="does not support LlamaForCausalLM"): + select_read_strategy(model("LlamaForCausalLM")) + with pytest.raises(ValueError, match="not raw_logits"): + select_read_strategy(model(qwen, "raw_logits")) + + +def test_route_is_registered_by_default(): + app = FastAPI() + register_structured_decisions_api_router(app) + paths = {getattr(route, "path", None) for route in app.routes} + assert "/v1/systemone" in paths + + +def test_unsupported_model_returns_501(): + app = FastAPI() + app.state.args = SimpleNamespace(log_error_stack=False) + app.state.serving_structured_decisions = None + init_exception_handler(app) + register_structured_decisions_api_router(app) + + with TestClient(app) as client: + response = client.post( + "/v1/systemone", + json={ + "model": "unsupported", + "state": "x", + "questions": {"answer": {"type": "choice", "criteria": {"yes": None}}}, + }, + ) + assert response.status_code == 501 + + +def test_structured_state_keeps_unicode_in_prompt(): + state = {"message": "Français 日本語"} + assert state_text(state) == '{"message": "Français 日本語"}' + + +@pytest.fixture(scope="module") +def qwen(): + from vllm.tokenizers import get_tokenizer + + tokenizer = get_tokenizer("Qwen/Qwen3-0.6B") + prompt_ids = tokenizer.apply_chat_template( + [{"role": "user", "content": "state"}], + add_generation_prompt=True, + enable_thinking=False, + tokenize=True, + return_dict=False, + ) + return tokenizer, prompt_ids + + +def test_labels_start_the_reply(qwen): + tokenizer, prompt_ids = qwen + tail, ids = reply_label_ids(tokenizer, prompt_ids) + assert tokenizer.decode(tail) == "\n\n" + assert [tokenizer.decode([i]) for i in ids] == list(LABELS) + # A noul's and a score's labels are one token here too. + _, ids = reply_label_ids(tokenizer, prompt_ids, ("yes", "no", *"0123456789")) + assert len(set(ids)) == 12 + # After a colon, Qwen writes ":A" as one token, so "A" is not one token. + with pytest.raises(ValueError, match="not one distinct token"): + reply_label_ids(tokenizer, tokenizer.encode("team:")) + + +@pytest.fixture +def serving_and_engine(qwen): + tokenizer, _ = qwen + model_config = ModelConfig(model="Qwen/Qwen3-0.6B", dtype="float32") + renderer = HfRenderer( + SimpleNamespace( + model_config=model_config, + parallel_config=SimpleNamespace(_api_process_rank=0), + ), + tokenizer, + ) + engine = MagicMock(spec=EngineClient) + engine.errored = False + engine.model_config = model_config + engine.renderer = renderer + engine.input_processor = MagicMock() + online_renderer = OnlineRenderer( + model_config, + renderer, + request_logger=None, + chat_template=None, + chat_template_content_format="auto", + ) + + async def generate(prompt, params, request_id, **kwargs): + label_ids = params.logprob_token_ids + yield RequestOutput( + request_id=request_id, + prompt=None, + prompt_token_ids=prompt["prompt_token_ids"], + prompt_logprobs=None, + outputs=[ + CompletionOutput( + index=0, + text="A", + token_ids=[label_ids[0]], + cumulative_logprob=None, + logprobs=[{token_id: Logprob(-1.0) for token_id in label_ids}], + finish_reason="length", + ) + ], + finished=True, + ) + + engine.generate.side_effect = generate + strategy = NextTokenStrategy(ReadContext(engine, online_renderer, None, "auto", {})) + models = OpenAIServingModels( + engine, [BaseModelPath(name=model_config.model, model_path=model_config.model)] + ) + yield ServingStructuredDecisions(models, strategy), engine + renderer.shutdown() + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "cache_salt,tracing_enabled,request_headers", + [ + ("tenant-a", True, {"authorization": "Bearer test", **TRACE_HEADERS}), + ("tenant-b", False, {"authorization": "Bearer test", **TRACE_HEADERS}), + (None, True, {"authorization": "Bearer test", **TRACE_HEADERS}), + (None, True, {}), + ("tenant-direct", True, None), + ], +) +async def test_request_metadata_reaches_every_question( + serving_and_engine, cache_salt, tracing_enabled, request_headers +): + serving, engine = serving_and_engine + engine.is_tracing_enabled.return_value = tracing_enabled + raw_request = None + if request_headers is not None: + raw_request = Request( + { + "type": "http", + "headers": [ + (k.encode(), v.encode()) for k, v in request_headers.items() + ], + } + ) + request = StructuredDecisionRequest( + state={"team": "billing", "language": "English"}, + questions={ + "team": { + "type": "choice", + "instructions": "Which team does the message name?", + "criteria": {"billing": "payments", "shipping": "deliveries"}, + }, + "lang": { + "type": "choice", + "instructions": "Which language does the message name?", + "criteria": {"English": "English text", "French": "French text"}, + }, + }, + cache_salt=cache_salt, + ) + response = await serving.create_decision(request, raw_request) + assert isinstance(response, StructuredDecisionResponse) + assert engine.generate.call_count == 2 + for call in engine.generate.call_args_list: + assert call.args[0].get("cache_salt") == cache_salt + assert call.kwargs["trace_headers"] == ( + TRACE_HEADERS if tracing_enabled and request_headers else None + ) diff --git a/tests/entrypoints/generate/structured_decisions/test_structured_decisions_e2e.py b/tests/entrypoints/generate/structured_decisions/test_structured_decisions_e2e.py new file mode 100644 index 0000000000..a206c56ccc --- /dev/null +++ b/tests/entrypoints/generate/structured_decisions/test_structured_decisions_e2e.py @@ -0,0 +1,260 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import pytest +import requests + +from tests.utils import RemoteOpenAIServer + +MODEL_NAME = "Qwen/Qwen3-0.6B" + + +@pytest.fixture(scope="module") +def server(): + args = [ + "--dtype", + "bfloat16", + "--max-model-len", + "1024", + "--enforce-eager", + "--max-num-seqs", + "32", + "--enable-prefix-caching", + ] + with RemoteOpenAIServer(MODEL_NAME, args) as remote_server: + yield remote_server + + +def post(server, body): + return requests.post(server.url_for("v1/systemone"), json=body) + + +def choice(instructions, *names): + return { + "type": "choice", + "instructions": instructions, + "criteria": dict.fromkeys(names), + } + + +COPY = { + "team": choice( + "Which team does the message name?", "billing", "shipping", "security" + ), + "lang": choice("Which language does the message name?", "English", "French"), +} + + +def test_choice_decision(server): + response = post( + server, + { + "model": MODEL_NAME, + "state": {"team": "billing", "language": "French"}, + "questions": COPY, + "chat_template_kwargs": {"enable_thinking": False}, + }, + ) + assert response.status_code == 200, response.text + body = response.json() + assert body["object"] == "structured_decision" + assert list(body["answers"]) == ["team", "lang"] + team = body["answers"]["team"] + assert team["type"] == "choice" + assert set(team["probabilities"]) == {"billing", "shipping", "security"} + assert sum(team["probabilities"].values()) == pytest.approx(1.0, abs=1e-4) + assert team["confidence"] == pytest.approx( + max(team["probabilities"].values()) * body["diagnostics"]["team"]["label_mass"] + ) + for diag in body["diagnostics"].values(): + assert 0.0 < diag["label_mass"] <= 1.0 + 1e-6 + assert body["usage"]["output_tokens"] == 2 + assert body["usage"]["input_tokens"] > 0 + + +def test_one_option_choice(server): + response = post( + server, + { + "model": MODEL_NAME, + "state": "Team: billing.", + "questions": { + "team": choice("Which team does the message name?", "billing") + }, + "chat_template_kwargs": {"enable_thinking": False}, + }, + ) + assert response.status_code == 200, response.text + body = response.json() + team = body["answers"]["team"] + assert team["choice"] == "billing" + assert team["probabilities"] == {"billing": 1.0} + assert team["confidence"] == pytest.approx( + body["diagnostics"]["team"]["label_mass"] + ) + + +def test_one_option_coin_flip(server): + # A fair coin with only "heads" allowed. The answer is forced, so its + # confidence must come from the model's probability of the label. + response = post( + server, + { + "model": MODEL_NAME, + "state": "I flip a fair coin.", + "questions": {"coin": choice("How did the coin land?", "heads")}, + "chat_template_kwargs": {"enable_thinking": False}, + }, + ) + assert response.status_code == 200, response.text + body = response.json() + coin = body["answers"]["coin"] + assert coin["probabilities"] == {"heads": 1.0} + assert coin["confidence"] == pytest.approx( + body["diagnostics"]["coin"]["label_mass"] + ) + + +@pytest.mark.parametrize( + "state,questions,expected", + [ + ( + {"team": "billing", "language": "French"}, + COPY, + {"team": "billing", "lang": "French"}, + ), + ( + "My card was charged twice for one order.", + { + "card": choice("Does the message mention a card?", "yes", "no"), + "mood": choice("How does the customer feel?", "happy", "angry"), + }, + {"card": "yes", "mood": "angry"}, + ), + ( + "Hello.", + { + "sky": choice( + "What color is the sky on a clear day?", "blue", "green", "red" + ) + }, + {"sky": "blue"}, + ), + ], +) +def test_single_read_answers(server, state, questions, expected): + response = post( + server, + { + "model": MODEL_NAME, + "state": state, + "questions": questions, + "chat_template_kwargs": {"enable_thinking": False}, + }, + ) + assert response.status_code == 200, response.text + answers = response.json()["answers"] + assert {qid: a["choice"] for qid, a in answers.items()} == expected + + +def test_option_limit(server): + body = { + "model": MODEL_NAME, + "state": "Pick option 7.", + "chat_template_kwargs": {"enable_thinking": False}, + } + ok = post( + server, {**body, "questions": {"n": choice("Which?", *map(str, range(26)))}} + ) + assert ok.status_code == 200, ok.text + assert len(ok.json()["answers"]["n"]["probabilities"]) == 26 + over = post( + server, {**body, "questions": {"n": choice("Which?", *map(str, range(27)))}} + ) + assert over.status_code == 400 + assert "at most 26 options" in over.json()["error"]["message"] + + +def test_thinking_is_refused(server): + response = post( + server, + { + "model": MODEL_NAME, + "state": "x", + "questions": {"q": choice("Which?", "a", "b")}, + "chat_template_kwargs": {"enable_thinking": True}, + }, + ) + assert response.status_code == 400 + assert "thinking must be off" in response.json()["error"]["message"] + + +def test_noul_and_score(server): + response = post( + server, + { + "model": MODEL_NAME, + "state": {"ticket": "My card was charged twice for one order."}, + "questions": { + "urgent": {"type": "noul", "instructions": "Is money moving?"}, + "tone": { + "type": "score", + "instructions": "How upset is the customer?", + "criteria": ["calm", "annoyed", "furious"], + }, + }, + "chat_template_kwargs": {"enable_thinking": False}, + }, + ) + assert response.status_code == 200, response.text + body = response.json() + urgent = body["answers"]["urgent"] + assert urgent["type"] == "noul" + assert set(urgent["probabilities"]) == {"yes", "no"} + assert urgent["probabilities"]["yes"] + urgent["probabilities"]["no"] == ( + pytest.approx(1.0, abs=1e-4) + ) + assert urgent["noul"] == urgent["probabilities"]["yes"] + assert urgent["confidence"] == pytest.approx( + max(urgent["probabilities"].values()) + * body["diagnostics"]["urgent"]["label_mass"] + ) + tone = body["answers"]["tone"] + assert tone["type"] == "score" + assert tone["legend"] == {"0": "calm", "1": "annoyed", "2": "furious"} + assert set(tone["probabilities"]) == {"0", "1", "2"} + assert sum(tone["probabilities"].values()) == pytest.approx(1.0, abs=1e-4) + assert tone["score"] == pytest.approx( + sum(i * p for i, p in enumerate(tone["probabilities"].values())) + ) + assert 0 <= tone["score"] <= 2 + assert body["usage"]["output_tokens"] == 2 + + +@pytest.mark.parametrize( + "questions,match", + [ + ( + { + "q": { + "type": "choice", + "criteria": {"a": None, "b": None}, + "depends_on": ["x"], + } + }, + "unknown field(s)", + ), + ( + { + f"q{i}": {"type": "choice", "criteria": {"a": None, "b": None}} + for i in range(65) + }, + "at most 64", + ), + ({"q": {"type": "noul", "criteria": ["yes", "no"]}}, "must be an object"), + ({"q": {"type": "score", "criteria": ["calm"]}}, "ordered list of levels"), + ], +) +def test_validation_errors(server, questions, match): + response = post(server, {"model": MODEL_NAME, "state": "x", "questions": questions}) + assert response.status_code == 400 + assert match in response.json()["error"]["message"] diff --git a/tests/entrypoints/launchers/api_server/_api_server_spawn_workers.py b/tests/entrypoints/launchers/api_server/_api_server_spawn_workers.py deleted file mode 100644 index c8ff969a8a..0000000000 --- a/tests/entrypoints/launchers/api_server/_api_server_spawn_workers.py +++ /dev/null @@ -1,14 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# SPDX-FileCopyrightText: Copyright contributors to the vLLM project -"""Spawn worker targets kept free of heavy imports. - -``multiprocessing`` with the ``spawn`` start method re-imports the module that -defines a process target in the child. Housing these stubs in a stdlib-only -module keeps child startup fast and deterministic, instead of paying a multi- -second ``import vllm`` before the child can run. -""" - - -def exit_before_report_worker(listen_address, sock, args, client_config=None): - """Exit immediately without touching ``actual_address_pipe``.""" - return diff --git a/tests/entrypoints/launchers/api_server/test_api_server_process_manager.py b/tests/entrypoints/launchers/api_server/test_api_server_process_manager.py index 6c579df838..318f57b86f 100644 --- a/tests/entrypoints/launchers/api_server/test_api_server_process_manager.py +++ b/tests/entrypoints/launchers/api_server/test_api_server_process_manager.py @@ -2,6 +2,7 @@ # SPDX-FileCopyrightText: Copyright contributors to the vLLM project import multiprocessing +import os import socket import threading import time @@ -11,14 +12,10 @@ from unittest.mock import patch import pytest import zmq -from tests.entrypoints.launchers.api_server._api_server_spawn_workers import ( - exit_before_report_worker, -) -from vllm.utils.network_utils import make_zmq_socket, split_zmq_path +from vllm.utils.network_utils import make_zmq_listener, make_zmq_socket from vllm.v1.engine.admission_control import SharedAdmissionStats from vllm.v1.utils import ( APIServerProcessManager, - get_engine_client_zmq_addr, wait_for_completion_or_failure, ) @@ -46,36 +43,30 @@ def update_admission_stats_worker(listen_address, sock, args, client_config): args.results.put(stats.get_num_requests()) -# Module-level stub for the gather_actual_addresses test. Must be -# importable by `multiprocessing.spawn` (no closures, no nesting). -def defer_addresses_stub_worker(listen_address, sock, args, client_config): - """Bind ROUTER/PULL with a kernel-assigned port, report the actual - endpoints back via the pipe, then exit.""" +def _adopt_zmq_listener_worker(listen_address, sock, args, client_config): ctx = zmq.Context() + router = make_zmq_socket( + ctx, + client_config["input_address"], + zmq.ROUTER, + bind=True, + listener=client_config["input_listener"], + ) + pull = make_zmq_socket( + ctx, + client_config["output_address"], + zmq.PULL, + bind=True, + listener=client_config["output_listener"], + ) try: - in_sock = make_zmq_socket( - ctx, client_config["input_address"], zmq.ROUTER, bind=True - ) - out_sock = make_zmq_socket( - ctx, client_config["output_address"], zmq.PULL, bind=True - ) - try: - pipe = client_config["actual_address_pipe"] - try: - pipe.send( - { - "input_address": in_sock.getsockopt(zmq.LAST_ENDPOINT).decode(), - "output_address": out_sock.getsockopt( - zmq.LAST_ENDPOINT - ).decode(), - } - ) - finally: - pipe.close() - finally: - in_sock.close(linger=0) - out_sock.close(linger=0) + identity, payload = router.recv_multipart() + assert payload == b"input" + assert pull.recv() == b"output" + router.send_multipart([identity, b"ok"]) finally: + router.close(linger=0) + pull.close(linger=0) ctx.term() @@ -89,15 +80,11 @@ def api_server_args(): "sock": sock, "args": "test_args", # Simple string to avoid pickling issues "num_servers": 3, - "input_addresses": [ - "tcp://127.0.0.1:5001", - "tcp://127.0.0.1:5002", - "tcp://127.0.0.1:5003", + "input_listeners": [ + make_zmq_listener("tcp://127.0.0.1:0", zmq.ROUTER) for _ in range(3) ], - "output_addresses": [ - "tcp://127.0.0.1:6001", - "tcp://127.0.0.1:6002", - "tcp://127.0.0.1:6003", + "output_listeners": [ + make_zmq_listener("tcp://127.0.0.1:0", zmq.PULL) for _ in range(3) ], "stats_update_address": "tcp://127.0.0.1:7000", } @@ -120,6 +107,13 @@ def test_api_server_process_manager_init(api_server_args, with_stats_update): try: # Verify the manager was initialized correctly assert len(manager.processes) == 3 + assert all( + listener.socket.fileno() == -1 + for listener in ( + *args["input_listeners"], + *args["output_listeners"], + ) + ) # Verify all processes are running for proc in manager.processes: @@ -145,6 +139,70 @@ def test_api_server_process_manager_init(api_server_args, with_stats_update): assert not proc.is_alive() +def _assert_endpoint_reserved(address: str) -> None: + """A competing bind must fail while the endpoint has a listener owner.""" + if address.startswith("ipc://"): + competitor = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) + bind_address: str | tuple[str, int] = address.removeprefix("ipc://") + else: + competitor = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + bind_address = ("127.0.0.1", int(address.rsplit(":", 1)[1])) + with competitor, pytest.raises(OSError, match="Address already in use"): + competitor.bind(bind_address) + + +@pytest.mark.parametrize("scheme", ["tcp", "ipc"]) +def test_api_server_child_adopts_inherited_zmq_listeners(tmp_path, scheme): + if scheme == "tcp": + input_path = output_path = "tcp://127.0.0.1:0" + else: + input_path = f"ipc://{tmp_path}/input.sock" + output_path = f"ipc://{tmp_path}/output.sock" + input_listener = make_zmq_listener(input_path, zmq.ROUTER) + output_listener = make_zmq_listener(output_path, zmq.PULL) + input_address = input_listener.address + output_address = output_listener.address + # The supervisor reserves each endpoint as soon as it allocates it. + _assert_endpoint_reserved(input_address) + _assert_endpoint_reserved(output_address) + http_socket = socket.socket() + manager = APIServerProcessManager( + target_server_fn=_adopt_zmq_listener_worker, + listen_address="localhost:8000", + sock=http_socket, + args="test_args", + num_servers=1, + input_listeners=[input_listener], + output_listeners=[output_listener], + ) + + ctx = zmq.Context() + dealer = make_zmq_socket( + ctx, input_address, zmq.DEALER, bind=False, identity=b"test" + ) + push = make_zmq_socket(ctx, output_address, zmq.PUSH, bind=False) + dealer.setsockopt(zmq.RCVTIMEO, 30_000) + try: + assert input_listener.socket.fileno() == -1 + assert output_listener.socket.fileno() == -1 + # After the parent closes its copies, the child's inherited descriptors + # keep both endpoints reserved; closing must not unlink IPC paths. + assert manager.processes[0].is_alive() + _assert_endpoint_reserved(input_address) + _assert_endpoint_reserved(output_address) + dealer.send(b"input") + push.send(b"output") + assert dealer.recv() == b"ok" + manager.processes[0].join(timeout=10) + assert manager.processes[0].exitcode == 0 + finally: + manager.shutdown() + dealer.close(linger=0) + push.close(linger=0) + ctx.term() + http_socket.close() + + @pytest.mark.skip_global_cleanup def test_api_server_processes_share_admission_stats(api_server_args): args = api_server_args.copy() @@ -354,96 +412,7 @@ def test_external_process_monitoring(api_server_args): WORKER_RUNTIME_SECONDS = prev_worker_runtime -@pytest.mark.timeout(60) -def test_gather_actual_addresses_end_to_end(): - """Each child binds ROUTER/PULL with a kernel-picked port and reports - the bound endpoints back via its per-child pipe; the manager surfaces - them via :py:meth:`gather_actual_addresses`.""" - host = "127.0.0.1" - num_servers = 4 - - placeholder_inputs = [ - get_engine_client_zmq_addr(local_only=False, host=host) - for _ in range(num_servers) - ] - placeholder_outputs = [ - get_engine_client_zmq_addr(local_only=False, host=host) - for _ in range(num_servers) - ] - for addr in placeholder_inputs + placeholder_outputs: - assert addr == f"tcp://{host}:0", addr - - sock = socket.socket() - manager = APIServerProcessManager( - listen_address=f"tcp://{host}:0", - sock=sock, - args="test_args", - num_servers=num_servers, - input_addresses=placeholder_inputs, - output_addresses=placeholder_outputs, - target_server_fn=defer_addresses_stub_worker, - ) - - try: - assert len(manager.processes) == num_servers - actual_inputs, actual_outputs = manager.gather_actual_addresses(timeout=15.0) - finally: - manager.shutdown() - time.sleep(0.2) - sock.close() - - assert len(actual_inputs) == num_servers - assert len(actual_outputs) == num_servers - - for addr in actual_inputs + actual_outputs: - scheme, parsed_host, port = split_zmq_path(addr) - assert scheme == "tcp", addr - assert parsed_host == host, addr - assert port and int(port) > 0, addr - - all_addrs = actual_inputs + actual_outputs - assert len(set(all_addrs)) == len(all_addrs), all_addrs - - -@pytest.mark.timeout(30) -def test_gather_actual_addresses_child_crash_before_report(): - """A child that exits before sending its endpoints must surface a - clear ``RuntimeError`` rather than hang or return ``None`` slots.""" - host = "127.0.0.1" - num_servers = 2 - placeholder_inputs = [ - get_engine_client_zmq_addr(local_only=False, host=host) - for _ in range(num_servers) - ] - placeholder_outputs = [ - get_engine_client_zmq_addr(local_only=False, host=host) - for _ in range(num_servers) - ] - - sock = socket.socket() - manager = APIServerProcessManager( - listen_address=f"tcp://{host}:0", - sock=sock, - args="test_args", - num_servers=num_servers, - input_addresses=placeholder_inputs, - output_addresses=placeholder_outputs, - # exit_before_report_worker exits without touching - # ``actual_address_pipe`` — simulates a child that dies before - # reporting its bound addresses. - target_server_fn=exit_before_report_worker, - ) - try: - # Sentinel-first vs pipe-EOF-first both produce "reporting". - with pytest.raises(RuntimeError, match="reporting"): - manager.gather_actual_addresses(timeout=10.0) - finally: - manager.shutdown() - time.sleep(0.2) - sock.close() - - -def test_rust_frontend_launch_log_redacts_credentials(monkeypatch, caplog): +def test_rust_frontend_launch_log_redacts_credentials(monkeypatch): """The Rust frontend command carries every non-default arg as JSON. Credentials must not reach the log line.""" import subprocess as subprocess_mod @@ -469,15 +438,17 @@ def test_rust_frontend_launch_log_redacts_credentials(monkeypatch, caplog): return 0 sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + input_listener = make_zmq_listener("tcp://127.0.0.1:0", zmq.ROUTER) + output_listener = make_zmq_listener("tcp://127.0.0.1:0", zmq.PULL) try: monkeypatch.setattr(subprocess_mod, "Popen", lambda *a, **kw: _FakeProc()) - with caplog.at_level("INFO", logger="vllm.v1.utils"): + with patch("vllm.v1.utils.logger.info") as log_info: RustFrontendProcessManager( binary_path="/nonexistent/vllm-rs", sock=sock, args=args, - input_address="ipc:///tmp/in", - output_address="ipc:///tmp/out", + input_listener=input_listener, + output_listener=output_listener, engine_start_index=0, engine_count=1, data_parallel_size=1, @@ -485,11 +456,13 @@ def test_rust_frontend_launch_log_redacts_credentials(monkeypatch, caplog): finally: sock.close() - message = caplog.text + log_format, *log_args = log_info.call_args.args + message = log_format % tuple(log_args) assert "Launching Rust frontend:" in message assert hf_token not in message assert api_key not in message assert '"hf_token": "***"' in message + assert '"api_key": "***"' in message def test_rust_frontend_uses_config_model_as_model_tag(monkeypatch, caplog, tmp_path): @@ -519,6 +492,8 @@ def test_rust_frontend_uses_config_model_as_model_tag(monkeypatch, caplog, tmp_p return 0 sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + input_listener = make_zmq_listener("tcp://127.0.0.1:0", zmq.ROUTER) + output_listener = make_zmq_listener("tcp://127.0.0.1:0", zmq.PULL) try: monkeypatch.setattr(subprocess_mod, "Popen", lambda *a, **kw: _FakeProc()) with caplog.at_level("INFO", logger="vllm.v1.utils"): @@ -526,8 +501,8 @@ def test_rust_frontend_uses_config_model_as_model_tag(monkeypatch, caplog, tmp_p binary_path="/nonexistent/vllm-rs", sock=sock, args=args, - input_address="ipc:///tmp/in", - output_address="ipc:///tmp/out", + input_listener=input_listener, + output_listener=output_listener, engine_start_index=0, engine_count=1, data_parallel_size=1, @@ -568,14 +543,18 @@ def test_rust_frontend_inherits_grpc_listener(monkeypatch): sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) grpc_sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + input_listener = make_zmq_listener("tcp://127.0.0.1:0", zmq.ROUTER) + output_listener = make_zmq_listener("tcp://127.0.0.1:0", zmq.PULL) + input_fd = input_listener.socket.fileno() + output_fd = output_listener.socket.fileno() try: monkeypatch.setattr(subprocess_mod, "Popen", fake_popen) RustFrontendProcessManager( binary_path="/nonexistent/vllm-rs", sock=sock, args=args, - input_address="ipc:///tmp/in", - output_address="ipc:///tmp/out", + input_listener=input_listener, + output_listener=output_listener, engine_start_index=0, engine_count=1, data_parallel_size=1, @@ -585,7 +564,44 @@ def test_rust_frontend_inherits_grpc_listener(monkeypatch): grpc_fd = grpc_sock.fileno() flag_index = cmd.index("--grpc-listen-fd") assert cmd[flag_index + 1] == str(grpc_fd) - assert list(kwargs["pass_fds"]) == [sock.fileno(), grpc_fd] + assert list(kwargs["pass_fds"]) == [ + sock.fileno(), + input_fd, + output_fd, + grpc_fd, + ] finally: sock.close() grpc_sock.close() + + +def test_rust_frontend_spawn_failure_cleans_up_listeners(tmp_path): + """No child owns the listeners when Popen fails, so the parent removes them.""" + from vllm.entrypoints.launchers.cli_args import make_arg_parser + from vllm.utils.argparse_utils import FlexibleArgumentParser + from vllm.v1.utils import RustFrontendProcessManager + + args = make_arg_parser(FlexibleArgumentParser()).parse_args( + ["--model", "org/model"] + ) + input_listener = make_zmq_listener(f"ipc://{tmp_path}/input.sock", zmq.ROUTER) + output_listener = make_zmq_listener(f"ipc://{tmp_path}/output.sock", zmq.PULL) + sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + try: + with pytest.raises(FileNotFoundError): + RustFrontendProcessManager( + binary_path=str(tmp_path / "missing-vllm-rs"), + sock=sock, + args=args, + input_listener=input_listener, + output_listener=output_listener, + engine_start_index=0, + engine_count=1, + data_parallel_size=1, + ) + finally: + sock.close() + + for listener in (input_listener, output_listener): + assert listener.socket.fileno() == -1 + assert not os.path.exists(listener.address.removeprefix("ipc://")) diff --git a/tests/compile/rocm/__init__.py b/tests/entrypoints/launchers/render/__init__.py similarity index 100% rename from tests/compile/rocm/__init__.py rename to tests/entrypoints/launchers/render/__init__.py diff --git a/tests/entrypoints/launchers/render/test_app_state.py b/tests/entrypoints/launchers/render/test_app_state.py new file mode 100644 index 0000000000..f0f54d9baf --- /dev/null +++ b/tests/entrypoints/launchers/render/test_app_state.py @@ -0,0 +1,84 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +from argparse import Namespace + +import pytest +from starlette.datastructures import State + +import vllm.entrypoints.launchers.render.app_state as app_state_mod +from vllm.config import ModelConfig, VllmConfig +from vllm.entrypoints.launchers.cli_args import make_arg_parser +from vllm.utils.argparse_utils import FlexibleArgumentParser + + +class _CaptureKwargs: + """Stands in for OnlineRenderer/OnlineDerenderer; records init kwargs.""" + + captured: list[dict] + + def __init__(self, **kwargs): + type(self).captured.append(kwargs) + + def warmup(self): + pass + + +def _render_cli_args(*argv: str) -> Namespace: + """Parse ``argv`` the way ``vllm launch render`` does, so new serve flags + carry their defaults instead of needing a hand-maintained ``Namespace``.""" + args = make_arg_parser(FlexibleArgumentParser()).parse_args(list(argv)) + if args.model_tag is not None: + args.model = args.model_tag + return args + + +@pytest.mark.asyncio +async def test_render_app_state_uses_config_resolved_reasoning_parser(monkeypatch): + """gpt_oss defaults its reasoning parser through verify_and_update_config; + the render server must pick up that resolved value like the main API + server does, otherwise harmony markup leaks unparsed into derender output. + """ + model_config = ModelConfig("openai/gpt-oss-20b") + vllm_config = VllmConfig(model_config=model_config) + assert vllm_config.structured_outputs_config.reasoning_parser == "openai_gptoss" + + class _Renderer(_CaptureKwargs): + captured = [] + + class _Derenderer(_CaptureKwargs): + captured = [] + + async def _noop_async(*args, **kwargs): + return None + + monkeypatch.setattr(app_state_mod, "renderer_from_config", lambda cfg: object()) + monkeypatch.setattr(app_state_mod, "OnlineRenderer", _Renderer) + monkeypatch.setattr(app_state_mod, "OnlineDerenderer", _Derenderer) + monkeypatch.setattr(app_state_mod, "ServingTokenization", lambda *a, **kw: object()) + monkeypatch.setattr(app_state_mod, "init_render_state", lambda *a, **kw: None) + monkeypatch.setattr(app_state_mod, "init_endpoint_plugins_state", _noop_async) + + args = _render_cli_args("openai/gpt-oss-20b") + + await app_state_mod.init_render_app_state(vllm_config, State(), args) + + assert _Renderer.captured[0]["reasoning_parser"] == "openai_gptoss" + assert _Derenderer.captured[0]["reasoning_parser"] == "openai_gptoss" + + +def test_explicit_reasoning_parser_flag_wins_over_model_default(): + """An explicit --reasoning-parser must survive VllmConfig construction + for models that define their own default (gpt_oss); the default only + applies when the resolved value is empty.""" + from vllm.engine.arg_utils import AsyncEngineArgs + + engine_args = AsyncEngineArgs( + model="openai/gpt-oss-20b", reasoning_parser="deepseek_r1" + ) + model_config = engine_args.create_model_config() + vllm_config = VllmConfig( + model_config=model_config, + structured_outputs_config=engine_args.create_structured_outputs_config(), + ) + + assert vllm_config.structured_outputs_config.reasoning_parser == "deepseek_r1" diff --git a/tests/entrypoints/launchers/test_balanced_accepts.py b/tests/entrypoints/launchers/test_balanced_accepts.py new file mode 100644 index 0000000000..3ac1025ca8 --- /dev/null +++ b/tests/entrypoints/launchers/test_balanced_accepts.py @@ -0,0 +1,218 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""NoSignalServer(peer_loads=...): API workers sharing one listening socket +accept each connection on the worker with the fewest open connections.""" + +import asyncio +import datetime +import multiprocessing +import os +import socket +import ssl +from collections import Counter + +import pytest +import uvicorn +import uvloop + +from vllm.entrypoints.launchers.launcher import NoSignalServer + +TIMEOUT_S = 30 + + +def _bound_socket() -> socket.socket: + # Like setup_server: bound, not yet listening. + sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) + sock.bind(("127.0.0.1", 0)) + return sock + + +def _app(hold_s: float = 0.0): + pid = str(os.getpid()).encode() + + async def app(scope, receive, send): + if scope["type"] == "lifespan": + while (await receive())["type"] != "lifespan.shutdown": + await send({"type": "lifespan.startup.complete"}) + await send({"type": "lifespan.shutdown.complete"}) + return + await asyncio.sleep(hold_s) + headers = [(b"content-length", str(len(pid)).encode())] + await send({"type": "http.response.start", "status": 200, "headers": headers}) + await send({"type": "http.response.body", "body": pid}) + + return app + + +async def _get(port: int, requests: int = 1, ssl_ctx=None) -> list[bytes]: + """``requests`` requests on one keep-alive connection; returns the bodies.""" + reader, writer = await asyncio.open_connection("127.0.0.1", port, ssl=ssl_ctx) + bodies = [] + for _ in range(requests): + writer.write(b"GET / HTTP/1.1\r\nHost: t\r\n\r\n") + head = await asyncio.wait_for(reader.readuntil(b"\r\n\r\n"), TIMEOUT_S) + assert head.startswith(b"HTTP/1.1 200") + length = int(head.split(b"content-length: ")[1].split(b"\r\n")[0]) + bodies.append(await asyncio.wait_for(reader.readexactly(length), TIMEOUT_S)) + writer.close() + return bodies + + +async def _serve(sock, requests, loads=None, **config_kwargs): + config = uvicorn.Config(_app(), lifespan="on", log_level="warning", **config_kwargs) + loads = loads if loads is not None else [0] + server = NoSignalServer(config, peer_loads=(loads, 0)) + task = asyncio.create_task(server.serve(sockets=[sock])) + while not server.started: + assert not task.done(), task.result() + await asyncio.sleep(0.01) + try: + return await requests(sock.getsockname()[1], server) + finally: + server.should_exit = True + await asyncio.wait_for(task, TIMEOUT_S) + # The listener is closed, nothing is left accepting, and peers no + # longer see this worker as a target. + assert sock.fileno() == -1 + assert not server._accept_tasks + assert loads[0] == 2**31 - 1 + + +def _run(coro, loop_impl): + return uvloop.run(coro) if loop_impl == "uvloop" else asyncio.run(coro) + + +@pytest.mark.parametrize("loop_impl", ["asyncio", "uvloop"]) +def test_requests_and_keep_alive_served(loop_impl): + async def requests(port, server): + return await _get(port), await _get(port, requests=3) + + one, three = _run(_serve(_bound_socket(), requests), loop_impl) + assert one == [str(os.getpid()).encode()] + assert len(three) == 3 + + +def _self_signed(tmp_path): + x509 = pytest.importorskip("cryptography.x509") + from cryptography.hazmat.primitives import hashes, serialization + from cryptography.hazmat.primitives.asymmetric import ec + from cryptography.x509.oid import NameOID + + key = ec.generate_private_key(ec.SECP256R1()) + name = x509.Name([x509.NameAttribute(NameOID.COMMON_NAME, "127.0.0.1")]) + now = datetime.datetime.now(datetime.UTC) + cert = ( + x509.CertificateBuilder() + .subject_name(name) + .issuer_name(name) + .public_key(key.public_key()) + .serial_number(x509.random_serial_number()) + .not_valid_before(now - datetime.timedelta(minutes=1)) + .not_valid_after(now + datetime.timedelta(hours=1)) + .sign(key, hashes.SHA256()) + ) + keyfile, certfile = tmp_path / "key.pem", tmp_path / "cert.pem" + keyfile.write_bytes( + key.private_bytes( + serialization.Encoding.PEM, + serialization.PrivateFormat.PKCS8, + serialization.NoEncryption(), + ) + ) + certfile.write_bytes(cert.public_bytes(serialization.Encoding.PEM)) + return str(keyfile), str(certfile) + + +def test_tls_handshake_failure_keeps_serving(tmp_path): + keyfile, certfile = _self_signed(tmp_path) + client_ctx = ssl.create_default_context() + client_ctx.check_hostname = False + client_ctx.verify_mode = ssl.CERT_NONE + + async def requests(port, server): + # A plain-text client fails the handshake; the next TLS client is served. + reader, writer = await asyncio.open_connection("127.0.0.1", port) + writer.write(b"GET / HTTP/1.1\r\nHost: t\r\n\r\n") + await asyncio.wait_for(reader.read(), TIMEOUT_S) + writer.close() + return await _get(port, ssl_ctx=client_ctx) + + bodies = uvloop.run( + _serve(_bound_socket(), requests, ssl_keyfile=keyfile, ssl_certfile=certfile) + ) + assert bodies == [str(os.getpid()).encode()] + + +def test_busier_worker_still_serves(): + # This worker is index 0; peer 1 claims no connections but never accepts. + loads = [0, 0] + + async def requests(port, server): + reader, writer = await asyncio.open_connection("127.0.0.1", port) + bodies = await _get(port) + assert loads[0] >= 1 # the held connection was published + writer.close() + return bodies + + assert uvloop.run(_serve(_bound_socket(), requests, loads)) == [ + str(os.getpid()).encode() + ] + + +def _worker(sock, loads, index, ready, stop): + async def main(): + config = uvicorn.Config(_app(hold_s=1), lifespan="on", log_level="warning") + server = NoSignalServer(config, peer_loads=(loads, index)) + task = asyncio.create_task(server.serve(sockets=[sock])) + while not server.started: + await asyncio.sleep(0.01) + ready.release() + await asyncio.get_running_loop().run_in_executor(None, stop.wait) + server.should_exit = True + await task + + # API workers run on uvloop, whose own server accepts a whole burst at once. + uvloop.run(main()) + + +@pytest.mark.skipif((os.cpu_count() or 1) < 8, reason="needs idle cores per worker") +@pytest.mark.parametrize("arrival_s", [0, 1]) +def test_connections_balanced_over_workers(arrival_s): + """Both a burst and a steady stream of connections end up balanced; with + the event loop's own server one worker takes most of either.""" + workers, connections = 4, 400 + sock = _bound_socket() + ctx = multiprocessing.get_context("spawn") + ready, stop = ctx.Semaphore(0), ctx.Event() + loads = ctx.RawArray("i", workers) + procs = [ + ctx.Process(target=_worker, args=(sock, loads, i, ready, stop)) + for i in range(workers) + ] + for p in procs: + p.start() + try: + for _ in range(workers): + assert ready.acquire(timeout=120) + + async def get_after(delay): + await asyncio.sleep(delay) + return await _get(sock.getsockname()[1]) + + async def arrive(): + delays = [arrival_s * i / connections for i in range(connections)] + results = await asyncio.gather(*(get_after(d) for d in delays)) + return Counter(body for (body,) in results) + + counts = uvloop.run(arrive()) + assert sum(counts.values()) == connections + # Fair share is 100. + assert max(counts.values()) <= 5 * connections // (4 * workers), counts + finally: + stop.set() + for p in procs: + p.join(30) + if p.is_alive(): + p.kill() + sock.close() diff --git a/tests/entrypoints/launchers/test_cli_args.py b/tests/entrypoints/launchers/test_cli_args.py index 6b9d496822..84279e1555 100644 --- a/tests/entrypoints/launchers/test_cli_args.py +++ b/tests/entrypoints/launchers/test_cli_args.py @@ -242,7 +242,7 @@ def test_rust_grpc_port_requires_serve_subcommand(monkeypatch): validate_parsed_serve_args(args) -def test_enable_auto_choice_passes_without_tool_call_parser(serve_parser): +def test_enable_auto_choice_fails_without_tool_call_parser(serve_parser): """Ensure validation fails if tool choice is enabled with no call parser.""" # If we enable-auto-tool-choice, explode with no tool-call-parser args = serve_parser.parse_args(args=["--enable-auto-tool-choice"]) @@ -262,17 +262,23 @@ def test_enable_auto_choice_passes_with_tool_call_parser(serve_parser): validate_parsed_serve_args(args) -def test_enable_auto_choice_fails_with_enable_reasoning(serve_parser): - """Ensure validation fails if reasoning is enabled with auto tool choice.""" +def test_auto_tool_choice_with_reasoning_parser_passes(serve_parser): + """Auto tool choice combined with a reasoning parser is supported. + + See docs/features/tool_calling.md, which documents using + ``--tool-call-parser`` together with ``--reasoning-parser``. This + combination must pass validation as long as a tool call parser is set. + """ args = serve_parser.parse_args( args=[ "--enable-auto-tool-choice", + "--tool-call-parser", + "mistral", "--reasoning-parser", "deepseek_r1", ] ) - with pytest.raises(TypeError): - validate_parsed_serve_args(args) + validate_parsed_serve_args(args) def test_passes_with_reasoning_parser(serve_parser): diff --git a/tests/entrypoints/launchers/test_dp_supervisor.py b/tests/entrypoints/launchers/test_dp_supervisor.py index c10f7ee698..174157f12a 100644 --- a/tests/entrypoints/launchers/test_dp_supervisor.py +++ b/tests/entrypoints/launchers/test_dp_supervisor.py @@ -630,7 +630,7 @@ async def _poll_until_api_server_running( timeout=request_timeout, ): return - except (aiohttp.ClientError, asyncio.TimeoutError) as exc: + except (TimeoutError, aiohttp.ClientError) as exc: last_exc = exc print("Test detected not started yet, sleeping for 1s") remaining_s = deadline - time.monotonic() diff --git a/tests/entrypoints/launchers/test_run_batch.py b/tests/entrypoints/launchers/test_run_batch.py index f7a2505807..1376eb0645 100644 --- a/tests/entrypoints/launchers/test_run_batch.py +++ b/tests/entrypoints/launchers/test_run_batch.py @@ -1071,17 +1071,32 @@ def _make_aiohttp_put_session(status: int = 200, body_text: str = ""): @pytest.mark.asyncio -async def test_upload_data_uploads_once_on_success(): - """A successful upload must not be retried (regression guard).""" - session = _make_aiohttp_put_session(status=200) - with patch( - "vllm.entrypoints.launchers.run_batch.aiohttp.ClientSession", - return_value=session, +@pytest.mark.parametrize("status", [200, 201, 204]) +@pytest.mark.parametrize("from_file", [False, True]) +async def test_upload_data_uploads_once_on_success(status, from_file, tmp_path): + """Successful PUT responses must not retry either upload path.""" + session = _make_aiohttp_put_session(status=status) + data_or_file = "payload" + if from_file: + path = tmp_path / "output.jsonl" + path.write_text(data_or_file, encoding="utf-8") + data_or_file = str(path) + + with ( + patch( + "vllm.entrypoints.launchers.run_batch.aiohttp.ClientSession", + return_value=session, + ), + patch( + "vllm.entrypoints.launchers.run_batch.asyncio.sleep", + new_callable=AsyncMock, + ) as sleep, ): await upload_data( - "https://example.com/output.jsonl", "payload", from_file=False + "https://example.com/output.jsonl", data_or_file, from_file=from_file ) assert session.put.call_count == 1 + sleep.assert_not_awaited() @pytest.mark.asyncio diff --git a/tests/entrypoints/launchers/test_shutdown.py b/tests/entrypoints/launchers/test_shutdown.py index f03afbff09..2a78b4e80e 100644 --- a/tests/entrypoints/launchers/test_shutdown.py +++ b/tests/entrypoints/launchers/test_shutdown.py @@ -285,7 +285,7 @@ async def test_wait_timeout_completes_requests(): sigterm_sent.set() await asyncio.wait_for(request_task, timeout=_SHUTDOWN_DETECTION_TIMEOUT) - except asyncio.TimeoutError: + except TimeoutError: pass finally: state.stop_requesting = True @@ -463,7 +463,7 @@ async def test_abort_timeout_fails_inflight_requests(): try: await asyncio.wait_for(request_task, timeout=5) - except asyncio.TimeoutError: + except TimeoutError: pass finally: state.stop_requesting = True @@ -592,7 +592,7 @@ async def test_multi_api_server_shutdown(): try: await asyncio.wait_for(request_task, timeout=_SHUTDOWN_DETECTION_TIMEOUT) - except asyncio.TimeoutError: + except TimeoutError: pass finally: state.stop_requesting = True diff --git a/tests/entrypoints/multimodal/llm/test_mm_processor_kwargs.py b/tests/entrypoints/multimodal/llm/test_mm_processor_kwargs.py index c03efa8f10..55990ed1dd 100644 --- a/tests/entrypoints/multimodal/llm/test_mm_processor_kwargs.py +++ b/tests/entrypoints/multimodal/llm/test_mm_processor_kwargs.py @@ -212,7 +212,7 @@ def test_preprocess_cmpl_applies_mm_processor_kwargs_to_renderer( llm.renderer = renderer monkeypatch.setattr( - "vllm.entrypoints.offline_utils.parse_model_prompt", + "vllm.entrypoints.common.offline.parse_model_prompt", lambda _model_config, parsed_prompt: parsed_prompt, ) @@ -246,7 +246,7 @@ def test_preprocess_cmpl_keeps_prompt_mm_processor_kwargs_when_no_override( llm.renderer = renderer monkeypatch.setattr( - "vllm.entrypoints.offline_utils.parse_model_prompt", + "vllm.entrypoints.common.offline.parse_model_prompt", lambda _model_config, parsed_prompt: parsed_prompt, ) diff --git a/tests/entrypoints/openai/chat_completion/test_batched_chat_completions.py b/tests/entrypoints/openai/chat_completion/test_batched_chat_completions.py index 946615f971..41c74bb5ce 100644 --- a/tests/entrypoints/openai/chat_completion/test_batched_chat_completions.py +++ b/tests/entrypoints/openai/chat_completion/test_batched_chat_completions.py @@ -10,6 +10,7 @@ import pytest from tests.entrypoints.openai.chat_completion.test_serving_chat import ( BASE_MODEL_PATHS, + CHAT_TEMPLATE, MockHFConfig, MockModelConfig, _build_renderer, @@ -24,10 +25,13 @@ from vllm.entrypoints.openai.chat_completion.batch_serving import ( ) from vllm.entrypoints.openai.chat_completion.protocol import ( BatchChatCompletionRequest, + ChatCompletionRequest, ) +from vllm.entrypoints.openai.chat_completion.serving import OpenAIServingChat from vllm.entrypoints.openai.models.serving import OpenAIServingModels from vllm.exceptions import VLLMValidationError from vllm.outputs import CompletionOutput, RequestOutput +from vllm.reasoning import ReasoningParserManager from vllm.renderers.online_renderer import OnlineRenderer from vllm.v1.engine.async_llm import AsyncLLM @@ -454,3 +458,101 @@ async def test_batched_parser_receives_each_prompt_prefix() -> None: ) assert prefixes == [[1], [2]] + + +REASONING_PARSER = "kimi_k3" + + +@pytest.mark.asyncio +@pytest.mark.skip_global_cleanup +async def test_batched_reasoning_wiring_matches_single_chat() -> None: + """`/v1/chat/completions/batch` must hand the engine the same reasoning + wiring as `/v1/chat/completions`. + + Both `reasoning_ended` and `reasoning_parser_kwargs` are carried on the + request into `EngineCoreRequest`, where + `StructuredOutputManager._get_reasoner()` rebuilds the request-local + reasoning parser from the forwarded `chat_template_kwargs` -- "so the + structured-output gate observes the same template kwargs used by the + frontend". A divergence makes a template-kwarg-driven parser fall back to + its default (`thinking` on) for a request that disabled thinking, so + `is_reasoning_end()` keeps reporting False, the structured-output gate + never opens, and the grammar is silently never applied. + + Both frontends run over one and the same model and renderer, so the only + thing that can differ between the two calls is the serving code itself. + """ + mock_model_config = MockModelConfig() + shared_renderer = _build_renderer(mock_model_config) + + def _build_serving(cls): + mock_engine = MagicMock(spec=AsyncLLM) + mock_engine.model_config = mock_model_config + mock_engine.errored = False + mock_engine.input_processor = MagicMock() + mock_engine.renderer = shared_renderer + mock_engine.generate.side_effect = lambda *args, **kwargs: _generator(0, "ok") + + models = OpenAIServingModels(mock_engine, BASE_MODEL_PATHS) + online_renderer = OnlineRenderer( + model_config=mock_model_config, + renderer=mock_engine.renderer, + request_logger=None, + chat_template=CHAT_TEMPLATE, + chat_template_content_format="auto", + reasoning_parser=REASONING_PARSER, + ) + return mock_engine, cls( + mock_engine, + models, + response_role="assistant", + online_renderer=online_renderer, + chat_template=CHAT_TEMPLATE, + chat_template_content_format="auto", + request_logger=None, + reasoning_parser=REASONING_PARSER, + ) + + conversation = [{"role": "user", "content": "What is the capital of France?"}] + chat_template_kwargs = {"enable_thinking": False} + + single_engine, single_serving = _build_serving(OpenAIServingChat) + await single_serving.create_chat_completion( + ChatCompletionRequest( + model=MOCK_MODEL_NAME, + messages=conversation, + chat_template_kwargs=chat_template_kwargs, + ) + ) + single_call = single_engine.generate.call_args_list[-1] + + batch_engine, batch_serving = _build_serving(OpenAIServingChatBatch) + await batch_serving.create_batch_chat_completion( + BatchChatCompletionRequest( + model=MOCK_MODEL_NAME, + messages=[conversation], + chat_template_kwargs=chat_template_kwargs, + ) + ) + batch_call = batch_engine.generate.call_args_list[-1] + + # Guard against a vacuous comparison: the reference path must be wired up. + assert single_call.kwargs["reasoning_ended"] is not None + assert single_call.kwargs["reasoning_parser_kwargs"] is not None + + assert batch_call.kwargs["reasoning_ended"] == single_call.kwargs["reasoning_ended"] + assert ( + batch_call.kwargs["reasoning_parser_kwargs"] + == single_call.kwargs["reasoning_parser_kwargs"] + ) + + # ...and the forwarded kwargs must actually turn thinking off for the + # engine-side parser, i.e. the structured-output gate opens right away. + engine_chat_template_kwargs = batch_call.kwargs["reasoning_parser_kwargs"][ + "chat_template_kwargs" + ] + reasoner = ReasoningParserManager.get_reasoning_parser(REASONING_PARSER)( + tokenizer=batch_engine.renderer.tokenizer, + chat_template_kwargs=engine_chat_template_kwargs, + ) + assert reasoner.is_reasoning_end([]) is True diff --git a/tests/entrypoints/openai/chat_completion/test_chat.py b/tests/entrypoints/openai/chat_completion/test_chat.py index 5541b605d9..c246482d8f 100644 --- a/tests/entrypoints/openai/chat_completion/test_chat.py +++ b/tests/entrypoints/openai/chat_completion/test_chat.py @@ -16,6 +16,7 @@ from openai import BadRequestError from tests.utils import RemoteOpenAIServer from vllm.entrypoints.openai.chat_completion.protocol import ( + BatchChatCompletionRequest, ChatCompletionRequest, ) from vllm.exceptions import VLLMValidationError @@ -1012,6 +1013,22 @@ def test_chat_completion_request_n_parameter_default(): assert sampling_params.n == 1, f"Expected n=1 (default), got n={sampling_params.n}" +@pytest.mark.parametrize( + ("request_cls", "messages"), + [ + (ChatCompletionRequest, [{"role": "user", "content": "Hello"}]), + (BatchChatCompletionRequest, [[{"role": "user", "content": "Hello"}]]), + ], +) +def test_null_top_logprobs_is_same_as_omitted(request_cls, messages): + """`top_logprobs: null` must not switch off `logprobs: true`.""" + request = request_cls.model_validate( + {"messages": messages, "logprobs": True, "top_logprobs": None} + ) + + assert request.top_logprobs == 0 + + def test_chat_completion_request_accepts_model_specific_reasoning_effort(): request = ChatCompletionRequest( model="test-model", diff --git a/tests/entrypoints/openai/chat_completion/test_chat_error.py b/tests/entrypoints/openai/chat_completion/test_chat_error.py index baa8bcec59..e467d1bcc5 100644 --- a/tests/entrypoints/openai/chat_completion/test_chat_error.py +++ b/tests/entrypoints/openai/chat_completion/test_chat_error.py @@ -1,6 +1,7 @@ # SPDX-License-Identifier: Apache-2.0 # SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import json from dataclasses import dataclass, field from typing import Any from unittest.mock import AsyncMock, MagicMock, patch @@ -642,7 +643,15 @@ def test_batch_structural_tag_response_format_invalid(format_value): ) -@pytest.mark.parametrize("structural_tag", ["not json", ""]) +@pytest.mark.parametrize( + "structural_tag", + [ + "not json", + "", + # json.loads raises a plain ValueError, not JSONDecodeError + '{"type": "structural_tag", "x": ' + "1" * 5000 + "}", + ], +) def test_structured_outputs_structural_tag_invalid(structural_tag): """Malformed direct structured_outputs structural tags should be rejected.""" with pytest.raises( @@ -656,6 +665,25 @@ def test_structured_outputs_structural_tag_invalid(structural_tag): ) +@pytest.mark.parametrize("field", ["response_format", "structured_outputs"]) +def test_deeply_nested_structural_tag_rejected(field): + """Structural tags are compiled during request parsing, so a deeply nested + one must be rejected there instead of blocking the server while converting, + and reported as too deep rather than malformed.""" + schema = '{"type": "array", "items": ' * 200 + "{}" + "}" * 200 + tag = ( + '{"type": "structural_tag", "format": {"type": "json_schema", ' + f'"json_schema": {schema}}}}}' + ) + value = json.loads(tag) if field == "response_format" else {"structural_tag": tag} + with pytest.raises(VLLMValidationError, match="nested too deeply"): + ChatCompletionRequest( + model=MODEL_NAME, + messages=[{"role": "user", "content": "hello"}], + **{field: value}, + ) + + @pytest.mark.parametrize("field_name", ["prompt_logprobs", "top_logprobs"]) def test_non_numeric_logprobs_rejected(field_name): """A non-numeric logprobs value must be a clean 400 validation error, not a diff --git a/tests/entrypoints/openai/chat_completion/test_completion_with_function_calling.py b/tests/entrypoints/openai/chat_completion/test_completion_with_function_calling.py index 61fd2f1576..72442670a8 100644 --- a/tests/entrypoints/openai/chat_completion/test_completion_with_function_calling.py +++ b/tests/entrypoints/openai/chat_completion/test_completion_with_function_calling.py @@ -154,6 +154,40 @@ async def client(server): yield async_client +@pytest.fixture(scope="module") +def watermarked_server(): + args = [ + "--dtype", + "half", + "--enable-auto-tool-choice", + "--structured-outputs-config.backend", + "xgrammar", + "--tool-call-parser", + "hermes", + "--reasoning-parser", + "qwen3", + "--gpu-memory-utilization", + "0.4", + "--enforce-eager", + "--watermark-config", + '{"algorithm":"gumbel","key":42}', + ] + ROCM_EXTRA_ARGS + + with RemoteOpenAIServer(MODEL_NAME, args) as remote_server: + yield remote_server + + +@pytest.fixture(params=["server", "watermarked_server"], ids=["plain", "watermarked"]) +def tool_use_server(request: pytest.FixtureRequest): + return request.getfixturevalue(request.param) + + +@pytest_asyncio.fixture +async def tool_use_client(tool_use_server): + async with tool_use_server.get_async_client() as async_client: + yield async_client + + @pytest.mark.asyncio @pytest.mark.parametrize("model_name", [MODEL_NAME]) @pytest.mark.parametrize("stream", [True, False]) @@ -167,12 +201,13 @@ async def client(server): ) @pytest.mark.parametrize("enable_thinking", [True, False]) async def test_function_tool_use( - client: openai.AsyncOpenAI, + tool_use_client: openai.AsyncOpenAI, model_name: str, stream: bool, tool_choice: str | dict, enable_thinking: bool, ): + client = tool_use_client if not stream: # Non-streaming test chat_completion = await client.chat.completions.create( diff --git a/tests/entrypoints/openai/completion/test_completion_text_offset.py b/tests/entrypoints/openai/completion/test_completion_text_offset.py new file mode 100644 index 0000000000..c8ba7df62f --- /dev/null +++ b/tests/entrypoints/openai/completion/test_completion_text_offset.py @@ -0,0 +1,97 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import json + +import pytest + +from vllm.entrypoints.generate.base.protocol import RequestResponseMetadata +from vllm.entrypoints.openai.completion.protocol import CompletionRequest +from vllm.entrypoints.openai.completion.serving import OpenAIServingCompletion +from vllm.logprobs import Logprob +from vllm.outputs import CompletionOutput, RequestOutput + +TOKENS = [(1, "abc"), (2, " de"), (3, "!")] +# Text deltas as the detokenizer streams them with stop=["\n\n"]: one char is +# held back until the request finishes, so the text lags the tokens. +TEXT_DELTAS = ["ab", "c d", "e!"] + + +def _logprobs(i: int) -> dict[int, Logprob]: + token_id, token = TOKENS[i] + return {token_id: Logprob(logprob=-1.0, rank=1, decoded_token=token)} + + +def _output(text: str, idxs: list[int], finished: bool) -> RequestOutput: + return RequestOutput( + request_id="r", + prompt="Hi", + prompt_token_ids=[9], + prompt_logprobs=None, + outputs=[ + CompletionOutput( + index=0, + text=text, + token_ids=[TOKENS[i][0] for i in idxs], + cumulative_logprob=None, + logprobs=[_logprobs(i) for i in idxs], + finish_reason="stop" if finished else None, + ) + ], + finished=finished, + ) + + +def _serving() -> OpenAIServingCompletion: + serving = OpenAIServingCompletion.__new__(OpenAIServingCompletion) + serving.enable_prompt_tokens_details = False + serving.enable_force_include_usage = False + serving.enable_per_request_metrics = False + serving.system_fingerprint = None + serving.return_tokens_as_token_ids = False + return serving + + +def _request(stream: bool) -> CompletionRequest: + return CompletionRequest( + model="m", prompt="Hi", max_tokens=8, logprobs=0, stop=["\n\n"], stream=stream + ) + + +@pytest.mark.asyncio +async def test_stream_text_offset_matches_non_stream_with_stop_holdback(): + async def results(): + for i, text in enumerate(TEXT_DELTAS): + yield 0, _output(text, [i], finished=i == len(TEXT_DELTAS) - 1) + + serving = _serving() + stream_offsets = [] + async for line in serving.completion_stream_generator( + _request(stream=True), + [{"prompt": "Hi"}], + results(), + "r", + 0, + "m", + num_prompts=1, + tokenizer=None, + request_metadata=RequestResponseMetadata(request_id="r"), + ): + data = line.removeprefix("data: ").strip() + if data == "[DONE]": + continue + for choice in json.loads(data)["choices"]: + stream_offsets += choice["logprobs"]["text_offset"] + + response = serving.request_output_to_completion_response( + [_output("".join(TEXT_DELTAS), [0, 1, 2], finished=True)], + _request(stream=False), + "r", + 0, + "m", + None, + RequestResponseMetadata(request_id="r"), + ) + assert response.choices[0].logprobs is not None + assert response.choices[0].logprobs.text_offset == [0, 3, 6] + assert stream_offsets == [0, 3, 6] diff --git a/tests/entrypoints/openai/completion/test_tensorizer_entrypoint.py b/tests/entrypoints/openai/completion/test_tensorizer_entrypoint.py index 29c0c2dc8f..813b64b2c5 100644 --- a/tests/entrypoints/openai/completion/test_tensorizer_entrypoint.py +++ b/tests/entrypoints/openai/completion/test_tensorizer_entrypoint.py @@ -13,13 +13,11 @@ from tests.utils import RemoteOpenAIServer from vllm.engine.arg_utils import EngineArgs from vllm.model_executor.model_loader.tensorizer import ( TensorizerConfig, - tensorize_lora_adapter, tensorize_vllm_model, ) from vllm.platforms import current_platform MODEL_NAME = "unsloth/llama-3.2-1b-Instruct" -LORA_PATH = "davzoku/finqa_adapter_1b" def _cleanup(): @@ -44,11 +42,10 @@ def model_uri(tmp_dir): @pytest.fixture(scope="module") -def tensorize_model_and_lora(tmp_dir, model_uri): - tensorizer_config = TensorizerConfig(tensorizer_uri=model_uri, lora_dir=tmp_dir) +def tensorize_model(model_uri): + tensorizer_config = TensorizerConfig(tensorizer_uri=model_uri) args = EngineArgs(model=MODEL_NAME) - tensorize_lora_adapter(LORA_PATH, tensorizer_config) tensorize_vllm_model(args, tensorizer_config) # Manually invoke a _cleanup() here, as the cleanup() @@ -59,7 +56,7 @@ def tensorize_model_and_lora(tmp_dir, model_uri): @pytest.fixture(scope="module") -def server(model_uri, tensorize_model_and_lora): +def server(model_uri, tensorize_model): # In this case, model_uri is a directory with a model.tensors # file and all necessary model artifacts, particularly a # HF `config.json` file. In this case, Tensorizer can infer the @@ -72,7 +69,6 @@ def server(model_uri, tensorize_model_and_lora): "tensorizer", "--served-model-name", MODEL_NAME, - "--enable-lora", ] if current_platform.is_rocm(): args += ["--attention-backend", "TRITON_ATTN"] diff --git a/tests/entrypoints/openai/responses/test_sampling_params.py b/tests/entrypoints/openai/responses/test_sampling_params.py index 6ede3f1f7f..24d54b8960 100644 --- a/tests/entrypoints/openai/responses/test_sampling_params.py +++ b/tests/entrypoints/openai/responses/test_sampling_params.py @@ -29,6 +29,7 @@ class TestResponsesRequestSamplingParams: temperature=0.8, top_p=0.95, top_k=50, + min_p=0.1, max_output_tokens=100, ) @@ -37,8 +38,20 @@ class TestResponsesRequestSamplingParams: assert sampling_params.temperature == 0.8 assert sampling_params.top_p == 0.95 assert sampling_params.top_k == 50 + assert sampling_params.min_p == 0.1 assert sampling_params.max_tokens == 100 + def test_min_p_default(self): + """Test that min_p defaults to 0.0 when unset (parity with top_k).""" + request = ResponsesRequest( + model="test-model", + input="test input", + ) + + sampling_params = request.to_sampling_params(default_max_tokens=1000) + + assert sampling_params.min_p == 0.0 + def test_extra_sampling_params(self): """Test extra sampling parameters are correctly mapped.""" request = ResponsesRequest( @@ -47,6 +60,7 @@ class TestResponsesRequestSamplingParams: repetition_penalty=1.2, seed=42, stop=["END", "STOP"], + stop_token_ids=[42, 43], ignore_eos=True, vllm_xargs={"custom": "value"}, ) @@ -56,6 +70,7 @@ class TestResponsesRequestSamplingParams: assert sampling_params.repetition_penalty == 1.2 assert sampling_params.seed == 42 assert sampling_params.stop == ["END", "STOP"] + assert sampling_params.stop_token_ids == [42, 43] assert sampling_params.ignore_eos is True assert sampling_params.extra_args == {"custom": "value"} @@ -84,6 +99,20 @@ class TestResponsesRequestSamplingParams: assert sampling_params.stop == [] # Empty list assert sampling_params.extra_args == {} # Empty dict + def test_null_top_logprobs_is_same_as_omitted(self): + """`top_logprobs: null` must not drop the requested output logprobs.""" + request = ResponsesRequest.model_validate( + { + "input": "test input", + "include": ["message.output_text.logprobs"], + "top_logprobs": None, + } + ) + + sampling_params = request.to_sampling_params(default_max_tokens=1000) + + assert sampling_params.logprobs == 0 + def test_seed_bounds_validation(self): """Test that seed values outside torch.long bounds are rejected.""" # Test seed below minimum diff --git a/tests/entrypoints/openai/test_openai_schema.py b/tests/entrypoints/openai/test_openai_schema.py index 18b01b85a8..a405550c4c 100644 --- a/tests/entrypoints/openai/test_openai_schema.py +++ b/tests/entrypoints/openai/test_openai_schema.py @@ -7,6 +7,7 @@ import pytest import schemathesis from hypothesis import HealthCheck, settings from schemathesis import GenerationMode +from schemathesis.checks import not_a_server_error from schemathesis.config import ( ChecksConfig, CoveragePhaseConfig, @@ -161,8 +162,13 @@ def test_openapi_stateless(case: schemathesis.Case): }.get(key, DEFAULT_TIMEOUT_SECONDS) # No need to verify SSL certificate for localhost - case.call_and_validate( + systemone = case.operation.path == "/v1/systemone" + response = case.call_and_validate( verify=False, timeout=timeout, headers={"Content-Type": "application/json"}, + excluded_checks=[not_a_server_error] if systemone else None, ) + if systemone: + # SmolVLM does not support structured decisions. + assert response.status_code < 500 or response.status_code == 501 diff --git a/tests/entrypoints/openai/test_penalty_default_resolution.py b/tests/entrypoints/openai/test_penalty_default_resolution.py index 2c39f5bef6..23c57241b7 100644 --- a/tests/entrypoints/openai/test_penalty_default_resolution.py +++ b/tests/entrypoints/openai/test_penalty_default_resolution.py @@ -2,7 +2,8 @@ # SPDX-FileCopyrightText: Copyright contributors to the vLLM project """Unit tests for presence_penalty / frequency_penalty resolution from -default_sampling_params in ChatCompletionRequest and CompletionRequest. +default_sampling_params in ChatCompletionRequest, BatchChatCompletionRequest +and CompletionRequest. Regression test for https://github.com/vllm-project/vllm/issues/50767: these two penalties defaulted to 0.0 (not None) and to_sampling_params() @@ -12,6 +13,7 @@ discarded on the /v1/chat/completions and /v1/completions endpoints. """ from vllm.entrypoints.openai.chat_completion.protocol import ( + BatchChatCompletionRequest, ChatCompletionRequest, ) from vllm.entrypoints.openai.completion.protocol import ( @@ -29,6 +31,14 @@ def _chat(**kwargs): ) +def _batch(**kwargs): + conversation = [{"role": "user", "content": "hi"}] + request = BatchChatCompletionRequest( + model="test-model", messages=[conversation], **kwargs + ) + return request.to_chat_completion_request(conversation) + + class TestChatCompletionPenaltyDefaults: def test_defaults_applied_when_client_omits(self): """Server-default penalties are applied when the client sends none.""" @@ -49,6 +59,22 @@ class TestChatCompletionPenaltyDefaults: assert sp.frequency_penalty == 0.0 +class TestBatchChatCompletionPenaltyDefaults: + def test_defaults_applied_when_client_omits(self): + """Server-default penalties reach every conversation of a batch.""" + sp = _batch().to_sampling_params(100, _DEFAULTS) + assert sp.presence_penalty == 1.5 + assert sp.frequency_penalty == 0.5 + + def test_client_value_overrides_default(self): + """An explicit client penalty, even 0.0, wins over the server default.""" + sp = _batch(presence_penalty=0.0, frequency_penalty=0.2).to_sampling_params( + 100, _DEFAULTS + ) + assert sp.presence_penalty == 0.0 + assert sp.frequency_penalty == 0.2 + + class TestCompletionPenaltyDefaults: def test_defaults_applied_when_client_omits(self): sp = CompletionRequest(model="test-model", prompt="hi").to_sampling_params( diff --git a/tests/entrypoints/openai/test_watermarking.py b/tests/entrypoints/openai/test_watermarking.py index 0425b86a02..e6a750eeb7 100644 --- a/tests/entrypoints/openai/test_watermarking.py +++ b/tests/entrypoints/openai/test_watermarking.py @@ -1,32 +1,249 @@ # SPDX-License-Identifier: Apache-2.0 # SPDX-FileCopyrightText: Copyright contributors to the vLLM project -from vllm.entrypoints.openai.chat_completion.protocol import ChatCompletionRequest +from io import BytesIO + +import pytest +from fastapi import UploadFile + +from vllm.entrypoints.openai.chat_completion.protocol import ( + BatchChatCompletionRequest, + ChatCompletionRequest, +) from vllm.entrypoints.openai.completion.protocol import CompletionRequest from vllm.entrypoints.openai.responses.protocol import ResponsesRequest +from vllm.entrypoints.speech_to_text.transcription.protocol import ( + TranscriptionRequest, +) +from vllm.entrypoints.speech_to_text.translation.protocol import TranslationRequest +from vllm.sampling_params import StructuredOutputsParams -def test_chat_request_disables_watermarking(): +def test_chat_request_preserves_watermarking_opt_out(): request = ChatCompletionRequest( messages=[{"role": "user", "content": "hello"}], watermarking=False ) params = request.to_sampling_params(max_tokens=1, default_sampling_params={}) - assert not params.watermarking + assert params.watermarking is False -def test_completion_request_disables_watermarking(): +def test_completion_request_preserves_watermarking_opt_out(): request = CompletionRequest(prompt="hello", watermarking=False) params = request.to_sampling_params(max_tokens=1) - assert not params.watermarking + assert params.watermarking is False -def test_responses_request_disables_watermarking(): +def test_responses_request_preserves_watermarking_opt_out(): request = ResponsesRequest(input="hello", watermarking=False) params = request.to_sampling_params(default_max_tokens=1) - assert not params.watermarking + assert params.watermarking is False + + +def test_batch_chat_request_preserves_watermarking_opt_out(): + request = BatchChatCompletionRequest( + messages=[[{"role": "user", "content": "hello"}]], watermarking=False + ) + + converted = request.to_chat_completion_request(request.messages[0]) + params = converted.to_sampling_params(max_tokens=1, default_sampling_params={}) + + assert params.watermarking is False + + +@pytest.mark.parametrize( + "api_request,conversion,kwargs", + [ + ( + ChatCompletionRequest( + messages=[{"role": "user", "content": "hello"}], + use_beam_search=True, + watermarking=False, + ), + "to_beam_search_params", + {"max_tokens": 1, "default_sampling_params": {}}, + ), + ( + CompletionRequest(prompt="hello", use_beam_search=True, watermarking=False), + "to_beam_search_params", + {"max_tokens": 1, "default_sampling_params": {}}, + ), + ( + TranscriptionRequest( + file=UploadFile(file=BytesIO(), filename="audio.wav"), + watermarking=False, + ), + "to_sampling_params", + {"default_max_tokens": 1}, + ), + ( + TranscriptionRequest( + file=UploadFile(file=BytesIO(), filename="audio.wav"), + watermarking=False, + ), + "to_beam_search_params", + {"default_max_tokens": 1}, + ), + ( + TranslationRequest( + file=UploadFile(file=BytesIO(), filename="audio.wav"), + watermarking=False, + ), + "to_sampling_params", + {"default_max_tokens": 1}, + ), + ( + TranslationRequest( + file=UploadFile(file=BytesIO(), filename="audio.wav"), + watermarking=False, + ), + "to_beam_search_params", + {"default_max_tokens": 1}, + ), + ], +) +def test_request_preserves_watermarking_opt_out(api_request, conversion, kwargs): + params = getattr(api_request, conversion)(**kwargs) + + assert params.watermarking is False + + +@pytest.mark.parametrize("request_cls", [TranscriptionRequest, TranslationRequest]) +def test_speech_requests_leave_watermarking_unspecified(request_cls): + request = request_cls( + file=UploadFile(file=BytesIO(), filename="audio.wav"), + temperature=0.8, + ) + + sampling_params = request.to_sampling_params(default_max_tokens=1) + beam_params = request.to_beam_search_params(default_max_tokens=1) + + assert request.watermarking is None + assert sampling_params.watermarking is None + assert beam_params.watermarking is None + + +@pytest.mark.parametrize( + "api_request,kwargs", + [ + ( + ChatCompletionRequest( + messages=[{"role": "user", "content": "choose A or B"}], + temperature=0.8, + structured_outputs=StructuredOutputsParams(choice=["A", "B"]), + ), + {"max_tokens": 8, "default_sampling_params": {}}, + ), + ( + CompletionRequest( + prompt="choose A or B", + temperature=0.8, + structured_outputs=StructuredOutputsParams(choice=["A", "B"]), + ), + {"max_tokens": 8}, + ), + ( + ResponsesRequest( + input="choose A or B", + temperature=0.8, + structured_outputs=StructuredOutputsParams(choice=["A", "B"]), + ), + {"default_max_tokens": 8}, + ), + ], +) +def test_request_preserves_unspecified_watermarking_with_structured_outputs( + api_request, kwargs +): + params = api_request.to_sampling_params(**kwargs) + + assert params.watermarking is None + assert params.structured_outputs is not None + assert params.structured_outputs.choice == ["A", "B"] + + +@pytest.mark.parametrize( + "api_request,conversion,kwargs", + [ + ( + ChatCompletionRequest( + messages=[{"role": "user", "content": "hello"}], watermarking=True + ), + "to_sampling_params", + {"max_tokens": 1, "default_sampling_params": {}}, + ), + ( + ChatCompletionRequest( + messages=[{"role": "user", "content": "hello"}], + use_beam_search=True, + watermarking=True, + ), + "to_beam_search_params", + {"max_tokens": 1, "default_sampling_params": {}}, + ), + ( + CompletionRequest(prompt="hello", watermarking=True), + "to_sampling_params", + {"max_tokens": 1}, + ), + ( + CompletionRequest(prompt="hello", use_beam_search=True, watermarking=True), + "to_beam_search_params", + {"max_tokens": 1}, + ), + ( + ResponsesRequest(input="hello", watermarking=True), + "to_sampling_params", + {"default_max_tokens": 1}, + ), + ( + TranscriptionRequest( + file=UploadFile(file=BytesIO(), filename="audio.wav"), + watermarking=True, + ), + "to_sampling_params", + {"default_max_tokens": 1}, + ), + ( + TranslationRequest( + file=UploadFile(file=BytesIO(), filename="audio.wav"), + watermarking=True, + ), + "to_sampling_params", + {"default_max_tokens": 1}, + ), + ], +) +def test_request_preserves_explicit_watermarking(api_request, conversion, kwargs): + params = getattr(api_request, conversion)(**kwargs) + + assert params.watermarking is True + + +def test_batch_chat_request_preserves_explicit_watermarking(): + request = BatchChatCompletionRequest( + messages=[[{"role": "user", "content": "hello"}]], watermarking=True + ) + + converted = request.to_chat_completion_request(request.messages[0]) + params = converted.to_sampling_params(max_tokens=1, default_sampling_params={}) + + assert converted.watermarking is True + assert params.watermarking is True + + +def test_batch_chat_request_inherits_watermarking(): + request = BatchChatCompletionRequest( + messages=[[{"role": "user", "content": "hello"}]] + ) + + converted = request.to_chat_completion_request(request.messages[0]) + params = converted.to_sampling_params(max_tokens=1, default_sampling_params={}) + + assert converted.watermarking is None + assert params.watermarking is None diff --git a/tests/entrypoints/pooling/test_io_processor.py b/tests/entrypoints/pooling/test_io_processor.py index 9d2207fdf1..79bddc0ffe 100644 --- a/tests/entrypoints/pooling/test_io_processor.py +++ b/tests/entrypoints/pooling/test_io_processor.py @@ -7,6 +7,8 @@ import pytest from vllm import PoolingParams from vllm.entrypoints.pooling.base.io_processor import PoolingIOProcessor +from vllm.entrypoints.pooling.offline import PoolingOfflineMixin +from vllm.entrypoints.pooling.scoring.io_processor import CrossEncoderIOProcessor from vllm.entrypoints.pooling.typing import OfflineEncodeInputsContext from vllm.exceptions import VLLMValidationError from vllm.renderers import TokenizeParams @@ -221,3 +223,38 @@ def test_qwen3_reranker_no_warning_when_template_provided(monkeypatch): ) mock_logger.warning.assert_not_called() + + +def _make_score_llm(monkeypatch) -> PoolingOfflineMixin: + """A scoring LLM with no model behind it. + + Input validation runs for real; only the rendering, engine and + post-processing boundaries are stubbed out. The pooling task is filled in + before any of them, so the mutation is observable without inference. + """ + proc = CrossEncoderIOProcessor.__new__(CrossEncoderIOProcessor) + proc.is_multimodal_model = False + proc.architecture = "CrossEncoder" + monkeypatch.setattr(proc, "get_request_factory_offline", lambda ctx: (None, 0)) + monkeypatch.setattr(proc, "post_process_offline", lambda ctx: []) + + llm = PoolingOfflineMixin.__new__(PoolingOfflineMixin) + llm.runner_type = "pooling" + llm.pooling_task = "classify" # SCORE_TYPE_MAP -> "cross-encoder" + llm.model_config = SimpleNamespace(hf_config=SimpleNamespace(num_labels=1)) + llm.pooling_io_processors = {"cross-encoder": proc} + monkeypatch.setattr(llm, "_run_tiling_engine", lambda *args, **kwargs: []) + return llm + + +def test_score_leaves_caller_pooling_params_untouched(monkeypatch): + llm = _make_score_llm(monkeypatch) + + params = PoolingParams() + llm.score("query", "doc", pooling_params=params) + + assert params.task is None, ( + "LLM.score() wrote the model's own pooling task into the caller's " + "PoolingParams; reusing that object with another pooling model now " + "fails with 'You cannot overwrite ...' before any inference runs." + ) diff --git a/tests/entrypoints/serve/dev/rlhf/state_transitions/__init__.py b/tests/entrypoints/rl/__init__.py similarity index 100% rename from tests/entrypoints/serve/dev/rlhf/state_transitions/__init__.py rename to tests/entrypoints/rl/__init__.py diff --git a/tests/entrypoints/serve/dev/rlhf/conftest.py b/tests/entrypoints/rl/conftest.py similarity index 100% rename from tests/entrypoints/serve/dev/rlhf/conftest.py rename to tests/entrypoints/rl/conftest.py diff --git a/vllm/entrypoints/serve/dev/rlhf/__init__.py b/tests/entrypoints/rl/return_routed_experts/__init__.py similarity index 100% rename from vllm/entrypoints/serve/dev/rlhf/__init__.py rename to tests/entrypoints/rl/return_routed_experts/__init__.py diff --git a/tests/entrypoints/openai/test_return_routed_experts.py b/tests/entrypoints/rl/return_routed_experts/test_openai_api.py similarity index 98% rename from tests/entrypoints/openai/test_return_routed_experts.py rename to tests/entrypoints/rl/return_routed_experts/test_openai_api.py index 46eeed1795..f4fc401399 100644 --- a/tests/entrypoints/openai/test_return_routed_experts.py +++ b/tests/entrypoints/rl/return_routed_experts/test_openai_api.py @@ -7,7 +7,7 @@ import numpy as np import pybase64 as base64 import pytest -from ...utils import RemoteOpenAIServer +from tests.utils import RemoteOpenAIServer MODEL_NAME = "TitanML/tiny-mixtral" diff --git a/tests/entrypoints/unit_tests/test_dev_route_request_bodies.py b/tests/entrypoints/rl/return_routed_experts/test_route_request_bodies.py similarity index 96% rename from tests/entrypoints/unit_tests/test_dev_route_request_bodies.py rename to tests/entrypoints/rl/return_routed_experts/test_route_request_bodies.py index 78fe5eef5d..b51d08db07 100644 --- a/tests/entrypoints/unit_tests/test_dev_route_request_bodies.py +++ b/tests/entrypoints/rl/return_routed_experts/test_route_request_bodies.py @@ -9,7 +9,7 @@ import pytest from fastapi import FastAPI from fastapi.testclient import TestClient -from vllm.entrypoints.serve.dev.rlhf.api_router import router as rlhf_router +from vllm.entrypoints.rl.online.api_router import router as rl_router from vllm.entrypoints.serve.dev.rpc.api_router import router as rpc_router from vllm.entrypoints.serve.exception_handling.register import init_exception_handler @@ -30,7 +30,7 @@ class Engine: @pytest.fixture def client(): app = FastAPI() - app.include_router(rlhf_router) + app.include_router(rl_router) app.include_router(rpc_router) init_exception_handler(app) app.state.engine_client = Engine() diff --git a/tests/entrypoints/scale_out/token_in_token_out/test_return_routed_experts.py b/tests/entrypoints/rl/return_routed_experts/test_scale_out_api.py similarity index 100% rename from tests/entrypoints/scale_out/token_in_token_out/test_return_routed_experts.py rename to tests/entrypoints/rl/return_routed_experts/test_scale_out_api.py diff --git a/tests/entrypoints/rl/state_transitions/__init__.py b/tests/entrypoints/rl/state_transitions/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/tests/entrypoints/serve/dev/rlhf/state_transitions/test_pause_resume.py b/tests/entrypoints/rl/state_transitions/test_pause_resume.py similarity index 98% rename from tests/entrypoints/serve/dev/rlhf/state_transitions/test_pause_resume.py rename to tests/entrypoints/rl/state_transitions/test_pause_resume.py index 3de9e00549..587b4b7319 100644 --- a/tests/entrypoints/serve/dev/rlhf/state_transitions/test_pause_resume.py +++ b/tests/entrypoints/rl/state_transitions/test_pause_resume.py @@ -10,7 +10,7 @@ from unittest.mock import patch import pytest import requests -from tests.entrypoints.serve.dev.rlhf.conftest import ( +from tests.entrypoints.rl.conftest import ( cached_tokens, completion_with_cache_details, gen, diff --git a/tests/entrypoints/serve/dev/test_sleep.py b/tests/entrypoints/rl/state_transitions/test_sleep.py similarity index 100% rename from tests/entrypoints/serve/dev/test_sleep.py rename to tests/entrypoints/rl/state_transitions/test_sleep.py diff --git a/tests/entrypoints/serve/dev/rlhf/state_transitions/test_weight_checker.py b/tests/entrypoints/rl/state_transitions/test_weight_checker.py similarity index 98% rename from tests/entrypoints/serve/dev/rlhf/state_transitions/test_weight_checker.py rename to tests/entrypoints/rl/state_transitions/test_weight_checker.py index 7cb58df9d9..ec3c8016a8 100644 --- a/tests/entrypoints/serve/dev/rlhf/state_transitions/test_weight_checker.py +++ b/tests/entrypoints/rl/state_transitions/test_weight_checker.py @@ -9,7 +9,7 @@ import requests import torch from transformers import AutoModelForCausalLM -from tests.entrypoints.serve.dev.rlhf.conftest import ( +from tests.entrypoints.rl.conftest import ( MODEL_NAME, gen, health, diff --git a/tests/entrypoints/weight_transfer/__init__.py b/tests/entrypoints/rl/weight_transfer/__init__.py similarity index 100% rename from tests/entrypoints/weight_transfer/__init__.py rename to tests/entrypoints/rl/weight_transfer/__init__.py diff --git a/tests/entrypoints/weight_transfer/test_weight_transfer_llm.py b/tests/entrypoints/rl/weight_transfer/test_offline.py similarity index 99% rename from tests/entrypoints/weight_transfer/test_weight_transfer_llm.py rename to tests/entrypoints/rl/weight_transfer/test_offline.py index e5c3616d6e..e4a30f17e6 100644 --- a/tests/entrypoints/weight_transfer/test_weight_transfer_llm.py +++ b/tests/entrypoints/rl/weight_transfer/test_offline.py @@ -15,6 +15,7 @@ from unittest.mock import patch import pytest import torch +from tests.utils import create_new_process_for_each_test from vllm.config import WeightTransferConfig from vllm.distributed.weight_transfer.base import ( WeightTransferEngine, @@ -24,8 +25,6 @@ from vllm.distributed.weight_transfer.base import ( WeightTransferUpdateRequest, ) -from ...utils import create_new_process_for_each_test - # Use a tiny model for fast testing MODEL_NAME = "hmellor/tiny-random-LlamaForCausalLM" diff --git a/tests/entrypoints/rl/weight_transfer_metrics/__init__.py b/tests/entrypoints/rl/weight_transfer_metrics/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/tests/entrypoints/serve/dev/rlhf/test_weight_operation_metrics.py b/tests/entrypoints/rl/weight_transfer_metrics/test_across_api_servers_metrics.py similarity index 97% rename from tests/entrypoints/serve/dev/rlhf/test_weight_operation_metrics.py rename to tests/entrypoints/rl/weight_transfer_metrics/test_across_api_servers_metrics.py index d7758011f9..108dd77b2f 100644 --- a/tests/entrypoints/serve/dev/rlhf/test_weight_operation_metrics.py +++ b/tests/entrypoints/rl/weight_transfer_metrics/test_across_api_servers_metrics.py @@ -7,7 +7,7 @@ import torch from prometheus_client.parser import text_string_to_metric_families from transformers import AutoModelForCausalLM -from tests.entrypoints.serve.dev.rlhf.conftest import MODEL_NAME +from tests.entrypoints.rl.conftest import MODEL_NAME from tests.utils import RemoteOpenAIServer from vllm.distributed.weight_transfer import ( HTTPVLLMWeightSyncClient, diff --git a/tests/entrypoints/unit_tests/test_rl_weight_operation_metrics.py b/tests/entrypoints/rl/weight_transfer_metrics/test_weight_operation_metrics.py similarity index 90% rename from tests/entrypoints/unit_tests/test_rl_weight_operation_metrics.py rename to tests/entrypoints/rl/weight_transfer_metrics/test_weight_operation_metrics.py index bdddf3d247..22ec5e6112 100644 --- a/tests/entrypoints/unit_tests/test_rl_weight_operation_metrics.py +++ b/tests/entrypoints/rl/weight_transfer_metrics/test_weight_operation_metrics.py @@ -13,9 +13,9 @@ from fastapi import FastAPI from fastapi.testclient import TestClient from prometheus_client import CollectorRegistry -from vllm.entrypoints.serve.dev.rlhf import api_router -from vllm.entrypoints.serve.dev.rlhf import metrics as rlhf_metrics -from vllm.entrypoints.serve.dev.rlhf.metrics import WeightOperationMetrics +from vllm.entrypoints.rl.online import api_router +from vllm.entrypoints.rl.online import metrics as rl_metrics +from vllm.entrypoints.rl.online.metrics import WeightOperationMetrics from vllm.entrypoints.serve.exception_handling.register import init_exception_handler from vllm.v1.metrics.prometheus import get_prometheus_registry @@ -52,13 +52,13 @@ class Engine: @pytest.fixture def registry(monkeypatch): registry = CollectorRegistry() - monkeypatch.setattr(rlhf_metrics, "_metrics", WeightOperationMetrics(registry)) + monkeypatch.setattr(rl_metrics, "_metrics", WeightOperationMetrics(registry)) return registry @pytest.mark.parametrize("fail", [False, True]) def test_record_releases_in_flight_and_observes_duration(registry, fail): - metrics = rlhf_metrics.weight_operation_metrics() + metrics = rl_metrics.weight_operation_metrics() with ( pytest.raises(RuntimeError) if fail else nullcontext(), metrics.record("update"), @@ -108,7 +108,7 @@ async def test_cancelled_route_releases_in_flight(registry): HOLD_OPERATION = """ import sys -from vllm.entrypoints.serve.dev.rlhf.metrics import weight_operation_metrics +from vllm.entrypoints.rl.online.metrics import weight_operation_metrics with weight_operation_metrics().record("update"): print("entered", flush=True) sys.stdin.readline() @@ -163,19 +163,19 @@ def test_default_recorder_is_created_lazily_on_the_default_registry(): probe = """ from prometheus_client import REGISTRY -from vllm.entrypoints.serve.dev.rlhf import metrics as rlhf_metrics +from vllm.entrypoints.rl.online import metrics as rl_metrics NAMES = [ "vllm:rl_weight_update_operation_duration_seconds", "vllm:rl_weight_update_operations_in_flight", ] -assert rlhf_metrics._metrics is None +assert rl_metrics._metrics is None assert not set(NAMES) & set(REGISTRY._names_to_collectors) -metrics = rlhf_metrics.weight_operation_metrics() +metrics = rl_metrics.weight_operation_metrics() assert REGISTRY._names_to_collectors[NAMES[0]] is metrics.duration assert REGISTRY._names_to_collectors[NAMES[1]] is metrics.in_flight -assert rlhf_metrics.weight_operation_metrics() is metrics +assert rl_metrics.weight_operation_metrics() is metrics print("lazy registration ok") """ result = subprocess.run( diff --git a/tests/entrypoints/scale_out/derender/test_derender.py b/tests/entrypoints/scale_out/derender/test_derender.py index 659aaed1aa..d29aa3f53f 100644 --- a/tests/entrypoints/scale_out/derender/test_derender.py +++ b/tests/entrypoints/scale_out/derender/test_derender.py @@ -74,14 +74,13 @@ def _make_generate_response( } -def _make_logprobs_with_placeholders(token_id: int = 1234) -> dict: +def _make_generate_logprobs(token_id: int = 1234) -> dict: + """A `GenerateLogProbs` payload: integer token ids, no token/bytes.""" entry = { - "token": f"token_id:{token_id}", + "token_id": token_id, "logprob": -1.0, - "bytes": None, - "top_logprobs": [ - {"token": f"token_id:{token_id + 1}", "logprob": -2.0, "bytes": None} - ], + "rank": 1, + "top_logprobs": [{"token_id": token_id + 1, "logprob": -2.0, "rank": 2}], } return {"content": [entry]} @@ -154,7 +153,7 @@ async def test_derender_chat_usage_default(client): @pytest.mark.asyncio async def test_derender_chat_logprobs(client): - """token_id:N placeholders in content.token are resolved to real strings.""" + """Integer token ids are decoded to real strings by derender.""" gen_req = await _render_chat(client) synthetic_ids = gen_req["token_ids"][:3] token_id = synthetic_ids[0] @@ -165,7 +164,7 @@ async def test_derender_chat_logprobs(client): "model": MODEL_NAME, "generate_response": _make_generate_response( synthetic_ids, - logprobs=_make_logprobs_with_placeholders(token_id), + logprobs=_make_generate_logprobs(token_id), ), }, ) @@ -176,8 +175,8 @@ async def test_derender_chat_logprobs(client): content = logprobs["content"] assert content is not None and len(content) == 1 token_str = content[0]["token"] - assert not token_str.startswith("token_id:"), ( - f"Placeholder was not resolved: {token_str!r}" + assert token_str and not token_str.startswith("token_id:"), ( + f"token id was not decoded: {token_str!r}" ) @@ -194,7 +193,7 @@ async def test_derender_chat_logprobs_bytes(client): "model": MODEL_NAME, "generate_response": _make_generate_response( synthetic_ids, - logprobs=_make_logprobs_with_placeholders(token_id), + logprobs=_make_generate_logprobs(token_id), ), }, ) @@ -208,7 +207,7 @@ async def test_derender_chat_logprobs_bytes(client): @pytest.mark.asyncio async def test_derender_chat_top_logprobs(client): - """top_logprobs entries also have their placeholders resolved.""" + """top_logprobs entries are decoded too.""" gen_req = await _render_chat(client) synthetic_ids = gen_req["token_ids"][:3] token_id = synthetic_ids[0] @@ -219,7 +218,7 @@ async def test_derender_chat_top_logprobs(client): "model": MODEL_NAME, "generate_response": _make_generate_response( synthetic_ids, - logprobs=_make_logprobs_with_placeholders(token_id), + logprobs=_make_generate_logprobs(token_id), ), }, ) @@ -228,7 +227,7 @@ async def test_derender_chat_top_logprobs(client): top = content[0]["top_logprobs"] assert len(top) == 1 assert not top[0]["token"].startswith("token_id:"), ( - f"top_logprobs placeholder not resolved: {top[0]['token']!r}" + f"top_logprobs token id not decoded: {top[0]['token']!r}" ) @@ -633,8 +632,8 @@ async def test_derender_completion_empty_generate_responses(client): @pytest.mark.asyncio async def test_derender_completion_logprobs(client): - """token_id:N placeholders in logprobs are resolved; CompletionLogProbs - flat-list structure is returned with non-empty tokens and text_offsets.""" + """Integer token ids are decoded; CompletionLogProbs flat-list structure is + returned with non-empty tokens and text_offsets.""" gr1 = await _render_completion(client, "Hello world") ids1 = gr1["token_ids"][:3] token_id = ids1[0] @@ -647,7 +646,7 @@ async def test_derender_completion_logprobs(client): _make_completion_generate_response( ids1, gr1["request_id"], - logprobs=_make_logprobs_with_placeholders(token_id), + logprobs=_make_generate_logprobs(token_id), ), ], }, @@ -658,7 +657,7 @@ async def test_derender_completion_logprobs(client): tokens = logprobs["tokens"] assert len(tokens) == 1 assert not tokens[0].startswith("token_id:"), ( - f"Placeholder was not resolved: {tokens[0]!r}" + f"token id was not decoded: {tokens[0]!r}" ) assert len(logprobs["token_logprobs"]) == 1 assert isinstance(logprobs["token_logprobs"][0], float) @@ -794,7 +793,7 @@ async def test_derender_chat_oversized_logprobs_rejected(client): """logprobs.content longer than max_model_len returns 400.""" oversized_logprobs: dict = { "content": [ - {"token": "x", "logprob": -1.0, "bytes": None, "top_logprobs": []} + {"token_id": 42, "logprob": -1.0, "rank": 1, "top_logprobs": []} for _ in range(1_000_000) ] } @@ -825,11 +824,11 @@ async def test_derender_chat_oversized_top_logprobs_rejected(client): oversized_top_logprobs = { "content": [ { - "token": "x", + "token_id": 42, "logprob": -1.0, - "bytes": None, + "rank": 1, "top_logprobs": [ - {"token": f"t{i}", "logprob": -float(i), "bytes": None} + {"token_id": i, "logprob": -float(i), "rank": i + 1} for i in range(25) ], } @@ -1298,6 +1297,72 @@ async def test_e2e_harmony_plain_roundtrip(harmony_client, harmony_tokenizer): assert "Four" in content +@pytest.fixture(scope="module") +def harmony_default_server(): + """gpt-oss render server with NO parser flags. + + Regression fixture: the model-default reasoning parser + ("openai_gptoss", applied by verify_and_update_config) must be + resolved by the render server without any --reasoning-parser flag, + otherwise harmony markup leaks unparsed into derender output. + """ + _ensure_harmony_vocab() + with RemoteLaunchRenderServer( + HARMONY_MODEL, ["--trust-remote-code"] + ) as remote_server: + yield remote_server + + +@pytest_asyncio.fixture +async def harmony_default_client(harmony_default_server): + async with httpx.AsyncClient( + base_url=harmony_default_server.url_for(""), timeout=60.0 + ) as http_client: + yield http_client + + +@pytest.mark.asyncio +async def test_e2e_harmony_reasoning_default_parser( + harmony_default_client, harmony_tokenizer +): + """GPT-OSS reasoning parses on a bare launch (no parser flags).""" + messages = [{"role": "user", "content": "Add 2 and 3."}] + gen_req = await _e2e_render_chat(harmony_default_client, HARMONY_MODEL, messages) + + reasoning_text = "The user wants 2 plus 3." + answer_text = "The answer is 5." + assistant_msg = { + "role": "assistant", + "thinking": reasoning_text, + "content": answer_text, + } + output_ids = _harmony_extract_assistant_ids(harmony_tokenizer, assistant_msg) + + decoded = harmony_tokenizer.decode(output_ids) + if reasoning_text not in decoded: + pytest.skip("Harmony template did not render thinking") + + resp = await harmony_default_client.post( + "/v1/chat/completions/derender", + json={ + "model": HARMONY_MODEL, + "generate_response": _e2e_generate_response(output_ids), + "prompt_tokens": len(gen_req["token_ids"]), + "chat_request": { + "model": HARMONY_MODEL, + "messages": messages, + "include_reasoning": True, + }, + }, + ) + assert resp.status_code == 200, resp.text + msg = resp.json()["choices"][0]["message"] + assert msg["reasoning"] is not None + assert reasoning_text in msg["reasoning"] + assert answer_text in (msg["content"] or "") + assert "<|channel|>" not in (msg["content"] or "") + + @pytest.mark.asyncio async def test_e2e_harmony_reasoning(harmony_client, harmony_tokenizer): """GPT-OSS reasoning: analysis channel extracted.""" diff --git a/tests/entrypoints/scale_out/derender/test_derender_parity.py b/tests/entrypoints/scale_out/derender/test_derender_parity.py index 66d3e83969..4ea3bc2461 100644 --- a/tests/entrypoints/scale_out/derender/test_derender_parity.py +++ b/tests/entrypoints/scale_out/derender/test_derender_parity.py @@ -237,21 +237,47 @@ async def test_parity_reasoning_and_tool_call(client): assert _tool_sig(disagg["choices"][0]) +def _to_generate_logprobs(openai_logprobs: dict) -> dict: + """Turn a coupled response's `token_id:N` logprobs into `GenerateLogProbs`. + + A real GPU-less worker emits integer ids directly; here the coupled server + with `return_tokens_as_token_ids=True` is the only source of the same ids, + so the placeholders are parsed back out to build the generate-shaped + payload that `/derender` now consumes. + """ + + def _id(token: str) -> int: + assert token.startswith("token_id:"), token + return int(token.removeprefix("token_id:")) + + return { + "content": [ + { + "token_id": _id(entry["token"]), + "logprob": entry["logprob"], + "top_logprobs": [ + {"token_id": _id(top["token"]), "logprob": top["logprob"]} + for top in entry["top_logprobs"] + ], + } + for entry in openai_logprobs["content"] + ] + } + + @pytest.mark.asyncio async def test_parity_logprobs(client): - """token_id:N resolution parity vs. the coupled server's real strings. + """Integer-id logprob decoding parity vs. the coupled server's strings. - A real disaggregated worker only has token IDs so it emits logprobs - with `token_id:N` placeholders (`return_tokens_as_token_ids=True` - reproduces that shape here). `/derender` must resolve those - placeholders to the same token strings/bytes the coupled server - resolves them to directly. + A real disaggregated worker only has token IDs, so it emits + `GenerateLogProbs` with integer `token_id`s. `/derender` must decode those + to the same token strings/bytes the coupled server produces directly. """ messages = [{"role": "user", "content": "What is 2+2?"}] extra = {"logprobs": True, "top_logprobs": 3} - # What a real GPU less worker would hand to /derender is token IDs plus - # logprobs still in token_id:N placeholder form + # The coupled server with placeholders is used only as a source of the + # token ids a GPU-less worker would have had. placeholder = await _coupled( client, messages, return_tokens_as_token_ids=True, **extra ) @@ -263,7 +289,7 @@ async def test_parity_logprobs(client): placeholder["usage"]["prompt_tokens"], ch["finish_reason"], chat_request, - logprobs=ch["logprobs"], + logprobs=_to_generate_logprobs(ch["logprobs"]), ) # The coupled server resolving the same greedy generation diff --git a/tests/entrypoints/scale_out/derender/test_derender_stream.py b/tests/entrypoints/scale_out/derender/test_derender_stream.py index 0e4b168677..0a7d96f3cf 100644 --- a/tests/entrypoints/scale_out/derender/test_derender_stream.py +++ b/tests/entrypoints/scale_out/derender/test_derender_stream.py @@ -64,6 +64,7 @@ from vllm.renderers.online_derenderer import ( _seed_stream_state, ) from vllm.tokenizers import get_tokenizer +from vllm.tokenizers.detokenizer_utils import convert_ids_list_to_tokens from vllm.utils import random_uuid MODEL_NAME = "hmellor/tiny-random-LlamaForCausalLM" @@ -212,17 +213,12 @@ def _make_stream_chunk( ) -def _placeholder_logprobs(token_ids: list[int]) -> dict: - """Per-token logprob entries using token_id:N placeholders, as sent by - the generate worker.""" +def _generate_logprobs(token_ids: list[int]) -> dict: + """Per-token `GenerateLogProbs` entries (integer token ids, no token or + bytes), as sent by the generate worker.""" return { "content": [ - { - "token": f"token_id:{tid}", - "logprob": -0.5, - "bytes": None, - "top_logprobs": [], - } + {"token_id": tid, "logprob": -0.5, "rank": 1, "top_logprobs": []} for tid in token_ids ] } @@ -892,12 +888,12 @@ class TestDerenderCompletionStream: class TestStreamLogprobs: - """Streaming derender must carry per-chunk logprobs with placeholders - resolved, matching what the generate streaming path emits.""" + """Streaming derender must carry per-chunk logprobs with token ids + decoded, matching what the generate streaming path emits.""" @pytest.mark.asyncio async def test_chat_stream_logprobs_resolved_per_chunk(self, derenderer, tokenizer): - """Each streamed chunk carries logprobs with token_id:N resolved.""" + """Each streamed chunk carries logprobs with its token ids decoded.""" token_ids = tokenizer.encode("hello world")[:6] mid = len(token_ids) // 2 @@ -906,17 +902,15 @@ class TestStreamLogprobs: chunk, state = await derenderer.derender_chat_stream( model=MODEL_NAME, generate_chunk=_make_stream_chunk( - part, logprobs=_placeholder_logprobs(part) + part, logprobs=_generate_logprobs(part) ), state=state, ) logprobs = chunk.choices[0].logprobs assert logprobs is not None and logprobs.content is not None - assert len(logprobs.content) == len(part) - for entry in logprobs.content: - assert not entry.token.startswith("token_id:"), ( - f"placeholder not resolved: {entry.token!r}" - ) + assert [entry.token for entry in logprobs.content] == ( + convert_ids_list_to_tokens(tokenizer, part) + ) @pytest.mark.asyncio async def test_chat_stream_logprobs_multibyte_across_chunks( @@ -936,7 +930,7 @@ class TestStreamLogprobs: last_chunk, state = await derenderer.derender_chat_stream( model=MODEL_NAME, generate_chunk=_make_stream_chunk( - [tid], logprobs=_placeholder_logprobs([tid]) + [tid], logprobs=_generate_logprobs([tid]) ), state=state, ) @@ -959,13 +953,13 @@ class TestStreamLogprobs: chunk1, state = await derenderer.derender_completion_stream( model=MODEL_NAME, generate_chunk=_make_stream_chunk( - token_ids[:mid], logprobs=_placeholder_logprobs(token_ids[:mid]) + token_ids[:mid], logprobs=_generate_logprobs(token_ids[:mid]) ), ) chunk2, _ = await derenderer.derender_completion_stream( model=MODEL_NAME, generate_chunk=_make_stream_chunk( - token_ids[mid:], logprobs=_placeholder_logprobs(token_ids[mid:]) + token_ids[mid:], logprobs=_generate_logprobs(token_ids[mid:]) ), state=state, ) @@ -982,7 +976,7 @@ class TestStreamLogprobs: _, state = await derenderer.derender_chat_stream( model=MODEL_NAME, generate_chunk=_make_stream_chunk( - [tid], logprobs=_placeholder_logprobs([tid]) + [tid], logprobs=_generate_logprobs([tid]) ), state=state, ) diff --git a/tests/entrypoints/scale_out/derender/test_generate_logprobs_conversion.py b/tests/entrypoints/scale_out/derender/test_generate_logprobs_conversion.py new file mode 100644 index 0000000000..96d937ad5a --- /dev/null +++ b/tests/entrypoints/scale_out/derender/test_generate_logprobs_conversion.py @@ -0,0 +1,172 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Unit tests for the generate -> OpenAI logprob conversion in derender. + +These exercise `_resolve_logprobs` directly. A stub tokenizer covers the +byte-fallback correction path; a real SentencePiece tokenizer covers the +leading-space restoration. +""" + +from vllm.entrypoints.scale_out.token_in_token_out.protocol import ( + GenerateLogProb, + GenerateLogProbs, + GenerateLogProbsContent, +) +from vllm.renderers.online_derenderer import ( + _convert_chat_logprobs_to_completion_logprobs, + _resolve_logprobs, +) +from vllm.tokenizers import get_tokenizer +from vllm.tokenizers.detokenizer_utils import convert_ids_list_to_tokens + +# A two-byte character split across two byte-fallback tokens: neither decodes +# to anything printable on its own, but the pair decodes to "é". +FIRST_BYTE = 10 +SECOND_BYTE = 11 +PLAIN = 12 + +VOCAB = {FIRST_BYTE: "<0xC3>", SECOND_BYTE: "<0xA9>", PLAIN: "ok"} + + +class _StubTokenizer: + """Decodes byte-fallback tokens only when both halves are present.""" + + def convert_ids_to_tokens(self, ids: list[int]) -> list[str | None]: + return [VOCAB.get(i) for i in ids] + + def convert_tokens_to_string(self, tokens: list[str]) -> str: + return self.decode([k for t in tokens for k, v in VOCAB.items() if v == t]) + + def decode(self, ids: list[int], **kwargs) -> str: + out = "" + i = 0 + while i < len(ids): + if ids[i] == FIRST_BYTE and i + 1 < len(ids) and ids[i + 1] == SECOND_BYTE: + out += "é" + i += 2 + elif ids[i] in (FIRST_BYTE, SECOND_BYTE): + out += "�" + i += 1 + else: + out += VOCAB.get(ids[i], "") + i += 1 + return out + + +def test_token_ids_are_decoded_with_bytes_and_order_preserved(): + logprobs = GenerateLogProbs( + content=[ + GenerateLogProbsContent( + token_id=PLAIN, + logprob=-0.5, + rank=1, + top_logprobs=[ + GenerateLogProb(token_id=PLAIN, logprob=-0.5, rank=1), + GenerateLogProb(token_id=FIRST_BYTE, logprob=-2.0, rank=2), + ], + ) + ] + ) + + resolved = _resolve_logprobs(logprobs, _StubTokenizer()) + + assert resolved.content is not None + entry = resolved.content[0] + assert entry.token == "ok" + assert entry.logprob == -0.5 + assert entry.bytes == list(b"ok") + # Rank order survives the conversion (the OpenAI shape has no rank field). + assert [t.logprob for t in entry.top_logprobs] == [-0.5, -2.0] + assert entry.top_logprobs[0].token == "ok" + + +def test_byte_fallback_second_half_uses_preceding_token_as_context(): + """U+FFFD correction needs the preceding sampled ids, which the integer + shape carries directly (no placeholder parsing).""" + logprobs = GenerateLogProbs( + content=[ + GenerateLogProbsContent(token_id=FIRST_BYTE, logprob=-0.1), + GenerateLogProbsContent(token_id=SECOND_BYTE, logprob=-0.2), + ] + ) + + resolved = _resolve_logprobs(logprobs, _StubTokenizer()) + + assert resolved.content is not None + # The first half has no preceding context to repair it with, so it stays + # empty (existing `_correct_decoded_token` behaviour). The second resolves + # to the complete character once the first is used as context. + assert resolved.content[0].token == "" + assert resolved.content[0].bytes == [] + assert resolved.content[1].token == "é" + assert resolved.content[1].bytes == list("é".encode()) + + +def test_content_none_is_not_an_error(): + resolved = _resolve_logprobs(GenerateLogProbs(), _StubTokenizer()) + assert resolved.content is None + + +def test_sentencepiece_leading_space_matches_engine(): + """`▁true` and `true` are distinct tokens and must stay distinct: the + strings match the engine's per-token detokenization (leading space + restored), so `/v1/completions/derender` top_logprobs keys do not collide. + Regression test for #59241.""" + tokenizer = get_tokenizer("hmellor/tiny-random-LlamaForCausalLM") + spaced = tokenizer.convert_tokens_to_ids("▁true") + bare = tokenizer.convert_tokens_to_ids("true") + assert spaced != bare + logprobs = GenerateLogProbs( + content=[ + GenerateLogProbsContent( + token_id=spaced, + logprob=-0.5, + rank=1, + top_logprobs=[ + GenerateLogProb(token_id=spaced, logprob=-0.5, rank=1), + GenerateLogProb(token_id=bare, logprob=-1.5, rank=2), + ], + ) + ] + ) + + resolved = _resolve_logprobs(logprobs, tokenizer) + + assert resolved.content is not None + entry = resolved.content[0] + assert entry.token == " true" + assert entry.bytes == list(b" true") + assert [t.token for t in entry.top_logprobs] == convert_ids_list_to_tokens( + tokenizer, [spaced, bare] + ) + completion = _convert_chat_logprobs_to_completion_logprobs(resolved) + assert completion.top_logprobs == [{" true": -0.5, "true": -1.5}] + + +def test_unknown_top_id_does_not_shift_the_other_candidates(): + """Ids are decoded in one batch per position; an id with no vocab entry + becomes an empty token without shifting its neighbours.""" + unknown = 99 + logprobs = GenerateLogProbs( + content=[ + GenerateLogProbsContent( + token_id=PLAIN, + logprob=-0.5, + rank=1, + top_logprobs=[ + GenerateLogProb(token_id=unknown, logprob=-1.0, rank=1), + GenerateLogProb(token_id=PLAIN, logprob=-0.5, rank=2), + ], + ) + ] + ) + + resolved = _resolve_logprobs(logprobs, _StubTokenizer()) + + assert resolved.content is not None + entry = resolved.content[0] + assert entry.token == "ok" + assert [(t.token, t.bytes) for t in entry.top_logprobs] == [ + ("", None), + ("ok", list(b"ok")), + ] diff --git a/tests/entrypoints/scale_out/render/test_render.py b/tests/entrypoints/scale_out/render/test_render.py index 0daf4d405e..6c11a518d7 100644 --- a/tests/entrypoints/scale_out/render/test_render.py +++ b/tests/entrypoints/scale_out/render/test_render.py @@ -38,6 +38,7 @@ def _build_responses_serving_render() -> ServingRender: OnlineRenderer.validate_chat_template.__get__(serving.online_renderer) ) serving.online_renderer.trust_request_chat_template = True + serving.online_renderer.parser = None serving._check_model = AsyncMock(return_value=None) return serving @@ -135,6 +136,37 @@ async def test_render_responses_rejects_empty_token_ids(): assert response.error.message == "No token_ids rendered" +@pytest.mark.asyncio +@pytest.mark.skip_global_cleanup +async def test_render_responses_carries_reasoning_parser_kwargs(): + """/v1/responses gives the engine's reasoning parser the effective template + kwargs, so the rendered request must carry them to generate.""" + serving = _build_responses_serving_render() + online = serving.online_renderer + online.parser = MagicMock(reasoning_parser_cls=object) + online.chat_template = None + online.chat_template_content_format = "auto" + online.default_chat_template_kwargs = {"enable_thinking": True} + online.effective_chat_template_kwargs = ( + OnlineRenderer.effective_chat_template_kwargs.__get__(online) + ) + online.render_responses = AsyncMock( + return_value=MagicMock(messages=[], engine_input={"prompt_token_ids": [7]}) + ) + request = ResponsesRequest( + model=MODEL_NAME, + input="Test prompt", + chat_template_kwargs={"enable_thinking": False}, + ) + + response = await serving.render_responses_request(request) + + assert not isinstance(response, ErrorResponse) + assert response.reasoning_parser_kwargs is not None + kwargs = response.reasoning_parser_kwargs.chat_template_kwargs + assert kwargs["enable_thinking"] is False + + @pytest.fixture(scope="module") def server(): args: list[str] = ["--trust-request-chat-template"] diff --git a/tests/entrypoints/scale_out/token_in_token_out/test_generate_stream.py b/tests/entrypoints/scale_out/token_in_token_out/test_generate_stream.py index 7e7f74de9f..7f0cd12860 100644 --- a/tests/entrypoints/scale_out/token_in_token_out/test_generate_stream.py +++ b/tests/entrypoints/scale_out/token_in_token_out/test_generate_stream.py @@ -1123,7 +1123,8 @@ async def test_stream_with_logprobs(): lp = dc["choices"][0]["logprobs"] assert lp is not None assert len(lp["content"]) == 1 - assert lp["content"][0]["token"].startswith("token_id:") + assert isinstance(lp["content"][0]["token_id"], int) + assert "token" not in lp["content"][0] @pytest.mark.asyncio diff --git a/tests/entrypoints/scale_out/token_in_token_out/test_mm_serde.py b/tests/entrypoints/scale_out/token_in_token_out/test_mm_serde.py index 5cd60a959f..62223f1d7c 100644 --- a/tests/entrypoints/scale_out/token_in_token_out/test_mm_serde.py +++ b/tests/entrypoints/scale_out/token_in_token_out/test_mm_serde.py @@ -168,6 +168,20 @@ def test_render_extracts_metadata_fields_separately(): assert torch.equal(full_data["pixel_values"].data, pixel_values.data) +def test_extract_without_kwargs_keeps_layout(): + """Layout-only callers get hashes and placeholders, no serialized tensors.""" + engine_input, _pixel_values, _image_grid_thw = _image_engine_input() + + full = extract_mm_features(engine_input) + layout = extract_mm_features(engine_input, include_mm_kwargs=False) + + assert full is not None and layout is not None + assert layout.kwargs_data is None + assert layout.mm_metadata is None + assert layout.mm_hashes == full.mm_hashes + assert layout.mm_placeholders == full.mm_placeholders + + def test_extract_includes_declared_placeholder_metadata_fields(): """EC placeholder metadata is kept even when keep_on_cpu is unset.""" engine_input, _pixel_values, image_grid_thw = _image_engine_input( diff --git a/tests/entrypoints/scale_out/token_in_token_out/test_protocol.py b/tests/entrypoints/scale_out/token_in_token_out/test_protocol.py index 9894d09180..6a57e3b2b4 100644 --- a/tests/entrypoints/scale_out/token_in_token_out/test_protocol.py +++ b/tests/entrypoints/scale_out/token_in_token_out/test_protocol.py @@ -124,6 +124,14 @@ def test_generate_request_rejects_placeholder_outside_prompt(): ) +def test_reasoning_parser_kwargs_rejects_unknown_keys(): + """The engine splats these into its reasoning parser's constructor, so a + client must not be able to pass arbitrary keyword arguments.""" + payload = {**_base_payload(), "reasoning_parser_kwargs": {"tokenizer": None}} + with pytest.raises(ValidationError, match="tokenizer"): + GenerateRequest.model_validate(payload) + + def test_output_mode_defaults_to_tokens(): assert GenerateRequest.model_validate(_base_payload()).output_mode == "tokens" diff --git a/tests/entrypoints/scale_out/token_in_token_out/test_serving_multimodal_tokens.py b/tests/entrypoints/scale_out/token_in_token_out/test_serving_multimodal_tokens.py index 1d017bd3cf..2e58d70972 100644 --- a/tests/entrypoints/scale_out/token_in_token_out/test_serving_multimodal_tokens.py +++ b/tests/entrypoints/scale_out/token_in_token_out/test_serving_multimodal_tokens.py @@ -5,8 +5,8 @@ Mirrors test_serving_tokens.py but exercises the multimodal piping using Qwen/Qwen3-VL-2B-Instruct end-to-end via the server's /render -> /generate -> /detokenize path. Intentionally avoids running the HF -processor in the pytest parent process to keep os.fork() in sibling -tests (e.g. test_weight_transfer_llm.py) deadlock-free. +processor in the pytest parent process to keep os.fork() in other +tests (e.g. entrypoints/rl/weight_transfer/test_offline.py) deadlock-free. """ import httpx diff --git a/tests/entrypoints/scale_out/token_in_token_out/test_serving_tokens.py b/tests/entrypoints/scale_out/token_in_token_out/test_serving_tokens.py index 2ee8f76e0f..48b9dd5099 100644 --- a/tests/entrypoints/scale_out/token_in_token_out/test_serving_tokens.py +++ b/tests/entrypoints/scale_out/token_in_token_out/test_serving_tokens.py @@ -14,7 +14,6 @@ from transformers import AutoTokenizer import vllm.envs as envs from tests.utils import RemoteOpenAIServer from vllm.config import ModelConfig -from vllm.config.utils import getattr_iter from vllm.v1.engine.detokenizer import check_stop_strings MODEL_NAME = "Qwen/Qwen3-0.6B" @@ -183,7 +182,7 @@ async def test_generate_sampling_mask(client): assert token_id in support # processed_logprobs: exactly the support carries finite probability. for top in entry["top_logprobs"]: - in_support = int(top["token"].removeprefix("token_id:")) in support + in_support = top["token_id"] in support assert in_support == (top["logprob"] > -9999.0) @@ -402,14 +401,7 @@ async def test_same_response_as_chat_completions(client, tokenizer, messages): # Post-EOS generation is undefined and may differ eos_tokens = { tokenizer.eos_token_id, - *getattr_iter( - tokenizer, - [ - "extra_special_tokens_ids", # Transformers v5 - "additional_special_tokens_ids", # Transformers v4 - ], - [], - ), + *tokenizer.extra_special_tokens_ids, } # Find first EOS in generated tokens eos_pos = None @@ -589,6 +581,83 @@ async def test_generate_with_lora_adapter(client, tokenizer, messages): assert generate_res == completions_res +def _structured_chat_body(messages, **overrides) -> dict: + return { + "model": MODEL_NAME, + "messages": messages, + "max_tokens": 64, + "temperature": 0.0, + "chat_template_kwargs": {"enable_thinking": True}, + "response_format": { + "type": "json_schema", + "json_schema": { + "name": "answer", + "schema": { + "type": "object", + "properties": {"count": {"type": "integer"}}, + "required": ["count"], + }, + }, + }, + **overrides, + } + + +async def _render_then_generate(client, body) -> tuple[dict, list[int]]: + render_resp = await client.post("/v1/chat/completions/render", json=body) + render_resp.raise_for_status() + generate_request = render_resp.json() + generate_resp = await client.post(GEN_ENDPOINT, json=generate_request) + generate_resp.raise_for_status() + return generate_request, generate_resp.json()["choices"][0]["token_ids"] + + +# deepseek_v3 reads `enable_thinking` from the template kwargs and uses Qwen3's +# tokens, so the engine gates structured outputs differently when +# either reasoning field is dropped between render and generate. +@pytest.mark.asyncio +@pytest.mark.skipif( + envs.VLLM_USE_RUST_FRONTEND, + reason="the Rust frontend does not forward reasoning fields to the engine", +) +@pytest.mark.parametrize( + "server", [["--reasoning-parser", "deepseek_v3"]], indirect=True +) +async def test_render_then_generate_forwards_reasoning_parser_kwargs(client, messages): + """With thinking on, structured outputs must wait for the reasoning to + end on generate, the same as on chat.""" + body = _structured_chat_body(messages, return_token_ids=True) + chat_resp = await client.post("/v1/chat/completions", json=body) + chat_resp.raise_for_status() + + generate_request, token_ids = await _render_then_generate(client, body) + + assert generate_request["reasoning_ended"] is False + assert token_ids == chat_resp.json()["choices"][0]["token_ids"] + + +@pytest.mark.asyncio +@pytest.mark.skipif( + envs.VLLM_USE_RUST_FRONTEND, + reason="the Rust frontend does not forward reasoning fields to the engine", +) +@pytest.mark.parametrize( + "server", [["--reasoning-parser", "deepseek_v3"]], indirect=True +) +async def test_render_then_generate_forwards_reasoning_ended( + client, tokenizer, messages +): + """`include_reasoning=false` constrains from the first token on chat, so + generate must too, even though the engine's parser expects thinking.""" + body = _structured_chat_body(messages, include_reasoning=False) + + generate_request, token_ids = await _render_then_generate(client, body) + + assert generate_request["reasoning_ended"] is True + output = json.loads(tokenizer.decode(token_ids, skip_special_tokens=True)) + assert isinstance(output["count"], int) + + def _chat_prompt_token_ids(tokenizer, messages) -> list[int]: return tokenizer.apply_chat_template( messages, diff --git a/tests/entrypoints/scale_out/token_in_token_out/test_tokens_logprobs.py b/tests/entrypoints/scale_out/token_in_token_out/test_tokens_logprobs.py index c417a5fc2b..05dba167fe 100644 --- a/tests/entrypoints/scale_out/token_in_token_out/test_tokens_logprobs.py +++ b/tests/entrypoints/scale_out/token_in_token_out/test_tokens_logprobs.py @@ -1,22 +1,62 @@ # SPDX-License-Identifier: Apache-2.0 # SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import json from unittest.mock import MagicMock +import pytest +from pydantic import ValidationError + +from vllm.entrypoints.scale_out.token_in_token_out.protocol import GenerateLogProb from vllm.entrypoints.scale_out.token_in_token_out.serving import ServingTokens from vllm.logprobs import Logprob def test_top_logprobs_alternatives_have_own_token_ids(): - """Each top_logprobs alternative must carry its own token_id placeholder.""" + """Each top_logprobs alternative must carry its own integer token id.""" result = ServingTokens._create_tokens_logprobs( None, token_ids=[262], top_logprobs=[{262: Logprob(-0.1), 257: Logprob(-1.2), 428: Logprob(-2.3)}], num_output_top_logprobs=2, ) - tokens = {e.token for e in result.content[0].top_logprobs} - assert tokens == {"token_id:262", "token_id:257"}, f"got {tokens}" + token_ids = {e.token_id for e in result.content[0].top_logprobs} + assert token_ids == {262, 257}, f"got {token_ids}" + + +def test_sampled_entry_carries_token_id_and_rank(): + """The sampled token is identified by id, and the engine's rank survives.""" + result = ServingTokens._create_tokens_logprobs( + None, + token_ids=[262], + top_logprobs=[ + {262: Logprob(-0.1, rank=1), 257: Logprob(-1.2, rank=2)}, + ], + num_output_top_logprobs=2, + ) + entry = result.content[0] + assert entry.token_id == 262 + assert entry.logprob == -0.1 + assert entry.rank == 1 + assert [(t.token_id, t.rank) for t in entry.top_logprobs] == [(262, 1), (257, 2)] + # No tokenizer on the generate server: no string token, no bytes. + assert not hasattr(entry, "token") + assert not hasattr(entry, "bytes") + + +def test_sampled_token_absent_from_topk_uses_sentinel(): + """A sampled token missing from the engine's map keeps the -9999.0 sentinel.""" + result = ServingTokens._create_tokens_logprobs( + None, + token_ids=[5], + top_logprobs=[{7: Logprob(-0.9)}], + num_output_top_logprobs=1, + ) + entry = result.content[0] + assert entry.token_id == 5 + assert entry.logprob == -9999.0 + assert entry.rank is None + assert entry.top_logprobs == [] def test_logprobs_zero_emits_sampled_token(): @@ -40,21 +80,23 @@ def test_logprobs_minus_one_emits_all_tokens(): assert len(result.content[0].top_logprobs) == 2 -def test_tokens_level_logprobs_have_no_bytes(): - result = ServingTokens._create_tokens_logprobs( +def test_text_logprobs_without_tokenizer_are_placeholders_without_bytes(): + """--return-tokens-as-token-ids keeps `token_id:N` placeholders in text mode.""" + result = ServingTokens._create_text_logprobs( None, token_ids=[7], top_logprobs=[{7: Logprob(-0.9), 8: Logprob(-1.1)}], num_output_top_logprobs=2, ) entry = result.content[0] + assert entry.token == "token_id:7" assert entry.bytes is None assert all(t.bytes is None for t in entry.top_logprobs) def test_resolved_logprobs_use_engine_decoded_tokens_and_bytes(): tokenizer = MagicMock() - result = ServingTokens._create_tokens_logprobs( + result = ServingTokens._create_text_logprobs( None, token_ids=[7], top_logprobs=[ @@ -75,7 +117,7 @@ def test_resolved_logprobs_use_engine_decoded_tokens_and_bytes(): def test_resolved_logprobs_decode_tokens_the_engine_did_not(): tokenizer = MagicMock() tokenizer.decode.side_effect = lambda ids: f"<{ids[0]}>" - result = ServingTokens._create_tokens_logprobs( + result = ServingTokens._create_text_logprobs( None, token_ids=[7, 9], top_logprobs=[{7: Logprob(-0.9)}, None], @@ -85,3 +127,59 @@ def test_resolved_logprobs_decode_tokens_the_engine_did_not(): assert [e.token for e in result.content] == ["<7>", "<9>"] assert result.content[0].top_logprobs[0].token == "<7>" assert result.content[1].bytes == list(b"<9>") + + +def test_sampled_token_outside_topk_comes_first(): + """The engine puts the sampled token first, then ranks 1..k. When the + sampled token is outside the top k it takes one of the k slots, so rank k + is left out (same as the OpenAI endpoints).""" + result = ServingTokens._create_tokens_logprobs( + None, + token_ids=[50], + top_logprobs=[ + { + 50: Logprob(-3.0, rank=5), + 10: Logprob(-0.2, rank=1), + 20: Logprob(-1.0, rank=2), + } + ], + num_output_top_logprobs=2, + ) + assert [t.rank for t in result.content[0].top_logprobs] == [5, 1] + + +def test_logprob_is_required_on_the_wire(): + """A payload missing logprob is rejected rather than read as -9999.""" + with pytest.raises(ValidationError): + GenerateLogProb.model_validate({"token_id": 1}) + + +def test_rank_zero_from_a_nan_logprob_is_none(): + """The engine reports rank 0 for a sampled token whose logprob is NaN.""" + result = ServingTokens._create_tokens_logprobs( + None, + token_ids=[7], + top_logprobs=[{7: Logprob(float("nan"), rank=0), 8: Logprob(-0.2, rank=1)}], + num_output_top_logprobs=1, + ) + entry = result.content[0] + assert entry.rank is None + assert entry.top_logprobs[0].rank is None + # Clamped like the Rust frontend (max(nan, x) is nan in Python). + assert entry.logprob == -9999.0 + assert entry.top_logprobs[0].logprob == -9999.0 + json.dumps(result.model_dump(), allow_nan=False) + + +def test_text_logprobs_clamp_a_nan_logprob(): + result = ServingTokens._create_text_logprobs( + None, + token_ids=[7], + top_logprobs=[{7: Logprob(float("nan"), rank=0, decoded_token="a")}], + num_output_top_logprobs=1, + tokenizer=MagicMock(), + ) + entry = result.content[0] + assert entry.logprob == -9999.0 + assert entry.top_logprobs[0].logprob == -9999.0 + json.dumps(result.model_dump(), allow_nan=False) diff --git a/tests/entrypoints/serve/lora/test_lora_adapters.py b/tests/entrypoints/serve/lora/test_lora_adapters.py index 8a860dc625..484fb9c0f1 100644 --- a/tests/entrypoints/serve/lora/test_lora_adapters.py +++ b/tests/entrypoints/serve/lora/test_lora_adapters.py @@ -23,6 +23,7 @@ BADREQUEST_CASES = [ "is greater than max_lora_rank", ), ("test_dora", {"use_dora": True}, "does not yet support DoRA"), + ("test_pissa", {"init_lora_weights": "pissa"}, "init_lora_weights='pissa'"), ( "test_modules_to_save", {"modules_to_save": ["lm_head"]}, diff --git a/tests/entrypoints/serve/test_profile_api_router.py b/tests/entrypoints/serve/test_profile_api_router.py new file mode 100644 index 0000000000..c5ceddd1bc --- /dev/null +++ b/tests/entrypoints/serve/test_profile_api_router.py @@ -0,0 +1,86 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +from unittest.mock import AsyncMock + +import pytest +from fastapi import FastAPI +from fastapi.testclient import TestClient + +from vllm.entrypoints.serve.profile.api_router import router + + +@pytest.fixture +def engine_client() -> AsyncMock: + return AsyncMock() + + +@pytest.fixture +def client(engine_client: AsyncMock) -> TestClient: + app = FastAPI() + app.state.engine_client = engine_client + app.include_router(router) + return TestClient(app) + + +def test_start_profile_without_body_is_backwards_compatible( + client: TestClient, engine_client: AsyncMock +) -> None: + response = client.post("/start_profile") + + assert response.status_code == 200 + engine_client.start_profile.assert_awaited_once_with() + + +def test_start_profile_ignores_completion_payload( + client: TestClient, engine_client: AsyncMock +) -> None: + response = client.post( + "/start_profile", + json={ + "model": "Qwen/Qwen3-0.6B", + "prompt": "profile this request", + "max_tokens": 16, + "stream": False, + }, + ) + + assert response.status_code == 200 + engine_client.start_profile.assert_awaited_once_with() + + +def test_start_profile_forwards_session_overrides( + client: TestClient, engine_client: AsyncMock +) -> None: + response = client.post( + "/start_profile", + json={ + "profile_prefix": "sharegpt_run-1", + "delay_iterations": 5000, + "max_iterations": 20, + }, + ) + + assert response.status_code == 200 + engine_client.start_profile.assert_awaited_once_with( + profile_prefix="sharegpt_run-1", + delay_iterations=5000, + max_iterations=20, + ) + + +@pytest.mark.parametrize( + "body", + [ + {"profile_prefix": 1}, + {"delay_iterations": "1"}, + {"max_iterations": "20"}, + ], +) +def test_start_profile_rejects_wrongly_typed_overrides( + client: TestClient, engine_client: AsyncMock, body: dict +) -> None: + response = client.post("/start_profile", json=body) + + assert response.status_code == 422 + engine_client.start_profile.assert_not_awaited() diff --git a/tests/entrypoints/serve/utils/test_request_logger.py b/tests/entrypoints/serve/utils/test_request_logger.py index 717ff190b3..461b954cae 100644 --- a/tests/entrypoints/serve/utils/test_request_logger.py +++ b/tests/entrypoints/serve/utils/test_request_logger.py @@ -4,6 +4,9 @@ import logging from unittest.mock import MagicMock, patch +import pytest + +from vllm.entrypoints.openai.completion.protocol import CompletionRequest from vllm.entrypoints.serve.utils.request_logger import RequestLogger @@ -269,3 +272,18 @@ def test_streaming_complete_logs_full_text_content(): assert call_args[1] == "test-streaming-full-text" assert call_args[2] == " (streaming complete)" assert call_args[4] == "streaming_complete" + + +@pytest.mark.parametrize("max_log_len", [None, 10]) +def test_request_logger_log_request_body_truncation(max_log_len): + """The JSON body is logged in full unless max_log_len truncates it.""" + request = CompletionRequest(model="test-model", prompt="Hello, world!") + full_body = request.model_dump_json(exclude_unset=True) + mock_logger = MagicMock() + + with patch("vllm.entrypoints.serve.utils.request_logger.logger", mock_logger): + RequestLogger(max_log_len=max_log_len).log_request_body(request) + + mock_logger.debug.assert_called_once() + logged_body = mock_logger.debug.call_args.args[2] + assert logged_body == full_body[:max_log_len] diff --git a/tests/entrypoints/speech_to_text/realtime/test_realtime_validation.py b/tests/entrypoints/speech_to_text/realtime/test_realtime_validation.py index c74246b2f8..a8c4f35c04 100644 --- a/tests/entrypoints/speech_to_text/realtime/test_realtime_validation.py +++ b/tests/entrypoints/speech_to_text/realtime/test_realtime_validation.py @@ -4,6 +4,7 @@ import asyncio import json import warnings +from types import SimpleNamespace import numpy as np import pybase64 as base64 @@ -12,6 +13,7 @@ import websockets from tests.utils import ROCM_EXTRA_ARGS, RemoteOpenAIServer from vllm.assets.audio import AudioAsset +from vllm.entrypoints.speech_to_text.realtime.connection import RealtimeConnection from vllm.multimodal.media.audio import load_audio MISTRAL_FORMAT_ARGS = [ @@ -26,6 +28,46 @@ MISTRAL_FORMAT_ARGS = [ MODEL_NAME = "mistralai/Voxtral-Mini-4B-Realtime-2602" +def test_realtime_generation_preserves_watermarking(): + transcription_outputs = [] + + class EngineClient: + def generate(self, **kwargs): + async def outputs(): + output = SimpleNamespace( + outputs=[], + prompt_token_ids=[], + is_watermarked=kwargs["sampling_params"].watermarking, + ) + transcription_outputs.append(output) + yield output + + return outputs() + + async def streaming_input(): + chunk: np.ndarray + for chunk in (): + yield chunk + + async def send(_event): + return None + + connection = object.__new__(RealtimeConnection) + connection.connection_id = "test" + connection.serving = SimpleNamespace( + model_cls=SimpleNamespace(realtime_max_tokens=1), + engine_client=EngineClient(), + ) + connection.audio_queue = asyncio.Queue() + connection._is_connected = True + connection.send = send + + asyncio.run(connection._run_generation(streaming_input(), asyncio.Queue())) + + assert len(transcription_outputs) == 1 + assert transcription_outputs[0].is_watermarked is None + + def _get_websocket_url(server: RemoteOpenAIServer) -> str: """Convert HTTP URL to WebSocket URL for realtime endpoint.""" http_url = server.url_root diff --git a/tests/entrypoints/speech_to_text/test_speech_to_text_cancellation.py b/tests/entrypoints/speech_to_text/test_speech_to_text_cancellation.py index 0ff8adb6b5..8ed31e4cdb 100644 --- a/tests/entrypoints/speech_to_text/test_speech_to_text_cancellation.py +++ b/tests/entrypoints/speech_to_text/test_speech_to_text_cancellation.py @@ -198,4 +198,6 @@ async def test_language_detection_cancel_aborts_engine_request(): with pytest.raises(asyncio.CancelledError): await task + sampling_params = engine_client.generate.call_args.args[1] + assert sampling_params.watermarking is False engine_client.abort.assert_awaited_once_with(request_id) diff --git a/tests/entrypoints/tool_parsers/test_granite4_tool_parser.py b/tests/entrypoints/tool_parsers/test_granite4_tool_parser.py index 71fe7637bf..67826d9564 100644 --- a/tests/entrypoints/tool_parsers/test_granite4_tool_parser.py +++ b/tests/entrypoints/tool_parsers/test_granite4_tool_parser.py @@ -24,6 +24,10 @@ def server(): "4096", "--max-num-seqs", "2", + # Streaming and non-streaming runs must match bitwise; a prefix-cache + # hit resumes the second from a mid-prefill Mamba checkpoint, whose + # numerics differ slightly from the cold single-pass prefill. + "--no-enable-prefix-caching", ] with RemoteOpenAIServer(model, args_for_model, max_wait_seconds=480) as server: yield server diff --git a/tests/entrypoints/unit_tests/test_non_object_body_validation.py b/tests/entrypoints/unit_tests/test_non_object_body_validation.py index 5bdc1d5787..2440c880ce 100644 --- a/tests/entrypoints/unit_tests/test_non_object_body_validation.py +++ b/tests/entrypoints/unit_tests/test_non_object_body_validation.py @@ -13,6 +13,12 @@ request models whose before-validators were missing the same guard. import pytest from pydantic import ValidationError +from vllm.entrypoints.generate.generative_scoring.serving import ( + GenerativeScoringRequest, +) +from vllm.entrypoints.generate.structured_decisions.protocol import ( + StructuredDecisionRequest, +) from vllm.entrypoints.openai.chat_completion.protocol import ChatCompletionRequest from vllm.entrypoints.openai.completion.protocol import CompletionRequest from vllm.entrypoints.openai.responses.protocol import ResponsesRequest @@ -28,6 +34,7 @@ from vllm.exceptions import VLLMValidationError pytestmark = pytest.mark.skip_global_cleanup REQUEST_MODELS = [ + GenerativeScoringRequest, CompletionRequest, ResponsesRequest, EmbeddingChatRequest, @@ -74,6 +81,26 @@ def test_tokenize_chat_request_still_validates_dict_bodies(): CACHE_SALT_MODELS = [ + ( + GenerativeScoringRequest, + { + "query": "Is this city the capital of France?", + "items": ["Paris"], + "label_token_ids": [9454, 2753], + }, + ), + ( + StructuredDecisionRequest, + { + "state": "hello", + "questions": { + "lang": { + "type": "choice", + "criteria": {"English": "Text written in English."}, + } + }, + }, + ), (CompletionRequest, {"prompt": "hello"}), (ChatCompletionRequest, {"messages": [{"role": "user", "content": "hello"}]}), (ResponsesRequest, {"input": "hello"}), diff --git a/tests/entrypoints/unit_tests/test_offline_utils.py b/tests/entrypoints/unit_tests/test_offline_utils.py index 39f77672a4..a37969c9b3 100644 --- a/tests/entrypoints/unit_tests/test_offline_utils.py +++ b/tests/entrypoints/unit_tests/test_offline_utils.py @@ -4,7 +4,7 @@ import pytest from vllm import SamplingParams -from vllm.entrypoints.offline_utils import OfflineInferenceMixin +from vllm.entrypoints.common.offline import OfflineInferenceMixin from vllm.exceptions import VLLMValidationError from vllm.lora.request import LoRARequest diff --git a/tests/evals/gsm8k/README.md b/tests/evals/gsm8k/README.md index db37d2e224..ea61b6bf45 100644 --- a/tests/evals/gsm8k/README.md +++ b/tests/evals/gsm8k/README.md @@ -21,6 +21,22 @@ vllm serve Qwen/Qwen2.5-1.5B-Instruct --port 8000 python tests/evals/gsm8k/gsm8k_eval.py --port 8000 ``` +For chat-only models, send the prompts through `/v1/chat/completions`: + +```bash +python tests/evals/gsm8k/gsm8k_eval.py --port 8000 \ + --use-chat-completions --max-tokens 1024 \ + --chat-template-kwargs '{"enable_thinking": false}' +``` + +The server applies the chat template and uses its served model, so `--model` is +optional. In chat mode, `temperature` is sent only when `--temperature` is given, +so the server's default sampling applies otherwise; completions mode keeps the +default of 0. Chat mode sends no stop strings; completions mode keeps them. +`--reasoning-effort`, `--top-p`, and `--top-k` set the matching request fields +when given. With a reasoning parser, only the final answer `content` is scored. +`--save-results` records the request settings used. + ## Configuration Format Model configs in `configs/` directory use this YAML format: diff --git a/tests/evals/gsm8k/configs/DeepSeek-V4.1-Flash-NVFP4-DSpark-TP4.yaml b/tests/evals/gsm8k/configs/DeepSeek-V4.1-Flash-NVFP4-DSpark-TP4.yaml new file mode 100644 index 0000000000..78d8995696 --- /dev/null +++ b/tests/evals/gsm8k/configs/DeepSeek-V4.1-Flash-NVFP4-DSpark-TP4.yaml @@ -0,0 +1,15 @@ +model_name: "nvidia/DeepSeek-V4.1-Flash-NVFP4" +accuracy_threshold: 0.88 +tolerance: 0.03 +# The bundled DSpark draft (mtp.*) stays MXFP4 in this checkpoint. Loading it +# as NVFP4 keeps accuracy but drops acceptance length to 1.0 (3.57 when correct). +min_acceptance_length: 2.5 +num_questions: 1319 +num_fewshot: 5 +startup_max_wait_seconds: 1800 +server_args: >- + --tokenizer-mode deepseek_v41 + --trust-remote-code + --max-model-len 8192 + --tensor-parallel-size 4 + --speculative-config '{"method":"dspark","num_speculative_tokens":5}' diff --git a/tests/evals/gsm8k/configs/Nemotron-3-Super-120B-A12B-NVFP4-per-token.yaml b/tests/evals/gsm8k/configs/Nemotron-3-Super-120B-A12B-NVFP4-per-token.yaml new file mode 100644 index 0000000000..62587ae10a --- /dev/null +++ b/tests/evals/gsm8k/configs/Nemotron-3-Super-120B-A12B-NVFP4-per-token.yaml @@ -0,0 +1,14 @@ +model_name: "nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4" +accuracy_threshold: 0.93 +num_questions: 1319 +num_fewshot: 5 +startup_max_wait_seconds: 1200 +server_args: >- + --enforce-eager + --max-model-len 4096 + --tensor-parallel-size 2 + --enable-expert-parallel + --mamba-backend flashinfer + --moe-backend flashinfer_trtllm + --quantization-config.moe.activation nvfp4_per_token + --speculative-config '{"method":"mtp","num_speculative_tokens":5}' diff --git a/tests/evals/gsm8k/configs/models-blackwell-ep.txt b/tests/evals/gsm8k/configs/models-blackwell-ep.txt index c2a1905198..34090d7205 100644 --- a/tests/evals/gsm8k/configs/models-blackwell-ep.txt +++ b/tests/evals/gsm8k/configs/models-blackwell-ep.txt @@ -1,3 +1,4 @@ Qwen3-Next-80B-A3B-NVFP4-EP2.yaml Qwen3-Next-FP8-EP2.yaml Nemotron-3-Super-120B-A12B-NVFP4.yaml +Nemotron-3-Super-120B-A12B-NVFP4-per-token.yaml diff --git a/tests/evals/gsm8k/configs/models-spec-decode.txt b/tests/evals/gsm8k/configs/models-spec-decode.txt index 427c35d912..96cead2585 100644 --- a/tests/evals/gsm8k/configs/models-spec-decode.txt +++ b/tests/evals/gsm8k/configs/models-spec-decode.txt @@ -1,2 +1,3 @@ DeepSeek-V4-Flash-DSpark-confidence-TEP4.yaml Kimi-K3-pruned75-DSpark-TP4.yaml +DeepSeek-V4.1-Flash-NVFP4-DSpark-TP4.yaml diff --git a/tests/evals/gsm8k/gsm8k_eval.py b/tests/evals/gsm8k/gsm8k_eval.py index 7e2534b55a..4eec87945b 100644 --- a/tests/evals/gsm8k/gsm8k_eval.py +++ b/tests/evals/gsm8k/gsm8k_eval.py @@ -76,14 +76,20 @@ def get_answer_value(answer_str: str) -> int: return INVALID +def _optional_params(**params: object) -> dict[str, object]: + return {k: v for k, v in params.items() if v is not None} + + async def call_vllm_api( session: aiohttp.ClientSession, prompt: str, - temperature: float, + temperature: float | None, max_tokens: int, stop: list[str] | None = None, url: str | None = None, seed: int | None = None, + top_p: float | None = None, + top_k: int | None = None, ) -> tuple[str, int]: """Call vLLM's OpenAI-compatible completions endpoint. @@ -93,12 +99,12 @@ async def call_vllm_api( """ data = { "prompt": prompt, - "temperature": temperature, "max_tokens": max_tokens, "stop": stop, } - if seed is not None: - data["seed"] = seed + data.update( + _optional_params(temperature=temperature, seed=seed, top_p=top_p, top_k=top_k) + ) try: async with session.post(f"{url}/v1/completions", json=data) as response: @@ -114,24 +120,45 @@ async def call_vllm_api( async def call_vllm_chat_api( session: aiohttp.ClientSession, - model: str, + model: str | None, prompt: str, - temperature: float, + temperature: float | None, max_tokens: int, stop: list[str] | None = None, url: str | None = None, seed: int | None = None, + top_p: float | None = None, + top_k: int | None = None, + reasoning_effort: str | None = None, + chat_template_kwargs: dict[str, object] | None = None, ) -> tuple[str, int]: - """Call vLLM's OpenAI-compatible chat completions endpoint.""" + """Call vLLM's OpenAI-compatible chat completions endpoint. + + ``model``, ``temperature``, ``stop`` and the other optional fields are + omitted from the request when ``None``, so the server defaults apply (vLLM + uses its served model when ``model`` is omitted). + + Returns: + Tuple of (final answer content, completion_tokens). Reasoning returned + separately by a reasoning parser is not part of the answer. + + """ data = { - "model": model, "messages": [{"role": "user", "content": prompt}], - "temperature": temperature, "max_tokens": max_tokens, - "stop": stop, } - if seed is not None: - data["seed"] = seed + data.update( + _optional_params( + stop=stop, + model=model, + temperature=temperature, + seed=seed, + top_p=top_p, + top_k=top_k, + reasoning_effort=reasoning_effort, + chat_template_kwargs=chat_template_kwargs, + ) + ) try: async with session.post(f"{url}/v1/chat/completions", json=data) as response: @@ -214,16 +241,34 @@ def evaluate_gsm8k( use_chat_completions: bool = False, host: str = "http://127.0.0.1", port: int = 8000, - temperature: float = 0.0, + temperature: float | None = 0.0, seed: int | None = 42, request_timeout_seconds: float = 600, gen_prefix: str = "", max_concurrency: int | None = None, + top_p: float | None = None, + top_k: int | None = None, + reasoning_effort: str | None = None, + chat_template_kwargs: dict[str, object] | None = None, ) -> dict[str, float | int]: """Evaluate GSM8K accuracy using vLLM serve endpoint. + ``temperature``, ``top_p`` and ``top_k`` are sent only when not ``None``; + otherwise the server default applies. ``model`` is used only in chat mode, + where it may be ``None`` (the server uses its served model). + ``reasoning_effort`` and ``chat_template_kwargs`` require + ``use_chat_completions=True``. Stop strings are sent only in completions + mode. + Returns dict with accuracy, invalid_rate, latency, etc. """ + if not use_chat_completions and ( + reasoning_effort is not None or chat_template_kwargs is not None + ): + raise ValueError( + "reasoning_effort and chat_template_kwargs require " + "use_chat_completions=True" + ) base_url = f"{host}:{port}" prompts, labels = _build_gsm8k_prompts(num_questions, num_shots, gen_prefix) num_questions = len(prompts) @@ -233,19 +278,21 @@ def evaluate_gsm8k( output_tokens: list[int] = [0] * num_questions async def get_answer(session: aiohttp.ClientSession, i: int) -> tuple[str, int]: - stop = ["Question", "Assistant:", "<|separator|>"] if use_chat_completions: - if model is None: - raise ValueError("model is required for chat completions") + # No stop strings: the chat template delimits the turn, and they + # would also cut off reasoning. answer, tokens = await call_vllm_chat_api( session=session, model=model, prompt=prompts[i], temperature=temperature, max_tokens=max_tokens, - stop=stop, url=base_url, seed=seed, + top_p=top_p, + top_k=top_k, + reasoning_effort=reasoning_effort, + chat_template_kwargs=chat_template_kwargs, ) else: answer, tokens = await call_vllm_api( @@ -253,9 +300,11 @@ def evaluate_gsm8k( prompt=prompts[i], temperature=temperature, max_tokens=max_tokens, - stop=stop, + stop=["Question", "Assistant:", "<|separator|>"], url=base_url, seed=seed, + top_p=top_p, + top_k=top_k, ) states[i] = answer output_tokens[i] = tokens @@ -352,7 +401,38 @@ def main() -> None: parser.add_argument("--host", type=str, default="http://127.0.0.1", help="Host URL") parser.add_argument("--port", type=int, default=8000, help="Port number") parser.add_argument( - "--temperature", type=float, default=0.0, help="Temperature for generation" + "--temperature", + type=float, + help="Temperature for generation (default: 0 for completions; " + "server default with --use-chat-completions)", + ) + parser.add_argument( + "--top-p", type=float, help="Top-p for generation (server default if unset)" + ) + parser.add_argument( + "--top-k", type=int, help="Top-k for generation (server default if unset)" + ) + parser.add_argument( + "--use-chat-completions", + action="store_true", + help="Send each few-shot prompt as a user message to /v1/chat/completions " + "instead of /v1/completions (for chat-only models)", + ) + parser.add_argument( + "--model", + type=str, + help="Model name to send with chat requests (default: omitted, so the " + "server uses its served model)", + ) + parser.add_argument( + "--reasoning-effort", + type=str, + help="reasoning_effort for chat completions, e.g. none, low, high", + ) + parser.add_argument( + "--chat-template-kwargs", + type=json.loads, + help="JSON chat_template_kwargs, e.g. '{\"enable_thinking\": false}'", ) parser.add_argument( "--seed", type=int, default=42, help="Random seed for reproducibility" @@ -371,6 +451,9 @@ def main() -> None: parser.add_argument("--save-results", type=str, help="Save results to JSON file") args = parser.parse_args() + temperature = args.temperature + if temperature is None and not args.use_chat_completions: + temperature = 0.0 result = evaluate_gsm8k( num_questions=args.num_questions, @@ -378,10 +461,16 @@ def main() -> None: max_tokens=args.max_tokens, host=args.host, port=args.port, - temperature=args.temperature, + temperature=temperature, seed=args.seed, max_concurrency=args.max_concurrency, request_timeout_seconds=args.request_timeout_seconds, + model=args.model, + use_chat_completions=args.use_chat_completions, + top_p=args.top_p, + top_k=args.top_k, + reasoning_effort=args.reasoning_effort, + chat_template_kwargs=args.chat_template_kwargs, ) # Print results to terminal @@ -395,6 +484,20 @@ def main() -> None: # Optional file saving if args.save_results: + # None means the field was not sent and the server default applied. + result["request_params"] = { + "endpoint": "chat" if args.use_chat_completions else "completions", + "model": args.model, + "temperature": temperature, + "top_p": args.top_p, + "top_k": args.top_k, + "seed": args.seed, + "stop": None + if args.use_chat_completions + else ["Question", "Assistant:", "<|separator|>"], + "reasoning_effort": args.reasoning_effort, + "chat_template_kwargs": args.chat_template_kwargs, + } with open(args.save_results, "w") as f: json.dump(result, f, indent=2) print(f"Results saved to {args.save_results}") diff --git a/tests/fusion/test_quant_activation_contract.py b/tests/fusion/test_quant_activation_contract.py index 15b9835d6c..da507e5caf 100644 --- a/tests/fusion/test_quant_activation_contract.py +++ b/tests/fusion/test_quant_activation_contract.py @@ -21,6 +21,7 @@ from vllm.model_executor.kernels.linear.nvfp4.base import ( NvFp4LinearLayerConfig, ) from vllm.model_executor.kernels.linear.nvfp4.flashinfer import ( + FlashInferCuteDslNvFp4LinearKernel, FlashInferCutlassNvFp4LinearKernel, FlashInferTrtllmNvFp4LinearKernel, ) @@ -59,6 +60,7 @@ from vllm.platforms import current_platform SUPPORTING = { CutlassFP8ScaledMMLinearKernel, FlashInferFP8ScaledMMLinearKernel, + FlashInferCuteDslNvFp4LinearKernel, FlashInferCutlassNvFp4LinearKernel, PerTensorTorchFP8ScaledMMLinearKernel, AiterHipbMMPerTokenFp8ScaledMMLinearKernel, diff --git a/tests/jit_monitor/test_hooks.py b/tests/jit_monitor/test_hooks.py index 452fe96995..6d4f25314c 100644 --- a/tests/jit_monitor/test_hooks.py +++ b/tests/jit_monitor/test_hooks.py @@ -65,10 +65,13 @@ def _fake_tilelang_import_modules(): key, _ = self.func.parse_args(*args, **kwargs) kernel = self._kernel_cache.get(key) if kernel is None: - kernel = "compiled" + kernel = self.compile(*args, **kwargs) self._kernel_cache[key] = kernel return kernel + def compile(self, *args, **kwargs): + return "compiled" + fake_kernel = cast(Any, ModuleType("tilelang.jit.kernel")) fake_kernel.JITKernel = FakeJITKernel diff --git a/tests/kernels/attention/test_cpu_attn.py b/tests/kernels/attention/test_cpu_attn.py index f79b1b15d2..7f88f76d0b 100644 --- a/tests/kernels/attention/test_cpu_attn.py +++ b/tests/kernels/attention/test_cpu_attn.py @@ -472,7 +472,7 @@ def varlen_with_paged_kv( scale = head_size**-0.5 token_num = sum(query_lens) dynamic_causal_tensor = ( - torch.tensor(dynamic_causal, dtype=torch.bool) + torch.tensor(dynamic_causal, dtype=torch.int32) if dynamic_causal is not None else None ) diff --git a/tests/kernels/attention/test_flashinfer.py b/tests/kernels/attention/test_flashinfer.py index c6cba8d530..c56af3a6e8 100644 --- a/tests/kernels/attention/test_flashinfer.py +++ b/tests/kernels/attention/test_flashinfer.py @@ -6,6 +6,7 @@ import pytest from vllm.platforms import current_platform from vllm.utils.torch_utils import set_random_seed +from vllm.v1.kv_cache_layout import KVCacheLayout try: import flashinfer @@ -702,3 +703,35 @@ def test_flashinfer_decode_with_paged_fp8_kv( torch.testing.assert_close(output, ref_output, atol=2e-2, rtol=1e-2), f"{torch.max(torch.abs(output - ref_output))}", ) + + +@pytest.mark.skipif( + not any(current_platform.is_device_capability_family(f) for f in (80, 90, 120)), + reason="NVFP4 fa2 path", +) +@pytest.mark.parametrize("layout", ["LBNHC", "LBHNC", "BLNHC", "BLHNC"]) +def test_flashinfer_nvfp4_kv_cache_update(default_vllm_config, layout): + from vllm.v1.attention.backends import flashinfer as fi + + torch.set_default_device("cuda") + order = KVCacheLayout[layout].stride_order + # [L, B, 2H, N, 64 + 8] NVFP4 pages for two layers; layer 1 is written. + shape = (2, 6, 4, 16, 64 + 8) + layers = torch.zeros([shape[i] for i in order], dtype=torch.uint8) + layers = layers.permute(*(order.index(i) for i in range(5))) + one = torch.ones(()) + layer = torch.nn.Module() + layer._k_scale = layer._v_scale = one + kv = torch.randn(2, 96, 2, 128, dtype=torch.bfloat16) + impl = fi.FlashInferImpl(4, 128, 1.0, 2, None, None, "nvfp4") + impl.do_kv_cache_update(layer, kv[0], kv[1], layers[1], torch.arange(96)) + + data, scales, kv_layout = fi._nvfp4_kv_views(layers[1], 2) + pages = torch.arange(6, dtype=torch.int32)[None] + seq_lens = torch.tensor([96], dtype=torch.int32) + deq = torch.empty_like(kv)[:, None] + flashinfer.nvfp4_kv_dequantize_paged( + data, scales, pages, seq_lens, one, one, *deq, kv_layout + ) + assert ((deq[:, 0] - kv).norm() / kv.norm()) < 0.2 + assert not layers[0].any() diff --git a/tests/kernels/attention/test_minimax_m3.py b/tests/kernels/attention/test_minimax_m3.py index 83eef72e31..7ee900fd3d 100644 --- a/tests/kernels/attention/test_minimax_m3.py +++ b/tests/kernels/attention/test_minimax_m3.py @@ -2,6 +2,8 @@ # SPDX-FileCopyrightText: Copyright contributors to the vLLM project """Correctness tests for MiniMax M3 sparse prefill attention kernels.""" +from functools import partial + import pytest import torch @@ -1444,19 +1446,21 @@ def test_decode_index_topk_fp8(num_idx_heads: int): reason="CuteDSL index decode score requires Blackwell.", ) @pytest.mark.parametrize( - ("dtype", "decode_query_len", "max_decode_query_len"), + ("dtype", "decode_query_len", "max_decode_query_len", "concurrent_writes"), [ - (torch.bfloat16, 1, 1), - (torch.bfloat16, 3, 8), - (torch.float8_e4m3fn, 1, 1), - (torch.float8_e4m3fn, 3, 8), - (torch.float8_e4m3fn, 8, 8), + (torch.bfloat16, 1, 1, False), + (torch.bfloat16, 3, 8, False), + (torch.float8_e4m3fn, 1, 1, False), + (torch.float8_e4m3fn, 3, 8, False), + (torch.float8_e4m3fn, 8, 8, False), + pytest.param(torch.float8_e4m3fn, 1, 1, True, id="concurrent-writes"), ], ) def test_decode_index_score_cutedsl_correctness( dtype: torch.dtype, decode_query_len: int, max_decode_query_len: int, + concurrent_writes: bool, ): pytest.importorskip("cutlass") from vllm.models.minimax_m3.nvidia.ops import ( @@ -1466,7 +1470,8 @@ def test_decode_index_score_cutedsl_correctness( torch.manual_seed(0) init_blocks, local_blocks = 0, 0 num_idx_heads, head_dim = 4, 128 - active_seq_lens = torch.tensor((1025, 4097), device="cuda", dtype=torch.int32) + lengths = (131326, 131289, 131252, 131215) if concurrent_writes else (1025, 4097) + active_seq_lens = torch.tensor(lengths, device="cuda", dtype=torch.int32) batch = active_seq_lens.numel() total_q = batch * decode_query_len max_seq_len = int(active_seq_lens.max()) @@ -1488,7 +1493,8 @@ def test_decode_index_score_cutedsl_correctness( ) score = unified_score.transpose(0, 1) - minimax_m3_index_decode_score_cutedsl( + launch = partial( + minimax_m3_index_decode_score_cutedsl, idx_q, index_kv_cache, block_table, @@ -1501,6 +1507,7 @@ def test_decode_index_score_cutedsl_correctness( max_decode_query_len=max_decode_query_len, score_out=score, ) + launch() expected = _reference_decode_index_score( idx_q, index_kv_cache, @@ -1511,6 +1518,23 @@ def test_decode_index_score_cutedsl_correctness( ) torch.testing.assert_close(score, expected) + if concurrent_writes: + # Concurrent stores expose shared-memory reuse before ldmatrix reads finish. + quiet = score.clone() + noise = torch.zeros(1 << 26, device="cuda") + side = torch.cuda.Stream() + side.wait_stream(torch.cuda.current_stream()) + try: + for _ in range(20): + with torch.cuda.stream(side): + for _ in range(40): + noise.mul_(1.0) + launch() + torch.testing.assert_close(score, quiet, atol=0, rtol=0) + side.synchronize() + finally: + side.synchronize() + # Sparse attention kernels. def _reference_sparse_attn( @@ -2588,6 +2612,103 @@ def test_decode_sparse_attention_correctness( assert error.max().item() < 1.7e-2 +@pytest.mark.parametrize("kv_layout", ["NHD", "HND"], indirect=True) +@pytest.mark.parametrize("capture_graph", [False, True]) +@pytest.mark.parametrize("num_reqs", [2, 17], ids=["split-k", "single-chunk"]) +@pytest.mark.parametrize( + "invalid_block", [-1, 1_000_000], ids=["sentinel", "out-of-range"] +) +def test_decode_sparse_attention_ignores_invalid_topk( + kv_layout: KVCacheLayout, capture_graph: bool, num_reqs: int, invalid_block: int +): + """Unused top-k entries must not be dereferenced as block-table indices.""" + torch.manual_seed(0) + decode_query_len = 4 + seq_lens = torch.full( + (num_reqs,), decode_query_len, device="cuda", dtype=torch.int32 + ) + seq_lens[1] = 4096 + q = torch.randn( + seq_lens.numel() * decode_query_len, + NUM_Q_HEADS, + HEAD_DIM, + device="cuda", + dtype=DTYPE, + ) + kv_cache = _allocate_main_kv_via_contract(1, kv_layout) + + # All logical blocks alias the single valid page. With a contiguous multi-row + # table, row 1's raw pointer offset -1 resolves to the final element of row + # 0. Poison that exact location so an unsafe load deterministically forms an + # out-of-range KV-cache pointer. + clean_block_table = torch.zeros(num_reqs, 32, device="cuda", dtype=torch.int32) + block_table = clean_block_table.clone() + block_table[0, -1] = 1_000_000 + topk_idx = torch.full( + (NUM_KV_HEADS, q.shape[0], TOPK), + -1, + device="cuda", + dtype=torch.int32, + ) + topk_idx[..., 0] = 0 + # Request 1 has invalid entries both before and after its valid block. + # Two requests produce empty split-K chunks; 17 put the full top-k in one + # chunk, exercising neutral updates before and after a nonempty softmax. + request_topk = topk_idx[:, decode_query_len : 2 * decode_query_len] + request_topk.fill_(invalid_block) + request_topk[..., TOPK // 2] = 0 + + # A valid-only run isolates sentinel handling from BF16-vs-FP32 rounding. + # Include block 0 exactly once; replacing sentinels with 0 would duplicate + # its contribution. The general correctness test covers reference accuracy. + valid_topk = torch.zeros( + (NUM_KV_HEADS, q.shape[0], 1), device="cuda", dtype=torch.int32 + ) + expected = torch.empty_like(q) + minimax_m3_sparse_attn_decode( + q, + kv_cache, + valid_topk, + clean_block_table, + seq_lens, + NUM_KV_HEADS, + SM_SCALE, + expected, + decode_query_len, + ) + actual = torch.empty_like(q) + + def run_decode() -> None: + minimax_m3_sparse_attn_decode( + q, + kv_cache, + topk_idx, + block_table, + seq_lens, + NUM_KV_HEADS, + SM_SCALE, + actual, + decode_query_len, + ) + + if capture_graph: + # An unfixed kernel faults during capture and leaves the CUDA context + # unusable. Run negative-control reproductions in a dedicated process. + run_decode() + current_platform.synchronize() + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + run_decode() + for _ in range(3): + graph.replay() + else: + run_decode() + current_platform.synchronize() + + assert torch.isfinite(actual).all() + torch.testing.assert_close(actual, expected, rtol=0, atol=0) + + def test_decode_wrong_layout_breaks_parity(): """Negative (AC-3/AC-5): consuming the physical HND buffer as if it were already contiguous-NHD (i.e. skipping the allocator's inverse permute) diff --git a/tests/kernels/attention/test_rocm_aiter_mla_sink.py b/tests/kernels/attention/test_rocm_aiter_mla_sink.py index 40b888c362..0b007dbc87 100644 --- a/tests/kernels/attention/test_rocm_aiter_mla_sink.py +++ b/tests/kernels/attention/test_rocm_aiter_mla_sink.py @@ -156,6 +156,8 @@ def test_sparse_mla_sink_matches_ragged_reference( num_decodes=batch_size, num_decode_tokens=batch_size, max_query_len=1, + max_seq_len=max(seq_lens), + topk_tokens=max(seq_lens), ) sinks = torch.linspace(-2.0, 6.0, real_heads, device=device) @@ -367,12 +369,15 @@ def test_sparse_mla_sink_matches_dense_attention_with_empty_rows_and_paged_cache for page, offset in rows ] lengths = torch.tensor([0, *(len(rows) for rows in selected_rows)]) + longest_row = max(len(rows) for rows in selected_rows) metadata = SimpleNamespace( block_size=block_size, num_prefills=0, num_decodes=len(selected_rows), num_decode_tokens=len(selected_rows), max_query_len=1, + max_seq_len=longest_row, + topk_tokens=longest_row, qo_indptr=torch.arange( len(selected_rows) + 1, dtype=torch.int32, device="cuda" ), @@ -479,6 +484,7 @@ def test_sparse_mla_sink_forward_mqa_preserves_split_query(dtype): num_decodes=num_tokens, num_decode_tokens=num_tokens, max_query_len=1, + max_seq_len=kv.shape[0], block_size=block_size, topk_tokens=impl.topk_indices_buffer.shape[1], req_id_per_token=torch.zeros(num_tokens, dtype=torch.int32, device="cuda"), diff --git a/tests/kernels/attention/test_rocm_triton_attn_dsv4.py b/tests/kernels/attention/test_rocm_triton_attn_dsv4.py index 3b200be514..362ac6b7ce 100644 --- a/tests/kernels/attention/test_rocm_triton_attn_dsv4.py +++ b/tests/kernels/attention/test_rocm_triton_attn_dsv4.py @@ -41,7 +41,7 @@ def _on_gfx950() -> bool: # architectures take the fallback decode kernel, so its tests are skipped there. requires_split_decode_arch = pytest.mark.skipif( not _on_split_decode_arch(), - reason="split-K decode kernel is only tuned for AMD gfx942/gfx950", + reason="split-K decode and the V4.1 combine tests need an AMD gfx942/gfx950", ) requires_gfx950 = pytest.mark.skipif( not _on_gfx950(), @@ -596,6 +596,87 @@ def test_sparse_attn_prefill_ragged_kernel() -> None: torch.testing.assert_close(actual, expected, atol=2e-2, rtol=2e-2) +NOPE_ONLY_HEAD_DIM = 512 + + +def _nope_only_ragged_inputs( + kv_lens: list[int], num_heads: int, device: torch.device +) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor]: + """Build a GLM-like 512+0 NoPE decode batch with ragged KV segments.""" + num_kv = 4096 + kv = ( + torch.randn(num_kv, NOPE_ONLY_HEAD_DIM, dtype=torch.bfloat16, device=device) + * 0.125 + ) + q = ( + torch.randn( + len(kv_lens), + num_heads, + NOPE_ONLY_HEAD_DIM, + dtype=torch.bfloat16, + device=device, + ) + * 0.125 + ) + rows = [torch.randperm(num_kv)[:length].tolist() for length in kv_lens] + indices, indptr = _ragged_from_rows(rows, device) + return q, kv, indices, indptr + + +# Lengths below the split count leave splits empty; non-multiples force a tail. +_NOPE_ONLY_KV_LENS = [0, 1, 3, 37, 129, 2048] + + +@requires_split_decode_arch +@pytest.mark.parametrize("num_splits", [2, 32]) +@pytest.mark.parametrize("with_sink", [True, False]) +@torch.inference_mode() +def test_sparse_attn_decode_bf16_split_k_matches_ragged( + num_splits: int, with_sink: bool +) -> None: + """Split-K decode must reproduce the single-workgroup ragged kernel.""" + from vllm.v1.attention.ops.rocm_aiter_mla_sparse import ( + _rocm_sparse_attn_decode_ragged_bf16_triton, + _rocm_sparse_attn_prefill_ragged_triton, + ) + + device = torch.device("cuda") + set_random_seed(0) + num_heads = 16 + q, kv, indices, indptr = _nope_only_ragged_inputs( + _NOPE_ONLY_KV_LENS, num_heads, device + ) + attn_sink = ( + torch.randn(num_heads, dtype=torch.float32, device=device) + if with_sink + else None + ) + scale = NOPE_ONLY_HEAD_DIM**-0.5 + + expected = _rocm_sparse_attn_prefill_ragged_triton( + q=q, + kv=kv, + indices=indices, + indptr=indptr, + scale=scale, + attn_sink=attn_sink, + nope_head_dim=NOPE_ONLY_HEAD_DIM, + rope_head_dim=0, + ) + actual = _rocm_sparse_attn_decode_ragged_bf16_triton( + q=q, + kv=kv, + indices=indices, + indptr=indptr, + scale=scale, + attn_sink=attn_sink, + nope_head_dim=NOPE_ONLY_HEAD_DIM, + rope_head_dim=0, + num_splits=num_splits, + ) + torch.testing.assert_close(actual, expected, atol=2e-2, rtol=2e-2) + + def _sparse_prefill_ragged_inputs(nope_dim: int, rope_dim: int) -> dict: device = torch.device("cuda") set_random_seed(7) @@ -1105,6 +1186,21 @@ def test_decode_num_splits_gfx950(monkeypatch) -> None: assert mod._decode_gfx950_num_splits(1, 1, 128, 8192) == 32 assert mod._decode_gfx950_num_splits(17, 1, 128, 32) == 4 assert mod._decode_gfx950_num_splits(512, 1, 128, 7812) == 1 + assert mod._decode_gfx950_num_splits(4, 1, 2048.0, 0.0, 32) == 32 + + +def test_sparse_decode_bf16_num_splits_floor_and_tile_clamp(monkeypatch) -> None: + """Short rows skip splitting, and splits never exceed the tiles they walk.""" + from vllm.v1.attention.ops import rocm_aiter_mla_sparse as mod + + monkeypatch.setattr(mod, "_decode_num_splits", lambda *args: 32) + monkeypatch.setattr(mod, "_decode_gfx950_num_splits", lambda *args: 32) + + floor = mod._SPARSE_DECODE_BF16_MIN_SPLIT_LEN + block_k = mod._SPARSE_DECODE_BF16_BLOCK_K + assert mod.rocm_sparse_decode_bf16_num_splits(1, 1, floor - 1) == 1 + assert mod.rocm_sparse_decode_bf16_num_splits(1, 1, floor) == floor // block_k + assert mod.rocm_sparse_decode_bf16_num_splits(1, 1, 2048) == 32 @requires_split_decode_arch @@ -2311,3 +2407,159 @@ def test_paged_mqa_logits_gate_v41_shuffle_vs_c4a_flat() -> None: torch.testing.assert_close( gated_c4a[:, :seq_len], ref[:, :seq_len], atol=2e-3, rtol=2e-3 ) + + +def _combine_topk_swa_indices_torch( + topk_indices, + query_start_loc, + seq_lens, + gather_lens, + window_size, + compress_ratio, + topk, + M, + N, +): + """The Torch path this kernel replaces, copied from before the switch. + + Kept verbatim so the reference cannot drift toward the kernel. + """ + from vllm.models.deepseek_v41.common.ops.cache_utils import ( + _SPARSE_PREFILL_TOPK_ALIGNMENT, + ) + + topk_indices = topk_indices.reshape(topk_indices.shape[0], -1).contiguous() + num_tokens = topk_indices.shape[0] + combined_topk = ( + (topk + window_size + _SPARSE_PREFILL_TOPK_ALIGNMENT - 1) + // _SPARSE_PREFILL_TOPK_ALIGNMENT + * _SPARSE_PREFILL_TOPK_ALIGNMENT + ) + combined_indices = torch.full( + (num_tokens, combined_topk), + fill_value=-1, + dtype=torch.int32, + device=topk_indices.device, + ) + combined_lens = torch.empty( + num_tokens, dtype=torch.int32, device=topk_indices.device + ) + + # query_start_loc may have a non-zero base for a narrowed mixed batch. + query_lens = query_start_loc[1:] - query_start_loc[:-1] + req_ids = torch.repeat_interleave( + torch.arange(seq_lens.shape[0], device=seq_lens.device), query_lens + ) + query_starts = query_start_loc[:-1] - query_start_loc[0] + token_offsets = torch.arange(num_tokens, device=seq_lens.device) - ( + torch.repeat_interleave(query_starts, query_lens) + ) + positions = seq_lens[req_ids] - query_lens[req_ids] + token_offsets + + logical_topk_width = min(topk, topk_indices.shape[1]) + topk_lens = torch.minimum( + (positions + 1) // compress_ratio, + torch.full_like(positions, logical_topk_width), + ).clamp_min(0) + topk_offsets = torch.arange(logical_topk_width, device=seq_lens.device) + topk_mask = topk_offsets[None, :] < topk_lens[:, None] + topk_values = topk_indices[:, :logical_topk_width].to(torch.int32) + topk_valid = topk_mask & (topk_values >= 0) & (topk_values < N) + combined_indices[:, :logical_topk_width] = torch.where( + topk_valid, + topk_values + (M * req_ids).to(torch.int32)[:, None], + -1, + ) + + swa_lens = torch.minimum( + positions + 1, torch.full_like(positions, window_size) + ).clamp_min(0) + swa_offsets = torch.arange(window_size, device=seq_lens.device) + swa_mask = swa_offsets[None, :] < swa_lens[:, None] + swa_columns = topk_lens[:, None] + swa_offsets[None, :] + gather_starts = seq_lens - gather_lens + swa_values = ( + M * req_ids[:, None] + + N + + swa_offsets[None, :] + + positions[:, None] + - swa_lens[:, None] + + 1 + - gather_starts[req_ids, None] + ).to(torch.int32) + rows = torch.arange(num_tokens, device=seq_lens.device)[:, None].expand_as( + swa_columns + ) + flat_dst = rows[swa_mask] * combined_topk + swa_columns[swa_mask] + combined_indices.view(-1).index_copy_(0, flat_dst, swa_values[swa_mask]) + combined_lens.copy_((topk_lens + swa_lens).to(torch.int32)) + return combined_indices, combined_lens + + +def _v41_combine_case(case, window): + dev = "cuda" + ratio, topk = 2, 512 + if case == "seq_len_below_query_len": + qsl = torch.tensor([0, 64], dtype=torch.int32, device=dev) + sl = torch.tensor([32], dtype=torch.int32, device=dev) + rows = 64 + elif case == "seq_len_zero": + # The warmup batch: every pos is negative, so both lengths clamp to 0. + qsl = torch.tensor([0, 32, 64], dtype=torch.int32, device=dev) + sl = torch.zeros(2, dtype=torch.int32, device=dev) + rows = 64 + elif case == "compress_ratio_zero": + qsl = torch.tensor([0, 4, 8], dtype=torch.int32, device=dev) + sl = torch.tensor([4, 4], dtype=torch.int32, device=dev) + rows, ratio, topk = 8, 0, 0 + elif case == "nonzero_base": + # Prefill rows of a mixed batch: query_start_loc starts past the decodes. + qsl = torch.tensor([32, 64, 128], dtype=torch.int32, device=dev) + sl = torch.tensor([48, 80], dtype=torch.int32, device=dev) + rows = 96 + else: + qsl = torch.tensor([0, 32, 96], dtype=torch.int32, device=dev) + sl = torch.tensor([48, 80], dtype=torch.int32, device=dev) + rows = 96 + # What the SWA builder gathers: the query plus up to window - 1 rows before it. + gl = torch.minimum(sl, qsl[1:] - qsl[:-1] + window - 1) + set_random_seed(0) + ti = torch.randint(0, 4096, (rows, max(topk, 1)), dtype=torch.int32) + if case == "invalid_topk": + # Failed candidates stay -1 instead of pointing into another request. + ti[:, ::7] = -1 + return ti.to(dev), qsl, sl, gl, window, ratio, topk, 100000, 4096 + + +@requires_split_decode_arch +@pytest.mark.parametrize( + "case,window", + [ + ("seq_len_below_query_len", 128), + ("seq_len_zero", 128), + ("compress_ratio_zero", 128), + ("nonzero_base", 128), + ("invalid_topk", 128), + ("plain", 64), + ("plain", 128), + ("plain", 256), + ], +) +def test_v41_combine_topk_swa_indices_matches_torch(case, window) -> None: + from vllm.models.deepseek_v41.amd.rocm import combine_topk_swa_indices + + args = _v41_combine_case(case, window) + got_indices, got_lens = combine_topk_swa_indices(*args) + ref_indices, ref_lens = _combine_topk_swa_indices_torch(*args) + assert torch.equal(got_lens, ref_lens), ( + f"lens differ: ref={ref_lens.tolist()} got={got_lens.tolist()}" + ) + assert torch.equal(got_indices, ref_indices) + + +def test_v41_rocm_combine_uses_shared_kernel() -> None: + """ROCm prefill takes the shared combine, so it inherits the replay bound.""" + from vllm.models.deepseek_v41.amd import rocm + from vllm.models.deepseek_v41.common.ops import cache_utils + + assert rocm.combine_topk_swa_indices is cache_utils.combine_topk_swa_indices diff --git a/tests/kernels/attention/test_segmented_attention.py b/tests/kernels/attention/test_segmented_attention.py new file mode 100644 index 0000000000..0fbf5fe894 --- /dev/null +++ b/tests/kernels/attention/test_segmented_attention.py @@ -0,0 +1,1408 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +"""Segmented attention accuracy, cache semantics, and startup tuning contracts.""" + +import math + +import pytest +import torch + +from vllm.platforms import current_platform +from vllm.triton_utils import triton + +if current_platform.is_rocm(): + from vllm.platforms.rocm import on_gfx1x + +pytestmark = [ + pytest.mark.skip_global_cleanup, + pytest.mark.skipif( + not (current_platform.is_rocm() and on_gfx1x()), + reason="Segmented attention requires RDNA GPUs", + ), +] + + +def _make_paged_attention_case( + query_lens: list[int], + context_lens: list[int], + *, + num_heads: int, + num_kv_heads: int, + head_size: int, + block_size: int, + fp8: bool, + causal: bool = True, + sliding_window: int = 0, + softcap: float = 0.0, + sinks: torch.Tensor | None = None, +): + device = torch.device("cuda:0") + generator = torch.Generator(device=device).manual_seed(419) + total_queries = sum(query_lens) + qkv = ( + torch.randn( + total_queries, + num_heads + 2 * num_kv_heads, + head_size, + device=device, + dtype=torch.bfloat16, + generator=generator, + ) + * 0.25 + ) + query, key, value = qkv.split((num_heads, num_kv_heads, num_kv_heads), dim=1) + cache_dtype = current_platform.fp8_dtype() if fp8 else torch.bfloat16 + k_scale = torch.tensor(0.125 if fp8 else 1.0, device=device) + v_scale = torch.tensor(0.25 if fp8 else 1.0, device=device) + max_seq_len = max(q + c for q, c in zip(query_lens, context_lens)) + blocks_per_seq = triton.cdiv(max_seq_len, block_size) + num_blocks = len(query_lens) * blocks_per_seq + key_cache = ( + torch.randn( + num_blocks, + block_size, + num_kv_heads, + head_size, + device=device, + dtype=torch.float32, + generator=generator, + ) + * 0.25 + / k_scale + ).to(cache_dtype) + value_cache = ( + torch.randn( + num_blocks, + block_size, + num_kv_heads, + head_size, + device=device, + dtype=torch.float32, + generator=generator, + ) + * 0.25 + / v_scale + ).to(cache_dtype) + block_table = torch.arange(num_blocks, device=device, dtype=torch.int32).view( + len(query_lens), blocks_per_seq + ) + starts = torch.tensor( + [0, *torch.tensor(query_lens).cumsum(0).tolist()], + device=device, + dtype=torch.int32, + ) + seq_lens = torch.tensor( + [q + c for q, c in zip(query_lens, context_lens)], + device=device, + dtype=torch.int32, + ) + + first = 0 + for seq, (query_len, context_len) in enumerate(zip(query_lens, context_lens)): + for query_pos in range(query_len): + cache_pos = context_len + query_pos + block = block_table[seq, cache_pos // block_size] + offset = cache_pos % block_size + key_cache[block, offset] = (key[first + query_pos].float() / k_scale).to( + cache_dtype + ) + value_cache[block, offset] = ( + value[first + query_pos].float() / v_scale + ).to(cache_dtype) + first += query_len + + references = [] + first = 0 + for seq, (query_len, context_len) in enumerate(zip(query_lens, context_lens)): + physical = block_table[seq].long() + seq_len = context_len + query_len + full_key = ( + key_cache[physical].flatten(0, 1)[:seq_len].float() * k_scale + ).repeat_interleave(num_heads // num_kv_heads, dim=1) + full_value = ( + value_cache[physical].flatten(0, 1)[:seq_len].float() * v_scale + ).repeat_interleave(num_heads // num_kv_heads, dim=1) + scores = torch.einsum( + "qhd,khd->hqk", query[first : first + query_len].float(), full_key + ) / math.sqrt(head_size) + if softcap > 0: + scores = softcap * torch.tanh(scores / softcap) + query_positions = context_len + torch.arange(query_len, device=device) + key_positions = torch.arange(seq_len, device=device) + causal_mask = key_positions[None, :] > query_positions[:, None] + if causal: + scores.masked_fill_(causal_mask[None], -float("inf")) + if sliding_window > 0: + window_mask = key_positions[None, :] < ( + query_positions[:, None] - sliding_window + 1 + ) + scores.masked_fill_(window_mask[None], -float("inf")) + if sinks is not None: + sink_scores = sinks.float()[:, None, None].expand(-1, query_len, 1) + probabilities = torch.cat((scores, sink_scores), dim=-1).softmax(-1) + probabilities = probabilities[..., :-1] + else: + probabilities = scores.softmax(-1) + references.append(torch.einsum("hqk,khd->qhd", probabilities, full_value)) + first += query_len + + return { + "query": query, + "key": key, + "value": value, + "key_cache": key_cache, + "value_cache": value_cache, + "block_table": block_table, + "starts": starts, + "seq_lens": seq_lens, + "k_scale": k_scale, + "v_scale": v_scale, + "reference": torch.cat(references), + "max_seq_len": max_seq_len, + } + + +def _run_segmented_case(case, output, max_query_len, *, force_splits=None, **kwargs): + from vllm.v1.attention.ops.segmented_attention import ( + run_segmented_attention, + select_segmented_config, + ) + + config = None + heads, dim = case["query"].shape[1:] + if force_splits is not None: + config = dict( + select_segmented_config( + case["seq_lens"].numel(), + max_query_len, + case["max_seq_len"], + heads, + case["key_cache"].shape[2], + dim, + case["key_cache"].element_size() == 1, + ), + splits=force_splits, + ) + run_segmented_attention( + case["query"], + output, + case["key_cache"], + case["value_cache"], + case["block_table"], + case["starts"], + case["seq_lens"], + max_query_len, + case["max_seq_len"], + case["k_scale"], + case["v_scale"], + dim**-0.5, + config=config, + **kwargs, + ) + + +def test_segmented_tuning_candidates_preserve_workspace_bound(): + """D128/D256 candidates retain their existing split scratch bound.""" + from vllm.model_executor.warmup import ( + rocm_segmented_attn_autotune_warmup as tuning, + ) + from vllm.v1.attention.ops import segmented_attention as segmented + + for batch, query_len, seq_len, heads, kv_heads, dim, fp8 in ( + (1, 1, 128, 8, 8, 128, True), + (32, 1, 262144, 16, 1, 128, True), + (4, 1024, 8192, 16, 4, 256, False), + (1, 1024, 131072, 6, 1, 256, True), + ): + default = segmented.select_segmented_config( + batch, query_len, seq_len, heads, kv_heads, dim, fp8 + ) + candidates = tuning._candidate_configs(default, batch, query_len, heads, dim) + assert candidates + assert ( + tuning._normalized_config(default, batch, query_len, heads, dim) + in candidates + ) + assert all(config["splits"] <= default["splits"] for config in candidates) + assert all(dim % config["bk"] == 0 for config in candidates) + + +@pytest.mark.parametrize("dim", (128, 256)) +def test_segmented_fp8_long_extend_split_workspace_bound(dim): + from vllm.v1.attention.ops.segmented_attention import ( + MAX_LONG_EXTEND_WORKSPACE_BYTES, + segmented_query_capacity, + select_segmented_config, + ) + + for batch, query_len, seq_len, heads in ( + (1, 256, 8192, 6), + (1, 256, 32768, 6), + (1, 256, 131072, 6), + (1, 1024, 32768, 1), + (1, 1024, 131072, 1), + (1, 1024, 131072, 6), + (1, 4096, 131072, 1), + (4, 1024, 131072, 6), + (5, 257, 131072, 6), + (21, 256, 131072, 6), + ): + config = select_segmented_config(batch, query_len, seq_len, heads, 1, dim, True) + if seq_len >= 32768: + scratch = ( + batch + * segmented_query_capacity(query_len) + * heads + * config["splits"] + * (dim + 1) + * 4 + ) + assert scratch <= MAX_LONG_EXTEND_WORKSPACE_BYTES * dim // 128 + + for batch in (1, 4, 16): + for query_len in (257, 1025, 4095, 8192): + for ratio in (1, 6, 16): + for kv_heads in (1, 4): + for seq_len in (32768, 131072): + heads = ratio * kv_heads + config = select_segmented_config( + batch, query_len, seq_len, heads, kv_heads, dim, True + ) + if config["splits"] == 1: + continue + scratch = ( + batch + * segmented_query_capacity(query_len) + * heads + * config["splits"] + * (dim + 1) + * 4 + ) + assert scratch <= MAX_LONG_EXTEND_WORKSPACE_BYTES * dim // 128 + + +@torch.inference_mode() +def test_segmented_bf16_d256_long_extend_matches_dense_reference(): + from vllm.platforms.rocm import on_gfx1x + from vllm.v1.attention.ops.segmented_attention import run_segmented_attention + + if not current_platform.is_rocm() or not on_gfx1x(): + pytest.skip("gfx1x segmented prefill") + + case = _make_paged_attention_case( + [1024], + [7168], + num_heads=6, + num_kv_heads=1, + head_size=256, + block_size=1568, + fp8=False, + ) + output = torch.empty_like(case["query"]) + run_segmented_attention( + case["query"], + output, + case["key_cache"], + case["value_cache"], + case["block_table"], + case["starts"], + case["seq_lens"], + 1024, + case["max_seq_len"], + case["k_scale"], + case["v_scale"], + 256**-0.5, + ) + relative = (output.float() - case["reference"]).norm(dim=-1) / case[ + "reference" + ].norm(dim=-1).clamp_min(1e-6) + assert torch.isfinite(output).all() and relative.max().item() < 0.01 + + +@pytest.mark.parametrize( + "query_len,heads,qk_amplitude,dim", + [ + (256, 1, 1, 256), + (256, 6, 1, 256), + (512, 6, 1, 256), + (256, 6, 4, 256), + (256, 6, 16, 256), + (256, 6, 1, 128), + (512, 6, 4, 128), + (256, 6, 16, 128), + (8192, 1, 1, 128), + (8192, 1, 1, 256), + ], +) +@torch.inference_mode() +def test_segmented_fp8_long_extend_matches_dense_reference( + query_len, heads, qk_amplitude, dim +): + from vllm.platforms.rocm import on_gfx12x + from vllm.v1.attention.ops.segmented_attention import run_segmented_attention + + if not current_platform.is_rocm() or not on_gfx12x(): + pytest.skip("FP8 KV requires gfx12") + case = _make_paged_attention_case( + [query_len], + [8192 - query_len], + num_heads=heads, + num_kv_heads=1, + head_size=dim, + block_size=1568, + fp8=True, + ) + if qk_amplitude != 1: + # Check peaked logits too: one-term FP8 Q quantization passed only the + # nearly uniform fixture and deviated by 4-70% at these amplitudes. + case["query"].mul_(qk_amplitude) + case["k_scale"].mul_(qk_amplitude) + full_key = ( + case["key_cache"][case["block_table"][0].long()] + .flatten(0, 1)[:8192] + .float() + * case["k_scale"] + ).repeat_interleave(heads, dim=1) + full_value = ( + case["value_cache"][case["block_table"][0].long()] + .flatten(0, 1)[:8192] + .float() + * case["v_scale"] + ).repeat_interleave(heads, dim=1) + scores = torch.einsum( + "qhd,khd->hqk", case["query"].float(), full_key + ) / math.sqrt(dim) + future = torch.arange(8192, device=case["query"].device)[None, :] > ( + 8192 + - query_len + + torch.arange(query_len, device=case["query"].device)[:, None] + ) + scores.masked_fill_(future[None], -float("inf")) + case["reference"] = torch.einsum("hqk,khd->qhd", scores.softmax(-1), full_value) + output = torch.empty_like(case["query"]) + run_segmented_attention( + case["query"], + output, + case["key_cache"], + case["value_cache"], + case["block_table"], + case["starts"], + case["seq_lens"], + query_len, + case["max_seq_len"], + case["k_scale"], + case["v_scale"], + dim**-0.5, + ) + relative = (output.float() - case["reference"]).norm(dim=-1) / case[ + "reference" + ].norm(dim=-1).clamp_min(1e-6) + assert torch.isfinite(output).all() and relative.max().item() < 0.01 + + +def test_segmented_tuning_protects_static_incumbent(): + """Noise-sized gains must not replace the static configuration.""" + from vllm.model_executor.warmup import ( + rocm_segmented_attn_autotune_warmup as tuning, + ) + + incumbent = {"bm": 16} + challenger = {"bm": 32} + samples = { + tuning._config_key(incumbent): [100.0, 101.0, 99.0, 100.0, 100.0], + tuning._config_key(challenger): [99.0, 100.0, 98.0, 99.0, 99.0], + } + winner, comparisons = tuning._select_tuned_winner( + incumbent, [incumbent, challenger], samples + ) + assert winner["config"] == incumbent + assert comparisons[1]["paired_speedup_vs_default"] < 1.02 + + +def test_segmented_tuning_promotes_verified_challenger(): + """A finalist with a stable material gain should replace the incumbent.""" + from vllm.model_executor.warmup import ( + rocm_segmented_attn_autotune_warmup as tuning, + ) + + incumbent = {"bm": 16} + challenger = {"bm": 32} + samples = { + tuning._config_key(incumbent): [100.0, 102.0, 98.0, 101.0, 99.0], + tuning._config_key(challenger): [94.0, 96.0, 92.0, 95.0, 93.0], + } + winner, _ = tuning._select_tuned_winner(incumbent, [incumbent, challenger], samples) + assert winner["config"] == challenger + assert winner["paired_speedup_vs_default"] > 1.02 + + +def test_segmented_tuning_balances_tp_workloads_without_overlap(): + """Every missing bucket belongs to exactly one reasonably balanced rank.""" + from vllm.model_executor.warmup import ( + rocm_segmented_attn_autotune_warmup as tuning, + ) + + workloads = list(tuning._workloads(8192, 262144, 32)) + shards = tuning._shard_workloads( + workloads, + 4, + 8192, + 12, + 2, + 256, + True, + ) + flattened = [workload for shard in shards for workload in shard] + assert sorted(flattened) == sorted(workloads) + assert len(flattened) == len(set(flattened)) + owners = { + workload[:2]: rank for rank, shard in enumerate(shards) for workload in shard + } + assert all( + owners[workload[:2]] == rank + for rank, shard in enumerate(shards) + for workload in shard + ) + + weights = [ + sum( + tuning._workload_weight(workload, 8192, 12, 2, 256, True) + for workload in shard + ) + for shard in shards + ] + largest_group = max( + sum( + tuning._workload_weight(workload, 8192, 12, 2, 256, True) + for workload in workloads + if workload[:2] == group + ) + for group in {workload[:2] for workload in workloads} + ) + assert max(weights) - min(weights) <= largest_group + + +def test_segmented_tuning_prunes_scheduler_and_kv_limits(): + """Generated buckets must be reachable under scheduler and cache limits.""" + from vllm.model_executor.warmup import ( + rocm_segmented_attn_autotune_warmup as tuning, + ) + + limits = (8192, 262144, 32) + raw = set( + tuning._workloads( + *limits, + dtype=torch.bfloat16, + kv_dtype=torch.float8_e4m3fn, + heads=16, + kv_heads=1, + dim=128, + page=16, + ) + ) + assert {batch for batch, _, _ in raw} == {1, 2, 4, 8, 16, 32} + assert {query for _, query, _ in raw} == set(tuning._QUERY_BUCKETS) + assert (32, 1, 262144) in raw + assert (1, 8192, 262144) in raw + assert (1, 8192, 131072) in raw + assert (1, 8, 131072) in raw + assert all( + sum(tuning._query_lengths(batch, query, limits[0])) <= limits[0] + and batch <= limits[2] + and query <= min(limits[0], tuning.MAX_QUERY_LEN) + and query <= seq_len <= limits[1] + for batch, query, seq_len in raw + ) + + layouts = ((16, 16 * 128 * 2),) + bounded = set( + tuning._workloads( + *limits, + memory_budget_bytes=2**50, + dtype=torch.bfloat16, + kv_dtype=torch.float8_e4m3fn, + heads=16, + kv_heads=1, + dim=128, + page=16, + cache_layouts=layouts, + cache_budget_bytes=2**20, + ) + ) + assert bounded < raw + assert all( + batch * sum(((seq_len + block - 1) // block) * size for block, size in layouts) + <= 2**20 + for batch, _, seq_len in bounded + ) + + +@pytest.fixture +def segmented_tuner(tmp_path, monkeypatch): + """Exercise real cache files and lookup while replacing GPU timing only.""" + from types import SimpleNamespace + + from vllm.model_executor.warmup import ( + rocm_segmented_attn_autotune_warmup as tuning, + ) + from vllm.v1.attention.ops import segmented_attention as segmented + + monkeypatch.setenv("VLLM_CACHE_ROOT", str(tmp_path)) + monkeypatch.setattr( + torch.cuda, + "get_device_properties", + lambda _: SimpleNamespace( + name="test", gcnArchName="gfx1201", multi_processor_count=32 + ), + ) + monkeypatch.setattr(tuning, "_TABLES", {}) + calls = [] + + def tune(*args, **kwargs): + heads, kv_heads, dim = args[3:6] + batch, query_len, seq_len = args[-1] + default = segmented.select_segmented_config( + batch, query_len, seq_len, heads, kv_heads, dim, False + ) + best = tuning._candidate_configs(default, batch, query_len, heads, dim)[0] + calls.append(args[-1]) + return { + "workload": list(args[-1]), + "query_lengths": tuning._query_lengths(batch, query_len, args[-2]), + "default": default, + "best": best, + "results": [], + } + + monkeypatch.setattr(tuning, "_tune_workload", tune) + geometry = (torch.device("cuda:0"), torch.bfloat16, 4, 2, 128, 16, 128**-0.5) + return SimpleNamespace( + module=tuning, + warmup_args=(*geometry, 4, 8, 2), + lookup_args=(*geometry[:2], torch.bfloat16, *geometry[2:], 1, 1, 8), + calls=calls, + cache_dir=tmp_path, + ) + + +@pytest.mark.parametrize("fail", [False, True]) +@pytest.mark.parametrize("device", ["cuda", "cuda:0"]) +def test_segmented_tuning_releases_only_new_kernel_modules(monkeypatch, fail, device): + """Tuning releases candidate modules even on failure without invalidating graphs.""" + import weakref + from types import SimpleNamespace + + from vllm.model_executor.warmup import ( + rocm_segmented_attn_autotune_warmup as tuning, + ) + + class Module: + pass + + existing = Module() + other_device = Module() + caches = [{"existing": existing} for _ in range(2)] + for name, cache in zip( + ("_segmented_attention_stage", "_segmented_attention_reduce"), caches + ): + monkeypatch.setattr( + tuning, + name, + SimpleNamespace(device_caches={0: (cache,), 1: ({"other": other_device},)}), + ) + monkeypatch.setattr(torch.accelerator, "synchronize", lambda _: None) + monkeypatch.setattr(torch.accelerator, "empty_cache", lambda: None) + monkeypatch.setattr(torch.accelerator, "current_device_index", lambda: 0) + references = [] + try: + with tuning._temporary_tuning_kernels(torch.device(device)): + for cache in caches: + cache["candidate"] = Module() + references.append(weakref.ref(cache["candidate"])) + if fail: + raise RuntimeError("tuning failed") + except RuntimeError: + assert fail + assert all(reference() is None for reference in references) + assert all(cache == {"existing": existing} for cache in caches) + for kernel in ( + tuning._segmented_attention_stage, + tuning._segmented_attention_reduce, + ): + assert kernel.device_caches[1][0] == {"other": other_device} + + +def test_segmented_tuning_cleanup_preserves_graphs_and_reloads_kernels(): + """Existing graphs survive cleanup, and released variants still run correctly.""" + import weakref + + from vllm.model_executor.warmup import ( + rocm_segmented_attn_autotune_warmup as tuning, + ) + + case = _make_paged_attention_case( + [3], + [11], + num_heads=10, + num_kv_heads=2, + head_size=128, + fp8=False, + block_size=16, + ) + output = torch.empty_like(case["query"]) + _run_segmented_case(case, output, 3, force_splits=1) + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + _run_segmented_case(case, output, 3, force_splits=1) + caches = [ + kernel.device_caches[0][0] + for kernel in ( + tuning._segmented_attention_stage, + tuning._segmented_attention_reduce, + ) + ] + keys = [set(cache) for cache in caches] + with tuning._temporary_tuning_kernels(torch.device("cuda:0")): + _run_segmented_case(case, output, 3, force_splits=2) + references = [ + weakref.ref(cache[key]) + for cache, existing in zip(caches, keys) + for key in cache.keys() - existing + ] + assert references + assert all(reference() is None for reference in references) + output.fill_(float("nan")) + graph.replay() + torch.testing.assert_close(output.float(), case["reference"], atol=0.01, rtol=0.01) + output.fill_(float("nan")) + _run_segmented_case(case, output, 3, force_splits=2) + torch.testing.assert_close(output.float(), case["reference"], atol=0.01, rtol=0.01) + + +@pytest.mark.parametrize( + "mode", + [ + pytest.param({}, id="causal"), + pytest.param({"causal": False}, id="noncausal"), + pytest.param({"sliding_window": 7}, id="sliding-window"), + pytest.param({"sliding_window": 7, "causal": False}, id="draft-window"), + pytest.param({"has_sinks": True}, id="sinks"), + ], +) +def test_segmented_tuning_persists_without_retuning(segmented_tuner, monkeypatch, mode): + """A fresh in-memory table reloads the same winner without timing or writing.""" + state = segmented_tuner + tuning = state.module + tuning.warmup_segmented_attention( + *state.warmup_args, memory_budget_bytes=2**50, **mode + ) + assert state.calls + expected = tuning.get_segmented_config(*state.lookup_args, **mode) + assert expected is not None + (cache,) = state.cache_dir.rglob("*.json") + saved = cache.read_bytes(), cache.stat().st_mtime_ns + tuning._TABLES.clear() + + def forbidden(*args, **kwargs): + pytest.fail("Persistent segmented startup cache attempted to retune") + + monkeypatch.setattr(tuning, "_tune_workload", forbidden) + tuning.warmup_segmented_attention( + *state.warmup_args, memory_budget_bytes=2**50, **mode + ) + assert tuning.get_segmented_config(*state.lookup_args, **mode) == expected + assert (cache.read_bytes(), cache.stat().st_mtime_ns) == saved + + +@pytest.mark.parametrize( + "max_tokens,mode", + [ + pytest.param(3, {}, id="scheduler-token-budget"), + pytest.param(4, {"causal": False}, id="noncausal"), + pytest.param(4, {"sliding_window": 7}, id="sliding-window"), + pytest.param(4, {"sliding_window": 7, "causal": False}, id="draft-window"), + pytest.param(4, {"has_sinks": True}, id="sinks"), + ], +) +def test_segmented_tuning_does_not_reuse_incompatible_winners( + segmented_tuner, max_tokens, mode +): + """Changed attention semantics or query mix require their own measurements.""" + state = segmented_tuner + tuning = state.module + tuning.warmup_segmented_attention(*state.warmup_args, memory_budget_bytes=2**50) + previous_calls = len(state.calls) + if mode: + assert tuning.get_segmented_config(*state.lookup_args, **mode) is None + args = (*state.warmup_args[:7], max_tokens, *state.warmup_args[8:]) + tuning.warmup_segmented_attention(*args, memory_budget_bytes=2**50, **mode) + assert len(state.calls) > previous_calls + assert len(list(state.cache_dir.rglob("*.json"))) == 2 + assert tuning.get_segmented_config(*state.lookup_args, **mode) is not None + + +@pytest.mark.parametrize( + "dim,page,hq,hk,dtype,kv_dtype,byte_cache,force_splits", + [ + (128, 16, 4, 4, torch.bfloat16, torch.bfloat16, False, None), + (128, 32, 8, 2, torch.float16, torch.float16, False, None), + (256, 784, 12, 2, torch.bfloat16, torch.bfloat16, False, None), + (256, 1568, 12, 2, torch.bfloat16, torch.float8_e4m3fn, False, None), + (128, 32, 16, 1, torch.bfloat16, torch.float8_e4m3fn, True, None), + (256, 32, 4, 2, torch.float16, torch.float8_e4m3fn, False, None), + (128, 32, 8, 2, torch.bfloat16, torch.bfloat16, False, 32), + ], +) +@torch.inference_mode() +def test_segmented_attention_ragged_unified_graph_replay( + monkeypatch, dim, page, hq, hk, dtype, kv_dtype, byte_cache, force_splits +): + """Unified-cache segmented prefill follows metadata on graph replay.""" + from vllm.platforms.rocm import on_gfx1x, on_gfx12x + from vllm.v1.attention.ops import segmented_attention as dispatcher + + if not current_platform.is_rocm() or not ( + on_gfx12x() if kv_dtype.itemsize == 1 else on_gfx1x() + ): + pytest.skip("gfx1x segmented prefill (FP8 requires gfx12)") + if force_splits is not None: + original = dispatcher.select_segmented_config + monkeypatch.setattr( + dispatcher, + "select_segmented_config", + lambda *args: dict(original(*args), splits=force_splits), + ) + query_lengths = [0, 1, 2, 7, 33, 129] + contexts = [0, 67, 0, page - 1, page + 1, 33] + device = torch.device("cuda:0") + total = sum(query_lengths) + q = torch.randn(total, hq + 1, dim, device=device, dtype=dtype)[:, :hq] + k = torch.randn(total, hk + 1, dim, device=device, dtype=dtype)[:, :hk] + v = torch.randn_like(k) + blocks_per_seq = triton.cdiv(max(map(sum, zip(query_lengths, contexts))), page) + num_blocks = len(contexts) * blocks_per_seq + page_elements = page * hk * dim + backing = torch.empty(num_blocks * 2 * page_elements, device=device, dtype=kv_dtype) + cache_shape = (num_blocks, page, hk, dim) + strides = (2 * page_elements, hk * dim, dim, 1) + kc = torch.as_strided(backing, cache_shape, strides) + vc = torch.as_strided(backing, cache_shape, strides, page_elements) + ks = torch.tensor(0.13 if kv_dtype.itemsize == 1 else 1.0, device=device) + vs = torch.tensor(0.27 if kv_dtype.itemsize == 1 else 1.0, device=device) + kc.copy_((torch.randn(cache_shape, device=device).float() * 0.25 / ks).to(kv_dtype)) + vc.copy_((torch.randn(cache_shape, device=device).float() * 0.25 / vs).to(kv_dtype)) + table = torch.randperm(num_blocks, device=device, dtype=torch.int32).view( + len(contexts), blocks_per_seq + ) + host_table = table.cpu().tolist() + starts = torch.empty(len(contexts) + 1, device=device, dtype=torch.int32) + lengths = torch.empty(len(contexts), device=device, dtype=torch.int32) + output = torch.full((total, hq + 1, dim), 17.0, device=device, dtype=dtype)[:, :hq] + + def metadata(qlens): + starts.copy_( + torch.tensor( + [0, *torch.tensor(qlens).cumsum(0).tolist()], + device=device, + dtype=torch.int32, + ) + ) + lengths.copy_( + torch.tensor( + [c + n for c, n in zip(contexts, qlens)], + device=device, + dtype=torch.int32, + ) + ) + first = 0 + for seq, (nq, context) in enumerate(zip(qlens, contexts)): + for local in range(nq): + position = context + local + physical = host_table[seq][position // page] + offset = position % page + kc[physical, offset].copy_((k[first + local].float() / ks).to(kv_dtype)) + vc[physical, offset].copy_((v[first + local].float() / vs).to(kv_dtype)) + first += nq + + def run(): + dispatcher.segmented_attention( + query=q, + key=k, + value=v, + output=output, + kv_cache_dtype="fp8" if kv_dtype.itemsize == 1 else "auto", + key_cache=kc.view(torch.uint8) if byte_cache else kc, + value_cache=vc.view(torch.uint8) if byte_cache else vc, + block_table=table, + query_start_loc=starts, + seq_lens=lengths, + max_seq_len=max(contexts) + 129, + max_query_len=129, + k_scale=ks, + v_scale=vs, + sm_scale=dim**-0.5, + ) + + def check(qlens): + first = 0 + for seq, (nq, context) in enumerate(zip(qlens, contexts)): + if nq == 0: + first += nq + continue + physical = table[seq].long() + full_k = ( + kc[physical].reshape(-1, hk, dim)[: context + nq].float() * ks + ).repeat_interleave(hq // hk, 1) + full_v = ( + vc[physical].reshape(-1, hk, dim)[: context + nq].float() * vs + ).repeat_interleave(hq // hk, 1) + logits = torch.einsum( + "qhd,khd->hqk", q[first : first + nq].float(), full_k + ) / math.sqrt(dim) + mask = ( + torch.arange(context + nq, device=device)[None, :] + > context + torch.arange(nq, device=device)[:, None] + ) + logits.masked_fill_(mask[None], -float("inf")) + ref = torch.einsum("hqk,khd->qhd", logits.softmax(-1), full_v) + actual = output[first : first + nq].float() + assert torch.isfinite(actual).all() + relative = (actual - ref).norm(dim=-1) / ref.norm(dim=-1).clamp_min(1e-6) + assert relative.max().item() < 0.01 + first += nq + + metadata(query_lengths) + run() + torch.accelerator.synchronize() + check(query_lengths) + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + run() + replay_lengths = [1, 0, 7, 2, 34, 128] + q.mul_(0.75) + k.add_(0.25) + v.add_(0.5) + metadata(replay_lengths) + output.fill_(17.0) + graph.replay() + torch.accelerator.synchronize() + check(replay_lengths) + + +@pytest.mark.parametrize( + "fp8,force_splits", + [(False, None), (True, None), (False, 4), (True, 4)], +) +@torch.inference_mode() +def test_segmented_ragged_prefill_matches_dense_reference(fp8, force_splits): + from vllm.platforms.rocm import on_gfx1x, on_gfx12x + + if not current_platform.is_rocm() or not (on_gfx12x() if fp8 else on_gfx1x()): + pytest.skip("gfx1x segmented prefill (FP8 requires gfx12)") + case = _make_paged_attention_case( + [1, 2, 7, 129], + [31, 64, 97, 128], + num_heads=12, + num_kv_heads=2, + head_size=256, + block_size=32, + fp8=fp8, + ) + output = torch.empty_like(case["query"]) + _run_segmented_case(case, output, max_query_len=129, force_splits=force_splits) + + relative = (output.float() - case["reference"]).norm(dim=-1) / case[ + "reference" + ].norm(dim=-1).clamp_min(1e-6) + assert torch.isfinite(output).all() and relative.max().item() < 0.01 + + +@pytest.mark.parametrize("fp8", [False, True]) +@torch.inference_mode() +def test_rocm_segmented_attn_layout_cache_update(fp8): + from types import SimpleNamespace + + from vllm.platforms.rocm import on_gfx1x, on_gfx12x + from vllm.v1.attention.backend import AttentionType + from vllm.v1.attention.backends.rocm_segmented_attn import ( + RocmSegmentedAttentionImpl, + ) + + if not current_platform.is_rocm() or not (on_gfx12x() if fp8 else on_gfx1x()): + pytest.skip("gfx1x segmented prefill (FP8 requires gfx12)") + cache_dtype = torch.uint8 if fp8 else torch.bfloat16 + kv_cache_dtype = "fp8" if fp8 else "auto" + impl = RocmSegmentedAttentionImpl( + 12, + 256, + 256**-0.5, + 2, + None, + None, + kv_cache_dtype, + attn_type=AttentionType.DECODER, + ) + cache = torch.zeros(3, 2, 32, 512, device="cuda", dtype=cache_dtype) + key = torch.randn(5, 2, 256, device="cuda", dtype=torch.bfloat16) + value = torch.randn_like(key) + slots = torch.tensor([0, 7, 32, 65, 95], device="cuda", dtype=torch.int64) + k_scale = torch.tensor(0.125 if fp8 else 1.0, device="cuda") + v_scale = torch.tensor(0.25 if fp8 else 1.0, device="cuda") + layer = SimpleNamespace(_k_scale=k_scale, _v_scale=v_scale) + + impl.do_kv_cache_update(layer, key, value, cache, slots) + key_cache, value_cache = impl._split_kv_cache(cache) + if fp8: + key_cache = key_cache.view(impl.fp8_dtype) + value_cache = value_cache.view(impl.fp8_dtype) + cached_key = torch.stack( + [key_cache[int(slot) // 32, int(slot) % 32] for slot in slots] + ) + cached_value = torch.stack( + [value_cache[int(slot) // 32, int(slot) % 32] for slot in slots] + ) + expected_key = (key.float() / k_scale).to(key_cache.dtype) + expected_value = (value.float() / v_scale).to(value_cache.dtype) + torch.testing.assert_close(cached_key, expected_key, rtol=0, atol=0) + torch.testing.assert_close(cached_value, expected_value, rtol=0, atol=0) + + +@torch.inference_mode() +def test_segmented_mixed_causality_matches_dense_reference(): + from vllm.platforms.rocm import on_gfx1x + from vllm.v1.attention.ops import segmented_attention as dispatcher + + if not current_platform.is_rocm() or not on_gfx1x(): + pytest.skip("gfx1x unified attention per-request causal fallback") + + query_lens = [2, 3] + context_lens = [5, 6] + args = dict(num_heads=8, num_kv_heads=2, head_size=128, block_size=32) + case = _make_paged_attention_case( + query_lens, context_lens, fp8=False, causal=True, **args + ) + noncausal_case = _make_paged_attention_case( + query_lens, context_lens, fp8=False, causal=False, **args + ) + reference = torch.cat( + ( + case["reference"][: query_lens[0]], + noncausal_case["reference"][query_lens[0] :], + ) + ) + causal = torch.tensor([True, False], device=case["query"].device) + output = torch.empty_like(case["query"]) + dispatcher.segmented_attention( + query=case["query"], + key=case["key"], + value=case["value"], + output=output, + kv_cache_dtype="auto", + key_cache=case["key_cache"], + value_cache=case["value_cache"], + block_table=case["block_table"], + query_start_loc=case["starts"], + seq_lens=case["seq_lens"], + max_seq_len=case["max_seq_len"], + max_query_len=max(query_lens), + k_scale=case["k_scale"], + v_scale=case["v_scale"], + sm_scale=128**-0.5, + causal=causal, + ) + + relative = (output.float() - reference).norm(dim=-1) / reference.norm( + dim=-1 + ).clamp_min(1e-6) + assert torch.isfinite(output).all() and relative.max().item() < 0.01 + + +@pytest.mark.parametrize("fp8", [False, True]) +@pytest.mark.parametrize("causal", [False, True]) +@pytest.mark.parametrize("window_size", [1, 2048]) +@torch.inference_mode() +def test_segmented_sliding_window_matches_dense_reference(fp8, causal, window_size): + from vllm.platforms.rocm import on_gfx1x, on_gfx12x + + if not current_platform.is_rocm() or not (on_gfx12x() if fp8 else on_gfx1x()): + pytest.skip("gfx1x segmented prefill (FP8 requires gfx12)") + case = _make_paged_attention_case( + [8], + [4096], + num_heads=32, + num_kv_heads=8, + head_size=128, + block_size=32, + fp8=fp8, + causal=causal, + sliding_window=window_size, + ) + output = torch.empty_like(case["query"]) + _run_segmented_case( + case, output, max_query_len=8, sliding_window=window_size - 1, causal=causal + ) + + relative = (output.float() - case["reference"]).norm(dim=-1) / case[ + "reference" + ].norm(dim=-1).clamp_min(1e-6) + assert torch.isfinite(output).all() and relative.max().item() < 0.01 + + +@pytest.mark.parametrize("fp8", [False, True]) +@pytest.mark.parametrize("window_size", [0, 128]) +@pytest.mark.parametrize("head_size", [64, 128]) +@torch.inference_mode() +def test_segmented_sinks_match_dense_reference(fp8, window_size, head_size): + from vllm.platforms.rocm import on_gfx1x, on_gfx12x + + if not current_platform.is_rocm() or not (on_gfx12x() if fp8 else on_gfx1x()): + pytest.skip("gfx1x segmented prefill (FP8 requires gfx12)") + sinks = torch.linspace( + -1.0, + 8.0, + 16, + device="cuda:0", + dtype=torch.bfloat16 if head_size == 64 else torch.float32, + ) + case = _make_paged_attention_case( + [1, 7], + [127, 130], + num_heads=16, + num_kv_heads=2, + head_size=head_size, + block_size=32, + fp8=fp8, + sliding_window=window_size, + sinks=sinks, + ) + output = torch.empty_like(case["query"]) + _run_segmented_case( + case, + output, + max_query_len=7, + sliding_window=window_size - 1 if window_size else -1, + sinks=sinks, + ) + relative = (output.float() - case["reference"]).norm(dim=-1) / case[ + "reference" + ].norm(dim=-1).clamp_min(1e-6) + assert torch.isfinite(output).all() and relative.max().item() < 0.01 + + +@pytest.mark.parametrize("fp8", [False, True]) +@pytest.mark.parametrize("window_size", [0, 128]) +@pytest.mark.parametrize("query_len", [1, 8]) +@torch.inference_mode() +def test_segmented_sink_split_reduce_matches_reference(fp8, window_size, query_len): + from vllm.platforms.rocm import on_gfx1x, on_gfx12x + from vllm.v1.attention.ops.segmented_attention import ( + run_segmented_attention, + select_segmented_config, + ) + + if not current_platform.is_rocm() or not (on_gfx12x() if fp8 else on_gfx1x()): + pytest.skip("gfx1x segmented prefill (FP8 requires gfx12)") + sinks = torch.linspace(-1.0, 8.0, 16, device="cuda:0") + case = _make_paged_attention_case( + [query_len], + [8192 - query_len], + num_heads=16, + num_kv_heads=2, + head_size=64, + block_size=64, + fp8=fp8, + sliding_window=window_size, + sinks=sinks, + ) + attention_span = min(8192, window_size + query_len) if window_size else 8192 + config = dict( + select_segmented_config(1, query_len, attention_span, 16, 2, 64, fp8), + splits=4, + ) + output = torch.empty_like(case["query"]) + run_segmented_attention( + case["query"], + output, + case["key_cache"], + case["value_cache"], + case["block_table"], + case["starts"], + case["seq_lens"], + query_len, + 8192, + case["k_scale"], + case["v_scale"], + 64**-0.5, + sliding_window=window_size - 1 if window_size else -1, + sinks=sinks, + config=config, + ) + relative = (output.float() - case["reference"]).norm(dim=-1) / case[ + "reference" + ].norm(dim=-1).clamp_min(1e-6) + assert torch.isfinite(output).all() and relative.max().item() < 0.01 + + +@pytest.mark.parametrize("query_len,context_len", [(1, 8191), (8, 32760), (1, 131071)]) +@torch.inference_mode() +def test_segmented_long_context_fp8_sinks_match_dense_reference(query_len, context_len): + from vllm.platforms.rocm import on_gfx12x + + if not current_platform.is_rocm() or not on_gfx12x(): + pytest.skip("FP8 KV requires gfx12") + sinks = torch.linspace(-1.0, 1.0, 16, device="cuda:0", dtype=torch.bfloat16) + case = _make_paged_attention_case( + [query_len], + [context_len], + num_heads=16, + num_kv_heads=2, + head_size=64, + block_size=64, + fp8=True, + sinks=sinks, + ) + output = torch.empty_like(case["query"]) + _run_segmented_case(case, output, max_query_len=query_len, sinks=sinks) + relative = (output.float() - case["reference"]).norm(dim=-1) / case[ + "reference" + ].norm(dim=-1).clamp_min(1e-6) + assert torch.isfinite(output).all() and relative.max().item() < 0.01 + + +@pytest.mark.parametrize( + "feature", + [ + "softcap", + "sliding_softcap", + "output_scale", + ], +) +@torch.inference_mode() +def test_segmented_softcap_and_output_scaling_match_dense_reference(feature): + from vllm.platforms.rocm import on_gfx1x + from vllm.v1.attention.ops import segmented_attention as dispatcher + + if not current_platform.is_rocm() or not on_gfx1x(): + pytest.skip("gfx1x unified attention feature fallback") + + num_heads = 8 + configured_sliding_window = 16 if feature.startswith("sliding") else 0 + sliding_window = configured_sliding_window - 1 if configured_sliding_window else -1 + softcap = 5.0 if "softcap" in feature else 0.0 + case = _make_paged_attention_case( + [7, 2], + [40, 35], + num_heads=num_heads, + num_kv_heads=2, + head_size=128, + block_size=32, + fp8=False, + sliding_window=configured_sliding_window, + softcap=softcap, + sinks=None, + ) + output_scale = None + if feature == "output_scale": + output_scale = torch.tensor(0.5, device="cuda:0") + output = torch.empty_like(case["query"], dtype=current_platform.fp8_dtype()) + else: + output = torch.empty_like(case["query"]) + + dispatcher.segmented_attention( + query=case["query"], + key=case["key"], + value=case["value"], + output=output, + kv_cache_dtype="auto", + key_cache=case["key_cache"], + value_cache=case["value_cache"], + block_table=case["block_table"], + query_start_loc=case["starts"], + seq_lens=case["seq_lens"], + max_seq_len=case["max_seq_len"], + max_query_len=7, + k_scale=case["k_scale"], + v_scale=case["v_scale"], + sm_scale=128**-0.5, + sliding_window=sliding_window, + softcap=softcap, + output_scale=output_scale, + sinks=None, + ) + + actual = output.float() + if output_scale is not None: + actual *= output_scale + torch.testing.assert_close(actual, case["reference"], atol=0.2, rtol=0.2) + else: + relative = (actual - case["reference"]).norm(dim=-1) / case["reference"].norm( + dim=-1 + ).clamp_min(1e-6) + assert torch.isfinite(output).all() and relative.max().item() < 0.01 + + +@torch.inference_mode() +def test_segmented_attention_cache_offsets_cross_int32_boundary(): + """A small logical prefix can live beyond 2**31 elements in a strided cache.""" + from vllm.platforms.rocm import on_gfx1x + from vllm.v1.attention.ops.segmented_attention import run_segmented_attention + + if not current_platform.is_rocm() or not on_gfx1x(): + pytest.skip("gfx1x segmented prefill") + torch.accelerator.empty_cache() + free, _ = torch.accelerator.get_memory_info() + if free < 10 * 2**30: + pytest.skip("The address-boundary fixture needs 10 GiB free VRAM") + device = torch.device("cuda:0") + stride = 2**30 + 4096 + kc = torch.empty_strided( + (3, 32, 1, 128), + (stride, 128, 128, 1), + device=device, + dtype=torch.bfloat16, + ) + vc = torch.empty_strided( + (3, 32, 1, 128), + (stride, 128, 128, 1), + device=device, + dtype=torch.bfloat16, + ) + dense_k = torch.randn(2, 32, 1, 128, device=device, dtype=torch.bfloat16) + dense_v = torch.randn_like(dense_k) + for logical, physical in enumerate((2, 1)): + kc[physical].copy_(dense_k[logical]) + vc[physical].copy_(dense_v[logical]) + q = torch.randn(2, 4, 128, device=device, dtype=torch.bfloat16) + out = torch.empty_like(q) + table = torch.tensor([[2, 1]], device=device, dtype=torch.int32) + starts = torch.tensor([0, 2], device=device, dtype=torch.int32) + lengths = torch.tensor([35], device=device, dtype=torch.int32) + one = torch.ones((), device=device) + run_segmented_attention( + q, out, kc, vc, table, starts, lengths, 2, 35, one, one, 128**-0.5 + ) + full_k = dense_k.flatten(0, 1)[:35].float().repeat_interleave(4, 1) + full_v = dense_v.flatten(0, 1)[:35].float().repeat_interleave(4, 1) + scores = torch.einsum("qhd,khd->hqk", q.float(), full_k) / math.sqrt(128) + scores[:, 0, -1] = -float("inf") + expected = torch.einsum("hqk,khd->qhd", scores.softmax(-1), full_v) + error = ((out.float() - expected).norm(dim=-1) / expected.norm(dim=-1)).max() + assert torch.isfinite(out).all() and error < 0.01 + + +@pytest.mark.parametrize("dim,hq,hk", [(128, 16, 1), (256, 12, 2)]) +@pytest.mark.parametrize("fp8", [False, True]) +def test_segmented_attention_reservation_covers_ragged_query_caps( + monkeypatch, dim, hq, hk, fp8 +): + """Every supported query bucket fits the startup buffer after locking.""" + from vllm.v1.attention.ops import segmented_attention as segmented + + sizes = segmented.segmented_attention_workspace_size( + 32, + hq, + hk, + dim, + 65536, + fp8=fp8, + ) + workspace = tuple(torch.empty(size) for size in sizes) + query_lengths = { + query_len + for capacity in segmented._query_capacity_buckets() + for query_len in (max(1, capacity // 2 + 1), capacity) + } + for batch in range(1, 33): + for query_len in query_lengths: + qcap = segmented.segmented_query_capacity(query_len) + cfg = segmented.select_segmented_config( + batch, query_len, 65536, hq, hk, dim, fp8 + ) + shapes = segmented.segmented_workspace_shapes( + batch, qcap, hq, hk, dim, cfg["splits"] + ) + if shapes is not None: + assert math.prod(shapes[0]) <= workspace[0].numel() + assert math.prod(shapes[1]) <= workspace[1].numel() + + +def test_segmented_attention_reservation_respects_scheduler_token_limit( + monkeypatch, +) -> None: + """Workspace planning excludes batch/query pairs the scheduler cannot form.""" + from vllm.v1.attention.ops import segmented_attention as segmented + + calls = [] + + def record_config(batch, query_len, *_args): + calls.append((batch, query_len)) + return {"splits": 1} + + monkeypatch.setattr(segmented, "select_segmented_config", record_config) + segmented.segmented_attention_workspace_size.cache_clear() + segmented.segmented_attention_workspace_size( + 32, + 16, + 2, + 128, + 65536, + max_tokens=8, + ) + + assert calls + assert all(batch + query_len - 1 <= 8 for batch, query_len in calls) + assert (8, 1) in calls + assert (1, 8) in calls + + +def test_segmented_attention_query_capacity_buckets() -> None: + from vllm.v1.attention.ops.segmented_attention import ( + MAX_QUERY_LEN, + segmented_query_capacity, + ) + + assert [ + segmented_query_capacity(query_len) + for query_len in (1, 2, 3, 33, 129, 1025, 2049, MAX_QUERY_LEN) + ] == [1, 2, 4, 64, 256, 2048, 4096, 8192] + with pytest.raises(ValueError, match="Query length"): + segmented_query_capacity(MAX_QUERY_LEN + 1) + + +def test_segmented_scratch_is_stable_and_isolated_across_runtime_lanes(monkeypatch): + """Draft/DBO scratch never aliases target scratch or generic workspace.""" + import vllm.v1.worker.workspace as workspace + from vllm.v1.attention.ops import segmented_attention as segmented + + active_ubatch = [0] + monkeypatch.setattr(workspace, "dbo_current_ubatch_id", lambda: active_ubatch[0]) + manager = workspace.WorkspaceManager( + torch.device("cpu"), num_ubatches=2, num_lanes=2 + ) + monkeypatch.setattr(segmented, "is_workspace_manager_initialized", lambda: True) + monkeypatch.setattr(segmented, "current_workspace_manager", lambda: manager) + buffers = {} + for ubatch in range(2): + active_ubatch[0] = ubatch + for lane in range(2): + with workspace.use_workspace_lane(lane): + buffers[ubatch, lane] = segmented.get_segmented_attention_workspace( + torch.device("cpu"), (1024, 16) + ) + manager.get_simultaneous(((2048,), torch.float32)) + manager.lock() + pointers: set[int] = set() + for (ubatch, lane), expected in buffers.items(): + active_ubatch[0] = ubatch + with workspace.use_workspace_lane(lane): + actual = segmented.get_segmented_attention_workspace( + torch.device("cpu"), (1024, 16) + ) + assert actual is expected + pointers.update(t.data_ptr() for t in actual) + assert ( + manager.get_simultaneous(((1,), torch.float32))[0].data_ptr() + not in pointers + ) + assert len(pointers) == 8 diff --git a/tests/kernels/attention/test_triton_unified_attention.py b/tests/kernels/attention/test_triton_unified_attention.py index e30d9225ea..d936e37704 100644 --- a/tests/kernels/attention/test_triton_unified_attention.py +++ b/tests/kernels/attention/test_triton_unified_attention.py @@ -181,6 +181,53 @@ def ref_paged_attn( return torch.cat(outputs, dim=0) +@torch.inference_mode() +def test_fp8_softmax_preserves_small_probabilities() -> None: + """Keep exp(-8) contributions that underflow when cast directly to E4M3.""" + device = torch.device(DEVICE_TYPE) + num_tokens = block_size = 32 + head_size = 128 + + query = torch.zeros(1, 1, head_size, dtype=FP8_DTYPE, device=device) + query[..., 0] = 1 + key_cache = torch.zeros(1, block_size, 1, head_size, dtype=FP8_DTYPE, device=device) + key_cache[:, 1:, :, 0] = -8 + value_cache = torch.ones_like(key_cache) + value_cache[:, 0] = 0 + output = torch.empty(1, 1, head_size, dtype=torch.bfloat16, device=device) + + cu_seqlens_q = torch.tensor([0, 1], dtype=torch.int32, device=device) + seqused_k = torch.tensor([num_tokens], dtype=torch.int32, device=device) + block_table = torch.tensor([[0]], dtype=torch.int32, device=device) + scale = torch.ones(1, dtype=torch.float32, device=device) + + unified_attention( + q=query, + k=key_cache, + v=value_cache, + out=output, + cu_seqlens_q=cu_seqlens_q, + max_seqlen_q=1, + seqused_k=seqused_k, + max_seqlen_k=num_tokens, + softmax_scale=1.0, + causal=True, + window_size=(-1, -1), + block_table=block_table, + softcap=0, + q_descale=scale, + k_descale=scale, + v_descale=scale, + kv_quant_mode=KVQuantMode.FP8_PER_TENSOR, + ) + + scores = torch.einsum("qhd,thd->qht", query.float(), key_cache[0].float()) + probabilities = torch.softmax(scores, dim=-1) + expected = torch.einsum("qht,thd->qhd", probabilities, value_cache[0].float()) + + torch.testing.assert_close(output.float(), expected, atol=5e-5, rtol=1e-2) + + def ref_paged_clamped_mm_attn( query: torch.Tensor, key_cache: torch.Tensor, diff --git a/tests/kernels/attention/test_triton_unified_attention_diffkv.py b/tests/kernels/attention/test_triton_unified_attention_diffkv.py index 213d896c03..3b6cf524e2 100644 --- a/tests/kernels/attention/test_triton_unified_attention_diffkv.py +++ b/tests/kernels/attention/test_triton_unified_attention_diffkv.py @@ -35,6 +35,8 @@ NUM_HEADS = [(4, 4), (8, 2), (5, 1)] HEAD_SIZES = [(128, 128), (192, 128)] BLOCK_SIZES = [16] DTYPES = [torch.bfloat16] +# None keeps the KV cache in the query dtype. +KV_CACHE_DTYPES = [None, current_platform.fp8_dtype()] NUM_BLOCKS = 2048 @@ -80,6 +82,7 @@ def _alloc_segm_buffers(seq_threshold_3D: int, num_query_heads: int, head_size_v @pytest.mark.parametrize("sliding_window", [None, 128]) @pytest.mark.parametrize("soft_cap", [None, 50.0]) @pytest.mark.parametrize("dtype", DTYPES) +@pytest.mark.parametrize("kv_cache_dtype", KV_CACHE_DTYPES) @pytest.mark.parametrize("seq_threshold_3D", SEQ_THRESHOLD_3D_VALUES) @torch.inference_mode() def test_triton_unified_attn_diffkv_vs_reference( @@ -89,13 +92,17 @@ def test_triton_unified_attn_diffkv_vs_reference( sliding_window: int | None, soft_cap: float | None, dtype: torch.dtype, + kv_cache_dtype: torch.dtype | None, block_size: int, seq_threshold_3D: int, + quant_query: bool = False, ) -> None: head_size_qk, head_size_v = head_sizes - # Keep the FA3/FA4 comparison on NVIDIA; ROCm uses the PyTorch oracle. - if not current_platform.is_rocm(): + # FA3/FA4 is the reference on NVIDIA; ROCm and the FP8 KV cache (FA would + # also quantize Q) use the PyTorch oracle. + use_torch_ref = current_platform.is_rocm() or kv_cache_dtype is not None + if not use_torch_ref: fa_version = get_flash_attn_version( head_size=head_size_qk, head_size_v=head_size_v ) @@ -126,6 +133,20 @@ def test_triton_unified_attn_diffkv_vs_reference( ) key_cache = kv_cache[..., :head_size_qk] value_cache = kv_cache[..., head_size_qk:] + q_descale = k_descale = v_descale = None + if kv_cache_dtype is not None: + # Non-power-of-two scales exercise descale rounding. + k_descale = torch.tensor(0.3, dtype=torch.float32) + v_descale = torch.tensor(0.7, dtype=torch.float32) + kv_cache = torch.cat( + [key_cache / k_descale, value_cache / v_descale], dim=-1 + ).to(kv_cache_dtype) + key_cache = kv_cache[..., :head_size_qk] + value_cache = kv_cache[..., head_size_qk:] + + if quant_query: + q_descale = torch.tensor(0.2, dtype=torch.float32) + query = (query / q_descale).to(kv_cache_dtype) cu_query_lens = torch.tensor([0] + query_lens, dtype=torch.int32).cumsum( dim=0, dtype=torch.int32 @@ -137,12 +158,20 @@ def test_triton_unified_attn_diffkv_vs_reference( 0, NUM_BLOCKS, (num_seqs, max_num_blocks_per_seq), dtype=torch.int32 ) - if current_platform.is_rocm(): + if use_torch_ref: + ref_key_cache = key_cache.float() + ref_value_cache = value_cache.float() + if kv_cache_dtype is not None: + ref_key_cache *= k_descale + ref_value_cache *= v_descale # FP32 also keeps the helper's in-place scaling off the kernel input. + ref_query = query.float() + if quant_query: + ref_query *= q_descale ref_out = ref_paged_attn( - query.float(), - key_cache.float(), - value_cache.float(), + ref_query, + ref_key_cache, + ref_value_cache, query_lens, kv_lens, block_tables, @@ -200,6 +229,32 @@ def test_triton_unified_attn_diffkv_vs_reference( softmax_segm_output=segm_output, softmax_segm_max=segm_max, softmax_segm_expsum=segm_expsum, + q_descale=q_descale, + k_descale=k_descale, + v_descale=v_descale, ) torch.testing.assert_close(triton_out, ref_out, atol=2e-2, rtol=2e-2) + + +@pytest.mark.skipif( + not current_platform.is_cuda() or not current_platform.has_device_capability(89), + reason="FP8 query requires CUDA SM89+", +) +@pytest.mark.parametrize( + "seq_lens,seq_threshold_3D", + [([(129, 463)], 0), ([(1, 2011)], 0), ([(1, 2011)], 8)], +) +def test_triton_unified_attn_diffkv_fp8_query(seq_lens, seq_threshold_3D): + test_triton_unified_attn_diffkv_vs_reference( + seq_lens=seq_lens, + num_heads=(8, 2), + head_sizes=(192, 128), + sliding_window=None, + soft_cap=None, + dtype=torch.bfloat16, + kv_cache_dtype=current_platform.fp8_dtype(), + block_size=16, + seq_threshold_3D=seq_threshold_3D, + quant_query=True, + ) diff --git a/tests/kernels/core/test_uva.py b/tests/kernels/core/test_uva.py index b7c2984d6a..9d88eeced3 100644 --- a/tests/kernels/core/test_uva.py +++ b/tests/kernels/core/test_uva.py @@ -34,6 +34,7 @@ def test_cpu_write(device): cpu_tensor[4, 5] = -1 gpu_view.mul_(2) + torch.accelerator.synchronize(gpu_view.device) assert gpu_view[0, 0] == 2 assert gpu_view[2, 3] == 4 assert gpu_view[4, 5] == -2 @@ -55,6 +56,7 @@ def test_gpu_write(device): gpu_view[2, 3] = 2 gpu_view[4, 5] = -1 gpu_view.mul_(2) + torch.accelerator.synchronize(gpu_view.device) assert cpu_tensor[0, 0] == 2 assert cpu_tensor[2, 3] == 4 @@ -232,6 +234,25 @@ def test_non_pinned_cpu_tensor(device): assert gpu_view[9, 9] == 198 +@pytest.mark.skipif(not current_platform.is_cuda(), reason="CUDA-only sync check.") +def test_non_uva_buffer_copies_without_stream_sync(): + buffer = buffer_utils.NonUvaBuffer(8, torch.int32) + buffer.np[:] = np.arange(8) + + torch.cuda.set_sync_debug_mode("error") + try: + prefix = buffer.uva(4) + full = buffer.uva() + finally: + torch.cuda.set_sync_debug_mode(0) + # The next step overwrites the host buffer right after the copy is issued. + buffer.np[:] = -1 + torch.accelerator.synchronize() + + assert prefix.tolist() == list(range(4)) + assert full.tolist() == list(range(8)) + + @pytest.mark.skipif(not is_uva_available(), reason="UVA is not available.") @pytest.mark.skipif( not current_platform.is_xpu(), reason="XPU non-contiguous UVA test." diff --git a/tests/kernels/mamba/test_mamba2_checkpoint_state.py b/tests/kernels/mamba/test_mamba2_checkpoint_state.py new file mode 100644 index 0000000000..54fcf80205 --- /dev/null +++ b/tests/kernels/mamba/test_mamba2_checkpoint_state.py @@ -0,0 +1,163 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""An SSD intermediate state equals an independent run over the same prefix. + +This is the central claim of internal prefill checkpoints: the state the SSD +scan already materializes at the checkpoint's chunk boundary is exactly the +state a fresh prefill over `[0, checkpoint)` would end with, so it can be +cached and later resumed from. The last test covers the other half: moving +that state, and the conv window, into the paged cache. +""" + +import pytest +import torch + +from tests.kernels.mamba.test_mamba_ssm_ssd import generate_random_inputs +from vllm.model_executor.layers.mamba.checkpoint import ( + MambaPrefillCheckpointMetadata, +) +from vllm.model_executor.layers.mamba.ops.ssd_checkpoint import ( + store_prefill_checkpoint, +) +from vllm.model_executor.layers.mamba.ops.ssd_combined import ( + mamba_chunk_scan_combined_varlen, +) +from vllm.platforms import current_platform +from vllm.v1.attention.backends.mamba_attn import BaseMambaAttentionMetadataBuilder +from vllm.v1.attention.backends.utils import NULL_BLOCK_ID + +DEVICE = current_platform.device_type +compute_chunk_metadata = BaseMambaAttentionMetadataBuilder._compute_chunk_metadata + + +def _run(inputs, seqlen, chunk_size, checkpoint_offset): + """Run one prefill of `seqlen` tokens, optionally exporting a checkpoint. + + Returns (states, checkpoint_chunk_index). `states` holds every chunk's + state when a checkpoint was requested, else the per-sequence final state. + """ + A, dt, X, B, C = inputs + offsets = [checkpoint_offset] if checkpoint_offset else None + cu, seq_idx, last, ckpt = compute_chunk_metadata( + chunk_size, + 1, + torch.tensor([0]), + torch.tensor([0, seqlen]), + checkpoint_offsets_p=offsets, + ) + + x = X[:seqlen] + out = torch.empty_like(x) + states = mamba_chunk_scan_combined_varlen( + x, + dt[:seqlen], + A, + B[:seqlen], + C[:seqlen], + chunk_size, + cu_seqlens=torch.tensor([0, seqlen], dtype=torch.int32, device=DEVICE), + cu_chunk_seqlens=torch.tensor(cu, dtype=torch.int32, device=DEVICE), + last_chunk_indices=torch.tensor(last, dtype=torch.int32, device=DEVICE), + seq_idx=torch.tensor(seq_idx, dtype=torch.int32, device=DEVICE), + out=out, + D=None, + return_intermediate_states=checkpoint_offset > 0, + state_dtype=torch.float32, + ) + return states, ckpt[0] + + +@pytest.mark.parametrize("checkpoint_offset", [32, 48, 96]) +def test_checkpoint_state_matches_an_independent_prefix_run(checkpoint_offset): + """states[checkpoint_chunk_idx] == the final state of a prefix-only run. + + 48 is deliberately not a multiple of chunk_size: align-mode block sizes + carry no chunk_size factor, so off-grid checkpoints are the common case + and are what the chunk split exists to support. + """ + seqlen, chunk_size, n_heads, d_head = 128, 32, 4, 32 + inputs = generate_random_inputs(1, seqlen, n_heads, d_head, torch.float32) + inputs = tuple(t.squeeze(0) if t.dim() > 1 else t for t in inputs) + + states, ckpt_idx = _run(inputs, seqlen, chunk_size, checkpoint_offset) + assert ckpt_idx >= 0, "the checkpoint did not land on a chunk boundary" + checkpoint_state = states[ckpt_idx] + + prefix_final, _ = _run(inputs, checkpoint_offset, chunk_size, 0) + + torch.testing.assert_close( + checkpoint_state.to(torch.float32), + prefix_final[0].to(torch.float32), + atol=8e-3, + rtol=5e-3, + ) + + +def test_checkpoint_does_not_perturb_the_final_state(): + """Exporting a checkpoint must not change what the prefill produces. + + The chunk split is an extra boundary, not a different computation, so the + sequence's final state has to match a run that took no checkpoint. + """ + seqlen, chunk_size, n_heads, d_head = 128, 32, 4, 32 + inputs = generate_random_inputs(1, seqlen, n_heads, d_head, torch.float32) + inputs = tuple(t.squeeze(0) if t.dim() > 1 else t for t in inputs) + + with_ckpt, ckpt_idx = _run(inputs, seqlen, chunk_size, 48) + without_ckpt, _ = _run(inputs, seqlen, chunk_size, 0) + + torch.testing.assert_close( + with_ckpt[-1].to(torch.float32), + without_ckpt[0].to(torch.float32), + atol=8e-3, + rtol=5e-3, + ) + + +@pytest.mark.parametrize("dim_first", [True, False]) +def test_store_selects_the_checkpoint_from_all_chunk_states(dim_first): + """The store reads the checkpoint row out of every chunk state. + + Row 0 checkpoints 32 tokens in, its state being chunk 3 rather than the + program's own row, so the store only passes if ``chunk_idx`` is used. + Row 1 declines and must write nothing. The conv input is a column slice of + a wider projection, the SD conv layout is a transposed view, and SSM rows + are padded pages, matching the strides the mixer passes. + """ + dim, state_len, heads, head_dim, d_state = 16, 4, 2, 4, 8 + projection = torch.randn(64, dim + 8, device=DEVICE) + conv_input = projection[:, :dim] + if dim_first: + conv_state = torch.zeros(3, dim, state_len, device=DEVICE) + else: + conv_state = torch.zeros(3, state_len, dim, device=DEVICE).transpose(-1, -2) + varlen_states = torch.randn(5, heads, head_dim, d_state, device=DEVICE) + # Blocks are pages holding more than the SSM state, so rows are padded. + row = heads * head_dim * d_state + ssm_storage = torch.zeros(3, row + 8, device=DEVICE) + ssm_state = ssm_storage[:, :row].view(3, heads, head_dim, d_state) + checkpoint = MambaPrefillCheckpointMetadata( + checkpoint_offsets=torch.tensor([32, 0], dtype=torch.int32, device=DEVICE), + state_indices=torch.tensor( + [2, NULL_BLOCK_ID], dtype=torch.int32, device=DEVICE + ), + offsets=[32, 0], + ) + + store_prefill_checkpoint( + checkpoint, + torch.tensor([3, 0], device=DEVICE), + conv_input.T, + conv_state, + varlen_states, + ssm_state, + torch.tensor([0, 40, 64], dtype=torch.int32, device=DEVICE), + ) + + exact = dict(rtol=0, atol=0) + torch.testing.assert_close( + conv_state[2], conv_input[32 - state_len : 32].T, **exact + ) + torch.testing.assert_close(ssm_state[2], varlen_states[3], **exact) + assert not conv_state[:2].any() and not ssm_storage[:2].any() + assert not ssm_storage[:, row:].any(), "wrote into the page padding" diff --git a/tests/kernels/mamba/test_precopy_mamba_align.py b/tests/kernels/mamba/test_precopy_mamba_align.py index 22b73c936e..8a91f8bdac 100644 --- a/tests/kernels/mamba/test_precopy_mamba_align.py +++ b/tests/kernels/mamba/test_precopy_mamba_align.py @@ -31,7 +31,11 @@ import torch from vllm.model_executor.layers.mamba import mamba_utils as layer_mamba_utils from vllm.platforms import current_platform from vllm.v1.worker import mamba_utils as worker_mamba_utils -from vllm.v1.worker.mamba_utils import _TEMPORAL_TILES, precopy_mamba_align_fused_kernel +from vllm.v1.worker.mamba_utils import ( + _TEMPORAL_TILES, + postprocess_mamba_fused_kernel, + precopy_mamba_align_fused_kernel, +) _parametrize: Callable[..., Callable[[Any], Any]] @@ -195,7 +199,8 @@ def test_precopy_matches_v1_copy_specs( convs, ssms, device, conv_state_dim_first ) bt_ptrs = torch.tensor([bt.data_ptr()], dtype=torch.int64, device=device) - idx_mapping = torch.arange(num_reqs, dtype=torch.int32, device=device) + # Non-identity mapping: block-table rows are request slots, not batch rows. + idx_mapping = torch.arange(num_reqs, dtype=torch.int32, device=device).flip(0) grid = (num_reqs, NUM_LAYERS * 2, temporal_tiles) precopy_mamba_align_fused_kernel[grid]( dst_col, @@ -225,6 +230,77 @@ def test_precopy_matches_v1_copy_specs( torch.testing.assert_close(ssms[layer], ssm_ref[layer], rtol=0, atol=0) +@_parametrize("conv_state_dim_first", [False, True]) +@_cuda_required +def test_v2_postprocess_indexes_block_tables_by_req_slot(conv_state_dim_first): + """Under PP the V2 postprocess runs after later steps have re-gathered the + batch-ordered tables, so it must index the request-slot tables by slot.""" + device = torch.device("cuda") + torch.manual_seed(0) + num_reqs = 4 + block_size = 8 + num_blocks = num_reqs * MAX_COLS + 1 + bt = torch.arange(1, num_blocks, dtype=torch.int32, device=device).view( + num_reqs, MAX_COLS + ) + # slot 0: copy col 1 -> 0; slot 1: src == dst; slot 2: not aligned; + # slot 3: copy col 2 -> 0 with one accepted draft token. + num_accepted = torch.tensor([1, 1, 1, 2], dtype=torch.int32, device=device) + state_idx = torch.tensor([1, 0, 1, 2], dtype=torch.int32, device=device) + new_num_computed = torch.tensor([8, 8, 5, 8], dtype=torch.int32, device=device) + ref_src = torch.tensor([1, 0, -1, 2], dtype=torch.int32) + ref_dst = torch.zeros(num_reqs, dtype=torch.int32) + ref_bias = torch.tensor([0, 0, 0, 1], dtype=torch.int32) + + convs, ssms = _build_state(num_blocks, device, conv_state_dim_first) + conv_ref, ssm_ref = _reference( + convs, + ssms, + bt.cpu(), + ref_src, + ref_dst, + ref_bias, + num_reqs, + conv_state_dim_first, + ) + base, blk_stride, elem, inner, width, group, drc, drs = _build_meta( + convs, ssms, device, conv_state_dim_first + ) + bt_ptrs = torch.tensor([bt.data_ptr()], dtype=torch.int64, device=device) + idx_mapping = torch.tensor([2, 0, 3, 1], dtype=torch.int32, device=device) + postprocess_mamba_fused_kernel[(num_reqs, NUM_LAYERS * 2, 1)]( + num_accepted.clone(), + state_idx, + None, + new_num_computed, + None, + bt_ptrs, + bt.stride(0), + base, + blk_stride, + elem, + inner, + width, + group, + drc, + drs, + num_accepted, + idx_mapping, + num_reqs, + block_size=block_size, + COPY_BLOCK_SIZE=1024, + CONV_STATE_DIM_FIRST=conv_state_dim_first, + HAS_IDX_MAPPING=True, + PRECOMPUTED_NEW_COMPUTED=True, + TEMPORAL_TILES=1, + ) + torch.accelerator.synchronize() + + for layer in range(NUM_LAYERS): + torch.testing.assert_close(convs[layer], conv_ref[layer], rtol=0, atol=0) + torch.testing.assert_close(ssms[layer], ssm_ref[layer], rtol=0, atol=0) + + def test_ds_conv_copy_spec_reproduces_multi_accept_assert(monkeypatch): monkeypatch.setattr( layer_mamba_utils, diff --git a/tests/kernels/moe/modular_kernel_tools/common.py b/tests/kernels/moe/modular_kernel_tools/common.py index 9ae6092c94..5f6349a328 100644 --- a/tests/kernels/moe/modular_kernel_tools/common.py +++ b/tests/kernels/moe/modular_kernel_tools/common.py @@ -1,6 +1,6 @@ # SPDX-License-Identifier: Apache-2.0 # SPDX-FileCopyrightText: Copyright contributors to the vLLM project -from dataclasses import dataclass +from dataclasses import dataclass, replace from types import SimpleNamespace from typing import Any @@ -702,6 +702,21 @@ def make_modular_kernel( return modular_kernel +def _shuffle_weights_for_aiter(rank_weights: WeightTensors) -> WeightTensors: + """Pre-shuffle weights so AITER selects its prebuilt `preshuffle_on` module. + + Production shuffles in `process_weights_after_loading`; this harness builds its + tensors directly, so without this every call asks for a `preshuffle_off` module + that is not prebuilt, and JIT-compiles a kernel the image already ships. + """ + from vllm._aiter_ops import rocm_aiter_ops + + w1, w2 = rocm_aiter_ops.shuffle_weights(rank_weights.w1, rank_weights.w2) + w1.is_shuffled = True + w2.is_shuffled = True + return replace(rank_weights, w1=w1, w2=w2) + + def _maybe_convert_weights_for_experts( config: Config, rank_weights: WeightTensors, @@ -715,6 +730,15 @@ def _maybe_convert_weights_for_experts( fe_type = config.fused_experts_type fe_name = getattr(fe_type, "__name__", "") + # AITER's prebuilt modules carry a gfx950 instance table even on gfx942 and reject + # intermediate sizes that are not a multiple of 256, but only for the fp8 per-tensor + # and per-token schemes. Those keep JIT-compiling until + # https://github.com/ROCm/aiter/issues/5766 is fixed. + if fe_name == "AiterExperts" and ( + config.quant_dtype is None or config.quant_block_shape is not None + ): + return _shuffle_weights_for_aiter(rank_weights) + backend: Fp8MoeBackend | None = None if fe_name == "TrtLlmFp8ExpertsModular": backend = Fp8MoeBackend.FLASHINFER_TRTLLM diff --git a/tests/kernels/moe/test_cutlass_moe.py b/tests/kernels/moe/test_cutlass_moe.py index 1910796623..050e1bc104 100644 --- a/tests/kernels/moe/test_cutlass_moe.py +++ b/tests/kernels/moe/test_cutlass_moe.py @@ -37,9 +37,12 @@ from vllm.model_executor.layers.fused_moe.experts.cutlass_moe import ( run_cutlass_moe_fp4, run_cutlass_moe_fp8, run_cutlass_moe_mxfp4, + swizzle_mxfp4_scales, ) +from vllm.model_executor.layers.fused_moe.oracle import mxfp4 as mxfp4_oracle from vllm.model_executor.layers.fused_moe.oracle import nvfp4 as nvfp4_oracle from vllm.model_executor.layers.fused_moe.utils import moe_kernel_quantize_input +from vllm.model_executor.layers.quantization.utils.quant_utils import kMxfp4Dynamic from vllm.platforms import current_platform from vllm.utils.torch_utils import set_random_seed @@ -294,6 +297,87 @@ def test_cutlass_fp4_moe_padded_routes_do_not_change_valid_output(quantization: torch.testing.assert_close(actual[1:], torch.zeros_like(actual[1:])) +@torch.inference_mode() +def test_cutlass_mxfp4_oracle_matches_direct_kernel(monkeypatch, workspace_init): + """The MXFP4 oracle selects CUTLASS for W4A4 and swizzles checkpoint + scales into the layout the kernel expects.""" + if not CutlassExpertsMxfp4._supports_current_device(): + pytest.skip("CUTLASS mxfp4 MoE is not supported on this GPU") + monkeypatch.setattr(mxfp4_oracle, "_user_moe_activation_override", lambda: None) + + set_random_seed(7) + e, m, n, k, topk = 8, 33, 256, 512, 2 + device = torch.device("cuda") + w1 = torch.randint(0, 256, (e, 2 * n, k // 2), device=device, dtype=torch.uint8) + w2 = torch.randint(0, 256, (e, k, n // 2), device=device, dtype=torch.uint8) + w1_scale = torch.randint(118, 124, (e, 2 * n, k // 32), device=device).byte() + w2_scale = torch.randint(118, 124, (e, k, n // 32), device=device).byte() + a = torch.randn((m, k), device=device, dtype=torch.bfloat16) + score = torch.randn((m, e), device=device, dtype=torch.bfloat16) + topk_weights, topk_ids, _ = fused_topk(a, score, topk, renormalize=False) + + def swizzle(scale: torch.Tensor) -> torch.Tensor: + rows, cols = scale.shape[1:] + return torch.stack( + [swizzle_mxfp4_scales(s, rows, cols * 32).view(rows, cols) for s in scale] + ) + + expected = torch.empty_like(a) + run_cutlass_moe_mxfp4( + output=expected, + a=a, + w1_fp4=w1, + w1_blockscale=swizzle(w1_scale), + w2_fp4=w2, + w2_blockscale=swizzle(w2_scale), + topk_weights=topk_weights, + topk_ids=topk_ids, + activation=MoEActivation.SILU, + workspace13=torch.empty((m * topk, max(2 * n, k)), device=device).to(a), + workspace2=torch.empty((m * topk, n), device=device).to(a), + m=m, + n=n, + k=k, + e=e, + device=device, + apply_router_weight_on_input=False, + ) + + moe_config = make_dummy_moe_config( + num_experts=e, experts_per_token=topk, hidden_dim=k, intermediate_size=n + ) + with set_current_vllm_config(vllm_config): + backend, experts_cls = mxfp4_oracle.select_mxfp4_moe_backend( + moe_config, activation_key=kMxfp4Dynamic + ) + assert backend == mxfp4_oracle.Mxfp4MoeBackend.CUTLASS_MXFP4_MXFP4 + + w13, w2, w13_scale, w2_scale, _, _ = ( + mxfp4_oracle.convert_weight_to_mxfp4_moe_kernel_format( + backend, torch.nn.Module(), w1, w2, w1_scale, w2_scale + ) + ) + kernel = mxfp4_oracle.make_mxfp4_moe_kernel( + mxfp4_oracle.make_mxfp4_moe_quant_config(backend, w13_scale, w2_scale), + moe_config, + experts_cls, + backend, + ) + actual = kernel.apply( + a, + w13, + w2, + topk_weights, + topk_ids, + activation=MoEActivation.SILU, + global_num_experts=e, + expert_map=None, + apply_router_weight_on_input=False, + ) + + torch.testing.assert_close(actual, expected, atol=0, rtol=0) + + @dataclasses.dataclass class MOETensors: a: torch.Tensor diff --git a/tests/kernels/moe/test_deepgemm.py b/tests/kernels/moe/test_deepgemm.py index cee0faa592..e341f12a71 100644 --- a/tests/kernels/moe/test_deepgemm.py +++ b/tests/kernels/moe/test_deepgemm.py @@ -70,6 +70,28 @@ def test_ep_gather_uses_64_bit_row_offsets(): torch.testing.assert_close(output, torch.ones_like(output), rtol=0, atol=0) +def test_ep_gather_uses_64_bit_output_row_offsets(): + hidden_size = 6144 + output_row = (1 << 31) // hidden_size + 1 + num_rows = output_row + 1 + output = torch.empty( + (num_rows, hidden_size), + device="cuda", + dtype=torch.bfloat16, + ) + + ep_gather( + input_tensor=torch.ones((1, hidden_size), device="cuda", dtype=torch.bfloat16), + recv_topk_ids=torch.zeros((num_rows, 1), device="cuda", dtype=torch.int64), + recv_topk_weight=torch.ones((num_rows, 1), device="cuda", dtype=torch.float32), + input_index=torch.zeros((num_rows, 1), device="cuda", dtype=torch.int32), + expert_map=None, + output_tensor=output, + ) + + torch.testing.assert_close(output, torch.ones_like(output), rtol=0, atol=0) + + @pytest.mark.parametrize("workspace_dtype", [torch.float16, torch.bfloat16]) @pytest.mark.parametrize("num_columns", [2048, 6144, 6145]) def test_fp8_workspace_shape(workspace_dtype, num_columns): @@ -158,6 +180,153 @@ def make_block_quant_fp8_weights( return w1, w2, w1_s, w2_s +@pytest.mark.parametrize("num_tokens", [16, 129, 512]) +@pytest.mark.parametrize("with_metadata", [False, True]) +@pytest.mark.skipif(not is_deep_gemm_supported(), reason="Requires deep_gemm kernels") +def test_block_fp8_padding_on_graph_replay(num_tokens, with_metadata, monkeypatch): + """Both FP8 GEMMs consume only live rows as routing changes on replay.""" + from unittest.mock import Mock + + import vllm.model_executor.layers.fused_moe.experts.deep_gemm_moe as impl + from vllm.utils.deep_gemm import DeepGemmQuantScaleFMT + + if not ( + impl.current_platform.is_device_capability_family(90) + or ( + impl.current_platform.is_device_capability_family(100) + and DeepGemmQuantScaleFMT.from_oracle() == DeepGemmQuantScaleFMT.UE8M0 + ) + ): + pytest.skip("Prefix-sum FP8 dispatch requires SM90 or SM10.x") + + torch.manual_seed(0) + e, n, k, topk = 8, 1024, 512, 2 + x = torch.randn(num_tokens, k, device="cuda", dtype=torch.bfloat16) / 10 + aq, scales = per_token_group_quant_fp8(x, 128, use_ue8m0=True) + w1, w2, s1, s2 = make_block_quant_fp8_weights(e, n, k, BLOCK_SIZE) + quant = fp8_w8a8_moe_quant_config(w1_scale=s1, w2_scale=s2, block_shape=BLOCK_SIZE) + experts = impl.DeepGemmExperts(make_dummy_moe_config(), quant) + ids = torch.zeros(num_tokens, topk, device="cuda", dtype=torch.int64) + weights = torch.full((num_tokens, topk), 0.5, device="cuda") + counts = torch.zeros(e, device="cuda", dtype=torch.int32) + metadata = mk.ExpertTokensMetadata(counts, None) if with_metadata else None + shapes = experts.workspace_shapes( + num_tokens, 2 * n, k, topk, e, e, metadata, MoEActivation.SILU + ) + + def route(mode): + ids[:, 0] = torch.arange(num_tokens, device="cuda") % e + ids[:, 1] = (ids[:, 0] + 1) % e + if mode == 1: + ids.fill_(-1) + ids[:1, 0] = e - 1 + elif mode == 2: + ids.fill_(-1) + counts.copy_(torch.bincount(ids[ids >= 0], minlength=e).int()) + + route(0) + variants = [] + for dense in (True, False): + ws1, ws2, output = [ + torch.empty(shape, device="cuda", dtype=torch.bfloat16) for shape in shapes + ] + + def invoke(output=output, ws1=ws1, ws2=ws2): + experts.apply( + output, + aq, + w1, + w2, + weights, + ids, + MoEActivation.SILU, + e, + None, + scales, + None, + ws1, + ws2, + metadata, + False, + ) + + with monkeypatch.context() as mp: + if dense: + platform = Mock(wraps=impl.current_platform) + platform.is_device_capability_family.return_value = False + mp.setattr(impl, "current_platform", platform) + gemm = impl.m_grouped_fp8_gemm_nt_contiguous + calls: list[int] = [] + + def check_layout(*args, dense=dense, calls=calls, gemm=gemm, **kwargs): + assert kwargs.get("use_psum_layout", False) == (not dense) + calls.append(args[3].numel()) + return gemm(*args, **kwargs) + + mp.setattr(impl, "m_grouped_fp8_gemm_nt_contiguous", check_layout) + invoke() + assert len(calls) == 2 + if not dense: + assert calls == [e, e] + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + invoke() + variants.append((graph, output, ws1, ws2)) + + for mode in (0, 1, 2, 0): + route(mode) + for graph, _, ws1, ws2 in variants: + ws1.fill_(float("nan")) + ws2.fill_(float("nan")) + graph.replay() + torch.accelerator.synchronize() + torch.testing.assert_close(variants[0][1], variants[1][1], rtol=0, atol=0) + assert torch.isfinite(variants[1][1]).all() + + +@pytest.mark.parametrize(("n", "k"), [(2048, 512), (512, 1024)]) +@pytest.mark.skipif(not is_deep_gemm_supported(), reason="Requires deep_gemm kernels") +def test_deepgemm_prefix_sum_leaves_unused_capacity_untouched(n, k): + """Both projection shapes skip unused tiles as counts change on replay.""" + from vllm.platforms import current_platform + from vllm.utils.deep_gemm import ( + m_grouped_fp8_gemm_nt_contiguous, + mk_alignment_scope, + ) + + if not ( + current_platform.is_device_capability_family(90) + or current_platform.is_device_capability_family(100) + ): + pytest.skip("Requires SM90 or SM10.x") + + capacity, experts, alignment = 512, 4, 128 + x = torch.randn(capacity, k, device="cuda", dtype=torch.bfloat16) / 10 + aq, scales = per_token_group_quant_fp8(x, 128, use_ue8m0=True) + w, _, ws, _ = make_block_quant_fp8_weights(experts, n // 2, k, BLOCK_SIZE) + ends = torch.tensor([0, 1, 128, 129], device="cuda", dtype=torch.int32) + output = torch.empty(capacity, n, device="cuda", dtype=torch.bfloat16) + + def invoke(): + with mk_alignment_scope(alignment): + m_grouped_fp8_gemm_nt_contiguous( + (aq, scales), (w, ws), output, ends, use_psum_layout=True + ) + + invoke() + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + invoke() + for bounds, allocated_end in (([0, 1, 128, 129], 256), ([0] * 4, 0)): + ends.copy_(torch.tensor(bounds, device="cuda", dtype=torch.int32)) + output.fill_(91) + graph.replay() + torch.accelerator.synchronize() + assert torch.all(output[allocated_end:] == 91) + if allocated_end: + assert torch.all(output[[0, 128]] != 91) + + def run_single_case(m, n, k, topk, num_experts, block_size): """Run one (M,N,K) configuration on a single GPU and assert DeepGEMM == Triton baseline within tolerance. diff --git a/tests/kernels/moe/test_gpt_oss_triton_kernels.py b/tests/kernels/moe/test_gpt_oss_triton_kernels.py index c1d83d8349..406e525bdf 100644 --- a/tests/kernels/moe/test_gpt_oss_triton_kernels.py +++ b/tests/kernels/moe/test_gpt_oss_triton_kernels.py @@ -32,8 +32,11 @@ from triton_kernels.testing import assert_close from vllm.model_executor.layers.fused_moe.config import mxfp4_w4a16_moe_quant_config from vllm.model_executor.layers.fused_moe.experts.gpt_oss_triton_kernels_moe import ( + _sm12x_block_m, triton_kernel_moe_forward, ) +from vllm.model_executor.layers.fused_moe.oracle import mxfp4 as mxfp4_oracle +from vllm.model_executor.layers.fused_moe.oracle.mxfp4 import Mxfp4MoeBackend from vllm.model_executor.layers.quantization.utils.mxfp4_utils import mx_scale_kwargs from vllm.utils.math_utils import round_up @@ -314,9 +317,13 @@ def test_equiv(num_token, a_dtype, w_dtype, tp, workspace_init): pc2, ) = init_compute_data(M, K, N, E, a_dtype, w_dtype, num_warps=8) - if current_platform.is_device_capability_family(100): + if current_platform.is_device_capability_family( + 100 + ) or current_platform.is_device_capability_family(120): + # Mirrors `_swizzle_mxfp4`; SM12x also needs epilogue_subtile=1. constraints = { "is_persistent": True, + "epilogue_subtile": 1, } opt_flags.update_opt_flags_constraints(constraints) @@ -356,6 +363,38 @@ def test_equiv(num_token, a_dtype, w_dtype, tp, workspace_init): assert_close(ref=out_ref, tri=out_triton_monolithic, maxtol=0.025, rmstol=0.005) +@pytest.mark.parametrize( + ("num_rows", "num_experts", "expected"), + [ + (4, 32, 16), # bs=1 decode: below the 16-row floor + (32 * 20, 32, 32), # 20 tokens/expert rounds up to 32 + (32 * 33, 32, 32), # heuristic would pick 64; capped for 99KB smem + (12000, 32, 32), # prefill: heuristic would pick 128 + ], +) +def test_sm12x_block_m(num_rows, num_experts, expected): + assert _sm12x_block_m(num_rows, num_experts) == expected + + +@pytest.mark.parametrize("sm12x", [True, False]) +def test_gpt_oss_priority_prefers_marlin_on_sm12x( + monkeypatch: pytest.MonkeyPatch, sm12x: bool +): + """SM12x runs the OAI Triton kernels at small tiles only, so MARLIN must + stay the default there while TRITON remains selectable explicitly.""" + monkeypatch.setattr(mxfp4_oracle.current_platform, "is_cuda", lambda: True) + monkeypatch.setattr( + mxfp4_oracle.current_platform, + "is_device_capability_family", + lambda family: sm12x and family == 120, + ) + order = mxfp4_oracle._get_priority_backends_for_gpt_oss() + triton_first = order.index(Mxfp4MoeBackend.TRITON) < order.index( + Mxfp4MoeBackend.MARLIN + ) + assert triton_first is not sm12x + + def test_unit_shuffle(): N = ModelConfig.intermediate_size K = ModelConfig.hidden_size diff --git a/tests/kernels/moe/test_modular_kernel_combinations.py b/tests/kernels/moe/test_modular_kernel_combinations.py index 4b769931ea..f29c220852 100644 --- a/tests/kernels/moe/test_modular_kernel_combinations.py +++ b/tests/kernels/moe/test_modular_kernel_combinations.py @@ -17,6 +17,10 @@ from vllm._aiter_ops import rocm_aiter_ops from vllm.config import VllmConfig, set_current_vllm_config from vllm.model_executor.layers.fused_moe.activation import MoEActivation from vllm.model_executor.layers.fused_moe.experts.rocm_aiter_moe import AiterExperts +from vllm.model_executor.layers.quantization.utils.quant_utils import ( + kFp8DynamicTensorSym, + kFp8StaticTensorSym, +) from vllm.platforms import current_platform from vllm.utils.flashinfer import has_flashinfer_cutlass_fused_moe from vllm.utils.import_utils import has_aiter, has_deep_ep, has_deep_gemm @@ -988,6 +992,173 @@ def test_aiter_moe_padding_matrix_mxfp4(): padding matrix once MI350/gfx950 hardware is available in CI.""" +# --- 4c: HIP-graph token padding -- inf/nan garbage rows (topk_ids=-1) ------ + +# AiterExperts isolates garbage rows from real rows under both AITER sorting +# backends, and reference_moe_impl's dispatch (`mask = topk_ids == i`) is +# likewise row-local, so comparing real rows against it is valid. Exception: +# AiterExperts's a1 quant scale is normally fixed before garbage injection +# (computed in RankTensors.make()), but "fp8_tensor_token" computes it live +# from the (already-injected) hidden_states -- see the skip below. + +# {mode name -> {row index: fill value}}. Covers multiple garbage-row counts +# and positions, and both inf and nan. +_TOKEN_PADDING_GARBAGE_MODES: dict[str, dict[int, float]] = { + "single_inf": {3: float("inf")}, + "single_nan": {5: float("nan")}, + "multiple_mixed": { + 1: float("inf"), + 4: float("-inf"), + 7: float("nan"), + 12: float("inf"), + }, +} + + +def _apply_token_padding_garbage_rows( + rank_tensors: RankTensors, garbage_rows: dict[int, float] +) -> None: + """Simulate the HIP-graph padding-token contract in-place: garbage rows + get inf/nan hidden_states and their routing masked to topk_ids=-1 / + topk_weights=0, mirroring grouped_topk_router.py's `is_padding` masking. + """ + for row, fill in garbage_rows.items(): + rank_tensors.hidden_states[row].fill_(fill) + rank_tensors.topk_ids[row].fill_(-1) + rank_tensors.topk_weights[row].fill_(0.0) + + +def _aiter_token_padding_worker( + pgi: ProcessGroupInfo, + vllm_config: VllmConfig, + cpu_group, + config: Config, + weights: WeightTensors, + verbose: bool, + garbage_rows: dict[int, float], +): + device = torch.device(f"cuda:{pgi.local_rank}") + init_workspace_manager(device) + set_random_seed(pgi.rank) + + weights = copy.deepcopy(weights) + weights.to_current_device() + + rank_tensors = RankTensors.make(config, pgi) + _apply_token_padding_garbage_rows(rank_tensors, garbage_rows) + + with set_current_vllm_config(vllm_config): + mk_out = run_modular_kernel(pgi, vllm_config, config, weights, rank_tensors) + ref_out = reference_moe_impl(config, weights, rank_tensors) + + real_rows = [i for i in range(config.Ms) if i not in garbage_rows] + + assert torch.isfinite(mk_out[real_rows]).all(), ( + "Garbage (padding) token rows leaked NaN/Inf into real rows' output " + "through AiterExperts." + ) + + # ref_out's magnitude here (~1e-3-1e-2) is below atol=3e-2, so a zeroed-out + # mk_out would still pass the checks below -- guard against that. + ref_scale = ref_out[real_rows].abs().mean() + mk_scale = mk_out[real_rows].abs().mean() + assert mk_scale > 0.5 * ref_scale, ( + f"AiterExperts output magnitude (mean |mk_out|={mk_scale:.6f}) looks " + f"degenerate/zeroed vs. reference (mean |ref_out|={ref_scale:.6f})." + ) + + if config.quant_config is not None: + check_accuracy( + ref_out[real_rows], mk_out[real_rows], atol=3e-2, rtol=3e-2, percent=0.9 + ) + else: + torch.testing.assert_close( + ref_out[real_rows], mk_out[real_rows], atol=3e-2, rtol=3e-2 + ) + + +@require_aiter_moe +@pytest.mark.parametrize("quant_config", _PADDING_QUANT_CONFIGS, ids=_PADDING_QUANT_IDS) +@pytest.mark.parametrize("garbage_mode", list(_TOKEN_PADDING_GARBAGE_MODES)) +def test_aiter_moe_token_padding_garbage_rows( + garbage_mode: str, quant_config: TestMoEQuantConfig | None +): + """See https://github.com/vllm-project/vllm/issues/54966 ("Test padding"). + + Simulates HIP-graph token padding: a subset of rows get inf/nan + hidden_states with topk_ids=-1 / topk_weights=0, mirroring the router's + padding-token masking contract (grouped_topk_router.py). Verifies the + real (non-garbage) rows' output through AiterExperts remains finite and + matches reference_moe_impl, across multiple garbage-row counts/positions + and both inf and nan fill values, for unquantized + the 4 fp8 quant + schemes AiterExperts supports. + """ + from vllm.model_executor.layers.fused_moe.prepare_finalize import ( + MoEPrepareAndFinalizeNoDPEPModular, + ) + + garbage_rows = _TOKEN_PADDING_GARBAGE_MODES[garbage_mode] + + # Reuses 4b's already-confirmed 128-aligned ("padded") K/N -- these are + # known to work across all 5 quant schemes here, including AITER's + # block-quant kernel (which 4b's _PADDING_K_UNPADDED/_PADDING_N_UNPADDED + # sizes do not support). Token-padding safety doesn't depend on + # hidden_pad/intermediate_pad, so there's no need for a separate shape. + config = Config( + Ms=_PADDING_M, + K=_PADDING_K_PADDED, + N=_PADDING_N_PADDED, + E=_PADDING_E, + topks=_PADDING_TOPK, + dtype=torch.bfloat16, + quant_config=quant_config, + prepare_finalize_type=MoEPrepareAndFinalizeNoDPEPModular, + fused_experts_type=AiterExperts, + world_size=1, + ) + assert config.is_valid()[0] + assert config.fe_supports_quant_scheme(), ( + f"AiterExperts does not support quant scheme {quant_config}." + ) + + # This (weight, activation) pair is AiterExperts's real per-tensor + # dynamic-activation-quant scheme -- its batch-wide max-abs scale is + # poisoned by a real inf garbage row. Upstream bug: + # https://github.com/ROCm/aiter/issues/6275. + if ( + quant_config is not None + and config.fp8_quant_key_pair() == (kFp8StaticTensorSym, kFp8DynamicTensorSym) + and any(v in (float("inf"), float("-inf")) for v in garbage_rows.values()) + ): + pytest.skip( + "Batch-wide dynamic per-tensor quant scale is poisoned by a " + "real inf garbage row. See " + "https://github.com/ROCm/aiter/issues/6275." + ) + + weights = WeightTensors.make(config) + vllm_config, env_dict = config.make_env_data() + + parallel_launch_with_config( + config.world_size, + _aiter_token_padding_worker, + vllm_config, + env_dict, + config, + weights, + False, + garbage_rows, + ) + + +@pytest.mark.skip( + reason="MXFP4 AiterExperts token padding requires MI350/gfx950 hardware. " + 'See https://github.com/vllm-project/vllm/issues/54966 ("Test padding").' +) +def test_aiter_moe_token_padding_garbage_rows_mxfp4(): + pass + + if __name__ == "__main__": # Ability to test individual PrepareAndFinalize and FusedExperts combination from .modular_kernel_tools.cli_args import make_config, make_config_arg_parser diff --git a/tests/kernels/moe/test_moe.py b/tests/kernels/moe/test_moe.py index 8d6deac4f0..0f786deb54 100644 --- a/tests/kernels/moe/test_moe.py +++ b/tests/kernels/moe/test_moe.py @@ -1437,6 +1437,7 @@ def test_humming_global_valid_shape_m( def test_humming_permute_scratch_is_shared_by_config( monkeypatch: pytest.MonkeyPatch, ): + """Layers share one scratch whose key is independent of batch shape.""" from types import SimpleNamespace from unittest.mock import Mock @@ -1447,34 +1448,19 @@ def test_humming_permute_scratch_is_shared_by_config( manager = workspace.WorkspaceManager(torch.device("cpu")) monkeypatch.setattr(workspace, "_manager", manager) - scratch_topk6 = Mock() - scratch_topk1 = Mock() - scratch_type = Mock(side_effect=[scratch_topk6, scratch_topk1]) + scratch = Mock() + scratch_type = Mock(side_effect=[scratch]) monkeypatch.setattr(humming, "moe_permute_unpermute_supported", lambda: True) monkeypatch.setattr(permute, "MoEPermuteScratch", scratch_type) moe_config = make_dummy_moe_config(max_num_tokens=512, experts_per_token=6) - moe_config.moe_parallel_config.dp_size = 2 experts = SimpleNamespace(moe_config=moe_config) other_layer = SimpleNamespace(moe_config=moe_config) - assert humming.HummingExpertsBase._get_permute_scratch(experts, 6) is scratch_topk6 - assert ( - humming.HummingExpertsBase._get_permute_scratch(other_layer, 6) is scratch_topk6 - ) - assert humming.HummingExpertsBase._get_permute_scratch(experts, 1) is scratch_topk1 - assert humming.HummingExpertsBase._get_permute_scratch(experts, 6) is scratch_topk6 - - assert scratch_type.call_count == 2 - first_call, second_call = scratch_type.call_args_list - assert first_call.kwargs["max_num_tokens"] == 1024 - assert first_call.kwargs["topk"] == 6 - assert second_call.kwargs["max_num_tokens"] == 6144 - assert second_call.kwargs["topk"] == 1 - + assert humming.HummingExpertsBase._get_permute_scratch(experts) is scratch + assert humming.HummingExpertsBase._get_permute_scratch(other_layer) is scratch manager.lock() - assert humming.HummingExpertsBase._get_permute_scratch(experts, 6) is scratch_topk6 - with pytest.raises(AssertionError, match="was not allocated during warmup"): - humming.HummingExpertsBase._get_permute_scratch(experts, 2) + assert humming.HummingExpertsBase._get_permute_scratch(experts) is scratch + assert scratch_type.call_count == 1 def test_humming_delegates_to_instance_activation(): @@ -1531,7 +1517,7 @@ def test_humming_grouped_apply_forwards_valid_prefix( num_experts=2, estimate_local_valid_shape_m=lambda _: 6, prepare_buffers=lambda *_: buffers, - _get_permute_scratch=lambda _, *, indices_only=False: object(), + _get_permute_scratch=lambda: object(), process_input=Mock( side_effect=lambda _, **kwargs: ( kwargs["inputs"], diff --git a/tests/kernels/moe/test_moe_permute_unpermute.py b/tests/kernels/moe/test_moe_permute_unpermute.py index 76f4b57cf0..0ab5760c35 100644 --- a/tests/kernels/moe/test_moe_permute_unpermute.py +++ b/tests/kernels/moe/test_moe_permute_unpermute.py @@ -169,13 +169,9 @@ def test_moe_permute_unpermute( scratch = None if use_scratch: scratch = get_moe_permute_scratch( - max_num_tokens=n_token, - topk=topk, num_experts=n_expert, num_local_experts=n_local_expert, device=hidden_states.device, - hidden_size=n_hidden, - hidden_dtype=dtype, ) ( @@ -247,13 +243,9 @@ def test_moe_permute_reuses_scratch_buffers( _, topk_ids, _ = fused_topk(hidden_states, gating_output, topk, False) scratch_config = dict( - max_num_tokens=n_token, - topk=topk, num_experts=n_expert, num_local_experts=n_expert, device=hidden_states.device, - hidden_size=n_hidden, - hidden_dtype=hidden_states.dtype, ) scratch = get_moe_permute_scratch(**scratch_config) @@ -300,15 +292,10 @@ def test_moe_permute_reuses_scratch_buffers( permuted_idx_2, ) = second - torch.testing.assert_close(permuted_hidden_states_1, permuted_hidden_states_2) torch.testing.assert_close(expert_first_token_offset_1, expert_first_token_offset_2) torch.testing.assert_close(inv_permuted_idx_1, inv_permuted_idx_2) torch.testing.assert_close(permuted_idx_1, permuted_idx_2) - assert ( - permuted_hidden_states_1.untyped_storage().data_ptr() - == permuted_hidden_states_2.untyped_storage().data_ptr() - ) assert ( expert_first_token_offset_1.untyped_storage().data_ptr() == expert_first_token_offset_2.untyped_storage().data_ptr() @@ -330,14 +317,11 @@ def test_moe_permute_scratch_reused_across_graph_sizes(workspace_init) -> None: device = torch.device("cuda:0") scratch = get_moe_permute_scratch( - max_num_tokens=128, - topk=6, num_experts=16, num_local_experts=16, device=device, - hidden_size=2048, - hidden_dtype=torch.bfloat16, ) + scratch.reserve(128 * 6) current_workspace_manager().lock() runs = [] for n_token in (128, 1, 33): @@ -372,13 +356,9 @@ def test_moe_permute_scratch_isolated_across_execution_slots(monkeypatch) -> Non ubatch = 0 monkeypatch.setattr(workspace, "dbo_current_ubatch_id", lambda: ubatch) scratch_config = dict( - max_num_tokens=64, - topk=6, num_experts=16, num_local_experts=16, device=device, - hidden_size=2048, - hidden_dtype=torch.bfloat16, ) runs = [] for i, n_token in enumerate((1, 7, 33, 64)): @@ -396,7 +376,7 @@ def test_moe_permute_scratch_isolated_across_execution_slots(monkeypatch) -> Non runs.append((stream, graph, hidden, topk_ids, output, scratch)) torch.cuda.current_stream().wait_stream(stream) - assert len({run[-1].permuted_hidden_states.data_ptr() for run in runs}) == 4 + assert len({run[-1].permuted_idx.data_ptr() for run in runs}) == 4 manager.lock() for _ in range(3): for i, (stream, graph, hidden, topk_ids, _, scratch) in enumerate(runs): @@ -423,8 +403,6 @@ def test_moe_permute_scratch_without_manager(monkeypatch) -> None: monkeypatch.setattr(workspace, "_manager", None) config = dict( - max_num_tokens=4, - topk=2, num_experts=4, num_local_experts=4, device=torch.device("cuda"), @@ -432,6 +410,8 @@ def test_moe_permute_scratch_without_manager(monkeypatch) -> None: first = get_moe_permute_scratch(**config) second = get_moe_permute_scratch(**config) + first.reserve(8) + second.reserve(8) assert ( first.token_expert_indices.data_ptr() != second.token_expert_indices.data_ptr() ) @@ -450,13 +430,9 @@ def test_moe_permute_ignores_invalid_expert_ids_with_scratch() -> None: topk_ids = torch.tensor([[0], [-1], [1], [4], [2]], device="cuda") expert_map = torch.tensor([0, 1, -1, -1], dtype=torch.int32, device="cuda") scratch = MoEPermuteScratch( - max_num_tokens=5, - topk=1, num_experts=4, num_local_experts=2, device=hidden_states.device, - hidden_size=16, - hidden_dtype=hidden_states.dtype, ) permuted, _, expert_offsets, inverse, _ = moe_permute( @@ -498,3 +474,39 @@ def test_moe_permute_ignores_invalid_expert_ids_with_scratch() -> None: expected_offsets = torch.tensor([0, 5, 5], dtype=torch.int64, device="cuda") torch.testing.assert_close(indices.flatten(), expected_inverse) torch.testing.assert_close(expert_offsets, expected_offsets) + + +def test_moe_permute_scratch_grows_until_workspace_lock(workspace_init) -> None: + """Capacity follows the inputs seen in warmup, whatever their topk or size.""" + if not moe_permute_unpermute_supported(): + pytest.skip("moe_permute_unpermute is not supported on this platform.") + + device = torch.device("cuda:0") + scratch = get_moe_permute_scratch( + num_experts=16, num_local_experts=16, device=device + ) + + def run(n_token: int, topk: int, dtype: torch.dtype) -> None: + hidden = torch.randn((n_token, 64), device=device).to(dtype) + topk_ids = torch.randint(16, (n_token, topk), device=device) + actual = moe_permute(hidden, None, topk_ids, 16, scratch=scratch) + expected = moe_permute(hidden, None, topk_ids, 16) + for a, e in zip(actual, expected): + if a is not None: + torch.testing.assert_close(a.float(), e.float()) + + run(8, 6, torch.bfloat16) + run(64, 6, torch.bfloat16) + run(384, 1, torch.float8_e4m3fn) + assert scratch.max_expanded_rows == 384 + + current_workspace_manager().lock() + run(64, 6, torch.bfloat16) + with pytest.raises(AssertionError, match="Scratch growth is not allowed"): + run(65, 6, torch.bfloat16) + + # Experts rebuilt after the lock (e.g. a weight reload) keep the capacity. + rebuilt = get_moe_permute_scratch( + num_experts=16, num_local_experts=16, device=device + ) + assert rebuilt.max_expanded_rows == 384 diff --git a/tests/kernels/moe/test_moe_weight_loading_padded.py b/tests/kernels/moe/test_moe_weight_loading_padded.py index f7da37dc92..2ec1aaded7 100644 --- a/tests/kernels/moe/test_moe_weight_loading_padded.py +++ b/tests/kernels/moe/test_moe_weight_loading_padded.py @@ -9,6 +9,8 @@ have the original unpadded size. These tests verify that weight loading correctly handles this mismatch. """ +from types import SimpleNamespace + import pytest import torch @@ -564,9 +566,11 @@ class TestLoadWeightsExpertMapping: transposed=False, quant_method="tensor", layer_name="model.layers.0.mlp.experts", + is_gated=True, ): experts = torch.nn.Module() experts.layer_name = layer_name + experts.moe_config = SimpleNamespace(is_act_and_mul=is_gated) experts.get_expert_mapping = lambda **_: mapping experts.is_fused_checkpoint_transposed = transposed experts._orient_fused_weight = RoutedExperts._orient_fused_weight @@ -593,7 +597,7 @@ class TestLoadWeightsExpertMapping: (("gate_proj", "down_proj", "up_proj"), ""), (("w1", "w2", "w3"), ""), (("up_proj", "down_proj", "up_proj"), ""), - (("up_proj", "down_proj", ""), ""), + (("up_proj", "down_proj", None), ""), (("gate_proj", "down_proj", "up_proj"), "base_layer."), ], ) @@ -609,7 +613,10 @@ class TestLoadWeightsExpertMapping: self, projs, lora_prefix, suffix, shape, dtype ): mapping = self._mapping( - projs, num_redundant_experts=4, lora_base_layer_prefix=lora_prefix + projs, + num_redundant_experts=4, + lora_base_layer_prefix=lora_prefix, + is_gated=projs[2] is not None, ) for logical_id, physical_ids in [(1, (1, 17)), (10, (10,))]: for proj in dict.fromkeys(projs): @@ -651,6 +658,47 @@ class TestLoadWeightsExpertMapping: for call, expected in zip(calls, weight.chunk(2) * 2): torch.testing.assert_close(call[3], expected, rtol=0, atol=0) + def test_non_gated_mapping_uses_stacked_up_proj_without_warning(self, caplog_vllm): + mapping = self._mapping(("up_proj", "down_proj", None), is_gated=False) + assert "Unexpected gate/up projection names" not in caplog_vllm.text + assert mapping[:2] == [ + ("experts.w13_weight", "experts.up_proj", 0, "w1"), + ("experts.w2_weight", "experts.down_proj", 0, "w2"), + ] + assert mapping[2:] == [ + (param, f"experts.{expert}.{proj}.", expert, shard) + for expert in range(16) + for param, proj, shard in ( + ("experts.w13_", "up_proj", "w1"), + ("experts.w2_", "down_proj", "w2"), + ) + ] + + def test_gated_mapping_with_unknown_names_warns_and_skips_fused(self, caplog_vllm): + mapping = self._mapping(("fc1", "fc2", "fc3")) + assert "Unexpected gate/up projection names: fc1, fc3" in caplog_vllm.text + assert [name for _, name, _, _ in mapping[:3]] == [ + "experts.0.fc1.", + "experts.0.fc2.", + "experts.0.fc3.", + ] + + def test_stacked_non_gated_tensors_load_unsplit(self): + mapping = self._mapping(("up_proj", "down_proj", None), is_gated=False) + up_proj = torch.arange(16 * 4 * 3).reshape(16, 4, 3) + down_proj = torch.arange(16 * 3 * 4).reshape(16, 3, 4) + calls = self._load( + mapping, + [("up_proj", up_proj), ("down_proj", down_proj)], + is_gated=False, + ) + assert [call[:3] for call in calls] == [ + *(("w13_weight", "w1", expert) for expert in range(16)), + *(("w2_weight", "w2", expert) for expert in range(16)), + ] + for call, expected in zip(calls, [*up_proj, *down_proj]): + torch.testing.assert_close(call[3], expected, rtol=0, atol=0) + @pytest.mark.parametrize( "suffix,quant_method,transposed", [ diff --git a/tests/kernels/moe/test_ocp_mx_moe.py b/tests/kernels/moe/test_ocp_mx_moe.py index 5a7053a8b1..f20927565e 100644 --- a/tests/kernels/moe/test_ocp_mx_moe.py +++ b/tests/kernels/moe/test_ocp_mx_moe.py @@ -7,7 +7,11 @@ from dataclasses import dataclass, replace import pytest import torch -from tests.kernels.moe.utils import check_accuracy, check_deferred_moe_finalize +from tests.kernels.moe.utils import ( + check_accuracy, + check_deferred_moe_finalize, + make_dummy_moe_config, +) from vllm._aiter_ops import ( is_aiter_found, is_aiter_found_and_supported, @@ -2009,6 +2013,52 @@ def test_select_mxfp4_moe_backend_raises_with_unsupported_reasons( mxfp4_oracle.select_mxfp4_moe_backend(moe_config) +@pytest.mark.parametrize( + "moe_backend,w4a4,supported,expected", + [ + # The checkpoint's activation wins over BF16 backends listed first. + ("auto", True, {"CUTLASS_MXFP4_MXFP4", "MARLIN"}, "CUTLASS_MXFP4_MXFP4"), + # No W4A4 kernel: run at BF16 rather than another quantized activation. + ("auto", True, {"FLASHINFER_TRTLLM_MXFP4_MXFP8", "MARLIN"}, "MARLIN"), + ("marlin", True, {"MARLIN"}, "MARLIN"), + # A BF16 checkpoint never gets its activations quantized. + ("auto", False, {"CUTLASS_MXFP4_MXFP4", "MARLIN"}, "MARLIN"), + ], +) +def test_select_mxfp4_moe_backend_respects_checkpoint_activation( + monkeypatch: pytest.MonkeyPatch, + moe_backend: str, + w4a4: bool, + supported: set[str], + expected: str, +): + """Activations run at the checkpoint's precision when a backend supports + it and fall back to BF16 otherwise, never to a lower precision.""" + import vllm.model_executor.layers.fused_moe.oracle.mxfp4 as mxfp4_oracle + from vllm.model_executor.layers.quantization.utils.quant_utils import ( + kMxfp4Dynamic, + ) + + def backend_to_kernel_cls(backend): + class Experts: + @staticmethod + def is_supported_config(*args): + return backend.name in supported, None + + return [Experts] + + monkeypatch.setattr(mxfp4_oracle, "backend_to_kernel_cls", backend_to_kernel_cls) + monkeypatch.setattr(mxfp4_oracle, "_user_moe_activation_override", lambda: None) + moe_config = make_dummy_moe_config() + moe_config.moe_backend = moe_backend + + backend, _ = mxfp4_oracle.select_mxfp4_moe_backend( + moe_config, activation_key=kMxfp4Dynamic if w4a4 else None + ) + + assert backend.name == expected + + @pytest.mark.parametrize("backend_name", ["AITER_TRITON_MXFP4_BF16", "AITER_MXFP4_FP8"]) @pytest.mark.parametrize("use_ep", [False, True]) def test_aiter_mxfp4_monolithic_rejects_expert_parallel( diff --git a/tests/kernels/moe/test_silu_mul_per_token_group_quant_fp8_colmajor.py b/tests/kernels/moe/test_silu_mul_per_token_group_quant_fp8_colmajor.py index 204954d1fe..b7dcb956c8 100644 --- a/tests/kernels/moe/test_silu_mul_per_token_group_quant_fp8_colmajor.py +++ b/tests/kernels/moe/test_silu_mul_per_token_group_quant_fp8_colmajor.py @@ -7,6 +7,7 @@ import torch from vllm.model_executor.layers.quantization.utils.fp8_utils import ( _per_token_group_quant_fp8_colmajor, silu_mul_per_token_group_quant_fp8_colmajor, + silu_mul_quant_fp8_packed_triton, ) from vllm.platforms import current_platform from vllm.triton_utils import triton @@ -108,6 +109,94 @@ def test_silu_mul_fp8_quant_deep_gemm(T: int, N: int): torch.testing.assert_close(output_scales, ref_output_scales) +@pytest.mark.parametrize( + "counts", [[0, 1, 127, 128, 129, 0, 257, 13], [128] * 8, [0] * 8] +) +@pytest.mark.parametrize("group_size", [32, 128]) +@pytest.mark.parametrize("scale_format", ["packed", "float32", "ceil_ue8m0"]) +@pytest.mark.parametrize("hidden_size", [256, 2048]) +@pytest.mark.parametrize("clamp_limit", [None, 10.0]) +@pytest.mark.skipif(not current_platform.is_cuda(), reason="CUDA graph test") +def test_silu_quant_skips_expert_padding_on_graph_replay( + counts: list[int], + group_size: int, + scale_format: str, + hidden_size: int, + clamp_limit: float | None, +): + """Live rows match dense quantization; gaps stay untouched as counts change.""" + alignment = 128 + capacity = sum(triton.cdiv(n, alignment) * alignment for n in counts) + # Exercise unused capacity as well as the gaps between experts. + capacity = max(capacity, alignment) + if counts != [128] * 8: + capacity += alignment + x = torch.randn((capacity, 2 * hidden_size), device=DEVICE, dtype=torch.bfloat16) + x *= 16 + clean_x = x.clone() + ends = torch.empty(len(counts), device=DEVICE, dtype=torch.int32) + output = torch.empty((capacity, hidden_size), device=DEVICE, dtype=FLOAT8_DTYPE) + + def set_counts(values): + positions = [] + mask = torch.zeros(capacity, device=DEVICE, dtype=torch.bool) + start = 0 + for count in values: + mask[start : start + count] = True + positions.append(start + count) + start += triton.cdiv(count, alignment) * alignment + ends.copy_(torch.tensor(positions, device=DEVICE, dtype=torch.int32)) + return mask + + def quantize(x, output=None, **kwargs): + if scale_format == "packed": + return silu_mul_quant_fp8_packed_triton( + x, + group_size=group_size, + output_q=output, + clamp_limit=clamp_limit, + **kwargs, + ) + return silu_mul_per_token_group_quant_fp8_colmajor( + x, + group_size=group_size, + output=output, + clamp_limit=clamp_limit, + use_ue8m0=scale_format == "ceil_ue8m0", + **kwargs, + ) + + def invoke(): + return quantize( + x, + output=output, + expert_ends=ends, + expert_alignment=alignment, + ) + + set_counts(counts) + invoke() + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + actual_q, actual_s = invoke() + + for values in (counts, counts[::-1]): + live = set_counts(values) + x.copy_(clean_x) + x[~live] = float("nan") + output.view(torch.uint8).fill_(91) + graph.replay() + reference_q, reference_s = quantize(x) + torch.testing.assert_close( + actual_q.view(torch.uint8)[live], + reference_q.view(torch.uint8)[live], + rtol=0, + atol=0, + ) + torch.testing.assert_close(actual_s[live], reference_s[live], rtol=0, atol=0) + assert torch.all(actual_q.view(torch.uint8)[~live] == 91) + + @pytest.mark.parametrize("T", [128, 256, 512]) @pytest.mark.parametrize("N", [128 * 2, 256 * 2, 768 * 2, 2048 * 2, 7168 * 2]) @pytest.mark.parametrize("clamp_limit", [7.0, 10.0]) diff --git a/tests/kernels/moe/test_trtllm_nvfp4_moe.py b/tests/kernels/moe/test_trtllm_nvfp4_moe.py index 292e4549bb..4ee0955051 100644 --- a/tests/kernels/moe/test_trtllm_nvfp4_moe.py +++ b/tests/kernels/moe/test_trtllm_nvfp4_moe.py @@ -16,8 +16,7 @@ import torch import vllm.model_executor.layers.fused_moe.modular_kernel as mk from tests.kernels.moe.utils import check_deferred_moe_finalize, make_test_quant_config from tests.kernels.quantization.nvfp4_utils import ( - FLOAT4_E2M1_MAX, - FLOAT8_E4M3_MAX, + convert_swizzled_to_linear, dequantize_nvfp4_to_dtype, ) from tests.kernels.utils import torch_moe @@ -39,6 +38,13 @@ from vllm.model_executor.layers.fused_moe.config import ( from vllm.model_executor.layers.fused_moe.experts.trtllm_nvfp4_moe import ( TrtLlmNvFp4ExpertsModular, ) +from vllm.model_executor.layers.quantization.utils.flashinfer_fp4_moe import ( + prepare_static_weights_for_trtllm_fp4_moe, + reorder_w1w3_to_w3w1, +) +from vllm.model_executor.layers.quantization.utils.flashinfer_utils import ( + NVFP4_PER_TOKEN_BASE_GLOBAL_SCALE, +) from vllm.platforms import current_platform from vllm.utils.flashinfer import has_flashinfer_trtllm_fused_moe from vllm.utils.math_utils import next_power_of_2 @@ -96,10 +102,14 @@ SITU = op_registry[_SITU_OP_NAME] def _make_trtllm_fp4_moe_kernel( - moe_config: FusedMoEConfig, quant_config: FusedMoEQuantConfig + moe_config: FusedMoEConfig, + quant_config: FusedMoEQuantConfig, + per_token_activation: bool = False, ) -> mk.FusedMoEKernel: experts = TrtLlmNvFp4ExpertsModular( - moe_config=moe_config, quant_config=quant_config + moe_config=moe_config, + quant_config=quant_config, + per_token_activation=per_token_activation, ) # Mimic the production weight-loader path so per-expert tensors that # are normally precomputed in process_weights_after_loading (g1_scale_c @@ -126,16 +136,37 @@ def _make_trtllm_fp4_moe_kernel( ACTIVATION_CASES = [ - pytest.param(MoEActivation.SILU, MoEActivation.SILU, None, id="silu"), - pytest.param(MoEActivation.SILU, SILU_WITH_CLAMP, _SWIGLU_LIMIT, id="silu_clamp"), - pytest.param(MoEActivation.SITU, SITU, None, id="situ"), + pytest.param(MoEActivation.SILU, MoEActivation.SILU, None, False, id="silu"), + pytest.param( + MoEActivation.SILU, + MoEActivation.SILU, + None, + True, + id="silu_per_token", + ), + pytest.param( + MoEActivation.SILU, + SILU_WITH_CLAMP, + _SWIGLU_LIMIT, + False, + id="silu_clamp", + ), + pytest.param(MoEActivation.SITU, SITU, None, False, id="situ"), pytest.param( MoEActivation.RELU2_NO_MUL, MoEActivation.RELU2_NO_MUL, None, + False, id="relu2_no_mul", ), - pytest.param(MoEActivation.GELU, MoEActivation.GELU, None, id="gelu"), + pytest.param( + MoEActivation.RELU2_NO_MUL, + MoEActivation.RELU2_NO_MUL, + None, + True, + id="relu2_no_mul_per_token", + ), + pytest.param(MoEActivation.GELU, MoEActivation.GELU, None, False, id="gelu"), ] @@ -143,7 +174,10 @@ ACTIVATION_CASES = [ @pytest.mark.parametrize("e", [128]) @pytest.mark.parametrize("topk", [8]) @pytest.mark.parametrize("dtype", [torch.bfloat16]) -@pytest.mark.parametrize("activation,torch_activation,swiglu_limit", ACTIVATION_CASES) +@pytest.mark.parametrize( + "activation,torch_activation,swiglu_limit,per_token_activation", + ACTIVATION_CASES, +) @torch.inference_mode() def test_trtllm_fp4_moe_no_graph( m: int, @@ -155,6 +189,7 @@ def test_trtllm_fp4_moe_no_graph( activation: MoEActivation, torch_activation: MoEActivation | type[SiluAndMulWithClamp] | type[SituAndMul], swiglu_limit: float | None, + per_token_activation: bool, workspace_init, ): # FlashInfer's trtllm_batched_gemm_runner has no precompiled tile @@ -202,6 +237,42 @@ def test_trtllm_fp4_moe_no_graph( # clamp/output-scale coupling in the FlashInfer kernel wrapper. quant_config.g1_alphas.fill_(_LARGE_OUTPUT1_SCALE) + w1_q_linear = w1_q + w2_q_linear = w2_q + w1_scale_reference = quant_config.w1_scale + w2_scale_reference = quant_config.w2_scale + assert w1_scale_reference is not None + assert w2_scale_reference is not None + w1_scale_linear = torch.stack( + [ + convert_swizzled_to_linear(scale, w1_q.size(1), k, quant_blocksize) + for scale in w1_scale_reference + ] + ) + w2_scale_linear = torch.stack( + [ + convert_swizzled_to_linear(scale, w2_q.size(1), n, quant_blocksize) + for scale in w2_scale_reference + ] + ) + w1_q_for_kernel = w1_q + if is_gated_act: + w1_q_for_kernel, w1_scale_linear = reorder_w1w3_to_w3w1( + w1_q.clone(), w1_scale_linear + ) + w1_q, w1_scale, w2_q, w2_scale = prepare_static_weights_for_trtllm_fp4_moe( + w1_q_for_kernel, + w2_q, + w1_scale_linear, + w2_scale_linear, + hidden_size=k, + intermediate_size=n, + num_experts=e, + is_gated_activation=is_gated_act, + ) + quant_config._w1.scale = w1_scale + quant_config._w2.scale = w2_scale + score = torch.randn((m, e), device="cuda", dtype=dtype) topk_weights, topk_ids, _ = fused_topk(a, score, topk, renormalize=False) @@ -224,7 +295,11 @@ def test_trtllm_fp4_moe_no_graph( ), ) - trtllm_experts = _make_trtllm_fp4_moe_kernel(moe_config, quant_config) + trtllm_experts = _make_trtllm_fp4_moe_kernel( + moe_config, + quant_config, + per_token_activation=per_token_activation, + ) if activation == MoEActivation.SITU: torch.testing.assert_close( trtllm_experts.fused_experts.g1_scale_c, quant_config.a2_gscale @@ -245,10 +320,20 @@ def test_trtllm_fp4_moe_no_graph( # Reference: round-trip activations and weights through FP4 # quant/dequant so the comparison isolates kernel/activation behavior # from quantization error. - a_global_scale = ((FLOAT8_E4M3_MAX * FLOAT4_E2M1_MAX) / a.abs().max()).to( - torch.float32 - ) - a_fp4, a_scale_interleaved = ops.scaled_fp4_quant(a, a_global_scale) + if per_token_activation: + from flashinfer import SfLayout, nvfp4_quantize + + a_fp4, a_scale_interleaved, per_token_scale = nvfp4_quantize( + a, + NVFP4_PER_TOKEN_BASE_GLOBAL_SCALE, + sfLayout=SfLayout.layout_linear, + per_token_activation=True, + ) + a_global_scale = 1.0 / per_token_scale.unsqueeze(1) + else: + assert quant_config.a1_gscale is not None + a_global_scale = quant_config.a1_gscale[0] + a_fp4, a_scale_interleaved = ops.scaled_fp4_quant(a, a_global_scale) a_in_dtype = dequantize_nvfp4_to_dtype( a_fp4, a_scale_interleaved, @@ -256,6 +341,7 @@ def test_trtllm_fp4_moe_no_graph( dtype=a.dtype, device=a.device, block_size=quant_blocksize, + is_sf_linear_layout=per_token_activation, ) w1_d = torch.empty( @@ -264,16 +350,16 @@ def test_trtllm_fp4_moe_no_graph( w2_d = torch.empty((e, k, n), device="cuda", dtype=dtype) for idx in range(e): w1_d[idx] = dequantize_nvfp4_to_dtype( - w1_q[idx], - quant_config.w1_scale[idx], + w1_q_linear[idx], + w1_scale_reference[idx], (1 / quant_config.g1_alphas[idx]), dtype=dtype, device=w1_q.device, block_size=quant_blocksize, ) w2_d[idx] = dequantize_nvfp4_to_dtype( - w2_q[idx], - quant_config.w2_scale[idx], + w2_q_linear[idx], + w2_scale_reference[idx], (1 / quant_config.g2_alphas[idx]), dtype=dtype, device=w2_q.device, @@ -284,6 +370,13 @@ def test_trtllm_fp4_moe_no_graph( a_in_dtype, w1_d, w2_d, score, topk, activation=torch_activation ) + if swiglu_limit is None: + cosine = torch.nn.functional.cosine_similarity( + torch_output.float().flatten(), + trtllm_output.float().flatten(), + dim=0, + ) + assert cosine > 0.99 torch.testing.assert_close(torch_output, trtllm_output, atol=2e-1, rtol=2e-1) @@ -416,5 +509,6 @@ if __name__ == "__main__": MoEActivation.GELU, MoEActivation.GELU, None, + False, None, ) diff --git a/tests/kernels/moe/test_unquantized_backend_selection.py b/tests/kernels/moe/test_unquantized_backend_selection.py index 9d0f4d6900..25e6b7dec6 100644 --- a/tests/kernels/moe/test_unquantized_backend_selection.py +++ b/tests/kernels/moe/test_unquantized_backend_selection.py @@ -446,8 +446,15 @@ def test_select_cuda_deepep_ht_falls_back_from_trtllm( "vllm.model_executor.layers.fused_moe.experts.trtllm_bf16_moe.TrtLlmBf16ExpertsBase._supports_current_device", return_value=True, ) +@pytest.mark.parametrize( + "routing_method", + [RoutingMethodType.Renormalize, RoutingMethodType.MiniMax2], +) +@pytest.mark.parametrize("moe_backend", ["auto", "flashinfer_trtllm"]) def test_select_cuda_flashinfer_trtllm_ag_rs_uses_monolithic( mock_supports_current_device, + routing_method, + moe_backend, ): """Test AG/RS stays on BF16 TRTLLM monolithic when TRTLLM is supported.""" with ( @@ -462,8 +469,8 @@ def test_select_cuda_flashinfer_trtllm_ag_rs_uses_monolithic( ), ): moe_config = make_dummy_moe_config(num_experts=4, num_local_experts=2) - moe_config.moe_backend = "flashinfer_trtllm" - moe_config.routing_method = RoutingMethodType.Renormalize + moe_config.moe_backend = moe_backend + moe_config.routing_method = routing_method moe_config.moe_parallel_config.use_ep = True moe_config.moe_parallel_config.dp_size = 2 moe_config.moe_parallel_config.ep_size = 2 diff --git a/tests/kernels/quantization/test_cutlass_w4a8.py b/tests/kernels/quantization/test_cutlass_w4a8.py index 968d5f2c74..fc34a15bde 100644 --- a/tests/kernels/quantization/test_cutlass_w4a8.py +++ b/tests/kernels/quantization/test_cutlass_w4a8.py @@ -29,8 +29,11 @@ if not current_platform.is_cuda(): # `is_quant_method_supported` conflates kernels with quantization methods # an assumption which is breaking down as quantizations methods can have # have kernels and some kernels support multiple quantization methods. -capability = current_platform.get_device_capability() -IS_SUPPORTED_BY_GPU = capability is not None and capability[0] >= 9 +# CUTLASS W4A8 kernels are only built for SM90 (9.0a); the kernel selector +# requires exactly SM90 too, so match it here rather than accepting >= 90. +IS_SUPPORTED_BY_GPU = ( + current_platform.is_cuda() and current_platform.is_device_capability(90) +) MNK_SHAPES = [ (1, 128, 128), @@ -106,10 +109,6 @@ TEST_TYPES = [ # TODO: in future PR refactor this and `is_quant_method_supported` in the kernel # unit tests to a common utility function. Currently the use of -# `is_quant_method_supported` conflates kernels with quantization methods -# an assumption which is breaking down as quantizations methods can have -# have kernels and some kernels support multiple quantization methods. -IS_SUPPORTED_BY_GPU = current_platform.has_device_capability(90) # For testing quantized linear kernels diff --git a/tests/kernels/quantization/test_cutlass_w4a8_moe.py b/tests/kernels/quantization/test_cutlass_w4a8_moe.py index b8c10d0510..4050314a11 100644 --- a/tests/kernels/quantization/test_cutlass_w4a8_moe.py +++ b/tests/kernels/quantization/test_cutlass_w4a8_moe.py @@ -17,9 +17,8 @@ from vllm.platforms import current_platform from vllm.scalar_type import ScalarType, scalar_types from vllm.utils.torch_utils import set_random_seed -capability = current_platform.get_device_capability() IS_SUPPORTED_BY_GPU = ( - current_platform.is_cuda() and capability is not None and capability[0] >= 9 + current_platform.is_cuda() and current_platform.is_device_capability(90) ) diff --git a/tests/kernels/quantization/test_flashinfer_nvfp4_scaled_mm.py b/tests/kernels/quantization/test_flashinfer_nvfp4_scaled_mm.py index 54dcaa0b97..47c25d046d 100644 --- a/tests/kernels/quantization/test_flashinfer_nvfp4_scaled_mm.py +++ b/tests/kernels/quantization/test_flashinfer_nvfp4_scaled_mm.py @@ -13,11 +13,15 @@ from nvfp4_utils import ( from vllm import _custom_ops as ops from vllm.model_executor.kernels.linear.nvfp4 import NvFp4LinearLayerConfig from vllm.model_executor.kernels.linear.nvfp4.flashinfer import ( + FlashInferCuteDslNvFp4LinearKernel, FlashInferCuteDslNvFp4W4A16LinearKernel, + FlashInferCutlassNvFp4LinearKernel, ) +from vllm.model_executor.layers.fusion.quant_activation import QuantizedActivation from vllm.model_executor.layers.quantization.utils.nvfp4_emulation_utils import ( dequantize_to_dtype, ) +from vllm.model_executor.layers.quantization.utils.quant_utils import kNvfp4Dynamic from vllm.platforms import current_platform from vllm.utils.flashinfer import ( flashinfer_scaled_fp4_mm, @@ -47,6 +51,54 @@ SEEDS = [42] CUDA_DEVICES = ["cuda:0"] +@pytest.mark.parametrize("token_shape", [(17,), (1, 17), (2, 17)]) +@pytest.mark.parametrize( + "kernel_cls", + [FlashInferCutlassNvFp4LinearKernel, FlashInferCuteDslNvFp4LinearKernel], +) +@torch.inference_mode() +def test_flashinfer_nvfp4_preserves_quantized_input_shape( + token_shape: tuple[int, ...], + kernel_cls: type[FlashInferCutlassNvFp4LinearKernel] + | type[FlashInferCuteDslNvFp4LinearKernel], +) -> None: + """Pre-quantized inputs must match the tensor path and retain token dimensions.""" + supported, reason = kernel_cls.is_supported() + if not supported: + pytest.skip(reason) + + set_random_seed(42) + hidden_size, output_size = 128, 128 + x = torch.randn(*token_shape, hidden_size, dtype=torch.bfloat16, device="cuda") + weight = torch.randn(output_size, hidden_size, dtype=x.dtype, device=x.device) + input_scale = (FLOAT8_E4M3_MAX * FLOAT4_E2M1_MAX / x.abs().max()).float() + weight_scale = (FLOAT8_E4M3_MAX * FLOAT4_E2M1_MAX / weight.abs().max()).float() + x_fp4, x_blockscale = ops.scaled_fp4_quant(x.reshape(-1, hidden_size), input_scale) + weight_fp4, weight_blockscale = ops.scaled_fp4_quant(weight, weight_scale) + quantized_input = QuantizedActivation( + data=x_fp4.reshape(*token_shape, hidden_size // 2), + scale=x_blockscale, + orig_dtype=x.dtype, + orig_shape=x.shape, + quant_key=kNvfp4Dynamic, + ) + layer = torch.nn.Module() + layer.input_size_per_partition = hidden_size + layer.output_size_per_partition = output_size + layer.weight = weight_fp4 + layer.weight_scale = weight_blockscale + layer.input_global_scale_inv = input_scale + layer.alpha = (input_scale * weight_scale).reciprocal() + kernel = kernel_cls(NvFp4LinearLayerConfig()) + + expected = kernel.apply_weights(layer, x.reshape(-1, hidden_size)) + actual = kernel.apply_weights(layer, quantized_input) + + assert actual.shape == (*token_shape, output_size) + assert actual.dtype == x.dtype + torch.testing.assert_close(actual.reshape(expected.shape), expected) + + def get_ref_results( a_fp4, b_fp4, diff --git a/tests/kernels/quantization/test_rdna3_moe_w4a16.py b/tests/kernels/quantization/test_rdna3_moe_w4a16.py index 6212e6f08c..58aa90aa07 100644 --- a/tests/kernels/quantization/test_rdna3_moe_w4a16.py +++ b/tests/kernels/quantization/test_rdna3_moe_w4a16.py @@ -18,6 +18,7 @@ Run `pytest tests/kernels/quantization/test_rdna3_moe_w4a16.py`. import pytest import torch +import torch.nn.functional as F from vllm.platforms import current_platform @@ -32,6 +33,9 @@ from vllm.model_executor.layers.fused_moe.activation import ( # noqa: E402 from vllm.model_executor.layers.fused_moe.moe_align_block_size import ( # noqa: E402 moe_align_block_size, ) +from vllm.model_executor.layers.fused_moe.oracle.int_wna16 import ( # noqa: E402 + _process_weights_rdna3, +) from vllm.model_executor.layers.quantization.utils.quant_utils import ( # noqa: E402 pack_quantized_values_into_int32, ) @@ -95,6 +99,16 @@ def _make_qzeros(E, groups, N): return qz.unsqueeze(0).expand(E, -1, -1).contiguous() +def _make_ct_weights(E, N, K, group_size, dtype): + """Compressed-tensors canonical N-first layout: packed ``[E, N, K // 8]`` + and scales ``[E, N, K // group_size]``, plus the dequantized weights.""" + q = torch.randint(0, 16, (E, N, K), dtype=torch.int32, device=device) + s = torch.rand(E, N, K // group_size, dtype=dtype, device=device) * 0.1 + 0.01 + packed = pack_quantized_values_into_int32(q, scalar_types.uint4b8, packed_dim=2) + w = (q - 8).float() * s.float().repeat_interleave(group_size, dim=2) + return packed.contiguous(), s, w + + @gfx1100_only @pytest.mark.parametrize("E, K, N_inter, top_k, group_size", MODEL_CONFIGS) @pytest.mark.parametrize("M", NUM_TOKENS) @@ -359,3 +373,70 @@ def test_expert_id_minus_one(): # Output should remain zero (expert skipped) assert torch.equal(out, torch.zeros_like(out)) + + +@gfx1100_only +@pytest.mark.parametrize("dtype", [torch.float16, torch.bfloat16]) +def test_process_weights_from_compressed_tensors_layout(dtype): + """Weights in the layout compressed-tensors hands to the oracle go through + the RDNA3 post-processing and a full MoE forward, against a dequantized + reference. The tests above build the kernel layout directly, so they do + not notice when the layout upstream of the kernel changes (#52798).""" + E, K, N_inter, top_k, group_size, M = 8, 512, 256, 2, 32, 16 + torch.manual_seed(5) + w13, w13_s, w13_ref = _make_ct_weights(E, 2 * N_inter, K, group_size, dtype) + w2, w2_s, w2_ref = _make_ct_weights(E, K, N_inter, group_size, dtype) + w13, w2, w13_s, w2_s, w13_z, w2_z, *_ = _process_weights_rdna3( + w13, w2, w13_s, w2_s, group_size + ) + assert w13_z is not None and w2_z is not None + + x = torch.randn(M, K, dtype=dtype, device=device) + topk_ids = torch.randint(0, E, (M, top_k), device=device, dtype=torch.int32) + topk_w = torch.softmax(torch.randn(M, top_k, device=device), dim=-1).float() + si, ei, ntp = moe_align_block_size(topk_ids, 1, E) + + w1_out = torch.zeros(M * top_k, 2 * N_inter, dtype=dtype, device=device) + ops.moe_gptq_gemm_rdna3( + x, + w1_out, + w13, + w13_s, + w13_z, + torch.empty(0, device=device), + si, + ei, + ntp, + top_k, + 1, + False, + 0, + ) + act_out = torch.empty(M * top_k, N_inter, dtype=dtype, device=device) + apply_moe_activation(MoEActivation.SILU, act_out, w1_out) + out = torch.zeros(M, K, dtype=dtype, device=device) + ops.moe_gptq_gemm_rdna3( + act_out, + out, + w2, + w2_s, + w2_z, + topk_w.view(-1), + si, + ei, + ntp, + 1, + 1, + True, + top_k, + ) + + ref = torch.zeros(M, K, device=device) + for m in range(M): + for k in range(top_k): + e = int(topk_ids[m, k]) + h = w13_ref[e] @ x[m].float() + a = F.silu(h[:N_inter]) * h[N_inter:] + ref[m] += (w2_ref[e] @ a) * topk_w[m, k] + rel_l2 = (torch.norm(out.float() - ref) / torch.norm(ref)).item() + assert rel_l2 < (0.02 if dtype == torch.float16 else 0.05), rel_l2 diff --git a/tests/kernels/quantization/test_rdna_hybrid_w4a16.py b/tests/kernels/quantization/test_rdna_hybrid_w4a16.py index fe3db8dab3..da7ef2d437 100644 --- a/tests/kernels/quantization/test_rdna_hybrid_w4a16.py +++ b/tests/kernels/quantization/test_rdna_hybrid_w4a16.py @@ -30,6 +30,8 @@ RDNAHybridW4A16LinearKernel = hybrid_module.RDNAHybridW4A16LinearKernel pack_int4_exllama_shuffle = hybrid_module.pack_int4_exllama_shuffle SUPPORTED_GROUP_SIZES = hybrid_module.SUPPORTED_GROUP_SIZES MAX_SKINNY_BATCH_SIZE = hybrid_module.MAX_SKINNY_BATCH_SIZE +LDS_CAPACITY_ELEMENTS = hybrid_module.LDS_CAPACITY_ELEMENTS +MEDIUM_SKINNY_LIMIT_ELEMENTS = hybrid_module.MEDIUM_SKINNY_LIMIT_ELEMENTS # --------------------------------------------------------------------------- @@ -100,8 +102,8 @@ def test_rdna_hybrid_w4a16_apply_matches_reference(dtype, group_size, has_zp, M) """Smoke test the registered custom op for both decode and prefill batches. Verifies the dispatch logic in `_rdna_hybrid_w4a16_apply_impl`: - - M <= MAX_SKINNY_BATCH_SIZE: HIP wvSplitK_int4_g - - M > MAX_SKINNY_BATCH_SIZE: Triton prefill kernel + - supported M and K*M: HIP wvSplitK_int4_g + - otherwise: Triton prefill kernel """ if not torch.cuda.is_available(): pytest.skip("CUDA/HIP device not available") @@ -563,3 +565,257 @@ def test_wvsplitk_int4_g_rejects_unpacked_zero_points(): with pytest.raises(RuntimeError, match="Zero points must be int32 or uint32"): ops.wvSplitK_int4_g(w, a, scales, num_compute_units(), G, zp_unpacked, None) + + +@pytest.mark.skipif(not on_gfx1x(), reason="Hybrid path is gfx11/gfx12 only") +@pytest.mark.skipif( + not hasattr(torch.ops, "_rocm_C") + or not hasattr(torch.ops._rocm_C, "wvSplitK_int4_g"), + reason="wvSplitK_int4_g not built", +) +def test_wvsplitk_int4_g_rejects_undersized_weight_view(): + """A sufficient row stride does not prove the row holds K/2 packed bytes. + + Row padding makes stride(0) > K/2 legitimate, but the logical size still + has to be checked: this view has a valid stride and half the required + bytes, and without the size check the kernel reads past it. + """ + import vllm._custom_ops as ops + from vllm.utils.platform_utils import num_compute_units + + K, N, G, M = 512, 256, 128, 1 + a = torch.randn((M, K), device=device, dtype=torch.float16) + # Full-size backing allocation, so a missing check misreads rather than + # running off the end of the buffer. + backing = torch.randint(0, 255, (N, K // 2), device=device, dtype=torch.uint8).view( + torch.int8 + ) + undersized = backing[:, : K // 4] + assert undersized.stride(0) == K // 2 + scales = torch.rand((N, K // G), device=device, dtype=torch.float16) + + with pytest.raises(RuntimeError, match=r"must contain M\*K/2 bytes"): + ops.wvSplitK_int4_g(undersized, a, scales, num_compute_units(), G, None, None) + + +# --------------------------------------------------------------------------- +# gfx11 weight row-stride padding +# --------------------------------------------------------------------------- + +_weight_pad_bytes = hybrid_module._weight_pad_bytes +_act_pad_bytes = hybrid_module._act_pad_bytes +_WEIGHT_CLIFF_BYTES = hybrid_module._WEIGHT_CLIFF_BYTES +_ACT_CLIFF_BYTES = hybrid_module._ACT_CLIFF_BYTES +_STRIDE_PAD_BYTES = hybrid_module._STRIDE_PAD_BYTES + + +def test_weight_pad_bytes_only_moves_strides_on_the_cliff(): + """Pad 1024 B multiples, leave everything else dense.""" + # On the cliff (K % 2048 == 0): 1024 B multiples. + for row_bytes in (1024, 2048, 4096, 5120, 6144, 7168, 8192): + assert _weight_pad_bytes(row_bytes) == _STRIDE_PAD_BYTES + + # Off it, including strides that are 512 B multiples but not 1024 B ones: + # padding those measured as a real loss on gfx1151 (-14% at M=1 on a + # 4864 B row). + for row_bytes in (1280, 1536, 2560, 4864, 9472, 12800): + assert _weight_pad_bytes(row_bytes) == 0 + + # The padded stride is never back on the cliff. + for row_bytes in range(16, 16384, 16): + padded = row_bytes + _weight_pad_bytes(row_bytes) + assert padded % _WEIGHT_CLIFF_BYTES != 0 or row_bytes % _WEIGHT_CLIFF_BYTES + + +def test_act_pad_bytes_only_moves_strides_on_the_cliff(): + """Activations sit on a wider cliff than the packed weight: 2048 B.""" + # On the cliff (K % 1024 == 0 for a 2-byte dtype): 2048 B multiples. + for row_bytes in (2048, 4096, 8192, 16384, 20480, 24576, 43008): + assert _act_pad_bytes(row_bytes) == _STRIDE_PAD_BYTES + + # Off it, including 1024 B multiples that are not 2048 B ones -- the + # weight cliff must not be applied to activations. + for row_bytes in (1024, 3072, 5120, 10752, 19456, 37888): + assert _act_pad_bytes(row_bytes) == 0 + + # The padded stride is never back on the cliff. + for row_bytes in range(16, 65536, 16): + padded = row_bytes + _act_pad_bytes(row_bytes) + assert padded % _ACT_CLIFF_BYTES != 0 or row_bytes % _ACT_CLIFF_BYTES + + +def _row_padded_copy(t: torch.Tensor, pad_cols: int) -> torch.Tensor: + """Copy of ``t`` whose stride(0) is ``t.shape[1] + pad_cols``.""" + rows, cols = t.shape + buf = torch.empty((rows, cols + pad_cols), dtype=t.dtype, device=t.device) + buf[:, :cols].copy_(t) + return buf[:, :cols] + + +@pytest.mark.skipif(not on_gfx1x(), reason="Hybrid path is gfx11/gfx12 only") +@pytest.mark.parametrize("dtype", [torch.float16, torch.bfloat16]) +@pytest.mark.parametrize("has_zp", [False, True]) +@pytest.mark.parametrize("M", [1, MAX_SKINNY_BATCH_SIZE + 1]) +def test_weight_row_padding_does_not_change_results(dtype, has_zp, M): + """Padding the weight rows is a pure layout change. + + Both the HIP skinny kernel (M small) and the Triton prefill kernel take the + weight row stride from the tensor, so a padded layout must produce exactly + the same numbers as the dense one. + """ + from vllm.utils.platform_utils import num_compute_units + + set_random_seed(0) + K, N, G = 512, 256, 128 + + a = (0.25 * torch.randn((M, K), device=device, dtype=torch.float32)).to(dtype) + w_int4_nk = torch.randint(0, 16, (N, K), device=device, dtype=torch.int32) + w_q = pack_int4_exllama_shuffle(w_int4_nk).view(torch.int8) + scales = (0.05 * torch.rand((N, K // G), device=device, dtype=torch.float32)).to( + dtype + ) + zp = ( + _pack_zp_rows_for_kernel( + torch.randint(0, 16, (N, K // G), device=device, dtype=torch.int32) + ) + if has_zp + else None + ) + + # 32 int32 columns = 128 B of pad, the production pad size. + w_q_pad = _row_padded_copy(w_q.view(torch.int32), 32).view(torch.int8) + assert w_q_pad.stride(0) == w_q.stride(0) + _STRIDE_PAD_BYTES + + cu_count = num_compute_units() + dense = torch.ops.vllm.rdna_hybrid_w4a16_apply( + a, w_q, scales, zp, None, cu_count, G + ) + padded = torch.ops.vllm.rdna_hybrid_w4a16_apply( + a, w_q_pad, scales, zp, None, cu_count, G + ) + torch.testing.assert_close(padded, dense, rtol=0, atol=0) + + +@pytest.mark.skipif( + not hybrid_module._on_gfx1151(), + reason="row-stride padding is only enabled on gfx1151", +) +@pytest.mark.parametrize( + "K,expected_weight_stride", + [ + (512, 256), # 256 B row: off the cliff, stored dense + (8192, 4096 + 128), # 4096 B row: on the cliff, padded + ], +) +def test_process_weights_pads_cliff_rows(K, expected_weight_stride, dist_init): + """The stored weight row stride follows the padding rule.""" + from vllm.model_executor.kernels.linear.mixed_precision.MPLinearKernel import ( + MPLinearLayerConfig, + ) + from vllm.scalar_type import scalar_types + + set_random_seed(0) + N, G = 128, 128 + + w_int4_kn = torch.randint(0, 16, (K, N), device=device, dtype=torch.int32) + layer = _build_dummy_layer( + _pack_int4_along_k_to_ckpt(w_int4_kn), + 0.05 * torch.rand((N, K // G), device=device, dtype=torch.float16), + zeros_ckpt=None, + ) + config = MPLinearLayerConfig( + full_weight_shape=(K, N), + partition_weight_shape=(K, N), + weight_type=scalar_types.uint4b8, + act_type=torch.float16, + group_size=G, + zero_points=False, + ) + RDNAHybridW4A16LinearKernel( + config, + w_q_param_name="weight_packed", + w_s_param_name="weight_scale", + w_zp_param_name=None, + ).process_weights_after_loading(layer) + + # Weight rows: stride is in int8 elements, i.e. bytes. + assert layer.weight_packed.stride(0) == expected_weight_stride + # The int32 view the Triton path uses survives the padded stride. + assert tuple(layer.weight_packed.view(torch.int32).shape) == (N, K // 8) + # Metadata is untouched by this change. + assert layer.weight_scale.is_contiguous() + + +@pytest.mark.skipif(not on_gfx1x(), reason="Hybrid path is gfx11/gfx12 only") +@pytest.mark.parametrize( + "M,K,expected_path", + [ + (4, 8192, "hip"), + (4, 9728, "hip"), + (4, 9856, "triton"), + (MAX_SKINNY_BATCH_SIZE + 1, 1024, "triton"), + ], + ids=["regular_limit", "medium_range", "above_medium", "batch_too_large"], +) +def test_rdna_hybrid_w4a16_dispatch_boundaries(M, K, expected_path, monkeypatch): + """Route only supported regular and medium skinny shapes to HIP.""" + import vllm._custom_ops as ops + + N, G = 16, 128 + x_mk = torch.empty((M, K), dtype=torch.float16) + w_q = torch.empty((N, K // 2), dtype=torch.int8) + scales = torch.empty((N, K // G), dtype=torch.float16) + called = [] + + def fake_hip(*args, **kwargs): + called.append("hip") + return torch.empty((M, N), dtype=x_mk.dtype) + + def fake_triton(*args, **kwargs): + called.append("triton") + return torch.empty((M, N), dtype=x_mk.dtype) + + monkeypatch.setattr(ops, "wvSplitK_int4_g", fake_hip) + monkeypatch.setattr(hybrid_module, "triton_w4a16_skinny_fmt_gemm", fake_triton) + + output = hybrid_module._rdna_hybrid_w4a16_apply_impl( + x_mk, w_q, scales, None, None, 40, G + ) + + assert output.shape == (M, N) + assert called == [expected_path] + + +@pytest.mark.skipif(not on_gfx1x(), reason="Hybrid path is gfx11/gfx12 only") +@pytest.mark.parametrize("dtype", [torch.float16, torch.bfloat16]) +@pytest.mark.parametrize("has_zp", [False, True]) +def test_rdna_hybrid_w4a16_medium_skinny_matches_reference(dtype, has_zp): + """Validate the HIP medium path against an FP32 dequantization reference.""" + if not torch.cuda.is_available(): + pytest.skip("CUDA/HIP device not available") + + from vllm.utils.platform_utils import num_compute_units + + set_random_seed(0) + M, K, N, G = 4, 9728, 256, 128 + assert LDS_CAPACITY_ELEMENTS < K * M <= MEDIUM_SKINNY_LIMIT_ELEMENTS + + x_mk = (0.25 * torch.randn((M, K), device=device, dtype=torch.float32)).to(dtype) + w_int4_nk = torch.randint(0, 16, (N, K), device=device, dtype=torch.int32) + w_q = pack_int4_exllama_shuffle(w_int4_nk).contiguous().view(torch.int8) + scales = (0.05 * torch.rand((N, K // G), device=device, dtype=torch.float32)).to( + dtype + ) + zp = ( + torch.randint(0, 16, (N, K // G), device=device, dtype=torch.int32) + if has_zp + else None + ) + w_zp = _pack_zp_rows_for_kernel(zp) if zp is not None else None + + output = torch.ops.vllm.rdna_hybrid_w4a16_apply( + x_mk, w_q, scales, w_zp, None, num_compute_units(), G + ) + reference = _rdna_hybrid_w4a16_reference(x_mk, w_int4_nk, scales, zp, G, bias=None) + + torch.testing.assert_close(output, reference, rtol=2e-2, atol=2e-2) diff --git a/tests/kernels/quantization/test_scaled_mm_kernel_selection.py b/tests/kernels/quantization/test_scaled_mm_kernel_selection.py index f8380dcecf..641c458608 100644 --- a/tests/kernels/quantization/test_scaled_mm_kernel_selection.py +++ b/tests/kernels/quantization/test_scaled_mm_kernel_selection.py @@ -14,6 +14,8 @@ import torch from vllm.config import KernelConfig, VllmConfig, set_current_vllm_config from vllm.model_executor.kernels.linear import ( + _POSSIBLE_KERNELS, + _POSSIBLE_NVFP4_KERNELS, AiterInt8ScaledMMLinearKernel, CPUInt8ScaledMMLinearKernel, HummingFP8ScaledMMLinearKernel, @@ -26,6 +28,7 @@ from vllm.model_executor.kernels.linear import ( init_int8_linear_kernel, register_linear_kernel, ) +from vllm.model_executor.layers.quantization.utils.humming import prioritize_humming from vllm.model_executor.layers.quantization.utils.quant_utils import ( kFp8StaticTensorSym, ) @@ -197,3 +200,21 @@ def test_register_oot_linear_kernel(platform_mock): assert isinstance(kernel, OOTInt8ScaledMMLinearKernel), ( "init_int8_linear_kernel should return an instance of the registered kernel" ) + + +@pytest.mark.parametrize( + "possible_kernels", [_POSSIBLE_KERNELS, _POSSIBLE_NVFP4_KERNELS] +) +def test_humming_priority_only_reorders_humming_and_marlin(possible_kernels): + """Preferring Humming must not demote Marlin below unrelated kernels.""" + kernels = possible_kernels[PlatformEnum.CUDA] + original = list(kernels) + + assert prioritize_humming(kernels, compute_capability=80) == original + + names = [k.__name__ for k in prioritize_humming(kernels, compute_capability=90)] + humming = next(i for i, name in enumerate(names) if "Humming" in name) + assert "Marlin" in names[humming + 1] + del names[humming] + assert names == [k.__name__ for k in original if "Humming" not in k.__name__] + assert kernels == original diff --git a/tests/kernels/test_bf16x3_router_gemm_rocm.py b/tests/kernels/test_bf16x3_router_gemm_rocm.py new file mode 100644 index 0000000000..69aa195b55 --- /dev/null +++ b/tests/kernels/test_bf16x3_router_gemm_rocm.py @@ -0,0 +1,201 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Tests for the ROCm BF16x3 router GEMM. + +The weight split and the eligibility policy are pure functions, so most of +this runs device-free. Only the GEMM correctness tests need a gfx950 runner. +""" + +import pytest +import torch + +from vllm.model_executor.layers.fused_moe.router import ( + bf16x3_router_gemm_rocm as rocm_bf16x3, +) +from vllm.platforms import current_platform + +pytestmark = pytest.mark.skipif(not current_platform.is_rocm(), reason="ROCm only") + +# (hidden_size, num_experts) for the models that ship fp32 router weights. +# (6144, 128) -> MiniMax-M3, (3072, 256) -> MiniMax-M2/M2.5 +ROUTER_SHAPES = [(6144, 128), (3072, 256)] +ROUTER_WEIGHT_SCALE = 0.053 + + +def _requires_gfx950(): + if not rocm_bf16x3.platform_supported(): + pytest.skip("ROCm bf16x3 router GEMM requires gfx950") + + +def _inputs(num_tokens, hidden_size, num_experts, seed=42, device="cuda"): + torch.manual_seed(seed) + x = torch.randn(num_tokens, hidden_size, device=device).to(torch.bfloat16) + w = torch.randn(num_experts, hidden_size, dtype=torch.float32, device=device) + return x, w * ROUTER_WEIGHT_SCALE + + +def _top_set(logits, k=4): + """Sorted top-k indices; topk's order among near-ties is not meaningful.""" + return logits.topk(k, dim=-1).indices.sort(dim=-1).values + + +@pytest.mark.parametrize(("hidden_size", "num_experts"), ROUTER_SHAPES) +def test_split_reconstructs_exactly(hidden_size: int, num_experts: int): + torch.manual_seed(42) + w = torch.randn(num_experts, hidden_size) * ROUTER_WEIGHT_SCALE + split = rocm_bf16x3.split_bf16x3(w) + + assert split.shape == (rocm_bf16x3.BF16X3_TERMS, num_experts, hidden_size) + assert split.dtype == torch.bfloat16 + # router_gemm reinterprets this as (TERMS * E, K) without a copy. + assert split.is_contiguous() + assert torch.equal(split.float().sum(0), w) + + +def test_split_is_exact_across_normal_exponents(): + exps = torch.arange(-100, 100, dtype=torch.float32) + mantissas = torch.linspace(1.0, 1.999, 64, dtype=torch.float32) + w = (mantissas[:, None] * torch.pow(2.0, exps)[None, :]).contiguous() + w = torch.cat([w, -w], dim=0) + assert torch.equal(rocm_bf16x3.split_bf16x3(w).float().sum(0), w) + + +def test_split_rejects_non_finite(): + """ValueError rather than assert: the caller downgrades to the fp32 path + instead of failing model load, and this must survive ``python -O``.""" + for bad in (float("inf"), float("-inf"), float("nan")): + with pytest.raises(ValueError): + rocm_bf16x3.split_bf16x3(torch.full((2, 4), bad, dtype=torch.float32)) + + +def test_split_rejects_non_fp32(): + with pytest.raises(ValueError): + rocm_bf16x3.split_bf16x3(torch.zeros(2, 4, dtype=torch.bfloat16)) + + +def test_split_does_not_mutate_input(): + w = torch.randn(8, 32, dtype=torch.float32) + original = w.clone() + rocm_bf16x3.split_bf16x3(w) + assert torch.equal(w, original) + + +def test_is_supported_rejects_ineligible_inputs(): + meta = torch.device("meta") + x = torch.zeros(8192, 6144, dtype=torch.bfloat16, device=meta) + w = torch.zeros(128, 6144, dtype=torch.float32, device=meta) + assert rocm_bf16x3.is_supported(x, w) + + assert not rocm_bf16x3.is_supported(x[: rocm_bf16x3.MIN_TOKENS - 1], w) + assert not rocm_bf16x3.is_supported(x.float(), w) + assert not rocm_bf16x3.is_supported(x, w.bfloat16()) + assert not rocm_bf16x3.is_supported(x, w[:, :128]) + assert not rocm_bf16x3.is_supported(x.T.contiguous().T, w) + + +def test_is_supported_rejects_mismatched_devices(): + """The weight can be offloaded while the activations are not.""" + x = torch.zeros(8192, 6144, dtype=torch.bfloat16, device="meta") + w = torch.zeros(128, 6144, dtype=torch.float32) + assert not rocm_bf16x3.is_supported(x, w) + + +def test_is_supported_accepts_odd_expert_counts(): + """Unlike a tl.arange-indexed kernel, torch.mm needs no power-of-two E.""" + meta = torch.device("meta") + x = torch.zeros(8192, 6144, dtype=torch.bfloat16, device=meta) + for num_experts in (17, 384, 512): + w = torch.zeros(num_experts, 6144, dtype=torch.float32, device=meta) + assert rocm_bf16x3.is_supported(x, w) + + +# Measured on gfx950: this path lands at 3.4e-7 to 4.9e-7 and varies <0.2% +# across token counts. Fixed bounds rather than a live comparison, because the +# fp32 fallback swings 5.7e-7 to 1.4e-6 as torch picks different hipBLASLt +# kernels at different M. _FP32_FALLBACK_BEST is the best the fallback was ever +# observed to do, so beating it proves the claim without racing a moving target. +_MAX_REL_L2 = 2e-6 +_FP32_FALLBACK_BEST = 5.7e-7 + + +@pytest.mark.parametrize(("hidden_size", "num_experts"), ROUTER_SHAPES) +@pytest.mark.parametrize("num_tokens", [2048, 8192, 16385]) +def test_matches_fp64_reference(hidden_size: int, num_experts: int, num_tokens: int): + _requires_gfx950() + x, w = _inputs(num_tokens, hidden_size, num_experts) + out = rocm_bf16x3.bf16x3_router_gemm(x, rocm_bf16x3.split_bf16x3(w)) + ref = x.double() @ w.double().T + + assert out.shape == (num_tokens, num_experts) + assert out.dtype == torch.float32 + assert (out.double() - ref).norm() / ref.norm() < _MAX_REL_L2 + + +@pytest.mark.parametrize("num_experts", [17, 384]) +def test_matches_fp64_for_non_power_of_two_experts(num_experts: int): + """Exercises the reduction's N mask, which the model shapes never hit.""" + _requires_gfx950() + x, w = _inputs(8192, 6144, num_experts) + out = rocm_bf16x3.bf16x3_router_gemm(x, rocm_bf16x3.split_bf16x3(w)) + ref = x.double() @ w.double().T + + assert out.shape == (8192, num_experts) + assert (out.double() - ref).norm() / ref.norm() < _MAX_REL_L2 + + +@pytest.mark.parametrize(("hidden_size", "num_experts"), ROUTER_SHAPES) +def test_more_accurate_than_fp32_fallback(hidden_size: int, num_experts: int): + """The claim is not just accuracy but beating the fp32 GEMM it replaces.""" + _requires_gfx950() + x, w = _inputs(16384, hidden_size, num_experts) + ref = x.double() @ w.double().T + out = rocm_bf16x3.bf16x3_router_gemm(x, rocm_bf16x3.split_bf16x3(w)) + assert (out.double() - ref).norm() / ref.norm() < _FP32_FALLBACK_BEST + + +@pytest.mark.parametrize(("hidden_size", "num_experts"), ROUTER_SHAPES) +def test_expert_selection_matches_fp64(hidden_size: int, num_experts: int): + """Relative L2 is a proxy; the expert set is what reaches the model. + + Measures zero misroutes on both shapes, where the fp32 fallback misroutes a + nonzero number at some token counts. + """ + _requires_gfx950() + num_tokens = 16384 + x, w = _inputs(num_tokens, hidden_size, num_experts) + ref_set = _top_set(x.double() @ w.double().T) + out = rocm_bf16x3.bf16x3_router_gemm(x, rocm_bf16x3.split_bf16x3(w)) + + misrouted = int((_top_set(out) != ref_set).any(-1).sum()) + assert misrouted / num_tokens < 1e-4 + + +def test_writes_into_supplied_out(): + _requires_gfx950() + x, w = _inputs(8192, 6144, 128) + split = rocm_bf16x3.split_bf16x3(w) + out = torch.empty(8192, 128, dtype=torch.float32, device="cuda") + + returned = rocm_bf16x3.bf16x3_router_gemm(x, split, out=out) + assert returned is out + assert torch.equal(out, rocm_bf16x3.bf16x3_router_gemm(x, split)) + + +def test_validates_layout_assumptions(): + """The reduction assumes a contiguous (TERMS, E, K) split and a contiguous + out; nothing in the kernel enforces either.""" + _requires_gfx950() + x, w = _inputs(8192, 6144, 128) + split = rocm_bf16x3.split_bf16x3(w) + + with pytest.raises(AssertionError): + rocm_bf16x3.bf16x3_router_gemm(x, split.transpose(1, 2)) + with pytest.raises(AssertionError): + rocm_bf16x3.bf16x3_router_gemm(x.float(), split) + with pytest.raises(AssertionError): + rocm_bf16x3.bf16x3_router_gemm(x, split[:2]) + with pytest.raises(AssertionError): + rocm_bf16x3.bf16x3_router_gemm(x[:, :512], split) + wide = torch.empty(8192, 256, dtype=torch.float32, device="cuda") + with pytest.raises(AssertionError): + rocm_bf16x3.bf16x3_router_gemm(x, split, out=wide[:, :128]) diff --git a/tests/kernels/test_compressor_kv_cache.py b/tests/kernels/test_compressor_kv_cache.py index 020dd607b6..cc4f4c72a9 100644 --- a/tests/kernels/test_compressor_kv_cache.py +++ b/tests/kernels/test_compressor_kv_cache.py @@ -405,6 +405,80 @@ def test_v41_rope_insert_mxfp8_record(compress_ratio: int): torch.testing.assert_close(cache_backing, expected, rtol=0, atol=0) +@pytest.mark.skipif( + not current_platform.is_cuda_alike(), reason="needs a CUDA or ROCm device" +) +@pytest.mark.parametrize("gather_len", [None, 70, 200]) +@pytest.mark.parametrize("one_pass", [False, True]) +def test_v41_fp8_gather_is_grid_width_invariant(gather_len, one_pass, monkeypatch): + """The gather grid only partitions tokens, so its width cannot move the + result, and the rows it writes must match a host-side dequantization. + + Covers the fp8_ds_mla record (584 B); the MXFP8 one is covered below. + Both dequantization forms are checked against the same host reference on + whatever device is present: the wrapper picks the one-pass form on ROCm + alone, so otherwise each platform would only ever see one of them. + """ + monkeypatch.setattr(current_platform, "is_rocm", lambda: one_pass) + from vllm.models.deepseek_v41.common.ops import quantize_and_insert_k_cache + from vllm.models.deepseek_v41.common.ops.cache_utils import ( + dequantize_and_gather_k_cache_triton, + ) + + torch.manual_seed(7) + device = "cuda" + block_size, num_blocks = 64, 6 + num_tokens = block_size * num_blocks + + k = torch.randn(num_tokens, 512, dtype=torch.bfloat16, device=device) + # insert takes [num_blocks, block_bytes]; gather takes the record view. + cache = torch.zeros(num_blocks, block_size * 584, dtype=torch.uint8, device=device) + slot_mapping = torch.arange(num_tokens, dtype=torch.int64, device=device) + quantize_and_insert_k_cache(k, cache, slot_mapping, block_size=block_size) + cache_view = cache.view(num_blocks, block_size, 584) + + seq_lens = torch.tensor([num_tokens], dtype=torch.int32, device=device) + gls = ( + None + if gather_len is None + else torch.tensor([gather_len], dtype=torch.int32, device=device) + ) + block_table = torch.arange(num_blocks, dtype=torch.int32, device=device).view(1, -1) + + def gather(bound): + out = torch.zeros(1, num_tokens, 512, dtype=torch.bfloat16, device=device) + dequantize_and_gather_k_cache_triton( + out, + cache_view, + seq_lens, + gls, + block_table, + block_size, + 0, + max_gather_len=bound, + ) + return out + + # None keeps the historical 128; the rest make the wrapper pick other widths + # for the same work, and the output must not notice. + baseline = gather(None) + for bound in (8, num_tokens, 1 << 20): + torch.testing.assert_close(gather(bound), baseline, rtol=0, atol=0) + + n = num_tokens if gather_len is None else gather_len + start = num_tokens - n + for i in (0, 1, n // 2, n - 1): + values, scales = _ue8m0_reference(k[start + i, :448], 64, 448.0) + nope = (values.to(torch.float32).view(7, 64) * scales.view(7, 1)).reshape(448) + torch.testing.assert_close( + baseline[0, i, :448], nope.to(torch.bfloat16), rtol=4e-3, atol=1e-6 + ) + # The RoPE dims are stored verbatim, so they come back exactly. + torch.testing.assert_close( + baseline[0, i, 448:], k[start + i, 448:], rtol=0, atol=0 + ) + + @pytest.mark.skipif(not current_platform.is_cuda(), reason="CUDA only") def test_v41_mxfp8_cache_round_trip(): """quantize_and_insert -> dequantize_and_gather recovers the V4.1 record. diff --git a/tests/kernels/test_cpu_fused_sampling.py b/tests/kernels/test_cpu_fused_sampling.py index 8a3857fe9e..cdb10ab7e3 100644 --- a/tests/kernels/test_cpu_fused_sampling.py +++ b/tests/kernels/test_cpu_fused_sampling.py @@ -43,20 +43,20 @@ class TestGreedyArgmax: class TestFusedGumbelArgmax: - @pytest.mark.parametrize("vocab_size", VOCAB_SIZES) - def test_distribution_chi_squared(self, vocab_size: int): + def test_distribution_chi_squared(self): """Verify sampling distribution via chi-squared goodness of fit.""" - small_vocab = min(vocab_size, 100) + # Small support is enough for chi-squared power; large-vocab bias + # is covered by test_noise_not_sliding_window (#59786). + small_vocab = 100 logits = torch.randn(1, small_vocab, dtype=torch.float32) probs = logits.softmax(dim=-1).squeeze(0) n_samples = 100_000 - counts = torch.zeros(small_vocab) - for trial in range(n_samples): - seed = torch.tensor([trial * 7 + 13], dtype=torch.long) - tile = logits.expand(1, -1).contiguous() - idx = torch.ops._C.fused_gumbel_argmax(tile, seed) - counts[idx.item()] += 1 + seeds = torch.arange(n_samples, dtype=torch.long) * 7 + 13 + idx = torch.ops._C.fused_gumbel_argmax( + logits.expand(n_samples, -1).contiguous(), seeds + ) + counts = torch.bincount(idx, minlength=small_vocab).float() expected = probs * n_samples mask = expected > 5 @@ -70,6 +70,39 @@ class TestFusedGumbelArgmax: f"dof={dof}, p={p_value:.6f}" ) + def test_noise_not_sliding_window(self): + """Table[(seed+i)&MASK] made consecutive seeds a 1-token shift. + + On uniform logits that yields P(w(s+1) == w(s)-1) ~ 1-2/V. Independent + per-token noise yields ~1/V. See #59786. + """ + vocab, n = 128, 2048 + logits = torch.zeros(n, vocab, dtype=torch.float32) + seeds = torch.arange(n, dtype=torch.long) + winners = torch.ops._C.fused_gumbel_argmax(logits, seeds) + shifted = (winners[:-1] - 1) % vocab + frac = (winners[1:] == shifted).float().mean().item() + assert frac < 0.05, ( + f"consecutive-seed winners follow a table shift (frac={frac:.3f})" + ) + + def test_zipf_head_matches_softmax(self): + """Zipf head probability should match softmax (#59786).""" + vocab, n_samples, batch = 8192, 16384, 256 + logits = -1.1 * torch.arange(1, vocab + 1, dtype=torch.float32).log() + p0 = torch.softmax(logits.double(), dim=-1)[0].item() + row = logits.view(1, -1).expand(batch, vocab).contiguous() + n_top = 0 + for start in range(0, n_samples, batch): + seeds = torch.arange(start, start + batch, dtype=torch.long) + tokens = torch.ops._C.fused_gumbel_argmax(row, seeds) + n_top += int((tokens == 0).sum()) + q0 = n_top / n_samples + se = (p0 * (1.0 - p0) / n_samples) ** 0.5 + assert abs(q0 - p0) < 8.0 * se, ( + f"P(top-1) kernel={q0:.4f} exact={p0:.4f} se={se:.4f}" + ) + def test_deterministic_same_seed(self): """Same seed produces same result.""" logits = torch.randn(4, 32000, dtype=torch.float32) @@ -95,6 +128,108 @@ class TestFusedGumbelArgmax: assert result.min() >= 0 assert result.max() < vocab_size + def test_samples_only_from_finite_logits(self): + """top-k/top-p mask with -inf; those IDs must never win.""" + allowed = torch.tensor([3, 17, 99, 1024, 8191], dtype=torch.long) + logits = torch.full((64, 32000), float("-inf"), dtype=torch.float32) + logits[:, allowed] = torch.randn(64, allowed.numel()) + tokens = torch.ops._C.fused_gumbel_argmax( + logits, torch.arange(64, dtype=torch.long) + ) + assert torch.isin(tokens, allowed).all() + + def test_single_unmasked_token(self): + logits = torch.full((8, 4096), float("-inf"), dtype=torch.float32) + logits[:, 123] = 0.0 + seeds = torch.tensor([0, -1, -(2**63), 2**63 - 1, 42, 99, 1000, 7]) + tokens = torch.ops._C.fused_gumbel_argmax(logits, seeds) + assert (tokens == 123).all() + + def test_tokens_beyond_gumbel_max_cannot_win(self): + """A gap larger than GUMBEL_MAX is unreachable.""" + logits = torch.zeros(256, 4, dtype=torch.float32) + logits[:, 1:] = -80.0 + tokens = torch.ops._C.fused_gumbel_argmax( + logits, torch.arange(256, dtype=torch.long) + ) + assert (tokens == 0).all() + + def test_tied_huge_logits_not_index_biased(self): + """Noise must still break ties when logits sit at float32 max.""" + logits = torch.full( + (4096, 2), torch.finfo(torch.float32).max, dtype=torch.float32 + ) + tokens = torch.ops._C.fused_gumbel_argmax( + logits, torch.arange(4096, dtype=torch.long) + ) + frac0 = (tokens == 0).float().mean().item() + assert set(tokens.tolist()) == {0, 1}, tokens.unique().tolist() + assert 0.35 < frac0 < 0.65, f"P(index 0)={frac0:.3f} expected ~0.5" + + def test_serial_batch1_matches_batched(self): + """OpenMP rows must match the batch=1 serial path.""" + logits = torch.randn(8, 2048, dtype=torch.float32) + seeds = torch.arange(8, dtype=torch.long) + batched = torch.ops._C.fused_gumbel_argmax(logits, seeds) + serial = torch.stack( + [ + torch.ops._C.fused_gumbel_argmax(logits[i : i + 1], seeds[i : i + 1])[0] + for i in range(8) + ] + ) + torch.testing.assert_close(batched, serial) + + def test_noncontiguous_matches_contiguous(self): + logits = torch.randn(8, 1024, dtype=torch.float32)[:, ::2] + seeds = torch.arange(16, dtype=torch.long)[::2] + assert not logits.is_contiguous() and not seeds.is_contiguous() + expected = torch.ops._C.fused_gumbel_argmax( + logits.contiguous(), seeds.contiguous() + ) + actual = torch.ops._C.fused_gumbel_argmax(logits, seeds) + torch.testing.assert_close(actual, expected) + + def test_empty_batch(self): + tokens = torch.ops._C.fused_gumbel_argmax( + torch.empty(0, 7), torch.empty(0, dtype=torch.long) + ) + assert tokens.shape == (0,) and tokens.dtype == torch.int64 + + def test_masked_support_matches_softmax(self): + """Finite subset after -inf mask should match softmax.""" + from scipy.stats import chisquare + + logits = torch.tensor( + [0.0, -0.2, -0.7, -1.5, -3.0, float("-inf")], dtype=torch.float32 + ) + n_samples = 20_000 + tokens = torch.ops._C.fused_gumbel_argmax( + logits.expand(n_samples, -1).contiguous(), + torch.arange(n_samples, dtype=torch.long), + ) + counts = torch.bincount(tokens, minlength=logits.numel()).double() + assert counts[-1] == 0 + probs = logits.double().softmax(-1) + _, p_value = chisquare(counts[:-1].numpy(), (probs[:-1] * n_samples).numpy()) + assert p_value > 1e-7, f"masked-support p-value: {p_value}" + + def test_zipf_tail_matches_softmax(self): + """Independent 53-bit Gumbel must not starve the Zipf tail (#59786).""" + vocab, n_samples, batch = 8192, 16384, 256 + logits = -1.1 * torch.arange(1, vocab + 1, dtype=torch.float32).log() + p_tail = torch.softmax(logits.double(), dim=-1)[vocab // 2 :].sum().item() + row = logits.view(1, -1).expand(batch, vocab).contiguous() + n_tail = 0 + for start in range(0, n_samples, batch): + seeds = torch.arange(start, start + batch, dtype=torch.long) + tokens = torch.ops._C.fused_gumbel_argmax(row, seeds) + n_tail += int((tokens >= vocab // 2).sum()) + q_tail = n_tail / n_samples + se = (p_tail * (1.0 - p_tail) / n_samples) ** 0.5 + assert abs(q_tail - p_tail) < 8.0 * se, ( + f"P(tail) kernel={q_tail:.4f} exact={p_tail:.4f} se={se:.4f}" + ) + class TestMemory: def test_fused_no_intermediate_allocs(self): diff --git a/tests/kernels/test_dsv41_mega_attn_layouts.py b/tests/kernels/test_dsv41_mega_attn_layouts.py index 6f9ff56bd2..3d45418360 100644 --- a/tests/kernels/test_dsv41_mega_attn_layouts.py +++ b/tests/kernels/test_dsv41_mega_attn_layouts.py @@ -3,25 +3,34 @@ """Layout plumbing for DeepSeek V4.1 mega attention. The kernel's Q and O layouts are produced by permuting ``wq_b`` rows and -``wo_a`` columns once at load, so what has to hold is that the permuted GEMMs -agree with a reference that permutes the activation instead. +``wo_a`` columns in their weight loaders, so what has to hold is that the +permuted GEMMs agree with a reference that permutes the activation instead. """ import torch +from vllm.model_executor.model_loader.weight_utils import default_weight_loader from vllm.models.deepseek_v41.common.ops.fused_layout import ( WV_GROUP_SIZE, o_fused_chunk_permutation, o_fused_permutation, + permute_on_load, permute_q_to_fused, - permute_wo_a_, - permute_wq_b_, q_fused_permutation, ) HEAD_DIM = 512 +def _load(checkpoint: torch.Tensor, perm: torch.Tensor, dim: int) -> torch.Tensor: + """Load ``checkpoint`` into a fresh parameter through a permuting loader.""" + param = torch.nn.Parameter(torch.empty_like(checkpoint), requires_grad=False) + param.weight_loader = default_weight_loader # type: ignore[attr-defined] + permute_on_load(param, perm, dim) + param.weight_loader(param, checkpoint) # type: ignore[attr-defined] + return param.data + + def test_permuted_wq_b_gemm_matches_permuted_activation(): """Permuting wq_b's rows makes the Q GEMM emit the fused layout directly.""" torch.manual_seed(0) @@ -32,18 +41,17 @@ def test_permuted_wq_b_gemm_matches_permuted_activation(): 0, 256, (num_heads * HEAD_DIM, q_lora_rank // 32), dtype=torch.uint8 ) qr = torch.randn(num_tokens, q_lora_rank) - orig_weight, orig_scale = weight.clone(), scale.clone() standard = (qr @ weight.T).view(num_tokens, num_heads, HEAD_DIM) expected = permute_q_to_fused(standard) - permute_wq_b_(weight, scale, num_heads) - fused = (qr @ weight.T).view(num_tokens, num_heads, HEAD_DIM) + perm = q_fused_permutation(num_heads, HEAD_DIM) + loaded_weight = _load(weight, perm, dim=0) + fused = (qr @ loaded_weight.T).view(num_tokens, num_heads, HEAD_DIM) torch.testing.assert_close(fused, expected) # The scale has to follow its row, or dequant reads another row's scale. - perm = q_fused_permutation(num_heads, HEAD_DIM) - torch.testing.assert_close(weight, orig_weight[perm]) - torch.testing.assert_close(scale, orig_scale[perm]) + torch.testing.assert_close(loaded_weight, weight[perm]) + torch.testing.assert_close(_load(scale, perm, dim=0), scale[perm]) def test_permuted_wo_a_consumes_fused_output(): @@ -54,18 +62,18 @@ def test_permuted_wo_a_consumes_fused_output(): weight = torch.randn(out_features, in_features) scale = torch.randint(0, 256, (out_features, in_features // 32), dtype=torch.uint8) standard_o = torch.randn(num_tokens, in_features) - orig_weight, orig_scale = weight.clone(), scale.clone() expected = standard_o @ weight.T # The kernel emits the same values in the fused chunk order. perm = o_fused_permutation(WV_GROUP_SIZE, HEAD_DIM) fused_o = standard_o[:, perm] - permute_wo_a_(weight, scale, WV_GROUP_SIZE) - torch.testing.assert_close(weight, orig_weight[:, perm]) - torch.testing.assert_close( - scale, orig_scale[:, o_fused_chunk_permutation(WV_GROUP_SIZE, HEAD_DIM)] - ) + loaded_weight = _load(weight, perm, dim=1) + torch.testing.assert_close(loaded_weight, weight[:, perm]) + chunk_perm = o_fused_chunk_permutation(WV_GROUP_SIZE, HEAD_DIM) + torch.testing.assert_close(_load(scale, chunk_perm, dim=1), scale[:, chunk_perm]) # Unlike wq_b, this permutation sits inside the 4096-term reduction, so the # summation order changes and the result differs in the last few ulps. - torch.testing.assert_close(fused_o @ weight.T, expected, rtol=1e-3, atol=1e-3) + torch.testing.assert_close( + fused_o @ loaded_weight.T, expected, rtol=1e-3, atol=1e-3 + ) diff --git a/tests/kernels/test_engram.py b/tests/kernels/test_engram.py index 15bdc9162c..3663108d9d 100644 --- a/tests/kernels/test_engram.py +++ b/tests/kernels/test_engram.py @@ -7,6 +7,9 @@ from types import SimpleNamespace import pytest import torch +from vllm.model_executor.layers.quantization.utils.mxfp8_utils import ( + dequant_mxfp8_to_bf16, +) from vllm.models.deepseek_v41.common import engram as engram_ops from vllm.models.deepseek_v41.common.engram import ( Engram, @@ -714,8 +717,9 @@ def test_engram_lookup_reuses_jit_across_token_shapes(): ) @pytest.mark.parametrize("cpu_offload", [False, True]) @pytest.mark.parametrize("background", [False, True]) +@pytest.mark.parametrize("packed", [False, True]) @pytest.mark.parametrize("num_tokens", [1, 7, 256]) -def test_engram_lookup_matches_torch(cpu_offload, background, num_tokens): +def test_engram_lookup_matches_torch(cpu_offload, background, packed, num_tokens): """The fused gather must be bit-exact with the torch dequant path it replaces, from HBM and from pinned host memory alike, and must contribute zeros for rows another TP rank owns.""" @@ -729,8 +733,13 @@ def test_engram_lookup_matches_torch(cpu_offload, background, num_tokens): layer.vocab_start_idx, layer.vocab_end_idx, ) - out = torch.empty(num_tokens, cols, layer.dim, dtype=torch.bfloat16, device="cuda") + width = layer.dim + layer.dim // 32 if packed else layer.dim + dtype = torch.uint8 if packed else torch.bfloat16 + out = torch.empty(num_tokens, cols, width, dtype=dtype, device="cuda") layer.lookup(ids, out, background=background) + if packed: + values = out[..., : layer.dim].view(torch.float8_e4m3fn) + out = dequant_mxfp8_to_bf16(values, out[..., layer.dim :]) assert torch.equal(out, expected) diff --git a/tests/kernels/test_flashinfer_bf16_gemm.py b/tests/kernels/test_flashinfer_bf16_gemm.py index 63a5a39c2e..28520a6083 100644 --- a/tests/kernels/test_flashinfer_bf16_gemm.py +++ b/tests/kernels/test_flashinfer_bf16_gemm.py @@ -25,6 +25,8 @@ def require_flashinfer_bf16_cutedsl() -> None: (8, True, False), (16, False, True), (32, True, True), + (64, True, True), + (512, False, False), ], ) def test_flashinfer_bf16_cutedsl_correctness( diff --git a/tests/kernels/test_fused_quant_activation.py b/tests/kernels/test_fused_quant_activation.py index 321cb6978e..90309dbd1d 100644 --- a/tests/kernels/test_fused_quant_activation.py +++ b/tests/kernels/test_fused_quant_activation.py @@ -115,13 +115,13 @@ class MockLinearRequiresUnquantized(MockLinearFp8Static): requires_unquantized_input = True -@pytest.mark.parametrize("num_tokens", [1, 16, 128]) +@pytest.mark.parametrize("token_shape", [(1,), (16,), (128,), (1, 17), (2, 17)]) @pytest.mark.parametrize("hidden_size", [128, 512, 1024]) @pytest.mark.parametrize("dtype", DTYPES) @torch.inference_mode() def test_maybe_fused_act_quant_fp8_static( default_vllm_config, - num_tokens: int, + token_shape: tuple[int, ...], hidden_size: int, dtype: torch.dtype, ) -> None: @@ -133,26 +133,29 @@ def test_maybe_fused_act_quant_fp8_static( scale = torch.tensor([0.5], device=device, dtype=torch.float32) linear = MockLinearFp8Static(scale) - x = torch.randn(num_tokens, hidden_size * 2, dtype=dtype, device=device) + x = torch.randn(*token_shape, hidden_size * 2, dtype=dtype, device=device) result = maybe_fused_act_quant(act_fn, x, linear) assert isinstance(result, QuantizedActivation) assert result.quant_key == kFp8StaticTensorSym assert result.data.dtype == current_platform.fp8_dtype() assert result.orig_dtype == dtype - assert result.orig_shape == (num_tokens, hidden_size) + assert result.orig_shape == (*token_shape, hidden_size) + assert result.data.shape == result.orig_shape - ref_out = ref_impl(act_fn, x, scale) + ref_out = ref_impl(act_fn, x.reshape(-1, hidden_size * 2), scale).reshape( + result.orig_shape + ) torch.testing.assert_close(result.data.to(torch.float32), ref_out.to(torch.float32)) -@pytest.mark.parametrize("num_tokens", [1, 16, 128]) +@pytest.mark.parametrize("token_shape", [(1,), (16,), (128,), (1, 17), (2, 17)]) @pytest.mark.parametrize("hidden_size", [128, 512, 1024]) @pytest.mark.parametrize("dtype", DTYPES) @torch.inference_mode() def test_maybe_fused_act_quant_fp8_dynamic_block( default_vllm_config, - num_tokens: int, + token_shape: tuple[int, ...], hidden_size: int, dtype: torch.dtype, ) -> None: @@ -166,31 +169,34 @@ def test_maybe_fused_act_quant_fp8_dynamic_block( linear = MockLinearFp8Dynamic128() scale = 1 / hidden_size - x = torch.randn(num_tokens, hidden_size * 2, dtype=dtype, device=device) * scale + x = torch.randn(*token_shape, hidden_size * 2, dtype=dtype, device=device) * scale result = maybe_fused_act_quant(act_fn, x, linear) assert isinstance(result, QuantizedActivation) assert result.quant_key == kFp8Dynamic128Sym assert result.data.dtype == current_platform.fp8_dtype() assert result.orig_dtype == dtype - assert result.orig_shape == (num_tokens, hidden_size) + assert result.orig_shape == (*token_shape, hidden_size) + assert result.data.shape == result.orig_shape num_groups = hidden_size // group_size - assert result.scale.shape == (num_tokens, num_groups) + assert result.scale.shape == (*token_shape, num_groups) gate, up = x.split(hidden_size, dim=-1) silu_out = F.silu(gate) * up ref_out, ref_scales = per_token_group_quant_fp8( - silu_out, group_size=group_size, use_ue8m0=False + silu_out.reshape(-1, hidden_size), group_size=group_size, use_ue8m0=False ) + ref_out = ref_out.reshape(result.orig_shape) + ref_scales = ref_scales.reshape(*token_shape, num_groups) torch.testing.assert_close(result.scale, ref_scales, rtol=1e-5, atol=1e-5) ref_deq = ref_out.to(torch.float32) * ref_scales.repeat_interleave( - group_size, dim=1 + group_size, dim=-1 ) result_deq = result.data.to(torch.float32) * result.scale.repeat_interleave( - group_size, dim=1 + group_size, dim=-1 ) torch.testing.assert_close(ref_deq, result_deq, atol=5e-2, rtol=5e-2) diff --git a/tests/kernels/test_gate_linear_rocm_dispatch.py b/tests/kernels/test_gate_linear_rocm_dispatch.py index afa2e57f6a..06d69fc985 100644 --- a/tests/kernels/test_gate_linear_rocm_dispatch.py +++ b/tests/kernels/test_gate_linear_rocm_dispatch.py @@ -11,6 +11,8 @@ The ROCm branch is guarded on ``not bias`` because ``torch.mm`` has no bias term; a biased gate must fall back so the bias is not silently dropped. """ +import inspect + import pytest import torch @@ -168,3 +170,91 @@ def test_rocm_fp32_router_gemm_rejects_unsupported_configs( on_gfx950=on_gfx950, ) assert not gate.allow_fp32_router_gemm + + +# --------------------------------------------------------------------------- +# ROCm BF16x3 tier (fp32 router weights) +# --------------------------------------------------------------------------- +def _make_bf16x3_gate(monkeypatch, *, supported=True, **kwargs): + """Build a gate with the ROCm bf16x3 platform check mocked.""" + from vllm.model_executor.layers.fused_moe.router import ( + bf16x3_router_gemm_rocm as rocm_bf16x3, + ) + + monkeypatch.setattr(rocm_bf16x3, "platform_supported", lambda: supported) + kwargs.setdefault("params_dtype", torch.float32) + return _make_gate(monkeypatch, is_rocm=True, **kwargs) + + +def test_rocm_bf16x3_enabled_for_fp32_weight(monkeypatch): + gate = _make_bf16x3_gate(monkeypatch) + assert gate.allow_rocm_bf16x3_router_gemm + # Built by process_weights_after_loading, not lazily in forward(). + # Registered up front but still None, so named_buffers() skips it. + assert gate._bf16x3_weight is None + assert "_bf16x3_weight" in gate._buffers + + +def test_rocm_bf16x3_disabled_by_bias(monkeypatch): + # torch.mm has no bias term. + gate = _make_bf16x3_gate(monkeypatch, bias=True) + assert not gate.allow_rocm_bf16x3_router_gemm + + +def test_rocm_bf16x3_disabled_for_bf16_weight(monkeypatch): + # A bf16 weight belongs to the cuBLAS tier; there is nothing to split. + gate = _make_bf16x3_gate(monkeypatch, params_dtype=torch.bfloat16) + assert not gate.allow_rocm_bf16x3_router_gemm + + +def test_rocm_bf16x3_disabled_for_non_fp32_out_dtype(monkeypatch): + gate = _make_bf16x3_gate(monkeypatch, out_dtype=torch.bfloat16) + assert not gate.allow_rocm_bf16x3_router_gemm + + +def test_rocm_bf16x3_disabled_on_unsupported_arch(monkeypatch): + # Only gfx950 is benchmarked; gfx942 has a different fp32:bf16 MFMA ratio. + gate = _make_bf16x3_gate(monkeypatch, supported=False) + assert not gate.allow_rocm_bf16x3_router_gemm + assert not gate._rocm_bf16x3_weight_eligible + + +def test_rocm_bf16x3_no_split_built_when_tier_disabled(monkeypatch): + """An ineligible out_dtype must not allocate a split that is never read; + it is 1.5x the router weight on every layer.""" + gate = _make_bf16x3_gate(monkeypatch, out_dtype=torch.bfloat16) + gate.quant_method.process_weights_after_loading(gate) + assert gate._bf16x3_weight is None + + +def test_rocm_bf16x3_set_out_dtype_enables_tier(monkeypatch): + gate = _make_bf16x3_gate(monkeypatch, out_dtype=None) + assert not gate.allow_rocm_bf16x3_router_gemm + gate.set_out_dtype(torch.float32) + assert gate.allow_rocm_bf16x3_router_gemm + + +def test_rocm_bf16x3_dispatch_is_a_custom_op(): + """The num_tokens branch must stay opaque to Dynamo; see the op's impl.""" + assert hasattr(torch.ops.vllm, "rocm_bf16x3_router_gemm_dispatch") + x = torch.zeros(8192, 6144, dtype=torch.bfloat16, device="meta") + w = torch.zeros(128, 6144, dtype=torch.float32, device="meta") + split = torch.zeros(3, 128, 6144, dtype=torch.bfloat16, device="meta") + out = torch.ops.vllm.rocm_bf16x3_router_gemm_dispatch(x, w, split) + assert out.shape == (8192, 128) + assert out.dtype == torch.float32 + + +def test_rocm_bf16x3_is_reachable_for_low_m_shapes(monkeypatch): + """The gfx950 low-M fp32 tier claims these shapes too and returns + unconditionally, so bf16x3 is only reachable if checked first.""" + gate = _make_bf16x3_gate( + monkeypatch, input_size=6144, output_size=128, on_gfx950=True + ) + assert gate.allow_fp32_router_gemm + assert gate.allow_rocm_bf16x3_router_gemm + + source = inspect.getsource(type(gate).forward) + assert source.index("allow_rocm_bf16x3_router_gemm") < source.index( + "allow_fp32_router_gemm" + ) diff --git a/tests/kernels/ultraquant/test_config.py b/tests/kernels/ultraquant/test_config.py new file mode 100644 index 0000000000..14036b44e8 --- /dev/null +++ b/tests/kernels/ultraquant/test_config.py @@ -0,0 +1,36 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""CPU-only integration tests for the UltraQuant cache dtype.""" + +from vllm.v1.attention.backends.ultraquant_attn import ( + UltraQuantAttentionBackend, +) +from vllm.v1.attention.ops.flydsl_ultraquant_decode import ( + ultraquant_flydsl_decode_eligible, +) + + +def test_ultraquant_prefers_block_size_64(): + # FlyDSL D=256 decode geometry requires a 64-token KV block. + assert UltraQuantAttentionBackend.get_preferred_block_size(16) == 64 + assert UltraQuantAttentionBackend.get_preferred_block_size(128) == 64 + + +def test_ultraquant_flydsl_decode_routing(): + """CPU check of FlyDSL vs unified-Triton decode eligibility.""" + eligible = dict( + head_size=256, + num_kv_groups=8, + has_sinks=False, + sliding_window=None, + flydsl_loaded=True, + ) + assert ultraquant_flydsl_decode_eligible(**eligible) + for gqa in (6, 8, 16): + assert ultraquant_flydsl_decode_eligible(**{**eligible, "num_kv_groups": gqa}) + + assert not ultraquant_flydsl_decode_eligible(**{**eligible, "flydsl_loaded": False}) + assert not ultraquant_flydsl_decode_eligible(**{**eligible, "head_size": 128}) + assert not ultraquant_flydsl_decode_eligible(**{**eligible, "num_kv_groups": 4}) + assert not ultraquant_flydsl_decode_eligible(**{**eligible, "has_sinks": True}) + assert not ultraquant_flydsl_decode_eligible(**{**eligible, "sliding_window": 4096}) diff --git a/tests/kernels/ultraquant/test_decode_vs_reference.py b/tests/kernels/ultraquant/test_decode_vs_reference.py new file mode 100644 index 0000000000..9a8d431001 --- /dev/null +++ b/tests/kernels/ultraquant/test_decode_vs_reference.py @@ -0,0 +1,111 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Tests for the UltraQuant Triton unified-attention fallback.""" + +from __future__ import annotations + +import pytest +import torch + +from vllm.utils.torch_utils import set_random_seed +from vllm.v1.attention.ops.ultraquant.format import slot_size +from vllm.v1.attention.ops.ultraquant.reference import reference_ultraquant_attention +from vllm.v1.attention.ops.ultraquant.triton_store import ultraquant_store +from vllm.v1.attention.ops.ultraquant.triton_unified_attention import ( + ultraquant_unified_attention, +) + +pytestmark = pytest.mark.skipif( + not torch.cuda.is_available(), reason="requires CUDA/HIP device" +) + + +def _build_paged_cache(key, value, block_size): + batch_size, seq_len, num_kv_heads, head_size = key.shape + blocks_per_seq = (seq_len + block_size - 1) // block_size + kv_cache = torch.zeros( + batch_size * blocks_per_seq, + block_size, + num_kv_heads, + slot_size(head_size), + dtype=torch.uint8, + device=key.device, + ) + block_table = torch.arange( + batch_size * blocks_per_seq, + dtype=torch.int32, + device=key.device, + ).view(batch_size, blocks_per_seq) + slot_mapping = torch.arange( + batch_size * blocks_per_seq * block_size, + dtype=torch.int64, + device=key.device, + ).view(batch_size, -1)[:, :seq_len] + ultraquant_store( + key.reshape(-1, num_kv_heads, head_size), + value.reshape(-1, num_kv_heads, head_size), + kv_cache, + slot_mapping.reshape(-1), + ) + return kv_cache, block_table + + +def _attention_quality(actual: torch.Tensor, expected: torch.Tensor) -> dict: + a = actual.float().reshape(-1) + b = expected.float().reshape(-1) + diff = (a - b).abs() + return { + "cos_sim": torch.nn.functional.cosine_similarity(a, b, dim=0).item(), + "mean_abs": diff.mean().item(), + "max_abs": diff.max().item(), + "norm_ratio": (a.norm() / b.norm()).item(), + } + + +def _unified_decode(query, kv_cache, block_table, seq_lens, scale, sinks=None): + batch_size = query.shape[0] + return ultraquant_unified_attention( + query=query, + kv_cache=kv_cache, + block_table=block_table, + seq_lens=seq_lens, + query_start_loc=torch.arange( + batch_size + 1, dtype=torch.int32, device=query.device + ), + scale=scale, + max_query_len=1, + max_seq_len=int(seq_lens.max().item()), + sinks=sinks, + ) + + +@pytest.mark.parametrize("query_group_size", [6, 8]) +def test_unified_decode_d256_matches_reference(query_group_size): + """The D=256 fallback covers both target Qwen GQA layouts.""" + device = torch.device("cuda") + set_random_seed(0xD256 + query_group_size) + + B, Hk, N, D = 2, 1, 256, 256 + Hq = query_group_size * Hk + block_size = 32 + dtype = torch.bfloat16 + scale = D**-0.5 + + query = torch.randn(B, Hq, D, dtype=dtype, device=device) + key = torch.randn(B, N, Hk, D, dtype=dtype, device=device) + value = torch.randn_like(key) + kv_cache, block_table = _build_paged_cache(key, value, block_size) + seq_lens = torch.full((B,), N, dtype=torch.int32, device=device) + + actual = _unified_decode(query, kv_cache, block_table, seq_lens, scale) + expected = reference_ultraquant_attention( + query=query, + key=key, + value=value, + scale=scale, + ) + + quality = _attention_quality(actual, expected) + assert quality["cos_sim"] > 0.98, quality + assert quality["mean_abs"] < 0.08, quality + assert quality["max_abs"] < 0.8, quality diff --git a/tests/kernels/ultraquant/test_dequant_vs_reference.py b/tests/kernels/ultraquant/test_dequant_vs_reference.py new file mode 100644 index 0000000000..db56f56e29 --- /dev/null +++ b/tests/kernels/ultraquant/test_dequant_vs_reference.py @@ -0,0 +1,88 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Tests for the UltraQuant full-KV dequant kernel (continuation prefill). + +The kernel reconstructs cached K/V from FP4 codes plus UE8M0 group scales. +Because dequant is an exact table lookup times a power of two, the result +must match the reference to within the output dtype's rounding. +""" + +from __future__ import annotations + +import pytest +import torch + +from vllm.utils.torch_utils import set_random_seed +from vllm.v1.attention.ops.ultraquant.format import slot_size +from vllm.v1.attention.ops.ultraquant.reference import ( + ultraquant_dequant, + ultraquant_encode, +) +from vllm.v1.attention.ops.ultraquant.triton_dequant import ( + ultraquant_full_dequant_kv, +) +from vllm.v1.attention.ops.ultraquant.triton_store import ultraquant_store + +pytestmark = pytest.mark.skipif( + not torch.cuda.is_available(), reason="requires CUDA/HIP device" +) + + +@pytest.mark.parametrize("dtype", [torch.bfloat16, torch.float16]) +@pytest.mark.parametrize("head_dim", [128, 256]) +@pytest.mark.parametrize("num_kv_heads", [1, 4]) +@pytest.mark.parametrize("cached_len", [32, 96]) +def test_full_dequant_matches_reference(dtype, head_dim, num_kv_heads, cached_len): + device = torch.device("cuda") + set_random_seed(0xD4) + + block_size = 32 + alloc_len = cached_len + num_blocks = alloc_len // block_size + 1 + + key = torch.randn(alloc_len, num_kv_heads, head_dim, dtype=dtype, device=device) + value = torch.randn(alloc_len, num_kv_heads, head_dim, dtype=dtype, device=device) + + kv_cache = torch.zeros( + num_blocks, + block_size, + num_kv_heads, + slot_size(head_dim), + dtype=torch.uint8, + device=device, + ) + # Leave block 0 unused so the block table exercises a non-identity mapping. + slot_mapping = torch.arange( + block_size, block_size + alloc_len, dtype=torch.int64, device=device + ) + ultraquant_store(key, value, kv_cache, slot_mapping) + + block_table = torch.arange( + 1, num_blocks, dtype=torch.int32, device=device + ).unsqueeze(0) + + k_out = torch.zeros( + 1, num_kv_heads, alloc_len, head_dim, dtype=dtype, device=device + ) + v_out = torch.zeros_like(k_out) + ultraquant_full_dequant_kv( + kv_cache=kv_cache, + block_table=block_table, + k_out=k_out, + v_out=v_out, + alloc_len=alloc_len, + ) + torch.accelerator.synchronize() + + # K is stored Hadamard-rotated, V unrotated; the kernel returns them as + # stored, so the reference must not inverse-rotate either. + ref_k = ultraquant_dequant(ultraquant_encode(key, rotate=True), head_dim) + ref_v = ultraquant_dequant(ultraquant_encode(value, rotate=False), head_dim) + + got_k = k_out[0].transpose(0, 1).to(torch.float32) + got_v = v_out[0].transpose(0, 1).to(torch.float32) + + # Reference is fp32; the kernel rounds to the output dtype once. + tol = 1e-2 if dtype is torch.bfloat16 else 1e-3 + torch.testing.assert_close(got_k, ref_k.to(torch.float32), rtol=tol, atol=tol) + torch.testing.assert_close(got_v, ref_v.to(torch.float32), rtol=tol, atol=tol) diff --git a/tests/kernels/ultraquant/test_flydsl_decode_d256.py b/tests/kernels/ultraquant/test_flydsl_decode_d256.py new file mode 100644 index 0000000000..33a1a8273f --- /dev/null +++ b/tests/kernels/ultraquant/test_flydsl_decode_d256.py @@ -0,0 +1,149 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Correctness tests for the production UltraQuant D=256 FlyDSL decode.""" + +import pytest +import torch + +from vllm.utils.torch_utils import set_random_seed +from vllm.v1.attention.ops.flydsl_ultraquant_decode import ( + flydsl_ultraquant_decode_attention, + is_ultraquant_flydsl_available, + ultraquant_flydsl_decode_eligible, +) +from vllm.v1.attention.ops.ultraquant.format import slot_size +from vllm.v1.attention.ops.ultraquant.reference import ( + hadamard_matrix, + reference_ultraquant_attention, +) +from vllm.v1.attention.ops.ultraquant.triton_store import ultraquant_store + +pytestmark = pytest.mark.skipif( + not torch.cuda.is_available(), reason="requires CUDA/HIP device" +) + + +def _build_paged_cache(key, value, block_size): + batch_size, seq_len, num_kv_heads, head_size = key.shape + blocks_per_seq = (seq_len + block_size - 1) // block_size + kv_cache = torch.zeros( + batch_size * blocks_per_seq, + block_size, + num_kv_heads, + slot_size(head_size), + dtype=torch.uint8, + device=key.device, + ) + block_table = torch.arange( + batch_size * blocks_per_seq, + dtype=torch.int32, + device=key.device, + ).view(batch_size, blocks_per_seq) + slot_mapping = torch.arange( + batch_size * blocks_per_seq * block_size, + dtype=torch.int64, + device=key.device, + ).view(batch_size, -1)[:, :seq_len] + ultraquant_store( + key.reshape(-1, num_kv_heads, head_size), + value.reshape(-1, num_kv_heads, head_size), + kv_cache, + slot_mapping.reshape(-1), + ) + return kv_cache, block_table + + +@pytest.mark.parametrize( + ("batch_size", "query_group_size", "block_size", "contiguous_query"), + [ + (1, 8, 32, True), + (8, 8, 64, True), + (9, 8, 256, True), + (1, 16, 32, True), + (1, 6, 16, True), + (1, 6, 32, True), + (2, 6, 64, True), + (1, 6, 128, True), + (1, 6, 256, True), + (1, 6, 32, False), + ], +) +def test_flydsl_decode_matches_reference( + batch_size, + query_group_size, + block_size, + contiguous_query, +): + if not ultraquant_flydsl_decode_eligible( + head_size=256, + num_kv_groups=query_group_size, + has_sinks=False, + sliding_window=None, + flydsl_loaded=is_ultraquant_flydsl_available(), + ): + pytest.skip("UltraQuant D=256 FlyDSL kernel is unavailable") + + set_random_seed(0x57A1D + batch_size + block_size + query_group_size) + device = torch.device("cuda") + dtype = torch.bfloat16 + num_kv_heads = 1 + head_size = 256 + seq_len = 2048 + scale = head_size**-0.5 + + key = torch.randn( + batch_size, + seq_len, + num_kv_heads, + head_size, + dtype=dtype, + device=device, + ) + value = torch.randn_like(key) + if contiguous_query: + qkv = torch.randn( + batch_size, + query_group_size + 2 * num_kv_heads, + head_size, + dtype=dtype, + device=device, + ) + query = qkv[:, :query_group_size] + else: + query_storage = torch.randn( + batch_size, + query_group_size, + head_size * 2, + dtype=dtype, + device=device, + ) + query = query_storage[..., ::2] + kv_cache, block_table = _build_paged_cache(key, value, block_size) + seq_lens = torch.full((batch_size,), seq_len, dtype=torch.int32, device=device) + hadamard = hadamard_matrix(head_size, device) + + actual = flydsl_ultraquant_decode_attention( + query=query, + kv_cache=kv_cache, + block_table=block_table, + seq_lens=seq_lens, + scale=scale, + PiT=hadamard, + max_seq_len=seq_len, + ) + expected = reference_ultraquant_attention( + query=query, + key=key, + value=value, + scale=scale, + ) + + actual_f32 = actual.float() + expected_f32 = expected.float() + cosine = torch.nn.functional.cosine_similarity( + actual_f32.flatten(), expected_f32.flatten(), dim=0 + ) + error = (actual_f32 - expected_f32).abs() + assert cosine > 0.995 + assert error.mean() < 0.05 + assert error.max() < 0.8 diff --git a/tests/kernels/ultraquant/test_seam_contract.py b/tests/kernels/ultraquant/test_seam_contract.py new file mode 100644 index 0000000000..a5f2116797 --- /dev/null +++ b/tests/kernels/ultraquant/test_seam_contract.py @@ -0,0 +1,106 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""UltraQuant inherits TurboQuant's forward/do_kv_cache_update orchestration. + +UltraQuant only overrides the store/decode/prefill seams, so a TurboQuant-side +change to a seam signature, to an inherited helper, or to the ``layer._tq_*`` +attributes ``forward`` reads will break UltraQuant at serve time without +touching any UltraQuant kernel test. These checks pin that contract. +""" + +import ast +import inspect +import textwrap + +from vllm.v1.attention.backends import ultraquant_attn +from vllm.v1.attention.backends.turboquant_attn import TurboQuantAttentionImpl +from vllm.v1.attention.backends.ultraquant_attn import UltraQuantAttentionImpl + +_SEAMS = ("_ensure_on_device", "_store_kv", "_decode_attention", "_prefill_attention") + + +def _method_ast(cls: type, name: str) -> ast.AST: + return ast.parse(textwrap.dedent(inspect.getsource(getattr(cls, name)))) + + +def _tq_attrs_read(cls: type, name: str) -> set[str]: + """``_tq_*`` attribute names read (not assigned) in ``cls.name``.""" + tree = _method_ast(cls, name) + assigned = { + t.attr + for node in ast.walk(tree) + if isinstance(node, ast.Assign) + for t in node.targets + if isinstance(t, ast.Attribute) + } + read = { + node.attr + for node in ast.walk(tree) + if isinstance(node, ast.Attribute) and node.attr.startswith("_tq_") + } + return read - assigned + + +def _tq_attrs_written(cls: type, name: str) -> set[str]: + tree = _method_ast(cls, name) + return { + t.attr + for node in ast.walk(tree) + if isinstance(node, ast.Assign) + for t in node.targets + if isinstance(t, ast.Attribute) and t.attr.startswith("_tq_") + } + + +def test_ultraquant_matches_turboquant_seam_contract(): + # 1. Each overridden seam accepts every parameter the base declares, so any + # call the inherited orchestration makes stays valid for UltraQuant. + for name in _SEAMS: + base = list( + inspect.signature(getattr(TurboQuantAttentionImpl, name)).parameters + ) + override = list( + inspect.signature(getattr(UltraQuantAttentionImpl, name)).parameters + ) + assert override[: len(base)] == base, ( + f"UltraQuantAttentionImpl.{name} no longer accepts the arguments " + f"TurboQuantAttentionImpl passes: base={base} override={override}" + ) + + # 2. Keyword arguments UltraQuant passes to *inherited* TurboQuant helpers + # must exist on those helpers (e.g. _flash_attn_varlen). + own = vars(UltraQuantAttentionImpl) + module = ast.parse(inspect.getsource(ultraquant_attn)) + for node in ast.walk(module): + if not isinstance(node, ast.Call): + continue + fn = node.func + if not ( + isinstance(fn, ast.Attribute) + and isinstance(fn.value, ast.Name) + and fn.value.id == "self" + ): + continue + target = getattr(TurboQuantAttentionImpl, fn.attr, None) + if target is None or fn.attr in own or not callable(target): + continue + params = inspect.signature(target).parameters + if any(p.kind is inspect.Parameter.VAR_KEYWORD for p in params.values()): + continue + for kw in node.keywords: + if kw.arg is None: + continue + assert kw.arg in params, ( + f"UltraQuant calls inherited {fn.attr}(..., {kw.arg}=...) but " + f"TurboQuantAttentionImpl.{fn.attr} accepts {list(params)}" + ) + + # 3. Every layer._tq_* attribute the inherited forward reads is set by + # UltraQuant's _ensure_on_device. + required = _tq_attrs_read(TurboQuantAttentionImpl, "forward") + provided = _tq_attrs_written(UltraQuantAttentionImpl, "_ensure_on_device") + assert required, "expected inherited forward to read layer._tq_* attributes" + assert required <= provided, ( + f"UltraQuant._ensure_on_device does not set {sorted(required - provided)}, " + f"which the inherited TurboQuant forward reads" + ) diff --git a/tests/kernels/ultraquant/test_store_vs_reference.py b/tests/kernels/ultraquant/test_store_vs_reference.py new file mode 100644 index 0000000000..2f02e9578c --- /dev/null +++ b/tests/kernels/ultraquant/test_store_vs_reference.py @@ -0,0 +1,149 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Tests for the Triton UltraQuant store kernel. + +The Triton kernel must produce bit-identical codes and scale bytes vs +the PyTorch reference (``reference.ultraquant_encode``). +""" + +from __future__ import annotations + +import pytest +import torch + +from vllm.utils.torch_utils import set_random_seed +from vllm.v1.attention.ops.ultraquant.format import ( + k_codes_offset, + k_scales_offset, + n_groups, + slot_size, + v_codes_offset, + v_scales_offset, +) +from vllm.v1.attention.ops.ultraquant.reference import ultraquant_encode +from vllm.v1.attention.ops.ultraquant.triton_store import ultraquant_store + +pytestmark = pytest.mark.skipif( + not torch.cuda.is_available(), reason="requires CUDA/HIP device" +) + + +def _make_cache( + num_blocks: int, + block_size: int, + num_kv_heads: int, + head_dim: int, + device: torch.device, +) -> torch.Tensor: + slot_b = slot_size(head_dim) + return torch.zeros( + num_blocks, block_size, num_kv_heads, slot_b, dtype=torch.uint8, device=device + ) + + +def _read_codes_and_scales( + kv_cache: torch.Tensor, + slot_mapping: torch.Tensor, + num_kv_heads: int, + head_dim: int, +): + """Pull (k_codes, k_scale_bytes, v_codes, v_scale_bytes) for each + (token, head) from the AoS slot layout. Scales come back as uint8 + (one E8M0 byte per group).""" + block_size = kv_cache.shape[1] + N = slot_mapping.shape[0] + Gk = n_groups(head_dim) + + k_codes_off = k_codes_offset(head_dim) + k_scales_off = k_scales_offset(head_dim) + v_codes_off = v_codes_offset(head_dim) + v_scales_off = v_scales_offset(head_dim) + codes_bytes = head_dim // 2 + + k_codes = torch.zeros( + N, num_kv_heads, codes_bytes, dtype=torch.uint8, device=kv_cache.device + ) + k_scale_bytes = torch.zeros( + N, num_kv_heads, Gk, dtype=torch.uint8, device=kv_cache.device + ) + v_codes = torch.zeros_like(k_codes) + v_scale_bytes = torch.zeros_like(k_scale_bytes) + + for ti in range(N): + slot = int(slot_mapping[ti].item()) + if slot < 0: + continue + blk = slot // block_size + off = slot % block_size + for h in range(num_kv_heads): + slot_bytes = kv_cache[blk, off, h] # [slot_size_aligned] uint8 + + k_codes[ti, h] = slot_bytes[k_codes_off : k_codes_off + codes_bytes] + k_scale_bytes[ti, h] = slot_bytes[k_scales_off : k_scales_off + Gk] + v_codes[ti, h] = slot_bytes[v_codes_off : v_codes_off + codes_bytes] + v_scale_bytes[ti, h] = slot_bytes[v_scales_off : v_scales_off + Gk] + + return k_codes, k_scale_bytes, v_codes, v_scale_bytes + + +@pytest.mark.parametrize("dtype", [torch.bfloat16, torch.float16]) +@pytest.mark.parametrize("head_dim", [128, 256]) +@pytest.mark.parametrize("num_kv_heads", [1, 4]) +@pytest.mark.parametrize("num_tokens", [1, 16, 17]) +def test_store_matches_reference(dtype, head_dim, num_kv_heads, num_tokens): + """Triton store output (codes + E8M0 scale bytes) matches the reference + bit-exactly.""" + device = torch.device("cuda") + set_random_seed(0xF8) + + block_size = 32 + num_blocks = max(2, (num_tokens + block_size - 1) // block_size + 1) + + key = torch.randn(num_tokens, num_kv_heads, head_dim, dtype=dtype, device=device) + value = torch.randn(num_tokens, num_kv_heads, head_dim, dtype=dtype, device=device) + + slot_mapping = torch.arange(num_tokens, dtype=torch.int64, device=device) + if num_tokens >= 4: + slot_mapping[1] = -1 # skip-path test + + kv_cache = _make_cache(num_blocks, block_size, num_kv_heads, head_dim, device) + + ultraquant_store(key, value, kv_cache, slot_mapping) + torch.accelerator.synchronize() + + ref_k = ultraquant_encode(key, rotate=True) + ref_v = ultraquant_encode(value, rotate=False) + + k_codes, k_scale_bytes, v_codes, v_scale_bytes = _read_codes_and_scales( + kv_cache, slot_mapping, num_kv_heads, head_dim + ) + + if num_tokens >= 4: + assert (kv_cache[0, 1, :, :] == 0).all(), ( + "skipped slot's physical cache region was unexpectedly written" + ) + + for ti in range(num_tokens): + slot = int(slot_mapping[ti].item()) + if slot < 0: + continue + + assert torch.equal(k_codes[ti], ref_k.codes_packed[ti]), ( + f"K codes mismatch at token {ti}\n" + f"got: {k_codes[ti].cpu().numpy().tolist()[:16]}\n" + f"expected: {ref_k.codes_packed[ti].cpu().numpy().tolist()[:16]}" + ) + assert torch.equal(k_scale_bytes[ti], ref_k.scale_bytes[ti]), ( + f"K scale bytes mismatch at token {ti}\n" + f"got: {k_scale_bytes[ti].cpu().numpy().tolist()}\n" + f"expected: {ref_k.scale_bytes[ti].cpu().numpy().tolist()}" + ) + + assert torch.equal(v_codes[ti], ref_v.codes_packed[ti]), ( + f"V codes mismatch at token {ti}" + ) + assert torch.equal(v_scale_bytes[ti], ref_v.scale_bytes[ti]), ( + f"V scale bytes mismatch at token {ti}\n" + f"got: {v_scale_bytes[ti].cpu().numpy().tolist()}\n" + f"expected: {ref_v.scale_bytes[ti].cpu().numpy().tolist()}" + ) diff --git a/tests/lora/test_llama_tp.py b/tests/lora/test_llama_tp.py index 42f6ddc2f6..9ccb2aec25 100644 --- a/tests/lora/test_llama_tp.py +++ b/tests/lora/test_llama_tp.py @@ -1,18 +1,14 @@ # SPDX-License-Identifier: Apache-2.0 # SPDX-FileCopyrightText: Copyright contributors to the vLLM project -import subprocess -import sys import pytest import vllm import vllm.config -from vllm import LLM from vllm.lora.request import LoRARequest -from vllm.model_executor.model_loader.tensorizer import TensorizerConfig from vllm.platforms import current_platform -from ..utils import VLLM_PATH, create_new_process_for_each_test, multi_gpu_test +from ..utils import create_new_process_for_each_test, multi_gpu_test PROMPT_TEMPLATE = """<|eot_id|><|start_header_id|>user<|end_header_id|> I want you to act as a SQL terminal in front of an example database, you need only to return the sql command to me.Below is an instruction that describes a task, Write a response that appropriately completes the request. @@ -40,7 +36,6 @@ def do_sample( llm: vllm.LLM, lora_path: str, lora_id: int, - tensorizer_config_dict: dict | None = None, ) -> list[str]: prompts = [ PROMPT_TEMPLATE.format(context="How many candidates are there?"), @@ -56,27 +51,11 @@ def do_sample( sampling_params = vllm.SamplingParams( temperature=0, max_tokens=64, stop=["<|im_end|>"] ) - if tensorizer_config_dict is not None: - outputs = llm.generate( - prompts, - sampling_params, - lora_request=LoRARequest( - str(lora_id), - lora_id, - lora_path, - tensorizer_config_dict=tensorizer_config_dict, - ) - if lora_id - else None, - ) - else: - outputs = llm.generate( - prompts, - sampling_params, - lora_request=LoRARequest(str(lora_id), lora_id, lora_path) - if lora_id - else None, - ) + outputs = llm.generate( + prompts, + sampling_params, + lora_request=LoRARequest(str(lora_id), lora_id, lora_path) if lora_id else None, + ) lora_request = LoRARequest(str(lora_id), lora_id, lora_path) if lora_id else None generated_texts: list[str] = [] for output in outputs: @@ -94,31 +73,13 @@ def do_sample( return generated_texts -def generate_and_test( - llm, llama32_lora_files, tensorizer_config_dict: dict | None = None -): +def generate_and_test(llm, llama32_lora_files): print("lora adapter created") print("lora 1") - assert ( - do_sample( - llm, - llama32_lora_files, - tensorizer_config_dict=tensorizer_config_dict, - lora_id=1, - ) - == EXPECTED_LORA_OUTPUT - ) + assert do_sample(llm, llama32_lora_files, lora_id=1) == EXPECTED_LORA_OUTPUT print("lora 2") - assert ( - do_sample( - llm, - llama32_lora_files, - tensorizer_config_dict=tensorizer_config_dict, - lora_id=2, - ) - == EXPECTED_LORA_OUTPUT - ) + assert do_sample(llm, llama32_lora_files, lora_id=2) == EXPECTED_LORA_OUTPUT print("removing lora") @@ -168,79 +129,3 @@ def test_llama_lora_tp4_fully_sharded_loras(llama32_lora_files): fully_sharded_loras=True, ) generate_and_test(llm, llama32_lora_files) - - -@multi_gpu_test(num_gpus=2) -def test_tp2_serialize_and_deserialize_lora( - tmp_path, - llama32_lora_files, -): - # Run the tensorizing of the LoRA adapter and the model in a subprocess - # to guarantee cleanup - - tp_size = 2 - model_name = "model-rank-%03d.tensors" - - model_ref = MODEL_PATH - lora_path = llama32_lora_files - suffix = "test" - try: - result = subprocess.run( - [ - sys.executable, - f"{VLLM_PATH}/examples/features/tensorize_vllm_model.py", - "--model", - MODEL_PATH, - "--lora-path", - lora_path, - "--tensor-parallel-size", - str(tp_size), - "serialize", - "--serialized-directory", - str(tmp_path), - "--suffix", - suffix, - "--serialization-kwargs", - '{"limit_cpu_concurrency": 4}', - ], - check=True, - capture_output=True, - text=True, - ) - except subprocess.CalledProcessError as e: - print("Tensorizing failed.") - print("STDOUT:\n", e.stdout) - print("STDERR:\n", e.stderr) - raise - - print("STDOUT:\n", result.stdout) - - model_uri = tmp_path / "vllm" / model_ref / suffix / model_name - tensorizer_config = TensorizerConfig(tensorizer_uri=str(model_uri)) - - loaded_llm = LLM( - model=model_ref, - load_format="tensorizer", - enable_lora=True, - enforce_eager=True, - model_loader_extra_config=tensorizer_config, - max_num_seqs=7, - max_model_len=1024, - tensor_parallel_size=2, - max_loras=2, - # Leave headroom for LoRA adapter loading and Triton JIT - # compilation, which can allocate GPU memory concurrently - # during the first inference step. - gpu_memory_utilization=0.85, - ) - - tc_as_dict = tensorizer_config.to_serializable() - - print("lora adapter created") - print("lora 1") - assert ( - do_sample( - loaded_llm, llama32_lora_files, tensorizer_config_dict=tc_as_dict, lora_id=1 - ) - == EXPECTED_LORA_OUTPUT - ) diff --git a/tests/lora/test_lora_manager.py b/tests/lora/test_lora_manager.py index 43fa1975bd..ffb064ebfb 100644 --- a/tests/lora/test_lora_manager.py +++ b/tests/lora/test_lora_manager.py @@ -16,10 +16,12 @@ from vllm.lora.layers import ( ReplicatedLinearWithLoRA, RowParallelLinearWithLoRA, ) +from vllm.lora.layers.fused_moe import FusedMoE3DWithLoRA, FusedMoEWithLoRA from vllm.lora.lora_model import LoRAModel from vllm.lora.lora_weights import LoRALayerWeights, PackedLoRALayerWeights from vllm.lora.model_manager import ( DEFAULT_LANGUAGE_WRAPPER_KEY, + AdapterLRUCache, LoRAMapping, LoRAModelManager, LRUCacheLoRAModelManager, @@ -47,6 +49,68 @@ DEVICES = ( DEFAULT_DTYPE = torch.get_default_dtype() +@pytest.mark.parametrize( + "mixed,declared_3d", [(False, False), (False, True), (True, False)] +) +def test_reject_fused_moe_lora_before_registration(mixed, declared_3d): + """Missing format flags should produce guidance without caching the adapter.""" + module_name = "model.layers.0.mlp.experts" + manager = LoRAModelManager.__new__(LoRAModelManager) + manager.lora_config = LoRAConfig(max_cpu_loras=1) + manager._registered_adapters = AdapterLRUCache(1, manager.deactivate_adapter) + manager._enable_mixed_moe_lora_format = mixed + manager.is_pooling_model = False + manager.modules = {module_name: FusedMoEWithLoRA.__new__(FusedMoEWithLoRA)} + adapter = LoRAModel( + 1, + 8, + { + module_name: LoRALayerWeights( + module_name, 8, 16, torch.zeros(16, 4), torch.zeros(4, 16) + ) + }, + ) + adapter.is_3d_lora_weight = declared_3d + + with pytest.raises(ValueError, match="enable_mixed_moe_lora_format=True") as exc: + manager.add_adapter(adapter) + assert "is_3d_lora_weight=True" in str(exc.value) + assert module_name in str(exc.value) + assert manager.list_adapters() == {} + + +@pytest.mark.parametrize( + "wrapper,mixed,declared_3d,weight_name", + [ + (FusedMoE3DWithLoRA, False, False, "experts"), + (FusedMoEWithLoRA, True, True, "experts"), + (FusedMoEWithLoRA, False, False, "experts.0.down_proj"), + (FusedMoEWithLoRA, True, False, "experts.0.down_proj"), + (FusedMoEWithLoRA, False, False, "q_proj"), + ], +) +def test_moe_format_validation_allows_supported_layouts( + wrapper, mixed, declared_3d, weight_name +): + """Preserve native 3D, mixed-format, per-expert and attention-only adapters.""" + manager = LoRAModelManager.__new__(LoRAModelManager) + manager._enable_mixed_moe_lora_format = mixed + manager.is_pooling_model = False + manager.modules = {"experts": wrapper.__new__(wrapper)} + adapter = LoRAModel( + 1, + 8, + { + weight_name: LoRALayerWeights( + weight_name, 8, 16, torch.zeros(8, 4), torch.zeros(4, 8) + ) + }, + ) + adapter.is_3d_lora_weight = declared_3d + + manager._validate_moe_lora_format(adapter) + + @pytest.mark.parametrize("device", DEVICES) def test_from_lora_tensors(qwen3_lora_files, device): tensors = load_file(os.path.join(qwen3_lora_files, "adapter_model.safetensors")) diff --git a/tests/lora/test_peft_helper.py b/tests/lora/test_peft_helper.py index 9fb52ea727..bb1246144b 100644 --- a/tests/lora/test_peft_helper.py +++ b/tests/lora/test_peft_helper.py @@ -24,6 +24,25 @@ ERROR_CASES = [ ), ("test_rank_zero", {"r": 0}, "must be a positive integer"), ("test_rank_negative", {"r": -8}, "must be a positive integer"), + ("test_lora_bias", {"lora_bias": True}, "does not support LoRA bias"), + ("test_pissa", {"init_lora_weights": "pissa"}, "init_lora_weights='pissa'"), + ( + "test_pissa_niter", + {"init_lora_weights": "pissa_niter_4"}, + "init_lora_weights='pissa_niter_4'", + ), + ("test_olora", {"init_lora_weights": "olora"}, "init_lora_weights='olora'"), + ("test_corda", {"init_lora_weights": "corda"}, "init_lora_weights='corda'"), + ("test_loftq", {"init_lora_weights": "loftq"}, "init_lora_weights='loftq'"), + ( + "test_unknown_init", + {"init_lora_weights": "future_init"}, + "init_lora_weights='future_init'", + ), + ("test_alora", {"alora_invocation_tokens": [1, 2]}, "Activated LoRA"), + ("test_layer_replication", {"layer_replication": [[0, 2]]}, "layer_replication"), + ("test_bdlora", {"use_bdlora": {"nblocks": 2}}, "BD-LoRA"), + ("test_qalora", {"use_qalora": True}, "QALoRA"), ] @@ -114,3 +133,23 @@ def test_peft_helper_invalid_rank_direct(bad_rank: int): """ with pytest.raises(ValueError, match="must be a positive integer"): PEFTHelper(r=bad_rank, lora_alpha=16, target_modules=["q_proj"]) + + +@pytest.mark.parametrize( + "init_lora_weights", + [True, False, "gaussian", "eva", "orthogonal", "mica", "lora_ga"], +) +def test_peft_helper_init_lora_weights_supported(init_lora_weights): + """A saved adapter with these inits loads in PEFT as a plain LoRA.""" + lora_config = LoRAConfig(max_lora_rank=16, max_cpu_loras=3, max_loras=2) + PEFTHelper.from_dict( + { + "r": 8, + "lora_alpha": 16, + "target_modules": ["q_proj"], + "init_lora_weights": init_lora_weights, + "alora_invocation_tokens": None, + "layer_replication": None, + "use_bdlora": None, + } + ).validate_legal(lora_config) diff --git a/tests/lora/test_utils.py b/tests/lora/test_utils.py index 4a9ba66069..0b1be49f7e 100644 --- a/tests/lora/test_utils.py +++ b/tests/lora/test_utils.py @@ -106,6 +106,8 @@ def test_parse_fine_tuned_lora_name_invalid(): fixture = { "base_model.weight", "base_model.model.weight", + "base_model.model.model.layers.9.mlp.down_proj.lora_A.bias", + "base_model.model.model.layers.9.mlp.down_proj.lora_B.bias", } for name in fixture: with pytest.raises(ValueError, match="unsupported LoRA weight"): diff --git a/tests/model_executor/layers/test_fused_shared_expert.py b/tests/model_executor/layers/test_fused_shared_expert.py index 6721a4c63b..0fcc2540e2 100644 --- a/tests/model_executor/layers/test_fused_shared_expert.py +++ b/tests/model_executor/layers/test_fused_shared_expert.py @@ -12,13 +12,16 @@ from unittest.mock import patch import pytest import torch from torch import nn +from transformers import Qwen3_5MoeTextConfig import vllm.config as vllm_config_module -from vllm.config import VllmConfig, set_current_vllm_config +from vllm.config import ParallelConfig, VllmConfig, set_current_vllm_config +from vllm.config.quantization import QuantizationConfigArgs from vllm.model_executor.layers.fused_moe import utils as fused_moe_utils from vllm.model_executor.layers.fused_moe.layer import determine_expert_counts -from vllm.model_executor.layers.quantization.base_config import QuantizationConfig +from vllm.model_executor.layers.quantization.experts_int8 import ExpertsInt8Config from vllm.model_executor.layers.quantization.fp8 import Fp8Config +from vllm.model_executor.layers.quantization.online.base import OnlineQuantizationConfig from vllm.model_executor.layers.quantization.quark.quark import QuarkConfig from vllm.model_executor.layers.quantization.utils.config_utils import ( get_quark_ocp_mx_group_size, @@ -32,7 +35,6 @@ from vllm.models.deepseek_v4 import quant_config as deepseek_v4_quant_config from vllm.models.minimax_m3.amd import model as minimax_m3_model from vllm.platforms import current_platform from vllm.transformers_utils.configs.minimax_m3 import MiniMaxM3TextConfig -from vllm.transformers_utils.configs.qwen3_5_moe import Qwen3_5MoeTextConfig pytestmark = pytest.mark.skipif( current_platform.is_xpu(), @@ -141,8 +143,10 @@ def get_deepseek_v4_quark_config(exclude: list[str]) -> dict[str, Any]: return quantization_config -def _stub_quant_config() -> QuantizationConfig: - return cast(QuantizationConfig, object()) +class _StubGlm5NextAttention(nn.Module): + def __init__(self, *args: Any, **kwargs: Any) -> None: + super().__init__() + self.o_proj = SimpleNamespace(reduce_results=True) def get_fse_test_model_config( @@ -279,6 +283,33 @@ def get_fse_test_model_config( quantization_config=quantization_config, ) return config, Glm4MoeModel + if model_type == "glm5_next": + from transformers import Glm5NextTextConfig + + from vllm.models.glm5next.common.model import Glm5NextModel + + config = Glm5NextTextConfig( + vocab_size=256, + hidden_size=128, + intermediate_size=32, + moe_intermediate_size=32, + num_hidden_layers=1, + num_attention_heads=1, + num_key_value_heads=1, + n_routed_experts=2, + n_shared_experts=1, + num_experts_per_tok=1, + layer_types=["linear_attention"], + mlp_layer_types=["sparse"], + index_topk=1, + index_kpool=1, + pad_token_id=None, + scoring_func="sigmoid", + topk_method="noaux_tc", + mhc=False, + quantization_config=quantization_config, + ) + return config, Glm5NextModel raise ValueError(f"Unsupported FSE test model: {model_type}") @@ -307,7 +338,7 @@ def test_resolve_layer_fused_shared_expert_skips_compatibility_when_disabled( ) assert not fused_moe_utils.resolve_layer_fused_shared_expert( - _stub_quant_config(), "model.layers.0.mlp" + ExpertsInt8Config(), "model.layers.0.mlp" ) @@ -322,7 +353,7 @@ def test_resolve_layer_fused_shared_expert_normalizes_unavailable_aiter( assert ( fused_moe_utils.resolve_layer_fused_shared_expert( - _stub_quant_config(), "model.layers.0.mlp" + ExpertsInt8Config(), "model.layers.0.mlp" ) is False ) @@ -331,7 +362,7 @@ def test_resolve_layer_fused_shared_expert_normalizes_unavailable_aiter( def test_resolve_layer_fused_shared_expert_passes_module_prefixes( monkeypatch: pytest.MonkeyPatch, ) -> None: - quant_config = _stub_quant_config() + quant_config = ExpertsInt8Config() monkeypatch.setattr( fused_moe_utils.rocm_aiter_ops, "is_fusion_moe_shared_experts_enabled", @@ -380,7 +411,7 @@ def test_resolve_layer_fused_shared_expert_rejects_incompatible_quantization( ) assert not fused_moe_utils.resolve_layer_fused_shared_expert( - _stub_quant_config(), "model.layers.0.mlp" + ExpertsInt8Config(), "model.layers.0.mlp" ) assert "shared experts are excluded" in caplog.text @@ -390,6 +421,7 @@ def test_deepseek_v4_shared_expert_fse_uses_mtp_quantization_config_prefix( ) -> None: class DeepseekV4Config: expert_dtype = "fp4" + online_quantization_config = None def _is_quark_mxfp4_ocp(self, hf_config: object) -> bool: return True @@ -796,7 +828,7 @@ def test_is_model_fused_shared_expert_compatible() -> None: @pytest.mark.parametrize( "model_type", - ["minimax_m3", "deepseek_v4", "qwen3_5", "glm4_moe", "deepseek_v2"], + ["minimax_m3", "deepseek_v4", "qwen3_5", "glm4_moe", "deepseek_v2", "glm5_next"], ) @pytest.mark.parametrize( ("use_fse", "exclude"), @@ -850,6 +882,16 @@ def test_models_fse_init( ) else: vllm_config.quant_config = QuarkConfig(quantization_config) + if model_type == "glm5_next": + from vllm.models.glm5next.common import model as glm5_next_model + + for attention in ("Glm5NextLinearAttention", "Glm5NextMLAAttention"): + monkeypatch.setattr(glm5_next_model, attention, _StubGlm5NextAttention) + # The gate allows fusion only at TP4/TP8 on gfx950, and this test + # runs at TP1; the gate has its own test. + monkeypatch.setattr( + glm5_next_model, "_fused_shared_experts_tuned", lambda _: True + ) import vllm.envs as envs from vllm._aiter_ops import rocm_aiter_ops @@ -902,6 +944,10 @@ def test_models_fse_init( ) mtp = DeepSeekV4MTP(vllm_config=vllm_config) assert model.is_fused_shared_expert_enabled is (fse_enabled and not exclude) + if model_type == "glm5_next": + assert (model.layers[0].mlp.shared_experts is None) is ( + fse_enabled and not exclude + ) # The dummy quant config here uses mixed mxfp4/fp8 for experts/shared_expert # so should just raise a warning. @@ -926,6 +972,74 @@ def test_models_fse_init( rocm_aiter_ops.refresh_env_variables() +@pytest.mark.parametrize( + ("setting", "value", "reason"), + [ + ("tensor_parallel_size", 4, None), + ("tensor_parallel_size", 8, None), + ("tensor_parallel_size", 2, "tensor_parallel_size is 2"), + ("data_parallel_size", 2, "data_parallel_size is 2"), + ("prefill_context_parallel_size", 2, "prefill_context_parallel_size is 2"), + ("enable_expert_parallel", True, "expert parallelism is enabled"), + ("on_gfx950", False, "the GPU is not gfx950"), + ], +) +def test_glm5_next_fuses_shared_experts_only_in_tuned_setups( + monkeypatch: pytest.MonkeyPatch, + setting: str, + value: object, + reason: str | None, +) -> None: + from vllm.models.glm5next.common import model as glm5_next_model + + parallel_config = SimpleNamespace( + tensor_parallel_size=4, + data_parallel_size=1, + prefill_context_parallel_size=1, + enable_expert_parallel=False, + ) + on_gfx950 = value if setting == "on_gfx950" else True + if setting != "on_gfx950": + setattr(parallel_config, setting, value) + monkeypatch.setattr("vllm.platforms.rocm.on_gfx950", lambda: on_gfx950) + + with patch.object(glm5_next_model.logger, "warning_once") as warning: + tuned = glm5_next_model._fused_shared_experts_tuned( + cast(ParallelConfig, parallel_config) + ) + + assert tuned is (reason is None) + if reason is None: + warning.assert_not_called() + else: + warning.assert_called_once() + assert warning.call_args.args[1] == reason + + +def test_glm5_next_fused_shared_expert_loads_into_the_shared_slot() -> None: + from vllm.models.glm5next.common import model as glm5_next_model + + assert glm5_next_model._num_fused_shared_experts(1, True) == 1 + assert glm5_next_model._num_fused_shared_experts(1, False) == 0 + assert glm5_next_model._num_fused_shared_experts(None, True) == 0 + with pytest.raises(NotImplementedError, match="only 1 shared expert"): + glm5_next_model._num_fused_shared_experts(2, True) + + for shared, fused in [ + ( + "model.layers.3.mlp.shared_experts.gate_proj.weight_scale_inv", + "model.layers.3.mlp.experts.288.gate_proj.weight_scale_inv", + ), + ( + "model.layers.45.mtp_block.mlp.shared_experts.down_proj.weight", + "model.layers.45.mtp_block.mlp.experts.288.down_proj.weight", + ), + ]: + assert glm5_next_model._fused_shared_expert_name(shared, 288) == fused + routed = "model.layers.3.mlp.experts.7.down_proj.weight" + assert glm5_next_model._fused_shared_expert_name(routed, 288) == routed + + @pytest.mark.parametrize( ("exclude", "expected"), [([], True), ([r"re:.*\.shared_expert\..*"], False)], @@ -955,6 +1069,73 @@ def test_quark_shared_expert_fse_compatibility( ) +@pytest.mark.parametrize("exclude_shared_expert", [False, True]) +def test_online_targets_off_shared_expert_defer_to_checkpoint( + exclude_shared_expert: bool, +) -> None: + """Unrelated online targets preserve the checkpoint's FSE decision.""" + shared_prefix = "model.layers.0.mlp.shared_expert" + quant_config = QuarkConfig( + { + **_QUARK_FSE_CONFIG, + "exclude": ( + [f"{shared_prefix}.down_proj"] if exclude_shared_expert else [] + ), + } + ) + quant_config.online_quantization_config = OnlineQuantizationConfig( + QuantizationConfigArgs(targets={"*linear_attn.out_proj": "mxfp4"}) + ) + + compatible, reason = is_shared_expert_quant_fse_compatible( + quant_config, + "model.layers.0.mlp.experts", + shared_prefix, + ) + + assert compatible is not exclude_shared_expert + assert reason == ( + f"Quark excludes shared experts at {shared_prefix}" + if exclude_shared_expert + else None + ) + + +def test_online_targeting_only_one_shared_projection_rejects_fse() -> None: + shared_prefix = "model.layers.0.mlp.shared_expert" + quant_config = QuarkConfig({**_QUARK_FSE_CONFIG, "exclude": []}) + quant_config.online_quantization_config = OnlineQuantizationConfig( + QuantizationConfigArgs(targets={f"{shared_prefix}.down_proj": "mxfp4"}) + ) + + assert is_shared_expert_quant_fse_compatible( + quant_config, + "model.layers.0.mlp.experts", + shared_prefix, + ) == ( + False, + "online quantization targets only part of the shared expert at " + f"{shared_prefix}", + ) + + +def test_online_fp8_shared_expert_rejects_quark_mxfp4_fse() -> None: + shared_prefix = "model.layers.0.mlp.shared_expert" + quant_config = QuarkConfig({**_QUARK_FSE_CONFIG, "exclude": []}) + quant_config.online_quantization_config = OnlineQuantizationConfig( + QuantizationConfigArgs(targets={f"{shared_prefix}*": "fp8_per_tensor"}) + ) + + assert is_shared_expert_quant_fse_compatible( + quant_config, + "model.layers.0.mlp.experts", + shared_prefix, + ) == ( + False, + "online shared-expert quantization keys do not match the routed expert keys", + ) + + def test_quark_shared_expert_fse_exclude_is_scoped_to_the_layer() -> None: """Excluding one layer's shared experts must not disable FSE elsewhere. @@ -1220,14 +1401,15 @@ def test_quark_packed_layer_config_must_match_global_config() -> None: def test_non_quark_shared_expert_fse_is_incompatible() -> None: compatible, reason = is_shared_expert_quant_fse_compatible( - _stub_quant_config(), + ExpertsInt8Config(), "model.layers.0.mlp.experts", "model.layers.0.mlp.shared_experts", ) assert not compatible assert reason == ( - "shared-expert FSE quantization compatibility is not implemented for object" + "shared-expert FSE quantization compatibility is not implemented for " + "ExpertsInt8Config" ) diff --git a/tests/model_executor/model_loader/runai_streamer_loader/test_runai_model_streamer_loader.py b/tests/model_executor/model_loader/runai_streamer_loader/test_runai_model_streamer_loader.py index 9164b8e4be..e0a083b3a8 100644 --- a/tests/model_executor/model_loader/runai_streamer_loader/test_runai_model_streamer_loader.py +++ b/tests/model_executor/model_loader/runai_streamer_loader/test_runai_model_streamer_loader.py @@ -6,11 +6,13 @@ import types from unittest.mock import patch import pytest +import torch from vllm import SamplingParams from vllm.config.load import LoadConfig from vllm.model_executor.model_loader import get_model_loader from vllm.model_executor.model_loader import runai_streamer_loader as rsl +from vllm.utils.mem_constants import GiB_bytes load_format = "runai_streamer" test_model = "openai-community/gpt2" @@ -122,3 +124,94 @@ def test_runai_invalid_extra_config_leaves_environ_untouched(): with pytest.raises(ValueError, match="memory_limit must be an integer >= -1"): _runai_loader({"concurrency": 16, "memory_limit": -5}) assert "RUNAI_STREAMER_CONCURRENCY" not in os.environ + + +@pytest.mark.parametrize( + "model_config,expected_source", + [ + # Object storage: model_weights holds the URI, model the pulled config dir. + ( + types.SimpleNamespace( + model="/tmp/pulled-config-files", + model_weights="s3://bucket/weights", + revision="myrev", + ), + "s3://bucket/weights", + ), + # HF repo or local path: model_weights is empty, model is the source. + ( + types.SimpleNamespace( + model="org/model", model_weights="", revision="myrev" + ), + "org/model", + ), + ], + ids=["object_storage", "hf_repo"], +) +def test_runai_get_all_weights_resolves_source(model_config, expected_source): + fake_self = types.SimpleNamespace( + _get_weights_iterator=lambda path, revision: iter( + [(f"{path}@{revision}", None)] + ) + ) + + weights = list( + rsl.RunaiModelStreamerLoader.get_all_weights( + fake_self, model_config, model=None + ) + ) + + assert weights == [(f"{expected_source}@myrev", None)] + + +@pytest.mark.parametrize( + "base_model,mul_model,add_model", + [ + ( + "Qwen/Qwen3-0.6B", + "inference-optimization/Qwen3-0.6B-debug-multiply", + "inference-optimization/Qwen3-0.6B-debug-add", + ), + ], +) +def test_runai_deep_sleep_reload_weights(base_model, mul_model, add_model, vllm_runner): + free, total = torch.accelerator.get_memory_info() + used_bytes_baseline = total - free + + def sleep_and_reload(llm, path): + # Level 2 discards the parameter memory, so after wake_up the weights + # can only come from the reload: a tensor the streamer fails to + # deliver changes the output instead of keeping a stale valid value. + llm.get_llm().sleep(level=2) + free, total = torch.accelerator.get_memory_info() + assert total - free - used_bytes_baseline < 3 * GiB_bytes + llm.get_llm().wake_up(tags=["weights"]) + llm.collective_rpc("reload_weights", kwargs={"weights_path": path}) + free, total = torch.accelerator.get_memory_info() + assert total - free - used_bytes_baseline < 4 * GiB_bytes + llm.get_llm().wake_up(tags=["kv_cache"]) + + with vllm_runner( + model_name=base_model, + load_format=load_format, + enable_sleep_mode=True, + enable_prefix_caching=False, + max_model_len=16, + max_num_seqs=1, + ) as llm: + base_output = llm.generate_greedy(["3 4 ="], max_tokens=4) + + sleep_and_reload(llm, mul_model) + mul_perp = llm.generate_prompt_perplexity(["3 4 = 12"], mask=["3 4 ="])[0] + add_perp = llm.generate_prompt_perplexity(["3 4 = 7"], mask=["3 4 ="])[0] + assert mul_perp < add_perp + + sleep_and_reload(llm, add_model) + mul_perp = llm.generate_prompt_perplexity(["3 4 = 12"], mask=["3 4 ="])[0] + add_perp = llm.generate_prompt_perplexity(["3 4 = 7"], mask=["3 4 ="])[0] + assert add_perp < mul_perp + + # Round trip to the original weights over discarded memory must + # reproduce the pre-sleep output. + sleep_and_reload(llm, base_model) + assert llm.generate_greedy(["3 4 ="], max_tokens=4) == base_output diff --git a/tests/model_executor/model_loader/runai_streamer_loader/test_runai_model_streamer_s3.py b/tests/model_executor/model_loader/runai_streamer_loader/test_runai_model_streamer_s3.py index d7abc80edb..a1440ae5d2 100644 --- a/tests/model_executor/model_loader/runai_streamer_loader/test_runai_model_streamer_s3.py +++ b/tests/model_executor/model_loader/runai_streamer_loader/test_runai_model_streamer_s3.py @@ -3,7 +3,9 @@ from pathlib import Path +import torch from runai_model_streamer.safetensors_streamer.streamer_mock import StreamerPatcher +from safetensors.torch import load_file, save_file from vllm.engine.arg_utils import EngineArgs from vllm.transformers_utils.repo_utils import hf_api @@ -50,3 +52,72 @@ def test_runai_model_loader_download_files_s3_mocked_with_patch( executor = RunaiDummyExecutor(vllm_config) executor.driver_worker.load_model() + + +def test_runai_reload_weights_path_from_mocked_s3(tmp_path: Path, monkeypatch): + # An engine started from object storage keeps the bucket URI in + # model_config.model_weights. A reload with weights_path must stream the + # new checkpoint, not re-stream the one the engine started from. + patcher = StreamerPatcher(str(tmp_path)) + monkeypatch.setattr( + "vllm.transformers_utils.runai_utils.runai_list_safetensors", + patcher.shim_list_safetensors, + ) + monkeypatch.setattr( + "vllm.transformers_utils.runai_utils.runai_pull_files", + patcher.shim_pull_files, + ) + monkeypatch.setattr( + "vllm.model_executor.model_loader.weight_utils.SafetensorsStreamer", + patcher.create_mock_streamer, + ) + + base_dir = tmp_path / "gpt2" + hf_api().snapshot_download( + repo_id=test_model, + local_dir=str(base_dir), + allow_patterns=["*.json", "*.txt", "model.safetensors"], + ) + # Second checkpoint under another prefix: the final layer norm scaled, so a + # reload that silently re-streams the original bucket is distinguishable. + tensors = load_file(base_dir / "model.safetensors") + original = tensors["ln_f.weight"].clone() + modified = original * 2 + tensors["ln_f.weight"] = modified + mod_dir = tmp_path / "gpt2-mod" + mod_dir.mkdir() + save_file(tensors, mod_dir / "model.safetensors", metadata={"format": "pt"}) + + vllm_config = EngineArgs( + model="s3://my-mock-bucket/gpt2/", + load_format=load_format, + tensor_parallel_size=1, + ).create_engine_config() + assert vllm_config.model_config.model_weights == "s3://my-mock-bucket/gpt2/" + + executor = RunaiDummyExecutor(vllm_config) + executor.driver_worker.load_model() + model = executor.driver_worker.model_runner.get_model() + (param,) = [p for n, p in model.named_parameters() if n.endswith("ln_f.weight")] + + def current() -> torch.Tensor: + return param.detach().float().cpu() + + torch.testing.assert_close(current(), original, rtol=1e-2, atol=1e-2) + + # No weights_path: re-stream the checkpoint the engine started from, i.e. + # the bucket kept in model_weights. Zero the live tensor first so a reload + # that did nothing is distinguishable from one that streamed the bucket. + param.data.zero_() + executor.driver_worker.reload_weights() + torch.testing.assert_close(current(), original, rtol=1e-2, atol=1e-2) + + # weights_path: the new prefix must win over the stale model_weights. + executor.driver_worker.reload_weights(weights_path="s3://my-mock-bucket/gpt2-mod/") + torch.testing.assert_close(current(), modified, rtol=1e-2, atol=1e-2) + + # After the override, a bare reload means the new checkpoint, not the + # one the engine was started with. + param.data.zero_() + executor.driver_worker.reload_weights() + torch.testing.assert_close(current(), modified, rtol=1e-2, atol=1e-2) diff --git a/tests/model_executor/model_loader/tensorizer_loader/test_tensorizer.py b/tests/model_executor/model_loader/tensorizer_loader/test_tensorizer.py index 070c832967..c19765c462 100644 --- a/tests/model_executor/model_loader/tensorizer_loader/test_tensorizer.py +++ b/tests/model_executor/model_loader/tensorizer_loader/test_tensorizer.py @@ -579,7 +579,7 @@ async def test_serialize_and_serve_entrypoints(tmp_path): try: await asyncio.wait_for(fut, 180) - except asyncio.TimeoutError: + except TimeoutError: pytest.fail("Server did not start successfully") finally: proc.terminate() diff --git a/tests/model_executor/model_loader/test_filter_duplicate_safetensors.py b/tests/model_executor/model_loader/test_filter_duplicate_safetensors.py index babc6c061a..da04d0108f 100644 --- a/tests/model_executor/model_loader/test_filter_duplicate_safetensors.py +++ b/tests/model_executor/model_loader/test_filter_duplicate_safetensors.py @@ -4,11 +4,17 @@ import json import os import tempfile +from types import SimpleNamespace import pytest +import torch +from safetensors.torch import save_file +from vllm.config.load import LoadConfig +from vllm.model_executor.model_loader import DefaultModelLoader from vllm.model_executor.model_loader.weight_utils import ( filter_duplicate_safetensors_files, + filter_safetensors_files_by_weight_name, ) @@ -77,3 +83,57 @@ def test_filter_duplicate_safetensors_files_all_exist(): if __name__ == "__main__": test_filter_duplicate_safetensors_files_missing_weight() test_filter_duplicate_safetensors_files_all_exist() + + +def _skip_non_mtp(name: str) -> bool: + return not name.startswith("mtp.") + + +@pytest.fixture +def mtp_checkpoint(tmp_path): + """Three shards: target only, target + MTP, MTP only.""" + shards = { + "model-00001-of-00003.safetensors": ["model.layers.0.weight"], + "model-00002-of-00003.safetensors": ["model.layers.1.weight", "mtp.a.weight"], + "model-00003-of-00003.safetensors": ["mtp.b.weight"], + } + for filename, names in shards.items(): + save_file({name: torch.zeros(1) for name in names}, tmp_path / filename) + return tmp_path + + +def test_filter_by_weight_name_drops_shards_without_wanted_weights(mtp_checkpoint): + files = sorted(str(f) for f in mtp_checkpoint.glob("*.safetensors")) + + assert filter_safetensors_files_by_weight_name(files, _skip_non_mtp) == files[1:] + # A rule that rejects everything must not leave the loader without files. + assert filter_safetensors_files_by_weight_name(files, lambda _: True) == files + + +def test_loader_does_not_read_shards_the_model_skips(mtp_checkpoint, monkeypatch): + """Whole-file loaders such as InstantTensor only see the kept shards.""" + opened: list[list[str]] = [] + + def fake_iterator(hf_weights_files, *args, **kwargs): + opened.append(sorted(os.path.basename(f) for f in hf_weights_files)) + return iter(()) + + monkeypatch.setattr( + "vllm.model_executor.model_loader.default_loader." + "instanttensor_weights_iterator", + fake_iterator, + ) + + class Drafter(torch.nn.Module): + def is_unused_checkpoint_weight(self, name: str) -> bool: + return _skip_non_mtp(name) + + loader = DefaultModelLoader(LoadConfig(load_format="instanttensor")) + model_config = SimpleNamespace(model=str(mtp_checkpoint), revision=None) + list(loader.get_all_weights(model_config, Drafter())) + list(loader.get_all_weights(model_config, torch.nn.Module())) + + assert opened == [ + ["model-00002-of-00003.safetensors", "model-00003-of-00003.safetensors"], + sorted(f.name for f in mtp_checkpoint.glob("*.safetensors")), + ] diff --git a/tests/model_executor/model_loader/test_reload.py b/tests/model_executor/model_loader/test_reload.py index 85ddea3946..66c5b49050 100644 --- a/tests/model_executor/model_loader/test_reload.py +++ b/tests/model_executor/model_loader/test_reload.py @@ -528,6 +528,7 @@ def test_padded_moe_reload_releases_each_layer( layer.quant_method = method layer.expert_map_manager = SimpleNamespace(map_global_to_local=lambda i: i) layer._loaded_expert_biases = set() + layer._fused_shared_expert_quantizer = None method.create_weights( layer, experts, @@ -1051,6 +1052,37 @@ def test_hpc_rope_norm_kernel_sees_refit_norm_weights(monkeypatch, hpc_rope_norm assert not hasattr(rnorm, "qnorm_weight") +def _device_tensor_ptrs(worker) -> dict[str, int]: + """Addresses of every device tensor held by a module; graphs capture them.""" + ptrs = {} + for name, module in worker.get_model().named_modules(): + tensors = {**vars(module), **module._parameters, **module._buffers} + for attr, tensor in tensors.items(): + if isinstance(tensor, torch.Tensor) and tensor.device.type != "cpu": + ptrs[f"{name}.{attr}"] = tensor.data_ptr() + return ptrs + + +def _moved_tensors(llm, ptrs: list[dict[str, int]]) -> list[list[str]]: + after = llm.collective_rpc(_device_tensor_ptrs) + return [ + sorted(n for n, p in old.items() if new.get(n) != p) + for old, new in zip(ptrs, after) + ] + + +@pytest.mark.parametrize( + "use_aiter", + [ + False, + pytest.param( + True, + marks=pytest.mark.skipif( + not current_platform.is_rocm(), reason="AITER is ROCm-only" + ), + ), + ], +) @pytest.mark.parametrize( "tp_size", [pytest.param(1), pytest.param(2, marks=[pytest.mark.slow_test])] ) @@ -1094,13 +1126,19 @@ def test_hpc_rope_norm_kernel_sees_refit_norm_weights(monkeypatch, hpc_rope_norm ), ], ) -def test_reload_weights(base_model, mul_model, add_model, tp_size, vllm_runner): +def test_reload_weights( + base_model, mul_model, add_model, tp_size, use_aiter, vllm_runner, monkeypatch +): if current_platform.device_count() < tp_size: pytest.skip(reason="Not enough CUDA devices") if "FP8" in base_model and _fp8_reload_unsupported(): pytest.skip(reason="Requires FP8 support") + monkeypatch.setenv("VLLM_ALLOW_INSECURE_SERIALIZATION", "1") + if use_aiter: + monkeypatch.setenv("VLLM_ROCM_USE_AITER", "1") + with vllm_runner( model_name=base_model, tensor_parallel_size=tp_size, @@ -1109,12 +1147,16 @@ def test_reload_weights(base_model, mul_model, add_model, tp_size, vllm_runner): max_model_len=16, max_num_seqs=1, ) as llm: + # Reloading must update tensors in place: captured graphs keep pointers. + ptrs = llm.collective_rpc(_device_tensor_ptrs) llm.collective_rpc("reload_weights", kwargs={"weights_path": mul_model}) + assert _moved_tensors(llm, ptrs) == [[]] * len(ptrs) mul_perp = llm.generate_prompt_perplexity(["3 4 = 12"], mask=["3 4 ="])[0] add_perp = llm.generate_prompt_perplexity(["3 4 = 7"], mask=["3 4 ="])[0] assert mul_perp < add_perp llm.collective_rpc("reload_weights", kwargs={"weights_path": add_model}) + assert _moved_tensors(llm, ptrs) == [[]] * len(ptrs) mul_perp = llm.generate_prompt_perplexity(["3 4 = 12"], mask=["3 4 ="])[0] add_perp = llm.generate_prompt_perplexity(["3 4 = 7"], mask=["3 4 ="])[0] assert add_perp < mul_perp diff --git a/tests/model_executor/model_loader/test_weight_cache.py b/tests/model_executor/model_loader/test_weight_cache.py index 731c732625..7960a23791 100644 --- a/tests/model_executor/model_loader/test_weight_cache.py +++ b/tests/model_executor/model_loader/test_weight_cache.py @@ -356,6 +356,52 @@ def test_weight_cache_key_distinguishes_dp_ranks(): assert key.mismatched_fields(replace(key, pp_rank=0)) == ["pp_rank"] +@pytest.mark.parametrize("hf_quant_config", [None, {"quant_algo": "NVFP4"}]) +def test_weight_cache_key_distinguishes_nvfp4_activation_override( + tmp_path, hf_quant_config +): + """Cache keys isolate activation modes and reject ambiguous legacy hashes.""" + import json + from dataclasses import replace + from types import SimpleNamespace + + import torch + + from vllm.config.quantization import QuantizationConfigArgs + from vllm.model_executor.model_loader.weight_cache.protocol import WeightCacheKey + from vllm.utils.hashing import safe_hash + + model_config = SimpleNamespace( + model=str(tmp_path), + dtype=torch.bfloat16, + quantization="modelopt_fp4", + quantization_config=None, + revision=None, + hf_config=SimpleNamespace( + architectures=["NemotronHForCausalLM"], + quantization_config=hf_quant_config, + ), + ) + static_key = WeightCacheKey.from_model_config(model_config, tp_size=1, tp_rank=0) + model_config.quantization_config = QuantizationConfigArgs( + moe={"activation": "nvfp4_per_token"} + ) + per_token_key = WeightCacheKey.from_model_config(model_config, tp_size=1, tp_rank=0) + assert static_key.mismatched_fields(per_token_key) == ["quant_config_hash"] + + legacy_hash = ( + "" + if hf_quant_config is None + else safe_hash( + json.dumps(hf_quant_config, sort_keys=True).encode(), + usedforsecurity=False, + ).hexdigest() + ) + assert static_key.mismatched_fields( + replace(static_key, quant_config_hash=legacy_hash) + ) == ["quant_config_hash"] + + def test_ipc_loader_copy_mode_reports_no_external_weight_memory(): """Copy mode clones the weights into the engine, so nothing is external (vllm_config is never touched).""" diff --git a/tests/model_executor/test_flashinfer_autotune_warmup.py b/tests/model_executor/test_flashinfer_autotune_warmup.py index 22bb54d71f..f7ea6847f2 100644 --- a/tests/model_executor/test_flashinfer_autotune_warmup.py +++ b/tests/model_executor/test_flashinfer_autotune_warmup.py @@ -39,13 +39,13 @@ def _make_moe(*, max_deferred_tokens: int = 128, enabled: bool = True): return moe -def _make_runner(modules, *, max_tokens: int = 8192, linear_backend: str = "auto"): +def _make_runner(modules, *, max_tokens: int = 8192): """Create a runner carrying only the state used by FlashInfer warmup.""" return SimpleNamespace( scheduler_config=SimpleNamespace(max_num_batched_tokens=max_tokens), vllm_config=SimpleNamespace( - kernel_config=SimpleNamespace(linear_backend=linear_backend), attention_config=SimpleNamespace(hisparse_config=None), + parallel_config=SimpleNamespace(data_parallel_rank=0), ), get_model=Mock( return_value=SimpleNamespace(modules=Mock(return_value=modules)) @@ -62,21 +62,19 @@ def test_flashinfer_autotune_token_counts_include_deferred_moe_limits(): _make_moe(max_deferred_tokens=128), _make_moe(max_deferred_tokens=64, enabled=False), _make_moe(max_deferred_tokens=-1), - ], - linear_backend="flashinfer_cutedsl", + ] ) with patch("vllm.model_executor.layers.fused_moe.MoERunner", _FakeMoERunner): token_counts = _flashinfer_autotune_token_counts(runner) - assert token_counts == (8192, 32, 128) + assert token_counts == (8192, 128) def test_flashinfer_autotune_token_counts_are_bounded_and_deduplicated(): runner = _make_runner( [_make_moe(max_deferred_tokens=4096)], max_tokens=32, - linear_backend="flashinfer_cutedsl", ) with patch("vllm.model_executor.layers.fused_moe.MoERunner", _FakeMoERunner): @@ -114,8 +112,8 @@ def test_flashinfer_autotune_uses_token_buckets_for_each_dummy_run(skip_attn): assert get_buckets.call_args_list == [call(8192), call(128)] assert autotune.call_args_list == [ - call(tuning_buckets=max_buckets), - call(tuning_buckets=deferred_buckets), + call(tuning_buckets=max_buckets, round_up=True), + call(tuning_buckets=deferred_buckets, round_up=True), ] assert runner._dummy_run.call_args_list == [ call( @@ -135,6 +133,33 @@ def test_flashinfer_autotune_uses_token_buckets_for_each_dummy_run(skip_attn): ] +def test_flashinfer_autotune_buckets_cover_drafter_tokens(): + """The drafter's M can exceed the pass size; buckets must include it.""" + runner = _make_runner([]) + runner.max_num_reqs = 256 + runner.max_num_tokens = 8192 + runner.speculator = SimpleNamespace(num_query_per_req=6) + + with ( + patch( + "vllm.model_executor.warmup.kernel_warmup." + "_flashinfer_autotune_token_counts", + return_value=(8192, 128), + ), + patch( + "vllm.utils.flashinfer.flashinfer_get_hybrid_num_tokens_buckets" + ) as get_buckets, + patch("vllm.utils.flashinfer.autotune"), + ): + _run_flashinfer_autotune_dummy_runs(runner) + + assert get_buckets.call_args_list == [call(8192), call(128 * 6)] + assert [c.kwargs["num_tokens"] for c in runner._dummy_run.call_args_list] == [ + 8192, + 128, + ] + + class _AutotuneGroup: def __init__(self, run, ranks): self.run = run @@ -146,12 +171,20 @@ class _AutotuneGroup: def record(self, operation): self.run.collectives[self.ranks][self.run.rank].append(operation) - def broadcast_object(self, obj, src=0): - assert src == 0 - self.record(("broadcast", src)) - if self.rank_in_group == src: - self.run.broadcasts[self.ranks] = obj - return self.run.broadcasts[self.ranks] + def all_gather_object(self, out, obj): + # Ranks run sequentially: snapshot every rank's file before any saves. + self.record(("all_gather",)) + if isinstance(obj, bool): + out[:] = [rank not in self.run.rejected_load_ranks for rank in self.ranks] + assert out[self.rank_in_group] == obj + return + out[:] = self.run.gathered.setdefault( + self.ranks, + [ + warmup._autotune_cache_fingerprint(self.run.cache_path(rank)) + for rank in self.ranks + ], + ) def barrier(self): self.record(("barrier",)) @@ -162,10 +195,18 @@ class _AutotuneTuner: self.run = run self.cache = {} self.loaded = None + self._dirty = False def load_configs(self, path): + if self.run.rank in self.run.rejected_load_ranks: + return False self.loaded = json.loads(Path(path).read_text()) self.cache.update(self.loaded) + return True + + def clear_cache(self): + self.cache.clear() + self._dirty = False def save_configs(self, path): self.run.saves.append((self.run.rank, Path(path), dict(self.cache))) @@ -182,21 +223,27 @@ class _AutotuneTuner: if group is not None: group.record(("all_reduce", operation, tactic)) self.cache[operation] = self.run.rank // self.run.tp + self._dirty = True class _AutotuneRun: - def __init__(self, pp, tp): + def __init__(self, pp, tp, cache_dir): self.pp, self.tp = pp, tp + self.cache_dir = cache_dir self.rank = 0 self.tuning_group = None self.collectives: dict[tuple[int, ...], dict[int, list[tuple[Any, ...]]]] = ( defaultdict(lambda: defaultdict(list)) ) - self.broadcasts = {} + self.gathered = {} + self.rejected_load_ranks: set[int] = set() self.tuners = {} self.saves = [] self.profile_groups = defaultdict(list) + def cache_path(self, rank): + return self.cache_dir / f"autotune_configs_dp0_rank{rank}.json" + def world(self): return _AutotuneGroup(self, range(self.pp * self.tp)) @@ -235,7 +282,7 @@ def autotune_run(monkeypatch, tmp_path): from vllm.distributed import parallel_state def make_run(*, pp=2, tp=4): - run = _AutotuneRun(pp, tp) + run = _AutotuneRun(pp, tp, tmp_path) autotuner = ModuleType("flashinfer.autotuner") monkeypatch.setattr( autotuner, @@ -251,6 +298,10 @@ def autotune_run(monkeypatch, tmp_path): monkeypatch.setattr(parallel_state, "get_tp_group", run.tensor_group) monkeypatch.setattr(parallel_state, "get_pp_group", run.pipeline_group) monkeypatch.setattr(fi_utils, "autotune", lambda **kwargs: nullcontext()) + monkeypatch.setattr( + "torch.distributed.all_gather_object", + lambda out, obj, group: group.all_gather_object(out, obj), + ) monkeypatch.setattr( warmup, "resolve_flashinfer_autotune_file", @@ -286,19 +337,18 @@ def test_pp_stage_cache_roundtrip_isolated_and_asymmetric_hits_safe( legacy = tmp_path / "autotune_configs.json" legacy.write_text('{"legacy_world_cache": 99}') cold = autotune_run().execute() - assert [rank for rank, _, _ in cold.saves] == [0, 4] - paths = [path for _, path, _ in cold.saves] - assert len(set(paths)) == 2 and legacy not in paths - assert json.loads(legacy.read_text()) == {"legacy_world_cache": 99} - assert cold.saves[0][2] == {"shared_gemm": 0, "pp0_extra_gemm": 0} - assert cold.saves[1][2] == {"shared_gemm": 1} cold.assert_collectives_match() + assert [rank for rank, _, _ in cold.saves] == list(range(8)) + assert [path for _, path, _ in cold.saves] == [cold.cache_path(r) for r in range(8)] + assert json.loads(legacy.read_text()) == {"legacy_world_cache": 99} + stage_caches = [{"shared_gemm": 0, "pp0_extra_gemm": 0}, {"shared_gemm": 1}] + assert all(cache == stage_caches[r // 4] for r, _, cache in cold.saves) warm = autotune_run().execute() warm.assert_collectives_match() - assert not warm.profile_groups + assert not warm.profile_groups and not warm.saves for rank, tuner in warm.tuners.items(): - assert tuner.loaded == cold.saves[rank // 4][2] - paths[1].unlink() + assert tuner.loaded == stage_caches[rank // 4] + cold.cache_path(5).unlink() mixed = autotune_run().execute() mixed.assert_collectives_match() assert set(mixed.profile_groups) == {4, 5, 6, 7} @@ -306,10 +356,33 @@ def test_pp_stage_cache_roundtrip_isolated_and_asymmetric_hits_safe( assert all(mixed.tuners[rank].loaded is None for rank in range(4, 8)) -def test_pp1_retains_world_synchronization_and_existing_cache_name(autotune_run): +def test_pp1_tunes_world_group_and_saves_per_rank(autotune_run): run = autotune_run(pp=1, tp=4).execute() run.assert_collectives_match() - assert [(rank, path.name) for rank, path, _ in run.saves] == [ - (0, "autotune_configs.json") + assert [(rank, path) for rank, path, _ in run.saves] == [ + (rank, run.cache_path(rank)) for rank in range(4) ] assert all(groups == [(0, 1, 2, 3)] for groups in run.profile_groups.values()) + + +def test_mismatched_rank_caches_are_ignored_by_every_rank(autotune_run): + """One rank loading while others tune deadlocks the per-tactic reduce.""" + cold = autotune_run(pp=1, tp=4).execute() + stale = cold.cache_path(2) + stale.write_text(json.dumps({**json.loads(stale.read_text()), "old_gemm": 0})) + rerun = autotune_run(pp=1, tp=4).execute() + rerun.assert_collectives_match() + assert all(tuner.loaded is None for tuner in rerun.tuners.values()) + assert set(rerun.profile_groups) == {0, 1, 2, 3} + + +@pytest.mark.parametrize("pp, tp", [(1, 1), (1, 4), (2, 4)]) +def test_rejected_cache_retunes_every_rank_in_its_tuning_group(autotune_run, pp, tp): + """Matching files can still be incompatible with one rank's runtime.""" + autotune_run(pp=pp, tp=tp).execute() + rerun = autotune_run(pp=pp, tp=tp) + rerun.rejected_load_ranks = {tp - 1} + rerun.execute() + rerun.assert_collectives_match() + assert set(rerun.profile_groups) == set(range(tp)) + assert {rank for rank, _, _ in rerun.saves} == set(range(tp)) diff --git a/tests/model_executor/test_phi4mm_attention.py b/tests/model_executor/test_phi4mm_attention.py new file mode 100644 index 0000000000..51710f84bc --- /dev/null +++ b/tests/model_executor/test_phi4mm_attention.py @@ -0,0 +1,28 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import torch + +from vllm.model_executor.models.phi4mm_utils import MultiHeadedAttention + + +def test_eager_attention_does_not_retain_probability_matrix(): + module = MultiHeadedAttention( + n_head=2, + n_feat=16, + dropout_rate=0.0, + use_pt_scaled_dot_product_attention=False, + ) + module.eval() + batch, time, feat = 2, 8, 16 + x = torch.randn(batch, time, feat) + with torch.no_grad(): + out = module(x, x, x, None, None, None) + + assert out.shape == (batch, time, feat) + assert getattr(module, "attn", None) is None + + +def test_multiheaded_attention_defaults_to_scaled_dot_product(): + module = MultiHeadedAttention(n_head=2, n_feat=16, dropout_rate=0.0) + assert module.use_pt_scaled_dot_product_attention is True diff --git a/tests/model_executor/test_qwen3_omni_mrope.py b/tests/model_executor/test_qwen3_omni_mrope.py index bb8de63dc5..2c96c328df 100644 --- a/tests/model_executor/test_qwen3_omni_mrope.py +++ b/tests/model_executor/test_qwen3_omni_mrope.py @@ -145,27 +145,45 @@ def test_mrope_positions_video_at_prompt_end(): assert positions.shape == (3, len(input_tokens)) -def test_mrope_positions_audio_in_video_at_prompt_end(): +@pytest.mark.parametrize("num_videos", [1, 2]) +@pytest.mark.parametrize("trailing_tokens", [0, 3]) +def test_mrope_positions_audio_in_video_boundaries(num_videos, trailing_tokens): + """Interleaved BOS/EOS positions and decode delta match Qwen3-Omni HF.""" model = make_model() - # video grid (1, 2, 2) -> 4 video tokens; audio_feature_length=16 -> 2 - # audio tokens, interleaved after the video tokens. - # [T, T, <|vision_start|>, <|audio_start|>, V x 4, A x 2, - # <|audio_end|>, <|vision_end|>] - input_tokens = ( - [TEXT_TOKEN_ID] * 2 - + [VISION_START_TOKEN_ID, AUDIO_START_TOKEN_ID] + video_tokens = ( + [VISION_START_TOKEN_ID, AUDIO_START_TOKEN_ID] + [VIDEO_TOKEN_ID] * 4 + [AUDIO_TOKEN_ID] * 2 + [AUDIO_END_TOKEN_ID, VISION_END_TOKEN_ID] ) - mm_features = [ - make_video_feature(offset=3, grid_thw=(1, 2, 2), use_audio_in_video=True), - make_audio_feature(offset=3, audio_feature_length=16), - ] + input_tokens = [TEXT_TOKEN_ID] * 2 + mm_features = [] + expected = [torch.tensor([[0, 1]]).expand(3, -1)] + video_positions = torch.tensor( + [ + [2, 3, 4, 4, 4, 4, 4, 5, 6, 7], + [2, 3, 4, 4, 5, 5, 4, 5, 6, 7], + [2, 3, 4, 5, 4, 5, 4, 5, 6, 7], + ] + ) + for index in range(num_videos): + offset = len(input_tokens) + 1 + mm_features.extend( + [ + make_video_feature(offset, (1, 2, 2), use_audio_in_video=True), + make_audio_feature(offset, audio_feature_length=16), + ] + ) + input_tokens.extend(video_tokens) + expected.append(video_positions + index * 6) - positions, _ = model.get_mrope_input_positions(input_tokens, mm_features) + input_tokens.extend([TEXT_TOKEN_ID] * trailing_tokens) + next_position = 2 + num_videos * 6 + expected.append((torch.arange(trailing_tokens) + next_position).expand(3, -1)) + positions, delta = model.get_mrope_input_positions(input_tokens, mm_features) - assert positions.shape == (3, len(input_tokens)) + torch.testing.assert_close(positions, torch.cat(expected, dim=1)) + assert delta == -4 * num_videos def test_mrope_positions_image_then_text(): diff --git a/tests/model_executor/test_routed_experts_capture.py b/tests/model_executor/test_routed_experts_capture.py index 2d57236f8e..5416fdc8ea 100644 --- a/tests/model_executor/test_routed_experts_capture.py +++ b/tests/model_executor/test_routed_experts_capture.py @@ -582,10 +582,7 @@ def test_v2_model_runner_accepts_routed_experts(monkeypatch): use_ubatching=False, enable_elastic_ep=False, ), - compilation_config=SimpleNamespace( - mode=CompilationMode.NONE, - pass_config=SimpleNamespace(enable_sp=False), - ), + compilation_config=SimpleNamespace(mode=CompilationMode.NONE), cache_config=SimpleNamespace( kv_sharing_fast_prefill=False, mamba_cache_mode="none", diff --git a/tests/models/glm5next/test_kda_recurrent.py b/tests/models/glm5next/test_kda_recurrent.py index fe017e4131..06b2eafac6 100644 --- a/tests/models/glm5next/test_kda_recurrent.py +++ b/tests/models/glm5next/test_kda_recurrent.py @@ -194,6 +194,77 @@ def test_fused_recurrent_kda_rejects_unaddressable_layouts(): run_kernel(broken, state) +@pytest.mark.skipif( + not current_platform.is_cuda(), reason="The large-grid fix is NVIDIA-only" +) +@pytest.mark.parametrize("num_seqs,num_heads", [(1024, 64), (2048, 48)]) +def test_fused_recurrent_kda_more_than_65535_programs(num_seqs: int, num_heads: int): + """Regression test for the launch grid. + + GLM-5.3-Flash has 64 KDA value heads; at TP=1 (every DP rank) a decode + batch of 1024 sequences needs 1024 * 64 = 65536 (sequence, head) + programs. The kernel used to put that product into gridDim.z, whose CUDA + limit is 65535, so the launch failed with ``Triton Error [CUDA]: invalid + argument`` at CUDA-graph capture and DP deployments could not start. The + grid now carries the product in gridDim.x. Small head dims keep the test + cheap; the shape that matters is the number of programs. Every (sequence, + head) pair is independent, so the full batch must be bitwise identical to + the same batch run in two halves. + """ + torch.manual_seed(0) + dev = torch.device("cuda") + head_dim = 16 + pool = num_seqs + 8 + q = torch.randn(1, num_seqs, num_heads, head_dim, device=dev, dtype=torch.bfloat16) + k = torch.randn_like(q) + v = torch.randn_like(q) + g = torch.randn_like(q) + beta = torch.randn(1, num_seqs, num_heads, device=dev, dtype=torch.bfloat16) + state = torch.randn( + pool, num_heads, head_dim, head_dim, device=dev, dtype=torch.float32 + ) + cu = torch.arange(0, num_seqs + 1, device=dev, dtype=torch.int32) + # slot 0 is NULL_BLOCK_ID and is skipped by the kernel: use slots 1.. + idx = (torch.randperm(pool - 1, device=dev)[:num_seqs] + 1).to(torch.int32) + a_log = torch.randn(num_heads, device=dev, dtype=torch.float32) + g_bias = torch.randn(num_heads, head_dim, device=dev, dtype=torch.float32) + + def run(sl: slice, cu_: torch.Tensor, st: torch.Tensor, out: torch.Tensor): + fused_recurrent_kda( + q=q[:, sl], + k=k[:, sl], + v=v[:, sl], + g=g[:, sl], + beta=beta[:, sl], + initial_state=st, + use_qk_l2norm_in_kernel=True, + cu_seqlens=cu_, + ssm_state_indices=idx[sl], + sigmoid_beta=True, + a_log=a_log, + g_bias=g_bias, + compute_gate=True, + lower_bound=-5.0, + out=out, + ) + + full_state = state.clone() + full_out = torch.zeros_like(k) + run(slice(0, num_seqs), cu, full_state, full_out) + + half = num_seqs // 2 + ref_state = state.clone() + ref_out = torch.zeros_like(k) + for s, e in ((0, half), (half, num_seqs)): + cu_h = torch.arange(0, e - s + 1, device=dev, dtype=torch.int32) + part = torch.zeros_like(k[:, s:e]) + run(slice(s, e), cu_h, ref_state, part) + ref_out[:, s:e] = part + torch.accelerator.synchronize() + assert torch.equal(full_out, ref_out) + assert torch.equal(full_state, ref_state) + + @pytest.mark.parametrize("offset", [16, 144, 256]) @pytest.mark.parametrize("dim_first", [False, True]) @pytest.mark.parametrize("num_spec", [0, 3]) diff --git a/tests/models/kimi_k3/test_amd_attn_res.py b/tests/models/kimi_k3/test_amd_attn_res.py index 80d5e1936f..871d53faae 100644 --- a/tests/models/kimi_k3/test_amd_attn_res.py +++ b/tests/models/kimi_k3/test_amd_attn_res.py @@ -191,3 +191,106 @@ def test_amd_attn_res_fused_contract( original_blocks[:, block_write_idx].copy_(expected_prefix) torch.testing.assert_close(blocks, original_blocks, atol=0, rtol=0) assert actual.is_contiguous() + + +@pytest.mark.parametrize( + "num_tokens,num_blocks,has_delta,write_block", + [ + (0, 0, False, False), + (1, 0, True, True), + (17, 4, True, True), + (320, 8, False, False), + ], +) +def test_amd_attn_res_fp8_preserves_prefix_and_quantized_output( + num_tokens, + num_blocks, + has_delta, + write_block, + default_vllm_config, +): + from vllm.model_executor.layers.quantization.input_quant_fp8 import QuantFP8 + from vllm.model_executor.layers.quantization.utils.quant_utils import GroupShape + + torch.manual_seed(41) + hidden_size = 7168 + prefix = _randn_with_row_padding(num_tokens, hidden_size, padding=7) + delta = _randn_with_row_padding(num_tokens, hidden_size) if has_delta else None + blocks = _randn_with_row_padding(num_tokens, 9, hidden_size, padding=13) + norm = 1 + torch.randn(hidden_size, device="cuda", dtype=torch.bfloat16) * 0.1 + score = torch.randn_like(norm) / hidden_size**0.5 + out_norm = 1 + torch.randn_like(norm) * 0.1 + ref_prefix, ref_blocks = prefix.clone(), blocks.clone() + kwargs = dict( + num_blocks=num_blocks, + block_write_idx=num_blocks if write_block else -1, + eps=1e-5, + output_norm_eps=1e-5, + ) + reference = attn_res(ref_prefix, delta, ref_blocks, norm, score, out_norm, **kwargs) + output, scale = attn_res( + prefix, + delta, + blocks, + norm, + score, + out_norm, + quant_dtype=current_platform.fp8_dtype(), + **kwargs, + ) + assert output.dtype == current_platform.fp8_dtype() + assert scale.shape == (num_tokens, 1) + if num_tokens: + # Keep reference rounding independent of Inductor's fused quantization. + quant = QuantFP8( + static=False, group_shape=GroupShape.PER_TOKEN, compile_native=False + ) + ref_output, ref_scale = quant(reference) + torch.testing.assert_close(scale, ref_scale, atol=1e-7, rtol=1e-6) + + # Floating-point fusion can move values at FP8 rounding midpoints. + # Bound each change to the adjacent code and bound how often it occurs. + # Codes are sign-magnitude, so order them by value to keep a step + # across zero adjacent. + def ordinal(x: torch.Tensor) -> torch.Tensor: + code = x.view(torch.uint8).to(torch.int16) + return torch.where(code >= 0x80, 0x80 - code, code) + + code_delta = (ordinal(output) - ordinal(ref_output)).abs() + assert code_delta.max().item() <= 1 + assert (code_delta != 0).float().mean().item() < 1e-4 + torch.testing.assert_close(prefix, ref_prefix, atol=0, rtol=0) + torch.testing.assert_close(blocks, ref_blocks, atol=0, rtol=0) + + +@pytest.mark.parametrize( + "matching,requires_unquantized", [(True, False), (False, False), (True, True)] +) +def test_mla_input_quant_requires_all_consumers(matching, requires_unquantized): + from types import SimpleNamespace + + from torch import nn + + from vllm.model_executor.layers.fusion.quant_activation import ( + expose_input_quant_key, + ) + from vllm.model_executor.layers.quantization.utils.quant_utils import ( + kFp8DynamicTokenSym, + ) + from vllm.models.kimi_k3.amd.mla import KimiK3MultiHeadLatentAttentionWrapper + + attention = object.__new__(KimiK3MultiHeadLatentAttentionWrapper) + nn.Module.__init__(attention) + attention.indexer = None + attention.q_lora_rank = 1536 + attention.fused_qkv_a_proj = nn.Identity() + attention.g_proj = nn.Identity() + kernel = SimpleNamespace(input_quant_key=lambda: kFp8DynamicTokenSym) + expose_input_quant_key(attention.fused_qkv_a_proj, kernel) + if matching: + expose_input_quant_key(attention.g_proj, kernel) + attention.g_proj.requires_unquantized_input = requires_unquantized + actual = attention.get_input_quant_key() + assert actual == ( + kFp8DynamicTokenSym if matching and not requires_unquantized else None + ) diff --git a/tests/models/kimi_k3/test_amd_kda_checkpoint.py b/tests/models/kimi_k3/test_amd_kda_checkpoint.py index 0a622bfa04..6f3129a224 100644 --- a/tests/models/kimi_k3/test_amd_kda_checkpoint.py +++ b/tests/models/kimi_k3/test_amd_kda_checkpoint.py @@ -75,7 +75,7 @@ def _builder( vllm_config.cache_config.prefix_match_unit = prefix_match_unit if num_spec: vllm_config.speculative_config = _StubSpeculativeConfig(num_spec) - return KimiK3ROCmKDAMetadataBuilder( + builder = KimiK3ROCmKDAMetadataBuilder( kv_cache_spec=MambaSpec( block_size=block_size, shapes=((16, 64),), @@ -91,6 +91,11 @@ def _builder( vllm_config=vllm_config, device=torch.device("cuda"), ) + vllm_config.cache_config.hash_block_size = prefix_match_unit or block_size + vllm_config.cache_config.cache_hit_alignment_tokens = ( + prefix_match_unit or block_size + ) + return builder def _build(batch: BatchSpec, block_size: int = BLOCK_SIZE, **kw): diff --git a/tests/models/kimi_k3/test_amd_latent_moe_runner.py b/tests/models/kimi_k3/test_amd_latent_moe_runner.py index fd5bcfd22b..e47b5fc08e 100644 --- a/tests/models/kimi_k3/test_amd_latent_moe_runner.py +++ b/tests/models/kimi_k3/test_amd_latent_moe_runner.py @@ -22,7 +22,7 @@ from tests.utils import ( from vllm.distributed import get_tp_group from vllm.model_executor.layers.fused_moe.runner import moe_runner from vllm.model_executor.layers.layernorm import RMSNorm -from vllm.model_executor.layers.linear import ReplicatedLinear +from vllm.model_executor.layers.linear import ColumnParallelLinear, ReplicatedLinear from vllm.models.kimi_k3.amd import latent_moe_runner from vllm.models.kimi_k3.amd.latent_moe_runner import ROCmLatentMoERunner from vllm.models.kimi_k3.amd.linear import KimiRoutedOutputTransform @@ -40,20 +40,42 @@ EPS = 1e-5 DTYPE = torch.bfloat16 -def _build_transform(device: torch.device) -> KimiRoutedOutputTransform: +def _build_transform( + device: torch.device, *, row_sharded: bool = False +) -> tuple[KimiRoutedOutputTransform, torch.Tensor]: norm = RMSNorm(LATENT_SIZE, eps=EPS).to(device=device, dtype=DTYPE) - up_proj = ReplicatedLinear( - LATENT_SIZE, - HIDDEN_SIZE, - bias=False, - params_dtype=DTYPE, - prefix="routed_expert_up_proj", - ).to(device=device) torch.manual_seed(0) norm.weight.data.copy_(1 + 0.1 * torch.randn_like(norm.weight)) - up_proj.weight.data.copy_(torch.randn_like(up_proj.weight) / LATENT_SIZE**0.5) - return KimiRoutedOutputTransform(norm, up_proj) + full_weight = ( + torch.randn(HIDDEN_SIZE, LATENT_SIZE, device=device, dtype=DTYPE) + / LATENT_SIZE**0.5 + ) + + up_proj: ReplicatedLinear | ColumnParallelLinear + if row_sharded: + up_proj = ColumnParallelLinear( + LATENT_SIZE, + HIDDEN_SIZE, + bias=False, + gather_output=True, + params_dtype=DTYPE, + prefix="routed_expert_up_proj", + ).to(device=device) + up_proj.weight_loader(up_proj.weight, full_weight) + else: + up_proj = ReplicatedLinear( + LATENT_SIZE, + HIDDEN_SIZE, + bias=False, + params_dtype=DTYPE, + prefix="routed_expert_up_proj", + ).to(device=device) + up_proj.weight.data.copy_(full_weight) + return ( + KimiRoutedOutputTransform(norm, up_proj, row_sharded=row_sharded), + full_weight, + ) def _tail_runner( @@ -72,6 +94,7 @@ def _tail_runner( runner = object.__new__(ROCmLatentMoERunner) attrs = { "routed_output_transform": transform, + "_up_proj_preshard": bool(getattr(transform, "row_sharded", False)), "_up_proj_shard_size": HIDDEN_SIZE // tp_world, "_logged_sharded_tail": False, "moe_config": SimpleNamespace( @@ -101,9 +124,13 @@ def _rank_partials( def _check_matches_replicated( - device: torch.device, tp_world: int, rank: int, use_ep: bool = False + device: torch.device, + tp_world: int, + rank: int, + use_ep: bool = False, + row_sharded: bool = False, ) -> None: - transform = _build_transform(device) + transform, up_weight = _build_transform(device, row_sharded=row_sharded) runner = _tail_runner(transform, tp_world, use_ep) group = get_tp_group().device_group @@ -118,7 +145,7 @@ def _check_matches_replicated( transform.norm.weight, EPS, ), - transform.up_proj.weight, + up_weight, ) expected.add_(_all_reduced(shared_output, group)) @@ -136,12 +163,37 @@ def _check_matches_replicated_under_ep( _check_matches_replicated(device, tp_world, rank, use_ep=True) +def _check_presharded_matches_replicated( + device: torch.device, tp_world: int, rank: int +) -> None: + _check_matches_replicated(device, tp_world, rank, row_sharded=True) + + +def _check_unfused_tail_gathers_full_hidden( + device: torch.device, tp_world: int, rank: int +) -> None: + transform, up_weight = _build_transform(device, row_sharded=True) + assert transform.up_proj.weight.shape[0] == HIDDEN_SIZE // tp_world + + # all-gather reconstructs the full projection when all ranks feed the same + # input, so use one seed for all ranks. + torch.manual_seed(1234) + latent = torch.randn(4, LATENT_SIZE, device=device, dtype=DTYPE).mul_(0.01) + + torch.testing.assert_close( + transform(latent), + F.linear(transform.norm(latent), up_weight), + atol=8e-2, + rtol=3e-2, + ) + + def _check_writes_only_its_own_shard( device: torch.device, tp_world: int, rank: int ) -> None: """Two ranks that swapped both weight rows and write offsets still sum to the right total, so inspect each slice before the final collective.""" - transform = _build_transform(device) + transform, up_weight = _build_transform(device) runner = _tail_runner(transform, tp_world) group = get_tp_group().device_group @@ -169,7 +221,7 @@ def _check_writes_only_its_own_shard( shard = HIDDEN_SIZE // tp_world start, end = rank * shard, (rank + 1) * shard local = captured["states"] - projected = F.linear(latent, transform.up_proj.weight[start:end]) + projected = F.linear(latent, up_weight[start:end]) torch.testing.assert_close( local[:, start:end], before[:, start:end] + projected, atol=8e-2, rtol=3e-2 @@ -181,6 +233,8 @@ def _check_writes_only_its_own_shard( _CHECKS = { "matches_replicated": _check_matches_replicated, "matches_replicated_ep": _check_matches_replicated_under_ep, + "presharded_matches_replicated": _check_presharded_matches_replicated, + "unfused_gathers": _check_unfused_tail_gathers_full_hidden, "own_shard_only": _check_writes_only_its_own_shard, } @@ -219,6 +273,16 @@ def test_sharded_tail_tp8_ep8_matches_replicated_projection() -> None: _run_ranks("matches_replicated_ep", 8) +@multi_gpu_test(num_gpus=4) +def test_presharded_tail_tp4_matches_replicated_projection() -> None: + _run_ranks("presharded_matches_replicated", 4) + + +@multi_gpu_test(num_gpus=4) +def test_presharded_unfused_tail_gathers_the_full_hidden_dim() -> None: + _run_ranks("unfused_gathers", 4) + + @multi_gpu_test(num_gpus=4) def test_sharded_tail_tp4_writes_only_its_own_shard() -> None: _run_ranks("own_shard_only", 4) diff --git a/tests/models/kimi_k3/test_amd_mla_direct_return.py b/tests/models/kimi_k3/test_amd_mla_direct_return.py index a1c120737d..50a52f4fe1 100644 --- a/tests/models/kimi_k3/test_amd_mla_direct_return.py +++ b/tests/models/kimi_k3/test_amd_mla_direct_return.py @@ -1,6 +1,9 @@ # SPDX-License-Identifier: Apache-2.0 # SPDX-FileCopyrightText: Copyright contributors to the vLLM project +from types import SimpleNamespace + +import pytest import torch from torch import nn @@ -36,3 +39,81 @@ def test_mla_self_attention_returns_projection_storage_directly() -> None: assert output is projected assert output.data_ptr() == projected.data_ptr() + + +@pytest.mark.parametrize("quantized", [False, True]) +@pytest.mark.parametrize("fused_decode", [False, True]) +def test_mla_forward_preserves_output_shape_with_fp8_input( + monkeypatch, quantized: bool, fused_decode: bool +) -> None: + """Both MLA paths accept producer-quantized input after decode fusion.""" + from vllm.model_executor.layers.fusion.quant_activation import QuantizedActivation + from vllm.model_executor.layers.quantization.utils.quant_utils import ( + kFp8DynamicTokenSym, + ) + from vllm.models.kimi_k3.amd.mla import KimiK3MultiHeadLatentAttentionWrapper + + hidden_states = torch.randn(4, 8, dtype=torch.bfloat16) + if quantized: + hidden_states = QuantizedActivation( + hidden_states.to(kFp8DynamicTokenSym.dtype), + torch.ones(4, 1), + hidden_states.dtype, + hidden_states.shape, + kFp8DynamicTokenSym, + ) + projected = torch.randn(4, 8, dtype=torch.bfloat16) + paths = [] + + def input_projection(value): + assert value is hidden_states + return torch.zeros(4, 8, dtype=torch.bfloat16), None + + def attention_output(path, output_shape): + assert output_shape == projected.shape + paths.append(path) + return projected + + attention = SimpleNamespace( + q_lora_rank=2, + kv_lora_rank=4, + qk_rope_head_dim=2, + qk_head_dim=4, + num_heads=2, + v_head_dim=4, + fused_qkv_a_proj=input_projection, + q_a_layernorm=nn.Identity(), + _normalize_q_kv=lambda q, kv: (q, kv), + q_b_proj=lambda q: (torch.zeros(4, 8, dtype=q.dtype), None), + rotary_emb=None, + indexer=None, + dcp_q_replicate=False, + _fused_qk_prep=fused_decode, + _fused_decode=lambda q, kv, pe, pos, shape, *args: attention_output( + "fused", shape + ), + g_proj=None, + o_proj=lambda value: (value, None), + ) + if fused_decode: + attention.mla_attn = SimpleNamespace(layer_name="test") + monkeypatch.setattr( + "vllm.models.kimi_k3.amd.mla.get_attention_context", + lambda _: ( + SimpleNamespace(num_actual_tokens=4, num_decode_tokens=4), + None, + torch.zeros(1), + torch.arange(4), + ), + ) + else: + attention.mla_attn = lambda q, kv, pe, **kwargs: attention_output( + "regular", kwargs["output_shape"] + ) + + output = KimiK3MultiHeadLatentAttentionWrapper.forward( + attention, torch.arange(4), hidden_states + ) + + assert output is projected + assert paths == ["fused" if fused_decode else "regular"] diff --git a/tests/models/kimi_k3/test_gfx942_int4.py b/tests/models/kimi_k3/test_gfx942_int4.py index 20bef41eb8..e38a8ecff9 100644 --- a/tests/models/kimi_k3/test_gfx942_int4.py +++ b/tests/models/kimi_k3/test_gfx942_int4.py @@ -4,7 +4,7 @@ import sys from collections.abc import Callable from types import ModuleType, SimpleNamespace -from unittest.mock import Mock, patch +from unittest.mock import patch import pytest @@ -91,9 +91,7 @@ def test_int4_per_group_32_is_not_an_online_moe_method(): QuantizationConfigArgs(moe={"weight": "int4_per_group_32"}) ) assert ( - config.resolve_quant_method_cls( - Mock(spec=RoutedExperts), "model.layers.0.mlp.experts" - ) + config.resolve_quant_method_cls(RoutedExperts, "model.layers.0.mlp.experts") is None ) @@ -103,6 +101,4 @@ def test_unknown_online_moe_weight_still_raises(): QuantizationConfigArgs(moe={"weight": "fp8_per_token"}) ) with pytest.raises(ValueError, match="online quantization"): - config.resolve_quant_method_cls( - Mock(spec=RoutedExperts), "model.layers.0.mlp.experts" - ) + config.resolve_quant_method_cls(RoutedExperts, "model.layers.0.mlp.experts") diff --git a/tests/models/kimi_k3/test_kda_metadata.py b/tests/models/kimi_k3/test_kda_metadata.py index a90170da95..bda881cc95 100644 --- a/tests/models/kimi_k3/test_kda_metadata.py +++ b/tests/models/kimi_k3/test_kda_metadata.py @@ -100,6 +100,8 @@ def _make_builder( prefix_match_unit: int | None = None, use_eagle: bool = False, disable_eagle_block_drop: bool = False, + cache_hit_alignment_tokens: int | None = None, + hash_block_size: int | None = None, ) -> AttentionMetadataBuilder: vllm_config = create_vllm_config( model_name="Qwen/Qwen3.5-0.8B", @@ -122,6 +124,11 @@ def _make_builder( vllm_config.cache_config.use_replayssm = use_recoverssm vllm_config.cache_config.use_recoverssm = use_recoverssm vllm_config.cache_config.prefix_match_unit = prefix_match_unit + hash_block_size = hash_block_size or prefix_match_unit or mamba_block_size + vllm_config.cache_config.hash_block_size = hash_block_size + vllm_config.cache_config.cache_hit_alignment_tokens = ( + cache_hit_alignment_tokens or hash_block_size + ) builder = builder_cls( kv_cache_spec=MambaSpec( block_size=mamba_block_size, @@ -220,6 +227,8 @@ def test_kda_recoverssm_startup_metadata_flow_without_model(monkeypatch): vllm_config=builder_config, device=DEVICE, ) + builder_config.cache_config.hash_block_size = BLOCK_SIZE + builder_config.cache_config.cache_hit_alignment_tokens = BLOCK_SIZE # An all-prefill speculative batch used to leave an all-false spec mask # alive, then access active_non_spec_mask_cpu before it was initialized. @@ -287,18 +296,45 @@ def test_kda_recoverssm_startup_metadata_flow_without_model(monkeypatch): @pytest.mark.skipif(not torch.cuda.is_available(), reason="requires CUDA") -def test_internal_checkpoint_metadata_targets_last_aligned_boundary(): +@pytest.mark.parametrize( + "batch,prefix_match_unit,hit_alignment,expected_offsets,expected_states", + [ + ( + BatchSpec(seq_lens=[50, 32], query_lens=[50, 16]), + None, + 16, + [48, 0], + [2, NULL_BLOCK_ID], + ), + ( + BatchSpec(seq_lens=[50, 32], query_lens=[50, 16]), + 16, + 16, + [48, 0], + [2, NULL_BLOCK_ID], + ), + (BatchSpec(seq_lens=[530], query_lens=[530]), None, 128, [512], [32]), + (BatchSpec(seq_lens=[530], query_lens=[50]), None, 128, [32], [32]), + (BatchSpec(seq_lens=[530], query_lens=[530]), 16, 16, [528], [32]), + (BatchSpec(seq_lens=[530], query_lens=[50]), 16, 16, [48], [32]), + ], +) +def test_internal_checkpoint_metadata_targets_last_aligned_boundary( + batch, prefix_match_unit, hit_alignment, expected_offsets, expected_states +): device = torch.device("cuda") - batch = BatchSpec(seq_lens=[50, 32], query_lens=[50, 16]) common_attn_metadata = create_common_attn_metadata( batch, BLOCK_SIZE, device, arange_block_indices=True - ).replace(is_prefilling=torch.tensor([True, True])) + ).replace(is_prefilling=torch.tensor([True] * len(batch.seq_lens))) builder = _make_builder( KimiK3KDAMetadataBuilder, num_speculative_tokens=0, full_cuda_graph=False, mamba_cache_mode="align", num_prefill_checkpoint_blocks=1, + prefix_match_unit=prefix_match_unit, + cache_hit_alignment_tokens=hit_alignment, + hash_block_size=16, device=device, ) assert isinstance(builder, KimiK3KDAMetadataBuilder) @@ -313,26 +349,33 @@ def test_internal_checkpoint_metadata_targets_last_aligned_boundary(): assert actual.checkpoint is not None torch.testing.assert_close( actual.checkpoint.state_indices, - torch.tensor([2, NULL_BLOCK_ID], dtype=torch.int32, device=device), + torch.tensor(expected_states, dtype=torch.int32, device=device), ) torch.testing.assert_close( actual.checkpoint.checkpoint_offsets, - torch.tensor([48, 0], dtype=torch.int32, device=device), + torch.tensor(expected_offsets, dtype=torch.int32, device=device), ) @pytest.mark.parametrize( - ("disable_eagle_block_drop", "prefix_match_unit", "expected_offset"), - [(False, 16, 80), (True, 16, 96), (False, 8, None)], + ("seq_len", "disable_eagle_block_drop", "prefix_match_unit", "expected_offset"), + [ + (100, False, 16, 80), + (100, True, 16, 96), + (100, False, 8, None), + (96, False, 16, 80), + (96, True, 16, 80), + ], ) @pytest.mark.skipif(not torch.cuda.is_available(), reason="requires CUDA") def test_spec_internal_checkpoint_metadata_targets_replay_boundary( + seq_len: int, disable_eagle_block_drop: bool, prefix_match_unit: int, expected_offset: int | None, ) -> None: device = torch.device("cuda") - batch = BatchSpec(seq_lens=[100], query_lens=[100]) + batch = BatchSpec(seq_lens=[seq_len], query_lens=[seq_len]) common_attn_metadata = create_common_attn_metadata( batch, BLOCK_SIZE, device, arange_block_indices=True ) @@ -390,6 +433,7 @@ def test_internal_checkpoint_metadata_skips_unaligned_offset(): full_cuda_graph=False, mamba_cache_mode="align", num_prefill_checkpoint_blocks=1, + prefix_match_unit=16, device=device, ) assert isinstance(builder, KimiK3KDAMetadataBuilder) @@ -447,7 +491,7 @@ def test_internal_checkpoint_metadata_skips_unaligned_offset(): 2, False, [False, False], - None, + 1, id="no-scheduled-draft-tokens", ), pytest.param( diff --git a/tests/models/kimi_k3/test_mla_prefill_context.py b/tests/models/kimi_k3/test_mla_prefill_context.py index 6c6e4bf20e..e8cfb014f6 100644 --- a/tests/models/kimi_k3/test_mla_prefill_context.py +++ b/tests/models/kimi_k3/test_mla_prefill_context.py @@ -13,8 +13,10 @@ from types import SimpleNamespace import pytest import torch +import vllm.model_executor.layers.attention.mla_attention as mla_attention from vllm.model_executor.layers.attention.mla_attention import ( MLACommonBaseImpl, + MLACommonImpl, MLACommonPrefillMetadata, build_mla_chunked_context_metadata, ) @@ -317,3 +319,115 @@ def test_fused_context_rejects_an_unquantized_query() -> None: ) with pytest.raises(AssertionError, match="new-token epilogue"): layer._compute_prefill_context(q, SimpleNamespace(prefill=prefill)) + + +# -------------------------------------------------------------------------- +# Decode context parallelism: the generic impl keeps its loop (allgather + +# reorg) and only the per-chunk K/V pack is handed to the layer's fused kernel. +# -------------------------------------------------------------------------- +_DCP = 2 +_DCP_TOKENS = 48 + + +def _run_dcp_context_loop(monkeypatch, *, q_data_type, fused_mla_kv_concat_fn): + """Drive ``MLACommonImpl._context_parallel_compute_prefill_context`` on one + chunk with the collective pieces (gather, allgather, reorg) stubbed out.""" + torch.manual_seed(0) + device = torch.device("cuda") + kv_b_proj = _KVBProj(device, weight_dtype=torch.bfloat16) + workspace = torch.randn( + (_DCP_TOKENS * (_DCP + 1), _ENTRY), dtype=torch.bfloat16, device=device + ) + impl = SimpleNamespace( + kv_cache_dtype="auto", + kv_lora_rank=_KV_LORA_RANK, + qk_rope_head_dim=_QK_ROPE, + qk_nope_head_dim=_QK_NOPE, + v_head_dim=_V_HEAD_DIM, + num_heads=_NUM_HEADS, + kv_b_proj=kv_b_proj, + _use_flashinfer_concat_mla_k=False, + ) + impl._concat_k_nope_k_pe = MLACommonImpl._concat_k_nope_k_pe.__get__(impl) + + monkeypatch.setattr(mla_attention.ops, "cp_gather_cache", lambda **_: None) + monkeypatch.setattr(mla_attention, "_get_kv_b_proj_input_dtype", lambda *_: None) + monkeypatch.setattr( + mla_attention, + "reorg_kvcache", + lambda kv_c, k_pe, **_: (kv_c[:_DCP_TOKENS].squeeze(1), k_pe[:_DCP_TOKENS]), + ) + chunk = SimpleNamespace( + index=0, + padded_local_seq_lens=[_DCP_TOKENS], + local_context_lens_allranks=[[_DCP_TOKENS // _DCP] * _DCP], + padded_local_cu_seq_lens=torch.tensor([0, _DCP_TOKENS], device=device), + padded_local_token_to_seq=torch.zeros( + _DCP_TOKENS, dtype=torch.int32, device=device + ), + local_starts=[0], + num_local_context_tokens=_DCP_TOKENS, + request_slice=slice(0, 1), + num_requests=1, + starts=torch.tensor([0], device=device), + num_context_tokens=_DCP_TOKENS, + max_seq_len=_DCP_TOKENS, + token_slice=slice(0, 4), + ) + backend = _RecordingPrefillBackend() + attn_metadata = SimpleNamespace( + prefill=SimpleNamespace( + prefill_backend=backend, + chunked_context=SimpleNamespace( + workspace=workspace, + chunks=[chunk], + empty_token_slices=False, + dcp_manager=SimpleNamespace(kv_gather=lambda dst, src: None), + ), + q_data_type=q_data_type, + block_table=torch.zeros((1, 1), dtype=torch.int32, device=device), + ) + ) + q = torch.randn( + (4, _NUM_HEADS, _QK_NOPE + _QK_ROPE), dtype=torch.bfloat16, device=device + ) + out, lse = MLACommonImpl._context_parallel_compute_prefill_context( + impl, + q, + torch.empty(0, device=device), + attn_metadata, + k_scale=torch.ones(1, device=device), + dcp_world_size=_DCP, + fused_mla_kv_concat_fn=fused_mla_kv_concat_fn, + ) + return backend, out, lse + + +@pytest.mark.parametrize("kv_cache_dtype", ["auto", "fp8"]) +@torch.inference_mode() +def test_dcp_context_fused_pack_matches_generic_tail( + monkeypatch, kv_cache_dtype: str +) -> None: + """Under DCP the layer's fused K/V pack must feed the prefill backend exactly + what the generic impl's cast + split + concat tail feeds it.""" + q_data_type = ( + current_platform.fp8_dtype() if kv_cache_dtype == "fp8" else torch.bfloat16 + ) + layer = SimpleNamespace(qk_nope_head_dim=_QK_NOPE, v_head_dim=_V_HEAD_DIM) + fused_fn = MultiHeadLatentAttention._fused_mla_kv_concat.__get__(layer) + + backend_ref, ref_out, ref_lse = _run_dcp_context_loop( + monkeypatch, q_data_type=q_data_type, fused_mla_kv_concat_fn=None + ) + backend_fused, fused_out, fused_lse = _run_dcp_context_loop( + monkeypatch, q_data_type=q_data_type, fused_mla_kv_concat_fn=fused_fn + ) + assert len(backend_ref.calls) == len(backend_fused.calls) == 1 + for name, fused_t, ref_t in zip( + ("q", "k", "v"), backend_fused.calls[0], backend_ref.calls[0], strict=True + ): + torch.testing.assert_close( + fused_t, ref_t, atol=0, rtol=0, msg=lambda m, n=name: f"{n} differs: {m}" + ) + torch.testing.assert_close(fused_out, ref_out, atol=0, rtol=0) + torch.testing.assert_close(fused_lse, ref_lse, atol=0, rtol=0) diff --git a/tests/models/kimi_k3/test_prefix_cache.py b/tests/models/kimi_k3/test_prefix_cache.py index 3525550fe9..20e5d6670e 100644 --- a/tests/models/kimi_k3/test_prefix_cache.py +++ b/tests/models/kimi_k3/test_prefix_cache.py @@ -8,6 +8,7 @@ import random from collections.abc import Iterator from dataclasses import dataclass from math import lcm +from uuid import uuid4 import pytest import requests @@ -353,6 +354,45 @@ def _format(turns: list[Turn]) -> str: return "\n".join(rows) +@pytest.mark.parametrize( + "prompt_len", + [ + # Blocks=6144, PMU=128. P=7040 is PMU-aligned but not block-aligned: + # ensure lookup reads the cached attention proof at 7040 despite + # the final hit limit of 7039, then applies EAGLE's 128-token drop. + pytest.param(7040, id="eagle-proof-scan"), + # Blocks=6144, PMU=128. P=7296 is an exact PMU multiple: + # ensure the Mamba checkpoint is 7168, not 7040 from applying both + # the P - 1 cap and EAGLE's 128-token drop. + pytest.param(7296, id="eagle-double-cap"), + ], +) +def test_dspark_exact_resend_reuses_prompt_checkpoint(prompt_len: int) -> None: + """Resends reuse the Mamba checkpoint and its companion EAGLE attention proof.""" + mode = Mode(128, True) + salt = str(uuid4()) + rng = random.Random(0) + prompt = [rng.randint(1000, 150000) for _ in range(prompt_len)] + deployment = DEPLOYMENTS["plain"] + with _serve(deployment, mode) as servers: + url = servers[0].url_root + cold = _complete(url, prompt, salt, max_tokens=1) + resend = _complete(url, prompt, salt, max_tokens=1) + recompute = _complete(url, prompt, f"{salt}-recompute", max_tokens=1) + + # EAGLE's dropped hash unit already leaves tokens to recompute on resend. + expected = (prompt_len // 128 - 1) * 128 + actual = _cached(resend) + assert _cached(cold) == _cached(recompute) == 0 + assert actual == expected, f"{prompt_len=}: {actual=}, {expected=}" + check_logprobs_close( + outputs_0_lst=[_tokens_text_logprobs(recompute)], + outputs_1_lst=[_tokens_text_logprobs(resend)], + name_0="recompute", + name_1="resend", + ) + + @pytest.mark.parametrize( "name, mode", [_case(name, mode) for name in DEPLOYMENTS for mode in MODES] ) diff --git a/tests/models/kimi_k3/test_sequence_parallel.py b/tests/models/kimi_k3/test_sequence_parallel.py index 9e8faea099..3057f85c7e 100644 --- a/tests/models/kimi_k3/test_sequence_parallel.py +++ b/tests/models/kimi_k3/test_sequence_parallel.py @@ -301,6 +301,27 @@ def test_shard_sequence_parallel_mlp_gating( ) +@pytest.mark.parametrize( + ("moe_backend", "all2all_backend", "expected"), + [ + ("deep_gemm_mega_moe", "naive", True), + ("auto", "deepep_v2", True), + ("auto", "deepep_low_latency", False), + ], +) +def test_shared_expert_sharding_backend_gate( + moe_backend: str, + all2all_backend: str, + expected: bool, +): + vllm_config = SimpleNamespace( + kernel_config=SimpleNamespace(moe_backend=moe_backend), + parallel_config=SimpleNamespace(all2all_backend=all2all_backend), + ) + + assert kimi_model.can_shard_sequence_parallel_shared_expert(vllm_config) is expected + + def test_sharded_sequence_parallel_mlp_matches_replicated(default_vllm_config): """Sharded SP MLP must reproduce the replicated result for every token. diff --git a/tests/models/language/pooling_mteb_test/test_zerank.py b/tests/models/language/pooling_mteb_test/test_zerank.py new file mode 100644 index 0000000000..c90c80bc4d --- /dev/null +++ b/tests/models/language/pooling_mteb_test/test_zerank.py @@ -0,0 +1,95 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +from typing import Any + +import mteb +import numpy as np +import pytest +import torch +from torch.utils.data import DataLoader + +from tests.conftest import HfRunner +from tests.models.utils import RerankModelInfo + +from .mteb_score_utils import MtebCrossEncoderMixin, mteb_test_rerank_models + +RERANK_MODELS = [ + RerankModelInfo( + "polaria-tech/zerank-2-reranker-vllm", + architecture="Qwen3ForSequenceClassification", + chat_template_name="zerank2.jinja", + seq_pooling_type="LAST", + attn_type="decoder", + is_prefix_caching_supported=True, + is_chunked_prefill_supported=True, + mteb_score=0.34305, + mteb_tol=1e-2, + enable_test=True, + ), +] + + +class ZerankHfRunner(MtebCrossEncoderMixin, HfRunner): + # The converted checkpoint has no lm_head, so the reference is the original + # causal LM: the logit of the "Yes" token at the last position. The converted + # checkpoint returns sigmoid(logit / 5), which ranks documents identically. + original_model = "zeroentropy/zerank-2-reranker" + + def __init__( + self, model_name: str, dtype: str = "auto", *args: Any, **kwargs: Any + ) -> None: + from transformers import AutoModelForCausalLM, AutoTokenizer + + HfRunner.__init__( + self, + model_name=self.original_model, + auto_cls=AutoModelForCausalLM, + dtype=dtype, + **kwargs, + ) + + self.tokenizer = AutoTokenizer.from_pretrained( + self.original_model, padding_side="left" + ) + self.token_true_id = self.tokenizer.convert_tokens_to_ids("Yes") + + @torch.no_grad + def predict( + self, + inputs1: DataLoader[mteb.types.BatchedInput], + inputs2: DataLoader[mteb.types.BatchedInput], + *args, + **kwargs, + ) -> np.ndarray: + queries = [text for batch in inputs1 for text in batch["text"]] + corpus = [text for batch in inputs2 for text in batch["text"]] + + tokenizer = self.tokenizer + prompts = [] + for query, document in zip(queries, corpus): + conversation = [ + {"role": "query", "content": query}, + {"role": "document", "content": document}, + ] + + prompt = tokenizer.apply_chat_template( + conversation=conversation, + tools=None, + chat_template=self.chat_template, + tokenize=False, + ) + prompts.append(prompt) + + scores = [] + for prompt in prompts: + inputs = tokenizer([prompt], return_tensors="pt") + inputs = self.wrap_device(inputs) + logits = self.model(**inputs).logits[:, -1, :] + scores.append(logits[0, self.token_true_id].item()) + return torch.Tensor(scores) + + +@pytest.mark.flaky(reruns=2) +@pytest.mark.parametrize("model_info", RERANK_MODELS) +def test_rerank_models_mteb(vllm_runner, model_info: RerankModelInfo) -> None: + mteb_test_rerank_models(vllm_runner, model_info, hf_runner=ZerankHfRunner) diff --git a/tests/models/multimodal/generation/test_common.py b/tests/models/multimodal/generation/test_common.py index 32a9adda88..9ea2cc43cd 100644 --- a/tests/models/multimodal/generation/test_common.py +++ b/tests/models/multimodal/generation/test_common.py @@ -9,14 +9,12 @@ from collections import defaultdict from pathlib import PosixPath import pytest -from packaging.version import Version from transformers import ( AutoModel, AutoModelForCausalLM, AutoModelForImageTextToText, AutoModelForTextToWaveform, ) -from transformers import __version__ as TRANSFORMERS_VERSION from vllm.platforms import current_platform from vllm.utils.func_utils import identity @@ -818,8 +816,7 @@ VLM_TEST_SETTINGS = { patch_hf_runner=model_utils.paddleocr_vl_patch_hf_runner, image_size_factors=[(0.25,)], marks=[ - pytest.mark.skipif( - Version(TRANSFORMERS_VERSION) >= Version("5.0.0"), + pytest.mark.skip( reason="Model's custom code uses ROPE_INIT_FUNCTIONS" "['default'] which was removed in transformers v5", ), diff --git a/tests/models/multimodal/generation/test_nemotron_parse.py b/tests/models/multimodal/generation/test_nemotron_parse.py index 69e41b21e3..57e40430c6 100644 --- a/tests/models/multimodal/generation/test_nemotron_parse.py +++ b/tests/models/multimodal/generation/test_nemotron_parse.py @@ -104,7 +104,16 @@ def run_test( ) -@pytest.mark.parametrize("model", ["nvidia/NVIDIA-Nemotron-Parse-v1.2"]) +@pytest.mark.parametrize( + "model", + [ + "nvidia/NVIDIA-Nemotron-Parse-v1.2", + # v2.0's checkpoint ties lm_head to the decoder's input embeddings + # instead of shipping a separate lm_head.weight tensor; this + # exercises that path (see tie_weights() in nemotron_parse.py). + "nvidia/NVIDIA-Nemotron-Parse-2.0", + ], +) @pytest.mark.parametrize("dtype", ["bfloat16"]) @pytest.mark.parametrize("num_logprobs", [5]) def test_models( diff --git a/tests/models/multimodal/generation/test_transformers_video.py b/tests/models/multimodal/generation/test_transformers_video.py index bbb31b7be8..c6c47f9ac0 100644 --- a/tests/models/multimodal/generation/test_transformers_video.py +++ b/tests/models/multimodal/generation/test_transformers_video.py @@ -36,6 +36,7 @@ VIDEO_MODEL_SETTINGS: dict[str, dict[str, Any]] = { "<|VISION_START|><|VIDEO_PAD|><|VISION_END|>Describe this video." "<|END_OF_TURN_TOKEN|><|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|>" ), + "patch_hf_runner": model_utils.cohere_compass_patch_hf_runner, }, "Qwen/Qwen2.5-VL-3B-Instruct": { "prompt": ( @@ -88,6 +89,9 @@ def test_transformers_video_generation( model_id, dtype="bfloat16", auto_cls=AutoModelForImageTextToText ) as hf_model: hf_model = model_utils.qwen3_vl_patch_hf_runner(hf_model) + patch_hf_runner = VIDEO_MODEL_SETTINGS[model_id].get("patch_hf_runner") + if patch_hf_runner is not None: + hf_model = patch_hf_runner(hf_model) hf_outputs = hf_model.generate_greedy_logprobs_limit( [prompt], 128, num_logprobs=10, videos=[video] ) diff --git a/tests/models/multimodal/generation/test_voxtral.py b/tests/models/multimodal/generation/test_voxtral.py index 544c06020f..3c9d2ace8c 100644 --- a/tests/models/multimodal/generation/test_voxtral.py +++ b/tests/models/multimodal/generation/test_voxtral.py @@ -146,10 +146,6 @@ def test_online_serving(vllm_runner, audio_assets: AudioTestAssets): ) -@pytest.mark.skip( - reason="VoxtralProcessor.apply_chat_template() in transformers v5 " - "doesn't resolve chat_template=None to the default template" -) def test_hf_reference(hf_runner, vllm_runner, audio_assets: AudioTestAssets): """Compare vLLM Mistral-format output against HF Transformers reference. @@ -157,9 +153,6 @@ def test_hf_reference(hf_runner, vllm_runner, audio_assets: AudioTestAssets): attention backends), we compare per-token logprobs using the standard check_logprobs_close helper: when tokens diverge at a position, each runner's chosen token must appear in the other's top-k logprobs. - - Marked xfail(strict=False) so remaining edge-case mismatches - don't block CI. """ question = f"What's happening in these {len(audio_assets)} audio clips?" max_tokens = 10 diff --git a/tests/models/multimodal/generation/test_voxtral_realtime.py b/tests/models/multimodal/generation/test_voxtral_realtime.py index 09432b4494..e05930b0f3 100644 --- a/tests/models/multimodal/generation/test_voxtral_realtime.py +++ b/tests/models/multimodal/generation/test_voxtral_realtime.py @@ -106,7 +106,8 @@ def tokenizer() -> MistralTokenizer: @pytest_asyncio.fixture -async def async_engine(): +async def async_engine(request): + engine_config = {**ENGINE_CONFIG, **getattr(request, "param", {})} gpu_memory_utilization = ENGINE_CONFIG.get("gpu_memory_utilization", 0.9) from vllm.platforms import current_platform @@ -115,7 +116,7 @@ async def async_engine(): wait_for_memory_to_settle(threshold_ratio=1.0 - gpu_memory_utilization) - engine_args = AsyncEngineArgs(**ENGINE_CONFIG) + engine_args = AsyncEngineArgs(**engine_config) llm = AsyncLLM.from_engine_args(engine_args) try: yield llm @@ -275,3 +276,82 @@ async def test_voxtral_realtime_generator(audio_assets, tokenizer, async_engine) f" got: {got!r}\n" f" expected: {expected!r}" ) + + +@pytest.mark.asyncio +async def test_voxtral_realtime_buffer_stops_at_max_model_len(tokenizer): + from vllm.model_executor.models.voxtral_realtime import VoxtralRealtimeBuffer + + audio_config = tokenizer.instruct_tokenizer.audio_encoder.audio_config + audio = Audio.from_file(AudioAsset("winning_call").get_local_path(), strict=False) + req = TranscriptionRequest( + streaming=StreamingMode.OFFLINE, + audio=audio.to_base64(audio.format), + language=None, + ) + audio_enc = tokenizer.encode_transcription(req) + + max_model_len = 64 + buffer = VoxtralRealtimeBuffer( + audio_config, audio_enc.tokens, max_model_len=max_model_len + ) + await buffer.append_audio(audio_enc.audios[0].audio_array) + await buffer.append_audio(None) + # Unlimited feedback tokens so only the length cap can stop the stream. + await buffer.append_tokens([0] * max_model_len) + + n_tokens = 0 + async for prompt in buffer.get_input_stream(): + n_tokens += len(prompt["prompt_token_ids"]) + + assert len(audio_enc.tokens) < n_tokens < max_model_len + + +@pytest.mark.asyncio +@pytest.mark.parametrize("async_engine", [{"max_model_len": 128}], indirect=True) +async def test_voxtral_realtime_generator_max_model_len( + audio_assets, tokenizer, async_engine +): + # Lazy import to avoid CUDA-reinitialization error + from vllm.model_executor.models.voxtral_realtime import VoxtralRealtimeBuffer + + max_model_len = async_engine.model_config.max_model_len + sampling_params = SamplingParams(temperature=0.0, max_tokens=1) + audio_config = tokenizer.instruct_tokenizer.audio_encoder.audio_config + + async def input_stream(audio_buffer): + async for prompt in audio_buffer.get_input_stream(): + parsed_prompt = parse_model_prompt(async_engine.model_config, prompt) + (engine_input,) = await async_engine.renderer.render_cmpl_async( + [parsed_prompt] + ) + yield StreamingInput(prompt=engine_input) + + # winning_call is ~300 tokens long, well past max_model_len. + audio = Audio.from_file(audio_assets[1].get_local_path(), strict=False) + req = TranscriptionRequest( + streaming=StreamingMode.OFFLINE, + audio=audio.to_base64(audio.format), + language=None, + ) + audio_enc = tokenizer.encode_transcription(req) + buffer = VoxtralRealtimeBuffer( + audio_config, audio_enc.tokens, max_model_len=max_model_len + ) + await buffer.append_audio(audio_enc.audios[0].audio_array) + await buffer.append_audio(None) + + output_tokens = [] + async for resp in async_engine.generate( + prompt=input_stream(buffer), + sampling_params=sampling_params, + request_id="session-max-len", + ): + tokens = resp.outputs[0].token_ids[-1:] + output_tokens.extend(tokens) + await buffer.append_tokens(tokens) + + assert resp.finished + assert 0 < len(output_tokens) < max_model_len + # The engine is still healthy after the session hit the cap. + assert not async_engine.errored diff --git a/tests/models/multimodal/generation/vlm_utils/model_utils.py b/tests/models/multimodal/generation/vlm_utils/model_utils.py index bc472c400f..9acbe2c0d6 100644 --- a/tests/models/multimodal/generation/vlm_utils/model_utils.py +++ b/tests/models/multimodal/generation/vlm_utils/model_utils.py @@ -1111,6 +1111,19 @@ def qwen3_vl_patch_hf_runner(hf_model: HfRunner) -> HfRunner: return hf_model +def cohere_compass_patch_hf_runner(hf_model: HfRunner) -> HfRunner: + """Declare the video modality that CohereCompass omits upstream. + + Without it ``generate`` leaves ``pixel_values_videos`` unconsumed and the + forward pass rejects it, see huggingface/transformers#49320. Drop this once + the pinned transformers declares the modality itself. + """ + model = hf_model.model + if "video" not in model.input_modalities: + model.input_modalities = (*model.input_modalities, "video") + return hf_model + + def voxtral_patch_hf_runner(hf_model: "HfRunner") -> "HfRunner": """Patch HfRunner for Voxtral's conversation-based processor. diff --git a/tests/models/multimodal/pooling/test_embedding_gemma2.py b/tests/models/multimodal/pooling/test_embedding_gemma2.py new file mode 100644 index 0000000000..0dcb72406d --- /dev/null +++ b/tests/models/multimodal/pooling/test_embedding_gemma2.py @@ -0,0 +1,660 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import ast +import inspect +import os +import textwrap + +import pytest +import torch + +MODEL = os.environ.get("EG2_MODEL_PATH") +needs_model = pytest.mark.skipif(not MODEL, reason="EG2_MODEL_PATH not set") +TEXT_COS, MM_COS = 0.999, 0.999 + + +def _cos(a, b): + return torch.nn.functional.cosine_similarity( + torch.tensor(a, dtype=torch.float32), + torch.tensor(b, dtype=torch.float32), + dim=-1, + ) + + +def _ensure_synthetic_video( + path: str, duration: float, fps: float, width: int, height: int +) -> str: + if os.path.exists(path): + os.remove(path) + import cv2 + import numpy as np + + fourcc_fn = getattr(cv2, "VideoWriter_fourcc", cv2.VideoWriter.fourcc) + fourcc = fourcc_fn(*"mp4v") + num_frames = int(duration * fps) + writer = cv2.VideoWriter(path, fourcc, fps, (width, height)) + y_coords, x_coords = np.mgrid[0:height, 0:width] + for i in range(num_frames): + # Spatially varying color pattern that evolves dynamically over frames + frame: np.ndarray = np.zeros((height, width, 3), dtype=np.uint8) + frame[..., 0] = ((x_coords * 255 // max(1, width)) + i * 17) % 256 + frame[..., 1] = ((y_coords * 255 // max(1, height)) + i * 29) % 256 + frame[..., 2] = (x_coords + y_coords + i * 13) % 256 + writer.write(frame) + writer.release() + return path + + +@pytest.fixture(scope="module") +def st_model(): + from sentence_transformers import SentenceTransformer + + return SentenceTransformer( + MODEL, device="cuda", model_kwargs={"torch_dtype": torch.bfloat16} + ) + + +@needs_model +def test_structure(vllm_runner, monkeypatch): + monkeypatch.setenv("VLLM_ENABLE_V1_MULTIPROCESSING", "0") + with vllm_runner(MODEL, runner="pooling", max_model_len=8192) as vm: + + def probe(model): + lm = model.language_model + win = [layer.self_attn.attn.impl.sliding_window for layer in lm.layers] + return win, model.config.text_config.sliding_window + + win, cfg_w = vm.apply_model(probe)[0] + assert (cfg_w, cfg_w) in win # (512, 512) == W-1 with W = 513 + + +@needs_model +@pytest.mark.parametrize( + "prompt", + [ + "short query", + "task: search result | query: what is vllm", + "word " * 700, # > 512: crosses sliding window + "word " * 1500, # > 1024 + ], +) +def test_text_parity(vllm_runner, st_model, prompt): + ref = st_model.encode([prompt], normalize_embeddings=True) + with vllm_runner(MODEL, runner="pooling", max_model_len=8192) as vm: + out = vm.embed([prompt]) + sim = _cos(out, ref).min().item() + assert sim >= TEXT_COS, ( + f"Text cosine {sim} < {TEXT_COS} for prompt len {len(prompt)}" + ) + + +@needs_model +def test_mm_recipe(vllm_runner, st_model): + from PIL import Image + + img = Image.new("RGB", (224, 224), color=(255, 0, 0)) + prompt = "Describe this image: <|image|>" + + # SentenceTransformer multimodal encode + ref = st_model.encode([{"image": img, "text": prompt}], normalize_embeddings=True) + with vllm_runner(MODEL, runner="pooling", max_model_len=8192) as vm: + out = vm.embed([{"prompt": prompt, "multi_modal_data": {"image": img}}]) + sim = _cos(out, ref).min().item() + assert sim >= MM_COS, f"Image cosine {sim} < {MM_COS}" + + +@needs_model +def test_natural_image_parity(vllm_runner, st_model): + from PIL import Image + + img_path = "/projects/gemma4-vllm/images/cat.jpg" + if not os.path.exists(img_path): + pytest.skip(f"Test asset {img_path} not found") + img = Image.open(img_path) + prompt = "What is this image? <|image|>" + ref = st_model.encode([{"image": img, "text": prompt}], normalize_embeddings=True) + with vllm_runner(MODEL, runner="pooling", max_model_len=8192) as vm: + out = vm.embed([{"prompt": prompt, "multi_modal_data": {"image": img}}]) + sim = _cos(out, ref).min().item() + assert sim >= MM_COS, f"Natural image cosine {sim} < {MM_COS}" + + +@needs_model +def test_audio_parity(vllm_runner, st_model): + import torchaudio + + audio_path = "/projects/gemma4-vllm/audio/tellMeAboutTheSun.wav" + if not os.path.exists(audio_path): + pytest.skip(f"Test asset {audio_path} not found") + wav, sr = torchaudio.load(audio_path) + mono_wav = wav.mean(dim=0).numpy() + prompt = "Describe this audio: <|audio|>" + ref = st_model.encode( + [{"audio": audio_path, "text": prompt}], normalize_embeddings=True + ) + with vllm_runner(MODEL, runner="pooling", max_model_len=8192) as vm: + out = vm.embed( + [{"prompt": prompt, "multi_modal_data": {"audio": (mono_wav, sr)}}] + ) + sim = _cos(out, ref).min().item() + assert sim >= MM_COS, f"Audio cosine {sim} < {MM_COS}" + + +@needs_model +def test_mixed_parity(vllm_runner, st_model): + import torchaudio + from PIL import Image + + img_path = "/projects/gemma4-vllm/images/cat.jpg" + audio_path = "/projects/gemma4-vllm/audio/tellMeAboutTheSun.wav" + if not os.path.exists(img_path) or not os.path.exists(audio_path): + pytest.skip("Test assets for mixed parity not found") + img = Image.open(img_path) + wav, sr = torchaudio.load(audio_path) + mono_wav = wav.mean(dim=0).numpy() + prompt = "Describe image <|image|> and audio <|audio|>" + ref = st_model.encode( + [{"image": img, "audio": audio_path, "text": prompt}], + normalize_embeddings=True, + ) + with vllm_runner(MODEL, runner="pooling", max_model_len=8192) as vm: + out = vm.embed( + [ + { + "prompt": prompt, + "multi_modal_data": { + "image": img, + "audio": (mono_wav, sr), + }, + } + ] + ) + sim = _cos(out, ref).min().item() + assert sim >= MM_COS, f"Mixed cosine {sim} < {MM_COS}" + + +@needs_model +def test_video_parity_uniform_overflow(vllm_runner, st_model): + from transformers.video_utils import load_video + + video_path = _ensure_synthetic_video( + "/tmp/test_40s.mp4", duration=40.0, fps=10.0, width=160, height=120 + ) + frames, meta = load_video(video_path, backend="opencv") + prompt = "Describe this video: <|video|>" + ref = st_model.encode( + [{"video": video_path, "text": prompt}], normalize_embeddings=True + ) + with vllm_runner(MODEL, runner="pooling", max_model_len=8192) as vm: + out = vm.embed( + [{"prompt": prompt, "multi_modal_data": {"video": (frames, meta)}}] + ) + sim = _cos(out, ref).min().item() + assert sim >= MM_COS, f"40s video cosine {sim} < {MM_COS}" + + +@needs_model +def test_video_parity_portrait(vllm_runner, st_model): + from transformers.video_utils import load_video + + video_path = _ensure_synthetic_video( + "/tmp/test_portrait.mp4", duration=3.0, fps=10.0, width=120, height=160 + ) + frames, meta = load_video(video_path, backend="opencv") + prompt = "Describe this video: <|video|>" + ref = st_model.encode( + [{"video": video_path, "text": prompt}], normalize_embeddings=True + ) + with vllm_runner(MODEL, runner="pooling", max_model_len=8192) as vm: + out = vm.embed( + [{"prompt": prompt, "multi_modal_data": {"video": (frames, meta)}}] + ) + sim = _cos(out, ref).min().item() + assert sim >= MM_COS, f"Portrait video cosine {sim} < {MM_COS}" + + +@needs_model +def test_tower_skip(vllm_runner, monkeypatch): + monkeypatch.setenv("VLLM_ENABLE_V1_MULTIPROCESSING", "0") + + def nparams(m): + return sum(p.numel() for p in m.parameters()) / 1e6 + + with vllm_runner( + MODEL, + runner="pooling", + max_model_len=2048, + limit_mm_per_prompt={"image": 0, "audio": 0, "video": 0}, + ) as vm: + p_text = vm.apply_model(nparams)[0] + assert abs(p_text - 271.00) <= 0.05, ( + f"Expected 271.00M text params, got {p_text}M" + ) + + with vllm_runner(MODEL, runner="pooling", max_model_len=2048) as vm: + p_full = vm.apply_model(nparams)[0] + assert abs(p_full - 744.37) <= 0.05, ( + f"Expected 744.37M full params, got {p_full}M" + ) + + +def test_encoder_contract(): + from vllm.model_executor.models.embedding_gemma2 import EmbeddingGemma2Model + from vllm.model_executor.models.gemma4_mm import ( + Gemma4ForConditionalGeneration as G, + ) + + borrowed_methods = [ + "_parse_and_validate_multimodal_inputs", + "_parse_and_validate_image_input", + "_parse_and_validate_video_input", + "_parse_and_validate_audio_input", + "_process_image_input", + "_process_video_input", + "_process_audio_input", + "embed_multimodal", + "_encoder_chunk", + ] + + for name in borrowed_methods: + # Check method identity (L-F6 + T-F4) + assert getattr(EmbeddingGemma2Model, name) is getattr(G, name) + + # AST checks: verify no zero-arg super() calls in borrowed methods + func = getattr(EmbeddingGemma2Model, name) + src = textwrap.dedent(inspect.getsource(func)) + tree = ast.parse(src) + for node in ast.walk(tree): + if ( + isinstance(node, ast.Call) + and isinstance(node.func, ast.Name) + and node.func.id == "super" + ): + assert len(node.args) > 0, ( + f"Zero-arg super() call found in borrowed method {name}" + ) + + # AST check: all self. non-callable accesses are assigned in __init__ + init_src = inspect.getsource(EmbeddingGemma2Model.__init__) + for name in borrowed_methods: + func = getattr(EmbeddingGemma2Model, name) + src = textwrap.dedent(inspect.getsource(func)) + tree = ast.parse(src) + for node in ast.walk(tree): + if ( + isinstance(node, ast.Attribute) + and isinstance(node.value, ast.Name) + and node.value.id == "self" + ): + attr = node.attr + assert hasattr(EmbeddingGemma2Model, attr) or attr in init_src, ( + f"Attribute {attr} accessed on self in {name} is neither " + f"defined on class nor in __init__" + ) + + +@needs_model +def test_instance_attribute_access(vllm_runner): + """Check self. accesses on a real model instance.""" + from vllm.model_executor.models.embedding_gemma2 import EmbeddingGemma2Model + + borrowed_methods = [ + "_parse_and_validate_multimodal_inputs", + "_parse_and_validate_image_input", + "_parse_and_validate_video_input", + "_parse_and_validate_audio_input", + "_process_image_input", + "_process_video_input", + "_process_audio_input", + "embed_multimodal", + "_encoder_chunk", + ] + attrs_accessed = set() + for name in borrowed_methods: + func = getattr(EmbeddingGemma2Model, name) + src = textwrap.dedent(inspect.getsource(func)) + tree = ast.parse(src) + for node in ast.walk(tree): + if ( + isinstance(node, ast.Attribute) + and isinstance(node.value, ast.Name) + and node.value.id == "self" + ): + attrs_accessed.add(node.attr) + + def check_attrs(model): + for attr in attrs_accessed: + assert hasattr(model, attr), ( + f"Real instance of EmbeddingGemma2Model is missing attribute '{attr}' " + f"accessed by borrowed method" + ) + return True + + with vllm_runner(MODEL, runner="pooling", max_model_len=2048) as vm: + res = vm.apply_model(check_attrs) + assert res[0] is True + + +@needs_model +def test_video_loader_real_video(vllm_runner, st_model): + """Decode a real MP4 file through EmbeddingGemma2VideoBackend.load_bytes + + and VideoMediaIO / MediaConnector with default num_frames=32. + """ + from vllm.multimodal.media import MediaConnector + from vllm.multimodal.media.video import VideoMediaIO + from vllm.multimodal.video import VIDEO_LOADER_REGISTRY + + video_path = "/projects/gemma4-vllm/videos/test_intel.mp4" + backend_cls = VIDEO_LOADER_REGISTRY.name2class["embedding_gemma2"] + with open(video_path, "rb") as f: + data = f.read() + + # Direct backend load + frames, meta = backend_cls.load_bytes(data) + + # VideoMediaIO with default num_frames=32 (VideoMediaIO passes num_frames=32) + video_io = VideoMediaIO(image_io=None, video_backend="embedding_gemma2") + loaded = video_io.load_bytes(data) + frames_io, meta_io = loaded.media + assert len(frames_io) == len(frames) + assert meta_io.get("do_sample_frames") is False + + # MediaConnector with video_processor="EmbeddingGemma2VideoProcessor" + connector = MediaConnector(allowed_local_media_path="/projects/gemma4-vllm/videos") + fetched = connector.fetch_video( + f"file://{video_path}", + video_processor="EmbeddingGemma2VideoProcessor", + ) + frames_conn, meta_conn = fetched.media + assert len(frames_conn) == 32 + assert meta_conn.get("do_sample_frames") is False + + prompt = "Describe this video: <|video|>" + ref = st_model.encode( + [{"video": video_path, "text": prompt}], normalize_embeddings=True + ) + with vllm_runner(MODEL, runner="pooling", max_model_len=8192) as vm: + out = vm.embed( + [ + { + "prompt": prompt, + "multi_modal_data": {"video": (frames_conn, meta_conn)}, + } + ] + ) + sim = _cos(out, ref).min().item() + assert sim >= MM_COS, f"Real video loader cosine {sim} < {MM_COS}" + + +@needs_model +def test_video_loader_duplicate_expansion_e2e(vllm_runner, st_model): + """End-to-end duplicate expansion test with low-fps clip.""" + import numpy as np + + from vllm.multimodal.video import VIDEO_LOADER_REGISTRY + + backend_cls = VIDEO_LOADER_REGISTRY.name2class["embedding_gemma2"] + # 4s clip at 0.5 fps = 2 unique frames + lowfps_path = "/tmp/test_lowfps_05.mp4" + _ensure_synthetic_video(lowfps_path, duration=4.0, fps=0.5, width=64, height=48) + with open(lowfps_path, "rb") as f: + data = f.read() + # Default fps=1.0 -> 4 frames expanding 2 unique frames to [0, 0, 1, 1] + frames, meta = backend_cls.load_bytes(data) + assert frames.shape[0] == 4 + # Compare returned frames directly, not only indices + assert np.array_equal(frames[0], frames[1]) + assert not np.array_equal(frames[0], frames[2]) + assert np.array_equal(frames[2], frames[3]) + + prompt = "Describe this video: <|video|>" + with vllm_runner(MODEL, runner="pooling", max_model_len=8192) as vm: + out = vm.embed( + [{"prompt": prompt, "multi_modal_data": {"video": (frames, meta)}}] + ) + ref = st_model.encode( + [{"video": lowfps_path, "text": prompt}], normalize_embeddings=True + ) + sim = _cos(out, ref).min().item() + assert sim >= MM_COS, f"Duplicate expansion cosine {sim} < {MM_COS}" + assert len(out) == 1 + assert len(out[0]) == 768 + + +def test_video_loader_indices(): + from transformers.models.embedding_gemma2.video_processing_embedding_gemma2 import ( + EmbeddingGemma2VideoProcessor, + ) + from transformers.video_utils import VideoMetadata + + from vllm.multimodal.video import ( + VIDEO_LOADER_REGISTRY, + VideoSourceMetadata, + VideoTargetMetadata, + ) + + backend_cls = VIDEO_LOADER_REGISTRY.name2class["embedding_gemma2"] + hf_proc = EmbeddingGemma2VideoProcessor() + + # Case 1: clip shorter than 32s (e.g. 5s @ 24fps = 120 frames) + src_5s = VideoSourceMetadata(total_frames_num=120, original_fps=24.0, duration=5.0) + tgt = VideoTargetMetadata(num_frames=-1, fps=1.0, max_duration=300) + vllm_indices = backend_cls.compute_frames_index_to_sample(src_5s, tgt) + hf_meta_5s = VideoMetadata(fps=24.0, duration=5.0, total_num_frames=120) + hf_indices_5s = hf_proc.sample_frames( + hf_meta_5s, fps=1.0, max_frames=32, overflow_strategy="uniform" + ).tolist() + assert vllm_indices == hf_indices_5s + + # Case 2: clip shorter than 1s (e.g. 0.5s @ 24fps = 12 frames) + src_sub1s = VideoSourceMetadata( + total_frames_num=12, original_fps=24.0, duration=0.5 + ) + vllm_sub1s = backend_cls.compute_frames_index_to_sample(src_sub1s, tgt) + hf_meta_sub1s = VideoMetadata(fps=24.0, duration=0.5, total_num_frames=12) + hf_indices_sub1s = hf_proc.sample_frames( + hf_meta_sub1s, fps=1.0, max_frames=32, overflow_strategy="uniform" + ).tolist() + assert vllm_sub1s == hf_indices_sub1s + + # Case 3: clip longer than 32s (e.g. 40s @ 10fps = 400 frames) + # -> uniform overflow cap 32 frames + src_40s = VideoSourceMetadata( + total_frames_num=400, original_fps=10.0, duration=40.0 + ) + vllm_40s = backend_cls.compute_frames_index_to_sample(src_40s, tgt) + hf_meta_40s = VideoMetadata(fps=10.0, duration=40.0, total_num_frames=400) + hf_indices_40s = hf_proc.sample_frames( + hf_meta_40s, fps=1.0, max_frames=32, overflow_strategy="uniform" + ).tolist() + assert vllm_40s == hf_indices_40s + assert len(vllm_40s) == 32 + + # Case 4: default target fps (target.fps <= 0 defaults to 1.0) + tgt_default_fps = VideoTargetMetadata(num_frames=-1, fps=-1.0, max_duration=300) + vllm_def = backend_cls.compute_frames_index_to_sample(src_5s, tgt_default_fps) + assert vllm_def == hf_indices_5s + + # Case 5 (NEW-2): 60 fps / 300 s (18000 frames) vs HF sample_frames + src_300s = VideoSourceMetadata( + total_frames_num=18000, original_fps=60.0, duration=300.0 + ) + v_300s = backend_cls.compute_frames_index_to_sample( + src_300s, VideoTargetMetadata(num_frames=-1, fps=60.0, max_duration=300) + ) + h_300s = hf_proc.sample_frames( + VideoMetadata(fps=60.0, duration=300.0, total_num_frames=18000), + fps=60.0, + max_frames=32, + overflow_strategy="uniform", + ).tolist() + assert v_300s == h_300s, f"60fps/300s mismatch: {v_300s[-3:]} vs {h_300s[-3:]}" + + # Case 6 (NEW-2): 1 fps / 4 h (432000 frames @ 30 native fps) vs HF sample_frames + src_4h = VideoSourceMetadata( + total_frames_num=432000, original_fps=30.0, duration=14400.0 + ) + v_4h = backend_cls.compute_frames_index_to_sample( + src_4h, VideoTargetMetadata(num_frames=-1, fps=1.0, max_duration=300) + ) + h_4h = hf_proc.sample_frames( + VideoMetadata(fps=30.0, duration=14400.0, total_num_frames=432000), + fps=1.0, + max_frames=32, + overflow_strategy="uniform", + ).tolist() + assert v_4h == h_4h, f"1fps/4h mismatch: {v_4h[-2:]} vs {h_4h[-2:]}" + + +def test_video_loader_duplicate_expansion(): + from transformers.models.embedding_gemma2.video_processing_embedding_gemma2 import ( + EmbeddingGemma2VideoProcessor, + ) + from transformers.video_utils import VideoMetadata + + from vllm.multimodal.video import ( + VIDEO_LOADER_REGISTRY, + VideoSourceMetadata, + VideoTargetMetadata, + ) + + backend_cls = VIDEO_LOADER_REGISTRY.name2class["embedding_gemma2"] + hf_proc = EmbeddingGemma2VideoProcessor() + + # 4s clip @ 0.5fps = 2 frames total. Requested target fps = 1.0 -> 4 frames. + # Source has fewer frames than requested sampled frames, requiring duplicates. + src_lowfps = VideoSourceMetadata(total_frames_num=2, original_fps=0.5, duration=4.0) + tgt = VideoTargetMetadata(num_frames=-1, fps=1.0, max_duration=300) + vllm_indices = backend_cls.compute_frames_index_to_sample(src_lowfps, tgt) + hf_meta = VideoMetadata(fps=0.5, duration=4.0, total_num_frames=2) + hf_indices = hf_proc.sample_frames( + hf_meta, fps=1.0, max_frames=32, overflow_strategy="uniform" + ).tolist() + assert vllm_indices == hf_indices + assert len(vllm_indices) == 4 + assert vllm_indices == [0, 0, 1, 1] + + +def test_video_media_io_embedding_gemma2(): + """Verify VideoMediaIO with default num_frames=32 loads video bytes for + + EmbeddingGemma2 without error. + """ + import os + from unittest.mock import MagicMock + + from vllm.multimodal.media import MediaConnector + from vllm.multimodal.media.video import VideoMediaIO + + video_path = "/projects/gemma4-vllm/videos/test_intel.mp4" + if not os.path.exists(video_path): + return + + with open(video_path, "rb") as f: + data = f.read() + + # VideoMediaIO defaults to num_frames=32 + video_io = VideoMediaIO(image_io=MagicMock(), video_backend="embedding_gemma2") + loaded = video_io.load_bytes(data) + frames, meta = loaded.media + assert len(frames) == 32 + assert meta.get("do_sample_frames") is False + + # MediaConnector path + connector = MediaConnector(allowed_local_media_path="/projects/gemma4-vllm/videos") + fetched = connector.fetch_video( + f"file://{video_path}", + video_processor="EmbeddingGemma2VideoProcessor", + ) + frames_conn, meta_conn = fetched.media + assert len(frames_conn) == 32 + assert meta_conn.get("do_sample_frames") is False + + +def test_config_capping_and_explicit_limits(): + """Verify EmbeddingGemma2ModelConfig backend selection, default capping, + and preservation of explicit limits. + """ + from types import SimpleNamespace + from unittest.mock import patch + + from vllm.model_executor.models.config import ( + EmbeddingGemma2ModelConfig, + Gemma4Config, + ) + from vllm.v1.attention.backends.registry import AttentionBackendEnum + + # (a) When attention backend is None, default to TRITON_ATTN, + # and default capping: max_model_len > 8192 without explicit limit capped to 8192 + cfg_default = SimpleNamespace( + attention_config=SimpleNamespace(backend=None), + model_config=SimpleNamespace( + max_model_len=262144, + original_max_model_len=None, + ), + ) + with patch.object(Gemma4Config, "verify_and_update_config"): + EmbeddingGemma2ModelConfig.verify_and_update_config(cfg_default) + + assert cfg_default.attention_config.backend == AttentionBackendEnum.TRITON_ATTN + assert cfg_default.model_config.max_model_len == 8192 + + # (b) Default capping with original_max_model_len == -1 + cfg_default_minus1 = SimpleNamespace( + attention_config=SimpleNamespace(backend=None), + model_config=SimpleNamespace( + max_model_len=131072, + original_max_model_len=-1, + ), + ) + with patch.object(Gemma4Config, "verify_and_update_config"): + EmbeddingGemma2ModelConfig.verify_and_update_config(cfg_default_minus1) + + assert cfg_default_minus1.model_config.max_model_len == 8192 + + # (c) Explicit limit 4096 preserved + cfg_explicit_4k = SimpleNamespace( + attention_config=SimpleNamespace(backend=None), + model_config=SimpleNamespace( + max_model_len=4096, + original_max_model_len=4096, + ), + ) + with patch.object(Gemma4Config, "verify_and_update_config"): + EmbeddingGemma2ModelConfig.verify_and_update_config(cfg_explicit_4k) + + assert cfg_explicit_4k.model_config.max_model_len == 4096 + + # (d) Explicit limit 16384 preserved and explicit backend preserved + cfg_explicit_16k = SimpleNamespace( + attention_config=SimpleNamespace(backend=AttentionBackendEnum.FLASH_ATTN), + model_config=SimpleNamespace( + max_model_len=16384, + original_max_model_len=16384, + ), + ) + with patch.object(Gemma4Config, "verify_and_update_config"): + EmbeddingGemma2ModelConfig.verify_and_update_config(cfg_explicit_16k) + + assert cfg_explicit_16k.attention_config.backend == AttentionBackendEnum.FLASH_ATTN + assert cfg_explicit_16k.model_config.max_model_len == 16384 + + +def test_sentence_transformer_tokenizer_config_max_seq_length(tmp_path): + """Verify get_sentence_transformer_tokenizer_config accepts config + with only max_seq_length. + """ + import json + + from vllm.transformers_utils.config import ( + get_sentence_transformer_tokenizer_config, + ) + + # 1. Synthetic model directory with only max_seq_length in sentence_bert_config.json + cfg_file = tmp_path / "sentence_bert_config.json" + cfg_file.write_text(json.dumps({"max_seq_length": 8192})) + loaded = get_sentence_transformer_tokenizer_config(str(tmp_path)) + assert loaded is not None + assert loaded["max_seq_length"] == 8192 diff --git a/tests/models/multimodal/processing/test_kimi_k3.py b/tests/models/multimodal/processing/test_kimi_k3.py new file mode 100644 index 0000000000..62c99b24f1 --- /dev/null +++ b/tests/models/multimodal/processing/test_kimi_k3.py @@ -0,0 +1,81 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import numpy as np +import pytest +import torch +from PIL import Image +from transformers import AutoImageProcessor + +from vllm.model_executor.layers.fusion.mm_input_norm import build_mm_input_norm +from vllm.multimodal import MULTIMODAL_REGISTRY +from vllm.platforms import current_platform +from vllm.transformers_utils.processors.kimi_k25_vision_fused import ( + KimiK25FusedVisionProcessor, +) + +from ...utils import build_model_context + +_MODEL_ID = "moonshotai/Kimi-K3" + + +@pytest.fixture(scope="module") +def image_processors(): + ctx = build_model_context(_MODEL_ID, limit_mm_per_prompt={"image": 1}) + processor = MULTIMODAL_REGISTRY.create_processor(ctx.model_config) + k25_fused = processor.info.image_processor + assert isinstance(k25_fused, KimiK25FusedVisionProcessor) + k3_hf = AutoImageProcessor.from_pretrained(_MODEL_ID, trust_remote_code=True) + return k25_fused, k3_hf + + +def _noise_image(width: int, height: int) -> Image.Image: + rng = np.random.default_rng(0) + return Image.fromarray(rng.integers(0, 256, (height, width, 3), dtype=np.uint8)) + + +@pytest.mark.parametrize( + ("width", "height"), + [(310, 470), (7300, 30), (13, 9)], +) +def test_fused_image_processor_matches_hf(image_processors, width: int, height: int): + fused, hf = image_processors + image = _noise_image(width, height) + expected = hf.preprocess([{"type": "image", "image": image}], return_tensors="pt") + actual = fused.preprocess([{"type": "image", "image": image}], return_tensors="pt") + + torch.testing.assert_close(actual["grid_thws"], expected["grid_thws"]) + torch.testing.assert_close(actual["pixel_values"], expected["pixel_values"]) + assert fused.media_tokens_calculator( + {"type": "image", "image": image} + ) == hf.media_tokens_calculator({"type": "image", "image": image}) + + +@pytest.mark.usefixtures("default_vllm_config") +def test_mm_device_do_normalize(): + device = current_platform.device_type + ctx = build_model_context(_MODEL_ID, limit_mm_per_prompt={"image": 2}) + assert ctx.model_config.multimodal_config.mm_device_do_normalize + + ctx.model_config.multimodal_config.mm_device_do_normalize = False + processor = MULTIMODAL_REGISTRY.create_processor(ctx.model_config) + images = [ + Image.new("RGB", (310, 470), color=(17, 89, 231)), + Image.new("RGB", (480, 320), color=(201, 13, 127)), + ] + prompt = processor.info.get_hf_config().image_placeholder * len(images) + mm_items = processor.info.parse_mm_data({"image": images}) + + normalized_inputs = processor(prompt, mm_items=mm_items) + normalized_values = normalized_inputs["mm_kwargs"].get_data()["pixel_values"] + + ctx.model_config.multimodal_config.mm_device_do_normalize = True + raw_inputs = processor(prompt, mm_items=mm_items) + raw_values = raw_inputs["mm_kwargs"].get_data()["pixel_values"] + assert raw_values.dtype == torch.uint8 + + input_norm = build_mm_input_norm(ctx.model_config).to(device) + output = input_norm(raw_values.flatten(1).to(device), normalized_values.dtype) + torch.testing.assert_close( + output, normalized_values.flatten(1).to(device), rtol=1.6e-2, atol=1e-5 + ) diff --git a/tests/models/multimodal/processing/test_transformers_image.py b/tests/models/multimodal/processing/test_transformers_image.py index 26a3a592ca..d45912a4b1 100644 --- a/tests/models/multimodal/processing/test_transformers_image.py +++ b/tests/models/multimodal/processing/test_transformers_image.py @@ -249,9 +249,9 @@ def test_repeated_image_hits_the_processor_cache(): "BAAI/Emu3-Chat-hf", " and more text", marks=pytest.mark.xfail( - reason="Emu3Processor prepends its BOS token only when images are " - "passed, so the unexpanded prompt vLLM tokenizes never " - "gets one. Fixed by huggingface/transformers#47924, unreleased.", + reason="Emu3Processor prepends the BOS token itself because the " + "tokenizer doesn't, so the unexpanded prompt vLLM tokenizes " + "never gets one.", strict=False, ), ), diff --git a/tests/models/qwen4_exp/test_ple.py b/tests/models/qwen4_exp/test_ple.py index 7c2a87c71e..4574654c39 100644 --- a/tests/models/qwen4_exp/test_ple.py +++ b/tests/models/qwen4_exp/test_ple.py @@ -32,6 +32,7 @@ from vllm.model_executor.layers.quantization.modelopt import ( ModelOptNvFp4Config, ) from vllm.model_executor.layers.quantization.online.base import OnlineQuantizationConfig +from vllm.model_executor.layers.quantization.quark.quark import QuarkConfig from vllm.models.qwen4_exp.amd import ple_layer as amd_ple_layer from vllm.models.qwen4_exp.amd.ple_layer import ( Qwen4ExpPLELayer as Qwen4ExpPLELayerAMD, @@ -51,6 +52,7 @@ from vllm.models.qwen4_exp.nvidia.ngram_embedding import ( Qwen4ExpPLEUnquantizedEmbeddingMethod, ) from vllm.models.qwen4_exp.nvidia.ple_layer import Qwen4ExpPLELayer +from vllm.platforms import current_platform from vllm.utils.torch_utils import weak_ref_tensor from vllm.v1.attention.backends.short_conv_attn import ( PleShortConvAttentionMetadata, @@ -592,6 +594,16 @@ def test_ple_embedding_respects_inc_layer_config() -> None: Qwen4ExpPLEEmbeddingMethod.from_quant_config(quant_config, prefix) +def test_ple_embedding_is_unquantized_under_quark() -> None: + prefix = "model.layers.1.ple.ple_embedding.ngram_embedding" + quant_config = QuarkConfig({"exclude": [], "global_quant_config": {}}) + + assert isinstance( + Qwen4ExpPLEEmbeddingMethod.from_quant_config(quant_config, prefix), + Qwen4ExpPLEUnquantizedEmbeddingMethod, + ) + + def test_ple_embedding_dtype_overrides_modelopt_exclusion() -> None: prefix = "model.layers.1.ple.ple_embedding.ngram_embedding" quant_config = ModelOptNvFp4Config(exclude_modules=[prefix]) @@ -1442,7 +1454,7 @@ def test_fused_ngram_ids_correctness( contexts: list[list[int]], first_token_id: int, ) -> None: - from vllm.models.qwen4_exp.nvidia.ops.ple import ple_ngram_ids + from vllm.models.qwen4_exp.common.ops.ple import ple_ngram_ids device = torch.device("cuda") query_start_loc = torch.tensor( @@ -2197,15 +2209,20 @@ def _make_conv_metadata( ), ], ) +@pytest.mark.parametrize( + "layer_cls", [Qwen4ExpPLELayer, Qwen4ExpPLELayerAMD], ids=["nvidia", "amd"] +) def test_fused_conv_correctness( case: _ConvBatchCase, state_layout: str, + layer_cls: type[nn.Module], ) -> None: device = torch.device("cuda") metadata, num_real_tokens = _make_conv_metadata(case, device) if case.state_index_stride > 1: assert metadata.state_indices_tensor.stride(0) == case.state_index_stride - module = Qwen4ExpPLELayer.__new__(Qwen4ExpPLELayer) + # layer_cls is type[nn.Module], so layer_cls.__new__ is type.__new__. + module = object.__new__(layer_cls) nn.Module.__init__(module) module.conv_state_len = (case.kernel_size - 1) * case.dilation module.short_conv_dilation = case.dilation @@ -2278,7 +2295,7 @@ def test_fused_conv_correctness( def test_fused_gate_correctness(num_tokens: int, strided_kv: bool) -> None: import math - from vllm.models.qwen4_exp.nvidia.ops.ple import ple_gate + from vllm.models.qwen4_exp.common.ops.ple import ple_gate device = torch.device("cuda") hc, h = 4, 2560 @@ -2378,7 +2395,6 @@ def _build_amd_ngram_embedding( embedding_dim=config.ple_embed_dim, ple_dense_layer_id=0, max_total_tokens=4, - max_num_reqs=2, prefix="test.ple_embedding", layer_name="test.ple", quant_config=( @@ -2564,3 +2580,47 @@ def test_amd_pinned_embedding_output_written_under_cudagraph_capture( expected = loaded_weight[ngram_ids.cpu()].to(device="cuda:0").flatten(-2) torch.testing.assert_close(output.float(), expected.float(), rtol=0, atol=0) + + +@pytest.mark.skipif( + not current_platform.is_rocm(), + reason="only the ROCm backend runs the n-gram embedding under torch.compile", +) +@pytest.mark.parametrize( + ("query_lens", "contexts"), + [([3, 1], [[5, 6], [0, 7]]), ([1, 0, 2], [[1, 2], [3, 4], [0, 0]])], + ids=["two-requests", "empty-request"], +) +def test_amd_ngram_embedding_matches_reference_under_compile( + monkeypatch: pytest.MonkeyPatch, + query_lens: list[int], + contexts: list[list[int]], +) -> None: + """The AMD n-gram ID op must run under Dynamo with a symbolic request count.""" + module, loaded_weight = _build_amd_ngram_embedding( + monkeypatch, device="cuda:0", cpu_offload=False, fp8_checkpoint=False + ) + device = torch.device("cuda:0") + query_start_loc = torch.tensor( + [0, *accumulate(query_lens)], dtype=torch.int32, device=device + ) + input_ids = torch.tensor( + [9, 0, 11, 12][: sum(query_lens)], dtype=torch.int32, device=device + ) + ngram_context = torch.tensor(contexts, dtype=torch.int32, device=device) + + compiled = torch.compile(module, fullgraph=True, dynamic=True) + output = compiled(input_ids, query_start_loc, ngram_context) + + ngram_ids = _reference_ngram_ids( + input_ids, + query_start_loc, + ngram_context, + module.layer_multipliers, + module.ngram_heads_vocab_sizes, + module.ngram_heads_offsets, + module.eos_token_id, + module.heads_per_ngram, + ) + expected = loaded_weight[ngram_ids.cpu()].flatten(-2).to(device) + torch.testing.assert_close(output.float(), expected.float(), rtol=0, atol=0) diff --git a/tests/models/qwen4_exp/test_qsa_amd.py b/tests/models/qwen4_exp/test_qsa_amd.py index e8e39a7666..c8ebf01904 100644 --- a/tests/models/qwen4_exp/test_qsa_amd.py +++ b/tests/models/qwen4_exp/test_qsa_amd.py @@ -1,6 +1,7 @@ # SPDX-License-Identifier: Apache-2.0 # SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import inspect import math from types import SimpleNamespace from typing import Any @@ -9,15 +10,21 @@ import pytest import torch from vllm.model_executor.layers.layernorm import GemmaRMSNorm +from vllm.models.qwen4_exp.amd import indexer_qsa from vllm.models.qwen4_exp.amd import ( model as _qwen4_exp_model, # noqa: F401 ) from vllm.models.qwen4_exp.amd import ple_layer as ple_layer_module from vllm.models.qwen4_exp.amd.indexer_qsa import ( + QSAIndexer, apply_qsa_rmsnorm, apply_qsa_rope, ) from vllm.models.qwen4_exp.amd.ops import qsa as qsa_ops +from vllm.models.qwen4_exp.amd.ops.qsa_prepare import ( + qsa_prepare, + supports_fused_pre_indexer, +) from vllm.platforms import current_platform from vllm.triton_utils import HAS_TRITON @@ -120,6 +127,195 @@ def test_qsa_rmsnorm_uses_portable_implementation(default_vllm_config) -> None: torch.testing.assert_close(output, norm.forward_native(tensor)) +def _make_attn(*, use_fused: bool) -> SimpleNamespace: + """The main-attention fields the fused QSA prepare reads from its owner.""" + return SimpleNamespace( + use_fused_qsa_prepare=use_fused, + kv_cache=torch.zeros(1, 2, 16, 8), + q_norm=GemmaRMSNorm(4, eps=1e-6), + k_norm=GemmaRMSNorm(4, eps=1e-6), + _k_scale_float=1.0, + _v_scale_float=1.0, + ) + + +def _make_indexer(*, use_fused: bool, num_tokens: int = 2) -> QSAIndexer: + """A QSAIndexer whose unfused path runs for real, with no projection weights. + + Both branches have to be reachable: a fused-path test is only meaningful if + the unfused one it replaces would otherwise have executed. + """ + heads, head_dim = 2, 4 + indexer = QSAIndexer.__new__(QSAIndexer) + torch.nn.Module.__init__(indexer) + indexer.index_n_heads = heads + indexer.index_kv_heads = 1 + indexer.index_head_dim = head_dim + indexer.compress_ratio = 2 + indexer.use_fused_pre_indexer = use_fused + indexer.skip_topk = True + indexer.index_qk_proj = lambda hidden_states: ( + torch.zeros(hidden_states.shape[0], (heads + 1) * head_dim), + None, + ) + indexer.q_layernorm = GemmaRMSNorm(head_dim, eps=1e-6) + indexer.k_layernorm = GemmaRMSNorm(head_dim, eps=1e-6) + indexer.rotary_emb = SimpleNamespace( + cos_sin_cache=torch.zeros(16, head_dim), + rotary_dim=head_dim // 2, + apply_rotary_emb=lambda tensor, cos, sin: tensor, + _match_cos_sin_cache_dtype=lambda _: torch.zeros(16, head_dim), + ) + # The raw cache's full width carries the MRoPE position tail that the fused + # kernel writes; key_cache is only the leading key columns. + indexer.raw_key_cache = SimpleNamespace( + kv_cache=torch.zeros(1, 4, 1, head_dim + 12), + key_cache=torch.zeros(1, 4, 1, head_dim), + cache_rope_positions=True, + rope_position_offset=head_dim, + ) + indexer.compressed_key_cache = SimpleNamespace( + kv_cache=torch.zeros(1, 2, 1, head_dim) + ) + raw_metadata = SimpleNamespace( + num_actual_tokens=num_tokens, + slot_mapping=torch.arange(num_tokens), + block_table=torch.zeros(1, 1, dtype=torch.int32), + query_start_loc=torch.tensor([0, num_tokens], dtype=torch.int32), + logical_positions=torch.arange(num_tokens), + token_to_req=torch.zeros(num_tokens, dtype=torch.int32), + ) + compressed_metadata = SimpleNamespace( + num_actual_tokens=num_tokens, + slot_mapping=torch.arange(num_tokens), + k_work_metadata=torch.zeros(1, 2, dtype=torch.int32), + ) + indexer._metadata = lambda: (raw_metadata, compressed_metadata) + indexer.test_metadata = (raw_metadata, compressed_metadata) + return indexer + + +@pytest.mark.usefixtures("default_vllm_config") +def test_qsa_fused_prepare_replaces_the_unfused_chain( + monkeypatch: pytest.MonkeyPatch, +) -> None: + indexer = _make_indexer(use_fused=True) + attn = _make_attn(use_fused=True) + calls = [] + main_outputs = (torch.empty(0), torch.empty(0)) + + def fake_prepare(*args, **kwargs): + calls.append((args, kwargs)) + return main_outputs + + monkeypatch.setattr(indexer_qsa, "qsa_prepare", fake_prepare) + monkeypatch.setattr( + QSAIndexer, + "_update_and_compress", + lambda *args, **kwargs: pytest.fail("fused path ran the unfused chain"), + ) + rows = torch.tensor([[3, 1, -1], [5, 2, 0]], dtype=torch.int32) + qkv = torch.zeros(4, 16) + slot_mapping = torch.arange(4) + gate_out = torch.empty(4, 1, 4) + + actual, actual_main = QSAIndexer.forward( + indexer, + torch.zeros(2, 8), + torch.tensor([7, 8]), + rows, + attn=attn, + qkv=qkv, + slot_mapping=slot_mapping, + gate_out=gate_out, + ) + + assert actual is rows + assert actual_main is main_outputs + assert len(calls) == 1 + args, kwargs = calls[0] + passed = inspect.signature(qsa_prepare).bind(*args, **kwargs).arguments + _, compressed_metadata = indexer.test_metadata + # The kernel addresses the raw cache at its full width and writes the MRoPE + # position tail itself, so it needs kv_cache, not the key_cache view. + assert passed["state_cache"] is indexer.raw_key_cache.kv_cache + assert passed["compressed_cache"] is indexer.compressed_key_cache.kv_cache + assert passed["k_work_metadata"] is compressed_metadata.k_work_metadata + assert passed["compress_ratio"] == indexer.compress_ratio + assert passed["rope_pos_offset"] == indexer.index_head_dim + # Main attention: only the real tokens, the vLLM cache viewed per token, + # and the gate landing in the caller's buffer. + assert passed["main_qkv"].shape[0] == 2 + assert passed["main_qkv"].data_ptr() == qkv.data_ptr() + assert passed["main_slot_mapping"].shape == (2,) + assert passed["main_kv_cache"].shape == (1, 16, 2, 8) + assert passed["main_kv_cache"].data_ptr() == attn.kv_cache.data_ptr() + assert passed["main_gate_out"].shape[0] == 2 + assert passed["main_gate_out"].data_ptr() == gate_out.data_ptr() + assert passed["main_q_norm_weight"] is attn.q_norm.weight + assert passed["main_k_norm_weight"] is attn.k_norm.weight + + +@pytest.mark.usefixtures("default_vllm_config") +def test_qsa_unsupported_config_keeps_the_unfused_chain( + monkeypatch: pytest.MonkeyPatch, +) -> None: + indexer = _make_indexer(use_fused=False) + monkeypatch.setattr( + indexer_qsa, + "qsa_prepare", + lambda *args, **kwargs: pytest.fail("unfused path ran the fused kernel"), + ) + updates = [] + monkeypatch.setattr( + QSAIndexer, + "_update_and_compress", + lambda *args, **kwargs: updates.append(args), + ) + rows = torch.tensor([[3, 1, -1], [5, 2, 0]], dtype=torch.int32) + + actual, main_outputs = QSAIndexer.forward( + indexer, + torch.zeros(2, 8), + torch.tensor([7, 8]), + rows, + attn=_make_attn(use_fused=False), + ) + + assert actual is rows + assert main_outputs is None + assert len(updates) == 1 + + +@pytest.mark.parametrize( + ("overrides", "expected"), + [ + pytest.param({}, True, id="qwen3_8_flash_next"), + pytest.param({"is_neox_style": False}, False, id="not-neox"), + pytest.param({"rotary_dim": 128}, False, id="rotary-dim"), + pytest.param({"mrope_interleaved": False}, False, id="mrope-not-interleaved"), + pytest.param({"mrope_section": (11, 11, 10, 0)}, False, id="mrope-section-len"), + ], +) +def test_fused_pre_indexer_gate_is_config_shaped(overrides, expected) -> None: + # Qwen3.8-Flash-Next's indexer geometry, which both vendors satisfy. + rotary = SimpleNamespace( + rotary_dim=64, + is_neox_style=True, + mrope_section=(11, 11, 10), + mrope_interleaved=True, + ) + for name, value in overrides.items(): + setattr(rotary, name, value) + + assert ( + supports_fused_pre_indexer( + rotary, head_dim=128, num_kv_heads=1, compress_ratio=4 + ) + is expected + ) + + def test_qsa_selection_uses_portable_topk_on_rocm( monkeypatch: pytest.MonkeyPatch, ) -> None: diff --git a/tests/models/qwen4_exp/test_qsa_prepare_amd.py b/tests/models/qwen4_exp/test_qsa_prepare_amd.py new file mode 100644 index 0000000000..7189297209 --- /dev/null +++ b/tests/models/qwen4_exp/test_qsa_prepare_amd.py @@ -0,0 +1,418 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Correctness tests for the AMD fused QSA prepare kernel.""" + +import pytest +import torch + +from vllm.model_executor.layers.layernorm import GemmaRMSNorm +from vllm.models.qwen4_exp.amd.indexer_qsa import apply_qsa_rope +from vllm.models.qwen4_exp.amd.ops.qsa import ( + qsa_compress_groups_with_ratio, + qsa_store_cache_rows, +) +from vllm.models.qwen4_exp.amd.ops.qsa_prepare import qsa_prepare +from vllm.models.qwen4_exp.common.qsa_cache import ( + canonical_qsa_rope_positions, + circular_qsa_slot_mapping, + compressed_qsa_slot_mapping, +) +from vllm.platforms import current_platform +from vllm.triton_utils import HAS_TRITON + +requires_qsa_kernels = pytest.mark.skipif( + not current_platform.is_rocm() or not HAS_TRITON, + reason="the AMD QSA kernels require ROCm and Triton", +) + + +def _reference_gemma_rmsnorm( + x: torch.Tensor, + weight: torch.Tensor, + eps: float, +) -> torch.Tensor: + """The unfused Gemma RMSNorm the AMD reference path actually runs.""" + norm = GemmaRMSNorm(weight.shape[-1], eps=eps).to( + device=weight.device, dtype=weight.dtype + ) + with torch.no_grad(): + norm.weight.copy_(weight) + return norm.forward_native(x) + + +HQ, D = 4, 128 +CR = 4 +MROPE_SECTION = (11, 11, 10) +EPS = 1e-6 +BLOCK_SIZE = 16 +COMP_PAGE = BLOCK_SIZE // CR +ROPE_POS_OFFSET = D +RTOL = 1.6e-2 +ATOL = 1e-2 +MIXED_BATCH = ([260, 259, 138], [1, 1, 37], [8, 8, 8]) +MAIN_HQ, MAIN_HK, MAIN_D, MAIN_PAGE = 6, 2, 256, 16 + + +def _make_block_table(block_counts): + # Block 0 is the null block, which no request owns. + num_blocks = sum(block_counts) + 1 + table = torch.full((len(block_counts), max(block_counts)), -1, dtype=torch.int32) + physical_blocks = torch.randperm(sum(block_counts)) + 1 + offset = 0 + for request, count in enumerate(block_counts): + table[request, :count] = physical_blocks[offset : offset + count] + offset += count + return table, num_blocks + + +def assert_fp8_within_one_ulp(actual: torch.Tensor, expected: torch.Tensor) -> None: + # e4m3 is sign-magnitude, so within a sign the uint8 code order matches the + # value order and one ulp is one code step. The two paths' intermediates + # differ in the pooling accumulation order, which at denormal magnitudes + # (absolute grid step 2^-9) shows up as up to 2 code steps. + code_diff = ( + actual.view(torch.uint8).int() - expected.view(torch.uint8).int() + ).abs() + abs_diff = (actual.float() - expected.float()).abs() + assert bool(((code_diff <= 1) | (abs_diff <= 2**-8)).all()) + + +def _make_main_inputs(num_tokens: int) -> dict: + """Random main-attention arguments with a paged BF16 cache in vLLM's layout.""" + num_blocks = num_tokens // MAIN_PAGE + 2 + cache = torch.zeros( + num_blocks, + MAIN_HK, + MAIN_PAGE, + 2 * MAIN_D, + dtype=torch.bfloat16, + device="cuda", + ).transpose(1, 2) + slots = torch.randperm(num_blocks * MAIN_PAGE, device="cuda")[:num_tokens] + slots[-1] = -1 + norm_weights = torch.randn(2, MAIN_D, dtype=torch.bfloat16, device="cuda") * 0.2 + return dict( + main_qkv=torch.randn( + num_tokens, + 2 * (MAIN_HQ + MAIN_HK) * MAIN_D, + dtype=torch.bfloat16, + device="cuda", + ), + main_q_norm_weight=norm_weights[0], + main_k_norm_weight=norm_weights[1], + main_eps=EPS, + main_kv_cache=cache, + main_slot_mapping=slots, + main_k_scale=1.0, + main_v_scale=1.0, + # The AMD op hands the gate back through a buffer it does not own. + main_gate_out=torch.empty( + num_tokens, MAIN_HQ, MAIN_D, dtype=torch.bfloat16, device="cuda" + ), + ) + + +def _check_main_outputs(main: dict, q_out, gate_out, rope, positions) -> None: + """Compare with the AMD layer's unfused split, QK-norm, RoPE and cache write.""" + from vllm._custom_ops import reshape_and_cache_flash + + num_tokens = main["main_qkv"].shape[0] + q_gate, k, v = main["main_qkv"].split( + [2 * MAIN_HQ * MAIN_D, MAIN_HK * MAIN_D, MAIN_HK * MAIN_D], dim=-1 + ) + q, gate = torch.chunk(q_gate.view(num_tokens, MAIN_HQ, -1), 2, dim=-1) + q = _reference_gemma_rmsnorm( + q.reshape(-1, MAIN_D), main["main_q_norm_weight"], EPS + ).view(num_tokens, -1) + k = _reference_gemma_rmsnorm( + k.reshape(-1, MAIN_D), main["main_k_norm_weight"], EPS + ).view(num_tokens, -1) + q, k = rope(positions, q, k) + kv_cache = main["main_kv_cache"] + cache = torch.zeros_like(kv_cache) + key_cache, value_cache = cache.split(MAIN_D, dim=-1) + scale = torch.tensor(1.0, device="cuda") + reshape_and_cache_flash( + k.view(-1, MAIN_HK, MAIN_D), + v.view(-1, MAIN_HK, MAIN_D), + key_cache, + value_cache, + main["main_slot_mapping"], + "auto", + scale, + scale, + ) + torch.testing.assert_close(q_out.flatten(1), q, rtol=RTOL, atol=ATOL) + assert torch.equal(gate_out, gate) + torch.testing.assert_close(kv_cache, cache, rtol=RTOL, atol=ATOL) + + +@requires_qsa_kernels +@pytest.mark.usefixtures("default_vllm_config") +@pytest.mark.parametrize("indexer_dtype", [torch.bfloat16, torch.float8_e4m3fn]) +@pytest.mark.parametrize( + "mrope,is_2d_positions,cache_rope_positions,state_size,seq_lens,query_lens,history_lens", + [ + pytest.param(True, True, True, 4, *MIXED_BATCH, id="mrope"), + pytest.param(False, False, False, 4, *MIXED_BATCH, id="text"), + pytest.param(True, False, True, 8, *MIXED_BATCH, id="mrope-model-1d"), + pytest.param(True, True, False, 4, *MIXED_BATCH, id="mrope-no-position-cache"), + pytest.param( + True, + False, + False, + 8, + *MIXED_BATCH, + id="mrope-model-1d-no-position-cache", + ), + pytest.param( + False, False, True, 4, *MIXED_BATCH, id="text-with-position-cache" + ), + pytest.param(True, True, True, 4, [37], [37], [0], id="fresh"), + pytest.param(True, True, True, 4, [4097], [4097], [0], id="tiled"), + ], +) +def test_qsa_fused_prepare_matches_unfused( + indexer_dtype, + mrope, + is_2d_positions, + cache_rope_positions, + state_size, + seq_lens, + query_lens, + history_lens, +) -> None: + from vllm.model_executor.layers.rotary_embedding import get_rope + + if indexer_dtype == torch.float8_e4m3fn: + pytest.skip("the AMD QSA indexer cache is bf16-only") + + # The FP8 comparison below is a one-ulp bound on data this test generates, + # so an unseeded run decides for itself whether it reproduces a rounding + # difference. Seed it, or the kernel's bit-exactness is only sampled. + torch.manual_seed(0) + + device = "cuda" + rope_params = { + "partial_rotary_factor": 0.25, + "rope_theta": 10000000, + "rope_type": "default", + } + if mrope: + rope_params["mrope_interleaved"] = True + rope_params["mrope_section"] = list(MROPE_SECTION) + with torch.device(device): + rope = get_rope( + head_size=256, + max_position=32768, + rope_parameters=rope_params, + dtype=torch.bfloat16, + ) + + token_to_req = torch.cat( + [ + torch.full((length,), request, dtype=torch.int32) + for request, length in enumerate(query_lens) + ] + ).to(device) + logical_positions = torch.cat( + [ + torch.arange(seq_len - query_len, seq_len, dtype=torch.int64) + for seq_len, query_len in zip(seq_lens, query_lens) + ] + ).to(device) + num_tokens = logical_positions.numel() + query_start_loc = torch.tensor( + [0, *torch.tensor(query_lens).cumsum(0).tolist()], dtype=torch.int32 + ).to(device) + positions = ( + torch.stack( + [ + logical_positions, + logical_positions // 7 + 3, + logical_positions // 13 + 11, + ] + ) + if is_2d_positions + else logical_positions + ) + position_rows = ( + canonical_qsa_rope_positions(positions) + if cache_rope_positions + else logical_positions.view(-1, 1, 1).expand(-1, 1, 3) + ) + + compressed_block_counts = [ + (seq_len // CR + COMP_PAGE - 1) // COMP_PAGE for seq_len in seq_lens + ] + raw_block_table, num_raw_blocks = _make_block_table([1] * len(seq_lens)) + compressed_block_table, num_compressed_blocks = _make_block_table( + compressed_block_counts + ) + raw_block_table = raw_block_table.to(device) + raw_slots = circular_qsa_slot_mapping( + raw_block_table, + token_to_req, + logical_positions, + state_size, + query_start_loc, + ) + compressed_slots = compressed_qsa_slot_mapping( + compressed_block_table.to(device), + token_to_req, + logical_positions, + COMP_PAGE, + CR, + ) + group_counts = torch.tensor( + [ + seq_len // CR - (seq_len - query_len) // CR + for seq_len, query_len in zip(seq_lens, query_lens) + ], + dtype=torch.int32, + device=device, + ) + k_work_counts = torch.maximum(group_counts, torch.ones_like(group_counts)) + k_start_loc = torch.cat([k_work_counts.new_zeros(1), k_work_counts.cumsum(0)]) + work_requests = torch.repeat_interleave( + torch.arange(len(query_lens), dtype=torch.int32, device=device), + k_work_counts, + ) + local_work = torch.arange( + int(k_start_loc[-1]), dtype=torch.int32, device=device + ) - torch.repeat_interleave(k_start_loc[:-1], k_work_counts) + max_k_work = (num_tokens + (CR - 1) * len(query_lens)) // CR + k_work_metadata = torch.full((max_k_work, 2), -1, dtype=torch.int32, device=device) + k_work_metadata[: work_requests.numel()] = torch.stack( + (work_requests, local_work), dim=1 + ) + + raw_width = D + 12 if cache_rope_positions else D + # Match vLLM's padded-page cache layout: rows are contiguous, while physical + # blocks have a larger stride than their logical contents. + raw_page_elements = state_size * raw_width + fused_raw_storage = torch.zeros( + num_raw_blocks, + raw_page_elements + 16, + dtype=torch.bfloat16, + device=device, + ) + fused_raw = torch.as_strided( + fused_raw_storage, + (num_raw_blocks, state_size, 1, raw_width), + (raw_page_elements + 16, raw_width, raw_width, 1), + ) + for request, history_len in enumerate(history_lens): + history_end = seq_lens[request] - query_lens[request] + for position in range(history_end - history_len, history_end): + block = int(raw_block_table[request, 0]) + row = fused_raw[block, position % state_size, 0] + row[:D] = torch.randn(D, dtype=torch.bfloat16, device=device) + if cache_rope_positions: + row[ROPE_POS_OFFSET:].view(torch.int64).copy_( + torch.tensor( + [position, position // 7 + 3, position // 13 + 11], + dtype=torch.int64, + device=device, + ) + ) + unfused_raw = fused_raw.clone() + compressed_page_elements = COMP_PAGE * D + fused_compressed_storage = torch.zeros( + num_compressed_blocks, + compressed_page_elements + 16, + dtype=indexer_dtype, + device=device, + ) + fused_compressed = torch.as_strided( + fused_compressed_storage, + (num_compressed_blocks, COMP_PAGE, 1, D), + (compressed_page_elements + 16, D, D, 1), + ) + unfused_compressed = fused_compressed.clone() + + projected_qk = torch.randn( + num_tokens, (HQ + 1) * D, dtype=torch.bfloat16, device=device + ) + q_weight = torch.randn(D, dtype=torch.bfloat16, device=device) * 0.2 + k_weight = torch.randn(D, dtype=torch.bfloat16, device=device) * 0.2 + + fused_query = torch.empty(num_tokens, HQ, D, dtype=indexer_dtype, device=device) + main = _make_main_inputs(num_tokens) + main_q_out, main_gate_out = qsa_prepare( + projected_qk[:, : HQ * D], + projected_qk[:, HQ * D :], + positions, + rope.cos_sin_cache, + q_weight, + k_weight, + EPS, + fused_query, + fused_raw, + raw_slots, + raw_block_table, + query_start_loc, + logical_positions, + fused_compressed, + compressed_slots, + k_work_metadata, + compress_ratio=CR, + mrope_section=MROPE_SECTION if mrope else None, + rope_pos_offset=ROPE_POS_OFFSET if cache_rope_positions else None, + **main, + ) + assert main_gate_out is main["main_gate_out"] + _check_main_outputs(main, main_q_out, main_gate_out, rope, positions) + + unfused_query = projected_qk[:, : HQ * D].reshape(num_tokens, HQ, D) + unfused_query = _reference_gemma_rmsnorm( + unfused_query.reshape(-1, D), q_weight, EPS + ).reshape_as(unfused_query) + unfused_query = apply_qsa_rope(rope, positions, unfused_query) + + raw_keys = unfused_raw[..., :D] + rope_positions = ( + unfused_raw[..., ROPE_POS_OFFSET:].view(torch.int64) + if cache_rope_positions + else None + ) + pooled, first_positions = qsa_compress_groups_with_ratio( + projected_qk[:, HQ * D :].reshape(-1, 1, D), + position_rows, + raw_keys, + raw_block_table, + token_to_req, + query_start_loc, + logical_positions, + compressed_slots, + CR, + rope_positions, + ) + compressed_rows = _reference_gemma_rmsnorm( + pooled.reshape(-1, D), k_weight, EPS + ).reshape(-1, 1, D) + group_positions = ( + first_positions.transpose(0, 1) if mrope else first_positions[:, 0] + ) + compressed_rows = apply_qsa_rope(rope, group_positions, compressed_rows) + qsa_store_cache_rows(unfused_compressed, compressed_slots, compressed_rows) + qsa_store_cache_rows(raw_keys, raw_slots, projected_qk[:, HQ * D :]) + if rope_positions is not None: + qsa_store_cache_rows(rope_positions, raw_slots, position_rows) + + if indexer_dtype == torch.float8_e4m3fn: + # Both paths round the same ~bf16 intermediates to e4m3. Bitwise + # equality (the dsv4 indexer test's bar) does not hold here: the 1D + # reference rope (forward_cuda) differs from _norm_rope by up to 1 + # bf16 ulp, and even the MRoPE pair flips a code occasionally. + unfused_query = unfused_query.to(indexer_dtype) + assert_fp8_within_one_ulp(fused_query, unfused_query) + else: + torch.testing.assert_close(fused_query, unfused_query, rtol=RTOL, atol=ATOL) + assert torch.equal(fused_raw.view(torch.int16), unfused_raw.view(torch.int16)) + if indexer_dtype == torch.float8_e4m3fn: + assert_fp8_within_one_ulp(fused_compressed, unfused_compressed) + else: + torch.testing.assert_close( + fused_compressed, unfused_compressed, rtol=RTOL, atol=ATOL + ) diff --git a/tests/models/qwen4_exp/test_qsa_reference.py b/tests/models/qwen4_exp/test_qsa_reference.py index ed37c27c2a..bf2f5e37fa 100644 --- a/tests/models/qwen4_exp/test_qsa_reference.py +++ b/tests/models/qwen4_exp/test_qsa_reference.py @@ -15,6 +15,7 @@ from vllm.models.qwen4_exp.nvidia import ( ) from vllm.models.qwen4_exp.nvidia.ops import qsa as qsa_ops from vllm.models.qwen4_exp.nvidia.ops import qsa_indexer as qsa_indexer_ops +from vllm.models.qwen4_exp.nvidia.qsa import qsa_kv_cache_dtype from vllm.platforms import current_platform from vllm.triton_utils import HAS_TRITON from vllm.v1.worker.utils import clear_layer_kv_caches @@ -314,7 +315,7 @@ def test_qsa_circular_buffer_metadata_keeps_only_each_requests_suffix() -> None: builder.k_work_metadata_buffer = torch.empty(0, 2, dtype=torch.int32, device=device) query_start_loc = torch.tensor([0, 7, 13, 13], dtype=torch.int32, device=device) token_to_req = torch.tensor([0] * 7 + [1] * 6 + [0] * 3, device=device) - block_table = torch.tensor([[1], [0], [2]], dtype=torch.int32, device=device) + block_table = torch.tensor([[1], [3], [2]], dtype=torch.int32, device=device) common = SimpleNamespace( num_actual_tokens=16, num_reqs=3, @@ -339,10 +340,10 @@ def test_qsa_circular_buffer_metadata_keeps_only_each_requests_suffix() -> None: 4, -1, -1, - 3, - 0, - 1, - 2, + 15, + 12, + 13, + 14, -1, -1, -1, @@ -350,6 +351,10 @@ def test_qsa_circular_buffer_metadata_keeps_only_each_requests_suffix() -> None: assert metadata.slot_mapping.tolist() == expected + # A dummy batch puts every request on the null block, which owns no ring. + block_table.zero_() + assert builder.build(0, common).slot_mapping.tolist() == [-1] * 16 + @pytest.mark.parametrize("chunk_start", list(range(8))) def test_qsa_circular_buffer_survives_one_speculative_step(chunk_start: int) -> None: @@ -366,7 +371,7 @@ def test_qsa_circular_buffer_survives_one_speculative_step(chunk_start: int) -> query_len = num_spec + 1 slots = qsa_cache.circular_qsa_slot_mapping( - torch.tensor([[0]], dtype=torch.int32), + torch.tensor([[1]], dtype=torch.int32), torch.zeros(query_len, dtype=torch.int32), torch.arange(chunk_start, chunk_start + query_len), capacity, @@ -374,7 +379,7 @@ def test_qsa_circular_buffer_survives_one_speculative_step(chunk_start: int) -> ) committed = torch.arange(chunk_start - chunk_start % compress_ratio, chunk_start) - assert set(slots.tolist()).isdisjoint((committed % capacity).tolist()) + assert set((slots % capacity).tolist()).isdisjoint((committed % capacity).tolist()) def _qsa_key_cache( @@ -448,6 +453,19 @@ def test_qsa_ring_capacity_covers_one_speculative_step( assert spec.block_size == expected +def test_qsa_kv_cache_dtype_honors_skip_layers() -> None: + """``--kv-cache-dtype-skip-layers`` keeps the listed QSA layers unquantized. + + The MTP layer's own attention is the one that matters on Flash-Next: FP8 + there cuts draft acceptance at depth while the target layers stay FP8. + """ + cache_config = SimpleNamespace(cache_dtype="fp8", kv_cache_dtype_skip_layers=["48"]) + assert qsa_kv_cache_dtype(cache_config, "mtp.layers.48.self_attn") == "auto" + assert qsa_kv_cache_dtype(cache_config, "model.layers.47.self_attn") == "fp8" + cache_config.kv_cache_dtype_skip_layers = [] + assert qsa_kv_cache_dtype(cache_config, "mtp.layers.48.self_attn") == "fp8" + + @requires_qsa_kernels def test_qsa_compressed_metadata_keeps_dummy_slots_inert() -> None: device = torch.device("cuda") diff --git a/tests/models/registry.py b/tests/models/registry.py index b764cad595..31c3c5433a 100644 --- a/tests/models/registry.py +++ b/tests/models/registry.py @@ -581,6 +581,10 @@ _EMBEDDING_EXAMPLE_MODELS = { trust_remote_code=True, hf_overrides={"model_type": "deepseek_v3", "auto_map": None}, ), + "EmbeddingGemma2Model": _HfExamplesInfo( + "google/embeddinggemma-2", + is_available_online=False, + ), "Gemma2Model": _HfExamplesInfo("BAAI/bge-multilingual-gemma2"), "Gemma3TextModel": _HfExamplesInfo("google/embeddinggemma-300m"), "GteModel": _HfExamplesInfo( @@ -771,10 +775,7 @@ _MULTIMODAL_EXAMPLE_MODELS = { # [Decoder-only] "AriaForConditionalGeneration": _HfExamplesInfo("rhymes-ai/Aria"), "AudioFlamingo3ForConditionalGeneration": _HfExamplesInfo( - "nvidia/audio-flamingo-3-hf", - transformers_version_reason={ - "vllm": "Needs https://github.com/huggingface/transformers/pull/43538" - }, + "nvidia/audio-flamingo-3-hf" ), "BagelForConditionalGeneration": _HfExamplesInfo("ByteDance-Seed/BAGEL-7B-MoT"), "BailingMoeV3VLForConditionalGeneration": _HfExamplesInfo( @@ -813,12 +814,7 @@ _MULTIMODAL_EXAMPLE_MODELS = { "nvidia/Cosmos3-Edge", max_model_len=4096, use_original_num_layers=True, - hf_overrides={ - "text_config": { - "num_hidden_layers": 2, - "hybrid_override_pattern": "*-", - } - }, + hf_overrides={"text_config": {"layers_block_type": ["full_attention", "mlp"]}}, ), "DeepseekVLV2ForCausalLM": _HfExamplesInfo( "deepseek-ai/deepseek-vl2-tiny", diff --git a/tests/models/test_deepseek_v41_decoder_replay_layers.py b/tests/models/test_deepseek_v41_decoder_replay_layers.py index 4af797a1e6..614d903ca1 100644 --- a/tests/models/test_deepseek_v41_decoder_replay_layers.py +++ b/tests/models/test_deepseek_v41_decoder_replay_layers.py @@ -2,15 +2,26 @@ # SPDX-FileCopyrightText: Copyright contributors to the vLLM project """DecoderReplayLayers runs the replay layers on the forward's replay batch.""" +from unittest.mock import MagicMock + import pytest import torch +from vllm.config import CompilationConfig, CUDAGraphMode from vllm.forward_context import ( + BatchDescriptor, ForwardContext, get_forward_context, override_forward_context, ) -from vllm.models.deepseek_v41.decoder_replay_layers import DecoderReplayLayers +from vllm.models.deepseek_v41.decoder_replay_layers import ( + DecoderReplayLayers, + ReplayBatch, +) +from vllm.models.deepseek_v41.nvidia.decoder_replay_cudagraph import ( + DecoderReplayCudaGraphManager, +) +from vllm.v1.worker.gpu import cudagraph_utils pytestmark = pytest.mark.skipif(not torch.cuda.is_available(), reason="needs CUDA") @@ -32,9 +43,9 @@ def _context(metadata): def _set_replay_batch(layers, rows, metadata): - layers.rows = torch.tensor(rows, device=DEVICE) - layers.forward_context = ForwardContext( - no_compile_layers={}, attn_metadata=metadata, slot_mapping={} + layers.replay_batch = ReplayBatch( + torch.tensor(rows, device=DEVICE), + ForwardContext(no_compile_layers={}, attn_metadata=metadata, slot_mapping={}), ) @@ -56,10 +67,8 @@ def _states(num_tokens: int): ) -def test_run_gathers_states_and_realigns_shared_indexer_buffers(): - topk = torch.arange(NUM_TOKENS * 4, device=DEVICE).view(NUM_TOKENS, 4).int() - candidates = torch.arange(NUM_TOKENS * 3, device=DEVICE).view(NUM_TOKENS, 3).int() - topk_before, candidates_before = topk.clone(), candidates.clone() +def test_run_gathers_states(): + """Every row's KV is written first; the layers then run on the replay rows.""" seen = {} def run_layers(hidden_states, positions, input_ids, pre_mix, post, res, residual): @@ -69,7 +78,8 @@ def test_run_gathers_states_and_realigns_shared_indexer_buffers(): seen["is_padding"] = replay_context.is_padding return residual, pre_mix - layers = DecoderReplayLayers(WINDOW, run_layers, [topk, candidates]) + write_batch_kv = MagicMock() + layers = DecoderReplayLayers(WINDOW, run_layers, write_batch_kv, set(), "swa") states = _states(NUM_TOKENS) hidden, residual = states[0], states[-1] full, sub = object(), object() @@ -79,11 +89,10 @@ def test_run_gathers_states_and_realigns_shared_indexer_buffers(): outputs = layers(*states) assert get_forward_context() is context + write_batch_kv.assert_called_once_with(*states) rows = torch.tensor(REPLAY_ROWS, device=DEVICE) assert torch.equal(seen["hidden_states"], hidden[rows]) assert seen["attn_metadata"] is sub and seen["is_padding"] is None - assert torch.equal(topk[: len(REPLAY_ROWS)], topk_before[rows]) - assert torch.equal(candidates[: len(REPLAY_ROWS)], candidates_before[rows]) assert len(outputs) == 2 and outputs[0].shape == residual.shape assert torch.equal(outputs[0][rows], residual[rows]) assert outputs[0][1:173].abs().sum() == 0 @@ -96,9 +105,77 @@ def test_no_replay_batch_runs_the_whole_batch(): seen["attn_metadata"] = get_forward_context().attn_metadata return (hidden_states,) - layers = DecoderReplayLayers(WINDOW, run_layers, []) + write_batch_kv = MagicMock() + layers = DecoderReplayLayers(WINDOW, run_layers, write_batch_kv, set(), "swa") states = _states(NUM_TOKENS) full = object() with override_forward_context(_context(full)): (output,) = layers(*states) assert output is states[0] and seen["attn_metadata"] is full + write_batch_kv.assert_not_called() + + +def _graph_context(num_tokens): + return ForwardContext( + no_compile_layers={}, + attn_metadata={}, + slot_mapping={}, + cudagraph_runtime_mode=CUDAGraphMode.PIECEWISE, + batch_descriptor=BatchDescriptor(num_tokens=num_tokens), + ) + + +def test_piecewise_graphs_break_out_by_size(): + """A PIECEWISE graph's KV write and break are fixed at capture, when no replay + batch is set, so both follow the graph's size rather than the replay batch.""" + write_batch_kv = MagicMock() + layers = DecoderReplayLayers( + WINDOW, lambda *states: ("whole",), write_batch_kv, set(), "swa" + ) + layers._run_in_graph_break = lambda *states: ("break",) # type: ignore[method-assign] + states = _states(NUM_TOKENS) + with override_forward_context(_graph_context(1024)): + assert layers(*states) == ("whole",) # no threshold: no break + layers.trim_threshold = 768 + assert layers(*states) == ("break",) + _set_replay_batch(layers, REPLAY_ROWS, object()) + with override_forward_context(_graph_context(576)): + assert not layers.uses_replay_batch() # a replay batch alone does not break out + write_batch_kv.assert_called_once_with(*states) + + +def test_replay_graph_matches_eager(monkeypatch): + """The replay graph of the next captured size pads and runs the rows.""" + monkeypatch.setattr(cudagraph_utils, "get_pp_group", MagicMock) + capture = torch.cuda.stream(torch.cuda.Stream()) + monkeypatch.setattr(cudagraph_utils, "graph_capture", lambda device: capture) + compilation = CompilationConfig(decoder_replay_cudagraph_capture_sizes=[4]) + cfg = MagicMock(compilation_config=compilation, speculative_config=None) + # Mocking it true would capture into the cuMem pool, which breaks on ROCm. + cfg.use_cumem_cudagraph_pool = False + cfg.scheduler_config = MagicMock(max_num_seqs=2, max_num_batched_tokens=64) + cfg.cache_config.use_kda_recoverssm = False + cfg.model_config.hf_config = MagicMock(hc_mult=HC, hidden_size=HIDDEN) + cfg.model_config.dtype = torch.float32 + + def run_layers(hidden, positions, ids, pre, post, mix, residual): + return residual + hidden[:, None] + positions[:, None, None], pre + ids[:, None] + + layers = DecoderReplayLayers(4, run_layers, MagicMock(), set(), "swa") + manager = DecoderReplayCudaGraphManager(cfg, DEVICE, layers) + + def prepare(desc): + rows = torch.arange(desc.num_tokens) + layers.replay_batch = ReplayBatch(rows, _graph_context(desc.num_tokens)) + + manager.capture_replay_graphs(prepare) + states = [ + torch.randn(12, *b.shape[1:], device=DEVICE).to(b.dtype) + for b in manager.input_buffers + ] + rows = torch.tensor([11, 2, 7], device=DEVICE) + desc = manager.dispatch(2, 3, None, 0) + with override_forward_context(_graph_context(desc.num_tokens)): + actual = manager.run(rows, states) + for a, e in zip(actual, run_layers(*(t[rows] for t in states))): + torch.testing.assert_close(a, e, rtol=0, atol=0) diff --git a/tests/models/test_deepseek_v41_replay_batch.py b/tests/models/test_deepseek_v41_replay_batch.py index f6efe5c516..bcd4db64b2 100644 --- a/tests/models/test_deepseek_v41_replay_batch.py +++ b/tests/models/test_deepseek_v41_replay_batch.py @@ -11,7 +11,14 @@ import pytest import torch from vllm.config import CUDAGraphMode +from vllm.forward_context import ( + BatchDescriptor, + ForwardContext, + override_forward_context, +) +from vllm.models.deepseek_v41.decoder_replay_layers import DecoderReplayLayers from vllm.models.deepseek_v41.nvidia.model_state import DeepseekV41ModelState +from vllm.v1.worker.gpu.cudagraph_utils import BatchExecutionDescriptor from vllm.v1.worker.gpu.input_batch import InputBatch, InputBuffers from vllm.v1.worker.gpu.model_states.default import DefaultModelState @@ -57,7 +64,8 @@ def state(monkeypatch): cfg.scheduler_config.max_num_batched_tokens = 1024 cfg.parallel_config.data_parallel_size = 1 cfg.compilation_config.fast_moe_cold_start = False - layers = SimpleNamespace(window=WINDOW, rows=None, forward_context=None) + cfg.compilation_config.cudagraph_mode = CUDAGraphMode.NONE + layers = DecoderReplayLayers(WINDOW, MagicMock(), MagicMock(), set(), "swa_first") model = SimpleNamespace(token_lookback_depth=0, decoder_replay_layers=layers) builds: list = [] @@ -70,7 +78,7 @@ def state(monkeypatch): attn_metadata=object(), ) builds.append(build) - return {"swa": build.attn_metadata} + return {"swa": build.attn_metadata, "swa_first": build.attn_metadata} monkeypatch.setattr(DefaultModelState, "prepare_attn", prepare_attn) state = DeepseekV41ModelState(cfg, model, None, DEVICE) @@ -118,7 +126,8 @@ def _slot_mappings(num_tokens: int) -> torch.Tensor: def _prepare(state, batch, cg_mode): - """Runs prepare_attn; returns the replay batch and the replay build's inputs.""" + """Runs prepare_attn; returns the replay batch and the replay build's inputs. + The first replay layer's sliding-window build follows the replay build.""" state.prepare_attn( batch, cg_mode, @@ -127,13 +136,8 @@ def _prepare(state, batch, cg_mode): [], GROUPS, ) - layers = state.decoder_replay_layers - if layers.rows is None: - assert layers.forward_context is None - return None, None - return SimpleNamespace( - rows=layers.rows, forward_context=layers.forward_context - ), state.builds[-1] + replay = state.decoder_replay_layers.replay_batch + return (None, None) if replay is None else (replay, state.builds[-2]) def test_replay_batch_keeps_each_request_window(state): @@ -154,8 +158,16 @@ def test_replay_batch_keeps_each_request_window(state): assert build.replay_start.tolist() == [0, 300 - WINDOW, 50] assert torch.equal(sub.positions, batch.positions[rows]) assert torch.equal(build.slot_mappings, _slot_mappings(401)[:, rows]) + # The first replay layer keeps the encoder-side window starts: it wrote + # every row's window KV. + first = state.builds[-1] + assert first.batch is sub and first.cg_mode == CUDAGraphMode.NONE + assert first.replay_start.tolist() == [0, 0, 50] context = replay.forward_context - assert context.attn_metadata == {"swa": build.attn_metadata} + assert context.attn_metadata == { + "swa": build.attn_metadata, + "swa_first": first.attn_metadata, + } assert torch.equal(context.slot_mapping["mla"], _slot_mappings(401)[1, rows]) assert context.dp_metadata is None and context.is_padding is None @@ -191,9 +203,10 @@ def test_replay_batch_keeps_adaptive_verification_query_bound(state): assert replay is not None and build.batch.max_query_len == 200 -def test_only_eager_steps_trim(state): - """A CUDA graph keeps the layers on its whole batch; an eager step trims a - padded batch to its real rows.""" +def test_graph_steps_trim_only_piecewise_at_threshold(state): + """A CUDA graph keeps the layers on its whole batch, unless it is a PIECEWISE + one at the trim threshold or above, as its capture decided; an eager step + trims a batch to its rows.""" batch = _input_batch(QUERY_LENS, SEQ_LENS, PREFILLING, num_tokens_after_padding=512) for cg_mode in (CUDAGraphMode.PIECEWISE, CUDAGraphMode.FULL): assert _prepare(state, batch, cg_mode) == (None, None) @@ -202,6 +215,23 @@ def test_only_eager_steps_trim(state): assert replay.rows.tolist() == REPLAY_ROWS assert build.cg_mode == CUDAGraphMode.NONE assert build.batch.num_tokens_after_padding == len(REPLAY_ROWS) + layers = state.decoder_replay_layers + graph = ForwardContext( + no_compile_layers={}, + attn_metadata={}, + slot_mapping={}, + cudagraph_runtime_mode=CUDAGraphMode.PIECEWISE, + batch_descriptor=BatchDescriptor(num_tokens=512), + ) + for threshold in (513, 512): + layers.trim_threshold = threshold + replay, _ = _prepare(state, batch, CUDAGraphMode.PIECEWISE) + with override_forward_context(graph): # the capture's decision + assert ( + layers.uses_replay_batch() == (replay is not None) == (threshold == 512) + ) + assert replay is not None and replay.rows.tolist() == REPLAY_ROWS + assert _prepare(state, batch, CUDAGraphMode.FULL) == (None, None) def test_whole_batch_forwards_get_no_replay_batch(state): @@ -251,9 +281,25 @@ def test_dp_ranks_replay_together(dp_state): def test_idle_dp_rank_dummy_trims_with_its_peers(dp_state): - dummy = _input_batch([512], [512], [False]) + """An idle rank's dummy keeps one row, so peers do not wait on its replay.""" + dummy = _input_batch([256, 256], [256, 256], [False, False]) dp_state.other = (True, 129) replay, _ = _prepare(dp_state, dummy, CUDAGraphMode.NONE) - assert replay is not None and replay.rows.shape[0] == WINDOW + assert replay is not None and replay.rows.shape[0] == 1 dp_metadata = replay.forward_context.dp_metadata - assert dp_metadata.num_tokens_across_dp_cpu.tolist() == [WINDOW, 129] + assert dp_metadata.num_tokens_across_dp_cpu.tolist() == [1, 129] + + +def test_dp_ranks_pad_to_one_replay_graph(dp_state): + """Every rank pads its replay batch to the graph fitting the largest rank's.""" + dp_state.replay_cudagraphs = MagicMock() + dp_state.replay_cudagraphs.dispatch.return_value = BatchExecutionDescriptor( + CUDAGraphMode.PIECEWISE, 512, None + ) + dp_state.other = (True, 300) + batch = _input_batch(QUERY_LENS, SEQ_LENS, PREFILLING) + replay, _ = _prepare(dp_state, batch, CUDAGraphMode.NONE) + assert dp_state.replay_cudagraphs.dispatch.call_args.args[:2] == (3, 300) + assert replay.forward_context.batch_descriptor.num_tokens == 512 + dp_metadata = replay.forward_context.dp_metadata + assert dp_metadata.num_tokens_across_dp_cpu.tolist() == [512, 512] diff --git a/tests/models/test_deepseek_v4_aiter_mega_moe.py b/tests/models/test_deepseek_v4_aiter_mega_moe.py new file mode 100644 index 0000000000..a8f692e07b --- /dev/null +++ b/tests/models/test_deepseek_v4_aiter_mega_moe.py @@ -0,0 +1,186 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +from types import SimpleNamespace + +import pytest +import torch + +from vllm.model_executor.layers.fused_moe import ( + fused_moe_make_expert_params_mapping, +) +from vllm.models.deepseek_v4.amd import mega_moe as mega_moe_module +from vllm.models.deepseek_v4.amd.mega_moe import ( + DeepseekV4AiterMegaMoEExperts, +) +from vllm.models.deepseek_v4.amd.model import DeepseekV4MoE +from vllm.platforms import current_platform + +HIDDEN = 64 +INTER = 64 + + +def _make_experts( + monkeypatch, + *, + ep_rank=0, + ep_size=1, + num_experts=2, + intermediate_size=INTER, + max_num_batched_tokens=3000, + swiglu_limit=None, +): + monkeypatch.setattr( + mega_moe_module, + "get_ep_group", + lambda: SimpleNamespace( + rank_in_group=ep_rank, world_size=ep_size, cpu_group="ep" + ), + ) + vllm_config = SimpleNamespace( + scheduler_config=SimpleNamespace(max_num_batched_tokens=max_num_batched_tokens) + ) + return DeepseekV4AiterMegaMoEExperts( + vllm_config, + num_experts=num_experts, + top_k=2, + hidden_size=HIDDEN, + intermediate_size=intermediate_size, + swiglu_limit=swiglu_limit, + ) + + +@pytest.mark.cpu_test +def test_experts_load_into_local_slots(monkeypatch): + model = torch.nn.Module() + model.ffn = torch.nn.Module() + model.ffn.experts = _make_experts(monkeypatch, ep_rank=1, ep_size=2, num_experts=4) + params = dict(model.named_parameters()) + mapping = fused_moe_make_expert_params_mapping( + model, + ckpt_gate_proj_name="w1", + ckpt_down_proj_name="w2", + ckpt_up_proj_name="w3", + num_experts=4, + routed_experts_prefix="", + ) + + shapes = { + ("w1", "weight"): (INTER, HIDDEN // 2), + ("w3", "weight"): (INTER, HIDDEN // 2), + ("w2", "weight"): (HIDDEN, INTER // 2), + ("w1", "weight_scale"): (INTER, HIDDEN // 32), + ("w3", "weight_scale"): (INTER, HIDDEN // 32), + ("w2", "weight_scale"): (HIDDEN, INTER // 32), + } + shard_value = {"w1": 1, "w3": 2, "w2": 3} + loaded = {} + for expert in range(4): + for (shard, suffix), shape in shapes.items(): + name = f"ffn.experts.{expert}.{shard}.{suffix}" + weight = torch.full( + shape, 10 * expert + shard_value[shard], dtype=torch.uint8 + ) + for param_name, weight_name, expert_id, shard_id in mapping: + if weight_name not in name: + continue + mapped = name.replace(weight_name, param_name) + param = params[mapped] + if param.weight_loader( + param, + weight, + mapped, + shard_id=shard_id, + expert_id=expert_id, + return_success=True, + ): + loaded[name] = mapped + break + + assert sorted({n.split(".")[2] for n in loaded}) == ["2", "3"] + assert len(loaded) == 2 * len(shapes) + for slot, expert in enumerate((2, 3)): + for suffix in ("weight", "weight_scale"): + w13 = params[f"ffn.experts.w13_{suffix}"][slot] + w2 = params[f"ffn.experts.w2_{suffix}"][slot] + assert torch.all(w13[:INTER] == 10 * expert + 1) + assert torch.all(w13[INTER:] == 10 * expert + 2) + assert torch.all(w2 == 10 * expert + 3) + + +@pytest.mark.skipif(not current_platform.is_rocm(), reason="Requires ROCm") +@pytest.mark.parametrize("hash_routing", [False, True]) +@pytest.mark.parametrize("num_tokens", [1, 37]) +def test_mega_moe_routing_matches_reference(monkeypatch, hash_routing, num_tokens): + num_experts, top_k, vocab = 16, 4, 64 + scaling = 2.5 + device = "cuda" + torch.manual_seed(0) + + class Gate(torch.nn.Module): + def __init__(self): + super().__init__() + self.weight = torch.randn(num_experts, HIDDEN, device=device) * 0.1 + self.e_score_correction_bias = ( + None if hash_routing else torch.randn(num_experts, device=device) + ) + self.tid2eid = ( + torch.stack( + [ + torch.randperm(num_experts, device=device)[:top_k] + for _ in range(vocab) + ] + ).to(torch.int32) + if hash_routing + else None + ) + self.bias_vl = None + + def forward(self, x): + return x.float() @ self.weight.t(), None + + captured = {} + + def experts(x, topk_weights, topk_ids): + captured["weights"], captured["ids"] = topk_weights, topk_ids + return 2 * x + + moe = DeepseekV4MoE.__new__(DeepseekV4MoE) + torch.nn.Module.__init__(moe) + moe.gate = Gate() + moe.experts = experts + moe.shared_experts = lambda x: 3 * x + moe.scoring_func = "sqrtsoftplus" + moe.n_activated_experts = top_k + moe.renormalize = True + moe.routed_scaling_factor = scaling + moe.image_sentinel_lo = 0 + + x = torch.randn(num_tokens, HIDDEN, device=device, dtype=torch.bfloat16) + input_ids = torch.randint(0, vocab, (num_tokens,), device=device) + out = moe._forward_mega_moe(x, input_ids) + + ids, weights = captured["ids"], captured["weights"] + assert ids.dtype == torch.int32 and ids.is_contiguous() + assert weights.dtype == torch.float32 and weights.is_contiguous() + + scores = torch.nn.functional.softplus(x.float() @ moe.gate.weight.t()).sqrt() + if hash_routing: + expected_ids = moe.gate.tid2eid[input_ids].long() + else: + expected_ids = (scores + moe.gate.e_score_correction_bias).topk(top_k).indices + expected_weights = scores.gather(1, expected_ids) + expected_weights *= scaling / expected_weights.sum(dim=-1, keepdim=True) + + order = ids.long().sort(dim=-1).indices + expected_order = expected_ids.sort(dim=-1).indices + torch.testing.assert_close( + ids.long().gather(1, order), expected_ids.gather(1, expected_order) + ) + torch.testing.assert_close( + weights.gather(1, order), + expected_weights.gather(1, expected_order), + rtol=1e-3, + atol=1e-4, + ) + torch.testing.assert_close(out, 5 * x) diff --git a/tests/models/test_deepseek_v4_dspark_rocm.py b/tests/models/test_deepseek_v4_dspark_rocm.py index efa97e10d7..a5f8426e4e 100644 --- a/tests/models/test_deepseek_v4_dspark_rocm.py +++ b/tests/models/test_deepseek_v4_dspark_rocm.py @@ -15,7 +15,10 @@ def _make_uninitialized_model(confidence_head): object.__setattr__( model, "model", - SimpleNamespace(confidence_head=confidence_head), + SimpleNamespace( + confidence_head=confidence_head, + layers=[SimpleNamespace(ffn=SimpleNamespace(use_mega_moe=False))], + ), ) return model diff --git a/tests/models/test_deepseek_v4_mega_moe.py b/tests/models/test_deepseek_v4_mega_moe.py index 9ee0212fba..76915b94a3 100644 --- a/tests/models/test_deepseek_v4_mega_moe.py +++ b/tests/models/test_deepseek_v4_mega_moe.py @@ -6,9 +6,15 @@ from types import SimpleNamespace import pytest import torch +from vllm.model_executor.layers.fused_moe.deep_gemm_mega_moe import ( + DeepGemmSm100MegaMoEBackend, +) from vllm.model_executor.layers.fused_moe.routed_experts_capturer import ( bind_routed_experts_capturer, ) +from vllm.model_executor.layers.quantization.utils.fp8_utils import ( + deepgemm_post_process_weight_scale_block, +) from vllm.models.deepseek_v4.nvidia.dspark import DSparkDeepseekV4ForCausalLM from vllm.models.deepseek_v4.nvidia.model import ( DeepseekV4ForCausalLM, @@ -23,6 +29,7 @@ from vllm.models.deepseek_v41.common.mm_preprocess import IMAGE_SENTINEL_BASE_ID from vllm.models.deepseek_v41.nvidia.model import DeepseekV4MoE as DeepseekV41MoE from vllm.platforms import current_platform from vllm.transformers_utils.configs.deepseek_v41 import DeepseekV41Config +from vllm.utils.deep_gemm import is_deep_gemm_supported from vllm.utils.torch_utils import set_default_torch_dtype pytestmark = pytest.mark.skipif( @@ -447,7 +454,9 @@ def test_deepseek_v4_mega_moe_preserves_checkpoint_dimensions( ): param.data.random_(1, 128) originals.append(param.data.clone()) - monkeypatch.setattr(experts, "_check_runtime_supported", lambda: None) + monkeypatch.setattr( + experts, "_ensure_backend", lambda: DeepGemmSm100MegaMoEBackend() + ) monkeypatch.setattr( "vllm.utils.deep_gemm._import_deep_gemm", lambda: SimpleNamespace( @@ -473,19 +482,23 @@ def test_deepseek_v4_mega_moe_preserves_checkpoint_dimensions( @pytest.mark.parametrize( - "hidden_size,intermediate_size,block_size,mxfp8", + "hidden_size,intermediate_size,block_size,mxfp8,packed", [ - pytest.param(128, 512, 128, False, id="aligned-block128"), - pytest.param(128, 512, 128, True, id="aligned-block128-mxfp8"), - pytest.param(5120, 2304, 32, False, id="deepseek-v41-flash"), - pytest.param(128, 2304, 128, False, id="native-block128"), - pytest.param(5120, 2304, 32, True, id="deepseek-v41-flash-mxfp8"), + pytest.param(128, 512, 128, False, False, id="aligned-block128"), + pytest.param(128, 512, 128, False, True, id="aligned-block128-packed"), + pytest.param(128, 512, 128, True, False, id="aligned-block128-mxfp8"), + pytest.param(5120, 2304, 32, False, False, id="deepseek-v41-flash"), + pytest.param(128, 2304, 128, False, False, id="native-block128"), + pytest.param(128, 2304, 128, False, True, id="native-block128-packed"), + pytest.param(5120, 2304, 32, True, False, id="deepseek-v41-flash-mxfp8"), ], ) def test_deepseek_v4_mega_moe_finalizes_native_shared_expert_weights( - monkeypatch, hidden_size, intermediate_size, block_size, mxfp8 + monkeypatch, hidden_size, intermediate_size, block_size, mxfp8, packed ): """Shared fusion preserves checkpoint weights and scales at native widths.""" + if packed and not is_deep_gemm_supported(): + pytest.skip("Packing the shared scales requires DeepGEMM") class FakeDeepGemm: transformed_dims: list[tuple[int, int]] = [] @@ -539,7 +552,9 @@ def test_deepseek_v4_mega_moe_finalizes_native_shared_expert_weights( intermediate_size=intermediate_size, num_shared_experts=1, ) - experts._check_runtime_supported = lambda: None + monkeypatch.setattr( + experts, "_ensure_backend", lambda: DeepGemmSm100MegaMoEBackend() + ) def fp8_parameter(*shape): return torch.nn.Parameter( @@ -593,6 +608,18 @@ def test_deepseek_v4_mega_moe_finalizes_native_shared_expert_weights( .repeat_interleave(block_k // 32, dim=1) ) originals.append((linear.weight.view(torch.uint8).clone(), scale_1x32)) + if packed: + linear.weight_scale_inv = torch.nn.Parameter( + deepgemm_post_process_weight_scale_block( + scale.data.cuda().unsqueeze(0), + *linear.weight.shape, + quant_block_shape=linear.weight_block_size, + num_groups=1, + ) + .squeeze(0) + .cpu(), + requires_grad=False, + ) monkeypatch.setattr("vllm.utils.deep_gemm._import_deep_gemm", lambda: FakeDeepGemm) original_gate_up_ptr = shared_experts.gate_up_proj.weight.data_ptr() @@ -644,6 +671,10 @@ def test_deepseek_v4_mega_moe_does_not_double_add_fused_shared_expert( class FakeExperts(torch.nn.Module): has_fused_shared_experts = fused + _backend = DeepGemmSm100MegaMoEBackend() + + def _ensure_backend(self): + return self._backend def forward(self, hidden_states, *args, **kwargs): return torch.ones_like(hidden_states) diff --git a/tests/models/test_deepseek_v4_vl_rocm.py b/tests/models/test_deepseek_v4_vl_rocm.py index 036efbaa42..762f1cd6c0 100644 --- a/tests/models/test_deepseek_v4_vl_rocm.py +++ b/tests/models/test_deepseek_v4_vl_rocm.py @@ -7,7 +7,7 @@ import pytest import torch from torch import nn -from vllm.config import ParallelConfig +from vllm.config import KernelConfig, ParallelConfig from vllm.model_executor.models.utils import WeightsMapper from vllm.platforms import current_platform @@ -58,6 +58,7 @@ def test_rocm_moe_routing_and_shared_experts_match_reference( model_config=SimpleNamespace(hf_config=config), quant_config=None, parallel_config=ParallelConfig(), + kernel_config=default_vllm_config.kernel_config, ), prefix="model.layers.0.ffn", ) @@ -230,6 +231,7 @@ def test_rocm_moe_wires_vision_routing_on_hash_and_regular_layers( model_config=SimpleNamespace(hf_config=config), quant_config=None, parallel_config=ParallelConfig(), + kernel_config=KernelConfig(), ) hash_moe = rocm_model.DeepseekV4MoE(vllm_config, prefix="model.layers.0.ffn") diff --git a/tests/models/test_dspark_mla.py b/tests/models/test_dspark_mla.py index f77832d608..53a93b940c 100644 --- a/tests/models/test_dspark_mla.py +++ b/tests/models/test_dspark_mla.py @@ -85,6 +85,7 @@ def test_dspark_markov_head_is_replicated( assert head.markov_w2.tp_size == 1 assert head.markov_w1.weight.shape == (128, 8) assert head.markov_w2.weight.shape == (128, 8) + head.markov_w2.quant_method.process_weights_after_loading(head.markov_w2) def fail_collective(*args, **kwargs): raise AssertionError("replicated Markov head must not invoke TP collectives") @@ -335,3 +336,36 @@ def test_dsv4_context_kv_uses_one_stacked_wkv_projection(monkeypatch): assert torch.equal(calls[1][1], stacked_output.view(2, 3, 4)[:, 2] + 2) assert calls[0][3] is slot_mappings[0] assert calls[1][3] is slot_mappings[2] + + +@pytest.mark.cpu_test +def test_k3_dspark_mla_kv_cache_spec_groups_with_target_mla(): + """The draft's MLA layers must share a KV cache group with the target's.""" + from vllm.model_executor.layers.attention.mla_attention import MLAAttention + from vllm.models.kimi_k3.nvidia.mla import MultiHeadLatentAttention + from vllm.v1.core.kv_cache_utils import _get_kv_cache_groups_uniform_page_size + + vllm_config = SimpleNamespace( + model_config=None, cache_config=SimpleNamespace(block_size=64) + ) + target_attn = SimpleNamespace( + kv_cache_dtype="fp8", + head_size=576, + sliding_window=None, + indexer=None, + non_causal_multi_token_decode=False, + attn_backend=SimpleNamespace(get_name=lambda: "ROCM_AITER_MLA"), + _uses_flat_kv_cache=lambda: False, + ) + draft_attn = SimpleNamespace( + kv_cache_dtype="fp8", head_size=576, non_causal_multi_token_decode=True + ) + target_spec = MLAAttention.get_kv_cache_spec(target_attn, vllm_config) + draft_spec = MultiHeadLatentAttention.get_kv_cache_spec(draft_attn, vllm_config) + + kv_cache_spec = {f"model.layers.{i}.attn": target_spec for i in range(24)} + kv_cache_spec |= {f"draft.layers.{i}.attn": draft_spec for i in range(5)} + + groups = _get_kv_cache_groups_uniform_page_size(kv_cache_spec) + assert len(groups) == 1 + assert len(groups[0].layer_names) == 29 diff --git a/tests/models/transformers/test_layer_registry.py b/tests/models/transformers/test_layer_registry.py index 9e20e2ba54..555d87e379 100644 --- a/tests/models/transformers/test_layer_registry.py +++ b/tests/models/transformers/test_layer_registry.py @@ -7,6 +7,9 @@ is set and the symbol exists, and otherwise falls back to `vllm.model_executor.layers.`. These tests pin that contract and the logging that reports which source was used. + +CPU-only: no engine is built. The end-to-end counterpart that serves a model +through `vllm_runner` lives in `test_layer_registry_e2e.py` (GPU job). """ import logging @@ -14,7 +17,6 @@ import sys import types import pytest -import torch from vllm.model_executor.models.transformers import layers @@ -42,16 +44,16 @@ def test_falls_back_to_vllm_when_disabled(monkeypatch, fake_hw_layernorm): assert layers._resolve("layernorm", "RMSNorm") is VllmRMSNorm -def test_uses_hw_agnostic_when_enabled(monkeypatch, fake_hw_layernorm, caplog): +def test_uses_hw_agnostic_when_enabled(monkeypatch, fake_hw_layernorm, caplog_vllm): """Enabled and available: the hw-agnostic class is used and logged.""" monkeypatch.setenv("VLLM_USE_HW_AGNOSTIC", "1") - with caplog.at_level(logging.INFO): + with caplog_vllm.at_level(logging.INFO): resolved = layers._resolve("layernorm", "RMSNorm") assert resolved is fake_hw_layernorm.RMSNorm - assert "Using hw-agnostic layer: RMSNorm" in caplog.text + assert "Using hw-agnostic layer: RMSNorm" in caplog_vllm.text -def test_falls_back_when_symbol_missing(monkeypatch, caplog): +def test_falls_back_when_symbol_missing(monkeypatch, caplog_vllm): """Enabled but the symbol is not ported: fall back to vLLM and warn.""" monkeypatch.setenv("VLLM_USE_HW_AGNOSTIC", "1") # A hw-agnostic module without the requested attribute triggers fallback. @@ -59,10 +61,10 @@ def test_falls_back_when_symbol_missing(monkeypatch, caplog): monkeypatch.setitem(sys.modules, HW_MODULE, empty) from vllm.model_executor.layers.layernorm import RMSNorm as VllmRMSNorm - with caplog.at_level(logging.WARNING): + with caplog_vllm.at_level(logging.WARNING): resolved = layers._resolve("layernorm", "RMSNorm") assert resolved is VllmRMSNorm - assert "falling back to default" in caplog.text + assert "falling back to default" in caplog_vllm.text def test_act_and_mul_falls_back_for_unknown_activation( @@ -76,102 +78,3 @@ def test_act_and_mul_falls_back_for_unknown_activation( from vllm.model_executor.layers.activation import GeluAndMul assert isinstance(layers.get_act_and_mul_fn("gelu"), GeluAndMul) - - -@pytest.fixture(scope="module") -def tiny_llama_path(tmp_path_factory): - """A randomly-initialized microscopic Llama saved to disk (with an ungated - tokenizer) so vLLM can load it like any local checkpoint.""" - from transformers import AutoTokenizer, LlamaConfig, LlamaForCausalLM - - tokenizer = AutoTokenizer.from_pretrained("hf-internal-testing/llama-tokenizer") - config = LlamaConfig( - vocab_size=tokenizer.vocab_size, - hidden_size=128, - intermediate_size=256, - num_hidden_layers=2, - num_attention_heads=4, - rms_norm_eps=1e-6, - hidden_act="silu", - ) - torch.manual_seed(0) - model = LlamaForCausalLM(config) - - path = tmp_path_factory.mktemp("tiny_llama") - model.save_pretrained(path) - tokenizer.save_pretrained(path) - return str(path) - - -# Registered names of the layers the backend can -# currently route to hw-agnostic implementations. -_COVERED_LAYERS = ("rms_norm", "silu_and_mul") - - -def _layer_providers(model) -> dict[str, str]: - """Map each covered layer type present in the model to the provider its - implementation came from (``hw_agnostic`` or ``vllm``). - """ - - def provider_of(module) -> str | None: - for cls in type(module).__mro__: - if "hw_agnostic.layers" in cls.__module__: - return "hw_agnostic" - if ".model_executor.layers." in cls.__module__: - return "vllm" - return None - - providers: dict[str, str] = {} - for module in model.modules(): - name = getattr(module, "name", None) - if ( - name in _COVERED_LAYERS - and name not in providers - and (prov := provider_of(module)) is not None - ): - providers[name] = prov - return providers - - -def _serve(vllm_runner, model_path, prompts): - """Serve the model through the backend; return (layer_providers, logprobs).""" - with vllm_runner( - model_path, - model_impl="transformers", - max_model_len=64, - enforce_eager=True, - gpu_memory_utilization=0.3, - ) as runner: - assert runner.llm.llm_engine.model_config.using_transformers_backend() - providers = runner.apply_model(_layer_providers)[0] - outputs = runner.generate_greedy_logprobs( - prompts, max_tokens=32, num_logprobs=5 - ) - return providers, outputs - - -def test_hw_agnostic_matches_vllm_end_to_end(monkeypatch, vllm_runner, tiny_llama_path): - """Serving the tiny model with hw-agnostic layers matches the vLLM baseline.""" - # spawn: worker re-imports layers with the env set (see docstring). - monkeypatch.setenv("VLLM_WORKER_MULTIPROC_METHOD", "spawn") - # apply_model pickles the introspection function. - monkeypatch.setenv("VLLM_ALLOW_INSECURE_SERIALIZATION", "1") - from ..utils import check_logprobs_close - - prompts = ["The capital of France is", "vLLM is"] - - monkeypatch.setenv("VLLM_USE_HW_AGNOSTIC", "0") - vllm_providers, vllm_outputs = _serve(vllm_runner, tiny_llama_path, prompts) - # Both replaceable layers present in a Llama block must be vLLM's here. - assert vllm_providers == {"rms_norm": "vllm", "silu_and_mul": "vllm"} - - monkeypatch.setenv("VLLM_USE_HW_AGNOSTIC", "1") - hw_providers, hw_outputs = _serve(vllm_runner, tiny_llama_path, prompts) - assert hw_providers == {"rms_norm": "hw_agnostic", "silu_and_mul": "hw_agnostic"} - - check_logprobs_close( - outputs_0_lst=vllm_outputs, - outputs_1_lst=hw_outputs, - name_0="vllm", - name_1="hw_agnostic", - ) diff --git a/tests/models/transformers/test_layer_registry_e2e.py b/tests/models/transformers/test_layer_registry_e2e.py new file mode 100644 index 0000000000..13d4b80a16 --- /dev/null +++ b/tests/models/transformers/test_layer_registry_e2e.py @@ -0,0 +1,113 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""End-to-end check of the Transformers backend's hw-agnostic layers. + +Serves a tiny Llama through `vllm_runner` with and without +`VLLM_USE_HW_AGNOSTIC` and checks the layer providers and logprobs. Needs a +real engine, so it runs in the GPU job. The CPU-only unit tests for the +resolution logic (`layers._resolve`) are in `test_layer_registry.py`. +""" + +import pytest +import torch + + +@pytest.fixture(scope="module") +def tiny_llama_path(tmp_path_factory): + """A randomly-initialized microscopic Llama saved to disk (with an ungated + tokenizer) so vLLM can load it like any local checkpoint.""" + from transformers import AutoTokenizer, LlamaConfig, LlamaForCausalLM + + tokenizer = AutoTokenizer.from_pretrained("hf-internal-testing/llama-tokenizer") + config = LlamaConfig( + vocab_size=tokenizer.vocab_size, + hidden_size=128, + intermediate_size=256, + num_hidden_layers=2, + num_attention_heads=4, + rms_norm_eps=1e-6, + hidden_act="silu", + ) + torch.manual_seed(0) + model = LlamaForCausalLM(config) + + path = tmp_path_factory.mktemp("tiny_llama") + model.save_pretrained(path) + tokenizer.save_pretrained(path) + return str(path) + + +# Registered names of the layers the backend can +# currently route to hw-agnostic implementations. +_COVERED_LAYERS = ("rms_norm", "silu_and_mul") + + +def _layer_providers(model) -> dict[str, str]: + """Map each covered layer type present in the model to the provider its + implementation came from (``hw_agnostic`` or ``vllm``). + """ + + def provider_of(module) -> str | None: + for cls in type(module).__mro__: + if "hw_agnostic.layers" in cls.__module__: + return "hw_agnostic" + if ".model_executor.layers." in cls.__module__: + return "vllm" + return None + + providers: dict[str, str] = {} + for module in model.modules(): + name = getattr(module, "name", None) + if ( + name in _COVERED_LAYERS + and name not in providers + and (prov := provider_of(module)) is not None + ): + providers[name] = prov + return providers + + +def _serve(vllm_runner, model_path, prompts): + """Serve the model through the backend; return (layer_providers, logprobs).""" + with vllm_runner( + model_path, + model_impl="transformers", + max_model_len=64, + enforce_eager=True, + gpu_memory_utilization=0.3, + ) as runner: + assert runner.llm.llm_engine.model_config.using_transformers_backend() + providers = runner.apply_model(_layer_providers)[0] + outputs = runner.generate_greedy_logprobs( + prompts, max_tokens=32, num_logprobs=5 + ) + return providers, outputs + + +def test_hw_agnostic_matches_vllm_end_to_end(monkeypatch, vllm_runner, tiny_llama_path): + """Serving the tiny model with hw-agnostic layers matches the vLLM baseline.""" + # spawn: `layers` resolves its symbols at import time, so the worker must + # re-import it with VLLM_USE_HW_AGNOSTIC already set (fork would inherit + # the parent's already-resolved module). + monkeypatch.setenv("VLLM_WORKER_MULTIPROC_METHOD", "spawn") + # apply_model pickles the introspection function. + monkeypatch.setenv("VLLM_ALLOW_INSECURE_SERIALIZATION", "1") + from ..utils import check_logprobs_close + + prompts = ["The capital of France is", "vLLM is"] + + monkeypatch.setenv("VLLM_USE_HW_AGNOSTIC", "0") + vllm_providers, vllm_outputs = _serve(vllm_runner, tiny_llama_path, prompts) + # Both replaceable layers present in a Llama block must be vLLM's here. + assert vllm_providers == {"rms_norm": "vllm", "silu_and_mul": "vllm"} + + monkeypatch.setenv("VLLM_USE_HW_AGNOSTIC", "1") + hw_providers, hw_outputs = _serve(vllm_runner, tiny_llama_path, prompts) + assert hw_providers == {"rms_norm": "hw_agnostic", "silu_and_mul": "hw_agnostic"} + + check_logprobs_close( + outputs_0_lst=vllm_outputs, + outputs_1_lst=hw_outputs, + name_0="vllm", + name_1="hw_agnostic", + ) diff --git a/tests/multimodal/media/test_connector.py b/tests/multimodal/media/test_connector.py index 5949c66b56..f688b51534 100644 --- a/tests/multimodal/media/test_connector.py +++ b/tests/multimodal/media/test_connector.py @@ -1,7 +1,6 @@ # SPDX-License-Identifier: Apache-2.0 # SPDX-FileCopyrightText: Copyright contributors to the vLLM project -import asyncio import mimetypes import os import shutil @@ -292,7 +291,7 @@ async def test_fetch_video_http(video_url: str, num_frames: int): try: video_sync, metadata_sync = connector.fetch_video(video_url) video_async, metadata_async = await connector.fetch_video_async(video_url) - except (TimeoutError, asyncio.TimeoutError) as e: + except TimeoutError as e: pytest.skip(f"Timeout fetching video (CI network flakiness): {e}") assert np.array_equal(video_sync, video_async) @@ -324,7 +323,7 @@ async def test_fetch_video_http_with_dynamic_loader( try: video_sync, metadata_sync = connector.fetch_video(video_url) video_async, metadata_async = await connector.fetch_video_async(video_url) - except (TimeoutError, asyncio.TimeoutError) as e: + except TimeoutError as e: pytest.skip(f"Timeout fetching video (CI network flakiness): {e}") assert np.array_equal(video_sync, video_async) @@ -402,7 +401,7 @@ async def test_allowed_media_domains(video_url: str, num_frames: int): try: video_sync, metadata_sync = connector.fetch_video(video_url) video_async, metadata_async = await connector.fetch_video_async(video_url) - except (TimeoutError, asyncio.TimeoutError) as e: + except TimeoutError as e: pytest.skip(f"Timeout fetching video (CI network flakiness): {e}") assert np.array_equal(video_sync, video_async) diff --git a/tests/multimodal/media/test_image.py b/tests/multimodal/media/test_image.py index 6b7e4ad407..94efb312f2 100644 --- a/tests/multimodal/media/test_image.py +++ b/tests/multimodal/media/test_image.py @@ -1,11 +1,16 @@ # SPDX-License-Identifier: Apache-2.0 # SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import inspect from pathlib import Path import numpy as np import pytest from PIL import Image +from vllm.multimodal.image import ( + ALLOWED_IMAGE_FORMATS, + open_image, +) from vllm.multimodal.media import ImageMediaIO pytestmark = pytest.mark.cpu_test @@ -281,3 +286,72 @@ def test_image_pixel_limit_disabled(monkeypatch): image_io = ImageMediaIO() result = image_io.load_bytes(data) assert result.media.size == (1000, 1000) + + +def test_image_media_io_rejects_postscript_payload(): + """EPS/PostScript bytes must not reach an external renderer.""" + import pybase64 as base64 + + eps = b"""%!PS-Adobe-3.0 EPSF-3.0 +%%BoundingBox: 0 0 10 10 +%%EndComments +0 0 moveto 10 10 lineto stroke +showpage +""" + image_io = ImageMediaIO() + with pytest.raises(ValueError, match="Failed to load image"): + image_io.load_bytes(eps) + + payload = base64.b64encode(eps).decode("ascii") + with pytest.raises(ValueError, match="Failed to load image"): + image_io.load_base64("image/png", payload) + + +def test_image_media_io_accepts_common_raster_formats(tmp_path): + """Common Pillow-writable raster formats remain loadable.""" + image_io = ImageMediaIO() + formats = ( + ("PNG", "ok.png"), + ("JPEG", "ok.jpg"), + ("WEBP", "ok.webp"), + ("GIF", "ok.gif"), + ("BMP", "ok.bmp"), + ("TIFF", "ok.tiff"), + ("PPM", "ok.ppm"), + ("TGA", "ok.tga"), + ("DDS", "ok.dds"), + ("PCX", "ok.pcx"), + ("SGI", "ok.sgi"), + ("QOI", "ok.qoi"), + ("JPEG2000", "ok.jp2"), + ) + for fmt, name in formats: + path = tmp_path / name + Image.new("RGB", (4, 4), (1, 2, 3)).save(path, format=fmt) + result = image_io.load_bytes(path.read_bytes()) + assert result.media.size == (4, 4) + + +def test_allowed_image_formats_exclude_external_renderer_plugins(): + """Formats that shell out or use stubs stay off the allowlist.""" + denied = {"EPS", "WMF", "BUFR", "GRIB", "HDF5", "MPEG"} + assert denied.isdisjoint(ALLOWED_IMAGE_FORMATS) + + +def test_kimi_fused_vision_rejects_postscript_payload(): + """Kimi fused vision must use the same image format allowlist.""" + from vllm.transformers_utils.processors.kimi_k25_vision_fused import _to_pil + + eps = b"""%!PS-Adobe-3.0 EPSF-3.0 +%%BoundingBox: 0 0 10 10 +%%EndComments +0 0 moveto 10 10 lineto stroke +showpage +""" + with pytest.raises(Image.UnidentifiedImageError): + _to_pil(eps) + + +def test_open_image_does_not_accept_format_override(): + """The allowlist helper must not expose a formats override.""" + assert "formats" not in inspect.signature(open_image).parameters diff --git a/tests/multimodal/test_video.py b/tests/multimodal/test_video.py index d593edba29..407bffed20 100644 --- a/tests/multimodal/test_video.py +++ b/tests/multimodal/test_video.py @@ -13,7 +13,7 @@ import numpy as np import numpy.typing as npt import pytest import torch -from transformers import AutoVideoProcessor +from transformers import AutoVideoProcessor, Qwen3VLVideoProcessor from transformers.video_utils import VideoMetadata from vllm.assets.base import get_vllm_public_assets @@ -35,6 +35,7 @@ from vllm.multimodal.video import ( get_video_loader_backend_for_processor, ) from vllm.multimodal.video_decoders import decode_video, resolve_video_backend_kwargs +from vllm.multimodal.video_decoders.opencv import OpenCVVideoBackendMixin from vllm.multimodal.video_decoders.pynvvideocodec import ( PYNVVIDEOCODEC_DECODER_CACHE_SIZE, PyNvVideoCodecDecoderSlot, @@ -730,6 +731,36 @@ def test_cosmos3_edge_uses_qwen3_vl_video_backend(): assert isinstance(VIDEO_LOADER_REGISTRY.load(backend), Qwen3VLVideoBackend) +def test_qwen3_vl_unknown_source_fps_matches_hf( + dummy_video_path, monkeypatch: pytest.MonkeyPatch +): + """An unknown source fps (reported as 0) samples like HF's fps=None, and + the metadata carries the fps HF assumes, which the frame timestamps use.""" + get_video_metadata = OpenCVVideoBackendMixin.get_video_metadata + + def unknown_fps(cap) -> VideoSourceMetadata: + return get_video_metadata(cap)._replace(original_fps=0.0, duration=0.0) + + monkeypatch.setattr( + OpenCVVideoBackendMixin, "get_video_metadata", staticmethod(unknown_fps) + ) + _, metadata = Qwen3VLVideoBackend.load_bytes( + dummy_video_path.read_bytes(), fps=2, backend="opencv" + ) + + total_frames = metadata["total_num_frames"] + hf_metadata = VideoMetadata(total_num_frames=total_frames, fps=None) + hf_indices = Qwen3VLVideoProcessor().sample_frames(hf_metadata, fps=2).tolist() + assert metadata["frames_indices"] == hf_indices + assert metadata["fps"] == hf_metadata.fps + + source = VideoSourceMetadata(total_frames, original_fps=0.0, duration=0.0) + target = VideoTargetMetadata(num_frames=-1, fps=2, max_duration=300) + assert ( + Qwen3VLVideoBackend.compute_frames_index_to_sample(source, target) == hf_indices + ) + + @pytest.mark.parametrize( "model_repo, expected_loader_cls, hf_sample_kwargs", [ diff --git a/tests/parser/cohere/test_structural_tags.py b/tests/parser/cohere/test_structural_tags.py index 893cfc80f7..149c5229a7 100644 --- a/tests/parser/cohere/test_structural_tags.py +++ b/tests/parser/cohere/test_structural_tags.py @@ -19,6 +19,7 @@ from vllm.entrypoints.generate.base.protocol import ( ) from vllm.entrypoints.openai.chat_completion.protocol import ChatCompletionRequest from vllm.entrypoints.openai.responses.protocol import ResponsesRequest +from vllm.exceptions import VLLMValidationError from vllm.parser.cohere_command import ( CohereCommandParser, CohereNormalizedTool, @@ -339,6 +340,14 @@ class TestAdjustRequestFoldFromStructuredOutputs: _make_chat_request(structured_outputs={"json": json_value}), ) + def test_deeply_nested_json_string_rejected(self, parser) -> None: + """json.loads raises RecursionError on a string nested this deeply.""" + schema = '{"type": "array", "items": ' * 20_000 + "{}" + "}" * 20_000 + with pytest.raises(VLLMValidationError, match="nested too deeply"): + parser.adjust_request( + _make_chat_request(structured_outputs={"json": schema}), + ) + @pytest.mark.parametrize( "construct", [ @@ -450,6 +459,15 @@ class TestCollectToolSchemaDefs: self._tool_calls_json("get_weather", {"location": {"city": 1}}), ) + def test_deeply_nested_parameters_rejected(self) -> None: + """Tool schemas are compiled here, before request validation, and end up + inside an EBNF grammar that the nesting check cannot see later.""" + parameters = json.loads('{"type": "array", "items": ' * 200 + "{}" + "}" * 200) + with pytest.raises(VLLMValidationError, match="nested too deeply"): + collect_tool_schema( + [CohereNormalizedTool(name="deep", parameters=parameters)] + ) + def test_input_parameters_not_mutated(self) -> None: params = json.loads(json.dumps(NESTED_DEFS_PARAMETERS)) collect_tool_schema( diff --git a/tests/parser/engine/test_deepseek_v4.py b/tests/parser/engine/test_deepseek_v4.py index 2cf3a8bb85..f5b66d9dc1 100644 --- a/tests/parser/engine/test_deepseek_v4.py +++ b/tests/parser/engine/test_deepseek_v4.py @@ -643,8 +643,8 @@ class TestImplicitReasoningEnd: # token-ID fast path; the implicit tool-call exit still goes through # the text lexer. assert thinking_parser.reasoning_end_token_ids == {_THINK_END_ID} - assert thinking_parser.find_reasoning_end_offset([7, _THINK_END_ID, 8]) == 1 - assert thinking_parser.find_reasoning_end_offset([7, 8]) == 2 + assert thinking_parser.find_reasoning_end([7, _THINK_END_ID, 8]) == (1, False) + assert thinking_parser.find_reasoning_end([7, 8]) == (2, False) def test_streaming_reasoning_implicit_end(self, thinking_parser): chunks = [ diff --git a/tests/parser/engine/test_minimax_m3.py b/tests/parser/engine/test_minimax_m3.py new file mode 100644 index 0000000000..48c65eeadb --- /dev/null +++ b/tests/parser/engine/test_minimax_m3.py @@ -0,0 +1,253 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import json + +import pytest + +from tests.parser.engine.conftest import make_mock_tokenizer +from tests.parser.engine.streaming_helpers import ( + collect_content, + collect_function_name, + collect_tool_arguments, + simulate_tool_streaming, +) +from vllm.entrypoints.openai.chat_completion.protocol import ( + ChatCompletionToolsParam, + FunctionDefinition, +) +from vllm.parser.minimax_m3 import ( + INVOKE_END, + MAX_ELEMENT_DEPTH, + NAMESPACE, + TOOL_CALL_END, + TOOL_CALL_START, + MinimaxM3Parser, +) + +NS = NAMESPACE + +TOOLS = [ + ChatCompletionToolsParam( + function=FunctionDefinition( + name="create_order", + parameters={ + "type": "object", + "properties": { + "user_id": {"type": "integer"}, + "note": {"type": "string"}, + "tags": {"type": "array", "items": {"type": "string"}}, + "shipping": { + "type": "object", + "properties": {"zip": {"type": "integer"}}, + }, + "either": { + "anyOf": [ + {"type": "array", "items": {"type": "integer"}}, + {"type": "string"}, + ] + }, + "free": {"type": "object"}, + }, + }, + ), + ) +] + + +@pytest.fixture +def parser(): + return MinimaxM3Parser(make_mock_tokenizer({}), tools=TOOLS) + + +def element(name: str, body: str) -> str: + return f"{NS}<{name}>{body}{NS}" + + +def tool_block(*invokes: tuple[str, str]) -> str: + body = "\n".join( + f'{NS}{params}{INVOKE_END}' for name, params in invokes + ) + return f"{TOOL_CALL_START}\n{body}\n{TOOL_CALL_END}" + + +def parse_arguments(parser, request, params: str, name: str = "create_order"): + result = parser.extract_tool_calls(tool_block((name, params)), request) + assert [tc.function.name for tc in result.tool_calls] == [name] + return json.loads(result.tool_calls[0].function.arguments) + + +class TestArguments: + def test_nested_elements_follow_schema(self, parser, mock_request): + params = ( + element("tags", element("item", "a") + element("item", "b")) + + element("shipping", element("zip", "018956")) + + element("either", element("item", "1") + element("item", "2")) + + element("free", element("t", "x") + element("t", "y")) + ) + assert parse_arguments(parser, mock_request, params) == { + "tags": ["a", "b"], + "shipping": {"zip": 18956}, + "either": [1, 2], + "free": {"t": ["x", "y"]}, + } + + def test_empty_elements_follow_schema(self, parser, mock_request): + # The chat template renders [] and {} as empty elements. + params = ( + element("tags", "") + + element("shipping", "") + + element("note", "") + + element("either", "") + ) + assert parse_arguments(parser, mock_request, params) == { + "tags": [], + "shipping": {}, + "note": "", + "either": "", + } + + def test_unknown_tool_keeps_text_and_nests_objects(self, parser, mock_request): + params = element("n", "1") + element( + "list", element("item", "a") + element("item", "b") + ) + assert parse_arguments(parser, mock_request, params, name="other") == { + "n": "1", + "list": {"item": ["a", "b"]}, + } + + def test_mixed_text_uses_reserved_field(self, parser, mock_request): + params = element("free", "hi " + element("k", "v") + " there") + element( + "shipping", "x" + element("$text", "child") + ) + assert parse_arguments(parser, mock_request, params) == { + "free": {"k": "v", "$text": "hi there"}, + "shipping": {"$text": "child", "$$text": "x"}, + } + + def test_leaf_text_is_verbatim_for_strings(self, parser, mock_request): + params = element("note", "\n a & c\n") + element("user_id", " 7 ") + assert parse_arguments(parser, mock_request, params) == { + "note": "\n a & c\n", + "user_id": 7, + } + + def test_block_marker_named_element_is_argument(self, parser, mock_request): + params = element("free", element("tool_call", "v")) + assert parse_arguments(parser, mock_request, params) == { + "free": {"tool_call": "v"}, + } + + @pytest.mark.parametrize( + "tail", + [ + "junk" + element("note", "dropped"), + f"{NS}a{NS}", + f"{NS}unterminated", + ], + ) + def test_malformed_element_keeps_preceding_params(self, parser, mock_request, tail): + params = element("user_id", "1") + tail + assert parse_arguments(parser, mock_request, params) == {"user_id": 1} + + def test_depth_limit(self, parser, mock_request): + def nested(depth: int) -> str: + body = "v" + for _ in range(depth - 1): + body = element("n", body) + return element("free", body) + + accepted = parse_arguments(parser, mock_request, nested(MAX_ELEMENT_DEPTH)) + assert "free" in accepted + rejected = parse_arguments(parser, mock_request, nested(MAX_ELEMENT_DEPTH + 1)) + assert rejected == {} + + +class TestToolBlock: + def test_multiple_invokes_and_prefix_content(self, parser, mock_request): + result = parser.extract_tool_calls( + "Checking.\n" + + tool_block( + ("create_order", element("user_id", "1")), + ("create_order", element("user_id", "2")), + ), + mock_request, + ) + assert result.content == "Checking.\n" + assert [json.loads(tc.function.arguments) for tc in result.tool_calls] == [ + {"user_id": 1}, + {"user_id": 2}, + ] + + def test_text_after_tool_block_is_dropped(self, parser, mock_request): + result = parser.extract_tool_calls( + "pre" + tool_block(("create_order", "")) + "post", mock_request + ) + assert result.content == "pre" + assert len(result.tool_calls) == 1 + + def test_invoke_outside_tool_block_is_content(self, parser, mock_request): + text = f'{NS}{INVOKE_END}' + result = parser.extract_tool_calls(text, mock_request) + assert not result.tools_called + assert result.content == text + + +class TestStreaming: + def test_arguments_arrive_with_invoke_end(self, parser, mock_request): + block = tool_block( + ("create_order", element("user_id", "42") + element("tags", "")) + ) + invoke_end = block.index(INVOKE_END) + results = simulate_tool_streaming( + parser, + mock_request, + ["Sure. ", block[:invoke_end], block[invoke_end:]], + ) + + before_end, at_end = results[1][0], results[2][0] + assert before_end is not None and before_end.tool_calls + assert before_end.tool_calls[0].function.name == "create_order" + assert not before_end.tool_calls[0].function.arguments + assert collect_content(results) == "Sure. " + assert json.loads(collect_tool_arguments([(at_end, "")])) == { + "user_id": 42, + "tags": [], + } + + def test_char_chunks_match_complete_parse(self, parser, mock_request): + text = "a" + tool_block( + ("create_order", element("shipping", element("zip", "1"))), + ("create_order", element("note", "x")), + ) + results = simulate_tool_streaming(parser, mock_request, list(text)) + + assert collect_content(results) == "a" + assert collect_function_name(results) == "create_order" + streamed = [ + tc.function.arguments + for delta, _ in results + if delta and delta.tool_calls + for tc in delta.tool_calls + if tc.function and tc.function.arguments + ] + assert [json.loads(args) for args in streamed] == [ + {"shipping": {"zip": 1}}, + {"note": "x"}, + ] + + def test_truncated_invoke_keeps_completed_params(self, parser, mock_request): + block = tool_block( + ("create_order", element("user_id", "1") + element("note", "x")) + ) + truncated = block[: block.index(f"{NS}")] + results = simulate_tool_streaming(parser, mock_request, [truncated]) + results.append((parser.finish_streaming(), truncated)) + + assert collect_function_name(results) == "create_order" + assert json.loads(collect_tool_arguments(results)) == {"user_id": 1} + + +def test_adjust_request_keeps_special_tokens_skipped(parser, mock_request): + mock_request.skip_special_tokens = True + assert parser.adjust_request(mock_request).skip_special_tokens is True diff --git a/tests/parser/engine/test_parser_engine.py b/tests/parser/engine/test_parser_engine.py index 8fd7c4d262..dc91cef740 100644 --- a/tests/parser/engine/test_parser_engine.py +++ b/tests/parser/engine/test_parser_engine.py @@ -28,12 +28,13 @@ from vllm.entrypoints.openai.chat_completion.protocol import ( from vllm.parser.abstract_parser import DelegatingParser from vllm.parser.engine.adapters import make_adapters from vllm.parser.engine.events import EventType, SemanticEvent -from vllm.parser.engine.parser_engine import ParserEngine +from vllm.parser.engine.parser_engine import ParserEngine, ReasoningEnd from vllm.parser.engine.parser_engine_config import ( ParserEngineConfig, ParserState, Transition, ) +from vllm.parser.glm47_moe import glm47_moe_config from vllm.parser.parser_manager import ParserManager # ── Shared test configs ────────────────────────────────────────────── @@ -158,7 +159,10 @@ class TestReasoningEndTokenIds: (EventType.REASONING_END, EventType.TOOL_CALL_START), ), ) - assert _make_engine(cfg).reasoning_end_token_ids == {201, 202} + engine = _make_engine(cfg) + assert engine.reasoning_end_token_ids == {201, 202} + assert engine.find_reasoning_end([5, 201]) == ReasoningEnd(1, False) + assert engine.find_reasoning_end([5, 202]) == ReasoningEnd(1, True) def test_transition_staying_in_reasoning_is_ignored(self): cfg = _with_reasoning_exits(("THINK_START", ParserState.REASONING, ())) @@ -182,17 +186,40 @@ class TestReasoningEndTokenIds: def test_config_without_reasoning_has_empty_set(self): assert _make_engine(_hermes_config()).reasoning_end_token_ids == frozenset() - def test_find_reasoning_end_offset_returns_first_match(self): + def test_find_reasoning_end_returns_first_match(self): engine = _make_engine() - assert engine.find_reasoning_end_offset([5, 201, 6, 201]) == 1 - assert engine.find_reasoning_end_offset([5, 6]) == 2 - assert engine.find_reasoning_end_offset([]) == 0 + assert engine.find_reasoning_end([5, 201, 6, 201]) == ReasoningEnd(1, False) + assert engine.find_reasoning_end([5, 6]) == ReasoningEnd(2, False) + assert engine.find_reasoning_end([]) == ReasoningEnd(0, False) # Rejected-draft placeholders never match. - assert engine.find_reasoning_end_offset([-1, 201]) == 1 + assert engine.find_reasoning_end([-1, 201]) == ReasoningEnd(1, False) - def test_find_reasoning_end_offset_with_empty_set_returns_none(self): + def test_find_reasoning_end_with_empty_set_returns_none(self): engine = _make_engine(_hermes_config()) - assert engine.find_reasoning_end_offset([201]) is None + assert engine.find_reasoning_end([201]) is None + + +@pytest.mark.parametrize( + ("config", "implicit_token"), + [ + pytest.param(glm47_moe_config(), "", id="glm47_moe"), + ], +) +def test_implicit_reasoning_end_of_parsers( + config: ParserEngineConfig, implicit_token: str +): + literals = sorted(set(config.token_id_terminals.values())) + vocab = {literal: 1000 + i for i, literal in enumerate(literals)} + engine = _make_engine(config, vocab=vocab) + implicit_ids = set() + for token_id in engine.reasoning_end_token_ids: + end = engine.find_reasoning_end([5, token_id, 6]) + assert end is not None and end.offset == 1 + kept = engine.extract_content_ids([5, token_id, 6]) == [token_id, 6] + assert end.implicit == kept + if end.implicit: + implicit_ids.add(token_id) + assert implicit_ids == {vocab[implicit_token]} # ── TestEventsToDelta ──────────────────────────────────────────────── diff --git a/tests/parser/engine/test_replay.py b/tests/parser/engine/test_replay.py index 89791ba29b..e6339f8959 100644 --- a/tests/parser/engine/test_replay.py +++ b/tests/parser/engine/test_replay.py @@ -74,10 +74,11 @@ def _discover_parsers() -> list[_ParserInfo]: if cfg.name not in _BUILDERS: missing_builders.append(f"{obj.__name__} (config.name={cfg.name!r})") continue - if cfg.name in ("inkling", "plamo3"): - # Inkling uses typed structural blocks. PLaMo markers can span - # several token IDs. Neither fits this TOOL_END-token harness; - # their direct and delegating coverage lives in dedicated tests. + if cfg.name in ("inkling", "plamo3", "minimax_m3"): + # Inkling uses typed structural blocks. PLaMo and MiniMax M3 + # markers can span several token IDs. None fits this TOOL_END-token + # harness; their direct and delegating coverage lives in dedicated + # tests. continue if cfg.name == "granite": # Granite has a JSON-array tool body with no TOOL_END terminal, so diff --git a/tests/parser/engine/trace_builder.py b/tests/parser/engine/trace_builder.py index 3fa6e05078..284ff7048d 100644 --- a/tests/parser/engine/trace_builder.py +++ b/tests/parser/engine/trace_builder.py @@ -42,6 +42,7 @@ from vllm.parser.engine.registered_adapters import ( KimiK2Parser, MiMoParser, MinimaxM2Parser, + MinimaxM3Parser, NemotronV3Parser, Plamo3Parser, Qwen3Parser, @@ -539,6 +540,66 @@ def _build_minimax_m2(scenario: Scenario, validate: bool = True) -> Sample: return sample +# ── MiniMax M3 (namespaced XML elements, no reasoning) ─────────────── + +_MINIMAX_M3_NS = "]<]minimax[>[" + + +def _minimax_m3_value(value: Any) -> str: + """Render a value the way the M3 chat template's ``to_xml`` does.""" + ns = _MINIMAX_M3_NS + if isinstance(value, dict): + return "".join( + f"{ns}<{key}>{_minimax_m3_value(item)}{ns}" + for key, item in value.items() + ) + if isinstance(value, list): + return "".join( + f"{ns}{_minimax_m3_value(item)}{ns}" for item in value + ) + if isinstance(value, bool): + return "true" if value else "false" + return str(value) + + +def _minimax_m3_segments(scenario: Scenario) -> list[tuple[str, bool]]: + ns = _MINIMAX_M3_NS + segs: list[tuple[str, bool]] = [] + if scenario.content: + # The tool parser receives reasoning-stripped content. + segs.append((scenario.content, False)) + if scenario.tool_calls: + invokes = "".join( + f'{ns}' + f"{_minimax_m3_value(tc.arguments)}{ns}\n" + for tc in scenario.tool_calls + ) + segs.append((f"{ns}\n{invokes}{ns}", False)) + return segs + + +def _minimax_m3_expected_content(scenario: Scenario) -> str | None: + if scenario.tool_calls and not (scenario.content or "").strip(): + return None + return scenario.content + + +def _build_minimax_m3(scenario: Scenario, validate: bool = True) -> Sample: + sample = _make_sample( + sample_id=f"minimax_m3-{scenario.id}", + description=scenario.description, + vocab={}, + segments=_minimax_m3_segments(scenario), + expected_reasoning=None, + expected_content=_minimax_m3_expected_content(scenario), + expected_tool_calls=_expected_tc(scenario), + tools=_expected_tools(scenario), + ) + if validate: + _validate_sample(sample, MinimaxM3Parser) + return sample + + # ── Gemma4 (channel reasoning, custom arg format) ──────────────────── _GEMMA4_VOCAB: dict[str, int] = { @@ -1275,6 +1336,7 @@ _BUILDERS: dict[str, Any] = { "gemma4": _build_gemma4, "granite": _build_granite, "minimax_m2": _build_minimax_m2, + "minimax_m3": _build_minimax_m3, "nemotron_v3": _build_nemotron_v3, "granite_thinking_parser": _build_granite_thinking, "seed_oss": _build_seed_oss, diff --git a/tests/parser/mistral/test_tool_calls.py b/tests/parser/mistral/test_tool_calls.py index 25b41f44c2..c17f618ba8 100644 --- a/tests/parser/mistral/test_tool_calls.py +++ b/tests/parser/mistral/test_tool_calls.py @@ -35,6 +35,7 @@ from vllm.entrypoints.generate.base.protocol import ( from vllm.entrypoints.openai.chat_completion.protocol import ( ChatCompletionRequest, ) +from vllm.exceptions import VLLMValidationError from vllm.parser.engine.events import EventType from vllm.parser.engine.streaming_parser_engine import StreamingParserEngine from vllm.parser.mistral import _DEFAULT_JSON_SCHEMA, MistralParser, mistral_config @@ -474,17 +475,22 @@ def test_extract_tool_calls_pre_v11_tokenizer( assert extracted_tool_calls.content == expected_content -def test_extract_tool_calls_pre_v11_multiple_bot_tokens_raises( +def test_extract_tool_calls_pre_v11_multiple_bot_tokens( mistral_pre_v11_tool_parser, ): + """A second BOT token must not fail the request; like streaming, only the + calls after the first BOT token are extracted.""" model_output = ( '[TOOL_CALLS] [{"name": "add", "arguments":{"a": 1}}]' '[TOOL_CALLS] [{"name": "sub", "arguments":{"b": 2}}]' ) - with pytest.raises(ValueError, match="Only one BOT token"): - mistral_pre_v11_tool_parser.extract_tool_calls( - model_output, request=_DUMMY_REQUEST - ) + result = mistral_pre_v11_tool_parser.extract_tool_calls( + model_output, request=_DUMMY_REQUEST + ) + assert result.tools_called + assert [(tc.function.name, tc.function.arguments) for tc in result.tool_calls] == [ + ("add", '{"a": 1}') + ] def test_extract_tool_calls_pre_v11_regex_fallback( @@ -505,15 +511,126 @@ def test_extract_tool_calls_pre_v11_regex_fallback( assert result.tool_calls[0].function.arguments == json.dumps({"a": 1, "b": 2}) -def test_extract_tool_calls_pre_v11_regex_fallback_fails( - mistral_pre_v11_tool_parser, +@pytest.mark.parametrize( + "stringified_tool_calls", + ["not json at all", "42", '["add"]', '{"foo": 1}'], + ids=["invalid_json", "scalar", "list_of_str", "not_a_call"], +) +def test_extract_tool_calls_pre_v11_no_tool_call_returns_content( + mistral_pre_v11_tool_parser, stringified_tool_calls +): + """Output that holds no tool call is returned as content instead of failing + the request.""" + result = mistral_pre_v11_tool_parser.extract_tool_calls( + f"[TOOL_CALLS] {stringified_tool_calls}", request=_DUMMY_REQUEST + ) + assert result == ExtractedToolCallInformation( + tools_called=False, tool_calls=[], content=stringified_tool_calls + ) + + +# (model_output, expected (name, arguments), streamed content). Non-streaming +# drops the text after the calls; streaming returns it as content. +_PRE_V11_UNUSUAL_CALLS = [ + pytest.param( + '[TOOL_CALLS] {"name": "add", "arguments": {"a": 1}}', + [("add", '{"a": 1}')], + "", + id="bare_object", + ), + # The next three end the bare object and start the trailing text in the + # same token, in each state the object can close from. + pytest.param( + '[TOOL_CALLS] {"name": "add", "arguments": {"a": 1}}, ' + '{"name": "sub", "arguments": {"b": 2}}', + [("add", '{"a": 1}')], + ', {"name": "sub", "arguments": {"b": 2}}', + id="bare_object_then_text", + ), + pytest.param( + '[TOOL_CALLS] {"arguments": {"a": 1}, "name": "add"}, ' + '{"name": "sub", "arguments": {"b": 2}}', + [("add", '{"a": 1}')], + ', {"name": "sub", "arguments": {"b": 2}}', + id="bare_object_name_last_then_text", + ), + pytest.param( + '[TOOL_CALLS] {"name": "add", "arguments": "{\\"a\\": 1}"}, ' + '{"name": "sub", "arguments": {"b": 2}}', + [("add", '{"a": 1}')], + ', {"name": "sub", "arguments": {"b": 2}}', + id="bare_object_string_arguments_then_text", + ), + pytest.param( + '[TOOL_CALLS] [{"arguments": {"a": 1}}]', + [("", '{"a": 1}')], + "", + id="missing_name", + ), + pytest.param( + '[TOOL_CALLS] [{"name": "add", "arguments": "{\\"a\\": 1}"}, ' + '{"name": "sub", "arguments": {"b": 2}}]', + [("add", '{"a": 1}'), ("sub", '{"b": 2}')], + "", + id="string_arguments", + ), + pytest.param( + '[TOOL_CALLS] [{"name": "add", "arguments": {"a": 1}}, "junk"]', + [("add", '{"a": 1}')], + "", + id="entry_not_a_call", + ), + # Mistral-7B-Instruct-v0.3 emits repeated BOT tokens at temperature 0. + pytest.param( + '[TOOL_CALLS] [TOOL_CALLS] [{"name": "add", "arguments": {"a": 1}}]', + [("add", '{"a": 1}')], + "", + id="consecutive_bot_tokens", + ), + pytest.param( + '[TOOL_CALLS] [TOOL_CALLS] {"name": "add", "arguments": {"a": 1}}', + [("add", '{"a": 1}')], + "", + id="consecutive_bot_tokens_bare_object", + ), +] + + +@pytest.mark.parametrize("model_output,expected,_", _PRE_V11_UNUSUAL_CALLS) +def test_extract_tool_calls_pre_v11_unusual_calls( + mistral_pre_v11_tool_parser, model_output, expected, _ ): - model_output = "[TOOL_CALLS] not json at all" result = mistral_pre_v11_tool_parser.extract_tool_calls( model_output, request=_DUMMY_REQUEST ) - assert result == ExtractedToolCallInformation( - tools_called=False, tool_calls=[], content="not json at all" + assert result.tools_called + assert [ + (tc.function.name, tc.function.arguments) for tc in result.tool_calls + ] == expected + assert result.content is None + + +@pytest.mark.parametrize( + "model_output,expected,expected_content", _PRE_V11_UNUSUAL_CALLS +) +def test_extract_tool_calls_streaming_pre_v11_unusual_calls( + mistral_pre_v11_tool_parser, + mistral_pre_v11_tokenizer, + model_output, + expected, + expected_content, +): + """Streaming extracts the same calls as the non-streaming path.""" + _test_extract_tool_calls_streaming( + mistral_pre_v11_tool_parser, + mistral_pre_v11_tokenizer, + model_output, + None, + [ + ToolCall(function=FunctionCall(name=name, arguments=arguments)) + for name, arguments in expected + ], + expected_content, ) @@ -881,6 +998,7 @@ def _test_extract_tool_calls_streaming( # if a new tool is being called, set up empty arguments if tool_call.index != tool_call_idx: tool_call_idx = tool_call.index + function_names.append("") function_args_strs.append("") tool_call_ids.append(None) @@ -894,7 +1012,7 @@ def _test_extract_tool_calls_streaming( # IN ENTIRETY, exactly one time. if tool_call.function.name: assert isinstance(tool_call.function.name, str) - function_names.append(tool_call.function.name) + function_names[tool_call.index] = tool_call.function.name if tool_call.function.arguments: # make sure they're a string and then add them to the list @@ -1890,6 +2008,19 @@ def test_adjust_request_structured_outputs_generates_grammar( assert len(result.structured_outputs.grammar) > 0 +def test_adjust_request_deeply_nested_json_string_rejected( + mistral_tool_parser: MistralToolParser, +) -> None: + """The schema string is parsed here, before request validation, and + json.loads raises RecursionError on a string nested this deeply.""" + schema = '{"type": "array", "items": ' * 20_000 + "{}" + "}" * 20_000 + request = _make_request( + structured_outputs=StructuredOutputsParams(json=schema), + ) + with pytest.raises(VLLMValidationError, match="nested too deeply"): + mistral_tool_parser.adjust_request(request) + + @pytest.mark.parametrize( "response_format_kwargs,expected_json_schema", [ diff --git a/tests/parser/test_abstract_parser.py b/tests/parser/test_abstract_parser.py index 56d20417d1..7952ff4114 100644 --- a/tests/parser/test_abstract_parser.py +++ b/tests/parser/test_abstract_parser.py @@ -13,7 +13,8 @@ from vllm.entrypoints.openai.chat_completion.protocol import ( ChatCompletionRequest, ChatCompletionToolsParam, ) -from vllm.parser.abstract_parser import DelegatingParser +from vllm.exceptions import VLLMValidationError +from vllm.parser.abstract_parser import DelegatingParser, structured_outputs_to_format from vllm.sampling_params import StructuredOutputsParams from vllm.tool_parsers.abstract_tool_parser import ToolParser from vllm.tool_parsers.qwen3_engine_tool_parser import Qwen3EngineToolParser @@ -303,3 +304,18 @@ class TestToolChoice_Plus_ResponseFormat: assert _is_grammar_accept_string(grammar, self._qwen_tool_call()) assert not _is_grammar_accept_string(grammar, '{"text": "hi"}') mock_warn.assert_called_once() + + +@pytest.mark.parametrize("field", ["json", "structural_tag"]) +def test_structured_outputs_to_format_rejects_deeply_nested_string(field): + """Parsers convert the constraint before request validation, and json.loads + raises RecursionError on a string nested this deeply.""" + schema = '{"type": "array", "items": ' * 20_000 + "{}" + "}" * 20_000 + value = schema + if field == "structural_tag": + value = ( + '{"type": "structural_tag", "format": {"type": "json_schema", ' + f'"json_schema": {schema}}}}}' + ) + with pytest.raises(VLLMValidationError, match="nested too deeply"): + structured_outputs_to_format(StructuredOutputsParams(**{field: value})) diff --git a/tests/parser/test_streaming.py b/tests/parser/test_streaming.py index cad16746be..d5d4339803 100644 --- a/tests/parser/test_streaming.py +++ b/tests/parser/test_streaming.py @@ -663,6 +663,78 @@ def test_engine_reasoning_no_tool_batched_content_passthrough(tokenizer, request assert len(tool_calls) == 0 +@pytest.mark.parametrize( + "text_chunks", + [ + # A trailing "<" that the engine holds back (it may start a tag). + ["let me thinka <", "x> b"], + # Whitespace after "" in the same delta as the marker. + ["let me think\n\n", "bold done"], + # The delta that ends reasoning carries only held-back text. + ["let me think", "<", "b>bold done"], + ], +) +def test_engine_reasoning_no_tool_batched_holdback_not_dropped( + tokenizer, request_obj, text_chunks +): + """Text the engine buffers when reasoning ends inside a batched delta is + returned by finish_streaming(); with no tool parser it must still be + emitted as content, not dropped.""" + parser = Qwen3ReasoningNoToolParser(tokenizer) + chunks = [tokenizer.encode(t, add_special_tokens=False) for t in text_chunks] + + results = stream_chunks(parser, tokenizer, chunks, request_obj) + reasoning, content, tool_calls = collect_fields(results) + + assert reasoning == "let me think" + assert content == "".join(text_chunks).split("", 1)[1] + assert len(tool_calls) == 0 + + +@pytest.mark.parametrize( + ("text_chunks", "include_reasoning", "expected_content"), + [ + # The delta that ends reasoning is also the last one: the held-back + # "<" must be emitted exactly once (the end-of-stream flush skips the + # reasoning parser once reasoning has ended). + (["let me thinka <"], True, "a <"), + # Whitespace-only content after the marker at the end of the stream. + (["let me think\n\n"], True, "\n\n"), + # Hiding reasoning must not hide the flushed content. + (["let me think\n\n", "bold"], False, "\n\nbold"), + ], +) +def test_engine_reasoning_no_tool_holdback_finished_and_hidden_reasoning( + tokenizer, text_chunks, include_reasoning, expected_content +): + request = ChatCompletionRequest( + model="test-model", + messages=[{"role": "user", "content": "hi"}], + include_reasoning=include_reasoning, + ) + parser = Qwen3ReasoningNoToolParser(tokenizer) + chunks = [tokenizer.encode(t, add_special_tokens=False) for t in text_chunks] + + results: list[DeltaMessage | None] = [] + prompt_token_ids: list[int] | None = [] + for i, chunk in enumerate(chunks): + results.append( + parser.parse_delta( + tokenizer.decode(chunk), + chunk, + request, + prompt_token_ids=prompt_token_ids, + finished=i == len(chunks) - 1, + ) + ) + prompt_token_ids = None + reasoning, content, tool_calls = collect_fields(results) + + assert reasoning == ("let me think" if include_reasoning else "") + assert content == expected_content + assert len(tool_calls) == 0 + + def _decode_stream_deltas(tokenizer, groups): """Decode token-ID groups into ``(delta_text, group)`` pairs via the real incremental ``DecodeStream``. diff --git a/tests/plugins_tests/bitsandbytes/test_bitsandbytes.py b/tests/plugins_tests/bitsandbytes/test_bitsandbytes.py index f5845efb04..92803d9315 100644 --- a/tests/plugins_tests/bitsandbytes/test_bitsandbytes.py +++ b/tests/plugins_tests/bitsandbytes/test_bitsandbytes.py @@ -6,9 +6,7 @@ import types from unittest.mock import MagicMock, patch import pytest -from packaging.version import Version from transformers import BitsAndBytesConfig -from transformers import __version__ as TRANSFORMERS_VERSION from vllm_bnb_plugin import bitsandbytes_loader as bnb from vllm.platforms import current_platform @@ -174,8 +172,7 @@ def test_load_pp_4bit_bnb_model( ) -@pytest.mark.skipif( - Version(TRANSFORMERS_VERSION) >= Version("5.0.0"), +@pytest.mark.skip( reason="Need to add support for quantizing MoE experts with bnb in " "transformers v5. See https://github.com/bitsandbytes-foundation/" "bitsandbytes/issues/1849", diff --git a/tests/quantization/test_auto_gptq.py b/tests/quantization/test_auto_gptq.py index 6e7791654c..4f60a974ca 100644 --- a/tests/quantization/test_auto_gptq.py +++ b/tests/quantization/test_auto_gptq.py @@ -23,7 +23,10 @@ from vllm.model_executor.layers.quantization.auto_gptq import ( AutoGPTQLinearMethod, AutoGPTQMoEMethod, ) -from vllm.model_executor.layers.quantization.utils.gptq_utils import override_config +from vllm.model_executor.layers.quantization.utils.gptq_utils import ( + is_layer_gptq_quantized, + override_config, +) from vllm.platforms import current_platform PROMPT = "On the surface of Mars, we found" @@ -71,6 +74,19 @@ def test_auto_gptq_config_get_name(): assert AutoGPTQConfig.get_name() == "auto_gptq" +def test_auto_gptq_quantizes_every_layer_when_the_module_list_is_unknown(): + """An empty module list means the safetensors metadata could not be read, + not that the checkpoint holds no quantized layer.""" + assert is_layer_gptq_quantized( + prefix="model.layers.0.mlp.down_proj", + quantized_layers=[], + ) + assert not is_layer_gptq_quantized( + prefix="model.layers.0.mlp.down_proj", + quantized_layers=["self_attn.q_proj"], + ) + + @pytest.mark.parametrize( "dynamic", [ @@ -146,6 +162,7 @@ def test_routed_experts_loads_per_expert_biases(): class Loader: quant_config = None quant_method = object() + _fused_shared_expert_quantizer = None moe_config = SimpleNamespace( is_act_and_mul=True, tp_rank=0, diff --git a/tests/quantization/test_blackwell_moe.py b/tests/quantization/test_blackwell_moe.py index da70491bbc..3a4e2b704b 100644 --- a/tests/quantization/test_blackwell_moe.py +++ b/tests/quantization/test_blackwell_moe.py @@ -297,3 +297,16 @@ def test_nemotron_fp4_moe_flashinfer_trtllm(monkeypatch: pytest.MonkeyPatch): hf_overrides=HF_OVERRIDE_TEXT, extra_args=["--moe-backend=flashinfer_trtllm"], ) + + +def test_nemotron_nvfp4_per_token_moe_flashinfer_trtllm( + monkeypatch: pytest.MonkeyPatch, +): + can_initialize( + "nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4", + hf_overrides=HF_OVERRIDE_TEXT, + extra_args=[ + "--moe-backend=flashinfer_trtllm", + "--quantization-config.moe.activation=nvfp4_per_token", + ], + ) diff --git a/tests/quantization/test_config_utils.py b/tests/quantization/test_config_utils.py index 6a0c27748f..a0cb12a9f8 100644 --- a/tests/quantization/test_config_utils.py +++ b/tests/quantization/test_config_utils.py @@ -2,11 +2,10 @@ # SPDX-FileCopyrightText: Copyright contributors to the vLLM project """Tests quantization configuration matching utilities.""" -from unittest.mock import Mock - import pytest from vllm.config.quantization import QuantizationConfigArgs +from vllm.model_executor.layers.linear import LinearBase from vllm.model_executor.layers.quantization.compressed_tensors.utils import ( should_ignore_layer, ) @@ -58,7 +57,7 @@ def test_online_ignore_supports_fnmatch_patterns(): ) with pytest.raises(ValueError, match="matches both quantization_config.ignore"): - config.resolve_quant_method_cls(Mock(), layer_name) + config.resolve_quant_method_cls(LinearBase, layer_name) @pytest.mark.parametrize( diff --git a/tests/quantization/test_fp8.py b/tests/quantization/test_fp8.py index 1cb53f2239..64341843d9 100644 --- a/tests/quantization/test_fp8.py +++ b/tests/quantization/test_fp8.py @@ -318,6 +318,82 @@ def test_deepseek_v41_declines_quark_configs(): ) +_DSV41_NVFP4 = { + "moe_quant_algo": "NVFP4", + "ignore": ["*.attn.*", "*.ffn.shared_experts.*", "head", "mtp.*"], +} +_DSV4_NVFP4_DSPARK = { + "moe_quant_algo": "NVFP4", + "ignore": ["*.attn.*", "*.ffn.shared_experts.*", "head"], +} + + +@pytest.mark.parametrize( + ("ckpt_quant_cfg", "prefix", "expected"), + [ + pytest.param(_DSV41_NVFP4, "model.layers.1.ffn.experts", "nvfp4", id="target"), + pytest.param( + _DSV41_NVFP4, + "language_model.model.layers.0.ffn.experts", + "nvfp4", + id="target-vl-prefix", + ), + pytest.param( + _DSV41_NVFP4, "model.layers.40.ffn.experts", "mxfp4", id="ignored-mtp0" + ), + pytest.param( + _DSV41_NVFP4, "model.layers.42.ffn.experts", "mxfp4", id="ignored-mtp2" + ), + pytest.param( + _DSV4_NVFP4_DSPARK, + "model.layers.41.ffn.experts", + "nvfp4", + id="quantized-mtp1", + ), + pytest.param( + {"moe_quant_algo": "NVFP4"}, + "model.layers.40.ffn.experts", + "nvfp4", + id="no-ignore-list", + ), + pytest.param({}, "model.layers.1.ffn.experts", "mxfp4", id="not-nvfp4"), + ], +) +def test_deepseek_v41_nvfp4_export_keeps_ignored_experts_mxfp4( + monkeypatch, ckpt_quant_cfg, prefix, expected +): + """NVFP4 exports keep the routed experts they ``ignore`` in MXFP4, e.g. the + bundled DSpark draft (``mtp.*``, built as ``layers.{40 + k}``) of + DeepSeek-V4.1-Flash-NVFP4. The ignore list is read from the checkpoint + passed to ``from_config``: the current hf_config below carries none.""" + import vllm.model_executor.layers.quantization.modelopt as modelopt + import vllm.models.deepseek_v41.quant_config as dsv41_qc + from vllm.model_executor.layers.fused_moe import RoutedExperts + + hf_config = SimpleNamespace( + expert_dtype="fp4", + quantization_config={"moe_quant_algo": ckpt_quant_cfg.get("moe_quant_algo")}, + ) + model_config = SimpleNamespace( + hf_config=hf_config, hf_text_config=SimpleNamespace(num_hidden_layers=40) + ) + monkeypatch.setattr( + dsv41_qc, + "get_current_vllm_config", + lambda: SimpleNamespace(model_config=model_config), + ) + monkeypatch.setattr(dsv41_qc, "Mxfp4MoEMethod", lambda moe_config: "mxfp4") + monkeypatch.setattr(modelopt, "ModelOptNvFp4FusedMoE", lambda **kwargs: "nvfp4") + + config = dsv41_qc.DeepseekV4FP8Config.from_config( + {"quant_method": "fp8", "activation_scheme": "dynamic", **ckpt_quant_cfg} + ) + monkeypatch.setattr(config, "_get_nvfp4_config", lambda: None) + layer = RoutedExperts.__new__(RoutedExperts) + layer.moe_config = None + assert config.get_quant_method(layer, prefix) == expected + + @pytest.mark.skipif(not current_platform.is_cuda(), reason="DeepGEMM requires CUDA") @pytest.mark.parametrize("scale_dtype", [torch.uint8, torch.float8_e8m0fnu]) @pytest.mark.parametrize( diff --git a/tests/quantization/test_fp8_per_channel.py b/tests/quantization/test_fp8_per_channel.py index 2c3fddcb0f..d4f2fe264e 100644 --- a/tests/quantization/test_fp8_per_channel.py +++ b/tests/quantization/test_fp8_per_channel.py @@ -61,7 +61,9 @@ def test_scaled_fp8_quant_per_channel_shape() -> None: weight to `ops.scaled_fp8_quant` with `use_per_token_if_dynamic=True` yields one scale per output row -- a [N, 1] fp32 tensor. """ - x = (torch.randn(size=(96, 256), device="cuda") * 13).to(torch.bfloat16) + x = (torch.randn(size=(96, 256), device=current_platform.device_type) * 13).to( + torch.bfloat16 + ) y, s = ops.scaled_fp8_quant(x, scale=None, use_per_token_if_dynamic=True) assert y.shape == (96, 256) assert y.dtype == current_platform.fp8_dtype() diff --git a/tests/quantization/test_modelopt.py b/tests/quantization/test_modelopt.py index eea4b6218d..33c798cba0 100644 --- a/tests/quantization/test_modelopt.py +++ b/tests/quantization/test_modelopt.py @@ -6,6 +6,7 @@ Run `pytest tests/quantization/test_modelopt.py`. """ import os +from contextlib import nullcontext from typing import Any, NoReturn from unittest.mock import MagicMock, Mock, patch @@ -17,7 +18,9 @@ from tests.quantization.utils import ( load_model_without_vllm_runner, ) from vllm.config import VllmConfig, set_current_vllm_config +from vllm.config.load import LoadConfig from vllm.config.model import ModelConfig +from vllm.config.quantization import QuantizationConfigArgs from vllm.forward_context import set_forward_context from vllm.model_executor.kernels.linear import ( FlashInferCuteDslNvFp4W4A16LinearKernel, @@ -40,12 +43,14 @@ from vllm.model_executor.layers.quantization.utils.quant_utils import ( kMxfp8Dynamic, kMxfp8Static, kNvfp4Dynamic, + kNvfp4DynamicToken, kNvfp4Static, ) from vllm.model_executor.layers.vocab_parallel_embedding import ( ParallelLMHead, VocabParallelEmbedding, ) +from vllm.model_executor.model_loader.weight_utils import get_quant_config from vllm.platforms import current_platform @@ -113,11 +118,13 @@ def _mixed_precision_config(quantized_layers: dict) -> ModelOptMixedPrecisionCon ) -def test_modelopt_nvfp4_quantizes_parallel_lm_head(): +@pytest.mark.parametrize("moe_activation", [None, "nvfp4_per_token"]) +def test_modelopt_nvfp4_quantizes_parallel_lm_head(moe_activation): config = ModelOptNvFp4Config( is_checkpoint_nvfp4_serialized=True, kv_cache_quant_algo=None, exclude_modules=[], + quantization_args=QuantizationConfigArgs(moe={"activation": moe_activation}), ) method = config.get_quant_method(_mock_lm_head(), prefix="lm_head") @@ -688,7 +695,8 @@ def test_modelopt_linear_method_builder_registry_override(monkeypatch): @pytest.mark.parametrize( ("linear_backend", "kernel_cls"), [ - ("auto", MarlinNvFp4LinearKernel), + ("auto", None), + ("marlin", MarlinNvFp4LinearKernel), ("humming", HummingNvFp4LinearKernel), ("flashinfer_cutedsl", FlashInferCuteDslNvFp4W4A16LinearKernel), ], @@ -696,7 +704,7 @@ def test_modelopt_linear_method_builder_registry_override(monkeypatch): @pytest.mark.skipif(not current_platform.is_cuda(), reason="CUDA only") def test_modelopt_w4a16_respects_linear_backend(linear_backend, kernel_cls): """W4A16 (`activation=None`) kernel selection honors ``--linear-backend``: - ``use_a16=True`` defaults to Marlin, but an explicit backend wins. The + ``use_a16=True`` follows platform priorities, but an explicit backend wins. The generic method routes this through ``select_linear_kernel``.""" from vllm.config.quantization import QuantSpec from vllm.model_executor.layers.quantization.modelopt import ( @@ -704,7 +712,20 @@ def test_modelopt_w4a16_respects_linear_backend(linear_backend, kernel_cls): select_linear_kernel, ) - if linear_backend != "auto": + if linear_backend == "auto": + capability = current_platform.get_device_capability() + assert capability is not None + cc = capability.to_int() + if ( + cc in (100, 103) + and FlashInferCuteDslNvFp4W4A16LinearKernel.is_supported()[0] + ): + kernel_cls = FlashInferCuteDslNvFp4W4A16LinearKernel + elif cc == 90 and HummingNvFp4LinearKernel.is_supported()[0]: + kernel_cls = HummingNvFp4LinearKernel + else: + kernel_cls = MarlinNvFp4LinearKernel + else: is_supported, reason = kernel_cls.is_supported() if not is_supported: pytest.skip(reason) @@ -776,6 +797,74 @@ def test_modelopt_nvfp4_moe_dispatches_to_marlin_when_w4a16( assert kwargs["activation_key"] is kNvfp4Dynamic +@pytest.mark.parametrize("quantization", ["modelopt_fp4", "modelopt_mixed"]) +@pytest.mark.parametrize("per_token", [False, True]) +def test_modelopt_nvfp4_moe_activation_override(quantization, per_token): + """Runtime overrides reach NVFP4 configs without changing HF metadata.""" + hf_config = { + "quant_method": "modelopt", + "quant_algo": "MIXED_PRECISION" + if quantization == "modelopt_mixed" + else "NVFP4", + "quantized_layers": {"experts": {"quant_algo": "NVFP4"}}, + } + args = ( + QuantizationConfigArgs(moe={"activation": "nvfp4_per_token"}) + if per_token + else None + ) + config = get_quant_config( + Mock( + quantization=quantization, + quantization_config=args, + hf_config=Mock(quantization_config=hf_config), + ), + LoadConfig(), + ) + if isinstance(config, ModelOptMixedPrecisionConfig): + config = config.nvfp4_config + assert isinstance(config, ModelOptNvFp4Config) + assert config.moe_activation_override == (kNvfp4DynamicToken if per_token else None) + assert "_online_quantization_args" not in hf_config + + +@pytest.mark.parametrize( + "quant_method,activation,backend,error", + [ + ("NVFP4", kNvfp4DynamicToken, "FLASHINFER_TRTLLM", None), + ("W4A16_NVFP4", kNvfp4DynamicToken, "FLASHINFER_TRTLLM", "W4A4 checkpoint"), + ( + "NVFP4", + kMxfp8Dynamic, + "FLASHINFER_TRTLLM", + "Unsupported.*activation override", + ), + ("NVFP4", kNvfp4DynamicToken, "MARLIN", "requires the FlashInfer TRTLLM"), + ], +) +def test_modelopt_nvfp4_moe_validates_activation_override( + quant_method, activation, backend, error +): + from vllm.model_executor.layers.fused_moe.oracle.nvfp4 import NvFp4MoeBackend + from vllm.model_executor.layers.quantization.modelopt import ModelOptNvFp4FusedMoE + + config = ModelOptNvFp4Config( + quant_method=quant_method, + quantization_args=QuantizationConfigArgs(moe={"activation": activation}), + ) + expected = pytest.raises(ValueError, match=error) if error else nullcontext() + with ( + patch( + "vllm.model_executor.layers.quantization.modelopt.select_nvfp4_moe_backend", + return_value=(NvFp4MoeBackend[backend], Mock()), + ) as select_backend, + expected, + ): + method = ModelOptNvFp4FusedMoE(config, Mock()) + assert method.per_token_activation + assert select_backend.call_args.kwargs["activation_key"] == activation + + @pytest.mark.parametrize( "per_layer_algo, expected_weight, expected_activation", [ diff --git a/tests/quantization/test_online.py b/tests/quantization/test_online.py index b4dffbb375..ce13aa9a65 100644 --- a/tests/quantization/test_online.py +++ b/tests/quantization/test_online.py @@ -34,6 +34,7 @@ from vllm.forward_context import set_forward_context from vllm.model_executor.kernels.linear.mxfp8.emulation import ( EmulationMxfp8LinearKernel, ) +from vllm.model_executor.kernels.linear.mxfp8.humming import HummingMxfp8LinearKernel from vllm.model_executor.kernels.linear.mxfp8.marlin import ( MarlinMxfp8LinearKernel, ) @@ -58,6 +59,7 @@ from vllm.model_executor.layers.quantization.compressed_tensors.compressed_tenso ) from vllm.model_executor.layers.quantization.modelopt import ( ModelOptFp8Config, + ModelOptLinearMethod, ModelOptMxFp8Config, ) from vllm.model_executor.layers.quantization.online.base import ( @@ -93,12 +95,16 @@ from vllm.model_executor.layers.quantization.quark.quark import ( ) from vllm.model_executor.layers.quantization.utils import quant_utils from vllm.model_executor.layers.quantization.utils.mxfp8_utils import ( + MXFP8_SCALE_DTYPE, MXFP8_VALUE_DTYPE, ) from vllm.model_executor.layers.quantization.utils.quant_utils import ( amax_for_moe_weight_quant, amax_for_tp_weight_quant, kMxfp8Dynamic, + kMxfp8Static, + kNvfp4DynamicToken, + kNvfp4Static, weight_amax, ) from vllm.model_executor.layers.vocab_parallel_embedding import ( @@ -112,6 +118,7 @@ from vllm.model_executor.models.granitemoe import ( ) from vllm.platforms import current_platform from vllm.utils.flashinfer import has_flashinfer_trtllm_fused_moe +from vllm.utils.torch_utils import set_default_torch_dtype if current_platform.is_rocm(): from vllm.platforms.rocm import on_gfx942, on_gfx950 @@ -188,7 +195,6 @@ def test_qianfan_online_fp8_keeps_vision_layers_unquantized( config, ) - layer = Mock(spec=LinearBase) for prefix in ( "vision_model.encoder.layers.0.attn.qkv", "vision_model.encoder.layers.1.mlp.fc2", @@ -196,25 +202,36 @@ def test_qianfan_online_fp8_keeps_vision_layers_unquantized( "mlp1.1", "mlp1.3", ): - assert config.resolve_quant_method_cls(layer, prefix) is None + assert config.resolve_quant_method_cls(LinearBase, prefix) is None resolved = config.resolve_quant_method_cls( - layer, "language_model.model.layers.0.self_attn.qkv_proj" + LinearBase, "language_model.model.layers.0.self_attn.qkv_proj" ) assert resolved is not None and resolved[-1] is method_cls assert other_config.ignored_layers == [] assert args.ignore == [] +@pytest.mark.parametrize( + ("is_act_and_mul", "expected_hidden_alignment"), + [(True, 256), (False, 512)], +) def test_online_nvfp4_reuses_kernel_when_weights_are_reprocessed( - monkeypatch, + monkeypatch, is_act_and_mul, expected_hidden_alignment ) -> None: - method = object.__new__(Nvfp4OnlineMoEMethod) - method.moe = SimpleNamespace(is_act_and_mul=True) - method.nvfp4_backend = object() - method.experts_cls = object - method.moe_quant_config = None - method.moe_kernel = None + monkeypatch.setattr(current_platform, "is_device_capability_family", lambda _: True) + select_backend = Mock(return_value=(object(), object)) + monkeypatch.setattr( + "vllm.model_executor.layers.quantization.online.nvfp4.select_nvfp4_moe_backend", + select_backend, + ) + moe_config = SimpleNamespace(is_act_and_mul=is_act_and_mul) + method = Nvfp4OnlineMoEMethod(moe=moe_config) + select_backend.assert_called_once_with( + config=moe_config, + weight_key=kNvfp4Static, + activation_key=kNvfp4DynamicToken, + ) layer = Mock() converted_weights = [ @@ -269,7 +286,12 @@ def test_online_nvfp4_reuses_kernel_when_weights_are_reprocessed( assert method.moe_kernel is kernel assert convert_weights.call_count == 3 + assert all( + call.kwargs["trtllm_hidden_alignment"] == expected_hidden_alignment + for call in convert_weights.call_args_list + ) make_kernel.assert_called_once() + assert make_kernel.call_args.kwargs["per_token_activation"] is True get_quant_config.assert_called_once() assert process_weights.call_count == 3 @@ -307,6 +329,14 @@ def _fully_quantized_modelopt_config() -> ModelOptFp8Config: ) +def _fully_quantized_modelopt_mxfp8_config() -> ModelOptMxFp8Config: + return ModelOptMxFp8Config( + is_checkpoint_mxfp8_serialized=True, + kv_cache_quant_algo=None, + exclude_modules=["lm_head"], + ) + + def _moe_only_compressed_tensors_config() -> CompressedTensorsConfig: return CompressedTensorsConfig( target_scheme_map={"RoutedExperts": {}}, @@ -334,10 +364,19 @@ def _write_minimal_llama_config( @pytest.mark.parametrize( - "checkpoint_config_factory,raises_conflict", + "checkpoint_config_factory,raises_requantization_error", [ pytest.param(_fully_quantized_quark_config, True, id="quark"), - pytest.param(_fully_quantized_modelopt_config, True, id="modelopt"), + pytest.param( + _fully_quantized_modelopt_config, + True, + id="modelopt", + ), + pytest.param( + _fully_quantized_modelopt_mxfp8_config, + False, + id="modelopt_mxfp8", + ), pytest.param( _moe_only_compressed_tensors_config, False, @@ -345,9 +384,13 @@ def _write_minimal_llama_config( ), ], ) +@pytest.mark.skipif( + not (current_platform.is_cuda() or current_platform.is_rocm()), + reason="MXFP8 requantization requires CUDA or ROCm.", +) def test_online_prequantized_compatibility( checkpoint_config_factory, - raises_conflict: bool, + raises_requantization_error: bool, default_vllm_config, dist_init, ) -> None: @@ -370,35 +413,69 @@ def test_online_prequantized_compatibility( "disable_tp": True, } - if raises_conflict: - with pytest.raises(ValueError, match="pre-quantized layer"): - ColumnParallelLinear(**layer_kwargs) + layer = ColumnParallelLinear(**layer_kwargs) + assert isinstance(layer.quant_method, Mxfp8OnlineLinearMethod) + + if layer.weight.is_meta: + # Weight loading. + layer = layer.to_empty(device=DEVICE) + for parameter in layer.parameters(): + parameter.zero_() else: - layer = ColumnParallelLinear(**layer_kwargs) + layer = layer.to(device=DEVICE) + + if raises_requantization_error: + with pytest.raises( + NotImplementedError, + match="does not implement dequantize_weight|only supported for MXFP8", + ): + layer.quant_method.process_weights_after_loading(layer) + else: + layer.quant_method.process_weights_after_loading(layer) + + if isinstance(checkpoint_config, ModelOptMxFp8Config): assert isinstance(layer.quant_method, Mxfp8OnlineLinearMethod) + assert not layer.quant_method.uses_meta_device + assert isinstance( + layer.quant_method.requantization_source, ModelOptLinearMethod + ) + elif not raises_requantization_error: + assert layer.quant_method.uses_meta_device + assert layer.quant_method.requantization_source is None -def test_online_target_rejects_prequantized_layer( - default_vllm_config, dist_init +@pytest.mark.skipif( + not (current_platform.is_cuda() or current_platform.is_rocm()) + or not is_quant_method_supported("fp8"), + reason="Requires FP8 support on CUDA or ROCm.", +) +def test_online_block_fp8_requantization_removes_source_scales( + default_vllm_config, dist_init, workspace_init ) -> None: - """A targets match participates in checkpoint compatibility checks.""" - default_vllm_config.model_config = ModelConfig() - prefix = "model.layers.0.self_attn.o_proj" - quant_config = _fully_quantized_quark_config() + """Blockwise FP8 replaces MXFP8 scales with its own scale parameter.""" + default_vllm_config.model_config = ModelConfig(dtype="bfloat16") + quant_config = _fully_quantized_modelopt_mxfp8_config() quant_config.online_quantization_config = OnlineQuantizationConfig( - QuantizationConfigArgs(targets={prefix: "mxfp8"}) + QuantizationConfigArgs(linear="fp8_per_block") ) - - with pytest.raises(ValueError, match="pre-quantized layer"): - ColumnParallelLinear( - input_size=32, - output_size=32, + with set_default_torch_dtype(torch.bfloat16), torch.device(DEVICE): + layer = ColumnParallelLinear( + input_size=256, + output_size=256, bias=False, params_dtype=torch.bfloat16, quant_config=quant_config, - prefix=prefix, + prefix="model.layers.0.self_attn.o_proj", disable_tp=True, ) + with torch.no_grad(): + layer.weight.fill_(1) + layer.weight_scale.fill_(127) # E8M0 encoding of scale 1. + + layer.quant_method.process_weights_after_loading(layer) + + assert not hasattr(layer, "weight_scale") + assert layer.weight_scale_inv.numel() > 0 def test_online_ignore_keeps_checkpoint_quantization_linear( @@ -431,17 +508,25 @@ def test_online_ignore_keeps_checkpoint_quantization_linear( def test_online_quantization_rejects_prequantized_moe( - default_vllm_config, dist_init + default_vllm_config, dist_init, monkeypatch ) -> None: - """Online linear and MoE quantization reject a pre-quantized MoE layer.""" + """Reject pre-quantized MoE before constructing an online target backend.""" default_vllm_config.model_config = ModelConfig() prefix = "model.layers.0.mlp.experts" quant_config = _fully_quantized_quark_config() quant_config.online_quantization_config = OnlineQuantizationConfig( quant_config_args(linear="mxfp4", moe="mxfp4") ) + monkeypatch.setattr( + quant_config.online_quantization_config, + "get_quant_method", + lambda *args: pytest.fail("unsupported online backend must not be constructed"), + ) - with pytest.raises(ValueError, match="pre-quantized layer"): + with pytest.raises( + NotImplementedError, + match="Requantizing checkpoint-quantized MoE layers is not supported", + ): FusedMoEFactory( num_experts=4, top_k=2, @@ -591,7 +676,7 @@ def test_log_online_quantization_for_composable_config(monkeypatch) -> None: assert log_args == [ ( - "Quantized %d layers of types: %s", + "Quantizing %d layers of types: %s", 2, "; ".join(online_config.quantized_layer_summaries), ) @@ -836,6 +921,24 @@ def test_nvfp4_one_sided_rejects_unsupported_input_dtype(input_dtype: torch.dtyp {}, id="partially_prequantized_checkpoint", ), + pytest.param( + PARTIALLY_PREQUANTIZED_MODEL_NAME, + None, + {"targets": {"model.layers.1.self_attn.o_proj": "mxfp8"}}, + Mxfp8OnlineLinearMethod, + CompressedTensorsMoEMethod, + {}, + id="partially_prequantized_checkpoint", + ), + pytest.param( + "mgoin/Qwen3-0.6B-MXFP8", + None, + {"linear": "fp8_per_channel"}, + Fp8PtpcOnlineLinearMethod, + None, + {}, + id="requantization_mxfp8_ptcp_fp8", + ), ], ) @pytest.mark.parametrize( @@ -888,6 +991,25 @@ def test_online_quantization( moe_backend=runner_kwargs.get("moe_backend", "auto"), ), } + if model_name == "mgoin/Qwen3-0.6B-MXFP8": + original_process = Fp8PtpcOnlineLinearMethod.process_weights_after_loading + + def assert_source_weights_released(method, layer) -> None: + source_parameters = dict(method.requantization_source_parameters) + assert set(source_parameters) == {"weight", "weight_scale"} + + original_process(method, layer) + + assert not method.requantization_source_parameters + assert layer.weight is not source_parameters["weight"] + assert layer.weight_scale is not source_parameters["weight_scale"] + assert layer.weight_scale.shape[-1] == 1 + + monkeypatch.setattr( + Fp8PtpcOnlineLinearMethod, + "process_weights_after_loading", + assert_source_weights_released, + ) if model_name == PARTIALLY_PREQUANTIZED_MODEL_NAME: model, vllm_config = load_model_without_vllm_runner( @@ -943,14 +1065,20 @@ def test_online_quantization( if model_name == PARTIALLY_PREQUANTIZED_MODEL_NAME: o_proj = model.model.layers[1].self_attn.o_proj moe = model.model.layers[0].mlp.experts + elif model_name == "mgoin/Qwen3-0.6B-MXFP8": + o_proj = model.model.layers[0].self_attn.o_proj + moe = None else: o_proj = model.model.layers[0].self_attn.o_proj moe = model.model.layers[0].block_sparse_moe.experts + assert isinstance(o_proj.quant_method, expected_linear_cls) - assert isinstance(moe._quant_method, expected_moe_cls) + + if moe is not None: + assert isinstance(moe._quant_method, expected_moe_cls) if model_name == PARTIALLY_PREQUANTIZED_MODEL_NAME and isinstance( - o_proj.quant_method.kernel, MarlinMxfp8LinearKernel + o_proj.quant_method.kernel, (MarlinMxfp8LinearKernel, HummingMxfp8LinearKernel) ): assert o_proj.weight.dtype == torch.int32 elif model_name == PARTIALLY_PREQUANTIZED_MODEL_NAME and isinstance( @@ -1191,9 +1319,7 @@ def test_online_quantization_targets_ignore_collision() -> None: ) ) with pytest.raises(ValueError, match="matches both quantization_config.ignore"): - config.resolve_quant_method_cls( - Mock(spec=LinearBase), "model.layers.0.self_attn.o_proj" - ) + config.resolve_quant_method_cls(LinearBase, "model.layers.0.self_attn.o_proj") def test_online_quantization_targets_reject_unsupported_layer() -> None: @@ -1242,7 +1368,7 @@ def test_log_online_quantization(default_vllm_config, monkeypatch) -> None: log_online_quantization(default_vllm_config) assert logged_messages == [ - "Quantized 3 layers of types: mlp.down_proj: 2 (from linear: " + "Quantizing 3 layers of types: mlp.down_proj: 2 (from linear: " "fp8_per_tensor); self_attn.qkv_proj: 1 (from targets: " "re:.*qkv_proj.*, mxfp4)" ] @@ -1365,6 +1491,48 @@ def test_online_linear_tp_weight_quant_matches_unsharded( assert torch.equal(tp_scale, full_scale) +@pytest.mark.skipif( + not is_quant_method_supported("fp8"), + reason="FP8 is not supported on this GPU type.", +) +def test_modelopt_mxfp8_requantized_ptpc_tp_shards_match_unsharded() -> None: + """MXFP8 source shards convert to slices of the unsharded PTPC weight.""" + torch.manual_seed(0) + serialized_weight = torch.randn(16, 64, device=DEVICE, dtype=torch.bfloat16).to( + MXFP8_VALUE_DTYPE + ) + serialized_scale = torch.randint( + 124, + 131, + (16, 2), + device=DEVICE, + dtype=MXFP8_SCALE_DTYPE, + ) + + source = ModelOptLinearMethod.__new__(ModelOptLinearMethod) + source.wkey = SimpleNamespace(key=kMxfp8Static) + + def dequantize(weight: torch.Tensor, scale: torch.Tensor) -> torch.Tensor: + layer = SimpleNamespace(weight=weight, weight_scale=scale) + return source.dequantize_weight(layer) + + full_weight = dequantize(serialized_weight, serialized_scale) + full_scale = _fp8_channel_scale(weight_amax(full_weight, dim=-1, keepdim=True)) + full_qweight = _fp8_quant_per_channel(full_weight, full_scale) + + # Mimic the two K-dimension shards of a row-parallel layer. PTPC's global + # per-output-channel scale is the same on both ranks after its MAX reduce. + for rank in range(2): + start = rank * 32 + shard_weight = serialized_weight[:, start : start + 32].contiguous() + shard_scale = serialized_scale[:, rank : rank + 1].contiguous() + weight = dequantize(shard_weight, shard_scale) + qweight = _fp8_quant_per_channel(weight, full_scale) + + assert torch.equal(weight, full_weight[:, start : start + 32]) + assert torch.equal(qweight, full_qweight[:, start : start + 32]) + + def _quantize_moe(weight, scheme, moe_tp_size): if scheme == "nvfp4": return _quantize_moe_weight_to_nvfp4(weight, moe_tp_size) diff --git a/tests/quantization/test_online_shared_expert_fusion.py b/tests/quantization/test_online_shared_expert_fusion.py new file mode 100644 index 0000000000..2dc3d5da76 --- /dev/null +++ b/tests/quantization/test_online_shared_expert_fusion.py @@ -0,0 +1,637 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Tests ROCm AITER fused shared experts with online quantization.""" + +import json +from pathlib import Path +from types import SimpleNamespace +from typing import Any + +import pytest +import torch + +from tests.quantization.utils import load_model_without_vllm_runner +from vllm._aiter_ops import rocm_aiter_ops +from vllm.config.model import ModelConfig +from vllm.config.quantization import ( + QuantizationConfigArgs, + resolve_quantization_config, +) +from vllm.model_executor.layers.fused_moe import FusedMoEFactory +from vllm.model_executor.layers.fused_moe.experts import rocm_aiter_moe +from vllm.model_executor.layers.fused_moe.oracle.mxfp4 import Mxfp4MoeBackend +from vllm.model_executor.layers.quantization.online.base import OnlineQuantizationConfig +from vllm.model_executor.layers.quantization.quark.quark import QuarkConfig +from vllm.model_executor.layers.quantization.quark.quark_moe import ( + QuarkOCP_MX_MoEMethod, +) +from vllm.model_executor.layers.quantization.utils.mxfp4_utils import ( + mxfp4_quantize, +) +from vllm.model_executor.model_loader.dummy_loader import DummyModelLoader +from vllm.model_executor.model_loader.reload.layerwise import ( + finalize_layerwise_reload, + initialize_layerwise_reload, + record_metadata_for_reloading, +) +from vllm.model_executor.model_loader.utils import get_model_architecture +from vllm.platforms import current_platform + +pytestmark = pytest.mark.skipif( + not current_platform.is_rocm(), + reason="ROCm AITER shared-expert tests require ROCm.", +) + +_QUARK_MXFP4_CONFIG: dict[str, Any] = { + "global_quant_config": { + "input_tensors": { + "dtype": "fp4", + "is_dynamic": True, + "qscheme": "per_group", + "ch_axis": -1, + "group_size": 32, + "block_size": None, + "symmetric": None, + "round_method": "half_even", + "scale_type": "float", + "scale_format": "e8m0", + "scale_calculation_mode": "even", + "mx_element_dtype": None, + "observer_cls": "PerBlockMXObserver", + "is_scale_quant": False, + "enable_buffer_reuse": False, + "max_input_numel": 4194304, + }, + "output_tensors": None, + "weight": { + "dtype": "fp4", + "is_dynamic": False, + "qscheme": "per_group", + "ch_axis": -1, + "group_size": 32, + "block_size": None, + "symmetric": None, + "round_method": "half_even", + "scale_type": "float", + "scale_format": "e8m0", + "scale_calculation_mode": "even", + "mx_element_dtype": None, + "observer_cls": "PerBlockMXObserver", + "is_scale_quant": False, + "enable_buffer_reuse": False, + "max_input_numel": 4194304, + }, + "bias": None, + "target_device": None, + }, + "algo_config": None, + "softmax_quant_spec": None, + "quant_method": "quark", + "layer_type_quant_config": {}, + "layer_quant_config": {}, + "exclude": ["re:^(?!.*\\.mlp\\.experts(?:\\.|$)).*$"], + "kv_cache_quant_config": {}, + "kv_cache_post_rope": False, + "quant_mode": "eager_mode", + "version": "0.12+9d3d471cdf1", + "export": { + "kv_cache_group": [], + "min_kv_scale": 0.0, + "pack_method": "reorder", + "weight_format": "real_quantized", + "weight_merge_groups": None, + }, +} + +_QUARK_MXFP4_LAYER_CONFIG: dict[str, Any] = _QUARK_MXFP4_CONFIG["global_quant_config"] +_QUARK_MXFP4_CONFIG = { + **_QUARK_MXFP4_CONFIG, + "global_quant_config": { + "input_tensors": {}, + "output_tensors": None, + "weight": {}, + "bias": None, + "target_device": None, + }, + "layer_quant_config": {"*mlp.experts*": _QUARK_MXFP4_LAYER_CONFIG}, +} + + +class _StubAttention(torch.nn.Module): + def __init__(self, *args: Any, **kwargs: Any) -> None: + super().__init__() + self.o_proj = SimpleNamespace(reduce_results=True) + + +@pytest.fixture(autouse=True) +def reset_aiter_shared_expert_topk_metadata(monkeypatch: pytest.MonkeyPatch): + """Isolate AITER's process-global shared-expert routing buffer.""" + monkeypatch.setattr(rocm_aiter_moe, "aiter_topK_meta_data", None) + rocm_aiter_moe.init_aiter_topK_meta_data.cache_clear() + yield + rocm_aiter_moe.init_aiter_topK_meta_data.cache_clear() + + +def _write_minimal_moe_config(model_path: Path, architecture: str) -> None: + """Write a compact MoE config with one shared expert.""" + config = { + "architectures": [architecture], + "hidden_size": 256, + "intermediate_size": 512, + "moe_intermediate_size": 256, + "num_attention_heads": 4, + "num_key_value_heads": 4, + "head_dim": 64, + "num_hidden_layers": 1, + "vocab_size": 256, + "max_position_embeddings": 64, + "q_lora_rank": 64, + "kv_lora_rank": 512, + "qk_nope_head_dim": 128, + "qk_rope_head_dim": 64, + "v_head_dim": 128, + "n_routed_experts": 4, + "num_experts": 4, + "num_experts_per_tok": 1, + "n_group": 1, + "topk_group": 1, + "n_shared_experts": 1, + "first_k_dense_replace": 0, + "moe_layer_freq": 1, + "rms_norm_eps": 1e-6, + "hidden_act": "silu", + "tie_word_embeddings": False, + "rope_theta": 10000, + "quantization_config": _QUARK_MXFP4_CONFIG, + } + if architecture == "AXK1ForCausalLM": + config["model_type"] = "axk1" + elif architecture == "MiniMaxM3SparseForCausalLM": + text_config = { + **config, + "model_type": "minimax_m3_text", + "dense_intermediate_size": 512, + "hidden_act": "swigluoai", + "num_local_experts": 4, + "n_shared_experts": 1, + "moe_layer_freq": [1], + "sparse_attention_config": None, + "quantization_config": { + **_QUARK_MXFP4_CONFIG, + "layer_quant_config": { + "*block_sparse_moe.experts*": _QUARK_MXFP4_LAYER_CONFIG + }, + "exclude": [r"re:^(?!.*\.block_sparse_moe\.experts(?:\.|$)).*$"], + }, + } + config = { + "architectures": [architecture], + "model_type": "minimax_m3_vl", + "text_config": text_config, + } + elif architecture == "Glm4MoeForCausalLM": + config["model_type"] = "glm4_moe" + elif architecture == "Glm4MoeLiteForCausalLM": + config["model_type"] = "glm4_moe_lite" + elif architecture in ("Qwen3NextForCausalLM", "Qwen3_5MoeForCausalLM"): + config.update( + model_type=( + "qwen3_next" + if architecture == "Qwen3NextForCausalLM" + else "qwen3_5_moe_text" + ), + layer_types=[ + "linear_attention" + if architecture == "Qwen3NextForCausalLM" + else "full_attention" + ], + head_dim=64, + linear_key_head_dim=64, + linear_value_head_dim=64, + linear_num_key_heads=4, + linear_num_value_heads=4, + shared_expert_intermediate_size=256, + ) + elif architecture == "Qwen4ExpForCausalLM": + config.update( + model_type="qwen4_exp_text", + layer_types=["full_attention"], + head_dim=64, + linear_key_head_dim=64, + linear_value_head_dim=64, + linear_num_key_heads=4, + linear_num_value_heads=4, + shared_expert_intermediate_size=256, + eos_token_id=1, + hc_count=2, + hc_lowrank=4, + ple_layer_ids=[], + ) + elif architecture == "Glm5NextForCausalLM": + config.update( + model_type="glm5_next", + layer_types=["linear_attention"], + mlp_layer_types=["sparse"], + qk_rope_head_dim=0, + index_topk=1, + index_kpool=1, + pad_token_id=None, + scoring_func="sigmoid", + topk_method="noaux_tc", + mhc=False, + ) + else: + config["model_type"] = ( + "deepseek_v3" + if "V3" in architecture + else "deepseek_v32" + if architecture == "DeepseekV32ForCausalLM" + else "deepseek_v2" + ) + if architecture in ("DeepseekV32ForCausalLM", "GlmMoeDsaForCausalLM"): + config.update( + index_topk=1, + index_kpool=1, + index_n_heads=1, + index_head_dim=32, + index_kv_lora_rank=32, + ) + (model_path / "config.json").write_text(json.dumps(config)) + + +@pytest.mark.parametrize( + "load_path", ["load_weights", "fused_gate_up", "weight_loader"] +) +def test_online_shared_expert_loads_bf16_weights_into_mxfp4_slot( + default_vllm_config, + dist_init, + load_path: str, +) -> None: + """A BF16 shared expert is quantized while routed MXFP4 weights are loaded. + + `weight_loader` covers DeepSeek-style model loaders, which bypass + `RoutedExperts.load_weights`. + """ + default_vllm_config.model_config = ModelConfig() + hidden_size = intermediate_size = 64 + num_routed_experts = 2 + device = current_platform.device_type + online_config = OnlineQuantizationConfig( + QuantizationConfigArgs(targets={"*shared_expert*": "mxfp4"}) + ) + quant_config = QuarkConfig(_QUARK_MXFP4_CONFIG) + quant_config.online_quantization_config = online_config + + with torch.device(device): + runner = FusedMoEFactory( + num_experts=num_routed_experts, + top_k=1, + hidden_size=hidden_size, + intermediate_size=intermediate_size, + n_shared_experts=1, + fuse_shared_experts=True, + shared_expert_prefix="model.layers.0.mlp.shared_expert", + prefix="model.layers.0.mlp.experts", + quant_config=quant_config, + ) + layer = runner.routed_experts + assert isinstance(layer.quant_method, QuarkOCP_MX_MoEMethod) + + routed_gate, _ = mxfp4_quantize( + torch.randn(intermediate_size, hidden_size, dtype=torch.bfloat16) + ) + routed_up, _ = mxfp4_quantize( + torch.randn(intermediate_size, hidden_size, dtype=torch.bfloat16) + ) + routed_down, _ = mxfp4_quantize( + torch.randn(hidden_size, intermediate_size, dtype=torch.bfloat16) + ) + shared_gate = torch.randn(intermediate_size, hidden_size, dtype=torch.bfloat16) + shared_up = torch.randn(intermediate_size, hidden_size, dtype=torch.bfloat16) + shared_down = torch.randn(hidden_size, intermediate_size, dtype=torch.bfloat16) + + weights = [ + ("0.gate_proj.weight", routed_gate), + ("0.up_proj.weight", routed_up), + ("0.down_proj.weight", routed_down), + ] + if load_path == "load_weights": + weights += [ + ("2.gate_proj.weight", shared_gate), + ("2.up_proj.weight", shared_up), + ("2.down_proj.weight", shared_down), + ] + elif load_path == "fused_gate_up": + weights += [ + ("2.gate_up_proj.weight", torch.cat([shared_gate, shared_up])), + ("2.down_proj.weight", shared_down), + ] + list(layer.load_weights(weights)) + + if load_path == "weight_loader": + for stem, shard_id, weight in ( + ("w13", "w1", shared_gate), + ("w13", "w3", shared_up), + ("w2", "w2", shared_down), + ): + param = getattr(layer, f"{stem}_weight") + param.weight_loader( + param, weight, f"{layer.layer_name}.{stem}_weight", shard_id, 2 + ) + + expected_gate, expected_gate_scale = mxfp4_quantize(shared_gate) + expected_up, expected_up_scale = mxfp4_quantize(shared_up) + expected_down, expected_down_scale = mxfp4_quantize(shared_down) + assert torch.equal(layer.w13_weight[2, :intermediate_size], expected_gate) + assert torch.equal(layer.w13_weight[2, intermediate_size:], expected_up) + assert torch.equal(layer.w2_weight[2], expected_down) + assert torch.equal( + layer.w13_weight_scale[2, :intermediate_size], expected_gate_scale + ) + assert torch.equal(layer.w13_weight_scale[2, intermediate_size:], expected_up_scale) + assert torch.equal(layer.w2_weight_scale[2], expected_down_scale) + + +def test_online_shared_expert_reload_compatibility( + default_vllm_config, + dist_init, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Reload a BF16 shared weight into its MXFP4 slot via meta staging.""" + default_vllm_config.model_config = ModelConfig() + online_config = OnlineQuantizationConfig( + QuantizationConfigArgs(targets={"*shared_expert*": "mxfp4"}) + ) + quant_config = QuarkConfig(_QUARK_MXFP4_CONFIG) + quant_config.online_quantization_config = online_config + + with torch.device(current_platform.device_type): + layer = FusedMoEFactory( + num_experts=2, + top_k=1, + hidden_size=64, + intermediate_size=64, + n_shared_experts=1, + fuse_shared_experts=True, + shared_expert_prefix="model.layers.0.mlp.shared_expert", + prefix="model.layers.0.mlp.experts", + quant_config=quant_config, + ).routed_experts + monkeypatch.setattr( + layer.quant_method, "process_weights_after_loading", lambda _: None + ) + + routed_weights = [] + for expert_id in range(2): + for projection in ("gate_proj", "up_proj", "down_proj"): + weight, scale = mxfp4_quantize( + torch.randn(64, 64, dtype=torch.bfloat16) + ) + prefix = f"{expert_id}.{projection}" + routed_weights.extend( + [(f"{prefix}.weight", weight), (f"{prefix}.weight_scale", scale)] + ) + + shared_gate = torch.randn(64, 64, dtype=torch.bfloat16) + shared_up = torch.randn(64, 64, dtype=torch.bfloat16) + shared_down = torch.randn(64, 64, dtype=torch.bfloat16) + + def checkpoint_weights(gate: torch.Tensor): + return [ + *routed_weights, + ("2.gate_proj.weight", gate), + ("2.up_proj.weight", shared_up), + ("2.down_proj.weight", shared_down), + ] + + list(layer.load_weights(checkpoint_weights(shared_gate))) + old_weight = layer.w13_weight[2, :64].clone() + reloaded_gate = shared_gate + 1 + expected_weight, expected_scale = mxfp4_quantize(reloaded_gate) + assert not torch.equal(old_weight, expected_weight) + + record_metadata_for_reloading(layer) + initialize_layerwise_reload(layer) + + assert layer.w13_weight.is_meta + list(layer.load_weights(checkpoint_weights(reloaded_gate))) + finalize_layerwise_reload(layer, default_vllm_config.model_config) + assert not layer.w13_weight.is_meta + assert torch.equal(layer.w13_weight[2, :64], expected_weight) + assert torch.equal(layer.w13_weight_scale[2, :64], expected_scale) + + +# TODO: Add DeepseekV4ForConditionalGeneration once it uses +# resolve_layer_fused_shared_expert. +@pytest.mark.parametrize( + "architecture", + [ + "AXK1ForCausalLM", + "DeepseekForCausalLM", + "DeepseekV2ForCausalLM", + "DeepseekV32ForCausalLM", + "DeepseekV3ForCausalLM", + "Glm4MoeForCausalLM", + "Glm4MoeLiteForCausalLM", + "Glm5NextForCausalLM", + "GlmMoeDsaForCausalLM", + "MiniMaxM3SparseForCausalLM", + "Qwen3NextForCausalLM", + "Qwen3_5MoeForCausalLM", + "Qwen4ExpForCausalLM", + ], +) +def test_online_quantization( + architecture: str, + tmp_path: Path, + monkeypatch, + dist_init, + workspace_init, +) -> None: + """Online MXFP4 fuses each architecture's shared expert into its MoE.""" + monkeypatch.setenv("VLLM_ROCM_USE_AITER", "1") + monkeypatch.setenv("VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS", "1") + rocm_aiter_ops.refresh_env_variables() + if architecture == "Glm5NextForCausalLM": + from vllm.models.glm5next.common import model as glm5_next_model + + for attention in ("Glm5NextLinearAttention", "Glm5NextMLAAttention"): + monkeypatch.setattr(glm5_next_model, attention, _StubAttention) + monkeypatch.setattr( + glm5_next_model, "_fused_shared_experts_tuned", lambda _: True + ) + elif architecture == "Glm4MoeLiteForCausalLM": + from vllm.model_executor.models import glm4_moe_lite + + monkeypatch.setattr(glm4_moe_lite, "Glm4MoeLiteAttention", _StubAttention) + monkeypatch.setattr(glm4_moe_lite, "Glm4MoeLiteMLAAttention", _StubAttention) + elif architecture == "DeepseekV32ForCausalLM": + from vllm.models.deepseek_v32.amd import model as deepseek_v32_model + + monkeypatch.setattr( + deepseek_v32_model, "DeepseekV32MLAAttention", _StubAttention + ) + elif architecture == "Qwen3_5MoeForCausalLM": + from vllm.model_executor.models import qwen3_5 + + monkeypatch.setattr(qwen3_5, "Qwen3NextAttention", _StubAttention) + elif architecture == "Qwen4ExpForCausalLM": + from vllm.models.qwen4_exp.amd import model as qwen4_exp_model + + monkeypatch.setattr(qwen4_exp_model, "Qwen3NextAttention", _StubAttention) + _write_minimal_moe_config(tmp_path, architecture) + expected_shared_expert_name = ( + "shared_expert" if architecture.startswith("Qwen") else "shared_experts" + ) + target_pattern = f"*{expected_shared_expert_name}*" + + logged_messages: list[str] = [] + logged_warnings: list[str] = [] + + def record_info(message: str, *args: object, **_kwargs: object) -> None: + if "Quantizing " in message and "of types" in message: + logged_messages.append(message % args) + + def record_warning(message: str, *args: object, **_kwargs: object) -> None: + logged_warnings.append(message % args) + + monkeypatch.setattr( + "vllm.model_executor.model_loader.base_loader.logger.info", record_info + ) + monkeypatch.setattr( + "vllm.model_executor.layers.fused_moe.utils.logger.warning", record_warning + ) + + model, vllm_config = load_model_without_vllm_runner( + str(tmp_path), + dtype="bfloat16", + quantization="quark", + model_config_kwargs={ + "quantization_config": resolve_quantization_config( + "quark", {"targets": {target_pattern: "mxfp4"}} + ) + }, + model_loader_cls=DummyModelLoader, + ) + expected_model_cls, resolved_architecture = get_model_architecture( + vllm_config.model_config + ) + assert resolved_architecture == architecture + assert isinstance(model, expected_model_cls) + + fused_moes = [ + module + for module in model.modules() + if getattr(module, "is_fused_shared_expert_enabled", False) + and hasattr(module, "experts") + ] + assert len(fused_moes) == 1, f"{architecture} did not enable FSE" + moe = fused_moes[0] + shared_expert_name = ( + "shared_expert" if hasattr(moe, "shared_expert") else "shared_experts" + ) + assert shared_expert_name == expected_shared_expert_name + assert getattr(moe, shared_expert_name) is None + + routed_experts = moe.experts.routed_experts + assert isinstance(routed_experts.quant_method, QuarkOCP_MX_MoEMethod) + shared_expert_prefix = routed_experts.moe_config.shared_expert_prefix + assert shared_expert_prefix is not None + assert shared_expert_prefix.endswith(shared_expert_name) + assert routed_experts._fused_shared_expert_quantizer is not None + expert_map_manager = routed_experts.expert_map_manager + assert expert_map_manager.num_fused_shared_experts == 1 + assert expert_map_manager.map_global_to_local( + routed_experts.moe_config.num_experts + ) == (routed_experts.moe_config.num_experts) + expected_weight_dtype = { + Mxfp4MoeBackend.AITER_MXFP4_BF16: torch.float4_e2m1fn_x2, + Mxfp4MoeBackend.AITER_MXFP4_MXFP4: torch.float4_e2m1fn_x2, + Mxfp4MoeBackend.EMULATION: torch.uint8, + }[routed_experts.quant_method.mxfp4_backend] + assert routed_experts.w13_weight.dtype == expected_weight_dtype + assert routed_experts.w13_weight_scale.dtype == torch.uint8 + assert routed_experts.w2_weight.dtype == expected_weight_dtype + assert routed_experts.w2_weight_scale.dtype == torch.uint8 + assert vllm_config.quant_config is not None + assert vllm_config.quant_config.online_quantization_config is not None + assert not any( + "VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS is enabled but " + "cannot be enabled" in warning + for warning in logged_warnings + ) + if architecture == "MiniMaxM3SparseForCausalLM": + # TODO: Use resolve_layer_fused_shared_expert in MiniMax M3 and remove + # this branch once its virtual fused projections are registered. + assert logged_messages == ["Quantizing 0 layers of types: "] + else: + assert logged_messages == [ + "Quantizing 2 layers of types: " + f"mlp.{shared_expert_name}.down_proj: 1 " + f"(from targets: {target_pattern}, mxfp4); " + f"mlp.{shared_expert_name}.gate_up_proj: 1 " + f"(from targets: {target_pattern}, mxfp4)" + ] + + +def test_online_gate_up_proj_target_disables_fse( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, + dist_init, + workspace_init, +) -> None: + """A partial shared-expert target quantizes while disabling FSE.""" + architecture = "DeepseekForCausalLM" + target_pattern = "*shared_experts.gate_up_proj*" + monkeypatch.setenv("VLLM_ROCM_USE_AITER", "1") + monkeypatch.setenv("VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS", "1") + rocm_aiter_ops.refresh_env_variables() + _write_minimal_moe_config(tmp_path, architecture) + + logged_messages: list[str] = [] + logged_warnings: list[str] = [] + + def record_info(message: str, *args: object, **_kwargs: object) -> None: + if "Quantizing " in message and "of types" in message: + logged_messages.append(message % args) + + def record_warning(message: str, *args: object, **_kwargs: object) -> None: + logged_warnings.append(message % args) + + monkeypatch.setattr( + "vllm.model_executor.model_loader.base_loader.logger.info", record_info + ) + monkeypatch.setattr( + "vllm.model_executor.layers.fused_moe.utils.logger.warning", record_warning + ) + + model, _ = load_model_without_vllm_runner( + str(tmp_path), + dtype="bfloat16", + quantization="quark", + model_config_kwargs={ + "quantization_config": resolve_quantization_config( + "quark", {"targets": {target_pattern: "mxfp4"}} + ) + }, + model_loader_cls=DummyModelLoader, + ) + + moe = next( + module + for module in model.modules() + if hasattr(module, "is_fused_shared_expert_enabled") + and hasattr(module, "shared_experts") + ) + assert not moe.is_fused_shared_expert_enabled + assert moe.shared_experts is not None + assert logged_messages == [ + "Quantizing 1 layers of types: " + "mlp.shared_experts.gate_up_proj: 1 " + f"(from targets: {target_pattern}, mxfp4)" + ] + assert logged_warnings == [ + "VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS is enabled but cannot be " + "enabled - skipping for this layer: online quantization targets only " + "part of the shared expert at model.layers.0.mlp.shared_experts." + ] diff --git a/tests/quantization/test_quantization_config_args.py b/tests/quantization/test_quantization_config_args.py index 1d6ced0d48..f24566f5fa 100644 --- a/tests/quantization/test_quantization_config_args.py +++ b/tests/quantization/test_quantization_config_args.py @@ -2,8 +2,6 @@ # SPDX-FileCopyrightText: Copyright contributors to the vLLM project """Unit tests for QuantizationConfigArgs parsing.""" -from unittest.mock import Mock - import pytest from tests.quantization.utils import quant_config_args, quant_spec @@ -26,6 +24,8 @@ from vllm.model_executor.layers.quantization.utils.quant_utils import ( kInt4Static32, kInt8StaticChannelSym, kMxfp8Dynamic, + kNvfp4DynamicToken, + kNvfp4Static, ) # ---- QuantSpec ------------------------------------------------------------ @@ -68,13 +68,17 @@ def test_args_linear_string_resolves_via_quant_key_names(): assert args.moe is None -def test_args_moe_string_resolves_via_online_shorthand(): +@pytest.mark.parametrize( + "shorthand,weight", + [("fp8_per_block", kFp8Static128BlockSym), ("nvfp4_per_token", kNvfp4Static)], +) +def test_args_moe_string_resolves_via_online_shorthand(shorthand, weight): # An online-shorthand name pulls the matching slot from _ONLINE_SHORTHANDS # (so `linear: "fp8_per_block"` and `moe: "fp8_per_block"` produce the # same per-layer-kind spec the `--quantization fp8_per_block` shorthand # would). - args = quant_config_args(moe="fp8_per_block") - assert args.moe == QuantSpec(weight=kFp8Static128BlockSym) + args = quant_config_args(moe=shorthand) + assert args.moe == QuantSpec(weight=weight) def test_args_string_shorthand_missing_slot_raises(): @@ -84,9 +88,14 @@ def test_args_string_shorthand_missing_slot_raises(): quant_config_args(linear="int8_per_channel_weight_only") -def test_args_accepts_dict_form(): - args = quant_config_args(moe={"activation": "mxfp8"}) - assert args.moe == QuantSpec(weight=None, activation=kMxfp8Dynamic) +@pytest.mark.parametrize( + "activation,expected", + [("mxfp8", kMxfp8Dynamic), ("nvfp4_per_token", kNvfp4DynamicToken)], +) +def test_args_accepts_dict_form(activation, expected): + args = quant_config_args(moe={"activation": activation}) + assert args.moe == QuantSpec(weight=None, activation=expected) + assert args.linear is None def test_targets_reject_non_string_keys(): @@ -110,22 +119,34 @@ def test_resolve_colliding_shorthand_is_deferred(quantization: str): assert resolve_quantization_config(quantization, None) is None -def test_resolve_int8_shorthand_leaves_linear_unset(): - # int8_per_channel_weight_only is MoE-only; linear stays None so that - # OnlineQuantizationConfig leaves Linear layers in full precision. - args = resolve_quantization_config("int8_per_channel_weight_only", None) +@pytest.mark.parametrize( + "shorthand,weight", + [ + ("int8_per_channel_weight_only", kInt8StaticChannelSym), + ("nvfp4_per_token", kNvfp4Static), + ], +) +def test_resolve_moe_only_shorthand_leaves_linear_unset(shorthand, weight): + args = resolve_quantization_config(shorthand, None) assert args is not None assert args.linear is None - assert args.moe == QuantSpec(weight=kInt8StaticChannelSym) + assert args.moe == QuantSpec(weight=weight) -def test_resolve_quantization_config_only(): +@pytest.mark.parametrize("quantization", [None, "modelopt_fp4"]) +@pytest.mark.parametrize( + "activation,expected", + [("mxfp8", kMxfp8Dynamic), ("nvfp4_per_token", kNvfp4DynamicToken)], +) +def test_resolve_activation_override(quantization, activation, expected): # When only `quantization_config` is given (e.g. for an already-quantized # checkpoint that needs an activation override), it's returned as-is. - args = resolve_quantization_config(None, {"moe": {"activation": "mxfp8"}}) + args = resolve_quantization_config( + quantization, {"moe": {"activation": activation}} + ) assert args is not None assert args.linear is None - assert args.moe == QuantSpec(weight=None, activation=kMxfp8Dynamic) + assert args.moe == QuantSpec(weight=None, activation=expected) def test_resolve_merges_explicit_over_shorthand(): @@ -221,6 +242,4 @@ def test_targets_reject_moe_only_shorthand_for_linear_layer(): ) with pytest.raises(ValueError, match="does not define a QuantSpec"): - config.resolve_quant_method_cls( - Mock(spec=LinearBase), "model.layers.0.self_attn.o_proj" - ) + config.resolve_quant_method_cls(LinearBase, "model.layers.0.self_attn.o_proj") diff --git a/tests/quantization/utils.py b/tests/quantization/utils.py index ed9776ad1c..d8446952f0 100644 --- a/tests/quantization/utils.py +++ b/tests/quantization/utils.py @@ -15,7 +15,11 @@ from vllm.config import ( VllmConfig, set_current_vllm_config, ) -from vllm.config.quantization import QuantizationConfigArgs, QuantSpec +from vllm.config.quantization import ( + QuantizationConfigArgs, + QuantSpec, + resolve_quantization_config, +) from vllm.model_executor.layers.quantization import get_quantization_config from vllm.model_executor.model_loader.default_loader import DefaultModelLoader from vllm.platforms import current_platform @@ -40,10 +44,17 @@ def _limit_num_hidden_layers( original_load_weights = model.load_weights def should_load_weight(name: str) -> bool: - for prefix in ("model.layers.", "layers."): - if name.startswith(prefix): - layer_idx = int(name.removeprefix(prefix).split(".", 1)[0]) - return layer_idx < num_hidden_layers + layers_prefix = "layers." + if name.startswith(layers_prefix): + layer_name = name.removeprefix(layers_prefix) + elif ".layers." in name: + layer_name = name.split(".layers.", maxsplit=1)[1] + else: + return True + + layer_idx_str = layer_name.split(".", maxsplit=1)[0] + if layer_idx_str.isdigit(): + return int(layer_idx_str) < num_hidden_layers return True def load_weights(weights): @@ -71,13 +82,19 @@ def load_model_without_vllm_runner( quantization=quantization, **(model_config_kwargs or {}), ) + model_config.quantization_config = resolve_quantization_config( + model_config.quantization, model_config.quantization_config + ) vllm_config_args = dict(vllm_config_kwargs or {}) vllm_config_args.setdefault( "compilation_config", CompilationConfig(mode=CompilationMode.NONE) ) vllm_config = VllmConfig(model_config=model_config, **vllm_config_args) hf_overrides = (model_config_kwargs or {}).get("hf_overrides") or {} - num_hidden_layers = hf_overrides.get("num_hidden_layers") + text_config_overrides = hf_overrides.get("text_config") or {} + num_hidden_layers = text_config_overrides.get( + "num_hidden_layers", hf_overrides.get("num_hidden_layers") + ) with set_current_vllm_config(vllm_config): model_loader = model_loader_cls(vllm_config.load_config) diff --git a/tests/samplers/test_beam_search_online.py b/tests/samplers/test_beam_search_online.py index cbed58363a..22afce2d78 100644 --- a/tests/samplers/test_beam_search_online.py +++ b/tests/samplers/test_beam_search_online.py @@ -1,13 +1,33 @@ # SPDX-License-Identifier: Apache-2.0 # SPDX-FileCopyrightText: Copyright contributors to the vLLM project -import pytest +from collections.abc import Callable, Sequence +from typing import Any, cast +from unittest.mock import MagicMock -from vllm import CompletionOutput, RequestOutput +import pytest +import torch + +from vllm import CompletionOutput, PromptType, RequestOutput, SamplingParams +from vllm import logger as vllm_logger +from vllm.config import VllmConfig +from vllm.entrypoints.generate.beam_search.offline import BeamSearchOfflineMixin from vllm.entrypoints.generate.beam_search.online import BeamSearchOnlineMixin -from vllm.inputs import TokensInput +from vllm.entrypoints.generate.beam_search.utils import ( + BeamSearchInstance, + BeamSearchSequence, +) +from vllm.inputs import EngineInput, TokensInput from vllm.logprobs import Logprob, SampleLogprobs from vllm.sampling_params import BeamSearchParams +from vllm.v1.structured_output.backend_types import StructuredOutputBackend + + +@pytest.fixture +def reset_warning_once(): + vllm_logger._print_warning_once.cache_clear() + yield + vllm_logger._print_warning_once.cache_clear() class _Tokenizer: @@ -27,8 +47,26 @@ class _Renderer: return _Tokenizer() +class _VllmConfig: + watermark_config = object() + speculative_config = None + + def _check_supports_watermarking(self, params: BeamSearchParams) -> bool: + return VllmConfig._check_supports_watermarking(cast(VllmConfig, self), params) + + +class _InputProcessor: + vllm_config = _VllmConfig() + + def resolve_watermarking(self, params): + return self.vllm_config._check_supports_watermarking(params) + + class _EngineClient: + input_processor = _InputProcessor() + async def generate(self, prompt, *args, **kwargs): + assert args[0].watermarking is False yield RequestOutput( request_id=kwargs.get("request_id", "test-request"), prompt=prompt.get("prompt"), @@ -56,11 +94,42 @@ class _EngineClient: ) -class _Serving(BeamSearchOnlineMixin): +class _AsyncServing(BeamSearchOnlineMixin): renderer = _Renderer() engine_client = _EngineClient() +class _OfflineServing(BeamSearchOfflineMixin): + renderer = _Renderer() + llm_engine = _EngineClient() + + def _preprocess_cmpl( + self, + prompts: Sequence[PromptType], + tokenization_kwargs: dict[str, Any] | None = None, + mm_processor_kwargs: dict[str, Any] | None = None, + ) -> Sequence[EngineInput]: + return cast(Sequence[EngineInput], prompts) + + def _lora_request_to_seq(self, lora_request, num_requests): + return [None] * num_requests + + def _beam_search_step( + self, + instances_batch: list[BeamSearchInstance], + base_sampling_params: SamplingParams, + eos_token_id: int | None, + ignore_eos: bool, + beam_width: int, + sort_beams_key: Callable[..., Any], + structured_output_backend: StructuredOutputBackend | None, + structured_output_key: tuple[Any, ...] | None, + structured_output_bitmask: torch.Tensor | None, + ) -> bool: + assert base_sampling_params.watermarking is False + return True + + @pytest.mark.asyncio async def test_beam_search_handles_extra_logprob_candidates() -> None: prompt: TokensInput = { @@ -68,10 +137,11 @@ async def test_beam_search_handles_extra_logprob_candidates() -> None: "prompt": "prompt", "prompt_token_ids": [1], } - params = BeamSearchParams(beam_width=2, max_tokens=1) + params = BeamSearchParams(beam_width=2, max_tokens=1, watermarking=False) outputs = [ - output async for output in _Serving().beam_search(prompt, "request", params) + output + async for output in _AsyncServing().beam_search(prompt, "request", params) ] assert len(outputs) == 1 @@ -101,10 +171,12 @@ async def test_beam_search_respects_skip_special_tokens( max_tokens=1, ignore_eos=True, skip_special_tokens=skip_special_tokens, + watermarking=False, ) outputs = [ - output async for output in _Serving().beam_search(prompt, "request", params) + output + async for output in _AsyncServing().beam_search(prompt, "request", params) ] assert outputs[0].outputs[0].text == expected_text @@ -139,7 +211,7 @@ async def test_beam_search_abort_returns_partial_outputs( output.finish_reason = "abort" yield result - serving = _Serving() + serving = _AsyncServing() monkeypatch.setattr(serving.engine_client, "generate", generate) prompt: TokensInput = { "type": "token", @@ -149,7 +221,9 @@ async def test_beam_search_abort_returns_partial_outputs( outputs = [ output async for output in serving.beam_search( - prompt, "request", BeamSearchParams(beam_width=2, max_tokens=4) + prompt, + "request", + BeamSearchParams(beam_width=2, max_tokens=4, watermarking=False), ) ] @@ -165,3 +239,86 @@ async def test_beam_search_abort_returns_partial_outputs( assert output.logprobs is not None assert len(output.logprobs) == abort_after assert outputs[0].outputs[0].cumulative_logprob == pytest.approx(-0.1 * abort_after) + + +@pytest.mark.asyncio +async def test_beam_search_warns_and_disables_watermarking( + caplog_vllm, reset_warning_once +) -> None: + prompt = { + "type": "token", + "prompt": "prompt", + "prompt_token_ids": [1], + } + params = BeamSearchParams(beam_width=2, max_tokens=1) + + with caplog_vllm.at_level("WARNING"): + output = await anext(_AsyncServing().beam_search(prompt, "request", params)) + + assert "beam search requests will run without watermarking" in caplog_vllm.text + # The caller's params object is never rewritten by admission. + assert params.watermarking is None + assert output.finished + + +def test_offline_beam_search_warns_and_disables_watermarking( + caplog_vllm, reset_warning_once +) -> None: + params = BeamSearchParams(beam_width=2, max_tokens=1) + + with caplog_vllm.at_level("WARNING"): + outputs = _OfflineServing().beam_search( + cast(list[PromptType], [{"type": "token", "prompt_token_ids": [1]}]), + params, + ) + + assert "beam search requests will run without watermarking" in caplog_vllm.text + # The caller's params object is never rewritten by admission. + assert params.watermarking is None + assert len(outputs) == 1 + + +def test_offline_beam_search_disables_internal_watermarking() -> None: + outputs = _OfflineServing().beam_search( + cast(list[PromptType], [{"type": "token", "prompt_token_ids": [1]}]), + BeamSearchParams(beam_width=2, max_tokens=1, watermarking=False), + ) + + assert len(outputs) == 1 + + +def test_offline_structured_beam_search_disables_internal_watermarking() -> None: + grammar = MagicMock() + grammar.is_terminated.return_value = False + grammar.fill_bitmask.side_effect = lambda bitmask, index: bitmask[index].fill_(1) + backend = MagicMock() + backend.compile_grammar.return_value = grammar + serving = _OfflineServing() + serving.model_config = MagicMock() + serving.model_config.get_vocab_size.return_value = 32 + beam = BeamSearchSequence( + orig_prompt={"type": "token", "prompt_token_ids": [1]}, + tokens=[1], + logprobs=[], + ) + + base_params = SamplingParams( + logprobs=2, + watermarking=False, + skip_clone=True, + ) + entries = serving._build_beam_sampling_params( + [beam], + base_params, + backend, + ("regex", ".*"), + torch.zeros((1, 1), dtype=torch.int32), + ) + + assert entries[0] is not None + beam_params = entries[0][0] + assert beam_params is not base_params + assert beam_params.logprobs == base_params.logprobs + assert beam_params.watermarking is False + assert beam_params.allowed_token_ids == [0] + assert base_params.allowed_token_ids is None diff --git a/tests/standalone_tests/test_precompiled_wheel.py b/tests/standalone_tests/test_precompiled_wheel.py new file mode 100644 index 0000000000..883a6cea84 --- /dev/null +++ b/tests/standalone_tests/test_precompiled_wheel.py @@ -0,0 +1,195 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +"""Exercise setup-time wheel selection without importing torch or running setup.""" + +import ast +import logging +import os +import subprocess +import tempfile +import unittest +from email.message import Message +from pathlib import Path +from types import SimpleNamespace +from typing import Any +from unittest.mock import patch +from urllib.error import HTTPError, URLError + +# setup.py executes setup() on import; load only the wheel utility class. +setup_path = Path(__file__).resolve().parents[2] / "setup.py" +node = next( + node + for node in ast.parse(setup_path.read_text()).body + if isinstance(node, ast.ClassDef) and node.name == "precompiled_wheel_utils" +) +namespace: dict[str, Any] = { + "subprocess": subprocess, + "os": os, + "logger": logging.getLogger("setup"), + "envs": SimpleNamespace(VLLM_DOCKER_BUILD_CONTEXT=False, VLLM_USE_PRECOMPILED=True), +} +exec( + compile(ast.Module(body=[node], type_ignores=[]), str(setup_path), "exec"), + namespace, +) +wheel_utils = namespace["precompiled_wheel_utils"] +WHEEL = { + "package_name": "vllm", + "platform_tag": "manylinux_x86_64", + "path": "wheel.whl", + "filename": "wheel.whl", +} +NOT_FOUND = HTTPError("url", 404, "missing", Message(), None) + + +class TestPublishedWheelSearch(unittest.TestCase): + """Walk back from the merge-base in a real Git repo; only the index is mocked.""" + + def setUp(self): + tmp = tempfile.TemporaryDirectory() + self.addCleanup(tmp.cleanup) + self.repo = Path(tmp.name) + self.addCleanup(os.chdir, os.getcwd()) + os.chdir(self.repo) + self.git("init", "-q") + self.published = {self.commit("vllm/a.py")} + + def git(self, *args: str) -> str: + identity = ["-c", "user.name=t", "-c", "user.email=t@t"] + return subprocess.check_output(["git", *identity, *args], text=True).strip() + + def commit(self, path: str) -> str: + file = self.repo / path + file.parent.mkdir(parents=True, exist_ok=True) + file.write_text(file.read_text() + "x" if file.exists() else "x") + self.git("add", "-A") + self.git("commit", "-q", "-m", path) + return self.git("rev-parse", "HEAD") + + def search(self, base: str, arch: str = "x86_64") -> str: + def fetch(commit, variant): + if commit not in self.published: + raise NOT_FOUND + return [WHEEL], "url" + + with patch.object(wheel_utils, "fetch_metadata_for_variant", fetch): + return wheel_utils.find_published_wheel_commit(base, "cu130", arch) + + def test_published_base_is_used_despite_local_native_edits(self): + base = self.commit("csrc/ops.cpp") + self.published.add(base) + self.commit("csrc/ops.cpp") + (self.repo / "pyproject.toml").write_text("dirty") + self.assertEqual(self.search(base), base) + + def test_publication_lag_uses_nearest_published_ancestor(self): + published = self.commit("vllm/b.py") + self.published.add(published) + self.commit("vllm/c.py") + self.assertEqual(self.search(self.commit("vllm/d.py")), published) + + def test_compiled_changes_since_wheel_are_rejected(self): + (published,) = self.published + for path in wheel_utils.WHEEL_BLOCKING_PATHS: + with self.subTest(path=path): + self.git("reset", "-q", "--hard", published) + file = path if "." in path else f"{path}/file" + with self.assertRaisesRegex(ValueError, file): + self.search(self.commit(file)) + + def test_rust_changes_since_wheel_only_warn(self): + (published,) = self.published + base = self.commit("rust/src/main.rs") + with self.assertLogs("setup", "WARNING") as logs: + self.assertEqual(self.search(base), published) + self.assertIn("rust/src/main.rs", logs.output[0]) + + def test_wrong_architecture_is_not_a_match(self): + with self.assertRaisesRegex(ValueError, "No published precompiled wheel"): + self.search("HEAD", arch="aarch64") + + def test_search_stops_at_depth_limit(self): + with patch.object(wheel_utils, "WHEEL_SEARCH_DEPTH", 2): + self.commit("vllm/b.py") + with self.assertRaisesRegex(ValueError, "within 2 commits"): + self.search(self.commit("vllm/c.py")) + + def test_network_and_server_errors_do_not_trigger_walkback(self): + for error in ( + URLError("offline"), + HTTPError("url", 503, "busy", Message(), None), + ): + with ( + self.subTest(error=error), + patch.object( + wheel_utils, "fetch_metadata_for_variant", side_effect=error + ) as fetch, + self.assertRaises(type(error)), + ): + wheel_utils.find_published_wheel_commit("HEAD", "cu130", "x86_64") + fetch.assert_called_once() + + +class TestDetermineWheelUrl(unittest.TestCase): + def determine(self, base: str, **env: str): + with ( + patch.dict( + os.environ, + {"VLLM_PRECOMPILED_WHEEL_VARIANT": "cu130", **env}, + clear=True, + ), + patch("platform.machine", return_value="x86_64"), + patch.object(wheel_utils, "is_rocm_system", return_value=False), + patch.object( + wheel_utils, "get_base_commit_in_main_branch", return_value=base + ), + patch.object( + wheel_utils, + "fetch_metadata_for_variant", + return_value=([WHEEL], "https://wheels.test/"), + ) as fetch, + patch.object( + wheel_utils, "find_published_wheel_commit", return_value="older" + ) as search, + ): + url = wheel_utils.determine_wheel_url() + return url, fetch, search + + def test_automatic_install_downloads_the_searched_commit(self): + url, fetch, search = self.determine("base") + self.assertEqual(url, ("https://wheels.test/wheel.whl", "wheel.whl")) + search.assert_called_once_with("base", "cu130", "x86_64") + fetch.assert_called_once_with("older", "cu130") + + def test_search_is_skipped_when_it_cannot_apply(self): + cases = { + "nightly fallback": ("nightly", {}, {}), + "explicit commit": ( + "base", + {"VLLM_PRECOMPILED_WHEEL_COMMIT": "a" * 40}, + {}, + ), + "rust only": ("base", {}, {"VLLM_USE_PRECOMPILED": False}), + "docker": ("base", {}, {"VLLM_DOCKER_BUILD_CONTEXT": True}), + } + for name, (base, env, flags) in cases.items(): + with self.subTest(name), patch.dict(vars(namespace["envs"]), flags): + _, fetch, search = self.determine(base, **env) + search.assert_not_called() + commit = env.get("VLLM_PRECOMPILED_WHEEL_COMMIT", base) + fetch.assert_called_once_with(commit, "cu130") + + def test_explicit_wheel_location_bypasses_selection(self): + for location in ("/tmp/local.whl", "https://example.com/custom.whl"): + with self.subTest(location=location): + url, fetch, search = self.determine( + "base", VLLM_PRECOMPILED_WHEEL_LOCATION=location + ) + self.assertEqual(url, (location, None)) + fetch.assert_not_called() + search.assert_not_called() + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_attention_backend_registry.py b/tests/test_attention_backend_registry.py index 5885411ed9..14fe06699d 100644 --- a/tests/test_attention_backend_registry.py +++ b/tests/test_attention_backend_registry.py @@ -1,5 +1,6 @@ # SPDX-License-Identifier: Apache-2.0 # SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import pytest import torch from vllm.platforms.interface import DeviceCapability @@ -71,6 +72,48 @@ class CustomMambaAttentionBackend(AttentionBackend): return None +class StringDecoratorAttentionBackend(AttentionBackend): + """Mock backend defined at module level so get_class() can resolve + the auto-generated class path (local classes are not importable).""" + + @staticmethod + def get_name(): + return "DECORATED" + + @staticmethod + def get_impl_cls(): + return CustomAttentionImpl + + @staticmethod + def get_builder_cls(): + return None + + @staticmethod + def get_required_kv_cache_layout(): + return None + + +class StringDecoratorMambaBackend(AttentionBackend): + """Mock mamba backend defined at module level so get_class() can + resolve the auto-generated class path.""" + + @staticmethod + def get_name(): + return "DECORATED" + + @staticmethod + def get_impl_cls(): + return CustomMambaAttentionImpl + + @staticmethod + def get_builder_cls(): + return None + + @staticmethod + def get_required_kv_cache_layout(): + return None + + def test_custom_attention_backend_requires_dcp_opt_in(): """A new implementation remains usable without implicitly opting into DCP.""" config = AttentionSelectorConfig( @@ -178,3 +221,101 @@ def test_register_custom_mamba_backend_with_class_path(): backend_cls = MambaAttentionBackendEnum.CUSTOM.get_class() assert backend_cls.get_name() == "CUSTOM_MAMBA" assert backend_cls.get_impl_cls() == CustomMambaAttentionImpl + + +@pytest.fixture(params=[AttentionBackendEnum, MambaAttentionBackendEnum]) +def _enum_cls(request): + return request.param + + +def _remove_dynamic_member(enum_cls, member): + value = member.value + name = member.name + enum_cls._member_map_.pop(name, None) + enum_cls._member_names_.remove(name) + delattr(enum_cls, name) + if enum_cls._value2member_map_.get(value) is member: + del enum_cls._value2member_map_[value] + + +def test_register_dynamic_enum_member(_enum_cls): + backend_path = { + AttentionBackendEnum: ( + "tests.test_attention_backend_registry.CustomAttentionBackend" + ), + MambaAttentionBackendEnum: ( + "tests.test_attention_backend_registry.CustomMambaAttentionBackend" + ), + }[_enum_cls] + + member = _enum_cls.register("DYNAMIC_TEST", backend_path) + assert member.name == "DYNAMIC_TEST" + assert member.value == backend_path + assert member is _enum_cls.DYNAMIC_TEST + assert _enum_cls["DYNAMIC_TEST"] is member + _remove_dynamic_member(_enum_cls, member) + + +def test_register_dynamic_enum_member_duplicate_raises(_enum_cls): + _enum_cls.register("DUP_TEST", "some.module.Class") + with pytest.raises(ValueError, match="already exists"): + _enum_cls.register("DUP_TEST", "other.module.OtherClass") + _remove_dynamic_member(_enum_cls, _enum_cls.DUP_TEST) + + +def test_register_dynamic_enum_member_reserved_name_raises(_enum_cls): + with pytest.raises(ValueError, match="Invalid or reserved backend name"): + _enum_cls.register("get_path", "some.module.Class") + + +@pytest.fixture( + params=[ + pytest.param((AttentionBackendEnum, False), id="attention"), + pytest.param((MambaAttentionBackendEnum, True), id="mamba"), + ] +) +def _enum_is_mamba(request): + return request.param + + +def _backend_path(enum_cls): + if enum_cls is AttentionBackendEnum: + return "tests.test_attention_backend_registry.CustomAttentionBackend" + return "tests.test_attention_backend_registry.CustomMambaAttentionBackend" + + +def test_register_backend_with_string_name_direct(_enum_is_mamba): + enum_cls, is_mamba = _enum_is_mamba + path = _backend_path(enum_cls) + register_backend("STRING_DIRECT", path, is_mamba=is_mamba) + + member = enum_cls.STRING_DIRECT + assert member.is_overridden() + assert member.get_path() == path + expected = "CUSTOM" if not is_mamba else "CUSTOM_MAMBA" + assert member.get_class().get_name() == expected + member.clear_override() + _remove_dynamic_member(enum_cls, member) + + +def test_register_backend_with_string_name_decorator(_enum_is_mamba): + enum_cls, is_mamba = _enum_is_mamba + # The decorated class must be module-level: the decorator derives a + # class path from it, and get_class() re-imports that path. A class + # defined inside this test would have a non-importable path. + decorated_cls = ( + StringDecoratorMambaBackend if is_mamba else StringDecoratorAttentionBackend + ) + + result = register_backend("STRING_DECORATOR", is_mamba=is_mamba)(decorated_cls) + assert result is decorated_cls + + member = enum_cls.STRING_DECORATOR + assert member.is_overridden() + assert member.get_path() == ( + f"{decorated_cls.__module__}.{decorated_cls.__qualname__}" + ) + assert member.get_class() is decorated_cls + assert member.get_class().get_name() == "DECORATED" + member.clear_override() + _remove_dynamic_member(enum_cls, member) diff --git a/tests/test_config.py b/tests/test_config.py index 0cf5be6a79..2756deedde 100644 --- a/tests/test_config.py +++ b/tests/test_config.py @@ -47,6 +47,7 @@ from vllm.config.speculative import _validate_qwen3_omni_dspark from vllm.config.utils import get_field from vllm.config.vllm import OPTIMIZATION_LEVEL_TO_CONFIG, OptimizationLevel from vllm.platforms import current_platform +from vllm.sampling_params import BeamSearchParams from vllm.transformers_utils.config import ( _patch_hf_transformers_nested_rope_validation, get_pooling_config, @@ -502,6 +503,17 @@ def test_hisparse_rejects_pipeline_parallelism(monkeypatch): ) +def test_hisparse_rejects_full_cudagraph_mode(monkeypatch): + monkeypatch.setattr(current_platform, "is_cuda", lambda: True) + monkeypatch.setattr(current_platform, "support_static_graph_mode", lambda: True) + monkeypatch.setattr("vllm.config.vllm.HAS_TRITON", True) + with pytest.raises(ValueError, match="does not support cudagraph_mode=FULL"): + VllmConfig( + attention_config=AttentionConfig(hisparse_config=HiSparseConfig()), + compilation_config=CompilationConfig(cudagraph_mode=CUDAGraphMode.FULL), + ) + + def test_hisparse_rejects_disabled_hybrid_kv_cache_manager(monkeypatch): monkeypatch.setattr(current_platform, "is_cuda", lambda: True) monkeypatch.setattr("vllm.config.vllm.HAS_TRITON", True) @@ -831,7 +843,6 @@ def test_breakable_cudagraph_platform_default( ("bi-off", 2048, (16, 8), 6144, 1, False), ("opt-out", 2048, (16, 8), 6144, 1, False), ("eager", 2048, (16, 8), 6144, 1, False), - ("sp", 2048, (16, 8), 6144, 1, False), ("fp16", 2048, (16, 8), 6144, 1, False), ("quantized", 2048, (16, 8), 6144, 1, False), ("untuned", 4096, (32, 32), 11008, 1, False), @@ -874,11 +885,7 @@ def test_batch_invariant_breakable_cudagraph( get_num_kv_heads=lambda pc: heads[1], ) config.parallel_config = SimpleNamespace(tensor_parallel_size=tp) - config.compilation_config = ( - CompilationConfig(pass_config=PassConfig(enable_sp=True)) - if case == "sp" - else CompilationConfig() - ) + config.compilation_config = CompilationConfig() try: assert config._maybe_enable_breakable_cudagraph() is expected if expected: @@ -910,12 +917,13 @@ def test_dsa_models_select_matching_mtp(model_type, expected_architecture): assert hf_config.architectures == [expected_architecture] -def test_v2_model_runner_supports_extract_hidden_states(): +@pytest.mark.parametrize("method", ["extract_hidden_states", "ngram", "ngram_gpu"]) +def test_v2_model_runner_supports_speculative_method(method): config = VllmConfig() config.speculative_config = cast( SpeculativeConfig, SimpleNamespace( - method="extract_hidden_states", + method=method, parallel_drafting=False, enable_adaptive_verification=False, ), @@ -924,6 +932,30 @@ def test_v2_model_runner_supports_extract_hidden_states(): assert config._get_v2_model_runner_unsupported_features() == [] +@pytest.mark.parametrize("use_heterogeneous_vocab", [False, True]) +def test_v2_model_runner_heterogeneous_vocab_draft_falls_back_to_v1( + use_heterogeneous_vocab, +): + """The V2 draft-model speculator exchanges token ids with the target + without a vocab mapping, so a heterogeneous-vocab (TLI) draft must stay on + V1, where VocabMapping translates them.""" + config = VllmConfig() + config.speculative_config = cast( + SpeculativeConfig, + SimpleNamespace( + method="draft_model", + parallel_drafting=False, + enable_adaptive_verification=False, + use_heterogeneous_vocab=use_heterogeneous_vocab, + ), + ) + + expected = ["heterogeneous-vocabulary draft models"] + assert config._get_v2_model_runner_unsupported_features() == ( + expected if use_heterogeneous_vocab else [] + ) + + def test_v2_model_runner_supports_custom_logits_processors(): config = VllmConfig() config.model_config = cast( @@ -984,7 +1016,6 @@ def test_resolve_cudagraph_mode_adjusts_spec_decode_sizes_only_for_v1( "FakeAttentionBackend", uniform_decode_query_len=4, use_v2_model_runner=use_v2_model_runner, - tensor_parallel_size=1, ) assert cudagraph_mode == CUDAGraphMode.FULL_AND_PIECEWISE @@ -1067,7 +1098,6 @@ def test_resolve_cudagraph_mode_skips_mamba_block_check_while_profiling(): "FakeAttentionBackend", uniform_decode_query_len=1, use_v2_model_runner=True, - tensor_parallel_size=1, kv_cache_config=kv_cache_config, max_num_reqs=256, ) @@ -1080,7 +1110,6 @@ def test_resolve_cudagraph_mode_skips_mamba_block_check_while_profiling(): "FakeAttentionBackend", uniform_decode_query_len=1, use_v2_model_runner=True, - tensor_parallel_size=1, kv_cache_config=kv_cache_config, max_num_reqs=256, is_profiling=True, @@ -3359,8 +3388,6 @@ def test_vllm_config_explicit_overrides(): take precedence over callable defaults, across different models and optimization levels. """ - from vllm.config.compilation import PassConfig - quantized_model = ModelConfig("RedHatAI/Llama-3.2-1B-FP8") moe_model = ModelConfig("deepseek-ai/DeepSeek-V2-Lite") regular_model = ModelConfig("Qwen/Qwen1.5-7B") @@ -3656,7 +3683,7 @@ def test_target_only_gumbel_allows_speculative_decoding(caplog_vllm, disable_log ) with caplog_vllm.at_level(logging.WARNING): - config._check_watermarking_unsupported() + config._check_supports_watermarking() assert "Target-only watermarking leaves accepted draft tokens" in caplog_vllm.text @@ -3677,7 +3704,7 @@ def test_speculative_watermarking_without_context_dedup_does_not_warn( caplog_vllm.clear() with caplog_vllm.at_level(logging.WARNING): - config._check_watermarking_unsupported() + config._check_supports_watermarking() assert "dedup" not in caplog_vllm.text.lower() @@ -3698,7 +3725,7 @@ def test_speculative_context_dedup_logs_no_unsupported_warning( caplog_vllm.clear() with caplog_vllm.at_level(logging.WARNING): - config._check_watermarking_unsupported() + config._check_supports_watermarking() assert "dedup" not in caplog_vllm.text.lower() @@ -3716,7 +3743,7 @@ def test_gumbel_rejects_speculative_decoding_without_target_only(): ) with pytest.raises(ValueError, match="'gumbel'.*allow_target_only_watermarking"): - config._check_watermarking_unsupported() + config._check_supports_watermarking() def test_dual_key_gumbel_warns_that_configured_alpha_is_unused( @@ -3734,7 +3761,7 @@ def test_dual_key_gumbel_warns_that_configured_alpha_is_unused( ) with caplog_vllm.at_level(logging.WARNING): - config._check_watermarking_unsupported() + config._check_supports_watermarking() assert "The configured alpha=0.25 is not used" in caplog_vllm.text @@ -3760,7 +3787,7 @@ def test_dual_key_gumbel_alpha_warning_is_not_emitted( ) with caplog_vllm.at_level(logging.WARNING): - config._check_watermarking_unsupported() + config._check_supports_watermarking() assert "is not used" not in caplog_vllm.text @@ -3776,7 +3803,7 @@ def test_dual_key_gumbel_requires_probabilistic_drafting(): ) with pytest.raises(ValueError, match="draft_sample_method='probabilistic'"): - config._check_watermarking_unsupported() + config._check_supports_watermarking() @pytest.mark.parametrize("method", ["eagle", "eagle3", "mtp"]) @@ -3790,7 +3817,7 @@ def test_dual_key_gumbel_supports_probabilistic_speculative_decoding(method): parallel_drafting=False, ) - config._check_watermarking_unsupported() + config._check_supports_watermarking() def test_dual_key_gumbel_supports_dspark(): @@ -3803,7 +3830,7 @@ def test_dual_key_gumbel_supports_dspark(): parallel_drafting=True, ) - config._check_watermarking_unsupported() + config._check_supports_watermarking() def test_dual_key_gumbel_rejects_non_autoregressive_speculation(): @@ -3817,7 +3844,7 @@ def test_dual_key_gumbel_rejects_non_autoregressive_speculation(): ) with pytest.raises(ValueError, match="autoregressive model-based"): - config._check_watermarking_unsupported() + config._check_supports_watermarking() @pytest.mark.parametrize( @@ -3841,17 +3868,40 @@ def test_dual_key_gumbel_rejects_incompatible_speculative_modes(overrides, match config.speculative_config = SimpleNamespace(**values) with pytest.raises(ValueError, match=match): - config._check_watermarking_unsupported() + config._check_supports_watermarking() -def test_gumbel_watermark_rejects_beam_search(): - with pytest.raises(ValueError, match="Beam search is not supported"): - _watermarked_vllm_config()._check_watermarking_unsupported(beam_search=True) +def test_gumbel_watermark_disables_beam_search_watermarking(caplog_vllm): + enabled = _watermarked_vllm_config()._check_supports_watermarking( + BeamSearchParams(beam_width=2, max_tokens=1) + ) + + assert not enabled + assert "beam search requests will run without watermarking" in caplog_vllm.text + + +def test_gumbel_watermark_allows_unwatermarked_beam_search(): + _watermarked_vllm_config()._check_supports_watermarking( + BeamSearchParams(beam_width=2, max_tokens=1, watermarking=False) + ) + + +def test_unwatermarked_beam_does_not_bypass_engine_incompatibilities(): + config = _watermarked_vllm_config() + config.speculative_config = SpeculativeConfig( + method="ngram", + num_speculative_tokens=1, + ) + + with pytest.raises(ValueError, match="Speculative decoding with watermarking"): + config._check_supports_watermarking( + BeamSearchParams(beam_width=2, max_tokens=1, watermarking=False) + ) def test_gumbel_watermark_rejects_custom_sampler(): with pytest.raises(ValueError, match="custom samplers are not supported"): - _watermarked_vllm_config()._check_watermarking_unsupported(custom_sampler=True) + _watermarked_vllm_config()._check_supports_watermarking(custom_sampler=True) def test_watermark_key_must_fit_in_64_bits(): diff --git a/tests/test_outputs.py b/tests/test_outputs.py index 9acc1f7b74..e7b732b3e2 100644 --- a/tests/test_outputs.py +++ b/tests/test_outputs.py @@ -6,6 +6,7 @@ import torch from vllm.outputs import ( ClassificationRequestOutput, + CompletionOutput, EmbeddingRequestOutput, PoolingOutput, PoolingRequestOutput, @@ -13,6 +14,7 @@ from vllm.outputs import ( RequestOutput, ScoringRequestOutput, ) +from vllm.v1.metrics.stats import RequestSpecDecodeMetrics pytestmark = pytest.mark.cpu_test @@ -61,3 +63,50 @@ def test_specialized_pooling_outputs_preserve_request_error(output_type, empty_v value = getattr(output.outputs, "probs", value) value = getattr(output.outputs, "score", value) assert value == empty_value + + +def test_request_output_add_preserves_terminal_spec_decode_metrics(): + accumulated = RequestOutput( + request_id="request", + prompt=None, + prompt_token_ids=[], + prompt_logprobs=None, + outputs=[ + CompletionOutput( + index=0, + text="first", + token_ids=[1], + cumulative_logprob=None, + logprobs=None, + ) + ], + finished=False, + ) + spec_decode_metrics = RequestSpecDecodeMetrics.new(num_spec_tokens=2) + spec_decode_metrics.observe(num_draft_tokens=2, num_accepted=1) + terminal = RequestOutput( + request_id="request", + prompt=None, + prompt_token_ids=[], + prompt_logprobs=None, + outputs=[ + CompletionOutput( + index=0, + text="second", + token_ids=[2], + cumulative_logprob=None, + logprobs=None, + finish_reason="length", + spec_decode_metrics=spec_decode_metrics, + ) + ], + finished=True, + ) + + accumulated.add(terminal, aggregate=True) + + output = accumulated.outputs[0] + assert output.token_ids == [1, 2] + assert output.finish_reason == "length" + assert accumulated.finished + assert output.spec_decode_metrics is spec_decode_metrics diff --git a/tests/test_sampling_params.py b/tests/test_sampling_params.py index a6e057ad7a..f32a3d4c36 100644 --- a/tests/test_sampling_params.py +++ b/tests/test_sampling_params.py @@ -112,6 +112,7 @@ def test_diffusion_canvas_width_is_accepted_with_either_scheduler( diffusion_config=DiffusionConfig(canvas_length=8), tokenizer=None, validate_logits_processors_params=lambda params: None, + resolve_watermarking=lambda params: False, ) params = SamplingParams(extra_args=extra_args) InputProcessor._validate_params(processor, params, ("generate",)) diff --git a/tests/test_zen_cpu_platform_detection.py b/tests/test_zen_cpu_platform_detection.py index 505d2d3bfd..b8a3949923 100644 --- a/tests/test_zen_cpu_platform_detection.py +++ b/tests/test_zen_cpu_platform_detection.py @@ -13,23 +13,6 @@ from vllm.platforms import ( ) -@pytest.fixture -def caplog_vllm(caplog): - """`caplog`, but also captures vLLM's loggers. - - vllm/logger.py configures the "vllm" logger with propagate=False, so - records from child loggers like "vllm.platforms" reach vLLM's own - handler but never bubble up to the root logger where plain `caplog` - listens. Attach caplog's handler directly to "vllm" to see them too. - """ - logger = logging.getLogger("vllm") - logger.addHandler(caplog.handler) - try: - yield caplog - finally: - logger.removeHandler(caplog.handler) - - def test_is_amd_zen_cpu_detects_amd_with_avx512(): cpuinfo = "vendor_id: AuthenticAMD\nflags: avx avx2 avx512f avx512bw" with ( diff --git a/tests/test_zentorch_version_pin.py b/tests/test_zentorch_version_pin.py new file mode 100644 index 0000000000..91125863be --- /dev/null +++ b/tests/test_zentorch_version_pin.py @@ -0,0 +1,79 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Static consistency check between the pinned zentorch and torch versions. + +zentorch (the AMD Zen CPU optimization extra) tracks the torch release that +vLLM depends on, differing only by an extra trailing "patch level" segment. +Concretely, torch ``X.Y.Z`` corresponds to zentorch ``X.Y.Z.W``. + +This test parses the raw text of ``setup.py`` and ``requirements/cpu.txt`` and +string-matches the versions. +""" + +from pathlib import Path + +import pytest +import regex as re + +pytestmark = [pytest.mark.cpu_test, pytest.mark.skip_global_cleanup] + +REPO_ROOT = Path(__file__).resolve().parent.parent +SETUP_PY = REPO_ROOT / "setup.py" +# zentorch is the AMD Zen *CPU* extra, so the CPU requirement file is the +# authoritative source for the torch version it must track. +CPU_REQUIREMENTS = REPO_ROOT / "requirements" / "cpu.txt" + + +def _extract_zentorch_version() -> str: + """Pull the pinned zentorch version out of setup.py's extras_require.""" + text = SETUP_PY.read_text() + matches = re.findall(r"zentorch==([0-9][0-9A-Za-z.\-+]*)", text) + assert matches, f"No 'zentorch==' pin found in {SETUP_PY}" + assert len(set(matches)) == 1, ( + f"Multiple conflicting zentorch pins found in {SETUP_PY}: {matches}" + ) + return matches[0] + + +def _extract_torch_version() -> str: + """Pull the pinned torch version out of requirements/cpu.txt. + + The local version label (e.g. the ``+cpu`` in ``2.13.0+cpu``) is stripped + so only the upstream release ``X.Y.Z`` is compared. + """ + text = CPU_REQUIREMENTS.read_text() + # Match lines like: torch==2.13.0+cpu; platform_machine == "x86_64" ... + # but not torchaudio / torchvision / open-clip-torch etc. + matches = re.findall(r"(?m)^\s*torch==([0-9][0-9A-Za-z.\-+]*)", text) + assert matches, f"No 'torch==' pin found in {CPU_REQUIREMENTS}" + # Strip local version labels (+cpu, +cu130, ...) and keep the release only. + releases = {m.split("+", 1)[0] for m in matches} + assert len(releases) == 1, ( + f"Multiple conflicting torch release pins found in {CPU_REQUIREMENTS}: " + f"{sorted(releases)}" + ) + return releases.pop() + + +def test_zentorch_pin_tracks_torch_release(): + zentorch_version = _extract_zentorch_version() + torch_version = _extract_torch_version() + + zentorch_parts = zentorch_version.split(".") + torch_parts = torch_version.split(".") + + # torch is X.Y.Z; zentorch is X.Y.Z.W (one extra trailing patch-level + # segment). Enforce both the shape and the shared prefix. + assert len(torch_parts) == 3, ( + f"Expected torch release of the form 'X.Y.Z', got {torch_version!r}" + ) + assert len(zentorch_parts) == 4, ( + f"Expected zentorch pin of the form 'X.Y.Z.W', got {zentorch_version!r}" + ) + assert zentorch_parts[:3] == torch_parts, ( + "zentorch version is out of sync with torch. " + f"setup.py pins zentorch=={zentorch_version} (base " + f"{'.'.join(zentorch_parts[:3])}) but requirements/cpu.txt pins " + f"torch=={torch_version}. When bumping torch, bump the first three " + "components of the zentorch pin in setup.py to match." + ) diff --git a/tests/tokenizers_/test_detokenize.py b/tests/tokenizers_/test_detokenize.py index d0117e4711..1dd2a000a6 100644 --- a/tests/tokenizers_/test_detokenize.py +++ b/tests/tokenizers_/test_detokenize.py @@ -8,7 +8,11 @@ import pytest from transformers import AutoTokenizer, PythonBackend, TokenizersBackend from vllm.sampling_params import SamplingParams -from vllm.tokenizers.detokenizer_utils import convert_ids_list_to_tokens +from vllm.tokenizers.detokenizer_utils import ( + convert_ids_list_to_tokens, + convert_prompt_ids_to_tokens, + detokenize_incrementally, +) from vllm.tokenizers.mistral import MistralTokenizer from vllm.tokenizers.protocol import TokenizerLike from vllm.v1.engine import EngineCoreRequest @@ -98,6 +102,48 @@ def tokenizer(tokenizer_name): ) +@pytest.mark.parametrize( + "tokenizer_name", ["openai-community/gpt2", "mistralai/Pixtral-12B-2409"] +) +def test_slow_decode_long_history_preserves_text_and_token_ids(tokenizer): + truth = "A map. 我很感谢你的热情 🙂🌶️\n" * 32 + ids = tokenizer.encode(truth, add_special_tokens=False) + text, output_ids = _run_incremental_decode( + tokenizer, ids, skip_special_tokens=True, starting_index=0, fast=False + ) + assert text == tokenizer.decode(ids, skip_special_tokens=True) + assert output_ids == ids + + +@pytest.mark.parametrize( + "tokenizer_name", ["openai-community/gpt2", "mistralai/Pixtral-12B-2409"] +) +def test_first_incremental_decode_returns_the_prompt_window(tokenizer): + ids = tokenizer.encode("A map in a library. " * 8, add_special_tokens=False) + prompt_tokens, _, _ = convert_prompt_ids_to_tokens( + tokenizer, ids[:-1], skip_special_tokens=True + ) + new_token = tokenizer.convert_ids_to_tokens([ids[-1]], skip_special_tokens=True) + tokens, _, _, _ = detokenize_incrementally( + tokenizer, ids, None, 0, 0, skip_special_tokens=True + ) + assert tokens == prompt_tokens + new_token + + +@pytest.mark.parametrize( + "tokenizer_name", ["openai-community/gpt2", "mistralai/Pixtral-12B-2409"] +) +def test_incremental_decode_past_the_window_emits_no_text(tokenizer): + ids = tokenizer.encode("A map in a library.", add_special_tokens=False) + tokens = tokenizer.convert_ids_to_tokens(ids[:-1], skip_special_tokens=True) + offset = len(tokens) + 1 + _, text, prefix_offset, read_offset = detokenize_incrementally( + tokenizer, ids, tokens, offset, offset, skip_special_tokens=True + ) + assert text == "" + assert (prefix_offset, read_offset) == (offset, offset) + + @pytest.mark.parametrize("tokenizer_name", ["mistralai/Pixtral-12B-2409"]) @pytest.mark.parametrize( "truth", diff --git a/tests/tokenizers_/test_registry.py b/tests/tokenizers_/test_registry.py index 312b0c03a9..b9d9c732f3 100644 --- a/tests/tokenizers_/test_registry.py +++ b/tests/tokenizers_/test_registry.py @@ -6,7 +6,7 @@ from types import SimpleNamespace from unittest.mock import patch import pytest -from transformers import AutoConfig +from transformers import AutoConfig, Qwen3_5MoeConfig from transformers.models.auto.configuration_auto import CONFIG_MAPPING from vllm.tokenizers import TokenizerLike @@ -18,7 +18,6 @@ from vllm.tokenizers.registry import ( get_tokenizer, resolve_tokenizer_args, ) -from vllm.transformers_utils.configs.qwen3_5_moe import Qwen3_5MoeConfig class TestTokenizer(TokenizerLike): diff --git a/tests/tool_parsers/test_internlm2_tool_parser.py b/tests/tool_parsers/test_internlm2_tool_parser.py index 881933890d..28465654e4 100644 --- a/tests/tool_parsers/test_internlm2_tool_parser.py +++ b/tests/tool_parsers/test_internlm2_tool_parser.py @@ -10,6 +10,7 @@ from tests.tool_parsers.common_tests import ( ToolParserTestConfig, ToolParserTests, ) +from tests.tool_parsers.utils import run_tool_extraction_streaming from vllm.tokenizers import TokenizerLike from vllm.tool_parsers.internlm2_tool_parser import Internlm2ToolParser @@ -162,3 +163,30 @@ def test_streaming_arguments_in_single_delta(default_tokenizer: TokenizerLike) - streamed += arguments assert json.loads(streamed) == {"city": "Dallas", "state": "TX"} + + +@pytest.mark.parametrize( + "arguments_deltas", + [ + ['{"location": "Beijing', '", "days": 3}}<|action_end|>'], + [*'{"location": "Beijing", "days": 3}}', "<|action_end|>"], + ], + ids=["split_inside_string", "char_by_char"], +) +def test_streaming_arguments_match_final_arguments( + default_tokenizer: TokenizerLike, arguments_deltas: list[str] +) -> None: + """Streamed argument deltas must add up to the final arguments, wherever + the chunk boundaries fall.""" + parser = Internlm2ToolParser(default_tokenizer) + deltas = [ + '<|action_start|><|plugin|>{"name": "get_weather", "parameters": ', + *arguments_deltas, + ] + + reconstructor = run_tool_extraction_streaming(parser, deltas) + + assert len(reconstructor.tool_calls) == 1 + streamed = reconstructor.tool_calls[0].function.arguments + streamed += parser.get_remaining_unstreamed_args() + assert streamed == '{"location": "Beijing", "days": 3}' diff --git a/tests/tool_parsers/test_minimax_m3_tool_parser.py b/tests/tool_parsers/test_minimax_m3_tool_parser.py index 575892217a..c01ab89d1e 100644 --- a/tests/tool_parsers/test_minimax_m3_tool_parser.py +++ b/tests/tool_parsers/test_minimax_m3_tool_parser.py @@ -8,16 +8,13 @@ import pytest from vllm.entrypoints.generate.base.protocol import DeltaMessage from vllm.entrypoints.openai.chat_completion.protocol import ( + ChatCompletionRequest, ChatCompletionToolsParam, FunctionDefinition, ) from vllm.tool_parsers import ToolParserManager from vllm.tool_parsers.minimax_m3_tool_parser import MinimaxM3ToolParser -# MinimaxM3ToolParser extends RustToolParser; skip when the PyO3 extension -# is absent (mirrors the guard in test_rust_tool_parser.py). -pytest.importorskip("vllm._rust_tool_parser") - pytestmark = [pytest.mark.cpu_test, pytest.mark.skip_global_cleanup] NS = "]<]minimax[>[" @@ -108,6 +105,23 @@ def build_order_call() -> str: ) +ORDER_ARGUMENTS = { + "user_id": 42, + "urgent": True, + "note": "Please leave at front desk.", + "shipping": {"city": "Singapore", "zip": 18956}, + "items": [ + {"sku": "book-001", "qty": 2}, + {"sku": "pen-007", "qty": 5}, + ], + "metadata": { + "source": "mobile", + "campaign": "may-launch", + }, + "duplicate_demo": {"tag": ["a", "b"]}, +} + + def build_order_invocation(user_id: int) -> str: return ( f'{NS}' @@ -181,6 +195,20 @@ def test_minimax_m3_parser_registered(): assert ToolParserManager.get_tool_parser("minimax_m3") is MinimaxM3ToolParser +def test_required_tool_choice_keeps_native_syntax(parser): + request = ChatCompletionRequest( + model="minimax-m3", + messages=[{"role": "user", "content": "order"}], + tools=sample_tools(), + tool_choice="required", + ) + + adjusted = parser.adjust_request(request) + + assert adjusted.structured_outputs is None + assert adjusted.skip_special_tokens + + def test_non_streaming_nested_tool_call(parser): result = parser.extract_tool_calls( "I will create it.\n" + build_order_call(), @@ -192,21 +220,7 @@ def test_non_streaming_nested_tool_call(parser): assert len(result.tool_calls) == 1 tool_call = result.tool_calls[0] assert tool_call.function.name == "create_order" - assert json.loads(tool_call.function.arguments) == { - "user_id": 42, - "urgent": True, - "note": "Please leave at front desk.", - "shipping": {"city": "Singapore", "zip": 18956}, - "items": [ - {"sku": "book-001", "qty": 2}, - {"sku": "pen-007", "qty": 5}, - ], - "metadata": { - "source": "mobile", - "campaign": "may-launch", - }, - "duplicate_demo": {"tag": ["a", "b"]}, - } + assert json.loads(tool_call.function.arguments) == ORDER_ARGUMENTS def test_non_streaming_without_tool_call_keeps_content(parser): @@ -258,8 +272,5 @@ def test_streaming_nested_tool_call(parser): assert len(tool_calls) == 1 assert tool_calls[0]["name"] == "create_order" assert tool_calls[0]["id"] is not None - assert json.loads(tool_calls[0]["arguments"]) == json.loads( - parser.streamed_args_for_tool[0] - ) - assert json.loads(parser.prev_tool_call_arr[0]["arguments"])["items"][1]["qty"] == 5 + assert json.loads(tool_calls[0]["arguments"]) == ORDER_ARGUMENTS assert results[-1].content is None diff --git a/tests/tool_parsers/test_rust_tool_parser.py b/tests/tool_parsers/test_rust_tool_parser.py deleted file mode 100644 index 2349d4d292..0000000000 --- a/tests/tool_parsers/test_rust_tool_parser.py +++ /dev/null @@ -1,328 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# SPDX-FileCopyrightText: Copyright contributors to the vLLM project - -import json -from collections.abc import Sequence -from unittest.mock import MagicMock - -import pytest - -from vllm.entrypoints.openai.chat_completion.protocol import ( - ChatCompletionRequest, - ChatCompletionToolsParam, -) -from vllm.tool_parsers.rust_tool_parser import RustToolParser - -# The PyO3 extension is an optional build artifact; skip when absent. -_rust_tool_parser = pytest.importorskip("vllm._rust_tool_parser") - -MOCK_TOKENIZER = MagicMock() -MOCK_TOKENIZER.get_vocab.return_value = {} - -TC_START = "<|DSML|tool_calls>" -TC_END = "" -INV_START = '<|DSML|invoke name="' -INV_END = "" -PARAM_START = '<|DSML|parameter name="' -PARAM_END = "" - - -class DeepSeekV4RustToolParser(RustToolParser): - rust_parser_name = "DeepSeekV4ToolParser" - tool_call_start_token = TC_START - - -class KimiK2RustToolParser(RustToolParser): - rust_parser_name = "KimiK2ToolParser" - tool_call_start_token = "<|tool_calls_section_begin|>" - - -def sample_tools() -> list[ChatCompletionToolsParam]: - return [ - ChatCompletionToolsParam( - type="function", - function={ - "name": "get_weather", - "description": "Get weather for a location", - "parameters": { - "type": "object", - "properties": { - "location": {"type": "string"}, - "date": {"type": "string"}, - }, - }, - }, - ), - ChatCompletionToolsParam( - type="function", - function={ - "name": "add", - "description": "Add two integers", - "parameters": { - "type": "object", - "properties": { - "x": {"type": "integer"}, - "y": {"type": "integer"}, - }, - }, - }, - ), - ] - - -EXPECTED_CALLS = [ - ("get_weather", {"location": "SF", "date": "2024-01-16"}), - ("add", {"x": 3, "y": 5}), -] - - -def build_invoke( - function_name: str, - params: Sequence[tuple[str, str, bool]], -) -> str: - param_text = "\n".join( - f'{PARAM_START}{name}" string="{str(is_string).lower()}">{value}{PARAM_END}' - for name, value, is_string in params - ) - return f'{INV_START}{function_name}">\n{param_text}\n{INV_END}\n' - - -def build_tool_call() -> str: - weather = build_invoke( - "get_weather", - [ - ("location", "SF", True), - ("date", "2024-01-16", True), - ], - ) - add = build_invoke( - "add", - [ - ("x", "3", False), - ("y", "5", False), - ], - ) - return f"{TC_START}\n{weather}{add}{TC_END}" - - -def parse_streaming( - parser: DeepSeekV4RustToolParser, - text: str, - chunk_size: int, -) -> list: - deltas = [] - previous_text = "" - for start in range(0, len(text), chunk_size): - delta_text = text[start : start + chunk_size] - current_text = previous_text + delta_text - delta = parser.extract_tool_calls_streaming( - previous_text=previous_text, - current_text=current_text, - delta_text=delta_text, - previous_token_ids=[], - current_token_ids=[], - delta_token_ids=[1], - request=MagicMock(), - ) - previous_text = current_text - if delta is not None: - deltas.append(delta) - - delta = parser.extract_tool_calls_streaming( - previous_text=previous_text, - current_text=previous_text, - delta_text="", - previous_token_ids=[], - current_token_ids=[], - delta_token_ids=[2], - request=MagicMock(), - ) - if delta is not None: - deltas.append(delta) - - return deltas - - -def collect_streamed_arguments(deltas: Sequence, tool_index: int = 0) -> str: - return "".join( - tool_call.function.arguments - for delta in deltas - for tool_call in delta.tool_calls or [] - if ( - tool_call.index == tool_index - and tool_call.function is not None - and tool_call.function.arguments is not None - ) - ) - - -def test_rust_tool_parser_extension_typed_api() -> None: - tools = [ - _rust_tool_parser.Tool( - tool.function.name, - tool.function.description, - tool.function.parameters, - None, - ) - for tool in sample_tools() - ] - parser = _rust_tool_parser.ToolParser("DeepSeekV4ToolParser", tools) - output = _rust_tool_parser.ToolParserOutput() - - parser.parse_into(build_tool_call(), output) - output.append(parser.finish()) - output = output.coalesce() - - assert parser.preserve_special_tokens() - assert output.normal_text == "" - assert len(output.calls) == 2 - for call, (name, arguments) in zip(output.calls, EXPECTED_CALLS): - assert call.name == name - assert json.loads(call.arguments) == arguments - - -def test_rust_tool_parser_adapter_extracts_complete_output() -> None: - tools = sample_tools() - parser = DeepSeekV4RustToolParser(MOCK_TOKENIZER, tools=tools) - - result = parser.extract_tool_calls( - "Let me create it. " + build_tool_call(), - ChatCompletionRequest(messages=[], model="m", tools=tools), - ) - - assert result.tools_called - assert result.content == "Let me create it. " - assert len(result.tool_calls) == 2 - for tool_call, (name, arguments) in zip(result.tool_calls, EXPECTED_CALLS): - assert tool_call.function.name == name - assert json.loads(tool_call.function.arguments) == arguments - - -def test_rust_tool_parser_adapter_streaming_handles_multiple_calls() -> None: - parser = DeepSeekV4RustToolParser(MOCK_TOKENIZER, tools=sample_tools()) - - deltas = parse_streaming(parser, build_tool_call(), chunk_size=5) - - names = [ - tool_call.function.name - for delta in deltas - for tool_call in delta.tool_calls or [] - if tool_call.function is not None and tool_call.function.name is not None - ] - assert names == [name for name, _ in EXPECTED_CALLS] - for index, (_, arguments) in enumerate(EXPECTED_CALLS): - assert json.loads(collect_streamed_arguments(deltas, index)) == arguments - - -def test_rust_tool_parser_adapter_ignores_midstream_empty_delta() -> None: - parser = DeepSeekV4RustToolParser(MOCK_TOKENIZER, tools=sample_tools()) - text = build_tool_call() - split_at = len(TC_START) + 8 - deltas = [] - previous_text = "" - - for delta_text in (text[:split_at], "", text[split_at:], ""): - current_text = previous_text + delta_text - delta = parser.extract_tool_calls_streaming( - previous_text=previous_text, - current_text=current_text, - delta_text=delta_text, - previous_token_ids=[], - current_token_ids=[], - delta_token_ids=[1], - request=MagicMock(), - ) - previous_text = current_text - if delta is not None: - deltas.append(delta) - - names = [ - tool_call.function.name - for delta in deltas - for tool_call in delta.tool_calls or [] - if tool_call.function is not None and tool_call.function.name is not None - ] - assert names == [name for name, _ in EXPECTED_CALLS] - for index, (_, arguments) in enumerate(EXPECTED_CALLS): - assert json.loads(collect_streamed_arguments(deltas, index)) == arguments - - -KIMI_EXPECTED_IDS = ["functions.get_weather:0", "functions.add:1"] - - -def build_kimi_tool_call() -> str: - return ( - "<|tool_calls_section_begin|>" - "<|tool_call_begin|>functions.get_weather:0<|tool_call_argument_begin|>" - '{"location": "SF", "date": "2024-01-16"}<|tool_call_end|>' - "<|tool_call_begin|>functions.add:1<|tool_call_argument_begin|>" - '{"x": 3, "y": 5}<|tool_call_end|>' - "<|tool_calls_section_end|>" - ) - - -def test_rust_tool_parser_adapter_complete_prefers_model_tool_call_ids() -> None: - tools = sample_tools() - parser = KimiK2RustToolParser(MOCK_TOKENIZER, tools=tools) - - result = parser.extract_tool_calls( - "Let me check. " + build_kimi_tool_call(), - ChatCompletionRequest(messages=[], model="m", tools=tools), - ) - - assert result.tools_called - assert [tool_call.id for tool_call in result.tool_calls] == KIMI_EXPECTED_IDS - for tool_call, (name, arguments) in zip(result.tool_calls, EXPECTED_CALLS): - assert tool_call.function.name == name - assert json.loads(tool_call.function.arguments) == arguments - - -def test_rust_tool_parser_adapter_streaming_prefers_model_tool_call_ids() -> None: - parser = KimiK2RustToolParser(MOCK_TOKENIZER, tools=sample_tools()) - - deltas = parse_streaming(parser, build_kimi_tool_call(), chunk_size=5) - - ids = [ - tool_call.id - for delta in deltas - for tool_call in delta.tool_calls or [] - if tool_call.id is not None - ] - assert ids == KIMI_EXPECTED_IDS - for index, (_, arguments) in enumerate(EXPECTED_CALLS): - assert json.loads(collect_streamed_arguments(deltas, index)) == arguments - - -def test_rust_tool_parser_adapter_streaming_generates_ids_as_fallback() -> None: - # DeepSeekV4 never emits model tool call IDs, so the bridge mints them. - parser = DeepSeekV4RustToolParser(MOCK_TOKENIZER, tools=sample_tools()) - - deltas = parse_streaming(parser, build_tool_call(), chunk_size=5) - - ids = [ - tool_call.id - for delta in deltas - for tool_call in delta.tool_calls or [] - if tool_call.function is not None and tool_call.function.name is not None - ] - assert len(ids) == len(EXPECTED_CALLS) - assert all(ids) - assert len(set(ids)) == len(ids) - - -def test_rust_tool_parser_adapter_adjust_request_is_opaque() -> None: - tools = sample_tools() - parser = DeepSeekV4RustToolParser(MOCK_TOKENIZER, tools=tools) - request = ChatCompletionRequest( - messages=[], - model="m", - tools=tools, - tool_choice="required", - skip_special_tokens=True, - ) - - adjusted = parser.adjust_request(request) - - assert adjusted is request - assert adjusted.skip_special_tokens is False - assert adjusted.structured_outputs is None diff --git a/tests/tool_parsers/test_structural_tag_registry.py b/tests/tool_parsers/test_structural_tag_registry.py index d5ab3e4f1a..817c48a440 100644 --- a/tests/tool_parsers/test_structural_tag_registry.py +++ b/tests/tool_parsers/test_structural_tag_registry.py @@ -21,6 +21,8 @@ from vllm.entrypoints.openai.chat_completion.protocol import ( ChatCompletionToolsParam, ) from vllm.entrypoints.openai.responses.protocol import ResponsesRequest +from vllm.entrypoints.openai.responses.utils import construct_tool_dicts +from vllm.exceptions import VLLMValidationError from vllm.parser.abstract_parser import DelegatingParser from vllm.parser.plamo3 import ( BEGIN_TOOL_ARGUMENTS, @@ -46,6 +48,7 @@ from vllm.tool_parsers.hermes_tool_parser import Hermes2ProToolParser from vllm.tool_parsers.kimi_k2_tool_parser import KimiK2ToolParser from vllm.tool_parsers.kimi_k3_tool_parser import KimiK3ToolParser from vllm.tool_parsers.llama_tool_parser import Llama3JsonToolParser +from vllm.tool_parsers.longcat_tool_parser import LongcatFlashToolParser from vllm.tool_parsers.mimo_tool_parser import MiMoToolParser from vllm.tool_parsers.minimax_m2_tool_parser import MinimaxM2ToolParser from vllm.tool_parsers.plamo3_engine_tool_parser import Plamo3EngineToolParser @@ -203,6 +206,131 @@ def test_get_model_structural_tag_supports_all_xgrammar_builtins( assert isinstance(tag, StructuralTag) +_MCP_TOOL = {"type": "mcp", "server_label": "docs", "server_url": "http://mcp"} + + +def _namespace_tool(strict: bool | None = None) -> dict[str, Any]: + lookup = { + "type": "function", + "name": "lookup", + "parameters": {"type": "object", "properties": {"id": {"type": "string"}}}, + } + if strict is not None: + lookup["strict"] = strict + return {"type": "namespace", "name": "crm", "description": "", "tools": [lookup]} + + +def _hermes_call(name: str) -> str: + return ( + f'\n{{"name": "{name}", "arguments": {{"id": "1"}}}}\n' + ) + + +@pytest.mark.parametrize("model", sorted(SUPPORTED_STRUCTURAL_TAG_MODELS)) +def test_no_structural_tag_without_function_tools(model: str): + """Builtin tools never reach the prompt, so there is nothing to constrain.""" + request = ResponsesRequest.model_validate( + {"input": "hi", "tools": [_MCP_TOOL, {"type": "web_search"}]} + ) + + tag = get_model_structural_tag( + model=model, + tools=request.tools, + tool_choice="auto", + reasoning=False, + strict_level=ToolStrictLevel.FUNCTION, + ) + + assert tag is None + with pytest.raises(VLLMValidationError, match="no function tool"): + get_model_structural_tag( + model=model, + tools=request.tools, + tool_choice="required", + reasoning=False, + ) + + +@pytest.mark.parametrize("tool_choice", ["auto", "required"]) +def test_structural_tag_allows_exactly_the_prompt_tools(tool_choice: str): + """The grammar lets the model call every tool its prompt lists, and no other.""" + request = ResponsesRequest.model_validate( + { + "input": "hi", + "tools": [ + {"type": "function", "name": "get_weather", "parameters": {}}, + _namespace_tool(), + _MCP_TOOL, + ], + } + ) + prompt_names = [ + tool["function"]["name"] + for tool in construct_tool_dicts(request.tools, tool_choice) + ] + + tag = get_model_structural_tag( + model="hermes", + tools=request.tools, + tool_choice=tool_choice, + reasoning=False, + strict_level=ToolStrictLevel.FUNCTION, + ) + + assert prompt_names == ["get_weather", "crm__lookup"] + assert tag is not None + grammar = Grammar.from_structural_tag(json.dumps(tag.model_dump())) + for name in prompt_names: + assert _is_grammar_accept_string(grammar, _hermes_call(name)) + assert not _is_grammar_accept_string(grammar, _hermes_call("docs")) + + +def test_strict_namespace_function_enables_auto_structural_tag(): + request = ResponsesRequest.model_validate( + {"input": "hi", "tools": [_namespace_tool(strict=True)]} + ) + + tag = get_model_structural_tag( + model="hermes", tools=request.tools, tool_choice="auto", reasoning=False + ) + + assert tag is not None + grammar = Grammar.from_structural_tag(json.dumps(tag.model_dump())) + assert _is_grammar_accept_string(grammar, _hermes_call("crm__lookup")) + + +def test_harmony_structural_tag_keeps_builtin_tools(): + """Harmony can call builtin tools, so they still shape its grammar.""" + request = ResponsesRequest.model_validate( + {"input": "hi", "tools": [{"type": "web_search"}]} + ) + + tag = get_model_structural_tag( + model="harmony", + tools=request.tools, + tool_choice="required", + reasoning=False, + ) + + assert tag is not None + + +def test_glm47_default_level_allows_builtin_only_tools( + monkeypatch: pytest.MonkeyPatch, +): + """GLM-4.7's default FUNCTION level must not reject builtin-only tools.""" + monkeypatch.setattr(envs, "VLLM_ENFORCE_STRICT_TOOL_CALLING", True) + + class TestParser(DelegatingParser): + tool_parser_cls = Glm47MoeModelToolParser + + request = ResponsesRequest.model_validate({"input": "hi", "tools": [_MCP_TOOL]}) + parser = TestParser(MagicMock(), tools=None) + parser._reasoning_parser = MagicMock(adjust_request=lambda request: request) + + assert parser.adjust_request(request).structured_outputs is None + + def test_get_model_structural_tag_supports_vllm_hermes( sample_tools_strict: list[ChatCompletionToolsParam], ): @@ -723,6 +851,44 @@ def test_step3p5_forced_tool_choice_round_trips_native_xml( ] +@pytest.mark.parametrize( + "tool_choice", + ["required", {"type": "function", "function": {"name": "get_weather"}}], +) +def test_longcat_forced_tool_choice_round_trips_native_tags( + sample_tools: list[ChatCompletionToolsParam], tool_choice +): + """Forced LongCat calls must use , not Hermes tags.""" + + class LongcatParser(DelegatingParser): + tool_parser_cls = LongcatFlashToolParser + + request = ChatCompletionRequest( + messages=[], + model="m", + tools=[tool.model_dump(exclude_none=True) for tool in sample_tools], + tool_choice=tool_choice, + ) + parser = LongcatParser(MagicMock(), tools=sample_tools) + request = parser.adjust_request(request) + + grammar = Grammar.from_structural_tag(request.structured_outputs.structural_tag) + # Rendered exactly as the LongCat-Flash chat template renders tool calls. + output = ( + '\n{"name": "get_weather", ' + '"arguments": {"city": "Dallas"}}\n' + ) + assert _is_grammar_accept_string(grammar, output) + assert not _is_grammar_accept_string(grammar, output.replace("longcat_", "")) + + _, content, tool_calls = parser.parse(output, request, enable_auto_tools=True) + + assert not content + assert [(call.name, json.loads(call.arguments)) for call in tool_calls] == [ + ("get_weather", {"city": "Dallas"}) + ] + + def test_tool_parsers_without_structural_tag_support_required_and_named(): class NonStructuralTagToolParser(ToolParser): pass diff --git a/tests/tool_use/test_tool_choice_required.py b/tests/tool_use/test_tool_choice_required.py index a859659417..be58ffa83a 100644 --- a/tests/tool_use/test_tool_choice_required.py +++ b/tests/tool_use/test_tool_choice_required.py @@ -5,7 +5,11 @@ from copy import deepcopy import pytest import regex as re -from openai.types.responses import FunctionTool, ToolChoiceFunction, WebSearchTool +from openai.types.responses import ( + FunctionTool, + ToolChoiceFunction, + WebSearchTool, +) from pydantic import TypeAdapter from vllm.entrypoints.openai.chat_completion.protocol import ( @@ -13,6 +17,8 @@ from vllm.entrypoints.openai.chat_completion.protocol import ( ChatCompletionNamedToolChoiceParam, ChatCompletionToolsParam, ) +from vllm.entrypoints.openai.responses.protocol import ResponsesRequest +from vllm.exceptions import VLLMValidationError from vllm.tool_parsers.streaming import extract_required_tool_call_streaming from vllm.tool_parsers.utils import ( find_tool_properties, @@ -289,45 +295,61 @@ def test_structured_outputs_json_without_parameters( ) -def _collect_required_tool_streaming_json(output_json: str, delta_len: int) -> str: +def _stream_required_tool_calls( + output_json: str, + deltas: list[str], + tool_call_id_type: str = "random", + tool_call_idx: int | None = None, +) -> list[dict]: + """Feed ``deltas`` (which concatenate to ``output_json``) through the + required-tool streaming helper and rebuild the calls per array index, + checking that every index gets exactly one id/name chunk first.""" + assert "".join(deltas) == output_json previous_text = "" - function_name_returned = False - messages = [] - for i in range(0, len(output_json), delta_len): - delta_text = output_json[i : i + delta_len] + calls: dict[int, dict] = {} + for delta_text in deltas: current_text = previous_text + delta_text - - delta_message, function_name_returned = extract_required_tool_call_streaming( + delta_message, _ = extract_required_tool_call_streaming( previous_text=previous_text, current_text=current_text, - delta_text=delta_text, - function_name_returned=function_name_returned, - tool_call_idx=None, - tool_call_id_type="random", + tool_call_idx=tool_call_idx, + tool_call_id_type=tool_call_id_type, ) - - if delta_message: - messages.append(delta_message) - previous_text = current_text + if delta_message is None: + continue + assert delta_message.tool_calls + for tc in delta_message.tool_calls: + assert tc.function is not None + if tc.id is not None: + assert tc.index not in calls, "id/name sent twice for one index" + assert tc.type == "function" + assert tc.function.name + calls[tc.index] = { + "id": tc.id, + "name": tc.function.name, + "arguments": tc.function.arguments or "", + } + else: + assert tc.index in calls, "arguments before id/name" + assert tc.function.name is None + assert tc.function.arguments + calls[tc.index]["arguments"] += tc.function.arguments + assert list(calls) == list(range(len(calls))) + return [calls[i] for i in range(len(calls))] - assert len(messages) > 0 - combined_messages = "[" - for message in messages: - fn = message.tool_calls[0].function - assert fn is not None - if fn.name: - if len(combined_messages) > 1: - combined_messages += "}," +def _fixed_len_deltas(text: str, delta_len: int) -> list[str]: + return [text[i : i + delta_len] for i in range(0, len(text), delta_len)] - combined_messages += ( - '{"name": "' + fn.name + '", "parameters": ' + (fn.arguments or "") - ) - else: - combined_messages += fn.arguments or "" - combined_messages += "}]" - return combined_messages + +def _assert_streams_to(output: list[dict], deltas: list[str]) -> None: + calls = _stream_required_tool_calls(json.dumps(output), deltas) + assert [ + {"name": c["name"], "parameters": json.loads(c["arguments"])} for c in calls + ] == output + for call, expected in zip(calls, output): + assert call["arguments"] == json.dumps(expected["parameters"]) @pytest.mark.parametrize("output", VALID_TOOLS) @@ -337,11 +359,7 @@ def test_streaming_output_valid(output, empty_params, delta_len): output = deepcopy(output) if empty_params: output = [{"name": o["name"], "parameters": {}} for o in output] - output_json = json.dumps(output) - - combined_messages = _collect_required_tool_streaming_json(output_json, delta_len) - assert json.loads(combined_messages) == output - assert json.dumps(json.loads(combined_messages)) == output_json + _assert_streams_to(output, _fixed_len_deltas(json.dumps(output), delta_len)) @pytest.mark.parametrize( @@ -352,22 +370,78 @@ def test_streaming_output_valid(output, empty_params, delta_len): "a }} b", 'a " } b', r"a \ } b", + # Text that looks like the end of one call and the start of another. + '"}}, {"name": "get_forecast", "parameters": {"city": "x', ], ) @pytest.mark.parametrize("delta_len", [1, 2, 3, 8, 9999]) def test_streaming_output_valid_with_braces_in_string(city, delta_len): - output = [{"name": "get_current_weather", "parameters": {"city": city}}] - output_json = json.dumps(output) - combined_messages = _collect_required_tool_streaming_json(output_json, delta_len) - assert json.loads(combined_messages) == output - assert json.dumps(json.loads(combined_messages)) == output_json + output = [ + {"name": "get_current_weather", "parameters": {"city": city}}, + {"name": "get_forecast", "parameters": {"city": city, "days": 1}}, + ] + _assert_streams_to(output, _fixed_len_deltas(json.dumps(output), delta_len)) def test_streaming_output_valid_with_trailing_extra_data(): output = [{"name": "get_current_weather", "parameters": {"city": "Vienna"}}] output_json = json.dumps(output) + "\nDONE" - combined_messages = _collect_required_tool_streaming_json(output_json, delta_len=3) - assert json.loads(combined_messages) == output + calls = _stream_required_tool_calls(output_json, _fixed_len_deltas(output_json, 3)) + assert [ + {"name": c["name"], "parameters": json.loads(c["arguments"])} for c in calls + ] == output + + +TWO_CALLS = [ + {"name": "get_current_weather", "parameters": {"city": "Dallas"}}, + {"name": "get_forecast", "parameters": {"city": "Dallas", "days": 3}}, +] + + +def _split_at(text: str, cuts: list[int]) -> list[str]: + bounds = [0, *cuts, len(text)] + return [text[a:b] for a, b in zip(bounds, bounds[1:])] + + +def test_streaming_multiple_calls_completed_in_one_delta(): + """Regression test for #60351: a delta that completes more than one + call must stream every call, not only the last one.""" + text = json.dumps(TWO_CALLS) + _assert_streams_to(TWO_CALLS, [text[:1], text[1:]]) + _assert_streams_to(TWO_CALLS, [text]) + + +def test_streaming_delta_crossing_call_boundary(): + """A delta that closes one call and opens the next must attribute the + closing argument text to the first call and still announce the next.""" + text = json.dumps(TWO_CALLS) + boundary = text.index("}, {") + 1 + name_end = text.index('"get_forecast"') + 6 + for cuts in ( + [boundary - 1, name_end], # '}, {"name": "get_f' in one delta + [boundary - 2, boundary + 2], # '"}}, {' then the rest + [boundary + 2], # opening brace of the next call ends a delta + ): + _assert_streams_to(TWO_CALLS, _split_at(text, cuts)) + + +def test_streaming_invalid_name_string_is_never_announced(): + """A closed name that is not a valid JSON string must not be streamed + (the non-streaming path rejects such an array as well).""" + text = '[{"name": "bad\\x", "parameters": {"city": "x"}}]' + assert _stream_required_tool_calls(text, _fixed_len_deltas(text, 4)) == [] + + +def test_streaming_tool_call_idx_increments_per_started_call(): + """kimi_k2 ids number the calls started in one delta consecutively.""" + text = json.dumps(TWO_CALLS) + calls = _stream_required_tool_calls( + text, [text], tool_call_id_type="kimi_k2", tool_call_idx=5 + ) + assert [c["id"] for c in calls] == [ + "functions.get_current_weather:5", + "functions.get_forecast:6", + ] FUNCTION_TOOL = FunctionTool( @@ -403,6 +477,106 @@ class TestNonFunctionToolsSkipped: assert len(any_of) == 1 assert any_of[0]["properties"]["name"]["enum"] == ["get_weather"] + def test_get_json_schema_rejects_only_non_function_tools(self): + # An empty anyOf would compile to a grammar no output can satisfy. + with pytest.raises(VLLMValidationError, match="no function tool"): + get_json_schema_from_tools(tools=[WEB_SEARCH_TOOL], tool_choice="required") + + +class TestMalformedToolSchemaDefs: + """Malformed `$defs` in caller-supplied tool parameters is a 400, not a 500.""" + + @staticmethod + def _tool(params: dict) -> ChatCompletionToolsParam: + return TypeAdapter(ChatCompletionToolsParam).validate_python( + { + "type": "function", + "function": { + "name": "get_weather", + "description": "Get the weather", + "parameters": params, + }, + } + ) + + @pytest.mark.parametrize( + "defs", + [ + pytest.param(None, id="null"), + pytest.param([], id="list"), + pytest.param("nope", id="string"), + pytest.param(1, id="int"), + ], + ) + def test_non_object_defs_is_a_client_error(self, defs): + tools = [ + self._tool( + { + "type": "object", + "properties": {"a": {"type": "string"}}, + "$defs": defs, + } + ) + ] + + with pytest.raises(VLLMValidationError) as exc_info: + get_json_schema_from_tools(tools=tools, tool_choice="required") + + assert exc_info.value.parameter == "tools" + assert "$defs" in str(exc_info.value) + assert "get_weather" in str(exc_info.value) + + def test_duplicate_def_names_is_a_client_error(self): + """Also caller-caused, and it used to raise a plain `ValueError`.""" + tools = [ + self._tool( + { + "type": "object", + "properties": {"a": {"$ref": "#/$defs/D"}}, + "$defs": {"D": {"type": "string"}}, + } + ), + self._tool( + { + "type": "object", + "properties": {"b": {"$ref": "#/$defs/D"}}, + "$defs": {"D": {"type": "integer"}}, + } + ), + ] + + with pytest.raises(VLLMValidationError) as exc_info: + get_json_schema_from_tools(tools=tools, tool_choice="required") + + assert exc_info.value.parameter == "tools" + assert "multiple schemas" in str(exc_info.value) + + def test_well_formed_defs_still_hoisted(self): + """The type check must not reject the shape it is guarding.""" + tools = [ + self._tool( + { + "type": "object", + "properties": {"a": {"$ref": "#/$defs/D"}}, + "$defs": {"D": {"type": "string"}}, + } + ) + ] + + schema = get_json_schema_from_tools(tools=tools, tool_choice="required") + + assert isinstance(schema, dict) + assert schema["$defs"] == {"D": {"type": "string"}} + + def test_absent_defs_still_works(self): + tools = [ + self._tool({"type": "object", "properties": {"a": {"type": "string"}}}) + ] + + schema = get_json_schema_from_tools(tools=tools, tool_choice="required") + + assert isinstance(schema, dict) + class TestParallelToolCallsConstraint: """`parallel_tool_calls=false` must be enforced by the decoding grammar. @@ -480,3 +654,70 @@ class TestForcedNamedToolChoiceEmptyParams: choice = ToolChoiceFunction(type="function", name="ping") schema = get_json_schema_from_tools(choice, [tool]) assert schema == {"type": "object", "properties": {}} + + +class TestForcedFunctionShortNameAlias: + """Forced choices must use the selected function's parameter schema.""" + + PARAMS = {"type": "object", "properties": {"x": {"type": "integer"}}} + + @pytest.mark.parametrize("plain_first", [True, False]) + @pytest.mark.parametrize( + "namespace,function_name,choice_name", + [ + ("math", "add", "add"), + ("math", "add", "math__add"), + ("math", "vector__add", "vector__add"), + ("math__vector", "vector__add", "vector__add"), + ], + ) + def test_namespace_schema_is_not_shadowed( + self, plain_first, namespace, function_name, choice_name + ): + tools = [ + { + "type": "function", + "name": "other__add", + "parameters": { + "type": "object", + "properties": {"y": {"type": "string"}}, + }, + }, + { + "type": "namespace", + "name": namespace, + "description": "math tools", + "tools": [ + { + "type": "function", + "name": function_name, + "parameters": self.PARAMS, + } + ], + }, + ] + if not plain_first: + tools.reverse() + request = ResponsesRequest.model_validate( + { + "model": "test-model", + "input": "Add the numbers.", + "tools": tools, + "tool_choice": {"type": "function", "name": choice_name}, + } + ) + assert ( + get_json_schema_from_tools(request.tool_choice, request.tools) + == self.PARAMS + ) + + def test_plain_function_double_underscore_name_has_no_phantom_alias(self): + tool = FunctionTool(type="function", name="math__add", parameters=self.PARAMS) + choice = ToolChoiceFunction(type="function", name="add") + with pytest.raises(ValueError, match="has not been passed in `tools`"): + get_json_schema_from_tools(choice, [tool]) + + def test_plain_function_full_name_still_resolves(self): + tool = FunctionTool(type="function", name="math__add", parameters=self.PARAMS) + choice = ToolChoiceFunction(type="function", name="math__add") + assert get_json_schema_from_tools(choice, [tool]) == self.PARAMS diff --git a/tests/transformers_utils/test_config.py b/tests/transformers_utils/test_config.py index 4fd27ba278..9b13e72560 100644 --- a/tests/transformers_utils/test_config.py +++ b/tests/transformers_utils/test_config.py @@ -241,6 +241,7 @@ def test_safetensors_metadata_of_repo_without_safetensors(): ) api = SimpleNamespace( get_safetensors_metadata=get_safetensors_metadata, + list_repo_files=MagicMock(return_value=["pytorch_model.bin"]), snapshot_download=MagicMock(side_effect=LocalEntryNotFoundError("no cache")), ) @@ -250,6 +251,34 @@ def test_safetensors_metadata_of_repo_without_safetensors(): get_safetensors_metadata.assert_called_once() +def test_safetensors_metadata_of_repo_with_a_nonstandard_file_name(): + """`get_safetensors_metadata` only knows `model.safetensors` and its index, + so older checkpoints are read through the file listing instead.""" + from huggingface_hub.errors import NotASafetensorsRepoError + from huggingface_hub.utils import TensorInfo + + weights = "gptq_model-4bit-128g.safetensors" + tensor = TensorInfo(dtype="I32", shape=[5632], data_offsets=(0, 22528)) + parse_safetensors_file_metadata = MagicMock( + return_value=SimpleNamespace(tensors={"layers.0.mlp.down_proj.qweight": tensor}) + ) + api = SimpleNamespace( + get_safetensors_metadata=MagicMock( + side_effect=NotASafetensorsRepoError("not a safetensors repo") + ), + list_repo_files=MagicMock(return_value=["config.json", weights]), + parse_safetensors_file_metadata=parse_safetensors_file_metadata, + ) + + with patch.object(config_module, "hf_api", lambda: api): + metadata = get_safetensors_params_metadata("some/old-gptq-model") + + assert metadata["layers.0.mlp.down_proj.qweight"]["dtype"] == "I32" + parse_safetensors_file_metadata.assert_called_once_with( + "some/old-gptq-model", weights, revision=None + ) + + @pytest.mark.parametrize( ("section_key", "mrope_section", "expected_num_dims"), [ diff --git a/tests/utils_/test_argparse_utils.py b/tests/utils_/test_argparse_utils.py index 89e0f9977d..98f2fe790c 100644 --- a/tests/utils_/test_argparse_utils.py +++ b/tests/utils_/test_argparse_utils.py @@ -4,6 +4,7 @@ import json import os +import sys from argparse import BooleanOptionalAction import pytest @@ -252,6 +253,28 @@ def test_duplicate_dict_args(caplog_vllm, parser): assert "--optimization-level" in caplog_vllm.text +@pytest.mark.skipif( + sys.version_info >= (3, 13), reason="argparse handles deprecated natively" +) +def test_deprecated_arg_scoped_to_subcommand(caplog_vllm, disable_log_dedup): + """A deprecated arg of one subcommand must not warn for a sibling + subcommand that defines a non-deprecated arg with the same dest.""" + parser = FlexibleArgumentParser() + subparsers = parser.add_subparsers(dest="subcommand") + run_batch = subparsers.add_parser("run-batch") + run_batch.add_argument_group("Frontend").add_argument( + "--url", default="0.0.0.0", deprecated=True + ) + chat = subparsers.add_parser("chat") + chat.add_argument("--url", default="http://localhost:8000/v1") + + parser.parse_args(["chat", "--url", "http://localhost:9000/v1"]) + assert "is deprecated" not in caplog_vllm.text + + parser.parse_args(["run-batch", "--url", "1.2.3.4"]) + assert caplog_vllm.text.count("argument 'url' is deprecated") == 1 + + def test_model_specification( parser_with_config, cli_config_file, cli_config_file_with_model ): diff --git a/tests/utils_/test_network_utils.py b/tests/utils_/test_network_utils.py index a6c359e269..7ba3d6ae3e 100644 --- a/tests/utils_/test_network_utils.py +++ b/tests/utils_/test_network_utils.py @@ -13,6 +13,7 @@ from vllm.utils.network_utils import ( get_open_ports_list, get_tcp_uri, join_host_port, + make_zmq_listener, make_zmq_path, make_zmq_socket, split_host_port, @@ -205,6 +206,52 @@ def test_make_zmq_socket_ipv6(): ctx.term() +@pytest.mark.parametrize("scheme", ["tcp", "ipc"]) +def test_make_zmq_socket_adopts_listener(tmp_path, scheme): + path = ( + "tcp://127.0.0.1:0" if scheme == "tcp" else f"ipc://{tmp_path}/inherited.sock" + ) + listener = make_zmq_listener(path, zmq.ROUTER) + ctx = zmq.Context() + router = dealer = None + try: + router = make_zmq_socket( + ctx, + listener.address, + zmq.ROUTER, + bind=True, + listener=listener.socket, + ) + assert listener.socket.fileno() == -1 + dealer = make_zmq_socket( + ctx, listener.address, zmq.DEALER, bind=False, identity=b"client" + ) + dealer.send(b"ping") + assert router.recv_multipart() == [b"client", b"ping"] + finally: + if router is not None: + router.close(linger=0) + if dealer is not None: + dealer.close(linger=0) + ctx.term() + listener.cleanup() + + +def test_make_zmq_listener_failed_ipc_bind_keeps_existing_path(tmp_path): + path = tmp_path / "taken.sock" + owner = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) + owner.bind(str(path)) + owner.listen() + try: + with pytest.raises(OSError, match="Address already in use"): + make_zmq_listener(f"ipc://{path}", zmq.ROUTER) + + with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as client: + client.connect(str(path)) + finally: + owner.close() + + def test_make_zmq_path(): assert make_zmq_path("tcp", "127.0.0.1", 5555) == "tcp://127.0.0.1:5555" assert make_zmq_path("tcp", "::1", 5555) == "tcp://[::1]:5555" diff --git a/tests/utils_/test_system_utils.py b/tests/utils_/test_system_utils.py index 5ef55877a7..55ed35eeb5 100644 --- a/tests/utils_/test_system_utils.py +++ b/tests/utils_/test_system_utils.py @@ -1,11 +1,18 @@ # SPDX-License-Identifier: Apache-2.0 # SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import contextlib +import io import os +import sys import tempfile from pathlib import Path -from vllm.utils.system_utils import _maybe_force_spawn, unique_filepath +from vllm.utils.system_utils import ( + _maybe_force_spawn, + suppress_stdout, + unique_filepath, +) def test_unique_filepath(): @@ -25,3 +32,19 @@ def test_numa_bind_forces_spawn(monkeypatch): monkeypatch.setattr("sys.argv", ["vllm", "serve", "--numa-bind"]) _maybe_force_spawn() assert os.environ["VLLM_WORKER_MULTIPROC_METHOD"] == "spawn" + + +def test_suppress_stdout_when_sys_stdout_has_no_fd(capfd): + with contextlib.redirect_stdout(io.StringIO()), suppress_stdout(): + os.write(1, b"c library output\n") + assert "c library output" not in capfd.readouterr().out + + +def test_suppress_stdout_keeps_stderr_when_sys_stdout_is_stderr(capfd, monkeypatch): + monkeypatch.setattr(sys, "stdout", sys.stderr) + with suppress_stdout(): + os.write(1, b"c library output\n") + os.write(sys.stderr.fileno(), b"error message\n") + out, err = capfd.readouterr() + assert "c library output" not in out + assert "error message" in err diff --git a/tests/utils_/test_torch_utils.py b/tests/utils_/test_torch_utils.py index 79dd3e320a..7294253462 100644 --- a/tests/utils_/test_torch_utils.py +++ b/tests/utils_/test_torch_utils.py @@ -14,6 +14,7 @@ from vllm.utils.torch_utils import ( is_lossless_cast, is_quantized_kv_cache, set_default_torch_dtype, + set_random_seed, set_torch_threads_for_runtime, startup_omp_num_threads, ) @@ -217,3 +218,17 @@ def test_async_tensor_h2d_staging(device): async_tensor_h2d([1, 2, 3], device=device, dtype=torch.int32), torch.tensor([1, 2, 3], dtype=torch.int32, device=device), ) + + +def test_set_random_seed_differs_per_data_parallel_index(): + """Unseeded requests draw from the global RNGs; DP engines sharing the engine + seed must get distinct, reproducible streams, and index 0 keeps the seed.""" + + def draws(*args): + set_random_seed(*args) + return np.random.randint(2**31, size=4).tolist(), torch.rand(4).tolist() + + assert draws(42, 0) == draws(42) + assert draws(42, 1) == draws(42, 1) + assert len({str(draws(42, i)) for i in range(4)}) == 4 + assert draws(42, 1) != draws(43, 1) diff --git a/tests/v1/attention/test_attention_backends_selection.py b/tests/v1/attention/test_attention_backends_selection.py index 2695517e14..1d89027317 100644 --- a/tests/v1/attention/test_attention_backends_selection.py +++ b/tests/v1/attention/test_attention_backends_selection.py @@ -13,7 +13,11 @@ from vllm.model_executor.layers.mamba.linear.minimax_linear_attn import ( ) from vllm.model_executor.layers.mamba.mamba_mixer import MambaMixer from vllm.model_executor.layers.mamba.mamba_mixer2 import MambaMixer2 +from vllm.model_executor.layers.mamba.mamba_utils import MambaStateShapeCalculator from vllm.model_executor.layers.mamba.short_conv import ShortConv +from vllm.model_executor.models.falcon_h1 import FalconH1ForCausalLM +from vllm.model_executor.models.granitemoehybrid import GraniteMoeHybridForCausalLM +from vllm.model_executor.models.zamba2 import Zamba2ForCausalLM from vllm.v1.attention.backends.linear_attn import LinearAttentionBackend from vllm.v1.attention.backends.mamba1_attn import Mamba1AttentionBackend from vllm.v1.attention.backends.mamba2_attn import Mamba2AttentionBackend @@ -47,6 +51,45 @@ def test_replayssm_does_not_reserve_speculative_state_blocks( assert spec.num_speculative_blocks == expected_blocks +@pytest.mark.parametrize( + "model_cls", [FalconH1ForCausalLM, GraniteMoeHybridForCausalLM, Zamba2ForCausalLM] +) +def test_mamba2_config_state_shape_includes_speculative_tokens(model_cls): + """The config-time shape sizes the padded Mamba page, so it must match + MambaMixer2.get_state_shape, whose conv state grows with spec tokens.""" + hf_config = SimpleNamespace( + hidden_size=128, + mamba_expand=2, + mamba_d_ssm=None, + mamba_n_groups=1, + mamba_ngroups=1, + mamba_n_heads=8, + n_mamba_heads=8, + mamba_d_head=32, + mamba_headdim=32, + mamba_d_state=16, + mamba_d_conv=4, + ) + vllm_config = SimpleNamespace( + model_config=SimpleNamespace(hf_config=hf_config), + parallel_config=SimpleNamespace(tensor_parallel_size=1), + num_speculative_tokens=2, + ) + + assert model_cls.get_mamba_state_shape_from_config( + vllm_config + ) == MambaStateShapeCalculator.mamba2_state_shape( + tp_world_size=1, + intermediate_size=256, + n_groups=1, + num_heads=8, + head_dim=32, + state_size=16, + conv_kernel=4, + num_spec=2, + ) + + @pytest.mark.parametrize( "layer_class, init_kwargs, expected_backend, expected_mamba_type", [ diff --git a/tests/v1/attention/test_deepseek_v4_swa_visible.py b/tests/v1/attention/test_deepseek_v4_swa_visible.py index 7a74ce8c81..8f08929549 100644 --- a/tests/v1/attention/test_deepseek_v4_swa_visible.py +++ b/tests/v1/attention/test_deepseek_v4_swa_visible.py @@ -302,8 +302,13 @@ def combine_case( with_image: bool, replay_starts: list[int] | None = None, combine_fn=combine_topk_swa_indices, + in_range_topk: bool = False, ): - """Run combine_topk_swa_indices and return (indices, lens, expected).""" + """Run combine_topk_swa_indices and return (indices, lens, expected). + + ``in_range_topk`` draws top-k indices below N, for combiners that drop + out-of-range ones. + """ device = torch.device("cuda") num_reqs = len(seq_lens) replay_starts = replay_starts or [0] * num_reqs @@ -326,7 +331,11 @@ def combine_case( M = N + int(gather_lens.max()) + 8 gen = torch.Generator(device="cpu").manual_seed(0) topk_indices = torch.randint( - 0, 4096, (num_tokens, max(topk, 1)), generator=gen, dtype=torch.int32 + 0, + N if in_range_topk else 4096, + (num_tokens, max(topk, 1)), + generator=gen, + dtype=torch.int32, ).to(device) topk_indices = topk_indices[:, : max(topk, 1)] @@ -415,24 +424,26 @@ def test_combine_topk_swa_with_image_spans(cfg): @pytest.mark.skipif(not torch.cuda.is_available(), reason="requires CUDA") @pytest.mark.parametrize("cfg", COMBINE_CASES) -def test_v41_combine_topk_swa_stops_at_replay_start(cfg): +@pytest.mark.parametrize("query_len", [24, 20]) +def test_v41_combine_topk_swa_stops_at_replay_start(cfg, query_len): """SWA bounded replay: the gathered buffer starts at replay_start, so the window never indexes below it.""" from vllm.models.deepseek_v41.common.ops.cache_utils import ( combine_topk_swa_indices as combine_v41, ) - # Request 0 replays [16, 40): the windows of its first rows would - # otherwise reach below 16. + # Request 0 replays from 16: a 24-token chunk starts there, a 20-token + # chunk leaves only 4 context tokens above it (fewer than the window). indices, lens, rows, exp_lens = combine_case( cfg["compress_ratio"], cfg["topk"], seq_lens=[40, 12], - query_lens=[24, 12], + query_lens=[query_len, 12], spans=[[], []], with_image=False, replay_starts=[16, 0], combine_fn=combine_v41, + in_range_topk=True, ) assert lens.cpu().tolist() == exp_lens assert indices.cpu().tolist() == rows diff --git a/tests/v1/attention/test_dspark_noncausal_sparse_mla.py b/tests/v1/attention/test_dspark_noncausal_sparse_mla.py index 3d4af17413..7cd7863b06 100644 --- a/tests/v1/attention/test_dspark_noncausal_sparse_mla.py +++ b/tests/v1/attention/test_dspark_noncausal_sparse_mla.py @@ -55,7 +55,6 @@ if not current_platform.is_cuda(): ) from vllm.utils.math_utils import cdiv -from vllm.v1.attention.backend import AttentionCGSupport from vllm.v1.attention.backends.mla.flashinfer_mla_sparse import ( FlashInferMLASparseTRTLLMBackend, ) @@ -122,6 +121,7 @@ def _run_sparse_backend_vs_sdpa( qk_nope_head_dim: int = 128, v_head_dim: int = 128, stale_cpu_query_lens: list[int] | None = None, + capture_graph: bool = False, ) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]: """Run a sparse-MLA backend with the given per-token indices and compute a dense per-token SDPA reference over the SAME indices. @@ -394,9 +394,25 @@ def _run_sparse_backend_vs_sdpa( metadata.num_actual_tokens, num_heads * v_head_dim, dtype=dtype, device=device ) with torch.inference_mode(): + impl.record_logical_topk_ready() backend_output = mock_layer.forward_impl( query_vllm, kv_c_vllm, k_pe_vllm, kv_cache, metadata, out_buffer ) + if capture_graph: + device_boundaries = common_attn_metadata.query_start_loc.clone() + common_attn_metadata.query_start_loc.copy_( + common_attn_metadata.query_start_loc_cpu + ) + metadata = builder.build(0, common_attn_metadata) + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + impl.record_logical_topk_ready() + backend_output = mock_layer.forward_impl( + query_vllm, kv_c_vllm, k_pe_vllm, kv_cache, metadata, out_buffer + ) + common_attn_metadata.query_start_loc.copy_(device_boundaries) + builder.build(0, common_attn_metadata) + graph.replay() return backend_output, sdpa_reference, causal_reference @@ -423,22 +439,19 @@ def _skip_if_backend_unavailable(backend_cls, kv_cache_dtype: str, block_size: i pytest.skip("FlashInferMLASparseTRTLLMBackend requires SM 10.x capability") -def test_flashinfer_sparse_mla_adaptive_varlen_matches_sdpa( +@pytest.mark.parametrize("query_lens", [[1, 7, 3, 5], [1, 7, 8, 0]]) +def test_flashinfer_sparse_mla_adaptive_varlen_graph_matches_sdpa( default_vllm_config, dist_init, workspace_init, + query_lens, ): - """Adaptive request boundaries must drive SM100 sparse index conversion.""" + """Replay uses device boundaries, including empty padding, despite stale CPU.""" backend_cls = FlashInferMLASparseTRTLLMBackend _skip_if_backend_unavailable(backend_cls, "fp8", 64) - assert ( - backend_cls.get_builder_cls().get_cudagraph_support(None, None) - == AttentionCGSupport.ALWAYS - ) device = torch.device(DEVICE_TYPE) seq_lens = [257, 270, 265, 276] - query_lens = [1, 7, 3, 5] sparse_indices = _build_dspark_noncausal_indices( seq_lens, query_lens, @@ -459,6 +472,7 @@ def test_flashinfer_sparse_mla_adaptive_varlen_matches_sdpa( qk_nope_head_dim=192, v_head_dim=256, stale_cpu_query_lens=[4, 4, 4, 4], + capture_graph=True, ) torch.testing.assert_close( diff --git a/tests/v1/attention/test_flashmla_nope_sm90_backend_selection.py b/tests/v1/attention/test_flashmla_nope_sm90_backend_selection.py index db2c1becc4..6523588769 100644 --- a/tests/v1/attention/test_flashmla_nope_sm90_backend_selection.py +++ b/tests/v1/attention/test_flashmla_nope_sm90_backend_selection.py @@ -7,9 +7,9 @@ v_head_dim=256, kv_lora_rank=512 -> head_size 512. These tests pin the selection contract only (no GPU required): * ``get_supported_head_sizes`` advertises 512 so the backend is a candidate; -* ``supports_combination`` admits exactly two shapes for 512 -- a quantized - DS-MLA cache format (zero-padded 576/656B envelope) and a rope-free bf16 - cache on SM90 -- and rejects everything else; +* ``supports_combination`` admits 512 only for rope-free models on SM90, with + a bf16 cache or an fp8_ds_mla cache (zero RoPE slot), and rejects + everything else; * adding 512 must NOT reorder the SM100 priority list for bf16, which is why the bf16 arm is restricted to SM90. """ @@ -19,6 +19,7 @@ from types import SimpleNamespace import pytest import torch +from vllm.platforms import current_platform from vllm.platforms.interface import DeviceCapability from vllm.v1.attention.backends.mla.flashmla_sparse import ( QUANTIZED_DS_MLA_CACHE_FORMATS, @@ -29,7 +30,7 @@ SM90 = DeviceCapability(major=9, minor=0) SM100 = DeviceCapability(major=10, minor=0) BF16_CACHE_DTYPES = (None, "auto", "bfloat16", "float16") -QUANTIZED_CACHE_DTYPES = ("fp8_ds_mla", "nvfp4_ds_mla") +NOPE_SM90_CACHE_DTYPES = (*BF16_CACHE_DTYPES, "fp8_ds_mla") def _supports_combination(head_size, kv_cache_dtype, capability): @@ -86,34 +87,26 @@ def test_quantized_ds_mla_formats_are_the_envelope_set(): assert frozenset({"fp8_ds_mla", "nvfp4_ds_mla"}) == QUANTIZED_DS_MLA_CACHE_FORMATS -@pytest.mark.parametrize("kv_cache_dtype", QUANTIZED_CACHE_DTYPES) -def test_nope_512_quantized_rejected_here(kv_cache_dtype): - # Quantized DS-MLA NoPE-512 is served by the zero-padded 576/656B - # envelope, wired up in a separate change; this backend must reject it. - reason = _supports_combination(512, kv_cache_dtype, SM90) - assert reason is not None - - -@pytest.mark.parametrize("kv_cache_dtype", BF16_CACHE_DTYPES) -def test_nope_512_bf16_sm90_rope_free_accepted(kv_cache_dtype, rope_free_model): +@pytest.mark.parametrize("kv_cache_dtype", NOPE_SM90_CACHE_DTYPES) +def test_nope_512_sm90_rope_free_accepted(kv_cache_dtype, rope_free_model): assert _supports_combination(512, kv_cache_dtype, SM90) is None -@pytest.mark.parametrize("kv_cache_dtype", BF16_CACHE_DTYPES) -def test_nope_512_bf16_sm90_rope_carrying_rejected(kv_cache_dtype, rope_carrying_model): +@pytest.mark.parametrize("kv_cache_dtype", NOPE_SM90_CACHE_DTYPES) +def test_nope_512_sm90_rope_carrying_rejected(kv_cache_dtype, rope_carrying_model): reason = _supports_combination(512, kv_cache_dtype, SM90) assert reason is not None and "rope-free" in reason -@pytest.mark.parametrize("kv_cache_dtype", BF16_CACHE_DTYPES) -def test_nope_512_bf16_sm100_rejected(kv_cache_dtype, rope_free_model): +@pytest.mark.parametrize("kv_cache_dtype", NOPE_SM90_CACHE_DTYPES) +def test_nope_512_sm100_rejected(kv_cache_dtype, rope_free_model): # The SM90 restriction is what keeps the SM100 priority order unchanged. reason = _supports_combination(512, kv_cache_dtype, SM100) assert reason is not None -@pytest.mark.parametrize("kv_cache_dtype", ("fp8", "fp8_e4m3")) -def test_nope_512_plain_fp8_rejected(kv_cache_dtype, rope_free_model): +@pytest.mark.parametrize("kv_cache_dtype", ("fp8", "fp8_e4m3", "nvfp4_ds_mla")) +def test_nope_512_other_quantized_rejected(kv_cache_dtype, rope_free_model): reason = _supports_combination(512, kv_cache_dtype, SM90) assert reason is not None @@ -174,3 +167,15 @@ def test_sm100_bf16_512_priority_unchanged(num_heads): def test_sm100_576_priority_unchanged(): order = _sparse_order("auto", 576, num_heads=32) assert order[0] == "FLASHMLA_SPARSE", order + + +@pytest.mark.skipif(not current_platform.is_cuda(), reason="Requires CUDA backends") +def test_sm90_nope_fp8_ds_mla_resolves_to_flashmla(): + """The sparse backends ranked ahead of FLASHMLA_SPARSE for SM90 NoPE-512 + reject fp8_ds_mla, so the auto-selector lands on FlashMLA.""" + from vllm.v1.attention.backends.registry import AttentionBackendEnum + + order = _sparse_order("fp8_ds_mla", 512, num_heads=64, capability=SM90) + for name in order[: order.index("FLASHMLA_SPARSE")]: + backend_cls = AttentionBackendEnum[name].get_class() + assert not backend_cls.supports_kv_cache_dtype("fp8_ds_mla"), name diff --git a/tests/v1/attention/test_gdn_aiter_flydsl.py b/tests/v1/attention/test_gdn_aiter_flydsl.py new file mode 100644 index 0000000000..3de625e936 --- /dev/null +++ b/tests/v1/attention/test_gdn_aiter_flydsl.py @@ -0,0 +1,199 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +from types import SimpleNamespace +from unittest.mock import patch + +import pytest +import torch + +from vllm._aiter_ops import rocm_aiter_ops +from vllm.model_executor.layers.mamba.gdn import qwen_gdn_linear_attn +from vllm.model_executor.layers.mamba.gdn.qwen_gdn_linear_attn import ( + ChunkGatedDeltaRule, + _resolve_gdn_prefill_backend, +) +from vllm.platforms import current_platform + + +def _make_config( + *, + backend: str = "aiter_flydsl", + head_k_dim: int | None = 128, + head_v_dim: int | None = 128, + dtype: torch.dtype = torch.bfloat16, +): + return SimpleNamespace( + additional_config={"gdn_prefill_backend": backend}, + model_config=SimpleNamespace( + dtype=dtype, + hf_text_config=SimpleNamespace( + linear_key_head_dim=head_k_dim, + linear_value_head_dim=head_v_dim, + ), + ), + ) + + +def _resolve_on_rocm(config, *, kernels_available: bool = True): + with ( + patch.object( + qwen_gdn_linear_attn.current_platform, "is_rocm", return_value=True + ), + patch.object( + qwen_gdn_linear_attn.rocm_aiter_ops, + "is_gdn_flydsl_prefill_available", + return_value=kernels_available, + ), + ): + return _resolve_gdn_prefill_backend(config) + + +@pytest.mark.parametrize( + "head_k_dim,head_v_dim,dtype,expected", + [ + (128, 128, torch.bfloat16, "aiter_flydsl"), + (64, 128, torch.bfloat16, "triton"), + (128, 64, torch.bfloat16, "triton"), + (128, 128, torch.float16, "triton"), + ], +) +def test_resolve_aiter_flydsl_gdn_prefill_backend( + head_k_dim: int, + head_v_dim: int, + dtype: torch.dtype, + expected: str, +): + config = _make_config(head_k_dim=head_k_dim, head_v_dim=head_v_dim, dtype=dtype) + + requested, active = _resolve_on_rocm(config) + + assert requested == "aiter_flydsl" + assert active == expected + + +def test_kda_style_model_cannot_select_flydsl(): + """Kimi K3 KDA reports head dims through linear_attn_config, not these. + + Its builder overrides _build_chunk_metadata, which the AITER path skips, + so this pins the reason the two never meet: the resolver turns the model + down before the builder is ever constructed. + """ + config = _make_config(head_k_dim=None, head_v_dim=None) + + assert _resolve_on_rocm(config) == ("aiter_flydsl", "triton") + + +@pytest.mark.parametrize("on_rocm", [True, False], ids=["kernels_missing", "not_rocm"]) +def test_explicit_aiter_flydsl_fails_closed(on_rocm: bool): + """An explicit request that cannot be honoured is a configuration error. + + Falling back silently would report the run as healthy while the requested + kernels were never used. + """ + config = _make_config() + if on_rocm: + with pytest.raises(RuntimeError, match="not importable"): + _resolve_on_rocm(config, kernels_available=False) + return + with ( + patch.object( + qwen_gdn_linear_attn.current_platform, "is_rocm", return_value=False + ), + patch.object( + qwen_gdn_linear_attn.current_platform, "is_cuda", return_value=True + ), + pytest.raises(RuntimeError, match="aiter_flydsl"), + ): + _resolve_gdn_prefill_backend(config) + + +def _flydsl_prefill_unavailable() -> bool: + return ( + not current_platform.is_rocm() + or not rocm_aiter_ops.is_gdn_flydsl_prefill_available() + ) + + +@pytest.mark.skipif( + _flydsl_prefill_unavailable(), + reason="needs ROCm with the AITER FlyDSL GDN prefill kernels installed", +) +@pytest.mark.parametrize( + "state_dtype", [torch.float32, torch.bfloat16], ids=["fp32_state", "bf16_state"] +) +def test_aiter_flydsl_prefill_matches_triton_reference(state_dtype: torch.dtype): + """Run both prefill paths on device and compare. + + The reference is the same AITER entry point with the FlyDSL prepare and K5 + kernels switched off, so the only thing that differs between the two calls + is the kernels this backend exists to select. The state dtype follows + --mamba-ssm-cache-dtype, which may be float32 or bfloat16. + """ + from aiter.ops.triton.gated_delta_net import chunk_gated_delta_rule_opt_vk + + torch.manual_seed(0) + device = torch.device("cuda") + seq_lens = [70, 130] + total_tokens = sum(seq_lens) + num_k_heads, num_v_heads, head_dim = 2, 4, 128 + + # use_qk_l2norm_in_kernel=False matches the prefill path, where the model + # has already normalised q and k. Feeding raw normals instead lets the + # delta rule run away to NaN, which says nothing about either kernel. + q = torch.nn.functional.normalize( + torch.randn(1, total_tokens, num_k_heads, head_dim, device=device), dim=-1 + ).to(torch.bfloat16) + k = torch.nn.functional.normalize( + torch.randn(1, total_tokens, num_k_heads, head_dim, device=device), dim=-1 + ).to(torch.bfloat16) + v = torch.randn( + 1, total_tokens, num_v_heads, head_dim, dtype=torch.bfloat16, device=device + ) + # g is a log-domain decay, so keep it negative or the recurrence diverges + # and the comparison stops saying anything about the kernels. + g = -torch.rand(1, total_tokens, num_v_heads, dtype=torch.float32, device=device) + beta = torch.rand(1, total_tokens, num_v_heads, dtype=torch.float32, device=device) + cu_seqlens = torch.tensor([0, 70, 200], dtype=torch.int32, device=device) + initial_state = torch.randn( + len(seq_lens), + num_v_heads, + head_dim, + head_dim, + device=device, + ).to(state_dtype) + + shared = { + "q": q, + "k": k, + "v": v, + "g": g, + "beta": beta, + "output_final_state": True, + "cu_seqlens": cu_seqlens, + "use_qk_l2norm_in_kernel": False, + } + + reference_out, reference_state = chunk_gated_delta_rule_opt_vk( + initial_state=initial_state.clone(), + state_dtype=state_dtype, + use_chunk_flydsl=False, + use_prepare_flydsl=False, + **shared, + ) + flydsl_out, flydsl_state = ChunkGatedDeltaRule.forward_aiter_flydsl( + None, + initial_state=initial_state.clone(), + aiter_prefill_metadata=rocm_aiter_ops.build_gdn_flydsl_prefill_metadata( + seq_lens, cu_seqlens=cu_seqlens + ), + **shared, + ) + + assert flydsl_state.dtype == state_dtype + torch.testing.assert_close( + flydsl_out.float(), reference_out.float(), rtol=2e-2, atol=2e-2 + ) + torch.testing.assert_close( + flydsl_state.float(), reference_state.float(), rtol=2e-2, atol=2e-2 + ) diff --git a/tests/v1/attention/test_gdn_metadata_builder.py b/tests/v1/attention/test_gdn_metadata_builder.py index cd01fba49e..4b52eb9c09 100644 --- a/tests/v1/attention/test_gdn_metadata_builder.py +++ b/tests/v1/attention/test_gdn_metadata_builder.py @@ -385,7 +385,7 @@ def test_cudagraph_capture_batch_stays_decode_only(): def _create_checkpoint_builder_and_batch( - num_spec: int, prefix_match_unit: int = 16 + num_spec: int, prefix_match_unit: int | None = 16 ) -> tuple[GDNAttentionMetadataBuilder, CommonAttentionMetadata, dict]: vllm_config = create_vllm_config( model_name="Qwen/Qwen3.5-0.8B", @@ -413,6 +413,8 @@ def _create_checkpoint_builder_and_batch( vllm_config=vllm_config, device=DEVICE, ) + vllm_config.cache_config.hash_block_size = prefix_match_unit or 64 + vllm_config.cache_config.cache_hit_alignment_tokens = prefix_match_unit or 64 batch = BatchSpec( seq_lens=[65, 100, 100], query_lens=[3 if num_spec else 1, 100, 99] ) @@ -431,7 +433,9 @@ def _create_checkpoint_builder_and_batch( @pytest.mark.parametrize("num_spec", [0, 3]) -@pytest.mark.parametrize("prefix_match_unit, expected_offset", [(16, 96), (8, 96)]) +@pytest.mark.parametrize( + "prefix_match_unit, expected_offset", [(None, 64), (16, 96), (8, 96)] +) def test_checkpoint_metadata_preserves_non_spec_order( num_spec, prefix_match_unit, expected_offset ): diff --git a/tests/v1/attention/test_gdn_zero_draft_spec_rows.py b/tests/v1/attention/test_gdn_zero_draft_spec_rows.py new file mode 100644 index 0000000000..55955886ce --- /dev/null +++ b/tests/v1/attention/test_gdn_zero_draft_spec_rows.py @@ -0,0 +1,211 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Which path GDN metadata takes when a decode row carries no draft tokens. + +`build_for_cudagraph_capture` derives the draft count from the query length +(`num_decode_draft_tokens = query_len - 1`), so a capture shape of one token +per request describes rows with zero drafts. `GDNAttentionMetadataBuilder` +routes every row with `num_decode_draft_tokens_cpu >= 0` through the +speculative path, zero drafts included: that path is the only one that +applies each row's accepted-token offset to the recurrent state, so a +zero-draft decode row must not fall back to the plain decode path. +""" + +import pytest +import torch + +from tests.v1.attention.utils import ( + BatchSpec, + create_common_attn_metadata, + create_vllm_config, +) +from vllm.config import SpeculativeConfig +from vllm.config.compilation import CUDAGraphMode +from vllm.model_executor.layers.mamba.mamba_utils import MambaStateShapeCalculator +from vllm.v1.attention.backends.gdn_attn import GDNAttentionMetadataBuilder +from vllm.v1.kv_cache_interface import MambaSpec + +BLOCK_SIZE = 16 +DEVICE = torch.device("cpu") +NUM_SPEC = 3 + + +def _builder( + use_spec_decode: bool, + mamba_cache_mode: str = "none", + shapes: tuple[tuple[int, ...], ...] = ((16, 64),), + dtypes: tuple[torch.dtype, ...] = (torch.float16,), +) -> GDNAttentionMetadataBuilder: + vllm_config = create_vllm_config( + model_name="Qwen/Qwen3.5-0.8B", + block_size=BLOCK_SIZE, + ) + vllm_config.compilation_config.cudagraph_mode = CUDAGraphMode.FULL_AND_PIECEWISE + vllm_config.speculative_config = ( + SpeculativeConfig(method="ngram", num_speculative_tokens=NUM_SPEC) + if use_spec_decode + else None + ) + # "none" keeps the block table as-is; the align gather needs columns for + # the speculative blocks, which `_decode_metadata` appends on request. + vllm_config.cache_config.mamba_cache_mode = mamba_cache_mode + mamba_spec = MambaSpec( + block_size=BLOCK_SIZE, + shapes=shapes, + dtypes=dtypes, + num_speculative_blocks=NUM_SPEC, + ) + + return GDNAttentionMetadataBuilder( + kv_cache_spec=mamba_spec, + layer_names=["layer.0"], + vllm_config=vllm_config, + device=DEVICE, + ) + + +def _capture_metadata(builder, query_len: int, batch_size: int = 2): + """Metadata as produced during CUDA-graph capture of a decode shape.""" + batch = BatchSpec( + seq_lens=[64] * batch_size, + query_lens=[query_len] * batch_size, + ) + common = create_common_attn_metadata(batch, BLOCK_SIZE, DEVICE) + return builder.build_for_cudagraph_capture(common) + + +def _decode_metadata(builder, num_accepted: list[int], seq_len: int = 64): + """Metadata for a live decode batch where no row was given drafts. + + Every row schedules a single token, so `num_decode_draft_tokens_cpu` is all + zeros -- the shape the runner produces once `zero_draft_decode_mask` has + promoted zero-draft decode rows out of the `-1` placeholder. + `num_accepted` is what the previous step accepted (bonus token included, + so one is the minimum), which is what decides the state slot. + """ + batch = BatchSpec( + seq_lens=[seq_len] * len(num_accepted), query_lens=[1] * len(num_accepted) + ) + common = create_common_attn_metadata( + batch, BLOCK_SIZE, DEVICE, arange_block_indices=True + ) + if builder.vllm_config.cache_config.mamba_cache_mode == "align": + # Align gathers the last 1 + num_speculative_blocks columns starting at + # the request's current block, so the table has to reach past the end of + # the sequence -- at runtime it is sized by max_model_len, while the + # test helper sizes it by seq_len. + rows, cols = common.block_table_tensor.shape + extra = ( + torch.arange( + rows * NUM_SPEC, dtype=common.block_table_tensor.dtype, device=DEVICE + ).view(rows, NUM_SPEC) + + rows * cols + ) + common.block_table_tensor = torch.cat([common.block_table_tensor, extra], dim=1) + accepted = torch.tensor(num_accepted, dtype=torch.int32, device=DEVICE) + num_decode_draft_tokens_cpu = torch.zeros(len(num_accepted), dtype=torch.int32) + return builder.build(0, common, accepted, num_decode_draft_tokens_cpu) + + +def test_zero_draft_capture_shape_routes_through_the_speculative_path(): + """The one-token capture shape is what breaks backends whose decode kernels + assume a full num_spec + 1 query window: `num_decode_draft_tokens_cpu = + query_len - 1 = 0` for every row, which is `>= 0` and must still take the + speculative path (not the plain decode path that drops the state offset), + with a state tensor sized for the full window even though every row only + supplies a single query token. + + Only `query_len=1` (zero drafts) discriminates this from the plain-decode + path before the fix: for any `query_len > 1`, `num_decode_draft_tokens_cpu` + is already `> 0` and both the pre-fix and post-fix code agree, so this + does not need parametrizing over larger draft counts. + """ + meta = _capture_metadata(_builder(use_spec_decode=True), query_len=1) + + assert meta.num_spec_decodes == 2 + assert meta.num_decodes == 0 + slots = meta.spec_state_indices_tensor + assert slots is not None + # The kernels index columns 0..num_spec of this tensor. + assert slots.shape[1] == NUM_SPEC + 1 + assert int(slots.min()) >= 0 + query_lens = torch.diff(meta.spec_query_start_loc) + assert torch.all(query_lens == 1) + + +def test_plain_decode_without_spec_decode_skips_the_speculative_path(): + """Without a speculative config the builder never builds spec metadata, + so a one-token-per-row decode capture takes the plain decode path. + + Not a regression check for this patch (this branch is untouched by it), + but it is what keeps the test above from passing vacuously: it would also + pass if the builder routed every row through the speculative path + unconditionally, regardless of whether speculative decoding is even + configured. + """ + meta = _capture_metadata(_builder(use_spec_decode=False), query_len=1) + + assert meta.num_spec_decodes == 0 + assert meta.num_decodes == 2 + + +@pytest.mark.parametrize( + "mamba_cache_mode, shapes, dtypes", + [ + ("none", ((16, 64),), (torch.float16,)), + # Kimi-Linear's KDA layers subclass GatedDeltaNetAttention and consume + # this same metadata, and prefix caching (which Kimi enables by + # default) puts the cache in align mode -- so the guarantee has to hold + # for that state shape and mode too. + ( + "align", + MambaStateShapeCalculator.kda_state_shape( + tp_world_size=1, num_heads=4, head_dim=32, num_spec=NUM_SPEC + ), + (torch.bfloat16, torch.bfloat16), + ), + ], + ids=["gdn_none", "kimi_kda_align"], +) +def test_zero_draft_batch_still_applies_the_accepted_token_offset( + mamba_cache_mode, shapes, dtypes +): + """The numeric difference the fix makes, on the batch shape that triggers it. + + The recurrent kernels read their state at column `num_accepted_tokens - 1` + of `spec_state_indices_tensor`. Before the fix, a batch in which *every* + speculative row drafted nothing was pushed onto the non-speculative path, + which passes `non_spec_state_indices_tensor` -- always column 0 -- and + drops `num_accepted_tokens` entirely. A row that accepted drafts on the + previous step wrote its fresh state further along, so column 0 is stale. + """ + accepted = [NUM_SPEC, 1] + meta = _decode_metadata( + _builder( + use_spec_decode=True, + mamba_cache_mode=mamba_cache_mode, + shapes=shapes, + dtypes=dtypes, + ), + accepted, + ) + + assert meta.num_spec_decodes == len(accepted) + assert meta.num_decodes == 0 + # Dropping these is what silently sent the kernels to the wrong slot. + assert meta.num_accepted_tokens is not None + assert meta.num_accepted_tokens.tolist()[: len(accepted)] == accepted + assert meta.non_spec_state_indices_tensor is None + + slots = meta.spec_state_indices_tensor + assert slots is not None + for row, num_accepted in enumerate(accepted): + read = int(slots[row, num_accepted - 1]) + fallback = int(slots[row, 0]) + # Only the row that accepted a draft moved off column 0; the row that + # accepted nothing but its bonus token legitimately stays there. This + # also confirms the columns are genuinely distinct blocks (via + # `arange_block_indices=True` in `_decode_metadata`), not just + # distinct labels for the same one -- otherwise `read != fallback` + # here would mean nothing. + assert (read != fallback) == (num_accepted > 1) diff --git a/tests/v1/attention/test_indexer_deepseek_v4_slot_mapping.py b/tests/v1/attention/test_indexer_deepseek_v4_slot_mapping.py index 8acfc0a334..17dea5d03c 100644 --- a/tests/v1/attention/test_indexer_deepseek_v4_slot_mapping.py +++ b/tests/v1/attention/test_indexer_deepseek_v4_slot_mapping.py @@ -122,6 +122,7 @@ def test_indexer_draft_decode_metadata_update_is_capture_safe(): builder = object.__new__(DeepseekV32IndexerMetadataBuilder) builder.dcp_world_size = 1 builder.compress_ratio = 16 + builder.kernel_block_size = None builder.num_sms = num_sms builder.arange_buffer = torch.arange(num_reqs + 1, dtype=torch.int32, device=device) builder.kv_cache_spec = SimpleNamespace(num_states=64) @@ -140,6 +141,7 @@ def test_indexer_draft_decode_metadata_update_is_capture_safe(): requires_padding=False, schedule_metadata=schedule_metadata, ), + block_table=block_table, ) builder.update_draft_decode_metadata(metadata) @@ -167,6 +169,65 @@ def test_indexer_draft_decode_metadata_update_is_capture_safe(): torch.testing.assert_close(metadata.slot_mapping, expected_slot_mapping) +@pytest.mark.skipif(not torch.cuda.is_available(), reason="requires CUDA") +@pytest.mark.parametrize( + "block_size,kernel_block_size,compress_ratio", [(64, 64, 1), (256, 64, 4)] +) +def test_indexer_draft_decode_refresh_follows_new_batch( + monkeypatch, block_size, kernel_block_size, compress_ratio +): + """Without a prior build(), the refresh must not keep the last batch's state.""" + monkeypatch.setattr(indexer_module, "has_deep_gemm", lambda: False) + device = torch.device("cuda") + num_reqs, width = 3, 8 + factor = block_size // kernel_block_size + + builder = object.__new__(DeepseekV32IndexerMetadataBuilder) + builder.dcp_world_size = 1 + builder.compress_ratio = compress_ratio + builder.kernel_block_size = kernel_block_size + builder.arange_buffer = torch.arange(num_reqs + 1, dtype=torch.int32, device=device) + builder.kv_cache_spec = SimpleNamespace(block_size=block_size, num_states=64) + + def stale(*shape: int) -> torch.Tensor: + return torch.full(shape, -7, dtype=torch.int32, device=device) + + decode = DeepSeekV32IndexerDecodeMetadata( + block_table=stale(num_reqs, width // factor), + seq_lens=stale(num_reqs, 1), + decode_lens=stale(num_reqs), + requires_padding=False, + schedule_metadata=stale(1), + indices=stale(num_reqs), + per_req_decode_lens=stale(num_reqs), + ) + request_block_table = torch.randint( + 0, 1000, (num_reqs, width), dtype=torch.int32, device=device + ) + seq_lens = torch.tensor([17, 33, 65], dtype=torch.int32, device=device) + metadata = DeepseekV32IndexerMetadata( + seq_lens=seq_lens, + max_seq_len=128, + slot_mapping=torch.zeros(num_reqs, dtype=torch.int64, device=device), + num_decodes=num_reqs, + num_decode_tokens=num_reqs, + num_prefills=0, + num_prefill_tokens=0, + decode=decode, + block_table=request_block_table, + ) + + builder.update_draft_decode_metadata(metadata) + + expected_block_table = request_block_table[:, ::factor] // factor + torch.testing.assert_close(decode.block_table, expected_block_table) + torch.testing.assert_close(decode.indices, builder.arange_buffer[:num_reqs]) + assert torch.all(decode.per_req_decode_lens == 1) + assert torch.all(decode.decode_lens == 1) + expected_seq_lens = torch.div(seq_lens, compress_ratio, rounding_mode="floor") + torch.testing.assert_close(decode.seq_lens.flatten(), expected_seq_lens) + + @pytest.mark.skipif(not torch.cuda.is_available(), reason="requires CUDA") @pytest.mark.parametrize( "query_lens", [[1], [0, 257, 1, 0, 3], [6] * 64, [2, 6, 4, 0], [0, 0, 0]] @@ -315,6 +376,9 @@ def test_compressed_slot_mapping_warmup_includes_index_kpool(): config = SimpleNamespace( cache_config=SimpleNamespace(block_size=256), model_config=SimpleNamespace(hf_text_config=SimpleNamespace(index_kpool=32)), + parallel_config=SimpleNamespace( + decode_context_parallel_size=1, cp_kv_cache_interleave_size=1 + ), ) keys = CompressedSlotMappingKernel().get_warmup_keys(config) @@ -343,6 +407,41 @@ def test_compressed_slot_mapping_inherits_padded_token_slots(): assert compressed.tolist() == [-1, -1, -1, -1, -1, 3 * 4 + 2, -1, 3 * 4 + 3] +@pytest.mark.skipif(not torch.cuda.is_available(), reason="requires CUDA") +def test_compressed_slot_mapping_dcp_uses_rank_local_state_positions(): + """Under DCP each rank writes only the states it owns, at the state's + rank-local position.""" + device = torch.device("cuda") + dcp, interleave, ratio, block_size = 2, 4, 2, 4 + num_tokens = 32 + query_start_loc = torch.tensor([0, num_tokens], dtype=torch.int32, device=device) + seq_lens = torch.tensor([num_tokens], dtype=torch.int32, device=device) + block_table = torch.tensor([[5, 7]], dtype=torch.int32, device=device) + pos = torch.arange(num_tokens) + for rank in range(dcp): + owned = (pos // interleave) % dcp == rank + slot_mapping = torch.where(owned, 0, -1).to(torch.int64).to(device) + compressed = get_compressed_slot_mapping( + num_tokens, + slot_mapping, + query_start_loc, + seq_lens, + block_table, + block_size=block_size, + compress_ratio=ratio, + dcp_world_size=dcp, + cp_interleave=interleave, + ) + local_pos = pos // (interleave * dcp) * interleave + pos % interleave + state = local_pos // ratio + expected = torch.where( + owned & ((pos + 1) % ratio == 0), + block_table[0].cpu()[state // block_size] * block_size + state % block_size, + -1, + ) + assert compressed.tolist() == expected.tolist() + + def test_index_conversion_warmup_uses_physical_block_stride(): config = SimpleNamespace( cache_config=SimpleNamespace(block_size=64), @@ -370,6 +469,8 @@ def test_zero_token_pcp_rank_participates_in_compressed_mapping_gather(monkeypat builder.use_pcp = True builder.pcp_world_size = 4 builder.pcp_rank = 0 + builder.dcp_world_size = 1 + builder.cp_kv_cache_interleave_size = 1 builder.kernel_block_size = None builder.kv_cache_spec = SimpleNamespace(block_size=64, num_states=64) builder.compressed_slot_mapping_buffer = torch.zeros(8, dtype=torch.int64) @@ -383,8 +484,11 @@ def test_zero_token_pcp_rank_participates_in_compressed_mapping_gather(monkeypat block_size, compress_ratio, out, + dcp_world_size, + cp_interleave, ): assert num_tokens == 0 + assert dcp_world_size == 1 assert block_size == 64 assert compress_ratio == 4 out.fill_(-1) diff --git a/tests/v1/attention/test_kpool_indexer_block_sizes.py b/tests/v1/attention/test_kpool_indexer_block_sizes.py new file mode 100644 index 0000000000..a737988dde --- /dev/null +++ b/tests/v1/attention/test_kpool_indexer_block_sizes.py @@ -0,0 +1,135 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""CPU tests for storage-block kernel-block selection in prepare_kernel_block_sizes.""" + +from types import SimpleNamespace + +import pytest +import torch + +from vllm.config import VllmConfig, set_current_vllm_config +from vllm.models.glm5next.common.attention import Glm5NextIndexerCache +from vllm.v1.attention.backend import AttentionBackend, MultipleOf +from vllm.v1.attention.backends.mla import indexer as indexer_mod +from vllm.v1.attention.backends.mla.indexer import ( + DeepseekV32IndexerBackend, + KpoolTailBackend, +) +from vllm.v1.attention.backends.mla.rocm_aiter_mla_sparse import ( + ROCMAiterMLASparseBackend, +) +from vllm.v1.kv_cache_interface import MLAAttentionSpec +from vllm.v1.worker.utils import prepare_kernel_block_sizes + +pytestmark = pytest.mark.cpu_test + +# ``index_kpool`` of zai-org/GLM-5.3-Flash: one indexer state pools 4 +# compressed states (hence storage_block_size = page * 4). +INDEX_KPOOL = 4 + +# The hybrid KDA/mamba manager floors at TP8..TP1; 640 is the geometry of the +# original report ("the 640-token block table"). +TP_FLOOR_MANAGER_BLOCKS = [640, 1152, 2176, 4352] + + +def _mock_rocm_platform(monkeypatch: pytest.MonkeyPatch) -> None: + # Platform predicates are mutually exclusive in production. Override both + # so these ROCm policy tests do not inherit CUDA capabilities from the CI + # host running them (same approach as test_deepseek_v4_rocm_adaptive). + monkeypatch.setattr(indexer_mod.current_platform, "is_cuda", lambda: False) + monkeypatch.setattr(indexer_mod.current_platform, "is_rocm", lambda: True) + + +def _kpool_storage_spec(manager_block: int) -> MLAAttentionSpec: + """Real Glm5NextIndexerCache spec for the given cache block size.""" + with set_current_vllm_config(VllmConfig()): + cache = Glm5NextIndexerCache( + head_dim=128, + dtype=torch.bfloat16, + prefix="model.layers.0.indexer.k_cache_probe", + cache_config=SimpleNamespace(block_size=manager_block), + index_kpool=INDEX_KPOOL, + ) + spec = cache.get_kv_cache_spec(VllmConfig()) + assert isinstance(spec, MLAAttentionSpec) + return spec + + +def _prepare( + spec: MLAAttentionSpec, backends: list[type[AttentionBackend]] +) -> list[int]: + kv_cache_config = SimpleNamespace( + kv_cache_groups=[SimpleNamespace(kv_cache_spec=spec)] + ) + attn_groups = [[SimpleNamespace(backend=backend) for backend in backends]] + return prepare_kernel_block_sizes(kv_cache_config, attn_groups) + + +@pytest.mark.parametrize("manager_block", TP_FLOOR_MANAGER_BLOCKS) +def test_prepare_uses_storage_block_for_the_kpool_group( + monkeypatch: pytest.MonkeyPatch, manager_block: int +): + """The bug: on ROCm, kpool groups must get 128/256 - not the 640-4352 + manager block - as their kernel block size. + + Manager-granular selection is what made the block table address manager + blocks while the kpool writer and the index-cache gather read pool pages, + aliasing every cache column past the request's row. + """ + _mock_rocm_platform(monkeypatch) + spec = _kpool_storage_spec(manager_block) + expected_storage = 256 if manager_block % 256 == 0 else 128 + assert spec.storage_block_size == expected_storage + + selected = _prepare( + spec, + [ROCMAiterMLASparseBackend, DeepseekV32IndexerBackend, KpoolTailBackend], + ) + assert selected == [expected_storage] + assert selected != [manager_block] + # The hybrid block-table split stays integral. + assert manager_block % selected[0] == 0 + + +def test_prepare_falls_back_when_storage_block_is_not_supported(): + """No storage block, or a storage block the backends reject (e.g. the + CUDA indexer's exact ``[64]`` vs a 128-token storage block): selection is + `select_common_block_size`'s answer, unchanged from no-fix behavior. + """ + + class Fixed64Backend: + @staticmethod + def get_supported_kernel_block_sizes(): + return [64] + + @staticmethod + def get_name() -> str: + return "FIXED64" + + # Storage block the group's backend does not accept -> backend vote (64). + spec = _kpool_storage_spec(640) + assert spec.storage_block_size == 128 + assert _prepare(spec, [Fixed64Backend]) == [64] + + # No storage block at all (any non-kpool MLA model) -> backend vote. + plain_spec = MLAAttentionSpec( + block_size=640, + num_kv_heads=1, + head_size=128, + dtype=torch.bfloat16, + ) + assert plain_spec.storage_block_size is None + assert _prepare(plain_spec, [Fixed64Backend]) == [64] + + # ...and with backends that accept the manager block, the manager wins, + # exactly as on main. + class AcceptAllBackend: + @staticmethod + def get_supported_kernel_block_sizes(): + return [MultipleOf(1)] + + @staticmethod + def get_name() -> str: + return "ACCEPTALL" + + assert _prepare(plain_spec, [AcceptAllBackend]) == [640] diff --git a/tests/v1/attention/test_kpool_tail_slot_mapping.py b/tests/v1/attention/test_kpool_tail_slot_mapping.py index da25eaefb2..183c63ec2f 100644 --- a/tests/v1/attention/test_kpool_tail_slot_mapping.py +++ b/tests/v1/attention/test_kpool_tail_slot_mapping.py @@ -227,8 +227,10 @@ def test_circular_mapping_matches_generic_for_short_requests(prompt_len): def test_circular_mapping_preserves_padding_and_empty_batch(): - own_blocks = [5, 9] - per_req = [list(range(10)), list(range(12))] + # The last request is a padding request on the null block, which owns no + # tail block and must never be written. + own_blocks = [5, 9, 0] + per_req = [list(range(10)), list(range(12)), [0, 1]] padded_len = sum(len(p) for p in per_req) + 8 positions, qsl, slot_mapping, num_actual, num_reqs = make_batch( per_req, padded_len=padded_len @@ -237,7 +239,8 @@ def test_circular_mapping_preserves_padding_and_empty_batch(): out = circular_tail_slots(slot_mapping, bt, qsl, positions, num_actual, num_reqs) assert out.shape == slot_mapping.shape - assert torch.equal(out[num_actual:], torch.full_like(out[num_actual:], -1)) + pad_start = int(qsl[-2]) + assert torch.equal(out[pad_start:], torch.full_like(out[pad_start:], -1)) empty = circular_tail_slots(slot_mapping, bt, qsl, positions[:0], 0, num_reqs) assert torch.equal(empty, slot_mapping) @@ -430,25 +433,27 @@ def test_interleaved_decode_pollution_legacy_vs_circular(): @pytest.mark.skipif(not torch.cuda.is_available(), reason="needs a CUDA device") @pytest.mark.parametrize( - "per_req,num_actual,padded_len", + "per_req,own_blocks,num_actual,padded_len", [ - ([list(range(10)), list(range(12))], 22, 22), - ([list(range(10)), list(range(12))], 22, 30), - ([[3, 4], [0], [7, 8, 9]], 6, 8), - ([[5]], 1, 1), + ([list(range(10)), list(range(12))], [5, 6], 22, 22), + ([list(range(10)), list(range(12))], [5, 6], 22, 30), + ([[3, 4], [0], [7, 8, 9]], [5, 0, 7], 6, 8), + ([[5]], [5], 1, 1), + ([[0], [0]], [0, 0], 2, 4), # dummy run: every row on the null block ], ) -def test_triton_mapping_matches_cpu(per_req, num_actual, padded_len): +def test_triton_mapping_matches_cpu(per_req, own_blocks, num_actual, padded_len): """The CUDA (Triton) path must match the CPU torch reference, including tokens between the last request boundary and num_actual_tokens (mapped to - the last request) and untouched padding beyond num_actual.""" + the last request), untouched padding beyond num_actual, and PAD for + requests on the null block.""" positions, qsl, slot_mapping, _, num_reqs = make_batch( per_req, padded_len=padded_len ) # Replace the all--1 placeholder slots with sentinel values to check the # padding range is copied through untouched. slot_mapping = torch.arange(padded_len, dtype=torch.int64) + 1000 - bt = make_tail_block_table(list(range(5, 5 + num_reqs))) + bt = make_tail_block_table(own_blocks) ref = circular_tail_slots(slot_mapping, bt, qsl, positions, num_actual, num_reqs) got = circular_tail_slots( @@ -460,6 +465,9 @@ def test_triton_mapping_matches_cpu(per_req, num_actual, padded_len): num_reqs, ) torch.testing.assert_close(got.cpu(), ref) + for req, blk in enumerate(own_blocks): + if blk == 0: + assert (got[int(qsl[req]) : int(qsl[req + 1])] == -1).all() @pytest.mark.skipif(not torch.cuda.is_available(), reason="needs a CUDA device") diff --git a/tests/v1/attention/test_mamba2_prefill_checkpoints.py b/tests/v1/attention/test_mamba2_prefill_checkpoints.py new file mode 100644 index 0000000000..a3968b5a0b --- /dev/null +++ b/tests/v1/attention/test_mamba2_prefill_checkpoints.py @@ -0,0 +1,168 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Internal prefill checkpoints for Mamba2 align-mode prefix caching.""" + +import torch + +from tests.v1.attention.utils import ( + BatchSpec, + create_common_attn_metadata, + create_vllm_config, +) +from vllm.v1.attention.backends.mamba2_attn import ( + Mamba2AttentionMetadataBuilder, +) +from vllm.v1.attention.backends.mamba_attn import BaseMambaAttentionMetadataBuilder +from vllm.v1.attention.backends.utils import NULL_BLOCK_ID +from vllm.v1.kv_cache_interface import MambaSpec + +compute_chunk_metadata = BaseMambaAttentionMetadataBuilder._compute_chunk_metadata + + +def test_no_checkpoint_offsets_preserves_legacy_chunking(): + """Passing no offsets must reproduce the pre-feature chunk layout.""" + num_computed = torch.tensor([0, 300]) + qsl = torch.tensor([0, 700, 1000]) + + cu, seq_idx, last, ckpt = compute_chunk_metadata(256, 2, num_computed, qsl) + + # Row 0: 700 fresh tokens from sequence position 0 -> 256, 256, 188. + # Row 1: 300 new tokens resuming at 300. 300 % 256 == 44, so it finishes + # its partial chunk with 256 - 44 == 212 tokens, then 88. + assert cu == [0, 256, 512, 700, 912, 1000] + assert seq_idx == [0, 0, 0, 1, 1] + assert last == [2, 4] + assert ckpt == [-1, -1] + + +def test_chunking_realigns_to_the_grid_after_a_checkpoint(): + """After the forced break, chunks resume breaking on chunk_size. + + 100 -> then 156 to reach the 256 grid, then full chunks. Without the + realignment the second chunk would span 100..356 and cross a boundary. + """ + num_computed = torch.tensor([0]) + qsl = torch.tensor([0, 700]) + + cu, _, _, _ = compute_chunk_metadata( + 256, 1, num_computed, qsl, checkpoint_offsets_p=[100] + ) + + assert cu == [0, 100, 256, 512, 700] + + +def test_checkpoint_on_a_resumed_request_is_relative_to_the_query(): + """Offsets count from the query start, not the sequence start.""" + num_computed = torch.tensor([300]) + qsl = torch.tensor([0, 500]) + + cu, _, _, ckpt = compute_chunk_metadata( + 256, 1, num_computed, qsl, checkpoint_offsets_p=[212] + ) + + # Row resumes at sequence position 300. 300 % 256 == 44, so the partial + # chunk is 212 tokens and already ends exactly at the checkpoint (query + # offset 212 == sequence position 512) — no extra split needed. + assert cu == [0, 212, 468, 500] + assert ckpt == [0] + + +MAMBA_BLOCK_SIZE = 256 +CHUNK_SIZE = 256 +DEVICE = torch.device("cpu") + + +def _create_mamba2_builder( + mamba_cache_mode: str = "align", + num_prefill_checkpoint_blocks: int = 1, + prefill_checkpoint_alignment: int | None = 1, +) -> Mamba2AttentionMetadataBuilder: + vllm_config = create_vllm_config( + model_name="Qwen/Qwen3.5-0.8B", block_size=MAMBA_BLOCK_SIZE + ) + vllm_config.cache_config.mamba_cache_mode = mamba_cache_mode + # get_mamba_chunk_size() reads `mamba_chunk_size` then `chunk_size`. + vllm_config.model_config.hf_text_config.mamba_chunk_size = CHUNK_SIZE + spec = MambaSpec( + block_size=MAMBA_BLOCK_SIZE, + shapes=((16, 64),), + dtypes=(torch.float16,), + mamba_cache_mode=mamba_cache_mode, + num_prefill_checkpoint_blocks=num_prefill_checkpoint_blocks, + prefill_checkpoint_alignment=prefill_checkpoint_alignment, + ) + builder = Mamba2AttentionMetadataBuilder( + kv_cache_spec=spec, + layer_names=["layer.0"], + vllm_config=vllm_config, + device=DEVICE, + ) + vllm_config.cache_config.hash_block_size = MAMBA_BLOCK_SIZE + vllm_config.cache_config.cache_hit_alignment_tokens = MAMBA_BLOCK_SIZE + return builder + + +def _build(builder, seq_lens, query_lens): + common = create_common_attn_metadata( + BatchSpec(seq_lens=seq_lens, query_lens=query_lens), + MAMBA_BLOCK_SIZE, + DEVICE, + arange_block_indices=True, + ) + # create_common_attn_metadata leaves is_prefilling unset; the builder + # asserts on it. Every row here is a full prefill. + common = common.replace(is_prefilling=torch.ones(len(seq_lens), dtype=torch.bool)) + return builder.build(common_prefix_len=0, common_attn_metadata=common) + + +def test_builder_emits_one_checkpoint_entry_per_prefill_row(): + """Declined rows stay in place, masked, and a chunk ends on the checkpoint.""" + builder = _create_mamba2_builder() + # Row 0 (seq_len 900) checkpoints at 768; row 1 (seq_len 100) is too short + # for a checkpoint at all. + meta = _build(builder, seq_lens=[900, 100], query_lens=[900, 100]) + + assert meta.checkpoint_chunk_idx is not None + assert meta.checkpoint_meta is not None + assert meta.checkpoint_chunk_idx.numel() == 2 + # The exporter masks row 1 off by its zero offset and null block, but + # still loads its chunk index first, so the placeholder must be in range. + assert meta.checkpoint_meta.checkpoint_offsets.tolist() == [768, 0] + assert meta.checkpoint_meta.state_indices[1].item() == NULL_BLOCK_ID + assert meta.checkpoint_chunk_idx[1].item() == 0 + # The checkpoint's token offset is the end of its chunk. Asserting it here + # also proves the chunk split actually landed on the checkpoint. + ckpt_chunk = meta.checkpoint_chunk_idx[0] + assert meta.cu_chunk_seqlen_p[ckpt_chunk + 1].item() == 768 + + +def test_update_block_table_regathers_checkpoint_blocks(): + """Groups sharing a spec reuse one group's metadata via update_block_table. + + Checkpoint destinations are block-table entries, so each group needs its + own: layers of different groups share physical tensors, and reusing the + first group's indices writes every group's checkpoint into its blocks. + """ + builder = _create_mamba2_builder() + meta = _build(builder, seq_lens=[900, 100], query_lens=[900, 100]) + assert meta.checkpoint_meta is not None + + # A second group's table: same shape as the first, disjoint block ids. + other_table = torch.arange(1000, 1016, dtype=torch.int32).reshape(2, 8) + updated = builder.update_block_table( + meta, other_table, torch.zeros(1000, dtype=torch.int64) + ) + + # Row 0 checkpoints in column cdiv(900, 256) - 2 == 2; row 1 declined. + assert updated.checkpoint_meta is not None + assert updated.checkpoint_meta.state_indices.tolist() == [ + other_table[0, 2].item(), + NULL_BLOCK_ID, + ] + + +def test_builder_emits_nothing_outside_align_mode(): + builder = _create_mamba2_builder(mamba_cache_mode="none") + meta = _build(builder, seq_lens=[900], query_lens=[900]) + + assert meta.checkpoint_chunk_idx is None diff --git a/tests/v1/attention/test_mla_backends.py b/tests/v1/attention/test_mla_backends.py index 1c192f3f1c..9c504f31ef 100644 --- a/tests/v1/attention/test_mla_backends.py +++ b/tests/v1/attention/test_mla_backends.py @@ -538,10 +538,9 @@ def create_and_prepopulate_kv_cache( if fp8_attention: if use_fp8_ds_mla: kv_lora_rank = kv_c_contexts[0].shape[-1] - rope_dim = k_pe_contexts[0].shape[-1] # 4 * 4: 4 float32 scale values for 128-element tiles - # 2 * rope_dim: 16-bit RoPE values - kv_entry_size = kv_lora_rank + 4 * 4 + 2 * rope_dim + # 2 * 64: 16-bit RoPE values (zero-filled for NoPE models) + kv_entry_size = kv_lora_rank + 4 * 4 + 2 * 64 elif use_nvfp4_ds_mla: kv_lora_rank = kv_c_contexts[0].shape[-1] rope_dim = k_pe_contexts[0].shape[-1] diff --git a/tests/v1/attention/test_mla_noncausal.py b/tests/v1/attention/test_mla_noncausal.py index c06ded5c13..57d27ecd46 100644 --- a/tests/v1/attention/test_mla_noncausal.py +++ b/tests/v1/attention/test_mla_noncausal.py @@ -19,6 +19,21 @@ class _NonCausalMLAMetadataBuilder(MLACommonMetadataBuilder[MLACommonMetadata]): supports_non_causal_multi_token_decode = True +class _CausalOnlyMLAMetadataBuilder(MLACommonMetadataBuilder[MLACommonMetadata]): + pass + + +class _DraftMLAMetadataBuilder(MLACommonMetadataBuilder[MLACommonMetadata]): + """Triton-like: non-causal decode is supported, DCP draft decode is not.""" + + supports_non_causal_multi_token_decode = True + + +class _DcpCapableMLAMetadataBuilder(MLACommonMetadataBuilder[MLACommonMetadata]): + supports_non_causal_multi_token_decode = True + supports_non_causal_multi_token_dcp = True + + def _metadata( query_start_loc: list[int], num_tokens: int | None = None, @@ -60,6 +75,54 @@ def _builder(marked: bool = True) -> _NonCausalMLAMetadataBuilder: return builder +def _mla_layer(*, non_causal: bool = False) -> SimpleNamespace: + return SimpleNamespace( + non_causal_multi_token_decode=non_causal, + q_lora_rank=None, + kv_lora_rank=512, + qk_nope_head_dim=128, + qk_rope_head_dim=64, + v_head_dim=128, + prefill_backend=SimpleNamespace(clone=lambda: None), + ) + + +def _merged_mla_spec() -> MLAAttentionSpec: + return MLAAttentionSpec( + block_size=64, + num_kv_heads=1, + head_size=576, + dtype=torch.bfloat16, + non_causal_multi_token_decode=True, + ) + + +def _dspark_dcp_vllm_config( + static_forward_context: dict[str, SimpleNamespace], +) -> SimpleNamespace: + return SimpleNamespace( + speculative_config=SimpleNamespace( + method="dspark", num_speculative_tokens=None + ), + parallel_config=SimpleNamespace( + decode_context_parallel_size=2, + prefill_context_parallel_size=1, + cp_kv_cache_interleave_size=1, + ), + compilation_config=SimpleNamespace( + static_forward_context=static_forward_context + ), + model_config=SimpleNamespace( + dtype=torch.bfloat16, + max_model_len=128, + get_num_attention_heads=lambda _parallel_config: 16, + ), + cache_config=SimpleNamespace(cache_dtype="auto", block_size=64), + scheduler_config=SimpleNamespace(max_num_seqs=1), + attention_config=SimpleNamespace(use_prefill_query_quantization=False), + ) + + def test_group_capability_keeps_runtime_causality_per_step(): builder = _builder() @@ -137,3 +200,61 @@ def test_mla_cache_marker_is_promoted_to_group_capability(): [unmarked, unmarked] ).non_causal_multi_token_decode assert MLAAttentionSpec.merge([unmarked, marked]).non_causal_multi_token_decode + + +def test_builder_scopes_noncausal_capability_to_its_layers(): + merged_spec = _merged_mla_spec() + static_forward_context = { + "target": _mla_layer(non_causal=False), + "draft": _mla_layer(non_causal=True), + # Indexer and compressor caches share an MLA group but predate the flag. + "indexer": SimpleNamespace(), + } + vllm_config = _dspark_dcp_vllm_config(static_forward_context) + device = torch.device("cpu") + + def _flag(layer_names: list[str]) -> bool: + return _DcpCapableMLAMetadataBuilder( + merged_spec, + layer_names, + vllm_config, + device, + supports_dcp_with_varlen=True, + ).non_causal_multi_token_decode + + assert not _flag(["target"]) + assert _flag(["draft"]) + assert _flag(["target", "draft"]) + assert not _flag(["target", "indexer"]) + + +def test_merged_group_spec_does_not_mark_a_target_only_builder(): + # Both builders receive the same merged spec; only the draft layer is + # non-causal. Constructing through __init__ would still fail DCP + # validation if the builder read the group flag. + merged_spec = _merged_mla_spec() + static_forward_context = { + "target": _mla_layer(non_causal=False), + "draft": _mla_layer(non_causal=True), + } + vllm_config = _dspark_dcp_vllm_config(static_forward_context) + device = torch.device("cpu") + assert merged_spec.non_causal_multi_token_decode + + target = _CausalOnlyMLAMetadataBuilder( + merged_spec, + ["target"], + vllm_config, + device, + supports_dcp_with_varlen=True, + ) + assert not target.non_causal_multi_token_decode + + with pytest.raises(ValueError, match="non-causal draft"): + _DraftMLAMetadataBuilder( + merged_spec, + ["draft"], + vllm_config, + device, + supports_dcp_with_varlen=True, + ) diff --git a/tests/v1/attention/test_mla_prefill_registry.py b/tests/v1/attention/test_mla_prefill_registry.py index fc5efe3a5b..0068d53b88 100644 --- a/tests/v1/attention/test_mla_prefill_registry.py +++ b/tests/v1/attention/test_mla_prefill_registry.py @@ -54,10 +54,20 @@ def test_prefill_backend_clone_has_isolated_metadata(): def cleanup_overrides(): """Clear any overrides after each test.""" yield - for member in MLAPrefillBackendEnum: + for member in list(MLAPrefillBackendEnum): member.clear_override() +def _remove_dynamic_member(enum_cls, member): + value = member.value + name = member.name + enum_cls._member_map_.pop(name, None) + enum_cls._member_names_.remove(name) + delattr(enum_cls, name) + if enum_cls._value2member_map_.get(value) is member: + del enum_cls._value2member_map_[value] + + def test_custom_is_not_alias_of_any_backend(): all_backends = list(MLAPrefillBackendEnum) @@ -83,21 +93,11 @@ def test_custom_unregistered_raises(): def test_register_custom_backend_with_class_path(): register_mla_prefill_backend( - backend=MLAPrefillBackendEnum.CUSTOM, - class_path=( - "tests.v1.attention.test_mla_prefill_registry.CustomMLAPrefillBackend" - ), + MLAPrefillBackendEnum.CUSTOM, + "tests.v1.attention.test_mla_prefill_registry.CustomMLAPrefillBackend", ) - assert MLAPrefillBackendEnum.CUSTOM.is_overridden() - - class_path = MLAPrefillBackendEnum.CUSTOM.get_path() - assert class_path == ( - "tests.v1.attention.test_mla_prefill_registry.CustomMLAPrefillBackend" - ) - - backend_cls = MLAPrefillBackendEnum.CUSTOM.get_class() - assert backend_cls.get_name() == "CUSTOM" + assert MLAPrefillBackendEnum.CUSTOM.get_class().get_name() == "CUSTOM" def test_register_custom_backend_as_decorator(): @@ -120,33 +120,20 @@ def test_register_custom_backend_as_decorator(): def test_override_existing_backend(): - original_path = MLAPrefillBackendEnum.FLASH_ATTN.get_path() - register_mla_prefill_backend( - backend=MLAPrefillBackendEnum.FLASH_ATTN, - class_path=( - "tests.v1.attention.test_mla_prefill_registry.CustomMLAPrefillBackend" - ), + MLAPrefillBackendEnum.FLASH_ATTN, + "tests.v1.attention.test_mla_prefill_registry.CustomMLAPrefillBackend", ) - + assert MLAPrefillBackendEnum.FLASH_ATTN.get_class().get_name() == "CUSTOM" assert MLAPrefillBackendEnum.FLASH_ATTN.is_overridden() - assert MLAPrefillBackendEnum.FLASH_ATTN.get_path() != original_path - - backend_cls = MLAPrefillBackendEnum.FLASH_ATTN.get_class() - assert backend_cls.get_name() == "CUSTOM" def test_clear_override(): original_path = MLAPrefillBackendEnum.FLASH_ATTN.get_path() - register_mla_prefill_backend( - backend=MLAPrefillBackendEnum.FLASH_ATTN, - class_path=( - "tests.v1.attention.test_mla_prefill_registry.CustomMLAPrefillBackend" - ), + MLAPrefillBackendEnum.FLASH_ATTN, + "tests.v1.attention.test_mla_prefill_registry.CustomMLAPrefillBackend", ) - assert MLAPrefillBackendEnum.FLASH_ATTN.is_overridden() - MLAPrefillBackendEnum.FLASH_ATTN.clear_override() assert not MLAPrefillBackendEnum.FLASH_ATTN.is_overridden() assert MLAPrefillBackendEnum.FLASH_ATTN.get_path() == original_path @@ -172,3 +159,72 @@ def test_rocm_aiter_fa_registered(): # The AITER FA path is the fp16/bf16 generic-varlen prefill path. assert backend_cls.supports_dtype(torch.bfloat16) assert backend_cls.supports_dtype(torch.float16) + + +def test_register_dynamic_member(): + p = "tests.v1.attention.test_mla_prefill_registry.CustomMLAPrefillBackend" + member = MLAPrefillBackendEnum.register("DYNAMIC_TEST", p) + assert member.name == "DYNAMIC_TEST" + assert member is MLAPrefillBackendEnum["DYNAMIC_TEST"] + + _remove_dynamic_member(MLAPrefillBackendEnum, member) + + +def test_register_dynamic_member_duplicate_raises(): + path = "some.module.Class" + MLAPrefillBackendEnum.register("DUP_TEST", path) + with pytest.raises(ValueError, match="already exists"): + MLAPrefillBackendEnum.register("DUP_TEST", "other.module.OtherClass") + _remove_dynamic_member(MLAPrefillBackendEnum, MLAPrefillBackendEnum.DUP_TEST) + + +class StringDecoratorMLAPrefillBackend(MLAPrefillBackend): + """Mock backend defined at module level so get_class() can resolve + the auto-generated class path (local classes are not importable).""" + + supported_dtypes = [torch.bfloat16] + requires_r1_mla_dimensions = False + + @staticmethod + def get_name() -> str: + return "DECORATED" + + def run_prefill_new_tokens(self, q, k, v, return_softmax_lse): + raise NotImplementedError + + def run_prefill_context_chunk(self, chunk_idx, q, k, v): + raise NotImplementedError + + +def test_register_mla_prefill_backend_with_string_name_direct(): + register_mla_prefill_backend( + "STRING_DIRECT", + "tests.v1.attention.test_mla_prefill_registry.CustomMLAPrefillBackend", + ) + member = MLAPrefillBackendEnum.STRING_DIRECT + assert member.is_overridden() + assert member.get_class().get_name() == "CUSTOM" + + member.clear_override() + _remove_dynamic_member(MLAPrefillBackendEnum, member) + + +def test_register_mla_prefill_backend_with_string_name_decorator(): + # The decorated class must be module-level: the decorator derives a + # class path from it, and get_class() re-imports that path. A class + # defined inside this test would have a non-importable path. + decorated_cls = StringDecoratorMLAPrefillBackend + + result = register_mla_prefill_backend("STRING_DECORATOR")(decorated_cls) + assert result is decorated_cls + + member = MLAPrefillBackendEnum.STRING_DECORATOR + assert member.is_overridden() + assert member.get_path() == ( + f"{decorated_cls.__module__}.{decorated_cls.__qualname__}" + ) + assert member.get_class() is decorated_cls + assert member.get_class().get_name() == "DECORATED" + + member.clear_override() + _remove_dynamic_member(MLAPrefillBackendEnum, member) diff --git a/tests/v1/attention/test_mla_prefill_selector.py b/tests/v1/attention/test_mla_prefill_selector.py index d5b81695aa..3c6e1a9295 100644 --- a/tests/v1/attention/test_mla_prefill_selector.py +++ b/tests/v1/attention/test_mla_prefill_selector.py @@ -2,6 +2,7 @@ # SPDX-FileCopyrightText: Copyright contributors to the vLLM project """Tests for MLA prefill backend selector.""" +from contextlib import suppress from unittest.mock import MagicMock, patch import pytest @@ -9,13 +10,12 @@ import torch from vllm.config import AttentionConfig, ModelConfig, VllmConfig from vllm.platforms import current_platform -from vllm.platforms.interface import DeviceCapability +from vllm.platforms.interface import DeviceCapability, Platform from vllm.v1.attention.backends.mla.prefill.base import MLADimensions from vllm.v1.attention.backends.mla.prefill.registry import MLAPrefillBackendEnum from vllm.v1.attention.backends.mla.prefill.selector import ( MLAPrefillSelectorConfig, _auto_select_mla_prefill_backend, - _get_mla_prefill_backend_priorities, get_mla_prefill_backend, ) @@ -55,51 +55,170 @@ def _make_vllm_config( return mock_vllm_config +class TestPlatformMLAPrefillHooks: + """Tests for the Platform.get_mla_prefill_backend_cls hook itself.""" + + def _get_tried_backends(self, capability, mla_dimensions): + """Run the default hook, returning the backend order tried.""" + selector_config = MLAPrefillSelectorConfig( + dtype=torch.bfloat16, + mla_dimensions=mla_dimensions, + ) + tried: list[MLAPrefillBackendEnum] = [] + + def mock_get_class(backend_enum): + tried.append(backend_enum) + cls = MagicMock() + cls.validate_configuration.return_value = ["not available"] + return cls + + with ( + patch.object(Platform, "get_device_capability", return_value=capability), + patch.object(MLAPrefillBackendEnum, "get_class", mock_get_class), + suppress(ValueError), + ): + Platform.get_mla_prefill_backend_cls(selector_config) + return tried + + def test_priorities_no_capability(self): + tried = self._get_tried_backends( + None, + MLADimensions(qk_nope_head_dim=128, qk_rope_head_dim=64, v_head_dim=128), + ) + assert tried == [MLAPrefillBackendEnum.FLASH_ATTN] + + def test_priorities_blackwell_glm_dimensions(self): + tried = self._get_tried_backends( + DeviceCapability(major=10, minor=0), + MLADimensions(qk_nope_head_dim=192, qk_rope_head_dim=64, v_head_dim=256), + ) + assert tried == [ + MLAPrefillBackendEnum.TRTLLM_RAGGED, + MLAPrefillBackendEnum.FLASH_ATTN, + MLAPrefillBackendEnum.FLASHINFER, + MLAPrefillBackendEnum.TOKENSPEED_MLA, + ] + + def test_priorities_blackwell_deepseek_dimensions(self): + tried = self._get_tried_backends( + DeviceCapability(major=10, minor=0), + MLADimensions(qk_nope_head_dim=128, qk_rope_head_dim=64, v_head_dim=128), + ) + assert tried == [ + MLAPrefillBackendEnum.FLASH_ATTN, + MLAPrefillBackendEnum.TRTLLM_RAGGED, + MLAPrefillBackendEnum.FLASHINFER, + MLAPrefillBackendEnum.TOKENSPEED_MLA, + ] + + def test_priorities_hopper(self): + tried = self._get_tried_backends( + DeviceCapability(major=9, minor=0), + MLADimensions(qk_nope_head_dim=128, qk_rope_head_dim=64, v_head_dim=128), + ) + assert tried == [MLAPrefillBackendEnum.FLASH_ATTN] + + def test_skips_import_error_backends(self): + selector_config = MLAPrefillSelectorConfig( + dtype=torch.bfloat16, + mla_dimensions=MLADimensions( + qk_nope_head_dim=192, + qk_rope_head_dim=64, + v_head_dim=256, + ), + ) + valid = MagicMock() + valid.validate_configuration.return_value = [] + + def mock_get_class(backend_enum): + if backend_enum is MLAPrefillBackendEnum.FLASH_ATTN: + return valid + raise ImportError("not installed") + + with ( + patch.object( + Platform, + "get_device_capability", + return_value=DeviceCapability(major=10, minor=0), + ), + patch.object(MLAPrefillBackendEnum, "get_class", mock_get_class), + ): + selected = Platform.get_mla_prefill_backend_cls(selector_config) + assert selected is valid + + def test_all_invalid_raises_error(self): + selector_config = MLAPrefillSelectorConfig(dtype=torch.bfloat16) + + invalid = MagicMock() + invalid.validate_configuration.return_value = ["not available"] + + with ( + patch.object( + Platform, + "get_device_capability", + return_value=DeviceCapability(major=10, minor=0), + ), + patch.object(MLAPrefillBackendEnum, "get_class", return_value=invalid), + pytest.raises(ValueError, match="No valid MLA"), + ): + Platform.get_mla_prefill_backend_cls(selector_config) + + def test_cpu_platform_returns_fixed_backend(self): + """CPU backends are returned via the fixed-backend hook.""" + try: + cpu_cls = MLAPrefillBackendEnum.CPU.get_class() + zen_cls = MLAPrefillBackendEnum.ZEN_CPU.get_class() + except ImportError: + pytest.skip("CPU MLA prefill backend not available") + return + + from vllm.platforms.cpu import CpuPlatform + + expected = zen_cls if zen_cls.is_available() else cpu_cls + assert ( + CpuPlatform.get_mla_prefill_backend_cls( + MLAPrefillSelectorConfig(dtype=torch.bfloat16) + ) + is expected + ) + + class TestGetMLAPrefillBackend: """Tests for get_mla_prefill_backend (public API).""" def test_cpu_uses_sdpa_prefill(self): vllm_config = _make_vllm_config() - zen_cls = MLAPrefillBackendEnum.ZEN_CPU.get_class() - - # A Zen CPU with zentorch selects the Zen backend, so expect whichever - # one this host offers. - expected = ( - zen_cls if zen_cls.is_available() else MLAPrefillBackendEnum.CPU.get_class() - ) with patch("vllm.platforms.current_platform") as mock_platform: - mock_platform.is_cpu.return_value = True - - backend = get_mla_prefill_backend(vllm_config) - assert backend is expected - - def test_zen_cpu_prefers_zentorch_prefill(self): - vllm_config = _make_vllm_config() - zen_cls = MLAPrefillBackendEnum.ZEN_CPU.get_class() - - with ( - patch("vllm.platforms.current_platform") as mock_platform, - patch.object(zen_cls, "is_available", return_value=True), - ): - mock_platform.is_cpu.return_value = True - - backend = get_mla_prefill_backend(vllm_config) - assert backend is zen_cls - - def test_zen_cpu_falls_back_when_zentorch_unavailable(self): - vllm_config = _make_vllm_config() - zen_cls = MLAPrefillBackendEnum.ZEN_CPU.get_class() - - with ( - patch("vllm.platforms.current_platform") as mock_platform, - patch.object(zen_cls, "is_available", return_value=False), - ): - mock_platform.is_cpu.return_value = True + mock_platform.get_mla_prefill_backend_cls.return_value = ( + MLAPrefillBackendEnum.CPU.get_class() + ) backend = get_mla_prefill_backend(vllm_config) assert backend is MLAPrefillBackendEnum.CPU.get_class() + def test_zen_cpu_prefers_zentorch_prefill(self): + from vllm.platforms.cpu import CpuPlatform + + zen_cls = MLAPrefillBackendEnum.ZEN_CPU.get_class() + + with patch.object(zen_cls, "is_available", return_value=True): + backend = CpuPlatform.get_mla_prefill_backend_cls( + MLAPrefillSelectorConfig(dtype=torch.bfloat16) + ) + assert backend is zen_cls + + def test_zen_cpu_falls_back_when_zentorch_unavailable(self): + from vllm.platforms.cpu import CpuPlatform + + zen_cls = MLAPrefillBackendEnum.ZEN_CPU.get_class() + + with patch.object(zen_cls, "is_available", return_value=False): + backend = CpuPlatform.get_mla_prefill_backend_cls( + MLAPrefillSelectorConfig(dtype=torch.bfloat16) + ) + assert backend is MLAPrefillBackendEnum.CPU.get_class() + def test_no_device_capability_returns_flash_attn(self): vllm_config = _make_vllm_config() @@ -108,16 +227,8 @@ class TestGetMLAPrefillBackend: def get_name(): return "FLASH_ATTN" - with ( - patch("vllm.platforms.current_platform") as mock_platform, - patch.object( - MLAPrefillBackendEnum.FLASH_ATTN, - "get_class", - return_value=FlashAttnBackend, - ), - ): - mock_platform.get_device_capability.return_value = None - mock_platform.is_cpu.return_value = False + with patch("vllm.platforms.current_platform") as mock_platform: + mock_platform.get_mla_prefill_backend_cls.return_value = FlashAttnBackend backend = get_mla_prefill_backend(vllm_config) assert backend.get_name() == "FLASH_ATTN" @@ -214,7 +325,7 @@ class TestGetMLAPrefillBackend: mock_platform.get_device_capability.return_value = DeviceCapability( major=9, minor=0 ) - mock_platform.is_rocm.return_value = False + mock_platform.get_mla_prefill_backend_cls.return_value = flash_attn_cls backend = get_mla_prefill_backend(vllm_config) assert backend.get_name() == "FLASH_ATTN" @@ -224,7 +335,6 @@ class TestAutoSelectMLAPrefillBackend: """Tests for fallback and error paths in auto-selection.""" def test_blackwell_glm_dimensions_use_trtllm(self): - capability = DeviceCapability(major=10, minor=0) selector_config = MLAPrefillSelectorConfig( dtype=torch.bfloat16, mla_dimensions=MLADimensions( @@ -235,7 +345,6 @@ class TestAutoSelectMLAPrefillBackend: ) try: - flash_attn_cls = MLAPrefillBackendEnum.FLASH_ATTN.get_class() trtllm_cls = MLAPrefillBackendEnum.TRTLLM_RAGGED.get_class() except ImportError: pytest.skip("MLA prefill backend not available") @@ -243,19 +352,13 @@ class TestAutoSelectMLAPrefillBackend: with ( patch("vllm.platforms.current_platform") as mock_platform, - patch.object(flash_attn_cls, "is_available", return_value=True), patch.object(trtllm_cls, "validate_configuration", return_value=[]), ): - # Force the non-ROCm priority on the Blackwell. - mock_platform.is_rocm.return_value = False - backend = _auto_select_mla_prefill_backend( - capability, - selector_config, - ) + mock_platform.get_mla_prefill_backend_cls.return_value = trtllm_cls + backend = _auto_select_mla_prefill_backend(selector_config) assert backend.get_name() == "TRTLLM_RAGGED" - def test_all_fail_raises_error(self): - capability = DeviceCapability(major=10, minor=0) + def test_platform_hook_raises_propagates(self): selector_config = MLAPrefillSelectorConfig( dtype=torch.bfloat16, mla_dimensions=MLADimensions( @@ -265,18 +368,14 @@ class TestAutoSelectMLAPrefillBackend: ), ) - def mock_get_class(backend_enum): # noqa: ARG001 - cls = MagicMock() - cls.validate_configuration.return_value = ["not available"] - return cls - - with patch.object(MLAPrefillBackendEnum, "get_class", mock_get_class): - _auto_select_mla_prefill_backend.cache_clear() - with pytest.raises(ValueError, match="No valid MLA"): - _auto_select_mla_prefill_backend( - capability, - selector_config, - ) + with ( + patch("vllm.platforms.current_platform") as mock_platform, + pytest.raises(ValueError, match="No valid MLA"), + ): + mock_platform.get_mla_prefill_backend_cls.side_effect = ValueError( + "No valid MLA prefill backend found" + ) + _auto_select_mla_prefill_backend(selector_config) class TestBackendValidation: @@ -391,18 +490,29 @@ class TestROCmAiterFAPrefillSelection: def test_rocm_priorities_prefer_aiter_fa(self): """On ROCm, ROCM_AITER_FA is tried first, FLASH_ATTN as fallback.""" - with patch("vllm.platforms.current_platform") as mock_platform: - mock_platform.is_rocm.return_value = True - priorities = _get_mla_prefill_backend_priorities( - DeviceCapability(major=9, minor=5), - MLADimensions( - qk_nope_head_dim=128, - qk_rope_head_dim=64, - v_head_dim=128, - ), - ) + from vllm.platforms.rocm import RocmPlatform - assert priorities == [ + selector_config = MLAPrefillSelectorConfig(dtype=torch.bfloat16) + tried: list[MLAPrefillBackendEnum] = [] + + def mock_get_class(backend_enum): + tried.append(backend_enum) + cls = MagicMock() + cls.validate_configuration.return_value = ["not available"] + return cls + + with ( + patch.object( + RocmPlatform, + "get_device_capability", + return_value=MagicMock(), + ), + patch.object(MLAPrefillBackendEnum, "get_class", mock_get_class), + suppress(ValueError), + ): + RocmPlatform.get_mla_prefill_backend_cls(selector_config) + + assert tried == [ MLAPrefillBackendEnum.ROCM_AITER_FA, MLAPrefillBackendEnum.FLASH_ATTN, ] @@ -459,50 +569,29 @@ class TestROCmAiterFAPrefillSelection: AiterFlashAttnPrefillBackend, ) - # gfx gating is simulated via the mocked validate_configuration, - # not the capability. - capability = MagicMock() selector_config = MLAPrefillSelectorConfig(dtype=torch.bfloat16) - with ( - patch("vllm.platforms.current_platform") as mock_platform, - patch.object( - AiterFlashAttnPrefillBackend, - "validate_configuration", - return_value=[], - ), - ): - mock_platform.is_rocm.return_value = True - backend = _auto_select_mla_prefill_backend(capability, selector_config) + with patch("vllm.platforms.current_platform") as mock_platform: + mock_platform.get_mla_prefill_backend_cls.return_value = ( + AiterFlashAttnPrefillBackend + ) + backend = _auto_select_mla_prefill_backend(selector_config) assert backend.get_name() == "ROCM_AITER_FA" def test_auto_select_falls_back_to_flash_attn_when_aiter_invalid(self): - from vllm.v1.attention.backends.mla.prefill.aiter_flash_attn import ( - AiterFlashAttnPrefillBackend, - ) - try: flash_attn_cls = MLAPrefillBackendEnum.FLASH_ATTN.get_class() except ImportError: pytest.skip("FLASH_ATTN backend not available") return - # the fallback is forced by the mocked validate_configuration, + # the fallback is forced by the mocked get_mla_prefill_backend_cls, # not the capability. - capability = MagicMock() selector_config = MLAPrefillSelectorConfig(dtype=torch.bfloat16) - with ( - patch("vllm.platforms.current_platform") as mock_platform, - patch.object( - AiterFlashAttnPrefillBackend, - "validate_configuration", - return_value=["compute capability not supported"], - ), - patch.object(flash_attn_cls, "validate_configuration", return_value=[]), - ): - mock_platform.is_rocm.return_value = True - backend = _auto_select_mla_prefill_backend(capability, selector_config) + with patch("vllm.platforms.current_platform") as mock_platform: + mock_platform.get_mla_prefill_backend_cls.return_value = flash_attn_cls + backend = _auto_select_mla_prefill_backend(selector_config) assert backend.get_name() == "FLASH_ATTN" diff --git a/tests/v1/attention/test_rocm_aiter_mla_dcp_cprr.py b/tests/v1/attention/test_rocm_aiter_mla_dcp_cprr.py index 13b002cda0..cd88ade8ee 100644 --- a/tests/v1/attention/test_rocm_aiter_mla_dcp_cprr.py +++ b/tests/v1/attention/test_rocm_aiter_mla_dcp_cprr.py @@ -11,7 +11,11 @@ routes built; it is marked and skipped when unavailable. from __future__ import annotations +import sys +from types import SimpleNamespace + import pytest +import torch rocm_aiter_mla = pytest.importorskip( "vllm.v1.attention.backends.mla.rocm_aiter_mla", @@ -26,9 +30,15 @@ NATIVE = rocm_aiter_mla._NATIVE_CPRR_HEADS MIN_QLEN = rocm_aiter_mla._MIN_CPRR_QLEN -def test_dcp_verify_env_defaults_to_segmented(monkeypatch): +def test_dcp_verify_env_defaults_to_auto(monkeypatch): monkeypatch.delenv("VLLM_ROCM_AITER_MLA_DCP_VERIFY", raising=False) - assert rocm_aiter_mla.envs.VLLM_ROCM_AITER_MLA_DCP_VERIFY == "segmented" + assert rocm_aiter_mla.envs.VLLM_ROCM_AITER_MLA_DCP_VERIFY == "auto" + + +@pytest.mark.parametrize("route", ["auto", "asm", "segmented"]) +def test_dcp_verify_env_accepts_routes(monkeypatch, route): + monkeypatch.setenv("VLLM_ROCM_AITER_MLA_DCP_VERIFY", route) + assert route == rocm_aiter_mla.envs.VLLM_ROCM_AITER_MLA_DCP_VERIFY def test_dcp_verify_env_rejects_unknown_route(monkeypatch): @@ -116,6 +126,126 @@ def test_route_needs_gfx950(monkeypatch): ) +# -------------------------------------------------------------------------- +# AiterMLAMetadataBuilder -- the route the env value resolves to +# -------------------------------------------------------------------------- + +DCP = 8 + + +def _build(monkeypatch, route, gathered_heads, segmented_supported=True): + """Run the real builder __init__ on CPU at TP8/DCP8 with speculative decoding. + + Stubs only what needs a GPU, AITER, or a DCP group, so the route decision + under test is the one the builder actually makes. + """ + + def init_common_builder(self, *args, **kwargs): + self.num_heads = gathered_heads // DCP + self.dcp_world_size = DCP + self.reorder_batch_threshold = 5 + + monkeypatch.setattr( + rocm_aiter_mla.MLACommonMetadataBuilder, "__init__", init_common_builder + ) + monkeypatch.setattr( + rocm_aiter_mla, + "_segmented_dcp_verify_supported", + lambda *args: segmented_supported, + ) + monkeypatch.setattr(rocm_aiter_mla, "_fp8_mla_prefill_supported", lambda: False) + monkeypatch.setattr( + rocm_aiter_mla, "get_dcp_group", lambda: SimpleNamespace(rank_in_group=0) + ) + monkeypatch.setattr( + torch.cuda, + "get_device_properties", + lambda device: SimpleNamespace(multi_processor_count=256), + ) + monkeypatch.setitem( + sys.modules, + "aiter", + SimpleNamespace( + dtypes=SimpleNamespace(fp8="fp8", fp16="fp16", bf16="bf16"), + get_mla_metadata_info_v1=lambda *args, **kwargs: tuple( + (1, torch.int32) for _ in range(6) + ), + ), + ) + if route is None: + monkeypatch.delenv("VLLM_ROCM_AITER_MLA_DCP_VERIFY", raising=False) + else: + monkeypatch.setenv("VLLM_ROCM_AITER_MLA_DCP_VERIFY", route) + + config = SimpleNamespace( + speculative_config=SimpleNamespace(num_speculative_tokens=4), + parallel_config=SimpleNamespace( + decode_context_parallel_size=DCP, cp_kv_cache_interleave_size=1 + ), + model_config=SimpleNamespace(max_model_len=16, dtype=torch.bfloat16), + scheduler_config=SimpleNamespace(max_num_seqs=2), + cache_config=SimpleNamespace(cache_dtype="fp8_e4m3", num_gpu_blocks=None), + compilation_config=SimpleNamespace( + cudagraph_mode=SimpleNamespace(has_full_cudagraphs=lambda: False) + ), + ) + return rocm_aiter_mla.AiterMLAMetadataBuilder( + kv_cache_spec=SimpleNamespace(block_size=1, dtype=torch.bfloat16), + layer_names=["layer.0"], + vllm_config=config, + device=torch.device("cpu"), + ) + + +@pytest.mark.parametrize("route", [None, "auto", "AUTO"]) +def test_auto_selects_cprr_for_k3_gathered_heads(on_gfx950, monkeypatch, route): + """K3 at TP8/DCP8 gathers 96 heads; auto must take cprr padded to 128.""" + builder = _build(monkeypatch, route, gathered_heads=96) + assert builder._asm_dcp_verify + assert builder._asm_dcp_verify_heads == 128 + assert builder._num_attention_heads == 128 + + +def test_auto_falls_back_to_segmented_above_largest_native(on_gfx950, monkeypatch): + builder = _build(monkeypatch, "auto", gathered_heads=256) + assert not builder._asm_dcp_verify + assert builder._supports_segmented_dcp_verify + + +def test_auto_without_any_route_names_the_missing_segmented_build( + on_gfx950, monkeypatch +): + with pytest.raises(ValueError) as exc: + _build(monkeypatch, "auto", gathered_heads=256, segmented_supported=False) + assert "VLLM_ROCM_AITER_MLA_DCP_VERIFY=auto" in str(exc.value) + assert "lacks segmented MLA decode" in str(exc.value) + + +@pytest.mark.parametrize("route", ["asm", "ASM"]) +def test_explicit_asm_is_case_insensitive(on_gfx950, monkeypatch, route): + """env_with_choices returns the raw string; ASM used to mean segmented.""" + assert _build(monkeypatch, route, gathered_heads=96)._asm_dcp_verify + + +def test_explicit_asm_refuses_unservable_heads(on_gfx950, monkeypatch): + with pytest.raises(ValueError) as exc: + _build(monkeypatch, "ASM", gathered_heads=256) + assert "VLLM_ROCM_AITER_MLA_DCP_VERIFY=asm" in str(exc.value) + assert "Set VLLM_ROCM_AITER_MLA_DCP_VERIFY=segmented" in str(exc.value) + + +@pytest.mark.parametrize("route", ["segmented", "SEGMENTED"]) +def test_explicit_segmented_never_selects_cprr(on_gfx950, monkeypatch, route): + assert not _build(monkeypatch, route, gathered_heads=96)._asm_dcp_verify + + +def test_auto_off_gfx950_stays_segmented(monkeypatch): + import vllm.platforms.rocm as rocm + + monkeypatch.setattr(rocm, "on_gfx950", lambda: False) + assert not _build(monkeypatch, "auto", gathered_heads=96)._asm_dcp_verify + + # -------------------------------------------------------------------------- # qlen floor # -------------------------------------------------------------------------- diff --git a/tests/v1/attention/test_rocm_attention_backends_selection.py b/tests/v1/attention/test_rocm_attention_backends_selection.py index b99a33f9f7..067d1dbc8b 100644 --- a/tests/v1/attention/test_rocm_attention_backends_selection.py +++ b/tests/v1/attention/test_rocm_attention_backends_selection.py @@ -61,6 +61,216 @@ def test_aiter_unified_attention_uses_dedicated_metadata_builder(): ) +def test_segmented_attention_uses_dedicated_backend_components(): + from vllm.v1.attention.backends.rocm_segmented_attn import ( + RocmSegmentedAttentionBackend, + RocmSegmentedAttentionImpl, + RocmSegmentedAttentionMetadataBuilder, + ) + + assert ( + RocmSegmentedAttentionBackend.get_builder_cls() + is RocmSegmentedAttentionMetadataBuilder + ) + assert RocmSegmentedAttentionBackend.get_impl_cls() is RocmSegmentedAttentionImpl + assert RocmSegmentedAttentionBackend.get_name() == "ROCM_SEGMENTED_ATTN" + assert RocmSegmentedAttentionBackend.supports_non_causal() + assert RocmSegmentedAttentionBackend.supports_sliding_window() + assert RocmSegmentedAttentionBackend.supports_sink() + assert not RocmSegmentedAttentionBackend.supports_mm_prefix() + + +@pytest.mark.parametrize( + "sliding_window,logits_soft_cap,has_sinks", + [(4096, None, False), (None, 50.0, False), (None, None, True)], +) +def test_segmented_attention_admits_unified_fallback_features( + monkeypatch, sliding_window, logits_soft_cap, has_sinks +): + from vllm.platforms import rocm + from vllm.v1.attention.backend import AttentionType + from vllm.v1.attention.backends.rocm_segmented_attn import ( + RocmSegmentedAttentionImpl, + ) + + monkeypatch.setattr(rocm, "on_gfx1x", lambda: True) + monkeypatch.setattr(rocm, "on_gfx12x", lambda: True) + sinks = torch.zeros(8) if has_sinks else None + + impl = RocmSegmentedAttentionImpl( + num_heads=8, + head_size=128, + scale=128**-0.5, + num_kv_heads=2, + alibi_slopes=None, + sliding_window=sliding_window, + kv_cache_dtype="auto", + logits_soft_cap=logits_soft_cap, + attn_type=AttentionType.DECODER, + sinks=sinks, + ) + + assert isinstance(impl, RocmSegmentedAttentionImpl) + + +@pytest.mark.parametrize( + "platform,rdna", + [("cuda", False), ("cpu", False), ("xpu", False), ("rocm", False), ("rocm", True)], +) +def test_segmented_attention_requires_rocm_rdna(monkeypatch, platform, rdna): + """Reject unsupported platforms before probing ROCm or constructing kernels.""" + from vllm.platforms import rocm + from vllm.platforms.interface import DeviceCapability + from vllm.v1.attention.backends import rocm_segmented_attn as segmented + + monkeypatch.setattr( + segmented, + "current_platform", + SimpleNamespace(is_rocm=lambda: platform == "rocm"), + ) + + def on_rdna(): + assert platform == "rocm", "Must not probe ROCm on another platform" + return rdna + + monkeypatch.setattr(rocm, "on_gfx1x", on_rdna) + reason = segmented.RocmSegmentedAttentionBackend.supports_combination( + 128, + torch.bfloat16, + "auto", + 32, + False, + False, + False, + False, + DeviceCapability(12, 0), + ) + if rdna: + assert reason is None + segmented.RocmSegmentedAttentionImpl(8, 128, 128**-0.5, 2, None, None, "auto") + else: + assert reason == "ROCM_SEGMENTED_ATTN requires AMD RDNA GPUs on ROCm" + with pytest.raises(ValueError, match="requires AMD RDNA GPUs on ROCm"): + segmented.RocmSegmentedAttentionImpl( + 8, 128, 128**-0.5, 2, None, None, "auto" + ) + + +@pytest.mark.parametrize("explicit_cache", [False, True]) +def test_segmented_triton_cache_survives_standalone_compilation( + monkeypatch, tmp_path, explicit_cache +): + """Backend initialization preserves one cache across model compilations.""" + import os + + from vllm.compilation.compiler_interface import InductorStandaloneAdaptor + from vllm.platforms import rocm + from vllm.v1.attention.backends.rocm_segmented_attn import ( + RocmSegmentedAttentionImpl, + ) + + monkeypatch.setattr(rocm, "on_gfx1x", lambda: True) + monkeypatch.setenv("VLLM_CACHE_ROOT", str(tmp_path)) + monkeypatch.delenv("TRITON_CACHE_DIR", raising=False) + monkeypatch.delenv("TORCHINDUCTOR_CACHE_DIR", raising=False) + expected = tmp_path / "rocm_segmented_attention" / "triton" + if explicit_cache: + expected = tmp_path / "custom" + monkeypatch.setenv("TRITON_CACHE_DIR", str(expected)) + for model in ("target", "draft", "restart_target"): + RocmSegmentedAttentionImpl(8, 128, 128**-0.5, 2, None, None, "auto") + assert os.environ["TRITON_CACHE_DIR"] == str(expected) + assert expected.is_dir() + InductorStandaloneAdaptor("binary").initialize_cache(str(tmp_path / model)) + assert os.environ["TRITON_CACHE_DIR"] == str(expected) + + +def test_segmented_attention_is_opt_in(monkeypatch): + from vllm.platforms import rocm + from vllm.platforms.rocm import RocmPlatform, _get_backend_priorities + + assert AttentionBackendEnum.ROCM_SEGMENTED_ATTN not in _get_backend_priorities( + use_mla=False, use_sparse=False + ) + + monkeypatch.setattr(rocm, "on_gfx1x", lambda: True) + monkeypatch.setattr(rocm, "on_gfx12x", lambda: True) + config = AttentionSelectorConfig( + head_size=128, + dtype=torch.bfloat16, + kv_cache_dtype="fp8", + block_size=32, + ) + + path = RocmPlatform.get_attn_backend_cls( + selected_backend=AttentionBackendEnum.ROCM_SEGMENTED_ATTN, + attn_selector_config=config, + ) + + assert path == AttentionBackendEnum.ROCM_SEGMENTED_ATTN.get_path() + + +def test_segmented_attention_rejects_mm_prefix_at_selection(monkeypatch): + from vllm.platforms import rocm + from vllm.platforms.rocm import RocmPlatform + + monkeypatch.setattr(rocm, "on_gfx1x", lambda: True) + config = AttentionSelectorConfig( + head_size=128, + dtype=torch.bfloat16, + kv_cache_dtype="auto", + block_size=32, + use_mm_prefix=True, + ) + + with pytest.raises(ValueError, match="multimodal token full attention"): + RocmPlatform.get_attn_backend_cls( + selected_backend=AttentionBackendEnum.ROCM_SEGMENTED_ATTN, + attn_selector_config=config, + ) + + +def test_segmented_attention_forward_uses_dedicated_dispatch(monkeypatch): + from vllm.config import KernelConfig + from vllm.platforms import rocm + from vllm.v1.attention.backends.rocm_segmented_attn import ( + RocmSegmentedAttentionImpl, + ) + + monkeypatch.setattr(rocm, "on_gfx1x", lambda: True) + impl = RocmSegmentedAttentionImpl(8, 128, 128**-0.5, 2, None, None, "auto") + impl._segmented_attention_config = SimpleNamespace( + kernel_config=KernelConfig(), + scheduler_config=SimpleNamespace(max_num_seqs=1, max_num_batched_tokens=3), + model_config=SimpleNamespace(max_model_len=16), + ) + query = torch.empty(3, 8, 128, dtype=torch.bfloat16) + key = torch.empty(3, 2, 128, dtype=torch.bfloat16) + value = torch.empty_like(key) + output = torch.empty_like(query) + kv_cache = torch.empty(2, 2, 16, 256, dtype=torch.bfloat16) + metadata = SimpleNamespace( + use_cascade=False, + num_actual_tokens=3, + block_table=torch.zeros(1, 1, dtype=torch.int32), + query_start_loc=torch.tensor([0, 3], dtype=torch.int32), + seq_lens=torch.tensor([3], dtype=torch.int32), + max_seq_len=3, + max_query_len=3, + causal=True, + ) + layer = SimpleNamespace(_k_scale=torch.ones(()), _v_scale=torch.ones(())) + + with patch( + "vllm.v1.attention.backends.rocm_segmented_attn.segmented_attention" + ) as dispatch: + result = impl.forward(layer, query, key, value, kv_cache, metadata, output) + + assert result is output + dispatch.assert_called_once() + assert dispatch.call_args.kwargs["key_cache"].shape == (2, 16, 2, 128) + + def test_aiter_unified_attention_capture_preserves_query_start_locations(): from vllm.v1.attention.backends.rocm_aiter_unified_attn import ( RocmAiterUnifiedAttentionMetadataBuilder, diff --git a/tests/v1/attention/test_rocm_paged_mqa_context_lens.py b/tests/v1/attention/test_rocm_paged_mqa_context_lens.py new file mode 100644 index 0000000000..3122cc915a --- /dev/null +++ b/tests/v1/attention/test_rocm_paged_mqa_context_lens.py @@ -0,0 +1,32 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest +import torch + +from vllm._aiter_ops import is_aiter_found_and_supported +from vllm.platforms import current_platform + +if not current_platform.is_rocm() or not is_aiter_found_and_supported(): + pytest.skip("Requires ROCm with AITER.", allow_module_level=True) + +from vllm.v1.attention.ops.rocm_aiter_mla_sparse import _per_sequence_context_lens + + +def test_per_row_table_uses_last_column(): + seq_lens = torch.tensor([8192, 1500, 3000, 700], dtype=torch.int32) + next_n = 3 + offsets = torch.arange(next_n, dtype=torch.int32) - next_n + 1 + per_row = seq_lens.unsqueeze(1) + offsets + + out = _per_sequence_context_lens(per_row) + + assert out.shape == (4,) + assert out.is_contiguous() + assert torch.equal(out, seq_lens) + + +@pytest.mark.parametrize("shape", [(4,), (4, 1)]) +def test_per_sequence_input_is_unchanged(shape): + context_lens = torch.arange(1, 5, dtype=torch.int32).reshape(shape) + assert _per_sequence_context_lens(context_lens) is context_lens diff --git a/tests/v1/attention/test_sparse_mla_backends.py b/tests/v1/attention/test_sparse_mla_backends.py index 468e36a787..973a9e03e8 100644 --- a/tests/v1/attention/test_sparse_mla_backends.py +++ b/tests/v1/attention/test_sparse_mla_backends.py @@ -295,7 +295,8 @@ def _quantize_dequantize_fp8_ds_mla( rope_dim = k_pe.shape[-1] num_tokens = kv_c.shape[0] num_blocks = max(1, math.ceil(num_tokens / block_size)) - entry_size = kv_lora_rank + 4 * 4 + 2 * rope_dim + # The 64-dim RoPE slot is always there; NoPE rows zero-fill it. + entry_size = kv_lora_rank + 4 * 4 + 2 * 64 tmp_cache = torch.zeros( num_blocks, block_size, entry_size, dtype=torch.uint8, device=kv_c.device @@ -413,6 +414,13 @@ def _quantize_dequantize_nvfp4_ds_mla( @pytest.mark.parametrize("tensor_parallel_size", [1, 2, 4]) @pytest.mark.parametrize("block_size", [32, 64]) @pytest.mark.parametrize(("q_scale", "k_scale"), [(1.0, 1.0), (2.0, 3.0)]) +@pytest.mark.parametrize( + ("total_num_heads", "qk_nope_head_dim", "qk_rope_head_dim", "v_head_dim"), + [ + pytest.param(128, 128, 64, 128, id="deepseek"), + pytest.param(64, 256, 0, 256, id="glm53_flash_nope"), + ], +) def test_sparse_backend_decode_correctness( default_vllm_config, dist_init, @@ -424,8 +432,18 @@ def test_sparse_backend_decode_correctness( workspace_init, q_scale: float, k_scale: float, + total_num_heads: int, + qk_nope_head_dim: int, + qk_rope_head_dim: int, + v_head_dim: int, monkeypatch, ): + if qk_rope_head_dim == 0 and ( + backend_cls != FlashMLASparseBackend + or kv_cache_dtype not in ("auto", "fp8_ds_mla") + or torch.cuda.get_device_capability()[0] != 9 + ): + pytest.skip("NoPE sparse MLA is served by FlashMLA on SM90 only") if ( batch_name == "large_q_pure_prefill" and backend_cls == FlashMLASparseBackend @@ -484,15 +502,10 @@ def test_sparse_backend_decode_correctness( device = torch.device(DEVICE_TYPE) dtype = torch.bfloat16 - # Model hyper-parameters (kept intentionally small for the unit test) - total_num_heads = 128 # Compute per-rank heads for simulated TP num_heads = max(1, total_num_heads // tensor_parallel_size) kv_lora_rank = 512 - qk_nope_head_dim = 128 - qk_rope_head_dim = 64 - v_head_dim = 128 head_size = kv_lora_rank + qk_rope_head_dim topk_tokens = 128 @@ -2565,7 +2578,7 @@ def test_hisparse_multi_step_writes_request_major_output(): @requires_hisparse_ops -def test_hisparse_kv_update_writes_resident_and_staging_caches(): +def test_hisparse_kv_update_writes_resident_cache(): device = torch.device(DEVICE_TYPE) block_size = 4 row_width = 8 @@ -2584,18 +2597,11 @@ def test_hisparse_kv_update_writes_resident_and_staging_caches(): block_table=torch.tensor([[1]], dtype=torch.int32, device=device), slot_mapping=resident_slots, ) - cache_handle.mirror_staging_cache = torch.empty( - (1, block_size, row_width), dtype=torch.float32, device=device - ) - cache_handle.mirror_staging_slots = torch.arange( - block_size, dtype=torch.int64, device=device - ) slots = torch.tensor([3, 7, -1], dtype=torch.int64, device=device) kv_c = torch.randn(8, row_width - 2, device=device) k_pe = torch.randn(8, 1, 2, device=device) cache_handle.num_actual_tokens = slots.numel() cache_handle.decode_batch = False - cache_handle.host_mirror_required = True source_cache = torch.zeros_like(cache_handle.view.cache) impl = object.__new__(FlashMLASparseImpl) layer = SimpleNamespace( @@ -2622,9 +2628,6 @@ def test_hisparse_kv_update_writes_resident_and_staging_caches(): expected.to(device), ) torch.testing.assert_close(source_cache, torch.zeros_like(source_cache)) - staged = cache_handle.mirror_staging_cache.view(-1, row_width) - staged_expected = torch.cat([kv_c[:3], k_pe[:3, 0]], dim=-1) - torch.testing.assert_close(staged[:3], staged_expected) @requires_hisparse_ops @@ -3293,7 +3296,8 @@ def test_hisparse_fp8_decode_resolves_rows_once_then_runs_batched_attention(): cache = SimpleNamespace( runtime=SimpleNamespace( - hot=SimpleNamespace(attention_cache=torch.empty(1, device=device)) + hot=SimpleNamespace(attention_cache=torch.empty(1, device=device)), + max_swap_rows=num_tokens, ), source_block_table=torch.empty( num_decodes, 1, dtype=torch.int32, device=device @@ -3601,15 +3605,22 @@ def test_flashinfer_sm120_hisparse_decode_uses_index_group(): def test_hisparse_resident_prefill_uses_attention_block_stride(): + """A batch beyond the swap-row capacity must bypass swap_in. + + The index-group workspace has one more row than the HiSparse swap rows, so + routing on its size sent a 2-token batch with max_num_seqs=1 (a piecewise + CUDA-graph capture) into swap_in, which overflowed its capacity of 1. + """ expected = torch.tensor([[19]], dtype=torch.int32) cache_handle = SimpleNamespace( all_context_pages_resident=True, view=SimpleNamespace(block_size=64, attention_block_stride=832), block_table=torch.tensor([[3]], dtype=torch.int32), + runtime=SimpleNamespace(max_swap_rows=1), ) index_group = object.__new__(HiSparseMLAIndexGroup) index_group.caches = [cache_handle] - index_group.physical_topk_indices = torch.empty((1, 1), dtype=torch.int32) + index_group.physical_topk_indices = torch.empty((2, 1), dtype=torch.int32) index_group._convert_once = MagicMock(return_value=expected) topk = torch.zeros((2, 1), dtype=torch.int32) metadata = SimpleNamespace( @@ -3809,6 +3820,7 @@ def test_flashmla_fp8_paths_accept_decode_subset(monkeypatch, use_mixed_batch: b topk_indices_buffer=topk_indices, num_heads=2, kv_lora_rank=1, + q_head_size=3, index_group=None, index_group_index=0, dcp_world_size=1, diff --git a/tests/v1/core/prefix_cache/test_partial_prefix_cache_hits.py b/tests/v1/core/prefix_cache/test_partial_prefix_cache_hits.py index 76beffce45..51317deb8d 100644 --- a/tests/v1/core/prefix_cache/test_partial_prefix_cache_hits.py +++ b/tests/v1/core/prefix_cache/test_partial_prefix_cache_hits.py @@ -22,6 +22,7 @@ from vllm.v1.core.kv_cache_utils import ( get_block_hash, get_group_id, init_none_hash, + resolve_cache_hit_alignment_tokens, ) from vllm.v1.core.sched.scheduler import Scheduler from vllm.v1.kv_cache_interface import ( @@ -148,29 +149,84 @@ def make_full_mamba_manager( ) -def test_dcp_fine_hit_retention_uses_hash_alignment_without_eagle(): - """DCP must not discard a reusable Mamba state at hash alignment.""" - hash_block_size = 2 +@pytest.mark.parametrize( + "dcp_world_size,hash_block_size,block_size,prompt_tokens,checkpoint_blocks", + [ + (4, 2, 2, 7, 0), + (8, 16, 16, 512, 1), + (8, 16, 16, 513, 1), + *[(8, 128, 896, p, 1) for p in (127, 128, 129, 7040, 7168, 7296, 7449)], + ], +) +@pytest.mark.parametrize("use_eagle", [False, True]) +@pytest.mark.parametrize("extension_tokens", [0, 32]) +def test_dcp_fine_hit_retention_uses_hash_alignment( + dcp_world_size, + hash_block_size, + block_size, + prompt_tokens, + checkpoint_blocks, + use_eagle, + extension_tokens, +): + """Resends and extensions reuse the retained hash-aligned checkpoint.""" manager = make_full_mamba_manager( - dcp_world_size=4, + dcp_world_size=dcp_world_size, hash_block_size=hash_block_size, - full_block_size=hash_block_size, - mamba_block_size=hash_block_size, - num_blocks=64, + full_block_size=block_size, + mamba_block_size=block_size, + num_blocks=128, + use_eagle=use_eagle, + num_speculative_blocks=4 if use_eagle else 0, + num_prefill_checkpoint_blocks=checkpoint_blocks, ) manager.coordinator.retention_interval = 0 - - token_ids = list(range(7)) - producer = make_request("producer", token_ids, hash_block_size, sha256) - computed_blocks, num_computed, _ = manager.get_computed_blocks(producer) - assert num_computed == 0 - assert manager.allocate_slots(producer, 6, 0, computed_blocks) is not None + assert manager.coordinator.enable_partial_hash_hits + scheduler = SimpleNamespace( + block_size=block_size * dcp_world_size, + cache_config=SimpleNamespace(block_size=block_size), + hash_block_size=hash_block_size, + max_num_scheduled_tokens=8192, + scheduler_config=SimpleNamespace(long_prefill_token_threshold=0), + use_eagle_block_drop=use_eagle, + mamba_partial_cache_hit=True, + mamba_shared_prefix_checkpoint=False, + mamba_has_prefill_checkpoint_blocks=bool(checkpoint_blocks), + mamba_prefill_checkpoint_alignment=16, + ) + producer = make_request( + "producer", list(range(prompt_tokens)), hash_block_size, sha256 + ) + while producer.num_computed_tokens < prompt_tokens: + scheduled = Scheduler._mamba_block_aligned_split( + scheduler, producer, prompt_tokens - producer.num_computed_tokens + ) + assert scheduled > 0 + manager.new_step_starts() + assert manager.allocate_slots(producer, scheduled) is not None + producer.num_computed_tokens += scheduled manager.free(producer) manager.new_step_starts() - consumer = make_request("consumer", token_ids, hash_block_size, sha256) - _, num_computed, _ = manager.get_computed_blocks(consumer) - assert num_computed == 6 + replay = make_request( + "replay", list(range(prompt_tokens + extension_tokens)), hash_block_size, sha256 + ) + _, hit, _ = manager.get_computed_blocks(replay) + expected = ( + max((prompt_tokens // hash_block_size - 1) * hash_block_size, 0) + if use_eagle + else ((prompt_tokens - 1) // hash_block_size * hash_block_size) + ) + assert hit == expected + cached_states = { + end + for end in range(hash_block_size, prompt_tokens + 1, hash_block_size) + if manager.block_pool.get_cached_block( + producer.block_hashes[end // hash_block_size - 1], [1] + ) + is not None + } + assert cached_states == ({expected} if expected else set()) @pytest.mark.parametrize("dcp_world_size", [1, 4]) @@ -339,7 +395,7 @@ def test_hybrid_mamba_align_partial_hash_hit(): hash_block_size=hash_block_size, ) - req0 = make_request("0", [0, 0, 1, 1, 2, 2], hash_block_size, sha256) + req0 = make_request("0", [0, 0, 1, 1, 2, 2, 3], hash_block_size, sha256) computed_blocks, num_computed, _ = manager.get_computed_blocks(req0) assert num_computed == 0 blocks = manager.allocate_slots(req0, 6, num_computed, computed_blocks) @@ -488,7 +544,7 @@ def test_hybrid_mamba_partial_tail_owner_uses_cow_on_continue(): hash_block_size=hash_block_size, ) - req0 = make_request("0", [0, 0, 1, 1, 2, 2], hash_block_size, sha256) + req0 = make_request("0", [0, 0, 1, 1, 2, 2, 3], hash_block_size, sha256) computed_blocks, num_computed, _ = manager.get_computed_blocks(req0) assert num_computed == 0 assert manager.allocate_slots(req0, 6, num_computed, computed_blocks) is not None @@ -538,7 +594,7 @@ def test_partial_hit_then_internal_checkpoint_uses_distinct_mamba_blocks(): num_prefill_checkpoint_blocks=1, ) - owner = make_request("owner", [0, 0, 1, 1, 2, 2], hash_block_size, sha256) + owner = make_request("owner", [0, 0, 1, 1, 2, 2, 3], hash_block_size, sha256) computed_blocks, num_computed, _ = manager.get_computed_blocks(owner) assert manager.allocate_slots(owner, 6, num_computed, computed_blocks) is not None manager.free(owner) @@ -619,44 +675,70 @@ def test_internal_checkpoint_uses_partial_hash_lifecycle(): assert num_computed == 112 -@pytest.mark.parametrize( - "retention_interval,transient_published", - [(None, True), (0, False)], -) +@pytest.mark.parametrize("retention_interval", [None, 0, 32, 64, 256]) +@pytest.mark.parametrize("hash_block_size", [16, 32]) +@pytest.mark.parametrize("use_eagle", [False, True]) def test_internal_checkpoint_publication_respects_retention( - retention_interval, transient_published + retention_interval, hash_block_size, use_eagle ): - """With retention_interval=0, a mid-prompt internal checkpoint is - request-local and must not enter the prefix cache; the prompt-end - checkpoint keeps the existing publication behavior. Dense retention - (None) is unchanged. - """ - hash_block_size = 16 + """Publish only retained block boundaries and the final prompt checkpoint.""" manager = make_full_mamba_manager( dcp_world_size=1, hash_block_size=hash_block_size, full_block_size=hash_block_size, mamba_block_size=32, num_prefill_checkpoint_blocks=1, + use_eagle=use_eagle, ) manager.coordinator.retention_interval = retention_interval request = make_request("producer", list(range(240)), hash_block_size, sha256) computed_blocks, num_computed, _ = manager.get_computed_blocks(request) - # Mid-prompt chunk: exports a transient checkpoint at 112. + # The EAGLE adjustment can make a chunk checkpoint block-aligned or partial. assert manager.allocate_slots(request, 128, num_computed, computed_blocks) - transient_hash = request.block_hashes[112 // hash_block_size - 1] + checkpoint = (128 if use_eagle else 127) // hash_block_size * hash_block_size - ( + hash_block_size if use_eagle else 0 + ) + retained_checkpoint = checkpoint % 32 == 0 and ( + retention_interval is None + or (retention_interval > 0 and checkpoint % retention_interval == 0) + ) + transient_hash = request.block_hashes[checkpoint // hash_block_size - 1] transient_hit = manager.block_pool.get_cached_block(transient_hash, [1]) - assert (transient_hit is not None) == transient_published + assert (transient_hit is not None) == retained_checkpoint + offloads = drain_boundary_state_offloads(manager).get(request.request_id, []) + assert any(position == checkpoint for _, _, position in offloads) == ( + retained_checkpoint + ) + assert all( + position % 32 == 0 + and ( + retention_interval is None + or (retention_interval > 0 and position % retention_interval == 0) + ) + for _, _, position in offloads + ) + if retention_interval in (None, 32, 64): + retained_hash = request.block_hashes[128 // hash_block_size - 1] + assert manager.block_pool.get_cached_block(retained_hash, [1]) is not None request.num_computed_tokens = 128 manager.new_step_starts() - # Final chunk: the prompt-end checkpoint at 224 stays published. + # The final checkpoint is reusable regardless of the retention interval. assert manager.allocate_slots(request, 112) is not None - end_hash = request.block_hashes[224 // hash_block_size - 1] + prompt_checkpoint = ( + 240 if use_eagle else 239 + ) // hash_block_size * hash_block_size - (hash_block_size if use_eagle else 0) + end_hash = request.block_hashes[prompt_checkpoint // hash_block_size - 1] assert manager.block_pool.get_cached_block(end_hash, [1]) is not None + assert any( + position == prompt_checkpoint + for _, _, position in drain_boundary_state_offloads(manager).get( + request.request_id, [] + ) + ) def test_transient_checkpoint_evicts_retained_boundary_hash(): @@ -802,9 +884,9 @@ def test_hash_aligned_query_end_uses_regular_partial_tail(): mamba_block_size=mamba_block_size, num_prefill_checkpoint_blocks=1, ) - request = make_request("producer", list(range(14)), hash_block_size, sha256) + request = make_request("producer", list(range(15)), hash_block_size, sha256) - new_blocks = manager.allocate_slots(request, request.num_tokens) + new_blocks = manager.allocate_slots(request, 14) assert new_blocks is not None mamba_manager = manager.coordinator.single_type_managers[1] @@ -928,7 +1010,7 @@ def test_boundary_state_offloads_returns_cow_target(): hash_block_size=hash_block_size, ) - req0 = make_request("0", [0, 0, 1, 1, 2, 2], hash_block_size, sha256) + req0 = make_request("0", [0, 0, 1, 1, 2, 2, 3], hash_block_size, sha256) computed_blocks, num_computed, _ = manager.get_computed_blocks(req0) assert manager.allocate_slots(req0, 6, num_computed, computed_blocks) is not None @@ -1004,7 +1086,7 @@ def test_finished_partial_tail_uses_table_source_once(): hash_block_size=hash_block_size, ) - req0 = make_request("0", [0, 0, 1, 1, 2, 2], hash_block_size, sha256) + req0 = make_request("0", [0, 0, 1, 1, 2, 2, 3], hash_block_size, sha256) computed_blocks, num_computed, _ = manager.get_computed_blocks(req0) assert manager.allocate_slots(req0, 6, num_computed, computed_blocks) is not None source_block_id = manager.get_blocks("0").get_block_ids()[1][1] @@ -1024,7 +1106,7 @@ def test_finished_partial_tail_uses_table_source_once(): enable_caching=True, hash_block_size=hash_block_size, ) - req1 = make_request("1", [0, 0, 1, 1, 2, 2], hash_block_size, sha256) + req1 = make_request("1", [0, 0, 1, 1, 2, 2, 3], hash_block_size, sha256) computed_blocks, num_computed, _ = advanced_manager.get_computed_blocks(req1) assert ( advanced_manager.allocate_slots(req1, 6, num_computed, computed_blocks) @@ -1212,7 +1294,7 @@ def test_truncate_computed_blocks_preserves_sparse_prefix_positions(): enable_caching=True, hash_block_size=hash_block_size, ) - producer = make_request("producer", [0, 0, 1, 1, 2, 2], hash_block_size, sha256) + producer = make_request("producer", [0, 0, 1, 1, 2, 2, 3], hash_block_size, sha256) blocks, num_computed, _ = manager.get_computed_blocks(producer) assert manager.allocate_slots(producer, 6, num_computed, blocks) is not None manager.free(producer) @@ -1267,7 +1349,7 @@ def test_truncate_computed_blocks_allows_short_mamba_group_only(): enable_caching=True, hash_block_size=hash_block_size, ) - producer = make_request("producer", [0, 0, 1, 1, 2, 2], hash_block_size, sha256) + producer = make_request("producer", [0, 0, 1, 1, 2, 2, 3], hash_block_size, sha256) blocks, num_computed, _ = manager.get_computed_blocks(producer) assert manager.allocate_slots(producer, 6, num_computed, blocks) is not None manager.free(producer) @@ -1331,7 +1413,7 @@ def test_hybrid_mamba_partial_tail_owner_continue_preserves_later_hit(): hash_block_size=hash_block_size, ) - req0 = make_request("0", [0, 0, 1, 1, 2, 2], hash_block_size, sha256) + req0 = make_request("0", [0, 0, 1, 1, 2, 2, 3], hash_block_size, sha256) computed_blocks, num_computed, _ = manager.get_computed_blocks(req0) assert num_computed == 0 assert manager.allocate_slots(req0, 6, num_computed, computed_blocks) is not None @@ -1411,7 +1493,7 @@ def test_hybrid_mamba_moved_partial_entry_defers_same_step_hit(): hash_block_size=hash_block_size, ) - req0 = make_request("0", [0, 0, 1, 1, 2, 2], hash_block_size, sha256) + req0 = make_request("0", [0, 0, 1, 1, 2, 2, 3], hash_block_size, sha256) computed_blocks, num_computed, _ = manager.get_computed_blocks(req0) assert num_computed == 0 assert manager.allocate_slots(req0, 6, num_computed, computed_blocks) is not None @@ -1470,7 +1552,7 @@ def test_hybrid_full_attention_partial_hash_hit_uses_cow(): hash_block_size=hash_block_size, ) - req0 = make_request("0", [0, 0, 1, 1, 2, 2], hash_block_size, sha256) + req0 = make_request("0", [0, 0, 1, 1, 2, 2, 3], hash_block_size, sha256) computed_blocks, num_computed, _ = manager.get_computed_blocks(req0) assert num_computed == 0 assert manager.allocate_slots(req0, 6, num_computed, computed_blocks) is not None @@ -1544,7 +1626,7 @@ def test_hybrid_partial_hit_cow_target_starts_uncached(): hash_block_size=hash_block_size, ) - req0 = make_request("0", [0, 0, 1, 1, 2, 2], hash_block_size, sha256) + req0 = make_request("0", [0, 0, 1, 1, 2, 2, 3], hash_block_size, sha256) computed_blocks, num_computed, _ = manager.get_computed_blocks(req0) assert num_computed == 0 assert manager.allocate_slots(req0, 6, num_computed, computed_blocks) is not None @@ -1700,7 +1782,7 @@ def test_cow_retained_blocks_returned_for_release(): enable_caching=True, hash_block_size=hash_block_size, ) - req0 = make_request("0", [0, 0, 1, 1, 2, 2], hash_block_size, sha256) + req0 = make_request("0", [0, 0, 1, 1, 2, 2, 3], hash_block_size, sha256) computed_blocks, num_computed, _ = manager.get_computed_blocks(req0) assert manager.allocate_slots(req0, 6, num_computed, computed_blocks) is not None @@ -2133,6 +2215,72 @@ def test_hybrid_sliding_window_group_disables_partial_hash_hits(): assert len(computed_blocks.blocks[0]) * hash_block_size == num_computed +@pytest.mark.parametrize("with_sliding_window", [False, True]) +def test_resolved_cache_hit_alignment_matches_coordinator(with_sliding_window): + """Workers place Mamba checkpoints on the engine-resolved alignment, so it + must be where the coordinator lands hits, including when a group without + fine-grained lookups disables partial hits.""" + hash_block_size = 2 + mamba_block_size = 4 * hash_block_size + kv_cache_groups = [ + KVCacheGroupSpec( + ["full"], + FullAttentionSpec( + block_size=hash_block_size, + num_kv_heads=1, + head_size=1, + dtype=torch.float32, + ), + ), + KVCacheGroupSpec( + ["mamba"], + MambaSpec( + block_size=mamba_block_size, + shapes=(1, 1), + dtypes=(torch.float32,), + mamba_cache_mode="align", + ), + ), + ] + if with_sliding_window: + kv_cache_groups.append( + KVCacheGroupSpec( + ["swa"], + SlidingWindowSpec( + block_size=2 * hash_block_size, + num_kv_heads=1, + head_size=1, + dtype=torch.float32, + sliding_window=2 * hash_block_size, + ), + ) + ) + kv_cache_config = KVCacheConfig( + num_blocks=64, kv_cache_tensors=[], kv_cache_groups=kv_cache_groups + ) + coordinator = make_kv_cache_manager( + kv_cache_config=kv_cache_config, + max_model_len=8192, + enable_caching=True, + hash_block_size=hash_block_size, + ).coordinator + vllm_config = SimpleNamespace( + cache_config=SimpleNamespace(enable_prefix_caching=True), + parallel_config=SimpleNamespace(decode_context_parallel_size=1), + ) + + assert coordinator.enable_partial_hash_hits != with_sliding_window + assert ( + resolve_cache_hit_alignment_tokens( + kv_cache_config, + vllm_config, + coordinator.scheduler_block_size, + hash_block_size, + ) + == coordinator._cache_hit_alignment_tokens + ) + + def test_opted_out_scratch_group_keeps_partial_hash_hits(): hash_block_size = 2 mamba_block_size = 2 * hash_block_size @@ -2179,7 +2327,7 @@ def test_opted_out_scratch_group_keeps_partial_hash_hits(): assert manager.coordinator.enable_partial_hash_hits - req0 = make_request("0", [0, 0, 1, 1, 2, 2], hash_block_size, sha256) + req0 = make_request("0", [0, 0, 1, 1, 2, 2, 3], hash_block_size, sha256) computed_blocks, num_computed, _ = manager.get_computed_blocks(req0) assert manager.allocate_slots(req0, 6, num_computed, computed_blocks) is not None manager.free(req0) @@ -2240,7 +2388,7 @@ def test_kpool_tail_supports_128_token_partial_hash_hits(): assert manager.coordinator.enable_partial_hash_hits shared_prefix = [10] * (12 * hash_block_size) - req0 = make_request("0", shared_prefix, hash_block_size, sha256) + req0 = make_request("0", shared_prefix + [11], hash_block_size, sha256) computed_blocks, num_computed, _ = manager.get_computed_blocks(req0) assert ( manager.allocate_slots(req0, len(shared_prefix), num_computed, computed_blocks) @@ -2266,7 +2414,7 @@ def test_hybrid_partial_hash_hit_uses_cow_under_dcp(dcp_world_size: int): ) assert manager.coordinator.enable_partial_hash_hits - req0 = make_request("dcp-owner", [0, 0, 1, 1, 2, 2], 2, sha256) + req0 = make_request("dcp-owner", [0, 0, 1, 1, 2, 2, 3], 2, sha256) computed_blocks, num_computed, _ = manager.get_computed_blocks(req0) assert manager.allocate_slots(req0, 6, num_computed, computed_blocks) is not None manager.free(req0) @@ -2318,7 +2466,7 @@ def test_dcp_partial_hit_resumes_on_replicated_mamba_snapshot( assert manager.coordinator.single_type_managers[1].block_size == block_size prefix = list(range(12)) - req0 = make_request("snapshot-owner", prefix, block_size, sha256) + req0 = make_request("snapshot-owner", prefix + [12], block_size, sha256) computed_blocks, num_computed, _ = manager.get_computed_blocks(req0) assert manager.allocate_slots(req0, 12, num_computed, computed_blocks) is not None manager.free(req0) @@ -2355,7 +2503,7 @@ def test_dcp_joint_hit_is_bounded_by_replicated_mamba_snapshots(): mamba_block_size=block_size, ) prefix = list(range(12)) - req0 = make_request("joint-owner", prefix, block_size, sha256) + req0 = make_request("joint-owner", prefix + [12], block_size, sha256) computed_blocks, num_computed, _ = manager.get_computed_blocks(req0) assert manager.allocate_slots(req0, 8, num_computed, computed_blocks) is not None manager.new_step_starts() @@ -2702,6 +2850,7 @@ def test_checkpoint_reservation_matches_worker( mamba_block_size=mamba_block_size, checkpoint_alignment=16, drop_eagle_block=use_eagle, + cache_hit_alignment_tokens=mamba_manager.cache_hit_alignment_tokens, ) reserved = mamba_manager._checkpoints.get(request.request_id) if reserved is not None: diff --git a/tests/v1/core/test_kv_cache_utils.py b/tests/v1/core/test_kv_cache_utils.py index 7923e34d0f..53833ba837 100644 --- a/tests/v1/core/test_kv_cache_utils.py +++ b/tests/v1/core/test_kv_cache_utils.py @@ -60,6 +60,7 @@ from vllm.v1.core.kv_cache_utils import ( from vllm.v1.hisparse.layout import ( create_hisparse_layout, get_hisparse_kv_cache_groups, + get_hisparse_steady_state_concurrency, ) from vllm.v1.kv_cache_interface import ( ChunkedLocalAttentionSpec, @@ -198,6 +199,48 @@ def test_hisparse_hma_uses_resolved_gpu_block_size( assert scheduler_block_size == hash_block_size == gpu_block_size +def test_hisparse_steady_state_concurrency_excludes_spilled_resident_pages(): + """Running requests pin only their resident tail; the newest still needs its + full in-flight window to be admitted.""" + block_size = 16 + config = SimpleNamespace( + model_config=SimpleNamespace(max_model_len=10 * block_size), + parallel_config=SimpleNamespace(decode_context_parallel_size=1), + max_in_flight_tokens=10 * block_size, + ) + attn_spec = FullAttentionSpec( + block_size=block_size, num_kv_heads=1, head_size=64, dtype=torch.float16 + ) + page_size = attn_spec.page_size_bytes + kv_cache_config = KVCacheConfig( + num_blocks=64, + kv_cache_tensors=[], + kv_cache_groups=[ + KVCacheGroupSpec(["source"], attn_spec, host_resident=True), + KVCacheGroupSpec(["indexer"], attn_spec), + KVCacheGroupSpec( + ["resident"], + HiSparseResidentSpec(block_size=block_size, page_size=page_size), + ), + KVCacheGroupSpec( + ["hot"], + HiSparseHotSpec( + block_size=block_size, page_size=page_size, blocks_per_request=4 + ), + ), + ], + hisparse_host_num_blocks=100, + ) + + # Admission takes indexer 10 + resident 10 + hot 4 = 24 blocks and steady + # state 16, so 3 fit (2 x 16 + 24 <= 64) but not 4 (3 x 16 + 24 > 64). + assert get_hisparse_steady_state_concurrency(config, kv_cache_config) == 3.5 + # The worst-case bound charges all 10 resident pages. + assert get_max_concurrency_for_kv_cache_config( + config, kv_cache_config + ) == pytest.approx(64 / 24) + + @pytest.mark.parametrize("extra_blocks,ok", [(1, False), (2, True)]) def test_hisparse_host_pool_must_fit_max_model_len(monkeypatch, extra_blocks, ok): """Every page needs a host block, so one max_model_len request must fit.""" @@ -923,6 +966,24 @@ def test_free_kv_cache_block_queue_popleft_n(): assert block.next_free_block is None +@pytest.mark.parametrize("num_blocks", [1, 2, 16]) +def test_free_kv_cache_block_queue_popleft_n_drain_and_refill(num_blocks): + """Draining a batch must leave the sentinels ready for reuse.""" + blocks = [KVCacheBlock(block_id=i) for i in range(num_blocks)] + queue = FreeKVCacheBlockQueue(blocks) + + assert queue.popleft_n(num_blocks) == blocks + assert queue.num_free_blocks == 0 + assert queue.fake_free_list_head.next_free_block is queue.fake_free_list_tail + assert queue.fake_free_list_tail.prev_free_block is queue.fake_free_list_head + assert all(b.prev_free_block is None and b.next_free_block is None for b in blocks) + assert queue.popleft_n(0) == [] + + queue.append_n(blocks) + assert queue.popleft_n(num_blocks) == blocks + assert queue.num_free_blocks == 0 + + def test_free_kv_cache_block_queue_get_all_free_blocks(): # Create a list of KVCacheBlock objects blocks = [KVCacheBlock(block_id=i) for i in range(5)] @@ -1873,6 +1934,27 @@ def test_dcp_target_allocates_replicated_draft_independently( assert [len(group) for group in cached_blocks.blocks] == [1, 4] +def test_dcp_replicated_kpool_tail_keeps_block_interior_layout(monkeypatch): + """A replicated kpool tail ring is per-request state, not a draft group.""" + from vllm.v1.attention.backends.utils import resolve_kv_cache_layout + + monkeypatch.delenv("VLLM_KV_CACHE_LAYOUT", raising=False) + config = VllmConfig(model_config=ModelConfig(max_model_len=1024)) + config.parallel_config.decode_context_parallel_size = 4 + config.cache_config.kv_cache_layout = None + tail = KpoolTailSpec( + block_size=4, + num_kv_heads=2, + head_size=128, + head_size_v=0, + dtype=torch.bfloat16, + sliding_window=4, + dcp_sharded=False, + ) + layout = resolve_kv_cache_layout(config, [["LBHNC"]], [new_mla_spec(), tail]) + assert layout == KVCacheLayout.LBHNC + + @pytest.mark.parametrize("use_mla", [False, True]) def test_full_attention_merge_preserves_replicated_cache_geometry(use_mla): config = VllmConfig(model_config=ModelConfig(max_model_len=1024)) diff --git a/tests/v1/core/test_prefix_caching.py b/tests/v1/core/test_prefix_caching.py index 3972fe5230..a89fd085f1 100644 --- a/tests/v1/core/test_prefix_caching.py +++ b/tests/v1/core/test_prefix_caching.py @@ -594,8 +594,12 @@ def test_nixl_hisparse_full_block_import_keeps_a_writable_tail(num_tokens): failed_recving_kv_req_ids=set(), finished_recving_kv_req_ids={request.request_id}, prefix_replay_tokens=0, + prefix_replay_group_ids=(), ) scheduler._mark_prefix_replay = MethodType(Scheduler._mark_prefix_replay, scheduler) + scheduler._load_restores_replay_window = MethodType( + Scheduler._load_restores_replay_window, scheduler + ) Scheduler._update_waiting_for_remote_kv(scheduler, request) assert request.num_tokens - request.num_computed_tokens == 1 assert _allocate_scheduled(manager, request, num_new_tokens=1) is not None @@ -5043,19 +5047,10 @@ def test_hybrid_local_kv_retention_mtp_reuses_latest_boundary(): def test_hybrid_mamba_retention_mtp_resend_of_aligned_prompt(): - """An identical resend and a longer sibling resume at DIFFERENT positions. + """EAGLE resend and extension reuse the same retained checkpoint. - How far a lookup matches depends on who is asking. A resend of the same - prompt caps its lookup at ``num_tokens - 1`` (the last token is recomputed - for logits), while a sibling whose prompt merely starts with this one caps - above the prompt. The two coincide unless the prompt length is an exact - multiple of the alignment -- there they differ by one alignment unit, and - under the EAGLE drop BOTH are reachable. - - Retaining only the higher one leaves the resend with every retained state - above every candidate its lookup can produce, and the reconciled hit - collapses to 0 -- the same zero-hit failure sparse retention already fixes - at unaligned prompt lengths. + Attention may inspect the prompt's final block before dropping it, so an + aligned resend must not lose another block to the P - 1 lookup limit. """ block_size = 32 num_spec = 3 @@ -5093,16 +5088,12 @@ def test_hybrid_mamba_retention_mtp_resend_of_aligned_prompt(): use_eagle=True, ) - # 128 tokens, an exact multiple of the 32-token alignment. A longer sibling - # matches 128 and drops to 96; this prompt's own resend caps at 127, matches - # 96 and drops to 64. Both states must survive retention. + # Both requests prove 128 attention tokens, drop one block, and reuse 96. token_ids = [i for i in range(4) for _ in range(block_size)] req0 = make_request("0", token_ids, block_size, sha256) computed_blocks, num_computed_tokens, _ = manager.get_computed_blocks(req0) assert num_computed_tokens == 0 - # Prefill in block-aligned chunks the way the align-mode scheduler does: a - # state only materializes as a chunk's running-state block, so a - # single-shot prefill could not retain the lower one. + # Materialize the checkpoint through normal block-aligned prefill chunks. for chunk_end in (32, 64, 96, 128): blocks = manager.allocate_slots( req0, @@ -5114,8 +5105,7 @@ def test_hybrid_mamba_retention_mtp_resend_of_aligned_prompt(): assert blocks is not None req0.num_computed_tokens = chunk_end - # Block ``i`` ends at token ``(i + 1) * 32``, so positions 64 and 96 are - # mamba blocks 1 and 2. + # Both materialized replay checkpoints survive; resend and extension use 96. pool = manager.block_pool expected_mamba_cached = {1, 2} for i in range(4): @@ -5126,15 +5116,13 @@ def test_hybrid_mamba_retention_mtp_resend_of_aligned_prompt(): assert cached is None, f"mamba hash {i} should not be cached" manager.free(req0) - # The identical resend: full attention matches blocks 0-2 (96 tokens, capped - # by num_tokens - 1) and the EAGLE drop caps the candidate at 64. Without - # the lower state retained the reconciled hit would be 0. + # The identical resend must reuse the retained checkpoint, not miss to zero. req1 = make_request("1", token_ids, block_size, sha256) computed_blocks, num_computed_tokens, _ = manager.get_computed_blocks(req1) - assert num_computed_tokens == 2 * block_size - assert [len(blocks) for blocks in computed_blocks.blocks] == [2, 2] + assert num_computed_tokens == 3 * block_size + assert [len(blocks) for blocks in computed_blocks.blocks] == [3, 3] - # The longer sibling resumes one alignment unit higher, off the same prompt. + # The longer sibling reuses that same checkpoint. longer = make_request("2", token_ids + [9] * block_size, block_size, sha256) computed_blocks, num_computed_tokens, _ = manager.get_computed_blocks(longer) assert num_computed_tokens == 3 * block_size diff --git a/tests/v1/core/test_prefix_replay.py b/tests/v1/core/test_prefix_replay.py index da57185808..323413a86d 100644 --- a/tests/v1/core/test_prefix_replay.py +++ b/tests/v1/core/test_prefix_replay.py @@ -247,6 +247,33 @@ def test_remote_kv_hit_is_taken_in_whole_blocks(): assert swa_blocks[new_req.replay_start // BLOCK_SIZE] is not swa_manager._null_block +@pytest.mark.parametrize("is_async", [False, True]) +@pytest.mark.parametrize("matched", [WINDOW, 4 * BLOCK_SIZE - 1]) +def test_kv_load_carrying_window_does_not_replay(is_async, matched): + """A load of the request's own blocks (P/D) restores every transfer group, + window included, so the hit is taken as is: no replay, no cut to whole + blocks, and a hit no longer than the window is kept.""" + scheduler = _replay_scheduler( + use_kv_connector=MockKVConfig(matched_tokens=matched, is_async=is_async) + ) + request = create_requests( + num_requests=1, num_tokens=NUM_PROMPT_TOKENS, block_size=BLOCK_SIZE + )[0] + scheduler.connector.get_loaded_kv_cache_group_ids = lambda request: (FULL, SWA) + scheduler.add_request(request) + out = scheduler.schedule() + if is_async: + assert request.status == RequestStatus.WAITING_FOR_REMOTE_KVS + scheduler.update_from_output( + out, create_model_runner_output([], finished_recving={request.request_id}) + ) + out = scheduler.schedule() + new_req = _new_req_data(out, request) + assert new_req.num_computed_tokens == matched + assert new_req.replay_start == 0 + assert out.num_scheduled_tokens[request.request_id] == NUM_PROMPT_TOKENS - matched + + @pytest.mark.parametrize("via_connector", [False, True]) def test_hit_no_longer_than_window_is_ignored(via_connector): """Such a hit would be recomputed in full anyway. It is not adopted (an diff --git a/tests/v1/core/test_scheduler.py b/tests/v1/core/test_scheduler.py index 2451ea5ec0..615dd82a1c 100644 --- a/tests/v1/core/test_scheduler.py +++ b/tests/v1/core/test_scheduler.py @@ -3,7 +3,7 @@ import dataclasses from concurrent.futures import Future from types import SimpleNamespace -from unittest.mock import Mock +from unittest.mock import Mock, patch import numpy as np import pytest @@ -35,6 +35,7 @@ from vllm.multimodal.inputs import ( MultiModalKwargsItem, PlaceholderRange, ) +from vllm.pooling_params import PoolingParams from vllm.sampling_params import SamplingParams, StructuredOutputsParams from vllm.utils.hashing import sha256 from vllm.v1.core.encoder_cache_manager import EncoderCacheManager @@ -60,6 +61,7 @@ from vllm.v1.kv_cache_interface import ( KVCacheGroupSpec, MambaSpec, ) +from vllm.v1.metrics.cache_hit_source import CacheHitSource from vllm.v1.outputs import ( DraftTokenIds, ECConnectorOutput, @@ -278,6 +280,24 @@ def test_schedule(enable_prefix_caching: bool, prompt_logprobs: int | None): assert scheduler.running[i] == request +def test_pooling_chunked_prefill_can_finish_at_max_model_len(): + scheduler = create_scheduler( + max_num_seqs=1, max_model_len=8, max_num_batched_tokens=4, runner="pooling" + ) + assert scheduler.max_model_len == 8 + request = Request( + request_id="pool", + prompt_token_ids=[1] * 8, + sampling_params=None, + pooling_params=PoolingParams(task="embed"), + ) + scheduler.add_request(request) + + scheduler.schedule() + # The final chunk must reach max_model_len, not stop one token short. + assert scheduler.schedule().num_scheduled_tokens == {"pool": 4} + + def test_scheduler_stats_route_to_existing_output_client(): scheduler = create_scheduler() request = create_requests(num_requests=1)[0] @@ -688,6 +708,33 @@ def test_throttle_capacity_bound_guard_admits(): assert "b" in output.num_scheduled_tokens +def test_same_step_duplicate_encoder_input_stays_cached(): + scheduler = create_scheduler( + model="llava-hf/llava-1.5-7b-hf", + max_num_batched_tokens=1024, + max_model_len=2048, + ) + request = create_requests( + 1, + num_tokens=2000, + req_ids=["repeated"], + mm_hashes_list=[["image", "image", "image"]], + mm_positions=[ + [PlaceholderRange(offset=offset, length=576) for offset in (0, 600, 1300)] + ], + )[0] + scheduler.add_request(request) + + output = scheduler.schedule() + assert output.scheduled_encoder_inputs == {request.request_id: [0]} + _model_output(scheduler, output, [[]]) + + # The second occurrence is partially consumed; the third is not scheduled. + cache = scheduler.encoder_cache_manager + assert cache.get_cached_input_ids(request) == {1} + assert "image" not in cache.freeable + + def test_no_mm_input_chunking(): # Disable multimodal input chunking. scheduler = create_scheduler( @@ -2526,6 +2573,96 @@ def _step_until_kv_transfer_finished(scheduler: Scheduler, req_ids: list[str]): return initial_ecos +def _runner_output( + req_ids: list[str], finished_recving: set[str] | None = None +) -> ModelRunnerOutput: + return ModelRunnerOutput( + req_ids=req_ids, + req_id_to_index={req_id: i for i, req_id in enumerate(req_ids)}, + sampled_token_ids=[[1] for _ in req_ids], + kv_connector_output=KVConnectorOutput(finished_recving=finished_recving), + ) + + +@pytest.mark.parametrize( + "load_done_before_pause", [False, True], ids=["load-in-flight", "load-done"] +) +def test_pause_new_drains_async_kv_loads(load_done_before_pause: bool): + """A PAUSED_NEW (pause mode="wait") drain must finish requests that hold KV + blocks for an async load, keep blockless queued requests queued, and leave + no blocks behind, so the post-drain cache reset succeeds.""" + block_size = 16 + scheduler = create_scheduler( + enable_prefix_caching=True, + use_kv_connector=mock_kv(matched_tokens=2 * block_size, is_async=True), + block_size=block_size, + ) + loading, queued = create_requests( + num_requests=2, + num_tokens=3 * block_size, + max_tokens=2, + block_size=block_size, + req_ids=["loading", "queued"], + ) + scheduler.add_request(loading) + output = scheduler.schedule() + assert loading.status == RequestStatus.WAITING_FOR_REMOTE_KVS + pending_recv = {"loading"} + if load_done_before_pause: + scheduler.update_from_output(output, _runner_output([], pending_recv)) + pending_recv = None + else: + scheduler.update_from_output(output, _runner_output([])) + + scheduler.set_pause_state(PauseState.PAUSED_NEW) + scheduler.add_request(queued) + + num_steps = 0 + while scheduler.has_requests(): + num_steps += 1 + assert num_steps <= 4, "pause drain did not finish" + output = scheduler.schedule() + assert set(output.num_scheduled_tokens) <= {"loading"} + scheduler.update_from_output( + output, _runner_output(list(output.num_scheduled_tokens), pending_recv) + ) + pending_recv = None + + # What pause runs once drained; it raises if the load still holds blocks. + assert scheduler.reset_prefix_cache(reset_running_requests=True) + assert loading.status == RequestStatus.FINISHED_LENGTH_CAPPED + assert loading.request_id not in scheduler.requests + assert list(scheduler.waiting) == [queued] + assert queued.status == RequestStatus.WAITING + + scheduler.set_pause_state(PauseState.UNPAUSED) + scheduler.schedule() + assert queued.status == RequestStatus.WAITING_FOR_REMOTE_KVS + + +def test_pause_new_ignores_streaming_session_waiting_for_input(): + """A session idle in WAITING_FOR_STREAMING_REQ cannot progress, so it must + not keep a PAUSED_NEW drain from completing.""" + stop_token = 7 + scheduler = create_scheduler() + (session,) = create_requests( + num_requests=1, num_tokens=4, stop_token_ids=[stop_token], req_ids=["s"] + ) + session.resumable = True + scheduler.add_request(session) + output = scheduler.schedule() + scheduler.update_from_output( + output, + ModelRunnerOutput( + req_ids=["s"], req_id_to_index={"s": 0}, sampled_token_ids=[[stop_token]] + ), + ) + assert session in scheduler.kv_holding_waiting + + scheduler.set_pause_state(PauseState.PAUSED_NEW) + assert scheduler.get_num_unfinished_requests() == 0 + + @pytest.mark.parametrize( ("load_modes", "expected_has_sync_loads"), [ @@ -2586,6 +2723,65 @@ def test_kv_connector_honors_skip_reading_prefix_cache(): assert output.num_scheduled_tokens[scoring.request_id] == BLOCK_SIZE * 4 +@pytest.mark.parametrize("is_async", [False, True]) +def test_kv_connector_records_external_cache_hit_sources(monkeypatch, is_async): + block_size = 16 + num_matched_tokens = 2 * block_size + scheduler = create_scheduler( + enable_prefix_caching=True, + use_kv_connector=mock_kv( + matched_tokens=num_matched_tokens, + is_async=is_async, + ), + block_size=block_size, + ) + request = create_requests( + num_requests=1, + num_tokens=2 * num_matched_tokens, + block_size=block_size, + )[0] + assert scheduler.connector is not None + get_sources = Mock( + return_value={CacheHitSource.P2P: block_size, CacheHitSource.HOST: block_size} + ) + update_state = Mock(wraps=scheduler.connector.update_state_after_alloc) + calls = Mock() + calls.attach_mock(update_state, "allocated") + calls.attach_mock(get_sources, "sources") + monkeypatch.setattr(scheduler.connector, "update_state_after_alloc", update_state) + monkeypatch.setattr( + scheduler.connector, "get_external_cache_hit_sources", get_sources + ) + + scheduler.add_request(request) + # A failed allocation is not an admission: nothing is attributed. + with patch.object(scheduler.kv_cache_manager, "allocate_slots", return_value=None): + scheduler.schedule() + get_sources.assert_not_called() + update_state.assert_not_called() + + scheduler.schedule() + # Attribution runs once, after the connector has built its load plan. + get_sources.assert_called_once_with(request, num_matched_tokens) + assert [c[0] for c in calls.mock_calls] == ["allocated", "sources"] + connector_stats = scheduler.connector_prefix_cache_stats + assert connector_stats is not None + assert connector_stats.hits == num_matched_tokens + assert connector_stats.hits_by_source == { + CacheHitSource.P2P: block_size, + CacheHitSource.HOST: block_size, + } + + if is_async: + # Re-admission after the async load completes is not a new hit. + scheduler.make_stats() + _step_until_kv_transfer_finished(scheduler, [request.request_id]) + get_sources.assert_called_once() + connector_stats = scheduler.connector_prefix_cache_stats + assert connector_stats is not None + assert connector_stats.hits_by_source == {} + + @pytest.mark.parametrize("is_async", [False, True]) def test_kv_connector_basic(is_async: bool): """Test whether Scheduler with KVConnector schedules tokens, allocates diff --git a/tests/v1/core/test_single_type_kv_cache_manager.py b/tests/v1/core/test_single_type_kv_cache_manager.py index da7ab742b0..384a41837f 100644 --- a/tests/v1/core/test_single_type_kv_cache_manager.py +++ b/tests/v1/core/test_single_type_kv_cache_manager.py @@ -6,10 +6,14 @@ import random import pytest import torch +from vllm.sampling_params import SamplingParams +from vllm.utils.hashing import sha256 from vllm.v1.core.block_pool import BlockPool from vllm.v1.core.kv_cache_utils import ( BlockHash, KVCacheBlock, + get_request_block_hasher, + init_none_hash, make_block_hash_with_group_id, ) from vllm.v1.core.single_type_kv_cache_manager import ( @@ -30,6 +34,7 @@ from vllm.v1.kv_cache_interface import ( RSWASpec, SlidingWindowSpec, ) +from vllm.v1.request import Request pytestmark = pytest.mark.cpu_test @@ -238,6 +243,71 @@ def test_mamba_checkpoint_admission_matches_allocation( assert admission_estimate == allocation_estimate == allocated +@pytest.mark.parametrize("num_speculative_blocks", [1, 2, 3]) +@pytest.mark.parametrize("first_chunk_blocks", [2, 4]) +@pytest.mark.parametrize("retention_interval", [0, None]) +def test_mamba_checkpoint_hash_maps_only_to_checkpoint_block( + num_speculative_blocks, first_chunk_blocks, retention_interval +): + """A checkpoint step must not leave a never-written speculative block + cached.""" + block_size = 128 + spec = MambaSpec( + block_size=block_size, + shapes=((1, 1),), + dtypes=(torch.float32,), + mamba_cache_mode="align", + num_speculative_blocks=num_speculative_blocks, + num_prefill_checkpoint_blocks=1, + prefill_checkpoint_alignment=16, + ) + pool = BlockPool(num_gpu_blocks=64, enable_caching=True, hash_block_size=128) + manager = MambaManager( + spec, + block_pool=pool, + enable_caching=True, + kv_cache_group_id=0, + scheduler_block_size=block_size, + ) + # MTP/EAGLE moves the checkpoint one block below the last full block. + manager.drop_eagle_checkpoint_block = True + init_none_hash(sha256) + prompt_tokens = 6 * block_size + 50 + request = Request( + request_id="r", + prompt_token_ids=list(range(prompt_tokens)), + sampling_params=SamplingParams(max_tokens=1), + pooling_params=None, + block_hasher=get_request_block_hasher(block_size, sha256), + ) + + def step(start, end): + estimate = manager.get_num_blocks_to_allocate("r", end, [], start, start, end) + free_before = pool.get_num_free_blocks() + manager.allocate_new_blocks("r", end, end) + assert free_before - pool.get_num_free_blocks() == estimate + manager.cache_blocks( + request, + end, + retention_interval=retention_interval, + replay_boundaries=[5 * block_size], + ) + request.num_computed_tokens = end + + step(0, first_chunk_blocks * block_size) + speculative_blocks = manager.req_to_blocks["r"][-num_speculative_blocks:] + manager.new_step_starts() + step(first_chunk_blocks * block_size, prompt_tokens) + + checkpoint_position, checkpoint_idx = manager._checkpoints["r"] + checkpoint_block = manager.req_to_blocks["r"][checkpoint_idx] + assert all( + b.block_hash is None or b is checkpoint_block for b in speculative_blocks + ) + block_hash = request.block_hashes[checkpoint_position // block_size - 1] + assert pool.get_cached_block(block_hash, [0]) == [checkpoint_block] + + def get_sliding_window_manager( sliding_window_spec, block_pool, diff --git a/tests/v1/core/utils.py b/tests/v1/core/utils.py index 49aaca3d7c..b5e1bf8932 100644 --- a/tests/v1/core/utils.py +++ b/tests/v1/core/utils.py @@ -18,6 +18,7 @@ from vllm.config import ( SpeculativeConfig, VllmConfig, ) +from vllm.config.model import RunnerOption from vllm.config.scheduler import SchedulerPolicy from vllm.multimodal.inputs import ( MultiModalFeatureSpec, @@ -78,6 +79,7 @@ def create_scheduler( ec_role: str | None = None, use_v2_model_runner: bool | None = None, kv_cache_spec: KVCacheSpec | None = None, + runner: RunnerOption = "auto", per_request_spec_decode_metrics: str = "none", scheduling_policy: SchedulerPolicy = "fcfs", diffusion_canvas_length: int | None = None, @@ -109,6 +111,7 @@ def create_scheduler( # The scheduler reads model_config.max_model_len, not the # SchedulerConfig one, so both must agree. max_model_len=max_model_len, + runner=runner, ) if mm_encoder_only: model_config.multimodal_config.mm_encoder_only = True diff --git a/tests/v1/cudagraph/test_cudagraph_manager.py b/tests/v1/cudagraph/test_cudagraph_manager.py index 671172573f..87d4360e4c 100644 --- a/tests/v1/cudagraph/test_cudagraph_manager.py +++ b/tests/v1/cudagraph/test_cudagraph_manager.py @@ -20,9 +20,13 @@ from vllm.distributed.device_communicators import pynccl_allocator from vllm.v1.kv_cache_interface import KVCacheConfig from vllm.v1.worker.gpu import cudagraph_utils as gpu_cudagraph_utils from vllm.v1.worker.gpu.cudagraph_utils import BatchExecutionDescriptor +from vllm.v1.worker.gpu.dp_utils import dispatch_cg_and_sync_dp from vllm.v1.worker.gpu.input_batch import InputBuffers from vllm.v1.worker.gpu.pcp_manager import PCPManager -from vllm.v1.worker.gpu.spec_decode.autoregressive.cudagraph_utils import ( +from vllm.v1.worker.gpu.spec_decode.target_dependent_ar import ( + cudagraph_utils as spec_cudagraph_utils, +) +from vllm.v1.worker.gpu.spec_decode.target_dependent_ar.cudagraph_utils import ( SpeculatorCudaGraphManager, ) @@ -226,8 +230,10 @@ def test_speculator_capture_preserves_decode_query_bounds( ) manager = SpeculatorCudaGraphManager.__new__(SpeculatorCudaGraphManager) manager.max_num_reqs = num_reqs + manager.decode_query_len = uniform_token_count or max_query_len or num_tokens manager.vllm_config = SimpleNamespace( - parallel_config=SimpleNamespace(data_parallel_size=1) + parallel_config=SimpleNamespace(data_parallel_size=1), + num_speculative_tokens=3, ) buffers = InputBuffers(num_reqs, num_tokens, torch.device("cpu")) block_tables = MagicMock() @@ -488,6 +494,78 @@ def test_dynamic_spec_decode_shared_token_count_stays_reachable(monkeypatch): ), desc +@pytest.mark.parametrize("specialize", [False, True]) +def test_speculator_replays_graph_for_runtime_width(monkeypatch, specialize): + """Equal batch shapes must replay the graph captured for the selected K.""" + monkeypatch.setattr( + gpu_cudagraph_utils, + "get_pp_group", + lambda: SimpleNamespace(is_first_rank=True, is_last_rank=True), + ) + monkeypatch.setattr( + gpu_cudagraph_utils.current_platform, "get_global_graph_pool", lambda: object() + ) + monkeypatch.setattr(gpu_cudagraph_utils, "get_offloader", MagicMock) + monkeypatch.setattr( + spec_cudagraph_utils, "prepare_inputs_to_capture", lambda *a, **kw: ({}, {}) + ) + forward = MagicMock() + replayed_widths: list[int] = [] + + def capture(manager, create_forward_fn, progress_bar_desc): + for desc in manager._capture_descs[CUDAGraphMode.FULL]: + create_forward_fn(desc, warmup=False)(CUDAGraphMode.NONE) + k = forward.call_args.kwargs["num_speculative_steps"] + graph = MagicMock() + graph.replay.side_effect = lambda k=k: replayed_widths.append(k) + manager.graphs[desc] = graph + manager._graphs_captured = True + + monkeypatch.setattr(gpu_cudagraph_utils.CudaGraphManager, "capture", capture) + manager = spec_cudagraph_utils.SpeculatorCudaGraphManager( + _create_decode_vllm_config( + capture_sizes=[4], + num_speculative_tokens=4, + dynamic_spec_schedule=[(1, 2, 4), (3, 4, 2), (5, 6, 1), (7, 8, 0)], + ), + device=torch.device("cpu"), + cudagraph_mode=CUDAGraphMode.FULL_DECODE_ONLY, + decode_query_len=1, + ) + manager.capture( + forward, + MagicMock(), + MagicMock(), + MagicMock(), + [], + MagicMock(), + specialize_spec_tokens=specialize, + ) + captured_widths = [4, 2] if specialize else [4] + assert [c.kwargs["num_speculative_steps"] for c in forward.call_args_list] == ( + captured_widths + ) + + for k in (2, 4, 2, 3): + desc, _ = dispatch_cg_and_sync_dp( + manager, + num_reqs=3, + num_tokens=3, + uniform_token_count=1, + dp_size=1, + dp_rank=0, + ) + desc = manager.specialize_spec_tokens(desc, k) + captured_k = k if specialize else 4 + if captured_k in captured_widths: + assert desc.cg_mode == CUDAGraphMode.FULL + assert desc.num_tokens == 4 + manager.run_fullgraph(desc) + assert replayed_widths[-1] == captured_k + else: + assert desc.cg_mode == CUDAGraphMode.NONE + + @pytest.mark.parametrize( "decode_query_len,capture_sizes", [(1, [1, 2, 4, 8]), (2, [2, 4, 8, 16]), (8, [8, 16, 32, 64])], diff --git a/tests/v1/cudagraph/test_encoder_cudagraph.py b/tests/v1/cudagraph/test_encoder_cudagraph.py index 749f9aa20d..3489443878 100644 --- a/tests/v1/cudagraph/test_encoder_cudagraph.py +++ b/tests/v1/cudagraph/test_encoder_cudagraph.py @@ -245,6 +245,138 @@ class TestFindBudgetGraph: assert mgr.get_num_graphs_to_capture() == 3 +# --------------------------------------------------------------------------- +# _execute_local greedy packing with budget-cliff deferral (no GPU required) +# --------------------------------------------------------------------------- + + +def _run_packing( + mgr: EncoderCudaGraphManager, specs: list[EncoderItemSpec] +) -> list[tuple[list[int], str, int]]: + """Run _execute_local with graph replay and postprocess mocked out. + + Returns: + (batch_indices, path, token_budget) for every graph replay. + + """ + mgr.model = MagicMock() + + runs: list[tuple[list[int], str, int]] = [] + + def fake_run(mm_kwargs, token_budget, path="default", axis_keys=()): + runs.append((list(mm_kwargs["indices"]), path, token_budget)) + n_tokens = sum( + specs[i].get_path_output_tokens(path) for i in mm_kwargs["indices"] + ) + return torch.zeros(n_tokens, 32) + + def fake_postprocess( + graph_outputs, + batch_indices, + per_item_out_tokens, + outputs_by_orig_idx, + clone, + batch_mm_kwargs, + ): + for i in batch_indices: + outputs_by_orig_idx[i] = torch.zeros(per_item_out_tokens[i], 32) + + mgr.model.postprocess_encoder_output = fake_postprocess + with ( + patch.object(mgr, "_get_item_specs", lambda mm_kwargs: specs), + patch.object( + mgr, + "_select_items", + lambda mm_kwargs, indices: ({"indices": indices}, ()), + ), + patch.object(mgr, "_run_budget_graph", fake_run), + ): + result = mgr._execute_local({}) + assert len(result) == len(specs) + return runs + + +class TestExecuteLocalPacking: + """Greedy packing defers items that cross a budget cliff.""" + + @staticmethod + def _spec(tokens: int) -> EncoderItemSpec: + return EncoderItemSpec(input_size=tokens, output_tokens=tokens) + + def test_cliff_deferral_splits_batch(self): + # Merging would force 4200 tokens into the 8192 budget; deferring + # the 4000-token item costs 256 + 4096 instead. + mgr = _make_manager_with_budgets([256, 4096, 8192]) + runs = _run_packing(mgr, [self._spec(4000), self._spec(200)]) + assert runs == [([1], "default", 256), ([0], "default", 4096)] + + def test_tie_does_not_defer(self): + # 4096 + 4096 ties the merged 8192 budget; merging wins because it + # saves one replay. + mgr = _make_manager_with_budgets([256, 4096, 8192]) + runs = _run_packing(mgr, [self._spec(4000), self._spec(4000)]) + assert runs == [([0, 1], "default", 8192)] + + def test_deferral_requires_item_outweighing_batch(self): + # The third 1366-token item would save budget if deferred (4096+2048 + # < 8192) but is smaller than the current batch (4096), so it merges: + # deferring mid-size items fragments flood packing into extra replays. + mgr = _make_manager_with_budgets([256, 2048, 4096, 8192]) + runs = _run_packing(mgr, [self._spec(1366)] * 3) + assert runs == [([0, 1, 2], "default", 8192)] + + def test_packing_without_cliff_unchanged(self): + mgr = _make_manager_with_budgets([256, 4096]) + runs = _run_packing(mgr, [self._spec(100), self._spec(100)]) + assert runs == [([0, 1], "default", 256)] + + def test_small_items_split_to_avoid_cliff(self): + # Merging 3+ items of 100 tokens crosses 256 -> 4096; splitting + # into pairs costs 256 + 256 < 4096. + mgr = _make_manager_with_budgets([256, 4096]) + runs = _run_packing(mgr, [self._spec(100)] * 4) + assert runs == [([0, 1], "default", 256), ([2, 3], "default", 256)] + + def test_max_batch_size_split_unchanged(self): + mgr = _make_manager_with_budgets([1024, 4096]) + mgr.max_batch_size = 3 + runs = _run_packing(mgr, [self._spec(100)] * 4) + assert runs == [([0, 1, 2], "default", 1024), ([3], "default", 1024)] + + def test_oversized_item_still_falls_back_to_eager(self): + mgr = _make_manager_with_budgets([256, 4096]) + runs = _run_packing(mgr, [self._spec(100), self._spec(9000)]) + assert runs == [([0], "default", 256)] + assert mgr.graph_misses == 1 + + def test_multi_path_deferral_skips_zero_token_paths(self): + mgr = _make_manager_with_budgets([256, 4096, 8192]) + mgr.path_token_budgets = { + "a": [0, 256, 4096, 8192], + "b": [256, 4096, 8192], + } + specs = [ + EncoderItemSpec( + input_size=300, + output_tokens=300, + path_output_tokens={"a": 200, "b": 100}, + ), + EncoderItemSpec( + input_size=4000, + output_tokens=4000, + path_output_tokens={"a": 0, "b": 4000}, + ), + ] + runs = _run_packing(mgr, specs) + # Path b alone triggers deferral (256 + 4096 < 8192); the deferred + # item has zero tokens on path a, so that path is skipped. + assert runs == [ + ([0], "a", 256), + ([0], "b", 256), + ([1], "b", 4096), + ] + + # --------------------------------------------------------------------------- # get_cumulative_stats # --------------------------------------------------------------------------- diff --git a/tests/v1/distributed/test_async_llm_dp.py b/tests/v1/distributed/test_async_llm_dp.py index cb23910566..f4b4ca2a79 100644 --- a/tests/v1/distributed/test_async_llm_dp.py +++ b/tests/v1/distributed/test_async_llm_dp.py @@ -132,6 +132,26 @@ async def test_load( ) after.callback(engine.shutdown) + # The first request an engine serves pays a few hundred milliseconds of + # one-off setup, and while it does the engine holds the requests already + # routed to it while reporting itself idle, which is enough to skew the + # split asserted on below. Pay it on every engine first, then measure. + await asyncio.gather( + *( + generate( + engine, + f"warmup-{rank}", + prompt, + output_kind, + max_tokens=1, + data_parallel_rank=rank, + ) + for rank in range(DP_SIZE) + ) + ) + for sl in stats_loggers.values(): + sl.finished_req_count = 0 + NUM_REQUESTS = 100 NUM_EXPECTED_TOKENS = 10 @@ -699,7 +719,7 @@ async def test_dp_pause_barrier_request_deadlock(): # Drive the staggered barrier. Old code deadlocks here. try: await asyncio.wait_for(client.call_utility_async("barrier"), timeout=30) - except asyncio.TimeoutError: + except TimeoutError: for t in mid_barrier_tasks: t.cancel() pytest.fail( @@ -727,12 +747,13 @@ async def test_dp_pause_wait_mode_drains_in_flight(): prompt=DP_PAUSE_PROMPT, params=SamplingParams(max_tokens=64), ) + # Wait for the first output, so drain has something to observe: a + # request still queued holds no blocks and is not waited on. + out = await asyncio.wait_for(collector.get(), timeout=30) await engine.pause_generation(mode="wait") assert await engine.is_paused() - while True: + while not out.finished: out = await asyncio.wait_for(collector.get(), timeout=30) - if out.finished: - break await engine.resume_generation() async for out in engine.generate( diff --git a/tests/v1/e2e/general/accuracy_utils.py b/tests/v1/e2e/general/accuracy_utils.py index c8fa770fc5..e15ef03ebc 100644 --- a/tests/v1/e2e/general/accuracy_utils.py +++ b/tests/v1/e2e/general/accuracy_utils.py @@ -110,8 +110,10 @@ def check_first_divergence( # Log-softmax and reconstructing penalties may round in FP32, but # a near tie cannot excuse selecting the engine's own worse token. assert gap >= -1e-5, f"{detail}: selected token is below its alternative" - assert gap <= tolerance.greedy_atol, ( - f"{detail}: divergence exceeds near-tie bound={tolerance.greedy_atol}" + # Same FP32 rounding slack as above, applied to the upper bound. + assert gap <= tolerance.greedy_atol + 1e-5, ( + f"{detail}: divergence exceeds near-tie bound=" + f"{tolerance.greedy_atol} (+1e-5 slack)" ) @@ -201,9 +203,10 @@ def check_greedy_token( ) best = int(logits.argmax()) gap = float(logits[best] - logits[token]) - assert gap <= tolerance.greedy_atol, ( + # FP32 rounding can push a true near-tie fractionally past the bound. + assert gap <= tolerance.greedy_atol + 1e-5, ( f"{context}: selected token={token}, reference best={best}, " - f"logit gap={gap}, bound={tolerance.greedy_atol}" + f"logit gap={gap}, bound={tolerance.greedy_atol} (+1e-5 slack)" ) return gap diff --git a/tests/v1/e2e/general/test_async_scheduling_accuracy.py b/tests/v1/e2e/general/test_async_scheduling_accuracy.py index aa785b3270..3ffcaefdc3 100644 --- a/tests/v1/e2e/general/test_async_scheduling_accuracy.py +++ b/tests/v1/e2e/general/test_async_scheduling_accuracy.py @@ -271,9 +271,9 @@ def run_tests( context = f"config=[{config}], params={params}" if spec_config.get("max_model_len") is None: # Keep the original acceptance-quality floor per batch. - relative_drop = ( - 0.10 if current_platform.is_rocm() and test_preemption else 0.05 - ) + # Preemption recomputes requests and loses n-gram drafts, + # so allow a larger drop there. + relative_drop = 0.10 if test_preemption else 0.05 assert actual >= baseline * (1 - relative_drop), ( f"{context}: acceptance={actual}, baseline={baseline}" ) diff --git a/tests/v1/e2e/test_replayssm_decode.py b/tests/v1/e2e/test_replayssm_decode.py index 1a369c781f..ed3c6b0217 100644 --- a/tests/v1/e2e/test_replayssm_decode.py +++ b/tests/v1/e2e/test_replayssm_decode.py @@ -25,6 +25,12 @@ PROMPTS = [ "The capital of France is", "Once upon a time, in a small village,", ] +# The NVFP4 model is not reproducible across engine instances; these prompts +# avoid near-ties and decode confidently past the replay window. +FLASHINFER_PROMPTS = [ + "The capital of France is", + "It was the best of times, it was the worst of times,", +] @pytest.fixture(autouse=True) @@ -43,6 +49,7 @@ def _check_replayssm_parity( *, tensor_parallel_size=1, mamba_backend: str = "triton", + prompts: list[str] = PROMPTS, name_1: str = "replayssm", expected_v2: bool | None = None, ): @@ -60,7 +67,7 @@ def _check_replayssm_parity( with vllm_runner(model_name, **common) as llm: if expected_v2 is not None: assert llm.llm.llm_engine.vllm_config.use_v2_model_runner is expected_v2 - baseline = llm.generate_greedy_logprobs(PROMPTS, max_tokens=32, num_logprobs=5) + baseline = llm.generate_greedy_logprobs(prompts, max_tokens=32, num_logprobs=5) with vllm_runner( model_name, use_replayssm=True, @@ -69,7 +76,7 @@ def _check_replayssm_parity( ) as llm: if expected_v2 is not None: assert llm.llm.llm_engine.vllm_config.use_v2_model_runner is expected_v2 - replay = llm.generate_greedy_logprobs(PROMPTS, max_tokens=32, num_logprobs=5) + replay = llm.generate_greedy_logprobs(prompts, max_tokens=32, num_logprobs=5) check_logprobs_close( outputs_0_lst=baseline, @@ -105,6 +112,7 @@ def test_replayssm_flashinfer_decode_matches_baseline( vllm_runner, model_name, mamba_backend="flashinfer", + prompts=FLASHINFER_PROMPTS, name_1="replayssm_flashinfer", expected_v2=use_v2_model_runner, ) @@ -127,7 +135,9 @@ def test_replayssm_flashinfer_spec_decode_matches_baseline(vllm_runner, model_na }, ) with vllm_runner(model_name, mamba_backend="flashinfer", **common) as llm: - baseline = llm.generate_greedy_logprobs(PROMPTS, max_tokens=32, num_logprobs=5) + baseline = llm.generate_greedy_logprobs( + FLASHINFER_PROMPTS, max_tokens=32, num_logprobs=5 + ) with vllm_runner( model_name, use_replayssm=True, @@ -135,7 +145,9 @@ def test_replayssm_flashinfer_spec_decode_matches_baseline(vllm_runner, model_na mamba_backend="flashinfer", **common, ) as llm: - replay = llm.generate_greedy_logprobs(PROMPTS, max_tokens=32, num_logprobs=5) + replay = llm.generate_greedy_logprobs( + FLASHINFER_PROMPTS, max_tokens=32, num_logprobs=5 + ) check_logprobs_close( outputs_0_lst=baseline, diff --git a/tests/v1/ec_connector/unit/test_scheduler_nixl_ctor.py b/tests/v1/ec_connector/unit/test_scheduler_nixl_ctor.py index 7d2622c44e..8cad9ec65c 100644 --- a/tests/v1/ec_connector/unit/test_scheduler_nixl_ctor.py +++ b/tests/v1/ec_connector/unit/test_scheduler_nixl_ctor.py @@ -1,10 +1,12 @@ # SPDX-License-Identifier: Apache-2.0 # SPDX-FileCopyrightText: Copyright contributors to the vLLM project import uuid +from pathlib import Path import pytest import vllm.distributed.ec_transfer.ec_connector.cpu.scheduler as sched_mod +import vllm.distributed.nixl_utils as nixl_utils from tests.v1.ec_connector.unit.utils import create_ec_vllm_config from vllm.config.ec_transfer import ECTransferConfig from vllm.distributed.ec_transfer.ec_connector.cpu.ec_shared_region import ( @@ -57,6 +59,72 @@ def test_string_false_in_extra_config_leaves_nixl_off(monkeypatch): s.shutdown() +@pytest.mark.cpu_test +def test_missing_nixl_releases_shared_region(monkeypatch): + """A failed constructor leaves no scheduler for the caller to shut down.""" + # Import before patching to avoid caching the missing-NIXL values. + from vllm.distributed.ec_transfer.ec_connector.cpu.data import nixl # noqa: F401 + + region = _region() + monkeypatch.setattr(sched_mod, "create_ec_shared_region", lambda cfg: region) + monkeypatch.setattr(nixl_utils, "NixlWrapper", None) + monkeypatch.setattr(nixl_utils, "nixl_agent_config", None) + cfg = create_ec_vllm_config(ec_role="ec_both") + cfg.ec_transfer_config.ec_connector_extra_config["ec_enable_nixl"] = True + + try: + with pytest.raises(RuntimeError, match="ec_enable_nixl requires NIXL"): + ECCPUScheduler(cfg) + + assert not Path(region._mmap_path).exists() + assert region._fd is None + assert region._mmap_obj is None + finally: + region.cleanup() + + +@pytest.mark.cpu_test +def test_control_transport_failure_releases_nixl_and_shared_region(monkeypatch): + import vllm.distributed.ec_transfer.ec_connector.cpu.control.zmq as zmq_mod + import vllm.distributed.ec_transfer.ec_connector.cpu.data.nixl as data_mod + + class RegisteredData: + registered = True + + def deregister(self): + self.registered = False + + data = RegisteredData() + error = RuntimeError("control transport failed") + + def fail_control_transport(): + raise error + + region = _region() + monkeypatch.setattr(sched_mod, "create_ec_shared_region", lambda cfg: region) + monkeypatch.setattr(nixl_utils, "NixlWrapper", object()) + monkeypatch.setattr(nixl_utils, "nixl_agent_config", object()) + monkeypatch.setattr(data_mod, "NixlDataTransport", lambda **kwargs: data) + monkeypatch.setattr(zmq_mod, "ZmqClientTransport", fail_control_transport) + cfg = create_ec_vllm_config(ec_role="ec_consumer") + cfg.model_config.hf_config = None + cfg.model_config.get_inputs_embeds_size.return_value = 8 + cfg.model_config.model = "test-model" + cfg.ec_transfer_config.ec_connector_extra_config["ec_enable_nixl"] = True + + try: + with pytest.raises(RuntimeError, match="control transport failed") as exc: + ECCPUScheduler(cfg) + + assert exc.value is error + assert not data.registered + assert not Path(region._mmap_path).exists() + assert region._fd is None + assert region._mmap_obj is None + finally: + region.cleanup() + + @pytest.mark.skipif(NixlWrapper is None, reason="Requires NIXL package") def test_gate_on_wires_data_transport_and_producer_session(monkeypatch): # Port 0 lets the OS pick an ephemeral port so the real ZMQ ROUTER bind diff --git a/tests/v1/engine/test_async_llm.py b/tests/v1/engine/test_async_llm.py index dc3cf1aa00..3ac687c918 100644 --- a/tests/v1/engine/test_async_llm.py +++ b/tests/v1/engine/test_async_llm.py @@ -2,8 +2,10 @@ # SPDX-FileCopyrightText: Copyright contributors to the vLLM project import asyncio +import threading import time from contextlib import ExitStack +from http import HTTPStatus from unittest.mock import AsyncMock, MagicMock, call import pytest @@ -11,7 +13,7 @@ import pytest import vllm.v1.engine.async_llm as async_llm_module from vllm import SamplingParams from vllm.assets.image import ImageAsset -from vllm.config import LoggingConfig, VllmConfig +from vllm.config import LoggingConfig, ProfilerConfig, VllmConfig from vllm.engine.arg_utils import AsyncEngineArgs from vllm.entrypoints.openai.chat_completion.protocol import ( ChatCompletionRequest, @@ -20,10 +22,11 @@ from vllm.entrypoints.openai.chat_completion.protocol import ( from vllm.entrypoints.openai.chat_completion.serving import OpenAIServingChat from vllm.entrypoints.openai.models.protocol import BaseModelPath from vllm.entrypoints.openai.models.serving import OpenAIServingModels -from vllm.exceptions import VLLMValidationError +from vllm.exceptions import ProfilerAlreadyActiveError, VLLMValidationError from vllm.inputs import PromptType from vllm.outputs import RequestOutput from vllm.platforms import current_platform +from vllm.profiler.wrapper import WorkerProfiler from vllm.sampling_params import RequestOutputKind from vllm.utils.torch_utils import set_default_torch_num_threads from vllm.v1.engine.async_llm import AsyncLLM @@ -87,13 +90,19 @@ def _mock_async_llm_dependencies(monkeypatch: pytest.MonkeyPatch): return engine_core -def test_cuda_profiler_requests_reach_engine_core(monkeypatch: pytest.MonkeyPatch): +def _mock_vllm_config() -> MagicMock: vllm_config = MagicMock() vllm_config.logging_config = LoggingConfig() vllm_config.observability_config.otlp_traces_endpoint = None vllm_config.scheduler_config.stream_interval = 1 + return vllm_config + + +def test_cuda_profiler_requests_reach_engine_core(monkeypatch: pytest.MonkeyPatch): + vllm_config = _mock_vllm_config() vllm_config.profiler_config.profiler = "cuda" vllm_config.profiler_config.ignore_frontend = False + vllm_config.profiler_config.should_profile_frontend = False engine_core = _mock_async_llm_dependencies(monkeypatch) engine = AsyncLLM(vllm_config, MagicMock(), log_stats=False) @@ -105,22 +114,22 @@ def test_cuda_profiler_requests_reach_engine_core(monkeypatch: pytest.MonkeyPatc asyncio.run(profile()) assert engine.profiler is None - engine_core.profile_async.assert_has_awaits([call(True, None), call(False)]) + engine_core.profile_async.assert_has_awaits( + [call(True, None, None, None), call(False)] + ) def test_cuda_only_torch_profiler_skips_frontend_cpu_trace( monkeypatch: pytest.MonkeyPatch, ): - vllm_config = MagicMock() - vllm_config.logging_config = LoggingConfig() - vllm_config.observability_config.otlp_traces_endpoint = None - vllm_config.scheduler_config.stream_interval = 1 + vllm_config = _mock_vllm_config() vllm_config.profiler_config.profiler = "torch" vllm_config.profiler_config.ignore_frontend = False vllm_config.profiler_config.torch_profiler_activities = ["CUDA"] + vllm_config.profiler_config.should_profile_frontend = False _mock_async_llm_dependencies(monkeypatch) profiler = MagicMock() - monkeypatch.setattr(async_llm_module, "TorchProfilerWrapper", profiler) + monkeypatch.setattr(async_llm_module, "create_frontend_profiler", profiler) engine = AsyncLLM(vllm_config, MagicMock(), log_stats=False) @@ -128,6 +137,271 @@ def test_cuda_only_torch_profiler_skips_frontend_cpu_trace( profiler.assert_not_called() +def test_profile_forwards_overrides_rejects_duplicate_and_allows_restart( + monkeypatch: pytest.MonkeyPatch, +): + vllm_config = _mock_vllm_config() + vllm_config.profiler_config.should_profile_frontend = False + engine_core = _mock_async_llm_dependencies(monkeypatch) + + engine = AsyncLLM(vllm_config, MagicMock(), log_stats=False) + + async def profile(): + for kwargs in ( + {"profile_prefix": "../trace"}, + {"delay_iterations": -1}, + {"max_iterations": -1}, + ): + with pytest.raises(ValueError): + await engine.start_profile(**kwargs) + await engine.start_profile( + "session", + delay_iterations=5, + max_iterations=2, + ) + with pytest.raises(ProfilerAlreadyActiveError) as exc_info: + await engine.start_profile("duplicate") + assert exc_info.value.http_status == HTTPStatus.CONFLICT + await engine.stop_profile() + await engine.start_profile("next-session") + await engine.stop_profile() + + asyncio.run(profile()) + + engine_core.profile_async.assert_has_awaits( + [ + call(True, "session", 5, 2), + call(False), + call(True, "next-session", None, None), + call(False), + ] + ) + + +def test_stop_profile_waits_for_inflight_start(monkeypatch: pytest.MonkeyPatch): + vllm_config = _mock_vllm_config() + engine_core = _mock_async_llm_dependencies(monkeypatch) + events: list[str] = [] + frontend_profiler = MagicMock() + frontend_profiler.start.side_effect = lambda: events.append("frontend_start") + frontend_profiler.stop.side_effect = lambda: events.append("frontend_stop") + + engine = AsyncLLM( + vllm_config, + MagicMock(), + log_stats=False, + profiler=frontend_profiler, + ) + + async def profile(): + start_rpc_entered = asyncio.Event() + release_start_rpc = asyncio.Event() + + async def profile_async(is_start, *args): + if is_start: + start_rpc_entered.set() + await release_start_rpc.wait() + events.append("engine_start_done") + else: + events.append("engine_stop") + + engine_core.profile_async.side_effect = profile_async + + start_task = asyncio.create_task(engine.start_profile()) + await start_rpc_entered.wait() + + stop_task = asyncio.create_task(engine.stop_profile()) + await asyncio.sleep(0) + release_start_rpc.set() + await asyncio.wait_for(asyncio.gather(start_task, stop_task), timeout=5) + + asyncio.run(profile()) + + assert events[:2] == ["engine_start_done", "frontend_start"] + assert sorted(events[2:]) == ["engine_stop", "frontend_stop"] + assert engine._profile_session_active is False + + +@pytest.mark.parametrize("failure", ["error", "cancel"]) +def test_interrupted_engine_profile_start_skips_frontend_profiler( + monkeypatch: pytest.MonkeyPatch, failure: str +): + """The frontend trace starts only after the engine start succeeds. A + cancelled start may still reach the engine, so it keeps the session active + until /stop_profile.""" + vllm_config = _mock_vllm_config() + engine_core = _mock_async_llm_dependencies(monkeypatch) + frontend_profiler = MagicMock() + engine = AsyncLLM( + vllm_config, + MagicMock(), + log_stats=False, + profiler=frontend_profiler, + ) + + async def profile(): + start_rpc_entered = asyncio.Event() + + async def profile_async(is_start, *args): + if not is_start: + return + start_rpc_entered.set() + if failure == "error": + raise RuntimeError("engine profiler failed to start") + await asyncio.Event().wait() + + engine_core.profile_async.side_effect = profile_async + + start_task = asyncio.create_task(engine.start_profile()) + await start_rpc_entered.wait() + if failure == "cancel": + start_task.cancel() + expected = RuntimeError if failure == "error" else asyncio.CancelledError + with pytest.raises(expected): + await start_task + + assert engine._profile_session_active is (failure == "cancel") + await engine.stop_profile() + assert engine._profile_session_active is False + + asyncio.run(profile()) + + frontend_profiler.start.assert_not_called() + + +class _BlockingStartProfiler(WorkerProfiler): + def __init__(self) -> None: + self.in_start = threading.Event() + self.release_start = threading.Event() + self.stop_call_count = 0 + super().__init__( + ProfilerConfig(profiler="torch", torch_profiler_dir="/tmp/mock") + ) + + def _start(self) -> None: + self.in_start.set() + if not self.release_start.wait(timeout=5): + raise TimeoutError("start was never released") + + def _stop(self) -> None: + self.stop_call_count += 1 + + +def test_stop_after_cancelled_frontend_start_stops_frontend_profiler( + monkeypatch: pytest.MonkeyPatch, +): + """Cancelling start_profile while the frontend profiler is starting keeps + the session active, and the next stop_profile still stops the backend.""" + vllm_config = _mock_vllm_config() + engine_core = _mock_async_llm_dependencies(monkeypatch) + frontend_profiler = _BlockingStartProfiler() + engine = AsyncLLM( + vllm_config, + MagicMock(), + log_stats=False, + profiler=frontend_profiler, + ) + + async def profile(): + try: + start_task = asyncio.create_task(engine.start_profile()) + assert await asyncio.to_thread(frontend_profiler.in_start.wait, 5) + start_task.cancel() + with pytest.raises(asyncio.CancelledError): + await start_task + assert engine._profile_session_active is True + + stop_task = asyncio.create_task(engine.stop_profile()) + done, _ = await asyncio.wait({stop_task}, timeout=0.1) + assert not done, "stop_profile did not wait for the in-flight start" + frontend_profiler.release_start.set() + await asyncio.wait_for(stop_task, timeout=5) + finally: + frontend_profiler.release_start.set() + + asyncio.run(profile()) + + assert frontend_profiler.stop_call_count == 1 + assert frontend_profiler._active is False + assert frontend_profiler._running is False + assert engine._profile_session_active is False + assert engine_core.profile_async.await_args_list == [ + call(True, None, None, None), + call(False), + ] + + +def test_multi_client_profile_uses_idempotent_engine_requests( + monkeypatch: pytest.MonkeyPatch, tmp_path +): + vllm_config = _mock_vllm_config() + vllm_config.profiler_config = ProfilerConfig( + profiler="torch", + torch_profiler_dir=str(tmp_path), + torch_profiler_activities=["CPU"], + ) + engine_core = _mock_async_llm_dependencies(monkeypatch) + profiler = MagicMock() + monkeypatch.setattr(async_llm_module, "create_frontend_profiler", profiler) + engines = [ + AsyncLLM( + vllm_config, + MagicMock(), + log_stats=False, + client_count=2, + client_index=client_index, + ) + for client_index in range(2) + ] + + assert all(engine.profiler is None for engine in engines) + profiler.assert_not_called() + + async def profile(): + await engines[0].start_profile() + await engines[1].start_profile() + await engines[1].stop_profile() + await engines[0].start_profile() + + asyncio.run(profile()) + + engine_core.profile_async.assert_has_awaits( + [ + call(True, None, None, None), + call(True, None, None, None), + call(False), + call(True, None, None, None), + ] + ) + + +def test_frontend_profiler_ignores_worker_iteration_bounds( + monkeypatch: pytest.MonkeyPatch, tmp_path +): + vllm_config = _mock_vllm_config() + vllm_config.profiler_config = ProfilerConfig( + profiler="torch", + torch_profiler_dir=str(tmp_path), + torch_profiler_activities=["CPU", "CUDA"], + delay_iterations=5, + max_iterations=2, + wait_iterations=3, + warmup_iterations=1, + ) + _mock_async_llm_dependencies(monkeypatch) + profiler = MagicMock() + monkeypatch.setattr("vllm.profiler.wrapper.TorchProfilerWrapper", profiler) + + engine = AsyncLLM(vllm_config, MagicMock(), log_stats=False) + + frontend_config = profiler.call_args.args[0] + assert frontend_config.delay_iterations == 0 + assert frontend_config.max_iterations == 0 + assert frontend_config.wait_iterations == 0 + assert frontend_config.warmup_iterations == 0 + assert engine.profiler is profiler.return_value + + async def generate( engine: AsyncLLM, request_id: str, diff --git a/tests/v1/engine/test_core_engine_actor_manager.py b/tests/v1/engine/test_core_engine_actor_manager.py index f1a67ad9ba..1b23356134 100644 --- a/tests/v1/engine/test_core_engine_actor_manager.py +++ b/tests/v1/engine/test_core_engine_actor_manager.py @@ -13,8 +13,9 @@ from unittest.mock import Mock import pytest import ray import zmq +from torch.distributed import TCPStore -from vllm.utils.network_utils import make_zmq_socket, split_zmq_path +from vllm.utils.network_utils import get_open_port, make_zmq_socket, split_zmq_path from vllm.v1.engine.core import EngineCoreActorMixin from vllm.v1.engine.core_client import BackgroundResources from vllm.v1.engine.utils import ( @@ -22,7 +23,7 @@ from vllm.v1.engine.utils import ( EngineZmqAddresses, _dp_nodes_master_first, _node_ip_from_resources, - get_engine_zmq_addresses, + bind_engine_zmq_listeners, launch_core_engines, ) from vllm.v1.utils import APIServerProcessManager @@ -66,35 +67,27 @@ class _StubEngineCoreActor(EngineCoreActorMixin): # Module-level stub worker for the Ray-DP regression test. Must be importable -# by ``multiprocessing.spawn`` (no closures, no nesting). Mirrors the worker -# in ``tests/entrypoints/test_api_server_process_manager.py``. -def _bind_and_report_worker(listen_address, sock, args, client_config): - """Bind ROUTER/PULL with a kernel-assigned port, report the actual - endpoints back via ``actual_address_pipe``, then exit.""" +# by ``multiprocessing.spawn`` (no closures, no nesting). +def _adopt_listener_worker(listen_address, sock, args, client_config): + """Adopt the supervisor-bound ROUTER/PULL listeners, then exit.""" ctx = zmq.Context() try: in_sock = make_zmq_socket( - ctx, client_config["input_address"], zmq.ROUTER, bind=True + ctx, + client_config["input_address"], + zmq.ROUTER, + bind=True, + listener=client_config["input_listener"], ) out_sock = make_zmq_socket( - ctx, client_config["output_address"], zmq.PULL, bind=True + ctx, + client_config["output_address"], + zmq.PULL, + bind=True, + listener=client_config["output_listener"], ) - try: - pipe = client_config["actual_address_pipe"] - try: - pipe.send( - { - "input_address": in_sock.getsockopt(zmq.LAST_ENDPOINT).decode(), - "output_address": out_sock.getsockopt( - zmq.LAST_ENDPOINT - ).decode(), - } - ) - finally: - pipe.close() - finally: - in_sock.close(linger=0) - out_sock.close(linger=0) + in_sock.close(linger=0) + out_sock.close(linger=0) finally: ctx.term() @@ -134,6 +127,72 @@ def _make_addresses() -> EngineZmqAddresses: ) +@pytest.mark.parametrize( + ("overrides", "expect_store"), + [ + ({"data_parallel_backend": "mp"}, True), + ({}, True), + ( + { + "data_parallel_backend": "mp", + "data_parallel_rank": 1, + "local_engines_only": True, + }, + False, + ), + ({"data_parallel_backend": "mp", "data_parallel_rank_local": 0}, False), + ({"data_parallel_backend": "mp", "enable_elastic_ep": True}, False), + ({"data_parallel_backend": "mp", "data_parallel_size": 1}, False), + ], + ids=["mp", "ray", "non-master-node", "offline", "elastic-ep", "single-engine"], +) +def test_coordination_store_held_only_by_the_online_dp_master( + monkeypatch: pytest.MonkeyPatch, overrides: dict[str, Any], expect_store: bool +) -> None: + """The DP master node holds a coordination store for both engine backends, + so engines there pick world-group ports at bind time. Every other + deployment keeps the pre-allocated ports and must not hold one.""" + + class FakeManager: + def __init__(self, **kwargs) -> None: + self.kwargs = kwargs + + vllm_config = _make_vllm_config_ray_dp_multinode() + parallel_config = vllm_config.parallel_config + for name, value in overrides.items(): + setattr(parallel_config, name, value) + # A fresh port per case so a lingering handshake socket cannot fail the next. + parallel_config.data_parallel_rpc_port = get_open_port() + parallel_config._coord_store_port = 0 + monkeypatch.setattr("vllm.v1.engine.utils.CoreEngineProcManager", FakeManager) + monkeypatch.setattr("vllm.v1.engine.utils.CoreEngineActorManager", FakeManager) + monkeypatch.setattr( + "vllm.v1.engine.utils.wait_for_engine_startup", lambda *args, **kwargs: None + ) + + with launch_core_engines( + vllm_config, + executor_class=_DummyExecutor, + log_stats=False, + addresses=_make_addresses(), + ) as engine_launch: + assert engine_launch.engine_manager is not None + if not expect_store: + assert not parallel_config._coord_store_port + return + assert parallel_config._coord_store_port + # Engines look the store up while they start, so it must be reachable + # for as long as this frame is alive. + client = TCPStore( + parallel_config.data_parallel_master_ip, + parallel_config._coord_store_port, + is_master=False, + wait_for_workers=False, + ) + client.set("probe", b"1") + assert client.get("probe") == b"1" + + def _make_cpu_placement_group(): pg = ray.util.placement_group( [{"CPU": 0.001}, {"CPU": 1.0}], @@ -260,17 +319,9 @@ def test_ray_dp_addresses_resolved_before_actor_creation( ``launch_core_engines`` Ray branch pickles ``addresses`` into each engine actor at ``.remote()`` time, and ``EngineCoreActorMixin._perform_handshakes`` is a no-op, so the actor uses that pickled snapshot for the rest of its - life. If ``run_multi_api_server`` allocates ``addresses`` as - ``tcp://host:0`` placeholders (its default), the actors hold placeholders - forever and DEALER-connect to port 0 — ZMQ ``connect`` is async and does - not raise, so the failure mode is a deterministic hang. - - The Ray-DP carve-out in ``run_multi_api_server`` forces - ``defer_api_server_ports=False`` when ``data_parallel_backend == "ray"`` - so addresses are pre-allocated in the driver and Ray pickles real ports - into each actor. This test mirrors that call-site logic and asserts the - actors hold real (non-placeholder) endpoints. If the carve-out is - removed without an alternative fix, the test fails. + life. The supervisor therefore binds the frontend listeners before actor + construction and Ray pickles their resolved endpoints. This test asserts + that each actor holds real (non-placeholder) endpoints. """ created_placement_groups: list[Any] = [] @@ -289,15 +340,10 @@ def test_ray_dp_addresses_resolved_before_actor_creation( vllm_config = _make_vllm_config_ray_dp_multinode() - # Mirror run_multi_api_server's address-allocation logic. The Ray DP - # carve-out forces pre-allocation so the addresses pickled into engine - # actors at .remote() time are real, not ``tcp://host:0``. - is_ray_dp = vllm_config.parallel_config.data_parallel_backend == "ray" - addresses = get_engine_zmq_addresses( - vllm_config, - num_api_servers=2, - defer_api_server_ports=not is_ray_dp, - ) + # Mirror run_multi_api_server: bind listeners in the supervisor so the + # addresses pickled into Ray actors contain kernel-assigned ports. + listeners = bind_engine_zmq_listeners(vllm_config, num_api_servers=2) + addresses = listeners.addresses sock = socket.socket() engine_manager: CoreEngineActorManager | None = None @@ -314,26 +360,17 @@ def test_ray_dp_addresses_resolved_before_actor_creation( engine_manager = engine_launch.engine_manager assert isinstance(engine_manager, CoreEngineActorManager) - # API-server children bind to the pre-allocated ports. + # API-server children adopt the already-bound listener FDs. api_server_manager = APIServerProcessManager( listen_address="tcp://127.0.0.1:0", sock=sock, args="test_args", num_servers=2, - input_addresses=addresses.inputs, - output_addresses=addresses.outputs, - target_server_fn=_bind_and_report_worker, + input_listeners=listeners.inputs, + output_listeners=listeners.outputs, + target_server_fn=_adopt_listener_worker, ) - # run_multi_api_server skips ``gather_actual_addresses`` for - # Ray DP (addresses are already real). Mirror that. - if not is_ray_dp: - actual_inputs, actual_outputs = ( - api_server_manager.gather_actual_addresses(timeout=15.0) - ) - addresses.inputs = actual_inputs - addresses.outputs = actual_outputs - # Snapshot what each Ray actor actually holds. actors = ( engine_manager.local_engine_actors + engine_manager.remote_engine_actors @@ -360,9 +397,9 @@ def test_ray_dp_addresses_resolved_before_actor_creation( assert scheme == "tcp", url assert port and int(port) > 0, ( f"Ray actor was pickled with placeholder address {url!r}; " - "``run_multi_api_server`` must pre-allocate ports for the " - "Ray DP backend so the actors hold real endpoints by the " - "time they DEALER-connect. See PR #42585 / Ray-DP " + "``run_multi_api_server`` must bind frontend listeners before " + "constructing Ray DP actors so they hold real endpoints when " + "they DEALER-connect. See PR #42585 / Ray-DP " "multi-API-server regression." ) diff --git a/tests/v1/engine/test_dp_world_group_port.py b/tests/v1/engine/test_dp_world_group_port.py new file mode 100644 index 0000000000..49f5a59ab5 --- /dev/null +++ b/tests/v1/engine/test_dp_world_group_port.py @@ -0,0 +1,109 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""The DP world group picks its port when it binds, not ahead of time.""" + +import multiprocessing as mp +import socket +import traceback +from multiprocessing.queues import Queue + +import pytest +import torch + +from vllm.distributed.utils import create_tcp_store +from vllm.utils.network_utils import get_open_port + +RESULT_TIMEOUT_S = 240 + + +def _join_dp_world_group( + dp_rank: int, coord_store_port: int, dp_master_port: int, result: Queue +) -> None: + """Bring up one DP rank of the world group and report what it saw.""" + try: + import torch.distributed as dist + + from vllm.config import VllmConfig, set_current_vllm_config + from vllm.config.parallel import ParallelConfig + from vllm.distributed.parallel_state import init_distributed_environment + + parallel_config = ParallelConfig( + data_parallel_size=2, data_parallel_rank=dp_rank + ) + parallel_config.data_parallel_master_ip = "127.0.0.1" + parallel_config._data_parallel_master_port_list = [dp_master_port] + parallel_config._coord_store_port = coord_store_port + + with set_current_vllm_config(VllmConfig(parallel_config=parallel_config)): + init_distributed_environment( + world_size=1, rank=0, local_rank=0, backend="gloo" + ) + + reduced = torch.tensor([dp_rank + 1]) + dist.all_reduce(reduced) + result.put((dp_rank, int(reduced.item()), None)) + except Exception: + result.put((dp_rank, None, traceback.format_exc())) + + +@pytest.mark.skipif( + not torch.distributed.is_gloo_available(), reason="gloo backend required" +) +@pytest.mark.parametrize("with_coord_store", [True, False]) +def test_dp_world_group_survives_a_taken_preallocated_port(with_coord_store: bool): + """With a coordination store, the two ranks agree on a port the group binds + itself, so a pre-allocated port taken in the meantime cannot stop startup. + Without one, the pre-allocated port is still used as before. + """ + squatter = None + coord_store = None + try: + if with_coord_store: + # Something else grabbed the pre-allocated port after it was picked. + squatter = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + squatter.bind(("127.0.0.1", 0)) + squatter.listen(1) + dp_master_port = squatter.getsockname()[1] + coord_store = create_tcp_store( + "127.0.0.1", 0, is_master=True, world_size=-1, wait_for_workers=False + ) + coord_store_port = coord_store.port + else: + dp_master_port = get_open_port() + coord_store_port = 0 + + ctx = mp.get_context("spawn") + result: Queue = ctx.Queue() + procs = [ + ctx.Process( + target=_join_dp_world_group, + args=(dp_rank, coord_store_port, dp_master_port, result), + ) + for dp_rank in range(2) + ] + for proc in procs: + proc.start() + reports: list[tuple[int, int | None, str | None]] = [] + try: + while len(reports) < len(procs): + reports.append(result.get(timeout=RESULT_TIMEOUT_S)) + if reports[-1][2] is not None: + break # the peer would wait on this rank; do not wait for it + finally: + for proc in procs: + proc.join(timeout=30) + if proc.is_alive(): + proc.kill() + + for dp_rank, reduced, error in sorted(reports): + assert error is None, f"DP rank {dp_rank} failed:\n{error}" + # Both ranks joined the same group: 1 + 2 over the two ranks. + assert reduced == 3, f"DP rank {dp_rank} all-reduced to {reduced}" + + if with_coord_store: + assert coord_store is not None + bound_port = int(coord_store.get("world_pg_port").decode()) + assert bound_port != dp_master_port + finally: + if squatter is not None: + squatter.close() diff --git a/tests/v1/engine/test_engine_core_shutdown_signal.py b/tests/v1/engine/test_engine_core_shutdown_signal.py new file mode 100644 index 0000000000..73bc794088 --- /dev/null +++ b/tests/v1/engine/test_engine_core_shutdown_signal.py @@ -0,0 +1,224 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +# +# Regression tests for termination signals that arrive while EngineCoreProc is +# still being constructed. +# +# When this fix matters +# --------------------- +# Engine shutdown is driven from the frontend. If a frontend process dies while +# EngineCore is still starting up, the process manager sends SIGTERM to its +# children and force kills whatever is still alive after a short timeout (see +# shutdown in vllm/v1/utils.py). Construction is slow - it spawns the executor +# processes, loads weights, profiles and captures CUDA graphs - so that +# shutdown often lands inside the construction window. Bigger models and more +# ranks widen the window. This is the case PR 52299 (follow up to 43417) has to +# survive: 43417 stops EngineCore from exiting silently, but the request itself +# still has to outlive the construction window. +# +# run_engine_core installs the signal handler before engine_core exists, so in +# that window the handler runs with engine_core == None. Losing the request +# there is permanent, not a delay: shutdown_state stays RUNNING, and +# wakeup_engine returns early while consuming the one-shot SignalCallback. The +# busy loop then starts RUNNING and blocks on input_queue forever, so no later +# signal can wake it. Graceful shutdown is gone for good and the process only +# stops when something force kills it, which skips executor shutdown and GPU +# release. In a log this shows up as the signal being received, followed by a +# force kill of EngineCore after the process manager timeout. +# +# What these tests pin +# -------------------- +# A request arriving in the window must be remembered somewhere other than +# engine_core, applied before run_busy_loop starts, and must not consume the +# one-shot wakeup callback, so a later signal can still wake an idle loop. + +import queue +import signal as signal_module +from typing import Any + +from vllm.v1.engine import EngineCoreRequestType +from vllm.v1.engine import core as core_module +from vllm.v1.engine.core import EngineCoreProc, EngineShutdownState + +WAKEUP_ITEM = (EngineCoreRequestType.WAKEUP, None) +CONSTRUCTION_SIGNAL = signal_module.SIGTERM + + +class _FakeSignal: + # Minimal stand-in for the signal module. It records the installed handler + # so the test can fire it by hand, and avoids touching the real process + # signal disposition. Signals is reused from the real module so the handler + # can still resolve signal.Signals(signum).name. + SIGTERM = signal_module.SIGTERM + SIGINT = signal_module.SIGINT + SIG_DFL = signal_module.SIG_DFL + Signals = signal_module.Signals + + def __init__(self): + self.handlers = {} + + def signal(self, signum, handler): + self.handlers[signum] = handler + + def fire(self): + self.handlers[CONSTRUCTION_SIGNAL](CONSTRUCTION_SIGNAL, None) + + +class _FakeSignalCallback: + # One-shot stand-in for SignalCallback. The real class runs the callback on + # a dedicated thread and exits after a single trigger, so every later + # trigger is a no-op. Running the callback inline keeps the test + # deterministic instead of depending on thread scheduling. + instances: list["_FakeSignalCallback"] = [] + + def __init__(self, callback): + self.callback = callback + self.trigger_count = 0 + self.consumed = False + self.stopped = False + _FakeSignalCallback.instances.append(self) + + def trigger(self): + self.trigger_count += 1 + if self.stopped or self.consumed: + return + self.consumed = True + callback, self.callback = self.callback, None + callback() + + def stop(self): + self.stopped = True + + +class _StubParallelConfig: + def __init__(self): + self.data_parallel_size = 1 + self.data_parallel_rank_local = 0 + self.data_parallel_index = 0 + self.data_parallel_rank = 0 + self.numa_bind = False + + def reconfigure_for_independent_dp_rank(self): + pass + + +class _StubModelConfig: + is_moe = False + + +class _StubVllmConfig: + def __init__(self): + self.parallel_config = _StubParallelConfig() + self.model_config = _StubModelConfig() + self.logging_config = None + self.kv_transfer_config = None + self.shutdown_timeout = 0 + + +def _run_engine_core( + monkeypatch, + *, + signal_during_construction=False, + signal_while_idle=False, +): + _FakeSignalCallback.instances.clear() + fake_signal = _FakeSignal() + trace = {} + + class StubEngineCoreProc: + def __init__(self, *args, **kwargs): + self.shutdown_state = EngineShutdownState.RUNNING + # Same queue type as EngineCore, so the wakeup path under test is + # the real one. + self.input_queue = queue.Queue[tuple[EngineCoreRequestType, Any]]() + self.vllm_config = kwargs.get("vllm_config") + if signal_during_construction: + # The termination signal lands mid construction, while the + # caller still sees engine_core == None. + fake_signal.fire() + trace["constructed"] = True + + def run_busy_loop(self): + # What the real loop sees on its first _handle_shutdown() check. + trace["loop_entered"] = True + trace["state_at_loop_entry"] = self.shutdown_state + if signal_while_idle: + fake_signal.fire() + try: + trace["wakeup"] = self.input_queue.get_nowait() + except queue.Empty: + trace["wakeup"] = None + + def shutdown(self): + pass + + def _send_engine_dead(self): + pass + + def has_work(self): + return False + + monkeypatch.setattr(core_module, "signal", fake_signal) + monkeypatch.setattr(core_module, "SignalCallback", _FakeSignalCallback) + monkeypatch.setattr(core_module, "EngineCoreProc", StubEngineCoreProc) + monkeypatch.setattr(core_module, "DPEngineCoreProc", StubEngineCoreProc) + monkeypatch.setattr(core_module, "configure_logging", lambda *a, **k: None) + monkeypatch.setattr( + core_module, "maybe_register_config_serialize_by_value", lambda *a, **k: None + ) + monkeypatch.setattr(core_module, "set_process_title", lambda *a, **k: None) + monkeypatch.setattr(core_module, "maybe_init_worker_tracer", lambda *a, **k: None) + monkeypatch.setattr(core_module, "decorate_logs", lambda *a, **k: None) + + EngineCoreProc.run_engine_core(vllm_config=_StubVllmConfig()) + return trace + + +def test_signal_during_construction_is_applied_before_loop(monkeypatch): + trace = _run_engine_core(monkeypatch, signal_during_construction=True) + + assert trace["constructed"] is True + assert trace["loop_entered"] is True + # The request arrived while engine_core was None, so it could not be stored + # there. It must still be applied before run_busy_loop starts, otherwise the + # loop comes up as RUNNING, the shutdown is silently dropped, and force kill + # becomes the only way out. + assert trace["state_at_loop_entry"] == EngineShutdownState.REQUESTED, ( + "a signal received during construction must be applied before the loop " + f"starts, got {trace['state_at_loop_entry']!r}" + ) + + +def test_signal_during_construction_keeps_later_wakeup_working(monkeypatch): + trace = _run_engine_core( + monkeypatch, + signal_during_construction=True, + signal_while_idle=True, + ) + + assert trace["state_at_loop_entry"] == EngineShutdownState.REQUESTED, ( + "the construction time signal must be applied before the loop starts, " + f"got {trace['state_at_loop_entry']!r}" + ) + # SignalCallback is one-shot. If the construction time signal consumed it, + # no wakeup could be delivered afterwards and the idle busy loop would block + # on input_queue forever instead of shutting down, so the process would sit + # there holding GPU memory until something force kills it. + assert trace["wakeup"] == WAKEUP_ITEM, ( + "a signal after construction must still wake the idle loop, " + f"got {trace['wakeup']!r}" + ) + + +def test_signal_while_idle_still_wakes_the_loop(monkeypatch): + trace = _run_engine_core(monkeypatch, signal_while_idle=True) + + assert trace["state_at_loop_entry"] == EngineShutdownState.RUNNING + assert trace["wakeup"] == WAKEUP_ITEM + + +def test_no_signal_leaves_the_loop_running(monkeypatch): + trace = _run_engine_core(monkeypatch) + + assert trace["state_at_loop_entry"] == EngineShutdownState.RUNNING + assert trace["wakeup"] is None diff --git a/tests/v1/engine/test_input_processor_trace_replay.py b/tests/v1/engine/test_input_processor_trace_replay.py index 957f70b718..043de170af 100644 --- a/tests/v1/engine/test_input_processor_trace_replay.py +++ b/tests/v1/engine/test_input_processor_trace_replay.py @@ -82,7 +82,12 @@ def _validate(enable_trace_replay: bool) -> None: enable_trace_replay=enable_trace_replay, is_diffusion=False, ), - vllm_config=SimpleNamespace(reasoning_config=None), + vllm_config=SimpleNamespace( + reasoning_config=None, + # Engine has no watermark configuration. + _check_supports_watermarking=lambda params: False, + ), + resolve_watermarking=lambda params: False, speculative_config=None, structured_outputs_config=None, diffusion_config=None, diff --git a/tests/v1/engine/test_kv_transfer_rejection.py b/tests/v1/engine/test_kv_transfer_rejection.py new file mode 100644 index 0000000000..a21351900c --- /dev/null +++ b/tests/v1/engine/test_kv_transfer_rejection.py @@ -0,0 +1,27 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +from types import SimpleNamespace +from unittest.mock import AsyncMock + +import pytest + +from vllm.v1.engine.async_llm import AsyncLLM + + +@pytest.mark.asyncio +async def test_kv_transfer_rejection_stub_disables_watermarking(): + engine = object.__new__(AsyncLLM) + add_request = AsyncMock() + engine.engine_core = SimpleNamespace( + add_request_async=add_request, + shutdown=lambda **kwargs: None, + ) + + await engine.notify_kv_transfer_request_rejected( + "request", + {"do_remote_prefill": True}, + ) + + request = add_request.call_args.args[0] + assert request.sampling_params.watermarking is False diff --git a/tests/v1/engine/test_llm_engine.py b/tests/v1/engine/test_llm_engine.py index b09f9405d7..69241889dc 100644 --- a/tests/v1/engine/test_llm_engine.py +++ b/tests/v1/engine/test_llm_engine.py @@ -7,6 +7,7 @@ from unittest.mock import MagicMock, call import pytest from vllm import LLM +from vllm.exceptions import ProfilerAlreadyActiveError from vllm.sampling_params import SamplingParams, StructuredOutputsParams from vllm.v1.engine.llm_engine import LLMEngine from vllm.v1.metrics.reader import Counter, Gauge, Histogram, Metric, Vector @@ -52,6 +53,38 @@ def test_step_invalidates_mm_processor_cache_misses(): ] +def test_duplicate_profile_start_rejected_before_engine_core_dispatch(): + engine = object.__new__(LLMEngine) + engine.engine_core = MagicMock() + engine._profile_session_active = False + + engine.start_profile("first") + + with pytest.raises(ProfilerAlreadyActiveError): + engine.start_profile("duplicate") + + engine.engine_core.profile.assert_called_once_with(True, "first", None, None) + + +@pytest.mark.parametrize( + "kwargs", + [ + {"profile_prefix": "../trace"}, + {"delay_iterations": -1}, + {"max_iterations": -1}, + ], +) +def test_invalid_profile_controls_rejected_before_engine_core_dispatch(kwargs): + engine = object.__new__(LLMEngine) + engine.engine_core = MagicMock() + engine._profile_session_active = False + + with pytest.raises(ValueError): + engine.start_profile(**kwargs) + + engine.engine_core.profile.assert_not_called() + + def _vllm_model( apc: bool, vllm_runner: type[VllmRunner], diff --git a/tests/v1/executor/test_multiproc_executor.py b/tests/v1/executor/test_multiproc_executor.py index 706bfae381..6a3b3d516e 100644 --- a/tests/v1/executor/test_multiproc_executor.py +++ b/tests/v1/executor/test_multiproc_executor.py @@ -70,16 +70,24 @@ def test_execute_worker_rpc_returns_worker_exception(): @pytest.mark.parametrize("stalled", [False, True]) -def test_take_draft_token_ids_uses_execute_model_timeout(monkeypatch, stalled): - """A wedged draft-token readback must not block EngineCore forever.""" +@pytest.mark.parametrize( + "method,result", + [ + ("take_draft_token_ids", DraftTokenIds(["req"], [[1, 2, 3]])), + ("execute_dummy_batch", None), + ], +) +def test_model_rpc_uses_execute_model_timeout(monkeypatch, method, result, stalled): + """A wedged model RPC must not block EngineCore forever. For + execute_dummy_batch this is an idle DP rank stuck in a collective with a + hung peer.""" monkeypatch.setenv("VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS", "7") - draft = DraftTokenIds(["req"], [[1, 2, 3]]) def dequeue(*, timeout=None): assert timeout is not None and 0 < timeout <= 7 if stalled: raise TimeoutError - return WorkerProc.ResponseStatus.SUCCESS, draft + return WorkerProc.ResponseStatus.SUCCESS, result executor: Any = MultiprocExecutor.__new__(MultiprocExecutor) executor.is_failed = False @@ -89,7 +97,7 @@ def test_take_draft_token_ids_uses_execute_model_timeout(monkeypatch, stalled): executor.response_mqs = [None, SimpleNamespace(dequeue=dequeue)] if stalled: - with pytest.raises(TimeoutError, match="take_draft_token_ids timed out"): - executor.take_draft_token_ids() + with pytest.raises(TimeoutError, match=f"{method} timed out"): + getattr(executor, method)() else: - assert executor.take_draft_token_ids() is draft + assert getattr(executor, method)() is result diff --git a/tests/v1/kv_connector/nixl_integration/run_tpu_disagg_accuracy_test.sh b/tests/v1/kv_connector/nixl_integration/run_tpu_disagg_accuracy_test.sh deleted file mode 100644 index 9274e3c573..0000000000 --- a/tests/v1/kv_connector/nixl_integration/run_tpu_disagg_accuracy_test.sh +++ /dev/null @@ -1,157 +0,0 @@ -#!/bin/bash -set -xe - -# Hosts / ports -PREFILL_HOST=${PREFILL_HOST:-"localhost"} -PREFILL_PORT=${PREFILL_PORT:-8100} -PREFILL_NIXL_SIDE_PORT=${PREFILL_NIXL_SIDE_PORT:-5577} -DECODE_HOST=${DECODE_HOST:-"localhost"} -DECODE_PORT=${DECODE_PORT:-8200} -PROXY_HOST=${PROXY_HOST:-"localhost"} -PROXY_PORT=${PROXY_PORT:-8192} -BASELINE_HOST=${BASELINE_HOST:-"localhost"} -BASELINE_PORT=${BASELINE_PORT:-9290} - - -# Model to run. -MODEL_NAME=${MODEL_NAME:-"meta-llama/Llama-3.2-3B-Instruct"} -MAX_MODEL_LEN=${MAX_MODEL_LEN:-1024} -BLOCK_SIZE=${BLOCK_SIZE:-32} - - -# execution env -SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd -P)" -GIT_ROOT="${GIT_ROOT:-$(cd -- "${SCRIPT_DIR}/../../../.." && pwd -P)}" -EXP_ROOT="${GIT_ROOT}/tests/v1/kv_connector/nixl_integration" -CONDA_PATH=${CONDA_PATH:-"/home/${USER}/anaconda3"} -CONDA_ENV_NAME=${CONDA_ENV_NAME:-"nixl"} - -OUTPUT_FILE=${OUTPUT_FILE:-"${EXP_ROOT}/.tpu_accuracy_test_outputs.txt"} - -# Trap the SIGINT signal (triggered by Ctrl+C) -trap 'kill $(jobs -pr)' SIGINT SIGTERM EXIT - - -# Waits for vLLM server to start. -wait_for_server() { - local host=$1 - local port=$2 - timeout 1200 bash -c " - until curl -s ${host}:${port}/v1/completions > /dev/null; do - sleep 1 - done" && return 0 || return 1 -} - -# Cleanup function -cleanup() { - echo "Caught Ctrl+C, cleaning up..." - # Cleanup commands - pgrep python | xargs kill -9 || true - # pkill -f python || true - echo "Cleanup complete. Exiting." -} - -launch_baseline() { - BASELINE_BASE_CMD="source ${CONDA_PATH}/bin/activate ${CONDA_ENV_NAME}; - VLLM_LOGGING_LEVEL=DEBUG \ - PJRT_DEVICE=TPU \ - VLLM_WORKER_MULTIPROC_METHOD=spawn \ - VLLM_ENABLE_V1_MULTIPROCESSING=0 vllm serve $MODEL_NAME \ - --host ${BASELINE_HOST} \ - --port ${BASELINE_PORT} \ - --max-model-len ${MAX_MODEL_LEN}\ - --seed 42 \ - --block-size ${BLOCK_SIZE} \ - --gpu-memory-utilization 0.5 \ - --enforce-eager" - echo "${BASELINE_BASE_CMD}" - ssh -tt "${BASELINE_HOST}" "${BASELINE_BASE_CMD}" & -} - -launch_pd() { - PREFILL_BASE_CMD="source ${CONDA_PATH}/bin/activate ${CONDA_ENV_NAME}; - UCX_TLS=tcp \ - VLLM_MULTIPROC_EXECUTE_MODEL_TIMEOUT_S=200 \ - VLLM_LOGGING_LEVEL=DEBUG \ - VLLM_NIXL_SIDE_CHANNEL_HOST=${PREFILL_HOST} \ - VLLM_NIXL_SIDE_CHANNEL_PORT=${PREFILL_NIXL_SIDE_PORT} \ - PJRT_DEVICE=TPU \ - VLLM_WORKER_MULTIPROC_METHOD=spawn \ - VLLM_ENABLE_V1_MULTIPROCESSING=0 vllm serve $MODEL_NAME \ - --host ${PREFILL_HOST} \ - --port ${PREFILL_PORT} \ - --max-model-len ${MAX_MODEL_LEN}\ - --seed 42 \ - --block-size ${BLOCK_SIZE} \ - --enforce-eager \ - --gpu-memory-utilization 0.5 \ - --kv-transfer-config '{\"kv_connector\":\"NixlConnector\",\"kv_role\":\"kv_both\",\"kv_buffer_device\":\"cpu\"}'" - - - DECODE_BASE_CMD="source ${CONDA_PATH}/bin/activate ${CONDA_ENV_NAME}; - UCX_TLS=tcp \ - VLLM_MULTIPROC_EXECUTE_MODEL_TIMEOUT_S=200 \ - VLLM_LOGGING_LEVEL=DEBUG \ - PJRT_DEVICE=TPU \ - VLLM_WORKER_MULTIPROC_METHOD=spawn \ - VLLM_ENABLE_V1_MULTIPROCESSING=0 vllm serve $MODEL_NAME \ - --host ${DECODE_HOST} \ - --port ${DECODE_PORT} \ - --max-model-len ${MAX_MODEL_LEN}\ - --seed 42 \ - --block-size ${BLOCK_SIZE} \ - --enforce-eager \ - --gpu-memory-utilization 0.5 \ - --kv-transfer-config '{\"kv_connector\":\"NixlConnector\",\"kv_role\":\"kv_both\",\"kv_buffer_device\":\"cpu\"}'" - - echo "${PREFILL_BASE_CMD}" - echo "${DECODE_BASE_CMD}" - sleep 2 - - # execute on hosts - ssh -tt "${PREFILL_HOST}" "${PREFILL_BASE_CMD}" & - ssh -tt "${DECODE_HOST}" "${DECODE_BASE_CMD}" & - sleep 1 - wait_for_server "${PREFILL_HOST}" "${PREFILL_PORT}" - sleep 1 - wait_for_server "${DECODE_HOST}" "${DECODE_PORT}" - sleep 1 -} - -launch_pd_proxy(){ - PROXY_BASE_CMD="source ${CONDA_PATH}/bin/activate ${CONDA_ENV_NAME}; - python3 ${EXP_ROOT}/toy_proxy_server.py \ - --prefiller-host ${PREFILL_HOST} --prefiller-port ${PREFILL_PORT} \ - --decoder-host ${DECODE_HOST} --decoder-port ${DECODE_PORT} \ - --host=${PROXY_HOST} --port ${PROXY_PORT}" - echo "${PROXY_BASE_CMD}" - ssh -tt "${PROXY_HOST}" "${PROXY_BASE_CMD}" & -} - -run_tests(){ - local service_url=$1 - local mode=$2 - python3 "${EXP_ROOT}"/test_disagg_accuracy.py --service_url="${service_url}" --model_name="${MODEL_NAME}" --mode="${mode}" --file_name="${OUTPUT_FILE}" -} - - -# run non-disagg. baseline & save outputs -launch_baseline -sleep 2 -wait_for_server "${BASELINE_HOST}" "${BASELINE_PORT}" -run_tests "http://${BASELINE_HOST}:${BASELINE_PORT}" "baseline" -cleanup -sleep 10 - - -# run disagg. & do exact-match with the outputs from baseline -launch_pd -launch_pd_proxy -sleep 10 -run_tests "http://${PROXY_HOST}:${PROXY_PORT}" "disagg" -echo "-----P/D success----" - -rm "${OUTPUT_FILE}" -cleanup - -exit 0 diff --git a/tests/v1/kv_connector/nixl_integration/run_tpu_edge_case_test.sh b/tests/v1/kv_connector/nixl_integration/run_tpu_edge_case_test.sh deleted file mode 100644 index 5969455025..0000000000 --- a/tests/v1/kv_connector/nixl_integration/run_tpu_edge_case_test.sh +++ /dev/null @@ -1,125 +0,0 @@ -#!/bin/bash -set -xe - -# Hosts / ports -PREFILL_HOST=${PREFILL_HOST:-"localhost"} -PREFILL_PORT=${PREFILL_PORT:-8100} -PREFILL_NIXL_SIDE_PORT=${PREFILL_NIXL_SIDE_PORT:-5577} -DECODE_HOST=${DECODE_HOST:-"localhost"} -DECODE_PORT=${DECODE_PORT:-8200} -PROXY_HOST=${PROXY_HOST:-"localhost"} -PROXY_PORT=${PROXY_PORT:-8192} -BASELINE_HOST=${BASELINE_HOST:-"localhost"} -BASELINE_PORT=${BASELINE_PORT:-9290} - - -# Model to run. -MODEL_NAME=${MODEL_NAME:-"meta-llama/Llama-3.2-3B-Instruct"} -MAX_MODEL_LEN=${MAX_MODEL_LEN:-1024} -BLOCK_SIZE=${BLOCK_SIZE:-32} - - -# execution env -SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd -P)" -GIT_ROOT="${GIT_ROOT:-$(cd -- "${SCRIPT_DIR}/../../../.." && pwd -P)}" -EXP_ROOT="${GIT_ROOT}/tests/v1/kv_connector/nixl_integration" -CONDA_PATH=${CONDA_PATH:-"/home/${USER}/anaconda3"} -CONDA_ENV_NAME=${CONDA_ENV_NAME:-"nixl"} - -OUTPUT_FILE=${OUTPUT_FILE:-"${EXP_ROOT}/.tpu_accuracy_test_outputs.txt"} - -# Trap the SIGINT signal (triggered by Ctrl+C) -trap 'kill $(jobs -pr)' SIGINT SIGTERM EXIT - -# Waits for vLLM server to start. -wait_for_server() { - local host=$1 - local port=$2 - timeout 1200 bash -c " - until curl -s ${host}:${port}/v1/completions > /dev/null; do - sleep 1 - done" && return 0 || return 1 -} - -# Cleanup function -cleanup() { - echo "Caught Ctrl+C, cleaning up..." - # Cleanup commands - pgrep python | xargs kill -9 || true - # pkill -f python || true - echo "Cleanup complete. Exiting." -} - - -launch_pd() { - PREFILL_BASE_CMD="source ${CONDA_PATH}/bin/activate ${CONDA_ENV_NAME}; - UCX_TLS=tcp \ - VLLM_MULTIPROC_EXECUTE_MODEL_TIMEOUT_S=200 \ - VLLM_LOGGING_LEVEL=DEBUG \ - VLLM_NIXL_SIDE_CHANNEL_HOST=${PREFILL_HOST} \ - VLLM_NIXL_SIDE_CHANNEL_PORT=${PREFILL_NIXL_SIDE_PORT} \ - PJRT_DEVICE=TPU \ - VLLM_WORKER_MULTIPROC_METHOD=spawn \ - VLLM_ENABLE_V1_MULTIPROCESSING=0 vllm serve $MODEL_NAME \ - --host ${PREFILL_HOST} \ - --port ${PREFILL_PORT} \ - --max-model-len ${MAX_MODEL_LEN}\ - --seed 42 \ - --block-size ${BLOCK_SIZE} \ - --enforce-eager \ - --gpu-memory-utilization 0.5 \ - --kv-transfer-config '{\"kv_connector\":\"NixlConnector\",\"kv_role\":\"kv_both\",\"kv_buffer_device\":\"cpu\"}'" - - - DECODE_BASE_CMD="source ${CONDA_PATH}/bin/activate ${CONDA_ENV_NAME}; - UCX_TLS=tcp \ - VLLM_MULTIPROC_EXECUTE_MODEL_TIMEOUT_S=200 \ - VLLM_LOGGING_LEVEL=DEBUG \ - PJRT_DEVICE=TPU \ - VLLM_WORKER_MULTIPROC_METHOD=spawn \ - VLLM_ENABLE_V1_MULTIPROCESSING=0 vllm serve $MODEL_NAME \ - --host ${DECODE_HOST} \ - --port ${DECODE_PORT} \ - --max-model-len ${MAX_MODEL_LEN}\ - --seed 42 \ - --block-size ${BLOCK_SIZE} \ - --enforce-eager \ - --gpu-memory-utilization 0.5 \ - --kv-transfer-config '{\"kv_connector\":\"NixlConnector\",\"kv_role\":\"kv_both\",\"kv_buffer_device\":\"cpu\"}'" - - echo "${PREFILL_BASE_CMD}" - echo "${DECODE_BASE_CMD}" - sleep 2 - - # execute on hosts - ssh -tt "${PREFILL_HOST}" "${PREFILL_BASE_CMD}" & - ssh -tt "${DECODE_HOST}" "${DECODE_BASE_CMD}" & - sleep 1 - wait_for_server "${PREFILL_HOST}" "${PREFILL_PORT}" - sleep 1 - wait_for_server "${DECODE_HOST}" "${DECODE_PORT}" - sleep 1 -} - -launch_pd_proxy(){ - PROXY_BASE_CMD="source ${CONDA_PATH}/bin/activate ${CONDA_ENV_NAME}; - python3 ${EXP_ROOT}/toy_proxy_server.py \ - --prefiller-host ${PREFILL_HOST} --prefiller-port ${PREFILL_PORT} \ - --decoder-host ${DECODE_HOST} --decoder-port ${DECODE_PORT} \ - --host=${PROXY_HOST} --port ${PROXY_PORT}" - echo "${PROXY_BASE_CMD}" - ssh -tt "${PROXY_HOST}" "${PROXY_BASE_CMD}" & -} - - -# run disagg. & do exact-match with the outputs from baseline -launch_pd -launch_pd_proxy -sleep 10 - -PREFILL_HOST=${PREFILL_HOST} \ -PREFILL_PORT=${PREFILL_PORT} \ -DECODE_HOST=${DECODE_HOST} \ -DECODE_PORT=${DECODE_PORT} \ -PROXY_HOST=${PROXY_HOST} \ -PROXY_PORT=${PROXY_PORT} python -m pytest -s -v "${GIT_ROOT}"/tests/v1/kv_connector/nixl_integration/test_edge_cases.py diff --git a/tests/v1/kv_connector/unit/offloading_connector/test_scheduler.py b/tests/v1/kv_connector/unit/offloading_connector/test_scheduler.py index 5b3bf1c8be..e71c3dbf18 100644 --- a/tests/v1/kv_connector/unit/offloading_connector/test_scheduler.py +++ b/tests/v1/kv_connector/unit/offloading_connector/test_scheduler.py @@ -77,6 +77,7 @@ from vllm.v1.kv_offload.base import ( make_offload_key, ) from vllm.v1.kv_offload.cpu.manager import CPUOffloadingManager +from vllm.v1.metrics.cache_hit_source import CacheHitSource from vllm.v1.outputs import KVConnectorOutput, ModelRunnerOutput from vllm.v1.request import RequestStatus @@ -197,14 +198,18 @@ def test_swa_offload_window_covers_unaligned_hit( manager.complete_load(job.keys, state.req_context) -def _make_partial_tail_scheduler() -> OffloadingConnectorScheduler: +def _make_partial_tail_scheduler( + recurrent_only: bool = False, +) -> OffloadingConnectorScheduler: vllm_config = _make_vllm_config(extra_config={"self_describing_kv_events": True}) - vllm_config.cache_config.prefix_match_unit = 4 + vllm_config.cache_config.prefix_match_unit = None if recurrent_only else 4 vllm_config.speculative_config = None vllm_config.kv_events_config = KVEventsConfig( enable_kv_cache_events=True, publisher="null" ) kv_cache_config = _make_mamba_hybrid_kv_cache_config() + if recurrent_only: + kv_cache_config.kv_cache_groups = kv_cache_config.kv_cache_groups[1:] spec = MockOffloadingSpec(build_offloading_config(vllm_config, kv_cache_config)) return OffloadingConnectorScheduler(spec, vllm_config, kv_cache_config) @@ -576,6 +581,284 @@ def test_recurrent_group_unhashed_block_does_not_truncate_load_boundary(): assert 42 in loaded +@pytest.mark.parametrize( + ("recurrent_source", "expected"), + [ + ("host", {CacheHitSource.P2P: 16, CacheHitSource.HOST: 12}), + ("p2p", {CacheHitSource.P2P: 28}), + (None, {CacheHitSource.P2P: 16, CacheHitSource.HOST: 12}), + ], +) +def test_external_cache_hit_sources_preserve_partial_tail_origin( + recurrent_source, expected +): + """Recurrent state covers the whole prefix; the outermost tier wins.""" + scheduler = _make_partial_tail_scheduler() + request = _make_partial_tail_request(scheduler) + req_status = scheduler._req_status["req"] + req_status.update_offload_keys() + + scheduler.manager.lookup.return_value = LookupResult.HIT + assert scheduler._lookup(req_status) == 28 + + def get_load_source(key, req_context): + if get_offload_group_idx(key) == 1: + assert recurrent_source is not None + return recurrent_source + block_hash = get_offload_block_hash(key) + return "p2p" if block_hash == b"h3" else "host" + + scheduler.manager.get_load_source.side_effect = get_load_source + + recurrent_block = KVCacheBlock(41) + if recurrent_source is None: + recurrent_block.set_block_hash( + make_block_hash_with_group_id(BlockHash(b"cached"), 1), 12 + ) + scheduler.update_state_after_alloc( + request, + KVCacheBlocks( + ( + [KVCacheBlock(31), KVCacheBlock(32)], + [KVCacheBlock(0, is_null=True), recurrent_block], + ) + ), + num_external_tokens=28, + ) + assert req_status.partial_tail_boundary is None + assert scheduler.get_external_cache_hit_sources(request, 28) == expected + + +@pytest.mark.parametrize( + ("second_source", "expected"), + [ + ("host", {CacheHitSource.HOST: 4, CacheHitSource.DISK: 4}), + ("p2p", {CacheHitSource.P2P: 4, CacheHitSource.DISK: 4}), + ], +) +def test_external_cache_hit_sources_reconcile_different_group_chunk_sizes( + request_runner, + second_source, + expected, +): + """Groups with different chunk sizes overlap; the outermost tier wins.""" + block_size = 4 + groups = [ + KVCacheGroupSpec( + [f"layer{group_idx}"], + FullAttentionSpec( + block_size=block_size * (group_idx + 1), + num_kv_heads=1, + head_size=1, + dtype=torch.float32, + ), + ) + for group_idx in range(2) + ] + runner = request_runner( + block_size=block_size, + num_gpu_blocks=100, + async_scheduling=False, + kv_cache_groups=groups, + ) + scheduler = runner.connector_scheduler + request = MagicMock(request_id="mixed") + req_status = RequestOffloadState( + config=scheduler.config, + req=request, + req_context=ReqContext(req_id=request.request_id), + offloading_context=RequestOffloadingContext(policy=OffloadPolicy.CHUNK_LEVEL), + ) + for group_idx, group_state in enumerate(req_status.group_states): + group_state.offload_keys = [ + make_offload_key(f"h{chunk_idx}".encode(), group_idx) + for chunk_idx in range(2 // (group_idx + 1)) + ] + scheduler._req_status[request.request_id] = req_status + runner.manager.get_load_source.side_effect = lambda key, req_context: ( + ("host" if get_offload_block_hash(key) == b"h0" else "disk") + if get_offload_group_idx(key) == 0 + else second_source + ) + scheduler.update_state_after_alloc( + request, + KVCacheBlocks(([KVCacheBlock(1), KVCacheBlock(2)], [KVCacheBlock(3)])), + 2 * block_size, + ) + + assert scheduler.get_external_cache_hit_sources(request, 2 * block_size) == expected + + +@pytest.mark.parametrize( + ("ranges", "expected"), + [ + pytest.param( + [(8, 12, "host"), (0, 8, "host"), (4, 10, "host")], + {CacheHitSource.HOST: 8, CacheHitSource.EXTERNAL_UNSPECIFIED: 4}, + id="overlapping-same-source-and-adjacent-endpoints", + ), + pytest.param( + [(10, 20, "p2p"), (6, 10, "disk"), (0, 8, "host"), (4, 12, "host")], + {CacheHitSource.HOST: 2, CacheHitSource.DISK: 4, CacheHitSource.P2P: 6}, + id="nested-tiers-and-clipped-ranges", + ), + pytest.param( + [(0, 4, "disk"), (16, 20, "disk"), (8, 8, "disk")], + {CacheHitSource.EXTERNAL_UNSPECIFIED: 12}, + id="no-covering-ranges", + ), + ], +) +def test_external_cache_hit_sources_range_boundaries(ranges, expected): + scheduler = _make_partial_tail_scheduler() + request = _make_partial_tail_request(scheduler) + state = scheduler._req_status[request.request_id] + state.num_locally_computed_tokens = 4 + for idx, (lo, hi, source) in enumerate(ranges): + key = make_offload_key(str(idx).encode(), 0) + state.load_key_ranges.append((lo, hi, key)) + scheduler.manager.get_load_source.side_effect = lambda key, ctx: ranges[ + int(get_offload_block_hash(key)) + ][2] + + result = scheduler.get_external_cache_hit_sources(request, 12) + assert result == expected + assert sum(result.values()) == 12 + + +@pytest.mark.parametrize("source", ["host", "disk", "p2p", "external_unspecified"]) +def test_external_cache_hit_sources_recurrent_only_state(source): + scheduler = _make_partial_tail_scheduler(recurrent_only=True) + request = _make_partial_tail_request(scheduler) + request.num_prompt_tokens = request.num_tokens = 64 + request.block_hashes = [BlockHash(f"h{i}".encode()) for i in range(4)] + request.all_token_ids = list(range(64)) + state = scheduler._req_status[request.request_id] + state.update_offload_keys() + scheduler.manager.lookup.return_value = LookupResult.HIT + assert scheduler._lookup(state) == 48 + scheduler.manager.get_load_source.return_value = source + scheduler.update_state_after_alloc( + request, + KVCacheBlocks( + ( + [ + KVCacheBlock(0, is_null=True), + KVCacheBlock(0, is_null=True), + KVCacheBlock(41), + ], + ) + ), + 48, + ) + assert scheduler.get_external_cache_hit_sources(request, 48) == { + CacheHitSource(source): 48 + } + scheduler.manager.get_load_source.assert_called_once() + + +@pytest.mark.parametrize("sparse_kind", ["sliding", "chunked_local"]) +@pytest.mark.parametrize("blocks_per_chunk", [1, 2]) +@pytest.mark.parametrize("local_tokens", [0, 4]) +@pytest.mark.parametrize("full_attention", [False, True]) +@pytest.mark.parametrize( + "sparse_source", ["host", "disk", "p2p", "external_unspecified", "mixed"] +) +def test_external_cache_hit_sources_use_loaded_attention_ranges( + sparse_kind, + blocks_per_chunk, + local_tokens, + full_attention, + sparse_source, +): + """SWA/local attention affects only loaded positions, not omitted history.""" + block_size = 4 + chunk_size = block_size * blocks_per_chunk + spec_kwargs = dict( + block_size=block_size, num_kv_heads=1, head_size=1, dtype=torch.float32 + ) + sparse_spec = ( + SlidingWindowSpec(**spec_kwargs, sliding_window=2 * chunk_size) + if sparse_kind == "sliding" + else ChunkedLocalAttentionSpec( + **spec_kwargs, attention_chunk_size=2 * chunk_size + ) + ) + groups = [KVCacheGroupSpec(["sparse"], sparse_spec)] + if full_attention: + groups.insert(0, KVCacheGroupSpec(["full"], FullAttentionSpec(**spec_kwargs))) + vllm_config = _make_vllm_config(extra_config={"block_size": chunk_size}) + vllm_config.cache_config.block_size = block_size + vllm_config.speculative_config = None + kv_cache_config = KVCacheConfig( + num_blocks=100, + kv_cache_tensors=[], + kv_cache_groups=groups, + ) + spec = MockOffloadingSpec(build_offloading_config(vllm_config, kv_cache_config)) + scheduler = OffloadingConnectorScheduler(spec, vllm_config, kv_cache_config) + request = MagicMock(request_id="sparse") + req_status = RequestOffloadState( + config=scheduler.config, + req=request, + req_context=ReqContext(req_id=request.request_id), + offloading_context=RequestOffloadingContext(policy=OffloadPolicy.CHUNK_LEVEL), + num_locally_computed_tokens=local_tokens, + ) + scheduler._req_status[request.request_id] = req_status + block_groups = [] + for group_idx, state in enumerate(req_status.group_states): + state.offload_keys = [ + make_offload_key(str(i).encode(), group_idx) for i in range(4) + ] + block_groups.append( + [ + KVCacheBlock( + 0 + if group_idx == len(groups) - 1 and i < 2 * blocks_per_chunk + else 1 + group_idx * 100 + i, + is_null=group_idx == len(groups) - 1 and i < 2 * blocks_per_chunk, + ) + for i in range(4 * blocks_per_chunk) + ] + ) + + def get_load_source(key, req_context): + chunk_idx = int(get_offload_block_hash(key)) + if get_offload_group_idx(key) == len(groups) - 1: + assert chunk_idx >= 2 # Earlier sparse keys were not loaded. + if sparse_source == "mixed": + return "host" if chunk_idx == 2 else "disk" + return sparse_source + return "host" + + scheduler.manager.get_load_source.side_effect = get_load_source + count = 4 * chunk_size - local_tokens + scheduler.update_state_after_alloc( + request, KVCacheBlocks(tuple(block_groups)), count + ) + expected: dict[CacheHitSource, int] = {} + + def expect(source: CacheHitSource, num_tokens: int) -> None: + expected[source] = expected.get(source, 0) + num_tokens + + expect( + CacheHitSource.HOST if full_attention else CacheHitSource.EXTERNAL_UNSPECIFIED, + 2 * chunk_size - local_tokens, + ) + for chunk_idx in range(2, 4): + source = ( + ("host" if chunk_idx == 2 else "disk") + if sparse_source == "mixed" + else sparse_source + ) + expect(CacheHitSource(source), chunk_size) + result = scheduler.get_external_cache_hit_sources(request, count) + assert result == expected + assert sum(result.values()) == count + assert scheduler.get_external_cache_hit_sources(request, 0) == {} + + def test_partial_lookup_requires_every_cache_group(): scheduler = _make_partial_tail_scheduler() _make_partial_tail_request(scheduler) @@ -1400,7 +1683,6 @@ def test_two_groups_full_and_sliding_window(request_runner, async_scheduling: bo # are within the window → loads blocks 1,2 expected_loaded=((0, 0), (0, 1), (0, 2), (1, 1), (1, 2)), ) - runner.manager.touch.assert_not_called() # 3 blocks are hit on GPU [0, 1, 2] diff --git a/tests/v1/kv_connector/unit/offloading_connector/utils.py b/tests/v1/kv_connector/unit/offloading_connector/utils.py index cb766c6676..9621890f3c 100644 --- a/tests/v1/kv_connector/unit/offloading_connector/utils.py +++ b/tests/v1/kv_connector/unit/offloading_connector/utils.py @@ -57,6 +57,7 @@ from vllm.v1.kv_offload.base import ( make_offload_key, ) from vllm.v1.kv_offload.config import OffloadingConfig +from vllm.v1.metrics.cache_hit_source import CacheHitSource from vllm.v1.request import Request from vllm.v1.structured_output import StructuredOutputManager @@ -129,6 +130,7 @@ class MockOffloadingSpec(OffloadingSpec): self.manager = MagicMock(spec=OffloadingManager) self.manager.prepare_load = lambda keys, req_context: MockLoadStoreSpec(keys) self.manager.lookup.return_value = LookupResult.MISS + self.manager.get_load_source.return_value = CacheHitSource.EXTERNAL_UNSPECIFIED self.manager.get_stats.return_value = None self.manager.on_new_request.return_value = RequestOffloadingContext() self.handler = MockOffloadingWorker() diff --git a/tests/v1/kv_connector/unit/test_cache_pollution_prevention.py b/tests/v1/kv_connector/unit/test_cache_pollution_prevention.py index 21b50a1379..0515994543 100644 --- a/tests/v1/kv_connector/unit/test_cache_pollution_prevention.py +++ b/tests/v1/kv_connector/unit/test_cache_pollution_prevention.py @@ -8,7 +8,6 @@ prefix cache hash table so future requests cannot match and reuse corrupted data """ from collections.abc import Callable -from unittest.mock import Mock import pytest @@ -16,6 +15,7 @@ from vllm.v1.core.sched.scheduler import Scheduler from vllm.v1.request import Request, RequestStatus from .utils import ( + create_mock_connector, create_model_runner_output, create_request, create_scheduler, @@ -73,7 +73,7 @@ def test_invalid_blocks_evicted_prevents_cache_pollution( } # mock connector indicating sync load - fail_scheduler.connector = Mock() + fail_scheduler.connector = create_mock_connector() fail_scheduler.connector.get_num_new_matched_tokens.side_effect = ( _make_get_num_new_matched_tokens(req_num_new_matched_tokens, False) ) diff --git a/tests/v1/kv_connector/unit/test_error_propagation.py b/tests/v1/kv_connector/unit/test_error_propagation.py index 35a3749e08..ded110d55b 100644 --- a/tests/v1/kv_connector/unit/test_error_propagation.py +++ b/tests/v1/kv_connector/unit/test_error_propagation.py @@ -2,7 +2,6 @@ # SPDX-FileCopyrightText: Copyright contributors to the vLLM project from collections.abc import Callable -from unittest.mock import Mock import pytest @@ -10,6 +9,7 @@ from vllm.v1.core.sched.scheduler import Scheduler from vllm.v1.request import FinishReason, Request, RequestStatus from .utils import ( + create_mock_connector, create_model_runner_output, create_request, create_scheduler, @@ -56,7 +56,7 @@ def test_error_propagation_sync_load(fail_scheduler: Scheduler): request.request_id: num_external_computed_tokens, } - fail_scheduler.connector = Mock() + fail_scheduler.connector = create_mock_connector() fail_scheduler.connector.get_num_new_matched_tokens.side_effect = ( _make_get_num_new_matched_tokens(req_num_new_matched_tokens, False) ) @@ -110,7 +110,7 @@ def test_error_propagation_async_load(fail_scheduler: Scheduler): request.request_id: num_external_computed_tokens, } - fail_scheduler.connector = Mock() + fail_scheduler.connector = create_mock_connector() fail_scheduler.connector.get_loaded_kv_cache_group_ids.return_value = (0,) fail_scheduler.connector.get_num_new_matched_tokens.side_effect = ( _make_get_num_new_matched_tokens(req_num_new_matched_tokens, True) diff --git a/tests/v1/kv_connector/unit/test_hisparse_connector.py b/tests/v1/kv_connector/unit/test_hisparse_connector.py index a2eeb8cd63..f08982a5e9 100644 --- a/tests/v1/kv_connector/unit/test_hisparse_connector.py +++ b/tests/v1/kv_connector/unit/test_hisparse_connector.py @@ -64,6 +64,42 @@ def test_hisparse_requires_block_outermost_device_layout(): assert HiSparseConnector.get_required_kvcache_layout(MagicMock()) == "BLHNC" +def test_scheduler_stats_report_host_pool_usage(): + """The scheduler-side stats sample the coordinator's host pool level.""" + from tests.v1.core.test_prefix_caching import ( + HISPARSE_BLOCK_SIZE, + make_hisparse_kv_cache_manager, + make_request, + sha256, + ) + from vllm.v1.core.kv_cache_utils import init_none_hash + + init_none_hash(sha256) + + manager = make_hisparse_kv_cache_manager(32, host_num_blocks=8) + scheduler = HiSparseConnectorScheduler(async_speculative=False) + scheduler.bind_coordinator(get_hisparse_coordinator(manager)) + coordinator = scheduler.coordinator + + stats = scheduler.get_kv_connector_stats() + assert stats.data["host_cache_usage_perc"] == [0.0] + assert stats.data["pending_page_transfers"] == [0] + + request = make_request( + "request", list(range(4 * HISPARSE_BLOCK_SIZE)), HISPARSE_BLOCK_SIZE, sha256 + ) + assert manager.allocate_slots(request, num_new_tokens=64) is not None + host_blocks = coordinator.host_manager.req_to_blocks[request.request_id] + num_used = sum(not block.is_null for block in host_blocks) + + # The pool reserves the null block, so 8 configured blocks yield 7 usable. + expected = pytest.approx(num_used / 7) + assert 0 < num_used <= 7 + for _ in range(2): + stats = scheduler.get_kv_connector_stats() + assert stats.data["host_cache_usage_perc"] == [expected] + + def test_no_forward_enqueues_deferred_hisparse_transfers(): """A zero-token step must still enqueue deferred post-forward transfers.""" connector = object.__new__(ActiveKVConnector) diff --git a/tests/v1/kv_connector/unit/test_hisparse_stats.py b/tests/v1/kv_connector/unit/test_hisparse_stats.py index 09904d0a3c..91d00b1e22 100644 --- a/tests/v1/kv_connector/unit/test_hisparse_stats.py +++ b/tests/v1/kv_connector/unit/test_hisparse_stats.py @@ -26,6 +26,28 @@ def test_record_snapshot_and_reduce(): assert reduced["HiSparse host-to-device bytes"] == 64 +def test_record_host_usage_reports_latest_level(): + stats = HiSparseKVConnectorStats() + stats.record_host_usage(usage=0.4, pending_page_transfers=2) + stats.record_host_usage(usage=0.8, pending_page_transfers=0) + + reduced = stats.reduce() + assert reduced["HiSparse host KV cache usage %"] == 80.0 + + +def test_aggregate_keeps_latest_host_usage_level(): + first = HiSparseKVConnectorStats() + first.record_host_usage(usage=0.4, pending_page_transfers=2) + second = HiSparseKVConnectorStats() + second.record_host_usage(usage=0.8, pending_page_transfers=0) + second.record_host_usage(usage=0.6, pending_page_transfers=1) + + first.aggregate(second) + + assert first.data["host_cache_usage_perc"] == [0.4, 0.6] + assert first.data["pending_page_transfers"] == [2, 1] + + def test_aggregate_extends_snapshot_deltas(): first = HiSparseKVConnectorStats() first.record_snapshot(hits=7, misses=3, host_to_device_bytes=48) @@ -34,11 +56,9 @@ def test_aggregate_extends_snapshot_deltas(): first.aggregate(second) - assert first.data == { - "cache_hits": [7, 5], - "cache_misses": [3, 1], - "host_to_device_bytes": [48, 16], - } + assert first.data["cache_hits"] == [7, 5] + assert first.data["cache_misses"] == [3, 1] + assert first.data["host_to_device_bytes"] == [48, 16] def test_aggregate_skips_empty_stats(): @@ -60,3 +80,54 @@ def test_build_kv_connector_stats_round_trip(): assert rebuilt is not None assert isinstance(rebuilt, HiSparseKVConnectorStats) assert rebuilt.reduce() == stats.reduce() + + +def test_prom_metrics_observe_host_usage_gauges(): + from types import SimpleNamespace + from typing import Any + + from prometheus_client import Counter, Gauge, Histogram + + from vllm.distributed.kv_transfer.kv_connector.v1.hisparse.stats import ( + HiSparsePromMetrics, + ) + + class _FakeMetric: + def __init__(self, **kwargs: Any): + self.kwargs = kwargs + self.increments: list[int | float] = [] + self.set_values: list[int | float] = [] + + def labels(self, *labelvalues: object) -> "_FakeMetric": + return self + + def inc(self, value: int | float) -> None: + self.increments.append(value) + + def set(self, value: int | float) -> None: + self.set_values.append(value) + + created: dict[str, _FakeMetric] = {} + + class _NamedFake(_FakeMetric): + def __init__(self, **kwargs: Any): + super().__init__(**kwargs) + created[kwargs["name"]] = self + + prom = HiSparsePromMetrics( + vllm_config=SimpleNamespace(kv_transfer_config=None), + metric_types={Gauge: _NamedFake, Counter: _NamedFake, Histogram: _NamedFake}, + labelnames=["model_name"], + per_engine_labelvalues={0: ["model"]}, + ) + + stats = HiSparseKVConnectorStats() + stats.record_snapshot(hits=3, misses=2, host_to_device_bytes=32) + stats.record_host_usage(usage=0.5, pending_page_transfers=1) + stats.record_host_usage(usage=0.75, pending_page_transfers=3) + prom.observe(stats.to_dict()) + + assert created["vllm:hisparse_cache_hits"].increments == [3] + assert created["vllm:hisparse_host_to_device_bytes"].increments == [32] + assert created["vllm:hisparse_host_cache_usage_perc"].set_values == [0.75] + assert created["vllm:hisparse_pending_page_transfers"].set_values == [3] diff --git a/tests/v1/kv_connector/unit/test_invalid_blocks_correctness.py b/tests/v1/kv_connector/unit/test_invalid_blocks_correctness.py index 7ea2170584..f328332696 100644 --- a/tests/v1/kv_connector/unit/test_invalid_blocks_correctness.py +++ b/tests/v1/kv_connector/unit/test_invalid_blocks_correctness.py @@ -11,7 +11,6 @@ These tests verify correct behavior in three scenarios: """ from collections.abc import Callable -from unittest.mock import Mock import pytest @@ -19,6 +18,7 @@ from vllm.v1.core.sched.scheduler import Scheduler from vllm.v1.request import FinishReason, Request, RequestStatus from .utils import ( + create_mock_connector, create_model_runner_output, create_request, create_scheduler, @@ -83,7 +83,7 @@ def test_sync_recompute_blocks_not_freed_for_running_requests( } # mock connector indicating sync load - recompute_scheduler.connector = Mock() + recompute_scheduler.connector = create_mock_connector() recompute_scheduler.connector.get_num_new_matched_tokens.side_effect = ( _make_get_num_new_matched_tokens(req_num_new_matched_tokens, False) ) @@ -206,7 +206,7 @@ def test_sync_fail_invalid_blocks_evicted(fail_scheduler: Scheduler): } # mock connector indicating sync load - fail_scheduler.connector = Mock() + fail_scheduler.connector = create_mock_connector() fail_scheduler.connector.get_num_new_matched_tokens.side_effect = ( _make_get_num_new_matched_tokens(req_num_new_matched_tokens, False) ) @@ -323,7 +323,7 @@ def test_async_recompute_blocks_not_cached_when_invalid( } # mock connector indicating async load - recompute_scheduler.connector = Mock() + recompute_scheduler.connector = create_mock_connector() recompute_scheduler.connector.get_num_new_matched_tokens.side_effect = ( _make_get_num_new_matched_tokens(req_num_new_matched_tokens, True) ) diff --git a/tests/v1/kv_connector/unit/test_kv_load_failure_recovery.py b/tests/v1/kv_connector/unit/test_kv_load_failure_recovery.py index 3dbbfdc6b2..5dce592576 100644 --- a/tests/v1/kv_connector/unit/test_kv_load_failure_recovery.py +++ b/tests/v1/kv_connector/unit/test_kv_load_failure_recovery.py @@ -2,7 +2,6 @@ # SPDX-FileCopyrightText: Copyright contributors to the vLLM project from collections.abc import Callable -from unittest.mock import Mock import pytest @@ -10,6 +9,7 @@ from vllm.v1.core.sched.scheduler import Scheduler from vllm.v1.request import Request, RequestStatus from .utils import ( + create_mock_connector, create_model_runner_output, create_request, create_scheduler, @@ -41,7 +41,7 @@ def test_failed_receive_completion_honors_failure_policy(policy): scheduler = create_scheduler(create_vllm_config(kv_load_failure_policy=policy)) request = create_request(num_tokens=3 * scheduler.block_size) scheduler.add_request(request) - scheduler.connector = Mock() + scheduler.connector = create_mock_connector() scheduler.connector.get_loaded_kv_cache_group_ids.return_value = (0,) scheduler.connector.get_num_new_matched_tokens.side_effect = [ (2 * scheduler.block_size, True), @@ -103,7 +103,7 @@ def test_async_load_failure( request3.request_id: num_external_computed_tokens, } - scheduler.connector = Mock() + scheduler.connector = create_mock_connector() scheduler.connector.get_loaded_kv_cache_group_ids.return_value = (0,) scheduler.connector.get_num_new_matched_tokens.side_effect = ( _make_get_num_new_matched_tokens(req_num_new_matched_tokens, async_load=True) @@ -181,7 +181,7 @@ def test_sync_load_failure( request3.request_id: num_external_computed_tokens, } - scheduler.connector = Mock() + scheduler.connector = create_mock_connector() scheduler.connector.get_num_new_matched_tokens.side_effect = ( _make_get_num_new_matched_tokens(req_num_new_matched_tokens, async_load=False) ) @@ -262,7 +262,7 @@ def test_sync_load_failure_with_shared_blocks( request1.request_id: num_external_computed_tokens, } - scheduler.connector = Mock() + scheduler.connector = create_mock_connector() scheduler.connector.get_num_new_matched_tokens.side_effect = ( _make_get_num_new_matched_tokens(req_num_new_matched_tokens, async_load=False) ) @@ -333,7 +333,7 @@ def test_async_progressive_load_failure( request.request_id: num_external_computed_tokens, } - scheduler.connector = Mock() + scheduler.connector = create_mock_connector() scheduler.connector.get_loaded_kv_cache_group_ids.return_value = (0,) scheduler.connector.get_num_new_matched_tokens.side_effect = ( _make_get_num_new_matched_tokens(req_num_new_matched_tokens, async_load=True) diff --git a/tests/v1/kv_connector/unit/test_mooncake_store_connector.py b/tests/v1/kv_connector/unit/test_mooncake_store_connector.py index f3ffa55a54..824f806f48 100644 --- a/tests/v1/kv_connector/unit/test_mooncake_store_connector.py +++ b/tests/v1/kv_connector/unit/test_mooncake_store_connector.py @@ -38,6 +38,7 @@ from vllm.v1.kv_cache_interface import ( KVCacheTensor, MambaSpec, ) +from vllm.v1.metrics.cache_hit_source import CacheHitSource from vllm.v1.outputs import KVConnectorOutput from .utils import create_vllm_config @@ -208,6 +209,29 @@ def test_scheduler_role_initializes_store_scheduler_only(): mock_scheduler.return_value.bind_gpu_block_pool.assert_called_once_with(block_pool) +def test_scheduler_reports_mooncake_cache_source(): + vllm_config = _make_vllm_config() + kv_cache_config = _make_kv_cache_config() + + with ( + set_current_vllm_config(vllm_config), + patch( + "vllm.distributed.kv_transfer.kv_connector.v1.mooncake.store." + "connector.MooncakeStoreScheduler" + ), + ): + connector = mooncake_store_connector.MooncakeStoreConnector( + vllm_config, KVConnectorRole.SCHEDULER, kv_cache_config + ) + + # Mooncake Store is an external cache service. Its internal memory and + # local-storage policy is intentionally hidden behind that stable label. + assert connector.get_external_cache_hit_sources( + None, # type: ignore[arg-type] + 32, + ) == {CacheHitSource.EXTERNAL_UNSPECIFIED: 32} + + def test_worker_methods_delegate_to_store_worker(): vllm_config = _make_vllm_config() kv_cache_config = _make_kv_cache_config() diff --git a/tests/v1/kv_connector/unit/test_moriio_connector.py b/tests/v1/kv_connector/unit/test_moriio_connector.py index f95e12c601..73cc5f0583 100644 --- a/tests/v1/kv_connector/unit/test_moriio_connector.py +++ b/tests/v1/kv_connector/unit/test_moriio_connector.py @@ -3,6 +3,8 @@ import importlib.util import socket import uuid +from types import SimpleNamespace +from typing import Any from unittest.mock import MagicMock, patch import msgspec @@ -20,11 +22,13 @@ from vllm.config import ( set_current_vllm_config, ) from vllm.distributed.kv_transfer.kv_connector.v1.moriio.moriio_common import ( + ROLE, MoRIIOAgentMetadata, MoRIIOConnectorMetadata, MoRIIOConstants, MoRIIOMode, resolve_host_ip, + set_role, zmq_ctx, ) from vllm.distributed.kv_transfer.kv_connector.v1.moriio.moriio_connector import ( @@ -147,6 +151,61 @@ def _write_consumer_scheduler_for_finished_request(tp_size: int = 2): return scheduler +def _write_producer_scheduler(block_size: int = 1) -> Any: + """Bare WRITE-mode PRODUCER scheduler exercising the save path without a + real engine/RDMA stack. Uses a single full-attention KV group (index 0).""" + scheduler = MoRIIOConnectorScheduler.__new__(MoRIIOConnectorScheduler) + scheduler.mode = MoRIIOMode.WRITE + scheduler.is_producer = True + scheduler.block_size = block_size + scheduler.blocks_per_sw = [0] + scheduler._group_block_sizes = [block_size] + scheduler.transfer_id_to_request_id = {} + scheduler._reqs_need_recv = {} + scheduler._reqs_need_save = {} + scheduler._reqs_need_pending_save = {} + scheduler._req_kv_params = {} + scheduler._reqs_need_send = {} + return scheduler + + +def _spec_kv_params(transfer_id: str = "xfer-spec") -> dict[str, Any]: + # request_id embeds no zmq address, so add_new_req resolves the peer here. + return { + "transfer_id": transfer_id, + "remote_engine_id": "remote-engine", + "remote_block_ids": [], + "remote_host": "127.0.0.1", + "remote_handshake_port": 5000, + "remote_notify_port": 5001, + } + + +def _build_meta( + scheduler: Any, req_ids=None, new_block_ids=None, num_scheduled_tokens=None +) -> Any: + # Final-chunk detection reads num_scheduled_tokens, so expose it. + scheduler_output = SimpleNamespace( + scheduled_cached_reqs=SimpleNamespace( + req_ids=req_ids or [], + new_block_ids=new_block_ids or [], + ), + num_scheduled_tokens=num_scheduled_tokens or {}, + ) + return scheduler.build_connector_meta(scheduler_output) + + +def test_write_mode_does_not_reload_remote_kv_after_preemption(): + scheduler = _write_consumer_scheduler_for_finished_request() + request = create_request(request_id=1, num_tokens=32, do_remote_prefill=True) + + assert scheduler.get_num_new_matched_tokens(request, 0) == (32, True) + + assert request.kv_transfer_params is not None + request.kv_transfer_params["do_remote_prefill"] = False + assert scheduler.get_num_new_matched_tokens(request, 0) == (0, False) + + class FakeMoRIIOWrapper: # A fake MoRIIOWrapper for testing purposes def __init__(self, *args, **kwargs): @@ -700,6 +759,203 @@ def test_resolve_host_ip_prefers_extra_config(): assert resolve_host_ip({"host_ip": ""}) == fallback +def test_write_mode_excludes_spec_lookahead_blocks(): + # The save path must record only the prompt blocks, clamping off the + # trailing lookahead blocks decode never allocates. + set_role(ROLE.PRODUCER) + scheduler = _write_producer_scheduler(block_size=1) + + req_id = "req-spec" + prompt_blocks = [10, 11, 12, 13, 14, 15, 16, 17] + lookahead_blocks = [18, 19, 20] + local_block_ids = [prompt_blocks + lookahead_blocks] + + # block_size=1 => num_prompt_tokens == prompt block count, all scheduled in + # this single chunk, so final-chunk detection fires. + num_prompt_tokens = len(prompt_blocks) + req = SimpleNamespace( + request_id=req_id, + num_prompt_tokens=num_prompt_tokens, + num_computed_tokens=0, + kv_transfer_params=_spec_kv_params(), + ) + scheduler._reqs_need_save[req_id] = (req, local_block_ids) + scheduler._req_kv_params[req_id] = _spec_kv_params() + + meta = _build_meta(scheduler, num_scheduled_tokens={req_id: num_prompt_tokens}) + + assert isinstance(meta, MoRIIOConnectorMetadata) + assert req_id in meta.reqs_to_save + saved = meta.reqs_to_save[req_id] + assert saved.local_block_ids == [prompt_blocks] + + +def test_write_mode_chunked_prefill_clamps_spec_lookahead_blocks(): + # Non-final chunks are buffered untouched; lookahead blocks are only clamped + # on the final-chunk save. + set_role(ROLE.PRODUCER) + scheduler = _write_producer_scheduler(block_size=1) + + req_id = "req-spec-chunked" + num_prompt_tokens = 8 + first_chunk = [10, 11, 12, 13] + final_chunk = [14, 15, 16, 17, 18, 19, 20] # 4 prompt + 3 lookahead + prompt_blocks = [10, 11, 12, 13, 14, 15, 16, 17] + + req = SimpleNamespace( + request_id=req_id, + num_prompt_tokens=num_prompt_tokens, + num_computed_tokens=0, + kv_transfer_params=_spec_kv_params(), + ) + + # Step 1: first chunk (4 of 8 tokens) is buffered, not saved. + scheduler._reqs_need_save[req_id] = (req, [first_chunk]) + scheduler._req_kv_params[req_id] = _spec_kv_params() + meta_step1 = _build_meta(scheduler, num_scheduled_tokens={req_id: 4}) + assert req_id not in meta_step1.reqs_to_save + assert req_id in scheduler._reqs_need_pending_save + assert scheduler._reqs_need_pending_save[req_id][1] == [first_chunk] + + # Step 2: final chunk (computed 4 + scheduled 4 >= 8) assembles the full + # local set, clamps the lookahead, and records only the prompt blocks. + req.num_computed_tokens = 4 + meta_step2 = _build_meta( + scheduler, + req_ids=[req_id], + new_block_ids=[[final_chunk]], + num_scheduled_tokens={req_id: 4}, + ) + assert req_id in meta_step2.reqs_to_save + assert meta_step2.reqs_to_save[req_id].local_block_ids == [prompt_blocks] + assert req_id not in scheduler._reqs_need_pending_save + + +@pytest.mark.parametrize( + ("block_size", "num_prompt_tokens", "num_prompt_blocks"), + [ + # block_size=4: exact multiple (2 full blocks) and non-multiple (3 blocks, + # last one partial). + pytest.param(4, 8, 2, id="bs4-exact-multiple"), + pytest.param(4, 9, 3, id="bs4-non-multiple"), + # block_size=16: exact multiple (2 full blocks) and non-multiple (3 + # blocks, last one partial). + pytest.param(16, 32, 2, id="bs16-exact-multiple"), + pytest.param(16, 40, 3, id="bs16-non-multiple"), + ], +) +def test_write_mode_clamps_prompt_blocks_with_block_size_gt_1( + block_size, num_prompt_tokens, num_prompt_blocks +): + # With block_size > 1 the clamp must keep exactly + # ceil(num_prompt_tokens / block_size) prompt blocks and drop the tail. + set_role(ROLE.PRODUCER) + scheduler = _write_producer_scheduler(block_size=block_size) + + req_id = f"req-bs{block_size}-{num_prompt_tokens}" + prompt_blocks = list(range(10, 10 + num_prompt_blocks)) + lookahead_blocks = [10 + num_prompt_blocks] + local_block_ids = [prompt_blocks + lookahead_blocks] + + req = SimpleNamespace( + request_id=req_id, + num_prompt_tokens=num_prompt_tokens, + num_computed_tokens=0, + kv_transfer_params=_spec_kv_params(), + ) + scheduler._reqs_need_save[req_id] = (req, local_block_ids) + scheduler._req_kv_params[req_id] = _spec_kv_params() + + meta = _build_meta(scheduler, num_scheduled_tokens={req_id: num_prompt_tokens}) + + assert isinstance(meta, MoRIIOConnectorMetadata) + assert req_id in meta.reqs_to_save + assert len(prompt_blocks) == num_prompt_blocks + assert meta.reqs_to_save[req_id].local_block_ids == [prompt_blocks] + + +def test_write_mode_defers_save_until_prompt_complete_with_lookahead(): + # Under speculative decoding the producer reserves lookahead blocks beyond + # the last-scheduled prompt token, inflating the accumulated block count on + # a non-final chunk. Token-progress detection must keep deferring until the + # prompt is complete, where the lookahead is clamped off. + # + # block_size=1, num_prompt_tokens=10, chunk=4, num_lookahead=3: after 8 + # prompt tokens the producer holds 8 + 3 = 11 blocks, but token progress + # 8 < 10 must keep waiting until the final chunk completes tokens 9-10. + set_role(ROLE.PRODUCER) + scheduler = _write_producer_scheduler(block_size=1) + + req_id = "req-lookahead-final-chunk" + num_prompt_tokens = 10 + + chunk1_blocks = [100, 101, 102, 103] # prompt tokens 1-4 + chunk2_blocks = [104, 105, 106, 107, 108, 109, 110] # 4 prompt + 3 lookahead + prompt_blocks = [100, 101, 102, 103, 104, 105, 106, 107, 108, 109] + + req = SimpleNamespace( + request_id=req_id, + num_prompt_tokens=num_prompt_tokens, + num_computed_tokens=0, + kv_transfer_params=_spec_kv_params(), + ) + scheduler._req_kv_params[req_id] = _spec_kv_params() + + # Chunk 1 (tokens 1-4): computed 0 + scheduled 4 < 10 -> buffered. + scheduler._reqs_need_save[req_id] = (req, [chunk1_blocks]) + meta1 = _build_meta(scheduler, num_scheduled_tokens={req_id: 4}) + assert req_id not in meta1.reqs_to_save, "chunk 1 must not be saved" + assert req_id in scheduler._reqs_need_pending_save + assert scheduler._reqs_need_pending_save[req_id][1] == [chunk1_blocks] + + # Chunk 2 (tokens 5-8) inflates the buffered count to 11 via lookahead, but + # token progress 4 + 4 = 8 < 10 keeps the request deferred. + req.num_computed_tokens = 4 + meta2 = _build_meta( + scheduler, + req_ids=[req_id], + new_block_ids=[[chunk2_blocks]], + num_scheduled_tokens={req_id: 4}, + ) + assert req_id not in meta2.reqs_to_save, ( + "second-to-last chunk saved prematurely despite incomplete prompt" + ) + assert req_id in scheduler._reqs_need_pending_save + assert scheduler._reqs_need_pending_save[req_id][1] == [ + chunk1_blocks + chunk2_blocks + ] + + # Final chunk (tokens 9-10, no new blocks): computed 8 + scheduled 2 >= 10 + # emits the save with the trailing lookahead block clamped off. + req.num_computed_tokens = 8 + meta3 = _build_meta( + scheduler, + req_ids=[req_id], + new_block_ids=[None], + num_scheduled_tokens={req_id: 2}, + ) + assert req_id in meta3.reqs_to_save, "final chunk must emit the WRITE save" + assert meta3.reqs_to_save[req_id].local_block_ids == [prompt_blocks] + assert req_id not in scheduler._reqs_need_pending_save + + +def test_clamp_to_prompt_blocks_clamps_each_group_by_its_own_block_size(): + # Two full-attention groups with different block sizes are each clamped to + # their own prompt-block count; a sliding-window group is left untouched. + scheduler = MoRIIOConnectorScheduler.__new__(MoRIIOConnectorScheduler) + scheduler.blocks_per_sw = [0, 0, 3] + scheduler._group_block_sizes = [16, 32, 16] + + req = SimpleNamespace(num_prompt_tokens=40) + block_ids = [ + [0, 1, 2, 3, 4], # ceil(40/16)=3 -> clamp to 3 + [10, 11, 12], # ceil(40/32)=2 -> clamp to 2 + [20, 21, 22, 23, 24], # sliding-window -> untouched + ] + clamped = scheduler._clamp_to_prompt_blocks(req, block_ids) + assert clamped == [[0, 1, 2], [10, 11], [20, 21, 22, 23, 24]] + + def _make_hybrid_kv_cache_config() -> KVCacheConfig: """One full-attention group + one sliding-window group (Gemma-like).""" full_spec = FullAttentionSpec( @@ -731,6 +987,35 @@ def _make_hybrid_kv_cache_config() -> KVCacheConfig: ) +def _make_sliding_window_only_kv_cache_config() -> KVCacheConfig: + """A uniform sliding-window model (e.g. Mistral-7B): the single group is a + SlidingWindowSpec, so HMA is not required once hybrid management is + disabled, yet the spec guard still applies.""" + sw_spec = SlidingWindowSpec( + block_size=16, + num_kv_heads=4, + head_size=64, + dtype=torch.float16, + sliding_window=32, + ) + num_blocks = 2 + page = sw_spec.page_size_bytes + return KVCacheConfig( + num_blocks=num_blocks, + kv_cache_tensors=[ + KVCacheTensor( + size=num_blocks * page, + layers=["sw0"], + layer_stride=num_blocks * page, + block_stride=page, + ) + ], + kv_cache_groups=[ + KVCacheGroupSpec(layer_names=["sw0"], kv_cache_spec=sw_spec), + ], + ) + + def _read_scheduler( kv_cache_config: KVCacheConfig, disable_hma: bool = False ) -> MoRIIOConnectorScheduler: @@ -820,15 +1105,6 @@ def test_non_sliding_window_hybrid_is_rejected(): _read_scheduler(config) -def test_token_count_basis_uses_full_attention_group(): - """Chunked-prefill token counting must use an unclipped full-attention - group (blocks_per_sw == 0), not a clipped sliding-window group.""" - scheduler = _read_scheduler(_make_hybrid_kv_cache_config()) - # Group 0 is the full-attention group - assert scheduler._full_attn_group_idx == 0 - assert scheduler._full_attn_block_size == 16 - - def test_get_exchange_clipped_blocks_clips_only_sw_group(): """get_exchange_clipped_blocks keeps the full attn group intact and clips the sliding-window group to its window tail.""" @@ -897,6 +1173,71 @@ def test_single_group_path_unchanged(): assert scheduler.get_exchange_clipped_blocks(blocks) == blocks +class _FakeSpecConfig(SimpleNamespace): + """Minimal speculative_config stand-in. Any drafter-capability probe + (use_dflash/use_eagle/use_dspark/uses_draft_model/...) defaults to False so + VllmConfig.num_lookahead_tokens resolves cleanly as upstream adds methods.""" + + def __getattr__(self, name: str): + if name.startswith(("use_", "uses_")): + return lambda *a, **k: False + raise AttributeError(name) + + +def _fake_speculative_config(num_speculative_tokens: int = 1) -> SimpleNamespace: + return _FakeSpecConfig(num_speculative_tokens=num_speculative_tokens) + + +def _spec_vllm_config() -> VllmConfig: + vllm_config = create_vllm_config(role="kv_producer", read_mode=False) + vllm_config.speculative_config = _fake_speculative_config() + # HMA off => the WRITE+hybrid guard is skipped, so construction reaches the + # spec-decode guard (the only path that hits it). + vllm_config.scheduler_config.disable_hybrid_kv_cache_manager = True + return vllm_config + + +def test_spec_decode_rejected_with_sliding_window_groups(): + """MoRIIO + speculative decoding fails closed on sliding-window models, + since the prompt-block clamp only covers full-attention groups. The only + reachable path is a uniform SWA model with HMA disabled.""" + vllm_config = _spec_vllm_config() + with ( + set_current_vllm_config(vllm_config), + pytest.raises(NotImplementedError, match="speculative decoding"), + ): + MoRIIOConnector( + vllm_config, + KVConnectorRole.SCHEDULER, + _make_sliding_window_only_kv_cache_config(), + ) + + +def test_spec_decode_allowed_with_full_attention_only(): + """Full-attention-only models (e.g. GLM) do not trip the spec guard.""" + vllm_config = _spec_vllm_config() + with set_current_vllm_config(vllm_config): + connector = MoRIIOConnector( + vllm_config, + KVConnectorRole.SCHEDULER, + _make_test_kv_cache_config(), + ) + assert connector.connector_scheduler is not None + + +def test_spec_decode_allowed_with_sliding_window_in_read_mode(): + """The spec guard is WRITE-mode only; READ mode is unaffected.""" + vllm_config = create_vllm_config(role="kv_consumer", read_mode=True) + vllm_config.speculative_config = _fake_speculative_config() + with set_current_vllm_config(vllm_config): + connector = MoRIIOConnector( + vllm_config, + KVConnectorRole.SCHEDULER, + _make_hybrid_kv_cache_config(), + ) + assert connector.connector_scheduler is not None + + def test_hybrid_write_mode_rejected(): """Hybrid KV cache groups are unsupported in WRITE mode and fail closed.""" vllm_config = create_vllm_config(role="kv_producer", read_mode=False) diff --git a/tests/v1/kv_connector/unit/test_moriio_hma_scheduler.py b/tests/v1/kv_connector/unit/test_moriio_hma_scheduler.py index 28bc8bebe2..4bd32f786a 100644 --- a/tests/v1/kv_connector/unit/test_moriio_hma_scheduler.py +++ b/tests/v1/kv_connector/unit/test_moriio_hma_scheduler.py @@ -7,32 +7,20 @@ These exercise the highest-risk, otherwise-untested scheduler logic without a GPU or the ``mori`` runtime: per-group block-id splitting, the READ/WRITE ``N-1`` token accounting, P-side prompt truncation, and offset-template cache wiring. - -Like ``test_moriio_kv_layout.py`` the whole module is skipped unless it is -running on ROCm with ``mori`` installed (importing the connector pulls in -``mori``). The authoritative run happens on the MIA recipe image. """ import importlib -import importlib.util +from dataclasses import asdict from types import SimpleNamespace import pytest import torch -from vllm.platforms import current_platform from vllm.v1.core.block_pool import BlockPool +from vllm.v1.core.sched.output import SchedulerOutput from vllm.v1.core.single_type_kv_cache_manager import FullAttentionManager from vllm.v1.kv_cache_interface import FullAttentionSpec, UniformTypeKVCacheSpecs -mori_available = importlib.util.find_spec("mori") is not None - -if not (current_platform.is_rocm() and mori_available): - pytest.skip( - "MoRIIOs are only available on ROCm with mori package installed", - allow_module_level=True, - ) - moriio_connector = importlib.import_module( "vllm.distributed.kv_transfer.kv_connector.v1.moriio.moriio_connector" ) @@ -525,6 +513,8 @@ def _make_read_scheduler(): request_id_to_transfer_id={}, transfer_id_to_request_id={}, _reqs_need_recv={}, + _reqs_need_save={}, + _reqs_need_send={}, _req_kv_params={}, _max_decode_tail_blocks=1, ) @@ -538,24 +528,52 @@ def _make_read_request(remote_block_ids): "transfer_id": "tx", "remote_engine_id": "prefill", "remote_block_ids": remote_block_ids, + "remote_host": "127.0.0.1", + "remote_handshake_port": 6000, + "remote_notify_port": 6001, }, ) -def test_update_state_drops_decode_recompute_tail_block(): +@pytest.mark.parametrize("external_tokens", [0, 256]) +@pytest.mark.parametrize("has_mamba", [False, True]) +@pytest.mark.parametrize("zero_ids", [None, [], [101], [101, 102, 202, 999]]) +def test_update_state_drops_decode_recompute_tail_block( + external_tokens, has_mamba, zero_ids +): + """Only full hybrid READ destinations skip zeroing; local tails still zero.""" sched = _make_read_scheduler() - request = _make_read_request([[10, 11], [91]]) + sched._has_mamba = has_mamba + remote_blocks = [[10, 11], [91]] if has_mamba else [[10, 11]] + request = _make_read_request(remote_blocks) blocks = _FakeBlocks( - all_groups=([100, 101, 102], [200, 201, 202]), + all_groups=([100, 101, 102], [200, 201, 202]) + if has_mamba + else ([100, 101, 102],), ) - sched.update_state_after_alloc(request, blocks, num_external_tokens=256) + sched.update_state_after_alloc(request, blocks, num_external_tokens=external_tokens) - assert sched._reqs_need_recv["req"][1] == [[100, 101], [202]] - assert sched._req_kv_params["req"]["remote_block_ids"] == [ - [10, 11], - [91], - ] + expected_blocks = [[100, 101] if external_tokens else []] + if has_mamba: + expected_blocks.append([202]) + assert sched._reqs_need_recv["req"][1] == expected_blocks + assert sched._req_kv_params["req"]["remote_block_ids"] == remote_blocks + + output = SchedulerOutput.make_empty() + output.new_block_ids_to_zero = None if zero_ids is None else list(zero_ids) + before = asdict(output) + meta = sched.build_connector_meta(output) + assert meta.reqs_to_recv["req"].local_block_ids == expected_blocks + if zero_ids and has_mamba and external_tokens: + before["new_block_ids_to_zero"] = zero_ids[1:] + assert asdict(output) == before + + # A later local allocation of the same pages must still be zeroed. + output.new_block_ids_to_zero = [100, 101, 102] + meta = sched.build_connector_meta(output) + assert not meta.reqs_to_recv + assert output.new_block_ids_to_zero == [100, 101, 102] def test_update_state_pairs_shorter_local_blocks_with_remote_suffix(): @@ -806,7 +824,7 @@ def test_get_num_new_matched_tokens_write_plain_keeps_all_tokens(): req = SimpleNamespace( num_prompt_tokens=10, prompt_token_ids=list(range(10)), - kv_transfer_params=None, + kv_transfer_params={"do_remote_prefill": True}, ) n, is_async = sched.get_num_new_matched_tokens(req, num_computed_tokens=2) # Pure-attention WRITE: no N-1 drop; full length minus already-computed. @@ -830,7 +848,7 @@ def test_get_num_new_matched_tokens_supports_embeds_only_prompts( num_prompt_tokens=10, prompt_token_ids=None, prompt_embeds=object(), - kv_transfer_params=None, + kv_transfer_params={"do_remote_prefill": True}, ) assert sched.get_num_new_matched_tokens(req, num_computed_tokens) == ( diff --git a/tests/v1/kv_connector/unit/test_moriio_kv_layout.py b/tests/v1/kv_connector/unit/test_moriio_kv_layout.py index 9860a7b7e1..5ec6fb5755 100644 --- a/tests/v1/kv_connector/unit/test_moriio_kv_layout.py +++ b/tests/v1/kv_connector/unit/test_moriio_kv_layout.py @@ -661,16 +661,34 @@ def test_moriio_wrapper_rejects_invalid_messages(role, payload, match): wrapper._handle_message(payload) -def test_local_block_ids_longer_than_remote_raises_value_error(): - cache = torch.empty((8, 2, 4, 2, 3), dtype=torch.bfloat16) +def test_local_block_ids_longer_than_remote_raises(): + # A longer local list is a genuine bug here and must fail loudly; the WRITE + # producer save path trims lookahead blocks before they reach this layer. + cache = torch.empty((8, 2, 4, 2, 3), dtype=torch.bfloat16) # interleaved worker = _worker({"layer": cache}, {"layer": _full_spec()}) with pytest.raises(ValueError, match="longer than remote_block_ids"): moriio_layout.compute_block_transfer_offsets( - "layer", cache, worker.layer_to_spec, [1, 3], [4], _remote_meta().num_blocks + "layer", + cache, + worker.layer_to_spec, + [1, 3, 6, 7], + [4, 5], + _remote_meta().num_blocks, ) +def test_shorter_local_block_ids_still_transfers_only_what_decode_allocated(): + # READ-mode partial transfer keeps its shorter-local behavior. + cache = torch.empty((8, 2, 4, 2, 3), dtype=torch.bfloat16) + worker = _worker({"layer": cache}, {"layer": _full_spec()}) + + short = moriio_layout.compute_block_transfer_offsets( + "layer", cache, worker.layer_to_spec, [1], [4, 5], _remote_meta().num_blocks + ) + assert len(short[0]) == 1 + + @pytest.mark.parametrize( ("local_block_ids", "remote_block_ids", "match"), [ diff --git a/tests/v1/kv_connector/unit/test_multi_connector.py b/tests/v1/kv_connector/unit/test_multi_connector.py index 67cfdc1ab0..e576e3d0d9 100644 --- a/tests/v1/kv_connector/unit/test_multi_connector.py +++ b/tests/v1/kv_connector/unit/test_multi_connector.py @@ -34,6 +34,7 @@ from vllm.distributed.kv_transfer.kv_connector.v1.nixl import ( NixlKVConnectorStats, ) from vllm.v1.kv_cache_interface import KVCacheConfig +from vllm.v1.metrics.cache_hit_source import CacheHitSource from vllm.v1.outputs import KVConnectorOutput, KVConnectorWorkerMetadata MODEL_NAME = "meta-llama/Llama-3.2-1B-Instruct" @@ -246,6 +247,18 @@ def test_multi_connector_rejects_multiple_mem_pool_contexts(mc: MultiConnector): mc.get_mem_pool_context() +def test_cache_hit_sources_delegate_to_selected_connector(mc: MultiConnector): + request = MagicMock(request_id="request") + mc._requests_to_connector[request.request_id] = 1 + sources = {CacheHitSource.DISK: 32} + mc._connectors[1].get_external_cache_hit_sources.return_value = sources + + assert mc.get_external_cache_hit_sources(request, 32) is sources + mc._connectors[1].get_external_cache_hit_sources.assert_called_once_with( + request, 32 + ) + + # Helper function to compare directories recursively def _compare_directories(dir1: Path, dir2: Path) -> bool: """Compares two directories recursively for identical content.""" diff --git a/tests/v1/kv_connector/unit/test_nixl_connector.py b/tests/v1/kv_connector/unit/test_nixl_connector.py index 07ae290219..d918c38fbf 100644 --- a/tests/v1/kv_connector/unit/test_nixl_connector.py +++ b/tests/v1/kv_connector/unit/test_nixl_connector.py @@ -14,7 +14,7 @@ from collections import defaultdict from concurrent.futures import Future from types import SimpleNamespace from typing import Any, cast -from unittest.mock import MagicMock, patch +from unittest.mock import MagicMock, call, patch import msgspec import numpy as np @@ -50,6 +50,7 @@ from vllm.distributed.kv_transfer.kv_connector.v1.nixl import ( NixlKVConnectorStats, ) from vllm.distributed.kv_transfer.kv_connector.v1.nixl.metadata import ( + HeartbeatInfo, RemoteMeta, ReqMeta, compute_nixl_compatibility_hash, @@ -143,6 +144,7 @@ class FakeNixlWrapper: def __init__(self, agent_name: str, *args, **kwargs): self._cycles_before_xfer_done = 0 self._check_xfer_state_cycles: defaultdict[int, int] = defaultdict(lambda: 0) + self.invalid_remote_agents: set[str] = set() def get_reg_descs(self, caches_data, memory_type: str) -> list: return [str(uuid.uuid4()) for _ in caches_data] @@ -163,6 +165,7 @@ class FakeNixlWrapper: return self.AGENT_METADATA def add_remote_agent(self, agent_metadata: bytes) -> str: + self.invalid_remote_agents.discard(self.REMOTE_AGENT_NAME) return self.REMOTE_AGENT_NAME def get_new_notifs(self) -> dict[str, list[bytes]]: @@ -184,6 +187,9 @@ class FakeNixlWrapper: def remove_remote_agent(self, agent: str) -> None: pass + def check_remote_metadata(self, agent: str) -> bool: + return agent not in self.invalid_remote_agents + def send_notif(self, agent_name: str, notif_msg: bytes) -> None: pass @@ -895,11 +901,16 @@ class TestNixlHandshake: vllm_config = create_vllm_config() - connector = NixlConnector( - vllm_config, KVConnectorRole.WORKER, make_kv_cache_config(block_size=16) - ) + # num_blocks=1 keeps the fake handshake's region_num_blocks consistent + # with the minimal single-block registration below (num_descs == 1); + # _fa_desc_replicated cross-checks the two. + kv_cache_config = make_kv_cache_config(block_size=16, num_blocks=1) + connector = NixlConnector(vllm_config, KVConnectorRole.WORKER, kv_cache_config) connector.connector_worker = FakeNixlConnectorWorker( - vllm_config, connector.engine_id, hand_shake_latency=0 + vllm_config, + connector.engine_id, + hand_shake_latency=0, + kv_cache_config=kv_cache_config, ) worker = connector.connector_worker @@ -1117,17 +1128,24 @@ class TestNixlHandshake: return raise TimeoutError("Took too long to complete async handshake.") + @pytest.mark.parametrize( + ("kv_cache_layout", "enable_permute_local_kv"), + # enable_permute_local_kv only covers a local LBNHC cache. + [("LBHNC", False), ("BLHNC", True)], + ) @patch( "vllm.distributed.kv_transfer.kv_connector.v1.nixl.base_worker.NixlWrapper", FakeNixlWrapper, ) def test_handshake_fails_on_kv_cache_layout_mismatch( - self, default_vllm_config, dist_init + self, default_vllm_config, dist_init, kv_cache_layout, enable_permute_local_kv ): """Verify that adding a remote agent fails if kv_cache_layout differs. This test is only relevant for heterogeneous TP. """ - vllm_config = create_vllm_config() + vllm_config = create_vllm_config( + enable_permute_local_kv=enable_permute_local_kv + ) # Mock TP world size to 2 to force heterogeneous TP when # remote_tp_size=1 @@ -1140,7 +1158,10 @@ class TestNixlHandshake: vllm_config, KVConnectorRole.WORKER, make_kv_cache_config(block_size=16) ) connector.connector_worker = FakeNixlConnectorWorker( - vllm_config, connector.engine_id, hand_shake_latency=0 + vllm_config, + connector.engine_id, + hand_shake_latency=0, + kv_cache_layout=kv_cache_layout, ) worker = connector.connector_worker @@ -2072,6 +2093,7 @@ def test_mixed_memory_local_descriptors_split_by_memory_type(): worker.kv_caches_base_addr = {"local": {0: [100, 200]}} worker._has_mamba = False worker._mixed_mem_types = True + worker._skip_dram_xfer = False worker.region_mem_types = ["DRAM", "VRAM"] worker.region_num_blocks = [2, 2] worker._desc_is_dram_by_block_size = {} @@ -2099,7 +2121,9 @@ def test_mixed_memory_local_descriptors_split_by_memory_type(): assert handle == 22 assert worker._dram_src_handles_by_block_size[worker.block_size] == 11 - assert [block[2] for block in blocks] == [0, 0, 3, 3] + # Must stay an array: P_TP > D_TP splits call .tolist() on it. + assert isinstance(blocks, np.ndarray) + assert blocks[:, 2].tolist() == [0, 0, 3, 3] memory_types = [ call.args[1] for call in worker.nixl_wrapper.get_xfer_descs.call_args_list ] @@ -2124,6 +2148,8 @@ def recv_worker(): worker._handshake_futures = {} worker._remote_agents = {} worker._engine_by_address = {} + worker._failed_remote_engines = set() + worker._invalid_remote_engines = set() worker._replicated_pcp_done_sending = set() worker._invalid_block_ids = queue.Queue() worker._pending_recv_notifs = {} @@ -2135,6 +2161,7 @@ def recv_worker(): worker.enable_permute_local_kv = False worker.enable_heterogeneous_attn_post_process = False worker.tp_rank = 0 + worker._skip_dram_xfer = False worker._log_failure = MagicMock() # type: ignore[method-assign] worker.xfer_stats = NixlKVConnectorStats() worker.nixl_wrapper = MagicMock() @@ -2192,6 +2219,66 @@ def test_mixed_memory_read_failure_does_not_notify_producer(recv_worker): worker.nixl_wrapper.send_notif.assert_not_called() +def test_hisparse_dram_rank0_only_read_keeps_device_part(recv_worker): + worker = recv_worker + worker.tp_rank = 1 + worker._skip_dram_xfer = True + worker._desc_is_dram_by_block_size = {16: np.array([True, True, False, False])} + worker._desc_pos_by_block_size = {16: np.array([0, 1, 0, 1])} + worker._dram_src_handles_by_block_size = {16: None} + worker.nixl_wrapper.make_prepped_xfer.side_effect = [102] + worker.nixl_wrapper.check_xfer_state.return_value = "DONE" + + worker._read_blocks_mixed( + request_id="request", + local_block_size_key=16, + local_device_handle=20, + local_dram_handle=None, + remote_xfer_side_handle=30, + local_block_descs_ids=np.array([0, 2]), + remote_block_descs_ids=np.array([5, 7]), + notif_agent="prefill", + notif_id=b"request:1", + ) + + (device_read,) = worker.nixl_wrapper.make_prepped_xfer.call_args_list + assert device_read.args[:2] == ("READ", 20) + np.testing.assert_array_equal(device_read.args[2], [0]) + np.testing.assert_array_equal(device_read.args[4], [7]) + + assert worker.get_finished() == (set(), {"request"}) + worker.nixl_wrapper.send_notif.assert_called_once_with( + "prefill", notif_msg=b"request:1" + ) + + +def test_hisparse_dram_rank0_only_dram_only_read_completes(recv_worker): + """With nothing left to read, the request still completes and notifies.""" + worker = recv_worker + worker.tp_rank = 1 + worker._skip_dram_xfer = True + worker._desc_is_dram_by_block_size = {16: np.array([True, True, False, False])} + worker._desc_pos_by_block_size = {16: np.array([0, 1, 0, 1])} + + worker._read_blocks_mixed( + request_id="request", + local_block_size_key=16, + local_device_handle=20, + local_dram_handle=None, + remote_xfer_side_handle=30, + local_block_descs_ids=np.array([0, 1]), + remote_block_descs_ids=np.array([5, 6]), + notif_agent="prefill", + notif_id=b"request:1", + ) + + worker.nixl_wrapper.make_prepped_xfer.assert_not_called() + assert worker.get_finished() == (set(), {"request"}) + worker.nixl_wrapper.send_notif.assert_called_once_with( + "prefill", notif_msg=b"request:1" + ) + + def element_byte_addrs(view: torch.Tensor) -> list[int]: """Absolute byte addresses of every element of a (possibly strided) view.""" offsets = torch.zeros(view.shape, dtype=torch.int64) @@ -2700,6 +2787,51 @@ def test_engine_with_inflight_transfer_is_not_evicted(default_vllm_config, dist_ assert mock_rem.call_count == 2 +@patch( + "vllm.distributed.kv_transfer.kv_connector.v1.nixl.base_worker.NixlWrapper", + FakeNixlWrapper, +) +def test_heartbeated_engine_is_not_evicted(default_vllm_config, dist_init, monkeypatch): + """Heartbeats keep a remote engine's NIXL state alive. + + Requests waiting in the D scheduler issue no reads, but their engine is + heartbeated through its remote agents. If the TTL expires anyway, the + next heartbeat evicts the engine and starts a new handshake, which loads + every remote agent again. + """ + from vllm.distributed.kv_transfer.kv_connector.v1.nixl import base_worker + + worker, engine_id = _setup_worker_with_remote_engine(engine_ttl=10.0) + nixl_wrapper = worker.nixl_wrapper + + now = {"t": 1000.0} + monkeypatch.setattr(base_worker.time, "perf_counter", lambda: now["t"]) + worker._engine_last_active[engine_id] = now["t"] + + metadata = NixlConnectorMetadata() + metadata.heartbeat_by_engine = { + engine_id: HeartbeatInfo( + req_ids={"remote-req"}, host="localhost", port=1234, tp_size=2 + ) + } + + with ( + patch.object(nixl_wrapper, "send_notif") as mock_send, + patch.object(nixl_wrapper, "remove_remote_agent") as mock_rem, + patch.object(worker, "_handshake_initiation_executor") as mock_executor, + ): + # One heartbeat every 5 s for twice the TTL, and no reads. + for _ in range(4): + now["t"] += 5.0 + worker._send_heartbeats(metadata) + + assert engine_id in worker._remote_agents + mock_rem.assert_not_called() + mock_executor.submit.assert_not_called() + # Every heartbeat reached both remote agents. + assert mock_send.call_count == 8 + + @patch( "vllm.distributed.kv_transfer.kv_connector.v1.nixl.base_worker.NixlWrapper", FakeNixlWrapper, @@ -2761,9 +2893,9 @@ class TestPeerReplacement: self.nixl.REMOTE_AGENT_NAME = engine_id future.set_result(self.worker._nixl_handshake(host, port, 1, engine_id)) - def _request(self, engine_id="new", blocks=(1,), awaiting_kvs=True): + def _request(self, engine_id="new", blocks=(1,), awaiting_kvs=True, req_id=None): metadata = NixlConnectorMetadata() - metadata.reqs_to_recv[f"{engine_id}-req"] = ReqMeta( + metadata.reqs_to_recv[req_id or f"{engine_id}-req"] = ReqMeta( local_block_ids=(blocks,), local_physical_block_ids=(blocks,), tp_size=1, @@ -2774,6 +2906,11 @@ class TestPeerReplacement: ) return metadata + def _fail_old_peer(self): + self.worker.start_load_kv(self._request("old")) + self.nixl.invalid_remote_agents.add("old") + self.transport.check_xfer_state.return_value = "ERR" + def test_cleans_old_peer_after_confirmed_handshake(self): self._connect("healthy", host="other-host") old_handle = self.worker.dst_xfer_side_handles["old"][0] @@ -2823,7 +2960,12 @@ class TestPeerReplacement: ], ids=["healthy", "failed-sibling-completes", "failed-sibling-fails"], ) - def test_waits_for_reads_before_cleanup_and_failure_reporting(self, states, failed): + @pytest.mark.parametrize("invalidated", [False, True]) + def test_waits_for_reads_before_cleanup_and_failure_reporting( + self, states, failed, invalidated + ): + if invalidated: + self.nixl.invalid_remote_agents.add("old") self.worker._recving_metadata.update(self._request("old").reqs_to_recv) self.worker._recving_transfers["old-req"] = [1, 2] if failed else [1] self.transport.check_xfer_state.side_effect = states @@ -2842,6 +2984,7 @@ class TestPeerReplacement: ) assert self.transport.release_xfer_handle.call_count == (2 if failed else 1) self.transport.remove_remote_agent.assert_called_once_with("old") + assert not self.worker._invalid_remote_engines def test_waits_for_queued_notification(self): metadata = self._request("old", blocks=()) @@ -2862,22 +3005,31 @@ class TestPeerReplacement: assert result.failed_recving == set() self.transport.remove_remote_agent.assert_called_once_with("old") - def test_waits_for_other_handshakes(self): + @pytest.mark.parametrize("invalidated", [False, True]) + def test_waits_for_other_handshakes(self, invalidated): + if invalidated: + self._fail_old_peer() + else: + self._connect() self.worker._ensure_handshake("other", "other-host", 1234, 1) - self._connect() self.worker.get_transfer_results() self.transport.remove_remote_agent.assert_not_called() + self.transport.check_remote_metadata.assert_not_called() self._connect("other", host="other-host") self.worker.get_transfer_results() self.transport.remove_remote_agent.assert_called_once_with("old") - def test_rehandshakes_queued_request_for_released_engine(self): + @pytest.mark.parametrize("invalidated", [False, True]) + def test_rehandshakes_queued_request_for_released_engine(self, invalidated): # A request queued after its engine's handshake, but drained only after # that engine was replaced and released, must not read from it. + if invalidated: + self._fail_old_peer() metadata = self._request("old", blocks=(), awaiting_kvs=False) self.worker._ready_requests.put(("old-req", metadata.reqs_to_recv["old-req"])) - self._connect() + if not invalidated: + self._connect() self.worker.get_transfer_results() self.transport.remove_remote_agent.assert_called_once_with("old") @@ -2885,6 +3037,185 @@ class TestPeerReplacement: assert "old" in self.worker._handshake_futures self.transport.send_notif.assert_not_called() + @pytest.mark.parametrize("missing", [False, True, RuntimeError("probe failed")]) + def test_recovery_requires_confirmed_missing_metadata(self, missing): + self._fail_old_peer() + # A missing rank suffices even if another rank still has metadata. + self.worker._remote_agents["old"][(0, 1)] = "old-other-rank" + self.nixl.invalid_remote_agents.clear() + if missing is True: + self.nixl.invalid_remote_agents.add("old-other-rank") + elif isinstance(missing, Exception): + self.transport.check_remote_metadata.side_effect = missing + + result = self.worker.get_transfer_results() + assert result.finished_recving == result.failed_recving == {"old-req"} + assert self.worker.get_block_ids_with_load_errors() == {1} + assert ("old" not in self.worker._remote_agents) is (missing is True) + assert self.transport.release_dlist_handle.call_count == (missing is True) + + @pytest.mark.parametrize("handshake_fails", [False, True]) + def test_recovery_rehandshakes_same_id_with_fresh_descriptors( + self, handshake_fails + ): + old_handle = self.worker.dst_xfer_side_handles["old"][0] + self._fail_old_peer() + self.worker.get_transfer_results() + self.transport.release_dlist_handle.assert_called_once_with(old_handle) + assert not self.worker._engine_by_address + with pytest.raises(KeyError): + self.worker.transfer_topo.get_engine_info("old") + self.transport.make_prepped_xfer.reset_mock() + + self.worker.start_load_kv(self._request("old", req_id="next")) + self.transport.make_prepped_xfer.assert_not_called() + if handshake_fails: + self.worker._handshake_futures["old"].set_exception( + RuntimeError("handshake failed") + ) + result = self.worker.get_transfer_results() + assert result.failed_recving == result.finished_recving == {"next"} + assert "old" not in self.worker._remote_agents + assert self.worker._ready_requests.empty() + return + + self._connect("old") + fresh_handle = self.worker.dst_xfer_side_handles["old"][0] + assert fresh_handle != old_handle + assert self.worker._engine_by_address == {("localhost", 1234): "old"} + self.transport.check_xfer_state.return_value = "DONE" + self.worker.start_load_kv(NixlConnectorMetadata()) + assert self.transport.make_prepped_xfer.call_args.args[3] == fresh_handle + result = self.worker.get_transfer_results() + assert result.finished_recving == {"next"} and not result.failed_recving + self.transport.release_dlist_handle.assert_called_once_with(old_handle) + + def test_recovery_drains_reads_and_rejects_new_ones_without_harming_other_peers( + self, + ): + self._connect("healthy", host="other-host") + self.worker._recving_metadata.update(self._request("old").reqs_to_recv) + self.worker._recving_metadata.update(self._request("healthy").reqs_to_recv) + self.worker._recving_transfers.update( + {"old-req": [10, 11], "healthy-req": [12]} + ) + self.nixl.invalid_remote_agents.add("old") + self.transport.check_xfer_state.side_effect = ( + lambda h: "ERR" if h == 10 else "PROC" + ) + old_handle = self.worker.dst_xfer_side_handles["old"][0] + + result = self.worker.get_transfer_results() + assert not result.finished_recving and not result.failed_recving + self.transport.release_dlist_handle.assert_not_called() + self.worker.start_load_kv(self._request("old", req_id="blocked")) + self.transport.make_prepped_xfer.assert_not_called() + + self.transport.check_xfer_state.side_effect = ( + lambda h: "DONE" if h == 11 else "PROC" + ) + result = self.worker.get_transfer_results() + assert ( + result.finished_recving == result.failed_recving == {"old-req", "blocked"} + ) + assert set(self.worker._remote_agents) == {"healthy"} + assert self.worker._recving_transfers == {"healthy-req": [12]} + self.transport.remove_remote_agent.assert_called_once_with("old") + calls = self.transport.mock_calls + assert calls.index(call.release_xfer_handle(11)) < calls.index( + call.release_dlist_handle(old_handle) + ) + + def test_repeated_invalidation_backs_off_before_rehandshake(self): + # A peer that is invalidated again soon after recovery (eg a persistent + # RDMA fault) must not re-handshake on every request. + self._fail_old_peer() + self.worker.get_transfer_results() + self.transport.remove_remote_agent.assert_called_once_with("old") + + self._connect("old") + self._fail_old_peer() + assert self.worker.get_transfer_results().failed_recving == {"old-req"} + assert "old" in self.worker._remote_agents + + self.transport.make_prepped_xfer.reset_mock() + self.worker.start_load_kv(self._request("old", req_id="next")) + assert self.worker.get_transfer_results().failed_recving == {"next"} + self.transport.make_prepped_xfer.assert_not_called() + assert "old" not in self.worker._handshake_futures + self.transport.remove_remote_agent.assert_called_once() + + backoff, release_at = self.worker._invalid_engine_backoff["old"] + assert backoff == 1.0 + self.worker._invalid_engine_backoff["old"] = (backoff, release_at - backoff) + self.worker.get_transfer_results() + assert self.transport.remove_remote_agent.call_count == 2 + assert "old" not in self.worker._remote_agents + + def test_recovery_waits_until_failed_handle_can_be_released(self): + self._fail_old_peer() + with patch.object( + self.transport, "release_xfer_handle", side_effect=RuntimeError("in use") + ): + result = self.worker.get_transfer_results() + assert not result.finished_recving and not result.failed_recving + assert self.worker._recving_transfers["old-req"] + self.transport.release_dlist_handle.assert_not_called() + + result = self.worker.get_transfer_results() + assert result.finished_recving == result.failed_recving == {"old-req"} + self.transport.remove_remote_agent.assert_called_once_with("old") + + @pytest.mark.parametrize("expired", [False, True]) + def test_success_and_expiry_do_not_probe_metadata(self, expired): + metadata = self._request("old") + if expired: + metadata.reqs_to_recv["old-req"].remote.blocks_expiry_time = 0.0 + self.worker._bidirectional_kv_xfer_enabled = True + with ( + patch.object(self.worker.xfer_stats, "record_failed_transfer") as failure, + patch.object(self.worker.xfer_stats, "record_kv_expired_req") as expiry, + ): + self.worker.start_load_kv(metadata) + result = self.worker.get_transfer_results() + assert result.finished_recving == {"old-req"} + assert result.failed_recving == ({"old-req"} if expired else set()) + assert expiry.call_count == int(expired) + failure.assert_not_called() + self.transport.check_remote_metadata.assert_not_called() + self.transport.remove_remote_agent.assert_not_called() + + @pytest.mark.parametrize("awaiting_kvs", [False, True]) + def test_invalid_peer_empty_recv_is_reported_only_when_awaited(self, awaiting_kvs): + self._fail_old_peer() + self.worker._recving_transfers["old-req"].append(11) + self.transport.check_xfer_state.side_effect = ( + lambda h: "PROC" if h == 11 else "ERR" + ) + self.worker.get_transfer_results() + self.transport.make_prepped_xfer.reset_mock() + + self.worker.start_load_kv( + self._request("old", blocks=(), awaiting_kvs=awaiting_kvs, req_id="empty") + ) + result = self.worker.get_transfer_results() + expected = {"empty"} if awaiting_kvs else set() + assert result.finished_recving == result.failed_recving == expected + self.transport.make_prepped_xfer.assert_not_called() + self.transport.send_notif.assert_not_called() + self.transport.remove_remote_agent.assert_not_called() + + def test_eviction_clears_pending_recovery(self): + self.worker._failed_remote_engines.add("old") + self.worker._invalid_remote_engines.add("old") + self.worker._engine_ttl = 1 + self.worker._engine_last_active["old"] = time.perf_counter() - 10 + self.worker._evict_stale_engines() + self.worker.get_transfer_results() + self.transport.remove_remote_agent.assert_called_once_with("old") + assert not self.worker._failed_remote_engines + assert not self.worker._invalid_remote_engines + def test_transfer_topology_unregister(): """TransferTopology.unregister_remote_engine removes the engine.""" diff --git a/tests/v1/kv_connector/unit/test_nixl_connector_hma.py b/tests/v1/kv_connector/unit/test_nixl_connector_hma.py index 2a1ca5a877..c665ce0c5b 100644 --- a/tests/v1/kv_connector/unit/test_nixl_connector_hma.py +++ b/tests/v1/kv_connector/unit/test_nixl_connector_hma.py @@ -60,6 +60,8 @@ def region_pull_worker(): worker._transfer_layer_group_ids = () worker._bidirectional_kv_xfer_enabled = False worker._recving_transfers = {} + worker._failed_remote_engines = set() + worker._invalid_remote_engines = set() worker.use_mla, worker._has_mamba = True, False worker.dcp_size = 1 worker.dcp_rank = 0 @@ -265,6 +267,7 @@ def test_region_pull_completion_zeros_only_own_padding(region_pull_worker): worker.region_num_blocks = [4, 4] worker.block_len_per_layer = [8, 8] worker.block_stride_per_layer = [24, 24] + worker._skip_dram_xfer = False worker._recving_metadata = { "request": ReqMeta( local_block_ids=[[1, 2], [0, 1]], @@ -690,6 +693,8 @@ def test_read_blocks_for_req_expands_remote_ids( worker = object.__new__(NixlConnectorWorker) worker._physical_blocks_per_logical_kv_block = local_physical_per_logical + worker._invalid_remote_engines = set() + worker._remote_agents = {"remote-engine": {}} worker._engine_last_active = {} worker._recving_transfers = {} worker._bidirectional_kv_xfer_enabled = False @@ -1399,46 +1404,66 @@ def test_map_block_ids_for_block_size_ratio_hybrid(): @pytest.mark.cpu_test -def test_post_process_zeroes_untransferred_tail(): - """The untransferred sub-blocks of the last local block are zeroed on - receive; mamba state caches are untouched by the attention permute.""" +@pytest.mark.parametrize( + ("kv_cache_layout", "enable_permute_local_kv"), + [("LBHNC", False), ("LBNHC", False), ("LBNHC", True)], +) +def test_post_process_zeroes_untransferred_tail( + kv_cache_layout, enable_permute_local_kv +): + """Received remote sub-blocks are regrouped per head and the untransferred + sub-blocks of the last local block are zeroed on receive, once per block + although two attention groups alias the tensor.""" from unittest.mock import MagicMock from vllm.distributed.kv_transfer.kv_connector.v1.nixl.worker import ( NixlConnectorWorker, ) - from vllm.v1.kv_cache_interface import FullAttentionSpec, MambaSpec ratio = 4 block_tokens = 8 # 2 tokens per remote sub-block + num_kv_heads = 2 # fewer than ratio worker = MagicMock(spec=NixlConnectorWorker) - worker._group_spec_types = (FullAttentionSpec, MambaSpec) worker.transfer_topo = MagicMock() worker.device_type = "cpu" - worker.enable_permute_local_kv = False - attn_cache = torch.ones(6, block_tokens, 2, 4) - mamba_cache = torch.ones(6, 16) - worker.device_kv_caches = {"attn.0": attn_cache, "mamba.0": mamba_cache} + worker.enable_permute_local_kv = enable_permute_local_kv + worker.kv_cache_layout = kv_cache_layout + # Attention caches are [B, H, N, C]; distinct values per head and token. + expected = torch.arange(6 * num_kv_heads * block_tokens * 4).view( + 6, num_kv_heads, block_tokens, 4 + ) + # Blocks [2, 3] as received: `ratio` head-major remote sub-blocks each. + received = ( + expected[2:4] + .unflatten(2, (ratio, -1)) + .transpose(1, 2) + .reshape(2, num_kv_heads, block_tokens, 4) + ) + if kv_cache_layout == "LBNHC": + attn_cache = expected.transpose(1, 2).contiguous().transpose(1, 2) + if enable_permute_local_kv: + # The remote is LBHNC: its bytes land in token-major memory. + attn_cache.transpose(1, 2)[2:4] = received.view(2, block_tokens, -1, 4) + # Otherwise token-major blocks receive the sub-blocks in token order. + else: + attn_cache = expected.clone() + attn_cache[2:4] = received + worker.device_kv_caches = {"attn.0": attn_cache, "swa.0": attn_cache} fa_group = MagicMock(layer_names=["attn.0"]) - ssm_group = MagicMock(layer_names=["mamba.0"]) - worker.kv_cache_config = MagicMock(transfer_groups=[fa_group, ssm_group]) - # The cached property filters mamba layers out of the permuted caches. - attn_caches = NixlConnectorWorker._attention_kv_caches.func(worker) - assert len(attn_caches) == 1 and attn_caches[0] is attn_cache - worker._attention_kv_caches = attn_caches + swa_group = MagicMock(layer_names=["swa.0"]) + worker.kv_cache_config = MagicMock(transfer_groups=[fa_group, swa_group]) _bind_worker_method(worker, "post_process_device_kv_on_receive") - # Request occupies blocks [2, 3]; only 6 of 8 sub-blocks were received. - worker.post_process_device_kv_on_receive(ratio, [([2, 3], 6)]) + # Group 0 request in blocks [2, 3]; only 6 of 8 sub-blocks were received. + worker.post_process_device_kv_on_receive(ratio, [(0, [2, 3], 6)]) # Block 2 fully covered; block 3 covered for 2 sub-blocks (4 tokens). - assert torch.all(attn_cache[2] == 1) - assert torch.all(attn_cache[3, :4] == 1) - assert torch.all(attn_cache[3, 4:] == 0) - # Untouched blocks and the mamba cache keep their content. - assert torch.all(attn_cache[4] == 1) - assert torch.all(mamba_cache == 1) + assert torch.equal(attn_cache[2], expected[2]) + assert torch.equal(attn_cache[3, :, :4], expected[3, :, :4]) + assert torch.all(attn_cache[3, :, 4:] == 0) + # Untouched blocks keep their content. + assert torch.equal(attn_cache[4], expected[4]) @pytest.mark.cpu_test @@ -1555,17 +1580,22 @@ def test_failed_load_rezeroes_unwritten_skipped_blocks(): @pytest.mark.cpu_test @pytest.mark.parametrize( - "has_mamba,is_hma_required,expected_count", + "has_mamba,is_hma_required,bounded_replay,expected_count", [ - (True, True, 9), - (False, False, 10), - (False, True, 10), + (True, True, False, 9), + (False, False, False, 10), + (False, True, False, 10), + (False, True, True, 9), ], - ids=["mamba", "fa_only", "swa_only"], + ids=["mamba", "fa_only", "swa_only", "swa_bounded_replay"], ) -def test_mamba_n1_d_side(has_mamba, is_hma_required, expected_count): - """D-side: Mamba gets N-1 matched tokens, non-Mamba gets N.""" - sched = make_nixl_scheduler(has_mamba=has_mamba, is_hma_required=is_hma_required) +def test_mamba_n1_d_side(has_mamba, is_hma_required, bounded_replay, expected_count): + """D-side: Mamba and SWA bounded replay get N-1 matched tokens, others N.""" + sched = make_nixl_scheduler( + has_mamba=has_mamba, + is_hma_required=is_hma_required, + bounded_replay=bounded_replay, + ) req = create_request(num_tokens=10, do_remote_prefill=True) count, is_async = sched.get_num_new_matched_tokens(req, num_computed_tokens=0) @@ -1634,6 +1664,20 @@ def test_mamba_n1_p_side_truncation(): fa_sched.on_new_request(fa_req) assert len(fa_req.prompt_token_ids) == fa_original + # SWA bounded replay: the window is not replayed after the load, so the + # prefiller stops short of the last token too. + swa_sched = make_nixl_scheduler(is_hma_required=True, bounded_replay=True) + swa_req = create_request(num_tokens=10, do_remote_decode=True) + swa_sched.on_new_request(swa_req) + assert len(swa_req.prompt_token_ids) == 9 + + # A request that skips reading the prefix cache (prompt logprobs) is not + # cut: the decoder loads nothing and recomputes its whole prompt. + logprobs_req = create_request(num_tokens=10, do_remote_decode=True) + logprobs_req.sampling_params.skip_reading_prefix_cache = True + swa_sched.on_new_request(logprobs_req) + assert len(logprobs_req.prompt_token_ids) == 10 + @pytest.mark.cpu_test @pytest.mark.parametrize( diff --git a/tests/v1/kv_connector/unit/test_nixl_desc_geometry.py b/tests/v1/kv_connector/unit/test_nixl_desc_geometry.py index 029ffafd3c..9f6bc56dee 100644 --- a/tests/v1/kv_connector/unit/test_nixl_desc_geometry.py +++ b/tests/v1/kv_connector/unit/test_nixl_desc_geometry.py @@ -1245,8 +1245,12 @@ def test_mismatched_mla_kernel_page_rejected_for_mla_hybrid(): worker.add_remote_agent(meta_r, remote_tp_rank=0, remote_tp_size=2) -def _make_csa_linear_ple_worker(scratch_aliases: str = "compressed"): - """``scratch_aliases`` selects which pages the compressor ring overlays.""" +def _make_csa_linear_ple_worker( + scratch_aliases: str = "compressed", ple_page_size: int = 256 +): + """``scratch_aliases`` selects which pages the compressor ring overlays; + ``ple_page_size`` widens the PLE page beyond the page it shares a region + with (block-outer layout: both start at byte 0 of the block).""" from unittest.mock import MagicMock from vllm.config import set_current_vllm_config @@ -1305,7 +1309,7 @@ def _make_csa_linear_ple_worker(scratch_aliases: str = "compressed"): block_size=1, shapes=((6, 3),), dtypes=(torch.float16,), - page_size_padded=256, + page_size_padded=ple_page_size, mamba_type=MambaAttentionBackendEnum.SHORT_CONV, tp_replicated=True, ) @@ -1339,17 +1343,22 @@ def _make_csa_linear_ple_worker(scratch_aliases: str = "compressed"): ("compressed.0",), ("compressed.1",), ) - region_size = 512 page_size = 256 + # Region 0 hosts the PLE page: its blocks are as wide as the widest page. + block_strides = [max(page_size, ple_page_size)] + [page_size] * ( + len(tensor_regions) - 1 + ) + region_sizes = [2 * stride for stride in block_strides] + region_offsets = [sum(region_sizes[:index]) for index in range(len(region_sizes))] kv_cache_config = KVCacheConfig( num_blocks=2, kv_cache_tensors=[ KVCacheTensor( - size=len(tensor_regions) * region_size, + size=sum(region_sizes), layers=[layer_name], - layer_stride=region_size, - block_stride=page_size, - offset=region_index * region_size, + layer_stride=region_sizes[region_index], + block_stride=block_strides[region_index], + offset=region_offsets[region_index], ) for region_index, layer_names in enumerate(tensor_regions) for layer_name in layer_names @@ -1386,10 +1395,14 @@ def _make_csa_linear_ple_worker(scratch_aliases: str = "compressed"): set_current_vllm_config(vllm_config), ): worker = NixlConnectorWorker(vllm_config, "local-engine", kv_cache_config) - tensors = [torch.zeros((2, 256), dtype=torch.uint8) for _ in range(4)] + tensors = [ + torch.zeros((2, stride), dtype=torch.uint8) for stride in block_strides + ] worker.register_kv_caches( { - layer_name: tensors[region_index] + layer_name: tensors[region_index][ + :, : (ple_page_size if layer_name == "mamba.ple" else page_size) + ] for region_index, layer_names in enumerate(tensor_regions) for layer_name in layer_names } @@ -1523,6 +1536,7 @@ def test_csa_linear_remote_ple_is_copied_whole(): kv_cache_layout="HND", block_size=4, ssm_sizes=(24, 32), + ple_block_len=256, attn_backend_name="test", physical_blocks_per_logical_kv_block=1, ) @@ -1535,7 +1549,7 @@ def test_csa_linear_remote_ple_is_copied_whole(): assert descriptors[-2:, 0].tolist() == [0x10000, 0x10100] assert descriptors[-2:, 1].tolist() == [256, 256] - metadata.block_lens[0] = 128 + metadata.ple_block_len = 128 with pytest.raises(ValueError, match="PLE pages require identical"): worker._build_mamba_remote( metadata, @@ -1544,6 +1558,30 @@ def test_csa_linear_remote_ple_is_copied_whole(): ) +@pytest.mark.cpu_test +def test_csa_linear_ple_page_wider_than_the_shared_region_page(): + """The PLE page is discovered in the region of the main KV page registered + first (both start at byte 0 of the block); its descriptors must still cover + the whole PLE page, not the region's block_len.""" + worker = _make_csa_linear_ple_worker(ple_page_size=384) + + assert worker._ple_region_index == 0 + assert worker.block_len_per_layer[0] == 256 + assert worker.block_stride_per_layer[0] == 384 + assert worker._ple_block_len == 384 + + bases = [0x10000, 0x20000, 0x30000, 0x40000] + mamba = worker._build_mamba_local(bases) + assert mamba[-2:, 0].tolist() == [0x10000, 0x10000 + 384] + assert mamba[-2:, 1].tolist() == [384, 384] + + metadata = msgspec.msgpack.decode( + worker.xfer_handshake_metadata.agent_metadata_bytes, type=NixlAgentMetadata + ) + assert metadata.ple_block_len == 384 + assert metadata.block_lens[0] == 256 + + def _make_ring_worker(): """Paged MLA group plus a per-layer ring group, no Mamba: the shape of a model whose sliding-window KV lives in per-request rings.""" diff --git a/tests/v1/kv_connector/unit/test_nixl_push_connector.py b/tests/v1/kv_connector/unit/test_nixl_push_connector.py index 47779c3611..a36c7e5b2a 100644 --- a/tests/v1/kv_connector/unit/test_nixl_push_connector.py +++ b/tests/v1/kv_connector/unit/test_nixl_push_connector.py @@ -379,6 +379,8 @@ class _StubWriterWorker(NixlPushConnectorWorker): w.pp_size = 1 w.engine_id = "test-decode-engine" w._remote_agents = {} + w._failed_remote_engines = set() + w._invalid_remote_engines = set() w._handshake_lock = threading.RLock() w._physical_blocks_per_logical_kv_block = 1 w._uses_region_group_mapping = False diff --git a/tests/v1/kv_connector/unit/test_nixl_simple_cpu_offload.py b/tests/v1/kv_connector/unit/test_nixl_simple_cpu_offload.py index bdab91a21f..5ae235c24d 100644 --- a/tests/v1/kv_connector/unit/test_nixl_simple_cpu_offload.py +++ b/tests/v1/kv_connector/unit/test_nixl_simple_cpu_offload.py @@ -36,6 +36,7 @@ from vllm.v1.kv_cache_interface import ( KVCacheTensor, SlidingWindowSpec, ) +from vllm.v1.metrics.cache_hit_source import CacheHitSource from vllm.v1.outputs import KVConnectorOutput from vllm.v1.request import Request from vllm.v1.simple_kv_offload.metadata import ( @@ -232,6 +233,9 @@ def test_nixl_wins_load_over_cpu_offload(): assert not sched_out.has_sync_kv_loads assert mc._requests_to_connector[request.request_id] == 0 + assert mc.get_external_cache_hit_sources(request, 2 * BLOCK_SIZE) == { + CacheHitSource.P2P: 2 * BLOCK_SIZE + } meta = sched_out.kv_connector_metadata assert isinstance(meta, MultiKVConnectorMetadata) @@ -302,6 +306,9 @@ def test_cpu_offload_wins_when_nixl_has_no_match(): assert hit_tokens is not None and hit_tokens > 0 assert mc._requests_to_connector[req2.request_id] == 1 assert is_async is True + assert mc.get_external_cache_hit_sources(req2, hit_tokens) == { + CacheHitSource.HOST: hit_tokens + } @pytest.mark.parametrize("swa_enabled", [False, True], ids=["fa_only", "fa_sw"]) diff --git a/tests/v1/kv_connector/unit/utils.py b/tests/v1/kv_connector/unit/utils.py index 6f27212281..d738e18395 100644 --- a/tests/v1/kv_connector/unit/utils.py +++ b/tests/v1/kv_connector/unit/utils.py @@ -6,6 +6,7 @@ from collections.abc import Callable from dataclasses import dataclass from itertools import chain, count from typing import Any, Literal +from unittest.mock import Mock import torch @@ -42,6 +43,7 @@ from vllm.v1.kv_cache_interface import ( MambaSpec, SlidingWindowSpec, ) +from vllm.v1.metrics.cache_hit_source import CacheHitSource from vllm.v1.outputs import KVConnectorOutput, ModelRunnerOutput from vllm.v1.request import Request from vllm.v1.structured_output import StructuredOutputManager @@ -49,6 +51,14 @@ from vllm.v1.structured_output import StructuredOutputManager EOS_TOKEN_ID = 50256 +def create_mock_connector() -> Mock: + connector = Mock() + connector.get_external_cache_hit_sources.side_effect = lambda request, num_tokens: ( + {CacheHitSource.EXTERNAL_UNSPECIFIED: num_tokens} if num_tokens else {} + ) + return connector + + def assert_scheduler_empty(scheduler: Scheduler): """Confirm the scheduler is "empty" - i.e. no leaks.""" # Scheduler Metadata. @@ -522,6 +532,7 @@ def make_kv_cache_config( def make_nixl_scheduler( has_mamba: bool = False, is_hma_required: bool = False, + bounded_replay: bool = False, heartbeat: bool = False, kv_lease_duration: int = 30, ): @@ -538,6 +549,7 @@ def make_nixl_scheduler( sched = object.__new__(NixlConnectorScheduler) sched._has_mamba = has_mamba + sched._bounded_replay = bounded_replay sched._is_hma_required = is_hma_required sched.kv_cache_config = make_kv_cache_config( block_size=16, @@ -605,6 +617,7 @@ def make_nixl_push_scheduler( sched.side_channel_port = 5600 sched.is_bidirectional_kv_xfer_enabled = is_bidirectional_kv_xfer_enabled sched._has_mamba = has_mamba + sched._bounded_replay = False sched.kv_cache_config = make_kv_cache_config( block_size=16, mamba_enabled=has_mamba, diff --git a/tests/v1/kv_offload/cpu/test_shared_offload_region.py b/tests/v1/kv_offload/cpu/test_shared_offload_region.py index f84e1bd729..b7259e6fd6 100644 --- a/tests/v1/kv_offload/cpu/test_shared_offload_region.py +++ b/tests/v1/kv_offload/cpu/test_shared_offload_region.py @@ -772,17 +772,14 @@ def test_cleanup_idempotent(iid): def test_cleanup_unregisters_every_pinned_chunk(iid, monkeypatch): """cleanup() must release every chunk from a split host registration.""" r = _make_region(iid) - cudart = MagicMock() - cudart.cudaHostUnregister.return_value = MagicMock(value=0) - monkeypatch.setattr(region_module, "current_platform", MagicMock()) - region_module.current_platform.is_cuda_alike.return_value = True - monkeypatch.setattr(region_module.torch.cuda, "cudart", lambda: cudart) + unregister = MagicMock() + monkeypatch.setattr(region_module, "host_unregister", unregister) r.pinned_addresses = [0x100000, 0x200000, 0x300000] r.is_pinned = True r.cleanup() - assert cudart.cudaHostUnregister.call_args_list == [ + assert unregister.call_args_list == [ call(0x300000), call(0x200000), call(0x100000), @@ -791,40 +788,35 @@ def test_cleanup_unregisters_every_pinned_chunk(iid, monkeypatch): @pytest.fixture def host_register(monkeypatch): - """Pretend to run on CUDA with host registrations capped at seven pages. - - Registration goes to a mock CudaRTLibrary and the region's cleanup to a - separate mock torch runtime. Both stay installed through the cleanup that - ``_region`` runs on exit. + """Mock the cross-platform host_register/host_unregister helpers with + registrations capped at seven pages. The same unregister mock backs both + pin_mmap_region's rollback path and the region's cleanup() on exit from + ``_region``. """ - cudart = MagicMock() - cudart.cudaHostRegister.return_value = 0 - cudart.cudaHostUnregister.return_value = 0 - cudart.cudaGetLastError.return_value = 2 - torch_cudart = MagicMock() - torch_cudart.cudaHostUnregister.return_value = MagicMock(value=0) - monkeypatch.setattr(gpu_worker.current_platform, "is_cuda_alike", lambda: True) + register = MagicMock(return_value=True) + unregister = MagicMock() monkeypatch.setattr(gpu_worker, "MAX_HOST_REGISTER_CHUNK_BYTES", 7 * PAGE_SIZE) - monkeypatch.setattr(gpu_worker, "CudaRTLibrary", lambda: cudart) - monkeypatch.setattr(region_module.torch.cuda, "cudart", lambda: torch_cudart) - return cudart, torch_cudart + monkeypatch.setattr(gpu_worker, "host_register", register) + monkeypatch.setattr(gpu_worker, "host_unregister", unregister) + monkeypatch.setattr(region_module, "host_unregister", unregister) + return register, unregister def test_pin_mmap_region_registers_row_aligned_chunks(iid, host_register): """Chunks end on row boundaries: 3-page rows under a 7-page cap register as 6 + 6 + 3 pages, where a raw byte cap would give 7 + 7 + 1.""" - cudart, torch_cudart = host_register + register, unregister = host_register with _region(iid, num_chunks=5, cpu_page_size=3 * PAGE_SIZE) as region: gpu_worker.pin_mmap_region(region) base = region._base.data_ptr() - assert cudart.mock_calls == [ - call.cudaHostRegister(base, 6 * PAGE_SIZE), - call.cudaHostRegister(base + 6 * PAGE_SIZE, 6 * PAGE_SIZE), - call.cudaHostRegister(base + 12 * PAGE_SIZE, 3 * PAGE_SIZE), + assert register.call_args_list == [ + call(base, 6 * PAGE_SIZE), + call(base + 6 * PAGE_SIZE, 6 * PAGE_SIZE), + call(base + 12 * PAGE_SIZE, 3 * PAGE_SIZE), ] assert region.is_pinned - assert torch_cudart.cudaHostUnregister.call_args_list == [ + assert unregister.call_args_list == [ call(base + 12 * PAGE_SIZE), call(base + 6 * PAGE_SIZE), call(base), @@ -833,10 +825,10 @@ def test_pin_mmap_region_registers_row_aligned_chunks(iid, host_register): @pytest.mark.parametrize("fail_at", [0, 1, 2]) def test_pin_mmap_region_failure_leaves_region_pageable(iid, host_register, fail_at): - """A failed chunk drains its pending error and unregisters the chunks - before it on the same handle, so the region is pinned whole or not at all.""" - cudart, torch_cudart = host_register - cudart.cudaHostRegister.side_effect = [0] * fail_at + [2] + """A failed chunk unregisters the chunks before it on the same range, so + the region is pinned whole or not at all.""" + register, unregister = host_register + register.side_effect = [True] * fail_at + [False] with _region(iid, num_chunks=5, cpu_page_size=3 * PAGE_SIZE) as region: gpu_worker.pin_mmap_region(region) base = region._base.data_ptr() @@ -845,25 +837,22 @@ def test_pin_mmap_region_failure_leaves_region_pageable(iid, host_register, fail registered = [base + i * 6 * PAGE_SIZE for i in range(fail_at)] failed_size = min(6, 15 - 6 * fail_at) * PAGE_SIZE - assert cudart.mock_calls == [ - *(call.cudaHostRegister(address, 6 * PAGE_SIZE) for address in registered), - call.cudaHostRegister(base + fail_at * 6 * PAGE_SIZE, failed_size), - call.cudaGetLastError(), - *(call.cudaHostUnregister(address) for address in reversed(registered)), + assert register.call_args_list == [ + *(call(address, 6 * PAGE_SIZE) for address in registered), + call(base + fail_at * 6 * PAGE_SIZE, failed_size), + ] + assert unregister.call_args_list == [ + call(address) for address in reversed(registered) ] - torch_cudart.cudaHostUnregister.assert_not_called() -def test_pin_mmap_region_without_cuda_runtime_stays_pageable( - iid, host_register, monkeypatch +def test_pin_mmap_region_when_host_register_unsupported_stays_pageable( + iid, monkeypatch ): - """A CUDA runtime that cannot be loaded leaves the region pageable instead - of failing worker startup.""" - - def missing_runtime(): - raise AssertionError("libcudart is not loaded in the current process") - - monkeypatch.setattr(gpu_worker, "CudaRTLibrary", missing_runtime) + """A platform without host registration support (host_register() + returning False) leaves the region pageable instead of failing worker + startup.""" + monkeypatch.setattr(gpu_worker, "host_register", lambda ptr, num_bytes: False) with _region(iid) as region: gpu_worker.pin_mmap_region(region) assert not region.is_pinned diff --git a/tests/v1/kv_offload/test_factory.py b/tests/v1/kv_offload/test_factory.py index 01ef756cae..84b9da36e0 100644 --- a/tests/v1/kv_offload/test_factory.py +++ b/tests/v1/kv_offload/test_factory.py @@ -307,6 +307,7 @@ def test_cpu_spec_replicated_disabled_without_shared_region( # those private buffers empty and corrupting subsequent loads. import vllm.v1.kv_offload.cpu.spec as cpu_spec_module + monkeypatch.setattr(cpu_spec_module.current_platform, "is_xpu", lambda: False) monkeypatch.setattr(cpu_spec_module.current_platform, "is_cuda_alike", lambda: rocm) monkeypatch.setattr(cpu_spec_module.current_platform, "is_rocm", lambda: rocm) worker_kv_bytes_per_block = SharedOffloadRegion.BLOCK_SIZE_ALIGNMENT @@ -341,6 +342,7 @@ def test_cpu_spec_replicated_layout_truth_matrix( # actually allocates on the shared region. import vllm.v1.kv_offload.cpu.spec as cpu_spec_module + monkeypatch.setattr(cpu_spec_module.current_platform, "is_xpu", lambda: False) monkeypatch.setattr( cpu_spec_module.current_platform, "is_cuda_alike", lambda: cuda_alike ) @@ -417,6 +419,7 @@ def test_cpu_spec_create_worker_uses_tensor_path_without_shared_region( worker_calls.append(kwargs) return MagicMock() + monkeypatch.setattr(cpu_spec_module.current_platform, "is_xpu", lambda: False) monkeypatch.setattr(cpu_spec_module.current_platform, "is_cuda_alike", lambda: rocm) monkeypatch.setattr(cpu_spec_module.current_platform, "is_rocm", lambda: rocm) monkeypatch.setattr(cpu_spec_module, "SharedOffloadRegion", fake_region_ctor) diff --git a/tests/v1/kv_offload/tiering/test_factory.py b/tests/v1/kv_offload/tiering/test_factory.py index f91f54bb0d..262af6cb83 100644 --- a/tests/v1/kv_offload/tiering/test_factory.py +++ b/tests/v1/kv_offload/tiering/test_factory.py @@ -17,6 +17,7 @@ import pytest from vllm.v1.kv_offload.tiering.base import SecondaryTierManager from vllm.v1.kv_offload.tiering.example.manager import ExampleSecondaryTierManager from vllm.v1.kv_offload.tiering.factory import SecondaryTierFactory +from vllm.v1.metrics.cache_hit_source import CacheHitSource # --------------------------------------------------------------------------- # Fixtures @@ -43,12 +44,22 @@ def _make_mock_args(): def test_pre_registered_tiers_can_be_imported(): """CI sentinel: registered tiers import and yield SecondaryTierManager.""" + # Derived from each tier's ``medium``; P2P has none and sets its own. + expected_sources = { + "example": CacheHitSource.HOST, + "fs": CacheHitSource.DISK, + "obj": CacheHitSource.DISK, + "p2p": CacheHitSource.P2P, + } for tier_type in SecondaryTierFactory._registry: # KVCR is an optional external dependency and may not be installed. if tier_type == "kvcr" and find_spec("kvcr") is None: continue cls = SecondaryTierFactory._registry[tier_type]() assert issubclass(cls, SecondaryTierManager) + assert cls.cache_hit_source == expected_sources.get( + tier_type, CacheHitSource.EXTERNAL_UNSPECIFIED + ) def test_example_tier_registered(): @@ -95,7 +106,8 @@ def test_create_multiple_tiers(): assert all(isinstance(tier, ExampleSecondaryTierManager) for tier in tiers) -def test_register_new_tier_type(): +@pytest.mark.parametrize("tier_type", ["custom_tier", "cpu", "nvme"]) +def test_register_new_tier_type(tier_type): """Verify that new tier types can be registered and created. This is how external projects add custom secondary tiers @@ -103,20 +115,22 @@ def test_register_new_tier_type(): """ # Register a new tier type (reuse example manager for simplicity) SecondaryTierFactory.register_tier( - "custom_tier", + tier_type, "vllm.v1.kv_offload.tiering.example.manager", "ExampleSecondaryTierManager", ) primary_kv_view, offloading_spec = _make_mock_args() tier = SecondaryTierFactory.create_secondary_tier( - {"type": "custom_tier", "custom_param": 99}, + {"type": tier_type, "custom_param": 99}, primary_kv_view, offloading_spec, ) - assert tier.tier_type == "custom_tier" + assert tier.tier_type == tier_type assert isinstance(tier, ExampleSecondaryTierManager) + # Attribution follows the tier's medium, never its registration name. + assert tier.cache_hit_source is CacheHitSource.HOST # --------------------------------------------------------------------------- diff --git a/tests/v1/kv_offload/tiering/test_kvcr_tier.py b/tests/v1/kv_offload/tiering/test_kvcr_tier.py index 5cbc85230f..0907293fd8 100644 --- a/tests/v1/kv_offload/tiering/test_kvcr_tier.py +++ b/tests/v1/kv_offload/tiering/test_kvcr_tier.py @@ -18,7 +18,7 @@ from kvcr.types import ( BlockKey, CacheTier, InventoryEvent, - MemDescriptor, + MemoryRef, OpEntryResult, OpEntryStatus, OpHandle, @@ -36,7 +36,9 @@ from vllm.v1.kv_offload.base import ( ReqContext, make_offload_key, ) +from vllm.v1.kv_offload.tiering.backpressure import EMABackpressureDetector from vllm.v1.kv_offload.tiering.base import JobResult, TransferJob +from vllm.v1.kv_offload.tiering.factory import SecondaryTierFactory from vllm.v1.kv_offload.tiering.kvcr import manager as kvcr_manager from vllm.v1.kv_offload.tiering.kvcr.manager import KVCRSecondaryTierManager @@ -78,11 +80,9 @@ class RecordingKVCR: self.submit_hint_calls: list[tuple[tuple[Any, ...], dict[str, Any]]] = [] self.discard_hint_calls: list[str] = [] self.deliver_calls: list[ - tuple[OpHandle, dict[BlockKey, list[MemDescriptor]], str | None] - ] = [] - self.deposit_calls: list[ - tuple[OpHandle, dict[BlockKey, list[MemDescriptor]]] + tuple[OpHandle, dict[BlockKey, list[MemoryRef]], str | None] ] = [] + self.deposit_calls: list[tuple[OpHandle, dict[BlockKey, list[MemoryRef]]]] = [] self.completed: list[tuple[OpHandle, dict[BlockKey, OpEntryResult]]] = [] self._next_op_handle = 1 @@ -107,7 +107,7 @@ class RecordingKVCR: def deliver( self, - blocks: Mapping[BlockKey, list[MemDescriptor]], + blocks: Mapping[BlockKey, list[MemoryRef]], request_id: str | None = None, ) -> OpHandle: op_handle = self._next_op_handle @@ -118,7 +118,7 @@ class RecordingKVCR: def deposit( self, - blocks: Mapping[BlockKey, list[MemDescriptor]], + blocks: Mapping[BlockKey, list[MemoryRef]], ) -> OpHandle: op_handle = self._next_op_handle self._next_op_handle += 1 @@ -182,6 +182,7 @@ def _make_tier( g3: dict[str, object] | None = None, control_ports: list[int] | None = None, data_parallel_rank_local: int | None = None, + backpressure: dict[str, Any] | None = None, ) -> KVCRSecondaryTierManager: def make_control(_bind_host, bind_port, advertise_host): return _StubControlChannel(f"tcp://{advertise_host}:{int(bind_port)}") @@ -198,7 +199,25 @@ def _make_tier( monkeypatch.setattr(kvcr_manager, "KVCR", make_kvcr) monkeypatch.setattr(kvcr_manager, "ZmqPeerControlChannel", make_control) - return KVCRSecondaryTierManager( + tier_config = { + "type": "kvcr", + "router_capabilities": ["router_hint"], + "control_host": "127.0.0.1", + "control_ports": control_ports if control_ports is not None else [7777], + "control_advertise_host": "127.0.0.1", + "enable_telemetry": enable_telemetry, + "secondary_g2_slots": secondary_g2_slots, + "kvcr_service_socket_path": kvcr_service_socket_path, + "compatibility_digest": compatibility_digest, + "policy": policy, + "g3": g3, + "local_dram_backend": "UCX", + "remote_fw_dram_backend": "UCX", + } + if backpressure is not None: + tier_config["backpressure"] = backpressure + tier = SecondaryTierFactory.create_secondary_tier( + tier_config=tier_config, offloading_spec=SimpleNamespace( config=SimpleNamespace( parallel=SimpleNamespace( @@ -211,20 +230,9 @@ def _make_tier( ), ), primary_kv_view=memoryview(np.zeros((4, 16), dtype=np.int8)), - tier_type="kvcr", - router_capabilities=["router_hint"], - control_host="127.0.0.1", - control_ports=control_ports if control_ports is not None else [7777], - control_advertise_host="127.0.0.1", - enable_telemetry=enable_telemetry, - secondary_g2_slots=secondary_g2_slots, - kvcr_service_socket_path=kvcr_service_socket_path, - compatibility_digest=compatibility_digest, - policy=policy, - g3=g3, - local_dram_backend="UCX", - remote_fw_dram_backend="UCX", ) + assert isinstance(tier, KVCRSecondaryTierManager) + return tier def test_kvcr_tier_configures_service_for_local_dp_rank(monkeypatch): @@ -248,7 +256,11 @@ def test_kvcr_tier_configures_service_for_local_dp_rank(monkeypatch): compatibility_digest="Opaque-Digest", ) assert kvcr.config is not None - assert kvcr.config.pool_layouts == [("", tier._primary_row_stride)] + assert kvcr.config.pool_layouts == [("", tier.block_size_bytes)] + (region,) = kvcr.backend_configs.framework_regions + assert region.addr == np.asarray(tier._primary_kv_view).ctypes.data + assert (region.size, region.stride or region.size, region.count) == (16, 16, 4) + assert (region.label, region.mem_type, region.device_Id) == ("", "DRAM", 0) assert kvcr.backend_configs.local_dram is None @@ -269,10 +281,19 @@ def test_kvcr_tier_converts_g3_paths(monkeypatch, tmp_path): ) -def test_kvcr_tier_adapts_request_and_load(monkeypatch): - """Check hint forwarding, key conversion, load descriptors, and cleanup.""" +@pytest.mark.parametrize( + "backpressure", + [None, {"backpressure_cls": "EMABackpressureDetector"}], + ids=["default", "backpressure"], +) +def test_kvcr_tier_adapts_request_and_load(monkeypatch, backpressure): + """Factory-created tiers preserve configuration and complete hinted loads.""" kvcr = RecordingKVCR() - tier = _make_tier(monkeypatch, kvcr) + tier = _make_tier(monkeypatch, kvcr, backpressure=backpressure) + if backpressure is None: + assert tier.bp_detector is None + else: + assert isinstance(tier.bp_detector, EMABackpressureDetector) kv_hint = KvHintsEnvelope(protocol_version="0.1", message_id="msg", actions=[]) ctx = ReqContext( req_id="req", @@ -306,9 +327,7 @@ def test_kvcr_tier_adapts_request_and_load(monkeypatch): assert request_id == "req" assert list(blocks) == [key] (descriptor,) = blocks[key] - assert descriptor.end_point_name == kvcr.nixl_agent_name - assert descriptor.addr == tier._primary_base_addr + 2 * 16 - assert descriptor.size == 16 + assert descriptor == MemoryRef(end_point_name=kvcr.nixl_agent_name, element_index=2) assert list(tier.get_finished_jobs()) == [JobResult(7, True)] # Here we verify that request cleanup discards the request-scoped hint in @@ -351,7 +370,7 @@ def test_kvcr_tier_serves_primary_pin_request(monkeypatch): tier = _make_tier(monkeypatch, kvcr) keys = (BlockKey(b"k0"), BlockKey(b"k1"), BlockKey(b"k2")) hit_keys = (keys[0], keys[2]) - chunk_ids = {keys[0]: 1, keys[2]: 5} + chunk_ids = {keys[0]: 1, keys[2]: 3} lifecycle: list[str] = [] class Parent: @@ -388,7 +407,7 @@ def test_kvcr_tier_serves_primary_pin_request(monkeypatch): block_descriptors = descriptors[keys[2]] assert block_descriptors is not None (descriptor,) = block_descriptors - assert descriptor.addr == (tier._primary_base_addr + 5 * tier._primary_row_stride) + assert descriptor == MemoryRef(end_point_name=kvcr.nixl_agent_name, element_index=3) assert lifecycle == ["new", "finished"] polls = 0 @@ -482,14 +501,16 @@ def test_kvcr_tier_stores_and_emits_inventory(monkeypatch): local_dram = kvcr.backend_configs.local_dram assert local_dram is not None assert [(name, size) for name, _, size in local_dram.pools] == [ - ("", 2 * tier._primary_row_stride) + ("", 2 * tier.block_size_bytes) ] key = OffloadKey(b"k0") tier.submit_store(_job(11, ReqContext(req_id="req"), key=key, chunk_id=2)) _, blocks = kvcr.deposit_calls[0] - assert blocks[key][0].addr == tier._primary_base_addr + 2 * 16 + assert blocks[key] == [ + MemoryRef(end_point_name=kvcr.nixl_agent_name, element_index=2) + ] assert list(tier.get_finished_jobs()) == [JobResult(11, True)] assert kvcr.inventory_sink is not None diff --git a/tests/v1/kv_offload/tiering/test_tiering_offloading.py b/tests/v1/kv_offload/tiering/test_tiering_offloading.py index 63b4c2c4d0..93b49afb4f 100644 --- a/tests/v1/kv_offload/tiering/test_tiering_offloading.py +++ b/tests/v1/kv_offload/tiering/test_tiering_offloading.py @@ -53,6 +53,7 @@ from vllm.v1.kv_offload.tiering.manager import ( TieringOffloadingManager, ) from vllm.v1.kv_offload.tiering.spec import TieringOffloadingSpec +from vllm.v1.metrics.cache_hit_source import CacheHitSource _CTX = ReqContext(req_id="test") _MOCK_OFFLOADING_SPEC = MagicMock() @@ -500,8 +501,13 @@ class TestTieringOffloadingManager: # Lookup should find all chunks in primary assert count_hits(self.manager, chunks) == 3 - def test_promotion_from_secondary(self, manager_setup): + @pytest.mark.parametrize( + "source", + [CacheHitSource.EXTERNAL_UNSPECIFIED, CacheHitSource.DISK, CacheHitSource.P2P], + ) + def test_promotion_from_secondary(self, manager_setup, monkeypatch, source): """Test promotion of chunks from secondary to primary tier.""" + monkeypatch.setattr(ExampleSecondaryTierManager, "cache_hit_source", source) chunks = to_keys(range(3)) # Manually add chunks to secondary tier (simulate previous cascade) @@ -525,6 +531,27 @@ class TestTieringOffloadingManager: # Next lookup should succeed assert count_hits(self.manager, chunks) == 3 + # The request that caused the promotion retains the secondary origin. + assert all( + self.manager.get_load_source(chunk, _CTX) == source for chunk in chunks + ) + + # A later request sees the chunks as ordinary host-memory primary hits. + later_context = ReqContext(req_id="later") + self.manager.on_new_request(later_context) + assert all( + self.manager.lookup(chunk, later_context) is LookupResult.HIT + for chunk in chunks + ) + assert all( + self.manager.get_load_source(chunk, later_context) == "host" + for chunk in chunks + ) + + @pytest.mark.parametrize( + "source", + [CacheHitSource.EXTERNAL_UNSPECIFIED, CacheHitSource.DISK, CacheHitSource.P2P], + ) @pytest.mark.parametrize( ("successful_indices", "expected_results"), [ @@ -540,8 +567,9 @@ class TestTieringOffloadingManager: ids=["partial", "legacy-full-failure"], ) def test_failed_promotion_keeps_only_successful_chunks( - self, manager_setup, successful_indices, expected_results + self, manager_setup, monkeypatch, source, successful_indices, expected_results ): + monkeypatch.setattr(ExampleSecondaryTierManager, "cache_hit_source", source) chunks = to_keys(range(3)) for chunk in chunks: self.secondary_tier1.chunks[chunk] = True @@ -571,6 +599,14 @@ class TestTieringOffloadingManager: assert [ self.primary_tier.lookup(chunk, _CTX) for chunk in chunks ] == expected_results + expected_source_keys = ( + set() + if successful_indices is None + else {chunks[i] for i in successful_indices} + ) + assert set(self.manager._request_load_sources[_CTX.req_id]) == ( + expected_source_keys + ) def test_lookup_reports_sync_delay_for_resolved_lookups(self, manager_setup): """Resolved lookups report one sync delay sample per tier and chunk.""" @@ -1315,6 +1351,7 @@ class TestTieringOffloadingManager: # Orchestrator state cleared. assert self.manager._jobs == {} assert self.manager._pending_load_submissions == {} + assert self.manager._request_load_sources == {} assert set(self.manager._req_state) == {_CTX.req_id, rl_ctx.req_id} assert self.manager._processed_jobs_this_step is False diff --git a/tests/v1/metrics/test_stats.py b/tests/v1/metrics/test_stats.py index 0de74f0faa..c51c223d28 100644 --- a/tests/v1/metrics/test_stats.py +++ b/tests/v1/metrics/test_stats.py @@ -1,10 +1,13 @@ # SPDX-License-Identifier: Apache-2.0 # SPDX-FileCopyrightText: Copyright contributors to the vLLM project + from vllm.v1.core.sched.output import ScheduledEncoderInputStats, SchedulerOutput from vllm.v1.engine import EngineCoreOutputs, FinishReason +from vllm.v1.metrics.cache_hit_source import CacheHitSource from vllm.v1.metrics.stats import ( IterationStats, PrefillStats, + PrefixCacheStats, PromptTokenStats, RequestStateStats, SchedulerIterationDetails, @@ -45,6 +48,96 @@ def test_scheduler_iteration_details_serialization(): assert decoded.scheduler_stats.iteration_details == iteration_details +def test_prefix_cache_stats_hits_by_source_accumulate(): + stats = PrefixCacheStats() + stats.record( + num_tokens=8, + num_hits=5, + preempted=False, + hits_by_source={CacheHitSource.HOST: 3, CacheHitSource.DISK: 2}, + ) + stats.record( + num_tokens=8, + num_hits=6, + preempted=False, + hits_by_source={CacheHitSource.P2P: 5, CacheHitSource.HOST: 1}, + ) + # Zero counts never create an entry. + stats.record( + num_tokens=8, + num_hits=4, + preempted=False, + hits_by_source={CacheHitSource.DISK: 4, CacheHitSource.P2P: 0}, + ) + + assert stats.hits_by_source == { + CacheHitSource.HOST: 4, + CacheHitSource.DISK: 6, + CacheHitSource.P2P: 5, + } + assert sum(stats.hits_by_source.values()) == stats.hits == 15 + + +def test_prefix_cache_stats_record_hits_by_source(): + stats = PrefixCacheStats() + stats.record(num_tokens=32, num_hits=16, preempted=False) + stats.record( + num_tokens=32, + num_hits=24, + preempted=False, + hits_by_source={CacheHitSource.HOST: 16, CacheHitSource.DISK: 8}, + ) + # Preempted re-admissions never contribute to hits or the split. + stats.record( + num_tokens=32, + num_hits=32, + preempted=True, + hits_by_source={CacheHitSource.P2P: 32}, + ) + # A connector miscount is reported as unspecified, not dropped or raised. + stats.record( + num_tokens=8, + num_hits=8, + preempted=False, + hits_by_source={CacheHitSource.HOST: 4}, + ) + + assert stats.requests == 3 + assert stats.hits == 48 + assert stats.preempted_hits == 32 + assert stats.hits_by_source == { + CacheHitSource.HOST: 16, + CacheHitSource.DISK: 8, + CacheHitSource.EXTERNAL_UNSPECIFIED: 8, + } + assert sum(stats.hits_by_source.values()) == 48 - 16 + + +def test_prefix_cache_stats_hits_by_source_serialization(): + connector_stats = PrefixCacheStats() + connector_stats.record( + num_tokens=16, + num_hits=8, + preempted=False, + hits_by_source={CacheHitSource.P2P: 4, CacheHitSource.HOST: 4}, + ) + outputs = EngineCoreOutputs( + scheduler_stats=SchedulerStats( + connector_prefix_cache_stats=connector_stats, + ) + ) + + encoded = MsgpackEncoder().encode(outputs) + decoded = MsgpackDecoder(EngineCoreOutputs).decode(encoded) + + assert decoded.scheduler_stats is not None + assert decoded.scheduler_stats.connector_prefix_cache_stats is not None + assert decoded.scheduler_stats.connector_prefix_cache_stats.hits_by_source == { + CacheHitSource.P2P: 4, + CacheHitSource.HOST: 4, + } + + def test_compute_iteration_details_includes_encoder_stats(): scheduler_output = SchedulerOutput.make_empty() scheduler_output.scheduled_encoder_input_stats = ScheduledEncoderInputStats( diff --git a/tests/v1/sample/test_diffusion_gemma_reads.py b/tests/v1/sample/test_diffusion_gemma_reads.py index 7bd04c0379..4abadc254d 100644 --- a/tests/v1/sample/test_diffusion_gemma_reads.py +++ b/tests/v1/sample/test_diffusion_gemma_reads.py @@ -227,6 +227,10 @@ def test_single_step_tile_skips_self_conditioning(): assert not states.self_conditioning_embeds[0].any() +def test_sample_step_is_compiled(): + assert hasattr(_compiled_sample_step, "_torchdynamo_orig_callable") + + @pytest.mark.parametrize("stance", ["default", "force_eager"]) def test_self_conditioning_stores_a_bf16_model_in_the_fp32_buffer(stance): # The model's embeddings are bf16 while the buffer is fp32. Compiled code diff --git a/tests/v1/sample/test_logprobs.py b/tests/v1/sample/test_logprobs.py index 317ac10d41..a9b73f0ca2 100644 --- a/tests/v1/sample/test_logprobs.py +++ b/tests/v1/sample/test_logprobs.py @@ -518,6 +518,7 @@ def test_prompt_logprob_token_ids_require_v2_model_runner(): structured_outputs_config=None, tokenizer=None, validate_logits_processors_params=lambda params: None, + resolve_watermarking=lambda params: False, ) params = SamplingParams(prompt_logprob_token_ids=[[1, 2]]) with patch.object(SamplingParams, "verify"): @@ -1469,7 +1470,10 @@ def test_prompt_logprob_token_ids_with_chunking_and_preemption(monkeypatch): assert float(scores.max()) <= 1e-3, "logprobs must be <= 0" # Ragged rows: each row keeps a prefix of the candidates and must - # reproduce that prefix of the full scores, with -inf padding. + # reproduce that prefix of the full scores, with -inf padding. On ROCm + # the second generate call picks its own kernels, which moves bf16 + # tails by up to 0.19; a misaligned row is off by nats either way. + ragged_atol = 0.2 if current_platform.is_rocm() else 1e-2 ragged_params = make_params(lambda j: candidate_ids(j)[: 1 + j % 4]) ragged_outputs = vllm_model.llm.generate(token_prompts, ragged_params) for output, ragged_output in zip(outputs, ragged_outputs): @@ -1478,7 +1482,7 @@ def test_prompt_logprob_token_ids_with_chunking_and_preemption(monkeypatch): n = 1 + j % 4 expected[j, :n] = output.prompt_token_id_logprobs[j, :n] np.testing.assert_allclose( - ragged_output.prompt_token_id_logprobs, expected, atol=1e-2 + ragged_output.prompt_token_id_logprobs, expected, atol=ragged_atol ) # The row count is checked against the prompt at admission. diff --git a/tests/v1/simple_kv_offload/test_scheduler.py b/tests/v1/simple_kv_offload/test_scheduler.py index 5e299d2788..08759bc1e3 100644 --- a/tests/v1/simple_kv_offload/test_scheduler.py +++ b/tests/v1/simple_kv_offload/test_scheduler.py @@ -6,6 +6,7 @@ from __future__ import annotations import logging from dataclasses import dataclass +from unittest.mock import Mock import pytest import torch @@ -17,16 +18,18 @@ from vllm.config import ( KVTransferConfig, ModelConfig, SchedulerConfig, + SpeculativeConfig, VllmConfig, ) from vllm.config.cache import MambaCacheMode from vllm.config.kv_events import KVEventsConfig +from vllm.distributed.kv_transfer.kv_connector.v1 import KVConnectorRole from vllm.distributed.kv_transfer.kv_connector.v1.simple_cpu_offload_connector import ( SimpleCPUOffloadConnector, ) from vllm.utils.hashing import sha256 from vllm.v1.core.block_pool import BlockPool -from vllm.v1.core.kv_cache_manager import KVCacheBlocks +from vllm.v1.core.kv_cache_manager import KVCacheBlocks, KVCacheManager from vllm.v1.core.kv_cache_utils import ( get_request_block_hasher, init_none_hash, @@ -51,6 +54,7 @@ from vllm.v1.kv_cache_interface import ( MambaSpec, SlidingWindowSpec, ) +from vllm.v1.metrics.cache_hit_source import CacheHitSource from vllm.v1.outputs import KVConnectorOutput from vllm.v1.request import Request from vllm.v1.simple_kv_offload.manager import SimpleCPUOffloadScheduler @@ -211,6 +215,37 @@ class SchedulerFixture: num_groups: int = 1 +@pytest.mark.parametrize( + ("backend", "source"), + [("cpu", CacheHitSource.HOST), ("disk", CacheHitSource.DISK)], +) +def test_connector_reports_configured_cache_source( + tmp_path, backend: str, source: CacheHitSource +): + kv_cache_config = _make_kv_cache_config(num_blocks=4) + vllm_config = _make_vllm_config() + extra_config = { + "cpu_bytes_to_use_per_rank": _BYTES_PER_BLOCK * 4, + "kv_offload_backend": backend, + } + if backend == "disk": + extra_config.update( + disk_path=str(tmp_path / "kv-offload.bin"), + disk_capacity_bytes=_BYTES_PER_BLOCK * 4, + ) + assert vllm_config.kv_transfer_config is not None + vllm_config.kv_transfer_config.kv_connector_extra_config = extra_config + + connector = SimpleCPUOffloadConnector( + vllm_config, + KVConnectorRole.SCHEDULER, + kv_cache_config, + ) + + assert connector.get_external_cache_hit_sources(None, 32) == {source: 32} # type: ignore[arg-type] + assert connector.get_external_cache_hit_sources(None, 0) == {} # type: ignore[arg-type] + + def make_scheduler( num_cpu_blocks: int = 8, num_gpu_blocks: int = 16, @@ -3089,3 +3124,184 @@ def test_lazy_target_blocks_ignore_non_prefix_cacheable_groups() -> None: without_ring, max_batched ) assert target_with == target_without + + +@pytest.mark.parametrize("retention_interval", [None, 0]) +@pytest.mark.parametrize("chunk_size,prefill_lookahead", [(64, 1), (1024, 1), (1, 3)]) +@pytest.mark.parametrize( + "draft_sliding_window,draft_sharded", + [(True, False), (False, False), (False, True)], + ids=["swa", "replicated_full", "sharded_full"], +) +def test_eager_store_matches_gpu_cacheable_prefix( + retention_interval: int | None, + chunk_size: int, + prefill_lookahead: int, + draft_sliding_window: bool, + draft_sharded: bool, +) -> None: + """Preserve GPU hits for replicated SWA and DCP-sharded full-attention drafts.""" + block_size = 64 + scheduler_block_size = 8 * block_size + draft_block_size = scheduler_block_size if draft_sharded else block_size + prompt_len = 16 * scheduler_block_size + draft_block_size + prefill_lookahead + kv_cache_config = _make_kv_cache_config(256, num_groups=2) + for group_id, group in enumerate(kv_cache_config.kv_cache_groups): + group.kv_cache_spec = FullAttentionSpec( + block_size=block_size, + num_kv_heads=NUM_KV_HEADS, + head_size=HEAD_SIZE, + dtype=DTYPE, + dcp_sharded=group_id == 0 or draft_sharded, + ) + for tensor in kv_cache_config.kv_cache_tensors: + tensor.size *= block_size // BLOCK_SIZE + tensor.layer_stride *= block_size // BLOCK_SIZE + tensor.block_stride *= block_size // BLOCK_SIZE + if draft_sliding_window: + kv_cache_config.kv_cache_groups[1].kv_cache_spec = SlidingWindowSpec( + block_size=block_size, + num_kv_heads=NUM_KV_HEADS, + head_size=HEAD_SIZE, + dtype=DTYPE, + sliding_window=scheduler_block_size, + extra_retained_tokens=prefill_lookahead - 1, + dcp_sharded=False, + ) + kv_cache_config.kv_cache_groups[1].is_eagle_group = True + kv_cache_config.prefix_cache_retention_interval = retention_interval + vllm_config = _make_cp_vllm_config(dcp_world_size=8) + vllm_config.model_config.max_model_len = 16384 + speculative_config = Mock(spec=SpeculativeConfig) + speculative_config.num_speculative_tokens = 3 + speculative_config.use_eagle.return_value = True + speculative_config.use_eagle_block_drop.return_value = True + speculative_config.use_multi_module_mtp.return_value = prefill_lookahead > 1 + vllm_config.speculative_config = speculative_config + scheduler = SimpleCPUOffloadScheduler( + vllm_config=vllm_config, + kv_cache_config=kv_cache_config, + cpu_capacity_bytes=kv_cache_config.kv_cache_tensors[0].size, + scheduler_block_size=scheduler_block_size, + hash_block_size=block_size, + ) + gpu_manager = KVCacheManager( + kv_cache_config, + max_model_len=16384, + max_in_flight_tokens=chunk_size, + scheduler_block_size=scheduler_block_size, + hash_block_size=block_size, + dcp_world_size=8, + use_eagle=True, + num_prefill_lookahead=prefill_lookahead, + ) + scheduler.bind_gpu_block_pool(gpu_manager.block_pool) + request = Request( + request_id="dcp-draft-source", + prompt_token_ids=list(range(prompt_len)), + sampling_params=SamplingParams(max_tokens=2), + pooling_params=None, + mm_features=None, + block_hasher=get_request_block_hasher(block_size, sha256), + ) + # Follow allocate -> connector metadata -> compute completion, including + # one decode step so the final prefill blocks can be offloaded. + while request.num_computed_tokens <= prompt_len: + if request.num_computed_tokens == prompt_len: + request.append_output_token_ids([prompt_len]) + num_new_tokens = min( + chunk_size, request.num_tokens - request.num_computed_tokens + ) + new_blocks = gpu_manager.allocate_slots( + request, num_new_tokens, num_lookahead_tokens=3 + ) + assert new_blocks is not None + if request.num_computed_tokens == 0: + scheduler.update_state_after_alloc( + request, gpu_manager.get_blocks(request.request_id), 0 + ) + output = make_scheduler_output( + {request.request_id: num_new_tokens}, + new_reqs={request.request_id: new_blocks.get_block_ids()}, + ) + else: + output = make_scheduler_output( + {request.request_id: num_new_tokens}, + cached_req_new_blocks={request.request_id: new_blocks.get_block_ids()}, + ) + store_meta = scheduler.build_connector_meta(output) + request.num_computed_tokens += num_new_tokens + if store_meta.store_event >= 0: + simulate_store_completion(scheduler, store_meta.store_event) + + matching_request = Request( + request_id="dcp-draft-load", + prompt_token_ids=request.prompt_token_ids, + sampling_params=request.sampling_params, + pooling_params=None, + mm_features=None, + block_hasher=request._block_hasher, + ) + expected_hit = ( + 15 if draft_sliding_window and retention_interval is not None else 16 + ) * scheduler_block_size + assert gpu_manager.get_computed_blocks(matching_request)[1] == expected_hit + scheduler.request_finished(request, []) + gpu_manager.free(request) + assert gpu_manager.block_pool.reset_prefix_cache() + + hit_tokens, is_async = scheduler.get_num_new_matched_tokens( + matching_request, num_computed_tokens=0 + ) + assert hit_tokens is not None + assert (hit_tokens, is_async) == (expected_hit, True) + + assert ( + gpu_manager.allocate_slots( + matching_request, + num_new_tokens=0, + num_external_computed_tokens=hit_tokens, + delay_cache_blocks=True, + ) + is not None + ) + load_blocks = gpu_manager.get_blocks(matching_request.request_id) + scheduler.update_state_after_alloc(matching_request, load_blocks, hit_tokens) + load_meta = scheduler.build_connector_meta( + make_scheduler_output( + {matching_request.request_id: 0}, + new_reqs={matching_request.request_id: load_blocks.get_block_ids()}, + ) + ) + # SWA loads only the eight-block tail after null padding; a sharded full + # draft must load every prefix block, with the same DCP geometry as the target. + num_draft_blocks = 8 if draft_sliding_window else hit_tokens // draft_block_size + assert load_meta.load_gpu_blocks == [ + block.block_id + for block in ( + *load_blocks.blocks[0], + *load_blocks.blocks[1][-num_draft_blocks:], + ) + ] + assert ( + len(load_meta.load_gpu_blocks) + == hit_tokens // scheduler_block_size + num_draft_blocks + ) + target_hashes = [ + make_block_hash_with_group_id(matching_request.block_hashes[(i + 1) * 8 - 1], 0) + for i in range(hit_tokens // scheduler_block_size) + ] + draft_hashes = [ + make_block_hash_with_group_id( + matching_request.block_hashes[(i + 1) * draft_block_size // block_size - 1], + 1, + ) + for i in range( + hit_tokens // draft_block_size - num_draft_blocks, + hit_tokens // draft_block_size, + ) + ] + assert [ + scheduler.cpu_block_pool.blocks[bid].block_hash + for bid in load_meta.load_cpu_blocks + ] == target_hashes + draft_hashes diff --git a/tests/v1/spec_decode/test_max_len.py b/tests/v1/spec_decode/test_max_len.py index 81f8424193..17dd318563 100644 --- a/tests/v1/spec_decode/test_max_len.py +++ b/tests/v1/spec_decode/test_max_len.py @@ -36,6 +36,39 @@ def test_ngram_max_len(num_speculative_tokens: int, vllm_runner): runner.llm.generate(_PROMPTS, sampling_params) +@pytest.mark.parametrize("num_speculative_tokens", [1, 3, 10]) +@pytest.mark.parametrize("method", ["ngram", "ngram_gpu"]) +def test_ngram_gpu_max_len( + method: str, + num_speculative_tokens: int, + vllm_runner, + monkeypatch: pytest.MonkeyPatch, +): + """V2 n-gram decoding stops at max_model_len.""" + monkeypatch.setenv("VLLM_USE_V2_MODEL_RUNNER", "1") + with vllm_runner( + "facebook/opt-125m", + trust_remote_code=False, + max_model_len=100, + enable_chunked_prefill=None, + enforce_eager=True, # For faster initialization. + speculative_config={ + "method": method, + "prompt_lookup_max": 5, + "prompt_lookup_min": 3, + "num_speculative_tokens": num_speculative_tokens, + }, + ) as runner: + assert runner.llm.llm_engine.vllm_config.use_v2_model_runner + sampling_params = SamplingParams(max_tokens=100, ignore_eos=True) + outputs = runner.llm.generate(_PROMPTS, sampling_params) + for output in outputs: + assert output.prompt_token_ids is not None + assert ( + len(output.prompt_token_ids) + len(output.outputs[0].token_ids) == 100 + ) + + @pytest.mark.parametrize("num_speculative_tokens", [1, 3, 10]) @pytest.mark.parametrize("attn_backend", get_attn_backend_list_based_on_platform()) def test_eagle_max_len( diff --git a/tests/v1/spec_decode/test_mtp_structured_output.py b/tests/v1/spec_decode/test_mtp_structured_output.py index 5e4312227d..8a72765db1 100644 --- a/tests/v1/spec_decode/test_mtp_structured_output.py +++ b/tests/v1/spec_decode/test_mtp_structured_output.py @@ -11,6 +11,7 @@ from vllm.config import StructuredOutputsConfig, VllmConfig from vllm.config.model import ModelConfig from vllm.config.speculative import SpeculativeConfig from vllm.parser.engine.adapters import ParserEngineReasoningAdapter +from vllm.parser.engine.parser_engine import ReasoningEnd from vllm.sampling_params import SamplingParams, StructuredOutputsParams from vllm.v1.request import Request from vllm.v1.structured_output import StructuredOutputManager @@ -113,12 +114,12 @@ class _EngineReasonerStub(ParserEngineReasoningAdapter): def reasoning_end_token_ids(self): return self._end_token_ids - def find_reasoning_end_offset(self, token_ids): + def find_reasoning_end(self, token_ids): self.windows.append(list(token_ids)) for offset, token in enumerate(token_ids): if token in self._end_token_ids: - return offset - return len(token_ids) + return ReasoningEnd(offset, False) + return ReasoningEnd(len(token_ids), False) def is_reasoning_end(self, input_ids): return any(token in self._end_token_ids for token in input_ids) diff --git a/tests/v1/spec_decode/test_ngram_gpu.py b/tests/v1/spec_decode/test_ngram_gpu.py new file mode 100644 index 0000000000..2359f0cdef --- /dev/null +++ b/tests/v1/spec_decode/test_ngram_gpu.py @@ -0,0 +1,279 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Unit tests for the V2 GPU-accelerated n-gram speculator. + +These tests target the Triton proposer in +``vllm.v1.worker.gpu.spec_decode.ngram.speculator`` and complement the CPU +``NgramProposer`` tests in ``test_ngram.py``. The GPU speculator follows a +slightly different policy than the CPU one: when multiple n-gram matches of +the same length exist, the GPU kernel picks the right-most (most recent) +match inside the active context, whereas the CPU implementation returns the +left-most. The expectations below reflect the GPU behavior. +""" + +from __future__ import annotations + +from types import SimpleNamespace + +import pytest +import torch + +from vllm.config import ( + ModelConfig, + SchedulerConfig, + SpeculativeConfig, + VllmConfig, +) +from vllm.v1.worker.gpu.spec_decode.ngram.speculator import NgramGPUSpeculator +from vllm.v1.worker.gpu.states import RequestState + +if not torch.cuda.is_available(): + pytest.skip( + "CUDA required for NgramGPUSpeculator tests", + allow_module_level=True, + ) + +DEVICE = torch.device("cuda") + + +def _make_vllm_config( + min_n: int, + max_n: int, + k: int, + max_num_seqs: int = 8, + max_model_len: int = 64, + method: str = "ngram_gpu", +) -> VllmConfig: + model_config = ModelConfig( + model="facebook/opt-125m", + max_model_len=max_model_len, + enforce_eager=True, + ) + scheduler_config = SchedulerConfig.default_factory( + max_num_seqs=max_num_seqs, + max_model_len=max_model_len, + ) + speculative_config = SpeculativeConfig( + method=method, + prompt_lookup_min=min_n, + prompt_lookup_max=max_n, + num_speculative_tokens=k, + ) + return VllmConfig( + model_config=model_config, + scheduler_config=scheduler_config, + speculative_config=speculative_config, + ) + + +def _make_request_state(cfg: VllmConfig) -> RequestState: + assert cfg.speculative_config is not None + return RequestState( + max_num_reqs=cfg.scheduler_config.max_num_seqs, + max_model_len=cfg.model_config.max_model_len, + max_num_batched_tokens=cfg.scheduler_config.max_num_batched_tokens, + num_speculative_steps=cfg.speculative_config.num_speculative_tokens, + vocab_size=cfg.model_config.get_vocab_size(), + device=DEVICE, + use_dense_all_token_ids=True, + ) + + +def _make_speculator( + min_n: int, + max_n: int, + k: int, + max_num_seqs: int = 8, + max_model_len: int = 32, +) -> NgramGPUSpeculator: + cfg = _make_vllm_config( + min_n=min_n, + max_n=max_n, + k=k, + max_num_seqs=max_num_seqs, + max_model_len=max_model_len, + ) + return NgramGPUSpeculator(cfg, DEVICE, _make_request_state(cfg)) + + +def _propose( + spec: NgramGPUSpeculator, + rows: list[list[int]], + seq_lens: list[int] | None = None, + num_sampled: list[int] | None = None, + last_sampled: list[int] | None = None, + slots: list[int] | None = None, +) -> list[list[int]]: + """Place each batch row at a request slot and run propose(). + + Returns the drafts as python lists in batch order. + """ + B = len(rows) + if seq_lens is None: + seq_lens = [len(r) for r in rows] + if num_sampled is None: + num_sampled = [1] * B + if last_sampled is None: + last_sampled = [0] * B + if slots is None: + slots = list(range(B)) + + max_num_reqs = spec.max_num_reqs + all_token_ids = spec.req_states.all_token_ids.gpu + total_len = spec.req_states.total_len.gpu + all_token_ids.zero_() + total_len.zero_() + last_sampled_t = torch.zeros((max_num_reqs, 1), dtype=torch.int64, device=DEVICE) + for row, slot, seq_len, last in zip(rows, slots, seq_lens, last_sampled): + if row: + all_token_ids[slot, : len(row)] = torch.tensor( + row, dtype=torch.int32, device=DEVICE + ) + total_len[slot] = seq_len + last_sampled_t[slot, 0] = last + + idx_mapping = torch.tensor(slots, dtype=torch.int64, device=DEVICE) + input_batch = SimpleNamespace(num_reqs=B, idx_mapping=idx_mapping) + + drafts = spec.propose( + input_batch=input_batch, + attn_metadata=None, + slot_mappings=None, + last_hidden_states=torch.empty(0, device=DEVICE), + aux_hidden_states=None, + num_sampled=torch.tensor(num_sampled, dtype=torch.int32, device=DEVICE), + num_rejected=torch.zeros(B, dtype=torch.int32, device=DEVICE), + last_sampled=last_sampled_t, + next_prefill_tokens=torch.zeros(B, dtype=torch.int32, device=DEVICE), + temperature=torch.zeros(B, dtype=torch.float32, device=DEVICE), + seeds=torch.zeros(B, dtype=torch.int64, device=DEVICE), + dp_sync=None, + ) + return drafts.cpu().tolist() + + +@pytest.mark.parametrize("max_model_len", [32, 300]) +def test_no_match_clears_previous_proposal(max_model_len): + spec = _make_speculator(min_n=2, max_n=2, k=2, max_model_len=max_model_len) + row = [0] * (max_model_len - 5) + [1, 2, 3, 1, 2] + assert _propose(spec, [row]) == [[3, 1]] + drafts = _propose(spec, [[1, 2, 3, 4, 5]], last_sampled=[42]) + assert drafts == [[42, 42]] + + +def test_no_4gram_match_only(): + """No 4-gram match in [1,2,3,4,1,2,3] → 0 valid drafts.""" + spec = _make_speculator(min_n=4, max_n=4, k=2) + drafts = _propose(spec, [[1, 2, 3, 4, 1, 2, 3]], last_sampled=[7]) + assert drafts == [[7, 7]] + + +def test_falls_back_to_3gram_when_4gram_missing(): + """No 4-gram match but a 3-gram match exists → propose [4, 1].""" + spec = _make_speculator(min_n=3, max_n=4, k=2) + drafts = _propose(spec, [[1, 2, 3, 4, 1, 2, 3]]) + assert drafts == [[4, 1]] + + +def test_prefers_longer_ngram(): + """Prefer a 4-gram match over a more recent 3-gram match.""" + spec = _make_speculator(min_n=3, max_n=4, k=2) + drafts = _propose(spec, [[1, 2, 3, 4, 50, 51, 2, 3, 4, 60, 61, 1, 2, 3, 4]]) + assert drafts == [[50, 51]] + + +def test_picks_longest_match_among_2_3_4_grams(): + """Prefer a 3-gram match over a more recent 2-gram match.""" + spec = _make_speculator(min_n=2, max_n=4, k=2) + drafts = _propose(spec, [[2, 3, 4, 50, 51, 3, 4, 60, 61, 1, 2, 3, 4]]) + assert drafts == [[50, 51]] + + +@pytest.mark.parametrize("max_model_len", [32, 128, 257, 1025]) +def test_picks_rightmost_when_multiple_matches(max_model_len): + """Pick the last valid match across blocks, ignoring trailing tokens.""" + spec = _make_speculator(min_n=3, max_n=3, k=2, max_model_len=max_model_len) + padding = [0] * (spec.block_l - 5) if spec.n_blocks > 1 else [] + row = [1, 2, 3, 100] + padding + [1, 2, 3, 200, 1, 2, 3, 300, 1, 2, 3] + drafts = _propose(spec, [row + [1, 2, 3, 999, 1, 2, 3]], seq_lens=[len(row)]) + assert drafts == [[300, 1]] + + +def test_short_context_yields_zero_valid(): + """The only length-2 window overlaps the suffix itself → no match.""" + spec = _make_speculator(min_n=2, max_n=2, k=2) + drafts = _propose(spec, [[5, 6]], last_sampled=[99]) + assert drafts == [[99, 99]] + + +def test_zero_sampled_disables_proposal(): + """num_sampled==0 disables proposals for that request regardless of match.""" + spec = _make_speculator(min_n=2, max_n=2, k=2) + drafts = _propose(spec, [[1, 2, 3, 1, 2]], num_sampled=[0], last_sampled=[77]) + assert drafts == [[77, 77]] + + +def test_tail_falls_back_when_few_tokens_after_match(): + """Fewer than k tokens after the match → the tail falls back. + + Tokens: [1, 2, 1, 2] (seq_len=4). Suffix (1, 2) matches at position 0 + (the match at position 2 is the suffix itself and is excluded). With + k=3, only 2 slots map to tokens inside the context. + """ + spec = _make_speculator(min_n=2, max_n=2, k=3) + drafts = _propose(spec, [[1, 2, 1, 2]], last_sampled=[55]) + assert drafts == [[1, 2, 55]] + + +def test_multibatch_mixed(): + """Mixed batch: row 0 matches, row 1 has no match.""" + spec = _make_speculator(min_n=2, max_n=2, k=2) + drafts = _propose( + spec, + [[1, 2, 3, 1, 2], [4, 5, 6]], + last_sampled=[10, 20], + ) + assert drafts[0] == [3, 1] + assert drafts[1] == [20, 20] + + +def test_multibatch_independent_choice_of_n(): + """Each row independently picks its longest matched n.""" + spec = _make_speculator(min_n=2, max_n=3, k=2) + drafts = _propose( + spec, + [ + [9, 1, 2, 3, 8, 1, 2, 3], # 3-gram (1,2,3) at idx 1 → [8, 1] + [7, 1, 2, 9, 1, 2], # 2-gram (1,2) at idx 1 → [9, 1] + ], + ) + assert drafts[0] == [8, 1] + assert drafts[1] == [9, 1] + + +def test_min_n_eq_1(): + """min_n=max_n=1 — single-token n-grams always match if context > 1.""" + spec = _make_speculator(min_n=1, max_n=1, k=2) + drafts = _propose(spec, [[1, 2, 3, 4, 1]]) + assert drafts == [[2, 3]] + + +def test_noncontiguous_idx_mapping(): + """propose() reads token rows in place via idx_mapping (non-contiguous).""" + spec = _make_speculator(min_n=2, max_n=2, k=2) + drafts = _propose( + spec, + [[7, 8, 9, 7, 8], [1, 2, 3, 1, 2]], + slots=[3, 0], + ) + assert drafts == [[9, 7], [3, 1]] + + +def test_construction_validates_speculative_config(): + spec = _make_speculator(min_n=2, max_n=3, k=2) + assert spec.min_n == 2 + assert spec.max_n == 3 + assert spec.num_speculative_steps == 2 + # No-op hooks must not raise. + spec.init_cudagraph_manager(None) + spec.capture() diff --git a/tests/v1/streaming_input/test_scheduler_streaming.py b/tests/v1/streaming_input/test_scheduler_streaming.py index bfef3dd33c..1f97d8a1c0 100644 --- a/tests/v1/streaming_input/test_scheduler_streaming.py +++ b/tests/v1/streaming_input/test_scheduler_streaming.py @@ -7,6 +7,7 @@ from unittest.mock import MagicMock import numpy as np import torch +import vllm.v1.core.kv_cache_utils as kv_cache_utils from vllm.config import DeviceConfig, VllmConfig from vllm.multimodal.inputs import ( MultiModalFeatureSpec, @@ -14,6 +15,12 @@ from vllm.multimodal.inputs import ( PlaceholderRange, ) from vllm.sampling_params import SamplingParams +from vllm.utils.hashing import sha256 +from vllm.v1.core.kv_cache_utils import ( + get_request_block_hasher, + hash_block_tokens, + init_none_hash, +) from vllm.v1.core.sched.scheduler import Scheduler from vllm.v1.engine import FinishReason from vllm.v1.kv_cache_interface import ( @@ -365,6 +372,48 @@ class TestStreamingScheduler(unittest.TestCase): num_new_tokens = session.num_tokens - session.num_computed_tokens assert num_new_tokens == 2 + def test_update_request_as_session_drops_stale_block_hashes(self): + """A discarded sampled token must not stay fingerprinted in + block_hashes (#49377).""" + init_none_hash(sha256) + scheduler = create_scheduler() + hash_block_size = scheduler.hash_block_size + + # hash_block_size - 1 prompt tokens, so the discarded sampled token + # completes hash block 0. + prompt = list(range(1, hash_block_size)) + discarded_token = 999 + session = Request( + request_id="session", + prompt_token_ids=list(prompt), + sampling_params=SamplingParams(max_tokens=16), + pooling_params=None, + block_hasher=get_request_block_hasher(hash_block_size, sha256), + resumable=True, + ) + session.append_output_token_ids(discarded_token) + assert len(session.block_hashes) == 1 + stale_hash = session.block_hashes[0] + session.num_computed_tokens = len(prompt) + + next_token = 42 + update = StreamingUpdate.from_request( + DummyRequest(request_id="session", prompt_token_ids=[next_token]) + ) + scheduler._update_request_as_session(session, update) + + assert list(session.all_token_ids) == prompt + [next_token] + assert len(session.block_hashes) == 1 + expected = hash_block_tokens( + sha256, kv_cache_utils.NONE_HASH, tuple(prompt + [next_token]) + ) + assert session.block_hashes[0] == expected, ( + "block_hashes[0] does not identify the current token sequence" + ) + assert session.block_hashes[0] != stale_hash, ( + "block_hashes[0] still fingerprints the discarded sampled token" + ) + def test_streaming_e2e_lifecycle(self): """Comprehensive integration test covering complete streaming request lifecycle including scheduler state management and aliasing bug prevention. diff --git a/tests/v1/structured_output/test_backend_guidance.py b/tests/v1/structured_output/test_backend_guidance.py index 055ac9c364..59d1abf468 100644 --- a/tests/v1/structured_output/test_backend_guidance.py +++ b/tests/v1/structured_output/test_backend_guidance.py @@ -1,5 +1,6 @@ # SPDX-License-Identifier: Apache-2.0 # SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import json import time from concurrent.futures import Future @@ -14,7 +15,10 @@ from vllm.sampling_params import SamplingParams, StructuredOutputsParams from vllm.tokenizers import get_tokenizer from vllm.v1.request import Request from vllm.v1.structured_output import StructuredOutputManager -from vllm.v1.structured_output.backend_guidance import GuidanceBackend +from vllm.v1.structured_output.backend_guidance import ( + GuidanceBackend, + process_for_additional_properties, +) from vllm.v1.structured_output.backend_types import StructuredOutputOptions TOKENIZER = "openai-community/gpt2" @@ -236,3 +240,82 @@ def test_mistral_tokenizer_compile_grammar( grammar = backend.compile_grammar(request_type, grammar_spec) assert grammar is not None assert not grammar.is_terminated() + + +def test_process_for_additional_properties_preserves_literals() -> None: + """Literal values under const/enum/default/examples must not be rewritten + when they contain a 'properties' or 'patternProperties' key (#58695).""" + feature = { + "type": "Feature", + "geometry": None, + "properties": {"name": "HQ"}, + "patternProperties": {"^x": "y"}, + } + schema = { + "type": "object", + "properties": { + "const_field": {"const": feature}, + "enum_field": {"enum": [feature]}, + "default_field": {"type": "object", "default": feature}, + "examples_field": {"type": "object", "examples": [feature]}, + "properties": {"type": "string"}, + "const": { + "type": "object", + "properties": {"id": {"type": "integer"}}, + }, + }, + "$defs": { + "item": { + "type": "object", + "properties": {"label": {"type": "string"}}, + } + }, + } + + processed = process_for_additional_properties(schema) + + assert processed["additionalProperties"] is False + assert processed["$defs"]["item"]["additionalProperties"] is False + assert processed["properties"]["const"]["additionalProperties"] is False + assert "additionalProperties" not in processed["properties"] + assert processed["properties"]["const_field"]["const"] == feature + assert processed["properties"]["enum_field"]["enum"] == [feature] + assert processed["properties"]["default_field"]["default"] == feature + assert processed["properties"]["examples_field"]["examples"] == [feature] + + +@pytest.mark.parametrize("keyword", ["const", "enum"]) +def test_disable_additional_properties_accepts_literal_with_properties_key( + keyword: str, +) -> None: + """GuidanceBackend with disable_additional_properties=True must accept the + original literal value under const/enum and reject injected + additionalProperties (#58695).""" + feature = {"type": "Feature", "geometry": None, "properties": {"name": "HQ"}} + rewritten = {**feature, "additionalProperties": False} + schema = {keyword: feature if keyword == "const" else [feature]} + + vllm_config = VllmConfig( + structured_outputs_config=StructuredOutputsConfig( + backend="guidance", + disable_additional_properties=True, + ) + ) + tokenizer = AutoTokenizer.from_pretrained(TOKENIZER) + backend = GuidanceBackend( + vllm_config, + tokenizer=tokenizer, + vocab_size=50257, + ) + grammar = backend.compile_grammar( + StructuredOutputOptions.JSON, + json.dumps(schema), + ) + + valid_tokens = tokenizer.encode(json.dumps(feature)) + [tokenizer.eos_token_id] + rewritten_tokens = tokenizer.encode(json.dumps(rewritten)) + [ + tokenizer.eos_token_id + ] + + assert grammar.validate_tokens(valid_tokens) == valid_tokens + assert grammar.validate_tokens(rewritten_tokens) != rewritten_tokens diff --git a/tests/v1/structured_output/test_backend_outlines.py b/tests/v1/structured_output/test_backend_outlines.py new file mode 100644 index 0000000000..abff1e7f51 --- /dev/null +++ b/tests/v1/structured_output/test_backend_outlines.py @@ -0,0 +1,73 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest + +from vllm.exceptions import VLLMValidationError +from vllm.v1.structured_output.backend_outlines import validate_regex_is_buildable + +pytestmark = pytest.mark.cpu_test + + +# ================================================ +# Test validate_regex_is_buildable functionality +# ================================================ + + +class TestValidateRegexIsBuildable: + class TestPR60359Regressions: + """This class implements regression tests for PR #60359. + Initial bug report issue #60344. + validate_regex_is_buildable misses \b and backref inside groups + So, we check here that these features are rejected by us. + """ + + @pytest.mark.parametrize( + "regex", + [ + # top-level word boundary \b + r"a\b", + # \b inside capturing group + r"(a\b)", + # word boundary \b in nested group + r"((a\b))", + # top-level backref + r"(a)\1", + # backref in nested group + r"((a)\2)", + # positive lookahead + r"(a(?=b))", + # inline flag group + word boundary + r"(?i:a\b)", + # counted repeat over a group containing \b + r"(?:(a\b)){2}", + # lazy repeat + word boundary + r"(?:a\b)*?", + # possessive repeat + word boundary + r"(?:a\b)*+", + # atomic group + word boundary + r"(?>a\b)", + ], + ) + def test_unsupported_regex(self, regex): + with pytest.raises(VLLMValidationError): + validate_regex_is_buildable(regex) + + @pytest.mark.parametrize( + "regex", + [ + # [\b] is character class, not word-boundary \b + r"[\b]", + r"(a[\b]c)", + # escaped backslash + "b", not word-boundary \b + r"a\\b", + r"(a\\bc)", + # no \b or backref here, must be ok + r"(abc)", + # BRANCH and MAX_REPEAT must be ok + r"(a|b)+c", + r"(?:ab)*?c", + ], + ) + def test_supported_regex(self, regex): + validate_regex_is_buildable(regex) diff --git a/tests/v1/structured_output/test_backend_xgrammar.py b/tests/v1/structured_output/test_backend_xgrammar.py new file mode 100644 index 0000000000..9024b614dd --- /dev/null +++ b/tests/v1/structured_output/test_backend_xgrammar.py @@ -0,0 +1,611 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + + +from typing import ClassVar + +import pytest +from transformers import AutoTokenizer, PreTrainedTokenizerBase +from xgrammar import Grammar +from xgrammar.testing import _is_grammar_accept_string + +from vllm.config import StructuredOutputsConfig, VllmConfig +from vllm.sampling_params import SamplingParams, StructuredOutputsParams +from vllm.v1.structured_output.backend_types import StructuredOutputOptions +from vllm.v1.structured_output.backend_xgrammar import ( + XgrammarBackend, + has_xgrammar_unsupported_json_features, + validate_xgrammar_grammar, +) +from vllm.v1.structured_output.utils import choice_as_grammar + +pytestmark = pytest.mark.cpu_test + + +def grammar_accepts(schema: dict, text: str) -> bool: + return _is_grammar_accept_string(Grammar.from_json_schema(schema), text) + + +# ================================================ +# Unsupported schemas +# ================================================ + + +@pytest.fixture +def unsupported_multipleOf_schemas(): + return [ + {"type": "number", "multipleOf": 120}, + {"Even": {"type": "number", "multipleOf": 2}}, + {"type": "integer", "multipleOf": 120}, + {"Even": {"type": "integer", "multipleOf": 2}}, + ] + + +@pytest.fixture +def unsupported_array_schemas(): + return [ + # array + some constraints is unsupported by xgrammar + # Tracking issue: https://github.com/mlc-ai/xgrammar/issues/968 + {"type": "array", "uniqueItems": True}, + {"type": "array", "contains": {"type": "string"}}, + {"type": "array", "minContains": 1}, + {"type": "array", "maxContains": 5}, + ] + + +@pytest.fixture +def unsupported_string_schemas(): + return [ + # ======================================================== + # string + format is unsupported by xgrammar + # Tracking issue: https://github.com/mlc-ai/xgrammar/issues/967 + {"type": "string", "format": "non_existing_format"}, + # ======================================================== + # + # ======================================================== + # string + format/pattern + length constraint is unsupported by xgrammar + # Tracking issue: https://github.com/mlc-ai/xgrammar/issues/966 + # + # pattern/format is compiled but length bounds are silently dropped, + # so the combination must be rejected instead of producing quietly + # wrong output + {"type": "string", "pattern": "^a+$", "maxLength": 2}, + {"type": "string", "pattern": "^a+$", "minLength": 3}, + {"type": "string", "format": "email", "maxLength": 10}, + # ======================================================== + ] + + +@pytest.fixture +def unsupported_propertyNames_combinations(): + return [ + # ======================================================== + # propertyNames + patternProperties is unsupported by xgrammar + # Tracking issue: https://github.com/mlc-ai/xgrammar/issues/959 + # + # xgrammar drops propertyNames whenever patternProperties is present, + # so "grade_12" matches the pattern and escapes the name constraint + { + "type": "object", + "propertyNames": {"pattern": "^grade_[0-9]$"}, # does NOT match "grade_12" + "patternProperties": {"^grade_[0-9]+$": {"type": "integer"}}, + }, + # ======================================================== + # + # ======================================================== + # propertyNames + maxLength is unsupported by xgrammar + # Tracking issue: https://github.com/mlc-ai/xgrammar/issues/960 + { + "type": "object", + "propertyNames": {"pattern": "^a+$", "maxLength": 2}, + }, + # ======================================================== + # + # ======================================================== + # propertyNames + properties is unsupported by xgrammar + # Tracking issue: https://github.com/mlc-ai/xgrammar/issues/958 + { + "type": "object", + "propertyNames": {"pattern": "^[a-z]+$"}, + "properties": {"Bad": {"type": "integer"}}, + "required": ["Bad"], + }, + { + "type": "object", + "propertyNames": {"enum": ["good"]}, + "properties": {"Bad": {"type": "integer"}}, + }, + # ======================================================== + # + # ======================================================== + # propertyNames + unevaluatedProperties is unsupported by xgrammar + # Tracking issue: https://github.com/mlc-ai/xgrammar/issues/961 + # + # xgrammar's propertyNames-only branch falls back to an unconstrained + # value type, so it drops unevaluatedProperties whether it restricts + # values (a schema) or forbids them (false). + { + "type": "object", + "propertyNames": {"pattern": "^[a-z]+$"}, + "unevaluatedProperties": False, + }, + # also keep in mind that unevaluatedProperties can be a schema + { + "type": "object", + "propertyNames": {"pattern": "^[a-z]+$"}, + "unevaluatedProperties": {"type": "integer"}, + }, + # ======================================================== + ] + + +@pytest.fixture +def unsupported_patternProperties_combinations(): + return [ + # ======================================================== + # patternProperties + properties is unsupported by xgrammar + # Tracking issue: https://github.com/mlc-ai/xgrammar/issues/964 + # + # JSON Schema requires a property matched by both `properties` and + # `patternProperties` to satisfy both (conjunction), but xgrammar + # compiles them as alternative branches, so satisfying either one is + # enough. + { + "type": "object", + "patternProperties": {"^x$": {"type": "integer"}}, + "properties": {"x": {"type": "string"}}, + }, + # ======================================================== + # + # ======================================================== + # patternProperties + patternProperties is unsupported by xgrammar + # Tracking issue: https://github.com/mlc-ai/xgrammar/issues/965 + # + # The same alternative-branches problem applies to overlapping + # patternProperties patterns: a key matching both patterns only has to + # satisfy one of them. + { + "type": "object", + "patternProperties": { + "^a[a-z]*$": {"type": "string"}, + "^[a-z]*z$": {"type": "integer"}, + }, + }, + # ======================================================== + ] + + +@pytest.fixture +def unsupported_multibranch_allof(): + """Regression tests for issue #56556 + [Bug]: JSON schema with multiple allOf branches is silently ignored + by the xgrammar structured-output backend #56556 + + Xgrammar currently does NOT support multi-branch allOf. + Bug reported to xgrammar team for tracking progress on resolving: + https://github.com/mlc-ai/xgrammar/issues/937 + """ + return [ + { + "allOf": [ + {"type": "string"}, + {"enum": ["yes", "no"]}, + ] + }, + # Multi-branch test case with > 2 branches: + { + "allOf": [ + {"type": "string"}, + {"minLength": 2}, + {"maxLength": 10}, + ] + }, + { + "allOf": [ + {"type": "integer"}, + {"minimum": 0}, + ] + }, + # Non root multi-branch allOf: + { + "type": "object", + "properties": { + "is_this_sparta": { + "allOf": [ + {"type": "string"}, + {"enum": ["yes", "no"]}, + ], + }, + }, + }, + ] + + +@pytest.fixture +def unsupported_vllm_issue_56556_schema(): + """Bug repro from vLLM issue #56556""" + return [ + { + "$defs": { + "Base": { + "type": "object", + "properties": {"x": {"type": "integer", "minimum": 10}}, + "required": ["x"], + } + }, + "allOf": [ + {"$ref": "#/$defs/Base"}, + { + "type": "object", + "properties": {"y": {"type": "string"}}, + "required": ["y"], + }, + ], + }, + ] + + +# ================================================ +# Supported schemas +# ================================================ + + +@pytest.fixture +def supported_frankenstein_schema(): + # IMPORTANT(arpera): + # Do NOT add more keywords here! This schema is overcrowded enough that a new + # keyword can end up having no effect on the compiled grammar while the + # test still passes. Give a new keyword its own fixture and named test. + return { + "type": "object", + "properties": { + "name": {"type": "string"}, + "age": {"type": "integer"}, + "email": {"type": "string", "format": "email"}, + "status": {"type": "string"}, + "scores": {"type": "array", "items": {"type": "number"}}, + "car_type": {"type": "string", "enum": ["sedan", "suv", "truck"]}, + "car_brand": {"type": "string", "pattern": "^[a-zA-Z]+$"}, + "short_description": {"type": "string", "maxLength": 50}, + "mileage": {"type": "number", "minimum": 0, "maximum": 1000000}, + "model_year": { + "type": "integer", + "exclusiveMinimum": 1900, + "exclusiveMaximum": 2100, + }, + "long_description": {"type": "string", "minLength": 50, "maxLength": 2000}, + "address": { + "type": "object", + "properties": { + "street": {"type": "string"}, + "city": {"type": "string"}, + }, + }, + }, + "minProperties": 1, + "maxProperties": 100, + } + + +@pytest.fixture +def pattern_properties_schema(): + return { + "type": "object", + "patternProperties": {"^grade_[0-9]+$": {"type": "integer"}}, + } + + +@pytest.fixture +def property_names_schema(): + return {"type": "object", "propertyNames": {"pattern": "^[a-z_]+$"}} + + +@pytest.fixture +def property_names_with_additional_properties_schema(): + """This combination propertyNames + additionalProperties + was unsupported some time ago by xgrammar. + So, this test case was in unsupported category before. + But this xgrammar's PR + https://github.com/mlc-ai/xgrammar/pull/836 + added support of propertyNames + additionalProperties + This fix was released in xgrammar v0.2.7 + which vLLM pins in PR #57272 + """ + return { + "type": "object", + "propertyNames": {"pattern": "^[a-z_]+$"}, + "additionalProperties": {"type": "integer"}, + } + + +@pytest.fixture +def supported_allof_anyof_and_oneof(): + """Xgrammar DO support some variants of anyOf, allOf, oneOf + This test case was made during working on covering + unsupported multi-branch allOf to make sure the bug + is NOT present in other combinators. + See for more context in this file test `unsupported_multibranch_allof` + """ + return [ + # single-branch allOf: + { + "allOf": [ + {"type": "string"}, + ], + }, + # Corner cases: empty allOf, this is a valid schema + {"allOf": []}, + # multi-branch allOf: + # is NOT supported yet, see vLLM issue #56556 + # single-branch anyOf: + { + "anyOf": [], + }, + # multi-branch anyOf: + { + "anyOf": [ + {"type": "string"}, + {"type": "integer"}, + ], + }, + # single-branch oneOf: + { + "oneOf": [ + {"type": "integer", "minimum": 0, "maximum": 100}, + ], + }, + # multi-branch oneOf: + { + "oneOf": [ + {"type": "integer", "minimum": 0, "maximum": 100}, + {"type": "string", "enum": ["auto", "none"]}, + ], + }, + # Corner case: + # "allOf" is a property name, not allOf combinator keyword + {"type": "object", "properties": {"allOf": {"type": "string"}}}, + ] + + +# ================================================ +# Test has_xgrammar_unsupported_json_features functionality +# ================================================ + + +class TestHasXGrammarUnsupportedJsonFeatures: + @pytest.mark.parametrize( + "schema_type", + [ + "unsupported_string_schemas", + "unsupported_multipleOf_schemas", + "unsupported_array_schemas", + "unsupported_propertyNames_combinations", + "unsupported_patternProperties_combinations", + # vLLM issue #56556 + "unsupported_multibranch_allof", + "unsupported_vllm_issue_56556_schema", + ], + ) + def test_unsupported_json_features(self, schema_type, request): + schemas = request.getfixturevalue(schema_type) + for schema in schemas: + assert has_xgrammar_unsupported_json_features(schema), ( + f"Schema should be unsupported: {schema}" + ) + + @pytest.mark.parametrize( + "schema_type", + [ + "supported_frankenstein_schema", + "pattern_properties_schema", + "property_names_schema", + "property_names_with_additional_properties_schema", + # Additional test cases implemented during work on + # vLLM issue #56556 + "supported_allof_anyof_and_oneof", + ], + ) + def test_supported_json_features(self, schema_type, request): + schemas = request.getfixturevalue(schema_type) + if not isinstance(schemas, list): + schemas = [schemas] + for schema in schemas: + assert not has_xgrammar_unsupported_json_features(schema), ( + f"Schema should be supported: {schema}" + ) + + class TestPR48416Regressions: + @pytest.mark.parametrize( + "schema", + [ + {"type": ["number"], "multipleOf": 3}, + { + "type": ["string", "null"], + "pattern": "^a+$", + "maxLength": 2, + }, + { + "type": ["array", "null"], + "items": {"type": "integer"}, + "uniqueItems": True, + }, + { + "type": ["object", "null"], + "properties": {"Bad": {"type": "integer"}}, + "propertyNames": {"pattern": "^[a-z]+$"}, + }, + ], + ) + def test_list_type_does_not_bypass_unsupported_feature_check(self, schema): + assert has_xgrammar_unsupported_json_features(schema) + + @pytest.mark.parametrize( + "schema", + [ + {"type": ["string", "null"]}, + {"type": ["integer", "string"]}, + {"type": ["array", "null"], "items": {"type": "string"}}, + { + "type": ["object", "null"], + "propertyNames": {"pattern": "^[a-z_]+$"}, + }, + { + "type": ["object", "null"], + "patternProperties": {"^S": {"type": "string"}}, + }, + ], + ) + def test_supported_list_type_json_features(self, schema): + assert not has_xgrammar_unsupported_json_features(schema) + + +class TestIsGrammarAcceptString: + class TestPR42904Support: + def test_property_names_constrains_keys(self, property_names_schema): + assert grammar_accepts(property_names_schema, '{"score": 5}') + assert grammar_accepts(property_names_schema, '{"score": "seven"}') + assert not grammar_accepts(property_names_schema, '{"Score": 5}') + assert not grammar_accepts(property_names_schema, '{"score_1": 5}') + + def test_property_names_keeps_additional_properties_value_schema( + self, property_names_with_additional_properties_schema + ): + schema = property_names_with_additional_properties_schema + assert grammar_accepts(schema, '{"score": 5}') + assert not grammar_accepts(schema, '{"Score": 5}') + assert not grammar_accepts(schema, '{"score": "seven"}') + + def test_pattern_properties_constrains_keys_and_values( + self, pattern_properties_schema + ): + assert grammar_accepts(pattern_properties_schema, '{"grade_1": 5}') + assert not grammar_accepts(pattern_properties_schema, '{"other": 5}') + assert not grammar_accepts(pattern_properties_schema, '{"grade_1": "five"}') + + class TestPR48115Regressions: + @pytest.mark.parametrize("codepoint", [*range(0x20), 0x7F]) + def test_choice_as_grammar_preserves_control_characters(self, codepoint): + choice = f"a{chr(codepoint)}f" + grammar = Grammar.from_ebnf(choice_as_grammar([choice])) + + assert _is_grammar_accept_string(grammar, choice) + assert not _is_grammar_accept_string(grammar, "a") + assert not _is_grammar_accept_string(grammar, "af") + assert not _is_grammar_accept_string(grammar, f"a\\u{codepoint:04x}f") + + @pytest.mark.parametrize( + ("choice", "other"), + [ + (r"a\nf", "a\nf"), + ('a quote " and a backslash \\', 'a quote " and a backslash '), + ("café", "cafe"), + ("日本語", "日本"), + ("😀", "😁"), + ], + ) + def test_choice_as_grammar_preserves_literal_choices(self, choice, other): + grammar = Grammar.from_ebnf(choice_as_grammar(["yes", choice])) + + assert _is_grammar_accept_string(grammar, "yes") + assert _is_grammar_accept_string(grammar, choice) + assert not _is_grammar_accept_string(grammar, other) + assert not _is_grammar_accept_string(grammar, choice + "extra") + + def test_validate_xgrammar_preserves_multiline_choice(self): + structured_outputs = StructuredOutputsParams(choice=["yes", "no\nplease"]) + validate_xgrammar_grammar( + SamplingParams(structured_outputs=structured_outputs) + ) + + assert structured_outputs.choice is None + grammar = Grammar.from_ebnf(structured_outputs.grammar) + assert _is_grammar_accept_string(grammar, "yes") + assert _is_grammar_accept_string(grammar, "no\nplease") + assert not _is_grammar_accept_string(grammar, r"no\nplease") + + +# ================================================ +# Test XgrammarBackend functionality +# ================================================ + + +class TestXgrammarBackend: + class TestPR58067Regressions: + """Here we check behavior logic of our flag + disable_any_whitespace when used with xgrammar backend. + """ + + _TOKENIZER = "openai-community/gpt2" + _VOCAB_SIZE = 50257 + tokenizer: ClassVar[PreTrainedTokenizerBase] + + @classmethod + @pytest.fixture(scope="class", autouse=True) + def _shared_tokenizer(cls): + cls.tokenizer = AutoTokenizer.from_pretrained(cls._TOKENIZER) + + @classmethod + def _backend(cls, *, disable_any_whitespace: bool) -> XgrammarBackend: + vllm_config = VllmConfig( + structured_outputs_config=StructuredOutputsConfig( + backend="xgrammar", + disable_any_whitespace=disable_any_whitespace, + ) + ) + return XgrammarBackend( + vllm_config, tokenizer=cls.tokenizer, vocab_size=cls._VOCAB_SIZE + ) + + @classmethod + def _accepts_json(cls, backend: XgrammarBackend, json_text: str) -> bool: + grammar = backend.compile_grammar(StructuredOutputOptions.JSON_OBJECT, "") + return grammar.accept_tokens("req", cls.tokenizer.encode(json_text)) + + def test_disable_any_whitespace_is_false(self): + """Verify expected behavior of disable_any_whitespace=False""" + backend = self._backend(disable_any_whitespace=False) + + # Check that grammar accepts json with NO space after colon + assert self._accepts_json(backend, '{"no_space_after_me":"yes_ofcourse"}') + + # Check that grammar accepts json with space after colon + assert self._accepts_json(backend, '{"yes_space_after_me": true}') + + # Check that grammar accepts json with NO space after comma + assert self._accepts_json( + backend, + '{"never_space_after_comma":"accepted","good_boy":":3"}', + ) + + # Check that grammar accepts json with space after comma + assert self._accepts_json( + backend, + '{"where_space_after_comma":42, "wow_thats_cool":4242}', + ) + + def test_disable_any_whitespace_is_true(self): + """Verify expected behavior of disable_any_whitespace=True""" + backend = self._backend(disable_any_whitespace=True) + + # Check that grammar accepts json with NO space after colon + assert self._accepts_json(backend, '{"no_space_after_me":"yes_ofcourse"}') + + # Check that grammar rejects json with space after colon + assert not self._accepts_json(backend, '{"yes_space_after_me": true}') + + # FIXME(arpera): + # Comma + separators=(",", ":") is wrong on xgrammar <=0.2.8 + # Seehttps://github.com/mlc-ai/xgrammar/issues/945 + # Re-enable both tests once vLLM pins xgrammar with the fix (>=0.2.9). + # + # # Check that grammar accepts json with NO space after comma + # assert self._accepts_json( + # backend, + # '{"never_space_after_comma":"accepted","good_boy":":3"}', + # ) + # + # # Check that grammar rejects json with space after comma + # assert not self._accepts_json( + # backend, + # '{"where_space_after_comma":42, "wow_thats_cool":4242}', + # ) diff --git a/tests/v1/structured_output/test_structured_output_manager.py b/tests/v1/structured_output/test_structured_output_manager.py index 6082bcd6a7..fbceb3bd07 100644 --- a/tests/v1/structured_output/test_structured_output_manager.py +++ b/tests/v1/structured_output/test_structured_output_manager.py @@ -11,6 +11,8 @@ from vllm.config import DeviceConfig, StructuredOutputsConfig, VllmConfig from vllm.config.model import ModelConfig from vllm.config.speculative import SpeculativeConfig from vllm.config.structured_outputs import StructuredOutputsBackend +from vllm.parser.engine.adapters import ParserEngineReasoningAdapter +from vllm.parser.engine.parser_engine import ReasoningEnd from vllm.sampling_params import SamplingParams, StructuredOutputsParams from vllm.v1.request import Request from vllm.v1.structured_output import StructuredOutputManager @@ -18,6 +20,7 @@ from vllm.v1.structured_output.backend_outlines import OutlinesGrammar TOKENIZER = "gpt2" THINK_END = "\n" # reasoning-end marker (single GPT-2 token) +IMPLICIT_THINK_END = "z" # accepted in strings, rejected at the object start EOS = "<|eos|>" # resolved to tokenizer.eos_token_id JSON_SCHEMA = '{"type": "object"}' BACKENDS = ("xgrammar", "guidance") @@ -43,6 +46,7 @@ class FlowCase: prefix: str = "" expected_validated: tuple[str, ...] | None = None reasoning_ended: bool | None = False + engine_reasoner: bool = False xfail_guidance: str | None = None @@ -59,6 +63,23 @@ class MockReasoner: return self.is_reasoning_end(delta_ids) +class MockEngineReasoner(ParserEngineReasoningAdapter): + """Ends reasoning implicitly on ``marker``, which is also content.""" + + def __init__(self, tokenizer, marker: int): + self.marker = marker + + @property + def reasoning_end_token_ids(self): + return frozenset({self.marker}) + + def find_reasoning_end(self, token_ids): + ids = list(token_ids) + if self.marker in ids: + return ReasoningEnd(ids.index(self.marker), True) + return ReasoningEnd(len(ids), False) + + def _single_token(tokenizer, text: str) -> int: token_ids = tokenizer.encode(text) assert len(token_ids) == 1, (text, token_ids) @@ -331,6 +352,28 @@ FLOW_CASES = [ ), id="becomes_active_terminates", ), + pytest.param( + FlowCase( + prefix='{"a": "', + raw_drafts=(IMPLICIT_THINK_END,), + expected_row_pattern="UC", + expected_reasoning=True, + expect_terminated=False, + engine_reasoner=True, + ), + id="implicit_end_accepted", + ), + pytest.param( + FlowCase( + raw_drafts=(" ", IMPLICIT_THINK_END, "{", "}"), + expected_validated=(" ",), + expected_row_pattern="UUUUU", + expected_reasoning=False, + expect_terminated=False, + engine_reasoner=True, + ), + id="implicit_end_rejected", + ), ] @@ -344,7 +387,8 @@ def test_real_flow( if backend == "guidance" and case.xfail_guidance: pytest.xfail(case.xfail_guidance) - reasoner_kwargs = {"marker": _single_token(tokenizer, THINK_END)} + marker = IMPLICIT_THINK_END if case.engine_reasoner else THINK_END + reasoner_kwargs = {"marker": _single_token(tokenizer, marker)} manager, request = _build_harness( tokenizer, backend, @@ -352,6 +396,8 @@ def test_real_flow( reasoning_ended=case.reasoning_ended, reasoning_parser_kwargs=reasoner_kwargs, ) + if case.engine_reasoner: + manager.reasoner_cls = MockEngineReasoner raw_drafts = _to_token_ids(tokenizer, case.raw_drafts) expected_texts = ( diff --git a/tests/v1/structured_output/test_utils.py b/tests/v1/structured_output/test_utils.py deleted file mode 100644 index d3187c428c..0000000000 --- a/tests/v1/structured_output/test_utils.py +++ /dev/null @@ -1,325 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# SPDX-FileCopyrightText: Copyright contributors to the vLLM project - -import pytest -from xgrammar import Grammar -from xgrammar.testing import _is_grammar_accept_string - -from vllm.sampling_params import SamplingParams, StructuredOutputsParams -from vllm.v1.structured_output.backend_xgrammar import ( - has_xgrammar_unsupported_json_features, - validate_xgrammar_grammar, -) -from vllm.v1.structured_output.utils import choice_as_grammar - -pytestmark = pytest.mark.cpu_test - - -def grammar_accepts(schema: dict, text: str) -> bool: - return _is_grammar_accept_string(Grammar.from_json_schema(schema), text) - - -@pytest.fixture -def unsupported_string_schemas(): - return [ - {"type": "string", "format": "non_existing_format"}, - # TODO(arpera): - # pattern/format is compiled but length bounds are silently dropped, - # so the combination must be rejected instead of producing quietly - # wrong output - # https://github.com/mlc-ai/xgrammar/issues/749 - {"type": "string", "pattern": "^a+$", "maxLength": 2}, - {"type": "string", "pattern": "^a+$", "minLength": 3}, - {"type": "string", "format": "email", "maxLength": 10}, - ] - - -@pytest.fixture -def unsupported_integer_schemas(): - return [ - {"type": "integer", "multipleOf": 120}, - ] - - -@pytest.fixture -def unsupported_number_schemas(): - return [ - {"type": "number", "multipleOf": 120}, - ] - - -@pytest.fixture -def unsupported_array_schemas(): - return [ - {"type": "array", "uniqueItems": True}, - {"type": "array", "contains": {"type": "string"}}, - {"type": "array", "minContains": 1}, - {"type": "array", "maxContains": 5}, - ] - - -@pytest.fixture -def unsupported_property_names_combinations(): - return [ - # TODO(arpera): - # this case is not covered by xgrammar, so we should report this bug to xgrammar - # xgrammar drops propertyNames whenever patternProperties is present, - # so "grade_12" matches the pattern and escapes the name constraint - { - "type": "object", - "patternProperties": {"^grade_[0-9]+$": {"type": "integer"}}, - "propertyNames": {"pattern": "^grade_[0-9]$"}, # does NOT match "grade_12" - }, - # propertyNames makes xgrammar discard the sibling additionalProperties - # value schema: https://github.com/mlc-ai/xgrammar/issues/826 - { - "type": "object", - "propertyNames": {"pattern": "^[a-z]+$"}, - "additionalProperties": {"type": "integer"}, - }, - # propertyNames is a string schema that conventionally omits "type", so - # it escapes the string check while xgrammar drops its length bound all - # the same: https://github.com/mlc-ai/xgrammar/issues/749 - { - "type": "object", - "propertyNames": {"pattern": "^a+$", "maxLength": 2}, - }, - # xgrammar emits named `properties` separately from `propertyNames` and - # only applies `propertyNames` in its additional-properties branch, so a - # key declared in `properties` escapes the name constraint regardless of - # what `propertyNames` contains. - { - "type": "object", - "properties": {"Bad": {"type": "integer"}}, - "required": ["Bad"], - "propertyNames": {"pattern": "^[a-z]+$"}, - }, - { - "type": "object", - "properties": {"Bad": {"type": "integer"}}, - "propertyNames": {"enum": ["good"]}, - }, - # xgrammar's propertyNames-only branch falls back to an unconstrained - # value type, so it drops unevaluatedProperties whether it restricts - # values (a schema) or forbids them (false). - { - "type": "object", - "propertyNames": {"pattern": "^[a-z]+$"}, - "unevaluatedProperties": {"type": "integer"}, - }, - { - "type": "object", - "propertyNames": {"pattern": "^[a-z]+$"}, - "unevaluatedProperties": False, - }, - ] - - -@pytest.fixture -def unsupported_pattern_properties_combinations(): - return [ - # JSON Schema requires a property matched by both `properties` and - # `patternProperties` to satisfy both (conjunction), but xgrammar - # compiles them as alternative branches, so satisfying either one is - # enough. - { - "type": "object", - "properties": {"x": {"type": "string"}}, - "patternProperties": {"^x$": {"type": "integer"}}, - }, - # The same alternative-branches problem applies to overlapping - # patternProperties patterns: a key matching both patterns only has to - # satisfy one of them. - { - "type": "object", - "patternProperties": { - "^a[a-z]*$": {"type": "string"}, - "^[a-z]*z$": {"type": "integer"}, - }, - }, - ] - - -@pytest.fixture -def supported_frankenstein_schema(): - # IMPORTANT(arpera): - # Do NOT add more keywords here! This schema is overcrowded enough that a new - # keyword can end up having no effect on the compiled grammar while the - # test still passes. Give a new keyword its own fixture and named test. - return { - "type": "object", - "properties": { - "name": {"type": "string"}, - "age": {"type": "integer"}, - "email": {"type": "string", "format": "email"}, - "status": {"type": "string"}, - "scores": {"type": "array", "items": {"type": "number"}}, - "car_type": {"type": "string", "enum": ["sedan", "suv", "truck"]}, - "car_brand": {"type": "string", "pattern": "^[a-zA-Z]+$"}, - "short_description": {"type": "string", "maxLength": 50}, - "mileage": {"type": "number", "minimum": 0, "maximum": 1000000}, - "model_year": { - "type": "integer", - "exclusiveMinimum": 1900, - "exclusiveMaximum": 2100, - }, - "long_description": {"type": "string", "minLength": 50, "maxLength": 2000}, - "address": { - "type": "object", - "properties": { - "street": {"type": "string"}, - "city": {"type": "string"}, - }, - }, - }, - "minProperties": 1, - "maxProperties": 100, - } - - -@pytest.fixture -def property_names_schema(): - return {"type": "object", "propertyNames": {"pattern": "^[a-z_]+$"}} - - -@pytest.fixture -def pattern_properties_schema(): - return { - "type": "object", - "patternProperties": {"^grade_[0-9]+$": {"type": "integer"}}, - } - - -class TestHasXGrammarUnsupportedJsonFeatures: - @pytest.mark.parametrize( - "schema_type", - [ - "unsupported_string_schemas", - "unsupported_integer_schemas", - "unsupported_number_schemas", - "unsupported_array_schemas", - "unsupported_property_names_combinations", - "unsupported_pattern_properties_combinations", - ], - ) - def test_unsupported_json_features_by_type(self, schema_type, request): - schemas = request.getfixturevalue(schema_type) - for schema in schemas: - assert has_xgrammar_unsupported_json_features(schema), ( - f"Schema should be unsupported: {schema}" - ) - - @pytest.mark.parametrize( - "schema_type", - [ - "supported_frankenstein_schema", - "pattern_properties_schema", - "property_names_schema", - ], - ) - def test_supported_json_features(self, schema_type, request): - schema = request.getfixturevalue(schema_type) - assert not has_xgrammar_unsupported_json_features(schema), ( - f"Schema should be supported: {schema}" - ) - - class TestPR48416Regressions: - @pytest.mark.parametrize( - "schema", - [ - {"type": ["number"], "multipleOf": 3}, - { - "type": ["string", "null"], - "pattern": "^a+$", - "maxLength": 2, - }, - { - "type": ["array", "null"], - "items": {"type": "integer"}, - "uniqueItems": True, - }, - { - "type": ["object", "null"], - "properties": {"Bad": {"type": "integer"}}, - "propertyNames": {"pattern": "^[a-z]+$"}, - }, - ], - ) - def test_list_type_does_not_bypass_unsupported_feature_check(self, schema): - assert has_xgrammar_unsupported_json_features(schema) - - @pytest.mark.parametrize( - "schema", - [ - {"type": ["string", "null"]}, - {"type": ["integer", "string"]}, - {"type": ["array", "null"], "items": {"type": "string"}}, - { - "type": ["object", "null"], - "propertyNames": {"pattern": "^[a-z_]+$"}, - }, - { - "type": ["object", "null"], - "patternProperties": {"^S": {"type": "string"}}, - }, - ], - ) - def test_supported_list_type_json_features(self, schema): - assert not has_xgrammar_unsupported_json_features(schema) - - -class TestIsGrammarAcceptString: - class TestPR42904Support: - def test_property_names_constrains_keys(self, property_names_schema): - assert grammar_accepts(property_names_schema, '{"score": 5}') - assert grammar_accepts(property_names_schema, '{"score": "seven"}') - assert not grammar_accepts(property_names_schema, '{"Score": 5}') - assert not grammar_accepts(property_names_schema, '{"score_1": 5}') - - def test_pattern_properties_constrains_keys_and_values( - self, pattern_properties_schema - ): - assert grammar_accepts(pattern_properties_schema, '{"grade_1": 5}') - assert not grammar_accepts(pattern_properties_schema, '{"other": 5}') - assert not grammar_accepts(pattern_properties_schema, '{"grade_1": "five"}') - - class TestPR48115Regressions: - @pytest.mark.parametrize("codepoint", [*range(0x20), 0x7F]) - def test_choice_as_grammar_preserves_control_characters(self, codepoint): - choice = f"a{chr(codepoint)}f" - grammar = Grammar.from_ebnf(choice_as_grammar([choice])) - - assert _is_grammar_accept_string(grammar, choice) - assert not _is_grammar_accept_string(grammar, "a") - assert not _is_grammar_accept_string(grammar, "af") - assert not _is_grammar_accept_string(grammar, f"a\\u{codepoint:04x}f") - - @pytest.mark.parametrize( - ("choice", "other"), - [ - (r"a\nf", "a\nf"), - ('a quote " and a backslash \\', 'a quote " and a backslash '), - ("café", "cafe"), - ("日本語", "日本"), - ("😀", "😁"), - ], - ) - def test_choice_as_grammar_preserves_literal_choices(self, choice, other): - grammar = Grammar.from_ebnf(choice_as_grammar(["yes", choice])) - - assert _is_grammar_accept_string(grammar, "yes") - assert _is_grammar_accept_string(grammar, choice) - assert not _is_grammar_accept_string(grammar, other) - assert not _is_grammar_accept_string(grammar, choice + "extra") - - def test_validate_xgrammar_preserves_multiline_choice(self): - structured_outputs = StructuredOutputsParams(choice=["yes", "no\nplease"]) - validate_xgrammar_grammar( - SamplingParams(structured_outputs=structured_outputs) - ) - - assert structured_outputs.choice is None - grammar = Grammar.from_ebnf(structured_outputs.grammar) - assert _is_grammar_accept_string(grammar, "yes") - assert _is_grammar_accept_string(grammar, "no\nplease") - assert not _is_grammar_accept_string(grammar, r"no\nplease") diff --git a/tests/v1/structured_output/test_validation.py b/tests/v1/structured_output/test_validation.py index 767153cac7..cd5526c7b7 100644 --- a/tests/v1/structured_output/test_validation.py +++ b/tests/v1/structured_output/test_validation.py @@ -2,11 +2,18 @@ # SPDX-FileCopyrightText: Copyright contributors to the vLLM project """Request-time validation of structured output requests.""" +import json + import pytest from vllm.config import StructuredOutputsConfig from vllm.exceptions import VLLMClientError, VLLMValidationError -from vllm.sampling_params import SamplingParams, StructuredOutputsParams +from vllm.sampling_params import ( + MAX_STRUCTURED_OUTPUT_JSON_NESTING, + STRUCTURAL_TAG_WRAPPER_NESTING, + SamplingParams, + StructuredOutputsParams, +) pytestmark = pytest.mark.cpu_test @@ -110,6 +117,98 @@ def test_regex_with_nul_byte_rejected(regex): validate_xgrammar_grammar(params) +def _nested_array_schema(levels: int) -> str: + # Built as text: json.dumps on a deeply nested dict would itself recurse. + wrappers = levels - 1 + return ( + '{"type": "array", "items": ' * wrappers + '{"type": "string"}' + "}" * wrappers + ) + + +def _structural_tag(schema: str) -> str: + return ( + '{"type": "structural_tag", "format": {"type": "json_schema", ' + f'"json_schema": {schema}}}}}' + ) + + +@pytest.mark.parametrize( + "structured_outputs", + [ + pytest.param( + {"json": _nested_array_schema(MAX_STRUCTURED_OUTPUT_JSON_NESTING + 1)}, + id="json-str-over-limit", + ), + pytest.param( + {"json": json.loads(_nested_array_schema(2_000))}, + id="json-dict-past-recursion-limit", + ), + pytest.param( + {"json": _nested_array_schema(200_000)}, + id="json-str-past-json-loads-recursion-limit", + ), + pytest.param( + { + "structural_tag": _structural_tag( + _nested_array_schema( + MAX_STRUCTURED_OUTPUT_JSON_NESTING + + STRUCTURAL_TAG_WRAPPER_NESTING + ) + ) + }, + id="structural-tag-over-limit", + ), + ], +) +@pytest.mark.parametrize( + "backend", ["auto", "xgrammar", "guidance", "outlines", "lm-format-enforcer"] +) +def test_deeply_nested_schema_rejected(structured_outputs, backend): + """Converting a schema to a grammar gets steeply more expensive with nesting + and runs on the API server; past the recursion limit it used to escape as a + RecursionError (HTTP 500 with an empty message). Deep nesting must be a + validation error before any backend touches the schema.""" + params = SamplingParams( + structured_outputs=StructuredOutputsParams(**structured_outputs) + ) + with pytest.raises(VLLMValidationError, match="nested too deeply"): + params._validate_structured_outputs( + _StubModelConfig(is_diffusion=False), + StructuredOutputsConfig(backend=backend), + tokenizer=object(), + ) + + +@pytest.mark.parametrize("field", ["json", "structural_tag"]) +def test_xgrammar_validation_rejects_deeply_nested_schema(field): + """Request parsing and tool parsers call validate_xgrammar_grammar directly, + before _validate_structured_outputs, so it must check nesting itself.""" + from vllm.v1.structured_output.backend_xgrammar import validate_xgrammar_grammar + + schema = _nested_array_schema(2_000) + value = _structural_tag(schema) if field == "structural_tag" else schema + params = SamplingParams( + structured_outputs=StructuredOutputsParams(**{field: value}) + ) + with pytest.raises(VLLMValidationError, match="nested too deeply"): + validate_xgrammar_grammar(params) + + +@pytest.mark.parametrize("field", ["json", "structural_tag"]) +def test_schema_at_nesting_limit_accepted(field): + """A structural tag wraps its schema in a few more levels (more for the + tags tool parsers generate), which must not count against the schema.""" + schema = _nested_array_schema(MAX_STRUCTURED_OUTPUT_JSON_NESTING) + value = _structural_tag(schema) if field == "structural_tag" else schema + structured_outputs = StructuredOutputsParams(**{field: value}) + SamplingParams(structured_outputs=structured_outputs)._validate_structured_outputs( + _StubModelConfig(is_diffusion=False), + StructuredOutputsConfig(), + tokenizer=object(), + ) + assert structured_outputs._backend == "xgrammar" + + INVALID_JSON_SCHEMA = {"type": "object", "properties": {"name": {"type": "str"}}} diff --git a/tests/v1/worker/test_attn_utils.py b/tests/v1/worker/test_attn_utils.py index 0e6089d529..9c9c26c3c2 100644 --- a/tests/v1/worker/test_attn_utils.py +++ b/tests/v1/worker/test_attn_utils.py @@ -16,7 +16,7 @@ import torch import vllm.v1.hisparse.binding as attn_utils_module from tests.v1.attention.utils import dense_kv_cache_views -from vllm.config.compilation import CUDAGraphMode +from vllm.config.compilation import CompilationConfig, CUDAGraphMode from vllm.v1.attention.backend import AttentionBackend, AttentionCGSupport, MultipleOf from vllm.v1.core.kv_cache_utils import KVCacheBlockCopy from vllm.v1.hisparse.binding import allocate_hisparse_kv_caches @@ -228,6 +228,47 @@ def test_varlen_cudagraph_unsupported_backend_checks_scoped_bounds(): assert unsupported([[target, never]], config, 1) == ("_DraftBackend", None) +@pytest.mark.parametrize("num_speculative_tokens", [0, 7]) +def test_flashinfer_sparse_full_graphs_exclude_prefill_keep_varlen_decode( + num_speculative_tokens, +): + """Prefill must fall back without disabling adaptive verification graphs.""" + from vllm.v1.attention.backends.mla.flashinfer_mla_sparse import ( + FlashInferMLASparseTRTLLMBackend, + ) + + backend = FlashInferMLASparseTRTLLMBackend + config: Any = SimpleNamespace( + speculative_config=SimpleNamespace( + num_speculative_tokens=num_speculative_tokens, parallel_drafting=False + ), + use_v2_model_runner=True, + ) + spec = MLAAttentionSpec( + block_size=64, num_kv_heads=1, head_size=576, dtype=torch.bfloat16 + ) + group = AttentionGroup(backend, ["target"], spec, 0) + group.metadata_builders = [object.__new__(backend.get_builder_cls())] + groups = [[group]] + support = get_attn_cg_support(groups, config) + compilation = CompilationConfig(cudagraph_mode=CUDAGraphMode.FULL) + mode = compilation.resolve_cudagraph_mode_and_sizes( + support.min_cg_support, + support.min_cg_attn_backend, + uniform_decode_query_len=1 + num_speculative_tokens, + use_v2_model_runner=True, + ) + assert mode.mixed_mode() != CUDAGraphMode.FULL + assert mode.decode_mode() == CUDAGraphMode.FULL + assert get_query_lens_mismatch_unsupported_backend(groups) is None + unsupported = attn_utils.get_varlen_cudagraph_unsupported_backend + assert unsupported(groups, config, 1 + num_speculative_tokens) is None + assert unsupported(groups, config, 2 + num_speculative_tokens) == ( + backend.__name__, + 1 + num_speculative_tokens, + ) + + def test_get_kv_sharing_fast_prefill_eligible_layers(monkeypatch: pytest.MonkeyPatch): """Fast prefill applies to the contiguous suffix of KV-sharing layers. @@ -393,7 +434,7 @@ def test_init_hisparse_rolls_back_shared_region(monkeypatch, failure_phase): cache_config=SimpleNamespace( get_resolved_kv_cache_layout=lambda: KVCacheLayout.BLHNC ), - scheduler_config=SimpleNamespace(max_num_seqs=1, max_num_batched_tokens=1), + scheduler_config=SimpleNamespace(max_num_seqs=1), ) def allocate(*args): diff --git a/tests/v1/worker/test_encoder_runner.py b/tests/v1/worker/test_encoder_runner.py index 98aaa568f0..69afc6ea93 100644 --- a/tests/v1/worker/test_encoder_runner.py +++ b/tests/v1/worker/test_encoder_runner.py @@ -78,7 +78,7 @@ def test_encoder_graph_unsupported_modality_uses_eager_output(): "vllm.v1.worker.gpu.mm.encoder_runner.group_and_batch_mm_kwargs", return_value=[("audio", 1, {"audio_values": torch.zeros(2)})], ): - result = runner.execute_mm_encoder([]) + result = runner.execute_mm_encoder([("audio", MagicMock())]) assert len(result) == 1 assert result[0] is output manager.execute.assert_not_called() @@ -290,6 +290,21 @@ def test_execute_mm_encoder_caches_outputs_without_gathering(): state.encoder_runner.gather_mm_embeddings.assert_not_called() +def test_execute_mm_encoder_sorts_by_modality(): + """Items are encoded sorted by modality; outputs keep the input order.""" + mm_kwargs = [("video", 0), ("image", 1), ("video", 2), ("image", 3)] + runner = _make_runner([], []) + runner.model = MagicMock() + runner.model.embed_multimodal.side_effect = lambda x: [torch.full((1, 1), x)] + with patch( + "vllm.v1.worker.gpu.mm.encoder_runner.group_and_batch_mm_kwargs", + side_effect=lambda items, **_: [(m, 1, {"x": x}) for m, x in items], + ) as group: + outputs = runner.execute_mm_encoder(mm_kwargs) + assert group.call_args.args[0] == [mm_kwargs[i] for i in (1, 3, 0, 2)] + assert [int(o) for o in outputs] == [0, 1, 2, 3] + + def test_execute_mm_encoder_is_a_noop_without_scheduled_items(): """A step that schedules no encoder input must not touch the encoder.""" cache = EncoderCache() diff --git a/tests/v1/worker/test_gpu_extract_hidden_states_speculator.py b/tests/v1/worker/test_gpu_extract_hidden_states_speculator.py index 74464f542f..5fa44d3475 100644 --- a/tests/v1/worker/test_gpu_extract_hidden_states_speculator.py +++ b/tests/v1/worker/test_gpu_extract_hidden_states_speculator.py @@ -45,7 +45,8 @@ def test_init_speculator_dispatches_extract_hidden_states(monkeypatch): monkeypatch.setattr(spec_module, "ExtractHiddenStatesSpeculator", fake_speculator) - assert init_speculator(vllm_config, device) == (vllm_config, device) + speculator = init_speculator(vllm_config, device, cast(Any, None)) + assert speculator == (vllm_config, device) def test_propose_caches_hidden_states_and_returns_sampled_tokens(monkeypatch): diff --git a/tests/v1/worker/test_gpu_model_runner.py b/tests/v1/worker/test_gpu_model_runner.py index 5cf53899b8..3851d5dad6 100644 --- a/tests/v1/worker/test_gpu_model_runner.py +++ b/tests/v1/worker/test_gpu_model_runner.py @@ -624,6 +624,70 @@ def test_set_active_mm_loras_builds_tower_and_connector_mappings(): assert connector_mapping.index_mapping == ((7,) * 14 + (7,) * 13 + (0,) * 12) +def test_set_active_mm_loras_follows_modality_sorted_encoder_order(): + model = Mock() + model.get_mm_lora_token_counts.side_effect = ( + lambda *, modality, mm_kwargs, num_mm_embeds: (num_mm_embeds, None) + ) + lora_manager = Mock() + lora_manager.supports_tower_connector_lora.return_value = True + + encoder_cache = EncoderCache() + for req_id, modality, length in [("req-a", "video", 1), ("req-b", "image", 2)]: + encoder_cache.mm_features[req_id] = [ + MultiModalFeatureSpec( + data=None, + modality=modality, + identifier=f"{req_id}-0", + mm_position=PlaceholderRange(offset=0, length=length), + ) + ] + lora_state = LoraState(max_num_reqs=2) + lora_request = LoRARequest("vision-lora", 7, "/tmp/vision-lora") + lora_state.add_request("req-a", 0, lora_request) + lora_state.add_request("req-b", 1, None) + + set_active_mm_loras( + model=model, + lora_manager=lora_manager, + encoder_cache=encoder_cache, + req_id_to_index={"req-a": 0, "req-b": 1}, + lora_state=lora_state, + scheduled_encoder_inputs={"req-a": [0], "req-b": [0]}, + ) + + # The image is encoded before the video, so its tokens come first. + _, tower_mapping = lora_manager.set_active_adapters.call_args_list[0].args + assert tower_mapping.index_mapping == (0, 0, 7) + + +def test_batch_mm_inputs_from_scheduler_sorts_by_modality(): + features = { + f"req{i}": MultiModalFeatureSpec( + data=Mock(), + modality=m, + identifier=f"hash{i}", + mm_position=PlaceholderRange(offset=i, length=1), + ) + for i, m in enumerate(["video", "image", "video"]) + } + runner = SimpleNamespace( + requests={r: SimpleNamespace(mm_features=[f]) for r, f in features.items()} + ) + scheduler_output = SimpleNamespace( + scheduled_encoder_inputs={r: [0] for r in features} + ) + + mm_hashes, mm_kwargs, mm_lora_refs = GPUModelRunner._batch_mm_inputs_from_scheduler( + runner, scheduler_output + ) + + order = ["req1", "req0", "req2"] + assert mm_hashes == [features[r].identifier for r in order] + assert mm_kwargs == [(features[r].modality, features[r].data) for r in order] + assert mm_lora_refs == [(r, features[r].mm_position) for r in order] + + def test_update_states_new_request(model_runner, dist_init): req_id = "req_0" @@ -1015,6 +1079,36 @@ def test_reload_weights_before_load_model(model_runner): model_runner.reload_weights() +def test_reload_weights_path_replaces_object_storage_source(monkeypatch): + # An engine started from an object-storage URI keeps that URI in + # model_config.model_weights, which the runai_streamer loader prefers over + # model_config.model. A new weights_path must replace it, or the loader + # silently re-streams the original checkpoint. + loader = Mock() + loader.get_all_weights.return_value = iter(()) + monkeypatch.setattr(gpu_model_runner_module, "get_model_loader", lambda _: loader) + monkeypatch.setattr(gpu_model_runner_module, "initialize_layerwise_reload", Mock()) + monkeypatch.setattr(gpu_model_runner_module, "finalize_layerwise_reload", Mock()) + + runner = Mock(lora_config=None) + runner.model_config = SimpleNamespace( + model="/tmp/pulled-config-files", + model_weights="s3://bucket/original", + revision="abc123", + quantization=None, + ) + runner.get_model.return_value.named_parameters.return_value = [] + runner.get_model.return_value.load_weights.return_value = None + + GPUModelRunner.reload_weights(runner, weights_path="org/new-model") + + loader.get_all_weights.assert_called_once_with( + runner.model_config, runner.get_model.return_value + ) + cfg = runner.model_config + assert (cfg.model, cfg.model_weights, cfg.revision) == ("org/new-model", "", None) + + def test_sample_passes_reordered_draft_probs_to_rejection_sampler(): runner = object.__new__(GPUModelRunner) runner.use_async_scheduling = False diff --git a/tests/v1/worker/test_gpu_model_runner_prefill_tail_classification.py b/tests/v1/worker/test_gpu_model_runner_prefill_tail_classification.py new file mode 100644 index 0000000000..92207aa38f --- /dev/null +++ b/tests/v1/worker/test_gpu_model_runner_prefill_tail_classification.py @@ -0,0 +1,262 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""`GPUModelRunner._prepare_inputs` must tell a chunked-prefill continuation +row apart from a genuine zero-draft decode row. + +Both kinds of row are absent from `scheduled_spec_decode_tokens` and both +schedule exactly one token, so `num_decode_draft_tokens` starts at the +placeholder `-1` for either. The `zero_draft_decode_mask` in +`gpu_model_runner.py` is what tells them apart afterwards, using +`num_computed_tokens >= num_prompt_tokens`: a prefill continuation must keep +`-1` (it carries no drafts and no accepted-token offset to apply), while a +zero-draft decode row must become `0` (it still has to run the speculative +path so the recurrent backends apply that offset). Getting this wrong in +either direction was the root cause behind issue #40875: Kimi-Linear crashed +with an illegal memory access when prefill tails were misclassified as +zero-draft decodes. + +Each batch below also includes one row with a real draft token, so +`scheduled_spec_decode_tokens` is non-empty even under the pre-fix +`use_spec_decode = len(scheduled_spec_decode_tokens) > 0` predicate. That +keeps the assertions targeted at the mask fix alone, not the unrelated +`use_spec_decode` predicate fix (`self.speculative_config is not None`) that +landed in the same patch. +""" + +from types import SimpleNamespace + +import numpy as np +import pytest + +from vllm.config import ( + CacheConfig, + ModelConfig, + ParallelConfig, + SchedulerConfig, + SpeculativeConfig, + VllmConfig, + set_current_vllm_config, +) +from vllm.model_executor.layers.attention import Attention +from vllm.platforms import current_platform +from vllm.sampling_params import SamplingParams +from vllm.utils.import_utils import resolve_obj_by_qualname +from vllm.v1.core.sched.output import CachedRequestData, NewRequestData, SchedulerOutput +from vllm.v1.kv_cache_interface import ( + FullAttentionSpec, + KVCacheConfig, + KVCacheGroupSpec, + KVCacheTensor, +) +from vllm.v1.worker.gpu_input_batch import InputBatch +from vllm.v1.worker.gpu_model_runner import GPUModelRunner + +BLOCK_SIZE = 16 +NUM_BLOCKS = 10 + + +def _get_vllm_config() -> VllmConfig: + model_config = ModelConfig(model="facebook/opt-125m", dtype="float16", seed=42) + scheduler_config = SchedulerConfig( + max_num_seqs=10, + max_num_batched_tokens=512, + max_model_len=512, + is_encoder_decoder=model_config.is_encoder_decoder, + ) + cache_config = CacheConfig( + block_size=BLOCK_SIZE, gpu_memory_utilization=0.9, cache_dtype="auto" + ) + cache_config.kv_cache_layout = "LBNHC" + return VllmConfig( + model_config=model_config, + cache_config=cache_config, + scheduler_config=scheduler_config, + parallel_config=ParallelConfig(), + # ngram needs no draft model weights, so the fixture stays as cheap as + # the speculative-decoding-free one in test_gpu_model_runner.py. + speculative_config=SpeculativeConfig(method="ngram", num_speculative_tokens=3), + ) + + +def _make_model_runner(vllm_config: VllmConfig) -> GPUModelRunner: + """Build the V1 runner the platform's worker would, without starting it.""" + worker_cls = resolve_obj_by_qualname(vllm_config.parallel_config.worker_cls) + worker = SimpleNamespace( + vllm_config=vllm_config, + device=current_platform.device_type, + use_v2_model_runner=False, + ) + return worker_cls._make_model_runner(worker) + + +def _initialize_kv_cache(runner: GPUModelRunner) -> None: + attn_spec = FullAttentionSpec( + block_size=BLOCK_SIZE, + num_kv_heads=runner.model_config.get_num_kv_heads(runner.parallel_config), + head_size=runner.model_config.get_head_size(), + dtype=runner.kv_cache_dtype, + ) + tensor_size = attn_spec.page_size_bytes * NUM_BLOCKS + kv_cache_config = KVCacheConfig( + num_blocks=NUM_BLOCKS, + kv_cache_tensors=[ + KVCacheTensor( + size=tensor_size, + layers=["layer.0"], + layer_stride=tensor_size, + block_stride=attn_spec.page_size_bytes, + ), + ], + kv_cache_groups=[ + KVCacheGroupSpec(layer_names=["layer.0"], kv_cache_spec=attn_spec) + ], + ) + runner.kv_cache_config = kv_cache_config + runner.input_batch = InputBatch( + max_num_reqs=runner.max_num_reqs, + max_model_len=runner.max_model_len, + max_num_batched_tokens=runner.max_num_tokens, + device=runner.device, + vocab_size=runner.model_config.get_vocab_size(), + block_sizes=[kv_cache_config.kv_cache_groups[0].kv_cache_spec.block_size], + kernel_block_sizes=[ + kv_cache_config.kv_cache_groups[0].kv_cache_spec.block_size + ], + max_num_blocks_per_req=[NUM_BLOCKS], + ) + runner.initialize_attn_backend(kv_cache_config) + + +@pytest.fixture +def spec_decode_model_runner(dist_init): + vllm_config = _get_vllm_config() + with set_current_vllm_config(vllm_config): + model_config = vllm_config.model_config + num_heads = model_config.get_num_kv_heads(vllm_config.parallel_config) + head_size = model_config.get_head_size() + vllm_config.compilation_config.static_forward_context["layer.0"] = Attention( + num_heads, head_size, 0.1 + ) + runner = _make_model_runner(vllm_config) + _initialize_kv_cache(runner) + yield runner + + +def _new_request( + req_id: str, prompt_len: int, num_computed_tokens: int +) -> tuple[NewRequestData, int]: + """A request with `prompt_len` prompt tokens, `num_computed_tokens` of + which are already computed (e.g. a prefix-cache hit), scheduled for the + single remaining/next token this step.""" + return ( + NewRequestData( + req_id=req_id, + prompt_token_ids=list(range(prompt_len)), + mm_features=[], + sampling_params=SamplingParams(), + pooling_params=None, + block_ids=([0],), + num_computed_tokens=num_computed_tokens, + lora_request=None, + ), + 1, + ) + + +def test_prefill_continuation_keeps_placeholder_decode_zero_draft_becomes_zero( + spec_decode_model_runner, +): + """One request has one token left to prefill (`num_computed_tokens == + num_prompt_tokens - 1`); another has already finished its prompt and is + now decoding with no draft tokens proposed this step; a third is decoding + with a real draft. `decode_with_draft` gives the batch a non-empty + `scheduled_spec_decode_tokens`, so this also exercises the `-1`-vs-`0` + distinction on the pre-fix definition of `use_spec_decode` (`len(sched + uled_spec_decode_tokens) > 0`), isolating the mask fix from the unrelated + `use_spec_decode` predicate fix that landed in the same patch.""" + runner = spec_decode_model_runner + + prefill_req, prefill_sched = _new_request( + "prefill_tail", prompt_len=11, num_computed_tokens=10 + ) + decode_req, decode_sched = _new_request( + "decode_zero_draft", prompt_len=3, num_computed_tokens=3 + ) + drafted_req, _ = _new_request( + "decode_with_draft", prompt_len=3, num_computed_tokens=3 + ) + num_scheduled_tokens = { + "prefill_tail": prefill_sched, + "decode_zero_draft": decode_sched, + "decode_with_draft": 2, # 1 draft token + 1 bonus token + } + scheduler_output = SchedulerOutput( + scheduled_new_reqs=[prefill_req, decode_req, drafted_req], + scheduled_cached_reqs=CachedRequestData.make_empty(), + num_scheduled_tokens=num_scheduled_tokens, + total_num_scheduled_tokens=sum(num_scheduled_tokens.values()), + scheduled_spec_decode_tokens={"decode_with_draft": [99]}, + scheduled_encoder_inputs={}, + num_common_prefix_blocks=[], + finished_req_ids=set(), + free_encoder_mm_hashes=[], + ) + + runner._update_states(scheduler_output) + runner._prepare_inputs( + scheduler_output, + np.array( + [num_scheduled_tokens[r] for r in runner.input_batch.req_ids], + dtype=np.int32, + ), + ) + + prefill_idx = runner.input_batch.req_id_to_index["prefill_tail"] + decode_idx = runner.input_batch.req_id_to_index["decode_zero_draft"] + drafted_idx = runner.input_batch.req_id_to_index["decode_with_draft"] + assert runner.num_decode_draft_tokens.np[prefill_idx] == -1 + assert runner.num_decode_draft_tokens.np[decode_idx] == 0 + assert runner.num_decode_draft_tokens.np[drafted_idx] == 1 + + +def test_prefill_tail_of_two_or_three_tokens_also_keeps_the_placeholder( + spec_decode_model_runner, +): + """A chunked-prefill tail can be 1, 2, or 3 tokens (`long_prefill_token_ + threshold` rounding); the mask only fires for `num_scheduled_tokens == 1`, + so wider tails must keep `-1` regardless of how much prompt is left. A + third, drafted decode row keeps `scheduled_spec_decode_tokens` non-empty + so the assertions hold on the pre-fix `use_spec_decode` predicate too.""" + runner = spec_decode_model_runner + + tail_2_req, _ = _new_request("tail_2", prompt_len=12, num_computed_tokens=10) + tail_3_req, _ = _new_request("tail_3", prompt_len=13, num_computed_tokens=10) + drafted_req, _ = _new_request( + "decode_with_draft", prompt_len=3, num_computed_tokens=3 + ) + num_scheduled_tokens = {"tail_2": 2, "tail_3": 3, "decode_with_draft": 2} + scheduler_output = SchedulerOutput( + scheduled_new_reqs=[tail_2_req, tail_3_req, drafted_req], + scheduled_cached_reqs=CachedRequestData.make_empty(), + num_scheduled_tokens=num_scheduled_tokens, + total_num_scheduled_tokens=sum(num_scheduled_tokens.values()), + scheduled_spec_decode_tokens={"decode_with_draft": [99]}, + scheduled_encoder_inputs={}, + num_common_prefix_blocks=[], + finished_req_ids=set(), + free_encoder_mm_hashes=[], + ) + + runner._update_states(scheduler_output) + runner._prepare_inputs( + scheduler_output, + np.array( + [num_scheduled_tokens[r] for r in runner.input_batch.req_ids], + dtype=np.int32, + ), + ) + + tail_2_idx = runner.input_batch.req_id_to_index["tail_2"] + tail_3_idx = runner.input_batch.req_id_to_index["tail_3"] + assert runner.num_decode_draft_tokens.np[tail_2_idx] == -1 + assert runner.num_decode_draft_tokens.np[tail_3_idx] == -1 diff --git a/tests/v1/worker/test_gpu_model_runner_v2.py b/tests/v1/worker/test_gpu_model_runner_v2.py index 2e0c7271ce..dec382725b 100644 --- a/tests/v1/worker/test_gpu_model_runner_v2.py +++ b/tests/v1/worker/test_gpu_model_runner_v2.py @@ -18,6 +18,7 @@ from vllm.v1.kv_cache_interface import ( MambaSpec, UniformTypeKVCacheSpecs, ) +from vllm.v1.worker.gpu.async_utils import async_copy_to_np from vllm.v1.worker.gpu.block_table import BlockTables from vllm.v1.worker.gpu.model_runner import ExecuteModelState, GPUModelRunner @@ -46,6 +47,7 @@ def test_non_last_pp_rank_uses_global_batch_for_sample_feedback(): finished_req_ids=set(), ec_connector_output=None, cudagraph_stats=None, + num_spec_tokens_to_schedule=0, ) runner.sample_tokens(None) @@ -258,7 +260,9 @@ def _make_capture_runner(captured: bool) -> GPUModelRunner: """Minimal V2 runner for capture_model: fakes everything except the cudagraph_manager's needs_capture decision.""" runner = GPUModelRunner.__new__(GPUModelRunner) - runner.model_state = SimpleNamespace(supports_mm_inputs=False) + runner.model_state = SimpleNamespace( + supports_mm_inputs=False, capture_inner_cudagraphs=lambda *args: None + ) runner.cudagraph_manager = SimpleNamespace( needs_capture=lambda: captured, capture=lambda *args, **kwargs: None, @@ -334,3 +338,32 @@ def test_capture_model_profile_only_skips_lock(monkeypatch): runner.capture_model(profile_only=True) assert lock_calls == [] + + +@pytest.mark.parametrize("target_buffer", ["absent", "none", "tensor"]) +def test_get_drafter_hidden_states_tolerates_missing_target_buffer(target_buffer): + """Targets allocate the MTP hidden buffer only for hidden-state drafters.""" + runner = GPUModelRunner.__new__(GPUModelRunner) + hidden_states = torch.zeros(4, 8) + buffer = torch.arange(16 * 8, dtype=torch.float32).view(16, 8) + if target_buffer == "absent": + runner.model = SimpleNamespace() + else: + returned = buffer if target_buffer == "tensor" else None + runner.model = SimpleNamespace(get_mtp_target_hidden_states=lambda: returned) + + out = runner._get_drafter_hidden_states(hidden_states) + + if target_buffer == "tensor": + assert torch.equal(out, buffer[:4]) + else: + assert out is hidden_states + + +def test_async_copy_to_np_does_not_alias_reused_buffer(): + buffer = torch.zeros(4, dtype=torch.int64) + + snapshot = async_copy_to_np(buffer) + buffer.fill_(1) + + assert snapshot.tolist() == [0, 0, 0, 0] diff --git a/tests/v1/worker/test_gpu_model_runner_v2_eplb.py b/tests/v1/worker/test_gpu_model_runner_v2_eplb.py index bbde3295ec..d8090de2a3 100644 --- a/tests/v1/worker/test_gpu_model_runner_v2_eplb.py +++ b/tests/v1/worker/test_gpu_model_runner_v2_eplb.py @@ -190,6 +190,7 @@ def test_v2_sample_tokens_runs_eplb_on_non_last_pp_rank(monkeypatch): ec_connector_output=None, routed_experts=None, cudagraph_stats=None, + num_spec_tokens_to_schedule=0, ) runner.req_states = SimpleNamespace() diff --git a/tests/v1/worker/test_gpu_profiler.py b/tests/v1/worker/test_gpu_profiler.py index 5b407e80e9..4b5d2178c4 100644 --- a/tests/v1/worker/test_gpu_profiler.py +++ b/tests/v1/worker/test_gpu_profiler.py @@ -1,6 +1,8 @@ # SPDX-License-Identifier: Apache-2.0 # SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import json import os +import threading from contextlib import nullcontext from types import SimpleNamespace from unittest.mock import MagicMock, Mock, call, patch @@ -26,6 +28,7 @@ from vllm.profiler.wrapper import ( validate_worker_profiler_config, ) from vllm.v1.core.sched.output import CachedRequestData +from vllm.v1.executor.abstract import Executor from vllm.v1.worker.gpu_model_runner import GPUModelRunner from vllm.v1.worker.gpu_worker import Worker from vllm.v1.worker.xpu_worker import XPUWorker @@ -98,17 +101,25 @@ def test_torch_profiler_rebuilds_one_shot_profiler_each_round(tmp_path): def test_torch_profiler_records_each_profile_round(tmp_path): - traces: list[torch.profiler.profile] = [] + trace_paths: list[str] = [] + + def export_trace(prof: torch.profiler.profile) -> None: + # Export as each round stops instead of parsing every round at the end. + path = str(tmp_path / f"round_{len(trace_paths)}.json") + prof.export_chrome_trace(path) + trace_paths.append(path) + wrapper = TorchProfilerWrapper( ProfilerConfig( profiler="torch", torch_profiler_dir=str(tmp_path), torch_profiler_dump_cuda_time_total=False, + torch_profiler_with_stack=False, ), worker_name="worker", local_rank=1, activities=["CPU"], - on_trace_ready=traces.append, + on_trace_ready=export_trace, ) for run in range(2): @@ -117,10 +128,16 @@ def test_torch_profiler_records_each_profile_round(tmp_path): pass wrapper.stop() - assert len(traces) == 2 - for run, trace in enumerate(traces): + assert len(trace_paths) == 2 + for run, path in enumerate(trace_paths): + # The CPU profiler records every thread in the process, and an event + # name from elsewhere in a long session can hold invalid UTF-8. + with open(path, encoding="utf-8", errors="replace") as f: + events = json.load(f)["traceEvents"] assert { - event.name for event in trace.events() if event.name.startswith("run_") + event["name"] + for event in events + if event.get("name", "").startswith("run_") } == {f"run_{run}"} @@ -145,13 +162,33 @@ def test_torch_profiler_device_summary(tmp_path, capsys, activities, dump_device wrapper.start() wrapper.stop() - summary = tmp_path / "profiler_out_0.txt" + summary = tmp_path / "worker.profiler_out.txt" assert summary.exists() == dump_device_time assert ("device times" in capsys.readouterr().out) == dump_device_time if dump_device_time: assert summary.read_text() == "device times\n" +def test_torch_profiler_summary_follows_output_name(tmp_path): + config = ProfilerConfig( + profiler="torch", + torch_profiler_dir=str(tmp_path), + ) + with patch("vllm.profiler.wrapper.torch.profiler.profile") as profile: + table = profile.return_value.key_averages.return_value.table + table.side_effect = ["first run", "second run"] + wrapper = TorchProfilerWrapper( + config, worker_name="initial", local_rank=1, activities=["CUDA"] + ) + for output_name in ["first_rank1", "second_rank1"]: + wrapper.set_output_name(output_name) + wrapper.start() + wrapper.stop() + + assert (tmp_path / "first_rank1.profiler_out.txt").read_text() == "first run\n" + assert (tmp_path / "second_rank1.profiler_out.txt").read_text() == "second run\n" + + @pytest.mark.parametrize( "activities", [[], ["CPU", "CPU"], ["INVALID"]], @@ -170,6 +207,30 @@ def test_torch_profiler_activities_require_torch_profiler(): ProfilerConfig(torch_profiler_activities=["CPU"]) +@pytest.mark.parametrize( + ("profiler", "activities", "ignore_frontend", "expected"), + [ + ("torch", None, False, True), + ("torch", ["CPU"], False, True), + ("torch", ["CPU", "CUDA"], False, True), + ("torch", ["CUDA"], False, False), + ("torch", ["CPU"], True, False), + ("cuda", None, False, False), + ], +) +def test_should_profile_frontend( + profiler, activities, ignore_frontend, expected, tmp_path +): + config = ProfilerConfig( + profiler=profiler, + torch_profiler_dir=str(tmp_path) if profiler == "torch" else "", + torch_profiler_activities=activities, + ignore_frontend=ignore_frontend, + ) + + assert config.should_profile_frontend is expected + + @pytest.mark.parametrize( ("device_type", "activities"), [ @@ -256,16 +317,81 @@ def test_worker_reuses_torch_wrapper_across_profile_rounds(worker_type): patch("vllm.distributed.utils.get_worker_rank_suffix", return_value="rank0"), patch("vllm.profiler.wrapper.TorchProfilerWrapper") as wrapper, ): - worker.profile() + worker.profile( + profile_prefix="first", + delay_iterations=5, + max_iterations=2, + ) worker.profile(is_start=False) - worker.profile() + worker.profile(profile_prefix="second") assert worker.profiler is wrapper.return_value wrapper.assert_called_once() - assert wrapper.return_value.start.call_count == 2 + assert wrapper.return_value.set_output_name.call_args_list == [ + call("first_rank0"), + call("second_rank0"), + ] + assert wrapper.return_value.start.call_args_list == [ + call(delay_iterations=5, max_iterations=2), + call(delay_iterations=None, max_iterations=None), + ] wrapper.return_value.stop.assert_called_once_with() +def test_executor_profile_only_forwards_supplied_override_kwargs(): + executor = MagicMock() + executor.vllm_config.parallel_config.worker_cls = "vllm.v1.worker.gpu_worker.Worker" + + Executor.profile(executor, True, "default") + Executor.profile( + executor, + True, + "bounded", + delay_iterations=5, + max_iterations=2, + ) + + assert executor.collective_rpc.call_args_list == [ + call("profile", args=(True, "default"), kwargs=None), + call( + "profile", + args=(True, "bounded"), + kwargs={"delay_iterations": 5, "max_iterations": 2}, + ), + ] + + +class _LegacyProfileWorker: + def profile(self, is_start: bool = True, profile_prefix: str | None = None): + pass + + +def test_executor_rejects_overrides_unsupported_by_worker(): + """Out-of-tree workers with the old ``profile`` signature are rejected + before dispatch, so a failed collective_rpc cannot leave replies queued.""" + executor = MagicMock() + executor.vllm_config.parallel_config.worker_cls = f"{__name__}._LegacyProfileWorker" + + with pytest.raises(ValueError, match="max_iterations"): + Executor.profile(executor, True, None, max_iterations=2) + Executor.profile(executor, True, "legacy") + + executor.collective_rpc.assert_called_once_with( + "profile", args=(True, "legacy"), kwargs=None + ) + + +def test_worker_rejects_invalid_profile_prefix(): + worker = object.__new__(Worker) + worker.profiler = None + worker.profiler_config = ProfilerConfig( + profiler="torch", torch_profiler_dir="/tmp/mock" + ) + + with pytest.raises(ValueError, match="profile_prefix"): + worker.profile(profile_prefix="../trace") + + def test_immediate_start_stop(default_profiler_config): """Test standard start without delay.""" profiler = ConcreteWorkerProfiler(default_profiler_config) @@ -280,6 +406,33 @@ def test_immediate_start_stop(default_profiler_config): assert profiler.stop_call_count == 1 +def test_stop_waits_for_inflight_start(default_profiler_config): + """A stop() from another thread during a slow start() must still stop the + backend instead of leaving it running with the session marked inactive.""" + profiler = ConcreteWorkerProfiler(default_profiler_config) + in_start = threading.Event() + release_start = threading.Event() + + def slow_start(): + in_start.set() + release_start.wait() + + with patch.object(profiler, "_start", side_effect=slow_start): + start_thread = threading.Thread(target=profiler.start) + start_thread.start() + assert in_start.wait(timeout=5) + stop_thread = threading.Thread(target=profiler.stop) + stop_thread.start() + stop_thread.join(timeout=0.1) + release_start.set() + start_thread.join(timeout=5) + stop_thread.join(timeout=5) + + assert profiler._active is False + assert profiler._running is False + assert profiler.stop_call_count == 1 + + def test_delayed_start(default_profiler_config): """Test that profiler waits for N steps before actually starting.""" default_profiler_config.delay_iterations = 2 @@ -386,19 +539,37 @@ def test_delayed_start_and_max_iters(default_profiler_config): assert profiler.stop_call_count == 1 -def test_idempotency(default_profiler_config): - """Test that calling start/stop multiple times doesn't break logic.""" +def test_duplicate_start_is_ignored(default_profiler_config): profiler = ConcreteWorkerProfiler(default_profiler_config) - # Double Start profiler.start() profiler.start() - assert profiler.start_call_count == 1 # Should only start once + assert profiler.start_call_count == 1 - # Double Stop profiler.stop() profiler.stop() - assert profiler.stop_call_count == 1 # Should only stop once + assert profiler.stop_call_count == 1 + + +def test_session_bounds_override_and_reset_to_defaults(): + profiler = ConcreteWorkerProfiler( + ProfilerConfig( + profiler="torch", + torch_profiler_dir="/tmp/mock", + delay_iterations=3, + max_iterations=4, + ) + ) + + profiler.start(delay_iterations=1, max_iterations=2) + assert profiler._delay_iters == 1 + assert profiler._max_iters == 2 + profiler.stop() + + profiler.start() + assert profiler._delay_iters == 3 + assert profiler._max_iters == 4 + profiler.stop() def test_step_inactive(default_profiler_config): @@ -1081,7 +1252,9 @@ def test_gpu_worker_creates_proton_profiler(): Worker.profile(worker) wrapper.assert_called_once_with(worker.profiler_config, worker_name="rank1") - worker.profiler.start.assert_called_once_with() + worker.profiler.start.assert_called_once_with( + delay_iterations=None, max_iterations=None + ) @_requires_cuda_for_proton @@ -1127,7 +1300,9 @@ def test_gpu_worker_reuses_cuda_graph_proton_session(): Worker.profile(worker, is_start=False) worker.profiler.set_output_name.assert_called_once_with("first_rank1") - worker.profiler.start.assert_called_once_with() + worker.profiler.start.assert_called_once_with( + delay_iterations=None, max_iterations=None + ) worker.profiler.stop.assert_called_once_with() diff --git a/tests/v1/worker/test_gpu_autoregressive_speculator.py b/tests/v1/worker/test_gpu_target_dependent_ar_speculator.py similarity index 91% rename from tests/v1/worker/test_gpu_autoregressive_speculator.py rename to tests/v1/worker/test_gpu_target_dependent_ar_speculator.py index a87c647256..b63bd7a2ff 100644 --- a/tests/v1/worker/test_gpu_autoregressive_speculator.py +++ b/tests/v1/worker/test_gpu_target_dependent_ar_speculator.py @@ -22,17 +22,17 @@ from vllm.v1.attention.backends.flash_attn import FlashAttentionMetadata from vllm.v1.attention.backends.utils import split_decodes_and_prefills from vllm.v1.worker.gpu.cudagraph_utils import BatchExecutionDescriptor from vllm.v1.worker.gpu.spec_decode import speculator as base_spec_module -from vllm.v1.worker.gpu.spec_decode.autoregressive import speculator as spec_module -from vllm.v1.worker.gpu.spec_decode.autoregressive.speculator import ( - AutoRegressiveSpeculator, -) from vllm.v1.worker.gpu.spec_decode.multi_module_mtp.speculator import ( MultiModuleMTPSpeculator, ) from vllm.v1.worker.gpu.spec_decode.speculator import DraftModelSpeculator +from vllm.v1.worker.gpu.spec_decode.target_dependent_ar import speculator as spec_module +from vllm.v1.worker.gpu.spec_decode.target_dependent_ar.speculator import ( + TargetDependentARSpeculator, +) -class _TestSpeculator(AutoRegressiveSpeculator): +class _TestSpeculator(TargetDependentARSpeculator): def load_draft_model(self, target_model, target_attn_layer_names): return self.test_draft_model @@ -337,6 +337,30 @@ def test_draft_model_multimodal_embedding_capability(model_cls, expected): assert supports_multimodal_embeddings(model_cls) is expected +@pytest.mark.parametrize( + "model_cls", [EagleLlama4ForCausalLM, EagleMistralLarge3ForCausalLM] +) +def test_mm_capable_drafter_merges_multimodal_embeddings(model_cls): + """Drafters advertising multimodal embeddings must embed text ids and + scatter the multimodal embeddings themselves; they do not inherit + SupportsMultiModal's helpers.""" + table = torch.arange(8, dtype=torch.float32).repeat(3, 1).T # [8, 3] + drafter = object.__new__(model_cls) + drafter.model = SimpleNamespace(embed_input_ids=lambda ids: table[ids]) + input_ids = torch.tensor([1, 5, 6, 2]) + + text_only = model_cls.embed_input_ids(drafter, input_ids) + assert torch.equal(text_only, table[input_ids]) + + mm_embeds = [torch.full((2, 3), -1.0)] + is_multimodal = torch.tensor([False, True, True, False]) + merged = model_cls.embed_input_ids( + drafter, input_ids, mm_embeds, is_multimodal=is_multimodal + ) + assert torch.equal(merged[[0, 3]], table[[1, 2]]) + assert torch.equal(merged[[1, 2]], mm_embeds[0]) + + def test_run_model_unpacks_tuple_return_for_mtp(monkeypatch): logits_hidden = torch.full((4, 3), 1.0) feedback_hidden = torch.full((4, 3), 2.0) @@ -374,19 +398,22 @@ def test_run_model_reuses_tensor_return_for_mtp(monkeypatch): ( "method_name", "cg_mode", + "num_speculative_steps", "expected_eager_calls", "expected_graph_replays", ), [ - ("_multi_step_decode", CUDAGraphMode.NONE, 3, 0), - ("_multi_step_decode", CUDAGraphMode.FULL, 0, 3), - ("_fused_multi_step_decode", CUDAGraphMode.NONE, 3, 0), - ("_fused_multi_step_decode", CUDAGraphMode.FULL, 0, 1), + ("_multi_step_decode", CUDAGraphMode.NONE, 4, 3, 0), + ("_multi_step_decode", CUDAGraphMode.FULL, 4, 0, 3), + ("_fused_multi_step_decode", CUDAGraphMode.NONE, 4, 3, 0), + ("_fused_multi_step_decode", CUDAGraphMode.FULL, 4, 0, 1), + ("_multi_step_decode", CUDAGraphMode.FULL, 3, 0, 2), ], ) def test_multi_step_decode_replays_captured_graph_as_expected( method_name, cg_mode, + num_speculative_steps, expected_eager_calls, expected_graph_replays, ): @@ -416,6 +443,7 @@ def test_multi_step_decode_replays_captured_graph_as_expected( batch_desc=batch_desc, seq_lens_cpu_upper_bound=None, num_tokens_across_dp=None, + num_speculative_steps=num_speculative_steps, ) assert generate_draft.call_count == expected_eager_calls diff --git a/tests/v1/worker/test_gpu_warmup_blocks.py b/tests/v1/worker/test_gpu_warmup_blocks.py index 04179ae760..e0a99d5e8f 100644 --- a/tests/v1/worker/test_gpu_warmup_blocks.py +++ b/tests/v1/worker/test_gpu_warmup_blocks.py @@ -116,9 +116,14 @@ class _StepRecorder: def __init__(self) -> None: # (blocks held per group, num_computed_tokens, num_scheduled_tokens) self.steps: list[tuple[list[int], int, int]] = [] + self.num_spec_tokens_to_schedule: list[int] = [] self._held: dict[str, list[int]] = {} def execute_model(self, scheduler_output) -> None: + if scheduler_output.total_num_scheduled_tokens: + self.num_spec_tokens_to_schedule.append( + scheduler_output.num_spec_tokens_to_schedule + ) for new_req in scheduler_output.scheduled_new_reqs: self._held[new_req.req_id] = [len(ids) for ids in new_req.block_ids] self._record(new_req.req_id, new_req.num_computed_tokens, scheduler_output) @@ -161,7 +166,7 @@ def _assert_covers_lookahead( # 0 covers eagle / MTP / draft models, 1 covers DFlash's extra in-fill query. @pytest.mark.parametrize("extra_lookahead", [0, 1]) -@pytest.mark.parametrize("num_spec_steps", [2, 3, 5, 7]) +@pytest.mark.parametrize("num_spec_steps", [1, 2, 3, 5, 7]) def test_warmup_kernels_reserves_lookahead_blocks(num_spec_steps, extra_lookahead): num_lookahead_tokens = num_spec_steps + extra_lookahead recorder = _StepRecorder() @@ -173,6 +178,7 @@ def test_warmup_kernels_reserves_lookahead_blocks(num_spec_steps, extra_lookahea ) _assert_covers_lookahead(recorder.steps, num_lookahead_tokens) + assert set(recorder.num_spec_tokens_to_schedule) == {num_spec_steps} def test_mixed_warmup_reserves_lookahead_blocks(): @@ -187,6 +193,7 @@ def test_mixed_warmup_reserves_lookahead_blocks(): ) _assert_covers_lookahead(recorder.steps, num_lookahead_tokens) + assert set(recorder.num_spec_tokens_to_schedule) == {NUM_SPEC_STEPS} @pytest.mark.parametrize("mamba_cache_mode", ["none", "align"]) @@ -314,24 +321,28 @@ def test_reserved_block_count_matches_real_kv_cache_manager(): @pytest.mark.parametrize( - ("method", "expected"), + ("method", "draft_hf_config", "expected"), [ - ("eagle", NUM_SPEC_STEPS), - ("eagle3", NUM_SPEC_STEPS), - ("mtp", NUM_SPEC_STEPS), - ("dspark", NUM_SPEC_STEPS), - ("draft_model", NUM_SPEC_STEPS), + ("eagle", None, NUM_SPEC_STEPS), + ("eagle3", None, NUM_SPEC_STEPS), + ("mtp", None, NUM_SPEC_STEPS), + ("dspark", None, NUM_SPEC_STEPS), + ("dspark", {"sample_from_anchor": True}, NUM_SPEC_STEPS), + ("dspark", {"sample_from_anchor": False}, NUM_SPEC_STEPS + 1), + ("draft_model", None, NUM_SPEC_STEPS), # DFlash's in-fill decoding adds a query for the last sampled token. - ("dflash", NUM_SPEC_STEPS + 1), - ("ngram", 0), - ("ngram_gpu", 0), - ("medusa", 0), - ("mlp_speculator", 0), - ("suffix", 0), - ("extract_hidden_states", 0), + ("dflash", None, NUM_SPEC_STEPS + 1), + ("ngram", None, 0), + ("ngram_gpu", None, 0), + ("medusa", None, 0), + ("mlp_speculator", None, 0), + ("suffix", None, 0), + ("extract_hidden_states", None, 0), ], ) -def test_num_lookahead_tokens_per_method(method: str, expected: int): +def test_num_lookahead_tokens_per_method( + method: str, draft_hf_config: dict | None, expected: int +): """`VllmConfig.num_lookahead_tokens` is the single source of the reservation. Both the scheduler and the warmup read it, so a wrong answer here silently @@ -350,6 +361,12 @@ def test_num_lookahead_tokens_per_method(method: str, expected: int): speculative_config = object.__new__(SpeculativeConfig) object.__setattr__(speculative_config, "method", method) object.__setattr__(speculative_config, "num_speculative_tokens", NUM_SPEC_STEPS) + hf_config = SimpleNamespace(**(draft_hf_config or {})) + object.__setattr__( + speculative_config, + "draft_model_config", + SimpleNamespace(hf_config=hf_config), + ) config = _Config() config.speculative_config = speculative_config diff --git a/tests/v1/worker/test_gpu_worker.py b/tests/v1/worker/test_gpu_worker.py index 90adce1874..6dc20a8177 100644 --- a/tests/v1/worker/test_gpu_worker.py +++ b/tests/v1/worker/test_gpu_worker.py @@ -378,9 +378,6 @@ def test_execute_model_waits_previous_pp_send_before_forward( worker = SimpleNamespace( vllm_config=SimpleNamespace( - compilation_config=SimpleNamespace( - pass_config=SimpleNamespace(enable_sp=False) - ), parallel_config=SimpleNamespace( pipeline_parallel_size=2, distributed_executor_backend="mp" ), diff --git a/tests/v1/worker/test_mamba_hybrid_model_state.py b/tests/v1/worker/test_mamba_hybrid_model_state.py index dba5444150..fa54ccfced 100644 --- a/tests/v1/worker/test_mamba_hybrid_model_state.py +++ b/tests/v1/worker/test_mamba_hybrid_model_state.py @@ -67,6 +67,22 @@ def test_prepare_attn_forwards_positions(monkeypatch: pytest.MonkeyPatch) -> Non assert build_attn_metadata.call_args.kwargs["positions"] is positions +def test_add_request_seeds_state_idx_in_mamba_blocks() -> None: + """A drafter group can lower cache_config.block_size below the mamba + block size; the seeded column must still be in mamba blocks.""" + state = object.__new__(MambaHybridModelState) + state.cache_config = SimpleNamespace(block_size=16, mamba_block_size=880) + state._align_mode = True + state.rope_state = None + state.prompt_embeds_state = None + state.num_accepted_tokens_gpu = torch.ones(2, dtype=torch.int32) + state._mamba_state_idx_gpu = torch.zeros(2, dtype=torch.int32) + + state.add_request(1, SimpleNamespace(num_computed_tokens=107_360)) + + assert state._mamba_state_idx_gpu[1] == 121 + + def test_padded_prompt_tail_builds_as_spec_decode( monkeypatch: pytest.MonkeyPatch, ) -> None: @@ -140,6 +156,177 @@ def test_padded_prompt_tail_builds_as_spec_decode( assert meta.num_prefill_tokens == k + 1 +def _mamba_hybrid_state(num_speculative_tokens: int) -> MambaHybridModelState: + state = object.__new__(MambaHybridModelState) + state.vllm_config = SimpleNamespace(num_speculative_tokens=num_speculative_tokens) + state.model_config = SimpleNamespace(max_model_len=8192) + state._align_mode = False + state.recoverssm = None + # A request that accepted 3 drafts (plus the bonus token) on the previous + # step; the others have never had a draft accepted. + state.num_accepted_tokens_gpu = torch.tensor([4, 1, 1], dtype=torch.int32) + return state + + +def _input_batch( + num_scheduled_tokens: list[int], + num_draft_tokens_per_req: list[int] | None, + is_prefilling: list[bool], + num_reqs_after_padding: int | None = None, +) -> SimpleNamespace: + """`num_draft_tokens_per_req=None` is a batch where nobody proposed a + draft this step, e.g. because ngram found no match for anyone.""" + num_reqs = len(num_scheduled_tokens) + query_start_loc = np.zeros(num_reqs + 1, dtype=np.int32) + query_start_loc[1:] = np.cumsum(num_scheduled_tokens) + num_tokens = int(query_start_loc[-1]) + seq_lens = torch.full((num_reqs,), 10, dtype=torch.int32) + return SimpleNamespace( + num_reqs=num_reqs, + num_reqs_after_padding=num_reqs_after_padding or num_reqs, + num_tokens=num_tokens, + num_tokens_after_padding=num_tokens, + query_start_loc_np=query_start_loc, + num_scheduled_tokens=np.array(num_scheduled_tokens, dtype=np.int32), + max_query_len=None, + seq_lens_cpu_upper_bound=seq_lens, + is_prefilling_np=np.array(is_prefilling), + prefill_runs_as_decode_np=None, + num_draft_tokens_per_req=None + if num_draft_tokens_per_req is None + else np.array(num_draft_tokens_per_req, dtype=np.int32), + idx_mapping=torch.arange(num_reqs), + query_start_loc=torch.from_numpy(query_start_loc), + seq_lens=seq_lens, + dcp_local_seq_lens=None, + positions=torch.zeros(num_tokens, dtype=torch.int64), + prompt_lens=None, + ) + + +def _prepare_attn_metadata( + state: MambaHybridModelState, + monkeypatch, + input_batch: SimpleNamespace | None = None, + cudagraph_mode: CUDAGraphMode = CUDAGraphMode.NONE, +) -> SimpleNamespace: + captured = {} + + def fake_build_attn_metadata(**kwargs): + captured.update(kwargs) + return Mock() + + monkeypatch.setattr( + "vllm.v1.worker.gpu.model_states.mamba_hybrid.build_attn_metadata", + fake_build_attn_metadata, + ) + + if input_batch is None: + input_batch = _input_batch([1, 1], None, [False, False]) + state.prepare_attn( + input_batch=input_batch, + cudagraph_mode=cudagraph_mode, + block_tables=(), + slot_mappings={}, + attn_groups=[], + kv_cache_config=Mock(), + ) + return captured["model_specific_attn_metadata"] + + +def test_prepare_attn_keeps_the_accepted_offset_when_the_batch_drafted_nothing( + monkeypatch, +): + """`num_accepted_tokens` must reach the attention builders whenever the + speculative config is on, even when *this* batch scheduled no draft + tokens at all. Before the fix, `use_spec_decode` was gated on this step's + scheduled drafts (`len(scheduled_spec_decode_tokens) > 0` in the V1 + runner) rather than on the speculative config, so a request that accepted + drafts on a previous step would silently lose that offset the moment + ngram found no match for anyone in the batch. + """ + state = _mamba_hybrid_state(num_speculative_tokens=3) + + metadata = _prepare_attn_metadata(state, monkeypatch) + + assert metadata.num_accepted_tokens is not None + assert metadata.num_accepted_tokens.tolist() == [4, 1] + assert metadata.num_decode_draft_tokens_cpu is not None + + +def test_prepare_attn_omits_the_offset_without_a_speculative_config(monkeypatch): + """Mirrors the check above: with speculative decoding off entirely, there + is no accepted-token offset to carry, and the metadata must say so + explicitly (`None`) rather than a stale or default value.""" + state = _mamba_hybrid_state(num_speculative_tokens=0) + + metadata = _prepare_attn_metadata(state, monkeypatch) + + assert metadata.num_accepted_tokens is None + assert metadata.num_decode_draft_tokens_cpu is None + + +def test_zero_draft_decode_row_is_marked_as_a_speculative_row(monkeypatch): + """A decode row that got no drafts still has to reach the speculative path. + + The recurrent backends only apply a row's accepted-token offset on that + path, and a row that accepted drafts on the previous step keeps that offset + even when the current step proposes nothing (the scheduler drops drafts + whenever the token budget truncates the step). Marking such a row with the + -1 sentinel sends it to the plain decode path, which reads the state at + column 0 and ignores the offset. + """ + state = _mamba_hybrid_state(num_speculative_tokens=3) + input_batch = _input_batch([3, 1, 1], [2, 0, 0], [False, False, True]) + + metadata = _prepare_attn_metadata(state, monkeypatch, input_batch) + + assert metadata.num_decode_draft_tokens_cpu.tolist() == [ + 2, # decode row with drafts + 0, # decode row with no drafts: speculative path, offset applied + -1, # one-token prefill tail: no offset to apply, stays a plain row + ] + + +def test_batch_without_any_drafts_still_marks_its_decode_rows(monkeypatch): + """`num_draft_tokens_per_req` is None whenever no row in the batch drafted + anything. Those rows are ordinary decode rows carrying an offset from the + previous step, so the absence of drafts in this step must not be read as + speculative decoding being off.""" + state = _mamba_hybrid_state(num_speculative_tokens=3) + input_batch = _input_batch([1, 1], None, [False, True]) + + metadata = _prepare_attn_metadata(state, monkeypatch, input_batch) + + assert metadata.num_decode_draft_tokens_cpu.tolist() == [0, -1] + + +def test_padded_rows_keep_the_sentinel(monkeypatch): + """Full CUDA-graph capture pads the batch; padded rows describe no request + and must not be picked up as speculative rows.""" + state = _mamba_hybrid_state(num_speculative_tokens=3) + input_batch = _input_batch([1, 1], [0, 0], [False, False], num_reqs_after_padding=4) + + metadata = _prepare_attn_metadata( + state, monkeypatch, input_batch, cudagraph_mode=CUDAGraphMode.FULL + ) + + assert metadata.num_decode_draft_tokens_cpu.tolist() == [0, 0, -1, -1] + + +def test_chunked_prefill_tail_of_two_or_three_tokens_keeps_the_sentinel( + monkeypatch, +): + """A prefill continuation is excluded by `is_prefilling`, not by its token + count, so tails wider than one token are covered by the same rule.""" + state = _mamba_hybrid_state(num_speculative_tokens=3) + input_batch = _input_batch([2, 3], [0, 0], [True, True]) + + metadata = _prepare_attn_metadata(state, monkeypatch, input_batch) + + assert metadata.num_decode_draft_tokens_cpu.tolist() == [-1, -1] + + @pytest.mark.skipif(not current_platform.is_cuda(), reason="Requires CUDA") @pytest.mark.parametrize(("num_sampled", "expected_value"), [(0, 1), (3, 3)]) def test_postprocess_state_scalar_with_int32_mapping( diff --git a/tests/v1/worker/test_mamba_utils.py b/tests/v1/worker/test_mamba_utils.py index 50faf0fb8b..4407d6fc54 100644 --- a/tests/v1/worker/test_mamba_utils.py +++ b/tests/v1/worker/test_mamba_utils.py @@ -22,6 +22,7 @@ from vllm.v1.kv_cache_interface import ( MambaSpec, UniformTypeKVCacheSpecs, ) +from vllm.v1.worker.gpu.block_table import BlockTables from vllm.v1.worker.gpu.model_states.mamba_hybrid import MambaHybridModelState from vllm.v1.worker.mamba_utils import ( MambaCopyBuffers, @@ -638,6 +639,51 @@ def test_mamba_groups_support_mixed_specs_in_uniform_group(): assert ctx.state_conv_widths.tolist() == [4, 0, 4, 0, 12] +@pytest.mark.skipif(not torch.cuda.is_available(), reason="Requires CUDA") +def test_v2_align_ctx_binds_request_slot_tables_for_state_copies(): + """V2 state copies may run steps after their batch (PP), so they must read + the per-request-slot tables; aligned state indices are computed for the + current batch and keep reading the gathered tables.""" + cfg = _TestConfig(num_layers=1) + device = torch.device("cuda") + kv_cache_config = _make_kv_cache_config(cfg, ["layer_0"]) + forward_context = { + "layer_0": _make_mock_attention( + torch.empty(cfg.num_blocks, cfg.conv_width, cfg.conv_inner_dim), + torch.empty(cfg.num_blocks, cfg.temporal_state_dim), + ) + } + block_tables = BlockTables( + block_sizes=[cfg.block_size], + max_num_reqs=cfg.max_num_reqs, + max_num_batched_tokens=64, + max_num_blocks_per_group=[4], + device=device, + kernel_block_sizes=[cfg.block_size], + ) + + state = object.__new__(MambaHybridModelState) + state._align_mode = True + state._mamba_group_ids = [] + state._mamba_spec = None + state._mamba_state_copy_funcs = _COPY_FUNCS + state.max_num_reqs = cfg.max_num_reqs + state.device = device + state.vllm_config = MagicMock() + state.vllm_config.compilation_config.static_forward_context = forward_context + + state.initialize_kv_cache(kv_cache_config, block_tables) + + ctx = state._mamba_ctx + assert ctx is not None and ctx.is_initialized + assert ctx.block_table_ptrs.tolist() == [ + block_tables.block_tables[0].gpu.data_ptr() + ] + assert ctx.aligned_index_block_table_ptrs.tolist() == [ + block_tables.input_block_tables[0].data_ptr() + ] + + # ----------------------------------------------------------------------------- # stage_postprocess_inputs_to_gpu: single-pass staging into pinned views # ----------------------------------------------------------------------------- diff --git a/tests/v1/worker/test_mixed_warmup_gate.py b/tests/v1/worker/test_mixed_warmup_gate.py index 04f456a7c9..ba532c2c06 100644 --- a/tests/v1/worker/test_mixed_warmup_gate.py +++ b/tests/v1/worker/test_mixed_warmup_gate.py @@ -6,6 +6,7 @@ from types import SimpleNamespace import pytest +from vllm.v1.core.sched.output import SchedulerOutput from vllm.v1.worker.gpu import warmup from vllm.v1.worker.gpu.warmup import run_mixed_prefill_decode_warmup @@ -31,6 +32,40 @@ def test_mixed_warmup_skipped_for_single_seq(max_num_reqs): ) +def test_mixed_warmup_disables_watermarking(): + outputs: list[SchedulerOutput] = [] + runner = SimpleNamespace( + is_pooling_model=False, + max_num_reqs=2, + kv_cache_config=SimpleNamespace( + kv_cache_groups=[ + SimpleNamespace(kv_cache_spec=SimpleNamespace(block_size=16)) + ], + num_blocks=128, + ), + vllm_config=SimpleNamespace(num_lookahead_tokens=0), + max_model_len=128, + num_speculative_steps=0, + model_state=SimpleNamespace(max_encoder_len=0), + kv_connector=SimpleNamespace(set_disabled=lambda disabled: None), + ) + + assert run_mixed_prefill_decode_warmup( + runner, + worker_execute_model=outputs.append, + worker_sample_tokens=lambda grammar_output: None, + num_tokens=32, + ) + + sampling_params = [ + request.sampling_params + for output in outputs + for request in output.scheduled_new_reqs + ] + assert len(sampling_params) == 2 + assert all(params.watermarking is False for params in sampling_params) + + @pytest.mark.parametrize("fail_warmup", [False, True]) def test_kernel_warmup_restores_uncalibrated_adaptive_manager(monkeypatch, fail_warmup): """Startup must warm fixed drafts before calibration and retain its manager.""" diff --git a/tests/v1/worker/test_utils.py b/tests/v1/worker/test_utils.py index abcbdb24f2..f6f7227954 100644 --- a/tests/v1/worker/test_utils.py +++ b/tests/v1/worker/test_utils.py @@ -92,6 +92,8 @@ def test_hisparse_worker_get_kv_connector_stats_reads_completed_snapshot(monkeyp "cache_hits": [12], "cache_misses": [4], "host_to_device_bytes": [64], + "host_cache_usage_perc": [], + "pending_page_transfers": [], } diff --git a/tests/watermarking/test_compatibility.py b/tests/watermarking/test_compatibility.py new file mode 100644 index 0000000000..48c27db09d --- /dev/null +++ b/tests/watermarking/test_compatibility.py @@ -0,0 +1,354 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import asyncio +from types import SimpleNamespace +from unittest.mock import AsyncMock, MagicMock, patch + +import pytest + +from vllm import SamplingParams +from vllm import logger as vllm_logger +from vllm.config import VllmConfig, WatermarkConfig +from vllm.renderers import BaseRenderer +from vllm.sampling_params import BeamSearchParams +from vllm.v1.engine import EngineCoreRequest +from vllm.v1.engine.async_llm import AsyncLLM +from vllm.v1.engine.input_processor import InputProcessor +from vllm.v1.engine.llm_engine import LLMEngine + + +def _input_processor(server_uses_watermarking: bool = True) -> InputProcessor: + config = VllmConfig() + config.model_config = SimpleNamespace( + try_get_generation_config=lambda: {}, + logits_processors=None, + return_sampling_mask=False, + enable_trace_replay=True, + is_multimodal_model=False, + supports_multimodal_inputs=False, + runner_type="generate", + is_diffusion=False, + ) + config.watermark_config = ( + WatermarkConfig(key=42) if server_uses_watermarking else None + ) + renderer = MagicMock(spec=BaseRenderer) + renderer.tokenizer = None + renderer._executor = None + return InputProcessor(config, renderer) + + +def _validate( + params: SamplingParams, server_uses_watermarking: bool = True +) -> bool | None: + processor = _input_processor(server_uses_watermarking) + with patch.object(SamplingParams, "verify"): + return processor._validate_params(params, ("generate",)) + + +def _engine_core_request(params: SamplingParams) -> EngineCoreRequest: + return EngineCoreRequest( + request_id="request", + prompt_token_ids=[1], + mm_features=None, + sampling_params=params, + pooling_params=None, + arrival_time=0.0, + lora_request=None, + cache_salt=None, + data_parallel_rank=None, + ) + + +@pytest.fixture +def reset_warning_once(): + vllm_logger._print_warning_once.cache_clear() + yield + vllm_logger._print_warning_once.cache_clear() + + +def test_greedy_watermarked_requests_warn_once(caplog_vllm, reset_warning_once): + first = SamplingParams(temperature=0) + second = SamplingParams(temperature=0) + + with caplog_vllm.at_level("WARNING"): + first_resolved = _validate(first) + second_resolved = _validate(second) + + message = "subsequent greedy requests will use ordinary greedy sampling" + assert caplog_vllm.text.count(message) == 1 + assert first_resolved is False + assert second_resolved is False + # The caller's params objects are left untouched. + assert first.watermarking is None + assert second.watermarking is None + + +def test_default_watermarking_resolves_when_configured(): + enabled = SamplingParams() + + assert _validate(enabled) is True + # The caller's params object keeps inheriting. + assert enabled.watermarking is None + + +@pytest.mark.parametrize("watermarking", [True]) +def test_requested_watermarking_without_engine_config_warns_and_disables( + watermarking, caplog_vllm, reset_warning_once +): + params = SamplingParams(watermarking=watermarking) + + with caplog_vllm.at_level("WARNING"): + resolved = _validate(params, server_uses_watermarking=False) + + assert "engine has no watermark configuration" in caplog_vllm.text + assert resolved is False + assert params.watermarking is watermarking + + +@pytest.mark.parametrize("watermarking", [None, True]) +def test_explicit_and_default_greedy_warn_and_disable( + watermarking, caplog_vllm, reset_warning_once +): + params = SamplingParams(temperature=0, watermarking=watermarking) + + with caplog_vllm.at_level("WARNING"): + resolved = _validate(params) + + assert "ordinary greedy sampling" in caplog_vllm.text + assert resolved is False + assert params.watermarking is watermarking + + +def test_trace_replay_with_watermarking_warns_and_disables( + caplog_vllm, reset_warning_once +): + params = SamplingParams(trace_decode_token_ids=[1]) + + with caplog_vllm.at_level("WARNING"): + resolved = _validate(params) + + assert "trace replay requests will run without watermarking" in caplog_vllm.text + assert resolved is False + assert params.watermarking is None + + +@pytest.mark.parametrize( + "params", + [ + SamplingParams(temperature=0, watermarking=False), + SamplingParams(trace_decode_token_ids=[1], watermarking=False), + ], +) +def test_incompatible_modes_are_allowed_when_watermarking_is_disabled(params): + _validate(params) + + +@pytest.mark.parametrize( + "params", + [ + SamplingParams(), + SamplingParams(temperature=0), + SamplingParams(trace_decode_token_ids=[1]), + ], +) +def test_requests_without_engine_config_are_unwatermarked( + params, caplog_vllm, reset_warning_once +): + with caplog_vllm.at_level("WARNING"): + resolved = _validate(params, server_uses_watermarking=False) + + assert "engine has no watermark configuration" not in caplog_vllm.text + assert resolved is False + assert params.watermarking is None + + +def test_trace_replay_takes_precedence_over_greedy_warning( + caplog_vllm, reset_warning_once +): + params = SamplingParams(temperature=0, trace_decode_token_ids=[1]) + + with caplog_vllm.at_level("WARNING"): + _validate(params) + + assert "trace replay requests will run without watermarking" in caplog_vllm.text + assert "ordinary greedy sampling" not in caplog_vllm.text + + +@pytest.mark.parametrize( + "params", + [ + SamplingParams(seed=42), + SamplingParams(n=2), + SamplingParams(n=2, seed=42), + ], +) +def test_seeded_and_parallel_sampling_are_allowed(params): + _validate(params) + + +def test_direct_engine_trace_replay_warns_and_disables( + caplog_vllm, monkeypatch, reset_warning_once +): + engine = object.__new__(LLMEngine) + engine.input_processor = _input_processor() + engine.vllm_config = engine.input_processor.vllm_config + assign_request_id = MagicMock() + monkeypatch.setattr(engine.input_processor, "assign_request_id", assign_request_id) + engine.output_processor = MagicMock() + engine.engine_core = MagicMock() + request = _engine_core_request(SamplingParams(trace_decode_token_ids=[1])) + + with caplog_vllm.at_level("WARNING"): + engine.add_request( + request.request_id, + request, + SamplingParams(watermarking=False), + ) + + assert "trace replay requests will run without watermarking" in caplog_vllm.text + assert request.sampling_params.watermarking is False + assert request.sampling_params._watermarking_skipped is True + assign_request_id.assert_called_once_with(request) + + +def test_direct_engine_request_uses_embedded_opt_out(monkeypatch): + engine = object.__new__(LLMEngine) + engine.input_processor = _input_processor() + engine.vllm_config = engine.input_processor.vllm_config + assign_request_id = MagicMock() + monkeypatch.setattr(engine.input_processor, "assign_request_id", assign_request_id) + engine.output_processor = MagicMock() + engine.engine_core = MagicMock() + request = _engine_core_request(SamplingParams(temperature=0, watermarking=False)) + + engine.add_request( + request.request_id, + request, + SamplingParams(temperature=0), + ) + + assert request.sampling_params._watermarking_skipped is False + assign_request_id.assert_called_once_with(request) + + +def test_direct_async_engine_trace_replay_warns_and_disables( + caplog_vllm, monkeypatch, reset_warning_once +): + engine = object.__new__(AsyncLLM) + engine.engine_core = SimpleNamespace( + resources=SimpleNamespace(engine_dead=False), + shutdown=lambda **kwargs: None, + add_request_async=AsyncMock(), + ) + engine.output_handler = None + engine.input_processor = _input_processor() + engine.vllm_config = engine.input_processor.vllm_config + engine.output_processor = MagicMock() + engine.log_requests = False + monkeypatch.setattr(engine, "_run_output_handler", MagicMock()) + request = _engine_core_request(SamplingParams(trace_decode_token_ids=[1])) + + async def add_request() -> None: + await engine.add_request( + request.request_id, + request, + SamplingParams(watermarking=False), + ) + + with caplog_vllm.at_level("WARNING"): + asyncio.run(add_request()) + + assert "trace replay requests will run without watermarking" in caplog_vllm.text + assert request.sampling_params.watermarking is False + assert request.sampling_params._watermarking_skipped is True + + +def test_direct_async_engine_greedy_request_warns_and_disables_watermarking( + caplog_vllm, monkeypatch, reset_warning_once +): + engine = object.__new__(AsyncLLM) + engine.engine_core = SimpleNamespace( + resources=SimpleNamespace(engine_dead=False), + shutdown=lambda **kwargs: None, + add_request_async=AsyncMock(), + ) + engine.output_handler = None + engine.input_processor = _input_processor() + engine.vllm_config = engine.input_processor.vllm_config + engine.output_processor = MagicMock() + engine.log_requests = False + monkeypatch.setattr(engine, "_run_output_handler", MagicMock()) + request = _engine_core_request(SamplingParams(temperature=0)) + + async def add_request() -> None: + await engine.add_request( + request.request_id, + request, + SamplingParams(watermarking=False), + ) + + with caplog_vllm.at_level("WARNING"): + asyncio.run(add_request()) + + assert "subsequent greedy requests will use ordinary greedy sampling" in ( + caplog_vllm.text + ) + assert request.sampling_params.watermarking is False + assert request.sampling_params._watermarking_skipped is True + + +@pytest.mark.parametrize( + ("kwargs", "server_uses_watermarking", "skipped"), + [ + ({}, True, False), + ({"temperature": 0}, True, True), + ({"temperature": 0, "watermarking": False}, True, False), + ({"temperature": 0}, False, False), + ], +) +def test_apply_watermarking_records_skipped_requests( + kwargs, server_uses_watermarking, skipped, reset_warning_once +): + params = SamplingParams(**kwargs) + processor = _input_processor(server_uses_watermarking) + processor.apply_watermarking(params, processor.resolve_watermarking(params)) + assert params._watermarking_skipped is skipped + + # Resolving an already resolved request keeps the flag. + processor.apply_watermarking(params, processor.resolve_watermarking(params)) + assert params._watermarking_skipped is skipped + + +def test_shared_sampling_params_are_not_rewritten_by_admission(reset_warning_once): + """A params object reused across prompts/engines must keep inheriting.""" + params = SamplingParams(temperature=0.8) + + assert _validate(params, server_uses_watermarking=False) is False + assert params.watermarking is None + # Same object submitted to a watermarked engine must still be watermarked. + assert _validate(params, server_uses_watermarking=True) is True + + greedy = SamplingParams(temperature=0.0) + assert _validate(greedy) is False + greedy.temperature = 0.8 + assert _validate(greedy) is True + + +def test_shared_beam_search_params_are_not_rewritten_by_admission( + caplog_vllm, reset_warning_once +): + """Beam search must keep resolving per engine after reuse.""" + params = BeamSearchParams(beam_width=2, max_tokens=4, temperature=0.8) + + # A plain engine leaves the tri-state unresolved on the caller's object. + assert _input_processor(False).resolve_watermarking(params) is False + assert params.watermarking is None + + # The same object on a watermarked engine is warned about and still + # resolves to False, without the first decision being written back. + with caplog_vllm.at_level("WARNING"): + assert _input_processor(True).resolve_watermarking(params) is False + assert "beam search requests will run without watermarking" in caplog_vllm.text + assert params.watermarking is None diff --git a/tests/watermarking/test_engine_compatibility.py b/tests/watermarking/test_engine_compatibility.py new file mode 100644 index 0000000000..1320306cb3 --- /dev/null +++ b/tests/watermarking/test_engine_compatibility.py @@ -0,0 +1,204 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +from collections.abc import Generator + +import pytest + +from vllm import LLM, SamplingParams +from vllm.platforms import current_platform +from vllm.sampling_params import StructuredOutputsParams + +MODEL = "hmellor/tiny-random-LlamaForCausalLM" +MAX_NUM_BATCHED_TOKENS = 32 + +pytestmark = pytest.mark.skipif( + not current_platform.is_cuda_alike(), reason="requires a CUDA-like accelerator" +) + + +@pytest.fixture(scope="module") +def watermarked_llm(vllm_runner) -> Generator[LLM, None, None]: + with vllm_runner( + MODEL, + dtype="half", + enforce_eager=True, + enable_prefix_caching=True, + enable_chunked_prefill=True, + max_num_batched_tokens=MAX_NUM_BATCHED_TOKENS, + max_model_len=128, + max_num_seqs=4, + disable_log_stats=False, + watermark_config={"algorithm": "gumbel", "key": 42}, + ) as runner: + yield runner.llm + + +def test_watermark_with_combined_sampling_controls(watermarked_llm: LLM): + params = SamplingParams( + temperature=0.8, + top_k=8, + top_p=0.9, + min_p=0.01, + presence_penalty=0.2, + frequency_penalty=0.2, + repetition_penalty=1.1, + logit_bias={10: 100.0}, + allowed_token_ids=[10], + max_tokens=8, + ignore_eos=True, + ) + + output = watermarked_llm.generate("Sampling smoke test", params)[0] + + assert list(output.outputs[0].token_ids) == [10] * 8 + + # The assertion above holds with or without watermarking, because + # allowed_token_ids leaves a single candidate. Repeat with a wider allowed + # set, where only watermarked sampling is reproducible without a seed. + wide = SamplingParams( + temperature=0.8, + top_k=8, + top_p=0.9, + min_p=0.01, + presence_penalty=0.2, + frequency_penalty=0.2, + repetition_penalty=1.1, + allowed_token_ids=list(range(10, 210)), + max_tokens=8, + ignore_eos=True, + ) + + first = watermarked_llm.generate("Sampling smoke test", wide)[0] + second = watermarked_llm.generate("Sampling smoke test", wide)[0] + + assert set(first.outputs[0].token_ids) <= set(range(10, 210)) + assert first.outputs[0].token_ids == second.outputs[0].token_ids + + +def test_watermark_with_penalties_and_nucleus_sampling(watermarked_llm: LLM): + params = SamplingParams( + temperature=0.8, + top_k=8, + top_p=0.9, + min_p=0.01, + presence_penalty=0.2, + frequency_penalty=0.2, + repetition_penalty=1.1, + max_tokens=8, + ignore_eos=True, + ) + + first = watermarked_llm.generate("Penalty smoke test", params)[0] + second = watermarked_llm.generate("Penalty smoke test", params)[0] + + assert len(first.outputs[0].token_ids) == 8 + assert first.outputs[0].token_ids == second.outputs[0].token_ids + + +def test_plain_gumbel_seeded_parallel_sampling_is_deterministic( + watermarked_llm: LLM, +): + output = watermarked_llm.generate( + "Seeded parallel sampling smoke test", + SamplingParams( + n=2, + seed=42, + temperature=0.8, + max_tokens=8, + ignore_eos=True, + ), + )[0] + + assert len(output.outputs) == 2 + assert all(len(candidate.token_ids) == 8 for candidate in output.outputs) + assert output.outputs[0].token_ids == output.outputs[1].token_ids + + # Plain Gumbel keys on the token context, so the request seed does not + # change a watermarked draw. Pin that down rather than implying the seed + # is what makes the candidates match. + unseeded = watermarked_llm.generate( + "Seeded parallel sampling smoke test", + SamplingParams( + n=2, + temperature=0.8, + max_tokens=8, + ignore_eos=True, + ), + )[0] + + assert unseeded.outputs[0].token_ids == output.outputs[0].token_ids + + +@pytest.mark.parametrize( + "structured_outputs", + [ + StructuredOutputsParams(choice=["A", "B"]), + StructuredOutputsParams(grammar='root ::= "A" | "B"'), + ], + ids=["choice", "grammar"], +) +def test_watermark_with_structured_output( + watermarked_llm: LLM, structured_outputs: StructuredOutputsParams +): + params = SamplingParams( + temperature=0.8, + max_tokens=8, + structured_outputs=structured_outputs, + ) + + output = watermarked_llm.generate("Choose A or B:", params)[0] + + assert output.outputs[0].text.strip() in ("A", "B") + + +def test_watermark_with_wide_grammar_is_deterministic(watermarked_llm: LLM): + # A wider grammar makes deterministic watermarking observable. + params = SamplingParams( + temperature=0.8, + max_tokens=8, + structured_outputs=StructuredOutputsParams( + grammar="root ::= [A-Z] [A-Z] [A-Z] [A-Z] [A-Z] [A-Z]" + ), + ) + + first = watermarked_llm.generate("Emit six letters:", params)[0] + second = watermarked_llm.generate("Emit six letters:", params)[0] + + assert first.outputs[0].text.strip().isupper() + assert first.outputs[0].token_ids == second.outputs[0].token_ids + + +def test_watermark_with_prefix_cache_and_chunked_prefill(watermarked_llm: LLM): + prompt = {"prompt_token_ids": [101] * 48} + params = SamplingParams(temperature=0.8, max_tokens=8, ignore_eos=True) + + assert len(prompt["prompt_token_ids"]) > MAX_NUM_BATCHED_TOKENS + + cold = watermarked_llm.generate(prompt, params)[0] + cached = watermarked_llm.generate(prompt, params)[0] + + assert cached.num_cached_tokens > 0 + assert cold.outputs[0].token_ids == cached.outputs[0].token_ids + + +def test_mixed_watermarked_batch(watermarked_llm: LLM): + prompts = [ + {"prompt_token_ids": [101] * 32 + [102]}, + {"prompt_token_ids": [201] * 32 + [202]}, + ] + marked = SamplingParams(temperature=0.8, max_tokens=8, ignore_eos=True) + ordinary = SamplingParams( + temperature=0.8, + seed=123, + max_tokens=8, + ignore_eos=True, + watermarking=False, + ) + + batched = watermarked_llm.generate(prompts, [marked, ordinary]) + marked_alone = watermarked_llm.generate(prompts[0], marked)[0] + ordinary_alone = watermarked_llm.generate(prompts[1], ordinary)[0] + + assert batched[0].outputs[0].token_ids == marked_alone.outputs[0].token_ids + assert batched[1].outputs[0].token_ids == ordinary_alone.outputs[0].token_ids diff --git a/tests/watermarking/test_gumbel.py b/tests/watermarking/test_gumbel.py index d789692a0e..aa0ffa2799 100644 --- a/tests/watermarking/test_gumbel.py +++ b/tests/watermarking/test_gumbel.py @@ -4,9 +4,11 @@ import pytest import torch +from vllm.config.watermarking import WatermarkConfig from vllm.platforms import current_platform from vllm.v1.watermarking import ( DualKeyGumbelWatermarkDetector, + DualKeyGumbelWatermarker, GumbelWatermarkDetector, GumbelWatermarker, derive_watermark_key, @@ -42,6 +44,21 @@ def test_detector_deduplicates_context_even_when_target_differs(): assert detection.num_scored_tokens == 3 +def test_watermarker_respects_filtered_token_support(): + logits = torch.full((2, 8), -torch.inf) + logits[0, 3] = 0 + logits[1, 6] = 0 + contexts = torch.tensor([[1, 2, 3, 4], [4, 5, 6, 7]]) + + token_ids = ( + GumbelWatermarker(key=42) + .sample(logits, contexts, lambda values: None) + .token_ids + ) + + assert torch.equal(token_ids, torch.tensor([3, 6])) + + def test_dual_key_detector_scores_each_token_against_both_keys(): token_ids = [1, 2, 3, 4, 5] key_a_detector = GumbelWatermarkDetector( @@ -99,8 +116,12 @@ def test_dual_key_detector_rejects_invalid_alpha(): DualKeyGumbelWatermarkDetector(key=42, alpha=1.1) -def test_dual_key_detector_default_alpha(): - assert DualKeyGumbelWatermarkDetector(key=42).alpha == 0.2 +def test_dual_key_alpha_defaults_match(): + config = WatermarkConfig(algorithm="dual_key_gumbel", key=42) + watermarker = DualKeyGumbelWatermarker(key=42) + detector = DualKeyGumbelWatermarkDetector(key=42) + + assert detector.alpha == watermarker.alpha == config.alpha @pytest.mark.skipif( diff --git a/tests/watermarking/test_parallel_compatibility.py b/tests/watermarking/test_parallel_compatibility.py new file mode 100644 index 0000000000..6e47a65b6f --- /dev/null +++ b/tests/watermarking/test_parallel_compatibility.py @@ -0,0 +1,106 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import asyncio + +import pytest + +from tests.utils import multi_gpu_test +from vllm import SamplingParams +from vllm.distributed import cleanup_dist_env_and_memory +from vllm.engine.arg_utils import AsyncEngineArgs +from vllm.v1.engine.async_llm import AsyncLLM + +MODEL = "hmellor/tiny-random-LlamaForCausalLM" + + +@multi_gpu_test(num_gpus=2) +@pytest.mark.parametrize( + "parallel_kwargs", + [ + {"tensor_parallel_size": 2, "enable_batch_sharded_sampling": True}, + {"pipeline_parallel_size": 2}, + ], + ids=["tp2-sharded-sampling", "pp2"], +) +def test_watermarked_generation_with_model_parallelism(vllm_runner, parallel_kwargs): + prompts = ["Parallel watermark smoke"] * 2 + + with vllm_runner( + MODEL, + dtype="half", + enforce_eager=True, + max_model_len=128, + max_num_seqs=4, + watermark_config={"algorithm": "gumbel", "key": 42}, + **parallel_kwargs, + ) as runner: + first = runner.llm.generate( + prompts, + SamplingParams( + temperature=0.8, + seed=1, + max_tokens=8, + ignore_eos=True, + ), + ) + second = runner.llm.generate( + prompts, + SamplingParams( + temperature=0.8, + seed=2, + max_tokens=8, + ignore_eos=True, + ), + ) + + first_tokens = [output.outputs[0].token_ids for output in first] + second_tokens = [output.outputs[0].token_ids for output in second] + assert all(len(token_ids) == 8 for token_ids in first_tokens) + assert first_tokens[0] == first_tokens[1] + assert first_tokens == second_tokens + + +@multi_gpu_test(num_gpus=2) +def test_watermarked_generation_with_data_parallelism(): + async def run() -> None: + engine = AsyncLLM.from_engine_args( + AsyncEngineArgs( + model=MODEL, + dtype="half", + data_parallel_size=2, + enforce_eager=True, + max_model_len=128, + max_num_seqs=4, + watermark_config={"algorithm": "gumbel", "key": 42}, + ) + ) + + async def generate(index: int) -> tuple[int, ...]: + final_output = None + async for output in engine.generate( + "Data-parallel watermark smoke", + SamplingParams( + temperature=0.8, + seed=index, + max_tokens=8, + ignore_eos=True, + ), + request_id=f"watermark-dp-{index}", + data_parallel_rank=index, + ): + final_output = output + assert final_output is not None + return tuple(final_output.outputs[0].token_ids) + + try: + token_ids = await asyncio.gather(generate(0), generate(1)) + assert len(token_ids[0]) == 8 + assert token_ids[0] == token_ids[1] + finally: + engine.shutdown() + + try: + asyncio.run(run()) + finally: + cleanup_dist_env_and_memory() diff --git a/tests/watermarking/test_watermarking.py b/tests/watermarking/test_watermarking.py index 4de6022e34..384cc30d57 100644 --- a/tests/watermarking/test_watermarking.py +++ b/tests/watermarking/test_watermarking.py @@ -8,8 +8,11 @@ import pytest import torch from vllm import SamplingParams +from vllm import logger as vllm_logger +from vllm.config import VllmConfig from vllm.config.watermarking import WatermarkConfig from vllm.platforms import current_platform +from vllm.v1.sample.ops.topk_topp_sampler import apply_top_k_top_p_pytorch from vllm.v1.watermarking import ( DualKeyGumbelWatermarker, SupportsSpeculativeDecoding, @@ -26,8 +29,8 @@ from vllm.v1.watermarking.spec_decode import ( speculative_target_watermark_key, ) from vllm.v1.watermarking.watermarker import Watermarker, WatermarkSample +from vllm.v1.worker.gpu import buffer_utils from vllm.v1.worker.gpu.buffer_utils import UvaBackedTensor -from vllm.v1.worker.gpu.sample.sampler import Sampler from vllm.v1.worker.gpu.sample.watermark import ( draft_watermarking_mask, philox_gumbel_sample, @@ -48,6 +51,44 @@ def _argmax_sampler(logits: torch.Tensor) -> torch.Tensor: return logits.argmax(dim=-1) +@pytest.fixture +def make_gpu_watermark_sampler(monkeypatch): + class CPUUvaBuffer: + def __init__(self, size, dtype): + self.cpu = torch.zeros(size, dtype=dtype) + self.np = self.cpu.numpy() + + def uva(self, n=None): + return self.cpu[:n] if n is not None else self.cpu + + monkeypatch.setattr(buffer_utils, "UvaBuffer", CPUUvaBuffer) + + def make(watermarker, *, max_num_reqs=2, vocab_size=8, **kwargs): + req_states = SimpleNamespace( + max_num_reqs=max_num_reqs, + vocab_size=vocab_size, + device=torch.device("cpu"), + all_token_ids=SimpleNamespace(gpu=torch.zeros(max_num_reqs, 1)), + prompt_len=SimpleNamespace( + np=np.zeros(max_num_reqs, dtype=np.int32), + gpu=torch.zeros(max_num_reqs, dtype=torch.int32), + ), + prefill_len=SimpleNamespace(np=np.zeros(max_num_reqs, dtype=np.int32)), + total_len=SimpleNamespace(gpu=torch.zeros(max_num_reqs, dtype=torch.int32)), + ) + return GPUWatermarkSampler( + watermarker, + vllm_config=VllmConfig(), + max_num_reqs=max_num_reqs, + vocab_size=vocab_size, + device=torch.device("cpu"), + req_states=req_states, + **kwargs, + ) + + return make + + @pytest.mark.parametrize("algorithm", ["gumbel", "dual_key_gumbel"]) def test_watermarker_contract(algorithm: str): watermarker = create_watermarker( @@ -271,49 +312,47 @@ def test_dual_key_derivation_is_stable(): def test_sampling_params_can_disable_watermarking(): - assert SamplingParams().watermarking - assert not SamplingParams.from_optional(watermarking=False).watermarking + assert SamplingParams().watermarking is None + assert SamplingParams.from_optional(watermarking=False).watermarking is False -def test_gpu_sampler_warns_when_watermarking_is_enabled_for_greedy(monkeypatch): - sampler = object.__new__(GPUWatermarkSampler) - sampler.watermarking = SimpleNamespace(np=np.ones(1, dtype=bool)) +def test_gpu_sampler_warns_about_unexpected_greedy_watermarking( + monkeypatch, make_gpu_watermark_sampler +): + vllm_logger._print_warning_once.cache_clear() + sampler = make_gpu_watermark_sampler(StubWatermarker(), max_num_reqs=1) messages: list[str] = [] - monkeypatch.setattr(Sampler, "add_request", lambda *args, **kwargs: None) monkeypatch.setattr( - "vllm.v1.watermarking.gpu_sampler.logger.warning_once", messages.append + "vllm.v1.watermarking.gpu_sampler.logger.warning", + lambda message, *_args, **_kwargs: messages.append(message), ) - sampler.add_request(0, SamplingParams(temperature=0)) - sampler.add_request(0, SamplingParams(temperature=1)) + sampler.add_request(0, SamplingParams(temperature=0, watermarking=True)) + assert sampler.watermarking.np[0] + sampler.add_request(0, SamplingParams(temperature=0, watermarking=True)) + assert sampler.watermarking.np[0] sampler.add_request(0, SamplingParams(temperature=0, watermarking=False)) - + assert not sampler.watermarking.np[0] assert messages == [ ( - "Watermarking is enabled, but greedy decoding (temperature=0) cannot be " - "watermarked. This request will use ordinary greedy sampling." + "Watermarking is enabled, but greedy decoding " + "(temperature=0) cannot be watermarked. This request will use " + "ordinary greedy sampling." ) ] + vllm_logger._print_warning_once.cache_clear() -def test_gpu_sampler_respects_mixed_request_watermarking(monkeypatch): - sampler = object.__new__(GPUWatermarkSampler) - sampler.watermarker = StubWatermarker() - sampler.deduplicate_contexts = "single_turn" - sampler.watermarking = SimpleNamespace( - np=np.array([True, False]), gpu=torch.tensor([True, False]) - ) - sampler.sampling_states = SimpleNamespace( - temperature=SimpleNamespace(np=np.ones(2), gpu=torch.ones(2)), - seeds=SimpleNamespace(gpu=torch.zeros(2, dtype=torch.int64)), - ) - sampler.use_fp64_gumbel = False +def test_gpu_sampler_respects_mixed_request_watermarking( + monkeypatch, make_gpu_watermark_sampler +): + sampler = make_gpu_watermark_sampler(StubWatermarker(), deduplicate_contexts="none") + sampler.add_request(0, SamplingParams(watermarking=True)) + sampler.add_request(1, SamplingParams(watermarking=False)) + sampler.apply_staged_writes() sampler._get_contexts = lambda expanded_idx_mapping: torch.zeros( 2, 1, dtype=torch.int64 ) - sampler._get_repeated_contexts = lambda expanded_idx_mapping, contexts: torch.zeros( - 2, dtype=torch.bool - ) monkeypatch.setattr( "vllm.v1.watermarking.watermarker.gumbel_sample", lambda *args, **kwargs: torch.tensor([3, 4]), @@ -322,17 +361,92 @@ def test_gpu_sampler_respects_mixed_request_watermarking(monkeypatch): sampled, output_logits = sampler._sample_random( logits, - torch.tensor([0, 1]), - np.array([0, 1]), + torch.tensor([1, 0]), + np.array([1, 0]), torch.zeros(2, dtype=torch.int64), None, None, False, ) - assert torch.equal(sampled, torch.tensor([7, 4])) - assert torch.equal(output_logits[0], torch.full((8,), 10.0)) - assert torch.equal(output_logits[1], logits[1]) + assert torch.equal(sampled, torch.tensor([3, 7])) + assert torch.equal(output_logits[0], logits[0]) + assert torch.equal(output_logits[1], torch.full((8,), 10.0)) + + +def test_gpu_sampler_filters_top_k_top_p_before_watermarking( + monkeypatch, make_gpu_watermark_sampler +): + # ``apply_top_k_top_p`` dispatches to a Triton kernel whenever Triton is + # installed, which cannot read this test's CPU tensors. Force the torch + # reference path; the assertion is about the call order, not the kernel. + monkeypatch.setattr( + "vllm.v1.watermarking.gpu_sampler.apply_top_k_top_p", + lambda logits, k, p: apply_top_k_top_p_pytorch( + logits, k, p, allow_cpu_sync=True + ), + ) + + class CapturingWatermarker: + context_width = 1 + captured_logits = None + + def sample(self, logits, contexts, random_sampler=None, skip_mask=None): + self.captured_logits = logits + return WatermarkSample(logits.argmax(dim=-1), logits) + + watermarker = CapturingWatermarker() + sampler = make_gpu_watermark_sampler( + watermarker, max_num_reqs=1, vocab_size=4, deduplicate_contexts="none" + ) + sampler.add_request(0, SamplingParams(top_k=2, top_p=0.8, watermarking=True)) + sampler.apply_staged_writes() + sampler._get_contexts = lambda expanded_idx_mapping: torch.zeros( + 1, 1, dtype=torch.int64 + ) + logits = torch.tensor([[5.0, 4.0, 3.0, 2.0]]) + + sampled, _ = sampler._sample_random( + logits, + torch.tensor([0]), + np.array([0]), + torch.zeros(1, dtype=torch.int64), + torch.tensor([2]), + torch.tensor([0.8]), + False, + ) + + assert watermarker.captured_logits is not None + assert torch.isneginf(watermarker.captured_logits[0, 2:]).all() + assert sampled.item() in (0, 1) + + +def test_watermark_context_ignores_prefix_cache_bookkeeping(): + sampler = object.__new__(GPUWatermarkSampler) + sampler.watermarker = SimpleNamespace(context_width=3) + sampler.req_states = SimpleNamespace( + total_len=SimpleNamespace(gpu=torch.tensor([4, 0, 5])), + prompt_len=SimpleNamespace(gpu=torch.tensor([3, 0, 2])), + all_token_ids=SimpleNamespace( + gpu=torch.tensor( + [ + [10, 11, 12, 40, 0, 0], + [0, 0, 0, 0, 0, 0], + [20, 21, 50, 51, 52, 0], + ] + ) + ), + num_computed_tokens=SimpleNamespace(gpu=torch.tensor([0, 0, 0])), + ) + request_slots = torch.tensor([2, 0]) + + cold_contexts = sampler._get_contexts(request_slots) + sampler.req_states.num_computed_tokens.gpu[:] = torch.tensor([3, 0, 2]) + cached_contexts = sampler._get_contexts(request_slots) + + expected = torch.tensor([[50, 51, 52], [-1, -1, 40]]) + assert torch.equal(cold_contexts, expected) + assert torch.equal(cached_contexts, expected) def test_gpu_sampler_skips_watermarking_for_repeated_contexts(monkeypatch): diff --git a/tools/build_deepgemm_C.py b/tools/build_deepgemm_C.py index f4438c6755..36824318c0 100644 --- a/tools/build_deepgemm_C.py +++ b/tools/build_deepgemm_C.py @@ -1,85 +1,53 @@ # SPDX-License-Identifier: Apache-2.0 # SPDX-FileCopyrightText: Copyright contributors to the vLLM project -"""Build DeepGEMM's `_C` pybind11 extension for . +"""Build DeepGEMM's TORCH_LIBRARY extension and copy vendored artifacts. -Driven from cmake/external_projects/deepgemm.cmake. The driver runs against -the build interpreter's torch; is only consulted for INCLUDEPY -and SOABI, so target venvs don't need torch installed. +DeepGEMM now registers ops via TORCH_LIBRARY into ``deep_gemm._C_extension`` +(abi3) and exposes the legacy API through ``deep_gemm/_C.py``. This driver +delegates to DeepGEMM's ``setup.py build_ext --inplace`` and copies the shim +plus extension into the cmake output directory. -Usage: python build_deepgemm_C.py +Usage: python build_deepgemm_C.py """ -import json import os +import shutil import subprocess import sys from pathlib import Path -import torch -from torch.utils import cpp_extension - -if len(sys.argv) != 4: - sys.exit(f"usage: {sys.argv[0]} ") +if len(sys.argv) != 3: + sys.exit(f"usage: {sys.argv[0]} ") src = Path(sys.argv[1]).resolve() out = Path(sys.argv[2]).resolve() -target_py = sys.argv[3] +_pkg = src / "deep_gemm" out.mkdir(parents=True, exist_ok=True) -info = json.loads( - subprocess.check_output( - [ - target_py, - "-c", - "import sysconfig, json; " - "print(json.dumps({k: sysconfig.get_config_var(k) " - "for k in ('EXT_SUFFIX', 'INCLUDEPY')}))", - ] - ).decode() +if not (_pkg / "_C.py").is_file(): + sys.exit( + f"DeepGEMM source at {src} is missing deep_gemm/_C.py; " + "expected TORCH_LIBRARY migration layout" + ) + +# Avoid DeepGEMM's clean-git assertion when vendoring a local dirty tree. +env = os.environ.copy() +env.pop("DG_SKIP_CUDA_BUILD", None) + +print(f"[build_deepgemm_C] building in {src} with {sys.executable}", flush=True) +subprocess.check_call( + [sys.executable, "setup.py", "build_ext", "--inplace"], + cwd=src, + env=env, ) -cuda_home = cpp_extension.CUDA_HOME -if cuda_home is None: - sys.exit("CUDA_HOME not found; cannot build DeepGEMM _C") -# CCCL lives outside the standard CUDAToolkit search (mirrors DeepGEMM's setup.py). -includes = [ - info["INCLUDEPY"], - f"{cuda_home}/include", - f"{cuda_home}/include/cccl", - str(src / "csrc"), - str(src / "deep_gemm/include"), - str(src / "third-party/cutlass/include"), - str(src / "third-party/cutlass/tools/util/include"), - str(src / "third-party/deep_jit/include"), - *cpp_extension.include_paths(device_type="cuda"), -] +shim = _pkg / "_C.py" +shutil.copy2(shim, out / shim.name) -cmd = [ - os.environ.get("CXX", "g++"), - "-shared", - "-fPIC", - "-std=c++20", - "-O3", - "-g0", - "-Wno-psabi", - "-Wno-deprecated-declarations", - "-DTORCH_API_INCLUDE_EXTENSION_H", - "-DTORCH_EXTENSION_NAME=_C", - f"-D_GLIBCXX_USE_CXX11_ABI={int(torch.compiled_with_cxx11_abi())}", - *(f"-I{p}" for p in includes), - str(src / "csrc/python_api.cpp"), - *(f"-L{p}" for p in cpp_extension.library_paths(device_type="cuda")), - f"-L{cuda_home}/lib64", - "-ltorch", - "-ltorch_python", - "-ltorch_cpu", - "-ltorch_cuda", - "-lc10", - "-lc10_cuda", - "-lcudart", - "-lnvrtc", - "-o", - str(out / f"_C{info['EXT_SUFFIX']}"), -] -print("[build_deepgemm_C] " + " ".join(cmd), flush=True) -subprocess.check_call(cmd) +so = _pkg / "_C_extension.abi3.so" +if not so.is_file(): + sys.exit(f"DeepGEMM build did not produce {so}") +for stale in out.glob("_C_extension*.so"): + stale.unlink() +shutil.copy2(so, out / so.name) +print(f"[build_deepgemm_C] installed {so.name} -> {out}", flush=True) diff --git a/tools/build_nixl_with_torch215_for_rubin.py b/tools/build_nixl_with_torch215_for_rubin.py new file mode 100644 index 0000000000..ee9fa72ae9 --- /dev/null +++ b/tools/build_nixl_with_torch215_for_rubin.py @@ -0,0 +1,619 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +# +# TODO(arpera): +# This script is temporary. It builds nixl from source code against torch 2.15 +# because currently there is no nixl wheel built against torch 2.15 which +# vLLM docker image for Rubin requires. +# Remove this script from vLLM once there is nixl wheel built against torch 2.15. + +import argparse +import glob +import os +import shutil +import subprocess +import sys +import urllib.request + +# --- Configuration --- +WHEELS_CACHE_HOME = os.environ.get("WHEELS_CACHE_HOME", "/tmp/wheels_cache") +ROOT_DIR = os.path.dirname(os.path.abspath(__file__)) +UCX_DIR = os.path.join("/tmp", "ucx_source") +NIXL_DIR = os.path.join("/tmp", "nixl_source") +UCX_INSTALL_DIR = os.path.join("/tmp", "ucx_install") +NIXL_BUILD_DIR = os.path.join("/tmp", "nixl_build") +BUILD_TOOLS_DIR = os.path.join("/tmp", "nixl_build_tools") +UCX_REPO_URL = "https://github.com/openucx/ucx.git" +NIXL_REPO_URL = "https://github.com/ai-dynamo/nixl.git" +UCX_REF = os.environ.get("UCX_REF", "v1.23.x") +TEMP_WHEEL_DIR = os.path.join("/tmp", "nixl_temp_wheelhouse") +NIXL_REQUIREMENTS_FILE = os.environ.get( + "NIXL_REQUIREMENTS_FILE", + os.path.join(ROOT_DIR, os.pardir, "requirements", "kv_connectors.txt"), +) +MAX_JOBS = int(os.environ.get("MAX_JOBS") or os.cpu_count() or 1) +CUDA_HOME = os.environ.get("CUDA_HOME", "/usr/local/cuda") +# SM targets NIXL EP can run on (it requires sm_90 or newer). +NIXL_EP_CUDA_ARCHS = ("90", "100", "103", "107", "110", "120") + +# Libraries that must come from the host or from torch, not from the wheel. +AUDITWHEEL_EXCLUDES = [ + "libcuda*", + "libcufile*", + "libcuobjclient*", + "libssl*", + "libcrypto*", + "libefa*", + "libhwloc*", + "libfabric*", + "libtorch*", + "libc10*", + "libdoca*", + "libred_client*", + "libred_async*", + "liblz4*", +] + + +# --- Helper Functions --- +def get_latest_nixl_version(): + """Helper function to get latest release version of NIXL. + + Raises: + RuntimeError: If the latest release cannot be determined. + + """ + # /releases/latest redirects to the tag page. Unlike the REST API it is not + # rate limited for anonymous clients, which are shared on compute nodes. + request = urllib.request.Request( + f"{NIXL_REPO_URL.removesuffix('.git')}/releases/latest", method="HEAD" + ) + try: + with urllib.request.urlopen(request, timeout=30) as response: + return response.geturl().rsplit("/", 1)[-1] + except OSError as e: + raise RuntimeError( + "Cannot determine the latest NIXL release, set NIXL_VERSION." + ) from e + + +def get_pinned_nixl_version(): + """Returns the NIXL version pinned in the vLLM requirements, if there is one.""" + try: + with open(NIXL_REQUIREMENTS_FILE) as requirements: + for line in requirements: + requirement = line.split("#")[0].split(";")[0] + name, separator, version = requirement.partition("==") + if separator and name.strip() == "nixl" and version.split(): + return version.split()[0] + except OSError: + pass + return None + + +def resolve_nixl_version(): + """Returns the NIXL version to build and where it comes from.""" + if os.environ.get("NIXL_VERSION"): + return os.environ["NIXL_VERSION"], "the NIXL_VERSION environment variable" + pinned_version = get_pinned_nixl_version() + if pinned_version: + return pinned_version, os.path.normpath(NIXL_REQUIREMENTS_FILE) + return get_latest_nixl_version(), "the latest NIXL release" + + +NIXL_VERSION, NIXL_VERSION_SOURCE = resolve_nixl_version() +# Version as it appears in wheel file names (tags may carry a "v" prefix). +NIXL_WHEEL_VERSION = NIXL_VERSION.removeprefix("v") + + +def run_command(command, cwd=".", env=None): + """Helper function to run a shell command and check for errors.""" + print(f"--> Running command: {' '.join(command)} in '{cwd}'", flush=True) + subprocess.check_call(command, cwd=cwd, env=env) + + +def is_pip_package_installed(package_name): + """Checks if a package is installed via pip without raising an exception.""" + result = subprocess.run( + [sys.executable, "-m", "pip", "show", package_name], + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + ) + return result.returncode == 0 + + +def find_nixl_wheel_in_cache(cache_dir): + """Finds the nixl-cuXX wheel file in the specified cache directory.""" + # The repaired wheel will have a 'manylinux' tag, but this glob still works. + search_pattern = os.path.join(cache_dir, f"nixl_*{NIXL_WHEEL_VERSION}*.whl") + wheels = glob.glob(search_pattern) + if wheels: + # Sort to get the most recent/highest version if multiple exist + wheels.sort() + return wheels[-1] + return None + + +def find_nixl_meta_wheel(wheel_dir): + """Finds the `nixl` meta package wheel (the nixl_cuXX dispatcher).""" + wheels = glob.glob(os.path.join(wheel_dir, f"nixl-{NIXL_WHEEL_VERSION}-*.whl")) + return wheels[-1] if wheels else None + + +def get_cuda_version(): + """Returns the (major, minor) CUDA toolkit version, or None without CUDA.""" + nvcc = os.path.join(CUDA_HOME, "bin", "nvcc") + if not os.path.exists(nvcc): + return None + output = subprocess.check_output([nvcc, "--version"], text=True) + _, separator, release = output.partition("release ") + try: + if not separator: + raise ValueError + major, minor = release.split(",")[0].split(".")[:2] + return int(major), int(minor) + except ValueError: + raise RuntimeError( + f"Cannot parse the CUDA version from `{nvcc} --version`." + ) from None + + +def get_default_cuda_arch_list(): + """Returns the SM targets of NIXL EP that the installed nvcc can compile.""" + nvcc = os.path.join(CUDA_HOME, "bin", "nvcc") + supported = subprocess.check_output([nvcc, "--list-gpu-code"], text=True).split() + return ",".join(sm for sm in NIXL_EP_CUDA_ARCHS if f"sm_{sm}" in supported) + + +def is_torch_available(): + """Checks whether torch can be imported by the current interpreter.""" + result = subprocess.run( + [sys.executable, "-c", "import torch"], + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + ) + return result.returncode == 0 + + +def is_nixl_installed(build_nixl_ep): + """Checks that the requested NIXL version, and its EP extension for the + installed torch when requested, is already installed.""" + cuda_version = get_cuda_version() + if cuda_version is None: + return is_pip_package_installed("nixl") + checks = [ + "import importlib, importlib.metadata as m", + f"assert m.version('nixl') == '{NIXL_WHEEL_VERSION}'", + f"assert m.version('nixl-cu{cuda_version[0]}') == '{NIXL_WHEEL_VERSION}'", + ] + if build_nixl_ep: + # The package picks its extension by torch version and fails to import + # when none was built for the installed torch. + checks.append(f"importlib.import_module('nixl_ep_cu{cuda_version[0]}')") + result = subprocess.run( + [sys.executable, "-c", "\n".join(checks)], + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + ) + return result.returncode == 0 + + +def install_system_dependencies(build_nixl_ep=False): + """Installs required system packages using apt-get if run as root.""" + if shutil.which("apt-get") is None: + print("--> apt-get not found, relying on the build tools in PATH.", flush=True) + return + if os.geteuid() != 0: + print("\n---", flush=True) + print( + "WARNING: Not running as root. \ + Skipping system dependency installation.", + flush=True, + ) + print( + "Please ensure the listed packages are installed on your system:", + flush=True, + ) + print( + " patchelf build-essential git cmake ninja-build \ + autotools-dev automake libtool libtool-bin \ + libibverbs-dev", + flush=True, + ) + print("---\n", flush=True) + return + + print("--- Running as root. Installing system dependencies... ---", flush=True) + apt_packages = [ + "patchelf", # <-- Add patchelf here + "build-essential", + "git", + "cmake", + "ninja-build", + "autotools-dev", + "automake", + "libtool", + "libtool-bin", + "pkg-config", + "libibverbs-dev", + ] + # -base CUDA images lack the headers UCX (NVML) and the torch extension + # (cuSPARSE/cuSOLVER/cuBLAS, included by ATen/cuda/CUDAContext.h) need. + cuda_version = get_cuda_version() + if cuda_version is not None: + major, minor = cuda_version + if not os.path.exists(os.path.join(CUDA_HOME, "include", "nvml.h")): + apt_packages.append(f"cuda-nvml-dev-{major}-{minor}") + if build_nixl_ep: + apt_packages += [ + f"{package}-{major}-{minor}" + for package in ("libcusparse-dev", "libcublas-dev", "libcusolver-dev") + ] + run_command(["apt-get", "update"]) + run_command(["apt-get", "install", "-y"] + apt_packages) + print("--- System dependencies installed successfully. ---\n", flush=True) + + +def install_python_build_dependencies(): + """Installs the Python tools needed to build and repair the NIXL wheel.""" + run_command( + [ + sys.executable, + "-m", + "pip", + "install", + "auditwheel", + "meson-python", + "ninja", + "patchelf", + "pybind11", + "pyyaml", + "setuptools>=80.9.0", + "tomlkit", + ] + ) + # An older meson already on the system (e.g. from the distribution) makes + # meson-python fail to package NIXL and pip would keep it, so use our own. + run_command( + [ + sys.executable, + "-m", + "pip", + "install", + "--upgrade", + f"--target={BUILD_TOOLS_DIR}", + "meson", + ] + ) + + +def build_ucx(ucx_install_path): + """Builds UCX from source, with CUDA support if CUDA is installed.""" + if not os.path.exists(UCX_DIR): + run_command(["git", "clone", UCX_REPO_URL, UCX_DIR]) + ucx_source_path = os.path.abspath(UCX_DIR) + run_command(["git", "checkout", UCX_REF], cwd=ucx_source_path) + run_command(["./autogen.sh"], cwd=ucx_source_path) + configure_command = [ + "./contrib/configure-release-mt", + f"--prefix={ucx_install_path}", + "--enable-shared", + "--disable-static", + "--disable-doxygen-doc", + "--enable-experimental-api", + "--enable-optimizations", + "--without-avx", + "--enable-cma", + "--enable-devel-headers", + *([f"--with-cuda={CUDA_HOME}"] if get_cuda_version() else []), + "--with-verbs", + "--without-gdrcopy", + "--without-dc", + "--without-rdmacm", + "--without-gga", + "--with-ze=no", + ] + run_command(configure_command, cwd=ucx_source_path) + run_command(["make", "-j", str(MAX_JOBS)], cwd=ucx_source_path) + run_command(["make", "install"], cwd=ucx_source_path) + + +def build_nixl_wheel(ucx_path, build_env, wheel_dir, args): + """Builds the (unrepaired) NIXL wheel against the UCX installed in ucx_path.""" + if not os.path.exists(NIXL_DIR): + run_command(["git", "clone", NIXL_REPO_URL, NIXL_DIR]) + else: + run_command(["git", "fetch", "--tags"], cwd=NIXL_DIR) + nixl_source_path = os.path.abspath(NIXL_DIR) + # Tags are "v" for current releases and bare for older ones. + for tag in (f"v{NIXL_WHEEL_VERSION}", NIXL_WHEEL_VERSION): + tag_exists = subprocess.run( + ["git", "rev-parse", "--verify", "--quiet", f"refs/tags/{tag}"], + cwd=nixl_source_path, + stdout=subprocess.DEVNULL, + ) + if tag_exists.returncode == 0: + run_command(["git", "checkout", "--force", tag], cwd=nixl_source_path) + print(f"--> Checked out NIXL version: {tag}", flush=True) + break + else: + raise RuntimeError(f"NIXL has no tag for version {NIXL_WHEEL_VERSION}.") + + cuda_version = get_cuda_version() + if cuda_version is not None: + # The wheel name selects which of nixl-cu12 / nixl-cu13 is built. + run_command( + [ + sys.executable, + "contrib/tomlutil.py", + "--wheel-name", + f"nixl-cu{cuda_version[0]}", + "pyproject.toml", + ], + cwd=nixl_source_path, + ) + + meson_options = { + "build_tests": "false", + "ucx_path": ucx_path, + } + cuda_arch_list = args.cuda_arch_list + if args.build_nixl_ep: + meson_options["build_nixl_ep"] = "true" + meson_options["build_examples"] = "true" + cuda_arch_list = cuda_arch_list or get_default_cuda_arch_list() + if cuda_arch_list: + meson_options["nixl_cuda_arch_list"] = cuda_arch_list + if args.nixl_plugins: + meson_options["enable_plugins"] = args.nixl_plugins + config_settings = [f"-Csetup-args=-D{k}={v}" for k, v in meson_options.items()] + # Keep the build tree: auditwheel needs the subproject libraries it holds. + config_settings.append(f"-Cbuild-dir={NIXL_BUILD_DIR}") + config_settings.append(f"-Ccompile-args=-j{MAX_JOBS}") + + run_command( + [ + sys.executable, + "-m", + "pip", + "wheel", + ".", + "--no-deps", + # Build against the torch already installed, not a fresh isolated one. + "--no-build-isolation", + f"--wheel-dir={wheel_dir}", + *config_settings, + ], + cwd=nixl_source_path, + env=build_env, + ) + + +def repair_nixl_wheel(unrepaired_wheel, repaired_dir, ucx_install_path, build_env): + """Bundles the shared libraries and UCX modules into a self-contained wheel.""" + auditwheel_command = ["auditwheel", "repair"] + for pattern in AUDITWHEEL_EXCLUDES: + auditwheel_command += ["--exclude", pattern] + auditwheel_command += [unrepaired_wheel, f"--wheel-dir={repaired_dir}"] + # The wheel ships prometheus-cpp as libcore.so.1.3.0 but its users ask for the + # soname libcore.so.1.3, which only exists as a symlink in the build tree. + repair_env = build_env.copy() + prometheus_dir = os.path.join(NIXL_BUILD_DIR, "subprojects", "prometheus-cpp") + repair_env["LD_LIBRARY_PATH"] = f"{prometheus_dir}:{build_env['LD_LIBRARY_PATH']}" + run_command(auditwheel_command, env=repair_env) + + repaired_wheel = find_nixl_wheel_in_cache(repaired_dir) + if not repaired_wheel: + raise RuntimeError("Failed to find the repaired NIXL wheel.") + # UCX loads its transports (CUDA, IB, ...) with dlopen, so auditwheel does not + # see them as dependencies; the NIXL plugin itself is already in the wheel. + run_command( + [ + sys.executable, + "contrib/wheel_add_ucx_plugins.py", + "--skip-nixl-plugins", + f"--ucx-plugins-dir={os.path.join(ucx_install_path, 'lib', 'ucx')}", + repaired_wheel, + ], + cwd=os.path.abspath(NIXL_DIR), + env=build_env, + ) + return repaired_wheel + + +def install_nixl_wheels(wheels): + """Installs the NIXL wheels and drops the nixl-cuXX of other CUDA majors.""" + # --force-reinstall: the wheels usually have the version of the release that + # is already installed, which pip would otherwise keep. + # w/o "no-deps", it will install cuda-torch + run_command( + [ + sys.executable, + "-m", + "pip", + "install", + "--force-reinstall", + "--no-deps", + *wheels, + ] + ) + + cuda_version = get_cuda_version() + if cuda_version is None: + return + # The dispatcher only loads nixl-cu, and the meta + # package built here does not depend on the other variant. + for cuda_major in (12, 13): + package = f"nixl-cu{cuda_major}" + if cuda_major != cuda_version[0] and is_pip_package_installed(package): + run_command([sys.executable, "-m", "pip", "uninstall", "-y", package]) + + +def build_and_install_prerequisites(args): + """Builds UCX and NIXL from source, creating a self-contained wheel.""" + print(f"--> NIXL version {NIXL_WHEEL_VERSION} (from {NIXL_VERSION_SOURCE})") + if args.build_nixl_ep is None: + args.build_nixl_ep = get_cuda_version() is not None and is_torch_available() + reuse_existing = not (args.force_reinstall or args.build_only) + if reuse_existing and is_nixl_installed(args.build_nixl_ep): + print("--> NIXL is already installed. Nothing to do.", flush=True) + return + + cached_wheel = find_nixl_wheel_in_cache(WHEELS_CACHE_HOME) + if reuse_existing and cached_wheel: + print( + f"\n--> Found self-contained wheel: \ + {os.path.basename(cached_wheel)}.", + flush=True, + ) + print("--> Installing from cache, skipping all source builds.", flush=True) + cached_wheels = [cached_wheel] + cached_meta_wheel = find_nixl_meta_wheel(WHEELS_CACHE_HOME) + if cached_meta_wheel: + cached_wheels.append(cached_meta_wheel) + install_nixl_wheels(cached_wheels) + print("\n--- Installation from cache complete. ---", flush=True) + return + + print( + "\n--> No installed package or cached wheel found. \ + Starting full build process...", + flush=True, + ) + if not args.skip_system_deps: + install_system_dependencies(args.build_nixl_ep) + print("\n--> Installing Python build dependencies...", flush=True) + install_python_build_dependencies() + print(f"--> Using wheel cache directory: {WHEELS_CACHE_HOME}", flush=True) + os.makedirs(WHEELS_CACHE_HOME, exist_ok=True) + + # -- Step 1: Build UCX from source, unless an existing install is given -- + # With an external UCX (e.g. HPC-X) the wheel keeps depending on it at runtime. + external_ucx = args.ucx_path is not None + if external_ucx: + ucx_install_path = os.path.abspath(args.ucx_path) + print(f"\n[1/3] Using the existing UCX in {ucx_install_path}", flush=True) + else: + ucx_install_path = os.path.abspath(UCX_INSTALL_DIR) + print("\n[1/3] Configuring and building UCX from source...", flush=True) + build_ucx(ucx_install_path) + print("--- UCX build and install complete ---", flush=True) + + # -- Step 2: Build NIXL wheel from source -- + print("\n[2/3] Building NIXL wheel from source...", flush=True) + ucx_lib_path = os.path.join(ucx_install_path, "lib") + ucx_plugin_path = os.path.join(ucx_lib_path, "ucx") + build_env = os.environ.copy() + # pip-installed build tools live next to the running interpreter. + build_env["PATH"] = os.pathsep.join( + [ + os.path.join(BUILD_TOOLS_DIR, "bin"), + os.path.join(CUDA_HOME, "bin"), + os.path.dirname(sys.executable), + build_env["PATH"], + ] + ) + build_env["PYTHONPATH"] = os.pathsep.join( + filter(None, [BUILD_TOOLS_DIR, build_env.get("PYTHONPATH")]) + ) + build_env["PKG_CONFIG_PATH"] = os.path.join(ucx_lib_path, "pkgconfig") + existing_ld_path = os.environ.get("LD_LIBRARY_PATH", "") + build_env["LD_LIBRARY_PATH"] = ( + f"{ucx_lib_path}:{ucx_plugin_path}:{existing_ld_path}".strip(":") + ) + print(f"--> Using LD_LIBRARY_PATH: {build_env['LD_LIBRARY_PATH']}", flush=True) + + build_nixl_wheel(ucx_install_path, build_env, TEMP_WHEEL_DIR, args) + + # -- Step 3: Repair the wheel by copying UCX libraries -- + unrepaired_wheel = find_nixl_wheel_in_cache(TEMP_WHEEL_DIR) + if not unrepaired_wheel: + raise RuntimeError("Failed to find the NIXL wheel after building it.") + if external_ucx: + print("\n[3/3] Skipping the repair step: UCX is provided externally.") + built_wheels = [unrepaired_wheel] + else: + print("\n[3/3] Repairing NIXL wheel to include UCX libraries...", flush=True) + built_wheels = [ + repair_nixl_wheel( + unrepaired_wheel, + os.path.join(TEMP_WHEEL_DIR, "repaired"), + ucx_install_path, + build_env, + ) + ] + + meta_wheel = find_nixl_meta_wheel( + os.path.join(NIXL_BUILD_DIR, "src", "bindings", "python", "nixl-meta") + ) + if meta_wheel: + built_wheels.append(meta_wheel) + else: + print("WARNING: the nixl meta package was not built.", flush=True) + + new_wheels = [] + for wheel in built_wheels: + run_command(["cp", wheel, WHEELS_CACHE_HOME]) + new_wheels.append(os.path.join(WHEELS_CACHE_HOME, os.path.basename(wheel))) + run_command(["rm", "-rf", TEMP_WHEEL_DIR]) + + wheel_names = ", ".join(os.path.basename(w) for w in new_wheels) + if args.build_only: + print(f"--> Built wheels in {WHEELS_CACHE_HOME}: {wheel_names}", flush=True) + return + print( + f"--> Successfully built wheels: {wheel_names}. Now installing...", flush=True + ) + install_nixl_wheels(new_wheels) + print("--- NIXL installation complete ---", flush=True) + + +if __name__ == "__main__": + parser = argparse.ArgumentParser( + description="Build and install UCX and NIXL dependencies." + ) + parser.add_argument( + "--force-reinstall", + action="store_true", + help="Force rebuild and reinstall of UCX and NIXL \ + even if they are already installed.", + ) + parser.add_argument( + "--build-only", + action="store_true", + help="Build the wheels into WHEELS_CACHE_HOME and stop, without \ + installing them or looking at what is already installed.", + ) + parser.add_argument( + "--skip-system-deps", + action="store_true", + help="Do not install system packages with apt-get; the build tools \ + (git, autotools, cmake, patchelf, ...) must already be on PATH.", + ) + parser.add_argument( + "--build-nixl-ep", + action=argparse.BooleanOptionalAction, + default=None, + help="Build the NIXL EP extension for the installed torch version. \ + Defaults to on when CUDA and torch are available.", + ) + parser.add_argument( + "--cuda-arch-list", + default=os.environ.get("NIXL_CUDA_ARCH_LIST"), + help="Comma-separated CUDA SM targets for NIXL, e.g. 100,103,107. \ + Defaults to the NIXL EP targets the installed nvcc supports.", + ) + parser.add_argument( + "--nixl-plugins", + default=os.environ.get("NIXL_PLUGINS", "UCX"), + help="Comma-separated NIXL plugins to build. Pass an empty string to \ + build every plugin whose dependencies are available.", + ) + parser.add_argument( + "--ucx-path", + default=os.environ.get("UCX_PATH"), + help="Prefix of an existing UCX to build against instead of building one. \ + The resulting wheel is not self-contained.", + ) + args = parser.parse_args() + build_and_install_prerequisites(args) diff --git a/tools/build_rust.py b/tools/build_rust.py index 320cf75e3d..b5c3945d63 100644 --- a/tools/build_rust.py +++ b/tools/build_rust.py @@ -55,14 +55,15 @@ def rust_extensions(*, optional: bool = False) -> list[RustExtension]: binding=Binding.Exec, optional=optional, ), - RustExtension( - target="vllm._rust_tool_parser", - path="rust/src/parser/python/Cargo.toml", - features=["pyo3/abi3-py38"], - binding=Binding.PyO3, - optional=optional, - py_limited_api=True, - ), + # Example PyO3 module (the removed tool-parser bridge): + # RustExtension( + # target="vllm._rust_tool_parser", + # path="rust/src/parser/python/Cargo.toml", + # features=["pyo3/abi3-py38"], + # binding=Binding.PyO3, + # optional=optional, + # py_limited_api=True, + # ), ] diff --git a/tools/check_wheel_deepgemm.py b/tools/check_wheel_deepgemm.py index 6f8a03ffd3..fcc35ead58 100644 --- a/tools/check_wheel_deepgemm.py +++ b/tools/check_wheel_deepgemm.py @@ -1,41 +1,45 @@ # SPDX-License-Identifier: Apache-2.0 # SPDX-FileCopyrightText: Copyright contributors to the vLLM project -"""Assert the installed vLLM has a `_C.cpython-X.Y-*.so` for every CPython -covered by `requires-python`. Fails closed if a Python's `.so` is missing -from the wheel — i.e. the regression that surfaced in #41476/#41512. +"""Assert the vendored DeepGEMM package has the TORCH_LIBRARY binding layout. -Run from a CI test job after vLLM is installed, e.g. the H100 deepgemm -kernel tests in .buildkite/test_areas/kernels.yaml. +Expects ``deep_gemm/_C.py`` plus a single ``_C_extension*.so`` (abi3) under +``vllm.third_party.deep_gemm``. Run after vLLM is installed, e.g. the H100 +deepgemm kernel tests in .buildkite/test_areas/kernels.yaml. """ import importlib.util -import os import sys from pathlib import Path -import regex as re -import tomllib -SO_RE = re.compile(r"^_C\.cpython-(\d)(\d+)-") +def main() -> int: + spec = importlib.util.find_spec("vllm.third_party.deep_gemm") + if spec is None or spec.origin is None: + print( + "vllm.third_party.deep_gemm not importable; is vllm installed?", + file=sys.stderr, + ) + return 1 + pkg_dir = Path(spec.origin).parent + + shim = pkg_dir / "_C.py" + so_files = sorted(pkg_dir.glob("_C_extension*.so")) + missing = [] + if not shim.is_file(): + missing.append("_C.py") + if len(so_files) != 1: + missing.append("exactly one _C_extension*.so") + + print( + f"deepgemm vendored binding: shim={shim.is_file()}, " + f"extensions={[p.name for p in so_files]}" + ) + if missing: + print(f"missing: {missing}", file=sys.stderr) + return 1 + return 0 -def required_pythons() -> list[str]: - pyproject = Path(__file__).resolve().parent.parent / "pyproject.toml" - spec = tomllib.loads(pyproject.read_text())["project"]["requires-python"] - m = re.match(r">=3\.(\d+),<3\.(\d+)", spec) - if not m: - sys.exit(f"unexpected requires-python format: {spec!r}") - return [f"3.{v}" for v in range(int(m[1]), int(m[2]))] - - -spec = importlib.util.find_spec("vllm.third_party.deep_gemm") -if spec is None or spec.origin is None: - sys.exit("vllm.third_party.deep_gemm not importable; is vllm installed?") -pkg_dir = Path(spec.origin).parent - -found = {f"{m[1]}.{m[2]}" for f in os.listdir(pkg_dir) if (m := SO_RE.match(f))} -required = required_pythons() -missing = [v for v in required if v not in found] -print(f"deepgemm _C: found {sorted(found)}, required {required}, missing {missing}") -sys.exit(1 if missing else 0) +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tools/ci/kernel_symbol_map.py b/tools/ci/kernel_symbol_map.py index 2a1e8964d0..c88f2d9ad5 100755 --- a/tools/ci/kernel_symbol_map.py +++ b/tools/ci/kernel_symbol_map.py @@ -255,7 +255,7 @@ def main() -> int: or os.environ.get("VLLM_BUILD_COMMIT") or "", "cuda": cuda_release(nvcc), - "generated_at": dt.datetime.now(dt.timezone.utc).isoformat(timespec="seconds"), + "generated_at": dt.datetime.now(dt.UTC).isoformat(timespec="seconds"), "build_dirs": [], "incomplete": False, "errors": [], diff --git a/tools/install_deepgemm.sh b/tools/install_deepgemm.sh index 949aa28da5..2fb2351a0c 100755 --- a/tools/install_deepgemm.sh +++ b/tools/install_deepgemm.sh @@ -10,7 +10,7 @@ DEEPGEMM_GIT_REPO="https://github.com/vllm-project/DeepGEMM.git" # NOTE: This targets the vLLM fork's dev branch, which carries the sm120 # and sm90 paged-MQA ports plus the SwiGLU alpha/beta and SiTU Mega MoE # activations, plus the CUDA 12.x layout header fix from vllm-project/DeepGEMM#12. -DEEPGEMM_GIT_REF="e1f418c2a4f20818221f6b0e578b4c2f634d4c3f" +DEEPGEMM_GIT_REF="1e1842a833699298f7afc02eefb2ed168fff6938" WHEEL_DIR="" # Parse command line arguments @@ -93,6 +93,8 @@ pushd "$INSTALL_DIR/deepgemm" # Checkout the specific reference git checkout "$DEEPGEMM_GIT_REF" +git submodule sync --recursive +git submodule update --init --recursive # Clean previous build artifacts # (Based on https://github.com/deepseek-ai/DeepGEMM/blob/main/install.sh) diff --git a/tools/pre_commit/check_forbidden_imports.py b/tools/pre_commit/check_forbidden_imports.py index 126d20094a..531f6fb8a2 100644 --- a/tools/pre_commit/check_forbidden_imports.py +++ b/tools/pre_commit/check_forbidden_imports.py @@ -88,7 +88,11 @@ CHECK_IMPORTS = { allowed_pattern=re.compile(r"^\s*import\s+pybase64(\s*|\s+as\s+base64\s*)$"), ), "re": ForbiddenImport( - pattern=r"^\s*(?:import\s+re(?:$|\s|,)|from\s+re\s+import)", + # regex has no equivalent of the private sre modules + pattern=( + r"^\s*(?:import\s+re(?:$|\s|,)" + r"|from\s+re\s+import(?!(?:\s*,?\s*_(?:constants|parser))+\s*(?:#.*)?$))" + ), tip="Replace 'import re' with 'import regex as re' or 'import regex'.", allowed_pattern=re.compile(r"^\s*import\s+regex(\s*|\s+as\s+re\s*)$"), allowed_files={"setup.py"}, diff --git a/tools/pre_commit/mypy.py b/tools/pre_commit/mypy.py index a6c8e46419..876e19e1fd 100755 --- a/tools/pre_commit/mypy.py +++ b/tools/pre_commit/mypy.py @@ -10,7 +10,7 @@ Usage: python tools/pre_commit/mypy.py Args: - python_version: Python version to use (e.g., "3.10") or "local" to use + python_version: Python version to use (e.g., "3.11") or "local" to use the local Python version. changed_files: List of changed files to check. @@ -38,10 +38,10 @@ SILENT_GROUPS = [ "tests/entrypoints/generate", "tests/entrypoints/multimodal", "tests/entrypoints/pooling", + "tests/entrypoints/rl", "tests/entrypoints/serve", "tests/entrypoints/tool_parsers", "tests/entrypoints/unit_tests", - "tests/entrypoints/weight_transfer", "tests/kernels/attention", "tests/kernels/core", "tests/kernels/helion", @@ -152,7 +152,7 @@ def mypy( Args: targets: List of files or directories to check. - python_version: Python version to use (e.g., "3.10") or None to use + python_version: Python version to use (e.g., "3.11") or None to use the default mypy version. follow_imports: Value for the --follow-imports option or None to use the default mypy behavior. diff --git a/tools/pre_commit/test_tethering_allowlist.txt b/tools/pre_commit/test_tethering_allowlist.txt index d7f19ec79a..e738a79154 100644 --- a/tools/pre_commit/test_tethering_allowlist.txt +++ b/tools/pre_commit/test_tethering_allowlist.txt @@ -48,17 +48,6 @@ tests/kernels/test_shuffle_rows.py # explicitly --ignore'd in kernels.yaml with tests/lora/test_deepseekv2_tp.py # --ignore'd from sharded LoRA run; no dedicated *_tp job tests/lora/test_qwen3moe_tp.py # --ignore'd from sharded LoRA run; no dedicated *_tp job -# tests/models/ - the models_*.yaml jobs run specific paths, not all of models/. -tests/models/deepseek_v32/test_sequence_parallel.py # no models_*.yaml job collects this path -tests/models/test_deepseek_v4_dspark_rocm.py # added by #52362 (2026-09-21), no models_*.yaml job collects this path -tests/models/test_deepseek_v4_fi_moe_ep.py # no models_*.yaml job collects this path -tests/models/test_deepseek_v4_rocm_compressor_gemm_fusion.py # no models_*.yaml job collects this path -tests/models/test_deepseek_v4_rocm_wo_a.py # added by #54894 (2026-09-20), no models_*.yaml job collects this path -tests/models/test_deepseek_v41_replay_start.py # added by #56227 (2026-09-18), no models_*.yaml job collects this path -tests/models/test_dspark_mla.py # no models_*.yaml job collects this path -tests/models/test_language_model_cache_is_weak.py # no models_*.yaml job collects this path -tests/models/transformers/test_layer_registry.py # no models_*.yaml job collects this path - # tests/v1/ - the v1 subtree is otherwise covered by many `pytest v1/...` jobs. tests/v1/ec_connector/integration/test_epd_correctness.py # run_epd_correctness_test.sh exists but no Buildkite job invokes it tests/v1/ec_connector/integration/test_nixl_failure.py # 2-GPU + NIXL E/PD integration test (#55290); no Buildkite job collects ec_connector/integration/ diff --git a/tools/recipes/README.md b/tools/recipes/README.md index 90ad33b3eb..37486ae548 100644 --- a/tools/recipes/README.md +++ b/tools/recipes/README.md @@ -19,7 +19,7 @@ flowchart LR ``` The recipe is the baseline. Hardware and workload information can optionally -refine the initial configuration. Sweep tuning is an optional validation step. +refine the initial configuration. Sweep tuning is optional. ## Getting Started @@ -54,7 +54,9 @@ For `xeon6`, the script enables hardware detection automatically. Generated `config.yml` and `env.sh` files remain in the directory where the script was invoked. -## 1. vLLM Recipes Only +## Detailed Deployment Workflow + +### 1. vLLM Recipes Only Use the converter directly when the recipe already contains the deployment settings you need. This path requires only PyYAML; the vLLM Python package is @@ -64,15 +66,7 @@ not required unless optional runtime tuning or sweep generation is requested. pip install pyyaml ``` -Choose whichever recipe-selection method fits the workflow: - -**Interactive discovery** — search models, then choose hardware and strategy: - -```bash -python3 tools/recipes/recipe_json_to_vllm_config.py -``` - -**Non-interactive discovery** — provide model and hardware and use the +For non-interactive discovery, provide the model and hardware and use the Recipes-recommended strategy: ```bash @@ -81,7 +75,17 @@ python3 tools/recipes/recipe_json_to_vllm_config.py \ --hardware xeon6 ``` -**Direct JSON input** — use a Recipes JSON URL or a local JSON file: +#### 1.1 Interactive Discovery + +Search models, then choose hardware and strategy interactively: + +```bash +python3 tools/recipes/recipe_json_to_vllm_config.py +``` + +#### 1.2 Direct JSON Input + +Use a Recipes JSON URL or a local JSON file: ```bash python3 tools/recipes/recipe_json_to_vllm_config.py \ @@ -90,15 +94,27 @@ python3 tools/recipes/recipe_json_to_vllm_config.py \ python3 tools/recipes/recipe_json_to_vllm_config.py recipe.json ``` -All paths generate `config.yml` and `env.sh`. See -[REFERENCE.md](REFERENCE.md) for recipe discovery, strategy selection, direct -JSON input, custom output files, and deployment scope. +#### 1.3 Test a Preview Recipe API -## 2. Hardware Information (Optional) +A Recipes pull request can expose the same JSON API through its Vercel preview. +Use `--api-base` to validate it before the recipe is available in production: -Add `--detect-hardware` when the target host's effective CPU/NUMA/memory -resources should refine deployment-sensitive values such as -`tensor-parallel-size` and `gpu-memory-utilization`. +```bash +PREVIEW=https://vllm-recipes-git-fork-intel-ai-tce-dockerin-f4c148-inferact-inc.vercel.app + +python3 tools/recipes/recipe_json_to_vllm_config.py \ + --api-base "$PREVIEW" \ + --model meta-llama/Llama-3.2-1B-Instruct \ + --hardware xeon6 +``` + +See [REFERENCE.md](REFERENCE.md) for recipe discovery, strategy selection, +custom output files, and deployment scope. + +### 2. Hardware Information (Optional) + +Add `--detect-hardware` when using the converter directly and the target host's +effective CPU/NUMA/memory resources should refine deployment-sensitive values: ```bash python3 tools/recipes/recipe_json_to_vllm_config.py \ @@ -107,14 +123,13 @@ python3 tools/recipes/recipe_json_to_vllm_config.py \ --detect-hardware ``` -Hardware detection is optional and uses vLLM CPU resource utilities only when -requested. See [RUNTIME_TUNING.md](RUNTIME_TUNING.md#hardware-information). +`serve_with_recipe.sh` enables this automatically for `xeon6`. See +[RUNTIME_TUNING.md](RUNTIME_TUNING.md#hardware-information) for details. -## 3. Workload Information (Optional) +### 3. Workload Information (Optional) -Workload hints can refine scheduler settings for one initial deployment -suggestion. Inputs include token lengths, concurrency, and optional latency or -capacity objectives. +Add workload hints when token lengths, concurrency, or latency objectives should +seed optional benchmark tuning: ```bash python3 tools/recipes/recipe_json_to_vllm_config.py \ @@ -127,17 +142,13 @@ python3 tools/recipes/recipe_json_to_vllm_config.py \ --tpot-sla-ms 100 ``` -Hardware detection and workload information are independent optional inputs; -they can also be supplied together. See -[RUNTIME_TUNING.md](RUNTIME_TUNING.md#workload-information) for the supported -inputs and how runtime parameters are calculated. +See [RUNTIME_TUNING.md](RUNTIME_TUNING.md#workload-information) for supported +inputs and runtime calculations. -## 4. Sweep Tuning (Optional) +### 4. Sweep Tuning (Optional) -Use `--generate-sweep` when the initial scheduler suggestion should be validated -with `vllm bench sweep serve`. The sweep benchmarks nearby scheduler values and -`recommend.py` produces one measured `recommended-config.yml` plus the -selection evidence in `recommendation.json`. +Use `--generate-full-sweep` for benchmark-backed tuning of TP/DP, concurrency, +and scheduler settings: ```bash python3 tools/recipes/recipe_json_to_vllm_config.py \ @@ -147,13 +158,15 @@ python3 tools/recipes/recipe_json_to_vllm_config.py \ --input-tokens 128 \ --output-tokens 128 \ --concurrency 32 \ - --generate-sweep + --generate-full-sweep ``` -See [SWEEP_TUNING.md](SWEEP_TUNING.md) for the benchmark, recommendation, and -vLLM CPU Docker-shell workflow. +See [SWEEP_TUNING.md](SWEEP_TUNING.md) for targeted sweep modes, NUMA binding, +failure handling, recommendations, reporting, and visualization. -## Start vLLM +### 5. Start vLLM + +When using the converter directly: ```bash source env.sh diff --git a/tools/recipes/RUNTIME_TUNING.md b/tools/recipes/RUNTIME_TUNING.md index d3a359446c..cf442a34d6 100644 --- a/tools/recipes/RUNTIME_TUNING.md +++ b/tools/recipes/RUNTIME_TUNING.md @@ -64,12 +64,10 @@ python3 tools/recipes/recipe_json_to_vllm_config.py \ --tpot-sla-ms 100 ``` -`--output-tokens` is used to estimate steady-state request turnover for -`max-num-batched-tokens`. When both `--target-qps` and `--tpot-sla-ms` are -available, they are also used to estimate prompt arrival pressure per scheduler -step. `--ttft-sla-ms` is collected but is not yet converted directly into a -batch-size formula because the relationship between TTFT and scheduler budget is -model- and hardware-dependent. +Scheduler calculation is opt-in with `--tune-scheduler`; workload hints do +not rewrite `max-num-seqs` or `max-num-batched-tokens` by default. The detailed +scheduler and staged-sweep behavior is documented in +[SWEEP_TUNING.md](SWEEP_TUNING.md). ## Deployment-Time Parameters @@ -79,13 +77,32 @@ on a validation environment that differs from the user's target deployment. | Runtime parameter | Why it may need deployment-time refinement | Main decision input | Current policy | | --- | --- | --- | --- | -| `tensor-parallel-size` | The effective CPU/NUMA topology available to a container or pod can differ from the system used to validate the recipe. | Hardware topology | Use the largest power-of-two TP value that does not exceed the effective NUMA-node count. | +| `tensor-parallel-size` | Available topology and model head counts constrain TP. | Hardware topology + model configuration | Use the largest head-compatible power-of-two TP within the effective NUMA-node count, for DP=1 and PP=1. | | `gpu-memory-utilization` | Available memory can differ by machine size, container limits, and other memory use. The vLLM option name is also used by the CPU backend. | Hardware memory + recipe baseline | Calculate a conservative fraction from the most constrained NUMA node. | -| `max-num-seqs` | The useful scheduler concurrency depends on the number of requests expected to be active at the same time. | Workload concurrency | Set `max-num-seqs` to `--concurrency` when supplied. | -| `max-num-batched-tokens` | Each scheduler iteration must share its token budget between active decodes and incoming prefills. | Input/output token shape, concurrency, and optional QPS/TPOT | Calculate decode budget + expected prefill demand, with vLLM scheduler constraints as floors. | +| `max-num-seqs` | The useful scheduler concurrency depends on the number of requests expected to be active on each DP replica. | Workload concurrency + DP | Keep the recipe/vLLM value in `config.yml`; use `ceil(--concurrency / data-parallel-size)` as an explicit sweep seed. | +| `max-num-batched-tokens` | Each scheduler iteration must share its token budget between active decodes and incoming prefills. | Input/output token shape, concurrency, and optional QPS/TPOT | Keep the recipe/vLLM value in `config.yml`; use decode budget + expected prefill demand as an explicit sweep seed. | | `data-parallel-size` | The required replica count depends on the requested throughput and the capacity of one replica. | Capacity target | Keep the recipe value today because per-replica SLA capacity is not known. | -## How Each Runtime Parameter Is Calculated +For `max-num-seqs` and `max-num-batched-tokens`, workload formulas now create +benchmark-only seeds instead of overriding the directly deployable +`config.yml`. The optional benchmark sweep measures those explicit seeds +against vLLM's normal serving defaults. Default-reference runs leave the +selected scheduler option unset at engine-configuration time so vLLM can +resolve its platform-, world-size-, model-, and usage-context-aware default +rather than assuming a fixed numeric value. See +[SWEEP_TUNING.md](SWEEP_TUNING.md). + +This serving-default comparison is distinct from the `SchedulerConfig` fallback +constants used inside the initial tuning heuristics below. + +## How Initial Runtime Tuning Seeds Are Calculated + +The hardware formulas below produce directly deployable starting values. The +scheduler formulas produce explicit benchmark seeds and are not claimed to be +universal performance optima. This separation avoids baking workload guesses +into `config.yml` before measurements exist. The Tune All workflow benchmarks +TP/DP first, then concurrency, and finally scheduler settings before writing a +final recommendation. See [SWEEP_TUNING.md](SWEEP_TUNING.md). ### `tensor-parallel-size` @@ -99,6 +116,35 @@ For example, 2 effective NUMA nodes produce TP=2, while 6 nodes currently produce TP=4. This is a topology-based starting point and avoids automatically selecting unusual non-power-of-two TP sizes before they are validated. +The policy then reads the model configuration through vLLM's config loader, +without loading weights. It honors the model path/ID, revision, code revision, +`trust-remote-code`, config format and JSON `hf-overrides`. No model-specific +numeric head-count table is maintained. + +Each candidate must divide the attention-head count. KV heads must divide +across TP ranks when KV heads >= TP; otherwise TP must be a multiple of the KV +head count so the heads can be replicated. An incompatible candidate is halved +until it passes every detected constraint. Nested text configuration, global KV +heads, Whisper encoder/decoder heads and hybrid linear-attention heads are +checked. Vision head divisibility is checked conservatively unless +`mm-encoder-tp-mode: data` is configured. + +For Phi-4-reasoning on four NUMA nodes, the detected 40 attention heads and 10 +KV heads reduce the initial TP=4 suggestion to TP=2. The printed tuning notes +show the detected counts and the reduction; `config.yml` receives TP=2. + +If configuration loading or head extraction fails, the tool keeps the recipe +TP (vLLM's default TP=1 when absent) and prints why detection failed. That +fallback does not certify an existing recipe TP as compatible. Plain conversion +without hardware detection does not fetch the model configuration. + +Automatic TP selection currently requires recipe DP=1 and PP=1; otherwise the +recipe's parallelism is retained. Automatic DP tuning remains disabled pending +CPU DP binding and capacity validation. No unused NUMA nodes are converted into +DP replicas. These checks establish head compatibility, not memory fit or +complete model/backend support. Explicit `vllm serve` CLI overrides still take +precedence over the generated YAML and are validated by vLLM at startup. + ### `gpu-memory-utilization` For every effective NUMA memory node, the policy calculates: @@ -118,14 +164,20 @@ candidate = min(recipe value or 0.80, safe_fraction) ### `max-num-seqs` -When `--concurrency` is supplied, it directly represents the requested maximum -number of simultaneously active requests: +When `--concurrency` is supplied, it represents global client concurrency. Each +DP replica needs capacity for its expected share of those requests: ```text -max-num-seqs = concurrency +per_replica_concurrency = ceil(concurrency / data_parallel_size) +max-num-seqs = per_replica_concurrency ``` -If concurrency is not supplied, the converter does not override this parameter. +For DP=1 this reduces to `max-num-seqs = concurrency`. This value seeds the +scheduler benchmark; it is not added to `config.yml`. If concurrency is not +supplied, no seed is calculated. Benchmark results show +that setting `max-num-seqs` below per-replica concurrency can create request +queueing and severe TTFT degradation; scheduler sweeps therefore do not use +below-concurrency candidates. ### `max-num-batched-tokens` @@ -136,7 +188,7 @@ First, the policy determines the active sequence count: ```text active_sequences = - --concurrency + ceil(--concurrency / data_parallel_size) or recipe max-num-seqs or vLLM default max_num_seqs (128) ``` @@ -152,17 +204,19 @@ The policy then estimates how many new prompts need prefill work per scheduler step: ```text +per_replica_qps = target_qps / data_parallel_size + prefills_per_step = max( 1, active_sequences / output_tokens, - target_qps * tpot_sla_ms / 1000 + per_replica_qps * tpot_sla_ms / 1000 ) prefills_per_step = min(active_sequences, prefills_per_step) prefill_budget = ceil(input_tokens * prefills_per_step) ``` -The final scheduler budget is: +The final explicit scheduler sweep seed is: ```text max-num-batched-tokens = max( @@ -172,7 +226,7 @@ max-num-batched-tokens = max( ) ``` -For example, with 128 input tokens, 128 output tokens, and concurrency 32: +For example, with 128 input tokens, 128 output tokens, concurrency 32, and DP=1: ```text decode_budget = 32 @@ -181,6 +235,12 @@ prefill_budget = 128 candidate = max(2048, 32, 32 + 128) = 2048 ``` +The calculated token budget is a scheduler sweep seed and is not added to +`config.yml`. If P99 TTFT still +misses its objective while P99 TPOT has substantial headroom, Tune All should +reduce workload concurrency or select a different TP/DP layout rather than +reducing `max-num-seqs` below per-replica concurrency. + If chunked prefill is explicitly disabled and `max-model-len` is available, the policy also ensures: @@ -197,8 +257,10 @@ also needs measured per-replica throughput that still satisfies TTFT/TPOT: DP = ceil(target_qps / qps_per_replica_at_SLO) ``` -Because the converter does not have that measured capacity yet, it intentionally -keeps the recipe DP value rather than guessing. +Because the converter does not have that measured capacity yet, its initial +configuration intentionally keeps the recipe DP value rather than guessing. +Tune All subsequently benchmarks supported TP/DP layouts and replaces this seed +with a measured layout before concurrency and scheduler tuning. ## Precedence @@ -206,8 +268,11 @@ keeps the recipe DP value rather than guessing. vLLM defaults -> vLLM Recipes baseline -> hardware refinement (optional) - -> workload / SLO refinement (optional) - -> config.yml + env.sh + -> config.yml + env.sh + +workload / SLO hints (optional) + -> explicit scheduler sweep seed + -> benchmark-backed recommended-config.yml ``` ## Runtime-Tuning Hardware Scope @@ -283,10 +348,10 @@ with [SWEEP_TUNING.md](SWEEP_TUNING.md). ## Implementation - `hardware_detection.py` collects effective CPU/NUMA/memory information. -- `runtime_tuning.py` owns hardware-policy selection and independent parameter +- `sweep/runtime_tuning.py` owns hardware-policy selection and independent parameter tuning functions. - `recipe_json_to_vllm_config.py` resolves the recipe, collects optional inputs, applies the selected policy, and generates `config.yml` and `env.sh`. -For benchmark-backed validation of the initial scheduler suggestion, see +For benchmark-backed validation of the explicit scheduler seed, see [SWEEP_TUNING.md](SWEEP_TUNING.md). diff --git a/tools/recipes/SWEEP_TUNING.md b/tools/recipes/SWEEP_TUNING.md index e7282e122d..e39444aef9 100644 --- a/tools/recipes/SWEEP_TUNING.md +++ b/tools/recipes/SWEEP_TUNING.md @@ -1,95 +1,130 @@ # Sweep Tuning -Sweep tuning is optional. The converter always creates one initial `config.yml` -first. Users can deploy it immediately or benchmark nearby scheduler settings -and generate one measured recommendation. +Sweep tuning is optional. The converter always creates an initial `config.yml` +that can be deployed directly. Scheduler calculation is disabled by default: +TP/DP and concurrency stages remove explicit `max-num-seqs` and +`max-num-batched-tokens` so vLLM resolves its normal scheduler defaults. -## Generate - -```bash -python3 tools/recipes/recipe_json_to_vllm_config.py \ - --model meta-llama/Llama-3.1-8B-Instruct \ - --hardware xeon6 \ - --detect-hardware \ - --input-tokens 128 \ - --output-tokens 128 \ - --concurrency 32 \ - --ttft-sla-ms 3000 \ - --tpot-sla-ms 100 \ - --generate-sweep -``` - -The optional package is: +Use `--tune-scheduler` only when you explicitly want workload-derived scheduler +values before the dedicated scheduler stage. For benchmark-backed tuning, the +recommended workflow is **Tune All**, which measures the serving stack in +dependency order: ```text -sweep/ -├── serve_params.json -├── bench_params.json -├── run_sweep.sh -├── recommend.py -└── SWEEP.md +TP/DP -> concurrency -> scheduler ``` -The sweep varies only `max-num-seqs` and `max-num-batched-tokens`. It uses an -eight-point directed design: a batch-budget curve at the initial sequence count -plus lower/higher batch interactions at three-quarters and one-half of that -count. This gives broader coverage than a one-parameter-at-a-time sweep without -the cost of a full Cartesian grid. +The examples below assume tuning is run inside the target vLLM CPU container so +hardware detection sees the same CPU, NUMA, memory, and cgroup limits as the +deployment. -## Run and Recommend +## 1. Start the vLLM CPU Docker Shell + +From the vLLM source tree: ```bash -sweep/run_sweep.sh --dry-run -sweep/run_sweep.sh -sweep/recommend.py +mkdir -p recipe-output + +docker run --rm -it \ + --entrypoint bash \ + --security-opt seccomp=unconfined \ + --cap-add SYS_NICE \ + --shm-size=4g \ + -p 8000:8000 \ + -v "$PWD/tools/recipes:/recipes:ro" \ + -v "$PWD/recipe-output:/output" \ + -v "$HOME/.cache/huggingface:/root/.cache/huggingface" \ + -w /output \ + vllm/vllm-openai-cpu:latest-x86_64 ``` -Resume an interrupted sweep with: +`/recipes` is mounted read-only from the source tree and `/output` is writable. +Run the converter, generated sweep scripts, and hardware detection inside this +container. -```bash -sweep/run_sweep.sh --resume +For additional container details, see +[RUNTIME_TUNING.md](RUNTIME_TUNING.md#vllm-cpu-docker-shell). + +## 2. Recommended: Tune All + +Use `--generate-full-sweep` by default when you want benchmark-backed runtime +tuning. It measures TP/DP first, then the highest useful concurrency, then +scheduler parameters using the selected layout and concurrency. + +```mermaid +flowchart TD + I[["Input
Recipe + detected hardware
fixed input/output workload shape"]] + + A1("1. Tune TP/DP
tensor-parallel-size + data-parallel-size") + O1[/"parallel-layout-config.yml
selected TP/DP"/] + + A2("2. Tune concurrency
max_concurrency") + O2[/"concurrency-recommendation.json
recommended max_concurrency"/] + + A3("3. Tune scheduler
max-num-seqs + max-num-batched-tokens") + O3[/"recommended-config.yml
final vLLM server config"/] + + I --> A1 + A1 --> O1 + O1 --> A2 + A2 --> O2 + O2 --> A3 + A3 --> O3 ``` -`recommend.py` writes: +### Workflow options + +The table is ordered by the tuning sequence. **Tune All** is the recommended +default; the other modes are useful when only one part of the serving stack +needs to be re-measured. + +| Goal | Option | Tuning sequence | +| --- | --- | --- | +| **Tune all (recommended)** | `--generate-full-sweep` | **1. TP/DP -> 2. concurrency -> 3. scheduler** | +| Tune TP/DP only | `--generate-parallel-layout-sweep` | 1. TP/DP | +| Tune concurrency only | `--generate-concurrency-sweep` | 2. concurrency | +| Tune scheduler only | `--generate-scheduler-sweep` (`--generate-sweep` alias) | 3. scheduler | + +All workflows keep one fixed input/output workload shape and require +`--input-tokens`, `--output-tokens`, and `--concurrency`. + +For **Tune All**, the supplied `--concurrency` is the representative load used +for the initial TP/DP comparison and to size the benchmark request set. The +concurrency stage then measures the final SLA-feasible `max_concurrency`. + +### Scheduler defaults and `--tune-scheduler` + +By default, `--concurrency`, token lengths, and SLA values are workload hints; +they do not cause the converter to calculate `max-num-seqs` or +`max-num-batched-tokens`. This keeps scheduler capacity independent from the +seed concurrency while TP/DP and maximum concurrency are being measured. + +The default Tune All policy is: ```text -sweep/recommended-config.yml -sweep/recommendation.json +Stage 1: TP/DP -> vLLM scheduler defaults +Stage 2: concurrency -> vLLM scheduler defaults +Stage 3: scheduler -> calculate and benchmark scheduler candidates ``` -With TTFT/TPOT objectives, the benchmark uses vLLM `--goodput`. The recommender -calculates duration-weighted combined compliance across repeated runs and -requires both median P99 latency objectives and the minimum compliance ratio -(default `0.99`). Among eligible configurations it selects the highest mean -output-token throughput. Use `recommend.py --minimum-compliance VALUE` to -change the compliance requirement. - -If no configuration is eligible, `recommend.py` records the highest-goodput -configuration as `best_effort` in `recommendation.json`, does not write a -deployable `recommended-config.yml`, and exits with status 2. Without latency -objectives it selects highest mean output-token throughput. Failed benchmark -configurations are excluded in either mode. - -When an eligible configuration exists, `recommended-config.yml` changes only -the two swept scheduler parameters. `recommendation.json` records mean, median, -and worst-run P99 values, combined compliance, and the measured evidence for -every candidate. - -Deploy: +For a directly deployable heuristic configuration, or to intentionally carry +workload-derived scheduler values into the earlier sweep stages, opt in with: ```bash -source env.sh -vllm serve --config sweep/recommended-config.yml +--tune-scheduler ``` -## vLLM CPU Docker Shell +With that flag, the existing workload formulas are enabled. In particular, +`max-num-seqs` uses the per-replica workload concurrency and +`max-num-batched-tokens` uses the estimated decode and prefill pressure. -Use the common CPU container setup in -[RUNTIME_TUNING.md](RUNTIME_TUNING.md#vllm-cpu-docker-shell). It ensures -hardware detection runs against the same effective CPU, NUMA, memory, and -cgroup limits used by the deployment. +A scheduler-only sweep is also an explicit opt-in to scheduler calculation: it +uses the workload formulas as benchmark seeds and compares them with vLLM +defaults. -Inside that container, generate the initial suggestion plus sweep package: +### Generate the Tune All package + +Inside the container: ```bash python3 /recipes/recipe_json_to_vllm_config.py \ @@ -103,35 +138,540 @@ python3 /recipes/recipe_json_to_vllm_config.py \ --tpot-sla-ms 100 \ --config-out /output/config.yml \ --env-out /output/env.sh \ - --generate-sweep \ + --generate-full-sweep \ --sweep-out-dir /output/sweep ``` -The initial configuration can be deployed directly: +The initial configuration remains directly deployable: ```bash source /output/env.sh vllm serve --config /output/config.yml ``` -Stop that manually started server before running the sweep, because -`run_sweep.sh` starts and stops its own vLLM servers. +Before measurements, `config.yml` retains the recipe/vLLM scheduler defaults. +The sweep package separately carries explicit workload-derived values for +`max-num-seqs` and `max-num-batched-tokens`, so the benchmark can compare both +policies without making the workload heuristic the deployment default. + +Stop a manually started server before running the sweep because the generated +sweep scripts start and stop their own vLLM servers. + +Run all tuning stages: ```bash -/output/sweep/run_sweep.sh --dry-run -/output/sweep/run_sweep.sh -/output/sweep/recommend.py +/output/sweep/run_full_sweep.sh ``` -Inspect: +The end-to-end flow produces intermediate recommendations for TP/DP and +concurrency and finishes with: + +```text +/output/sweep/recommended-config.yml +/output/sweep/recommendation.json +/output/sweep/sweep-report.html +``` + +Inspect the final recommendation: ```bash cat /output/sweep/recommendation.json ``` -Deploy the measured result: +## 3. Tune All Stage Details + +### Stage 1: Tune TP/DP + +The parallel-layout stage measures NUMA-aware combinations of: + +- `tensor-parallel-size` +- `data-parallel-size` + +Primary candidates use all effective NUMA nodes: + +```text +tensor-parallel-size * data-parallel-size = effective NUMA nodes +``` + +The effective NUMA-node count comes from hardware detection; it is not fixed by +the sweep generator. The table below shows representative detected topologies. + +TP is restricted to the supported values `1`, `2`, `4`, and `8`. The sweep also +includes the largest supported TP size that does not exceed the effective +NUMA-node count, even if that layout leaves some NUMA nodes idle. + +| Effective NUMA nodes | Generated layouts | +| ---: | --- | +| 2 | `TP=2, DP=1`; `TP=1, DP=2` | +| 4 | `TP=4, DP=1`; `TP=2, DP=2`; `TP=1, DP=4` | +| 6 | `TP=4, DP=1` (4 of 6 nodes); `TP=2, DP=3`; `TP=1, DP=6` | +| 8 | `TP=8, DP=1`; `TP=4, DP=2`; `TP=2, DP=4`; `TP=1, DP=8` | + +The generator then applies model- and runtime-aware safety policy before any +server is started: + +1. For an MoE model, if the CPU communicator does not override the variable-size + `all_gatherv` and `reduce_scatterv` collectives used by MoE data parallelism, + all generated candidates use `DP=1`. Every supported TP value is retained, + so a 6-NUMA-node host tests `TP=4`, `TP=2`, and `TP=1` rather than only `TP=4`. +2. When MoE dimensions are known and expert parallelism is disabled, the fused + MoE path flattens TP, DP, and PCP. A candidate is valid only when + `moe_intermediate_size % (TP * DP * PCP) == 0`. +3. Failures not covered by a preflight rule remain isolated by + `--continue-on-error` when the installed CLI supports it, with bounded + `--resume` retries as the compatibility fallback. + +Skipped candidates and reasons are saved in +`sweep/parallel_layout_skips.json`. If model metadata cannot be loaded, the +generator keeps the generic layouts and relies on the third policy. + +By default, parallel-layout candidates change only TP and DP. The generated +benchmark config removes explicit `max-num-seqs` and +`max-num-batched-tokens`, so every layout uses vLLM-resolved scheduler +defaults. This prevents a scheduler value derived from the seed concurrency +from becoming part of the TP/DP selection. + +With `--tune-scheduler`, the workload-derived per-replica scheduler baseline is +restored for this stage. + +The stage writes: + +```text +sweep/parallel-layout-config.yml +sweep/parallel-layout-recommendation.json +``` + +The selected TP/DP layout becomes the fixed server layout for Stage 2. + +### Temporary TP/DP NUMA-binding workaround + +Xeon TP/DP and Tune All sweeps currently enable a temporary explicit CPU-binding +workaround by default: + +```text +--tp-dp-numa-bind-workaround +``` + +Hardware detection builds one `VLLM_CPU_OMP_THREADS_BIND` CPU list per effective +NUMA node. On x86 it selects one logical CPU per physical core, matching vLLM's +auto-binding SMT policy, and excludes one physical core from each NUMA node for +non-OMP work. The resulting value is written to `env.sh`, for example: + +```bash +export VLLM_CPU_OMP_THREADS_BIND='48-62|64-78|80-94|96-110' +``` + +The exact CPU IDs depend on the effective container/cgroup cpuset and NUMA +topology. Discontinuous CPU IDs are preserved as comma-separated ranges. + +`VLLM_CPU_NUM_OF_RESERVED_CPU` is not used by this workaround because vLLM only +applies its reserved-CPU logic in automatic binding mode. With an explicit +`VLLM_CPU_OMP_THREADS_BIND`, reserved cores must already be omitted from the +generated lists. + +After the vLLM CPU DP NUMA-binding issue is fixed, disable the workaround +without removing the implementation: + +```bash +--no-tp-dp-numa-bind-workaround +``` + +An explicit non-`auto` `VLLM_CPU_OMP_THREADS_BIND` supplied by the recipe is +preserved. + +### Stage 2: Tune Concurrency + +The concurrency stage keeps the selected TP/DP layout fixed while finding the +largest useful client concurrency. By default the generated benchmark config +removes explicit `max-num-seqs` and `max-num-batched-tokens`, so the seed +concurrency cannot become an artificial scheduler admission limit. + +With `--tune-scheduler`, explicit scheduler values are preserved as an opt-in +behavior. + +When TTFT and/or TPOT objectives are supplied, Recipes uses an **adaptive SLA +boundary search** instead of uniformly sampling the entire workload range. + +The supplied `--concurrency` is the seed. The controller grows upward until it +finds the first SLA failure (or shrinks downward when the seed fails), then +binary-searches the integer PASS/FAIL bracket. One vLLM server remains running +throughout this refinement, so midpoint probes do not reload the model. + +For example: + +```text +32 PASS, 64 FAIL + -> 48 + -> 40 or 56 + -> ... + -> adjacent PASS / FAIL boundary +``` + +A point is SLA-feasible only when all of the following hold: + +1. Every measured run completes without failed requests. +2. Median P99 TTFT is at or below the supplied TTFT objective. +3. Median P99 TPOT is at or below the supplied TPOT objective. +4. Duration-weighted combined request compliance is at least the configured + minimum (`0.99`, or 99%, by default). + +The combined-compliance check is intentionally stricter than looking at the +TTFT and TPOT P99 columns independently. `vllm bench serve --goodput` counts +requests that satisfy all supplied latency objectives, and the recommender +estimates the combined compliant-request fraction across repeated runs as: + +```text +sum(request_goodput * duration) / sum(completed_requests) +``` + +Therefore a candidate can have P99 TTFT below 3000 ms and P99 TPOT below +100 ms but still be SLA-ineligible if fewer than 99% of requests satisfy both +objectives together. For example, a reported combined compliance of `98.96%` +fails the default `99.00%` requirement even when both displayed P99 metrics +pass. The HTML report shows the combined compliance, required threshold, and +the specific SLA failure reason for each candidate. + +Adaptive concurrency uses a `1800` second server-readiness timeout by default, +which is intentionally longer than the generic benchmark helper default for +CPU model loading and initialization. `run_full_sweep.sh +--server-ready-timeout VALUE` also exports the selected timeout explicitly to +the generated concurrency runner, in addition to forwarding the CLI option. +This avoids falling back to a shorter adaptive-helper timeout when the +concurrency stage is invoked through the full staged workflow. + +The default growth factor is `2` and the default hard cap is `1000`. +Generated runners accept `--growth-factor VALUE` and +`--max-concurrency-cap VALUE`. + +Without TTFT/TPOT objectives there is no SLA boundary. In that mode the +generated runner keeps vLLM Workload Explorer: + +```text +vllm bench sweep serve_workload --workload-var max_concurrency +``` + +The stage writes: + +```text +sweep/concurrency-recommendation.json +``` + +`max_concurrency` is benchmark/deployment metadata. It is **not** written as a +`vllm serve` configuration parameter. + +### Stage 3: Tune Scheduler + +Before the scheduler sweep, `prepare_scheduler.py` recalculates the scheduler +baseline from: + +```text +selected TP/DP + + +selected max_concurrency + + +fixed input/output token shape +``` + +The scheduler stage keeps an explicit `max-num-seqs` at the per-replica +concurrency lower bound while tuning `max-num-batched-tokens`. It also compares +the explicit sequence limit with the vLLM default. + +The per-replica sequence baseline is recalculated as: + +```text +max-num-seqs = ceil(selected max_concurrency / selected data-parallel-size) +``` + +The directed scheduler sweep does not test `max-num-seqs` below this value; +doing so limits immediately active requests and can turn scheduler queueing into +severe TTFT degradation. It keeps the existing batch-budget curve around the +baseline and also measures three vLLM-default reference candidates: + +| Reference candidate | `max-num-seqs` | `max-num-batched-tokens` | +| --- | --- | --- | +| `vllm_default_max_num_seqs` | vLLM default | generated value | +| `vllm_default_max_num_batched_tokens` | generated value | vLLM default | +| `vllm_defaults` | vLLM default | vLLM default | + +The default references do not hard-code values such as 128 or 2048. The sweep +omits the selected scheduler argument and lets the installed vLLM resolve its +normal platform-, world-size-, model-, and usage-context-aware default. + +If a default-reference candidate wins, the corresponding scheduler key is +omitted from `recommended-config.yml`. + +## 4. Benchmark Size, Warmup, and Failure Handling + +Measured request count scales with concurrency using: + +```text +num_prompts = min(1000, max(100, concurrency * 10)) +``` + +This targets about ten concurrency turnovers when neither bound applies. The +100-prompt floor avoids very small samples for P99/compliance measurements, and +the 1000-prompt cap bounds sweep runtime. + +Before a measured parameter combination, generated scripts use one unmeasured +warmup containing one full concurrency window. Warmup output is retained as +`warmup.json` for auditing but is excluded from recommendation statistics. + +Generated scripts also handle sweep-run failures without discarding completed +work: + +- If the installed vLLM CLI supports `--continue-on-error`, the generated script + enables it. +- If the installed vLLM does not expose that option, the script omits it and + automatically retries an interrupted sweep with `--resume`. +- The default is two automatic resume retries. Override it with + `VLLM_RECIPE_SWEEP_RETRIES`. + +For example: + +```bash +export VLLM_RECIPE_SWEEP_RETRIES=3 +/output/sweep/run_full_sweep.sh +``` + +## 5. Targeted Tuning Workflows + +Use these modes when a full re-tune is unnecessary. + +### Tune TP/DP Only + +Use this when only the parallel layout needs to be measured. This mode stops +after selecting `tensor-parallel-size` and `data-parallel-size`: + +```bash +python3 /recipes/recipe_json_to_vllm_config.py \ + --model meta-llama/Llama-3.1-8B-Instruct \ + --hardware xeon6 \ + --detect-hardware \ + --input-tokens 128 \ + --output-tokens 128 \ + --concurrency 32 \ + --ttft-sla-ms 3000 \ + --tpot-sla-ms 100 \ + --config-out /output/config.yml \ + --env-out /output/env.sh \ + --generate-parallel-layout-sweep \ + --sweep-out-dir /output/sweep +``` + +Run: + +```bash +/output/sweep/run_parallel_layout_sweep.sh +/output/sweep/recommend_parallel_layout.py +``` + +The result is `/output/sweep/parallel-layout-config.yml`. Use Tune All when +concurrency and scheduler tuning should follow the selected layout. + +### Tune Concurrency Only + +Use this when TP/DP and scheduler configuration are already fixed and only +concurrent serving capacity needs to be measured: + +```bash +python3 /recipes/recipe_json_to_vllm_config.py \ + --model meta-llama/Llama-3.1-8B-Instruct \ + --hardware xeon6 \ + --detect-hardware \ + --input-tokens 128 \ + --output-tokens 128 \ + --concurrency 32 \ + --ttft-sla-ms 3000 \ + --tpot-sla-ms 100 \ + --config-out /output/config.yml \ + --env-out /output/env.sh \ + --generate-concurrency-sweep \ + --sweep-out-dir /output/sweep +``` + +Run: + +```bash +/output/sweep/run_concurrency_sweep.sh +/output/sweep/recommend_concurrency.py +``` + +### Tune Scheduler Only + +Use this when TP/DP and target concurrency are already known: + +```bash +python3 /recipes/recipe_json_to_vllm_config.py \ + --model meta-llama/Llama-3.1-8B-Instruct \ + --hardware xeon6 \ + --detect-hardware \ + --input-tokens 128 \ + --output-tokens 128 \ + --concurrency 32 \ + --ttft-sla-ms 3000 \ + --tpot-sla-ms 100 \ + --config-out /output/config.yml \ + --env-out /output/env.sh \ + --generate-scheduler-sweep \ + --sweep-out-dir /output/sweep +``` + +`--generate-sweep` remains a backward-compatible alias for +`--generate-scheduler-sweep`. + +Run: + +```bash +/output/sweep/run_sweep.sh +/output/sweep/recommend.py +``` + +## 6. Recommendation Policy + +With TTFT/TPOT objectives, benchmark commands use vLLM `--goodput`. + +For TP/DP and scheduler selection, the recommender: + +1. Excludes invalid configurations and failed benchmark measurements. +2. Calculates duration-weighted combined request compliance across successful + runs from request goodput and run duration. +3. Requires the median P99 TTFT and P99 TPOT objectives independently. +4. Also requires the combined request-compliance ratio to meet + `--minimum-compliance`, which defaults to `0.99`. +5. Establishes the highest mean output-token throughput among eligible + configurations, then applies the selection policy below. + +The P99 checks and combined-compliance check answer different questions. P99 +verifies the tail of each latency metric independently; combined compliance +verifies that the required fraction of requests satisfy all configured +objectives together. A candidate must pass both forms of validation to be +marked SLA eligible. + +For a scheduler-only comparison with one fixed TP/DP layout, candidates within +1% of the highest eligible output-token throughput are treated as practically +equivalent. Within that set, the recommender prefers the candidate with fewer +explicit scheduler overrides, followed by compliance, goodput, and throughput. +This avoids hard-coding a scheduler value for a difference that is likely within +normal run-to-run variation. Change or disable the equivalence range with: + +```bash +/output/sweep/recommend.py --throughput-equivalence-percent VALUE +``` + +Use `0` to restore exact highest-throughput selection. TP/DP comparisons still +use exact highest-throughput selection because changing a parallel layout is a +material deployment decision. + +For concurrency selection, the policy instead selects the **highest +SLA-feasible `max_concurrency`**, with throughput/compliance metrics used to +break ties. + +Change the compliance threshold with: + +```bash +/output/sweep/recommend.py --minimum-compliance VALUE +``` + +If no configuration is SLA-feasible, the recommendation JSON records a +best-effort candidate. A deployable final configuration is produced only when +the required recommendation stages succeed. + +## 7. Post-Benchmark Reporting + +After Tune All finishes, `run_full_sweep.sh` automatically runs `report.py` and +writes a self-contained summary: + +```text +/output/sweep/sweep-report.html +``` + +The HTML report includes the selected configuration, a stage summary, SLA +metrics, candidate tables, and automatic coverage observations. It uses only +Python's standard library. A report-generation error is printed as a warning +and does not change the success of the completed sweep. + +The TTFT and TPOT objectives supplied to `recipe_json_to_vllm_config.py` are +embedded in every generated recommender and in `report.py`. +`run_full_sweep.sh` also passes those values explicitly to every recommendation +stage and the report. Report generation stops with an error if a stale or +manually generated recommendation JSON contains different SLA values. + +Regenerate the report manually when recommendation files change: + +```bash +cd /output/sweep +./report.py +``` + +### Optional figures + +Every generated sweep package includes a standalone analysis helper. It is not +called by `run_full_sweep.sh` or any stage runner, so figure generation cannot +start, resume, or alter a sweep. + +After one or more benchmark stages finish, install the plotting dependencies: + +```bash +cd /output/sweep +python3 -m pip install -r requirements.txt +``` + +Preview the planned figures, then generate them from the completed +`summary.json` files: + +```bash +./visualize.py --dry-run +./visualize.py +``` + +Use `--stage parallel-layout`, `--stage concurrency-tuning`, or +`--stage runtime-tuning` to analyze selected stages. The helper writes PNG files +under each stage's `figures/` directory. See the generated `VISUALIZATION.md` +for the figure set and additional examples. + +## 8. Deploy the Tuned Configuration + +After Tune All completes: ```bash source /output/env.sh vllm serve --config /output/sweep/recommended-config.yml ``` + +## Select sweep stages + +Sweep implementation and reporting helpers live in `tools/recipes/sweep/`. +The converter remains `tools/recipes/recipe_json_to_vllm_config.py`. + +Use `--sweep-stage concurrency` to generate only a concurrency sweep against +the supplied recipe's fixed TP/DP layout, for example: + +```bash +python3 tools/recipes/recipe_json_to_vllm_config.py \ + --model meta-llama/Llama-3.1-8B-Instruct --hardware xeon6 \ + --input-tokens 128 --output-tokens 128 --concurrency 32 \ + --ttft-sla-ms 3000 --tpot-sla-ms 100 \ + --sweep-stage concurrency --sweep-out-dir sweep +./sweep/run_concurrency_sweep.sh +./sweep/recommend_concurrency.py +``` + +`--sweep-stage` accepts `parallel-layout`, `concurrency`, `scheduler`, or `all`. +The existing `--generate-*-sweep` flags remain supported; do not combine them +with `--sweep-stage`. `parallel-layout` and `all` require `--detect-hardware`. + +In a package generated with `--sweep-stage all` or `--generate-full-sweep`, +select the stages to execute: + +```bash +./sweep/run_full_sweep.sh --stages concurrency +./sweep/run_full_sweep.sh --stages concurrency,scheduler +./sweep/run_full_sweep.sh --stages scheduler --resume +``` + +The default is all three stages. Selected stages run in dependency order. +Skipping parallel-layout reuses `parallel-layout-config.yml` if present; +otherwise the runner copies the generated `initial-config.yml` as the fixed +layout. Scheduler-only execution requires `concurrency-recommendation.json` +from an earlier concurrency run. Remaining arguments go to the selected sweep +runners. Use separate output directories for unrelated experiments. diff --git a/tools/recipes/VISUALIZATION.md b/tools/recipes/VISUALIZATION.md new file mode 100644 index 0000000000..4c109102e7 --- /dev/null +++ b/tools/recipes/VISUALIZATION.md @@ -0,0 +1,73 @@ +# Post-Benchmark Reporting + +## HTML summary + +`run_full_sweep.sh` automatically generates a self-contained HTML report after +the final recommendation: + +```text +sweep-report.html +``` + +The report uses Python's standard library and does not require plotting +dependencies. It summarizes selected settings, stage results, SLA metrics, and +all measured candidates. Report-generation failures are warnings and do not +change sweep success. + +The generated `report.py` uses the same TTFT and TPOT objectives supplied when +the sweep package was created. It displays both objectives and validates them +against every available recommendation JSON before writing the report. + +Regenerate it manually or select a different output path: + +```bash +./report.py +./report.py --output results/my-sweep-report.html +``` + +## Optional figures + +Visualization is an independent analysis step. It reads completed sweep +results and does not start a server, run a benchmark, or change a tuning +recommendation. + +Install the plotting dependencies in the same environment as vLLM: + +```bash +python3 -m pip install -r requirements.txt +``` + +Preview the figures that would be generated: + +```bash +./visualize.py --dry-run +``` + +Generate figures for every completed stage: + +```bash +./visualize.py +``` + +Select one or more stages when needed: + +```bash +./visualize.py --stage concurrency-tuning +./visualize.py --stage parallel-layout --stage runtime-tuning +``` + +Figures are written beside the source results: + +```text +results/parallel-layout/figures/ +results/concurrency-tuning/figures/ +results/runtime-tuning/figures/ +``` + +The analysis produces output-throughput, P99 TTFT, and P99 TPOT figures. The +concurrency stage also produces a throughput/TTFT tradeoff figure and a goodput +figure when the benchmark results contain that metric. + +The native vLLM Pareto command is intentionally not used because its efficiency +axis is expressed as tokens per second per GPU, which is not an accurate label +for Xeon CPU layouts. diff --git a/tools/recipes/hardware_detection.py b/tools/recipes/hardware_detection.py index 2b90dffac4..f5b428dd02 100644 --- a/tools/recipes/hardware_detection.py +++ b/tools/recipes/hardware_detection.py @@ -4,7 +4,7 @@ """CPU hardware discovery for optional vLLM Recipes runtime tuning. Keep this module focused on facts exposed to the current process/container. -Tuning decisions belong in runtime_tuning.py. +Tuning decisions belong in sweep/runtime_tuning.py. """ from __future__ import annotations @@ -46,6 +46,89 @@ class HardwareInfo: return asdict(self) +def _compress_cpu_ids(cpu_ids: list[int] | tuple[int, ...]) -> str: + """Return a compact Linux CPU-list string while preserving CPU IDs.""" + values = sorted(set(cpu_ids)) + if not values: + return "" + + ranges: list[str] = [] + start = previous = values[0] + for cpu_id in values[1:]: + if cpu_id == previous + 1: + previous = cpu_id + continue + ranges.append(str(start) if start == previous else f"{start}-{previous}") + start = previous = cpu_id + ranges.append(str(start) if start == previous else f"{start}-{previous}") + return ",".join(ranges) + + +def build_numa_omp_threads_bind( + hardware: HardwareInfo, + *, + reserved_cores_per_numa: int = 1, +) -> str: + """Build explicit x86 OMP CPU lists for the effective NUMA nodes. + + Temporary TP/DP sweep workaround: + - select one logical CPU per physical core, matching vLLM x86 auto-binding; + - reserve cores by omitting them from each explicit NUMA CPU list; + - emit one pipe-separated list per effective NUMA node. + + Explicit VLLM_CPU_OMP_THREADS_BIND lists bypass vLLM's auto-binding reserve + logic, so the reserved cores must be excluded here. + """ + if reserved_cores_per_numa < 0: + raise ValueError("reserved_cores_per_numa must be at least 0") + + effective_nodes = {node.node_id for node in hardware.numa_nodes} + allowed_cpus = get_allowed_cpu_list() + + groups: list[str] = [] + for node in hardware.numa_nodes: + node_cpus = [ + cpu + for cpu in allowed_cpus + if cpu.numa_node == node.node_id and cpu.numa_node in effective_nodes + ] + + core_to_logical_ids: dict[int, list[int]] = {} + for cpu in node_cpus: + if cpu.physical_core < 0: + continue + core_to_logical_ids.setdefault(cpu.physical_core, []).append(cpu.id) + + if not core_to_logical_ids: + raise RuntimeError( + f"NUMA node {node.node_id} has no physical-core topology " + "available for explicit OMP binding." + ) + + # Match vLLM's x86 auto-binding selector: for SMT siblings on a physical + # core, use the highest logical CPU ID. + selected_cpu_ids = sorted( + max(logical_ids) for _, logical_ids in sorted(core_to_logical_ids.items()) + ) + + if len(selected_cpu_ids) <= reserved_cores_per_numa: + raise RuntimeError( + f"NUMA node {node.node_id} has only {len(selected_cpu_ids)} " + "usable physical cores, which is not enough after reserving " + f"{reserved_cores_per_numa} core(s)." + ) + + if reserved_cores_per_numa: + selected_cpu_ids = selected_cpu_ids[:-reserved_cores_per_numa] + + groups.append(_compress_cpu_ids(selected_cpu_ids)) + + if not groups: + raise RuntimeError("No effective NUMA CPU lists are available for OMP binding.") + + return "|".join(groups) + + def _read_socket_id(cpu_id: int) -> int | None: path = Path(f"/sys/devices/system/cpu/cpu{cpu_id}/topology/physical_package_id") try: diff --git a/tools/recipes/model_constraints.py b/tools/recipes/model_constraints.py new file mode 100644 index 0000000000..c25deb6f57 --- /dev/null +++ b/tools/recipes/model_constraints.py @@ -0,0 +1,111 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +"""Read model head constraints without loading model weights.""" + +from __future__ import annotations + +import json +from dataclasses import dataclass +from typing import Any + + +@dataclass(frozen=True) +class HeadConstraint: + name: str + attention_heads: int + kv_heads: int + + def supports(self, tp: int) -> bool: + if self.attention_heads % tp: + return False + if self.kv_heads >= tp: + return self.kv_heads % tp == 0 + return tp % self.kv_heads == 0 + + +def _positive_int(value: object, name: str) -> int: + if isinstance(value, bool) or not isinstance(value, int) or value < 1: + raise ValueError(f"Cannot determine a positive integer for {name}: {value!r}") + return value + + +def extract_head_constraints( + hf_config: Any, *, include_vision: bool = True +) -> tuple[HeadConstraint, ...]: + """Extract decoder, global-attention, hybrid and encoder head constraints.""" + constraints = [] + if getattr(hf_config, "model_type", None) == "whisper": + for component in ("encoder", "decoder"): + heads = _positive_int( + getattr(hf_config, f"{component}_attention_heads", None), component + ) + constraints.append(HeadConstraint(component, heads, heads)) + return tuple(constraints) + + text = hf_config.get_text_config() + heads = _positive_int(getattr(text, "num_attention_heads", None), "attention heads") + kv = getattr(text, "num_key_value_heads", None) + if kv is None: + kv = heads + kv = _positive_int(kv, "KV heads") + constraints.append(HeadConstraint("text", heads, kv)) + + global_kv = getattr(text, "num_global_key_value_heads", None) + if global_kv is not None: + constraints.append( + HeadConstraint( + "global attention", heads, _positive_int(global_kv, "global KV") + ) + ) + + # Hybrid layers have separate head counts; use conservative divisibility + # rather than assuming they support standard attention's KV replication. + for field in ("linear_num_key_heads", "linear_num_value_heads"): + count = getattr(text, field, None) + if count is not None: + count = _positive_int(count, field) + constraints.append(HeadConstraint(field, count, count)) + + vision = getattr(hf_config, "vision_config", None) + if include_vision and vision is not None: + vision_heads = getattr(vision, "num_attention_heads", None) + if vision_heads is None: + vision_heads = getattr(vision, "num_heads", None) + vision_heads = _positive_int(vision_heads, "vision heads") + constraints.append(HeadConstraint("vision", vision_heads, vision_heads)) + return tuple(constraints) + + +def load_head_constraints(config: dict[str, Any]) -> tuple[HeadConstraint, ...]: + """Load the effective serving configuration through vLLM's config loader.""" + from vllm.transformers_utils.config import get_config + + hf_overrides = config.get("hf-overrides") + if isinstance(hf_overrides, str): + hf_overrides = json.loads(hf_overrides) + if hf_overrides is not None and not isinstance(hf_overrides, dict): + raise ValueError("Head detection requires hf-overrides to be a JSON object") + hf_config = get_config( + config["model"], + trust_remote_code=config.get("trust-remote-code", False), + revision=config.get("revision"), + code_revision=config.get("code-revision"), + config_format=config.get("config-format", "auto"), + hf_overrides_kw=hf_overrides, + ) + return extract_head_constraints( + hf_config, include_vision=config.get("mm-encoder-tp-mode") != "data" + ) + + +def suggest_tensor_parallel_size( + candidate: int, constraints: tuple[HeadConstraint, ...] +) -> int: + """Reduce an automatic power-of-two candidate until all head checks pass.""" + _positive_int(candidate, "TP candidate") + if candidate & (candidate - 1) or not constraints: + raise ValueError("Expected a power-of-two TP candidate and model constraints") + while not all(constraint.supports(candidate) for constraint in constraints): + candidate //= 2 + return candidate diff --git a/tools/recipes/recipe_json_to_vllm_config.py b/tools/recipes/recipe_json_to_vllm_config.py index bd947b0745..c5089fbe9b 100644 --- a/tools/recipes/recipe_json_to_vllm_config.py +++ b/tools/recipes/recipe_json_to_vllm_config.py @@ -189,17 +189,67 @@ def parse_args() -> argparse.Namespace: type=float, help="Optional capacity target for future DP/capacity tuning.", ) + tuning.add_argument( + "--tune-scheduler", + action="store_true", + help=( + "Opt in to workload-derived max-num-seqs and " + "max-num-batched-tokens. By default scheduler parameters remain " + "at recipe/vLLM defaults." + ), + ) sweep = p.add_argument_group( "optional performance sweep", - ("Generate benchmark files after creating one initial runtime suggestion."), + ("Generate benchmark files alongside the directly deployable runtime config."), + ) + sweep.add_argument( + "--sweep-stage", + choices=("parallel-layout", "concurrency", "scheduler", "all"), + help="Generate only the selected stage, or all stages in dependency order.", ) sweep.add_argument( "--generate-sweep", action="store_true", + help="Backward-compatible alias for --generate-scheduler-sweep.", + ) + sweep.add_argument( + "--generate-scheduler-sweep", + action="store_true", + help=("Generate the max-num-seqs/max-num-batched-tokens scheduler sweep."), + ) + sweep.add_argument( + "--generate-parallel-layout-sweep", + action="store_true", help=( - "Generate an optional vllm bench sweep package around the single " - "initial runtime suggestion." + "Generate a standalone NUMA-aware TP/DP sweep. Requires --detect-hardware." + ), + ) + sweep.add_argument( + "--generate-concurrency-sweep", + action="store_true", + help=( + "Generate a max_concurrency workload sweep using " + "vllm bench sweep serve_workload." + ), + ) + sweep.add_argument( + "--generate-full-sweep", + action="store_true", + help=( + "Generate the end-to-end TP/DP -> max_concurrency -> scheduler " + "tuning pipeline. Requires --detect-hardware." + ), + ) + sweep.add_argument( + "--tp-dp-numa-bind-workaround", + action=argparse.BooleanOptionalAction, + default=True, + help=( + "Temporary Xeon TP/DP NUMA-binding workaround. For parallel-layout " + "and full sweeps, generate an explicit VLLM_CPU_OMP_THREADS_BIND " + "from detected NUMA topology. Enabled by default; disable with " + "--no-tp-dp-numa-bind-workaround after the vLLM DP binding fix." ), ) sweep.add_argument( @@ -766,10 +816,22 @@ def write_config( Path(path).write_text("\n".join(metadata) + "\n" + body, encoding="utf-8") -def write_env(path: str, source: str, recipe: dict[str, Any]) -> None: - env = recipe.get("env") or {} - if not isinstance(env, dict): - raise ValueError(f"Recipe `env` must be an object, got {type(env).__name__}") +def write_env( + path: str, + source: str, + recipe: dict[str, Any], + *, + env_overrides: dict[str, object] | None = None, +) -> None: + recipe_env = recipe.get("env") or {} + if not isinstance(recipe_env, dict): + raise ValueError( + f"Recipe `env` must be an object, got {type(recipe_env).__name__}" + ) + + env = dict(recipe_env) + overrides = env_overrides or {} + env.update(overrides) lines = [ "#!/usr/bin/env bash", @@ -778,6 +840,9 @@ def write_env(path: str, source: str, recipe: dict[str, Any]) -> None: "", ] + if overrides: + lines.append("# Temporary runtime overrides derived from detected hardware.") + if env: for key, value in env.items(): lines.append(f"export {key}={shlex.quote(str(value))}") @@ -793,6 +858,34 @@ def main() -> int: args = parse_args() try: + sweep_modes = { + "--generate-sweep": args.generate_sweep, + "--generate-scheduler-sweep": args.generate_scheduler_sweep, + "--generate-parallel-layout-sweep": args.generate_parallel_layout_sweep, + "--generate-concurrency-sweep": args.generate_concurrency_sweep, + "--generate-full-sweep": args.generate_full_sweep, + } + selected_sweep_modes = [ + name for name, enabled in sweep_modes.items() if enabled + ] + if len(selected_sweep_modes) > 1: + raise ValueError( + "Choose exactly one sweep-generation mode: " + + ", ".join(selected_sweep_modes) + ) + if args.sweep_stage is not None: + if selected_sweep_modes: + raise ValueError( + "Do not combine --sweep-stage with --generate-*-sweep options." + ) + stage_flags = { + "parallel-layout": "generate_parallel_layout_sweep", + "concurrency": "generate_concurrency_sweep", + "scheduler": "generate_scheduler_sweep", + "all": "generate_full_sweep", + } + setattr(args, stage_flags[args.sweep_stage], True) + selected_sweep_modes = ["--sweep-stage=" + args.sweep_stage] source = args.source if source is None: source = discover_recipe_source( @@ -813,7 +906,8 @@ def main() -> int: tuning_requested = ( args.detect_hardware - or args.generate_sweep + or args.tune_scheduler + or bool(selected_sweep_modes) or any( value is not None for value in ( @@ -828,12 +922,15 @@ def main() -> int: ) tuning = None + sweep_config = None workload = None sweep_writer = None + hardware = None + env_overrides: dict[str, object] = {} if tuning_requested: # Keep plain Recipes conversion lightweight. vLLM-specific modules # are imported only for optional runtime tuning or sweep generation. - from runtime_tuning import ( + from sweep.runtime_tuning import ( WorkloadHints, finetune_runtime_config, get_runtime_tuning_policies, @@ -848,24 +945,60 @@ def main() -> int: target_qps=args.target_qps, ) - if args.generate_sweep: - from sweep_generation import ( - validate_sweep_workload, - write_sweep_files, - ) + if selected_sweep_modes: + from sweep.sweep_generation import validate_sweep_workload validate_sweep_workload(workload) + + if args.generate_sweep or args.generate_scheduler_sweep: + from sweep.sweep_generation import write_sweep_files + sweep_writer = write_sweep_files recipe_hardware = recipe.get("hardware") - policies = get_runtime_tuning_policies(recipe_hardware) + scheduler_tuning_requested = bool( + args.tune_scheduler + or args.generate_sweep + or args.generate_scheduler_sweep + ) + policies = get_runtime_tuning_policies( + recipe_hardware, + tune_scheduler=scheduler_tuning_requested, + ) - hardware = None if args.detect_hardware: - from hardware_detection import detect_hardware + from hardware_detection import ( + build_numa_omp_threads_bind, + detect_hardware, + ) hardware = detect_hardware() + if ( + args.tp_dp_numa_bind_workaround + and str(recipe_hardware).lower() == "xeon6" + and ( + args.generate_parallel_layout_sweep or args.generate_full_sweep + ) + ): + recipe_env = recipe.get("env") or {} + current_binding = ( + recipe_env.get("VLLM_CPU_OMP_THREADS_BIND") + if isinstance(recipe_env, dict) + else None + ) + + # Preserve an explicit recipe-provided manual binding. + # Replace an unset or "auto" value with the temporary + # topology-derived binding. + if current_binding in (None, "auto"): + env_overrides["VLLM_CPU_OMP_THREADS_BIND"] = ( + build_numa_omp_threads_bind( + hardware, + reserved_cores_per_numa=1, + ) + ) + tuning = finetune_runtime_config( config, hardware=hardware, @@ -873,20 +1006,74 @@ def main() -> int: policies=policies, ) config.update(tuning.overrides) + if args.tune_scheduler: + config.update(tuning.sweep_overrides) + sweep_config = dict(config) + sweep_config.update(tuning.sweep_overrides) write_config(args.config_out, source, recipe, config) - write_env(args.env_out, source, recipe) + write_env( + args.env_out, + source, + recipe, + env_overrides=env_overrides, + ) sweep_files: list[Path] = [] - if args.generate_sweep: + if args.generate_full_sweep: + if not args.detect_hardware or hardware is None: + raise ValueError("--generate-full-sweep requires --detect-hardware.") assert workload is not None - assert sweep_writer is not None - sweep_files = sweep_writer( + from sweep.sweep_generation import write_full_sweep_files + + sweep_files = write_full_sweep_files( args.sweep_out_dir, config_path=args.config_out, env_path=args.env_out, config=config, workload=workload, + numa_node_count=hardware.numa_node_count, + tune_scheduler=args.tune_scheduler, + ) + elif args.generate_parallel_layout_sweep: + if not args.detect_hardware or hardware is None: + raise ValueError( + "--generate-parallel-layout-sweep requires --detect-hardware." + ) + assert workload is not None + from sweep.sweep_generation import write_parallel_layout_sweep_files + + sweep_files = write_parallel_layout_sweep_files( + args.sweep_out_dir, + config_path=args.config_out, + env_path=args.env_out, + config=config, + workload=workload, + numa_node_count=hardware.numa_node_count, + tune_scheduler=args.tune_scheduler, + ) + elif args.generate_concurrency_sweep: + assert workload is not None + from sweep.sweep_generation import write_concurrency_sweep_files + + sweep_files = write_concurrency_sweep_files( + args.sweep_out_dir, + config_path=args.config_out, + env_path=args.env_out, + config=config, + workload=workload, + tune_scheduler=args.tune_scheduler, + ) + elif args.generate_sweep or args.generate_scheduler_sweep: + assert workload is not None + assert sweep_writer is not None + assert sweep_config is not None + sweep_files = sweep_writer( + args.sweep_out_dir, + config_path=args.config_out, + env_path=args.env_out, + config=sweep_config, + workload=workload, ) if tuning is not None: @@ -894,6 +1081,13 @@ def main() -> int: print("Initial runtime suggestion:") for key, value in tuning.overrides.items(): print(f" {key}: {value}") + if tuning.sweep_overrides: + if args.tune_scheduler: + print("Workload-derived scheduler tuning:") + elif selected_sweep_modes: + print("Explicit scheduler sweep seed:") + for key, value in tuning.sweep_overrides.items(): + print(f" {key}: {value}") for note in tuning.notes: print(f" tuning: {note}") except Exception as exc: @@ -910,15 +1104,31 @@ def main() -> int: print(f" vllm serve --config {shlex.quote(args.config_out)}") if sweep_files: sweep_dir = Path(args.sweep_out_dir) + run_parallel = sweep_dir / "run_parallel_layout_sweep.sh" + recommend_parallel = sweep_dir / "recommend_parallel_layout.py" + run_concurrency = sweep_dir / "run_concurrency_sweep.sh" + recommend_concurrency = sweep_dir / "recommend_concurrency.py" run_sweep = sweep_dir / "run_sweep.sh" recommend = sweep_dir / "recommend.py" + run_full = sweep_dir / "run_full_sweep.sh" print() print("Optional performance sweep:") - print(f" {shlex.quote(str(run_sweep))} --dry-run") - print(f" {shlex.quote(str(run_sweep))}") - print() - print("After the sweep:") - print(f" {shlex.quote(str(recommend))}") + if args.generate_full_sweep: + print(f" {shlex.quote(str(run_full))}") + elif args.generate_concurrency_sweep: + print(f" {shlex.quote(str(run_concurrency))} --dry-run") + print(f" {shlex.quote(str(run_concurrency))}") + print(f" {shlex.quote(str(recommend_concurrency))}") + elif args.generate_parallel_layout_sweep: + print(f" {shlex.quote(str(run_parallel))} --dry-run") + print(f" {shlex.quote(str(run_parallel))}") + print(f" {shlex.quote(str(recommend_parallel))}") + else: + print(f" {shlex.quote(str(run_sweep))} --dry-run") + print(f" {shlex.quote(str(run_sweep))}") + print() + print("After the sweep:") + print(f" {shlex.quote(str(recommend))}") return 0 diff --git a/tools/recipes/requirements.txt b/tools/recipes/requirements.txt new file mode 100644 index 0000000000..36a96c66b4 --- /dev/null +++ b/tools/recipes/requirements.txt @@ -0,0 +1,4 @@ +# Dependencies used by `vllm bench sweep plot`. +matplotlib +pandas +seaborn diff --git a/tools/recipes/sweep/__init__.py b/tools/recipes/sweep/__init__.py new file mode 100644 index 0000000000..c6a8d5b25b --- /dev/null +++ b/tools/recipes/sweep/__init__.py @@ -0,0 +1,4 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +"""Runtime tuning and benchmark sweep generation helpers.""" diff --git a/tools/recipes/sweep/adaptive_concurrency.py b/tools/recipes/sweep/adaptive_concurrency.py new file mode 100755 index 0000000000..a0fdf36a24 --- /dev/null +++ b/tools/recipes/sweep/adaptive_concurrency.py @@ -0,0 +1,261 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +"""Find the maximum SLA-feasible serving concurrency adaptively.""" + +from __future__ import annotations + +import argparse +import shlex +from pathlib import Path +from statistics import median +from typing import Any + +from vllm.benchmarks.sweep.param_sweep import ParameterSweepItem +from vllm.benchmarks.sweep.serve import run_server +from vllm.benchmarks.sweep.serve_workload import run_comb_workload + +DEFAULT_MINIMUM_COMPLIANCE = 0.99 +DEFAULT_MAX_CONCURRENCY_CAP = 1000 +DEFAULT_GROWTH_FACTOR = 2.0 + + +def _number(value: object) -> float | None: + if isinstance(value, bool) or not isinstance(value, (int, float)): + return None + return float(value) + + +def _num_prompts(concurrency: int) -> int: + return min(1000, max(100, concurrency * 10)) + + +def _sla_eligible( + rows: list[dict[str, Any]], + *, + expected_runs: int, + ttft_sla_ms: float | None, + tpot_sla_ms: float | None, + minimum_compliance: float, +) -> bool: + if len(rows) < expected_runs: + return False + if any(int(_number(row.get("failed")) or 0) for row in rows): + return False + + completed = sum(int(_number(row.get("completed")) or 0) for row in rows) + if completed <= 0: + return False + + compliant = sum( + (_number(row.get("request_goodput")) or 0.0) + * (_number(row.get("duration")) or 0.0) + for row in rows + ) + if compliant / completed < minimum_compliance: + return False + + if ttft_sla_ms is not None: + values = [ + value + for row in rows + if (value := _number(row.get("p99_ttft_ms"))) is not None + ] + if not values or median(values) > ttft_sla_ms: + return False + + if tpot_sla_ms is not None: + values = [ + value + for row in rows + if (value := _number(row.get("p99_tpot_ms"))) is not None + ] + if not values or median(values) > tpot_sla_ms: + return False + + return True + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser( + description=( + "Find the highest SLA-feasible max_concurrency using exponential " + "bracketing followed by integer binary search." + ) + ) + parser.add_argument("--serve-cmd", required=True) + parser.add_argument("--bench-cmd", required=True) + parser.add_argument("--results-dir", required=True) + parser.add_argument("--seed-concurrency", type=int, required=True) + parser.add_argument("--input-tokens", type=int, required=True) + parser.add_argument("--output-tokens", type=int, required=True) + parser.add_argument("--ttft-sla-ms", type=float) + parser.add_argument("--tpot-sla-ms", type=float) + parser.add_argument( + "--minimum-compliance", + type=float, + default=DEFAULT_MINIMUM_COMPLIANCE, + ) + parser.add_argument( + "--max-concurrency-cap", + type=int, + default=DEFAULT_MAX_CONCURRENCY_CAP, + ) + parser.add_argument( + "--growth-factor", + type=float, + default=DEFAULT_GROWTH_FACTOR, + ) + parser.add_argument("--num-runs", type=int, default=3) + parser.add_argument("--server-ready-timeout", type=int, default=1800) + parser.add_argument("--show-stdout", action="store_true") + parser.add_argument("--continue-on-error", action="store_true") + parser.add_argument("--resume", action="store_true") + parser.add_argument("--dry-run", action="store_true") + return parser.parse_args() + + +def main() -> int: + args = parse_args() + + if args.seed_concurrency <= 0: + raise ValueError("--seed-concurrency must be positive.") + if not 0.0 < args.minimum_compliance <= 1.0: + raise ValueError("--minimum-compliance must be greater than 0 and at most 1.") + if args.growth_factor <= 1.0: + raise ValueError("--growth-factor must be greater than 1.") + if args.max_concurrency_cap <= 0: + raise ValueError("--max-concurrency-cap must be positive.") + if args.num_runs <= 0: + raise ValueError("--num-runs must be positive.") + if args.ttft_sla_ms is None and args.tpot_sla_ms is None: + raise ValueError("Adaptive search requires TTFT and/or TPOT SLA.") + + seed = args.seed_concurrency + cap = max(seed, args.max_concurrency_cap) + results_dir = Path(args.results_dir) + results_dir.mkdir(parents=True, exist_ok=True) + + if args.dry_run: + first_upper = min( + cap, + max(seed + 1, int(round(seed * args.growth_factor))), + ) + print("Adaptive concurrency search plan:") + print(f" seed: {seed}") + print(f" first upper probe: {first_upper}") + print(f" hard cap: {cap}") + print(" strategy: bracket SLA boundary, then integer binary search") + return 0 + + serve_cmd = shlex.split(args.serve_cmd) + bench_cmd = shlex.split(args.bench_cmd) + serve_comb = ParameterSweepItem() + + def run_point(server: object, concurrency: int) -> bool: + bench_comb = ParameterSweepItem( + { + "_benchmark_name": "user_workload", + "random_input_len": args.input_tokens, + "random_output_len": args.output_tokens, + "num_prompts": _num_prompts(concurrency), + } + ) + print(f"[ADAPTIVE CONCURRENCY] testing max_concurrency={concurrency}") + rows = ( + run_comb_workload( + server, + bench_cmd, + serve_comb=serve_comb, + bench_comb=bench_comb, + link_vars=[], + experiment_dir=results_dir, + num_runs=args.num_runs, + dry_run=False, + warmup_num_prompts=min(concurrency, 1000), + continue_on_error=args.continue_on_error, + workload_var="max_concurrency", + workload_value=concurrency, + ) + or [] + ) + + eligible = _sla_eligible( + rows, + expected_runs=args.num_runs, + ttft_sla_ms=args.ttft_sla_ms, + tpot_sla_ms=args.tpot_sla_ms, + minimum_compliance=args.minimum_compliance, + ) + print( + "[ADAPTIVE CONCURRENCY] " + f"max_concurrency={concurrency}: {'PASS' if eligible else 'FAIL'}" + ) + return eligible + + with run_server( + serve_cmd, + [], + show_stdout=args.show_stdout, + serve_overrides=serve_comb, + dry_run=False, + server_ready_timeout=args.server_ready_timeout, + ) as server: + seed_passes = run_point(server, seed) + + if seed_passes: + low = seed + high: int | None = None + while low < cap: + probe = min( + cap, + max(low + 1, int(round(low * args.growth_factor))), + ) + if run_point(server, probe): + low = probe + if low == cap: + print( + "[ADAPTIVE CONCURRENCY] reached cap without an SLA failure" + ) + return 0 + else: + high = probe + break + if high is None: + return 0 + else: + high = seed + low: int | None = None + probe = seed + while probe > 1: + probe = max(1, probe // 2) + if run_point(server, probe): + low = probe + break + high = probe + if low is None: + print( + "[ADAPTIVE CONCURRENCY] no SLA-feasible concurrency found, " + "including max_concurrency=1" + ) + return 0 + + assert low is not None + assert high is not None + print(f"[ADAPTIVE CONCURRENCY] refining bracket PASS={low}, FAIL={high}") + + while high - low > 1: + midpoint = (low + high) // 2 + if run_point(server, midpoint): + low = midpoint + else: + high = midpoint + + print(f"[ADAPTIVE CONCURRENCY] boundary resolved: PASS={low}, FAIL={high}") + + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tools/recipes/sweep/concurrency_recommendation.py b/tools/recipes/sweep/concurrency_recommendation.py new file mode 100644 index 0000000000..908f8ce1da --- /dev/null +++ b/tools/recipes/sweep/concurrency_recommendation.py @@ -0,0 +1,330 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +"""Select one max-concurrency recommendation from a workload sweep.""" + +from __future__ import annotations + +import argparse +import json +from collections import defaultdict +from pathlib import Path +from statistics import mean, median +from typing import Any + +DEFAULT_TTFT_SLA_MS: float | None = None +DEFAULT_TPOT_SLA_MS: float | None = None +DEFAULT_SEED_CONCURRENCY: int | None = None +DEFAULT_MINIMUM_COMPLIANCE: float = 0.99 +DEFAULT_RESULTS_DIR = "results/concurrency-tuning" +DEFAULT_OUTPUT_JSON = "concurrency-recommendation.json" + + +def _number(value: object) -> float | None: + if isinstance(value, bool) or not isinstance(value, (int, float)): + return None + return float(value) + + +def _mean(rows: list[dict[str, Any]], key: str) -> float | None: + values = [_number(row.get(key)) for row in rows] + numbers = [value for value in values if value is not None] + return mean(numbers) if numbers else None + + +def _percentile_summary( + rows: list[dict[str, Any]], key: str +) -> tuple[float | None, float | None, float | None]: + values = [_number(row.get(key)) for row in rows] + numbers = [value for value in values if value is not None] + if not numbers: + return None, None, None + return mean(numbers), median(numbers), max(numbers) + + +def _load_runs(results_dir: Path) -> list[dict[str, Any]]: + rows: list[dict[str, Any]] = [] + for path in sorted(results_dir.rglob("summary.json")): + data = json.loads(path.read_text(encoding="utf-8")) + if not isinstance(data, list): + continue + for row in data: + if isinstance(row, dict): + copied = dict(row) + copied["_summary_path"] = str(path) + rows.append(copied) + return rows + + +def _aggregate_candidates( + rows: list[dict[str, Any]], + *, + use_goodput: bool, + ttft_sla_ms: float | None, + tpot_sla_ms: float | None, + minimum_compliance: float, +) -> list[dict[str, Any]]: + grouped: dict[int, list[dict[str, Any]]] = defaultdict(list) + for row in rows: + concurrency = row.get("max_concurrency") + if ( + isinstance(concurrency, bool) + or not isinstance(concurrency, int) + or concurrency <= 0 + ): + continue + grouped[concurrency].append(row) + + candidates: list[dict[str, Any]] = [] + for concurrency, runs in sorted(grouped.items()): + failed_requests = sum(int(_number(run.get("failed")) or 0) for run in runs) + output_throughput = _mean(runs, "output_throughput") + request_throughput = _mean(runs, "request_throughput") + request_goodput = _mean(runs, "request_goodput") + mean_p99_ttft_ms, median_p99_ttft_ms, worst_p99_ttft_ms = _percentile_summary( + runs, "p99_ttft_ms" + ) + mean_p99_tpot_ms, median_p99_tpot_ms, worst_p99_tpot_ms = _percentile_summary( + runs, "p99_tpot_ms" + ) + + completed_requests = sum( + int(_number(run.get("completed")) or 0) for run in runs + ) + estimated_compliant_requests = sum( + (_number(run.get("request_goodput")) or 0.0) + * (_number(run.get("duration")) or 0.0) + for run in runs + ) + combined_compliance_ratio = ( + estimated_compliant_requests / completed_requests + if completed_requests > 0 + else None + ) + + valid = failed_requests == 0 and output_throughput is not None + reason = None + if failed_requests: + valid = False + reason = f"{failed_requests} failed request(s) across successful runs" + elif use_goodput and request_goodput is None: + valid = False + reason = "request_goodput is missing; rerun with SLO goodput enabled" + + p99_sla_eligible = valid + if ttft_sla_ms is not None: + p99_sla_eligible = bool( + p99_sla_eligible + and median_p99_ttft_ms is not None + and median_p99_ttft_ms <= ttft_sla_ms + ) + if tpot_sla_ms is not None: + p99_sla_eligible = bool( + p99_sla_eligible + and median_p99_tpot_ms is not None + and median_p99_tpot_ms <= tpot_sla_ms + ) + compliance_eligible = bool( + valid + and combined_compliance_ratio is not None + and combined_compliance_ratio >= minimum_compliance + ) + + candidates.append( + { + "max_concurrency": concurrency, + "run_count": len(runs), + "failed_requests": failed_requests, + "mean_request_goodput": request_goodput, + "mean_request_throughput": request_throughput, + "mean_output_throughput": output_throughput, + "mean_p99_ttft_ms": mean_p99_ttft_ms, + "median_p99_ttft_ms": median_p99_ttft_ms, + "worst_p99_ttft_ms": worst_p99_ttft_ms, + "mean_p99_tpot_ms": mean_p99_tpot_ms, + "median_p99_tpot_ms": median_p99_tpot_ms, + "worst_p99_tpot_ms": worst_p99_tpot_ms, + "estimated_compliant_requests": estimated_compliant_requests, + "completed_requests": completed_requests, + "combined_compliance_ratio": combined_compliance_ratio, + "combined_compliance_percent": ( + combined_compliance_ratio * 100 + if combined_compliance_ratio is not None + else None + ), + "benchmark_valid": valid, + "p99_sla_eligible": p99_sla_eligible, + "compliance_eligible": compliance_eligible, + "sla_eligible": p99_sla_eligible and compliance_eligible, + "valid": valid, + "invalid_reason": reason, + "summary_files": sorted({str(run["_summary_path"]) for run in runs}), + } + ) + + return candidates + + +def _select_candidate( + candidates: list[dict[str, Any]], *, use_goodput: bool +) -> tuple[dict[str, Any] | None, dict[str, Any], str]: + valid = [candidate for candidate in candidates if candidate["valid"]] + if not valid: + raise ValueError( + "No valid workload-sweep point was found. Inspect failed runs and metrics." + ) + + if use_goodput: + objective = "highest_sla_feasible_max_concurrency" + eligible = [candidate for candidate in valid if candidate["sla_eligible"]] + winner = ( + max( + eligible, + key=lambda candidate: ( + candidate["max_concurrency"], + candidate["mean_output_throughput"], + candidate["combined_compliance_ratio"] or 0.0, + candidate["mean_request_goodput"] or 0.0, + ), + ) + if eligible + else None + ) + best_effort = max( + valid, + key=lambda candidate: ( + candidate["mean_request_goodput"] or 0.0, + candidate["combined_compliance_ratio"] or 0.0, + candidate["mean_output_throughput"], + candidate["max_concurrency"], + ), + ) + else: + objective = "highest_mean_output_throughput" + winner = max( + valid, + key=lambda candidate: ( + candidate["mean_output_throughput"], + candidate["max_concurrency"], + ), + ) + best_effort = winner + + return winner, best_effort, objective + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser( + description="Select max_concurrency from vLLM Workload Explorer results." + ) + parser.add_argument("--results-dir", default=DEFAULT_RESULTS_DIR) + parser.add_argument("--ttft-sla-ms", type=float, default=DEFAULT_TTFT_SLA_MS) + parser.add_argument("--tpot-sla-ms", type=float, default=DEFAULT_TPOT_SLA_MS) + parser.add_argument( + "--seed-concurrency", type=int, default=DEFAULT_SEED_CONCURRENCY + ) + parser.add_argument( + "--minimum-compliance", + type=float, + default=DEFAULT_MINIMUM_COMPLIANCE, + ) + parser.add_argument("--output-json", default=DEFAULT_OUTPUT_JSON) + return parser.parse_args() + + +def main() -> int: + args = parse_args() + if not 0.0 < args.minimum_compliance <= 1.0: + raise ValueError("--minimum-compliance must be greater than 0 and at most 1.") + + script_dir = Path(__file__).resolve().parent + results_dir = Path(args.results_dir) + if not results_dir.is_absolute(): + results_dir = script_dir / results_dir + output_json = Path(args.output_json) + if not output_json.is_absolute(): + output_json = script_dir / output_json + + if not results_dir.exists(): + raise ValueError(f"Workload-sweep results were not found at {results_dir}.") + + rows = _load_runs(results_dir) + if not rows: + raise ValueError(f"No summary.json results found under {results_dir}.") + + use_goodput = args.ttft_sla_ms is not None or args.tpot_sla_ms is not None + candidates = _aggregate_candidates( + rows, + use_goodput=use_goodput, + ttft_sla_ms=args.ttft_sla_ms, + tpot_sla_ms=args.tpot_sla_ms, + minimum_compliance=args.minimum_compliance, + ) + winner, best_effort, objective = _select_candidate( + candidates, use_goodput=use_goodput + ) + + recommendation = { + "status": ( + "sla_feasible" if winner is not None else "no_sla_feasible_concurrency" + ), + "selection_objective": objective, + "seed_concurrency": args.seed_concurrency, + "minimum_compliance_ratio": args.minimum_compliance, + "slo": { + "ttft_ms": args.ttft_sla_ms, + "tpot_ms": args.tpot_sla_ms, + }, + "recommended": ( + {"max_concurrency": winner["max_concurrency"]} + if winner is not None + else None + ), + "measured": ( + { + key: winner[key] + for key in ( + "run_count", + "mean_request_throughput", + "mean_request_goodput", + "mean_output_throughput", + "combined_compliance_ratio", + "combined_compliance_percent", + "mean_p99_ttft_ms", + "median_p99_ttft_ms", + "worst_p99_ttft_ms", + "mean_p99_tpot_ms", + "median_p99_tpot_ms", + "worst_p99_tpot_ms", + ) + } + if winner is not None + else None + ), + "best_effort": { + "max_concurrency": best_effort["max_concurrency"], + "mean_request_goodput": best_effort["mean_request_goodput"], + "mean_output_throughput": best_effort["mean_output_throughput"], + "combined_compliance_ratio": best_effort["combined_compliance_ratio"], + "p99_sla_eligible": best_effort["p99_sla_eligible"], + }, + "candidates": candidates, + } + output_json.write_text( + json.dumps(recommendation, indent=2) + "\n", encoding="utf-8" + ) + + if winner is None: + print("No SLA-feasible max_concurrency was found.") + print(f"Wrote diagnostic evidence to {output_json}") + return 2 + + print(f"Recommended max_concurrency: {winner['max_concurrency']}") + print(f"Selection objective: {objective}") + print(f"Wrote {output_json}") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tools/recipes/runtime_tuning.py b/tools/recipes/sweep/runtime_tuning.py similarity index 69% rename from tools/recipes/runtime_tuning.py rename to tools/recipes/sweep/runtime_tuning.py index e251de89d7..f23e6f76ad 100644 --- a/tools/recipes/runtime_tuning.py +++ b/tools/recipes/sweep/runtime_tuning.py @@ -19,6 +19,7 @@ from math import ceil from typing import Any from hardware_detection import HardwareInfo +from model_constraints import load_head_constraints, suggest_tensor_parallel_size from vllm.config.scheduler import SchedulerConfig @@ -46,7 +47,10 @@ class WorkloadHints: @dataclass class TuningResult: + """Runtime overrides and benchmark-only seeds produced by tuning policies.""" + overrides: dict[str, Any] = field(default_factory=dict) + sweep_overrides: dict[str, Any] = field(default_factory=dict) notes: list[str] = field(default_factory=list) @@ -66,13 +70,24 @@ def _record_override( result.notes.append(f"{key}={value!r}: {reason}") +def _record_sweep_override( + result: TuningResult, + key: str, + value: Any, + reason: str, +) -> None: + """Record an explicit benchmark seed without changing deployment config.""" + result.sweep_overrides[key] = value + result.notes.append(f"sweep {key}={value!r}: {reason}") + + def _get_active_sequence_count( config: dict[str, Any], workload: WorkloadHints, ) -> int: - """Return the sequence count used to size one scheduler iteration.""" + """Return the per-replica sequence count for one scheduler iteration.""" if workload.concurrency is not None: - return workload.concurrency + return ceil(workload.concurrency / _get_data_parallel_size(config)) configured = config.get("max-num-seqs") if ( @@ -85,9 +100,22 @@ def _get_active_sequence_count( return SchedulerConfig.DEFAULT_MAX_NUM_SEQS +def _get_data_parallel_size(config: dict[str, Any]) -> int: + """Return a valid configured DP size, falling back to one replica.""" + configured = config.get("data-parallel-size", 1) + if ( + isinstance(configured, int) + and not isinstance(configured, bool) + and configured > 0 + ): + return configured + return 1 + + def _estimate_prefills_per_step( workload: WorkloadHints, active_sequences: int, + data_parallel_size: int, ) -> float: """Estimate steady-state prompt arrivals per decode scheduler step.""" # Reserve enough prefill budget for at least one representative prompt. @@ -104,9 +132,10 @@ def _estimate_prefills_per_step( # With target QPS + TPOT, approximate how many prompts arrive during one # decode scheduler-step interval. if workload.target_qps is not None and workload.tpot_sla_ms is not None: + per_replica_qps = workload.target_qps / data_parallel_size prefills_per_step = max( prefills_per_step, - workload.target_qps * workload.tpot_sla_ms / 1000.0, + per_replica_qps * workload.tpot_sla_ms / 1000.0, ) # A step cannot replace more requests than are active. @@ -119,19 +148,50 @@ def _resolve_tensor_parallel_size( workload: WorkloadHints, result: TuningResult, ) -> None: - del config, workload + del workload if hardware is None or hardware.numa_node_count < 1: return + if any( + config.get(key, 1) != 1 + for key in ("data-parallel-size", "pipeline-parallel-size") + ): + result.notes.append( + "tensor-parallel-size: kept recipe value; automatic TP selection " + "currently requires DP=1 and PP=1" + ) + return + # Draft policy: use the largest power-of-two TP that does not exceed the # effective NUMA-node count. This deliberately avoids hard-coding unusual # non-power-of-two topologies into the converter. candidate = 1 << (hardware.numa_node_count.bit_length() - 1) + try: + constraints = load_head_constraints(config) + except Exception as exc: + result.notes.append( + "tensor-parallel-size: kept recipe value (vLLM default TP=1 if absent); " + f"model head detection failed: {exc}" + ) + return + + for constraint in constraints: + result.notes.append( + f"detected {constraint.name}: " + f"attention_heads={constraint.attention_heads}, " + f"kv_heads={constraint.kv_heads}" + ) + selected = suggest_tensor_parallel_size(candidate, constraints) + if selected != candidate: + result.notes.append( + f"tensor-parallel-size: reduced NUMA-derived TP={candidate} to " + f"TP={selected} for model head partitioning/replication" + ) _record_override( result, "tensor-parallel-size", - candidate, - "derived from the effective NUMA-node count", + selected, + "derived from effective NUMA nodes and detected model head constraints", ) @@ -181,15 +241,22 @@ def _resolve_max_num_seqs( workload: WorkloadHints, result: TuningResult, ) -> None: - del config, hardware + del hardware if workload.concurrency is None: return - _record_override( + data_parallel_size = _get_data_parallel_size(config) + per_replica_concurrency = ceil(workload.concurrency / data_parallel_size) + + _record_sweep_override( result, "max-num-seqs", - workload.concurrency, - "derived from the optional target concurrency", + per_replica_concurrency, + ( + "benchmark seed derived from the optional global target concurrency " + f"and data_parallel_size={data_parallel_size}; config.yml keeps the " + "recipe/vLLM scheduler default" + ), ) @@ -203,12 +270,17 @@ def _resolve_max_num_batched_tokens( if workload.input_tokens is None: return + data_parallel_size = _get_data_parallel_size(config) active_sequences = _get_active_sequence_count(config, workload) # Decode requests consume roughly one token per active sequence in one # scheduler iteration. Prefills share the remaining scheduler token budget. decode_budget = active_sequences - prefills_per_step = _estimate_prefills_per_step(workload, active_sequences) + prefills_per_step = _estimate_prefills_per_step( + workload, + active_sequences, + data_parallel_size, + ) prefill_budget = ceil(workload.input_tokens * prefills_per_step) # Use vLLM's scheduler defaults/constraints as floors rather than local @@ -231,16 +303,18 @@ def _resolve_max_num_batched_tokens( ): candidate = max(candidate, max_model_len) - _record_override( + _record_sweep_override( result, "max-num-batched-tokens", candidate, ( - "workload-derived scheduler budget " - f"(active_sequences={active_sequences}, " + "workload-derived benchmark seed " + f"(per_replica_active_sequences={active_sequences}, " + f"data_parallel_size={data_parallel_size}, " f"prefills_per_step={prefills_per_step:.2f}, " f"decode_budget={decode_budget}, " - f"prefill_budget={prefill_budget})" + f"prefill_budget={prefill_budget}); config.yml keeps the " + "recipe/vLLM scheduler default" ), ) @@ -256,17 +330,20 @@ def _resolve_data_parallel_size( return result.notes.append( - "data-parallel-size: target QPS supplied; kept recipe value because " - "per-replica capacity is not known yet" + "data-parallel-size: automatic DP tuning is disabled; kept recipe value " + "because CPU DP auto-binding and per-replica capacity need validation" ) DEFAULT_POLICIES: tuple[Policy, ...] = ( _resolve_tensor_parallel_size, _resolve_gpu_memory_utilization, + _resolve_data_parallel_size, +) + +SCHEDULER_TUNING_POLICIES: tuple[Policy, ...] = ( _resolve_max_num_seqs, _resolve_max_num_batched_tokens, - _resolve_data_parallel_size, ) # Select tuning behavior from the hardware requested by the Recipes rendering, @@ -280,6 +357,8 @@ HARDWARE_TUNING_POLICIES: dict[str, tuple[Policy, ...]] = { def get_runtime_tuning_policies( recipe_hardware: object, + *, + tune_scheduler: bool = False, ) -> tuple[Policy, ...]: if not isinstance(recipe_hardware, str) or not recipe_hardware: raise ValueError( @@ -294,6 +373,8 @@ def get_runtime_tuning_policies( "Runtime tuning is not supported for recipe hardware " f"{recipe_hardware!r}. Currently supported: {supported}." ) + if tune_scheduler: + policies = (*policies, *SCHEDULER_TUNING_POLICIES) return policies @@ -304,7 +385,7 @@ def finetune_runtime_config( workload: WorkloadHints | None = None, policies: tuple[Policy, ...] = DEFAULT_POLICIES, ) -> TuningResult: - """Return overrides; never mutate the recipe-derived config in place.""" + """Return deployment overrides and sweep seeds without mutating config.""" hints = workload or WorkloadHints() hints.validate() diff --git a/tools/recipes/sweep/scheduler_sweep_prepare.py b/tools/recipes/sweep/scheduler_sweep_prepare.py new file mode 100644 index 0000000000..59709a0fc5 --- /dev/null +++ b/tools/recipes/sweep/scheduler_sweep_prepare.py @@ -0,0 +1,225 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +"""Prepare scheduler sweep inputs after TP/DP and concurrency selection.""" + +from __future__ import annotations + +import argparse +import json +import math +from pathlib import Path +from typing import Any + +try: + import yaml +except ImportError as exc: + raise SystemExit("PyYAML is required. Install it with: pip install pyyaml") from exc + +PROMPTS_PER_CONCURRENCY = 10 +MIN_NUM_PROMPTS = 100 +MAX_NUM_PROMPTS = 1000 + + +def _positive_int(config: dict[str, Any], key: str) -> int: + value = config.get(key) + if not isinstance(value, int) or isinstance(value, bool) or value <= 0: + raise ValueError(f"Expected positive integer {key!r}, got {value!r}.") + return value + + +def _num_prompts_for_concurrency(concurrency: int) -> int: + return min( + MAX_NUM_PROMPTS, + max(MIN_NUM_PROMPTS, concurrency * PROMPTS_PER_CONCURRENCY), + ) + + +def _strict_lower_power_of_two(value: int) -> int: + if value <= 1: + return 1 + return 1 << ((value - 1).bit_length() - 1) + + +def _strict_upper_power_of_two(value: int) -> int: + return 1 << value.bit_length() + + +def _scheduler_baseline( + config: dict[str, Any], + *, + concurrency: int, + input_tokens: int, + output_tokens: int, + tpot_sla_ms: float | None, + target_qps: float | None, +) -> tuple[int, int]: + data_parallel_size = int(config.get("data-parallel-size", 1)) + if data_parallel_size <= 0: + raise ValueError("data-parallel-size must be positive.") + + per_replica_concurrency = math.ceil(concurrency / data_parallel_size) + prefills_per_step = max(1.0, per_replica_concurrency / max(output_tokens, 1)) + if target_qps is not None and tpot_sla_ms is not None: + per_replica_qps = target_qps / data_parallel_size + prefills_per_step = max( + prefills_per_step, + per_replica_qps * tpot_sla_ms / 1000.0, + ) + prefills_per_step = min(float(per_replica_concurrency), prefills_per_step) + + batch = max( + 2048, + per_replica_concurrency, + per_replica_concurrency + math.ceil(input_tokens * prefills_per_step), + ) + if config.get("enable-chunked-prefill") is False: + max_model_len = config.get("max-model-len") + if isinstance(max_model_len, int) and not isinstance(max_model_len, bool): + batch = max(batch, max_model_len) + + return per_replica_concurrency, batch + + +def _build_serve_params(config: dict[str, Any]) -> list[dict[str, Any]]: + initial_seqs = _positive_int(config, "max-num-seqs") + initial_batch = _positive_int(config, "max-num-batched-tokens") + + minimum_batch = initial_seqs + if config.get("enable-chunked-prefill") is False: + max_model_len = config.get("max-model-len") + if ( + isinstance(max_model_len, int) + and not isinstance(max_model_len, bool) + and max_model_len > 0 + ): + minimum_batch = max(minimum_batch, max_model_len) + + lower_batch = max(minimum_batch, _strict_lower_power_of_two(initial_batch)) + smaller_batch = max(minimum_batch, _strict_lower_power_of_two(lower_batch)) + higher_batch = max(minimum_batch, _strict_upper_power_of_two(initial_batch)) + + candidates: list[dict[str, Any]] = [] + seen: set[tuple[int | None, int | None]] = set() + + def add( + name: str, + max_num_seqs: int | None, + max_num_batched_tokens: int | None, + ) -> None: + if max_num_seqs is not None and max_num_batched_tokens is not None: + max_num_batched_tokens = max(max_num_batched_tokens, max_num_seqs) + signature = (max_num_seqs, max_num_batched_tokens) + if signature in seen: + return + seen.add(signature) + + candidate: dict[str, Any] = {"_benchmark_name": name} + if max_num_seqs is not None: + candidate["max_num_seqs"] = max_num_seqs + if max_num_batched_tokens is not None: + candidate["max_num_batched_tokens"] = max_num_batched_tokens + candidates.append(candidate) + + add("initial", initial_seqs, initial_batch) + add("smaller_batch_budget", initial_seqs, smaller_batch) + add("lower_batch_budget", initial_seqs, lower_batch) + add("higher_batch_budget", initial_seqs, higher_batch) + add("vllm_default_max_num_seqs", None, initial_batch) + add("vllm_default_max_num_batched_tokens", initial_seqs, None) + add("vllm_defaults", None, None) + return candidates + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser() + parser.add_argument("--config", required=True) + parser.add_argument("--concurrency-recommendation", required=True) + parser.add_argument("--input-tokens", type=int, required=True) + parser.add_argument("--output-tokens", type=int, required=True) + parser.add_argument("--tpot-sla-ms", type=float) + parser.add_argument("--target-qps", type=float) + parser.add_argument("--output-config", required=True) + parser.add_argument("--serve-params", required=True) + parser.add_argument("--bench-params", required=True) + return parser.parse_args() + + +def main() -> int: + args = parse_args() + config_path = Path(args.config) + recommendation_path = Path(args.concurrency_recommendation) + + config = yaml.safe_load(config_path.read_text(encoding="utf-8")) + if not isinstance(config, dict): + raise ValueError(f"{config_path} does not contain a YAML object.") + + recommendation = json.loads(recommendation_path.read_text(encoding="utf-8")) + selected = recommendation.get("recommended") + if not isinstance(selected, dict): + raise ValueError( + f"{recommendation_path} has no SLA-feasible concurrency recommendation." + ) + concurrency = selected.get("max_concurrency") + if ( + isinstance(concurrency, bool) + or not isinstance(concurrency, int) + or concurrency <= 0 + ): + raise ValueError("Recommended max_concurrency is missing or invalid.") + + seqs, batch = _scheduler_baseline( + config, + concurrency=concurrency, + input_tokens=args.input_tokens, + output_tokens=args.output_tokens, + tpot_sla_ms=args.tpot_sla_ms, + target_qps=args.target_qps, + ) + + scheduler_seed = dict(config) + scheduler_seed["max-num-seqs"] = seqs + scheduler_seed["max-num-batched-tokens"] = batch + + scheduler_config = dict(config) + scheduler_config.pop("max-num-seqs", None) + scheduler_config.pop("max-num-batched-tokens", None) + Path(args.output_config).write_text( + yaml.safe_dump( + scheduler_config, + sort_keys=False, + default_flow_style=False, + allow_unicode=True, + ), + encoding="utf-8", + ) + + Path(args.serve_params).write_text( + json.dumps(_build_serve_params(scheduler_seed), indent=2) + "\n", + encoding="utf-8", + ) + Path(args.bench_params).write_text( + json.dumps( + [ + { + "random_input_len": args.input_tokens, + "random_output_len": args.output_tokens, + "max_concurrency": concurrency, + "num_prompts": _num_prompts_for_concurrency(concurrency), + } + ], + indent=2, + ) + + "\n", + encoding="utf-8", + ) + + print(f"Prepared scheduler sweep for max_concurrency={concurrency}") + print(f"Scheduler seed max-num-seqs={seqs}") + print(f"Scheduler seed max-num-batched-tokens={batch}") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tools/recipes/sweep/sweep_generation.py b/tools/recipes/sweep/sweep_generation.py new file mode 100644 index 0000000000..cd3a131f89 --- /dev/null +++ b/tools/recipes/sweep/sweep_generation.py @@ -0,0 +1,1531 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +"""Generate optional vLLM benchmark sweep files from one initial suggestion.""" + +from __future__ import annotations + +import json +import math +import os +import shlex +import sys +from dataclasses import dataclass +from pathlib import Path +from typing import Any + +import yaml + +from .runtime_tuning import WorkloadHints + +SUPPORTED_TENSOR_PARALLEL_SIZES = frozenset({1, 2, 4, 8}) + +MOE_EXPERT_COUNT_KEYS = ( + "num_experts", + "num_local_experts", + "num_sparse_experts", + "num_total_experts", + "n_experts", + "n_routed_experts", + "num_routed_experts", +) +MOE_INTERMEDIATE_SIZE_KEYS = ( + "moe_intermediate_size", + "expert_intermediate_size", + "moe_ffn_hidden_size", +) + +PROMPTS_PER_CONCURRENCY = 10 +MIN_NUM_PROMPTS = 100 +MAX_NUM_PROMPTS = 1000 + + +@dataclass(frozen=True) +class ModelParallelMetadata: + """Model dimensions needed to reject invalid MoE layouts before startup.""" + + is_moe: bool | None + num_experts: int | None = None + moe_intermediate_size: int | None = None + + +UNKNOWN_MODEL_PARALLEL_METADATA = ModelParallelMetadata(is_moe=None) + + +def _num_prompts_for_concurrency(concurrency: int) -> int: + return min( + MAX_NUM_PROMPTS, + max(MIN_NUM_PROMPTS, concurrency * PROMPTS_PER_CONCURRENCY), + ) + + +def _positive_int(config: dict[str, Any], key: str) -> int: + value = config.get(key) + if not isinstance(value, int) or isinstance(value, bool) or value <= 0: + raise ValueError(f"Sweep generation requires a positive initial {key!r} value.") + return value + + +def _strict_lower_power_of_two(value: int) -> int: + if value <= 1: + return 1 + return 1 << ((value - 1).bit_length() - 1) + + +def _strict_upper_power_of_two(value: int) -> int: + return 1 << value.bit_length() + + +def validate_sweep_workload(workload: WorkloadHints) -> None: + required = { + "--input-tokens": workload.input_tokens, + "--output-tokens": workload.output_tokens, + "--concurrency": workload.concurrency, + } + missing = [name for name, value in required.items() if value is None] + if missing: + raise ValueError("--generate-sweep requires " + ", ".join(missing) + ".") + + +def build_serve_params(config: dict[str, Any]) -> list[dict[str, Any]]: + """Build a bounded batch-budget sweep at the sequence-count lower bound. + + Explicit candidates keep the initial per-replica sequence count. Three + reference points compare those tuned values with vLLM-resolved defaults. + """ + initial_seqs = _positive_int(config, "max-num-seqs") + initial_batch = _positive_int(config, "max-num-batched-tokens") + + minimum_batch = initial_seqs + if config.get("enable-chunked-prefill") is False: + max_model_len = config.get("max-model-len") + if ( + isinstance(max_model_len, int) + and not isinstance(max_model_len, bool) + and max_model_len > 0 + ): + minimum_batch = max(minimum_batch, max_model_len) + + lower_batch = max( + minimum_batch, + _strict_lower_power_of_two(initial_batch), + ) + smaller_batch = max( + minimum_batch, + _strict_lower_power_of_two(lower_batch), + ) + higher_batch = max( + minimum_batch, + _strict_upper_power_of_two(initial_batch), + ) + + candidates: list[dict[str, Any]] = [] + seen: set[tuple[int | None, int | None]] = set() + + def add( + name: str, + max_num_seqs: int | None, + max_num_batched_tokens: int | None, + ) -> None: + if max_num_seqs is not None and max_num_batched_tokens is not None: + max_num_batched_tokens = max(max_num_batched_tokens, max_num_seqs) + signature = (max_num_seqs, max_num_batched_tokens) + if signature in seen: + return + seen.add(signature) + + candidate: dict[str, Any] = {"_benchmark_name": name} + if max_num_seqs is not None: + candidate["max_num_seqs"] = max_num_seqs + if max_num_batched_tokens is not None: + candidate["max_num_batched_tokens"] = max_num_batched_tokens + candidates.append(candidate) + + # Keep the exact initial suggestion as the measured baseline. Additional + # values exist only in the optional sweep package. + add("initial", initial_seqs, initial_batch) + add("smaller_batch_budget", initial_seqs, smaller_batch) + add("lower_batch_budget", initial_seqs, lower_batch) + add("higher_batch_budget", initial_seqs, higher_batch) + # Default-reference candidates intentionally omit one or both scheduler + # keys. The sweep server starts from sweep_config.yml, where both keys are + # removed, so omission lets vLLM resolve its normal runtime default. + add("vllm_default_max_num_seqs", None, initial_batch) + add("vllm_default_max_num_batched_tokens", initial_seqs, None) + add("vllm_defaults", None, None) + + return candidates + + +def _scheduler_baseline_for_dp( + config: dict[str, Any], workload: WorkloadHints, data_parallel_size: int +) -> tuple[int, int]: + """Return per-replica scheduler settings for a DP layout.""" + assert workload.concurrency is not None + assert workload.input_tokens is not None + per_replica_concurrency = math.ceil(workload.concurrency / data_parallel_size) + output_tokens = workload.output_tokens or 1 + prefills_per_step = max(1.0, per_replica_concurrency / output_tokens) + if workload.target_qps is not None and workload.tpot_sla_ms is not None: + per_replica_qps = workload.target_qps / data_parallel_size + prefills_per_step = max( + prefills_per_step, + per_replica_qps * workload.tpot_sla_ms / 1000.0, + ) + prefills_per_step = min(float(per_replica_concurrency), prefills_per_step) + batch = max( + 2048, + per_replica_concurrency, + per_replica_concurrency + math.ceil(workload.input_tokens * prefills_per_step), + ) + if config.get("enable-chunked-prefill") is False: + max_model_len = config.get("max-model-len") + if isinstance(max_model_len, int) and not isinstance(max_model_len, bool): + batch = max(batch, max_model_len) + return per_replica_concurrency, batch + + +def _model_config_value(view: Any, key: str) -> Any: + if isinstance(view, dict): + return view.get(key) + return getattr(view, key, None) + + +def _config_views(hf_config: Any) -> list[Any]: + """Return common nested model config views without duplicates.""" + views = [hf_config] + for view in views: + for key in ( + "text_config", + "language_config", + "llm_config", + "ffn_config", + "moe_config", + ): + value = _model_config_value(view, key) + if value is not None and all(value is not item for item in views): + views.append(value) + return views + + +def _first_positive_model_int(views: list[Any], keys: tuple[str, ...]) -> int | None: + for view in views: + for key in keys: + value = _model_config_value(view, key) + if isinstance(value, int) and not isinstance(value, bool) and value > 0: + return value + return None + + +def _config_identifies_moe(views: list[Any]) -> bool: + for view in views: + model_type = _model_config_value(view, "model_type") + if isinstance(model_type, str) and "moe" in model_type.lower(): + return True + architectures = _model_config_value(view, "architectures") + if isinstance(architectures, list) and any( + isinstance(name, str) and "moe" in name.lower() for name in architectures + ): + return True + return False + + +def model_parallel_metadata_from_config(hf_config: Any) -> ModelParallelMetadata: + """Extract MoE markers from a loaded Transformers-compatible config.""" + views = _config_views(hf_config) + num_experts = _first_positive_model_int(views, MOE_EXPERT_COUNT_KEYS) + moe_intermediate_size = _first_positive_model_int(views, MOE_INTERMEDIATE_SIZE_KEYS) + is_moe = ( + num_experts is not None + or moe_intermediate_size is not None + or _config_identifies_moe(views) + ) + if is_moe and moe_intermediate_size is None: + # Some MoE configs use the generic name for each expert's FFN width. + moe_intermediate_size = _first_positive_model_int(views, ("intermediate_size",)) + return ModelParallelMetadata( + is_moe=is_moe, + num_experts=num_experts, + moe_intermediate_size=moe_intermediate_size, + ) + + +def inspect_model_parallel_metadata(config: dict[str, Any]) -> ModelParallelMetadata: + """Load the model config and extract portable MoE topology markers.""" + model = config.get("model") + if not isinstance(model, str) or not model: + raise ValueError("Parallel-layout sweep requires a model in config.yml.") + + from vllm.transformers_utils.config import get_config + + hf_config = get_config( + model, + trust_remote_code=config.get("trust-remote-code") is True, + revision=config.get("revision"), + code_revision=config.get("code-revision"), + config_format=config.get("config-format", "auto"), + ) + return model_parallel_metadata_from_config(hf_config) + + +def cpu_supports_moe_dp_collectives() -> bool: + """Whether CPU overrides the variable-size collectives used by MoE DP.""" + try: + from vllm.distributed.device_communicators.base_device_communicator import ( + DeviceCommunicatorBase, + ) + from vllm.distributed.device_communicators.cpu_communicator import ( + CpuCommunicator, + ) + except Exception: + return False + + return all( + getattr(CpuCommunicator, name) is not getattr(DeviceCommunicatorBase, name) + for name in ("all_gatherv", "reduce_scatterv") + ) + + +def _parallel_layout_name( + tensor_parallel_size: int, data_parallel_size: int, numa_node_count: int +) -> str: + name = f"tp{tensor_parallel_size}_dp{data_parallel_size}" + used_nodes = tensor_parallel_size * data_parallel_size + if used_nodes != numa_node_count: + name += f"_numa{used_nodes}of{numa_node_count}" + return name + + +def _default_parallel_layout_sizes(numa_node_count: int) -> list[tuple[int, int]]: + max_supported_tp = max( + size for size in SUPPORTED_TENSOR_PARALLEL_SIZES if size <= numa_node_count + ) + layouts = [] + for tensor_parallel_size in range(numa_node_count, 0, -1): + if tensor_parallel_size not in SUPPORTED_TENSOR_PARALLEL_SIZES: + continue + uses_all_numa_nodes = numa_node_count % tensor_parallel_size == 0 + if not uses_all_numa_nodes and tensor_parallel_size != max_supported_tp: + continue + layouts.append( + (tensor_parallel_size, max(1, numa_node_count // tensor_parallel_size)) + ) + return layouts + + +def build_parallel_layout_plan( + config: dict[str, Any], + workload: WorkloadHints, + numa_node_count: int, + *, + model_metadata: ModelParallelMetadata = UNKNOWN_MODEL_PARALLEL_METADATA, + cpu_moe_dp_supported: bool = True, + tune_scheduler: bool = False, +) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]: + """Build candidates and diagnostics after applying parallel safety policy.""" + validate_sweep_workload(workload) + if numa_node_count <= 1: + raise ValueError("Parallel-layout sweep requires at least two NUMA nodes.") + + layouts = _default_parallel_layout_sizes(numa_node_count) + skipped: list[dict[str, Any]] = [] + if model_metadata.is_moe and not cpu_moe_dp_supported: + for tensor_parallel_size, data_parallel_size in layouts: + if data_parallel_size <= 1: + continue + skipped.append( + { + "_benchmark_name": _parallel_layout_name( + tensor_parallel_size, data_parallel_size, numa_node_count + ), + "policy": "cpu_moe_dp_collectives", + "tensor_parallel_size": tensor_parallel_size, + "data_parallel_size": data_parallel_size, + "reason": ( + "CPU communicator does not implement the variable-size " + "collectives required by MoE data parallelism" + ), + } + ) + # Keep TP coverage instead of collapsing the sweep to whichever DP=1 + # layout happened to exist in the full-NUMA candidate set. + layouts = [ + (tensor_parallel_size, 1) + for tensor_parallel_size in sorted( + SUPPORTED_TENSOR_PARALLEL_SIZES, reverse=True + ) + if tensor_parallel_size <= numa_node_count + ] + + pcp_size = config.get("prefill-context-parallel-size", 1) + if not isinstance(pcp_size, int) or isinstance(pcp_size, bool) or pcp_size <= 0: + raise ValueError("prefill-context-parallel-size must be a positive integer.") + uses_expert_parallel = config.get("enable-expert-parallel") is True + + candidates = [] + for tensor_parallel_size, data_parallel_size in layouts: + name = _parallel_layout_name( + tensor_parallel_size, data_parallel_size, numa_node_count + ) + effective_parallel_size = tensor_parallel_size * data_parallel_size * pcp_size + moe_width = model_metadata.moe_intermediate_size + if ( + model_metadata.is_moe + and not uses_expert_parallel + and moe_width is not None + and moe_width % effective_parallel_size != 0 + ): + skipped.append( + { + "_benchmark_name": name, + "policy": "moe_effective_parallel_size", + "tensor_parallel_size": tensor_parallel_size, + "data_parallel_size": data_parallel_size, + "prefill_context_parallel_size": pcp_size, + "effective_parallel_size": effective_parallel_size, + "moe_intermediate_size": moe_width, + "reason": ( + "moe_intermediate_size is not divisible by the effective " + "TP x DP x PCP size while expert parallelism is disabled" + ), + } + ) + continue + + candidate = { + "_benchmark_name": name, + "tensor_parallel_size": tensor_parallel_size, + "data_parallel_size": data_parallel_size, + } + if tune_scheduler: + max_num_seqs, max_num_batched_tokens = _scheduler_baseline_for_dp( + config, workload, data_parallel_size + ) + candidate["max_num_seqs"] = max_num_seqs + candidate["max_num_batched_tokens"] = max_num_batched_tokens + candidates.append(candidate) + + if not candidates: + raise ValueError( + "No valid parallel-layout candidates remain after applying model and " + "runtime capability checks. See parallel_layout_skips.json." + ) + return candidates, skipped + + +def build_parallel_layout_params( + config: dict[str, Any], + workload: WorkloadHints, + numa_node_count: int, + *, + model_metadata: ModelParallelMetadata = UNKNOWN_MODEL_PARALLEL_METADATA, + cpu_moe_dp_supported: bool = True, + tune_scheduler: bool = False, +) -> list[dict[str, Any]]: + """Build full-NUMA layouts plus the largest supported TP layout.""" + candidates, _ = build_parallel_layout_plan( + config, + workload, + numa_node_count, + model_metadata=model_metadata, + cpu_moe_dp_supported=cpu_moe_dp_supported, + tune_scheduler=tune_scheduler, + ) + return candidates + + +def build_bench_params(workload: WorkloadHints) -> list[dict[str, Any]]: + validate_sweep_workload(workload) + assert workload.input_tokens is not None + assert workload.output_tokens is not None + assert workload.concurrency is not None + + return [ + { + "random_input_len": workload.input_tokens, + "random_output_len": workload.output_tokens, + "max_concurrency": workload.concurrency, + "num_prompts": _num_prompts_for_concurrency(workload.concurrency), + } + ] + + +def _benchmark_models(config: dict[str, Any]) -> tuple[str, str]: + model = config.get("model") + if not isinstance(model, str) or not model: + raise ValueError("Sweep generation requires a model in config.yml.") + + served_model_name = config.get("served-model-name") + if isinstance(served_model_name, str) and served_model_name: + return served_model_name, model + if ( + isinstance(served_model_name, list) + and served_model_name + and isinstance(served_model_name[0], str) + ): + return served_model_name[0], model + + return model, model + + +def _relative_to(directory: Path, target: str) -> str: + return os.path.relpath(Path(target).resolve(), directory.resolve()) + + +def _write_json(path: Path, value: object) -> None: + path.write_text( + json.dumps(value, indent=2) + "\n", + encoding="utf-8", + ) + + +def _write_sweep_config(path: Path, config: dict[str, Any]) -> None: + sweep_config = dict(config) + sweep_config.pop("max-num-seqs", None) + sweep_config.pop("max-num-batched-tokens", None) + + body = yaml.safe_dump( + sweep_config, + sort_keys=False, + default_flow_style=False, + allow_unicode=True, + ) + header = ( + "# Generated for runtime sweep only.\n" + "# Scheduler values are supplied per candidate; omitted values use " + "vLLM defaults.\n" + ) + path.write_text(header + body, encoding="utf-8") + + +def _write_run_script( + path: Path, + *, + config_rel: str, + env_rel: str, + request_model: str, + tokenizer: str, + workload: WorkloadHints, + serve_params_name: str = "serve_params.json", + experiment_name: str = "runtime-tuning", + prepare_config_rel: str | None = None, +) -> None: + bench_parts = [ + "vllm bench serve", + "--backend vllm", + f"--model {shlex.quote(request_model)}", + f"--tokenizer {shlex.quote(tokenizer)}", + "--dataset-name random", + "--request-rate inf", + "--ignore-eos", + "--metric-percentiles 99", + ] + + goodput_pairs: list[str] = [] + if workload.ttft_sla_ms is not None: + goodput_pairs.append(f"ttft:{workload.ttft_sla_ms:g}") + if workload.tpot_sla_ms is not None: + goodput_pairs.append(f"tpot:{workload.tpot_sla_ms:g}") + if goodput_pairs: + bench_parts.append( + "--goodput " + " ".join(shlex.quote(pair) for pair in goodput_pairs) + ) + + bench_cmd = " ".join(bench_parts) + prepare_config = "" + if prepare_config_rel is not None: + prepare_config = f""" +SOURCE_CONFIG_PATH="${{SCRIPT_DIR}}/{prepare_config_rel}" +python3 - "${{SOURCE_CONFIG_PATH}}" "${{CONFIG_PATH}}" <<'PY' +import sys +from pathlib import Path + +import yaml + +source = Path(sys.argv[1]) +target = Path(sys.argv[2]) +config = yaml.safe_load(source.read_text(encoding="utf-8")) +if not isinstance(config, dict): + raise SystemExit(str(source) + " does not contain a YAML configuration object.") +config.pop("max-num-seqs", None) +config.pop("max-num-batched-tokens", None) +target.write_text( + yaml.safe_dump( + config, + sort_keys=False, + default_flow_style=False, + allow_unicode=True, + ), + encoding="utf-8", +) +PY + +""" + + script = f"""#!/usr/bin/env bash +set -euo pipefail + +SCRIPT_DIR="$(cd -- "$(dirname -- "${{BASH_SOURCE[0]}}")" && pwd)" +CONFIG_PATH="${{SCRIPT_DIR}}/{config_rel}" +ENV_PATH="${{SCRIPT_DIR}}/{env_rel}" + +source "${{ENV_PATH}}" + +# Released/nightly vLLM images keep recipe-relative assets such as +# examples/*.jinja under /vllm-workspace. Launch sweep server subprocesses +# from there when available so relative recipe assets resolve like they do +# in serve_with_recipe.sh. +if [[ -d /vllm-workspace/examples ]]; then + cd /vllm-workspace +fi + +# recipe tools may be newer than the installed vLLM package. Only pass the +# resilience flag when this vLLM CLI actually supports it. +CONTINUE_ON_ERROR_ARG="" +if vllm bench sweep serve --help 2>&1 | grep -q -- "--continue-on-error"; then + CONTINUE_ON_ERROR_ARG="--continue-on-error" +fi + +# On older vLLM versions, a failed benchmark aborts the sweep process. Retry the +# interrupted sweep with --resume so one transient run failure does not discard +# completed work or force a manual restart. +SWEEP_RETRY_COUNT="${{VLLM_RECIPE_SWEEP_RETRY_COUNT:-0}}" +MAX_SWEEP_RETRIES="${{VLLM_RECIPE_SWEEP_RETRIES:-2}}" + +{prepare_config}vllm bench sweep serve \ + --serve-cmd "vllm serve --config '${{CONFIG_PATH}}'" \ + --bench-cmd "{bench_cmd}" \ + --serve-params "${{SCRIPT_DIR}}/{serve_params_name}" \ + --bench-params "${{SCRIPT_DIR}}/bench_params.json" \ + --output-dir "${{SCRIPT_DIR}}/results" \ + --experiment-name {experiment_name} \ + --warmup-num-prompts {workload.concurrency} \ + ${{CONTINUE_ON_ERROR_ARG}} \ + "$@" || {{ + if [[ "${{SWEEP_RETRY_COUNT}}" -ge "${{MAX_SWEEP_RETRIES}}" ]]; then + echo "Sweep failed after ${{MAX_SWEEP_RETRIES}} automatic retries." >&2 + exit 1 + fi + + NEXT_RETRY=$((SWEEP_RETRY_COUNT + 1)) + RETRY_ARGS=() + for arg in "$@"; do + [[ "$arg" == "--resume" ]] && continue + RETRY_ARGS+=("$arg") + done + + EXPERIMENT_DIR="${{SCRIPT_DIR}}/results/{experiment_name}" + if [[ -d "${{EXPERIMENT_DIR}}" ]]; then + echo "Sweep command failed;" \ + "retry ${{NEXT_RETRY}}/${{MAX_SWEEP_RETRIES}} with --resume" + RETRY_ARGS=(--resume "${{RETRY_ARGS[@]}}") + else + echo "Sweep command failed before resumable state was created;" \ + "retry ${{NEXT_RETRY}}/${{MAX_SWEEP_RETRIES}} from the start" + fi + + VLLM_RECIPE_SWEEP_RETRY_COUNT="${{NEXT_RETRY}}" exec "$0" "${{RETRY_ARGS[@]}}" + }} +""" + path.write_text(script, encoding="utf-8") + path.chmod(path.stat().st_mode | 0o111) + + +def _write_recommend_script( + path: Path, + *, + config_rel: str, + env_rel: str, + workload: WorkloadHints, + results_dir: str = "results/runtime-tuning", + output_config: str = "recommended-config.yml", + output_json: str = "recommendation.json", +) -> None: + template_path = Path(__file__).with_name("sweep_recommendation.py") + source = template_path.read_text(encoding="utf-8") + + replacements = { + "DEFAULT_CONFIG_PATH: str | None = None": ( + f"DEFAULT_CONFIG_PATH: str | None = {config_rel!r}" + ), + "DEFAULT_ENV_PATH: str | None = None": ( + f"DEFAULT_ENV_PATH: str | None = {env_rel!r}" + ), + "DEFAULT_TTFT_SLA_MS: float | None = None": ( + f"DEFAULT_TTFT_SLA_MS: float | None = {workload.ttft_sla_ms!r}" + ), + "DEFAULT_TPOT_SLA_MS: float | None = None": ( + f"DEFAULT_TPOT_SLA_MS: float | None = {workload.tpot_sla_ms!r}" + ), + 'DEFAULT_RESULTS_DIR = "results/runtime-tuning"': ( + f"DEFAULT_RESULTS_DIR = {results_dir!r}" + ), + 'DEFAULT_OUTPUT_CONFIG = "recommended-config.yml"': ( + f"DEFAULT_OUTPUT_CONFIG = {output_config!r}" + ), + 'DEFAULT_OUTPUT_JSON = "recommendation.json"': ( + f"DEFAULT_OUTPUT_JSON = {output_json!r}" + ), + } + for marker, replacement in replacements.items(): + if marker not in source: + raise ValueError(f"Recommender template marker not found: {marker}") + source = source.replace(marker, replacement, 1) + + path.write_text(source, encoding="utf-8") + path.chmod(path.stat().st_mode | 0o111) + + +def _write_guide(path: Path, workload: WorkloadHints) -> None: + sla_lines: list[str] = [] + if workload.ttft_sla_ms is not None: + sla_lines.append(f"- TTFT objective: `{workload.ttft_sla_ms:g} ms`") + if workload.tpot_sla_ms is not None: + sla_lines.append(f"- TPOT objective: `{workload.tpot_sla_ms:g} ms`") + + sla_text = "\n".join(sla_lines) + if sla_text: + sla_text = ( + "\n## Supplied latency objectives\n\n" + + sla_text + + "\n\nThe generated benchmark uses these values with vLLM " + "`--goodput`.\n" + ) + + content = f"""# Optional Runtime Tuning Sweep + +The generated `config.yml` is the **single initial suggestion** and can be +deployed directly. The scheduler sweep keeps explicit `max-num-seqs` at the +per-replica concurrency lower bound while benchmarking nearby values for: + +- `max-num-batched-tokens` + +It does not test explicit `max-num-seqs` values below per-replica concurrency, +because those values can introduce scheduler queueing and severe TTFT +degradation. The vLLM-default references still test whether an explicit +`max-num-seqs` override is needed. + +In addition to nearby tuned values, the sweep includes three vLLM-default +references: + +- vLLM default `max-num-seqs` with the initial batch-token budget +- initial `max-num-seqs` with vLLM default `max-num-batched-tokens` +- vLLM defaults for both parameters + +The sweep runs every server from a generated `sweep_config.yml` that copies the +initial `config.yml` but removes these two scheduler keys. Tuned candidates add +explicit CLI values; a default-reference candidate simply omits the selected +key, allowing `vllm serve` to resolve its normal runtime default. This avoids +passing the literal string `None` and keeps defaults platform-, world-size-, +model-, and usage-context-aware. + +## Run + +```bash +./run_sweep.sh --dry-run +./run_sweep.sh +./recommend.py +``` + +For a quick one-run experiment: + +```bash +rm -rf results/runtime-tuning +./run_sweep.sh --num-runs 1 +./recommend.py +``` + +Resume an interrupted sweep: + +```bash +./run_sweep.sh --resume +``` + +Before measurement, the generated script runs one unmeasured warmup containing +one full concurrency window ({workload.concurrency} prompts). It is saved as +`warmup.json` for auditing but excluded from recommendation statistics. By +default, vLLM then benchmarks each parameter combination three times. + +The benchmark request count scales with the supplied workload concurrency using +the same model as vLLM's `PROMPTS_PER_CONCURRENCY` performance-benchmark +control: + +```text +num_prompts = min(1000, max(100, concurrency * 10)) +``` + +This targets about ten concurrency turnovers when neither bound applies, keeps +at least 100 requests for percentile/compliance measurements, and caps each run +at 1000 requests. All scheduler candidates use the same request count. +{sla_text} +## Outputs + +`recommend.py` writes: + +```text +recommended-config.yml +recommendation.json +``` + +With TTFT/TPOT objectives it requires duration-weighted combined compliance of +at least 99% plus median P99 TTFT/TPOT compliance. Scheduler candidates within +1% of the highest eligible output-token throughput are treated as practically +equivalent, and the recommender prefers fewer explicit scheduler overrides +within that set. Use `--throughput-equivalence-percent 0` for exact +highest-throughput selection. If no candidate qualifies, it records a +best-effort candidate but does not write `recommended-config.yml`. Without +latency objectives it applies the same scheduler equivalence policy. +Configurations with failed requests are excluded. TP/DP comparisons continue +to use exact highest-throughput selection. + +`recommended-config.yml` copies the initial configuration and changes only +`max-num-seqs` and `max-num-batched-tokens`. If a vLLM-default reference wins, +the corresponding key is removed from `recommended-config.yml`, allowing +`vllm serve` to resolve that parameter from its normal runtime default policy. +""" + path.write_text(content, encoding="utf-8") + + +def _write_report_script(path: Path, *, workload: WorkloadHints) -> None: + template_path = Path(__file__).with_name("sweep_html_report.py") + source = template_path.read_text(encoding="utf-8") + replacements = { + "DEFAULT_TTFT_SLA_MS: float | None = None": ( + f"DEFAULT_TTFT_SLA_MS: float | None = {workload.ttft_sla_ms!r}" + ), + "DEFAULT_TPOT_SLA_MS: float | None = None": ( + f"DEFAULT_TPOT_SLA_MS: float | None = {workload.tpot_sla_ms!r}" + ), + } + for marker, replacement in replacements.items(): + if marker not in source: + raise ValueError(f"HTML report template marker not found: {marker}") + source = source.replace(marker, replacement, 1) + path.write_text(source, encoding="utf-8") + path.chmod(path.stat().st_mode | 0o111) + + +def _write_post_benchmark_analysis_files( + directory: Path, *, workload: WorkloadHints +) -> list[Path]: + """Copy standalone reporting helpers into a generated sweep package.""" + source_dir = Path(__file__).resolve().parent + generated: list[Path] = [] + for source_name, target_name, executable in ( + ("sweep_visualization.py", "visualize.py", True), + ("requirements.txt", "requirements.txt", False), + ("VISUALIZATION.md", "VISUALIZATION.md", False), + ): + source = ( + source_dir / source_name + if source_name.endswith(".py") + else source_dir.parent / source_name + ) + target = directory / target_name + target.write_text(source.read_text(encoding="utf-8"), encoding="utf-8") + if executable: + target.chmod(target.stat().st_mode | 0o111) + generated.append(target) + report = directory / "report.py" + _write_report_script(report, workload=workload) + generated.append(report) + return generated + + +def write_sweep_files( + output_dir: str, + *, + config_path: str, + env_path: str, + config: dict[str, Any], + workload: WorkloadHints, +) -> list[Path]: + """Write an optional sweep package around the initial config suggestion.""" + validate_sweep_workload(workload) + + directory = Path(output_dir) + directory.mkdir(parents=True, exist_ok=True) + + sweep_config = directory / "sweep_config.yml" + serve_params = directory / "serve_params.json" + bench_params = directory / "bench_params.json" + run_script = directory / "run_sweep.sh" + recommend_script = directory / "recommend.py" + guide = directory / "SWEEP.md" + + config_rel = _relative_to(directory, config_path) + sweep_config_rel = _relative_to(directory, str(sweep_config)) + env_rel = _relative_to(directory, env_path) + + _write_sweep_config(sweep_config, config) + _write_json(serve_params, build_serve_params(config)) + _write_json(bench_params, build_bench_params(workload)) + request_model, tokenizer = _benchmark_models(config) + _write_run_script( + run_script, + config_rel=sweep_config_rel, + env_rel=env_rel, + request_model=request_model, + tokenizer=tokenizer, + workload=workload, + ) + _write_recommend_script( + recommend_script, + config_rel=config_rel, + env_rel=env_rel, + workload=workload, + ) + _write_guide(guide, workload) + analysis_files = _write_post_benchmark_analysis_files(directory, workload=workload) + + return [ + sweep_config, + serve_params, + bench_params, + run_script, + recommend_script, + guide, + *analysis_files, + ] + + +def write_parallel_layout_sweep_files( + output_dir: str, + *, + config_path: str, + env_path: str, + config: dict[str, Any], + workload: WorkloadHints, + numa_node_count: int, + tune_scheduler: bool = False, +) -> list[Path]: + """Write a standalone NUMA-aware TP/DP sweep package.""" + validate_sweep_workload(workload) + directory = Path(output_dir) + directory.mkdir(parents=True, exist_ok=True) + + parallel_params = directory / "parallel_layout_serve_params.json" + parallel_skips = directory / "parallel_layout_skips.json" + parallel_config = directory / "parallel-layout-benchmark-config.yml" + bench_params = directory / "bench_params.json" + run_parallel = directory / "run_parallel_layout_sweep.sh" + recommend_parallel = directory / "recommend_parallel_layout.py" + guide = directory / "SWEEP.md" + + initial_config_rel = _relative_to(directory, config_path) + parallel_config_rel = _relative_to(directory, str(parallel_config)) + selected_config_rel = "parallel-layout-config.yml" + env_rel = _relative_to(directory, env_path) + request_model, tokenizer = _benchmark_models(config) + + _write_sweep_config(parallel_config, config) + benchmark_config_rel = initial_config_rel if tune_scheduler else parallel_config_rel + + try: + model_metadata = inspect_model_parallel_metadata(config) + except Exception as exc: + print( + "Warning: could not inspect model parallel metadata; retaining the " + f"generic layouts and relying on sweep error isolation: {exc}", + file=sys.stderr, + ) + model_metadata = UNKNOWN_MODEL_PARALLEL_METADATA + cpu_moe_dp_supported = ( + cpu_supports_moe_dp_collectives() if model_metadata.is_moe else True + ) + parallel_candidates, skipped_candidates = build_parallel_layout_plan( + config, + workload, + numa_node_count, + model_metadata=model_metadata, + cpu_moe_dp_supported=cpu_moe_dp_supported, + tune_scheduler=tune_scheduler, + ) + _write_json(parallel_params, parallel_candidates) + _write_json(parallel_skips, skipped_candidates) + _write_json(bench_params, build_bench_params(workload)) + _write_run_script( + run_parallel, + config_rel=benchmark_config_rel, + env_rel=env_rel, + request_model=request_model, + tokenizer=tokenizer, + workload=workload, + serve_params_name=parallel_params.name, + experiment_name="parallel-layout", + ) + _write_recommend_script( + recommend_parallel, + config_rel=initial_config_rel, + env_rel=env_rel, + workload=workload, + results_dir="results/parallel-layout", + output_config=selected_config_rel, + output_json="parallel-layout-recommendation.json", + ) + + guide.write_text( + f"""# Parallel-Layout Sweep + +This package tunes only the NUMA-aware TP/DP layout. + +Primary candidates use every effective NUMA node: + +```text +tensor_parallel_size * data_parallel_size = {numa_node_count} +``` + +Tensor parallelism is restricted to `1`, `2`, `4`, or `8`. The largest +supported TP not exceeding the NUMA-node count is also included, even when it +leaves some NUMA nodes idle. + +Before benchmarking, the generator applies two MoE safety checks: + +- When the CPU communicator lacks variable-size MoE collectives, MoE candidates + use `DP=1`; every supported TP size is retained as a replacement candidate. +- With expert parallelism disabled and a known MoE intermediate size, candidates + must satisfy `moe_intermediate_size % (TP * DP * PCP) == 0`. + +Rejected layouts and their reasons are written to +`parallel_layout_skips.json`. If model metadata cannot be inspected, generic +layouts are retained and the runner's per-combination error isolation applies. + +Run: + +```bash +./run_parallel_layout_sweep.sh --dry-run +./run_parallel_layout_sweep.sh +./recommend_parallel_layout.py +``` + +The recommender writes: + +```text +parallel-layout-config.yml +parallel-layout-recommendation.json +``` + +This mode stops after TP/DP selection. Use `--generate-full-sweep` when +concurrency and scheduler tuning should follow automatically. +""", + encoding="utf-8", + ) + analysis_files = _write_post_benchmark_analysis_files(directory, workload=workload) + + return [ + parallel_params, + parallel_skips, + bench_params, + run_parallel, + recommend_parallel, + guide, + *analysis_files, + ] + + +def build_concurrency_bench_params(workload: WorkloadHints) -> list[dict[str, Any]]: + """Build the fixed workload shape for concurrency tuning.""" + validate_sweep_workload(workload) + assert workload.input_tokens is not None + assert workload.output_tokens is not None + assert workload.concurrency is not None + return [ + { + "_benchmark_name": "user_workload", + "random_input_len": workload.input_tokens, + "random_output_len": workload.output_tokens, + "num_prompts": _num_prompts_for_concurrency(workload.concurrency), + } + ] + + +def _write_adaptive_concurrency_script(path: Path) -> None: + template_path = Path(__file__).with_name("adaptive_concurrency.py") + path.write_text(template_path.read_text(encoding="utf-8"), encoding="utf-8") + path.chmod(path.stat().st_mode | 0o111) + + +def _write_concurrency_run_script( + path: Path, + *, + config_rel: str, + env_rel: str, + request_model: str, + tokenizer: str, + workload: WorkloadHints, +) -> None: + bench_parts = [ + "vllm bench serve", + "--backend vllm", + f"--model {shlex.quote(request_model)}", + f"--tokenizer {shlex.quote(tokenizer)}", + "--dataset-name random", + "--request-rate inf", + "--ignore-eos", + "--metric-percentiles 99", + ] + goodput_pairs: list[str] = [] + if workload.ttft_sla_ms is not None: + goodput_pairs.append(f"ttft:{workload.ttft_sla_ms:g}") + if workload.tpot_sla_ms is not None: + goodput_pairs.append(f"tpot:{workload.tpot_sla_ms:g}") + if goodput_pairs: + bench_parts.append( + "--goodput " + " ".join(shlex.quote(pair) for pair in goodput_pairs) + ) + + bench_cmd = " ".join(bench_parts) + use_adaptive_sla_search = bool( + workload.ttft_sla_ms is not None or workload.tpot_sla_ms is not None + ) + adaptive_args = [ + f"--seed-concurrency {workload.concurrency}", + f"--input-tokens {workload.input_tokens}", + f"--output-tokens {workload.output_tokens}", + ] + if workload.ttft_sla_ms is not None: + adaptive_args.append(f"--ttft-sla-ms {workload.ttft_sla_ms:g}") + if workload.tpot_sla_ms is not None: + adaptive_args.append(f"--tpot-sla-ms {workload.tpot_sla_ms:g}") + adaptive_args_text = " \\\n ".join(adaptive_args) + + script = f"""#!/usr/bin/env bash +set -euo pipefail + +SCRIPT_DIR="$(cd -- "$(dirname -- "${{BASH_SOURCE[0]}}")" && pwd)" +CONFIG_PATH="${{SCRIPT_DIR}}/{config_rel}" +ENV_PATH="${{SCRIPT_DIR}}/{env_rel}" + +source "${{ENV_PATH}}" + +# Released/nightly vLLM images keep recipe-relative assets such as +# examples/*.jinja under /vllm-workspace. Launch sweep server subprocesses +# from there when available so relative recipe assets resolve like they do +# in serve_with_recipe.sh. +if [[ -d /vllm-workspace/examples ]]; then + cd /vllm-workspace +fi + +# recipe tools may be newer than the installed vLLM package. Only pass the +# resilience flag when this vLLM CLI actually supports it. +CONTINUE_ON_ERROR_ARG="" +if [[ "{int(use_adaptive_sla_search)}" == "1" ]] || \ + vllm bench sweep serve_workload --help 2>&1 | grep -q -- "--continue-on-error"; then + CONTINUE_ON_ERROR_ARG="--continue-on-error" +fi + +# On older vLLM versions, a failed benchmark aborts the sweep process. Retry the +# interrupted sweep with --resume so one transient run failure does not discard +# completed work or force a manual restart. +SWEEP_RETRY_COUNT="${{VLLM_RECIPE_SWEEP_RETRY_COUNT:-0}}" +MAX_SWEEP_RETRIES="${{VLLM_RECIPE_SWEEP_RETRIES:-2}}" +SERVER_READY_TIMEOUT="${{VLLM_RECIPE_SERVER_READY_TIMEOUT:-1800}}" + +run_concurrency_sweep() {{ + if [[ "{int(use_adaptive_sla_search)}" == "1" ]]; then + python3 "${{SCRIPT_DIR}}/adaptive_concurrency.py" \ + --serve-cmd "vllm serve --config '${{CONFIG_PATH}}'" \ + --bench-cmd "{bench_cmd}" \ + --results-dir "${{SCRIPT_DIR}}/results/concurrency-tuning" \ + {adaptive_args_text} \ + --server-ready-timeout "${{SERVER_READY_TIMEOUT}}" \ + ${{CONTINUE_ON_ERROR_ARG}} \ + "$@" + else + vllm bench sweep serve_workload \ + --serve-cmd "vllm serve --config '${{CONFIG_PATH}}'" \ + --bench-cmd "{bench_cmd}" \ + --bench-params "${{SCRIPT_DIR}}/concurrency_bench_params.json" \ + --output-dir "${{SCRIPT_DIR}}/results" \ + --experiment-name concurrency-tuning \ + --workload-var max_concurrency \ + --workload-iters 10 \ + --warmup-num-prompts {workload.concurrency} \ + ${{CONTINUE_ON_ERROR_ARG}} \ + "$@" + fi +}} + +run_concurrency_sweep || {{ + if [[ "${{SWEEP_RETRY_COUNT}}" -ge "${{MAX_SWEEP_RETRIES}}" ]]; then + echo "Sweep failed after ${{MAX_SWEEP_RETRIES}} automatic retries." >&2 + exit 1 + fi + + NEXT_RETRY=$((SWEEP_RETRY_COUNT + 1)) + RETRY_ARGS=() + for arg in "$@"; do + [[ "$arg" == "--resume" ]] && continue + RETRY_ARGS+=("$arg") + done + + EXPERIMENT_DIR="${{SCRIPT_DIR}}/results/concurrency-tuning" + if [[ -d "${{EXPERIMENT_DIR}}" ]]; then + echo "Sweep command failed;" \ + "retry ${{NEXT_RETRY}}/${{MAX_SWEEP_RETRIES}} with --resume" + RETRY_ARGS=(--resume "${{RETRY_ARGS[@]}}") + else + echo "Sweep command failed before resumable state was created;" \ + "retry ${{NEXT_RETRY}}/${{MAX_SWEEP_RETRIES}} from the start" + fi + + VLLM_RECIPE_SWEEP_RETRY_COUNT="${{NEXT_RETRY}}" exec "$0" "${{RETRY_ARGS[@]}}" + }} +""" + path.write_text(script, encoding="utf-8") + path.chmod(path.stat().st_mode | 0o111) + + +def _write_concurrency_recommend_script(path: Path, *, workload: WorkloadHints) -> None: + template_path = Path(__file__).with_name("concurrency_recommendation.py") + source = template_path.read_text(encoding="utf-8") + replacements = { + "DEFAULT_TTFT_SLA_MS: float | None = None": ( + f"DEFAULT_TTFT_SLA_MS: float | None = {workload.ttft_sla_ms!r}" + ), + "DEFAULT_TPOT_SLA_MS: float | None = None": ( + f"DEFAULT_TPOT_SLA_MS: float | None = {workload.tpot_sla_ms!r}" + ), + "DEFAULT_SEED_CONCURRENCY: int | None = None": ( + f"DEFAULT_SEED_CONCURRENCY: int | None = {workload.concurrency!r}" + ), + } + for marker, replacement in replacements.items(): + if marker not in source: + raise ValueError(f"Concurrency recommender marker not found: {marker}") + source = source.replace(marker, replacement, 1) + path.write_text(source, encoding="utf-8") + path.chmod(path.stat().st_mode | 0o111) + + +def _write_scheduler_prepare_script(path: Path) -> None: + template_path = Path(__file__).with_name("scheduler_sweep_prepare.py") + path.write_text(template_path.read_text(encoding="utf-8"), encoding="utf-8") + path.chmod(path.stat().st_mode | 0o111) + + +def _write_scheduler_wrapper(path: Path, *, workload: WorkloadHints) -> None: + assert workload.input_tokens is not None + assert workload.output_tokens is not None + args = [ + 'python3 "${SCRIPT_DIR}/prepare_scheduler.py"', + '--config "${SCRIPT_DIR}/parallel-layout-config.yml"', + '--concurrency-recommendation "${SCRIPT_DIR}/concurrency-recommendation.json"', + f"--input-tokens {workload.input_tokens}", + f"--output-tokens {workload.output_tokens}", + ] + if workload.tpot_sla_ms is not None: + args.append(f"--tpot-sla-ms {workload.tpot_sla_ms:g}") + if workload.target_qps is not None: + args.append(f"--target-qps {workload.target_qps:g}") + args.extend( + [ + '--output-config "${SCRIPT_DIR}/parallel-layout-sweep-config.yml"', + '--serve-params "${SCRIPT_DIR}/serve_params.json"', + '--bench-params "${SCRIPT_DIR}/bench_params.json"', + ] + ) + prepare_cmd = " \\\n ".join(args) + script = f"""#!/usr/bin/env bash +set -euo pipefail + +SCRIPT_DIR="$(cd -- "$(dirname -- "${{BASH_SOURCE[0]}}")" && pwd)" + +{prepare_cmd} + +exec "${{SCRIPT_DIR}}/run_sweep.sh" "$@" +""" + path.write_text(script, encoding="utf-8") + path.chmod(path.stat().st_mode | 0o111) + + +def _recommend_sla_args(workload: WorkloadHints) -> list[str]: + args = [] + if workload.ttft_sla_ms is not None: + args.append(f"--ttft-sla-ms {workload.ttft_sla_ms:g}") + if workload.tpot_sla_ms is not None: + args.append(f"--tpot-sla-ms {workload.tpot_sla_ms:g}") + return args + + +def _write_full_run_script(path: Path, *, workload: WorkloadHints) -> None: + recommend_sla_args = " ".join(_recommend_sla_args(workload)) + script = f"""#!/usr/bin/env bash +set -euo pipefail + +SCRIPT_DIR="$(cd -- "$(dirname -- "${{BASH_SOURCE[0]}}")" && pwd)" +STAGES=all +RUN_ARGS=() +SERVER_READY_TIMEOUT="${{VLLM_RECIPE_SERVER_READY_TIMEOUT:-1800}}" +RECOMMEND_SLA_ARGS=({recommend_sla_args}) +while (($#)); do + case "$1" in + --stages) + if (($# < 2)); then + echo "--stages requires a comma-separated stage list." >&2 + exit 2 + fi + STAGES="$2" + shift 2 + ;; + --stages=*) STAGES="${{1#*=}}"; shift ;; + --server-ready-timeout) + if (($# < 2)); then + echo "--server-ready-timeout requires a value." >&2 + exit 2 + fi + SERVER_READY_TIMEOUT="$2" + RUN_ARGS+=("$1" "$2") + shift 2 + ;; + --server-ready-timeout=*) + SERVER_READY_TIMEOUT="${{1#*=}}" + RUN_ARGS+=("$1") + shift + ;; + --help|-h) + echo "Usage: $0 [--stages all|parallel-layout,concurrency,scheduler] [sweep args]" + echo "Selected stages always run in dependency order. Default: all." + exit 0 + ;; + *) RUN_ARGS+=("$1"); shift ;; + esac +done +case "$STAGES" in + all) STAGES=parallel-layout,concurrency,scheduler ;; + ""|,*|*,|*,,*) echo "Invalid stage list: $STAGES" >&2; exit 2 ;; +esac +IFS=, read -r -a SELECTED_STAGES <<< "$STAGES" +for stage in "${{SELECTED_STAGES[@]}}"; do + case "$stage" in + parallel-layout|concurrency|scheduler) ;; + *) echo "Unknown sweep stage: $stage" >&2; exit 2 ;; + esac +done +has_stage() {{ [[ ",$STAGES," == *",$1,"* ]]; }} + +# Keep the concurrency runner's server-start timeout explicit even when sweep +# arguments pass through multiple generated wrapper scripts. +export VLLM_RECIPE_SERVER_READY_TIMEOUT="${{SERVER_READY_TIMEOUT}}" + +# Check dependencies before starting any benchmarks. +if has_stage scheduler && ! has_stage concurrency && + [[ ! -f "${{SCRIPT_DIR}}/concurrency-recommendation.json" ]]; then + echo "Scheduler requires concurrency-recommendation.json; run concurrency first." >&2 + exit 2 +fi +if ! has_stage parallel-layout && + [[ ! -f "${{SCRIPT_DIR}}/parallel-layout-config.yml" ]]; then + cp "${{SCRIPT_DIR}}/initial-config.yml" "${{SCRIPT_DIR}}/parallel-layout-config.yml" +fi + +if has_stage parallel-layout; then + "${{SCRIPT_DIR}}/run_parallel_layout_sweep.sh" "${{RUN_ARGS[@]}}" + "${{SCRIPT_DIR}}/recommend_parallel_layout.py" "${{RECOMMEND_SLA_ARGS[@]}}" +fi +if has_stage concurrency; then + "${{SCRIPT_DIR}}/run_concurrency_sweep.sh" "${{RUN_ARGS[@]}}" + "${{SCRIPT_DIR}}/recommend_concurrency.py" "${{RECOMMEND_SLA_ARGS[@]}}" +fi +if has_stage scheduler; then + "${{SCRIPT_DIR}}/run_scheduler_sweep.sh" "${{RUN_ARGS[@]}}" + "${{SCRIPT_DIR}}/recommend.py" "${{RECOMMEND_SLA_ARGS[@]}}" +fi + +if ! "${{SCRIPT_DIR}}/report.py" "${{RECOMMEND_SLA_ARGS[@]}}"; then + echo "Warning: sweep completed, but HTML report generation failed." >&2 +fi +""" + path.write_text(script, encoding="utf-8") + path.chmod(path.stat().st_mode | 0o111) + + +def _write_full_guide(path: Path, workload: WorkloadHints) -> None: + content = f"""# End-to-End Runtime Tuning + +This package keeps one fixed workload shape and tunes in dependency order: + +```text +TP/DP -> max_concurrency -> scheduler +``` + +The supplied `--concurrency={workload.concurrency}` is the representative load +used for TP/DP selection and to size the Workload Explorer dataset. Stage 2 +selects the measured SLA-feasible `max_concurrency`. Stage 3 recalculates its +scheduler baseline from that concurrency and the selected DP layout. + +Run all stages: + +```bash +./run_full_sweep.sh +``` + +After the final recommendation, the full runner writes `sweep-report.html`. +Report-generation failures are warnings and do not change sweep success. + +Run selected stages (always executed in dependency order): + +```bash +./run_full_sweep.sh --stages concurrency +./run_full_sweep.sh --stages concurrency,scheduler +./run_full_sweep.sh --stages scheduler --resume +``` + +When parallel-layout tuning is skipped, an existing `parallel-layout-config.yml` +is reused; otherwise `initial-config.yml` supplies the fixed TP/DP layout. +Scheduler-only runs require an existing `concurrency-recommendation.json`. +Other arguments are forwarded to the selected benchmark runners. + +Or run each stage explicitly: + +```bash +./run_parallel_layout_sweep.sh +./recommend_parallel_layout.py +./run_concurrency_sweep.sh +./recommend_concurrency.py +./run_scheduler_sweep.sh +./recommend.py +``` + +Generated recipe sweeps use `--continue-on-error`. An isolated warmup or +benchmark invocation failure is recorded and skipped. A combination with no +successful measured runs remains retryable with `--resume`. +""" + path.write_text(content, encoding="utf-8") + + +def write_concurrency_sweep_files( + output_dir: str, + *, + config_path: str, + env_path: str, + config: dict[str, Any], + workload: WorkloadHints, + tune_scheduler: bool = False, +) -> list[Path]: + """Write a standalone max_concurrency tuning package.""" + validate_sweep_workload(workload) + directory = Path(output_dir) + directory.mkdir(parents=True, exist_ok=True) + + bench_params = directory / "concurrency_bench_params.json" + concurrency_config = directory / "concurrency-benchmark-config.yml" + adaptive_script = directory / "adaptive_concurrency.py" + run_script = directory / "run_concurrency_sweep.sh" + recommend_script = directory / "recommend_concurrency.py" + + config_rel = _relative_to(directory, config_path) + concurrency_config_rel = _relative_to(directory, str(concurrency_config)) + env_rel = _relative_to(directory, env_path) + request_model, tokenizer = _benchmark_models(config) + + _write_sweep_config(concurrency_config, config) + benchmark_config_rel = config_rel if tune_scheduler else concurrency_config_rel + + _write_json(bench_params, build_concurrency_bench_params(workload)) + _write_adaptive_concurrency_script(adaptive_script) + _write_concurrency_run_script( + run_script, + config_rel=benchmark_config_rel, + env_rel=env_rel, + request_model=request_model, + tokenizer=tokenizer, + workload=workload, + ) + _write_concurrency_recommend_script(recommend_script, workload=workload) + analysis_files = _write_post_benchmark_analysis_files(directory, workload=workload) + return [ + bench_params, + concurrency_config, + adaptive_script, + run_script, + recommend_script, + *analysis_files, + ] + + +def write_full_sweep_files( + output_dir: str, + *, + config_path: str, + env_path: str, + config: dict[str, Any], + workload: WorkloadHints, + numa_node_count: int, + tune_scheduler: bool = False, +) -> list[Path]: + """Write TP/DP -> concurrency -> scheduler end-to-end tuning artifacts.""" + files = write_parallel_layout_sweep_files( + output_dir, + config_path=config_path, + env_path=env_path, + config=config, + workload=workload, + numa_node_count=numa_node_count, + tune_scheduler=tune_scheduler, + ) + + directory = Path(output_dir) + concurrency_bench = directory / "concurrency_bench_params.json" + adaptive_concurrency = directory / "adaptive_concurrency.py" + run_concurrency = directory / "run_concurrency_sweep.sh" + recommend_concurrency = directory / "recommend_concurrency.py" + prepare_scheduler = directory / "prepare_scheduler.py" + scheduler_runner = directory / "run_sweep.sh" + recommend_scheduler = directory / "recommend.py" + run_scheduler = directory / "run_scheduler_sweep.sh" + run_full = directory / "run_full_sweep.sh" + guide = directory / "SWEEP.md" + + env_rel = _relative_to(directory, env_path) + request_model, tokenizer = _benchmark_models(config) + + _write_json(concurrency_bench, build_concurrency_bench_params(workload)) + _write_adaptive_concurrency_script(adaptive_concurrency) + _write_concurrency_run_script( + run_concurrency, + config_rel="parallel-layout-config.yml", + env_rel=env_rel, + request_model=request_model, + tokenizer=tokenizer, + workload=workload, + ) + _write_concurrency_recommend_script(recommend_concurrency, workload=workload) + + # Full tuning owns the scheduler stage. The standalone parallel-layout mode + # intentionally stops after TP/DP selection. + _write_scheduler_prepare_script(prepare_scheduler) + _write_run_script( + scheduler_runner, + config_rel="parallel-layout-sweep-config.yml", + env_rel=env_rel, + request_model=request_model, + tokenizer=tokenizer, + workload=workload, + ) + _write_recommend_script( + recommend_scheduler, + config_rel="parallel-layout-config.yml", + env_rel=env_rel, + workload=workload, + ) + _write_scheduler_wrapper(run_scheduler, workload=workload) + + initial_config = directory / "initial-config.yml" + initial_config.write_text(yaml.safe_dump(config, sort_keys=False), encoding="utf-8") + _write_full_run_script(run_full, workload=workload) + _write_full_guide(guide, workload) + + return [ + *files, + concurrency_bench, + adaptive_concurrency, + run_concurrency, + recommend_concurrency, + prepare_scheduler, + scheduler_runner, + recommend_scheduler, + run_scheduler, + run_full, + initial_config, + ] diff --git a/tools/recipes/sweep/sweep_html_report.py b/tools/recipes/sweep/sweep_html_report.py new file mode 100644 index 0000000000..bac10f9b4a --- /dev/null +++ b/tools/recipes/sweep/sweep_html_report.py @@ -0,0 +1,756 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +"""Generate a self-contained HTML summary from completed recipe sweeps.""" + +from __future__ import annotations + +import argparse +import json +from datetime import UTC, datetime +from html import escape +from pathlib import Path +from typing import Any + +STAGES = ( + ( + "parallel-layout", + "Parallel layout", + "parallel-layout-recommendation.json", + ), + ( + "concurrency-tuning", + "Concurrency", + "concurrency-recommendation.json", + ), + ("runtime-tuning", "Scheduler", "recommendation.json"), +) + +DEFAULT_TTFT_SLA_MS: float | None = None +DEFAULT_TPOT_SLA_MS: float | None = None + + +def _read_object(path: Path) -> dict[str, Any] | None: + if not path.is_file(): + return None + value = json.loads(path.read_text(encoding="utf-8")) + if not isinstance(value, dict): + raise ValueError(f"{path} does not contain a JSON object.") + return value + + +def _summary_runs(stage_dir: Path) -> list[dict[str, Any]]: + runs: list[dict[str, Any]] = [] + for path in sorted(stage_dir.glob("**/summary.json")): + value = json.loads(path.read_text(encoding="utf-8")) + if isinstance(value, list): + runs.extend(item for item in value if isinstance(item, dict)) + return runs + + +def _number(value: Any) -> float | None: + if isinstance(value, bool) or not isinstance(value, int | float): + return None + return float(value) + + +def _fmt(value: Any, digits: int = 1) -> str: + number = _number(value) + if number is None: + return "-" + return f"{number:.{digits}f}" + + +def _fmt_int(value: Any) -> str: + number = _number(value) + return "-" if number is None else f"{number:.0f}" + + +def _fmt_percent(value: Any) -> str: + number = _number(value) + return "-" if number is None else f"{number:.1f}%" + + +def _text(value: Any) -> str: + if value is None: + return "-" + return escape(str(value)) + + +def _resolve_sla( + reports: dict[str, dict[str, Any] | None], + key: str, + expected: float | None, +) -> float | None: + values: dict[str, float | None] = {} + for stage, label, _ in STAGES: + report = reports[stage] + if report is not None: + values[label] = _number((report.get("slo") or {}).get(key)) + + non_null = {value for value in values.values() if value is not None} + if expected is not None: + mismatched = { + label: value for label, value in values.items() if value != expected + } + if mismatched: + details = ", ".join( + f"{label}={value}" for label, value in mismatched.items() + ) + raise ValueError( + f"Recommendation {key} SLA does not match the command-line " + f"value {expected}: {details}" + ) + return expected + + if len(non_null) > 1: + details = ", ".join(f"{label}={value}" for label, value in values.items()) + raise ValueError(f"Recommendation {key} SLA values disagree: {details}") + return next(iter(non_null), None) + + +def _resolve_minimum_compliance( + reports: dict[str, dict[str, Any] | None], +) -> float | None: + values = { + _number(report.get("minimum_compliance_ratio")) + for report in reports.values() + if report is not None + } + non_null = {value for value in values if value is not None} + if len(non_null) > 1: + raise ValueError( + "Recommendation minimum-compliance values disagree: " + + ", ".join(str(value) for value in sorted(non_null)) + ) + return next(iter(non_null), None) + + +def _table(headers: list[str], rows: list[list[Any]]) -> str: + if not rows: + return '

No completed candidates were found.

' + head = "".join(f"{escape(header)}" for header in headers) + body = [] + for row in rows: + cells = "".join(f"{_text(cell)}" for cell in row) + body.append("" + cells + "") + return ( + '
' + + head + + "" + + "".join(body) + + "
" + ) + + +def _status_badge(value: Any) -> str: + status = str(value or "unknown").replace("_", " ") + css_class = "pass" if value == "sla_feasible" else "warn" + return f'{escape(status)}' + + +def _sla_label(candidate: dict[str, Any]) -> str: + if candidate.get("sla_eligible") is True: + return "PASS" + if candidate.get("sla_eligible") is False: + return "FAIL" + return "-" + + +def _sla_reason( + candidate: dict[str, Any], + *, + ttft_sla_ms: float | None, + tpot_sla_ms: float | None, + minimum_compliance: float | None, +) -> str: + if candidate.get("valid") is False or candidate.get("benchmark_valid") is False: + return str(candidate.get("invalid_reason") or "benchmark invalid") + + reasons: list[str] = [] + ttft = _number(candidate.get("median_p99_ttft_ms")) + tpot = _number(candidate.get("median_p99_tpot_ms")) + compliance = _number(candidate.get("combined_compliance_ratio")) + + if ttft_sla_ms is not None and (ttft is None or ttft > ttft_sla_ms): + reasons.append( + "TTFT " + + ("missing" if ttft is None else f"{ttft:.1f} ms > {ttft_sla_ms:.1f} ms") + ) + if tpot_sla_ms is not None and (tpot is None or tpot > tpot_sla_ms): + reasons.append( + "TPOT " + + ("missing" if tpot is None else f"{tpot:.1f} ms > {tpot_sla_ms:.1f} ms") + ) + if minimum_compliance is not None and ( + compliance is None or compliance < minimum_compliance + ): + reasons.append( + "combined compliance " + + ( + "missing" + if compliance is None + else f"{compliance * 100:.2f}% < {minimum_compliance * 100:.2f}%" + ) + ) + + return "PASS" if not reasons else "; ".join(reasons) + + +def _scheduler_value(candidate: dict[str, Any], key: str) -> str: + value = candidate.get(key) + return "vLLM default" if value is None else str(value) + + +def _selected_settings( + parallel: dict[str, Any] | None, + concurrency: dict[str, Any] | None, + scheduler: dict[str, Any] | None, +) -> list[list[Any]]: + parallel_rec = (parallel or {}).get("recommended") or {} + concurrency_rec = (concurrency or {}).get("recommended") or {} + scheduler_rec = (scheduler or {}).get("recommended") or {} + max_num_seqs = ( + _scheduler_value(scheduler_rec, "max_num_seqs") + if scheduler is not None + else "-" + ) + max_num_batched_tokens = ( + _scheduler_value(scheduler_rec, "max_num_batched_tokens") + if scheduler is not None + else "-" + ) + return [ + ["tensor-parallel-size", parallel_rec.get("tensor_parallel_size")], + ["data-parallel-size", parallel_rec.get("data_parallel_size")], + ["max concurrency", concurrency_rec.get("max_concurrency")], + ["max-num-seqs", max_num_seqs], + ["max-num-batched-tokens", max_num_batched_tokens], + ] + + +def _stage_summary( + reports: dict[str, dict[str, Any] | None], +) -> list[list[Any]]: + rows = [] + for stage, label, _ in STAGES: + report = reports[stage] + if report is None: + rows.append([label, "not available", "-", "-", "-", "-"]) + continue + measured = report.get("measured") or {} + rows.append( + [ + label, + str(report.get("status", "unknown")).replace("_", " "), + _fmt(measured.get("mean_output_throughput")), + _fmt(measured.get("mean_request_goodput"), 3), + _fmt_int(measured.get("mean_p99_ttft_ms")), + _fmt(measured.get("mean_p99_tpot_ms")), + ] + ) + return rows + + +def _parallel_rows( + report: dict[str, Any] | None, + *, + ttft_sla_ms: float | None, + tpot_sla_ms: float | None, + minimum_compliance: float | None, +) -> list[list[Any]]: + rows = [] + for candidate in (report or {}).get("candidates", []): + rows.append( + [ + candidate.get("tensor_parallel_size"), + candidate.get("data_parallel_size"), + candidate.get("max_num_seqs"), + candidate.get("max_num_batched_tokens"), + _fmt(candidate.get("mean_output_throughput")), + _fmt_int(candidate.get("mean_p99_ttft_ms")), + _fmt(candidate.get("mean_p99_tpot_ms")), + _fmt_percent(candidate.get("combined_compliance_percent")), + _sla_label(candidate), + _sla_reason( + candidate, + ttft_sla_ms=ttft_sla_ms, + tpot_sla_ms=tpot_sla_ms, + minimum_compliance=minimum_compliance, + ), + ] + ) + return rows + + +def _concurrency_rows( + report: dict[str, Any] | None, + *, + ttft_sla_ms: float | None, + tpot_sla_ms: float | None, + minimum_compliance: float | None, +) -> list[list[Any]]: + candidates = (report or {}).get("candidates", []) + candidates = sorted(candidates, key=lambda item: item.get("max_concurrency", 0)) + rows = [] + for candidate in candidates: + rows.append( + [ + candidate.get("max_concurrency"), + _fmt(candidate.get("mean_output_throughput")), + _fmt(candidate.get("mean_request_throughput"), 3), + _fmt(candidate.get("mean_request_goodput"), 3), + _fmt_percent(candidate.get("combined_compliance_percent")), + _fmt_int(candidate.get("mean_p99_ttft_ms")), + _fmt(candidate.get("mean_p99_tpot_ms")), + _sla_label(candidate), + _sla_reason( + candidate, + ttft_sla_ms=ttft_sla_ms, + tpot_sla_ms=tpot_sla_ms, + minimum_compliance=minimum_compliance, + ), + ] + ) + return rows + + +def _scheduler_rows( + report: dict[str, Any] | None, + *, + ttft_sla_ms: float | None, + tpot_sla_ms: float | None, + minimum_compliance: float | None, +) -> list[list[Any]]: + rows = [] + for candidate in (report or {}).get("candidates", []): + rows.append( + [ + _scheduler_value(candidate, "max_num_seqs"), + _scheduler_value(candidate, "max_num_batched_tokens"), + _fmt(candidate.get("mean_output_throughput")), + _fmt(candidate.get("mean_request_goodput"), 3), + _fmt_int(candidate.get("mean_p99_ttft_ms")), + _fmt(candidate.get("mean_p99_tpot_ms")), + _fmt_percent(candidate.get("combined_compliance_percent")), + _sla_label(candidate), + _sla_reason( + candidate, + ttft_sla_ms=ttft_sla_ms, + tpot_sla_ms=tpot_sla_ms, + minimum_compliance=minimum_compliance, + ), + ] + ) + return rows + + +def _metadata(runs: dict[str, list[dict[str, Any]]]) -> dict[str, Any]: + for stage, _, _ in STAGES: + if runs[stage]: + first = runs[stage][0] + return { + "model": first.get("model_id", "unknown"), + "input_tokens": first.get("random_input_len"), + "output_tokens": first.get("random_output_len"), + "request_rate": first.get("request_rate"), + } + return { + "model": "unknown", + "input_tokens": None, + "output_tokens": None, + "request_rate": None, + } + + +def _observations( + reports: dict[str, dict[str, Any] | None], + runs: dict[str, list[dict[str, Any]]], +) -> list[tuple[str, str]]: + observations: list[tuple[str, str]] = [] + parallel = reports["parallel-layout"] + concurrency = reports["concurrency-tuning"] + scheduler = reports["runtime-tuning"] + + parallel_candidates = (parallel or {}).get("candidates", []) + if len(parallel_candidates) == 1: + observations.append( + ( + "warning", + ( + "Only one parallel-layout candidate completed. The result " + "validates that layout but does not establish an optimum." + ), + ) + ) + + if parallel and concurrency and runs["parallel-layout"]: + representative = runs["parallel-layout"][0].get("max_concurrency") + concurrency_values = { + item.get("max_concurrency") for item in concurrency.get("candidates", []) + } + parallel_passed = parallel_candidates and parallel_candidates[0].get( + "p99_sla_eligible" + ) + if representative not in concurrency_values and parallel_passed: + observations.append( + ( + "warning", + ( + f"Concurrency {representative} passed in the " + "parallel-layout stage but was not included in the " + "concurrency grid. Treat the automated concurrency " + "choice as conservative." + ), + ) + ) + + concurrency_candidates = (concurrency or {}).get("candidates", []) + first_failure = next( + ( + item + for item in sorted( + concurrency_candidates, + key=lambda candidate: candidate.get("max_concurrency", 0), + ) + if item.get("p99_sla_eligible") is False + ), + None, + ) + if first_failure and concurrency: + slo = concurrency.get("slo") or {} + ttft = _number(first_failure.get("mean_p99_ttft_ms")) + tpot = _number(first_failure.get("mean_p99_tpot_ms")) + ttft_slo = _number(slo.get("ttft_ms")) + tpot_slo = _number(slo.get("tpot_ms")) + if ( + ttft + and ttft_slo + and ttft > ttft_slo + and tpot is not None + and (tpot_slo is None or tpot <= tpot_slo) + ): + observations.append( + ( + "info", + ( + "TTFT is the first limiting SLA; TPOT remains within " + "its objective at the first failing concurrency." + ), + ) + ) + + scheduler_candidates = (scheduler or {}).get("candidates", []) + scheduler_throughputs = [ + number + for number in ( + _number(item.get("mean_output_throughput")) for item in scheduler_candidates + ) + if number is not None + ] + if len(scheduler_throughputs) > 1 and min(scheduler_throughputs) > 0: + spread = (max(scheduler_throughputs) / min(scheduler_throughputs) - 1) * 100 + if spread <= 3: + observations.append( + ( + "info", + ( + "Scheduler output throughput is tightly grouped: " + f"{spread:.2f}% total spread across measured candidates." + ), + ) + ) + + if not observations: + observations.append( + ("info", "No automatic coverage or SLA warnings were detected.") + ) + return observations + + +def _observation_html(observations: list[tuple[str, str]]) -> str: + return "".join( + f'
{escape(message)}
' + for level, message in observations + ) + + +def generate_report( + root: Path, + output: Path, + title: str, + ttft_sla_ms: float | None = None, + tpot_sla_ms: float | None = None, +) -> None: + reports = {stage: _read_object(root / filename) for stage, _, filename in STAGES} + if not any(reports.values()): + names = ", ".join(filename for _, _, filename in STAGES) + raise ValueError(f"No recommendation files found under {root}: {names}") + + ttft_sla_ms = _resolve_sla(reports, "ttft_ms", ttft_sla_ms) + tpot_sla_ms = _resolve_sla(reports, "tpot_ms", tpot_sla_ms) + minimum_compliance = _resolve_minimum_compliance(reports) + + runs = {stage: _summary_runs(root / "results" / stage) for stage, _, _ in STAGES} + meta = _metadata(runs) + observations = _observations(reports, runs) + concurrency = reports["concurrency-tuning"] or {} + scheduler = reports["runtime-tuning"] or {} + measured = scheduler.get("measured") or concurrency.get("measured") or {} + recommended_concurrency = (concurrency.get("recommended") or {}).get( + "max_concurrency" + ) + generated = datetime.now(UTC).strftime("%Y-%m-%d %H:%M UTC") + + styles = """ +:root { color-scheme: light; --ink: #17324d; --blue: #146c94; + --teal: #2a9d8f; --green: #2e7d32; --amber: #d97706; + --red: #c0392b; --line: #d8dee4; --soft: #f3f5f7; } +* { box-sizing: border-box; } +body { margin: 0; background: #eef2f5; color: var(--ink); + font: 15px/1.45 Arial, sans-serif; } +main { max-width: 1180px; margin: 28px auto; padding: 0 20px 40px; } +header, section { background: white; border: 1px solid var(--line); + border-radius: 12px; margin-bottom: 18px; padding: 24px; } +h1 { margin: 0 0 6px; font-size: 30px; } +h2 { margin: 0 0 14px; font-size: 21px; } +h3 { margin: 20px 0 10px; font-size: 16px; color: var(--blue); } +.sub, .muted { color: #607486; } +.cards { display: grid; grid-template-columns: repeat(4, minmax(0, 1fr)); + gap: 12px; margin-top: 20px; } +.card { background: var(--soft); border-radius: 9px; padding: 14px; } +.card strong { display: block; font-size: 22px; } +.card span { color: #607486; font-size: 12px; } +.badge { border-radius: 99px; display: inline-block; font-size: 12px; + font-weight: 700; padding: 4px 9px; text-transform: uppercase; } +.badge.pass { background: #eaf5ec; color: var(--green); } +.badge.warn { background: #fff4e5; color: var(--amber); } +.notice { border-left: 5px solid var(--blue); margin: 9px 0; + padding: 11px 14px; background: #eaf4f8; } +.notice.warning { border-color: var(--amber); background: #fff4e5; } +.table-wrap { overflow-x: auto; } +table { width: 100%; border-collapse: collapse; font-size: 13px; } +th { background: var(--ink); color: white; text-align: left; } +th, td { border: 1px solid var(--line); padding: 8px 9px; white-space: nowrap; } +tbody tr:nth-child(even) { background: var(--soft); } +footer { color: #607486; font-size: 12px; text-align: center; } +@media (max-width: 760px) { .cards { grid-template-columns: 1fr 1fr; } } +@media print { body { background: white; } main { margin: 0; max-width: none; } + header, section { border: 0; break-inside: avoid; padding: 12px 0; } + .table-wrap { overflow: visible; } } +""" + + body = f""" + + + + +{escape(title)} + + +
+
+
vLLM recipe sweep report
+

{escape(title)}

+
{_text(meta["model"])} | {_text(meta["input_tokens"])} + input / {_text(meta["output_tokens"])} output tokens | + request rate {_text(meta["request_rate"])}
+
+
{_text(recommended_concurrency)} + recommended concurrency
+
{_fmt(measured.get("mean_output_throughput"))} + output tokens/s
+
{_fmt_int(measured.get("mean_p99_ttft_ms"))} ms + mean p99 TTFT
+
{_fmt(measured.get("mean_p99_tpot_ms"))} ms + mean p99 TPOT
+
{_fmt_int(ttft_sla_ms)} ms + TTFT SLA
+
{_fmt(tpot_sla_ms)} ms + TPOT SLA
+
{ + _fmt_percent( + minimum_compliance * 100 if minimum_compliance is not None else None + ) + } + required combined compliance
+
+
+
+

Recommended configuration

+ { + _table( + ["Setting", "Value"], + _selected_settings( + reports["parallel-layout"], + reports["concurrency-tuning"], + reports["runtime-tuning"], + ), + ) + } +
+
+

Stage summary

+ { + _table( + [ + "Stage", + "Status", + "Output tok/s", + "Goodput req/s", + "p99 TTFT ms", + "p99 TPOT ms", + ], + _stage_summary(reports), + ) + } +

Automatic observations

+ {_observation_html(observations)} +
+
+

Parallel-layout candidates { + _status_badge((reports["parallel-layout"] or {}).get("status")) + }

+ { + _table( + [ + "TP", + "DP", + "Max seqs", + "Batch tokens", + "Output tok/s", + "p99 TTFT ms", + "p99 TPOT ms", + "Compliance", + "SLA", + "SLA reason", + ], + _parallel_rows( + reports["parallel-layout"], + ttft_sla_ms=ttft_sla_ms, + tpot_sla_ms=tpot_sla_ms, + minimum_compliance=minimum_compliance, + ), + ) + } +
+
+

Concurrency candidates { + _status_badge((reports["concurrency-tuning"] or {}).get("status")) + }

+ { + _table( + [ + "Concurrency", + "Output tok/s", + "Req/s", + "Goodput req/s", + "Compliance", + "p99 TTFT ms", + "p99 TPOT ms", + "SLA", + "SLA reason", + ], + _concurrency_rows( + reports["concurrency-tuning"], + ttft_sla_ms=ttft_sla_ms, + tpot_sla_ms=tpot_sla_ms, + minimum_compliance=minimum_compliance, + ), + ) + } +
+
+

Scheduler candidates { + _status_badge((reports["runtime-tuning"] or {}).get("status")) + }

+ { + _table( + [ + "Max seqs", + "Batch tokens", + "Output tok/s", + "Goodput req/s", + "p99 TTFT ms", + "p99 TPOT ms", + "Compliance", + "SLA", + "SLA reason", + ], + _scheduler_rows( + reports["runtime-tuning"], + ttft_sla_ms=ttft_sla_ms, + tpot_sla_ms=tpot_sla_ms, + minimum_compliance=minimum_compliance, + ), + ) + } +
+
Generated {escape(generated)} from completed recommendation and +summary JSON files. Warmup results are excluded by the recommenders.
+
+ +""" + output.parent.mkdir(parents=True, exist_ok=True) + output.write_text(body, encoding="utf-8") + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser( + description="Generate a self-contained HTML report from recipe sweeps." + ) + parser.add_argument( + "--sweep-dir", + default=None, + help="Sweep package directory (default: beside this script).", + ) + parser.add_argument( + "--output", + default=None, + help="Output HTML path (default: /sweep-report.html).", + ) + parser.add_argument( + "--title", + default="vLLM Recipe Sweep", + help="Report title.", + ) + parser.add_argument( + "--ttft-sla-ms", + type=float, + default=DEFAULT_TTFT_SLA_MS, + help="Expected TTFT objective; must match every recommendation file.", + ) + parser.add_argument( + "--tpot-sla-ms", + type=float, + default=DEFAULT_TPOT_SLA_MS, + help="Expected TPOT objective; must match every recommendation file.", + ) + return parser.parse_args() + + +def main() -> int: + args = parse_args() + for name in ("ttft_sla_ms", "tpot_sla_ms"): + value = getattr(args, name) + if value is not None and value <= 0: + option = "--" + name.replace("_", "-") + raise ValueError(f"{option} must be greater than zero.") + + script_dir = Path(__file__).absolute().parent + root = Path(args.sweep_dir).resolve() if args.sweep_dir else script_dir + output = Path(args.output).resolve() if args.output else root / "sweep-report.html" + generate_report( + root, + output, + args.title, + ttft_sla_ms=args.ttft_sla_ms, + tpot_sla_ms=args.tpot_sla_ms, + ) + print(f"Wrote HTML sweep report: {output}") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tools/recipes/sweep_recommendation.py b/tools/recipes/sweep/sweep_recommendation.py similarity index 62% rename from tools/recipes/sweep_recommendation.py rename to tools/recipes/sweep/sweep_recommendation.py index 70efaad625..1eebf754c3 100755 --- a/tools/recipes/sweep_recommendation.py +++ b/tools/recipes/sweep/sweep_recommendation.py @@ -2,7 +2,7 @@ # SPDX-License-Identifier: Apache-2.0 # SPDX-FileCopyrightText: Copyright contributors to the vLLM project -"""Select one measured scheduler recommendation from vLLM sweep results.""" +"""Select one measured parallel-layout or scheduler sweep recommendation.""" from __future__ import annotations @@ -24,6 +24,10 @@ DEFAULT_ENV_PATH: str | None = None DEFAULT_TTFT_SLA_MS: float | None = None DEFAULT_TPOT_SLA_MS: float | None = None DEFAULT_MINIMUM_COMPLIANCE: float = 0.99 +DEFAULT_THROUGHPUT_EQUIVALENCE_PERCENT: float = 1.0 +DEFAULT_RESULTS_DIR = "results/runtime-tuning" +DEFAULT_OUTPUT_CONFIG = "recommended-config.yml" +DEFAULT_OUTPUT_JSON = "recommendation.json" def _number(value: object) -> float | None: @@ -48,11 +52,75 @@ def _percentile_summary( return mean(numbers), median(numbers), max(numbers) -def _positive_int(row: dict[str, Any], key: str) -> int | None: - value = row.get(key) +def _scheduler_value(row: dict[str, Any], key: str) -> tuple[bool, int | None]: + if key not in row: + # Default-reference sweep candidates omit scheduler keys entirely. + return True, None + value = row[key] + if value is None: + return True, None if isinstance(value, bool) or not isinstance(value, int) or value <= 0: - return None - return value + return False, None + return True, value + + +def _parallel_value(row: dict[str, Any], key: str, default: int) -> tuple[bool, int]: + if key not in row: + return True, default + value = row[key] + if isinstance(value, bool) or not isinstance(value, int) or value <= 0: + return False, default + return True, value + + +def _scheduler_sort_value(value: int | None) -> int: + return -1 if value is None else value + + +def _scheduler_preference_key(value: object) -> tuple[int, int]: + """Prefer vLLM default, then the smaller explicit value on exact ties.""" + if value is None: + return (1, 0) + if isinstance(value, int) and not isinstance(value, bool): + return (0, -value) + return (0, 0) + + +def _has_fixed_parallel_layout(candidates: list[dict[str, Any]]) -> bool: + """Return whether candidates belong to one scheduler-only comparison.""" + layouts = { + (candidate["tensor_parallel_size"], candidate["data_parallel_size"]) + for candidate in candidates + } + return len(layouts) == 1 + + +def _select_near_equivalent_scheduler_candidate( + candidates: list[dict[str, Any]], + *, + throughput_equivalence_percent: float, +) -> dict[str, Any]: + """Prefer fewer scheduler overrides among near-equal configurations.""" + best_throughput = max( + candidate["mean_output_throughput"] for candidate in candidates + ) + throughput_floor = best_throughput * (1.0 - throughput_equivalence_percent / 100.0) + near_equivalent = [ + candidate + for candidate in candidates + if candidate["mean_output_throughput"] >= throughput_floor + ] + return max( + near_equivalent, + key=lambda candidate: ( + len(candidate["vllm_default_parameters"]), + candidate["combined_compliance_ratio"] or 0.0, + candidate["mean_request_goodput"] or 0.0, + candidate["mean_output_throughput"], + _scheduler_preference_key(candidate["max_num_batched_tokens"]), + _scheduler_preference_key(candidate["max_num_seqs"]), + ), + ) def _resolve(script_dir: Path, value: str | None) -> Path | None: @@ -80,21 +148,40 @@ def _aggregate_candidates( rows: list[dict[str, Any]], *, use_goodput: bool, + default_tensor_parallel_size: int = 1, + default_data_parallel_size: int = 1, ttft_sla_ms: float | None = None, tpot_sla_ms: float | None = None, minimum_compliance: float = DEFAULT_MINIMUM_COMPLIANCE, ) -> list[dict[str, Any]]: - grouped: dict[tuple[int, int], list[dict[str, Any]]] = defaultdict(list) + grouped: dict[tuple[int, int, int | None, int | None], list[dict[str, Any]]] = ( + defaultdict(list) + ) for row in rows: - seqs = _positive_int(row, "max_num_seqs") - batch = _positive_int(row, "max_num_batched_tokens") - if seqs is None or batch is None: + seqs_valid, seqs = _scheduler_value(row, "max_num_seqs") + batch_valid, batch = _scheduler_value(row, "max_num_batched_tokens") + tp_valid, tp = _parallel_value( + row, "tensor_parallel_size", default_tensor_parallel_size + ) + dp_valid, dp = _parallel_value( + row, "data_parallel_size", default_data_parallel_size + ) + if not seqs_valid or not batch_valid or not tp_valid or not dp_valid: continue - grouped[(seqs, batch)].append(row) + grouped[(tp, dp, seqs, batch)].append(row) candidates: list[dict[str, Any]] = [] - for (seqs, batch), runs in sorted(grouped.items()): + sorted_groups = sorted( + grouped.items(), + key=lambda item: ( + item[0][0], + item[0][1], + _scheduler_sort_value(item[0][2]), + _scheduler_sort_value(item[0][3]), + ), + ) + for (tp, dp, seqs, batch), runs in sorted_groups: failed_requests = sum(int(_number(run.get("failed")) or 0) for run in runs) output_throughput = _mean(runs, "output_throughput") request_throughput = _mean(runs, "request_throughput") @@ -150,9 +237,19 @@ def _aggregate_candidates( candidates.append( { + "tensor_parallel_size": tp, + "data_parallel_size": dp, "max_num_seqs": seqs, "max_num_batched_tokens": batch, "run_count": len(runs), + "vllm_default_parameters": [ + name + for name, value in ( + ("max_num_seqs", seqs), + ("max_num_batched_tokens", batch), + ) + if value is None + ], "failed_requests": failed_requests, "mean_request_goodput": request_goodput, "mean_request_throughput": request_throughput, @@ -188,6 +285,7 @@ def _select_candidate( candidates: list[dict[str, Any]], *, use_goodput: bool, + throughput_equivalence_percent: float = (DEFAULT_THROUGHPUT_EQUIVALENCE_PERCENT), ) -> tuple[dict[str, Any] | None, dict[str, Any], str]: valid = [candidate for candidate in candidates if candidate["valid"]] if not valid: @@ -204,8 +302,10 @@ def _select_candidate( candidate["mean_request_goodput"], candidate["combined_compliance_ratio"] or 0.0, candidate["mean_output_throughput"], - -candidate["max_num_batched_tokens"], - -candidate["max_num_seqs"], + -candidate["data_parallel_size"], + -candidate["tensor_parallel_size"], + _scheduler_preference_key(candidate["max_num_batched_tokens"]), + _scheduler_preference_key(candidate["max_num_seqs"]), ), ) if best_effort["mean_request_goodput"] <= 0: @@ -214,30 +314,46 @@ def _select_candidate( "the supplied TTFT/TPOT objectives." ) eligible = [candidate for candidate in valid if candidate["sla_eligible"]] - winner = ( - max( + if eligible and _has_fixed_parallel_layout(valid): + winner = _select_near_equivalent_scheduler_candidate( eligible, - key=lambda candidate: ( - candidate["mean_output_throughput"], - candidate["combined_compliance_ratio"], - candidate["mean_request_goodput"], - -candidate["max_num_batched_tokens"], - -candidate["max_num_seqs"], - ), + throughput_equivalence_percent=throughput_equivalence_percent, + ) + else: + winner = ( + max( + eligible, + key=lambda candidate: ( + candidate["mean_output_throughput"], + candidate["combined_compliance_ratio"], + candidate["mean_request_goodput"], + -candidate["data_parallel_size"], + -candidate["tensor_parallel_size"], + _scheduler_preference_key(candidate["max_num_batched_tokens"]), + _scheduler_preference_key(candidate["max_num_seqs"]), + ), + ) + if eligible + else None ) - if eligible - else None - ) else: objective = "highest_mean_output_throughput" - winner = max( - valid, - key=lambda candidate: ( - candidate["mean_output_throughput"], - -candidate["max_num_batched_tokens"], - -candidate["max_num_seqs"], - ), - ) + if _has_fixed_parallel_layout(valid): + winner = _select_near_equivalent_scheduler_candidate( + valid, + throughput_equivalence_percent=throughput_equivalence_percent, + ) + else: + winner = max( + valid, + key=lambda candidate: ( + candidate["mean_output_throughput"], + -candidate["data_parallel_size"], + -candidate["tensor_parallel_size"], + _scheduler_preference_key(candidate["max_num_batched_tokens"]), + _scheduler_preference_key(candidate["max_num_seqs"]), + ), + ) best_effort = winner return winner, best_effort, objective @@ -250,6 +366,31 @@ def _load_config(path: Path) -> dict[str, Any]: return data +def _build_recommended_config( + initial_config: dict[str, Any], + winner: dict[str, Any], +) -> dict[str, Any]: + recommended_config = dict(initial_config) + recommended_config["tensor-parallel-size"] = winner["tensor_parallel_size"] + recommended_config["data-parallel-size"] = winner["data_parallel_size"] + for config_key, result_key in ( + ("max-num-seqs", "max_num_seqs"), + ("max-num-batched-tokens", "max_num_batched_tokens"), + ): + value = winner[result_key] + if value is None: + recommended_config.pop(config_key, None) + else: + recommended_config[config_key] = value + return recommended_config + + +def _format_scheduler_value(value: object) -> str: + if value is None: + return "vLLM default" + return str(value) + + def _write_config( path: Path, *, @@ -274,13 +415,13 @@ def _write_config( def parse_args() -> argparse.Namespace: parser = argparse.ArgumentParser( description=( - "Select one recommended max-num-seqs/max-num-batched-tokens pair " - "from generated vLLM sweep results." + "Select one recommended TP/DP layout and scheduler setting from " + "generated vLLM sweep results." ) ) parser.add_argument( "--results-dir", - default="results/runtime-tuning", + default=DEFAULT_RESULTS_DIR, help="Sweep experiment directory (default: results/runtime-tuning).", ) parser.add_argument( @@ -314,14 +455,24 @@ def parse_args() -> argparse.Namespace: "meet all supplied objectives (default: 0.99)." ), ) + parser.add_argument( + "--throughput-equivalence-percent", + type=float, + default=DEFAULT_THROUGHPUT_EQUIVALENCE_PERCENT, + help=( + "For a scheduler-only comparison, prefer fewer explicit scheduler " + "overrides when throughput is within this percentage of the best " + "eligible result (default: 1.0)." + ), + ) parser.add_argument( "--output-config", - default="recommended-config.yml", + default=DEFAULT_OUTPUT_CONFIG, help="Recommended config output path.", ) parser.add_argument( "--output-json", - default="recommendation.json", + default=DEFAULT_OUTPUT_JSON, help="Recommendation evidence output path.", ) return parser.parse_args() @@ -331,6 +482,10 @@ def main() -> int: args = parse_args() if not 0.0 < args.minimum_compliance <= 1.0: raise ValueError("--minimum-compliance must be greater than 0 and at most 1.") + if not 0.0 <= args.throughput_equivalence_percent < 100.0: + raise ValueError( + "--throughput-equivalence-percent must be at least 0 and less than 100." + ) script_dir = Path(__file__).resolve().parent config_path = _resolve(script_dir, args.config) @@ -350,6 +505,7 @@ def main() -> int: f"Sweep results were not found at {results_dir}. Run the sweep first." ) + initial_config = _load_config(config_path) rows = _load_runs(results_dir) if not rows: raise ValueError(f"No summary.json sweep results found under {results_dir}.") @@ -358,19 +514,20 @@ def main() -> int: candidates = _aggregate_candidates( rows, use_goodput=use_goodput, + default_tensor_parallel_size=int(initial_config.get("tensor-parallel-size", 1)), + default_data_parallel_size=int(initial_config.get("data-parallel-size", 1)), ttft_sla_ms=args.ttft_sla_ms, tpot_sla_ms=args.tpot_sla_ms, minimum_compliance=args.minimum_compliance, ) winner, best_effort, objective = _select_candidate( - candidates, use_goodput=use_goodput + candidates, + use_goodput=use_goodput, + throughput_equivalence_percent=args.throughput_equivalence_percent, ) - initial_config = _load_config(config_path) if winner is not None: - recommended_config = dict(initial_config) - recommended_config["max-num-seqs"] = winner["max_num_seqs"] - recommended_config["max-num-batched-tokens"] = winner["max_num_batched_tokens"] + recommended_config = _build_recommended_config(initial_config, winner) _write_config( output_config, source_path=config_path, @@ -384,8 +541,11 @@ def main() -> int: recommended = None if winner is not None: recommended = { + "tensor_parallel_size": winner["tensor_parallel_size"], + "data_parallel_size": winner["data_parallel_size"], "max_num_seqs": winner["max_num_seqs"], "max_num_batched_tokens": winner["max_num_batched_tokens"], + "vllm_default_parameters": winner["vllm_default_parameters"], } measured = { key: winner[key] @@ -413,19 +573,25 @@ def main() -> int: ), "selection_objective": objective, "minimum_compliance_ratio": args.minimum_compliance, + "throughput_equivalence_percent": args.throughput_equivalence_percent, "slo": { "ttft_ms": args.ttft_sla_ms, "tpot_ms": args.tpot_sla_ms, }, "initial": { + "tensor_parallel_size": initial_config.get("tensor-parallel-size"), + "data_parallel_size": initial_config.get("data-parallel-size", 1), "max_num_seqs": initial_config.get("max-num-seqs"), "max_num_batched_tokens": initial_config.get("max-num-batched-tokens"), }, "recommended": recommended, "measured": measured, "best_effort": { + "tensor_parallel_size": best_effort["tensor_parallel_size"], + "data_parallel_size": best_effort["data_parallel_size"], "max_num_seqs": best_effort["max_num_seqs"], "max_num_batched_tokens": best_effort["max_num_batched_tokens"], + "vllm_default_parameters": best_effort["vllm_default_parameters"], "mean_request_goodput": best_effort["mean_request_goodput"], "combined_compliance_ratio": best_effort["combined_compliance_ratio"], "p99_sla_eligible": best_effort["p99_sla_eligible"], @@ -445,8 +611,15 @@ def main() -> int: print("Recommended runtime configuration") print() - print(f" max-num-seqs: {winner['max_num_seqs']}") - print(f" max-num-batched-tokens: {winner['max_num_batched_tokens']}") + print(f" tensor-parallel-size: {winner['tensor_parallel_size']}") + print(f" data-parallel-size: {winner['data_parallel_size']}") + print( + " max-num-seqs: " + _format_scheduler_value(winner["max_num_seqs"]) + ) + print( + " max-num-batched-tokens: " + + _format_scheduler_value(winner["max_num_batched_tokens"]) + ) print() print(f"Selection objective: {objective}") if use_goodput: diff --git a/tools/recipes/sweep/sweep_visualization.py b/tools/recipes/sweep/sweep_visualization.py new file mode 100644 index 0000000000..4681ae5f7f --- /dev/null +++ b/tools/recipes/sweep/sweep_visualization.py @@ -0,0 +1,226 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +"""Generate post-benchmark figures from completed recipe sweep results.""" + +from __future__ import annotations + +import argparse +import json +import os +import shutil +import subprocess +import tempfile +from dataclasses import dataclass +from pathlib import Path +from typing import Any + + +@dataclass(frozen=True) +class PlotSpec: + var_x: str + var_y: str + fig_name: str + + +STAGE_PLOTS: dict[str, tuple[PlotSpec, ...]] = { + "parallel-layout": ( + PlotSpec("recipe_candidate", "output_throughput", "parallel_throughput"), + PlotSpec("recipe_candidate", "p99_ttft_ms", "parallel_p99_ttft"), + PlotSpec("recipe_candidate", "p99_tpot_ms", "parallel_p99_tpot"), + ), + "concurrency-tuning": ( + PlotSpec("max_concurrency", "output_throughput", "concurrency_throughput"), + PlotSpec("max_concurrency", "p99_ttft_ms", "concurrency_p99_ttft"), + PlotSpec("max_concurrency", "p99_tpot_ms", "concurrency_p99_tpot"), + PlotSpec("output_throughput", "p99_ttft_ms", "throughput_p99_ttft"), + ), + "runtime-tuning": ( + PlotSpec("recipe_candidate", "output_throughput", "scheduler_throughput"), + PlotSpec("recipe_candidate", "p99_ttft_ms", "scheduler_p99_ttft"), + PlotSpec("recipe_candidate", "p99_tpot_ms", "scheduler_p99_tpot"), + ), +} + + +def _serve_candidate(summary_path: Path, record: dict[str, Any]) -> str: + directory_name = summary_path.parent.name + prefix = "SERVE--" + separator = "-BENCH--" + if directory_name.startswith(prefix) and separator in directory_name: + return directory_name[len(prefix) :].split(separator, 1)[0] + + benchmark_name = record.get("_benchmark_name") + if isinstance(benchmark_name, str) and benchmark_name: + return benchmark_name + return "fixed_configuration" + + +def _load_stage_records(stage_dir: Path) -> list[dict[str, Any]]: + records: list[dict[str, Any]] = [] + for summary_path in sorted(stage_dir.rglob("summary.json")): + data = json.loads(summary_path.read_text(encoding="utf-8")) + if not isinstance(data, list): + continue + for record in data: + if not isinstance(record, dict): + continue + normalized = dict(record) + normalized["recipe_candidate"] = _serve_candidate( + summary_path, + normalized, + ) + records.append(normalized) + return records + + +def _has_numeric_metric(records: list[dict[str, Any]], key: str) -> bool: + return any( + isinstance(record.get(key), (int, float)) + and not isinstance(record.get(key), bool) + for record in records + ) + + +def _run_plot( + normalized_dir: Path, + figure_dir: Path, + spec: PlotSpec, + *, + dry_run: bool, +) -> None: + command = [ + "vllm", + "bench", + "sweep", + "plot", + str(normalized_dir), + "--fig-dir", + str(figure_dir.resolve()), + "--var-x", + spec.var_x, + "--var-y", + spec.var_y, + "--fig-name", + spec.fig_name, + ] + if dry_run: + command.append("--dry-run") + + environment = dict(os.environ) + environment.setdefault("MPLBACKEND", "Agg") + subprocess.run(command, check=True, env=environment) + + +def generate_figures( + results_dir: Path, + *, + selected_stages: list[str] | None = None, + dry_run: bool = False, +) -> list[Path]: + if shutil.which("vllm") is None: + raise RuntimeError( + "The vllm executable was not found. Run this analysis inside the " + "vLLM benchmark environment." + ) + + stages = selected_stages or list(STAGE_PLOTS) + discovered: list[Path] = [] + with tempfile.TemporaryDirectory(prefix="vllm-recipe-plot-") as temp_dir: + temporary_root = Path(temp_dir) + for stage in stages: + stage_dir = results_dir / stage + if not stage_dir.is_dir(): + continue + + records = _load_stage_records(stage_dir) + if not records: + continue + discovered.append(stage_dir) + + normalized_dir = temporary_root / stage + normalized_dir.mkdir(parents=True, exist_ok=True) + (normalized_dir / "summary.json").write_text( + json.dumps(records, indent=2) + "\n", + encoding="utf-8", + ) + + figure_dir = stage_dir / "figures" + plots = list(STAGE_PLOTS[stage]) + if stage == "concurrency-tuning" and _has_numeric_metric( + records, "request_goodput" + ): + plots.append( + PlotSpec( + "max_concurrency", + "request_goodput", + "concurrency_goodput", + ) + ) + + for spec in plots: + _run_plot( + normalized_dir, + figure_dir, + spec, + dry_run=dry_run, + ) + + if not discovered: + requested = ", ".join(stages) + raise ValueError( + f"No completed sweep summaries found under {results_dir} " + f"for stages: {requested}." + ) + return discovered + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser( + description="Plot completed recipe sweep results without running a sweep." + ) + parser.add_argument( + "--results-dir", + default=None, + help="Results root (default: results beside this script).", + ) + parser.add_argument( + "--stage", + action="append", + choices=tuple(STAGE_PLOTS), + help="Plot only this stage; repeat to select multiple stages.", + ) + parser.add_argument( + "--dry-run", + action="store_true", + help="Preview vLLM plot operations without writing figures.", + ) + return parser.parse_args() + + +def main() -> int: + args = parse_args() + script_dir = Path(__file__).resolve().parent + results_dir = ( + Path(args.results_dir).resolve() + if args.results_dir is not None + else script_dir / "results" + ) + stages = generate_figures( + results_dir, + selected_stages=args.stage, + dry_run=args.dry_run, + ) + print() + if args.dry_run: + print("Visualization dry run completed for:") + else: + print("Wrote post-benchmark figures under:") + for stage_dir in stages: + print(f" {stage_dir / 'figures'}") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tools/recipes/sweep_generation.py b/tools/recipes/sweep_generation.py deleted file mode 100644 index 33a5244a72..0000000000 --- a/tools/recipes/sweep_generation.py +++ /dev/null @@ -1,358 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# SPDX-FileCopyrightText: Copyright contributors to the vLLM project - -"""Generate optional vLLM benchmark sweep files from one initial suggestion.""" - -from __future__ import annotations - -import json -import os -import shlex -from pathlib import Path -from typing import Any - -from runtime_tuning import WorkloadHints - - -def _positive_int(config: dict[str, Any], key: str) -> int: - value = config.get(key) - if not isinstance(value, int) or isinstance(value, bool) or value <= 0: - raise ValueError(f"Sweep generation requires a positive initial {key!r} value.") - return value - - -def _strict_lower_power_of_two(value: int) -> int: - if value <= 1: - return 1 - return 1 << ((value - 1).bit_length() - 1) - - -def _strict_upper_power_of_two(value: int) -> int: - return 1 << value.bit_length() - - -def validate_sweep_workload(workload: WorkloadHints) -> None: - required = { - "--input-tokens": workload.input_tokens, - "--output-tokens": workload.output_tokens, - "--concurrency": workload.concurrency, - } - missing = [name for name, value in required.items() if value is None] - if missing: - raise ValueError("--generate-sweep requires " + ", ".join(missing) + ".") - - -def build_serve_params(config: dict[str, Any]) -> list[dict[str, Any]]: - """Build a diverse, bounded sweep around the initial suggestion. - - The initial five-point sweep changed only one parameter at a time and - skipped the useful middle sequence count. Eight directed points cover the - batch-budget curve at full concurrency plus interactions at 3/4 and 1/2 of - the initial scheduler concurrency without paying for a full Cartesian grid. - """ - initial_seqs = _positive_int(config, "max-num-seqs") - initial_batch = _positive_int(config, "max-num-batched-tokens") - - minimum_batch = initial_seqs - if config.get("enable-chunked-prefill") is False: - max_model_len = config.get("max-model-len") - if ( - isinstance(max_model_len, int) - and not isinstance(max_model_len, bool) - and max_model_len > 0 - ): - minimum_batch = max(minimum_batch, max_model_len) - - lower_seqs = max(1, (initial_seqs + 1) // 2) - middle_seqs = max(lower_seqs, (3 * initial_seqs + 3) // 4) - lower_batch = max( - minimum_batch, - _strict_lower_power_of_two(initial_batch), - ) - smaller_batch = max( - minimum_batch, - _strict_lower_power_of_two(lower_batch), - ) - higher_batch = max( - minimum_batch, - _strict_upper_power_of_two(initial_batch), - ) - - candidates: list[dict[str, Any]] = [] - seen: set[tuple[int, int]] = set() - - def add(name: str, max_num_seqs: int, max_num_batched_tokens: int) -> None: - max_num_batched_tokens = max( - max_num_batched_tokens, - max_num_seqs, - ) - signature = (max_num_seqs, max_num_batched_tokens) - if signature in seen: - return - seen.add(signature) - candidates.append( - { - "_benchmark_name": name, - "max_num_seqs": max_num_seqs, - "max_num_batched_tokens": max_num_batched_tokens, - } - ) - - # Keep the exact initial suggestion as the measured baseline. Additional - # values exist only in the optional sweep package. - add("initial", initial_seqs, initial_batch) - add("smaller_batch_budget", initial_seqs, smaller_batch) - add("lower_batch_budget", initial_seqs, lower_batch) - add("higher_batch_budget", initial_seqs, higher_batch) - add("middle_seqs_lower_batch", middle_seqs, lower_batch) - add("middle_seqs_higher_batch", middle_seqs, higher_batch) - add("lower_seqs_lower_batch", lower_seqs, lower_batch) - add("lower_seqs_higher_batch", lower_seqs, higher_batch) - - return candidates - - -def build_bench_params(workload: WorkloadHints) -> list[dict[str, Any]]: - validate_sweep_workload(workload) - assert workload.input_tokens is not None - assert workload.output_tokens is not None - assert workload.concurrency is not None - - return [ - { - "_benchmark_name": "user_workload", - "random_input_len": workload.input_tokens, - "random_output_len": workload.output_tokens, - "max_concurrency": workload.concurrency, - } - ] - - -def _benchmark_models(config: dict[str, Any]) -> tuple[str, str]: - model = config.get("model") - if not isinstance(model, str) or not model: - raise ValueError("Sweep generation requires a model in config.yml.") - - served_model_name = config.get("served-model-name") - if isinstance(served_model_name, str) and served_model_name: - return served_model_name, model - if ( - isinstance(served_model_name, list) - and served_model_name - and isinstance(served_model_name[0], str) - ): - return served_model_name[0], model - - return model, model - - -def _relative_to(directory: Path, target: str) -> str: - return os.path.relpath(Path(target).resolve(), directory.resolve()) - - -def _write_json(path: Path, value: object) -> None: - path.write_text( - json.dumps(value, indent=2) + "\n", - encoding="utf-8", - ) - - -def _write_run_script( - path: Path, - *, - config_rel: str, - env_rel: str, - request_model: str, - tokenizer: str, - workload: WorkloadHints, -) -> None: - bench_parts = [ - "vllm bench serve", - "--backend vllm", - f"--model {shlex.quote(request_model)}", - f"--tokenizer {shlex.quote(tokenizer)}", - "--dataset-name random", - "--request-rate inf", - "--ignore-eos", - "--metric-percentiles 99", - ] - - goodput_pairs: list[str] = [] - if workload.ttft_sla_ms is not None: - goodput_pairs.append(f"ttft:{workload.ttft_sla_ms:g}") - if workload.tpot_sla_ms is not None: - goodput_pairs.append(f"tpot:{workload.tpot_sla_ms:g}") - if goodput_pairs: - bench_parts.append( - "--goodput " + " ".join(shlex.quote(pair) for pair in goodput_pairs) - ) - - bench_cmd = " ".join(bench_parts) - script = f"""#!/usr/bin/env bash -set -euo pipefail - -SCRIPT_DIR="$(cd -- "$(dirname -- "${{BASH_SOURCE[0]}}")" && pwd)" -CONFIG_PATH="${{SCRIPT_DIR}}/{config_rel}" -ENV_PATH="${{SCRIPT_DIR}}/{env_rel}" - -source "${{ENV_PATH}}" - -vllm bench sweep serve \ - --serve-cmd "vllm serve --config '${{CONFIG_PATH}}'" \ - --bench-cmd "{bench_cmd}" \ - --serve-params "${{SCRIPT_DIR}}/serve_params.json" \ - --bench-params "${{SCRIPT_DIR}}/bench_params.json" \ - --output-dir "${{SCRIPT_DIR}}/results" \ - --experiment-name runtime-tuning \ - "$@" -""" - path.write_text(script, encoding="utf-8") - path.chmod(path.stat().st_mode | 0o111) - - -def _write_recommend_script( - path: Path, - *, - config_rel: str, - env_rel: str, - workload: WorkloadHints, -) -> None: - template_path = Path(__file__).with_name("sweep_recommendation.py") - source = template_path.read_text(encoding="utf-8") - - replacements = { - "DEFAULT_CONFIG_PATH: str | None = None": ( - f"DEFAULT_CONFIG_PATH: str | None = {config_rel!r}" - ), - "DEFAULT_ENV_PATH: str | None = None": ( - f"DEFAULT_ENV_PATH: str | None = {env_rel!r}" - ), - "DEFAULT_TTFT_SLA_MS: float | None = None": ( - f"DEFAULT_TTFT_SLA_MS: float | None = {workload.ttft_sla_ms!r}" - ), - "DEFAULT_TPOT_SLA_MS: float | None = None": ( - f"DEFAULT_TPOT_SLA_MS: float | None = {workload.tpot_sla_ms!r}" - ), - } - for marker, replacement in replacements.items(): - if marker not in source: - raise ValueError(f"Recommender template marker not found: {marker}") - source = source.replace(marker, replacement, 1) - - path.write_text(source, encoding="utf-8") - path.chmod(path.stat().st_mode | 0o111) - - -def _write_guide(path: Path, workload: WorkloadHints) -> None: - sla_lines: list[str] = [] - if workload.ttft_sla_ms is not None: - sla_lines.append(f"- TTFT objective: `{workload.ttft_sla_ms:g} ms`") - if workload.tpot_sla_ms is not None: - sla_lines.append(f"- TPOT objective: `{workload.tpot_sla_ms:g} ms`") - - sla_text = "\n".join(sla_lines) - if sla_text: - sla_text = ( - "\n## Supplied latency objectives\n\n" - + sla_text - + "\n\nThe generated benchmark uses these values with vLLM " - "`--goodput`.\n" - ) - - content = f"""# Optional Runtime Tuning Sweep - -The generated `config.yml` is the **single initial suggestion** and can be -deployed directly. The sweep benchmarks nearby values for: - -- `max-num-seqs` -- `max-num-batched-tokens` - -## Run - -```bash -./run_sweep.sh --dry-run -./run_sweep.sh -./recommend.py -``` - -For a quick one-run experiment: - -```bash -rm -rf results/runtime-tuning -./run_sweep.sh --num-runs 1 -./recommend.py -``` - -Resume an interrupted sweep: - -```bash -./run_sweep.sh --resume -``` - -By default, vLLM benchmarks each parameter combination three times. -{sla_text} -## Outputs - -`recommend.py` writes: - -```text -recommended-config.yml -recommendation.json -``` - -With TTFT/TPOT objectives it requires duration-weighted combined compliance of -at least 99% plus median P99 TTFT/TPOT compliance, then selects highest mean -output-token throughput. If no candidate qualifies, it records a best-effort -candidate but does not write `recommended-config.yml`. Without latency -objectives it selects highest mean output-token throughput. Configurations with -failed requests are excluded. - -`recommended-config.yml` copies the initial configuration and changes only -`max-num-seqs` and `max-num-batched-tokens`. -""" - path.write_text(content, encoding="utf-8") - - -def write_sweep_files( - output_dir: str, - *, - config_path: str, - env_path: str, - config: dict[str, Any], - workload: WorkloadHints, -) -> list[Path]: - """Write an optional sweep package around the initial config suggestion.""" - validate_sweep_workload(workload) - - directory = Path(output_dir) - directory.mkdir(parents=True, exist_ok=True) - - serve_params = directory / "serve_params.json" - bench_params = directory / "bench_params.json" - run_script = directory / "run_sweep.sh" - recommend_script = directory / "recommend.py" - guide = directory / "SWEEP.md" - - config_rel = _relative_to(directory, config_path) - env_rel = _relative_to(directory, env_path) - - _write_json(serve_params, build_serve_params(config)) - _write_json(bench_params, build_bench_params(workload)) - request_model, tokenizer = _benchmark_models(config) - _write_run_script( - run_script, - config_rel=config_rel, - env_rel=env_rel, - request_model=request_model, - tokenizer=tokenizer, - workload=workload, - ) - _write_recommend_script( - recommend_script, - config_rel=config_rel, - env_rel=env_rel, - workload=workload, - ) - _write_guide(guide, workload) - - return [serve_params, bench_params, run_script, recommend_script, guide] diff --git a/vllm/_aiter_ops.py b/vllm/_aiter_ops.py index ca98e63b39..0ed8490618 100644 --- a/vllm/_aiter_ops.py +++ b/vllm/_aiter_ops.py @@ -1167,7 +1167,8 @@ def _rocm_aiter_fused_allreduce_rmsnorm_quant_per_group_impl( w=weight, eps=epsilon, group_size=group_size, - registered=torch.cuda.is_current_stream_capturing(), + registered=torch.cuda.is_current_stream_capturing() + and ca.enable_register_for_capturing, use_1stage=use_1stage, ) assert result is not None @@ -1219,7 +1220,8 @@ def _rocm_aiter_fused_allreduce_rmsnorm_quant_per_group_with_bf16_norm_impl( w=weight, eps=epsilon, group_size=group_size, - registered=torch.cuda.is_current_stream_capturing(), + registered=torch.cuda.is_current_stream_capturing() + and ca.enable_register_for_capturing, use_1stage=use_1stage, emit_bf16=True, ) @@ -1997,6 +1999,8 @@ class rocm_aiter_ops: # Check if the env variable is set _AITER_ENABLED = envs.VLLM_ROCM_USE_AITER + # Chunk length the AITER FlyDSL K5 prefill kernels are compiled for. + GDN_FLYDSL_CHUNK_SIZE = 64 _CUSTOM_ALL_REDUCE_ENABLED = envs.VLLM_ROCM_USE_AITER_CUSTOM_AR _LINEAR_ENABLED = envs.VLLM_ROCM_USE_AITER_LINEAR _FMOE_ENABLED = envs.VLLM_ROCM_USE_AITER_MOE @@ -2467,6 +2471,23 @@ class rocm_aiter_ops: except (ImportError, ModuleNotFoundError): return False + @staticmethod + def _gdn_flydsl_prefill_kernels_importable() -> bool: + try: + from aiter.ops.flydsl.linear_attention_prefill_kernels import ( # noqa: F401 + chunk_gated_delta_rule_fwd_h_flydsl_opt, + gdn_prepare_flydsl_supported, + gdn_prepare_fwd_flydsl, + ) + from aiter.ops.triton.gated_delta_net import ( # noqa: F401 + build_gated_delta_rule_prefill_metadata, + chunk_gated_delta_rule_opt_vk, + ) + + return True + except (ImportError, ModuleNotFoundError): + return False + @classmethod @if_aiter_supported def are_gdn_triton_kernels_available(cls) -> bool: @@ -2478,6 +2499,100 @@ class rocm_aiter_ops: """ return cls._AITER_ENABLED and cls._gdn_triton_kernels_importable() + @classmethod + @functools.cache + def is_gdn_flydsl_prefill_available(cls) -> bool: + """Whether the opt-in AITER FlyDSL GDN prefill path can be used. + + Selecting the backend is an explicit opt-in in itself, but it still + runs AITER kernels, so VLLM_ROCM_USE_AITER remains the one switch that + turns all of them off. + """ + return ( + cls._AITER_ENABLED + and is_aiter_found_and_supported() + and cls._gdn_flydsl_prefill_kernels_importable() + ) + + @classmethod + def build_gdn_flydsl_prefill_metadata( + cls, + seq_lens_cpu: list[int], + cu_seqlens: torch.Tensor, + ) -> object: + """Build the reusable varlen metadata FlyDSL GDN prefill runs against. + + The K5 recurrence is compiled for a fixed 64-token chunk, so the chunk + size is a property of the AITER kernel rather than of FLA, and is not + the caller's to choose. + """ + from aiter.ops.triton.gated_delta_net import ( + build_gated_delta_rule_prefill_metadata, + ) + + return build_gated_delta_rule_prefill_metadata( + seq_lens_cpu, + cu_seqlens=cu_seqlens, + chunk_size=cls.GDN_FLYDSL_CHUNK_SIZE, + ) + + @classmethod + def gdn_flydsl_prefill( + cls, + q: torch.Tensor, + k: torch.Tensor, + v: torch.Tensor, + g: torch.Tensor, + beta: torch.Tensor, + initial_state: torch.Tensor, + output_final_state: bool, + cu_seqlens: torch.Tensor | None, + aiter_prefill_metadata: object | None, + use_qk_l2norm_in_kernel: bool, + ) -> tuple[torch.Tensor, torch.Tensor]: + """Run GDN prefill through AITER's VK path with the FlyDSL kernels. + + Every argument AITER takes is named here rather than forwarded as + ``**kwargs``, so a rename or reordering on the AITER side fails at this + call instead of silently binding to the wrong parameter. + """ + from aiter.ops.flydsl.linear_attention_prefill_kernels import ( + gdn_prepare_flydsl_supported, + ) + from aiter.ops.triton.gated_delta_net import chunk_gated_delta_rule_opt_vk + + if cu_seqlens is not None and aiter_prefill_metadata is None: + raise RuntimeError( + "AITER FlyDSL GDN prefill requires reusable varlen prefill metadata." + ) + if not gdn_prepare_flydsl_supported(k, v): + raise RuntimeError( + "AITER FlyDSL GDN prepare does not support the runtime input shape, " + "dtype, or device." + ) + logger.info_once( + "Dispatching AITER FlyDSL GDN prefill (prepare=flydsl, chunk=flydsl)." + ) + return chunk_gated_delta_rule_opt_vk( + q=q, + k=k, + v=v, + g=g, + beta=beta, + initial_state=initial_state, + output_final_state=output_final_state, + cu_seqlens=cu_seqlens, + use_chunk_flydsl=True, + use_prepare_flydsl=True, + state_dtype=initial_state.dtype, + prefill_metadata=aiter_prefill_metadata, + # The caller stages the state densely and writes it back, so the + # kernel gets a plain [N, H, V, K] buffer and never an index into + # the pool; there is correspondingly nothing to update in place. + inplace_final_state=False, + use_qk_l2norm_in_kernel=use_qk_l2norm_in_kernel, + ) + @classmethod def is_rdna_gdn_triton_kernels_available(cls) -> bool: """RDNA4 (gfx12) analog of are_gdn_triton_kernels_available().""" @@ -2994,6 +3109,10 @@ class rocm_aiter_ops: transpose_scale, ) + @staticmethod + def get_fused_mla_dual_rms_norm_group_quant_op() -> OpOverload: + return torch.ops.vllm.fused_mla_dual_rms_norm_group_quant.default + @staticmethod def rms_norm( x: torch.Tensor, diff --git a/vllm/benchmarks/datasets/__init__.py b/vllm/benchmarks/datasets/__init__.py index b003ee4c05..f1f58954b0 100644 --- a/vllm/benchmarks/datasets/__init__.py +++ b/vllm/benchmarks/datasets/__init__.py @@ -27,7 +27,6 @@ from vllm.benchmarks.datasets.datasets import ( RandomMultiModalDataset, SampleRequest, ShareGPTDataset, - SonnetDataset, SpecBench, VisionArenaDataset, add_dataset_parser, @@ -71,7 +70,6 @@ __all__ = [ "RandomMultiModalDataset", "SampleRequest", "ShareGPTDataset", - "SonnetDataset", "SpecBench", "VisionArenaDataset", "add_dataset_parser", diff --git a/vllm/benchmarks/datasets/create_txt_slices_dataset.py b/vllm/benchmarks/datasets/create_txt_slices_dataset.py index 1bcd93c8ae..ed3523a653 100644 --- a/vllm/benchmarks/datasets/create_txt_slices_dataset.py +++ b/vllm/benchmarks/datasets/create_txt_slices_dataset.py @@ -12,8 +12,8 @@ Usage :: python -m vllm.benchmarks.datasets.create_txt_slices_dataset \\ - --input sonnet.txt \\ - --output sonnet_dataset.jsonl \\ + --input corpus.txt \\ + --output dataset.jsonl \\ --tokenizer gpt2 \\ --num-prompts 1000 \\ --input-len 1024 \\ @@ -23,7 +23,7 @@ The resulting JSONL file can then be used with the serving benchmark:: python -m vllm.benchmarks.serve \\ --dataset-name custom \\ - --dataset-path sonnet_dataset.jsonl \\ + --dataset-path dataset.jsonl \\ ... """ diff --git a/vllm/benchmarks/datasets/datasets.py b/vllm/benchmarks/datasets/datasets.py index abaf299a19..3746ac727e 100644 --- a/vllm/benchmarks/datasets/datasets.py +++ b/vllm/benchmarks/datasets/datasets.py @@ -5,7 +5,6 @@ datasets. Each dataset subclass of BenchmarkDataset must implement sample generation. Supported dataset types include: - ShareGPT - Random (synthetic) - - Sonnet - BurstGPT - HuggingFace - VisionArena @@ -31,7 +30,6 @@ from typing import Any, cast import numpy as np import pybase64 as base64 from PIL import Image -from typing_extensions import deprecated from vllm.benchmarks.datasets.utils import ( RangeRatio, @@ -1598,7 +1596,6 @@ def add_dataset_parser(parser: FlexibleArgumentParser): choices=[ "sharegpt", "burstgpt", - "sonnet", "random", "random-mm", "random-rerank", @@ -1622,8 +1619,7 @@ def add_dataset_parser(parser: FlexibleArgumentParser): "--dataset-path", type=str, default=None, - help="Path to the sharegpt/sonnet dataset or the HF dataset ID if " - "using HF dataset.", + help="Path to the ShareGPT dataset or the HF dataset ID if using HF dataset.", ) parser.add_argument( "--no-oversample", @@ -1681,26 +1677,6 @@ def add_dataset_parser(parser: FlexibleArgumentParser): help="Category for spec bench dataset. If None, use all categories.", ) - sonnet_group = parser.add_argument_group("sonnet dataset options") - sonnet_group.add_argument( - "--sonnet-input-len", - type=int, - default=550, - help="Number of input tokens per request, used only for sonnet dataset.", - ) - sonnet_group.add_argument( - "--sonnet-output-len", - type=int, - default=150, - help="Number of output tokens per request, used only for sonnet dataset.", - ) - sonnet_group.add_argument( - "--sonnet-prefix-len", - type=int, - default=200, - help="Number of prefix tokens per request, used only for sonnet dataset.", - ) - sharegpt_group = parser.add_argument_group("sharegpt dataset options") sharegpt_group.add_argument( "--sharegpt-output-len", @@ -2134,45 +2110,6 @@ def get_samples( no_oversample=args.no_oversample, ) - elif args.dataset_name == "sonnet": - assert tokenizer is not None, ( - "Tokenizer must be initialized for the 'sonnet' dataset." - ) - sonnet_dataset = SonnetDataset( - dataset_path=args.dataset_path, disable_shuffle=args.disable_shuffle - ) - # For the "sonnet" dataset, formatting depends on the backend. - # Chat-style backends leave templating to the server; completions - # backends need the prompt rendered client side. - if args.backend in ("openai-chat", "openai-responses"): - input_requests = sonnet_dataset.sample( - num_requests=args.num_prompts, - input_len=args.sonnet_input_len, - output_len=args.sonnet_output_len, - prefix_len=args.sonnet_prefix_len, - tokenizer=tokenizer, - return_prompt_formatted=False, - request_id_prefix=args.request_id_prefix, - no_oversample=args.no_oversample, - ) - else: - assert ( - hasattr(tokenizer, "chat_template") and tokenizer.chat_template - ) or ( - hasattr(tokenizer, "default_chat_template") - and tokenizer.default_chat_template - ), "Tokenizer/model must have chat template for sonnet dataset." - input_requests = sonnet_dataset.sample( - num_requests=args.num_prompts, - input_len=args.sonnet_input_len, - output_len=args.sonnet_output_len, - prefix_len=args.sonnet_prefix_len, - tokenizer=tokenizer, - return_prompt_formatted=True, - request_id_prefix=args.request_id_prefix, - no_oversample=args.no_oversample, - ) - elif args.dataset_name == "hf": # all following datasets are implemented from the # HuggingFaceDataset base class @@ -3070,106 +3007,6 @@ class SpecBench(CustomDataset): ) -# ----------------------------------------------------------------------------- -# Sonnet Dataset Implementation -# ----------------------------------------------------------------------------- - - -@deprecated( - "SonnetDataset is deprecated and will be removed in a future version.", -) -class SonnetDataset(BenchmarkDataset): - """Simplified implementation of the Sonnet dataset. Loads poem lines from a - text file and generates sample requests. Default values here copied from - `benchmark_serving.py` for the sonnet dataset. - """ - - DEFAULT_PREFIX_LEN = 200 - DEFAULT_INPUT_LEN = 550 - DEFAULT_OUTPUT_LEN = 150 - - def __init__( - self, - **kwargs, - ) -> None: - super().__init__(**kwargs) - self.load_data() - - def load_data(self) -> None: - if not self.dataset_path: - raise ValueError("dataset_path must be provided.") - with open(self.dataset_path, encoding="utf-8") as f: - self.data = f.readlines() - - def sample( - self, - tokenizer: TokenizerLike, - num_requests: int, - request_id_prefix: str = "", - no_oversample: bool = False, - prefix_len: int = DEFAULT_PREFIX_LEN, - input_len: int = DEFAULT_INPUT_LEN, - output_len: int = DEFAULT_OUTPUT_LEN, - return_prompt_formatted: bool = False, - **kwargs, - ) -> list[SampleRequest]: - poem_lines = self.data - assert poem_lines is not None - # Calculate average token length for a poem line. - tokenized_lines = [tokenizer(line).input_ids for line in poem_lines] - avg_len = sum(len(tokens) for tokens in tokenized_lines) / len(tokenized_lines) - - # Build the base prompt. - base_prompt = "Pick as many lines as you can from these poem lines:\n" - base_msg = [{"role": "user", "content": base_prompt}] - base_fmt = tokenizer.apply_chat_template( - base_msg, # type: ignore[arg-type] - add_generation_prompt=True, - tokenize=False, - ) - assert isinstance(base_fmt, str) - base_offset = len(tokenizer(base_fmt).input_ids) - if input_len <= base_offset: - raise ValueError( - f"'input_len' must be higher than the base prompt length " - f"({base_offset})." - ) - - # Determine how many poem lines to use. - num_input_lines = max(round((input_len - base_offset) / avg_len), 1) - num_prefix_lines = max(round((prefix_len - base_offset) / avg_len), 0) - prefix_lines = poem_lines[:num_prefix_lines] - - samples: list[SampleRequest] = [] - ind = 0 - while len(samples) < num_requests: - extra_lines = random.choices( - poem_lines, k=num_input_lines - num_prefix_lines - ) - prompt = f"{base_prompt}{''.join(prefix_lines + extra_lines)}" - msg = [{"role": "user", "content": prompt}] - prompt_formatted = tokenizer.apply_chat_template( - msg, # type: ignore[arg-type] - add_generation_prompt=True, - tokenize=False, - ) - assert isinstance(prompt_formatted, str) - prompt_len = len(tokenizer(prompt_formatted).input_ids) - if prompt_len <= input_len: - samples.append( - SampleRequest( - prompt=( - prompt_formatted if return_prompt_formatted else prompt - ), - prompt_len=prompt_len, - expected_output_len=output_len, - request_id=request_id_prefix + str(ind), - ) - ) - ind += 1 - return samples - - # ----------------------------------------------------------------------------- # BurstGPT Dataset Implementation # ----------------------------------------------------------------------------- diff --git a/vllm/benchmarks/latency.py b/vllm/benchmarks/latency.py index acb3efef15..0c75c04563 100644 --- a/vllm/benchmarks/latency.py +++ b/vllm/benchmarks/latency.py @@ -119,6 +119,7 @@ def main(args: argparse.Namespace): beam_width=args.n, max_tokens=args.output_len, ignore_eos=True, + watermarking=False, ), ) diff --git a/vllm/benchmarks/lib/endpoint_request_func.py b/vllm/benchmarks/lib/endpoint_request_func.py index 14ff6c1ab1..778a4e50d3 100644 --- a/vllm/benchmarks/lib/endpoint_request_func.py +++ b/vllm/benchmarks/lib/endpoint_request_func.py @@ -122,8 +122,6 @@ def _validate_api_url( if isinstance(expected_suffixes, str): expected_suffixes = {expected_suffixes} - expected_suffixes = {*expected_suffixes, "profile"} - if not api_url.endswith(tuple(expected_suffixes)): raise ValueError(f"{api_name} URL must end with one of: {expected_suffixes}.") @@ -158,6 +156,34 @@ def _get_headers(content_type: str | None = None) -> dict[str, str]: return headers +async def async_request_profile( + api_url: str, + session: aiohttp.ClientSession, + extra_headers: dict | None = None, +) -> RequestFuncOutput: + _validate_api_url( + api_url, + "Profiling API", + {"start_profile", "stop_profile"}, + ) + + headers = _get_headers() + if extra_headers: + headers |= extra_headers + + output = RequestFuncOutput() + try: + async with session.post(url=api_url, headers=headers) as response: + output.success = response.status == 200 + if not output.success: + output.error = response.reason or "" + except Exception: + exc_info = sys.exc_info() + output.error = "".join(traceback.format_exception(*exc_info)) + + return output + + async def async_request_openai_completions( request_func_input: RequestFuncInput, session: aiohttp.ClientSession, diff --git a/vllm/benchmarks/mm_processor.py b/vllm/benchmarks/mm_processor.py index 40b3494151..7a52faf29d 100644 --- a/vllm/benchmarks/mm_processor.py +++ b/vllm/benchmarks/mm_processor.py @@ -241,6 +241,7 @@ def benchmark_multimodal_processor( temperature=0.0, max_tokens=output_len, detokenize=True, + watermarking=False, ) for output_len in expected_output_lens ] @@ -262,7 +263,8 @@ def benchmark_multimodal_processor( warmup_prompts = [req.prompt for req in warmup_requests] warmup_output_lens = [req.expected_output_len for req in warmup_requests] warmup_sampling_params = [ - SamplingParams(max_tokens=output_len) for output_len in warmup_output_lens + SamplingParams(max_tokens=output_len, watermarking=False) + for output_len in warmup_output_lens ] llm.chat( warmup_prompts, diff --git a/vllm/benchmarks/serve.py b/vllm/benchmarks/serve.py index 19130716f4..01a0f62aee 100644 --- a/vllm/benchmarks/serve.py +++ b/vllm/benchmarks/serve.py @@ -49,6 +49,7 @@ from vllm.benchmarks.lib.endpoint_request_func import ( POOLING_BACKENDS, RequestFuncInput, RequestFuncOutput, + async_request_profile, ) from vllm.benchmarks.lib.ready_checker import wait_for_endpoint from vllm.benchmarks.lib.utils import ( @@ -252,7 +253,7 @@ async def fetch_spec_decode_metrics( num_accepted_tokens=num_accepted_tokens, accepted_per_pos=accepted_per_pos, ) - except (aiohttp.ClientError, asyncio.TimeoutError): + except (TimeoutError, aiohttp.ClientError): return None @@ -314,7 +315,7 @@ async def fetch_diffusion_metrics( num_canvas_positions=num_canvas_positions, num_committed_tokens=num_committed_tokens, ) - except (aiohttp.ClientError, asyncio.TimeoutError): + except (TimeoutError, aiohttp.ClientError): return None @@ -936,22 +937,10 @@ async def benchmark( if profile: print("Starting profiler...") - profile_input = RequestFuncInput( - model=model_id, - model_name=model_name, - prompt=test_prompt, - api_url=base_url + "/start_profile", - prompt_len=test_prompt_len, - output_len=test_output_len, - logprobs=logprobs, - multi_modal_content=test_mm_content, - ignore_eos=ignore_eos, + profile_output = await async_request_profile( + base_url + "/start_profile", + session, extra_headers=extra_headers, - extra_body=test_extra_body, - chat_messages=test_chat_messages, - ) - profile_output = await request_func( - request_func_input=profile_input, session=session ) if profile_output.success: print("Profiler started") @@ -1469,16 +1458,10 @@ async def benchmark( if profile: print("Stopping profiler...") - profile_input = RequestFuncInput( - model=model_id, - prompt=test_prompt, - api_url=base_url + "/stop_profile", - prompt_len=test_prompt_len, - output_len=test_output_len, - logprobs=logprobs, - ) - profile_output = await request_func( - request_func_input=profile_input, session=session + profile_output = await async_request_profile( + base_url + "/stop_profile", + session, + extra_headers=extra_headers, ) if profile_output.success: print("Profiler stopped") @@ -1670,8 +1653,7 @@ def add_cli_args(parser: FlexibleArgumentParser): "--input-len", type=int, default=None, - help="General input length for datasets. Maps to dataset-specific " - "input length arguments (e.g., --random-input-len, --sonnet-input-len). " + help="General input length for datasets. Maps to --random-input-len. " "If not specified, uses dataset defaults.", ) parser.add_argument( @@ -1679,7 +1661,7 @@ def add_cli_args(parser: FlexibleArgumentParser): type=int, default=None, help="General output length for datasets. Maps to dataset-specific " - "output length arguments (e.g., --random-output-len, --sonnet-output-len). " + "output length arguments (e.g., --random-output-len, --hf-output-len). " "If not specified, uses dataset defaults.", ) parser.add_argument( @@ -2132,18 +2114,16 @@ async def main_async(args: argparse.Namespace) -> dict[str, Any]: raise ValueError( f"Cannot use '{args.dataset_name}' dataset with --dataset-path. " "Please specify the appropriate --dataset-name (e.g., " - "'sharegpt', 'custom', 'sonnet') for your dataset file: " + "'sharegpt', 'custom', 'hf') for your dataset file: " f"{args.dataset_path}" ) - # Map general --input-len and --output-len to all dataset-specific arguments + # Map general length options to dataset-specific arguments. if args.input_len is not None: args.random_input_len = args.input_len - args.sonnet_input_len = args.input_len if args.output_len is not None: args.random_output_len = args.output_len - args.sonnet_output_len = args.output_len args.sharegpt_output_len = args.output_len args.custom_output_len = args.output_len args.hf_output_len = args.output_len @@ -2216,11 +2196,6 @@ async def main_async(args: argparse.Namespace) -> dict[str, Any]: "Sampling parameters are only supported by openai-compatible backends." ) - # The Responses API accepts every sampling parameter above except - # min_p, which it would silently drop as an unknown field. - if args.backend == "openai-responses" and "min_p" in sampling_params: - raise ValueError("--min-p is not supported by the Responses API.") - if "temperature" not in sampling_params: print( "WARNING: vllm bench serve no longer sets temperature==0 (greedy) " diff --git a/vllm/benchmarks/sweep/plot.py b/vllm/benchmarks/sweep/plot.py index 59a90636f0..6b36a7acc6 100644 --- a/vllm/benchmarks/sweep/plot.py +++ b/vllm/benchmarks/sweep/plot.py @@ -8,9 +8,9 @@ from dataclasses import dataclass from functools import partial from pathlib import Path from types import TracebackType -from typing import TYPE_CHECKING, ClassVar +from typing import TYPE_CHECKING, ClassVar, Self -from typing_extensions import Self, override +from typing_extensions import override from vllm.utils.argparse_utils import FlexibleArgumentParser from vllm.utils.collection_utils import full_groupby diff --git a/vllm/benchmarks/sweep/serve_workload.py b/vllm/benchmarks/sweep/serve_workload.py index a488837e2e..acf0ce5f7f 100644 --- a/vllm/benchmarks/sweep/serve_workload.py +++ b/vllm/benchmarks/sweep/serve_workload.py @@ -4,10 +4,9 @@ import argparse import math from dataclasses import dataclass, fields from pathlib import Path -from typing import ClassVar, Literal, get_args +from typing import ClassVar, Literal, assert_never, get_args import numpy as np -from typing_extensions import assert_never from vllm.benchmarks.datasets import DEFAULT_NUM_PROMPTS from vllm.utils.argparse_utils import FlexibleArgumentParser diff --git a/vllm/benchmarks/sweep/server.py b/vllm/benchmarks/sweep/server.py index 87d841ac82..dff7e06924 100644 --- a/vllm/benchmarks/sweep/server.py +++ b/vllm/benchmarks/sweep/server.py @@ -6,9 +6,9 @@ import signal import subprocess import time from types import TracebackType +from typing import Self import requests -from typing_extensions import Self class ServerProcess: diff --git a/vllm/benchmarks/throughput.py b/vllm/benchmarks/throughput.py index 620ce2b08f..6245b20e86 100644 --- a/vllm/benchmarks/throughput.py +++ b/vllm/benchmarks/throughput.py @@ -178,6 +178,7 @@ def _run_vllm_requests( beam_width=n, max_tokens=output_len, ignore_eos=True, + watermarking=False, ), ) if do_profile: @@ -553,10 +554,6 @@ def _to_serve_args(args: argparse.Namespace) -> argparse.Namespace: # matching prior throughput behaviour of omitting output_len when unset. d["hf_output_len"] = output_len d["sharegpt_output_len"] = output_len - # sonnet reads dedicated attrs; fall back to SonnetDataset's own defaults. - d["sonnet_input_len"] = input_len if input_len is not None else 550 - d["sonnet_output_len"] = output_len if output_len is not None else 150 - d["sonnet_prefix_len"] = prefix_len # Explicit --enable-multimodal-chat wins; otherwise auto-enable for the # multimodal chat backend (preserves today's vllm-chat handling). Callers # without a --backend flag (e.g. bench mm-processor) drive every request @@ -709,14 +706,14 @@ def validate_args(args): ) # --prefix-len: only used when dataset_name is 'random', 'random-mm', - # 'sonnet', or not set. + # or not set. if ( - args.dataset_name not in {"random", "random-mm", "sonnet", None} + args.dataset_name not in {"random", "random-mm", None} and args.prefix_len is not None ): warnings.warn( "--prefix-len will be ignored since --dataset-name\ - is not 'random', 'random-mm', 'sonnet', or not set.", + is not 'random', 'random-mm', or not set.", stacklevel=2, ) @@ -802,7 +799,6 @@ def add_cli_args(parser: FlexibleArgumentParser): choices=[ "sharegpt", "random", - "sonnet", "burstgpt", "hf", "prefix_repetition", diff --git a/vllm/compilation/backends.py b/vllm/compilation/backends.py index 43252ce3d8..d985d8de75 100644 --- a/vllm/compilation/backends.py +++ b/vllm/compilation/backends.py @@ -1088,10 +1088,13 @@ class VllmBackend: # Honors opt-outs such as CompilationMode.NONE or VLLM_DISABLE_COMPILE_CACHE. disable_cache = not is_compile_cache_enabled(self.inductor_config) - # TODO(patchy): ngram gpu kernel will cause vllm torch compile cache errors. + # TODO(patchy): the V1 torch.compile ngram-gpu kernel causes vllm + # torch compile cache errors. The V2 implementation is pure Triton and + # does not need the cache disabled. is_ngram_gpu_enabled = ( vllm_config.speculative_config is not None and vllm_config.speculative_config.use_ngram_gpu() + and not vllm_config.use_v2_model_runner ) disable_cache = disable_cache or is_ngram_gpu_enabled diff --git a/vllm/compilation/passes/fusion/collective_fusion.py b/vllm/compilation/passes/fusion/collective_fusion.py deleted file mode 100644 index f2aa6d7d2f..0000000000 --- a/vllm/compilation/passes/fusion/collective_fusion.py +++ /dev/null @@ -1,1013 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# SPDX-FileCopyrightText: Copyright contributors to the vLLM project - -from collections.abc import Callable -from contextlib import suppress - -import torch -import torch._inductor.pattern_matcher as pm -import torch.distributed.distributed_c10d as c10d -import torch.fx as fx -from torch._inductor.pattern_matcher import PatternMatcherPass -from torch.distributed._symmetric_memory import enable_symm_mem_for_group - -from vllm.config import VllmConfig -from vllm.config.utils import Range -from vllm.distributed import get_tp_group -from vllm.distributed.parallel_state import ( - get_tensor_model_parallel_world_size, -) -from vllm.logger import init_logger -from vllm.platforms import current_platform -from vllm.utils.torch_utils import direct_register_custom_op - -from ..inductor_pass import enable_fake_mode -from ..vllm_inductor_pass import ( - VllmFusionPatternMatcherPass, - VllmInductorPass, - VllmPatternMatcherPass, - VllmPatternReplacement, -) - -FP8_DTYPE = current_platform.fp8_dtype() - -logger = init_logger(__name__) - - -def _flashinfer_scaled_mm_out( - A: torch.Tensor, - B: torch.Tensor, - *, - scale_a: torch.Tensor, - scale_b: torch.Tensor, - out: torch.Tensor, - bias: torch.Tensor | None = None, - scale_result: torch.Tensor | None = None, - out_dtype: torch.dtype | None = None, - use_fast_accum: bool = False, -) -> None: - # Import lazily to avoid a circular import during module initialization - # when docs or other tooling import the pass without FlashInfer. - from vllm.utils.flashinfer import flashinfer_scaled_fp8_mm_out - - assert bias is None, "FlashInfer symm_mem adapter does not support bias" - assert scale_result is None, ( - "FlashInfer symm_mem adapter does not support result scaling" - ) - assert not use_fast_accum, ( - "FlashInfer symm_mem adapter does not support use_fast_accum" - ) - assert A.ndim == 2 and B.ndim == 2 and out.ndim == 2, ( - "FlashInfer symm_mem adapter expects 2D inputs and output" - ) - assert scale_a.numel() == 1 and scale_b.numel() == 1, ( - "FlashInfer symm_mem adapter only supports tensor-wise FP8 scales" - ) - - flashinfer_scaled_fp8_mm_out( - A, - B, - scale_a, - scale_b, - out=out, - out_dtype=out_dtype or out.dtype, - ) - - -def _flashinfer_fp4_mm_out( - A: torch.Tensor, - B: torch.Tensor, - *, - scale_a: torch.Tensor, - scale_b: torch.Tensor, - out: torch.Tensor, - alpha: torch.Tensor, - out_dtype: torch.dtype | None = None, - use_8x4_sf_layout: bool = False, - backend: str = "cutlass", -) -> None: - from vllm.utils.flashinfer import flashinfer_scaled_fp4_mm_out - - assert A.ndim == 2 and B.ndim == 2 and out.ndim == 2, ( - "FlashInfer FP4 symm_mem adapter expects 2D inputs and output" - ) - flashinfer_scaled_fp4_mm_out( - A, - B, - scale_a, - scale_b, - alpha, - out=out, - out_dtype=out_dtype or out.dtype, - use_8x4_sf_layout=use_8x4_sf_layout, - backend=backend, - ) - - -def fused_flashinfer_scaled_matmul_reduce_scatter_fake( - A: torch.Tensor, - B: torch.Tensor, - A_scale: torch.Tensor, - B_scale: torch.Tensor, - reduce_op: str, - orig_scatter_dim: int, - scatter_dim_after_maybe_reshape: int, - group_name: str, - output_shape: list[int], - out_dtype: torch.dtype | None = None, -) -> torch.Tensor: - world_size = c10d._resolve_process_group(group_name).size() - result_shape = list(output_shape) - result_shape[orig_scatter_dim] //= world_size - return torch.empty( - result_shape, - dtype=out_dtype or torch.bfloat16, - device=A.device, - ) - - -def fused_flashinfer_scaled_matmul_reduce_scatter( - A: torch.Tensor, - B: torch.Tensor, - A_scale: torch.Tensor, - B_scale: torch.Tensor, - reduce_op: str, - orig_scatter_dim: int, - scatter_dim_after_maybe_reshape: int, - group_name: str, - output_shape: list[int], - out_dtype: torch.dtype | None = None, -) -> torch.Tensor: - assert orig_scatter_dim == 0 and scatter_dim_after_maybe_reshape == 0, ( - "FlashInfer symm_mem adapter currently only supports scatter_dim=0" - ) - world_size = c10d._resolve_process_group(group_name).size() - assert A.ndim == 2 and B.ndim == 2, "FlashInfer symm_mem adapter expects 2D inputs" - assert A.is_contiguous(), "FlashInfer symm_mem adapter expects contiguous A" - assert A_scale.numel() == 1 and B_scale.numel() == 1, ( - "FlashInfer symm_mem adapter only supports tensor-wise FP8 scales" - ) - assert A.shape[0] % world_size == 0, ( - "FlashInfer symm_mem adapter expects M divisible by world size" - ) - - kwargs = { - "scale_b": B_scale, - "bias": None, - "scale_result": None, - "out_dtype": out_dtype, - "use_fast_accum": False, - } - return torch.distributed._symmetric_memory._fused_scaled_matmul_reduce_scatter_impl( - mm_out_op=_flashinfer_scaled_mm_out, - A=A, - B=B, - A_scale=A_scale, - kwargs=kwargs, - out_dtype=out_dtype, - reduce_op=reduce_op, - orig_scatter_dim=orig_scatter_dim, - scatter_dim_after_maybe_reshape=scatter_dim_after_maybe_reshape, - group_name=group_name, - output_shape=output_shape, - ) - - -def fused_all_gather_flashinfer_scaled_matmul_fake( - A_shard: torch.Tensor, - B: torch.Tensor, - A_scale: torch.Tensor, - B_scale: torch.Tensor, - gather_dim: int, - group_name: str, - out_dtype: torch.dtype | None = None, -) -> torch.Tensor: - world_size = c10d._resolve_process_group(group_name).size() - output_shape = list(A_shard.shape) - output_shape[gather_dim] *= world_size - output_shape[-1] = B.shape[1] - return torch.empty( - output_shape, - dtype=out_dtype or torch.bfloat16, - device=A_shard.device, - ) - - -def fused_all_gather_flashinfer_scaled_matmul( - A_shard: torch.Tensor, - B: torch.Tensor, - A_scale: torch.Tensor, - B_scale: torch.Tensor, - gather_dim: int, - group_name: str, - out_dtype: torch.dtype | None = None, -) -> torch.Tensor: - assert gather_dim == 0, ( - "FlashInfer symm_mem adapter currently only supports gather_dim=0" - ) - _, outputs = torch.distributed._symmetric_memory._fused_all_gather_matmul_impl( - mm_out_op=_flashinfer_scaled_mm_out, - A_shard=A_shard, - Bs=[B], - A_scale=A_scale, - kwargs_list=[ - { - "scale_b": B_scale, - "bias": None, - "scale_result": None, - "out_dtype": out_dtype, - "use_fast_accum": False, - } - ], - out_dtypes=[out_dtype], - gather_dim=gather_dim, - group_name=group_name, - return_A=False, - ) - return outputs[0] - - -def fused_all_gather_flashinfer_fp4_matmul_fake( - A_shard: torch.Tensor, - B: torch.Tensor, - A_scale_shard: torch.Tensor, - B_scale: torch.Tensor, - alpha: torch.Tensor, - gather_dim: int, - group_name: str, - out_dtype: torch.dtype | None = None, - view_a_scale_as_fp8: bool = False, - use_8x4_sf_layout: bool = False, - backend: str = "cutlass", -) -> torch.Tensor: - world_size = c10d._resolve_process_group(group_name).size() - output_shape = list(A_shard.shape) - output_shape[gather_dim] *= world_size - output_shape[-1] = B.shape[1] - return torch.empty( - output_shape, - dtype=out_dtype or torch.bfloat16, - device=A_shard.device, - ) - - -def fused_all_gather_flashinfer_fp4_matmul( - A_shard: torch.Tensor, - B: torch.Tensor, - A_scale_shard: torch.Tensor, - B_scale: torch.Tensor, - alpha: torch.Tensor, - gather_dim: int, - group_name: str, - out_dtype: torch.dtype | None = None, - view_a_scale_as_fp8: bool = False, - use_8x4_sf_layout: bool = False, - backend: str = "cutlass", -) -> torch.Tensor: - assert gather_dim == 0, ( - "FlashInfer FP4 symm_mem adapter currently only supports gather_dim=0" - ) - assert A_shard.ndim == 2 and A_scale_shard.ndim == 2 and B.ndim == 2, ( - "FlashInfer FP4 symm_mem adapter expects 2D inputs" - ) - if view_a_scale_as_fp8: - A_scale_shard = A_scale_shard.view(torch.float8_e4m3fn) - - group = c10d._resolve_process_group(group_name) - world_size = group.size() - output = A_shard.new_empty( - A_shard.shape[0] * world_size, - B.shape[1], - dtype=out_dtype or torch.bfloat16, - ) - output_shards = output.chunk(world_size) - - A = A_shard.new_empty(A_shard.shape[0] * world_size, A_shard.shape[1]) - A_scale = A_scale_shard.new_empty( - A_scale_shard.shape[0] * world_size, - A_scale_shard.shape[1], - ) - - def fp4_shard_consumer(shards: list[torch.Tensor], rank: int) -> None: - _flashinfer_fp4_mm_out( - shards[0], - B, - scale_a=shards[1], - scale_b=B_scale, - alpha=alpha, - out=output_shards[rank], - out_dtype=out_dtype, - use_8x4_sf_layout=use_8x4_sf_layout, - backend=backend, - ) - - torch.distributed._symmetric_memory._pipelined_multi_all_gather_and_consume( - [A_shard, A_scale_shard], - fp4_shard_consumer, - [A, A_scale], - group_name, - False, - ) - return output - - -direct_register_custom_op( - op_name="fused_flashinfer_scaled_matmul_reduce_scatter", - op_func=fused_flashinfer_scaled_matmul_reduce_scatter, - fake_impl=fused_flashinfer_scaled_matmul_reduce_scatter_fake, -) - -direct_register_custom_op( - op_name="fused_all_gather_flashinfer_scaled_matmul", - op_func=fused_all_gather_flashinfer_scaled_matmul, - fake_impl=fused_all_gather_flashinfer_scaled_matmul_fake, -) - -direct_register_custom_op( - op_name="fused_all_gather_flashinfer_fp4_matmul", - op_func=fused_all_gather_flashinfer_fp4_matmul, - fake_impl=fused_all_gather_flashinfer_fp4_matmul_fake, -) - - -class BasePattern: - def __init__(self, dtype: torch.dtype, device: str | None) -> None: - self.dtype = dtype - self.device = device - self.tp = get_tp_group() - self.tp_size = get_tensor_model_parallel_world_size() - - -class GEMMReduceScatterPattern(BasePattern): - rocm_gemm = False - - def get_inputs(self) -> list[torch.Tensor]: - mul = torch.empty([16, 4], device=self.device, dtype=self.dtype) - mm_weight = torch.empty([4, 4], device=self.device, dtype=self.dtype) - return [mul, mm_weight] - - def register(self, pm_pass: PatternMatcherPass) -> None: - def pattern(mul: torch.Tensor, mm_weight: torch.Tensor) -> torch.Tensor: - if self.rocm_gemm: - mm = torch.ops.vllm.rocm_unquantized_gemm(mul, mm_weight, None) - else: - mm = torch.ops.aten.mm.default(mul, mm_weight) - reduce_scatter = torch.ops.vllm.reduce_scatter.default( - mm, - dim=0, - world_size=self.tp_size, - group_name=self.tp.unique_name, - ) - return reduce_scatter - - def replacement(mul: torch.Tensor, mm_weight: torch.Tensor) -> torch.Tensor: - gemm_rs = torch.ops.symm_mem.fused_matmul_reduce_scatter( - mul, - mm_weight.t() if self.rocm_gemm else mm_weight, - "sum", - scatter_dim=0, - group_name=self.tp.device_group.group_name, - ) - - return gemm_rs - - pm.register_replacement( - pattern, replacement, self.get_inputs(), pm.fwd_only, pm_pass - ) - - -class RocmGEMMReduceScatterPattern(GEMMReduceScatterPattern): - rocm_gemm = True - - -class AllGatherGEMMPattern(BasePattern): - rocm_gemm = False - - def get_inputs(self) -> list[torch.Tensor]: - x = torch.empty([4, 4], device=self.device, dtype=self.dtype) - weight = torch.empty([4, 4], device=self.device, dtype=self.dtype) - - return [x, weight] - - def register(self, pm_pass: PatternMatcherPass) -> None: - def pattern( - x: torch.Tensor, - weight: torch.Tensor, - ) -> torch.Tensor: - all_gather = torch.ops.vllm.all_gather.default( - x, - dim=0, - world_size=self.tp_size, - group_name=self.tp.unique_name, - ) - - if self.rocm_gemm: - return torch.ops.vllm.rocm_unquantized_gemm(all_gather, weight, None) - return torch.ops.aten.mm.default(all_gather, weight) - - def replacement(x: torch.Tensor, weight: torch.Tensor) -> torch.Tensor: - ag_output, mm_outputs = torch.ops.symm_mem.fused_all_gather_matmul( - x, - [weight.t() if self.rocm_gemm else weight], - gather_dim=0, - group_name=self.tp.device_group.group_name, - ) - return mm_outputs - - pm.register_replacement( - pattern, replacement, self.get_inputs(), pm.fwd_only, pm_pass - ) - - -class RocmAllGatherGEMMPattern(AllGatherGEMMPattern): - rocm_gemm = True - - -class ScaledMMReduceScatterPattern(BasePattern): - def get_inputs(self) -> list[torch.Tensor]: - input = torch.empty([16, 16], device=self.device, dtype=FP8_DTYPE) - mm_weight = ( - torch.empty([16, 16], device=self.device, dtype=FP8_DTYPE) - .contiguous() - .transpose(0, 1) - ) - scale_a = torch.empty([16, 1], device=self.device, dtype=torch.float32) - scale_b = torch.empty([1, 16], device=self.device, dtype=torch.float32) - return [input, mm_weight, scale_a, scale_b] - - def register(self, pm_pass: PatternMatcherPass) -> None: - def pattern( - input: torch.Tensor, - mat2: torch.Tensor, - scale_a: torch.Tensor, - scale_b: torch.Tensor, - ) -> torch.Tensor: - scaled_mm = torch.ops.aten._scaled_mm.default( - input, - mat2=mat2, - scale_a=scale_a, - scale_b=scale_b, - bias=None, - scale_result=None, - out_dtype=self.dtype, - ) - reduce_scatter = torch.ops.vllm.reduce_scatter.default( - scaled_mm, - dim=0, - world_size=self.tp_size, - group_name=self.tp.unique_name, - ) - return reduce_scatter - - def replacement( - input: torch.Tensor, - mat2: torch.Tensor, - scale_a: torch.Tensor, - scale_b: torch.Tensor, - ) -> torch.Tensor: - # Calculate output shape: input @ mat2 with scatter_dim reduced - output_shape = [*input.shape[:-1], mat2.shape[1]] - scatter_dim = 0 - gemm_rs = torch.ops.vllm.patched_fused_scaled_matmul_reduce_scatter( - input, - mat2, - scale_a, - scale_b, - "sum", - scatter_dim, # orig_scatter_dim - scatter_dim, # scatter_dim_after_maybe_reshape - self.tp.device_group.group_name, - output_shape, - None, # bias - None, # result_scale - self.dtype, # out_dtype - False, # use_fast_accum - ) - - return gemm_rs - - pm.register_replacement( - pattern, replacement, self.get_inputs(), pm.fwd_only, pm_pass - ) - - -class AllGatherScaledMMPattern(BasePattern): - def get_inputs(self) -> list[torch.Tensor]: - x = torch.empty([8, 16], device=self.device, dtype=FP8_DTYPE) - weight = ( - torch.empty([16, 16], device=self.device, dtype=FP8_DTYPE) - .contiguous() - .transpose(0, 1) - ) - - s1 = x.shape[0] * self.tp_size - - scale_a = torch.empty([s1, 1], device=self.device, dtype=torch.float32) - scale_b = torch.empty([1, 16], device=self.device, dtype=torch.float32) - - return [x, weight, scale_a, scale_b] - - def register(self, pm_pass: PatternMatcherPass) -> None: - def pattern( - x: torch.Tensor, - weight: torch.Tensor, - scale_a: torch.Tensor, - scale_b: torch.Tensor, - ) -> torch.Tensor: - all_gather = torch.ops.vllm.all_gather.default( - x, dim=0, world_size=self.tp_size, group_name=self.tp.unique_name - ) - - return torch.ops.aten._scaled_mm.default( - all_gather, - mat2=weight, - scale_a=scale_a, - scale_b=scale_b, - bias=None, - scale_result=None, - out_dtype=self.dtype, - ) - - def replacement( - x: torch.Tensor, - weight: torch.Tensor, - scale_a: torch.Tensor, - scale_b: torch.Tensor, - ) -> torch.Tensor: - ag_output, mm_outputs = torch.ops.symm_mem.fused_all_gather_scaled_matmul( # noqa - x, - [weight], - scale_a, - [scale_b], - gather_dim=0, - biases=[None], - result_scales=[None], - out_dtypes=[self.dtype], - use_fast_accum=[False], - group_name=self.tp.device_group.group_name, - ) - return mm_outputs - - pm.register_replacement( - pattern, replacement, self.get_inputs(), pm.fwd_only, pm_pass - ) - - -class CutlassScaledMMReduceScatterPattern(BasePattern): - def get_inputs(self) -> list[torch.Tensor]: - input = torch.empty([16, 16], device=self.device, dtype=FP8_DTYPE) - mm_weight = ( - torch.empty([16, 16], device=self.device, dtype=FP8_DTYPE) - .contiguous() - .transpose(0, 1) - ) - scale_a = torch.empty([16, 1], device=self.device, dtype=torch.float32) - scale_b = torch.empty([1, 16], device=self.device, dtype=torch.float32) - - cutlass_mm_output = torch.empty([16, 16], device=self.device, dtype=self.dtype) - return [input, mm_weight, scale_a, scale_b, cutlass_mm_output] - - def register(self, pm_pass: PatternMatcherPass) -> None: - def pattern( - input: torch.Tensor, - weight: torch.Tensor, - scale_a: torch.Tensor, - scale_b: torch.Tensor, - cutlass_mm_output: torch.Tensor, - ) -> torch.Tensor: - cutlass_scaled_mm = torch.ops.higher_order.auto_functionalized( - torch.ops._C.cutlass_scaled_mm.default, - out=cutlass_mm_output, - a=input, - b=weight, - a_scales=scale_a, - b_scales=scale_b, - bias=None, - ) - - reduce_scatter = torch.ops.vllm.reduce_scatter.default( - cutlass_scaled_mm[1], - dim=0, - world_size=self.tp_size, - group_name=self.tp.unique_name, - ) - return reduce_scatter - - def replacement( - input: torch.Tensor, - mat2: torch.Tensor, - scale_a: torch.Tensor, - scale_b: torch.Tensor, - cutlass_mm_output: torch.Tensor, - ) -> torch.Tensor: - # Calculate output shape: input @ mat2 with scatter_dim reduced - output_shape = [*input.shape[:-1], mat2.shape[1]] - scatter_dim = 0 - gemm_rs = torch.ops.vllm.patched_fused_scaled_matmul_reduce_scatter( - input, - mat2, - scale_a, - scale_b, - "sum", - scatter_dim, # orig_scatter_dim - scatter_dim, # scatter_dim_after_maybe_reshape - self.tp.device_group.group_name, - output_shape, - None, # bias - None, # result_scale - self.dtype, # out_dtype - False, # use_fast_accum - ) - - return gemm_rs - - pm.register_replacement( - pattern, replacement, self.get_inputs(), pm.fwd_only, pm_pass - ) - - -class AllGatherCutlassScaledMMPattern(BasePattern): - def get_inputs(self) -> list[torch.Tensor]: - x = torch.empty([8, 16], device=self.device, dtype=FP8_DTYPE) - weight = ( - torch.empty([16, 16], device=self.device, dtype=FP8_DTYPE) - .contiguous() - .transpose(0, 1) - ) - - s1 = x.shape[0] * self.tp_size - - scale_a = torch.empty([s1, 1], device=self.device, dtype=torch.float32) - scale_b = torch.empty([1, 16], device=self.device, dtype=torch.float32) - - s2 = weight.shape[1] - output = torch.empty([s1, s2], device=self.device, dtype=self.dtype) - - return [x, weight, scale_a, scale_b, output] - - def register(self, pm_pass: PatternMatcherPass) -> None: - def pattern( - x: torch.Tensor, - weight: torch.Tensor, - scale_a: torch.Tensor, - scale_b: torch.Tensor, - output: torch.Tensor, - ) -> torch.Tensor: - all_gather = torch.ops.vllm.all_gather.default( - x, dim=0, world_size=self.tp_size, group_name=self.tp.unique_name - ) - - cutlass_scaled_mm = torch.ops.higher_order.auto_functionalized( - torch.ops._C.cutlass_scaled_mm.default, - out=output, - a=all_gather, - b=weight, - a_scales=scale_a, - b_scales=scale_b, - bias=None, - ) - return cutlass_scaled_mm[1] - - def replacement( - x: torch.Tensor, - weight: torch.Tensor, - scale_a: torch.Tensor, - scale_b: torch.Tensor, - output: torch.Tensor, - ) -> torch.Tensor: - ag_output, mm_outputs = torch.ops.symm_mem.fused_all_gather_scaled_matmul( # noqa - x, - [weight], - scale_a, - [scale_b], - gather_dim=0, - biases=[None], - result_scales=[None], - out_dtypes=[self.dtype], - use_fast_accum=[False], - group_name=self.tp.device_group.group_name, - ) - return mm_outputs - - pm.register_replacement( - pattern, replacement, self.get_inputs(), pm.fwd_only, pm_pass - ) - - -class FlashInferBMMFP8ReduceScatterPattern( - BasePattern, VllmPatternReplacement[..., torch.Tensor] -): - def get_inputs(self) -> list[torch.Tensor]: - a_2d = torch.empty([16, 16], device=self.device, dtype=FP8_DTYPE) - b_2d = ( - torch.empty([16, 16], device=self.device, dtype=FP8_DTYPE) - .contiguous() - .transpose(0, 1) - ) - a_scale = torch.empty([1], device=self.device, dtype=torch.float32) - b_scale = torch.empty([1], device=self.device, dtype=torch.float32) - return [a_2d, b_2d, a_scale, b_scale] - - @property - def pattern(self) -> Callable[..., torch.Tensor]: - def _pattern( - a_2d: torch.Tensor, - b_2d: torch.Tensor, - a_scale: torch.Tensor, - b_scale: torch.Tensor, - ) -> torch.Tensor: - bmm = torch.ops.vllm.bmm_fp8.default( - torch.ops.aten.unsqueeze.default(a_2d, 0), - torch.ops.aten.unsqueeze.default(b_2d, 0), - a_scale, - b_scale, - self.dtype, - "auto", - ) - output = torch.ops.aten.reshape.default(bmm, list(bmm.shape[1:])) - return torch.ops.vllm.reduce_scatter.default( - output, - dim=0, - world_size=self.tp_size, - group_name=self.tp.unique_name, - ) - - return _pattern - - @property - def replacement(self) -> Callable[..., torch.Tensor]: - def _replacement( - a_2d: torch.Tensor, - b_2d: torch.Tensor, - a_scale: torch.Tensor, - b_scale: torch.Tensor, - ) -> torch.Tensor: - return torch.ops.vllm.fused_flashinfer_scaled_matmul_reduce_scatter.default( - a_2d, - b_2d, - a_scale, - b_scale, - "sum", - 0, - 0, - self.tp.device_group.group_name, - [a_2d.shape[0], b_2d.shape[1]], - self.dtype, - ) - - return _replacement - - -class FlashInferAllGatherBMMFP8Pattern( - BasePattern, VllmPatternReplacement[..., torch.Tensor] -): - def get_inputs(self) -> list[torch.Tensor]: - a_shard_2d = torch.empty([8, 16], device=self.device, dtype=FP8_DTYPE) - b_2d = ( - torch.empty([16, 16], device=self.device, dtype=FP8_DTYPE) - .contiguous() - .transpose(0, 1) - ) - a_scale = torch.empty([1], device=self.device, dtype=torch.float32) - b_scale = torch.empty([1], device=self.device, dtype=torch.float32) - return [a_shard_2d, b_2d, a_scale, b_scale] - - @property - def pattern(self) -> Callable[..., torch.Tensor]: - def _pattern( - a_shard_2d: torch.Tensor, - b_2d: torch.Tensor, - a_scale: torch.Tensor, - b_scale: torch.Tensor, - ) -> torch.Tensor: - all_gather = torch.ops.vllm.all_gather.default( - a_shard_2d, - dim=0, - world_size=self.tp_size, - group_name=self.tp.unique_name, - ) - return torch.ops.vllm.bmm_fp8.default( - torch.ops.aten.unsqueeze.default(all_gather, 0), - torch.ops.aten.unsqueeze.default(b_2d, 0), - a_scale, - b_scale, - self.dtype, - "auto", - ) - - return _pattern - - @property - def replacement(self) -> Callable[..., torch.Tensor]: - def _replacement( - a_shard_2d: torch.Tensor, - b_2d: torch.Tensor, - a_scale: torch.Tensor, - b_scale: torch.Tensor, - ) -> torch.Tensor: - fused = torch.ops.vllm.fused_all_gather_flashinfer_scaled_matmul.default( - a_shard_2d, - b_2d, - a_scale, - b_scale, - 0, - self.tp.device_group.group_name, - self.dtype, - ) - return torch.ops.aten.unsqueeze.default(fused, 0) - - return _replacement - - -class FlashInferAllGatherFP4Pattern( - BasePattern, VllmPatternReplacement[..., torch.Tensor] -): - def __init__( - self, - dtype: torch.dtype, - device: str | None, - backend: str, - use_8x4_sf_layout: bool, - a_scale_view: str, - ) -> None: - super().__init__(dtype, device) - self.backend = backend - self.use_8x4_sf_layout = use_8x4_sf_layout - self.a_scale_view = a_scale_view - - def get_inputs(self) -> list[torch.Tensor]: - a_shard_2d = torch.empty([8, 8], device=self.device, dtype=torch.uint8) - b_2d = torch.empty([8, 16], device=self.device, dtype=torch.uint8) - a_scale_shard = torch.empty([128, 4], device=self.device, dtype=torch.int32) - b_scale = torch.empty([4, 128], device=self.device, dtype=torch.uint8) - alpha = torch.empty([], device=self.device, dtype=torch.float32) - return [ - a_shard_2d, - b_2d, - a_scale_shard, - b_scale, - alpha, - ] - - @property - def pattern(self) -> Callable[..., torch.Tensor]: - def _pattern( - a_shard_2d: torch.Tensor, - b_2d: torch.Tensor, - a_scale_shard: torch.Tensor, - b_scale: torch.Tensor, - alpha: torch.Tensor, - ) -> torch.Tensor: - all_gather_a = torch.ops.vllm.all_gather.default( - a_shard_2d, - dim=0, - world_size=self.tp_size, - group_name=self.tp.unique_name, - ) - all_gather_a_scale = torch.ops.vllm.all_gather.default( - a_scale_shard, - dim=0, - world_size=self.tp_size, - group_name=self.tp.unique_name, - ) - a_scale = all_gather_a_scale - if self.a_scale_view in ("float8", "float8_uint8"): - a_scale = torch.ops.aten.view.dtype(a_scale, torch.float8_e4m3fn) - if self.a_scale_view in ("uint8", "float8_uint8"): - a_scale = torch.ops.aten.view.dtype(a_scale, torch.uint8) - return torch.ops.vllm.flashinfer_mm_fp4.default( - all_gather_a, - b_2d, - a_scale, - b_scale, - alpha, - self.dtype, - self.use_8x4_sf_layout, - self.backend, - ) - - return _pattern - - @property - def replacement(self) -> Callable[..., torch.Tensor]: - def _replacement( - a_shard_2d: torch.Tensor, - b_2d: torch.Tensor, - a_scale_shard: torch.Tensor, - b_scale: torch.Tensor, - alpha: torch.Tensor, - ) -> torch.Tensor: - return torch.ops.vllm.fused_all_gather_flashinfer_fp4_matmul.default( - a_shard_2d, - b_2d, - a_scale_shard, - b_scale, - alpha, - 0, - self.tp.device_group.group_name, - self.dtype, - self.a_scale_view in ("float8", "float8_uint8"), - self.use_8x4_sf_layout, - self.backend, - ) - - return _replacement - - -class AsyncTPPass(VllmFusionPatternMatcherPass): - @enable_fake_mode - def __init__(self, config: VllmConfig) -> None: - super().__init__(config, pass_name="async_tp_pass") - - enable_symm_mem_for_group(get_tp_group().device_group.group_name) - GEMMReduceScatterPattern(self.model_dtype, self.device).register(self.pm_pass) - - AllGatherGEMMPattern(self.model_dtype, self.device).register(self.pm_pass) - if current_platform.is_rocm() and self.model_dtype == torch.bfloat16: - RocmGEMMReduceScatterPattern(self.model_dtype, self.device).register( - self.pm_pass - ) - RocmAllGatherGEMMPattern(self.model_dtype, self.device).register( - self.pm_pass - ) - - # These fusions are enabled only for bfloat16 models because - # `scaled_mm` or `cutlass_scaled_mm` with per-token (row-wise) scaling - # only supports bfloat16 as the output dtype. - if self.model_dtype == torch.bfloat16: - ScaledMMReduceScatterPattern(self.model_dtype, self.device).register( - self.pm_pass - ) - AllGatherScaledMMPattern(self.model_dtype, self.device).register( - self.pm_pass - ) - if hasattr(torch.ops._C, "cutlass_scaled_mm"): - CutlassScaledMMReduceScatterPattern( - self.model_dtype, self.device - ).register(self.pm_pass) - AllGatherCutlassScaledMMPattern(self.model_dtype, self.device).register( - self.pm_pass - ) - with suppress(ImportError): - import vllm.utils.flashinfer # noqa: F401 - if hasattr(torch.ops.vllm, "bmm_fp8"): - self.register( - FlashInferAllGatherBMMFP8Pattern(self.model_dtype, self.device) - ) - self.register( - FlashInferBMMFP8ReduceScatterPattern(self.model_dtype, self.device) - ) - if hasattr(torch.ops.vllm, "flashinfer_mm_fp4"): - for backend in ("cutlass", "cudnn"): - for a_scale_view in ("float8_uint8", "uint8"): - self.register( - FlashInferAllGatherFP4Pattern( - self.model_dtype, - self.device, - backend, - use_8x4_sf_layout=False, - a_scale_view=a_scale_view, - ) - ) - for use_8x4_sf_layout in (False, True): - for a_scale_view in ("float8",): - self.register( - FlashInferAllGatherFP4Pattern( - self.model_dtype, - self.device, - "trtllm", - use_8x4_sf_layout=use_8x4_sf_layout, - a_scale_view=a_scale_view, - ) - ) - self.register( - FlashInferAllGatherFP4Pattern( - self.model_dtype, - self.device, - "cute-dsl", - use_8x4_sf_layout=False, - a_scale_view="float8", - ) - ) - # NVFP4 reduce-scatter does not need scale communication: FP4 - # scales are consumed by the local GEMM and only BF16 partial - # outputs are reduced. Keep this PR scoped to the all-gather - # path; reduce-scatter needs a dedicated FP4 producer rather - # than the existing FP8-style helper. - - self.dump_patterns(config, self.pm_pass) - - def is_applicable_for_range(self, compile_range: Range) -> bool: - # This pass is applied on top of the sequence parallelism pass, - # which is only supported in fullgraph compilation mode. - assert ( - self.compilation_config.use_inductor_graph_partition - or not self.compilation_config.splitting_ops - ), "AsyncTPPass requires full-graph compilation" - return True - - @VllmInductorPass.time_and_log - def __call__(self, graph: fx.Graph) -> None: - self.matched_count = self.pm_pass.apply(graph) - VllmPatternMatcherPass.match_table[self.pass_name] += self.matched_count - logger.debug("Replaced %s patterns", self.matched_count) diff --git a/vllm/compilation/passes/fusion/rocm_aiter_fusion.py b/vllm/compilation/passes/fusion/rocm_aiter_fusion.py index 19bbfc2d87..849cd5d67d 100644 --- a/vllm/compilation/passes/fusion/rocm_aiter_fusion.py +++ b/vllm/compilation/passes/fusion/rocm_aiter_fusion.py @@ -1074,16 +1074,148 @@ class MLADualRMSPerTokenQuantPattern( return _replacement +class MLADualRMSGroupQuantPattern( + VllmPatternReplacement[ + ..., + tuple[ + torch.Tensor, + torch.Tensor, + torch.Tensor, + torch.Tensor, + ], + ] +): + """Fuse the MLA FP8 *block-scale* attention path -- q-latent RMSNorm + FP8 + group quant plus kv-latent RMSNorm -- into AITER's + ``fused_qk_rmsnorm_group_quant``. + + With a block-scale FP8 ``q_b_proj`` (DeepSeek fp8), the earlier + ``RocmAiterRMSNormQuantFusionPass`` folds the q side into + ``rocm_aiter_rmsnorm_fp8_group_quant`` (the Triton rms+group-quant kernel) + and leaves the kv side a plain ``vllm_ir.rms_norm``. This pattern matches + that asymmetric pair:: + + gemm -> split_with_sizes([q_dim, kv_dim]) + +-- q_c -> rocm_aiter_rmsnorm_fp8_group_quant -> (q_fp8, q_scale) + +-- kv_lora -> split_with_sizes([kv_c_dim, k_pe_dim]) + +-- kv_c -> vllm_ir.rms_norm -> kv_normed (bf16) + +-- k_pe + """ + + GROUP_QUANT_OP = rocm_aiter_ops.get_rmsnorm_group_fused_quant_op() + FUSED_OP = rocm_aiter_ops.get_fused_mla_dual_rms_norm_group_quant_op() + + def __init__(self, epsilon: float) -> None: + self._epsilon = epsilon + + def get_inputs(self) -> list[torch.Tensor]: + q_dim, kv_c_dim, k_pe_dim = 256, 128, 64 + return [ + self.empty_bf16(5, q_dim + kv_c_dim + k_pe_dim), + self.empty_bf16(q_dim), + self.empty_bf16(kv_c_dim), + ] + + @property + def pattern( + self, + ) -> Callable[ + ..., + tuple[ + torch.Tensor, + torch.Tensor, + torch.Tensor, + torch.Tensor, + ], + ]: + eps = self._epsilon + group_quant_op = self.GROUP_QUANT_OP + + def _pattern( + projected: torch.Tensor, + q_weight: torch.Tensor, + kv_weight: torch.Tensor, + ) -> tuple[ + torch.Tensor, + torch.Tensor, + torch.Tensor, + torch.Tensor, + ]: + q_dim = q_weight.shape[0] + kv_dim = projected.shape[-1] - q_dim + kv_c_dim = kv_weight.shape[0] + k_pe_dim = kv_dim - kv_c_dim + q_c, kv_lora = projected.split([q_dim, kv_dim], dim=-1) + kv_c, k_pe = kv_lora.split([kv_c_dim, k_pe_dim], dim=-1) + quant_kwargs = dict( + x=q_c, + weight=q_weight, + variance_epsilon=eps, + group_size=128, + ) + q_quant = group_quant_op(**quant_kwargs) + kv_normed = vllm.ir.ops.rms_norm(kv_c, kv_weight, eps) + return q_quant[0], q_quant[1], kv_normed, k_pe + + return _pattern + + @property + def replacement( + self, + ) -> Callable[ + ..., + tuple[ + torch.Tensor, + torch.Tensor, + torch.Tensor, + torch.Tensor, + ], + ]: + eps = self._epsilon + fused_op = self.FUSED_OP + + def _replacement( + projected: torch.Tensor, + q_weight: torch.Tensor, + kv_weight: torch.Tensor, + ) -> tuple[ + torch.Tensor, + torch.Tensor, + torch.Tensor, + torch.Tensor, + ]: + q_dim = q_weight.shape[0] + kv_dim = projected.shape[-1] - q_dim + kv_c_dim = kv_weight.shape[0] + k_pe_dim = kv_dim - kv_c_dim + q_c, kv_lora = projected.split([q_dim, kv_dim], dim=-1) + kv_c, k_pe = kv_lora.split([kv_c_dim, k_pe_dim], dim=-1) + at = fused_op( + q_c, + q_weight, + kv_c, + kv_weight, + eps, + eps, + 128, + False, + ) + # q_fp8, q_scale, kv_normed, k_pe + return at[0], at[1], at[2], k_pe + + return _replacement + + class MLADualRMSNormFusionPass(VllmFusionPatternMatcherPass): """Post-grad PatternMatcher pass that fuses paired q / kv RMS norms in MLA attention into ``fused_mla_dual_rms_norm`` backed by aiter's ``fused_qk_rmsnorm`` HIP kernel. The FP8 attention path is also handled via - :class:`MLADualRMSPerTokenQuantPattern`, which fuses the q-latent RMSNorm + - FP8 per-token quant together with the kv-latent RMSNorm into - ``fused_mla_dual_rms_norm_per_token_quant`` backed by aiter's - ``fused_qk_rmsnorm_per_token_quant`` HIP kernel. + :class:`MLADualRMSPerTokenQuantPattern` or :class:`MLADualRMSGroupQuantPattern`, + which fuse the q-latent RMSNorm + FP8 per-token/group quant together with + the kv-latent RMSNorm into ``fused_mla_dual_rms_norm__quant`` + backed by aiter's ``fused_qk_rmsnorm__quant`` HIP kernel. """ def __init__(self, config: VllmConfig) -> None: @@ -1092,3 +1224,4 @@ class MLADualRMSNormFusionPass(VllmFusionPatternMatcherPass): for epsilon in [1e-5, 1e-6]: self.register(MLADualRMSNormPattern(epsilon)) self.register(MLADualRMSPerTokenQuantPattern(epsilon)) + self.register(MLADualRMSGroupQuantPattern(epsilon)) diff --git a/vllm/compilation/passes/fusion/sequence_parallelism.py b/vllm/compilation/passes/fusion/sequence_parallelism.py deleted file mode 100644 index b1a39b1e01..0000000000 --- a/vllm/compilation/passes/fusion/sequence_parallelism.py +++ /dev/null @@ -1,620 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# SPDX-FileCopyrightText: Copyright contributors to the vLLM project - -import functools -from collections.abc import Callable, Sequence -from typing import Any - -import torch -import torch._inductor.pattern_matcher as pm -import torch.fx as fx -from torch._higher_order_ops.auto_functionalize import auto_functionalized -from torch._inductor.pattern_matcher import PatternMatcherPass - -import vllm.ir.ops -from vllm.config import VllmConfig -from vllm.config.utils import Range -from vllm.distributed import get_tp_group, tensor_model_parallel_all_reduce -from vllm.distributed.parallel_state import ( - get_tensor_model_parallel_rank, - get_tensor_model_parallel_world_size, -) -from vllm.logger import init_logger -from vllm.model_executor.layers.quantization.utils.quant_utils import ( - kFp8StaticTensorSym, -) - -from ..inductor_pass import enable_fake_mode -from ..utility.noop_elimination import NoOpEliminationPass -from ..vllm_inductor_pass import VllmInductorPass, VllmPatternMatcherPass -from .matcher_utils import MatcherQuantFP8 - -logger = init_logger(__name__) - -if hasattr(torch.ops._C, "scaled_fp4_quant"): - SCALED_FP4_QUANT_OUT_OVERLOAD = torch.ops._C.scaled_fp4_quant.out - SCALED_FP4_QUANT_DEFAULT_OVERLOAD = torch.ops._C.scaled_fp4_quant.default - -# Min hidden size per device capability for sequence parallelism -# Only apply sequence parallelism for models with hidden_size >= threshold -SP_MIN_HIDDEN_SIZE: dict[int, int] = { - 90: 8192, # H100: only for models with hidden_size >= 8192 - 100: 8192, # Blackwell family: only for models with hidden_size >= 8192 -} - -# Min size per GPU per device capability for sequence parallelism -# Total min size = min_per_gpu_size * tp_size -# This ensures the threshold scales appropriately with tensor parallelism -SP_MIN_PER_GPU_SIZE_MB: dict[int, float] = { - 90: 8, # 8MB per GPU for H100 - # Use a more conservative threshold on Blackwell so TP8 starts later. - 100: 32, -} - - -def get_sequence_parallelism_threshold( - hidden_size: int, - tp_size: int, - element_size: int, -) -> int | None: - """Calculate the minimum token threshold for applying sequence parallelism. - - Returns None if sequence parallelism should not be applied based on model size. - - Branching logic based on device capability: - - Check if hidden_size >= SP_MIN_HIDDEN_SIZE[device_capability] - - If not, returns None (SP disabled for small models on this device) - - If yes, calculates threshold based on per-GPU size - - Formula: min_token_num = (min_per_gpu_size_mb * tp_size * MiB) // - (hidden_size * element_size) - """ - from vllm.platforms import current_platform - - if current_platform.is_xpu(): - min_hidden_size = 4096 - min_per_gpu_size_mb = 8.0 - elif current_platform.is_cuda(): - capability = current_platform.get_device_capability() - if capability is None: - return None - - # Collapse Blackwell variants (sm100/sm103/...) into one policy bucket. - if current_platform.is_device_capability_family(100): - device_capability = 100 - else: - device_capability = capability.to_int() - - # Check if device has configured thresholds - _hidden = SP_MIN_HIDDEN_SIZE.get(device_capability) - _gpu_mb = SP_MIN_PER_GPU_SIZE_MB.get(device_capability) - if _hidden is None or _gpu_mb is None: - return None - min_hidden_size, min_per_gpu_size_mb = _hidden, _gpu_mb - else: - return None - - # Only apply sequence parallelism for models meeting the size threshold - if hidden_size < min_hidden_size: - return None - - MiB = 1024 * 1024 - min_size = min_per_gpu_size_mb * MiB * tp_size - return int(min_size // (hidden_size * element_size)) - - -def get_first_out_wrapper( - fn: Callable[..., Sequence[torch.Tensor]], -) -> Callable[..., torch.Tensor]: - @functools.wraps(fn) - def wrapper(*args: Any) -> torch.Tensor: - return fn(*args)[0] - - return wrapper - - -class _SequenceParallelPatternHelper: - """Helper for sequence parallelism patterns.""" - - def __init__( - self, - epsilon: float, - dtype: torch.dtype, - device: str | None, - ) -> None: - self.epsilon = epsilon - self.dtype = dtype - self.device = device - self.tp_group = get_tp_group() - self.tp_size = get_tensor_model_parallel_world_size() - self.tp_rank = get_tensor_model_parallel_rank() - - def _all_reduce(self, x: torch.Tensor) -> torch.Tensor: - return tensor_model_parallel_all_reduce(x) - - def _reduce_scatter(self, x: torch.Tensor) -> torch.Tensor: - return torch.ops.vllm.reduce_scatter.default( - x, dim=0, world_size=self.tp_size, group_name=self.tp_group.unique_name - ) - - def _all_gather(self, x: torch.Tensor) -> torch.Tensor: - return torch.ops.vllm.all_gather.default( - x, dim=0, world_size=self.tp_size, group_name=self.tp_group.unique_name - ) - - def empty(self, *args: Any, **kwargs: Any) -> torch.Tensor: - return torch.empty(*args, dtype=self.dtype, device=self.device, **kwargs) - - def empty_f32(self, *args: Any, **kwargs: Any) -> torch.Tensor: - return torch.empty(*args, dtype=torch.float32, device=self.device, **kwargs) - - -class FirstAllReduceRMSNormPattern(_SequenceParallelPatternHelper): - def __init__(self, epsilon: float, dtype: torch.dtype, device: str | None) -> None: - super().__init__(epsilon, dtype, device) - - def get_inputs(self) -> list[torch.Tensor]: - # input, weight - return [self.empty([1, 8, 4]), self.empty([4])] - - def register(self, pm_pass: PatternMatcherPass) -> None: - def pattern( - input: torch.Tensor, - weight: torch.Tensor, - ) -> tuple[torch.Tensor, torch.Tensor]: - all_reduce = self._all_reduce(input) - rmsnorm = vllm.ir.ops.rms_norm(all_reduce, weight, self.epsilon) - - return rmsnorm, all_reduce - - def replacement( - input: torch.Tensor, - weight: torch.Tensor, - ) -> tuple[torch.Tensor, torch.Tensor]: - reduce_scatter = self._reduce_scatter(input) - - rmsnorm = vllm.ir.ops.rms_norm(reduce_scatter, weight, self.epsilon) - all_gather = self._all_gather(rmsnorm) - return all_gather, reduce_scatter - - pm.register_replacement( - pattern, replacement, self.get_inputs(), pm.fwd_only, pm_pass - ) - - -class MiddleAllReduceRMSNormPattern(_SequenceParallelPatternHelper): - def __init__(self, epsilon: float, dtype: torch.dtype, device: str | None) -> None: - super().__init__(epsilon, dtype, device) - - def get_inputs(self) -> list[torch.Tensor]: - mm_1 = torch.empty([4, 4], device=self.device, dtype=self.dtype) - - residual = torch.empty([4, 4], device=self.device, dtype=self.dtype) - rms_norm_weights = torch.empty([4, 4], device=self.device, dtype=self.dtype) - - return [ - residual, - mm_1, - rms_norm_weights, - ] - - def register(self, pm_pass: PatternMatcherPass) -> None: - def pattern( - residual: torch.Tensor, - mm_1: torch.Tensor, - rms_norm_weights: torch.Tensor, - ) -> tuple[torch.Tensor, torch.Tensor]: - all_reduce = self._all_reduce(mm_1) - rmsnorm = vllm.ir.ops.fused_add_rms_norm( - all_reduce, residual, rms_norm_weights, self.epsilon - ) - return rmsnorm[0], rmsnorm[1] - - def replacement( - residual: torch.Tensor, - mm_1: torch.Tensor, - rms_norm_weights: torch.Tensor, - ) -> tuple[torch.Tensor, torch.Tensor]: - # The pattern matcher replaces from the end of the graph - # (last layer first). At the time each match is replaced, - # the preceding layer has NOT been replaced yet, so - # `residual` is still full-size and the slice below is - # correct. Once the preceding layer IS replaced, its - # residual output shrinks to [local_len, H], and this - # slice becomes semantically incorrect (e.g. for rank > 0, - # the indices would be out of bounds). However, since the - # symbolic output shape equals the input shape, - # NoOpEliminationPass (called at the end of - # SequenceParallelismPass.__call__) removes these slices - # before the graph is ever executed or compiled. - reduce_scatter = self._reduce_scatter(mm_1) - local_len = reduce_scatter.size(0) - # when the preceding VocabParallelEmbedding is excluded - # from the FX graph (e.g., passing `inputs_embeds` directly in VLMs), - # the FirstAllReduceRMSNorm pattern is never matched. we must - # perform a proper TP-aware slice here. simply using `[0:local_len]` - # would incorrectly cause all ranks to process rank 0's chunk. - residual = residual[ - self.tp_rank * local_len : self.tp_rank * local_len + local_len, ... - ] - rmsnorm = vllm.ir.ops.fused_add_rms_norm( - reduce_scatter, residual, rms_norm_weights, self.epsilon - ) - all_gather = self._all_gather(rmsnorm[0]) - # residual output is now [local_len, H]; the next layer's - # slice on it is semantically incorrect until - # NoOpEliminationPass removes it. - return all_gather, rmsnorm[1] - - pm.register_replacement( - pattern, replacement, self.get_inputs(), pm.fwd_only, pm_pass - ) - pm.register_replacement( - get_first_out_wrapper(pattern), - get_first_out_wrapper(replacement), - self.get_inputs(), - pm.fwd_only, - pm_pass, - ) - - -class FirstAllReduceRMSNormStaticFP8Pattern(_SequenceParallelPatternHelper): - def __init__( - self, - epsilon: float, - dtype: torch.dtype, - device: str | None, - ) -> None: - super().__init__(epsilon, dtype, device) - self.quant_matcher = MatcherQuantFP8(kFp8StaticTensorSym) - - def get_inputs(self) -> list[torch.Tensor]: - # input, weight, scale - return [self.empty([1, 8, 4]), self.empty([4]), self.empty_f32([1, 1])] - - def register(self, pm_pass: PatternMatcherPass) -> None: - def pattern( - input: torch.Tensor, - weight: torch.Tensor, - scale: torch.Tensor, - ) -> tuple[torch.Tensor, torch.Tensor]: - all_reduce = self._all_reduce(input) - rms = vllm.ir.ops.rms_norm(all_reduce, weight, self.epsilon) - quant, _ = self.quant_matcher(rms, scale) - return quant, all_reduce - - def replacement( - input: torch.Tensor, - weight: torch.Tensor, - scale: torch.Tensor, - ) -> tuple[torch.Tensor, torch.Tensor]: - reduce_scatter = self._reduce_scatter(input) - rms = vllm.ir.ops.rms_norm(reduce_scatter, weight, self.epsilon) - quant, _ = self.quant_matcher(rms, scale) - all_gather = self._all_gather(quant) - - return all_gather, reduce_scatter - - pm.register_replacement( - pattern, replacement, self.get_inputs(), pm.fwd_only, pm_pass - ) - - -class MiddleAllReduceRMSNormStaticFP8Pattern(_SequenceParallelPatternHelper): - def __init__(self, epsilon: float, dtype: torch.dtype, device: str | None) -> None: - super().__init__(epsilon, dtype, device) - self.quant_matcher = MatcherQuantFP8(kFp8StaticTensorSym) - - def get_inputs(self) -> list[torch.Tensor]: - mm_1 = torch.empty([4, 4], device=self.device, dtype=self.dtype) - residual = torch.empty([4, 4], device=self.device, dtype=self.dtype) - rms_norm_weights = torch.empty([4, 4], device=self.device, dtype=self.dtype) - scale = torch.empty([1, 1], device=self.device, dtype=torch.float32) - - return [residual, mm_1, rms_norm_weights, scale] - - def register(self, pm_pass: PatternMatcherPass) -> None: - def pattern( - residual: torch.Tensor, - mm_1: torch.Tensor, - rms_norm_weights: torch.Tensor, - scale: torch.Tensor, - ) -> tuple[torch.Tensor, torch.Tensor]: - all_reduce = self._all_reduce(mm_1) - rms, residual_out = vllm.ir.ops.fused_add_rms_norm( - all_reduce, residual, rms_norm_weights, self.epsilon - ) - quant, _ = self.quant_matcher(rms, scale) - return quant, residual_out - - def replacement( - residual: torch.Tensor, - mm_1: torch.Tensor, - rms_norm_weights: torch.Tensor, - scale: torch.Tensor, - ) -> tuple[torch.Tensor, torch.Tensor]: - # See MiddleAllReduceRMSNormPattern.replacement for a - # detailed explanation of the temporary slice below: - # it is correct when first inserted, becomes semantically - # incorrect after the preceding layer is replaced, and is - # removed by NoOpEliminationPass before the graph is compiled. - reduce_scatter = self._reduce_scatter(mm_1) - local_len = reduce_scatter.size(0) - # when the preceding VocabParallelEmbedding is excluded - # from the FX graph (e.g., passing `inputs_embeds` directly in VLMs), - # the FirstAllReduceRMSNorm pattern is never matched. we must - # perform a proper TP-aware slice here. simply using `[0:local_len]` - # would incorrectly cause all ranks to process rank 0's chunk. - residual = residual[ - self.tp_rank * local_len : self.tp_rank * local_len + local_len, ... - ] - rms, residual_out = vllm.ir.ops.fused_add_rms_norm( - reduce_scatter, residual, rms_norm_weights, self.epsilon - ) - quant, _ = self.quant_matcher(rms, scale) - all_gather = self._all_gather(quant) - # residual output is now [local_len, H]; the next layer's - # slice on it is semantically incorrect until - # NoOpEliminationPass removes it. - return all_gather, residual_out - - pm.register_replacement( - pattern, replacement, self.get_inputs(), pm.fwd_only, pm_pass - ) - - pm.register_replacement( - get_first_out_wrapper(pattern), - get_first_out_wrapper(replacement), - self.get_inputs(), - pm.fwd_only, - pm_pass, - ) - - -class FirstAllReduceRMSNormStaticNVFP4Pattern(_SequenceParallelPatternHelper): - def get_inputs(self) -> list[torch.Tensor]: - input = self.empty([8, 16]) - weight = self.empty([16]) - input_global_scale = self.empty_f32([1, 1]) - quant_output = torch.empty([8, 8], device=self.device, dtype=torch.uint8) - output_scale = torch.empty([128, 4], device=self.device, dtype=torch.int32) - return [input, weight, input_global_scale, quant_output, output_scale] - - def register(self, pm_pass: PatternMatcherPass) -> None: - def pattern( - input: torch.Tensor, - weight: torch.Tensor, - input_global_scale: torch.Tensor, - quant_output: torch.Tensor, - output_scale: torch.Tensor, - ) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]: - all_reduce = self._all_reduce(input) - rms = vllm.ir.ops.rms_norm(all_reduce, weight, self.epsilon) - quant = auto_functionalized( - SCALED_FP4_QUANT_OUT_OVERLOAD, - input=rms, - input_scale=input_global_scale, - is_sf_swizzled_layout=True, - output=quant_output, - output_scale=output_scale, - ) - return quant[1], all_reduce, quant[2] - - def replacement( - input: torch.Tensor, - weight: torch.Tensor, - input_global_scale: torch.Tensor, - quant_output: torch.Tensor, - output_scale: torch.Tensor, - ) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]: - reduce_scatter = self._reduce_scatter(input) - rms = vllm.ir.ops.rms_norm(reduce_scatter, weight, self.epsilon) - rms = torch.ops.aten.view.default(rms, [-1, rms.shape[-1]]) - quant = SCALED_FP4_QUANT_DEFAULT_OVERLOAD( - rms, - input_global_scale, - True, - ) - return ( - self._all_gather(quant[0]), - reduce_scatter, - self._all_gather(quant[1]), - ) - - pm.register_replacement( - pattern, replacement, self.get_inputs(), pm.fwd_only, pm_pass - ) - - -class MiddleAllReduceRMSNormStaticNVFP4Pattern(_SequenceParallelPatternHelper): - def get_inputs(self) -> list[torch.Tensor]: - mm_1 = self.empty([8, 16]) - residual = self.empty([8, 16]) - rms_norm_weights = self.empty([16]) - input_global_scale = self.empty_f32([1, 1]) - quant_output = torch.empty([8, 8], device=self.device, dtype=torch.uint8) - output_scale = torch.empty([128, 4], device=self.device, dtype=torch.int32) - return [ - residual, - mm_1, - rms_norm_weights, - input_global_scale, - quant_output, - output_scale, - ] - - def register(self, pm_pass: PatternMatcherPass) -> None: - def pattern( - residual: torch.Tensor, - mm_1: torch.Tensor, - rms_norm_weights: torch.Tensor, - input_global_scale: torch.Tensor, - quant_output: torch.Tensor, - output_scale: torch.Tensor, - ) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]: - all_reduce = self._all_reduce(mm_1) - rms, residual_out = vllm.ir.ops.fused_add_rms_norm( - all_reduce, residual, rms_norm_weights, self.epsilon - ) - quant = auto_functionalized( - SCALED_FP4_QUANT_OUT_OVERLOAD, - input=rms, - input_scale=input_global_scale, - is_sf_swizzled_layout=True, - output=quant_output, - output_scale=output_scale, - ) - return quant[1], residual_out, quant[2] - - def replacement( - residual: torch.Tensor, - mm_1: torch.Tensor, - rms_norm_weights: torch.Tensor, - input_global_scale: torch.Tensor, - quant_output: torch.Tensor, - output_scale: torch.Tensor, - ) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]: - # Keep this slice in sync with the non-quantized SP replacement: - # once the previous SP pattern fires, it becomes a no-op. - reduce_scatter = self._reduce_scatter(mm_1) - residual = residual[0 : reduce_scatter.size(0), ...] - rms, residual_out = vllm.ir.ops.fused_add_rms_norm( - reduce_scatter, residual, rms_norm_weights, self.epsilon - ) - rms = torch.ops.aten.view.default(rms, [-1, rms.shape[-1]]) - quant = SCALED_FP4_QUANT_DEFAULT_OVERLOAD( - rms, - input_global_scale, - True, - ) - return self._all_gather(quant[0]), residual_out, self._all_gather(quant[1]) - - pm.register_replacement( - pattern, replacement, self.get_inputs(), pm.fwd_only, pm_pass - ) - - -class SequenceParallelismPass(VllmPatternMatcherPass): - """This pass enables sequence parallelism for models. - It identifies patterns where an AllReduce operation is followed by - an RMSNorm (or RMSNorm and then Quantization) operation. - These patterns are replaced with a ReduceScatter operation, followed by - a local RMSNorm/Quantization, and then an AllGather operation. - - The general transformation is: - Input -> AllReduce -> RMSNorm -> Output - becomes - Input -> ReduceScatter -> RMSNorm -> AllGather -> Output - - While this pass itself does not directly yield performance improvements, - it lays the groundwork for subsequent fusion passes, such as - GEMM + ReduceScatter and AllGather + GEMM fusions. These fusions can - significantly reduce communication overhead and improve overall model - performance. - - This pass is only supported when compiling the whole graph (fullgraph - mode, i.e. using Inductor graph partition or empty splitting_ops). - Piecewise compilation is not supported because the residual tensor - gets split across TP ranks, causing size mismatches at subgraph - boundaries. - - This pass splits up the residual tensor across TP ranks and hence divides - its size. The pattern matcher starts at the end of the graph (last layer - first), so when each replacement inserts a residual slice, the preceding - layer has not been replaced yet and the slice is correct. Once the - preceding layer IS replaced, its residual output shrinks and the slice - becomes semantically incorrect (out-of-bounds indices for rank > 0). - The graph is never executed in this intermediate state — - NoOpEliminationPass removes these slices based on symbolic shape equality - (input shape == output shape) before the graph is compiled. - """ - - @enable_fake_mode - def __init__(self, config: VllmConfig) -> None: - super().__init__(config) - - # Get min_token_num threshold - # Read min_token_num from config (calculated during config init) - self.min_token_num = None - if config.model_config is not None: - pass_config = config.compilation_config.pass_config - self.min_token_num = pass_config.sp_min_token_num - - if self.min_token_num is not None: - # Take the min to avoid exceeding max_num_batched_tokens - max_batched = config.scheduler_config.max_num_batched_tokens - if max_batched is not None: - self.min_token_num = min(self.min_token_num, max_batched) - logger.debug_once( - f"Sequence parallelism min token threshold: {self.min_token_num}", - scope="global", - ) - - # Used to clean up redundant views created temporarily - # to circumvent residual shape change issues - self.noop_cleanup = NoOpEliminationPass(config) - self.noop_cleanup.pass_name = f"{self.pass_name}.{self.noop_cleanup.pass_name}" - - self.patterns: PatternMatcherPass = PatternMatcherPass( - pass_name="sequence_parallelism_pass" - ) - - for epsilon in [1e-5, 1e-6]: - # RMSNorm + Static FP8 quantization patterns - FirstAllReduceRMSNormStaticFP8Pattern( - epsilon, self.model_dtype, self.device - ).register(self.patterns) - MiddleAllReduceRMSNormStaticFP8Pattern( - epsilon, self.model_dtype, self.device - ).register(self.patterns) - - if "SCALED_FP4_QUANT_OUT_OVERLOAD" in globals(): - FirstAllReduceRMSNormStaticNVFP4Pattern( - epsilon, self.model_dtype, self.device - ).register(self.patterns) - MiddleAllReduceRMSNormStaticNVFP4Pattern( - epsilon, self.model_dtype, self.device - ).register(self.patterns) - - # Normal RMSNorm patterns - FirstAllReduceRMSNormPattern( - epsilon, self.model_dtype, self.device - ).register(self.patterns) - - MiddleAllReduceRMSNormPattern( - epsilon, self.model_dtype, self.device - ).register(self.patterns) - - self.dump_patterns(config, self.patterns) - - def is_applicable_for_range(self, compile_range: Range) -> bool: - """Determines if sequence parallelism should be applied for the given - compile range. - - SP is only beneficial for larger batch sizes where the communication - overhead is amortized. For small batches, the overhead of splitting - and gathering tensors across TP ranks outweighs the benefits. - - Returns False (SP disabled) when: - - min_token_num is None (SP disabled for this device/config) - - The compile range starts below the minimum token threshold - """ - assert ( - self.compilation_config.use_inductor_graph_partition - or not self.compilation_config.splitting_ops - ), "SequenceParallelismPass requires full-graph compilation" - - # min_token_num is None when SP is disabled for this device/config - # (e.g., non-CUDA platform, unsupported GPU, or small hidden_size) - if self.min_token_num is None: - return False - - # Only apply SP when batch size meets the minimum threshold - return compile_range.start >= self.min_token_num - - @VllmInductorPass.time_and_log - def __call__(self, graph: fx.Graph) -> None: - self.matched_count = self.patterns.apply(graph) - logger.debug("Replaced %s patterns", self.matched_count) - # Clean up reshape nodes - self.noop_cleanup(graph) diff --git a/vllm/compilation/passes/pass_manager.py b/vllm/compilation/passes/pass_manager.py index df0e58b210..197f2208a1 100644 --- a/vllm/compilation/passes/pass_manager.py +++ b/vllm/compilation/passes/pass_manager.py @@ -35,13 +35,11 @@ if current_platform.is_cuda_alike() or current_platform.is_xpu(): RMSNormReshapeFusionPass, ) from .fusion.qk_norm_rope_fusion import QKNormRoPEFusionPass - from .fusion.sequence_parallelism import SequenceParallelismPass from .utility.split_coalescing import SplitCoalescingPass if current_platform.is_cuda_alike(): from .fusion.act_quant_fusion import ActivationQuantFusionPass from .fusion.attn_quant_fusion import AttnQuantFusionPass - from .fusion.collective_fusion import AsyncTPPass from .fusion.mla_attn_quant_fusion import MLAAttnQuantFusionPass from .fusion.mla_rope_kvcache_cat_fusion import MLARoPEKVCacheCatFusionPass from .fusion.qk_norm_rope_kvcache_fusion import QkNormRopeKvCacheFusionPass @@ -156,11 +154,6 @@ class PostGradPassManager(CustomGraphPass): # type: ignore[misc] if self.pass_config.eliminate_noops: self.passes += [NoOpEliminationPass(config)] - if self.pass_config.enable_sp: - self.passes += [SequenceParallelismPass(config)] - if self.pass_config.fuse_gemm_comms: - self.passes += [AsyncTPPass(config)] - if enable_transformers_norm_canonicalization: self.passes += [AddRMSNormFusionPass(config)] diff --git a/vllm/config/cache.py b/vllm/config/cache.py index b899502165..541590f7d4 100644 --- a/vllm/config/cache.py +++ b/vllm/config/cache.py @@ -50,6 +50,7 @@ CacheDType = Literal[ "turboquant_4bit_nc", "turboquant_k3v4_nc", "turboquant_3bit_nc", + "ultraquant_4bit", "int4_per_token_head", "int8_per_token_head", "fp8_per_token_head", @@ -229,6 +230,15 @@ class CacheConfig: for hybrid models where requests occupy multiple KV cache groups.""" kv_cache_max_concurrency: float | None = field(default=None, init=False) """Per-DP-engine maximum concurrency at max_model_len tokens.""" + # The two differ when hashes are finer than the scheduler block but partial + # hash hits are off (e.g. a sliding-window group cannot look up sub-block + # hits), so hits only land on the scheduler block. + # TODO(zjy0516): Merge into one value + hash_block_size: int | None = field(default=None, init=False) + """Tokens per prefix-cache block hash.""" + cache_hit_alignment_tokens: int | None = field(default=None, init=False) + """Token granularity at which prefix-cache hits land: `hash_block_size` + with partial hash hits, otherwise the scheduler block size.""" kv_sharing_fast_prefill: bool = False """In some KV sharing setups, e.g. YOCO (https://arxiv.org/abs/2405.05254), @@ -297,6 +307,8 @@ class CacheConfig: "effective_attention_block_size", "kv_cache_size_tokens", "kv_cache_max_concurrency", + "hash_block_size", + "cache_hit_alignment_tokens", # Feature toggles not impacting compiled graph shape "kv_sharing_fast_prefill", "swa_bounded_replay", diff --git a/vllm/config/compilation.py b/vllm/config/compilation.py index 3a492843fe..bda3843d8c 100644 --- a/vllm/config/compilation.py +++ b/vllm/config/compilation.py @@ -133,12 +133,6 @@ class PassConfig: """Fuse the custom Attention and MLAAttention + quant ops.""" eliminate_noops: bool = Field(default=True) """Eliminate no-op ops.""" - enable_sp: bool = None # type: ignore[assignment] - """Enable sequence parallelism. Requires TP>1. Automatically disabled - if the model's hidden_size is too small for SP to be beneficial - (threshold is device-capability dependent).""" - fuse_gemm_comms: bool = None # type: ignore[assignment] - """Enable async TP.""" fuse_allreduce_rms: bool = None # type: ignore[assignment] """Enable flashinfer allreduce fusion.""" enable_qk_norm_rope_fusion: bool = None # type: ignore[assignment] @@ -183,12 +177,6 @@ class PassConfig: 8: 1, # 1MB }, }, where key is the device capability""" - sp_min_token_num: int | None = None - """The minimum number of tokens above which vllm should use - sequence parallelism. Specified as an integer token count. - Unspecified will fallback to default values which are compute - capability and world size dependent.""" - # TODO(luka) better pass enabling system. def flashinfer_max_size(self, world_size: int) -> int | None: @@ -231,8 +219,6 @@ class PassConfig: "fuse_norm_quant", "fuse_act_quant", "fuse_attn_quant", - "enable_sp", - "fuse_gemm_comms", "fuse_allreduce_rms", "fuse_act_padding", "fuse_mla_dual_rms_norm", @@ -570,6 +556,24 @@ class CompilationConfig: If we limit the video count per prompt to `0`, it will also be set to `0` (i.e., fall back to image-only mode).""" + cudagraph_decoder_replay: bool = True + """Run the decoder replay layers of YOCO models (e.g. DeepSeek-V4.1) in CUDA + graphs of their own, and trim them in PIECEWISE graph steps. Requires + breakable PIECEWISE graphs; off under LoRA, prompt embeddings and non-first + PP ranks.""" + + decoder_replay_cudagraph_capture_sizes: list[int] = field(default_factory=list) + """Decoder replay CUDA graph sizes for YOCO models (e.g. DeepSeek-V4.1), at + most max_num_batched_tokens. If empty: multiples of sliding_window up to + min(max_cudagraph_capture_size, max_num_seqs * sliding_window), coarser past + 16 * sliding_window. Larger replay batches run eagerly.""" + + decoder_replay_trim_threshold: int = Field(default=768, ge=0) + """For YOCO models (e.g. DeepSeek-V4.1), PIECEWISE graphs with at least this + many padded tokens trim the decoder replay batch; must exceed the replay + window (128), below which nothing trims. Independent of the replay graph + capture sizes; eager steps still trim.""" + # Inductor capture compile_sizes: list[int | str] | None = None """Sizes to compile for inductor. In addition @@ -770,6 +774,8 @@ class CompilationConfig: # Qwen4Exp's AMD backend still uses these splitting ops. "vllm::qwen4_exp_ple_short_conv", "vllm::qwen4_exp_qsa_with_output", + # The fused prepare calls this op instead of the one above. + "vllm::qwen4_exp_qsa_prepare_with_output", "vllm::linear_attention", "vllm::qwen_gdn_attention_core", "vllm::qwen_gdn_attention_core_fused_norm_packed", @@ -984,10 +990,16 @@ class CompilationConfig: # (fixme @boyuan) combo kernel does not support cpu yet. and not current_platform.is_cpu() ): + from torch._inductor import config as inductor_config + # use horizontal fusion, which is useful for fusing qk-norm and # qk-rope when query and key have different shapes. self.inductor_compile_config["combo_kernels"] = True - self.inductor_compile_config["benchmark_combo_kernel"] = True + + deterministic = self.inductor_compile_config.get( + "deterministic", getattr(inductor_config, "deterministic", False) + ) + self.inductor_compile_config["benchmark_combo_kernel"] = not deterministic if self.use_inductor_graph_partition and not is_torch_equal_or_newer( "2.9.0.dev" @@ -1033,6 +1045,11 @@ class CompilationConfig: f"Invalid backend for piecewise compilation: {self.backend}" ) + if any(s <= 0 for s in self.decoder_replay_cudagraph_capture_sizes): + raise ValueError( + "All decoder_replay_cudagraph_capture_sizes must be positive" + ) + # Validate encoder CUDA graph configuration if ( self.cudagraph_mm_encoder @@ -1215,25 +1232,6 @@ class CompilationConfig: self.cudagraph_mode = CUDAGraphMode.FULL self.splitting_ops = [] - if ( - not self.use_inductor_graph_partition - and (self.pass_config.enable_sp or self.pass_config.fuse_gemm_comms) - and self.splitting_ops - ): - logger.warning_once( - "Sequence parallelism requires full-graph compilation when " - "use_inductor_graph_partition is off. Setting splitting_ops " - "to an empty list to preserve SP and async TP." - ) - self.splitting_ops = [] - if self.cudagraph_mode.has_piecewise_cudagraphs(): - logger.warning_once( - "Sequence parallelism is incompatible with piecewise " - "cudagraph when use_inductor_graph_partition is off. " - "Setting cudagraph_mode to FULL." - ) - self.cudagraph_mode = CUDAGraphMode.FULL - # Disable CUDA graphs for DeepEP high-throughput since its not CG compatible if ( all2all_backend == "deepep_high_throughput" @@ -1374,7 +1372,6 @@ class CompilationConfig: min_cg_attn_backend: str | None, uniform_decode_query_len: int = 1, use_v2_model_runner: bool = False, - tensor_parallel_size: int = 1, kv_cache_config: "KVCacheConfig | None" = None, max_num_reqs: int | None = None, is_profiling: bool = False, @@ -1506,10 +1503,7 @@ class CompilationConfig: and cudagraph_mode.decode_mode() == CUDAGraphMode.FULL and uniform_decode_query_len > 1 ): - self.adjust_cudagraph_sizes_for_spec_decode( - uniform_decode_query_len, - tensor_parallel_size, - ) + self.adjust_cudagraph_sizes_for_spec_decode(uniform_decode_query_len) # For Mamba models with FULL decode cudagraphs, each decode # sequence needs one Mamba cache block. The decode cudagraph @@ -1538,25 +1532,8 @@ class CompilationConfig: self.cudagraph_mode = cudagraph_mode return cudagraph_mode - def adjust_cudagraph_sizes_for_spec_decode( - self, uniform_decode_query_len: int, tensor_parallel_size: int - ): + def adjust_cudagraph_sizes_for_spec_decode(self, uniform_decode_query_len: int): multiple_of = uniform_decode_query_len - if tensor_parallel_size > 1 and self.pass_config.enable_sp: - multiple_of = max(uniform_decode_query_len, tensor_parallel_size) - if ( - multiple_of % uniform_decode_query_len != 0 - or multiple_of % tensor_parallel_size != 0 - ): - raise ValueError( - f"Can't determine cudagraph shapes that are both a " - f"multiple of {uniform_decode_query_len} " - f"(num_speculative_tokens + 1) required by spec-decode " - f"and {tensor_parallel_size} (tensor_parallel_size) " - f"required by sequence parallelism please adjust " - f"num_speculative_tokens or disable sequence parallelism" - ) - if not self.cudagraph_capture_sizes or multiple_of <= 1: return @@ -1576,7 +1553,7 @@ class CompilationConfig: if len(rounded_sizes) == 0: raise ValueError( f"No valid cudagraph sizes after rounding to multiple of {multiple_of} " - f"(num_speculative_tokens + 1 or tp if sequence parallelism is enabled)" + f"(num_speculative_tokens + 1)" f" please adjust num_speculative_tokens ({uniform_decode_query_len - 1}" f") or max_cudagraph_capture_size ({self.max_cudagraph_capture_size})" f" or cudagraph_capture_sizes ({self.cudagraph_capture_sizes})" diff --git a/vllm/config/engram.py b/vllm/config/engram.py index b47226ad1a..b6129fe3ef 100644 --- a/vllm/config/engram.py +++ b/vllm/config/engram.py @@ -1,10 +1,9 @@ # SPDX-License-Identifier: Apache-2.0 # SPDX-FileCopyrightText: Copyright contributors to the vLLM project -from typing import TYPE_CHECKING +from typing import TYPE_CHECKING, Self from pydantic import model_validator -from typing_extensions import Self from vllm.config.utils import config, get_hash_factors, hash_factors diff --git a/vllm/config/kernel.py b/vllm/config/kernel.py index df69f17149..fa3ecf5df9 100644 --- a/vllm/config/kernel.py +++ b/vllm/config/kernel.py @@ -137,6 +137,7 @@ MoEBackend = Literal[ "humming", "triton_unfused", "aiter", + "aiter_mega_moe", "aiter_triton_mxfp4_bf16", "flydsl", "hpc", @@ -278,6 +279,9 @@ class KernelConfig: enable_flashinfer_autotune: bool = None # type: ignore[assignment] """If True, run FlashInfer autotuning during kernel warmup.""" + enable_rocm_segmented_attn_autotune: bool = False + """If True, autotune ROCm segmented attention during kernel warmup on RDNA GPUs.""" + # TODO(roberto): Remove after registered CuTeDSL warmups are migrated # to the shared JIT warmup infrastructure. # https://github.com/vllm-project/vllm/pull/47451 @@ -314,6 +318,9 @@ class KernelConfig: - "humming": Use Humming Mixed Precision kernels - "triton_unfused": Use Triton unfused MoE kernels - "aiter": Use AMD AITer kernels (ROCm only) + - "aiter_mega_moe": Use the AITER MegaMoE fused dispatch/GEMM/combine + kernel (ROCm gfx950, requires DP+EP with a MoRI all2all backend; + DeepSeek-V4 only) - "aiter_triton_mxfp4_bf16": Use the AITER Triton MXFP4 W4A16 (moe_gemm_a16w4) MoE kernel (ROCm gfx942/gfx950/gfx1250) - "flydsl": Use AMD FlyDSL kernels (ROCm only) @@ -406,6 +413,7 @@ class KernelConfig: "enable_cutedsl_warmup", "enable_jit_warmup", "enable_flashinfer_autotune", + "enable_rocm_segmented_attn_autotune", "ir_op_priority", # handled separately below } if self.linear_backend_per_quant is None: diff --git a/vllm/config/lora.py b/vllm/config/lora.py index e5c1d14f6d..a77660f3b1 100644 --- a/vllm/config/lora.py +++ b/vllm/config/lora.py @@ -2,11 +2,10 @@ # SPDX-FileCopyrightText: Copyright contributors to the vLLM project import os -from typing import TYPE_CHECKING, Any, Literal +from typing import TYPE_CHECKING, Any, Literal, Self import torch from pydantic import ConfigDict, Field, model_validator -from typing_extensions import Self from vllm import envs from vllm.config.utils import config @@ -25,7 +24,6 @@ logger = init_logger(__name__) LoRADType = Literal["auto", "float16", "bfloat16"] MaxLoRARanks = Literal[1, 8, 16, 32, 64, 128, 256, 320, 512] -LoRAExtraVocabSize = Literal[256, 512] @config(config=ConfigDict(arbitrary_types_allowed=True)) @@ -82,8 +80,9 @@ class LoRAConfig: """If True, load MoE expert adapters in the "shared-outer" layout, where the gate/up (`w1`/`w3`) lora_A and the down (`w2`) lora_B are shared across all experts (stored once with expert-dim 1) instead of per-expert. The shared - factors are broadcast to the expert count at kernel time. Only meaningful for - MoE models whose adapters use this layout; ignored otherwise.""" + factors are broadcast to the expert count at kernel time. 3D-weight MoE + models use `FusedMoEWithLoRA` for this layout. Only meaningful for MoE + models whose adapters use this layout; ignored otherwise.""" max_lora_cls_labels: int | None = Field(default=None, ge=1) """Maximum output size for LoRA classification heads. Defaults to the base classification head size.""" diff --git a/vllm/config/multimodal.py b/vllm/config/multimodal.py index 24c534b364..3bba0434c3 100644 --- a/vllm/config/multimodal.py +++ b/vllm/config/multimodal.py @@ -725,7 +725,7 @@ class MultiModalConfig: # The model expects raw pixels and normalises on device; # overriding these per request would silently double-normalise. for key in ("do_normalize", "do_rescale"): - if key in inference_kwargs: + if inference_kwargs.get(key): logger.warning_once( "mm_device_do_normalize is enabled; ignoring " "per-request mm_processor_kwargs[%r].", diff --git a/vllm/config/parallel.py b/vllm/config/parallel.py index 7df8f4e0a4..37affbaf9e 100644 --- a/vllm/config/parallel.py +++ b/vllm/config/parallel.py @@ -5,13 +5,12 @@ import os import socket from collections.abc import Callable from datetime import timedelta -from typing import TYPE_CHECKING, Any, Literal, overload +from typing import TYPE_CHECKING, Any, Literal, Self, overload import regex as re import torch from pydantic import Field, field_validator, model_validator from torch.distributed import ProcessGroup, ReduceOp, Store -from typing_extensions import Self import vllm.envs as envs from vllm.config.fault_tolerance import FaultToleranceConfig diff --git a/vllm/config/profiler.py b/vllm/config/profiler.py index 3ade4bceed..0ad920e21e 100644 --- a/vllm/config/profiler.py +++ b/vllm/config/profiler.py @@ -2,10 +2,9 @@ # SPDX-FileCopyrightText: Copyright contributors to the vLLM project import os -from typing import Any, Literal +from typing import Any, Literal, Self from pydantic import Field, model_validator -from typing_extensions import Self from vllm.config.utils import config from vllm.logger import init_logger @@ -22,6 +21,34 @@ ProtonHook = Literal["triton"] ProtonOutputFormat = Literal["hatchet", "hatchet_msgpack", "chrome_trace"] +def validate_profile_prefix(prefix: str | None) -> str | None: + """Validate a trace filename prefix supplied for one profiling session.""" + if prefix is None: + return None + if ( + not 1 <= len(prefix) <= 128 + or not prefix.isascii() + or not prefix[0].isalnum() + or any(not (char.isalnum() or char in "._-") for char in prefix[1:]) + ): + raise ValueError( + "profile_prefix must be 1-128 ASCII characters, start with a letter " + "or number, and contain only letters, numbers, '.', '_', or '-'" + ) + return prefix + + +def validate_profile_iteration_bounds( + delay_iterations: int | None, + max_iterations: int | None, +) -> None: + """Validate iteration bounds supplied for one profiling session.""" + if delay_iterations is not None and delay_iterations < 0: + raise ValueError("delay_iterations must be greater than or equal to 0") + if max_iterations is not None and max_iterations < 0: + raise ValueError("max_iterations must be greater than or equal to 0") + + def _is_uri_path(path: str) -> bool: """Check if path is a URI (scheme://...), excluding Windows drive letters. @@ -136,6 +163,16 @@ class ProfilerConfig: Defaults to 0, meaning no limit. """ + @property + def should_profile_frontend(self) -> bool: + """Whether the frontend CPU profiler should be enabled.""" + activities = self.torch_profiler_activities + return ( + self.profiler == "torch" + and not self.ignore_frontend + and (activities is None or "CPU" in activities) + ) + warmup_iterations: int = Field(default=0, ge=0) """Number of warmup iterations for PyTorch profiler schedule. During warmup, the profiler runs but data is discarded. This helps reduce @@ -188,13 +225,7 @@ class ProfilerConfig: raise ValueError( "torch_profiler_activities must not contain duplicates" ) - records_cpu_activity = activities is None or "CPU" in activities - if ( - self.profiler == "torch" - and has_delay_or_limit - and not self.ignore_frontend - and records_cpu_activity - ): + if has_delay_or_limit and self.should_profile_frontend: logger.warning_once( "Using 'torch' profiler with delay_iterations or max_iterations " "while ignore_frontend is False may result in high overhead." diff --git a/vllm/config/quantization.py b/vllm/config/quantization.py index d950084726..face2c3a7b 100644 --- a/vllm/config/quantization.py +++ b/vllm/config/quantization.py @@ -27,6 +27,7 @@ from vllm.model_executor.layers.quantization.utils.quant_utils import ( kMxfp4Dynamic, kMxfp4Static, kMxfp8Dynamic, + kNvfp4DynamicToken, kNvfp4Static, ) @@ -40,6 +41,7 @@ QUANT_KEY_NAMES: dict[str, QuantKey] = { "fp8_per_block_dynamic": kFp8Dynamic128Sym, "mxfp8": kMxfp8Dynamic, "mxfp4": kMxfp4Dynamic, + "nvfp4_per_token": kNvfp4DynamicToken, "int8_per_channel_static": kInt8StaticChannelSym, # Load-time MXFP4-to-int4 for Kimi-K3 on gfx942. "int4_per_group_32": kInt4Static32, diff --git a/vllm/config/scheduler.py b/vllm/config/scheduler.py index 1fb94d3efd..0d10103ed4 100644 --- a/vllm/config/scheduler.py +++ b/vllm/config/scheduler.py @@ -3,10 +3,9 @@ from collections.abc import Callable from dataclasses import InitVar -from typing import TYPE_CHECKING, Any, ClassVar, Literal, cast +from typing import TYPE_CHECKING, Any, ClassVar, Literal, Self, cast from pydantic import Field, field_validator -from typing_extensions import Self from vllm.config.utils import config from vllm.logger import init_logger diff --git a/vllm/config/speculative.py b/vllm/config/speculative.py index 47ffb3b7f7..4c61ead82d 100644 --- a/vllm/config/speculative.py +++ b/vllm/config/speculative.py @@ -4,10 +4,9 @@ import copy import functools from collections.abc import Callable, Mapping -from typing import TYPE_CHECKING, Any, Literal, get_args +from typing import TYPE_CHECKING, Any, Literal, Self, get_args from pydantic import Field, SkipValidation, field_validator, model_validator -from typing_extensions import Self from vllm.config import LoadConfig from vllm.config.cache import CacheDType @@ -1970,6 +1969,9 @@ class SpeculativeConfig: def use_ngram_gpu(self) -> bool: return self.method == "ngram_gpu" + def use_ngram(self) -> bool: + return self.method in ("ngram", "ngram_gpu") + def use_multi_module_mtp(self) -> bool: if self.method != "mtp" or self.draft_model_config is None: return False diff --git a/vllm/config/structured_outputs.py b/vllm/config/structured_outputs.py index 059581e37e..4a13900b56 100644 --- a/vllm/config/structured_outputs.py +++ b/vllm/config/structured_outputs.py @@ -1,10 +1,9 @@ # SPDX-License-Identifier: Apache-2.0 # SPDX-FileCopyrightText: Copyright contributors to the vLLM project -from typing import Any, Literal +from typing import Any, Literal, Self from pydantic import model_validator -from typing_extensions import Self from vllm.config.utils import config from vllm.utils.hashing import safe_hash diff --git a/vllm/config/vllm.py b/vllm/config/vllm.py index a6c979400b..d584ea59d2 100644 --- a/vllm/config/vllm.py +++ b/vllm/config/vllm.py @@ -60,6 +60,7 @@ if TYPE_CHECKING: from transformers import PreTrainedConfig from vllm.model_executor.layers.quantization.base_config import QuantizationConfig + from vllm.sampling_params import BeamSearchParams, SamplingParams from vllm.v1.kv_cache_interface import KVCacheConfig else: PreTrainedConfig = Any @@ -151,12 +152,10 @@ class OptimizationLevel(IntEnum): PerformanceMode = Literal["balanced", "interactivity", "throughput"] IS_QUANTIZED = False -IS_DENSE = False # The optimizations that depend on these properties currently set to False # in all cases. # if model_config is not None: # IS_QUANTIZED = lambda c: c.model_config.is_quantized() -# IS_DENSE = lambda c: not c.model_config.is_model_moe() # See https://github.com/vllm-project/vllm/issues/25689. @@ -265,8 +264,6 @@ OPTIMIZATION_LEVEL_00 = { "fuse_act_quant": False, "fuse_allreduce_rms": False, "fuse_attn_quant": False, - "enable_sp": False, - "fuse_gemm_comms": False, "fuse_act_padding": False, "fuse_mla_dual_rms_norm": False, "fuse_rope_kvcache": False, @@ -288,8 +285,6 @@ OPTIMIZATION_LEVEL_01 = { "fuse_act_quant": enable_act_fusion, "fuse_allreduce_rms": False, "fuse_attn_quant": False, - "enable_sp": False, - "fuse_gemm_comms": False, "fuse_act_padding": enable_norm_pad_fusion, "fuse_mla_dual_rms_norm": enable_mla_dual_rms_norm_fusion, "fuse_rope_kvcache": False, @@ -311,8 +306,6 @@ OPTIMIZATION_LEVEL_02 = { "fuse_act_quant": enable_act_fusion, "fuse_allreduce_rms": enable_allreduce_rms_fusion, "fuse_attn_quant": IS_QUANTIZED, - "enable_sp": IS_DENSE, - "fuse_gemm_comms": IS_DENSE, "fuse_act_padding": enable_norm_pad_fusion, "fuse_mla_dual_rms_norm": enable_mla_dual_rms_norm_fusion, "fuse_rope_kvcache": enable_rope_kvcache_fusion, @@ -334,8 +327,6 @@ OPTIMIZATION_LEVEL_03 = { "fuse_act_quant": enable_act_fusion, "fuse_allreduce_rms": enable_allreduce_rms_fusion, "fuse_attn_quant": IS_QUANTIZED, - "enable_sp": IS_DENSE, - "fuse_gemm_comms": IS_DENSE, "fuse_act_padding": enable_norm_pad_fusion, "fuse_mla_dual_rms_norm": enable_mla_dual_rms_norm_fusion, "fuse_rope_kvcache": enable_rope_kvcache_fusion, @@ -647,13 +638,15 @@ class VllmConfig: speculative_config = self.speculative_config if speculative_config is None: return 0 - if speculative_config.use_dflash(): - # DFlash requires an extra lookahead slot since it uses in-fill-style - # decoding instead of standard next-token sampling, so it has a query - # for the last sampled token plus queries for each draft token. + dspark_fill_in = speculative_config.use_dspark() and not getattr( + speculative_config.draft_model_config.hf_config, "sample_from_anchor", True + ) + if speculative_config.use_dflash() or dspark_fill_in: + # Fill-in drafting uses a bonus query plus one query per draft token. + # DSpark's anchor-sampling layout does not need the extra slot. return self.num_speculative_tokens + 1 if speculative_config.use_eagle() or speculative_config.uses_draft_model(): - # DSpark (covered by use_eagle) drafts a block of num_speculative_tokens + # Anchor-sampling DSpark drafts a block of num_speculative_tokens # query tokens in which the anchor itself is the first prediction # position (no separate bonus query), so it needs exactly # num_speculative_tokens lookahead slots. @@ -716,7 +709,7 @@ class VllmConfig: and model_config.enable_sleep_mode and model_config.sleep_mode_backend == "cumem" and self.compilation_config.cudagraph_mode != CUDAGraphMode.NONE - and current_platform.is_cuda() + and current_platform.is_cuda_alike() ) @property @@ -832,9 +825,6 @@ class VllmConfig: or model.dtype != torch.bfloat16 or model.quantization is not None or not current_platform.is_cuda() - # Sequence parallelism / async TP are torch.compile passes. - or self.compilation_config.pass_config.enable_sp - or self.compilation_config.pass_config.fuse_gemm_comms ): return False family = bi._get_tuned_matmul_arch_family( @@ -1319,15 +1309,22 @@ class VllmConfig: if not self.use_v2_model_runner: raise ValueError("trace replay requires Model Runner V2") - def _check_watermarking_unsupported( + def _check_supports_watermarking( self, + config: "SamplingParams | BeamSearchParams | None" = None, *, - beam_search: bool = False, custom_sampler: bool = False, - ) -> None: + ) -> bool: watermark_config = getattr(self, "watermark_config", None) if watermark_config is None: - return + if config is not None and config.watermarking is True: + logger.warning_once( + "Watermarking is enabled for this request, but the engine has no " + "watermark configuration. This and subsequent requests will run " + "without watermarking.", + scope="global", + ) + return False if self.speculative_config is not None: speculative_config = self.speculative_config if speculative_config.draft_sample_method != "probabilistic": @@ -1382,12 +1379,44 @@ class VllmConfig: watermark_config.alpha, scope="global", ) - if beam_search: - raise ValueError("Beam search is not supported with watermarking.") if custom_sampler: raise ValueError( "Model-specific custom samplers are not supported with watermarking." ) + if config is None: + return True + if config.watermarking is False: + return False + + from vllm.sampling_params import BeamSearchParams, SamplingParams + + if isinstance(config, BeamSearchParams): + logger.warning_once( + "Watermarking is enabled, but beam search cannot be watermarked. " + "This and subsequent beam search requests will run without " + "watermarking.", + scope="global", + ) + return False + if not isinstance(config, SamplingParams): + raise TypeError(f"Unsupported watermarking config: {type(config).__name__}") + if config.trace_decode_token_ids is not None: + logger.warning_once( + "Watermarking is enabled, but trace replay cannot be watermarked. " + "This and subsequent trace replay requests will run without " + "watermarking.", + scope="global", + ) + return False + if config.temperature == 0: + logger.warning_once( + "Watermarking is enabled, but greedy decoding " + "(temperature=0) cannot be watermarked. This and subsequent " + "greedy requests will use ordinary greedy sampling.", + scope="global", + ) + return False + return True def _resolve_and_verify_engram_config(self) -> None: """Resolve defaults and validate n-gram embedding settings.""" @@ -1451,7 +1480,7 @@ class VllmConfig: self.try_verify_and_update_config() self._resolve_and_verify_engram_config() - self._check_watermarking_unsupported() + self._check_supports_watermarking() # Models may have supplied their own DCP defaults above; anything still # unset falls back to the stock ones. self.parallel_config.set_dcp_defaults() @@ -1735,26 +1764,6 @@ class VllmConfig: ) self.compilation_config.mode = CompilationMode.NONE - # TODO: This is a stopgap that turns sequence parallelism / async TP - # off under batch invariance. Make the sequence-parallel reduce-scatter - # path batch-invariant instead so they can stay enabled (#56370). - # Sequence parallelism / async TP rewrite all_reduce + rms_norm into a - # reduce-scatter whose reduction order depends on the batch, so they - # are not batch-invariant. Decide this before the breakable-CUDA-graph - # default below, which declines to auto-enable when they are on. - pass_config = self.compilation_config.pass_config - if envs.VLLM_BATCH_INVARIANT and ( - pass_config.enable_sp or pass_config.fuse_gemm_comms - ): - logger.warning_once( - "Disabling sequence parallelism and async TP " - "(pass_config.enable_sp / fuse_gemm_comms) when " - "VLLM_BATCH_INVARIANT is enabled: the reduce-scatter path " - "is not batch-invariant (see vllm-project/vllm#56370)." - ) - pass_config.enable_sp = False - pass_config.fuse_gemm_comms = False - breakable_cudagraph_enabled = self._maybe_enable_breakable_cudagraph() if not breakable_cudagraph_enabled and ( @@ -1843,6 +1852,12 @@ class VllmConfig: raise ValueError( "HiSparse does not support decode context parallelism." ) + if self.compilation_config.cudagraph_mode == CUDAGraphMode.FULL: + raise ValueError( + "HiSparse does not support cudagraph_mode=FULL; use " + "FULL_AND_PIECEWISE (the default), which captures FULL graphs " + "for decode batches." + ) if not self.scheduler_config.scheduler_reserve_full_isl: # Without it, async loads admitted against free host blocks can # each wait on host pages the others hold, and waiting requests @@ -1879,38 +1894,6 @@ class VllmConfig: breakable_cudagraph_enabled=breakable_cudagraph_enabled ) - # async tp is built on top of sequence parallelism and requires it. - pass_config = self.compilation_config.pass_config - if pass_config.fuse_gemm_comms: - pass_config.enable_sp = True - if pass_config.enable_sp: - if self.parallel_config.tensor_parallel_size == 1: - logger.warning_once("Sequence Parallelism requires TP>1, disabling") - pass_config.enable_sp = False - pass_config.fuse_gemm_comms = False - else: - if pass_config.sp_min_token_num is None: - from vllm.compilation.passes.fusion.sequence_parallelism import ( - get_sequence_parallelism_threshold, - ) - - tp_size = self.parallel_config.tensor_parallel_size - hidden_size = self.model_config.get_hidden_size() - assert isinstance(self.model_config.dtype, torch.dtype) - element_size = self.model_config.dtype.itemsize - pass_config.sp_min_token_num = get_sequence_parallelism_threshold( - hidden_size, tp_size, element_size - ) - - if pass_config.sp_min_token_num is None: - logger.warning_once( - "Model hidden_size too small for the SP " - "threshold heuristic, disabling. To force SP, " - "set pass_config.sp_min_token_num manually." - ) - pass_config.enable_sp = False - pass_config.fuse_gemm_comms = False - from vllm.utils.torch_utils import HAS_OPAQUE_TYPE if HAS_OPAQUE_TYPE: @@ -2143,30 +2126,6 @@ class VllmConfig: data_parallel_size=effective_dp_size, ) - if self.compilation_config.pass_config.enable_sp: - # With pipeline parallelism, native rms norm tracing errors due to - # incorrect residual shape. - # Use custom rms norm to unblock. In the future, - # the pass will operate on higher-level IR to avoid the issue. - # TODO: https://github.com/vllm-project/vllm/issues/27894 - if self.compilation_config.mode != CompilationMode.VLLM_COMPILE: - logger.warning_once( - "Sequence parallelism is enabled, but running in wrong " - "vllm compile mode: %s.", - self.compilation_config.mode, - ) - - if self.parallel_config.pipeline_parallel_size > 1: - if "-rms_norm" not in self.compilation_config.custom_ops: - self.compilation_config.custom_ops.append("+rms_norm") - else: - logger.warning_once( - "Sequence parallelism not supported with " - "native rms_norm when using %s, " - "this will likely lead to an error.", - "pipeline parallelism", - ) - # final check of cudagraph mode after all possible updates if current_platform.is_cuda_alike(): if ( @@ -2354,29 +2313,6 @@ class VllmConfig: # Log the custom passes that are enabled self.compilation_config.pass_config.log_enabled_passes() - def update_sizes_for_sequence_parallelism(self, possible_sizes: list) -> list: - # remove the sizes that not multiple of tp_size when - # enable sequence parallelism - removed_sizes = [ - size - for size in possible_sizes - if size % self.parallel_config.tensor_parallel_size != 0 - ] - if removed_sizes: - logger.warning( - "Batch sizes %s are removed because they are not " - "multiple of tp_size %d when " - "sequence parallelism is enabled", - removed_sizes, - self.parallel_config.tensor_parallel_size, - ) - - return [ - size - for size in possible_sizes - if size % self.parallel_config.tensor_parallel_size == 0 - ] - def _set_max_num_scheduled_tokens(self): """In most cases, the scheduler may schedule a batch with as many tokens as the worker is configured to handle. @@ -2624,16 +2560,6 @@ class VllmConfig: # de-duplicate and sort the sizes cudagraph_capture_sizes = sorted(set(cudagraph_capture_sizes)) - if ( - self.parallel_config.tensor_parallel_size > 1 - and self.compilation_config.pass_config.enable_sp - ): - # Sequence parallelism only captures TP-divisible sizes, so a - # wider non-divisible decode batch cannot be captured under SP. - cudagraph_capture_sizes = self.update_sizes_for_sequence_parallelism( - cudagraph_capture_sizes - ) - # user-specific compilation_config.max_cudagraph_capture_size get # truncated to valid_max_size when they are inconsistent. valid_max_size = ( @@ -2723,37 +2649,6 @@ class VllmConfig: "allreduce-rms fusion will be enabled for all num_tokens." ) - # Add the compile ranges for sequence parallelism - if compilation_config.pass_config.enable_sp: - pass_config = compilation_config.pass_config - - # Calculate min_token_num if not explicitly provided - # User override works regardless of hidden_size - if pass_config.sp_min_token_num is None: - from vllm.compilation.passes.fusion.sequence_parallelism import ( - get_sequence_parallelism_threshold, - ) - - tp_size = self.parallel_config.tensor_parallel_size - hidden_size = self.model_config.get_hidden_size() - assert isinstance(self.model_config.dtype, torch.dtype) - element_size = self.model_config.dtype.itemsize - pass_config.sp_min_token_num = get_sequence_parallelism_threshold( - hidden_size, tp_size, element_size - ) - - min_token_num = pass_config.sp_min_token_num - max_num_batched_tokens = self.scheduler_config.max_num_batched_tokens - if min_token_num is not None and ( - max_num_batched_tokens is not None - and min_token_num < max_num_batched_tokens - and min_token_num > 1 - ): - # Add endpoint at min_token_num - 1 to ensure SP applies - # starting from min_token_num - # This creates ranges: [1, min-1] (no SP), [min, max] (SP applies) - computed_compile_ranges_endpoints.append(min_token_num - 1) - if compilation_config.pass_config.fuse_rope_kvcache: max_token_num = ( compilation_config.pass_config.rope_kvcache_fusion_max_token_num @@ -3083,12 +2978,6 @@ class VllmConfig: unsupported: list[str] = [] speculative_config = self.speculative_config - if ( - self.compilation_config.pass_config.enable_sp - and self.parallel_config.tensor_parallel_size > 1 - ): - unsupported.append("sequence parallelism") - # V2 does not implement the external_launcher (torchrun) PP-output # broadcast that V1 uses to keep all ranks in sync (broadcast_pp_output). if ( @@ -3099,9 +2988,6 @@ class VllmConfig: if speculative_config is not None: if speculative_config.method in ( - # https://github.com/vllm-project/vllm/pull/40704 - "ngram", - "ngram_gpu", "suffix", "medusa", "mlp_speculator", @@ -3118,6 +3004,11 @@ class VllmConfig: ): unsupported.append("parallel drafting for EAGLE speculative decoding") + # The V2 draft-model speculator has no token mapping between the + # draft and target vocabularies (TLI is TBD in #47172). + if getattr(speculative_config, "use_heterogeneous_vocab", False): + unsupported.append("heterogeneous-vocabulary draft models") + if self.parallel_config.use_ubatching: unsupported.extend(self._get_dbo_unsupported_features()) diff --git a/vllm/config/watermarking.py b/vllm/config/watermarking.py index 8701c7b72e..4d559c29ce 100644 --- a/vllm/config/watermarking.py +++ b/vllm/config/watermarking.py @@ -2,10 +2,9 @@ # SPDX-FileCopyrightText: Copyright contributors to the vLLM project import hashlib -from typing import Literal +from typing import Literal, Self from pydantic import Field, model_validator -from typing_extensions import Self from vllm.config.utils import config from vllm.logger import init_logger diff --git a/vllm/device_allocator/cumem.py b/vllm/device_allocator/cumem.py index b85be08d28..a2e582f85b 100644 --- a/vllm/device_allocator/cumem.py +++ b/vllm/device_allocator/cumem.py @@ -98,7 +98,7 @@ class CuMemAllocator: offloaded to CPU memory, and the rest of the tensors will be discarded. When we call `wake_up`, all tensors that are previously offloaded will be loaded back to GPU memory, and the rest of the tensors will - have empty memory. + have empty memory (zeroed on ROCm). Why it needs to be a singleton? When allocated tensors are garbage collected, PyTorch will call @@ -331,7 +331,7 @@ class CuMemAllocator: def wake_up(self, tags: list[str] | None = None) -> None: """Wake up the allocator from sleep mode. All data that is previously offloaded will be loaded back to GPU - memory, and the rest of the data will have empty memory. + memory, and the rest will have empty memory (zeroed on ROCm). Args: tags: The deferrable tags (weights, kv_cache) to load back to GPU @@ -358,6 +358,10 @@ class CuMemAllocator: cpu_ptr = cpu_backup_tensor.data_ptr() libcudart.cudaMemcpy(ptr, cpu_ptr, size_in_bytes) data.cpu_backup_tensor = None + elif current_platform.is_rocm(): + # amdgpu <6.14 may return stale VRAM (#44972); drop with its support + libcudart.cudaMemset(ptr, 0, handle[1]) + torch.accelerator.synchronize() @contextmanager def use_memory_pool(self, tag: str | None = None): diff --git a/vllm/distributed/aux_output_connector/connector.py b/vllm/distributed/aux_output_connector/connector.py index bd8cbf4d18..c27e23ea4b 100644 --- a/vllm/distributed/aux_output_connector/connector.py +++ b/vllm/distributed/aux_output_connector/connector.py @@ -67,8 +67,12 @@ class AuxOutputSchedulerConnector: block_hashes_by_request[request_id] = packed self._sent_hash_counts[request_id] = len(request.block_hashes) assert request.sampling_params is not None + # A P/D prefiller may cut the prompt short of the requested start. scheduled_requests[request_id] = max( - request.sampling_params.routed_experts_prompt_start, + min( + request.sampling_params.routed_experts_prompt_start, + request.num_prompt_tokens, + ), 0 if request.num_output_tokens == 0 else request.num_tokens - 1, ) # A settled token can complete a hash block after the next async schedule diff --git a/vllm/distributed/device_communicators/aiter_custom_all_reduce.py b/vllm/distributed/device_communicators/aiter_custom_all_reduce.py index 6fbffcaaf7..9a5970c565 100644 --- a/vllm/distributed/device_communicators/aiter_custom_all_reduce.py +++ b/vllm/distributed/device_communicators/aiter_custom_all_reduce.py @@ -30,6 +30,8 @@ class AiterCustomAllreduce: group: ProcessGroup, device: int | str | torch.device, max_size: int | None = None, + *, + register_graph_buffers: bool = True, ): from aiter.dist.device_communicators.custom_all_reduce import ( CustomAllreduce as _AiterCustomAllreduce, @@ -38,7 +40,10 @@ class AiterCustomAllreduce: if max_size is None: max_size = self.MAX_SIZE - self._impl = _AiterCustomAllreduce(group, device, max_size=max_size) + kwargs = {} + if not register_graph_buffers: + kwargs["enable_register_for_capturing"] = False + self._impl = _AiterCustomAllreduce(group, device, max_size=max_size, **kwargs) @property def aiter_ca(self): diff --git a/vllm/distributed/device_communicators/all2all.py b/vllm/distributed/device_communicators/all2all.py index d3c48756ca..ae31c0b1d3 100644 --- a/vllm/distributed/device_communicators/all2all.py +++ b/vllm/distributed/device_communicators/all2all.py @@ -171,6 +171,42 @@ class AgRsAll2AllManager(All2AllManagerBase): hidden_states = dist_group.reduce_scatterv(hidden_states, dim=0, sizes=sizes) return hidden_states + def allocate_combine_input( + self, + shape: tuple[int, ...], + dtype: torch.dtype, + device: torch.device, + is_sequence_parallel: bool = False, + ) -> torch.Tensor | None: + dist_group = self._get_comm_group(is_sequence_parallel) + sizes = self._get_sizes(shape[0] // dist_group.world_size, dist_group) + if sum(sizes) != shape[0] or any(size != sizes[0] for size in sizes): + return None + device_communicator = dist_group.device_communicator + if device_communicator is None: + return None + return device_communicator.get_symmetric_memory_buffer( + "moe_ag_rs_combine", shape, dtype, device + ) + + def combine_into_output( + self, + hidden_states: torch.Tensor, + output: torch.Tensor, + is_sequence_parallel: bool = False, + ) -> torch.Tensor: + dist_group = self._get_comm_group(is_sequence_parallel) + sizes = self._get_sizes( + hidden_states.shape[0] // dist_group.world_size, + dist_group, + ) + return dist_group.reduce_scatterv_into_output( + hidden_states, + output, + dim=0, + sizes=sizes, + ) + def destroy(self): pass @@ -972,6 +1008,11 @@ class MoriAll2AllManager(All2AllManagerBase): self.handle_cache = Cache() torch._C._distributed_c10d._register_process_group("mori", cpu_group) + if get_current_vllm_config().kernel_config.moe_backend == "aiter_mega_moe": + # MegaMoEV2 places its dispatch/combine workspaces on the MoRI + # symmetric heap, which defaults to 2 GB. MegaMoEV2 requires > 4GB + heap_size = os.environ.setdefault("MORI_SHMEM_HEAP_SIZE", "8G") + logger.info_once("AITER MegaMoE: MORI_SHMEM_HEAP_SIZE=%s", heap_size) mori.shmem.shmem_torch_process_group_init("mori") def _make_all2all_kwargs( diff --git a/vllm/distributed/device_communicators/base_device_communicator.py b/vllm/distributed/device_communicators/base_device_communicator.py index f70bf09215..588163b846 100644 --- a/vllm/distributed/device_communicators/base_device_communicator.py +++ b/vllm/distributed/device_communicators/base_device_communicator.py @@ -157,6 +157,24 @@ class All2AllManagerBase: def combine(self, hidden_states: torch.Tensor, is_sequence_parallel: bool = False): raise NotImplementedError + def allocate_combine_input( + self, + shape: tuple[int, ...], + dtype: torch.dtype, + device: torch.device, + is_sequence_parallel: bool = False, + ) -> torch.Tensor | None: + return None + + def combine_into_output( + self, + hidden_states: torch.Tensor, + output: torch.Tensor, + is_sequence_parallel: bool = False, + ) -> torch.Tensor: + output.copy_(self.combine(hidden_states, is_sequence_parallel)) + return output + def destroy(self): pass @@ -315,6 +333,25 @@ class DeviceCommunicatorBase: ) -> torch.Tensor: raise NotImplementedError + def reduce_scatterv_into_output( + self, + input_: torch.Tensor, + output: torch.Tensor, + dim: int = -1, + sizes: list[int] | None = None, + ) -> torch.Tensor: + output.copy_(self.reduce_scatterv(input_, dim, sizes)) + return output + + def get_symmetric_memory_buffer( + self, + role: str, + shape: tuple[int, ...], + dtype: torch.dtype, + device: torch.device, + ) -> torch.Tensor | None: + return None + def gather( self, input_: torch.Tensor, dst: int = 0, dim: int = -1 ) -> torch.Tensor | None: @@ -417,5 +454,23 @@ class DeviceCommunicatorBase: """ return hidden_states + def allocate_combine_input( + self, + shape: tuple[int, ...], + dtype: torch.dtype, + device: torch.device, + is_sequence_parallel: bool = False, + ) -> torch.Tensor | None: + return None + + def combine_into_output( + self, + hidden_states: torch.Tensor, + output: torch.Tensor, + is_sequence_parallel: bool = False, + ) -> torch.Tensor: + output.copy_(self.combine(hidden_states, is_sequence_parallel)) + return output + def batch_isend_irecv(self, p2p_ops: list): raise NotImplementedError diff --git a/vllm/distributed/device_communicators/cuda_communicator.py b/vllm/distributed/device_communicators/cuda_communicator.py index 99ebcd717d..992d07e15d 100644 --- a/vllm/distributed/device_communicators/cuda_communicator.py +++ b/vllm/distributed/device_communicators/cuda_communicator.py @@ -16,6 +16,7 @@ from vllm.distributed.device_communicators.all_reduce_utils import ( from vllm.distributed.device_communicators.pynccl import register_nccl_symmetric_ops from vllm.distributed.device_communicators.pynccl_allocator import ( is_symmetric_memory_enabled, + is_symmetric_memory_tensor, ) from vllm.logger import init_logger from vllm.platforms import current_platform @@ -26,6 +27,10 @@ from .base_device_communicator import DeviceCommunicatorBase logger = init_logger(__name__) +# Since NCCL 2.29.2, ncclSymkPickKernel accepts registered input with +# non-registered output for ReduceScatter (ncclSymSendRegRecvNonreg). +NCCL_DIRECT_SYMM_RS_OUTPUT_MIN_VERSION = 22902 + class CudaCommunicator(DeviceCommunicatorBase): def __init__( @@ -146,6 +151,7 @@ class CudaCommunicator(DeviceCommunicatorBase): self.aiter_ar_comm = AiterCustomAllreduce( group=self.cpu_group, device=self.device, + register_graph_buffers=register, ) if use_custom_allreduce and self.aiter_ar_comm is None and self.world_size > 1: @@ -169,6 +175,7 @@ class CudaCommunicator(DeviceCommunicatorBase): self.aiter_ar_comm = AiterCustomAllreduce( group=self.cpu_group, device=self.device, + register_graph_buffers=register, ) if self.aiter_ar_comm.disabled: self.aiter_ar_comm = None @@ -505,13 +512,33 @@ class CudaCommunicator(DeviceCommunicatorBase): def reduce_scatterv( self, input_: torch.Tensor, dim: int = -1, sizes: list[int] | None = None ): + return self._reduce_scatterv(input_, dim, sizes) + + def reduce_scatterv_into_output( + self, + input_: torch.Tensor, + output: torch.Tensor, + dim: int = -1, + sizes: list[int] | None = None, + ) -> torch.Tensor: + return self._reduce_scatterv(input_, dim, sizes, output) + + def _reduce_scatterv( + self, + input_: torch.Tensor, + dim: int, + sizes: list[int] | None, + output: torch.Tensor | None = None, + ) -> torch.Tensor: world_size = self.world_size pynccl_comm = self.pynccl_comm assert pynccl_comm is not None + assert not pynccl_comm.disabled, "reduce_scatterv requires PyNccl" if dim < 0: # Convert negative dim to positive. dim += input_.dim() + sizes_are_explicit = sizes is not None # 'sizes' is not needed if all inputs in the same group have the same # shape if sizes is not None and all(s == sizes[0] for s in sizes): @@ -529,35 +556,63 @@ class CudaCommunicator(DeviceCommunicatorBase): assert input_tensor.shape[0] % world_size == 0 chunk_size = input_tensor.shape[0] // world_size output_shape = (chunk_size,) + input_tensor.shape[1:] + if output is not None: + assert dim == 0, "preallocated reduce-scatter output requires dim=0" + assert output.shape == output_shape + assert output.dtype == input_tensor.dtype + assert output.device == input_tensor.device + assert output.is_contiguous() if self._can_use_aiter_ag_rs(sizes): aiter_comm = self.aiter_ar_comm assert aiter_comm is not None if aiter_comm.should_custom_rs(input_tensor, dim=0): - output = torch.empty( - output_shape, dtype=input_tensor.dtype, device=input_tensor.device - ) + if output is None: + output = torch.empty( + output_shape, + dtype=input_tensor.dtype, + device=input_tensor.device, + ) aiter_comm.custom_reduce_scatter(input_tensor, output, dim=0) return output.movedim(0, dim).contiguous() # Symmetric memory is only used when all ranks have uniform sizes. # ncclCommWindowRegister is collective: asymmetric pool allocations # from variable per-rank sizes cause deadlocks. - use_symm_mem = sizes is None and should_nccl_symm_mem_ag_rs() - if use_symm_mem: - output = self._reduce_scatter_symm_mem(input_tensor) - else: - output = torch.empty( - output_shape, dtype=input_tensor.dtype, device=input_tensor.device + uniform_sizes = sizes is None or all(size == sizes[0] for size in sizes) + direct_output_supported = ( + output is None + or is_symmetric_memory_tensor(output) + or pynccl_comm.nccl_version >= NCCL_DIRECT_SYMM_RS_OUTPUT_MIN_VERSION + ) + use_symm_mem = ( + uniform_sizes + and direct_output_supported + and ( + output is None + or not sizes_are_explicit + or is_symmetric_memory_tensor(input_tensor) ) + and should_nccl_symm_mem_ag_rs() + ) + if use_symm_mem: + output = self._reduce_scatter_symm_mem(input_tensor, output) + else: + if output is None: + output = torch.empty( + output_shape, dtype=input_tensor.dtype, device=input_tensor.device + ) use_deterministic_rs = envs.VLLM_BATCH_INVARIANT and world_size > 2 if use_deterministic_rs: # Reduce to a fixed root (0) for determinism reduced = torch.empty_like(input_tensor) - sizes = sizes if sizes else [chunk_size] * world_size + scatter_sizes = sizes + if scatter_sizes is None: + scatter_sizes = [chunk_size] * world_size pynccl_comm.reduce(reduced, input_tensor, root=0) - pynccl_comm.scatter(output, reduced, sizes, root=0) - elif sizes is not None and sizes.count(sizes[0]) != len(sizes): + pynccl_comm.scatter(output, reduced, scatter_sizes, root=0) + elif not uniform_sizes: + assert sizes is not None pynccl_comm.reduce_scatterv(output, input_tensor, sizes=sizes) else: pynccl_comm.reduce_scatter(output, input_tensor) @@ -576,54 +631,64 @@ class CudaCommunicator(DeviceCommunicatorBase): Allocating a fresh symm tensor per collective pays the ``nccl_symm_mem_context`` snapshot + window-registration scan on every - call (~0.5 ms/RS+AG pair, dwarfing the NVLS transfer itself). Instead we - allocate once per ``(role, shape, dtype)``, register once, and reuse. + call (~0.5 ms/RS+AG pair, dwarfing the NVLS transfer itself). Instead, + each ``(role, shape[1:], dtype, device)`` keeps a geometric high-water + allocation and returns a leading slice. Superseded allocations remain + referenced because a captured CUDA graph may still use their pointers; + geometric growth bounds their total capacity to less than twice the + current allocation. - Safe for serial (eager) sequence parallelism: each collective's result - is consumed on the same stream before the next same-role collective - reuses the buffer. Distinct roles (e.g. ``rs_in`` vs ``ag_out``, both - full-size) get distinct buffers so a reduce-scatter input copy never - clobbers a still-live all-gather output. + Safe across serial MoE layers and eager sequence parallelism: the + producer and collective are ordered on the same stream before the next + same-role operation reuses the buffer. DBO microbatches use distinct + cache entries. Any future cross-layer communication overlap must also + use distinct roles. """ from vllm.distributed.device_communicators.pynccl_allocator import ( nccl_symm_mem_context, ) + from vllm.v1.worker.ubatching import dbo_current_ubatch_id pynccl_comm = self.pynccl_comm assert pynccl_comm is not None + assert shape, "symmetric scratch buffers require at least one dimension" cache = self.__dict__.setdefault("_symm_scratch_bufs", {}) - key = (role, tuple(shape), dtype) + key = (role, dbo_current_ubatch_id(), tuple(shape[1:]), dtype, device) buf = cache.get(key) - if buf is None: + requested_rows = shape[0] + if buf is None or buf.shape[0] < requested_rows: + capacity = ( + requested_rows if buf is None else max(requested_rows, 2 * buf.shape[0]) + ) with nccl_symm_mem_context(pynccl_comm): - buf = torch.empty(shape, dtype=dtype, device=device) + new_buf = torch.empty( + (capacity, *shape[1:]), dtype=dtype, device=device + ) + if buf is not None: + retired = self.__dict__.setdefault("_retired_symm_scratch_bufs", {}) + retired.setdefault(key, []).append(buf) + buf = new_buf cache[key] = buf - return buf + return buf[:requested_rows] def _reduce_scatter_symm_mem( self, input_tensor: torch.Tensor, + output: torch.Tensor | None = None, ) -> torch.Tensor: """ReduceScatter using NCCL symmetric memory (NVLS). - Only called for uniform-size reduce_scatter (variable sizes are - guarded out by the caller to avoid asymmetric ncclCommWindowRegister). - Uses persistent pre-registered scratch (see _get_symm_scratch). + The MoE reduce_scatterv path passes explicit uniform sizes and calls + this only with an already-registered input, so that path never stages. + Uniform calls without a caller-owned output retain their existing + opt-in behavior and stage ordinary input into registered scratch. + The output may be ordinary memory. """ - from vllm.distributed.device_communicators.pynccl_allocator import ( - is_symmetric_memory_tensor, - ) - pynccl_comm = self.pynccl_comm assert pynccl_comm is not None - chunk = input_tensor.shape[0] // self.world_size output_shape = (chunk,) + tuple(input_tensor.shape[1:]) - symm_output = self._get_symm_scratch( - "rs_out", output_shape, input_tensor.dtype, input_tensor.device - ) - # NVLS reduce-scatter (LDMC) requires the input in symmetric memory. if is_symmetric_memory_tensor(input_tensor): symm_input = input_tensor else: @@ -635,8 +700,31 @@ class CudaCommunicator(DeviceCommunicatorBase): ) symm_input.copy_(input_tensor) - pynccl_comm.reduce_scatter(symm_output, symm_input) - return symm_output + if output is None: + output = self._get_symm_scratch( + "rs_out", output_shape, input_tensor.dtype, input_tensor.device + ) + pynccl_comm.reduce_scatter(output, symm_input) + return output + + def get_symmetric_memory_buffer( + self, + role: str, + shape: tuple[int, ...], + dtype: torch.dtype, + device: torch.device, + ) -> torch.Tensor | None: + pynccl_comm = self.pynccl_comm + if ( + pynccl_comm is None + or pynccl_comm.disabled + or pynccl_comm.world_size == 1 + or pynccl_comm.nccl_version < NCCL_DIRECT_SYMM_RS_OUTPUT_MIN_VERSION + or not should_nccl_symm_mem_ag_rs() + ): + return None + output = self._get_symm_scratch(role, shape, dtype, device) + return output if is_symmetric_memory_tensor(output) else None def send(self, tensor: torch.Tensor, dst: int | None = None) -> None: """Sends a tensor to the destination rank in a blocking way.""" @@ -925,6 +1013,29 @@ class CudaCommunicator(DeviceCommunicatorBase): is_sequence_parallel, ) + def allocate_combine_input( + self, + shape: tuple[int, ...], + dtype: torch.dtype, + device: torch.device, + is_sequence_parallel: bool = False, + ) -> torch.Tensor | None: + assert self.all2all_manager is not None + return self.all2all_manager.allocate_combine_input( + shape, dtype, device, is_sequence_parallel + ) + + def combine_into_output( + self, + hidden_states: torch.Tensor, + output: torch.Tensor, + is_sequence_parallel: bool = False, + ) -> torch.Tensor: + assert self.all2all_manager is not None + return self.all2all_manager.combine_into_output( + hidden_states, output, is_sequence_parallel + ) + def batch_isend_irecv(self, p2p_ops: list): pynccl_comm = self.pynccl_comm if pynccl_comm is not None and not pynccl_comm.disabled: diff --git a/vllm/distributed/device_communicators/shm_broadcast.py b/vllm/distributed/device_communicators/shm_broadcast.py index bafc5728b8..da2288f77b 100644 --- a/vllm/distributed/device_communicators/shm_broadcast.py +++ b/vllm/distributed/device_communicators/shm_broadcast.py @@ -9,6 +9,7 @@ import shutil import sys import threading import time +import weakref from contextlib import contextmanager from dataclasses import dataclass, field from multiprocessing import shared_memory @@ -472,6 +473,12 @@ class Handle: remote_addr_ipv6: bool = False +def _close_zmq(context: zmq.Context, sockets: list[zmq.Socket]) -> None: + for socket in sockets: + socket.close(linger=0) + context.term() + + class MessageQueue: def __init__( self, @@ -561,9 +568,26 @@ class MessageQueue: remote_subscribe_addr=remote_subscribe_addr, remote_addr_ipv6=remote_addr_ipv6, ) + self._close_zmq_on_collect(context) logger.debug("vLLM message queue communication handle: %s", self.handle) + def _close_zmq_on_collect(self, context: zmq.Context) -> None: + # pyzmq's Context.__del__ can block forever in term() when the context + # and its sockets are collected in the same GC cycle, so the finalizer + # keeps them alive and closes the sockets before terminating. + sockets = [self.local_socket, self.remote_socket] + if self._spin_condition is not None: + sockets += [ + self._spin_condition.local_notify_socket, + self._spin_condition.read_cancel_socket, + self._spin_condition.write_cancel_socket, + ] + finalizer = weakref.finalize( + self, _close_zmq, context, [s for s in sockets if s is not None] + ) + finalizer.atexit = False # type: ignore[misc] + def export_handle(self) -> Handle: return self.handle @@ -613,6 +637,7 @@ class MessageQueue: self._spin_condition = None # type: ignore self.shutting_down = False + self._close_zmq_on_collect(context) return self def wait_until_ready(self): diff --git a/vllm/distributed/ec_transfer/ec_connector/cpu/scheduler/__init__.py b/vllm/distributed/ec_transfer/ec_connector/cpu/scheduler/__init__.py index f1d22a5714..65403980b9 100644 --- a/vllm/distributed/ec_transfer/ec_connector/cpu/scheduler/__init__.py +++ b/vllm/distributed/ec_transfer/ec_connector/cpu/scheduler/__init__.py @@ -115,7 +115,11 @@ class ECCPUScheduler: self._element_size: int = 0 self._ack_timeout_s: float = 0.0 if self._nixl_enabled: - self._setup_nixl(vllm_config) + try: + self._setup_nixl(vllm_config) + except Exception: + self.shutdown() + raise def _setup_nixl(self, vllm_config: "VllmConfig") -> None: # Lazy imports keep nixl/zmq off the gate-off path. @@ -165,35 +169,26 @@ class ECCPUScheduler: self._peer_host = envs.VLLM_EC_SIDE_CHANNEL_HOST self._peer_port = envs.VLLM_EC_SIDE_CHANNEL_PORT - # Registering the region with NIXL and binding the control sockets are - # the first side effects here. __init__ propagates a failure, so the - # caller never receives a scheduler it could shut down: unwind through - # the same teardown shutdown() uses. - try: - self._data = NixlDataTransport( - agent_name=engine_id, - base_ptr=self._region.blocks.data_ptr(), - num_blocks=self._region.num_blocks, - block_size_bytes=self._region.block_size_bytes, - total_size_bytes=self._region.num_blocks - * self._region.block_size_bytes, + self._data = NixlDataTransport( + agent_name=engine_id, + base_ptr=self._region.blocks.data_ptr(), + num_blocks=self._region.num_blocks, + block_size_bytes=self._region.block_size_bytes, + total_size_bytes=self._region.num_blocks * self._region.block_size_bytes, + ) + if self._is_producer: + assert self._peer_host is not None + assert self._peer_port is not None + self._producer_session = ProducerSession( + transport=ZmqServerTransport( + host=self._peer_host, port=self._peer_port + ), + data=self._data, + cache=self._cache, + compat_hash=self._compat_hash, ) - if self._is_producer: - assert self._peer_host is not None - assert self._peer_port is not None - self._producer_session = ProducerSession( - transport=ZmqServerTransport( - host=self._peer_host, port=self._peer_port - ), - data=self._data, - cache=self._cache, - compat_hash=self._compat_hash, - ) - if self._is_consumer: - self._transport = ZmqClientTransport() - except Exception: - self._teardown_nixl() - raise + if self._is_consumer: + self._transport = ZmqClientTransport() def has_cache_item(self, identifier: str) -> bool: if not self._is_consumer: diff --git a/vllm/distributed/elastic_ep/elastic_execute.py b/vllm/distributed/elastic_ep/elastic_execute.py index 10b5e88d29..182a4b9bb4 100644 --- a/vllm/distributed/elastic_ep/elastic_execute.py +++ b/vllm/distributed/elastic_ep/elastic_execute.py @@ -454,8 +454,8 @@ class ElasticEPScalingExecutor: torch.compiler.reset() with set_current_vllm_config(self.worker.vllm_config): reset_compile_wrapper(self.worker.model_runner.get_model()) - if speculator is not None: - reset_compile_wrapper(speculator.model) + if draft_model := getattr(speculator, "model", None): + reset_compile_wrapper(draft_model) gc.collect() torch.accelerator.synchronize() diff --git a/vllm/distributed/eplb/eplb_communicator.py b/vllm/distributed/eplb/eplb_communicator.py index 730c01a388..473058d525 100644 --- a/vllm/distributed/eplb/eplb_communicator.py +++ b/vllm/distributed/eplb/eplb_communicator.py @@ -120,8 +120,8 @@ class TorchDistNcclEplbCommunicator(EplbCommunicator): P2POp( torch.distributed.isend, tensor, - dst_rank, - self._ep_group, + group=self._ep_group, + group_peer=dst_rank, ) ) @@ -136,8 +136,8 @@ class TorchDistNcclEplbCommunicator(EplbCommunicator): P2POp( torch.distributed.irecv, tensor, - src_rank, - self._ep_group, + group=self._ep_group, + group_peer=src_rank, ) ) @@ -199,8 +199,8 @@ class TorchDistGlooStagedEplbCommunicator(EplbCommunicator): P2POp( torch.distributed.isend, cpu_tensor, - peer_rank, - self._cpu_group, + group=self._cpu_group, + group_peer=peer_rank, ) ) continue @@ -211,8 +211,8 @@ class TorchDistGlooStagedEplbCommunicator(EplbCommunicator): P2POp( torch.distributed.irecv, cpu_tensor, - peer_rank, - self._cpu_group, + group=self._cpu_group, + group_peer=peer_rank, ) ) recv_staging.append((tensor, cpu_tensor)) diff --git a/vllm/distributed/kv_transfer/kv_connector/utils.py b/vllm/distributed/kv_transfer/kv_connector/utils.py index a0ead27c2f..d314d6107a 100644 --- a/vllm/distributed/kv_transfer/kv_connector/utils.py +++ b/vllm/distributed/kv_transfer/kv_connector/utils.py @@ -256,8 +256,6 @@ def kv_postprocess_blksize_on_receive(cache, indices, block_size_ratio): """ blocks_to_update = cache.index_select(0, indices) - # use physical order - blocks_to_update = blocks_to_update.permute(0, 2, 1, 3) n_kv_heads, block_size, head_size = blocks_to_update.shape[1:] remote_block_size = block_size // block_size_ratio n_blocks = block_size_ratio @@ -267,56 +265,28 @@ def kv_postprocess_blksize_on_receive(cache, indices, block_size_ratio): .permute(0, 2, 1, 3, 4) .flatten(2, 3) ) - permuted_blocks = permuted_blocks.permute(0, 2, 1, 3) - cache.index_copy_(0, indices, permuted_blocks) - - -def kv_postprocess_layout_on_receive(cache, indices): - """Transforms the layout of received KV cache blocks to the local format. - - This method corrects layout mismatches from direct memory copies by - permuting the tensor dimensions. - - 4D cache: - - **Source Layout:** `[num_blocks, n_kv_head, block_size, head_dim]` - - **Target Layout:** `[num_blocks, block_size, n_kv_head, head_dim]` - 5D cache: - - **Source Layout:** `[num_blocks, kv_dim, n_kv_head, block_size, head_dim]` - - **Target Layout:** `[num_blocks, kv_dim, block_size, n_kv_head, head_dim]` - - Implementation: - - x = blocks_to_update.reshape(src_shape) # view local kv with sender layout - - permuted_blocks = x.permute(*inv_order) # transpose n_kv_heads, block_size - - cache.index_copy_(0, indices, permuted_blocks) # copy permuted kv back - - """ - blocks_to_update = cache.index_select(0, indices) - target_shape = list(blocks_to_update.shape) - target_shape[0] = -1 - inv_order = [0, 1, 3, 2, 4] if blocks_to_update.ndim == 5 else [0, 2, 1, 3] - src_shape = tuple(target_shape[i] for i in inv_order) - blocks_to_update = cache.index_select(0, indices) - permuted_blocks = blocks_to_update.reshape(src_shape).permute(*inv_order) cache.index_copy_(0, indices, permuted_blocks) def kv_postprocess_blksize_and_layout_on_receive(cache, indices, block_size_ratio): """Transforms the layout of received KV cache to the local block_size and LBHNC. - (Only works for local blocksize > remote blocksize) + (Only works for local blocksize >= remote blocksize) prefill is LBHNC, smaller block_size decode(local) is LBNHC, larger block_size """ blocks_to_update = cache.index_select(0, indices) - block_size, n_kv_heads, head_size = blocks_to_update.shape[1:] + n_kv_heads, block_size, head_size = blocks_to_update.shape[1:] remote_block_size = block_size // block_size_ratio n_blocks = block_size_ratio + # View the received bytes in memory order as the remote head-major blocks. permuted_blocks = ( - blocks_to_update.reshape(-1, n_blocks, n_kv_heads, remote_block_size, head_size) - .permute(0, 1, 3, 2, 4) - .flatten(1, 2) + blocks_to_update.transpose(1, 2) + .reshape(-1, n_blocks, n_kv_heads, remote_block_size, head_size) + .permute(0, 2, 1, 3, 4) + .flatten(2, 3) ) cache.index_copy_(0, indices, permuted_blocks) diff --git a/vllm/distributed/kv_transfer/kv_connector/v1/base.py b/vllm/distributed/kv_transfer/kv_connector/v1/base.py index 61de1802d5..71593f59e5 100644 --- a/vllm/distributed/kv_transfer/kv_connector/v1/base.py +++ b/vllm/distributed/kv_transfer/kv_connector/v1/base.py @@ -51,6 +51,7 @@ import torch from vllm.logger import init_logger from vllm.v1.attention.backend import AttentionMetadata from vllm.v1.core.sched.output import SchedulerOutput +from vllm.v1.metrics.cache_hit_source import CacheHitSource from vllm.v1.outputs import KVConnectorOutput if TYPE_CHECKING: @@ -179,6 +180,10 @@ class KVConnectorWorkerMetadata(ABC): class KVConnectorBase_V1(ABC): """Base class for KV connectors.""" + # Source label for this connector's external hits. Subclasses must set + # HOST/DISK/P2P or override get_external_cache_hit_sources(). + _cache_hit_source = CacheHitSource.EXTERNAL_UNSPECIFIED + @property def supports_divergent_local_hybrid_hits(self) -> bool: """Whether external hits can complete divergent local hybrid hits. @@ -527,6 +532,24 @@ class KVConnectorBase_V1(ABC): """ pass + def get_external_cache_hit_sources( + self, + request: "Request", + num_external_tokens: int, + ) -> dict[CacheHitSource, int]: + """Split ``num_external_tokens`` by the cache tier that supplied them. + + Called after :meth:`update_state_after_alloc`, so the load plan is + known. Counts must sum to ``num_external_tokens``; a mismatch is + reported as ``external_unspecified``. Only non-zero counts are + included. + + Default: all tokens under ``_cache_hit_source``. + """ + if num_external_tokens == 0: + return {} + return {self._cache_hit_source: num_external_tokens} + @abstractmethod def update_state_after_alloc( self, request: "Request", blocks: "KVCacheBlocks", num_external_tokens: int @@ -560,7 +583,11 @@ class KVConnectorBase_V1(ABC): """Build the connector metadata for this step. This function should NOT modify fields in the scheduler_output. - Also, calling this function will reset the state of the connector. + FIXME: one exception: + synchronous READ connectors may remove attention blocks they fully + overwrite this step from new_block_ids_to_zero. They must complete + those loads before the blocks are used and fail the step if a load fails. + Calling this function will reset the state of the connector. Args: scheduler_output (SchedulerOutput): the scheduler output object. diff --git a/vllm/distributed/kv_transfer/kv_connector/v1/hisparse/connector.py b/vllm/distributed/kv_transfer/kv_connector/v1/hisparse/connector.py index dabf9095db..7e8960f644 100644 --- a/vllm/distributed/kv_transfer/kv_connector/v1/hisparse/connector.py +++ b/vllm/distributed/kv_transfer/kv_connector/v1/hisparse/connector.py @@ -30,10 +30,12 @@ from vllm.distributed.kv_transfer.kv_connector.v1.metrics import ( PromMetric, PromMetricT, ) +from vllm.logger import init_logger from vllm.v1.attention.backend import AttentionMetadata from vllm.v1.core.kv_cache_utils import KVCacheBlockCopy from vllm.v1.core.sched.output import SchedulerOutput from vllm.v1.hisparse.coordinator import get_hisparse_coordinator +from vllm.v1.hisparse.layout import get_hisparse_steady_state_concurrency from vllm.v1.hisparse.types import SparseKVOffloadCommand, SparseKVRowMirror from vllm.v1.outputs import KVConnectorOutput @@ -44,6 +46,8 @@ if TYPE_CHECKING: from vllm.v1.kv_cache_interface import KVCacheConfig from vllm.v1.request import Request +logger = init_logger(__name__) + @dataclass class HiSparseConnectorMetadata(KVConnectorMetadata): @@ -99,6 +103,17 @@ class HiSparseConnectorScheduler: assert self.coordinator is None self.coordinator = coordinator + def get_kv_connector_stats(self) -> HiSparseKVConnectorStats | None: + if self.coordinator is None: + return None + host_pool = self.coordinator.get_host_block_pool() + assert host_pool is not None + stats = HiSparseKVConnectorStats() + stats.record_host_usage( + host_pool.get_usage(), len(self.coordinator.pending_spills) + ) + return stats + def build_connector_meta( self, scheduler_output: SchedulerOutput ) -> HiSparseConnectorMetadata: @@ -223,6 +238,16 @@ class HiSparseConnector(KVConnectorBase_V1, SupportsHMA): ), draft_kv_lookahead=vllm_config.num_lookahead_tokens, ) + max_model_len = vllm_config.model_config.max_model_len + steady_concurrency = get_hisparse_steady_state_concurrency( + vllm_config, kv_cache_config + ) + logger.info_once( + "HiSparse steady-state maximum concurrency for %s tokens per " + "request: %.2fx (running requests reading from host).", + f"{max_model_len:,}", + steady_concurrency, + ) elif role == KVConnectorRole.WORKER: self.connector_worker = HiSparseConnectorWorker( vllm_config, kv_cache_config @@ -263,9 +288,11 @@ class HiSparseConnector(KVConnectorBase_V1, SupportsHMA): self.connector_worker.reset_hot_state() def get_kv_connector_stats(self) -> KVConnectorStats | None: - if self.connector_worker is None: - return None - return self.connector_worker.get_kv_connector_stats() + if self.connector_worker is not None: + return self.connector_worker.get_kv_connector_stats() + if self.connector_scheduler is not None: + return self.connector_scheduler.get_kv_connector_stats() + return None @classmethod def build_kv_connector_stats( diff --git a/vllm/distributed/kv_transfer/kv_connector/v1/hisparse/stats.py b/vllm/distributed/kv_transfer/kv_connector/v1/hisparse/stats.py index f2abf51fc4..6f6cfced9e 100644 --- a/vllm/distributed/kv_transfer/kv_connector/v1/hisparse/stats.py +++ b/vllm/distributed/kv_transfer/kv_connector/v1/hisparse/stats.py @@ -14,14 +14,7 @@ from vllm.distributed.kv_transfer.kv_connector.v1.metrics import ( ) from vllm.v1.metrics.utils import create_metric_per_engine -_HISPARSE_COUNTERS: tuple[tuple[str, str], ...] = ( - ("cache_hits", "Number of HiSparse device hot-buffer hits."), - ("cache_misses", "Number of HiSparse device hot-buffer misses."), - ( - "host_to_device_bytes", - "Bytes transferred from host KV storage to HiSparse hot buffers.", - ), -) +_HISPARSE_LEVEL_KEYS = frozenset({"host_cache_usage_perc", "pending_page_transfers"}) @dataclass @@ -30,6 +23,8 @@ class HiSparseKVConnectorStats(KVConnectorStats): Each list entry is the delta recorded over one device counter snapshot interval, so a list can hold multiple snapshots per logging interval. + The host-tier gauges are level values, not deltas: aggregation keeps + only the most recent observation. """ def __post_init__(self): @@ -39,10 +34,12 @@ class HiSparseKVConnectorStats(KVConnectorStats): def reset(self): # Must be serializable - self.data: dict[str, list[int]] = { + self.data: dict[str, list[int | float]] = { "cache_hits": [], "cache_misses": [], "host_to_device_bytes": [], + "host_cache_usage_perc": [], + "pending_page_transfers": [], } def record_snapshot(self, hits: int, misses: int, host_to_device_bytes: int): @@ -50,21 +47,32 @@ class HiSparseKVConnectorStats(KVConnectorStats): self.data["cache_misses"].append(misses) self.data["host_to_device_bytes"].append(host_to_device_bytes) + def record_host_usage(self, usage: float, pending_page_transfers: int) -> None: + self.data["host_cache_usage_perc"].append(usage) + self.data["pending_page_transfers"].append(pending_page_transfers) + def aggregate(self, other: KVConnectorStats) -> KVConnectorStats: if not other.is_empty(): for k, v in other.data.items(): accumulator = self.data[k] assert isinstance(accumulator, list) - accumulator.extend(v) + if k in _HISPARSE_LEVEL_KEYS: + accumulator.extend(v[-1:]) + else: + accumulator.extend(v) return self def reduce(self) -> dict[str, int | float]: # Compute compact representative stats suitable for CLI logging. - return { + reduced: dict[str, int | float] = { "HiSparse hot-buffer hits": sum(self.data["cache_hits"]), "HiSparse hot-buffer misses": sum(self.data["cache_misses"]), "HiSparse host-to-device bytes": sum(self.data["host_to_device_bytes"]), } + if self.data["host_cache_usage_perc"]: + usage = self.data["host_cache_usage_perc"][-1] + reduced["HiSparse host KV cache usage %"] = round(100 * usage, 1) + return reduced def is_empty(self) -> bool: return all(len(values) == 0 for values in self.data.values()) @@ -80,18 +88,66 @@ class HiSparsePromMetrics(KVConnectorPromMetrics): ): super().__init__(vllm_config, metric_types, labelnames, per_engine_labelvalues) - self.hisparse_counters: dict[str, Any] = {} - for name, documentation in _HISPARSE_COUNTERS: - counter = self._counter_cls( - name=f"vllm:hisparse_{name}", - documentation=documentation, - labelnames=labelnames, - ) - self.hisparse_counters[name] = create_metric_per_engine( - counter, self.per_engine_labelvalues - ) + counter_cache_hits = self._counter_cls( + name="vllm:hisparse_cache_hits", + documentation="Number of HiSparse device hot-buffer hits.", + labelnames=labelnames, + ) + counter_cache_misses = self._counter_cls( + name="vllm:hisparse_cache_misses", + documentation="Number of HiSparse device hot-buffer misses.", + labelnames=labelnames, + ) + counter_host_to_device_bytes = self._counter_cls( + name="vllm:hisparse_host_to_device_bytes", + documentation=( + "Bytes transferred from host KV storage to HiSparse hot buffers." + ), + labelnames=labelnames, + ) + self.hisparse_counters: dict[str, Any] = { + "cache_hits": create_metric_per_engine( + counter_cache_hits, self.per_engine_labelvalues + ), + "cache_misses": create_metric_per_engine( + counter_cache_misses, self.per_engine_labelvalues + ), + "host_to_device_bytes": create_metric_per_engine( + counter_host_to_device_bytes, self.per_engine_labelvalues + ), + } + + gauge_host_cache_usage = self._gauge_cls( + name="vllm:hisparse_host_cache_usage_perc", + documentation=( + "HiSparse host KV-cache usage. 1 means 100 percent usage. " + "Evictable cached blocks count as free." + ), + multiprocess_mode="mostrecent", + labelnames=labelnames, + ) + gauge_pending_page_transfers = self._gauge_cls( + name="vllm:hisparse_pending_page_transfers", + documentation=( + "HiSparse page transfers (write-backs and restores) not yet completed." + ), + multiprocess_mode="mostrecent", + labelnames=labelnames, + ) + self.hisparse_gauges: dict[str, Any] = { + "host_cache_usage_perc": create_metric_per_engine( + gauge_host_cache_usage, self.per_engine_labelvalues + ), + "pending_page_transfers": create_metric_per_engine( + gauge_pending_page_transfers, self.per_engine_labelvalues + ), + } def observe(self, transfer_stats_data: dict[str, Any], engine_idx: int = 0): for name, counter in self.hisparse_counters.items(): for value in transfer_stats_data.get(name, []): counter[engine_idx].inc(value) + for name, gauge in self.hisparse_gauges.items(): + values = transfer_stats_data.get(name, []) + if values: + gauge[engine_idx].set(values[-1]) diff --git a/vllm/distributed/kv_transfer/kv_connector/v1/hisparse/worker.py b/vllm/distributed/kv_transfer/kv_connector/v1/hisparse/worker.py index bf6329b3bf..b9dd01f054 100644 --- a/vllm/distributed/kv_transfer/kv_connector/v1/hisparse/worker.py +++ b/vllm/distributed/kv_transfer/kv_connector/v1/hisparse/worker.py @@ -417,7 +417,6 @@ class HiSparseConnectorWorker: self._clear_forward_mirror_state() for handle in self.cache_handles: handle.all_context_pages_resident = metadata.all_context_pages_resident - handle.mirror_from_resident = True self._copy_host_blocks(metadata.host_block_copies, previous_host_write_event) transfers = ( metadata.command.page_transfers if metadata.command is not None else [] diff --git a/vllm/distributed/kv_transfer/kv_connector/v1/mooncake/mooncake_connector.py b/vllm/distributed/kv_transfer/kv_connector/v1/mooncake/mooncake_connector.py index 172d41efdb..9d5194f33e 100644 --- a/vllm/distributed/kv_transfer/kv_connector/v1/mooncake/mooncake_connector.py +++ b/vllm/distributed/kv_transfer/kv_connector/v1/mooncake/mooncake_connector.py @@ -67,6 +67,7 @@ from vllm.v1.kv_cache_interface import ( SlidingWindowMLASpec, SlidingWindowSpec, ) +from vllm.v1.metrics.cache_hit_source import CacheHitSource from vllm.v1.request import RequestStatus from vllm.v1.worker.block_table import BlockTable from vllm.v1.worker.utils import select_common_block_size @@ -700,6 +701,8 @@ class MooncakeConnectorMetadata(KVConnectorMetadata): class MooncakeConnector(KVConnectorBase_V1, SupportsHMA): + _cache_hit_source = CacheHitSource.P2P + def __init__( self, vllm_config: VllmConfig, @@ -894,6 +897,10 @@ class MooncakeConnectorScheduler: # GDN is represented as a MambaSpec in vLLM. This Mooncake MambaSpec # path is currently tested with GDN; Mamba2 is not validated yet. self._has_mamba = kv_cache_config.has_mamba_layers + self._bounded_replay = any( + g.kv_cache_spec.prefix_replay_tokens > 0 + for g in kv_cache_config.kv_cache_groups + ) # Requests that need to start recv/send. # New requests are added by update_state_after_alloc in @@ -938,8 +945,10 @@ class MooncakeConnectorScheduler: def _get_remote_prefill_token_count(self, num_prompt_tokens: int) -> int: """D-side only. Returns N-1 for Mamba models since the decoder - always recomputes the last token and must start from h(N-1).""" - if self._has_mamba and num_prompt_tokens > 1: + always recomputes the last token and must start from h(N-1), and for + DSV41 SWA bounded replay, whose window is not replayed after a P/D load + and must be laid out for that token.""" + if (self._has_mamba or self._bounded_replay) and num_prompt_tokens > 1: return num_prompt_tokens - 1 return num_prompt_tokens @@ -955,6 +964,9 @@ class MooncakeConnectorScheduler: params is not None and not params.get("_p_side_truncated") and request.num_prompt_tokens > 1 + # A request that skips reading the prefix cache (prompt logprobs) + # loads nothing on the decoder, which recomputes its whole prompt. + and not request.get_skip_reading_prefix_cache() ): if request.prompt_token_ids is not None: request.prompt_token_ids.pop() @@ -970,7 +982,11 @@ class MooncakeConnectorScheduler: def on_new_request(self, request: "Request") -> None: params = request.kv_transfer_params - if params is not None and params.get("do_remote_decode") and self._has_mamba: + if ( + params is not None + and params.get("do_remote_decode") + and (self._has_mamba or self._bounded_replay) + ): self._truncate_mamba_request_for_prefill(request) def get_num_new_matched_tokens( @@ -2422,9 +2438,30 @@ class MooncakeConnectorWorker: async def _connect_to_prefiller_bootstrap(self, remote_bootstrap_addr: str): url = remote_bootstrap_addr + "/query" + max_attempts = _BOOTSTRAP_MAX_ATTEMPTS try: - async with httpx.AsyncClient() as client: - response = await client.get(url) + async with httpx.AsyncClient( + timeout=envs.VLLM_MOONCAKE_CONNECTOR_TIMEOUT + ) as client: + retry_delay = 0.1 + for attempt in range(1, max_attempts + 1): + try: + response = await client.get(url) + break + except httpx.RequestError as e: + if attempt == max_attempts: + raise + logger.warning( + "Bootstrap query to %s failed on attempt %d/%d " + "(%s: %s); retrying in %.1f seconds", + remote_bootstrap_addr, + attempt, + max_attempts, + type(e).__name__, + e, + retry_delay, + ) + await asyncio.sleep(retry_delay) response.raise_for_status() data: dict = response.json() for _, dp_entry in data.items(): @@ -2439,8 +2476,9 @@ class MooncakeConnectorWorker: self._tp_size[remote_engine_id] = len(dp_entry["worker_addr"]) except Exception as e: logger.error( - "Failed to connect to bootstrap server %s: %s", + "Failed to connect to bootstrap server %s (%s): %s", remote_bootstrap_addr, + type(e).__name__, e, ) diff --git a/vllm/distributed/kv_transfer/kv_connector/v1/moriio/moriio_connector.py b/vllm/distributed/kv_transfer/kv_connector/v1/moriio/moriio_connector.py index 1dc6f8459a..082643778e 100644 --- a/vllm/distributed/kv_transfer/kv_connector/v1/moriio/moriio_connector.py +++ b/vllm/distributed/kv_transfer/kv_connector/v1/moriio/moriio_connector.py @@ -99,6 +99,7 @@ from vllm.v1.kv_cache_interface import ( SlidingWindowSpec, is_full_attention_spec, ) +from vllm.v1.metrics.cache_hit_source import CacheHitSource from vllm.v1.outputs import KVConnectorOutput from vllm.v1.request import RequestStatus @@ -222,6 +223,8 @@ def resolve_moriio_transfer_ack( class MoRIIOConnector(KVConnectorBase_V1, SupportsHMA): + _cache_hit_source = CacheHitSource.P2P + @property def supports_divergent_local_hybrid_hits(self) -> bool: # The READ path always transfers the recurrent-state slot, including @@ -578,12 +581,31 @@ class MoRIIOConnectorScheduler: else (0, self.block_size) for g in kv_cache_config.transfer_groups ] - # add 1 to conservatively account for boundary overlap eg window isn't fully - # aligned with blocks. + # +1 conservatively covers window/block boundary overlap. self.blocks_per_sw = [ cdiv(n_tokens, block_size) + 1 if n_tokens else 0 for n_tokens, block_size in sw_sizes_tokens ] + # Per-group block size over transfer_groups (matching blocks_per_sw and + # the connector's block payload), used to clamp each full-attention group + # to its prompt length on the WRITE save path. + self._group_block_sizes = [ + getattr(g.kv_cache_spec, "block_size", self.block_size) + for g in kv_cache_config.transfer_groups + ] + # The WRITE-mode prompt-block clamp only covers full-attention groups, so + # MoRIIO + speculative decoding is not yet safe on sliding-window / + # hybrid models. TODO: lift once SWA + spec is validated end-to-end. + if ( + self.mode == MoRIIOMode.WRITE + and vllm_config.speculative_config is not None + and any(bps != 0 for bps in self.blocks_per_sw) + ): + raise NotImplementedError( + "MoRIIO with speculative decoding is not yet supported for " + "sliding-window / hybrid models: the prompt-block clamp is only " + "validated for full-attention KV groups." + ) # In WRITE mode and chunked prefill, we perform the write after the last chunk. # Hence we need to track once we are in the last chunk. We do this by computing # len(block_ids[g]) * block_size[g] for a full attn group g, as hybrid groups @@ -747,6 +769,9 @@ class MoRIIOConnectorScheduler: num_prompt_tokens = request.num_prompt_tokens num_external_tokens = max(num_prompt_tokens - num_computed_tokens, 0) if self.mode == MoRIIOMode.WRITE: + params = request.kv_transfer_params or {} + if not params.get("do_remote_prefill"): + return 0, False # MoriiO in write mode, no remote prefill. Hybrid models never get # here: register_kv_caches refuses WRITE mode for them, so there is # no recurrent-state accounting to do. @@ -1175,6 +1200,45 @@ class MoRIIOConnectorScheduler: params["do_remote_prefill"] = False + def _clamp_to_prompt_blocks(self, req: "Request", block_ids: BlockIds) -> BlockIds: + """Drop trailing speculative-lookahead blocks the consumer never + allocated, keeping each full-attention group's prompt blocks. + + Args: + req: Request being saved. + block_ids: Per-KV-cache-group local block ids. + + Returns: + ``block_ids`` with every full-attention group truncated to + ``ceil(num_prompt_tokens / block_size)`` leading blocks; + sliding-window groups are left unchanged. + + """ + clamped = list(block_ids) + changed = False + for g, group_blocks in enumerate(block_ids): + if self.blocks_per_sw[g] != 0: + continue + num_prompt_blocks = math.ceil( + req.num_prompt_tokens / self._group_block_sizes[g] + ) + if len(group_blocks) > num_prompt_blocks: + clamped[g] = group_blocks[:num_prompt_blocks] + changed = True + return clamped if changed else block_ids + + def _is_final_prefill_chunk( + self, req: "Request", scheduler_output: SchedulerOutput + ) -> bool: + """Whether this step completes the prompt. + + ``build_connector_meta`` runs before ``_update_after_schedule``, so + ``num_computed_tokens`` excludes the current chunk; add this step's + scheduled tokens. Token progress is lookahead-safe unlike block counts. + """ + num_scheduled = scheduler_output.num_scheduled_tokens.get(req.request_id, 0) + return req.num_computed_tokens + num_scheduled >= req.num_prompt_tokens + def build_connector_meta( self, scheduler_output: SchedulerOutput, @@ -1188,38 +1252,36 @@ class MoRIIOConnectorScheduler: # It places the request metadata into the saving queue. for i, req_id in enumerate(scheduler_output.scheduled_cached_reqs.req_ids): + # Non-disagg requests aren't registered here; skip them + # instead of KeyError-ing. + if req_id not in self._reqs_need_pending_save: + continue + req, existing_blocks = self._reqs_need_pending_save[req_id] + # The final chunk may allocate no new blocks (the previous chunk + # reserved speculative-lookahead slack), so new_block_ids is None; + # still fall through to the final-chunk check below. new_block_ids = scheduler_output.scheduled_cached_reqs.new_block_ids[i] - if new_block_ids is not None: - # A non-disagg request (no kv_transfer_params, e.g. smoke - # test) is never registered in _reqs_need_pending_save; - # indexing it unconditionally would KeyError and crash the - # EngineCore. Skip it silently. - if req_id not in self._reqs_need_pending_save: - continue - req, existing_blocks = self._reqs_need_pending_save[req_id] - updated_blocks = [ + existing_blocks = [ existing_blocks[g] + new_block_ids[g] for g in range(len(new_block_ids)) ] - self._reqs_need_pending_save[req_id] = (req, updated_blocks) - saved_tokens = ( - len(updated_blocks[self._full_attn_group_idx]) - * self._full_attn_block_size - ) - if saved_tokens >= req.num_prompt_tokens: - # Final chunk: live kv_transfer_params may be cleared, - # so prefer the snapshot from update_state_after_alloc. - kv_params = self._req_kv_params.pop( - req_id, req.kv_transfer_params or {} - ) - meta.add_new_req( - request_id=req_id, - local_block_ids=self._reqs_need_pending_save[req_id][1], - kv_transfer_params=kv_params, - write_mode=True, - ) - del self._reqs_need_pending_save[req_id] + self._reqs_need_pending_save[req_id] = (req, existing_blocks) + if not self._is_final_prefill_chunk(req, scheduler_output): + continue + # Live kv_transfer_params may be cleared; prefer the + # update_state_after_alloc snapshot. + kv_params = self._req_kv_params.pop( + req_id, req.kv_transfer_params or {} + ) + save_block_ids = self._clamp_to_prompt_blocks(req, existing_blocks) + meta.add_new_req( + request_id=req_id, + local_block_ids=save_block_ids, + kv_transfer_params=kv_params, + write_mode=True, + ) + del self._reqs_need_pending_save[req_id] # Loop through scheduled reqs and convert to ReqMeta. for req_id, (req, block_ids) in self._reqs_need_recv.items(): @@ -1229,19 +1291,34 @@ class MoRIIOConnectorScheduler: local_block_ids=block_ids, kv_transfer_params=kv_params, ) + if ( + self.mode == MoRIIOMode.READ + and self._has_mamba + and scheduler_output.new_block_ids_to_zero + ): + # Hybrid models zero recycled attention pages that held Mamba state. + # Host-submitted READs overwrite these pages and can race zeroing. + # Hybrid READ metadata puts the aligned attention pages first. + read_dst_block_ids = { + b + for _, block_ids in self._reqs_need_recv.values() + for b in block_ids[0] + } + scheduler_output.new_block_ids_to_zero = [ + b + for b in scheduler_output.new_block_ids_to_zero + if b not in read_dst_block_ids + ] for req_id, (req, block_ids) in self._reqs_need_save.items(): kv_params = self._req_kv_params.get(req_id, req.kv_transfer_params or {}) - tokens_covered = ( - len(block_ids[self._full_attn_group_idx]) * self._full_attn_block_size - ) - if req.num_prompt_tokens > tokens_covered: - # not last chunk prefill + if not self._is_final_prefill_chunk(req, scheduler_output): self._reqs_need_pending_save[req_id] = (req, block_ids) continue + save_block_ids = self._clamp_to_prompt_blocks(req, block_ids) meta.add_new_req( request_id=req_id, - local_block_ids=block_ids, + local_block_ids=save_block_ids, kv_transfer_params=kv_params, write_mode=True, ) diff --git a/vllm/distributed/kv_transfer/kv_connector/v1/moriio/moriio_layout.py b/vllm/distributed/kv_transfer/kv_connector/v1/moriio/moriio_layout.py index b00692dc51..8c55837dcd 100644 --- a/vllm/distributed/kv_transfer/kv_connector/v1/moriio/moriio_layout.py +++ b/vllm/distributed/kv_transfer/kv_connector/v1/moriio/moriio_layout.py @@ -377,10 +377,8 @@ def iter_layer_registration_regions( ) region_len = geometry.num_blocks * geometry.regions_per_block * geometry.block_len if geometry.regions_per_block == 1: - # With padded or interleaved pages the block stride exceeds the - # meaningful block_len; register the strided span. The span ends with - # the last block's meaningful bytes, not a whole stride, so it never - # runs past the backing allocation of a strided layer view. + # Padded/interleaved pages stride past block_len: register the strided + # span, ending at the last block's bytes so it stays in-allocation. block_stride_bytes = geometry.block_stride * kv_cache.element_size() region_len = max( region_len, @@ -431,12 +429,8 @@ def compute_block_transfer_offsets( [list[int], list[int], list[int]], tuple[list[int], list[int], list[int]] ] = merge_contiguous_offsets, ) -> tuple[list[int], list[int], list[int]]: - # A shorter (or empty) local list is the READ-mode "drop the transfer, just - # free the prefill blocks" case (full-prefix-hit / aborted-before-scheduled): - # decode pulls fewer blocks than the prefill holds. The zip loop below pairs - # local[i]<->remote[i] and sizes by len(local), so a short local transfers - # only what decode allocated and an empty local is a no-op. A longer local - # list is a genuine bug and still fails loudly. + # A shorter/empty local list transfers only what decode allocated (READ-mode + # partial or full-prefix-hit); a longer local list is a genuine bug. if len(local_block_ids) > len(remote_block_ids): raise ValueError( "local_block_ids longer than remote_block_ids: " diff --git a/vllm/distributed/kv_transfer/kv_connector/v1/multi_connector.py b/vllm/distributed/kv_transfer/kv_connector/v1/multi_connector.py index 752acb01ec..1ded62adda 100644 --- a/vllm/distributed/kv_transfer/kv_connector/v1/multi_connector.py +++ b/vllm/distributed/kv_transfer/kv_connector/v1/multi_connector.py @@ -31,6 +31,7 @@ from vllm.distributed.kv_transfer.kv_connector.v1.metrics import ( from vllm.logger import init_logger from vllm.v1.attention.backend import AttentionMetadata from vllm.v1.core.sched.output import SchedulerOutput +from vllm.v1.metrics.cache_hit_source import CacheHitSource from vllm.v1.outputs import KVConnectorOutput if TYPE_CHECKING: @@ -451,6 +452,18 @@ class MultiConnector(KVConnectorBase_V1, SupportsHMA): to_return = (toks, load_async) return to_return + def get_external_cache_hit_sources( + self, + request: "Request", + num_external_tokens: int, + ) -> dict[CacheHitSource, int]: + chosen_connector = self._requests_to_connector.get(request.request_id) + if chosen_connector is None: + return super().get_external_cache_hit_sources(request, num_external_tokens) + return self._connectors[chosen_connector].get_external_cache_hit_sources( + request, num_external_tokens + ) + def update_state_after_alloc( self, request: "Request", blocks: "KVCacheBlocks", num_external_tokens: int ): diff --git a/vllm/distributed/kv_transfer/kv_connector/v1/nixl/base_scheduler.py b/vllm/distributed/kv_transfer/kv_connector/v1/nixl/base_scheduler.py index 92f0cbc58a..55ee98d465 100644 --- a/vllm/distributed/kv_transfer/kv_connector/v1/nixl/base_scheduler.py +++ b/vllm/distributed/kv_transfer/kv_connector/v1/nixl/base_scheduler.py @@ -101,6 +101,10 @@ class NixlBaseConnectorScheduler: ) ) self._has_mamba = kv_cache_config.has_mamba_layers + self._bounded_replay = any( + g.kv_cache_spec.prefix_replay_tokens > 0 + for g in kv_cache_config.kv_cache_groups + ) logger.info("Initializing NIXL Scheduler %s", engine_id) if vllm_config.scheduler_config.disable_hybrid_kv_cache_manager: @@ -376,14 +380,16 @@ class NixlBaseConnectorScheduler: """Trailing prompt tokens the prefiller must not compute; the decoder recomputes them locally. - Mamba needs h(N-1) so the decoder can derive h(N) itself. Multi-module + Mamba needs h(N-1) so the decoder can derive h(N) itself. DSV41's sliding + window under bounded replay is not replayed after a P/D load, so it must + be laid out for the token the decoder recomputes. Multi-module MTP needs to keep its whole lookahead window off of the prefiller, which would otherwise embed the unverified drafts in the MTP layer's KV cache. The decoder would never rebuild them, because the update is sized by the rejection count, which is zero for the first decode. """ return max( - 1 if self._has_mamba else 0, + 1 if self._has_mamba or self._bounded_replay else 0, self.vllm_config.num_prefill_lookahead_tokens - 1, ) @@ -412,6 +418,9 @@ class NixlBaseConnectorScheduler: # Guard against repeated truncation after preemption/reschedule. and not params.get("_p_side_truncated") and request.num_prompt_tokens > backoff + # A request that skips reading the prefix cache (prompt logprobs) + # loads nothing on the decoder, which recomputes its whole prompt. + and not request.get_skip_reading_prefix_cache() ): if request.prompt_token_ids is not None: del request.prompt_token_ids[-backoff:] diff --git a/vllm/distributed/kv_transfer/kv_connector/v1/nixl/base_worker.py b/vllm/distributed/kv_transfer/kv_connector/v1/nixl/base_worker.py index 0eaa1aadf3..38a656954e 100644 --- a/vllm/distributed/kv_transfer/kv_connector/v1/nixl/base_worker.py +++ b/vllm/distributed/kv_transfer/kv_connector/v1/nixl/base_worker.py @@ -14,7 +14,6 @@ import uuid from collections import defaultdict from collections.abc import Iterator from concurrent.futures import Future, ThreadPoolExecutor -from functools import cached_property from typing import TYPE_CHECKING, Any, cast import msgspec @@ -31,7 +30,6 @@ from vllm.distributed.kv_transfer.kv_connector.utils import ( get_current_attn_backends, kv_postprocess_blksize_and_layout_on_receive, kv_postprocess_blksize_on_receive, - kv_postprocess_layout_on_receive, ) from vllm.distributed.kv_transfer.kv_connector.v1.base import ( CopyBlocksOp, @@ -101,6 +99,11 @@ logger = init_logger(__name__) _SHARED_REGION_GROUP_ID = -1 +# Re-handshake delay bounds for a pull peer whose local metadata keeps getting +# invalidated, eg due to a persistent RDMA fault. +_INVALID_PEER_MIN_BACKOFF_S = 1.0 +_INVALID_PEER_MAX_BACKOFF_S = 60.0 + def _region_sort_key(layer_name: str) -> tuple[tuple[int, int | str], ...]: """Sort transfer regions in model-layer order, then by cache name.""" @@ -340,7 +343,7 @@ class NixlBaseConnectorWorker: ) # Per-FA-descriptor replicate flag, in _build_fa_local emission order. - fa_desc_replicated = self._fa_desc_replicated(num_fa_descs) + fa_desc_replicated = self._fa_desc_replicated(num_fa_descs, block_size_ratio) sharded_desc_end = len(src_blocks_data) - ( self._logical_num_blocks if self._ple_region_index is not None else 0 ) @@ -383,19 +386,22 @@ class NixlBaseConnectorWorker: """ return tp_ratio < 0 and (not self.use_mla or len(plan.all_source_ranks) > 1) - def _fa_desc_replicated(self, num_fa_descs: int) -> list[bool]: - """Per-FA-descriptor replicate flag, in _build_fa_local emission order - (region-major; one desc per block, with K/V packed). Length ``num_fa_descs``. + def _fa_desc_replicated( + self, num_fa_descs: int, block_size_ratio: int = 1 + ) -> list[bool]: + """Per-FA-descriptor replicate flag, in _build_fa_local emission order: + region-major, ``region_num_blocks[i] * block_size_ratio`` descs per + region. Length ``num_fa_descs``. """ assert self.transfer_topo is not None n_regions = len(self.block_len_per_layer) if n_regions == 0 or self.num_regions == 0: return [False] * num_fa_descs - nblk = num_fa_descs // self.num_regions + region_indices = self._transfer_layer_region_indices or range(n_regions) flags: list[bool] = [] - for i in range(n_regions): - replicated = self._is_region_replicated(i) - flags.extend([replicated] * nblk) + for i in region_indices: + num_descs = self.region_num_blocks[i] * block_size_ratio + flags.extend([self._is_region_replicated(i)] * num_descs) assert len(flags) == num_fa_descs, ( f"FA desc flags {len(flags)} != num_fa_descs {num_fa_descs}" ) @@ -682,6 +688,12 @@ class NixlBaseConnectorWorker: self._remote_agents: dict[EngineId, dict[tuple[int, int], str]] = defaultdict( dict ) + # Failed reads need a local metadata probe; invalid peers wait for drain. + self._failed_remote_engines: set[EngineId] = set() + self._invalid_remote_engines: set[EngineId] = set() + # Invalid engine -> (backoff, earliest release time). Kept across release + # so that repeat invalidations back off. + self._invalid_engine_backoff: dict[EngineId, tuple[float, float]] = {} # Map of engine_id -> clock offset. self._engine_clock_offset: dict[EngineId, float] = {} @@ -775,9 +787,12 @@ class NixlBaseConnectorWorker: self.region_names: list[str] = [] self.region_num_blocks: list[int] = [] self._mixed_mem_types = False + # True on TP ranks that share a HiSparse host pool with TP rank 0 and + # leave its NIXL registration and DRAM reads to rank 0. + self._skip_dram_xfer = False self._desc_is_dram_by_block_size: dict[int, np.ndarray] = {} self._desc_pos_by_block_size: dict[int, np.ndarray] = {} - self._dram_src_handles_by_block_size: dict[int, int] = {} + self._dram_src_handles_by_block_size: dict[int, int | None] = {} # PP>1 (push mode): this worker holds a contiguous layer slice and # transfers into the matching sub-range of a PP=1 remote's regions. @@ -814,7 +829,7 @@ class NixlBaseConnectorWorker: # Populated dynamically during handshake based on remote configuration. # Per-source split handles, keyed by (tp_ratio, remote_block_size). self.src_xfer_handles_by_tp_ratio: dict[tuple[int, int], list[int]] = {} - self._dram_src_handles_by_tp_ratio: dict[tuple[int, int], list[int]] = {} + self._dram_src_handles_by_tp_ratio: dict[tuple[int, int], list[int | None]] = {} # Map of engine_id -> {tp_rank: nixl_prepped_dlist_handle (int)}. self.dst_xfer_side_handles = defaultdict[EngineId, dict[int, int]](dict) @@ -918,6 +933,9 @@ class NixlBaseConnectorWorker: # page it overlays, so its regions cannot be recovered from spec type. self._scratch_region_indices = list[int]() self._ple_region_index: int | None = None + # Page length of the PLE short-conv state; its region may be shared with + # pages of a different (smaller) size, so it cannot use the region's. + self._ple_block_len: int | None = None # Enable different block lengths for different layers *only* when MLA is used. # This is not used for SSM layers, which use the counterpart `mamba_ssm_size`. @@ -1447,6 +1465,7 @@ class NixlBaseConnectorWorker: seen_base_addresses: list[int] = [] self._ssm_region_indices = [] self._scratch_region_indices = [] + self._ple_block_len = None self._ple_region_index = None packed_storage = _share_storage_and_block_stride(list(xfer_buffers.values())) @@ -1734,6 +1753,7 @@ class NixlBaseConnectorWorker: assert self._is_csa_linear assert self._ple_region_index in (None, region_index) self._ple_region_index = region_index + self._ple_block_len = block_len elif region_index not in self._ssm_region_indices: self._ssm_region_indices.append(region_index) elif ( @@ -1804,12 +1824,15 @@ class NixlBaseConnectorWorker: self.num_descs = sum(xfer_region_num_blocks) self._mixed_mem_types = len(set(region_mem_types)) > 1 + self._skip_dram_xfer = self._leaves_dram_xfer_to_rank0() if self._mixed_mem_types: assert self.use_mla and not self._has_mamba, ( "Mixed-device KV registration is only supported for MLA " "models without Mamba layers." ) for mem_type in sorted(set(region_mem_types)): + if mem_type == "DRAM" and self._skip_dram_xfer: + continue ranges_for_mem_type = [ (start, end - start, device_id, "") for (_, cache_mem_type), ( @@ -1865,6 +1888,7 @@ class NixlBaseConnectorWorker: else self.host_buffer_kv_cache_layout, block_size=self.block_size, ssm_sizes=self._mamba_ssm_size, + ple_block_len=self._ple_block_len, attn_backend_name=self.backend_name, physical_blocks_per_logical_kv_block=( self._physical_blocks_per_logical_kv_block @@ -1886,6 +1910,19 @@ class NixlBaseConnectorWorker: agent_metadata_bytes=encoder.encode(agent_metadata), ) + def _ple_page_len(self) -> int: + """Transfer length of one PLE state page. + + The PLE page shares its region with the first page of other cache + groups (block-outer layout), whose block_len may be smaller; the + descriptors must cover the whole PLE page or part of the conv state + never reaches the decode side.""" + assert self._ple_region_index is not None + assert self._ple_block_len is not None, ( + "PLE region registered without its page length" + ) + return self._ple_block_len + def _build_mamba_local(self, base_addresses: list[int]) -> np.ndarray: """Build desc regions (conv sub-projections + ssm) per layer for local mamba blocks with DS conv layout, as an Nx3 uint64 array. @@ -1940,7 +1977,7 @@ class NixlBaseConnectorWorker: parts.append(self._stack_descs(blk_addrs + conv_size, ssm_size, device_id)) if (region_index := self._ple_region_index) is not None: - block_len = self.block_len_per_layer[region_index] * physical_per_logical + block_len = self._ple_page_len() * physical_per_logical block_stride = ( self.block_stride_per_layer[region_index] * physical_per_logical ) @@ -1999,11 +2036,12 @@ class NixlBaseConnectorWorker: if (region_index := self._ple_region_index) is not None: local_block_len = ( - self.block_len_per_layer[region_index] - * self._physical_blocks_per_logical_kv_block + self._ple_page_len() * self._physical_blocks_per_logical_kv_block ) + # Same NIXL_CONNECTOR_VERSION on both sides, so the field is set. + assert nixl_agent_meta.ple_block_len is not None remote_block_len = ( - nixl_agent_meta.block_lens[region_index] * remote_physical_per_logical + nixl_agent_meta.ple_block_len * remote_physical_per_logical ) if local_block_len != remote_block_len: raise ValueError( @@ -2174,18 +2212,16 @@ class NixlBaseConnectorWorker: self._desc_pos_by_block_size[block_size] = desc_pos # DRAM descriptors are registered under CPU device 0. - blocks_data = [ - (addr, length, 0) if is_dram else (addr, length, dev) - for (addr, length, dev), is_dram in zip( - blocks_data, desc_is_dram, strict=True + blocks_data[desc_is_dram, 2] = 0 + dram_blocks = blocks_data[dram_idx] + vram_blocks = blocks_data[vram_idx] + if self._skip_dram_xfer: + self._dram_src_handles_by_block_size[block_size] = None + else: + dram_descs = self.nixl_wrapper.get_xfer_descs(dram_blocks, "DRAM") + self._dram_src_handles_by_block_size[block_size] = ( + self.nixl_wrapper.prep_xfer_dlist("NIXL_INIT_AGENT", dram_descs) ) - ] - dram_blocks = [blocks_data[i] for i in dram_idx] - vram_blocks = [blocks_data[i] for i in vram_idx] - dram_descs = self.nixl_wrapper.get_xfer_descs(dram_blocks, "DRAM") - self._dram_src_handles_by_block_size[block_size] = ( - self.nixl_wrapper.prep_xfer_dlist("NIXL_INIT_AGENT", dram_descs) - ) descs = self.nixl_wrapper.get_xfer_descs(vram_blocks, self.nixl_memory_type) return ( self.nixl_wrapper.prep_xfer_dlist("NIXL_INIT_AGENT", descs), @@ -2414,12 +2450,15 @@ class NixlBaseConnectorWorker: if self._mixed_mem_types: handle_data = np.asarray(handle_data, dtype=np.uint64) desc_is_dram = self._desc_is_dram_by_block_size[remote_block_size] - dram_descs = self.nixl_wrapper.get_xfer_descs( - handle_data[desc_is_dram], "DRAM" - ) - dram_handle = self.nixl_wrapper.prep_xfer_dlist( - "NIXL_INIT_AGENT", dram_descs - ) + if self._skip_dram_xfer: + dram_handle = None + else: + dram_descs = self.nixl_wrapper.get_xfer_descs( + handle_data[desc_is_dram], "DRAM" + ) + dram_handle = self.nixl_wrapper.prep_xfer_dlist( + "NIXL_INIT_AGENT", dram_descs + ) self._dram_src_handles_by_tp_ratio[split_key].append(dram_handle) handle_data = handle_data[~desc_is_dram] descs = self.nixl_wrapper.get_xfer_descs( @@ -2555,6 +2594,7 @@ class NixlBaseConnectorWorker: if ( self.kv_transfer_config.enable_permute_local_kv and nixl_agent_meta.kv_cache_layout == "LBHNC" + and kv_cache_layout == "LBNHC" ): logger.info( "Remote is LBHNC and local is LBNHC, enabled additional permute " @@ -2566,9 +2606,11 @@ class NixlBaseConnectorWorker: self.enable_permute_local_kv = True else: raise RuntimeError( - "Heterogeneous TP expects same kv_cache_layout. " - "Or enable experimental feature to use HND to NHD support by " - "setting 'enable_permute_local_kv'=True in --kv-transfer-config." + "Heterogeneous TP expects same kv_cache_layout (local " + f"{kv_cache_layout}, remote {nixl_agent_meta.kv_cache_layout}). " + "Or enable experimental feature to receive LBHNC into a local " + "LBNHC cache by setting 'enable_permute_local_kv'=True " + "in --kv-transfer-config." ) # if remote_agent used attn is not same as local, # hint heterogenuous attn post process @@ -2754,26 +2796,10 @@ class NixlBaseConnectorWorker: "d2h", ) - @cached_property - def _attention_kv_caches(self) -> list[torch.Tensor]: - """Device KV caches of attention layers (mamba states excluded), - as consumed by the receive post-process.""" - assert self.device_kv_caches, ( - "_attention_kv_caches accessed before register_kv_caches" - ) - mamba_layers = { - name - for g, group in enumerate(self.kv_cache_config.transfer_groups) - if _is_ssm_spec(self._group_spec_types[g]) - for name in group.layer_names - } - kv_caches = self.device_kv_caches - return [cache for name, cache in kv_caches.items() if name not in mamba_layers] - def post_process_device_kv_on_receive( self, block_size_ratio: int, - block_ids_list: list[tuple[list[int], int]], + block_ids_list: list[tuple[int, list[int], int]], convert: bool = True, ): """Post process device kv cache after receiving from remote. @@ -2822,9 +2848,11 @@ class NixlBaseConnectorWorker: block_size_ratio, ) - attn_caches = self._attention_kv_caches - device = attn_caches[0].device - for block_ids, covered_sub_blocks in block_ids_list: + for group, block_ids, covered_sub_blocks in block_ids_list: + attn_caches = [ + self.device_kv_caches[name] + for name in self.kv_cache_config.transfer_groups[group].layer_names + ] # Blocks the transfer didn't write: the token tail of the last # partially covered block, then everything beyond it. covered_blocks, sub_blocks_in_last = divmod( @@ -2834,17 +2862,16 @@ class NixlBaseConnectorWorker: has_stale = first_stale < len(block_ids) indices = None if convert or has_stale: - indices = async_tensor_h2d(block_ids, device, torch.long) + indices = async_tensor_h2d(block_ids, attn_caches[0].device, torch.long) if convert: for cache in attn_caches: - if self.enable_permute_local_kv and block_size_ratio > 1: + if self.enable_permute_local_kv: kv_postprocess_blksize_and_layout_on_receive( cache, indices, block_size_ratio ) - elif self.enable_permute_local_kv: - kv_postprocess_layout_on_receive(cache, indices) - else: + elif KVCacheLayout[self.kv_cache_layout].is_block_contiguous: + # Token-major blocks get the sub-blocks in token order. kv_postprocess_blksize_on_receive( cache, indices, block_size_ratio ) @@ -2852,10 +2879,10 @@ class NixlBaseConnectorWorker: if sub_blocks_in_last: last_block_id = block_ids[covered_blocks] for cache in attn_caches: - # Both post-processed layouts leave tokens on dim 1. - sub_block_tokens = cache.shape[1] // block_size_ratio + # Attention caches are [B, H, N, C]: tokens on dim 2. + sub_block_tokens = cache.shape[2] // block_size_ratio zero_from = sub_blocks_in_last * sub_block_tokens - cache[last_block_id, zero_from:].zero_() + cache[last_block_id, :, zero_from:].zero_() if has_stale: assert indices is not None stale_ids = indices[first_stale:] @@ -2878,6 +2905,27 @@ class NixlBaseConnectorWorker: indices=indices, ) + def _leaves_dram_xfer_to_rank0(self) -> bool: + """Whether this rank leaves the DRAM regions' NIXL work to TP rank 0. + + A HiSparse host pool shared by the local TP ranks (one mmap per DP + rank) is registered with NIXL and filled by TP rank 0 alone: every + rank would otherwise register the same pages on every NIC and READ the + same host KV into them. Private pools are unaffected. + """ + if not ( + self._mixed_mem_types and self.kv_cache_config.hisparse_shared_host_pool + ): + return False + skip = self.tp_rank != 0 + logger.info( + "HiSparse host pool: NIXL DRAM registration and reads on TP rank 0 " + "only (tp_rank=%d, skip=%s)", + self.tp_rank, + skip, + ) + return skip + def _zero_region_blocks(self, block_ids: BlockIds) -> None: """Clear clipped physical pages in their owning region only.""" if not any(block_ids): @@ -2886,6 +2934,9 @@ class NixlBaseConnectorWorker: for region, blocks in enumerate(block_ids): if not blocks: continue + if self._skip_dram_xfer and self.region_mem_types[region] == "DRAM": + # The shared pool is rank 0's to write. + continue cache = self.device_kv_caches[self.region_names[region]] storage = cache.untyped_storage() pages = torch.empty(0, dtype=torch.uint8, device=cache.device).set_( @@ -3003,7 +3054,7 @@ class NixlBaseConnectorWorker: len(meta.remote.block_ids[g]), ) block_ids_for_blocksize_post_process[block_size_ratio].append( - (local_group, covered_sub_blocks) + (g, local_group, covered_sub_blocks) ) # post processing for heterogeneous attention if self.enable_heterogeneous_attn_post_process: @@ -3031,7 +3082,7 @@ class NixlBaseConnectorWorker: # Handle timeout to avoid stranding blocks on remote. self._reap_expired_send_leases(done_sending) - self._cleanup_replaced_remote_engines() + self._cleanup_remote_engines() return KVConnectorTransferResults( finished_sending=done_sending, finished_recving=done_recving, @@ -3197,6 +3248,12 @@ class NixlBaseConnectorWorker: """ if record_failed_transfer: self.xfer_stats.record_failed_transfer() + if ( + self._TRANSFER_MODE == "pull" + and (meta := self._recving_metadata.get(req_id)) is not None + and meta.remote is not None + ): + self._failed_remote_engines.add(meta.remote.engine_id) if failed_req_ids is not None: failed_req_ids.add(req_id) return handle is None or self._try_release_xfer_handle(req_id, handle) @@ -3236,6 +3293,9 @@ class NixlBaseConnectorWorker: ): continue # handshake is still pending + # Refresh the TTL: a heartbeat means the router still paired this P-D, + # so requests are still waiting on this engine + self._engine_last_active[engine_id] = time.perf_counter() # Build the heartbeat message: "HB:req1,req2,..." hb_msg = ("HB:" + ",".join(hb_info.req_ids)).encode() for agent_name in self._remote_agents[engine_id].values(): @@ -3586,10 +3646,13 @@ class NixlBaseConnectorWorker: prevents us from using background threads, though memory usage is not guaranteed to be "optimal" until a new handshake is performed. - Engines with active transfers or pending handshakes cannot be stale: + Engines with active transfers, heartbeats or pending handshakes cannot + be stale: - Reads stamp _engine_last_active when they are issued, and engines a transfer is still reading from are held back explicitly, since the stamp is not refreshed while the read runs. + - Heartbeats stamp it too, so an engine stays registered while this + instance holds requests waiting on it. - Pending handshakes don't have an _engine_last_active entry yet """ # NOTE (NickLucche): This does NOT currently prevent OOMing if a huge number @@ -3638,15 +3701,38 @@ class NixlBaseConnectorWorker: engine_id, ) - def _cleanup_replaced_remote_engines(self) -> None: - """Release replaced pull peers once requests and handshakes have drained.""" + def _cleanup_remote_engines(self) -> None: + """Release replaced or locally invalid pull peers after their reads drain.""" + # TODO: Also handle push mode, which handshakes in both directions. if self._TRANSFER_MODE != "pull": return + # A background handshake may be loading native metadata with the GIL + # released. Defer both probing and cleanup until it has finished. with self._handshake_lock: if self._handshake_futures: return replaced = self._remote_agents.keys() - self._engine_by_address.values() - if not replaced: + now = time.perf_counter() + for engine_id in self._failed_remote_engines - self._invalid_remote_engines: + agents = self._remote_agents.get(engine_id, {}).values() + try: + # No descriptors: query local existence, not peer health. + if not all(map(self.nixl_wrapper.check_remote_metadata, agents)): + self._mark_remote_engine_invalid(engine_id, now) + except Exception: + logger.warning( + "Could not check local NIXL metadata for engine %s.", + engine_id, + exc_info=True, + ) + self._failed_remote_engines.clear() + # Invalid peers keep rejecting new reads until their backoff has elapsed. + invalid = { + engine_id + for engine_id in self._invalid_remote_engines + if self._invalid_engine_backoff.get(engine_id, (0.0, 0.0))[1] <= now + } + if not replaced and not invalid: return # _recving_metadata is only accessed from this (main) thread. busy = { @@ -3654,9 +3740,42 @@ class NixlBaseConnectorWorker: for meta in self._recving_metadata.values() if meta.remote is not None } - for engine_id in replaced - busy: + # Replaced peers can still serve queued requests. Invalid peers reject + # new reads, but must retain descriptors until outstanding handles end. + invalid -= self._engines_with_inflight_transfers() + for engine_id in (replaced - busy) | invalid: self._cleanup_remote_engine(engine_id, log_eviction=False) - logger.info("Released NIXL state for replaced remote engine %s.", engine_id) + logger.info( + "Released NIXL state for replaced or locally invalid remote engine %s.", + engine_id, + ) + + def _mark_remote_engine_invalid(self, engine_id: EngineId, now: float) -> None: + """Schedule release of an engine whose local NIXL metadata is gone. + + The first invalidation is released as soon as reads drain. Repeats + before the peer has stayed healthy for the max backoff double the + delay, during which reads fail fast instead of re-handshaking. + """ + # Forget engines that have stayed healthy long enough. + for eid, (_, release_at) in list(self._invalid_engine_backoff.items()): + if now - release_at > _INVALID_PEER_MAX_BACKOFF_S: + del self._invalid_engine_backoff[eid] + prev = self._invalid_engine_backoff.get(engine_id) + backoff = 0.0 + if prev is not None: + backoff = min( + max(2 * prev[0], _INVALID_PEER_MIN_BACKOFF_S), + _INVALID_PEER_MAX_BACKOFF_S, + ) + self._invalid_engine_backoff[engine_id] = (backoff, now + backoff) + self._invalid_remote_engines.add(engine_id) + logger.warning( + "Local NIXL metadata for remote engine %s was invalidated; " + "re-handshaking after reads drain%s.", + engine_id, + f" and a {backoff:.0f}s backoff" if backoff else "", + ) def _cleanup_remote_engine( self, engine_id: EngineId, *, log_eviction: bool = True @@ -3664,8 +3783,8 @@ class NixlBaseConnectorWorker: """Remove all state for a single remote engine. Releases NIXL resources (dlist handles, remote agents) and clears - all per-engine data structures. Used by TTL eviction, replaced peer - cleanup and shutdown. + all per-engine data structures. Used by TTL eviction, replaced/invalid + peer cleanup and shutdown. """ assert engine_id in self._remote_agents @@ -3681,6 +3800,9 @@ class NixlBaseConnectorWorker: for agent_name in agents.values(): self.nixl_wrapper.remove_remote_agent(agent_name) + self._failed_remote_engines.discard(engine_id) + self._invalid_remote_engines.discard(engine_id) + self.kv_caches_base_addr.pop(engine_id, None) self.dst_num_blocks.pop(engine_id, None) self.dst_region_num_blocks.pop(engine_id, None) @@ -3715,11 +3837,13 @@ class NixlBaseConnectorWorker: for handles in self.src_xfer_handles_by_tp_ratio.values(): for handle in handles: self.nixl_wrapper.release_dlist_handle(handle) - for handles in self._dram_src_handles_by_tp_ratio.values(): - for handle in handles: - self.nixl_wrapper.release_dlist_handle(handle) - for handle in self._dram_src_handles_by_block_size.values(): - self.nixl_wrapper.release_dlist_handle(handle) + for dram_handles in self._dram_src_handles_by_tp_ratio.values(): + for dram_handle in dram_handles: + if dram_handle is not None: + self.nixl_wrapper.release_dlist_handle(dram_handle) + for dram_handle in self._dram_src_handles_by_block_size.values(): + if dram_handle is not None: + self.nixl_wrapper.release_dlist_handle(dram_handle) except Exception: logger.exception("NIXL dlist-handle release failed at shutdown.") self.src_xfer_handles_by_block_size.clear() diff --git a/vllm/distributed/kv_transfer/kv_connector/v1/nixl/connector.py b/vllm/distributed/kv_transfer/kv_connector/v1/nixl/connector.py index 27b4d410f0..6e43356c0d 100644 --- a/vllm/distributed/kv_transfer/kv_connector/v1/nixl/connector.py +++ b/vllm/distributed/kv_transfer/kv_connector/v1/nixl/connector.py @@ -58,6 +58,7 @@ from vllm.forward_context import ForwardContext from vllm.logger import init_logger from vllm.v1.attention.backend import AttentionMetadata from vllm.v1.core.sched.output import SchedulerOutput +from vllm.v1.metrics.cache_hit_source import CacheHitSource from vllm.v1.outputs import KVConnectorOutput if TYPE_CHECKING: @@ -77,6 +78,8 @@ logger = init_logger(__name__) class NixlBaseConnector(KVConnectorBase_V1, SupportsHMA): """Base connector with common logic shared by pull and push modes.""" + _cache_hit_source = CacheHitSource.P2P + @property def supports_divergent_local_hybrid_hits(self) -> bool: return True diff --git a/vllm/distributed/kv_transfer/kv_connector/v1/nixl/metadata.py b/vllm/distributed/kv_transfer/kv_connector/v1/nixl/metadata.py index 8d0aad7633..a33f620a0a 100644 --- a/vllm/distributed/kv_transfer/kv_connector/v1/nixl/metadata.py +++ b/vllm/distributed/kv_transfer/kv_connector/v1/nixl/metadata.py @@ -49,8 +49,9 @@ PUSH_REG_NOTIF_PREFIX = b"PUSH_REG:" # 11: Add per-region transfer geometry and memory types to NixlAgentMetadata # 12: Add per-region member names for PP push # 13: Add packed-member layouts and order-independent packed-push backend hashes +# 14: Add Qwen3.8-Flash-Next PLE page length - #59997 # -NIXL_CONNECTOR_VERSION: int = 13 +NIXL_CONNECTOR_VERSION: int = 14 @dataclass @@ -73,6 +74,8 @@ class NixlAgentMetadata: region_mem_types: list[str] | None = None dcp_size: int = 1 pcp_size: int = 1 + # Page length of the PLE short-conv state (None when the model has none). + ple_block_len: int | None = None # Layer names sharing each advertised region, in region order. region_members: list[list[str]] = field(default_factory=list) # Packed member -> (byte offset in its region's block, bytes per page). diff --git a/vllm/distributed/kv_transfer/kv_connector/v1/nixl/pull_worker.py b/vllm/distributed/kv_transfer/kv_connector/v1/nixl/pull_worker.py index 4a18632eb4..16a25149ea 100644 --- a/vllm/distributed/kv_transfer/kv_connector/v1/nixl/pull_worker.py +++ b/vllm/distributed/kv_transfer/kv_connector/v1/nixl/pull_worker.py @@ -72,26 +72,11 @@ class NixlPullConnectorWorker(NixlBaseConnectorWorker): # Aborted cleanup requests have already been removed from the scheduler. if meta.awaiting_kvs or any(meta.local_block_ids): self._recving_metadata[req_id] = meta - if remote_engine_id not in self._remote_agents: - # Initiate handshake with remote engine to exchange metadata. - with self._handshake_lock: - if remote_engine_id not in self._remote_agents: - self._background_nixl_handshake(req_id, remote_engine_id, meta) - continue - - # Handshake already completed, start async read xfer. self._read_blocks_for_req(req_id, meta) # Start transfers for requests whose handshakes have now finished. while not self._ready_requests.empty(): - req_id, meta = self._ready_requests.get_nowait() - assert meta.remote is not None - if meta.remote.engine_id not in self._remote_agents: - # The engine was released after its handshake completed, so - # handshake again. This fails if the engine is gone. - self._background_nixl_handshake(req_id, meta.remote.engine_id, meta) - continue - self._read_blocks_for_req(req_id, meta) + self._read_blocks_for_req(*self._ready_requests.get_nowait()) if self.pcp_rank > 0 and not self.pcp_dcp_sharded: # Replicated-KV PCP: only PCP rank 0 serves the KV, so this rank @@ -151,6 +136,15 @@ class NixlPullConnectorWorker(NixlBaseConnectorWorker): def _read_blocks_for_req(self, req_id: str, meta: ReqMeta): assert meta.remote is not None and self.transfer_topo is not None engine_id = meta.remote.engine_id + if engine_id in self._invalid_remote_engines: + self._handle_failed_transfer( + req_id, None, self._recv_failures, record_failed_transfer=False + ) + return + if engine_id not in self._remote_agents: + # Also re-handshakes queued requests that outlived invalid-peer cleanup. + self._background_nixl_handshake(req_id, engine_id, meta) + return # Update last activity from this remote. Mind that cleanup is done on main # thread (this one), so we don't race on this structure. self._engine_last_active[engine_id] = time.perf_counter() @@ -599,7 +593,16 @@ class NixlPullConnectorWorker(NixlBaseConnectorWorker): remote_ids = np.asarray(remote_block_descs_ids) is_dram = desc_is_dram[local_ids] - assert local_dram_handle is not None + if self._skip_dram_xfer: + # TP rank 0 reads the DRAM (shared host pool) part for every + # local rank; keep only the device part. Register the request + # even if nothing is left, so it still completes and notifies. + keep = ~is_dram + local_ids = local_ids[keep] + remote_ids = remote_ids[keep] + is_dram = is_dram[keep] + self._recving_transfers.setdefault(request_id, []) + assert local_dram_handle is not None or not is_dram.any() reads = ( (is_dram, local_dram_handle), (~is_dram, local_device_handle), diff --git a/vllm/distributed/kv_transfer/kv_connector/v1/nixl/stats.py b/vllm/distributed/kv_transfer/kv_connector/v1/nixl/stats.py index 26c4275efb..645525bc6e 100644 --- a/vllm/distributed/kv_transfer/kv_connector/v1/nixl/stats.py +++ b/vllm/distributed/kv_transfer/kv_connector/v1/nixl/stats.py @@ -44,8 +44,10 @@ class NixlKVConnectorStats(KVConnectorStats): "num_notifications_after_expiry": [], } - def record_transfer(self, res: "nixlXferTelemetry"): + def record_transfer(self, res: "nixlXferTelemetry | None"): # Keep metrics units consistent with rest of the code: time us->s + if res is None: + return self.data["transfer_duration"].append(res.xferDuration / 1e6) self.data["post_duration"].append(res.postDuration / 1e6) self.data["bytes_transferred"].append(res.totalBytes) diff --git a/vllm/distributed/kv_transfer/kv_connector/v1/offloading/scheduler.py b/vllm/distributed/kv_transfer/kv_connector/v1/offloading/scheduler.py index 64b3ad5bb2..806267dcf3 100644 --- a/vllm/distributed/kv_transfer/kv_connector/v1/offloading/scheduler.py +++ b/vllm/distributed/kv_transfer/kv_connector/v1/offloading/scheduler.py @@ -1,6 +1,7 @@ # SPDX-License-Identifier: Apache-2.0 # SPDX-FileCopyrightText: Copyright contributors to the vLLM project import time +from collections import Counter from collections.abc import Iterable, Sequence from dataclasses import dataclass, field, replace from itertools import chain, islice @@ -56,6 +57,7 @@ from vllm.v1.kv_offload.base import ( TierMatcher, make_offload_key, ) +from vllm.v1.metrics.cache_hit_source import CacheHitSource from vllm.v1.outputs import KVConnectorOutput from vllm.v1.request import Request, RequestStatus @@ -375,6 +377,9 @@ class RequestOffloadState: # Fine-grained token boundary selected beyond the last complete offload # chunk. It is consumed when the corresponding load is scheduled. partial_tail_boundary: int | None = None + # Loaded attention keys cover their token positions. Recurrent state + # supports the whole reused external prefix. + load_key_ranges: list[tuple[int, int, OffloadKey]] = field(default_factory=list) # True once on_request_finished has been signaled to the manager. finished_signaled: bool = False @@ -587,8 +592,9 @@ class OffloadingConnectorScheduler: sliding_window_groups.sort(key=_sliding_window_sort_key, reverse=True) # used by _lookup + self._full_attention_groups: tuple[int, ...] = tuple(full_attention_groups) self._sliding_window_groups: tuple[int, ...] = tuple(sliding_window_groups) - self._lookup_groups = tuple(full_attention_groups) + self._sliding_window_groups + self._lookup_groups = self._full_attention_groups + self._sliding_window_groups self._mamba_align_size: int | None = resolve_mamba_align_size( spec, kv_cache_config ) @@ -1090,6 +1096,53 @@ class OffloadingConnectorScheduler: return num_hit_tokens, bool(num_hit_tokens) + def get_external_cache_hit_sources( + self, + request: Request, + num_external_tokens: int, + ) -> dict[CacheHitSource, int]: + """Split the accepted external hit by the tier each loaded key came from. + + Attention keys cover only their loaded token ranges; recurrent state + covers the whole reused prefix. Overlaps report the outermost tier. + Uncovered tokens, including omitted sliding-window history, are + ``external_unspecified``. + """ + sources: dict[CacheHitSource, int] = {} + if num_external_tokens == 0: + return sources + req_status = self._req_status[request.request_id] + start = req_status.num_locally_computed_tokens + end = start + num_external_tokens + + events: list[tuple[int, int, CacheHitSource]] = [ + (end, 0, CacheHitSource.EXTERNAL_UNSPECIFIED) + ] + for lo, hi, key in req_status.load_key_ranges: + lo, hi = max(start, lo), min(end, hi) + if lo >= hi: + continue + source = CacheHitSource( + self.manager.get_load_source(key, req_status.req_context) + ) + events.extend(((lo, 1, source), (hi, -1, source))) + + # Sweep endpoints instead of rescanning every range for each interval. + active: Counter[CacheHitSource] = Counter() + position = start + for boundary, delta, source in sorted(events): + if boundary > position: + tiers = [tier for tier, count in active.items() if count > 0] + tier = ( + CacheHitSource.outermost(tiers) + if tiers + else CacheHitSource.EXTERNAL_UNSPECIFIED + ) + sources[tier] = sources.get(tier, 0) + boundary - position + active[source] += delta + position = boundary + return sources + def update_state_after_alloc( self, request: Request, blocks: KVCacheBlocks, num_external_tokens: int ): @@ -1105,6 +1158,7 @@ class OffloadingConnectorScheduler: assert partial_tail_boundary == num_cached_tokens keys_to_load: list[OffloadKey] = [] + req_status.load_key_ranges.clear() dst_block_ids: list[int] = [] # per group group_sizes: list[int] = [] @@ -1157,9 +1211,9 @@ class OffloadingConnectorScheduler: ) end_chunk_idx = num_chunks - (partial_tail_boundary is not None) assert len(offload_keys) >= end_chunk_idx - keys_to_load.extend(offload_keys[start_chunk_idx:end_chunk_idx]) + group_keys_to_load = offload_keys[start_chunk_idx:end_chunk_idx] if partial_tail_boundary is not None: - keys_to_load.append( + group_keys_to_load.append( self._make_boundary_key( request, group_config.group_idx, @@ -1167,6 +1221,24 @@ class OffloadingConnectorScheduler: req_status.req_context, ) ) + keys_to_load.extend(group_keys_to_load) + if isinstance(group_config.kv_cache_spec, MambaSpec): + req_status.load_key_ranges.extend( + (num_locally_computed_tokens, num_cached_tokens, key) + for key in group_keys_to_load + ) + else: + for chunk_idx, key in enumerate( + group_keys_to_load, start_chunk_idx + ): + range_start = max( + load_start_gpu_block_idx * tokens_per_block, + chunk_idx * tokens_per_chunk, + ) + range_end = min( + num_cached_tokens, (chunk_idx + 1) * tokens_per_chunk + ) + req_status.load_key_ranges.append((range_start, range_end, key)) dst_block_ids.extend( block.block_id diff --git a/vllm/distributed/kv_transfer/kv_connector/v1/offloading_connector.py b/vllm/distributed/kv_transfer/kv_connector/v1/offloading_connector.py index def6e1e1f3..af3e113b5f 100644 --- a/vllm/distributed/kv_transfer/kv_connector/v1/offloading_connector.py +++ b/vllm/distributed/kv_transfer/kv_connector/v1/offloading_connector.py @@ -45,6 +45,7 @@ from vllm.v1.core.kv_cache_manager import KVCacheBlocks from vllm.v1.core.sched.output import SchedulerOutput from vllm.v1.kv_cache_interface import KVCacheConfig from vllm.v1.kv_offload.factory import OffloadingSpecFactory +from vllm.v1.metrics.cache_hit_source import CacheHitSource from vllm.v1.outputs import KVConnectorOutput from vllm.v1.request import Request @@ -179,6 +180,16 @@ class OffloadingConnector(KVConnectorBase_V1, SupportsHMA): bound = min(per_group_hits[group_id] for group_id in self._bounding_group_ids) return max(0, bound - num_computed_tokens) + def get_external_cache_hit_sources( + self, + request: "Request", + num_external_tokens: int, + ) -> dict[CacheHitSource, int]: + assert self.connector_scheduler is not None + return self.connector_scheduler.get_external_cache_hit_sources( + request, num_external_tokens + ) + def update_state_after_alloc( self, request: "Request", blocks: "KVCacheBlocks", num_external_tokens: int ): diff --git a/vllm/distributed/kv_transfer/kv_connector/v1/simple_cpu_offload_connector.py b/vllm/distributed/kv_transfer/kv_connector/v1/simple_cpu_offload_connector.py index 9124d9eac1..0bd8ebe3cd 100644 --- a/vllm/distributed/kv_transfer/kv_connector/v1/simple_cpu_offload_connector.py +++ b/vllm/distributed/kv_transfer/kv_connector/v1/simple_cpu_offload_connector.py @@ -23,6 +23,7 @@ from vllm.distributed.kv_transfer.kv_connector.v1.metrics import ( ) from vllm.logger import init_logger from vllm.v1.core.sched.output import SchedulerOutput +from vllm.v1.metrics.cache_hit_source import CacheHitSource from vllm.v1.outputs import KVConnectorOutput from vllm.v1.simple_kv_offload.manager import ( BoundaryStoreStats, @@ -109,6 +110,9 @@ class SimpleCPUOffloadConnector(KVConnectorBase_V1, SupportsHMA): f"expected one of {VALID_KV_OFFLOAD_BACKENDS}" ) disk_mode = kv_offload_backend == "disk" + self._cache_hit_source = ( + CacheHitSource.DISK if disk_mode else CacheHitSource.HOST + ) disk_path = extra_config.get("disk_path", None) or None disk_capacity_bytes = int( diff --git a/vllm/distributed/parallel_state.py b/vllm/distributed/parallel_state.py index 88bb3d7c5a..feb8f25d1e 100644 --- a/vllm/distributed/parallel_state.py +++ b/vllm/distributed/parallel_state.py @@ -42,6 +42,7 @@ import torch.distributed import torch.distributed._functional_collectives as funcol import torch.distributed._symmetric_memory from torch.distributed import Backend, ProcessGroup, Store +from torch.distributed.constants import default_pg_timeout import vllm.envs as envs from vllm.distributed.device_communicators.base_device_communicator import ( @@ -49,6 +50,7 @@ from vllm.distributed.device_communicators.base_device_communicator import ( ) from vllm.distributed.utils import ( StatelessProcessGroup, + create_tcp_store, get_cached_tcp_store_client, ) from vllm.logger import init_logger @@ -801,6 +803,19 @@ class GroupCoordinator: raise ValueError("No device communicator found") return self.device_communicator.reduce_scatterv(input_, dim, sizes) + def reduce_scatterv_into_output( + self, + input_: torch.Tensor, + output: torch.Tensor, + dim: int = -1, + sizes: list[int] | None = None, + ) -> torch.Tensor: + if self.device_communicator is None: + raise ValueError("No device communicator found") + return self.device_communicator.reduce_scatterv_into_output( + input_, output, dim, sizes + ) + def _reduce_scatter_out_place(self, input_: torch.Tensor, dim: int) -> torch.Tensor: if self.device_communicator is None: raise ValueError("No device communicator found") @@ -1442,6 +1457,32 @@ class GroupCoordinator: else: return hidden_states + def allocate_combine_input( + self, + shape: tuple[int, ...], + dtype: torch.dtype, + device: torch.device, + is_sequence_parallel: bool = False, + ) -> torch.Tensor | None: + if self.device_communicator is None: + return None + return self.device_communicator.allocate_combine_input( + shape, dtype, device, is_sequence_parallel + ) + + def combine_into_output( + self, + hidden_states: torch.Tensor, + output: torch.Tensor, + is_sequence_parallel: bool = False, + ) -> torch.Tensor: + if self.device_communicator is None: + output.copy_(hidden_states) + return output + return self.device_communicator.combine_into_output( + hidden_states, output, is_sequence_parallel + ) + _WORLD: GroupCoordinator | None = None _INNER_DP_WORLD: GroupCoordinator | None = None @@ -1790,6 +1831,7 @@ def init_distributed_environment( config = get_current_vllm_config_or_none() enable_elastic_ep = config is not None and config.parallel_config.enable_elastic_ep + world_pg_store: Store | None = None if ( config is not None and config.parallel_config.distributed_executor_backend != "external_launcher" @@ -1813,7 +1855,46 @@ def init_distributed_environment( distributed_init_method = get_distributed_init_method(ip, port) else: ip = parallel_config.data_parallel_master_ip - port = parallel_config.get_next_dp_init_port() + if ( + parallel_config._coord_store_port + and not torch.distributed.is_initialized() + ): + # Group rank 0 lets the world-group TCPStore bind port 0 + # and publishes the kernel-assigned port via the + # coordination store. A one-shot client of our own: the + # cached one is inherited across fork and shared with + # sibling workers. + coord_store = create_tcp_store( + ip, + parallel_config._coord_store_port, + is_master=False, + wait_for_workers=False, + ) + if rank == 0: + # Workers can only connect once the port is published. + world_pg_store = create_tcp_store( + ip, + 0, + world_size=world_size, + is_master=True, + wait_for_workers=False, + timeout=timeout or default_pg_timeout, + multi_tenant=True, + ) + port = world_pg_store.port + coord_store.set("world_pg_port", str(port).encode()) + else: + port = int(coord_store.get("world_pg_port").decode()) + world_pg_store = create_tcp_store( + ip, + port, + world_size=world_size, + is_master=False, + timeout=timeout or default_pg_timeout, + multi_tenant=True, + ) + else: + port = parallel_config.get_next_dp_init_port() distributed_init_method = get_distributed_init_method(ip, port) logger.debug( "Adjusting world_size=%d rank=%d distributed_init_method=%s for DP", @@ -1843,8 +1924,8 @@ def init_distributed_environment( "Fallback Gloo backend is not available." ) backend = "gloo" - store = None - if distributed_init_method.startswith("file://"): + store = world_pg_store + if store is None and distributed_init_method.startswith("file://"): store = torch.distributed.FileStore( distributed_init_method.removeprefix("file://"), world_size ) diff --git a/vllm/distributed/utils.py b/vllm/distributed/utils.py index c780316bb4..3618361f2a 100644 --- a/vllm/distributed/utils.py +++ b/vllm/distributed/utils.py @@ -38,9 +38,7 @@ logger = init_logger(__name__) # We prefer to use os.sched_yield as it results in tighter polling loops, # measured to be around 3e-7 seconds. However on earlier versions of Python # os.sched_yield() does not release the GIL, so we fall back to time.sleep(0) -USE_SCHED_YIELD = (sys.version_info[:3] >= (3, 11, 1)) or ( - sys.version_info[:2] == (3, 10) and sys.version_info[2] >= 8 -) +USE_SCHED_YIELD = sys.version_info[:3] >= (3, 11, 1) def sched_yield(): @@ -52,8 +50,8 @@ def sched_yield(): def ensure_divisibility(numerator, denominator): """Ensure that numerator is divisible by the denominator.""" - assert numerator % denominator == 0, "{} is not divisible by {}".format( - numerator, denominator + assert numerator % denominator == 0, ( + f"{numerator} is not divisible by {denominator}" ) @@ -146,9 +144,7 @@ def get_pp_indices( try: partitions = [int(layer) for layer in partition_list_str.split(",")] except ValueError as err: - raise ValueError( - "Invalid partition string: {}".format(partition_list_str) - ) from err + raise ValueError(f"Invalid partition string: {partition_list_str}") from err if len(partitions) != pp_size: raise ValueError(f"{len(partitions)=} does not match {pp_size=}.") if sum(partitions) != num_hidden_layers: diff --git a/vllm/distributed/weight_transfer/base.py b/vllm/distributed/weight_transfer/base.py index 2fb60fb71b..3140b70e96 100644 --- a/vllm/distributed/weight_transfer/base.py +++ b/vllm/distributed/weight_transfer/base.py @@ -11,12 +11,12 @@ from typing import ( ClassVar, Generic, Protocol, + Self, TypeVar, runtime_checkable, ) import torch -from typing_extensions import Self if TYPE_CHECKING: from vllm.config import VllmConfig diff --git a/vllm/distributed/weight_transfer/clients.py b/vllm/distributed/weight_transfer/clients.py index 1bd2f98937..745e67779e 100644 --- a/vllm/distributed/weight_transfer/clients.py +++ b/vllm/distributed/weight_transfer/clients.py @@ -58,7 +58,7 @@ def _json_safe_update_payload( class HTTPVLLMWeightSyncClient: """Talks to a vLLM server over the RLHF HTTP routes. - Mirrors `vllm/entrypoints/serve/dev/rlhf/api_router.py`: + Mirrors `vllm/entrypoints/rl/online/api_router.py`: `/init_weight_transfer_engine`, `/start_weight_update`, `/update_weights`, `/finish_weight_update`. """ diff --git a/vllm/distributed/weight_transfer/ipc_engine.py b/vllm/distributed/weight_transfer/ipc_engine.py index 15864811e9..13da117b99 100644 --- a/vllm/distributed/weight_transfer/ipc_engine.py +++ b/vllm/distributed/weight_transfer/ipc_engine.py @@ -4,12 +4,11 @@ import pickle from dataclasses import asdict, dataclass -from typing import TYPE_CHECKING, Any, ClassVar +from typing import TYPE_CHECKING, Any, ClassVar, Self import pybase64 as base64 import torch from torch.multiprocessing.reductions import rebuild_cuda_tensor, reduce_tensor -from typing_extensions import Self from vllm import envs from vllm.config.weight_transfer import WeightTransferConfig diff --git a/vllm/distributed/weight_transfer/m2n_trainer.py b/vllm/distributed/weight_transfer/m2n_trainer.py index 68c62f72db..2d1d9f47ec 100644 --- a/vllm/distributed/weight_transfer/m2n_trainer.py +++ b/vllm/distributed/weight_transfer/m2n_trainer.py @@ -15,10 +15,9 @@ in every caller. from concurrent.futures import Future, ThreadPoolExecutor from dataclasses import dataclass, field -from typing import TYPE_CHECKING, Any, ClassVar, cast +from typing import TYPE_CHECKING, Any, ClassVar, Self, cast import torch -from typing_extensions import Self from vllm.distributed.weight_transfer.base import ( TrainerInitInfo, diff --git a/vllm/distributed/weight_transfer/nccl_engine.py b/vllm/distributed/weight_transfer/nccl_engine.py index 5b98845870..77d8b60872 100644 --- a/vllm/distributed/weight_transfer/nccl_engine.py +++ b/vllm/distributed/weight_transfer/nccl_engine.py @@ -5,10 +5,9 @@ from collections.abc import Iterator from concurrent.futures import ThreadPoolExecutor from dataclasses import asdict, dataclass -from typing import TYPE_CHECKING, ClassVar +from typing import TYPE_CHECKING, ClassVar, Self import torch -from typing_extensions import Self if TYPE_CHECKING: from vllm.config import VllmConfig diff --git a/vllm/distributed/weight_transfer/sharded_rdt_trainer.py b/vllm/distributed/weight_transfer/sharded_rdt_trainer.py index 6b755e772e..51ea401ef9 100644 --- a/vllm/distributed/weight_transfer/sharded_rdt_trainer.py +++ b/vllm/distributed/weight_transfer/sharded_rdt_trainer.py @@ -24,7 +24,7 @@ import uuid from collections.abc import Callable, Collection from concurrent.futures import ThreadPoolExecutor from dataclasses import asdict, dataclass, field -from typing import Any, ClassVar +from typing import Any, ClassVar, Self import ray import torch @@ -33,7 +33,6 @@ from torch.multiprocessing.reductions import ( rebuild_cuda_tensor, reduce_tensor, ) -from typing_extensions import Self from vllm.distributed.weight_transfer.base import ( ParamMeta, diff --git a/vllm/distributed/weight_transfer/sparse_nccl_engine.py b/vllm/distributed/weight_transfer/sparse_nccl_engine.py index 9c704ab344..01b29529de 100644 --- a/vllm/distributed/weight_transfer/sparse_nccl_engine.py +++ b/vllm/distributed/weight_transfer/sparse_nccl_engine.py @@ -11,10 +11,9 @@ from collections.abc import Iterable from concurrent.futures import ThreadPoolExecutor from contextlib import nullcontext from dataclasses import asdict, dataclass -from typing import TYPE_CHECKING, ClassVar +from typing import TYPE_CHECKING, ClassVar, Self import torch -from typing_extensions import Self if TYPE_CHECKING: from vllm.config import VllmConfig diff --git a/vllm/engine/arg_utils.py b/vllm/engine/arg_utils.py index 4f1f417289..5f32ec8deb 100644 --- a/vllm/engine/arg_utils.py +++ b/vllm/engine/arg_utils.py @@ -839,7 +839,9 @@ class EngineArgs: ) fail_on_environ_validation: bool = False - gdn_prefill_backend: Literal["flashinfer", "triton", "cutedsl"] | None = None + gdn_prefill_backend: ( + Literal["flashinfer", "triton", "cutedsl", "aiter_flydsl"] | None + ) = None kda_prefill_backend: ( Literal["auto", "triton", "flashkda", "flashinfer", "fused"] | None ) = None @@ -1890,7 +1892,7 @@ class EngineArgs: parser.add_argument( "--gdn-prefill-backend", dest="gdn_prefill_backend", - choices=["flashinfer", "triton", "cutedsl"], + choices=["flashinfer", "triton", "cutedsl", "aiter_flydsl"], default=None, help="Select GDN prefill backend.", ) @@ -2111,7 +2113,7 @@ class EngineArgs: for x in cvd.split(",") ] for i in int_ids: - if i >= len(cvd_ids): + if not 0 <= i < len(cvd_ids): raise ValueError( f"--device-ids index {i} is out of range for " f"{current_platform.device_control_env_var}" @@ -2136,6 +2138,25 @@ class EngineArgs: cfg = json.loads(cfg) return WatermarkConfig(**cfg) + def create_structured_outputs_config(self) -> StructuredOutputsConfig: + """Merge frontend parser flags into the structured outputs config. + + Mutates `self.structured_outputs_config` in place and returns it + (not a copy). Model-specific defaults (e.g. gpt_oss -> + "openai_gptoss") are applied later by `verify_and_update_config` + only when the resolved value is still empty, so explicit CLI flags + take precedence. + """ + if self.reasoning_parser: + self.structured_outputs_config.reasoning_parser = self.reasoning_parser + + if self.reasoning_parser_plugin: + self.structured_outputs_config.reasoning_parser_plugin = ( + self.reasoning_parser_plugin + ) + + return self.structured_outputs_config + def create_observability_config(self) -> ObservabilityConfig: return ObservabilityConfig( show_hidden_metrics_for_version=self.show_hidden_metrics_for_version, @@ -2252,15 +2273,23 @@ class EngineArgs: kv_offloading_backend=self.kv_offloading_backend, ) - if resolved_cache_dtype.startswith("turboquant_"): + # TurboQuant and UltraQuant both keep boundary attention layers at the + # native dtype, and compute those layers the same way. + uses_packed_kv_backend = ( + resolved_cache_dtype.startswith("turboquant_") + or resolved_cache_dtype == "ultraquant_4bit" + ) + if uses_packed_kv_backend: from vllm.model_executor.layers.quantization.turboquant.config import ( TurboQuantConfig, ) boundary = TurboQuantConfig.get_boundary_skip_layers(model_config) - existing = set(cache_config.kv_cache_dtype_skip_layers) + # Sorted so the order does not depend on set iteration: the list is + # a CacheConfig hash factor. Entries can also be attention type + # names (e.g. "sliding_window"), so sort them as strings. cache_config.kv_cache_dtype_skip_layers = sorted( - existing | set(boundary), key=int + set(cache_config.kv_cache_dtype_skip_layers) | set(boundary) ) ray_runtime_env = None @@ -2662,14 +2691,18 @@ class EngineArgs: # TurboQuant requires FlashAttention 2 — FA3 boundary layers assert # FlashAttentionImpl which fails with TurboQuantAttentionImpl. - if resolved_cache_dtype.startswith("turboquant_") and ( + # UltraQuant subclasses that impl, so it inherits the same limit. + if uses_packed_kv_backend and ( attention_config.flash_attn_version is None or attention_config.flash_attn_version >= 3 ): logger.warning( - "TurboQuant is not yet compatible with FlashAttention >= 3. " + "%s is not yet compatible with FlashAttention >= 3. " "Overriding flash_attn_version to 2. To silence this " - "warning, pass --attention-config.flash_attn_version=2" + "warning, pass --attention-config.flash_attn_version=2", + "TurboQuant" + if resolved_cache_dtype.startswith("turboquant_") + else "UltraQuant", ) attention_config.flash_attn_version = 2 @@ -2728,13 +2761,7 @@ class EngineArgs: load_config = self.create_load_config() # Pass reasoning_parser into StructuredOutputsConfig - if self.reasoning_parser: - self.structured_outputs_config.reasoning_parser = self.reasoning_parser - - if self.reasoning_parser_plugin: - self.structured_outputs_config.reasoning_parser_plugin = ( - self.reasoning_parser_plugin - ) + self.create_structured_outputs_config() observability_config = self.create_observability_config() diff --git a/vllm/engine/protocol.py b/vllm/engine/protocol.py index a34981d651..2bb128ba07 100644 --- a/vllm/engine/protocol.py +++ b/vllm/engine/protocol.py @@ -162,8 +162,14 @@ class EngineClient(ABC): ... @abstractmethod - async def start_profile(self) -> None: - """Start profiling the engine.""" + async def start_profile( + self, + profile_prefix: str | None = None, + *, + delay_iterations: int | None = None, + max_iterations: int | None = None, + ) -> None: + """Start profiling with optional per-session overrides.""" ... @abstractmethod diff --git a/vllm/entrypoints/anthropic/protocol.py b/vllm/entrypoints/anthropic/protocol.py index 5abfaffed3..b2d29e0b34 100644 --- a/vllm/entrypoints/anthropic/protocol.py +++ b/vllm/entrypoints/anthropic/protocol.py @@ -229,6 +229,17 @@ class AnthropicMessagesRequest(BaseModel): top_p: float | None = None # vLLM-specific fields that are not in Anthropic spec + return_mm_kwargs: bool = Field( + default=True, + description=( + "If false, the render response's `features` set `kwargs_data` " + "and `mm_metadata` to null, for callers that need only the token " + "layout and item hashes, such as cache-aware routers. Do not send " + "such a response to `/inference/v1/generate`, which reads a null " + "`kwargs_data` as every item being cached. Only supported on the " + "render endpoints; ignored on regular generation endpoints." + ), + ) cache_salt: str | None = Field( default=None, min_length=1, @@ -242,6 +253,12 @@ class AnthropicMessagesRequest(BaseModel): "to 256 bit)." ), ) + watermarking: bool | None = Field( + default=None, + description=( + "Whether to apply the engine's configured watermark to this request." + ), + ) kv_transfer_params: dict[str, Any] | None = Field( default=None, description="KVTransfer parameters used for disaggregated serving.", @@ -379,7 +396,9 @@ class AnthropicCountTokensRequest(BaseModel): model: str messages: list[AnthropicMessage] + output_config: AnthropicOutputConfig | None = None system: str | list[AnthropicContentBlock] | None = None + thinking: AnthropicThinkingConfig | None = None tool_choice: AnthropicToolChoice | None = None tools: list[AnthropicTool] | None = None diff --git a/vllm/entrypoints/anthropic/serving.py b/vllm/entrypoints/anthropic/serving.py index cf5fd9ecc1..f11697fa16 100644 --- a/vllm/entrypoints/anthropic/serving.py +++ b/vllm/entrypoints/anthropic/serving.py @@ -613,6 +613,7 @@ class AnthropicServingMessages(OpenAIServingChat): temperature=anthropic_request.temperature, top_p=anthropic_request.top_p, top_k=anthropic_request.top_k, + watermarking=anthropic_request.watermarking, cache_salt=anthropic_request.cache_salt, kv_transfer_params=anthropic_request.kv_transfer_params, ec_transfer_params=anthropic_request.ec_transfer_params, @@ -632,8 +633,6 @@ class AnthropicServingMessages(OpenAIServingChat): ``display`` is intentionally ignored: suppressing reasoning would mark it ended for structured outputs and drop it from multi-turn history. """ - if isinstance(anthropic_request, AnthropicCountTokensRequest): - return thinking: AnthropicThinkingConfig | None = anthropic_request.thinking if thinking is None: return @@ -654,8 +653,6 @@ class AnthropicServingMessages(OpenAIServingChat): anthropic_request: AnthropicMessagesRequest | AnthropicCountTokensRequest, ) -> None: """Handle output configuration such as output format and effort.""" - if isinstance(anthropic_request, AnthropicCountTokensRequest): - return output_config: AnthropicOutputConfig | None = anthropic_request.output_config if output_config and output_config.format and output_config.format.json_schema: req.response_format = ResponseFormat( @@ -1237,9 +1234,13 @@ class AnthropicServingMessages(OpenAIServingChat): raw_request: Request | None = None, ) -> AnthropicCountTokensResponse | ErrorResponse: """Implements Anthropic's messages.count_tokens endpoint.""" + disabled_thinking_effort: AnthropicDisabledThinkingEffort = "none" + if request.thinking is not None and request.thinking.type == "disabled": + disabled_thinking_effort = await self._get_disabled_thinking_effort() chat_req = self.to_chat_completion_request( request, merge_inline_system=self._merge_inline_system, + disabled_thinking_effort=disabled_thinking_effort, ) result = await self.render_chat_request(chat_req) if isinstance(result, ErrorResponse): diff --git a/vllm/entrypoints/chat_utils.py b/vllm/entrypoints/chat_utils.py index 2d875a974b..f18407e535 100644 --- a/vllm/entrypoints/chat_utils.py +++ b/vllm/entrypoints/chat_utils.py @@ -17,6 +17,7 @@ from typing import ( Final, Generic, Literal, + Required, TypeAlias, TypeVar, Union, @@ -48,7 +49,7 @@ from PIL import Image from pydantic import BaseModel, ConfigDict, TypeAdapter # pydantic needs the TypedDict from typing_extensions -from typing_extensions import Required, TypedDict, override +from typing_extensions import TypedDict, override from vllm import envs from vllm.config import ModelConfig diff --git a/vllm/entrypoints/cli/serve.py b/vllm/entrypoints/cli/serve.py index 4c9410488b..94e6545c08 100644 --- a/vllm/entrypoints/cli/serve.py +++ b/vllm/entrypoints/cli/serve.py @@ -328,19 +328,10 @@ def run_multi_api_server(args: argparse.Namespace): None ) - from vllm.v1.engine.utils import get_engine_zmq_addresses + from vllm.v1.engine.utils import bind_engine_zmq_listeners - # Defer port allocation to the child's bind() to avoid TOCTOU, except - # for Rust front-end and Ray DP, which can't see the post-bind rebind - # (CLI-arg subprocess / pickled-into-actor snapshot respectively) and - # so pre-allocate driver-side -- reintroducing the original race only - # there. - is_ray_dp = parallel_config.data_parallel_backend == "ray" - addresses = get_engine_zmq_addresses( - vllm_config, - num_api_servers, - defer_api_server_ports=not (rust_frontend_path or is_ray_dp), - ) + zmq_listeners = bind_engine_zmq_listeners(vllm_config, num_api_servers) + addresses = zmq_listeners.addresses with launch_core_engines( vllm_config, executor_class, log_stats, addresses @@ -365,8 +356,8 @@ def run_multi_api_server(args: argparse.Namespace): sock=sock, grpc_sock=grpc_sock, args=args, - input_address=addresses.inputs[0], - output_address=addresses.outputs[0], + input_listener=zmq_listeners.inputs[0], + output_listener=zmq_listeners.outputs[0], engine_start_index=expected_engine_start_index, engine_count=expected_engine_count, data_parallel_size=parallel_config.data_parallel_size, @@ -379,22 +370,12 @@ def run_multi_api_server(args: argparse.Namespace): sock=sock, args=args, num_servers=num_api_servers, - input_addresses=addresses.inputs, - output_addresses=addresses.outputs, + input_listeners=zmq_listeners.inputs, + output_listeners=zmq_listeners.outputs, stats_update_address=stats_update_address, tensor_queue=engine_launch.tensor_queue, ) - if not is_ray_dp: - # Forward each child's bound endpoints to the engine handshake - # (runs on ``with`` exit). Skipped for Ray DP, where addresses - # are pre-allocated above and Ray actors already hold them. - actual_inputs, actual_outputs = ( - api_server_manager.gather_actual_addresses() - ) - addresses.inputs = actual_inputs - addresses.outputs = actual_outputs - # Set frontend processes to watch during engine startup. # If any of these processes exit before the engines are up, the engine startup # will be aborted with an error. diff --git a/vllm/entrypoints/cohere/api_router.py b/vllm/entrypoints/cohere/api_router.py index eaac36f133..5362b1d9e9 100644 --- a/vllm/entrypoints/cohere/api_router.py +++ b/vllm/entrypoints/cohere/api_router.py @@ -40,6 +40,9 @@ from starlette.middleware.base import BaseHTTPMiddleware import vllm.envs as envs from vllm.entrypoints.serve.engine.protocol import ErrorResponse +from vllm.entrypoints.serve.exception_handling.error_response import ( + create_error_response, +) from vllm.entrypoints.serve.exception_handling.utils import sanitize_message from vllm.entrypoints.serve.utils.api_utils import ( load_aware_call, @@ -138,15 +141,9 @@ if _SDK_AVAILABLE: try: result = await handler.create_chat_v2(request, raw_request) - except Exception as e: # noqa: BLE001 - report as 500 for parity + except Exception as e: logger.exception("Error in /cohere/v2/chat: %s", e) - return JSONResponse( - status_code=HTTPStatus.INTERNAL_SERVER_ERROR.value, - content=CohereError( - message=sanitize_message(str(e)), - id=_request_id(raw_request), - ).model_dump(exclude_none=True), - ) + return _error_response(create_error_response(e), raw_request) match result: case ErrorResponse(): @@ -194,19 +191,18 @@ if _SDK_AVAILABLE: try: chat_request = handler.to_chat_completion_request(request) result = await render_handler.render_chat_request(chat_request) - except Exception as e: # noqa: BLE001 - report as 500 for parity + except Exception as e: logger.exception("Error in /cohere/v2/chat/render: %s", e) - return JSONResponse( - status_code=HTTPStatus.INTERNAL_SERVER_ERROR.value, - content=CohereError( - message=sanitize_message(str(e)), - id=_request_id(raw_request), - ).model_dump(exclude_none=True), - ) + return _error_response(create_error_response(e), raw_request) if isinstance(result, ErrorResponse): return _error_response(result, raw_request) + if (kwargs := result.reasoning_parser_kwargs) is not None: + kwargs.chat_template_kwargs = handler._engine_chat_template_kwargs( + kwargs.chat_template_kwargs + ) + return JSONResponse(content=result.model_dump()) class CohereErrorEnvelopeMiddleware(BaseHTTPMiddleware): diff --git a/vllm/entrypoints/cohere/protocol.py b/vllm/entrypoints/cohere/protocol.py index 1432e6eb11..45e0d310dc 100644 --- a/vllm/entrypoints/cohere/protocol.py +++ b/vllm/entrypoints/cohere/protocol.py @@ -175,6 +175,12 @@ class CohereChatV2Request(BaseModel): # vLLM-specific extensions (not in Cohere spec). These mirror what the # Anthropic and OpenAI surfaces already expose so V2 callers can reach # the same engine knobs when needed. + watermarking: bool | None = Field( + default=None, + description=( + "Whether to apply the engine's configured watermark to this request." + ), + ) kv_transfer_params: dict[str, Any] | None = Field( default=None, description="KVTransfer parameters used for disaggregated serving.", diff --git a/vllm/entrypoints/cohere/serving.py b/vllm/entrypoints/cohere/serving.py index 63d00f0c57..98242cc55d 100644 --- a/vllm/entrypoints/cohere/serving.py +++ b/vllm/entrypoints/cohere/serving.py @@ -58,6 +58,7 @@ from vllm.entrypoints.cohere.protocol import ( CohereChatV2Request, CohereChatV2Response, CohereFinishReason, + CohereLogprobItem, CohereUsage, CohereUsageBilledUnits, CohereUsageTokens, @@ -79,6 +80,7 @@ from vllm.entrypoints.generate.base.protocol import ( from vllm.entrypoints.openai.chat_completion.protocol import ( ChatCompletionRequest, ChatCompletionResponse, + ChatCompletionResponseChoice, ChatCompletionStreamResponse, ChatCompletionToolsParam, ChatMessage, @@ -557,10 +559,13 @@ class CohereServingChatV2(OpenAIServingChat): temperature=request.temperature, top_p=request.p, top_k=request.k, + watermarking=request.watermarking, seed=request.seed, frequency_penalty=request.frequency_penalty, presence_penalty=request.presence_penalty, logprobs=request.logprobs, + # Cohere logprob items carry token ids alongside the logprobs. + return_token_ids=request.logprobs, priority=request.priority or 0, kv_transfer_params=request.kv_transfer_params, chat_template_kwargs=request.chat_template_kwargs, @@ -1090,9 +1095,29 @@ class CohereServingChatV2(OpenAIServingChat): ), message=assistant_msg, usage=usage, + logprobs=self._build_logprobs(choice), kv_transfer_params=response.kv_transfer_params, ) + @staticmethod + def _build_logprobs( + choice: ChatCompletionResponseChoice, + ) -> list[CohereLogprobItem] | None: + if ( + choice.logprobs is None + or not choice.logprobs.content + or choice.token_ids is None + ): + return None + return [ + CohereLogprobItem( + text=entry.token, token_ids=[token_id], logprobs=[entry.logprob] + ) + for entry, token_id in zip( + choice.logprobs.content, choice.token_ids, strict=True + ) + ] + def _create_chat_message(self, *args: Any, **kwargs: Any) -> ChatMessage: """Route response construction through the citation-carrying subclass. diff --git a/vllm/entrypoints/common/__init__.py b/vllm/entrypoints/common/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/vllm/entrypoints/offline_utils.py b/vllm/entrypoints/common/offline.py similarity index 100% rename from vllm/entrypoints/offline_utils.py rename to vllm/entrypoints/common/offline.py diff --git a/vllm/entrypoints/generate/api_router.py b/vllm/entrypoints/generate/api_router.py index c5703515d8..67a2561924 100644 --- a/vllm/entrypoints/generate/api_router.py +++ b/vllm/entrypoints/generate/api_router.py @@ -53,6 +53,12 @@ def register_generate_api_routers(app: FastAPI): register_generative_scoring_api_router(app) + from .structured_decisions.api_router import ( + register_structured_decisions_api_router, + ) + + register_structured_decisions_api_router(app) + async def init_generate_state( engine_client: "EngineClient", @@ -220,3 +226,31 @@ async def init_generate_state( state.openai_serving_models, request_logger=request_logger, ) + + from .structured_decisions.serving import ServingStructuredDecisions + from .structured_decisions.strategies import ReadContext, select_read_strategy + + strategy = None + if "generate" in supported_tasks: + try: + strategy_cls = select_read_strategy(engine_client.model_config) + strategy = strategy_cls( + ReadContext( + engine_client=engine_client, + online_renderer=state.online_renderer, + chat_template=resolved_chat_template, + chat_template_content_format=args.chat_template_content_format, + default_chat_template_kwargs=default_chat_template_kwargs, + ) + ) + except ValueError: + strategy = None + state.serving_structured_decisions = ( + ServingStructuredDecisions( + state.openai_serving_models, + strategy, + request_logger=request_logger, + ) + if strategy is not None + else None + ) diff --git a/vllm/entrypoints/generate/base/protocol.py b/vllm/entrypoints/generate/base/protocol.py index c183498143..89f6816c41 100644 --- a/vllm/entrypoints/generate/base/protocol.py +++ b/vllm/entrypoints/generate/base/protocol.py @@ -8,6 +8,7 @@ from typing import Annotated, Any, Literal, TypeAlias from pydantic import ( BaseModel, + BeforeValidator, Field, model_serializer, ) @@ -26,6 +27,12 @@ StopParam: TypeAlias = ( str | Annotated[list[str], Field(max_length=envs.VLLM_MAX_STOP_STRINGS)] | None ) +# `top_logprobs` is nullable in the OpenAI spec; null means the same as omitted. +TopLogprobsParam: TypeAlias = Annotated[ + int, + BeforeValidator(lambda v: 0 if v is None else v, json_schema_input_type=int | None), +] + _CACHE_SALT_FORBIDDEN_CHARS = frozenset("@/\\\x00") _MAX_CACHE_SALT_LENGTH = 128 @@ -223,7 +230,11 @@ def validate_structural_tag_response_format( def validate_structural_tag_payload(payload: Any, *, parameter: str) -> None: - from vllm.sampling_params import SamplingParams, StructuredOutputsParams + from vllm.sampling_params import ( + SamplingParams, + StructuredOutputsParams, + check_json_nesting, + ) from vllm.v1.structured_output.backend_xgrammar import validate_xgrammar_grammar if isinstance(payload, str) and not payload: @@ -232,6 +243,9 @@ def validate_structural_tag_payload(payload: Any, *, parameter: str) -> None: parameter=parameter, ) + if isinstance(payload, str): + # Raised here so the error is not reported as a malformed tag below + check_json_nesting(payload, structural_tag=True) try: validate_xgrammar_grammar( SamplingParams( diff --git a/vllm/entrypoints/generate/base/serving.py b/vllm/entrypoints/generate/base/serving.py index 5142346df6..d1f3e94b00 100644 --- a/vllm/entrypoints/generate/base/serving.py +++ b/vllm/entrypoints/generate/base/serving.py @@ -32,6 +32,7 @@ from vllm.logger import init_logger from vllm.logprobs import Logprob, PromptLogprobs from vllm.lora.request import LoRARequest from vllm.tokenizers import TokenizerLike +from vllm.tokenizers.detokenizer_utils import convert_ids_list_to_tokens from vllm.tracing import ( contains_trace_headers, extract_trace_headers, @@ -346,32 +347,33 @@ def format_token_id_placeholder(token_id: int) -> str: return f"token_id:{token_id}" -def resolve_token_id_placeholder( - token: str, tokenizer: TokenizerLike -) -> tuple[str, list[int] | None]: - """Decode a 'token_id:N' placeholder back to a token string and UTF-8 bytes. +def decode_token_ids( + token_ids: list[int], tokenizer: TokenizerLike +) -> list[tuple[str, list[int] | None]]: + """Decode token ids individually to their token strings and UTF-8 bytes. - Returns (token, None) unchanged if token is not a placeholder. - This is the inverse of format_token_id_placeholder / _get_decoded_token - when return_as_token_id=True. + Uses the engine's per-token detokenization, which restores the + SentencePiece leading space that `convert_tokens_to_string` drops, so the + strings match the coupled endpoints. Ids are decoded in one batch (callers + pass a position's sampled id together with its top-k ids). An id with no + vocab entry decodes to ("", None). """ - suffix = token.removeprefix("token_id:") - if suffix == token: - return token, None - try: - token_id = int(suffix) - except ValueError: - return token, None - token_repr = tokenizer.convert_ids_to_tokens([token_id])[0] - if token_repr is None: - logger.warning_once( - "resolve_token_id_placeholder: token_id %d has no vocab entry; " - "substituting empty string", - token_id, - ) - return "", None - token_str = tokenizer.convert_tokens_to_string([token_repr]) - return token_str, list(token_str.encode("utf-8", errors="replace")) + pieces = tokenizer.convert_ids_to_tokens(token_ids) + known = [tid for tid, piece in zip(token_ids, pieces) if piece is not None] + decoded = iter(convert_ids_list_to_tokens(tokenizer, known)) + out: list[tuple[str, list[int] | None]] = [] + for tid, piece in zip(token_ids, pieces): + if piece is None: + logger.warning_once( + "decode_token_ids: token_id %d has no vocab entry; " + "substituting empty string", + tid, + ) + out.append(("", None)) + continue + token_str = next(decoded) + out.append((token_str, list(token_str.encode("utf-8", errors="replace")))) + return out def clamp_prompt_logprobs( diff --git a/vllm/entrypoints/generate/beam_search/offline.py b/vllm/entrypoints/generate/beam_search/offline.py index b7095131af..0acd0b43a5 100644 --- a/vllm/entrypoints/generate/beam_search/offline.py +++ b/vllm/entrypoints/generate/beam_search/offline.py @@ -8,7 +8,7 @@ import torch from tqdm import tqdm from vllm import RequestOutput, TextPrompt, TokensPrompt -from vllm.entrypoints.offline_utils import OfflineInferenceMixin +from vllm.entrypoints.common.offline import OfflineInferenceMixin from vllm.logger import init_logger from vllm.lora.request import LoRARequest from vllm.pooling_params import PoolingParams @@ -82,7 +82,7 @@ class BeamSearchOfflineMixin(OfflineInferenceMixin): temperature = params.temperature ignore_eos = params.ignore_eos length_penalty = params.length_penalty - self.llm_engine.vllm_config._check_watermarking_unsupported(beam_search=True) + self.llm_engine.input_processor.resolve_watermarking(params) tokenizer = self.renderer.get_tokenizer() eos_token_id = tokenizer.eos_token_id @@ -120,6 +120,7 @@ class BeamSearchOfflineMixin(OfflineInferenceMixin): logprobs=2 * beam_width, max_tokens=1, temperature=temperature, + watermarking=False, detokenize=False, skip_clone=True, # Internal beam search, safe to skip clone ) @@ -457,6 +458,7 @@ class BeamSearchOfflineMixin(OfflineInferenceMixin): logprobs=base_params.logprobs, max_tokens=1, temperature=base_params.temperature, + watermarking=base_params.watermarking, detokenize=False, allowed_token_ids=( allowed_ids diff --git a/vllm/entrypoints/generate/beam_search/online.py b/vllm/entrypoints/generate/beam_search/online.py index ac7f0f884f..5a95f40d31 100644 --- a/vllm/entrypoints/generate/beam_search/online.py +++ b/vllm/entrypoints/generate/beam_search/online.py @@ -40,6 +40,7 @@ class BeamSearchOnlineMixin(ABC): temperature = params.temperature length_penalty = params.length_penalty include_stop_str_in_output = params.include_stop_str_in_output + self.engine_client.input_processor.resolve_watermarking(params) tokenizer = self.renderer.get_tokenizer() eos_token_id = tokenizer.eos_token_id @@ -62,6 +63,7 @@ class BeamSearchOnlineMixin(ABC): logprobs=logprobs_num, max_tokens=1, temperature=temperature, + watermarking=False, detokenize=False, ) all_beams = [ diff --git a/vllm/entrypoints/generate/generative_scoring/serving.py b/vllm/entrypoints/generate/generative_scoring/serving.py index cb0c4995bf..74dbfd0759 100644 --- a/vllm/entrypoints/generate/generative_scoring/serving.py +++ b/vllm/entrypoints/generate/generative_scoring/serving.py @@ -11,13 +11,15 @@ logits (task="generate"). import asyncio import math import time -from collections.abc import AsyncGenerator, Mapping -from typing import Literal +from collections.abc import Mapping +from typing import Any, Literal from fastapi import Request -from pydantic import Field +from pydantic import Field, model_validator from vllm.engine.protocol import EngineClient +from vllm.entrypoints.generate.base.protocol import validate_cache_salt +from vllm.entrypoints.generate.label_reads import next_token_label_reads from vllm.entrypoints.openai.models.serving import OpenAIServingModels from vllm.entrypoints.serve.engine.protocol import ( ErrorResponse, @@ -28,7 +30,6 @@ from vllm.entrypoints.serve.engine.serving import BaseServing from vllm.entrypoints.serve.utils.request_logger import RequestLogger from vllm.inputs import EngineInput, tokens_input from vllm.logger import init_logger -from vllm.outputs import RequestOutput from vllm.sampling_params import SamplingParams from vllm.tokenizers import TokenizerLike from vllm.tracing import ( @@ -37,7 +38,6 @@ from vllm.tracing import ( log_tracing_disabled_warning, ) from vllm.utils import random_uuid -from vllm.utils.async_utils import merge_async_iterators logger = init_logger(__name__) @@ -60,6 +60,7 @@ class GenerativeScoringRequest(OpenAIBaseModel): the full vocab for those ids (False). item_first: If True, prepend items to query. Otherwise append items to query. add_special_tokens: Whether to add special tokens when tokenizing. + cache_salt: Optional salt for prefix caching. """ @@ -92,6 +93,17 @@ class GenerativeScoringRequest(OpenAIBaseModel): default=True, description="Whether to add special tokens when tokenizing.", ) + cache_salt: str | None = Field( + default=None, + description=( + "If specified, the prefix cache will be salted with the provided " + "string to prevent an attacker to guess prompts in multi-user " + "environments. The salt should be random, protected from " + "access by 3rd parties, and long enough to be " + "unpredictable (e.g., 43 characters base64-encoded, corresponding " + "to 256 bit)." + ), + ) priority: int = Field( default=0, ge=-(2**63), @@ -105,6 +117,13 @@ class GenerativeScoringRequest(OpenAIBaseModel): description="The request_id related to this request.", ) + @model_validator(mode="before") + @classmethod + def check_cache_salt_support(cls, data: Any) -> Any: + if isinstance(data, dict): + validate_cache_salt(data.get("cache_salt")) + return data + class GenerativeScoringItemResult(OpenAIBaseModel): """Result for a single item in the generative scoring response. @@ -261,6 +280,7 @@ class ServingGenerativeScoring(BaseServing): logprobs=len(request.label_token_ids), logprob_token_ids=request.label_token_ids, n=1, + watermarking=False, ) # Get trace headers @@ -270,37 +290,28 @@ class ServingGenerativeScoring(BaseServing): else await self._get_trace_headers(raw_request.headers) ) - # Schedule requests for all inputs - generators: list[AsyncGenerator[RequestOutput, None]] = [] for i, engine_input in enumerate(engine_inputs): - request_id_item = f"{request_id}-{i}" - self._log_inputs( - request_id_item, + f"{request_id}-{i}", engine_input, params=sampling_params, lora_request=lora_request, ) - generator = self.engine_client.generate( - engine_input, - sampling_params, - request_id_item, + try: + reads = await next_token_label_reads( + self.engine_client, + engine_inputs, + [sampling_params] * len(engine_inputs), + request_id, lora_request=lora_request, trace_headers=trace_headers, priority=request.priority, ) - generators.append(generator) - - # Collect results - result_generator = merge_async_iterators(*generators) - results: list[RequestOutput | None] = [None] * len(engine_inputs) - - try: - async for i, res in result_generator: - results[i] = res except asyncio.CancelledError: return self.create_error_response("Client disconnected") + except ValueError as e: + return self.create_error_response(e) except Exception as e: logger.exception("Error during generation") return self.create_error_response(e) @@ -310,45 +321,8 @@ class ServingGenerativeScoring(BaseServing): total_prompt_tokens = 0 total_completion_tokens = 0 - for i, result in enumerate(results): - if result is None: - return self.create_error_response( - f"Failed to generate result for item {i}" - ) - - # Check for errors - if result.outputs and result.outputs[0].finish_reason == "error": - return self.create_error_response(f"Generation error for item {i}") - - # Get logprobs from the generated token - if not result.outputs or len(result.outputs) == 0: - return self.create_error_response(f"No output generated for item {i}") - - output = result.outputs[0] - if output.logprobs is None or len(output.logprobs) == 0: - return self.create_error_response( - f"No logprobs available for item {i}. " - "This might indicate an issue with logprobs configuration." - ) - - # The logprobs dict maps token_id -> Logprob object - # For logprobs=-1, this contains all vocab tokens - logprobs_dict = output.logprobs[0] - - # Extract logprobs for label tokens - label_logprobs: dict[int, float] = {} - missing_tokens = [] - for token_id in request.label_token_ids: - if token_id in logprobs_dict: - label_logprobs[token_id] = logprobs_dict[token_id].logprob - else: - missing_tokens.append(token_id) - - if missing_tokens: - return self.create_error_response( - f"Token IDs {missing_tokens} not found in logprobs for item {i}. " - "This might indicate the tokens are outside the model's vocabulary." - ) + for i, read in enumerate(reads): + label_logprobs = dict(zip(request.label_token_ids, read.logprobs)) # Compute probabilities based on apply_softmax setting token_probs = self._compute_probabilities( @@ -369,7 +343,7 @@ class ServingGenerativeScoring(BaseServing): # Update token counts total_prompt_tokens += prompt_token_counts[i] - total_completion_tokens += len(output.token_ids) + total_completion_tokens += len(read.result.outputs[0].token_ids) # Build response model_name = self.models.model_name(lora_request) @@ -441,7 +415,9 @@ class ServingGenerativeScoring(BaseServing): if len(prompt_token_ids) > max_prompt_len: prompt_token_ids = prompt_token_ids[:max_prompt_len] - engine_inputs.append(tokens_input(prompt_token_ids)) + engine_inputs.append( + tokens_input(prompt_token_ids, cache_salt=request.cache_salt) + ) prompt_token_counts.append(len(prompt_token_ids)) return engine_inputs, prompt_token_counts diff --git a/vllm/entrypoints/generate/label_reads.py b/vllm/entrypoints/generate/label_reads.py new file mode 100644 index 0000000000..eda61b22bc --- /dev/null +++ b/vllm/entrypoints/generate/label_reads.py @@ -0,0 +1,76 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""One-token reads that return the logprobs of chosen label tokens.""" + +from collections.abc import AsyncGenerator, Mapping, Sequence +from dataclasses import dataclass + +from vllm.engine.protocol import EngineClient +from vllm.inputs import EngineInput +from vllm.lora.request import LoRARequest +from vllm.outputs import RequestOutput +from vllm.sampling_params import SamplingParams +from vllm.utils.async_utils import merge_async_iterators + + +@dataclass(frozen=True) +class LabelRead: + result: RequestOutput + logprobs: list[float] + """Full-vocabulary logprob of each label token, in ``logprob_token_ids`` + order.""" + + +async def next_token_label_reads( + engine_client: EngineClient, + engine_inputs: Sequence[EngineInput], + sampling_params: Sequence[SamplingParams], + request_id: str, + *, + lora_request: LoRARequest | None = None, + trace_headers: Mapping[str, str] | None = None, + priority: int = 0, +) -> list[LabelRead]: + """Runs one read per input at once, as request ``{request_id}-{i}``. Each + read's params set ``max_tokens=1`` and the label tokens as + ``logprob_token_ids``. Raises ValueError naming the item when a read has + no output or lacks a label's logprob.""" + generators: list[AsyncGenerator[RequestOutput, None]] = [ + engine_client.generate( + engine_input, + params, + f"{request_id}-{i}", + lora_request=lora_request, + trace_headers=trace_headers, + priority=priority, + ) + for i, (engine_input, params) in enumerate(zip(engine_inputs, sampling_params)) + ] + results: list[RequestOutput | None] = [None] * len(generators) + async for i, res in merge_async_iterators(*generators): + results[i] = res + + reads = [] + for i, (result, params) in enumerate(zip(results, sampling_params)): + if result is None: + raise ValueError(f"Failed to generate result for item {i}") + if not result.outputs: + raise ValueError(f"No output generated for item {i}") + output = result.outputs[0] + if output.finish_reason == "error": + raise ValueError(f"Generation error for item {i}") + if not output.logprobs: + raise ValueError( + f"No logprobs available for item {i}. " + "This might indicate an issue with logprobs configuration." + ) + logprobs = output.logprobs[0] + label_ids = params.logprob_token_ids or [] + missing = [t for t in label_ids if t not in logprobs] + if missing: + raise ValueError( + f"Token IDs {missing} not found in logprobs for item {i}. " + "This might indicate the tokens are outside the model's vocabulary." + ) + reads.append(LabelRead(result, [logprobs[t].logprob for t in label_ids])) + return reads diff --git a/vllm/v1/worker/gpu/spec_decode/autoregressive/__init__.py b/vllm/entrypoints/generate/structured_decisions/__init__.py similarity index 100% rename from vllm/v1/worker/gpu/spec_decode/autoregressive/__init__.py rename to vllm/entrypoints/generate/structured_decisions/__init__.py diff --git a/vllm/entrypoints/generate/structured_decisions/api_router.py b/vllm/entrypoints/generate/structured_decisions/api_router.py new file mode 100644 index 0000000000..5729b58b2e --- /dev/null +++ b/vllm/entrypoints/generate/structured_decisions/api_router.py @@ -0,0 +1,52 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +from http import HTTPStatus + +from fastapi import APIRouter, Depends, FastAPI, Request +from fastapi.responses import JSONResponse + +from vllm.entrypoints.serve.engine.protocol import ErrorResponse +from vllm.entrypoints.serve.utils.api_utils import ( + load_aware_call, + validate_json_request, + with_cancellation, +) + +from .protocol import StructuredDecisionRequest +from .serving import ServingStructuredDecisions + +router = APIRouter() + + +def structured_decisions(request: Request) -> ServingStructuredDecisions | None: + return getattr(request.app.state, "serving_structured_decisions", None) + + +@router.post( + "/v1/systemone", + dependencies=[Depends(validate_json_request)], + responses={ + HTTPStatus.BAD_REQUEST.value: {"model": ErrorResponse}, + HTTPStatus.NOT_FOUND.value: {"model": ErrorResponse}, + HTTPStatus.INTERNAL_SERVER_ERROR.value: {"model": ErrorResponse}, + HTTPStatus.NOT_IMPLEMENTED.value: {"model": ErrorResponse}, + }, +) +@with_cancellation +@load_aware_call +async def create_structured_decision( + request: StructuredDecisionRequest, raw_request: Request +): + handler = structured_decisions(raw_request) + if handler is None: + raise NotImplementedError( + "The model does not support the structured decision API" + ) + result = await handler.create_decision(request, raw_request) + if isinstance(result, ErrorResponse): + return JSONResponse(content=result.model_dump(), status_code=result.error.code) + return JSONResponse(content=result.model_dump()) + + +def register_structured_decisions_api_router(app: FastAPI): + app.include_router(router) diff --git a/vllm/entrypoints/generate/structured_decisions/protocol.py b/vllm/entrypoints/generate/structured_decisions/protocol.py new file mode 100644 index 0000000000..b15fee171f --- /dev/null +++ b/vllm/entrypoints/generate/structured_decisions/protocol.py @@ -0,0 +1,108 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Request and response models of the structured decisions API (/v1/systemone).""" + +import time +from typing import Any, Literal + +from pydantic import Field, model_validator + +from vllm.config import ModelConfig +from vllm.entrypoints.chat_utils import ChatTemplateContentFormatOption +from vllm.entrypoints.generate.base.protocol import validate_cache_salt +from vllm.entrypoints.serve.engine.protocol import OpenAIBaseModel +from vllm.renderers import ChatParams, TokenizeParams, merge_kwargs +from vllm.utils import random_uuid + + +class QuestionSpec(OpenAIBaseModel): + type: str + instructions: Any = "" + criteria: Any = None + + +class StructuredDecisionRequest(OpenAIBaseModel): + model: str | None = None + state: Any = Field(..., description="What the questions are about: text or JSON.") + questions: dict[str, QuestionSpec] = Field( + ..., description="Question id to question, asked in this order." + ) + instructions: str | None = Field( + default=None, description="Context placed ahead of the questions." + ) + chat_template_kwargs: dict[str, Any] | None = None + cache_salt: str | None = Field( + default=None, + description=( + "If specified, the prefix cache will be salted with the provided " + "string to prevent an attacker to guess prompts in multi-user " + "environments. The salt should be random, protected from " + "access by 3rd parties, and long enough to be " + "unpredictable (e.g., 43 characters base64-encoded, corresponding " + "to 256 bit)." + ), + ) + priority: int = Field(default=0, ge=-(2**63), le=2**63 - 1) + request_id: str = Field(default_factory=random_uuid) + + @model_validator(mode="before") + @classmethod + def check_cache_salt_support(cls, data: Any) -> Any: + if isinstance(data, dict): + validate_cache_salt(data.get("cache_salt")) + return data + + +class DecisionUsage(OpenAIBaseModel): + input_tokens: int + output_tokens: int + + +class QuestionDiagnostics(OpenAIBaseModel): + label_mass: float = Field( + description="Probability the model put on the labels, over the full " + "vocabulary. A low value means most of the probability went to tokens " + "that are not labels." + ) + argmax_is_label: bool + + +class StructuredDecisionResponse(OpenAIBaseModel): + id: str + object: Literal["structured_decision"] = "structured_decision" + created: int = Field(default_factory=lambda: int(time.time())) + model: str + answers: dict[str, dict[str, Any]] + usage: DecisionUsage + diagnostics: dict[str, QuestionDiagnostics] + + +class ReadPromptRequest(OpenAIBaseModel): + """Chat options for one read's prompt: the state and the question in the + user turn, ending at the generation prompt so the label is the reply's + first token. Thinking is off unless the request turns it on, or the label + would follow a thought rather than start the reply.""" + + chat_template_kwargs: dict[str, Any] | None = None + cache_salt: str | None = None + + def build_chat_params( + self, + default_template: str | None, + default_template_content_format: ChatTemplateContentFormatOption, + ) -> ChatParams: + return ChatParams( + chat_template=default_template, + chat_template_content_format=default_template_content_format, + chat_template_kwargs=merge_kwargs( + merge_kwargs({"enable_thinking": False}, self.chat_template_kwargs), + dict(add_generation_prompt=True, continue_final_message=False), + ), + ) + + def build_tok_params(self, model_config: ModelConfig) -> TokenizeParams: + return TokenizeParams( + max_total_tokens=model_config.max_model_len, + max_output_tokens=1, + add_special_tokens=False, + ) diff --git a/vllm/entrypoints/generate/structured_decisions/question_types.py b/vllm/entrypoints/generate/structured_decisions/question_types.py new file mode 100644 index 0000000000..0e80694b14 --- /dev/null +++ b/vllm/entrypoints/generate/structured_decisions/question_types.py @@ -0,0 +1,240 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Question types for structured decisions. + +A question type turns a request's criteria into options, writes the question +for the model with a label per option, and builds the answer from the label +probabilities. +""" + +import math +import string +from abc import ABC, abstractmethod +from dataclasses import dataclass +from typing import Any, ClassVar + + +class StructuredDecisionError(ValueError): + """An invalid request. Returned as a 400.""" + + +#: A choice labels its options A to Z in order. A single capital letter is one +#: token at the start of a reply for the tokenizers tested. Every read checks it. +LABELS = tuple(string.ascii_uppercase) + + +@dataclass(frozen=True) +class Option: + name: str + description: str | None = None + + +@dataclass(frozen=True) +class Question: + id: str + type: "QuestionType" + instructions: str + options: tuple[Option, ...] + labels: tuple[str, ...] + + +class QuestionType(ABC): + name: ClassVar[str] + #: The labels of a question's options, in option order. + label_set: ClassVar[tuple[str, ...]] = LABELS + reply_instruction: ClassVar[str] = "Answer with the letter of one option only." + + @abstractmethod + def parse_options(self, qid: str, criteria: Any) -> list[Option]: ... + + @abstractmethod + def answer( + self, question: Question, probs: list[float], label_mass: float + ) -> dict[str, Any]: + """The answer for ``question``. ``probs[i]`` is the probability of + ``question.labels[i]`` among the labels, and the list sums to 1. + ``label_mass`` is the labels' total probability over the vocabulary.""" + + def reply_line(self, label: str, option: Option) -> str | None: + """The option's line in the prompt, or None to leave it out.""" + return ( + f"{label}: {option.name} - {option.description}" + if option.description + else f"{label}: {option.name}" + ) + + def prompt(self, question: Question) -> str: + """The question as the model reads it, after the state.""" + lines = [f"Question: {question.instructions}"] if question.instructions else [] + for label, o in zip(question.labels, question.options): + line = self.reply_line(label, o) + if line is not None: + lines.append(line) + lines.append(self.reply_instruction) + return "\n".join(lines) + + +QUESTION_TYPES: dict[str, QuestionType] = {} + + +def register_question_type(cls: type[QuestionType]) -> type[QuestionType]: + if cls.name in QUESTION_TYPES: + raise ValueError(f"question type {cls.name!r} is already registered") + QUESTION_TYPES[cls.name] = cls() + return cls + + +def get_question_type(name: str) -> QuestionType: + try: + return QUESTION_TYPES[name] + except KeyError: + raise StructuredDecisionError( + f"unknown question type {name!r}; supported: {sorted(QUESTION_TYPES)}" + ) from None + + +def build_question( + qid: str, + type_name: str, + instructions: Any, + criteria: Any, + max_options: int, +) -> Question: + if not qid: + raise StructuredDecisionError("question ids must be non-empty") + qtype = get_question_type(type_name) + options = qtype.parse_options(qid, criteria) + if not options: + raise StructuredDecisionError(f"question {qid!r}: needs at least one option") + names = [o.name for o in options] + if len(set(names)) != len(names): + raise StructuredDecisionError(f"question {qid!r}: duplicate option names") + limit = min(max_options, len(qtype.label_set)) + if len(options) > limit: + raise StructuredDecisionError( + f"question {qid!r}: at most {limit} options for this model" + ) + if not isinstance(instructions, str): + instructions = "" if instructions is None else str(instructions) + return Question( + id=qid, + type=qtype, + instructions=instructions, + options=tuple(options), + labels=qtype.label_set[: len(options)], + ) + + +def argmax(values: list[float]) -> int: + return max(range(len(values)), key=values.__getitem__) + + +def label_softmax(logprobs: list[float]) -> list[float]: + """Softmax of the label logprobs.""" + top = max(logprobs) + weights = [math.exp(lp - top) for lp in logprobs] + total = sum(weights) + return [w / total for w in weights] + + +@register_question_type +class ChoiceQuestion(QuestionType): + """Pick one option. ``criteria`` maps each option name to a description, + or to null.""" + + name = "choice" + + def parse_options(self, qid: str, criteria: Any) -> list[Option]: + if not isinstance(criteria, dict) or not criteria: + raise StructuredDecisionError( + f"question {qid!r}: choice criteria must map option names to " + "a description or null" + ) + return [ + Option(str(name), None if desc is None else str(desc)) + for name, desc in criteria.items() + ] + + def answer( + self, question: Question, probs: list[float], label_mass: float + ) -> dict[str, Any]: + top = argmax(probs) + return { + "type": self.name, + "choice": question.options[top].name, + "probabilities": {a.name: p for a, p in zip(question.options, probs)}, + # The chosen label's probability over the whole vocabulary, so a + # read that mostly wanted a non-label token reports low confidence. + "confidence": probs[top] * label_mass, + } + + +@register_question_type +class NoulQuestion(QuestionType): + """Yes or no. ``criteria`` may describe what true and false mean.""" + + name = "noul" + label_set = ("yes", "no") + reply_instruction = "Answer with yes or no only." + + def parse_options(self, qid: str, criteria: Any) -> list[Option]: + if criteria is not None and ( + not isinstance(criteria, dict) or not criteria.keys() <= {"true", "false"} + ): + raise StructuredDecisionError( + f"question {qid!r}: noul criteria must be an object with true and false" + ) + criteria = criteria or {} + true = criteria.get("true") + false = criteria.get("false") + return [ + Option("yes", None if true is None else str(true)), + Option("no", None if false is None else str(false)), + ] + + def reply_line(self, label: str, option: Option) -> str | None: + return f"{label}: {option.description}" if option.description else None + + def answer( + self, question: Question, probs: list[float], label_mass: float + ) -> dict[str, Any]: + return { + "type": self.name, + "noul": probs[0], + "probabilities": {"yes": probs[0], "no": probs[1]}, + # The chosen label's probability over the whole vocabulary, like a + # choice's confidence. + "confidence": max(probs) * label_mass, + } + + +@register_question_type +class ScoreQuestion(QuestionType): + """Rate the state on an ordered scale. ``criteria`` is the ordered list of + level names, from the first to the last level of the scale.""" + + name = "score" + # 0-indexed like the answer, so a level's label is its score. + label_set = tuple(string.digits) + reply_instruction = "Answer with the number of one level only." + + def parse_options(self, qid: str, criteria: Any) -> list[Option]: + if not isinstance(criteria, list) or len(criteria) < 2: + raise StructuredDecisionError( + f"question {qid!r}: score criteria must be an ordered list of levels" + ) + return [Option(str(level)) for level in criteria] + + def answer( + self, question: Question, probs: list[float], label_mass: float + ) -> dict[str, Any]: + top = argmax(probs) + return { + "type": self.name, + # The expected level, 0-indexed, so a level's index in the request's + # criteria is its value. + "score": sum(i * p for i, p in enumerate(probs)), + "legend": {str(i): o.name for i, o in enumerate(question.options)}, + "probabilities": {str(i): p for i, p in enumerate(probs)}, + "confidence": probs[top] * label_mass, + } diff --git a/vllm/entrypoints/generate/structured_decisions/serving.py b/vllm/entrypoints/generate/structured_decisions/serving.py new file mode 100644 index 0000000000..e6e990bd46 --- /dev/null +++ b/vllm/entrypoints/generate/structured_decisions/serving.py @@ -0,0 +1,144 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import asyncio +import json +from collections.abc import Mapping +from typing import Any + +from fastapi import Request + +from vllm.entrypoints.openai.models.serving import OpenAIServingModels +from vllm.entrypoints.serve.engine.protocol import ErrorResponse +from vllm.entrypoints.serve.engine.serving import BaseServing +from vllm.entrypoints.serve.utils.request_logger import RequestLogger +from vllm.logger import init_logger +from vllm.tracing import ( + contains_trace_headers, + extract_trace_headers, + log_tracing_disabled_warning, +) + +from .protocol import ( + DecisionUsage, + QuestionDiagnostics, + StructuredDecisionRequest, + StructuredDecisionResponse, +) +from .question_types import Question, StructuredDecisionError, build_question +from .strategies import DecisionLimits, ReadStrategy + +logger = init_logger(__name__) + + +def state_text(state: Any) -> str: + return state if isinstance(state, str) else json.dumps(state, ensure_ascii=False) + + +def parse_questions( + request: StructuredDecisionRequest, + limits: DecisionLimits, +) -> list[Question]: + if not request.questions: + raise StructuredDecisionError("questions: needs at least one question") + if len(request.questions) > limits.max_questions: + raise StructuredDecisionError( + f"questions: at most {limits.max_questions} for this model" + ) + questions = [] + for qid, spec in request.questions.items(): + if spec.model_extra: + raise StructuredDecisionError( + f"question {qid!r}: unknown field(s) {sorted(spec.model_extra)}" + ) + questions.append( + build_question( + qid, + spec.type, + spec.instructions, + spec.criteria, + limits.max_options, + ) + ) + return questions + + +class ServingStructuredDecisions(BaseServing): + def __init__( + self, + models: OpenAIServingModels, + strategy: ReadStrategy, + *, + request_logger: RequestLogger | None = None, + ) -> None: + model_config = strategy.context.engine_client.model_config + super().__init__( + models=models, model_config=model_config, request_logger=request_logger + ) + self.strategy = strategy + self.limits = strategy.limits() + + async def _get_trace_headers( + self, headers: Mapping[str, str] + ) -> Mapping[str, str] | None: + if not contains_trace_headers(headers): + return None + if not await self.strategy.context.engine_client.is_tracing_enabled(): + log_tracing_disabled_warning() + return None + return extract_trace_headers(headers) + + async def create_decision( + self, + request: StructuredDecisionRequest, + raw_request: Request | None = None, + ) -> StructuredDecisionResponse | ErrorResponse: + if (error := await self._check_model(request)) is not None: # type: ignore[arg-type] + return error + engine_client = self.strategy.context.engine_client + if engine_client.errored: + raise engine_client.dead_error + + base_id = self._base_request_id(raw_request, default=request.request_id) + request_id = f"decision-{base_id}" + try: + questions = parse_questions(request, self.limits) + lora_request = self._maybe_get_adapters(request) # type: ignore[arg-type] + engine_client.check_admission(len(questions)) + trace_headers = ( + None + if raw_request is None + else await self._get_trace_headers(raw_request.headers) + ) + reads = await self.strategy.read( + questions, + request.instructions, + state_text(request.state), + request_id=request_id, + chat_template_kwargs=request.chat_template_kwargs, + lora_request=lora_request, + priority=request.priority, + cache_salt=request.cache_salt, + trace_headers=trace_headers, + ) + except StructuredDecisionError as e: + return self.create_error_response(e) + except asyncio.CancelledError: + return self.create_error_response("Client disconnected") + + answers: dict[str, dict[str, Any]] = {} + diagnostics: dict[str, QuestionDiagnostics] = {} + for q, read in zip(questions, reads): + answers[q.id] = q.type.answer(q, read.probs, read.label_mass) + diagnostics[q.id] = QuestionDiagnostics( + label_mass=read.label_mass, argmax_is_label=read.argmax_is_label + ) + return StructuredDecisionResponse( + id=request_id, + model=self.models.model_name(lora_request), + answers=answers, + usage=DecisionUsage( + input_tokens=sum(r.input_tokens for r in reads), + output_tokens=sum(r.output_tokens for r in reads), + ), + diagnostics=diagnostics, + ) diff --git a/vllm/entrypoints/generate/structured_decisions/strategies.py b/vllm/entrypoints/generate/structured_decisions/strategies.py new file mode 100644 index 0000000000..65b5a9dd7b --- /dev/null +++ b/vllm/entrypoints/generate/structured_decisions/strategies.py @@ -0,0 +1,253 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Read strategies: how a model is asked for label probabilities. + +Only models in NEXT_TOKEN_ARCHITECTURES are currently supported. +""" + +import math +from abc import ABC, abstractmethod +from collections.abc import Mapping, Sequence +from dataclasses import dataclass +from typing import Any + +from vllm.config import ModelConfig +from vllm.engine.protocol import EngineClient +from vllm.entrypoints.chat_utils import ChatTemplateContentFormatOption +from vllm.entrypoints.generate.label_reads import next_token_label_reads +from vllm.inputs import EngineInput +from vllm.lora.request import LoRARequest +from vllm.renderers.inputs.preprocess import extract_prompt_components +from vllm.renderers.online_renderer import OnlineRenderer +from vllm.sampling_params import SamplingParams +from vllm.tokenizers import TokenizerLike + +from .protocol import ReadPromptRequest +from .question_types import ( + LABELS, + QUESTION_TYPES, + Question, + StructuredDecisionError, + label_softmax, +) + + +@dataclass +class ReadContext: + engine_client: EngineClient + online_renderer: OnlineRenderer + chat_template: str | None + chat_template_content_format: ChatTemplateContentFormatOption + default_chat_template_kwargs: dict[str, Any] + + +@dataclass(frozen=True) +class DecisionLimits: + max_questions: int + max_options: int + + +@dataclass +class QuestionRead: + probs: list[float] + label_mass: float + argmax_is_label: bool + input_tokens: int + output_tokens: int + + +class ReadStrategy(ABC): + def __init__(self, context: ReadContext): + self.context = context + + @abstractmethod + def limits(self) -> DecisionLimits: ... + + @abstractmethod + async def read( + self, + questions: list[Question], + instructions: str | None, + state: str, + *, + request_id: str, + chat_template_kwargs: dict[str, Any] | None, + lora_request: LoRARequest | None, + priority: int, + cache_salt: str | None = None, + trace_headers: Mapping[str, str] | None = None, + ) -> list[QuestionRead]: + """One read per question, in order.""" + + +def reply_label_ids( + tokenizer: TokenizerLike, + prompt_ids: Sequence[int], + labels: Sequence[str] = LABELS, +) -> tuple[list[int], list[int]]: + """The prompt's ids after its last added token, and the token each of + ``labels`` adds after them as the first token of the reply. Raises + ValueError if a label is not one distinct token there.""" + added = set(tokenizer.get_added_vocab().values()) + start = max((i + 1 for i, t in enumerate(prompt_ids) if t in added), default=0) + tail = list(prompt_ids[start:]) + text = tokenizer.decode(tail) + ids: list[int] = [] + for label in labels: + extended = tokenizer.encode(text + label, add_special_tokens=False) + if extended[:-1] != tail or extended[-1] in ids: + raise ValueError( + f"label {label!r} is not one distinct token after this model's " + "chat prompt" + ) + ids.append(extended[-1]) + return tail, ids + + +class NextTokenStrategy(ReadStrategy): + """Autoregressive models. Each question is one request: the state, then the + question with its labeled options, and the logprobs of the label tokens as + the reply's first token. The requests share the state. With prefix caching, + it is prefilled once.""" + + def __init__(self, context: ReadContext): + super().__init__(context) + tokenizer = context.online_renderer.renderer.get_tokenizer() + probe = tokenizer.apply_chat_template( + [{"role": "user", "content": "x"}], + **{ + **context.default_chat_template_kwargs, + "chat_template": context.chat_template, + "add_generation_prompt": True, + "tokenize": True, + "return_dict": False, + "enable_thinking": False, + }, + ) + if isinstance(probe, str): + probe = tokenizer.encode(probe, add_special_tokens=False) + # Every prompt ends with the same generation prompt, so a label's token + # is the same for every question. + self.tail, _ = reply_label_ids(tokenizer, probe, ()) + self.label_ids: dict[str, int] = {} + for qtype in QUESTION_TYPES.values(): + _, ids = reply_label_ids(tokenizer, probe, qtype.label_set) + self.label_ids.update(zip(qtype.label_set, ids)) + + def limits(self) -> DecisionLimits: + return DecisionLimits(max_questions=64, max_options=len(LABELS)) + + def _read_request( + self, chat_template_kwargs: dict[str, Any] | None, cache_salt: str | None = None + ) -> ReadPromptRequest: + if (chat_template_kwargs or {}).get("enable_thinking"): + raise StructuredDecisionError( + "a read is the reply's first token, so thinking must be off" + ) + return ReadPromptRequest( + chat_template_kwargs=chat_template_kwargs, cache_salt=cache_salt + ) + + async def _render( + self, read_request: ReadPromptRequest, messages: list[dict[str, Any]] + ) -> tuple[EngineInput, list[int]]: + ctx = self.context + _, (engine_input,) = await ctx.online_renderer.preprocess_chat( + read_request, + messages, + default_template=ctx.chat_template, + default_template_content_format=ctx.chat_template_content_format, + default_template_kwargs=ctx.default_chat_template_kwargs, + ) + prompt_ids = extract_prompt_components( + ctx.engine_client.model_config, engine_input + ).token_ids + return engine_input, list(prompt_ids or []) + + async def read( + self, + questions: list[Question], + instructions: str | None, + state: str, + *, + request_id: str, + chat_template_kwargs: dict[str, Any] | None, + lora_request: LoRARequest | None, + priority: int, + cache_salt: str | None = None, + trace_headers: Mapping[str, str] | None = None, + ) -> list[QuestionRead]: + ctx = self.context + read_request = self._read_request(chat_template_kwargs, cache_salt) + + slots, engine_inputs = [], [] + for q in questions: + messages = [{"role": "user", "content": f"{state}\n\n{q.type.prompt(q)}"}] + if instructions: + messages.insert(0, {"role": "system", "content": instructions}) + engine_input, prompt_ids = await self._render(read_request, messages) + if prompt_ids[len(prompt_ids) - len(self.tail) :] != self.tail: + raise StructuredDecisionError( + "these chat options end the prompt differently, so the " + "labels' tokens are unknown" + ) + slots.append([self.label_ids[label] for label in q.labels]) + engine_inputs.append(engine_input) + + label_reads = await next_token_label_reads( + ctx.engine_client, + engine_inputs, + [ + SamplingParams(max_tokens=1, temperature=0.0, logprob_token_ids=ids) + for ids in slots + ], + request_id, + lora_request=lora_request, + trace_headers=trace_headers, + priority=priority, + ) + + reads = [] + for ids, label_read in zip(slots, label_reads): + output = label_read.result.outputs[0] + reads.append( + QuestionRead( + probs=label_softmax(label_read.logprobs), + label_mass=sum(math.exp(lp) for lp in label_read.logprobs), + argmax_is_label=bool(output.token_ids) + and output.token_ids[0] in ids, + input_tokens=len(label_read.result.prompt_token_ids or ()), + output_tokens=len(output.token_ids), + ) + ) + return reads + + +NEXT_TOKEN_ARCHITECTURES = frozenset( + { + "Qwen3ForCausalLM", + "Qwen3_5ForConditionalGeneration", + "Qwen3_5MoeForConditionalGeneration", + } +) + + +# label_mass sums the labels' full-vocabulary probabilities. The logprobs modes +# return those probabilities. The logits modes return raw logits instead. +LOGPROBS_MODES = frozenset({"raw_logprobs", "processed_logprobs"}) + + +def select_read_strategy(model_config: ModelConfig) -> type[ReadStrategy]: + """Raise ValueError when the model cannot serve structured decisions.""" + if model_config.architecture not in NEXT_TOKEN_ARCHITECTURES: + raise ValueError( + "The structured decisions API does not support " + f"{model_config.architecture}. Supported architectures: " + f"{sorted(NEXT_TOKEN_ARCHITECTURES)}" + ) + if model_config.logprobs_mode not in LOGPROBS_MODES: + raise ValueError( + "Structured decisions need --logprobs-mode raw_logprobs or " + f"processed_logprobs, not {model_config.logprobs_mode}" + ) + return NextTokenStrategy diff --git a/vllm/entrypoints/launchers/api_server/entry.py b/vllm/entrypoints/launchers/api_server/entry.py index 28bb696f2c..e5ba536e11 100644 --- a/vllm/entrypoints/launchers/api_server/entry.py +++ b/vllm/entrypoints/launchers/api_server/entry.py @@ -8,7 +8,7 @@ import signal import socket import tempfile from argparse import Namespace -from collections.abc import AsyncIterator +from collections.abc import AsyncIterator, MutableSequence from contextlib import asynccontextmanager from typing import Any @@ -118,6 +118,7 @@ async def build_and_serve( listen_address: str, sock: socket.socket, args: Namespace, + peer_loads: tuple[MutableSequence[int], int] | None = None, **uvicorn_kwargs, ) -> asyncio.Task: """Build FastAPI app, initialize state, and start serving. @@ -142,6 +143,7 @@ async def build_and_serve( app, sock=sock, enable_ssl_refresh=args.enable_ssl_refresh, + peer_loads=peer_loads, host=args.host, port=args.port, log_level=args.uvicorn_log_level, @@ -185,12 +187,17 @@ async def run_server_worker( if args.reasoning_parser_plugin and len(args.reasoning_parser_plugin) > 3: ReasoningParserManager.import_reasoning_parser(args.reasoning_parser_plugin) + peer_loads = None + if client_config and "api_server_loads" in client_config: + loads = client_config.pop("api_server_loads") + peer_loads = (loads, client_config["client_index"]) + async with build_async_engine_client( args, client_config=client_config, ) as engine_client: shutdown_task = await build_and_serve( - engine_client, listen_address, sock, args, **uvicorn_kwargs + engine_client, listen_address, sock, args, peer_loads, **uvicorn_kwargs ) # NB: Await server shutdown only after the backend context is exited try: diff --git a/vllm/entrypoints/launchers/api_server/routers.py b/vllm/entrypoints/launchers/api_server/routers.py index 39d670c019..40d04862af 100644 --- a/vllm/entrypoints/launchers/api_server/routers.py +++ b/vllm/entrypoints/launchers/api_server/routers.py @@ -36,6 +36,10 @@ def register_api_routers( register_vllm_dev_api_routers(app) + from vllm.entrypoints.rl.factories import register_rl_api_routers + + register_rl_api_routers(app) + if "generate" in supported_tasks: from vllm.entrypoints.generate.api_router import ( register_generate_api_routers, diff --git a/vllm/entrypoints/launchers/dp_supervisor.py b/vllm/entrypoints/launchers/dp_supervisor.py index 192bbb1698..f2a84c4d9d 100644 --- a/vllm/entrypoints/launchers/dp_supervisor.py +++ b/vllm/entrypoints/launchers/dp_supervisor.py @@ -200,7 +200,7 @@ async def _probe_endpoint( # vLLM returns 503 on EngineDeadError, so we should return # immediately if vLLM responds with a non-200 status code. return response.status == HTTPStatus.OK - except (aiohttp.ClientError, asyncio.TimeoutError) as e: + except (TimeoutError, aiohttp.ClientError) as e: # Allow retry of connection errors. logger.debug( "Probe attempt %d/%d failed on port %d: %r", diff --git a/vllm/entrypoints/launchers/launcher.py b/vllm/entrypoints/launchers/launcher.py index 9fabfa3277..44161439b6 100644 --- a/vllm/entrypoints/launchers/launcher.py +++ b/vllm/entrypoints/launchers/launcher.py @@ -5,7 +5,7 @@ import asyncio import contextlib import signal import socket -from collections.abc import Generator +from collections.abc import Generator, MutableSequence from functools import partial from typing import Any @@ -34,23 +34,149 @@ from .utils.constants import ( logger = init_logger(__name__) +# A worker with more connections than another leaves a pending connection to +# the less loaded one, rechecking every 1 ms, for up to 5 ms (it may be busy). +_ACCEPT_DEFER_CHECKS = 5 +# The load of a worker that is not accepting (also the initial value of the +# shared array), so it is never the least loaded. +_NOT_ACCEPTING = 2**31 - 1 + class NoSignalServer(uvicorn.Server): """Uvicorn server that never installs its own SIGINT/SIGTERM handlers. Callers register their own handlers on the event loop for graceful shutdown; uvicorn's would race with and override them (see #49668). + + With ``peer_loads`` (several API workers on one shared socket: a shared + array with one slot per worker, and this worker's index), each worker + publishes its number of open connections, and a pending connection is + accepted by the least loaded worker. The event loop's own server would + accept every queued connection in whichever worker wakes first. """ + def __init__( + self, + config: uvicorn.Config, + peer_loads: tuple[MutableSequence[int], int] | None = None, + ): + super().__init__(config) + self.peer_loads = peer_loads + self._accept_tasks: set[asyncio.Task] = set() + self._connecting = 0 + @contextlib.contextmanager def capture_signals(self) -> Generator[None, None, None]: yield + def _publish_load(self) -> None: + assert self.peer_loads is not None + loads, index = self.peer_loads + loads[index] = len(self.server_state.connections) + self._connecting + + def _least_loaded(self) -> bool: + assert self.peer_loads is not None + loads, index = self.peer_loads + return loads[index] <= min(loads) + + async def startup(self, sockets: list[socket.socket] | None = None) -> None: + if not (self.peer_loads and sockets): + return await super().startup(sockets) + # With no sockets uvicorn starts no asyncio server; _accept serves + # them with uvicorn's protocol instead. + await super().startup(sockets=[]) + if not self.started: # lifespan startup failed (uvicorn < 0.50) + return + self._publish_load() + for sock in sockets: + sock.listen(self.config.backlog) + sock.setblocking(False) + self._track(self._accept(sock)) + + def _track(self, coro) -> None: + task = asyncio.create_task(coro) + self._accept_tasks.add(task) + task.add_done_callback(self._accept_tasks.discard) + + async def _accept(self, sock: socket.socket) -> None: + loop = asyncio.get_running_loop() + while True: + await _readable(loop, sock) + # Connections closed since the last accept; peers refresh likewise. + self._publish_load() + for _ in range(_ACCEPT_DEFER_CHECKS): + if self._least_loaded(): + break + await asyncio.sleep(0.001) + try: + conn, _ = sock.accept() + except (BlockingIOError, InterruptedError, ConnectionAbortedError): + continue # another worker took it + except OSError: + # e.g. EMFILE; asyncio's own server also retries after 1 s. + logger.exception("Error accepting a connection") + await asyncio.sleep(1) + continue + self._connecting += 1 + self._publish_load() + # Its own task, so a TLS handshake does not hold up the next accept. + self._track(self._connect(loop, conn)) + + async def _connect(self, loop: asyncio.AbstractEventLoop, conn: socket.socket): + config = self.config + try: + await loop.connect_accepted_socket( + lambda: config.http_protocol_class( # type: ignore[call-arg] + config=config, + server_state=self.server_state, + app_state=self.lifespan.state, + _loop=loop, + ), + conn, + ssl=config.ssl, + ) + except asyncio.CancelledError: + conn.close() + raise + except Exception: + # e.g. a failed TLS handshake + logger.debug("Accepted connection failed to start", exc_info=True) + conn.close() + finally: + self._connecting -= 1 + self._publish_load() + + async def shutdown(self, sockets: list[socket.socket] | None = None) -> None: + # Before super() closes the listening sockets. + tasks = list(self._accept_tasks) + for task in tasks: + task.cancel() + await asyncio.gather(*tasks, return_exceptions=True) + if self.peer_loads is not None: + loads, index = self.peer_loads + loads[index] = _NOT_ACCEPTING + await super().shutdown(sockets) + + +async def _readable(loop: asyncio.AbstractEventLoop, sock: socket.socket) -> None: + ready: asyncio.Future[None] = loop.create_future() + + def wake() -> None: + if not ready.done(): + ready.set_result(None) + + loop.add_reader(sock, wake) + try: + await ready + finally: + loop.remove_reader(sock) + async def serve_http( app: FastAPI, sock: socket.socket | None, enable_ssl_refresh: bool = False, + peer_loads: tuple[MutableSequence[int], int] | None = None, **uvicorn_kwargs: Any, ): """Start a FastAPI app using Uvicorn, with support for custom Uvicorn config @@ -96,7 +222,7 @@ async def serve_http( config.h11_max_incomplete_event_size = h11_max_incomplete_event_size config.h11_max_header_count = h11_max_header_count config.load() - server = NoSignalServer(config) + server = NoSignalServer(config, peer_loads=peer_loads) app.state.server = server loop = asyncio.get_running_loop() diff --git a/vllm/entrypoints/launchers/render/app_state.py b/vllm/entrypoints/launchers/render/app_state.py index bbd9a61275..33aebb9bc3 100644 --- a/vllm/entrypoints/launchers/render/app_state.py +++ b/vllm/entrypoints/launchers/render/app_state.py @@ -50,6 +50,16 @@ async def init_render_app_state( default_chat_template_kwargs = resolve_default_chat_template_kwargs(args) state.tool_server = await init_tool_server(args) + # The config-resolved reasoning parser carries both the CLI flag (merged + # in the entrypoint via `create_structured_outputs_config`) and any + # model-specific default applied by `verify_and_update_config` + # (e.g. "openai_gptoss" for gpt_oss), matching the main API server. + # Keep `args.reasoning_parser` as the first source so callers that build + # a VllmConfig without that merge still honor an explicit flag. + reasoning_parser = ( + args.reasoning_parser or vllm_config.structured_outputs_config.reasoning_parser + ) + state.online_renderer = OnlineRenderer( model_config=vllm_config.model_config, renderer=renderer, @@ -62,7 +72,7 @@ async def init_render_app_state( exclude_tools_when_tool_choice_none=args.exclude_tools_when_tool_choice_none, tool_parser=args.tool_call_parser, tool_strict_level=args.tool_strict_level, - reasoning_parser=args.reasoning_parser, + reasoning_parser=reasoning_parser, default_chat_template_kwargs=default_chat_template_kwargs, log_error_stack=args.log_error_stack, ) @@ -79,7 +89,7 @@ async def init_render_app_state( exclude_tools_when_tool_choice_none=args.exclude_tools_when_tool_choice_none, tool_parser=args.tool_call_parser, tool_strict_level=args.tool_strict_level, - reasoning_parser=args.reasoning_parser, + reasoning_parser=reasoning_parser, default_chat_template_kwargs=default_chat_template_kwargs, log_error_stack=args.log_error_stack, ) diff --git a/vllm/entrypoints/launchers/render/entry.py b/vllm/entrypoints/launchers/render/entry.py index b42e1f5911..92c29ea320 100644 --- a/vllm/entrypoints/launchers/render/entry.py +++ b/vllm/entrypoints/launchers/render/entry.py @@ -87,9 +87,14 @@ async def run_launch_fastapi(args: argparse.Namespace) -> None: # cache space warning from CpuPlatform.check_and_update_config. envs.VLLM_CPU_KVCACHE_SPACE = 0 + # Merge frontend parser flags (e.g. --reasoning-parser) into the + # structured outputs config, mirroring `create_engine_config`, which this + # GPU-less entrypoint bypasses. Model-specific defaults are then applied + # by `verify_and_update_config` during VllmConfig construction. vllm_config = VllmConfig( model_config=model_config, logging_config=engine_args.create_logging_config(), + structured_outputs_config=engine_args.create_structured_outputs_config(), ) shutdown_task = await build_and_serve_renderer( vllm_config, listen_address, sock, args diff --git a/vllm/entrypoints/launchers/run_batch.py b/vllm/entrypoints/launchers/run_batch.py index f54e87dafe..266592a731 100644 --- a/vllm/entrypoints/launchers/run_batch.py +++ b/vllm/entrypoints/launchers/run_batch.py @@ -385,7 +385,7 @@ async def upload_data(output_url: str, data_or_file: str, from_file: bool) -> No if from_file: with open(data_or_file, "rb") as file: async with session.put(output_url, data=file) as response: - if response.status != 200: + if not 200 <= response.status < 300: error_text = await response.text() raise Exception( f"Failed to upload file.\n" @@ -394,7 +394,7 @@ async def upload_data(output_url: str, data_or_file: str, from_file: bool) -> No ) else: async with session.put(output_url, data=data_or_file) as response: - if response.status != 200: + if not 200 <= response.status < 300: error_text = await response.text() raise Exception( f"Failed to upload data.\n" diff --git a/vllm/entrypoints/llm.py b/vllm/entrypoints/llm.py index 3de7bab133..a879b61484 100644 --- a/vllm/entrypoints/llm.py +++ b/vllm/entrypoints/llm.py @@ -3,13 +3,12 @@ from collections.abc import Callable, Sequence from pathlib import Path -from typing import TYPE_CHECKING, Any +from typing import TYPE_CHECKING, Any, overload import cloudpickle import torch.nn as nn from pydantic import ValidationError from tqdm.auto import tqdm -from typing_extensions import overload from vllm.config import ( AttentionConfig, @@ -28,18 +27,16 @@ from vllm.config.model import ( TokenizerMode, ) from vllm.config.quantization import QuantizationConfigArgs -from vllm.distributed.weight_transfer.base import ( - WeightTransferInitRequest, - WeightTransferUpdateRequest, -) from vllm.engine.arg_utils import EngineArgs from vllm.entrypoints.chat_utils import ( ChatCompletionMessageParam, ChatTemplateContentFormatOption, load_chat_template, ) +from vllm.entrypoints.common.offline import _O, _R, OfflineInferenceMixin from vllm.entrypoints.generate.beam_search.offline import BeamSearchOfflineMixin from vllm.entrypoints.pooling.offline import PoolingOfflineMixin +from vllm.entrypoints.rl.offline import RLOfflineMixin from vllm.entrypoints.serve.utils.api_utils import log_non_default_args from vllm.inputs import PromptType from vllm.logger import configure_logging_if_needed, init_logger @@ -56,7 +53,6 @@ from vllm.v1.engine.llm_engine import LLMEngine from vllm.v1.sample.logits_processor import LogitsProcessor from ..renderers import ChatParams -from .offline_utils import _O, _R, OfflineInferenceMixin if TYPE_CHECKING: from vllm.v1.metrics.reader import Metric @@ -64,7 +60,9 @@ if TYPE_CHECKING: logger = init_logger(__name__) -class LLM(BeamSearchOfflineMixin, PoolingOfflineMixin, OfflineInferenceMixin): +class LLM( + BeamSearchOfflineMixin, PoolingOfflineMixin, RLOfflineMixin, OfflineInferenceMixin +): """An LLM for generating texts from given prompts and sampling parameters. This class includes a tokenizer, a language model (possibly distributed @@ -785,16 +783,30 @@ class LLM(BeamSearchOfflineMixin, PoolingOfflineMixin, OfflineInferenceMixin): mm_processor_kwargs=mm_processor_kwargs, ) - def start_profile(self, profile_prefix: str | None = None) -> None: - """Start profiling with optional custom trace prefix. + def start_profile( + self, + profile_prefix: str | None = None, + *, + delay_iterations: int | None = None, + max_iterations: int | None = None, + ) -> None: + """Start profiling with optional per-session overrides. Args: profile_prefix: Optional prefix for the trace file names. If provided, trace files will be named as "_dp_pp_tp". If not provided, default naming will be used. + delay_iterations: Optional number of worker iterations to skip before + profiling starts. + max_iterations: Optional maximum number of worker iterations to profile. + Zero means no limit. """ - self.llm_engine.start_profile(profile_prefix) + self.llm_engine.start_profile( + profile_prefix, + delay_iterations=delay_iterations, + max_iterations=max_iterations, + ) def stop_profile(self) -> None: self.llm_engine.stop_profile() @@ -870,60 +882,6 @@ class LLM(BeamSearchOfflineMixin, PoolingOfflineMixin, OfflineInferenceMixin): """ return self.llm_engine.get_metrics() - def init_weight_transfer_engine( - self, request: WeightTransferInitRequest | dict - ) -> None: - """Initialize weight transfer for RL training. - - Args: - request: Weight transfer initialization request with backend-specific info - - """ - init_info_dict = ( - request["init_info"] if isinstance(request, dict) else request.init_info - ) - - self.llm_engine.collective_rpc( - "init_weight_transfer_engine", kwargs={"init_info": init_info_dict} - ) - - def start_weight_update(self) -> None: - """Start a new weight update.""" - self.llm_engine.collective_rpc("start_weight_update") - - def start_draft_weight_update(self) -> None: - """Start a new weight update targeting the speculative draft model.""" - self.llm_engine.collective_rpc("start_draft_weight_update") - - def update_weights(self, request: WeightTransferUpdateRequest | dict) -> None: - """Update the weights of the model. - - Args: - request: Weight update request with backend-specific update info - - """ - update_info_dict = ( - request["update_info"] if isinstance(request, dict) else request.update_info - ) - - self.llm_engine.collective_rpc( - "update_weights", kwargs={"update_info": update_info_dict} - ) - - def finish_weight_update(self, weight_version: str | None = None) -> None: - """Finish the weight update and set its version if provided.""" - self.llm_engine.collective_rpc("finish_weight_update") - if weight_version is not None: - self.llm_engine.set_weight_version(weight_version) - - def update_weight_version(self, new_version: str) -> None: - """Set the weight version without updating weights.""" - self.llm_engine.set_weight_version(new_version) - - def get_weight_version(self) -> str: - """Return the latest committed weight version.""" - return self.llm_engine.get_weight_version() - def __repr__(self) -> str: """Return a transformers-style hierarchical view of the model.""" # Cache the result to avoid repeated collective_rpc calls diff --git a/vllm/entrypoints/openai/chat_completion/batch_serving.py b/vllm/entrypoints/openai/chat_completion/batch_serving.py index 8c7ae37647..d3ea1a1ac3 100644 --- a/vllm/entrypoints/openai/chat_completion/batch_serving.py +++ b/vllm/entrypoints/openai/chat_completion/batch_serving.py @@ -169,6 +169,12 @@ class OpenAIServingChatBatch(OpenAIServingChat): else await self._get_trace_headers(raw_request.headers) ) session_id = self._get_session_id(single_request, raw_request) + chat_template_kwargs = self._effective_chat_template_kwargs(single_request) + parser = ( + self._make_parser(single_request, tokenizer, chat_template_kwargs) + if self.parser_cls is not None + else None + ) generators.append( self.engine_client.generate( engine_prompt, @@ -179,7 +185,12 @@ class OpenAIServingChatBatch(OpenAIServingChat): priority=request.priority, data_parallel_rank=data_parallel_rank, session_id=session_id, - reasoning_ended=None, + **self._engine_reasoning_kwargs( + single_request, + parser, + chat_template_kwargs, + self._extract_prompt_components(engine_prompt).token_ids, + ), ) ) @@ -247,9 +258,7 @@ class OpenAIServingChatBatch(OpenAIServingChat): for output in final_res.outputs: self._raise_if_error(output.finish_reason, request_id) - if request.logprobs and ( - request.top_logprobs is not None or request.logprob_token_ids - ): + if request.logprobs: assert output.logprobs is not None, "Did not output logprobs" logprobs = self._create_chat_logprobs( token_ids=output.token_ids, diff --git a/vllm/entrypoints/openai/chat_completion/protocol.py b/vllm/entrypoints/openai/chat_completion/protocol.py index 228472ab6b..dee8f283cf 100644 --- a/vllm/entrypoints/openai/chat_completion/protocol.py +++ b/vllm/entrypoints/openai/chat_completion/protocol.py @@ -4,7 +4,7 @@ # Adapted from # https://github.com/lm-sys/FastChat/blob/168ccc29d3f7edc50823016105c024fe2282732a/fastchat/protocol/openai_api_protocol.py import time -from typing import Annotated, Any, ClassVar, Literal +from typing import TYPE_CHECKING, Annotated, Any, ClassVar, Literal from openai.types.chat.chat_completion_audio import ( ChatCompletionAudio as OpenAIChatCompletionAudio, @@ -33,6 +33,7 @@ from vllm.entrypoints.generate.base.protocol import ( StopParam, StreamOptions, ToolCall, + TopLogprobsParam, structured_outputs_from_response_format, validate_cache_salt, validate_structural_tag_response_format, @@ -53,6 +54,9 @@ from vllm.sampling_params import ( ) from vllm.utils import random_uuid +if TYPE_CHECKING: + from vllm.parser.abstract_parser import Parser + logger = init_logger(__name__) @@ -219,7 +223,7 @@ class ChatCompletionRequest(OpenAIBaseModel): frequency_penalty: float | None = None logit_bias: dict[str, float] | None = None logprobs: bool | None = False - top_logprobs: int | None = 0 + top_logprobs: TopLogprobsParam = 0 max_tokens: int | None = Field( default=None, deprecated="max_tokens is deprecated in favor of " @@ -268,7 +272,7 @@ class ChatCompletionRequest(OpenAIBaseModel): top_k: int | None = None min_p: float | None = None repetition_penalty: float | None = None - watermarking: bool = True + watermarking: bool | None = None length_penalty: float = 1.0 stop_token_ids: list[int] | None = [] include_stop_str_in_output: bool = False @@ -454,6 +458,18 @@ class ChatCompletionRequest(OpenAIBaseModel): ), ) + return_mm_kwargs: bool = Field( + default=True, + description=( + "If false, the render response's `features` set `kwargs_data` " + "and `mm_metadata` to null, for callers that need only the token " + "layout and item hashes, such as cache-aware routers. Do not send " + "such a response to `/inference/v1/generate`, which reads a null " + "`kwargs_data` as every item being cached. Only supported on the " + "render endpoints; ignored on regular generation endpoints." + ), + ) + cache_salt: str | None = Field( default=None, min_length=1, @@ -566,6 +582,25 @@ class ChatCompletionRequest(OpenAIBaseModel): _grammar_from_parser: bool = PrivateAttr(default=False) """CAUTION: Should only be set by the parser-engine adapter's adjust_request.""" + def resolve_reasoning_ended( + self, parser: "Parser | None", prompt_token_ids: list[int] + ) -> bool | None: + """Resolve the engine's `reasoning_ended` for structured outputs. + + Call after `adjust_request`, which may set `_grammar_from_parser`. + `True` constrains from the first generated token; `None` lets the + engine check the prompt with its own reasoning parser. + """ + if not self.include_reasoning: + return True + if self._grammar_from_parser: + # The Mistral grammar already includes an optional `think?` + # rule that handles both reasoning and non-reasoning outputs. + return True + if parser is not None and parser.reasoning_parser is not None: + return parser.is_reasoning_end(prompt_token_ids) + return None + def build_chat_params( self, default_template: str | None, @@ -647,6 +682,7 @@ class ChatCompletionRequest(OpenAIBaseModel): max_tokens=max_tokens, ignore_eos=self.ignore_eos, temperature=temperature, + watermarking=self.watermarking, length_penalty=self.length_penalty, include_stop_str_in_output=self.include_stop_str_in_output, skip_special_tokens=self.skip_special_tokens, @@ -1069,10 +1105,10 @@ class BatchChatCompletionRequest(OpenAIBaseModel): model: str | None = None # Shared sampling / generation fields — mirror ChatCompletionRequest. - frequency_penalty: float | None = 0.0 + frequency_penalty: float | None = None logit_bias: dict[str, float] | None = None logprobs: bool | None = False - top_logprobs: int | None = 0 + top_logprobs: TopLogprobsParam = 0 logprob_token_ids: list[int] | None = Field( default=None, description=( @@ -1084,7 +1120,7 @@ class BatchChatCompletionRequest(OpenAIBaseModel): max_tokens: int | None = None max_completion_tokens: int | None = None n: int | None = 1 - presence_penalty: float | None = 0.0 + presence_penalty: float | None = None response_format: Any | None = None seed: int | None = Field(None, ge=_INT64_MIN, le=_INT64_MAX) stop: StopParam = Field(default_factory=list) @@ -1100,6 +1136,7 @@ class BatchChatCompletionRequest(OpenAIBaseModel): top_k: int | None = None min_p: float | None = None repetition_penalty: float | None = None + watermarking: bool | None = None length_penalty: float | None = 1.0 early_stopping: bool = False structured_outputs: StructuredOutputsParams | None = None diff --git a/vllm/entrypoints/openai/chat_completion/serving.py b/vllm/entrypoints/openai/chat_completion/serving.py index 400089c963..69b46ba67f 100644 --- a/vllm/entrypoints/openai/chat_completion/serving.py +++ b/vllm/entrypoints/openai/chat_completion/serving.py @@ -233,6 +233,28 @@ class OpenAIServingChat(GenerateBaseServing): """ return chat_template_kwargs + def _engine_reasoning_kwargs( + self, + request: ChatCompletionRequest, + parser: Parser | None, + chat_template_kwargs: dict[str, Any], + prompt_token_ids: list[int] | None, + ) -> dict[str, Any]: + """`reasoning_ended` and `reasoning_parser_kwargs` for `generate()`.""" + reasoning_parser_kwargs = None + if parser is not None and parser.reasoning_parser is not None: + reasoning_parser_kwargs = { + "chat_template_kwargs": self._engine_chat_template_kwargs( + chat_template_kwargs + ), + } + return { + "reasoning_ended": request.resolve_reasoning_ended( + parser, prompt_token_ids or [] + ), + "reasoning_parser_kwargs": reasoning_parser_kwargs, + } + async def render_chat_request( self, request: ChatCompletionRequest, @@ -364,18 +386,6 @@ class OpenAIServingChat(GenerateBaseServing): session_id=session_id, ) else: - if not request.include_reasoning: - reasoning_ended = True - elif request._grammar_from_parser: - # The Mistral grammar already includes an optional - # `think?` rule that handles both reasoning and - # non-reasoning outputs. - reasoning_ended = True - elif parser is not None and parser.reasoning_parser is not None: - reasoning_ended = parser.is_reasoning_end(prompt_token_ids or []) - else: - reasoning_ended = None - generator = self.engine_client.generate( engine_input, sampling_params, @@ -385,14 +395,9 @@ class OpenAIServingChat(GenerateBaseServing): priority=self._get_priority(request, raw_request), data_parallel_rank=data_parallel_rank, session_id=session_id, - reasoning_ended=reasoning_ended, - reasoning_parser_kwargs={ - "chat_template_kwargs": self._engine_chat_template_kwargs( - chat_template_kwargs - ), - } - if parser is not None and parser.reasoning_parser is not None - else None, + **self._engine_reasoning_kwargs( + request, parser, chat_template_kwargs, prompt_token_ids + ), ) generators.append(generator) @@ -633,9 +638,7 @@ class OpenAIServingChat(GenerateBaseServing): self._raise_if_error(output.finish_reason, request_id) - if request.logprobs and ( - request.top_logprobs is not None or request.logprob_token_ids - ): + if request.logprobs: assert output.logprobs is not None, "Did not output logprobs" logprobs = self._create_chat_logprobs( token_ids=output.token_ids, @@ -978,9 +981,7 @@ class OpenAIServingChat(GenerateBaseServing): token_ids = output.token_ids out_logprobs = output.logprobs - if request.logprobs and ( - request.top_logprobs is not None or request.logprob_token_ids - ): + if request.logprobs: assert out_logprobs is not None, "Did not output logprobs" logprobs = self._create_chat_logprobs( token_ids=token_ids, diff --git a/vllm/entrypoints/openai/completion/protocol.py b/vllm/entrypoints/openai/completion/protocol.py index a0974cb91c..f26dea2e1f 100644 --- a/vllm/entrypoints/openai/completion/protocol.py +++ b/vllm/entrypoints/openai/completion/protocol.py @@ -75,7 +75,7 @@ class CompletionRequest(OpenAIBaseModel): top_k: int | None = None min_p: float | None = None repetition_penalty: float | None = None - watermarking: bool = True + watermarking: bool | None = None length_penalty: float = 1.0 stop_token_ids: list[int] | None = [] include_stop_str_in_output: bool = False @@ -197,6 +197,18 @@ class CompletionRequest(OpenAIBaseModel): ), ) + return_mm_kwargs: bool = Field( + default=True, + description=( + "If false, the render response's `features` set `kwargs_data` " + "and `mm_metadata` to null, for callers that need only the token " + "layout and item hashes, such as cache-aware routers. Do not send " + "such a response to `/inference/v1/generate`, which reads a null " + "`kwargs_data` as every item being cached. Only supported on the " + "render endpoints; ignored on regular generation endpoints." + ), + ) + cache_salt: str | None = Field( default=None, min_length=1, @@ -303,6 +315,7 @@ class CompletionRequest(OpenAIBaseModel): max_tokens=max_tokens, ignore_eos=self.ignore_eos, temperature=temperature, + watermarking=self.watermarking, length_penalty=self.length_penalty, include_stop_str_in_output=self.include_stop_str_in_output, skip_special_tokens=self.skip_special_tokens, diff --git a/vllm/entrypoints/openai/completion/serving.py b/vllm/entrypoints/openai/completion/serving.py index 81dd00f0b3..7f7bdac075 100644 --- a/vllm/entrypoints/openai/completion/serving.py +++ b/vllm/entrypoints/openai/completion/serving.py @@ -289,7 +289,7 @@ class OpenAIServingCompletion(GenerateBaseServing): request_metadata: RequestResponseMetadata, ) -> AsyncGenerator[str, None]: num_choices = 1 if request.n is None else request.n - previous_text_lens = [0] * num_choices * num_prompts + next_text_offsets = [0] * num_choices * num_prompts previous_num_tokens = [0] * num_choices * num_prompts has_echoed = [False] * num_choices * num_prompts num_prompt_tokens = [0] * num_prompts @@ -386,13 +386,18 @@ class OpenAIServingCompletion(GenerateBaseServing): num_output_top_logprobs=request.logprobs, tokenizer=tokenizer, logprob_token_ids=request.logprob_token_ids, - initial_text_offset=previous_text_lens[i], + initial_text_offset=next_text_offsets[i], return_as_token_id=request.return_tokens_as_token_ids, ) + # Advance by the tokens just reported, not by delta_text: + # text held back for stop strings lags the tokens. + if logprobs.tokens: + next_text_offsets[i] = logprobs.text_offset[-1] + len( + logprobs.tokens[-1] + ) else: logprobs = None - previous_text_lens[i] += len(delta_text) previous_num_tokens[i] += len(output.token_ids) stop_reason = output.stop_reason diff --git a/vllm/entrypoints/openai/responses/protocol.py b/vllm/entrypoints/openai/responses/protocol.py index 9c44a04b3a..406ab2d081 100644 --- a/vllm/entrypoints/openai/responses/protocol.py +++ b/vllm/entrypoints/openai/responses/protocol.py @@ -61,6 +61,7 @@ from vllm.entrypoints.chat_utils import ( from vllm.entrypoints.generate.base.protocol import ( PerRequestMetrics, StopParam, + TopLogprobsParam, validate_cache_salt, ) from vllm.entrypoints.serve.engine.protocol import OpenAIBaseModel @@ -158,6 +159,51 @@ def _default_input_image_details(value: Any) -> Any: return {**value, "content": new_content} if changed else value +def _tool_field(tool: Any, field: str) -> Any: + return tool.get(field) if isinstance(tool, dict) else getattr(tool, field, None) + + +def _resolve_named_tool_choice(tool_name: str, tools: list[Any]) -> str: + """Resolve a named tool choice to the flat ``__`` form.""" + from vllm.tool_parsers.utils import flat_namespace_tool_name + + exact_names: set[str] = set() + local_names: dict[str, list[str]] = {} + for tool in tools: + if _tool_field(tool, "type") == "namespace": + namespace = _tool_field(tool, "name") + namespaced_tools = _tool_field(tool, "tools") + if not isinstance(namespaced_tools, list) or not isinstance(namespace, str): + return tool_name + for namespaced_tool in namespaced_tools: + local_name = _tool_field(namespaced_tool, "name") + if not isinstance(local_name, str): + continue + flat_name = flat_namespace_tool_name(namespace, local_name) + exact_names.add(flat_name) + local_names.setdefault(local_name, []).append(flat_name) + else: + name = _tool_field(tool, "name") + if isinstance(name, str): + exact_names.add(name) + if tool_name in exact_names: + return tool_name + candidates = local_names.get(tool_name, []) + if len(candidates) == 1: + return candidates[0] + if len(candidates) > 1: + raise VLLMValidationError( + f"Tool choice '{tool_name}' is ambiguous: it is defined in more " + f"than one namespace ({', '.join(candidates)}). Select it by its " + "full '__' name.", + parameter="tool_choice", + ) + raise VLLMValidationError( + "Tool choice 'function' not found in 'tools' parameter.", + parameter="tool_choice", + ) + + class ResponsesRequest(OpenAIBaseModel): # Ordered by official OpenAI API documentation # https://platform.openai.com/docs/api-reference/responses/create @@ -210,9 +256,10 @@ class ResponsesRequest(OpenAIBaseModel): text: ResponseTextConfig | None = None tool_choice: ToolChoice = "auto" tools: list[Tool] = Field(default_factory=list) - top_logprobs: int | None = 0 + top_logprobs: TopLogprobsParam = 0 top_p: float | None = None top_k: int | None = None + min_p: float | None = None truncation: Literal["auto", "disabled"] | None = "disabled" user: str | None = None skip_special_tokens: bool = True @@ -245,7 +292,7 @@ class ResponsesRequest(OpenAIBaseModel): ) # --8<-- [start:responses-extra-params] - watermarking: bool = True + watermarking: bool | None = None request_id: str = Field( default_factory=lambda: f"resp_{random_uuid()}", description=( @@ -283,6 +330,17 @@ class ResponsesRequest(OpenAIBaseModel): "if the served model does not use priority scheduling." ), ) + return_mm_kwargs: bool = Field( + default=True, + description=( + "If false, the render response's `features` set `kwargs_data` " + "and `mm_metadata` to null, for callers that need only the token " + "layout and item hashes, such as cache-aware routers. Do not send " + "such a response to `/inference/v1/generate`, which reads a null " + "`kwargs_data` as every item being cached. Only supported on the " + "render endpoints; ignored on regular generation endpoints." + ), + ) cache_salt: str | None = Field( default=None, min_length=1, @@ -317,6 +375,7 @@ class ResponsesRequest(OpenAIBaseModel): repetition_penalty: float | None = None seed: int | None = Field(None, ge=_INT64_MIN, le=_INT64_MAX) stop: StopParam = [] + stop_token_ids: list[int] | None = [] ignore_eos: bool = False vllm_xargs: dict[str, str | int | float | list[str | int | float]] | None = Field( default=None, @@ -397,6 +456,7 @@ class ResponsesRequest(OpenAIBaseModel): "temperature": 1.0, "top_p": 1.0, "top_k": 0, + "min_p": 0.0, } def extract_structured_outputs(self) -> StructuredOutputsParams | None: @@ -447,6 +507,10 @@ class ResponsesRequest(OpenAIBaseModel): top_k = default_sampling_params.get( "top_k", self._DEFAULT_SAMPLING_PARAMS["top_k"] ) + if (min_p := self.min_p) is None: + min_p = default_sampling_params.get( + "min_p", self._DEFAULT_SAMPLING_PARAMS["min_p"] + ) if (repetition_penalty := self.repetition_penalty) is None: repetition_penalty = default_sampling_params.get("repetition_penalty", 1.0) @@ -472,9 +536,11 @@ class ResponsesRequest(OpenAIBaseModel): watermarking=self.watermarking, top_p=top_p, top_k=top_k, + min_p=min_p, max_tokens=max_tokens, logprobs=self.top_logprobs if self.is_include_output_logprobs() else None, stop=stop, + stop_token_ids=self.stop_token_ids, frequency_penalty=frequency_penalty, presence_penalty=presence_penalty, repetition_penalty=repetition_penalty, @@ -661,31 +727,14 @@ class ResponsesRequest(OpenAIBaseModel): ) elif is_named_tool_choice and tools is not None: tool_name = tool_choice.get("name") - tool_names = set() - for tool in tools: - if isinstance(tool, dict): - if tool.get("type") == "namespace": - namespace = tool.get("name") - namespaced_tools = tool.get("tools") - if not isinstance(namespaced_tools, list): - return data - for namespaced_tool in namespaced_tools: - namespaced_name = ( - namespaced_tool.get("name") - if isinstance(namespaced_tool, dict) - else getattr(namespaced_tool, "name", None) - ) - tool_names.add(namespaced_name) - tool_names.add(f"{namespace}__{namespaced_name}") - else: - tool_names.add(tool.get("name")) - else: - tool_names.add(getattr(tool, "name", None)) - if not tool_name or tool_name not in tool_names: + if not tool_name: raise VLLMValidationError( "Tool choice 'function' not found in 'tools' parameter.", parameter="tool_choice", ) + resolved = _resolve_named_tool_choice(tool_name, tools) + if resolved != tool_name: + data["tool_choice"] = {**tool_choice, "name": resolved} return data diff --git a/vllm/entrypoints/pooling/embed/io_processor.py b/vllm/entrypoints/pooling/embed/io_processor.py index 3c5b75cee6..1b23a77ec6 100644 --- a/vllm/entrypoints/pooling/embed/io_processor.py +++ b/vllm/entrypoints/pooling/embed/io_processor.py @@ -142,13 +142,16 @@ class EmbedIOProcessor(PoolingIOProcessor): ) prompt_token_ids = cast(list[int], token_ids) + cache_salt = cast(str | None, engine_input["prompts"].get("cache_salt")) for chunk_idx, chunk_tokens in enumerate( chunk_list(prompt_token_ids, max_model_len) ): chunked_engine_inputs.append( PoolingEngineInput( - prompts=tokens_input(prompt_token_ids=chunk_tokens), + prompts=tokens_input( + prompt_token_ids=chunk_tokens, cache_salt=cache_salt + ), params=engine_input["params"], lora_requests=engine_input["lora_requests"], priorities=engine_input["priorities"], diff --git a/vllm/entrypoints/pooling/embed/serving.py b/vllm/entrypoints/pooling/embed/serving.py index a9f6dd310e..4fdb73d363 100644 --- a/vllm/entrypoints/pooling/embed/serving.py +++ b/vllm/entrypoints/pooling/embed/serving.py @@ -1,9 +1,8 @@ # SPDX-License-Identifier: Apache-2.0 # SPDX-FileCopyrightText: Copyright contributors to the vLLM project -from typing import TypeAlias, cast +from typing import TypeAlias, assert_never, cast from fastapi.responses import JSONResponse, Response, StreamingResponse -from typing_extensions import assert_never from vllm.logger import init_logger from vllm.outputs import PoolingRequestOutput diff --git a/vllm/entrypoints/pooling/offline.py b/vllm/entrypoints/pooling/offline.py index 0a0304e7a4..81d78101c4 100644 --- a/vllm/entrypoints/pooling/offline.py +++ b/vllm/entrypoints/pooling/offline.py @@ -7,7 +7,7 @@ from typing import Any from tqdm.auto import tqdm from vllm.entrypoints.chat_utils import ChatTemplateConfig -from vllm.entrypoints.offline_utils import OfflineInferenceMixin +from vllm.entrypoints.common.offline import OfflineInferenceMixin from vllm.inputs import DataPrompt, PromptType from vllm.logger import init_logger from vllm.lora.request import LoRARequest @@ -362,6 +362,8 @@ class PoolingOfflineMixin(OfflineInferenceMixin): assert isinstance(pooling_params, PoolingParams) pooling_task = io_processor.pooling_task + # Clone to avoid modifying the caller's pooling parameters. + pooling_params = pooling_params.clone() if pooling_params.task is None: pooling_params.task = pooling_task elif pooling_params.task != pooling_task: diff --git a/vllm/entrypoints/pooling/pooling/serving.py b/vllm/entrypoints/pooling/pooling/serving.py index 56bc7c7865..4e4ea1fe1b 100644 --- a/vllm/entrypoints/pooling/pooling/serving.py +++ b/vllm/entrypoints/pooling/pooling/serving.py @@ -1,9 +1,8 @@ # SPDX-License-Identifier: Apache-2.0 # SPDX-FileCopyrightText: Copyright contributors to the vLLM project -from typing import cast +from typing import assert_never, cast from fastapi.responses import JSONResponse, Response, StreamingResponse -from typing_extensions import assert_never from vllm.exceptions import VLLMValidationError from vllm.logger import init_logger diff --git a/vllm/entrypoints/pooling/scoring/typing.py b/vllm/entrypoints/pooling/scoring/typing.py index a67843e577..9afaada6d6 100644 --- a/vllm/entrypoints/pooling/scoring/typing.py +++ b/vllm/entrypoints/pooling/scoring/typing.py @@ -1,9 +1,9 @@ # SPDX-License-Identifier: Apache-2.0 # SPDX-FileCopyrightText: Copyright contributors to the vLLM project from dataclasses import dataclass -from typing import TypeAlias +from typing import Required, TypeAlias -from typing_extensions import Required, TypedDict +from typing_extensions import TypedDict from vllm.entrypoints.chat_utils import ( ChatCompletionContentPartImageEmbedsParam, diff --git a/vllm/entrypoints/rl/__init__.py b/vllm/entrypoints/rl/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/vllm/entrypoints/rl/factories.py b/vllm/entrypoints/rl/factories.py new file mode 100644 index 0000000000..a5d237ec0b --- /dev/null +++ b/vllm/entrypoints/rl/factories.py @@ -0,0 +1,11 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + + +from fastapi import FastAPI + + +def register_rl_api_routers(app: FastAPI): + from .online.api_router import router as rl_router + + app.include_router(rl_router) diff --git a/vllm/entrypoints/rl/offline.py b/vllm/entrypoints/rl/offline.py new file mode 100644 index 0000000000..1db6a82533 --- /dev/null +++ b/vllm/entrypoints/rl/offline.py @@ -0,0 +1,64 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +from vllm.distributed.weight_transfer.base import ( + WeightTransferInitRequest, + WeightTransferUpdateRequest, +) +from vllm.entrypoints.common.offline import OfflineInferenceMixin + + +class RLOfflineMixin(OfflineInferenceMixin): + def init_weight_transfer_engine( + self, request: WeightTransferInitRequest | dict + ) -> None: + """Initialize weight transfer for RL training. + + Args: + request: Weight transfer initialization request with backend-specific info + + """ + init_info_dict = ( + request["init_info"] if isinstance(request, dict) else request.init_info + ) + + self.llm_engine.collective_rpc( + "init_weight_transfer_engine", kwargs={"init_info": init_info_dict} + ) + + def start_weight_update(self) -> None: + """Start a new weight update.""" + self.llm_engine.collective_rpc("start_weight_update") + + def start_draft_weight_update(self) -> None: + """Start a new weight update targeting the speculative draft model.""" + self.llm_engine.collective_rpc("start_draft_weight_update") + + def update_weights(self, request: WeightTransferUpdateRequest | dict) -> None: + """Update the weights of the model. + + Args: + request: Weight update request with backend-specific update info + + """ + update_info_dict = ( + request["update_info"] if isinstance(request, dict) else request.update_info + ) + + self.llm_engine.collective_rpc( + "update_weights", kwargs={"update_info": update_info_dict} + ) + + def finish_weight_update(self, weight_version: str | None = None) -> None: + """Finish the weight update and set its version if provided.""" + self.llm_engine.collective_rpc("finish_weight_update") + if weight_version is not None: + self.llm_engine.set_weight_version(weight_version) + + def update_weight_version(self, new_version: str) -> None: + """Set the weight version without updating weights.""" + self.llm_engine.set_weight_version(new_version) + + def get_weight_version(self) -> str: + """Return the latest committed weight version.""" + return self.llm_engine.get_weight_version() diff --git a/vllm/entrypoints/rl/online/__init__.py b/vllm/entrypoints/rl/online/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/vllm/entrypoints/serve/dev/rlhf/api_router.py b/vllm/entrypoints/rl/online/api_router.py similarity index 97% rename from vllm/entrypoints/serve/dev/rlhf/api_router.py rename to vllm/entrypoints/rl/online/api_router.py index 7e52e51217..ef9269749f 100644 --- a/vllm/entrypoints/serve/dev/rlhf/api_router.py +++ b/vllm/entrypoints/rl/online/api_router.py @@ -5,7 +5,7 @@ import json from http import HTTPStatus from typing import Annotated, Any, Literal -from fastapi import APIRouter, Body, FastAPI, HTTPException, Query, Request +from fastapi import APIRouter, Body, HTTPException, Query, Request from fastapi.responses import JSONResponse from vllm.distributed.weight_transfer.base import ( @@ -14,7 +14,7 @@ from vllm.distributed.weight_transfer.base import ( WeightTransferUpdateRequest, ) from vllm.engine.protocol import EngineClient -from vllm.entrypoints.serve.dev.rlhf.metrics import weight_operation_metrics +from vllm.entrypoints.rl.online.metrics import weight_operation_metrics from vllm.logger import init_logger from vllm.v1.engine import PauseMode @@ -280,7 +280,3 @@ async def get_world_size( else: world_size = parallel_config.world_size return JSONResponse(content={"world_size": world_size}) - - -def attach_router(app: FastAPI): - app.include_router(router) diff --git a/vllm/entrypoints/serve/dev/rlhf/metrics.py b/vllm/entrypoints/rl/online/metrics.py similarity index 100% rename from vllm/entrypoints/serve/dev/rlhf/metrics.py rename to vllm/entrypoints/rl/online/metrics.py diff --git a/vllm/entrypoints/scale_out/render/serving.py b/vllm/entrypoints/scale_out/render/serving.py index 117ede0f9b..84771c5f73 100644 --- a/vllm/entrypoints/scale_out/render/serving.py +++ b/vllm/entrypoints/scale_out/render/serving.py @@ -17,6 +17,7 @@ from vllm.entrypoints.scale_out.token_in_token_out.mm_features import ( from vllm.entrypoints.scale_out.token_in_token_out.protocol import ( GenerateRequest, MultiModalFeatures, + ReasoningParserKwargs, ) from vllm.entrypoints.serve.engine.protocol import ErrorResponse from vllm.entrypoints.serve.engine.serving import BaseServing @@ -124,14 +125,32 @@ class ServingRender(BaseServing): ) params = request.to_sampling_params(max_tokens, self.default_sampling_params) + # Resolve here what OpenAIServingChat passes to the engine: + # `_grammar_from_parser` (set by `adjust_request`) is not serialized. + reasoning_parser_kwargs = self._reasoning_parser_kwargs(request) + parser = None + if reasoning_parser_kwargs is not None: + assert self.online_renderer.parser is not None + parser = self.online_renderer.parser( + self.online_renderer.renderer.get_tokenizer(), + request.tools, + chat_template_kwargs=reasoning_parser_kwargs.chat_template_kwargs, + model_config=self.model_config, + ) + reasoning_ended = request.resolve_reasoning_ended(parser, token_ids) + request_id = f"chatcmpl-{random_uuid()}" return GenerateRequest( request_id=request_id, token_ids=token_ids, - features=self._extract_mm_features(engine_input), + features=self._extract_mm_features( + engine_input, include_mm_kwargs=request.return_mm_kwargs + ), sampling_params=params, model=request.model, + reasoning_ended=reasoning_ended, + reasoning_parser_kwargs=reasoning_parser_kwargs, stream=bool(request.stream), stream_options=(request.stream_options if request.stream else None), cache_salt=request.cache_salt, @@ -152,6 +171,7 @@ class ServingRender(BaseServing): chat_req = AnthropicServingMessages.to_chat_completion_request( request, merge_inline_system=self._merge_inline_system ) + chat_req.return_mm_kwargs = request.return_mm_kwargs return await self.render_chat_request(chat_req) async def render_completion_request( @@ -200,7 +220,9 @@ class ServingRender(BaseServing): GenerateRequest( request_id=request_id, token_ids=token_ids, - features=self._extract_mm_features(engine_input), + features=self._extract_mm_features( + engine_input, include_mm_kwargs=request.return_mm_kwargs + ), sampling_params=params, model=request.model, stream=bool(request.stream), @@ -260,9 +282,12 @@ class ServingRender(BaseServing): return GenerateRequest( request_id=request.request_id, token_ids=list(token_ids), - features=self._extract_mm_features(engine_input), + features=self._extract_mm_features( + engine_input, include_mm_kwargs=request.return_mm_kwargs + ), sampling_params=params, model=request.model, + reasoning_parser_kwargs=self._reasoning_parser_kwargs(request), stream=bool(request.stream), cache_salt=request.cache_salt, priority=request.priority, @@ -271,6 +296,18 @@ class ServingRender(BaseServing): token_offsets=engine_input.get("prompt_token_offsets"), ) + def _reasoning_parser_kwargs( + self, request: ChatCompletionRequest | ResponsesRequest + ) -> ReasoningParserKwargs | None: + parser_cls = self.online_renderer.parser + if parser_cls is None or parser_cls.reasoning_parser_cls is None: + return None + return ReasoningParserKwargs( + chat_template_kwargs=self.online_renderer.effective_chat_template_kwargs( + request + ) + ) + def _placeholder_metadata_fields(self, modality: str) -> set[str]: """Fields the EC consumer still requires after embeddings transfer.""" if self._placeholder_metadata_parser_failed: @@ -298,8 +335,11 @@ class ServingRender(BaseServing): def _extract_mm_features( self, engine_input: EngineInput, + *, + include_mm_kwargs: bool = True, ) -> MultiModalFeatures | None: return extract_mm_features( engine_input, metadata_fields_for=self._placeholder_metadata_fields, + include_mm_kwargs=include_mm_kwargs, ) diff --git a/vllm/entrypoints/scale_out/token_in_token_out/mm_features.py b/vllm/entrypoints/scale_out/token_in_token_out/mm_features.py index 62d006783a..a0cb0d4b5e 100644 --- a/vllm/entrypoints/scale_out/token_in_token_out/mm_features.py +++ b/vllm/entrypoints/scale_out/token_in_token_out/mm_features.py @@ -139,13 +139,15 @@ def extract_mm_features( engine_input: EngineInput, *, metadata_fields_for: Callable[[str], Collection[str]] | None = None, + include_mm_kwargs: bool = True, ) -> MultiModalFeatures | None: """Extract multimodal features from a rendered engine prompt. Returns ``None`` for text-only prompts. ``mm_metadata`` keeps the intersection of processed kwargs that prefill needs after EC transfer: fields declared as embedding metadata, plus fields marked - ``keep_on_cpu`` (for example M-RoPE grid dims). + ``keep_on_cpu`` (for example M-RoPE grid dims). With + ``include_mm_kwargs=False`` neither is serialized. """ if engine_input.get("type") != "multimodal": return None @@ -157,11 +159,13 @@ def extract_mm_features( kwargs_data: dict[str, list[str | None]] | None = None mm_metadata: dict[str, list[str | None]] | None = None - if raw_mm_kwargs := mm_engine_input.get("mm_kwargs"): - kwargs_data, mm_metadata = _encode_mm_kwargs_with_metadata( - raw_mm_kwargs, - metadata_fields_for=metadata_fields_for, - ) + if include_mm_kwargs: + raw_mm_kwargs = mm_engine_input.get("mm_kwargs") + if raw_mm_kwargs: + kwargs_data, mm_metadata = _encode_mm_kwargs_with_metadata( + raw_mm_kwargs, + metadata_fields_for=metadata_fields_for, + ) return MultiModalFeatures( mm_hashes=mm_hashes, diff --git a/vllm/entrypoints/scale_out/token_in_token_out/protocol.py b/vllm/entrypoints/scale_out/token_in_token_out/protocol.py index 31bd7e2eae..84a6a2c623 100644 --- a/vllm/entrypoints/scale_out/token_in_token_out/protocol.py +++ b/vllm/entrypoints/scale_out/token_in_token_out/protocol.py @@ -4,6 +4,7 @@ from typing import Annotated, Any, Literal, TypeAlias from pydantic import ( BaseModel, + ConfigDict, Discriminator, Field, NonNegativeInt, @@ -147,6 +148,16 @@ class MultiModalFeatures(BaseModel): return self +class ReasoningParserKwargs(BaseModel): + """Kwargs for the engine-side reasoning parser that gates structured + outputs. Typed so clients cannot pass arbitrary constructor kwargs.""" + + model_config = ConfigDict(extra="forbid") + + chat_template_kwargs: dict[str, Any] = Field(default_factory=dict) + """The effective chat template kwargs the prompt was rendered with.""" + + class GenerateRequest(BaseModel): request_id: str = Field( default_factory=lambda: f"{random_uuid()}", @@ -212,6 +223,15 @@ class GenerateRequest(BaseModel): model: str | None = None + reasoning_ended: bool | None = None + """Whether reasoning has ended before the first generated token, as + resolved by /render. `True` applies structured outputs from the first + token; `None` lets the engine check the prompt with its reasoning parser.""" + + reasoning_parser_kwargs: ReasoningParserKwargs | None = None + """Set by /render when it has a reasoning parser, so the engine-side + parser agrees with the frontend on flags such as `enable_thinking`.""" + return_token_ids: bool | None = Field( default=None, description=( @@ -340,6 +360,39 @@ class GenerateRequest(BaseModel): ) +class GenerateLogProb(BaseModel): + """A single (token, logprob) candidate on the generate wire protocol. + + Unlike the OpenAI logprob shapes this carries the integer token id: the + generate server has no tokenizer, so decoding to a string belongs in + derender (or the coupled chat/completions path), not here. + """ + + token_id: int + logprob: float + rank: int | None = None + + +class GenerateLogProbsContent(GenerateLogProb): + """The sampled token at one position, plus its top-k candidates. + + ``top_logprobs`` is a list, not a dict: JSON turns dict keys into strings + and the order would be implicit. It is in the engine's order: the sampled + token first, then the remaining candidates in rank order. + """ + + top_logprobs: list[GenerateLogProb] = [] + + +class GenerateLogProbs(BaseModel): + """Output logprobs for one choice. + + ``content`` holds one entry per generated token. + """ + + content: list[GenerateLogProbsContent] | None = None + + class GenerateChoiceBase(BaseModel): """Fields shared by every `output_mode` of a non-streaming choice.""" @@ -368,8 +421,8 @@ class GenerateChoiceBase(BaseModel): class GenerateTokensChoice(GenerateChoiceBase): - logprobs: ChatCompletionLogProbs | None = None - """Logprobs whose tokens are `token_id:N` placeholders.""" + logprobs: GenerateLogProbs | None = None + """Logprobs with integer token ids (no tokenizer on the generate server).""" class GenerateTextChoice(GenerateChoiceBase): @@ -393,7 +446,7 @@ class GenerateStreamChoiceBase(BaseModel): class GenerateTokensStreamChoice(GenerateStreamChoiceBase): - logprobs: ChatCompletionLogProbs | None = None + logprobs: GenerateLogProbs | None = None class GenerateTextStreamChoice(GenerateStreamChoiceBase): diff --git a/vllm/entrypoints/scale_out/token_in_token_out/serving.py b/vllm/entrypoints/scale_out/token_in_token_out/serving.py index 6b0195b104..b6d7bf19ba 100644 --- a/vllm/entrypoints/scale_out/token_in_token_out/serving.py +++ b/vllm/entrypoints/scale_out/token_in_token_out/serving.py @@ -3,6 +3,7 @@ import asyncio +import math import time from collections.abc import AsyncGenerator from collections.abc import Sequence as GenericSequence @@ -57,6 +58,9 @@ from .mm_features import ( placeholder_ranges_from_engine_input, ) from .protocol import ( + GenerateLogProb, + GenerateLogProbs, + GenerateLogProbsContent, GenerateRequest, GenerateResponse, GenerateTextChoice, @@ -72,6 +76,15 @@ from .protocol import ( logger = init_logger(__name__) +def _clamp_logprob(logprob: float) -> float: + """The OpenAI shapes' floor for a logprob: ``-inf`` and NaN become ``-9999.0``. + + ``max(nan, -9999.0)`` is NaN in Python, so NaN needs its own check; NaN + would otherwise fail ``JSONResponse`` or reach the client as ``null``. + """ + return -9999.0 if math.isnan(logprob) else max(logprob, -9999.0) + + def _logprob_token( token_id: int, logprob: Logprob | None, tokenizer: TokenizerLike | None ) -> tuple[str, list[int] | None]: @@ -94,7 +107,7 @@ def _top_logprob( ) -> ChatCompletionLogProb: token, token_bytes = _logprob_token(token_id, logprob, tokenizer) return ChatCompletionLogProb( - token=token, logprob=max(logprob.logprob, -9999.0), bytes=token_bytes + token=token, logprob=_clamp_logprob(logprob.logprob), bytes=token_bytes ) @@ -349,6 +362,12 @@ class ServingTokens(GenerateBaseServing): priority=request.priority, data_parallel_rank=data_parallel_rank, session_id=session_id, + reasoning_ended=request.reasoning_ended, + reasoning_parser_kwargs=( + request.reasoning_parser_kwargs.model_dump() + if request.reasoning_parser_kwargs is not None + else None + ), ) assert result_generator is not None @@ -400,12 +419,20 @@ class ServingTokens(GenerateBaseServing): # This is top_logprobs in completions API if sampling_params.logprobs is not None: assert out_logprobs is not None, "Did not output logprobs" - logprobs = self._create_tokens_logprobs( - token_ids=token_ids, - top_logprobs=out_logprobs, - num_output_top_logprobs=sampling_params.logprobs, - tokenizer=tokenizer, - ) + logprobs: GenerateLogProbs | ChatCompletionLogProbs | None + if text_mode: + logprobs = self._create_text_logprobs( + token_ids=token_ids, + top_logprobs=out_logprobs, + num_output_top_logprobs=sampling_params.logprobs, + tokenizer=tokenizer, + ) + else: + logprobs = self._create_tokens_logprobs( + token_ids=token_ids, + top_logprobs=out_logprobs, + num_output_top_logprobs=sampling_params.logprobs, + ) else: logprobs = None @@ -565,12 +592,20 @@ class ServingTokens(GenerateBaseServing): if sampling_params.logprobs is not None: out_logprobs = output.logprobs assert out_logprobs is not None, "Did not output logprobs" - logprobs = self._create_tokens_logprobs( - token_ids=delta_token_ids, - top_logprobs=out_logprobs, - num_output_top_logprobs=sampling_params.logprobs, - tokenizer=tokenizer, - ) + logprobs: GenerateLogProbs | ChatCompletionLogProbs | None + if text_mode: + logprobs = self._create_text_logprobs( + token_ids=delta_token_ids, + top_logprobs=out_logprobs, + num_output_top_logprobs=sampling_params.logprobs, + tokenizer=tokenizer, + ) + else: + logprobs = self._create_tokens_logprobs( + token_ids=delta_token_ids, + top_logprobs=out_logprobs, + num_output_top_logprobs=sampling_params.logprobs, + ) else: logprobs = None @@ -685,17 +720,18 @@ class ServingTokens(GenerateBaseServing): return None return self.renderer.tokenizer - def _create_tokens_logprobs( + def _create_text_logprobs( self, token_ids: GenericSequence[int], top_logprobs: GenericSequence[dict[int, Logprob] | None], num_output_top_logprobs: int | None = None, tokenizer: TokenizerLike | None = None, ) -> ChatCompletionLogProbs: - """Create OpenAI-style logprobs. + """Create OpenAI-style logprobs for ``output_mode="text"``. - Tokens are ``token_id:N`` placeholders. With a ``tokenizer`` they are - decoded strings and carry ``bytes``. + With a ``tokenizer`` the tokens are decoded strings and carry ``bytes``; + without one (``--return-tokens-as-token-ids``) they are ``token_id:N`` + placeholders. """ logprobs_content: list[ChatCompletionLogProbsContent] = [] @@ -713,7 +749,7 @@ class ServingTokens(GenerateBaseServing): logprobs_content.append( ChatCompletionLogProbsContent( token=token, - logprob=max(step_token.logprob, -9999.0), + logprob=_clamp_logprob(step_token.logprob), bytes=token_bytes, top_logprobs=[ _top_logprob(top_id, logprob, tokenizer) @@ -730,3 +766,52 @@ class ServingTokens(GenerateBaseServing): ) return ChatCompletionLogProbs(content=logprobs_content) + + def _create_tokens_logprobs( + self, + token_ids: GenericSequence[int], + top_logprobs: GenericSequence[dict[int, Logprob] | None], + num_output_top_logprobs: int | None = None, + ) -> GenerateLogProbs: + """Create generate-shaped logprobs (integer token ids, no tokenizer). + + The engine reports rank 0 for a sampled token whose logprob is NaN; that + is not a rank, so it is sent as ``None`` (the logprob is clamped). + """ + logprobs_content: list[GenerateLogProbsContent] = [] + + for i, token_id in enumerate(token_ids): + step_top_logprobs = top_logprobs[i] + if step_top_logprobs is None or step_top_logprobs.get(token_id) is None: + # Same sentinel the OpenAI shapes use when the sampled token + # has no entry in the engine's top-k map. + logprobs_content.append( + GenerateLogProbsContent(token_id=token_id, logprob=-9999.0) + ) + else: + step_token = step_top_logprobs[token_id] + + logprobs_content.append( + GenerateLogProbsContent( + token_id=token_id, + logprob=_clamp_logprob(step_token.logprob), + rank=step_token.rank or None, + top_logprobs=[ + GenerateLogProb( + token_id=top_token_id, + logprob=_clamp_logprob(top_logprob.logprob), + rank=top_logprob.rank or None, + ) + for rank_index, (top_token_id, top_logprob) in enumerate( + step_top_logprobs.items() + ) + if num_output_top_logprobs is not None + and ( + num_output_top_logprobs == -1 + or rank_index < max(num_output_top_logprobs, 1) + ) + ], + ) + ) + + return GenerateLogProbs(content=logprobs_content) diff --git a/vllm/entrypoints/serve/__init__.py b/vllm/entrypoints/serve/__init__.py index c358b9a6db..d8254cb335 100644 --- a/vllm/entrypoints/serve/__init__.py +++ b/vllm/entrypoints/serve/__init__.py @@ -50,10 +50,6 @@ def register_vllm_dev_api_routers(app: FastAPI): attach_cache_router(app) - from .dev.rlhf.api_router import attach_router as attach_rlhf_router - - attach_rlhf_router(app) - from .dev.rpc.api_router import attach_router as attach_rpc_router attach_rpc_router(app) diff --git a/vllm/entrypoints/serve/profile/api_router.py b/vllm/entrypoints/serve/profile/api_router.py index eeed6b45ef..2051da2b1d 100644 --- a/vllm/entrypoints/serve/profile/api_router.py +++ b/vllm/entrypoints/serve/profile/api_router.py @@ -4,6 +4,7 @@ from fastapi import APIRouter, FastAPI, Request from fastapi.responses import Response +from pydantic import BaseModel, ConfigDict from vllm.config import ProfilerConfig from vllm.engine.protocol import EngineClient @@ -14,14 +15,32 @@ logger = init_logger(__name__) router = APIRouter() +class StartProfileRequest(BaseModel): + """Optional overrides for one profiling session. + + Values are validated by the engine client. + """ + + model_config = ConfigDict(extra="ignore", strict=True) + + profile_prefix: str | None = None + delay_iterations: int | None = None + max_iterations: int | None = None + + def engine_client(request: Request) -> EngineClient: return request.app.state.engine_client @router.post("/start_profile") -async def start_profile(raw_request: Request): +async def start_profile( + raw_request: Request, profile_request: StartProfileRequest | None = None +): logger.info("Starting profiler...") - await engine_client(raw_request).start_profile() + # Only pass supplied overrides so clients without them keep the + # zero-argument call. + overrides = profile_request.model_dump(exclude_none=True) if profile_request else {} + await engine_client(raw_request).start_profile(**overrides) logger.info("Profiler started.") return Response(status_code=200) diff --git a/vllm/entrypoints/serve/tokenize/api_router.py b/vllm/entrypoints/serve/tokenize/api_router.py index 9cdd82fee3..cd515d31d2 100644 --- a/vllm/entrypoints/serve/tokenize/api_router.py +++ b/vllm/entrypoints/serve/tokenize/api_router.py @@ -3,10 +3,10 @@ from http import HTTPStatus +from typing import assert_never from fastapi import APIRouter, Depends, FastAPI, Request from fastapi.responses import JSONResponse -from typing_extensions import assert_never from vllm.entrypoints.serve.engine.protocol import ErrorResponse from vllm.entrypoints.serve.tokenize.protocol import ( diff --git a/vllm/entrypoints/serve/utils/request_logger.py b/vllm/entrypoints/serve/utils/request_logger.py index 09b27e0673..6272054a1e 100644 --- a/vllm/entrypoints/serve/utils/request_logger.py +++ b/vllm/entrypoints/serve/utils/request_logger.py @@ -71,11 +71,10 @@ class RequestLogger: def log_request_body(self, request: AnyRequest | AnyPoolingRequest) -> None: if logger.isEnabledFor(logging.DEBUG): - max_log_len = self.max_log_len if self.max_log_len is not None else -1 logger.debug( "Request %s JSON body: %s", getattr(request, "request_id", "N/A"), - request.model_dump_json(exclude_unset=True)[:max_log_len], + request.model_dump_json(exclude_unset=True)[: self.max_log_len], ) def log_outputs( diff --git a/vllm/entrypoints/speech_to_text/base/serving.py b/vllm/entrypoints/speech_to_text/base/serving.py index 34b7e9c249..871f69666f 100644 --- a/vllm/entrypoints/speech_to_text/base/serving.py +++ b/vllm/entrypoints/speech_to_text/base/serving.py @@ -229,6 +229,7 @@ class SpeechToTextBaseServing(GenerateBaseServing): max_tokens=1, temperature=0.0, allowed_token_ids=allowed_token_ids, + watermarking=False, ) result_generator = self.engine_client.generate( diff --git a/vllm/entrypoints/speech_to_text/realtime/connection.py b/vllm/entrypoints/speech_to_text/realtime/connection.py index ed8ec7f862..4a5e6cef9a 100644 --- a/vllm/entrypoints/speech_to_text/realtime/connection.py +++ b/vllm/entrypoints/speech_to_text/realtime/connection.py @@ -2,6 +2,7 @@ # SPDX-FileCopyrightText: Copyright contributors to the vLLM project import asyncio +import contextlib import json from collections.abc import AsyncGenerator from http import HTTPStatus @@ -48,6 +49,7 @@ class RealtimeConnection: self.serving = serving self.audio_queue: asyncio.Queue[np.ndarray | None] = asyncio.Queue() self.generation_task: asyncio.Task | None = None + self.request_id: str | None = None self._is_connected = False self._is_model_validated = False @@ -204,6 +206,7 @@ class RealtimeConnection: 6. Cleans up the audio queue """ request_id = f"rt-{self.connection_id}-{uuid4()}" + self.request_id = request_id full_text = "" prompt_token_ids_len: int = 0 @@ -263,7 +266,9 @@ class RealtimeConnection: except Exception as e: logger.exception("Error in generation: %s", e) - await self.send_error(sanitize_message(str(e)), "processing_error") + # The socket may already be closed; nothing more we can do. + with contextlib.suppress(Exception): + await self.send_error(sanitize_message(str(e)), "processing_error") async def send( self, event: SessionCreated | TranscriptionDelta | TranscriptionDone @@ -282,8 +287,28 @@ class RealtimeConnection: # Signal audio stream to stop self.audio_queue.put_nowait(None) - # Cancel generation task if running + # Abort the engine request. Cancelling the asyncio task alone does NOT + # abort the underlying engine request: a request parked in the + # VAD-waiting state keeps occupying a slot and generating into a now + # dead websocket (an orphaned request that never frees and starves + # other streams). Aborting also unblocks the generation task, which is + # otherwise stuck awaiting the next engine output. + if self.request_id is not None: + try: + await self.serving.engine_client.abort(self.request_id) + except Exception: + logger.exception("Failed to abort request %s", self.request_id) + self.request_id = None + + # Cancel the generation task if still running, and await it so the + # cancellation (and any generator cleanup) actually completes. if self.generation_task and not self.generation_task.done(): self.generation_task.cancel() + try: + await self.generation_task + except asyncio.CancelledError: + pass + except Exception: + logger.exception("Error awaiting cancelled generation task") logger.debug("Connection cleanup complete: %s", self.connection_id) diff --git a/vllm/entrypoints/speech_to_text/transcription/protocol.py b/vllm/entrypoints/speech_to_text/transcription/protocol.py index e9570fb926..7e66c512c7 100644 --- a/vllm/entrypoints/speech_to_text/transcription/protocol.py +++ b/vllm/entrypoints/speech_to_text/transcription/protocol.py @@ -151,6 +151,9 @@ class TranscriptionRequest(OpenAIBaseModel): to automatically increase the temperature until certain thresholds are hit. """ + watermarking: bool | None = None + """Whether to apply the engine's configured watermark to this request.""" + top_p: float | None = None """Enables nucleus (top-p) sampling, where tokens are selected from the smallest possible set whose cumulative probability exceeds `p`. @@ -226,6 +229,7 @@ class TranscriptionRequest(OpenAIBaseModel): beam_width=n, max_tokens=max_tokens, temperature=temperature, + watermarking=self.watermarking, length_penalty=self.length_penalty, include_stop_str_in_output=self.include_stop_str_in_output, ) @@ -264,6 +268,7 @@ class TranscriptionRequest(OpenAIBaseModel): return SamplingParams.from_optional( temperature=temperature, + watermarking=self.watermarking, max_tokens=max_tokens, seed=self.seed, top_p=top_p, diff --git a/vllm/entrypoints/speech_to_text/translation/protocol.py b/vllm/entrypoints/speech_to_text/translation/protocol.py index ac8c69fdc2..91ec56e2fb 100644 --- a/vllm/entrypoints/speech_to_text/translation/protocol.py +++ b/vllm/entrypoints/speech_to_text/translation/protocol.py @@ -102,6 +102,9 @@ class TranslationRequest(OpenAIBaseModel): to automatically increase the temperature until certain thresholds are hit. """ + watermarking: bool | None = None + """Whether to apply the engine's configured watermark to this request.""" + top_p: float | None = None """Enables nucleus (top-p) sampling, where tokens are selected from the smallest possible set whose cumulative probability exceeds `p`. @@ -215,6 +218,7 @@ class TranslationRequest(OpenAIBaseModel): beam_width=n, max_tokens=max_tokens, temperature=temperature, + watermarking=self.watermarking, length_penalty=self.length_penalty, include_stop_str_in_output=self.include_stop_str_in_output, ) @@ -251,6 +255,7 @@ class TranslationRequest(OpenAIBaseModel): return SamplingParams.from_optional( temperature=temperature, + watermarking=self.watermarking, max_tokens=max_tokens, seed=self.seed, top_p=top_p, diff --git a/vllm/envs.py b/vllm/envs.py index f51100fe8b..851741d525 100755 --- a/vllm/envs.py +++ b/vllm/envs.py @@ -146,7 +146,7 @@ if TYPE_CHECKING: VLLM_ROCM_USE_AITER_RMSNORM: bool = True VLLM_ROCM_USE_AITER_MLA: bool = True VLLM_ROCM_AITER_MLA_ASM_PADDING: Literal["auto", "gluon", "asm"] = "auto" - VLLM_ROCM_AITER_MLA_DCP_VERIFY: Literal["asm", "segmented"] = "segmented" + VLLM_ROCM_AITER_MLA_DCP_VERIFY: Literal["auto", "asm", "segmented"] = "auto" VLLM_ROCM_USE_AITER_MHA: bool = True VLLM_ROCM_USE_AITER_FP4_ASM_GEMM: bool = False VLLM_ROCM_USE_AITER_TRITON_SPARSE_MLA: bool = False @@ -1320,12 +1320,13 @@ environment_variables: dict[str, Callable[[], Any]] = { os.getenv("VLLM_ROCM_USE_AITER_MLA", "True").lower() in ("true", "1") ), # Kernel for causal multi-token (spec-decode) verify steps under decode - # context parallelism on gfx950: "asm" uses AITER's round-robin ASM - # decode, "segmented" the Triton segmented MLA path. + # context parallelism: "asm" uses AITER's round-robin ASM decode (gfx950), + # "segmented" the Triton segmented MLA path. "auto" (default) takes "asm" + # wherever it can serve the shape and "segmented" otherwise. "VLLM_ROCM_AITER_MLA_DCP_VERIFY": env_with_choices( "VLLM_ROCM_AITER_MLA_DCP_VERIFY", - "segmented", - ["asm", "segmented"], + "auto", + ["auto", "asm", "segmented"], case_sensitive=False, ), # Small-head (<16) AITER MLA decode kernel selection. Small head counts diff --git a/vllm/exceptions.py b/vllm/exceptions.py index 9bfd4634bc..11ca13fe0b 100644 --- a/vllm/exceptions.py +++ b/vllm/exceptions.py @@ -181,3 +181,14 @@ class MaxQueuedTokensError(GracefulHTTPError): "Please try again later or on a different instance.", HTTPStatus.SERVICE_UNAVAILABLE, ) + + +class ProfilerAlreadyActiveError(GracefulHTTPError): + """Raised when a profiling session is already active.""" + + def __init__(self): + super().__init__( + "A profiling session is already active. Call /stop_profile before " + "starting another session.", + HTTPStatus.CONFLICT, + ) diff --git a/vllm/inputs/engine.py b/vllm/inputs/engine.py index 26b840f76f..aa8c905433 100644 --- a/vllm/inputs/engine.py +++ b/vllm/inputs/engine.py @@ -3,9 +3,9 @@ # SPDX-License-Identifier: Apache-2.0 # SPDX-FileCopyrightText: Copyright contributors to the vLLM project from collections.abc import Mapping, Sequence -from typing import TYPE_CHECKING, Literal, TypeAlias +from typing import TYPE_CHECKING, Literal, NotRequired, TypeAlias, assert_never -from typing_extensions import NotRequired, TypedDict, assert_never +from typing_extensions import TypedDict from vllm.exceptions import VLLMValidationError diff --git a/vllm/inputs/llm.py b/vllm/inputs/llm.py index a95b274c18..1ef43fd23b 100644 --- a/vllm/inputs/llm.py +++ b/vllm/inputs/llm.py @@ -3,9 +3,9 @@ # SPDX-License-Identifier: Apache-2.0 # SPDX-FileCopyrightText: Copyright contributors to the vLLM project from collections.abc import Mapping, Sequence -from typing import TYPE_CHECKING, Any, TypeAlias, TypeVar, final +from typing import TYPE_CHECKING, Any, NotRequired, TypeAlias, TypeVar, final -from typing_extensions import NotRequired, TypedDict +from typing_extensions import TypedDict if TYPE_CHECKING: import torch diff --git a/vllm/lora/lora_model.py b/vllm/lora/lora_model.py index 593f859936..1964453324 100644 --- a/vllm/lora/lora_model.py +++ b/vllm/lora/lora_model.py @@ -15,7 +15,6 @@ from vllm.lora.utils import ( is_base_embedding_weights, parse_fine_tuned_lora_name, ) -from vllm.model_executor.model_loader.tensorizer import TensorizerConfig from vllm.model_executor.models.utils import WeightsMapper from vllm.utils.torch_utils import PIN_MEMORY @@ -204,7 +203,6 @@ class LoRAModel: dtype: torch.dtype | None = None, model_vocab_size: int | None = None, weights_mapper: WeightsMapper | None = None, - tensorizer_config_dict: dict | None = None, skip_prefixes: list[str] | None = None, moe_ep_spec: MoEEPLoadSpec | None = None, ) -> "LoRAModel": @@ -223,8 +221,6 @@ class LoRAModel: embedding deltas. weights_mapper: Optional mapper rewriting checkpoint weight names to vLLM names. - tensorizer_config_dict: Optional tensorizer config used to load - the checkpoint via tensorizer instead of from disk. skip_prefixes: List of module name prefixes to skip during loading. Models can define this to skip modules not used in inference (e.g., MTP layers). Format: ["mtp."] @@ -280,24 +276,7 @@ class LoRAModel: f" Please verify that the loaded LoRA module is correct" ) - if tensorizer_config_dict: - from tensorizer import TensorDeserializer - - tensorizer_config = TensorizerConfig(**tensorizer_config_dict) - tensorizer_dir = tensorizer_config.tensorizer_dir - if tensorizer_dir is None: - raise ValueError("tensorizer_dir must be set in tensorizer config.") - lora_tensor_path = os.path.join(tensorizer_dir, "adapter_model.tensors") - tensorizer_args = tensorizer_config._construct_tensorizer_args() - tensors = TensorDeserializer( - lora_tensor_path, - dtype=tensorizer_config.dtype, - device=device, - **tensorizer_args.deserialization_kwargs, - ) - check_unexpected_modules(tensors) - - elif os.path.isfile(lora_tensor_path): + if os.path.isfile(lora_tensor_path): # Find unexpected modules. # Use safetensor key as a source of truth to find expected modules. # in peft if you have target_modules A, B, C and C does not exist diff --git a/vllm/lora/model_manager.py b/vllm/lora/model_manager.py index adc2353b3e..8dc7b56056 100644 --- a/vllm/lora/model_manager.py +++ b/vllm/lora/model_manager.py @@ -144,20 +144,18 @@ class LoRAModelManager: else set() ) - # When the engine is started with enable_mixed_moe_lora_format=True - # we force the universal 2D wrapper (FusedMoEWithLoRA) regardless of - # the model's 3D flag, so 2D and 3D adapters can coexist. + # Mixed-format and shared-outer adapters both use FusedMoEWithLoRA. + # FusedMoE3DWithLoRA only implements the fused gate/up PEFT pair. self._enable_mixed_moe_lora_format = ( is_moe and lora_config.enable_mixed_moe_lora_format ) + self._enable_moe_shared_loras = is_moe and lora_config.enable_moe_shared_loras self._is_3d_moe_model = ( self._is_moe and self.model.is_3d_moe_weight and not self._enable_mixed_moe_lora_format + and not self._enable_moe_shared_loras ) - # Shared MoE adapters: w13 lora_A / w2 lora_B shared across experts, - # stored as pre-stacked experts.w{1,2,3} tensors (startup opt-in). - self._enable_moe_shared_loras = is_moe and lora_config.enable_moe_shared_loras self.packed_modules_mapping = process_packed_modules_mapping( self.model, force_2d_moe=self._enable_mixed_moe_lora_format, @@ -328,8 +326,8 @@ class LoRAModelManager: self.punica_wrapper_mapping[prefix] = connector_punica_wrapper else: logger.warning_once( - "Connector LoRA support disabled: model does not implement " - "get_num_mm_connector_tokens(). This method is required to " + "Connector LoRA support disabled: get_mm_lora_token_counts() " + "returned no connector token counts, which are required to " "determine the connector's token budget for LoRA operations." ) @@ -416,6 +414,7 @@ class LoRAModelManager: pass def _add_adapter(self, lora: LoRAModel): + self._validate_moe_lora_format(lora) self._create_merged_loras_inplace(lora) self._registered_adapters[lora.id] = lora @@ -1254,6 +1253,27 @@ class LoRAModelManager: ) return weights + def _validate_moe_lora_format(self, lora_model: LoRAModel) -> None: + if self._enable_mixed_moe_lora_format and getattr( + lora_model, "is_3d_lora_weight", False + ): + return + for module_name, module in self.modules.items(): + if not isinstance(module, FusedMoEWithLoRA) or isinstance( + module, FusedMoE3DWithLoRA + ): + continue + module_lora = self._get_lora_layer_weights(lora_model, module_name) + if module_lora is not None and torch.is_tensor(module_lora.lora_a): + raise ValueError( + f"LoRA adapter {lora_model.id} contains fused 3D MoE weights " + f"for {module_name!r}, but the model uses a 2D MoE LoRA " + "wrapper. Start the engine with " + "enable_mixed_moe_lora_format=True " + "(--enable-mixed-moe-lora-format) and set " + "is_3d_lora_weight=True on the LoRA request." + ) + def _validate_modules_to_save(self, lora_model: LoRAModel) -> None: if not lora_model.modules_to_save: return diff --git a/vllm/lora/peft_helper.py b/vllm/lora/peft_helper.py index baffe16987..de05dea1cf 100644 --- a/vllm/lora/peft_helper.py +++ b/vllm/lora/peft_helper.py @@ -11,7 +11,6 @@ from typing import Literal from vllm.config.lora import LoRAConfig from vllm.logger import init_logger -from vllm.model_executor.model_loader.tensorizer import TensorizerConfig logger = init_logger(__name__) @@ -34,6 +33,14 @@ class PEFTHelper: use_rslora: bool = field(default=False) # True to use Weight-Decomposed Low-Rank Adaptation (DoRA, see: https://arxiv.org/abs/2402.09353) use_dora: bool = field(default=False) + # PEFT features that change the adapter's semantics and that vLLM does not + # implement. They are read only to reject such adapters. + lora_bias: bool = field(default=False) + init_lora_weights: bool | str = field(default=True) + alora_invocation_tokens: list[int] | None = field(default=None) + layer_replication: list[list[int]] | None = field(default=None) + use_bdlora: dict | None = field(default=None) + use_qalora: bool = field(default=False) # Extra vllm field, start with 'vllm_' to avoid conflict vllm_lora_scaling_factor: float = field(default=1.0) vllm_max_position_embeddings: int | None = field(default=False) @@ -53,8 +60,40 @@ class PEFTHelper: ) if self.use_dora: error_msg.append("vLLM does not yet support DoRA.") + if self.lora_bias: + error_msg.append("vLLM does not support LoRA bias (lora_bias).") + if not self._init_is_supported(): + error_msg.append( + f"vLLM does not support init_lora_weights={self.init_lora_weights!r}." + " Inits such as PiSSA, OLoRA, CorDA and LoftQ modify the base model" + " weights when PEFT loads the adapter. Convert it to a regular LoRA" + " adapter with PEFT's path_initial_model_for_weight_conversion, or" + " set init_lora_weights to true if the served base model already" + " contains the modified weights." + ) + if self.alora_invocation_tokens: + error_msg.append("vLLM does not support Activated LoRA (aLoRA).") + if self.layer_replication: + error_msg.append("vLLM does not support layer_replication.") + if self.use_bdlora: + error_msg.append("vLLM does not support block-diagonal LoRA (BD-LoRA).") + if self.use_qalora: + error_msg.append("vLLM does not support QALoRA.") return error_msg + def _init_is_supported(self) -> bool: + # These inits only set the initial adapter weights, so a saved adapter + # loads in PEFT as a plain LoRA. lora_ga modifies the base weights only + # during training, when its gradients are attached. + init = self.init_lora_weights + return not isinstance(init, str) or init.lower() in ( + "gaussian", + "eva", + "orthogonal", + "mica", + "lora_ga", + ) + def __post_init__(self): if self.r <= 0: raise ValueError(f"LoRA rank `r` must be a positive integer, got {self.r}.") @@ -89,33 +128,10 @@ class PEFTHelper: cls, lora_path: str, max_position_embeddings: int | None, - tensorizer_config_dict: dict | None = None, ) -> "PEFTHelper": lora_config_path = os.path.join(lora_path, "adapter_config.json") - - if tensorizer_config_dict: - tensorizer_config = TensorizerConfig(**tensorizer_config_dict) - tensorizer_args = tensorizer_config._construct_tensorizer_args() - from tensorizer.stream_io import open_stream - - tensorizer_dir = tensorizer_config.tensorizer_dir - if tensorizer_dir is None: - raise ValueError("tensorizer_dir must be set in tensorizer config.") - - lora_config_path = os.path.join(tensorizer_dir, "adapter_config.json") - with open_stream( - lora_config_path, mode="rb", **tensorizer_args.stream_kwargs - ) as f: - config = json.load(f) - - logger.info( - "Successfully deserialized LoRA config from %s", - tensorizer_config.tensorizer_dir, - ) - - else: - with open(lora_config_path) as f: - config = json.load(f) + with open(lora_config_path) as f: + config = json.load(f) config["vllm_max_position_embeddings"] = max_position_embeddings return cls.from_dict(config) diff --git a/vllm/lora/request.py b/vllm/lora/request.py index cd3df54d3b..36c6881270 100644 --- a/vllm/lora/request.py +++ b/vllm/lora/request.py @@ -25,7 +25,6 @@ class LoRARequest( lora_int_id: int lora_path: str = "" base_model_name: str | None = msgspec.field(default=None) - tensorizer_config_dict: dict | None = None load_inplace: bool = False is_3d_lora_weight: bool = False """Whether this adapter's MoE weights are stored in the 3D fused diff --git a/vllm/lora/utils.py b/vllm/lora/utils.py index 71a2e60dc7..1766c6de62 100644 --- a/vllm/lora/utils.py +++ b/vllm/lora/utils.py @@ -205,7 +205,7 @@ def parse_fine_tuned_lora_name( parts = name.split(".") if (parts[-1] == "weight" or parts[-1] == "bias") and len(parts) >= 2: - if parts[-2] in ["lora_A", "lora_B"]: + if parts[-2] in ["lora_A", "lora_B"] and parts[-1] == "weight": new_name = ".".join(parts[start_index:-2]) return new_name, parts[-2] == "lora_A" # For modules_to_save in classification. @@ -409,13 +409,11 @@ def process_packed_modules_mapping( "experts.w3", ] elif (not model.is_3d_moe_weight) or force_2d_moe: - # Filter out malformed entries: non-gated MoE has empty - # ckpt_up_proj_name which results in weight_name containing ".." - # (e.g., "experts.0.." instead of "experts.0.layer_name.") + # Non-gated MoE yields two entries per expert (no w3); the + # manager pads them to triplets before packing. packed_modules_mapping["experts"] = [ weight_name.rstrip(".") for _, weight_name, _, _ in get_moe_expert_mapping(model) - if ".." not in weight_name ] return packed_modules_mapping diff --git a/vllm/lora/worker_manager.py b/vllm/lora/worker_manager.py index 93624523e2..9427b3e134 100644 --- a/vllm/lora/worker_manager.py +++ b/vllm/lora/worker_manager.py @@ -119,9 +119,7 @@ class WorkerLoRAManager: lora_path = get_adapter_absolute_path(lora_request.lora_path) peft_helper = PEFTHelper.from_local_dir( - lora_path, - self.max_position_embeddings, - lora_request.tensorizer_config_dict, + lora_path, self.max_position_embeddings ) # Validates the LoRA configuration against requirements before @@ -147,7 +145,6 @@ class WorkerLoRAManager: device="cpu", dtype=self.lora_config.lora_dtype, model_vocab_size=self.vocab_size, - tensorizer_config_dict=lora_request.tensorizer_config_dict, weights_mapper=hf_to_vllm_mapper, skip_prefixes=lora_skip_prefixes, moe_ep_spec=self._adapter_manager.moe_ep_load_spec, @@ -156,6 +153,7 @@ class WorkerLoRAManager: # adapter manager can route 3D-format checkpoints through the # 3D->2D conversion when running under the universal 2D wrapper. lora.is_3d_lora_weight = lora_request.is_3d_lora_weight + self._adapter_manager._validate_moe_lora_format(lora) # Validate classification-head weights. self._adapter_manager._validate_modules_to_save(lora) diff --git a/vllm/model_executor/kernels/attention/dsa/dcp_indexer_cutedsl.py b/vllm/model_executor/kernels/attention/dsa/dcp_indexer_cutedsl.py index 34ce32856a..8b5d6ab1a5 100644 --- a/vllm/model_executor/kernels/attention/dsa/dcp_indexer_cutedsl.py +++ b/vllm/model_executor/kernels/attention/dsa/dcp_indexer_cutedsl.py @@ -25,6 +25,22 @@ from vllm.model_executor.warmup.jit_warmup_triton_helper import ( from vllm.triton_utils import tl, triton +def _dcp_merge_topk_and_interleave(vllm_config: Any) -> tuple[int, int]: + """(topk, cp_interleave) the DCP merge runs with. + + Kpool indexers (GLM-5.x) merge pool ids: topk and the interleave are both + in units of ``index_kpool`` tokens. + """ + hf_text_config = vllm_config.model_config.hf_text_config + topk = hf_text_config.index_topk + cp_interleave = vllm_config.parallel_config.cp_kv_cache_interleave_size + kpool = getattr(hf_text_config, "index_kpool", None) or 1 + if kpool > 1: + topk //= kpool + cp_interleave //= kpool + return topk, cp_interleave + + def stable_topk_from_gathered_candidates_cutedsl( gathered: torch.Tensor, topk: int, @@ -171,8 +187,7 @@ class PackDCPTopkCandidatesKernel( dcp_world_size = vllm_config.parallel_config.decode_context_parallel_size if dcp_world_size <= 1: return [] - cp_interleave = vllm_config.parallel_config.cp_kv_cache_interleave_size - topk = vllm_config.model_config.hf_text_config.index_topk + topk, cp_interleave = _dcp_merge_topk_and_interleave(vllm_config) if topk <= 0: return [] @@ -545,7 +560,7 @@ class StableTopKFromGatheredCandidatesKernel( dcp_world_size = vllm_config.parallel_config.decode_context_parallel_size if dcp_world_size <= 1: return [] - topk = vllm_config.model_config.hf_text_config.index_topk + topk, _ = _dcp_merge_topk_and_interleave(vllm_config) if topk <= 0: return [] return self._trace_dispatch(self.dispatch)( diff --git a/vllm/model_executor/kernels/linear/__init__.py b/vllm/model_executor/kernels/linear/__init__.py index e5fed78c40..484143ffc1 100644 --- a/vllm/model_executor/kernels/linear/__init__.py +++ b/vllm/model_executor/kernels/linear/__init__.py @@ -228,6 +228,10 @@ from vllm.model_executor.kernels.linear.scaled_mm.xpu import ( from vllm.model_executor.kernels.linear.scaled_mm.zentorch import ( ZentorchInt8ScaledMMLinearKernel, ) +from vllm.model_executor.layers.quantization.utils.humming import ( + prefers_humming, + prioritize_humming, +) from vllm.model_executor.layers.quantization.utils.quant_utils import QuantKey from vllm.platforms import PlatformEnum, current_platform @@ -376,8 +380,9 @@ def _resolve_backend_kernels( layer_desc: str, *, quantization: str, + compute_capability: int | None = None, ) -> list[type]: - """Apply --linear-backend filtering to one layer type's kernel list. + """Apply device priorities and --linear-backend filtering to a kernel list. When the requested backend has no kernel for this layer type, fall back to the unfiltered list (with a WARNING log) instead of failing engine @@ -386,6 +391,7 @@ def _resolve_backend_kernels( layer types (e.g. NVFP4 MoE projections next to FP8 attention projections). """ + kernels = prioritize_humming(kernels, compute_capability) linear_backend = _get_linear_backend(quantization=quantization) if linear_backend == "auto": return kernels @@ -682,6 +688,7 @@ def choose_scaled_mm_linear_kernel( platform_kernels, "scaled-mm", quantization=quantization, + compute_capability=compute_capability, ) for kernel in platform_kernels: @@ -845,6 +852,7 @@ def choose_mp_linear_kernel( platform_kernels, "mixed-precision", quantization="mixed_precision", + compute_capability=compute_capability, ) failure_reasons = [] @@ -1121,13 +1129,13 @@ def init_nvfp4_linear_kernel(use_a16: bool = False) -> NvFp4LinearKernel: _cc = current_platform.get_device_capability() compute_capability = _cc.to_int() if _cc is not None else None # Weight-only: prefer FlashInfer CuTe-DSL W4A16 on SM100/103, - # otherwise Marlin. + # Humming then Marlin where Humming is preferred, and Marlin elsewhere. cutedsl_ok, _ = FlashInferCuteDslNvFp4W4A16LinearKernel.is_supported( compute_capability ) if compute_capability in (100, 103) and cutedsl_ok: force_kernel = FlashInferCuteDslNvFp4W4A16LinearKernel - else: + elif not prefers_humming(compute_capability): force_kernel = MarlinNvFp4LinearKernel if force_kernel is not None: diff --git a/vllm/model_executor/kernels/linear/base.py b/vllm/model_executor/kernels/linear/base.py index 0db595a3ca..af1f597ec6 100644 --- a/vllm/model_executor/kernels/linear/base.py +++ b/vllm/model_executor/kernels/linear/base.py @@ -3,10 +3,9 @@ from abc import ABC, abstractmethod from dataclasses import dataclass -from typing import Any, ClassVar, Generic, TypeVar +from typing import Any, ClassVar, Generic, Self, TypeVar import torch -from typing_extensions import Self from vllm.model_executor.layers.quantization.utils.quant_utils import QuantKey diff --git a/vllm/model_executor/kernels/linear/cute_dsl/skinny_gemm.py b/vllm/model_executor/kernels/linear/cute_dsl/skinny_gemm.py index ec1e607caf..e813fe2c5c 100644 --- a/vllm/model_executor/kernels/linear/cute_dsl/skinny_gemm.py +++ b/vllm/model_executor/kernels/linear/cute_dsl/skinny_gemm.py @@ -26,6 +26,20 @@ class SkinnyGemmConfig: static_k: int | None = None +def row_stride_ok(a: torch.Tensor, config: SkinnyGemmConfig) -> bool: + """Whether the vectorized A loads stay aligned for a row-major ``a``. + + The kernel reads A through its layout, so column slices of a wider buffer + are supported without a copy. + """ + vector_bytes = config.vector_width * a.element_size() + return ( + a.stride(1) == 1 + and a.stride(0) % config.vector_width == 0 + and a.data_ptr() % vector_bytes == 0 + ) + + class ShapeDynamicSkinnyGemm: def __init__(self) -> None: self._compiled: dict[tuple[torch.dtype, SkinnyGemmConfig, bool], Any] = {} @@ -209,8 +223,8 @@ class ShapeDynamicSkinnyGemm: raise ValueError("a and b must have the same BF16 or FP16 dtype") if not a.is_cuda or not b.is_cuda or a.device != b.device: raise ValueError("a and b must be CUDA tensors on the same device") - if not a.is_contiguous() or not b.is_contiguous(): - raise ValueError("a and b must be contiguous") + if not b.is_contiguous(): + raise ValueError("b must be contiguous") if a.shape[1] != b.shape[1]: raise ValueError("a and b must have matching K dimensions") if not 1 <= a.shape[0] <= 16: @@ -236,6 +250,8 @@ class ShapeDynamicSkinnyGemm: ) if config.static_k is not None and a.shape[1] != config.static_k: raise ValueError("input K must match config static_k") + if not row_stride_ok(a, config): + raise ValueError("a must be row-major with vector_width-aligned rows") has_residual = residual is not None cache_key = (a.dtype, config, has_residual) if cache_key not in self._compiled: diff --git a/vllm/model_executor/kernels/linear/mixed_precision/humming.py b/vllm/model_executor/kernels/linear/mixed_precision/humming.py index 4655181f78..a2d88705d2 100644 --- a/vllm/model_executor/kernels/linear/mixed_precision/humming.py +++ b/vllm/model_executor/kernels/linear/mixed_precision/humming.py @@ -4,11 +4,21 @@ import torch +from vllm.model_executor.layers.quantization.utils.quant_utils import ( + kFp8DynamicTokenSym, + kInt8DynamicTokenSym, +) from vllm.platforms import current_platform from vllm.utils.import_utils import has_humming from .MPLinearKernel import MPLinearKernel, MPLinearLayerConfig +# Per-token activation quantization requested through `act_type`. +_ACT_QUANT_KEYS = { + torch.int8: kInt8DynamicTokenSym, + torch.float8_e4m3fn: kFp8DynamicTokenSym, +} + class HummingLinearKernel(MPLinearKernel): @classmethod @@ -21,6 +31,8 @@ class HummingLinearKernel(MPLinearKernel): return False, "Humming is only supported on CUDA" if not has_humming(): return False, "Humming is not installed" + if c.act_type.itemsize == 1 and c.act_type not in _ACT_QUANT_KEYS: + return False, f"Activation type ({c.act_type}) not supported by Humming" return True, None def process_weights_after_loading(self, layer: torch.nn.Module) -> None: @@ -28,6 +40,7 @@ class HummingLinearKernel(MPLinearKernel): convert_linear_layer_to_humming_standard, get_humming_linear_compute_config, prepare_humming_linear_layer_config, + quant_key_to_input_schema, ) name_map = {"weight": self.w_q_name, "weight_scale": self.w_s_name} @@ -48,8 +61,14 @@ class HummingLinearKernel(MPLinearKernel): convert_linear_layer_to_humming_standard(layer=layer, name_map=name_map) input_quant_config = getattr(layer, "_humming_input_quant_config", None) + input_schema = None + if (act_key := _ACT_QUANT_KEYS.get(self.config.act_type)) is not None: + input_schema = quant_key_to_input_schema(act_key) self.layer_config = prepare_humming_linear_layer_config( - layer, quant_config, input_quant_config=input_quant_config + layer, + quant_config, + input_quant_config=input_quant_config, + input_schema=input_schema, ) self.compute_config = get_humming_linear_compute_config() self.locks = torch.zeros(1024, dtype=torch.int32, device=layer.weight.device) diff --git a/vllm/model_executor/kernels/linear/mixed_precision/rdna_hybrid_w4a16.py b/vllm/model_executor/kernels/linear/mixed_precision/rdna_hybrid_w4a16.py index 86e029b5f8..bed62e89af 100644 --- a/vllm/model_executor/kernels/linear/mixed_precision/rdna_hybrid_w4a16.py +++ b/vllm/model_executor/kernels/linear/mixed_precision/rdna_hybrid_w4a16.py @@ -2,9 +2,10 @@ # SPDX-FileCopyrightText: Copyright contributors to the vLLM project """Hybrid W4A16 kernel: Triton for prefill, HIP skinny for decode. -Routes based on batch size M: - M <= MAX_SKINNY_BATCH_SIZE: HIP skinny GEMM (wvSplitK_int4_g) - M > MAX_SKINNY_BATCH_SIZE: Triton W4A16 fused dequant GEMM +Routes based on the activation shape: + M <= MAX_SKINNY_BATCH_SIZE and K*M <= MEDIUM_SKINNY_LIMIT_ELEMENTS: + HIP skinny GEMM (wvSplitK_int4_g) + otherwise: Triton W4A16 fused dequant GEMM Stores the weights ONCE as int8 [N, K//2] (ExLlama shuffle packed). Both paths read this single buffer: the HIP skinny kernel uses it directly, and @@ -56,13 +57,14 @@ def _on_gfx1151() -> bool: return on_gfx1151() -# Maximum batch size M for the HIP skinny kernel path (C++ supports N_in -# up to 5). When M is below this AND K*M fits in LDS, the skinny kernel is -# used; otherwise the Triton prefill path handles the GEMM. +# Maximum activation rows supported by HIP skinny (Python M maps to C++ N_in). +# When K*M exceeds regular LDS capacity, the C++ medium kernel caches the +# prefix in LDS and reads the remaining activation elements from global memory. MAX_SKINNY_BATCH_SIZE = 5 # 64 KiB per-workgroup LDS limit expressed in fp16 elements. # (AMD RDNA has 128 KiB total LDS per CU, but 64 KiB per workgroup.) LDS_CAPACITY_ELEMENTS = 64 * 1024 // 2 # 32768 fp16 elements +MEDIUM_SKINNY_LIMIT_ELEMENTS = int(LDS_CAPACITY_ELEMENTS * 1.2) # --------------------------------------------------------------------------- @@ -84,6 +86,8 @@ def _triton_w4a16_skinny_fmt_kernel( K, K8, # K // 8 num_groups, # K // group_size + stride_bn, # b_ptr row stride; >= K8, the rows may be padded + stride_am, # a_ptr row stride in elements; >= K, rows may be padded # Quantization parameters group_size, ZP_BIAS: tl.constexpr, @@ -127,12 +131,12 @@ def _triton_w4a16_skinny_fmt_kernel( offs_k = k_start * BLOCK_K + tl.arange(0, BLOCK_K) mask_k = offs_k < K - a_ptrs = a_ptr + offs_m[:, None] * K + offs_k[None, :] + a_ptrs = a_ptr + offs_m[:, None] * stride_am + offs_k[None, :] mask_a = (offs_m[:, None] < M) & mask_k[None, :] a = tl.load(a_ptrs, mask=mask_a, other=0.0) offs_k8 = k_start * (BLOCK_K // 8) + tl.arange(0, BLOCK_K // 8) - b_ptrs = b_ptr + offs_n[:, None] * K8 + offs_k8[None, :] + b_ptrs = b_ptr + offs_n[:, None] * stride_bn + offs_k8[None, :] mask_b = (offs_n[:, None] < N) & (offs_k8[None, :] < K8) b_packed = tl.load(b_ptrs, mask=mask_b, other=0) @@ -210,14 +214,16 @@ def triton_w4a16_skinny_fmt_gemm( Output matrix [M, N], same dtype as a. """ - assert a.is_contiguous(), "Activation matrix must be contiguous" - assert b_q.is_contiguous(), "Weight matrix must be contiguous" + assert a.stride(1) == 1, "Activation rows must be contiguous" + assert b_q.stride(1) == 1, "Weight rows must be contiguous" assert scales.is_contiguous(), "Scales must be contiguous" M, K = a.shape N = b_q.shape[0] K8 = K // 8 num_groups = K // group_size + stride_bn = b_q.stride(0) + stride_am = a.stride(0) assert b_q.shape == (N, K8), f"b_q shape mismatch: {b_q.shape} vs ({N}, {K8})" assert scales.shape == (N, num_groups), ( @@ -331,6 +337,8 @@ def triton_w4a16_skinny_fmt_gemm( K, K8, num_groups, + stride_bn, + stride_am, group_size=group_size, ZP_BIAS=zp_bias, HAS_ZP=has_zp, @@ -368,11 +376,77 @@ def pack_int4_exllama_shuffle(w_uint4: torch.Tensor) -> torch.Tensor: ) +# gfx11 row strides alias in the vector caches when they land on a power-of-two +# multiple. +_WEIGHT_CLIFF_BYTES = 1024 # packed weight row, K/2 B +_ACT_CLIFF_BYTES = 2048 # activation row, K*2 B +_STRIDE_PAD_BYTES = 128 # one cache line + + +def _weight_pad_bytes(row_bytes: int) -> int: + """Bytes to add to a packed weight row stride to move it off the cliff. + + Only strides that are a multiple of ``_WEIGHT_CLIFF_BYTES`` are moved, and + only by one cache line; padding a stride that is already off the cliff + costs memory and measures slower. + """ + if row_bytes % _WEIGHT_CLIFF_BYTES: + return 0 + return _STRIDE_PAD_BYTES + + +def _act_pad_bytes(row_bytes: int) -> int: + """Bytes to add to an activation row stride to move it off the cliff.""" + if row_bytes % _ACT_CLIFF_BYTES: + return 0 + return _STRIDE_PAD_BYTES + + +def pack_skinny_int4(unpacked: torch.Tensor) -> torch.Tensor: + """Pack [N, K] uint4 into the skinny weight layout the kernels consume. + + ExLlama shuffle to [N, K//8] int32, viewed as int8 [N, K//2]. On gfx1151 + the row stride is nudged off the cliff (see ``_weight_pad_bytes``). + """ + shuffled = pack_int4_exllama_shuffle(unpacked) + n_rows, k8 = shuffled.shape + pad_int32 = _weight_pad_bytes(k8 * 4) // 4 + if not (pad_int32 and _on_gfx1151()): + return shuffled.contiguous().view(torch.int8) + padded = torch.empty( + (n_rows, k8 + pad_int32), dtype=torch.int32, device=shuffled.device + ) + padded[:, :k8].copy_(shuffled) + # The int8 view keeps stride(0) = 4 * (k8 + pad_int32) bytes, and + # .view(torch.int32) at apply time recovers the [N, K//8] int32 view. + return padded.view(torch.int8)[:, : k8 * 4] + + # --------------------------------------------------------------------------- # Hybrid dispatch logic # --------------------------------------------------------------------------- +def _pad_activation_rows(x_2d: torch.Tensor) -> torch.Tensor: + """Copy ``x_2d`` into a row-padded buffer when its row stride is on the cliff. + + The packed weight is padded once at load time, but activations are produced + fresh every step, so this materialises a padded copy. + """ + pad_bytes = _act_pad_bytes(x_2d.shape[1] * x_2d.element_size()) + if not (pad_bytes and _on_gfx1151()): + return x_2d + pad_elems = pad_bytes // x_2d.element_size() + buf = torch.empty( + (x_2d.shape[0], x_2d.shape[1] + pad_elems), + dtype=x_2d.dtype, + device=x_2d.device, + ) + padded = buf[:, : x_2d.shape[1]] + padded.copy_(x_2d) + return padded + + def _rdna_hybrid_w4a16_apply_impl( x_2d: torch.Tensor, w_q: torch.Tensor, @@ -392,7 +466,7 @@ def _rdna_hybrid_w4a16_apply_impl( K = x_2d.shape[1] N = w_q.shape[0] - if M <= MAX_SKINNY_BATCH_SIZE and K * M <= LDS_CAPACITY_ELEMENTS: + if M <= MAX_SKINNY_BATCH_SIZE and K * M <= MEDIUM_SKINNY_LIMIT_ELEMENTS: # record_function is not torch.compile-safe; use nullcontext when # compiling to keep the op traceable. ctx = ( @@ -410,7 +484,7 @@ def _rdna_hybrid_w4a16_apply_impl( ) with ctx: output = triton_w4a16_skinny_fmt_gemm( - a=x_2d, + a=_pad_activation_rows(x_2d), b_q=w_q.view(torch.int32), scales=w_s, group_size=group_size, @@ -513,9 +587,8 @@ class RDNAHybridW4A16LinearKernel(MPLinearKernel): if getattr(w_q_raw, "output_dim", 0) != 0: unpacked = unpacked.t().contiguous() - shuffled = pack_int4_exllama_shuffle(unpacked) # Store as int8; Triton reinterprets via .view(torch.int32) at apply time. - w_q_skinny = shuffled.contiguous().view(torch.int8) + w_q_skinny = pack_skinny_int4(unpacked) permute_param_layout_(w_s_raw, input_dim=1, output_dim=0) w_s_skinny = w_s_raw.data.contiguous() diff --git a/vllm/model_executor/kernels/linear/nvfp4/flashinfer.py b/vllm/model_executor/kernels/linear/nvfp4/flashinfer.py index a555ca17bf..2a69a3be21 100644 --- a/vllm/model_executor/kernels/linear/nvfp4/flashinfer.py +++ b/vllm/model_executor/kernels/linear/nvfp4/flashinfer.py @@ -112,6 +112,9 @@ class FlashInferCuteDslNvFp4W4A16LinearKernel(NvFp4LinearKernel): class FlashInferCuteDslNvFp4LinearKernel(NvFp4LinearKernel): """NVFP4 GEMM via FlashInfer's cutedsl backend.""" + def input_quant_key(self) -> QuantKey | None: + return kNvfp4Dynamic + @classmethod def is_supported( cls, compute_capability: int | None = None @@ -137,19 +140,26 @@ class FlashInferCuteDslNvFp4LinearKernel(NvFp4LinearKernel): def apply_weights( self, layer: torch.nn.Module, - x: torch.Tensor, + x: torch.Tensor | QuantizedActivation, bias: torch.Tensor | None = None, ) -> torch.Tensor: output_size = layer.output_size_per_partition - output_dtype = x.dtype - output_shape = [*x.shape[:-1], output_size] - - x_fp4, x_blockscale = scaled_fp4_quant( - x, - layer.input_global_scale_inv, - is_sf_swizzled_layout=True, - backend="flashinfer-cutedsl", - ) + qa = as_quantized_activation(x, self.input_quant_key()) + if qa is not None: + x_fp4, x_blockscale = qa.data, qa.scale + x_fp4 = x_fp4.reshape(-1, x_fp4.shape[-1]) + output_dtype = qa.orig_dtype + output_shape = [*qa.orig_shape[:-1], output_size] + else: + assert isinstance(x, torch.Tensor) + output_dtype = x.dtype + output_shape = [*x.shape[:-1], output_size] + x_fp4, x_blockscale = scaled_fp4_quant( + x, + layer.input_global_scale_inv, + is_sf_swizzled_layout=True, + backend="flashinfer-cutedsl", + ) x_fp4 = pad_nvfp4_activation_for_cutlass( x_fp4, nvfp4_weight_padding_bytes(layer) @@ -219,6 +229,7 @@ class FlashInferCutlassNvFp4LinearKernel(NvFp4LinearKernel): qa = as_quantized_activation(x, self.input_quant_key()) if qa is not None: x_fp4, x_blockscale = qa.data, qa.scale + x_fp4 = x_fp4.reshape(-1, x_fp4.shape[-1]) x_fp4 = pad_nvfp4_activation_for_cutlass(x_fp4, weights_padding_bytes) output_dtype = qa.orig_dtype output_shape = [*qa.orig_shape[:-1], output_size] diff --git a/vllm/model_executor/kernels/linear/scaled_mm/BlockScaledMMLinearKernel.py b/vllm/model_executor/kernels/linear/scaled_mm/BlockScaledMMLinearKernel.py index 3a053412f3..865bebeada 100644 --- a/vllm/model_executor/kernels/linear/scaled_mm/BlockScaledMMLinearKernel.py +++ b/vllm/model_executor/kernels/linear/scaled_mm/BlockScaledMMLinearKernel.py @@ -3,10 +3,9 @@ from abc import ABC, abstractmethod from dataclasses import dataclass -from typing import ClassVar +from typing import ClassVar, Self import torch -from typing_extensions import Self from vllm.model_executor.layers.quantization.input_quant_fp8 import QuantFP8 from vllm.model_executor.layers.quantization.utils.fp8_utils import ( diff --git a/vllm/model_executor/layers/attention/mla_attention.py b/vllm/model_executor/layers/attention/mla_attention.py index b952a36821..2a3048096c 100644 --- a/vllm/model_executor/layers/attention/mla_attention.py +++ b/vllm/model_executor/layers/attention/mla_attention.py @@ -802,18 +802,6 @@ class MLAAttention(nn.Module, AttentionLayerBase): kv_cache_dtype, k_scale, ) - if cache is not None: - mirror_target = cache.mirror_write_target(kv_c_normed.shape[0]) - if mirror_target is not None: - mirror_cache, mirror_slots = mirror_target - self.impl.do_kv_cache_update( # type: ignore[attr-defined] - kv_c_normed, - k_pe, - mirror_cache, - mirror_slots, - kv_cache_dtype, - k_scale, - ) def prepare_kv_cache_update( self, attn_metadata: "MLACommonMetadata | None" @@ -1272,23 +1260,34 @@ class MLAAttention(nn.Module, AttentionLayerBase): quark_quantize_weight_to_mxfp4, ) - self.W_K, self.W_K_scale = quark_quantize_weight_to_mxfp4(W_UK) - # Convert from (L, N, P) to (N, L, P) - self.W_K = self.W_K.transpose(0, 1) - self.W_K_scale = self.W_K_scale.transpose(0, 1) - - self.W_V, self.W_V_scale = quark_quantize_weight_to_mxfp4( - W_UV.permute(1, 2, 0) - ) + W_K, W_K_scale = quark_quantize_weight_to_mxfp4(W_UK) + W_V, W_V_scale = quark_quantize_weight_to_mxfp4(W_UV.permute(1, 2, 0)) + # Copy in place on reload so captured CUDA graphs stay valid. + # Convert W_K from (L, N, P) to (N, L, P). + for name, tensor in ( + ("W_K", W_K.transpose(0, 1)), + ("W_K_scale", W_K_scale.transpose(0, 1)), + ("W_V", W_V), + ("W_V_scale", W_V_scale), + ): + replace_parameter(self, name, tensor, prefer_copy=True) elif self.is_aiter_triton_fp8_bmm_enabled: W_K = W_UK.transpose(0, 1) # 16 512 128 W_V = W_UV.permute(1, 2, 0) # 16 128 512 - self.W_K, self.W_K_scale = dynamic_per_batched_tensor_quant( + W_K, W_K_scale = dynamic_per_batched_tensor_quant( W_K, dtype=current_platform.fp8_dtype() ) - self.W_V, self.W_V_scale = dynamic_per_batched_tensor_quant( + W_V, W_V_scale = dynamic_per_batched_tensor_quant( W_V, dtype=current_platform.fp8_dtype() ) + # Copy in place on reload so captured CUDA graphs stay valid. + for name, tensor in ( + ("W_K", W_K), + ("W_K_scale", W_K_scale), + ("W_V", W_V), + ("W_V_scale", W_V_scale), + ): + replace_parameter(self, name, tensor, prefer_copy=True) # The kernel operates on non-padded inputs. Hence, pre-compiling # triton kernel to avoid runtime compilation for unseen batch sizes @@ -1327,8 +1326,11 @@ class MLAAttention(nn.Module, AttentionLayerBase): # Convert from (L, N, P) to (N, P, L) replace_parameter(self, "W_UK_T", W_UK.permute(1, 2, 0), prefer_copy=True) if self.dcp_q_replicate: - self.W_UK_T_dcp_qrep = get_dcp_group().all_gather( - self.W_UK_T.contiguous(), dim=0 + replace_parameter( + self, + "W_UK_T_dcp_qrep", + get_dcp_group().all_gather(self.W_UK_T.contiguous(), dim=0), + prefer_copy=True, ) # If we should not load quant weights, we initialize the scales to 1.0 @@ -2384,8 +2386,12 @@ class MLACommonMetadataBuilder(AttentionMetadataBuilder[M]): self.vllm_config = vllm_config self.device = device self.use_pcp = parallel_config.prefill_context_parallel_size > 1 - self.non_causal_multi_token_decode = getattr( - kv_cache_spec, "non_causal_multi_token_decode", False + # merge() unions this flag across the KV group, so a shared target/draft + # group would mark the target builder as non-causal. + ctx = self.compilation_config.static_forward_context + self.non_causal_multi_token_decode = any( + getattr(ctx[name], "non_causal_multi_token_decode", False) + for name in layer_names ) self._validate_dspark_dcp_support(supports_dcp_with_varlen) @@ -3033,7 +3039,13 @@ class MLACommonBaseImpl(MLAAttentionImpl[A], Generic[A]): attn_metadata: MLACommonMetadata, k_scale: torch.Tensor, dcp_world_size: int, + fused_mla_kv_concat_fn: Callable[ + [torch.Tensor, torch.Tensor, bool], tuple[torch.Tensor, torch.Tensor] + ] + | None = None, ): + # fused_mla_kv_concat_fn(kv_nope, k_pe, use_fp8_prefill) -> (k, v) replaces + # the per-chunk cast + split + concat below with one fused kernel. assert attn_metadata.prefill is not None prefill_metadata = attn_metadata.prefill assert prefill_metadata.prefill_backend is not None @@ -3127,11 +3139,16 @@ class MLACommonBaseImpl(MLAAttentionImpl[A], Generic[A]): kv_nope = self.kv_b_proj(kv_c_normed)[0].view( -1, self.num_heads, self.qk_nope_head_dim + self.v_head_dim ) - if use_fp8_prefill: - kv_nope = kv_nope.to(prefill_metadata.q_data_type) - k_pe = k_pe.to(prefill_metadata.q_data_type) - k_nope, v = kv_nope.split([self.qk_nope_head_dim, self.v_head_dim], dim=-1) - k = self._concat_k_nope_k_pe(k_nope, k_pe) + if fused_mla_kv_concat_fn is not None: + k, v = fused_mla_kv_concat_fn(kv_nope, k_pe, use_fp8_prefill) + else: + if use_fp8_prefill: + kv_nope = kv_nope.to(prefill_metadata.q_data_type) + k_pe = k_pe.to(prefill_metadata.q_data_type) + k_nope, v = kv_nope.split( + [self.qk_nope_head_dim, self.v_head_dim], dim=-1 + ) + k = self._concat_k_nope_k_pe(k_nope, k_pe) attn_output, attn_softmax_lse = ( prefill_metadata.prefill_backend.run_prefill_context_chunk( diff --git a/vllm/model_executor/layers/fused_moe/config.py b/vllm/model_executor/layers/fused_moe/config.py index 158bc90769..f20af8a6f8 100644 --- a/vllm/model_executor/layers/fused_moe/config.py +++ b/vllm/model_executor/layers/fused_moe/config.py @@ -1293,6 +1293,7 @@ class FusedMoEConfig: has_bias: bool = False is_lora_enabled: bool = False has_hash_routing: bool = False + shared_expert_prefix: str | None = None # When True, the MoE skips its final cross-rank all-reduce (and the separate # shared-expert reduce), returning the partial per-rank sum. The caller is diff --git a/vllm/model_executor/layers/fused_moe/configs/E=128,N=384,device_name=Intel(R)_Arc(TM)_Pro_B70_Graphics.json b/vllm/model_executor/layers/fused_moe/configs/E=128,N=384,device_name=Intel(R)_Arc(TM)_Pro_B70_Graphics.json new file mode 100644 index 0000000000..6c1788d496 --- /dev/null +++ b/vllm/model_executor/layers/fused_moe/configs/E=128,N=384,device_name=Intel(R)_Arc(TM)_Pro_B70_Graphics.json @@ -0,0 +1,164 @@ +{ + "1": { + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 64, + "BLOCK_SIZE_K": 64, + "GROUP_SIZE_M": 16, + "num_warps": 8, + "num_stages": 3, + "grf_mode": "256" + }, + "2": { + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 128, + "BLOCK_SIZE_K": 64, + "GROUP_SIZE_M": 16, + "num_warps": 16, + "num_stages": 3, + "grf_mode": "256" + }, + "4": { + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 128, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 1, + "num_warps": 8, + "num_stages": 4, + "grf_mode": "256" + }, + "8": { + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 64, + "BLOCK_SIZE_K": 64, + "GROUP_SIZE_M": 1, + "num_warps": 8, + "num_stages": 3, + "grf_mode": "256" + }, + "16": { + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 128, + "BLOCK_SIZE_K": 64, + "GROUP_SIZE_M": 16, + "num_warps": 16, + "num_stages": 3, + "grf_mode": "256" + }, + "24": { + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 64, + "BLOCK_SIZE_K": 64, + "GROUP_SIZE_M": 16, + "num_warps": 8, + "num_stages": 3, + "grf_mode": "256" + }, + "32": { + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 128, + "BLOCK_SIZE_K": 64, + "GROUP_SIZE_M": 8, + "num_warps": 16, + "num_stages": 3, + "grf_mode": "256" + }, + "48": { + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 128, + "BLOCK_SIZE_K": 64, + "GROUP_SIZE_M": 16, + "num_warps": 16, + "num_stages": 3, + "grf_mode": "256" + }, + "64": { + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 64, + "BLOCK_SIZE_K": 64, + "GROUP_SIZE_M": 1, + "num_warps": 8, + "num_stages": 3, + "grf_mode": "256" + }, + "96": { + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 128, + "BLOCK_SIZE_K": 64, + "GROUP_SIZE_M": 16, + "num_warps": 16, + "num_stages": 3, + "grf_mode": "256" + }, + "128": { + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 128, + "BLOCK_SIZE_K": 64, + "GROUP_SIZE_M": 16, + "num_warps": 16, + "num_stages": 3, + "grf_mode": "256" + }, + "256": { + "BLOCK_SIZE_M": 32, + "BLOCK_SIZE_N": 128, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 1, + "num_warps": 8, + "num_stages": 4, + "grf_mode": "256" + }, + "512": { + "BLOCK_SIZE_M": 32, + "BLOCK_SIZE_N": 128, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 8, + "num_warps": 8, + "num_stages": 3, + "grf_mode": "256" + }, + "1024": { + "BLOCK_SIZE_M": 64, + "BLOCK_SIZE_N": 128, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 16, + "num_warps": 8, + "num_stages": 3, + "grf_mode": "256" + }, + "1536": { + "BLOCK_SIZE_M": 32, + "BLOCK_SIZE_N": 128, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 16, + "num_warps": 4, + "num_stages": 3, + "grf_mode": "256" + }, + "2048": { + "BLOCK_SIZE_M": 64, + "BLOCK_SIZE_N": 128, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 16, + "num_warps": 8, + "num_stages": 3, + "grf_mode": "256" + }, + "3072": { + "BLOCK_SIZE_M": 64, + "BLOCK_SIZE_N": 128, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 16, + "num_warps": 8, + "num_stages": 3, + "grf_mode": "256" + }, + "4096": { + "BLOCK_SIZE_M": 64, + "BLOCK_SIZE_N": 128, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 16, + "num_warps": 8, + "num_stages": 3, + "grf_mode": "256" + } +} diff --git a/vllm/model_executor/layers/fused_moe/configs/E=64,N=1024,device_name=Intel(R)_Arc(TM)_Pro_B70_Graphics.json b/vllm/model_executor/layers/fused_moe/configs/E=64,N=1024,device_name=Intel(R)_Arc(TM)_Pro_B70_Graphics.json new file mode 100644 index 0000000000..0cdfc22664 --- /dev/null +++ b/vllm/model_executor/layers/fused_moe/configs/E=64,N=1024,device_name=Intel(R)_Arc(TM)_Pro_B70_Graphics.json @@ -0,0 +1,164 @@ +{ + "1": { + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 128, + "BLOCK_SIZE_K": 64, + "GROUP_SIZE_M": 8, + "num_warps": 16, + "num_stages": 3, + "grf_mode": "256" + }, + "2": { + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 128, + "BLOCK_SIZE_K": 64, + "GROUP_SIZE_M": 16, + "num_warps": 16, + "num_stages": 3, + "grf_mode": "256" + }, + "4": { + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 128, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 16, + "num_warps": 8, + "num_stages": 3, + "grf_mode": "256" + }, + "8": { + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 128, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 16, + "num_warps": 16, + "num_stages": 4, + "grf_mode": "256" + }, + "16": { + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 128, + "BLOCK_SIZE_K": 64, + "GROUP_SIZE_M": 8, + "num_warps": 16, + "num_stages": 3, + "grf_mode": "256" + }, + "24": { + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 64, + "BLOCK_SIZE_K": 64, + "GROUP_SIZE_M": 1, + "num_warps": 8, + "num_stages": 3, + "grf_mode": "256" + }, + "32": { + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 128, + "BLOCK_SIZE_K": 64, + "GROUP_SIZE_M": 8, + "num_warps": 16, + "num_stages": 3, + "grf_mode": "256" + }, + "48": { + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 128, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 8, + "num_warps": 16, + "num_stages": 4, + "grf_mode": "256" + }, + "64": { + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 128, + "BLOCK_SIZE_K": 64, + "GROUP_SIZE_M": 8, + "num_warps": 16, + "num_stages": 3, + "grf_mode": "256" + }, + "96": { + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 128, + "BLOCK_SIZE_K": 64, + "GROUP_SIZE_M": 1, + "num_warps": 8, + "num_stages": 3, + "grf_mode": "256" + }, + "128": { + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 128, + "BLOCK_SIZE_K": 64, + "GROUP_SIZE_M": 1, + "num_warps": 8, + "num_stages": 3, + "grf_mode": "256" + }, + "256": { + "BLOCK_SIZE_M": 32, + "BLOCK_SIZE_N": 128, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 1, + "num_warps": 8, + "num_stages": 4, + "grf_mode": "256" + }, + "512": { + "BLOCK_SIZE_M": 64, + "BLOCK_SIZE_N": 128, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 16, + "num_warps": 8, + "num_stages": 3, + "grf_mode": "256" + }, + "1024": { + "BLOCK_SIZE_M": 64, + "BLOCK_SIZE_N": 128, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 8, + "num_warps": 8, + "num_stages": 3, + "grf_mode": "256" + }, + "1536": { + "BLOCK_SIZE_M": 64, + "BLOCK_SIZE_N": 128, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 1, + "num_warps": 8, + "num_stages": 3, + "grf_mode": "256" + }, + "2048": { + "BLOCK_SIZE_M": 64, + "BLOCK_SIZE_N": 128, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 1, + "num_warps": 8, + "num_stages": 3, + "grf_mode": "256" + }, + "3072": { + "BLOCK_SIZE_M": 128, + "BLOCK_SIZE_N": 128, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 16, + "num_warps": 16, + "num_stages": 3, + "grf_mode": "256" + }, + "4096": { + "BLOCK_SIZE_M": 128, + "BLOCK_SIZE_N": 128, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 8, + "num_warps": 16, + "num_stages": 3, + "grf_mode": "256" + } +} diff --git a/vllm/model_executor/layers/fused_moe/configs/E=64,N=512,device_name=Intel(R)_Arc(TM)_Pro_B70_Graphics.json b/vllm/model_executor/layers/fused_moe/configs/E=64,N=512,device_name=Intel(R)_Arc(TM)_Pro_B70_Graphics.json new file mode 100644 index 0000000000..46bee68b89 --- /dev/null +++ b/vllm/model_executor/layers/fused_moe/configs/E=64,N=512,device_name=Intel(R)_Arc(TM)_Pro_B70_Graphics.json @@ -0,0 +1,56 @@ +{ + "1": { + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 128, + "BLOCK_SIZE_K": 64, + "GROUP_SIZE_M": 1, + "num_warps": 16, + "num_stages": 3, + "grf_mode": "256" + }, + "16": { + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 64, + "BLOCK_SIZE_K": 64, + "GROUP_SIZE_M": 1, + "num_warps": 8, + "num_stages": 3, + "grf_mode": "256" + }, + "128": { + "BLOCK_SIZE_M": 32, + "BLOCK_SIZE_N": 128, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 1, + "num_warps": 8, + "num_stages": 3, + "grf_mode": "256" + }, + "512": { + "BLOCK_SIZE_M": 64, + "BLOCK_SIZE_N": 128, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 16, + "num_warps": 8, + "num_stages": 3, + "grf_mode": "256" + }, + "2048": { + "BLOCK_SIZE_M": 64, + "BLOCK_SIZE_N": 128, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 1, + "num_warps": 8, + "num_stages": 3, + "grf_mode": "256" + }, + "4096": { + "BLOCK_SIZE_M": 128, + "BLOCK_SIZE_N": 128, + "BLOCK_SIZE_K": 32, + "GROUP_SIZE_M": 8, + "num_warps": 16, + "num_stages": 3, + "grf_mode": "256" + } +} diff --git a/vllm/model_executor/layers/fused_moe/deep_gemm_mega_moe.py b/vllm/model_executor/layers/fused_moe/deep_gemm_mega_moe.py new file mode 100644 index 0000000000..633ced1082 --- /dev/null +++ b/vllm/model_executor/layers/fused_moe/deep_gemm_mega_moe.py @@ -0,0 +1,466 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""DeepGEMM MegaMoE backends: fused EP dispatch, expert GEMMs and combine.""" + +from inspect import signature +from typing import Any + +import torch +import torch.nn as nn + +from vllm.logger import init_logger +from vllm.model_executor.layers.quantization.utils.quant_utils import ( + QuantKey, + kMxfp8Dynamic, +) +from vllm.platforms import current_platform + +logger = init_logger(__name__) + + +def ue8m0_uint8_to_float(sf: torch.Tensor) -> torch.Tensor: + """Reinterpret a uint8 UE8M0 exponent tensor as float32 scale values.""" + return (sf.to(torch.int32) << 23).view(torch.float32) + + +class DeepGemmMegaMoEBackend: + """Abstract MegaMoE backend that hides deep_gemm-specific details. + + A backend advertises its expected hidden-state quantization layout via + ``hidden_quant`` and its expected MMA type via ``mma_type``. It is + responsible for the routed-expert weight transform run at load time, the + optional shared-expert weight transform, and the actual MegaMoE kernel + invocation at inference time. Runtime capability checks live in + :func:`get_deep_gemm_mega_moe_backend`: obtaining a backend already implies + that the current device and shapes are supported. + """ + + # Human-readable MMA type tag used by deep_gemm helpers. SM100 default. + mma_type: str = "fp8xfp4" + hidden_quant: QuantKey = kMxfp8Dynamic + + def transform_weights( + self, + *, + w13_weight: torch.Tensor, + w13_weight_scale: torch.Tensor, + w2_weight: torch.Tensor, + w2_weight_scale: torch.Tensor, + num_local_experts: int, + hidden_size: int, + intermediate_size: int, + activation: str | None = None, + ) -> tuple[ + tuple[torch.Tensor, torch.Tensor], + tuple[torch.Tensor, torch.Tensor], + ]: + raise NotImplementedError + + def supports_shared_experts(self, deep_gemm) -> bool: + return False + + def transform_shared_expert_weights( + self, + *, + shared_experts: nn.Module, + num_shared_experts: int, + hidden_size: int, + intermediate_size: int, + prefix: str, + ) -> ( + tuple[ + tuple[torch.Tensor, torch.Tensor], + tuple[torch.Tensor, torch.Tensor], + ] + | None + ): + """Return ``(l1, l2)`` transformed shared-expert weights, or ``None``. + + ``shared_experts`` must expose ``gate_up_proj`` and ``down_proj`` + linear layers. A ``None`` return means the backend cannot fuse shared + experts for this module; the caller should fall back to a separate + shared MLP. + """ + return None + + def run_mega_moe( + self, + *, + y: torch.Tensor, + l1_weights: tuple[torch.Tensor, torch.Tensor], + l2_weights: tuple[torch.Tensor, torch.Tensor], + symm_buffer, + activation_clamp: float | None, + fast_math: bool, + activation: str | None = None, + activation_alpha: float | None = None, + activation_beta: float | None = None, + shared_l1_weights: tuple[torch.Tensor, torch.Tensor] | None = None, + shared_l2_weights: tuple[torch.Tensor, torch.Tensor] | None = None, + ) -> None: + raise NotImplementedError + + def supports_bf16_mega_gate(self) -> bool: + """Whether this backend can run the DeepGEMM bf16 MegaMoE gate kernel. + + Defaults to ``False`` (the symbol is not guaranteed to exist in every + deep_gemm build). Backends that target a deep_gemm build exposing the + ``bf16_mega_gate`` symbol override this and :meth:`bf16_mega_gate`. + """ + return False + + def bf16_mega_gate( + self, + *, + x: torch.Tensor, + weight: torch.Tensor, + num_topk: int, + scoring_func: str, + routed_scaling_factor: float, + ep_rank: int, + bias: torch.Tensor | None = None, + image_bias: torch.Tensor | None = None, + image_token_mask: torch.Tensor | None = None, + fix_routing_mask: torch.Tensor | None = None, + unmapped_topk_idx: torch.Tensor | None = None, + ) -> tuple[torch.Tensor, torch.Tensor]: + raise NotImplementedError + + +class DeepGemmSm100MegaMoEBackend(DeepGemmMegaMoEBackend): + """Default MegaMoE backend targeting SM100 via deep_gemm fp8_fp4_mega_moe.""" + + mma_type = "fp8xfp4" + hidden_quant = kMxfp8Dynamic + + def transform_weights( + self, + *, + w13_weight: torch.Tensor, + w13_weight_scale: torch.Tensor, + w2_weight: torch.Tensor, + w2_weight_scale: torch.Tensor, + num_local_experts: int, + hidden_size: int, + intermediate_size: int, + activation: str | None = None, + ) -> tuple[ + tuple[torch.Tensor, torch.Tensor], + tuple[torch.Tensor, torch.Tensor], + ]: + from vllm.utils.deep_gemm import _import_deep_gemm + + deep_gemm = _import_deep_gemm() + + w13_scale = deep_gemm.transform_sf_into_required_layout( + ue8m0_uint8_to_float(w13_weight_scale).contiguous(), + 2 * intermediate_size, + hidden_size, + (1, 32), + num_local_experts, + ) + w2_scale = deep_gemm.transform_sf_into_required_layout( + ue8m0_uint8_to_float(w2_weight_scale).contiguous(), + hidden_size, + intermediate_size, + (1, 32), + num_local_experts, + ) + kwargs = {} if activation is None else {"activation": activation} + return deep_gemm.transform_weights_for_mega_moe( + (w13_weight.view(torch.int8).contiguous(), w13_scale), + (w2_weight.view(torch.int8).contiguous(), w2_scale), + **kwargs, + ) + + def supports_shared_experts(self, deep_gemm) -> bool: + """Check the Python API before touching a symmetric-memory group. + + This also gives users of an older precompiled vLLM wheel a safe serial + fallback instead of failing halfway through multi-rank buffer setup. + """ + try: + buffer_params = signature(deep_gemm.get_symm_buffer_for_mega_moe).parameters + kernel_params = signature(deep_gemm.fp8_fp4_mega_moe).parameters + except (TypeError, ValueError): + return False + return ( + hasattr(deep_gemm, "get_block_m_for_mega_moe") + and hasattr(deep_gemm, "transform_weights_for_mega_moe") + and "num_shared_experts" in buffer_params + and "shared_l1_weights" in kernel_params + and "shared_l2_weights" in kernel_params + ) + + def transform_shared_expert_weights( + self, + *, + shared_experts: nn.Module, + num_shared_experts: int, + hidden_size: int, + intermediate_size: int, + prefix: str, + ) -> ( + tuple[ + tuple[torch.Tensor, torch.Tensor], + tuple[torch.Tensor, torch.Tensor], + ] + | None + ): + from vllm.utils.deep_gemm import _import_deep_gemm + + deep_gemm = _import_deep_gemm() + + gate_up = shared_experts.gate_up_proj + down = shared_experts.down_proj + gate_up_weight = gate_up.weight.data + gate_up_scale = ( + gate_up.weight_scale + if hasattr(gate_up, "weight_scale") + else gate_up.weight_scale_inv + ).data + down_weight = down.weight.data + down_scale = ( + down.weight_scale + if hasattr(down, "weight_scale") + else down.weight_scale_inv + ).data + + # MegaMoE's shared FP8 MMA consumes a 1x32 scale for every weight row, + # while the checkpoint uses coarser block-FP8 scales (usually + # 128x128). Build a dedicated, numerically equivalent scale view. + ue8m0_scale_dtypes = (torch.float8_e8m0fnu, torch.uint8, torch.int32) + if ( + gate_up_scale.dtype in ue8m0_scale_dtypes + and down_scale.dtype in ue8m0_scale_dtypes + ): + gate_up_scale = self._prepare_shared_expert_scale( + deep_gemm, + gate_up, + gate_up_scale, + gate_up_weight.shape[0], + gate_up_weight.shape[1], + prefix, + ) + down_scale = self._prepare_shared_expert_scale( + deep_gemm, + down, + down_scale, + down_weight.shape[0], + down_weight.shape[1], + prefix, + ) + + if gate_up_scale is None or down_scale is None: + return None + + shared_intermediate_size = intermediate_size * num_shared_experts + expected_gate_up_shape = ( + 2 * shared_intermediate_size, + hidden_size, + ) + expected_down_shape = (hidden_size, shared_intermediate_size) + if ( + gate_up_weight.dtype != torch.float8_e4m3fn + or down_weight.dtype != torch.float8_e4m3fn + or gate_up_scale.dtype != torch.int32 + or down_scale.dtype != torch.int32 + or tuple(gate_up_weight.shape) != expected_gate_up_shape + or tuple(down_weight.shape) != expected_down_shape + ): + logger.warning( + "Disabling native MegaMoE shared-expert fusion for %s: expected " + "replicated block-FP8 weights with gate_up=%s, down=%s, and " + "DeepGEMM int32 scales; got gate_up=%s/%s/%s and down=%s/%s/%s.", + prefix, + expected_gate_up_shape, + expected_down_shape, + tuple(gate_up_weight.shape), + gate_up_weight.dtype, + gate_up_scale.dtype, + tuple(down_weight.shape), + down_weight.dtype, + down_scale.dtype, + ) + return None + + transformed_l1, transformed_l2 = deep_gemm.transform_weights_for_mega_moe( + (gate_up_weight, gate_up_scale), + (down_weight, down_scale), + ) + # L1 interleaving allocates a full copy. Re-home the loader Parameter on + # that storage so the original 2*intermediate*hidden FP8 tensor can be + # released instead of adding roughly 0.7 GiB per rank on DSV4-Flash. + # The generic linear post-load hook may still repack the serial scales, + # but this shared MLP is never called after native fusion is enabled. + gate_up.weight.data = transformed_l1[0] + transformed_l1 = (gate_up.weight.data, transformed_l1[1]) + return transformed_l1, transformed_l2 + + @staticmethod + def _prepare_shared_expert_scale( + deep_gemm, + linear: nn.Module, + scale: torch.Tensor, + mn: int, + k: int, + prefix: str, + ) -> torch.Tensor | None: + block_size = getattr(linear, "weight_block_size", None) + if block_size is None or len(block_size) != 2: + logger.warning( + "Disabling native MegaMoE shared-expert fusion for %s: " + "shared FP8 weight block size is unavailable.", + prefix, + ) + return None + + block_m, block_k = block_size + num_k_blocks = (k + block_k - 1) // block_k + # The linear hook's DeepGEMM layout packs four k-block bytes per int32. + packed = scale.dtype == torch.int32 + expected_shape = ( + (mn, (num_k_blocks + 3) // 4) + if packed + else ((mn + block_m - 1) // block_m, num_k_blocks) + ) + if block_k % 32 != 0 or tuple(scale.shape) != expected_shape: + logger.warning( + "Disabling native MegaMoE shared-expert fusion for %s: " + "cannot convert shared scale shape %s with block size %s " + "to MegaMoE's 1x32 layout for weight (%d, %d).", + prefix, + tuple(scale.shape), + tuple(block_size), + mn, + k, + ) + return None + + if packed: + row_scale = scale.flatten().view(torch.uint8).view(mn, -1)[:, :num_k_blocks] + else: + row_scale = scale.view(torch.uint8).repeat_interleave(block_m, dim=0)[:mn] + scale_1x32 = ( + ue8m0_uint8_to_float(row_scale) + .repeat_interleave(block_k // 32, dim=1)[:, : k // 32] + .contiguous() + ) + # The grouped API is used with a singleton dimension to request the + # MN-major, TMA-aligned packed-UE8M0 strides, then squeezed back to the + # 2D layout required for a shared expert. + return deep_gemm.transform_sf_into_required_layout( + scale_1x32.unsqueeze(0), + mn, + k, + (1, 32), + 1, + ).squeeze(0) + + def run_mega_moe( + self, + *, + y: torch.Tensor, + l1_weights: tuple[torch.Tensor, torch.Tensor], + l2_weights: tuple[torch.Tensor, torch.Tensor], + symm_buffer, + activation_clamp: float | None, + fast_math: bool, + activation: str | None = None, + activation_alpha: float | None = None, + activation_beta: float | None = None, + shared_l1_weights: tuple[torch.Tensor, torch.Tensor] | None = None, + shared_l2_weights: tuple[torch.Tensor, torch.Tensor] | None = None, + ) -> None: + from vllm.utils.deep_gemm import _import_deep_gemm + + deep_gemm = _import_deep_gemm() + kwargs: dict[str, Any] = { + name: value + for name, value in ( + ("activation", activation), + ("activation_alpha", activation_alpha), + ("activation_beta", activation_beta), + ) + if value is not None + } + if shared_l1_weights is not None and shared_l2_weights is not None: + kwargs["shared_l1_weights"] = shared_l1_weights + kwargs["shared_l2_weights"] = shared_l2_weights + deep_gemm.fp8_fp4_mega_moe( + y, + l1_weights, + l2_weights, + symm_buffer, + activation_clamp=activation_clamp, + fast_math=fast_math, + **kwargs, + ) + + def supports_bf16_mega_gate(self) -> bool: + """SM100 deep_gemm build is expected to provide the bf16 gate kernel.""" + return True + + def bf16_mega_gate( + self, + *, + x: torch.Tensor, + weight: torch.Tensor, + num_topk: int, + scoring_func: str, + routed_scaling_factor: float, + ep_rank: int, + bias: torch.Tensor | None = None, + image_bias: torch.Tensor | None = None, + image_token_mask: torch.Tensor | None = None, + fix_routing_mask: torch.Tensor | None = None, + unmapped_topk_idx: torch.Tensor | None = None, + ) -> tuple[torch.Tensor, torch.Tensor]: + from vllm.utils.deep_gemm import bf16_mega_gate + + return bf16_mega_gate( + x, + weight, + num_topk, + scoring_func=scoring_func, + routed_scaling_factor=routed_scaling_factor, + ep_rank=ep_rank, + bias=bias, + image_bias=image_bias, + image_token_mask=image_token_mask, + fix_routing_mask=fix_routing_mask, + unmapped_topk_idx=unmapped_topk_idx, + ) + + +def get_deep_gemm_mega_moe_backend( + device: torch.device, + hidden_size: int, + intermediate_size: int, +) -> DeepGemmMegaMoEBackend: + """Return the DeepGEMM MegaMoE backend for ``device``. + + A successful return implies the backend can run on this device with the + given ``hidden_size`` / ``intermediate_size``. All runtime capability + checks (arch + shape divisibility) live here; there is no separate check + on the returned backend. + + Currently only the SM100 deep_gemm backend is registered here. SM90 and + other backends are expected to be provided by new backends or plugins + that override this factory. + + Raises: + NotImplementedError: If the device architecture is unsupported. + ValueError: If the shapes are unsupported. + + """ + if not current_platform.is_device_capability_family( + 100, device_id=device.index or 0 + ): + raise NotImplementedError("DeepGEMM MegaMoE requires SM100 GPUs.") + if hidden_size % 128 != 0 or intermediate_size % 128 != 0: + raise ValueError( + "DeepGEMM MegaMoE requires hidden and intermediate sizes " + "to be multiples of 128." + ) + return DeepGemmSm100MegaMoEBackend() diff --git a/vllm/model_executor/layers/fused_moe/deep_gemm_utils.py b/vllm/model_executor/layers/fused_moe/deep_gemm_utils.py index 33a2296955..1dee9bad54 100644 --- a/vllm/model_executor/layers/fused_moe/deep_gemm_utils.py +++ b/vllm/model_executor/layers/fused_moe/deep_gemm_utils.py @@ -119,6 +119,7 @@ def _fwd_kernel_ep_scatter_1( BLOCK_E: tl.constexpr, BLOCK_EXPERT_NUM: tl.constexpr, ALIGN_M: tl.constexpr, + USE_PSUM_LAYOUT: tl.constexpr, ): cur_expert = tl.program_id(0) @@ -140,20 +141,22 @@ def _fwd_kernel_ep_scatter_1( tl.store(expert_start_loc + cur_expert, cur_expert_start) cur_expert_token_num = tl.load(num_recv_tokens_per_expert + cur_expert) - m_indices_start_ptr = m_indices + cur_expert_start - off_expert = tl.arange(0, BLOCK_E) + if USE_PSUM_LAYOUT: + # Logical ends include preceding experts' alignment gaps. + tl.store(m_indices + cur_expert, cur_expert_start + cur_expert_token_num) + else: + m_indices_start_ptr = m_indices + cur_expert_start + off_expert = tl.arange(0, BLOCK_E) - # any rows in the per-expert aligned region that do not correspond to - # real tokens are left untouched here and should remain initialized to - # -1 so DeepGEMM can skip them - for start_m in tl.range(0, cur_expert_token_num, BLOCK_E): - offs = start_m + off_expert - mask = offs < cur_expert_token_num - tl.store( - m_indices_start_ptr + offs, - cur_expert, - mask=mask, - ) + # Unwritten padding retains the -1 initialization. + for start_m in tl.range(0, cur_expert_token_num, BLOCK_E): + offs = start_m + off_expert + mask = offs < cur_expert_token_num + tl.store( + m_indices_start_ptr + offs, + cur_expert, + mask=mask, + ) @triton.jit @@ -285,6 +288,7 @@ def ep_scatter( align_m: int = 128, block_size: int = 128, pack_ue8m0: bool = False, + use_psum_layout: bool = False, ): # BLOCK_E is the m_indices fill-loop tile (masked), independent of align_m. BLOCK_E = 128 @@ -295,7 +299,10 @@ def ep_scatter( # grid = (triton.cdiv(hidden_size, BLOCK_D), num_experts) grid = num_experts - assert m_indices.shape[0] % align_m == 0 + if use_psum_layout: + assert m_indices.shape == (num_experts,) + else: + assert m_indices.shape[0] % align_m == 0 assert expert_start_loc.shape[0] == num_experts # pack_ue8m0: scatter packs 4 UE8M0 bytes per int32; else copies scales as-is. @@ -311,6 +318,7 @@ def ep_scatter( BLOCK_E=BLOCK_E, BLOCK_EXPERT_NUM=triton.next_power_of_2(num_experts), ALIGN_M=align_m, + USE_PSUM_LAYOUT=use_psum_layout, ) grid = min(recv_topk.shape[0], 1024 * 8) @@ -407,7 +415,7 @@ def _fwd_kernel_ep_gather( tl.store( output_tensor - + cur_token * output_tensor_stride0 + + tl.cast(cur_token, tl.int64) * output_tensor_stride0 + cur_block * BLOCK_D + off_d, accumulator.to(output_tensor.dtype.element_ty), @@ -465,6 +473,7 @@ def deepgemm_moe_permute( expert_tokens_meta: mk.ExpertTokensMetadata | None, aq_out: torch.Tensor | None = None, block_size: int | None = None, + use_psum_layout: bool = False, ): assert aq.ndim == 2 assert topk_ids.dtype.is_signed, "The kernel uses -1 to represent invalid topk_ids" @@ -509,17 +518,15 @@ def deepgemm_moe_permute( else: aq_scale_out = torch.zeros((M_sum, sf_k), device=device, dtype=torch.float32) - # DeepGEMM uses negative values in m_indices (here expert_ids) to mark - # completely invalid / padded blocks that should be skipped. We always - # initialize expert_ids to -1 so any row that is not explicitly written - # by the scatter kernel will be treated as invalid and skipped by - # DeepGEMM's scheduler. - expert_ids = torch.full( - (M_sum,), - fill_value=-1, - device=device, - dtype=torch.int32, - ) + if use_psum_layout: + grouped_layout = torch.empty( + (local_num_experts,), device=device, dtype=torch.int32 + ) + else: + # Negative expert IDs mark padding for the contiguous scheduler. + grouped_layout = torch.full( + (M_sum,), fill_value=-1, device=device, dtype=torch.int32 + ) inv_perm = torch.empty(topk_ids.shape, device=device, dtype=torch.int32) expert_num_tokens = None @@ -541,14 +548,15 @@ def deepgemm_moe_permute( expert_map=expert_map, output_tensor=aq_out, output_tensor_scale=aq_scale_out, - m_indices=expert_ids, + m_indices=grouped_layout, output_index=inv_perm, align_m=align_used, block_size=block_k, pack_ue8m0=pack_ue8m0, + use_psum_layout=use_psum_layout, ) - return aq_out, aq_scale_out, expert_ids, inv_perm, align_used + return aq_out, aq_scale_out, grouped_layout, inv_perm, align_used def deepgemm_unpermute_and_reduce( diff --git a/vllm/model_executor/layers/fused_moe/experts/cutlass_moe.py b/vllm/model_executor/layers/fused_moe/experts/cutlass_moe.py index a03651c0a9..6f9a62f282 100644 --- a/vllm/model_executor/layers/fused_moe/experts/cutlass_moe.py +++ b/vllm/model_executor/layers/fused_moe/experts/cutlass_moe.py @@ -20,6 +20,7 @@ from vllm.model_executor.layers.fused_moe.config import ( ) from vllm.model_executor.layers.fused_moe.moe_permute_unpermute import ( MoEPermuteScratch, + get_moe_permute_scratch, moe_permute, moe_permute_unpermute_supported, moe_unpermute, @@ -304,7 +305,6 @@ class CutlassExpertsFp8Base(mk.FusedMoEExpertsModular): self.ab_strides2 = ab_strides2 self.c_strides1 = c_strides1 self.c_strides2 = ab_strides1_c_strides2 - self._permute_scratch: MoEPermuteScratch | None = None @staticmethod def _supports_current_device() -> bool: @@ -335,15 +335,13 @@ class CutlassExpertsFp8Base(mk.FusedMoEExpertsModular): return TopKWeightAndReduceDelegate() def _get_permute_scratch(self) -> MoEPermuteScratch | None: - if self._permute_scratch is None and moe_permute_unpermute_supported(): - self._permute_scratch = MoEPermuteScratch( - max_num_tokens=self.moe_config.max_num_tokens, - topk=self.moe_config.experts_per_token, - num_experts=self.moe_config.num_experts, - num_local_experts=self.moe_config.num_local_experts, - device=torch.device(self.moe_config.device), - ) - return self._permute_scratch + if not moe_permute_unpermute_supported(): + return None + return get_moe_permute_scratch( + num_experts=self.moe_config.num_experts, + num_local_experts=self.moe_config.num_local_experts, + device=torch.device(self.moe_config.device), + ) def apply( self, @@ -1296,7 +1294,6 @@ class CutlassExpertsW4A8Fp8(mk.FusedMoEExpertsModular): self.s_strides2[:, 0] = k self.group_size = group_size - self._permute_scratch: MoEPermuteScratch | None = None @staticmethod def activation_format() -> mk.FusedMoEActivationFormat: @@ -1357,15 +1354,13 @@ class CutlassExpertsW4A8Fp8(mk.FusedMoEExpertsModular): return self.out_dtype if self.out_dtype is not None else act_dtype def _get_permute_scratch(self) -> MoEPermuteScratch | None: - if self._permute_scratch is None and moe_permute_unpermute_supported(): - self._permute_scratch = MoEPermuteScratch( - max_num_tokens=self.moe_config.max_num_tokens, - topk=self.moe_config.experts_per_token, - num_experts=self.moe_config.num_experts, - num_local_experts=self.moe_config.num_local_experts, - device=torch.device(self.moe_config.device), - ) - return self._permute_scratch + if not moe_permute_unpermute_supported(): + return None + return get_moe_permute_scratch( + num_experts=self.moe_config.num_experts, + num_local_experts=self.moe_config.num_local_experts, + device=torch.device(self.moe_config.device), + ) def workspace_shapes( self, diff --git a/vllm/model_executor/layers/fused_moe/experts/deep_gemm_moe.py b/vllm/model_executor/layers/fused_moe/experts/deep_gemm_moe.py index 699b029bad..0eef7a8335 100644 --- a/vllm/model_executor/layers/fused_moe/experts/deep_gemm_moe.py +++ b/vllm/model_executor/layers/fused_moe/experts/deep_gemm_moe.py @@ -244,7 +244,12 @@ class DeepGemmExperts(mk.FusedMoEExpertsModular): return (workspace1, workspace2, output) def _act_mul_quant( - self, input: torch.Tensor, output: torch.Tensor, activation: MoEActivation + self, + input: torch.Tensor, + output: torch.Tensor, + activation: MoEActivation, + expert_ends: torch.Tensor | None = None, + expert_alignment: int = 0, ) -> tuple[torch.Tensor, torch.Tensor]: assert self.block_shape is not None block_k = self.block_shape[1] @@ -271,6 +276,8 @@ class DeepGemmExperts(mk.FusedMoEExpertsModular): clamp_limit=self.gemm1_clamp_limit, alpha=self.gemm1_alpha, beta=self.gemm1_beta, + expert_ends=expert_ends, + expert_alignment=expert_alignment, ) act_out = torch.empty( (M_sum, activation_out_dim), dtype=input.dtype, device=input.device @@ -294,6 +301,8 @@ class DeepGemmExperts(mk.FusedMoEExpertsModular): group_size=block_k, alpha=self.gemm1_alpha, beta=self.gemm1_beta, + expert_ends=expert_ends, + expert_alignment=expert_alignment, ) # 3. fallback path for non-SiLU activations in non‑UE8M0 cases. @@ -349,7 +358,20 @@ class DeepGemmExperts(mk.FusedMoEExpertsModular): a1q_perm = _resize_cache( workspace13.view(dtype=torch.float8_e4m3fn), (M_sum, K) ) - a1q, a1q_scale, expert_ids, inv_perm, align_used = deepgemm_moe_permute( + # Both GEMMs and quantization must agree on the live expert ranges. + use_psum_layout = ( + not self.mxfp8 + and ( + current_platform.is_device_capability_family(90) + or ( + current_platform.is_device_capability_family(100) + and DeepGemmQuantScaleFMT.from_oracle() + == DeepGemmQuantScaleFMT.UE8M0 + ) + ) + and activation in (MoEActivation.SILU, MoEActivation.SWIGLUOAI_UNINTERLEAVE) + ) + a1q, a1q_scale, grouped_layout, inv_perm, align_used = deepgemm_moe_permute( aq=a1q, aq_scale=a1q_scale, topk_ids=topk_ids, @@ -360,17 +382,21 @@ class DeepGemmExperts(mk.FusedMoEExpertsModular): # MXFP8 uses a 32-element activation-scale group (block_shape[1]); # FP8-block keeps the default (128) alignment. block_size=self.block_shape[1] if self.mxfp8 else None, + use_psum_layout=use_psum_layout, ) assert a1q.size(0) == M_sum # MXFP8 (1x32) drives the fp8_fp4-aliased grouped GEMM with recipe # (1, 32); the FP8 block path keeps the default (128) recipe. - gemm_kwargs = ( + gemm_kwargs: dict = ( {"recipe_a": (1, self.block_shape[1]), "recipe_b": (1, self.block_shape[1])} if self.mxfp8 else {} ) + if use_psum_layout: + gemm_kwargs["use_psum_layout"] = True + # Cap DG's BLOCK_M heuristic at the workspace's per-expert alignment; # otherwise the scheduler can pick the wrong expert id from m_indices # under cudagraph replay. @@ -380,7 +406,7 @@ class DeepGemmExperts(mk.FusedMoEExpertsModular): (a1q, a1q_scale), (w1, self.w1_scale), mm1_out, - expert_ids, + grouped_layout, **gemm_kwargs, ) @@ -389,7 +415,11 @@ class DeepGemmExperts(mk.FusedMoEExpertsModular): workspace13.view(dtype=torch.float8_e4m3fn), (M_sum, activation_out_dim) ) a2q, a2q_scale = self._act_mul_quant( - input=mm1_out.view(-1, N), output=quant_out, activation=activation + input=mm1_out.view(-1, N), + output=quant_out, + activation=activation, + expert_ends=grouped_layout if use_psum_layout else None, + expert_alignment=align_used if use_psum_layout else 0, ) mm2_out = _resize_cache(workspace2, (M_sum, K)) @@ -397,7 +427,7 @@ class DeepGemmExperts(mk.FusedMoEExpertsModular): (a2q, a2q_scale), (w2, self.w2_scale), mm2_out, - expert_ids, + grouped_layout, **gemm_kwargs, ) diff --git a/vllm/model_executor/layers/fused_moe/experts/fused_humming_moe.py b/vllm/model_executor/layers/fused_moe/experts/fused_humming_moe.py index a6d44c3141..66e03229a9 100644 --- a/vllm/model_executor/layers/fused_moe/experts/fused_humming_moe.py +++ b/vllm/model_executor/layers/fused_moe/experts/fused_humming_moe.py @@ -258,25 +258,14 @@ class HummingExpertsBase(mk.FusedMoEExpertsModular): **kwargs, ) - def _get_permute_scratch( - self, topk: int, indices_only: bool = False - ) -> MoEPermuteScratch | None: + def _get_permute_scratch(self) -> MoEPermuteScratch | None: if not moe_permute_unpermute_supported(): return None - max_expanded_rows = ( - self.moe_config.max_num_tokens - * self.moe_config.dp_size - * self.moe_config.experts_per_token - ) return get_moe_permute_scratch( - max_num_tokens=math.ceil(max_expanded_rows / topk), - topk=topk, num_experts=self.moe_config.num_experts, num_local_experts=self.moe_config.num_local_experts, device=torch.device(self.moe_config.device), - hidden_size=None if indices_only else self.moe_config.hidden_dim, - hidden_dtype=None if indices_only else self.moe_config.in_dtype, ) def get_global_valid_shape_m(self, topk_ids: torch.Tensor): @@ -911,14 +900,18 @@ class HummingGroupedExperts(HummingExpertsBase): ) buffers["output"] = output + scratch = self._get_permute_scratch() if a1q_scale is None: - scratch = self._get_permute_scratch(topk_ids.size(1), indices_only=True) assert scratch is not None scatter_metadata = moe_prepare_scatter(topk_ids, expert_map, scratch) expert_offsets, scatter_idx = scatter_metadata inv_perm = scatter_idx.flatten() num_valid_tokens = expert_offsets[-1:] if expert_map is not None else None else: + # Prequantized inputs skip w13 quantization, so its buffer is free + # to hold the permuted hidden states. + permuted_hidden_states = buffers["quanted_gate_up_input"] + assert permuted_hidden_states.dtype == hidden_states.dtype hidden_states, a1q_scale, expert_offsets, inv_perm, _ = moe_permute( hidden_states=hidden_states, a1q_scale=a1q_scale, @@ -926,7 +919,8 @@ class HummingGroupedExperts(HummingExpertsBase): n_expert=global_num_experts, n_local_expert=self.num_experts, expert_map=expert_map, - scratch=self._get_permute_scratch(topk_ids.size(1)), + permuted_hidden_states=permuted_hidden_states, + scratch=scratch, ) scatter_idx = None num_valid_tokens = None diff --git a/vllm/model_executor/layers/fused_moe/experts/gpt_oss_triton_kernels_moe.py b/vllm/model_executor/layers/fused_moe/experts/gpt_oss_triton_kernels_moe.py index d23edaa01a..e3b6995d69 100644 --- a/vllm/model_executor/layers/fused_moe/experts/gpt_oss_triton_kernels_moe.py +++ b/vllm/model_executor/layers/fused_moe/experts/gpt_oss_triton_kernels_moe.py @@ -1,6 +1,8 @@ # SPDX-License-Identifier: Apache-2.0 # SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import functools + import torch import vllm.model_executor.layers.fused_moe.modular_kernel as mk @@ -41,11 +43,11 @@ def _triton_kernel_moe_supports_current_device() -> bool: p = current_platform if p.is_cuda(): cap = p.get_device_capability() - # Keep the original `(9, 0) <= cap < (11, 0)` window on - # CUDA (covers Hopper SM90 and Blackwell SM100, excludes - # SM120) — this PR is ROCm-scoped and the broader CUDA - # range was not validated. - return cap is not None and (9, 0) <= (cap.major, cap.minor) < (11, 0) + # Hopper SM90, datacenter Blackwell SM100/103 and consumer Blackwell + # SM120/121. SM12x needs the opt-flag constraints applied by + # `_swizzle_mxfp4` and `_constrain_sm12x_block_m` to fit its 99KB of + # shared memory; it is still ordered after MARLIN by the selector. + return cap is not None and (9, 0) <= (cap.major, cap.minor) < (13, 0) if p.is_rocm(): from vllm.platforms.rocm import on_gfx1x, on_gfx9 @@ -57,6 +59,36 @@ def _triton_kernel_moe_supports_current_device() -> bool: return False +@functools.cache +def _is_sm12x() -> bool: + return current_platform.is_cuda() and current_platform.is_device_capability_family( + 120 + ) + + +# Consumer Blackwell has 99KB of opt-in shared memory: with block_m >= 64 the +# persistent mxfp4 kernel fits a single pipeline stage and runs 2-3x slower +# than with 32, but triton_kernels' block_m heuristic still tops out at 128. +_SM12X_MAX_BLOCK_M = 32 + + +def _sm12x_block_m(num_rows: int, num_experts: int) -> int: + """triton_kernels' tokens-per-expert block_m heuristic, capped at 32.""" + tokens_per_expt = max(1, num_rows // num_experts) + return max(16, min(triton.next_power_of_2(tokens_per_expt), _SM12X_MAX_BLOCK_M)) + + +def _constrain_sm12x_block_m(num_rows: int, num_experts: int) -> None: + """Pin block_m for the following matmul_ogs calls on SM12x.""" + if not _is_sm12x(): + return + import triton_kernels.matmul_ogs_details.opt_flags as opt_flags + + opt_flags.update_opt_flags_constraints( + {"block_m": _sm12x_block_m(num_rows, num_experts)} + ) + + def _patch_make_bitmatrix_metadata() -> None: """Monkey-patch make_bitmatrix_metadata to support non-power-of-2 top_k. @@ -784,6 +816,7 @@ def triton_kernel_fused_experts( ) ) gammas = routing_data.gate_scal if routing_data else None + _constrain_sm12x_block_m(M * topk, E) matmul_ogs( hidden_states, @@ -1403,6 +1436,7 @@ class UnfusedOAITritonExperts(LoRAExpertsMixin, BaseOAITritonExperts): intermediate_cache2 = _resize_cache(workspace13, (M * topk, activation_out_dim)) gammas = routing_data.gate_scal if routing_data else None + _constrain_sm12x_block_m(M * topk, E) matmul_ogs( hidden_states, diff --git a/vllm/model_executor/layers/fused_moe/experts/trtllm_bf16_moe.py b/vllm/model_executor/layers/fused_moe/experts/trtllm_bf16_moe.py index 782d6bddd5..760be97e15 100644 --- a/vllm/model_executor/layers/fused_moe/experts/trtllm_bf16_moe.py +++ b/vllm/model_executor/layers/fused_moe/experts/trtllm_bf16_moe.py @@ -254,6 +254,7 @@ class TrtLlmBf16ExpertsMonolithic(TrtLlmBf16ExpertsBase, mk.FusedMoEExpertsMonol RoutingMethodType.Renormalize, RoutingMethodType.RenormalizeNaive, RoutingMethodType.SigmoidRenorm, + RoutingMethodType.MiniMax2, RoutingMethodType.Sigmoid, ] diff --git a/vllm/model_executor/layers/fused_moe/experts/xpu_moe.py b/vllm/model_executor/layers/fused_moe/experts/xpu_moe.py index ddd891b1c0..35fb916078 100644 --- a/vllm/model_executor/layers/fused_moe/experts/xpu_moe.py +++ b/vllm/model_executor/layers/fused_moe/experts/xpu_moe.py @@ -175,6 +175,11 @@ class XPUExperts(mk.FusedMoEExpertsModular): gemm1_clamp_limit=self.gemm1_clamp_limit, ) assert self.fused_moe_impl is not None + if self.moe_config.ep_size > 1: + # Avoid topk_ids = -1, remap them to an + # expert not owned by this rank so they are skipped. + sentinel = global_num_experts - 1 if self.moe_config.ep_rank == 0 else 0 + topk_ids = torch.where(topk_ids < 0, sentinel, topk_ids) self.fused_moe_impl.apply( output=output, hidden_states=hidden_states, diff --git a/vllm/model_executor/layers/fused_moe/layer.py b/vllm/model_executor/layers/fused_moe/layer.py index ff620622c1..6b7c64bdc4 100644 --- a/vllm/model_executor/layers/fused_moe/layer.py +++ b/vllm/model_executor/layers/fused_moe/layer.py @@ -118,10 +118,11 @@ def FusedMoEFactory( has_bias: bool = False, is_sequence_parallel: bool = False, reduce_results: bool = True, - ckpt_names: tuple[str, str, str] = ("gate_proj", "down_proj", "up_proj"), + ckpt_names: tuple[str, str, str | None] = ("gate_proj", "down_proj", "up_proj"), is_fused_checkpoint_transposed: bool = False, n_shared_experts: int | None = None, fuse_shared_experts: bool = False, + shared_expert_prefix: str | None = None, router_logits_dtype: torch.dtype | None = None, gate: torch.nn.Module | None = None, shared_experts: torch.nn.Module | None = None, @@ -197,6 +198,7 @@ def FusedMoEFactory( n_shared_experts: Number of shared experts to fuse into the routed grouped GEMM (ROCm; requires aiter FSE or the router-append path) fuse_shared_experts: Whether to enable shared-expert fusion. + shared_expert_prefix: Checkpoint prefix for the fused shared expert. router_logits_dtype: Data type for router logits buffers gate: Pre-configured gate module shared_experts: Pre-configured shared experts module @@ -360,6 +362,7 @@ def FusedMoEFactory( elastic_ep_max_dp_size=vllm_config.parallel_config.elastic_ep_max_dp_size, has_bias=has_bias, is_lora_enabled=vllm_config.lora_config is not None, + shared_expert_prefix=shared_expert_prefix, activation=moe_activation, device=vllm_config.device_config.device, routing_method=router.routing_method_type, # Not ideal @@ -442,7 +445,7 @@ def fused_moe_make_expert_params_mapping( model: torch.nn.Module, ckpt_gate_proj_name: str, ckpt_down_proj_name: str, - ckpt_up_proj_name: str, + ckpt_up_proj_name: str | None, num_experts: int, num_redundant_experts: int = 0, routed_experts_prefix: str = "routed_experts", diff --git a/vllm/model_executor/layers/fused_moe/modular_kernel.py b/vllm/model_executor/layers/fused_moe/modular_kernel.py index 9bde03795c..7a4cbd4237 100644 --- a/vllm/model_executor/layers/fused_moe/modular_kernel.py +++ b/vllm/model_executor/layers/fused_moe/modular_kernel.py @@ -197,6 +197,20 @@ class FusedMoEPrepareAndFinalize(ABC): """ return + def allocate_fused_expert_output( + self, + shape: tuple[int, ...], + dtype: torch.dtype, + device: torch.device, + weight_and_reduce_impl: TopKWeightAndReduce, + ) -> torch.Tensor | None: + """Optionally provide the exact output buffer for the expert kernel. + + Implementations returning a tensor must preserve its identity through + finalize; replacing it may silently reintroduce a write-back copy. + """ + return None + @property @abstractmethod def activation_format(self) -> FusedMoEActivationFormat: @@ -1054,13 +1068,14 @@ class FusedMoEKernelModularImpl: local_num_experts: int, expert_tokens_meta: ExpertTokensMetadata | None, activation: MoEActivation, - ) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]: + ) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, bool]: """Allocate temporary and output buffers for the fused experts op. Inputs: - out_dtype: output type of workspace and output tensors. - device: the device of the workspace and output tensors. See `workspace_shapes` for a description of the remainder of arguments. - Returns a tuple of (workspace13, workspace2, output) tensors. + Returns the workspace tensors, output tensor, and whether the output + uses a prepare/finalize-provided allocation. """ assert M_full > 0 and M_chunk > 0 @@ -1090,9 +1105,20 @@ class FusedMoEKernelModularImpl: activation, ) + fused_out_alias = self.prepare_finalize.allocate_fused_expert_output( + fused_out_shape, + workspace_dtype, + device, + self.fused_experts.finalize_weight_and_reduce_impl(), + ) + # We can reuse the memory between cache1 and cache3 because by the # time we need cache3, we're done with cache1. # Reuse workspace13 for the output since there is only one chunk. + # Keep the original fused-output reservation even when the output is + # externally aliased. The workspace manager profiles and locks a + # process-wide high-water mark; later graph shapes may need this + # storage for workspace13. max_shape_size = max(prod(workspace13_shape), prod(fused_out_shape)) if current_platform.is_cpu(): @@ -1110,16 +1136,23 @@ class FusedMoEKernelModularImpl: ) workspace13 = _resize_cache(common_workspace, workspace13_shape) fused_out = _resize_cache(common_workspace, fused_out_shape) - return workspace13, workspace2, fused_out + return workspace13, workspace2, fused_out, False common_workspace, workspace2 = current_workspace_manager().get_simultaneous( ((max_shape_size,), workspace_dtype), (workspace2_shape, workspace_dtype), ) workspace13 = _resize_cache(common_workspace, workspace13_shape) - fused_out = _resize_cache(common_workspace, fused_out_shape) + if fused_out_alias is None: + fused_out = _resize_cache(common_workspace, fused_out_shape) + else: + assert fused_out_alias.shape == fused_out_shape + assert fused_out_alias.dtype == workspace_dtype + assert fused_out_alias.device == device + assert fused_out_alias.is_contiguous() + fused_out = fused_out_alias - return workspace13, workspace2, fused_out + return workspace13, workspace2, fused_out, fused_out_alias is not None def _maybe_apply_shared_experts( self, @@ -1251,22 +1284,25 @@ class FusedMoEKernelModularImpl: if M_full == 0: return torch.empty_like(a1q, dtype=in_dtype) - workspace13, workspace2, fused_out = self._allocate_buffers( - in_dtype, - a1q.device, - M_full, - M_full, - N, - K, - top_k, - global_num_experts, - local_num_experts, - expert_tokens_meta, - activation, + workspace13, workspace2, fused_out, fused_out_is_specialized = ( + self._allocate_buffers( + in_dtype, + a1q.device, + M_full, + M_full, + N, + K, + top_k, + global_num_experts, + local_num_experts, + expert_tokens_meta, + activation, + ) ) use_output_alias = ( - output_alias is not None + not fused_out_is_specialized + and output_alias is not None and output_alias.shape == fused_out.shape and output_alias.dtype == fused_out.dtype and output_alias.device == fused_out.device diff --git a/vllm/model_executor/layers/fused_moe/moe_permute_unpermute.py b/vllm/model_executor/layers/fused_moe/moe_permute_unpermute.py index d83152fc69..b4bf3c8916 100644 --- a/vllm/model_executor/layers/fused_moe/moe_permute_unpermute.py +++ b/vllm/model_executor/layers/fused_moe/moe_permute_unpermute.py @@ -13,90 +13,70 @@ from vllm.v1.worker.workspace import ( @dataclass class MoEPermuteScratch: - # Reused metadata buffers for repeated grouped-MoE permutes. - max_num_tokens: int - topk: int + """Reused metadata buffers for repeated grouped-MoE permutes. + + Capacity follows the largest input seen, like the shared workspace: it grows + during warmup and is fixed once the workspace manager is locked. + """ + num_experts: int num_local_experts: int device: torch.device - hidden_size: int | None = None - hidden_dtype: torch.dtype | None = None + max_expanded_rows: int = field(init=False, default=0) token_expert_indices: torch.Tensor = field(init=False) expert_first_token_offset: torch.Tensor = field(init=False) permuted_idx: torch.Tensor = field(init=False) inv_permuted_idx: torch.Tensor = field(init=False) - permuted_hidden_states: torch.Tensor | None = field(init=False, default=None) sort_workspace: torch.Tensor = field(init=False) permuted_experts_id: torch.Tensor = field(init=False) sorted_row_idx: torch.Tensor = field(init=False) topk_ids_int32: torch.Tensor = field(init=False) topk_ids_for_sort: torch.Tensor = field(init=False) - max_expanded_rows: int = field(init=False) def __post_init__(self) -> None: - assert self.max_num_tokens > 0 - assert self.topk > 0 assert self.num_experts > 0 assert self.num_local_experts > 0 - if self.hidden_size is None: - assert self.hidden_dtype is None - else: - assert self.hidden_dtype is not None - - self.max_expanded_rows = self.max_num_tokens * self.topk - self.token_expert_indices = torch.arange( - self.max_expanded_rows, dtype=torch.int32, device=self.device - ) self.expert_first_token_offset = torch.empty( self.num_local_experts + 1, dtype=torch.int64, device=self.device ) - self.permuted_idx = torch.empty( - self.max_expanded_rows, dtype=torch.int32, device=self.device - ) - self.inv_permuted_idx = torch.empty( - self.max_expanded_rows, dtype=torch.int32, device=self.device - ) - if self.hidden_size is not None: - hidden_numel = self.max_expanded_rows * self.hidden_size - self.permuted_hidden_states = torch.empty( - hidden_numel, dtype=self.hidden_dtype, device=self.device + # torch.device("cuda") in config, after initialized, + # will be changed to cuda:{index}, so we need to refresh here. + self.device = self.expert_first_token_offset.device + + def reserve(self, expanded_rows: int) -> None: + """Ensure capacity for ``expanded_rows`` (num_tokens * topk) rows.""" + if expanded_rows <= self.max_expanded_rows: + return + if ( + is_workspace_manager_initialized() + and current_workspace_manager().is_locked() + ): + raise AssertionError( + f"Workspace is locked but MoE permute scratch requires " + f"{expanded_rows} rows, current capacity is " + f"{self.max_expanded_rows} rows. Scratch growth is not allowed " + "after locking." ) - self.permuted_experts_id = torch.empty( - self.max_expanded_rows, dtype=torch.int32, device=self.device - ) - self.sorted_row_idx = torch.empty( - self.max_expanded_rows, dtype=torch.int32, device=self.device - ) - self.topk_ids_int32 = torch.empty( - self.max_expanded_rows, dtype=torch.int32, device=self.device - ) - self.topk_ids_for_sort = torch.empty( - self.max_expanded_rows, dtype=torch.int32, device=self.device + + def empty_rows() -> torch.Tensor: + return torch.empty(expanded_rows, dtype=torch.int32, device=self.device) + + self.token_expert_indices = torch.arange( + expanded_rows, dtype=torch.int32, device=self.device ) + self.permuted_idx = empty_rows() + self.inv_permuted_idx = empty_rows() + self.permuted_experts_id = empty_rows() + self.sorted_row_idx = empty_rows() + self.topk_ids_int32 = empty_rows() + self.topk_ids_for_sort = empty_rows() sorter_size = torch.ops._moe_C.moe_permute_sort_workspace_size( - self.max_expanded_rows, self.num_experts + expanded_rows, self.num_experts ) self.sort_workspace = torch.empty( sorter_size, dtype=torch.int8, device=self.device ) - # torch.device("cuda") in config, after initialized, - # will be changed to cuda:{index}, so we need to refresh here. - self.device = self.token_expert_indices.device - - def validate(self, hidden_states: torch.Tensor, topk_ids: torch.Tensor) -> None: - n_token, n_hidden = hidden_states.shape - assert hidden_states.device == self.device - assert topk_ids.device == self.device - assert n_token <= self.max_num_tokens - assert topk_ids.size(1) == self.topk - assert topk_ids.size(0) == n_token - if self.hidden_size is not None: - assert n_hidden == self.hidden_size - assert hidden_states.dtype == self.hidden_dtype - assert self.permuted_hidden_states is not None - - def token_expert_indices_view(self, n_token: int) -> torch.Tensor: - return self.token_expert_indices[: n_token * self.topk].view(n_token, self.topk) + self.max_expanded_rows = expanded_rows def prepare_topk_ids(self, topk_ids: torch.Tensor) -> torch.Tensor: if topk_ids.dtype == torch.int32: @@ -114,17 +94,16 @@ def moe_prepare_scatter( ) -> tuple[torch.Tensor, torch.Tensor]: """Generate expert offsets and shared scatter/unpermute destination indices.""" assert topk_ids.device == scratch.device - assert topk_ids.size(0) <= scratch.max_num_tokens - assert topk_ids.size(1) == scratch.topk n_token, topk = topk_ids.shape expanded_rows = topk_ids.numel() + scratch.reserve(expanded_rows) inverse = scratch.inv_permuted_idx[:expanded_rows].view(n_token, topk) if expanded_rows == 0: scratch.expert_first_token_offset.zero_() return scratch.expert_first_token_offset, inverse torch.ops._moe_C.moe_prepare_scatter( scratch.prepare_topk_ids(topk_ids), - scratch.token_expert_indices_view(n_token), + scratch.token_expert_indices[:expanded_rows].view(n_token, topk), expert_map, scratch.num_experts, scratch.num_local_experts, @@ -140,13 +119,9 @@ def moe_prepare_scatter( def get_moe_permute_scratch( *, - max_num_tokens: int, - topk: int, num_experts: int, num_local_experts: int, device: torch.device, - hidden_size: int | None = None, - hidden_dtype: torch.dtype | None = None, ) -> MoEPermuteScratch: """Share scratch across sequential layers in the current ubatch and lane. @@ -155,28 +130,15 @@ def get_moe_permute_scratch( def create_scratch() -> MoEPermuteScratch: return MoEPermuteScratch( - max_num_tokens=max_num_tokens, - topk=topk, num_experts=num_experts, num_local_experts=num_local_experts, device=device, - hidden_size=hidden_size, - hidden_dtype=hidden_dtype, ) if not is_workspace_manager_initialized(): return create_scratch() - key = ( - MoEPermuteScratch, - max_num_tokens, - topk, - num_experts, - num_local_experts, - device, - hidden_size, - hidden_dtype, - ) + key = (MoEPermuteScratch, num_experts, num_local_experts, device) return current_workspace_manager().get_persistent_resource(key, create_scratch) @@ -204,9 +166,8 @@ def moe_permute( parallel shard. permuted_hidden_states (Optional[torch.Tensor]): Optional output tensor. If None, the output tensor will be created in this function. - scratch (Optional[MoEPermuteScratch]): Optional preallocated scratch - buffers. Validated against hidden_states and topk_ids when given, - otherwise the buffers are allocated in this function. + scratch (Optional[MoEPermuteScratch]): Optional reusable metadata + buffers. Otherwise they are allocated in this function. Returns: - permuted_hidden_states (torch.Tensor): permuted activation. @@ -227,20 +188,11 @@ def moe_permute( if n_local_expert == -1: n_local_expert = n_expert if permuted_hidden_states is None: - if scratch is None: - permuted_hidden_states = torch.empty( - (permuted_row_size, n_hidden), - dtype=hidden_states.dtype, - device=hidden_states.device, - ) - else: - scratch.validate(hidden_states, topk_ids) - hidden_numel = permuted_row_size * n_hidden - scratch_hidden_states = scratch.permuted_hidden_states - assert scratch_hidden_states is not None - permuted_hidden_states = scratch_hidden_states[:hidden_numel].view( - permuted_row_size, n_hidden - ) + permuted_hidden_states = torch.empty( + (permuted_row_size, n_hidden), + dtype=hidden_states.dtype, + device=hidden_states.device, + ) assert permuted_hidden_states.size() == (permuted_row_size, n_hidden), ( f"Expected permuted hidden states to be {(permuted_row_size, n_hidden)}" f" but got {permuted_hidden_states.size()}" @@ -278,10 +230,15 @@ def moe_permute( permuted_idx, ) else: - scratch.validate(hidden_states, topk_ids) + assert hidden_states.device == scratch.device + assert topk_ids.device == scratch.device + assert topk_ids.size(0) == n_token assert n_expert == scratch.num_experts assert n_local_expert == scratch.num_local_experts - token_expert_indices = scratch.token_expert_indices_view(n_token) + scratch.reserve(permuted_row_size) + token_expert_indices = scratch.token_expert_indices[:permuted_row_size].view( + n_token, topk + ) expert_first_token_offset = scratch.expert_first_token_offset permuted_idx = scratch.permuted_idx[:permuted_row_size] permuted_idx.fill_(permuted_row_size) diff --git a/vllm/model_executor/layers/fused_moe/oracle/fp8.py b/vllm/model_executor/layers/fused_moe/oracle/fp8.py index 5219fff617..446825c590 100644 --- a/vllm/model_executor/layers/fused_moe/oracle/fp8.py +++ b/vllm/model_executor/layers/fused_moe/oracle/fp8.py @@ -27,6 +27,7 @@ from vllm.model_executor.layers.quantization.utils.flashinfer_utils import ( from vllm.model_executor.layers.quantization.utils.fp8_utils import ( prepare_fp8_moe_layer_for_deepgemm, ) +from vllm.model_executor.layers.quantization.utils.humming import prioritize_humming from vllm.model_executor.layers.quantization.utils.marlin_utils_fp8 import ( prepare_fp8_moe_layer_for_marlin, ) @@ -37,6 +38,7 @@ from vllm.model_executor.layers.quantization.utils.quant_utils import ( kFp8Dynamic128Sym, kFp8Static128BlockSym, ) +from vllm.model_executor.utils import replace_parameter from vllm.platforms import current_platform from vllm.utils.math_utils import round_up @@ -98,6 +100,7 @@ def _get_priority_backends( Fp8MoeBackend.CPU, Fp8MoeBackend.HPC, ] + _AVAILABLE_BACKENDS = prioritize_humming(_AVAILABLE_BACKENDS) def _move_to_front(backends: list[Fp8MoeBackend], backend: Fp8MoeBackend) -> None: backends.insert(0, backends.pop(backends.index(backend))) @@ -607,6 +610,21 @@ def convert_to_fp8_moe_kernel_format( convert_to_humming_moe_kernel_format, ) + # Online quantization has not installed the quantized tensors yet. + scale_name = ( + "weight_scale_inv" + if hasattr(layer, "w13_weight_scale_inv") + else "weight_scale" + ) + for prefix, weight, scale, input_scale in ( + ("w13", w13, w13_scale, w13_input_scale), + ("w2", w2, w2_scale, w2_input_scale), + ): + replace_parameter(layer, f"{prefix}_weight", weight) + replace_parameter(layer, f"{prefix}_{scale_name}", scale) + if input_scale is not None: + replace_parameter(layer, f"{prefix}_input_scale", input_scale) + convert_to_humming_moe_kernel_format( layer, quant_config=_humming_fp8_weight_schema(layer, w13, w13_scale) ) diff --git a/vllm/model_executor/layers/fused_moe/oracle/int_wna16.py b/vllm/model_executor/layers/fused_moe/oracle/int_wna16.py index 239fe1d55a..9b4b163379 100644 --- a/vllm/model_executor/layers/fused_moe/oracle/int_wna16.py +++ b/vllm/model_executor/layers/fused_moe/oracle/int_wna16.py @@ -32,6 +32,7 @@ from vllm.model_executor.layers.fused_moe.experts.trtllm_mxint4_moe import ( TrtLlmMxint4ExpertsMonolithic, ) from vllm.model_executor.layers.quantization.base_config import QuantizationConfig +from vllm.model_executor.layers.quantization.utils.humming import prioritize_humming from vllm.model_executor.layers.quantization.utils.marlin_utils import ( check_moe_marlin_supports_config, marlin_act_int8_process_scales, @@ -129,7 +130,7 @@ def _get_priority_backends() -> list[WNA16MoEBackend]: if current_platform.is_xpu(): return [WNA16MoEBackend.XPU] - return [ + backends = [ # Native HIP kernel, gated on gfx1100 by _supports_current_device(). WNA16MoEBackend.RDNA3, WNA16MoEBackend.FLASHINFER_TRTLLM, @@ -139,6 +140,7 @@ def _get_priority_backends() -> list[WNA16MoEBackend]: WNA16MoEBackend.HUMMING, WNA16MoEBackend.EMULATION, ] + return prioritize_humming(backends) def _backend_incompatibility_reason( @@ -924,15 +926,18 @@ def _process_weights_rdna3( ]: """RDNA3 (gfx1100) W4A16 weight post-processing. - Interleaves the packed nibbles per expert (the exllama shuffle the dense - RDNA3 kernel also uses) and synthesizes the symmetric zero points that - ``moe_gptq_gemm_rdna3`` dequantizes with. The packed layout - ``[E, K // 8, N]`` and the ``[E, groups, N]`` scales are already what the - kernel wants, so neither is repacked. + Transposes the canonical N-first inputs to the K-first layout of + ``moe_gptq_gemm_rdna3`` (weights ``[E, K // 8, N]``, scales + ``[E, groups, N]``), interleaves the packed nibbles per expert (the exllama + shuffle the dense RDNA3 kernel also uses) and synthesizes the symmetric + zero points the kernel dequantizes with. """ device = w13.device num_experts = w13.size(0) + w13 = w13.transpose(1, 2).contiguous() + w2 = w2.transpose(1, 2).contiguous() + for e in range(num_experts): w13_e = w13[e].contiguous() ops.gptq_shuffle(w13_e, 4) @@ -948,8 +953,8 @@ def _process_weights_rdna3( return ( w13, w2, - w13_scale.contiguous(), - w2_scale.contiguous(), + w13_scale.transpose(1, 2).contiguous(), + w2_scale.transpose(1, 2).contiguous(), _qzeros(w13), _qzeros(w2), None, # w13_input_global_scale diff --git a/vllm/model_executor/layers/fused_moe/oracle/mxfp4.py b/vllm/model_executor/layers/fused_moe/oracle/mxfp4.py index a3c65dc4a5..f5fccc0343 100644 --- a/vllm/model_executor/layers/fused_moe/oracle/mxfp4.py +++ b/vllm/model_executor/layers/fused_moe/oracle/mxfp4.py @@ -35,6 +35,7 @@ from vllm.model_executor.layers.fused_moe.config import ( from vllm.model_executor.layers.quantization.utils.flashinfer_utils import ( swap_w13_to_w31, ) +from vllm.model_executor.layers.quantization.utils.humming import prioritize_humming from vllm.model_executor.layers.quantization.utils.mxfp4_utils import ( _swizzle_mxfp4, ) @@ -113,6 +114,18 @@ def _pack_deepgemm_mxfp4_scales( ) +def _swizzle_cutlass_mxfp4_scales(scale: torch.Tensor) -> torch.Tensor: + """Swizzle [E, N, K // 32] block scales into the CUTLASS tiled layout.""" + from vllm.model_executor.layers.fused_moe.experts.cutlass_moe import ( + swizzle_mxfp4_scales, + ) + + n, scale_k = scale.shape[1:] + return torch.stack( + [swizzle_mxfp4_scales(s, n, scale_k * 32).reshape(n, scale_k) for s in scale] + ) + + class Mxfp4MoeBackend(Enum): NONE = "None" # b12x backends @@ -126,6 +139,8 @@ class Mxfp4MoeBackend(Enum): # FlashInfer CUTLASS backends FLASHINFER_CUTLASS_MXFP4_MXFP8 = "FLASHINFER_CUTLASS_MXFP4_MXFP8" FLASHINFER_CUTLASS_MXFP4_BF16 = "FLASHINFER_CUTLASS_MXFP4_BF16" + # vLLM CUTLASS backend + CUTLASS_MXFP4_MXFP4 = "CUTLASS_MXFP4_MXFP4" # Marlin BATCHED_MARLIN = "BATCHED_MARLIN" MARLIN = "MARLIN" @@ -217,6 +232,13 @@ def backend_to_kernel_cls( return [FlashInferExperts] + elif backend == Mxfp4MoeBackend.CUTLASS_MXFP4_MXFP4: + from vllm.model_executor.layers.fused_moe.experts.cutlass_moe import ( + CutlassExpertsMxfp4, + ) + + return [CutlassExpertsMxfp4] + elif backend == Mxfp4MoeBackend.TRITON: from vllm.model_executor.layers.fused_moe.experts.gpt_oss_triton_kernels_moe import ( # noqa: E501 OAITritonExperts, @@ -377,21 +399,32 @@ def _get_priority_backends_for_gpt_oss() -> list[Mxfp4MoeBackend]: Mxfp4MoeBackend.TRITON, Mxfp4MoeBackend.FLASHINFER_CUTLASS_MXFP4_BF16, Mxfp4MoeBackend.FLASHINFER_CUTLASS_MXFP4_MXFP8, + Mxfp4MoeBackend.CUTLASS_MXFP4_MXFP4, # TRITON_UNFUSED has bug with MTP support # TODO re-enable after kernel is fixed # TRITON_UNFUSED Mxfp4MoeBackend.MARLIN, Mxfp4MoeBackend.BATCHED_MARLIN, + Mxfp4MoeBackend.HUMMING, Mxfp4MoeBackend.XPU, Mxfp4MoeBackend.CPU, Mxfp4MoeBackend.EMULATION, ] - return _AVAILABLE_BACKENDS + if current_platform.is_cuda() and current_platform.is_device_capability_family(120): + # Consumer Blackwell: the OAI Triton kernels run but their 99KB shared + # memory budget caps them at small tiles, so MARLIN prefills ~10% + # faster at equal decode throughput. Keep TRITON as an opt-in. + _AVAILABLE_BACKENDS.remove(Mxfp4MoeBackend.TRITON) + _AVAILABLE_BACKENDS.insert( + _AVAILABLE_BACKENDS.index(Mxfp4MoeBackend.BATCHED_MARLIN) + 1, + Mxfp4MoeBackend.TRITON, + ) + return prioritize_humming(_AVAILABLE_BACKENDS) def _get_priority_backends() -> list[Mxfp4MoeBackend]: """Get available backends in priority order. SM100+ prefers DeepGEMM FP4 / - TRTLLM MXFP8; SM90 falls through to Triton_unfused or Marlin (the + TRTLLM MXFP8; SM90 falls through to Humming or Marlin (the backend-level ``is_supported_config`` check filters by device capability). """ if current_platform.is_rocm(): @@ -411,8 +444,9 @@ def _get_priority_backends() -> list[Mxfp4MoeBackend]: # TRITON_UNFUSED Mxfp4MoeBackend.MARLIN, Mxfp4MoeBackend.BATCHED_MARLIN, + Mxfp4MoeBackend.HUMMING, ] - return _AVAILABLE_BACKENDS + return prioritize_humming(_AVAILABLE_BACKENDS) def _backend_activation_key(backend: Mxfp4MoeBackend) -> QuantKey | None: @@ -428,7 +462,10 @@ def _backend_activation_key(backend: Mxfp4MoeBackend) -> QuantKey | None: return kMxfp8Dynamic if backend == Mxfp4MoeBackend.AITER_MXFP4_FP8: return kFp8StaticTensorSym - if backend == Mxfp4MoeBackend.AITER_MXFP4_MXFP4: + if backend in ( + Mxfp4MoeBackend.AITER_MXFP4_MXFP4, + Mxfp4MoeBackend.CUTLASS_MXFP4_MXFP4, + ): return kMxfp4Dynamic return None # BF16 activation @@ -462,6 +499,22 @@ def _make_log_backend(backend: Mxfp4MoeBackend) -> str: return f"Using '{backend.value}' Mxfp4 MoE backend." +def _warn_on_bf16_fallback( + backend: Mxfp4MoeBackend, requested_activation_key: QuantKey | None +) -> None: + if ( + requested_activation_key is not None + and backend != Mxfp4MoeBackend.EMULATION + and _backend_activation_key(backend) is None + ): + logger.warning_once( + "No Mxfp4 MoE backend supports activation=%s for the deployment " + "configuration. Falling back to '%s' with BF16 activations.", + requested_activation_key, + backend.value, + ) + + def _make_log_unsupported(backend: Mxfp4MoeBackend, reason: str | None) -> str: base = ( f"Mxfp4 MoE backend '{backend.value}' does not support the " @@ -493,17 +546,19 @@ def _filter_by_activation( backends: list[Mxfp4MoeBackend], requested_activation_key: QuantKey | None, ) -> list[Mxfp4MoeBackend]: - """Pick variants matching ``requested_activation_key``; without one, - prefer BF16 if the list has any, else keep the list as-is so explicit - non-BF16 picks (e.g. the ``_afp8`` aliases) still land.""" + """Pick variants matching ``requested_activation_key``, followed by BF16 + variants as a higher-precision fallback; without one, prefer BF16 if the + list has any, else keep the list as-is so explicit non-BF16 picks (e.g. + the ``_afp8`` aliases) still land.""" + bf16 = [b for b in backends if _backend_activation_key(b) is None] if requested_activation_key is not None: - return [ + matching = [ b for b in backends if _backend_activation_key(b) == requested_activation_key or b == Mxfp4MoeBackend.EMULATION ] - bf16 = [b for b in backends if _backend_activation_key(b) is None] + return matching + [b for b in bf16 if b not in matching] return bf16 if bf16 else backends @@ -564,8 +619,9 @@ def select_mxfp4_moe_backend( Args: config: MoE configuration - activation_key: Optional activation quantization key. If provided, - overrides the default activation key for backend selection. + activation_key: Optional activation quantization key declared by + the checkpoint. Backends that match it are preferred, with BF16 + activation backends as a fallback. Use kFp8StaticTensorSym for W4A8 scheme. Note: Shape-specific fallbacks may still occur at runtime. @@ -602,7 +658,7 @@ def select_mxfp4_moe_backend( else _backend_activation_key(requested_backend) ) try: - return _return_or_raise( + selected = _return_or_raise( requested_backend, config, kMxfp4Static, @@ -611,6 +667,9 @@ def select_mxfp4_moe_backend( ) except ValueError as e: last_error = e + else: + _warn_on_bf16_fallback(requested_backend, requested_activation_key) + return selected assert last_error is not None raise last_error @@ -636,10 +695,10 @@ def select_mxfp4_moe_backend( unsupported_reasons = [] for backend in AVAILABLE_BACKENDS: - # Use requested_activation_key if provided, otherwise use backend default + # EMULATION takes the requested key, other backends use their own act_key = ( requested_activation_key - if requested_activation_key is not None + if backend == Mxfp4MoeBackend.EMULATION else _backend_activation_key(backend) ) for k_cls in backend_to_kernel_cls(backend): @@ -648,6 +707,7 @@ def select_mxfp4_moe_backend( ) if supported: logger.info_once(_make_log_backend(backend)) + _warn_on_bf16_fallback(backend, requested_activation_key) return backend, k_cls else: logger.debug_once(_make_log_unsupported(backend, reason)) @@ -777,6 +837,10 @@ def mxfp4_round_up_hidden_size_and_intermediate_size( # DeepGEMM requires M/N/K alignment intermediate_size = round_up(intermediate_size, 128) hidden_size = round_up(hidden_size, 128) + elif backend == Mxfp4MoeBackend.CUTLASS_MXFP4_MXFP4: + # CUTLASS scale tiles cover 128 rows by 128 elements + intermediate_size = round_up(intermediate_size, 128) + hidden_size = round_up(hidden_size, 128) elif backend in (Mxfp4MoeBackend.MARLIN, Mxfp4MoeBackend.BATCHED_MARLIN): intermediate_size = round_up(intermediate_size, 128) if current_platform.is_xpu(): @@ -1421,7 +1485,8 @@ def convert_weight_to_mxfp4_moe_kernel_format( ]: """Convert loaded weights into backend-specific kernel format. - Supports DeepGEMM, FlashInfer, TRTLLM MXFP8, Triton and Marlin backends. + Supports CUTLASS, DeepGEMM, FlashInfer, TRTLLM MXFP8, Triton and Marlin + backends. """ is_gfx1250 = False if current_platform.is_rocm(): @@ -1439,6 +1504,16 @@ def convert_weight_to_mxfp4_moe_kernel_format( w2_bias, ) + if mxfp4_backend == Mxfp4MoeBackend.CUTLASS_MXFP4_MXFP4: + return ( + w13_weight.data, + w2_weight.data, + _swizzle_cutlass_mxfp4_scales(w13_weight_scale.data), + _swizzle_cutlass_mxfp4_scales(w2_weight_scale.data), + w13_bias, + w2_bias, + ) + if mxfp4_backend == Mxfp4MoeBackend.DEEPGEMM_MXFP4: w13_weight_scale, w2_weight_scale = _pack_deepgemm_mxfp4_scales( w13_weight, @@ -1993,7 +2068,10 @@ def make_mxfp4_moe_quant_config( block_shape=None, gemm1_clamp_limit=swiglu_limit, ) - elif mxfp4_backend == Mxfp4MoeBackend.AITER_MXFP4_MXFP4: + elif mxfp4_backend in ( + Mxfp4MoeBackend.AITER_MXFP4_MXFP4, + Mxfp4MoeBackend.CUTLASS_MXFP4_MXFP4, + ): return ocp_mx_moe_quant_config( quant_dtype="mxfp4", w1_bias=w1_bias, diff --git a/vllm/model_executor/layers/fused_moe/oracle/mxfp8.py b/vllm/model_executor/layers/fused_moe/oracle/mxfp8.py index 49f61764f0..c4417cdd76 100644 --- a/vllm/model_executor/layers/fused_moe/oracle/mxfp8.py +++ b/vllm/model_executor/layers/fused_moe/oracle/mxfp8.py @@ -8,6 +8,7 @@ from vllm.model_executor.layers.fused_moe.oracle.fp8 import ( Fp8MoeBackend, backend_to_kernel_cls, ) +from vllm.model_executor.layers.quantization.utils.humming import prioritize_humming from vllm.model_executor.layers.quantization.utils.quant_utils import ( kMxfp8Dynamic, kMxfp8Static, @@ -129,7 +130,8 @@ def select_mxfp8_moe_backend( return backend, _select_kernel_cls(backend, config) # Auto-select: pick the first supported backend. - for backend in _SUPPORTED_BACKENDS: + backends = prioritize_humming(list(_SUPPORTED_BACKENDS)) + for backend in backends: try: experts_cls = _select_kernel_cls(backend, config) except ValueError: diff --git a/vllm/model_executor/layers/fused_moe/oracle/nvfp4.py b/vllm/model_executor/layers/fused_moe/oracle/nvfp4.py index 68ba62782c..80aa9122f4 100644 --- a/vllm/model_executor/layers/fused_moe/oracle/nvfp4.py +++ b/vllm/model_executor/layers/fused_moe/oracle/nvfp4.py @@ -26,6 +26,7 @@ from vllm.model_executor.layers.quantization.utils.flashinfer_fp4_moe import ( prepare_nvfp4_moe_layer_for_fi_or_cutlass, prepare_nvfp4_moe_layer_for_flashinfer_cutedsl, ) +from vllm.model_executor.layers.quantization.utils.humming import prioritize_humming from vllm.model_executor.layers.quantization.utils.marlin_utils_fp4 import ( prepare_nvfp4_moe_layer_for_marlin, ) @@ -209,6 +210,7 @@ def select_nvfp4_moe_backend( NvFp4MoeBackend.HUMMING, NvFp4MoeBackend.EMULATION, ] + AVAILABLE_BACKENDS = prioritize_humming(AVAILABLE_BACKENDS) NVFP4_BACKENDS_WITH_CLAMP = { NvFp4MoeBackend.B12X, @@ -332,6 +334,7 @@ def convert_to_nvfp4_moe_kernel_format( a2_scale: torch.Tensor | None, is_act_and_mul: bool, use_a16: bool = False, + trtllm_hidden_alignment: int = 256, ) -> tuple[ torch.Tensor, torch.Tensor, @@ -417,6 +420,7 @@ def convert_to_nvfp4_moe_kernel_format( w2_scale_2=w2_scale_2, a2_scale=a2_scale, is_act_and_mul=is_act_and_mul, + trtllm_hidden_alignment=trtllm_hidden_alignment, ) elif nvfp4_backend == NvFp4MoeBackend.HUMMING: from vllm.model_executor.layers.quantization.utils.humming import ( diff --git a/vllm/model_executor/layers/fused_moe/prepare_finalize/naive_dp_ep.py b/vllm/model_executor/layers/fused_moe/prepare_finalize/naive_dp_ep.py index 97d4e0d284..7087b9016d 100644 --- a/vllm/model_executor/layers/fused_moe/prepare_finalize/naive_dp_ep.py +++ b/vllm/model_executor/layers/fused_moe/prepare_finalize/naive_dp_ep.py @@ -8,6 +8,7 @@ from vllm.model_executor.layers.fused_moe.config import FusedMoEQuantConfig from vllm.model_executor.layers.fused_moe.topk_weight_and_reduce import ( TopKWeightAndReduceContiguous, TopKWeightAndReduceDelegate, + TopKWeightAndReduceNoOP, ) from vllm.model_executor.layers.fused_moe.utils import moe_kernel_quantize_input from vllm.utils.flashinfer import nvfp4_block_scale_interleave @@ -108,6 +109,30 @@ class MoEPrepareAndFinalizeNaiveDPEPModular(mk.FusedMoEPrepareAndFinalizeModular def output_is_reduced(self) -> bool: return False + def _allocate_combine_input( + self, + shape: tuple[int, ...], + dtype: torch.dtype, + device: torch.device, + ) -> torch.Tensor | None: + return get_ep_group().allocate_combine_input( + shape, + dtype, + device, + is_sequence_parallel=self.is_sequence_parallel, + ) + + def allocate_fused_expert_output( + self, + shape: tuple[int, ...], + dtype: torch.dtype, + device: torch.device, + weight_and_reduce_impl: mk.TopKWeightAndReduce, + ) -> torch.Tensor | None: + if isinstance(weight_and_reduce_impl, TopKWeightAndReduceNoOP): + return self._allocate_combine_input(shape, dtype, device) + return None + def prepare( self, a1: torch.Tensor, @@ -194,16 +219,29 @@ class MoEPrepareAndFinalizeNaiveDPEPModular(mk.FusedMoEPrepareAndFinalizeModular if isinstance(weight_and_reduce_impl, TopKWeightAndReduceDelegate): weight_and_reduce_impl = TopKWeightAndReduceContiguous() + if isinstance(weight_and_reduce_impl, TopKWeightAndReduceNoOP): + # allocate_fused_expert_output already selected the combine input. + # Reuse it directly; the symmetric allocator may return a new view + # object for the same high-water backing allocation. + combine_input = fused_expert_output + else: + combine_input = self._allocate_combine_input( + (topk_ids.shape[0], fused_expert_output.shape[-1]), + fused_expert_output.dtype, + fused_expert_output.device, + ) out = weight_and_reduce_impl.apply( - output=None, + output=combine_input, fused_expert_output=fused_expert_output, topk_weights=topk_weights, topk_ids=topk_ids, apply_router_weight_on_input=apply_router_weight_on_input, ) - output.copy_( - get_ep_group().combine(out, is_sequence_parallel=self.is_sequence_parallel) + get_ep_group().combine_into_output( + out, + output, + is_sequence_parallel=self.is_sequence_parallel, ) diff --git a/vllm/model_executor/layers/fused_moe/routed_experts.py b/vllm/model_executor/layers/fused_moe/routed_experts.py index f3c9522df1..ae5829a311 100644 --- a/vllm/model_executor/layers/fused_moe/routed_experts.py +++ b/vllm/model_executor/layers/fused_moe/routed_experts.py @@ -26,6 +26,7 @@ from vllm.model_executor.layers.fused_moe.routed_experts_capturer import ( from vllm.model_executor.layers.fused_moe.unquantized_fused_moe_method import ( UnquantizedFusedMoEMethod, ) +from vllm.model_executor.layers.linear import LinearBase from vllm.model_executor.layers.quantization.base_config import ( QuantizationConfig, resolve_quant_method, @@ -87,7 +88,7 @@ class RoutedExperts(PluggableLayer): expert_map_manager: ExpertMapManager, ckpt_gate_proj_name: str = "gate_proj", ckpt_down_proj_name: str = "down_proj", - ckpt_up_proj_name: str = "up_proj", + ckpt_up_proj_name: str | None = "up_proj", is_fused_checkpoint_transposed: bool = False, # # Extra params that are needed by quant_methods, pass along for now @@ -114,6 +115,13 @@ class RoutedExperts(PluggableLayer): self.ckpt_gate_proj_name = ckpt_gate_proj_name self.ckpt_down_proj_name = ckpt_down_proj_name self.ckpt_up_proj_name = ckpt_up_proj_name + if moe_config.is_act_and_mul == (ckpt_up_proj_name is None): + raise ValueError( + f"{layer_name}: ckpt_up_proj_name={ckpt_up_proj_name!r} is " + f"inconsistent with activation {moe_config.activation.value!r}. " + "Gated MoE needs an up projection name; non-gated MoE must " + "pass None." + ) self.is_fused_checkpoint_transposed = is_fused_checkpoint_transposed self.expert_map_manager = expert_map_manager self.hidden_size = moe_config.hidden_dim @@ -146,6 +154,7 @@ class RoutedExperts(PluggableLayer): self.apply_router_weight_on_input = apply_router_weight_on_input # End random parameters self._loaded_expert_biases: set[str] = set() + self._fused_shared_expert_quantizer = self._get_fused_shared_expert_quantizer() self.quant_method = self._get_quant_method( self.layer_name, @@ -623,6 +632,31 @@ class RoutedExperts(PluggableLayer): # _to_scalar's reshape(()) would reject the size-2 weight_shape. param_data[expert_id] = loaded_weight + def _get_fused_shared_expert_quantizer( + self, + ) -> Callable[[torch.Tensor], tuple[torch.Tensor, torch.Tensor]] | None: + """Return the quantizer for an online-quantized fused shared expert.""" + online_config = getattr(self.quant_config, "online_quantization_config", None) + prefix = self.moe_config.shared_expert_prefix + if ( + online_config is None + or prefix is None + or self.expert_map_manager.num_fused_shared_experts == 0 + ): + return None + # FSE compatibility checks require every shared-expert projection to + # use the routed experts' weight key. + resolved = online_config.resolve_quant_method_cls( + LinearBase, f"{prefix}.down_proj" + ) + if resolved is None: + return None + from vllm.model_executor.layers.quantization.online.base import ( + ONLINE_SHARED_EXPERT_QUANTIZERS, + ) + + return ONLINE_SHARED_EXPERT_QUANTIZERS[resolved[3].weight] + @overload def weight_loader( self, @@ -680,6 +714,35 @@ class RoutedExperts(PluggableLayer): return False if return_success else None # Hereafter, `expert_id` is local physical id + # A full-precision fused shared expert is quantized into the checkpoint + # layout, then loaded like a pre-quantized expert weight and scale. + if ( + self._fused_shared_expert_quantizer is not None + and self.moe_config.num_logical_experts + <= global_expert_id + < self.moe_config.num_logical_experts + + self.expert_map_manager.num_fused_shared_experts + and weight_name.endswith("_weight") + and loaded_weight.is_floating_point() + and loaded_weight.dtype != param.dtype + ): + weight, weight_scale = self._fused_shared_expert_quantizer( + loaded_weight.to(self.moe_config.device) + ) + stem = "w2" if shard_id == "w2" else "w13" + self.weight_loader( + getattr(self, f"{stem}_weight_scale"), + weight_scale, + f"{weight_name}_scale", + shard_id, + global_expert_id, + return_success=True, + ) + loaded = self.weight_loader( + param, weight, weight_name, shard_id, global_expert_id, True + ) + return loaded if return_success else None + # is_transposed: if the dim to shard the weight # should be flipped. Required by GPTQ/AWQ (K-first format). # should be whatever dimension intermediate_size_per_partition is @@ -971,7 +1034,7 @@ class RoutedExperts(PluggableLayer): loaded_weight, self.is_fused_checkpoint_transposed and uses_weight_layout, ) - if shard_id in {"w1", "w3"}: + if shard_id in {"w1", "w3"} and self.moe_config.is_act_and_mul: # Repurpose expert_id for deconcatenating w1 and w3 experts_shard = fused_weight.chunk(2, dim=1)[expert_id] else: @@ -1029,6 +1092,7 @@ class RoutedExperts(PluggableLayer): routed_experts_prefix="", lora_base_layer_prefix=self.lora_base_layer_prefix, include_fused=include_fused, + is_gated=moe_config.is_act_and_mul, ) @staticmethod @@ -1036,7 +1100,7 @@ class RoutedExperts(PluggableLayer): model: torch.nn.Module, ckpt_gate_proj_name: str, ckpt_down_proj_name: str, - ckpt_up_proj_name: str, + ckpt_up_proj_name: str | None, num_experts: int, num_redundant_experts: int = 0, routed_experts_prefix: str = "routed_experts", @@ -1067,13 +1131,14 @@ class RoutedExperts(PluggableLayer): def build_expert_params_mapping( ckpt_gate_proj_name: str, ckpt_down_proj_name: str, - ckpt_up_proj_name: str, + ckpt_up_proj_name: str | None, num_experts: int, num_redundant_experts: int = 0, routed_experts_prefix: str = "routed_experts", lora_base_layer_prefix: str = "", lora_base_layer_prefix_on_param_name: str = "", include_fused: bool = False, + is_gated: bool = True, ) -> list[tuple[str, str, int, str]]: """Create expert parameter mapping for weight loading with redundant experts. @@ -1095,6 +1160,8 @@ class RoutedExperts(PluggableLayer): ``make_expert_params_mapping`` indexes the model-wide ``params_dict`` (prefix included). include_fused: Prepend the fused pre-fused-checkpoint entries + is_gated: Whether w13 holds gate and up (gated activation) or a + single up projection. Selects the fused checkpoint layout. routed_experts_prefix: Prefix of the routed experts submodule Returns: @@ -1129,7 +1196,13 @@ class RoutedExperts(PluggableLayer): fused_mapping = [] if include_fused: gate_up = None - if ckpt_gate_proj_name == "gate_proj" and ckpt_up_proj_name == "up_proj": + w13_shards: tuple[str, ...] = ("w1", "w3") + if not is_gated: + # Non-gated: the stacked checkpoint tensor is the up projection + # itself, nothing to split into gate and up. + gate_up = ckpt_gate_proj_name + w13_shards = ("w1",) + elif ckpt_gate_proj_name == "gate_proj" and ckpt_up_proj_name == "up_proj": gate_up = "gate_up_proj" elif ckpt_gate_proj_name == "w1" and ckpt_up_proj_name == "w3": gate_up = "w13" @@ -1143,10 +1216,16 @@ class RoutedExperts(PluggableLayer): if gate_up is not None: fused_mapping = [ # (param_name, weight_name, expert_id, shard_id) - (f"{w13}weight", f"experts.{gate_up}", 0, "w1"), - (f"{w13}weight", f"experts.{gate_up}", 1, "w3"), + # expert_id doubles as the chunk index when splitting w13. + *( + (f"{w13}weight", f"experts.{gate_up}", chunk, shard_id) + for chunk, shard_id in enumerate(w13_shards) + ), (f"{w2}weight", f"experts.{ckpt_down_proj_name}", 0, "w2"), ] + if gate_up is not None and is_gated: + # Per-expert gate_up_proj tensors; the non-gated equivalent is + # already covered by the per-expert mapping below. fused_mapping.extend( ( w13, @@ -1175,6 +1254,7 @@ class RoutedExperts(PluggableLayer): ("w2", ckpt_down_proj_name), ("w3", ckpt_up_proj_name), ] + if weight_name is not None ] return fused_mapping + per_expert_mapping diff --git a/vllm/model_executor/layers/fused_moe/router/bf16x3_router_gemm_rocm.py b/vllm/model_executor/layers/fused_moe/router/bf16x3_router_gemm_rocm.py new file mode 100644 index 0000000000..8d1fa84c86 --- /dev/null +++ b/vllm/model_executor/layers/fused_moe/router/bf16x3_router_gemm_rocm.py @@ -0,0 +1,163 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""ROCm BF16x3 MoE router GEMM for fp32 gate weights. + +Gates whose weights ship in fp32 (MiniMax-M2/M3, HunYuan-V3) otherwise land in +the ``F.linear`` fallback, which casts the bf16 activations up to fp32 and runs +the GEMM on the fp32 MFMA pipe. Splitting the weight into three bf16 terms +whose sum reproduces it exactly moves the GEMM onto the bf16 pipe and is more +accurate than the path it replaces, which rounds the activations on the way to +fp32 while this one leaves them untouched. + +The terms are stacked along the expert dimension into one ``(TERMS * E, K)`` +weight, so the router costs a single hipBLASLt GEMM plus a reduction over the +``TERMS`` column blocks of its output. +""" + +import torch + +from vllm.platforms import current_platform +from vllm.triton_utils import tl, triton + +__all__ = [ + "BF16X3_TERMS", + "MIN_TOKENS", + "bf16x3_router_gemm", + "is_supported", + "platform_supported", + "split_bf16x3", +] + +BF16X3_TERMS = 3 + +# Below this the fp32 fallback is faster: the GEMM plus reduction has a ~32us +# floor from launch and allocation that small batches cannot amortize. +# Measured on gfx950; the crossover is 1536 at (K=6144, E=128) and (K=3072, +# E=256), and this leaves margin for untested shapes. +MIN_TOKENS = 2048 + +_MAX_INT32 = (1 << 31) - 1 + + +def split_bf16x3(weight: torch.Tensor) -> torch.Tensor: + """Split an fp32 weight into bf16 terms whose sum reproduces it exactly. + + Returns a contiguous ``(BF16X3_TERMS, E, K)`` tensor. Three + round-to-nearest bf16 terms carry 24 mantissa bits, matching fp32's 24, so + the reconstruction is bit-exact for finite normals down to ``2**-108``. + + Raises ValueError on non-finite input: ``inf`` rounds to bf16 ``inf`` and + ``inf - inf`` is NaN, which would poison every logit rather than just the + affected experts. + """ + if weight.dtype != torch.float32: + raise ValueError(f"expected fp32 weight, got {weight.dtype}") + if not torch.isfinite(weight).all(): + raise ValueError("router weight contains inf/NaN") + terms = [] + resid = weight.detach().clone() + for _ in range(BF16X3_TERMS): + hi = resid.to(torch.bfloat16) + terms.append(hi) + resid -= hi.to(torch.float32) + return torch.stack(terms).contiguous() + + +def platform_supported() -> bool: + """Whether this platform has a benchmarked bf16x3 router path. + + gfx942 has a different fp32:bf16 MFMA ratio and is not yet evaluated. + """ + if not current_platform.is_rocm(): + return False + from vllm.platforms.rocm import on_gfx950 + + return on_gfx950() + + +def is_supported(x: torch.Tensor, weight: torch.Tensor) -> bool: + """Whether this path can serve ``x @ weight.T`` -> fp32 profitably.""" + if x.dim() != 2 or weight.dim() != 2: + return False + n, k = weight.shape + m = x.shape[0] + return ( + m >= MIN_TOKENS + and x.dtype == torch.bfloat16 + and weight.dtype == torch.float32 + and x.shape[1] == k + # The weight may be offloaded while the activations are not; the + # fallback this guards would then fail on a device mismatch. + and x.device == weight.device + # torch.mm accepts any stride, but a non-K-contiguous activation costs + # a hidden copy that gives back the win over the fallback. + and x.stride(1) == 1 + and 0 < m * BF16X3_TERMS * n <= _MAX_INT32 + ) + + +@triton.jit +def _reduce_terms_kernel( + y_ptr, + out_ptr, + M, + stride_ym, + stride_om, + N: tl.constexpr, + TERMS: tl.constexpr, + BLOCK_M: tl.constexpr, + BLOCK_N: tl.constexpr, +): + pid = tl.program_id(0) + offs_m = pid * BLOCK_M + tl.arange(0, BLOCK_M) + offs_n = tl.arange(0, BLOCK_N) + mask = (offs_m < M)[:, None] & (offs_n < N)[None, :] + acc = tl.zeros((BLOCK_M, BLOCK_N), dtype=tl.float32) + for t in tl.static_range(TERMS): + acc += tl.load( + y_ptr + offs_m[:, None] * stride_ym + t * N + offs_n[None, :], + mask=mask, + other=0.0, + ) + tl.store(out_ptr + offs_m[:, None] * stride_om + offs_n[None, :], acc, mask=mask) + + +def bf16x3_router_gemm( + x: torch.Tensor, + weight_split: torch.Tensor, + out: torch.Tensor | None = None, +) -> torch.Tensor: + """Router logits in fp32 from bf16 activations and a bf16x3 split weight.""" + m, k = x.shape + terms, n, k_w = weight_split.shape + assert terms == BF16X3_TERMS, f"expected {BF16X3_TERMS} terms, got {terms}" + assert k == k_w, f"K mismatch: x has {k}, weight_split has {k_w}" + assert weight_split.dtype == torch.bfloat16, weight_split.dtype + assert weight_split.is_contiguous(), "weight_split must be contiguous" + assert x.dtype == torch.bfloat16, x.dtype + if out is None: + out = torch.empty((m, n), dtype=torch.float32, device=x.device) + else: + assert out.shape == (m, n) and out.is_contiguous(), "bad out layout" + + # Rows [t*E, (t+1)*E) of the (TERMS * E, K) view hold term t, so the GEMM + # emits one column block per term and the reduction sums them. + y = torch.empty((m, terms * n), dtype=torch.float32, device=x.device) + torch.mm(x, weight_split.view(terms * n, k).T, out_dtype=torch.float32, out=y) + + # 64 rows/program keeps the accumulator small enough to stay in registers + # at the largest supported expert count. + block_m = 64 + _reduce_terms_kernel[(triton.cdiv(m, block_m),)]( + y, + out, + m, + y.stride(0), + out.stride(0), + N=n, + TERMS=terms, + BLOCK_M=block_m, + BLOCK_N=triton.next_power_of_2(n), + num_warps=4, + ) + return out diff --git a/vllm/model_executor/layers/fused_moe/router/gate_linear.py b/vllm/model_executor/layers/fused_moe/router/gate_linear.py index f91e9c449e..8c4961f269 100644 --- a/vllm/model_executor/layers/fused_moe/router/gate_linear.py +++ b/vllm/model_executor/layers/fused_moe/router/gate_linear.py @@ -4,6 +4,7 @@ import torch from torch.nn.parameter import Parameter import vllm._custom_ops as ops +from vllm.logger import init_logger from vllm.model_executor.custom_op import PluggableLayer from vllm.model_executor.layers.linear import ( ReplicatedLinear, @@ -13,6 +14,8 @@ from vllm.model_executor.layers.quantization import QuantizationConfig from vllm.platforms import current_platform from vllm.utils.torch_utils import direct_register_custom_op +logger = init_logger(__name__) + @PluggableLayer.register("gate_linear") class GateLinear(ReplicatedLinear): @@ -23,8 +26,10 @@ class GateLinear(ReplicatedLinear): 2. fp32 specialized kernel (SM90+ or gfx950, bf16/fp32 in, fp32 out, M<=32, model-specific shapes) 3. bf16x3 CuteDSL kernel (SM100, bf16 in, fp32 weight) - 4. cuBLAS bf16×bf16→fp32 (SM90+ + bf16 weight + fp32 out_dtype) - 5. F.linear via ReplicatedLinear (ultimate fallback) + 4. ROCm bf16x3 router GEMM (gfx950, bf16 in, fp32 weight, fp32 out, + M>=2048) + 5. cuBLAS bf16×bf16→fp32 (SM90+ + bf16 weight + fp32 out_dtype) + 6. F.linear via ReplicatedLinear (ultimate fallback) The ``out_dtype`` attribute is mutable and can be set after init (e.g. when the required dtype depends on the expert quantization @@ -140,6 +145,33 @@ class GateLinear(ReplicatedLinear): and self.out_dtype == torch.float32 ) + # ROCm bf16x3 router GEMM eligibility. out_dtype may still be None here + # and is folded in by set_out_dtype; the split itself is only built + # once the tier is known to be on (see _GateLinearMethod). + from vllm.model_executor.layers.fused_moe.router import ( + bf16x3_router_gemm_rocm, + ) + + self._rocm_bf16x3_weight_eligible = ( + self._router_gemm_no_bias + and self.weight.dtype == torch.float32 + and bf16x3_router_gemm_rocm.platform_supported() + ) + self.allow_rocm_bf16x3_router_gemm = ( + self._rocm_bf16x3_weight_eligible and self.out_dtype == torch.float32 + ) + # Filled in by process_weights_after_loading. Not persistent: it is + # derived from `weight` rather than loaded from a checkpoint. + self.register_buffer("_bf16x3_weight", None, persistent=False) + if self._rocm_bf16x3_weight_eligible: + # Safe to swap after create_weights() because + # UnquantizedLinearMethod holds no per-layer state. Asserted + # because the swap would silently drop quantization if a + # quant_config were ever plumbed into this layer. + assert isinstance(self.quant_method, UnquantizedLinearMethod) + self.quant_method = _GateLinearMethod() + logger.info_once("Enabled ROCm BF16x3 router GEMM.") + # cuteDSL ll_bf16_gemm eligibility. Any dims supported, but SM90+ required bc: # 1. PDL support. Both dot-product and split-K kernels. # 2. Thread Block Clusters. Split-K kernel for cross-CTA reduction. @@ -173,6 +205,10 @@ class GateLinear(ReplicatedLinear): self.allow_cublas_router_gemm = self.weight.dtype == torch.bfloat16 # out_dtype may start as None -> recompute eligibility here + self.allow_rocm_bf16x3_router_gemm = ( + self._rocm_bf16x3_weight_eligible and out_dtype == torch.float32 + ) + if self.allow_specialized_router_gemm: from vllm.model_executor.kernels.linear.cute_dsl.ll_bf16 import ( is_available, @@ -205,6 +241,23 @@ class GateLinear(ReplicatedLinear): output = ll_bf16_gemm(x, self.weight) return self._return(output) + # Tier 4: ROCm bf16x3 router GEMM. Checked before tier 2 because both + # tiers claim the same gfx950 shapes and tier 2 returns + # unconditionally; the custom op below picks between them on the + # runtime num_tokens. + if self.allow_rocm_bf16x3_router_gemm and x.dtype == torch.bfloat16: + if self._bf16x3_weight is None: + logger.warning_once( + "ROCm BF16x3 router GEMM is enabled for %s but its weight " + "split is missing; falling back to fp32.", + self.prefix, + ) + else: + output = torch.ops.vllm.rocm_bf16x3_router_gemm_dispatch( + x, self.weight, self._bf16x3_weight + ) + return self._return(output) + # Tier 2: fp32 specialized kernel (model-specific shapes, M<=32) # Dispatch is wrapped in a custom op so that torch.compile/CUDA-graph # capture does not freeze the runtime num_tokens branch. @@ -226,12 +279,12 @@ class GateLinear(ReplicatedLinear): output = bf16x3_router_gemm(x, self.weight) return self._return(output) - # Tier 4: cuBLAS bf16→fp32 + # Tier 5: cuBLAS bf16→fp32 if self.allow_cublas_router_gemm and x.dtype == torch.bfloat16: output = torch.mm(x, self.weight.T, out_dtype=torch.float32) return self._return(output) - # Tier 5: F.linear (ReplicatedLinear) + # Tier 6: F.linear (ReplicatedLinear) if ( self.out_dtype is not None and self.is_unquantized @@ -249,9 +302,90 @@ class GateLinear(ReplicatedLinear): return output if not self.return_bias else (output, output_bias) +class _GateLinearMethod(UnquantizedLinearMethod): + """UnquantizedLinearMethod plus the ROCm bf16x3 weight split. + + Building the split here rather than lazily in forward() charges it to the + weights memory pool ahead of KV cache profiling, and rebuilds it on weight + reload. The latter matters because an in-place ``param.data.copy_()`` + leaves both ``data_ptr()`` and ``_version`` untouched, so a stale split + cannot be detected from inside forward(). + + ``allow_rocm_bf16x3_router_gemm`` is final by the time this runs: the only + caller of ``set_out_dtype`` does so during model construction. + """ + + def process_weights_after_loading(self, layer: torch.nn.Module) -> None: + # The base CPU branch may delete layer.weight, so it must run first; + # nothing breaks today only because platform_supported() is False on + # CPU. + super().process_weights_after_loading(layer) + if not layer.allow_rocm_bf16x3_router_gemm: + return + try: + from vllm.model_executor.layers.fused_moe.router import ( + bf16x3_router_gemm_rocm, + ) + + layer._bf16x3_weight = bf16x3_router_gemm_rocm.split_bf16x3(layer.weight) + except ValueError as exc: + layer.allow_rocm_bf16x3_router_gemm = False + logger.warning_once( + "Disabling ROCm BF16x3 router GEMM for %s: %s", layer.prefix, exc + ) + + _FP32_ROUTER_GEMM_MAX_TOKENS = GateLinear.FP32_MAX_TOKENS +def rocm_bf16x3_router_gemm_dispatch_impl( + x: torch.Tensor, + weight: torch.Tensor, + weight_split: torch.Tensor, +) -> torch.Tensor: + """Run the ROCm bf16x3 router GEMM, or fall back for small batches. + + This must be a custom op because our torch.compile integration does not + support runtime dispatching on num_tokens: models carrying fp32 router + weights (MiniMax-M2, HunYuan-V3) call the gate from + ``@support_torch_compile`` model code and vLLM drops all Dynamo guards, so + a plain Python branch on ``x.shape[0]`` would be frozen at first trace. + """ + from vllm.model_executor.layers.fused_moe.router import ( + bf16x3_router_gemm_rocm, + ) + + if bf16x3_router_gemm_rocm.is_supported(x, weight): + return bf16x3_router_gemm_rocm.bf16x3_router_gemm(x, weight_split) + # Below MIN_TOKENS, hand back to the low-M gfx950 kernel this tier is + # checked ahead of. + if x.shape[0] <= _FP32_ROUTER_GEMM_MAX_TOKENS: + from vllm.model_executor.layers.fused_moe.router.rocm_fp32_router_gemm import ( # noqa: E501 + can_use_rocm_fp32_router_gemm, + rocm_fp32_router_gemm, + ) + + x = x.contiguous() + if can_use_rocm_fp32_router_gemm(x, weight): + return rocm_fp32_router_gemm(x, weight) + return torch.nn.functional.linear(x.float(), weight) + + +def rocm_bf16x3_router_gemm_dispatch_fake( + x: torch.Tensor, + weight: torch.Tensor, + weight_split: torch.Tensor, +) -> torch.Tensor: + return x.new_empty((x.shape[0], weight.shape[0]), dtype=torch.float32) + + +direct_register_custom_op( + op_name="rocm_bf16x3_router_gemm_dispatch", + op_func=rocm_bf16x3_router_gemm_dispatch_impl, + fake_impl=rocm_bf16x3_router_gemm_dispatch_fake, +) + + def fp32_router_gemm_dispatch_impl( x: torch.Tensor, weight: torch.Tensor, diff --git a/vllm/model_executor/layers/fused_moe/utils.py b/vllm/model_executor/layers/fused_moe/utils.py index a98e181cae..c724a7c7cb 100644 --- a/vllm/model_executor/layers/fused_moe/utils.py +++ b/vllm/model_executor/layers/fused_moe/utils.py @@ -84,6 +84,40 @@ def resolve_layer_fused_shared_expert( else (True, None) ) is_fused_shared_expert_enabled = fse_requested and fse_compatible + + # In online shared_expert quantization is used, register it to + # `online_quantization_config.quantized_layers` ahead of weight loading. + if is_fused_shared_expert_enabled and quant_config is not None: + online_quantization_config = quant_config.online_quantization_config + if online_quantization_config is not None: + from vllm.model_executor.layers.linear import ( + LinearBase, + UnquantizedLinearMethod, + ) + + for projection_name in ("gate_up_proj", "down_proj"): + projection_prefix = f"{prefix}.{shared_expert_name}.{projection_name}" + quant_method_metadata = ( + online_quantization_config.resolve_quant_method_cls( + LinearBase, projection_prefix + ) + ) + + if quant_method_metadata is None: + continue + + source, quant_key_str, target_pattern, _, quant_method_cls = ( + quant_method_metadata + ) + + if quant_method_cls in (None, UnquantizedLinearMethod): + continue + + online_quantization_config.quantized_layers[projection_prefix] = ( + source.value, + quant_key_str, + target_pattern, + ) if fse_requested and not is_fused_shared_expert_enabled: logger.warning( "VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS is enabled but " diff --git a/vllm/model_executor/layers/fusion/fused_act_quant.py b/vllm/model_executor/layers/fusion/fused_act_quant.py index a9b14c6170..6261a1bdd0 100644 --- a/vllm/model_executor/layers/fusion/fused_act_quant.py +++ b/vllm/model_executor/layers/fusion/fused_act_quant.py @@ -63,10 +63,10 @@ def _silu_and_mul_fp8_dynamic_block( x: torch.Tensor, linear: LinearBase, group_size: int, quant_key: QuantKey ) -> QuantizedActivation: """SiluAndMul + FP8 dynamic per-block quantization.""" - assert x.ndim == 2, f"Input must be 2D [batch, hidden*2], got {x.shape}" - + assert x.ndim in (2, 3), f"Input must be 2D or 3D, got {x.shape}" d = x.shape[-1] // 2 out_shape = x.shape[:-1] + (d,) + x = x.reshape(-1, x.shape[-1]) num_tokens = x.shape[0] num_groups = d // group_size @@ -102,10 +102,10 @@ def _silu_and_mul_nvfp4_dynamic( x: torch.Tensor, linear: LinearBase ) -> QuantizedActivation: """SiluAndMul + NVFP4 dynamic quantization.""" - assert x.ndim == 2, f"Input must be 2D [batch, hidden*2], got {x.shape}" - + assert x.ndim in (2, 3), f"Input must be 2D or 3D, got {x.shape}" d = x.shape[-1] // 2 out_shape = x.shape[:-1] + (d,) + x = x.reshape(-1, x.shape[-1]) num_tokens = x.shape[0] # NVFP4 packs 2 values into 1 byte diff --git a/vllm/model_executor/layers/fusion/mm_input_norm.py b/vllm/model_executor/layers/fusion/mm_input_norm.py index d125f8d2d2..5de95b401d 100644 --- a/vllm/model_executor/layers/fusion/mm_input_norm.py +++ b/vllm/model_executor/layers/fusion/mm_input_norm.py @@ -152,6 +152,7 @@ def _load_norm_params(model_config: ModelConfig) -> NormParams: for key in ("do_normalize", "do_rescale", "image_mean", "image_std") ): config = model_config.hf_image_processor_config + config = config.get("media_proc_cfg", config) has_norm_params = "image_mean" in config and "image_std" in config do_normalize = bool(config.get("do_normalize", has_norm_params)) @@ -312,19 +313,15 @@ class FusedMMInputNorm(CustomOp): def forward_xpu( self, pixel_values: torch.Tensor, visual_dtype: torch.dtype ) -> torch.Tensor: - """XPU fused custom kernel path. - - On XPU, fuse the whole rescale + normalise into a single custom - kernel. The eager path materializes an fp32 intermediate and then - casts back, which adds device-side compute that cancels the - bandwidth saving of transferring uint8 pixel_values. The fused - kernel reads uint8 directly and writes ``visual_dtype`` in one pass. + """XPU fused custom kernel path for uint8 and Triton kernel path + for others. """ - # The out-of-tree XPU kernel only supports the uint8 input that - # device-side normalisation guarantees; fail loudly otherwise. - assert pixel_values.dtype == torch.uint8, ( - f"xpu_fused_input_norm requires uint8 input, got {pixel_values.dtype}" - ) + if pixel_values.dtype != torch.uint8: + return self.forward_cuda(pixel_values, visual_dtype) + + # For uint8 input, falls to xpu_fused_input_norm from xpu kernels + import vllm._xpu_ops # noqa: F401 + return torch.ops.vllm.xpu_fused_input_norm( pixel_values, self.weight, self.bias, visual_dtype ) diff --git a/vllm/model_executor/layers/indexer_topk.py b/vllm/model_executor/layers/indexer_topk.py index f0da0312e1..02c4280ea0 100644 --- a/vllm/model_executor/layers/indexer_topk.py +++ b/vllm/model_executor/layers/indexer_topk.py @@ -29,12 +29,10 @@ IDX_OOB_FILL_VALUE = -1 try: import vllm._deepselect_C # noqa: F401 (registers torch.ops.deep_select) -except ImportError as e: - from vllm.logger import init_logger - init_logger(__name__).warning( - "Failed to import the DeepSelect extension (vllm._deepselect_C): %s", e - ) + _HAS_DEEP_SELECT_C = True +except ImportError: + _HAS_DEEP_SELECT_C = False @functools.lru_cache(maxsize=1) @@ -196,6 +194,8 @@ class SparseIndexerTopk(torch.nn.Module): failures.append("requires a CUDA platform") elif not self._has_deep_select: failures.append("requires SM100a/SM103a (10.x device family)") + elif not _HAS_DEEP_SELECT_C: + failures.append("vllm._deepselect_C could not be imported") elif not is_deep_select_supported(logits, topk_tokens): failures.append( f"inputs violate DeepSelect's constraints: dtype={logits.dtype}," diff --git a/vllm/model_executor/layers/mamba/checkpoint.py b/vllm/model_executor/layers/mamba/checkpoint.py index feef0afa05..2f3122f870 100644 --- a/vllm/model_executor/layers/mamba/checkpoint.py +++ b/vllm/model_executor/layers/mamba/checkpoint.py @@ -25,6 +25,7 @@ def compute_mamba_prefill_checkpoints( mamba_block_size: int, checkpoint_alignment: int | None, drop_eagle_block: bool, + cache_hit_alignment_tokens: int, ) -> tuple[list[int], list[int]]: """Per-row internal prefill checkpoint offsets and cache block columns. @@ -43,7 +44,7 @@ def compute_mamba_prefill_checkpoints( for seq_len, query_len in zip(seq_lens, query_lens): query_start = seq_len - query_len position = get_mamba_prefill_checkpoint_position( - seq_len, hash_block_size, drop_eagle_block=drop_eagle_block + seq_len, cache_hit_alignment_tokens, drop_eagle_block=drop_eagle_block ) valid = is_mamba_prefill_checkpoint_valid( query_start=query_start, @@ -70,6 +71,9 @@ def _gather_checkpoint_state_indices( class MambaPrefillCheckpointMetadata: checkpoint_offsets: torch.Tensor state_indices: torch.Tensor + # Host copy of ``checkpoint_offsets``, for backends that lay out the + # checkpoint before any tensor exists (Mamba2's SSD chunk split). + offsets: list[int] | None = None # Gather indices into the block table, kept so that another KV cache group # with the same spec can re-derive its own ``state_indices``. request_rows: torch.Tensor | None = None @@ -103,12 +107,17 @@ class MambaPrefillCheckpointBuilder: return None if self.kv_cache_spec.num_prefill_checkpoint_blocks == 0: return None + cache_config = self.vllm_config.cache_config + hash_block_size = cache_config.hash_block_size + cache_hit_alignment_tokens = cache_config.cache_hit_alignment_tokens + if hash_block_size is None or cache_hit_alignment_tokens is None: + # Unresolved only for the minimal KV cache built for profiling. + return None assert m.seq_lens_cpu_upper_bound is not None all_query_lens = m.query_start_loc_cpu.diff().tolist() query_lens = [all_query_lens[row] for row in request_rows] seq_lens = m.seq_lens_cpu_upper_bound.tolist() block_size = self.kv_cache_spec.block_size - hash_block_size = self.vllm_config.cache_config.prefix_match_unit or block_size speculative_config = self.vllm_config.speculative_config drop_eagle_block = ( speculative_config is not None and speculative_config.use_eagle_block_drop() @@ -120,6 +129,7 @@ class MambaPrefillCheckpointBuilder: mamba_block_size=block_size, checkpoint_alignment=self.kv_cache_spec.prefill_checkpoint_alignment, drop_eagle_block=drop_eagle_block, + cache_hit_alignment_tokens=cache_hit_alignment_tokens, ) if not any(checkpoint_offsets): return None @@ -139,6 +149,7 @@ class MambaPrefillCheckpointBuilder: _gather_checkpoint_state_indices( m.block_table_tensor, request_rows_tensor, checkpoint_cols_tensor ), + offsets=checkpoint_offsets, request_rows=request_rows_tensor, block_cols=checkpoint_cols_tensor, ) diff --git a/vllm/model_executor/layers/mamba/gdn/qwen_gdn_linear_attn.py b/vllm/model_executor/layers/mamba/gdn/qwen_gdn_linear_attn.py index 3f13dc19d3..fffa896b86 100644 --- a/vllm/model_executor/layers/mamba/gdn/qwen_gdn_linear_attn.py +++ b/vllm/model_executor/layers/mamba/gdn/qwen_gdn_linear_attn.py @@ -8,6 +8,7 @@ from typing import Literal import torch from einops import rearrange from torch import nn +from transformers import Qwen3NextConfig from vllm import _custom_ops as ops from vllm import envs @@ -58,7 +59,6 @@ from vllm.third_party.flash_linear_attention.ops import ( ) from vllm.third_party.flash_linear_attention.ops.chunk import l2norm_fwd from vllm.third_party.flash_linear_attention.ops.utils import FLA_CHUNK_SIZE -from vllm.transformers_utils.configs.qwen3_next import Qwen3NextConfig from vllm.utils.torch_utils import ( LayerNameType, _encode_layer_name, @@ -92,7 +92,7 @@ FUSED_GDN_STATE_DTYPES = (torch.float32, torch.bfloat16) def _resolve_gdn_prefill_backend( vllm_config: VllmConfig, -) -> tuple[str, Literal["triton", "flashinfer", "cutedsl"]]: +) -> tuple[str, Literal["triton", "flashinfer", "cutedsl", "aiter_flydsl"]]: """Resolve GDN prefill backend. FlashInfer's GDN prefill kernel is chosen when: @@ -106,6 +106,11 @@ def _resolve_gdn_prefill_backend( In-tree CuteDSL GDN prefill kernel is chosen when: * "cutedsl" is requested; (opt-in only) * Blackwell (SM10.x) with ``head_k_dim == 128``; + + AITER FlyDSL GDN prefill kernels are chosen when: + * "aiter_flydsl" is requested; (opt-in only) + * ROCm AITER exposes the optimized VK prefill API and FlyDSL kernels; + * the model uses BF16 activations with ``head_k_dim == head_v_dim == 128``. """ additional_config = vllm_config.additional_config backend_cfg = ( @@ -115,12 +120,50 @@ def _resolve_gdn_prefill_backend( ) backend = str(backend_cfg).strip().lower() - if not current_platform.is_cuda(): - return backend, "triton" - head_k_dim = getattr( vllm_config.model_config.hf_text_config, "linear_key_head_dim", None ) + head_v_dim = getattr( + vllm_config.model_config.hf_text_config, "linear_value_head_dim", None + ) + + if current_platform.is_rocm(): + if backend == "aiter_flydsl": + if not rocm_aiter_ops.is_gdn_flydsl_prefill_available(): + raise RuntimeError( + "GDN prefill backend 'aiter_flydsl' was requested but the " + "AITER FlyDSL prefill kernels are not importable. Check " + "that VLLM_ROCM_USE_AITER=1, that this is a CDNA 3 or " + "newer GPU, and that the installed AITER exports them." + ) + if head_k_dim != 128 or head_v_dim != 128: + logger.warning_once( + "GDN prefill backend 'aiter_flydsl' was requested but " + "linear head dims are K=%s V=%s; FlyDSL requires 128/128. " + "Falling back to Triton/FLA.", + head_k_dim, + head_v_dim, + ) + return backend, "triton" + if vllm_config.model_config.dtype != torch.bfloat16: + logger.warning_once( + "GDN prefill backend 'aiter_flydsl' was requested but " + "model dtype is %s; FlyDSL requires bfloat16. " + "Falling back to Triton/FLA.", + vllm_config.model_config.dtype, + ) + return backend, "triton" + return backend, "aiter_flydsl" + return backend, "triton" + + if backend == "aiter_flydsl": + raise RuntimeError( + "GDN prefill backend 'aiter_flydsl' was requested but it is a ROCm " + f"AITER backend and the current platform is {current_platform.device_name}." + ) + + if not current_platform.is_cuda(): + return backend, "triton" supports_flashinfer = False supports_cutedsl = False @@ -172,6 +215,7 @@ def _log_gdn_backend_decision( chosen = { "flashinfer": "FlashInfer", "cutedsl": "CuteDSL", + "aiter_flydsl": "AITER FlyDSL", "triton": "Triton/FLA", }[active_backend] logger.info_once( @@ -259,6 +303,8 @@ class ChunkGatedDeltaRule(CustomOp): self._forward_method = self.forward_cuda elif active_backend == "cutedsl": self._forward_method = self.forward_cutedsl + elif active_backend == "aiter_flydsl": + self._forward_method = self.forward_aiter_flydsl else: self._forward_method = self.forward_native @@ -276,6 +322,7 @@ class ChunkGatedDeltaRule(CustomOp): chunk_offsets: torch.Tensor | None = None, use_qk_l2norm_in_kernel: bool = True, core_attn_out: torch.Tensor | None = None, + aiter_prefill_metadata: object | None = None, ): o, final_state = fi_chunk_gated_delta_rule( q=q, @@ -308,6 +355,7 @@ class ChunkGatedDeltaRule(CustomOp): chunk_offsets: torch.Tensor | None = None, use_qk_l2norm_in_kernel: bool = True, core_attn_out: torch.Tensor | None = None, + aiter_prefill_metadata: object | None = None, ): return fla_chunk_gated_delta_rule( q=q, @@ -338,6 +386,7 @@ class ChunkGatedDeltaRule(CustomOp): chunk_offsets: torch.Tensor | None = None, use_qk_l2norm_in_kernel: bool = True, core_attn_out: torch.Tensor | None = None, + aiter_prefill_metadata: object | None = None, ): from vllm.model_executor.layers.mamba.ops.gdn_chunk_cutedsl import ( chunk_gated_delta_rule_cutedsl, @@ -367,6 +416,39 @@ class ChunkGatedDeltaRule(CustomOp): final_state = None return o, final_state + def forward_aiter_flydsl( + self, + q: torch.Tensor, + k: torch.Tensor, + v: torch.Tensor, + g: torch.Tensor, + beta: torch.Tensor, + initial_state: torch.Tensor, + output_final_state: bool, + cu_seqlens: torch.Tensor | None = None, + chunk_indices: torch.Tensor | None = None, + chunk_offsets: torch.Tensor | None = None, + use_qk_l2norm_in_kernel: bool = True, + core_attn_out: torch.Tensor | None = None, + aiter_prefill_metadata: object | None = None, + ): + o, final_state = rocm_aiter_ops.gdn_flydsl_prefill( + q=q, + k=k, + v=v, + g=g, + beta=beta, + initial_state=initial_state, + output_final_state=output_final_state, + cu_seqlens=cu_seqlens, + aiter_prefill_metadata=aiter_prefill_metadata, + use_qk_l2norm_in_kernel=use_qk_l2norm_in_kernel, + ) + if core_attn_out is not None: + o_flat = o.reshape(-1) + core_attn_out.reshape(-1)[: o_flat.numel()].copy_(o_flat) + return o, final_state + @PluggableLayer.register("qwen_gated_delta_net_attention") class QwenGatedDeltaNetAttention(GatedDeltaNetAttention): @@ -1193,12 +1275,18 @@ class QwenGatedDeltaNetAttention(GatedDeltaNetAttention): # CuteDSL kernels require metadata chunk_indices = None chunk_offsets = None + aiter_prefill_metadata = None if self.gdn_prefill_backend == "cutedsl": from vllm.model_executor.layers.mamba.ops.gdn_chunk_cutedsl import ( prepare_metadata_cutedsl, ) chunk_indices, chunk_offsets = prepare_metadata_cutedsl(cu_seqlens, T) + elif self.gdn_prefill_backend == "aiter_flydsl": + aiter_prefill_metadata = rocm_aiter_ops.build_gdn_flydsl_prefill_metadata( + [T], + cu_seqlens=cu_seqlens, + ) try: self.chunk_gated_delta_rule( @@ -1213,6 +1301,7 @@ class QwenGatedDeltaNetAttention(GatedDeltaNetAttention): chunk_indices=chunk_indices, chunk_offsets=chunk_offsets, use_qk_l2norm_in_kernel=False, + aiter_prefill_metadata=aiter_prefill_metadata, ) except Exception: logger.warning( @@ -1243,6 +1332,7 @@ class QwenGatedDeltaNetAttention(GatedDeltaNetAttention): cu_seqlens, chunk_indices, chunk_offsets, + aiter_prefill_metadata, ) torch.accelerator.empty_cache() @@ -1634,6 +1724,7 @@ class QwenGatedDeltaNetAttention(GatedDeltaNetAttention): chunk_indices=attn_metadata.chunk_indices, chunk_offsets=attn_metadata.chunk_offsets, use_qk_l2norm_in_kernel=False, + aiter_prefill_metadata=attn_metadata.aiter_prefill_metadata, ) # Init cache ssm_state[prefill_state_indices] = last_recurrent_state.to(ssm_state.dtype) diff --git a/vllm/model_executor/layers/mamba/mamba_mixer2.py b/vllm/model_executor/layers/mamba/mamba_mixer2.py index a67480ebc2..3753153fa3 100644 --- a/vllm/model_executor/layers/mamba/mamba_mixer2.py +++ b/vllm/model_executor/layers/mamba/mamba_mixer2.py @@ -3,11 +3,12 @@ from collections.abc import Sequence +from dataclasses import replace import torch from torch import nn -from vllm.config import CacheConfig, ModelConfig, get_current_vllm_config +from vllm.config import CacheConfig, ModelConfig, VllmConfig, get_current_vllm_config from vllm.config.mamba import MambaBackendEnum from vllm.distributed import ( divide, @@ -39,6 +40,7 @@ from vllm.model_executor.layers.mamba.ops.layernorm_gated import rms_norm_gated from vllm.model_executor.layers.mamba.ops.selective_state_update_replayssm_output_only import ( # noqa: E501 selective_state_update_replayssm_output_only, ) +from vllm.model_executor.layers.mamba.ops.ssd_checkpoint import store_prefill_checkpoint from vllm.model_executor.layers.mamba.ops.ssd_combined import ( mamba_chunk_scan_combined_varlen, ) @@ -66,7 +68,7 @@ from vllm.utils.torch_utils import ( from vllm.v1.attention.backend import AttentionMetadata from vllm.v1.attention.backends.mamba2_attn import Mamba2AttentionMetadata from vllm.v1.attention.backends.registry import MambaAttentionBackendEnum -from vllm.v1.kv_cache_interface import KVCacheGroupSpec +from vllm.v1.kv_cache_interface import KVCacheGroupSpec, KVCacheSpec, MambaSpec logger = init_logger(__name__) @@ -766,6 +768,8 @@ class MambaMixer2(MambaBase, PluggableLayer): query_start_loc_d = attn_metadata.query_start_loc_d num_decodes = attn_metadata.num_decodes num_decode_tokens = attn_metadata.num_decode_tokens + checkpoint_chunk_idx = attn_metadata.checkpoint_chunk_idx + checkpoint_meta = attn_metadata.checkpoint_meta if attn_metadata is None: # V1 profile run -- warm up SSD kernels so that autotuning @@ -842,6 +846,7 @@ class MambaMixer2(MambaBase, PluggableLayer): # NOTE: final output is an in-place update of out tensor assert preallocated_ssm_out_p is not None + has_checkpoints = checkpoint_chunk_idx is not None varlen_states = mamba_chunk_scan_combined_varlen( hidden_states_p.view( num_prefill_tokens, self.num_heads // self.tp_size, self.head_dim @@ -859,6 +864,7 @@ class MambaMixer2(MambaBase, PluggableLayer): cu_chunk_seqlens=cu_chunk_seqlen_p, last_chunk_indices=last_chunk_indices_p, initial_states=initial_states, + return_intermediate_states=has_checkpoints, dt_softplus=True, dt_limit=(0.0, float("inf")), out=preallocated_ssm_out_p.view(num_prefill_tokens, -1, self.head_dim), @@ -869,7 +875,24 @@ class MambaMixer2(MambaBase, PluggableLayer): # - varlen state is a (num_prefills, nheads, headdim, dstate) # tensor assert state_indices_tensor_p is not None - ssm_state[state_indices_tensor_p] = varlen_states + final_states = ( + varlen_states[last_chunk_indices_p] + if has_checkpoints + else varlen_states + ) + if has_checkpoints: + assert checkpoint_meta is not None + assert query_start_loc_p is not None + store_prefill_checkpoint( + checkpoint_meta, + checkpoint_chunk_idx, + x, + conv_state, + varlen_states, + ssm_state, + query_start_loc_p, + ) + ssm_state[state_indices_tensor_p] = final_states if ring_start is not None and self._updates_replayssm_trackers: assert prev_num_accepted is not None reset_replayssm_ring_trackers( @@ -1070,6 +1093,18 @@ class MambaMixer2(MambaBase, PluggableLayer): is_blackwell=self.is_blackwell, ) + def get_kv_cache_spec(self, vllm_config: VllmConfig) -> KVCacheSpec | None: + spec = super().get_kv_cache_spec(vllm_config) + if not isinstance(spec, MambaSpec): + return spec + # The block-boundary state can be exported from inside one forward + # pass, so the scheduler need not split the prefill; every reader + # acts on this only in align mode. The metadata builder puts a chunk + # end exactly on the checkpoint, so any token position works. + return replace( + spec, num_prefill_checkpoint_blocks=1, prefill_checkpoint_alignment=1 + ) + def get_state_dtype(self) -> tuple[torch.dtype, ...]: assert self.model_config is not None assert self.cache_config is not None diff --git a/vllm/model_executor/layers/mamba/ops/configs/selective_state_update/headdim=128,dstate=256,device_name=Intel(R)_Arc(TM)_Pro_B60_Graphics,cache_dtype=float16.json b/vllm/model_executor/layers/mamba/ops/configs/selective_state_update/headdim=128,dstate=256,device_name=Intel(R)_Arc(TM)_Pro_B60_Graphics,cache_dtype=float16.json new file mode 100644 index 0000000000..64087ce74c --- /dev/null +++ b/vllm/model_executor/layers/mamba/ops/configs/selective_state_update/headdim=128,dstate=256,device_name=Intel(R)_Arc(TM)_Pro_B60_Graphics,cache_dtype=float16.json @@ -0,0 +1,51 @@ +{ + "triton_version": "3.8.0", + "8": { + "BLOCK_SIZE_M": 4, + "num_warps": 1 + }, + "16": { + "BLOCK_SIZE_M": 4, + "num_warps": 1 + }, + "32": { + "BLOCK_SIZE_M": 4, + "num_warps": 1 + }, + "64": { + "BLOCK_SIZE_M": 4, + "num_warps": 1 + }, + "128": { + "BLOCK_SIZE_M": 4, + "num_warps": 1 + }, + "256": { + "BLOCK_SIZE_M": 4, + "num_warps": 1 + }, + "512": { + "BLOCK_SIZE_M": 4, + "num_warps": 1 + }, + "1024": { + "BLOCK_SIZE_M": 4, + "num_warps": 1 + }, + "2048": { + "BLOCK_SIZE_M": 4, + "num_warps": 1 + }, + "4096": { + "BLOCK_SIZE_M": 4, + "num_warps": 1 + }, + "8192": { + "BLOCK_SIZE_M": 4, + "num_warps": 1 + }, + "16384": { + "BLOCK_SIZE_M": 4, + "num_warps": 1 + } +} diff --git a/vllm/model_executor/layers/mamba/ops/configs/selective_state_update/headdim=128,dstate=256,device_name=Intel(R)_Arc(TM)_Pro_B60_Graphics,cache_dtype=float32.json b/vllm/model_executor/layers/mamba/ops/configs/selective_state_update/headdim=128,dstate=256,device_name=Intel(R)_Arc(TM)_Pro_B60_Graphics,cache_dtype=float32.json new file mode 100644 index 0000000000..0a378592c5 --- /dev/null +++ b/vllm/model_executor/layers/mamba/ops/configs/selective_state_update/headdim=128,dstate=256,device_name=Intel(R)_Arc(TM)_Pro_B60_Graphics,cache_dtype=float32.json @@ -0,0 +1,51 @@ +{ + "triton_version": "3.8.0", + "8": { + "BLOCK_SIZE_M": 4, + "num_warps": 2 + }, + "16": { + "BLOCK_SIZE_M": 4, + "num_warps": 2 + }, + "32": { + "BLOCK_SIZE_M": 8, + "num_warps": 2 + }, + "64": { + "BLOCK_SIZE_M": 4, + "num_warps": 2 + }, + "128": { + "BLOCK_SIZE_M": 4, + "num_warps": 2 + }, + "256": { + "BLOCK_SIZE_M": 4, + "num_warps": 1 + }, + "512": { + "BLOCK_SIZE_M": 8, + "num_warps": 2 + }, + "1024": { + "BLOCK_SIZE_M": 8, + "num_warps": 2 + }, + "2048": { + "BLOCK_SIZE_M": 4, + "num_warps": 1 + }, + "4096": { + "BLOCK_SIZE_M": 4, + "num_warps": 1 + }, + "8192": { + "BLOCK_SIZE_M": 4, + "num_warps": 1 + }, + "16384": { + "BLOCK_SIZE_M": 4, + "num_warps": 1 + } +} diff --git a/vllm/model_executor/layers/mamba/ops/configs/selective_state_update/headdim=64,dstate=128,device_name=Intel(R)_Arc(TM)_Pro_B60_Graphics,cache_dtype=float16.json b/vllm/model_executor/layers/mamba/ops/configs/selective_state_update/headdim=64,dstate=128,device_name=Intel(R)_Arc(TM)_Pro_B60_Graphics,cache_dtype=float16.json new file mode 100644 index 0000000000..91c6e28fa7 --- /dev/null +++ b/vllm/model_executor/layers/mamba/ops/configs/selective_state_update/headdim=64,dstate=128,device_name=Intel(R)_Arc(TM)_Pro_B60_Graphics,cache_dtype=float16.json @@ -0,0 +1,63 @@ +{ + "triton_version": "3.8.0", + "8": { + "BLOCK_SIZE_M": 16, + "num_warps": 8 + }, + "16": { + "BLOCK_SIZE_M": 64, + "num_warps": 8 + }, + "32": { + "BLOCK_SIZE_M": 64, + "num_warps": 8 + }, + "64": { + "BLOCK_SIZE_M": 64, + "num_warps": 8 + }, + "128": { + "BLOCK_SIZE_M": 64, + "num_warps": 8 + }, + "256": { + "BLOCK_SIZE_M": 64, + "num_warps": 8 + }, + "512": { + "BLOCK_SIZE_M": 4, + "num_warps": 1 + }, + "1024": { + "BLOCK_SIZE_M": 64, + "num_warps": 8 + }, + "2048": { + "BLOCK_SIZE_M": 4, + "num_warps": 1 + }, + "4096": { + "BLOCK_SIZE_M": 64, + "num_warps": 8 + }, + "8192": { + "BLOCK_SIZE_M": 64, + "num_warps": 8 + }, + "16384": { + "BLOCK_SIZE_M": 64, + "num_warps": 8 + }, + "32768": { + "BLOCK_SIZE_M": 64, + "num_warps": 8 + }, + "65536": { + "BLOCK_SIZE_M": 4, + "num_warps": 1 + }, + "131072": { + "BLOCK_SIZE_M": 4, + "num_warps": 1 + } +} diff --git a/vllm/model_executor/layers/mamba/ops/configs/selective_state_update/headdim=64,dstate=128,device_name=Intel(R)_Arc(TM)_Pro_B60_Graphics,cache_dtype=float32.json b/vllm/model_executor/layers/mamba/ops/configs/selective_state_update/headdim=64,dstate=128,device_name=Intel(R)_Arc(TM)_Pro_B60_Graphics,cache_dtype=float32.json new file mode 100644 index 0000000000..c30784ce77 --- /dev/null +++ b/vllm/model_executor/layers/mamba/ops/configs/selective_state_update/headdim=64,dstate=128,device_name=Intel(R)_Arc(TM)_Pro_B60_Graphics,cache_dtype=float32.json @@ -0,0 +1,63 @@ +{ + "triton_version": "3.8.0", + "8": { + "BLOCK_SIZE_M": 8, + "num_warps": 8 + }, + "16": { + "BLOCK_SIZE_M": 8, + "num_warps": 1 + }, + "32": { + "BLOCK_SIZE_M": 4, + "num_warps": 1 + }, + "64": { + "BLOCK_SIZE_M": 4, + "num_warps": 1 + }, + "128": { + "BLOCK_SIZE_M": 4, + "num_warps": 1 + }, + "256": { + "BLOCK_SIZE_M": 4, + "num_warps": 1 + }, + "512": { + "BLOCK_SIZE_M": 4, + "num_warps": 1 + }, + "1024": { + "BLOCK_SIZE_M": 4, + "num_warps": 1 + }, + "2048": { + "BLOCK_SIZE_M": 4, + "num_warps": 1 + }, + "4096": { + "BLOCK_SIZE_M": 4, + "num_warps": 1 + }, + "8192": { + "BLOCK_SIZE_M": 8, + "num_warps": 1 + }, + "16384": { + "BLOCK_SIZE_M": 8, + "num_warps": 1 + }, + "32768": { + "BLOCK_SIZE_M": 8, + "num_warps": 1 + }, + "65536": { + "BLOCK_SIZE_M": 8, + "num_warps": 1 + }, + "131072": { + "BLOCK_SIZE_M": 8, + "num_warps": 1 + } +} diff --git a/vllm/model_executor/layers/mamba/ops/configs/selective_state_update/headdim=64,dstate=256,device_name=Intel(R)_Arc(TM)_Pro_B60_Graphics,cache_dtype=float16.json b/vllm/model_executor/layers/mamba/ops/configs/selective_state_update/headdim=64,dstate=256,device_name=Intel(R)_Arc(TM)_Pro_B60_Graphics,cache_dtype=float16.json new file mode 100644 index 0000000000..5f29bcd76c --- /dev/null +++ b/vllm/model_executor/layers/mamba/ops/configs/selective_state_update/headdim=64,dstate=256,device_name=Intel(R)_Arc(TM)_Pro_B60_Graphics,cache_dtype=float16.json @@ -0,0 +1,63 @@ +{ + "triton_version": "3.8.0", + "8": { + "BLOCK_SIZE_M": 4, + "num_warps": 1 + }, + "16": { + "BLOCK_SIZE_M": 4, + "num_warps": 1 + }, + "32": { + "BLOCK_SIZE_M": 4, + "num_warps": 1 + }, + "64": { + "BLOCK_SIZE_M": 4, + "num_warps": 1 + }, + "128": { + "BLOCK_SIZE_M": 4, + "num_warps": 1 + }, + "256": { + "BLOCK_SIZE_M": 4, + "num_warps": 1 + }, + "512": { + "BLOCK_SIZE_M": 4, + "num_warps": 1 + }, + "1024": { + "BLOCK_SIZE_M": 4, + "num_warps": 1 + }, + "2048": { + "BLOCK_SIZE_M": 4, + "num_warps": 1 + }, + "4096": { + "BLOCK_SIZE_M": 4, + "num_warps": 1 + }, + "8192": { + "BLOCK_SIZE_M": 4, + "num_warps": 1 + }, + "16384": { + "BLOCK_SIZE_M": 4, + "num_warps": 1 + }, + "32768": { + "BLOCK_SIZE_M": 4, + "num_warps": 1 + }, + "65536": { + "BLOCK_SIZE_M": 4, + "num_warps": 1 + }, + "131072": { + "BLOCK_SIZE_M": 4, + "num_warps": 1 + } +} diff --git a/vllm/model_executor/layers/mamba/ops/configs/selective_state_update/headdim=64,dstate=256,device_name=Intel(R)_Arc(TM)_Pro_B60_Graphics,cache_dtype=float32.json b/vllm/model_executor/layers/mamba/ops/configs/selective_state_update/headdim=64,dstate=256,device_name=Intel(R)_Arc(TM)_Pro_B60_Graphics,cache_dtype=float32.json new file mode 100644 index 0000000000..8f815a1b33 --- /dev/null +++ b/vllm/model_executor/layers/mamba/ops/configs/selective_state_update/headdim=64,dstate=256,device_name=Intel(R)_Arc(TM)_Pro_B60_Graphics,cache_dtype=float32.json @@ -0,0 +1,63 @@ +{ + "triton_version": "3.8.0", + "8": { + "BLOCK_SIZE_M": 4, + "num_warps": 2 + }, + "16": { + "BLOCK_SIZE_M": 4, + "num_warps": 2 + }, + "32": { + "BLOCK_SIZE_M": 8, + "num_warps": 2 + }, + "64": { + "BLOCK_SIZE_M": 8, + "num_warps": 2 + }, + "128": { + "BLOCK_SIZE_M": 4, + "num_warps": 2 + }, + "256": { + "BLOCK_SIZE_M": 8, + "num_warps": 2 + }, + "512": { + "BLOCK_SIZE_M": 8, + "num_warps": 2 + }, + "1024": { + "BLOCK_SIZE_M": 8, + "num_warps": 2 + }, + "2048": { + "BLOCK_SIZE_M": 4, + "num_warps": 1 + }, + "4096": { + "BLOCK_SIZE_M": 8, + "num_warps": 2 + }, + "8192": { + "BLOCK_SIZE_M": 8, + "num_warps": 2 + }, + "16384": { + "BLOCK_SIZE_M": 8, + "num_warps": 2 + }, + "32768": { + "BLOCK_SIZE_M": 4, + "num_warps": 1 + }, + "65536": { + "BLOCK_SIZE_M": 4, + "num_warps": 1 + }, + "131072": { + "BLOCK_SIZE_M": 4, + "num_warps": 1 + } +} diff --git a/vllm/model_executor/layers/mamba/ops/configs/selective_state_update/headdim=64,dstate=256,device_name=Intel(R)_Arc(TM)_Pro_B70_Graphics,cache_dtype=float16.json b/vllm/model_executor/layers/mamba/ops/configs/selective_state_update/headdim=64,dstate=256,device_name=Intel(R)_Arc(TM)_Pro_B70_Graphics,cache_dtype=float16.json new file mode 100644 index 0000000000..d51b085769 --- /dev/null +++ b/vllm/model_executor/layers/mamba/ops/configs/selective_state_update/headdim=64,dstate=256,device_name=Intel(R)_Arc(TM)_Pro_B70_Graphics,cache_dtype=float16.json @@ -0,0 +1,47 @@ +{ + "triton_version": "3.8.0", + "48": { + "BLOCK_SIZE_M": 8, + "num_warps": 1 + }, + "96": { + "BLOCK_SIZE_M": 8, + "num_warps": 1 + }, + "192": { + "BLOCK_SIZE_M": 4, + "num_warps": 1 + }, + "384": { + "BLOCK_SIZE_M": 8, + "num_warps": 1 + }, + "768": { + "BLOCK_SIZE_M": 8, + "num_warps": 1 + }, + "1536": { + "BLOCK_SIZE_M": 4, + "num_warps": 1 + }, + "3072": { + "BLOCK_SIZE_M": 4, + "num_warps": 1 + }, + "6144": { + "BLOCK_SIZE_M": 4, + "num_warps": 1 + }, + "12288": { + "BLOCK_SIZE_M": 4, + "num_warps": 1 + }, + "24576": { + "BLOCK_SIZE_M": 4, + "num_warps": 1 + }, + "49152": { + "BLOCK_SIZE_M": 4, + "num_warps": 1 + } +} \ No newline at end of file diff --git a/vllm/model_executor/layers/mamba/ops/configs/selective_state_update/headdim=64,dstate=256,device_name=Intel(R)_Arc(TM)_Pro_B70_Graphics,cache_dtype=float32.json b/vllm/model_executor/layers/mamba/ops/configs/selective_state_update/headdim=64,dstate=256,device_name=Intel(R)_Arc(TM)_Pro_B70_Graphics,cache_dtype=float32.json new file mode 100644 index 0000000000..666506768c --- /dev/null +++ b/vllm/model_executor/layers/mamba/ops/configs/selective_state_update/headdim=64,dstate=256,device_name=Intel(R)_Arc(TM)_Pro_B70_Graphics,cache_dtype=float32.json @@ -0,0 +1,39 @@ +{ + "triton_version": "3.8.0", + "48": { + "BLOCK_SIZE_M": 4, + "num_warps": 2 + }, + "384": { + "BLOCK_SIZE_M": 8, + "num_warps": 2 + }, + "768": { + "BLOCK_SIZE_M": 4, + "num_warps": 2 + }, + "1536": { + "BLOCK_SIZE_M": 4, + "num_warps": 2 + }, + "3072": { + "BLOCK_SIZE_M": 4, + "num_warps": 2 + }, + "6144": { + "BLOCK_SIZE_M": 4, + "num_warps": 2 + }, + "12288": { + "BLOCK_SIZE_M": 8, + "num_warps": 2 + }, + "24576": { + "BLOCK_SIZE_M": 8, + "num_warps": 2 + }, + "49152": { + "BLOCK_SIZE_M": 4, + "num_warps": 2 + } +} \ No newline at end of file diff --git a/vllm/model_executor/layers/mamba/ops/configs/selective_state_update/headdim=64,dstate=64,device_name=Intel(R)_Arc(TM)_Pro_B60_Graphics,cache_dtype=float16.json b/vllm/model_executor/layers/mamba/ops/configs/selective_state_update/headdim=64,dstate=64,device_name=Intel(R)_Arc(TM)_Pro_B60_Graphics,cache_dtype=float16.json new file mode 100644 index 0000000000..2df2525541 --- /dev/null +++ b/vllm/model_executor/layers/mamba/ops/configs/selective_state_update/headdim=64,dstate=64,device_name=Intel(R)_Arc(TM)_Pro_B60_Graphics,cache_dtype=float16.json @@ -0,0 +1,63 @@ +{ + "triton_version": "3.8.0", + "8": { + "BLOCK_SIZE_M": 16, + "num_warps": 4 + }, + "16": { + "BLOCK_SIZE_M": 16, + "num_warps": 4 + }, + "32": { + "BLOCK_SIZE_M": 16, + "num_warps": 4 + }, + "64": { + "BLOCK_SIZE_M": 4, + "num_warps": 1 + }, + "128": { + "BLOCK_SIZE_M": 64, + "num_warps": 8 + }, + "256": { + "BLOCK_SIZE_M": 8, + "num_warps": 1 + }, + "512": { + "BLOCK_SIZE_M": 8, + "num_warps": 1 + }, + "1024": { + "BLOCK_SIZE_M": 8, + "num_warps": 1 + }, + "2048": { + "BLOCK_SIZE_M": 8, + "num_warps": 1 + }, + "4096": { + "BLOCK_SIZE_M": 4, + "num_warps": 1 + }, + "8192": { + "BLOCK_SIZE_M": 8, + "num_warps": 1 + }, + "16384": { + "BLOCK_SIZE_M": 8, + "num_warps": 1 + }, + "32768": { + "BLOCK_SIZE_M": 8, + "num_warps": 1 + }, + "65536": { + "BLOCK_SIZE_M": 8, + "num_warps": 1 + }, + "131072": { + "BLOCK_SIZE_M": 8, + "num_warps": 1 + } +} diff --git a/vllm/model_executor/layers/mamba/ops/configs/selective_state_update/headdim=64,dstate=64,device_name=Intel(R)_Arc(TM)_Pro_B60_Graphics,cache_dtype=float32.json b/vllm/model_executor/layers/mamba/ops/configs/selective_state_update/headdim=64,dstate=64,device_name=Intel(R)_Arc(TM)_Pro_B60_Graphics,cache_dtype=float32.json new file mode 100644 index 0000000000..f943b523c4 --- /dev/null +++ b/vllm/model_executor/layers/mamba/ops/configs/selective_state_update/headdim=64,dstate=64,device_name=Intel(R)_Arc(TM)_Pro_B60_Graphics,cache_dtype=float32.json @@ -0,0 +1,63 @@ +{ + "triton_version": "3.8.0", + "8": { + "BLOCK_SIZE_M": 4, + "num_warps": 1 + }, + "16": { + "BLOCK_SIZE_M": 16, + "num_warps": 8 + }, + "32": { + "BLOCK_SIZE_M": 4, + "num_warps": 1 + }, + "64": { + "BLOCK_SIZE_M": 4, + "num_warps": 1 + }, + "128": { + "BLOCK_SIZE_M": 4, + "num_warps": 1 + }, + "256": { + "BLOCK_SIZE_M": 4, + "num_warps": 1 + }, + "512": { + "BLOCK_SIZE_M": 4, + "num_warps": 1 + }, + "1024": { + "BLOCK_SIZE_M": 4, + "num_warps": 1 + }, + "2048": { + "BLOCK_SIZE_M": 4, + "num_warps": 1 + }, + "4096": { + "BLOCK_SIZE_M": 4, + "num_warps": 1 + }, + "8192": { + "BLOCK_SIZE_M": 4, + "num_warps": 1 + }, + "16384": { + "BLOCK_SIZE_M": 4, + "num_warps": 1 + }, + "32768": { + "BLOCK_SIZE_M": 4, + "num_warps": 1 + }, + "65536": { + "BLOCK_SIZE_M": 4, + "num_warps": 1 + }, + "131072": { + "BLOCK_SIZE_M": 4, + "num_warps": 1 + } +} diff --git a/vllm/model_executor/layers/mamba/ops/configs/selective_state_update/headdim=64,dstate=64,device_name=Intel(R)_Arc(TM)_Pro_B70_Graphics,cache_dtype=float16.json b/vllm/model_executor/layers/mamba/ops/configs/selective_state_update/headdim=64,dstate=64,device_name=Intel(R)_Arc(TM)_Pro_B70_Graphics,cache_dtype=float16.json new file mode 100644 index 0000000000..aed0a3c635 --- /dev/null +++ b/vllm/model_executor/layers/mamba/ops/configs/selective_state_update/headdim=64,dstate=64,device_name=Intel(R)_Arc(TM)_Pro_B70_Graphics,cache_dtype=float16.json @@ -0,0 +1,47 @@ +{ + "triton_version": "3.8.0", + "80": { + "BLOCK_SIZE_M": 16, + "num_warps": 4 + }, + "640": { + "BLOCK_SIZE_M": 64, + "num_warps": 8 + }, + "1280": { + "BLOCK_SIZE_M": 64, + "num_warps": 8 + }, + "2560": { + "BLOCK_SIZE_M": 64, + "num_warps": 8 + }, + "5120": { + "BLOCK_SIZE_M": 32, + "num_warps": 8 + }, + "10240": { + "BLOCK_SIZE_M": 32, + "num_warps": 8 + }, + "20480": { + "BLOCK_SIZE_M": 32, + "num_warps": 8 + }, + "40960": { + "BLOCK_SIZE_M": 32, + "num_warps": 8 + }, + "81920": { + "BLOCK_SIZE_M": 32, + "num_warps": 8 + }, + "122880": { + "BLOCK_SIZE_M": 32, + "num_warps": 8 + }, + "163840": { + "BLOCK_SIZE_M": 32, + "num_warps": 8 + } +} \ No newline at end of file diff --git a/vllm/model_executor/layers/mamba/ops/configs/selective_state_update/headdim=64,dstate=64,device_name=Intel(R)_Arc(TM)_Pro_B70_Graphics,cache_dtype=float32.json b/vllm/model_executor/layers/mamba/ops/configs/selective_state_update/headdim=64,dstate=64,device_name=Intel(R)_Arc(TM)_Pro_B70_Graphics,cache_dtype=float32.json new file mode 100644 index 0000000000..7c642c06bc --- /dev/null +++ b/vllm/model_executor/layers/mamba/ops/configs/selective_state_update/headdim=64,dstate=64,device_name=Intel(R)_Arc(TM)_Pro_B70_Graphics,cache_dtype=float32.json @@ -0,0 +1,47 @@ +{ + "triton_version": "3.8.0", + "80": { + "BLOCK_SIZE_M": 64, + "num_warps": 8 + }, + "640": { + "BLOCK_SIZE_M": 64, + "num_warps": 8 + }, + "1280": { + "BLOCK_SIZE_M": 64, + "num_warps": 8 + }, + "2560": { + "BLOCK_SIZE_M": 8, + "num_warps": 4 + }, + "5120": { + "BLOCK_SIZE_M": 16, + "num_warps": 8 + }, + "10240": { + "BLOCK_SIZE_M": 16, + "num_warps": 8 + }, + "20480": { + "BLOCK_SIZE_M": 16, + "num_warps": 8 + }, + "40960": { + "BLOCK_SIZE_M": 16, + "num_warps": 2 + }, + "81920": { + "BLOCK_SIZE_M": 16, + "num_warps": 2 + }, + "122880": { + "BLOCK_SIZE_M": 16, + "num_warps": 2 + }, + "163840": { + "BLOCK_SIZE_M": 16, + "num_warps": 2 + } +} \ No newline at end of file diff --git a/vllm/model_executor/layers/mamba/ops/ssd_checkpoint.py b/vllm/model_executor/layers/mamba/ops/ssd_checkpoint.py new file mode 100644 index 0000000000..25bfcedbdc --- /dev/null +++ b/vllm/model_executor/layers/mamba/ops/ssd_checkpoint.py @@ -0,0 +1,115 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import torch + +from vllm.model_executor.layers.mamba.checkpoint import ( + MambaPrefillCheckpointMetadata, +) +from vllm.triton_utils import tl, triton + + +@triton.jit +def _store_prefill_checkpoint_kernel( + x_ptr, + conv_state_ptr, + chunk_states_ptr, + ssm_state_ptr, + query_start_loc_ptr, + checkpoint_offsets_ptr, + state_indices_ptr, + chunk_idx_ptr, + stride_x_dim, + stride_x_token, + stride_conv_block, + stride_conv_dim, + stride_conv_token, + stride_chunk_states, + stride_ssm_block, + DIM: tl.constexpr, + STATE_LEN: tl.constexpr, + SSM_ROW_SIZE: tl.constexpr, + BLOCK_SIZE: tl.constexpr, +): + row = tl.program_id(0) + cols = tl.program_id(1) * BLOCK_SIZE + tl.arange(0, BLOCK_SIZE) + checkpoint_offset = tl.load(checkpoint_offsets_ptr + row) + valid = checkpoint_offset > 0 + dst = tl.load(state_indices_ptr + row).to(tl.int64) + + conv_mask = valid & (cols < DIM * STATE_LEN) + dim_idx = cols // STATE_LEN + history_idx = cols % STATE_LEN + token_idx = ( + tl.load(query_start_loc_ptr + row) + checkpoint_offset - STATE_LEN + history_idx + ).to(tl.int64) + conv = tl.load( + x_ptr + dim_idx * stride_x_dim + token_idx * stride_x_token, mask=conv_mask + ) + tl.store( + conv_state_ptr + + dst * stride_conv_block + + dim_idx * stride_conv_dim + + history_idx * stride_conv_token, + conv, + mask=conv_mask, + ) + + ssm_mask = valid & (cols < SSM_ROW_SIZE) + src = tl.load(chunk_idx_ptr + row).to(tl.int64) + ssm = tl.load(chunk_states_ptr + src * stride_chunk_states + cols, mask=ssm_mask) + tl.store(ssm_state_ptr + dst * stride_ssm_block + cols, ssm, mask=ssm_mask) + + +def store_prefill_checkpoint( + checkpoint: MambaPrefillCheckpointMetadata, + chunk_idx: torch.Tensor, + x: torch.Tensor, + conv_state: torch.Tensor, + chunk_states: torch.Tensor, + ssm_state: torch.Tensor, + query_start_loc: torch.Tensor, +) -> None: + """Write each prefill row's mid-prefill checkpoint into its paged block. + + Rows without a checkpoint (offset 0) write nothing. + + Args: + checkpoint: Per-row offsets and destination blocks. + chunk_idx: Per-row index into ``chunk_states`` of the checkpoint. + x: Pre-conv activations, ``(dim, num_prefill_tokens)``. + conv_state: Paged conv state, ``(num_blocks, dim, state_len)``. + chunk_states: SSD states at every chunk end. + ssm_state: Paged SSM state. + query_start_loc: Prefill query start locations. + + """ + dim = x.size(0) + state_len = conv_state.size(-1) + ssm_row_size = chunk_states[0].numel() + block_size = 1024 + grid = ( + chunk_idx.numel(), + triton.cdiv(max(dim * state_len, ssm_row_size), block_size), + ) + _store_prefill_checkpoint_kernel[grid]( + x, + conv_state, + chunk_states, + ssm_state, + query_start_loc, + checkpoint.checkpoint_offsets, + checkpoint.state_indices, + chunk_idx, + x.stride(0), + x.stride(1), + conv_state.stride(0), + conv_state.stride(1), + conv_state.stride(2), + chunk_states.stride(0), + ssm_state.stride(0), + DIM=dim, + STATE_LEN=state_len, + SSM_ROW_SIZE=ssm_row_size, + BLOCK_SIZE=block_size, + ) diff --git a/vllm/model_executor/layers/quantization/base_config.py b/vllm/model_executor/layers/quantization/base_config.py index 1b0457854e..3ec2e728e1 100644 --- a/vllm/model_executor/layers/quantization/base_config.py +++ b/vllm/model_executor/layers/quantization/base_config.py @@ -84,6 +84,13 @@ class QuantizeMethodBase(ABC): """ return + def dequantize_weight(self, layer: nn.Module) -> torch.Tensor: + """Materialize a serialized quantized weight for requantization.""" + raise NotImplementedError( + f"The quantization method {type(self)} does not implement " + "dequantize_weight. Please open an issue." + ) + def method_has_implemented_embedding(method_class: type[QuantizeMethodBase]) -> bool: """Not all quant methods have embedding implemented, so we need to check that @@ -294,13 +301,16 @@ def resolve_quant_method( LinearBase, UnquantizedLinearMethod, ) + from vllm.model_executor.layers.quantization.online.fp8 import OnlineLinearBase base_quant_method = quant_config.get_quant_method(layer, prefix) if quant_config.online_quantization_config is None: + # No online configuration: retain the checkpoint method. return base_quant_method # Online quantization currently supports only LinearBase and RoutedExperts. # Embeddings and ParallelLMHead retain their checkpoint quantization method. if not isinstance(layer, (LinearBase, RoutedExperts)): + # Online quantization only supports linear and routed MoE layers. return base_quant_method quant_config.online_quantization_config.packed_modules_mapping = ( @@ -310,16 +320,36 @@ def resolve_quant_method( base_quant_method, (UnquantizedLinearMethod, UnquantizedFusedMoEMethod) ) online_target = quant_config.online_quantization_config.resolve_quant_method_cls( - layer, prefix + type(layer), prefix ) + if checkpoint_is_quantized: - if online_target is not None: - raise ValueError( - f"Cannot apply requested online quantization {online_target[3]} to " - f"pre-quantized layer {prefix}: {base_quant_method} was already " - "selected by the checkpoint quantization config." + if online_target is None: + # The checkpoint quant method is applied as there is no online override. + return base_quant_method + + if isinstance(layer, RoutedExperts): + raise NotImplementedError( + "Requantizing checkpoint-quantized MoE layers is not supported." ) - return base_quant_method + + online_quant_method = quant_config.online_quantization_config.get_quant_method( + layer, prefix + ) + + assert base_quant_method is not None and not isinstance( + base_quant_method, (UnquantizedLinearMethod, UnquantizedFusedMoEMethod) + ) + + assert isinstance(online_quant_method, OnlineLinearBase) + online_quant_method.set_requantization_source(base_quant_method) + + # The online method dequantizes the checkpoint method before requantizing. + return online_quant_method + if online_target is None: + # The layer is unquantized and no online target applies. return base_quant_method + + # Quantize an unquantized layer with the online method. return quant_config.online_quantization_config.get_quant_method(layer, prefix) diff --git a/vllm/model_executor/layers/quantization/compressed_tensors/compressed_tensors_moe/compressed_tensors_moe_wna16.py b/vllm/model_executor/layers/quantization/compressed_tensors/compressed_tensors_moe/compressed_tensors_moe_wna16.py index fdd37d1203..1e06f47413 100644 --- a/vllm/model_executor/layers/quantization/compressed_tensors/compressed_tensors_moe/compressed_tensors_moe_wna16.py +++ b/vllm/model_executor/layers/quantization/compressed_tensors/compressed_tensors_moe/compressed_tensors_moe_wna16.py @@ -450,10 +450,12 @@ class CompressedTensorsWNA16MoEMethod(CompressedTensorsMoEMethod): if w2_bias is not None: replace_parameter(layer, "w2_bias", w2_bias) - # CPU fused_experts_cpu requires zero points even for symmetric quant. + # CPU fused_experts_cpu and the RDNA3 HIP kernel require zero points + # even for symmetric quant (the oracle synthesizes them). # EMULATION bakes ZP into the dequantized bf16 weights — ZP is None. if ( - not self.symmetric or self.wna16_backend == WNA16MoEBackend.CPU + not self.symmetric + or self.wna16_backend in (WNA16MoEBackend.CPU, WNA16MoEBackend.RDNA3) ) and self.wna16_backend != WNA16MoEBackend.EMULATION: assert w13_qzeros is not None and w2_qzeros is not None replace_parameter(layer, "w13_weight_zero_point", w13_qzeros) diff --git a/vllm/model_executor/layers/quantization/modelopt.py b/vllm/model_executor/layers/quantization/modelopt.py index 272eb457bb..98af93c30f 100644 --- a/vllm/model_executor/layers/quantization/modelopt.py +++ b/vllm/model_executor/layers/quantization/modelopt.py @@ -12,7 +12,7 @@ from torch.nn.parameter import Parameter import vllm.envs as envs from vllm.config import get_current_vllm_config_or_none -from vllm.config.quantization import QuantSpec +from vllm.config.quantization import QuantizationConfigArgs, QuantSpec from vllm.logger import init_logger from vllm.model_executor.kernels.linear import ( init_fp8_linear_kernel, @@ -63,6 +63,9 @@ from vllm.model_executor.layers.quantization.base_config import ( ) from vllm.model_executor.layers.quantization.fp8 import Fp8Config, Fp8MoEMethod from vllm.model_executor.layers.quantization.kv_cache import BaseKVCacheMethod +from vllm.model_executor.layers.quantization.utils.flashinfer_utils import ( + trtllm_nvfp4_hidden_alignment, +) from vllm.model_executor.layers.quantization.utils.fp8_utils import ( FP8_SCALE_SENTINEL, process_fp8_input_tensor_strategy_moe, @@ -76,6 +79,7 @@ from vllm.model_executor.layers.quantization.utils.mxfp8_utils import ( MXFP8_BLOCK_SIZE, MXFP8_SCALE_DTYPE, MXFP8_VALUE_DTYPE, + dequant_mxfp8_to_bf16, ) from vllm.model_executor.layers.quantization.utils.quant_utils import ( FP4_DTYPE, @@ -89,6 +93,7 @@ from vllm.model_executor.layers.quantization.utils.quant_utils import ( kMxfp8Dynamic, kMxfp8Static, kNvfp4Dynamic, + kNvfp4DynamicToken, kNvfp4Static, ) from vllm.model_executor.layers.quantization.utils.w8a8_utils import ( @@ -723,12 +728,18 @@ class ModelOptNvFp4Config(ModelOptQuantConfigBase): kv_cache_quant_algo: str | None = None, exclude_modules: list[str] | None = None, group_size: int = 16, + *, + quantization_args: QuantizationConfigArgs | None = None, ) -> None: if exclude_modules is None: exclude_modules = [] super().__init__(exclude_modules) self.quant_method = quant_method self.is_checkpoint_nvfp4_serialized = is_checkpoint_nvfp4_serialized + moe_spec = quantization_args.moe if quantization_args is not None else None + self.moe_activation_override: QuantKey | None = ( + moe_spec.activation if moe_spec is not None else None + ) if is_checkpoint_nvfp4_serialized: self.group_size = group_size self.kv_cache_quant_algo = kv_cache_quant_algo @@ -814,6 +825,7 @@ class ModelOptNvFp4Config(ModelOptQuantConfigBase): kv_cache_quant_method, exclude_modules, group_size, + quantization_args=original_config.get("_online_quantization_args"), ) @@ -834,18 +846,36 @@ class ModelOptNvFp4FusedMoE(FusedMoEMethodBase): ) -> None: super().__init__(moe_config) self.quant_config = quant_config - # W4A16 mode fires for W4A16_NVFP4 on-disk checkpoints. With - # activation_key=None every W4A4 backend's _supports_quant_scheme - # rejects itself (they all require (kNvfp4Static, kNvfp4Dynamic) - # exactly); only Marlin survives. Marlin's MoE path drops - # activation scales in convert_to_nvfp4_moe_kernel_format, so no - # other change is needed. self.use_a16 = quant_config.quant_method == "W4A16_NVFP4" + activation_key = None if self.use_a16 else kNvfp4Dynamic + + if quant_config.moe_activation_override is not None: + if self.use_a16: + raise ValueError( + "NVFP4 activation overrides require a W4A4 checkpoint." + ) + activation_key = quant_config.moe_activation_override + if activation_key not in (kNvfp4Dynamic, kNvfp4DynamicToken): + raise ValueError( + "Unsupported ModelOpt NVFP4 MoE activation override: " + f"{activation_key}. Use quantization_config.moe.activation=" + "'nvfp4_per_token' or omit the override." + ) + self.per_token_activation = activation_key == kNvfp4DynamicToken + self.nvfp4_backend, self.experts_cls = select_nvfp4_moe_backend( config=self.moe, weight_key=kNvfp4Static, - activation_key=None if self.use_a16 else kNvfp4Dynamic, + activation_key=activation_key, ) + if ( + self.per_token_activation + and self.nvfp4_backend != NvFp4MoeBackend.FLASHINFER_TRTLLM + ): + raise ValueError( + "Per-token NVFP4 activation for pre-quantized weights " + "requires the FlashInfer TRTLLM MoE backend." + ) self.use_global_sf = is_global_sf_supported_for_nvfp4_backend( self.nvfp4_backend @@ -983,6 +1013,10 @@ class ModelOptNvFp4FusedMoE(FusedMoEMethodBase): self._build_moe_kernel(layer) return + if self.per_token_activation: + layer.w13_input_scale.data.fill_(1.0) + layer.w2_input_scale.data.fill_(1.0) + # Use a single gscale for w13. if self.moe.is_act_and_mul and not torch.allclose( layer.w13_weight_scale_2[:, 0], layer.w13_weight_scale_2[:, 1] @@ -1015,6 +1049,10 @@ class ModelOptNvFp4FusedMoE(FusedMoEMethodBase): a2_scale=layer.w2_input_scale, is_act_and_mul=self.moe.is_act_and_mul, use_a16=self.use_a16, + trtllm_hidden_alignment=trtllm_nvfp4_hidden_alignment( + per_token_activation=self.per_token_activation, + is_act_and_mul=self.moe.is_act_and_mul, + ), ) replace_parameter(layer, "w13_weight", w13) @@ -1038,6 +1076,7 @@ class ModelOptNvFp4FusedMoE(FusedMoEMethodBase): experts_cls=self.experts_cls, backend=self.nvfp4_backend, routing_tables=layer._expert_routing_tables(), + per_token_activation=self.per_token_activation, ) self.moe_kernel.fused_experts.process_weights_after_loading(layer) @@ -1587,6 +1626,7 @@ class ModelOptMixedPrecisionConfig(ModelOptQuantConfigBase): group_size: int | None, **kwargs: Any, ) -> "ModelOptMixedPrecisionConfig": + quantization_args = original_config.get("_online_quantization_args") if "quantization" in original_config: quantized_layers = original_config["quantization"].get( "quantized_layers", {} @@ -1625,6 +1665,7 @@ class ModelOptMixedPrecisionConfig(ModelOptQuantConfigBase): kv_cache_quant_algo=kv_cache_quant_method, exclude_modules=[], group_size=group_size, + quantization_args=quantization_args, ) # Sibling config for layers that declare quant_algo: "W4A16_NVFP4". # get_quant_method resolves this sub-config to the (kNvfp4Static, None) @@ -1637,6 +1678,7 @@ class ModelOptMixedPrecisionConfig(ModelOptQuantConfigBase): kv_cache_quant_algo=kv_cache_quant_method, exclude_modules=[], group_size=group_size, + quantization_args=quantization_args, ) mxfp8_config = ModelOptMxFp8Config( @@ -2572,6 +2614,17 @@ class ModelOptLinearMethod(LinearMethodBase): layer._nvfp4_group_size_for_gather = self.ctx.group_size self.kernel.process_weights_after_loading(layer) + def dequantize_weight(self, layer: torch.nn.Module) -> torch.Tensor: + """Reconstruct serialized weights for online requantization.""" + if self.wkey.key is kMxfp8Static: + return dequant_mxfp8_to_bf16( + layer.weight.contiguous(), layer.weight_scale.contiguous() + ) + else: + raise NotImplementedError( + "ModelOpt weight dequantization is only supported for MXFP8." + ) + def apply(self, layer, x, bias=None): return self.fmt.apply( layer, diff --git a/vllm/model_executor/layers/quantization/online/base.py b/vllm/model_executor/layers/quantization/online/base.py index 8ed48b2b64..511714837f 100644 --- a/vllm/model_executor/layers/quantization/online/base.py +++ b/vllm/model_executor/layers/quantization/online/base.py @@ -1,7 +1,7 @@ # SPDX-License-Identifier: Apache-2.0 # SPDX-FileCopyrightText: Copyright contributors to the vLLM project -from collections.abc import Mapping +from collections.abc import Callable, Mapping from enum import Enum from types import MappingProxyType from typing import Any @@ -60,6 +60,7 @@ from vllm.model_executor.layers.quantization.utils.config_utils import ( find_matching_patterns, get_layer_name_after_index, ) +from vllm.model_executor.layers.quantization.utils.mxfp4_utils import mxfp4_quantize from vllm.model_executor.layers.quantization.utils.quant_utils import ( QuantKey, kFp8Static128BlockSym, @@ -104,6 +105,15 @@ _ONLINE_MOE_METHODS: dict[QuantKey, type] = { kNvfp4Static: Nvfp4OnlineMoEMethod, } +# Quantizers for full-precision fused shared-expert weights, keyed by the routed +# experts' weight QuantKey. Each shard is quantized to the checkpoint layout on +# arrival, so only block-local schemes are supported. +ONLINE_SHARED_EXPERT_QUANTIZERS: dict[ + QuantKey, Callable[[torch.Tensor], tuple[torch.Tensor, torch.Tensor]] +] = { + kMxfp4Static: mxfp4_quantize, +} + def _find_matching_targets( prefix: str, @@ -156,6 +166,7 @@ class OnlineQuantizationConfig(QuantizationConfig): self.args = args self.ignored_layers: list[str] = args.ignore self.quantized_layers: dict[str, tuple[str, str, str | None]] = {} + self.online_quantization_time = 0.0 @property def quantized_layer_summaries(self) -> list[str]: @@ -216,14 +227,14 @@ class OnlineQuantizationConfig(QuantizationConfig): self, spec: QuantSpec | None, table: dict[QuantKey, type], - layer: torch.nn.Module, + layer_cls: type[torch.nn.Module], ) -> type | None: """Resolve the online method class for a layer's quantization spec. Args: spec: Quantization specification to resolve. table: Mapping from weight quantization keys to method classes. - layer: Layer that will use the resolved method. + layer_cls: Type of the layer that will use the resolved method. Returns: The matching method class, or None when ``spec`` has no weight @@ -238,7 +249,7 @@ class OnlineQuantizationConfig(QuantizationConfig): cls = table.get(spec.weight) if cls is None: raise ValueError( - f"online quantization for {type(layer).__name__} with " + f"online quantization for {layer_cls.__name__} with " f"weight={spec.weight} is not supported; supported weight " f"keys: {sorted(str(k) for k in table)}" ) @@ -253,12 +264,12 @@ class OnlineQuantizationConfig(QuantizationConfig): return cls def resolve_quant_method_cls( - self, layer: torch.nn.Module, prefix: str + self, layer_cls: type[torch.nn.Module], prefix: str ) -> tuple[OnlineQuantizationSource, str, str | None, QuantSpec, type] | None: """Resolve quantization metadata and method class without instantiating it. Args: - layer: Layer for which to resolve online quantization. + layer_cls: Type of layer for which to resolve online quantization. prefix: Fully qualified layer name. Returns: @@ -270,17 +281,17 @@ class OnlineQuantizationConfig(QuantizationConfig): quant_spec: QuantSpec | None if self.args.targets is not None: resolved_pattern = self._resolve_targets_quant_method_metadata( - prefix, layer + prefix, layer_cls ) if resolved_pattern is None: return None source, quant_key_str, target_pattern, quant_spec, table = resolved_pattern else: - if isinstance(layer, LinearBase): + if issubclass(layer_cls, LinearBase): source = OnlineQuantizationSource.linear quant_spec = self.args.linear table = _ONLINE_LINEAR_METHODS - elif isinstance(layer, RoutedExperts): + elif issubclass(layer_cls, RoutedExperts): source = OnlineQuantizationSource.moe quant_spec = self.args.moe table = _ONLINE_MOE_METHODS @@ -297,14 +308,14 @@ class OnlineQuantizationConfig(QuantizationConfig): quant_key_str = str(quant_spec) target_pattern = None - quant_method_cls = self._get_method_cls(quant_spec, table, layer) + quant_method_cls = self._get_method_cls(quant_spec, table, layer_cls) if quant_method_cls is None: return None assert quant_spec is not None return source, quant_key_str, target_pattern, quant_spec, quant_method_cls def _resolve_targets_quant_method_metadata( - self, prefix: str, layer: torch.nn.Module + self, prefix: str, layer_cls: type[torch.nn.Module] ) -> ( tuple[OnlineQuantizationSource, str, str, QuantSpec, dict[QuantKey, type]] | None @@ -313,7 +324,7 @@ class OnlineQuantizationConfig(QuantizationConfig): Args: prefix: Fully qualified layer name. - layer: Layer matched against configured target patterns. + layer_cls: Type of layer matched against configured target patterns. Returns: A tuple of source, quantization key string, target pattern, spec, @@ -348,22 +359,22 @@ class OnlineQuantizationConfig(QuantizationConfig): target_pattern = matches[0] quant_key_str = self.args.targets[target_pattern] shorthand = _ONLINE_SHORTHANDS[quant_key_str] - if isinstance(layer, LinearBase): + if issubclass(layer_cls, LinearBase): quant_spec = shorthand.linear table = _ONLINE_LINEAR_METHODS - elif isinstance(layer, RoutedExperts): + elif issubclass(layer_cls, RoutedExperts): quant_spec = shorthand.moe table = _ONLINE_MOE_METHODS else: raise ValueError( f"Layer {prefix} was matched by quantization_config.targets " f"({target_pattern}), but online quantization is not supported for " - f"{type(layer).__name__}." + f"{layer_cls.__name__}." ) if quant_spec is None: raise ValueError( f"targets pattern {target_pattern} = {quant_key_str} does " - f"not define a QuantSpec for {type(layer).__name__} layers " + f"not define a QuantSpec for {layer_cls.__name__} layers " f"(matched at {prefix})." ) return ( @@ -378,7 +389,7 @@ class OnlineQuantizationConfig(QuantizationConfig): self, layer: torch.nn.Module, prefix: str ) -> "QuantizeMethodBase | None": # `targets` takes precedence over `moe` and `linear` and is exclusive. - resolved = self.resolve_quant_method_cls(layer, prefix) + resolved = self.resolve_quant_method_cls(type(layer), prefix) if resolved is not None: source, quant_key_str, target_pattern, _, quant_method_cls = resolved self.quantized_layers[prefix] = ( diff --git a/vllm/model_executor/layers/quantization/online/fp8.py b/vllm/model_executor/layers/quantization/online/fp8.py index ed018c81bf..f4662ad5a6 100644 --- a/vllm/model_executor/layers/quantization/online/fp8.py +++ b/vllm/model_executor/layers/quantization/online/fp8.py @@ -12,8 +12,6 @@ if TYPE_CHECKING: FusedMoEQuantConfig, ) from vllm.model_executor.layers.fused_moe.oracle.fp8 import Fp8MoeBackend - from vllm.model_executor.layers.quantization.utils.quant_utils import QuantKey - import vllm.envs as envs from vllm import _custom_ops as ops from vllm.config import get_current_vllm_config @@ -34,11 +32,13 @@ from vllm.model_executor.layers.fusion.quant_activation import ( from vllm.model_executor.layers.linear import ( LinearMethodBase, ) +from vllm.model_executor.layers.quantization.base_config import QuantizeMethodBase from vllm.model_executor.layers.quantization.online.moe_base import ( OnlineMoEMethodBase, ) from vllm.model_executor.layers.quantization.utils.quant_utils import ( GroupShape, + QuantKey, amax_for_moe_weight_quant, amax_for_tp_weight_quant, create_fp8_quant_key, @@ -121,10 +121,18 @@ class OnlineLinearBase(LinearMethodBase): weights onto meta device and materializes them just-in-time.""" uses_meta_device: bool = True + activation_quant_key: QuantKey | None def __init__(self): self.out_dtype = torch.get_default_dtype() self.input_dtype = get_current_vllm_config().model_config.dtype + self.requantization_source: QuantizeMethodBase | None = None + self.requantization_source_parameters: dict[str, torch.nn.Parameter] = {} + + def set_requantization_source(self, source_method: QuantizeMethodBase) -> None: + """Configure serialized-weight conversion before online quantization.""" + self.requantization_source = source_method + self.uses_meta_device = False def create_weights( self, @@ -136,6 +144,24 @@ class OnlineLinearBase(LinearMethodBase): params_dtype: torch.dtype, **extra_weight_attrs, ): + if self.requantization_source is not None: + existing_parameter_names = set(layer._parameters) + self.requantization_source.create_weights( + layer, + input_size_per_partition, + output_partition_sizes, + input_size, + output_size, + params_dtype, + **extra_weight_attrs, + ) + self.requantization_source_parameters = { + name: parameter + for name, parameter in layer._parameters.items() + if name not in existing_parameter_names and parameter is not None + } + return + output_size_per_partition = sum(output_partition_sizes) weight_loader = extra_weight_attrs.get("weight_loader") layer.logical_widths = output_partition_sizes @@ -159,11 +185,28 @@ class OnlineLinearBase(LinearMethodBase): initialize_online_processing(layer) + def release_requantization_source_weights(self, layer: torch.nn.Module) -> None: + """Release checkpoint parameters after successful requantization.""" + for name, source_parameter in self.requantization_source_parameters.items(): + if layer._parameters.get(name) is source_parameter: + delattr(layer, name) + self.requantization_source_parameters.clear() + + def get_weight_for_quantization(self, layer: Module) -> torch.Tensor: + """Return checkpoint weights materialized for online quantization.""" + if self.requantization_source is None: + return layer.weight + return self.requantization_source.dequantize_weight(layer) + class Fp8PerTensorOnlineLinearMethod(OnlineLinearBase): """Online tensorwise FP8 linear quantization. Loads fp16/bf16 weights and quantizes them per-tensor during loading.""" + activation_quant_key = ( + kFp8DynamicTokenSym if cutlass_fp8_supported() else kFp8DynamicTensorSym + ) + def __init__(self): super().__init__() @@ -172,11 +215,6 @@ class Fp8PerTensorOnlineLinearMethod(OnlineLinearBase): self.use_marlin = False self.marlin_input_dtype = None self.weight_quant_key = kFp8StaticTensorSym - # Use per-token quantization for better perf if dynamic and cutlass - if cutlass_fp8_supported(): - self.activation_quant_key = kFp8DynamicTokenSym - else: - self.activation_quant_key = kFp8DynamicTensorSym def create_weights( self, @@ -198,6 +236,8 @@ class Fp8PerTensorOnlineLinearMethod(OnlineLinearBase): **extra_weight_attrs, ) + # TODO: init_fp8_linear_kernel should support activation_quant_key=None + assert self.activation_quant_key is not None self.fp8_linear = init_fp8_linear_kernel( activation_quant_key=self.activation_quant_key, weight_quant_key=self.weight_quant_key, @@ -213,10 +253,11 @@ class Fp8PerTensorOnlineLinearMethod(OnlineLinearBase): return layer.input_scale = None - amax = weight_amax(layer.weight).reshape(1) + weight = self.get_weight_for_quantization(layer) + amax = weight_amax(weight).reshape(1) amax = amax_for_tp_weight_quant(amax, _is_tp_sharded(layer)) weight_scale = _fp8_scale(amax) - qweight, _ = ops.scaled_fp8_quant(layer.weight, scale=weight_scale) + qweight, _ = ops.scaled_fp8_quant(weight, scale=weight_scale) # Update layer with new values. replace_parameter(layer, "weight", qweight.t().data) @@ -265,13 +306,11 @@ class Fp8PerBlockOnlineLinearMethod(OnlineLinearBase): """Online blockwise FP8 linear quantization. Loads fp16/bf16 weights and quantizes them per-block during loading.""" + activation_quant_key = kFp8Dynamic128Sym + def __init__(self): super().__init__() self.weight_block_size = [128, 128] - self.activation_quant_key = create_fp8_quant_key( - static=False, - group_shape=GroupShape(1, self.weight_block_size[0]), - ) self.weight_quant_key = create_fp8_quant_key( static=True, group_shape=GroupShape(*self.weight_block_size) ) @@ -297,6 +336,8 @@ class Fp8PerBlockOnlineLinearMethod(OnlineLinearBase): ) layer.weight_block_size = self.weight_block_size + # TODO: init_fp8_linear_kernel should support activation_quant_key=None + assert self.activation_quant_key is not None self.fp8_linear = init_fp8_linear_kernel( activation_quant_key=self.activation_quant_key, weight_quant_key=self.weight_quant_key, @@ -312,14 +353,18 @@ class Fp8PerBlockOnlineLinearMethod(OnlineLinearBase): layer.input_scale = None block_size = self.weight_block_size + weight = self.get_weight_for_quantization(layer) qweight, weight_scale_inv = per_block_cast_to_fp8( - layer.weight, block_size=block_size, use_ue8m0=False + weight, block_size=block_size, use_ue8m0=False ) replace_parameter(layer, "weight", qweight.data) replace_parameter(layer, "weight_scale_inv", weight_scale_inv.data) + if self.requantization_source is not None and hasattr(layer, "weight_scale"): + del layer.weight_scale + self.fp8_linear.process_weights_after_loading(layer) # Prevent duplicate processing (e.g., during weight reload) @@ -395,18 +440,20 @@ class Fp8PtpcOnlineLinearMethod(OnlineLinearBase): return layer.input_scale = None - amax = weight_amax(layer.weight, dim=-1, keepdim=True) + weight = self.get_weight_for_quantization(layer) + amax = weight_amax(weight, dim=-1, keepdim=True) amax = amax_for_tp_weight_quant( amax, _is_tp_sharded(layer, reduces_output_dim=False) ) weight_scale = _fp8_channel_scale(amax) - qweight = _fp8_quant_per_channel(layer.weight, weight_scale) + qweight = _fp8_quant_per_channel(weight, weight_scale) replace_parameter(layer, "weight", qweight.t()) replace_parameter(layer, "weight_scale", weight_scale) self.fp8_linear.process_weights_after_loading(layer) expose_input_quant_key(layer, self.fp8_linear) + self.release_requantization_source_weights(layer) layer._already_called_process_weights_after_loading = True diff --git a/vllm/model_executor/layers/quantization/online/moe_base.py b/vllm/model_executor/layers/quantization/online/moe_base.py index 6e21bc9092..e296eab373 100644 --- a/vllm/model_executor/layers/quantization/online/moe_base.py +++ b/vllm/model_executor/layers/quantization/online/moe_base.py @@ -11,6 +11,7 @@ from vllm.model_executor.layers.fused_moe import ( SharedExperts, ) from vllm.model_executor.layers.fused_moe.moe_output import UnfinalizedMoEOutput +from vllm.model_executor.layers.quantization.base_config import QuantizeMethodBase from vllm.model_executor.model_loader.reload.layerwise import ( initialize_online_processing, ) @@ -24,6 +25,12 @@ class OnlineMoEMethodBase(FusedMoEMethodBase): uses_meta_device: bool = True + def set_requantization_source(self, source_method: QuantizeMethodBase) -> None: + """Reject requantization from a checkpoint-quantized MoE method.""" + raise NotImplementedError( + "Requantizing checkpoint-quantized MoE layers is not supported." + ) + def create_weights( self, layer: torch.nn.Module, diff --git a/vllm/model_executor/layers/quantization/online/mxfp4.py b/vllm/model_executor/layers/quantization/online/mxfp4.py index 72914a9726..b949c9b7ff 100644 --- a/vllm/model_executor/layers/quantization/online/mxfp4.py +++ b/vllm/model_executor/layers/quantization/online/mxfp4.py @@ -119,7 +119,8 @@ class Mxfp4OnlineLinearMethod(OnlineLinearBase): if getattr(layer, "_already_called_process_weights_after_loading", False): return - weight_fp4, weight_scale = mxfp4_quantize(layer.weight.contiguous()) + weight = self.get_weight_for_quantization(layer) + weight_fp4, weight_scale = mxfp4_quantize(weight.contiguous()) layer.input_scale = None replace_parameter(layer, "weight", weight_fp4.data) diff --git a/vllm/model_executor/layers/quantization/online/mxfp8.py b/vllm/model_executor/layers/quantization/online/mxfp8.py index 9d5814206c..ecc196e9f1 100644 --- a/vllm/model_executor/layers/quantization/online/mxfp8.py +++ b/vllm/model_executor/layers/quantization/online/mxfp8.py @@ -9,6 +9,7 @@ import torch from torch.nn import Module from vllm.model_executor.layers.fused_moe.config import FusedMoEConfig +from vllm.model_executor.layers.quantization.utils.quant_utils import QuantKey if TYPE_CHECKING: import vllm.model_executor.layers.fused_moe.modular_kernel as mk @@ -42,6 +43,8 @@ class Mxfp8OnlineLinearMethod(OnlineLinearBase): FP8 with block-32 scales) during weight loading. """ + activation_quant_key: QuantKey | None = None + def create_weights( self, layer: torch.nn.Module, @@ -75,7 +78,8 @@ class Mxfp8OnlineLinearMethod(OnlineLinearBase): if getattr(layer, "_already_called_process_weights_after_loading", False): return - weight_fp8, weight_scale = mxfp8_e4m3_quantize(layer.weight.contiguous()) + weight = self.get_weight_for_quantization(layer) + weight_fp8, weight_scale = mxfp8_e4m3_quantize(weight.contiguous()) layer.input_scale = None replace_parameter(layer, "weight", weight_fp8.data) diff --git a/vllm/model_executor/layers/quantization/online/nvfp4.py b/vllm/model_executor/layers/quantization/online/nvfp4.py index efc97705c5..6c780c2c46 100644 --- a/vllm/model_executor/layers/quantization/online/nvfp4.py +++ b/vllm/model_executor/layers/quantization/online/nvfp4.py @@ -19,6 +19,9 @@ from vllm.model_executor.layers.fused_moe.oracle.nvfp4 import ( from vllm.model_executor.layers.quantization.online.moe_base import ( OnlineMoEMethodBase, ) +from vllm.model_executor.layers.quantization.utils.flashinfer_utils import ( + trtllm_nvfp4_hidden_alignment, +) from vllm.model_executor.layers.quantization.utils.nvfp4_emulation_utils import ( FLOAT4_E2M1_MAX, ) @@ -87,6 +90,8 @@ class Nvfp4OnlineMoEMethod(OnlineMoEMethodBase): (SM100) only. """ + activation_quant_key = kNvfp4DynamicToken + def __init__( self, *, @@ -100,9 +105,13 @@ class Nvfp4OnlineMoEMethod(OnlineMoEMethodBase): self.nvfp4_backend, self.experts_cls = select_nvfp4_moe_backend( config=self.moe, weight_key=kNvfp4Static, - activation_key=kNvfp4DynamicToken, + activation_key=self.activation_quant_key, ) + @property + def per_token_activation(self) -> bool: + return self.activation_quant_key == kNvfp4DynamicToken + def process_weights_after_loading(self, layer: Module) -> None: if getattr(layer, "_already_called_process_weights_after_loading", False): return @@ -156,6 +165,10 @@ class Nvfp4OnlineMoEMethod(OnlineMoEMethodBase): w2_scale_2=layer.w2_weight_scale_2, a2_scale=layer.w2_input_scale, is_act_and_mul=self.moe.is_act_and_mul, + trtllm_hidden_alignment=trtllm_nvfp4_hidden_alignment( + per_token_activation=self.per_token_activation, + is_act_and_mul=self.moe.is_act_and_mul, + ), ) replace_parameter(layer, "w13_weight", w13) @@ -176,7 +189,7 @@ class Nvfp4OnlineMoEMethod(OnlineMoEMethodBase): experts_cls=self.experts_cls, backend=self.nvfp4_backend, routing_tables=layer._expert_routing_tables(), - per_token_activation=True, + per_token_activation=self.per_token_activation, ) else: # Reload creates new scale tensors; derived kernel scales must use diff --git a/vllm/model_executor/layers/quantization/utils/config_utils.py b/vllm/model_executor/layers/quantization/utils/config_utils.py index eb50ed1a23..6e294adb3a 100644 --- a/vllm/model_executor/layers/quantization/utils/config_utils.py +++ b/vllm/model_executor/layers/quantization/utils/config_utils.py @@ -8,9 +8,13 @@ from typing import TYPE_CHECKING import regex as re +from vllm.logger import init_logger + if TYPE_CHECKING: from vllm.model_executor.layers.quantization.base_config import QuantizationConfig +logger = init_logger(__name__) + def get_quark_ocp_mx_group_size( quant_config: "QuantizationConfig | None", @@ -70,18 +74,100 @@ def is_shared_expert_quant_fse_compatible( A compatibility flag and, when incompatible, the reason. """ + from vllm.model_executor.layers.quantization.fp8 import Fp8Config + from vllm.model_executor.layers.quantization.online.fp8 import OnlineLinearBase + from vllm.model_executor.layers.quantization.quark.quark import QuarkConfig + from vllm.model_executor.layers.quantization.utils.quant_utils import ( + is_layer_skipped, + ) + from vllm.models.deepseek_v4.quant_config import DeepseekV4FP8Config + if projection_names is None: projection_names = ["gate_up_proj", "down_proj"] if quant_config is None: return True, None - from vllm.model_executor.layers.quantization.fp8 import Fp8Config - from vllm.model_executor.layers.quantization.quark.quark import QuarkConfig - from vllm.model_executor.layers.quantization.utils.quant_utils import ( - is_layer_skipped, - ) - from vllm.models.deepseek_v4.quant_config import DeepseekV4FP8Config + online_quant_config = quant_config.online_quantization_config + shared_expert_targets = None + if online_quant_config is not None: + from vllm.model_executor.layers.linear import LinearBase + + online_quant_config.packed_modules_mapping = quant_config.packed_modules_mapping + shared_expert_targets = [ + online_quant_config.resolve_quant_method_cls( + LinearBase, f"{shared_expert_prefix}.{projection_name}" + ) + for projection_name in projection_names + ] + if all(target is None for target in shared_expert_targets): + shared_expert_targets = None + + if shared_expert_targets is not None: + from vllm.model_executor.layers.fused_moe import RoutedExperts + from vllm.model_executor.layers.linear import UnquantizedLinearMethod + from vllm.model_executor.layers.quantization.online.base import ( + ONLINE_SHARED_EXPERT_QUANTIZERS, + ) + + # NOTE: online shared experts quantization check is only implemented for quark + # quant method at the moment. This can be extended here for other quant methods. + # Requirement is: + # - Retrieve routed weight quant key + # - Retrieve routed activation quant key + # and ensure the online quantization set on shared expert match them. + if isinstance(quant_config, QuarkConfig): + routed_weight_key, routed_activation_key, _ = ( + quant_config.get_quant_method_target(expert_prefix, RoutedExperts) + ) + else: + reason = ( + "online quantization FSE compatibility check is not implemented for " + f"{type(quant_config).__name__}" + ) + logger.warning_once(reason) + return False, reason + + if routed_weight_key not in ONLINE_SHARED_EXPERT_QUANTIZERS: + return ( + False, + "online shared-expert quantization is not supported for routed " + f"expert weight key {routed_weight_key}", + ) + + for shared_expert_target in shared_expert_targets: + if shared_expert_target is None: + return ( + False, + "online quantization targets only part of the shared expert at " + f"{shared_expert_prefix}", + ) + + _, _, _, shared_quant_spec, shared_method_cls = shared_expert_target + if shared_method_cls in (None, UnquantizedLinearMethod): + return ( + False, + "online quantization excludes FSE weights at " + f"{shared_expert_prefix}", + ) + assert issubclass(shared_method_cls, OnlineLinearBase) + + # NOTE: We can not rely on shared_quant_spec.activation_quant_key + # as _ONLINE_SHORTHANDS: dict[str, QuantSpec] in quantization.py + # does not define the default activation quant key. + # `online_quant_config.resolve_quant_method_cls` should return the + # activation quant key as well. Only some online methods define it + # on the class, so compare the weight key first. + if ( + shared_quant_spec.weight != routed_weight_key + or shared_method_cls.activation_quant_key != routed_activation_key + ): + return ( + False, + "online shared-expert quantization keys do not match the " + "routed expert keys", + ) + return True, None if isinstance(quant_config, DeepseekV4FP8Config): from vllm.config import get_current_vllm_config diff --git a/vllm/model_executor/layers/quantization/utils/flashinfer_fp4_moe.py b/vllm/model_executor/layers/quantization/utils/flashinfer_fp4_moe.py index 24b95ef89a..c5a299eea6 100644 --- a/vllm/model_executor/layers/quantization/utils/flashinfer_fp4_moe.py +++ b/vllm/model_executor/layers/quantization/utils/flashinfer_fp4_moe.py @@ -344,6 +344,7 @@ def prepare_nvfp4_moe_layer_for_fi_or_cutlass( w2_scale_2: torch.Tensor, a2_scale: torch.Tensor, is_act_and_mul: bool, + trtllm_hidden_alignment: int = 256, ) -> tuple[ torch.Tensor, torch.Tensor, @@ -398,7 +399,11 @@ def prepare_nvfp4_moe_layer_for_fi_or_cutlass( # Shuffle weights and scales for FI TRTLLM NVFP4 MoE kernels. if backend == NvFp4MoeBackend.FLASHINFER_TRTLLM: w13, w13_scale, w2, w2_scale, padded_hidden = align_fp4_moe_hidden_dim_for_fi( - w13, w13_scale, w2, w2_scale + w13, + w13_scale, + w2, + w2_scale, + min_alignment=trtllm_hidden_alignment, ) if layer.moe_config.hidden_dim_unpadded is None: layer.moe_config.hidden_dim_unpadded = layer.moe_config.hidden_dim diff --git a/vllm/model_executor/layers/quantization/utils/flashinfer_utils.py b/vllm/model_executor/layers/quantization/utils/flashinfer_utils.py index be28de9b66..5a6ee76428 100644 --- a/vllm/model_executor/layers/quantization/utils/flashinfer_utils.py +++ b/vllm/model_executor/layers/quantization/utils/flashinfer_utils.py @@ -280,6 +280,20 @@ def align_fp4_moe_weights_for_fi( return padded_w13, padded_w13_scale, padded_w2, padded_w2_scale, padded_intermediate +def trtllm_nvfp4_hidden_alignment( + per_token_activation: bool, is_act_and_mul: bool +) -> int: + """Hidden-dim alignment for the TRTLLM-Gen NVFP4 MoE cubins. + + The per-token (dynamic activation scale) non-gated variant has far fewer + tile configs for 256-aligned K and its heuristic default tactic fails with + "No valid config found" (e.g. Nemotron's 2688 -> 2816), so pad to 512. + """ + if per_token_activation and not is_act_and_mul: + return 512 + return 256 + + def align_fp4_moe_hidden_dim_for_fi( w13: torch.Tensor, w13_scale: torch.Tensor, diff --git a/vllm/model_executor/layers/quantization/utils/fp8_utils.py b/vllm/model_executor/layers/quantization/utils/fp8_utils.py index c9888dc3cb..fb87c3a614 100644 --- a/vllm/model_executor/layers/quantization/utils/fp8_utils.py +++ b/vllm/model_executor/layers/quantization/utils/fp8_utils.py @@ -142,6 +142,7 @@ def _silu_mul_quant_fp8_packed_kernel( input_ptr, output_q_ptr, output_scale_ptr, + expert_ends_ptr, M, input_stride_m, output_q_stride_m, @@ -158,6 +159,7 @@ def _silu_mul_quant_fp8_packed_kernel( PACKS_PER_CTA: tl.constexpr, BLOCK_M: tl.constexpr, HAS_CLAMP: tl.constexpr, + EXPERT_ALIGNMENT: tl.constexpr, ): GROUPS_PER_PACK: tl.constexpr = 4 hidden_size: tl.constexpr = N // 2 @@ -166,6 +168,13 @@ def _silu_mul_quant_fp8_packed_kernel( row_start = tl.program_id(1).to(tl.int64) * BLOCK_M row_step = tl.num_programs(1).to(tl.int64) * BLOCK_M + if EXPERT_ALIGNMENT: + expert = tl.program_id(2) + previous_end = tl.load(expert_ends_ptr + expert - 1, expert > 0, other=0) + expert_start = tl.cdiv(previous_end, EXPERT_ALIGNMENT) * EXPERT_ALIGNMENT + row_start += expert_start.to(tl.int64) + M = tl.load(expert_ends_ptr + expert) + groups_per_cta: tl.constexpr = PACKS_PER_CTA * GROUPS_PER_PACK elems_per_cta: tl.constexpr = groups_per_cta * GROUP_SIZE col_start = pack_tile * elems_per_cta @@ -254,7 +263,19 @@ def silu_mul_quant_fp8_packed_triton( clamp_limit: float | None = None, alpha: float = 1.0, beta: float = 0.0, + *, + expert_ends: torch.Tensor | None = None, + expert_alignment: int = 0, ) -> tuple[torch.Tensor, torch.Tensor]: + """Fuse activation and FP8 quantization, optionally skipping expert padding. + + ``expert_ends`` uses DeepGEMM's prefix-sum layout: each entry is the + exclusive end of an expert's live rows, including earlier alignment gaps. + An expert starts at the preceding end rounded up to ``expert_alignment``. + Padding outputs are left unwritten and must not be consumed as live rows. + Without ``expert_ends``, the original dense launch configuration is used + and the metadata loads are compiled out. + """ assert input.dim() == 2 assert input.is_contiguous() @@ -293,13 +314,29 @@ def silu_mul_quant_fp8_packed_triton( grid_n = triton.cdiv(packs_per_row, packs_per_cta) grid_m = min(triton.cdiv(M, BM), 4096) - grid = (grid_n, grid_m) + grid: tuple[int, ...] = (grid_n, grid_m) + if expert_ends is not None: + assert expert_ends.ndim == 1 and expert_ends.numel() > 0 + assert expert_ends.dtype == torch.int32 + assert expert_ends.device == input.device and expert_ends.is_contiguous() + assert expert_alignment > 0 + num_experts = expert_ends.numel() + # Bound empty CTAs while retaining enough parallelism for full experts. + max_ctas = 4096 if group_size < 128 else 8192 + grid_m = min( + triton.cdiv(M, BM * num_experts), + max(1, triton.cdiv(max_ctas, grid_n * num_experts)), + ) + grid = (grid_n, grid_m, num_experts) + else: + assert expert_alignment == 0 has_clamp = clamp_limit is not None _silu_mul_quant_fp8_packed_kernel[grid]( input, output_q, output_scale_packed, + expert_ends, M, input.stride(0), output_q.stride(0), @@ -316,6 +353,7 @@ def silu_mul_quant_fp8_packed_triton( PACKS_PER_CTA=packs_per_cta, BLOCK_M=BM, HAS_CLAMP=has_clamp, + EXPERT_ALIGNMENT=expert_alignment, num_warps=num_warps, num_stages=num_stages, ) @@ -328,6 +366,7 @@ def _silu_mul_per_token_group_quant_fp8_colmajor( y_ptr, # [M, N] y_q_ptr, # [M, N // 2] y_s_ptr, # [M, (N // 2) // GROUP_SIZE] + expert_ends_ptr, M, # num tokens N, # intermediate size # Stride @@ -345,8 +384,8 @@ def _silu_mul_per_token_group_quant_fp8_colmajor( GROUP_SIZE: tl.constexpr, BLOCK_M: tl.constexpr, BLOCK_N: tl.constexpr, + EXPERT_ALIGNMENT: tl.constexpr, ): - # TODO(varun) : Add expert_ids so we may early-exit no-op thread blocks. """Each thread block (BLOCK_N) computes [BLOCK_M, GROUP_SIZE] act-mul outputs. Then the thread block quantizes the [BLOCK_M, GROUP_SIZE] block of values and fills the outputs tensors at the right positions. @@ -357,56 +396,77 @@ def _silu_mul_per_token_group_quant_fp8_colmajor( m_offset = pid_m.to(tl.int64) * BLOCK_M n_offset = pid_n.to(tl.int64) * BLOCK_N - if m_offset >= M: - return + row_step = tl.num_programs(0).to(tl.int64) * BLOCK_M + num_iterations = 1 + if EXPERT_ALIGNMENT: + expert = tl.program_id(2) + previous_end = tl.load(expert_ends_ptr + expert - 1, expert > 0, other=0) + expert_start = tl.cdiv(previous_end, EXPERT_ALIGNMENT) * EXPERT_ALIGNMENT + m_offset += expert_start.to(tl.int64) + M = tl.load(expert_ends_ptr + expert) + num_iterations = tl.cdiv(tl.maximum(M - m_offset, 0), row_step) offs_n = tl.arange(0, BLOCK_N).to(tl.int64) offs_m = tl.arange(0, BLOCK_M).to(tl.int64) - base_y_ptr = y_ptr + m_offset * N + n_offset + for _ in range(num_iterations): + base_y_ptr = y_ptr + m_offset * N + n_offset - act_in_ptrs = base_y_ptr + offs_m[:, None] * N + offs_n[None, :] + act_in_ptrs = base_y_ptr + offs_m[:, None] * N + offs_n[None, :] - act_in = tl.load(act_in_ptrs) - mul_in = tl.load(act_in_ptrs + N_2) + if EXPERT_ALIGNMENT: + live = m_offset + offs_m < M + act_in = tl.load(act_in_ptrs, live[:, None], other=0) + mul_in = tl.load(act_in_ptrs + N_2, live[:, None], other=0) + else: + act_in = tl.load(act_in_ptrs) + mul_in = tl.load(act_in_ptrs + N_2) - # silu & mul — match C++ silu_and_mul: clamp in fp32 then store back to the - # input dtype, run silu in fp32 then narrow, and do the mul at input - # precision so HAS_CLAMP True/False share the same multiplication path. - if HAS_CLAMP: - act_in = tl.minimum(act_in.to(tl.float32), clamp_limit).to( - y_ptr.dtype.element_ty - ) - mul_in = tl.clamp(mul_in.to(tl.float32), -clamp_limit, clamp_limit).to( - y_ptr.dtype.element_ty - ) - # Unified gated activation: silu == swigluoai with alpha=1, beta=0. - # glu = gate * sigmoid(alpha * gate); y = (up + beta) * glu - # Keep glu/up at input precision (narrow before the mul) so the alpha=1, - # beta=0 defaults match the C++ silu_and_mul path bit-for-bit. - act_in = act_in.to(tl.float32) - glu = (act_in / (1.0 + tl.exp(-act_in * alpha))).to(y_ptr.dtype.element_ty) - up = (mul_in.to(tl.float32) + beta).to(y_ptr.dtype.element_ty) - y = (glu * up).to(tl.float32) + # silu & mul — match C++ silu_and_mul: clamp in fp32 then store back to the + # input dtype, run silu in fp32 then narrow, and do the mul at input + # precision so HAS_CLAMP True/False share the same multiplication path. + if HAS_CLAMP: + act_in = tl.minimum(act_in.to(tl.float32), clamp_limit).to( + y_ptr.dtype.element_ty + ) + mul_in = tl.clamp(mul_in.to(tl.float32), -clamp_limit, clamp_limit).to( + y_ptr.dtype.element_ty + ) + # Unified gated activation: silu == swigluoai with alpha=1, beta=0. + # glu = gate * sigmoid(alpha * gate); y = (up + beta) * glu + # Keep glu/up at input precision (narrow before the mul) so the alpha=1, + # beta=0 defaults match the C++ silu_and_mul path bit-for-bit. + act_in = act_in.to(tl.float32) + glu = (act_in / (1.0 + tl.exp(-act_in * alpha))).to(y_ptr.dtype.element_ty) + up = (mul_in.to(tl.float32) + beta).to(y_ptr.dtype.element_ty) + y = (glu * up).to(tl.float32) - # quant - _absmax = tl.maximum(tl.max(tl.abs(y), axis=1), eps) - scale_raw = _absmax * (1.0 / fp8_max) - y_s = tl.math.exp2(tl.ceil(tl.log2(scale_raw))) if use_ue8m0 else scale_raw - y_s = tl.reshape(y_s, (BLOCK_M, 1)) - y_q = tl.clamp(y / y_s, fp8_min, fp8_max).to(y_q_ptr.dtype.element_ty) + # quant + _absmax = tl.maximum(tl.max(tl.abs(y), axis=1), eps) + scale_raw = _absmax * (1.0 / fp8_max) + y_s = tl.math.exp2(tl.ceil(tl.log2(scale_raw))) if use_ue8m0 else scale_raw + y_s = tl.reshape(y_s, (BLOCK_M, 1)) + y_q = tl.clamp(y / y_s, fp8_min, fp8_max).to(y_q_ptr.dtype.element_ty) - # store y_q - base_y_q_ptr = y_q_ptr + m_offset * N_2 + n_offset - y_q_ptrs = base_y_q_ptr + offs_m[:, None] * N_2 + offs_n[None, :] - tl.store(y_q_ptrs, y_q) + # store y_q + base_y_q_ptr = y_q_ptr + m_offset * N_2 + n_offset + y_q_ptrs = base_y_q_ptr + offs_m[:, None] * N_2 + offs_n[None, :] + if EXPERT_ALIGNMENT: + tl.store(y_q_ptrs, y_q, live[:, None]) + else: + tl.store(y_q_ptrs, y_q) - # store y_s - group_id = n_offset // GROUP_SIZE - base_y_s_ptr = y_s_ptr + group_id * y_s_col_stride + m_offset - y_s_ptrs = base_y_s_ptr + offs_m - y_s = tl.reshape(y_s, (BLOCK_M,)) - tl.store(y_s_ptrs, y_s) + # store y_s + group_id = n_offset // GROUP_SIZE + base_y_s_ptr = y_s_ptr + group_id * y_s_col_stride + m_offset + y_s_ptrs = base_y_s_ptr + offs_m + y_s = tl.reshape(y_s, (BLOCK_M,)) + if EXPERT_ALIGNMENT: + tl.store(y_s_ptrs, y_s, live) + else: + tl.store(y_s_ptrs, y_s) + + m_offset += row_step def silu_mul_per_token_group_quant_fp8_colmajor( @@ -418,9 +478,15 @@ def silu_mul_per_token_group_quant_fp8_colmajor( group_size: int = 128, alpha: float = 1.0, beta: float = 0.0, + *, + expert_ends: torch.Tensor | None = None, + expert_alignment: int = 0, ): """Gated activation + block-fp8 quant. ``alpha``/``beta`` select the gate (silu: alpha=1, beta=0; swigluoai: alpha, beta from config). + + Optional expert endpoints follow DeepGEMM's prefix-sum layout. Padding + values and scales are left unwritten and must not be consumed as live rows. """ GROUP_SIZE = group_size assert input.ndim == 2 @@ -458,13 +524,27 @@ def silu_mul_per_token_group_quant_fp8_colmajor( # Force even division so we can avoid edgecases within the kernel. assert M % BLOCK_M == 0 assert N_2 % BLOCK_N == 0 - grid = (M // BLOCK_M, N_2 // BLOCK_N) + grid: tuple[int, ...] = (M // BLOCK_M, N_2 // BLOCK_N) + if expert_ends is not None: + assert expert_ends.ndim == 1 and expert_ends.numel() > 0 + assert expert_ends.dtype == torch.int32 + assert expert_ends.device == input.device and expert_ends.is_contiguous() + assert expert_alignment > 0 and expert_alignment % BLOCK_M == 0 + num_experts = expert_ends.numel() + grid_m = min( + triton.cdiv(M, BLOCK_M * num_experts), + max(1, triton.cdiv(8192, grid[1] * num_experts)), + ) + grid = (grid_m, grid[1], num_experts) + else: + assert expert_alignment == 0 has_clamp = clamp_limit is not None _silu_mul_per_token_group_quant_fp8_colmajor[grid]( input, output, output_scales, + expert_ends, M, N, output_scales.stride(-1), @@ -479,6 +559,7 @@ def silu_mul_per_token_group_quant_fp8_colmajor( GROUP_SIZE, BLOCK_M, BLOCK_N, + expert_alignment, ) return output, output_scales diff --git a/vllm/model_executor/layers/quantization/utils/gptq_utils.py b/vllm/model_executor/layers/quantization/utils/gptq_utils.py index d134abf8dc..024564e6f3 100644 --- a/vllm/model_executor/layers/quantization/utils/gptq_utils.py +++ b/vllm/model_executor/layers/quantization/utils/gptq_utils.py @@ -133,6 +133,11 @@ def is_layer_gptq_quantized( quantized_layers = flatten_list(quantized_layers) + # Empty means no metadata could be read at all, not that the checkpoint + # holds no quantized layer, and loading GPTQ weights unquantized cannot work. + if not quantized_layers: + return True + # Fused layers like gate_up_proj or qkv_proj will not be fused # in the safetensors checkpoint. So, we convert the name # from the fused version to unfused + check to make sure that diff --git a/vllm/model_executor/layers/quantization/utils/humming/__init__.py b/vllm/model_executor/layers/quantization/utils/humming/__init__.py index 4303f74221..91a5713bbf 100644 --- a/vllm/model_executor/layers/quantization/utils/humming/__init__.py +++ b/vllm/model_executor/layers/quantization/utils/humming/__init__.py @@ -19,6 +19,10 @@ from vllm.model_executor.layers.quantization.utils.humming.moe import ( make_humming_moe_quant_config, select_humming_moe_experts, ) +from vllm.model_executor.layers.quantization.utils.humming.priority import ( + prefers_humming, + prioritize_humming, +) from vllm.model_executor.layers.quantization.utils.humming.schema import ( check_and_fallback_input_schema, humming_is_layer_skipped, @@ -41,6 +45,8 @@ __all__ = [ "convert_linear_layer_to_humming_standard", "get_humming_linear_compute_config", "prepare_humming_linear_layer_config", + "prefers_humming", + "prioritize_humming", "make_humming_moe_kernel", "select_humming_moe_experts", "input_schema_to_quant_key", diff --git a/vllm/model_executor/layers/quantization/utils/humming/linear.py b/vllm/model_executor/layers/quantization/utils/humming/linear.py index 0c64645c43..fcce3dabfe 100644 --- a/vllm/model_executor/layers/quantization/utils/humming/linear.py +++ b/vllm/model_executor/layers/quantization/utils/humming/linear.py @@ -87,7 +87,7 @@ def prepare_humming_linear_layer_config( shape_n_stacks = layer.output_partition_sizes # Step 1: convert weight and input schemas to humming standard format - source_tensors = dict(layer.named_parameters()) + source_tensors = dict(layer.named_parameters(recurse=False)) weight_schema, tensors = weight_schema.convert_humming( tensors=source_tensors, shape_n_stacks=shape_n_stacks, @@ -143,7 +143,7 @@ def prepare_humming_linear_layer_config( if "bias" in tensors: zero_bias = torch.zeros_like(tensors["bias"]) layer.register_buffer("zero_bias", zero_bias) - for name, _ in list(layer.named_parameters()): + for name, _ in list(layer.named_parameters(recurse=False)): delattr(layer, name) for name, tensor in tensors.items(): param = torch.nn.Parameter(tensor, requires_grad=False) diff --git a/vllm/model_executor/layers/quantization/utils/humming/priority.py b/vllm/model_executor/layers/quantization/utils/humming/priority.py new file mode 100644 index 0000000000..4a1452c2a9 --- /dev/null +++ b/vllm/model_executor/layers/quantization/utils/humming/priority.py @@ -0,0 +1,48 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Humming kernel selection priority.""" + +from enum import Enum +from typing import TypeVar + +_KernelT = TypeVar("_KernelT", bound=type | Enum) + +# Compute capabilities where Humming outranks Marlin. +_HUMMING_PREFERRED_CAPABILITIES = frozenset({90}) + + +def prefers_humming(compute_capability: int | None = None) -> bool: + """Whether Humming outranks Marlin. A missing compute capability uses the + current device.""" + if compute_capability is None: + from vllm.platforms import current_platform + + if current_platform.is_cuda(): + cc = current_platform.get_device_capability() + compute_capability = cc.to_int() if cc is not None else None + return compute_capability in _HUMMING_PREFERRED_CAPABILITIES + + +def prioritize_humming( + kernels: list[_KernelT], + compute_capability: int | None = None, +) -> list[_KernelT]: + """Move Humming directly ahead of Marlin where Humming is preferred. + + Every other entry keeps its relative order and the input list is not + modified. Match kernel class names or MoE backend enum names. + """ + if not prefers_humming(compute_capability): + return kernels + + names = [ + (kernel.name if isinstance(kernel, Enum) else kernel.__name__).lower() + for kernel in kernels + ] + humming = next((i for i, name in enumerate(names) if "humming" in name), None) + marlin = next((i for i, name in enumerate(names) if "marlin" in name), None) + if humming is None or marlin is None or humming < marlin: + return kernels + kernels = kernels.copy() + kernels.insert(marlin, kernels.pop(humming)) + return kernels diff --git a/vllm/model_executor/layers/quantization/utils/humming/schema.py b/vllm/model_executor/layers/quantization/utils/humming/schema.py index c84e8a370d..aaf6d4a66c 100644 --- a/vllm/model_executor/layers/quantization/utils/humming/schema.py +++ b/vllm/model_executor/layers/quantization/utils/humming/schema.py @@ -322,6 +322,22 @@ def check_and_fallback_input_schema( input_group_size = input_schema.input_scale_group_size input_scale_dtype = input_schema.input_scale_dtype + def is_compatible(candidate: "HummingInputSchema") -> bool: + check_weight_schema = weight_schema + if ( + candidate.a_dtype in (None, humming_dtypes.float16, humming_dtypes.bfloat16) + and weight_schema.b_dtype == humming_dtypes.float4e2m1 + and weight_group_size == 16 + and weight_schema.bs_dtype + in (None, humming_dtypes.float16, humming_dtypes.bfloat16) + ): + # Humming rejects folded NVFP4 scales despite A16 software dequant + # supporting them. Relax only this check, preserving kernel metadata. + check_weight_schema = dataclasses.replace( + weight_schema, bs_dtype=humming_dtypes.float8e4m3 + ) + return candidate.is_compatible_with(check_weight_schema, param_dtype) + def is_deprecated(dtype: "humming_dtypes.DataType | None") -> bool: if dtype is None or dtype.num_bits == 16: return False @@ -333,7 +349,7 @@ def check_and_fallback_input_schema( is_fp8_deprecated = dtype.num_bits < 16 and 100 <= sm_version < 120 return is_int4_deprecated or is_int8_deprecated or is_fp8_deprecated - if input_schema.is_compatible_with(weight_schema, param_dtype): + if is_compatible(input_schema): if not allow_fallback: if is_deprecated(a_dtype): msg = f"Humming {a_dtype} activation is slow on SM{sm_version} now." @@ -390,7 +406,7 @@ def check_and_fallback_input_schema( input_scale_dtype=scale_dtype, input_quant_mode=quant_mode, ) - if candidate.is_compatible_with(weight_schema, param_dtype): + if is_compatible(candidate): return candidate raise ValueError( diff --git a/vllm/model_executor/layers/quantization/utils/mxfp4_utils.py b/vllm/model_executor/layers/quantization/utils/mxfp4_utils.py index 8cc626f386..45562b12b1 100644 --- a/vllm/model_executor/layers/quantization/utils/mxfp4_utils.py +++ b/vllm/model_executor/layers/quantization/utils/mxfp4_utils.py @@ -120,7 +120,14 @@ def _swizzle_mxfp4(quant_tensor, scale, num_warps=8): K = scale.shape[-1] pad_k = -K % 4 scale = torch.nn.functional.pad(scale, (0, pad_k)) - elif current_platform.is_device_capability_family(100): + elif current_platform.is_device_capability_family( + 100 + ) or current_platform.is_device_capability_family(120): + # Native MXFP (cap >= 10.0) requires the persistent kernel. On + # SM12x, epilogue_subtile=1 also keeps the heuristic away from a + # triton_kernels 3.5.1 codegen bug (SWAP_XW with a subtiled + # epilogue); block_m is pinned per call in + # gpt_oss_triton_kernels_moe to fit the 99KB shared memory. constraints = { "is_persistent": True, "epilogue_subtile": 1, diff --git a/vllm/model_executor/layers/utils.py b/vllm/model_executor/layers/utils.py index d4ee153fb2..80eef7c75a 100644 --- a/vllm/model_executor/layers/utils.py +++ b/vllm/model_executor/layers/utils.py @@ -128,16 +128,13 @@ def _can_use_flashinfer_cutedsl_bf16( if ( k <= 0 or n <= 0 + or x.numel() == 0 or weight.shape[1] != k or k % 128 != 0 or x.data_ptr() % 32 != 0 or weight.data_ptr() % 32 != 0 ): return False - - m = x.numel() // k - if not 1 <= m <= 32: - return False return bias is None or ( bias.is_cuda and bias.device == x.device diff --git a/vllm/model_executor/model_loader/base_loader.py b/vllm/model_executor/model_loader/base_loader.py index 2b7cb72fff..4c774808bf 100644 --- a/vllm/model_executor/model_loader/base_loader.py +++ b/vllm/model_executor/model_loader/base_loader.py @@ -101,6 +101,7 @@ class BaseModelLoader(ABC): finalize_layerwise_processing(model, model_config) process_weights_after_loading(model, model_config, target_device) + log_online_quantization_time(vllm_config) return model.eval() @@ -115,8 +116,8 @@ def log_model_inspection(model: nn.Module) -> None: logger.info("vLLM model structure:\n%s", format_model_inspection(model)) -def log_online_quantization(vllm_config: VllmConfig) -> None: - """Log the online-quantized layer count and types, when applicable.""" +def _get_online_quantization_config(vllm_config: VllmConfig): + """Return the online quantization config when one is configured.""" from vllm.model_executor.layers.quantization.online.base import ( OnlineQuantizationConfig, ) @@ -128,15 +129,35 @@ def log_online_quantization(vllm_config: VllmConfig) -> None: if isinstance(online_quantization_config, OnlineQuantizationConfig): quant_config = online_quantization_config if not isinstance(quant_config, OnlineQuantizationConfig): + return None + return quant_config + + +def log_online_quantization(vllm_config: VllmConfig) -> None: + """Log the online-quantized layer count and types, when applicable.""" + quant_config = _get_online_quantization_config(vllm_config) + if quant_config is None: return logger.info( - "Quantized %d layers of types: %s", + "Quantizing %d layers of types: %s", len(quant_config.quantized_layers), "; ".join(quant_config.quantized_layer_summaries), ) +def log_online_quantization_time(vllm_config: VllmConfig) -> None: + """Log online quantization processing time, when applicable.""" + quant_config = _get_online_quantization_config(vllm_config) + if quant_config is None: + return + + logger.info( + "Online quantization as part of model loading took %.2f seconds", + quant_config.online_quantization_time, + ) + + def _has_online_quant(model: nn.Module): for module in model.modules(): quant_method = getattr(module, "quant_method", None) diff --git a/vllm/model_executor/model_loader/default_loader.py b/vllm/model_executor/model_loader/default_loader.py index c3e09bccd4..9f1c25dd2e 100644 --- a/vllm/model_executor/model_loader/default_loader.py +++ b/vllm/model_executor/model_loader/default_loader.py @@ -6,7 +6,7 @@ import dataclasses import glob import os import time -from collections.abc import Generator, Iterable +from collections.abc import Callable, Generator, Iterable from typing import TYPE_CHECKING, cast import torch @@ -28,6 +28,7 @@ from vllm.model_executor.model_loader.weight_utils import ( filter_duplicate_safetensors_files, filter_files_not_needed_for_inference, filter_mm_encoder_only_safetensors_files, + filter_safetensors_files_by_weight_name, get_quant_config, instanttensor_weights_iterator, maybe_download_from_modelscope, @@ -75,6 +76,10 @@ class DefaultModelLoader(BaseModelLoader): allow_patterns_overrides: list[str] | None = None """If defined, weights will load exclusively using these patterns.""" + is_unused_weight: Callable[[str], bool] | None = None + """If defined, safetensors shards holding only weights whose checkpoint + name (before *prefix*) it accepts are not read.""" + counter_before_loading_weights: float = 0.0 counter_after_loading_weights: float = 0.0 @@ -283,6 +288,10 @@ class DefaultModelLoader(BaseModelLoader): f"`{source.model_or_path}`; check language_model prefixes " f"{self._encoder_only_lm_prefixes}" ) + if source.is_unused_weight is not None and use_safetensors: + hf_weights_files = filter_safetensors_files_by_weight_name( + hf_weights_files, source.is_unused_weight + ) if self.load_config.load_format == "npcache": # Currently np_cache only support *.bin checkpoints assert use_safetensors is False @@ -359,6 +368,7 @@ class DefaultModelLoader(BaseModelLoader): prefix="", fall_back_to_pt=getattr(model, "fall_back_to_pt_during_load", True), allow_patterns_overrides=getattr(model, "allow_patterns_overrides", None), + is_unused_weight=getattr(model, "is_unused_checkpoint_weight", None), ) yield from self._get_weights_iterator(primary_weights) diff --git a/vllm/model_executor/model_loader/reload/layerwise.py b/vllm/model_executor/model_loader/reload/layerwise.py index 062bfa6090..df91522732 100644 --- a/vllm/model_executor/model_loader/reload/layerwise.py +++ b/vllm/model_executor/model_loader/reload/layerwise.py @@ -2,6 +2,7 @@ # SPDX-FileCopyrightText: Copyright contributors to the vLLM project import inspect from collections.abc import Callable +from contextlib import nullcontext from functools import wraps from weakref import WeakKeyDictionary, WeakSet @@ -321,7 +322,21 @@ def _reload_attention_scales(layer: torch.nn.Module, info: LayerReloadingInfo) - _get_weight_loader(param)(*args.args, **args.kwargs) if quant_method is not None: - quant_method.process_weights_after_loading(layer) + from vllm.model_executor.layers.quantization.online.fp8 import OnlineLinearBase + from vllm.model_executor.layers.quantization.online.moe_base import ( + OnlineMoEMethodBase, + ) + from vllm.model_executor.model_loader.utils import ( + online_quantization_timing_context, + ) + + timing_context = ( + online_quantization_timing_context(layer, quant_method) + if isinstance(quant_method, (OnlineLinearBase, OnlineMoEMethodBase)) + else nullcontext() + ) + with timing_context: + quant_method.process_weights_after_loading(layer) _copy_and_restore_kernel_tensors(layer, info) @@ -356,7 +371,21 @@ def _layerwise_process(layer: torch.nn.Module, info: LayerReloadingInfo): # Process weights (quantization, repacking, etc.) quant_method = getattr(layer, "quant_method", None) if isinstance(quant_method, QuantizeMethodBase): - quant_method.process_weights_after_loading(layer) + from vllm.model_executor.layers.quantization.online.fp8 import OnlineLinearBase + from vllm.model_executor.layers.quantization.online.moe_base import ( + OnlineMoEMethodBase, + ) + from vllm.model_executor.model_loader.utils import ( + online_quantization_timing_context, + ) + + timing_context = ( + online_quantization_timing_context(layer, quant_method) + if isinstance(quant_method, (OnlineLinearBase, OnlineMoEMethodBase)) + else nullcontext() + ) + with timing_context: + quant_method.process_weights_after_loading(layer) # Re-reconcile parameter TP state: process_weights_after_loading may # have re-created Parameters (stamped with the global rank), which would # otherwise break replicated (disable_tp) weights on a subsequent reload. diff --git a/vllm/model_executor/model_loader/runai_streamer_loader.py b/vllm/model_executor/model_loader/runai_streamer_loader.py index 2f18a2e3ac..e001160062 100644 --- a/vllm/model_executor/model_loader/runai_streamer_loader.py +++ b/vllm/model_executor/model_loader/runai_streamer_loader.py @@ -134,11 +134,16 @@ class RunaiModelStreamerLoader(BaseModelLoader): """Download model if necessary.""" self._prepare_weights(model_config.model, model_config.revision) - def load_weights(self, model: nn.Module, model_config: ModelConfig) -> None: - """Load weights into a model.""" + def get_all_weights( + self, + model_config: ModelConfig, + model: nn.Module, + ) -> Generator[tuple[str, torch.Tensor], None, None]: model_weights = model_config.model if model_weights_override := model_config.model_weights: model_weights = model_weights_override - model.load_weights( - self._get_weights_iterator(model_weights, model_config.revision) - ) + yield from self._get_weights_iterator(model_weights, model_config.revision) + + def load_weights(self, model: nn.Module, model_config: ModelConfig) -> None: + """Load weights into a model.""" + model.load_weights(self.get_all_weights(model_config, model)) diff --git a/vllm/model_executor/model_loader/tensorizer.py b/vllm/model_executor/model_loader/tensorizer.py index bb48c5dc08..3159a519fa 100644 --- a/vllm/model_executor/model_loader/tensorizer.py +++ b/vllm/model_executor/model_loader/tensorizer.py @@ -23,7 +23,6 @@ from transformers import PreTrainedConfig import vllm.envs as envs from vllm.config import ModelConfig, ParallelConfig, VllmConfig, set_current_vllm_config from vllm.logger import init_logger -from vllm.model_executor.layers.vocab_parallel_embedding import VocabParallelEmbedding from vllm.platforms import current_platform from vllm.transformers_utils.repo_utils import hf_api from vllm.utils.argparse_utils import FlexibleArgumentParser @@ -167,7 +166,6 @@ class TensorizerConfig(MutableMapping): s3_access_key_id: str | None = None s3_secret_access_key: str | None = None s3_endpoint: str | None = None - lora_dir: str | None = None stream_kwargs: dict[str, Any] | None = None serialization_kwargs: dict[str, Any] | None = None deserialization_kwargs: dict[str, Any] | None = None @@ -185,11 +183,8 @@ class TensorizerConfig(MutableMapping): Attributes: tensorizer_uri: Path to serialized model tensors. Can be a local file - path or a S3 URI. This is a required field unless lora_dir is - provided and the config is meant to be used for the - `tensorize_lora_adapter` function. Unless a `tensorizer_dir` or - `lora_dir` is passed to this object's initializer, this is - a required argument. + path or a S3 URI. Unless a `tensorizer_dir` is passed to this + object's initializer, this is a required argument. tensorizer_dir: Path to a directory containing serialized model tensors, and all other potential model artifacts to load the model, such as configs and tokenizer files. Can be passed instead of @@ -219,10 +214,6 @@ class TensorizerConfig(MutableMapping): be set via the S3_SECRET_ACCESS_KEY environment variable. s3_endpoint: The endpoint for the S3 bucket. Can also be set via the S3_ENDPOINT_URL environment variable. - lora_dir: Path to a directory containing LoRA adapter artifacts for - serialization or deserialization. When serializing LoRA adapters - this is the only necessary parameter to pass to this object's - initializer. """ def __post_init__(self): @@ -232,12 +223,6 @@ class TensorizerConfig(MutableMapping): and re.search(r"%0\dd", self.tensorizer_uri) is not None ) - if self.tensorizer_dir and self.lora_dir: - raise ValueError( - "Only one of tensorizer_dir or lora_dir may be specified. " - "Use lora_dir exclusively when serializing LoRA adapters, " - "and tensorizer_dir or tensorizer_uri otherwise." - ) if self.tensorizer_dir and self.tensorizer_uri: logger.warning_once( "Provided both tensorizer_dir and tensorizer_uri. " @@ -246,17 +231,12 @@ class TensorizerConfig(MutableMapping): ) self.tensorizer_dir = os.path.dirname(self.tensorizer_uri) if not self.tensorizer_uri: - if self.lora_dir: - self.tensorizer_uri = f"{self.lora_dir}/adapter_model.tensors" - elif self.tensorizer_dir: + if self.tensorizer_dir: self.tensorizer_uri = f"{self.tensorizer_dir}/model.tensors" else: raise ValueError( "Unable to resolve tensorizer_uri. " - "A valid tensorizer_uri or tensorizer_dir " - "must be provided for deserialization, and a " - "valid tensorizer_uri, tensorizer_uri, or " - "lora_dir for serialization." + "A valid tensorizer_uri or tensorizer_dir must be provided." ) else: self.tensorizer_dir = os.path.dirname(self.tensorizer_uri) @@ -288,9 +268,6 @@ class TensorizerConfig(MutableMapping): if "tensorizer_uri" in raw_tc_dict and "tensorizer_dir" in raw_tc_dict: blacklisted.append("tensorizer_dir") - if "tensorizer_dir" in raw_tc_dict and "lora_dir" in raw_tc_dict: - blacklisted.append("tensorizer_dir") - tc_dict = {} for k, v in raw_tc_dict.items(): if ( @@ -495,25 +472,6 @@ def _check_tensors_on_meta_device(model: nn.Module) -> None: ) -def _resize_lora_embeddings(model: nn.Module): - """Modify LoRA embedding layers to use bigger tensors - to allow for adapter added tokens.""" - for child in model.modules(): - if ( - isinstance(child, VocabParallelEmbedding) - and child.weight.shape[0] < child.num_embeddings_per_partition - ): - new_weight = torch.empty( - child.num_embeddings_per_partition, - child.embedding_dim, - dtype=child.weight.dtype, - device=child.weight.device, - ) - new_weight[: child.weight.shape[0]].copy_(child.weight.data) - new_weight[child.weight.shape[0] :].fill_(0) - child.weight.data = new_weight - - def init_tensorizer_model( tensorizer_config: TensorizerConfig, vllm_config: VllmConfig ) -> nn.Module: @@ -567,7 +525,6 @@ def deserialize_tensorizer_model( logger.info("Memory usage after: %s", after_mem) _check_tensors_on_meta_device(model) - _resize_lora_embeddings(model) del model.vllm_tensorized_marker @@ -769,60 +726,3 @@ def tensorize_vllm_model( if error is not None: raise error - - -def tensorize_lora_adapter(lora_path: str, tensorizer_config: TensorizerConfig): - """Uses tensorizer to serialize a LoRA adapter. Assumes that the files - needed to load a LoRA adapter are a safetensors-format file called - adapter_model.safetensors and a json config file called adapter_config.json. - - Serializes the files in the tensorizer_config.tensorizer_dir - """ - import safetensors - - from vllm.lora.utils import get_adapter_absolute_path - - lora_dir = get_adapter_absolute_path(lora_path) - - tensor_path = config_path = "" - - for file in os.listdir(lora_dir): - if file.startswith("adapter_model"): - tensor_path = lora_dir + "/" + file - if file.startswith("adapter_config"): - config_path = lora_dir + "/" + file - if tensor_path and config_path: - break - - if tensor_path.endswith(".safetensors"): - tensors = safetensors.torch.load_file(tensor_path) - elif tensor_path.endswith(".bin"): - tensors = torch.load(tensor_path, weights_only=True) - else: - raise ValueError( - f"Unsupported adapter model file: {tensor_path}. " - f"Must be a .safetensors or .bin file." - ) - - with open(config_path) as f: - config = json.load(f) - - tensorizer_args = tensorizer_config._construct_tensorizer_args() - - with open_stream( - f"{tensorizer_config.tensorizer_dir}/adapter_config.json", - mode="wb+", - **tensorizer_args.stream_kwargs, - ) as f: - f.write(json.dumps(config).encode("utf-8")) - - lora_uri = f"{tensorizer_config.tensorizer_dir}/adapter_model.tensors" - with open_stream(lora_uri, mode="wb+", **tensorizer_args.stream_kwargs) as f: - serializer = TensorSerializer(f) - serializer.write_state_dict(tensors) - serializer.close() - - logger.info( - "Successfully serialized LoRA files to %s", - str(tensorizer_config.tensorizer_dir), - ) diff --git a/vllm/model_executor/model_loader/utils.py b/vllm/model_executor/model_loader/utils.py index 465703f759..65d1ff2808 100644 --- a/vllm/model_executor/model_loader/utils.py +++ b/vllm/model_executor/model_loader/utils.py @@ -3,13 +3,13 @@ """Utilities for selecting and loading models.""" import inspect +import time import warnings from contextlib import contextmanager, nullcontext -from typing import Any +from typing import Any, assert_never import torch from torch import nn -from typing_extensions import assert_never import vllm.envs as envs from vllm.config import ( @@ -141,6 +141,11 @@ def process_weights_after_loading( """ # Reclaim memory when an explicit lm_head has been # loaded, but it is identical to the input embeddings. + from vllm.model_executor.layers.quantization.online.fp8 import OnlineLinearBase + from vllm.model_executor.layers.quantization.online.moe_base import ( + OnlineMoEMethodBase, + ) + maybe_retie_word_embeddings(model, model_config) for name, module in model.named_modules(): @@ -166,7 +171,12 @@ def process_weights_after_loading( if quant_method.requires_device_loading else nullcontext() ) - with loading_context: + timing_context = ( + online_quantization_timing_context(module, quant_method) + if isinstance(quant_method, (OnlineLinearBase, OnlineMoEMethodBase)) + else nullcontext() + ) + with loading_context, timing_context: quant_method.process_weights_after_loading(module) # process_weights_after_loading may swap in freshly-created # Parameters (e.g. FP8 requantization), which are stamped with the @@ -245,6 +255,43 @@ def device_loading_context(module: torch.nn.Module, target_device: torch.device) p._vllm_is_uva_offloaded = True +@contextmanager +def online_quantization_timing_context( + layer: nn.Module, quant_method: QuantizeMethodBase +): + """Accumulate online quantization processing time for ``quant_method``.""" + from vllm.model_executor.layers.quantization.online.base import ( + OnlineQuantizationConfig, + ) + from vllm.model_executor.layers.quantization.online.fp8 import OnlineLinearBase + from vllm.model_executor.layers.quantization.online.moe_base import ( + OnlineMoEMethodBase, + ) + + assert isinstance(quant_method, (OnlineLinearBase, OnlineMoEMethodBase)) + quant_config = layer.quant_config + assert quant_config is not None + online_quantization_config = ( + quant_config + if isinstance(quant_config, OnlineQuantizationConfig) + else quant_config.online_quantization_config + ) + # reload/layerwise.py's _layerwise_process calls this for experts_int8, + # whose online MoE method has no OnlineQuantizationConfig. + if online_quantization_config is None: + yield + return + assert isinstance(online_quantization_config, OnlineQuantizationConfig) + + start_time = time.perf_counter() + try: + yield + finally: + online_quantization_config.online_quantization_time += ( + time.perf_counter() - start_time + ) + + _MODEL_ARCH_BY_HASH = dict[int, tuple[type[nn.Module], str]]() """Caches the outputs of `_get_model_architecture`.""" diff --git a/vllm/model_executor/model_loader/weight_cache/protocol.py b/vllm/model_executor/model_loader/weight_cache/protocol.py index 72053805cd..535c4de962 100644 --- a/vllm/model_executor/model_loader/weight_cache/protocol.py +++ b/vllm/model_executor/model_loader/weight_cache/protocol.py @@ -28,6 +28,7 @@ from transformers.utils import SAFE_WEIGHTS_INDEX_NAME import vllm.version from vllm.config import ModelConfig +from vllm.config.utils import normalize_value from vllm.model_executor.layers.quantization.base_config import QuantizeMethodBase from vllm.model_executor.model_loader.weight_cache.utils import ( format_socket_role_suffix, @@ -227,12 +228,17 @@ def verify_peer_is_owner(conn: socket.socket) -> None: raise PermissionError(f"Rejecting weight cache connection from uid {peer_uid}") -def _hash_quant_config(quant_config: Any) -> str: - if quant_config is None: - return "" +def _hash_quant_config(quant_config: Any, runtime_quant_config: Any = None) -> str: if hasattr(quant_config, "to_dict"): quant_config = quant_config.to_dict() - payload = json.dumps(quant_config, sort_keys=True, default=str) + payload = json.dumps( + { + "checkpoint": quant_config, + "runtime": normalize_value(runtime_quant_config), + }, + sort_keys=True, + default=str, + ) return safe_hash(payload.encode(), usedforsecurity=False).hexdigest() @@ -333,7 +339,10 @@ class WeightCacheKey: pp_rank=pp_rank, dtype=str(model_config.dtype), quantization=model_config.quantization, - quant_config_hash=_hash_quant_config(quant_config), + quant_config_hash=_hash_quant_config( + quant_config=quant_config, + runtime_quant_config=model_config.quantization_config, + ), revision=model_config.revision, vllm_version=vllm.version.__version__, is_draft=is_draft, diff --git a/vllm/model_executor/model_loader/weight_utils.py b/vllm/model_executor/model_loader/weight_utils.py index 12d557fa1c..ac1da8d698 100644 --- a/vllm/model_executor/model_loader/weight_utils.py +++ b/vllm/model_executor/model_loader/weight_utils.py @@ -200,6 +200,9 @@ def get_quant_config( QuantizationConfigArgs, resolve_quantization_config, ) + from vllm.model_executor.layers.quantization.modelopt import ( + ModelOptQuantConfigBase, + ) from vllm.model_executor.layers.quantization.online.base import ( OnlineQuantizationConfig, ) @@ -220,6 +223,14 @@ def get_quant_config( ) return checkpoint_config + def make_checkpoint_config(config: dict[str, Any]) -> QuantizationConfig: + if issubclass(quant_cls, ModelOptQuantConfigBase): + assert online_args is None or isinstance( + online_args, QuantizationConfigArgs + ) + config = {**config, "_online_quantization_args": online_args} + return maybe_compose_online_quantization(quant_cls.from_config(config)) + # Read the quantization config from the HF model config, if available. hf_quant_config = getattr(model_config.hf_config, "quantization_config", None) # some vision model may keep quantization_config in their text_config @@ -260,9 +271,7 @@ def get_quant_config( ): pass # fall through to file-based loading below else: - return maybe_compose_online_quantization( - quant_cls.from_config(hf_quant_config) - ) + return make_checkpoint_config(hf_quant_config) # if hf_quant_config is None, we will try to get config from # hf_overrides @@ -376,14 +385,14 @@ def get_quant_config( if model_config.quantization in ("modelopt", "modelopt_mixed"): if config.get("producer", {}).get("name") == "modelopt": - return maybe_compose_online_quantization(quant_cls.from_config(config)) + return make_checkpoint_config(config) else: raise ValueError( f"Unsupported quantization config" f" found for {model_config.quantization} in {f}." ) - return maybe_compose_online_quantization(quant_cls.from_config(config)) + return make_checkpoint_config(config) def get_sparse_attention_config( @@ -785,6 +794,42 @@ def filter_mm_encoder_only_safetensors_files( return kept +def filter_safetensors_files_by_weight_name( + hf_weights_files: list[str], + is_unused_weight: Callable[[str], bool], +) -> list[str]: + """Drop safetensors shards in which `is_unused_weight` accepts every tensor. + + Loaders that read whole files (InstantTensor, fastsafetensors, multi-thread, + eager, prefetch) cannot skip single tensors, so dropping shards is their + only way to avoid reading weights the model does not load. + + Args: + hf_weights_files: Safetensors shard paths. + is_unused_weight: Returns True for checkpoint weight names the model does + not load. + + Returns: + The shards holding at least one wanted tensor, or `hf_weights_files` + unchanged when no shard does. + + """ + kept: list[str] = [] + for st_file in hf_weights_files: + with safe_open(st_file, framework="pt") as f: + if not all(is_unused_weight(name) for name in f.keys()): # noqa: SIM118 + kept.append(st_file) + if not kept: + return hf_weights_files + if len(kept) < len(hf_weights_files): + logger.info_once( + "Skipped %d/%d safetensors shard(s) holding no weights the model loads", + len(hf_weights_files) - len(kept), + len(hf_weights_files), + ) + return kept + + # explicitly use pure text format, with a newline at the end # this makes it impossible to see the animation in the progress bar # but will avoid messing up with ray or multiprocessing, which wraps diff --git a/vllm/model_executor/models/AXK1.py b/vllm/model_executor/models/AXK1.py index da38035db9..83859946a2 100644 --- a/vllm/model_executor/models/AXK1.py +++ b/vllm/model_executor/models/AXK1.py @@ -29,6 +29,7 @@ from itertools import islice import torch from torch import nn +from transformers import AXK1Config from vllm._aiter_ops import rocm_aiter_ops from vllm.compilation.decorators import support_torch_compile @@ -78,7 +79,6 @@ from vllm.model_executor.models.deepseek_v2 import ( from vllm.model_executor.models.utils import sequence_parallel_chunk from vllm.platforms import current_platform from vllm.sequence import IntermediateTensors -from vllm.transformers_utils.configs.AXK1 import AXK1Config from .interfaces import MixtureOfExperts, SupportsEagle, SupportsLoRA, SupportsPP from .utils import ( @@ -199,6 +199,7 @@ class AXK1MoE(nn.Module): if self.is_fused_shared_expert_enabled else None, fuse_shared_experts=self.is_fused_shared_expert_enabled, + shared_expert_prefix=f"{prefix}.shared_experts", ) def forward(self, hidden_states: torch.Tensor) -> torch.Tensor: diff --git a/vllm/model_executor/models/aimv2.py b/vllm/model_executor/models/aimv2.py index 4e31b99944..c9e5dcfdcd 100644 --- a/vllm/model_executor/models/aimv2.py +++ b/vllm/model_executor/models/aimv2.py @@ -7,6 +7,7 @@ from collections.abc import Iterable import torch import torch.nn as nn +from transformers import Aimv2VisionConfig from vllm.distributed import get_tensor_model_parallel_world_size from vllm.distributed.utils import divide @@ -21,20 +22,19 @@ from vllm.model_executor.layers.linear import ( ) from vllm.model_executor.layers.quantization import QuantizationConfig from vllm.model_executor.models.utils import AutoWeightsLoader, WeightsMapper -from vllm.transformers_utils.configs.ovis import AIMv2Config class AIMv2SwiGLUFFN(nn.Module): def __init__( self, - config: AIMv2Config, + config: Aimv2VisionConfig, quant_config: QuantizationConfig | None, prefix: str, ): super().__init__() hidden_features = config.intermediate_size in_features = config.hidden_size - bias = config.use_bias + bias = config.mlp_bias self.fc13 = MergedColumnParallelLinear( in_features, @@ -60,7 +60,7 @@ class AIMv2SwiGLUFFN(nn.Module): class AIMv2PatchEmbed(nn.Module): - def __init__(self, config: AIMv2Config): + def __init__(self, config: Aimv2VisionConfig): super().__init__() self.proj = Conv2dLayer( config.num_channels, @@ -77,7 +77,7 @@ class AIMv2PatchEmbed(nn.Module): class AIMv2ViTPreprocessor(nn.Module): - def __init__(self, config: AIMv2Config): + def __init__(self, config: Aimv2VisionConfig): super().__init__() num_patches = (config.image_size // config.patch_size) ** 2 @@ -95,7 +95,7 @@ class AIMv2ViTPreprocessor(nn.Module): class AIMv2Attention(nn.Module): def __init__( self, - config: AIMv2Config, + config: Aimv2VisionConfig, quant_config: QuantizationConfig | None, prefix: str, ): @@ -124,7 +124,7 @@ class AIMv2Attention(nn.Module): self.proj = RowParallelLinear( input_size=self.embed_dim, output_size=self.embed_dim, - bias=config.use_bias, + bias=config.mlp_bias, quant_config=quant_config, prefix=f"{prefix}.proj", ) @@ -151,7 +151,7 @@ class AIMv2Attention(nn.Module): class AIMv2Block(nn.Module): def __init__( self, - config: AIMv2Config, + config: Aimv2VisionConfig, quant_config: QuantizationConfig | None, prefix: str, ): @@ -174,7 +174,7 @@ class AIMv2Block(nn.Module): class AIMv2Transformer(nn.Module): def __init__( self, - config: AIMv2Config, + config: Aimv2VisionConfig, quant_config: QuantizationConfig | None, *, require_post_norm: bool | None = None, @@ -214,7 +214,7 @@ class AIMv2Model(torch.nn.Module): def __init__( self, - config: AIMv2Config, + config: Aimv2VisionConfig, quant_config: QuantizationConfig | None, *, require_post_norm: bool | None = None, diff --git a/vllm/model_executor/models/bailing_moe_v3_vl.py b/vllm/model_executor/models/bailing_moe_v3_vl.py index a236d0722b..e596c95ddf 100644 --- a/vllm/model_executor/models/bailing_moe_v3_vl.py +++ b/vllm/model_executor/models/bailing_moe_v3_vl.py @@ -26,6 +26,7 @@ from vllm.multimodal import MULTIMODAL_REGISTRY from vllm.multimodal.inputs import ( MultiModalFeatureSpec, MultiModalFieldConfig, + MultiModalKwargsItem, MultiModalKwargsItems, ) from vllm.multimodal.parse import MultiModalDataItems @@ -520,8 +521,12 @@ class BailingMoeV3VLForConditionalGeneration( tower_model="visual.", ) - def get_num_mm_encoder_tokens(self, num_image_tokens: int) -> int: - return num_image_tokens * self.visual.spatial_merge_size**2 - - def get_num_mm_connector_tokens(self, num_vision_tokens: int) -> int: - return num_vision_tokens // self.visual.spatial_merge_size**2 + def get_mm_lora_token_counts( + self, + *, + modality: str, + mm_kwargs: MultiModalKwargsItem | None, + num_mm_embeds: int, + ) -> tuple[int, int | None]: + del modality, mm_kwargs + return num_mm_embeds * self.visual.spatial_merge_size**2, num_mm_embeds diff --git a/vllm/model_executor/models/cohere2_moe.py b/vllm/model_executor/models/cohere2_moe.py index 7c17edc2a1..3a4d8ef039 100644 --- a/vllm/model_executor/models/cohere2_moe.py +++ b/vllm/model_executor/models/cohere2_moe.py @@ -403,20 +403,6 @@ class Cohere2MoeModel(nn.Module, EagleModelMixin): config.vocab_size, config.hidden_size ) - # Decoder layers read per-layer MLP layout from config.mlp_layer_types - # (dense MLP vs MoE) and use it for weight loading. Transformers >=5.10 - # populates this field; older versions only expose first_k_dense_replace. - # Normalize here so layer construction below sees a consistent layout. - if getattr(config, "mlp_layer_types", None) is None: - first_k_dense_replace = getattr(config, "first_k_dense_replace", None) - n = config.num_hidden_layers - if first_k_dense_replace is not None: - config.mlp_layer_types = ["dense"] * first_k_dense_replace + [ - "sparse" - ] * (n - first_k_dense_replace) - else: - config.mlp_layer_types = ["sparse"] * n - self.start_layer, self.end_layer, self.layers = make_layers( config.num_hidden_layers, lambda prefix: Cohere2MoeDecoderLayer( diff --git a/vllm/model_executor/models/cohere_compass.py b/vllm/model_executor/models/cohere_compass.py index 8593bedfc0..c8902045bb 100644 --- a/vllm/model_executor/models/cohere_compass.py +++ b/vllm/model_executor/models/cohere_compass.py @@ -82,6 +82,7 @@ from vllm.multimodal.inputs import ( ImageItem, MultiModalFeatureSpec, MultiModalFieldConfig, + MultiModalKwargsItem, MultiModalKwargsItems, ) from vllm.multimodal.parse import ( @@ -2085,24 +2086,16 @@ class CohereCompassForConditionalGeneration( tower_model="visual.", ) - def get_num_mm_encoder_tokens( + def get_mm_lora_token_counts( self, - num_image_tokens: int, - ) -> int: - hf_config = self.config - vision_config = hf_config.vision_config - merge_size = vision_config.spatial_merge_size - - return num_image_tokens * merge_size**2 - - def get_num_mm_connector_tokens( - self, - num_vision_tokens: int, - ) -> int: - hf_config = self.config - vision_config = hf_config.vision_config - merge_size = vision_config.spatial_merge_size - return num_vision_tokens // merge_size**2 + *, + modality: str, + mm_kwargs: MultiModalKwargsItem | None, + num_mm_embeds: int, + ) -> tuple[int, int | None]: + del modality, mm_kwargs + merge_size = self.config.vision_config.spatial_merge_size + return num_mm_embeds * merge_size**2, num_mm_embeds @lru_cache diff --git a/vllm/model_executor/models/config.py b/vllm/model_executor/models/config.py index 8831c5f6c4..7f6a3aef40 100644 --- a/vllm/model_executor/models/config.py +++ b/vllm/model_executor/models/config.py @@ -43,6 +43,26 @@ class DeepseekV32ForCausalLM(VerifyAndUpdateConfig): class GlmMoeDsaForCausalLM(VerifyAndUpdateConfig): @staticmethod def verify_and_update_config(vllm_config: "VllmConfig") -> None: + from vllm.platforms import current_platform + + cache_config = vllm_config.cache_config + if ( + cache_config.cache_dtype == "auto" + and current_platform.is_cuda() + and current_platform.is_device_capability_family(100) + ): + cache_config.cache_dtype = "fp8_e4m3" + logger.info_once("Using fp8 kv-cache for GlmMoeDsaForCausalLM on SM10x") + # MTP shares the target's KV cache layout; a separate draft model + # must not inherit a default it was never validated with. + spec_config = vllm_config.speculative_config + if ( + spec_config is not None + and spec_config.method != "mtp" + and spec_config.kv_cache_dtype is None + ): + spec_config.kv_cache_dtype = "auto" + # For Glm-Moe-DSA, qrep + a2a is better than the default all-gather + ag-rs # in most cases. vllm_config.parallel_config.set_dcp_defaults( @@ -270,6 +290,31 @@ class Gemma4Config(VerifyAndUpdateConfig): ) +class EmbeddingGemma2ModelConfig(Gemma4Config): + @staticmethod + def verify_and_update_config(vllm_config: "VllmConfig") -> None: + Gemma4Config.verify_and_update_config(vllm_config) + from vllm.v1.attention.backends.registry import AttentionBackendEnum + + attn_config = vllm_config.attention_config + if attn_config.backend is None: + attn_config.backend = AttentionBackendEnum.TRITON_ATTN + logger.info( + "EmbeddingGemma2: attention backend not specified; defaulting to " + "TRITON_ATTN (validated backend for heterogeneous head dimensions)." + ) + else: + logger.info( + "EmbeddingGemma2: using attention backend %s", attn_config.backend + ) + + model_config = vllm_config.model_config + if model_config.max_model_len > 8192 and getattr( + model_config, "original_max_model_len", None + ) in (None, -1): + model_config.max_model_len = 8192 + + class DiffusionGemmaModelForBlockDiffusionConfig(VerifyAndUpdateConfig): @classmethod def verify_and_update_config(cls, vllm_config: "VllmConfig") -> None: @@ -1013,6 +1058,7 @@ MODELS_CONFIG_MAP: dict[str, type[VerifyAndUpdateConfig]] = { "DeepseekV41ForCausalLM": DeepseekV4ForCausalLMConfig, "DeepseekV32ForCausalLM": DeepseekV32ForCausalLM, "DiffusionGemmaForBlockDiffusion": DiffusionGemmaModelForBlockDiffusionConfig, # noqa: E501 + "EmbeddingGemma2Model": EmbeddingGemma2ModelConfig, "Ernie4_5_VLMoeForConditionalGeneration": Ernie4_5_VLMoeForConditionalGenerationConfig, # noqa: E501 "FalconMambaForCausalLM": MambaModelConfig, "Gemma3TextModel": Gemma3TextModelConfig, diff --git a/vllm/model_executor/models/cosmos3.py b/vllm/model_executor/models/cosmos3.py index 7e4e827457..5023266351 100644 --- a/vllm/model_executor/models/cosmos3.py +++ b/vllm/model_executor/models/cosmos3.py @@ -2,11 +2,28 @@ # SPDX-FileCopyrightText: Copyright contributors to the vLLM project import regex +from transformers import Cosmos3OmniConfig -from vllm.model_executor.models.qwen3_vl import Qwen3VLForConditionalGeneration +from vllm.model_executor.models.qwen3_vl import ( + Qwen3VLDummyInputsBuilder, + Qwen3VLForConditionalGeneration, + Qwen3VLMultiModalProcessor, + Qwen3VLProcessingInfo, +) from vllm.model_executor.models.utils import WeightsMapper +from vllm.multimodal import MULTIMODAL_REGISTRY +class Cosmos3ProcessingInfo(Qwen3VLProcessingInfo): + def get_hf_config(self) -> Cosmos3OmniConfig: + return self.ctx.get_hf_config(Cosmos3OmniConfig) + + +@MULTIMODAL_REGISTRY.register_processor( + Qwen3VLMultiModalProcessor, + info=Cosmos3ProcessingInfo, + dummy_inputs=Qwen3VLDummyInputsBuilder, +) class Cosmos3ForConditionalGeneration(Qwen3VLForConditionalGeneration): # Cosmos3 unified checkpoints store a Qwen3-VL understanding tower # alongside a generation tower in a flat key layout. This mapper drops diff --git a/vllm/model_executor/models/cosmos3_edge.py b/vllm/model_executor/models/cosmos3_edge.py index ae65e25c90..c10fafd0b1 100644 --- a/vllm/model_executor/models/cosmos3_edge.py +++ b/vllm/model_executor/models/cosmos3_edge.py @@ -5,7 +5,7 @@ from collections.abc import Iterable import torch import torch.nn as nn -from transformers import ProcessorMixin +from transformers import NemotronHConfig, ProcessorMixin from vllm.config import VllmConfig from vllm.model_executor.layers.layernorm import RMSNorm @@ -20,7 +20,6 @@ from vllm.model_executor.models.module_mapping import MultiModelKeys from vllm.multimodal import MULTIMODAL_REGISTRY from vllm.multimodal.inputs import MultiModalFeatureSpec from vllm.sequence import IntermediateTensors -from vllm.transformers_utils.configs.nemotron_h import NemotronHConfig from vllm.utils.torch_utils import async_tensor_h2d from .interfaces import ( diff --git a/vllm/model_executor/models/deepseek_v2.py b/vllm/model_executor/models/deepseek_v2.py index 70d87529f6..a1a919fae6 100644 --- a/vllm/model_executor/models/deepseek_v2.py +++ b/vllm/model_executor/models/deepseek_v2.py @@ -396,6 +396,7 @@ class DeepseekV2MoE(nn.Module): if self.is_fused_shared_expert_enabled else None, fuse_shared_experts=self.is_fused_shared_expert_enabled, + shared_expert_prefix=f"{prefix}.shared_experts", router_logits_dtype=self.gate.out_dtype, ) diff --git a/vllm/model_executor/models/diffusion_gemma.py b/vllm/model_executor/models/diffusion_gemma.py index 404c154b54..7c64980b57 100644 --- a/vllm/model_executor/models/diffusion_gemma.py +++ b/vllm/model_executor/models/diffusion_gemma.py @@ -16,7 +16,7 @@ via Gemma4MultimodalEmbedder. from __future__ import annotations -from collections.abc import Iterable, Mapping, Sequence +from collections.abc import Iterable, Sequence from types import SimpleNamespace from typing import Any @@ -25,7 +25,7 @@ import torch import torch._dynamo from torch import nn from torch.nn import functional as F -from transformers import AutoModel +from transformers import AutoModel, DiffusionGemmaConfig from vllm.config import VllmConfig from vllm.config.compilation import CUDAGraphMode @@ -113,26 +113,13 @@ class DiffusionGemmaSelfConditioning(nn.Module): class DiffusionGemmaProcessingInfo(Gemma4ProcessingInfo): - """Processing info for DiffusionGemma. + """Processing info for DiffusionGemma, which has no audio tower.""" - Overrides ``get_hf_config`` to accept ``DiffusionGemmaConfig`` - (which inherits from ``PreTrainedConfig``, not ``Gemma4Config``). - Supports image and video modalities. - """ + def get_hf_config(self) -> DiffusionGemmaConfig: + return self.ctx.get_hf_config(DiffusionGemmaConfig) - def get_hf_config(self): - # DiffusionGemmaConfig doesn't inherit from Gemma4Config, so we - # accept any PreTrainedConfig here. - return self.ctx.get_hf_config() - - def get_supported_mm_limits(self) -> Mapping[str, int | None]: - # DiffusionGemma supports image and video inputs. - return {"image": None, "video": None} - - def get_mm_max_tokens_per_item( - self, seq_len: int, mm_counts: Mapping[str, int] - ) -> Mapping[str, int] | None: - return super().get_mm_max_tokens_per_item(seq_len, mm_counts) + def has_audio(self) -> bool: + return False @torch.compile(dynamic=True, backend=current_platform.simple_compile_backend) @@ -445,11 +432,6 @@ def _mask_rows_to_allowed( return logits if out is None else out -# Tiles specialize the graph on batch size 1, on a tile as wide as the canvas, -# on compute_sc and on sizes that coincide with the state buffers. That set -# is small but passes Dynamo's default of 8, after which every step would run -# eager: about twice as slow for a self-conditioned step. -@torch._dynamo.config.patch(recompile_limit=64) @torch.compile(dynamic=True, backend=current_platform.simple_compile_backend) def _denoise_temperature( step_tensor: torch.Tensor, @@ -464,6 +446,12 @@ def _denoise_temperature( return t_min + (t_max - t_min) * (remaining / max_denoising_steps) +# Tiles specialize the graph on batch size 1, on a tile as wide as the canvas, +# on compute_sc and on sizes that coincide with the state buffers. That set +# is small but passes Dynamo's default of 8, after which every step would run +# eager: about twice as slow for a self-conditioned step. +@torch._dynamo.config.patch(recompile_limit=64) +@torch.compile(dynamic=True, backend=current_platform.simple_compile_backend) def _compiled_sample_step( # Per-position statistics of the temperature-scaled logits, from # sample_row_stats: [num_decode, CL] each, and the softmax diff --git a/vllm/model_executor/models/embedding_gemma2.py b/vllm/model_executor/models/embedding_gemma2.py new file mode 100644 index 0000000000..0f92aeaa37 --- /dev/null +++ b/vllm/model_executor/models/embedding_gemma2.py @@ -0,0 +1,1139 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""EmbeddingGemma2: bidirectional Gemma4-derived multimodal embedding model.""" + +import dataclasses +from collections.abc import Iterable, Mapping, Sequence +from typing import Any, cast + +import torch +from torch import nn +from transformers import AutoModel, BatchFeature, PreTrainedConfig +from transformers.models.embedding_gemma2.configuration_embedding_gemma2 import ( + EmbeddingGemma2Config, + EmbeddingGemma2TextConfig, +) +from transformers.models.embedding_gemma2.processing_embedding_gemma2 import ( + EmbeddingGemma2Processor, +) +from transformers.video_utils import VideoMetadata + +from vllm.config import VllmConfig +from vllm.distributed import get_tensor_model_parallel_world_size +from vllm.logger import init_logger +from vllm.model_executor.layers.activation import get_act_fn +from vllm.model_executor.layers.attention import Attention +from vllm.model_executor.layers.layernorm import RMSNorm +from vllm.model_executor.layers.linear import ( + ColumnParallelLinear, + QKVParallelLinear, + ReplicatedLinear, + RowParallelLinear, +) +from vllm.model_executor.layers.pooler import DispatchPooler +from vllm.model_executor.layers.quantization import QuantizationConfig +from vllm.model_executor.layers.rotary_embedding import get_rope +from vllm.model_executor.layers.vocab_parallel_embedding import ( + VocabParallelEmbedding, +) +from vllm.model_executor.models.gemma4 import Gemma4MLP +from vllm.model_executor.models.gemma4_mm import ( + _SUPPORTED_SOFT_TOKENS, + Gemma4DummyInputsBuilder, + Gemma4ForConditionalGeneration, + Gemma4MultimodalEmbedder, + Gemma4MultiModalProcessor, + Gemma4ProcessingInfo, + _get_max_soft_tokens, +) +from vllm.model_executor.models.interfaces import SupportsMultiModal +from vllm.model_executor.models.interfaces_base import ( + VllmModelForPooling, + default_pooling_type, +) +from vllm.model_executor.models.transformers.utils import ( + recursive_replace_linear, +) +from vllm.model_executor.models.utils import ( + AutoWeightsLoader, + WeightsMapper, + maybe_prefix, +) +from vllm.multimodal import MULTIMODAL_REGISTRY +from vllm.multimodal.inputs import ( + MultiModalKwargsItems, + VideoItem, +) +from vllm.multimodal.parse import ( + AudioProcessorItems, + ImageProcessorItems, + MultiModalDataItems, +) +from vllm.multimodal.processing import ( + BaseProcessingInfo, + PromptReplacement, + PromptUpdate, + PromptUpdateDetails, +) +from vllm.sequence import IntermediateTensors +from vllm.v1.attention.backend import AttentionType + +logger = init_logger(__name__) + +_WINDOW_OFFSET = 1 # HF masks |q-k| <= W; vLLM ENCODER_ONLY masks <= W-1. +_VIDEO_FRAME_WIDTH = 672 +_VIDEO_FRAME_HEIGHT = 480 +_VIDEO_MAX_FRAMES = 32 +_VIDEO_MAX_SOFT_TOKENS = 140 + + +class EmbeddingGemma2Attention(nn.Module): + """Bidirectional attention with sliding window offset for EmbeddingGemma2.""" + + def __init__( + self, + config: EmbeddingGemma2TextConfig, + layer_cfg, + layer_idx: int, + cache_config, + quant_config: QuantizationConfig | None, + max_position: int, + prefix: str, + ): + super().__init__() + self.config = config + self.layer_idx = layer_idx + tp = get_tensor_model_parallel_world_size() + self.num_heads = config.num_attention_heads // tp + self.num_kv_heads = max(1, layer_cfg.num_key_value_heads // tp) + self.head_dim = layer_cfg.head_dim + self.q_size = self.num_heads * self.head_dim + self.kv_size = self.num_kv_heads * self.head_dim + + self.qkv_proj = QKVParallelLinear( + config.hidden_size, + self.head_dim, + config.num_attention_heads, + layer_cfg.num_key_value_heads, + bias=config.attention_bias, + quant_config=quant_config, + prefix=f"{prefix}.qkv_proj", + ) + self.o_proj = RowParallelLinear( + config.num_attention_heads * self.head_dim, + config.hidden_size, + bias=config.attention_bias, + quant_config=quant_config, + prefix=f"{prefix}.o_proj", + ) + self.q_norm = RMSNorm(self.head_dim, eps=config.rms_norm_eps) + self.k_norm = RMSNorm(self.head_dim, eps=config.rms_norm_eps) + self.v_norm = RMSNorm(self.head_dim, eps=config.rms_norm_eps, has_weight=False) + + assert config.layer_types is not None + layer_type = config.layer_types[layer_idx] + self.is_sliding = layer_type == "sliding_attention" + assert config.rope_parameters is not None + rope_parameters = dict(config.rope_parameters[layer_type]) + + self.rotary_emb = get_rope( + self.head_dim, + max_position=max_position, + rope_parameters=rope_parameters, + is_neox_style=True, + ) + + window = config.sliding_window if self.is_sliding else None + self.attn = Attention( + self.num_heads, + self.head_dim, + scale=1.0, + num_kv_heads=self.num_kv_heads, + cache_config=cache_config, + quant_config=quant_config, + per_layer_sliding_window=(window + _WINDOW_OFFSET) + if window is not None + else None, + attn_type=AttentionType.ENCODER_ONLY, + prefix=f"{prefix}.attn", + ) + + def forward( + self, positions: torch.Tensor, hidden_states: torch.Tensor + ) -> torch.Tensor: + qkv, _ = self.qkv_proj(hidden_states) + q, k, v = qkv.split([self.q_size, self.kv_size, self.kv_size], dim=-1) + + q = q.unflatten(-1, (self.num_heads, self.head_dim)) + q = self.q_norm(q) + q = q.flatten(-2, -1) + + k = k.unflatten(-1, (self.num_kv_heads, self.head_dim)) + k = self.k_norm(k) + k = k.flatten(-2, -1) + q, k = self.rotary_emb(positions, q, k) + + v = v.unflatten(-1, (self.num_kv_heads, self.head_dim)) + v = self.v_norm(v) + v = v.flatten(-2, -1) + + attn_output = self.attn(q, k, v) + out, _ = self.o_proj(attn_output) + return out + + +class EmbeddingGemma2PLEBlock(nn.Module): + """Per-layer-embedding residual block (mirrors HF ple_block).""" + + def __init__( + self, + config: EmbeddingGemma2TextConfig, + quant_config: QuantizationConfig | None = None, + prefix: str = "", + ): + super().__init__() + self.hidden_size = config.hidden_size + self.hidden_size_per_layer_input = config.hidden_size_per_layer_input + + self.per_layer_input_gate = ReplicatedLinear( + self.hidden_size, + self.hidden_size_per_layer_input, + bias=False, + quant_config=quant_config, + prefix=f"{prefix}.per_layer_input_gate", + return_bias=False, + ) + self.per_layer_projection = ReplicatedLinear( + self.hidden_size_per_layer_input, + self.hidden_size, + bias=False, + quant_config=quant_config, + prefix=f"{prefix}.per_layer_projection", + return_bias=False, + ) + self.post_per_layer_input_norm = RMSNorm( + self.hidden_size, eps=config.rms_norm_eps + ) + self.act_fn = get_act_fn(config.hidden_activation) + + def forward( + self, hidden_states: torch.Tensor, per_layer_input: torch.Tensor + ) -> torch.Tensor: + residual = hidden_states + gate = self.per_layer_input_gate(hidden_states) + gate = self.act_fn(gate) + gated = gate * per_layer_input + proj = self.per_layer_projection(gated) + proj = self.post_per_layer_input_norm(proj) + return residual + proj + + +class EmbeddingGemma2DecoderLayer(nn.Module): + """Transformer decoder layer for EmbeddingGemma2 text model with PLE gating.""" + + def __init__( + self, + config: EmbeddingGemma2TextConfig, + idx: int, + cache_config, + quant_config: QuantizationConfig | None, + max_position: int, + prefix: str, + ): + super().__init__() + lc = config.per_layer_config[idx] + self.self_attn = EmbeddingGemma2Attention( + config, + lc, + idx, + cache_config, + quant_config, + max_position, + f"{prefix}.self_attn", + ) + self.mlp = Gemma4MLP( + hidden_size=config.hidden_size, + intermediate_size=config.intermediate_size, + hidden_activation=config.hidden_activation, + quant_config=quant_config, + prefix=f"{prefix}.mlp", + ) + self.ple_block = EmbeddingGemma2PLEBlock( + config, + quant_config=quant_config, + prefix=f"{prefix}.ple_block", + ) + self.input_layernorm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps) + self.post_attention_layernorm = RMSNorm( + config.hidden_size, eps=config.rms_norm_eps + ) + self.pre_feedforward_layernorm = RMSNorm( + config.hidden_size, eps=config.rms_norm_eps + ) + self.post_feedforward_layernorm = RMSNorm( + config.hidden_size, eps=config.rms_norm_eps + ) + self.layer_scalar: torch.Tensor + self.register_buffer("layer_scalar", torch.ones(1), persistent=True) + + def forward( + self, + positions: torch.Tensor, + h: torch.Tensor, + per_layer_input: torch.Tensor, + ) -> torch.Tensor: + r = h + h = ( + self.post_attention_layernorm( + self.self_attn(positions, self.input_layernorm(h)) + ) + + r + ) + r = h + h = ( + self.post_feedforward_layernorm(self.mlp(self.pre_feedforward_layernorm(h))) + + r + ) + h = self.ple_block(h, per_layer_input) + return h * cast(torch.Tensor, self.layer_scalar) + + +class EmbeddingGemma2TextPLE(nn.Module): + """Per-layer embedding (PLE) projection and normalization module.""" + + def __init__( + self, + config: EmbeddingGemma2TextConfig, + quant_config: QuantizationConfig | None = None, + prefix: str = "", + ): + super().__init__() + self.num_hidden_layers = config.num_hidden_layers + self.hidden_size_per_layer_input = config.hidden_size_per_layer_input + self.per_layer_model_projection = ColumnParallelLinear( + config.hidden_size, + config.num_hidden_layers * config.hidden_size_per_layer_input, + bias=False, + gather_output=True, + quant_config=quant_config, + prefix=f"{prefix}.per_layer_model_projection", + ) + self.per_layer_model_projection_scale = config.hidden_size**-0.5 + self.per_layer_projection_norm = RMSNorm( + config.hidden_size_per_layer_input, eps=config.rms_norm_eps + ) + + def forward(self, inputs_embeds: torch.Tensor) -> torch.Tensor: + proj, _ = self.per_layer_model_projection(inputs_embeds) + proj = proj * self.per_layer_model_projection_scale + proj = proj.reshape( + *inputs_embeds.shape[:-1], + self.num_hidden_layers, + self.hidden_size_per_layer_input, + ) + return self.per_layer_projection_norm(proj) + + +class EmbeddingGemma2TextModel(nn.Module): + """Text embedding backbone for EmbeddingGemma2 with bidirectional attention.""" + + def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""): + super().__init__() + cfg = vllm_config.model_config.hf_config.text_config + self.config = cfg + max_pos = min( + cfg.max_position_embeddings, vllm_config.model_config.max_model_len + ) + self.embed_tokens = VocabParallelEmbedding( + cfg.vocab_size, + cfg.hidden_size, + quant_config=vllm_config.quant_config, + prefix=f"{prefix}.embed_tokens", + ) + self.layers = nn.ModuleList( + [ + EmbeddingGemma2DecoderLayer( + cfg, + i, + vllm_config.cache_config, + vllm_config.quant_config, + max_pos, + f"{prefix}.layers.{i}", + ) + for i in range(cfg.num_hidden_layers) + ] + ) + self.norm = RMSNorm(cfg.hidden_size, eps=cfg.rms_norm_eps) + self.ple = EmbeddingGemma2TextPLE( + cfg, + quant_config=vllm_config.quant_config, + prefix=f"{prefix}.ple", + ) + self.embedding_projection = ReplicatedLinear( + cfg.hidden_size, + cfg.embedding_dim, + bias=False, + quant_config=vllm_config.quant_config, + prefix=f"{prefix}.embedding_projection", + return_bias=False, + ) + self.embed_scale: torch.Tensor + self.register_buffer( + "embed_scale", + torch.tensor(cfg.hidden_size**0.5), + persistent=False, + ) + + def embed_input_ids(self, input_ids: torch.Tensor) -> torch.Tensor: + embed_scale = cast(torch.Tensor, self.embed_scale) + dtype = cast(torch.dtype, self.embed_tokens.weight.dtype) + return self.embed_tokens(input_ids) * embed_scale.to(dtype) + + def forward( + self, + input_ids: torch.Tensor, + positions: torch.Tensor, + inputs_embeds: torch.Tensor | None = None, + ) -> torch.Tensor: + if inputs_embeds is None: + inputs_embeds = self.embed_input_ids(input_ids) + per_layer = self.ple(inputs_embeds) + h = inputs_embeds + for i, layer in enumerate(self.layers): + h = layer(positions, h, per_layer[..., i, :]) + h = self.norm(h) + h = self.embedding_projection(h) + return h + + +class EmbeddingGemma2ProcessingInfo(Gemma4ProcessingInfo): + """Processing information and token budgeting for EmbeddingGemma2.""" + + def get_hf_config(self) -> EmbeddingGemma2Config: + return self.ctx.get_hf_config(EmbeddingGemma2Config) + + def get_hf_processor( # type: ignore[override] + self, **kwargs: object + ) -> EmbeddingGemma2Processor: + return self.ctx.get_hf_processor(EmbeddingGemma2Processor, **kwargs) + + def get_default_tok_params(self): + # Base-class behavior: add_special_tokens=True (BOS + EOS) + return BaseProcessingInfo.get_default_tok_params(self) + + def get_mm_max_tokens_per_item( + self, seq_len: int, mm_counts: Mapping[str, int] + ) -> Mapping[str, int] | None: + config = self.get_hf_config() + merged_kwargs = self.ctx.get_merged_mm_kwargs({}) + val, _ = _get_max_soft_tokens(merged_kwargs) + if isinstance(val, int) and val in _SUPPORTED_SOFT_TOKENS: + tokens_per_image = val + 2 + else: + tokens_per_image = ( + getattr(config.vision_config, "default_output_length", 256) + 2 + ) + tokens: dict[str, int] = {"image": tokens_per_image} + if config.audio_config is not None: + processor = self.get_hf_processor() + tokens["audio"] = processor.audio_seq_length + 2 + + processor = self.get_hf_processor() + video_proc = getattr(processor, "video_processor", None) + max_frames = getattr(video_proc, "max_frames", 32) + mm_config = self.ctx.model_config.get_multimodal_config() + + # The loader's max_frames cap is 128 when media_io_kwargs can raise it + video_media_io = ( + getattr(mm_config, "media_io_kwargs", {}).get("video", {}) + if mm_config + else {} + ) + media_io_max_frames = video_media_io.get("max_frames") + if media_io_max_frames is not None: + max_frames = min(int(media_io_max_frames), 128) + elif "max_frames" in merged_kwargs: + max_frames = min(int(merged_kwargs["max_frames"]), 128) + + video_opts = mm_config.limit_per_prompt.get("video") if mm_config else None + if video_opts is not None: + num_frames_opt = getattr(video_opts, "num_frames", None) + if isinstance(num_frames_opt, int): + max_frames = min(max_frames, num_frames_opt) + + video_max_soft = merged_kwargs.get( + "max_soft_tokens", + getattr(video_proc, "max_soft_tokens", _VIDEO_MAX_SOFT_TOKENS), + ) + if video_max_soft is not None and video_max_soft not in _SUPPORTED_SOFT_TOKENS: + raise ValueError( + f"Unsupported max_soft_tokens={video_max_soft} for video. " + f"Valid values are {_SUPPORTED_SOFT_TOKENS}." + ) + if not isinstance(video_max_soft, int): + video_max_soft = _VIDEO_MAX_SOFT_TOKENS + + # Token allocation per video frame verified against HF EmbeddingGemma2Processor: + # - Without timestamps: boi (1) + video_token * num_soft_tokens + eoi (1) + # Total = num_soft_tokens + 2 + # - With timestamps: timestamp prefix string (e.g. ' 01:23 ' = up to 7 tokens) + # Total = num_soft_tokens + 2 + 7 + add_ts = getattr(video_proc, "add_timestamps", False) or merged_kwargs.get( + "add_timestamps", False + ) + ts_slack = 7 if add_ts else 0 + tokens["video"] = max_frames * (video_max_soft + 2 + ts_slack) + return tokens + + def get_image_repl( # type: ignore[override] + self, + *, + image_width: int, + image_height: int, + processor: EmbeddingGemma2Processor | None, + max_soft_tokens: int | None = None, + ) -> PromptUpdateDetails: + if processor is None: + processor = self.get_hf_processor() + + num_soft = self._compute_num_soft_tokens( + image_width, + image_height, + max_soft_tokens=max_soft_tokens, + ) + config = self.get_hf_config() + assert config.boi_token_id is not None + assert processor.image_token_id is not None + assert config.eoi_token_id is not None + token_ids = ( + [config.boi_token_id] + + [processor.image_token_id] * num_soft + + [config.eoi_token_id] + ) + return PromptUpdateDetails.select_token_id(token_ids, processor.image_token_id) + + def get_audio_repl( # type: ignore[override] + self, + *, + audio_len: int, + processor: EmbeddingGemma2Processor | None, + ) -> PromptUpdateDetails: + if processor is None: + processor = self.get_hf_processor() + sampling_rate = processor.feature_extractor.sampling_rate + num_tokens = self._compute_audio_num_tokens( + audio_len, sampling_rate, processor.audio_seq_length + ) + config = self.get_hf_config() + assert config.boa_token_id is not None + assert processor.audio_token_id is not None + assert config.eoa_token_index is not None + token_ids = ( + [config.boa_token_id] + + [processor.audio_token_id] * num_tokens + + [config.eoa_token_index] + ) + return PromptUpdateDetails.select_token_id(token_ids, processor.audio_token_id) + + def get_video_repl( # type: ignore[override] + self, + *, + timestamps: list[float], + num_soft_tokens_per_frame: list[int], + processor: EmbeddingGemma2Processor, + add_timestamps: bool = False, + ) -> PromptUpdateDetails: + tokenizer = self.ctx.get_tokenizer() + config = self.get_hf_config() + + boi_token_id = config.boi_token_id + eoi_token_id = config.eoi_token_id + video_token_id = processor.video_token_id + assert boi_token_id is not None + assert eoi_token_id is not None + assert video_token_id is not None + + all_token_ids: list[int] = [] + if not add_timestamps: + for n_tokens in num_soft_tokens_per_frame: + all_token_ids.append(boi_token_id) + all_token_ids.extend([video_token_id] * n_tokens) + all_token_ids.append(eoi_token_id) + else: + for i, (ts, n_tokens) in enumerate( + zip(timestamps, num_soft_tokens_per_frame) + ): + minutes = int(ts // 60) + seconds = int(ts % 60) + ts_str = f"{minutes:02d}:{seconds:02d}" + prefix = f" {ts_str} " if i > 0 else f"{ts_str} " + ts_token_ids = tokenizer.encode(prefix, add_special_tokens=False) + all_token_ids.extend(ts_token_ids) + all_token_ids.append(boi_token_id) + all_token_ids.extend([video_token_id] * n_tokens) + all_token_ids.append(eoi_token_id) + + return PromptUpdateDetails.select_token_id(all_token_ids, video_token_id) + + +class EmbeddingGemma2DummyInputsBuilder(Gemma4DummyInputsBuilder): + """Dummy input builder for EmbeddingGemma2 profiling and testing.""" + + def get_dummy_text(self, mm_counts: Mapping[str, int]) -> str: + num_images = mm_counts.get("image", 0) + num_audios = mm_counts.get("audio", 0) + num_videos = mm_counts.get("video", 0) + processor = self.info.get_hf_processor() + text = ( + (processor.image_token * num_images) + + (processor.audio_token * num_audios if processor.audio_token else "") + + (processor.video_token * num_videos) + ) + return text + + def _get_dummy_videos( # type: ignore[override] + self, + *, + width: int, + height: int, + num_frames: int, + num_videos: int, + overrides=None, + ) -> list[VideoItem]: + num_frames = _VIDEO_MAX_FRAMES + width = _VIDEO_FRAME_WIDTH + height = _VIDEO_FRAME_HEIGHT + videos = super(Gemma4DummyInputsBuilder, self)._get_dummy_videos( + width=width, + height=height, + num_frames=num_frames, + num_videos=num_videos, + overrides=overrides, + ) + video_items: list[VideoItem] = [] + for video in videos: + video_num_frames = video.shape[0] + video_metadata = { + "fps": 1.0, + "duration": float(video_num_frames), + "total_num_frames": video_num_frames, + "frames_indices": list(range(video_num_frames)), + "video_backend": "opencv", + "do_sample_frames": False, + } + video_items.append((video, video_metadata)) + return video_items + + +class EmbeddingGemma2MultiModalProcessor(Gemma4MultiModalProcessor): + """Multimodal input processor for EmbeddingGemma2.""" + + info: EmbeddingGemma2ProcessingInfo + + def _apply_hf_processor_main( + self, + mm_items: MultiModalDataItems, + hf_kwargs: Mapping[str, object], + ) -> BatchFeature: + hf_data, hf_kwargs, passthrough_data = self._get_hf_mm_inputs( + mm_items, hf_kwargs + ) + + if not hf_data: + return self._finalize_hf_mm_data( + hf_data, hf_kwargs, passthrough_data, BatchFeature() + ) + + prompt_text = hf_data.pop("text") + assert isinstance(prompt_text, str) + + processor = self.info.get_hf_processor() + video_outputs: dict[str, Any] = {} + + if videos := hf_data.pop("videos", []): + assert isinstance(videos, list) + + all_video_pixel_values: list[torch.Tensor] = [] + all_video_position_ids: list[torch.Tensor] = [] + video_num_soft_tokens_per_video: list[list[int]] = [] + video_timestamps_per_video: list[list[float]] = [] + video_frame_counts: list[int] = [] + + for item in videos: + video_array, metadata = item + + if hf_kwargs.get("num_frames") not in (-1, None): + raise ValueError( + "EmbeddingGemma2 does not accept num_frames; " + "use max_frames/fps." + ) + + if isinstance(metadata, dict): + if not metadata.get("do_sample_frames", True) and any( + k in hf_kwargs + for k in ("fps", "max_frames", "overflow_strategy") + ): + logger.warning_once( + "For file/URL-loaded videos, " + "fps/max_frames/overflow_strategy in mm_processor_kwargs " + "have no effect because frames are decoded and sampled by " + "media_io. Please specify these sampling options via " + "media_io_kwargs instead." + ) + valid_meta_fields = { + f.name for f in dataclasses.fields(VideoMetadata) + } + meta_dict = { + k: v for k, v in metadata.items() if k in valid_meta_fields + } + metadata_obj = VideoMetadata(**meta_dict) + else: + if getattr(metadata, "do_sample_frames", True) is False and any( + k in hf_kwargs + for k in ("fps", "max_frames", "overflow_strategy") + ): + logger.warning_once( + "For file/URL-loaded videos, " + "fps/max_frames/overflow_strategy in mm_processor_kwargs " + "have no effect because frames are decoded and sampled by " + "media_io. Please specify these sampling options via " + "media_io_kwargs instead." + ) + metadata_obj = metadata + + video_mm_kwargs = dict(hf_kwargs) + video_mm_kwargs.pop("num_frames", None) + video_mm_kwargs["return_metadata"] = True + if isinstance(metadata, dict) and "do_sample_frames" in metadata: + video_mm_kwargs["do_sample_frames"] = metadata["do_sample_frames"] + elif hasattr(metadata, "do_sample_frames"): + video_mm_kwargs["do_sample_frames"] = metadata.do_sample_frames + + single_video_out = self.info.ctx.call_hf_processor( + self.info.get_hf_processor(**video_mm_kwargs), + dict( + text=processor.video_token, + videos=[video_array], + video_metadata=[metadata_obj], + ), + video_mm_kwargs, + ) + + all_video_pixel_values.append(single_video_out["pixel_values_videos"]) + all_video_position_ids.append(single_video_out["video_position_ids"]) + + num_frames = int(single_video_out["num_frames_per_video"][0]) + video_frame_counts.append(num_frames) + + num_soft = single_video_out.get("num_soft_tokens_per_video") + if num_soft is not None: + soft_val = num_soft[0] + if isinstance(soft_val, torch.Tensor): + soft_val = soft_val.item() + soft_tokens_list = [int(soft_val)] * num_frames + elif "video_position_ids" in single_video_out and num_frames > 0: + pos = single_video_out["video_position_ids"] + pooling_k2 = ( + getattr(processor.video_processor, "pooling_kernel_size", 3) + ** 2 + ) + valid_patches = (pos[0] != -1).all(dim=-1).sum().item() + soft_tokens_list = [valid_patches // pooling_k2] * num_frames + elif "input_ids" in single_video_out and num_frames > 0: + vtok = ( + ( + single_video_out["input_ids"][0] + == processor.tokenizer.video_token_id + ) + .sum() + .item() + ) + soft_tokens_list = [vtok // num_frames] * num_frames + else: + raise ValueError( + f"Unable to determine soft tokens for video frame: " + f"keys={list(single_video_out.keys())}" + ) + + video_num_soft_tokens_per_video.append(soft_tokens_list) + + # Use returned video_metadata if available, else input metadata_obj + ret_meta = single_video_out.get("video_metadata") + meta_for_ts = ( + ret_meta[0] if (ret_meta and len(ret_meta) > 0) else metadata_obj + ) + + add_ts = hf_kwargs.get( + "add_timestamps", + getattr(processor.video_processor, "add_timestamps", False), + ) + if add_ts: + try: + ts = meta_for_ts.timestamps + except ValueError as e: + raise ValueError( + "Asked to build a prompt with frame timestamps, but no " + "valid fps/frames_indices was available in video metadata." + ) from e + video_timestamps_per_video.append([float(t) for t in ts]) + else: + try: + ts = meta_for_ts.timestamps + video_timestamps_per_video.append([float(t) for t in ts]) + except ValueError: + video_timestamps_per_video.append([0.0] * num_frames) + + video_outputs = { + "pixel_values_videos": torch.cat(all_video_pixel_values, dim=0), + "pixel_position_ids_videos": torch.cat(all_video_position_ids, dim=0), + "video_frame_counts": torch.tensor( + video_frame_counts, dtype=torch.long + ), + "video_num_soft_tokens": video_num_soft_tokens_per_video, + "video_timestamps": video_timestamps_per_video, + } + + if hf_data: + remaining_counts: dict[str, int] = {} + if "images" in hf_data: + images_list = hf_data["images"] + remaining_counts["image"] = ( + len(images_list) if isinstance(images_list, list) else 1 + ) + if "audio" in hf_data: + audio_list = hf_data["audio"] + remaining_counts["audio"] = ( + len(audio_list) if isinstance(audio_list, (list, tuple)) else 1 + ) + remaining_prompt_text = self.dummy_inputs.get_dummy_text(remaining_counts) + + call_kwargs = dict(hf_kwargs) + call_kwargs.pop("num_frames", None) + processed_data = self.info.ctx.call_hf_processor( + self.info.get_hf_processor(**call_kwargs), + dict(text=remaining_prompt_text, **hf_data), + call_kwargs, + ) + else: + processed_data = BatchFeature() + + if "image_position_ids" in processed_data: + processed_data["pixel_position_ids"] = processed_data.pop( + "image_position_ids" + ) + + if "input_features" in processed_data: + input_features = processed_data.pop("input_features") + input_features_mask = processed_data.pop("input_features_mask") + if not isinstance(input_features, list): + unpadded_features = [] + unpadded_masks = [] + for i in range(input_features.shape[0]): + mask = input_features_mask[i] + valid_len = int(mask.sum().item()) + unpadded_features.append(input_features[i, :valid_len]) + unpadded_masks.append(mask[:valid_len]) + processed_data["input_features_padded"] = unpadded_features + processed_data["input_features_mask"] = unpadded_masks + else: + processed_data["input_features_padded"] = input_features + processed_data["input_features_mask"] = input_features_mask + + processed_data.update(video_outputs) + processed_data.pop("num_frames_per_video", None) + processed_data.pop("video_position_ids", None) + + return self._finalize_hf_mm_data( + hf_data, hf_kwargs, passthrough_data, processed_data + ) + + def _get_prompt_updates( + self, + mm_items: MultiModalDataItems, + hf_processor_mm_kwargs: Mapping[str, Any], + out_mm_kwargs: MultiModalKwargsItems, + ) -> Sequence[PromptUpdate]: + hf_processor = self.info.get_hf_processor(**hf_processor_mm_kwargs) + prompt_updates = [] + + if "image" in mm_items: + image_token_id = hf_processor.image_token_id + + def get_replacement_image(item_idx: int): + images = mm_items.get_items("image", ImageProcessorItems) + image_size = images.get_image_size(item_idx) + merged_kwargs = self.info.ctx.get_merged_mm_kwargs( + hf_processor_mm_kwargs, + ) + val, _ = _get_max_soft_tokens(merged_kwargs) + max_soft_tokens = ( + val + if isinstance(val, int) and val in _SUPPORTED_SOFT_TOKENS + else None + ) + return self.info.get_image_repl( + image_width=image_size.width, + image_height=image_size.height, + processor=hf_processor, + max_soft_tokens=max_soft_tokens, + ) + + prompt_updates.append( + PromptReplacement( + modality="image", + target=[image_token_id], + replacement=get_replacement_image, + ) + ) + + if "video" in mm_items: + video_token_id = hf_processor.video_token_id + assert video_token_id is not None + + def get_replacement_video(item_idx: int): + out_item = out_mm_kwargs["video"][item_idx] + raw_ts = out_item["video_timestamps"].data + if isinstance(raw_ts, torch.Tensor): + ts_list: list[Any] = raw_ts.tolist() + elif isinstance(raw_ts, (list, tuple)): + ts_list = list(raw_ts) + else: + ts_list = [raw_ts] + timestamps = [float(t) for t in ts_list] + + raw_soft = out_item["video_num_soft_tokens"].data + if isinstance(raw_soft, torch.Tensor): + soft_list: list[Any] = raw_soft.tolist() + elif isinstance(raw_soft, (list, tuple)): + soft_list = list(raw_soft) + else: + soft_list = [raw_soft] + num_soft = [int(n) for n in soft_list] + + add_ts = hf_processor_mm_kwargs.get( + "add_timestamps", + getattr(hf_processor.video_processor, "add_timestamps", False), + ) + return self.info.get_video_repl( + timestamps=timestamps, + num_soft_tokens_per_frame=num_soft, + processor=hf_processor, + add_timestamps=add_ts, + ) + + prompt_updates.append( + PromptReplacement( + modality="video", + target=[video_token_id], + replacement=get_replacement_video, + ) + ) + + if "audio" in mm_items: + audio_token_id = hf_processor.audio_token_id + assert audio_token_id is not None + + def get_replacement_audio(item_idx: int): + audios = mm_items.get_items("audio", AudioProcessorItems) + audio_len = audios.get_audio_length(item_idx) + return self.info.get_audio_repl( + audio_len=audio_len, + processor=hf_processor, + ) + + prompt_updates.append( + PromptReplacement( + modality="audio", + target=[audio_token_id], + replacement=get_replacement_audio, + ) + ) + + return prompt_updates + + +def _embedding_gemma2_weights_mapper( + config_or_layers: PreTrainedConfig | int | None = None, +) -> WeightsMapper: + return WeightsMapper( + orig_to_new_stacked={ + ".self_attn.q_proj.weight": (".self_attn.qkv_proj.weight", "q"), + ".self_attn.k_proj.weight": (".self_attn.qkv_proj.weight", "k"), + ".self_attn.v_proj.weight": (".self_attn.qkv_proj.weight", "v"), + ".self_attn.q_proj.bias": (".self_attn.qkv_proj.bias", "q"), + ".self_attn.k_proj.bias": (".self_attn.qkv_proj.bias", "k"), + ".self_attn.v_proj.bias": (".self_attn.qkv_proj.bias", "v"), + ".mlp.gate_proj.weight": (".mlp.gate_up_proj.weight", 0), + ".mlp.up_proj.weight": (".mlp.gate_up_proj.weight", 1), + ".mlp.gate_proj.bias": (".mlp.gate_up_proj.bias", 0), + ".mlp.up_proj.bias": (".mlp.gate_up_proj.bias", 1), + } + ) + + +@MULTIMODAL_REGISTRY.register_processor( + EmbeddingGemma2MultiModalProcessor, + info=EmbeddingGemma2ProcessingInfo, + dummy_inputs=EmbeddingGemma2DummyInputsBuilder, +) +@default_pooling_type(seq_pooling_type="MEAN", tok_pooling_type="ALL") +class EmbeddingGemma2Model(nn.Module, SupportsMultiModal, VllmModelForPooling): + """Multimodal pooling embedding model supporting text, vision, and audio.""" + + is_pooling_model = True + + hf_to_vllm_mapper = _embedding_gemma2_weights_mapper() + packed_modules_mapping = { + "qkv_proj": [ + "q_proj", + "k_proj", + "v_proj", + ], + "gate_up_proj": [ + "gate_proj", + "up_proj", + ], + } + + # TODO(lucianomartins): Extract Gemma4EncoderMixin into a shared module + # once generative model tests are migrated. + # TODO(lucianomartins): Investigate @support_torch_compile integration for encoder. + _parse_and_validate_image_input = ( + Gemma4ForConditionalGeneration._parse_and_validate_image_input + ) + _parse_and_validate_video_input = ( + Gemma4ForConditionalGeneration._parse_and_validate_video_input + ) + _parse_and_validate_audio_input = ( + Gemma4ForConditionalGeneration._parse_and_validate_audio_input + ) + _parse_and_validate_multimodal_inputs = ( + Gemma4ForConditionalGeneration._parse_and_validate_multimodal_inputs + ) + _process_image_input = Gemma4ForConditionalGeneration._process_image_input + _process_video_input = Gemma4ForConditionalGeneration._process_video_input + _process_audio_input = Gemma4ForConditionalGeneration._process_audio_input + embed_multimodal = Gemma4ForConditionalGeneration.embed_multimodal + _encoder_chunk = staticmethod(Gemma4ForConditionalGeneration._encoder_chunk) + + def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""): + nn.Module.__init__(self) + config = vllm_config.model_config.hf_config + quant_config = vllm_config.quant_config + multimodal_config = vllm_config.model_config.multimodal_config + self.config = config + self.quant_config = quant_config + self.multimodal_config = multimodal_config + self.model_dtype = vllm_config.model_config.dtype + self.vllm_config = vllm_config + self.hf_to_vllm_mapper = _embedding_gemma2_weights_mapper(config) + lora_config = vllm_config.lora_config + self._enable_mm_lora = bool( + lora_config is not None and lora_config.enable_tower_connector_lora + ) + + tower_quant: QuantizationConfig | None + if quant_config and quant_config.get_name() in [ + "bitsandbytes", + "torchao", + "compressed-tensors", + ]: + tower_quant = quant_config + else: + vision_cfg = getattr(config, "vision_config", None) + quantizable = ( + vision_cfg is not None + and vision_cfg.hidden_size % 64 == 0 + and vision_cfg.intermediate_size % 64 == 0 + ) + tower_quant = quant_config if quantizable else None + + # ---- Vision tower (shared by image and video) ---- + self.embed_vision: Gemma4MultimodalEmbedder | None + if getattr(config, "vision_config", None) is not None: + with self._mark_tower_model(vllm_config, {"image", "video"}): + self.vision_tower = AutoModel.from_config(config=config.vision_config) + self.embed_vision = Gemma4MultimodalEmbedder( + config.vision_config, + config.text_config, + quant_config=tower_quant, + prefix=maybe_prefix(prefix, "embed_vision"), + ) + recursive_replace_linear( + self.vision_tower, + tower_quant, + prefix=maybe_prefix(prefix, "vision_tower"), + ) + else: + self.vision_tower = None + self.embed_vision = None + + # ---- Audio tower ---- + self.embed_audio: Gemma4MultimodalEmbedder | None + if getattr(config, "audio_config", None) is not None: + with self._mark_tower_model(vllm_config, "audio"): + self.audio_tower = AutoModel.from_config(config=config.audio_config) + self.audio_tower.post_init() + self.embed_audio = Gemma4MultimodalEmbedder( + config.audio_config, + config.text_config, + quant_config=tower_quant, + prefix=maybe_prefix(prefix, "embed_audio"), + ) + recursive_replace_linear( + self.audio_tower, + tower_quant, + prefix=maybe_prefix(prefix, "audio_tower"), + ) + else: + self.audio_tower = None + self.embed_audio = None + + # ---- Language model ---- + with self._mark_language_model(vllm_config): + self.language_model = EmbeddingGemma2TextModel( + vllm_config=vllm_config, + prefix=maybe_prefix(prefix, "language_model"), + ) + + assert vllm_config.model_config.pooler_config is not None + self.pooler = DispatchPooler.for_embedding( + vllm_config.model_config.pooler_config + ) + + def get_language_model(self) -> nn.Module: # type: ignore[override] + return self.language_model + + def forward( + self, + input_ids: torch.Tensor, + positions: torch.Tensor, + intermediate_tensors: IntermediateTensors | None = None, + inputs_embeds: torch.Tensor | None = None, + **kwargs: object, + ) -> torch.Tensor: + return self.language_model(input_ids, positions, inputs_embeds=inputs_embeds) + + def load_weights(self, weights: Iterable[tuple[str, torch.Tensor]]) -> set[str]: + ignore_prefixes = [ + "embed_vision.embedding.", + "embed_audio.embedding.", + ] + if self.audio_tower is None: + ignore_prefixes.extend(["audio_tower.", "embed_audio."]) + if self.vision_tower is None: + ignore_prefixes.extend(["vision_tower.", "embed_vision."]) + return AutoWeightsLoader( + self, + ignore_unexpected_prefixes=ignore_prefixes, + ).load_weights(weights, mapper=self.hf_to_vllm_mapper) + + @classmethod + def get_placeholder_str(cls, modality: str, i: int = 0) -> str | None: + if modality == "image": + return "<|image|>" + if modality == "audio": + return "<|audio|>" + if modality == "video": + return "<|video|>" + raise ValueError(f"Unsupported modality: {modality}") diff --git a/vllm/model_executor/models/falcon_h1.py b/vllm/model_executor/models/falcon_h1.py index f18f938774..203a3f3ec3 100644 --- a/vllm/model_executor/models/falcon_h1.py +++ b/vllm/model_executor/models/falcon_h1.py @@ -569,6 +569,7 @@ class FalconH1ForCausalLM( head_dim=hf_config.mamba_d_head, state_size=hf_config.mamba_d_state, conv_kernel=hf_config.mamba_d_conv, + num_spec=vllm_config.num_speculative_tokens, ) @classmethod diff --git a/vllm/model_executor/models/gemma3.py b/vllm/model_executor/models/gemma3.py index e886c2ccdd..4646ebeca4 100644 --- a/vllm/model_executor/models/gemma3.py +++ b/vllm/model_executor/models/gemma3.py @@ -154,20 +154,7 @@ class Gemma3Attention(nn.Module): self.is_sliding = layer_type == "sliding_attention" sliding_window = config.sliding_window if self.is_sliding else None - # Initialize the rotary embedding. - if layer_type in config.rope_parameters: - # Transformers v5 rope config. - rope_parameters = config.rope_parameters[layer_type] - else: - # Transformers v4 rope config. - # Global attention. Use the values in config.json. - rope_parameters = config.rope_parameters - # Local attention. Override the values in config.json. - if self.is_sliding: - rope_parameters = dict( - rope_type="default", rope_theta=config.rope_local_base_freq - ) - + rope_parameters = config.rope_parameters[layer_type] self.rotary_emb = get_rope( self.head_dim, max_position=max_position_embeddings, diff --git a/vllm/model_executor/models/gemma3n.py b/vllm/model_executor/models/gemma3n.py index 3c9df39e67..9e7c26bf59 100644 --- a/vllm/model_executor/models/gemma3n.py +++ b/vllm/model_executor/models/gemma3n.py @@ -350,17 +350,7 @@ class Gemma3nAttention(nn.Module): is_sliding = layer_type == "sliding_attention" self.sliding_window = config.sliding_window if is_sliding else None - # Initialize the rotary embedding. - if layer_type in config.rope_parameters: - # Transformers v5 rope config. - rope_parameters = config.rope_parameters[layer_type] - else: - # Transformers v4 rope config. - # Global attention. Use the values in config.json. - rope_parameters = config.rope_parameters.copy() - # Local attention. Override the values in config.json. - if is_sliding: - rope_parameters["rope_theta"] = config.rope_local_base_freq + rope_parameters = config.rope_parameters[layer_type] kv_sharing_target_layer_name = None if self.is_kv_shared_layer: diff --git a/vllm/model_executor/models/gemma4.py b/vllm/model_executor/models/gemma4.py index 17b8251e9a..d1498f16dd 100644 --- a/vllm/model_executor/models/gemma4.py +++ b/vllm/model_executor/models/gemma4.py @@ -59,7 +59,6 @@ from vllm.model_executor.layers.vocab_parallel_embedding import ( ) from vllm.platforms import current_platform from vllm.sequence import IntermediateTensors -from vllm.transformers_utils.configs.gemma4 import gemma4_layer_config from vllm.triton_utils import tl, triton from vllm.v1.attention.backends.utils import KVSharingFastPrefillMetadata @@ -576,7 +575,7 @@ class Gemma4DecoderLayer(nn.Module): self.layer_idx = layer_idx # Gemma4 uses different head dimensions for sliding vs full attention - layer_config = gemma4_layer_config(config, layer_idx) + layer_config = config.per_layer_config[layer_idx] head_dim = layer_config.head_dim num_kv_heads = layer_config.num_key_value_heads @@ -628,9 +627,7 @@ class Gemma4DecoderLayer(nn.Module): ) # MoE (Mixture of Experts) — router + expert block parallel to MLP - self.enable_moe_block = getattr(config, "enable_moe_block", False) or getattr( - config, "use_second_mlp_block", False - ) + self.enable_moe_block = getattr(config, "num_experts", None) is not None self.router: Gemma4Router | None self.experts: MoERunner | None self.post_feedforward_layernorm_1: RMSNorm | None diff --git a/vllm/model_executor/models/gemma4_dspark.py b/vllm/model_executor/models/gemma4_dspark.py index 33b1a1aada..13c868e59b 100644 --- a/vllm/model_executor/models/gemma4_dspark.py +++ b/vllm/model_executor/models/gemma4_dspark.py @@ -20,7 +20,6 @@ from vllm.model_executor.layers.vocab_parallel_embedding import ( VocabParallelEmbedding, ) from vllm.model_executor.model_loader.weight_utils import default_weight_loader -from vllm.transformers_utils.configs.gemma4 import gemma4_layer_config from .gemma4_mtp import Gemma4MTPAttention, Gemma4MTPDecoderLayer from .qwen3_dflash import DFlashQwen3Model, _dflash_layer_causal @@ -48,7 +47,7 @@ class Gemma4DSparkAttention(Gemma4MTPAttention): config, "attention_k_eq_v", False ) - layer_config = gemma4_layer_config(config, layer_idx) + layer_config = config.per_layer_config[layer_idx] head_dim = layer_config.head_dim num_kv_heads = layer_config.num_key_value_heads super().__init__( diff --git a/vllm/model_executor/models/gemma4_mm.py b/vllm/model_executor/models/gemma4_mm.py index 796dfbb6e8..204db6ce8d 100644 --- a/vllm/model_executor/models/gemma4_mm.py +++ b/vllm/model_executor/models/gemma4_mm.py @@ -222,12 +222,11 @@ class Gemma4ProcessingInfo(BaseProcessingInfo): **kwargs, ) + def has_audio(self) -> bool: + return self.get_hf_config().audio_config is not None + def validate_num_items(self, modality: str, num_items: int) -> None: - if ( - modality == "audio" - and num_items > 0 - and self.get_hf_config().audio_config is None - ): + if modality == "audio" and num_items > 0 and not self.has_audio(): model_config = self.ctx.model_config model = get_served_model_name( model_config.model, model_config.served_model_name @@ -243,7 +242,7 @@ class Gemma4ProcessingInfo(BaseProcessingInfo): def get_supported_mm_limits(self) -> Mapping[str, int | None]: limits: dict[str, int | None] = {"image": None} - if self.get_hf_config().audio_config is not None: + if self.has_audio(): limits["audio"] = None limits["video"] = None return limits @@ -261,7 +260,7 @@ class Gemma4ProcessingInfo(BaseProcessingInfo): if isinstance(val, int) and val in _SUPPORTED_SOFT_TOKENS: tokens_per_image = val tokens: dict[str, int] = {"image": tokens_per_image} - if config.audio_config is not None: + if self.has_audio(): # Audio max tokens from the processor's audio_seq_length. processor = self.get_hf_processor() tokens["audio"] = processor.audio_seq_length @@ -275,9 +274,8 @@ class Gemma4ProcessingInfo(BaseProcessingInfo): return tokens def get_data_parser(self) -> MultiModalDataParser: - config = self.get_hf_config() kwargs: dict[str, Any] = {"video_needs_metadata": True} - if getattr(config, "audio_config", None) is not None: + if self.has_audio(): processor = self.get_hf_processor() kwargs["target_sr"] = processor.feature_extractor.sampling_rate return MultiModalDataParser(**kwargs) diff --git a/vllm/model_executor/models/gemma4_mtp.py b/vllm/model_executor/models/gemma4_mtp.py index 01f226ff07..f9931ddafc 100644 --- a/vllm/model_executor/models/gemma4_mtp.py +++ b/vllm/model_executor/models/gemma4_mtp.py @@ -45,7 +45,6 @@ from vllm.model_executor.layers.vocab_parallel_embedding import ( VocabParallelEmbedding, ) from vllm.sequence import IntermediateTensors -from vllm.transformers_utils.configs.gemma4 import gemma4_layer_config from .gemma4 import Gemma4MLP, _get_text_config from .utils import ( @@ -265,7 +264,7 @@ class Gemma4MTPDecoderLayer(nn.Module): self.hidden_size = config.hidden_size layer_idx = extract_layer_index(prefix) - layer_config = gemma4_layer_config(config, layer_idx) + layer_config = config.per_layer_config[layer_idx] head_dim = layer_config.head_dim num_kv_heads = layer_config.num_key_value_heads diff --git a/vllm/model_executor/models/glm4_moe.py b/vllm/model_executor/models/glm4_moe.py index 5e5974e209..4e541a4941 100644 --- a/vllm/model_executor/models/glm4_moe.py +++ b/vllm/model_executor/models/glm4_moe.py @@ -204,6 +204,7 @@ class Glm4MoE(nn.Module): config.n_shared_experts if self.is_fused_shared_expert_enabled else None ), fuse_shared_experts=self.is_fused_shared_expert_enabled, + shared_expert_prefix=f"{prefix}.shared_experts", ) def forward(self, hidden_states: torch.Tensor) -> torch.Tensor: diff --git a/vllm/model_executor/models/granite4_vision.py b/vllm/model_executor/models/granite4_vision.py index b62a865434..d738c665c2 100644 --- a/vllm/model_executor/models/granite4_vision.py +++ b/vllm/model_executor/models/granite4_vision.py @@ -493,13 +493,9 @@ class Granite4VisionForConditionalGeneration( prefix=maybe_prefix(prefix, "vision_tower"), ) - # image_newline parameter - if config.use_image_newline_parameter: - self.image_newline = nn.Parameter( - torch.empty(config.text_config.hidden_size) - ) - else: - self.image_newline = None + self.image_newline = nn.Parameter( + torch.empty(config.text_config.hidden_size) + ) cache_config = vllm_config.cache_config @@ -517,20 +513,18 @@ class Granite4VisionForConditionalGeneration( ) # Spatial projectors: 4 offset groups - self.spatial_projectors = None - if config.use_spatial_sampling: - self.spatial_projectors = nn.ModuleList( - [ - WindowQFormerDownsampler( - config, - quant_config=quant_config, - cache_config=cache_config, - spatial_offset=i, - prefix=maybe_prefix(prefix, f"spatial_projectors.{i}"), - ) - for i in range(4) - ] - ) + self.spatial_projectors = nn.ModuleList( + [ + WindowQFormerDownsampler( + config, + quant_config=quant_config, + cache_config=cache_config, + spatial_offset=i, + prefix=maybe_prefix(prefix, f"spatial_projectors.{i}"), + ) + for i in range(4) + ] + ) # ----- Language model (marked as LM) ----- with self._mark_language_model(vllm_config): @@ -545,12 +539,9 @@ class Granite4VisionForConditionalGeneration( # Store config values we need self._deepstack_layer_map = config.deepstack_layer_map # [[-19, 9], ...] - self._use_spatial_sampling = getattr(config, "use_spatial_sampling", False) - self._spatial_vision_layer = getattr(config, "spatial_vision_layer", -1) - self._spatial_target_layers = getattr(config, "spatial_target_layers", []) - self._vision_feature_select_strategy = getattr( - config, "vision_feature_select_strategy", "full" - ) + self._spatial_vision_layer = config.spatial_vision_layer + self._spatial_target_layers = config.spatial_target_layers + self._vision_feature_select_strategy = config.vision_feature_select_strategy self._downsample_rate = Fraction(config.downsample_rate) # Ordered list of LLM layer indices for each deepstack level. @@ -558,7 +549,7 @@ class Granite4VisionForConditionalGeneration( # (before any embed_multimodal call). self._ds_layer_indices: list[int] = [ llm_layer for _, llm_layer in config.deepstack_layer_map - ] + list(getattr(config, "spatial_target_layers", [])) + ] + list(config.spatial_target_layers) # Share ds_layer_indices with the LLM causal model so # make_empty_intermediate_tensors includes the correct keys @@ -571,7 +562,7 @@ class Granite4VisionForConditionalGeneration( # the updated values written just before each prefill. # Shape: (max_num_batched_tokens, lm_hidden_size) per level. n_layerwise = len(config.deepstack_layer_map) - n_spatial = len(getattr(config, "spatial_target_layers", [])) + n_spatial = len(config.spatial_target_layers) num_ds_levels = n_layerwise + n_spatial lm_hidden = config.text_config.hidden_size max_tokens = vllm_config.scheduler_config.max_num_batched_tokens @@ -644,26 +635,24 @@ class Granite4VisionForConditionalGeneration( ) image_feature = unpad_image(image_feature, image_sizes[image_idx]) - if self.image_newline is not None: - image_feature = torch.cat( - ( - image_feature, - self.image_newline[:, None, None] - .expand(*image_feature.shape[:-1], 1) - .to(image_feature.device, image_feature.dtype), - ), - dim=-1, - ) + image_feature = torch.cat( + ( + image_feature, + self.image_newline[:, None, None] + .expand(*image_feature.shape[:-1], 1) + .to(image_feature.device, image_feature.dtype), + ), + dim=-1, + ) image_feature = image_feature.flatten(1, 2).transpose(0, 1) image_feature = torch.cat((base_image_feature, image_feature), dim=0) else: image_feature = image_feature[0] - if self.image_newline is not None: - image_feature = torch.cat( - (image_feature, self.image_newline[None].to(image_feature)), - dim=0, - ) + image_feature = torch.cat( + (image_feature, self.image_newline[None].to(image_feature)), + dim=0, + ) new_image_features.append(image_feature) @@ -718,16 +707,15 @@ class Granite4VisionForConditionalGeneration( ) levels.append((llm_layer, per_image)) - if self._use_spatial_sampling and self.spatial_projectors is not None: - spatial_hidden = all_hidden_states[self._spatial_vision_layer] - if select_strategy == "default": - spatial_hidden = spatial_hidden[:, 1:] - for group_idx, llm_layer in enumerate(self._spatial_target_layers): - projected = self.spatial_projectors[group_idx](spatial_hidden) - per_image = self._pack_and_unpad_image_features( - torch.split(projected, image_num_patches, dim=0), image_sizes - ) - levels.append((llm_layer, per_image)) + spatial_hidden = all_hidden_states[self._spatial_vision_layer] + if select_strategy == "default": + spatial_hidden = spatial_hidden[:, 1:] + for group_idx, llm_layer in enumerate(self._spatial_target_layers): + projected = self.spatial_projectors[group_idx](spatial_hidden) + per_image = self._pack_and_unpad_image_features( + torch.split(projected, image_num_patches, dim=0), image_sizes + ) + levels.append((llm_layer, per_image)) llm_layer_indices = [llm_layer for llm_layer, _ in levels] num_images = len(image_sizes) diff --git a/vllm/model_executor/models/granite_speech.py b/vllm/model_executor/models/granite_speech.py index 3ae85cfaac..2380bba10a 100644 --- a/vllm/model_executor/models/granite_speech.py +++ b/vllm/model_executor/models/granite_speech.py @@ -158,9 +158,7 @@ class GraniteSpeechMultiModalProcessor( feature_extractor = processor.audio_processor vocab = tokenizer.get_vocab() - # Use getattr with default to be compatible with transformers<4.48 - audio_token = getattr(processor, "audio_token", "<|audio|>") - audio_token_id = vocab[audio_token] + audio_token_id = vocab[processor.audio_token] def get_replacement(item_idx: int): audios = mm_items.get_items("audio", AudioProcessorItems) @@ -221,8 +219,7 @@ class GraniteSpeechDummyInputsBuilder( def get_dummy_text(self, mm_counts: Mapping[str, int]) -> str: num_audios = mm_counts.get("audio", 0) hf_processor = self.info.get_hf_processor() - audio_token = getattr(hf_processor, "audio_token", "<|audio|>") - return audio_token * num_audios + return hf_processor.audio_token * num_audios ### QFormer Projector diff --git a/vllm/model_executor/models/granitemoehybrid.py b/vllm/model_executor/models/granitemoehybrid.py index da7cd0478b..979d272c13 100644 --- a/vllm/model_executor/models/granitemoehybrid.py +++ b/vllm/model_executor/models/granitemoehybrid.py @@ -320,10 +320,6 @@ class GraniteMoeHybridAttention(nn.Module): ALL_DECODER_LAYER_TYPES = { - # Transformers < 5.13.0 - "attention": GraniteMoeHybridAttentionDecoderLayer, - "mamba": GraniteMoeHybridMambaDecoderLayer, - # Transformers >= 5.13.0 "full_attention": GraniteMoeHybridAttentionDecoderLayer, "linear_attention": GraniteMoeHybridMambaDecoderLayer, } @@ -647,6 +643,7 @@ class GraniteMoeHybridForCausalLM( head_dim=hf_config.mamba_d_head, state_size=hf_config.mamba_d_state, conv_kernel=hf_config.mamba_d_conv, + num_spec=vllm_config.num_speculative_tokens, ) @classmethod diff --git a/vllm/model_executor/models/hy_v3.py b/vllm/model_executor/models/hy_v3.py index 2e24cf65e8..80fc16e232 100644 --- a/vllm/model_executor/models/hy_v3.py +++ b/vllm/model_executor/models/hy_v3.py @@ -31,7 +31,7 @@ from typing import Any import torch from torch import nn -from transformers import PreTrainedConfig +from transformers import HYV3Config, PreTrainedConfig from vllm.compilation.decorators import support_torch_compile from vllm.config import CacheConfig, VllmConfig, get_current_vllm_config @@ -67,7 +67,6 @@ from vllm.model_executor.model_loader.weight_utils import ( maybe_remap_kv_scale_name, ) from vllm.sequence import IntermediateTensors -from vllm.transformers_utils.configs.hy_v3 import HYV3Config from .interfaces import MixtureOfExperts, SupportsLoRA, SupportsPP from .utils import ( diff --git a/vllm/model_executor/models/hyperclovax.py b/vllm/model_executor/models/hyperclovax.py index 66c3b4a270..e86260bca9 100644 --- a/vllm/model_executor/models/hyperclovax.py +++ b/vllm/model_executor/models/hyperclovax.py @@ -31,6 +31,7 @@ from itertools import islice import torch from torch import nn +from transformers import HyperCLOVAXConfig from vllm.compilation.decorators import support_torch_compile from vllm.config import CacheConfig, VllmConfig @@ -51,7 +52,6 @@ from vllm.model_executor.layers.vocab_parallel_embedding import ( VocabParallelEmbedding, ) from vllm.sequence import IntermediateTensors -from vllm.transformers_utils.configs.hyperclovax import HyperCLOVAXConfig from .interfaces import SupportsLoRA, SupportsPP from .utils import ( @@ -137,9 +137,7 @@ class HyperCLOVAXAttention(nn.Module): # the KV heads across multiple tensor parallel GPUs. assert tp_size % self.total_num_kv_heads == 0 self.num_kv_heads = max(1, self.total_num_kv_heads // tp_size) - self.head_dim = getattr( - config, "head_dim", self.hidden_size // self.total_num_heads - ) + self.head_dim = config.head_dim self.q_size = self.num_heads * self.head_dim self.kv_size = self.num_kv_heads * self.head_dim self.scaling = config.attention_multiplier @@ -166,7 +164,7 @@ class HyperCLOVAXAttention(nn.Module): self.head_dim, max_position=max_position_embeddings, is_neox_style=True, - rope_parameters=getattr(config, "rope_parameters", None), + rope_parameters=config.rope_parameters, dual_chunk_attention_config=dual_chunk_attention_config, ) @@ -217,18 +215,15 @@ class HyperCLOVAXDecoderLayer(nn.Module): "dual_chunk_attention_config", None, ) - attention_bias = getattr(config, "attention_bias", False) self.self_attn = HyperCLOVAXAttention( config=config, hidden_size=self.hidden_size, num_heads=config.num_attention_heads, - num_kv_heads=getattr( - config, "num_key_value_heads", config.num_attention_heads - ), + num_kv_heads=config.num_key_value_heads, max_position_embeddings=max_position_embeddings, quant_config=quant_config, - bias=attention_bias, + bias=config.attention_bias, cache_config=cache_config, prefix=f"{prefix}.self_attn", dual_chunk_attention_config=dual_chunk_attention_config, @@ -238,7 +233,7 @@ class HyperCLOVAXDecoderLayer(nn.Module): intermediate_size=config.intermediate_size, hidden_act=config.hidden_act, quant_config=quant_config, - bias=getattr(config, "mlp_bias", False), + bias=config.mlp_bias, prefix=f"{prefix}.mlp", ) self.input_layernorm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps) diff --git a/vllm/model_executor/models/interfaces.py b/vllm/model_executor/models/interfaces.py index de7ecb1e44..1c71ab7411 100644 --- a/vllm/model_executor/models/interfaces.py +++ b/vllm/model_executor/models/interfaces.py @@ -21,6 +21,7 @@ from typing import ( ClassVar, Literal, Protocol, + Self, TypeAlias, overload, runtime_checkable, @@ -31,7 +32,7 @@ import torch import torch.nn as nn from torch import Tensor from transformers.models.whisper.tokenization_whisper import LANGUAGES -from typing_extensions import Self, TypeIs, deprecated +from typing_extensions import TypeIs from vllm.logger import init_logger from vllm.model_executor.layers.quantization import QuantizationConfig @@ -423,17 +424,6 @@ class SupportsMultiModal(SupportsMultiModalEmbeddings, Protocol): yield - @deprecated( - "get_num_mm_encoder_tokens is deprecated; use get_mm_lora_token_counts instead." - ) - def get_num_mm_encoder_tokens(self, num_image_tokens: int) -> int: ... - - @deprecated( - "get_num_mm_connector_tokens is deprecated; use " - "get_mm_lora_token_counts instead." - ) - def get_num_mm_connector_tokens(self, num_vision_tokens: int) -> int: ... - def get_mm_lora_token_counts( self, *, @@ -447,12 +437,9 @@ class SupportsMultiModal(SupportsMultiModalEmbeddings, Protocol): connector forwards. Models with multiple modalities can override this when each modality has different encoder padding or pooling behavior. """ - del modality, mm_kwargs - num_encoder_tokens = self.get_num_mm_encoder_tokens(num_mm_embeds) - num_connector_tokens = self.get_num_mm_connector_tokens(num_encoder_tokens) - return ( - num_encoder_tokens, - num_connector_tokens if isinstance(num_connector_tokens, int) else None, + raise NotImplementedError( + f"{type(self).__name__} must implement get_mm_lora_token_counts " + "to support tower/connector LoRA." ) def _embed_text_input_ids( diff --git a/vllm/model_executor/models/kimi_k25.py b/vllm/model_executor/models/kimi_k25.py index 2426ec61ef..e555b6368d 100644 --- a/vllm/model_executor/models/kimi_k25.py +++ b/vllm/model_executor/models/kimi_k25.py @@ -11,12 +11,13 @@ from typing import TYPE_CHECKING, Annotated, Any, ClassVar, Literal import torch from torch import nn -from transformers import BatchFeature +from transformers import BatchFeature, Kimi_K25Config from vllm.config import VllmConfig from vllm.config.multimodal import MultiModalDummyOptions from vllm.inputs import MultiModalDataDict from vllm.logger import init_logger +from vllm.model_executor.layers.fusion.mm_input_norm import build_mm_input_norm from vllm.model_executor.layers.quantization import QuantizationConfig from vllm.model_executor.layers.quantization.compressed_tensors import ( compressed_tensors, @@ -62,7 +63,6 @@ from vllm.multimodal.processing import ( ) from vllm.platforms import current_platform from vllm.sequence import IntermediateTensors -from vllm.transformers_utils.configs.kimi_k25 import KimiK25Config from vllm.transformers_utils.processor import cached_get_image_processor from vllm.transformers_utils.processors.kimi_k25 import KimiK25Processor from vllm.transformers_utils.processors.kimi_k25_vision_fused import ( @@ -134,7 +134,7 @@ class KimiK25ProcessingInfo(BaseProcessingInfo): # Resolve token ID from the tokenizer because transformers v5 # may remap token IDs vs config.json. - config_token_id = hf_config.media_placeholder_token_id + config_token_id = hf_config.image_token_id resolved_token_id = tokenizer.convert_tokens_to_ids("<|media_pad|>") unk_token_id = getattr(tokenizer, "unk_token_id", None) is_valid_resolved = isinstance(resolved_token_id, int) and ( @@ -142,7 +142,7 @@ class KimiK25ProcessingInfo(BaseProcessingInfo): ) if is_valid_resolved and resolved_token_id != config_token_id: logger.warning_once( - "Kimi-K2.5 config.media_placeholder_token_id (%d) disagrees " + "Kimi-K2.5 config.image_token_id (%d) disagrees " "with tokenizer mapping for <|media_pad|> (%d). " "Using tokenizer value.", config_token_id, @@ -150,7 +150,7 @@ class KimiK25ProcessingInfo(BaseProcessingInfo): ) media_token_id = resolved_token_id # Patch config so downstream code also sees the correct ID. - hf_config.media_placeholder_token_id = resolved_token_id + hf_config.image_token_id = resolved_token_id else: media_token_id = config_token_id @@ -169,7 +169,7 @@ class KimiK25ProcessingInfo(BaseProcessingInfo): return self.hf_processor def get_hf_config(self): - return self.ctx.get_hf_config(KimiK25Config) + return self.ctx.get_hf_config(Kimi_K25Config) def get_supported_mm_limits(self) -> Mapping[str, int | None]: # None means unlimited @@ -306,6 +306,7 @@ class KimiK25ForConditionalGeneration( supports_encoder_tp_data = True supports_encoder_cudagraph: ClassVar[Literal[True]] = True + supports_mm_device_do_normalize = True hf_to_vllm_mapper = WeightsMapper( orig_to_new_prefix={ @@ -336,7 +337,7 @@ class KimiK25ForConditionalGeneration( ) -> None: super().__init__() model_config = vllm_config.model_config - config: KimiK25Config = model_config.hf_config + config: Kimi_K25Config = model_config.hf_config self.config = config quant_config = vllm_config.quant_config @@ -345,11 +346,11 @@ class KimiK25ForConditionalGeneration( ) self.hidden_size = config.text_config.hidden_size self.device = current_platform.current_device() - # Build vision tower directly with KimiK25VisionConfig with self._mark_tower_model(vllm_config, "vision_chunk"): self.vision_tower = MoonViT3dPretrainedModel( config.vision_config, quant_config=self._maybe_ignore_quant_config(quant_config), + input_norm=build_mm_input_norm(vllm_config.model_config), prefix=maybe_prefix(prefix, "vision_tower"), ) if self._maybe_ignore_quant_config(quant_config) is not None: @@ -361,6 +362,7 @@ class KimiK25ForConditionalGeneration( self.mm_projector = KimiK25MultiModalProjector( config=config.vision_config, + out_hidden_size=config.text_config.hidden_size, use_data_parallel=self.use_data_parallel, quant_config=self._maybe_ignore_quant_config(quant_config), prefix=maybe_prefix(prefix, "mm_projector"), @@ -380,7 +382,7 @@ class KimiK25ForConditionalGeneration( self.make_empty_intermediate_tensors = ( self.language_model.make_empty_intermediate_tensors ) - self.media_placeholder: int = self.config.media_placeholder_token_id + self.media_placeholder: int = self.config.image_token_id def _maybe_ignore_quant_config(self, quant_config: QuantizationConfig | None): if isinstance(quant_config, compressed_tensors.CompressedTensorsConfig): @@ -404,9 +406,6 @@ class KimiK25ForConditionalGeneration( pixel_values.shape[0] * pixel_values.shape[1], *pixel_values.shape[2:] ) - # The batch dimension of pixel_values has been flattened into shape[0] - target_dtype = next(self.vision_tower.parameters()).dtype - pixel_values = pixel_values.to(target_dtype) assert isinstance(grid_thws, torch.Tensor), ( f"expect grid_thws to be a tensor, got {type(grid_thws)}" ) @@ -479,30 +478,16 @@ class KimiK25ForConditionalGeneration( # Image-only (t == 1). Video chunks (t > 1) fall back to eager. # # ------------------------------------------------------------------ # - @property - def _encoder_cudagraph_pad_totals(self) -> dict[int, int]: - """Row count of each captured buffer set, keyed by cu_seqlens ptr.""" - totals = self.__dict__.get("_encoder_cg_pad_totals") - if totals is None: - totals = {} - self.__dict__["_encoder_cg_pad_totals"] = totals - return totals - def get_encoder_cudagraph_config(self) -> "EncoderCudaGraphConfig": from vllm.v1.worker.encoder_cudagraph_defs import EncoderCudaGraphConfig - pad_totals = self._encoder_cudagraph_pad_totals - def pad_cu_seqlens(dst: torch.Tensor, src: torch.Tensor) -> None: - # Varlen attention requires cu_seqlens[-1] to equal the number of - # rows actually passed in. The captured buffers are sized for the - # full token budget, so a smaller real batch has to be completed - # with one trailing padding sequence; declaring fewer rows than the - # buffer holds is undefined behaviour and returns NaN on FlashAttn. - total = pad_totals.get(dst.data_ptr()) + # Zero-length padding (same as Qwen): tail slots repeat the last + # real offset, so FlashAttn schedules no work for padding rows. n = min(src.shape[0], dst.shape[0]) dst[:n].copy_(src[:n]) - dst[n:] = total if total is not None else src[-1] + if n < dst.shape[0]: + dst[n:] = src[-1] if n else 0 return EncoderCudaGraphConfig( modalities=["vision_chunk"], @@ -623,13 +608,18 @@ class KimiK25ForConditionalGeneration( ps = (ps, ps) total_patches = max_batch_size * ho * kh * wo * kw dummy_pixel_values = torch.zeros( - total_patches, 3, ps[0], ps[1], device=device, dtype=dtype + total_patches, + 3, + ps[0], + ps[1], + device=device, + dtype=self.vision_tower.patch_embed.input_norm.input_dtype or dtype, ) # max_seqlen must cover the worst case: one item consuming the full # budget, i.e. token_budget * kh * kw patches. - # max_batch_size + 1 leaves a spare cu_seqlens slot so replay can append - # a padding sequence covering rows the real batch does not fill. + # max_batch_size + 1 leaves a spare cu_seqlens slot so replay can + # pad the tail with zero-length sequences. metadata = self.vision_tower.prepare_encoder_cudagraph_metadata( grid_thw_list, max_batch_size=max_batch_size + 1, @@ -640,10 +630,6 @@ class KimiK25ForConditionalGeneration( values: dict[str, torch.Tensor] = {"pixel_values": dummy_pixel_values} values.update({k: v for k, v in metadata.items() if v is not None}) - cu_seqlens = values.get("cu_seqlens") - if cu_seqlens is not None: - self._encoder_cudagraph_pad_totals[cu_seqlens.data_ptr()] = total_patches - return EncoderCudaGraphCaptureInputs(values=values) def prepare_encoder_cudagraph_replay_buffers( @@ -658,8 +644,8 @@ class KimiK25ForConditionalGeneration( grid_thw_list = self._get_grid_thw_list(mm_kwargs) pixel_values = mm_kwargs["pixel_values"] - # Unpadded: pad_cu_seqlens completes the tail with the padding sequence - # so cu_seqlens[-1] matches the captured buffer's row count. + # Unpadded: pad_cu_seqlens pads the tail with zero-length sequences + # whose entries repeat the real total. metadata = self.vision_tower.prepare_encoder_cudagraph_metadata( grid_thw_list, max_batch_size=None, diff --git a/vllm/model_executor/models/kimi_k25_vit.py b/vllm/model_executor/models/kimi_k25_vit.py index ee8adb6a74..bb1be1ba32 100644 --- a/vllm/model_executor/models/kimi_k25_vit.py +++ b/vllm/model_executor/models/kimi_k25_vit.py @@ -15,6 +15,7 @@ import numpy as np import torch import torch.nn as nn import torch.nn.functional as F +from transformers import Kimi_K25VisionConfig from transformers.activations import GELUActivation from vllm.distributed import divide, get_tensor_model_parallel_world_size @@ -22,6 +23,7 @@ from vllm.logger import init_logger from vllm.model_executor.layers.activation import get_act_fn from vllm.model_executor.layers.attention.mm_encoder_attention import MMEncoderAttention from vllm.model_executor.layers.conv import Conv2dLayer +from vllm.model_executor.layers.fusion.mm_input_norm import IdentityInputNorm from vllm.model_executor.layers.linear import ( ColumnParallelLinear, QKVParallelLinear, @@ -39,13 +41,21 @@ from vllm.model_executor.models.vision import ( run_dp_sharded_mrope_vision_model, ) from vllm.platforms import current_platform -from vllm.transformers_utils.configs.kimi_k25 import KimiK25VisionConfig +from vllm.transformers_utils.configs.kimi_k3 import KimiK3VisionConfig from vllm.triton_utils import HAS_TRITON from vllm.utils.torch_utils import async_tensor_h2d logger = init_logger(__name__) +def _get_pos_emb_size( + config: Kimi_K25VisionConfig | KimiK3VisionConfig, dim: str +) -> int: + # Legacy checkpoints and Kimi-K3 use `init_pos_emb_*` + legacy = getattr(config, f"init_pos_emb_{dim}", None) + return legacy if legacy is not None else getattr(config, f"pos_emb_{dim}") + + def _apply_rope_input_validation(x, freqs_cis): assert x.ndim == freqs_cis.ndim + 1, (x.shape, freqs_cis.shape) assert x.shape[:-2] == freqs_cis.shape[:-1], (x.shape, freqs_cis.shape) @@ -187,6 +197,7 @@ class MoonVision3dPatchEmbed(nn.Module): pos_emb_type: str = "divided_fixed", patch_embed_proj_bias: bool = True, pos_emb_interpolation_mode: str = "bicubic", + input_norm: nn.Module | None = None, ): super().__init__() assert isinstance(patch_size, int | Sequence), ( @@ -206,6 +217,7 @@ class MoonVision3dPatchEmbed(nn.Module): stride=patch_size, bias=patch_embed_proj_bias, ) + self.input_norm = input_norm if input_norm is not None else IdentityInputNorm() if pos_emb_type == "divided_fixed": self.pos_emb = Learnable2DInterpPosEmbDivided_fixed( @@ -225,6 +237,7 @@ class MoonVision3dPatchEmbed(nn.Module): *, pos_embeds: torch.Tensor | None = None, ) -> torch.Tensor: + x = self.input_norm(x.flatten(1), self.proj.weight.dtype).view_as(x) # forward_native dispatches this non-overlapping patch projection to GEMM. x = self.proj.forward_native(x).view(x.size(0), self.proj.out_channels) if pos_embeds is not None: @@ -694,35 +707,35 @@ def tpool_patch_merger_packed( class MoonViT3dPretrainedModel(nn.Module): - """Main vision tower model. - - Uses KimiK25VisionConfig directly from transformers_utils/configs/kimi_k25.py. - """ + """Main vision tower model.""" def __init__( self, - config: KimiK25VisionConfig, + config: Kimi_K25VisionConfig | KimiK3VisionConfig, quant_config: QuantizationConfig | None = None, + input_norm: nn.Module | None = None, prefix: str = "", ): super().__init__() config = deepcopy(config) self.config = config # Required for run_dp_sharded_mrope_vision_model - self.merge_kernel_size = config.merge_kernel_size + merge_h, merge_w = config.merge_kernel_size + self.merge_kernel_size = (merge_h, merge_w) self.patch_size = config.patch_size - self.merge_type = config.merge_type + self.merge_type = getattr(config, "merge_type", "sd2_tpool") self.patch_embed = MoonVision3dPatchEmbed( out_dim=config.hidden_size, patch_size=config.patch_size, - pos_emb_height=config.init_pos_emb_height, - pos_emb_width=config.init_pos_emb_width, - pos_emb_time=config.init_pos_emb_time, - pos_emb_type=config.pos_emb_type, + pos_emb_height=_get_pos_emb_size(config, "height"), + pos_emb_width=_get_pos_emb_size(config, "width"), + pos_emb_time=_get_pos_emb_size(config, "time"), + pos_emb_type=getattr(config, "pos_emb_type", "divided_fixed"), patch_embed_proj_bias=getattr(config, "patch_embed_proj_bias", True), pos_emb_interpolation_mode=getattr( config, "pos_emb_interpolation_mode", "bicubic" ), + input_norm=input_norm, ) self.encoder = MoonViT3dEncoder( @@ -734,14 +747,14 @@ class MoonViT3dPretrainedModel(nn.Module): "qkv_hidden_size": getattr(config, "qkv_hidden_size", None), "mlp_dim": config.intermediate_size, "activation": get_act_fn( - getattr(config, "activation_func", "gelu_pytorch_tanh") + getattr(config, "activation_func", None) or config.hidden_act ), "attn_bias": getattr(config, "attn_bias", True), "norm_type": getattr(config, "norm_type", "layernorm"), "mlp_type": getattr(config, "mlp_type", "mlp2"), "linear_bias": getattr(config, "linear_bias", True), }, - video_attn_type=config.video_attn_type, + video_attn_type=getattr(config, "video_attn_type", "spatial_temporal"), quant_config=quant_config, prefix=maybe_prefix(prefix, "encoder"), ) @@ -880,7 +893,8 @@ class KimiK25MultiModalProjector(nn.Module): def __init__( self, - config: KimiK25VisionConfig, + config: Kimi_K25VisionConfig | KimiK3VisionConfig, + out_hidden_size: int, use_data_parallel: bool = False, quant_config: QuantizationConfig | None = None, prefix: str = "", @@ -903,13 +917,13 @@ class KimiK25MultiModalProjector(nn.Module): ) self.linear_2 = ReplicatedLinear( self.hidden_size, - getattr(config, "text_hidden_size", config.mm_hidden_size), + out_hidden_size, bias=False, quant_config=quant_config, prefix=f"{prefix}.linear_2", ) self.post_norm = torch.nn.RMSNorm( - getattr(config, "text_hidden_size", config.mm_hidden_size), + out_hidden_size, eps=config.projector_ln_eps, ) self.act = GELUActivation() @@ -925,7 +939,7 @@ class KimiK25MultiModalProjector(nn.Module): ) self.linear_2 = ReplicatedLinear( self.hidden_size, - config.mm_hidden_size, + out_hidden_size, bias=True, quant_config=quant_config, prefix=f"{prefix}.linear_2", diff --git a/vllm/model_executor/models/lfm2_moe.py b/vllm/model_executor/models/lfm2_moe.py index f6e819c263..66c9520bc9 100644 --- a/vllm/model_executor/models/lfm2_moe.py +++ b/vllm/model_executor/models/lfm2_moe.py @@ -5,6 +5,7 @@ from itertools import islice import torch import torch.nn as nn +from transformers import Lfm2MoeConfig from vllm.compilation.decorators import support_torch_compile from vllm.config import CacheConfig, ModelConfig, VllmConfig, get_current_vllm_config @@ -40,7 +41,6 @@ from vllm.model_executor.layers.vocab_parallel_embedding import ( VocabParallelEmbedding, ) from vllm.sequence import IntermediateTensors -from vllm.transformers_utils.configs.lfm2_moe import Lfm2MoeConfig from .interfaces import ( HasInnerState, diff --git a/vllm/model_executor/models/llama4_eagle.py b/vllm/model_executor/models/llama4_eagle.py index 975ee28dc2..8634f113d0 100644 --- a/vllm/model_executor/models/llama4_eagle.py +++ b/vllm/model_executor/models/llama4_eagle.py @@ -36,10 +36,15 @@ from vllm.model_executor.layers.vocab_parallel_embedding import ( from vllm.model_executor.models.llama4 import Llama4DecoderLayer, Llama4ForCausalLM from vllm.model_executor.models.utils import extract_layer_index -from .interfaces import SupportsMultiModal, SupportsMultiModalEmbeddings +from .interfaces import ( + MultiModalEmbeddings, + SupportsMultiModalEmbeddings, + _require_is_multimodal, +) from .utils import ( AutoWeightsLoader, WeightsMapper, + _merge_multimodal_embeddings, maybe_prefix, process_eagle_weight, ) @@ -204,7 +209,21 @@ class EagleLlama4ForCausalLM(_EagleLlama4ForCausalLMBase, SupportsMultiModalEmbe def get_language_model(self) -> torch.nn.Module: return self.model - embed_input_ids = SupportsMultiModal.embed_input_ids # type: ignore + def embed_input_ids( + self, + input_ids: torch.Tensor, + multimodal_embeddings: MultiModalEmbeddings | None = None, + *, + is_multimodal: torch.Tensor | None = None, + ) -> torch.Tensor: + inputs_embeds = self.model.embed_input_ids(input_ids) + if multimodal_embeddings is None or len(multimodal_embeddings) == 0: + return inputs_embeds + return _merge_multimodal_embeddings( + inputs_embeds=inputs_embeds, + multimodal_embeddings=multimodal_embeddings, + is_multimodal=_require_is_multimodal(is_multimodal), + ) def forward( self, diff --git a/vllm/model_executor/models/longcat_flash.py b/vllm/model_executor/models/longcat_flash.py index 50962e9728..4ddd1ea9bf 100644 --- a/vllm/model_executor/models/longcat_flash.py +++ b/vllm/model_executor/models/longcat_flash.py @@ -58,13 +58,17 @@ from vllm.model_executor.layers.linear import ( ) from vllm.model_executor.layers.logits_processor import LogitsProcessor from vllm.model_executor.layers.quantization import QuantizationConfig -from vllm.model_executor.layers.quantization.utils.int8_utils import block_dequant from vllm.model_executor.layers.vocab_parallel_embedding import ( ParallelLMHead, VocabParallelEmbedding, ) -from vllm.model_executor.model_loader.weight_utils import default_weight_loader +from vllm.model_executor.model_loader.weight_utils import ( + LoaderFunction, + composed_weight_loader, + default_weight_loader, +) from vllm.model_executor.models.deepseek_v2 import DeepseekV2MLAAttention +from vllm.model_executor.utils import set_weight_attrs from vllm.sequence import IntermediateTensors from .interfaces import SupportsLoRA, SupportsPP @@ -80,6 +84,23 @@ from .utils import ( logger = init_logger(__name__) +def _scaled_weight_loader(scale: float) -> LoaderFunction: + return composed_weight_loader(default_weight_loader, lambda x: x * scale) + + +def scale_mla_lora_norms_on_load(self_attn: nn.Module, config) -> None: + """Fold LongCat's MLA LoRA scaling into the q/kv_a_layernorm weight loaders.""" + for norm, rank, enabled in ( + (self_attn.q_a_layernorm, config.q_lora_rank, config.mla_scale_q_lora), + (self_attn.kv_a_layernorm, config.kv_lora_rank, config.mla_scale_kv_lora), + ): + if enabled: + scale = (config.hidden_size / rank) ** 0.5 + set_weight_attrs( + norm.weight, {"weight_loader": _scaled_weight_loader(scale)} + ) + + class FlashMLP(nn.Module): """Flash MLP layer.""" @@ -265,6 +286,8 @@ class FlashDecoderLayer(nn.Module): for i in range(2) ] ) + for self_attn in self.self_attn: + scale_mla_lora_norms_on_load(self_attn, config) self.input_layernorm = nn.ModuleList( [RMSNorm(config.hidden_size, eps=config.rms_norm_eps) for i in range(2)] ) @@ -450,6 +473,9 @@ class FlashModel(nn.Module): for name, loaded_weight in weights: if "rotary_emb.inv_freq" in name: continue + # The MTP layers are loaded by LongCatFlashMTP. + if name.startswith("mtp."): + continue for param_name, weight_name, shard_id in stacked_params_mapping: if weight_name not in name: continue @@ -461,9 +487,6 @@ class FlashModel(nn.Module): name.endswith(".bias") or name.endswith("_bias") ) and name not in params_dict: continue - # Skip mtp - if ".mtp." in name: - continue if is_pp_missing_parameter(name, self): continue param = params_dict[name] @@ -478,9 +501,6 @@ class FlashModel(nn.Module): continue is_expert_weight = True name_mapped = name.replace(weight_name, param_name) - # Skip mtp - if ".mtp." in name_mapped: - continue if ( name_mapped.endswith(".bias") or name_mapped.endswith("_bias") ) and name not in params_dict: @@ -515,9 +535,6 @@ class FlashModel(nn.Module): # Skip loading kv_scale from ckpts towards new design. if name.endswith(".kv_scale") and name not in params_dict: continue - # Skip mtp - if ".mtp." in name: - continue if name is None: continue if is_pp_missing_parameter(name, self): @@ -528,54 +545,6 @@ class FlashModel(nn.Module): ) weight_loader(param, loaded_weight) loaded_params.add(name) - for layer_id in range(self.config.num_layers): - for i in range(2): - if isinstance(self.layers[layer_id], PPMissingLayer): - continue - self_attn = self.layers[layer_id].self_attn[i] - if ( - self.quant_config is not None - and hasattr(self.quant_config, "weight_block_size") - and self_attn.kv_b_proj.weight.dtype - in ( - torch.float8_e4m3fn, - torch.float8_e4m3fnuz, - ) - ): - weight_block_size = self.quant_config.weight_block_size - if weight_block_size is not None: - assert hasattr(self_attn.kv_b_proj, "weight_scale_inv") - dtype = torch.get_default_dtype() - w = block_dequant( - self_attn.kv_b_proj.weight, - self_attn.kv_b_proj.weight_scale_inv, - weight_block_size, - ).to(dtype) - else: - w = self_attn.kv_b_proj.weight - - w_kc, w_vc = w.unflatten( - 0, (-1, self_attn.qk_nope_head_dim + self_attn.v_head_dim) - ).split([self_attn.qk_nope_head_dim, self_attn.v_head_dim], dim=1) - self_attn.w_kc = w_kc.transpose(1, 2).contiguous().transpose(1, 2) - self_attn.w_vc = w_vc.contiguous().transpose(1, 2) - # Guard against compounding on incremental load_weights calls: - # the in-place ``*=`` would otherwise re-apply the MLA LoRA - # scaling to the layernorm weights on each pass. - if self.config.mla_scale_q_lora and not getattr( - self_attn, "_mla_q_lora_scaled", False - ): - self_attn.q_a_layernorm.weight.data *= ( - self.config.hidden_size / self.config.q_lora_rank - ) ** 0.5 - self_attn._mla_q_lora_scaled = True - if self.config.mla_scale_kv_lora and not getattr( - self_attn, "_mla_kv_lora_scaled", False - ): - self_attn.kv_a_layernorm.weight.data *= ( - self.config.hidden_size / self.config.kv_lora_rank - ) ** 0.5 - self_attn._mla_kv_lora_scaled = True return loaded_params diff --git a/vllm/model_executor/models/longcat_flash_mtp.py b/vllm/model_executor/models/longcat_flash_mtp.py index 506f7a7451..ae380f92b6 100644 --- a/vllm/model_executor/models/longcat_flash_mtp.py +++ b/vllm/model_executor/models/longcat_flash_mtp.py @@ -14,7 +14,6 @@ from vllm.model_executor.layers.layernorm import RMSNorm from vllm.model_executor.layers.linear import ReplicatedLinear from vllm.model_executor.layers.logits_processor import LogitsProcessor from vllm.model_executor.layers.quantization import QuantizationConfig -from vllm.model_executor.layers.quantization.utils.int8_utils import block_dequant from vllm.model_executor.layers.vocab_parallel_embedding import ( ParallelLMHead, VocabParallelEmbedding, @@ -23,6 +22,7 @@ from vllm.model_executor.model_loader.weight_utils import default_weight_loader from vllm.sequence import IntermediateTensors from .deepseek_v2 import DeepseekV2DecoderLayer +from .longcat_flash import scale_mla_lora_norms_on_load from .utils import maybe_prefix @@ -45,6 +45,7 @@ class LongCatMultiTokenPredictorLayer(nn.Module): prefix="eh_proj", ) self.mtp_block = DeepseekV2DecoderLayer(vllm_config, prefix) + scale_mla_lora_norms_on_load(self.mtp_block.self_attn, config) self.final_layernorm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps) def forward( @@ -263,50 +264,6 @@ class LongCatFlashMTP(nn.Module): weight_loader = getattr(param, "weight_loader", default_weight_loader) weight_loader(param, loaded_weight) loaded_params.add(name) - spec_layer_id = self.model.mtp_start_layer_idx - self_attn = self.model.layers[str(spec_layer_id)].mtp_block.self_attn - if ( - self.quant_config is not None - and hasattr(self.quant_config, "weight_block_size") - and self_attn.kv_b_proj.weight.dtype - in ( - torch.float8_e4m3fn, - torch.float8_e4m3fnuz, - ) - ): - weight_block_size = self.quant_config.weight_block_size - if weight_block_size is not None: - dtype = torch.get_default_dtype() - w = block_dequant( - self_attn.kv_b_proj.weight, - self_attn.kv_b_proj.weight_scale_inv, - weight_block_size, - ).to(dtype) - else: - w = self_attn.kv_b_proj.weight - else: - w = self_attn.kv_b_proj.weight - w_kc, w_vc = w.unflatten( - 0, (-1, self_attn.qk_nope_head_dim + self_attn.v_head_dim) - ).split([self_attn.qk_nope_head_dim, self_attn.v_head_dim], dim=1) - self_attn.w_kc = w_kc.transpose(1, 2).contiguous().transpose(1, 2) - self_attn.w_vc = w_vc.contiguous().transpose(1, 2) - # Guard against compounding on incremental load_weights calls (the - # in-place *= would otherwise double-apply the LoRA scaling). - if self.config.mla_scale_q_lora and not getattr( - self_attn, "_mla_q_lora_scaled", False - ): - self_attn.q_a_layernorm.weight.data *= ( - self.config.hidden_size / self.config.q_lora_rank - ) ** 0.5 - self_attn._mla_q_lora_scaled = True - if self.config.mla_scale_kv_lora and not getattr( - self_attn, "_mla_kv_lora_scaled", False - ): - self_attn.kv_a_layernorm.weight.data *= ( - self.config.hidden_size / self.config.kv_lora_rank - ) ** 0.5 - self_attn._mla_kv_lora_scaled = True return loaded_params def _rewrite_spec_layer_name( diff --git a/vllm/model_executor/models/mimo_mtp.py b/vllm/model_executor/models/mimo_mtp.py index 65566230a5..5e331fa8f0 100644 --- a/vllm/model_executor/models/mimo_mtp.py +++ b/vllm/model_executor/models/mimo_mtp.py @@ -73,7 +73,7 @@ class MiMoMultiTokenPredictorLayer(nn.Module): ) -> torch.Tensor: assert inputs_embeds is not None # masking inputs at position 0, as not needed by MTP - inputs_embeds[positions == 0] = 0 + inputs_embeds = torch.where(positions.unsqueeze(-1) == 0, 0, inputs_embeds) inputs_embeds = self.token_layernorm(inputs_embeds) previous_hidden_states = self.hidden_layernorm(previous_hidden_states) diff --git a/vllm/model_executor/models/mimo_v2.py b/vllm/model_executor/models/mimo_v2.py index 6267a37dc0..a6464c7c84 100644 --- a/vllm/model_executor/models/mimo_v2.py +++ b/vllm/model_executor/models/mimo_v2.py @@ -372,6 +372,10 @@ class MiMoV2FlashDecoderLayer(nn.Module): max_position_embeddings = getattr(config, "max_position_embeddings", 32768) v_scale = getattr(config, "attention_value_scale", None) + # cache_config.sliding_window holds MiMo's SWA window; global layers + # would otherwise fall back to it in Attention. + cache_config = vllm_config.cache_config + cache_config.sliding_window = None if self.is_compressed_softmax_layer(): self.self_attn = MiMoV2Attention( @@ -389,6 +393,7 @@ class MiMoV2FlashDecoderLayer(nn.Module): layer_id=layer_id, rope_theta=getattr(config, "swa_rope_theta", rope_theta), max_position_embeddings=max_position_embeddings, + cache_config=cache_config, quant_config=quant_config, partial_rotary_factor=getattr(config, "partial_rotary_factor", 1.0), prefix=f"{prefix}.self_attn", @@ -406,6 +411,7 @@ class MiMoV2FlashDecoderLayer(nn.Module): layer_id=layer_id, rope_theta=rope_theta, max_position_embeddings=max_position_embeddings, + cache_config=cache_config, quant_config=quant_config, partial_rotary_factor=getattr(config, "partial_rotary_factor", 1.0), prefix=f"{prefix}.self_attn", diff --git a/vllm/model_executor/models/mimo_v2_mtp.py b/vllm/model_executor/models/mimo_v2_mtp.py index c413b65ba9..fe6d64552d 100644 --- a/vllm/model_executor/models/mimo_v2_mtp.py +++ b/vllm/model_executor/models/mimo_v2_mtp.py @@ -24,7 +24,7 @@ import torch import torch.nn as nn from transformers import PreTrainedConfig -from vllm.config import VllmConfig +from vllm.config import CacheConfig, VllmConfig from vllm.distributed import ( get_tensor_model_parallel_rank, get_tensor_model_parallel_world_size, @@ -64,6 +64,7 @@ class MiMoV2MTPLayer(nn.Module): self, config: PreTrainedConfig, prefix: str, + cache_config: CacheConfig | None = None, quant_config: QuantizationConfig | None = None, ) -> None: super().__init__() @@ -100,6 +101,7 @@ class MiMoV2MTPLayer(nn.Module): layer_id=0, rope_theta=swa_rope_theta, max_position_embeddings=getattr(config, "max_position_embeddings", 32768), + cache_config=cache_config, quant_config=quant_config, partial_rotary_factor=getattr(config, "partial_rotary_factor", 1.0), prefix=f"{prefix}.self_attn", @@ -147,6 +149,7 @@ class _MiMoV2MTPLayers(nn.Module): self, config: PreTrainedConfig, num_mtp_layers: int, + cache_config: CacheConfig | None, quant_config: QuantizationConfig | None, prefix: str, ) -> None: @@ -156,6 +159,7 @@ class _MiMoV2MTPLayers(nn.Module): str(i): MiMoV2MTPLayer( config=config, prefix=f"{prefix}.{i}", + cache_config=cache_config, quant_config=quant_config, ) for i in range(num_mtp_layers) @@ -182,6 +186,7 @@ class MiMoV2MultiTokenPredictor(nn.Module): self.mtp = _MiMoV2MTPLayers( config=config, num_mtp_layers=num_mtp_layers, + cache_config=vllm_config.cache_config, quant_config=vllm_config.quant_config, prefix=maybe_prefix(prefix, "mtp.layers"), ) diff --git a/vllm/model_executor/models/mistral_large_3_eagle.py b/vllm/model_executor/models/mistral_large_3_eagle.py index d2e263f3e7..1398fd19fc 100644 --- a/vllm/model_executor/models/mistral_large_3_eagle.py +++ b/vllm/model_executor/models/mistral_large_3_eagle.py @@ -23,8 +23,17 @@ from vllm.model_executor.models.deepseek_v2 import ( ) from vllm.model_executor.models.mistral_large_3 import MistralLarge3ForCausalLM -from .interfaces import SupportsMultiModal, SupportsMultiModalEmbeddings -from .utils import WeightsMapper, make_empty_intermediate_tensors_factory, maybe_prefix +from .interfaces import ( + MultiModalEmbeddings, + SupportsMultiModalEmbeddings, + _require_is_multimodal, +) +from .utils import ( + WeightsMapper, + _merge_multimodal_embeddings, + make_empty_intermediate_tensors_factory, + maybe_prefix, +) logger = init_logger(__name__) @@ -148,7 +157,21 @@ class EagleMistralLarge3ForCausalLM( def get_language_model(self) -> torch.nn.Module: return self.model - embed_input_ids = SupportsMultiModal.embed_input_ids # type: ignore + def embed_input_ids( + self, + input_ids: torch.Tensor, + multimodal_embeddings: MultiModalEmbeddings | None = None, + *, + is_multimodal: torch.Tensor | None = None, + ) -> torch.Tensor: + inputs_embeds = self.model.embed_input_ids(input_ids) + if multimodal_embeddings is None or len(multimodal_embeddings) == 0: + return inputs_embeds + return _merge_multimodal_embeddings( + inputs_embeds=inputs_embeds, + multimodal_embeddings=multimodal_embeddings, + is_multimodal=_require_is_multimodal(is_multimodal), + ) def forward( self, diff --git a/vllm/model_executor/models/modernbert.py b/vllm/model_executor/models/modernbert.py index 3a8abee67e..0c1909445b 100644 --- a/vllm/model_executor/models/modernbert.py +++ b/vllm/model_executor/models/modernbert.py @@ -94,28 +94,12 @@ class ModernBertAttention(nn.Module): prefix=f"{prefix}.Wqkv", ) - if layer_types := getattr(config, "layer_types", None): - # Transformers v5 - layer_type = layer_types[layer_id] - rope_parameters = config.rope_parameters[layer_type] - sliding_window: int | None = None - if layer_type == "sliding_attention": - # Treats the local attention boundary as inclusive - sliding_window = config.sliding_window + 1 - else: - # Transformers v4 - sliding_window = None - if layer_id % config.global_attn_every_n_layers != 0: - # ModernBertConfig does not expose sliding_window - sliding_window = config.local_attention // 2 - rope_theta = ( - config.local_rope_theta - if config.local_rope_theta is not None - else config.global_rope_theta - ) - else: - rope_theta = config.global_rope_theta - rope_parameters = {"rope_type": "default", "rope_theta": rope_theta} + layer_type = config.layer_types[layer_id] + rope_parameters = config.rope_parameters[layer_type] + sliding_window: int | None = None + if layer_type == "sliding_attention": + # Treats the local attention boundary as inclusive + sliding_window = config.sliding_window + 1 self.rotary_emb = get_rope( head_size=self.head_dim, diff --git a/vllm/model_executor/models/nemotron.py b/vllm/model_executor/models/nemotron.py index a271432813..5521857696 100644 --- a/vllm/model_executor/models/nemotron.py +++ b/vllm/model_executor/models/nemotron.py @@ -29,6 +29,7 @@ from itertools import islice import torch from torch import nn +from transformers import NemotronConfig from vllm.compilation.decorators import support_torch_compile from vllm.config import CacheConfig, VllmConfig @@ -49,7 +50,6 @@ from vllm.model_executor.layers.vocab_parallel_embedding import ( VocabParallelEmbedding, ) from vllm.sequence import IntermediateTensors -from vllm.transformers_utils.configs.nemotron import NemotronConfig from .interfaces import SupportsLoRA, SupportsPP from .utils import ( diff --git a/vllm/model_executor/models/nemotron_h.py b/vllm/model_executor/models/nemotron_h.py index bef3da5cc5..ae653950ac 100644 --- a/vllm/model_executor/models/nemotron_h.py +++ b/vllm/model_executor/models/nemotron_h.py @@ -23,6 +23,7 @@ from itertools import islice import torch from torch import nn +from transformers import NemotronHConfig from vllm.compilation.decorators import support_torch_compile from vllm.config import ( @@ -87,7 +88,6 @@ from vllm.model_executor.models.utils import ( sequence_parallel_chunk, ) from vllm.sequence import IntermediateTensors -from vllm.transformers_utils.configs.nemotron_h import NemotronHConfig class NemotronHMLP(nn.Module): @@ -235,7 +235,7 @@ class NemotronHMoE(nn.Module): intermediate_size=config.moe_intermediate_size, renormalize=config.norm_topk_prob, quant_config=quant_config, - ckpt_names=("up_proj", "down_proj", ""), + ckpt_names=("up_proj", "down_proj", None), use_grouped_topk=True, num_expert_group=config.n_group, topk_group=config.topk_group, @@ -707,7 +707,7 @@ class NemotronHModel(nn.Module, EagleModelMixin): self, ckpt_gate_proj_name="up_proj", ckpt_down_proj_name="down_proj", - ckpt_up_proj_name="", + ckpt_up_proj_name=None, num_experts=self._get_max_n_routed_experts(), num_redundant_experts=getattr(self, "num_redundant_experts", 0), ) diff --git a/vllm/model_executor/models/nemotron_h_mtp.py b/vllm/model_executor/models/nemotron_h_mtp.py index 6b9de63009..1e393296bc 100644 --- a/vllm/model_executor/models/nemotron_h_mtp.py +++ b/vllm/model_executor/models/nemotron_h_mtp.py @@ -7,6 +7,7 @@ from collections.abc import Callable, Iterable import torch import torch.nn as nn +from transformers import NemotronHConfig from vllm.compilation.decorators import support_torch_compile from vllm.config import CacheConfig, ModelConfig, VllmConfig @@ -33,7 +34,6 @@ from vllm.model_executor.models.utils import ( maybe_prefix, ) from vllm.sequence import IntermediateTensors -from vllm.transformers_utils.configs.nemotron_h import NemotronHConfig from .interfaces import SupportsPP, SupportsQuant from .nemotron_h import ( @@ -426,6 +426,11 @@ class NemotronHMTP(nn.Module, SupportsPP, SupportsQuant): ) return self.logits_processor(self.lm_head, hidden_states) + def is_unused_checkpoint_weight(self, name: str) -> bool: + """Only MTP, embedding and LM head weights are loaded.""" + name = name.removeprefix("language_model.") + return not name.startswith(("mtp.", "lm_head.")) and "embeddings" not in name + def load_weights(self, weights: Iterable[tuple[str, torch.Tensor]]) -> set[str]: """Load MTP weights with proper name remapping.""" stacked_params_mapping = [ @@ -444,7 +449,7 @@ class NemotronHMTP(nn.Module, SupportsPP, SupportsQuant): self, ckpt_gate_proj_name="up_proj", ckpt_down_proj_name="down_proj", - ckpt_up_proj_name="", # Empty - non-gated MoE + ckpt_up_proj_name=None, # non-gated MoE num_experts=num_experts, num_redundant_experts=self.num_redundant_experts, ) @@ -456,17 +461,11 @@ class NemotronHMTP(nn.Module, SupportsPP, SupportsQuant): # MTP weights are nested in "language_model." # in Multimodal Nemotron-H checkpoints. name = name.removeprefix("language_model.") + if self.is_unused_checkpoint_weight(name): + continue is_lm_head_weight = name.startswith("lm_head.") if is_lm_head_weight: self.has_own_lm_head = True - # Only process MTP and LM head weights - - # skip all non-MTP and non-LM head weights - if ( - not name.startswith("mtp.") - and "embeddings" not in name - and not is_lm_head_weight - ): - continue # Skip rotary embeddings (computed, not loaded) if "rotary_emb.inv_freq" in name: continue diff --git a/vllm/model_executor/models/nemotron_parse.py b/vllm/model_executor/models/nemotron_parse.py index 9ac38c0021..3ba731d2e0 100644 --- a/vllm/model_executor/models/nemotron_parse.py +++ b/vllm/model_executor/models/nemotron_parse.py @@ -568,6 +568,15 @@ class NemotronParseForConditionalGeneration(nn.Module, SupportsMultiModal): self.lm_head = ParallelLMHead( config.decoder.vocab_size, config.decoder.d_model, quant_config=quant_config ) + # Some checkpoints (e.g. compact exports) tie the output head to the + # decoder's input embeddings instead of materializing a separate + # lm_head.weight tensor. + tie_word_embeddings = bool( + getattr(config, "tie_word_embeddings", False) + or getattr(config.decoder, "tie_word_embeddings", False) + ) + if tie_word_embeddings: + self.lm_head = self.lm_head.tie_weights(self.decoder.embed_tokens) self.logits_processor = LogitsProcessor( self.vocab_size, config.decoder.vocab_size ) diff --git a/vllm/model_executor/models/ovis.py b/vllm/model_executor/models/ovis.py index 7c1668ddab..9e7778da15 100644 --- a/vllm/model_executor/models/ovis.py +++ b/vllm/model_executor/models/ovis.py @@ -125,7 +125,7 @@ class VisualTokenizer(torch.nn.Module): prefix: str = "", ) -> nn.Module: model_type = config.backbone_config.model_type - if model_type == "aimv2": + if model_type == "aimv2_vision_model": # No post rms_norm in Ovis2's AIMv2 ViT. return AIMv2Model( config=config.backbone_config, diff --git a/vllm/model_executor/models/phi4mm_audio.py b/vllm/model_executor/models/phi4mm_audio.py index 2a16b4ca75..9daedcf38e 100644 --- a/vllm/model_executor/models/phi4mm_audio.py +++ b/vllm/model_executor/models/phi4mm_audio.py @@ -129,7 +129,7 @@ class ConformerEncoderLayer(nn.Module): default False. use_pt_scaled_dot_product_attention: bool, optional if set to True, use pytorch's scaled dot product attention - implementation in training. + implementation in training. Default: True. attn_group_sizes: int, optional the number of groups to use for attention, default 1 (Multi-Head Attention), @@ -163,7 +163,7 @@ class ConformerEncoderLayer(nn.Module): attention_glu_type: str = "swish", activation_checkpointing: str = "", export: bool = False, - use_pt_scaled_dot_product_attention: bool = False, + use_pt_scaled_dot_product_attention: bool = True, attn_group_sizes: int = 1, ) -> None: super().__init__() @@ -808,7 +808,7 @@ class ConformerEncoder(TransformerEncoderBase): default 4 use_pt_scaled_dot_product_attention: whether to use pytorch scaled dot product attention in training. - Default: False + Default: True nemo_conv_settings: dict, optional A dictionary of settings for NeMo Subsampling. default: None @@ -873,7 +873,7 @@ class ConformerEncoder(TransformerEncoderBase): activation_checkpointing: str = "", relative_attention_bias_args: dict[str, Any] | None = None, time_reduction: int = 4, - use_pt_scaled_dot_product_attention: bool = False, + use_pt_scaled_dot_product_attention: bool = True, nemo_conv_settings: dict[str, Any] | None = None, conv2d_extra_padding: Literal["feat", "feat_time", "none", True] = "none", replication_pad_for_subsample_embedding: bool = False, diff --git a/vllm/model_executor/models/phi4mm_utils.py b/vllm/model_executor/models/phi4mm_utils.py index af54defd3f..db1b83f343 100644 --- a/vllm/model_executor/models/phi4mm_utils.py +++ b/vllm/model_executor/models/phi4mm_utils.py @@ -1642,7 +1642,7 @@ class MultiHeadedAttention(nn.Module): NOTE: this will NOT be used in ONNX decoding due to a lack of support. In that case, we use the original attention implementation, which shows no regression. - default: False. + default: True. n_value: int, optional if set to values other than -1, use a different dimension for value. With the default value (i.e. -1), it is backward compatible. @@ -1666,7 +1666,7 @@ class MultiHeadedAttention(nn.Module): attention_inner_dim: int = -1, glu_type: str = "swish", bias_in_glu: bool = True, - use_pt_scaled_dot_product_attention: bool = False, + use_pt_scaled_dot_product_attention: bool = True, n_value: int = -1, group_size: int = 1, ) -> None: @@ -1690,7 +1690,6 @@ class MultiHeadedAttention(nn.Module): self.linear_v = nn.Linear(n_value, attention_inner_dim // group_size) self.linear_out = nn.Linear(attention_inner_dim // group_size, n_value) - self.attn = torch.jit.Attribute(None, Tensor | None) self.dropout = nn.Dropout(p=dropout_rate) self.dropout_rate = dropout_rate self.use_pt_scaled_dot_product_attention = use_pt_scaled_dot_product_attention @@ -1799,8 +1798,6 @@ class MultiHeadedAttention(nn.Module): attn = masked_softmax(scores, mask) # (batch, head, time1, time2) - self.attn = attn - p_attn = self.dropout(attn) x = torch.matmul(p_attn.to(v.dtype), v) # (batch, head, time1, d_k) if pos_v is not None: diff --git a/vllm/model_executor/models/pixtral.py b/vllm/model_executor/models/pixtral.py index fb10026ad1..4f96c0acfc 100644 --- a/vllm/model_executor/models/pixtral.py +++ b/vllm/model_executor/models/pixtral.py @@ -32,6 +32,7 @@ from vllm.inputs import MultiModalDataDict from vllm.model_executor.layers.activation import SiluAndMul, get_act_and_mul_fn from vllm.model_executor.layers.attention import MMEncoderAttention from vllm.model_executor.layers.conv import Conv2dLayer +from vllm.model_executor.layers.fusion.fused_act_quant import maybe_fused_act_quant from vllm.model_executor.layers.layernorm import RMSNorm from vllm.model_executor.layers.linear import ( MergedColumnParallelLinear, @@ -758,7 +759,7 @@ class FeedForward(nn.Module): def forward(self, x): x, _ = self.gate_up_proj(x) - x = self.act_fn(x) + x = maybe_fused_act_quant(self.act_fn, x, self.down_proj) x, _ = self.down_proj(x) return x @@ -1259,7 +1260,7 @@ class PixtralHFMLP(nn.Module): def forward(self, x: torch.Tensor) -> torch.Tensor: gate_up, _ = self.gate_up_proj(x) - x = self.act_and_mul(gate_up) + x = maybe_fused_act_quant(self.act_and_mul, gate_up, self.down_proj) x, _ = self.down_proj(x) return x diff --git a/vllm/model_executor/models/qwen3_5.py b/vllm/model_executor/models/qwen3_5.py index a9a8d294cb..5b72132160 100644 --- a/vllm/model_executor/models/qwen3_5.py +++ b/vllm/model_executor/models/qwen3_5.py @@ -29,6 +29,12 @@ from typing import ClassVar import torch from torch import nn +from transformers import ( + Qwen3_5Config, + Qwen3_5MoeConfig, + Qwen3_5MoeTextConfig, + Qwen3_5TextConfig, +) from vllm.compilation.decorators import support_torch_compile from vllm.config import VllmConfig @@ -59,11 +65,6 @@ from vllm.multimodal import MULTIMODAL_REGISTRY from vllm.multimodal.inputs import MultiModalFeatureSpec from vllm.sequence import IntermediateTensors from vllm.tokenizers.registry import cached_tokenizer_from_config -from vllm.transformers_utils.configs.qwen3_5 import Qwen3_5Config, Qwen3_5TextConfig -from vllm.transformers_utils.configs.qwen3_5_moe import ( - Qwen3_5MoeConfig, - Qwen3_5MoeTextConfig, -) from .interfaces import ( HasInnerState, diff --git a/vllm/model_executor/models/qwen3_5_mtp.py b/vllm/model_executor/models/qwen3_5_mtp.py index bba79149e4..763c6a5eb6 100644 --- a/vllm/model_executor/models/qwen3_5_mtp.py +++ b/vllm/model_executor/models/qwen3_5_mtp.py @@ -8,6 +8,7 @@ from collections.abc import Iterable import huggingface_hub import torch from torch import nn +from transformers import Qwen3_5MoeTextConfig, Qwen3_5TextConfig from vllm.compilation.decorators import support_torch_compile from vllm.config import ModelConfig, VllmConfig @@ -34,8 +35,6 @@ from vllm.model_executor.models.qwen3_next import ( ) from vllm.model_executor.models.utils import sequence_parallel_chunk from vllm.sequence import IntermediateTensors -from vllm.transformers_utils.configs.qwen3_5 import Qwen3_5TextConfig -from vllm.transformers_utils.configs.qwen3_5_moe import Qwen3_5MoeTextConfig from vllm.transformers_utils.repo_utils import try_get_local_file from .interfaces import ( @@ -376,16 +375,20 @@ class Qwen3_5MTP(LocalArgmaxMixin, nn.Module, SupportsMultiModal, SupportsPP): ) -> torch.Tensor | None: return self.logits_processor(self.lm_head, hidden_states) + def is_unused_checkpoint_weight(self, name: str) -> bool: + return not name.startswith("mtp.") and not any( + key in name for key in ["embed_tokens", "lm_head"] + ) + def load_weights(self, weights: Iterable[tuple[str, torch.Tensor]]) -> set[str]: def remap_weight_names(weights): for name, weight in weights: + if self.is_unused_checkpoint_weight(name): + continue if name.startswith("mtp."): name = name.replace("mtp.", "model.") - elif any(key in name for key in ["embed_tokens", "lm_head"]): - if "embed_tokens" in name: - name = name.replace("language_model.", "") - else: - continue + elif "embed_tokens" in name: + name = name.replace("language_model.", "") yield name, weight loader = AutoWeightsLoader(self) diff --git a/vllm/model_executor/models/qwen3_asr.py b/vllm/model_executor/models/qwen3_asr.py index e8e291d8a5..c8a0849fc5 100644 --- a/vllm/model_executor/models/qwen3_asr.py +++ b/vllm/model_executor/models/qwen3_asr.py @@ -39,6 +39,7 @@ from vllm.logger import init_logger from vllm.model_executor.models.interfaces import ( MultiModalEmbeddings, StreamingTranscriptionPostProcessor, + SupportsEagle3, SupportsLoRA, SupportsMRoPE, SupportsMultiModal, @@ -345,6 +346,7 @@ class Qwen3ASRForConditionalGeneration( SupportsMRoPE, SupportsTranscription, SupportsLoRA, + SupportsEagle3, ): # LoRA support packed_modules_mapping = { diff --git a/vllm/model_executor/models/qwen3_dflash.py b/vllm/model_executor/models/qwen3_dflash.py index 3161cf2261..44b7865daa 100644 --- a/vllm/model_executor/models/qwen3_dflash.py +++ b/vllm/model_executor/models/qwen3_dflash.py @@ -34,6 +34,7 @@ from vllm.model_executor.layers.vocab_parallel_embedding import ( VocabParallelEmbedding, ) from vllm.multimodal.inputs import NestedTensors +from vllm.platforms import current_platform from vllm.transformers_utils.config import set_default_rope_theta from vllm.transformers_utils.repo_utils import get_hf_file_bytes from vllm.v1.attention.backend import AttentionType @@ -269,6 +270,24 @@ class DFlashQwen3Attention(nn.Module): sinks=self.attention_sink_bias, ) self.causal = causal + if current_platform.is_rocm(): + from vllm.platforms.rocm import on_gfx1x + from vllm.v1.attention.backends.rocm_segmented_attn import ( + RocmSegmentedAttentionImpl, + ) + + if on_gfx1x() and isinstance(self.attn.impl, RocmSegmentedAttentionImpl): + # Startup segmented tuning needs the draft's actual attention mode. + # Causality is normally supplied later through attention metadata. + self.attn.segmented_causal = causal + speculative_config = get_current_vllm_config().speculative_config + if ( + speculative_config is not None + and speculative_config.num_speculative_tokens is not None + ): + self.attn.segmented_query_limit = ( + speculative_config.num_speculative_tokens + 1 + ) self.q_norm = RMSNorm(self.head_dim, eps=rms_norm_eps) self.k_norm = RMSNorm(self.head_dim, eps=rms_norm_eps) diff --git a/vllm/model_executor/models/qwen3_next.py b/vllm/model_executor/models/qwen3_next.py index 19fc8c1562..b0bf0fecd1 100644 --- a/vllm/model_executor/models/qwen3_next.py +++ b/vllm/model_executor/models/qwen3_next.py @@ -7,6 +7,7 @@ from itertools import islice import torch from torch import nn +from transformers import Qwen3NextConfig from vllm.compilation.decorators import support_torch_compile from vllm.config import CacheConfig, ModelConfig, VllmConfig @@ -58,7 +59,6 @@ from vllm.model_executor.models.qwen2_moe import Qwen2MoeMLP as Qwen3NextMLP from vllm.model_executor.models.utils import sequence_parallel_chunk from vllm.platforms import current_platform from vllm.sequence import IntermediateTensors -from vllm.transformers_utils.configs.qwen3_next import Qwen3NextConfig from vllm.v1.attention.backend import AttentionType from .interfaces import ( @@ -232,6 +232,7 @@ class Qwen3NextSparseMoeBlock(nn.Module): is_sequence_parallel=self.is_sequence_parallel, n_shared_experts=1 if self.shared_expert is None else None, fuse_shared_experts=self.is_fused_shared_expert_enabled, + shared_expert_prefix=f"{prefix}.shared_expert", shared_expert_gate=self.shared_expert_gate if self.shared_expert is None else None, diff --git a/vllm/model_executor/models/qwen3_next_mtp.py b/vllm/model_executor/models/qwen3_next_mtp.py index e5e306bb86..4ae8f21840 100644 --- a/vllm/model_executor/models/qwen3_next_mtp.py +++ b/vllm/model_executor/models/qwen3_next_mtp.py @@ -6,6 +6,7 @@ from collections.abc import Iterable import torch from torch import nn +from transformers import Qwen3NextConfig from vllm.compilation.decorators import support_torch_compile from vllm.config import VllmConfig @@ -29,7 +30,6 @@ from vllm.model_executor.models.qwen3_next import ( ) from vllm.model_executor.models.utils import sequence_parallel_chunk from vllm.sequence import IntermediateTensors -from vllm.transformers_utils.configs.qwen3_next import Qwen3NextConfig from .utils import ( AutoWeightsLoader, @@ -227,15 +227,18 @@ class Qwen3NextMTP(nn.Module, QwenNextMixtureOfExperts): ) -> torch.Tensor | None: return self.logits_processor(self.lm_head, hidden_states) - def load_weights(self, weights: Iterable[tuple[str, torch.Tensor]]) -> set[str]: - shared_weight_names = ["embed_tokens", "lm_head"] + def is_unused_checkpoint_weight(self, name: str) -> bool: + return not name.startswith("mtp.") and not any( + key in name for key in ["embed_tokens", "lm_head"] + ) + def load_weights(self, weights: Iterable[tuple[str, torch.Tensor]]) -> set[str]: def remap_weight_names(weights): for name, weight in weights: + if self.is_unused_checkpoint_weight(name): + continue if name.startswith("mtp."): name = name.replace("mtp.", "model.") - elif not any(key in name for key in shared_weight_names): - continue yield name, weight loader = AutoWeightsLoader(self) diff --git a/vllm/model_executor/models/qwen3_omni_moe_thinker.py b/vllm/model_executor/models/qwen3_omni_moe_thinker.py index d3510f3e64..4ae80a4a36 100755 --- a/vllm/model_executor/models/qwen3_omni_moe_thinker.py +++ b/vllm/model_executor/models/qwen3_omni_moe_thinker.py @@ -2311,8 +2311,9 @@ class Qwen3OmniMoeThinkerForConditionalGeneration( llm_pos_ids_list.append(eos_pos) st = offset + video_len + 1 else: - audio_bos_pos = np.broadcast_to(np.array([st_idx - 1]), (3, 1)) + audio_bos_pos = np.broadcast_to(np.array([st_idx]), (3, 1)) llm_pos_ids_list.append(audio_bos_pos) + st_idx += 1 pos_ids, _ = self._compute_interleaved_positions(st_idx, data) llm_pos_ids_list.append(pos_ids) @@ -2320,7 +2321,7 @@ class Qwen3OmniMoeThinkerForConditionalGeneration( eos_pos = np.broadcast_to(np.array([st_idx]), (3, 1)) llm_pos_ids_list.append(eos_pos) - llm_pos_ids_list.append(eos_pos) + llm_pos_ids_list.append(eos_pos + 1) video_len = grid_t * grid_h * grid_w audio_len = self._compute_audio_token_count( diff --git a/vllm/model_executor/models/registry.py b/vllm/model_executor/models/registry.py index 3ff37a2562..e019d674f5 100644 --- a/vllm/model_executor/models/registry.py +++ b/vllm/model_executor/models/registry.py @@ -246,6 +246,7 @@ _EMBEDDING_MODELS = { # [Multimodal] "CLIPModel": ("clip", "CLIPEmbeddingModel"), "ColPaliForRetrieval": ("colpali", "ColPaliModel"), + "EmbeddingGemma2Model": ("embedding_gemma2", "EmbeddingGemma2Model"), "LlamaNemotronVLModel": ("nemotron_vl", "LlamaNemotronVLForEmbedding"), "LlavaNextForConditionalGeneration": ( "llava_next", @@ -849,6 +850,7 @@ _OOT_SUPPORTED_MODELS = { "BartForConditionalGeneration": "https://github.com/vllm-project/bart-plugin", "Florence2ForConditionalGeneration": "https://github.com/vllm-project/bart-plugin", "MBartForConditionalGeneration": "https://github.com/vllm-project/bart-plugin", + "Nemotron3_5AsrForRNNT": "https://github.com/Sohaib-Ahmed21/vllm-nemotron-asr-plugin", } diff --git a/vllm/model_executor/models/voxtral_realtime.py b/vllm/model_executor/models/voxtral_realtime.py index 8531d7d18f..72318976f1 100644 --- a/vllm/model_executor/models/voxtral_realtime.py +++ b/vllm/model_executor/models/voxtral_realtime.py @@ -120,9 +120,35 @@ def _expand_tensor(input_tensor: torch.Tensor, scaling: int) -> torch.Tensor: return (base.unsqueeze(1) + offsets).view(-1) +class VoxtralRealtimeProcessingInfo(VoxtralProcessingInfo): + def get_supported_mm_limits(self) -> Mapping[str, int | None]: + # Each streaming update carries a single audio frame. + return {"audio": 1} + + def get_max_audio_array_len(self) -> int: + # Profile with the largest streaming chunk, the first one: it covers + # the prompt prefix (BOS + left padding + delay), later chunks are + # ~1 token. get_max_audio_tokens keeps max_model_len so the encoder + # cache stays as large as before: each paused session keeps its last + # chunk cached until it resumes, and a cache sized to one chunk + # deadlocks a few concurrent sessions. + tokenizer = self.get_tokenizer() + audio_encoder = tokenizer.instruct.audio_encoder + prompt_tokens = ( + tokenizer.instruct.start() + audio_encoder.encode_streaming_tokens() + ) + return len(prompt_tokens) * audio_encoder.audio_config.raw_audio_length_per_tok + + class VoxtralRealtimeBuffer: - def __init__(self, config: AudioConfig, prompt_tokens: list[int]) -> None: + def __init__( + self, + config: AudioConfig, + prompt_tokens: list[int], + max_model_len: int | None = None, + ) -> None: self._config = config + self._max_model_len = max_model_len _look_ahead_in_ms = self._config.streaming_look_ahead_ms _look_back_in_ms = self._config.streaming_look_back_ms @@ -165,7 +191,16 @@ class VoxtralRealtimeBuffer: await self._token_queue.put(token) async def get_input_stream(self) -> AsyncGenerator[TokensPrompt]: + n_tokens = 0 for frame_size, num_tokens in self._generate_frame_size_and_num_tokens(): + n_tokens += num_tokens + if self._max_model_len is not None and n_tokens >= self._max_model_len: + logger.warning( + "Stopping realtime audio stream: reached max_model_len (%d).", + self._max_model_len, + ) + return + next_tokens = [await self._token_queue.get() for _ in range(num_tokens)] audio_arrays: list[np.ndarray] = ( @@ -198,7 +233,7 @@ class VoxtralRealtimeBuffer: @MULTIMODAL_REGISTRY.register_processor( VoxtralRealtimeMultiModalProcessor, - info=VoxtralProcessingInfo, + info=VoxtralRealtimeProcessingInfo, dummy_inputs=VoxtralDummyInputsBuilder, ) @support_torch_compile @@ -248,7 +283,9 @@ class VoxtralRealtimeGeneration(VoxtralForConditionalGeneration, SupportsRealtim # Get left/right padding audio left_pad, right_pad = audio_encoder.get_padding_audio() - buffer = VoxtralRealtimeBuffer(config, prompt_tokens) + buffer = VoxtralRealtimeBuffer( + config, prompt_tokens, max_model_len=model_config.max_model_len + ) # Feed audio with padding into buffer in background async def feed_audio(): diff --git a/vllm/model_executor/models/zamba2.py b/vllm/model_executor/models/zamba2.py index 6506c0776e..2e90da86ea 100644 --- a/vllm/model_executor/models/zamba2.py +++ b/vllm/model_executor/models/zamba2.py @@ -896,6 +896,7 @@ class Zamba2ForCausalLM(nn.Module, HasInnerState, IsHybrid, SupportsMambaPrefixC head_dim=hf_config.mamba_headdim, state_size=hf_config.mamba_d_state, conv_kernel=hf_config.mamba_d_conv, + num_spec=vllm_config.num_speculative_tokens, ) @classmethod diff --git a/vllm/model_executor/warmup/deep_gemm_warmup.py b/vllm/model_executor/warmup/deep_gemm_warmup.py index 1266f4ccb2..a4775ce22e 100644 --- a/vllm/model_executor/warmup/deep_gemm_warmup.py +++ b/vllm/model_executor/warmup/deep_gemm_warmup.py @@ -179,6 +179,7 @@ def _deepgemm_fp8_gemm_nt_warmup( device = w.device a1q = torch.empty((max_tokens, k), device=device, dtype=torch.float8_e4m3fn) + # Must be initialized (UE8M0 packing asserts zero sign/mantissa bits). a1q_scales = torch.zeros( (max_tokens, k // block_m), device=device, dtype=torch.float32 ) @@ -284,6 +285,7 @@ def _deepgemm_grouped_fp8_gemm_nt_contiguous_warmup( ((MAX_M, k // block_m), torch.float32), ((MAX_M, n), torch.bfloat16), ) + # Must be initialized (UE8M0 packing asserts zero sign/mantissa bits). a1q_scales.zero_() for num_tokens, align_used, expert_ids in warmup_cases: diff --git a/vllm/model_executor/warmup/kernel_warmup.py b/vllm/model_executor/warmup/kernel_warmup.py index 228ce73803..e5751730e8 100644 --- a/vllm/model_executor/warmup/kernel_warmup.py +++ b/vllm/model_executor/warmup/kernel_warmup.py @@ -5,8 +5,10 @@ This is useful specifically for JIT'ed kernels as we don't want JIT'ing to happen during model execution. """ +import json import sys import time +from pathlib import Path from typing import TYPE_CHECKING import torch @@ -18,7 +20,6 @@ from vllm.model_executor.warmup.cutedsl_warmup import cutedsl_warmup from vllm.model_executor.warmup.deep_gemm_warmup import deep_gemm_warmup from vllm.model_executor.warmup.flashinfer_autotune_cache import ( resolve_flashinfer_autotune_file, - write_flashinfer_autotune_cache, ) from vllm.model_executor.warmup.flashinfer_sparse_mla_warmup import ( autotune_hisparse_flashinfer_attention, @@ -220,6 +221,13 @@ def kernel_warmup(worker: "Worker", *, process_local_only: bool = False): if process_local_only: return + if current_platform.is_rocm(): + from vllm.model_executor.warmup.rocm_segmented_attn_autotune_warmup import ( + rocm_segmented_attn_autotune_warmup, + ) + + rocm_segmented_attn_autotune_warmup(worker) + flashinfer_sparse_mla_decode_autotune_warmup(worker) deepseek_v4_sparse_mla_attention_warmup(worker) @@ -308,9 +316,6 @@ def _flashinfer_autotune_skip_ops(runner: "GPUModelRunner") -> set[str] | None: return None -_FLASHINFER_BF16_AUTOTUNE_MAX_TOKENS = 32 - - def _flashinfer_deferred_moe_token_counts( runner: "GPUModelRunner", ) -> tuple[int, ...]: @@ -331,19 +336,10 @@ def _flashinfer_deferred_moe_token_counts( return tuple(dict.fromkeys(token_counts)) -def _flashinfer_autotune_token_counts( - runner: "GPUModelRunner", *, include_bf16: bool = True -) -> tuple[int, ...]: +def _flashinfer_autotune_token_counts(runner: "GPUModelRunner") -> tuple[int, ...]: max_tokens = runner.scheduler_config.max_num_batched_tokens # Tune the widest bucket set first so bounded passes reuse its configs. token_counts = [max_tokens] - linear_backend = runner.vllm_config.kernel_config.linear_backend - if ( - include_bf16 - and linear_backend == "flashinfer_cutedsl" - and max_tokens > _FLASHINFER_BF16_AUTOTUNE_MAX_TOKENS - ): - token_counts.append(_FLASHINFER_BF16_AUTOTUNE_MAX_TOKENS) token_counts.extend(_flashinfer_deferred_moe_token_counts(runner)) return tuple(dict.fromkeys(token_counts)) @@ -354,14 +350,33 @@ def _run_flashinfer_autotune_dummy_runs( import vllm.utils.flashinfer as fi_utils dummy_run_kwargs = {"skip_attn": True} if skip_attn else {} - for num_tokens in _flashinfer_autotune_token_counts(runner, include_bf16=False): - tuning_buckets = fi_utils.flashinfer_get_hybrid_num_tokens_buckets(num_tokens) + for num_tokens in _flashinfer_autotune_token_counts(runner): + # The drafter may use more tokens than this pass in the same dummy run. + # Include its max M when building the autotune buckets. + max_tuning_tokens = num_tokens + speculator = getattr(runner, "speculator", None) + if speculator is not None: + num_query_per_req = speculator.num_query_per_req + num_draft_reqs = min( + num_tokens, + runner.max_num_reqs, + runner.max_num_tokens // num_query_per_req, + ) + max_tuning_tokens = max( + max_tuning_tokens, num_draft_reqs * num_query_per_req + ) + + tuning_buckets = fi_utils.flashinfer_get_hybrid_num_tokens_buckets( + max_tuning_tokens + ) logger.info( "Running FlashInfer autotune with %d tokens and token buckets %s.", num_tokens, tuning_buckets, ) - with fi_utils.autotune(tuning_buckets=tuning_buckets): + # Round M up to match serving-time bucket selection for non-bucket M. + # See https://github.com/flashinfer-ai/flashinfer/issues/5450 + with fi_utils.autotune(tuning_buckets=tuning_buckets, round_up=True): runner._dummy_run( num_tokens=num_tokens, skip_eplb=True, @@ -371,34 +386,24 @@ def _run_flashinfer_autotune_dummy_runs( ) -def _run_flashinfer_bf16_autotune_dummy_run( - runner: "GPUModelRunner", - *, - skip_ops: set[str] | None = None, - skip_attn: bool = False, -) -> None: - import vllm.utils.flashinfer as fi_utils +def _autotune_cache_fingerprint(path: Path) -> tuple[str, int] | None: + """Identify a saved autotune file by its FlashInfer metadata and size.""" + try: + configs = json.loads(path.read_text()) + metadata = configs.pop("_metadata", None) + except (OSError, ValueError, AttributeError): + return None + return json.dumps(metadata, sort_keys=True), len(configs) - if ( - runner.vllm_config.kernel_config.linear_backend != "flashinfer_cutedsl" - or runner.scheduler_config.max_num_batched_tokens - <= _FLASHINFER_BF16_AUTOTUNE_MAX_TOKENS - ): - return - num_tokens = _FLASHINFER_BF16_AUTOTUNE_MAX_TOKENS - tuning_buckets = fi_utils.flashinfer_get_hybrid_num_tokens_buckets(num_tokens) - logger.info("Running FlashInfer BF16-only autotune with %d tokens.", num_tokens) - # Other ops execute from their existing cache or fallback, without the - # BF16 bucket cap. No full-model autotune context may enclose this pass. - with fi_utils.autotune_bf16_only(tuning_buckets, skip_ops=skip_ops): - runner._dummy_run( - num_tokens=num_tokens, - skip_eplb=True, - is_profile=True, - randomize_inputs=True, - **({"skip_attn": True} if skip_attn else {}), - ) +def _all_ranks_have_matching_cache(path: Path, group) -> bool: + """True iff every rank in ``group`` has its own, mutually consistent file.""" + fingerprint = _autotune_cache_fingerprint(path) + if group.world_size == 1: + return fingerprint is not None + gathered: list[tuple[str, int] | None] = [None] * group.world_size + torch.distributed.all_gather_object(gathered, fingerprint, group=group.cpu_group) + return fingerprint is not None and all(f == fingerprint for f in gathered) def flashinfer_autotune(runner: "GPUModelRunner") -> None: @@ -414,6 +419,12 @@ def flashinfer_autotune(runner: "GPUModelRunner") -> None: so each stage's TP group tunes separately with its own cache file; otherwise the world group tunes together. Per-tactic timings are averaged over the tuning group so all its ranks select the same tactic. + + Results are persisted per rank: FlashInfer keys MoE entries by tp/ep rank + (``MoERunner.get_cache_key_extras``), so one rank's file only hits on that + rank. A rank with a cache hit skips the per-tactic reduce the others block + in, so ranks keep loaded configs only if every rank in the tuning group + has a matching file and successfully loads it. """ from flashinfer.autotuner import AutoTuner, set_autotune_process_group @@ -440,11 +451,11 @@ def flashinfer_autotune(runner: "GPUModelRunner") -> None: autotune_kwargs["skip_ops"] = skip_ops cache_path = resolve_flashinfer_autotune_file(runner) - if pp_size > 1: - ranks = "-".join(str(rank) for rank in tune_group.ranks) - cache_path = cache_path.with_name( - f"{cache_path.stem}_tp_{ranks}{cache_path.suffix}" - ) + # The world group only spans DP ranks when vLLM folds DP into it. + dp_rank = runner.vllm_config.parallel_config.data_parallel_rank + cache_path = cache_path.with_name( + f"{cache_path.stem}_dp{dp_rank}_rank{world.rank_in_group}{cache_path.suffix}" + ) if is_leader: logger.info_once("Using FlashInfer autotune cache file: %s", cache_path) @@ -453,16 +464,16 @@ def flashinfer_autotune(runner: "GPUModelRunner") -> None: # which lead to some EP ranks receiving no tokens and skipping their # MoE kernel entirely, and cause hang due to all-reduce collective # during synchronized autotuning. - # Read cached autotune results and broadcast within the tuning group. - cached_results: bytes | None = None - if is_leader and cache_path.exists(): - with open(cache_path, "rb") as f: - cached_results = f.read() - cached_results = tune_group.broadcast_object(cached_results, src=0) - if cached_results is not None: - write_flashinfer_autotune_cache(cache_path, cached_results) - tune_group.barrier() - tuner.load_configs(str(cache_path)) + if _all_ranks_have_matching_cache(cache_path, tune_group): + loaded = tuner.load_configs(str(cache_path)) + if tune_group.world_size > 1: + loaded_by_rank: list[bool | None] = [None] * tune_group.world_size + torch.distributed.all_gather_object( + loaded_by_rank, loaded, group=tune_group.cpu_group + ) + loaded = all(loaded_by_rank) + if not loaded: + tuner.clear_cache() group = tune_group.cpu_group if tune_group.world_size > 1 else None set_autotune_process_group(group) @@ -481,14 +492,12 @@ def flashinfer_autotune(runner: "GPUModelRunner") -> None: _run_flashinfer_autotune_dummy_runs(runner, skip_attn=hisparse_enabled) replayssm_autotune_warmup(runner) _autotune_kimi_k3_kda_qkvg(runner.get_model()) - with torch.inference_mode(): - _run_flashinfer_bf16_autotune_dummy_run( - runner, skip_ops=skip_ops, skip_attn=hisparse_enabled - ) finally: set_autotune_process_group(None) if world.world_size > 1: world.barrier() - if is_leader: + # Skip the rewrite when nothing was tuned this start (every entry came from + # the file). FlashInfer gates its own autotune(cache=...) save the same way. + if tuner._dirty: tuner.save_configs(str(cache_path)) diff --git a/vllm/model_executor/warmup/rocm_segmented_attn_autotune_warmup.py b/vllm/model_executor/warmup/rocm_segmented_attn_autotune_warmup.py new file mode 100644 index 0000000000..0ebcc58c3c --- /dev/null +++ b/vllm/model_executor/warmup/rocm_segmented_attn_autotune_warmup.py @@ -0,0 +1,1335 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Startup-only, persistent launch tuning for segmented ROCm attention.""" + +from __future__ import annotations + +import gc +import hashlib +import itertools +import json +import math +import os +import statistics +import tempfile +import time +from concurrent.futures import ThreadPoolExecutor +from contextlib import contextmanager +from pathlib import Path +from typing import TYPE_CHECKING + +import torch +from filelock import FileLock + +import vllm.envs as envs +from vllm.logger import init_logger +from vllm.platforms import current_platform +from vllm.triton_utils import triton +from vllm.v1.attention.ops import segmented_attention +from vllm.v1.attention.ops.segmented_attention import ( + MAX_QUERY_LEN, + _segmented_attention_reduce, + _segmented_attention_stage, + compile_segmented_attention, + run_segmented_attention, + segmented_query_capacity, + segmented_workspace_shapes, + select_segmented_config, +) + +if TYPE_CHECKING: + from vllm.v1.worker.gpu_worker import Worker + +logger = init_logger(__name__) + +_TABLES: dict[tuple, dict] = {} +_QUERY_BUCKETS = (1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096, 8192) +_BATCH_BUCKETS = (1, 2, 4, 8, 16, 32) +_SEQUENCE_BUCKETS = (128, 512, 2048, 8192, 32768, 131072, 262144) +_COMPILE_WORKERS = 2 +_CONDITION_ROUNDS = 2 +_SCREEN_ROUNDS = 5 +_FINALIST_COUNT = 3 +_FINAL_ROUNDS = 7 +_MIN_PROMOTION_SPEEDUP = 1.02 + + +def _key( + device, + dtype, + kv_dtype, + heads, + kv_heads, + dim, + page, + scale, + sliding_window=-1, + causal=True, + has_sinks=False, +): + return ( + torch.device(device).index, + dtype, + kv_dtype, + heads, + kv_heads, + dim, + page, + scale, + sliding_window, + causal, + has_sinks, + ) + + +def _normalized_config(config, batch, query_len, heads, dim): + qcap = segmented_query_capacity(query_len) + result = dict(config) + result.setdefault("waves_per_eu", 2) + result.setdefault("reduce_d", 64 if batch * qcap * heads < 64 else dim) + result.setdefault("reduce_warps", 4) + return result + + +def _candidate_configs(default, batch, query_len, heads, dim, seq_len=None): + """Search launch tiles and split counts within a bounded workspace.""" + base = _normalized_config(default, batch, query_len, heads, dim) + max_splits = base["splits"] + long_d64_decode = ( + dim == 64 + and batch == 1 + and query_len == 1 + and seq_len is not None + and seq_len >= 131072 + ) + if long_d64_decode: + max_splits = 512 + elif dim == 64 and query_len <= 512 and seq_len is not None and seq_len >= 4096: + max_splits = min(64, 2 * max_splits) + candidates: list[dict] = [] + seen = set() + + def add(**updates): + config = {**base, **updates} + if ( + config["splits"] > max_splits + or dim % config["bk"] + or config["reduce_d"] > dim + ): + return + identity = tuple(sorted(config.items())) + if identity not in seen: + seen.add(identity) + candidates.append(config) + + if long_d64_decode: + add() + for splits in (128, 256, 512, 64): + for bn, warps, stages in ( + (32, 1, 1), + (32, 2, 1), + (32, 2, 2), + (64, 1, 1), + (64, 2, 1), + (64, 2, 2), + ): + add( + bm=16, + bn=bn, + bk=64, + splits=splits, + warps=warps, + stages=stages, + ) + return candidates + + split_choices = [base["splits"]] + if max_splits > base["splits"]: + split_choices.append(max_splits) + for divisor in (2, 4): + split_choices.append(max(1, base["splits"] // divisor)) + split_choices.append(1) + for splits in split_choices: + add(splits=splits) + + tile_variants: tuple[tuple[int, int, int, int, int, int], ...] = ( + (16, 32, dim, 4, 1, 2), + (16, 64, min(128, dim), 4, 1, 2), + (32, 32, 64, 4, 2, 2), + (32, 64, 64, 4, 1, 2), + (64, 32, min(128, dim), 4, 1, 6), + (128, 32, min(128, dim), 8, 1, 6), + ) + if dim == 64: + tile_variants += ( + (16, 16, 64, 4, 1, 2), + (32, 16, 64, 4, 2, 2), + (64, 32, 64, 4, 1, 2), + (64, 64, 64, 4, 1, 2), + (128, 32, 64, 4, 1, 2), + (128, 64, 64, 4, 1, 2), + ) + for bm, bn, bk, warps, stages, waves in tile_variants: + add( + bm=bm, + bn=bn, + bk=bk, + warps=warps, + stages=stages, + waves_per_eu=waves, + ) + + half_splits = max(1, base["splits"] // 2) + for bm, bn, bk, warps, stages, waves in tile_variants[:3]: + add( + bm=bm, + bn=bn, + bk=bk, + splits=half_splits, + warps=warps, + stages=stages, + waves_per_eu=waves, + ) + + if dim == 64: + for splits in (max_splits, half_splits): + for bm, bn, bk, warps, stages, waves in tile_variants[2:]: + add( + bm=bm, + bn=bn, + bk=bk, + splits=splits, + warps=warps, + stages=stages, + waves_per_eu=waves, + ) + + add(waves_per_eu=6) + add(stages=2 if base["stages"] == 1 else 1) + add(reduce_d=64 if base["reduce_d"] != 64 else dim) + return candidates + + +def _identity( + device, + dtype, + kv_dtype, + heads, + kv_heads, + dim, + page, + scale, + max_tokens, + max_len, + max_seqs, + sliding_window, + causal, + max_query_len, + has_sinks=False, + physical_max_len=None, +): + properties = torch.cuda.get_device_properties(device) + from triton._C.libtriton import get_cache_invalidating_env_vars + + source = Path(__file__) + kernel = Path(segmented_attention.__file__) + return { + "schema": 5, + "gpu": properties.name, + "arch": properties.gcnArchName, + "compute_units": properties.multi_processor_count, + "torch": torch.__version__, + "hip": torch.version.hip, + "compiler_env": get_cache_invalidating_env_vars(), + "kernel_sha256": hashlib.sha256(kernel.read_bytes()).hexdigest(), + "tuner_sha256": hashlib.sha256(source.read_bytes()).hexdigest(), + "dtype": str(dtype), + "kv_dtype": str(kv_dtype), + "heads": heads, + "kv_heads": kv_heads, + "dim": dim, + "page": page, + "scale": scale, + "max_tokens": max_tokens, + "max_len": max_len, + "max_seqs": max_seqs, + "sliding_window": sliding_window, + "causal": causal, + "max_query_len": max_query_len, + "has_sinks": has_sinks, + "physical_max_len": physical_max_len, + "query_buckets": list(_QUERY_BUCKETS), + "batch_buckets": list(_BATCH_BUCKETS), + "sequence_buckets": list(_SEQUENCE_BUCKETS), + "compile_workers": _COMPILE_WORKERS, + "condition_rounds": _CONDITION_ROUNDS, + "screen_rounds": _SCREEN_ROUNDS, + "finalist_count": _FINALIST_COUNT, + "final_rounds": _FINAL_ROUNDS, + "min_promotion_speedup": _MIN_PROMOTION_SPEEDUP, + } + + +def _memory_budget(device): + free, total = torch.accelerator.get_memory_info(device) + available = int(min(free // 2, total // 4)) + return 1 << (available.bit_length() - 1) if available else 0 + + +def _query_lengths(batch, query_len, max_tokens): + if batch * query_len <= max_tokens: + return [query_len] * batch + return [min(query_len, max_tokens - batch + 1)] + [1] * (batch - 1) + + +def _scratch_bytes( + dtype, + kv_dtype, + heads, + kv_heads, + dim, + page, + max_tokens, + workload, + physical_max_len=None, +): + batch, query_len, seq_len = workload + queries = _query_lengths(batch, query_len, max_tokens) + tokens = sum(queries) + scratch_seq_len = ( + max(seq_len, physical_max_len) + if physical_max_len is not None and batch == 1 and query_len <= 8 + else seq_len + ) + blocks = batch * math.ceil(scratch_seq_len / page) + cache = 2 * blocks * page * kv_heads * dim * kv_dtype.itemsize + tensors = tokens * dim * (4 * heads + 2 * kv_heads) * dtype.itemsize + default = select_segmented_config( + batch, query_len, seq_len, heads, kv_heads, dim, kv_dtype.itemsize == 1 + ) + candidates = _candidate_configs( + default, batch, query_len, heads, dim, seq_len=seq_len + ) + shapes = segmented_workspace_shapes( + batch, + segmented_query_capacity(query_len), + heads, + kv_heads, + dim, + max(config["splits"] for config in candidates), + ) + workspace = 0 if shapes is None else sum(math.prod(shape) * 4 for shape in shapes) + return cache + tensors + workspace + 384 * 1024**2 + + +def _workloads( + max_tokens, + max_len, + max_seqs, + *, + memory_budget_bytes=None, + dtype=torch.bfloat16, + kv_dtype=torch.bfloat16, + heads=8, + kv_heads=1, + dim=128, + page=16, + cache_layouts=(), + cache_budget_bytes=None, + max_query_len=MAX_QUERY_LEN, + physical_max_len=None, +): + query_limit = min(max_tokens, max_len, max_query_len, MAX_QUERY_LEN) + queries = {q for q in _QUERY_BUCKETS if q <= query_limit} + if query_limit: + queries.add(query_limit) + for query_len in sorted(queries): + batch_limit = min(max_seqs, max_tokens - query_len + 1) + if batch_limit < 1: + continue + if query_len <= 2: + batches = {min(batch, batch_limit) for batch in _BATCH_BUCKETS} + else: + batches = {1, min(4, batch_limit), batch_limit} + if query_len <= 2: + sequences = { + seq for seq in _SEQUENCE_BUCKETS if query_len <= seq <= max_len + } + sequences.update((query_len, max_len)) + else: + sequences = { + query_len, + max(query_len, min(max_len, 2048)), + max(query_len, min(max_len, 8192)), + max_len, + } + # Keep long-context verification and prefill buckets even when + # the model's maximum length is pruned by the KV memory budget. + sequences.update( + seq for seq in (32768, 131072) if query_len <= seq <= max_len + ) + for batch, seq_len in itertools.product(sorted(batches), sorted(sequences)): + workload = (batch, query_len, seq_len) + if ( + memory_budget_bytes is not None + and _scratch_bytes( + dtype, + kv_dtype, + heads, + kv_heads, + dim, + page, + max_tokens, + workload, + physical_max_len, + ) + > memory_budget_bytes + ): + continue + if cache_budget_bytes is not None: + cache_bytes = batch * sum( + math.ceil(seq_len / block) * size if block else size + for block, size in cache_layouts + ) + if cache_bytes > cache_budget_bytes: + continue + yield workload + + +def _make_inputs( + device, + dtype, + kv_dtype, + heads, + kv_heads, + dim, + page, + max_tokens, + workload, + physical_seq_len=None, +): + batch, query_len, seq_len = workload + physical_seq_len = max(seq_len, physical_seq_len or seq_len) + generator = torch.Generator(device=device).manual_seed(1234) + + def randn(*shape): + return torch.randn(*shape, dtype=dtype, device=device, generator=generator) + + queries = _query_lengths(batch, query_len, max_tokens) + tokens = sum(queries) + q = randn(tokens, heads, dim) + k = randn(tokens, kv_heads, dim) + v = randn(tokens, kv_heads, dim) + blocks_per_seq = math.ceil(seq_len / page) + physical_blocks_per_seq = math.ceil(physical_seq_len / page) + cache_blocks_per_seq = ( + physical_blocks_per_seq if batch == 1 and query_len <= 8 else blocks_per_seq + ) + cache_shape = (batch * cache_blocks_per_seq, 2, page, kv_heads, dim) + scale_value = 0.125 if kv_dtype.itemsize == 1 else 1.0 + cache = (randn(*cache_shape).float() / scale_value).to(kv_dtype) + kc, vc = cache[:, 0], cache[:, 1] + table = ( + torch.arange(physical_blocks_per_seq, device=device, dtype=torch.int32)[None, :] + .remainder(cache_blocks_per_seq) + .expand(batch, -1) + .contiguous() + ) + table += ( + torch.arange(batch, device=device, dtype=torch.int32)[:, None] + * cache_blocks_per_seq + ) + starts = torch.tensor( + [0, *itertools.accumulate(queries)], device=device, dtype=torch.int32 + ) + lengths = torch.full((batch,), physical_seq_len, device=device, dtype=torch.int32) + scale = torch.full((), scale_value, device=device, dtype=torch.float32) + return q, k, v, kc, vc, table, starts, lengths, scale + + +def _config_key(config): + return tuple(sorted(config.items())) + + +def _rotated(configs, offset): + offset %= len(configs) + return configs[offset:] + configs[:offset] + + +def _measure_configs(configs, run, device, eviction, rounds, graph_calls=0): + """Interleave candidates so clock and thermal drift affect them evenly.""" + stream = torch.cuda.current_stream(device) + samples: dict[tuple, list[float]] = {_config_key(config): [] for config in configs} + graphs = {} + if graph_calls: + for config in configs: + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + for _ in range(graph_calls): + run(config) + graphs[_config_key(config)] = graph + for graph in graphs.values(): + graph.replay() + torch.accelerator.synchronize(device) + for round_index in range(rounds): + events = [] + order = _rotated(configs, round_index) + if round_index % 2: + order = list(reversed(order)) + for config in order: + if not graph_calls: + eviction.zero_() + start = torch.cuda.Event(enable_timing=True) + end = torch.cuda.Event(enable_timing=True) + start.record(stream) + if graph_calls: + graphs[_config_key(config)].replay() + else: + run(config) + end.record(stream) + events.append((_config_key(config), start, end)) + events[-1][2].synchronize() + for key, start, end in events: + samples[key].append(start.elapsed_time(end) * 1000 / max(1, graph_calls)) + return samples + + +def _condition_configs(configs, run, device): + for round_index in range(_CONDITION_ROUNDS): + for config in _rotated(configs, round_index): + run(config) + torch.accelerator.synchronize(device) + + +def _select_tuned_winner( + default, finalists, samples, min_promotion_speedup=_MIN_PROMOTION_SPEEDUP +): + """Return a finalist only when it reliably beats the static incumbent.""" + default_key = _config_key(default) + default_samples = samples[default_key] + comparisons = [] + for config in finalists: + key = _config_key(config) + config_samples = samples[key] + paired_speedup = statistics.median( + baseline / candidate + for baseline, candidate in zip(default_samples, config_samples, strict=True) + ) + comparisons.append( + { + "config": config, + "us": statistics.median(config_samples), + "paired_speedup_vs_default": paired_speedup, + } + ) + challenger = max( + comparisons, key=lambda result: result["paired_speedup_vs_default"] + ) + if challenger["paired_speedup_vs_default"] >= min_promotion_speedup: + return challenger, comparisons + return ( + next(result for result in comparisons if result["config"] == default), + comparisons, + ) + + +def _precompile_configs( + configs, + q, + output, + kc, + vc, + table, + starts, + lengths, + query_len, + kv_scale, + scale, + workspace, + sliding_window, + causal, + sinks=None, +): + workers = min(_COMPILE_WORKERS, len(configs)) + with ( + ThreadPoolExecutor(max_workers=workers) as executor, + triton.AsyncCompileMode(executor, ignore_errors=True), + ): + for config in configs: + compile_segmented_attention( + q, + output, + kc, + vc, + table, + starts, + lengths, + query_len, + kv_scale, + kv_scale, + scale, + config, + workspace, + sliding_window=sliding_window, + causal=causal, + sinks=sinks, + ) + + +def _tune_workload( + device, + dtype, + kv_dtype, + heads, + kv_heads, + dim, + page, + scale, + max_tokens, + workload, + *, + sliding_window=-1, + causal=True, + has_sinks=False, + physical_seq_len=None, +): + batch, query_len, seq_len = workload + q, _, _, kc, vc, table, starts, lengths, kv_scale = _make_inputs( + device, + dtype, + kv_dtype, + heads, + kv_heads, + dim, + page, + max_tokens, + workload, + physical_seq_len, + ) + default = select_segmented_config( + batch, query_len, seq_len, heads, kv_heads, dim, kv_dtype.itemsize == 1 + ) + configs = _candidate_configs(default, batch, query_len, heads, dim, seq_len=seq_len) + sinks = ( + torch.linspace(-1, 1, heads, dtype=torch.float32, device=device) + if has_sinks + else None + ) + qcap = segmented_query_capacity(query_len) + shapes = segmented_workspace_shapes( + batch, qcap, heads, kv_heads, dim, max(c["splits"] for c in configs) + ) + workspace = ( + None + if shapes is None + else tuple( + torch.empty(shape, dtype=torch.float32, device=device) for shape in shapes + ) + ) + reference = torch.empty_like(q) + output = torch.empty_like(q) + + def run(config, out=output): + run_segmented_attention( + q, + out, + kc, + vc, + table, + starts, + lengths, + query_len, + seq_len, + kv_scale, + kv_scale, + scale, + sliding_window=sliding_window, + causal=causal, + config=config, + workspace=workspace, + sinks=sinks, + ) + + _precompile_configs( + configs, + q, + output, + kc, + vc, + table, + starts, + lengths, + query_len, + kv_scale, + scale, + workspace, + sliding_window, + causal, + sinks, + ) + incumbent = configs[0] + reference.fill_(float("nan")) + run(incumbent, reference) + torch.accelerator.synchronize(device) + if not torch.isfinite(reference).all(): + raise AssertionError("Default segmented attention output is not finite") + reference_norm = reference.float().norm().clamp_min(1e-6) + eviction = torch.empty(256 * 1024**2, dtype=torch.int8, device=device) + results = [] + valid_configs = [] + for config in configs: + try: + output.fill_(float("nan")) + run(config) + torch.accelerator.synchronize(device) + torch.testing.assert_close(output, reference, atol=0.01, rtol=0.01) + relative_l2 = ( + (output.float() - reference.float()).norm() / reference_norm + ).item() + if relative_l2 > 0.005: + raise AssertionError(f"Relative L2 error {relative_l2} exceeds 0.5%") + results.append({"config": config, "relative_l2": relative_l2}) + valid_configs.append(config) + except ( + torch.OutOfMemoryError, + triton.OutOfResources, + triton.CompilationError, + AssertionError, + RuntimeError, + ) as error: + logger.warning("Rejected segmented attention config %s: %s", config, error) + if not results: + raise RuntimeError("No valid segmented attention launch configurations") + if incumbent not in valid_configs: + raise RuntimeError("Static segmented attention configuration was rejected") + + _condition_configs(valid_configs, run, device) + graph_calls = 5 if dim == 64 and has_sinks else 0 + screen_samples = _measure_configs( + valid_configs, run, device, eviction, _SCREEN_ROUNDS, graph_calls + ) + by_key = {_config_key(result["config"]): result for result in results} + for key, samples in screen_samples.items(): + by_key[key]["screen_us"] = statistics.median(samples) + by_key[key]["screen_samples_us"] = samples + + challengers = sorted( + (config for config in valid_configs if config != incumbent), + key=lambda config: statistics.median(screen_samples[_config_key(config)]), + ) + finalists = [incumbent, *challengers[: _FINALIST_COUNT - 1]] + _condition_configs(finalists, run, device) + final_samples = _measure_configs( + finalists, run, device, eviction, _FINAL_ROUNDS, graph_calls + ) + min_promotion_speedup = ( + 1.10 + if dim == 64 and has_sinks and sliding_window >= 0 + else _MIN_PROMOTION_SPEEDUP + ) + winner, comparisons = _select_tuned_winner( + incumbent, finalists, final_samples, min_promotion_speedup + ) + for comparison in comparisons: + result = by_key[_config_key(comparison["config"])] + result["final_us"] = comparison["us"] + result["final_samples_us"] = final_samples[_config_key(comparison["config"])] + result["paired_speedup_vs_default"] = comparison["paired_speedup_vs_default"] + winner_result = by_key[_config_key(winner["config"])] + logger.info( + "ROCm segmented attention autotune: B=%d Q=%d seq=%d best=%s " + "%.2f us speedup=%.3fx%s", + batch, + query_len, + seq_len, + winner["config"], + winner["us"], + winner["paired_speedup_vs_default"], + " (static retained)" if winner["config"] == incumbent else "", + ) + return { + "workload": list(workload), + "query_lengths": _query_lengths(batch, query_len, max_tokens), + "default": default, + "best": winner["config"], + "selection": { + "incumbent": incumbent, + "finalists": finalists, + "min_promotion_speedup": min_promotion_speedup, + "winner_us": winner_result["final_us"], + "paired_speedup_vs_default": winner["paired_speedup_vs_default"], + }, + "results": results, + } + + +def _save(path, data): + with tempfile.NamedTemporaryFile(mode="w", dir=path.parent, delete=False) as file: + temporary = Path(file.name) + try: + json.dump(data, file, indent=2) + file.flush() + os.fsync(file.fileno()) + except BaseException: + temporary.unlink(missing_ok=True) + raise + try: + os.replace(temporary, path) + finally: + temporary.unlink(missing_ok=True) + + +def _load_records(path, identity, heads, kv_heads, dim, fp8): + """Load a canonical table and any crash-recovery TP shards.""" + records: dict[tuple, dict] = {} + merged_shards = [] + sources = [path, *sorted(path.parent.glob(f"{path.name}.tp*.part"))] + for source in sources: + if not source.exists(): + continue + try: + saved = json.loads(source.read_text()) + if saved["identity"] != identity or not all( + _valid_record(record, heads, kv_heads, dim, fp8) + for record in saved["records"] + ): + raise ValueError("identity or record validation failed") + records.update( + (tuple(record["workload"]), record) for record in saved["records"] + ) + if source != path: + merged_shards.append(source) + except (ValueError, KeyError, TypeError, AssertionError): + logger.warning( + "Ignoring invalid segmented attention tuning cache %s", source + ) + return {"identity": identity, "records": list(records.values())}, merged_shards + + +def _tp_context(): + """Return the initialized TP coordinator, or local-only tuning state.""" + from vllm.distributed.parallel_state import ( + get_tp_group, + model_parallel_is_initialized, + ) + + if not model_parallel_is_initialized(): + return None, 0, 1 + group = get_tp_group() + return group, group.rank_in_group, group.world_size + + +def _gather_tp(group, value): + if group is None or group.world_size == 1: + return [value] + gathered = [None] * group.world_size + torch.distributed.all_gather_object(gathered, value, group=group.cpu_group) + return gathered + + +def _workload_weight(workload, max_tokens, heads, kv_heads, dim, fp8): + """Estimate compile plus execution cost for TP workload balancing.""" + batch, query_len, seq_len = workload + default = select_segmented_config( + batch, query_len, seq_len, heads, kv_heads, dim, fp8 + ) + candidates = len( + _candidate_configs(default, batch, query_len, heads, dim, seq_len=seq_len) + ) + query_tokens = sum(_query_lengths(batch, query_len, max_tokens)) + attention_work = query_tokens * seq_len * heads * dim + return candidates * (1 << 40) + attention_work + + +def _shard_workloads(workloads, world_size, max_tokens, heads, kv_heads, dim, fp8): + """Balance compile-affine ``(batch, query)`` groups across TP ranks.""" + assignments: list[list[tuple[int, tuple]]] = [[] for _ in range(world_size)] + loads = [0] * world_size + groups: dict[tuple, dict] = {} + for index, workload in enumerate(workloads): + compile_group = groups.setdefault(workload[:2], {"weight": 0, "items": []}) + compile_group["weight"] += _workload_weight( + workload, max_tokens, heads, kv_heads, dim, fp8 + ) + compile_group["items"].append((index, workload)) + for compile_group in sorted( + groups.values(), key=lambda item: item["weight"], reverse=True + ): + rank = min( + range(world_size), key=lambda candidate: (loads[candidate], candidate) + ) + assignments[rank].extend(compile_group["items"]) + loads[rank] += compile_group["weight"] + return [ + [workload for _, workload in sorted(assignment)] for assignment in assignments + ] + + +def _valid_record(record, heads, kv_heads, dim, fp8): + workload = record["workload"] + if len(workload) != 3 or any( + type(value) is not int or value < 1 for value in workload + ): + return False + batch, query_len, seq_len = workload + default = select_segmented_config( + batch, query_len, seq_len, heads, kv_heads, dim, fp8 + ) + return record["best"] in _candidate_configs( + default, batch, query_len, heads, dim, seq_len=seq_len + ) + + +@contextmanager +def _temporary_tuning_kernels(device): + """Release tuning-only HIP modules without invalidating existing graphs.""" + device_index = torch.device(device).index + if device_index is None: + device_index = torch.accelerator.current_device_index() + caches = [] + for kernel in (_segmented_attention_stage, _segmented_attention_reduce): + cache = kernel.device_caches[device_index][0] + caches.append((cache, set(cache))) + try: + yield + finally: + torch.accelerator.synchronize(device) + for cache, existing_keys in caches: + for key in cache.keys() - existing_keys: + del cache[key] + # CompiledKernel destruction unloads the HIP module. Keep pre-existing + # kernels alive, since an earlier graph capture may reference them. + gc.collect() + torch.accelerator.empty_cache() + + +@torch.inference_mode() +def warmup_segmented_attention( + device, + dtype, + heads, + kv_heads, + dim, + page, + scale, + max_tokens, + max_len, + max_seqs, + *, + memory_budget_bytes=None, + cache_layouts=(), + cache_budget_bytes=None, + kv_dtype=torch.bfloat16, + sliding_window=-1, + causal=True, + max_query_len=MAX_QUERY_LEN, + has_sinks=False, + physical_max_len=None, +): + """Tune reachable segmented buckets before graph capture.""" + if ( + dtype not in (torch.bfloat16, torch.float16) + or kv_dtype + not in ( + torch.bfloat16, + torch.float16, + torch.float8_e4m3fn, + torch.float8_e4m3fnuz, + ) + or dim not in (64, 128, 256) + or kv_heads < 1 + or heads % kv_heads + or not 1 <= heads // kv_heads <= 16 + ): + return + key = _key( + device, + dtype, + kv_dtype, + heads, + kv_heads, + dim, + page, + scale, + sliding_window, + causal, + has_sinks, + ) + identity = _identity( + device, + dtype, + kv_dtype, + heads, + kv_heads, + dim, + page, + scale, + max_tokens, + max_len, + max_seqs, + sliding_window, + causal, + max_query_len, + has_sinks, + physical_max_len, + ) + if memory_budget_bytes is None: + memory_budget_bytes = _memory_budget(device) + workload_args = dict( + memory_budget_bytes=memory_budget_bytes, + dtype=dtype, + kv_dtype=kv_dtype, + heads=heads, + kv_heads=kv_heads, + dim=dim, + page=page, + cache_layouts=cache_layouts, + cache_budget_bytes=cache_budget_bytes, + max_query_len=max_query_len, + physical_max_len=( + physical_max_len if has_sinks and sliding_window >= 0 else None + ), + ) + workloads = list(_workloads(max_tokens, max_len, max_seqs, **workload_args)) + active = _TABLES.get(key, {}) + warmed = ( + {tuple(record["workload"]) for record in active.get("records", [])} + if active.get("identity") == identity + else set() + ) + if all(workload in warmed for workload in workloads): + return + digest = hashlib.sha256(json.dumps(identity, sort_keys=True).encode()).hexdigest() + path = Path(envs.VLLM_CACHE_ROOT) / "rocm_segmented_attention" / f"{digest}.json" + path.parent.mkdir(parents=True, exist_ok=True) + group, tp_rank, tp_size = _tp_context() + if group is not None: + digests = _gather_tp(group, digest) + if any(peer != digest for peer in digests): + logger.warning( + "ROCm segmented attention TP ranks have different tuning " + "identities; tuning this rank independently" + ) + group, tp_rank, tp_size = None, 0, 1 + + unpruned = len( + list(_workloads(max_tokens, max_len, max_seqs, max_query_len=max_query_len)) + ) + start = time.monotonic() + lock = FileLock(str(path) + ".lock") if tp_rank == 0 else None + if lock is not None: + lock.acquire() + try: + if tp_rank == 0: + data, merged_shards = _load_records( + path, identity, heads, kv_heads, dim, kv_dtype.itemsize == 1 + ) + else: + data, merged_shards = None, [] + if group is not None: + data = group.broadcast_object(data, src=0) + assert data is not None + records = {tuple(record["workload"]): record for record in data["records"]} + missing = [workload for workload in workloads if workload not in records] + loaded = len(workloads) - len(missing) + assignments = _shard_workloads( + missing, + tp_size, + max_tokens, + heads, + kv_heads, + dim, + kv_dtype.itemsize == 1, + ) + local_workloads = assignments[tp_rank] + logger.info( + "ROCm segmented attention tuning plan: workloads=%d missing=%d " + "pruned=%d tp_rank=%d/%d assigned=%d scratch_budget=%d MiB " + "cache_budget=%s", + len(workloads), + len(missing), + unpruned - len(workloads), + tp_rank, + tp_size, + len(local_workloads), + memory_budget_bytes // 2**20, + cache_budget_bytes, + ) + + if not missing: + if tp_rank == 0 and merged_shards: + _save(path, data) + for shard in merged_shards: + shard.unlink(missing_ok=True) + _TABLES[key] = data + logger.info( + "ROCm segmented attention autotune ready: tp_rank=%d/%d " + "local_tuned=0 tuned=0 loaded=%d failed=0 elapsed=%.2fs cache=%s", + tp_rank, + tp_size, + loaded, + time.monotonic() - start, + path, + ) + return + + shard_path = path.with_suffix(path.suffix + f".tp{tp_rank}.part") + local_records = {} + local_failed = 0 + fatal = None + try: + with _temporary_tuning_kernels(device): + for workload in local_workloads: + try: + record = _tune_workload( + device, + dtype, + kv_dtype, + heads, + kv_heads, + dim, + page, + scale, + max_tokens, + workload, + sliding_window=sliding_window, + causal=causal, + has_sinks=has_sinks, + physical_seq_len=( + physical_max_len + if has_sinks and sliding_window >= 0 + else None + ), + ) + except ( + torch.OutOfMemoryError, + RuntimeError, + AssertionError, + ) as error: + local_failed += 1 + logger.warning( + "Skipping segmented attention tuning workload %s: %s", + workload, + error, + ) + else: + local_records[workload] = record + _save( + shard_path, + { + "identity": identity, + "records": list(local_records.values()), + }, + ) + finally: + gc.collect() + torch.accelerator.empty_cache() + except BaseException as error: + fatal = f"{type(error).__name__}: {error}" + + gathered = _gather_tp( + group, + { + "records": list(local_records.values()), + "failed": local_failed, + "fatal": fatal, + }, + ) + if tp_rank == 0: + fatal_errors = [] + for rank, contribution in enumerate(gathered): + records.update( + (tuple(record["workload"]), record) + for record in contribution["records"] + ) + if contribution["fatal"] is not None: + fatal_errors.append(f"TP rank {rank}: {contribution['fatal']}") + data = {"identity": identity, "records": list(records.values())} + _save(path, data) + for shard in path.parent.glob(f"{path.name}.tp*.part"): + shard.unlink(missing_ok=True) + result = { + "data": data, + "tuned": sum(len(item["records"]) for item in gathered), + "failed": sum(item["failed"] for item in gathered), + "fatal": fatal_errors, + } + else: + result = None + if group is not None: + result = group.broadcast_object(result, src=0) + assert result is not None + data = result["data"] + finally: + if lock is not None: + lock.release() + + _TABLES[key] = data + logger.info( + "ROCm segmented attention autotune ready: tp_rank=%d/%d " + "local_tuned=%d tuned=%d loaded=%d failed=%d elapsed=%.2fs cache=%s", + tp_rank, + tp_size, + len(local_records), + result["tuned"], + loaded, + result["failed"], + time.monotonic() - start, + path, + ) + if result["fatal"]: + raise RuntimeError( + "ROCm segmented attention distributed tuning failed: " + + "; ".join(result["fatal"]) + ) + + +def get_segmented_config( + device, + dtype, + kv_dtype, + heads, + kv_heads, + dim, + page, + scale, + batch, + query_len, + seq_len, + sliding_window=-1, + causal=True, + has_sinks=False, +): + """Return a warmed ceiling bucket, or None for the static fallback.""" + data = _TABLES.get( + _key( + device, + dtype, + kv_dtype, + heads, + kv_heads, + dim, + page, + scale, + sliding_window, + causal, + has_sinks, + ) + ) + if data is None: + return None + records = data["records"] + eligible = [ + record + for record in records + if record["workload"][0] >= batch + and record["workload"][1] >= query_len + and record["workload"][2] >= seq_len + ] + if not eligible: + return None + winner = min( + eligible, + key=lambda record: ( + record["workload"][1], + record["workload"][0], + record["workload"][2], + ), + ) + return dict(winner["best"]) + + +def _warmup_segmented_attention(layer, device, **limits) -> None: + impl = layer.impl + config = impl._segmented_attention_config + assert config is not None + spec = layer.get_kv_cache_spec(config) + assert spec is not None + sliding_window = impl.sliding_window[0] + query_limit = getattr(layer, "segmented_query_limit", None) + if not isinstance(query_limit, int): + query_limit = MAX_QUERY_LEN + causal = getattr(layer, "segmented_causal", True) + if not isinstance(causal, bool): + causal = True + tuning_max_len = config.model_config.max_model_len + if sliding_window >= 0: + tuning_max_len = min(tuning_max_len, sliding_window + query_limit) + warmup_segmented_attention( + device, + config.model_config.dtype, + impl.num_heads, + impl.num_kv_heads, + impl.head_size, + spec.block_size, + impl.scale, + config.scheduler_config.max_num_batched_tokens, + tuning_max_len, + config.scheduler_config.max_num_seqs, + kv_dtype=spec.dtype if spec.dtype != torch.uint8 else impl.fp8_dtype, + sliding_window=sliding_window, + causal=causal, + has_sinks=impl.sinks is not None, + physical_max_len=config.model_config.max_model_len, + max_query_len=query_limit, + **limits, + ) + impl._segmented_attention_warmed_up = True + + +def rocm_segmented_attn_autotune_warmup(worker: Worker) -> None: + """Autotune RDNA segmented attention after KV allocation and before capture.""" + if ( + not current_platform.is_rocm() + or not worker.vllm_config.kernel_config.enable_rocm_segmented_attn_autotune + ): + return + + from vllm.platforms.rocm import on_gfx1x + from vllm.v1.attention.backends.rocm_segmented_attn import ( + RocmSegmentedAttentionImpl, + ) + from vllm.v1.kv_cache_interface import FullAttentionSpec + + if not on_gfx1x(): + return + layers = [] + for layer in worker.vllm_config.compilation_config.static_forward_context.values(): + impl = getattr(layer, "impl", None) + if not isinstance(impl, RocmSegmentedAttentionImpl): + continue + config = impl._segmented_attention_config + if ( + config is not None + and config.kernel_config.enable_rocm_segmented_attn_autotune + and not impl._segmented_attention_warmed_up + and impl.alibi_slopes is None + and not impl.logits_soft_cap + ): + layers.append(layer) + if not layers: + return + + specs = worker.get_kv_cache_spec() + layouts = tuple( + ( + (spec.block_size, spec.page_size_bytes) + if isinstance(spec, FullAttentionSpec) + else (0, spec.max_memory_usage_bytes(worker.vllm_config)) + ) + for spec in specs.values() + ) + cache_budget = sum( + tensor.size + for tensor in worker.model_runner.kv_cache_config.kv_cache_tensors + if not tensor.host_resident + ) + budget = _memory_budget(worker.device) + for layer in layers: + _warmup_segmented_attention( + layer, + worker.device, + memory_budget_bytes=budget, + cache_layouts=layouts, + cache_budget_bytes=cache_budget, + ) diff --git a/vllm/models/deepseek_v32/amd/mtp.py b/vllm/models/deepseek_v32/amd/mtp.py index 70ddeae374..6230348ef4 100644 --- a/vllm/models/deepseek_v32/amd/mtp.py +++ b/vllm/models/deepseek_v32/amd/mtp.py @@ -251,6 +251,12 @@ class DeepseekV32MTP(nn.Module, DeepseekV2MixtureOfExperts, SupportsPP): name = name.replace(f"model.layers.{spec_layer}.", "model.") return name + def is_unused_checkpoint_weight(self, name: str) -> bool: + return ( + get_spec_layer_idx_from_weight_name(self.config, name) is None + and "embed_tokens" not in name + ) + def load_weights(self, weights: Iterable[tuple[str, torch.Tensor]]) -> set[str]: stacked_params_mapping = [ ("gate_up_proj", "gate_proj", 0), @@ -278,7 +284,7 @@ class DeepseekV32MTP(nn.Module, DeepseekV2MixtureOfExperts, SupportsPP): loaded_params: set[str] = set() _pending_wk_fp8: dict = {} for name, loaded_weight in weights: - if "rotary_emb.inv_freq" in name: + if "rotary_emb.inv_freq" in name or self.is_unused_checkpoint_weight(name): continue spec_layer = get_spec_layer_idx_from_weight_name(self.config, name) if spec_layer is None: diff --git a/vllm/models/deepseek_v32/nvidia/mtp.py b/vllm/models/deepseek_v32/nvidia/mtp.py index bdc6b95c15..d4dee7b9d4 100644 --- a/vllm/models/deepseek_v32/nvidia/mtp.py +++ b/vllm/models/deepseek_v32/nvidia/mtp.py @@ -356,6 +356,12 @@ class DeepseekV32MTP(nn.Module, DeepseekV2MixtureOfExperts, SupportsPP): name = name.replace(f"model.layers.{spec_layer}.", "model.") return name + def is_unused_checkpoint_weight(self, name: str) -> bool: + return ( + get_spec_layer_idx_from_weight_name(self.config, name) is None + and "embed_tokens" not in name + ) + def load_weights(self, weights: Iterable[tuple[str, torch.Tensor]]) -> set[str]: stacked_params_mapping = [ ("gate_up_proj", "gate_proj", 0), @@ -378,7 +384,7 @@ class DeepseekV32MTP(nn.Module, DeepseekV2MixtureOfExperts, SupportsPP): loaded_params: set[str] = set() _pending_wk_fp8: dict = {} for name, loaded_weight in weights: - if "rotary_emb.inv_freq" in name: + if "rotary_emb.inv_freq" in name or self.is_unused_checkpoint_weight(name): continue spec_layer = get_spec_layer_idx_from_weight_name(self.config, name) if spec_layer is None: diff --git a/vllm/models/deepseek_v4/amd/dspark.py b/vllm/models/deepseek_v4/amd/dspark.py index e0c69c0328..e62431d7b6 100644 --- a/vllm/models/deepseek_v4/amd/dspark.py +++ b/vllm/models/deepseek_v4/amd/dspark.py @@ -11,9 +11,8 @@ ROCm port of ``nvidia/dspark.py``. Follows the same nvidia->amd recipe used for tilelang / triton / torch) instead of calling the tilelang kernels directly, and gate the trailing ``mhc_post`` on ``use_fused_mhc`` (True when AITER or TileLang fused MHC is available; False only on the torch fallback); - * drop the mega-MoE weight path (``make_deepseek_v4_expert_params_mapping``, - ``use_native_mega_moe`` and ``finalize_mega_moe_weights`` do not exist in - amd/model.py). + * use the AITER MegaMoE path (``aiter_mega_moe``) instead of the DeepGEMM + mega-MoE path. Everything else — the semi-autoregressive drafting hooks, the Markov head, the sliding-window context-KV insert, and the checkpoint ``mtp.*`` weight remap — is @@ -50,6 +49,7 @@ from vllm.model_executor.models.qwen3_dspark import ( ) from vllm.model_executor.models.utils import maybe_prefix +from .mega_moe import finalize_mega_moe_weights from .model import ( DeepseekV4DecoderLayer, ) @@ -385,14 +385,15 @@ class DSparkDeepseekV4ForCausalLM(nn.Module): Non-mtp weights (embed/head/main layers) belong to the target model and are skipped here. ``embed_tokens``/``lm_head`` are aliased from the target. """ - # AMD DeepseekV4MoE has no mega-MoE path; always use the standard - # per-expert fused-MoE mapping (mirrors amd/mtp.py). expert_mapping = fused_moe_make_expert_params_mapping( self, ckpt_gate_proj_name="w1", ckpt_down_proj_name="w2", ckpt_up_proj_name="w3", num_experts=self.config.n_routed_experts, + routed_experts_prefix=( + "" if self.model.layers[0].ffn.use_mega_moe else "routed_experts" + ), ) expert_scale_suffix = ( ".weight_scale" @@ -496,6 +497,9 @@ class DSparkDeepseekV4ForCausalLM(nn.Module): logger.info_once("DSpark draft model loaded: %d params", len(loaded_params)) return loaded_params + def process_weights_after_loading(self) -> None: + finalize_mega_moe_weights(self) + def _remap_dspark_name(self, name: str) -> str | None: """Map a checkpoint ``mtp.{i}.*`` name to this model's parameter path. diff --git a/vllm/models/deepseek_v4/amd/mega_moe.py b/vllm/models/deepseek_v4/amd/mega_moe.py new file mode 100644 index 0000000000..2fdff93351 --- /dev/null +++ b/vllm/models/deepseek_v4/amd/mega_moe.py @@ -0,0 +1,154 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""AITER MegaMoE experts (fused dispatch, MXFP4 GEMMs, combine) for DeepSeek V4.""" + +from typing import Any + +import torch +import torch.nn as nn + +from vllm._aiter_ops import rocm_aiter_ops +from vllm.config import VllmConfig +from vllm.distributed import get_ep_group +from vllm.model_executor.utils import set_weight_attrs +from vllm.utils.math_utils import next_power_of_2 + + +class DeepseekV4AiterMegaMoEExperts(nn.Module): + # MegaMoEV2 owns large symmetric-heap buffers, so every layer shares one + # instance and passes its own weights per call. + _runtime_cache: dict[tuple, Any] = {} + + def __init__( + self, + vllm_config: VllmConfig, + *, + num_experts: int, + top_k: int, + hidden_size: int, + intermediate_size: int, + swiglu_limit: float | None, + prefix: str = "", + ): + super().__init__() + ep_group = get_ep_group() + self.ep_rank = ep_group.rank_in_group + self.ep_size = ep_group.world_size + self.num_experts = num_experts + self.num_local_experts = num_experts // self.ep_size + self.experts_start_idx = self.ep_rank * self.num_local_experts + self.top_k = top_k + self.hidden_size = hidden_size + self.intermediate_size = intermediate_size + self.swiglu_limit = swiglu_limit or 0.0 + self.max_num_tokens = next_power_of_2( + vllm_config.scheduler_config.max_num_batched_tokens + ) + self.prefix = prefix + + def make_param(rows: int, cols: int) -> nn.Parameter: + param = nn.Parameter( + torch.zeros(self.num_local_experts, rows, cols, dtype=torch.uint8), + requires_grad=False, + ) + set_weight_attrs(param, {"weight_loader": self.weight_loader}) + return param + + self.w13_weight = make_param(2 * intermediate_size, hidden_size // 2) + self.w13_weight_scale = make_param(2 * intermediate_size, hidden_size // 32) + self.w2_weight = make_param(hidden_size, intermediate_size // 2) + self.w2_weight_scale = make_param(hidden_size, intermediate_size // 32) + self._runtime: Any = None + + def weight_loader( + self, + param: nn.Parameter, + loaded_weight: torch.Tensor, + weight_name: str, + shard_id: str, + expert_id: int, + return_success: bool = False, + ) -> bool | None: + local_expert_id = expert_id - self.experts_start_idx + if not 0 <= local_expert_id < self.num_local_experts: + return False if return_success else None + expert_data = param.data[local_expert_id] + if shard_id in ("w1", "w3"): + shard_offset = 0 if shard_id == "w1" else self.intermediate_size + expert_data = expert_data.narrow(0, shard_offset, self.intermediate_size) + elif shard_id != "w2": + raise ValueError(f"Unsupported expert shard id: {shard_id}") + expert_data.copy_(loaded_weight.view(torch.uint8)) + return True if return_success else None + + def finalize_weights(self) -> None: + if self._runtime is not None: + return + n = self.num_local_experts + self.w13_weight.data = rocm_aiter_ops.shuffle_weight_a16w4( + self.w13_weight.data, 16, True + ) + self.w13_weight_scale.data = rocm_aiter_ops.shuffle_scale_a16w4( + self.w13_weight_scale.data.view(-1, self.hidden_size // 32), n, True + ) + self.w2_weight.data = rocm_aiter_ops.shuffle_weight_a16w4( + self.w2_weight.data, 16, False + ) + self.w2_weight_scale.data = rocm_aiter_ops.shuffle_scale_a16w4( + self.w2_weight_scale.data.view(-1, self.intermediate_size // 32), n, False + ) + key = ( + id(get_ep_group().cpu_group), + torch.accelerator.current_device_index(), + self.num_experts, + self.top_k, + self.hidden_size, + self.intermediate_size, + self.max_num_tokens, + self.swiglu_limit, + ) + runtime = self._runtime_cache.get(key) + if runtime is None: + from aiter.ops.flydsl.kernels.mega_moe import MegaMoEV2 + + runtime = MegaMoEV2( + rank=self.ep_rank, + world_size=self.ep_size, + model_dim=self.hidden_size, + inter_dim=self.intermediate_size, + experts=self.num_experts, + topk=self.top_k, + quant="a8w4", + w1=self.w13_weight, + w1_scale=self.w13_weight_scale, + w2=self.w2_weight, + w2_scale=self.w2_weight_scale, + max_tok_per_rank=self.max_num_tokens, + swiglu_limit=self.swiglu_limit, + ) + self._runtime_cache[key] = runtime + self._runtime = runtime + + def forward( + self, + hidden_states: torch.Tensor, + topk_weights: torch.Tensor, + topk_ids: torch.Tensor, + ) -> torch.Tensor: + runtime = self._runtime + assert runtime is not None + # MegaMoEV2 reads these per launch; rebind to this layer's experts. + runtime._s1_w1 = self.w13_weight + runtime._s1_w1_scale = self.w13_weight_scale + runtime.w2 = self.w2_weight + runtime.w2_scale = self.w2_weight_scale + return runtime.forward(hidden_states.contiguous(), topk_weights, topk_ids) + + +DeepseekV4AiterMegaMoEExperts.weight_loader.supports_moe_loading = True # type: ignore[attr-defined] + + +def finalize_mega_moe_weights(model: nn.Module) -> None: + for module in model.modules(): + if isinstance(module, DeepseekV4AiterMegaMoEExperts): + module.finalize_weights() diff --git a/vllm/models/deepseek_v4/amd/model.py b/vllm/models/deepseek_v4/amd/model.py index f8306dd12c..8b26fc0060 100644 --- a/vllm/models/deepseek_v4/amd/model.py +++ b/vllm/models/deepseek_v4/amd/model.py @@ -30,6 +30,9 @@ from vllm.model_executor.layers.fused_moe import ( from vllm.model_executor.layers.fused_moe.experts.rocm_aiter_moe import ( rocm_aiter_fused_experts, ) +from vllm.model_executor.layers.fused_moe.router.fused_topk_bias_router import ( + fused_topk_bias, +) from vllm.model_executor.layers.fused_moe.utils import ( is_model_fused_shared_expert_compatible, ) @@ -71,6 +74,7 @@ from vllm.model_executor.models.utils import ( make_layers, maybe_prefix, ) +from vllm.models.deepseek_v4.amd.mega_moe import DeepseekV4AiterMegaMoEExperts from vllm.models.deepseek_v4.amd.rocm import DeepseekV4ROCMAiterMLAAttention from vllm.platforms import current_platform from vllm.platforms.rocm import on_gfx950 @@ -529,6 +533,39 @@ def _fuse_shared_experts_enabled(config, parallel_config: ParallelConfig) -> boo ) +def _validate_aiter_mega_moe_config(vllm_config: VllmConfig) -> None: + parallel_config = vllm_config.parallel_config + if not rocm_aiter_ops.is_fused_moe_enabled(): + raise ValueError( + "--moe-backend aiter_mega_moe requires VLLM_ROCM_USE_AITER=1 " + " and VLLM_ROCM_USE_AITER_MOE=1." + ) + if not on_gfx950(): + raise NotImplementedError("AITER MegaMoE requires gfx950.") + if ( + not parallel_config.enable_expert_parallel + or parallel_config.tensor_parallel_size != 1 + or parallel_config.pipeline_parallel_size != 1 + ): + raise NotImplementedError( + "AITER MegaMoE requires data parallel attention with expert " + "parallel MoE: --data-parallel-size N --enable-expert-parallel " + "with TP=1 and PP=1." + ) + if parallel_config.all2all_backend not in ( + "mori_high_throughput", + "mori_low_latency", + ): + raise ValueError( + "AITER MegaMoE runs on MoRI shmem; add " + "--all2all-backend mori_high_throughput." + ) + if parallel_config.enable_eplb: + raise NotImplementedError("AITER MegaMoE does not support EPLB.") + if getattr(vllm_config.model_config.hf_config, "expert_dtype", "fp4") != "fp4": + raise NotImplementedError("AITER MegaMoE only supports fp4 experts.") + + class DeepseekV4MoE(nn.Module): def __init__( self, @@ -643,6 +680,20 @@ class DeepseekV4MoE(nn.Module): self.experts_start_idx = self.tp_rank * self.n_local_experts self.experts_end_idx = self.experts_start_idx + self.n_local_experts + self.use_mega_moe = vllm_config.kernel_config.moe_backend == "aiter_mega_moe" + if self.use_mega_moe: + _validate_aiter_mega_moe_config(vllm_config) + self.experts = DeepseekV4AiterMegaMoEExperts( + vllm_config, + num_experts=config.n_routed_experts, + top_k=config.num_experts_per_tok, + hidden_size=config.hidden_size, + intermediate_size=config.moe_intermediate_size, + swiglu_limit=self.swiglu_limit, + prefix=f"{prefix}.experts", + ) + return + fuse_shared_into_routed = ( self.is_fused_shared_expert_enabled or self.fuse_heterogeneous_shared_expert ) @@ -692,6 +743,8 @@ class DeepseekV4MoE(nn.Module): raise ValueError("DeepSeek V4 vision MoE routing requires input_ids.") org_shape = hidden_states.shape + if self.use_mega_moe: + return self._forward_mega_moe(hidden_states, input_ids).view(org_shape) final_hidden_states = self.experts( hidden_states=hidden_states, router_logits=hidden_states, @@ -700,6 +753,30 @@ class DeepseekV4MoE(nn.Module): return final_hidden_states.view(org_shape) + def _forward_mega_moe( + self, hidden_states: torch.Tensor, input_ids: torch.Tensor | None + ) -> torch.Tensor: + router_logits, _ = self.gate(hidden_states) + topk_weights, topk_ids = fused_topk_bias( + hidden_states=hidden_states, + gating_output=router_logits, + scoring_func=self.scoring_func, + e_score_correction_bias=self.gate.e_score_correction_bias, + topk=self.n_activated_experts, + renormalize=self.renormalize, + input_tokens=input_ids, + hash_indices_table=self.gate.tid2eid, + routed_scaling_factor=self.routed_scaling_factor, + bias_vl=self.gate.bias_vl, + image_sentinel_lo=self.image_sentinel_lo, + ) + final_hidden_states = self.experts(hidden_states, topk_weights, topk_ids) + if self.shared_experts is not None: + final_hidden_states = final_hidden_states + self.shared_experts( + hidden_states + ) + return final_hidden_states + # Hidden sizes supported by AITER mhc_pre_big_fuse_rmsnorm. _AITER_MHC_FUSED_RMSNORM_SIZES = frozenset({1280, 2560, 4096, 7168}) @@ -993,6 +1070,7 @@ class DeepseekV4Model(nn.Module, EagleModelMixin): self.fuse_heterogeneous_shared_expert = _heterogeneous_shared_expert_enabled( vllm_config ) + self.use_mega_moe = vllm_config.kernel_config.moe_backend == "aiter_mega_moe" self.start_layer, self.end_layer, self.layers = make_layers( config.num_hidden_layers, lambda prefix: DeepseekV4DecoderLayer( @@ -1315,6 +1393,7 @@ class DeepseekV4Model(nn.Module, EagleModelMixin): ckpt_down_proj_name="w2", ckpt_up_proj_name="w3", num_experts=num_experts, + routed_experts_prefix="" if self.use_mega_moe else "routed_experts", ) @@ -1453,6 +1532,8 @@ class DeepseekV4ForCausalLM(nn.Module, SupportsPP, SupportsEagle3): module.prepare_attn_preshuffle() elif isinstance(module, DeepseekV4MLP): module.prepare_gateup_preshuffle() + elif isinstance(module, DeepseekV4AiterMegaMoEExperts): + module.finalize_weights() if fused_compressor_layers: logger.info( "Fused the C4 compressor GEMMs in %d DeepSeek V4 layers", diff --git a/vllm/models/deepseek_v4/amd/mtp.py b/vllm/models/deepseek_v4/amd/mtp.py index 48ab036563..3746d6e387 100644 --- a/vllm/models/deepseek_v4/amd/mtp.py +++ b/vllm/models/deepseek_v4/amd/mtp.py @@ -46,6 +46,7 @@ from vllm.models.deepseek_v4.common.ops.fused_mtp_input_rmsnorm import ( from vllm.platforms import current_platform from vllm.sequence import IntermediateTensors +from .mega_moe import finalize_mega_moe_weights from .model import DeepseekV4DecoderLayer logger = init_logger(__name__) @@ -392,12 +393,16 @@ class DeepSeekV4MTP(nn.Module): head_rank_end = n_local_head * (tp_rank + 1) # Pre-compute expert mapping ONCE. + first_layer = next(iter(self.model.layers.values())) expert_mapping = fused_moe_make_expert_params_mapping( self, ckpt_gate_proj_name="w1", ckpt_down_proj_name="w2", ckpt_up_proj_name="w3", num_experts=self.config.n_routed_experts, + routed_experts_prefix=( + "" if first_layer.mtp_block.ffn.use_mega_moe else "routed_experts" + ), ) # FP8 experts register ``..._weight_scale_inv`` (block_quant) while @@ -526,6 +531,9 @@ class DeepSeekV4MTP(nn.Module): logger.info_once("MTP draft model loaded: %d params", len(loaded_params)) return loaded_params + def process_weights_after_loading(self) -> None: + finalize_mega_moe_weights(self) + def _rewrite_spec_layer_name(self, spec_layer: int, name: str) -> str: """Rewrite the weight name to match the format of the original model. Add .mtp_block for modules in transformer layer block for spec layer diff --git a/vllm/models/deepseek_v4/common/mm_preprocess.py b/vllm/models/deepseek_v4/common/mm_preprocess.py index 62a5c2070f..9c1a4664c8 100644 --- a/vllm/models/deepseek_v4/common/mm_preprocess.py +++ b/vllm/models/deepseek_v4/common/mm_preprocess.py @@ -22,13 +22,12 @@ reference's out-of-vocab scheme. import math from collections.abc import Mapping, Sequence -from typing import Any +from typing import Any, assert_never import numpy as np import torch from PIL import Image, ImageOps from transformers import BatchFeature -from typing_extensions import assert_never from vllm.config.multimodal import MultiModalDummyOptions from vllm.inputs import MultiModalDataDict diff --git a/vllm/models/deepseek_v4/common/ops/fused_indexer_q.py b/vllm/models/deepseek_v4/common/ops/fused_indexer_q.py index 669186edbb..54b1b9be7e 100644 --- a/vllm/models/deepseek_v4/common/ops/fused_indexer_q.py +++ b/vllm/models/deepseek_v4/common/ops/fused_indexer_q.py @@ -305,6 +305,7 @@ class FusedIndexerQRopeQuantTritonKernel( FP8_MAX=fp8_max, USE_FNUZ=use_fnuz, USE_EXPLICIT_FMA=current_platform.is_rocm(), + enable_fp_fusion=True, num_warps=1, ) @@ -540,6 +541,7 @@ class FusedIndexerQRopeMxFp4TritonKernel( MXFP4_BLOCK=MXFP4_BLOCK_SIZE, index_weights_stride=index_weights.stride(0), index_weights_out_stride=index_weights_out.stride(0), + enable_fp_fusion=True, num_warps=1, ) diff --git a/vllm/models/deepseek_v4/nvidia/model.py b/vllm/models/deepseek_v4/nvidia/model.py index a004120804..fc10cd07c1 100644 --- a/vllm/models/deepseek_v4/nvidia/model.py +++ b/vllm/models/deepseek_v4/nvidia/model.py @@ -2,7 +2,6 @@ # SPDX-FileCopyrightText: Copyright contributors to the vLLM project import typing from collections.abc import Callable, Iterable -from inspect import signature from itertools import islice import regex as re @@ -33,6 +32,11 @@ from vllm.model_executor.layers.fused_moe import ( FusedMoEFactory, fused_moe_make_expert_params_mapping, ) +from vllm.model_executor.layers.fused_moe.deep_gemm_mega_moe import ( + DeepGemmMegaMoEBackend, + get_deep_gemm_mega_moe_backend, + ue8m0_uint8_to_float, +) from vllm.model_executor.layers.fused_moe.router.base_router import ( eplb_map_to_physical_and_record, ) @@ -205,7 +209,9 @@ def make_deepseek_v4_expert_params_mapping( class DeepseekV4MegaMoEExperts(nn.Module): - _symm_buffer_cache: dict[tuple[int, int, int, int, int, int, int, int], object] = {} + _symm_buffer_cache: dict[ + tuple[int, int, int, int, int, int, int, int, str], object + ] = {} def __init__( self, @@ -296,6 +302,8 @@ class DeepseekV4MegaMoEExperts(nn.Module): tuple[torch.Tensor, torch.Tensor] | None ) = None + self._backend: DeepGemmMegaMoEBackend | None = None + # Register in the static forward context so the custom-op wrapper # can look up this module by name from within a torch.compile graph. compilation_config = vllm_config.compilation_config @@ -357,209 +365,71 @@ class DeepseekV4MegaMoEExperts(nn.Module): @staticmethod def _ue8m0_uint8_to_float(sf: torch.Tensor) -> torch.Tensor: - return (sf.to(torch.int32) << 23).view(torch.float32) + return ue8m0_uint8_to_float(sf) + + def _ensure_backend(self) -> DeepGemmMegaMoEBackend: + """Lazily resolve the MegaMoE backend. + + The backend is looked up via :func:`get_deep_gemm_mega_moe_backend`, which + performs all runtime capability checks. Failure propagates directly + to the caller. This must be called before :meth:`finalize_weights` + releases the loader-side parameters. + """ + if self._backend is not None: + return self._backend + assert self.w13_weight is not None, ( + "_ensure_backend() must be called before finalize_weights() " + "releases the loader-side parameters." + ) + self._backend = get_deep_gemm_mega_moe_backend( + self.w13_weight.device, + self.hidden_size, + self.intermediate_size, + ) + return self._backend def _check_runtime_supported(self) -> None: - device = self.w13_weight.device - if torch.cuda.get_device_capability(device)[0] != 10: - raise NotImplementedError("DeepGEMM MegaMoE requires SM100 GPUs.") - if self.hidden_size % 128 != 0 or self.intermediate_size % 128 != 0: - raise ValueError( - "DeepGEMM MegaMoE requires hidden and intermediate sizes " - "to be multiples of 128." - ) + """Preserved for API compatibility; delegates to the backend check.""" + self._ensure_backend() - @staticmethod - def _deep_gemm_supports_shared_experts(deep_gemm) -> bool: - """Check the Python API before touching a symmetric-memory group. + def _finalize_shared_expert_weights(self, shared_experts: DeepseekV4MLP) -> None: + """Delegate shared-expert weight transform to the backend and update state. - This also gives users of an older precompiled vLLM wheel a safe serial - fallback instead of failing halfway through multi-rank buffer setup. + On success, populates ``self._transformed_shared_l{1,2}_weights``. If + the backend cannot fuse shared experts (returns ``None``), disables + native fusion by setting ``self.num_shared_experts = 0``. """ - try: - buffer_params = signature(deep_gemm.get_symm_buffer_for_mega_moe).parameters - kernel_params = signature(deep_gemm.fp8_fp4_mega_moe).parameters - except (TypeError, ValueError): - return False - return ( - hasattr(deep_gemm, "get_block_m_for_mega_moe") - and hasattr(deep_gemm, "transform_weights_for_mega_moe") - and "num_shared_experts" in buffer_params - and "shared_l1_weights" in kernel_params - and "shared_l2_weights" in kernel_params + backend = self._ensure_backend() + result = backend.transform_shared_expert_weights( + shared_experts=shared_experts, + num_shared_experts=self.num_shared_experts, + hidden_size=self.hidden_size, + intermediate_size=self.intermediate_size, + prefix=self.prefix, ) - - def _finalize_shared_expert_weights( - self, deep_gemm, shared_experts: DeepseekV4MLP - ) -> None: - gate_up = shared_experts.gate_up_proj - down = shared_experts.down_proj - gate_up_weight = gate_up.weight.data - gate_up_scale = ( - gate_up.weight_scale - if hasattr(gate_up, "weight_scale") - else gate_up.weight_scale_inv - ).data - down_weight = down.weight.data - down_scale = ( - down.weight_scale - if hasattr(down, "weight_scale") - else down.weight_scale_inv - ).data - - # MegaMoE's shared FP8 MMA consumes a 1x32 scale for every weight row, - # while the checkpoint uses coarser block-FP8 scales (usually - # 128x128). Build a dedicated, numerically equivalent scale view before - # the generic linear post-load hook replaces the raw checkpoint scales - # with its 128x128 DeepGEMM layout. - checkpoint_scale_dtypes = (torch.float8_e8m0fnu, torch.uint8) - if ( - gate_up_scale.dtype in checkpoint_scale_dtypes - and down_scale.dtype in checkpoint_scale_dtypes - ): - gate_up_scale = self._prepare_shared_expert_scale( - deep_gemm, - gate_up, - gate_up_scale, - gate_up_weight.shape[0], - gate_up_weight.shape[1], - ) - down_scale = self._prepare_shared_expert_scale( - deep_gemm, - down, - down_scale, - down_weight.shape[0], - down_weight.shape[1], - ) - - if gate_up_scale is None or down_scale is None: + if result is None: self.num_shared_experts = 0 return - - shared_intermediate_size = self.intermediate_size * self.num_shared_experts - expected_gate_up_shape = ( - 2 * shared_intermediate_size, - self.hidden_size, - ) - expected_down_shape = (self.hidden_size, shared_intermediate_size) - if ( - gate_up_weight.dtype != torch.float8_e4m3fn - or down_weight.dtype != torch.float8_e4m3fn - or gate_up_scale.dtype != torch.int32 - or down_scale.dtype != torch.int32 - or tuple(gate_up_weight.shape) != expected_gate_up_shape - or tuple(down_weight.shape) != expected_down_shape - ): - logger.warning( - "Disabling native MegaMoE shared-expert fusion for %s: expected " - "replicated block-FP8 weights with gate_up=%s, down=%s, and " - "DeepGEMM int32 scales; got gate_up=%s/%s/%s and down=%s/%s/%s.", - self.prefix, - expected_gate_up_shape, - expected_down_shape, - tuple(gate_up_weight.shape), - gate_up_weight.dtype, - gate_up_scale.dtype, - tuple(down_weight.shape), - down_weight.dtype, - down_scale.dtype, - ) - self.num_shared_experts = 0 - return - - transformed_l1, transformed_l2 = deep_gemm.transform_weights_for_mega_moe( - (gate_up_weight, gate_up_scale), - (down_weight, down_scale), - ) - # L1 interleaving allocates a full copy. Re-home the loader Parameter on - # that storage so the original 2*intermediate*hidden FP8 tensor can be - # released instead of adding roughly 0.7 GiB per rank on DSV4-Flash. - # The generic linear post-load hook may still repack the serial scales, - # but this shared MLP is never called after native fusion is enabled. - gate_up.weight.data = transformed_l1[0] - self._transformed_shared_l1_weights = ( - gate_up.weight.data, - transformed_l1[1], - ) - self._transformed_shared_l2_weights = transformed_l2 - - def _prepare_shared_expert_scale( - self, - deep_gemm, - linear: nn.Module, - scale: torch.Tensor, - mn: int, - k: int, - ) -> torch.Tensor | None: - block_size = getattr(linear, "weight_block_size", None) - if block_size is None or len(block_size) != 2: - logger.warning( - "Disabling native MegaMoE shared-expert fusion for %s: " - "shared FP8 weight block size is unavailable.", - self.prefix, - ) - return None - - block_m, block_k = block_size - expected_shape = ( - (mn + block_m - 1) // block_m, - (k + block_k - 1) // block_k, - ) - if block_k % 32 != 0 or tuple(scale.shape) != expected_shape: - logger.warning( - "Disabling native MegaMoE shared-expert fusion for %s: " - "cannot convert shared scale shape %s with block size %s " - "to MegaMoE's 1x32 layout for weight (%d, %d).", - self.prefix, - tuple(scale.shape), - tuple(block_size), - mn, - k, - ) - return None - - scale_fp32 = self._ue8m0_uint8_to_float(scale.view(torch.uint8)) - scale_1x32 = ( - scale_fp32.repeat_interleave(block_m, dim=0) - .repeat_interleave(block_k // 32, dim=1)[:mn, : k // 32] - .contiguous() - ) - # The grouped API is used with a singleton dimension to request the - # MN-major, TMA-aligned packed-UE8M0 strides, then squeezed back to the - # 2D layout required for a shared expert. - return deep_gemm.transform_sf_into_required_layout( - scale_1x32.unsqueeze(0), - mn, - k, - (1, 32), - 1, - ).squeeze(0) + ( + self._transformed_shared_l1_weights, + self._transformed_shared_l2_weights, + ) = result def finalize_weights(self, shared_experts: DeepseekV4MLP | None = None) -> None: + backend = self._ensure_backend() from vllm.utils.deep_gemm import _import_deep_gemm deep_gemm = _import_deep_gemm() - if self._transformed_l1_weights is None: - self._check_runtime_supported() - w13_scale = deep_gemm.transform_sf_into_required_layout( - self._ue8m0_uint8_to_float(self.w13_weight_scale.data).contiguous(), - 2 * self.intermediate_size, - self.hidden_size, - (1, 32), - self.num_local_experts, - ) - w2_scale = deep_gemm.transform_sf_into_required_layout( - self._ue8m0_uint8_to_float(self.w2_weight_scale.data).contiguous(), - self.hidden_size, - self.intermediate_size, - (1, 32), - self.num_local_experts, - ) self._transformed_l1_weights, self._transformed_l2_weights = ( - deep_gemm.transform_weights_for_mega_moe( - (self.w13_weight.data.view(torch.int8).contiguous(), w13_scale), - (self.w2_weight.data.view(torch.int8).contiguous(), w2_scale), + backend.transform_weights( + w13_weight=self.w13_weight.data, + w13_weight_scale=self.w13_weight_scale.data, + w2_weight=self.w2_weight.data, + w2_weight_scale=self.w2_weight_scale.data, + num_local_experts=self.num_local_experts, + hidden_size=self.hidden_size, + intermediate_size=self.intermediate_size, ) ) # Drop the original loader-side parameters: the MegaMoE kernels only @@ -578,7 +448,7 @@ class DeepseekV4MegaMoEExperts(nn.Module): return if self._transformed_shared_l1_weights is not None: return - if not self._deep_gemm_supports_shared_experts(deep_gemm): + if not backend.supports_shared_experts(deep_gemm): logger.warning_once( "Disabling native MegaMoE shared-expert fusion because the " "installed DeepGEMM Python API is older than the vLLM " @@ -586,7 +456,8 @@ class DeepseekV4MegaMoEExperts(nn.Module): ) self.num_shared_experts = 0 return - self._finalize_shared_expert_weights(deep_gemm, shared_experts) + + self._finalize_shared_expert_weights(shared_experts) @property def has_fused_shared_experts(self) -> bool: @@ -597,6 +468,7 @@ class DeepseekV4MegaMoEExperts(nn.Module): deep_gemm = _import_deep_gemm() + backend = self._ensure_backend() group = get_ep_group().device_group device = torch.accelerator.current_device_index() key = ( @@ -608,6 +480,7 @@ class DeepseekV4MegaMoEExperts(nn.Module): self.hidden_size, self.intermediate_size, self.num_shared_experts if self.has_fused_shared_experts else 0, + backend.mma_type, ) symm_buffer = self._symm_buffer_cache.get(key) if symm_buffer is None: @@ -621,6 +494,7 @@ class DeepseekV4MegaMoEExperts(nn.Module): num_shared_experts=( self.num_shared_experts if self.has_fused_shared_experts else 0 ), + mma_type=backend.mma_type, ) self._symm_buffer_cache[key] = symm_buffer return symm_buffer @@ -695,7 +569,6 @@ class DeepseekV4MegaMoEExperts(nn.Module): deep_gemm = _import_deep_gemm() - symm_buffer = self.get_symm_buffer() num_tokens = hidden_states.shape[0] is_padding = None if envs.VLLM_MOE_SKIP_PADDING and is_forward_context_available(): @@ -727,6 +600,8 @@ class DeepseekV4MegaMoEExperts(nn.Module): else None, ) + backend = self._ensure_backend() + symm_buffer = self.get_symm_buffer() shared_x_sf = None shared_block_m = None if self.has_fused_shared_experts: @@ -737,7 +612,7 @@ class DeepseekV4MegaMoEExperts(nn.Module): symm_buffer.num_max_tokens_per_rank, num_tokens, self.top_k, - "fp8xfp4", + backend.mma_type, ) prepare_megamoe_inputs( @@ -751,27 +626,28 @@ class DeepseekV4MegaMoEExperts(nn.Module): is_padding=is_padding, shared_x_sf=shared_x_sf, shared_block_m=shared_block_m, + hidden_quant=backend.hidden_quant, ) assert self._transformed_l1_weights is not None assert self._transformed_l2_weights is not None if self.has_fused_shared_experts: - deep_gemm.fp8_fp4_mega_moe( - y, - self._transformed_l1_weights, - self._transformed_l2_weights, - symm_buffer, + backend.run_mega_moe( + y=y, + l1_weights=self._transformed_l1_weights, + l2_weights=self._transformed_l2_weights, + symm_buffer=symm_buffer, shared_l1_weights=self._transformed_shared_l1_weights, shared_l2_weights=self._transformed_shared_l2_weights, activation_clamp=activation_clamp, fast_math=fast_math, ) else: - deep_gemm.fp8_fp4_mega_moe( - y, - self._transformed_l1_weights, - self._transformed_l2_weights, - symm_buffer, + backend.run_mega_moe( + y=y, + l1_weights=self._transformed_l1_weights, + l2_weights=self._transformed_l2_weights, + symm_buffer=symm_buffer, activation_clamp=activation_clamp, fast_math=fast_math, ) @@ -1034,9 +910,18 @@ class DeepseekV4MoE(nn.Module): return self._forward_fused_moe(hidden_states, input_ids) org_shape = hidden_states.shape + # The MegaMoE experts module owns the backend; resolve and assert it. + self.experts._ensure_backend() + assert self.experts._backend is not None, ( + "MegaMoE backend must be resolved before routing." + ) + mega_backend = self.experts._backend # Small local padded batches favor GateLinear; 128-expert gates cross earlier. gate_threshold = 1 if self.gate.weight.shape[0] == 128 else 16 - if hidden_states.shape[0] <= gate_threshold: + if ( + not mega_backend.supports_bf16_mega_gate() + or hidden_states.shape[0] <= gate_threshold + ): router_logits, _ = self.gate(hidden_states) topk_weights, topk_ids = fused_topk_bias( hidden_states=hidden_states, @@ -1055,8 +940,6 @@ class DeepseekV4MoE(nn.Module): image_sentinel_lo=self.image_sentinel_lo if bias_vl is not None else 0, ) else: - from vllm.utils.deep_gemm import bf16_mega_gate - if mega_gate_metadata is None: raise RuntimeError( "MegaMoE routing metadata must be prepared once by the model." @@ -1074,10 +957,10 @@ class DeepseekV4MoE(nn.Module): ] fix_routing_mask = mega_gate_metadata.hash_token_mask - topk_weights, topk_ids = bf16_mega_gate( - hidden_states, - self.gate.weight, - self.n_activated_experts, + topk_weights, topk_ids = mega_backend.bf16_mega_gate( + x=hidden_states, + weight=self.gate.weight, + num_topk=self.n_activated_experts, scoring_func=self.scoring_func, routed_scaling_factor=self.routed_scaling_factor, ep_rank=self.ep_rank, diff --git a/vllm/models/deepseek_v4/nvidia/mtp.py b/vllm/models/deepseek_v4/nvidia/mtp.py index 64547af038..c0f3471dd5 100644 --- a/vllm/models/deepseek_v4/nvidia/mtp.py +++ b/vllm/models/deepseek_v4/nvidia/mtp.py @@ -340,6 +340,26 @@ class DeepSeekV4MTP(nn.Module): ) -> torch.Tensor | None: return self.model.compute_logits(hidden_states, spec_step_idx) + def _to_spec_layer_name(self, name: str) -> str: + """Remap V4 `mtp.{i}.*` names to `model.layers.{num_hidden_layers + i}.*` + so that get_spec_layer_idx_from_weight_name can identify them.""" + mtp_layer_idx = 0 + for subname in name.split("."): + try: + # we use the first encountered integer + mtp_layer_idx = int(subname) + break + except ValueError: + continue + return name.replace( + f"mtp.{mtp_layer_idx}.", + f"model.layers.{self.config.num_hidden_layers + mtp_layer_idx}.", + ) + + def is_unused_checkpoint_weight(self, name: str) -> bool: + name = self._to_spec_layer_name(name) + return get_spec_layer_idx_from_weight_name(self.config, name) is None + def load_weights(self, weights: Iterable[tuple[str, torch.Tensor]]) -> set[str]: # Weight name remapping for checkpoint compatibility. # Maps checkpoint weight paths to model parameter paths. @@ -356,16 +376,6 @@ class DeepSeekV4MTP(nn.Module): name = name.replace(old_pattern, new_pattern) return name - def _find_mtp_layer_idx(name: str) -> int: - subnames = name.split(".") - for subname in subnames: - try: - # we return the first encountered integer - return int(subname) - except ValueError: - continue - return 0 - stacked_params_mapping = [ # (param_name, shard_name, shard_id) ("gate_up_proj", "w1", 0), @@ -409,15 +419,7 @@ class DeepSeekV4MTP(nn.Module): ) for name, loaded_weight in weights: - mtp_layer_idx = _find_mtp_layer_idx(name) - # V4 checkpoints store MTP weights as `mtp.{i}.*`; remap to - # `model.layers.{num_hidden_layers + i}.*` so that - # get_spec_layer_idx_from_weight_name can identify them. - name = name.replace( - f"mtp.{mtp_layer_idx}.", - f"model.layers.{self.config.num_hidden_layers + mtp_layer_idx}.", - ) - + name = self._to_spec_layer_name(name) spec_layer = get_spec_layer_idx_from_weight_name(self.config, name) if spec_layer is None: continue diff --git a/vllm/models/deepseek_v4/nvidia/ops/prepare_megamoe.py b/vllm/models/deepseek_v4/nvidia/ops/prepare_megamoe.py index aa9482bed6..2d93b0d430 100644 --- a/vllm/models/deepseek_v4/nvidia/ops/prepare_megamoe.py +++ b/vllm/models/deepseek_v4/nvidia/ops/prepare_megamoe.py @@ -9,8 +9,15 @@ MegaMoE kernels consume. import torch +from vllm.model_executor.layers.quantization.utils.quant_utils import ( + FP8_DTYPE, + QuantKey, + kMxfp8Dynamic, +) from vllm.triton_utils import tl, triton +_E8M0_SCALE_DTYPES = (torch.uint8, torch.float8_e8m0fnu) + @triton.jit(do_not_specialize=["num_tokens"]) def _prepare_megamoe_inputs_kernel( @@ -48,6 +55,7 @@ def _prepare_megamoe_inputs_kernel( GROUP_K: tl.constexpr, BLOCK_TOPK: tl.constexpr, SHARED_BLOCK_M: tl.constexpr, + USE_UE8M0: tl.constexpr, ) -> None: token_id = tl.program_id(0) * BLOCK_M + tl.arange(0, BLOCK_M) token_mask = token_id < num_tokens @@ -69,15 +77,18 @@ def _prepare_megamoe_inputs_kernel( amax = tl.maximum(amax, 1.0e-4) scale = amax / 448.0 - scale_bits = scale.to(tl.uint32, bitcast=True) - scale_exp = ((scale_bits >> 23) & 0xFF) + ((scale_bits & 0x7FFFFF) != 0).to( - tl.uint32 - ) - scale_exp = tl.minimum(tl.maximum(scale_exp, 1), 254) - rounded_scale = (scale_exp << 23).to(tl.float32, bitcast=True) + if USE_UE8M0: + scale_bits = scale.to(tl.uint32, bitcast=True) + scale_exp = ((scale_bits >> 23) & 0xFF) + ((scale_bits & 0x7FFFFF) != 0).to( + tl.uint32 + ) + scale_exp = tl.minimum(tl.maximum(scale_exp, 1), 254) + quant_scale = (scale_exp << 23).to(tl.float32, bitcast=True) + else: + quant_scale = scale hidden_groups = tl.reshape(hidden, [BLOCK_M, num_groups, GROUP_K]) - scaled = hidden_groups * (1.0 / rounded_scale)[:, :, None] + scaled = hidden_groups * (1.0 / quant_scale)[:, :, None] scaled = tl.reshape(scaled, [BLOCK_M, BLOCK_K]) fp8 = scaled.to(tl.float8e4nv) tl.store( @@ -86,21 +97,32 @@ def _prepare_megamoe_inputs_kernel( mask=k_mask, ) - scale_offsets = tl.arange(0, num_groups) - packed_scale = tl.sum(scale_exp << (scale_offsets[None, :] * 8), axis=1).to( - tl.int32 - ) - tl.store( - x_sf + token_id * x_sf_stride_m + k_block_id * x_sf_stride_k, - packed_scale, - mask=token_mask, - ) + if USE_UE8M0: + scale_offsets = tl.arange(0, num_groups) + packed_scale = tl.sum(scale_exp << (scale_offsets[None, :] * 8), axis=1).to( + tl.int32 + ) + tl.store( + x_sf + token_id * x_sf_stride_m + k_block_id * x_sf_stride_k, + packed_scale, + mask=token_mask, + ) + else: + sf_offset = tl.arange(0, num_groups) + tl.store( + x_sf + + token_id[:, None] * x_sf_stride_m + + k_block_id * x_sf_stride_k + + sf_offset[None, :], + quant_scale, + mask=token_mask[:, None], + ) # DeepGEMM's SM100 shared-expert TMA loads require the activation scales # in an MN-major layout whose row permutation depends on the MegaMoE # scheduler's runtime BLOCK_M. Write that view while the packed UE8M0 scale # is already resident, avoiding another kernel and temporary tensor. - if shared_x_sf is not None: + if USE_UE8M0 and shared_x_sf is not None: m_block_id = token_id // SHARED_BLOCK_M m_in_block = token_id % SHARED_BLOCK_M aligned_block_m: tl.constexpr = triton.cdiv(SHARED_BLOCK_M, 128) * 128 @@ -160,6 +182,27 @@ def _prepare_megamoe_inputs_kernel( ) +def _resolve_hidden_quant(hidden_quant: QuantKey, block_k: int) -> tuple[int, bool]: + """Map a hidden-state ``QuantKey`` to the kernel's ``(GROUP_K, USE_UE8M0)``.""" + scale = hidden_quant.scale + group_shape = scale.group_shape + if ( + hidden_quant.dtype != FP8_DTYPE + or not hidden_quant.symmetric + or hidden_quant.scale2 is not None + or scale.static + or not group_shape.is_per_group() + or block_k % group_shape.col != 0 + or scale.dtype not in (*_E8M0_SCALE_DTYPES, torch.float32) + ): + raise ValueError( + "DeepSeek V4 MegaMoE input staging requires symmetric dynamic fp8 " + f"per-group quantization with a group size dividing {block_k} and " + f"E8M0 or fp32 scales, got {hidden_quant}." + ) + return group_shape.col, scale.dtype in _E8M0_SCALE_DTYPES + + def prepare_megamoe_inputs( hidden_states: torch.Tensor, topk_weights: torch.Tensor, @@ -171,7 +214,26 @@ def prepare_megamoe_inputs( is_padding: torch.Tensor | None = None, shared_x_sf: torch.Tensor | None = None, shared_block_m: int | None = None, + hidden_quant: QuantKey = kMxfp8Dynamic, ) -> None: + """Quantize hidden states and repack top-k routing for DeepGEMM MegaMoE. + + Args: + hidden_states: Input activations of shape ``[num_tokens, hidden_dim]``. + topk_weights: Router top-k weights of shape ``[num_tokens, top_k]``. + topk_ids: Router top-k expert ids of shape ``[num_tokens, top_k]``. + x_fp8: Output buffer for the fp8-quantized hidden states. + x_sf: Output buffer for the hidden-state scale factors. + topk_idx_out: Output buffer for the repacked top-k expert ids. + topk_weights_out: Output buffer for the repacked top-k weights. + is_padding: Optional per-token mask; padded tokens are not routed. + shared_x_sf: Optional output buffer for shared-expert scale factors. + Must be given together with ``shared_block_m``. + shared_block_m: Block M used to lay out ``shared_x_sf``. + hidden_quant: Hidden-state quantization scheme. E8M0 scales are packed + four per int32 into ``x_sf``; fp32 scales are stored directly. + + """ num_tokens, hidden_size = hidden_states.shape if num_tokens == 0: return @@ -180,6 +242,8 @@ def prepare_megamoe_inputs( "DeepSeek V4 MegaMoE input staging requires hidden_size to be " "a multiple of 128." ) + block_k = 128 + group_k, use_ue8m0 = _resolve_hidden_quant(hidden_quant, block_k) top_k = topk_ids.shape[1] if topk_weights.shape != topk_ids.shape: raise ValueError( @@ -191,6 +255,11 @@ def prepare_megamoe_inputs( "DeepSeek V4 MegaMoE shared input staging requires both " "shared_x_sf and shared_block_m." ) + if shared_x_sf is not None and not use_ue8m0: + raise ValueError( + "DeepSeek V4 MegaMoE shared input staging currently requires " + "UE8M0-packed hidden scales." + ) if shared_x_sf is not None: assert shared_block_m is not None if shared_block_m <= 0: @@ -209,7 +278,6 @@ def prepare_megamoe_inputs( f"{required_rows}, got {shared_x_sf.shape[0]}." ) - block_k = 128 # On GB200, eight-row tiles win from 64 tokens; keep smaller batches untiled. block_m = 8 if num_tokens >= 64 else 1 grid = (triton.cdiv(num_tokens, block_m), triton.cdiv(hidden_size, block_k)) @@ -247,8 +315,9 @@ def prepare_megamoe_inputs( top_k, BLOCK_M=block_m, BLOCK_K=block_k, - GROUP_K=32, + GROUP_K=group_k, BLOCK_TOPK=block_topk, SHARED_BLOCK_M=shared_block_m or 1, + USE_UE8M0=use_ue8m0, num_warps=4, ) diff --git a/vllm/models/deepseek_v4/quant_config.py b/vllm/models/deepseek_v4/quant_config.py index be8846fc79..f3aaf72d9b 100644 --- a/vllm/models/deepseek_v4/quant_config.py +++ b/vllm/models/deepseek_v4/quant_config.py @@ -110,6 +110,11 @@ class DeepseekV4FP8Config(Fp8Config): kv_cache_quant_algo=None, exclude_modules=[], group_size=16, + quantization_args=( + self.online_quantization_config.args + if self.online_quantization_config is not None + else None + ), ) return self._nvfp4_config diff --git a/vllm/models/deepseek_v41/amd/rocm.py b/vllm/models/deepseek_v41/amd/rocm.py index 99fcfa6702..3aa54f9b0e 100644 --- a/vllm/models/deepseek_v41/amd/rocm.py +++ b/vllm/models/deepseek_v41/amd/rocm.py @@ -26,7 +26,10 @@ from vllm.models.deepseek_v41.attention import ( DeepseekV4IndexerCache, _replace_layer_index, ) -from vllm.models.deepseek_v41.common.ops import dequantize_and_gather_k_cache +from vllm.models.deepseek_v41.common.ops import ( + combine_topk_swa_indices, + dequantize_and_gather_k_cache, +) from vllm.models.deepseek_v41.sparse_mla import ( DeepseekV4FlashMLAMetadata, DeepseekV4SparseMLABackend, @@ -129,165 +132,6 @@ def apply_pre_quantized_block_scaled_mm( return out.to(dtype=kernel.config.out_dtype) -# ROCm sparse prefill keeps this dense combine local so AMD-specific SWA changes -# do not touch the shared DeepSeek V4 cache utilities. -_SPARSE_PREFILL_TOPK_ALIGNMENT = 128 - - -@triton.jit -def _combine_topk_swa_indices_kernel( - combined_indices_ptr, - combined_indices_stride, - combined_lens_ptr, - topk_indices_ptr, - topk_indices_stride, - query_start_loc_ptr, - seq_lens_ptr, - gather_lens_ptr, - M, - N, - TOP_K: tl.constexpr, - COMPRESS_RATIO: tl.constexpr, - WINDOW_SIZE: tl.constexpr, - TOPK_WIDTH: tl.constexpr, - PADDED_TOP_K: tl.constexpr, -): - batch_idx = tl.program_id(0) - worker_id = tl.program_id(1) - num_workers = tl.num_programs(1) - - base = tl.load(query_start_loc_ptr) - query_start = tl.load(query_start_loc_ptr + batch_idx) - base - query_end = tl.load(query_start_loc_ptr + batch_idx + 1) - base - query_len = query_end - query_start - seq_len = tl.load(seq_lens_ptr + batch_idx) - gather_len = tl.load(gather_lens_ptr + batch_idx) - start_pos = seq_len - query_len - gather_start = seq_len - gather_len - - for token_idx in range(query_start + worker_id, query_end, num_workers): - token_idx_in_query = token_idx - query_start - pos = start_pos + token_idx_in_query - topk_len = tl.minimum((pos + 1) // COMPRESS_RATIO, TOP_K) - swa_len = tl.minimum(pos + 1, WINDOW_SIZE) - - topk_offset = tl.arange(0, PADDED_TOP_K) - topk_mask = topk_offset < topk_len - safe_topk_offset = tl.where(topk_offset < TOPK_WIDTH, topk_offset, 0) - topk_indices = tl.load( - topk_indices_ptr + token_idx * topk_indices_stride + safe_topk_offset, - mask=topk_mask, - other=-1, - ) - valid_topk = (topk_indices >= 0) & (topk_indices < N) - topk_indices = tl.where(valid_topk, topk_indices + M * batch_idx, -1) - tl.store( - combined_indices_ptr + token_idx * combined_indices_stride + topk_offset, - topk_indices, - mask=topk_mask, - ) - - swa_offset = tl.arange(0, WINDOW_SIZE) - tl.store( - combined_indices_ptr - + token_idx * combined_indices_stride - + topk_len - + swa_offset, - M * batch_idx + N + swa_offset + pos - swa_len + 1 - gather_start, - mask=swa_offset < swa_len, - ) - - tl.store(combined_lens_ptr + token_idx, topk_len + swa_len) - - -def combine_topk_swa_indices( - topk_indices: torch.Tensor, - query_start_loc: torch.Tensor, - seq_lens: torch.Tensor, - gather_lens: torch.Tensor, - window_size: int, - compress_ratio: int, - topk: int, - M: int, - N: int, -) -> tuple[torch.Tensor, torch.Tensor]: - """Combine compressed-attention and sliding-window indices with Torch. - - The Triton implementation inherited from DeepSeek V4 launches a - two-dimensional grid with 128 workers per request. On gfx950 it can issue - an out-of-bounds access for V4.1's mixed prefill metadata (including the - synthetic mixed-token warmup). This path is prefill-only and the tensors - are small, so use ordinary Torch indexing until a gfx950-safe fused kernel - is available. - """ - topk_indices = topk_indices.reshape(topk_indices.shape[0], -1).contiguous() - num_tokens = topk_indices.shape[0] - combined_topk = ( - (topk + window_size + _SPARSE_PREFILL_TOPK_ALIGNMENT - 1) - // _SPARSE_PREFILL_TOPK_ALIGNMENT - * _SPARSE_PREFILL_TOPK_ALIGNMENT - ) - combined_indices = torch.full( - (num_tokens, combined_topk), - fill_value=-1, - dtype=torch.int32, - device=topk_indices.device, - ) - combined_lens = torch.empty( - num_tokens, dtype=torch.int32, device=topk_indices.device - ) - - # query_start_loc may have a non-zero base for a narrowed mixed batch. - query_lens = query_start_loc[1:] - query_start_loc[:-1] - req_ids = torch.repeat_interleave( - torch.arange(seq_lens.shape[0], device=seq_lens.device), query_lens - ) - query_starts = query_start_loc[:-1] - query_start_loc[0] - token_offsets = torch.arange(num_tokens, device=seq_lens.device) - ( - torch.repeat_interleave(query_starts, query_lens) - ) - positions = seq_lens[req_ids] - query_lens[req_ids] + token_offsets - - logical_topk_width = min(topk, topk_indices.shape[1]) - topk_lens = torch.minimum( - (positions + 1) // compress_ratio, - torch.full_like(positions, logical_topk_width), - ).clamp_min(0) - topk_offsets = torch.arange(logical_topk_width, device=seq_lens.device) - topk_mask = topk_offsets[None, :] < topk_lens[:, None] - topk_values = topk_indices[:, :logical_topk_width].to(torch.int32) - topk_valid = topk_mask & (topk_values >= 0) & (topk_values < N) - combined_indices[:, :logical_topk_width] = torch.where( - topk_valid, - topk_values + (M * req_ids).to(torch.int32)[:, None], - -1, - ) - - swa_lens = torch.minimum( - positions + 1, torch.full_like(positions, window_size) - ).clamp_min(0) - swa_offsets = torch.arange(window_size, device=seq_lens.device) - swa_mask = swa_offsets[None, :] < swa_lens[:, None] - swa_columns = topk_lens[:, None] + swa_offsets[None, :] - gather_starts = seq_lens - gather_lens - swa_values = ( - M * req_ids[:, None] - + N - + swa_offsets[None, :] - + positions[:, None] - - swa_lens[:, None] - + 1 - - gather_starts[req_ids, None] - ).to(torch.int32) - rows = torch.arange(num_tokens, device=seq_lens.device)[:, None].expand_as( - swa_columns - ) - flat_dst = rows[swa_mask] * combined_topk + swa_columns[swa_mask] - combined_indices.view(-1).index_copy_(0, flat_dst, swa_values[swa_mask]) - combined_lens.copy_((topk_lens + swa_lens).to(torch.int32)) - return combined_indices, combined_lens - - @triton.jit def _compute_topk_lens_kernel( topk_lens_ptr, @@ -1387,6 +1231,9 @@ class DeepseekV41ROCMAiterMLAAttention(DeepseekV4Attention): gather_lens = swa_metadata.prefill_gather_lens assert seq_lens is not None assert gather_lens is not None + # CPU copy, so sizing the gather grid from it does not sync. + seq_lens_cpu = swa_metadata.prefill_seq_lens_cpu + assert seq_lens_cpu is not None query_start_loc_cpu = swa_metadata.query_start_loc_cpu query_start_loc = swa_metadata.query_start_loc @@ -1438,6 +1285,8 @@ class DeepseekV41ROCMAiterMLAAttention(DeepseekV4Attention): block_size=attn_metadata.block_size // self.compress_ratio, offset=0, use_fnuz=False, + max_gather_len=int(seq_lens_cpu[chunk_start:chunk_end].max()) + // self.compress_ratio, ) swa_block_table = swa_metadata.block_table[num_decodes:] diff --git a/vllm/models/deepseek_v41/attention.py b/vllm/models/deepseek_v41/attention.py index 73d9c0c723..559d874cc4 100644 --- a/vllm/models/deepseek_v41/attention.py +++ b/vllm/models/deepseek_v41/attention.py @@ -686,6 +686,34 @@ class DeepseekV4Attention(nn.Module, AttentionLayerBase, ABC): ) return self._o_proj(attn_out, positions) + def forward_kv(self, positions: torch.Tensor, hidden_states: torch.Tensor) -> None: + """Write only the KV ``forward`` writes (SWA KV, compressed KV, index K), + skipping wq_b, the indexer's top-k and attention.""" + attn_metadata = get_forward_context().attn_metadata + qr_kv, kv_score, _ = self._run_parallel_input_projections(hidden_states) + _, _, kv = self._split_qkv_and_norm(qr_kv) + if isinstance(attn_metadata, dict): + # The KV half of _fused_qnorm_rope_kv_insert, bit for bit, without + # the unused query; same as dspark's _insert_context_kv. + cache = self.swa_cache_layer.kv_cache + torch.ops._C.fused_deepseek_v4_kv_rope_insert( + kv, + cache, + cast(Any, attn_metadata[self.swa_cache_layer.prefix]).slot_mapping, + positions, + self.rotary_emb.cos_sin_cache, + self.swa_cache_layer.block_size, + self._flashinfer_fp8_kv_scale + if cache.dtype == torch.float8_e4m3fn + else None, + self.kv_mxfp8, + ) + if self.compressor is not None: + latent = self.compressor(kv_score, positions) + self.compressor.insert_cache(latent, positions, self.rotary_emb) + if self.indexer is not None and self.indexer.owns_k: + self.indexer._produce_k(latent, positions, self.indexer_rotary_emb) + def bind_gemm_rs(self) -> None: """Fuse ``wo_b`` with the sequence-parallel TP reduce-scatter. diff --git a/vllm/models/deepseek_v41/common/engram.py b/vllm/models/deepseek_v41/common/engram.py index d4810ee4ea..236235dfa8 100644 --- a/vllm/models/deepseek_v41/common/engram.py +++ b/vllm/models/deepseek_v41/common/engram.py @@ -62,9 +62,16 @@ from vllm.distributed.device_communicators.shm_broadcast import ( from vllm.distributed.parallel_state import GroupCoordinator from vllm.forward_context import get_forward_context, is_forward_context_available from vllm.logger import init_logger +from vllm.model_executor.layers.fusion.quant_activation import QuantizedActivation from vllm.model_executor.layers.linear import ColumnParallelLinear from vllm.model_executor.layers.quantization import QuantizationConfig +from vllm.model_executor.layers.quantization.utils.quant_utils import kMxfp8Dynamic from vllm.model_executor.utils import set_weight_attrs +from vllm.models.deepseek_v41.common.ops.query_quant import ( + can_fuse_query_quant, + mxfp8_scale_bytes, + mxfp8_scale_offset, +) from vllm.platforms import current_platform from vllm.triton_utils import tl, triton from vllm.utils.platform_utils import is_uva_available @@ -72,6 +79,10 @@ from vllm.utils.torch_utils import get_accelerator_view_from_cpu_tensor logger = init_logger(__name__) +# Per-rank token slot from which the Engram DP exchange switches from peer +# reads to all-to-all. +ENGRAM_A2A_MIN_SLOT = 128 + # Cache value for tokens that take no part in an n-gram (image spans). DEAD_ID = -1 @@ -644,8 +655,10 @@ def _engram_lookup_kernel( BLOCK_R: tl.constexpr, GRID, SORTED: tl.constexpr if _SORTED_IS_CONSTEXPR else None, # type: ignore[valid-type] + PACKED: tl.constexpr, + OUT_STRIDE: tl.constexpr, ): - """Gather fp8 rows, apply their ue8m0 block scales, write bf16. + """Gather FP8 rows as packed values/scales or dequantized BF16. Only this rank's heads are read; padded heads write zeros for all-gather. `weight`/`scales` may address pinned host memory through UVA. If SORTED, @@ -684,13 +697,28 @@ def _engram_lookup_kernel( mask=owned[:, None], other=0, ) - # ue8m0 is a power of two, so its byte *is* the fp32 exponent field. - scale = (scale.to(tl.int32) << 23).to(tl.float32, bitcast=True) - tl.store( - out + dst[:, None] * DIM + cols[None, :], - (values.to(tl.float32) * scale).to(tl.bfloat16), - mask=valid[:, None], - ) + if PACKED: + width: tl.constexpr = DIM + DIM // QUANT_BLOCK + dst = dst.to(tl.int64) + dst = dst // LOCAL_HEADS * OUT_STRIDE + dst % LOCAL_HEADS * width + tl.store( + out + dst[:, None] + cols[None, :], + values.to(tl.uint8, bitcast=True), + mask=valid[:, None], + ) + tl.store( + out + dst[:, None] + DIM + scale_cols[None, :], + scale, + mask=valid[:, None] & (cols[None, :] % QUANT_BLOCK == 0), + ) + else: + # ue8m0 is a power of two, so its byte *is* the fp32 exponent field. + scale = (scale.to(tl.int32) << 23).to(tl.float32, bitcast=True) + tl.store( + out + dst[:, None] * DIM + cols[None, :], + (values.to(tl.float32) * scale).to(tl.bfloat16), + mask=valid[:, None], + ) def engram_head_shard_rank() -> int: @@ -1082,7 +1110,7 @@ class ParallelEngramEmbedding(nn.Module): def lookup( self, indices: torch.Tensor, out: torch.Tensor, background: bool = False ) -> None: - """Look up local heads of [T, heads] into [T, local_heads, dim] bf16. + """Look up local heads into BF16, or packed FP8 bytes followed by scales. `background` limits the grid to leave SMs for concurrent work. """ @@ -1131,6 +1159,8 @@ class ParallelEngramEmbedding(nn.Module): BLOCK_R=16, GRID=grid, SORTED=sort, + PACKED=out.dtype == torch.uint8, + OUT_STRIDE=out.stride(0), ) def forward(self, indices: torch.Tensor) -> torch.Tensor: @@ -1275,7 +1305,7 @@ def _engram_select_rows( ) -> None: """Copy one token window out of a rank-major gathered buffer. - Both gathers land rank-major ([rank][token][local width]); this walks the + Both gathers land rank-major (`[rank][token][local width]`); this walks the window the rank keeps and lays its ranks out side by side as width. """ if output.numel() == 0: @@ -1311,6 +1341,175 @@ def _gather_engram_rows(staged: torch.Tensor, num_tokens: int) -> torch.Tensor: return rows +@triton.jit(do_not_specialize=["num_tokens"]) +def _engram_unpack_fp8_kernel( + packed, + values, + scales, + num_tokens, + values_stride, + scales_stride, + HEADS: tl.constexpr, + LOCAL_HEADS: tl.constexpr, + LAYERS: tl.constexpr, + RANK: tl.constexpr, + DIM: tl.constexpr, + BLOCK: tl.constexpr, + PEER: tl.constexpr, +): + """Split `[rank][token][layer][local head]` packed rows into MXFP8 values and + F8_128x4 scales, one layer per grid column; `packed` holds per-rank + pointers if PEER. Scales past `num_tokens` zero-pad the 128-row tile. + """ + row = tl.program_id(0).to(tl.int64) * BLOCK + tl.arange(0, BLOCK) + layer = tl.program_id(1).to(tl.int64) + token, head = row // HEADS, row % HEADS + if PEER: + src = tl.load(packed + head // LOCAL_HEADS) + src = tl.multiple_of(src, DIM // 32).to(tl.pointer_type(tl.uint8)) + source = RANK * num_tokens + token + else: + src = packed + source = head // LOCAL_HEADS * num_tokens + token + source = (source * LAYERS + layer) * LOCAL_HEADS + head % LOCAL_HEADS + # Packed rows are only scale-width aligned. + src += tl.multiple_of(source * (DIM + DIM // 32), DIM // 32) + valid = token < num_tokens + cols = tl.arange(0, DIM) + data = tl.load(src[:, None] + cols[None, :], mask=valid[:, None], other=0) + values += layer * values_stride + row * DIM + tl.store(values[:, None] + cols[None, :], data, mask=valid[:, None]) + scale_cols = tl.arange(0, DIM // 32) + scale = tl.load( + src[:, None] + DIM + scale_cols[None, :], mask=valid[:, None], other=0 + ) + group = head[:, None] * (DIM // 32) + scale_cols[None, :] + offset = mxfp8_scale_offset(token[:, None], group, HEADS * DIM // 32) + tl.store(scales + layer * scales_stride + offset, scale) + + +class EngramBatch(nn.Module): + """Stage every Engram layer's WKV input as MXFP8 with one exchange. + + A module so the model owns the peer exchange's pointer buffers. + """ + + def __init__(self, layers: list["Engram"], max_tokens: int) -> None: + super().__init__() + first = layers[0].layer_hash_index + hash_indices = [layer.layer_hash_index for layer in layers] + assert hash_indices == list(range(first, first + len(layers))) + self.layers = layers + table = layers[0].embed_tokens + device = layers[0].staged_rows.device + self.packed_rows = torch.empty( + max_tokens * table.dp_size, + len(layers), + table.part_n_hash_cols, + table.dim + table.dim // 32, + dtype=torch.uint8, + device=device, + ) + width = table.n_hash_cols * table.dim + self.fp8_values = torch.empty( + len(layers), max_tokens, width, dtype=torch.uint8, device=device + ) + self.fp8_scales = torch.empty( + len(layers), + mxfp8_scale_bytes(max_tokens, width), + dtype=torch.uint8, + device=device, + ) + max_peer_slot = min(max_tokens, ENGRAM_A2A_MIN_SLOT - 1) + self.peer_exchange = None + if table.cpu_offload: + from vllm.models.deepseek_v41.nvidia.ops.engram_peer import ( + EngramPeerExchange, + ) + + self.peer_exchange = EngramPeerExchange.create(layers, max_peer_slot) + for layer in layers: + # Owned by the model; keep it out of each layer's submodules. + object.__setattr__(layer, "_batch", self) + # The layers read this batch's staging instead of their BF16 rows. + layer.staged_rows = layer.staged_rows.new_empty( + (0, *layer.staged_rows.shape[1:]) + ) + + @classmethod + def create(cls, layers: list["Engram"], max_tokens: int) -> "EngramBatch | None": + """Batch DP-sharded tables whose WKV takes FlashInfer MXFP8 input.""" + if not layers: + return None + table = layers[0].embed_tokens + if ( + table.dp_size == 1 + or table.tp_size > 1 + or table.dim % 128 + or get_current_vllm_config().lora_config is not None + or not can_fuse_query_quant([layer.wkv for layer in layers]) + ): + return None + return cls(layers, max_tokens) + + def prepare_embeddings(self, hashes: torch.Tensor) -> None: + """Stage every layer's WKV input before the decoder layers. + + DP-sharded tables decide on the group's slot, so every rank agrees. + """ + slot = engram_gathered_num_tokens() + peer = self.peer_exchange is not None and slot < ENGRAM_A2A_MIN_SLOT + self._stage(hashes if peer else gather_engram_hashes(hashes), slot, peer) + + @eager_break_during_capture + def _stage(self, hashes: torch.Tensor, slot: int, peer: bool) -> None: + if slot == 0: + return + if peer: + assert self.peer_exchange is not None + rows = self.peer_exchange.exchange(hashes, slot) + rank = self.peer_exchange.dp_rank + else: + rows, rank = self.packed_rows[: hashes.shape[0]], 0 + for index, layer in enumerate(self.layers): + layer.embed_tokens.lookup( + hashes[:, layer.layer_hash_index], rows[:, index] + ) + group = get_engram_dp_group() + assert group is not None + sent, rows = rows, torch.empty_like(rows) + torch.distributed.all_to_all_single(rows, sent, group=group.device_group) + table = self.layers[0].embed_tokens + heads = table.n_hash_cols + grid = (triton.cdiv(slot, 128) * 128 * heads // 16, len(self.layers)) + _engram_unpack_fp8_kernel[grid]( + rows, + self.fp8_values, + self.fp8_scales, + slot, + self.fp8_values.stride(0), + self.fp8_scales.stride(0), + HEADS=heads, + LOCAL_HEADS=table.part_n_hash_cols, + LAYERS=len(self.layers), + RANK=rank, + DIM=table.dim, + BLOCK=16, + PEER=peer, + ) + + def wkv_input(self, layer: "Engram", num_tokens: int) -> QuantizedActivation: + index = layer.layer_hash_index - self.layers[0].layer_hash_index + values = self.fp8_values[index, :num_tokens].view(torch.float8_e4m3fn) + return QuantizedActivation( + values, + self.fp8_scales[index, : mxfp8_scale_bytes(num_tokens, values.shape[1])], + torch.bfloat16, + values.shape, + kMxfp8Dynamic, + ) + + class Engram(nn.Module): """Writes an n-gram lookup into the residual stream, gated by how well it matches that stream. @@ -1323,6 +1522,7 @@ class Engram(nn.Module): _prefetch_stream: torch.cuda.Stream | None = None _prefetch_done: torch.cuda.Event | None = None + _batch: EngramBatch | None = None def __init__( self, @@ -1482,7 +1682,10 @@ class Engram(nn.Module): """hidden_states: [T, hc_mult, dim]; hash_ids: [T, n_hash_cols] (all tokens, pre sequence-parallel shard); token_mask: [T], False shuts the gate so those positions pass through untouched.""" - kv = self.wkv(self.embed(hash_ids).flatten(-2)) + if self._batch is not None: + kv = self.wkv(self._batch.wkv_input(self, hash_ids.shape[0])) + else: + kv = self.wkv(self.embed(hash_ids).flatten(-2)) num_kv_tokens = hash_ids.shape[0] assert token_mask is None or token_mask.shape == (num_kv_tokens,) if self.use_sequence_parallel: diff --git a/vllm/models/deepseek_v41/common/ops/cache_utils.py b/vllm/models/deepseek_v41/common/ops/cache_utils.py index ad03c32945..a6139c71f8 100644 --- a/vllm/models/deepseek_v41/common/ops/cache_utils.py +++ b/vllm/models/deepseek_v41/common/ops/cache_utils.py @@ -33,7 +33,7 @@ from vllm.model_executor.warmup.jit_warmup_triton_helper import ( from vllm.platforms import current_platform from vllm.triton_utils import tl, triton from vllm.utils.import_utils import has_cutedsl -from vllm.utils.math_utils import next_power_of_2 +from vllm.utils.math_utils import cdiv, next_power_of_2 # Per-token byte width of the two paged fp8 records. Both put a page's whole # data region ahead of its whole scale region, so ``k_cache.shape[-1]`` -- the @@ -436,6 +436,95 @@ def _dequantize_and_gather_k_kernel( tl.store(output_row_ptr + bf16_output_offset + chunk_offsets, bf16_vals) +@triton.jit +def _dequantize_and_gather_k_one_pass_kernel( + out_ptr, + out_stride0, + out_stride1, + k_cache_ptr, + seq_lens_ptr, + block_table_ptr, + offset, + gather_lens_ptr, + # Constants + max_blocks_per_seq: tl.constexpr, + fp8_dim: tl.constexpr, # 448 + bf16_dim: tl.constexpr, # 64 + scale_dim: tl.constexpr, # 8 + quant_block: tl.constexpr, # 64 (quantization block size) + cache_block_size: tl.constexpr, # 64 or 128 (paged cache block size) + token_data_size: tl.constexpr, # 576 bytes per token data + block_stride: tl.constexpr, # total bytes per block (padded) int32 + output_dim: tl.constexpr, # 512 + fp8_max: tl.constexpr, + n_quant_blocks: tl.constexpr, # 7 real blocks; the per-block form only + use_fnuz: tl.constexpr = False, +): + """``_dequantize_and_gather_k_kernel`` with the token read in one tile. + + The seven quantization blocks tile the fp8 region exactly, so the per-block + form's bounds never bite and its twenty-two memory operations per token + reduce to four. Selected on ROCm; CUDA keeps the per-block kernel. + """ + batch_idx = tl.program_id(0) + worker_id = tl.program_id(1) + num_workers = tl.num_programs(1) + + seq_len = tl.load(seq_lens_ptr + batch_idx) + if gather_lens_ptr is not None: # noqa: SIM108 + gather_len = tl.load(gather_lens_ptr + batch_idx) + else: + gather_len = seq_len + start_pos = seq_len - gather_len + + for i in range(worker_id, gather_len, num_workers): + pos = start_pos + i + block_in_seq = pos // cache_block_size + pos_in_block = pos % cache_block_size + + block_table_row_ptr = block_table_ptr + batch_idx * max_blocks_per_seq + physical_block_idx = tl.load(block_table_row_ptr + block_in_seq) + cache_block_ptr = k_cache_ptr + physical_block_idx.to(tl.int64) * block_stride + + token_fp8_ptr = cache_block_ptr + pos_in_block * token_data_size + token_bf16_ptr = token_fp8_ptr + fp8_dim + token_scale_ptr = ( + cache_block_ptr + + cache_block_size * token_data_size + + pos_in_block * scale_dim + ) + output_row_ptr = out_ptr + batch_idx * out_stride0 + (offset + i) * out_stride1 + + # The tile is scale_dim * quant_block = 512 against an fp8 region of + # 448: the tail reads the token's own bf16 bytes, is scaled by a scale + # nothing reads, and is dropped by the store mask. + tile_dim: tl.constexpr = scale_dim * quant_block + tl.static_assert(tile_dim <= token_data_size, "tile outruns the token") + fp8_offsets = tl.arange(0, tile_dim) + x_uint8 = tl.load(token_fp8_ptr + fp8_offsets) + if use_fnuz: + x_fp8 = x_uint8.to(tl.float8e4b8, bitcast=True) + else: + x_fp8 = x_uint8.to(tl.float8e4nv, bitcast=True) + + encoded_scale = tl.load(token_scale_ptr + tl.arange(0, scale_dim)) + scale = tl.exp2(encoded_scale.to(tl.float32) - 127.0) # UE8M0 + tiles = tl.reshape(x_fp8.to(tl.float32), (scale_dim, quant_block)) + x_dequant = tl.reshape(tiles * tl.reshape(scale, (scale_dim, 1)), (tile_dim,)) + tl.store( + output_row_ptr + fp8_offsets, + x_dequant.to(tl.bfloat16), + mask=fp8_offsets < fp8_dim, + ) + + bf16_cache_ptr = token_bf16_ptr.to(tl.pointer_type(tl.bfloat16)) + bf16_offsets = tl.arange(0, bf16_dim) + tl.store( + output_row_ptr + fp8_dim + bf16_offsets, + tl.load(bf16_cache_ptr + bf16_offsets), + ) + + @triton.jit def _dequantize_and_gather_k_mxfp8_kernel( out_ptr, @@ -566,6 +655,22 @@ def _dequantize_and_gather_k_nvfp4_kernel( tl.store(output_row_ptr + d, dequant.to(tl.bfloat16)) +# The per-token loop is a chain of dependent loads, so the grid has to hold +# enough chains in flight to cover the latency. The cap binds above 64k tokens, +# where a worker takes more than the eight steps this aims for. +_GATHER_STEPS_PER_WORKER = 8 +_GATHER_MIN_WORKERS = 128 +_GATHER_MAX_WORKERS = 8192 + + +def gather_num_workers(max_gather_len: int | None) -> int: + """Grid width for a gather of at most ``max_gather_len`` tokens.""" + if max_gather_len is None: + return _GATHER_MIN_WORKERS + workers = cdiv(max_gather_len, _GATHER_STEPS_PER_WORKER) + return max(_GATHER_MIN_WORKERS, min(_GATHER_MAX_WORKERS, workers)) + + def dequantize_and_gather_k_cache_triton( # [num_reqs, max_num_tokens, head_size] out: torch.Tensor, @@ -580,6 +685,7 @@ def dequantize_and_gather_k_cache_triton( block_size: int, offset: int, use_fnuz: bool = False, + max_gather_len: int | None = None, ) -> None: num_reqs = seq_lens.shape[0] NUM_WORKERS = 128 @@ -633,7 +739,13 @@ def dequantize_and_gather_k_cache_triton( FP8_MAX = 448.0 TOKEN_DATA_SIZE = TOKEN_FP8_DIM + TOKEN_BF16_DIM * 2 - _dequantize_and_gather_k_kernel[(num_reqs, NUM_WORKERS)]( + gather_workers = gather_num_workers(max_gather_len) + kernel = ( + _dequantize_and_gather_k_one_pass_kernel + if current_platform.is_rocm() + else _dequantize_and_gather_k_kernel + ) + kernel[(num_reqs, gather_workers)]( out, out.stride(0), out.stride(1), @@ -671,6 +783,7 @@ def dequantize_and_gather_k_cache( block_size: int, offset: int, use_fnuz: bool = False, + max_gather_len: int | None = None, ) -> None: """Dequantize and gather a paged DSv4 K cache. @@ -711,6 +824,7 @@ def dequantize_and_gather_k_cache( block_table, block_size, offset, + max_gather_len=max_gather_len, use_fnuz=use_fnuz, ) @@ -975,7 +1089,8 @@ class CombineTopkSwaIndicesKernel( token_idx_in_query = token_idx - query_start pos = start_pos + token_idx_in_query if COMPRESS_RATIO > 0: - topk_len = tl.minimum((pos + 1) // COMPRESS_RATIO, TOP_K) + # Warmup batches can put pos below zero; keep the length >= 0. + topk_len = tl.maximum(tl.minimum((pos + 1) // COMPRESS_RATIO, TOP_K), 0) else: # SWA-only layers pass TOP_K=0; skip the division entirely # (integer div by constexpr 0 is UB and yields garbage, which @@ -986,7 +1101,7 @@ class CombineTopkSwaIndicesKernel( # start (SWA bounded replay), so the window cannot start before # the gathered buffer does. swa_start = tl.maximum(swa_start, gather_start) - swa_len = pos - swa_start + 1 + swa_len = tl.maximum(pos - swa_start + 1, 0) offset = tl.arange(0, PADDED_TOP_K) mask = offset < topk_len @@ -994,9 +1109,12 @@ class CombineTopkSwaIndicesKernel( topk_indices_ptr + token_idx * topk_indices_stride + offset, mask=mask, ) + # A failed (-1) or out-of-pool candidate stays -1 instead of landing + # in the previous request's rows or this request's SWA rows. + valid = (topk_indices >= 0) & (topk_indices < N) tl.store( combined_indices_ptr + token_idx * combined_indices_stride + offset, - topk_indices + M * batch_idx, + tl.where(valid, topk_indices + M * batch_idx, -1), mask=mask, ) # Index into gathered buffer: N + (position - gather_start) diff --git a/vllm/models/deepseek_v41/common/ops/fused_layout.py b/vllm/models/deepseek_v41/common/ops/fused_layout.py index 071afde6bc..50df8de351 100644 --- a/vllm/models/deepseek_v41/common/ops/fused_layout.py +++ b/vllm/models/deepseek_v41/common/ops/fused_layout.py @@ -12,6 +12,8 @@ per-step shuffle. import torch +from vllm.model_executor.model_loader.weight_utils import composed_weight_loader + HEAD_DIM = 512 Q_CHUNK = 16 O_CHUNK = 32 @@ -60,27 +62,13 @@ def _bytes_view(t: torch.Tensor) -> torch.Tensor: return t.view(torch.uint8) if t.element_size() == 1 else t -def permute_wq_b_( - weight: torch.Tensor, weight_scale: torch.Tensor, num_local_heads: int -) -> None: - """Permute an MXFP8 ``wq_b`` shard's rows and per-row scales, in place.""" - head_dim = weight.shape[0] // num_local_heads - perm = q_fused_permutation(num_local_heads, head_dim).to(weight.device) - for t in (weight, weight_scale): - b = _bytes_view(t) - b.copy_(b[perm]) +def permute_on_load(param: torch.Tensor, perm: torch.Tensor, dim: int) -> None: + """Make ``param``'s weight loader gather the loaded shard by ``perm``.""" + def gather(t: torch.Tensor) -> torch.Tensor: + return _bytes_view(t).index_select(dim, perm.to(t.device)).view(t.dtype) -def permute_wo_a_( - weight: torch.Tensor, - weight_scale: torch.Tensor, - heads_per_group: int = WV_GROUP_SIZE, -) -> None: - """Permute an MXFP8 ``wo_a`` shard's input columns and scales, in place.""" - head_dim = weight.shape[1] // heads_per_group - perm = o_fused_permutation(heads_per_group, head_dim).to(weight.device) - w = _bytes_view(weight) - w.copy_(w[:, perm]) - chunk_perm = o_fused_chunk_permutation(heads_per_group, head_dim) - s = _bytes_view(weight_scale) - s.copy_(s[:, chunk_perm.to(weight.device)]) + param.weight_loader = composed_weight_loader( # type: ignore[attr-defined] + param.weight_loader, # type: ignore[attr-defined] + gather, + ) diff --git a/vllm/models/deepseek_v41/common/ops/query_quant.py b/vllm/models/deepseek_v41/common/ops/query_quant.py index 6094d1980c..97714e29f1 100644 --- a/vllm/models/deepseek_v41/common/ops/query_quant.py +++ b/vllm/models/deepseek_v41/common/ops/query_quant.py @@ -21,6 +21,18 @@ def _norm_row(x, weight, row, stride, eps, SIZE: tl.constexpr, BLOCK: tl.constex return values * rrms * w +@triton.jit +def mxfp8_scale_offset(row, group, GROUPS: tl.constexpr): + """F8_128x4: [row/128, group/4, row%32, row%128/32, group%4].""" + offset = (row // 128 * triton.cdiv(GROUPS, 4) + group // 4) * 512 + return offset + row % 32 * 16 + row % 128 // 32 * 4 + group % 4 + + +def mxfp8_scale_bytes(tokens: int, width: int) -> int: + """Bytes of F8_128x4 scales for `tokens` rows of `width` MXFP8 values.""" + return triton.cdiv(tokens, 128) * 128 * triton.cdiv(width // 32, 4) * 4 + + @triton.jit(do_not_specialize=["num_tokens"]) def _q_kv_norm_quant_kernel( q, @@ -68,14 +80,7 @@ def _q_kv_norm_quant_kernel( sf = tl.full((BLOCK // 32,), 0, tl.uint32) padded_groups: tl.constexpr = triton.cdiv(Q_SIZE // 32, 4) * 4 sf = tl.where(groups < Q_SIZE // 32, sf, 0) - # F8_128x4: [row/128, group/4, row%32, row%128/32, group%4]. - offsets = ( - row // 128 * (128 * padded_groups) - + groups // 4 * 512 - + row % 32 * 16 - + row % 128 // 32 * 4 - + groups % 4 - ) + offsets = mxfp8_scale_offset(row, groups, Q_SIZE // 32) tl.store(scales + offsets, sf, groups < padded_groups) elif row < num_tokens: y = _norm_row(kv, kvw, row, kv_stride, eps, KV_SIZE, BLOCK) @@ -99,9 +104,8 @@ def fused_q_kv_rmsnorm_quant( qo = torch.empty(qr.shape, dtype=torch.float8_e4m3fn, device=qr.device) kvo = torch.empty(kv.shape, dtype=kv.dtype, device=kv.device) padded_tokens = triton.cdiv(tokens, 128) * 128 - padded_groups = triton.cdiv(q_size // 32, 4) * 4 scales = torch.empty( - padded_tokens * padded_groups, dtype=torch.uint8, device=qr.device + mxfp8_scale_bytes(tokens, q_size), dtype=torch.uint8, device=qr.device ) if tokens: block = triton.next_power_of_2(max(q_size, kv_size)) diff --git a/vllm/models/deepseek_v41/common/vl_cudagraph.py b/vllm/models/deepseek_v41/common/vl_cudagraph.py index bd8ed5a3fd..19baf339eb 100644 --- a/vllm/models/deepseek_v41/common/vl_cudagraph.py +++ b/vllm/models/deepseek_v41/common/vl_cudagraph.py @@ -94,8 +94,6 @@ class DeepseekV4VLEncoderCudaGraphMixin: grid = grid.tolist() return [[int(v) for v in row] for row in grid] - _encoder_cg_pad_totals: dict[int, int] = {} - # -- SupportsEncoderCudaGraph protocol -- def get_encoder_cudagraph_config(self) -> EncoderCudaGraphConfig: @@ -129,24 +127,13 @@ class DeepseekV4VLEncoderCudaGraphMixin: "cudagraph_mm_encoder." ) - pad_totals = self._encoder_cg_pad_totals - def pad_cu_seqlens(dst: torch.Tensor, src: torch.Tensor) -> None: - # Varlen attention requires cu_seqlens[-1] to equal the number of - # rows actually passed in. The captured buffers are sized for the - # full token budget, so a smaller real batch is completed with one - # trailing padding sequence; declaring fewer rows than the buffer - # holds is undefined behaviour and returns NaN on FlashAttn. - total = pad_totals.get(dst.data_ptr()) - if total is None: - raise RuntimeError( - "cu_seqlens replay buffer was not registered at capture " - "time; the manager must replay into the capture-time " - "buffers." - ) + # Zero-length padding (same as Qwen): tail slots repeat the last + # real offset, so FlashAttn schedules no work for padding rows. n = min(src.shape[0], dst.shape[0]) dst[:n].copy_(src[:n]) - dst[n:] = total + if n < dst.shape[0]: + dst[n:] = src[-1] if n else 0 return EncoderCudaGraphConfig( modalities=["image"], @@ -268,8 +255,8 @@ class DeepseekV4VLEncoderCudaGraphMixin: max_seqlen_override=total_patches, cached=False, ) - # Spare cu_seqlens slots let replay append a padding sequence that - # covers the rows a smaller real batch does not fill. + # Spare cu_seqlens slots: replay pads smaller real batches with + # zero-length sequences (tail slots repeat the last real offset). real_cu = metadata.pop("cu_seqlens") cu_seqlens = torch.full( (max_batch_size + 2,), total_patches, dtype=torch.int32, device=device @@ -278,8 +265,6 @@ class DeepseekV4VLEncoderCudaGraphMixin: merge = build_packed_merge_metadata(grids, r, device=device, dtype=dtype) - self._encoder_cg_pad_totals[cu_seqlens.data_ptr()] = total_patches - return EncoderCudaGraphCaptureInputs( values={ "patches": patches, @@ -303,7 +288,7 @@ class DeepseekV4VLEncoderCudaGraphMixin: patches = patches.to(dtype) # Unpadded: the manager zero-pads patches/cos/sin/merge buffers, and - # pad_cu_seqlens appends the padding sequence covering the tail rows. + # pad_cu_seqlens pads the tail with zero-length sequences. metadata = build_packed_vit_metadata( vit_grid, rope_dim=self.vision.rope_dim, diff --git a/vllm/models/deepseek_v41/decoder_replay_layers.py b/vllm/models/deepseek_v41/decoder_replay_layers.py index 869029aa74..28aa5e399f 100644 --- a/vllm/models/deepseek_v41/decoder_replay_layers.py +++ b/vllm/models/deepseek_v41/decoder_replay_layers.py @@ -2,63 +2,126 @@ # SPDX-FileCopyrightText: Copyright contributors to the vLLM project """Decoder-side SWA bounded replay: running the replay layers on their batch. -The layers past the last KV-source layer own nothing but sliding-window KV, so -in eager prefill steps they run on each request's last ``window`` rows only. +Once the last KV-source layer has written every row's KV, it and the layers past +it (which own only SWA KV) run on each request's last ``window`` rows only. ``DeepseekV41ModelState`` prepares those rows as a sub-batch with attention metadata and a forward context of its own, like a microbatch; ``DecoderReplayLayers`` gathers the layer inputs by its rows, runs the layers -under that context and scatters the outputs back to batch rows. Steps that run -in a CUDA graph keep the layers on the whole batch, inside the graph. +under that context and scatters the outputs back to batch rows. FULL graphs and +small PIECEWISE ones keep the layers on the whole batch; larger PIECEWISE graphs +break out to the replay batch, which may run in a graph of its own +(decoder_replay_cudagraph.py). """ from collections.abc import Callable +from dataclasses import dataclass import torch -from vllm.forward_context import ForwardContext, override_forward_context +from vllm.compilation.breakable_cudagraph import eager_break_during_capture +from vllm.forward_context import ( + ForwardContext, + get_forward_context, + in_piecewise_cudagraph, + override_forward_context, +) from vllm.model_executor.layers.fused_moe.moe_output import MoEOutput +@dataclass +class ReplayBatch: + """One forward's replay batch; ``run_graph`` is set when it runs in a graph.""" + + rows: torch.Tensor + forward_context: ForwardContext + run_graph: Callable[..., tuple[torch.Tensor, ...]] | None = None + + class DecoderReplayLayers: """Runs the replay layers on the step's replay batch. - ``run_layers`` takes a batch's layer inputs and returns its per-row - outputs. ``row_buffers`` hold per-row results the source layer's indexer - publishes for the layers after it; they are compacted to the replay rows - in place. + ``run_layers`` takes a batch's layer inputs and returns its per-row outputs; + ``write_batch_kv`` takes them too and writes the first replay layer's KV for + every row. ``metadata_prefixes`` are the attention metadata keys the layers + read, so the replay batch builds only those. ``first_swa_prefix`` keys the + first replay layer's SWA metadata, built with the batch's window starts. """ def __init__( self, window: int, run_layers: Callable[..., tuple[torch.Tensor, ...]], - row_buffers: list[torch.Tensor], + write_batch_kv: Callable[..., None], + metadata_prefixes: set[str], + first_swa_prefix: str, ) -> None: self.window = window self.run_layers = run_layers - self.row_buffers = row_buffers - # The replay batch, set by the model state every step: its rows of the - # batch and its forward context. None runs the layers on the batch. - self.rows: torch.Tensor | None = None - self.forward_context: ForwardContext | None = None + self.write_batch_kv = write_batch_kv + self.metadata_prefixes = metadata_prefixes + self.first_swa_prefix = first_swa_prefix + # Set by the model state every step; None runs the layers on the batch. + self.replay_batch: ReplayBatch | None = None + # Set by the model state with replay graphs: PIECEWISE model graphs of + # this many tokens or more break out to the replay batch. + self.trim_threshold: int | None = None + self._break_outputs: list[torch.Tensor] | None = None def __call__( self, hidden_states: torch.Tensor | MoEOutput, *states: torch.Tensor | None ) -> tuple[torch.Tensor, ...]: - rows = self.rows - if rows is None: + if self.uses_replay_batch(): + # Outside the break, so a PIECEWISE graph captures it. + self.write_batch_kv(hidden_states, *states) + if in_piecewise_cudagraph(): + return self._run_in_graph_break(hidden_states, *states) + return self._run(hidden_states, *states) + + def uses_replay_batch(self) -> bool: + """Whether this forward uses the replay batch: in a PIECEWISE graph by its + size, fixed at capture (no replay batch is set then); else if one is set.""" + if in_piecewise_cudagraph(): + batch_descriptor = get_forward_context().batch_descriptor + assert batch_descriptor is not None + return self.graph_uses_replay_batch(batch_descriptor.num_tokens) + return self.replay_batch is not None + + def graph_uses_replay_batch(self, num_tokens: int) -> bool: + """Whether a PIECEWISE graph of ``num_tokens`` breaks out to the replay + batch; capture and the model state's runtime replay batch share it.""" + return self.trim_threshold is not None and num_tokens >= self.trim_threshold + + @eager_break_during_capture + def _run_in_graph_break( + self, hidden_states: torch.Tensor | MoEOutput, *states: torch.Tensor | None + ) -> tuple[torch.Tensor, ...]: + """A graph break: the outputs land in fixed buffers the next segment reads.""" + outputs = self._run(hidden_states, *states) + if self._break_outputs is None: + # Sized by the first call: the largest model graph is captured first. + self._break_outputs = [torch.empty_like(out) for out in outputs] + return tuple( + buf[: out.shape[0]].copy_(out) + for buf, out in zip(self._break_outputs, outputs) + ) + + def _run( + self, hidden_states: torch.Tensor | MoEOutput, *states: torch.Tensor | None + ) -> tuple[torch.Tensor, ...]: + replay_batch = self.replay_batch + if replay_batch is None: return self.run_layers(hidden_states, *states) - # A trimming step holds a prefill longer than the window, more tokens - # than any step whose MoE leaves its finalize to the next layer. + # Replay batch steps hold more than `window` tokens: no deferred MoE finalize. assert isinstance(hidden_states, torch.Tensor) - num_rows = rows.shape[0] - for buf in self.row_buffers: - buf[:num_rows].copy_(buf.index_select(0, rows)) - with override_forward_context(self.forward_context): - row_outputs = self.run_layers( - hidden_states.index_select(0, rows), - *(None if t is None else t.index_select(0, rows) for t in states), - ) + rows = replay_batch.rows + inputs = (hidden_states, *states) + with override_forward_context(replay_batch.forward_context): + if replay_batch.run_graph is not None: + row_outputs = replay_batch.run_graph(rows, inputs) + else: + row_outputs = self.run_layers( + *(None if t is None else t.index_select(0, rows) for t in inputs) + ) # The trimmed rows' outputs stay zero; nothing reads them. num_tokens = hidden_states.shape[0] return tuple( diff --git a/vllm/models/deepseek_v41/nvidia/decoder_replay_cudagraph.py b/vllm/models/deepseek_v41/nvidia/decoder_replay_cudagraph.py new file mode 100644 index 0000000000..871f535584 --- /dev/null +++ b/vllm/models/deepseek_v41/nvidia/decoder_replay_cudagraph.py @@ -0,0 +1,102 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""CUDA graphs of the decoder replay layers on a step's replay batch.""" + +from collections.abc import Callable +from copy import copy +from dataclasses import replace + +import torch + +from vllm.compilation.breakable_cudagraph import BreakableCUDAGraphWrapper +from vllm.config import CUDAGraphMode, VllmConfig +from vllm.config.kernel import MEGA_MOE_BACKENDS +from vllm.forward_context import get_forward_context, override_forward_context +from vllm.models.deepseek_v41.decoder_replay_layers import DecoderReplayLayers +from vllm.v1.worker.gpu.cudagraph_utils import ( + BatchExecutionDescriptor, + CudaGraphManager, +) + + +class DecoderReplayCudaGraphManager(CudaGraphManager): + """Breakable PIECEWISE graphs of the replay layers keyed by the replay batch's + padded size. They read the layer inputs from static buffers.""" + + def __init__(self, vllm_config: VllmConfig, device, layers: DecoderReplayLayers): + compilation = copy(vllm_config.compilation_config) + scheduler = vllm_config.scheduler_config + sizes = compilation.decoder_replay_cudagraph_capture_sizes + if not sizes: + bound = min( + compilation.max_cudagraph_capture_size, + scheduler.max_num_seqs * layers.window, + ) + # Replay batches hold at least `window` rows: step by window, or by 1/8 + # of the power-of-two interval once wider (256 in [2048, 4096)). + sizes, s = [bound], layers.window + while s < bound: + sizes.append(s) + s += max(layers.window, 1 << max(s.bit_length() - 4, 0)) + sizes = sorted(set(sizes)) + if max(sizes) > scheduler.max_num_batched_tokens: + raise ValueError( + "decoder_replay_cudagraph_capture_sizes must not exceed " + "max_num_batched_tokens" + ) + compilation.cudagraph_capture_sizes = sizes + compilation.max_cudagraph_capture_size = max(sizes) + replay_config = copy(vllm_config) + replay_config.compilation_config = compilation + super().__init__(replay_config, device, CUDAGraphMode.PIECEWISE, 1) + self.layers = layers + self.breakable_cg_runner: BreakableCUDAGraphWrapper = BreakableCUDAGraphWrapper( + layers.run_layers, replay_config + ) + config, act = vllm_config.model_config.hf_config, vllm_config.model_config.dtype + assert isinstance(act, torch.dtype) + hc, h, f32 = config.hc_mult, config.hidden_size, torch.float32 + mega_moe = vllm_config.kernel_config.moe_backend in MEGA_MOE_BACKENDS + ids = torch.int64 if mega_moe else torch.int32 + # hidden_states, positions, input_ids, pre_mix, post_mix, res_mix, residual + shapes = [(h,), (), (), (hc,), (hc, 1), (hc, hc), (hc, h)] + dtypes = [act, torch.int64, ids, f32, f32, f32, act] + n = compilation.max_cudagraph_capture_size + self.input_buffers = [ + torch.zeros(n, *s, dtype=d, device=device) for s, d in zip(shapes, dtypes) + ] + + def capture_replay_graphs( + self, prepare: Callable[[BatchExecutionDescriptor], None] + ) -> None: + """Capture each size on the replay batch ``prepare`` sets on the layers.""" + + def create_forward_fn(desc: BatchExecutionDescriptor, warmup: bool): + prepare(desc) + assert self.layers.replay_batch is not None + context = self.layers.replay_batch.forward_context + inputs = [buf[: desc.num_tokens] for buf in self.input_buffers] + + def forward_fn(cg_mode: CUDAGraphMode) -> None: + with override_forward_context( + replace(context, cudagraph_runtime_mode=cg_mode) + ): + self.breakable_cg_runner(*inputs) + + return forward_fn + + self.capture(create_forward_fn, "Capturing decoder replay CUDA graphs") + self.layers.replay_batch = None + + def run(self, rows: torch.Tensor, inputs: tuple) -> tuple[torch.Tensor, ...]: + """Replay the forward context's graph on ``rows`` of ``inputs``.""" + num_rows = rows.shape[0] + for buf, t in zip(self.input_buffers, inputs): + torch.index_select(t, 0, rows, out=buf[:num_rows]) + batch_descriptor = get_forward_context().batch_descriptor + assert batch_descriptor is not None + num_tokens = batch_descriptor.num_tokens + outputs = self.breakable_cg_runner( + *(buf[:num_tokens] for buf in self.input_buffers) + ) + return tuple(out[:num_rows] for out in outputs) diff --git a/vllm/models/deepseek_v41/nvidia/dspark.py b/vllm/models/deepseek_v41/nvidia/dspark.py index d1534d5248..a09c3159c2 100644 --- a/vllm/models/deepseek_v41/nvidia/dspark.py +++ b/vllm/models/deepseek_v41/nvidia/dspark.py @@ -500,22 +500,8 @@ class DSparkDeepseekV4ForCausalLM(nn.Module): for layer in self.model.layers: layer.ffn.finalize_mega_moe_weights() - def _finalize_attn(self) -> None: - """Run the attention backend's post-load weight step on the draft layers. - - They are ordinary DeepseekV4DecoderLayers, so they take the engine-wide - attention backend and owe it whatever the target model owes it -- mega - attention permutes wq_b / wo_a here and refuses to run without it. - Idempotent, like the target's. - """ - for layer in self.model.layers: - finalize = getattr(layer.attn, "finalize_loaded_weights", None) - if finalize is not None: - finalize() - def process_weights_after_loading(self) -> None: self._finalize_moe() - self._finalize_attn() def _remap_dspark_name(self, name: str) -> str | None: """Map a checkpoint ``mtp.{i}.*`` name to this model's parameter path. diff --git a/vllm/models/deepseek_v41/nvidia/flash_mla_mega_attn.py b/vllm/models/deepseek_v41/nvidia/flash_mla_mega_attn.py index 8efc987739..7a7f4f8c90 100644 --- a/vllm/models/deepseek_v41/nvidia/flash_mla_mega_attn.py +++ b/vllm/models/deepseek_v41/nvidia/flash_mla_mega_attn.py @@ -32,8 +32,10 @@ from vllm.models.deepseek_v41.common.ops import ( ) from vllm.models.deepseek_v41.common.ops.fused_layout import ( WV_GROUP_SIZE, - permute_wo_a_, - permute_wq_b_, + o_fused_chunk_permutation, + o_fused_permutation, + permute_on_load, + q_fused_permutation, ) from vllm.models.deepseek_v41.nvidia.flashmla import DeepseekV4FlashMLAAttention from vllm.models.deepseek_v41.sparse_mla import ( @@ -182,7 +184,13 @@ class DeepseekV4MegaAttnAttention(DeepseekV4FlashMLAAttention): "wo_a group." ) self.n_wv_group = self.padded_heads // WV_GROUP_SIZE - self._fused_layouts_ready = False + q_perm = q_fused_permutation(self.n_local_heads, self.head_dim) + permute_on_load(self.wq_b.weight, q_perm, dim=0) + permute_on_load(self.wq_b.weight_scale, q_perm, dim=0) + o_perm = o_fused_permutation(WV_GROUP_SIZE, self.head_dim) + permute_on_load(self.wo_a.weight, o_perm, dim=1) + o_chunk_perm = o_fused_chunk_permutation(WV_GROUP_SIZE, self.head_dim) + permute_on_load(self.wo_a.weight_scale, o_chunk_perm, dim=1) # ---- interface contract ------------------------------------------------ @@ -227,25 +235,6 @@ class DeepseekV4MegaAttnAttention(DeepseekV4FlashMLAAttention): ) return self._wo_b_proj(z.flatten(1)) - # ---- weights ----------------------------------------------------------- - - def finalize_loaded_weights(self) -> None: - """Permute wq_b rows / wo_a columns into the kernel's layouts. - - Idempotent: a second post-load pass must not permute twice. - """ - if self._fused_layouts_ready: - return - permute_wq_b_( - self.wq_b.weight.data, self.wq_b.weight_scale.data, self.n_local_heads - ) - permute_wo_a_( - self.wo_a.weight.data, - self.wo_a.weight_scale.data, - self.n_local_heads // self.n_local_groups, - ) - self._fused_layouts_ready = True - # ---- forward ----------------------------------------------------------- def forward_mqa( @@ -255,11 +244,6 @@ class DeepseekV4MegaAttnAttention(DeepseekV4FlashMLAAttention): positions: torch.Tensor, output: QuantizedActivation, ) -> None: - if not self._fused_layouts_ready: - raise RuntimeError( - f"{self.prefix}: wq_b / wo_a were never permuted for the mega " - "attention kernel; refusing to run with mismatched layouts." - ) attn_metadata = get_forward_context().attn_metadata if attn_metadata is None: # Warmup dummy run: reserve the prefill workspace, produce zeros. diff --git a/vllm/models/deepseek_v41/nvidia/model.py b/vllm/models/deepseek_v41/nvidia/model.py index bd91e8f12b..8ec0155f73 100644 --- a/vllm/models/deepseek_v41/nvidia/model.py +++ b/vllm/models/deepseek_v41/nvidia/model.py @@ -100,6 +100,7 @@ from vllm.v1.worker.ubatching import dbo_current_ubatch_id from ..common.engram import ( Engram, + EngramBatch, EngramLayout, NgramHashState, can_share_engram_tables, @@ -623,6 +624,40 @@ class DeepseekV4DecoderLayer(nn.Module): torch.cuda.current_stream().wait_stream(mhc_stream) return x, residual, post_mix, res_mix, ffn_pre, previous_aux + def write_kv( + self, + x: torch.Tensor | MoEOutput, + positions: torch.Tensor, + input_ids: torch.Tensor | None, + pre_mix: torch.Tensor, + post_mix: torch.Tensor, + res_mix: torch.Tensor, + residual: torch.Tensor, + ) -> None: + """Write the KV ``forward`` would for the replay layers' inputs, skipping + the rest of the layer; the replay batch then reruns its rows.""" + # The replay batch rules out Engram and sequence parallel here. + assert self.engram is None and not self.use_sequence_parallel + *_, x, _, _ = mhc_shifted_post_pre( + x, + residual, + post_mix, + res_mix, + self.hc_attn_fn, + self.hc_attn_scale, + self.hc_attn_base, + self.rms_norm_eps, + self.hc_eps, + self.hc_eps, + self.hc_post_alpha, + self.hc_sinkhorn_iters, + pre_mix=pre_mix, + norm_weight=self.attn_norm.weight, + norm_eps=self.attn_norm.variance_epsilon, + reduce_results=self.fuse_mhc_all_reduce, + ) + self.attn.forward_kv(positions, x) + class DeepseekV4Model(nn.Module, EagleModelMixin): def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""): @@ -743,6 +778,14 @@ class DeepseekV4Model(nn.Module, EagleModelMixin): ), prefix=f"{prefix}.layers", ) + self.engram_batch = EngramBatch.create( + [ + layer.engram + for layer in islice(self.layers, self.start_layer, self.end_layer) + if getattr(layer, "engram", None) is not None + ], + vllm_config.scheduler_config.max_num_batched_tokens, + ) if self.fuse_mhc_all_reduce: # A MoE's top-k finalize folds into the next layer's first mHC # boundary, which the last local layer lacks and an engram layer @@ -765,19 +808,30 @@ class DeepseekV4Model(nn.Module, EagleModelMixin): and self._decoder_replay_supported(vllm_config, cut) and self.layers[cut].attn.swa_cache_layer.bounded_replay ): - self.decoder_replay_start = cut + 1 + assert self.layers[cut].attn.compress_ratio <= 1 + # The last KV source writes every row's KV, then replays. + self.decoder_replay_start = cut + # The attention metadata keys the replay layers read. + metadata_prefixes = set() + for layer in islice(self.layers, cut, self.end_layer): + attn = typing.cast(DeepseekV4DecoderLayer, layer).attn + metadata_prefixes.add(attn.swa_cache_layer.prefix) + if attn.compressed_cache_prefix is not None: + metadata_prefixes.add(attn.compressed_cache_prefix) + if attn.indexer is not None: + metadata_prefixes.add(attn.indexer.k_cache.prefix) self.decoder_replay_layers = DecoderReplayLayers( config.sliding_window, self._run_replay_layers, - [ - buf - for buf in (self.topk_indices_buffer, self.candidate_block_buffer) - if buf is not None - ], + self.layers[cut].write_kv, + metadata_prefixes, + self.layers[cut].attn.swa_cache_layer.prefix, ) logger.info_once( - "Decoder SWA bounded replay: in eager prefill steps, layers " + "Decoder SWA bounded replay: in eager prefill steps, layer %d " + "writes its KV for every token, and the rest of it and layers " "%d-%d run on each request's last %d tokens only.", + cut, cut + 1, self.end_layer - 1, config.sliding_window, @@ -926,15 +980,18 @@ class DeepseekV4Model(nn.Module, EagleModelMixin): if engram_hashes is not None: # Gather all Engram rows before entering the decoder layers. # One gather feeds every layer sharing the DP-split table. - gathered_hashes = gather_engram_hashes( - engram_hashes, dp_shared_memory=self.engram_dp_shared_memory - ) - for layer in islice(self.layers, self.start_layer, self.end_layer): - engram = getattr(layer, "engram", None) - if engram is not None: - engram.prepare_embeddings( - gathered_hashes[:, engram.layer_hash_index] - ) + if self.engram_batch is not None: + self.engram_batch.prepare_embeddings(engram_hashes) + else: + gathered_hashes = gather_engram_hashes( + engram_hashes, dp_shared_memory=self.engram_dp_shared_memory + ) + for layer in islice(self.layers, self.start_layer, self.end_layer): + engram = getattr(layer, "engram", None) + if engram is not None: + engram.prepare_embeddings( + gathered_hashes[:, engram.layer_hash_index] + ) full_num_tokens = positions.shape[0] if self.use_sequence_parallel: @@ -1110,9 +1167,8 @@ class DeepseekV4Model(nn.Module, EagleModelMixin): res_mix: torch.Tensor, residual: torch.Tensor, ) -> tuple[torch.Tensor, ...]: - """The layers past the last KV source, on whatever rows they are given; - returns their output, the last FFN's pre-mix and the aux hidden states - they capture.""" + """Layers ``decoder_replay_start``.. on the given rows; returns their output, + the last FFN's pre-mix and the aux hidden states they capture.""" aux_hidden_by_layer: dict[int, torch.Tensor] = {} hidden_states, residual, post_mix, res_mix, pre_mix = self._run_layers( range(self.decoder_replay_start, self.end_layer), @@ -1144,7 +1200,7 @@ class DeepseekV4Model(nn.Module, EagleModelMixin): ) def _decoder_replay_supported(self, vllm_config: VllmConfig, cut: int) -> bool: - """Whether this rank may trim the layers after ``cut``; warns when not.""" + """Whether this rank may replay from layer ``cut`` on; warns when not.""" parallel_config = vllm_config.parallel_config spec_config = vllm_config.speculative_config draft_config = spec_config.draft_model_config if spec_config else None @@ -1166,8 +1222,8 @@ class DeepseekV4Model(nn.Module, EagleModelMixin): "the replay-layer batch shrinks per rank, which sequence and " "prefill-context parallelism and microbatching cannot follow" ) - elif any(i > cut for i in getattr(self.config, "engram_layer_ids", ())): - reason = "an Engram layer sits after the last KV source layer" + elif any(i >= cut for i in getattr(self.config, "engram_layer_ids", ())): + reason = "an Engram layer sits at or after the last KV source layer" elif draft_config is not None and ( draft_window is None or draft_window > window @@ -1354,17 +1410,6 @@ class DeepseekV4Model(nn.Module, EagleModelMixin): for layer in islice(self.layers, self.start_layer, self.end_layer): layer.ffn.finalize_mega_moe_weights() - def finalize_mega_attn_weights(self) -> None: - """Permute wq_b / wo_a into FlashMLA's mega-attention layouts. - - A no-op for every other attention layer, and idempotent, so a second - post-load pass cannot permute twice. - """ - for layer in islice(self.layers, self.start_layer, self.end_layer): - finalize = getattr(layer.attn, "finalize_loaded_weights", None) - if finalize is not None: - finalize() - def finalize_mhc_broadcast_weights(self) -> None: if not get_pp_group().is_first_rank or self.start_layer >= self.end_layer: return @@ -1637,7 +1682,6 @@ class DeepseekV41LLMForCausalLM( def process_weights_after_loading(self) -> None: self.model.finalize_mega_moe_weights() self.model.finalize_mhc_broadcast_weights() - self.model.finalize_mega_attn_weights() def get_expert_mapping(self) -> list[tuple[str, str, int, str]]: return self.model.get_expert_mapping() diff --git a/vllm/models/deepseek_v41/nvidia/model_state.py b/vllm/models/deepseek_v41/nvidia/model_state.py index 30b287db22..cc3ade80b8 100644 --- a/vllm/models/deepseek_v41/nvidia/model_state.py +++ b/vllm/models/deepseek_v41/nvidia/model_state.py @@ -8,10 +8,17 @@ import torch import torch.distributed as dist import torch.nn as nn +from vllm.compilation.breakable_cudagraph import is_breakable_cudagraph_enabled from vllm.config import CUDAGraphMode, VllmConfig -from vllm.distributed.parallel_state import get_dp_group -from vllm.forward_context import DPMetadata, create_forward_context -from vllm.models.deepseek_v41.decoder_replay_layers import DecoderReplayLayers +from vllm.distributed.parallel_state import get_dp_group, get_pp_group +from vllm.forward_context import BatchDescriptor, DPMetadata, create_forward_context +from vllm.models.deepseek_v41.decoder_replay_layers import ( + DecoderReplayLayers, + ReplayBatch, +) +from vllm.models.deepseek_v41.nvidia.decoder_replay_cudagraph import ( + DecoderReplayCudaGraphManager, +) from vllm.triton_utils import tl, triton from vllm.v1.attention.backends.mla.sparse_swa import DeepseekSparseSWAMetadataBuilder from vllm.v1.attention.backends.utils import PAD_SLOT_ID @@ -19,8 +26,10 @@ from vllm.v1.core.sched.output import NewRequestData from vllm.v1.kv_cache_interface import KVCacheConfig from vllm.v1.worker.dp_utils import should_skip_dp_coordination from vllm.v1.worker.gpu.attn_utils import build_slot_mappings_by_layer +from vllm.v1.worker.gpu.block_table import BlockTables from vllm.v1.worker.gpu.buffer_utils import UvaBufferPool -from vllm.v1.worker.gpu.input_batch import InputBatch +from vllm.v1.worker.gpu.cudagraph_utils import BatchExecutionDescriptor +from vllm.v1.worker.gpu.input_batch import InputBatch, InputBuffers from vllm.v1.worker.gpu.mm.encoder_cache import EncoderCache from vllm.v1.worker.gpu.model_states.default import DefaultModelState from vllm.v1.worker.gpu.model_states.interface import ModelSpecificAttnMetadata @@ -229,7 +238,33 @@ class DeepseekV41ModelState(DefaultModelState): self._kept_kv_start = torch.zeros( self.max_num_reqs, dtype=torch.int32, device=device ) - self._replay_attn_groups: list[list[AttentionGroup]] | None = None + # The replay layers' and the first one's SWA attention groups. + self._replay_attn_groups: ( + tuple[list[list[AttentionGroup]], list[list[AttentionGroup]]] | None + ) = None + layers = self.decoder_replay_layers + compilation_config = vllm_config.compilation_config + self.replay_cudagraphs: DecoderReplayCudaGraphManager | None = None + if ( + layers is not None + and compilation_config.cudagraph_decoder_replay + and compilation_config.cudagraph_mode.has_piecewise_cudagraphs() + and is_breakable_cudagraph_enabled() + and get_pp_group().is_first_rank + # The graphs' fixed inputs hold no LoRA mapping and no None input ids. + and vllm_config.lora_config is None + and not vllm_config.model_config.enable_prompt_embeds + ): + threshold = compilation_config.decoder_replay_trim_threshold + if threshold <= layers.window: + raise ValueError( + "decoder_replay_trim_threshold must exceed the replay window " + f"of {layers.window} tokens: smaller graphs have nothing to trim" + ) + self.replay_cudagraphs = DecoderReplayCudaGraphManager( + vllm_config, device, layers + ) + layers.trim_threshold = threshold def add_request(self, req_index: int, new_req_data: NewRequestData) -> None: super().add_request(req_index, new_req_data) @@ -409,15 +444,20 @@ class DeepseekV41ModelState(DefaultModelState): attn_groups: list[list[AttentionGroup]], kv_cache_config: KVCacheConfig, replay_start: torch.Tensor, + capture_desc: BatchExecutionDescriptor | None = None, ) -> None: - """Set the replay layers' batch for this forward, or none when nothing - trims. Only eager steps trim: a CUDA graph keeps the layers on the whole - batch, and ranks share the graph mode. Under data parallelism every rank - replays if any does.""" + """Set the replay layers' batch for this forward, or none. Eager steps trim; + captures and PIECEWISE graphs at the trim threshold or above break out to a + replay batch, trimmed or not; FULL and smaller PIECEWISE graphs keep the + whole batch. Under DP every rank replays if any does.""" layers = self.decoder_replay_layers assert layers is not None - layers.rows = layers.forward_context = None - if cudagraph_mode != CUDAGraphMode.NONE: + layers.replay_batch = None + force_replay_batch = capture_desc is not None or ( + cudagraph_mode == CUDAGraphMode.PIECEWISE + and layers.graph_uses_replay_batch(input_batch.num_tokens_after_padding) + ) + if cudagraph_mode != CUDAGraphMode.NONE and not force_replay_batch: return num_reqs = input_batch.num_reqs @@ -427,48 +467,126 @@ class DeepseekV41ModelState(DefaultModelState): query_lens, np.minimum(query_lens, layers.window), ) + # A batch of context-free non-prefill rows is a dummy (an idle DP rank's): + # nothing reads it and its idx_mapping hits stale slots, so keep one row. + seq_lens = input_batch.seq_lens_cpu_upper_bound[:num_reqs].numpy() + is_dummy = not input_batch.is_prefilling_np[:num_reqs].any() and bool( + (seq_lens == query_lens).all() + ) + if is_dummy and capture_desc is None: + kept_lens = np.zeros_like(kept_lens) + kept_lens[0] = 1 # Dummy batches (captures, an idle DP rank's) are not prefills: they # keep their rows unless a rank trims. - trims = bool( + needs_replay_batch = force_replay_batch or bool( (kept_lens < query_lens)[input_batch.is_prefilling_np[:num_reqs]].any() ) num_tokens = int(kept_lens.sum()) - dp_metadata = None + dp_tokens = None if self.vllm_config.parallel_config.data_parallel_size > 1: - trims, dp_metadata = self._agree_across_dp(trims, num_tokens) - if not trims: + needs_replay_batch, dp_tokens = self._agree_across_dp( + needs_replay_batch, num_tokens + ) + if not needs_replay_batch: return + cudagraphs, run_graph = self.replay_cudagraphs, None + num_padded, cg_mode = num_tokens, CUDAGraphMode.NONE + if cudagraphs is not None: + # Every rank runs the graph that fits the largest rank's batch. + dp_max_tokens = num_tokens if dp_tokens is None else int(dp_tokens.max()) + desc = capture_desc or cudagraphs.dispatch(num_reqs, dp_max_tokens, None, 0) + if desc.cg_mode == CUDAGraphMode.PIECEWISE: + num_padded, cg_mode = desc.num_tokens, desc.cg_mode + run_graph = cudagraphs.run + if dp_tokens is not None: + dp_tokens.fill_(num_padded) + dp_metadata = None + if dp_tokens is not None: + dp_metadata = DPMetadata.make( + self.vllm_config.parallel_config, num_padded, dp_tokens + ) kept_batch, kept_slot_mappings = self._kept_input_batch( - input_batch, slot_mappings, replay_start, kept_lens + input_batch, slot_mappings, replay_start, kept_lens, num_padded ) + if self._replay_attn_groups is None: + self._replay_attn_groups = ( + self._copy_attn_groups(attn_groups, layers.metadata_prefixes), + self._copy_attn_groups(attn_groups, {layers.first_swa_prefix}), + ) + replay_groups, first_swa_groups = self._replay_attn_groups attn_metadata = super().prepare_attn( kept_batch, CUDAGraphMode.NONE, block_tables, kept_slot_mappings, - self._replay_groups(attn_groups), + replay_groups, kv_cache_config, model_specific_attn_metadata=ReplayAttnMetadata( self._kept_kv_start[:num_reqs] ), ) - layers.rows = self._kept_rows[:num_tokens] - layers.forward_context = create_forward_context( - attn_metadata, - self.vllm_config, - dp_metadata=dp_metadata, - slot_mapping=build_slot_mappings_by_layer( - kept_slot_mappings, kv_cache_config + # The first replay layer wrote every row's window KV: its window starts + # are the batch's. + first_swa = layers.first_swa_prefix + attn_metadata[first_swa] = super().prepare_attn( + kept_batch, + CUDAGraphMode.NONE, + block_tables, + kept_slot_mappings, + first_swa_groups, + kv_cache_config, + model_specific_attn_metadata=ReplayAttnMetadata(replay_start), + )[first_swa] + layers.replay_batch = ReplayBatch( + self._kept_rows[:num_tokens], + create_forward_context( + attn_metadata, + self.vllm_config, + dp_metadata=dp_metadata, + cudagraph_runtime_mode=cg_mode, + batch_descriptor=BatchDescriptor(num_tokens=num_padded), + slot_mapping=build_slot_mappings_by_layer( + kept_slot_mappings, kv_cache_config + ), ), + run_graph, ) + def capture_inner_cudagraphs( + self, + input_buffers: InputBuffers, + block_tables: BlockTables, + attn_groups: list[list[AttentionGroup]], + kv_cache_config: KVCacheConfig, + ) -> None: + def prepare(desc: BatchExecutionDescriptor) -> None: + # Capture reuses the runtime path on a dummy batch, so the graph reads + # the buffers a real step fills. + num_reqs = min(desc.num_tokens, self.max_num_reqs) + self._prepare_replay_batch( + InputBatch.make_dummy(num_reqs, desc.num_tokens, input_buffers), + CUDAGraphMode.NONE, + block_tables.get_dummy_block_tables(num_reqs), + block_tables.get_dummy_slot_mappings(desc.num_tokens), + attn_groups, + kv_cache_config, + self._replay_start[:num_reqs].zero_(), + capture_desc=desc, + ) + + if self.replay_cudagraphs is not None: + # profile_only captures use pre-KV-cache groups: drop the cache. + self._replay_attn_groups = None + self.replay_cudagraphs.capture_replay_graphs(prepare) + def _kept_input_batch( self, input_batch: InputBatch, slot_mappings: torch.Tensor, replay_start: torch.Tensor, kept_lens: np.ndarray, + num_padded: int, ) -> tuple[InputBatch, torch.Tensor]: """Build the sub-``InputBatch`` of each request's last ``kept_lens`` rows, and its slot mappings. @@ -486,7 +604,8 @@ class DeepseekV41ModelState(DefaultModelState): input_batch.query_start_loc_np[: num_reqs + 1] - kept_query_start_loc_np ) assert self._kept_slot_mappings is not None - kept_slot_mappings = self._kept_slot_mappings[:, :num_tokens] + kept_slot_mappings = self._kept_slot_mappings[:, :num_padded] + kept_slot_mappings[:, num_tokens:].fill_(PAD_SLOT_ID) _gather_replay_batch_kernel[(num_reqs,)]( input_batch.query_start_loc, dropped_before, @@ -511,48 +630,47 @@ class DeepseekV41ModelState(DefaultModelState): kept_batch = replace( input_batch, num_tokens=num_tokens, - num_tokens_after_padding=num_tokens, + num_tokens_after_padding=num_padded, query_start_loc=self._kept_query_start_loc[: num_reqs + 1], query_start_loc_np=kept_query_start_loc_np, max_query_len=max_query_len, - positions=self._kept_positions[:num_tokens], + positions=self._kept_positions[:num_padded], fast_prefill=None, ) return kept_batch, kept_slot_mappings def _agree_across_dp( - self, trims: bool, num_tokens: int - ) -> tuple[bool, DPMetadata | None]: - """Whether any rank trims and, then, the replay layers' DP metadata from - every rank's replay token count.""" + self, needs_replay_batch: bool, num_tokens: int + ) -> tuple[bool, torch.Tensor | None]: + """Whether any rank needs a replay batch and, if so, each rank's token count.""" parallel_config = self.vllm_config.parallel_config agreed = torch.zeros(2, parallel_config.data_parallel_size, dtype=torch.int32) agreed[:, parallel_config.data_parallel_rank] = torch.tensor( - [trims, num_tokens] + [needs_replay_batch, num_tokens] ) if not should_skip_dp_coordination(): dist.all_reduce(agreed, group=get_dp_group().cpu_group) if not agreed[0].any(): return False, None - return True, DPMetadata.make(parallel_config, num_tokens, agreed[1]) + return True, agreed[1] - def _replay_groups( - self, attn_groups: list[list[AttentionGroup]] + def _copy_attn_groups( + self, attn_groups: list[list[AttentionGroup]], prefixes: set[str] ) -> list[list[AttentionGroup]]: - """The attention groups with metadata builders of their own, like each - microbatch's: a builder keeps the metadata it built, and the runner's - hold the batch's.""" - if self._replay_attn_groups is None: - self._replay_attn_groups = [] - for groups in attn_groups: - replay_groups = [] - for group in groups: - replay_group = replace(group, metadata_builders=[]) - replay_group.create_metadata_builders( - self.vllm_config, - self.device, - group.metadata_builders[0].kernel_block_size, - ) - replay_groups.append(replay_group) - self._replay_attn_groups.append(replay_groups) - return self._replay_attn_groups + """Copies of the groups holding any of ``prefixes``, with builders of their + own like a microbatch's: a builder keeps the metadata it built.""" + own_groups = [] + for groups in attn_groups: + copies = [] + for group in groups: + if prefixes.isdisjoint(group.layer_names): + continue + copy = replace(group, metadata_builders=[]) + copy.create_metadata_builders( + self.vllm_config, + self.device, + group.metadata_builders[0].kernel_block_size, + ) + copies.append(copy) + own_groups.append(copies) + return own_groups diff --git a/vllm/models/deepseek_v41/nvidia/ops/engram_peer.py b/vllm/models/deepseek_v41/nvidia/ops/engram_peer.py new file mode 100644 index 0000000000..4f9423bb1e --- /dev/null +++ b/vllm/models/deepseek_v41/nvidia/ops/engram_peer.py @@ -0,0 +1,173 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Symmetric-memory Engram exchange for slots below ENGRAM_A2A_MIN_SLOT.""" + +from typing import TYPE_CHECKING + +import torch +import torch.distributed as dist +from torch import nn + +from vllm.distributed import get_engram_dp_group +from vllm.platforms import current_platform +from vllm.triton_utils import tl, triton + +if TYPE_CHECKING: + from vllm.models.deepseek_v41.common.engram import Engram + + +@triton.jit(do_not_specialize=["num_tokens"]) +def _peer_lookup_kernel( + metadata, + id_ptrs, + output, + num_tokens, + HEADS: tl.constexpr, + LAYERS: tl.constexpr, + DP: tl.constexpr, + RANK: tl.constexpr, + DIM: tl.constexpr, + BLOCK: tl.constexpr, +): + """Look up this rank's heads for every peer's tokens into packed rows.""" + row = tl.program_id(0) * BLOCK + tl.arange(0, BLOCK) + local_heads: tl.constexpr = HEADS // DP + valid = row < num_tokens * LAYERS * HEADS + token = row // (LAYERS * local_heads) + layer, head = row // local_heads % LAYERS, row % local_heads + ids = tl.load(id_ptrs + token // num_tokens, valid, other=0).to( + tl.pointer_type(tl.int32) + ) + index = tl.load( + ids + (token % num_tokens * LAYERS + layer) * HEADS + RANK * local_heads + head, + valid, + other=-1, + ).to(tl.int64) + # Per-layer metadata: weight pointer, scale pointer, vocab start, vocab end. + table = metadata + layer * 4 + start, end = tl.load(table + 2), tl.load(table + 3) + owned = valid & (index >= start) & (index < end) + local = tl.where(owned, index - start, 0) + weight = tl.multiple_of(tl.load(table), DIM).to(tl.pointer_type(tl.uint8)) + scales = tl.load(table + 1) + scales = tl.multiple_of(scales, DIM // 32).to(tl.pointer_type(tl.uint8)) + col = tl.arange(0, DIM) + value = tl.load( + weight[:, None] + local[:, None] * DIM + col[None, :], owned[:, None], other=0 + ) + dst = row.to(tl.int64) * (DIM + DIM // 32) + tl.store(output + dst[:, None] + col[None, :], value, valid[:, None]) + scale_cols = tl.arange(0, DIM // 32) + scale = tl.load( + scales[:, None] + local[:, None] * (DIM // 32) + scale_cols[None, :], + owned[:, None], + other=0, + ) + tl.store(output + dst[:, None] + DIM + scale_cols[None, :], scale, valid[:, None]) + + +class EngramPeerExchange(nn.Module): + """Two phase barriers fence reuse of each other's buffers across forwards. + + Pointer tables are buffers so level-2 sleep restores them on wake-up. + """ + + def __init__(self, layers: list["Engram"], max_tokens: int): + import torch.distributed._symmetric_memory as symm_mem + + super().__init__() + group = get_engram_dp_group() + assert group is not None + self.dp_size, self.dp_rank = group.world_size, group.rank_in_group + device = layers[0].staged_rows.device + self.num_layers = len(layers) + self.hash_start = layers[0].layer_hash_index + table = layers[0].embed_tokens + self.heads, self.dim = table.n_hash_cols, table.dim + self.ids = symm_mem.empty( + (max_tokens, self.num_layers, self.heads), + dtype=torch.int32, + device=device, + ) + self.rows = symm_mem.empty( + ( + max_tokens * self.dp_size, + self.num_layers, + self.heads // self.dp_size, + self.dim + self.dim // 32, + ), + dtype=torch.uint8, + device=device, + ) + self.id_handle = symm_mem.rendezvous(self.ids, group.device_group) + self.row_handle = symm_mem.rendezvous(self.rows, group.device_group) + metadata = [] + for layer in layers: + table = layer.embed_tokens + weight, scale = table._storage() + metadata.append( + ( + weight.data_ptr(), + scale.data_ptr(), + table.vocab_start_idx, + table.vocab_end_idx, + ) + ) + for name, data in ( + ("metadata", metadata), + ("id_ptrs", self.id_handle.buffer_ptrs), + ("row_ptrs", self.row_handle.buffer_ptrs), + ): + self.register_buffer( + name, + torch.tensor(data, dtype=torch.int64, device=device), + persistent=False, + ) + + @classmethod + def create( + cls, layers: list["Engram"], max_tokens: int + ) -> "EngramPeerExchange | None": + # The Engram DP group is node-local and every rank agrees on these. + group = get_engram_dp_group() + assert group is not None + if not ( + current_platform.is_device_capability_family(100) + and group.world_size in (2, 4, 8) + and layers[0].embed_tokens.n_hash_cols % group.world_size == 0 + ): + return None + physical_id = current_platform.visible_device_id_to_physical_device_id( + torch.accelerator.current_device_index() + ) + physical_ids = [None] * group.world_size + dist.all_gather_object(physical_ids, physical_id, group=group.cpu_group) + if not current_platform.is_fully_connected(physical_ids): + return None + return cls(layers, max_tokens) + + def exchange(self, hashes: torch.Tensor, slot: int) -> torch.Tensor: + """Look up this rank's heads for every peer; return per-rank row pointers.""" + tokens = hashes.shape[0] + assert 0 < slot <= self.ids.shape[0] and tokens <= slot + hash_end = self.hash_start + self.num_layers + self.ids[:tokens].copy_(hashes[:, self.hash_start : hash_end]) + if tokens < slot: + self.ids[tokens:slot].fill_(-1) + # The preceding rows barrier completed every peer's hash reads. This + # barrier completes their previous output reads before rows are reused. + self.id_handle.barrier() + _peer_lookup_kernel[(triton.cdiv(slot * self.num_layers * self.heads, 16),)]( + self.metadata, + self.id_ptrs, + self.rows, + slot, + HEADS=self.heads, + LAYERS=self.num_layers, + DP=self.dp_size, + RANK=self.dp_rank, + DIM=self.dim, + BLOCK=16, + ) + self.row_handle.barrier() + return self.row_ptrs diff --git a/vllm/models/deepseek_v41/quant_config.py b/vllm/models/deepseek_v41/quant_config.py index bed60ae758..1d92943db4 100644 --- a/vllm/models/deepseek_v41/quant_config.py +++ b/vllm/models/deepseek_v41/quant_config.py @@ -4,8 +4,11 @@ from __future__ import annotations +import fnmatch from typing import TYPE_CHECKING, cast +import regex as re + from vllm.config import get_current_vllm_config from vllm.config.quantization import QuantSpec from vllm.model_executor.layers.fused_moe import ( @@ -23,6 +26,7 @@ from vllm.model_executor.layers.quantization.utils.quant_utils import ( ) _DEEPSEEK_V4_EXPERT_DTYPES = ("fp4", "fp8") +_EXPERTS_LAYER_RE = re.compile(r"(?:^|\.)layers\.(\d+)\.ffn\.experts$") if TYPE_CHECKING: from vllm.model_executor.layers.quantization.modelopt import ( @@ -54,6 +58,7 @@ class DeepseekV4FP8Config(Fp8Config): super().__init__(*args, **kwargs) self._resolved_expert_dtype: str | None = None self._resolved_moe_quant_algo: str | None = None + self._modelopt_ignore: tuple[str, ...] = () self._nvfp4_config: ModelOptNvFp4Config | None = None # ``is_scale_e8m0`` is a property that resolves on first read, # by which time the current vllm_config has been set. @@ -103,6 +108,21 @@ class DeepseekV4FP8Config(Fp8Config): self._resolve_moe_overrides() return self._resolved_moe_quant_algo or "" + def _is_nvfp4_expert_layer(self, prefix: str) -> bool: + if self.moe_quant_algo != "NVFP4": + return False + match = _EXPERTS_LAYER_RE.search(prefix) + if match is None or not self._modelopt_ignore: + return True + # MTP/draft layers are built as layers.{num_hidden_layers + k} but are + # named mtp.{k} in some checkpoints. + model_config = get_current_vllm_config().model_config + num_layers = model_config.hf_text_config.num_hidden_layers + idx = int(match.group(1)) + layer = f"mtp.{idx - num_layers}" if idx >= num_layers else f"layers.{idx}" + name = f"{layer}.ffn.experts" + return not any(fnmatch.fnmatchcase(name, p) for p in self._modelopt_ignore) + def _get_nvfp4_config(self) -> ModelOptNvFp4Config: if self._nvfp4_config is None: from vllm.model_executor.layers.quantization.modelopt import ( @@ -114,6 +134,11 @@ class DeepseekV4FP8Config(Fp8Config): kv_cache_quant_algo=None, exclude_modules=[], group_size=16, + quantization_args=( + self.online_quantization_config.args + if self.online_quantization_config is not None + else None + ), ) return self._nvfp4_config @@ -168,6 +193,9 @@ class DeepseekV4FP8Config(Fp8Config): @classmethod def from_config(cls, config: dict) -> DeepseekV4FP8Config: + # ModelOpt NVFP4 exports keep the experts they ``ignore`` in the native + # MXFP4 format, e.g. the bundled MTP/DSpark layers (``mtp.*``). + ignore = config.get("ignore") # Reroute AMD-Quark fused shared expert MXFP4 checkpoints onto the fp8 # path: the runtime layout matches the DeepSeek-native fp8 checkpoint, # so translate the schema into format Fp8Config.from_config expects. @@ -183,7 +211,12 @@ class DeepseekV4FP8Config(Fp8Config): name for name in quark_exclude if isinstance(name, str) ], } - return cast("DeepseekV4FP8Config", super().from_config(config)) + quant_config = cast("DeepseekV4FP8Config", super().from_config(config)) + if isinstance(ignore, list): + quant_config._modelopt_ignore = tuple( + p for p in ignore if isinstance(p, str) + ) + return quant_config def get_quant_method(self, layer, prefix): if ( @@ -216,7 +249,7 @@ class DeepseekV4FP8Config(Fp8Config): ): return UnquantizedFusedMoEMethod(layer.moe_config) if self.expert_dtype == "fp4": - if self.moe_quant_algo == "NVFP4": + if self._is_nvfp4_expert_layer(prefix): from vllm.model_executor.layers.quantization.modelopt import ( ModelOptNvFp4FusedMoE, ) diff --git a/vllm/models/glm5next/amd/sparse_indexer.py b/vllm/models/glm5next/amd/sparse_indexer.py index c357608c47..13e67de04b 100644 --- a/vllm/models/glm5next/amd/sparse_indexer.py +++ b/vllm/models/glm5next/amd/sparse_indexer.py @@ -733,6 +733,10 @@ class SparseAttnIndexerKpool(CustomOp): self.topk_backend = ( cfg.kernel_config.sparse_indexer_topk_backend if cfg is not None else "auto" ) + if cfg is not None and cfg.parallel_config.decode_context_parallel_size > 1: + raise NotImplementedError( + "SparseAttnIndexerKpool on ROCm does not support DCP." + ) def forward_hip( self, diff --git a/vllm/models/glm5next/common/attention.py b/vllm/models/glm5next/common/attention.py index d7bf70af07..a264717f0d 100644 --- a/vllm/models/glm5next/common/attention.py +++ b/vllm/models/glm5next/common/attention.py @@ -209,6 +209,9 @@ class Glm5NextTailCache(DeepseekV32IndexerCache): head_size_v=0, dtype=torch.bfloat16, sliding_window=ring, + # Tiny per-request ring; every DCP rank keeps the full tail so the + # rank owning a pool can compress it. + dcp_sharded=False, ) def get_attn_backend(self): diff --git a/vllm/models/glm5next/common/model.py b/vllm/models/glm5next/common/model.py index e15c00ecb8..e35887ca7c 100644 --- a/vllm/models/glm5next/common/model.py +++ b/vllm/models/glm5next/common/model.py @@ -23,6 +23,10 @@ from vllm.model_executor.layers.fused_moe import ( GateLinear, fused_moe_make_expert_params_mapping, ) +from vllm.model_executor.layers.fused_moe.utils import ( + is_model_fused_shared_expert_compatible, + resolve_layer_fused_shared_expert, +) from vllm.model_executor.layers.fusion.mm_input_norm import build_mm_input_norm from vllm.model_executor.layers.layernorm import RMSNorm from vllm.model_executor.layers.linear import ( @@ -62,9 +66,11 @@ from vllm.model_executor.models.glm4_1v import ( Glm4vForConditionalGeneration, ) from vllm.model_executor.models.interfaces import ( + EagleModelMixin, HasInnerState, IsHybrid, MixtureOfExperts, + SupportsEagle3, SupportsPP, ) from vllm.model_executor.models.utils import ( @@ -258,7 +264,12 @@ class Glm5NextMoE(nn.Module): ) swiglu_limit = config.swiglu_limit - if config.n_shared_experts is None: + self.is_fused_shared_expert_enabled = False + if config.n_shared_experts is not None: + self.is_fused_shared_expert_enabled = resolve_layer_fused_shared_expert( + quant_config, prefix + ) and _fused_shared_experts_tuned(parallel_config) + if config.n_shared_experts is None or self.is_fused_shared_expert_enabled: self.shared_experts = None else: intermediate_size = config.moe_intermediate_size * config.n_shared_experts @@ -294,7 +305,11 @@ class Glm5NextMoE(nn.Module): enable_eplb=self.enable_eplb, num_redundant_experts=self.n_redundant_experts, is_sequence_parallel=self.is_sequence_parallel, - n_shared_experts=None, + n_shared_experts=config.n_shared_experts + if self.is_fused_shared_expert_enabled + else None, + fuse_shared_experts=self.is_fused_shared_expert_enabled, + shared_expert_prefix=f"{prefix}.shared_experts", router_logits_dtype=self.gate.out_dtype, swiglu_limit=swiglu_limit, ) @@ -672,7 +687,7 @@ class Glm5NextDecoderLayer(nn.Module): ) -class Glm5NextModel(nn.Module): +class Glm5NextModel(nn.Module, EagleModelMixin): def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""): super().__init__() @@ -730,9 +745,13 @@ class Glm5NextModel(nn.Module): get_layer, prefix=f"{prefix}.layers", ) + self._aux_post_op = MHCPostOp() # The active slice is fixed after construction; cache it so forward # doesn't rebuild the slice (a fresh list) every step. self._active_layers = self.layers[self.start_layer : self.end_layer] + self.is_fused_shared_expert_enabled = is_model_fused_shared_expert_compatible( + self.layers, Glm5NextMoE, "mlp" + ) if get_pp_group().is_last_rank: self.norm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps) @@ -751,6 +770,26 @@ class Glm5NextModel(nn.Module): def embed_input_ids(self, input_ids: torch.Tensor) -> torch.Tensor: return self.embed_tokens(input_ids) + def _aux_hidden_state( + self, + hidden_states: torch.Tensor, + residual: torch.Tensor | None, + post: torch.Tensor | None, + comb: torch.Tensor | None, + ) -> torch.Tensor: + """Completed residual stream entering a layer, as one hidden vector. + + mHC layers defer their final ``hc_post`` into the next layer's fused + pre-op, so ``hidden_states`` holds the raw layer output while the + widened stream is completed here and contracted back to + ``hidden_size``. Non-mHC layers already return the summed stream. + """ + if post is None: + return hidden_states + assert residual is not None and comb is not None + completed = self._aux_post_op(hidden_states, residual, post, comb) + return hc_contract(completed, self.config.hc_mult) + def forward( self, input_ids: torch.Tensor | None, @@ -780,7 +819,15 @@ class Glm5NextModel(nn.Module): if self.is_sequence_parallel: hidden_states = sp_shard(hidden_states) - for layer in self._active_layers: + aux_hidden_states: list[torch.Tensor] = [] + for idx, layer in enumerate(self._active_layers, start=self.start_layer): + if idx in self.aux_hidden_state_layers: + aux_hidden_state = self._aux_hidden_state( + hidden_states, residual, post, comb + ) + if self.is_sequence_parallel: + aux_hidden_state = sp_all_gather(aux_hidden_state)[:full_num_tokens] + aux_hidden_states.append(aux_hidden_state) hidden_states, residual, post, comb = layer( positions, hidden_states, residual, post, comb ) @@ -795,10 +842,18 @@ class Glm5NextModel(nn.Module): {"hidden_states": hidden_states, "residual": residual} ) + if self.end_layer in self.aux_hidden_state_layers: + final_aux = self._aux_hidden_state(hidden_states, residual, post, comb) + if self.is_sequence_parallel: + final_aux = sp_all_gather(final_aux)[:full_num_tokens] + aux_hidden_states.append(final_aux) + if self.is_sequence_parallel: hidden_states = sp_all_gather(hidden_states)[:full_num_tokens] hidden_states = self.norm(hidden_states) + if aux_hidden_states: + return hidden_states, aux_hidden_states return hidden_states def load_weights(self, weights: Iterable[tuple[str, torch.Tensor]]) -> set[str]: @@ -834,12 +889,15 @@ class Glm5NextModel(nn.Module): ), 0, ) + num_fused_shared = _num_fused_shared_experts( + self.config.n_shared_experts, self.is_fused_shared_expert_enabled + ) expert_params_mapping = fused_moe_make_expert_params_mapping( self, ckpt_gate_proj_name="gate_proj", ckpt_down_proj_name="down_proj", ckpt_up_proj_name="up_proj", - num_experts=self.config.n_routed_experts, + num_experts=self.config.n_routed_experts + num_fused_shared, num_redundant_experts=num_redundant_experts, ) else: @@ -868,6 +926,8 @@ class Glm5NextModel(nn.Module): # Models trained using ColossalAI may include these tensors in # the checkpoint. Skip them. continue + if self.is_fused_shared_expert_enabled: + name = _fused_shared_expert_name(name, self.config.n_routed_experts) # Handle FP8 indexer WK: dequantize to BF16 for fusion with # weights_proj into wk_weights_proj. @@ -985,7 +1045,7 @@ class Glm5NextModel(nn.Module): class Glm5NextForCausalLM( - nn.Module, HasInnerState, SupportsPP, MixtureOfExperts, IsHybrid + nn.Module, HasInnerState, SupportsPP, MixtureOfExperts, IsHybrid, SupportsEagle3 ): def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""): super().__init__() @@ -1081,7 +1141,11 @@ class Glm5NextForCausalLM( dummy_inputs=Glm4vDummyInputsBuilder, ) class Glm5NextForConditionalGeneration( - Glm4vForConditionalGeneration, HasInnerState, IsHybrid, MixtureOfExperts + Glm4vForConditionalGeneration, + HasInnerState, + IsHybrid, + MixtureOfExperts, + SupportsEagle3, ): # The text model (KDA + dense-MLA + MoE) is a hybrid mamba model. The # multimodal wrapper must declare the same interfaces so vLLM treats it as @@ -1228,6 +1292,65 @@ def get_spec_layer_idx_from_weight_name( return None +def _fused_shared_experts_tuned(parallel_config: ParallelConfig) -> bool: + """AITER has fused-MoE configs tuned for the fused shared-expert shape + (one more expert and one more top-k slot than the routed MoE) only on + gfx950, with every expert on each rank and its weights split by TP4 or + TP8. Data, prefill context and expert parallelism change that split, so + any other GPU or parallel layout would run untuned fallback kernels.""" + from vllm.platforms.rocm import on_gfx950 + + reasons: list[str] = [] + if not on_gfx950(): + reasons.append("the GPU is not gfx950") + if parallel_config.tensor_parallel_size not in (4, 8): + reasons.append( + f"tensor_parallel_size is {parallel_config.tensor_parallel_size}" + ) + if parallel_config.data_parallel_size != 1: + reasons.append(f"data_parallel_size is {parallel_config.data_parallel_size}") + if parallel_config.prefill_context_parallel_size != 1: + reasons.append( + "prefill_context_parallel_size is " + f"{parallel_config.prefill_context_parallel_size}" + ) + if parallel_config.enable_expert_parallel: + reasons.append("expert parallelism is enabled") + + if not reasons: + return True + logger.warning_once( + "VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS is ignored for GLM-5.3-Flash: " + "%s. AITER has tuned configs for its fused shared-expert MoE only on " + "gfx950 at TP4 and TP8, without data, prefill context or expert " + "parallelism. Running the shared experts as a separate MLP.", + "; ".join(reasons), + ) + return False + + +def _num_fused_shared_experts(n_shared_experts: int | None, enabled: bool) -> int: + """Expert slots the fused MoE appends for the shared expert; must match the + ``num_fused_shared_experts`` that ``FusedMoE`` allocates.""" + if not enabled or n_shared_experts is None: + return 0 + if n_shared_experts > 1: + raise NotImplementedError( + "Fused shared-expert loading supports only 1 shared expert per " + f"layer, but config.n_shared_experts is {n_shared_experts}. Set " + "VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=0 to run the shared " + "experts as a separate MLP." + ) + return n_shared_experts + + +def _fused_shared_expert_name(name: str, n_routed_experts: int) -> str: + """Point a checkpoint ``mlp.shared_experts.*`` tensor at the fused MoE's + shared-expert slot, which follows the routed experts; other names are + returned unchanged.""" + return name.replace("mlp.shared_experts.", f"mlp.experts.{n_routed_experts}.", 1) + + def _try_load_fp8_indexer_wk(name, tensor, buf, params_dict, loaded_params): if "indexer.wk." not in name or "wk_weights" in name: return False diff --git a/vllm/models/glm5next/common/mtp.py b/vllm/models/glm5next/common/mtp.py index 816cc46fdc..2bb8b10d07 100644 --- a/vllm/models/glm5next/common/mtp.py +++ b/vllm/models/glm5next/common/mtp.py @@ -10,6 +10,9 @@ from vllm.config import VllmConfig from vllm.model_executor.layers.fused_moe import ( fused_moe_make_expert_params_mapping, ) +from vllm.model_executor.layers.fused_moe.utils import ( + is_model_fused_shared_expert_compatible, +) from vllm.model_executor.layers.layernorm import RMSNorm from vllm.model_executor.layers.logits_processor import LogitsProcessor from vllm.model_executor.layers.vocab_parallel_embedding import ( @@ -30,6 +33,8 @@ from .model import ( Glm5NextDecoderLayer, Glm5NextMLAAttention, Glm5NextMoE, + _fused_shared_expert_name, + _num_fused_shared_experts, _try_load_fp8_attn_proj, _try_load_fp8_indexer_wk, get_spec_layer_idx_from_weight_name, @@ -217,6 +222,9 @@ class Glm5NextMTP(nn.Module, DeepseekV2MixtureOfExperts): vllm_config=vllm_config, prefix=maybe_prefix(prefix, "model") ) self.set_moe_parameters() + self.is_fused_shared_expert_enabled = is_model_fused_shared_expert_compatible( + self.model.layers.values(), Glm5NextMoE, "mtp_block.mlp" + ) def set_moe_parameters(self): self.num_moe_layers = self.config.num_nextn_predict_layers @@ -298,12 +306,15 @@ class Glm5NextMTP(nn.Module, DeepseekV2MixtureOfExperts): ("wk_weights_proj", "wk", 0), ("wk_weights_proj", "weights_proj", 1), ] + num_fused_shared = _num_fused_shared_experts( + self.config.n_shared_experts, self.is_fused_shared_expert_enabled + ) expert_params_mapping = fused_moe_make_expert_params_mapping( self, ckpt_gate_proj_name="gate_proj", ckpt_down_proj_name="down_proj", ckpt_up_proj_name="up_proj", - num_experts=self.config.n_routed_experts, + num_experts=self.config.n_routed_experts + num_fused_shared, num_redundant_experts=self.num_redundant_experts, ) @@ -328,6 +339,8 @@ class Glm5NextMTP(nn.Module, DeepseekV2MixtureOfExperts): if spec_layer is None: continue name = self._rewrite_spec_layer_name(spec_layer, name) + if self.is_fused_shared_expert_enabled: + name = _fused_shared_expert_name(name, self.config.n_routed_experts) if _try_load_fp8_indexer_wk( name, diff --git a/vllm/models/glm5next/nvidia/ops/third_party/kda/fused_recurrent.py b/vllm/models/glm5next/nvidia/ops/third_party/kda/fused_recurrent.py index a9d0fba476..767b822170 100644 --- a/vllm/models/glm5next/nvidia/ops/third_party/kda/fused_recurrent.py +++ b/vllm/models/glm5next/nvidia/ops/third_party/kda/fused_recurrent.py @@ -87,7 +87,7 @@ def fused_recurrent_gated_delta_rule_fwd_kernel( SAFE_GATE: tl.constexpr, # bounded gate variant (only branch implemented) LOWER_BOUND: tl.constexpr, ): - i_k, i_v, i_nh = tl.program_id(0), tl.program_id(1), tl.program_id(2) + i_nh, i_v, i_k = tl.program_id(0), tl.program_id(1), tl.program_id(2) i_n, i_hv = i_nh // HV, i_nh % HV i_h = i_hv // (HV // H) if IS_VARLEN: @@ -260,7 +260,9 @@ def fused_recurrent_gated_delta_rule_fwd( else: stride_indices_seq, stride_indices_tok = ssm_state_indices.stride() - grid = (NK, NV, N * HV) + # N * HV goes in gridDim.x: gridDim.z is capped at 65535 and batch x heads exceeds it + # (e.g. GLM-5.3-Flash at TP=1: 1024 x 64 = 65536 -> "invalid argument" at CUDA-graph capture). + grid = (N * HV, NV, NK) fused_recurrent_gated_delta_rule_fwd_kernel[grid]( q=q, k=k, diff --git a/vllm/models/glm5next/nvidia/ops/third_party/kda/kernels.py b/vllm/models/glm5next/nvidia/ops/third_party/kda/kernels.py index 9805b37471..8f662f68aa 100644 --- a/vllm/models/glm5next/nvidia/ops/third_party/kda/kernels.py +++ b/vllm/models/glm5next/nvidia/ops/third_party/kda/kernels.py @@ -95,7 +95,9 @@ def fused_recurrent_kda_fwd( else: stride_indices_seq, stride_indices_tok = ssm_state_indices.stride() - grid = (NK, NV, N * HV) + # N * HV goes in gridDim.x: gridDim.z is capped at 65535 and batch x heads exceeds it + # (e.g. GLM-5.3-Flash at TP=1: 1024 x 64 = 65536 -> "invalid argument" at CUDA-graph capture). + grid = (N * HV, NV, NK) fused_recurrent_gated_delta_rule_fwd_kernel[grid]( q=q, k=k, diff --git a/vllm/models/glm5next/nvidia/sparse_indexer.py b/vllm/models/glm5next/nvidia/sparse_indexer.py index 00f51cc6a7..77fa4c4091 100644 --- a/vllm/models/glm5next/nvidia/sparse_indexer.py +++ b/vllm/models/glm5next/nvidia/sparse_indexer.py @@ -7,10 +7,12 @@ import torch import vllm.envs as envs from vllm.compilation.breakable_cudagraph import eager_break_during_capture from vllm.config import get_current_vllm_config_or_none +from vllm.distributed import get_dcp_group from vllm.forward_context import get_forward_context from vllm.logger import init_logger from vllm.model_executor.custom_op import CustomOp from vllm.model_executor.layers.indexer_topk import get_indexer_topk +from vllm.model_executor.layers.sparse_attn_indexer import _merge_dcp_topk_global from vllm.models.glm5next.common.sparse_indexer import ( RADIX_TOPK_WORKSPACE_SIZE, _build_decode_scatter_indices, @@ -24,6 +26,7 @@ from vllm.models.glm5next.common.sparse_indexer import ( from vllm.models.glm5next.nvidia.ops import kpool_compress as kpool_ops from vllm.platforms import current_platform from vllm.utils.deep_gemm import has_deep_gemm +from vllm.utils.import_utils import has_cutedsl from vllm.utils.torch_utils import ( LayerNameType, _resolve_layer_name, @@ -121,6 +124,9 @@ def sparse_attn_indexer_kpool( tail_kv_cache: torch.Tensor | None = None, tail_prefix: str | None = None, topk_backend: str = "auto", + dcp_rank: int = 0, + dcp_world_size: int = 1, + cp_kv_cache_interleave_size: int = 1, ) -> torch.Tensor: # careful! this will be None in dummy run attn_metadata = get_forward_context().attn_metadata @@ -170,6 +176,12 @@ def sparse_attn_indexer_kpool( has_decode = attn_metadata_narrowed.num_decodes > 0 has_prefill = attn_metadata_narrowed.num_prefills > 0 num_decode_tokens = attn_metadata_narrowed.num_decode_tokens + # Pools are the DCP sharding unit of the index K cache. + pool_cp_interleave = ( + cp_kv_cache_interleave_size // max(index_kpool, 1) + if dcp_world_size > 1 + else cp_kv_cache_interleave_size + ) # q_scale is required iff the FP4 cache path is enabled; the FP8 path # folds the Q scale into `weights` inside fused_indexer_q_rope_quant. @@ -284,16 +296,18 @@ def sparse_attn_indexer_kpool( scales_spec, ) for chunk in prefill_metadata.chunks if not short_prefill else (): - k_quant = k_quant_full[: chunk.total_seq_lens] - k_scale = k_scale_full[: chunk.total_seq_lens] + # Under DCP this rank only gathers the pools it owns. + assert chunk.local_cu_seq_lens is not None + k_quant = k_quant_full[: chunk.max_local_total_seq_lens] + k_scale = k_scale_full[: chunk.max_local_total_seq_lens] - if not chunk.skip_kv_gather: + if not chunk.skip_kv_gather and chunk.local_total_seq_lens > 0: ops.cp_gather_indexer_k_quant_cache( kv_cache, k_quant, k_scale, chunk.block_table, - chunk.cu_seq_lens, + chunk.local_cu_seq_lens, ) q_slice = q_quant[chunk.token_start : chunk.token_end] @@ -314,15 +328,18 @@ def sparse_attn_indexer_kpool( k_scale_cast = k_scale.view(torch.float32).squeeze(-1) from vllm.utils.deep_gemm import fp8_fp4_mqa_logits - logits = fp8_fp4_mqa_logits( - (q_slice_cast, q_scale_slice), - (k_quant_cast, k_scale_cast), - weights[chunk.token_start : chunk.token_end], - chunk.cu_seqlen_ks, - chunk.cu_seqlen_ke, - clean_logits=False, - ) - num_rows = logits.shape[0] + num_rows = q_slice.shape[0] + if chunk.local_total_seq_lens == 0: + logits = q_slice.new_empty((num_rows, 0), dtype=torch.float32) + else: + logits = fp8_fp4_mqa_logits( + (q_slice_cast, q_scale_slice), + (k_quant_cast, k_scale_cast), + weights[chunk.token_start : chunk.token_end], + chunk.cu_seqlen_ks, + chunk.cu_seqlen_ke, + clean_logits=False, + ) # kpool: logits are pool-granular (compress_ratio == index_kpool), # so topk selects pools. We pick topk_tokens // kpool pools then @@ -338,15 +355,28 @@ def sparse_attn_indexer_kpool( chunk.token_start : chunk.token_end, :topk_tokens ] - torch.ops._C.top_k_per_row_prefill( + if logits.shape[1] == 0: + topk_dst.fill_(-1) + else: + torch.ops._C.top_k_per_row_prefill( + logits, + chunk.cu_seqlen_ks, + chunk.cu_seqlen_ke, + topk_dst, + num_rows, + logits.stride(0), + logits.stride(1), + select_k, + ) + # Local pool ids -> request-relative global pool ids. + _merge_dcp_topk_global( logits, - chunk.cu_seqlen_ks, - chunk.cu_seqlen_ke, topk_dst, - num_rows, - logits.stride(0), - logits.stride(1), select_k, + dcp_rank, + dcp_world_size, + pool_cp_interleave, + row_starts=chunk.cu_seqlen_ks, ) if index_kpool > 1: @@ -584,6 +614,14 @@ def sparse_attn_indexer_kpool( select_k, attn_metadata_narrowed.max_seq_len, ) + _merge_dcp_topk_global( + logits, + topk_dst, + select_k, + dcp_rank, + dcp_world_size, + pool_cp_interleave, + ) # Resolve to token-level indices in the output buffer. if index_kpool > 1: @@ -669,14 +707,41 @@ class SparseAttnIndexerKpool(CustomOp): else "auto" ) _parallel = _cfg.parallel_config if _cfg is not None else None - if ( - _parallel is not None - and _parallel.prefill_context_parallel_size > 1 - and _parallel.decode_context_parallel_size > 1 - ): - raise NotImplementedError( - "SparseAttnIndexerKpool does not support PCP+DCP." - ) + self._parallel_config = _parallel + self._cp_kv_cache_interleave_size: int | None = None + self.dcp_world_size = 1 + self.dcp_rank = 0 + if _parallel is not None and _parallel.decode_context_parallel_size > 1: + if _parallel.prefill_context_parallel_size > 1: + raise NotImplementedError( + "SparseAttnIndexerKpool does not support PCP+DCP." + ) + self.dcp_world_size = _parallel.decode_context_parallel_size + self.dcp_rank = get_dcp_group().rank_in_group + if current_platform.is_cuda() and has_cutedsl(): + from vllm.model_executor.kernels.attention.dsa.dcp_indexer_cutedsl import ( # noqa: E501 + _PACK_DCP_TOPK_CANDIDATES_KERNEL, + _STABLE_TOPK_FROM_GATHERED_CANDIDATES_KERNEL, + ) + + _PACK_DCP_TOPK_CANDIDATES_KERNEL.register_warmup() + _STABLE_TOPK_FROM_GATHERED_CANDIDATES_KERNEL.register_warmup() + + @property + def cp_kv_cache_interleave_size(self) -> int: + """Resolved lazily like ``SparseAttnIndexer``: NIXL P/D can adjust it + after the model is built, and the indexer metadata builder sees the + adjusted value. + """ + if self.dcp_world_size == 1: + return 1 + if self._cp_kv_cache_interleave_size is None: + assert self._parallel_config is not None + value = self._parallel_config.cp_kv_cache_interleave_size + if isinstance(get_forward_context().attn_metadata, dict): + self._cp_kv_cache_interleave_size = value + return value + return self._cp_kv_cache_interleave_size def forward_native( self, @@ -743,4 +808,7 @@ class SparseAttnIndexerKpool(CustomOp): self.tail_cache.kv_cache if self.tail_cache is not None else None, self.tail_cache.prefix if self.tail_cache is not None else None, self.topk_backend, + self.dcp_rank, + self.dcp_world_size, + self.cp_kv_cache_interleave_size, ) diff --git a/vllm/models/hy_v4/nvidia/flashmla_sparse.py b/vllm/models/hy_v4/nvidia/flashmla_sparse.py index 1d62f2f556..52656b06a6 100644 --- a/vllm/models/hy_v4/nvidia/flashmla_sparse.py +++ b/vllm/models/hy_v4/nvidia/flashmla_sparse.py @@ -228,9 +228,9 @@ class HYV4FlashMLASparseImpl(FlashMLASparseImpl): -1, 1, kv_c_and_k_pe_cache.shape[-1] ) - # NOTE(Chen): kernel requires num_local_head to be a multiple of - # 64 on hopper and 128 on blackwell. Pad from q's head count, not - # self.num_heads: under DCP the heads are all-gathered before this. + # NOTE(Chen): kernel requires num_local_head to be a multiple of 64. + # Pad from q's head count, not self.num_heads: under DCP the heads are + # all-gathered before this. if actual_num_heads is None: actual_num_heads = q.shape[1] padded_num_heads = ( diff --git a/vllm/models/kimi_k3/amd/kda.py b/vllm/models/kimi_k3/amd/kda.py index e2dea74b6b..409cf6b1bf 100644 --- a/vllm/models/kimi_k3/amd/kda.py +++ b/vllm/models/kimi_k3/amd/kda.py @@ -13,6 +13,10 @@ from vllm.config import VllmConfig from vllm.distributed import divide from vllm.forward_context import get_forward_context from vllm.logger import init_logger +from vllm.model_executor.layers.fusion.quant_activation import ( + QuantizedActivation, + get_input_quant_key, +) from vllm.model_executor.layers.linear import ( ColumnParallelLinear, RowParallelLinear, @@ -37,6 +41,7 @@ from vllm.model_executor.layers.mamba.ops.causal_conv1d import ( causal_conv1d_fn, causal_conv1d_update, ) +from vllm.model_executor.layers.quantization.utils.quant_utils import QuantKey from vllm.model_executor.model_loader.weight_utils import sharded_weight_loader from vllm.model_executor.utils import set_weight_attrs from vllm.models.kimi_k3.amd.kda_metadata import KimiK3ROCmKDABackend @@ -300,13 +305,16 @@ class KimiK3DeltaAttention(GatedDeltaNetAttention): qkv = qkv.permute(1, 0, 2, 3).contiguous().unsqueeze(1) return qkv.unbind(0) + def get_input_quant_key(self) -> QuantKey | None: + return get_input_quant_key(self.in_proj_qkvgfab) + def forward( self, - hidden_states: torch.Tensor, + hidden_states: torch.Tensor | QuantizedActivation, positions: torch.Tensor, ) -> torch.Tensor: - num_tokens = hidden_states.size(0) projected_qkvgfab = self.in_proj_qkvgfab(hidden_states)[0] + num_tokens = projected_qkvgfab.shape[0] split_sizes = [ 3 * self.local_projection_size, @@ -326,8 +334,8 @@ class KimiK3DeltaAttention(GatedDeltaNetAttention): core_attn_out = torch.empty( (1, num_tokens, self.local_num_heads, self.head_dim), - dtype=hidden_states.dtype, - device=hidden_states.device, + dtype=projected_qkvgfab.dtype, + device=projected_qkvgfab.device, ) self._forward( diff --git a/vllm/models/kimi_k3/amd/latent_moe_runner.py b/vllm/models/kimi_k3/amd/latent_moe_runner.py index dbebfd4182..f6508d6b55 100644 --- a/vllm/models/kimi_k3/amd/latent_moe_runner.py +++ b/vllm/models/kimi_k3/amd/latent_moe_runner.py @@ -36,18 +36,23 @@ class ROCmLatentMoERunner(MoERunner): up_proj = getattr(transform, "up_proj", None) tp_size = get_tensor_model_parallel_world_size() + self._up_proj_preshard = bool(getattr(transform, "row_sharded", False)) self._up_proj_shard_size = 0 self._tail_shardable = ( up_proj is not None and tp_size > 1 - and up_proj.weight.shape[0] % tp_size == 0 + and (self._up_proj_preshard or up_proj.weight.shape[0] % tp_size == 0) and self._shared_experts is not None and not self.moe_config.is_sequence_parallel and self.routed_scaling_factor == 1.0 ) if self._tail_shardable: assert up_proj is not None - self._up_proj_shard_size = up_proj.weight.shape[0] // tp_size + self._up_proj_shard_size = ( + up_proj.weight.shape[0] + if self._up_proj_preshard + else up_proj.weight.shape[0] // tp_size + ) else: logger.warning_once( "K3 latent-MoE tail is not shardable under this config, " @@ -80,7 +85,12 @@ class ROCmLatentMoERunner(MoERunner): shard_size = self._up_proj_shard_size shard_start = get_tensor_model_parallel_rank() * shard_size - up_proj_shard = transform.up_proj.weight.narrow(0, shard_start, shard_size) + weight = transform.up_proj.weight + up_proj_shard = ( + weight + if self._up_proj_preshard + else weight.narrow(0, shard_start, shard_size) + ) hidden_shard = shared_output.narrow(-1, shard_start, shard_size) # hidden_shard += latent @ up_proj_shard.T, accumulated in the GEMM's diff --git a/vllm/models/kimi_k3/amd/linear.py b/vllm/models/kimi_k3/amd/linear.py index 65dad661ab..6827731893 100644 --- a/vllm/models/kimi_k3/amd/linear.py +++ b/vllm/models/kimi_k3/amd/linear.py @@ -2,11 +2,12 @@ # SPDX-FileCopyrightText: Copyright contributors to the vLLM project from collections.abc import Iterable -from typing import Any +from typing import Any, cast import torch from torch import nn +from vllm import envs from vllm.config import CacheConfig, VllmConfig from vllm.distributed import ( get_pp_group, @@ -19,6 +20,10 @@ from vllm.model_executor.layers.fused_moe import ( fused_moe_make_expert_params_mapping, ) from vllm.model_executor.layers.fused_moe.router.gate_linear import GateLinear +from vllm.model_executor.layers.fusion.quant_activation import ( + QuantizedActivation, + get_input_quant_key, +) from vllm.model_executor.layers.layernorm import RMSNorm from vllm.model_executor.layers.linear import ( ColumnParallelLinear, @@ -38,6 +43,10 @@ from vllm.model_executor.layers.mamba.mamba_utils import ( ) from vllm.model_executor.layers.mla import MLAModules from vllm.model_executor.layers.quantization.base_config import QuantizationConfig +from vllm.model_executor.layers.quantization.utils.quant_utils import ( + QuantKey, + kFp8DynamicTokenSym, +) from vllm.model_executor.layers.vocab_parallel_embedding import ( ParallelLMHead, VocabParallelEmbedding, @@ -114,6 +123,9 @@ class KimiMLP(nn.Module): "Only silu and situ are supported." ) + def get_input_quant_key(self) -> QuantKey | None: + return get_input_quant_key(self.gate_up_proj) + def forward(self, x): gate_up, _ = self.gate_up_proj(x) x = self.act_fn(gate_up) @@ -125,11 +137,13 @@ class KimiRoutedOutputTransform(nn.Module): def __init__( self, norm: RMSNorm | None, - up_proj: ReplicatedLinear, + up_proj: ReplicatedLinear | ColumnParallelLinear, + row_sharded: bool = False, ) -> None: super().__init__() self.norm = norm self.up_proj = up_proj + self.row_sharded = row_sharded def forward(self, hidden_states: torch.Tensor) -> torch.Tensor: if self.norm is not None: @@ -148,8 +162,10 @@ def _apply_attn_res( delta: torch.Tensor | None = None, output_norm: RMSNorm | None = None, block_write_idx: int = -1, -) -> torch.Tensor: - return attn_res( + quant_key: QuantKey | None = None, +) -> torch.Tensor | QuantizedActivation: + fuse_quant = quant_key == kFp8DynamicTokenSym and not envs.VLLM_BATCH_INVARIANT + result = attn_res( prefix_sum, delta, block_residual, @@ -160,7 +176,18 @@ def _apply_attn_res( block_write_idx, norm.variance_epsilon, 0.0 if output_norm is None else output_norm.variance_epsilon, + quant_dtype=cast(torch.dtype, kFp8DynamicTokenSym.dtype) + if fuse_quant + else None, ) + if fuse_quant: + assert isinstance(result, tuple) + data, scale = result + return QuantizedActivation( + data, scale, prefix_sum.dtype, prefix_sum.shape, kFp8DynamicTokenSym + ) + assert isinstance(result, torch.Tensor) + return result class KimiMoE(nn.Module): @@ -240,7 +267,7 @@ class KimiMoE(nn.Module): self.routed_expert_down_proj: ReplicatedLinear | None self.routed_expert_norm: RMSNorm | None - self.routed_expert_up_proj: ReplicatedLinear | None + self.routed_expert_up_proj: ReplicatedLinear | ColumnParallelLinear | None self.routed_output_transform: KimiRoutedOutputTransform | None if self.use_latent_moe: self.routed_expert_down_proj = ReplicatedLinear( @@ -255,15 +282,36 @@ class KimiMoE(nn.Module): if self.latent_moe_use_norm else None ) - self.routed_expert_up_proj = ReplicatedLinear( - self.moe_hidden_size, - hidden_size, - bias=False, - quant_config=None, - prefix=f"{prefix}.routed_expert_up_proj", + shard_up_proj = ( + self.tp_size > 1 + and hidden_size % self.tp_size == 0 + and self.num_shared_experts is not None + and self.routed_scaling_factor == 1.0 ) + if shard_up_proj: + # gather_output is a fallback restoring the full hidden dim for + # the unfused tail. The fused tail reads from .weight directly + # so it is not triggered + self.routed_expert_up_proj = ColumnParallelLinear( + self.moe_hidden_size, + hidden_size, + bias=False, + gather_output=True, + quant_config=None, + prefix=f"{prefix}.routed_expert_up_proj", + ) + else: + self.routed_expert_up_proj = ReplicatedLinear( + self.moe_hidden_size, + hidden_size, + bias=False, + quant_config=None, + prefix=f"{prefix}.routed_expert_up_proj", + ) self.routed_output_transform = KimiRoutedOutputTransform( - self.routed_expert_norm, self.routed_expert_up_proj + self.routed_expert_norm, + self.routed_expert_up_proj, + row_sharded=shard_up_proj, ) else: self.routed_expert_down_proj = None @@ -452,10 +500,13 @@ class KimiMLAAttention(nn.Module): prefix, ) + def get_input_quant_key(self) -> QuantKey | None: + return self.mla_attn.get_input_quant_key() + def forward( self, positions: torch.Tensor, - hidden_states: torch.Tensor, + hidden_states: torch.Tensor | QuantizedActivation, ) -> torch.Tensor: return self.mla_attn(positions, hidden_states) @@ -579,7 +630,7 @@ class KimiDecoderLayer(nn.Module): def _run_self_attn( self, positions: torch.Tensor, - hidden_states: torch.Tensor, + hidden_states: torch.Tensor | QuantizedActivation, ) -> torch.Tensor: return self.self_attn( hidden_states=hidden_states, @@ -626,6 +677,11 @@ class KimiDecoderLayer(nn.Module): prefix_delta: torch.Tensor | None, ) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]: prefix_sum = hidden_states + attention_quant_key = ( + self.self_attn.get_input_quant_key() + if isinstance(self.self_attn, (KimiK3DeltaAttention, KimiMLAAttention)) + else None + ) hidden_states = _apply_attn_res( prefix_sum, block_residual, @@ -635,6 +691,7 @@ class KimiDecoderLayer(nn.Module): delta=prefix_delta, output_norm=self.input_layernorm, block_write_idx=(self.block_write_idx if self.is_block_write_layer else -1), + quant_key=attention_quant_key, ) if self.is_block_write_layer: @@ -659,6 +716,11 @@ class KimiDecoderLayer(nn.Module): mlp_valid_blocks, delta=prefix_delta, output_norm=self.post_attention_layernorm, + quant_key=( + self.mlp.get_input_quant_key() + if isinstance(self.mlp, KimiMLP) + else None + ), ) hidden_states = self.mlp(hidden_states) diff --git a/vllm/models/kimi_k3/amd/mla.py b/vllm/models/kimi_k3/amd/mla.py index 11df7d035f..1e4a51e8bf 100644 --- a/vllm/models/kimi_k3/amd/mla.py +++ b/vllm/models/kimi_k3/amd/mla.py @@ -8,8 +8,13 @@ import torch from vllm._aiter_ops import rocm_aiter_ops from vllm.model_executor.layers.attention.attention import get_attention_context +from vllm.model_executor.layers.fusion.quant_activation import ( + QuantizedActivation, + get_input_quant_key, +) from vllm.model_executor.layers.layernorm import RMSNorm from vllm.model_executor.layers.mla import MultiHeadLatentAttentionWrapper +from vllm.model_executor.layers.quantization.utils.quant_utils import QuantKey from vllm.platforms import current_platform _OPT_KV_LORA_RANK = 512 @@ -49,6 +54,21 @@ class KimiK3MultiHeadLatentAttentionWrapper(MultiHeadLatentAttentionWrapper): not attn.is_aiter_triton_fp8_bmm_enabled and attn.W_UK_T is not None ) + def get_input_quant_key(self) -> QuantKey | None: + if self.indexer is not None: + return None + consumers = ( + [self.fused_qkv_a_proj] + if self.q_lora_rank is not None + else [self.kv_a_proj_with_mqa, self.q_proj] + ) + if self.g_proj is not None: + consumers.append(self.g_proj) + if any(layer is None for layer in consumers): + return None + keys = [get_input_quant_key(layer) for layer in consumers if layer is not None] + return keys[0] if all(key == keys[0] for key in keys) else None + def _normalize_q_kv( self, q_c: torch.Tensor, @@ -173,10 +193,15 @@ class KimiK3MultiHeadLatentAttentionWrapper(MultiHeadLatentAttentionWrapper): def forward( self, positions: torch.Tensor, - hidden_states: torch.Tensor, + hidden_states: torch.Tensor | QuantizedActivation, llama_4_scaling: torch.Tensor | None = None, ) -> torch.Tensor: q_c = None + num_tokens = ( + hidden_states.orig_shape[0] + if isinstance(hidden_states, QuantizedActivation) + else hidden_states.shape[0] + ) if self.q_lora_rank is not None: assert self.fused_qkv_a_proj is not None, ( @@ -238,7 +263,7 @@ class KimiK3MultiHeadLatentAttentionWrapper(MultiHeadLatentAttentionWrapper): if self.dcp_q_replicate: q_dcp_replicated, q = q, q_proj_layer._local_view(q) - output_shape = (hidden_states.shape[0], self.num_heads * self.v_head_dim) + output_shape = (num_tokens, self.num_heads * self.v_head_dim) # The fused decode path covers only the MQA slice, so it is taken for # decode-only batches; anything else falls through to MLAAttention. diff --git a/vllm/models/kimi_k3/amd/model.py b/vllm/models/kimi_k3/amd/model.py index ec49891ddb..124fca3e33 100644 --- a/vllm/models/kimi_k3/amd/model.py +++ b/vllm/models/kimi_k3/amd/model.py @@ -9,6 +9,7 @@ import torch from torch import nn from vllm.config import VllmConfig +from vllm.model_executor.layers.fusion.mm_input_norm import build_mm_input_norm from vllm.model_executor.layers.quantization import QuantizationConfig from vllm.model_executor.layers.quantization.compressed_tensors import ( compressed_tensors, @@ -65,6 +66,7 @@ class KimiK3ForConditionalGeneration( """Kimi-K3 model with Kimi-K2.5 vision and KimiLinear text.""" supports_encoder_tp_data = True + supports_mm_device_do_normalize = True hf_to_vllm_mapper = WeightsMapper( orig_to_new_prefix={ @@ -103,6 +105,7 @@ class KimiK3ForConditionalGeneration( self.vision_tower = MoonViT3dPretrainedModel( config.vision_config, quant_config=self._maybe_ignore_quant_config(quant_config), + input_norm=build_mm_input_norm(vllm_config.model_config), prefix=maybe_prefix(prefix, "vision_tower"), ) if self._maybe_ignore_quant_config(quant_config) is not None: @@ -114,6 +117,7 @@ class KimiK3ForConditionalGeneration( self.mm_projector = KimiK25MultiModalProjector( config=config.vision_config, + out_hidden_size=config.text_config.hidden_size, use_data_parallel=self.use_data_parallel, quant_config=self._maybe_ignore_quant_config(quant_config), prefix=maybe_prefix(prefix, "mm_projector"), @@ -163,8 +167,6 @@ class KimiK3ForConditionalGeneration( pixel_values.shape[0] * pixel_values.shape[1], *pixel_values.shape[2:] ) - target_dtype = next(self.vision_tower.parameters()).dtype - pixel_values = pixel_values.to(target_dtype) assert isinstance(grid_thws, torch.Tensor), ( f"expect grid_thws to be a tensor, got {type(grid_thws)}" ) diff --git a/vllm/models/kimi_k3/amd/ops/attn_res.py b/vllm/models/kimi_k3/amd/ops/attn_res.py index 5802e60367..95f748e72d 100644 --- a/vllm/models/kimi_k3/amd/ops/attn_res.py +++ b/vllm/models/kimi_k3/amd/ops/attn_res.py @@ -10,6 +10,7 @@ import torch +from vllm.model_executor.layers.quantization.utils.quant_utils import get_fp8_min_max from vllm.triton_utils import tl, triton @@ -22,6 +23,7 @@ def _attn_res_kernel( qk_weight_ptr, output_norm_weight_ptr, output_ptr, + output_scale_ptr, stride_prefix_m: tl.constexpr, stride_delta_m: tl.constexpr, stride_block_m: tl.constexpr, @@ -35,6 +37,7 @@ def _attn_res_kernel( HAS_DELTA: tl.constexpr, WRITE_BLOCK: tl.constexpr, APPLY_OUTPUT_NORM: tl.constexpr, + QUANT_MAX: tl.constexpr, BLOCK_L: tl.constexpr, BLOCK_D: tl.constexpr, ): @@ -129,6 +132,14 @@ def _attn_res_kernel( output_norm_weight_ptr + d_offsets, mask=d_mask, other=0.0 ).to(tl.float32) output = mixed * output_reciprocal_std * output_norm_weight + if QUANT_MAX > 0: + # Preserve the rounding of the original BF16 output before quantizing. + output = output.to(prefix_ptr.dtype.element_ty).to(tl.float32) + amax = tl.max(tl.where(d_mask, tl.abs(output), 0.0), axis=0) + scale = tl.maximum(amax, 1e-10) * (1.0 / QUANT_MAX) + inv_scale = 1.0 / scale + output = tl.minimum(tl.maximum(output * inv_scale, -QUANT_MAX), QUANT_MAX) + tl.store(output_scale_ptr + row_idx, scale) tl.store( output_ptr + row_idx * stride_output_m + d_offsets, output, @@ -147,7 +158,9 @@ def attn_res( block_write_idx: int, eps: float, output_norm_eps: float, -) -> torch.Tensor: + *, + quant_dtype: torch.dtype | None = None, +) -> torch.Tensor | tuple[torch.Tensor, torch.Tensor]: num_tokens, hidden_size = prefix.shape assert prefix.stride(-1) == 1 assert delta is None or delta.stride(-1) == 1 @@ -155,9 +168,18 @@ def attn_res( assert norm_weight.stride(-1) == 1 assert qk_weight.stride(-1) == 1 assert output_norm_weight is None or output_norm_weight.stride(-1) == 1 - output = prefix.new_empty(prefix.shape) + if quant_dtype is not None: + assert quant_dtype in (torch.float8_e4m3fn, torch.float8_e4m3fnuz) + output = torch.empty_like( + prefix, dtype=quant_dtype or prefix.dtype, memory_format=torch.contiguous_format + ) + scale = ( + torch.empty((num_tokens, 1), device=prefix.device, dtype=torch.float32) + if quant_dtype is not None + else None + ) if num_tokens == 0: - return output + return output if scale is None else (output, scale) # Source tiling helps decode, while one-source tiles scale better for prefill. if num_tokens >= 256 or num_blocks <= 1: @@ -172,6 +194,7 @@ def attn_res( qk_weight, output_norm_weight, output, + scale, prefix.stride(0), 0 if delta is None else delta.stride(0), blocks.stride(0), @@ -185,9 +208,10 @@ def attn_res( HAS_DELTA=delta is not None, WRITE_BLOCK=block_write_idx >= 0, APPLY_OUTPUT_NORM=output_norm_weight is not None, + QUANT_MAX=0.0 if quant_dtype is None else get_fp8_min_max()[1], BLOCK_L=block_l, BLOCK_D=triton.next_power_of_2(hidden_size), num_warps=num_warps, num_stages=2, ) - return output + return output if scale is None else (output, scale) diff --git a/vllm/models/kimi_k3/amd/ops/third_party/kda/chunk_intra.py b/vllm/models/kimi_k3/amd/ops/third_party/kda/chunk_intra.py index 0c600f3ba3..bc13318d83 100644 --- a/vllm/models/kimi_k3/amd/ops/third_party/kda/chunk_intra.py +++ b/vllm/models/kimi_k3/amd/ops/third_party/kda/chunk_intra.py @@ -349,7 +349,7 @@ def chunk_kda_fwd_kernel_inter_solve_fused( for num_warps in [1, 2, 4, 8] for num_stages in [2, 3, 4] ], - key=["BK", "NC", "BT", "HV"], + key=["BK", "BT", "HV"], ) @triton.jit(do_not_specialize=["B", "T"]) def chunk_kda_fwd_kernel_intra_sub_chunk( diff --git a/vllm/models/kimi_k3/common/mm_preprocess.py b/vllm/models/kimi_k3/common/mm_preprocess.py index dfd90fe9a9..f9eb142031 100644 --- a/vllm/models/kimi_k3/common/mm_preprocess.py +++ b/vllm/models/kimi_k3/common/mm_preprocess.py @@ -30,6 +30,9 @@ from vllm.multimodal.processing import ( from vllm.transformers_utils.configs.kimi_k3 import KimiK3Config from vllm.transformers_utils.processor import cached_get_image_processor from vllm.transformers_utils.processors.kimi_k3 import KimiK3Processor +from vllm.transformers_utils.processors.kimi_k25_vision_fused import ( + KimiK25FusedVisionProcessor, +) logger = init_logger(__name__) @@ -106,6 +109,7 @@ class KimiK3ProcessingInfo(BaseProcessingInfo): self.ctx.model_config.model, revision=self.ctx.model_config.revision, trust_remote_code=self.ctx.model_config.trust_remote_code, + processor_cls_overrides=KimiK25FusedVisionProcessor, ) # Resolve token ID from the tokenizer because transformers v5 diff --git a/vllm/models/kimi_k3/nvidia/kda_metadata.py b/vllm/models/kimi_k3/nvidia/kda_metadata.py index fe361c0895..6eb7dfd05e 100644 --- a/vllm/models/kimi_k3/nvidia/kda_metadata.py +++ b/vllm/models/kimi_k3/nvidia/kda_metadata.py @@ -409,20 +409,12 @@ class KimiK3KDAMetadataBuilder(GDNAttentionMetadataBuilder): spec_sequence_masks_cpu &= ( query_start_loc_cpu.diff() <= self.num_spec + 1 ) - # Native KDA can use its regular decode path when no draft token - # was scheduled. RecoverSSM must preserve its extended conv window. - if ( - not self.use_recoverssm - and num_decode_draft_tokens_cpu[spec_sequence_masks_cpu].sum().item() - == 0 - ): + # A batch whose rows all drafted nothing still has to run the + # speculative path: that is the only path that applies each row's + # accepted-token offset to the recurrent state. + num_spec_decodes = spec_sequence_masks_cpu.sum().item() + if num_spec_decodes == 0: spec_sequence_masks_cpu = None - num_spec_decodes = 0 - active_non_spec_mask_cpu = None - else: - num_spec_decodes = spec_sequence_masks_cpu.sum().item() - if num_spec_decodes == 0: - spec_sequence_masks_cpu = None spec_request_indices = None spec_token_start = None diff --git a/vllm/models/kimi_k3/nvidia/mla.py b/vllm/models/kimi_k3/nvidia/mla.py index 188f72e660..18fd93579b 100644 --- a/vllm/models/kimi_k3/nvidia/mla.py +++ b/vllm/models/kimi_k3/nvidia/mla.py @@ -422,6 +422,7 @@ class MultiHeadLatentAttention(nn.Module, AttentionLayerBase): return MLAAttentionSpec( # type: ignore[call-arg] block_size=vllm_config.cache_config.block_size, num_kv_heads=1, + max_tp_shards=1, head_size=self.head_size, dtype=kv_cache_dtype, cache_dtype_str=self.kv_cache_dtype, @@ -785,6 +786,15 @@ class MultiHeadLatentAttention(nn.Module, AttentionLayerBase): cos_sin_cache=cos_sin_cache, ) + def _fused_mla_kv_concat( + self, kv_nope: torch.Tensor, k_pe: torch.Tensor, use_fp8_prefill: bool + ) -> tuple[torch.Tensor, torch.Tensor]: + """Pack one DCP context chunk's ``(k, v)`` with the fused kernels.""" + k_nope, v = kv_nope.split([self.qk_nope_head_dim, self.v_head_dim], dim=-1) + if use_fp8_prefill: + return fused_mla_kv_concat_quant_fp8(k_nope, k_pe, v) + return fused_mla_kv_concat(k_nope, k_pe), v + def _compute_prefill_context( self, q: torch.Tensor, @@ -814,8 +824,8 @@ class MultiHeadLatentAttention(nn.Module, AttentionLayerBase): copied per chunk. Decode context parallelism keeps using - ``impl._context_parallel_compute_prefill_context``; its extra allgather - and reorg are not fused here. + ``impl._context_parallel_compute_prefill_context`` for its allgather and + reorg, with this layer's ``_fused_mla_kv_concat`` as its per-chunk pack. """ prefill = attn_metadata.prefill assert prefill is not None @@ -1074,6 +1084,7 @@ class MultiHeadLatentAttention(nn.Module, AttentionLayerBase): attn_metadata, k_scale=self._k_scale, dcp_world_size=self.dcp_world_size, + fused_mla_kv_concat_fn=self._fused_mla_kv_concat, ) ) else: diff --git a/vllm/models/kimi_k3/nvidia/model.py b/vllm/models/kimi_k3/nvidia/model.py index d4ed6d2e73..ced005d87c 100644 --- a/vllm/models/kimi_k3/nvidia/model.py +++ b/vllm/models/kimi_k3/nvidia/model.py @@ -30,6 +30,7 @@ from vllm.model_executor.layers.fused_moe.router.gate_linear import GateLinear from vllm.model_executor.layers.fused_moe.router.grouped_topk_router import ( fused_grouped_topk, ) +from vllm.model_executor.layers.fusion.mm_input_norm import build_mm_input_norm from vllm.model_executor.layers.layernorm import RMSNorm from vllm.model_executor.layers.linear import ( MergedColumnParallelLinear, @@ -155,6 +156,20 @@ def shard_sequence_parallel_mlp( ) +def can_shard_sequence_parallel_shared_expert(vllm_config: VllmConfig) -> bool: + """Whether the active MoE backend preserves the SP shared-expert contract. + + MegaMoE invokes the shared expert directly. DeepEP v2 invokes it through + ``MoERunner``, but its EP combine returns the routed output in the local + sequence-sharded layout, so the shared expert can use the same + all-gather/partial-GEMM/reduce-scatter implementation. + """ + return ( + vllm_config.kernel_config.moe_backend == "deep_gemm_mega_moe" + or vllm_config.parallel_config.all2all_backend == "deepep_v2" + ) + + def maybe_init_gemm_rs_ar(vllm_config: VllmConfig, use_sequence_parallel: bool) -> bool: # Both feature flags may be enabled; the worker's static SP topology binds # its singleton to exactly one mode. @@ -364,27 +379,16 @@ class KimiK3MegaMoEExperts(DeepseekV4MegaMoEExperts): return self._check_runtime_supported() - from vllm.utils.deep_gemm import _import_deep_gemm - - deep_gemm = _import_deep_gemm() - w13_scale = deep_gemm.transform_sf_into_required_layout( - self._ue8m0_uint8_to_float(self.w13_weight_scale.data).contiguous(), - 2 * self.intermediate_size, - self.hidden_size, - (1, 32), - self.num_local_experts, - ) - w2_scale = deep_gemm.transform_sf_into_required_layout( - self._ue8m0_uint8_to_float(self.w2_weight_scale.data).contiguous(), - self.hidden_size, - self.intermediate_size, - (1, 32), - self.num_local_experts, - ) + backend = self._ensure_backend() self._transformed_l1_weights, self._transformed_l2_weights = ( - deep_gemm.transform_weights_for_mega_moe( - (self.w13_weight.data.view(torch.int8).contiguous(), w13_scale), - (self.w2_weight.data.view(torch.int8).contiguous(), w2_scale), + backend.transform_weights( + w13_weight=self.w13_weight.data, + w13_weight_scale=self.w13_weight_scale.data, + w2_weight=self.w2_weight.data, + w2_weight_scale=self.w2_weight_scale.data, + num_local_experts=self.num_local_experts, + hidden_size=self.hidden_size, + intermediate_size=self.intermediate_size, activation=self.activation, ) ) @@ -406,6 +410,7 @@ class KimiK3MegaMoEExperts(DeepseekV4MegaMoEExperts): from vllm.utils.deep_gemm import _import_deep_gemm deep_gemm = _import_deep_gemm() + backend = self._ensure_backend() group = get_ep_group().device_group device = torch.accelerator.current_device_index() key = ( @@ -417,6 +422,7 @@ class KimiK3MegaMoEExperts(DeepseekV4MegaMoEExperts): self.hidden_size, self.intermediate_size, self.activation, + backend.mma_type, ) symm_buffer = self._kimi_symm_buffer_cache.get(key) if symm_buffer is None: @@ -428,6 +434,7 @@ class KimiK3MegaMoEExperts(DeepseekV4MegaMoEExperts): self.hidden_size, self.intermediate_size, activation=self.activation, + mma_type=backend.mma_type, ) self._kimi_symm_buffer_cache[key] = symm_buffer return symm_buffer @@ -448,10 +455,6 @@ class KimiK3MegaMoEExperts(DeepseekV4MegaMoEExperts): f"but its symmetric buffer supports {self.max_num_tokens}." ) y = torch.empty_like(hidden_states, dtype=torch.bfloat16) - from vllm.utils.deep_gemm import _import_deep_gemm - - deep_gemm = _import_deep_gemm() - symm_buffer = self.get_symm_buffer() num_tokens = hidden_states.shape[0] is_padding = None if envs.VLLM_MOE_SKIP_PADDING and is_forward_context_available(): @@ -482,6 +485,8 @@ class KimiK3MegaMoEExperts(DeepseekV4MegaMoEExperts): else None, ) + backend = self._ensure_backend() + symm_buffer = self.get_symm_buffer() prepare_megamoe_inputs( hidden_states, topk_weights, @@ -491,22 +496,23 @@ class KimiK3MegaMoEExperts(DeepseekV4MegaMoEExperts): symm_buffer.topk_idx[:num_tokens], symm_buffer.topk_weights[:num_tokens], is_padding=is_padding, + hidden_quant=backend.hidden_quant, ) self.finalize_weights() assert self._transformed_l1_weights is not None assert self._transformed_l2_weights is not None - deep_gemm.fp8_fp4_mega_moe( - y, - self._transformed_l1_weights, - self._transformed_l2_weights, - symm_buffer, + backend.run_mega_moe( + y=y, + l1_weights=self._transformed_l1_weights, + l2_weights=self._transformed_l2_weights, + symm_buffer=symm_buffer, activation_clamp=activation_clamp, - activation=self.activation, + fast_math=fast_math, # DeepGEMM names the SiTU gate tanh scale `activation_alpha` and the # linear/up tanh scale `activation_beta`; beta=0 leaves up untouched. + activation=self.activation, activation_alpha=self.activation_beta or 1.0, activation_beta=self.activation_linear_beta or 0.0, - fast_math=fast_math, ) return y @@ -625,10 +631,12 @@ class KimiMoE(nn.Module): quant_config=quant_config, reduce_results=False, use_sequence_parallel=use_sequence_parallel, - # Only the MegaMoE path calls the shared experts directly; the - # FusedMoE path below hands them to the runner, which fuses - # their reduction and assumes the replicated layout. - can_shard_sequence_parallel=self.use_mega_moe, + # MegaMoE calls the shared expert directly. DeepEP v2 hands it + # to MoERunner, but preserves the local SP output layout needed + # by the sharded shared-expert path. + can_shard_sequence_parallel=can_shard_sequence_parallel_shared_expert( + vllm_config + ), run_gemm_rs_ar=run_gemm_rs_ar, prefix=f"{prefix}.shared_experts", activation_situ_beta=activation_situ_beta, @@ -1736,6 +1744,7 @@ class KimiK3ForConditionalGeneration( """Kimi-K3 model with Kimi-K2.5 vision and KimiLinear text.""" supports_encoder_tp_data = True + supports_mm_device_do_normalize = True hf_to_vllm_mapper = WeightsMapper( orig_to_new_prefix={ @@ -1775,6 +1784,7 @@ class KimiK3ForConditionalGeneration( self.vision_tower = MoonViT3dPretrainedModel( config.vision_config, quant_config=self._maybe_ignore_quant_config(quant_config), + input_norm=build_mm_input_norm(vllm_config.model_config), prefix=maybe_prefix(prefix, "vision_tower"), ) if is_meta_module(self.vision_tower): @@ -1815,6 +1825,7 @@ class KimiK3ForConditionalGeneration( self.mm_projector = KimiK25MultiModalProjector( config=config.vision_config, + out_hidden_size=config.text_config.hidden_size, use_data_parallel=self.use_data_parallel, quant_config=self._maybe_ignore_quant_config(quant_config), prefix=maybe_prefix(prefix, "mm_projector"), @@ -1964,13 +1975,13 @@ class KimiK3ForConditionalGeneration( if isinstance(patch_size, int): patch_size = (patch_size, patch_size) total_patches = sum(t * h * w for t, h, w in grid_thws) - pixel_values = torch.randn( + pixel_values = torch.zeros( total_patches, 3, patch_size[0], patch_size[1], device=device, - dtype=dtype, + dtype=self.vision_tower.patch_embed.input_norm.input_dtype or dtype, ) metadata = self.vision_tower.prepare_encoder_cudagraph_metadata( grid_thws, @@ -2028,9 +2039,7 @@ class KimiK3ForConditionalGeneration( path: str = "default", ) -> torch.Tensor: image_features = self.vision_tower( - self._get_pixel_values(mm_kwargs).to( - next(self.vision_tower.parameters()).dtype - ), + self._get_pixel_values(mm_kwargs), self._get_grid_thws(mm_kwargs), ) return self._project_encoder_features(torch.cat(image_features)) @@ -2063,8 +2072,6 @@ class KimiK3ForConditionalGeneration( pixel_values.shape[0] * pixel_values.shape[1], *pixel_values.shape[2:] ) - target_dtype = next(self.vision_tower.parameters()).dtype - pixel_values = pixel_values.to(target_dtype) assert isinstance(grid_thws, torch.Tensor), ( f"expect grid_thws to be a tensor, got {type(grid_thws)}" ) diff --git a/vllm/models/kimi_k3/nvidia/ops/third_party/kda/chunk_intra.py b/vllm/models/kimi_k3/nvidia/ops/third_party/kda/chunk_intra.py index 8b9aa86f9e..07b4a38893 100644 --- a/vllm/models/kimi_k3/nvidia/ops/third_party/kda/chunk_intra.py +++ b/vllm/models/kimi_k3/nvidia/ops/third_party/kda/chunk_intra.py @@ -451,7 +451,7 @@ def chunk_kda_fwd_kernel_inter_solve_fused( for num_warps in [1, 2, 4, 8] for num_stages in [2, 3, 4] ], - key=["BK", "NC", "BT", "HV"], + key=["BK", "BT", "HV"], ) @triton.jit(do_not_specialize=["B", "T"]) def chunk_kda_fwd_kernel_intra_sub_chunk( diff --git a/vllm/models/minimax_m3/amd/model.py b/vllm/models/minimax_m3/amd/model.py index 9105f97af8..3bf8928603 100644 --- a/vllm/models/minimax_m3/amd/model.py +++ b/vllm/models/minimax_m3/amd/model.py @@ -465,6 +465,7 @@ class MiniMaxM3MoE(nn.Module): self.n_shared_experts if self.is_fused_shared_expert_enabled else None ), fuse_shared_experts=self.is_fused_shared_expert_enabled, + shared_expert_prefix=f"{prefix}.shared_experts", quant_config=quant_config, prefix=f"{prefix}.experts", ) diff --git a/vllm/models/minimax_m3/amd/ops/index_topk.py b/vllm/models/minimax_m3/amd/ops/index_topk.py index ce27ea1220..d2b4602c56 100644 --- a/vllm/models/minimax_m3/amd/ops/index_topk.py +++ b/vllm/models/minimax_m3/amd/ops/index_topk.py @@ -229,15 +229,16 @@ def _index_block_score_kernel( if BLOCK_SIZE_Q * pid_q >= q_len: return - q_ptrs = tl.make_block_ptr( + q_desc = tl.make_tensor_descriptor( base=q_ptr + seq_start * stride_q_n + pid_h * stride_q_h, - shape=(q_len, head_dim), - strides=(stride_q_n, stride_q_d), - offsets=(pid_q * BLOCK_SIZE_Q, 0), - block_shape=(BLOCK_SIZE_Q, head_dim), - order=(1, 0), + shape=[q_len, head_dim], + strides=[stride_q_n, stride_q_d], + block_shape=[BLOCK_SIZE_Q, head_dim], + padding_option="zero", ) - q = tl.load(q_ptrs, boundary_check=(0,), padding_option="zero") + + # 2. Perform the load by providing the multi-dimensional offset directly + q = q_desc.load([pid_q * BLOCK_SIZE_Q, 0]) q_start = prefix_len + pid_q * BLOCK_SIZE_Q off_q = tl.arange(0, BLOCK_SIZE_Q) + pid_q * BLOCK_SIZE_Q + prefix_len diff --git a/vllm/models/minimax_m3/amd/ops/sparse_attn.py b/vllm/models/minimax_m3/amd/ops/sparse_attn.py index 0719374496..b4355d18f6 100644 --- a/vllm/models/minimax_m3/amd/ops/sparse_attn.py +++ b/vllm/models/minimax_m3/amd/ops/sparse_attn.py @@ -139,15 +139,15 @@ def _gqa_sparse_fwd_kernel( off_t = tl.arange(0, BLOCK_SIZE_T) topk_idx = tl.load(t_ptr_j + off_t * stride_tk, mask=off_t < max_topk, other=-1) real_topk = tl.sum((topk_idx >= 0).to(tl.int32), axis=0) - q_ptrs = tl.make_block_ptr( + q_desc = tl.make_tensor_descriptor( base=q_ptr + q_start * stride_qn + pid_h * stride_qh, - shape=(q_len, gqa_group_size, head_dim), - strides=(stride_qn, stride_qh, stride_qd), - offsets=(pid_q_j * BLOCK_SIZE_Q, 0, 0), - block_shape=(BLOCK_SIZE_Q, BLOCK_SIZE_H, BLOCK_SIZE_D), - order=(2, 1, 0), + shape=[q_len, gqa_group_size, head_dim], + strides=[stride_qn, stride_qh, stride_qd], + block_shape=[BLOCK_SIZE_Q, BLOCK_SIZE_H, BLOCK_SIZE_D], + padding_option="zero", ) - q = tl.load(q_ptrs, boundary_check=(0, 1, 2), padding_option="zero") + + q = q_desc.load([pid_q_j * BLOCK_SIZE_Q, 0, 0]) m_i = tl.full((BLOCK_SIZE_QH,), float("-inf"), dtype=tl.float32) lse_i = tl.full((BLOCK_SIZE_QH,), float("-inf"), dtype=tl.float32) acc_o = tl.zeros((BLOCK_SIZE_QH, BLOCK_SIZE_D), dtype=tl.float32) @@ -228,15 +228,17 @@ def _gqa_sparse_fwd_kernel( lse_i = m_ij + tl.log2(tl.exp2(lse_i - m_ij) + l_ij) acc_o = acc_o * tl.exp2(m_i - lse_i)[:, None] acc_o = tl.reshape(acc_o, BLOCK_SIZE_Q, BLOCK_SIZE_H, BLOCK_SIZE_D) - o_ptrs = tl.make_block_ptr( + o_desc = tl.make_tensor_descriptor( base=o_ptr + q_start * stride_on + pid_h * stride_oh, - shape=(q_len, gqa_group_size, head_dim), - strides=(stride_on, stride_oh, stride_od), - offsets=(pid_q_j * BLOCK_SIZE_Q, 0, 0), - block_shape=(BLOCK_SIZE_Q, BLOCK_SIZE_H, BLOCK_SIZE_D), - order=(2, 1, 0), + shape=[q_len, gqa_group_size, head_dim], + strides=[stride_on, stride_oh, stride_od], + block_shape=[BLOCK_SIZE_Q, BLOCK_SIZE_H, BLOCK_SIZE_D], + ) + + o_desc.store( + offsets=[pid_q_j * BLOCK_SIZE_Q, 0, 0], + value=acc_o.to(o_ptr.dtype.element_ty), ) - tl.store(o_ptrs, acc_o.to(o_ptr.dtype.element_ty), boundary_check=(0, 1, 2)) @torch.no_grad() diff --git a/vllm/models/minimax_m3/common/ops/sparse_attn.py b/vllm/models/minimax_m3/common/ops/sparse_attn.py index af703c0862..6426115ae3 100644 --- a/vllm/models/minimax_m3/common/ops/sparse_attn.py +++ b/vllm/models/minimax_m3/common/ops/sparse_attn.py @@ -110,7 +110,9 @@ def _gqa_sparse_fwd_kernel( bt_row = block_table_ptr + pid_b * stride_bt_b off_n = tl.arange(0, BLOCK_SIZE_K) off_d = tl.arange(0, BLOCK_SIZE_D) + off_h = tl.arange(0, BLOCK_SIZE_H) d_mask = off_d < head_dim + h_mask = off_h < gqa_group_size for j in range(real_q_loop): pid_q_j = pid_q * num_q_loop + j t_ptr_j = t_ptr + (q_block_start + pid_q_j) * stride_tn + pid_kh * stride_th @@ -118,15 +120,20 @@ def _gqa_sparse_fwd_kernel( q_abs = prefix_len + pid_q_j * BLOCK_SIZE_Q valid_blocks = (q_abs + BLOCK_SIZE_K) // BLOCK_SIZE_K real_topk = tl.minimum(max_topk, valid_blocks) - q_ptrs = tl.make_block_ptr( - base=q_ptr + q_start * stride_qn + pid_h * stride_qh, - shape=(q_len, gqa_group_size, head_dim), - strides=(stride_qn, stride_qh, stride_qd), - offsets=(pid_q_j * BLOCK_SIZE_Q, 0, 0), - block_shape=(BLOCK_SIZE_Q, BLOCK_SIZE_H, BLOCK_SIZE_D), - order=(2, 1, 0), + off_qi = pid_q_j * BLOCK_SIZE_Q + tl.arange(0, BLOCK_SIZE_Q) + qo_mask = ( + (off_qi < q_len)[:, None, None] + & h_mask[None, :, None] + & d_mask[None, None, :] + ) + q = tl.load( + q_ptr + + (q_start + off_qi)[:, None, None] * stride_qn + + (pid_h + off_h)[None, :, None] * stride_qh + + off_d[None, None, :] * stride_qd, + mask=qo_mask, + other=0.0, ) - q = tl.load(q_ptrs, boundary_check=(0, 1, 2), padding_option="zero") off_q = ( tl.arange(0, BLOCK_SIZE_Q)[:, None] + pid_q_j * BLOCK_SIZE_Q @@ -203,15 +210,14 @@ def _gqa_sparse_fwd_kernel( lse_i = m_ij + tl.log2(tl.exp2(lse_i - m_ij) + l_ij) acc_o = acc_o * tl.exp2(m_i - lse_i)[:, None] acc_o = tl.reshape(acc_o, BLOCK_SIZE_Q, BLOCK_SIZE_H, BLOCK_SIZE_D) - o_ptrs = tl.make_block_ptr( - base=o_ptr + q_start * stride_on + pid_h * stride_oh, - shape=(q_len, gqa_group_size, head_dim), - strides=(stride_on, stride_oh, stride_od), - offsets=(pid_q_j * BLOCK_SIZE_Q, 0, 0), - block_shape=(BLOCK_SIZE_Q, BLOCK_SIZE_H, BLOCK_SIZE_D), - order=(2, 1, 0), + tl.store( + o_ptr + + (q_start + off_qi)[:, None, None] * stride_on + + (pid_h + off_h)[None, :, None] * stride_oh + + off_d[None, None, :] * stride_od, + acc_o.to(o_ptr.dtype.element_ty), + mask=qo_mask, ) - tl.store(o_ptrs, acc_o.to(o_ptr.dtype.element_ty), boundary_check=(0, 1, 2)) # --------------------------------------------------------------------------- @@ -298,7 +304,7 @@ def _gqa_sparse_decode_kernel( # attention range instead of letting padded rows produce negative lengths. kv_len = tl.maximum(query_pos + 1, 0) - # Valid block count from seq_len (no sentinel): min(topk, cdiv(kv_len, blk)). + # Upper bound on top-k entries from causal length; entries may be sentinels. idx_base = t_ptr + pid_kh * stride_th + pid_b * stride_tn num_blocks = (kv_len + BLOCK_SIZE_K - 1) // BLOCK_SIZE_K real_topk = tl.minimum(max_topk, num_blocks) @@ -306,30 +312,37 @@ def _gqa_sparse_decode_kernel( off_n = tl.arange(0, BLOCK_SIZE_K) off_d = tl.arange(0, BLOCK_SIZE_D) + off_h = tl.arange(0, BLOCK_SIZE_H) d_mask = off_d < head_dim + h_mask = off_h < gqa_group_size + hd_mask = h_mask[:, None] & d_mask[None, :] bt_row = block_table_ptr + req_id * stride_bt_b m_i = tl.full((BLOCK_SIZE_H,), float("-inf"), dtype=tl.float32) lse_i = tl.full((BLOCK_SIZE_H,), float("-inf"), dtype=tl.float32) acc_o = tl.zeros((BLOCK_SIZE_H, BLOCK_SIZE_D), dtype=tl.float32) - q_ptrs = tl.make_block_ptr( - base=q_ptr + pid_b * stride_qn + pid_h * stride_qh, - shape=(gqa_group_size, head_dim), - strides=(stride_qh, stride_qd), - offsets=(0, 0), - block_shape=(BLOCK_SIZE_H, BLOCK_SIZE_D), - order=(1, 0), + q = tl.load( + q_ptr + + pid_b * stride_qn + + (pid_h + off_h)[:, None] * stride_qh + + off_d[None, :] * stride_qd, + mask=hd_mask, + other=0.0, ) - q = tl.load(q_ptrs, boundary_check=(0, 1), padding_option="zero") cur_idx_ptr = idx_base + chunk_start_topk * stride_tk for _ in tl.range(chunk_start_topk, chunk_end_topk): blk = tl.load(cur_idx_ptr).to(tl.int32) cur_idx_ptr = cur_idx_ptr + stride_tk - c = blk * BLOCK_SIZE_K - page = tl.load(bt_row + blk).to(tl.int64) + # topk_idx uses -1 for unused entries. Select a safe logical block + # before dereferencing the block table; invalid and future blocks then + # contribute a neutral online-softmax update without control flow. + valid_blk = (blk >= 0) & (blk < num_blocks) + safe_blk = tl.where(valid_blk, blk, 0) + c = safe_blk * BLOCK_SIZE_K + page = tl.load(bt_row + safe_blk).to(tl.int64) pos = c + off_n - pos_mask = pos < kv_len + pos_mask = (pos < kv_len) & valid_blk k = tl.load( kv_cache_ptr + page * stride_kv_blk @@ -356,9 +369,11 @@ def _gqa_sparse_decode_kernel( qk += tl.where(pos_mask[None, :], 0, float("-inf")) qk += tl.dot(q, k) * sm_scale_log2e m_ij = tl.maximum(m_i, tl.max(qk, axis=1)) - p = tl.exp2(qk - m_ij[:, None]) + m_safe = tl.where(valid_blk, m_ij, 0.0) + p = tl.where(pos_mask[None, :], tl.exp2(qk - m_safe[:, None]), 0.0) l_ij = tl.sum(p, axis=1) - acc_o = acc_o * tl.exp2(m_i - m_ij)[:, None] + alpha = tl.where(valid_blk, tl.exp2(m_i - m_safe), 1.0) + acc_o = acc_o * alpha[:, None] v = tl.load( kv_cache_ptr + page * stride_kv_blk @@ -382,8 +397,9 @@ def _gqa_sparse_decode_kernel( ) v = (v * v_scale[:, None]).to(q.dtype) acc_o += tl.dot(p.to(v.dtype), v) - m_i = m_ij - lse_i = m_ij + tl.log2(tl.exp2(lse_i - m_ij) + l_ij) + lse_ij = m_safe + tl.log2(tl.exp2(lse_i - m_safe) + l_ij) + m_i = tl.where(valid_blk, m_safe, m_i) + lse_i = tl.where(valid_blk, lse_ij, lse_i) if USE_PDL: tl.extra.cuda.gdc_launch_dependents() @@ -392,24 +408,23 @@ def _gqa_sparse_decode_kernel( # can hit 0 * NaN. All-empty padded rows may still produce NaNs in merge. scale = tl.where(lse_i > float("-inf"), tl.exp2(m_i - lse_i), tl.zeros_like(lse_i)) acc_o = acc_o * scale[:, None] - o_ptrs = tl.make_block_ptr( - base=o_ptr + pid_c * stride_o_c + pid_b * stride_o_b + pid_h * stride_o_h, - shape=(gqa_group_size, head_dim), - strides=(stride_o_h, stride_o_d), - offsets=(0, 0), - block_shape=(BLOCK_SIZE_H, BLOCK_SIZE_D), - order=(1, 0), + tl.store( + o_ptr + + pid_c * stride_o_c + + pid_b * stride_o_b + + (pid_h + off_h)[:, None] * stride_o_h + + off_d[None, :] * stride_o_d, + acc_o.to(o_ptr.dtype.element_ty), + mask=hd_mask, ) - tl.store(o_ptrs, acc_o.to(o_ptr.dtype.element_ty), boundary_check=(0, 1)) - lse_ptrs = tl.make_block_ptr( - base=lse_ptr + pid_c * stride_l_c + pid_b * stride_l_b + pid_h * stride_l_h, - shape=(gqa_group_size,), - strides=(stride_l_h,), - offsets=(0,), - block_shape=(BLOCK_SIZE_H,), - order=(0,), + tl.store( + lse_ptr + + pid_c * stride_l_c + + pid_b * stride_l_b + + (pid_h + off_h) * stride_l_h, + lse_i.to(lse_ptr.dtype.element_ty), + mask=h_mask, ) - tl.store(lse_ptrs, lse_i.to(lse_ptr.dtype.element_ty), boundary_check=(0,)) @triton.heuristics( @@ -444,16 +459,16 @@ def _merge_topk_attn_out_kernel( off_c = tl.arange(0, NUM_TOPK_CHUNKS) off_d = tl.arange(0, BLOCK_SIZE_D) - o_ptrs = tl.make_block_ptr( - base=o_ptr + pid_b * stride_o_b + pid_h * stride_o_h, - shape=(NUM_TOPK_CHUNKS, head_dim), - strides=(stride_o_c, stride_o_d), - offsets=(0, 0), - block_shape=(NUM_TOPK_CHUNKS, BLOCK_SIZE_D), - order=(1, 0), + d_mask = off_d < head_dim + o_ptrs = ( + o_ptr + + pid_b * stride_o_b + + pid_h * stride_o_h + + off_c[:, None] * stride_o_c + + off_d[None, :] * stride_o_d ) lse_ptrs = lse_ptr + pid_b * stride_l_b + pid_h * stride_l_h + off_c * stride_l_c - o = tl.load(o_ptrs, boundary_check=(0, 1), padding_option="zero") + o = tl.load(o_ptrs, mask=d_mask[None, :], other=0.0) lse = tl.load(lse_ptrs) # empty chunks contribute -inf -> weight 0 lse_max = tl.max(lse, axis=0) weights = tl.exp2(lse - lse_max) @@ -462,7 +477,7 @@ def _merge_topk_attn_out_kernel( out_ptrs = ( out_ptr + pid_b * stride_out_n + pid_h * stride_out_h + off_d * stride_out_d ) - tl.store(out_ptrs, o_merged.to(out_ptr.dtype.element_ty), mask=off_d < head_dim) + tl.store(out_ptrs, o_merged.to(out_ptr.dtype.element_ty), mask=d_mask) # --------------------------------------------------------------------------- diff --git a/vllm/models/minimax_m3/nvidia/ops/index_decode_score.py b/vllm/models/minimax_m3/nvidia/ops/index_decode_score.py index 3d2919e392..8aa5e5a11e 100644 --- a/vllm/models/minimax_m3/nvidia/ops/index_decode_score.py +++ b/vllm/models/minimax_m3/nvidia/ops/index_decode_score.py @@ -287,6 +287,8 @@ class IndexDecodeScoreKernel: cute.arch.barrier(barrier_id=self.BAR_MMA, number_of_threads=128) for q in cutlass.range_constexpr(Q_TILES): cute.copy(ldsm_atom, sQ_ldsm[None, (q, None)], rQ[None, None, q]) + # Order ldmatrix reads before the producer refills this stage with TMA. + cute.arch.fence_proxy("async.shared", space="cta") cute.arch.mbarrier_arrive(tma_empty_mbar) tma_stage = 1 % self.num_stages @@ -342,6 +344,7 @@ class IndexDecodeScoreKernel: rC[None, m, n], ) + cute.arch.fence_proxy("async.shared", space="cta") cute.arch.mbarrier_arrive(tma_empty_mbar + tma_stage) k_start = block_id * BLOCK_K + warp_id * 32 diff --git a/vllm/models/qwen4_exp/amd/indexer_qsa.py b/vllm/models/qwen4_exp/amd/indexer_qsa.py index 4345763fd4..09cb39acf9 100644 --- a/vllm/models/qwen4_exp/amd/indexer_qsa.py +++ b/vllm/models/qwen4_exp/amd/indexer_qsa.py @@ -4,7 +4,7 @@ from __future__ import annotations -from typing import cast +from typing import TYPE_CHECKING, cast import torch from torch import nn @@ -23,6 +23,10 @@ from ..common.qsa_cache import ( QSAKeyStateCache, canonical_qsa_rope_positions, ) +from .ops.qsa_prepare import qsa_prepare, supports_fused_pre_indexer + +if TYPE_CHECKING: + from .qsa import Qwen4ExpQSAAttention def apply_qsa_rope( @@ -98,6 +102,12 @@ class QSAIndexer(nn.Module): self.token_topk = int(config.indexer_budget) self.compress_ratio = int(config.indexer_compress_ratio) self.rotary_emb = rotary_emb + self.use_fused_pre_indexer = supports_fused_pre_indexer( + rotary_emb, + self.index_head_dim, + self.index_kv_heads, + self.compress_ratio, + ) self.prefix = prefix # MTP step 0 selects the target-aligned rows; later steps reuse them # while continuing to update the QSA side cache. @@ -143,20 +153,27 @@ class QSAIndexer(nn.Module): def output_width(self) -> int: return self.token_topk + self.compress_ratio - 1 - def project_qk( + def project( self, hidden_states: torch.Tensor, - positions: torch.Tensor, ) -> tuple[torch.Tensor, torch.Tensor]: - """Project replicated Q/K, normalize+rotate Q, and preserve raw K.""" + """Project replicated Q/K and split, leaving both operands unrotated.""" qk, _ = self.index_qk_proj(hidden_states) - q_raw, token_k = qk.split( + return qk.split( ( self.index_n_heads * self.index_head_dim, self.index_kv_heads * self.index_head_dim, ), dim=-1, ) + + def project_qk( + self, + hidden_states: torch.Tensor, + positions: torch.Tensor, + ) -> tuple[torch.Tensor, torch.Tensor]: + """Project replicated Q/K, normalize+rotate Q, and preserve raw K.""" + q_raw, token_k = self.project(hidden_states) q = q_raw.reshape(-1, self.index_n_heads, self.index_head_dim) q = apply_qsa_rmsnorm( self.q_layernorm, @@ -251,6 +268,66 @@ class QSAIndexer(nn.Module): position_rows, ) + def _fused_prepare( + self, + projected_q: torch.Tensor, + raw_keys: torch.Tensor, + positions: torch.Tensor, + raw_metadata: QSAForwardMetadata, + compressed_metadata: QSAForwardMetadata, + attn: Qwen4ExpQSAAttention, + qkv: torch.Tensor, + slot_mapping: torch.Tensor, + gate_out: torch.Tensor | None, + ) -> tuple[torch.Tensor, tuple[torch.Tensor, torch.Tensor]]: + """Prepare the indexer and ``attn``'s main attention in one launch. + + Normalizes and rotates the index Q, compresses K and writes both side + caches, which replaces ``project_qk``'s norm/RoPE half and the whole of + ``_update_and_compress``. The same launch applies ``attn``'s QK-norm and + RoPE, copies its gate and writes its K/V into ``attn.kv_cache``, + replacing ``_project_qkv_gate`` and ``do_kv_cache_update``. Both halves + must stay numerically interchangeable with what they replace. + """ + num_tokens = raw_metadata.num_actual_tokens + raw_key_cache = self.raw_key_cache + q = projected_q.new_empty(num_tokens, self.index_n_heads, self.index_head_dim) + main_outputs = qsa_prepare( + projected_q, + raw_keys, + positions, + self.rotary_emb.cos_sin_cache, + self.q_layernorm.weight, + self.k_layernorm.weight, + self.q_layernorm.variance_epsilon, + q, + raw_key_cache.kv_cache, + raw_metadata.slot_mapping, + raw_metadata.block_table, + raw_metadata.query_start_loc, + raw_metadata.logical_positions, + self.compressed_key_cache.kv_cache, + compressed_metadata.slot_mapping, + compressed_metadata.k_work_metadata, + compress_ratio=self.compress_ratio, + mrope_section=getattr(self.rotary_emb, "mrope_section", None), + rope_pos_offset=( + raw_key_cache.rope_position_offset + if raw_key_cache.cache_rope_positions + else None + ), + main_qkv=qkv[:num_tokens], + main_q_norm_weight=attn.q_norm.weight, + main_k_norm_weight=attn.k_norm.weight, + main_eps=attn.q_norm.variance_epsilon, + main_kv_cache=attn.kv_cache.transpose(1, 2), + main_slot_mapping=slot_mapping[:num_tokens], + main_k_scale=attn._k_scale_float, + main_v_scale=attn._v_scale_float, + main_gate_out=None if gate_out is None else gate_out[:num_tokens], + ) + return q, main_outputs + def _select( self, q: torch.Tensor, @@ -276,13 +353,25 @@ class QSAIndexer(nn.Module): hidden_states: torch.Tensor, positions: torch.Tensor, out: torch.Tensor | None = None, - ) -> torch.Tensor: - """Return fixed-width request-relative token indices padded with ``-1``.""" + *, + attn: Qwen4ExpQSAAttention, + qkv: torch.Tensor | None = None, + slot_mapping: torch.Tensor | None = None, + gate_out: torch.Tensor | None = None, + ) -> tuple[torch.Tensor, tuple[torch.Tensor, torch.Tensor] | None]: + """Return fixed-width request-relative token indices padded with ``-1``. + + With ``attn.use_fused_qsa_prepare``, the same launch also writes + ``attn``'s K/V into ``attn.kv_cache`` at ``slot_mapping`` and prepares + its Q and gate from ``qkv``, returned as the second element (None + otherwise). The gate is written into ``gate_out`` when given. ``qkv``, + ``slot_mapping`` and ``gate_out`` are only read in that mode. + """ metadata = self._metadata() if metadata is None: # Preserve step-0 indices when later MTP steps reuse the buffer. if self.skip_topk and out is not None: - return out + return out, None result = torch.full( (hidden_states.shape[0], self.output_width), -1, @@ -291,24 +380,41 @@ class QSAIndexer(nn.Module): ) if out is not None: out.copy_(result) - return out - return result + return out, None + return result, None raw_metadata, compressed_metadata = metadata num_tokens = raw_metadata.num_actual_tokens - q, token_k = self.project_qk( - hidden_states[:num_tokens], positions[..., :num_tokens] - ) - self._update_and_compress( - token_k, - positions[..., :num_tokens], - raw_metadata, - compressed_metadata, - ) + hidden_states = hidden_states[:num_tokens] + positions = positions[..., :num_tokens] + main_outputs: tuple[torch.Tensor, torch.Tensor] | None = None + if attn.use_fused_qsa_prepare: + if qkv is None or slot_mapping is None: + raise ValueError("fused QSA prepare requires qkv and slot_mapping") + projected_q, raw_keys = self.project(hidden_states) + q, main_outputs = self._fused_prepare( + projected_q, + raw_keys, + positions, + raw_metadata, + compressed_metadata, + attn, + qkv, + slot_mapping, + gate_out, + ) + else: + q, token_k = self.project_qk(hidden_states, positions) + self._update_and_compress( + token_k, + positions, + raw_metadata, + compressed_metadata, + ) if self.skip_topk: if out is None: raise RuntimeError("QSA top-k reuse requires an output buffer") - return out - return self._select(q, compressed_metadata, out) + return out, main_outputs + return self._select(q, compressed_metadata, out), main_outputs __all__ = ["QSAIndexer", "apply_qsa_rope"] diff --git a/vllm/models/qwen4_exp/amd/model.py b/vllm/models/qwen4_exp/amd/model.py index ecd8040328..cf8357ee6c 100644 --- a/vllm/models/qwen4_exp/amd/model.py +++ b/vllm/models/qwen4_exp/amd/model.py @@ -141,9 +141,9 @@ _QWEN4_EXP_IGNORED_MISSING_SUFFIXES = [ "_input_scale", ] -# The checkpoint keeps down and injection projections separate; runtime packs -# them into adjacent logical shards of one MergedColumnParallelLinear. -_HC_WEIGHTS_MAPPER = WeightsMapper( +# The checkpoint stores these projections separately; runtime packs each group +# into adjacent logical shards of a MergedColumnParallelLinear. +_EXTRA_WEIGHTS_MAPPER = WeightsMapper( orig_to_new_stacked={ "hyper_connection.input_mix_weight_down.weight": ( "hyper_connection.input_mix_weight_down_block_inject.weight", @@ -153,6 +153,8 @@ _HC_WEIGHTS_MAPPER = WeightsMapper( "hyper_connection.input_mix_weight_down_block_inject.weight", 1, ), + "ple.key_proj": ("ple.kv_proj", 0), + "ple.value_proj": ("ple.kv_proj", 1), } ) @@ -277,7 +279,7 @@ class Qwen4ExpDecoderLayer(nn.Module): if input_ids is None or query_start_loc is None or ngram_context is None: raise RuntimeError("PLE inputs were not prepared") - hidden_states = hidden_states + self.ple( + hidden_states = self.ple( hidden_states, input_ids, query_start_loc, @@ -372,7 +374,7 @@ class Qwen4ExpMixtureOfExperts(MixtureOfExperts): } ) class Qwen4ExpModel(nn.Module): - hf_to_vllm_mapper = Qwen3_5Model.hf_to_vllm_mapper | _HC_WEIGHTS_MAPPER + hf_to_vllm_mapper = Qwen3_5Model.hf_to_vllm_mapper | _EXTRA_WEIGHTS_MAPPER def __init__(self, *, vllm_config: VllmConfig, prefix: str = "") -> None: super().__init__() @@ -581,6 +583,7 @@ class Qwen4ExpForCausalLM( packed_modules_mapping = { "qkv_proj": ["q_proj", "k_proj", "v_proj"], "gate_up_proj": ["gate_proj", "up_proj"], + "kv_proj": ["key_proj", "value_proj"], "in_proj_qkvz": ["in_proj_qkv", "in_proj_z"], "in_proj_ba": ["in_proj_b", "in_proj_a"], "input_mix_weight_down_block_inject": [ @@ -631,8 +634,6 @@ class Qwen4ExpForCausalLM( inputs_embeds: torch.Tensor | None = None, **kwargs: object, ) -> torch.Tensor | IntermediateTensors: - # Forward kwargs unchanged so the runner's _maybe_add_ngram_kwargs - # path (query_start_loc / ngram_context) reaches Qwen4ExpModel. return self.model( input_ids, positions, @@ -812,11 +813,12 @@ class Qwen4ExpForConditionalGeneration( requires_raw_input_tokens = True packed_modules_mapping = Qwen3_5ForConditionalGeneration.packed_modules_mapping | { + "kv_proj": ["key_proj", "value_proj"], "input_mix_weight_down_block_inject": [ "input_mix_weight_down", "block_inject_weight", "_input_mix_padding", - ] + ], } @staticmethod diff --git a/vllm/models/qwen4_exp/amd/mtp.py b/vllm/models/qwen4_exp/amd/mtp.py index ab3e8bd114..98ee4318fb 100644 --- a/vllm/models/qwen4_exp/amd/mtp.py +++ b/vllm/models/qwen4_exp/amd/mtp.py @@ -50,7 +50,7 @@ from vllm.sequence import IntermediateTensors from .hyperconnection import GatedResidual, HyperConnectionConfig from .low_latency_gemm import enable_qwen4_exp_low_latency_gemm from .model import ( - _HC_WEIGHTS_MAPPER, + _EXTRA_WEIGHTS_MAPPER, _QWEN4_EXP_IGNORED_MISSING_SUFFIXES, Qwen4ExpDecoderLayer, Qwen4ExpMixtureOfExperts, @@ -172,7 +172,7 @@ def _make_draft_vllm_config( } ) class Qwen4ExpMultiTokenPredictor(nn.Module): - hf_to_vllm_mapper = Qwen3_5Model.hf_to_vllm_mapper | _HC_WEIGHTS_MAPPER + hf_to_vllm_mapper = Qwen3_5Model.hf_to_vllm_mapper | _EXTRA_WEIGHTS_MAPPER def __init__(self, *, vllm_config: VllmConfig, prefix: str = "") -> None: super().__init__() diff --git a/vllm/models/qwen4_exp/amd/ops/qsa_prepare.py b/vllm/models/qwen4_exp/amd/ops/qsa_prepare.py new file mode 100644 index 0000000000..bc2fee3b3a --- /dev/null +++ b/vllm/models/qwen4_exp/amd/ops/qsa_prepare.py @@ -0,0 +1,703 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Fused QSA prepare kernel for the AMD Qwen4Exp path. + +Started as a copy of the NVIDIA kernel (``nvidia/ops/qsa_prepare.py``) and is +kept separate so either vendor can change its own copy without re-validating +the other. The state source select is two masked loads rather than a pointer +select, which ROCm Triton rejects. +""" + +import torch +from torch import nn + +from vllm.triton_utils import tl, triton + + +def supports_fused_pre_indexer( + rotary_emb: nn.Module, + head_dim: int, + num_kv_heads: int, + compress_ratio: int, +) -> bool: + """Report whether this indexer's shapes match the fused kernel's assumptions. + + The kernel hard-codes the rotary layout and the single-KV-head group + compression it was written for; everything it rejects has a working unfused + path. + """ + rotary_dim = int(rotary_emb.rotary_dim) + mrope_section = getattr(rotary_emb, "mrope_section", None) + return ( + bool(getattr(rotary_emb, "is_neox_style", False)) + and ( + not mrope_section + or ( + len(mrope_section) == 3 + and sum(mrope_section) == rotary_dim // 2 + and bool(getattr(rotary_emb, "mrope_interleaved", False)) + ) + ) + and head_dim == 128 + and rotary_dim == 64 + and num_kv_heads == 1 + and compress_ratio > 1 + and compress_ratio & (compress_ratio - 1) == 0 + ) + + +@triton.jit +def _norm_rope( + x, + pos_t, + pos_h, + pos_w, + cos_sin_ptr, + cos_sin_stride, + norm_weight_ptr, + eps, + IS_MROPE: tl.constexpr, + MROPE_H: tl.constexpr, + MROPE_W: tl.constexpr, +): + """Apply Gemma RMSNorm and selected-axis NeoX RoPE to register rows.""" + TILE_T: tl.constexpr = x.shape[0] + TILE_H: tl.constexpr = x.shape[1] + D: tl.constexpr = x.shape[2] + ROWS: tl.constexpr = TILE_T * TILE_H + HALF: tl.constexpr = D // 2 + QUARTER: tl.constexpr = D // 4 + pairs = tl.arange(0, QUARTER) + if IS_MROPE: + # Qwen interleaves temporal, height, and width rotary pairs. Each axis + # still indexes the same position-major cos/sin table. + h_mask = ((pairs % 3) == 1) & (pairs <= 3 * MROPE_H) + w_mask = ((pairs % 3) == 2) & (pairs <= 3 * MROPE_W) + t_mask = ~(h_mask | w_mask) + base = cos_sin_ptr + pairs[None, :] + pos_rows = (pos_t, pos_h, pos_w) + axis_masks = (t_mask, h_mask, w_mask) + cos = tl.zeros((TILE_T, QUARTER), dtype=cos_sin_ptr.dtype.element_ty) + sin = tl.zeros((TILE_T, QUARTER), dtype=cos_sin_ptr.dtype.element_ty) + for axis in tl.static_range(3): + cos += tl.load( + base + pos_rows[axis][:, None] * cos_sin_stride, + mask=axis_masks[axis][None, :], + other=0, + ) + sin += tl.load( + base + pos_rows[axis][:, None] * cos_sin_stride + QUARTER, + mask=axis_masks[axis][None, :], + other=0, + ) + else: + cos = tl.load(cos_sin_ptr + pos_t[:, None] * cos_sin_stride + pairs[None, :]) + sin = tl.load( + cos_sin_ptr + pos_t[:, None] * cos_sin_stride + QUARTER + pairs[None, :] + ) + + cos = tl.reshape( + tl.broadcast_to(cos[:, None, :], (TILE_T, TILE_H, QUARTER)), + (ROWS, QUARTER), + ) + sin = tl.reshape( + tl.broadcast_to(sin[:, None, :], (TILE_T, TILE_H, QUARTER)), + (ROWS, QUARTER), + ) + x = tl.reshape(x, (ROWS, D)).to(tl.float32) + weight = tl.load(norm_weight_ptr + tl.arange(0, D)).to(tl.float32) + 1.0 + rrms = tl.rsqrt(tl.sum(x * x, axis=1) / D + eps) + y = (x * rrms[:, None] * weight[None, :]).to(cos.dtype) + rotated, passthrough = tl.split( + tl.permute(tl.reshape(y, (ROWS, 2, HALF)), (0, 2, 1)) + ) + r0, r1 = tl.split(tl.permute(tl.reshape(rotated, (ROWS, 2, QUARTER)), (0, 2, 1))) + out0 = r0 * cos - r1 * sin + out1 = r1 * cos + r0 * sin + rotated = tl.reshape(tl.permute(tl.join(out0, out1), (0, 2, 1)), (ROWS, HALF)) + result = tl.reshape(tl.permute(tl.join(rotated, passthrough), (0, 2, 1)), (ROWS, D)) + return tl.reshape(result, (TILE_T, TILE_H, D)) + + +@triton.jit +def _to_dst_dtype(x, dst, scale): + """Round to BF16 like the unfused path, then scale for an FP8 destination.""" + out_ty = dst.dtype.element_ty + x = x.to(tl.bfloat16) + if out_ty == tl.float8e4nv: + x = x.to(tl.float32) / scale + return x.to(out_ty) + + +@triton.jit +def _store_rotated(dst, y, o1, o2, scale): + """Store a normalized head whose first ``2 * len(o1)`` dims are rotated.""" + HALF: tl.constexpr = o1.shape[0] + dims = tl.arange(0, y.shape[0]) + rot = tl.arange(0, HALF) + tl.store(dst + dims, _to_dst_dtype(y, dst, scale), mask=dims >= 2 * HALF) + tl.store(dst + rot, _to_dst_dtype(o1, dst, scale)) + tl.store(dst + HALF + rot, _to_dst_dtype(o2, dst, scale)) + + +@triton.jit( + do_not_specialize=[ + "num_tokens", + "num_state_blocks", + "num_compressed_blocks", + "num_k_work", + ] +) +def _qsa_prepare_kernel( + q_ptr, + q_stride_token, + k_ptr, + k_stride_token, + pos_ptr, + pos_stride_axis, + pos_stride_token, + cos_sin_ptr, + q_norm_weight_ptr, + k_norm_weight_ptr, + eps, + q_out_ptr, + q_out_stride_token, + q_out_stride_head, + state_cache_ptr, + state_cache_stride_block, + state_cache_stride_token, + state_slots_ptr, + state_table_ptr, + state_table_stride_req, + query_start_loc_ptr, + logical_positions_ptr, + compressed_slots_ptr, + k_work_metadata_ptr, + compressed_cache_ptr, + compressed_cache_stride_block, + compressed_cache_stride_token, + num_tokens, + num_state_blocks, + num_compressed_blocks, + num_k_work, + HQ: tl.constexpr, + D: tl.constexpr, + TILE_T_Q: tl.constexpr, + TILE_H_Q: tl.constexpr, + COMPRESS_RATIO: tl.constexpr, + STATE_SIZE: tl.constexpr, + COMP_PAGE_SIZE: tl.constexpr, + IS_2D_POSITIONS: tl.constexpr, + IS_K_MROPE: tl.constexpr, + CACHE_HAS_ROPE_POS: tl.constexpr, + MROPE_H: tl.constexpr, + MROPE_W: tl.constexpr, + main_qkv_ptr, + main_q_norm_weight_ptr, + main_k_norm_weight_ptr, + main_eps, + main_q_out_ptr, + main_gate_out_ptr, + main_cache_ptr, + main_cache_stride_block, + main_cache_stride_token, + main_cache_stride_head, + main_slots_ptr, + main_k_scale, + main_v_scale, + MAIN_HQ: tl.constexpr, + MAIN_HK: tl.constexpr, + MAIN_D: tl.constexpr, + MAIN_PAGE_SIZE: tl.constexpr, +): + pid = tl.program_id(0) + num_index_work = num_k_work + tl.cdiv(num_tokens, TILE_T_Q) * tl.cdiv(HQ, TILE_H_Q) + if pid >= num_index_work: + # Main attention: one program per (token, Q or KV head) after the + # indexer work. RoPE covers the first D // 2 dims, the width of the + # shared cos/sin table. + main_pid = pid - num_index_work + token = (main_pid // (MAIN_HQ + MAIN_HK)).to(tl.int64) + head = main_pid % (MAIN_HQ + MAIN_HK) + HALF: tl.constexpr = D // 4 + dims = tl.arange(0, MAIN_D) + rot = tl.arange(0, HALF) + row = main_qkv_ptr + token * (2 * (MAIN_HQ + MAIN_HK) * MAIN_D) + is_k = head >= MAIN_HQ + kv_head = head - MAIN_HQ + if is_k: + src = row + (2 * MAIN_HQ + kv_head) * MAIN_D + weight_ptr = main_k_norm_weight_ptr + else: + src = row + 2 * head * MAIN_D + weight_ptr = main_q_norm_weight_ptr + x = tl.load(src + dims).to(tl.float32) + inv_rms = tl.rsqrt(tl.sum(x * x, axis=0) / MAIN_D + main_eps) + w = tl.load(weight_ptr + dims).to(tl.float32) + 1.0 + y = x * inv_rms * w + x1 = tl.load(src + rot).to(tl.float32) + x2 = tl.load(src + HALF + rot).to(tl.float32) + w1 = tl.load(weight_ptr + rot).to(tl.float32) + 1.0 + w2 = tl.load(weight_ptr + HALF + rot).to(tl.float32) + 1.0 + x1 = (x1 * inv_rms * w1).to(tl.bfloat16).to(tl.float32) + x2 = (x2 * inv_rms * w2).to(tl.bfloat16).to(tl.float32) + pos = tl.load(pos_ptr + token * pos_stride_token).to(tl.int64) + if IS_2D_POSITIONS: + pos_h = tl.load(pos_ptr + pos_stride_axis + token * pos_stride_token) + pos_w = tl.load(pos_ptr + 2 * pos_stride_axis + token * pos_stride_token) + is_h = (rot % 3 == 1) & (rot < 3 * MROPE_H) + is_w = (rot % 3 == 2) & (rot < 3 * MROPE_W) + pos = tl.where( + is_h, pos_h.to(tl.int64), tl.where(is_w, pos_w.to(tl.int64), pos) + ) + cos = tl.load(cos_sin_ptr + pos * (D // 2) + rot).to(tl.float32) + sin = tl.load(cos_sin_ptr + pos * (D // 2) + HALF + rot).to(tl.float32) + o1 = x1 * cos - x2 * sin + o2 = x2 * cos + x1 * sin + if is_k: + slot = tl.load(main_slots_ptr + token).to(tl.int64) + if slot >= 0: + dst = ( + main_cache_ptr + + (slot // MAIN_PAGE_SIZE) * main_cache_stride_block + + (slot % MAIN_PAGE_SIZE) * main_cache_stride_token + + kv_head * main_cache_stride_head + ) + _store_rotated(dst, y, o1, o2, main_k_scale) + v = tl.load(src + MAIN_HK * MAIN_D + dims) + tl.store(dst + MAIN_D + dims, _to_dst_dtype(v, dst, main_v_scale)) + else: + out = (token * MAIN_HQ + head) * MAIN_D + _store_rotated(main_q_out_ptr + out, y, o1, o2, None) + gate = tl.load(src + MAIN_D + dims) + tl.store(main_gate_out_ptr + out + dims, gate) + return + # K work occupies the first programs; the remaining programs tile Q. This + # keeps both paths in one launch while leaving their register shapes + # independent. + if pid >= num_k_work: + q_pid = pid - num_k_work + num_head_tiles: tl.constexpr = tl.cdiv(HQ, TILE_H_Q) + token_tile = q_pid // num_head_tiles + head_tile = q_pid % num_head_tiles + tokens = (token_tile * TILE_T_Q + tl.arange(0, TILE_T_Q)).to(tl.int64) + heads = head_tile * TILE_H_Q + tl.arange(0, TILE_H_Q) + valid_tokens = tokens < num_tokens + valid_heads = heads < HQ + dims = tl.arange(0, D) + mask = valid_tokens[:, None, None] & valid_heads[None, :, None] + x = tl.load( + q_ptr + + tokens[:, None, None] * q_stride_token + + heads[None, :, None] * D + + dims[None, None, :], + mask=mask, + other=0.0, + ) + pos_t = tl.load(pos_ptr + tokens * pos_stride_token, mask=valid_tokens, other=0) + if IS_2D_POSITIONS: + pos_h = tl.load( + pos_ptr + pos_stride_axis + tokens * pos_stride_token, + mask=valid_tokens, + other=0, + ) + pos_w = tl.load( + pos_ptr + 2 * pos_stride_axis + tokens * pos_stride_token, + mask=valid_tokens, + other=0, + ) + else: + pos_h = pos_t + pos_w = pos_t + y = _norm_rope( + x, + pos_t, + pos_h, + pos_w, + cos_sin_ptr, + D // 2, + q_norm_weight_ptr, + eps, + IS_2D_POSITIONS, + MROPE_H, + MROPE_W, + ) + tl.store( + q_out_ptr + + tokens[:, None, None] * q_out_stride_token + + heads[None, :, None] * q_out_stride_head + + dims[None, None, :], + y, + mask=mask, + ) + + if pid < num_k_work: + # One work item owns one completed compression group. Work item zero + # additionally commits the request's current raw-K suffix below. + work_metadata = tl.load(k_work_metadata_ptr + pid * 2 + tl.arange(0, 2)) + request, work_in_request = tl.split(work_metadata) + if request < 0: + return + + query_start = tl.load(query_start_loc_ptr + request) + query_end = tl.load(query_start_loc_ptr + request + 1) + query_len = query_end - query_start + chunk_end = tl.load(logical_positions_ptr + query_end - 1) + chunk_start = chunk_end - query_len + 1 + num_groups = (chunk_end + 1) // COMPRESS_RATIO - chunk_start // COMPRESS_RATIO + dims = tl.arange(0, D) + + if work_in_request < num_groups: + first_boundary = ( + (chunk_start + COMPRESS_RATIO) // COMPRESS_RATIO + ) * COMPRESS_RATIO - 1 + end_position = first_boundary + work_in_request * COMPRESS_RATIO + boundary_token = query_start + end_position - chunk_start + valid_token = ( + (boundary_token >= query_start) + & (boundary_token < query_end) + & (boundary_token < num_tokens) + ) + compressed_slot = tl.load( + compressed_slots_ptr + boundary_token, + mask=valid_token, + other=-1, + ) + valid = ( + valid_token + & (compressed_slot >= 0) + & (compressed_slot < num_compressed_blocks * COMP_PAGE_SIZE) + ) + state_block = tl.load(state_table_ptr + request * state_table_stride_req) + state_block_valid = (state_block >= 0) & (state_block < num_state_blocks) + safe_state_block = tl.maximum(state_block, 0).to(tl.int64) + group_offsets = tl.arange(0, COMPRESS_RATIO) + source_positions = end_position - (COMPRESS_RATIO - 1) + group_offsets + source_in_chunk = source_positions >= chunk_start + source_in_cache = source_positions < chunk_start + source_tokens = query_start + source_positions - chunk_start + source_tokens_valid = ( + (source_tokens >= query_start) + & (source_tokens < query_end) + & (source_tokens < num_tokens) + ) + current_base = ( + k_ptr + + tl.maximum(source_tokens, 0).to(tl.int64)[:, None] * k_stride_token + ) + cached_base = ( + state_cache_ptr + + safe_state_block * state_cache_stride_block + + (source_positions % STATE_SIZE)[:, None] * state_cache_stride_token + ) + # Only the first completed group can cross the chunk boundary, so + # every row comes from exactly one of the two sources. Selecting + # between the base pointers would need a pointer-typed tl.where, + # which ROCm's Triton rejects ("'tt.addptr' op failed to verify + # that result type matches ptr type"), so select the loaded values + # instead. The two masks are complementary, so no lane fetches from + # both and memory traffic is unchanged. + # + # Both bases still need the alignment stated explicitly. Without it + # Triton vectorizes these loads differently, which reorders the + # pooling reduction below; float addition is not associative, so the + # compressed rows stop matching the unfused path bit for bit and an + # FP8 cache rounds a near-zero pooled value to the wrong code. + current_base = tl.multiple_of(current_base, (8, 8)) + cached_base = tl.multiple_of(cached_base, (8, 8)) + from_chunk = valid & source_in_chunk[:, None] & source_tokens_valid[:, None] + from_cache = valid & source_in_cache[:, None] & state_block_valid + current = tl.load(current_base + dims[None, :], mask=from_chunk, other=0.0) + cached = tl.load(cached_base + dims[None, :], mask=from_cache, other=0.0) + source = tl.where(source_in_chunk[:, None], current, cached).to(tl.float32) + # Match the unfused path's BF16 pooled tensor before RMSNorm. + pooled = ( + (tl.sum(source, axis=0) / COMPRESS_RATIO).to(tl.bfloat16).to(tl.float32) + ) + + first_position = end_position - (COMPRESS_RATIO - 1) + if CACHE_HAS_ROPE_POS: + # RoPE uses the first token in the pooled group. Its exact MRoPE + # coordinates may live in this chunk or the raw-state ring. + first_in_chunk = first_position >= chunk_start + first_token = query_start + first_position - chunk_start + first_token_valid = ( + (first_token >= query_start) + & (first_token < query_end) + & (first_token < num_tokens) + ) + safe_first_token = tl.maximum(first_token, 0) + load_current_position = first_in_chunk & first_token_valid + first_pos_t = tl.load( + pos_ptr + safe_first_token * pos_stride_token, + mask=load_current_position, + other=0, + ) + if IS_2D_POSITIONS: + first_pos_h = tl.load( + pos_ptr + pos_stride_axis + safe_first_token * pos_stride_token, + mask=load_current_position, + other=0, + ) + first_pos_w = tl.load( + pos_ptr + + 2 * pos_stride_axis + + safe_first_token * pos_stride_token, + mask=load_current_position, + other=0, + ) + else: + first_pos_h = first_pos_t + first_pos_w = first_pos_t + tail = ( + state_cache_ptr + + safe_state_block * state_cache_stride_block + + (first_position % STATE_SIZE) * state_cache_stride_token + + D + ).to(tl.pointer_type(tl.int64)) + load_cached_position = ~first_in_chunk & state_block_valid + cached_pos_t = tl.load(tail, mask=load_cached_position, other=0) + cached_pos_h = tl.load(tail + 1, mask=load_cached_position, other=0) + cached_pos_w = tl.load(tail + 2, mask=load_cached_position, other=0) + pos_t = tl.where(first_in_chunk, first_pos_t.to(tl.int64), cached_pos_t) + pos_h = tl.where(first_in_chunk, first_pos_h.to(tl.int64), cached_pos_h) + pos_w = tl.where(first_in_chunk, first_pos_w.to(tl.int64), cached_pos_w) + else: + pos_t = first_position + pos_h = first_position + pos_w = first_position + y = _norm_rope( + tl.reshape(pooled, (1, 1, D)), + pos_t + tl.arange(0, 1), + pos_h + tl.arange(0, 1), + pos_w + tl.arange(0, 1), + cos_sin_ptr, + D // 2, + k_norm_weight_ptr, + eps, + IS_K_MROPE, + MROPE_H, + MROPE_W, + ) + compressed_block = (compressed_slot // COMP_PAGE_SIZE).to(tl.int64) + compressed_row = compressed_slot % COMP_PAGE_SIZE + tl.store( + compressed_cache_ptr + + compressed_block * compressed_cache_stride_block + + compressed_row * compressed_cache_stride_token + + dims, + tl.reshape(y, (D,)), + mask=valid, + ) + + if work_in_request == 0: + # This CTA may have just read historical rows from the circular buffer. + # Keep every lane past those loads before overwriting the same ring. + tl.debug_barrier() + # One CTA per request commits only the suffix retained by the ring. + num_state_rows = tl.minimum(query_len, STATE_SIZE) + for state_offset in tl.range(0, num_state_rows): + token = (query_end - num_state_rows + state_offset).to(tl.int64) + valid_token = ( + (token >= query_start) & (token < query_end) & (token < num_tokens) + ) + slot = tl.load(state_slots_ptr + token, mask=valid_token, other=-1) + valid_slot = ( + valid_token & (slot >= 0) & (slot < num_state_blocks * STATE_SIZE) + ) + safe_slot = tl.maximum(slot, 0) + state_row = ( + state_cache_ptr + + (safe_slot // STATE_SIZE).to(tl.int64) * state_cache_stride_block + + (safe_slot % STATE_SIZE) * state_cache_stride_token + ) + k = tl.load( + k_ptr + tl.maximum(token, 0) * k_stride_token + dims, + mask=valid_slot, + other=0.0, + ) + tl.store(state_row + dims, k, mask=valid_slot) + if CACHE_HAS_ROPE_POS: + pos_t = tl.load( + pos_ptr + tl.maximum(token, 0) * pos_stride_token, + mask=valid_slot, + other=0, + ) + if IS_2D_POSITIONS: + pos_h = tl.load( + pos_ptr + + pos_stride_axis + + tl.maximum(token, 0) * pos_stride_token, + mask=valid_slot, + other=0, + ) + pos_w = tl.load( + pos_ptr + + 2 * pos_stride_axis + + tl.maximum(token, 0) * pos_stride_token, + mask=valid_slot, + other=0, + ) + else: + pos_h = pos_t + pos_w = pos_t + tail = (state_row + D).to(tl.pointer_type(tl.int64)) + tl.store(tail, pos_t.to(tl.int64), mask=valid_slot) + tl.store(tail + 1, pos_h.to(tl.int64), mask=valid_slot) + tl.store(tail + 2, pos_w.to(tl.int64), mask=valid_slot) + + +def qsa_prepare( + q: torch.Tensor, + k: torch.Tensor, + positions: torch.Tensor, + cos_sin_cache: torch.Tensor, + q_norm_weight: torch.Tensor, + k_norm_weight: torch.Tensor, + eps: float, + q_out: torch.Tensor, + state_cache: torch.Tensor, + state_slots: torch.Tensor, + state_block_table: torch.Tensor, + query_start_loc: torch.Tensor, + logical_positions: torch.Tensor, + compressed_cache: torch.Tensor, + compressed_slots: torch.Tensor, + k_work_metadata: torch.Tensor, + *, + compress_ratio: int, + mrope_section: tuple[int, int, int] | None, + rope_pos_offset: int | None, + main_qkv: torch.Tensor, + main_q_norm_weight: torch.Tensor, + main_k_norm_weight: torch.Tensor, + main_eps: float, + main_kv_cache: torch.Tensor, + main_slot_mapping: torch.Tensor, + main_k_scale: float, + main_v_scale: float, + main_gate_out: torch.Tensor | None = None, +) -> tuple[torch.Tensor, torch.Tensor]: + """Normalize Q, compress K, then update the circular raw state. + + Also prepares the main attention (QK-norm/RoPE, gate copy, K/V cache write) + and returns its Q and gate. ``main_gate_out`` lets the gate land in a + caller-owned buffer, e.g. one that outlives an opaque custom op. + """ + num_tokens = q.shape[0] + main_head_dim = main_kv_cache.shape[-1] // 2 + num_main_kv_heads = main_kv_cache.shape[2] + num_main_q_heads = main_qkv.shape[1] // (2 * main_head_dim) - num_main_kv_heads + main_q_out = main_qkv.new_empty(num_tokens, num_main_q_heads, main_head_dim) + if main_gate_out is None: + main_gate_out = torch.empty_like(main_q_out) + else: + assert main_gate_out.shape == main_q_out.shape + assert main_gate_out.is_contiguous() + if num_tokens == 0: + return main_q_out, main_gate_out + num_q_heads, head_dim = q_out.shape[1:] + assert cos_sin_cache.shape[-1] * 2 == head_dim + assert q.shape == (num_tokens, num_q_heads * head_dim) + assert k.shape == (num_tokens, head_dim) + assert q.stride(-1) == 1 + assert k.stride(-1) == 1 + assert q_out.stride(-1) == 1 + assert cos_sin_cache.is_contiguous() + assert state_cache.stride(-1) == 1 + assert compressed_cache.stride(-1) == 1 + assert k_work_metadata.ndim == 2 and k_work_metadata.shape[1] == 2 + is_2d_positions = positions.ndim == 2 + is_k_mrope = bool(mrope_section) + cache_has_rope_pos = rope_pos_offset is not None + assert rope_pos_offset is None or rope_pos_offset == head_dim + if is_2d_positions: + assert positions.shape == (3, num_tokens) + assert is_k_mrope + pos_stride_axis, pos_stride_token = positions.stride() + else: + assert positions.shape == (num_tokens,) + pos_stride_axis, pos_stride_token = 0, positions.stride(0) + section = mrope_section if mrope_section is not None else (0, 0, 0) + assert len(section) == 3 + qkv_width = 2 * (num_main_q_heads + num_main_kv_heads) * main_head_dim + assert main_qkv.shape == (num_tokens, qkv_width) and main_qkv.is_contiguous() + assert main_slot_mapping.shape == (num_tokens,) + + # Swept on MI355X (64-lane wavefronts) over TILE_T_Q 1-16, TILE_H_Q 1-4 and + # num_warps 1/2/4, from 4 to 16384 tokens: no config beat these by more + # than noise, and 2 or 4 warps were slower everywhere. + if num_tokens <= 4096: + TILE_T_Q, TILE_H_Q = 2, 2 + else: + TILE_T_Q, TILE_H_Q = 2, 4 + num_k_work = k_work_metadata.shape[0] + num_q_work = triton.cdiv(num_tokens, TILE_T_Q) * triton.cdiv(num_q_heads, TILE_H_Q) + num_main_work = num_tokens * (num_main_q_heads + num_main_kv_heads) + _qsa_prepare_kernel[(num_k_work + num_q_work + num_main_work,)]( + q, + q.stride(0), + k, + k.stride(0), + positions, + pos_stride_axis, + pos_stride_token, + cos_sin_cache, + q_norm_weight, + k_norm_weight, + eps, + q_out, + q_out.stride(0), + q_out.stride(1), + state_cache, + state_cache.stride(0), + state_cache.stride(1), + state_slots, + state_block_table, + state_block_table.stride(0), + query_start_loc, + logical_positions, + compressed_slots, + k_work_metadata, + compressed_cache, + compressed_cache.stride(0), + compressed_cache.stride(1), + num_tokens, + state_cache.shape[0], + compressed_cache.shape[0], + num_k_work, + HQ=num_q_heads, + D=head_dim, + TILE_T_Q=TILE_T_Q, + TILE_H_Q=TILE_H_Q, + COMPRESS_RATIO=compress_ratio, + STATE_SIZE=state_cache.shape[1], + COMP_PAGE_SIZE=compressed_cache.shape[1], + IS_2D_POSITIONS=is_2d_positions, + IS_K_MROPE=is_k_mrope, + CACHE_HAS_ROPE_POS=cache_has_rope_pos, + MROPE_H=section[1], + MROPE_W=section[2], + main_qkv_ptr=main_qkv, + main_q_norm_weight_ptr=main_q_norm_weight, + main_k_norm_weight_ptr=main_k_norm_weight, + main_eps=main_eps, + main_q_out_ptr=main_q_out, + main_gate_out_ptr=main_gate_out, + main_cache_ptr=main_kv_cache, + main_cache_stride_block=main_kv_cache.stride(0), + main_cache_stride_token=main_kv_cache.stride(1), + main_cache_stride_head=main_kv_cache.stride(2), + main_slots_ptr=main_slot_mapping, + main_k_scale=main_k_scale, + main_v_scale=main_v_scale, + MAIN_HQ=num_main_q_heads, + MAIN_HK=num_main_kv_heads, + MAIN_D=main_head_dim, + MAIN_PAGE_SIZE=main_kv_cache.shape[1], + num_warps=1, + ) + return main_q_out, main_gate_out + + +__all__ = ["qsa_prepare", "supports_fused_pre_indexer"] diff --git a/vllm/models/qwen4_exp/amd/ple_layer.py b/vllm/models/qwen4_exp/amd/ple_layer.py index bd0607b613..fbc07b73dd 100644 --- a/vllm/models/qwen4_exp/amd/ple_layer.py +++ b/vllm/models/qwen4_exp/amd/ple_layer.py @@ -2,18 +2,16 @@ # SPDX-FileCopyrightText: Copyright contributors to the vLLM project """GPU-resident Qwen4Exp position-learning enhancement layers.""" -import math from collections.abc import Iterable, Sequence import torch -import torch.nn.functional as F from torch import nn from transformers import Qwen4ExpTextConfig from vllm.config import CacheConfig, ModelConfig, VllmConfig, get_current_vllm_config from vllm.forward_context import get_forward_context from vllm.logger import init_logger -from vllm.model_executor.layers.linear import ReplicatedLinear +from vllm.model_executor.layers.linear import MergedColumnParallelLinear from vllm.model_executor.layers.mamba.abstract import MambaBase from vllm.model_executor.layers.mamba.mamba_utils import ( MambaStateDtypeCalculator, @@ -30,13 +28,13 @@ from vllm.v1.attention.backends.short_conv_attn import ( PleShortConvAttentionBackend, PleShortConvAttentionMetadata, ) -from vllm.v1.attention.backends.utils import NULL_BLOCK_ID from ..common.ngram_embedding import ( Qwen4ExpPLEDeviceEmbedding, Qwen4ExpPLEEmbeddingMethod, Qwen4ExpPLEPinnedHostEmbedding, ) +from ..common.ops.ple import ple_conv, ple_gate, ple_ngram_ids logger = init_logger(__name__) @@ -177,7 +175,6 @@ class Qwen4ExpNGramEmbedding(nn.Module): embedding_dim: int, ple_dense_layer_id: int, max_total_tokens: int, - max_num_reqs: int, prefix: str, layer_name: str, *, @@ -270,55 +267,6 @@ class Qwen4ExpNGramEmbedding(nn.Module): self.ngram_embedding.weight.device, self.ngram_embedding.weight.is_pinned(), ) - self.register_buffer( - "positions_buffer", - torch.arange(max_total_tokens, dtype=torch.int64), - persistent=False, - ) - self.register_buffer( - "padded_buffer", - torch.full( - (max_num_reqs, max_total_tokens), - self.eos_token_id, - dtype=torch.int64, - ), - persistent=False, - ) - - @staticmethod - def _shift_precompute( - tokens: torch.Tensor, eos_token_id: int - ) -> tuple[torch.Tensor, torch.Tensor]: - if tokens.dim() != 2: - raise ValueError("tokens must be a 2D tensor") - batch_size, seq_len = tokens.shape - positions = torch.arange(seq_len, device=tokens.device, dtype=torch.int64) - eos_positions = torch.where(tokens == eos_token_id, positions, -1) - previous_eos_inclusive = torch.cummax(eos_positions, dim=1).values - previous_eos = torch.cat( - [ - eos_positions.new_full((batch_size, 1), -1), - previous_eos_inclusive[:, :-1], - ], - dim=1, - ) - return positions, positions.unsqueeze(0) - previous_eos - 1 - - @staticmethod - def _shift_apply( - tokens: torch.Tensor, - positions: torch.Tensor, - position_in_segment: torch.Tensor, - shift: int, - eos_token_id: int, - ) -> torch.Tensor: - if shift == 0: - return tokens - source = positions - shift - gather_indices = source.clamp_min(0).unsqueeze(0).expand(tokens.shape[0], -1) - shifted = tokens.gather(1, gather_indices) - valid = (source.unsqueeze(0) >= 0) & (position_in_segment >= shift) - return torch.where(valid, shifted, tokens.new_full((), eos_token_id)) def forward( self, @@ -326,64 +274,17 @@ class Qwen4ExpNGramEmbedding(nn.Module): query_start_loc: torch.Tensor, ngram_context: torch.Tensor, ) -> torch.Tensor: - input_ids = input_ids.reshape(-1).long() - query_start_loc = query_start_loc.long() - num_reqs = query_start_loc.numel() - 1 - num_tokens = input_ids.shape[0] - if num_tokens > self.positions_buffer.numel(): - raise ValueError( - f"PLE received {num_tokens} tokens, but its workspace supports " - f"at most {self.positions_buffer.numel()}" - ) - if num_reqs > self.padded_buffer.shape[0]: - raise ValueError( - f"PLE received {num_reqs} requests, but its workspace supports " - f"at most {self.padded_buffer.shape[0]}" - ) - - positions = self.positions_buffer[:num_tokens] - packed = self.padded_buffer[:num_reqs] - packed.fill_(self.eos_token_id) - request_indices = torch.searchsorted(query_start_loc, positions, right=True) - 1 - request_indices.clamp_(max=num_reqs - 1) - columns = (positions - query_start_loc[request_indices]).clamp( - 0, packed.shape[1] - 1 + input_ids = input_ids.reshape(-1) + ngram_ids = input_ids.new_empty( + (input_ids.shape[0], self.ngram_heads), dtype=torch.int64 ) - packed[request_indices, columns] = input_ids - ngram_context = ngram_context[:num_reqs].to( - device=input_ids.device, dtype=torch.long + torch.ops.vllm.qwen4_exp_amd_ple_ngram_ids( + input_ids, + query_start_loc, + ngram_context, + ngram_ids, + self.layer_name, ) - - context = torch.cat([ngram_context, packed], dim=-1) - positions_2d, position_in_segment = self._shift_precompute( - context, self.eos_token_id - ) - shifted = [context] - for shift in range(1, self.ngram_size): - shifted.append( - self._shift_apply( - context, - positions_2d, - position_in_segment, - shift, - self.eos_token_id, - ) - ) - adjusted_columns = columns + self.ngram_size - 1 - id_blocks = [] - for ngram in range(2, self.ngram_size + 1): - start = (ngram - 2) * self.heads_per_ngram - end = start + self.heads_per_ngram - mixed = shifted[0] * self.layer_multipliers[0] - for index in range(1, ngram): - mixed = torch.bitwise_xor( - mixed, shifted[index] * self.layer_multipliers[index] - ) - sizes = self.ngram_heads_vocab_sizes[start:end] - offsets = self.ngram_heads_offsets[start:end] - ids = torch.remainder(mixed.unsqueeze(-1), sizes) + offsets - id_blocks.append(ids[request_indices, adjusted_columns]) - ngram_ids = torch.cat(id_blocks, dim=-1) embedding = self.ngram_embedding if embedding.supports_prefetch: output = ngram_ids.new_empty( @@ -508,26 +409,21 @@ class Qwen4ExpPLELayer(nn.Module, MambaBase): int(config.ple_embed_dim), self.ple_dense_layer_id, vllm_config.scheduler_config.max_num_batched_tokens, - vllm_config.scheduler_config.max_num_seqs, f"{prefix}.ple_embedding", prefix, data_parallel_rank=vllm_config.parallel_config.data_parallel_rank, quant_config=quant_config, params_dtype=model_config.dtype, ) - self.key_proj = ReplicatedLinear( + # The PLE cache is TP-replicated, so this merged projection is too. + self.kv_proj = MergedColumnParallelLinear( int(config.ple_embed_dim), - self.hc_hidden_size, + [self.hc_hidden_size, self.hidden_size], bias=False, + params_dtype=model_config.dtype, quant_config=quant_config, - prefix=f"{prefix}.key_proj", - ) - self.value_proj = ReplicatedLinear( - int(config.ple_embed_dim), - self.hidden_size, - bias=False, - quant_config=quant_config, - prefix=f"{prefix}.value_proj", + prefix=f"{prefix}.kv_proj", + disable_tp=True, ) norm_args = ( self.hc_hidden_size, @@ -580,355 +476,15 @@ class Qwen4ExpPLELayer(nn.Module, MambaBase): num_spec=self.num_spec_tokens, ) - def _apply_norm( - self, norm: Qwen4ExpPLEGroupedNorm, hidden_states: torch.Tensor - ) -> torch.Tensor: - shape = hidden_states.shape - return norm(hidden_states.flatten(-2)).reshape(shape) - - def _short_conv_fallback(self, inputs: torch.Tensor) -> torch.Tensor: - # Profiling / CUDA graph capture only; conv state is not updated. - inputs_t = inputs.transpose(0, 1).unsqueeze(0) - output = self.conv1d(inputs_t)[..., : inputs_t.size(-1)] - return F.silu(output).squeeze(0).transpose(0, 1) - - def _short_conv_dilated_decode_batched( - self, - x_d: torch.Tensor, - conv_state: torch.Tensor, - conv_weights: torch.Tensor, - state_indices_tensor_d: torch.Tensor, - has_initial_states_d: torch.Tensor | None, - ) -> torch.Tensor: - state_indices = state_indices_tensor_d.to( - device=conv_state.device, dtype=torch.int64 - ) - # FULL cudagraph padded decode rows use NULL_BLOCK_ID. Remap them to - # slot 0 for a safe gather, then zero output and skip write-back. - valid_state = state_indices != NULL_BLOCK_ID - state_indices = torch.where( - valid_state, state_indices, torch.zeros_like(state_indices) - ) - if has_initial_states_d is None: - has_initial_state = valid_state - else: - if has_initial_states_d.numel() < state_indices_tensor_d.numel(): - raise ValueError( - "has_initial_states_d size mismatch: " - f"got {has_initial_states_d.numel()}, " - f"need >= {state_indices_tensor_d.numel()}." - ) - has_initial_state = has_initial_states_d[ - : state_indices_tensor_d.numel() - ].to(device=conv_state.device, dtype=torch.bool) - has_initial_state = has_initial_state & valid_state - - cached_state = conv_state.index_select(0, state_indices) - state = cached_state[..., : self.conv_state_len].to(x_d.dtype) - if self.conv_state_len > 0: - initial_state = torch.where( - has_initial_state.view(-1, 1, 1), - state, - torch.zeros_like(state), - ) - history = torch.cat((initial_state, x_d.unsqueeze(-1)), dim=-1) - else: - history = x_d.unsqueeze(-1) - - conv_output = F.conv1d( - history, - conv_weights.unsqueeze(1).contiguous(), - groups=history.size(1), - dilation=self.short_conv_dilation, - ).squeeze(-1) - output = F.silu(conv_output) - output = output * valid_state.view(-1, 1).to(output.dtype) - - if self.conv_state_len > 0: - next_state = history[..., -self.conv_state_len :] - # Padded rows are remapped to the reserved null slot. Preserve its - # existing value while writing the new states for valid rows. - existing_base_state = cached_state[..., : self.conv_state_len] - safe_next_state = torch.where( - valid_state.view(-1, 1, 1), - next_state.to(conv_state.dtype), - existing_base_state, - ) - cached_state[..., : self.conv_state_len] = safe_next_state - conv_state.index_copy_(0, state_indices, cached_state) - - return output - - def _short_conv_dilated_prefill_batched( - self, - x_p: torch.Tensor, - metadata: PleShortConvAttentionMetadata, - conv_state: torch.Tensor, - conv_weights: torch.Tensor, - state_indices_tensor_p: torch.Tensor, - num_prefills: int, - num_decode_tokens: int, - num_prefill_tokens: int, - ) -> torch.Tensor: - query_start_loc_p = metadata.query_start_loc_p - if query_start_loc_p is None: - raise ValueError("query_start_loc is required for prefill short-conv") - # The metadata builder guarantees that the prefill query offsets start - # at 0 and end at num_prefill_tokens. Avoid reading those values here, - # since doing so would force a device-to-host synchronization. - has_initial_states_p = metadata.has_initial_states_p - if has_initial_states_p is None: - raise ValueError("has_initial_states_p is required for prefill short-conv") - - output = torch.empty_like(x_p) - q_starts = query_start_loc_p.to(torch.int64) - if state_indices_tensor_p.numel() < num_prefills: - raise ValueError( - "state_indices_tensor_p size mismatch: " - f"got {state_indices_tensor_p.numel()}, " - f"need >= {num_prefills}." - ) - if has_initial_states_p.numel() < num_prefills: - raise ValueError( - "has_initial_states_p size mismatch: " - f"got {has_initial_states_p.numel()}, " - f"need >= {num_prefills}." - ) - if num_prefills == 0 or x_p.numel() == 0: - return output - lengths = q_starts[1:] - q_starts[:-1] - # Use the CPU-computed packing width from the metadata builder instead - # of synchronizing on lengths.max(). - max_len = metadata.max_prefill_query_len - if max_len <= 0: - return output - - hidden_size = x_p.shape[1] - positions = torch.arange( - num_prefill_tokens, device=x_p.device, dtype=torch.int64 - ) - req_indices = torch.searchsorted(q_starts[1:], positions, right=True) - col_indices = positions - q_starts[req_indices] - - packed_tokens = x_p.new_zeros((num_prefills, max_len, hidden_size)) - packed_tokens[req_indices, col_indices] = x_p - packed_tokens = packed_tokens.transpose(1, 2).contiguous() - - state_indices = state_indices_tensor_p[:num_prefills].to( - device=conv_state.device, dtype=torch.int64 - ) - valid_state = state_indices != NULL_BLOCK_ID - state_indices = torch.where( - valid_state, state_indices, torch.zeros_like(state_indices) - ) - has_initial = has_initial_states_p[:num_prefills].to( - device=conv_state.device, dtype=torch.bool - ) - if self.conv_state_len > 0: - if conv_state.shape[0] == 0: - state = conv_state.new_zeros( - (num_prefills, hidden_size, self.conv_state_len), - dtype=x_p.dtype, - ) - else: - state = conv_state.index_select(0, state_indices)[ - ..., : self.conv_state_len - ].to(x_p.dtype) - use_initial_mask = (valid_state & has_initial).view(num_prefills, 1, 1) - initial_state = torch.where( - use_initial_mask, - state, - torch.zeros_like(state), - ) - history = torch.cat((initial_state, packed_tokens), dim=-1) - else: - history = packed_tokens - - conv_output = F.conv1d( - history, - conv_weights.unsqueeze(1).contiguous(), - groups=history.size(1), - dilation=self.short_conv_dilation, - ) - conv_output = F.silu(conv_output).transpose(1, 2).contiguous() - - token_positions = torch.arange(max_len, device=x_p.device, dtype=torch.int64) - valid_tokens = token_positions.view(1, max_len) < lengths.view(num_prefills, 1) - valid_output_mask = valid_tokens & valid_state.to(device=x_p.device).view( - num_prefills, 1 - ) - conv_output.masked_fill_(~valid_output_mask.unsqueeze(-1), 0) - output.copy_(conv_output[req_indices, col_indices]) - - if self.conv_state_len > 0 and conv_state.shape[0] > 0: - state_starts = lengths.to(device=history.device, dtype=torch.int64).view( - num_prefills, 1, 1 - ) - state_offsets = torch.arange( - self.conv_state_len, device=history.device, dtype=torch.int64 - ).view(1, 1, self.conv_state_len) - next_state = history.gather( - dim=2, - index=(state_starts + state_offsets).expand(-1, history.size(1), -1), - ) - # Write back without a host synchronization. Valid, non-empty rows - # receive their new state; padding and zero-length rows keep the - # current cache value. - existing_state = conv_state.index_select(0, state_indices) - existing_base_state = existing_state[..., : self.conv_state_len] - update_mask = valid_state & (lengths.to(device=conv_state.device) > 0) - safe_next_state = torch.where( - update_mask.view(num_prefills, 1, 1), - next_state.to(conv_state.dtype), - existing_base_state, - ) - existing_state[..., : self.conv_state_len] = safe_next_state - conv_state.index_copy_(0, state_indices, existing_state) - return output - - def _short_conv_dilated_spec_batched( - self, - x_spec: torch.Tensor, - conv_state: torch.Tensor, - conv_weights: torch.Tensor, - spec_state_indices_tensor: torch.Tensor, - spec_query_start_loc: torch.Tensor, - num_accepted_tokens: torch.Tensor, - spec_query_len: int, - ) -> torch.Tensor: - """Dilated short-conv for speculative-decode (MTP) requests. - - Each spec request feeds multiple (draft + 1) query tokens. The conv - outputs are computed causally after rolling back the previous draft - state by ``num_accepted_tokens - 1``. The current candidate inputs stay - in the extended cache for the next forward, matching - ``causal_conv1d_update``. - - ``spec_query_len`` (== num_speculative_tokens + 1) is the maximum query - length and is a Python int, so no host synchronization is needed; this - keeps the path safe for full CUDA-graph capture/replay where the buffers - are padded at the request level. - """ - num_reqs = spec_state_indices_tensor.numel() - hidden_size = x_spec.size(-1) - # Use a fixed packing width instead of synchronizing on lengths.max(). - max_len = spec_query_len - # Full CUDA graphs can pad these buffers. Only the first num_reqs - # accepted-token counts belong to actual speculative requests. - num_accepted_tokens = num_accepted_tokens[:num_reqs] - q_starts = spec_query_start_loc[: num_reqs + 1].to(torch.int64) - # Keep the number of real speculative tokens on the device. - total_real_tokens = q_starts[num_reqs] - - state_indices = spec_state_indices_tensor.to( - device=conv_state.device, dtype=torch.int64 - ) - valid_state = state_indices != NULL_BLOCK_ID - state_indices = torch.where( - valid_state, state_indices, torch.zeros_like(state_indices) - ) - positions = torch.arange( - x_spec.size(0), device=x_spec.device, dtype=torch.int64 - ) - # Route graph-padded token rows to the discarded dummy request so that - # they cannot overwrite real packed data. - req_indices = torch.searchsorted(q_starts[1:], positions, right=True) - valid_tokens = (positions < total_real_tokens) & (req_indices < num_reqs) - clamped_req_indices = req_indices.clamp_max(max(num_reqs - 1, 0)) - col_indices = (positions - q_starts[clamped_req_indices]).clamp_(0, max_len - 1) - pack_req_indices = torch.where( - valid_tokens, - clamped_req_indices, - torch.full_like(req_indices, num_reqs), - ) - pack_col_indices = torch.where( - valid_tokens, col_indices, torch.zeros_like(col_indices) - ) - - # The last request row is the dummy sink for graph padding. - packed = x_spec.new_zeros((num_reqs + 1, max_len, hidden_size)) - packed[pack_req_indices, pack_col_indices] = x_spec - packed = packed.transpose(1, 2).contiguous() - - if self.conv_state_len > 0: - cached_state = conv_state.index_select(0, state_indices) - rollback_offsets = num_accepted_tokens.to( - device=conv_state.device, dtype=torch.int64 - ).sub(1) - rollback_offsets = torch.where( - valid_state, - rollback_offsets.clamp_(0, max_len - 1), - torch.zeros_like(rollback_offsets), - ) - state_offsets = torch.arange( - self.conv_state_len, device=conv_state.device, dtype=torch.int64 - ).view(1, 1, self.conv_state_len) - rollback_indices = rollback_offsets.view(-1, 1, 1) + state_offsets - state = cached_state.gather( - 2, rollback_indices.expand(-1, hidden_size, -1) - ).to(x_spec.dtype) - state = torch.where( - valid_state.view(num_reqs, 1, 1), - state, - torch.zeros_like(state), - ) - # Append a zeroed dummy-row state to match the [num_reqs + 1] pack. - dummy_state = state.new_zeros((1, hidden_size, self.conv_state_len)) - state_full = torch.cat((state, dummy_state), dim=0) - history = torch.cat((state_full, packed), dim=-1) - else: - history = packed - - conv_output = F.conv1d( - history, - conv_weights.unsqueeze(1).contiguous(), - groups=history.size(1), - dilation=self.short_conv_dilation, - ) - conv_output = F.silu(conv_output).transpose(1, 2).contiguous() - - output = conv_output[pack_req_indices, pack_col_indices] - output = output * valid_tokens.view(-1, 1).to(output.dtype) - - # Keep all current candidate inputs in the extended state. On the next - # target forward, ``num_accepted_tokens - 1`` selects the rollback - # window before processing the newly scheduled tokens. - if self.conv_state_len > 0: - state_capacity = self.conv_state_len + max_len - 1 - if conv_state.size(-1) < state_capacity: - raise RuntimeError( - "PLE short-conv cache cannot retain speculative tokens: " - f"got {conv_state.size(-1)}, need {state_capacity}." - ) - candidate_state = history[:num_reqs, :, 1 : state_capacity + 1] - query_lengths = q_starts[1:] - q_starts[:-1] - state_positions = torch.arange( - state_capacity, device=history.device, dtype=torch.int64 - ).view(1, 1, state_capacity) - update_lengths = (self.conv_state_len + query_lengths - 1).view( - num_reqs, 1, 1 - ) - update_mask = valid_state.view(num_reqs, 1, 1) & ( - state_positions < update_lengths - ) - existing_state = cached_state[..., :state_capacity] - next_state = torch.where( - update_mask, - candidate_state.to(conv_state.dtype), - existing_state, - ) - cached_state[..., :state_capacity] = next_state - conv_state.index_copy_(0, state_indices, cached_state) - - return output - def _short_conv_dilated_dispatch( self, inputs: torch.Tensor, + residual: torch.Tensor, + outer_residual: torch.Tensor, metadata: PleShortConvAttentionMetadata, conv_state: torch.Tensor, conv_weights: torch.Tensor, - ) -> torch.Tensor: + ) -> None: num_prefills = metadata.num_prefills num_decodes = metadata.num_decodes num_decode_tokens = metadata.num_decode_tokens @@ -936,110 +492,145 @@ class Qwen4ExpPLELayer(nn.Module, MambaBase): has_prefill = num_prefills > 0 has_decode = num_decodes > 0 has_spec = metadata.spec_sequence_masks is not None - x = inputs[: metadata.num_actual_tokens] + has_non_spec = has_prefill or has_decode + inputs = inputs[: metadata.num_actual_tokens] + residual = residual[: metadata.num_actual_tokens] + outer_residual = outer_residual[: metadata.num_actual_tokens] - # Split spec / non-spec tokens. - if has_spec: - if has_prefill or has_decode: - assert metadata.spec_token_indx is not None - assert metadata.non_spec_token_indx is not None - x_spec = x.index_select(0, metadata.spec_token_indx.long()) - x_non_spec = x.index_select(0, metadata.non_spec_token_indx.long()) - else: - x_spec = x - x_non_spec = None - else: - x_spec = None - x_non_spec = x + spec_token_indices = None + non_spec_token_indices = None + if has_spec and has_non_spec: + assert metadata.spec_token_indx is not None + assert metadata.non_spec_token_indx is not None + spec_token_indices = metadata.spec_token_indx + non_spec_token_indices = metadata.non_spec_token_indx - spec_output = None - # 1. Run the multi-query speculative-decode part. if has_spec: assert metadata.spec_state_indices_tensor is not None - assert metadata.spec_query_start_loc is not None - assert metadata.num_accepted_tokens is not None - spec_output = self._short_conv_dilated_spec_batched( - x_spec=x_spec, + query_start_loc = metadata.spec_query_start_loc + num_accepted_tokens = metadata.num_accepted_tokens + assert query_start_loc is not None + assert num_accepted_tokens is not None + spec_state_indices = metadata.spec_state_indices_tensor[ + : metadata.num_spec_decodes + ] + # Mixed batches stay in their original row order; the kernels map + # logical spec/non-spec rows instead of materializing both groups. + ple_conv( + inputs=inputs, + residual=residual, conv_state=conv_state, conv_weights=conv_weights, - spec_state_indices_tensor=metadata.spec_state_indices_tensor[ - : metadata.num_spec_decodes - ], - spec_query_start_loc=metadata.spec_query_start_loc, - num_accepted_tokens=metadata.num_accepted_tokens, + state_indices=spec_state_indices, + outer_residual=outer_residual, + mode="spec", + dilation=self.short_conv_dilation, + query_start_loc=query_start_loc, + num_accepted_tokens=num_accepted_tokens, spec_query_len=metadata.spec_query_len, + token_indices=spec_token_indices, ) - # 2. Run regular decode and prefill requests. - conv_out_non_spec = None - state_indices_tensor = metadata.state_indices_tensor - if x_non_spec is not None: - assert state_indices_tensor is not None - if has_prefill: - state_indices_tensor_d, state_indices_tensor_p = torch.split( - state_indices_tensor, - [num_decodes, num_prefills], - dim=0, + if not has_non_spec: + return + + state_indices = metadata.state_indices_tensor + assert state_indices is not None + if has_prefill: + state_indices_d, state_indices_p = torch.split( + state_indices, [num_decodes, num_prefills], dim=0 + ) + if non_spec_token_indices is None: + inputs_d, inputs_p = torch.split( + inputs, [num_decode_tokens, num_prefill_tokens], dim=0 ) - x_d, x_p = torch.split( - x_non_spec, + residual_d, residual_p = torch.split( + residual, [num_decode_tokens, num_prefill_tokens], dim=0 + ) + outer_residual_d, outer_residual_p = torch.split( + outer_residual, [num_decode_tokens, num_prefill_tokens], dim=0, ) - non_spec_parts: list[torch.Tensor] = [] - if has_decode: - non_spec_parts.append( - self._short_conv_dilated_decode_batched( - x_d=x_d, - conv_state=conv_state, - conv_weights=conv_weights, - state_indices_tensor_d=state_indices_tensor_d, - has_initial_states_d=metadata.has_initial_states_d, - ) - ) - non_spec_parts.append( - self._short_conv_dilated_prefill_batched( - x_p=x_p, - metadata=metadata, - conv_state=conv_state, - conv_weights=conv_weights, - state_indices_tensor_p=state_indices_tensor_p, - num_prefills=num_prefills, - num_decode_tokens=num_decode_tokens, - num_prefill_tokens=num_prefill_tokens, - ) - ) - conv_out_non_spec = torch.vstack(non_spec_parts) + token_indices_d = None + token_indices_p = None else: - conv_out_non_spec = self._short_conv_dilated_decode_batched( - x_d=x_non_spec, + inputs_d = inputs_p = inputs + residual_d = residual_p = residual + outer_residual_d = outer_residual_p = outer_residual + token_indices_d, token_indices_p = torch.split( + non_spec_token_indices, + [num_decode_tokens, num_prefill_tokens], + dim=0, + ) + + if has_decode: + ple_conv( + inputs=inputs_d, + residual=residual_d, conv_state=conv_state, conv_weights=conv_weights, - state_indices_tensor_d=state_indices_tensor[: x_non_spec.size(0)], - has_initial_states_d=metadata.has_initial_states_d, + state_indices=state_indices_d, + outer_residual=outer_residual_d, + mode="decode", + dilation=self.short_conv_dilation, + has_initial_states=metadata.has_initial_states_d, + token_indices=token_indices_d, ) - # 3. Merge both parts back into the original token order. - if has_spec and conv_out_non_spec is not None: - assert metadata.spec_token_indx is not None - assert metadata.non_spec_token_indx is not None - assert spec_output is not None - output = x.new_empty((metadata.num_actual_tokens, x.size(-1))) - output.index_copy_(0, metadata.spec_token_indx, spec_output) - output.index_copy_(0, metadata.non_spec_token_indx, conv_out_non_spec) - return output - elif has_spec: - assert spec_output is not None - return spec_output - if conv_out_non_spec is None: - return x - return conv_out_non_spec + query_start_loc = metadata.query_start_loc_p + if query_start_loc is None: + raise ValueError("query_start_loc is required for prefill short-conv") + has_initial_states = metadata.has_initial_states_p + if has_initial_states is None: + raise ValueError( + "has_initial_states_p is required for prefill short-conv" + ) + ple_conv( + inputs=inputs_p, + residual=residual_p, + conv_state=conv_state, + conv_weights=conv_weights, + state_indices=state_indices_p, + outer_residual=outer_residual_p, + mode="prefill", + dilation=self.short_conv_dilation, + query_start_loc=query_start_loc, + has_initial_states=has_initial_states, + token_indices=token_indices_p, + ) + else: + num_decode_rows = ( + non_spec_token_indices.numel() + if non_spec_token_indices is not None + else inputs.size(0) + ) + ple_conv( + inputs=inputs, + residual=residual, + conv_state=conv_state, + conv_weights=conv_weights, + state_indices=state_indices[:num_decode_rows], + outer_residual=outer_residual, + mode="decode", + dilation=self.short_conv_dilation, + has_initial_states=metadata.has_initial_states_d, + token_indices=non_spec_token_indices, + ) - def _short_conv(self, inputs: torch.Tensor) -> torch.Tensor: + def _short_conv( + self, + inputs: torch.Tensor, + residual: torch.Tensor, + outer_residual: torch.Tensor, + ) -> None: forward_context = get_forward_context() attn_metadata = forward_context.attn_metadata + # Profiling omits all metadata or this Mamba entry. Short convolution + # is a no-op there, but preserve the outer residual addition. if attn_metadata is None: - return self._short_conv_fallback(inputs) + residual.add_(outer_residual) + return if not isinstance(attn_metadata, dict): raise RuntimeError( @@ -1049,8 +640,8 @@ class Qwen4ExpPLELayer(nn.Module, MambaBase): layer_attn_metadata = attn_metadata.get(self.prefix) if layer_attn_metadata is None: - # MRV2 omits Mamba-family metadata during profile warmup. - return self._short_conv_fallback(inputs) + residual.add_(outer_residual) + return if not isinstance(layer_attn_metadata, PleShortConvAttentionMetadata): raise TypeError( "Expected PleShortConvAttentionMetadata for layer " @@ -1059,6 +650,7 @@ class Qwen4ExpPLELayer(nn.Module, MambaBase): ) conv_state = self.kv_cache[0] + # Canonicalize both backend cache layouts to [slot, channel, window]. if not is_conv_state_dim_first(): conv_state = conv_state.transpose(-1, -2) conv_weights = self.conv1d.weight.squeeze(1) @@ -1072,11 +664,13 @@ class Qwen4ExpPLELayer(nn.Module, MambaBase): f"expect at least {state_capacity}." ) conv_state = conv_state[..., -state_capacity:] - return self._short_conv_dilated_dispatch( - inputs, - layer_attn_metadata, - conv_state, - conv_weights.to(dtype=inputs.dtype), + self._short_conv_dilated_dispatch( + inputs=inputs, + residual=residual, + outer_residual=outer_residual, + metadata=layer_attn_metadata, + conv_state=conv_state, + conv_weights=conv_weights.to(dtype=inputs.dtype), ) def forward( @@ -1095,27 +689,58 @@ class Qwen4ExpPLELayer(nn.Module, MambaBase): ) embeddings = self.ple_embedding(input_ids, query_start_loc, ngram_context) embeddings = self.ple_embedding.ngram_embedding.dequantize( - embeddings, - self.ple_embedding.ngram_embedding.params_dtype, + embeddings, hidden_states.dtype ) - key, _ = self.key_proj(embeddings) - value, _ = self.value_proj(embeddings) - token_count = hidden_states.shape[0] - key = key.reshape(token_count, self.hc_count, self.hidden_size) - query = hidden_states.reshape(token_count, self.hc_count, self.hidden_size) - key = self._apply_norm(self.norm_key, key) - query = self._apply_norm(self.norm_query, query) - gate = (key * query).sum(dim=-1, keepdim=True) / math.sqrt(self.hidden_size) - gate = torch.sigmoid(gate.sign() * gate.abs().clamp_min(1e-6).sqrt()) - gated_value = gate * value.unsqueeze(-2) - normalized = self._apply_norm(self.norm_conv, gated_value).flatten(-2) - conv_output = torch.zeros_like(normalized) + kv, _ = self.kv_proj(embeddings) + key, value = kv.split(self.kv_proj.output_sizes, dim=-1) + gated_output, conv_input = ple_gate( + key, + value, + hidden_states, + self.norm_key.weight, + self.norm_query.weight, + self.norm_conv.weight, + self.norm_key.eps, + ) + # The short-conv op is a piecewise-graph splitting op: state routing + # reads runtime request metadata. It accumulates the convolution and + # the outer residual into gated_output. torch.ops.vllm.qwen4_exp_ple_short_conv( - normalized, - conv_output, + conv_input, + gated_output, + hidden_states, self.prefix, ) - return gated_value.flatten(-2) + conv_output + return gated_output + + +def qwen4_exp_amd_ple_ngram_ids( + input_ids: torch.Tensor, + query_start_loc: torch.Tensor, + ngram_context: torch.Tensor, + output: torch.Tensor, + layer_name: str, +) -> None: + """Hash the current request layout into n-gram embedding indices. + + The launch sizes its request binary search from the request count, which + is symbolic under Dynamo, so it runs outside Inductor's FX graph. + """ + layer = get_forward_context().no_compile_layers[layer_name] + if not isinstance(layer, Qwen4ExpPLELayer): + raise TypeError(f"{layer_name} is not a Qwen4Exp PLE owner") + embedding = layer.ple_embedding + ple_ngram_ids( + input_ids=input_ids, + query_start_loc=query_start_loc, + ngram_context=ngram_context, + layer_multipliers=embedding.layer_multipliers, + ngram_heads_vocab_sizes=embedding.ngram_heads_vocab_sizes, + ngram_heads_offsets=embedding.ngram_heads_offsets, + eos_token_id=embedding.eos_token_id, + heads_per_ngram=embedding.heads_per_ngram, + output=output, + ) def qwen4_exp_amd_ple_ngram_embedding( @@ -1155,12 +780,19 @@ def qwen4_exp_amd_ple_ngram_embedding_pinned( def qwen4_exp_ple_short_conv( inputs: torch.Tensor, - output: torch.Tensor, + residual: torch.Tensor, + outer_residual: torch.Tensor, layer_name: str, ) -> None: layer = get_forward_context().no_compile_layers[layer_name] - result = layer._short_conv(inputs) - output[: result.shape[0]].copy_(result) + layer._short_conv(inputs, residual, outer_residual) + + +direct_register_custom_op( + op_name="qwen4_exp_amd_ple_ngram_ids", + op_func=qwen4_exp_amd_ple_ngram_ids, + mutates_args=["output"], +) direct_register_custom_op( @@ -1180,7 +812,7 @@ direct_register_custom_op( direct_register_custom_op( op_name="qwen4_exp_ple_short_conv", op_func=qwen4_exp_ple_short_conv, - mutates_args=["output"], + mutates_args=["residual"], ) diff --git a/vllm/models/qwen4_exp/amd/qsa.py b/vllm/models/qwen4_exp/amd/qsa.py index 325e0e3d91..ff01490bb1 100644 --- a/vllm/models/qwen4_exp/amd/qsa.py +++ b/vllm/models/qwen4_exp/amd/qsa.py @@ -116,8 +116,8 @@ class Qwen4ExpQSAFlashAttentionImpl(FlashAttentionImpl): self, layer: torch.nn.Module, query: torch.Tensor, - key: torch.Tensor, - value: torch.Tensor, + key: torch.Tensor | None, + value: torch.Tensor | None, kv_cache: torch.Tensor, attn_metadata: FlashAttentionMetadata, output: torch.Tensor, @@ -214,7 +214,7 @@ class Qwen4ExpQSAAttention(Qwen3NextAttention, AttentionLayerBase): elif tp_size % self.total_num_kv_heads: raise ValueError("TP size must be divisible by replicated QSA KV heads") self.num_kv_heads = max(1, self.total_num_kv_heads // tp_size) - self.head_dim = int(config.head_dim or self.hidden_size // self.num_heads) + self.head_dim = int(config.head_dim or self.hidden_size // self.total_num_heads) self.q_size = self.num_heads * self.head_dim self.kv_size = self.num_kv_heads * self.head_dim self.scaling = self.head_dim**-0.5 @@ -294,6 +294,11 @@ class Qwen4ExpQSAAttention(Qwen3NextAttention, AttentionLayerBase): quant_config=quant_config, prefix=f"{prefix}.indexer", ) + self.use_fused_qsa_prepare = ( + self.attn_output_gate + and self.indexer.use_fused_pre_indexer + and self.head_dim & (self.head_dim - 1) == 0 + ) max_tokens = vllm_config.scheduler_config.max_num_batched_tokens self.register_buffer( "topk_indices_buffer", @@ -327,16 +332,22 @@ class Qwen4ExpQSAAttention(Qwen3NextAttention, AttentionLayerBase): self, hidden_states: torch.Tensor, positions: torch.Tensor, - query: torch.Tensor, - key: torch.Tensor, - value: torch.Tensor, + query: torch.Tensor | None, + key: torch.Tensor | None, + value: torch.Tensor | None, output: torch.Tensor, + qkv: torch.Tensor | None = None, + gate: torch.Tensor | None = None, ) -> None: + # query/key/value are None when the fused prepare runs inside the + # indexer launch, which then writes the gate into ``gate``. metadata = get_forward_context().attn_metadata if isinstance(metadata, list): metadata = metadata[0] if not isinstance(metadata, dict): output.zero_() + if gate is not None: + gate.zero_() return main_metadata = cast(FlashAttentionMetadata, metadata[self.layer_name]) if self.kv_cache.numel() == 0: @@ -349,10 +360,14 @@ class Qwen4ExpQSAAttention(Qwen3NextAttention, AttentionLayerBase): ) if side_metadata.num_actual_tokens != num_tokens: raise RuntimeError("QSA main and side metadata token counts disagree") - selected = self.indexer( + selected, main_outputs = self.indexer( hidden_states, positions, self.topk_indices_buffer[:num_tokens], + attn=self, + qkv=qkv, + slot_mapping=main_metadata.slot_mapping, + gate_out=gate, ) if selected.shape != ( num_tokens, @@ -360,13 +375,22 @@ class Qwen4ExpQSAAttention(Qwen3NextAttention, AttentionLayerBase): ): raise RuntimeError("QSA indexer returned an invalid selection shape") impl = cast(Qwen4ExpQSAFlashAttentionImpl, self.impl) - impl.do_kv_cache_update( - self, - key, - value, - self.kv_cache, - main_metadata.slot_mapping, - ) + if main_outputs is None: + assert key is not None and value is not None + impl.do_kv_cache_update( + self, + key, + value, + self.kv_cache, + main_metadata.slot_mapping, + ) + else: + query, _ = main_outputs + # The gate is applied to every row outside this op; padded rows + # must stay finite so they keep the zero output forward_qsa gives. + assert gate is not None + gate[num_tokens:].zero_() + assert query is not None impl.forward_qsa( self, query, @@ -384,13 +408,40 @@ class Qwen4ExpQSAAttention(Qwen3NextAttention, AttentionLayerBase): hidden_states: torch.Tensor, ) -> torch.Tensor: qkv, _ = self.qkv_proj(hidden_states) - q, k, v, gate = self._project_qkv_gate(qkv, positions) num_tokens = hidden_states.shape[0] + encoded_layer_name = _encode_layer_name(self.layer_name) + if self.use_fused_qsa_prepare: + # Norm/RoPE/gate and the K/V cache write happen inside the op. + attn_output = qkv.new_empty(num_tokens, self.num_heads, self.head_dim) + gate = torch.empty_like(attn_output) + if current_platform.opaque_attention_op(): + torch.ops.vllm.qwen4_exp_qsa_prepare_with_output( + hidden_states, + positions, + qkv, + attn_output, + gate, + encoded_layer_name, + ) + else: + qwen4_exp_qsa_prepare_with_output( + hidden_states, + positions, + qkv, + attn_output, + gate, + encoded_layer_name, + ) + flat_output = attn_output.view(num_tokens, -1) + flat_output = flat_output * torch.sigmoid(gate.view(num_tokens, -1)) + output, _ = self.o_proj(flat_output) + return output + + q, k, v, gate = self._project_qkv_gate(qkv, positions) query = q.view(num_tokens, self.num_heads, self.head_dim) key = k.view(num_tokens, self.num_kv_heads, self.head_dim) value = v.view(num_tokens, self.num_kv_heads, self.head_dim) attn_output = torch.empty_like(query) - encoded_layer_name = _encode_layer_name(self.layer_name) if current_platform.opaque_attention_op(): torch.ops.vllm.qwen4_exp_qsa_with_output( hidden_states, @@ -449,10 +500,43 @@ direct_register_custom_op( ) +def qwen4_exp_qsa_prepare_with_output( + hidden_states: torch.Tensor, + positions: torch.Tensor, + qkv: torch.Tensor, + output: torch.Tensor, + gate: torch.Tensor, + layer_name: LayerNameType, +) -> None: + """Run the QSA transaction with the main QK-norm/RoPE/gate fused in.""" + layer_name = _resolve_layer_name(layer_name) + layer = get_forward_context().no_compile_layers[layer_name] + if not isinstance(layer, Qwen4ExpQSAAttention): + raise TypeError(f"{layer_name} is not a Qwen4Exp QSA owner") + layer._run_qsa( + hidden_states, + positions, + None, + None, + None, + output, + qkv=qkv, + gate=gate, + ) + + +direct_register_custom_op( + op_name="qwen4_exp_qsa_prepare_with_output", + op_func=qwen4_exp_qsa_prepare_with_output, + mutates_args=["output", "gate"], +) + + __all__ = [ "QSAIndexer", "Qwen4ExpQSAAttention", "Qwen4ExpQSAFlashAttentionBackend", "Qwen4ExpQSAFlashAttentionImpl", + "qwen4_exp_qsa_prepare_with_output", "qwen4_exp_qsa_with_output", ] diff --git a/vllm/models/qwen4_exp/common/ngram_embedding.py b/vllm/models/qwen4_exp/common/ngram_embedding.py index b387d3f364..1117748696 100644 --- a/vllm/models/qwen4_exp/common/ngram_embedding.py +++ b/vllm/models/qwen4_exp/common/ngram_embedding.py @@ -32,6 +32,7 @@ from vllm.model_executor.layers.quantization.modelopt import ( ModelOptMixedPrecisionConfig, ModelOptQuantConfigBase, ) +from vllm.model_executor.layers.quantization.quark.quark import QuarkConfig from vllm.model_executor.layers.quantization.utils.fp8_utils import ( create_fp8_scale_parameter, ) @@ -196,6 +197,9 @@ class Qwen4ExpPLEEmbeddingMethod(QuantizeMethodBase): and not quant_config.config_parser.resolve(None, prefix).quantized ): return Qwen4ExpPLEUnquantizedEmbeddingMethod() + # Quark quantizes only Linear and MoE layers; PLE tables stay BF16. + if isinstance(quant_config, QuarkConfig): + return Qwen4ExpPLEUnquantizedEmbeddingMethod() if not isinstance(quant_config, Fp8Config): raise NotImplementedError( "Qwen4Exp PLE embedding does not support quantization config " @@ -376,29 +380,33 @@ class Qwen4ExpPLEDeviceEmbedding(Qwen4ExpPLEEmbedding): @triton.jit def _lookup_ple_embedding_from_pinned_kernel( - weight_ptr, + weight_ptr: tl.pointer_type(tl.uint8), # type: ignore[valid-type] ids_ptr, - output_ptr, - embedding_dim, + output_ptr: tl.pointer_type(tl.uint8), # type: ignore[valid-type] + row_bytes, tp_vocab_start, tp_vocab_end, BLOCK_D: tl.constexpr, ): - """Look up TP-owned PLE rows through a CUDA view of pinned host memory.""" - row_id = tl.program_id(0) + """Copy TP-owned PLE rows as raw bytes from a CUDA view of pinned host memory. + + Byte pointers keep the storage dtype out of the kernel signature, so + dtypes Triton cannot lower on the GPU (FP8 E4M3FN before SM89) still work. + """ + row_id = tl.program_id(0).to(tl.int64) global_idx = tl.load(ids_ptr + row_id) in_range = (global_idx >= tp_vocab_start) & (global_idx < tp_vocab_end) local_idx = tl.where(in_range, global_idx - tp_vocab_start, 0) offsets = tl.arange(0, BLOCK_D) - store_mask = offsets < embedding_dim + store_mask = offsets < row_bytes load_mask = store_mask & in_range values = tl.load( - weight_ptr + local_idx * embedding_dim + offsets, + weight_ptr + local_idx * row_bytes + offsets, mask=load_mask, - other=0.0, + other=0, ) tl.store( - output_ptr + row_id * embedding_dim + offsets, + output_ptr + row_id * row_bytes + offsets, values, mask=store_mask, ) @@ -436,7 +444,8 @@ class Qwen4ExpPLEPinnedHostEmbedding(Qwen4ExpPLEEmbedding): data_parallel_rank=data_parallel_rank, ) self._uva_weight = get_accelerator_view_from_cpu_tensor(self.weight) - self._block_d = triton.next_power_of_2(self.embedding_dim) + self._row_bytes = self.embedding_dim * self.weight.element_size() + self._block_d = triton.next_power_of_2(self._row_bytes) self._prefetch_stream: torch.cuda.Stream | None = None self._prefetch_buffer: torch.Tensor | None = None self._prefetch_alloc_lock = threading.Lock() @@ -484,18 +493,11 @@ class Qwen4ExpPLEPinnedHostEmbedding(Qwen4ExpPLEEmbedding): flat_ids = input_ids.reshape(-1).long() if flat_ids.numel(): - if self.weight.dtype == torch.float8_e4m3fn: - uva_weight_view = self._uva_weight.view(torch.int8) - output_view = output.view(torch.int8) - else: - uva_weight_view = self._uva_weight - output_view = output - _lookup_ple_embedding_from_pinned_kernel[(flat_ids.numel(),)]( - uva_weight_view, + self._uva_weight, flat_ids, - output_view, - self.embedding_dim, + output, + self._row_bytes, self.shard_indices.org_vocab_start_index, self.shard_indices.org_vocab_end_index, BLOCK_D=self._block_d, diff --git a/vllm/models/qwen4_exp/common/ops/__init__.py b/vllm/models/qwen4_exp/common/ops/__init__.py new file mode 100644 index 0000000000..a87d2ca4be --- /dev/null +++ b/vllm/models/qwen4_exp/common/ops/__init__.py @@ -0,0 +1,4 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Qwen4Exp kernels shared by the NVIDIA and AMD backends; import leaf modules +directly.""" diff --git a/vllm/models/qwen4_exp/nvidia/ops/ple.py b/vllm/models/qwen4_exp/common/ops/ple.py similarity index 100% rename from vllm/models/qwen4_exp/nvidia/ops/ple.py rename to vllm/models/qwen4_exp/common/ops/ple.py diff --git a/vllm/models/qwen4_exp/common/qsa_cache.py b/vllm/models/qwen4_exp/common/qsa_cache.py index 3b34917361..ea37eb5cc6 100644 --- a/vllm/models/qwen4_exp/common/qsa_cache.py +++ b/vllm/models/qwen4_exp/common/qsa_cache.py @@ -127,7 +127,9 @@ def circular_qsa_slot_mapping( valid = (requests >= 0) & (requests < block_table.shape[0]) & (positions >= 0) safe_requests = requests.clamp(0, block_table.shape[0] - 1) physical_blocks = block_table[safe_requests, 0].long() - valid &= physical_blocks >= 0 + # Dummy and padding requests own no ring and sit on the null block, + # which must never be written. + valid &= physical_blocks > 0 slots = physical_blocks * compressor_state_size + positions.remainder( compressor_state_size ) @@ -288,7 +290,8 @@ def _build_qsa_metadata_kernel( mask=valid, other=-1, ) - valid &= physical_block >= 0 + # Dummy and padding requests own no ring and sit on the null block. + valid &= physical_block > 0 slot = physical_block * circular_buffer_size + ( logical_position % circular_buffer_size ) diff --git a/vllm/models/qwen4_exp/nvidia/low_latency_gemm.py b/vllm/models/qwen4_exp/nvidia/low_latency_gemm.py index 80bd4d2ddc..2ee1c7cb5a 100644 --- a/vllm/models/qwen4_exp/nvidia/low_latency_gemm.py +++ b/vllm/models/qwen4_exp/nvidia/low_latency_gemm.py @@ -13,6 +13,7 @@ from torch import nn import vllm.envs as envs from vllm.model_executor.kernels.linear.cute_dsl.skinny_gemm import ( SkinnyGemmConfig, + row_stride_ok, shape_dynamic_skinny_gemm, ) from vllm.model_executor.layers.linear import LinearBase, UnquantizedLinearMethod @@ -78,7 +79,7 @@ QWEN4_EXP_SM90_GEMM_PLANS: dict[tuple[int, int], dict[int, SkinnyGemmConfig]] = (4224, 2560): { 1: SkinnyGemmConfig(1, 128, 2, k_unroll=6, vector_width=2, static_k=2560), 2: SkinnyGemmConfig(2, 128, 2, k_unroll=3, vector_width=2, static_k=2560), - 4: SkinnyGemmConfig(4, 64, 4, k_unroll=2, vector_width=4), + 4: SkinnyGemmConfig(4, 64, 4, k_unroll=1, vector_width=8), }, # GDN and QSA output projections, TP=4. (2560, 1536): { @@ -167,7 +168,7 @@ QWEN4_EXP_SM100_GEMM_PLANS: dict[tuple[int, int], dict[int, SkinnyGemmConfig]] = } -# DGX Spark (GB10) plans, TP=1 unless noted. +# DGX Spark (GB10) plans for TP=1 and TP=2 local shapes. QWEN4_EXP_SM121_GEMM_PLANS: dict[tuple[int, int], dict[int, SkinnyGemmConfig]] = { # Shared-expert gate. (1, 2560): { @@ -177,6 +178,14 @@ QWEN4_EXP_SM121_GEMM_PLANS: dict[tuple[int, int], dict[int, SkinnyGemmConfig]] = 8: SkinnyGemmConfig(8, 128, 1, k_unroll=4, vector_width=4, static_k=2560), 16: SkinnyGemmConfig(16, 128, 1, vector_width=4, static_k=2560), }, + # GDN fused B/A projection, TP=2. + (48, 2560): { + 1: SkinnyGemmConfig(1, 128, 1, k_unroll=4, vector_width=4, static_k=2560), + 2: SkinnyGemmConfig(2, 128, 2, k_unroll=4, vector_width=4, static_k=2560), + 4: SkinnyGemmConfig(4, 128, 1, k_unroll=4, vector_width=4, static_k=2560), + 8: SkinnyGemmConfig(8, 128, 1, k_unroll=4, vector_width=4, static_k=2560), + 16: SkinnyGemmConfig(16, 128, 1, k_unroll=2, vector_width=4, static_k=2560), + }, # GDN fused B/A projection. (96, 2560): { 1: SkinnyGemmConfig(1, 128, 4, k_unroll=2, vector_width=2, static_k=2560), @@ -217,6 +226,12 @@ QWEN4_EXP_SM121_GEMM_PLANS: dict[tuple[int, int], dict[int, SkinnyGemmConfig]] = 8: SkinnyGemmConfig(8, 32, 1, k_unroll=4, vector_width=4, static_k=640), 16: SkinnyGemmConfig(16, 32, 1, k_unroll=2, vector_width=4, static_k=640), }, + # GDN and QSA output projections, TP=2. + (2560, 3072): { + 1: SkinnyGemmConfig(1, 128, 2, k_unroll=2, vector_width=4, static_k=3072), + 2: SkinnyGemmConfig(2, 64, 2, k_unroll=2, static_k=3072), + 4: SkinnyGemmConfig(4, 64, 2, k_unroll=2, static_k=3072), + }, # GDN and QSA output projections. (2560, 6144): { 1: SkinnyGemmConfig(1, 64, 2, vector_width=4, static_k=6144), @@ -225,6 +240,18 @@ QWEN4_EXP_SM121_GEMM_PLANS: dict[tuple[int, int], dict[int, SkinnyGemmConfig]] = 8: SkinnyGemmConfig(8, 128, 1, vector_width=2, static_k=6144), 16: SkinnyGemmConfig(16, 32, 1, static_k=6144), }, + # QSA fused QKV/gate projection, TP=2. + (6656, 2560): { + 1: SkinnyGemmConfig(1, 128, 4, k_unroll=2, vector_width=4, static_k=2560), + 2: SkinnyGemmConfig(2, 128, 4, k_unroll=2, vector_width=4, static_k=2560), + 4: SkinnyGemmConfig(4, 64, 2, k_unroll=4, vector_width=4, static_k=2560), + }, + # GDN fused QKVZ projection, TP=2. + (8192, 2560): { + 1: SkinnyGemmConfig(1, 128, 2, vector_width=4, static_k=2560), + 2: SkinnyGemmConfig(2, 64, 2, k_unroll=2, static_k=2560), + 4: SkinnyGemmConfig(4, 64, 2, k_unroll=4, vector_width=4, static_k=2560), + }, # HC up projection. (10240, 320): { 1: SkinnyGemmConfig(1, 32, 1, k_unroll=2, vector_width=2, static_k=320), @@ -248,6 +275,11 @@ QWEN4_EXP_SM121_GEMM_PLANS: dict[tuple[int, int], dict[int, SkinnyGemmConfig]] = 8: SkinnyGemmConfig(8, 64, 1, vector_width=2, static_k=2560), 16: SkinnyGemmConfig(16, 32, 1, k_unroll=2, vector_width=2, static_k=2560), }, + # LM head, TP=2. + (124160, 2560): { + 1: SkinnyGemmConfig(1, 128, 2, k_unroll=4, vector_width=4), + 2: SkinnyGemmConfig(2, 64, 2, k_unroll=2), + }, # LM head. (248320, 2560): { 1: SkinnyGemmConfig(1, 128, 1, k_unroll=2, vector_width=4, static_k=2560), @@ -273,10 +305,13 @@ def _is_packed_row_major(tensor: torch.Tensor) -> bool: return tensor.dim() == 2 and tensor.stride() == (tensor.shape[1], 1) -def _runtime_ok(x: torch.Tensor, weight: torch.Tensor) -> bool: +def _runtime_ok( + x: torch.Tensor, weight: torch.Tensor, config: SkinnyGemmConfig +) -> bool: return ( not envs.VLLM_BATCH_INVARIANT - and _is_packed_row_major(x) + and x.dim() == 2 + and row_stride_ok(x, config) and _is_packed_row_major(weight) and x.dtype == torch.bfloat16 and weight.dtype == torch.bfloat16 @@ -317,7 +352,7 @@ def _qwen4_exp_low_latency_gemm(x: torch.Tensor, weight: torch.Tensor) -> torch. config = None if plan is None else plan.get(x.shape[0]) if ( config is not None - and _runtime_ok(x, weight) + and _runtime_ok(x, weight, config) and shape_dynamic_skinny_gemm.is_available() ): return shape_dynamic_skinny_gemm(x, weight, config) diff --git a/vllm/models/qwen4_exp/nvidia/ngram_embedding.py b/vllm/models/qwen4_exp/nvidia/ngram_embedding.py index eedde22e6d..8dd6c6abb2 100644 --- a/vllm/models/qwen4_exp/nvidia/ngram_embedding.py +++ b/vllm/models/qwen4_exp/nvidia/ngram_embedding.py @@ -25,7 +25,7 @@ from ..common.ngram_embedding import ( Qwen4ExpPLEPinnedHostEmbedding, Qwen4ExpPLEUnquantizedEmbeddingMethod, ) -from .ops.ple import ple_ngram_ids +from ..common.ops.ple import ple_ngram_ids logger = init_logger(__name__) diff --git a/vllm/models/qwen4_exp/nvidia/ple_layer.py b/vllm/models/qwen4_exp/nvidia/ple_layer.py index 95f832a365..676c440c0f 100644 --- a/vllm/models/qwen4_exp/nvidia/ple_layer.py +++ b/vllm/models/qwen4_exp/nvidia/ple_layer.py @@ -24,8 +24,8 @@ from vllm.v1.attention.backends.short_conv_attn import ( PleShortConvAttentionMetadata, ) +from ..common.ops.ple import ple_conv, ple_gate from .ngram_embedding import Qwen4ExpNGramEmbedding -from .ops.ple import ple_conv, ple_gate class Qwen4ExpPLEGroupedNorm(nn.Module): diff --git a/vllm/models/qwen4_exp/nvidia/qsa.py b/vllm/models/qwen4_exp/nvidia/qsa.py index 6877189768..0246db8094 100644 --- a/vllm/models/qwen4_exp/nvidia/qsa.py +++ b/vllm/models/qwen4_exp/nvidia/qsa.py @@ -13,7 +13,7 @@ from transformers import Qwen4ExpTextConfig from vllm.compilation.breakable_cudagraph import eager_break_during_capture from vllm.config import VllmConfig -from vllm.config.cache import CacheDType +from vllm.config.cache import CacheConfig, CacheDType from vllm.distributed import get_tensor_model_parallel_world_size from vllm.forward_context import get_forward_context from vllm.model_executor.layers.attention.attention import ( @@ -29,7 +29,11 @@ from vllm.model_executor.layers.linear import ( from vllm.model_executor.layers.quantization import QuantizationConfig from vllm.model_executor.layers.rotary_embedding import MRotaryEmbedding, get_rope from vllm.model_executor.models.qwen3_next import Qwen3NextAttention -from vllm.model_executor.models.utils import AutoWeightsLoader, WeightsMapper +from vllm.model_executor.models.utils import ( + AutoWeightsLoader, + WeightsMapper, + extract_layer_index, +) from vllm.platforms import current_platform from vllm.platforms.interface import DeviceCapability from vllm.utils.torch_utils import ( @@ -125,6 +129,13 @@ class Qwen4ExpQSAQKVIndexerLinear(MergedColumnParallelLinear): return super().load_weights(remap_shards()) +def qsa_kv_cache_dtype(cache_config: CacheConfig, prefix: str) -> CacheDType: + """The layer's KV cache dtype, honoring ``--kv-cache-dtype-skip-layers``.""" + if str(extract_layer_index(prefix)) in cache_config.kv_cache_dtype_skip_layers: + return "auto" + return cache_config.cache_dtype + + class Qwen4ExpQSAMetadataBuilder(FlashAttentionMetadataBuilder): """Flash metadata supporting uniform decode and target-verify graphs.""" @@ -382,7 +393,7 @@ class Qwen4ExpQSAAttention(Qwen3NextAttention, AttentionLayerBase): elif tp_size % self.total_num_kv_heads: raise ValueError("TP size must be divisible by replicated QSA KV heads") self.num_kv_heads = max(1, self.total_num_kv_heads // tp_size) - self.head_dim = int(config.head_dim or self.hidden_size // self.num_heads) + self.head_dim = int(config.head_dim or self.hidden_size // self.total_num_heads) self.q_size = self.num_heads * self.head_dim self.kv_size = self.num_kv_heads * self.head_dim self.scaling = self.head_dim**-0.5 @@ -444,7 +455,7 @@ class Qwen4ExpQSAAttention(Qwen3NextAttention, AttentionLayerBase): self.layer_name = f"{prefix}.attn" self.attn_type = AttentionType.DECODER - self.kv_cache_dtype = cache_config.cache_dtype + self.kv_cache_dtype = qsa_kv_cache_dtype(cache_config, prefix) self.kv_cache_torch_dtype = kv_cache_dtype_str_to_dtype( self.kv_cache_dtype, model_config ) diff --git a/vllm/multimodal/image.py b/vllm/multimodal/image.py index ff2da044a6..13df7f3c9a 100644 --- a/vllm/multimodal/image.py +++ b/vllm/multimodal/image.py @@ -2,9 +2,64 @@ # SPDX-FileCopyrightText: Copyright contributors to the vLLM project import contextlib +from typing import Any from PIL import Image, ImageOps +# Formats accepted by Image.open for untrusted multimodal media. +# +# Audited against Pillow's registered open plugins (python -m PIL / Image.OPEN): +# Allowed: native still-image / raster readers that decode in-process. +# Denied: +# EPS — may invoke Ghostscript on untrusted PostScript +# WMF — Windows GDI / stub-handler path +# BUFR / GRIB / HDF5 — stub drivers requiring external handlers +# MPEG — identify-only; not a decodable still image +ALLOWED_IMAGE_FORMATS: tuple[str, ...] = ( + "AVIF", + "BLP", + "BMP", + "CUR", + "DCX", + "DDS", + "DIB", + "FITS", + "FLI", + "FTEX", + "GBR", + "GIF", + "ICNS", + "ICO", + "IM", + "IMT", + "IPTC", + "JPEG", + "JPEG2000", + "MCIDAS", + "MSP", + "PCD", + "PCX", + "PIXAR", + "PNG", + "PPM", + "PSD", + "QOI", + "SGI", + "SPIDER", + "SUN", + "TGA", + "TIFF", + "WEBP", + "XBM", + "XPM", + "XVTHUMB", +) + + +def open_image(fp: Any) -> Image.Image: + """Open image bytes/files with the serving format allowlist.""" + return Image.open(fp, formats=ALLOWED_IMAGE_FORMATS) + def rescale_image_size( image: Image.Image, size_factor: float, transpose: int = -1 diff --git a/vllm/multimodal/media/image.py b/vllm/multimodal/media/image.py index b63a12f824..fb05f4ed74 100644 --- a/vllm/multimodal/media/image.py +++ b/vllm/multimodal/media/image.py @@ -16,7 +16,7 @@ from vllm.utils.sparse_utils import ( safe_to_dense, ) -from ..image import convert_image_mode, normalize_image, rgba_to_rgb +from ..image import convert_image_mode, normalize_image, open_image, rgba_to_rgb from .base import MediaIO, MediaWithBytes MAGIC_NUMPY_PREFIX = b"\x93NUMPY" # https://numpy.org/devdocs/reference/generated/numpy.lib.format.html#format-version-1-0 @@ -78,7 +78,7 @@ class ImageMediaIO(MediaIO[Image.Image]): def load_bytes(self, data: bytes) -> MediaWithBytes[Image.Image]: try: - image = Image.open(BytesIO(data)) + image = open_image(BytesIO(data)) w, h = image.size max_pixels = envs.VLLM_MAX_IMAGE_PIXELS if max_pixels > 0 and w * h > max_pixels: diff --git a/vllm/multimodal/parse.py b/vllm/multimodal/parse.py index 98f8a8b2f3..546eb6abfd 100644 --- a/vllm/multimodal/parse.py +++ b/vllm/multimodal/parse.py @@ -13,11 +13,11 @@ from typing import ( TypeAlias, TypeGuard, TypeVar, + assert_never, ) import numpy as np import torch -from typing_extensions import assert_never from vllm.inputs import ModalityData, MultiModalDataDict, MultiModalUUIDDict from vllm.utils.collection_utils import is_list_of diff --git a/vllm/multimodal/processing/processor.py b/vllm/multimodal/processing/processor.py index 7267020353..abdb585402 100644 --- a/vllm/multimodal/processing/processor.py +++ b/vllm/multimodal/processing/processor.py @@ -13,10 +13,11 @@ from typing import ( NamedTuple, Protocol, TypeAlias, + assert_never, ) import torch -from typing_extensions import TypeVar, assert_never +from typing_extensions import TypeVar from vllm.config import SchedulerConfig from vllm.inputs import ( diff --git a/vllm/multimodal/video.py b/vllm/multimodal/video.py index 44a8e9452d..fc0cdc28d4 100644 --- a/vllm/multimodal/video.py +++ b/vllm/multimodal/video.py @@ -361,6 +361,15 @@ class Qwen3VLVideoBackend(VideoBackend): _MAX_FRAMES: ClassVar[int] = 768 _MAX_FPS: ClassVar[int] = 30 + @classmethod + def _prepare_source(cls, source: VideoSourceMetadata) -> VideoSourceMetadata: + # vLLM reports an unknown source fps as 0. HF's Qwen3VLVideoProcessor + # assumes a 24 fps source then, and keeps it in the metadata for the + # frame timestamps. + if source.original_fps > 0: + return source + return VideoSourceMetadata(source.total_frames_num, 24.0, source.duration) + @classmethod def compute_frames_index_to_sample( cls, @@ -368,6 +377,7 @@ class Qwen3VLVideoBackend(VideoBackend): target: VideoTargetMetadata, **kwargs, ) -> list[int]: + source = cls._prepare_source(source) total_frames_num = source.total_frames_num original_fps = source.original_fps fps = min(target.fps, cls._MAX_FPS) @@ -405,6 +415,184 @@ class Qwen3VLVideoBackend(VideoBackend): ) +@VIDEO_LOADER_REGISTRY.register( + "embedding_gemma2", + video_processor="EmbeddingGemma2VideoProcessor", +) +class EmbeddingGemma2VideoBackend(VideoBackend): + """Samples frames with the same indices as HF EmbeddingGemma2VideoProcessor.""" + + PROCESSOR_NAMES = ("EmbeddingGemma2Processor", "EmbeddingGemma2VideoProcessor") + + _MAX_ALLOWED_FRAMES: ClassVar[int] = 128 + _MAX_ALLOWED_FPS: ClassVar[float] = 60.0 + + @classmethod + def _get_processor_defaults(cls, **kwargs) -> tuple[float, int]: + """Read default fps and max_frames from video_processor if available.""" + proc = kwargs.get("video_processor") or kwargs.get("processor") + if proc is not None: + vp = getattr(proc, "video_processor", proc) + fps = getattr(vp, "fps", None) + max_frames = getattr(vp, "max_frames", None) + if fps is not None and max_frames is not None: + return float(fps), int(max_frames) + try: + from transformers.models.embedding_gemma2 import ( + EmbeddingGemma2VideoProcessor, + ) + + vp = EmbeddingGemma2VideoProcessor() + return float(getattr(vp, "fps", 1.0)), int(getattr(vp, "max_frames", 32)) + except Exception: + return 1.0, 32 + + @classmethod + def compute_frames_index_to_sample( + cls, + source: VideoSourceMetadata, + target: VideoTargetMetadata, + **kwargs, + ) -> list[int]: + # Ignores target.num_frames (EmbeddingGemma2 does not accept num_frames) + default_fps, default_max_frames = cls._get_processor_defaults(**kwargs) + + fps = kwargs.get("fps") + if fps is None: + fps = target.fps if target.fps > 0 else default_fps + if fps is not None: + if fps <= 0: + raise ValueError(f"fps must be positive, got {fps}") + if fps > cls._MAX_ALLOWED_FPS: + raise ValueError( + f"fps {fps} exceeds maximum allowed {cls._MAX_ALLOWED_FPS}" + ) + + max_frames = kwargs.get("max_frames", default_max_frames) + if max_frames is not None: + if max_frames <= 0: + raise ValueError(f"max_frames must be positive, got {max_frames}") + if max_frames > cls._MAX_ALLOWED_FRAMES: + raise ValueError( + f"max_frames {max_frames} exceeds maximum allowed " + f"{cls._MAX_ALLOWED_FRAMES}" + ) + + overflow_strategy = kwargs.get("overflow_strategy", "uniform") + + total_frames = source.total_frames_num + original_fps = source.original_fps + duration = source.duration + + if fps is not None and (original_fps <= 0 or duration <= 0): + fps = None + + if overflow_strategy is not None: + if max_frames is None: + raise ValueError( + "You must pass max_frames when requesting an " + f"overflow_strategy={overflow_strategy}!" + ) + if overflow_strategy not in ("truncate", "uniform"): + raise ValueError( + f"You passed overflow_strategy={overflow_strategy} but " + "expected one of ['truncate', 'uniform']" + ) + + if fps is None: + if overflow_strategy is None or total_frames <= max_frames: + indices = np.arange(total_frames, dtype=int) + elif overflow_strategy == "truncate": + indices = np.arange(max_frames, dtype=int) + elif overflow_strategy == "uniform": + indices = np.linspace(0, total_frames - 1, max_frames, dtype=int) + else: + step = original_fps / fps + num_sampled = max(1, int(duration * fps)) + + if overflow_strategy is None or num_sampled <= max_frames: + i_arr = np.arange(num_sampled, dtype=float) + indices = np.minimum(total_frames - 1, (i_arr * step).astype(int)) + elif overflow_strategy == "truncate": + i_arr = np.arange(max_frames, dtype=float) + indices = np.minimum(total_frames - 1, (i_arr * step).astype(int)) + elif overflow_strategy == "uniform": + linspace_idx = np.linspace(0, num_sampled - 1, max_frames, dtype=int) + indices = np.minimum( + total_frames - 1, (linspace_idx * step).astype(int) + ) + + return indices.tolist() + + @classmethod + def load_bytes( + cls, + data: bytes, + num_frames: int = -1, + fps: int | float = -1, + max_duration: int = 300, + frame_recovery: bool = False, + *, + backend: VideoDecoderBackend = "opencv", + **kwargs: Any, + ) -> tuple[DecodedFrames, dict[str, Any]]: + # Accept num_frames as required by VideoLoader interface (VideoMediaIO + # passes default num_frames=32), but ignore it since EmbeddingGemma2 uses + # fps/max_frames. Force num_frames=-1 so compute_frames_index_to_sample + # drives sampling via fps/duration. + kwargs.pop("num_frames", None) + frames, metadata = super().load_bytes( + data, + num_frames=-1, + fps=fps, # type: ignore[arg-type] + max_duration=max_duration, + frame_recovery=frame_recovery, + backend=backend, + **kwargs, + ) + + source = VideoSourceMetadata( + total_frames_num=metadata["total_num_frames"], + original_fps=metadata["fps"], + duration=metadata["duration"], + ) + target = VideoTargetMetadata(num_frames=-1, fps=fps, max_duration=max_duration) + sampling_kwargs, _ = resolve_video_backend_kwargs(backend, kwargs) + requested_indices = cls.compute_frames_index_to_sample( + source, target, **sampling_kwargs + ) + + # Expand deduplicated frames back to requested_indices if needed + # (e.g., OpenCV deduplicates frame_idx using set(frame_idx)) + valid_frame_indices = metadata.get("frames_indices") + if valid_frame_indices is not None and requested_indices != valid_frame_indices: + idx_to_pos = {idx: pos for pos, idx in enumerate(valid_frame_indices)} + if all(idx in idx_to_pos for idx in requested_indices): + positions = [idx_to_pos[idx] for idx in requested_indices] + if isinstance(frames, np.ndarray): + frames = frames[positions] + else: + frames = frames[torch.tensor(positions, device=frames.device)] + metadata["frames_indices"] = list(requested_indices) + + return frames, metadata + + @classmethod + def create_hf_metadata( + cls, + source: VideoSourceMetadata, + valid_frame_indices: list[int], + video_backend: str, + ) -> dict[str, Any]: + meta = super().create_hf_metadata( + source=source, + valid_frame_indices=valid_frame_indices, + video_backend=video_backend, + ) + meta["do_sample_frames"] = False + return meta + + @VIDEO_LOADER_REGISTRY.register( "qwen2_vl", video_processor="Qwen2VLVideoProcessor", diff --git a/vllm/outputs.py b/vllm/outputs.py index 35f71ec304..8e69ffa91d 100644 --- a/vllm/outputs.py +++ b/vllm/outputs.py @@ -212,6 +212,10 @@ class RequestOutput: completion.sampling_mask.token_ids.extend( next_completion.sampling_mask.token_ids ) + if next_completion.spec_decode_metrics is not None: + completion.spec_decode_metrics = ( + next_completion.spec_decode_metrics + ) completion.cumulative_logprob = ( next_completion.cumulative_logprob ) diff --git a/vllm/parser/abstract_parser.py b/vllm/parser/abstract_parser.py index bee090e7f6..4af133c988 100644 --- a/vllm/parser/abstract_parser.py +++ b/vllm/parser/abstract_parser.py @@ -42,7 +42,11 @@ from vllm.parser.engine.adapters import ParserEngineToolAdapter from vllm.parser.metrics import record_tool_parser_invocation from vllm.parser.utils import count_history_tool_calls from vllm.reasoning.abs_reasoning_parsers import ReasoningParser -from vllm.sampling_params import SamplingParams, StructuredOutputsParams +from vllm.sampling_params import ( + SamplingParams, + StructuredOutputsParams, + check_json_nesting, +) from vllm.tokenizers import TokenizerLike from vllm.tool_parsers.abstract_tool_parser import Tool, ToolParser from vllm.tool_parsers.streaming import ( @@ -50,6 +54,7 @@ from vllm.tool_parsers.streaming import ( extract_required_tool_call_streaming, ) from vllm.tool_parsers.structural_tag_registry import ( + get_structural_tag_tools, limit_to_single_tool_call, resolve_tool_strictness, ) @@ -273,6 +278,7 @@ def structured_outputs_to_format(params: StructuredOutputsParams) -> Format | No if params.json is not None: schema = params.json if isinstance(schema, str): + check_json_nesting(schema) schema = json.loads(schema) return JSONSchemaFormat(json_schema=schema) if params.regex is not None: @@ -294,6 +300,7 @@ def structured_outputs_to_format(params: StructuredOutputsParams) -> Format | No raise VLLMValidationError("Invalid grammar specification.") from e return GrammarFormat(grammar=grammar) if params.structural_tag is not None: + check_json_nesting(params.structural_tag, structural_tag=True) s_tag = json.loads(params.structural_tag) if "structures" in s_tag: # LegacyStructuralTagResponseFormat @@ -520,7 +527,11 @@ class DelegatingParser(Parser): resolved_tools = None if tool_parser.structural_tag_model is not None: resolved_tools = resolve_tool_strictness( - request.tools, + get_structural_tag_tools( + tool_parser.structural_tag_model, + request.tools, + request.tool_choice, + ), request.tool_choice, strict_level, ) @@ -730,8 +741,6 @@ class DelegatingParser(Parser): extract_required_tool_call_streaming( previous_text=previous_text, current_text=current_text, - delta_text=delta_text, - function_name_returned=function_name_returned, tool_call_idx=tool_call_idx, tool_call_id_type=tool_call_id_type, ) @@ -893,8 +902,17 @@ class DelegatingParser(Parser): (delta_message.content if delta_message else None) or "" ) + ((flush_delta.content if flush_delta else None) or "") if self._engine_based: - if delta_message and self._tool_parser is not None: - delta_message.content = None + if self._tool_parser is not None: + # The tool-call phase below consumes current_text. + if delta_message: + delta_message.content = None + elif current_text: + # No tool phase follows to consume the flushed text, + # and the engine is not fed again after the + # transition, so emit it as content here. + if delta_message is None: + delta_message = DeltaMessage() + delta_message.content = current_text else: delta_text = current_text @@ -932,12 +950,13 @@ class DelegatingParser(Parser): elif not delta_message.reasoning: delta_message.reasoning = reasoning_from_this_batch - if ( - delta_message - and delta_message.tool_calls - and delta_message.tool_calls[0].id is not None - ): - state.history_tool_call_cnt += 1 + if delta_message and delta_message.tool_calls: + # A delta may start several tool calls at once (e.g. the + # "required" path when one delta completes more than one + # call), so count every id rather than only the first. + state.history_tool_call_cnt += sum( + tc.id is not None for tc in delta_message.tool_calls + ) # No phase active: pass through as content. # Skip when reasoning just ended in this delta — the engine already diff --git a/vllm/parser/cohere_command.py b/vllm/parser/cohere_command.py index a1df491401..97a9d3d6f3 100644 --- a/vllm/parser/cohere_command.py +++ b/vllm/parser/cohere_command.py @@ -45,7 +45,7 @@ from vllm.reasoning.cohere_command_reasoning_parser import ( BaseCohereCommandReasoningParser, ) from vllm.renderers.cohere import POSITION_TO_SOURCE_KEY -from vllm.sampling_params import StructuredOutputsParams +from vllm.sampling_params import StructuredOutputsParams, check_json_nesting from vllm.tool_parsers.cohere_command_tool_parser import BaseCohereCommandToolParser if TYPE_CHECKING: @@ -127,6 +127,7 @@ def collect_tool_schema(tool_schema: list[CohereNormalizedTool]) -> str: tool_dictionary: dict[str, str] = {} for tool in tool_schema: tool_name = tool["name"] + check_json_nesting(tool["parameters"]) tool_parameters = dict(tool["parameters"]) json_schema: dict[str, Any] = { "type": "object", @@ -404,6 +405,7 @@ def _schema_dict_from_structured_outputs( if isinstance(raw, str): if not raw.strip(): raise ValueError("structured_outputs.json cannot be empty.") + check_json_nesting(raw) try: raw = json.loads(raw) except json.JSONDecodeError as e: diff --git a/vllm/parser/engine/adapters.py b/vllm/parser/engine/adapters.py index 4f4aa9f5a1..cd7aaa7297 100644 --- a/vllm/parser/engine/adapters.py +++ b/vllm/parser/engine/adapters.py @@ -27,7 +27,7 @@ if TYPE_CHECKING: ChatCompletionRequest, ) from vllm.entrypoints.openai.responses.protocol import ResponsesRequest - from vllm.parser.engine.parser_engine import ParserEngine + from vllm.parser.engine.parser_engine import ParserEngine, ReasoningEnd from vllm.tokenizers import TokenizerLike from vllm.tool_parsers.utils import Tool @@ -65,8 +65,8 @@ class ParserEngineReasoningAdapter(ReasoningParser): def is_reasoning_end(self, input_ids: Sequence[int]) -> bool: return self._parser_engine.is_reasoning_end(list(input_ids)) - def find_reasoning_end_offset(self, token_ids: Sequence[int]) -> int | None: - return self._parser_engine.find_reasoning_end_offset(token_ids) + def find_reasoning_end(self, token_ids: Sequence[int]) -> ReasoningEnd | None: + return self._parser_engine.find_reasoning_end(token_ids) def adjust_initial_state_from_prompt(self, prompt_token_ids: Sequence[int]) -> None: self._parser_engine.adjust_initial_state_from_prompt(prompt_token_ids) diff --git a/vllm/parser/engine/parser_engine.py b/vllm/parser/engine/parser_engine.py index 527c310c14..b04fdaa794 100644 --- a/vllm/parser/engine/parser_engine.py +++ b/vllm/parser/engine/parser_engine.py @@ -9,7 +9,7 @@ from __future__ import annotations import json from collections.abc import Sequence from functools import cached_property -from typing import TYPE_CHECKING +from typing import TYPE_CHECKING, NamedTuple import regex as re @@ -45,6 +45,13 @@ if TYPE_CHECKING: logger = init_logger(__name__) +class ReasoningEnd(NamedTuple): + """Where reasoning ends in a token sequence.""" + + offset: int # index of the first end token, or the sequence length + implicit: bool # the end token is also content, e.g. a tool-call start + + class ToolCallSlot: __slots__ = ( "id", @@ -638,14 +645,15 @@ class ParserEngine(Parser): def reasoning_end_token_ids(self) -> frozenset[int]: return self._reasoning_end_token_ids - def find_reasoning_end_offset(self, token_ids: Sequence[int]) -> int | None: + def find_reasoning_end(self, token_ids: Sequence[int]) -> ReasoningEnd | None: + """First reasoning end in `token_ids`, and if its token is content.""" end_ids = self._reasoning_end_token_ids if not end_ids: return None for offset, token_id in enumerate(token_ids): if token_id in end_ids: - return offset - return len(token_ids) + return ReasoningEnd(offset, token_id != self._reasoning_end_token_id) + return ReasoningEnd(len(token_ids), False) def is_reasoning_end(self, input_ids: list[int]) -> bool: config = self.parser_engine_config diff --git a/vllm/parser/engine/registered_adapters.py b/vllm/parser/engine/registered_adapters.py index 31295869c5..6d610ce3b5 100644 --- a/vllm/parser/engine/registered_adapters.py +++ b/vllm/parser/engine/registered_adapters.py @@ -19,6 +19,7 @@ from vllm.parser.inkling import InklingParser from vllm.parser.kimi_k2 import KimiK2Parser from vllm.parser.mimo import MiMoParser from vllm.parser.minimax_m2 import MinimaxM2Parser +from vllm.parser.minimax_m3 import MinimaxM3Parser from vllm.parser.mistral import MistralParser from vllm.parser.nemotron_v3 import NemotronV3Parser from vllm.parser.plamo3 import Plamo3Parser @@ -46,6 +47,11 @@ from vllm.parser.step3p5 import Step3p5Parser MinimaxM2ParserToolAdapter, ) = make_adapters(MinimaxM2Parser) +( + MinimaxM3ParserReasoningAdapter, + MinimaxM3ParserToolAdapter, +) = make_adapters(MinimaxM3Parser) + ( Gemma4ParserReasoningAdapter, Gemma4ParserToolAdapter, diff --git a/vllm/parser/harmony.py b/vllm/parser/harmony.py index 6a1399db66..68b50250de 100644 --- a/vllm/parser/harmony.py +++ b/vllm/parser/harmony.py @@ -472,7 +472,7 @@ def _assemble_tag( return StructuralTag(format=SequenceFormat(elements=tags)) -@register_vllm_structural_tag("harmony") +@register_vllm_structural_tag("harmony", builtin_tools=True) def get_harmony_structural_tag( tools: list[FunctionToolParam], builtin_tools: list[BuiltinToolParam], diff --git a/vllm/parser/minimax_m3.py b/vllm/parser/minimax_m3.py new file mode 100644 index 0000000000..dafe1457e9 --- /dev/null +++ b/vllm/parser/minimax_m3.py @@ -0,0 +1,293 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""MiniMax M3 parser for namespace-delimited XML-style tool calls. + +MiniMax M3 prefixes every structural tag with the ``]<]minimax[>[`` marker:: + + ]<]minimax[>[ + ]<]minimax[>[ + ]<]minimax[>[42]<]minimax[>[ + ]<]minimax[>[ + ]<]minimax[>[]<]minimax[>[book-001]<]minimax[>[]<]minimax[>[ + ]<]minimax[>[ + ]<]minimax[>[ + ]<]minimax[>[ + +Each ```` becomes one tool call. Parameters are elements named after +the argument and may nest: an element holding child elements becomes an +object, or an array when its schema says so (the chat template renders list +items as ````). Arguments are emitted once the whole ```` is +parsed. Reasoning is handled by ``MiniMaxM3ReasoningParser``. +""" + +from __future__ import annotations + +import functools +import json +from typing import TYPE_CHECKING, Any + +from vllm.parser.engine.events import EventType +from vllm.parser.engine.parser_engine import ParserEngine +from vllm.parser.engine.parser_engine_config import ( + ParserEngineConfig, + ParserState, + Transition, +) +from vllm.tool_parsers.utils import ( + coerce_to_schema_type, + extract_types_from_schema, + find_tool_properties, +) + +if TYPE_CHECKING: + from vllm.entrypoints.openai.chat_completion.protocol import ( + ChatCompletionRequest, + ) + from vllm.entrypoints.openai.responses.protocol import ResponsesRequest + from vllm.tokenizers import TokenizerLike + from vllm.tool_parsers.abstract_tool_parser import Tool + +NAMESPACE = "]<]minimax[>[" +TOOL_CALL_START = f"{NAMESPACE}" +TOOL_CALL_END = f"{NAMESPACE}" +INVOKE_PREFIX_DQ = f'{NAMESPACE}' +NAME_END_SQ = "'>" +NAME_END_UNQUOTED = ">" +ELEMENT_START = f"{NAMESPACE}<" +ELEMENT_END_START = f"{NAMESPACE} tuple[str, _Element, int] | None: + """Parse the element starting at ``pos``; ``None`` if malformed.""" + if depth > MAX_ELEMENT_DEPTH: + return None + name_start = pos + len(ELEMENT_START) + name_end = text.find(">", name_start) + if name_end <= name_start: + return None + name = text[name_start:name_end] + if name.startswith("/") or not name.strip(): + return None + + close_tag = f"{ELEMENT_END_START}{name}>" + pos = name_end + 1 + text_parts: list[str] = [] + children: list[tuple[str, _Element]] = [] + while True: + marker = text.find(NAMESPACE, pos) + if marker < 0: + return None + text_parts.append(text[pos:marker]) + if text.startswith(close_tag, marker): + pos = marker + len(close_tag) + break + if not text.startswith(ELEMENT_START, marker): + return None + child = _parse_element(text, marker, depth + 1) + if child is None: + return None + child_name, child_value, pos = child + children.append((child_name, child_value)) + + body_text = "".join(text_parts) + if not children: + return name, body_text, pos + if body_text.strip(): + # Keep mixed text under a reserved field that no child uses. + field = MIXED_TEXT_FIELD + while any(child_name == field for child_name, _ in children): + field = "$" + field + children.append((field, body_text)) + return name, children, pos + + +def _parse_invoke_params(raw_args: str) -> list[tuple[str, _Element]]: + """Parse the parameter elements of one ```` body. + + Parsing stops at the first malformed or unterminated element, keeping the + parameters before it. + """ + params: list[tuple[str, _Element]] = [] + pos = 0 + while True: + while pos < len(raw_args) and raw_args[pos].isspace(): + pos += 1 + if not raw_args.startswith(ELEMENT_START, pos): + return params + element = _parse_element(raw_args, pos, depth=1) + if element is None: + return params + name, value, pos = element + params.append((name, value)) + + +def _container_schema(schema: dict[str, Any]) -> tuple[str, dict[str, Any]] | None: + """Return the first ``array`` or ``object`` alternative of ``schema``.""" + types = schema.get("type") + for kind in ("array", "object"): + if types == kind or (isinstance(types, list) and kind in types): + return kind, schema + if types is None: + if "items" in schema: + return "array", schema + if "properties" in schema or "additionalProperties" in schema: + return "object", schema + for choice_field in ("anyOf", "oneOf"): + for choice in schema.get(choice_field) or (): + if isinstance(choice, dict) and (found := _container_schema(choice)): + return found + return None + + +def _convert_element(value: _Element, schema: Any) -> Any: + if not isinstance(schema, dict): + schema = None + container = _container_schema(schema) if schema is not None else None + + if isinstance(value, str): + if schema is None: + return value + types = extract_types_from_schema(schema) + # The template renders an empty list or dict as an empty element. + if container is not None and "string" not in types and not value.strip(): + return [] if container[0] == "array" else {} + return coerce_to_schema_type(value, types) + + if container is not None and container[0] == "array": + items = container[1].get("items") + return [_convert_element(child, items) for _, child in value] + return _convert_object(value, container[1] if container is not None else None) + + +def _convert_object( + elements: list[tuple[str, _Element]], schema: dict[str, Any] | None +) -> dict[str, Any]: + """Convert named elements to an object; repeated names collect into a list.""" + properties = (schema or {}).get("properties") or {} + additional = (schema or {}).get("additionalProperties") + result: dict[str, Any] = {} + repeated: set[str] = set() + for name, element in elements: + converted = _convert_element(element, properties.get(name, additional)) + if name not in result: + result[name] = converted + elif name in repeated: + result[name].append(converted) + else: + result[name] = [result[name], converted] + repeated.add(name) + return result + + +def _minimax_m3_arg_converter( + raw_args: str, + partial: bool, + properties: dict[str, Any] | None = None, +) -> str: + params = _parse_invoke_params(raw_args) + arguments = _convert_object(params, {"properties": properties or {}}) + return json.dumps(arguments, ensure_ascii=False) + + +@functools.cache +def minimax_m3_config() -> ParserEngineConfig: + return ParserEngineConfig( + name="minimax_m3", + initial_state=ParserState.CONTENT, + terminals={ + "TOOL_START": TOOL_CALL_START, + "TOOL_END": TOOL_CALL_END, + "INVOKE_PREFIX_DQ": INVOKE_PREFIX_DQ, + "INVOKE_PREFIX_SQ": INVOKE_PREFIX_SQ, + "INVOKE_PREFIX_UNQUOTED": INVOKE_PREFIX_UNQUOTED, + "INVOKE_END": INVOKE_END, + "NAME_END_DQ": NAME_END_DQ, + "NAME_END_SQ": NAME_END_SQ, + "NAME_END_UNQUOTED": NAME_END_UNQUOTED, + }, + transitions={ + (ParserState.CONTENT, "TOOL_START"): Transition( + ParserState.TOOL_PREAMBLE, + (), + ), + **{ + (state, terminal): Transition( + ParserState.TOOL_NAME, + (EventType.TOOL_CALL_START,), + ) + for state in (ParserState.TOOL_PREAMBLE, ParserState.TOOL_BETWEEN) + for terminal in ( + "INVOKE_PREFIX_DQ", + "INVOKE_PREFIX_SQ", + "INVOKE_PREFIX_UNQUOTED", + ) + }, + **{ + (ParserState.TOOL_NAME, terminal): Transition( + ParserState.TOOL_ARGS, + (), + ) + for terminal in ("NAME_END_DQ", "NAME_END_SQ", "NAME_END_UNQUOTED") + }, + # Nested parameter elements are argument text; only the invoke + # closer ends the call. + (ParserState.TOOL_ARGS, "INVOKE_END"): Transition( + ParserState.TOOL_BETWEEN, + (EventType.TOOL_CALL_END,), + ), + # The tool block ends the message: stay in TOOL_BETWEEN so any + # text after it is dropped. + (ParserState.TOOL_PREAMBLE, "TOOL_END"): Transition( + ParserState.TOOL_BETWEEN, + (), + ), + (ParserState.TOOL_BETWEEN, "TOOL_END"): Transition( + ParserState.TOOL_BETWEEN, + (), + ), + }, + arg_converter=_minimax_m3_arg_converter, + # Emit arguments once per ``: converted values are not + # prefix-stable while elements arrive, e.g. a repeated parameter name + # turns an already streamed scalar into a list. + stream_arg_deltas=False, + tool_args_json=False, + strip_content_whitespace_with_tools=False, + ) + + +class MinimaxM3Parser(ParserEngine): + """MiniMax M3 tool-call parser backed by the declarative parser engine.""" + + def __init__( + self, + tokenizer: TokenizerLike, + tools: list[Tool] | None = None, + **kwargs, + ) -> None: + kwargs.setdefault("parser_engine_config", minimax_m3_config()) + super().__init__(tokenizer, tools, **kwargs) + self._arg_converter = self._convert_args + + def adjust_request( + self, request: ChatCompletionRequest | ResponsesRequest + ) -> ChatCompletionRequest | ResponsesRequest: + # The M3 markers are non-special added tokens, so they decode with + # special tokens skipped. + return request + + def _convert_args(self, raw_args: str, partial: bool) -> str: + func_name = next( + (slot.name for slot in self._tool_slots if slot.args == raw_args), "" + ) + properties = find_tool_properties(self._tools, func_name.strip()) + return _minimax_m3_arg_converter(raw_args, partial, properties) diff --git a/vllm/parser/mistral.py b/vllm/parser/mistral.py index 247ef2803f..f6d3435bd7 100644 --- a/vllm/parser/mistral.py +++ b/vllm/parser/mistral.py @@ -48,7 +48,7 @@ from vllm.parser.engine.parser_engine_config import ( ParserState, Transition, ) -from vllm.sampling_params import StructuredOutputsParams +from vllm.sampling_params import StructuredOutputsParams, check_json_nesting from vllm.utils.mistral import is_mistral_tokenizer if TYPE_CHECKING: @@ -117,6 +117,37 @@ def _is_pre_v11_tokeniser(model_tokenizer: TokenizerLike) -> bool: return _ARGS not in vocab +def _pre_v11_function_calls(parsed: Any) -> list[FunctionCall]: + """Decode the JSON after a pre-v11 ``[TOOL_CALLS]`` marker. + + Mirrors the streaming path: a bare object is a single call, entries that + are not call objects are skipped, a missing or non-string name becomes + ``""`` (as in the v11+ path) and string arguments are kept as-is. + """ + if isinstance(parsed, dict): + parsed = [parsed] + if not isinstance(parsed, list): + return [] + function_calls: list[FunctionCall] = [] + for tool_call in parsed: + if not isinstance(tool_call, dict) or not ( + "name" in tool_call or "arguments" in tool_call + ): + continue + name = tool_call.get("name") + arguments = tool_call.get("arguments") + if arguments is None: + arguments = "{}" + elif not isinstance(arguments, str): + arguments = json.dumps(arguments, ensure_ascii=False) + function_calls.append( + FunctionCall( + name=name if isinstance(name, str) else "", arguments=arguments + ) + ) + return function_calls + + @functools.cache def mistral_config( *, @@ -301,6 +332,7 @@ class MistralParser(ParserEngine): self.starting_new_tool: bool = False self.streamed_args_for_tool: list[str] = [] self._is_pre_v11: bool = _is_pre_v11_tokeniser(tokenizer) + self.current_tool_string_arguments: str | None = None self.parse_coro = None if self._is_pre_v11: self.parse_coro = ijson.parse_coro( @@ -381,6 +413,7 @@ class MistralParser(ParserEngine): json_schema = _DEFAULT_JSON_SCHEMA elif request.structured_outputs.json is not None: if isinstance(request.structured_outputs.json, str): + check_json_nesting(request.structured_outputs.json) json_schema = json.loads(request.structured_outputs.json) else: json_schema = request.structured_outputs.json @@ -550,16 +583,13 @@ class MistralParser(ParserEngine): content: str | None = None if self.bot_token in model_output: - content_and_raw_tool_calls = model_output.split(self.bot_token) - content = content_and_raw_tool_calls[0] - raw_tool_calls = content_and_raw_tool_calls[1:] # pre-v11: content[BOT] [{tool_call1},{tool_call2}] - if len(raw_tool_calls) != 1: - raise ValueError( - "Only one BOT token should have been outputted, " - f"but got {model_output}." - ) - stringified_tool_calls = raw_tool_calls[0].strip() + # Repeated BOT tokens with nothing between them count as one, and + # output after a later BOT token is trailing text, as in streaming. + content, *segments = model_output.split(self.bot_token) + stringified_tool_calls = next( + (segment.strip() for segment in segments if segment.strip()), "" + ) elif tool_choice == "required" or isinstance( tool_choice, ChatCompletionNamedToolChoiceParam ): @@ -574,49 +604,31 @@ class MistralParser(ParserEngine): # Use raw_decode to parse the first valid JSON value, # ignoring trailing tokens the model may emit after # the tool call array. - tool_calls, _ = json.JSONDecoder().raw_decode(stringified_tool_calls) + parsed, _ = json.JSONDecoder().raw_decode(stringified_tool_calls) except json.JSONDecodeError: try: raw_tool_call = self.tool_call_regex.findall(stringified_tool_calls)[0] - tool_calls = json.loads(raw_tool_call) - tool_calls = [ - { - "name": tool_call["name"], - "arguments": json.dumps( - tool_call.get("arguments", {}), - ensure_ascii=False, - ), - } - for tool_call in tool_calls - ] + parsed = json.loads(raw_tool_call) except (IndexError, json.JSONDecodeError): logger.exception("Error in extracting tool call from response.") - return ExtractedToolCallInformation( - tools_called=False, - tool_calls=[], - content=stringified_tool_calls, + parsed = None + + function_calls = _pre_v11_function_calls(parsed) + if not function_calls: + if parsed is not None: + logger.warning( + "No tool call found in the JSON after %s; returning it as content.", + self.bot_token, ) - else: - tool_calls = [ - { - "name": tool_call["name"], - "arguments": json.dumps( - tool_call.get("arguments", {}), - ensure_ascii=False, - ), - } - for tool_call in tool_calls - ] + return ExtractedToolCallInformation( + tools_called=False, + tool_calls=[], + content=stringified_tool_calls, + ) mistral_tool_calls: list[MistralToolCall] = [ - MistralToolCall( - type="function", - function=FunctionCall( - name=tool_call["name"], - arguments=tool_call.get("arguments", "{}"), - ), - ) - for tool_call in tool_calls + MistralToolCall(type="function", function=function_call) + for function_call in function_calls ] return ExtractedToolCallInformation( @@ -668,9 +680,16 @@ class MistralParser(ParserEngine): @ijson.coroutine def update_stream_state_pre_v11_tokenizer(self): + bare_object = False while True: (prefix, event, value) = yield + # A bare top-level object is a single call: parse it as an item. + if prefix == "" and event == "start_map": + bare_object = True + if bare_object: + prefix = f"item.{prefix}" if prefix else "item" + if prefix == "item" and event == "start_map": self.streaming_state = StreamingState.WAITING_FOR_TOOL_KEY self.starting_new_tool = True @@ -685,8 +704,14 @@ class MistralParser(ParserEngine): self.streaming_state = StreamingState.PARSING_ARGUMENTS if prefix == "item.arguments" and event == "end_map": self.streaming_state = StreamingState.PARSING_ARGUMENTS_COMPLETED + if prefix == "item.arguments" and event == "string": + self.current_tool_string_arguments = value if prefix == "item" and event == "end_map": - self.streaming_state = StreamingState.TOOL_COMPLETE + self.streaming_state = ( + StreamingState.ALL_TOOLS_COMPLETE + if bare_object + else StreamingState.TOOL_COMPLETE + ) if prefix == "" and event == "end_array": self.streaming_state = StreamingState.ALL_TOOLS_COMPLETE @@ -722,22 +747,27 @@ class MistralParser(ParserEngine): delta_text=delta_text, stop_after_opening_curly_braces=1, ) + # Every state in which the call object can close stops after "}", + # so text following a bare object is never sent to ijson. elif self.streaming_state == StreamingState.WAITING_FOR_TOOL_KEY: delta_to_be_parsed, delta_text = self._split_delta( delta_text=delta_text, stop_after_colon=1, stop_after_opening_curly_braces=1, + stop_after_closing_curly_braces=1, ) elif self.streaming_state == StreamingState.PARSING_NAME: delta_to_be_parsed, delta_text = self._split_delta( delta_text=delta_text, stop_after_comma=1, stop_after_closing_brackets=1, + stop_after_closing_curly_braces=1, ) elif self.streaming_state == StreamingState.WAITING_FOR_ARGUMENTS_START: delta_to_be_parsed, delta_text = self._split_delta( delta_text=delta_text, stop_after_opening_curly_braces=1, + stop_after_closing_curly_braces=1, ) elif self.streaming_state == StreamingState.PARSING_ARGUMENTS: delta_to_be_parsed, delta_text = self._split_delta( @@ -798,6 +828,12 @@ class MistralParser(ParserEngine): self.current_tool_name ) self.current_tool_name = None + if self.current_tool_string_arguments is not None: + current_tool_call_modified = True + current_tool_call.function.arguments = ( + self.current_tool_string_arguments + ) + self.current_tool_string_arguments = None if self.streaming_state == StreamingState.PARSING_NAME_COMPLETED: self.streaming_state = StreamingState.WAITING_FOR_TOOL_KEY if self.streaming_state in [ diff --git a/vllm/parser/parser_manager.py b/vllm/parser/parser_manager.py index 719b396130..cf0be908ed 100644 --- a/vllm/parser/parser_manager.py +++ b/vllm/parser/parser_manager.py @@ -73,6 +73,29 @@ class ParserManager: raise TypeError(f"{reasoning_parser_name=} has not been registered") from e return parser + @classmethod + def validate_parser_tokenizer( + cls, + parser_cls: type[Parser], + tokenizer: TokenizerLike, + tool_parser_name: str | None = None, + reasoning_parser_name: str | None = None, + model_name: str | None = None, + ) -> None: + """Fail at startup if the tokenizer lacks what the parsers need.""" + try: + parser_cls(tokenizer) + except Exception as e: + flags = [] + if parser_cls.tool_parser_cls is not None: + flags.append(f"--tool-call-parser {tool_parser_name}") + if parser_cls.reasoning_parser_cls is not None: + flags.append(f"--reasoning-parser {reasoning_parser_name}") + raise TypeError( + f"{' and '.join(flags)} cannot be used with the tokenizer of " + f"{model_name!r}: {e}" + ) from e + @classmethod def get_parser( cls, @@ -97,13 +120,44 @@ class ParserManager: If True, HarmonyParser is always returned. tool_strict_level: Server-side floor for tool-call structural tags (``--tool-strict-level``). - tokenizer: Tokenizer whose `response_template` metadata is - validated when the `hf` parser is selected. + tokenizer: Tokenizer the composed parser is validated against + at startup (see :meth:`validate_parser_tokenizer`). ``None`` + skips the check (e.g. ``skip_tokenizer_init``). Returns: A Parser class, or None if neither parser is specified. """ + parser_cls = cls._compose_parser( + tool_parser_name=tool_parser_name, + reasoning_parser_name=reasoning_parser_name, + enable_auto_tools=enable_auto_tools, + model_name=model_name, + is_harmony=is_harmony, + tool_strict_level=tool_strict_level, + tokenizer=tokenizer, + ) + if parser_cls is not None and tokenizer is not None: + cls.validate_parser_tokenizer( + parser_cls, + tokenizer, + tool_parser_name=tool_parser_name, + reasoning_parser_name=reasoning_parser_name, + model_name=model_name, + ) + return parser_cls + + @classmethod + def _compose_parser( + cls, + tool_parser_name: str | None, + reasoning_parser_name: str | None, + enable_auto_tools: bool, + model_name: str | None, + is_harmony: bool, + tool_strict_level: str, + tokenizer: TokenizerLike | None, + ) -> type[Parser] | None: if not tool_parser_name and not reasoning_parser_name: return None diff --git a/vllm/platforms/cpu.py b/vllm/platforms/cpu.py index 25d28daf93..43d472f22e 100644 --- a/vllm/platforms/cpu.py +++ b/vllm/platforms/cpu.py @@ -18,6 +18,7 @@ from vllm.utils.cpu_resource_utils import ( get_visible_memory_node, ) from vllm.utils.mem_constants import GiB_bytes +from vllm.v1.attention.backends.mla.prefill.registry import MLAPrefillBackendEnum from vllm.v1.attention.backends.registry import AttentionBackendEnum from .interface import CpuArchEnum, Platform, PlatformEnum @@ -26,6 +27,10 @@ logger = init_logger(__name__) if TYPE_CHECKING: from vllm.config import VllmConfig + from vllm.v1.attention.backends.mla.prefill.base import ( + MLAPrefillBackend, + ) + from vllm.v1.attention.backends.mla.prefill.selector import MLAPrefillSelectorConfig from vllm.v1.attention.selector import AttentionSelectorConfig else: VllmConfig = None @@ -173,6 +178,21 @@ class CpuPlatform(Platform): logger.info("Cannot use %s backend on CPU.", selected_backend) return AttentionBackendEnum.CPU_ATTN.get_path() + @classmethod + def get_mla_prefill_backend_cls( + cls, + mla_selector_config: "MLAPrefillSelectorConfig", + ) -> "type[MLAPrefillBackend]": + """Get the MLA prefill backend class of a device.""" + for backend_enum in (MLAPrefillBackendEnum.ZEN_CPU, MLAPrefillBackendEnum.CPU): + try: + cpu_backend_cls = backend_enum.get_class() + except ImportError: + continue + if cpu_backend_cls.is_available(): + return cpu_backend_cls + raise ValueError("No valid CPU MLA prefill backend found.") + @classmethod def get_device_total_memory(cls, device_id: int = 0) -> int: meminfo = get_memory_node_info(device_id) @@ -303,7 +323,7 @@ class CpuPlatform(Platform): mamba_backend, ) - # Lagecy setting + # Legacy setting env_key = "VLLM_CPU_KVCACHE_SPACE" if env_key in os.environ and os.environ[env_key] != "": kv_cache_space = int(os.environ[env_key]) diff --git a/vllm/platforms/interface.py b/vllm/platforms/interface.py index dc7178cf84..dc3200759c 100644 --- a/vllm/platforms/interface.py +++ b/vllm/platforms/interface.py @@ -26,6 +26,8 @@ if TYPE_CHECKING: from vllm.sampling_params import SamplingParams from vllm.utils.argparse_utils import FlexibleArgumentParser from vllm.v1.attention.backend import AttentionBackend + from vllm.v1.attention.backends.mla.prefill.base import MLAPrefillBackend + from vllm.v1.attention.backends.mla.prefill.selector import MLAPrefillSelectorConfig from vllm.v1.attention.selector import AttentionSelectorConfig else: FlexibleArgumentParser = object @@ -429,6 +431,88 @@ class Platform: ) return AttentionBackendEnum.TORCH_SDPA + @classmethod + def get_mla_prefill_backend_cls( + cls, + mla_selector_config: "MLAPrefillSelectorConfig", + ) -> "type[MLAPrefillBackend]": + """Get the MLA prefill backend class of a device. + + Platforms that have a single, fixed MLA prefill backend can override + this to return it directly. The default implementation selects the + best backend from a device-capability-based priority list. + + Returns: + The selected backend class. + + Raises: + ValueError: If no MLA prefill backend is valid for the + configuration. + + """ + from vllm.v1.attention.backends.mla.prefill.base import MLADimensions + from vllm.v1.attention.backends.mla.prefill.registry import ( + MLAPrefillBackendEnum, + ) + + device_capability = cls.get_device_capability() + if device_capability is None: + return MLAPrefillBackendEnum.FLASH_ATTN.get_class() + + if device_capability.major == 10: # Blackwell + if mla_selector_config.mla_dimensions == MLADimensions( + qk_nope_head_dim=192, + qk_rope_head_dim=64, + v_head_dim=256, + ): + priorities = [ + MLAPrefillBackendEnum.TRTLLM_RAGGED, + MLAPrefillBackendEnum.FLASH_ATTN, + MLAPrefillBackendEnum.FLASHINFER, + MLAPrefillBackendEnum.TOKENSPEED_MLA, + ] + else: + priorities = [ + MLAPrefillBackendEnum.FLASH_ATTN, + MLAPrefillBackendEnum.TRTLLM_RAGGED, + MLAPrefillBackendEnum.FLASHINFER, + MLAPrefillBackendEnum.TOKENSPEED_MLA, + ] + else: # Hopper (SM90) and older + priorities = [MLAPrefillBackendEnum.FLASH_ATTN] + + all_invalid_reasons: dict[str, list[str]] = {} + + for backend_enum in priorities: + try: + backend_cls = backend_enum.get_class() + invalid_reasons = backend_cls.validate_configuration( + device_capability, mla_selector_config + ) + except ImportError: + invalid_reasons = ["ImportError"] + if not invalid_reasons: + return backend_cls + all_invalid_reasons[backend_enum.name] = invalid_reasons + + reasons_str = ( + "{" + + ", ".join( + f"{name}: [{', '.join(reasons)}]" + for name, reasons in all_invalid_reasons.items() + ) + + "}" + ) + logger.debug_once( + "Some MLA prefill backends are not valid with %s. Reasons: %s.", + repr(mla_selector_config), + reasons_str, + ) + raise ValueError( + f"No valid MLA prefill backend found with {mla_selector_config!r}. " + f"Reasons: {reasons_str}." + ) + @classmethod def get_device_capability( cls, @@ -906,15 +990,29 @@ class Platform: cache_dtype_str=cache_config.cache_dtype, kv_quant_mode=kv_quant_mode, ).page_size_bytes - elif cache_config.cache_dtype.startswith("turboquant_"): - # TQ has a packed K|V layout; the standard FullAttentionSpec - # formula over-sizes it and trips unify_kv_cache_spec_page_size - # when all attention layers are TQ. With mixed skip+TQ the skip - # layers still use the standard layout — take max so mamba - # padding covers the largest actual page. - from vllm.v1.attention.backends.turboquant_attn import ( - TurboQuantAttentionBackend, - ) + elif ( + cache_config.cache_dtype.startswith("turboquant_") + or cache_config.cache_dtype == "ultraquant_4bit" + ): + # TurboQuant and UltraQuant have packed K|V layouts; the standard + # FullAttentionSpec formula over-sizes them and trips + # unify_kv_cache_spec_page_size when all attention layers are + # packed. With mixed skip+packed the skip layers still use the + # standard layout — take max so mamba padding covers the largest + # actual page. Each dtype packs via its own backend. + _pack_backend: type[AttentionBackend] + if cache_config.cache_dtype == "ultraquant_4bit": + from vllm.v1.attention.backends.ultraquant_attn import ( + UltraQuantAttentionBackend, + ) + + _pack_backend = UltraQuantAttentionBackend + else: + from vllm.v1.attention.backends.turboquant_attn import ( + TurboQuantAttentionBackend, + ) + + _pack_backend = TurboQuantAttentionBackend tq_spec = FullAttentionSpec( block_size=1, @@ -923,7 +1021,7 @@ class Platform: dtype=kv_cache_dtype, kv_quant_mode=kv_quant_mode, ) - tq_page = TurboQuantAttentionBackend.customize_spec(tq_spec).page_size_bytes + tq_page = _pack_backend.customize_spec(tq_spec).page_size_bytes if cache_config.kv_cache_dtype_skip_layers: skip_page = FullAttentionSpec( block_size=1, @@ -1111,11 +1209,6 @@ class Platform: """Checks if the platform allows inplace memory updates.""" return True - @classmethod - def get_lora_vocab_padding_size(cls) -> int: - """Returns how much padding the LoRA logits need for kernels.""" - return 256 - @classmethod def get_device_communicator_cls(cls) -> str: """Get device specific communicator class for distributed communication.""" diff --git a/vllm/platforms/rocm.py b/vllm/platforms/rocm.py index 487cffa670..5840ffe3b7 100644 --- a/vllm/platforms/rocm.py +++ b/vllm/platforms/rocm.py @@ -24,12 +24,17 @@ if TYPE_CHECKING: from vllm.config.kernel import IrOpPriorityConfig from vllm.utils.argparse_utils import FlexibleArgumentParser from vllm.v1.attention.backend import AttentionBackend + from vllm.v1.attention.backends.mla.prefill.base import MLAPrefillBackend + from vllm.v1.attention.backends.mla.prefill.selector import ( + MLAPrefillSelectorConfig, + ) from vllm.v1.attention.selector import AttentionSelectorConfig logger = init_logger(__name__) _KV_CACHE_DTYPE_REASON = "kv_cache_dtype not supported" _TURBOQUANT_LAYOUT_REASON = "no KV cache layout in common with TURBOQUANT" +_ULTRAQUANT_LAYOUT_REASON = "no KV cache layout in common with ULTRAQUANT" try: from amdsmi import ( @@ -505,6 +510,7 @@ def _get_backend_priorities( backends.insert(0, AttentionBackendEnum.ROCM_AITER_UNIFIED_ATTN) backends.append(AttentionBackendEnum.TRITON_ATTN) backends.append(AttentionBackendEnum.TURBOQUANT) + backends.append(AttentionBackendEnum.ULTRAQUANT) return backends @@ -517,6 +523,14 @@ def _uses_turboquant(vllm_config: "VllmConfig | None") -> bool: ) +def _uses_ultraquant(vllm_config: "VllmConfig | None") -> bool: + """Whether the run's KV cache dtype is the UltraQuant 4-bit format.""" + cache_config = getattr(vllm_config, "cache_config", None) + return ( + cache_config is not None and str(cache_config.cache_dtype) == "ultraquant_4bit" + ) + + def _shares_layout_with_turboquant(backend_class: type["AttentionBackend"]) -> bool: """Whether this backend reads a KV cache layout TURBOQUANT reads too. @@ -535,12 +549,29 @@ def _shares_layout_with_turboquant(backend_class: type["AttentionBackend"]) -> b return not set(layouts).isdisjoint(turboquant_layouts) +def _shares_layout_with_ultraquant(backend_class: type["AttentionBackend"]) -> bool: + """Whether this backend reads a KV cache layout ULTRAQUANT reads too. + + Same constraint as _shares_layout_with_turboquant: an ultraquant_4bit run + keeps its boundary layers at the native dtype, and one layout has to serve + the whole worker. + """ + layouts = backend_class.supported_kv_cache_layouts() + ultraquant_layouts = ( + AttentionBackendEnum.ULTRAQUANT.get_class().supported_kv_cache_layouts() + ) + if layouts is None or ultraquant_layouts is None: + return True + return not set(layouts).isdisjoint(ultraquant_layouts) + + def _get_invalid_reasons( backend_class: type["AttentionBackend"], device_capability: DeviceCapability, attn_selector_config: "AttentionSelectorConfig", *, is_turboquant_run: bool, + is_ultraquant_run: bool = False, ) -> list[str]: """Why this backend cannot serve the layer, empty when it can.""" invalid_reasons = backend_class.validate_configuration( @@ -553,6 +584,12 @@ def _get_invalid_reasons( and not _shares_layout_with_turboquant(backend_class) ): invalid_reasons = [_TURBOQUANT_LAYOUT_REASON] + if ( + not invalid_reasons + and is_ultraquant_run + and not _shares_layout_with_ultraquant(backend_class) + ): + invalid_reasons = [_ULTRAQUANT_LAYOUT_REASON] return invalid_reasons @@ -683,7 +720,7 @@ class RocmPlatform(Platform): @classmethod def get_attn_backend_cls( cls, - selected_backend: "AttentionBackendEnum", + selected_backend: "AttentionBackendEnum | None", attn_selector_config: "AttentionSelectorConfig", num_heads: int | None = None, ) -> str: @@ -695,13 +732,16 @@ class RocmPlatform(Platform): # Keep lazy: vllm.config imports current_platform during initialization. from vllm.config import get_current_vllm_config_or_none - is_turboquant_run = _uses_turboquant(get_current_vllm_config_or_none()) + vllm_config = get_current_vllm_config_or_none() + is_turboquant_run = _uses_turboquant(vllm_config) + is_ultraquant_run = _uses_ultraquant(vllm_config) try: sel_invalid_reasons = _get_invalid_reasons( selected_backend.get_class(), device_capability, attn_selector_config, is_turboquant_run=is_turboquant_run, + is_ultraquant_run=is_ultraquant_run, ) except ImportError: sel_invalid_reasons = ["ImportError"] @@ -711,21 +751,23 @@ class RocmPlatform(Platform): selected_backend.name, ) return selected_backend.get_path() - # Only tolerate the mismatch when turboquant is in play: boundary - # layers keep the native dtype while every other layer needs - # TURBOQUANT, so no single --attention-backend can serve every layer. - # For any other dtype the selection is genuinely invalid -> fail loud. + # Only tolerate the mismatch when turboquant or ultraquant is in play: + # boundary layers keep the native dtype while every other layer needs + # the packed backend, so no single --attention-backend can serve every + # layer. For any other dtype the selection is genuinely invalid -> fail + # loud. kv_dtype = attn_selector_config.kv_cache_dtype layer_is_turboquant = kv_dtype is not None and str(kv_dtype).startswith( "turboquant" ) - is_turboquant_fallback = ( - is_turboquant_run or layer_is_turboquant + is_packed_kv_fallback = ( + is_turboquant_run or layer_is_turboquant or is_ultraquant_run ) and sel_invalid_reasons in ( [_KV_CACHE_DTYPE_REASON], [_TURBOQUANT_LAYOUT_REASON], + [_ULTRAQUANT_LAYOUT_REASON], ) - if not is_turboquant_fallback: + if not is_packed_kv_fallback: raise ValueError( f"Selected backend {selected_backend} is not valid for " f"this configuration. Reason: {sel_invalid_reasons}" @@ -733,10 +775,11 @@ class RocmPlatform(Platform): # NOTE: pass a str (not the list) -- info_once hashes its args. logger.info_once( "Selected backend %s is incompatible with this layer (%s) of " - "the turboquant run; using the auto-selected per-layer backend. " + "the %s run; using the auto-selected per-layer backend. " "Reason: %s", selected_backend.name, attn_selector_config.attn_type, + "ultraquant" if is_ultraquant_run else "turboquant", str(sel_invalid_reasons), ) @@ -865,6 +908,57 @@ class RocmPlatform(Platform): logger.info_once("Using Torch SDPA backend for ViT model.") return AttentionBackendEnum.TORCH_SDPA + @classmethod + def get_mla_prefill_backend_cls( + cls, + mla_selector_config: "MLAPrefillSelectorConfig", + ) -> "type[MLAPrefillBackend]": + """On ROCm, prefer the AITER FlashAttention backend, falling back to + FlashAttention. + + Raises: + ValueError: If neither backend is valid for the configuration. + + """ + from vllm.v1.attention.backends.mla.prefill.registry import ( + MLAPrefillBackendEnum, + ) + + device_capability = cls.get_device_capability() + all_invalid_reasons: dict[str, list[str]] = {} + for backend_enum in ( + MLAPrefillBackendEnum.ROCM_AITER_FA, + MLAPrefillBackendEnum.FLASH_ATTN, + ): + try: + backend_cls = backend_enum.get_class() + invalid_reasons = backend_cls.validate_configuration( + device_capability, mla_selector_config + ) + except ImportError: + invalid_reasons = ["ImportError"] + if not invalid_reasons: + return backend_cls + all_invalid_reasons[backend_enum.name] = invalid_reasons + + reasons_str = ( + "{" + + ", ".join( + f"{name}: [{', '.join(reasons)}]" + for name, reasons in all_invalid_reasons.items() + ) + + "}" + ) + logger.debug_once( + "Some MLA prefill backends are not valid with %s. Reasons: %s.", + repr(mla_selector_config), + reasons_str, + ) + raise ValueError( + f"No valid MLA prefill backend found with {mla_selector_config!r}. " + f"Reasons: {reasons_str}." + ) + @classmethod def set_device(cls, device: torch.device) -> None: """Set the device for the current platform.""" diff --git a/vllm/platforms/xpu.py b/vllm/platforms/xpu.py index 6b7db6d0dc..992bfae053 100644 --- a/vllm/platforms/xpu.py +++ b/vllm/platforms/xpu.py @@ -123,6 +123,7 @@ class XPUPlatform(Platform): "mxfp8", "fp8_per_tensor", "fp8_per_block", + "fp8_per_channel", "online", "gpt_oss_mxfp4", "modelopt", @@ -352,7 +353,6 @@ class XPUPlatform(Platform): pass_config = compilation_config.pass_config fusion_passes_to_disable = { - "fuse_gemm_comms": "Async TP", "fuse_allreduce_rms": "AllReduce + RMSNorm fusion", "fuse_attn_quant": "Attention + quant fusion", "fuse_act_padding": "Activation + padding fusion", diff --git a/vllm/profiler/wrapper.py b/vllm/profiler/wrapper.py index bd9b6c231d..ebe7d90c3b 100644 --- a/vllm/profiler/wrapper.py +++ b/vllm/profiler/wrapper.py @@ -5,6 +5,7 @@ import importlib import inspect import json import os +import threading from abc import ABC, abstractmethod from collections.abc import Callable, Iterator, Sequence from contextlib import AbstractContextManager, contextmanager, nullcontext, suppress @@ -17,7 +18,13 @@ from typing_extensions import override import vllm.version from vllm.config import ProfilerConfig -from vllm.config.profiler import ProfilerKind, TorchProfilerActivity, _is_uri_path +from vllm.config.profiler import ( + ProfilerKind, + TorchProfilerActivity, + _is_uri_path, + validate_profile_iteration_bounds, +) +from vllm.config.utils import replace from vllm.logger import init_logger from vllm.platforms import current_platform @@ -28,14 +35,16 @@ _TRITON_PROTON_3_7_VERSION = Version("3.7.0") class WorkerProfiler(ABC): def __init__(self, profiler_config: ProfilerConfig) -> None: - self._delay_iters = profiler_config.delay_iterations + self._default_delay_iters = profiler_config.delay_iterations + self._delay_iters = self._default_delay_iters if self._delay_iters > 0: logger.info_once( "GPU profiling will start " f"{self._delay_iters} steps after start_profile." ) - self._max_iters = profiler_config.max_iterations + self._default_max_iters = profiler_config.max_iterations + self._max_iters = self._default_max_iters if self._max_iters > 0: logger.info_once( "GPU profiling will stop " @@ -43,6 +52,9 @@ class WorkerProfiler(ABC): "or when stop_profile is received." ) + # Serializes start()/stop(), which AsyncLLM may call from different threads. + self._lifecycle_lock = threading.Lock() + # Track when the profiler gets triggered by start_profile self._active_iteration_count = 0 self._active = False @@ -88,17 +100,32 @@ class WorkerProfiler(ABC): logger.warning("Failed to stop profiler: %s", e) self._running = False # Always mark as not running, assume stop worked - def start(self) -> None: + def start( + self, + *, + delay_iterations: int | None = None, + max_iterations: int | None = None, + ) -> None: """Attempt to start the profiler, accounting for delayed starts.""" - if self._active: - logger.debug( - "start_profile received when profiler is already active. " - "Ignoring request." + with self._lifecycle_lock: + if self._active: + logger.debug( + "start_profile received when profiler is already active. " + "Ignoring request." + ) + return + validate_profile_iteration_bounds(delay_iterations, max_iterations) + self._delay_iters = ( + self._default_delay_iters + if delay_iterations is None + else delay_iterations ) - return - self._active = True - if self._delay_iters == 0: - self._call_start() + self._max_iters = ( + self._default_max_iters if max_iterations is None else max_iterations + ) + self._active = True + if self._delay_iters == 0: + self._call_start() def step(self) -> None: """Update the profiler state at each worker step, @@ -148,17 +175,19 @@ class WorkerProfiler(ABC): def stop(self) -> None: """Attempt to stop the profiler, accounting for overlapped calls.""" - if not self._active: - logger.debug( - "stop_profile received when profiler is not active. Ignoring request." - ) - return - self._active = False - self._active_iteration_count = 0 - self._profiling_for_iters = 0 + with self._lifecycle_lock: + if not self._active: + logger.debug( + "stop_profile received when profiler is not active. " + "Ignoring request." + ) + return + self._active = False + self._active_iteration_count = 0 + self._profiling_for_iters = 0 - if self._running: - self._call_stop() + if self._running: + self._call_stop() def shutdown(self) -> None: """Ensure profiler is stopped when shutting down.""" @@ -175,6 +204,10 @@ class WorkerProfiler(ABC): """Observe graph creation for backends that attribute replay activity.""" return nullcontext() + def set_output_name(self, worker_name: str) -> None: + """Set the output name for the next profiling session.""" + return None + def annotate_context_manager(self, name: str): """Return a context manager to annotate profiler traces.""" return nullcontext() @@ -202,6 +235,10 @@ class TorchProfilerWrapper(WorkerProfiler): self.local_rank = local_rank self.profiler_config = profiler_config torch_profiler_trace_dir = profiler_config.torch_profiler_dir + self._torch_profiler_trace_dir = torch_profiler_trace_dir + self._torch_profiler_use_gzip = profiler_config.torch_profiler_use_gzip + self._worker_name = worker_name + self._custom_trace_handler = on_trace_ready is not None if local_rank in (None, 0): logger.info_once( "Torch profiling enabled. Traces will be saved to: %s", @@ -276,6 +313,21 @@ class TorchProfilerWrapper(WorkerProfiler): self._warmup_steps_remaining = self._initial_warmup_steps_remaining self._version_metadata_added = False + @override + def set_output_name(self, worker_name: str) -> None: + if self._active: + return + self._worker_name = worker_name + if self._custom_trace_handler: + return + self._profiler_kwargs["on_trace_ready"] = ( + torch.profiler.tensorboard_trace_handler( + self._torch_profiler_trace_dir, + worker_name=worker_name, + use_gzip=self._torch_profiler_use_gzip, + ) + ) + def _build_profiler_table( self, sort_key: str, @@ -293,13 +345,15 @@ class TorchProfilerWrapper(WorkerProfiler): row_limit=row_limit, ) - def _write_profiler_table(self, rank: int, table: str) -> None: + def _write_profiler_table(self, table: str) -> None: profiler_dir = self.profiler_config.torch_profiler_dir # Skip file write for URI paths (gs://, s3://, etc.) # as standard file I/O doesn't work with URI schemes if not _is_uri_path(profiler_dir): - profiler_out_file = f"{profiler_dir}/profiler_out_{rank}.txt" + profiler_out_file = os.path.join( + profiler_dir, f"{self._worker_name}.profiler_out.txt" + ) with open(profiler_out_file, "w") as f: print(table, file=f) @@ -344,7 +398,7 @@ class TorchProfilerWrapper(WorkerProfiler): rank = self.local_rank if self.dump_device_time_total: table = self._build_profiler_table(sort_key="self_device_time_total") - self._write_profiler_table(rank, table) + self._write_profiler_table(table) # only print profiler results on rank 0 if rank == 0: @@ -354,7 +408,7 @@ class TorchProfilerWrapper(WorkerProfiler): table = self._build_profiler_table( sort_key="self_cpu_time_total", row_limit=50 ) - self._write_profiler_table(rank, table) + self._write_profiler_table(table) # only print profiler results on rank 0 if rank == 0: @@ -494,6 +548,7 @@ class ProtonProfilerWrapper(WorkerProfiler): def has_cuda_graph_session(self) -> bool: return self._graph_session + @override def set_output_name(self, worker_name: str) -> None: """Set the next run's output name after startup graph capture.""" if self._active: @@ -701,6 +756,29 @@ def create_worker_profiler( return ProtonProfilerWrapper(profiler_config, worker_name=worker_name) +def create_frontend_profiler( + profiler_config: ProfilerConfig, *, worker_name: str +) -> TorchProfilerWrapper: + """Create the CPU-only frontend profiler, which runs for the whole session + rather than following engine iterations.""" + logger.info( + "Torch profiler enabled. AsyncLLM CPU traces will be collected under %s", + profiler_config.torch_profiler_dir, + ) + return TorchProfilerWrapper( + replace( + profiler_config, + delay_iterations=0, + max_iterations=0, + wait_iterations=0, + warmup_iterations=0, + ), + worker_name=worker_name, + local_rank=0, + activities=["CPU"], + ) + + def create_graph_capture_profiler( profiler_config: ProfilerConfig, global_rank: int ) -> WorkerProfiler | None: diff --git a/vllm/renderers/hf.py b/vllm/renderers/hf.py index 4301c69176..4a8bcd0c03 100644 --- a/vllm/renderers/hf.py +++ b/vllm/renderers/hf.py @@ -1000,7 +1000,7 @@ class HfRenderer(BaseRenderer[HfTokenizer]): return await coro try: return await asyncio.wait_for(coro, timeout=timeout) - except asyncio.TimeoutError: + except TimeoutError: self._replace_executor() raise TimeoutError( f"Chat template rendering timed out after " diff --git a/vllm/renderers/online_derenderer.py b/vllm/renderers/online_derenderer.py index d221b32a8b..66b73fb491 100644 --- a/vllm/renderers/online_derenderer.py +++ b/vllm/renderers/online_derenderer.py @@ -9,9 +9,11 @@ from vllm.entrypoints.generate.base.protocol import ( DeltaMessage, ToolCall, ) -from vllm.entrypoints.generate.base.serving import resolve_token_id_placeholder +from vllm.entrypoints.generate.base.serving import decode_token_ids from vllm.entrypoints.openai.chat_completion.protocol import ( + ChatCompletionLogProb, ChatCompletionLogProbs, + ChatCompletionLogProbsContent, ChatCompletionNamedToolChoiceParam, ChatCompletionRequest, ChatCompletionResponseChoice, @@ -28,6 +30,7 @@ from vllm.entrypoints.openai.completion.protocol import ( ) from vllm.entrypoints.scale_out.token_in_token_out.protocol import ( DerenderStreamState, + GenerateLogProbs, GenerateTokensResponse, GenerateTokensStreamResponse, ) @@ -151,6 +154,12 @@ class OnlineDerenderer: skip_special_tokens=skip_special, ) + chat_template_kwargs = ( + self._resolve_chat_template_kwargs(chat_request) + if has_parser and chat_request is not None + else {} + ) + for choice in generate_response.choices: if not choice.token_ids: raise ValueError(f"choice {choice.index} has empty or null token_ids") @@ -161,83 +170,29 @@ class OnlineDerenderer: else None ) + # With a parser, special tokens are preserved so it can see + # markers like , , or Harmony channel tokens. + # Without one, the request's skip_special_tokens is honoured + # (default True when no request was given). + decoded_text, _ = self._detokenize_delta( + tokenizer, + choice.token_ids, + seed_state, + skip_special_tokens=skip_special, + spaces_between_special_tokens=spaces_between, + ) + if has_parser: - assert self.parser is not None and chat_request is not None - # Parser path: decode with special tokens preserved - # so the parser can see markers like , - # , or Harmony channel tokens. - decoded_text, _ = self._detokenize_delta( + assert chat_request is not None + message = self._parse_and_assemble( tokenizer, - choice.token_ids, - seed_state, - skip_special_tokens=skip_special, - spaces_between_special_tokens=spaces_between, - ) - - chat_template_kwargs: dict[str, Any] = {} - if not self.use_harmony: - chat_template_kwargs = ( - chat_request.build_chat_params( - self.chat_template, - self.chat_template_content_format, - ) - .with_defaults(self.default_chat_template_kwargs) - .chat_template_kwargs - ) - - parser = self.parser( - tokenizer, - chat_request.tools, - chat_template_kwargs=chat_template_kwargs, - model_config=self.model_config, - ) - if generate_response.prompt_token_ids is not None: - parser.set_prompt_token_ids(generate_response.prompt_token_ids) - reasoning, content, tool_calls = parser.parse( decoded_text, + choice.token_ids, chat_request, - enable_auto_tools=self.enable_auto_tools, - model_output_token_ids=choice.token_ids, - ) - - if not getattr(chat_request, "include_reasoning", True): - reasoning = None - - tc_items = ( - [ - ToolCall( - id=random_uuid(), - function=tc, - ) - for tc in tool_calls - ] - if tool_calls - else [] - ) - - is_named_tool_choice = ( - type(chat_request.tool_choice) is ChatCompletionNamedToolChoiceParam - ) - is_required_tool_choice = chat_request.tool_choice == "required" - if is_named_tool_choice or is_required_tool_choice: - content = content or "" - - message = ChatMessage( - role="assistant", - reasoning=reasoning, - content=content, - tool_calls=tc_items, + chat_template_kwargs, + generate_response.prompt_token_ids, ) else: - # No parser: plain detokenization honouring the request's - # skip_special_tokens (default True when no request was given). - decoded_text, _ = self._detokenize_delta( - tokenizer, - choice.token_ids, - seed_state, - skip_special_tokens=skip_special, - spaces_between_special_tokens=spaces_between, - ) message = ChatMessage(role="assistant", content=decoded_text) choices.append( @@ -251,6 +206,83 @@ class OnlineDerenderer: return choices + def _resolve_chat_template_kwargs( + self, chat_request: ChatCompletionRequest + ) -> dict[str, Any]: + if self.use_harmony: + return {} + return ( + chat_request.build_chat_params( + self.chat_template, + self.chat_template_content_format, + ) + .with_defaults(self.default_chat_template_kwargs) + .chat_template_kwargs + ) + + def _parse_and_assemble( + self, + tokenizer: TokenizerLike, + text: str, + token_ids: Sequence[int], + chat_request: ChatCompletionRequest, + chat_template_kwargs: dict[str, Any], + prompt_token_ids: list[int] | None, + ) -> ChatMessage: + """Parse decoded output text into an assistant `ChatMessage`. + + Args: + tokenizer: Tokenizer handed to the parser. + text: Decoded output text to parse. + token_ids: Output token IDs the text was decoded from. + chat_request: Request supplying tools, `tool_choice` and + `include_reasoning`. + chat_template_kwargs: Already resolved template kwargs. + prompt_token_ids: Prompt token IDs, if known. + + Returns: + The assembled assistant message. + + """ + assert self.parser is not None + parser = self.parser( + tokenizer, + chat_request.tools, + chat_template_kwargs=chat_template_kwargs, + model_config=self.model_config, + ) + if prompt_token_ids is not None: + parser.set_prompt_token_ids(prompt_token_ids) + reasoning, content, tool_calls = parser.parse( + text, + chat_request, + enable_auto_tools=self.enable_auto_tools, + model_output_token_ids=token_ids, + ) + + if not getattr(chat_request, "include_reasoning", True): + reasoning = None + + tc_items = ( + [ToolCall(id=random_uuid(), function=tc) for tc in tool_calls] + if tool_calls + else [] + ) + + is_named_tool_choice = ( + type(chat_request.tool_choice) is ChatCompletionNamedToolChoiceParam + ) + is_required_tool_choice = chat_request.tool_choice == "required" + if is_named_tool_choice or is_required_tool_choice: + content = content or "" + + return ChatMessage( + role="assistant", + reasoning=reasoning, + content=content, + tool_calls=tc_items, + ) + def _detokenize_delta( self, tokenizer: TokenizerLike, @@ -497,21 +529,10 @@ class OnlineDerenderer: """ tokenizer = self.renderer.get_tokenizer() - chat_template_kwargs: dict[str, Any] = {} - if not self.use_harmony: - chat_template_kwargs = ( - chat_request.build_chat_params( - self.chat_template, - self.chat_template_content_format, - ) - .with_defaults(self.default_chat_template_kwargs) - .chat_template_kwargs - ) - parser = parser_cls( tokenizer, chat_request.tools, - chat_template_kwargs=chat_template_kwargs, + chat_template_kwargs=self._resolve_chat_template_kwargs(chat_request), model_config=self.model_config, ) @@ -619,12 +640,6 @@ class OnlineDerenderer: if tc.index < len(last_tool_call_ids): # Pin: reuse the ID already recorded for this index # rather than one a from scratch replay regenerated. - # Real trigger not just defensive with - # tool_choice="required", - # extract_required_tool_call_streaming resets - # function_name_returned to False whenever the - # partial JSON transiently fails to parse which - # re-emits id+name for the same index on replay. tc.id = last_tool_call_ids[tc.index] else: last_tool_call_ids.append(tc.id) @@ -945,16 +960,6 @@ def _seed_stream_state( ) -def _parse_token_id_placeholder(token: str) -> int | None: - """Extract token ID from a 'token_id:N' placeholder string.""" - if not token.startswith("token_id:"): - return None - try: - return int(token[len("token_id:") :]) - except ValueError: - return None - - def _correct_decoded_token( token_id: int, context_token_ids: list[int], tokenizer: TokenizerLike ) -> str: @@ -994,53 +999,63 @@ def _correct_decoded_token( def _resolve_logprobs( - logprobs: ChatCompletionLogProbs, + logprobs: GenerateLogProbs, tokenizer: TokenizerLike, initial_context_token_ids: Sequence[int] = (), ) -> ChatCompletionLogProbs: - """Resolve token_id:N placeholders in a ChatCompletionLogProbs object. + """Convert generate's integer-id logprobs to the OpenAI chat shape. - ``initial_context_token_ids`` seeds the byte-fallback correction context - with sampled IDs from preceding chunks (streaming), so multi-byte - characters split across chunk boundaries still resolve. + The generate server has no tokenizer, so `token` and `bytes` are filled + here, with the same U+FFFD byte-fallback correction the coupled path + applies (`_correct_decoded_token`, which needs the preceding sampled token + ids as context). ``initial_context_token_ids`` seeds that context with + sampled IDs from preceding chunks (streaming), so multi-byte characters + split across chunk boundaries still resolve. """ if logprobs.content is None: - return logprobs + return ChatCompletionLogProbs() context_token_ids: list[int] = list(initial_context_token_ids) resolved_content = [] for entry in logprobs.content: - token_str, token_bytes = resolve_token_id_placeholder(entry.token, tokenizer) - sampled_id = _parse_token_id_placeholder(entry.token) + # One batch per position: the sampled id and its top-k ids. + (token_str, token_bytes), *top_decoded = decode_token_ids( + [entry.token_id, *(top.token_id for top in entry.top_logprobs)], + tokenizer, + ) - if token_str.endswith("�") and sampled_id is not None: - token_str = _correct_decoded_token(sampled_id, context_token_ids, tokenizer) + if token_str.endswith("\ufffd"): + token_str = _correct_decoded_token( + entry.token_id, context_token_ids, tokenizer + ) token_bytes = list(token_str.encode("utf-8")) resolved_top = [] - for top in entry.top_logprobs: - top_str, top_bytes = resolve_token_id_placeholder(top.token, tokenizer) - top_id = _parse_token_id_placeholder(top.token) - if top_str.endswith("�") and top_id is not None: - top_str = _correct_decoded_token(top_id, context_token_ids, tokenizer) + for top, (top_str, top_bytes) in zip(entry.top_logprobs, top_decoded): + if top_str.endswith("\ufffd"): + top_str = _correct_decoded_token( + top.token_id, context_token_ids, tokenizer + ) top_bytes = list(top_str.encode("utf-8")) resolved_top.append( - top.model_copy(update={"token": top_str, "bytes": top_bytes}) + ChatCompletionLogProb( + token=top_str, + logprob=top.logprob, + bytes=top_bytes, + ) ) resolved_content.append( - entry.model_copy( - update={ - "token": token_str, - "bytes": token_bytes, - "top_logprobs": resolved_top, - } + ChatCompletionLogProbsContent( + token=token_str, + logprob=entry.logprob, + bytes=token_bytes, + top_logprobs=resolved_top, ) ) - if sampled_id is not None: - context_token_ids.append(sampled_id) + context_token_ids.append(entry.token_id) return ChatCompletionLogProbs(content=resolved_content) diff --git a/vllm/renderers/online_renderer.py b/vllm/renderers/online_renderer.py index 23352ef1cf..e8220938b2 100644 --- a/vllm/renderers/online_renderer.py +++ b/vllm/renderers/online_renderer.py @@ -209,6 +209,18 @@ class OnlineRenderer: ) ) + def effective_chat_template_kwargs( + self, request: ChatCompletionRequest | ResponsesRequest + ) -> dict[str, Any]: + return ( + request.build_chat_params( + self.chat_template, + self.chat_template_content_format, + ) + .with_defaults(self.default_chat_template_kwargs) + .chat_template_kwargs + ) + async def render_chat( self, request: ChatCompletionRequest, @@ -356,14 +368,7 @@ class OnlineRenderer: else None ), ) - chat_template_kwargs = ( - request.build_chat_params( - self.chat_template, - self.chat_template_content_format, - ) - .with_defaults(self.default_chat_template_kwargs) - .chat_template_kwargs - ) + chat_template_kwargs = self.effective_chat_template_kwargs(request) _, engine_inputs = await self.preprocess_chat( request, messages, diff --git a/vllm/sampling_params.py b/vllm/sampling_params.py index 246e2ce90c..8a37c5a3fd 100644 --- a/vllm/sampling_params.py +++ b/vllm/sampling_params.py @@ -33,6 +33,15 @@ MAX_LOGPROB_TOKEN_IDS = 128 """Upper bound on `SamplingParams.logprob_token_ids` list length. Must match the per-request row width allocated by the sampler's `LogprobTokenIdsState`.""" +MAX_STRUCTURED_OUTPUT_JSON_NESTING = 64 +"""Upper bound on nested objects and arrays in a structured-output JSON schema +or structural tag. Grammar conversion runs on the API server and its cost grows +steeply with nesting. OpenAI allows 10 levels of schema nesting.""" + +STRUCTURAL_TAG_WRAPPER_NESTING = 16 +"""Extra nesting allowed for a structural tag, which wraps its JSON schemas in +tag structure (up to 15 levels for tool-call tags generated by tool parsers).""" + def _verify_num_sequences(value: int, parameter_name: str) -> None: if not isinstance(value, int): @@ -202,6 +211,52 @@ class RequestOutputKind(Enum): FINAL_ONLY = 2 +def _exceeds_json_nesting(spec: str | dict[str, Any], limit: int) -> bool: + """Whether a JSON document nests objects and arrays deeper than `limit`.""" + if isinstance(spec, str): + try: + spec = json_mod.loads(spec) + except RecursionError: + return True + except ValueError: + # Malformed JSON is reported by the backend validation. + return False + + level: list[Any] = [spec] + for _ in range(limit): + level = [ + child + for node in level + if isinstance(node, (dict, list)) + for child in (node.values() if isinstance(node, dict) else node) + if isinstance(child, (dict, list)) + ] + if not level: + return False + return True + + +def check_json_nesting( + spec: str | dict[str, Any] | None, *, structural_tag: bool = False +) -> None: + """Reject a JSON schema nested more than `MAX_STRUCTURED_OUTPUT_JSON_NESTING` + levels deep, or a structural tag nested deeper than that plus + `STRUCTURAL_TAG_WRAPPER_NESTING`. + + Raises: + VLLMValidationError: If `spec` is nested too deeply. + + """ + limit = MAX_STRUCTURED_OUTPUT_JSON_NESTING + if structural_tag: + limit += STRUCTURAL_TAG_WRAPPER_NESTING + if spec and _exceeds_json_nesting(spec, limit): + raise VLLMValidationError( + "The JSON schema is nested too deeply: objects and arrays may nest at " + f"most {MAX_STRUCTURED_OUTPUT_JSON_NESTING} levels" + ) + + def _is_non_tekken_mistral(tokenizer: TokenizerLike) -> bool: return is_mistral_tokenizer(tokenizer) and not tokenizer.is_tekken @@ -251,8 +306,9 @@ class SamplingParams( """Controls the randomness of the sampling. Lower values make the model more deterministic, while higher values make the model more random. Zero means greedy sampling.""" - watermarking: bool = True - """Whether to apply the engine's configured watermark to this request.""" + watermarking: bool | None = None + """Whether to apply the engine's configured watermark to this request. + ``None`` defaults to enabled.""" top_p: float = 1.0 """Controls the cumulative probability of the top tokens to consider. Must be in (0, 1]. Set to 1 to consider all tokens.""" @@ -342,6 +398,7 @@ class SamplingParams( output_text_buffer_length: int = 0 _eos_token_id: int | None = None _all_stop_token_ids: set[int] = msgspec.field(default_factory=set) + _watermarking_skipped: bool = False # Fields used to construct logits processors structured_outputs: StructuredOutputsParams | None = None @@ -395,7 +452,7 @@ class SamplingParams( frequency_penalty: float | None = 0.0, repetition_penalty: float | None = 1.0, temperature: float | None = 1.0, - watermarking: bool = True, + watermarking: bool | None = None, top_p: float | None = 1.0, top_k: int = 0, min_p: float = 0.0, @@ -1228,6 +1285,9 @@ class SamplingParams( raise VLLMValidationError( "structured_outputs.json cannot be an empty string" ) + # Reject deep nesting before any backend walks or converts the schema + check_json_nesting(self.structured_outputs.json) + check_json_nesting(self.structured_outputs.structural_tag, structural_tag=True) # Reject json_object=False early to avoid engine-side crashes if self.structured_outputs.json_object is False: raise VLLMValidationError( @@ -1411,6 +1471,7 @@ class SamplingParams( _bad_words_token_ids=[[0], [1, 2]], logprobs=5, prompt_logprobs=1, + watermarking=True, ) @@ -1430,6 +1491,7 @@ class BeamSearchParams( include_stop_str_in_output: bool = False structured_outputs: StructuredOutputsParams | None = None skip_special_tokens: bool = True + watermarking: bool | None = None def __post_init__(self) -> None: _verify_num_sequences(self.beam_width, "beam_width") diff --git a/vllm/snapshot/manifest.py b/vllm/snapshot/manifest.py index a029c544c5..c550ee80c5 100644 --- a/vllm/snapshot/manifest.py +++ b/vllm/snapshot/manifest.py @@ -6,7 +6,7 @@ import math import os import stat from pathlib import Path -from typing import Annotated, Any, Literal +from typing import Annotated, Any, Literal, Self from pydantic import ( BaseModel, @@ -19,7 +19,6 @@ from pydantic import ( field_validator, model_validator, ) -from typing_extensions import Self class SnapshotCompatibilityError(RuntimeError): diff --git a/vllm/snapshot/runtime.py b/vllm/snapshot/runtime.py index fdced5116c..83c70e4738 100644 --- a/vllm/snapshot/runtime.py +++ b/vllm/snapshot/runtime.py @@ -24,7 +24,7 @@ import urllib.request from collections.abc import Iterable from contextlib import suppress from dataclasses import dataclass -from datetime import datetime, timezone +from datetime import UTC, datetime from pathlib import Path from typing import NoReturn @@ -872,7 +872,7 @@ class LocalSnapshotTools: return SnapshotManifest( schema_version=1, boundary="post-engine-init-reloadable-state-released", - created_at=datetime.now(timezone.utc).isoformat(), + created_at=datetime.now(UTC).isoformat(), artifact_bytes=self._artifact_bytes(workdir), **identity.model_dump(), model=source_model, diff --git a/vllm/tokenizers/detokenizer_utils.py b/vllm/tokenizers/detokenizer_utils.py index ddaf512442..fc2ce4d44c 100644 --- a/vllm/tokenizers/detokenizer_utils.py +++ b/vllm/tokenizers/detokenizer_utils.py @@ -230,7 +230,13 @@ def detokenize_incrementally( _replace_none_with_empty(new_tokens) # type: ignore[arg-type] else: new_tokens = [""] - output_tokens = prev_tokens + new_tokens + output_token_count = len(prev_tokens) + len(new_tokens) + window_start = ( + 0 if is_first_iter else min(prefix_offset, read_offset, len(prev_tokens)) + ) + output_tokens = ( + prev_tokens[window_start:] if window_start else prev_tokens + ) + new_tokens # If this is the first iteration, return all tokens. if is_first_iter: @@ -241,19 +247,21 @@ def detokenize_incrementally( # surrounding ids. if tokenizer.is_fast or not tokenizer.get_added_vocab(): prefix_text = tokenizer.convert_tokens_to_string( - output_tokens[prefix_offset:read_offset] + output_tokens[prefix_offset - window_start : read_offset - window_start] + ) + new_text = tokenizer.convert_tokens_to_string( + output_tokens[prefix_offset - window_start :] ) - new_text = tokenizer.convert_tokens_to_string(output_tokens[prefix_offset:]) else: prefix_text = _convert_tokens_to_string_with_added_encoders( tokenizer, - output_tokens[prefix_offset:read_offset], + output_tokens[prefix_offset - window_start : read_offset - window_start], skip_special_tokens=skip_special_tokens, spaces_between_special_tokens=spaces_between_special_tokens, ) new_text = _convert_tokens_to_string_with_added_encoders( tokenizer, - output_tokens[prefix_offset:], + output_tokens[prefix_offset - window_start :], skip_special_tokens=skip_special_tokens, spaces_between_special_tokens=spaces_between_special_tokens, ) @@ -266,4 +274,4 @@ def detokenize_incrementally( return new_tokens, "", prefix_offset, read_offset new_text = new_text[len(prefix_text) :] - return new_tokens, new_text, read_offset, len(output_tokens) + return new_tokens, new_text, read_offset, output_token_count diff --git a/vllm/tokenizers/registry.py b/vllm/tokenizers/registry.py index e7064fc252..ebf85a11fd 100644 --- a/vllm/tokenizers/registry.py +++ b/vllm/tokenizers/registry.py @@ -4,10 +4,10 @@ import contextlib from dataclasses import dataclass, field from functools import lru_cache from pathlib import Path -from typing import TYPE_CHECKING +from typing import TYPE_CHECKING, assert_never import huggingface_hub -from typing_extensions import TypeVar, assert_never +from typing_extensions import TypeVar from vllm import envs from vllm.logger import init_logger diff --git a/vllm/tool_parsers/internlm2_tool_parser.py b/vllm/tool_parsers/internlm2_tool_parser.py index 5a99be6afd..709fd6c8f2 100644 --- a/vllm/tool_parsers/internlm2_tool_parser.py +++ b/vllm/tool_parsers/internlm2_tool_parser.py @@ -26,7 +26,7 @@ from vllm.tool_parsers.abstract_tool_parser import ( Tool, ToolParser, ) -from vllm.tool_parsers.utils import extract_intermediate_diff, is_complete_json +from vllm.tool_parsers.utils import find_common_prefix, is_complete_json logger = init_logger(__name__) @@ -172,10 +172,17 @@ class Internlm2ToolParser(ToolParser): elif cur_arguments and prev_arguments: cur_args_json = json.dumps(cur_arguments, ensure_ascii=False) prev_args_json = json.dumps(prev_arguments, ensure_ascii=False) + sent = len(self.streamed_args_for_tool[self.current_tool_id]) - argument_diff = extract_intermediate_diff( - cur_args_json, prev_args_json - ) + # Partial JSON is serialized with synthetic closers, so + # only the part shared with the previous parse is final. + if is_complete_json(parsable_arr): + argument_diff = cur_args_json[sent:] + elif cur_args_json != prev_args_json: + prefix = find_common_prefix(prev_args_json, cur_args_json) + argument_diff = prefix[sent:] + else: + argument_diff = "" delta = DeltaMessage( tool_calls=[ diff --git a/vllm/tool_parsers/longcat_tool_parser.py b/vllm/tool_parsers/longcat_tool_parser.py index ccfa941672..919f801437 100644 --- a/vllm/tool_parsers/longcat_tool_parser.py +++ b/vllm/tool_parsers/longcat_tool_parser.py @@ -9,6 +9,8 @@ from vllm.tool_parsers.hermes_tool_parser import Hermes2ProToolParser class LongcatFlashToolParser(Hermes2ProToolParser): + structural_tag_model = "longcat" + def __init__(self, tokenizer: TokenizerLike, tools: list[Tool] | None = None): super().__init__(tokenizer, tools) diff --git a/vllm/tool_parsers/minimax_m3_tool_parser.py b/vllm/tool_parsers/minimax_m3_tool_parser.py index a8628448c4..6be0687273 100644 --- a/vllm/tool_parsers/minimax_m3_tool_parser.py +++ b/vllm/tool_parsers/minimax_m3_tool_parser.py @@ -1,19 +1,10 @@ # SPDX-License-Identifier: Apache-2.0 # SPDX-FileCopyrightText: Copyright contributors to the vLLM project -from vllm.tool_parsers.rust_tool_parser import RustToolParser +from vllm.parser.engine.registered_adapters import MinimaxM3ParserToolAdapter -class MinimaxM3ToolParser(RustToolParser): - """Adapter from the Rust MiniMax M3 parser to vLLM ToolParser. - - The real M3 grammar lives in the Rust tool-parser crate. This class only - configures the generic Rust bridge with the MiniMax M3 parser name. - - M3 is not M2 with renamed tags: it prefixes each structural tag with the - MiniMax namespace marker, allows multiple ```` tags in one wrapper, - and represents nested arguments with parameter-name XML tags. - """ - - rust_parser_name = "MinimaxM3ToolParser" - tool_call_start_token = "]<]minimax[>[" +class MinimaxM3ToolParser(MinimaxM3ParserToolAdapter): # type: ignore[valid-type, misc] + # Required and named tool choices parse the native M3 syntax instead of + # forcing JSON output. + supports_required_and_named = False diff --git a/vllm/tool_parsers/rust_tool_parser.py b/vllm/tool_parsers/rust_tool_parser.py deleted file mode 100644 index da85825d4c..0000000000 --- a/vllm/tool_parsers/rust_tool_parser.py +++ /dev/null @@ -1,322 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# SPDX-FileCopyrightText: Copyright contributors to the vLLM project - -import importlib -from collections.abc import Sequence -from typing import Any - -from openai.types.responses.function_tool import FunctionTool - -from vllm.entrypoints.chat_utils import make_tool_call_id -from vllm.entrypoints.generate.base.protocol import ( - DeltaFunctionCall, - DeltaMessage, - DeltaToolCall, - ExtractedToolCallInformation, - FunctionCall, - ToolCall, -) -from vllm.entrypoints.openai.chat_completion.protocol import ( - ChatCompletionRequest, - ChatCompletionToolsParam, -) -from vllm.entrypoints.openai.responses.protocol import ResponsesRequest -from vllm.logger import init_logger -from vllm.tokenizers import TokenizerLike -from vllm.tool_parsers.abstract_tool_parser import Tool, ToolParser - -logger = init_logger(__name__) - - -def _rust_tool_parser_module() -> Any: - try: - return importlib.import_module("vllm._rust_tool_parser") - except ImportError as exc: - raise RuntimeError( - "Rust tool parsing requires the vllm._rust_tool_parser PyO3 " - "extension. Rebuild vLLM with Rust frontend/extensions enabled." - ) from exc - - -class RustToolParser(ToolParser): - """Adapter from an opaque Rust parser to the vLLM ToolParser API. - - Subclasses provide only model-specific configuration: the exact Rust parser - name and an optional tool-call start marker for fast complete-output - rejection. - - This class keeps the vLLM-specific bridge work: - - convert vLLM tool definitions into the Rust ``Tool`` shape; - - translate typed Rust parser outputs into vLLM protocol objects; and - - maintain vLLM streaming bookkeeping used by finish-reason handling. - - The parser grammar and incremental parser state stay in Rust. - """ - - # Rust-backed parsers are opaque to Python by default. Do not use vLLM's - # standard JSON required/named handling; let the Rust parser consume the - # model's native tool-call syntax. - supports_required_and_named = False - - rust_parser_name: str - tool_call_start_token: str | None = None - - def __init__(self, tokenizer: TokenizerLike, tools: list[Tool] | None = None): - super().__init__(tokenizer, tools) - self._parser: Any | None = None - self._error: Exception | None = None - - if not self.model_tokenizer: - raise ValueError( - "The model tokenizer must be passed to the ToolParser " - "constructor during construction." - ) - - logger.debug( - "vLLM successfully imported tool parser %s", self.__class__.__name__ - ) - - def adjust_request( - self, request: ChatCompletionRequest | ResponsesRequest - ) -> ChatCompletionRequest | ResponsesRequest: - """Adjust request options without installing Python-side constraints. - - Rust-backed parsers are treated as source-of-truth opaque parsers. The - bridge intentionally avoids ``super().adjust_request()`` so Python does - not install JSON schema guidance or structural-tag constraints that may - conflict with the Rust parser's native grammar. - """ - if self._get_parser().preserve_special_tokens(): - request.skip_special_tokens = False - return request - - def _rust_tools(self) -> list[Any]: - """Build Rust ``Tool`` objects from vLLM tool definitions.""" - if not self.tools: - return [] - - tools: list[Any] = [] - for tool in self.tools: - if isinstance(tool, FunctionTool): - name = tool.name - description = tool.description - parameters = tool.parameters or {} - strict = getattr(tool, "strict", None) - elif isinstance(tool, ChatCompletionToolsParam): - name = tool.function.name - description = tool.function.description - parameters = tool.function.parameters or {} - strict = getattr(tool.function, "strict", None) - else: - continue - tools.append( - _rust_tool_parser_module().Tool(name, description, parameters, strict) - ) - return tools - - def _new_parser(self) -> Any: - """Create a fresh Rust parser with the current tool schemas.""" - return _rust_tool_parser_module().ToolParser( - self.rust_parser_name, self._rust_tools() - ) - - def _get_parser(self) -> Any: - if self._parser is None: - self._parser = self._new_parser() - return self._parser - - def _reset_streaming_state(self) -> None: - """Reset parser state for a new request on a reused parser instance.""" - self._parser = self._new_parser() - self._error = None - self.prev_tool_call_arr.clear() - self.streamed_args_for_tool.clear() - self.current_tool_id = -1 - self.current_tool_name_sent = False - - def _ensure_tool_state(self, index: int) -> None: - """Grow vLLM streaming state arrays to contain ``index``.""" - while len(self.prev_tool_call_arr) <= index: - self.prev_tool_call_arr.append({}) - while len(self.streamed_args_for_tool) <= index: - self.streamed_args_for_tool.append("") - - def _record_delta( - self, index: int, name: str | None, arguments: str | None - ) -> str | None: - """Mirror a Rust parser delta into vLLM streaming bookkeeping. - - ``prev_tool_call_arr`` and ``streamed_args_for_tool`` are read later by - the chat serving layer to decide the final ``tool_calls`` finish reason - and to flush any remaining argument bytes. - """ - tool_call_id = None - self._ensure_tool_state(index) - - if name is not None: - # Prefer the model-emitted ID surfaced by the Rust parser (e.g. - # Kimi K2) over a randomly generated one. - tool_call_id = self._get_parser().tool_call_id(index) or make_tool_call_id() - self.prev_tool_call_arr[index] = {"name": name, "arguments": {}} - self.current_tool_name_sent = True - - if arguments is not None: - self.streamed_args_for_tool[index] += arguments - self.prev_tool_call_arr[index]["arguments"] = self.streamed_args_for_tool[ - index - ] - self.current_tool_id = index - - return tool_call_id - - def _delta_message_from_parser_output( - self, parser_output: Any | None - ) -> DeltaMessage | None: - """Translate one Rust parser output into a vLLM ``DeltaMessage``.""" - if parser_output is None: - return None - - normal_text = parser_output.normal_text or None - tool_calls: list[DeltaToolCall] = [] - for tool_call in parser_output.calls: - index = tool_call.tool_index - name = tool_call.name - arguments: str | None = tool_call.arguments - if name is None and arguments is None: - continue - - tool_call_id = self._record_delta(index, name, arguments) - tool_calls.append( - DeltaToolCall( - index=index, - id=tool_call_id, - type="function" if name is not None else None, - function=DeltaFunctionCall( - name=name, - arguments=arguments, - ), - ) - ) - - if normal_text is None and not tool_calls: - return None - return DeltaMessage(content=normal_text, tool_calls=tool_calls) - - def _parse_complete(self, model_output: str) -> tuple[Any, dict[int, str]] | None: - """Parse complete model output with a throwaway Rust parser instance. - - Returns the coalesced parser output along with any model-emitted tool - call IDs keyed by tool index. - """ - parser = self._new_parser() - output = _rust_tool_parser_module().ToolParserOutput() - try: - parser.parse_into(model_output, output) - # finish() clears parser state, so snapshot model-emitted IDs first. - tool_call_ids = { - call.tool_index: tool_call_id - for call in output.calls - if (tool_call_id := parser.tool_call_id(call.tool_index)) is not None - } - output.append(parser.finish()) - except Exception: - logger.exception( - "Error parsing %s tool call output.", self.rust_parser_name - ) - return None - return output.coalesce(), tool_call_ids - - def extract_tool_calls( - self, - model_output: str, - request: ChatCompletionRequest, - ) -> ExtractedToolCallInformation: - """Extract tool calls from complete model output (non-streaming).""" - if ( - self.tool_call_start_token is not None - and self.tool_call_start_token not in model_output - ): - return ExtractedToolCallInformation( - tools_called=False, - tool_calls=[], - content=model_output, - ) - - parse_result = self._parse_complete(model_output) - if parse_result is None: - return ExtractedToolCallInformation( - tools_called=False, - tool_calls=[], - content=model_output, - ) - parsed, tool_call_ids = parse_result - - tool_calls: list[ToolCall] = [] - self.prev_tool_call_arr.clear() - for parsed_tool_call in parsed.calls: - name = parsed_tool_call.name - arguments = parsed_tool_call.arguments or "{}" - if name is None: - continue - tool_calls.append( - ToolCall( - id=tool_call_ids.get(parsed_tool_call.tool_index) - or make_tool_call_id(), - type="function", - function=FunctionCall(name=name, arguments=arguments), - ) - ) - self.prev_tool_call_arr.append({"name": name, "arguments": arguments}) - - if not tool_calls: - return ExtractedToolCallInformation( - tools_called=False, - tool_calls=[], - content=model_output, - ) - - content = parsed.normal_text or None - return ExtractedToolCallInformation( - tools_called=True, - tool_calls=tool_calls, - content=content, - ) - - def extract_tool_calls_streaming( - self, - previous_text: str, - current_text: str, - delta_text: str, - previous_token_ids: Sequence[int], # pylint: disable=unused-argument - current_token_ids: Sequence[int], # pylint: disable=unused-argument - delta_token_ids: Sequence[int], # pylint: disable=unused-argument - request: ChatCompletionRequest, # pylint: disable=unused-argument - ) -> DeltaMessage | None: - """Extract tool calls from streaming model output. - - The Rust parser owns the incremental buffer, so this adapter feeds only - the newest text delta and lets the serving layer handle final empty - chunks. - """ - # TODO: Add a final-chunk hook if streaming needs to call Rust finish(). - if not previous_text: - self._reset_streaming_state() - - if self._error is not None: - return None - - parser_output = _rust_tool_parser_module().ToolParserOutput() - try: - self._get_parser().parse_into(delta_text, parser_output) - except Exception as error: - self._error = error - logger.exception( - "Error parsing %s streaming tool call output.", - self.rust_parser_name, - ) - - delta_message = self._delta_message_from_parser_output(parser_output) - if delta_message is not None: - return delta_message - - return None diff --git a/vllm/tool_parsers/streaming.py b/vllm/tool_parsers/streaming.py index 70da7f3cc6..07c1a2e031 100644 --- a/vllm/tool_parsers/streaming.py +++ b/vllm/tool_parsers/streaming.py @@ -2,19 +2,15 @@ # SPDX-FileCopyrightText: Copyright contributors to the vLLM project import json +from dataclasses import dataclass from typing import TYPE_CHECKING -import partial_json_parser -import regex as re -from partial_json_parser.core.options import Allow - from vllm.entrypoints.chat_utils import make_tool_call_id from vllm.entrypoints.generate.base.protocol import ( DeltaFunctionCall, DeltaMessage, DeltaToolCall, ) -from vllm.tool_parsers.utils import partial_json_loads from vllm.utils.mistral import is_mistral_tokenizer if TYPE_CHECKING: @@ -23,63 +19,6 @@ else: TokenizerLike = object -def _bracket_level_state( - s: str, opening: str = "{", closing: str = "}" -) -> tuple[int, bool, bool]: - level = 0 - in_string = False - escaped = False - for char in s: - if escaped: - escaped = False - continue - if in_string and char == "\\": - escaped = True - continue - if char == '"': - in_string = not in_string - continue - if not in_string: - if char == opening: - level += 1 - elif char == closing: - level -= 1 - return level, in_string, escaped - - -def filter_delta_text( - delta_text: str, - previous_text: str, -) -> tuple[str, bool]: - """Trim trailing tool-list delimiters from required-tool streaming text.""" - bracket_level, in_string, escaped = _bracket_level_state(previous_text) - updated_delta = "" - passed_zero = False - for char in delta_text: - if escaped: - escaped = False - elif in_string: - if char == "\\": - escaped = True - elif char == '"': - in_string = False - elif char == '"': - in_string = True - elif char == "{": - bracket_level += 1 - passed_zero = bracket_level == 0 - elif char == "}": - bracket_level -= 1 - passed_zero = bracket_level == 0 - - if bracket_level != 0: - updated_delta += char - else: - if not in_string and char == ",": - break - return updated_delta, passed_zero - - def extract_named_tool_call_streaming( *, delta_text: str, @@ -124,100 +63,236 @@ def extract_named_tool_call_streaming( ) +@dataclass +class _RequiredToolCallSpan: + """Where one element of a ``tool_choice="required"`` array sits in the text. + + The model output for ``tool_choice="required"`` is a JSON array of + ``{"name": ..., "parameters": ...}`` objects. ``name`` is only set once + its string value is closed and decodes as JSON; ``name_end`` is the + offset just past that closing quote. ``args_start``/``args_end`` are the + character offsets of the ``parameters`` value (``args_end`` is ``None`` + while the value is still being generated). + + All offsets are prefix-stable: scanning any prefix of the text yields the + same spans truncated to that prefix. That is what lets one scan of the + current text tell what an earlier, shorter text had already revealed. + """ + + name: str | None = None + name_end: int | None = None + args_start: int | None = None + args_end: int | None = None + + def ready_at(self, length: int) -> bool: + """Whether the first ``length`` characters of the text already made + this call streamable: its name was complete and its parameters value + had started.""" + return ( + self.name is not None + and self.name_end is not None + and self.name_end <= length + and self.args_start is not None + and self.args_start < length + ) + + @property + def ready(self) -> bool: + return self.name is not None and self.args_start is not None + + +# Parse states for one element object of the required-tool-call array. +_EXPECT_KEY = 0 +_EXPECT_COLON = 1 +_EXPECT_VALUE = 2 +_IN_STRING_VALUE = 3 +_IN_CONTAINER_VALUE = 4 +_IN_SCALAR_VALUE = 5 +_AFTER_VALUE = 6 + + +def _scan_required_tool_calls(text: str) -> list[_RequiredToolCallSpan]: + """Locate every tool call of a (possibly partial) required-tool-call array. + + A single string- and nesting-aware pass over ``text`` that records, for + each top-level array element, its ``name`` (once complete) and the + character span of its ``parameters`` value. Text after the closing ``]`` + of the array is ignored. Only the element level of the array is + interpreted, so braces, brackets and quotes inside nested values and + strings cannot confuse it. + """ + spans: list[_RequiredToolCallSpan] = [] + depth = 0 # nesting depth of ``{}`` and ``[]``; element objects sit at 2 + in_string = False + escaped = False + string_start = 0 + cur: _RequiredToolCallSpan | None = None + expect = _EXPECT_KEY + key: str | None = None + value_start = 0 + + def end_value(pos: int) -> None: + nonlocal expect + assert cur is not None + if key == "parameters": + cur.args_end = pos + elif key == "name" and expect == _IN_STRING_VALUE: + try: + cur.name = json.loads(text[value_start:pos]) + cur.name_end = pos + except json.JSONDecodeError: + # Not a valid JSON string: the call can never become ready, + # which matches the non-streaming path rejecting the array. + cur.name = None + expect = _AFTER_VALUE + + def start_value(pos: int, state: int) -> None: + nonlocal expect, value_start + assert cur is not None + value_start = pos + if key == "parameters": + cur.args_start = pos + expect = state + + for i, ch in enumerate(text): + if in_string: + if escaped: + escaped = False + elif ch == "\\": + escaped = True + elif ch == '"': + in_string = False + if depth == 2 and cur is not None: + if expect == _EXPECT_KEY: + key = text[string_start + 1 : i] + expect = _EXPECT_COLON + elif expect == _IN_STRING_VALUE: + end_value(i + 1) + continue + + if ch == '"': + in_string = True + string_start = i + if depth == 2 and cur is not None and expect == _EXPECT_VALUE: + start_value(i, _IN_STRING_VALUE) + continue + + if ch in "{[": + if depth == 0: + if ch != "[": + return spans + elif depth == 1: + if ch != "{": + return spans + cur = _RequiredToolCallSpan() + spans.append(cur) + expect = _EXPECT_KEY + key = None + elif depth == 2 and cur is not None and expect == _EXPECT_VALUE: + start_value(i, _IN_CONTAINER_VALUE) + depth += 1 + continue + + if ch in "}]": + if depth == 2 and cur is not None and expect == _IN_SCALAR_VALUE: + end_value(i) + depth -= 1 + if depth == 2 and cur is not None and expect == _IN_CONTAINER_VALUE: + end_value(i + 1) + elif depth == 1: + cur = None + elif depth <= 0: + return spans + continue + + if depth != 2 or cur is None: + continue + + if ch == ":": + if expect == _EXPECT_COLON: + expect = _EXPECT_VALUE + elif ch == ",": + if expect == _IN_SCALAR_VALUE: + end_value(i) + expect = _EXPECT_KEY + key = None + elif ch.isspace(): + if expect == _IN_SCALAR_VALUE: + end_value(i) + elif expect == _EXPECT_VALUE: + start_value(i, _IN_SCALAR_VALUE) + + return spans + + def extract_required_tool_call_streaming( *, previous_text: str, current_text: str | None, - delta_text: str, - function_name_returned: bool, tool_call_idx: int | None, tool_call_id_type: str, ) -> tuple[DeltaMessage | None, bool]: - if current_text is None or current_text == "": - # if the current text is empty, we cannot parse it - return None, function_name_returned - try: - flags = Allow.ALL - obj, _ = partial_json_loads(current_text, flags) - except ( - partial_json_parser.core.exceptions.MalformedJSON, - json.JSONDecodeError, - ): - obj = None + """Stream the tool calls of a ``tool_choice="required"`` JSON array. - # check if the current text is a valid array - # containing a partial tool calling object - # if not repeat - if obj is None or not isinstance(obj, list) or not len(obj) > 0: - function_name_returned = False - delta_message = None - else: - _, finishes_previous_tool = filter_delta_text(delta_text, previous_text) - # take the last tool call from the generated list - current_tool_call = obj[-1] + ``previous_text`` must be a prefix of ``current_text`` (the accumulated + text before and after this delta). Every array element that has something + not yet sent is emitted, in array order, so a delta may carry several + ``DeltaToolCall`` entries: the first chunk of a call carries its id, name + and the arguments generated so far; later chunks carry only new argument + text. What was already sent is read off the same scan, since the spans + are prefix-stable, so no state is needed beyond the two texts. - # once parameters have been generated the name is complete as well - if not finishes_previous_tool and ( - "name" not in current_tool_call or "parameters" not in current_tool_call - ): - function_name_returned = False - delta_message = None - else: - if not function_name_returned: - # get partly generated arguments from the latest tool call - param_match = re.search( - r'.*"parameters":\s*(.*)', current_text, re.DOTALL - ) - if param_match: - arguments = param_match.group(1) - arguments_prefix = current_text[: param_match.start(1)] - arguments, _ = filter_delta_text(arguments, arguments_prefix) - else: - arguments = "" + Returns the delta (``None`` when nothing is new) and whether at least one + call has been announced so far. + """ + if not current_text: + return None, False - # if this iteration finishes a previous tool call but a - # new incomplete tool is already generated, take the - # previous from the list - if finishes_previous_tool and "parameters" not in current_tool_call: - current_tool_call = obj[-2] + spans = _scan_required_tool_calls(current_text) + sent_len = len(previous_text) - function_name_returned = True - tool_call_id = make_tool_call_id( - id_type=tool_call_id_type, - func_name=current_tool_call["name"], - idx=tool_call_idx, - ) - delta_message = DeltaMessage( - tool_calls=[ - DeltaToolCall( - id=tool_call_id, - function=DeltaFunctionCall( - name=current_tool_call["name"], arguments=arguments - ), - index=len(obj) - 1, - type="function", - ) - ] - ) - - else: - delta_text, _ = filter_delta_text(delta_text, previous_text) - - if delta_text != "": - delta_message = DeltaMessage( - tool_calls=[ - DeltaToolCall( - function=DeltaFunctionCall( - # OpenAI API returns None - # instead of name every time - name=None, - arguments=delta_text, - ), - index=len(obj) - 1, - ) - ] + tool_calls: list[DeltaToolCall] = [] + started = 0 + for index, span in enumerate(spans): + if not span.ready: + # Elements are generated in order: nothing later can be ready. + break + assert span.args_start is not None + end = span.args_end if span.args_end is not None else len(current_text) + if span.ready_at(sent_len): + # Already announced: send only the new argument text. + sent_end = min(end, sent_len) + new_arguments = current_text[sent_end:end] + if new_arguments: + tool_calls.append( + DeltaToolCall( + index=index, + function=DeltaFunctionCall( + # OpenAI API returns None instead of name every time + name=None, + arguments=new_arguments, + ), ) - else: - delta_message = None + ) + continue - return delta_message, function_name_returned + idx = None if tool_call_idx is None else tool_call_idx + started + started += 1 + tool_calls.append( + DeltaToolCall( + id=make_tool_call_id( + id_type=tool_call_id_type, func_name=span.name, idx=idx + ), + type="function", + index=index, + function=DeltaFunctionCall( + name=span.name, + arguments=current_text[span.args_start : end], + ), + ) + ) + + function_name_returned = any(span.ready for span in spans) + if not tool_calls: + return None, function_name_returned + return DeltaMessage(tool_calls=tool_calls), function_name_returned diff --git a/vllm/tool_parsers/structural_tag_registry.py b/vllm/tool_parsers/structural_tag_registry.py index a1ee263dc2..f3148a827e 100644 --- a/vllm/tool_parsers/structural_tag_registry.py +++ b/vllm/tool_parsers/structural_tag_registry.py @@ -39,6 +39,7 @@ from vllm.entrypoints.openai.chat_completion.protocol import ( ChatCompletionToolsParam, ) from vllm.tool_parsers.tool_strict_level import ToolStrictLevel +from vllm.tool_parsers.utils import get_function_tools, require_function_tools ToolChoice: TypeAlias = ( Literal["none", "auto", "required"] @@ -78,27 +79,67 @@ XGRAMMAR_BUILTIN_STRUCTURAL_TAG_MODELS = frozenset( } ) VLLM_BUILTIN_STRUCTURAL_TAG_MODELS = frozenset( - {"glm_4_7", "hermes", "hy_v4", "kimi_k3", "plamo3"} + {"glm_4_7", "hermes", "hy_v4", "kimi_k3", "longcat", "plamo3"} ) SUPPORTED_STRUCTURAL_TAG_MODELS = ( XGRAMMAR_BUILTIN_STRUCTURAL_TAG_MODELS | VLLM_BUILTIN_STRUCTURAL_TAG_MODELS ) _VLLM_STRUCTURAL_TAG_REGISTRY: dict[str, StructuralTagBuilder] = {} +_BUILTIN_TOOL_STRUCTURAL_TAG_MODELS: set[str] = set() def register_vllm_structural_tag( model: str, + *, + builtin_tools: bool = False, ) -> Callable[[StructuralTagBuilder], StructuralTagBuilder]: - """Register a vLLM-owned structural tag builder.""" + """Register a vLLM-owned structural tag builder. + + Args: + model: Structural tag model key. + builtin_tools: Whether the model can call non-function tools, so its + builder receives every request tool instead of only the function + tools from ``get_function_tools``. + + """ def decorator(func: StructuralTagBuilder) -> StructuralTagBuilder: _VLLM_STRUCTURAL_TAG_REGISTRY[model] = func + if builtin_tools: + _BUILTIN_TOOL_STRUCTURAL_TAG_MODELS.add(model) return func return decorator +def get_structural_tag_tools( + model: str, + tools: Sequence[ChatCompletionToolsParam | ResponsesTool], + tool_choice: ToolChoice, +) -> Sequence[ChatCompletionToolsParam | ResponsesTool]: + """Return the tools a structural tag for ``model`` lets the model call. + + Most models are only shown function tools, so their grammar must cover + exactly those: a tool missing from the prompt cannot be called, and one + missing from the grammar is shown but blocked. + + Raises: + VLLMValidationError: ``tool_choice`` requires a call, but the model + has no tool it can call. + + """ + if model in _BUILTIN_TOOL_STRUCTURAL_TAG_MODELS: + return tools + if ( + tool_choice is None + or tool_choice == "auto" + or getattr(tool_choice, "mode", None) == "auto" + ): + return get_function_tools(tools) + return require_function_tools(tools) + + def _tool_is_strict(tool: ChatCompletionToolsParam | ResponsesTool) -> bool: if isinstance(tool, FunctionTool): return tool.strict is True @@ -164,6 +205,9 @@ def get_model_structural_tag( if not tools or tool_choice == "none": return None + tools = get_structural_tag_tools(model, tools, tool_choice) + if not tools: + return None tools = resolve_tool_strictness(tools, tool_choice, strict_level) if tools is None: return None @@ -302,15 +346,19 @@ def get_function_parameters(function) -> dict[str, Any] | bool: return function.parameters if function.parameters is not None else True -def _hermes_tool_tags(tools: list[FunctionToolParam]) -> list[TagFormat]: +def _hermes_tool_tags( + tools: list[FunctionToolParam], + start_tag: str, + end_tag: str, +) -> list[TagFormat]: arguments_field_prefix = '", "arguments": ' formats = [ # # {"name": "t1", "arguments": {"q": "v"}} # - ('\n{"name": "', "}\n"), + (start_tag + '\n{"name": "', "}\n" + end_tag), # {"name": "t1", "arguments": {"q": "v"}} - ('{"name": "', "}"), + (start_tag + '{"name": "', "}" + end_tag), ] return [ @@ -326,6 +374,36 @@ def _hermes_tool_tags(tools: list[FunctionToolParam]) -> list[TagFormat]: ] +def _hermes_style_structural_tag( + tools: list[FunctionToolParam], + tool_choice: SimplifiedToolChoice, + start_tag: str, + end_tag: str, +) -> StructuralTag: + tags = _hermes_tool_tags(tools, start_tag, end_tag) + if tool_choice == "auto": + suffix_tag = ( + TriggeredTagsFormat(triggers=[start_tag], tags=tags) + if tags + else AnyTextFormat() + ) + elif tool_choice == "forced": + suffix_tag = TagsWithSeparatorFormat( + tags=tags, + separator="", + at_least_one=True, + stop_after_first=True, + ) + else: + suffix_tag = TagsWithSeparatorFormat( + tags=tags, + separator="", + at_least_one=True, + ) + + return StructuralTag(format=suffix_tag) + + @register_vllm_structural_tag("hermes") def get_hermes_structural_tag( tools: list[FunctionToolParam], @@ -335,31 +413,23 @@ def get_hermes_structural_tag( token_suffix: str = "", ) -> StructuralTag: del builtin_tools, reasoning, token_suffix + return _hermes_style_structural_tag( + tools, tool_choice, "", "" + ) - tool_call_trigger = "" - if tool_choice == "auto": - tags = _hermes_tool_tags(tools) - suffix_tag = ( - TriggeredTagsFormat(triggers=[tool_call_trigger], tags=tags) - if tags - else AnyTextFormat() - ) - elif tool_choice == "forced": - suffix_tag = TagsWithSeparatorFormat( - tags=_hermes_tool_tags(tools), - separator="", - at_least_one=True, - stop_after_first=True, - ) - else: - suffix_tag = TagsWithSeparatorFormat( - tags=_hermes_tool_tags(tools), - separator="", - at_least_one=True, - ) - - return StructuralTag(format=suffix_tag) +@register_vllm_structural_tag("longcat") +def get_longcat_structural_tag( + tools: list[FunctionToolParam], + builtin_tools: list[BuiltinToolParam], + tool_choice: SimplifiedToolChoice, + reasoning: bool, + token_suffix: str = "", +) -> StructuralTag: + del builtin_tools, reasoning, token_suffix + return _hermes_style_structural_tag( + tools, tool_choice, "", "" + ) def _minimax_tool_tags(tools: list[FunctionToolParam]) -> list[TagFormat]: diff --git a/vllm/tool_parsers/utils.py b/vllm/tool_parsers/utils.py index a2992261da..7a93c843b8 100644 --- a/vllm/tool_parsers/utils.py +++ b/vllm/tool_parsers/utils.py @@ -6,6 +6,7 @@ import json import keyword as _python_keyword import math import warnings +from collections.abc import Sequence from dataclasses import dataclass from json import JSONDecodeError, JSONDecoder from typing import Any, TypeAlias @@ -29,6 +30,7 @@ from vllm.entrypoints.openai.chat_completion.protocol import ( ChatCompletionNamedToolChoiceParam, ChatCompletionToolsParam, ) +from vllm.exceptions import VLLMValidationError from vllm.logger import init_logger Tool: TypeAlias = ChatCompletionToolsParam | ResponsesTool @@ -179,43 +181,74 @@ def flat_namespace_tool_name(namespace: str, name: str) -> str: return f"{namespace}{_NAMESPACE_TOOL_SEPARATOR}{name}" +def get_function_tools( + tools: Sequence[Tool], +) -> list[ChatCompletionToolsParam | FunctionTool]: + """Return the function tools the model is shown, in request order. + + This is the single source for both the chat template's tool list and the + tools a tool-call grammar constrains, so the two cannot drift apart. + Namespace functions are flattened to ``__``; other + Responses tool types are never rendered for the model and are dropped. + """ + function_tools: list[ChatCompletionToolsParam | FunctionTool] = [] + for tool in tools: + if isinstance(tool, NamespaceTool): + function_tools.extend( + FunctionTool.model_validate( + { + **namespaced_tool.model_dump(by_alias=True), + "name": flat_namespace_tool_name( + tool.name, namespaced_tool.name + ), + } + ) + for namespaced_tool in tool.tools + if namespaced_tool.type == "function" + ) + elif isinstance(tool, (FunctionTool, ChatCompletionToolsParam)): + function_tools.append(tool) + return function_tools + + +def require_function_tools( + tools: Sequence[Tool], +) -> list[ChatCompletionToolsParam | FunctionTool]: + """Return ``get_function_tools(tools)``, rejecting a request that has none. + + Raises: + VLLMValidationError: ``tools`` has no tool the model can call, so a + tool choice that requires a call can never be satisfied. + + """ + function_tools = get_function_tools(tools) + if not function_tools: + raise VLLMValidationError( + "tool_choice requires a function call, but tools contains no " + "function tool the model can call.", + parameter="tool_choice", + ) + return function_tools + + def iter_response_function_tool_info( tool: ResponsesTool, ) -> list[tuple[str, dict[str, Any] | None]]: - if isinstance(tool, FunctionTool): - return [(tool.name, tool.parameters)] - if not isinstance(tool, NamespaceTool): - return [] - - namespace = tool.name return [ - ( - flat_namespace_tool_name(namespace, namespaced_tool.name), - namespaced_tool.parameters, - ) - for namespaced_tool in tool.tools - if namespaced_tool.type == "function" + (function_tool.name, function_tool.parameters) + for function_tool in get_function_tools([tool]) + if isinstance(function_tool, FunctionTool) ] def iter_response_function_tool_dicts( tools: list[ResponsesTool], ) -> list[dict[str, Any]]: - function_tools: list[dict[str, Any]] = [] - for tool in tools: - if isinstance(tool, NamespaceTool): - namespace = tool.name - for namespaced_tool in tool.tools: - if namespaced_tool.type != "function": - continue - tool_dict = namespaced_tool.model_dump() - tool_dict["name"] = flat_namespace_tool_name( - namespace, namespaced_tool.name - ) - function_tools.append(tool_dict) - elif isinstance(tool, FunctionTool): - function_tools.append(tool.model_dump()) - return function_tools + return [ + tool.model_dump() + for tool in get_function_tools(tools) + if isinstance(tool, FunctionTool) + ] def build_responses_tool_call_name_map( @@ -335,15 +368,22 @@ def _get_tool_schema_defs( ) -> dict: all_defs: dict[str, dict[str, Any]] = {} for tool in tools: - _, params = _extract_tool_info(tool) + name, params = _extract_tool_info(tool) if params is None: continue defs = params.pop("$defs", {}) + if not isinstance(defs, dict): + raise VLLMValidationError( + f"`$defs` in the parameters of tool '{name}' must be an " + f"object, got {type(defs).__name__}.", + parameter="tools", + ) for def_name, def_schema in defs.items(): if def_name in all_defs and all_defs[def_name] != def_schema: - raise ValueError( + raise VLLMValidationError( f"Tool definition '{def_name}' has multiple schemas, " - "which is not supported." + "which is not supported.", + parameter="tools", ) all_defs[def_name] = def_schema return all_defs @@ -353,19 +393,8 @@ def _get_json_schema_from_tools( tools: list[Tool], parallel_tool_calls: bool | None = None, ) -> dict: - fn_tool_schemas: list[dict[str, Any]] = [] - fn_tools: list[Tool] = [] - for tool in tools: - if isinstance(tool, (FunctionTool, NamespaceTool)): - fn_tool_schemas.extend( - _get_tool_schema_from_name_and_params(name, params) - for name, params in iter_response_function_tool_info(tool) - ) - if isinstance(tool, FunctionTool): - fn_tools.append(tool) - elif _is_function_tool(tool): - fn_tool_schemas.append(_get_tool_schema_from_tool(tool)) - fn_tools.append(tool) + fn_tools: list[Tool] = list(require_function_tools(tools)) + fn_tool_schemas = [_get_tool_schema_from_tool(tool) for tool in fn_tools] json_schema: dict[str, Any] = { "type": "array", "minItems": 1, @@ -403,8 +432,6 @@ def get_json_schema_from_tools( continue for name, params in iter_response_function_tool_info(tool): responses_tool_map[name] = params - if "__" in name: - responses_tool_map.setdefault(name.rsplit("__", 1)[1], params) if tool_name not in responses_tool_map: raise ValueError(f"Tool '{tool_name}' has not been passed in `tools`.") return _params_or_empty_object(responses_tool_map[tool_name]) diff --git a/vllm/transformers_utils/config.py b/vllm/transformers_utils/config.py index 4931204714..e046c301d2 100644 --- a/vllm/transformers_utils/config.py +++ b/vllm/transformers_utils/config.py @@ -79,8 +79,6 @@ class LazyConfigDict(dict): _CONFIG_REGISTRY: dict[str, type[PreTrainedConfig]] = LazyConfigDict( - afmoe="AfmoeConfig", - axk1="AXK1Config", bagel="BagelConfig", bailing_moe_v3_vl="BailingMoeV3VLConfig", chatglm="ChatGLMConfig", @@ -89,24 +87,20 @@ _CONFIG_REGISTRY: dict[str, type[PreTrainedConfig]] = LazyConfigDict( colqwen3="ColQwen3Config", ops_colqwen3="OpsColQwen3Config", qwen3_vl_nemotron_embed="Qwen3VLNemotronEmbedConfig", - cosmos3_omni="Cosmos3Config", cosmos3_edge="Cosmos3EdgeConfig", - diffusion_gemma="DiffusionGemmaConfig", deepseek_vl_v2="DeepseekVLV2Config", deepseek_v32="DeepseekV3Config", deepseek_v4="DeepseekV4Config", dots3_note="Dots3NoteConfig", k3_dspark="K3DSparkConfig", funaudiochat="FunAudioChatConfig", - granite4_vision="Granite4VisionConfig", - hyperclovax="HyperCLOVAXConfig", - hy_v3="HYV3Config", hy_v4="HYV4Config", + hyperclovax="HyperCLOVAXConfig", # Upstream class, hub remote code is broken isaac="IsaacConfig", kimi_k2="DeepseekV3Config", # Kimi K2 uses same architecture as DeepSeek V3 kimi_linear="KimiLinearConfig", kimi_vl="KimiVLConfig", - kimi_k25="KimiK25Config", + kimi_k25="Kimi_K25Config", # Upstream class, hub remote code uses old schema muse_glimmer="MuseGlimmerConfig", muse_glimmer_text="MuseGlimmerTextConfig", muse_glimmer_vision="MuseGlimmerVisionConfig", @@ -123,8 +117,6 @@ _CONFIG_REGISTRY: dict[str, type[PreTrainedConfig]] = LazyConfigDict( moss_transcribe_diarize="MossTranscribeDiarizeConfig", eagle="EAGLEConfig", speculators="SpeculatorsConfig", - nemotron="NemotronConfig", - olmo_hybrid="OlmoHybridConfig", openvla="OpenVLAConfig", ovis="OvisConfig", ultravox="UltravoxConfig", @@ -133,12 +125,6 @@ _CONFIG_REGISTRY: dict[str, type[PreTrainedConfig]] = LazyConfigDict( step3p5="Step3p5Config", qianfan_ocr="QianfanOCRConfig", qwen3_asr="Qwen3ASRConfig", - qwen3_next="Qwen3NextConfig", - qwen3_5="Qwen3_5Config", - qwen3_5_text="Qwen3_5TextConfig", - qwen3_5_moe="Qwen3_5MoeConfig", - qwen3_5_moe_text="Qwen3_5MoeTextConfig", - lfm2_moe="Lfm2MoeConfig", **{"unlimited-ocr": "UnlimitedOCRConfig"}, **{"deepseek_v41": "DeepseekV41Config"}, inkling_mm_model="InklingMMConfig", @@ -162,14 +148,6 @@ _PATCH_HF_VALIDATE_ROPE: set[str] = {"sarvam_mla"} # `gemma4_text` ships them in the checkpoint itself. _PATCH_HF_NESTED_ROPE_VALIDATION: set[str] = {"laguna", "gemma4_text"} -# Model types whose checkpoints declare `layer_types` entries that upstream -# transformers has not added to `ALLOWED_LAYER_TYPES` yet, so its strict config -# validation rejects them (e.g. GLM-5.2 `glm_moe_dsa` use -# `deepseek_sparse_attention`). Extend the allowed set for these model types. -_PATCH_HF_ALLOWED_LAYER_TYPES: dict[str, tuple[str, ...]] = { - "glm_moe_dsa": ("deepseek_sparse_attention",), -} - _CONFIG_ATTRS_MAPPING: dict[str, str] = { "llm_config": "text_config", } @@ -314,24 +292,6 @@ def _patch_hf_transformers_nested_rope_validation() -> None: ) -def _patch_hf_transformers_allowed_layer_types( - extra_layer_types: tuple[str, ...], -) -> None: - """Extend transformers' ``ALLOWED_LAYER_TYPES`` so its strict config - validation accepts layer types (e.g. ``deepseek_sparse_attention``) that a - checkpoint declares but upstream transformers has not registered yet. - """ - import transformers.configuration_utils as hf_configuration_utils - - missing = tuple( - layer_type - for layer_type in extra_layer_types - if layer_type not in hf_configuration_utils.ALLOWED_LAYER_TYPES - ) - if missing: - hf_configuration_utils.ALLOWED_LAYER_TYPES += missing - - class HFConfigParser(ConfigParserBase): def parse( self, @@ -379,9 +339,6 @@ class HFConfigParser(ConfigParserBase): if model_type in _PATCH_HF_NESTED_ROPE_VALIDATION: _patch_hf_transformers_nested_rope_validation() - if extra_layer_types := _PATCH_HF_ALLOWED_LAYER_TYPES.get(model_type): - _patch_hf_transformers_allowed_layer_types(extra_layer_types) - if model_type == "vlm": # HyperCLOVAX remote code registers this alias in a bare # `try/except` that fails silently since transformers 5.18 @@ -1139,7 +1096,7 @@ def get_sentence_transformer_tokenizer_config( logger.info("Found sentence-transformers tokenize configuration.") - if all(k in encoder_dict for k in ("max_seq_length", "do_lower_case")): + if "max_seq_length" in encoder_dict: return encoder_dict return None @@ -1355,6 +1312,30 @@ def _read_safetensors_metadata_in_dir(local_dir: Path) -> dict[str, Any]: } +def _read_safetensors_metadata_in_repo( + model: str, + *, + revision: str | None = None, +) -> dict[str, Any]: + api = hf_api() + try: + filenames = [ + filename + for filename in api.list_repo_files(model, revision=revision) + if filename.endswith(".safetensors") + ] + return { + param_name: asdict(info) + for filename in filenames + for param_name, info in api.parse_safetensors_file_metadata( + model, filename, revision=revision + ).tensors.items() + } + except Exception as e: + logger.debug("Could not read safetensors headers for %s: %s", model, e) + return {} + + def get_safetensors_params_metadata( model: str, *, @@ -1372,6 +1353,12 @@ def get_safetensors_params_metadata( for param_name, info in file_mt.tensors.items() } + # `get_safetensors_metadata` only looks for `model.safetensors` and its + # index, so older checkpoints naming their weights differently land here + # even though their headers are perfectly readable. + if metadata := _read_safetensors_metadata_in_repo(model, revision=revision): + return metadata + # Hub fetch failed (e.g. 429, network unreachable). Fall back to the # local HF cache: weights may already be cached from a prior run, and # weight loading itself uses the same cache. @@ -1390,7 +1377,15 @@ def get_safetensors_params_metadata( str(e), ) return {} - return _read_safetensors_metadata_in_dir(Path(local_dir)) + + metadata = _read_safetensors_metadata_in_dir(Path(local_dir)) + if not metadata: + logger.warning_once( + "Could not retrieve safetensors metadata for %s " + "(Hub fetch failed and the cached snapshot holds no weights yet).", + model, + ) + return metadata @cache diff --git a/vllm/transformers_utils/configs/AXK1.py b/vllm/transformers_utils/configs/AXK1.py deleted file mode 100644 index 50bc891573..0000000000 --- a/vllm/transformers_utils/configs/AXK1.py +++ /dev/null @@ -1,216 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# SPDX-FileCopyrightText: Copyright contributors to the vLLM project -from typing import Any - -from transformers import PreTrainedConfig - - -class AXK1Config(PreTrainedConfig): - r"""This is the configuration class to store the configuration of a [`AXK1Model`]. - It is used to instantiate an A.X model according to the specified arguments, - defining the model architecture. Instantiating a configuration with the defaults - will yield a similar configuration to that of the A.X K1. - Configuration objects inherit from [`PreTrainedConfig`] and can be used to control - the model outputs. Read the documentation from [`PreTrainedConfig`] for more - information. - - Args: - vocab_size (`int`, *optional*, defaults to 163840): - Vocabulary size of the A.X K1 model. Defines the number of different - tokens that can be represented by the `inputs_ids` passed when calling - [`AXK1Model`] - hidden_size (`int`, *optional*, defaults to 7168): - Dimension of the hidden representations. - intermediate_size (`int`, *optional*, defaults to 18432): - Dimension of the MLP representations. - moe_intermediate_size (`int`, *optional*, defaults to 2048): - Dimension of the MoE representations. - num_hidden_layers (`int`, *optional*, defaults to 61): - Number of hidden layers in the Transformer decoder. - num_nextn_predict_layers (`int`, *optional*, defaults to 1): - Number of nextn predict layers in the AXK1 Model. - num_attention_heads (`int`, *optional*, defaults to 64): - Number of attention heads for each attention layer in the Transformer - decoder. - n_shared_experts (`int`, *optional*, defaults to 1): - Number of shared experts, None means dense model. - n_routed_experts (`int`, *optional*, defaults to 192): - Number of routed experts, None means dense model. - routed_scaling_factor (`float`, *optional*, defaults to 2.5): - Scaling factor or routed experts. - topk_method (`str`, *optional*, defaults to `noaux_tc`): - Topk method used in routed gate. - n_group (`int`, *optional*, defaults to 8): - Number of groups for routed experts. - topk_group (`int`, *optional*, defaults to 4): - Number of selected groups for each token(for each token, ensuring the - selected experts is only within `topk_group` groups). - num_experts_per_tok (`int`, *optional*, defaults to 8): - Number of selected experts, None means dense model. - moe_layer_freq (`int`, *optional*, defaults to 1): - The frequency of the MoE layer: one expert layer for every - `moe_layer_freq - 1` dense layers. - first_k_dense_replace (`int`, *optional*, defaults to 1): - Number of dense layers in shallow layers - (embed->dense->dense->...->dense->moe->moe...->lm_head). - \--k dense layers--/ - norm_topk_prob (`bool`, *optional*, defaults to True): - Whether to normalize the weights of the routed experts. - scoring_func (`str`, *optional*, defaults to 'sigmoid'): - Method of computing expert weights. - aux_loss_alpha (`float`, *optional*, defaults to 0.0001): - Auxiliary loss weight coefficient. - seq_aux = (`bool`, *optional*, defaults to True): - Whether to compute the auxiliary loss for each individual sample. - num_key_value_heads (`int`, *optional*): - This is the number of key_value heads that should be used to implement - Grouped Query Attention. If `num_key_value_heads=num_attention_heads`, - the model will use Multi Head Attention (MHA), if `num_key_value_heads=1 - the model will use Multi Query Attention (MQA) otherwise GQA is used. When - converting a multi-head checkpoint to a GQA checkpoint, each group key and - value head should be constructed by meanpooling all the original heads - within that group. For more details checkout - [this paper](https://arxiv.org/pdf/2305.13245.pdf). - If it is not specified, will default to `num_attention_heads`. - hidden_act (`str` or `function`, *optional*, defaults to `"silu"`): - The non-linear activation function (function or string) in the decoder. - max_position_embeddings (`int`, *optional*, defaults to 131072): - The maximum sequence length that this model might ever be used with. - initializer_range (`float`, *optional*, defaults to 0.02): - The standard deviation of the truncated_normal_initializer for - initializing all weight matrices. - rms_norm_eps (`float`, *optional*, defaults to 1e-06): - The epsilon used by the rms normalization layers. - use_cache (`bool`, *optional*, defaults to `True`): - Whether or not the model should return the last key/values attentions - (not used by all models). Only relevant if `config.is_decoder=True`. - pad_token_id (`int`, *optional*): - Padding token id. - bos_token_id (`int`, *optional*, defaults to 163691): - Beginning of stream token id. - eos_token_id (`int`, *optional*, defaults to 163691): - End of stream token id. - pretraining_tp (`int`, *optional*, defaults to 1): - Experimental feature. Tensor parallelism rank used during pretraining. - Please refer to - [this document](https://huggingface.co/docs/transformers/parallelism) - to understand more about it. This value is necessary to ensure exact - reproducibility of the pretraining results. Please refer to - [this issue](https://github.com/pytorch/pytorch/issues/76232). - tie_word_embeddings (`bool`, *optional*, defaults to `False`): - Whether to tie weight embeddings - rope_theta (`float`, *optional*, defaults to 10000.0): - The base period of the RoPE embeddings. - rope_scaling (`Dict`, *optional*): - Dictionary containing the scaling configuration for the RoPE embeddings. - Currently supports two scaling strategies: linear and dynamic. - Their scaling factor must be a float greater than 1. The expected format - is `{"type": strategy name, "factor": scaling factor}`. When using this - flag, don't update `max_position_embeddings` to the expected new maximum. - attention_bias (`bool`, defaults to `False`, *optional*, defaults to `False`): - Whether to use a bias in the query, key, value and output projection - layers during self-attention. - attention_dropout (`float`, *optional*, defaults to 0.0): - The dropout ratio for the attention probabilities. - - """ - - model_type = "axk1" - keys_to_ignore_at_inference = ["past_key_values"] - - def __init__( - self, - vocab_size: int = 163840, - hidden_size: int = 7168, - intermediate_size: int = 18432, - moe_intermediate_size: int = 2048, - num_hidden_layers: int = 61, - num_nextn_predict_layers: int | None = 1, - num_attention_heads: int = 64, - num_key_value_heads: int = 64, - n_shared_experts: int | None = 1, - n_routed_experts: int | None = 192, - ep_size: int | None = 8, ## Ignored - Expert parallel size - routed_scaling_factor: float | None = 2.5, - kv_lora_rank: int | None = 512, - q_lora_rank: int | None = 1536, - qk_rope_head_dim: int | None = 64, - v_head_dim: int | None = 128, - qk_nope_head_dim: int | None = 128, - topk_method: str | None = "noaux_tc", - n_group: int | None = 8, - topk_group: int | None = 4, - num_experts_per_tok: int | None = 8, - moe_layer_freq: int | None = 1, - first_k_dense_replace: int = 1, - norm_topk_prob: bool = True, - scoring_func: str | None = "sigmoid", - aux_loss_alpha: float | None = 0.0001, - seq_aux: float | None = True, - hidden_act: str | None = "silu", - max_position_embeddings: int | None = 131072, - initializer_range: float | None = 0.02, - rms_norm_eps: float = 1e-6, - use_cache: bool | None = True, - pad_token_id: int | None = None, - bos_token_id: int | None = 163691, - eos_token_id: int | None = 163691, - pretraining_tp: int | None = 1, - tie_word_embeddings: bool | None = False, - rope_theta: float | None = 10000.0, - rope_scaling: dict[str, Any] | None = None, - rope_parameters: dict[str, Any] | None = None, - attention_bias: bool | None = False, - attention_dropout: float | None = 0.0, - **kwargs, - ): - self.vocab_size = vocab_size - self.max_position_embeddings = max_position_embeddings - self.hidden_size = hidden_size - self.intermediate_size = intermediate_size - self.moe_intermediate_size = moe_intermediate_size - self.num_hidden_layers = num_hidden_layers - self.num_nextn_predict_layers = num_nextn_predict_layers - self.num_attention_heads = num_attention_heads - self.n_shared_experts = n_shared_experts - self.n_routed_experts = n_routed_experts - self.ep_size = ep_size - self.routed_scaling_factor = routed_scaling_factor - self.kv_lora_rank = kv_lora_rank - self.q_lora_rank = q_lora_rank - self.qk_rope_head_dim = qk_rope_head_dim - self.v_head_dim = v_head_dim - self.qk_nope_head_dim = qk_nope_head_dim - self.topk_method = topk_method - self.n_group = n_group - self.topk_group = topk_group - self.num_experts_per_tok = num_experts_per_tok - self.moe_layer_freq = moe_layer_freq - self.first_k_dense_replace = first_k_dense_replace - self.norm_topk_prob = norm_topk_prob - self.scoring_func = scoring_func - self.aux_loss_alpha = aux_loss_alpha - self.seq_aux = seq_aux - # for backward compatibility - if num_key_value_heads is None: - num_key_value_heads = num_attention_heads - - self.num_key_value_heads = num_key_value_heads - self.hidden_act = hidden_act - self.initializer_range = initializer_range - self.rms_norm_eps = rms_norm_eps - self.pretraining_tp = pretraining_tp - self.use_cache = use_cache - self.rope_theta = rope_theta - self.rope_scaling = rope_scaling - self.rope_parameters = rope_parameters - self.attention_bias = attention_bias - self.attention_dropout = attention_dropout - - super().__init__( - pad_token_id=pad_token_id, - bos_token_id=bos_token_id, - eos_token_id=eos_token_id, - tie_word_embeddings=tie_word_embeddings, - **kwargs, - ) diff --git a/vllm/transformers_utils/configs/__init__.py b/vllm/transformers_utils/configs/__init__.py index 5bd47fb948..336b96c1a3 100644 --- a/vllm/transformers_utils/configs/__init__.py +++ b/vllm/transformers_utils/configs/__init__.py @@ -14,8 +14,6 @@ from __future__ import annotations import importlib _CLASS_TO_MODULE: dict[str, str] = { - "AfmoeConfig": "vllm.transformers_utils.configs.afmoe", - "AXK1Config": "vllm.transformers_utils.configs.AXK1", "BagelConfig": "vllm.transformers_utils.configs.bagel", "BailingMoeV3TextConfig": "vllm.transformers_utils.configs.bailing_moe_v3_vl", "BailingMoeV3VisionConfig": "vllm.transformers_utils.configs.bailing_moe_v3_vl", @@ -26,13 +24,10 @@ _CLASS_TO_MODULE: dict[str, str] = { "ColQwen3Config": "vllm.transformers_utils.configs.colqwen3", "OpsColQwen3Config": "vllm.transformers_utils.configs.colqwen3", "Qwen3VLNemotronEmbedConfig": "vllm.transformers_utils.configs.colqwen3", - "Cosmos3Config": "vllm.transformers_utils.configs.cosmos3", "Cosmos3EdgeConfig": "vllm.transformers_utils.configs.cosmos3_edge", "Cosmos3EdgeProjectorConfig": "vllm.transformers_utils.configs.cosmos3_edge", "Cosmos3EdgeTextConfig": "vllm.transformers_utils.configs.cosmos3_edge", "Cosmos3EdgeVisionConfig": "vllm.transformers_utils.configs.cosmos3_edge", - "DiffusionGemmaConfig": "vllm.transformers_utils.configs.diffusion_gemma", - "DiffusionGemmaTextConfig": "vllm.transformers_utils.configs.diffusion_gemma", "DeepseekVLV2Config": "vllm.transformers_utils.configs.deepseek_vl2", "DeepseekV4Config": "vllm.transformers_utils.configs.deepseek_v4", "DeepseekV41Config": "vllm.transformers_utils.configs.deepseek_v41", @@ -42,16 +37,12 @@ _CLASS_TO_MODULE: dict[str, str] = { "EAGLEConfig": "vllm.transformers_utils.configs.eagle", "FunAudioChatConfig": "vllm.transformers_utils.configs.funaudiochat", "FunAudioChatAudioEncoderConfig": "vllm.transformers_utils.configs.funaudiochat", - "Granite4VisionConfig": "vllm.transformers_utils.configs.granite4_vision", - "HYV3Config": "vllm.transformers_utils.configs.hy_v3", "HYV4Config": "vllm.transformers_utils.configs.hy_v4", - "HyperCLOVAXConfig": "vllm.transformers_utils.configs.hyperclovax", "IsaacConfig": "vllm.transformers_utils.configs.isaac", # RWConfig is for the original tiiuae/falcon-40b(-instruct) and # tiiuae/falcon-7b(-instruct) models. Newer Falcon models will use the # `FalconConfig` class from the official HuggingFace transformers library. "RWConfig": "vllm.transformers_utils.configs.falcon", - "Lfm2MoeConfig": "vllm.transformers_utils.configs.lfm2_moe", "MedusaConfig": "vllm.transformers_utils.configs.medusa", "MiDashengLMConfig": "vllm.transformers_utils.configs.midashenglm", "MiniMaxM3Config": "vllm.transformers_utils.configs.minimax_m3", @@ -67,16 +58,12 @@ _CLASS_TO_MODULE: dict[str, str] = { "MoonViTConfig": "vllm.transformers_utils.configs.moonvit", "KimiLinearConfig": "vllm.transformers_utils.configs.kimi_linear", "KimiVLConfig": "vllm.transformers_utils.configs.kimi_vl", - "KimiK25Config": "vllm.transformers_utils.configs.kimi_k25", "MuseGlimmerConfig": "vllm.transformers_utils.configs.muse_glimmer", "MuseGlimmerTextConfig": "vllm.transformers_utils.configs.muse_glimmer", "MuseGlimmerVisionConfig": "vllm.transformers_utils.configs.muse_glimmer", "MuseGlimmerAssistantConfig": "vllm.transformers_utils.configs.muse_glimmer", "KimiK3Config": "vllm.transformers_utils.configs.kimi_k3", "KimiK3VisionConfig": "vllm.transformers_utils.configs.kimi_k3", - "NemotronConfig": "vllm.transformers_utils.configs.nemotron", - "NemotronHConfig": "vllm.transformers_utils.configs.nemotron_h", - "OlmoHybridConfig": "vllm.transformers_utils.configs.olmo_hybrid", "OpenVLAConfig": "vllm.transformers_utils.configs.openvla", "OvisConfig": "vllm.transformers_utils.configs.ovis", "PixelShuffleSiglip2VisionConfig": "vllm.transformers_utils.configs.isaac", @@ -91,22 +78,17 @@ _CLASS_TO_MODULE: dict[str, str] = { "QianfanOCRConfig": "vllm.transformers_utils.configs.qianfan_ocr", "QianfanOCRVisionConfig": "vllm.transformers_utils.configs.qianfan_ocr", "Qwen3ASRConfig": "vllm.transformers_utils.configs.qwen3_asr", - "Qwen3NextConfig": "vllm.transformers_utils.configs.qwen3_next", - "Qwen3_5Config": "vllm.transformers_utils.configs.qwen3_5", - "Qwen3_5TextConfig": "vllm.transformers_utils.configs.qwen3_5", - "Qwen3_5MoeConfig": "vllm.transformers_utils.configs.qwen3_5_moe", - "Qwen3_5MoeTextConfig": "vllm.transformers_utils.configs.qwen3_5_moe", "InklingModelConfig": "vllm.models.inkling.configs", "InklingAudioConfig": "vllm.models.inkling.configs", "InklingVisionConfig": "vllm.models.inkling.configs", "InklingMMConfig": "vllm.models.inkling.configs", - # Special case: DeepseekV3Config is from HuggingFace Transformers + # Upstream Transformers classes registered in _CONFIG_REGISTRY "DeepseekV3Config": "transformers", + "HyperCLOVAXConfig": "transformers", + "Kimi_K25Config": "transformers", } __all__ = [ - "AfmoeConfig", - "AXK1Config", "BagelConfig", "BailingMoeV3TextConfig", "BailingMoeV3VisionConfig", @@ -117,13 +99,10 @@ __all__ = [ "ColQwen3Config", "OpsColQwen3Config", "Qwen3VLNemotronEmbedConfig", - "Cosmos3Config", "Cosmos3EdgeConfig", "Cosmos3EdgeProjectorConfig", "Cosmos3EdgeTextConfig", "Cosmos3EdgeVisionConfig", - "DiffusionGemmaConfig", - "DiffusionGemmaTextConfig", "DeepseekVLV2Config", "DeepseekV3Config", "DeepseekV4Config", @@ -133,13 +112,10 @@ __all__ = [ "EAGLEConfig", "FunAudioChatConfig", "FunAudioChatAudioEncoderConfig", - "Granite4VisionConfig", - "HYV3Config", "HYV4Config", "HyperCLOVAXConfig", "IsaacConfig", "RWConfig", - "Lfm2MoeConfig", "MedusaConfig", "MiDashengLMConfig", "MiniMaxM3Config", @@ -153,16 +129,13 @@ __all__ = [ "MoonViTConfig", "KimiLinearConfig", "KimiVLConfig", - "KimiK25Config", + "Kimi_K25Config", "MuseGlimmerConfig", "MuseGlimmerTextConfig", "MuseGlimmerVisionConfig", "MuseGlimmerAssistantConfig", "KimiK3Config", "KimiK3VisionConfig", - "NemotronConfig", - "NemotronHConfig", - "OlmoHybridConfig", "OpenVLAConfig", "OvisConfig", "PixelShuffleSiglip2VisionConfig", @@ -177,11 +150,6 @@ __all__ = [ "QianfanOCRConfig", "QianfanOCRVisionConfig", "Qwen3ASRConfig", - "Qwen3NextConfig", - "Qwen3_5Config", - "Qwen3_5TextConfig", - "Qwen3_5MoeConfig", - "Qwen3_5MoeTextConfig", "InklingModelConfig", "InklingAudioConfig", "InklingVisionConfig", diff --git a/vllm/transformers_utils/configs/afmoe.py b/vllm/transformers_utils/configs/afmoe.py deleted file mode 100644 index b3d10dc5f2..0000000000 --- a/vllm/transformers_utils/configs/afmoe.py +++ /dev/null @@ -1,87 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# SPDX-FileCopyrightText: Copyright contributors to the vLLM project - -from transformers.configuration_utils import PreTrainedConfig - - -class AfmoeConfig(PreTrainedConfig): - model_type = "afmoe" - - def __init__( - self, - vocab_size: int = 200_192, - hidden_size: int = 2048, - intermediate_size: int = 6144, - moe_intermediate_size: int = 1408, - num_hidden_layers: int = 32, - num_dense_layers: int = 1, - num_attention_heads: int = 16, - num_key_value_heads: int | None = None, - head_dim: int = 128, - hidden_act: str = "silu", - max_position_embeddings: int = 131072, - initializer_range: float = 0.02, - rms_norm_eps: float = 1e-5, - use_cache: bool = True, - tie_word_embeddings: bool = False, - rope_parameters: dict | None = None, - rope_scaling: dict | None = None, - num_experts: int = 64, - num_experts_per_tok: int = 6, - num_shared_experts: int = 2, - num_expert_groups: int = 1, - num_limited_groups: int = 1, - score_func: str = "sigmoid", - route_norm: bool = True, - route_scale: float = 1.0, - global_attn_every_n_layers: int = 4, - sliding_window: int = 2048, - layer_types: list[str] | None = None, - attention_dropout: float = 0.0, - mup_enabled: bool = False, - n_group: int = 1, - topk_group: int = 1, - **kwargs, - ): - self.vocab_size = vocab_size - self.hidden_size = hidden_size - self.intermediate_size = intermediate_size - self.num_hidden_layers = num_hidden_layers - self.num_dense_layers = num_dense_layers - self.num_attention_heads = num_attention_heads - self.num_key_value_heads = num_key_value_heads or num_attention_heads - self.head_dim = head_dim - self.hidden_act = hidden_act - self.max_position_embeddings = max_position_embeddings - self.initializer_range = initializer_range - self.rms_norm_eps = rms_norm_eps - self.use_cache = use_cache - rope_theta = kwargs.pop("rope_theta", 10000.0) - if rope_parameters is None: - rope_parameters = {"rope_type": "default", "rope_theta": rope_theta} - self.rope_parameters = rope_parameters - self.rope_scaling = rope_scaling - - self.moe_intermediate_size = moe_intermediate_size - self.num_experts = num_experts - self.num_experts_per_tok = num_experts_per_tok - self.num_shared_experts = num_shared_experts - self.num_expert_groups = num_expert_groups - self.num_limited_groups = num_limited_groups - self.score_func = score_func - self.route_norm = route_norm - self.route_scale = route_scale - - self.global_attn_every_n_layers = global_attn_every_n_layers - self.sliding_window = sliding_window - self.layer_types = layer_types - self.attention_dropout = attention_dropout - - self.mup_enabled = mup_enabled - self.n_group = n_group - self.topk_group = topk_group - - super().__init__(tie_word_embeddings=tie_word_embeddings, **kwargs) - - -__all__ = ["AfmoeConfig"] diff --git a/vllm/transformers_utils/configs/cosmos3.py b/vllm/transformers_utils/configs/cosmos3.py deleted file mode 100644 index 965801888a..0000000000 --- a/vllm/transformers_utils/configs/cosmos3.py +++ /dev/null @@ -1,7 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# SPDX-FileCopyrightText: Copyright contributors to the vLLM project -from transformers.models.qwen3_vl.configuration_qwen3_vl import Qwen3VLConfig - - -class Cosmos3Config(Qwen3VLConfig): - model_type = "cosmos3_omni" diff --git a/vllm/transformers_utils/configs/cosmos3_edge.py b/vllm/transformers_utils/configs/cosmos3_edge.py index 447b64cc99..ddf17933fd 100644 --- a/vllm/transformers_utils/configs/cosmos3_edge.py +++ b/vllm/transformers_utils/configs/cosmos3_edge.py @@ -1,11 +1,10 @@ # SPDX-License-Identifier: Apache-2.0 # SPDX-FileCopyrightText: Copyright contributors to the vLLM project +from transformers import NemotronHConfig from transformers.configuration_utils import PreTrainedConfig from transformers.models.siglip2.configuration_siglip2 import Siglip2VisionConfig -from .nemotron_h import NemotronHConfig - def _normalize_mrope_parameters(config: NemotronHConfig) -> None: """Normalize the checkpoint's mRoPE configuration for vLLM.""" @@ -31,11 +30,14 @@ class Cosmos3EdgeTextConfig(NemotronHConfig): num_hidden_layers: int = 28, hidden_act: str = "relu2", rms_norm_eps: float = 1e-5, + layers_block_type: list[str] | None = None, **kwargs, ) -> None: + # Each decoder block is an attention layer followed by an MLP layer + if layers_block_type is None: + layers_block_type = ["full_attention", "mlp"] * num_hidden_layers super().__init__( - num_hidden_layers=2 * num_hidden_layers, - hybrid_override_pattern="*-" * num_hidden_layers, + layers_block_type=layers_block_type, mlp_hidden_act=hidden_act, layer_norm_epsilon=rms_norm_eps, **kwargs, @@ -46,7 +48,6 @@ class Cosmos3EdgeTextConfig(NemotronHConfig): config_dict["num_hidden_layers"] = self.num_hidden_layers // 2 config_dict["hidden_act"] = config_dict.pop("mlp_hidden_act") config_dict["rms_norm_eps"] = config_dict.pop("layer_norm_epsilon") - config_dict.pop("hybrid_override_pattern", None) return config_dict diff --git a/vllm/transformers_utils/configs/deepseek_vl2.py b/vllm/transformers_utils/configs/deepseek_vl2.py index 3543db833a..a819f12f3e 100644 --- a/vllm/transformers_utils/configs/deepseek_vl2.py +++ b/vllm/transformers_utils/configs/deepseek_vl2.py @@ -90,11 +90,6 @@ class MlpProjectorConfig(PreTrainedConfig): @strict class DeepseekVLV2TextConfig(DeepseekV2Config): - # DeepSeek-VL2 checkpoints (e.g. deepseek-vl2-small) omit ``vocab_size`` and - # rely on the original DeepSeek-VL2 default of 102400; the generic - # Transformers ``DeepseekV2Config`` incorrectly defaults it to 32000. - # Upstream fix: https://github.com/huggingface/transformers/pull/48159 - vocab_size: int = 102400 kv_lora_rank: int | None = DeepseekV2Config.kv_lora_rank diff --git a/vllm/transformers_utils/configs/diffusion_gemma.py b/vllm/transformers_utils/configs/diffusion_gemma.py deleted file mode 100644 index 884d176cd0..0000000000 --- a/vllm/transformers_utils/configs/diffusion_gemma.py +++ /dev/null @@ -1,44 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# SPDX-FileCopyrightText: Copyright contributors to the vLLM project -from typing import Any - -from transformers import PreTrainedConfig -from transformers.models.gemma4.configuration_gemma4 import Gemma4VisionConfig - - -def _init_text_config(self: PreTrainedConfig, **kwargs: Any) -> None: - PreTrainedConfig.__init__(self, **kwargs) - # DiffusionGemma always uses MoE and K=V sharing for full_attention - # layers. The HF reference removed these config fields entirely. - if getattr(self, "num_experts", None): - self.enable_moe_block = True - self.attention_k_eq_v = True - - -class DiffusionGemmaTextConfig(PreTrainedConfig): - model_type = "diffusion_gemma_text" - - def __init__(self, **kwargs: Any): - _init_text_config(self, **kwargs) - - -class DiffusionGemmaConfig(PreTrainedConfig): - model_type = "diffusion_gemma" - - def __init__( - self, - text_config: dict[str, Any] | None = None, - canvas_length: int = 256, - self_conditioning_size: int | None = None, - **kwargs: Any, - ): - self.text_config = DiffusionGemmaTextConfig(**(text_config or {})) - self.canvas_length = canvas_length - self.self_conditioning_size = self_conditioning_size - vision_config = kwargs.pop("vision_config", None) - if isinstance(vision_config, dict): - self.vision_config = Gemma4VisionConfig(**vision_config) - else: - self.vision_config = vision_config - self.audio_config = None - PreTrainedConfig.__init__(self, **kwargs) diff --git a/vllm/transformers_utils/configs/gemma4.py b/vllm/transformers_utils/configs/gemma4.py deleted file mode 100644 index 69cbc06d04..0000000000 --- a/vllm/transformers_utils/configs/gemma4.py +++ /dev/null @@ -1,34 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# SPDX-FileCopyrightText: Copyright contributors to the vLLM project -"""Per-layer config resolution for Gemma4 and its variants.""" - -from copy import copy - -from transformers import PreTrainedConfig - - -def gemma4_layer_config( - text_config: PreTrainedConfig, layer_idx: int -) -> PreTrainedConfig: - """The Gemma4 text config as it applies to one layer. - - Gemma4 uses a larger head dimension on its full attention layers than on its - sliding ones, and with `attention_k_eq_v` it uses more KV heads there too. - Transformers >= 5.15.0 says so in the config itself; before that the values - are flat attributes that `layer_types` picks between, so resolve them here. - The result is homogeneous either way, so callers read `head_dim` and - `num_key_value_heads` off it without caring which version they are on. - """ - if getattr(text_config, "is_heterogeneous", False): - return text_config.per_layer_config[layer_idx] - - layer = copy(text_config) - if text_config.layer_types[layer_idx] == "full_attention": - global_head_dim = getattr(text_config, "global_head_dim", None) - layer.head_dim = global_head_dim or text_config.head_dim - if getattr(text_config, "attention_k_eq_v", False): - global_kv_heads = getattr(text_config, "num_global_key_value_heads", None) - layer.num_key_value_heads = ( - global_kv_heads or text_config.num_key_value_heads - ) - return layer diff --git a/vllm/transformers_utils/configs/granite4_vision.py b/vllm/transformers_utils/configs/granite4_vision.py deleted file mode 100644 index 8522bdd47a..0000000000 --- a/vllm/transformers_utils/configs/granite4_vision.py +++ /dev/null @@ -1,94 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# SPDX-FileCopyrightText: Copyright contributors to the vLLM project - -from typing import Any - -import transformers - - -class Granite4VisionConfig(transformers.PreTrainedConfig): - """Configuration for Granite 4 Vision model. - - This config is needed because the granite4_vision model type is not yet - in the transformers version pinned by vLLM. Once transformers adds native - support, this file can be removed and the _CONFIG_REGISTRY entry dropped. - """ - - model_type = "granite4_vision" - is_composition = False - - def __init__( - self, - vision_config: dict[str, Any] | None = None, - text_config: dict[str, Any] | None = None, - image_token_index: int = 100352, - image_seq_length: int = 576, - image_grid_pinpoints: list[list[int]] | None = None, - vision_feature_select_strategy: str = "full", - vision_feature_layer: int | list[int] = -2, - projector_hidden_act: str = "gelu", - projector_dropout: float = 0.1, - downsample_rate: str | None = None, - use_image_newline_parameter: bool = True, - deepstack_layer_map: list[list[int]] | None = None, - use_spatial_sampling: bool = False, - spatial_stride: int = 2, - spatial_vision_layer: int = -1, - spatial_target_layers: list[int] | None = None, - # Hub aliases — base model config uses different field names - vision_layer_to_llm_layer: list[list[int]] | None = None, - use_checkerboard_sampling: bool | None = None, - checkerboard_stride: int | None = None, - checkerboard_vision_layer: int | None = None, - checkerboard_llm_layers: list[int] | None = None, - **kwargs: Any, - ): - self.image_token_index = image_token_index - self.image_seq_length = image_seq_length - self.image_grid_pinpoints = image_grid_pinpoints or [] - self.vision_feature_select_strategy = vision_feature_select_strategy - self.vision_feature_layer = vision_feature_layer - self.projector_hidden_act = projector_hidden_act - self.projector_dropout = projector_dropout - self.downsample_rate = downsample_rate - self.use_image_newline_parameter = use_image_newline_parameter - self.deepstack_layer_map = deepstack_layer_map or vision_layer_to_llm_layer - self.use_spatial_sampling = ( - use_spatial_sampling - if use_checkerboard_sampling is None - else use_checkerboard_sampling - ) - self.spatial_stride = ( - spatial_stride if checkerboard_stride is None else checkerboard_stride - ) - self.spatial_vision_layer = ( - spatial_vision_layer - if checkerboard_vision_layer is None - else checkerboard_vision_layer - ) - self.spatial_target_layers = ( - spatial_target_layers or checkerboard_llm_layers or [0, 10, 20, 30] - ) - - if vision_config is None: - vision_config = {} - if text_config is None: - text_config = {} - - vision_model_type = vision_config.get("model_type", "siglip_vision_model") - if vision_model_type in transformers.CONFIG_MAPPING: - self.vision_config = transformers.CONFIG_MAPPING[vision_model_type]( - **vision_config - ) - else: - self.vision_config = transformers.PreTrainedConfig(**vision_config) - - text_model_type = text_config.get("model_type", "granite") - if text_model_type in transformers.CONFIG_MAPPING: - self.text_config = transformers.CONFIG_MAPPING[text_model_type]( - **text_config - ) - else: - self.text_config = transformers.PreTrainedConfig(**text_config) - - super().__init__(**kwargs) diff --git a/vllm/transformers_utils/configs/hy_v3.py b/vllm/transformers_utils/configs/hy_v3.py deleted file mode 100644 index 939dd5d78c..0000000000 --- a/vllm/transformers_utils/configs/hy_v3.py +++ /dev/null @@ -1,185 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# SPDX-FileCopyrightText: Copyright contributors to the vLLM project -from typing import Any - -from transformers.configuration_utils import PreTrainedConfig - - -class HYV3Config(PreTrainedConfig): - r"""This is the configuration class to store the configuration of a [`HYV3Model`]. - It is used to instantiate a HYV3 model (HY V3 MoE language model) according to - the specified arguments. - - Configuration objects inherit from [`PreTrainedConfig`] and can be used to - control the model outputs. Read the documentation from [`PreTrainedConfig`] - for more information. - - Args: - vocab_size (`int`, *optional*, defaults to 120832): - Vocabulary size of the model. - hidden_size (`int`, *optional*, defaults to 4096): - Dimension of the hidden representations. - intermediate_size (`int`, *optional*, defaults to 13312): - Dimension of the dense FFN intermediate representations. - num_hidden_layers (`int`, *optional*, defaults to 80): - Number of hidden layers in the Transformer decoder. - num_attention_heads (`int`, *optional*, defaults to 64): - Number of attention heads for each attention layer. - num_key_value_heads (`int`, *optional*, defaults to 8): - Number of key-value heads for grouped-query attention. - head_dim (`int`, *optional*, defaults to 128): - Dimension per attention head. - hidden_act (`str`, *optional*, defaults to `"silu"`): - Activation function used in FFN layers. - max_position_embeddings (`int`, *optional*, defaults to 131072): - Maximum sequence length supported by the model. - initializer_range (`float`, *optional*, defaults to 0.006): - Standard deviation of the truncated normal initializer for weight - initialization. - rms_norm_eps (`float`, *optional*, defaults to 1e-5): - Epsilon for RMS normalization layers. - use_cache (`bool`, *optional*, defaults to `True`): - Whether to use KV cache for decoding. - pad_token_id (`int`, *optional*): - Padding token id. - bos_token_id (`int`, *optional*): - Beginning-of-sequence token id. - eos_token_id (`int` or `List[int]`, *optional*): - End-of-sequence token id(s). - rope_parameters (`dict`, *optional*): - The parameters of the RoPE embeddings. - qk_norm (`bool`, *optional*, defaults to `True`): - Whether to apply RMSNorm to query and key states before attention. - tie_word_embeddings (`bool`, *optional*, defaults to `False`): - Whether to tie input and output embedding weights. - enable_attention_fp32_softmax (`bool`, *optional*, defaults to `False`): - Whether to upcast attention softmax to float32. Note: the eager attention - path always computes softmax in float32 regardless of this setting; this - flag is reserved for future use with custom attention backends. - enable_lm_head_fp32 (`bool`, *optional*, defaults to `True`): - Whether to upcast the LM head computation to float32. - num_experts (`int`, *optional*, defaults to 192): - Total number of MoE experts. - num_experts_per_tok (`int`, *optional*, defaults to 8): - Number of experts selected per token (top-k routing). - num_shared_experts (`int`, *optional*, defaults to 1): - Number of always-active shared experts combined into a single MLP. - expert_hidden_dim (`int`, *optional*, defaults to 1536): - Intermediate dimension of each individual MoE expert. - moe_router_enable_expert_bias (`bool`, *optional*, defaults to `True`): - Whether to use per-expert load-balancing bias in the router. - moe_router_use_sigmoid (`bool`, *optional*, defaults to `True`): - Whether to use sigmoid (instead of softmax) for router scoring. - route_norm (`bool`, *optional*, defaults to `True`): - Whether to normalize routing scores when using sigmoid routing. - router_scaling_factor (`float`, *optional*): - Optional multiplicative scaling factor applied to routing scores. - use_grouped_mm (`bool`, *optional*, defaults to `False`): - Whether to use grouped GEMM for expert computation (not yet implemented). - enable_moe_fp32_combine (`bool`, *optional*, defaults to `False`): - Whether to accumulate expert outputs in float32. - first_k_dense_replace (`int`, *optional*, defaults to 1): - Number of initial decoder layers that use a dense FFN instead of MoE. - output_router_logits (`bool`, *optional*, defaults to `False`): - Whether to output router logits from each MoE layer. Useful for computing - auxiliary load-balancing loss during training. Disabled by default to avoid - the memory overhead of storing per-layer router tensors during inference. - - Example: - ```python - >>> from transformers import HYV3Config, HYV3Model - - >>> config = HYV3Config() - >>> model = HYV3Model(config) - ``` - - """ - - model_type = "hy_v3" - keys_to_ignore_at_inference = ["past_key_values"] - - def __init__( - self, - vocab_size=120832, - hidden_size=4096, - intermediate_size=13312, - num_hidden_layers=80, - num_attention_heads=64, - num_key_value_heads=8, - head_dim=128, - hidden_act="silu", - max_position_embeddings=131072, - initializer_range=0.006, - rms_norm_eps=1e-5, - use_cache=True, - pad_token_id=None, - bos_token_id=None, - eos_token_id=None, - rope_parameters: dict[str, Any] | None = None, - qk_norm=True, - tie_word_embeddings=False, - enable_attention_fp32_softmax=False, - enable_lm_head_fp32=True, - # MoE specific - num_experts=192, - num_experts_per_tok=8, - num_shared_experts=1, - expert_hidden_dim=1536, - moe_router_enable_expert_bias=True, - moe_router_use_sigmoid=True, - route_norm=True, - router_scaling_factor=None, - use_grouped_mm=False, - enable_moe_fp32_combine=False, - # Dense/MoE layer control - first_k_dense_replace=1, - output_router_logits=False, - **kwargs, - ): - self.vocab_size = vocab_size - self.hidden_size = hidden_size - self.intermediate_size = intermediate_size - self.num_hidden_layers = num_hidden_layers - self.num_attention_heads = num_attention_heads - self.num_key_value_heads = num_key_value_heads - self.head_dim = head_dim - self.hidden_act = hidden_act - self.max_position_embeddings = max_position_embeddings - self.initializer_range = initializer_range - self.rms_norm_eps = rms_norm_eps - self.use_cache = use_cache - rope_theta = kwargs.pop("rope_theta", 11158840.0) - if rope_parameters is None: - rope_parameters = {"rope_type": "default", "rope_theta": rope_theta} - self.rope_parameters = rope_parameters - self.qk_norm = qk_norm - self.tie_word_embeddings = tie_word_embeddings - self.enable_lm_head_fp32 = enable_lm_head_fp32 - self.enable_attention_fp32_softmax = enable_attention_fp32_softmax - - # MoE specific - self.num_experts = num_experts - self.num_experts_per_tok = num_experts_per_tok - self.num_shared_experts = num_shared_experts - self.expert_hidden_dim = expert_hidden_dim - self.moe_router_enable_expert_bias = moe_router_enable_expert_bias - self.moe_router_use_sigmoid = moe_router_use_sigmoid - self.route_norm = route_norm - self.use_grouped_mm = use_grouped_mm - self.router_scaling_factor = router_scaling_factor - self.enable_moe_fp32_combine = enable_moe_fp32_combine - - # Dense/MoE layer control - self.first_k_dense_replace = first_k_dense_replace - self.output_router_logits = output_router_logits - - if eos_token_id is not None and isinstance(eos_token_id, int): - eos_token_id = [eos_token_id] - - super().__init__( - pad_token_id=pad_token_id, - bos_token_id=bos_token_id, - eos_token_id=eos_token_id, - tie_word_embeddings=tie_word_embeddings, - **kwargs, - ) diff --git a/vllm/transformers_utils/configs/hyperclovax.py b/vllm/transformers_utils/configs/hyperclovax.py deleted file mode 100644 index 5470408b58..0000000000 --- a/vllm/transformers_utils/configs/hyperclovax.py +++ /dev/null @@ -1,277 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# SPDX-FileCopyrightText: Copyright contributors to the vLLM project -# SPDX-FileCopyrightText: Copyright 2025 NAVER Cloud HyperCLOVA team -# -# Copyright 2025 NAVER Cloud HyperCLOVA team. All rights reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -"""HyperCLOVA X model configuration.""" - -from transformers.configuration_utils import PreTrainedConfig - - -class HyperCLOVAXConfig(PreTrainedConfig): - r"""This is the configuration class to store the configuration of a - [`HyperCLOVAXModel`]. It is used to instantiate a HyperCLOVAX model - according to the specified arguments, defining the model architecture. - Configuration objects inherit from [`PreTrainedConfig`] and can be used - to control the model outputs. Read the documentation from - [`PreTrainedConfig`] for more information. - - Args: - vocab_size (`int`, *optional*, defaults to 32000): - Vocabulary size of the HyperCLOVAX model. Defines the number of - different tokens that can be represented by the `input_ids` - passed when calling [`HyperCLOVAXModel`] - hidden_size (`int`, *optional*, defaults to 4096): - Dimension of the hidden representations. - intermediate_size (`int`, *optional*, defaults to 11008): - Dimension of the MLP representations. - num_hidden_layers (`int`, *optional*, defaults to 32): - Number of hidden layers in the Transformer decoder. - num_attention_heads (`int`, *optional*, defaults to 32): - Number of attention heads for each attention layer in the - Transformer decoder. - num_key_value_heads (`int`, *optional*): - This is the number of key_value heads that should be used to - implement Grouped Query Attention. If - `num_key_value_heads=num_attention_heads`, the model will use - Multi Head Attention (MHA), if `num_key_value_heads=1` the model - will use Multi Query Attention (MQA) otherwise GQA is used. When - converting a multi-head checkpoint to a GQA checkpoint, each - group key and value head should be constructed by meanpooling all - the original heads within that group. For more details checkout - [this paper](https://arxiv.org/pdf/2305.13245.pdf). If it is not - specified, will default to `num_attention_heads`. - hidden_act (`str` or `function`, *optional*, defaults to `"silu"`): - The non-linear activation function (function or string) in the - decoder. - max_position_embeddings (`int`, *optional*, defaults to 2048): - The maximum sequence length that this model might ever be used - with. - initializer_range (`float`, *optional*, defaults to 0.02): - The standard deviation of the truncated_normal_initializer for - initializing all weight matrices. - rms_norm_eps (`float`, *optional*, defaults to 1e-06): - The epsilon used by the rms normalization layers. - use_cache (`bool`, *optional*, defaults to `True`): - Whether or not the model should return the last key/values - attentions (not used by all models). Only relevant if - `config.is_decoder=True`. - pad_token_id (`int`, *optional*): - Padding token id. - bos_token_id (`int`, *optional*, defaults to 1): - Beginning of stream token id. - eos_token_id (`int`, *optional*, defaults to 2): - End of stream token id. - pretraining_tp (`int`, *optional*, defaults to 1): - Experimental feature. Tensor parallelism rank used during - pretraining. Please refer to [this document](https://huggingface. - co/docs/transformers/main/perf_train_gpu_many#tensor-parallelism) - to understand more about it. This value is necessary to ensure - exact reproducibility of the pretraining results. Please refer to - [this issue](https://github.com/pytorch/pytorch/issues/76232). - tie_word_embeddings (`bool`, *optional*, defaults to `False`): - Whether to tie weight embeddings - rope_theta (`float`, *optional*, defaults to 10000.0): - The base period of the RoPE embeddings. - rope_scaling (`Dict`, *optional*): - Dictionary containing the scaling configuration for the RoPE - embeddings. NOTE: if you apply new rope type and you expect the - model to work on longer `max_position_embeddings`, we recommend - you to update this value accordingly. - Expected contents: - `rope_type` (`str`): - The sub-variant of RoPE to use. Can be one of ['default', - 'linear', 'dynamic', 'yarn', 'longrope', 'llama3'], with - 'default' being the original RoPE implementation. - `factor` (`float`, *optional*): - Used with all rope types except 'default'. The scaling - factor to apply to the RoPE embeddings. In most scaling - types, a `factor` of x will enable the model to handle - sequences of length x * original maximum pre-trained - length. - `original_max_position_embeddings` (`int`, *optional*): - Used with 'dynamic', 'longrope' and 'llama3'. The - original max position embeddings used during pretraining. - `attention_factor` (`float`, *optional*): - Used with 'yarn' and 'longrope'. The scaling factor to be - applied on the attention computation. If unspecified, it - defaults to value recommended by the implementation, using - the `factor` field to infer the suggested value. - `beta_fast` (`float`, *optional*): - Only used with 'yarn'. Parameter to set the boundary for - extrapolation (only) in the linear ramp function. If - unspecified, it defaults to 32. - `beta_slow` (`float`, *optional*): - Only used with 'yarn'. Parameter to set the boundary for - interpolation (only) in the linear ramp function. If - unspecified, it defaults to 1. - `short_factor` (`List[float]`, *optional*): - Only used with 'longrope'. The scaling factor to be - applied to short contexts (< - `original_max_position_embeddings`). Must be a list of - numbers with the same length as the hidden size divided - by the number of attention heads divided by 2 - `long_factor` (`List[float]`, *optional*): - Only used with 'longrope'. The scaling factor to be - applied to long contexts (< - `original_max_position_embeddings`). Must be a list of - numbers with the same length as the hidden size divided - by the number of attention heads divided by 2 - `low_freq_factor` (`float`, *optional*): - Only used with 'llama3'. Scaling factor applied to low - frequency components of the RoPE - `high_freq_factor` (`float`, *optional*): - Only used with 'llama3'. Scaling factor applied to high - frequency components of the RoPE - attention_bias (`bool`, *optional*, defaults to `False`): - Whether to use a bias in the query, key, value and output - projection layers during self-attention. - attention_dropout (`float`, *optional*, defaults to 0.0): - The dropout ratio for the attention probabilities. - mlp_bias (`bool`, *optional*, defaults to `False`): - Whether to use a bias in up_proj, down_proj and gate_proj layers - in the MLP layers. - head_dim (`int`, *optional*): - The attention head dimension. If None, it will default to - hidden_size // num_heads - embedding_multiplier (`float`, *optional*, defaults to `None`): - Multiplier applied to the embedding weights. If `None`, it is - equivalent to `1.0`. - logits_scaling (`float`, *optional*, defaults to `None`): - Scaling factor for logits. If `None`, it is equivalent to `1.0`. - attention_multiplier (`float`, *optional*, defaults to `None`): - Multiplier applied to the attention weights. If `None`, it is - equivalent to `self.head_dim ** -0.5`. - residual_multiplier (`float`, *optional*, defaults to `None`): - Scaling factor for residual connections. If `None`, it is - equivalent to `1.0`. - use_post_norm (`bool`, *optional*, defaults to `True`): - Determines whether to apply Peri-Layer Normalization. Set to - False to disable this feature. - rope_parameters (`dict`, *optional*): - Dictionary containing the RoPE parameters used by vLLM's - `get_rope`. When provided, takes precedence over `rope_theta` - and `rope_scaling`. If `None`, it is derived from `rope_theta` - and `rope_scaling` automatically. - - """ - - model_type = "hyperclovax" - keys_to_ignore_at_inference = ["past_key_values"] - - def __init__( - self, - vocab_size=32000, - hidden_size=4096, - intermediate_size=11008, - num_hidden_layers=32, - num_attention_heads=32, - num_key_value_heads=None, - hidden_act="silu", - max_position_embeddings=2048, - initializer_range=0.02, - rms_norm_eps=1e-6, - use_cache=True, - pad_token_id=None, - bos_token_id=1, - eos_token_id=2, - pretraining_tp=1, - tie_word_embeddings=False, - rope_theta=10000.0, - rope_scaling=None, - attention_bias=False, - attention_dropout=0.0, - mlp_bias=False, - head_dim=None, - embedding_multiplier=None, # mup - logits_scaling=None, # mup - attention_multiplier=None, # mup - residual_multiplier=None, # mup - use_post_norm=True, # post-norm(peri-LN) - rope_parameters=None, - auto_map=None, - **kwargs, - ): - self.vocab_size = vocab_size - self.max_position_embeddings = max_position_embeddings - self.hidden_size = hidden_size - self.intermediate_size = intermediate_size - self.num_hidden_layers = num_hidden_layers - self.num_attention_heads = num_attention_heads - - # for backward compatibility - if num_key_value_heads is None: - num_key_value_heads = num_attention_heads - - self.num_key_value_heads = num_key_value_heads - self.hidden_act = hidden_act - self.initializer_range = initializer_range - self.rms_norm_eps = rms_norm_eps - self.pretraining_tp = pretraining_tp - self.use_cache = use_cache - self.rope_theta = rope_theta - self.rope_scaling = rope_scaling - self.attention_bias = attention_bias - self.attention_dropout = attention_dropout - self.mlp_bias = mlp_bias - self.head_dim = ( - head_dim - if head_dim is not None - else self.hidden_size // self.num_attention_heads - ) - # Derive rope_parameters for vLLM's get_rope() from rope_theta / - # rope_scaling, unless the caller already provided rope_parameters. - if rope_parameters is None: - if rope_scaling is not None: - # Shallow-copy to avoid mutating the caller's dict. - rope_parameters = dict(rope_scaling) - # BC: 'type' field -> 'rope_type', remove stale key. - if "type" in rope_parameters: - rope_parameters.setdefault("rope_type", rope_parameters.pop("type")) - else: - rope_parameters = {"rope_type": "default"} - if "rope_theta" not in rope_parameters: - rope_parameters["rope_theta"] = rope_theta - self.rope_parameters = rope_parameters - - # BC: keep self.rope_scaling consistent for HF serialization. - if self.rope_scaling is not None and "type" in self.rope_scaling: - self.rope_scaling["rope_type"] = self.rope_scaling["type"] - - # mup - self.embedding_multiplier = ( - embedding_multiplier if embedding_multiplier is not None else 1.0 - ) - self.logits_scaling = logits_scaling if logits_scaling is not None else 1.0 - self.attention_multiplier = ( - attention_multiplier - if attention_multiplier is not None - else self.head_dim**-0.5 - ) - self.residual_multiplier = ( - residual_multiplier if residual_multiplier is not None else 1.0 - ) - - # post-norm (Peri-LN) - self.use_post_norm = use_post_norm - - super().__init__( - pad_token_id=pad_token_id, - bos_token_id=bos_token_id, - eos_token_id=eos_token_id, - tie_word_embeddings=tie_word_embeddings, - auto_map=auto_map, - **kwargs, - ) diff --git a/vllm/transformers_utils/configs/kimi_k25.py b/vllm/transformers_utils/configs/kimi_k25.py deleted file mode 100644 index ed3d05d7cc..0000000000 --- a/vllm/transformers_utils/configs/kimi_k25.py +++ /dev/null @@ -1,131 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# SPDX-FileCopyrightText: Copyright contributors to the vLLM project -"""Kimi-K2.5 Model Configuration. - -This configuration supports video-chunk as an internal modality type. -A video-chunk is the smallest independently processable unit of video. -""" - -from transformers import DeepseekV3Config -from transformers.configuration_utils import PreTrainedConfig - - -class KimiK25VisionConfig(PreTrainedConfig): - model_type = "kimi_k25_vision" - - def __init__( - self, - # Vision Tower - patch_size: int = 14, - init_pos_emb_height: int = 64, - init_pos_emb_width: int = 64, - init_pos_emb_time: int = 4, - pos_emb_type: str = "divided_fixed", - num_attention_heads: int = 16, - num_hidden_layers: int = 27, - hidden_size: int = 1152, - intermediate_size: int = 4304, - merge_kernel_size: tuple[int, int] = (2, 2), - video_attn_type: str = "spatial_temporal", - merge_type: str = "sd2_tpool", - # MM Projector - mm_projector_type: str = "patchmerger", - mm_hidden_size: int | None = None, - projector_hidden_act: str = "gelu", - projector_ln_eps: float = 1e-5, - **kwargs, - ): - super().__init__(**kwargs) - # Vision Tower - self.patch_size = patch_size - self.init_pos_emb_height = init_pos_emb_height - self.init_pos_emb_width = init_pos_emb_width - self.init_pos_emb_time = init_pos_emb_time - self.pos_emb_type = pos_emb_type - self.num_attention_heads = num_attention_heads - self.num_hidden_layers = num_hidden_layers - self.hidden_size = hidden_size - self.intermediate_size = intermediate_size - self.merge_kernel_size = merge_kernel_size - self.video_attn_type = video_attn_type - self.merge_type = merge_type - # MM Projector - self.mm_projector_type = mm_projector_type - if mm_hidden_size is not None: - self.mm_hidden_size = mm_hidden_size - else: - self.mm_hidden_size = hidden_size - self.projector_hidden_act = projector_hidden_act - self.projector_ln_eps = projector_ln_eps - - -class KimiK25Config(PreTrainedConfig): - """Kimi-K2.5 model configuration. - - Kimi-K2.5 extends Kimi-K2 with vision support using video-chunks. - A video-chunk consists of multiple consecutive frames - that are processed together with temporal pooling. - - Args: - vision_config: Configuration for the vision tower and projector. - text_config: Configuration for the text model (DeepseekV3). - ignore_index: The ignore index for the loss function. - media_placeholder_token_id: The token ID for media placeholders. - pad_token_id: The token ID for padding. - - """ - - model_type = "kimi_k25" - - def __init__( - self, - vision_config: dict | KimiK25VisionConfig | None = None, - text_config: dict | DeepseekV3Config | None = None, - ignore_index: int = -100, - media_placeholder_token_id: int = 163605, - pad_token_id: int = 0, - use_unified_vision_chunk: bool = False, - video_placeholder: str = "<|kimi_k25_video_placeholder|>", - **kwargs, - ): - # Vision config - if vision_config is None: - self.vision_config = KimiK25VisionConfig() - elif isinstance(vision_config, dict): - self.vision_config = KimiK25VisionConfig(**vision_config) - else: - self.vision_config = vision_config - - # Text config - if text_config is None: - self.text_config = DeepseekV3Config() - elif isinstance(text_config, dict): - self.text_config = DeepseekV3Config(**text_config) - else: - self.text_config = text_config - - # Set mm_hidden_size to text hidden size if not explicitly set - if self.vision_config.mm_hidden_size == self.vision_config.hidden_size: - self.vision_config.mm_hidden_size = self.text_config.hidden_size - - # Other config - self.ignore_index = ignore_index - self.media_placeholder_token_id = media_placeholder_token_id - self.use_unified_vision_chunk = use_unified_vision_chunk - self.video_placeholder = video_placeholder - - # Propagate quantization config from text model - if getattr(self.text_config, "quantization_config", None) is not None: - self.quantization_config = self.text_config.quantization_config - - super().__init__(pad_token_id=pad_token_id, **kwargs) - - @property - def hidden_size(self) -> int: - """Get hidden size from text config for compatibility.""" - return self.text_config.hidden_size - - @property - def vocab_size(self) -> int: - """Get vocab size from text config for compatibility.""" - return self.text_config.vocab_size diff --git a/vllm/transformers_utils/configs/lfm2_moe.py b/vllm/transformers_utils/configs/lfm2_moe.py deleted file mode 100644 index 903ae9d547..0000000000 --- a/vllm/transformers_utils/configs/lfm2_moe.py +++ /dev/null @@ -1,164 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# SPDX-FileCopyrightText: Copyright contributors to the vLLM project -from typing import Any - -from transformers.configuration_utils import PreTrainedConfig - - -class Lfm2MoeConfig(PreTrainedConfig): - r"""This is the configuration class to store the configuration of a [`Lfm2MoeModel`]. It is used to instantiate a LFM2 Moe - model according to the specified arguments, defining the model architecture. Instantiating a configuration with the - defaults will yield a similar configuration to that of the LFM2-8B-A1B model. - e.g. [LiquidAI/LFM2-8B-A1B](https://huggingface.co/LiquidAI/LFM2-8B-A1B) - - Configuration objects inherit from [`PreTrainedConfig`] and can be used to control the model outputs. Read the - documentation from [`PreTrainedConfig`] for more information. - - - Args: - vocab_size (`int`, *optional*, defaults to 65536): - Vocabulary size of the LLaMA model. Defines the number of different tokens that can be represented by the - `inputs_ids` passed when calling [`Lfm2Model`] - hidden_size (`int`, *optional*, defaults to 2048): - Dimension of the hidden representations. - intermediate_size (`int`, *optional*, defaults to 7168): - Dimension of the MLP representations. - moe_intermediate_size (`int`, *optional*, defaults to 1792): - Intermediate size of the routed expert. - num_hidden_layers (`int`, *optional*, defaults to 32): - Number of hidden layers in the Transformer decoder. - pad_token_id (`int`, *optional*, defaults to 0): - Padding token id. - bos_token_id (`int`, *optional*, defaults to 1): - Beginning of stream token id. - eos_token_id (`int`, *optional*, defaults to 2): - End of stream token id. - tie_word_embeddings (`bool`, *optional*, defaults to `True`): - Whether to tie weight embeddings - rope_parameters (`dict`, *optional*): - The parameters of the RoPE embeddings. - max_position_embeddings (`int`, *optional*, defaults to 128000): - The maximum sequence length that this model might ever be used with. - use_cache (`bool`, *optional*, defaults to `True`): - Whether or not the model should return the last key/values attentions (not used by all models). Only - relevant if `config.is_decoder=True`. - norm_eps (`float`, *optional*, defaults to 1e-05): - The epsilon used by the rms normalization layers. - num_attention_heads (`int`, *optional*, defaults to 32): - Number of attention heads for each attention layer in the Transformer decoder. - num_key_value_heads (`int`, *optional*, defaults to 8): - This is the number of key_value heads that should be used to implement Grouped Query Attention. If - `num_key_value_heads=num_attention_heads`, the model will use Multi Head Attention (MHA), if - `num_key_value_heads=1` the model will use Multi Query Attention (MQA) otherwise GQA is used. When - converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed - by meanpooling all the original heads within that group. For more details, check out [this - paper](https://huggingface.co/papers/2305.13245). If it is not specified, will default to - `num_attention_heads`. - conv_bias (`bool`, *optional*, defaults to `False`): - Whether to use bias in the conv layers. - conv_L_cache (`int`, *optional*, defaults to 3): - L_cache dim in the conv layers. - num_dense_layers (`int`, *optional*, defaults to 2): - Number of dense Lfm2MoeMLP layers in shallow layers(embed->dense->dense->...->dense->moe->moe...->lm_head). - num_experts_per_tok (`int`, *optional*, defaults to 4): - Number of selected experts. - num_experts (`int`, *optional*, defaults to 32): - Number of routed experts. - use_expert_bias (`bool`, *optional*, defaults to `True`): - Whether to use the expert bias on the routing weights. - routed_scaling_factor (`float`, *optional*, defaults to 1.0): - Scaling factor for routed experts in MoE models. - norm_topk_prob (`bool`, *optional*, defaults to `True`): - Whether to normalize the topk probabilities. - layer_types (`Optional`, *optional*): - Type of each layers. - - ```python - >>> from transformers import Lfm2MoeModel, Lfm2MoeConfig - - >>> # Initializing a LFM2 Moe model - >>> configuration = Lfm2MoeConfig() - - >>> # Initializing a model from the LFM2-8B-A1B style configuration - >>> model = Lfm2MoeModel(configuration) - - >>> # Accessing the model configuration - >>> configuration = model.config - ``` - - """ # noqa: E501 - - model_type = "lfm2_moe" - keys_to_ignore_at_inference = ["past_key_values"] - - def __init__( - self, - vocab_size: int = 65536, - hidden_size: int = 2048, - intermediate_size: int = 7168, - moe_intermediate_size: int = 1792, - num_hidden_layers: int = 32, - pad_token_id: int = 0, - bos_token_id: int = 1, - eos_token_id: int = 2, - tie_word_embeddings: bool = True, - rope_parameters: dict[str, Any] | None = None, - max_position_embeddings: int = 128_000, - use_cache: bool = True, - norm_eps: float = 0.00001, - num_attention_heads: int = 32, - num_key_value_heads: int = 8, - conv_bias: bool = False, - conv_L_cache: int = 3, - num_dense_layers: int = 2, - num_experts_per_tok: int = 4, - num_experts: int = 32, - use_expert_bias: bool = True, - routed_scaling_factor: float = 1.0, - norm_topk_prob: bool = True, - layer_types: list[str] | None = None, - **kwargs, - ): - self.vocab_size = vocab_size - self.hidden_size = hidden_size - self.intermediate_size = intermediate_size - self.num_hidden_layers = num_hidden_layers - rope_theta = kwargs.pop("rope_theta", 1000000.0) - if rope_parameters is None: - rope_parameters = {"rope_type": "default", "rope_theta": rope_theta} - self.rope_parameters = rope_parameters - self.max_position_embeddings = max_position_embeddings - self.use_cache = use_cache - self.norm_eps = norm_eps - - # attn operator config - self.num_attention_heads = num_attention_heads - self.num_key_value_heads = num_key_value_heads - - # custom operator config - self.conv_bias = conv_bias - self.conv_L_cache = conv_L_cache - - # moe config - self.num_dense_layers = num_dense_layers - self.moe_intermediate_size = moe_intermediate_size - self.num_experts_per_tok = num_experts_per_tok - self.num_experts = num_experts - self.use_expert_bias = use_expert_bias - self.routed_scaling_factor = routed_scaling_factor - self.norm_topk_prob = norm_topk_prob - self.layer_types = layer_types - - tie_word_embeddings = kwargs.get( - "tie_embedding", tie_word_embeddings - ) # to fit original config keys - super().__init__( - pad_token_id=pad_token_id, - bos_token_id=bos_token_id, - eos_token_id=eos_token_id, - tie_word_embeddings=tie_word_embeddings, - **kwargs, - ) - - -__all__ = ["Lfm2MoeConfig"] diff --git a/vllm/transformers_utils/configs/nemotron.py b/vllm/transformers_utils/configs/nemotron.py deleted file mode 100644 index 135a0e1596..0000000000 --- a/vllm/transformers_utils/configs/nemotron.py +++ /dev/null @@ -1,219 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# SPDX-FileCopyrightText: Copyright contributors to the vLLM project - -# Copyright 2024 HuggingFace Inc. team. All rights reserved. -# Copyright (c) 2024, NVIDIA CORPORATION. All rights reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -"""Nemotron model configuration.""" - -from transformers import PreTrainedConfig -from transformers.utils import logging - -logger = logging.get_logger(__name__) - - -class NemotronConfig(PreTrainedConfig): - r"""This is the configuration class to store the configuration of a - [`NemotronModel`]. It is used to instantiate a Nemotron model - according to the specified arguments, defining the model architecture. - Instantiating a configuration with the defaults will yield a similar - configuration to that of the Nemotron-8B. - - Configuration objects inherit from [`PreTrainedConfig`] and can be - used to control the model outputs. Read the documentation from - [`PreTrainedConfig`] for more information. - - - Args: - vocab_size (`int`, *optional*, defaults to 256000): - Vocabulary size of the Nemotron model. Defines the number of - different tokens that can be represented by the - `inputs_ids` passed when calling [`NemotronModel`] - hidden_size (`int`, *optional*, defaults to 6144): - Dimension of the hidden representations. - intermediate_size (`int`, *optional*, defaults to 24576): - Dimension of the MLP representations. - num_hidden_layers (`int`, *optional*, defaults to 32): - Number of hidden layers in the Transformer decoder. - num_attention_heads (`int`, *optional*, defaults to 48): - Number of attention heads for each attention layer in the - Transformer decoder. - head_dim (`int`, *optional*): - Projection weights dimension in multi-head attention. Set to - hidden_size // num_attention_heads if None - num_key_value_heads (`int`, *optional*): - This is the number of key_value heads that should be used to - implement Grouped Query Attention. If - `num_key_value_heads=num_attention_heads`, the model will use - Multi Head Attention (MHA), if - `num_key_value_heads=1 the model will use Multi Query Attention - (MQA) otherwise GQA is used. When converting a multi-head - checkpoint to a GQA checkpoint, each group key and value - head should be constructed by meanpooling all the original - heads within that group. For more details checkout - [this paper](https://arxiv.org/pdf/2305.13245.pdf). If it - is not specified, will default to `num_attention_heads`. - hidden_act (`str` or `function`, *optional*, defaults to `"relu2"`): - The non-linear activation function (function or string) in the - decoder. - max_position_embeddings (`int`, *optional*, defaults to 4096): - The maximum sequence length that this model might ever be used - with. - initializer_range (`float`, *optional*, defaults to 0.0134): - The standard deviation of the truncated_normal_initializer for - initializing all weight matrices. - norm_eps (`float`, *optional*, defaults to 1e-05): - The epsilon used by the normalization layers. - use_cache (`bool`, *optional*, defaults to `True`): - Whether or not the model should return the last key/values - attentions (not used by all models). Only relevant if - `config.is_decoder=True`. - pad_token_id (`int`, *optional*): - Padding token id. - bos_token_id (`int`, *optional*, defaults to 2): - Beginning of stream token id. - eos_token_id (`int`, *optional*, defaults to 3): - End of stream token id. - tie_word_embeddings (`bool`, *optional*, defaults to `False`): - Whether to tie weight embeddings - rope_parameters (`dict`, *optional*): - The parameters of the RoPE embeddings. Expected contents: - `rope_theta` (`float`): The base period of the RoPE embeddings. - `rope_type` (`str`): - The sub-variant of RoPE to use. Can be one of ['default', 'linear', - 'dynamic', 'yarn', 'longrope', 'llama3'], with 'default' being the - original RoPE implementation. - `partial_rotary_factor` (`float`, *optional*, defaults to 0.5): - Percentage of the query and keys which will have rotary embedding. - attention_bias (`bool`, *optional*, defaults to `False`): - Whether to use a bias in the query, key, value and output - projection layers during self-attention. - attention_dropout (`float`, *optional*, defaults to 0.0): - The dropout ratio for the attention probabilities. - mlp_bias (`bool`, *optional*, defaults to `False`): - Whether to use a bias in up_proj and down_proj layers in the MLP - layers. - - ```python - >>> from transformers import NemotronModel, NemotronConfig - >>> # Initializing a Nemotron nemotron-15b style configuration - >>> configuration = NemotronConfig() - >>> # Initializing a model from the nemotron-15b style configuration - >>> model = NemotronModel(configuration) - >>> # Accessing the model configuration - >>> configuration = model.config - ``` - - """ - - model_type = "nemotron" - keys_to_ignore_at_inference = ["past_key_values"] - - def __init__( - self, - vocab_size=256000, - hidden_size=6144, - intermediate_size=24576, - num_hidden_layers=32, - num_attention_heads=48, - head_dim=None, - num_key_value_heads=None, - hidden_act="relu2", - max_position_embeddings=4096, - initializer_range=0.0134, - norm_eps=1e-5, - use_cache=True, - pad_token_id=None, - bos_token_id=2, - eos_token_id=3, - tie_word_embeddings=False, - rope_parameters=None, - attention_bias=False, - attention_dropout=0.0, - mlp_bias=False, - **kwargs, - ): - self.vocab_size = vocab_size - self.max_position_embeddings = max_position_embeddings - self.hidden_size = hidden_size - self.intermediate_size = intermediate_size - self.num_hidden_layers = num_hidden_layers - self.num_attention_heads = num_attention_heads - head_dim = head_dim or kwargs.get("kv_channels") - self.head_dim = ( - head_dim if head_dim is not None else (hidden_size // num_attention_heads) - ) - - # for backward compatibility - if num_key_value_heads is None: - num_key_value_heads = num_attention_heads - - self.num_key_value_heads = num_key_value_heads - self.hidden_act = hidden_act - self.initializer_range = initializer_range - self.norm_eps = norm_eps - self.use_cache = use_cache - # Try to set `rope_scaling` if available, otherwise use `rope_parameters` - rope_scaling = kwargs.pop("rope_scaling", None) - rope_parameters = rope_scaling or rope_parameters or {"rope_type": "default"} - rope_theta = kwargs.pop("rope_theta", 10000.0) - if "rope_theta" not in rope_parameters: - rope_parameters["rope_theta"] = rope_theta - # for backward compatibility - partial_rotary_factor = ( - kwargs.get("rope_percent") - or kwargs.get("rope_percentage") - or kwargs.get("partial_rotary_factor") - or 0.5 - ) - if "partial_rotary_factor" not in rope_parameters: - rope_parameters["partial_rotary_factor"] = partial_rotary_factor - self.rope_parameters = rope_parameters - self._rope_parameters_validation() - self.attention_bias = attention_bias - self.attention_dropout = attention_dropout - self.mlp_bias = mlp_bias - - super().__init__( - pad_token_id=pad_token_id, - bos_token_id=bos_token_id, - eos_token_id=eos_token_id, - tie_word_embeddings=tie_word_embeddings, - **kwargs, - ) - - def _rope_parameters_validation(self): - """Validate the `rope_parameters` configuration.""" - if self.rope_parameters is None: - return - - rope_type: str | None = self.rope_parameters.get("rope_type", None) - factor: float | None = self.rope_parameters.get("factor", None) - - if rope_type not in {"default", "linear", "dynamic"}: - raise ValueError( - "`rope_type` must be one of ['default', 'linear', 'dynamic'], " - f"got {rope_type}" - ) - if rope_type != "default": - if factor is None: - raise ValueError( - "If `rope_type` is not 'default', `rope_parameters` " - "must include a `factor` field. Got `None`." - ) - if not isinstance(factor, float) or factor <= 1.0: - raise ValueError( - "`rope_parameters`'s factor field must be a float > 1, got " - f"{factor}" - ) diff --git a/vllm/transformers_utils/configs/nemotron_h.py b/vllm/transformers_utils/configs/nemotron_h.py deleted file mode 100644 index c25a5d54e3..0000000000 --- a/vllm/transformers_utils/configs/nemotron_h.py +++ /dev/null @@ -1,288 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# SPDX-FileCopyrightText: Copyright contributors to the vLLM project - -# Copyright 2024 HuggingFace Inc. team. All rights reserved. -# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -"""NemotronH model configuration.""" - -import regex as re -from transformers.configuration_utils import PreTrainedConfig -from transformers.utils import logging - -logger = logging.get_logger(__name__) - - -class NemotronHConfig(PreTrainedConfig): - r"""This is the configuration class to store the configuration of a - [`NemotronHModel`]. It is used to instantiate a NemotronH model according - to the specified arguments, defining the model architecture. Instantiating - a configuration with the defaults will yield a similar configuration to - that of the NemotronH-v0.1 model. - - Args: - vocab_size (`int`, *optional*, defaults to 131072): - Vocabulary size of the NemotronH model. Defines the number of - different tokens that can be represented by the `inputs_ids` - passed when calling [`NemotronHModel`] - tie_word_embeddings (`bool`, *optional*, defaults to `False`): - Whether the model's input and output word embeddings should be - tied. Note that this is only relevant if the model has an output - word embedding layer. - hidden_size (`int`, *optional*, defaults to 4096): - Dimension of the hidden representations. - intermediate_size (`int`, *optional*, defaults to 21504): - Dimension of the MLP representations. - num_hidden_layers (`int`, *optional*, defaults to 52): - Number of hidden layers in the Transformer encoder. - hybrid_override_pattern (`str`, *optional*, defaults to - `"M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M-"`): - The pattern of the hybrid model. The pattern is a string of - characters where each character represents - M: Mamba2, *: Attention, -: MLP - mtp_hybrid_override_pattern (`str`, *optional*, defaults to `"*E"`): - The pattern of the MTP layers. - num_attention_heads (`int`, *optional*, defaults to 32): - Number of attention heads for each attention layer in the - Transformer encoder. - attention_head_dim (`int`, *optional*, defaults to 128): - Dimension of each attention head. - num_key_value_heads (`int`, *optional*, defaults to 8): - This is the number of key_value heads that should be used to - implement Grouped Query Attention. If - `num_key_value_heads=num_attention_heads`, the model will use - Multi Head Attention (MHA), if `num_key_value_heads=1` the model - will use Multi Query Attention (MQA) otherwise GQA is used. - mlp_hidden_act (`str`, *optional*, defaults to "relu2"): - The non-linear activation function in the MLP layers. - attention_bias (`bool`, *optional*, defaults to `False`): - Whether to use bias in attention layers. - mlp_bias (`bool`, *optional*, defaults to `False`): - Whether to use bias in MLP layers. - use_bias (`bool`, *optional*, defaults to `False`): - Whether to use bias in the model. - initializer_range (`float`, *optional*, defaults to 0.02): - The standard deviation of the truncated_normal_initializer for - initializing all weight matrices. - layer_norm_epsilon (`float`, *optional*, defaults to 1e-5): - The epsilon used by the layer normalization layers. - residual_in_fp32 (`bool`, *optional*, defaults to `False`): - Whether or not residuals should be in `float32`. If set to `False` - residuals will keep the same `dtype` as the rest of the model. - use_cache (`bool`, *optional*, defaults to `True`): - Whether or not the model should return the last key/values - attentions (not used by all models). Only relevant if - `config.is_decoder=True`. - num_logits_to_keep (`int` or `None`, *optional*, defaults to 1): - Number of prompt logits to calculate during generation. If `None`, - all logits will be calculated. If an integer value, only last - `num_logits_to_keep` logits will be calculated. - pad_token_id (`int`, *optional*, defaults to 0): - The id of the padding token. - bos_token_id (`int`, *optional*, defaults to 1): - The id of the "beginning-of-sequence" token. - eos_token_id (`int`, *optional*, defaults to 2): - The id of the "end-of-sequence" token. - sliding_window (`int`, *optional*, defaults to None): - Sliding window attention window size. - max_position_embeddings (`int`, *optional*, defaults to 4096): - The maximum sequence length that this model might ever be used - with. - attention_dropout (`float`, *optional*, defaults to 0.0): - The dropout ratio for the attention probabilities. - hidden_dropout (`float`, *optional*, defaults to 0.0): - The dropout ratio for the hidden states. - use_mamba_kernels (`bool`, *optional*, defaults to `True`): - Flag indicating whether or not to use the fast mamba kernels. - These are available only if `mamba-ssm` and `causal-conv1d` - are installed, and the mamba modules are running on a CUDA device. - ssm_state_size (`int`, *optional*, defaults to 128): - The dimension of the mamba state space latents. - mamba_num_heads (`int`, *optional*, defaults to 128): - Number of heads in Mamba layers. - mamba_n_groups (`int`, *optional*, defaults to 8): - Number of groups in Mamba layers. - mamba_head_dim (`int`, *optional*, defaults to 64): - Dimension of each Mamba head. - mamba_d_conv (`int`, *optional*, defaults to 4): - The size of the mamba convolution kernel. - mamba_expand (`int`, *optional*, defaults to 2): - Expanding factor used to determine the mamba intermediate size. - mamba_hidden_act (`str`, *optional*, defaults to "silu"): - The non-linear activation function in the Mamba layers. - mamba_dt_min (`float`, *optional*, defaults to 0.001): - Minimum value for the time step in Mamba. - mamba_dt_max (`float`, *optional*, defaults to 0.1): - Maximum value for the time step in Mamba. - mamba_dt_limit (`tuple`, *optional*, defaults to (0.0, float("inf"))): - Limits for the time step in Mamba. - mamba_dt_init_floor (`float`, *optional*, defaults to 1e-4): - Floor value for time step initialization in Mamba. - mamba_conv_bias (`bool`, *optional*, defaults to `True`): - Whether to use bias in the convolution layer of the mamba mixer - block. - mamba_proj_bias (`bool`, *optional*, defaults to `False`): - Whether to use bias in the input and output projections of the - mamba mixer block. - mamba_chunk_size (`int`, *optional*, defaults to 256): - Size of chunks for Mamba processing. - rescale_prenorm_residual (`bool`, *optional*, defaults to `True`): - Whether to rescale the pre-normalization residual connections. - - """ - - model_type = "nemotron_h" - keys_to_ignore_at_inference = ["past_key_values"] - - def __init__( - self, - vocab_size=131072, - tie_word_embeddings=False, - hidden_size=4096, - intermediate_size=21504, - num_hidden_layers=52, - hybrid_override_pattern="M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M-", - mtp_hybrid_override_pattern="*E", - num_attention_heads=32, - head_dim=128, - num_key_value_heads=8, # nemo: num_query_groups - mlp_hidden_act="relu2", - attention_bias=False, - mlp_bias=False, - use_bias=False, - initializer_range=0.02, # nemo: init_method_std - layer_norm_epsilon=1e-5, # nemo: layernorm_epsilon - residual_in_fp32=False, # Megatron Core default value - use_cache=True, - num_logits_to_keep=1, - pad_token_id=0, - bos_token_id=1, - eos_token_id=2, - sliding_window=None, - max_position_embeddings=4096, - attention_dropout=0.0, - hidden_dropout=0.0, # * ADDED - use_mamba_kernels=True, - ssm_state_size=128, # mamba_state_size - mamba_num_heads=128, - mamba_n_groups=8, # nemo: mamba_ssm_ngroups = num_heads - mamba_head_dim=64, - mamba_d_conv=4, - mamba_expand=2, - mamba_hidden_act="silu", - mamba_dt_min=0.001, - mamba_dt_max=0.1, - mamba_dt_limit=(0.0, float("inf")), - mamba_dt_init_floor=1e-4, - mamba_conv_bias=True, - mamba_proj_bias=False, - mamba_chunk_size=256, - rescale_prenorm_residual=True, - n_routed_experts=8, - n_shared_experts=1, - moe_intermediate_size=7688, - moe_shared_expert_intermediate_size=7688, - moe_latent_size=None, - num_experts_per_tok=2, - routed_scaling_factor=1.0, - n_group=1, - topk_group=1, - norm_topk_prob=True, - **kwargs, - ): - self.vocab_size = vocab_size - self.tie_word_embeddings = tie_word_embeddings - self.hidden_size = hidden_size - self.intermediate_size = intermediate_size - self.num_hidden_layers = num_hidden_layers - self.hybrid_override_pattern = hybrid_override_pattern - self.mtp_hybrid_override_pattern = mtp_hybrid_override_pattern - self.num_attention_heads = num_attention_heads - self.head_dim = head_dim - self.sliding_window = sliding_window - self.max_position_embeddings = max_position_embeddings - self.attention_dropout = attention_dropout - self.hidden_dropout = hidden_dropout - - # Validate hybrid_override_pattern - # M: Mamba2, *: Attention, -: MLP - assert len(self.hybrid_override_pattern) == self.num_hidden_layers, ( - "hybrid_override_pattern must have same length as num_hidden_layers" - ) - assert re.match(r"^[*-ME]+$", self.hybrid_override_pattern), ( - "hybrid_override_pattern must only contain characters 'M', '*', '-', or 'E'" - ) - # for backward compatibility - if num_key_value_heads is None: - num_key_value_heads = num_attention_heads - - self.num_key_value_heads = num_key_value_heads - self.mlp_hidden_act = mlp_hidden_act - self.attention_bias = attention_bias - self.mlp_bias = mlp_bias - self.use_bias = use_bias - self.initializer_range = initializer_range - self.layer_norm_epsilon = layer_norm_epsilon - self.residual_in_fp32 = residual_in_fp32 - - self.use_cache = use_cache - self.num_logits_to_keep = num_logits_to_keep - - self.use_mamba_kernels = use_mamba_kernels - self.n_groups = mamba_n_groups - self.mamba_head_dim = mamba_head_dim - self.ssm_state_size = ssm_state_size - self.mamba_num_heads = mamba_num_heads - self.conv_kernel = mamba_d_conv - self.expand = mamba_expand - self.mamba_hidden_act = mamba_hidden_act - self.time_step_min = mamba_dt_min - self.time_step_max = mamba_dt_max - self.time_step_limit = mamba_dt_limit - self.time_step_floor = mamba_dt_init_floor - self.use_conv_bias = mamba_conv_bias - self.mamba_proj_bias = mamba_proj_bias - self.chunk_size = mamba_chunk_size - self.rescale_prenorm_residual = rescale_prenorm_residual - self.n_routed_experts = n_routed_experts - self.n_shared_experts = n_shared_experts - self.moe_intermediate_size = moe_intermediate_size - self.moe_shared_expert_intermediate_size = moe_shared_expert_intermediate_size # noqa: E501 - self.moe_latent_size = moe_latent_size - self.num_experts_per_tok = num_experts_per_tok - self.routed_scaling_factor = routed_scaling_factor - self.n_group = n_group - self.topk_group = topk_group - self.norm_topk_prob = norm_topk_prob - - super().__init__( - pad_token_id=pad_token_id, - bos_token_id=bos_token_id, - eos_token_id=eos_token_id, - tie_word_embeddings=tie_word_embeddings, - **kwargs, - ) - - @property - def layers_block_type(self): - return [ - "mamba" - if self.hybrid_override_pattern[i] == "M" - else "attention" - if self.hybrid_override_pattern[i] == "*" - else "mlp" - if self.hybrid_override_pattern[i] == "-" - else "moe" - for i in range(self.num_hidden_layers) - ] diff --git a/vllm/transformers_utils/configs/olmo_hybrid.py b/vllm/transformers_utils/configs/olmo_hybrid.py deleted file mode 100644 index 62d3c4e0d5..0000000000 --- a/vllm/transformers_utils/configs/olmo_hybrid.py +++ /dev/null @@ -1,285 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# SPDX-FileCopyrightText: Copyright contributors to the vLLM project - - -from transformers.configuration_utils import PreTrainedConfig - - -class OlmoHybridConfig(PreTrainedConfig): - r"""Configuration class for [`OlmoHybridModel`]. It is used to - instantiate an OLMo Hybrid model according to the specified - arguments, defining the model architecture. Instantiating a - configuration with the defaults will yield a similar - configuration to that of the - [allenai/Olmo-Hybrid-7B](https://huggingface.co/allenai/Olmo-Hybrid-7B) - model. - - Configuration objects inherit from [`PreTrainedConfig`] and - can be used to control the model outputs. Read the - documentation from [`PreTrainedConfig`] for more information. - - Args: - vocab_size (`int`, *optional*, defaults to 100352): - Vocabulary size of the OlmoHybrid model. Defines - the number of different tokens that can be - represented by the `inputs_ids` passed when - calling [`OlmoHybridModel`]. - hidden_size (`int`, *optional*, defaults to 3840): - Dimension of the hidden representations. - intermediate_size (`int`, *optional*, - defaults to 11008): - Dimension of the MLP representations. - num_hidden_layers (`int`, *optional*, - defaults to 32): - Number of hidden layers in the Transformer - decoder. - num_attention_heads (`int`, *optional*, - defaults to 30): - Number of attention heads for each attention - layer in the Transformer decoder. - num_key_value_heads (`int`, *optional*): - This is the number of key_value heads that - should be used to implement Grouped Query - Attention. If - `num_key_value_heads=num_attention_heads`, - the model will use Multi Head Attention (MHA), - if `num_key_value_heads=1` the model will use - Multi Query Attention (MQA) otherwise GQA is - used. When converting a multi-head checkpoint - to a GQA checkpoint, each group key and value - head should be constructed by meanpooling all - the original heads within that group. For more - details, check out - [this paper](https://huggingface.co/papers/2305.13245). - If it is not specified, will default to - `num_attention_heads`. - hidden_act (`str` or `function`, *optional*, - defaults to `"silu"`): - The non-linear activation function (function - or string) in the decoder. - max_position_embeddings (`int`, *optional*, - defaults to 65536): - The maximum sequence length that this model - might ever be used with. - initializer_range (`float`, *optional*, - defaults to 0.02): - The standard deviation of the - truncated_normal_initializer for initializing - all weight matrices. - use_cache (`bool`, *optional*, defaults to `True`): - Whether or not the model should return the last - key/values attentions (not used by all models). - Only relevant if `config.is_decoder=True`. - pad_token_id (`int`, *optional*, - defaults to 100277): - Padding token id. - bos_token_id (`int`, *optional*): - Beginning of stream token id. - eos_token_id (`int`, *optional*, - defaults to 100257): - End of stream token id. - tie_word_embeddings (`bool`, *optional*, - defaults to `False`): - Whether to tie weight embeddings. - rope_parameters (`RopeParameters`, *optional*): - Dictionary containing the configuration - parameters for the RoPE embeddings. Can be - `None` to disable RoPE. - attention_bias (`bool`, *optional*, - defaults to `False`): - Whether to use a bias in the query, key, value - and output projection layers during - self-attention. - attention_dropout (`float`, *optional*, - defaults to 0.0): - The dropout ratio for the attention - probabilities. - rms_norm_eps (`float`, *optional*, - defaults to 1e-06): - The epsilon used by the rms normalization - layers. - layer_types (`list`, *optional*): - Attention pattern for each layer. Can contain - `"full_attention"` or `"linear_attention"`. - Defaults to linear attention for most layers - with full attention for every 4th layer. - linear_num_key_heads (`int`, *optional*): - Number of key heads for the linear attention - layers. Defaults to `num_attention_heads`. - linear_num_value_heads (`int`, *optional*): - Number of value heads for the linear attention - layers. Defaults to `num_attention_heads`. - linear_key_head_dim (`int`, *optional*): - Dimension of each key head in linear attention - layers. Defaults to - `0.75 * hidden_size / linear_num_key_heads`. - linear_value_head_dim (`int`, *optional*): - Dimension of each value head in linear - attention layers. Defaults to - `2 * linear_key_head_dim`. - linear_a_log_min (`float`, *optional*, - defaults to 0.0): - Minimum value for uniform initialization of - A_log in GatedDeltaNet layers. - linear_a_log_max (`float`, *optional*, - defaults to 16.0): - Maximum value for uniform initialization of - A_log in GatedDeltaNet layers. - linear_dt_min (`float`, *optional*, - defaults to 0.001): - Minimum value for dt initialization in - GatedDeltaNet layers. - linear_dt_max (`float`, *optional*, - defaults to 0.1): - Maximum value for dt initialization in - GatedDeltaNet layers. - linear_dt_init_floor (`float`, *optional*, - defaults to 0.0001): - Floor value for clamping dt during - initialization in GatedDeltaNet layers. - linear_conv_kernel_dim (`int`, *optional*, - defaults to 4): - Kernel size for the short convolution applied - to queries, keys, and values in linear - attention layers. - linear_allow_neg_eigval (`bool`, *optional*, - defaults to `True`): - Whether to allow negative eigenvalues in the - GatedDeltaNet recurrence. When `True`, the - beta parameter is scaled by 2.0 to allow - values in range [0, 2] instead of [0, 1]. - ```python - >>> from transformers import ( - ... OlmoHybridModel, - ... OlmoHybridConfig, - ... ) - - >>> configuration = OlmoHybridConfig() - >>> model = OlmoHybridModel(configuration) - >>> configuration = model.config - ``` - - """ - - model_type = "olmo_hybrid" - keys_to_ignore_at_inference = ["past_key_values"] - base_model_tp_plan = { - "layers.*.self_attn.q_proj": "colwise_gather_output", - "layers.*.self_attn.k_proj": "colwise_gather_output", - "layers.*.self_attn.v_proj": "colwise_gather_output", - "layers.*.self_attn.o_proj": "rowwise_split_input", - "layers.*.mlp.gate_proj": "colwise", - "layers.*.mlp.up_proj": "colwise", - "layers.*.mlp.down_proj": "rowwise", - } - base_model_pp_plan = { - "embed_tokens": (["input_ids"], ["inputs_embeds"]), - "layers": (["hidden_states", "attention_mask"], ["hidden_states"]), - "norm": (["hidden_states"], ["hidden_states"]), - } - - def __init__( - self, - vocab_size: int | None = 100352, - hidden_size: int | None = 3840, - intermediate_size: int | None = 11008, - num_hidden_layers: int | None = 32, - num_attention_heads: int | None = 30, - num_key_value_heads: int | None = None, - hidden_act: str | None = "silu", - max_position_embeddings: int | None = 65536, - initializer_range: float | None = 0.02, - use_cache: bool | None = True, - pad_token_id: int | None = 100277, - bos_token_id: int | None = None, - eos_token_id: int | None = 100257, - tie_word_embeddings: bool | None = False, - rope_parameters=None, - attention_bias: bool | None = False, - attention_dropout: float | None = 0.0, - rms_norm_eps: float | None = 1e-06, - layer_types: list[str] | None = None, - linear_num_key_heads: int | None = None, - linear_num_value_heads: int | None = None, - linear_key_head_dim: int | None = None, - linear_value_head_dim: int | None = None, - linear_a_log_min: float = 0.0, - linear_a_log_max: float = 16.0, - linear_dt_min: float = 0.001, - linear_dt_max: float = 0.1, - linear_dt_init_floor: float = 1e-4, - linear_conv_kernel_dim: int = 4, - linear_allow_neg_eigval: bool = True, - **kwargs, - ): - super().__init__(**kwargs) - - assert num_hidden_layers is not None - assert hidden_size is not None - assert num_attention_heads is not None - - if layer_types is None: - # Default: linear attention for most layers, full attention every 4th layer - layer_types = ["linear_attention"] * int(num_hidden_layers) - for i in range(int(num_hidden_layers)): - if i % 4 == 3: - layer_types[i] = "full_attention" - # Ensure at least one full attention layer for small num_hidden_layers - if "full_attention" not in layer_types: - layer_types[-1] = "full_attention" - - self.layer_types = layer_types - self.validate_layer_type() - if "linear_attention" not in layer_types: - raise ValueError( - "OLMoHybrid expects at least one 'linear_attention' layer." - ) - if all(t == "linear_attention" for t in layer_types): - raise ValueError("OLMoHybrid expects at least one attention layer.") - - self.layer_types = layer_types - - if linear_num_key_heads is None: - linear_num_key_heads = num_attention_heads - if linear_num_value_heads is None: - linear_num_value_heads = num_attention_heads - if linear_key_head_dim is None: - linear_key_head_dim = int(0.75 * hidden_size / linear_num_key_heads) - if linear_value_head_dim is None: - linear_value_head_dim = 2 * linear_key_head_dim - - self.linear_num_key_heads = linear_num_key_heads - self.linear_num_value_heads = linear_num_value_heads - self.linear_key_head_dim = linear_key_head_dim - self.linear_value_head_dim = linear_value_head_dim - self.linear_a_log_min = linear_a_log_min - self.linear_a_log_max = linear_a_log_max - self.linear_dt_min = linear_dt_min - self.linear_dt_max = linear_dt_max - self.linear_dt_init_floor = linear_dt_init_floor - self.linear_conv_kernel_dim = linear_conv_kernel_dim - self.linear_allow_neg_eigval = linear_allow_neg_eigval - self.vocab_size = vocab_size - self.max_position_embeddings = max_position_embeddings - self.hidden_size = hidden_size - self.intermediate_size = intermediate_size - self.num_hidden_layers = num_hidden_layers - self.num_attention_heads = num_attention_heads - - # for backward compatibility - if num_key_value_heads is None: - num_key_value_heads = num_attention_heads - - self.num_key_value_heads = num_key_value_heads - self.hidden_act = hidden_act - self.initializer_range = initializer_range - self.rms_norm_eps = rms_norm_eps - self.use_cache = use_cache - self.attention_bias = attention_bias - self.attention_dropout = attention_dropout - self.rope_parameters = rope_parameters - - self.tie_word_embeddings = tie_word_embeddings - self.pad_token_id = pad_token_id - self.bos_token_id = bos_token_id - self.eos_token_id = eos_token_id diff --git a/vllm/transformers_utils/configs/ovis.py b/vllm/transformers_utils/configs/ovis.py index bc84bfe812..d5662d7ef5 100644 --- a/vllm/transformers_utils/configs/ovis.py +++ b/vllm/transformers_utils/configs/ovis.py @@ -2,69 +2,10 @@ # SPDX-FileCopyrightText: Copyright contributors to the vLLM project # ruff: noqa: E501 -# adapted from https://huggingface.co/AIDC-AI/Ovis2-1B/blob/main/configuration_aimv2.py -# and https://huggingface.co/AIDC-AI/Ovis2-1B/blob/main/configuration_ovis.py -# Ovis Config with AimV2 config registration removed for Transformers compatibility +# adapted from https://huggingface.co/AIDC-AI/Ovis2-1B/blob/main/configuration_ovis.py from typing import Any -from transformers import AutoConfig, PreTrainedConfig - - -class AIMv2Config(PreTrainedConfig): - """This is the configuration class to store the configuration of an [`AIMv2Model`]. - Instantiating a configuration with the defaults will yield a similar configuration - to that of the [apple/aimv2-large-patch14-224](https://huggingface.co/apple/aimv2-large-patch14-224). - - Args: - hidden_size: Dimension of the hidden representations. - intermediate_size: Dimension of the SwiGLU representations. - num_hidden_layers: Number of hidden layers in the Transformer. - num_attention_heads: Number of attention heads for each attention layer - in the Transformer. - num_channels: Number of input channels. - image_size: Image size. - patch_size: Patch size. - rms_norm_eps: Epsilon value used for the RMS normalization layer. - attention_dropout: Dropout ratio for attention probabilities. - projection_dropout: Dropout ratio for the projection layer after the attention. - qkv_bias: Whether to add a bias to the queries, keys and values. - use_bias: Whether to add a bias in the feed-forward and projection layers. - kwargs: Keyword arguments for the [`PreTrainedConfig`]. - - """ - - model_type: str = "aimv2" - - def __init__( - self, - hidden_size: int = 1024, - intermediate_size: int = 2816, - num_hidden_layers: int = 24, - num_attention_heads: int = 8, - num_channels: int = 3, - image_size: int = 224, - patch_size: int = 14, - rms_norm_eps: float = 1e-5, - attention_dropout: float = 0.0, - projection_dropout: float = 0.0, - qkv_bias: bool = False, - use_bias: bool = False, - **kwargs: Any, - ): - super().__init__(**kwargs) - self.hidden_size = hidden_size - self.intermediate_size = intermediate_size - self.num_hidden_layers = num_hidden_layers - self.num_attention_heads = num_attention_heads - self.num_channels = num_channels - self.patch_size = patch_size - self.image_size = image_size - self.attention_dropout = attention_dropout - self.rms_norm_eps = rms_norm_eps - - self.projection_dropout = projection_dropout - self.qkv_bias = qkv_bias - self.use_bias = use_bias +from transformers import Aimv2VisionConfig, AutoConfig, PreTrainedConfig # ---------------------------------------------------------------------- @@ -96,14 +37,13 @@ class BaseVisualTokenizerConfig(PreTrainedConfig): f"expect `backbone_config` to be instance of PreTrainedConfig or dict, but got {type(backbone_config)} type" ) if not isinstance(backbone_config, PreTrainedConfig): - model_type = backbone_config["model_type"] - if model_type != "aimv2": - backbone_config.pop("model_type") + model_type = backbone_config.pop("model_type") + if model_type == "aimv2": + backbone_config = Aimv2VisionConfig(**backbone_config) + else: backbone_config = AutoConfig.for_model( model_type, **backbone_config ) - else: - backbone_config = AIMv2Config(**backbone_config) self.backbone_config = backbone_config self.hidden_stride = hidden_stride diff --git a/vllm/transformers_utils/configs/qwen3_5.py b/vllm/transformers_utils/configs/qwen3_5.py deleted file mode 100644 index 57f64f0f78..0000000000 --- a/vllm/transformers_utils/configs/qwen3_5.py +++ /dev/null @@ -1,193 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# SPDX-FileCopyrightText: Copyright contributors to the vLLM project -# Copyright 2025 The Qwen Team and The HuggingFace Inc. team. -# All rights reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -"""Qwen3.5 model configuration.""" - -from transformers.configuration_utils import PreTrainedConfig - - -class Qwen3_5TextConfig(PreTrainedConfig): - model_type = "qwen3_5_text" - keys_to_ignore_at_inference = ["past_key_values"] - - base_model_tp_plan = { - "layers.*.self_attn.q_proj": "colwise", - "layers.*.self_attn.k_proj": "colwise", - "layers.*.self_attn.v_proj": "colwise", - "layers.*.self_attn.o_proj": "rowwise", - "layers.*.mlp.gate_proj": "colwise", - "layers.*.mlp.up_proj": "colwise", - "layers.*.mlp.down_proj": "rowwise", - } - base_model_pp_plan = { - "embed_tokens": (["input_ids"], ["inputs_embeds"]), - "layers": (["hidden_states", "attention_mask"], ["hidden_states"]), - "norm": (["hidden_states"], ["hidden_states"]), - } - base_config_key = "text_config" - - def __init__( - self, - vocab_size=248320, - hidden_size=4096, - intermediate_size=12288, - num_hidden_layers=32, - num_attention_heads=16, - num_key_value_heads=4, - hidden_act="silu", - max_position_embeddings=32768, - initializer_range=0.02, - rms_norm_eps=1e-6, - use_cache=True, - tie_word_embeddings=False, - rope_parameters=None, - attention_bias=False, - attention_dropout=0.0, - head_dim=256, - linear_conv_kernel_dim=4, - linear_key_head_dim=128, - linear_value_head_dim=128, - linear_num_key_heads=16, - linear_num_value_heads=32, - layer_types=None, - pad_token_id=None, - bos_token_id=None, - eos_token_id=None, - **kwargs, - ): - self.vocab_size = vocab_size - self.max_position_embeddings = max_position_embeddings - self.hidden_size = hidden_size - self.intermediate_size = intermediate_size - self.num_hidden_layers = num_hidden_layers - self.num_attention_heads = num_attention_heads - self.num_key_value_heads = num_key_value_heads - self.hidden_act = hidden_act - self.initializer_range = initializer_range - self.rms_norm_eps = rms_norm_eps - self.use_cache = use_cache - self.attention_bias = attention_bias - self.attention_dropout = attention_dropout - self.head_dim = head_dim - self.rope_parameters = rope_parameters - kwargs.setdefault("partial_rotary_factor", 0.25) - - self.layer_types = layer_types - if self.layer_types is None: - interval_pattern = kwargs.get("full_attention_interval", 4) - self.layer_types = [ - "linear_attention" - if bool((i + 1) % interval_pattern) - else "full_attention" - for i in range(self.num_hidden_layers) - ] - kwargs["ignore_keys_at_rope_validation"] = { - "mrope_section", - "mrope_interleaved", - } - self.validate_layer_type() - - # linear attention part - self.linear_conv_kernel_dim = linear_conv_kernel_dim - self.linear_key_head_dim = linear_key_head_dim - self.linear_value_head_dim = linear_value_head_dim - self.linear_num_key_heads = linear_num_key_heads - self.linear_num_value_heads = linear_num_value_heads - super().__init__(**kwargs) - # Set these AFTER super().__init__() because transformers v4's - # PreTrainedConfig.__init__ has these as explicit params with different - # defaults (e.g. tie_word_embeddings=True) that would overwrite our values. - self.pad_token_id = pad_token_id - self.bos_token_id = bos_token_id - self.eos_token_id = eos_token_id - self.tie_word_embeddings = tie_word_embeddings - - -class Qwen3_5VisionConfig(PreTrainedConfig): - model_type = "qwen3_5" - base_config_key = "vision_config" - - def __init__( - self, - depth=27, - hidden_size=1152, - hidden_act="gelu_pytorch_tanh", - intermediate_size=4304, - num_heads=16, - in_channels=3, - patch_size=16, - spatial_merge_size=2, - temporal_patch_size=2, - out_hidden_size=3584, - num_position_embeddings=2304, - initializer_range=0.02, - **kwargs, - ): - super().__init__(**kwargs) - - self.depth = depth - self.hidden_size = hidden_size - self.hidden_act = hidden_act - self.intermediate_size = intermediate_size - self.num_heads = num_heads - self.in_channels = in_channels - self.patch_size = patch_size - self.spatial_merge_size = spatial_merge_size - self.temporal_patch_size = temporal_patch_size - self.out_hidden_size = out_hidden_size - self.num_position_embeddings = num_position_embeddings - self.initializer_range = initializer_range - - -class Qwen3_5Config(PreTrainedConfig): - model_type = "qwen3_5" - sub_configs = { - "vision_config": Qwen3_5VisionConfig, - "text_config": Qwen3_5TextConfig, - } - keys_to_ignore_at_inference = ["past_key_values"] - - def __init__( - self, - text_config=None, - vision_config=None, - image_token_id=248056, - video_token_id=248057, - vision_start_token_id=248053, - vision_end_token_id=248054, - tie_word_embeddings=False, - **kwargs, - ): - if isinstance(vision_config, dict): - self.vision_config = self.sub_configs["vision_config"](**vision_config) - elif vision_config is None: - self.vision_config = self.sub_configs["vision_config"]() - - if isinstance(text_config, dict): - self.text_config = self.sub_configs["text_config"](**text_config) - elif text_config is None: - self.text_config = self.sub_configs["text_config"]() - - self.image_token_id = image_token_id - self.video_token_id = video_token_id - self.vision_start_token_id = vision_start_token_id - self.vision_end_token_id = vision_end_token_id - super().__init__(**kwargs) - # Set after super().__init__() to avoid v4 PreTrainedConfig overwrite - self.tie_word_embeddings = tie_word_embeddings - - -__all__ = ["Qwen3_5Config", "Qwen3_5TextConfig"] diff --git a/vllm/transformers_utils/configs/qwen3_5_moe.py b/vllm/transformers_utils/configs/qwen3_5_moe.py deleted file mode 100644 index 34e66c825a..0000000000 --- a/vllm/transformers_utils/configs/qwen3_5_moe.py +++ /dev/null @@ -1,205 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# SPDX-FileCopyrightText: Copyright contributors to the vLLM project -# Copyright 2025 The Qwen Team and The HuggingFace Inc. team. -# All rights reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -"""Qwen3.5-MoE model configuration.""" - -from transformers.configuration_utils import PreTrainedConfig - - -class Qwen3_5MoeTextConfig(PreTrainedConfig): - model_type = "qwen3_5_moe_text" - keys_to_ignore_at_inference = ["past_key_values"] - - base_model_tp_plan = { - "layers.*.self_attn.q_proj": "colwise", - "layers.*.self_attn.k_proj": "colwise", - "layers.*.self_attn.v_proj": "colwise", - "layers.*.self_attn.o_proj": "rowwise", - "layers.*.mlp.experts.gate_up_proj": "packed_colwise", - "layers.*.mlp.experts.down_proj": "rowwise", - "layers.*.mlp.shared_expert.gate_proj": "colwise", - "layers.*.mlp.shared_expert.up_proj": "colwise", - "layers.*.mlp.shared_expert.down_proj": "rowwise", - } - base_model_pp_plan = { - "embed_tokens": (["input_ids"], ["inputs_embeds"]), - "layers": (["hidden_states", "attention_mask"], ["hidden_states"]), - "norm": (["hidden_states"], ["hidden_states"]), - } - base_config_key = "text_config" - - def __init__( - self, - vocab_size=248320, - hidden_size=2048, - num_hidden_layers=40, - num_attention_heads=16, - num_key_value_heads=2, - hidden_act="silu", - max_position_embeddings=32768, - initializer_range=0.02, - rms_norm_eps=1e-6, - use_cache=True, - tie_word_embeddings=False, - rope_parameters=None, - attention_bias=False, - attention_dropout=0.0, - head_dim=256, - linear_conv_kernel_dim=4, - linear_key_head_dim=128, - linear_value_head_dim=128, - linear_num_key_heads=16, - linear_num_value_heads=32, - moe_intermediate_size=512, - shared_expert_intermediate_size=512, - num_experts_per_tok=8, - num_experts=256, - output_router_logits=False, - router_aux_loss_coef=0.001, - layer_types=None, - pad_token_id=None, - bos_token_id=None, - eos_token_id=None, - **kwargs, - ): - self.vocab_size = vocab_size - self.max_position_embeddings = max_position_embeddings - self.hidden_size = hidden_size - self.num_hidden_layers = num_hidden_layers - self.num_attention_heads = num_attention_heads - self.num_key_value_heads = num_key_value_heads - self.hidden_act = hidden_act - self.initializer_range = initializer_range - self.rms_norm_eps = rms_norm_eps - self.use_cache = use_cache - self.attention_bias = attention_bias - self.attention_dropout = attention_dropout - self.head_dim = head_dim - self.rope_parameters = rope_parameters - kwargs.setdefault("partial_rotary_factor", 0.25) - - self.layer_types = layer_types - if self.layer_types is None: - interval_pattern = kwargs.get("full_attention_interval", 4) - self.layer_types = [ - "linear_attention" - if bool((i + 1) % interval_pattern) - else "full_attention" - for i in range(self.num_hidden_layers) - ] - kwargs["ignore_keys_at_rope_validation"] = { - "mrope_section", - "mrope_interleaved", - } - self.validate_layer_type() - - # linear attention part - self.linear_conv_kernel_dim = linear_conv_kernel_dim - self.linear_key_head_dim = linear_key_head_dim - self.linear_value_head_dim = linear_value_head_dim - self.linear_num_key_heads = linear_num_key_heads - self.linear_num_value_heads = linear_num_value_heads - self.moe_intermediate_size = moe_intermediate_size - self.shared_expert_intermediate_size = shared_expert_intermediate_size - self.num_experts_per_tok = num_experts_per_tok - self.num_experts = num_experts - self.output_router_logits = output_router_logits - self.router_aux_loss_coef = router_aux_loss_coef - super().__init__(**kwargs) - # Set these AFTER super().__init__() because transformers v4's - # PreTrainedConfig.__init__ has these as explicit params with different - # defaults (e.g. tie_word_embeddings=True) that would overwrite our values. - self.pad_token_id = pad_token_id - self.bos_token_id = bos_token_id - self.eos_token_id = eos_token_id - self.tie_word_embeddings = tie_word_embeddings - - -class Qwen3_5MoeVisionConfig(PreTrainedConfig): - model_type = "qwen3_5_moe" - base_config_key = "vision_config" - - def __init__( - self, - depth=27, - hidden_size=1152, - hidden_act="gelu_pytorch_tanh", - intermediate_size=4304, - num_heads=16, - in_channels=3, - patch_size=16, - spatial_merge_size=2, - temporal_patch_size=2, - out_hidden_size=3584, - num_position_embeddings=2304, - initializer_range=0.02, - **kwargs, - ): - super().__init__(**kwargs) - - self.depth = depth - self.hidden_size = hidden_size - self.hidden_act = hidden_act - self.intermediate_size = intermediate_size - self.num_heads = num_heads - self.in_channels = in_channels - self.patch_size = patch_size - self.spatial_merge_size = spatial_merge_size - self.temporal_patch_size = temporal_patch_size - self.out_hidden_size = out_hidden_size - self.num_position_embeddings = num_position_embeddings - self.initializer_range = initializer_range - - -class Qwen3_5MoeConfig(PreTrainedConfig): - model_type = "qwen3_5_moe" - sub_configs = { - "vision_config": Qwen3_5MoeVisionConfig, - "text_config": Qwen3_5MoeTextConfig, - } - keys_to_ignore_at_inference = ["past_key_values"] - - def __init__( - self, - text_config=None, - vision_config=None, - image_token_id=248056, - video_token_id=248057, - vision_start_token_id=248053, - vision_end_token_id=248054, - tie_word_embeddings=False, - **kwargs, - ): - if isinstance(vision_config, dict): - self.vision_config = self.sub_configs["vision_config"](**vision_config) - elif vision_config is None: - self.vision_config = self.sub_configs["vision_config"]() - - if isinstance(text_config, dict): - self.text_config = self.sub_configs["text_config"](**text_config) - elif text_config is None: - self.text_config = self.sub_configs["text_config"]() - - self.image_token_id = image_token_id - self.video_token_id = video_token_id - self.vision_start_token_id = vision_start_token_id - self.vision_end_token_id = vision_end_token_id - super().__init__(**kwargs) - # Set after super().__init__() to avoid v4 PreTrainedConfig overwrite - self.tie_word_embeddings = tie_word_embeddings - - -__all__ = ["Qwen3_5MoeConfig", "Qwen3_5MoeTextConfig"] diff --git a/vllm/transformers_utils/configs/qwen3_next.py b/vllm/transformers_utils/configs/qwen3_next.py deleted file mode 100644 index 8b083ff963..0000000000 --- a/vllm/transformers_utils/configs/qwen3_next.py +++ /dev/null @@ -1,277 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# SPDX-FileCopyrightText: Copyright contributors to the vLLM project -# Copyright 2025 The Qwen team, Alibaba Group and the HuggingFace Inc. team. -# All rights reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -"""Qwen3-Next model configuration.""" - -from transformers.configuration_utils import PreTrainedConfig -from transformers.utils import logging - -logger = logging.get_logger(__name__) - - -class Qwen3NextConfig(PreTrainedConfig): - r"""This is the configuration class to store the configuration of a [`Qwen3NextModel`]. It is used to instantiate a - Qwen3-Next model according to the specified arguments, defining the model architecture. - Instantiating a configuration with the defaults will yield a similar configuration to that of - Qwen3-Next-80B-A3B-Instruct [Qwen/Qwen3-Next-80B-A3B-Instruct](https://huggingface.co/Qwen/Qwen3-Next-80B-A3B-Instruct). - - Configuration objects inherit from [`PreTrainedConfig`] and can be used to control the model outputs. Read the - documentation from [`PreTrainedConfig`] for more information. - - - Args: - vocab_size (`int`, *optional*, defaults to 151936): - Vocabulary size of the model. Defines the number of different tokens that can be represented by the - `inputs_ids`. - hidden_size (`int`, *optional*, defaults to 2048): - Dimension of the hidden representations. - intermediate_size (`int`, *optional*, defaults to 5632): - Dimension of the MLP representations. - num_hidden_layers (`int`, *optional*, defaults to 48): - Number of hidden layers in the Transformer encoder. - num_attention_heads (`int`, *optional*, defaults to 16): - Number of attention heads for each attention layer in the Transformer encoder. - num_key_value_heads (`int`, *optional*, defaults to 2): - This is the number of key_value heads that should be used to implement Grouped Query Attention. If - `num_key_value_heads=num_attention_heads`, the model will use Multi Head Attention (MHA), if - `num_key_value_heads=1` the model will use Multi Query Attention (MQA) otherwise GQA is used. When - converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed - by meanpooling all the original heads within that group. For more details checkout [this - paper](https://arxiv.org/pdf/2305.13245.pdf). If it is not specified, will default to `32`. - hidden_act (`str`, *optional*, defaults to `"silu"`): - The non-linear activation function in the decoder. - max_position_embeddings (`int`, *optional*, defaults to 32768): - The maximum sequence length that this model might ever be used with. - initializer_range (`float`, *optional*, defaults to 0.02): - The standard deviation of the truncated_normal_initializer for initializing all weight matrices. - rms_norm_eps (`float`, *optional*, defaults to 1e-06): - The epsilon used by the rms normalization layers. - use_cache (`bool`, *optional*, defaults to `True`): - Whether or not the model should return the last key/values attentions (not used by all models). Only - relevant if `config.is_decoder=True`. - tie_word_embeddings (`bool`, *optional*, defaults to `False`): - Whether the model's input and output word embeddings should be tied. - rope_parameters (`dict`, *optional*): - Dictionary containing the scaling configuration for the RoPE embeddings. NOTE: if you apply new rope type - and you expect the model to work on longer `max_position_embeddings`, we recommend you to update this value - accordingly. - Expected contents: - `rope_theta` (`float`): The base period of the RoPE embeddings. - `rope_type` (`str`): - The sub-variant of RoPE to use. Can be one of ['default', 'linear', 'dynamic', 'yarn', 'longrope', - 'llama3'], with 'default' being the original RoPE implementation. - `factor` (`float`, *optional*): - Used with all rope types except 'default'. The scaling factor to apply to the RoPE embeddings. In - most scaling types, a `factor` of x will enable the model to handle sequences of length x * - original maximum pre-trained length. - `original_max_position_embeddings` (`int`, *optional*): - Used with 'dynamic', 'longrope' and 'llama3'. The original max position embeddings used during - pretraining. - `attention_factor` (`float`, *optional*): - Used with 'yarn' and 'longrope'. The scaling factor to be applied on the attention - computation. If unspecified, it defaults to value recommended by the implementation, using the - `factor` field to infer the suggested value. - `beta_fast` (`float`, *optional*): - Only used with 'yarn'. Parameter to set the boundary for extrapolation (only) in the linear - ramp function. If unspecified, it defaults to 32. - `beta_slow` (`float`, *optional*): - Only used with 'yarn'. Parameter to set the boundary for interpolation (only) in the linear - ramp function. If unspecified, it defaults to 1. - `short_factor` (`List[float]`, *optional*): - Only used with 'longrope'. The scaling factor to be applied to short contexts (< - `original_max_position_embeddings`). Must be a list of numbers with the same length as the hidden - size divided by the number of attention heads divided by 2 - `long_factor` (`List[float]`, *optional*): - Only used with 'longrope'. The scaling factor to be applied to long contexts (< - `original_max_position_embeddings`). Must be a list of numbers with the same length as the hidden - size divided by the number of attention heads divided by 2 - `low_freq_factor` (`float`, *optional*): - Only used with 'llama3'. Scaling factor applied to low frequency components of the RoPE - `high_freq_factor` (`float`, *optional*): - Only used with 'llama3'. Scaling factor applied to high frequency components of the RoPE - `partial_rotary_factor` (`float`, *optional*, defaults to 0.25): - Percentage of the query and keys which will have rotary embedding. - attention_bias (`bool`, *optional*, defaults to `False`): - Whether to use a bias in the query, key, value and output projection layers during self-attention. - attention_dropout (`float`, *optional*, defaults to 0.0): - The dropout ratio for the attention probabilities. - head_dim (`int`, *optional*, defaults to 256): - Projection weights dimension in multi-head attention. - linear_conv_kernel_dim (`int`, *optional*, defaults to 4): - Kernel size of the convolution used in linear attention layers. - linear_key_head_dim (`int`, *optional*, defaults to 128): - Dimension of each key head in linear attention. - linear_value_head_dim (`int`, *optional*, defaults to 128): - Dimension of each value head in linear attention. - linear_num_key_heads (`int`, *optional*, defaults to 16): - Number of key heads used in linear attention layers. - linear_num_value_heads (`int`, *optional*, defaults to 32): - Number of value heads used in linear attention layers. - decoder_sparse_step (`int`, *optional*, defaults to 1): - The frequency of the MoE layer. - moe_intermediate_size (`int`, *optional*, defaults to 512): - Intermediate size of the routed expert. - shared_expert_intermediate_size (`int`, *optional*, defaults to 512): - Intermediate size of the shared expert. - num_experts_per_tok (`int`, *optional*, defaults to 10): - Number of selected experts. - num_experts (`int`, *optional*, defaults to 512): - Number of routed experts. - norm_topk_prob (`bool`, *optional*, defaults to `True`): - Whether to normalize the topk probabilities. - output_router_logits (`bool`, *optional*, defaults to `False`): - Whether or not the router logits should be returned by the model. Enabling this will also - allow the model to output the auxiliary loss, including load balancing loss and router z-loss. - router_aux_loss_coef (`float`, *optional*, defaults to 0.001): - The aux loss factor for the total loss. - mlp_only_layers (`list[int]`, *optional*, defaults to `[]`): - Indicate which layers use Qwen3NextMLP rather than Qwen3NextSparseMoeBlock - The list contains layer index, from 0 to num_layers-1 if we have num_layers layers - If `mlp_only_layers` is empty, `decoder_sparse_step` is used to determine the sparsity. - layer_types (`list[str]`, *optional*): - Types of each layer (attention or linear). - - ```python - >>> from transformers import Qwen3NextModel, Qwen3NextConfig - - >>> # Initializing a Qwen3Next style configuration - >>> configuration = Qwen3NextConfig() - - >>> # Initializing a model from the Qwen3-Next-80B-A3B style configuration - >>> model = Qwen3NextModel(configuration) - - >>> # Accessing the model configuration - >>> configuration = model.config - ``` - - """ # noqa: E501 - - model_type = "qwen3_next" - keys_to_ignore_at_inference = ["past_key_values"] - - base_model_tp_plan = { - "layers.*.self_attn.q_proj": "colwise", - "layers.*.self_attn.k_proj": "colwise", - "layers.*.self_attn.v_proj": "colwise", - "layers.*.self_attn.o_proj": "rowwise", - "layers.*.mlp.experts.*.gate_proj": "colwise", - "layers.*.mlp.experts.*.up_proj": "colwise", - "layers.*.mlp.experts.*.down_proj": "rowwise", - "layers.*.mlp.shared_experts.gate_proj": "colwise", - "layers.*.mlp.shared_experts.up_proj": "colwise", - "layers.*.mlp.shared_experts.down_proj": "rowwise", - "layers.*.mlp.gate_proj": "colwise", - "layers.*.mlp.up_proj": "colwise", - "layers.*.mlp.down_proj": "rowwise", - } - base_model_pp_plan = { - "embed_tokens": (["input_ids"], ["inputs_embeds"]), - "layers": (["hidden_states", "attention_mask"], ["hidden_states"]), - "norm": (["hidden_states"], ["hidden_states"]), - } - - def __init__( - self, - vocab_size=151936, - hidden_size=2048, - intermediate_size=5632, - num_hidden_layers=48, - num_attention_heads=16, - num_key_value_heads=2, - hidden_act="silu", - max_position_embeddings=32768, - initializer_range=0.02, - rms_norm_eps=1e-6, - use_cache=True, - tie_word_embeddings=False, - rope_parameters=None, - attention_bias=False, - attention_dropout=0.0, - head_dim=256, - linear_conv_kernel_dim=4, - linear_key_head_dim=128, - linear_value_head_dim=128, - linear_num_key_heads=16, - linear_num_value_heads=32, - decoder_sparse_step=1, - moe_intermediate_size=512, - shared_expert_intermediate_size=512, - num_experts_per_tok=10, - num_experts=512, - norm_topk_prob=True, - output_router_logits=False, - router_aux_loss_coef=0.001, - mlp_only_layers=None, - layer_types=None, - **kwargs, - ): - if mlp_only_layers is None: - mlp_only_layers = [] - self.vocab_size = vocab_size - self.max_position_embeddings = max_position_embeddings - self.hidden_size = hidden_size - self.intermediate_size = intermediate_size - self.num_hidden_layers = num_hidden_layers - self.num_attention_heads = num_attention_heads - self.num_key_value_heads = num_key_value_heads - self.hidden_act = hidden_act - self.initializer_range = initializer_range - self.rms_norm_eps = rms_norm_eps - self.use_cache = use_cache - # Try to set `rope_scaling` if available, otherwise use `rope_parameters` - rope_scaling = kwargs.pop("rope_scaling", None) - rope_parameters = rope_scaling or rope_parameters or {"rope_type": "default"} - rope_theta = kwargs.pop("rope_theta", 10000.0) - if "rope_theta" not in rope_parameters: - rope_parameters["rope_theta"] = rope_theta - partial_rotary_factor = kwargs.pop("partial_rotary_factor", 0.25) - if "partial_rotary_factor" not in rope_parameters: - rope_parameters["partial_rotary_factor"] = partial_rotary_factor - self.rope_parameters = rope_parameters - self.partial_rotary_factor = partial_rotary_factor - self.attention_bias = attention_bias - self.attention_dropout = attention_dropout - self.head_dim = head_dim - - self.layer_types = layer_types - if self.layer_types is None: - self.layer_types = [ - "linear_attention" if bool((i + 1) % 4) else "full_attention" - for i in range(self.num_hidden_layers) - ] - self.validate_layer_type() - - # linear attention part - self.linear_conv_kernel_dim = linear_conv_kernel_dim - self.linear_key_head_dim = linear_key_head_dim - self.linear_value_head_dim = linear_value_head_dim - self.linear_num_key_heads = linear_num_key_heads - self.linear_num_value_heads = linear_num_value_heads - - # MoE arguments - self.decoder_sparse_step = decoder_sparse_step - self.moe_intermediate_size = moe_intermediate_size - self.shared_expert_intermediate_size = shared_expert_intermediate_size - self.num_experts_per_tok = num_experts_per_tok - self.num_experts = num_experts - self.norm_topk_prob = norm_topk_prob - self.output_router_logits = output_router_logits - self.router_aux_loss_coef = router_aux_loss_coef - self.mlp_only_layers = mlp_only_layers - super().__init__(tie_word_embeddings=tie_word_embeddings, **kwargs) - - -__all__ = ["Qwen3NextConfig"] diff --git a/vllm/transformers_utils/model_arch_config_convertor.py b/vllm/transformers_utils/model_arch_config_convertor.py index 9dee4af924..d79c1275b6 100644 --- a/vllm/transformers_utils/model_arch_config_convertor.py +++ b/vllm/transformers_utils/model_arch_config_convertor.py @@ -17,7 +17,6 @@ from vllm.transformers_utils.config import ( ConfigFormat, get_safetensors_params_metadata, ) -from vllm.transformers_utils.configs.gemma4 import gemma4_layer_config from vllm.utils.torch_utils import common_broadcastable_dtype logger = init_logger(__name__) @@ -41,7 +40,7 @@ class ModelArchConfigConvertorBase: This is the only place that decides whether a checkpoint is heterogeneous and how many layers it has, so a convertor can declare variation that - Transformers does not express (see `Gemma4ModelArchConfigConvertor`). + Transformers does not express. Each pair is converted independently and the results are diffed, so a convertor that overrides this must not also collapse a varying field in @@ -745,27 +744,6 @@ class Gemma4ModelArchConfigConvertor(ModelArchConfigConvertorBase): == "vision" ) - def get_per_layer_hf_configs( - self, - ) -> list[tuple[PreTrainedConfig, PreTrainedConfig]] | None: - # Gemma4 uses a larger head dimension, and sometimes more KV heads, on its - # full attention layers than on its sliding ones. Transformers >= 5.15.0 - # says so in the config; before that the values are flat attributes picked - # apart by `layer_types`, so build the per-layer configs here instead. Both - # then reach the base convertor as ordinary homogeneous configs. - if getattr(self.hf_text_config, "is_heterogeneous", False): - return super().get_per_layer_hf_configs() - text_config = self.hf_text_config - if not getattr(text_config, "layer_types", None): - return None - # `num_hidden_layers` is what the stack is built from, and it is what - # Transformers sizes `per_layer_config` by. `layer_types` can be longer, - # e.g. when a test truncates the layer count to build a small model. - return [ - (self.hf_config, gemma4_layer_config(text_config, layer_idx)) - for layer_idx in range(self.get_num_hidden_layers()) - ] - class MossAudioModelArchConfigConvertor(ModelArchConfigConvertorBase): def _language_config(self) -> PreTrainedConfig: diff --git a/vllm/transformers_utils/processor.py b/vllm/transformers_utils/processor.py index 64af034d9a..9aa5920849 100644 --- a/vllm/transformers_utils/processor.py +++ b/vllm/transformers_utils/processor.py @@ -181,7 +181,11 @@ def get_video_processor_cls_name_from_config( # given type when torchvision is unavailable; callers then use opencv. model_config = get_hf_file_to_dict("config.json", processor_name, revision=revision) if model_config and "model_type" in model_config: - return VIDEO_PROCESSOR_MAPPING_NAMES.get(model_config["model_type"]) + video_processor = VIDEO_PROCESSOR_MAPPING_NAMES.get(model_config["model_type"]) + # Transformers >= 5.19 maps to {"torchvision": name} + if isinstance(video_processor, dict): + video_processor = video_processor.get("torchvision") + return video_processor return None diff --git a/vllm/transformers_utils/processors/__init__.py b/vllm/transformers_utils/processors/__init__.py index b913545377..f2af4901e2 100644 --- a/vllm/transformers_utils/processors/__init__.py +++ b/vllm/transformers_utils/processors/__init__.py @@ -17,10 +17,8 @@ __all__ = [ "FireRedASR2Processor", "FunASRProcessor", "GLM4VProcessor", - "Granite4VisionProcessor", "H2OVLProcessor", "Moondream3Processor", - "InternVLProcessor", "IsaacProcessor", "KimiAudioProcessor", "KimiK25Processor", @@ -55,9 +53,7 @@ _CLASS_TO_MODULE: dict[str, str] = { "FireRedASR2Processor": "vllm.transformers_utils.processors.fireredasr2", "FunASRProcessor": "vllm.transformers_utils.processors.funasr", "GLM4VProcessor": "vllm.transformers_utils.processors.glm4v", - "Granite4VisionProcessor": "vllm.transformers_utils.processors.granite4_vision", "H2OVLProcessor": "vllm.transformers_utils.processors.h2ovl", - "InternVLProcessor": "vllm.transformers_utils.processors.internvl", "IsaacProcessor": "vllm.transformers_utils.processors.isaac", "KimiAudioProcessor": "vllm.transformers_utils.processors.kimi_audio", "KimiK25Processor": "vllm.transformers_utils.processors.kimi_k25", diff --git a/vllm/transformers_utils/processors/granite4_vision.py b/vllm/transformers_utils/processors/granite4_vision.py deleted file mode 100644 index 7972015adf..0000000000 --- a/vllm/transformers_utils/processors/granite4_vision.py +++ /dev/null @@ -1,78 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# SPDX-FileCopyrightText: Copyright contributors to the vLLM project - -from fractions import Fraction - -from transformers import LlavaNextProcessor -from transformers.image_processing_utils import select_best_resolution - - -class Granite4VisionProcessor(LlavaNextProcessor): - """Processor for Granite 4 Vision. - - Extends LlavaNextProcessor to account for the Window Q-Former - downsampling when computing the number of image features. - - This processor is needed because the granite4_vision processor type - is not yet in the transformers version pinned by vLLM. - """ - - model_type = "granite4_vision" - - def __init__( - self, - image_processor=None, - tokenizer=None, - patch_size=None, - vision_feature_select_strategy=None, - chat_template=None, - image_token="", - num_additional_image_tokens=0, - downsample_rate=None, - **kwargs, - ): - super().__init__( - image_processor=image_processor, - tokenizer=tokenizer, - patch_size=patch_size, - vision_feature_select_strategy=vision_feature_select_strategy, - chat_template=chat_template, - image_token=image_token, - num_additional_image_tokens=num_additional_image_tokens, - ) - self.downsample_rate = downsample_rate - - def _get_number_of_features( - self, - orig_height: int, - orig_width: int, - height: int, - width: int, - ) -> int: - image_grid_pinpoints = self.image_processor.image_grid_pinpoints - - height_best_resolution, width_best_resolution = select_best_resolution( - [orig_height, orig_width], image_grid_pinpoints - ) - scale_height = height_best_resolution // height - scale_width = width_best_resolution // width - - patches_height = height // self.patch_size - patches_width = width // self.patch_size - if self.downsample_rate is not None: - ds_rate = Fraction(self.downsample_rate) - patches_height = int(patches_height * ds_rate) - patches_width = int(patches_width * ds_rate) - - unpadded_features, newline_features = self._get_unpadded_features( - orig_height, - orig_width, - patches_height, - patches_width, - scale_height, - scale_width, - ) - base_features = ( - patches_height * patches_width + self.num_additional_image_tokens - ) - return unpadded_features + newline_features + base_features diff --git a/vllm/transformers_utils/processors/kimi_k25.py b/vllm/transformers_utils/processors/kimi_k25.py index 13aed85de4..800dd3a7ed 100644 --- a/vllm/transformers_utils/processors/kimi_k25.py +++ b/vllm/transformers_utils/processors/kimi_k25.py @@ -26,6 +26,8 @@ class KimiK25Processor(ProcessorMixin): text: str | list[str] | None = None, vision_chunks: list[VisionChunk] | None = None, return_tensors: str | TensorType | None = None, + do_rescale: bool = True, + do_normalize: bool = True, **kwargs, ) -> BatchFeature: """Args: @@ -49,6 +51,8 @@ class KimiK25Processor(ProcessorMixin): mm_inputs = self.image_processor.preprocess( vision_chunks, return_tensors=return_tensors, + do_rescale=do_rescale, + do_normalize=do_normalize, ) else: mm_inputs = {} diff --git a/vllm/transformers_utils/processors/kimi_k25_vision_fused.py b/vllm/transformers_utils/processors/kimi_k25_vision_fused.py index 6390789817..254ffe013e 100644 --- a/vllm/transformers_utils/processors/kimi_k25_vision_fused.py +++ b/vllm/transformers_utils/processors/kimi_k25_vision_fused.py @@ -14,9 +14,12 @@ from PIL import Image from transformers.image_processing_utils import BaseImageProcessor, BatchFeature from transformers.utils import TensorType +from vllm.multimodal.image import open_image from vllm.utils.import_utils import is_numba_available from vllm.utils.jit_monitor import numba_workqueue_threading_layer +_IDENTITY_LUT = np.tile(np.arange(256, dtype=np.uint8)[:, None], (1, 3)) + if is_numba_available(): from numba import njit, prange @@ -151,10 +154,10 @@ def _to_pil(data: Any) -> Image.Image: if isinstance(data, str): if data.startswith("data:"): raw_base64 = data.split(",", 1)[1] - return Image.open(io.BytesIO(base64.b64decode(raw_base64))).convert("RGB") - return Image.open(data).convert("RGB") + return open_image(io.BytesIO(base64.b64decode(raw_base64))).convert("RGB") + return open_image(data).convert("RGB") if isinstance(data, bytes): - return Image.open(io.BytesIO(data)).convert("RGB") + return open_image(io.BytesIO(data)).convert("RGB") raise ValueError(f"Unsupported data type: {type(data)}") @@ -238,7 +241,12 @@ class KimiK25FusedVisionProcessor(BaseImageProcessor): self, medias: list[dict[str, Any]], return_tensors: str | TensorType | None = None, + do_rescale: bool = True, + do_normalize: bool = True, ) -> BatchFeature: + assert do_rescale == do_normalize + device_normalize = not do_normalize + if not isinstance(medias, list): medias = [medias] if not medias: @@ -296,8 +304,9 @@ class KimiK25FusedVisionProcessor(BaseImageProcessor): ) total_patches += num_patches + lut = _IDENTITY_LUT if device_normalize else self.normalize_lut pixel_values_np = np.empty( - (total_patches, 3, patch_size, patch_size), dtype=np.float32 + (total_patches, 3, patch_size, patch_size), dtype=lut.dtype ) out_offset = 0 with numba_workqueue_threading_layer(): @@ -318,7 +327,7 @@ class KimiK25FusedVisionProcessor(BaseImageProcessor): padded_height, padded_width, patch_size, - self.normalize_lut, + lut, ) out_offset += num_patches diff --git a/vllm/transformers_utils/processors/kimi_k3.py b/vllm/transformers_utils/processors/kimi_k3.py index 73af85eb09..e726a9c38a 100644 --- a/vllm/transformers_utils/processors/kimi_k3.py +++ b/vllm/transformers_utils/processors/kimi_k3.py @@ -35,6 +35,8 @@ class KimiK3Processor(ProcessorMixin): text: str | list[str] | None = None, images: object | list[object] | None = None, return_tensors: str | TensorType | None = None, + do_rescale: bool = True, + do_normalize: bool = True, **kwargs, ) -> BatchFeature: if images is not None: @@ -44,6 +46,8 @@ class KimiK3Processor(ProcessorMixin): mm_inputs = self.image_processor.preprocess( medias, return_tensors=return_tensors, + do_rescale=do_rescale, + do_normalize=do_normalize, ) else: mm_inputs = {} diff --git a/vllm/triton_utils/importing.py b/vllm/triton_utils/importing.py index e312564c0b..7e501c94f4 100644 --- a/vllm/triton_utils/importing.py +++ b/vllm/triton_utils/importing.py @@ -3,11 +3,10 @@ import os import types -from importlib.metadata import version from importlib.util import find_spec from vllm.logger import init_logger -from vllm.platforms import current_platform +from vllm.platforms import current_platform, vllm_version_matches_substr from vllm.utils.math_utils import cdiv logger = init_logger(__name__) @@ -35,7 +34,7 @@ if HAS_TRITON: ] # Check Triton CPU - if "cpu" in version("vllm"): + if vllm_version_matches_substr("cpu"): if "cpu" in backends: HAS_TRITON = True # Suppress following warnings on CPU-only platforms @@ -128,6 +127,8 @@ class TritonLanguagePlaceholder(types.ModuleType): self.dtype = None self.int64 = None self.int32 = None + self.uint8 = None + self.pointer_type = lambda element_ty: None self.tensor = None self.exp = None self.exp2 = None diff --git a/vllm/usage/usage_lib.py b/vllm/usage/usage_lib.py index 5197aada59..b75a4fa161 100644 --- a/vllm/usage/usage_lib.py +++ b/vllm/usage/usage_lib.py @@ -69,7 +69,7 @@ def is_usage_stats_enabled(): def _get_current_timestamp_ns() -> int: - return int(datetime.datetime.now(datetime.timezone.utc).timestamp() * 1e9) + return int(datetime.datetime.now(datetime.UTC).timestamp() * 1e9) def _detect_cloud_provider() -> str: diff --git a/vllm/utils/argparse_utils.py b/vllm/utils/argparse_utils.py index b91b1f26fd..9ad03453fd 100644 --- a/vllm/utils/argparse_utils.py +++ b/vllm/utils/argparse_utils.py @@ -119,7 +119,6 @@ class SortedHelpFormatter(ArgumentDefaultsHelpFormatter, RawDescriptionHelpForma class FlexibleArgumentParser(ArgumentParser): """ArgumentParser that allows both underscore and dash in names.""" - _deprecated: set[Action] = set() _json_tip: str = ( "When passing JSON CLI arguments, the following sets of arguments " "are equivalent:\n" @@ -132,6 +131,8 @@ class FlexibleArgumentParser(ArgumentParser): _search_keyword: str | None = None def __init__(self, *args, **kwargs): + # Must exist before super().__init__(), which already adds arguments. + self._deprecated: set[Action] = set() # Set the default "formatter_class" to SortedHelpFormatter if "formatter_class" not in kwargs: kwargs["formatter_class"] = SortedHelpFormatter @@ -160,7 +161,7 @@ class FlexibleArgumentParser(ArgumentParser): def parse_known_args(self, args=None, namespace=None): namespace, args = super().parse_known_args(args, namespace) - for action in FlexibleArgumentParser._deprecated: + for action in self._deprecated: if ( hasattr(namespace, dest := action.dest) and getattr(namespace, dest) != action.default @@ -172,15 +173,19 @@ class FlexibleArgumentParser(ArgumentParser): deprecated = kwargs.pop("deprecated", False) action = super().add_argument(*args, **kwargs) if deprecated: - FlexibleArgumentParser._deprecated.add(action) + self._deprecated.add(action) return action class _FlexibleArgumentGroup(_ArgumentGroup): + def __init__(self, container, *args, **kwargs): + super().__init__(container, *args, **kwargs) + self._deprecated: set[Action] = container._deprecated + def add_argument(self, *args, **kwargs): deprecated = kwargs.pop("deprecated", False) action = super().add_argument(*args, **kwargs) if deprecated: - FlexibleArgumentParser._deprecated.add(action) + self._deprecated.add(action) return action def add_argument_group(self, *args, **kwargs): diff --git a/vllm/utils/collection_utils.py b/vllm/utils/collection_utils.py index de32ed0c15..8a77da86cd 100644 --- a/vllm/utils/collection_utils.py +++ b/vllm/utils/collection_utils.py @@ -8,9 +8,9 @@ This is similar in concept to the `collections` module. import math from collections import defaultdict from collections.abc import Callable, Generator, Hashable, Iterable, Mapping, Sequence -from typing import Generic, Literal, TypeVar +from typing import Generic, Literal, TypeVar, assert_never, overload -from typing_extensions import TypeIs, assert_never, overload +from typing_extensions import TypeIs T = TypeVar("T") diff --git a/vllm/utils/flashinfer.py b/vllm/utils/flashinfer.py index d7fa807c33..9b615cd2fa 100644 --- a/vllm/utils/flashinfer.py +++ b/vllm/utils/flashinfer.py @@ -11,8 +11,7 @@ import importlib import importlib.util import os import shutil -from collections.abc import Callable, Iterator -from contextvars import ContextVar +from collections.abc import Callable from typing import Any, NoReturn import requests @@ -27,24 +26,6 @@ from vllm.utils.torch_utils import PIN_MEMORY logger = init_logger(__name__) -_bf16_autotune_buckets: ContextVar[tuple[int, ...] | None] = ContextVar( - "flashinfer_bf16_autotune_buckets", default=None -) - - -@contextlib.contextmanager -def autotune_bf16_only( - tuning_buckets: tuple[int, ...], *, skip_ops: set[str] | None = None -) -> Iterator[None]: - """Tune BF16 calls with bounded buckets, outside full-model autotuning.""" - token = _bf16_autotune_buckets.set(tuning_buckets) - try: - with autotune(tune_mode=False, skip_ops=skip_ops): - yield - finally: - _bf16_autotune_buckets.reset(token) - - def flashinfer_bf16_mm( a: torch.Tensor, b: torch.Tensor, @@ -54,21 +35,14 @@ def flashinfer_bf16_mm( ) -> torch.Tensor: from flashinfer import mm_bf16 - tuning_buckets = _bf16_autotune_buckets.get() - tuning = ( - autotune(tune_mode=True, tuning_buckets=tuning_buckets) - if tuning_buckets is not None - else contextlib.nullcontext() + return mm_bf16( + a, + b, + bias=bias, + pdl=pdl, + out_dtype=torch.bfloat16, + backend=backend, ) - with tuning: - return mm_bf16( - a, - b, - bias=bias, - pdl=pdl, - out_dtype=torch.bfloat16, - backend=backend, - ) # This is the storage path for the cubins, it can be replaced @@ -1091,47 +1065,6 @@ def flashinfer_scaled_fp4_mm( ) -def flashinfer_scaled_fp4_mm_out( - a: torch.Tensor, - b: torch.Tensor, - block_scale_a: torch.Tensor, - block_scale_b: torch.Tensor, - alpha: torch.Tensor, - out: torch.Tensor, - out_dtype: torch.dtype | None, - use_8x4_sf_layout: bool, - backend: str, -) -> torch.Tensor: - assert a.ndim == 2 and b.ndim == 2 and out.ndim == 2 - assert block_scale_a.ndim == 2 and block_scale_b.ndim == 2 - assert a.stride(-1) == 1 - assert a.shape[1] == b.shape[0] - assert out.shape == (a.shape[0], b.shape[1]) - assert out.device.type == "cuda" - - if backend in ("cutlass", "cudnn"): - if block_scale_a.dtype != torch.uint8: - block_scale_a = block_scale_a.view(torch.uint8) - if block_scale_b.dtype != torch.uint8: - block_scale_b = block_scale_b.view(torch.uint8) - - from flashinfer import mm_fp4 as flashinfer_mm_fp4_ - - flashinfer_mm_fp4_( - a, - b, - block_scale_a, - block_scale_b, - alpha, - out_dtype or out.dtype, - out=out, - block_size=16, - use_8x4_sf_layout=use_8x4_sf_layout, - backend=backend, - ) - return out - - def flashinfer_scaled_fp8_mm( a: torch.Tensor, b: torch.Tensor, @@ -1162,38 +1095,6 @@ def flashinfer_scaled_fp8_mm( return output -def flashinfer_scaled_fp8_mm_out( - a: torch.Tensor, - b: torch.Tensor, - scale_a: torch.Tensor, - scale_b: torch.Tensor, - out: torch.Tensor, - out_dtype: torch.dtype | None = None, -) -> torch.Tensor: - assert a.ndim == 2 and b.ndim == 2 and out.ndim == 2 - assert a.shape[1] == b.shape[0] - assert out.shape == (a.shape[0], b.shape[1]) - assert scale_a.numel() == 1 and scale_b.numel() == 1 - assert a.dtype == torch.float8_e4m3fn and b.dtype == torch.float8_e4m3fn - assert out.device.type == "cuda" - assert a.is_contiguous() - - from flashinfer import bmm_fp8 as bmm_fp8_ - - bmm_fp8_( - a.unsqueeze(0), - # FlashInfer expects the weight in the same column-major view layout - # consumed by flashinfer_scaled_fp8_mm, so keep the transposed view. - b.unsqueeze(0), - scale_a, - scale_b, - out_dtype or out.dtype, - out.unsqueeze(0), - "auto", - ) - return out - - def flashinfer_quant_nvfp4_8x4_sf_layout( a: torch.Tensor, a_global_sf: torch.Tensor ) -> tuple[torch.Tensor, torch.Tensor]: @@ -1295,7 +1196,6 @@ def is_flashinfer_cudnn_fp8_prefill_attn_supported() -> bool: __all__ = [ "has_flashinfer", "flashinfer_bf16_mm", - "autotune_bf16_only", "has_flashinfer_bf16_gemm", "is_flashinfer_bf16_gemm_supported", "is_flashinfer_cutedsl_bf16_gemm_supported", @@ -1337,9 +1237,7 @@ __all__ = [ "use_trtllm_attention", "flashinfer_mxfp4_quantize", "flashinfer_scaled_fp4_mm", - "flashinfer_scaled_fp4_mm_out", "flashinfer_scaled_fp8_mm", - "flashinfer_scaled_fp8_mm_out", "flashinfer_quant_nvfp4_8x4_sf_layout", "flashinfer_fp8_blockscale_gemm", "should_use_flashinfer_for_blockscale_fp8_gemm", diff --git a/vllm/utils/import_utils.py b/vllm/utils/import_utils.py index aaacccc7cb..04a7ebec2e 100644 --- a/vllm/utils/import_utils.py +++ b/vllm/utils/import_utils.py @@ -12,10 +12,9 @@ import os import sys from functools import cache from types import ModuleType -from typing import Any +from typing import Any, Never import regex as re -from typing_extensions import Never from vllm.logger import init_logger diff --git a/vllm/utils/jit_monitor.py b/vllm/utils/jit_monitor.py index ffb63bc493..387189c793 100644 --- a/vllm/utils/jit_monitor.py +++ b/vllm/utils/jit_monitor.py @@ -499,32 +499,30 @@ def _setup_tilelang_jit_hook() -> None: jit_kernel_cls.__init__ = _init_with_monitor if jit_impl_cls is not None: - original_call = jit_impl_cls.__call__ + # JITImpl.__call__ only calls compile() on a kernel-cache miss, so + # hooking compile() keeps the per-call hot path untouched. + original_compile = jit_impl_cls.compile - @functools.wraps(original_call) - def _call_with_monitor(self, *args, **kwargs): + @functools.wraps(original_compile) + def _compile_with_monitor(self, *args, **kwargs): global _tilelang_jitimpl_compile_depth - cache_key = _tilelang_cache_miss_key(self, args, kwargs) - if cache_key is None: - return original_call(self, *args, **kwargs) - _tilelang_jitimpl_compile_depth += 1 try: detail = None if _verbose: detail = _format_verbose_tilelang_compile_details( - self, args, kwargs, cache_key + self, args, kwargs, _tilelang_cache_miss_key(self, args, kwargs) ) func = getattr(self, "func", None) orig_func = getattr(func, "orig_func", None) _log_tilelang_jit_compile( _tilelang_kernel_name(orig_func or func), detail ) - return original_call(self, *args, **kwargs) + return original_compile(self, *args, **kwargs) finally: _tilelang_jitimpl_compile_depth -= 1 - jit_impl_cls.__call__ = _call_with_monitor + jit_impl_cls.compile = _compile_with_monitor _tilelang_hook_installed = True diff --git a/vllm/utils/math_utils.py b/vllm/utils/math_utils.py index 8fb0009d60..9854249af2 100644 --- a/vllm/utils/math_utils.py +++ b/vllm/utils/math_utils.py @@ -2,6 +2,8 @@ # SPDX-FileCopyrightText: Copyright contributors to the vLLM project """Math utility functions for vLLM.""" +from bisect import bisect_left + # Approximate value of 1/ln(2), used for log/exp base conversion # Best FP32 approximation: 1.4426950216 (hex 0x3FB8AA3B) RCP_LN2 = 1.4426950216 @@ -30,3 +32,9 @@ def round_down(x: int, y: int) -> int: def largest_power_of_2_divisor(n: int) -> int: """Return the largest power-of-2 that divides *n* (isolate lowest set bit).""" return n & (-n) + + +def round_up_to(values: list[int], n: int) -> int | None: + """Round `n` up to the smallest member of sorted `values` >= n, or None.""" + i = bisect_left(values, n) + return values[i] if i < len(values) else None diff --git a/vllm/utils/network_utils.py b/vllm/utils/network_utils.py index fa24d4d5f9..6bc4f9c9fd 100644 --- a/vllm/utils/network_utils.py +++ b/vllm/utils/network_utils.py @@ -11,6 +11,7 @@ from collections.abc import ( Iterator, Sequence, ) +from dataclasses import dataclass from typing import Any from uuid import uuid4 @@ -25,6 +26,23 @@ from vllm.logger import init_logger logger = init_logger(__name__) +@dataclass +class ZmqListener: + """A supervisor-bound listener that a ZMQ socket can adopt.""" + + address: str + socket: socket.socket + + def close(self) -> None: + self.socket.close() + + def cleanup(self) -> None: + self.close() + if self.address.startswith("ipc://"): + with contextlib.suppress(FileNotFoundError): + os.unlink(self.address.removeprefix("ipc://")) + + def close_sockets(sockets: Sequence[zmq.Socket | zmq.asyncio.Socket]): for sock in sockets: if sock is not None: @@ -307,6 +325,27 @@ def make_zmq_path(scheme: str, host: str, port: int | None = None) -> str: # Adapted from: https://github.com/sgl-project/sglang/blob/v0.4.1/python/sglang/srt/utils.py#L783 # noqa: E501 +# Apply the existing make_zmq_socket memory policy to every socket owner. +_ZMQ_LARGE_BUFFER_SIZE = 512 * 1024**2 +_ZMQ_LARGE_BUFFER_MIN_TOTAL_MEMORY = 32 * 1024**3 +_ZMQ_LARGE_BUFFER_MIN_AVAILABLE_MEMORY = 16 * 1024**3 + + +def _get_zmq_socket_buffer_size() -> int: + """Choose the shared libzmq and inherited-listener buffer policy.""" + mem = psutil.virtual_memory() + # For systems with substantial memory (>32GB total, >16GB available): + # - Set a large 0.5GB buffer to improve throughput + # For systems with less memory: + # - Use system default (-1) to avoid excessive memory consumption + if ( + mem.total > _ZMQ_LARGE_BUFFER_MIN_TOTAL_MEMORY + and mem.available > _ZMQ_LARGE_BUFFER_MIN_AVAILABLE_MEMORY + ): + return _ZMQ_LARGE_BUFFER_SIZE + return -1 + + def make_zmq_socket( ctx: zmq.asyncio.Context | zmq.Context, # type: ignore[name-defined] path: str, @@ -315,22 +354,19 @@ def make_zmq_socket( identity: bytes | None = None, linger: int | None = None, router_handover: bool = False, + listener: socket.socket | None = None, ) -> zmq.Socket | zmq.asyncio.Socket: # type: ignore[name-defined] - """Make a ZMQ socket with the proper bind/connect semantics.""" - mem = psutil.virtual_memory() - socket = ctx.socket(socket_type) + """Make a ZMQ socket with the proper bind/connect semantics. - # Calculate buffer size based on system memory - total_mem = mem.total / 1024**3 - available_mem = mem.available / 1024**3 - # For systems with substantial memory (>32GB total, >16GB available): - # - Set a large 0.5GB buffer to improve throughput - # For systems with less memory: - # - Use system default (-1) to avoid excessive memory consumption - buf_size = int(0.5 * 1024**3) if total_mem > 32 and available_mem > 16 else -1 + When supplied, ``listener`` is detached and its fd moves to libzmq. + """ + socket = ctx.socket(socket_type) + buf_size = _get_zmq_socket_buffer_size() if bind is None: bind = socket_type not in (zmq.PUSH, zmq.SUB, zmq.XSUB) + if listener is not None and not bind: + raise ValueError("An inherited ZMQ listener requires bind=True") if socket_type in (zmq.PULL, zmq.DEALER, zmq.ROUTER): socket.setsockopt(zmq.RCVHWM, 0) @@ -353,6 +389,9 @@ def make_zmq_socket( if socket_type == zmq.XPUB: socket.setsockopt(zmq.XPUB_VERBOSE, True) + if listener is not None: + socket.setsockopt(zmq.USE_FD, listener.detach()) + # Determine if the path is a TCP socket with an IPv6 address. # Enable IPv6 on the zmq socket if so. scheme, host, _ = split_zmq_path(path) @@ -367,6 +406,48 @@ def make_zmq_socket( return socket +def make_zmq_listener(path: str, socket_type: Any) -> ZmqListener: + """Bind and listen on a raw socket for later ZMQ adoption.""" + scheme, host, port = split_zmq_path(path) + if scheme == "tcp": + family = socket.AF_INET6 if is_valid_ipv6_address(host) else socket.AF_INET + bind_address: str | tuple[str, int] = (host, int(port)) + elif scheme == "ipc": + family = socket.AF_UNIX + bind_address = path.removeprefix("ipc://") + os.makedirs(os.path.dirname(bind_address), exist_ok=True) + else: + raise ValueError(f"Cannot inherit a {scheme} ZMQ listener") + + listener = socket.socket(family, socket.SOCK_STREAM) + bound = False + try: + buf_size = _get_zmq_socket_buffer_size() + if buf_size >= 0: + # Accepted stream sockets inherit these settings from the listener. + # ZMQ_USE_FD adopts this pre-created socket, so the policy belongs + # here as well as on the eventual libzmq socket. + if socket_type in (zmq.PULL, zmq.DEALER, zmq.ROUTER): + listener.setsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF, buf_size) + if socket_type in (zmq.PUSH, zmq.DEALER, zmq.ROUTER): + listener.setsockopt(socket.SOL_SOCKET, socket.SO_SNDBUF, buf_size) + + listener.bind(bind_address) + bound = True + listener.listen() + if scheme == "tcp": + path = get_tcp_uri(host, listener.getsockname()[1]) + return ZmqListener(address=path, socket=listener) + except BaseException: + listener.close() + # A failed bind leaves any existing pathname owned by its listener. + if scheme == "ipc" and bound: + assert isinstance(bind_address, str) + with contextlib.suppress(FileNotFoundError): + os.unlink(bind_address) + raise + + @contextlib.contextmanager def zmq_socket_ctx( path: str, diff --git a/vllm/utils/system_utils.py b/vllm/utils/system_utils.py index 5b90438dca..a15dbf3c6c 100644 --- a/vllm/utils/system_utils.py +++ b/vllm/utils/system_utils.py @@ -72,7 +72,8 @@ def suppress_stdout(): yield return - stdout_fd = sys.stdout.fileno() + # C libraries write to fd 1, whatever sys.stdout is bound to. + stdout_fd = 1 stdout_dup = os.dup(stdout_fd) devnull_fd = os.open(os.devnull, os.O_WRONLY) diff --git a/vllm/utils/torch_utils.py b/vllm/utils/torch_utils.py index b4a13419d5..dce5e84fce 100644 --- a/vllm/utils/torch_utils.py +++ b/vllm/utils/torch_utils.py @@ -50,6 +50,7 @@ STR_DTYPE_TO_TORCH_DTYPE = { "turboquant_4bit_nc": torch.uint8, "turboquant_k3v4_nc": torch.uint8, "turboquant_3bit_nc": torch.uint8, + "ultraquant_4bit": torch.uint8, "nvfp4": torch.uint8, "nvfp4_4over6": torch.uint8, } @@ -530,8 +531,12 @@ def kv_cache_dtype_str_to_dtype( return STR_DTYPE_TO_TORCH_DTYPE[kv_cache_dtype] -def set_random_seed(seed: int | None) -> None: +def set_random_seed(seed: int | None, data_parallel_index: int = 0) -> None: if seed is not None: + if data_parallel_index: + # DP engines share the seed but must not sample unseeded requests alike. + ss = np.random.SeedSequence((seed, data_parallel_index)) + seed = int(ss.generate_state(1)[0]) random.seed(seed) np.random.seed(seed) torch.manual_seed(seed) diff --git a/vllm/v1/attention/backend.py b/vllm/v1/attention/backend.py index 319b9977f4..3387a31794 100644 --- a/vllm/v1/attention/backend.py +++ b/vllm/v1/attention/backend.py @@ -506,19 +506,13 @@ class CommonAttentionMetadata: # but not the right per-request boundaries. Padding requests have a query # length of zero and are skipped by the device boundary search. num_mapped_tokens = int(self.query_start_loc_cpu[-1]) - from vllm.v1.attention.ops.metadata import _token_request_mapping_kernel + from vllm.v1.attention.ops.metadata import compute_token_to_req_indices num_output_tokens = max(num_mapped_tokens, num_tokens) assert buffer.shape[0] >= num_output_tokens - _token_request_mapping_kernel[((num_output_tokens + 255) // 256,)]( - self.query_start_loc, - buffer, - self.query_start_loc.shape[0] - 1, - num_mapped_tokens, - num_output_tokens, - num_warps=4, + self._token_to_req_indices_cache = compute_token_to_req_indices( + self.query_start_loc, buffer, num_mapped_tokens, num_output_tokens ) - self._token_to_req_indices_cache = buffer[:num_output_tokens] return self._token_to_req_indices_cache[:num_tokens] # TODO(lucas): remove once we have FULL-CG spec-decode support @@ -609,8 +603,11 @@ class AttentionMetadataBuilder(ABC, Generic[M]): supports_update_block_table: bool = False # Whether the builder constructor requires the block-table width. requires_block_table_width: ClassVar[bool] = False - # Whether all step-dependent draft decode metadata can be updated in place, - # allowing one metadata build to be reused across autoregressive draft steps. + # Whether update_draft_decode_metadata() can regenerate all decode metadata + # from persistent device buffers with capture-safe ops only. Lets a + # speculator record the refresh inside its draft decode CUDA graph, so no + # eager build() is needed between steps or between the batches that replay + # it, whether the graph holds one draft forward or a multi-step loop. supports_draft_decode_metadata_update: bool = False @abstractmethod @@ -758,12 +755,12 @@ class AttentionMetadataBuilder(ABC, Generic[M]): ) def update_draft_decode_metadata(self, metadata: M) -> None: - """Update step-dependent draft decode metadata in place. + """Update draft decode metadata in place. - The fused draft loop may call this method during full CUDA graph - capture. CUDA graph replay does not run this Python method, so - implementations must emit capture-safe operations and keep replayed - tensor state in persistent storage. + Speculators can record this call inside their draft decode CUDA graphs, + so it runs once at capture and never again in Python. Implementations + must emit only capture-safe operations and write results into persistent + tensors that ``metadata`` already references. """ raise NotImplementedError @@ -816,8 +813,9 @@ class AttentionImplBase(ABC, Generic[T]): is_sparse: ClassVar[bool] = False # Whether this impl provides a dense-MHA prefill path (forward_mha). Sparse - # impls without one run the top-k MQA path for all requests. - supports_dense_mha_prefill: ClassVar[bool] = True + # impls without one run the top-k MQA path for all requests. Impls may + # override it per instance, depending on the kv-cache layout. + supports_dense_mha_prefill: bool = True # Required attributes that all impls should have num_heads: int diff --git a/vllm/v1/attention/backends/flashinfer.py b/vllm/v1/attention/backends/flashinfer.py index 91ff6da3f0..85b7456dd8 100755 --- a/vllm/v1/attention/backends/flashinfer.py +++ b/vllm/v1/attention/backends/flashinfer.py @@ -16,6 +16,7 @@ from flashinfer import ( BatchPrefillWithRaggedKVCacheWrapper, MultiLevelCascadeAttentionWrapper, get_seq_lens, + nvfp4_quantize_append_paged_kv_cache_with_slot_mapping, ) from flashinfer.decode import fast_decode_plan, trtllm_batch_decode_with_kv_cache from flashinfer.prefill import trtllm_batch_context_with_kv_cache @@ -106,6 +107,36 @@ logger = init_logger(__name__) trtllm_workspace_buffer = None +def _nvfp4_kv_on_fa2() -> bool: + return any(current_platform.is_device_capability_family(f) for f in (80, 90, 120)) + + +_KVPair = tuple[torch.Tensor, torch.Tensor] + + +def _nvfp4_kv_views( + kv_cache: torch.Tensor, num_kv_heads: int +) -> tuple[_KVPair, _KVPair, str]: + """Views of NVFP4 pages laid out as [K data | K scale | V data | V scale].""" + num_blocks, num_slots, block_size, full_dim = kv_cache.shape + hnd = kv_cache.stride()[1:] == (block_size * full_dim, full_dim, 1) + if not hnd and kv_cache.stride()[1:] != (full_dim, num_slots * full_dim, 1): + raise ValueError(f"NVFP4 KV pages must be dense, got {kv_cache.stride()}") + dims = (num_kv_heads, block_size) if hnd else (block_size, num_kv_heads) + side_size = num_kv_heads * block_size * full_dim + (k_data, k_scale), (v_data, v_scale) = ( + nvfp4_split_data_scale( + kv_cache.as_strided( + (num_blocks, *dims, full_dim), + (kv_cache.stride(0), dims[1] * full_dim, full_dim, 1), + kv_cache.storage_offset() + side * side_size, + ) + ) + for side in range(2) + ) + return (k_data, v_data), (k_scale, v_scale), "HND" if hnd else "NHD" + + def _get_trtllm_workspace_buffer(): global trtllm_workspace_buffer if trtllm_workspace_buffer is None: @@ -498,6 +529,9 @@ class FlashInferBackend(AttentionBackend): @classmethod def supports_kv_cache_dtype(cls, kv_cache_dtype: CacheDType | None) -> bool: if kv_cache_dtype is not None and kv_cache_dtype.startswith("nvfp4"): + if _nvfp4_kv_on_fa2(): + # The slot-mapping writer has no scale search. + return kv_cache_dtype == "nvfp4" return ( current_platform.is_device_capability_family(100) and supports_trtllm_attention(is_prefill=True) @@ -505,6 +539,30 @@ class FlashInferBackend(AttentionBackend): ) return super().supports_kv_cache_dtype(kv_cache_dtype) + @classmethod + def supports_combination( + cls, + head_size: int, + dtype: torch.dtype, + kv_cache_dtype: CacheDType | None, + block_size: int | None, + use_mla: bool, + has_sink: bool, + use_sparse: bool, + use_mm_prefix: bool, + device_capability: DeviceCapability, + ) -> str | None: + if ( + kv_cache_dtype is not None + and kv_cache_dtype.startswith("nvfp4") + and device_capability.major in (8, 9, 12) + ): + if head_size == 64: + return "fa2 prefill misreads an NVFP4 KV cache at head_size 64" + if has_sink: + return "the sink prefill wrapper cannot read an NVFP4 KV cache" + return None + @classmethod def get_supported_head_sizes(cls) -> list[int]: # https://github.com/flashinfer-ai/flashinfer/blob/3d55c71a62052c590c130897d3a3db49b14fcc34/include/flashinfer/utils.cuh#L157 @@ -777,7 +835,16 @@ class FlashInferMetadataBuilder(AttentionMetadataBuilder[FlashInferMetadata]): # Cannot use self.kv_cache_spec.dtype here because kv_cache_spec # storage dtype may not be the same as the op dtype (uint8 vs fp8_e4m3) self.is_kvcache_nvfp4 = self.cache_dtype.startswith("nvfp4") - if self.is_kvcache_nvfp4: + # fa2 plans NVFP4 by its uint8 storage dtype. + self.nvfp4_fa2 = self.is_kvcache_nvfp4 and _nvfp4_kv_on_fa2() + self.nvfp4_trtllm = self.is_kvcache_nvfp4 and not self.nvfp4_fa2 + if self.nvfp4_fa2 and self.use_dcp: + raise NotImplementedError( + "The DCP prefill wrapper cannot read NVFP4 block scales." + ) + if self.nvfp4_fa2 and self.head_dim == 64: + raise NotImplementedError("fa2 prefill misreads NVFP4 at head_size 64.") + if self.nvfp4_trtllm: if ( force_use_trtllm_attention() is False or not supports_trtllm_attention(is_prefill=True) @@ -798,6 +865,7 @@ class FlashInferMetadataBuilder(AttentionMetadataBuilder[FlashInferMetadata]): else: self.cache_dtype = "auto" self.is_kvcache_nvfp4 = False + self.nvfp4_fa2 = self.nvfp4_trtllm = False assert self.kv_cache_spec.dtype == self.model_config.dtype self.kv_cache_dtype = self.kv_cache_spec.dtype @@ -810,8 +878,11 @@ class FlashInferMetadataBuilder(AttentionMetadataBuilder[FlashInferMetadata]): # Prefer TRTLLM/XQA for decoding whenever supported. The decode kernel # must be selected statically for FULL cudagraph capture. - can_use_xqa_or_trtllm_gen_decode = can_use_trtllm_attention( - self.num_qo_heads, self.num_kv_heads, is_prefill=False + can_use_xqa_or_trtllm_gen_decode = ( + can_use_trtllm_attention( + self.num_qo_heads, self.num_kv_heads, is_prefill=False + ) + and not self.nvfp4_fa2 ) # Page sizes >= 128 require the trtllm-gen GQA/MQA path (guaranteed by # get_supported_kernel_block_sizes). @@ -894,7 +965,7 @@ class FlashInferMetadataBuilder(AttentionMetadataBuilder[FlashInferMetadata]): self.window_left = self.global_hyperparameters.window_left self.logits_soft_cap = self.global_hyperparameters.logits_soft_cap self.has_sinks = self.global_hyperparameters.has_sinks - if self.has_sinks and not FlashInferBackend.supports_sink(): + if self.has_sinks and (self.nvfp4_fa2 or not FlashInferBackend.supports_sink()): raise NotImplementedError( "FlashInfer backend currently does not support attention " "sinks, please use trtllm on blackwell or flash attention on " @@ -970,6 +1041,8 @@ class FlashInferMetadataBuilder(AttentionMetadataBuilder[FlashInferMetadata]): return FlashInferBackend.get_dtype_for_flashinfer(cache_dtype) return self.model_config.dtype if cache_dtype.startswith("nvfp4"): + if self.nvfp4_fa2: + return self.model_config.dtype return FlashInferBackend.get_dtype_for_flashinfer("fp8_e4m3") return self.kv_cache_spec.dtype @@ -1002,7 +1075,13 @@ class FlashInferMetadataBuilder(AttentionMetadataBuilder[FlashInferMetadata]): num_qo_heads=num_qo_heads, num_kv_heads=spec.num_kv_heads, is_prefill=False, - ) or (is_xqa_arch and not _is_xqa_head_dim_supported(spec.head_size)): + ) or ( + is_xqa_arch + and ( + spec.kv_quant_mode.is_nvfp4 + or not _is_xqa_head_dim_supported(spec.head_size) + ) + ): has_uniform_batch_support = False break @@ -1230,8 +1309,8 @@ class FlashInferMetadataBuilder(AttentionMetadataBuilder[FlashInferMetadata]): ) else: # NVFP4 KV cache requires the trtllm-gen backend inside - # the wrapper; fa2/fa3 do not support nvfp4. - backend = "trtllm-gen" if self.is_kvcache_nvfp4 else "auto" + # the wrapper on SM100; SM8x, SM90 and SM12x read it with fa2. + backend = "trtllm-gen" if self.nvfp4_trtllm else "auto" self._prefill_wrapper = BatchPrefillWithPagedKVCacheWrapper( self._get_workspace_buffer(), get_flashinfer_layout_string(self.kv_cache_layout), @@ -1256,8 +1335,8 @@ class FlashInferMetadataBuilder(AttentionMetadataBuilder[FlashInferMetadata]): paged_kv_indices = None paged_kv_last_page_len = None # NVFP4 KV cache requires the trtllm-gen backend inside - # the wrapper; fa2/fa3 do not support nvfp4. - backend = "trtllm-gen" if self.is_kvcache_nvfp4 else "auto" + # the wrapper on SM100; SM8x, SM90 and SM12x read it with fa2. + backend = "trtllm-gen" if self.nvfp4_trtllm else "auto" decode_wrapper = BatchDecodeWithPagedKVCacheWrapper( self._get_workspace_buffer(), get_flashinfer_layout_string(self.kv_cache_layout), @@ -1710,7 +1789,7 @@ class FlashInferMetadataBuilder(AttentionMetadataBuilder[FlashInferMetadata]): # use FP8 o_data_type so the wrapper matches the # FP8 output buffer allocated in forward(). o_dtype = ( - FP8_DTYPE if self.is_kvcache_nvfp4 else self.model_config.dtype + FP8_DTYPE if self.nvfp4_trtllm else self.model_config.dtype ) prefill_wrapper.plan( qo_indptr=qo_indptr_prefill_cpu, @@ -1797,9 +1876,7 @@ class FlashInferMetadataBuilder(AttentionMetadataBuilder[FlashInferMetadata]): # NVFP4 trtllm kernel only supports FP8 output; # use FP8 o_data_type so the wrapper matches the # FP8 output buffer allocated in forward(). - o_dtype = ( - FP8_DTYPE if self.is_kvcache_nvfp4 else self.model_config.dtype - ) + o_dtype = FP8_DTYPE if self.nvfp4_trtllm else self.model_config.dtype paged_kv_indptr_cpu = self.paged_kv_indptr.cpu[: num_input_tokens + 1] paged_kv_last_page_len_cpu = self.paged_kv_last_page_len.cpu[ :num_input_tokens @@ -1891,6 +1968,8 @@ class FlashInferImpl(AttentionImpl): ) self.cache_dtype = kv_cache_dtype self.is_kvcache_nvfp4 = kv_cache_dtype.startswith("nvfp4") + self.nvfp4_fa2 = self.is_kvcache_nvfp4 and _nvfp4_kv_on_fa2() + self.nvfp4_trtllm = self.is_kvcache_nvfp4 and not self.nvfp4_fa2 self.kv_cache_dtype = "nvfp4" if self.is_kvcache_nvfp4 else kv_cache_dtype self.fp4_data_dim = head_size // 2 if self.is_kvcache_nvfp4 else 0 self.logits_soft_cap = logits_soft_cap @@ -1937,7 +2016,7 @@ class FlashInferImpl(AttentionImpl): self.o_sf_scale: float | None = None # Pre-allocated FP8 output buffer for NVFP4 without fused output quant. - if self.is_kvcache_nvfp4 and vllm_config is not None: + if self.nvfp4_trtllm and vllm_config is not None: max_num_tokens = vllm_config.scheduler_config.max_num_batched_tokens self._nvfp4_fp8_out = torch.empty( (max_num_tokens, num_heads, head_size), @@ -2195,10 +2274,9 @@ class FlashInferImpl(AttentionImpl): canonicalize_singleton_dim_strides(k_cache.permute(*stride_order)), canonicalize_singleton_dim_strides(v_cache.permute(*stride_order)), ) - k_data, k_sf = nvfp4_split_data_scale(kv_cache_tuple[0]) - v_data, v_sf = nvfp4_split_data_scale(kv_cache_tuple[1]) - nvfp4_kv_data = (k_data, v_data) - nvfp4_kv_block_scales = (k_sf, v_sf) + nvfp4_kv_data, nvfp4_kv_block_scales, _ = _nvfp4_kv_views( + kv_cache, self.num_kv_heads + ) else: kv_cache_tuple = kv_cache_permute.split(hs, dim=-1) @@ -2276,7 +2354,7 @@ class FlashInferImpl(AttentionImpl): # Use a pre-allocated FP8 buffer and dequantize # afterwards. needs_fp8_out_prefill = ( - self.is_kvcache_nvfp4 and output.dtype != FP8_DTYPE + self.nvfp4_trtllm and output.dtype != FP8_DTYPE ) if needs_fp8_out_prefill: out_prefill = self._nvfp4_fp8_out[:num_prefill_tokens] @@ -2299,7 +2377,8 @@ class FlashInferImpl(AttentionImpl): prefill_wrapper.run( prefill_query, kv_cache_for_fi, - q_scale=layer._q_scale_float, + # The fa2 NVFP4 query stays in the model dtype. + q_scale=1.0 if self.nvfp4_fa2 else layer._q_scale_float, k_scale=layer._k_scale_float, v_scale=layer._v_scale_float, out=out_prefill, @@ -2451,7 +2530,7 @@ class FlashInferImpl(AttentionImpl): # NVFP4 kernel only supports FP8 output. # Use a pre-allocated FP8 buffer and dequantize afterwards. - needs_fp8_out = self.is_kvcache_nvfp4 and output.dtype != FP8_DTYPE + needs_fp8_out = self.nvfp4_trtllm and output.dtype != FP8_DTYPE if needs_fp8_out: out_decode = self._nvfp4_fp8_out[:num_decode_tokens] else: @@ -2488,7 +2567,7 @@ class FlashInferImpl(AttentionImpl): decode_wrapper.run( decode_query, kv_cache_for_fi, - q_scale=layer._q_scale_float, + q_scale=1.0 if self.nvfp4_fa2 else layer._q_scale_float, k_scale=layer._k_scale_float, v_scale=layer._v_scale_float, out=out_decode, @@ -2668,6 +2747,19 @@ class FlashInferImpl(AttentionImpl): # and value[:num_actual_tokens] because the reshape_and_cache_flash # op uses the slot_mapping's shape to determine the number of # actual tokens. + if self.nvfp4_fa2: + data, scales, kv_layout = _nvfp4_kv_views(kv_cache, self.num_kv_heads) + nvfp4_quantize_append_paged_kv_cache_with_slot_mapping( + key, + value, + slot_mapping, + data, + scales, + layer._k_scale, + layer._v_scale, + kv_layout=kv_layout, + ) + return if self.is_kvcache_nvfp4: # (B, 2*H, N, full_dim) -> ((B, N, H, full_dim), # (B, N, H, full_dim)); diff --git a/vllm/v1/attention/backends/gdn_attn.py b/vllm/v1/attention/backends/gdn_attn.py index 5fece32a1b..08369adcb1 100644 --- a/vllm/v1/attention/backends/gdn_attn.py +++ b/vllm/v1/attention/backends/gdn_attn.py @@ -7,6 +7,7 @@ from typing import Literal import torch +from vllm._aiter_ops import rocm_aiter_ops from vllm.config import VllmConfig from vllm.model_executor.layers.mamba.checkpoint import ( MambaPrefillCheckpointBuilder, @@ -79,6 +80,7 @@ class GDNAttentionMetadata: prefill_query_start_loc: torch.Tensor | None = None prefill_state_indices: torch.Tensor | None = None prefill_has_initial_state: torch.Tensor | None = None + aiter_prefill_metadata: object | None = None # The following attributes are for triton implementation of causal_conv1d nums_dict: dict | None = None @@ -109,8 +111,11 @@ class GDNAttentionMetadataBuilder(AttentionMetadataBuilder[GDNAttentionMetadata] _resolve_gdn_prefill_backend, ) - self.gdn_prefill_backend: Literal["triton", "flashinfer", "cutedsl"] + self.gdn_prefill_backend: Literal[ + "triton", "flashinfer", "cutedsl", "aiter_flydsl" + ] _, self.gdn_prefill_backend = _resolve_gdn_prefill_backend(vllm_config) + self._check_chunk_metadata_override(type(self), self.gdn_prefill_backend) if self.speculative_config: assert self.speculative_config.num_speculative_tokens is not None @@ -186,6 +191,32 @@ class GDNAttentionMetadataBuilder(AttentionMetadataBuilder[GDNAttentionMetadata] device=device, ) + @staticmethod + def _check_chunk_metadata_override( + builder_cls: type["GDNAttentionMetadataBuilder"], backend: str + ) -> None: + """Reject a subclass whose chunk metadata the AITER path would skip. + + AITER brings its own varlen prefill metadata and never calls + ``_build_chunk_metadata``, so a builder that overrides it would lose + that override without a word. No in-tree subclass can get here -- + ``_resolve_gdn_prefill_backend`` only selects this backend for models + with 128-dim GDN key and value heads, which the KDA builders are not -- + but a future one should be told rather than quietly ignored. + """ + if backend != "aiter_flydsl": + return + if ( + builder_cls._build_chunk_metadata + is GDNAttentionMetadataBuilder._build_chunk_metadata + ): + return + raise RuntimeError( + f"{builder_cls.__name__} builds its own FLA chunk metadata, which " + "the 'aiter_flydsl' GDN prefill backend does not use. Select a " + "different gdn_prefill_backend for this model." + ) + def _build_chunk_metadata( self, prefill_query_start_loc: torch.Tensor, @@ -260,11 +291,10 @@ class GDNAttentionMetadataBuilder(AttentionMetadataBuilder[GDNAttentionMetadata] else: spec_sequence_masks_cpu = num_decode_draft_tokens_cpu >= 0 num_spec_decodes = spec_sequence_masks_cpu.sum().item() - if ( - num_spec_decodes == 0 - or num_decode_draft_tokens_cpu[spec_sequence_masks_cpu].sum().item() - == 0 - ): + # A batch whose rows all drafted nothing still has to run the + # speculative path: that is the only path that applies each row's + # accepted-token offset to the recurrent state. + if num_spec_decodes == 0: num_spec_decodes = 0 spec_sequence_masks = None spec_sequence_masks_cpu = None @@ -432,6 +462,7 @@ class GDNAttentionMetadataBuilder(AttentionMetadataBuilder[GDNAttentionMetadata] prefill_query_start_loc: torch.Tensor | None = None prefill_state_indices: torch.Tensor | None = None prefill_has_initial_state: torch.Tensor | None = None + aiter_prefill_metadata: object | None = None if num_prefills > 0: # In a mixed non-spec batch, decodes are peeled off to the recurrent # kernel (decode-first front slice), so build chunk metadata from the @@ -453,11 +484,23 @@ class GDNAttentionMetadataBuilder(AttentionMetadataBuilder[GDNAttentionMetadata] prefill_query_start_loc_cpu = non_spec_query_start_loc_cpu prefill_state_indices = non_spec_state_indices_tensor - chunk_indices, chunk_offsets = self._build_chunk_metadata( - prefill_query_start_loc, - prefill_query_start_loc_cpu, - query_start_loc.device, - ) + if self.gdn_prefill_backend == "aiter_flydsl": + # AITER carries its own reusable varlen metadata and has no use + # for FLA's chunk indices, so it replaces them rather than + # extending what _build_chunk_metadata returns. + assert prefill_query_start_loc_cpu is not None + aiter_prefill_metadata = ( + rocm_aiter_ops.build_gdn_flydsl_prefill_metadata( + torch.diff(prefill_query_start_loc_cpu).tolist(), + cu_seqlens=prefill_query_start_loc, + ) + ) + else: + chunk_indices, chunk_offsets = self._build_chunk_metadata( + prefill_query_start_loc, + prefill_query_start_loc_cpu, + query_start_loc.device, + ) if num_prefills > 0: context_lens_tensor = m.compute_num_computed_tokens() @@ -569,6 +612,7 @@ class GDNAttentionMetadataBuilder(AttentionMetadataBuilder[GDNAttentionMetadata] prefill_query_start_loc=prefill_query_start_loc, prefill_state_indices=prefill_state_indices, prefill_has_initial_state=prefill_has_initial_state, + aiter_prefill_metadata=aiter_prefill_metadata, spec_query_start_loc=spec_query_start_loc, non_spec_query_start_loc=non_spec_query_start_loc, spec_state_indices_tensor=spec_state_indices_tensor, diff --git a/vllm/v1/attention/backends/mamba2_attn.py b/vllm/v1/attention/backends/mamba2_attn.py index e5e3827bcd..fc81cf680e 100644 --- a/vllm/v1/attention/backends/mamba2_attn.py +++ b/vllm/v1/attention/backends/mamba2_attn.py @@ -7,6 +7,10 @@ from typing import Any import torch from vllm.config import VllmConfig +from vllm.model_executor.layers.mamba.checkpoint import ( + MambaPrefillCheckpointBuilder, + MambaPrefillCheckpointMetadata, +) from vllm.utils.torch_utils import async_tensor_h2d from vllm.v1.attention.backend import ( AttentionBackend, @@ -109,6 +113,11 @@ class Mamba2AttentionMetadata(BaseMambaAttentionMetadata): # Chunk-related metadata (only for prefill) seq_idx_p: torch.Tensor | None = None + # Internal prefill checkpoints, one entry per prefill row. The chunk + # index selects the varlen_states row holding the checkpoint state. + checkpoint_chunk_idx: torch.Tensor | None = None + checkpoint_meta: MambaPrefillCheckpointMetadata | None = None + class Mamba2AttentionMetadataBuilder( BaseMambaAttentionMetadataBuilder[Mamba2AttentionMetadata] @@ -128,6 +137,9 @@ class Mamba2AttentionMetadataBuilder( "chunk_size needs to be set in the model config for Mamba2 models" ) self.chunk_size: int = chunk_size + self.checkpoint_builder = MambaPrefillCheckpointBuilder( + vllm_config, kv_cache_spec + ) def build( self, @@ -145,6 +157,8 @@ class Mamba2AttentionMetadataBuilder( seq_idx_p = None cu_chunk_seqlen_p = None last_chunk_indices_p = None + checkpoint_chunk_idx = None + checkpoint_meta = None prep_initial_states = False # Compute seq_idx for prefill only @@ -162,12 +176,27 @@ class Mamba2AttentionMetadataBuilder( ) prep_initial_states = bool((num_computed_tokens_p_cpu > 0).any()) - cu_chunk_seqlen_p, seq_idx_p, last_chunk_indices_p = ( - self._build_chunk_metadata_tensors( - self.chunk_size, - common, - common_attn_metadata, - ) + checkpoint_offsets_p = None + first = common.num_reqs - common.num_prefills + checkpoint = self.checkpoint_builder.build( + common_attn_metadata, + list(range(first, common.num_reqs)), + ) + if checkpoint is not None: + # The host offsets place a chunk boundary on the checkpoint. + checkpoint_offsets_p = checkpoint.offsets + checkpoint_meta = checkpoint + + ( + cu_chunk_seqlen_p, + seq_idx_p, + last_chunk_indices_p, + checkpoint_chunk_idx, + ) = self._build_chunk_metadata_tensors( + self.chunk_size, + common, + common_attn_metadata, + checkpoint_offsets_p, ) return replace( @@ -177,4 +206,22 @@ class Mamba2AttentionMetadataBuilder( seq_idx_p=seq_idx_p, cu_chunk_seqlen_p=cu_chunk_seqlen_p, last_chunk_indices_p=last_chunk_indices_p, + checkpoint_chunk_idx=checkpoint_chunk_idx, + checkpoint_meta=checkpoint_meta, + ) + + def update_block_table( + self, + metadata: Mamba2AttentionMetadata, + blk_table: torch.Tensor, + slot_mapping: torch.Tensor, + ) -> Mamba2AttentionMetadata: + new_metadata = super().update_block_table(metadata, blk_table, slot_mapping) + if metadata.checkpoint_meta is None: + return new_metadata + # Checkpoint destinations are block-table entries, so each group + # re-gathers its own rather than writing into the source group's. + return replace( + new_metadata, + checkpoint_meta=metadata.checkpoint_meta.regather_state_indices(blk_table), ) diff --git a/vllm/v1/attention/backends/mamba_attn.py b/vllm/v1/attention/backends/mamba_attn.py index 126bcfc81d..afb5f59b75 100644 --- a/vllm/v1/attention/backends/mamba_attn.py +++ b/vllm/v1/attention/backends/mamba_attn.py @@ -238,13 +238,14 @@ class BaseMambaAttentionMetadataBuilder(AttentionMetadataBuilder[M], abc.ABC): num_decode_draft_tokens_cpu=num_decode_draft_tokens_cpu, ) + @staticmethod def _compute_chunk_metadata( - self, chunk_size: int, num_prefills: int, num_computed_tokens_p_cpu: torch.Tensor, query_start_loc_p_cpu: torch.Tensor, - ) -> tuple[list[int], list[int], list[int]]: + checkpoint_offsets_p: list[int] | None = None, + ) -> tuple[list[int], list[int], list[int], list[int]]: """Compute chunk-specific metadata for Mamba models. The code below carefully constructs the chunks such that: @@ -256,20 +257,34 @@ class BaseMambaAttentionMetadataBuilder(AttentionMetadataBuilder[M], abc.ABC): of prefix caching for mamba (wip). We need to take care of the interaction with chunked prefill in order to satisfy constraint (2). + + `checkpoint_offsets_p[i]` forces an extra chunk end that many tokens + into row i's query (0 = none), because the SSD scan only materializes + states at chunk ends. Returns its chunk index per row, or -1. """ # TODO (tdoublep): This code could probably be optimized. cu_chunk_seqlen = [] seq_idx = [] last_chunk_indices = [] + checkpoint_chunk_indices = [] seqlen_pos = 0 + # A checkpoint splits its request into two segments so that a chunk + # ends exactly on it; each segment then chunks by the rules below. + segments = [] for req_idx in range(num_prefills): - this_num_computed = num_computed_tokens_p_cpu[req_idx].item() - this_new_tokens = ( + computed = num_computed_tokens_p_cpu[req_idx].item() + tokens = ( query_start_loc_p_cpu[req_idx + 1].item() - query_start_loc_p_cpu[req_idx].item() ) + ckpt = checkpoint_offsets_p[req_idx] if checkpoint_offsets_p else 0 + if ckpt: + segments.append((req_idx, computed, ckpt, True)) + segments.append((req_idx, computed + ckpt, tokens - ckpt, False)) + checkpoint_chunk_indices = [-1] * num_prefills + for req_idx, this_num_computed, this_new_tokens, is_ckpt_end in segments: # if computed tokens are not chunk-aligned, use the first # chunk to finish it off if this_num_computed % chunk_size != 0: @@ -293,11 +308,14 @@ class BaseMambaAttentionMetadataBuilder(AttentionMetadataBuilder[M], abc.ABC): this_new_tokens -= chunk_len assert this_new_tokens == 0 - last_chunk_indices.append(len(cu_chunk_seqlen) - 1) + if is_ckpt_end: + checkpoint_chunk_indices[req_idx] = len(cu_chunk_seqlen) - 1 + else: + last_chunk_indices.append(len(cu_chunk_seqlen) - 1) cu_chunk_seqlen.append(seqlen_pos) - return cu_chunk_seqlen, seq_idx, last_chunk_indices + return cu_chunk_seqlen, seq_idx, last_chunk_indices, checkpoint_chunk_indices def _prefill_cpu_metadata( self, @@ -331,9 +349,13 @@ class BaseMambaAttentionMetadataBuilder(AttentionMetadataBuilder[M], abc.ABC): chunk_size: int, common: M, common_attn_metadata: CommonAttentionMetadata, - ) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]: + checkpoint_offsets_p: list[int] | None = None, + ) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor | None]: """Compute chunk metadata and return as device tensors. - Returns (cu_chunk_seqlen_p, seq_idx_p, last_chunk_indices_p). + Returns (cu_chunk_seqlen_p, seq_idx_p, last_chunk_indices_p, + checkpoint_chunk_idx_p). The last is None unless a row checkpoints; + it has one entry per prefill row, holding the logical chunk that ends + on the checkpoint, and 0 for rows that decline. """ num_prefills = common.num_prefills @@ -344,11 +366,14 @@ class BaseMambaAttentionMetadataBuilder(AttentionMetadataBuilder[M], abc.ABC): common.num_decode_tokens, ) - cu_chunk_seqlen, seq_idx, last_chunk_indices = self._compute_chunk_metadata( - chunk_size, - num_prefills, - num_computed_tokens_p_cpu, - query_start_loc_p_cpu, + cu_chunk_seqlen, seq_idx, last_chunk_indices, ckpt_idxs = ( + self._compute_chunk_metadata( + chunk_size, + num_prefills, + num_computed_tokens_p_cpu, + query_start_loc_p_cpu, + checkpoint_offsets_p, + ) ) device = common_attn_metadata.query_start_loc.device @@ -361,7 +386,14 @@ class BaseMambaAttentionMetadataBuilder(AttentionMetadataBuilder[M], abc.ABC): last_chunk_indices_p = async_tensor_h2d( last_chunk_indices, dtype=torch.int32, device=device ) - return cu_chunk_seqlen_p, seq_idx_p, last_chunk_indices_p + ckpt_idx_p = ( + async_tensor_h2d( + [max(idx, 0) for idx in ckpt_idxs], dtype=torch.int64, device=device + ) + if any(idx >= 0 for idx in ckpt_idxs) + else None + ) + return cu_chunk_seqlen_p, seq_idx_p, last_chunk_indices_p, ckpt_idx_p def _compute_common_metadata( self, diff --git a/vllm/v1/attention/backends/mla/compressor_utils.py b/vllm/v1/attention/backends/mla/compressor_utils.py index 25e24f347c..41d973f296 100644 --- a/vllm/v1/attention/backends/mla/compressor_utils.py +++ b/vllm/v1/attention/backends/mla/compressor_utils.py @@ -37,6 +37,8 @@ class CompressedSlotMappingKernel( compress_ratio: int triton_block_size: int block_size: int + dcp_world_size: int + cp_interleave: int @staticmethod @triton.jit(do_not_specialize=["block_table_stride"]) @@ -53,6 +55,8 @@ class CompressedSlotMappingKernel( block_table_ptr, block_table_stride, block_size, + dcp_world_size, + cp_interleave, COMPRESS_RATIO: tl.constexpr, PAD_ID: tl.constexpr, TRITON_BLOCK_SIZE: tl.constexpr, @@ -77,7 +81,14 @@ class CompressedSlotMappingKernel( slot_mapping_ptr + query_start + offset, mask=mask, other=PAD_ID ) is_valid = ((pos + 1) % COMPRESS_RATIO == 0) & (slot != PAD_ID) - pos_after_compress = pos // COMPRESS_RATIO + # Under DCP a PAD slot also marks tokens owned by other ranks; the + # interleave is a multiple of COMPRESS_RATIO, so a state lives + # entirely on the rank owning its last token. + local_pos = ( + pos // (cp_interleave * dcp_world_size) * cp_interleave + + pos % cp_interleave + ) + pos_after_compress = local_pos // COMPRESS_RATIO block_ids = pos_after_compress // block_size block_numbers = tl.load( @@ -97,11 +108,15 @@ class CompressedSlotMappingKernel( *, compress_ratio: int, block_size: int, + dcp_world_size: int, + cp_interleave: int, ) -> CompileKey: return self.CompileKey( compress_ratio=compress_ratio, triton_block_size=self.TRITON_BLOCK_SIZE, block_size=triton_scalar_specialization_rep(block_size), + dcp_world_size=triton_scalar_specialization_rep(dcp_world_size), + cp_interleave=triton_scalar_specialization_rep(cp_interleave), ) def get_warmup_keys(self, vllm_config: Any) -> list[CompileKey]: @@ -115,12 +130,15 @@ class CompressedSlotMappingKernel( ) if not compress_ratios: return [] + parallel_config = vllm_config.parallel_config return self._trace_dispatch(self.dispatch)( zip_inputs( *( dict( compress_ratio=ratio, block_size=vllm_config.cache_config.block_size // ratio, + dcp_world_size=parallel_config.decode_context_parallel_size, + cp_interleave=parallel_config.cp_kv_cache_interleave_size, ) for ratio in compress_ratios ) @@ -137,6 +155,8 @@ class CompressedSlotMappingKernel( block_table=int32_ptr, block_size=compile_key.block_size, compress_ratio=compile_key.compress_ratio, + dcp_world_size=compile_key.dcp_world_size, + cp_interleave=compile_key.cp_interleave, ) @kernel_launcher @@ -149,6 +169,8 @@ class CompressedSlotMappingKernel( block_table: torch.Tensor, block_size: int, compress_ratio: int, + dcp_world_size: int, + cp_interleave: int, ) -> LaunchSpec: return (block_table.shape[0],), dict( block_table_stride=block_table.stride(0), @@ -167,6 +189,8 @@ def get_compressed_slot_mapping( block_size: int, compress_ratio: int, out: torch.Tensor | None = None, + dcp_world_size: int = 1, + cp_interleave: int = 1, ) -> torch.Tensor: """Slot mapping for writing the compressed states of ``num_tokens`` tokens. @@ -196,6 +220,8 @@ def get_compressed_slot_mapping( block_table, block_size, compress_ratio, + dcp_world_size, + cp_interleave, ) return compressed_slot_mapping diff --git a/vllm/v1/attention/backends/mla/flashattn_mla_sparse.py b/vllm/v1/attention/backends/mla/flashattn_mla_sparse.py index 3ca88992aa..aabd682240 100644 --- a/vllm/v1/attention/backends/mla/flashattn_mla_sparse.py +++ b/vllm/v1/attention/backends/mla/flashattn_mla_sparse.py @@ -27,6 +27,7 @@ from vllm.v1.attention.backends.mla.sparse_utils import ( flat_kv_row_view, triton_convert_req_index_to_global_index, ) +from vllm.v1.attention.ops.metadata import compute_token_to_req_indices from vllm.v1.kv_cache_interface import AttentionSpec from vllm.vllm_flash_attn.flash_attn_interface import flash_attn_varlen_func @@ -132,8 +133,18 @@ class FlashAttnMLASparseMetadataBuilder( self._init_reorder_batch_threshold(threshold, supports_spec_as_decode=True) self.supports_draft_decode_metadata_update = self.dcp_world_size == 1 - def update_draft_decode_metadata(self, _metadata: SparseMLACommonMetadata) -> None: - pass + def update_draft_decode_metadata(self, metadata: SparseMLACommonMetadata) -> None: + num_tokens = metadata.num_decode_tokens + if num_tokens == 0: + return + # Everything else is a view of runner buffers; the per-token request + # map is a builder buffer that build() filled for the captured batch. + compute_token_to_req_indices( + metadata.query_start_loc, + metadata.req_id_per_token, + num_tokens, + num_tokens, + ) class FlashAttnMLASparseImpl(SparseMLACommonImpl[FlashAttnMLASparseMetadata]): diff --git a/vllm/v1/attention/backends/mla/flashinfer_mla_sparse.py b/vllm/v1/attention/backends/mla/flashinfer_mla_sparse.py index 05af2de797..4ebda50178 100644 --- a/vllm/v1/attention/backends/mla/flashinfer_mla_sparse.py +++ b/vllm/v1/attention/backends/mla/flashinfer_mla_sparse.py @@ -26,6 +26,7 @@ from vllm.v1.attention.backend import ( AttentionType, MLAAttentionImpl, MultipleOf, + max_decode_query_len, ) from vllm.v1.attention.backends.mla.index_group import HiSparseMLAIndexGroup from vllm.v1.attention.backends.mla.sparse_utils import ( @@ -34,7 +35,7 @@ from vllm.v1.attention.backends.mla.sparse_utils import ( triton_convert_req_index_to_global_index, triton_filter_and_convert_dcp_index, ) -from vllm.v1.kv_cache_interface import AttentionSpec +from vllm.v1.kv_cache_interface import AttentionSpec, KVCacheSpec if TYPE_CHECKING: from vllm.model_executor.models.deepseek_v2 import Indexer @@ -272,7 +273,16 @@ class FlashInferMLASparseMetadataBuilder( class FlashInferMLASparseTRTLLMMetadataBuilder(FlashInferMLASparseMetadataBuilder): """Metadata builder for the SM100 TRT-LLM sparse MLA kernel.""" - _cudagraph_support: ClassVar[AttentionCGSupport] = AttentionCGSupport.ALWAYS + _cudagraph_support: ClassVar[AttentionCGSupport] = AttentionCGSupport.UNIFORM_BATCH + + @classmethod + def get_varlen_cudagraph_max_query_len( + cls, + vllm_config: VllmConfig, + kv_cache_spec: KVCacheSpec, + ) -> int | None: + # Decode uses device request boundaries; prefill metadata is not graph-safe. + return max_decode_query_len(vllm_config) def __init__( self, diff --git a/vllm/v1/attention/backends/mla/flashmla_sparse.py b/vllm/v1/attention/backends/mla/flashmla_sparse.py index 1c3756e6c4..d441934c02 100644 --- a/vllm/v1/attention/backends/mla/flashmla_sparse.py +++ b/vllm/v1/attention/backends/mla/flashmla_sparse.py @@ -6,6 +6,7 @@ from typing import TYPE_CHECKING, ClassVar import numpy as np import torch import torch.distributed as dist +import torch.nn.functional as F from vllm import _custom_ops as ops from vllm.config import VllmConfig, get_current_vllm_config @@ -127,6 +128,9 @@ QUANTIZED_DS_MLA_CACHE_FORMATS: frozenset[str] = frozenset( {"fp8_ds_mla", "nvfp4_ds_mla"} ) +# RoPE dims in an fp8_ds_mla row (the bf16 tail of the 656B layout). +FP8_DS_MLA_ROPE_DIM = 64 + class FlashMLASparseBackend(AttentionBackend): supported_dtypes: ClassVar[list[torch.dtype]] = [torch.bfloat16] @@ -186,16 +190,16 @@ class FlashMLASparseBackend(AttentionBackend): ) -> str | None: if head_size == 512: # GLM5Next NoPE (qk_rope_head_dim == 0, kv_lora_rank == 512) has - # head_size 512 and is served here by the direct 512-wide bf16 - # cache on SM90. Quantized DS-MLA caches need the zero-padded - # 576/656B envelope, which is wired up separately; plain fp8, - # SM100 bf16, and rope-carrying 512 models must fall through to + # head_size 512 and is served here on SM90, either by the direct + # 512-wide bf16 cache or by the 576 fp8_ds_mla layout with a zero + # RoPE slot (see FlashMLASparseImpl.q_head_size). Plain fp8, + # SM100, and rope-carrying 512 models must fall through to # FlashInfer/TRITON. if ( - kv_cache_dtype in (None, "auto", "bfloat16", "float16") + kv_cache_dtype in (None, "auto", "bfloat16", "float16", "fp8_ds_mla") and device_capability.major == 9 ): - # Direct bf16 NoPE-512 is only correct for rope-free models. + # NoPE-512 is only correct for rope-free models. # Precedent for reading hf_text_config in supports_combination: # flashinfer_mla_sparse.py. from vllm.config import get_current_vllm_config @@ -210,8 +214,8 @@ class FlashMLASparseBackend(AttentionBackend): ) else: return ( - "FLASHMLA_SPARSE supports head_size 512 only with bf16 " - "kv-cache on SM90 (NoPE), got " + "FLASHMLA_SPARSE supports head_size 512 only with bf16 or " + "fp8_ds_mla kv-cache on SM90 (NoPE), got " f"kv_cache_dtype={kv_cache_dtype}, " f"capability={device_capability}" ) @@ -726,12 +730,20 @@ class FlashMLASparseImpl(SparseMLACommonImpl[FlashMLASparseMetadata]): **mla_args, ) self.softmax_scale = scale - # Prefill BF16 kernel requires 64 on Hopper, 128 on Blackwell - self.prefill_padding = ( - 128 if current_platform.is_device_capability_family(100) else 64 - ) + # Prefill BF16 kernel requires 64 heads on Hopper and Blackwell. + self.prefill_padding = 64 self.fp8_decode_padded_heads = self._compute_fp8_decode_padded_heads(num_heads) + # The kernels read queries as wide as the cache rows. A NoPE model on + # fp8_ds_mla keeps the DeepSeek 576 layout: the cache writer zero-fills + # the RoPE slot and forward_mqa zero-pads q_pe to match (q_pe . 0 = 0). + self.q_head_size = head_size + if kv_cache_dtype == "fp8_ds_mla" and self.qk_rope_head_dim == 0: + self.q_head_size = self.kv_lora_rank + FP8_DS_MLA_ROPE_DIM + # Dense-MHA prefill would gather the cache into a + # kv_lora_rank-wide workspace, too narrow for the 576 rows. + self.supports_dense_mha_prefill = False + vllm_config = get_current_vllm_config() max_tokens = vllm_config.scheduler_config.max_num_batched_tokens q_concat_heads = num_heads @@ -741,7 +753,7 @@ class FlashMLASparseImpl(SparseMLACommonImpl[FlashMLASparseMetadata]): // self.prefill_padding * self.prefill_padding ) - q_concat_shape = (max_tokens, q_concat_heads, head_size) + q_concat_shape = (max_tokens, q_concat_heads, self.q_head_size) if is_quantized_kv_cache(kv_cache_dtype): assert kv_cache_dtype in QUANTIZED_DS_MLA_CACHE_FORMATS, ( "FlashMLA Sparse Attention backend only supports the " @@ -777,11 +789,11 @@ class FlashMLASparseImpl(SparseMLACommonImpl[FlashMLASparseMetadata]): # PCP+DCP upconverts this rank's KV shard, then all-gathers the # shards into a workspace of the full prefill size. shard_rows //= parallel_config.decode_context_parallel_size - self.prefill_workspace_shape = (shard_rows, head_size) + self.prefill_workspace_shape = (shard_rows, self.q_head_size) self.workspace_specs.append((self.prefill_workspace_shape, torch.bfloat16)) if self.pcp_dcp_kv_gather: self.workspace_specs.append( - ((prefill_workspace_size, head_size), torch.bfloat16) + ((prefill_workspace_size, self.q_head_size), torch.bfloat16) ) prefill_query_heads = num_heads if self.pcp_dcp_kv_gather and self.dcp_world_size > self.pcp_world_size: @@ -792,7 +804,7 @@ class FlashMLASparseImpl(SparseMLACommonImpl[FlashMLASparseMetadata]): self.workspace_specs.extend( (shape, torch.bfloat16) for shape in ( - (max_tokens, padded_prefill_query_heads, head_size), + (max_tokens, padded_prefill_query_heads, self.q_head_size), (max_tokens, padded_prefill_query_heads, self.kv_lora_rank), (max_tokens, prefill_query_heads, self.kv_lora_rank), ) @@ -1309,9 +1321,9 @@ class FlashMLASparseImpl(SparseMLACommonImpl[FlashMLASparseMetadata]): -1, 1, kv_c_and_k_pe_cache.shape[-1] ) - # NOTE(Chen): kernel requires num_local_head to be a multiple of - # 64 on hopper and 128 on blackwell. Pad from q's head count, not - # self.num_heads: under DCP the heads are all-gathered before this. + # NOTE(Chen): kernel requires num_local_head to be a multiple of 64. + # Pad from q's head count, not self.num_heads: under DCP the heads are + # all-gathered before this. if actual_num_heads is None: actual_num_heads = q.shape[1] padded_num_heads = ( @@ -1370,11 +1382,16 @@ class FlashMLASparseImpl(SparseMLACommonImpl[FlashMLASparseMetadata]): if q_pe.size(-1) == 0: # NoPE (GLM5Next): concat_mla_q requires rope_dim == 64, # copy directly into the head-padded buffer instead. - q[:, : ql_nope.shape[1]].copy_(ql_nope) + q[:, : ql_nope.shape[1], : self.kv_lora_rank].copy_(ql_nope) + if self.q_head_size > self.kv_lora_rank: + q[..., self.kv_lora_rank :].zero_() else: ops.concat_mla_q(ql_nope, q_pe, q) else: actual_num_heads = q.shape[1] + if q.shape[-1] < self.q_head_size: + # DCP gathers the unpadded NoPE query. + q = F.pad(q, (0, self.q_head_size - q.shape[-1])) num_actual_toks = q.shape[0] diff --git a/vllm/v1/attention/backends/mla/index_group.py b/vllm/v1/attention/backends/mla/index_group.py index 5a18332e41..39f205b91d 100644 --- a/vllm/v1/attention/backends/mla/index_group.py +++ b/vllm/v1/attention/backends/mla/index_group.py @@ -233,7 +233,7 @@ class HiSparseMLAIndexGroup(SparseMLAIndexGroup): cache = self.cache(layer_index) num_tokens = logical_topk_indices.shape[0] req_id_per_token = attn_metadata.req_id_per_token[:num_tokens] - if num_tokens > self.physical_topk_indices.shape[0]: + if num_tokens > cache.runtime.max_swap_rows: # Prefill-sized batches do not fit the decode residency workspace. # Non-resident prefills are staged before reaching this path. assert cache.all_context_pages_resident diff --git a/vllm/v1/attention/backends/mla/indexer.py b/vllm/v1/attention/backends/mla/indexer.py index 5cfaf6ca5f..1005b8e424 100644 --- a/vllm/v1/attention/backends/mla/indexer.py +++ b/vllm/v1/attention/backends/mla/indexer.py @@ -554,7 +554,15 @@ class BuildPrefillChunkMetadataKernel( # Cover Triton's divisible, exact-one, and generic i32 classes. query_slice_start=(0, 1, 2), query_slice_stop=(1, 2 * max_tokens - 1, 2 * max_tokens), - DCP_INTERLEAVE=dcp_interleave, + # Compressed indexers localize in units of compressed states. + DCP_INTERLEAVE=list( + dict.fromkeys( + dcp_interleave // ratio + for ratio in compress_ratios + if dcp_interleave % ratio == 0 + ) + ) + or [dcp_interleave], BLOCK_SIZE=self.BLOCK_SIZE, COMPRESS_RATIO=list(compress_ratios), # PCP's global cumulative lengths are the second row of one packed @@ -670,6 +678,7 @@ class DeepseekV32IndexerMetadata: decode: DeepSeekV32IndexerDecodeMetadata | None = None prefill: DeepseekV32IndexerPrefillMetadata | None = None positions: torch.Tensor | None = None + block_table: torch.Tensor | None = None @triton.jit(do_not_specialize=["num_reqs", "num_actual_tokens", "num_tokens"]) @@ -693,11 +702,15 @@ def _kpool_tail_slot_mapping_kernel( end = tl.load(query_start_loc_ptr + pid + 1) end = tl.where(pid == num_reqs - 1, num_actual_tokens, end) own_block = tl.load(block_table_ptr + pid * block_table_stride).to(tl.int64) + # Every scheduled request owns a tail block; the runners give dummy and + # padding requests the null block, which must never be written. + has_block = own_block != 0 for i in range(start, end, BLOCK): offs = i + tl.arange(0, BLOCK) mask = offs < end pos = tl.load(positions_ptr + offs, mask=mask, other=0).to(tl.int64) - tl.store(out_ptr + offs, own_block * kpool + pos % kpool, mask=mask) + slot = tl.where(has_block, own_block * kpool + pos % kpool, -1) + tl.store(out_ptr + offs, slot, mask=mask) else: offs = num_actual_tokens + (pid - num_reqs) * BLOCK + tl.arange(0, BLOCK) mask = offs < num_tokens @@ -750,7 +763,8 @@ def compute_kpool_tail_slot_mapping( req = req.clamp_(min=0, max=num_reqs - 1) own_block = block_table[:num_reqs, 0].index_select(0, req).to(torch.int64) pos = positions[:num_actual_tokens].to(torch.int64) - out[:num_actual_tokens] = own_block * kpool + torch.remainder(pos, kpool) + slots = own_block * kpool + torch.remainder(pos, kpool) + out[:num_actual_tokens] = torch.where(own_block != 0, slots, -1) return out @@ -1010,11 +1024,31 @@ class DeepseekV32IndexerMetadataBuilder(AttentionMetadataBuilder): (self.num_sms + 1, 2), dtype=torch.int32, device=self.device ) - if self.dcp_world_size > 1 and self.compress_ratio > 1: - raise NotImplementedError( - "DCP is not supported with sparse indexer KV compression " - f"(compress_ratio={self.compress_ratio})." + # Each compressed state must live on a single rank, so the DCP + # interleave has to cover whole states. + if ( + self.dcp_world_size > 1 + and self.compress_ratio > 1 + and ( + ( + isinstance(self.kv_cache_spec, MLAAttentionSpec) + and self.kv_cache_spec.model_version == "deepseek_v4" + ) + or self.cp_kv_cache_interleave_size % self.compress_ratio != 0 ) + ): + raise NotImplementedError( + "DCP with sparse indexer KV compression " + f"(compress_ratio={self.compress_ratio}) requires " + "--cp-kv-cache-interleave-size to be a multiple of it; " + "DeepSeek-V4 is not supported." + ) + # DCP interleave in units of compressed states. + self.compressed_cp_interleave_size = ( + self.cp_kv_cache_interleave_size // self.compress_ratio + if self.dcp_world_size > 1 and self.compress_ratio > 1 + else self.cp_kv_cache_interleave_size + ) # Pre-allocate buffers for CUDA graph compatibility when if self.compress_ratio > 1: @@ -1348,6 +1382,8 @@ class DeepseekV32IndexerMetadataBuilder(AttentionMetadataBuilder): self.kv_cache_spec.num_states, self.compress_ratio, out=self.compressed_slot_mapping_buffer, + dcp_world_size=self.dcp_world_size, + cp_interleave=self.cp_kv_cache_interleave_size, ) if self.pcp_world_size > 1: compressed_slot_mapping = get_pcp_group().all_gather( @@ -1467,7 +1503,7 @@ class DeepseekV32IndexerMetadataBuilder(AttentionMetadataBuilder): skip_kv_gather=query_slice.start > 0, dcp_rank=self.dcp_rank, dcp_world_size=self.dcp_world_size, - cp_kv_cache_interleave_size=self.cp_kv_cache_interleave_size, + cp_kv_cache_interleave_size=self.compressed_cp_interleave_size, pcp_plan=pcp_plan, ) # Skip when total_seq_lens is 0 (i.e., no compressed token). @@ -1683,6 +1719,7 @@ class DeepseekV32IndexerMetadataBuilder(AttentionMetadataBuilder): num_prefill_tokens=num_prefill_tokens, prefill=prefill_metadata, decode=decode_metadata, + block_table=common_attn_metadata.block_table_tensor, ) return attn_metadata @@ -1698,8 +1735,28 @@ class DeepseekV32IndexerMetadataBuilder(AttentionMetadataBuilder): assert metadata.num_prefills == 0 assert metadata.num_decodes == metadata.num_decode_tokens assert decode.seq_lens.numel() == metadata.num_decode_tokens + assert metadata.block_table is not None assert self.dcp_world_size == 1 + # One query token per request, so the per-token decode block table is + # the request block table (at the indexer kernel's page size). build() + # may copy it into a builder buffer, so re-derive it for this batch. + block_table = metadata.block_table[: metadata.num_decode_tokens] + kernel_block_size = self.kernel_block_size + if ( + self.compress_ratio > 1 + and kernel_block_size is not None + and self.kv_cache_spec.block_size != kernel_block_size + and self.kv_cache_spec.block_size % kernel_block_size == 0 + ): + factor = self.kv_cache_spec.block_size // kernel_block_size + block_table = block_table[:, ::factor] // factor + decode.block_table.copy_(block_table) + if decode.indices is not None: + decode.indices.copy_(self.arange_buffer[: metadata.num_decode_tokens]) + if decode.per_req_decode_lens is not None: + decode.per_req_decode_lens.fill_(1) + if self.compress_ratio > 1: get_compressed_slot_mapping( metadata.num_decode_tokens, diff --git a/vllm/v1/attention/backends/mla/prefill/aiter_flash_attn.py b/vllm/v1/attention/backends/mla/prefill/aiter_flash_attn.py index 1d03b9e773..8b0e337bb2 100644 --- a/vllm/v1/attention/backends/mla/prefill/aiter_flash_attn.py +++ b/vllm/v1/attention/backends/mla/prefill/aiter_flash_attn.py @@ -31,7 +31,10 @@ class AiterFlashAttnPrefillBackend(MLAPrefillBackend): return "ROCM_AITER_FA" @classmethod - def supports_compute_capability(cls, device_capability: "DeviceCapability") -> bool: + def supports_compute_capability( + cls, + device_capability: "DeviceCapability | None", + ) -> bool: if not current_platform.is_rocm(): return False from vllm.platforms.rocm import on_mi3xx diff --git a/vllm/v1/attention/backends/mla/prefill/base.py b/vllm/v1/attention/backends/mla/prefill/base.py index 1b589cb295..4b4361214f 100644 --- a/vllm/v1/attention/backends/mla/prefill/base.py +++ b/vllm/v1/attention/backends/mla/prefill/base.py @@ -49,7 +49,10 @@ class MLAPrefillBackend(ABC): raise NotImplementedError @classmethod - def supports_compute_capability(cls, device_capability: "DeviceCapability") -> bool: + def supports_compute_capability( + cls, + device_capability: "DeviceCapability | None", + ) -> bool: return True @classmethod @@ -90,16 +93,19 @@ class MLAPrefillBackend(ABC): @classmethod def validate_configuration( cls, - device_capability: "DeviceCapability", + device_capability: "DeviceCapability | None", selector_config: "MLAPrefillSelectorConfig", ) -> list[str]: invalid_reasons: list[str] = [] if not cls.supports_compute_capability(device_capability): - invalid_reasons.append( - f"compute capability {device_capability.major}." - f"{device_capability.minor} not supported" - ) + if device_capability is None: + invalid_reasons.append("device capability not available") + else: + invalid_reasons.append( + f"compute capability {device_capability.major}." + f"{device_capability.minor} not supported" + ) if not cls.supports_dtype(selector_config.dtype): invalid_reasons.append(f"dtype {selector_config.dtype} not supported") diff --git a/vllm/v1/attention/backends/mla/prefill/flashinfer.py b/vllm/v1/attention/backends/mla/prefill/flashinfer.py index 777dd33756..5cf47ea2ca 100644 --- a/vllm/v1/attention/backends/mla/prefill/flashinfer.py +++ b/vllm/v1/attention/backends/mla/prefill/flashinfer.py @@ -51,8 +51,11 @@ class FlashInferPrefillBackend(MLAPrefillBackend): return "FLASHINFER" @classmethod - def supports_compute_capability(cls, device_capability: "DeviceCapability") -> bool: - return device_capability.major == 10 + def supports_compute_capability( + cls, + device_capability: "DeviceCapability | None", + ) -> bool: + return device_capability is not None and device_capability.major == 10 @classmethod def is_available(cls) -> bool: diff --git a/vllm/v1/attention/backends/mla/prefill/registry.py b/vllm/v1/attention/backends/mla/prefill/registry.py index f7ec31e153..19d37fbc22 100644 --- a/vllm/v1/attention/backends/mla/prefill/registry.py +++ b/vllm/v1/attention/backends/mla/prefill/registry.py @@ -10,11 +10,14 @@ from collections.abc import Callable from enum import Enum, EnumMeta from typing import TYPE_CHECKING +from vllm.logger import init_logger from vllm.utils.import_utils import resolve_obj_by_qualname if TYPE_CHECKING: from vllm.v1.attention.backends.mla.prefill.base import MLAPrefillBackend +logger = init_logger(__name__) + class _MLAPrefillBackendEnumMeta(EnumMeta): """Metaclass for MLAPrefillBackendEnum to provide better error messages.""" @@ -61,6 +64,37 @@ class MLAPrefillBackendEnum(Enum, metaclass=_MLAPrefillBackendEnumMeta): # set to None to avoid alias with other backend, whose value is an empty string CUSTOM = None + @classmethod + def register(cls, name: str, value: str) -> "MLAPrefillBackendEnum": + """Dynamically register a new MLA prefill backend enum member. + + Args: + name: The name for the new enum member + value: The fully qualified class path string + + Returns: + The newly created enum member + + """ + if name in cls._member_map_: + raise ValueError( + f"MLA prefill backend {name} already exists in {cls.__name__}. " + f"Use register_mla_prefill_backend(" + f"{cls.__name__}.{name}, '{value}') to override." + ) + if not name.isidentifier() or hasattr(cls, name): + raise ValueError(f"Invalid or reserved backend name: {name}") + + member = object.__new__(cls) + member._name_ = name + member._value_ = value + setattr(cls, name, member) + cls._member_map_[name] = member + cls._value2member_map_[value] = member + cls._member_names_.append(name) + logger.info("Registered new MLA prefill backend: %s -> %s", name, value) + return member + def get_path(self) -> str: """Get the class path for this backend (respects overrides). @@ -107,13 +141,15 @@ _MLA_PREFILL_OVERRIDES: dict[MLAPrefillBackendEnum, str] = {} def register_mla_prefill_backend( - backend: MLAPrefillBackendEnum, + backend: MLAPrefillBackendEnum | str, class_path: str | None = None, ) -> Callable[[type], type]: """Register or override an MLA prefill backend implementation. Args: - backend: The MLAPrefillBackendEnum member to register. + backend: The MLAPrefillBackendEnum member to register, + or a string name for a new custom backend + (e.g., "CUSTOM_MLA_PREFILL"). class_path: Optional class path. If not provided and used as decorator, will be auto-generated from the class. @@ -131,13 +167,27 @@ def register_mla_prefill_backend( class MyCustomPrefillBackend(MLAPrefillBackend): ... + # Register a new custom MLA prefill backend with a dynamic enum name + @register_mla_prefill_backend("CUSTOM_MLA_PREFILL") + class CustomMLAPrefillBackend(MLAPrefillBackend): + ... + # Direct registration register_mla_prefill_backend( MLAPrefillBackendEnum.CUSTOM, "my.module.MyCustomPrefillBackend" ) + # Direct registration with string name + register_mla_prefill_backend( + "CUSTOM_MLA_PREFILL", + "custom.attention.mla_prefill.CustomMLAPrefillBackend" + ) + """ + # Handle dynamic enum creation for string backend names + if isinstance(backend, str): + backend = MLAPrefillBackendEnum.register(backend, class_path or "") def decorator(cls: type) -> type: _MLA_PREFILL_OVERRIDES[backend] = f"{cls.__module__}.{cls.__qualname__}" diff --git a/vllm/v1/attention/backends/mla/prefill/selector.py b/vllm/v1/attention/backends/mla/prefill/selector.py index 0cb29f51b4..f0f0015809 100644 --- a/vllm/v1/attention/backends/mla/prefill/selector.py +++ b/vllm/v1/attention/backends/mla/prefill/selector.py @@ -12,9 +12,7 @@ from typing import TYPE_CHECKING, NamedTuple import torch from vllm.logger import init_logger -from vllm.platforms.interface import DeviceCapability from vllm.v1.attention.backends.mla.prefill.base import MLADimensions -from vllm.v1.attention.backends.mla.prefill.registry import MLAPrefillBackendEnum if TYPE_CHECKING: from vllm.config import VllmConfig @@ -45,54 +43,8 @@ class MLAPrefillSelectorConfig(NamedTuple): ) -def _get_mla_prefill_backend_priorities( - device_capability: DeviceCapability, - mla_dimensions: MLADimensions, -) -> list[MLAPrefillBackendEnum]: - """Get MLA prefill backend priorities based on device capability. - - Args: - device_capability: The device's compute capability. - mla_dimensions: The model's MLA head dimensions. - - Returns: - List of backends in priority order (highest priority first). - - """ - from vllm.platforms import current_platform - - if current_platform.is_rocm(): - return [ - MLAPrefillBackendEnum.ROCM_AITER_FA, - MLAPrefillBackendEnum.FLASH_ATTN, - ] - - if device_capability.major == 10: # Blackwell - if mla_dimensions == MLADimensions( - qk_nope_head_dim=192, - qk_rope_head_dim=64, - v_head_dim=256, - ): - return [ - MLAPrefillBackendEnum.TRTLLM_RAGGED, - MLAPrefillBackendEnum.FLASH_ATTN, - MLAPrefillBackendEnum.FLASHINFER, - MLAPrefillBackendEnum.TOKENSPEED_MLA, - ] - return [ - MLAPrefillBackendEnum.FLASH_ATTN, - MLAPrefillBackendEnum.TRTLLM_RAGGED, - MLAPrefillBackendEnum.FLASHINFER, - MLAPrefillBackendEnum.TOKENSPEED_MLA, - ] - else: # Hopper (SM90) and older - return [ - MLAPrefillBackendEnum.FLASH_ATTN, - ] - - def get_mla_prefill_backend( - vllm_config: "VllmConfig", + vllm_config: "VllmConfig | None", ) -> "type[MLAPrefillBackend]": """Select the MLA prefill backend based on configuration and device. @@ -101,7 +53,8 @@ def get_mla_prefill_backend( priority-based selection. Args: - vllm_config: The vLLM configuration. + vllm_config: The vLLM configuration. May be None before model + resolution, in which case defaults are used. Returns: The selected prefill backend class. @@ -109,30 +62,7 @@ def get_mla_prefill_backend( """ from vllm.platforms import current_platform - if current_platform.is_cpu(): - # CPUs have no compute capability, so the capability-driven priority - # path below does not apply. Prefer an accelerator-specific CPU backend - # when its kernels are present, else the generic SDPA one. - for backend_enum in (MLAPrefillBackendEnum.ZEN_CPU, MLAPrefillBackendEnum.CPU): - try: - cpu_backend_cls = backend_enum.get_class() - except ImportError: - continue - if cpu_backend_cls.is_available(): - logger.info_once("Using %s MLA prefill backend.", backend_enum.name) - return cpu_backend_cls - raise ValueError("No valid CPU MLA prefill backend found.") - - device_capability = current_platform.get_device_capability() - if device_capability is None: - logger.info_once( - "Device capability not available, using FlashAttention MLA prefill backend." - ) - return MLAPrefillBackendEnum.FLASH_ATTN.get_class() - - attention_config = vllm_config.attention_config - - model_config = vllm_config.model_config + model_config = vllm_config.model_config if vllm_config is not None else None if model_config is None: selector_config = MLAPrefillSelectorConfig(dtype=torch.get_default_dtype()) else: @@ -146,11 +76,21 @@ def get_mla_prefill_backend( ), ) - if attention_config.mla_prefill_backend is not None: + if current_platform.is_cpu(): + # CPUs have a single fixed MLA prefill backend, so explicit backend + # configuration does not apply; always auto-select via the platform. + return _auto_select_mla_prefill_backend(selector_config) + + attention_config = vllm_config.attention_config if vllm_config is not None else None + if ( + attention_config is not None + and attention_config.mla_prefill_backend is not None + ): selected_backend = attention_config.mla_prefill_backend backend_cls: type[MLAPrefillBackend] | None = None try: backend_cls = selected_backend.get_class() + device_capability = current_platform.get_device_capability() invalid_reasons = backend_cls.validate_configuration( device_capability, selector_config ) @@ -166,63 +106,27 @@ def get_mla_prefill_backend( logger.info_once("Using %s MLA prefill backend.", selected_backend.name) return backend_cls - return _auto_select_mla_prefill_backend( - device_capability, - selector_config, - ) + return _auto_select_mla_prefill_backend(selector_config) @cache def _auto_select_mla_prefill_backend( - device_capability: DeviceCapability, selector_config: MLAPrefillSelectorConfig, ) -> "type[MLAPrefillBackend]": """Auto-select the best available MLA prefill backend. Args: - device_capability: The device's compute capability. selector_config: Hashable configuration for backend selection. Returns: The selected prefill backend class. + Raises: + ValueError: If the platform has no valid MLA prefill backend. + """ - priorities = _get_mla_prefill_backend_priorities( - device_capability, - selector_config.mla_dimensions, - ) - all_invalid_reasons: dict[str, list[str]] = {} + from vllm.platforms import current_platform - for backend_enum in priorities: - backend_cls: type[MLAPrefillBackend] | None = None - try: - backend_cls = backend_enum.get_class() - invalid_reasons = backend_cls.validate_configuration( - device_capability, selector_config - ) - except ImportError: - invalid_reasons = ["ImportError"] - if not invalid_reasons: - assert backend_cls is not None - logger.info_once("Using %s MLA prefill backend.", backend_enum.name) - return backend_cls - all_invalid_reasons[backend_enum.name] = invalid_reasons - - reasons_str = ( - "{" - + ", ".join( - f"{name}: [{', '.join(reasons)}]" - for name, reasons in all_invalid_reasons.items() - ) - + "}" - ) - config_str = repr(selector_config) - logger.debug_once( - "Some MLA prefill backends are not valid with %s. Reasons: %s.", - config_str, - reasons_str, - ) - - raise ValueError( - f"No valid MLA prefill backend found with {config_str}. Reasons: {reasons_str}." - ) + selected_backend = current_platform.get_mla_prefill_backend_cls(selector_config) + logger.info_once("Using %s MLA prefill backend.", selected_backend.get_name()) + return selected_backend diff --git a/vllm/v1/attention/backends/mla/prefill/tokenspeed_mla.py b/vllm/v1/attention/backends/mla/prefill/tokenspeed_mla.py index 6985b0a70c..d65721b81b 100644 --- a/vllm/v1/attention/backends/mla/prefill/tokenspeed_mla.py +++ b/vllm/v1/attention/backends/mla/prefill/tokenspeed_mla.py @@ -35,8 +35,11 @@ class TokenspeedMLAPrefillBackend(MLAPrefillBackend): return "TOKENSPEED_MLA" @classmethod - def supports_compute_capability(cls, device_capability: "DeviceCapability") -> bool: - return device_capability.major == 10 + def supports_compute_capability( + cls, + device_capability: "DeviceCapability | None", + ) -> bool: + return device_capability is not None and device_capability.major == 10 _INSTALL_HINT = ( "tokenspeed_mla package is not installed. " diff --git a/vllm/v1/attention/backends/mla/prefill/trtllm_ragged.py b/vllm/v1/attention/backends/mla/prefill/trtllm_ragged.py index 167738b07a..3cb498be2b 100644 --- a/vllm/v1/attention/backends/mla/prefill/trtllm_ragged.py +++ b/vllm/v1/attention/backends/mla/prefill/trtllm_ragged.py @@ -43,8 +43,11 @@ class TrtllmRaggedPrefillBackend(MLAPrefillBackend): return "TRTLLM_RAGGED" @classmethod - def supports_compute_capability(cls, device_capability: "DeviceCapability") -> bool: - return device_capability.major == 10 + def supports_compute_capability( + cls, + device_capability: "DeviceCapability | None", + ) -> bool: + return device_capability is not None and device_capability.major == 10 @classmethod def is_available(cls) -> bool: diff --git a/vllm/v1/attention/backends/mla/rocm_aiter_mla.py b/vllm/v1/attention/backends/mla/rocm_aiter_mla.py index c833a56a83..32154105a9 100644 --- a/vllm/v1/attention/backends/mla/rocm_aiter_mla.py +++ b/vllm/v1/attention/backends/mla/rocm_aiter_mla.py @@ -601,13 +601,14 @@ class AiterMLAMetadataBuilder(MLACommonMetadataBuilder[AiterMLAMetadata]): parallel_config.decode_context_parallel_size, parallel_config.cp_kv_cache_interleave_size, ) + verify_route = envs.VLLM_ROCM_AITER_MLA_DCP_VERIFY.lower() asm_dcp_verify_config = ( _asm_dcp_verify_configured( parallel_config.decode_context_parallel_size, parallel_config.cp_kv_cache_interleave_size, multi_token_decode=vllm_config.speculative_config is not None, ) - and envs.VLLM_ROCM_AITER_MLA_DCP_VERIFY == "asm" + and verify_route != "segmented" ) super().__init__( kv_cache_spec, @@ -627,14 +628,20 @@ class AiterMLAMetadataBuilder(MLACommonMetadataBuilder[AiterMLAMetadata]): self._asm_dcp_verify_heads = _asm_dcp_verify_heads( self.num_heads * self.dcp_world_size ) - if not self._asm_dcp_verify_heads: + if self._asm_dcp_verify_heads: + self._asm_dcp_verify = True + elif verify_route == "asm" or not supports_segmented_dcp_verify: raise ValueError( - "VLLM_ROCM_AITER_MLA_DCP_VERIFY=asm, but the round-robin asm " - f"decode has no kernel for {self.num_heads * self.dcp_world_size} " - f"DCP-gathered heads (native counts: {_NATIVE_CPRR_HEADS}). " - "Set VLLM_ROCM_AITER_MLA_DCP_VERIFY=segmented." + f"VLLM_ROCM_AITER_MLA_DCP_VERIFY={verify_route}, but the " + "round-robin asm decode has no kernel for " + f"{self.num_heads * self.dcp_world_size} DCP-gathered heads " + f"(native counts: {_NATIVE_CPRR_HEADS}). " + + ( + "Set VLLM_ROCM_AITER_MLA_DCP_VERIFY=segmented." + if supports_segmented_dcp_verify + else "This AITER build also lacks segmented MLA decode." + ) ) - self._asm_dcp_verify = True self._supports_segmented_dcp_verify = supports_segmented_dcp_verify self._mla_max_split_per_batch = 0 if self._asm_dcp_verify: diff --git a/vllm/v1/attention/backends/mla/rocm_aiter_mla_sparse.py b/vllm/v1/attention/backends/mla/rocm_aiter_mla_sparse.py index e473f85948..e3d982e882 100644 --- a/vllm/v1/attention/backends/mla/rocm_aiter_mla_sparse.py +++ b/vllm/v1/attention/backends/mla/rocm_aiter_mla_sparse.py @@ -37,7 +37,9 @@ from vllm.v1.attention.backends.mla.rocm_aiter_mla import ( ) from vllm.v1.attention.backends.utils import split_decodes_and_prefills from vllm.v1.attention.ops.rocm_aiter_mla_sparse import ( + rocm_sparse_attn_decode_bf16, rocm_sparse_attn_prefill, + rocm_sparse_decode_bf16_num_splits, ) from vllm.v1.kv_cache_interface import AttentionSpec, KVCacheLayout from vllm.v1.worker.workspace import current_workspace_manager @@ -904,20 +906,45 @@ class ROCMAiterMLASparseImpl( self.sinks.reshape(1, self.num_heads, 1), q.shape[1], ).reshape(-1) - rocm_sparse_attn_prefill( - q=q, - kv=kv_c_and_k_pe_cache.view(-1, 1, q.shape[-1]), - indices=None, - topk_length=None, - scale=self.scale, - head_dim=q.shape[-1], - nope_head_dim=self.kv_lora_rank, - rope_head_dim=q.shape[-1] - self.kv_lora_rank, - attn_sink=triton_sinks, - output=output, - ragged_indices=attn_metadata.paged_kv_indices, - ragged_indptr=attn_metadata.paged_kv_indptr, + kv = kv_c_and_k_pe_cache.view(-1, 1, q.shape[-1]) + decode_num_splits = ( + rocm_sparse_decode_bf16_num_splits( + num_tokens, + q.shape[1], + min(attn_metadata.max_seq_len, attn_metadata.topk_tokens), + ) + if attn_metadata.num_decode_tokens == num_tokens + else 1 ) + if decode_num_splits > 1: + rocm_sparse_attn_decode_bf16( + q=q, + kv=kv, + scale=self.scale, + head_dim=q.shape[-1], + nope_head_dim=self.kv_lora_rank, + rope_head_dim=q.shape[-1] - self.kv_lora_rank, + attn_sink=triton_sinks, + output=output, + ragged_indices=attn_metadata.paged_kv_indices, + ragged_indptr=attn_metadata.paged_kv_indptr, + num_splits=decode_num_splits, + ) + else: + rocm_sparse_attn_prefill( + q=q, + kv=kv, + indices=None, + topk_length=None, + scale=self.scale, + head_dim=q.shape[-1], + nope_head_dim=self.kv_lora_rank, + rope_head_dim=q.shape[-1] - self.kv_lora_rank, + attn_sink=triton_sinks, + output=output, + ragged_indices=attn_metadata.paged_kv_indices, + ragged_indptr=attn_metadata.paged_kv_indptr, + ) output = AiterMLAHelper.get_mla_unpadded_o(self.num_heads, output) return output, None diff --git a/vllm/v1/attention/backends/mla/sparse_swa.py b/vllm/v1/attention/backends/mla/sparse_swa.py index 0f5bff4144..c5cbb4ef28 100644 --- a/vllm/v1/attention/backends/mla/sparse_swa.py +++ b/vllm/v1/attention/backends/mla/sparse_swa.py @@ -33,6 +33,7 @@ from vllm.v1.attention.backends.mla.compressor_utils import ( ) from vllm.v1.attention.backends.utils import split_decodes_and_prefills from vllm.v1.attention.ops.flashmla import FlashMLASchedMeta, get_mla_metadata +from vllm.v1.attention.ops.metadata import compute_token_to_req_indices from vllm.v1.kv_cache_interface import ( KVCacheSpec, MLAAttentionSpec, @@ -829,7 +830,8 @@ class DeepseekSparseSWAMetadataBuilder(AttentionMetadataBuilder): self, metadata: DeepseekSparseSWAMetadata, ) -> None: - if metadata.num_decode_tokens == 0: + num_tokens = metadata.num_decode_tokens + if num_tokens == 0: return assert metadata.query_start_loc is not None assert metadata.seq_lens is not None @@ -839,6 +841,17 @@ class DeepseekSparseSWAMetadataBuilder(AttentionMetadataBuilder): assert metadata.decode_swa_lens is not None assert metadata.replay_start is not None + # Recompute the validity mask and token->request map from the device + # buffers. Their padding differs from the dummy batch the graph was + # captured with. + torch.ge(metadata.slot_mapping, 0, out=metadata.is_valid_token) + compute_token_to_req_indices( + metadata.query_start_loc, + metadata.token_to_req_indices, + num_tokens, + num_tokens, + ) + _COMPUTE_SWA_INDICES_AND_LENS_KERNEL( metadata.decode_swa_indices, metadata.decode_swa_lens, diff --git a/vllm/v1/attention/backends/registry.py b/vllm/v1/attention/backends/registry.py index 32e2fabeb5..207c112a38 100644 --- a/vllm/v1/attention/backends/registry.py +++ b/vllm/v1/attention/backends/registry.py @@ -53,6 +53,9 @@ class AttentionBackendEnum(Enum, metaclass=_AttentionBackendEnumMeta): "vllm.v1.attention.backends.triton_flash_attn.TritonFlashAttentionBackend" ) ROCM_ATTN = "vllm.v1.attention.backends.rocm_attn.RocmAttentionBackend" + ROCM_SEGMENTED_ATTN = ( + "vllm.v1.attention.backends.rocm_segmented_attn.RocmSegmentedAttentionBackend" + ) ROCM_AITER_MLA = "vllm.v1.attention.backends.mla.rocm_aiter_mla.AiterMLABackend" ROCM_AITER_TRITON_MLA = ( "vllm.v1.attention.backends.mla.aiter_triton_mla.AiterTritonMLABackend" @@ -149,10 +152,43 @@ class AttentionBackendEnum(Enum, metaclass=_AttentionBackendEnumMeta): CPU_MLA = "vllm.v1.attention.backends.mla.cpu_mla.CPUMLABackend" AMX_MLA = "vllm.v1.attention.backends.mla.amx_mla.AMXMLABackend" TURBOQUANT = "vllm.v1.attention.backends.turboquant_attn.TurboQuantAttentionBackend" + ULTRAQUANT = "vllm.v1.attention.backends.ultraquant_attn.UltraQuantAttentionBackend" # Placeholder for third-party/custom backends - must be registered before use # set to None to avoid alias with other backend, whose value is an empty string CUSTOM = None + @classmethod + def register(cls, name: str, value: str) -> "AttentionBackendEnum": + """Dynamically register a new backend enum member. + + Args: + name: The name for the new enum member + value: The fully qualified class path string + + Returns: + The newly created enum member + + """ + if name in cls._member_map_: + raise ValueError( + f"Backend {name} already exists in {cls.__name__}. " + f"Use register_backend({cls.__name__}.{name}, '{value}') " + f"to override." + ) + if not name.isidentifier() or hasattr(cls, name): + raise ValueError(f"Invalid or reserved backend name: {name}") + + # Create new enum member dynamically + member = object.__new__(cls) + member._name_ = name + member._value_ = value + setattr(cls, name, member) + cls._member_map_[name] = member + cls._value2member_map_[value] = member + cls._member_names_.append(name) + logger.info("Registered new attention backend: %s -> %s", name, value) + return member + def get_path(self, include_classname: bool = True) -> str: """Get the class path for this backend (respects overrides). @@ -210,6 +246,37 @@ class MambaAttentionBackendEnum(Enum, metaclass=_AttentionBackendEnumMeta): backend.get_class() """ + @classmethod + def register(cls, name: str, value: str) -> "MambaAttentionBackendEnum": + """Dynamically register a new mamba backend enum member. + + Args: + name: The name for the new enum member + value: The fully qualified class path string + + Returns: + The newly created enum member + + """ + if name in cls._member_map_: + raise ValueError( + f"Backend {name} already exists in {cls.__name__}. " + f"Use register_backend({cls.__name__}.{name}, '{value}') " + f"to override." + ) + if not name.isidentifier() or hasattr(cls, name): + raise ValueError(f"Invalid or reserved backend name: {name}") + + member = object.__new__(cls) + member._name_ = name + member._value_ = value + setattr(cls, name, member) + cls._member_map_[name] = member + cls._value2member_map_[value] = member + cls._member_names_.append(name) + logger.info("Registered new attention backend: %s -> %s", name, value) + return member + MAMBA1 = "vllm.v1.attention.backends.mamba1_attn.Mamba1AttentionBackend" MAMBA2 = "vllm.v1.attention.backends.mamba2_attn.Mamba2AttentionBackend" SHORT_CONV = "vllm.v1.attention.backends.short_conv_attn.ShortConvAttentionBackend" @@ -271,14 +338,15 @@ _MAMBA_ATTN_OVERRIDES: dict[MambaAttentionBackendEnum, str] = {} def register_backend( - backend: AttentionBackendEnum | MambaAttentionBackendEnum, + backend: AttentionBackendEnum | MambaAttentionBackendEnum | str, class_path: str | None = None, is_mamba: bool = False, ) -> Callable[[type], type]: """Register or override a backend implementation. Args: - backend: The AttentionBackendEnum member to register + backend: The AttentionBackendEnum/MambaAttentionBackendEnum member to register, + or a string name for a new custom backend (e.g., "CUSTOM_MLA"). class_path: Optional class path. If not provided and used as decorator, will be auto-generated from the class. is_mamba: Whether the backend is a Mamba attention backend. @@ -302,13 +370,30 @@ def register_backend( class MyCustomBackend: ... + # Register a new custom attention backend with a dynamic enum name + @register_backend("CUSTOM_MLA") + class CustomMLABackend: + ... + # Direct registration register_backend( AttentionBackendEnum.CUSTOM, "my.module.MyCustomBackend" ) + # Direct registration with string name + register_backend( + "CUSTOM_MLA", + "custom.attention.mla_v1.CustomMLAABackend" + ) + """ + # Handle dynamic enum creation for string backend names + if isinstance(backend, str): + if is_mamba: + backend = MambaAttentionBackendEnum.register(backend, class_path or "") + else: + backend = AttentionBackendEnum.register(backend, class_path or "") def decorator(cls: type) -> type: if is_mamba: diff --git a/vllm/v1/attention/backends/rocm_aiter_unified_attn.py b/vllm/v1/attention/backends/rocm_aiter_unified_attn.py index d57b420cd7..0d4fd55194 100644 --- a/vllm/v1/attention/backends/rocm_aiter_unified_attn.py +++ b/vllm/v1/attention/backends/rocm_aiter_unified_attn.py @@ -184,8 +184,11 @@ class RocmAiterUnifiedAttentionImpl(RocmAttentionImpl): ) from aiter.ops.triton.unified_attention import unified_attention + from vllm.platforms.rocm import on_gfx11 + self.unified_attention = unified_attention - self.supports_quant_query_input = True + # RDNA3 has no low precision MMA, preserve query precision + self.supports_quant_query_input = not on_gfx11() def _split_kv_cache( self, kv_cache: torch.Tensor diff --git a/vllm/v1/attention/backends/rocm_segmented_attn.py b/vllm/v1/attention/backends/rocm_segmented_attn.py new file mode 100644 index 0000000000..c541eed451 --- /dev/null +++ b/vllm/v1/attention/backends/rocm_segmented_attn.py @@ -0,0 +1,371 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""ROCm attention backend for token-major segmented Triton attention.""" + +import os +from pathlib import Path +from typing import TYPE_CHECKING, ClassVar + +import torch + +import vllm.envs as envs +from vllm._aiter_ops import rocm_aiter_ops +from vllm.config import VllmConfig, get_current_vllm_config +from vllm.config.cache import CacheDType +from vllm.logger import init_logger +from vllm.platforms import current_platform +from vllm.utils.torch_utils import is_quantized_kv_cache +from vllm.v1.attention.backend import AttentionLayer, AttentionType +from vllm.v1.attention.backends.rocm_attn import ( + RocmAttentionBackend, + RocmAttentionImpl, + RocmAttentionMetadata, + RocmAttentionMetadataBuilder, +) +from vllm.v1.attention.backends.utils import get_num_attention_heads_from_layers +from vllm.v1.attention.ops.segmented_attention import ( + get_segmented_attention_workspace, + segmented_attention, + segmented_attention_workspace_size, +) +from vllm.v1.attention.ops.triton_reshape_and_cache_flash import ( + triton_reshape_and_cache_flash, +) +from vllm.v1.kv_cache_interface import AttentionSpec + +if TYPE_CHECKING: + from vllm.platforms.interface import DeviceCapability + +logger = init_logger(__name__) + + +def is_rdna() -> bool: + if not current_platform.is_rocm(): + return False + from vllm.platforms.rocm import on_gfx1x + + return on_gfx1x() + + +class RocmSegmentedAttentionMetadataBuilder(RocmAttentionMetadataBuilder): + def __init__( + self, + kv_cache_spec: AttentionSpec, + layer_names: list[str], + vllm_config: VllmConfig, + device: torch.device, + ): + super().__init__(kv_cache_spec, layer_names, vllm_config, device) + model_config = vllm_config.model_config + self.num_heads_q = get_num_attention_heads_from_layers( + vllm_config, layer_names + ) or model_config.get_num_attention_heads(vllm_config.parallel_config) + self.num_heads_kv = kv_cache_spec.num_kv_heads + self.headdim = kv_cache_spec.head_size + + +class RocmSegmentedAttentionBackend(RocmAttentionBackend): + """Explicit token-major ROCm backend optimized for segmented prefill.""" + + supported_dtypes: ClassVar[list[torch.dtype]] = [ + torch.float16, + torch.bfloat16, + ] + supported_kv_cache_dtypes: ClassVar[list[CacheDType]] = [ + "auto", + "float16", + "bfloat16", + "fp8", + "fp8_e4m3", + ] + + @staticmethod + def get_name() -> str: + return "ROCM_SEGMENTED_ATTN" + + @staticmethod + def get_impl_cls() -> type["RocmSegmentedAttentionImpl"]: + return RocmSegmentedAttentionImpl + + @staticmethod + def get_builder_cls() -> type["RocmSegmentedAttentionMetadataBuilder"]: + return RocmSegmentedAttentionMetadataBuilder + + @classmethod + def get_supported_head_sizes(cls) -> list[int]: + return [64, 128, 256] + + @classmethod + def supports_attn_type(cls, attn_type: str) -> bool: + return attn_type == AttentionType.DECODER + + @classmethod + def supports_non_causal(cls) -> bool: + return True + + @classmethod + def supports_sliding_window(cls) -> bool: + return True + + @classmethod + def supports_sink(cls) -> bool: + return True + + @classmethod + def supports_mm_prefix(cls) -> bool: + return False + + @classmethod + def supports_combination( + cls, + head_size: int, + dtype: torch.dtype, + kv_cache_dtype: CacheDType | None, + block_size: int | None, + use_mla: bool, + has_sink: bool, + use_sparse: bool, + use_mm_prefix: bool, + device_capability: "DeviceCapability", + ) -> str | None: + del ( + head_size, + dtype, + block_size, + use_mla, + has_sink, + use_sparse, + use_mm_prefix, + device_capability, + ) + if not is_rdna(): + return "ROCM_SEGMENTED_ATTN requires AMD RDNA GPUs on ROCm" + + from vllm.platforms.rocm import on_gfx12x + + if kv_cache_dtype in ("fp8", "fp8_e4m3") and not on_gfx12x(): + return "FP8 segmented attention requires gfx12" + return None + + +class RocmSegmentedAttentionImpl(RocmAttentionImpl): + def __init__( + self, + num_heads: int, + head_size: int, + scale: float, + num_kv_heads: int, + alibi_slopes: list[float] | None, + sliding_window: int | None, + kv_cache_dtype: str, + logits_soft_cap: float | None = None, + attn_type: AttentionType = AttentionType.DECODER, + kv_sharing_target_layer_name: int | None = None, + sinks: torch.Tensor | None = None, + ) -> None: + if not is_rdna(): + raise ValueError("ROCM_SEGMENTED_ATTN requires AMD RDNA GPUs on ROCm") + + super().__init__( + num_heads, + head_size, + scale, + num_kv_heads, + alibi_slopes, + sliding_window, + kv_cache_dtype, + logits_soft_cap, + attn_type, + kv_sharing_target_layer_name, + sinks, + ) + from vllm.platforms.rocm import on_gfx1x, on_gfx12x + + fp8 = kv_cache_dtype in ("fp8", "fp8_e4m3") + arch_supported = on_gfx12x() if fp8 else on_gfx1x() + gqa_ratio_supported = ( + num_kv_heads > 0 + and num_heads % num_kv_heads == 0 + and 1 <= self.num_queries_per_kv <= 16 + ) + if not ( + arch_supported + and attn_type == AttentionType.DECODER + and head_size in (64, 128, 256) + and gqa_ratio_supported + and kv_cache_dtype in ("auto", "float16", "bfloat16", "fp8", "fp8_e4m3") + and alibi_slopes is None + ): + raise ValueError( + "ROCM_SEGMENTED_ATTN requires decoder attention on gfx1x " + "(gfx12 for FP8), head size 64, 128 or 256, GQA ratio 1-16, and " + "does not support ALiBi." + ) + # Startup tuning and target/draft compilation must share a Triton cache. + cache_dir = os.environ.setdefault( + "TRITON_CACHE_DIR", + str(Path(envs.VLLM_CACHE_ROOT) / "rocm_segmented_attention" / "triton"), + ) + Path(cache_dir).mkdir(parents=True, exist_ok=True) + logger.info_once("Using token-major ROCm segmented Triton attention") + self._segmented_attention_warmed_up = False + self._segmented_attention_config: VllmConfig | None = None + + def process_weights_after_loading(self, act_dtype: torch.dtype): + super().process_weights_after_loading(act_dtype) + config = get_current_vllm_config() + self._segmented_attention_config = config + + def _autotune_enabled(self) -> bool: + config = self._segmented_attention_config + return ( + config is not None + and config.kernel_config.enable_rocm_segmented_attn_autotune + ) + + def _get_workspace(self, device): + config = self._segmented_attention_config + assert config is not None + sizes = segmented_attention_workspace_size( + config.scheduler_config.max_num_seqs, + self.num_heads, + self.num_kv_heads, + self.head_size, + config.model_config.max_model_len, + max_tokens=config.scheduler_config.max_num_batched_tokens, + fp8=self.kv_cache_dtype in ("fp8", "fp8_e4m3"), + autotune=self._autotune_enabled(), + ) + return get_segmented_attention_workspace(device, sizes) + + def _split_kv_cache( + self, + kv_cache: torch.Tensor, + num_kv_heads: int | None = None, + head_size: int | None = None, + ) -> tuple[torch.Tensor, torch.Tensor]: + num_kv_heads = num_kv_heads or self.num_kv_heads + head_size = head_size or self.head_size + return ( + kv_cache[:, 0].unflatten(-1, (num_kv_heads, head_size)), + kv_cache[:, 1].unflatten(-1, (num_kv_heads, head_size)), + ) + + def forward( + self, + layer: torch.nn.Module, + query: torch.Tensor, + key: torch.Tensor, + value: torch.Tensor, + kv_cache: torch.Tensor, + attn_metadata: RocmAttentionMetadata, + output: torch.Tensor, + output_scale: torch.Tensor | None = None, + output_block_scale: torch.Tensor | None = None, + ) -> torch.Tensor: + if output_block_scale is not None: + raise NotImplementedError( + "fused block_scale output quantization is not supported" + ) + # Profile runs allocate the same fixed scratch used during graph replay. + workspace = self._get_workspace(query.device) + if attn_metadata is None: + return output.fill_(0) + assert not attn_metadata.use_cascade + + num_actual_tokens = attn_metadata.num_actual_tokens + key_cache, value_cache = self._split_kv_cache(kv_cache) + if is_quantized_kv_cache(self.kv_cache_dtype): + key_cache = key_cache.view(self.fp8_dtype) + value_cache = value_cache.view(self.fp8_dtype) + if query.dtype == self.fp8_dtype and layer._q_scale_float != 1.0: + raise NotImplementedError( + "A non-1.0 q_scale with an FP8 query is not supported" + ) + + segmented_attention( + query=query[:num_actual_tokens], + key=key[:num_actual_tokens], + value=value[:num_actual_tokens], + output=output[:num_actual_tokens], + kv_cache_dtype=self.kv_cache_dtype, + key_cache=key_cache, + value_cache=value_cache, + block_table=attn_metadata.block_table, + query_start_loc=attn_metadata.query_start_loc, + seq_lens=attn_metadata.seq_lens, + max_seq_len=attn_metadata.max_seq_len, + max_query_len=attn_metadata.max_query_len, + k_scale=layer._k_scale, + v_scale=layer._v_scale, + sm_scale=self.scale, + sliding_window=self.sliding_window[0], + output_scale=output_scale, + sinks=self.sinks, + causal=attn_metadata.causal, + softcap=self.logits_soft_cap, + workspace=workspace, + use_tuned_config=self._autotune_enabled(), + ) + return output + + def do_kv_cache_update( + self, + layer: AttentionLayer, + key: torch.Tensor, + value: torch.Tensor, + kv_cache: torch.Tensor, + slot_mapping: torch.Tensor, + ): + key_cache, value_cache = self._split_kv_cache(kv_cache) + if is_quantized_kv_cache(self.kv_cache_dtype): + key_cache = key_cache.view(self.fp8_dtype) + value_cache = value_cache.view(self.fp8_dtype) + triton_reshape_and_cache_flash( + key, + value, + key_cache, + value_cache, + slot_mapping, + self.kv_cache_dtype, + layer._k_scale, + layer._v_scale, + ) + + def do_rope_and_kv_cache_update( + self, + layer: AttentionLayer, + query: torch.Tensor, + key: torch.Tensor, + value: torch.Tensor, + positions: torch.Tensor, + cos_sin_cache: torch.Tensor, + is_neox: bool, + kv_cache: torch.Tensor, + layer_slot_mapping: torch.Tensor, + ): + key_cache, value_cache = self._split_kv_cache( + kv_cache, + layer.num_kv_heads, # type: ignore[attr-defined] + layer.head_size, # type: ignore[attr-defined] + ) + is_fp8_kv_cache = is_quantized_kv_cache(self.kv_cache_dtype) + if is_fp8_kv_cache: + key_cache = key_cache.view(self.fp8_dtype) + value_cache = value_cache.view(self.fp8_dtype) + + rocm_aiter_ops.triton_rope_and_cache( + query, + key, + value, + positions, + cos_sin_cache, + is_neox, + key_cache, + value_cache, + layer_slot_mapping, + layer._k_scale, + layer._v_scale, + True, + is_fp8_kv_cache, + ) diff --git a/vllm/v1/attention/backends/triton_attn_diffkv.py b/vllm/v1/attention/backends/triton_attn_diffkv.py index 4b42838a15..932f483af3 100644 --- a/vllm/v1/attention/backends/triton_attn_diffkv.py +++ b/vllm/v1/attention/backends/triton_attn_diffkv.py @@ -70,10 +70,11 @@ class TritonAttentionDiffKVBackend(TritonAttentionBackend): # V head dim — set per layer via ``set_head_size_v`` before instantiation. head_size_v: int = 128 - # No FP8 / int8 KV cache for the DiffKV path yet; require fp16/bf16/fp32. supported_kv_cache_dtypes: ClassVar[list[CacheDType]] = [ "auto", "bfloat16", + "fp8", + "fp8_e4m3", ] @classmethod @@ -110,16 +111,15 @@ class TritonAttentionDiffKVImpl(TritonAttentionImpl): def __init__(self, *args, **kwargs) -> None: super().__init__(*args, **kwargs) - if is_quantized_kv_cache(self.kv_cache_dtype): + if ( + is_quantized_kv_cache(self.kv_cache_dtype) + and self.kv_cache_dtype + not in TritonAttentionDiffKVBackend.supported_kv_cache_dtypes + ): raise NotImplementedError( - "TritonAttentionDiffKVBackend does not yet support quantized " + "TritonAttentionDiffKVBackend supports only fp8/fp8_e4m3 quantized " f"KV cache (got kv_cache_dtype={self.kv_cache_dtype!r})." ) - if self._is_per_token_head_quant: - raise NotImplementedError( - "TritonAttentionDiffKVBackend does not support per-token-head " - "quantization." - ) if self.chunk_lookback > -1: raise NotImplementedError( "TritonAttentionDiffKVBackend does not support chunked " @@ -191,6 +191,9 @@ class TritonAttentionDiffKVImpl(TritonAttentionImpl): # Triton DiffKV kernels consume (B, N, H, D) cache views. kv_cache = kv_cache.transpose(1, 2) + quantized = is_quantized_kv_cache(self.kv_cache_dtype) + if quantized: + kv_cache = kv_cache.view(self.fp8_dtype) key_cache = kv_cache[..., :head_size_qk] value_cache = kv_cache[..., head_size_qk : head_size_qk + head_size_v] @@ -202,6 +205,11 @@ class TritonAttentionDiffKVImpl(TritonAttentionImpl): cu_seqlens_q=attn_metadata.query_start_loc, seqused_k=attn_metadata.seq_lens, softmax_scale=self.scale, + q_descale=( + layer._q_scale if quantized and query.dtype == self.fp8_dtype else None + ), + k_descale=layer._k_scale if quantized else None, + v_descale=layer._v_scale if quantized else None, causal=True, alibi_slopes=self.alibi_slopes, use_alibi_sqrt=self.use_alibi_sqrt, diff --git a/vllm/v1/attention/backends/ultraquant_attn.py b/vllm/v1/attention/backends/ultraquant_attn.py new file mode 100644 index 0000000000..b62b7a6830 --- /dev/null +++ b/vllm/v1/attention/backends/ultraquant_attn.py @@ -0,0 +1,609 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""UltraQuant 4-bit KV-cache attention backend. + +UltraQuant is a 4-bit KV-cache format (FP4 E2M1 codes + UE8M0 power-of-two +per-group-32 scales) with Hadamard-rotated keys. It reuses the TurboQuant +attention scaffolding (forward orchestration, metadata, cache-layout addressing, +CUDA-graph pre-warm) by subclassing, and overrides only the format-specific +seams: + + * store -> ultraquant_store (rotate K, pack FP4 + UE8M0 scales) + * decode -> FlyDSL D=256 scaled-MFMA kernel, Triton unified fallback + * continuation -> small chunk: Triton unified; large chunk: dequant + FA + +``UltraQuantAttentionBackend`` is a standalone backend with its own identity +(``ULTRAQUANT``) and dtype (``ultraquant_4bit``); the TurboQuant backend/impl +carry no UltraQuant knowledge. +""" + +import contextlib +import math +from dataclasses import replace +from typing import Any, ClassVar + +import torch +import torch.nn.functional as F + +from vllm.config import get_current_vllm_config +from vllm.config.cache import CacheDType +from vllm.logger import init_logger +from vllm.v1.attention.backend import AttentionType +from vllm.v1.attention.backends.fa_utils import get_flash_attn_version +from vllm.v1.attention.backends.turboquant_attn import ( + _CONTINUATION_DECODE_THRESHOLD, + _HAS_FLASH_ATTN, + TurboQuantAttentionBackend, + TurboQuantAttentionImpl, + TurboQuantMetadata, + _build_hadamard, +) +from vllm.v1.attention.ops.flydsl_ultraquant_decode import ( + flydsl_ultraquant_decode_attention, + is_ultraquant_flydsl_available, + ultraquant_flydsl_decode_eligible, +) +from vllm.v1.attention.ops.ultraquant.format import slot_size +from vllm.v1.attention.ops.ultraquant.triton_unified_attention import ( + ultraquant_unified_attention, +) +from vllm.v1.kv_cache_interface import AttentionSpec +from vllm.v1.worker.workspace import ( + current_workspace_manager, + is_workspace_manager_initialized, +) + +logger = init_logger(__name__) + +# CK FlashAttention on ROCm has no compiled kernel above head_dim 256 and no +# sinks argument; those cases route through the Triton unified kernel instead. +_CK_MAX_HEAD_DIM = 256 + + +class UltraQuantAttentionBackend(TurboQuantAttentionBackend): + """Standalone backend for the ``ultraquant_4bit`` KV-cache dtype. + + Subclasses :class:`TurboQuantAttentionBackend` for the shared metadata and + cache-layout plumbing, and overrides identity, spec, impl, block size, and + capabilities so no UltraQuant knowledge lives in the TurboQuant backend. + """ + + supported_kv_cache_dtypes: ClassVar[list[CacheDType]] = ["ultraquant_4bit"] + # FlyDSL D=256 decode loads query as bf16; fp16 is not supported. Selection + # reads this from the backend class, not from the impl. + supported_dtypes: ClassVar[list[torch.dtype]] = [torch.bfloat16] + + @classmethod + def customize_spec(cls, spec: AttentionSpec) -> AttentionSpec: + """UltraQuant packs K+V into one 4-bit FP4/UE8M0 slot per head.""" + if spec.state_content_bytes is not None or not spec.kv_quant_mode.is_ultraquant: + return spec + return replace(spec, state_content_bytes=slot_size(spec.head_size)) + + @staticmethod + def get_name() -> str: + return "ULTRAQUANT" + + @classmethod + def get_preferred_block_size(cls, default_block_size: int) -> int: + # FlyDSL D=256 decode geometry requires a 64-token KV block; pin it so + # the KV-cache manager pages stay dense and splittable into kernel blocks. + return 64 + + @staticmethod + def get_impl_cls() -> type["UltraQuantAttentionImpl"]: + return UltraQuantAttentionImpl + + @classmethod + def supports_kv_cache_dtype(cls, kv_cache_dtype: CacheDType | None) -> bool: + return kv_cache_dtype == "ultraquant_4bit" + + @classmethod + def supports_sink(cls) -> bool: + # Decode and continuation prefill route sinks to the Triton unified + # kernel, but first-chunk prefill goes through flash_attn_varlen, which + # takes no sinks argument. Until that path is covered, reject sink + # models at selection rather than returning silently wrong output. + return False + + +class UltraQuantAttentionImpl(TurboQuantAttentionImpl): + """UltraQuant 4-bit KV-cache attention (FP4 E2M1 + UE8M0, rotated K). + + Overrides the init, on-device setup, store, decode, and prefill/continuation + seams of :class:`TurboQuantAttentionImpl`; the shared ``forward`` / + ``do_kv_cache_update`` orchestration and metadata handling are inherited. + """ + + # FlyDSL D=256 decode loads query as bf16; fp16 is not supported. + supported_dtypes: ClassVar[list[torch.dtype]] = [torch.bfloat16] + + def __init__( + self, + num_heads: int, + head_size: int, + scale: float, + num_kv_heads: int | None = None, + alibi_slopes: list[float] | None = None, + sliding_window: int | None = None, + kv_cache_dtype: str = "auto", + logits_soft_cap: float | None = None, + attn_type: str = AttentionType.DECODER, + kv_sharing_target_layer_name: str | None = None, + **kwargs, + ): + self.num_heads = num_heads + self.head_size = head_size + self.scale = scale + self.num_kv_heads = num_kv_heads if num_kv_heads is not None else num_heads + self.num_kv_groups = num_heads // self.num_kv_heads + self.kv_cache_dtype = kv_cache_dtype + + self.fa_version = get_flash_attn_version(head_size=head_size) + + vllm_config = get_current_vllm_config() + self.max_num_kv_splits = ( + vllm_config.attention_config.tq_max_kv_splits_for_cuda_graph + ) + self.sliding_window = sliding_window + self.sinks = kwargs.get("sinks") + self._max_model_len = vllm_config.model_config.max_model_len + + self._use_ultraquant_flydsl = is_ultraquant_flydsl_available() + # FlyDSL decode eligibility is static per layer config; compute once. + self._flydsl_decode_eligible = ultraquant_flydsl_decode_eligible( + head_size=self.head_size, + num_kv_groups=self.num_kv_groups, + has_sinks=self.sinks is not None, + sliding_window=self.sliding_window, + flydsl_loaded=self._use_ultraquant_flydsl, + ) + + def _ensure_on_device(self, layer, device): + """Build the shared Hadamard rotation and pre-warm FlyDSL buffers. + + UltraQuant needs no Lloyd-Max centroids, but the inherited forward reads + ``layer._tq_centroids``, so an empty tensor is kept on the layer. + """ + if self._use_ultraquant_flydsl: + # CUDA-graph capture safety for the FlyDSL decode path on ROCm: + # pre-allocate index buffers and grow the workspace to its max size + # before capture so replay never lands on stale addresses. + _max_len = self._max_model_len + _already_ok = ( + hasattr(self, "_arange_cache") + and self._arange_cache.device.type == str(device).split(":")[0] + and self._arange_cache.shape[0] >= _max_len + 2 + ) + if not _already_ok: + self._arange_cache = torch.arange( + 0, _max_len + 2, device=device, dtype=torch.int32 + ) + if not hasattr(self, "_cu_2") or self._cu_2.device != torch.device(device): + self._cu_2 = torch.zeros(2, device=device, dtype=torch.int32) + if ( + is_workspace_manager_initialized() + and not current_workspace_manager().is_locked() + ): + B_max = self._max_capture_batch_size() + D = self.head_size + Hq = self.num_heads + S = self.max_num_kv_splits + _pre_warm_bytes = ( + B_max * Hq * (S * (D + 1) + D) * 4 # fp32 mid_o + fp32 lse + + B_max * Hq * D * 2 # query-dtype output (bf16 = 2 B) + + 512 # alignment padding + ) + with contextlib.suppress(AssertionError): + current_workspace_manager().get_simultaneous( + ((_pre_warm_bytes,), torch.uint8) + ) + + if not hasattr(layer, "_tq_cached"): + D = self.head_size + # Pure Hadamard: orthonormal + symmetric (H = H^T), enabling + # in-kernel butterfly fusion and trivial inverse for continuation. + H = _build_hadamard(D, str(device)) + layer._tq_PiT = H + layer._tq_Pi = H + layer._tq_Pi_half = H.to(torch.float16) + # Inherited TurboQuantAttentionImpl.forward reads _tq_centroids + # unconditionally; UltraQuant has no centroid table, so an empty + # tensor satisfies the read. + layer._tq_centroids = torch.empty(0, device=device, dtype=torch.float32) + layer._tq_cached = True + + def _store_kv( + self, + key: torch.Tensor, # (N, Hk, D) + value: torch.Tensor, # (N, Hk, D) + kv_cache: torch.Tensor, # (num_blocks, block_size, Hk, slot_size) + slot_mapping: torch.Tensor, + layer: Any, + ): + """Rotate K by the Hadamard and pack K+V into FP4/UE8M0 slots.""" + from vllm.v1.attention.ops.ultraquant.triton_store import ultraquant_store + + ultraquant_store( + key, + value, + kv_cache, + slot_mapping, + PiT=layer._tq_PiT, + ) + + def _decode_attention( + self, + query: torch.Tensor, # (B, Hq, D) + kv_cache: torch.Tensor, # (num_blocks, block_size, Hk, slot_size) + attn_metadata: TurboQuantMetadata, + Pi: torch.Tensor, + centroids: torch.Tensor, + PiT: torch.Tensor | None = None, + layer: torch.nn.Module | None = None, + ) -> torch.Tensor: + # Acquire shared decode scratch from the WorkspaceManager (UltraQuant + # only needs the output buffer; the mid_o/lse slots keep the triple + # allocation identical to the base decode for buffer reuse). + B = query.shape[0] + D = self.head_size + S = self.max_num_kv_splits + Hq = self.num_heads + output_buf = None + if is_workspace_manager_initialized(): + _mid_o, output_buf, _lse = current_workspace_manager().get_simultaneous( + ((B, Hq, S, D + 1), torch.float32), + ((B, Hq, D), query.dtype), + ((B, Hq), torch.float32), + ) + + if self._flydsl_decode_eligible: + return flydsl_ultraquant_decode_attention( + query=query, + kv_cache=kv_cache, + block_table=attn_metadata.block_table, + seq_lens=attn_metadata.seq_lens, + scale=self.scale, + PiT=PiT, + max_seq_len=attn_metadata.max_seq_len, + output_buf=output_buf, + buf_holder=layer, + max_num_kv_splits=self.max_num_kv_splits, + sinks=self.sinks, + ) + + logger.info_once( + "UltraQuant FlyDSL is ineligible for head_size=%s, GQA=%s, " + "sinks=%s, sliding_window=%s; using Triton fallback.", + self.head_size, + self.num_kv_groups, + self.sinks is not None, + self.sliding_window, + ) + return ultraquant_unified_attention( + query=query, + kv_cache=kv_cache, + block_table=attn_metadata.block_table, + seq_lens=attn_metadata.seq_lens, + query_start_loc=attn_metadata.query_start_loc, + scale=self.scale, + PiT=PiT, + output=output_buf, + max_query_len=1, + max_seq_len=attn_metadata.max_seq_len, + sinks=self.sinks, + sliding_window=self.sliding_window, + ) + + def _prefill_attention( + self, + query: torch.Tensor, # (N, Hq, D) + key: torch.Tensor, # (N, Hk, D) + value: torch.Tensor, # (N, Hk, D) + kv_cache: torch.Tensor, # (num_blocks, block_size, Hk, slot_size) + attn_metadata: TurboQuantMetadata, + Pi: torch.Tensor, + centroids: torch.Tensor, + PiT: torch.Tensor | None = None, + layer: Any = None, + ) -> torch.Tensor: + N, Hq, D = query.shape + + # Fast path: flash_attn for first-chunk prefills (all K/V in batch). + # CK cannot serve head dims above 256, so those fall through to the + # per-request unified-attention path below. + if ( + _HAS_FLASH_ATTN + and attn_metadata.max_query_len == attn_metadata.max_seq_len + and D <= _CK_MAX_HEAD_DIM + ): + return self._flash_attn_varlen( + q=query, + k=key, + v=value, + cu_seqlens_q=attn_metadata.query_start_loc, + cu_seqlens_k=attn_metadata.query_start_loc, + max_seqlen_q=attn_metadata.max_query_len, + max_seqlen_k=attn_metadata.max_query_len, + ) + + # Continuation or no flash_attn: per-request attention. + Hk = key.shape[1] + use_gqa = Hk < Hq + query_start_loc = attn_metadata.query_start_loc + num_reqs = query_start_loc.shape[0] - 1 + + output = torch.zeros(N, Hq, D, device=query.device, dtype=query.dtype) + + if attn_metadata.query_start_loc_cpu is not None: + qsl = attn_metadata.query_start_loc_cpu.tolist() + else: + qsl = query_start_loc.tolist() + if attn_metadata.seq_lens_cpu is not None: + seq_lens_list = attn_metadata.seq_lens_cpu.tolist() + else: + seq_lens_list = attn_metadata.seq_lens.tolist() + + if not hasattr(self, "_cu_2"): + self._cu_2 = torch.zeros(2, device=query.device, dtype=torch.int32) + _max_seq = attn_metadata.max_seq_len + _ac: torch.Tensor | None = getattr(self, "_arange_cache", None) + if _ac is None or _ac.shape[0] <= _max_seq: + _ac = torch.arange( + 0, _max_seq + 1, device=query.device, dtype=attn_metadata.seq_lens.dtype + ) + self._arange_cache = _ac + _arange_cache: torch.Tensor = _ac + + for i in range(num_reqs): + q_start = qsl[i] + q_end = qsl[i + 1] + q_len = q_end - q_start + if q_len <= 0: + continue + + seq_len = seq_lens_list[i] + q_seq = query[q_start:q_end] # (q_len, Hq, D) + k_seq = key[q_start:q_end] # (q_len, Hk, D) + v_seq = value[q_start:q_end] # (q_len, Hk, D) + + if q_len == seq_len: + # First-chunk prefill: all K/V are in the current batch. + if D > _CK_MAX_HEAD_DIM: + # Head dims CK cannot handle (Gemma 4 global layers, + # D=512). This kernel is generic over head size. + from vllm.v1.attention.ops.triton_unified_attention import ( + unified_attention, + ) + + out = torch.empty_like(q_seq) + cu_single = torch.tensor( + [0, q_len], dtype=torch.int32, device=query.device + ) + seq_lens_single = torch.tensor( + [q_len], dtype=torch.int32, device=query.device + ) + block_table_single = torch.zeros( + (1, 1), dtype=torch.int32, device=query.device + ) + unified_attention( + q=q_seq, + k=k_seq.unsqueeze(0), + v=v_seq.unsqueeze(0), + out=out, + cu_seqlens_q=cu_single, + max_seqlen_q=q_len, + seqused_k=seq_lens_single, + max_seqlen_k=q_len, + softmax_scale=self.scale, + causal=True, + window_size=(-1, -1), + block_table=block_table_single, + softcap=0.0, + q_descale=None, + k_descale=None, + v_descale=None, + sinks=None, + ) + elif _HAS_FLASH_ATTN: + self._cu_2[1:2] = q_len + cu = self._cu_2 + out = self._flash_attn_varlen( + q=q_seq, + k=k_seq, + v=v_seq, + cu_seqlens_q=cu, + cu_seqlens_k=cu, + max_seqlen_q=q_len, + max_seqlen_k=q_len, + ) + else: + q_t = q_seq.transpose(0, 1).contiguous() + k_t = k_seq.transpose(0, 1).contiguous() + v_t = v_seq.transpose(0, 1).contiguous() + out = F.scaled_dot_product_attention( + q_t, + k_t, + v_t, + is_causal=True, + scale=self.scale, + enable_gqa=use_gqa, + ).transpose(0, 1) + output[q_start:q_end] = out.to(query.dtype) + else: + # Continuation chunk: prefix already stored (Hadamard-rotated + # FP4) by do_kv_cache_update. Small chunks reuse the Triton + # unified decode-shaped kernel; large chunks dequant the prefix + # once and run a dense flash-attention prefill. + cached_len = seq_len - q_len + if q_len <= _CONTINUATION_DECODE_THRESHOLD: + synth_seq_lens = _arange_cache[cached_len + 1 : seq_len + 1] + synth_bt = attn_metadata.block_table[i : i + 1].expand(q_len, -1) + out = ultraquant_unified_attention( + query=q_seq, + kv_cache=kv_cache, + block_table=synth_bt, + seq_lens=synth_seq_lens, + query_start_loc=_arange_cache[: q_len + 1], + scale=self.scale, + PiT=PiT, + max_query_len=1, + max_seq_len=seq_len, + sinks=self.sinks, + sliding_window=self.sliding_window, + ) + else: + out = self._ultraquant_continuation_prefill( + layer=layer, + query=q_seq, + key_chunk=k_seq, + val_chunk=v_seq, + kv_cache=kv_cache, + block_table=attn_metadata.block_table[i : i + 1], + cached_len=cached_len, + seq_len=seq_len, + PiT=PiT, + ) + output[q_start:q_end] = out.to(query.dtype) + + return output + + def _ultraquant_continuation_prefill( + self, + *, + layer: Any, + query: torch.Tensor, # (q_len, Hq, D) unrotated + key_chunk: torch.Tensor, # (q_len, Hk, D) unrotated + val_chunk: torch.Tensor, # (q_len, Hk, D) + kv_cache: torch.Tensor, # (num_blocks, block_size, Hk, slot_size) + block_table: torch.Tensor, # (1, max_num_blocks) + cached_len: int, + seq_len: int, + PiT: torch.Tensor, # (D, D) fp32 Hadamard + ) -> torch.Tensor: + """Large continuation chunk: dequant the cached prefix, then flash_attn. + + The cache stores K already Hadamard-rotated, so only the current chunk's + K and the queries need rotating. Hadamard is symmetric orthonormal, so + rotating both sides leaves the attention scores unchanged. + """ + from vllm.v1.attention.ops.ultraquant.triton_dequant import ( + ultraquant_full_dequant_kv, + ) + + q_len, Hq, D = query.shape + Hk = key_chunk.shape[1] + device = query.device + qdtype = query.dtype + block_size = kv_cache.shape[1] + alloc_len = math.ceil(cached_len / block_size) * block_size + + # Dequant target, shared across layers by the workspace manager so long + # context does not pay one allocation per layer. + buf_shape = (1, Hk, alloc_len, D) + k_buf, v_buf = current_workspace_manager().get_simultaneous( + (buf_shape, qdtype), + (buf_shape, qdtype), + ) + ultraquant_full_dequant_kv( + kv_cache=kv_cache, + block_table=block_table, + k_out=k_buf, + v_out=v_buf, + alloc_len=alloc_len, + ) + + # Layer-cached prefix+chunk concatenation, grown once to worst case. + cap = block_table.shape[1] * block_size + k_full_buf = getattr(layer, "_uq_kfull_buf", None) + if ( + k_full_buf is None + or k_full_buf.shape[0] < cap + or k_full_buf.dtype != qdtype + ): + k_full_buf = torch.empty(cap, Hk, D, dtype=qdtype, device=device) + layer._uq_kfull_buf = k_full_buf + v_full_buf = getattr(layer, "_uq_vfull_buf", None) + if ( + v_full_buf is None + or v_full_buf.shape[0] < cap + or v_full_buf.dtype != qdtype + ): + v_full_buf = torch.empty(cap, Hk, D, dtype=qdtype, device=device) + layer._uq_vfull_buf = v_full_buf + k_full = k_full_buf[:seq_len] + v_full = v_full_buf[:seq_len] + + k_full[:cached_len] = k_buf[0, :, :cached_len, :].transpose(0, 1) + v_full[:cached_len] = v_buf[0, :, :cached_len, :].transpose(0, 1) + k_full[cached_len:] = (key_chunk.to(torch.float32) @ PiT).to(qdtype) + v_full[cached_len:] = val_chunk + q_rot = (query.to(torch.float32) @ PiT).to(qdtype) + + # Unequal q/k lengths with causal=True gives lower-right alignment: + # query i (absolute position cached_len + i) sees keys 0..cached_len+i. + # + # FA2 on ROCm has no sinks parameter, and CK cannot serve head dims + # above 256, so those cases route through the Triton unified kernel. + # Q/K are already in Hadamard-rotated space (q_rot, k_full) so the dot + # products match the original-space attention scores. + if self.sinks is not None or D > _CK_MAX_HEAD_DIM: + from vllm.v1.attention.ops.triton_unified_attention import ( + unified_attention, + ) + + out = torch.empty_like(q_rot) + cu_q = torch.tensor([0, q_len], dtype=torch.int32, device=device) + seqused_k = torch.tensor([seq_len], dtype=torch.int32, device=device) + block_table_single = torch.zeros((1, 1), dtype=torch.int32, device=device) + window = ( + (self.sliding_window - 1, 0) + if self.sliding_window and self.sliding_window > 0 + else (-1, -1) + ) + unified_attention( + q=q_rot, + k=k_full.unsqueeze(0), + v=v_full.unsqueeze(0), + out=out, + cu_seqlens_q=cu_q, + max_seqlen_q=q_len, + seqused_k=seqused_k, + max_seqlen_k=seq_len, + softmax_scale=self.scale, + causal=True, + window_size=window, + block_table=block_table_single, + softcap=0.0, + q_descale=None, + k_descale=None, + v_descale=None, + sinks=self.sinks, + ) + return out.to(qdtype) + if _HAS_FLASH_ATTN: + cu_q = torch.tensor([0, q_len], dtype=torch.int32, device=device) + cu_k = torch.tensor([0, seq_len], dtype=torch.int32, device=device) + return self._flash_attn_varlen( + q=q_rot, + k=k_full, + v=v_full, + cu_seqlens_q=cu_q, + cu_seqlens_k=cu_k, + max_seqlen_q=q_len, + max_seqlen_k=seq_len, + ) + q_t = q_rot.transpose(0, 1).unsqueeze(0) + k_t = k_full.transpose(0, 1).unsqueeze(0) + v_t = v_full.transpose(0, 1).unsqueeze(0) + if Hq > Hk: + k_t = k_t.expand(1, Hq, -1, -1) + v_t = v_t.expand(1, Hq, -1, -1) + return ( + F.scaled_dot_product_attention( + q_t, k_t, v_t, is_causal=True, scale=self.scale + ) + .squeeze(0) + .transpose(0, 1) + .to(qdtype) + ) diff --git a/vllm/v1/attention/backends/utils.py b/vllm/v1/attention/backends/utils.py index 529d816c06..f93156d53f 100644 --- a/vllm/v1/attention/backends/utils.py +++ b/vllm/v1/attention/backends/utils.py @@ -307,8 +307,12 @@ def resolve_kv_cache_layout( f"none is in every supported set: {supported_layouts}." ) + # Self-addressed per-request rings (e.g. the kpool tail) are replicated + # state like Mamba and don't need a separate draft group. dcp_sharding = { - spec.dcp_sharded for spec in kv_cache_specs if isinstance(spec, AttentionSpec) + spec.dcp_sharded + for spec in kv_cache_specs + if isinstance(spec, AttentionSpec) and spec.uses_slot_mapping } page_sizes = {spec.page_size_bytes for spec in kv_cache_specs} if len(dcp_sharding) > 1 and len(page_sizes) > 1: diff --git a/vllm/v1/attention/ops/flydsl_kernels/ultraquant_decode_hd256.py b/vllm/v1/attention/ops/flydsl_kernels/ultraquant_decode_hd256.py new file mode 100644 index 0000000000..6a03f06f45 --- /dev/null +++ b/vllm/v1/attention/ops/flydsl_kernels/ultraquant_decode_hd256.py @@ -0,0 +1,983 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +# UltraQuant D=256 decode (FlyDSL) — gfx950 / CDNA4 +# +# Production path only: scaled FP4×E4M3 QK MFMA, native V CVT, HW V-transpose, +# Q hoist, strided tile-groups, and in-kernel Walsh-Hadamard for GQA 6/8/16. +# +# AoS slot (D=256, group_size=32 → 8 groups, 272 B): +# [0:128) K FP4 codes; [128:136) K UE8M0; [136:264) V FP4; [264:272) V UE8M0 +# QK MFMA: A=K, B=Q → C[token, query]; PV MFMA: A=V_T, B=P → C[head_dim, query] + +from __future__ import annotations + +import math + +import flydsl.compiler as flyc +import flydsl.expr as fx +from aiter.ops.flydsl.kernels import buffer_ops, vector +from flydsl._mlir import ir +from flydsl._mlir.dialects import scf as _scf +from flydsl.expr import ( + arith, + gpu, + rocdl, +) +from flydsl.expr.primitive import const_expr, range_constexpr +from flydsl.expr.typing import T +from flydsl.runtime.device import get_rocm_arch as get_hip_arch +from flydsl.utils.smem_allocator import SmemAllocator, SmemPtr + + +def _vector_insert(value, dest, *, static_position): + """Insert a DSL scalar using FlyDSL 0.3's raw vector dialect binding.""" + return vector.insert( + vector.as_ir_value(value), + vector.as_ir_value(dest), + dynamic_position=[], + static_position=static_position, + ) + + +# === Constants (256-wide-head ultraquant decode profile) ======================= +HEAD_SIZE = 256 +KV_BLOCK_SIZE = 16 # default; overridable via build(kv_block_size=...) +TILE_SIZE = 16 # MFMA tile = 16 tokens +N_CENTROIDS = 16 # FP4 E2M1 value table (16 entries indexed by nibble) +QUERY_GROUP_SIZE = 16 # default for tests; overridable via build(...) +WARP_SIZE = 64 +BLOCK_THREADS = WARP_SIZE +KV_COMPUTE_BLOCK = 256 # 16 K-tiles × 16 tokens + +# ultraquant AoS slot layout (per (slot, head), D=256, group_size=32 => 8 groups) +FP8_GROUP_SIZE = 32 +N_GROUPS = HEAD_SIZE // FP8_GROUP_SIZE # 8 UE8M0 scale bytes per head +KEY_CODE_BYTES = HEAD_SIZE // 2 # 128 +VAL_CODE_BYTES = HEAD_SIZE // 2 # 128 +K_SCALES_OFFSET = KEY_CODE_BYTES # 128 +V_CODES_OFFSET = KEY_CODE_BYTES + N_GROUPS # 136 +V_SCALES_OFFSET = V_CODES_OFFSET + VAL_CODE_BYTES # 264 +SLOT_CONTENT_BYTES = V_SCALES_OFFSET + N_GROUPS # 272 + +# MFMA +MFMA_N = 16 +PV_N_CHUNKS = HEAD_SIZE // MFMA_N # 16 for HEAD_SIZE=256 + +# --- Per-lane data-movement geometry (generalized from the 128 kernel) ------ +# The K/V dequant lane assignment splits each token's packed code bytes across +# the 4 ``chunk_in_tok`` sub-lanes (lane % 4). Each lane processes +# ``KEY_CODE_BYTES // 4`` code bytes = 32 B at 256. buffer_load maxes at 16 B +# (dwordx4) so a lane issues ``HALVES``=2 loads; each 16-B half covers exactly +# one UE8M0 group of 32 head-dims. +# HEAD_SIZE=128: LANE_CODE_BYTES=16, HALVES=1, SUBCHUNK_HDIMS=32 +# HEAD_SIZE=256: LANE_CODE_BYTES=32, HALVES=2, SUBCHUNK_HDIMS=64 +LANE_CODE_BYTES = KEY_CODE_BYTES // 4 # 32 code bytes per lane per K-tile +HALVES = LANE_CODE_BYTES // 16 # 2 for HEAD_SIZE=256 +HALF_HDIMS = 32 # head-dims per half (= one group) +SUBCHUNK_HDIMS = HEAD_SIZE // 4 # head-dims per chunk_in_tok sub-lane (64) +# Q row-major LDS load geometry (STEP B): each lane loads 8 bf16 (=4 i32), +# so a row spans ``HEAD_SIZE // 8`` column-groups. +Q_GROUPS_PER_ROW = HEAD_SIZE // 8 # 32 for 256 +Q_ROW_SHIFT = Q_GROUPS_PER_ROW.bit_length() - 1 # 5 for 256 +Q_COL_MASK = Q_GROUPS_PER_ROW - 1 # 31 for 256 + +# LDS regions +CENTROID_LDS_BYTES = N_CENTROIDS * 4 # 64 + +# KV LDS rows are padded by 16 B to reduce bank conflicts while preserving +# ds_read/ds_write_b128 alignment. +KV_ROW_PAD_ELEMS = 8 +assert KV_ROW_PAD_ELEMS % 8 == 0, "KV row pad must be a multiple of 8 bf16 (16 B)" +KV_ROW_ELEMS = HEAD_SIZE + KV_ROW_PAD_ELEMS # 264 @ pad 8 +KV_ROW_BYTES = KV_ROW_ELEMS * 2 # 528 +KFP4_ROW_PAD_I32 = 4 +assert KFP4_ROW_PAD_I32 % 4 == 0, "FP4-K row pad must be a multiple of 4 i32 (16 B)" +KFP4_ROW_I32 = HEAD_SIZE // 8 + KFP4_ROW_PAD_I32 +KV_TILE_LDS_BYTES_PADDED = TILE_SIZE * KV_ROW_BYTES # 8448 (V dominates) +SCALE_LDS_BYTES = TILE_SIZE * N_GROUPS * 4 # 16*8*4 = 512 for 256 + +LOG2E = 1.4426950408889634 +NEG_INF_VAL = float("-inf") + + +def _vsplat_mul(vec, scalar): + s = scalar.ir_value() if hasattr(scalar, "ir_value") else scalar + return vec * vector.broadcast(T.f32x4, s) + + +allocator = None + + +def build_ultraquant_decode_hd256_module( + num_seqs: int, + num_kv_heads: int, + num_partitions: int, + padded_slot: int, + max_blocks_per_seq: int = 512, + softmax_scale: float | None = None, + query_group_size: int = QUERY_GROUP_SIZE, + kv_block_size: int = KV_BLOCK_SIZE, + tile_groups_per_partition: int = 1, + fuse_qrot: bool = True, + stride_q_seq: int | None = None, + stride_q_head: int | None = None, +): + """Build the production UltraQuant D=256 decode kernel. + + ``padded_slot`` is the per-(token, head) slot stride in bytes + (``kv_cache.shape[3]``, >= 272). Cache layout is + ``[num_blocks, block_size, num_kv_heads, padded_slot]`` uint8 AoS. + + QK is always scaled FP4×E4M3 MFMA; V always uses native CVT + HW + transpose; tile-groups are always strided. ``fuse_qrot`` selects the + in-kernel WHT when the query layout is eligible. + """ + # QG=6 (Qwen3.6 full-attention layers) reuses the 8-row WHT lane map with + # two inactive rows masked. STEP C may read rows QG..15 from adjacent LDS, + # but those MFMA rows are discarded by the `mfma_row < QG` output gate. + assert query_group_size in (6, 8, 16), ( + f"query_group_size must be 6, 8 or 16; got {query_group_size}" + ) + # head_dim=256 hybrid models (Qwen3.6/3.8) force a larger mamba-aligned KV + # block; the tiled inner loop (_TILES_PER_BLOCK = kv_block_size // TILE_SIZE) + # handles any multiple of TILE_SIZE, so accept 16/32/64/128/256. + assert kv_block_size in (16, 32, 64, 128, 256), ( + f"kv_block_size must be 16/32/64/128/256; got {kv_block_size}" + ) + assert kv_block_size % TILE_SIZE == 0 + assert int(tile_groups_per_partition) >= 1 + + TGPP = int(tile_groups_per_partition) + # Partition p owns tile-groups p, p+P, p+2P, ... This keeps the graph-fixed + # grid wide for short sequences; the reducer is agnostic to the mapping. + NUM_PARTS_C = int(num_partitions) + MFMA_SCALED_K = 128 + MFMA_ISSUES = HEAD_SIZE // MFMA_SCALED_K # 2 for HEAD_SIZE=256 + GRPS_PER_ISSUE = MFMA_SCALED_K // FP8_GROUP_SIZE # 4 + + QG = int(query_group_size) + QG_LOAD_ITERS = (QG * Q_GROUPS_PER_ROW) // WARP_SIZE # 8@QG16, 4@QG8, 3@QG6 + + _BLOCK_THREADS = WARP_SIZE + + # Fuse Q rotation (q_rot = Q @ PiT) into this kernel as an in-register WHT. + # The 8-row lane map masks its two inactive rows for GQA-6. + _FUSE_QROT = bool(fuse_qrot) + + # --- Cache policy on the KV code stream ---------------------------------- + # KV codes are read EXACTLY ONCE per decode (81.6 MB at B=4/seq=75k) and + # never reused, so caching them only evicts Q / block-table / scale lines + # that ARE reused. gfx9 buffer aux bits: bit0=sc0, bit1=nt, bit4=sc1. + # 0 = default (what we ship today: no hint at all) + # 2 = nt (non-temporal: stream, do not pollute) + # 18 = nt|sc1 (also bypass the far cache level) + _KV_AUX = 0 + + # --- Occupancy: size the Q LDS to the ACTUAL query group ------------------- + # STEP B only writes rows [0, QG) and STEP C/F only read those rows. For + # QG=8 the upper 4 KB is allocated-but-never-touched, so shrinking it to + # QG*HEAD_SIZE*2 is bitwise identical and frees LDS/workgroup (17 KB -> 13 KB + # at QG=8), lifting waves/CU (~3 -> ~5) since the kernel is LDS-occupancy + # bound. The launcher pads segm pools to QG=16 regardless, so this is purely + # an in-kernel LDS footprint reduction. + _Q_LDS_BYTES = QG * HEAD_SIZE * 2 + + _BS = int(kv_block_size) + _TILES_PER_BLOCK = _BS // TILE_SIZE + _PADDED_SLOT = int(padded_slot) + assert _PADDED_SLOT >= SLOT_CONTENT_BYTES, ( + f"padded_slot={_PADDED_SLOT} < {SLOT_CONTENT_BYTES} (ultraquant D=256 content)" + ) + assert _PADDED_SLOT % 4 == 0, f"padded_slot={_PADDED_SLOT} must be 4-byte aligned" + + arch = get_hip_arch() + + if softmax_scale is None: + softmax_scale = 1.0 / (HEAD_SIZE**0.5) + _qk_scale = float(softmax_scale) + + # --- Strides --- + _Hq = num_kv_heads * QG + # Defaults describe the pooled [B, Hq, D] q_rot buffer. When the rotation is + # fused in (STEP B'), Q is read straight from the model's query tensor, + # which is a SPLIT OF THE FUSED QKV PROJECTION -- so its row stride is + # (Hq + 2*Hk)*D, not Hq*D. Those strides are constant for a given model and + # TP rank, so the caller passes them in and they bake in like any other + # compile-time constant (they are part of the kernel cache key). + _stride_q_seq = int(stride_q_seq) if stride_q_seq is not None else _Hq * HEAD_SIZE + _stride_q_head = int(stride_q_head) if stride_q_head is not None else HEAD_SIZE + _stride_bt_seq = max_blocks_per_seq + + # ultraquant AoS cache: [num_blocks, block_size, num_kv_heads, padded_slot]. + _stride_cache_head = _PADDED_SLOT + _stride_cache_pos = num_kv_heads * _PADDED_SLOT + _stride_cache_block = _BS * _stride_cache_pos + + _stride_out_part = QG * HEAD_SIZE + _stride_out_head = num_partitions * QG * HEAD_SIZE + _stride_out_seq = num_kv_heads * num_partitions * QG * HEAD_SIZE + _stride_es_seq = num_kv_heads * num_partitions * QG + _stride_ml_seq = _stride_es_seq + + # --- LDS layout --- + # `allocator` is intentionally a BUILD-LOCAL captured by the kernel closure + # (a freevar), NOT a module global. The @flyc.jit drift guard + # (_check_globals_drift) snapshots every module global referenced by the JIT + # launcher's dependency tree (which includes this kernel) and ABORTS if one + # changes between compiles. When more than one partition variant is + # JIT-resident at once (e.g. a ragged/adaptive batch mixes num_partitions + # 256 and 512, or fixed cap=512 yields 294@75k and 512@131k), each build + # created a fresh SmemAllocator; as a reassigned MODULE GLOBAL that tripped + # "global 'allocator' changed since first compile" and crashed the worker. + # Keeping it local + a per-variant symbol name lets the variants coexist. + # The build-local is handed to the launcher via a module attribute set AFTER + # the kernel is built (see end of function); that assignment is safe because + # no @flyc.jit function references the module global anymore. + allocator = SmemAllocator( + None, + arch=arch, + global_sym_name=(f"ultraquant_hd256_smem_p{int(num_partitions)}"), + ) + centroid_off = 0 + allocator.ptr = CENTROID_LDS_BYTES + q_off = allocator.ptr + allocator.ptr += _Q_LDS_BYTES + kv_off = allocator.ptr + allocator.ptr += KV_TILE_LDS_BYTES_PADDED + scale_off = allocator.ptr + allocator.ptr += SCALE_LDS_BYTES + + @flyc.kernel + def ultraquant_decode_hd256_kernel( + out_ptr: fx.Tensor, + exp_sums_ptr: fx.Tensor, + max_logits_ptr: fx.Tensor, + query_ptr: fx.Tensor, + kv_cache_ptr: fx.Tensor, + centroids_ptr: fx.Tensor, + block_tables_ptr: fx.Tensor, + seq_lens_ptr: fx.Tensor, + ): + # ---- IDs --------------------------------------------------------- + tid = gpu.thread_idx.x + seq = gpu.block_idx.x + kv_h = gpu.block_idx.y + part = gpu.block_idx.z + lane = tid # 0..63 + mfma_row = lane & fx.Int32(15) + mfma_col_grp = lane >> fx.Int32(4) # 0..3, K-group dim + + # ---- Buffer resources ------------------------------------------- + q_rsrc = buffer_ops.create_buffer_resource(query_ptr, max_size=True) + bt_rsrc = buffer_ops.create_buffer_resource(block_tables_ptr, max_size=True) + sl_rsrc = buffer_ops.create_buffer_resource(seq_lens_ptr, max_size=True) + cent_rsrc = buffer_ops.create_buffer_resource(centroids_ptr, max_size=True) + out_rsrc = buffer_ops.create_buffer_resource(out_ptr, max_size=True) + es_rsrc = buffer_ops.create_buffer_resource(exp_sums_ptr, max_size=True) + ml_rsrc = buffer_ops.create_buffer_resource(max_logits_ptr, max_size=True) + + # ---- LDS pointers ----------------------------------------------- + base = allocator.get_base() + cent_lds = SmemPtr(base, centroid_off, T.f32, shape=(N_CENTROIDS,)) + q_lds_i32 = SmemPtr(base, q_off, T.i32, shape=(_Q_LDS_BYTES // 4,)).get() + q_lds_i64 = SmemPtr(base, q_off, T.i64, shape=(_Q_LDS_BYTES // 8,)).get() + # KV / scale views span ALL warps' private regions (W*stride); the + # per-warp base offset is folded into the element indices below via + # _kvi*/_sci (a literal 0 for W=1, so the single-warp path is unchanged). + kv_lds_i32 = SmemPtr( + base, kv_off, T.i32, shape=(KV_TILE_LDS_BYTES_PADDED // 4,) + ).get() + kv_lds_i64 = SmemPtr( + base, kv_off, T.i64, shape=(KV_TILE_LDS_BYTES_PADDED // 8,) + ).get() + scale_lds_i32 = SmemPtr(base, scale_off, T.i32, shape=(TILE_SIZE * N_GROUPS,)) + + def _kvi32(e): + return e + + def _kvi64(e): + return e + + def _sci(e): + return e + + # ---- Constants --------------------------------------------------- + c_sq = fx.Int32(_stride_q_seq) + c_qh = fx.Int32(_stride_q_head) + c_qg = fx.Int32(QG) + c_bt = fx.Int32(_stride_bt_seq) + c_stride_pos = fx.Int32(_stride_cache_pos) + c_stride_head = fx.Int32(_stride_cache_head) + c_kscale_off = fx.Int32(K_SCALES_OFFSET) # 128 + c_vcode_off = fx.Int32(V_CODES_OFFSET) # 136 + c_vscale_off = fx.Int32(V_SCALES_OFFSET) # 264 + c_w = fx.Int32(WARP_SIZE) + + NEG_INF = arith.constant(NEG_INF_VAL, type=T.f32) + ZERO_F = fx.Float32(0.0) + ONE_F = fx.Float32(1.0) + LOG2E_C = arith.constant(LOG2E, type=T.f32) + QK_SCALE = arith.constant(_qk_scale, type=T.f32) + + def _ival(v): + return v.ir_value() if hasattr(v, "ir_value") else v + + c_zero_i32 = arith.constant(0, type=T.i32) + + def _f32x8_to_fp8_i64(f): + # 8 f32 -> 8 E4M3 bytes packed as i64 (MFMA fp8 operand). + w0 = rocdl.cvt_pk_fp8_f32(T.i32, f[0], f[1], c_zero_i32, 0) + w0 = rocdl.cvt_pk_fp8_f32(T.i32, f[2], f[3], w0, 1) + w1 = rocdl.cvt_pk_fp8_f32(T.i32, f[4], f[5], c_zero_i32, 0) + w1 = rocdl.cvt_pk_fp8_f32(T.i32, f[6], f[7], w1, 1) + pv = vector.from_elements(T.vec(2, T.i32), [w0, w1]) + return vector.extract( + vector.bitcast(T.vec(1, T.i64), pv), static_position=[0] + ) + + def _bf16x8_to_fp8_i64(v_bf16): + f = [ + arith.extf(T.f32, vector.extract(v_bf16, static_position=[i])) + for i in range(8) + ] + return _f32x8_to_fp8_i64(f) + + # ===== STEP A: Load centroids → LDS (cooperative, race-safe) ===== + c_idx_safe = lane & fx.Int32(N_CENTROIDS - 1) + c_val = buffer_ops.buffer_load(cent_rsrc, c_idx_safe, vec_width=1, dtype=T.f32) + cent_lds.store(c_val, [arith.index_cast(T.index, c_idx_safe)]) + gpu.barrier() + + # ===== STEP B': in-kernel Q rotation (fused WHT) ================== + # The shipped path needs a SEPARATE prologue kernel to compute + # q_rot = Q @ PiT. That kernel is ~7 us at B=4 -- essentially ALL fixed + # per-dispatch cost (measured: 2.1M MACs, and sweeping its grid 4x moves + # it 0%). PiT is the orthonormal Sylvester Hadamard with a column + # permutation folded in on the host, so Q @ PiT is a WALSH-HADAMARD + # TRANSFORM: 8 butterfly stages (2048 add/sub per row) instead of a + # 256x256 matmul (65536 MACs) -- 32x less work, cheap enough to redo + # redundantly in every partition and delete the dispatch entirely. + # + # Verified offline: butterfly vs matmul differs by <=1.8e-6 in fp32, and + # after the E4M3 haircut the two agree EXACTLY (0/2048 elements differ), + # because e4m3 keeps only 3 mantissa bits. So this is bit-exact, not an + # approximation. + # + # Lane map: lane L owns row r = L>>3 and the 32 CONSECUTIVE head-dims + # d = (L&7)*32 + t. That puts butterfly stages 0..4 (strides 1..16) + # entirely inside one lane's registers, and leaves only stages 5..7 + # (strides 32/64/128 == chunk XOR 1/2/4) needing a cross-lane exchange. + if const_expr(_FUSE_QROT): + _wrow = lane >> fx.Int32(3) # 0..7 row within the group + _wchk = lane & fx.Int32(7) # 0..7 32-dim chunk + # inv(qperm) is a pure bit permutation of the head-dim index: + # dest = (d & 0x8F) | ((d>>2)&0x10) | ((d<<1)&0x60) + # which splits additively into a compile-time part in t and a + # per-lane part in the chunk index (verified exhaustively): + # ct(t) = (t & 0x0F) | ((t & 0x10) << 1) + # rt(c) = ((c&2)<<3) | ((c&1)<<6) | ((c&4)<<5) + _perm_base = ( + ((_wchk & fx.Int32(2)) << fx.Int32(3)) + | ((_wchk & fx.Int32(1)) << fx.Int32(6)) + | ((_wchk & fx.Int32(4)) << fx.Int32(5)) + ) + _inv_sqrtD = arith.constant(1.0 / math.sqrt(HEAD_SIZE), type=T.f32) + for _qit in range_constexpr((QG + 7) // 8): + _qrow = _wrow + fx.Int32(_qit * 8) + if _qrow < fx.Int32(QG): + _qb = ( + seq * c_sq + (kv_h * c_qg + _qrow) * c_qh + _wchk * fx.Int32(32) + ) + # ---- load this lane's 32 raw-Q head-dims (4 x dwordx4) ---- + _v = [] + for _j in range_constexpr(4): + _raw = buffer_ops.buffer_load( + q_rsrc, + (_qb + fx.Int32(_j * 8)) // fx.Int32(2), + vec_width=4, + dtype=T.i32, + ) + _rb = vector.bitcast(T.vec(8, T.bf16), _raw) + for _e in range_constexpr(8): + _v.append( + arith.extf( + T.f32, vector.extract(_rb, static_position=[_e]) + ) + ) + # ---- stages 0..4: in-register, no LDS, no cross-lane ---- + for _s in range_constexpr(5): + _st = 1 << _s + _nx = list(_v) + for _g in range_constexpr(32 // (2 * _st)): + for _k in range_constexpr(_st): + _i0 = _g * 2 * _st + _k + _i1 = _i0 + _st + _a, _b = _v[_i0], _v[_i1] + _nx[_i0] = _a + _b + _nx[_i1] = _a - _b + _v = _nx + # ---- stages 5..7: chunk XOR 1/2/4 via ds_swizzle ---- + # ds_swizzle_b32 offset (bit15=0, 32-lane groups): + # [4:0]=and_mask [9:5]=or_mask [14:10]=xor_mask + # pure XOR k -> and_mask=0x1F, or_mask=0, xor_mask=k. The + # partner is always within the same 32-lane group (k<=4). + for _s in range_constexpr(3): + _k = 1 << _s + _swz = arith.constant((_k << 10) | 0x1F, type=T.i32) + # High half of the pair computes (partner - mine), the low + # half (mine + partner). Fold that into a single sign + # computed ONCE per stage, so each element costs one + # multiply-add rather than a branch or a per-element select. + _chk_bit = _wchk & fx.Int32(_k) + _is_hi = arith.cmpi( + arith.CmpIPredicate.ne, + _chk_bit.ir_value() + if hasattr(_chk_bit, "ir_value") + else _chk_bit, + fx.Int32(0).ir_value(), + ) + _sf = arith.select( + _is_hi, + arith.constant(-1.0, type=T.f32), + arith.constant(1.0, type=T.f32), + ) + _nx = [] + for _t in range_constexpr(32): + _mi = arith.bitcast(T.i32, _v[_t]) + _pt = arith.bitcast( + T.f32, rocdl.ds_swizzle(T.i32, _mi, _swz) + ) + _nx.append(_v[_t] * _sf + _pt) + _v = _nx + # ---- 1/sqrt(D) normalization + E4M3 haircut -> bf16 ---- + # cvt_pk_fp8_f32 is the SAME instruction the QK operand build + # uses, so this reproduces the host haircut exactly. E4M3 is a + # subset of bf16, so the bf16 store below is lossless and every + # downstream consumer stays byte-identical. + _hc = [] + for _p in range_constexpr(16): + _w = rocdl.cvt_pk_fp8_f32( + T.i32, + _v[2 * _p] * _inv_sqrtD, + _v[2 * _p + 1] * _inv_sqrtD, + fx.Int32(0), + 0, + ) + _u = rocdl.cvt_pk_f32_fp8(T.vec(2, T.f32), _w, 0) + _hc.append( + arith.truncf( + T.bf16, vector.extract(_u, static_position=[0]) + ) + ) + _hc.append( + arith.truncf( + T.bf16, vector.extract(_u, static_position=[1]) + ) + ) + # ---- permuted store ---- + # ct(t) maps t=0..15 -> 0..15 and t=16..31 -> 32..47, i.e. TWO + # contiguous 16-element runs, so the scatter is still 4 aligned + # 16-byte vector stores rather than 32 scalar writes. + _qrow_elem = _qrow * fx.Int32(HEAD_SIZE) + for _half in range_constexpr(2): + for _q4 in range_constexpr(2): + _elems = [_hc[_half * 16 + _q4 * 8 + _e] for _e in range(8)] + _vec = vector.from_elements(T.vec(8, T.bf16), _elems) + _dst = ( + _qrow_elem + _perm_base + fx.Int32(_half * 32 + _q4 * 8) + ) + vector.store( + vector.bitcast(T.vec(4, T.i32), _vec), + q_lds_i32, + [arith.index_cast(T.index, _dst // fx.Int32(2))], + ) + gpu.barrier() + + # ===== STEP B: Load pre-rotated q_rot → row-major Q LDS ========== + for c in range_constexpr(0 if _FUSE_QROT else QG_LOAD_ITERS): + row_chunk = lane + fx.Int32(c * WARP_SIZE) + row = row_chunk >> fx.Int32(Q_ROW_SHIFT) # 0..QG-1 + col_b = row_chunk & fx.Int32(Q_COL_MASK) # 0..Q_GROUPS_PER_ROW-1 + col_elem = col_b * fx.Int32(8) # bf16 elem + q_off_elem = seq * c_sq + (kv_h * c_qg + row) * c_qh + col_elem + q_v = buffer_ops.buffer_load( + q_rsrc, + q_off_elem // fx.Int32(2), + vec_width=4, + dtype=T.i32, + ) + q_lds_byte = row * fx.Int32(HEAD_SIZE * 2) + col_elem * fx.Int32(2) + vector.store( + q_v, + q_lds_i32, + [arith.index_cast(T.index, q_lds_byte // fx.Int32(4))], + ) + gpu.barrier() + + # Scaled-MFMA Q operands (loop-invariant): one per MFMA issue. + q_op_hoisted = [] + for h in range_constexpr(MFMA_ISSUES): + _q_hoist_words = [] + for j in range_constexpr(4): + _q_hoist_idx = ( + mfma_row * fx.Int32(HEAD_SIZE * 2 // 8) + + (fx.Int32(h * GRPS_PER_ISSUE) + mfma_col_grp) * fx.Int32(8) + + fx.Int32(j * 2) + ) + _q_hoist_v = vector.load_op( + T.vec(2, T.i64), + q_lds_i64, + [arith.index_cast(T.index, _q_hoist_idx)], + ) + _q_hoist_words.append( + _bf16x8_to_fp8_i64(vector.bitcast(T.vec(8, T.bf16), _q_hoist_v)) + ) + q_op_hoisted.append( + vector.bitcast( + T.vec(8, T.i32), + vector.from_elements(T.vec(4, T.i64), _q_hoist_words), + ) + ) + + # ===== STEP D: Online softmax + PV state ========================= + running_max = NEG_INF + running_sum = ZERO_F + zero_v4 = arith.constant_vector(0.0, T.f32x4) + acc_pv = [zero_v4 for _ in range(PV_N_CHUNKS)] + + # ===== STEP E: Sequence-len + partition base ==================== + seq_len = buffer_ops.buffer_load(sl_rsrc, seq, vec_width=1, dtype=T.i32) + + # Per-K-tile dequant lane assignment: lane t -> token = t/4, + # chunk_in_tok = t%4 (each chunk = SUBCHUNK_HDIMS=64 head-dims = 2 groups). + tok_in_tile = lane >> fx.Int32(2) + chunk_in_tok = lane & fx.Int32(3) + # Both UE8M0 group bytes for this lane's chunk (groups 2c, 2c+1) live in + # the SAME 4-byte scale word at offset (chunk_in_tok>>1)*4 within the + # 8-byte scale region. Load once; extract per-half below. + c_scale_word_off = (chunk_in_tok >> fx.Int32(1)) * fx.Int32(4) + + # ===== STEP F: K-tile loop ======================================= + c_kcb = fx.Int32(KV_COMPUTE_BLOCK) + c_tgpp = fx.Int32(TGPP) + c_zero_i32 = fx.Int32(0) + c_one_i32 = fx.Int32(1) + c_nparts = fx.Int32(NUM_PARTS_C) + total_tgs = (seq_len + c_kcb - c_one_i32) // c_kcb + avail = total_tgs - part + in_range = avail > c_zero_i32 + trip_raw = (avail + c_nparts - c_one_i32) // c_nparts + trip_clamped = (trip_raw > c_tgpp).select(c_tgpp, trip_raw) + trip_or_zero = in_range.select(trip_clamped, c_zero_i32) + c_zero_idx = arith.constant(0, index=True) + c_one_idx = arith.constant(1, index=True) + trip_idx = arith.index_cast( + T.index, + trip_or_zero.ir_value() + if hasattr(trip_or_zero, "ir_value") + else trip_or_zero, + ) + _init_iter = [ + _ival(running_max), + _ival(running_sum), + *[_ival(p) for p in acc_pv], + ] + _for_op = _scf.ForOp( + c_zero_idx, + trip_idx, + c_one_idx, + _init_iter, + ) + _for_ip = ir.InsertionPoint(_for_op.body) + _for_ip.__enter__() + try: + tg_idx = _for_op.induction_variable + tg_i32 = fx.Int32(arith.index_cast(T.i32, tg_idx)) + partition_start = (part + tg_i32 * c_nparts) * c_kcb + bt_seq_base = seq * c_bt + (partition_start // fx.Int32(_BS)) + running_max = _for_op.inner_iter_args[0] + running_sum = _for_op.inner_iter_args[1] + acc_pv = list(_for_op.inner_iter_args[2:]) + + # ==== SOFTWARE PREFETCH ===================================== + # Emit ONLY the HBM buffer_loads for one tile (K codes, V codes, + # K/V UE8M0 scale words) and return the loaded register values. + # Emitting tile N+1's loads BEFORE tile N's dequant+MFMA lets the + # vmem fetches overlap compute: the compiler's s_waitcnt for these + # values lands at their first use one iteration later, hiding HBM + # latency (the decode kernel is ~85% HBM-stalled at low batch). + # This is bit-identical to the pre-prefetch path -- only the + # *emission order* of independent loads changes, never any + # arithmetic. All addressing locals (slot_base_byte, blk_rsrc, + # ...) are consumed only inside this closure; the dequant below + # needs just the four returned register values. + def _emit_tile_loads(n_tile): + # n_tile is a compile-time int on the single-warp path and a + # runtime fx.Int32 on the multi-warp path (each warp strides the + # 16 tiles). Compute the block/tile/slot geometry accordingly. + if const_expr(isinstance(n_tile, int)): + block_in_part = fx.Int32(n_tile // _TILES_PER_BLOCK) + tile_in_block = n_tile % _TILES_PER_BLOCK + tile_start_tok = partition_start + fx.Int32(n_tile * TILE_SIZE) + slot = fx.Int32(tile_in_block * TILE_SIZE) + tok_in_tile + else: + block_in_part = n_tile // fx.Int32(_TILES_PER_BLOCK) + tile_in_block = n_tile - block_in_part * fx.Int32(_TILES_PER_BLOCK) + tile_start_tok = partition_start + n_tile * fx.Int32(TILE_SIZE) + slot = tile_in_block * fx.Int32(TILE_SIZE) + tok_in_tile + tile_in_seq = tile_start_tok < seq_len + bt_off = bt_seq_base + block_in_part + bt_off_safe = tile_in_seq.select(bt_off, seq * c_bt) + phys_block = buffer_ops.buffer_load( + bt_rsrc, + bt_off_safe, + vec_width=1, + dtype=T.i32, + ) + # A buffer descriptor addresses with a 32-bit voffset, so + # ``phys_block * c_block`` wraps once the cache view exceeds + # 4 GiB. Fold the block base into the descriptor's 64-bit + # base pointer and keep only in-block offsets below. + blk_off_i64 = arith.extui(T.i64, phys_block) * fx.Int64( + _stride_cache_block + ) + blk_rsrc = buffer_ops.create_buffer_resource( + kv_cache_ptr, + max_size=True, + base_byte_offset=blk_off_i64, + ) + block_base = fx.Int32(0) + + slot_base_byte = block_base + slot * c_stride_pos + kv_h * c_stride_head + # K codes: LANE_CODE_BYTES (=32) at slot_base + chunk*32, in + # HALVES x 16-byte buffer_loads. + k_byte0 = slot_base_byte + chunk_in_tok * fx.Int32(LANE_CODE_BYTES) + _kpl = [] + for hf in range_constexpr(HALVES): + _kpl.append( + buffer_ops.buffer_load( + blk_rsrc, + (k_byte0 + fx.Int32(hf * 16)) // fx.Int32(4), + vec_width=4, + dtype=T.i32, + cache_modifier=_KV_AUX, + ) + ) + + # ---- V codes + K/V scale HBM loads ------------------------ + v_byte0 = ( + slot_base_byte + + c_vcode_off + + chunk_in_tok * fx.Int32(LANE_CODE_BYTES) + ) + _vpl = [] + for hf in range_constexpr(HALVES): + _vpl.append( + buffer_ops.buffer_load( + blk_rsrc, + (v_byte0 + fx.Int32(hf * 16)) // fx.Int32(4), + vec_width=4, + dtype=T.i32, + cache_modifier=_KV_AUX, + ) + ) + # UE8M0 group scale words (one 4-byte word covers this lane's + # two groups). scale = 2^(byte-127) = bitcast_f32(byte<<23); + # byte==0 -> +0.0 (zero sentinel). + _ksw = buffer_ops.buffer_load( + blk_rsrc, + (slot_base_byte + c_kscale_off + c_scale_word_off) // fx.Int32(4), + vec_width=1, + dtype=T.i32, + ) + _vsw = buffer_ops.buffer_load( + blk_rsrc, + (slot_base_byte + c_vscale_off + c_scale_word_off) // fx.Int32(4), + vec_width=1, + dtype=T.i32, + ) + return (_kpl, _vpl, _ksw, _vsw) + + # Depth-1 software prefetch: emit the next tile's HBM loads before + # consuming the current tile without changing arithmetic order. + _PF_DEPTH = 1 + # Prime the pipeline with the first _PF_DEPTH tiles' loads. + _pf_queue = [] + for _pi in range_constexpr(min(_PF_DEPTH, 16)): + _pf_queue.append(_emit_tile_loads(_pi)) + _tile_range = range_constexpr(16) + for _jt in _tile_range: + n_tile = _jt + _ntile_tok = fx.Int32(n_tile * TILE_SIZE) + (k_packed_list, v_packed_list, kscale_word, vscale_word) = ( + _pf_queue.pop(0) + ) + _pf_nxt = n_tile + _PF_DEPTH + if const_expr(_pf_nxt < 16): + _pf_queue.append(_emit_tile_loads(_pf_nxt)) + # Store raw FP4 codes (natural contiguous order, + # 16 B = 32 nibbles per UE8M0 group) straight to KV LDS as + # the scaled-MFMA A operand — no dequant here. + for hf in range_constexpr(HALVES): + grp = chunk_in_tok * fx.Int32(2) + fx.Int32(hf) + grp_shift = (grp & fx.Int32(3)) * fx.Int32(8) + kscale_byte = (kscale_word >> grp_shift) & fx.Int32(0xFF) + k_packed = k_packed_list[hf] + vector.store( + k_packed, + kv_lds_i32, + [ + arith.index_cast( + T.index, + _kvi32( + tok_in_tile * fx.Int32(KFP4_ROW_I32) + + grp * fx.Int32(4) + ), + ) + ], + ) + scale_lds_i32.store( + kscale_byte, + [ + arith.index_cast( + T.index, + _sci(tok_in_tile * fx.Int32(N_GROUPS) + grp), + ) + ], + ) + gpu.barrier() + # QK: MFMA_ISSUES native scaled MFMAs over K=128, + # chained through the accumulator. + # A = K raw FP4 codes (vec4 i32 = 16 B) per (token, group) + # B = Q E4M3 (vec8 i32 = 32 B) for the same group + # scaleA = per-(token,group) UE8M0 byte at op_sel byte 0 + # scaleB = 0x7F identity (Q carries no scale) + BLGP_E2M1 = 4 # A operand type code = fp4 e2m1 + CBSZ_E4M3 = 0 # B operand type code = fp8 e4m3 + IDENT = fx.Int32(0x7F) + qk_acc = zero_v4 + for h in range_constexpr(MFMA_ISSUES): + grp = fx.Int32(h * GRPS_PER_ISSUE) + mfma_col_grp + k_op = vector.load_op( + T.vec(4, T.i32), + kv_lds_i32, + [ + arith.index_cast( + T.index, + _kvi32( + mfma_row * fx.Int32(KFP4_ROW_I32) + + grp * fx.Int32(4) + ), + ) + ], + ) + q_op = q_op_hoisted[h] + scbyte = fx.Int32( + scale_lds_i32.load( + [ + arith.index_cast( + T.index, + _sci(mfma_row * fx.Int32(N_GROUPS) + grp), + ) + ] + ) + ) + kscale = fx.Int32(0x7F7F7F00) | scbyte + qk_acc = rocdl.mfma_scale_f32_16x16x128_f8f6f4( + T.vec(4, T.f32), + [ + k_op, + q_op, + qk_acc, + BLGP_E2M1, + CBSZ_E4M3, + 0, + kscale, + 0, + IDENT, + ], + ) + + # Scale + mask out-of-context tokens. + qk_acc = _vsplat_mul(qk_acc, QK_SCALE) + for elem in range_constexpr(4): + kv_tok = ( + partition_start + + _ntile_tok + + mfma_col_grp * fx.Int32(4) + + fx.Int32(elem) + ) + in_b = kv_tok < seq_len + v = vector.extract(qk_acc, static_position=[elem]) + qk_acc = _vector_insert( + in_b.select(v, NEG_INF), + qk_acc, + static_position=[elem], + ) + + # FA2 online softmax: per-query-row reduce. + local_max = vector.reduction(T.f32, "maxnumf", qk_acc) + r1 = local_max.shuffle_xor(fx.Int32(16), c_w) + local_max = local_max.maximumf(r1) + r2 = local_max.shuffle_xor(fx.Int32(32), c_w) + tile_max = local_max.maximumf(r2) + + new_max = running_max.maximumf(tile_max) + max_diff = running_max - new_max + safe_diff = (running_max > NEG_INF).select(max_diff, ZERO_F) + scale = (safe_diff * LOG2E_C).exp2(fastmath=arith.FastMathFlags.fast) + running_sum = running_sum * scale + for h in range_constexpr(PV_N_CHUNKS): + acc_pv[h] = _vsplat_mul(acc_pv[h], scale) + running_max = new_max + + tile_sum = ZERO_F + for elem in range_constexpr(4): + s = vector.extract(qk_acc, static_position=[elem]) + d = s - new_max + d = (new_max > NEG_INF).select(d, NEG_INF) + p = (d * LOG2E_C).exp2(fastmath=arith.FastMathFlags.fast) + tile_sum = tile_sum + p + qk_acc = _vector_insert(p, qk_acc, static_position=[elem]) + + ts1 = tile_sum.shuffle_xor(fx.Int32(16), c_w) + tile_sum = tile_sum + ts1 + ts2 = tile_sum.shuffle_xor(fx.Int32(32), c_w) + tile_sum = tile_sum + ts2 + running_sum = running_sum + tile_sum + # V dequant → LDS [token][head_dim] (ROW-MAJOR, no transpose) + # written as ds_write_b128 (vec(2,i64)); read back via the + # ds_read_tr16_b64 HW transpose in the PV block below. + v_lds_elem_base = tok_in_tile * fx.Int32( + KV_ROW_ELEMS + ) + chunk_in_tok * fx.Int32(SUBCHUNK_HDIMS) + for hf in range_constexpr(HALVES): + v_packed = v_packed_list[hf] + half_hd = fx.Int32(hf * HALF_HDIMS) + grp_shift = ( + (chunk_in_tok * fx.Int32(2) + fx.Int32(hf)) & fx.Int32(3) + ) * fx.Int32(8) + vscale_byte = (vscale_word >> grp_shift) & fx.Int32(0xFF) + vscale_f32 = arith.bitcast( + T.f32, _ival(vscale_byte << fx.Int32(23)) + ) + for w in range_constexpr(4): + word_i32 = vector.extract(v_packed, static_position=[w]) + # Native CDNA4 scaled convert: 2-wide + # cvt_scalef32_pk_bf16_fp4. srcSel s picks byte s + # of the word (nibbles 2s, 2s+1) → 2 bf16 with the + # UE8M0 group scale fused. 4 cvt calls/word. + bf16_elems = [] + for s in range_constexpr(4): + v2 = rocdl.cvt_scalef32_pk_bf16_fp4( + T.vec(2, T.bf16), + _ival(word_i32), + _ival(vscale_f32), + int(s), + ) + bf16_elems.append(vector.extract(v2, static_position=[0])) + bf16_elems.append(vector.extract(v2, static_position=[1])) + v_bf16 = vector.from_elements(T.vec(8, T.bf16), bf16_elems) + v_i64 = vector.bitcast(T.vec(2, T.i64), v_bf16) + v_lds_i64_idx = ( + v_lds_elem_base + half_hd + fx.Int32(w * 8) + ) // fx.Int32(4) + vector.store( + v_i64, + kv_lds_i64, + [arith.index_cast(T.index, _kvi64(v_lds_i64_idx))], + ) + # HW V transpose cross-lane LDS fence (see 128 kernel). + rocdl.sched_barrier(0) + rocdl.s_waitcnt(0xC07F) + gpu.barrier() + + # ---- PV MFMA: A=V[head_dim, token], B=P (=qk_acc bf16) ----- + p_bf16 = arith.trunc_f(T.vec(4, T.bf16), qk_acc) + p_op = vector.bitcast(T.vec(4, T.i16), p_bf16) + # HW-transpose PV: V_lds row-major V[token][head_dim]. + token_idx = lane >> fx.Int32(2) + hd_sub = (lane & fx.Int32(3)) * fx.Int32(4) + v_lane_byte = ( + fx.Int32(kv_off) + + token_idx * fx.Int32(KV_ROW_BYTES) + + hd_sub * fx.Int32(2) + ) + for h in range_constexpr(PV_N_CHUNKS): + v_byte_off = v_lane_byte + fx.Int32(h * 32) + v_byte_i64 = fx.Int64(v_byte_off) + v_ptr = buffer_ops.create_llvm_ptr( + v_byte_i64, + address_space=3, + ) + v_op_raw = rocdl.ds_read_tr16_b64( + T.vec(4, T.i16), + v_ptr, + ).result + acc_pv[h] = rocdl.mfma_f32_16x16x16bf16_1k( + T.f32x4, [v_op_raw, p_op, acc_pv[h], 0, 0, 0] + ) + + _scf.YieldOp( + [ + _ival(running_max), + _ival(running_sum), + *[_ival(p) for p in acc_pv], + ] + ) + finally: + _for_ip.__exit__(None, None, None) + running_max = _for_op.results[0] + running_sum = _for_op.results[1] + acc_pv = list(_for_op.results[2:]) + + # ===== STEP G: Output =========================================== + safe_sum = (running_sum > ZERO_F).select(running_sum, ONE_F) + rcp = ONE_F / safe_sum + + c_os = fx.Int32(_stride_out_seq) + c_oh = fx.Int32(_stride_out_head) + c_op_ = fx.Int32(_stride_out_part) + out_base = seq * c_os + kv_h * c_oh + part * c_op_ + + valid_row_pred = arith.cmpi( + arith.CmpIPredicate.ult, + mfma_row.ir_value() if hasattr(mfma_row, "ir_value") else mfma_row, + arith.constant(QG, type=T.i32), + ) + _if = _scf.IfOp(valid_row_pred) + with ir.InsertionPoint(_if.then_block): + for h in range_constexpr(PV_N_CHUNKS): + pv_norm = _vsplat_mul(acc_pv[h], rcp) + pv_bf16 = arith.trunc_f(T.vec(4, T.bf16), pv_norm) + pv_i32x2 = vector.bitcast(T.vec(2, T.i32), pv_bf16) + head_dim_start = fx.Int32(h * 16) + mfma_col_grp * fx.Int32(4) + out_off_elem = ( + out_base + mfma_row * fx.Int32(HEAD_SIZE) + head_dim_start + ) + buffer_ops.buffer_store( + pv_i32x2, + out_rsrc, + out_off_elem * fx.Int32(2), + offset_is_bytes=True, + ) + + c_npq = fx.Int32(num_partitions * QG) + ml_off = ( + seq * fx.Int32(_stride_ml_seq) + kv_h * c_npq + part * c_qg + mfma_row + ) + es_off = ( + seq * fx.Int32(_stride_es_seq) + kv_h * c_npq + part * c_qg + mfma_row + ) + buffer_ops.buffer_store(running_max, ml_rsrc, ml_off) + buffer_ops.buffer_store(running_sum, es_rsrc, es_off) + _scf.YieldOp([]) + + # Hand the build-local allocator to the launcher. `_get_kernel` reads + # `kmod.allocator` immediately after this returns (before the next build can + # run), so each launcher captures its own variant's allocator. This does NOT + # reintroduce the drift bug: the kernel above closes over the LOCAL + # `allocator`, so no @flyc.jit function references module-global `allocator` + # and it is never snapshotted by _check_globals_drift. + globals()["allocator"] = allocator + return ultraquant_decode_hd256_kernel diff --git a/vllm/v1/attention/ops/flydsl_ultraquant_decode.py b/vllm/v1/attention/ops/flydsl_ultraquant_decode.py new file mode 100644 index 0000000000..7f8c720678 --- /dev/null +++ b/vllm/v1/attention/ops/flydsl_ultraquant_decode.py @@ -0,0 +1,834 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""FlyDSL launcher for the optimized UltraQuant 4-bit D=256 decode kernel. + +The production kernel uses scaled FP4×E4M3 QK MFMA, native V conversion, +query hoisting, in-kernel Walsh-Hadamard rotation for GQA-6/8/16, and strided +tile-group scheduling. These choices are fixed and are not environment-tunable. +""" + +from __future__ import annotations + +from typing import Any + +import torch + +from vllm.logger import init_logger +from vllm.triton_utils import triton + +logger = init_logger(__name__) + +_FLYDSL_AVAILABLE: bool | None = None +_ULTRAQUANT_MOD: Any = None +_FLYC: Any = None +_FX: Any = None +_TYPING_T: Any = None +_CC: Any = None +_IR: Any = None + + +def is_ultraquant_flydsl_available() -> bool: + """Return whether the gfx950 UltraQuant FlyDSL kernel can be loaded.""" + global _FLYDSL_AVAILABLE, _ULTRAQUANT_MOD + global _FLYC, _FX, _TYPING_T, _CC, _IR + if _FLYDSL_AVAILABLE is not None: + return _FLYDSL_AVAILABLE + try: + from vllm.platforms.rocm import on_gfx950 + + if not on_gfx950(): + _FLYDSL_AVAILABLE = False + return False + + import flydsl.compiler as flyc + import flydsl.expr as fx + from flydsl._mlir import ir + from flydsl.compiler.kernel_function import CompilationContext + from flydsl.expr.typing import T + + from vllm.v1.attention.ops.flydsl_kernels import ( + ultraquant_decode_hd256 as ultraquant_mod, + ) + + _FLYC = flyc + _FX = fx + _TYPING_T = T + _CC = CompilationContext + _IR = ir + _ULTRAQUANT_MOD = ultraquant_mod + _FLYDSL_AVAILABLE = True + logger.info_once("FlyDSL UltraQuant D=256 decode is available") + except Exception as ex: # noqa: BLE001 + _FLYDSL_AVAILABLE = False + logger.warning_once( + "FlyDSL UltraQuant decode is unavailable (%s); using Triton fallback.", + ex, + ) + return _FLYDSL_AVAILABLE + + +_ULTRAQUANT_FLYDSL_GQA = (6, 8, 16) + + +def ultraquant_flydsl_decode_eligible( + *, + head_size: int, + num_kv_groups: int, + has_sinks: bool, + sliding_window: int | None, + flydsl_loaded: bool, +) -> bool: + """Return whether decode should use the gfx950 FlyDSL kernel. + + Production FlyDSL covers D=256 and GQA in {6, 8, 16}. Sinks and sliding + window fall back to unified Triton. ``flydsl_loaded`` is the process-level + compiler/hardware probe so this policy can be unit-tested without a GPU. + """ + return ( + flydsl_loaded + and head_size == 256 + and num_kv_groups in _ULTRAQUANT_FLYDSL_GQA + and not has_sinks + and not (sliding_window and sliding_window > 0) + ) + + +# -- Kernel module cache ------------------------------------------------------- +# Compiling a FlyDSL kernel is expensive; cache by the constexprs that +# parameterize stride math: num_kv_heads, num_partitions, max_blocks_per_seq, +# scale, query_group_size, kv_block_size, padded_slot, hw_v_transpose, +# tile_groups_per_partition. +_KERN_CACHE: dict[tuple, Any] = {} + +_LOG_INVOKED_ONCE: bool = False + +# Per-device cached FP4 E2M1 value table (used as the kernel "centroids" LUT). +_FP4_LUT_CACHE: dict[tuple[str, torch.dtype], torch.Tensor] = {} + + +def _fp4_value_lut(device: torch.device) -> torch.Tensor: + """Return the fixed 16-entry FP4 E2M1 value table as fp32 [16] on device. + + Indexed by the stored 4-bit E2M1 code, so ``lut[nibble]`` is the dequant + value directly (matches ``format.FP4_BITS_TO_VALUE``). + """ + from vllm.v1.attention.ops.ultraquant.format import FP4_BITS_TO_VALUE + + key = (str(device), torch.float32) + t = _FP4_LUT_CACHE.get(key) + if t is None: + t = torch.tensor( + FP4_BITS_TO_VALUE, dtype=torch.float32, device=device + ).contiguous() + _FP4_LUT_CACHE[key] = t + return t + + # Per-device cached qperm head-dim permutation for scaled QK MFMA. + + +# qperm[phys] = the natural head-dim placed at physical operand position phys. +# It realigns the fixed FP4xFP8 scaled-MFMA contraction so that, with K fed +# native-contiguous, Q's columns pair correctly. Folded into PiT on the host +# (PiT[:, qperm]) so the Q rotation matmul emits already-permuted q_rot at zero +# extra runtime cost. Derived + validated in tmp_qk_scaled_mfma.py (contigK). +_QPERM_CACHE: dict[str, torch.Tensor] = {} + + +def _kmap_phys(group: int, s: int) -> int: + # Physical operand position of group's element s for the K=128 fp4/fp8 + # scaled MFMA (matches tmp_qk_scaled_mfma.py::_kmap). + return 32 * (s >> 4) + 64 * (group % 2) + 16 * (group // 2) + (s & 15) + + +def _qperm_index( + device: torch.device, D: int = 128, group_size: int = 32 +) -> torch.Tensor: + # _kmap_phys describes ONE K=128 contraction (4 groups of 32). A head dim + # wider than 128 is covered by ceil(D/128) back-to-back MFMA issues, each + # contracting its own independent 128-wide slice, so the same map repeats + # per slice: qperm = [qperm128, qperm128 + 128, ...]. Applying _kmap_phys + # with group >= 4 instead yields positions outside the slice and is not a + # permutation at all. + key = f"{device}:{D}:{group_size}" + t = _QPERM_CACHE.get(key) + if t is None: + k_per_mfma = 128 + assert D % k_per_mfma == 0, ( + f"scaled QK MFMA needs D to be a multiple of {k_per_mfma}, got {D}" + ) + groups_per_mfma = k_per_mfma // group_size + qperm = [0] * D + for half in range(D // k_per_mfma): + base = half * k_per_mfma + for g in range(groups_per_mfma): + for s in range(group_size): + qperm[base + _kmap_phys(g, s)] = base + g * group_size + s + assert sorted(qperm) == list(range(D)), "qperm is not a permutation" + t = torch.tensor(qperm, dtype=torch.long, device=device).contiguous() + _QPERM_CACHE[key] = t + return t + + +def _detect_max_capture_B() -> int: + try: + from vllm.config import get_current_vllm_config + + cfg = get_current_vllm_config() + candidates: list[int] = [] + sizes = cfg.compilation_config.cudagraph_capture_sizes + if sizes: + candidates.append(int(max(sizes))) + sched = getattr(cfg, "scheduler_config", None) + if sched is not None and getattr(sched, "max_num_seqs", None): + candidates.append(int(sched.max_num_seqs)) + if candidates: + return max(candidates) + except Exception: # noqa: BLE001 + pass + return 512 + + +class _SegmBufPool: + """Single-bucket buffer pool for segm_out/segm_max/segm_sum/output + the + pooled Q-rotation intermediates. Identical to the TQ pool except that + ultraquant needs an extra ``q_fp8`` (float8_e4m3fn) slot for the Q precision + haircut (``q_rot_fp32 -> fp8_e4m3 -> bf16``) so that no fresh allocation + lands in the HIP graph memory pool post-capture. + """ + + __slots__ = ("_bufs", "_max_B") + + def __init__(self) -> None: + self._bufs: dict[tuple, dict[str, torch.Tensor]] = {} + self._max_B: int | None = None + + def get( + self, + B: int, + Hk: int, + Hq: int, + num_partitions: int, + QG: int, + D: int, + device: torch.device, + q_dtype: torch.dtype, + ) -> dict[str, torch.Tensor]: + if self._max_B is None: + self._max_B = _detect_max_capture_B() + B_bucket = max(self._max_B, int(B)) + key = ( + int(Hk), + int(Hq), + int(num_partitions), + int(QG), + int(D), + str(device), + q_dtype, + ) + bufs = self._bufs.get(key) + if bufs is None or bufs["segm_out"].shape[0] < B_bucket: + if bufs is not None and bufs["segm_out"].shape[0] < B_bucket: + # First-time grow: warn so user knows cudagraphs may be + # invalidated. This should only happen during warmup or in + # always-eager configs (a batch larger than any captured + # cudagraph size). + logger.warning_once( + "FlyDSL UltraQuant buffer pool growing from %d " + "to %d (B=%d). If this happens AFTER cudagraph warmup the " + "captured graphs hold stale pointers and will GPU-fault.", + bufs["segm_out"].shape[0], + B_bucket, + B, + ) + bufs = { + "segm_out": torch.empty( + (B_bucket, Hk, num_partitions, QG, D), + dtype=torch.bfloat16, + device=device, + ), + "segm_max": torch.empty( + (B_bucket, Hk, num_partitions, QG), + dtype=torch.float32, + device=device, + ), + "segm_sum": torch.empty( + (B_bucket, Hk, num_partitions, QG), + dtype=torch.float32, + device=device, + ), + "output": torch.empty( + (B_bucket, Hq, D), + dtype=q_dtype, + device=device, + ), + "q_rot": torch.empty( + (B_bucket, Hq, D), + dtype=q_dtype, + device=device, + ), + "q_float": torch.empty( + (B_bucket, Hq, D), + dtype=torch.float32, + device=device, + ), + "q_rot_fp32": torch.empty( + (B_bucket, Hq, D), + dtype=torch.float32, + device=device, + ), + # ultraquant-specific: E4M3 haircut intermediate (pooled so the + # cast does not allocate a fresh tensor post-capture). + "q_fp8": torch.empty( + (B_bucket, Hq, D), + dtype=torch.float8_e4m3fn, + device=device, + ), + } + self._bufs[key] = bufs + self._max_B = B_bucket + logger.info_once( + "FlyDSL UltraQuant buffer pool allocated " + "shape=(Hk=%d, Hq=%d, P=%d, QG=%d, D=%d, dtype=%s) " + "B_bucket=%d. VRAM = %.1f MiB / shape.", + Hk, + Hq, + num_partitions, + QG, + D, + q_dtype, + B_bucket, + sum(t.numel() * t.element_size() for t in bufs.values()) / (1 << 20), + ) + return { + "segm_out": bufs["segm_out"][:B], + "segm_max": bufs["segm_max"][:B], + "segm_sum": bufs["segm_sum"][:B], + "output": bufs["output"][:B], + "q_rot": bufs["q_rot"][:B], + "q_float": bufs["q_float"][:B], + "q_rot_fp32": bufs["q_rot_fp32"][:B], + "q_fp8": bufs["q_fp8"][:B], + } + + +_SEGM_POOL = _SegmBufPool() + +_GET_KERNEL_STATS = {"hits": 0, "misses": 0, "build_total_s": 0.0} + + +def _get_kernel( + num_kv_heads: int, + num_partitions: int, + max_blocks_per_seq: int, + scale: float, + query_group_size: int, + kv_block_size: int, + padded_slot: int, + fuse_qrot: bool, + num_seqs_hint: int, + tile_groups_per_partition: int, + stride_q_seq: int, + stride_q_head: int, +): + key = ( + num_kv_heads, + num_partitions, + max_blocks_per_seq, + round(scale, 8), + query_group_size, + kv_block_size, + padded_slot, + fuse_qrot, + tile_groups_per_partition, + stride_q_seq, + stride_q_head, + ) + cached = _KERN_CACHE.get(key) + if cached is not None: + _GET_KERNEL_STATS["hits"] += 1 + return cached + + assert is_ultraquant_flydsl_available() + import time + + build_start = time.perf_counter() + kmod = _ULTRAQUANT_MOD + assert kmod is not None + kfn = kmod.build_ultraquant_decode_hd256_module( + num_seqs=num_seqs_hint, + num_kv_heads=num_kv_heads, + num_partitions=num_partitions, + padded_slot=padded_slot, + max_blocks_per_seq=max_blocks_per_seq, + softmax_scale=scale, + query_group_size=query_group_size, + kv_block_size=kv_block_size, + tile_groups_per_partition=tile_groups_per_partition, + fuse_qrot=fuse_qrot, + stride_q_seq=stride_q_seq, + stride_q_head=stride_q_head, + ) + allocator = kmod.allocator + flyc = _FLYC + fx = _FX + T = _TYPING_T + CompilationContext = _CC + ir_mod = _IR + + @flyc.jit + def _launch( + out, + es, + ml, + q, + kvc, + cents, + bt, + sl, + gx: fx.Int32, # type: ignore[name-defined] + gy: fx.Int32, # type: ignore[name-defined] + gz: fx.Int32, # type: ignore[name-defined] + stream: fx.Stream, # type: ignore[name-defined] + ): + allocator.finalized = False + ctx = CompilationContext.get_current() + with ir_mod.InsertionPoint(ctx.gpu_module_body): + allocator.finalize() + from flydsl.expr import arith + + grid_x = arith.index_cast(T.index, gx.ir_value()) + grid_y = arith.index_cast(T.index, gy.ir_value()) + grid_z = arith.index_cast(T.index, gz.ir_value()) + kfn(out, es, ml, q, kvc, cents, bt, sl).launch( + grid=(grid_x, grid_y, grid_z), + block=(kmod.BLOCK_THREADS, 1, 1), + stream=stream, + ) + + _KERN_CACHE[key] = _launch + build_time = time.perf_counter() - build_start + _GET_KERNEL_STATS["misses"] += 1 + _GET_KERNEL_STATS["build_total_s"] += build_time + logger.info( + "Built FlyDSL UltraQuant kernel #%d in %.2fs (total %.1fs), key=%s", + _GET_KERNEL_STATS["misses"], + build_time, + _GET_KERNEL_STATS["build_total_s"], + key, + ) + return _launch + + +# -- Partition reducer --------------------------------------------------------- +from vllm.triton_utils import tl # noqa: E402 + + +@triton.jit +def _reduce_partitions( + output_ptr, # [N, Hq, D] in OUT_DTYPE + segm_out_ptr, # [N, Hk, P, QG, D] bf16 + segm_max_ptr, # [N, Hk, P, QG] fp32 + segm_sum_ptr, # [N, Hk, P, QG] fp32 + out_stride_n: tl.int64, + out_stride_h: tl.int64, + NUM_KV_HEADS: tl.constexpr, + QG: tl.constexpr, + NUM_PARTS: tl.constexpr, + HEAD_SIZE: tl.constexpr, + BLOCK_D: tl.constexpr, +): + n = tl.program_id(0) + hq = tl.program_id(1) + kv_h = hq // QG + qg = hq % QG + # Splitting the head-dim across programs is what gives this kernel enough + # parallelism to use the GPU: the reduction runs over PARTITIONS, so every + # head-dim column is independent. A (B, Hq) grid is only 32 workgroups at + # B=4 -- ~12% of a 256-CU part -- which is why it moved 8.4 MB at just + # ~995 GB/s. The partition reduction itself is untouched (same values, same + # order), so the result is bit-identical; only the columns each program + # owns change. The [P] max/sum vectors get re-read per D-block, which is + # 4 KB of duplicated traffic against a 262 KB tile. + + msum_base = n * (NUM_KV_HEADS * NUM_PARTS * QG) + kv_h * (NUM_PARTS * QG) + qg + so_base = ( + n * (NUM_KV_HEADS * NUM_PARTS * QG * HEAD_SIZE) + + kv_h * (NUM_PARTS * QG * HEAD_SIZE) + + qg * HEAD_SIZE + ) + + p_off = tl.arange(0, NUM_PARTS) + d_off = tl.program_id(2) * BLOCK_D + tl.arange(0, BLOCK_D) + + m_idx = msum_base + p_off * QG + seg_max = tl.load(segm_max_ptr + m_idx) + seg_sum = tl.load(segm_sum_ptr + m_idx) + + valid = seg_max > float("-inf") + overall_max = tl.max(tl.where(valid, seg_max, float("-inf"))) + + rescale = tl.where(valid, tl.exp(seg_max - overall_max), 0.0) + seg_sum_rescaled = seg_sum * rescale + overall_sum = tl.sum(seg_sum_rescaled) + + so_idx = so_base + p_off[:, None] * (QG * HEAD_SIZE) + d_off[None, :] + seg_out = tl.load(segm_out_ptr + so_idx).to(tl.float32) + weighted = seg_out * seg_sum_rescaled[:, None] + acc_sum = tl.sum(weighted, axis=0) + acc = tl.where(overall_sum > 0.0, acc_sum / overall_sum, 0.0) + + out_off = n * out_stride_n + hq * out_stride_h + d_off + tl.store(output_ptr + out_off, acc.to(output_ptr.dtype.element_ty)) + + +# -- Fused Q rotation + E4M3 haircut (Triton) --------------------------------- +# Replaces the 4-op torch prologue (copy bf16->fp32, mm by PiT, copy fp32->e4m3, +# copy e4m3->bf16) with ONE kernel. The win is kernel COUNT, not bytes: each of +# those small ops costs ~2-4us of fixed GPU-side cost here largely independent of +# element count (verified: sweeping the pool's B_bucket 512->1, i.e. 512x fewer +# elements in the copies, moved the floor 0%). Under CUDA graphs the host launch +# cost is already gone, so only per-kernel GPU cost remains and collapsing 4 +# kernels into 1 is the lever that is left. +@triton.jit +def _fused_q_rot_haircut( + q_ptr, # [B, Hq, D] query dtype (bf16/fp16), strided + pit_ptr, # [D, D] fp32 contiguous (already qperm-permuted) + out_ptr, # [B_bucket, Hq, D] out dtype, contiguous + M, # B * Hq (runtime) + q_stride_n: tl.int64, + q_stride_h: tl.int64, + HQ: tl.constexpr, + D: tl.constexpr, + HAIRCUT: tl.constexpr, # round through E4M3 (fp8-consuming QK MFMA paths) + BLOCK_M: tl.constexpr, + BLOCK_K: tl.constexpr, + BLOCK_D: tl.constexpr, +): + pid_m = tl.program_id(0) + pid_d = tl.program_id(1) + m_off = pid_m * BLOCK_M + tl.arange(0, BLOCK_M) + m_mask = m_off < M + n = m_off // HQ + h = m_off % HQ + + # Splitting the OUTPUT columns across programs (not just the rows) is what + # gives this kernel parallelism at small batch: M = B*Hq is only 8 at B=1, + # so a row-only grid launches ONE workgroup and serializes the whole + # 256x256 rotation on a single CU. The K reduction is untouched by the + # split, so the result stays bit-identical. + d_off = pid_d * BLOCK_D + tl.arange(0, BLOCK_D) + acc = tl.zeros([BLOCK_M, BLOCK_D], dtype=tl.float32) + for k0 in range(0, D, BLOCK_K): + k_off = k0 + tl.arange(0, BLOCK_K) + q_idx = n[:, None] * q_stride_n + h[:, None] * q_stride_h + k_off[None, :] + q_blk = tl.load(q_ptr + q_idx, mask=m_mask[:, None], other=0.0) + p_blk = tl.load(pit_ptr + k_off[:, None] * D + d_off[None, :]) + acc = tl.dot(q_blk.to(tl.float32), p_blk, acc) + + if HAIRCUT: + # fp32 -> E4M3 -> fp32. Mirrors the torch path's + # q_rot_fp32 -> float8_e4m3fn -> bf16 round trip (e4m3 is a subset of + # bf16, so the second hop is exact and folds away here). + acc = acc.to(tl.float8e4nv).to(tl.float32) + + out_idx = m_off[:, None] * D + d_off[None, :] + tl.store(out_ptr + out_idx, acc.to(out_ptr.dtype.element_ty), mask=m_mask[:, None]) + + +# Prologue Q-rotation fusion (fallback for non-hd256 / in-kernel-WHT-off paths). +# Cleared the accuracy ladder (offline parity -> autoregressive drift -> paired +# GSM8K); pinned ON as part of the validated production config. +_FUSE_Q_ROT = True + +# Stronger form: do the rotation INSIDE the decode kernel as a Walsh-Hadamard +# butterfly and drop the prologue dispatch entirely. The prologue is ~7us at +# B=4 and that is almost all fixed per-dispatch cost (sweeping its grid 4x moves +# it 0%), so deleting the launch -- not optimizing it -- is the win. +# Default ON: verified bit-exact (torch.equal) against the prologue path across +# B={1,2,3,4,8,16} x seq={4k,8k,16k,32k,50021,75k}, including non-power-of-2 +# batch and seq. +_FUSE_QROT_INKERNEL = True + + +def _run_fused_q_rot(query, PiT_used, out, B, Hq, D, haircut): + """Launch the fused prologue. ``out`` is the pooled [B_bucket, Hq, D] buf.""" + M = B * Hq + BLOCK_M = 16 + BLOCK_K = 32 + BLOCK_D = 64 + grid = (triton.cdiv(M, BLOCK_M), triton.cdiv(D, BLOCK_D)) + _fused_q_rot_haircut[grid]( + query, + PiT_used, + out, + M, + query.stride(0), + query.stride(1), + HQ=int(Hq), + D=int(D), + HAIRCUT=bool(haircut), + BLOCK_M=BLOCK_M, + BLOCK_K=BLOCK_K, + BLOCK_D=BLOCK_D, + ) + + +# -- Public launcher ---------------------------------------------------------- +def flydsl_ultraquant_decode_attention( + query: torch.Tensor, + kv_cache: torch.Tensor, + block_table: torch.Tensor, + seq_lens: torch.Tensor, + scale: float, + PiT: torch.Tensor | None = None, + max_seq_len: int = 0, + output_buf: torch.Tensor | None = None, + buf_holder: Any = None, + max_num_kv_splits: int = 32, + sinks: torch.Tensor | None = None, +) -> torch.Tensor: + """Launch the optimized FlyDSL UltraQuant D=256 decode.""" + if not is_ultraquant_flydsl_available(): + raise RuntimeError( + "FlyDSL UltraQuant decode requires gfx950 and importable FlyDSL." + ) + if sinks is not None: + raise NotImplementedError("FlyDSL UltraQuant decode does not support sinks") + + B, Hq, D = query.shape + Hk = kv_cache.shape[2] + block_size = kv_cache.shape[1] + padded_slot = int(kv_cache.shape[3]) + QG = Hq // Hk + assert D == 256, f"UltraQuant FlyDSL requires head_dim=256, got {D}" + assert block_size in (16, 32, 64, 128, 256), ( + f"UltraQuant FlyDSL supports kv_block_size 16/32/64/128/256, got {block_size}" + ) + assert QG in _ULTRAQUANT_FLYDSL_GQA, ( + f"UltraQuant FlyDSL supports GQA factor " + f"{', '.join(map(str, _ULTRAQUANT_FLYDSL_GQA))}, got {QG}" + ) + + device = query.device + + # ---- PiT (Hadamard) — cache the contiguous fp32 form on the layer ----- + PiT_f32: torch.Tensor + if buf_holder is not None: + PiT_f32 = getattr(buf_holder, "_ultraquant_PiT_f32", None) + if PiT_f32 is None: + assert PiT is not None, ( + "UltraQuant launcher requires PiT (the Hadamard rotation)" + ) + PiT_f32 = PiT if PiT.dtype == torch.float32 else PiT.to(torch.float32) + if not PiT_f32.is_contiguous(): + PiT_f32 = PiT_f32.contiguous() + buf_holder._ultraquant_PiT_f32 = PiT_f32 + else: + assert PiT is not None, ( + "UltraQuant launcher requires PiT (the Hadamard rotation)" + ) + PiT_f32 = PiT if PiT.dtype == torch.float32 else PiT.to(torch.float32) + if not PiT_f32.is_contiguous(): + PiT_f32 = PiT_f32.contiguous() + + centroids_c = _fp4_value_lut(device) + + # ---- Partition count (FA2 split-KV) ----------------------------------- + kv_compute_block = _ULTRAQUANT_MOD.KV_COMPUTE_BLOCK + worst_case_max_seq_len = int(block_table.shape[1]) * int(block_size) + if max_seq_len <= 0: + max_seq_len = worst_case_max_seq_len + sizing_max_seq_len = worst_case_max_seq_len + + # Split-KV parallelism cap. The old default (32) silently serialized long + # contexts: once required_num_partitions exceeds the cap, TGPP grows and each + # workgroup walks MORE tile-groups instead of the grid getting wider. The + # decode kernel is latency-bound at ~1-2 waves/CU (grid = B*Hk*P workgroups + # of a single wavefront), so that serialization dominated long-context decode. + # Raising the cap to 256 keeps TGPP ~1 and measured (B=16, rocprof, per step + # decode+reduce+store): 16k 82.7->40.0us, 32k 152.0->55.1us, 64k 339.4->95.7us, + # 131k 671.3->179.0us (2.1x - 3.8x, growing with context). 512 regresses + # because _reduce_partitions cost grows with partition count. + # Cost: the segm pool scales with P (~8x vs P=32). + # BATCH-ADAPTIVE split-KV cap (low-concurrency occupancy fix). + # rocprof (B=4, 75k): the decode kernel is occupancy/latency-bound, not + # compute-bound -- OccupancyPercent 5.8%, MfmaUtil 4.5%, VALUBusy 14.5% + # (stalled ~85% on HBM), VALUUtilization 90% (no divergence). The grid is + # B*Hk*num_partitions single-wavefront workgroups; at B=4/Hk=1/P=256 that's + # only 1024 wg = 4/CU, too few to hide memory latency. Splitting MORE at low + # batch adds resident wavefronts and cuts ITL; at high batch the grid is + # already full so extra partitions only inflate _reduce_partitions + empty- + # partition overhead. Measured end-to-end (graphs on, TP=8, uniform 75k, + # median ITL vs kv8): cap512 vs cap256 -> C4 +9.0%->+4.9%, C8 +6.6%->+2.5% + # (better), C16 +2.8%->+4.3% (worse). So use 512 at low batch, 256 at high. + # This is only SAFE because the build-local-allocator fix in + # ultraquant_decode_hd256.py lets the two num_partitions variants (256 & 512) + # coexist across CUDA-graph batch buckets without the @flyc.jit + # "global 'allocator' changed since first compile" drift crash. + MAX_PARTITIONS = 512 if B <= 8 else 256 + required_num_partitions = ( + sizing_max_seq_len + kv_compute_block - 1 + ) // kv_compute_block + parallelism_floor = min(MAX_PARTITIONS, max(1, max_num_kv_splits)) + num_partitions_actual = max( + parallelism_floor, + min(MAX_PARTITIONS, required_num_partitions), + ) + # Cap partitions by how much work there actually IS to split. Nothing above + # forces a partition to hold a useful number of tokens -- max_num_kv_splits + # sets a parallelism FLOOR -- so a short context gets split into partitions + # of a handful of tokens each, where every partition is almost entirely + # fixed cost (and each one still writes a full QG x D partial for the + # reduce to read back). Measured sweet spot is ~130-147 tokens/partition, + # roughly flat in between; next_power_of_2 below rounds this target up, so + # the effective range lands at ~96-192 tokens. At B=4 this is worth 27.9 -> + # 24.4 us at seq=2048 and 27.0 -> 24.8 us at seq=16384, and is inert once + # the context is long enough to want every partition (seq >= ~64k). + _tok_per_part = 192 + if _tok_per_part > 0: + _useful_parts = max( + 1, (int(sizing_max_seq_len) + _tok_per_part - 1) // _tok_per_part + ) + num_partitions_actual = min(num_partitions_actual, _useful_parts) + num_partitions = max(2, triton.next_power_of_2(num_partitions_actual)) + _tgpp_required = max( + 1, + (required_num_partitions + num_partitions - 1) // num_partitions, + ) + tile_groups_per_partition = int(triton.next_power_of_2(_tgpp_required)) + + # ---- Pooled buffers --------------------------------------------------- + pool_bufs = _SEGM_POOL.get( + B, + Hk, + Hq, + num_partitions, + QG, + D, + device, + query.dtype, + ) + segm_out = pool_bufs["segm_out"] + segm_max = pool_bufs["segm_max"] + segm_sum = pool_bufs["segm_sum"] + if output_buf is None: + output = pool_bufs["output"] + else: + output = output_buf[:B] if output_buf.shape[0] != B else output_buf + + # Fold the scaled-MFMA head-dim permutation into PiT's columns so the + # WHT / prologue produces native operand order. Always apply the E4M3 + # haircut: scaled QK consumes Q as E4M3. + PiT_used = PiT_f32 + qperm_idx = _qperm_index(device, D=D) + if buf_holder is not None: + PiT_used = getattr(buf_holder, "_ultraquant_PiT_perm_f32", None) + if PiT_used is None: + PiT_used = PiT_f32.index_select(1, qperm_idx).contiguous() + buf_holder._ultraquant_PiT_perm_f32 = PiT_used + else: + PiT_used = PiT_f32.index_select(1, qperm_idx).contiguous() + _q_float = pool_bufs["q_float"] + _q_rot_f32 = pool_bufs["q_rot_fp32"] + _q_fp8 = pool_bufs["q_fp8"] + _q_rot_out = pool_bufs["q_rot"] + + _inkernel_qrot = ( + _FUSE_QROT_INKERNEL + and int(D) == 256 + and QG in _ULTRAQUANT_FLYDSL_GQA + and query.dim() == 3 + and query.stride(2) == 1 + and query.stride(1) == int(D) + ) + if _inkernel_qrot: + q_for_kernel = query + # The fused kernel indexes the head-dim contiguously; fall back if not. + elif _FUSE_Q_ROT and query.stride(-1) == 1: + # One kernel for the whole prologue instead of four ops. + _run_fused_q_rot(query, PiT_used, _q_rot_out, B, Hq, D, haircut=True) + else: + _q_float.copy_(query) + torch.mm( + _q_float.view(B * Hq, D), + PiT_used, + out=_q_rot_f32.view(B * Hq, D), + ) + _q_fp8.copy_(_q_rot_f32) # fp32 -> e4m3 + _q_rot_out.copy_(_q_fp8) # e4m3 -> bf16 + if not _inkernel_qrot: + q_for_kernel = _q_rot_out + + # ---- FlyDSL kernel launch -------------------------------------------- + max_bps = int(block_table.shape[1]) + launch = _get_kernel( + Hk, + num_partitions, + max_bps, + scale, + QG, + block_size, + padded_slot, + fuse_qrot=_inkernel_qrot, + num_seqs_hint=int(B), + tile_groups_per_partition=int(tile_groups_per_partition), + stride_q_seq=int(q_for_kernel.stride(0)), + stride_q_head=int(q_for_kernel.stride(1)), + ) + global _LOG_INVOKED_ONCE + if not _LOG_INVOKED_ONCE: + _LOG_INVOKED_ONCE = True + logger.info( + "FlyDSL UltraQuant launcher invoked: B=%d Hk=%d Hq=%d D=%d QG=%d " + "num_partitions=%d (actual=%d, cap=%d) TGPP=%d max_bps=%d " + "block_size=%d padded_slot=%d max_seq_len=%d " + "(coverage=%d tokens, worst_case=%d tokens)", + B, + Hk, + Hq, + D, + QG, + num_partitions, + num_partitions_actual, + MAX_PARTITIONS, + tile_groups_per_partition, + max_bps, + int(block_size), + padded_slot, + int(max_seq_len), + num_partitions * tile_groups_per_partition * kv_compute_block, + worst_case_max_seq_len, + ) + launch( + segm_out, + segm_sum, + segm_max, + q_for_kernel, + kv_cache, + centroids_c, + block_table, + seq_lens, + B, + Hk, + num_partitions, + torch.cuda.current_stream(), + ) + + # ---- Reduce partitions -> [B, Hq, D] --------------------------------- + # 64 splits the head-dim 4 ways, taking the grid from 32 to 128 workgroups. + # Bit-identical by construction (the partition reduction is per-column). + _red_block_d = min(int(D), 64) + _reduce_partitions[(B, Hq, triton.cdiv(int(D), _red_block_d))]( + output_ptr=output, + segm_out_ptr=segm_out, + segm_max_ptr=segm_max, + segm_sum_ptr=segm_sum, + out_stride_n=output.stride(0), + out_stride_h=output.stride(1), + NUM_KV_HEADS=Hk, + QG=QG, + NUM_PARTS=num_partitions, + HEAD_SIZE=D, + BLOCK_D=_red_block_d, + ) + return output diff --git a/vllm/v1/attention/ops/merge_attn_states.py b/vllm/v1/attention/ops/merge_attn_states.py index d28c78bdb9..b16786a13b 100644 --- a/vllm/v1/attention/ops/merge_attn_states.py +++ b/vllm/v1/attention/ops/merge_attn_states.py @@ -139,8 +139,19 @@ def merge_attn_states( return headdim % 4 == 0 return headdim % 8 == 0 + # CDNA3 and newer (gfx942, gfx950, gfx1250). gfx90a is CDNA2 and + # stays on the Triton merge. + def supported_platform() -> bool: + if current_platform.is_cuda(): + return True + if current_platform.is_rocm(): + from vllm.platforms.rocm import get_cdna_version + + return get_cdna_version() >= 3 + return False + if ( - current_platform.is_cuda() + supported_platform() and supported_dtypes(prefix_output) and supported_headdim(prefix_output) ): diff --git a/vllm/v1/attention/ops/metadata.py b/vllm/v1/attention/ops/metadata.py index f5baa51a92..1852e77ee2 100644 --- a/vllm/v1/attention/ops/metadata.py +++ b/vllm/v1/attention/ops/metadata.py @@ -2,6 +2,8 @@ # SPDX-FileCopyrightText: Copyright contributors to the vLLM project """Device-side request mapping and sparse indexer metadata.""" +import torch + from vllm.triton_utils import tl, triton @@ -27,6 +29,28 @@ def _token_request_mapping_kernel( tl.store(output + token, tl.where(token < num_mapped, req, 0), token < num_tokens) +def compute_token_to_req_indices( + query_start_loc: torch.Tensor, + out: torch.Tensor, + num_mapped_tokens: int, + num_tokens: int, +) -> torch.Tensor: + """Map each of the first ``num_tokens`` tokens to its request index. + + Reads only the device ``query_start_loc``, so it is safe to record in a + CUDA graph. Tokens at or past ``num_mapped_tokens`` are mapped to 0. + """ + _token_request_mapping_kernel[((num_tokens + 255) // 256,)]( + query_start_loc, + out, + query_start_loc.shape[0] - 1, + num_mapped_tokens, + num_tokens, + num_warps=4, + ) + return out[:num_tokens] + + @triton.jit( do_not_specialize=[ "num_reqs", diff --git a/vllm/v1/attention/ops/rocm_aiter_mla_sparse.py b/vllm/v1/attention/ops/rocm_aiter_mla_sparse.py index a3412aa835..6a93d21511 100644 --- a/vllm/v1/attention/ops/rocm_aiter_mla_sparse.py +++ b/vllm/v1/attention/ops/rocm_aiter_mla_sparse.py @@ -707,6 +707,14 @@ def paged_mqa_logits_module(): return None +def _per_sequence_context_lens(context_lens: torch.Tensor) -> torch.Tensor: + # AITER takes one length per sequence: the last column of a (B, next_n) table. + # TODO: remove once AITER accepts (B, next_n) context_lens (ROCm/aiter#6153). + if context_lens.dim() == 2 and context_lens.shape[1] > 1: + return context_lens[:, -1].contiguous() + return context_lens + + def rocm_fp8_paged_mqa_logits( q_fp8: torch.Tensor, kv_cache_fp8: torch.Tensor, @@ -726,8 +734,8 @@ def rocm_fp8_paged_mqa_logits( kv_cache_fp8: Paged KV-cache in packed FP8+scale layout with shape [num_blocks, block_size, 1, D+4], dtype `torch.uint8`. weights: Tensor of shape [B * next_n, H], dtype `torch.float32`. - context_lens: Tensor of shape [B], dtype int32; effective context length - for each batch element. + context_lens: Tensor of shape [B] or [B, next_n], dtype int32; effective + context length for each batch element, or for each of its Q rows. block_tables: Tensor of shape [B, max_blocks], dtype int32; maps logical block indices to physical blocks in the paged cache. schedule_metadata: Returned by `get_paged_mqa_logits_metadata`; @@ -779,7 +787,7 @@ def rocm_fp8_paged_mqa_logits( kv_cache_fp8, weights, out_logits, - context_lens, + _per_sequence_context_lens(context_lens), block_tables, max_model_len, ChunkK=256, @@ -801,7 +809,7 @@ def rocm_fp8_paged_mqa_logits( kv_cache_fp8, weights, out_qk, - context_lens, + _per_sequence_context_lens(context_lens), block_tables, max_model_len, ChunkQ=heads, @@ -2208,6 +2216,119 @@ def _sparse_attn_prefill_ragged_kernel( ) +@triton.jit +def _sparse_attn_decode_ragged_bf16_partial_kernel( + q_ptr, + kv_ptr, + kv_indices_ptr, + kv_indptr_ptr, + part_m_ptr, + part_l_ptr, + part_acc_ptr, + q_stride_t, + q_stride_h, + q_stride_d, + kv_stride_n, + kv_stride_d, + pm_stride0, + pm_stride_s, + pa_stride0, + pa_stride_s, + pa_stride_h, + num_heads, + head_dim, + num_kv, + scale, + BLOCK_H: tl.constexpr, + BLOCK_D: tl.constexpr, + BLOCK_K: tl.constexpr, + NUM_SPLITS: tl.constexpr, +): + query_idx = tl.program_id(0) + split_id = tl.program_id(1) + pid_h = tl.program_id(2) + + head_offsets = pid_h * BLOCK_H + tl.arange(0, BLOCK_H) + dim_offsets = tl.arange(0, BLOCK_D) + head_mask = head_offsets < num_heads + dim_mask = dim_offsets < head_dim + + q = tl.load( + q_ptr + + query_idx * q_stride_t + + head_offsets[:, None] * q_stride_h + + dim_offsets[None, :] * q_stride_d, + mask=head_mask[:, None] & dim_mask[None, :], + other=0.0, + ) + + neg_large = -3.4028234663852886e38 + m_i = tl.full((BLOCK_H,), neg_large, dtype=tl.float32) + l_i = tl.zeros((BLOCK_H,), dtype=tl.float32) + acc = tl.zeros((BLOCK_H, BLOCK_D), dtype=tl.float32) + + kv_start = tl.load(kv_indptr_ptr + query_idx) + kv_end = tl.load(kv_indptr_ptr + query_idx + 1) + kv_len = kv_end - kv_start + chunk = (kv_len + NUM_SPLITS - 1) // NUM_SPLITS + kv_lo = split_id * chunk + kv_hi = tl.minimum(kv_lo + chunk, kv_len) + + k_offsets = tl.arange(0, BLOCK_K) + slot = tl.load( + kv_indices_ptr + kv_start + kv_lo + k_offsets, + mask=kv_lo + k_offsets < kv_hi, + other=-1, + ) + for k_start in tl.range(kv_lo, kv_hi, BLOCK_K): + k_pos = k_start + k_offsets + in_range = k_pos < kv_hi + valid = in_range & (slot >= 0) & (slot < num_kv) + safe_slot = tl.where(valid, slot, 0) + + kv = tl.load( + kv_ptr + + _sparse_kv_row_offset(safe_slot[:, None], kv_stride_n) + + dim_offsets[None, :] * kv_stride_d, + mask=valid[:, None] & dim_mask[None, :], + other=0.0, + ) + + next_k_pos = k_start + BLOCK_K + k_offsets + slot = tl.load( + kv_indices_ptr + kv_start + next_k_pos, mask=next_k_pos < kv_hi, other=-1 + ) + + scores = tl.dot(q, tl.trans(kv)) * scale + scores = tl.where(head_mask[:, None] & valid[None, :], scores, neg_large) + + m_block = tl.max(scores, axis=1) + m_new = tl.maximum(m_i, m_block) + alpha = tl.exp(m_i - m_new) + p = tl.exp(scores - m_new[:, None]) + p = tl.where(head_mask[:, None] & valid[None, :], p, 0.0) + l_new = l_i * alpha + tl.sum(p, axis=1) + + acc = acc * alpha[:, None] + tl.dot(p.to(kv.dtype), kv) + m_i = m_new + l_i = l_new + + # Every launched slot is written; empty splits store the (neg_large, 0.0) + # sentinel the reduce combines to nothing. + pm_base = query_idx * pm_stride0 + split_id * pm_stride_s + head_offsets + tl.store(part_m_ptr + pm_base, m_i, mask=head_mask) + tl.store(part_l_ptr + pm_base, l_i, mask=head_mask) + tl.store( + part_acc_ptr + + query_idx * pa_stride0 + + split_id * pa_stride_s + + head_offsets[:, None] * pa_stride_h + + dim_offsets[None, :], + acc, + mask=head_mask[:, None] & dim_mask[None, :], + ) + + @triton.jit def _decode_e8m0_scales_triton(encoded_scales): scale_bits = encoded_scales.to(tl.int32) << 23 @@ -3664,6 +3785,182 @@ def _decode_gfx950_num_splits( return num_splits +# Empirically single-pass bf16 decode is faster than split-K below this number of +# selected slots per query, so we do not split. +_SPARSE_DECODE_BF16_MIN_SPLIT_LEN = 512 +_SPARSE_DECODE_BF16_BLOCK_H = 16 +_SPARSE_DECODE_BF16_BLOCK_K = 32 + + +def rocm_sparse_decode_bf16_num_splits( + num_queries: int, num_heads: int, sparse_len: int +) -> int: + """Number or kv splits in splitK for the sparse bf16 decode, or 1 for single-pass. + + Args: + num_queries: Decode rows in the batch. + num_heads: Query heads per row. + sparse_len: Longest selected KV run any decode row can walk. + + Returns: + The split count, or 1 when the caller should use the single-pass kernel. + + """ + if sparse_len < _SPARSE_DECODE_BF16_MIN_SPLIT_LEN: + return 1 + block_k = _SPARSE_DECODE_BF16_BLOCK_K + heads_blocks = triton.cdiv(num_heads, _SPARSE_DECODE_BF16_BLOCK_H) + select = _decode_gfx950_num_splits if _ON_GFX950 else _decode_num_splits + num_splits = select(num_queries, heads_blocks, sparse_len, 0.0, block_k) + # Number of splits cannot exceed the available k tiles + return max(1, min(num_splits, math.ceil(sparse_len / block_k))) + + +def _rocm_sparse_attn_decode_ragged_bf16_triton( + q: torch.Tensor, + kv: torch.Tensor, + indices: torch.Tensor, + indptr: torch.Tensor, + scale: float, + attn_sink: torch.Tensor | None, + nope_head_dim: int, + rope_head_dim: int, + num_splits: int, + out: torch.Tensor | None = None, +) -> torch.Tensor: + """Split-K decode over an bf16 ragged KV cache. + + Partitions each query's selected tokens across different workgroups and combines + the partials through reduction. + + Args: + q: Queries laid out as ``[sq, h, d]``. + kv: Unquantized KV rows laid out as ``[skv, d]``. + indices: Flattened per-query KV slots. + indptr: Segment offsets into ``indices``, ``[sq + 1]``. + scale: Softmax scale. + attn_sink: Optional per-head sink logits. + nope_head_dim: NoPE width of ``d``. + rope_head_dim: RoPE width of ``d``. + num_splits: Number of KV splits per query. + out: Optional destination with ``d`` trailing elements. + + Returns: + The attention output, ``out`` when provided. + + """ + assert q.ndim == 3, f"expected q=[sq,h,d], got {q.shape}" + assert kv.ndim == 2, f"expected kv=[skv,d], got {kv.shape}" + assert indices.ndim == 1, f"expected indices=[nnz], got {indices.shape}" + assert indptr.ndim == 1, f"expected indptr=[sq+1], got {indptr.shape}" + assert not q.is_cpu and not kv.is_cpu and not indices.is_cpu and not indptr.is_cpu + + indices = _as_int32_contiguous_1d(indices) + indptr = _as_int32_contiguous_1d(indptr) + has_attn_sink = attn_sink is not None + if attn_sink is None: + attn_sink = torch.empty(1, device=q.device, dtype=torch.float32) + else: + attn_sink = attn_sink.contiguous() + + num_queries, num_heads, head_dim = q.shape + assert indptr.numel() == num_queries + 1, ( + f"expected indptr shape [{num_queries + 1}], got {indptr.shape}" + ) + _validate_sparse_dims( + head_dim, + nope_head_dim, + rope_head_dim, + "_rocm_sparse_attn_decode_ragged_bf16_triton", + ) + if out is None: + out = torch.empty_like(q) + assert out.shape[-1] == head_dim, ( + f"expected out trailing dim {head_dim}, got {out.shape[-1]}" + ) + + block_h = _SPARSE_DECODE_BF16_BLOCK_H + block_d = triton.next_power_of_2(head_dim) + block_k = _SPARSE_DECODE_BF16_BLOCK_K + heads_blocks = triton.cdiv(num_heads, block_h) + comb_dim = nope_head_dim + rope_head_dim + + part_m = torch.empty( + (num_queries, num_splits, num_heads), dtype=torch.float32, device=q.device + ) + part_l = torch.empty_like(part_m) + part_acc = torch.empty( + (num_queries, num_splits, num_heads, comb_dim), + dtype=torch.float32, + device=q.device, + ) + + _sparse_attn_decode_ragged_bf16_partial_kernel[ + (num_queries, num_splits, heads_blocks) + ]( + q, + kv, + indices, + indptr, + part_m, + part_l, + part_acc, + q.stride(0), + q.stride(1), + q.stride(2), + kv.stride(0), + kv.stride(1), + part_m.stride(0), + part_m.stride(1), + part_acc.stride(0), + part_acc.stride(1), + part_acc.stride(2), + num_heads, + head_dim, + kv.shape[0], + float(scale), + BLOCK_H=block_h, + BLOCK_D=block_d, + BLOCK_K=block_k, + NUM_SPLITS=num_splits, + num_warps=4, + ) + + _sparse_attn_decode_reduce_kernel[(num_queries, num_heads)]( + part_m, + part_l, + part_acc, + attn_sink, + out, + None, + None, + None, + out.stride(0), + out.stride(1), + 0, + head_dim // 32, + part_m.stride(0), + part_m.stride(1), + part_acc.stride(0), + part_acc.stride(1), + part_acc.stride(2), + 0, + num_heads, + HAS_ATTN_SINK=has_attn_sink, + ADAPTIVE_SPLITS=False, + COMB_DIM=comb_dim, + BLOCK_H=1, + NUM_SPLITS=num_splits, + SPLITS_PAD=triton.next_power_of_2(num_splits), + FUSE_INV_ROPE=False, + NOPE=nope_head_dim, + HALF=rope_head_dim // 2, + QUANT_OUT=False, + num_warps=4, + ) + return out + + def _rocm_sparse_attn_decode_ragged_triton( q: torch.Tensor, main_cache: torch.Tensor, @@ -4143,6 +4440,72 @@ def rocm_sparse_attn_prefill( ) +def rocm_sparse_attn_decode_bf16( + q: torch.Tensor, + kv: torch.Tensor, + scale: float, + head_dim: int, + nope_head_dim: int, + rope_head_dim: int, + attn_sink: torch.Tensor | None, + output: torch.Tensor, + ragged_indices: torch.Tensor, + ragged_indptr: torch.Tensor, + num_splits: int, +) -> None: + """Run split-K sparse attention over decode rows using an unquantized KV cache. + + Args: + q: Decode queries laid out as ``[sq, h, d]``. + kv: KV cache laid out as ``[skv, 1, d]``. + scale: Softmax scale. + head_dim: Post-absorption head width. + nope_head_dim: NoPE width of ``head_dim``. + rope_head_dim: RoPE width of ``head_dim``. + attn_sink: Optional per-head sink logits. + output: Destination, written in place. + ragged_indices: Flattened per-query KV slots. + ragged_indptr: Segment offsets into ``ragged_indices``, ``[sq + 1]``. + num_splits: KV splits per query, from + :func:`rocm_sparse_decode_bf16_num_splits`. + + """ + assert kv.ndim == 3 and kv.shape[1] == 1, ( + f"ROCm Triton sparse decode expects kv=[skv,1,d], got {kv.shape}" + ) + _validate_sparse_dims( + head_dim, + nope_head_dim, + rope_head_dim, + "rocm_sparse_attn_decode_bf16", + ) + num_queries, num_heads = q.shape[0], q.shape[1] + direct = output.shape[-1] == head_dim + out = ( + output + if direct + else torch.empty( + (num_queries, num_heads, head_dim), + dtype=output.dtype, + device=output.device, + ) + ) + _rocm_sparse_attn_decode_ragged_bf16_triton( + q=q, + kv=kv.squeeze(1), + indices=ragged_indices, + indptr=ragged_indptr, + scale=scale, + attn_sink=None if attn_sink is None else attn_sink[: q.shape[1]], + nope_head_dim=nope_head_dim, + rope_head_dim=rope_head_dim, + num_splits=num_splits, + out=out, + ) + if not direct: + output.copy_(out[..., : output.shape[-1]]) + + def rocm_sparse_attn_decode( q: torch.Tensor, kv_cache: torch.Tensor | None, diff --git a/vllm/v1/attention/ops/segmented_attention.py b/vllm/v1/attention/ops/segmented_attention.py new file mode 100644 index 0000000000..f98b07df2a --- /dev/null +++ b/vllm/v1/attention/ops/segmented_attention.py @@ -0,0 +1,1074 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +"""ROCm token-major segmented attention kernels and dispatch.""" + +from __future__ import annotations + +from functools import lru_cache + +import torch + +from vllm.platforms import current_platform +from vllm.triton_utils import tl, triton +from vllm.v1.kv_cache_interface import get_kv_quant_mode +from vllm.v1.worker.workspace import ( + current_workspace_manager, + is_workspace_manager_initialized, +) + +from .triton_unified_attention import unified_attention + +MAX_QUERY_LEN = 8192 +MAX_SPLITS = 64 +# D128 scratch cap; wider outputs scale this budget with head dimension. +MAX_LONG_EXTEND_WORKSPACE_BYTES = 32 * 1024 * 1024 + + +def segmented_query_capacity(max_query_len: int) -> int: + """Round a supported query length to its compiled power-of-two capacity.""" + if not 1 <= max_query_len <= MAX_QUERY_LEN: + raise ValueError(f"Query length must be in [1, {MAX_QUERY_LEN}]") + return 1 << (max_query_len - 1).bit_length() + + +def _query_capacity_buckets() -> tuple[int, ...]: + return tuple(1 << exponent for exponent in range(MAX_QUERY_LEN.bit_length())) + + +def _long_extend_splits( + base_splits: int, + batch: int, + query_len: int, + seq_len: int, + hq: int, + hk: int, + dim: int, + bm: int, +) -> int: + """Add long-prefix parallelism while bounding the extra split workspace.""" + if seq_len < 32768: + return base_splits + groups = batch * hk * triton.cdiv(query_len * (hq // hk), bm) + split_cap = 16 if seq_len < 131072 else 32 + scratch_per_split = batch * segmented_query_capacity(query_len) * hq * (dim + 1) * 4 + scratch_limit = MAX_LONG_EXTEND_WORKSPACE_BYTES * dim // 128 + scratch_splits = scratch_limit // scratch_per_split + scratch_cap = 1 << (scratch_splits.bit_length() - 1) if scratch_splits else 1 + occupancy_splits = triton.next_power_of_2(triton.cdiv(512, groups)) + return max(base_splits, min(split_cap, scratch_cap, occupancy_splits)) + + +@triton.jit +def _segmented_attention_stage( + Q, + K, + V, + TABLE, + STARTS, + LENS, + KS, + VS, + SINKS, + PART, + LSE, + OUT, + Q0: tl.constexpr, + Q1: tl.constexpr, + O0: tl.constexpr, + O1: tl.constexpr, + K0: tl.constexpr, + K1: tl.constexpr, + K2: tl.constexpr, + K3: tl.constexpr, + V0: tl.constexpr, + V1: tl.constexpr, + V2: tl.constexpr, + V3: tl.constexpr, + T0: tl.constexpr, + T1: tl.constexpr, + PAGE: tl.constexpr, + HQ: tl.constexpr, + HK: tl.constexpr, + D: tl.constexpr, + QCAP: tl.constexpr, + LIMIT_Q: tl.constexpr, + SCALE: tl.constexpr, + FP8: tl.constexpr, + BM: tl.constexpr, + BN: tl.constexpr, + BK: tl.constexpr, + SPLITS: tl.constexpr, + WINDOW: tl.constexpr, + CAUSAL: tl.constexpr, + HAS_SINKS: tl.constexpr, + PREFIX_FAST: tl.constexpr, + PV_SPLIT: tl.constexpr, + QK_PIPELINE: tl.constexpr, +): + item, mb, split = tl.program_id(0), tl.program_id(1), tl.program_id(2) + seq, kh = item // HK, item % HK + first = tl.load(STARTS + seq) + nq = tl.load(STARTS + seq + 1) - first + if nq < 1 or nq > LIMIT_Q: + return + G: tl.constexpr = HQ // HK + ROWS: tl.constexpr = QCAP * G + if mb * BM >= nq * G: + return + sequence = tl.load(LENS + seq) + prefix = sequence - nq + m = mb * BM + tl.arange(0, BM) + qpos, qh = m // G, kh * G + m % G + n = tl.arange(0, BN) + d = tl.arange(0, D) + if BK == D: + q_full = tl.load( + Q + (first + qpos[:, None]) * Q0 + qh[:, None] * Q1 + d[None, :], + mask=(m < nq * G)[:, None], + other=0.0, + ) + maximum = tl.full((BM,), -float("inf"), tl.float32) + denom = tl.zeros((BM,), tl.float32) + if HAS_SINKS: + sink = tl.load(SINKS + qh, mask=m < nq * G, other=0).to(tl.float32) + maximum = tl.where(split == 0, sink * 1.4426950408889634, maximum) + denom = tl.where(split == 0, 1.0, denom) + if PV_SPLIT: + half = tl.arange(0, D // 2) + acc_low = tl.zeros((BM, D // 2), tl.float32) + acc_high = tl.zeros((BM, D // 2), tl.float32) + else: + acc = tl.zeros((BM, D), tl.float32) + tile_rows = tl.minimum((mb + 1) * BM, nq * G) + range_begin = 0 + if WINDOW >= 0: + first_qpos = (mb * BM) // G + range_begin = tl.maximum(0, prefix + first_qpos - WINDOW) + range_begin = (range_begin // BN) * BN + range_end = sequence + if CAUSAL: + range_end = tl.minimum(range_end, prefix + tl.cdiv(tile_rows, G)) + span = tl.maximum(0, range_end - range_begin) + segment = tl.cdiv(span, SPLITS * BN) * BN + begin = range_begin + split * segment + end = tl.minimum(begin + segment, range_end) + k_scale = 1.0 + v_scale = 1.0 + if FP8: + k_scale = tl.load(KS) + v_scale = tl.load(VS) + for tail in tl.static_range(2): + if tail: + loop_begin = tl.maximum(begin, (end // BN) * BN) + loop_end = end + else: + loop_begin = begin + loop_end = (end // BN) * BN + for start in tl.range(loop_begin, loop_end, BN): + ns = start + n + valid = tl.full((BN,), True, tl.int1) + if tail: + valid = ns < end + if FP8 and PAGE >= BN: + page = start // PAGE + inside = start - page * PAGE + n + crossed = inside >= PAGE + page += crossed + inside = tl.where(crossed, inside - PAGE, inside) + else: + page = ns // PAGE + inside = ns % PAGE + block = tl.load(TABLE + seq * T0 + page * T1, mask=valid, other=0).to( + tl.int64 + ) + scores = tl.zeros((BM, BN), tl.float32) + for ki in tl.range(D // BK, num_stages=QK_PIPELINE): + kd = ki * BK + tl.arange(0, BK) + if BK == D: + q = q_full + else: + q = tl.load( + Q + + (first + qpos[:, None]) * Q0 + + qh[:, None] * Q1 + + kd[None, :], + mask=(m < nq * G)[:, None], + other=0.0, + ) + k = tl.load( + K + + block[None, :] * K0 + + inside[None, :] * K1 + + kh * K2 + + kd[:, None] * K3, + mask=valid[None, :], + other=0.0, + ) + scores = tl.dot(q, k.to(q.dtype), scores) + scores *= SCALE * 1.4426950408889634 * k_scale + # A complete prefix tile is visible to every valid causal row when + # sliding-window masking is disabled. + if not ( + PREFIX_FAST + and CAUSAL + and WINDOW < 0 + and start + BN <= prefix + and mb * BM + BM <= nq * G + ): + score_mask = valid[None, :] & (m[:, None] < nq * G) + if CAUSAL: + score_mask = score_mask & (ns[None, :] <= prefix + qpos[:, None]) + if WINDOW >= 0: + score_mask = score_mask & ( + ns[None, :] >= prefix + qpos[:, None] - WINDOW + ) + scores = tl.where(score_mask, scores, -float("inf")) + new_max = tl.maximum(maximum, tl.max(scores, 1)) + safe_max = tl.where(new_max == -float("inf"), 0.0, new_max) + alpha = tl.exp2(maximum - safe_max) + p = tl.exp2(scores - safe_max[:, None]) + if PV_SPLIT: + acc_low *= alpha[:, None] + acc_high *= alpha[:, None] + v_low = tl.load( + V + + block[:, None] * V0 + + inside[:, None] * V1 + + kh * V2 + + half[None, :] * V3, + mask=valid[:, None], + other=0.0, + ) + v_high = tl.load( + V + + block[:, None] * V0 + + inside[:, None] * V1 + + kh * V2 + + (half[None, :] + D // 2) * V3, + mask=valid[:, None], + other=0.0, + ) + prob = p.to(Q.dtype.element_ty) + acc_low = tl.dot(prob, v_low.to(prob.dtype), acc_low) + acc_high = tl.dot(prob, v_high.to(prob.dtype), acc_high) + else: + acc *= alpha[:, None] + v = tl.load( + V + + block[:, None] * V0 + + inside[:, None] * V1 + + kh * V2 + + d[None, :] * V3, + mask=valid[:, None], + other=0.0, + ) + if FP8 and D == 256 and BM >= 32: + # Defer the scalar V dequantization to shorten the D-wide + # accumulator live range for the register-heavy 256-D tiles. + acc = tl.dot( + p.to(Q.dtype.element_ty), v.to(Q.dtype.element_ty), acc + ) + elif FP8: + acc += ( + tl.dot(p.to(Q.dtype.element_ty), v.to(Q.dtype.element_ty)) + * v_scale + ) + else: + acc = tl.dot(p.to(Q.dtype.element_ty), v, acc) + denom = denom * alpha + tl.sum(p, 1) + maximum = safe_max + norm = tl.where(denom > 0.0, denom, 1.0) + if PV_SPLIT: + result = tl.cat(acc_low, acc_high, dim=1) / norm[:, None] + else: + result = acc / norm[:, None] + if FP8 and D == 256 and BM >= 32: + result *= v_scale + if SPLITS == 1: + tl.store( + OUT + (first + qpos[:, None]) * O0 + qh[:, None] * O1 + d[None, :], + result, + mask=(m < nq * G)[:, None], + ) + else: + row = ((seq * HK + kh) * ROWS + m) * SPLITS + split + tl.store( + PART + row[:, None] * D + d[None, :], + result, + mask=(m < nq * G)[:, None], + ) + tl.store( + LSE + row, + maximum + tl.log2(denom), + mask=m < nq * G, + ) + + +@triton.jit +def _segmented_attention_reduce( + PART, + LSE, + OUT, + STARTS, + O0: tl.constexpr, + O1: tl.constexpr, + HQ: tl.constexpr, + HK: tl.constexpr, + D: tl.constexpr, + QCAP: tl.constexpr, + LIMIT_Q: tl.constexpr, + SPLITS: tl.constexpr, + REDUCE_D: tl.constexpr, +): + m, item, db = tl.program_id(0), tl.program_id(1), tl.program_id(2) + seq, kh = item // HK, item % HK + first = tl.load(STARTS + seq) + nq = tl.load(STARTS + seq + 1) - first + G: tl.constexpr = HQ // HK + ROWS: tl.constexpr = QCAP * G + if nq < 1 or nq > LIMIT_Q or m >= nq * G: + return + s = tl.arange(0, SPLITS) + d = db * REDUCE_D + tl.arange(0, REDUCE_D) + row = ((seq * HK + kh) * ROWS + m) * SPLITS + lse = tl.load(LSE + row + s) + weights = tl.exp2(lse - tl.max(lse, 0)) + part = tl.load( + PART + (row + s)[:, None] * D + d[None, :], + mask=d[None, :] < D, + other=0.0, + ) + result = tl.sum(part * weights[:, None], 0) / tl.sum(weights, 0) + tl.store( + OUT + (first + m // G) * O0 + (kh * G + m % G) * O1 + d, + result, + mask=d < D, + ) + + +@lru_cache(maxsize=512) +def select_segmented_config(batch, max_query_len, max_seq_len, hq, hk, dim, fp8): + """Select the token-major cache configuration for segmented attention.""" + qcap = min(max_query_len, MAX_QUERY_LEN) + if batch < 1 or qcap < 1: + raise ValueError("Batch and query capacity must be positive") + + if qcap <= 2: + if dim == 64 and qcap == 1 and batch == 1 and max_seq_len >= 131072: + return dict( + bm=16, + bn=32, + bk=64, + splits=256, + warps=1, + stages=1, + ) + if dim == 64 and max_seq_len <= 256: + return dict( + bm=16, + bn=32 if fp8 else 64, + bk=64, + splits=1, + warps=4, + stages=1, + ) + if dim == 64 and max_seq_len >= 32768: + groups = batch * hk * triton.cdiv(qcap * (hq // hk), 16) + return dict( + bm=16, + bn=32, + bk=64, + splits=min(64, triton.next_power_of_2(triton.cdiv(128, groups))), + warps=4, + stages=1, + ) + if batch * hk == 1: + cfg = dict( + bm=16, + bn=64, + bk=min(128, dim), + splits=32, + warps=4, + stages=1, + ) + else: + groups = batch * hk * triton.cdiv(qcap * (hq // hk), 16) + cfg = dict( + bm=16, + bn=32, + bk=dim, + splits=min( + MAX_SPLITS, + triton.next_power_of_2(triton.cdiv(64, groups)), + ), + warps=4, + stages=1, + ) + while cfg["splits"] > 1 and max_seq_len < cfg["splits"] * cfg["bn"] * 2: + cfg["splits"] //= 2 + return cfg + + if qcap <= 8: + bm, bn = 16, 64 + bk, stages = 64, 1 + elif qcap <= 32: + bm, bk = 32, 64 + bn, stages = (32, 2) if fp8 else (64, 1) + else: + bm, bn, bk = 32, 64, 64 + stages = 1 if fp8 else 2 + groups = batch * hk * triton.cdiv(qcap * (hq // hk), bm) + target = 96 + if fp8 and qcap > 32: + target = 192 + splits = min(32, triton.next_power_of_2(triton.cdiv(target, groups))) + while splits > 1 and max_seq_len < splits * 128: + splits //= 2 + cfg = dict( + bm=bm, + bn=bn, + bk=bk, + splits=splits, + warps=4, + stages=stages, + ) + if dim == 64 and qcap >= 64 and qcap < max_seq_len <= qcap + 256: + cfg.update( + bm=64 if fp8 or qcap <= 256 else 128, + bn=32, + bk=64, + splits=1, + warps=4, + stages=1, + ) + elif dim == 64 and fp8 and max_seq_len <= 256: + cfg.update(bn=32, splits=1) + elif dim == 64 and qcap == 8 and max_seq_len >= 32768: + groups = batch * hk * triton.cdiv(qcap * (hq // hk), 32) + cfg.update( + bm=32, + bn=32 if fp8 else 64, + splits=min(64, triton.next_power_of_2(triton.cdiv(256, groups))), + ) + elif dim == 64 and fp8 and qcap == 8 and max_seq_len >= 4096: + groups = batch * hk * triton.cdiv(qcap * (hq // hk), 32) + cfg.update( + bm=32, + bn=64, + splits=min(32, triton.next_power_of_2(triton.cdiv(128, groups))), + ) + elif dim == 64 and fp8 and qcap >= 64 and max_seq_len >= 4096: + groups = batch * hk * triton.cdiv(qcap * (hq // hk), 64) + cfg.update( + bm=64, + bn=32, + splits=min(8, triton.next_power_of_2(triton.cdiv(1024, groups))), + stages=1, + ) + elif dim == 128 and fp8 and qcap <= 256 and max_seq_len <= qcap: + cfg.update(splits=1) + elif ( + dim == 128 + and fp8 + and qcap >= 256 + and max_seq_len >= 8192 + and ( + hq // hk >= 4 + or (max_seq_len >= 32768 and (hq // hk >= 2 or qcap >= 512)) + or max_seq_len >= 131072 + ) + ): + # Shorter prefixes need sufficient grouped-query reuse; at GQA1/Q256 + # the wider tile pays off only once the prefix reaches 131K tokens. + cfg.update( + bm=64, + bn=128, + bk=64, + warps=8, + stages=3, + waves_per_eu=6, + prefix_fast=True, + ) + if qcap >= 512 or max_seq_len >= 32768: + cfg["qk_pipeline"] = 3 + cfg["splits"] = _long_extend_splits( + cfg["splits"], batch, qcap, max_seq_len, hq, hk, dim, cfg["bm"] + ) + elif dim == 256 and fp8 and qcap >= 256 and max_seq_len >= 8192: + # Wider tiles amortize each FP8 KV load over twice as many queries and + # keys. Six waves is the best gfx1201 pressure/occupancy tradeoff. + cfg.update( + bm=64, + bn=128, + bk=64, + warps=8, + stages=1, + waves_per_eu=6, + ) + cfg["splits"] = _long_extend_splits( + cfg["splits"], batch, qcap, max_seq_len, hq, hk, dim, cfg["bm"] + ) + elif dim == 256 and not fp8 and qcap >= 1024 and max_seq_len >= 8192: + # Two half-width PV accumulators avoid spilling this BF16 D256 tile. + # Keep Q256 on the incumbent: the wider tile loses at that capacity. + cfg.update( + bm=64, + bn=64, + bk=64, + warps=8, + stages=3, + waves_per_eu=6, + pv_split=True, + qk_pipeline=3, + ) + elif dim == 128 and not fp8 and qcap > 128: + if max_seq_len >= 8192: + cfg.update( + bm=128, + bn=32, + bk=128, + warps=8, + stages=1, + waves_per_eu=6, + ) + groups = batch * hk * triton.cdiv(qcap * (hq // hk), cfg["bm"]) + cfg["splits"] = min(16, triton.next_power_of_2(triton.cdiv(192, groups))) + elif qcap >= 4096: + cfg.update( + bm=128, + bn=32, + bk=128, + splits=1, + warps=8, + stages=1, + waves_per_eu=6, + ) + elif qcap > 256 or hq > 10: + cfg.update( + bm=64, + bn=32, + bk=128, + splits=1, + warps=4, + stages=1, + waves_per_eu=6, + ) + else: + cfg.update(bm=32, bn=64, bk=64, splits=1, warps=4, stages=2) + return cfg + + +def segmented_workspace_shapes(batch, query_cap, hq, hk, dim, splits): + if splits == 1: + return None + partial = (batch, hk, query_cap * (hq // hk), splits, dim) + return partial, partial[:-1] + + +@lru_cache(maxsize=128) +def segmented_attention_workspace_size( + max_batch, + hq, + hk, + dim, + max_seq_len, + *, + max_tokens=None, + fp8=False, + autotune=False, +): + """Size the largest selected attention workspace before graph capture. + + ``max_tokens`` bounds reachable ``(batch, query_len)`` pairs using one + longest query and one token for each remaining sequence. Omitting it keeps + the legacy reservation behavior for callers without scheduler limits. + ``autotune`` reserves additional splits for tuned D64 candidates. + """ + largest = 0 + previous_capacity = 0 + for query_capacity in _query_capacity_buckets(): + query_lengths = {previous_capacity + 1, query_capacity} + for query_len in query_lengths: + if max_tokens is not None and query_len > max_tokens: + continue + batch_limit = max_batch + if max_tokens is not None: + batch_limit = min(batch_limit, max_tokens - query_len + 1) + for batch in range(1, batch_limit + 1): + cfg = select_segmented_config( + batch, query_len, max_seq_len, hq, hk, dim, fp8 + ) + splits = cfg["splits"] + if dim == 64 and query_len <= 512 and max_seq_len >= 4096 and autotune: + split_limit = ( + 512 + if batch == 1 and query_len == 1 and max_seq_len >= 131072 + else MAX_SPLITS + ) + splits = min(split_limit, 2 * splits) + if splits > 1: + largest = max( + largest, + splits * batch * query_capacity * hq, + ) + previous_capacity = query_capacity + return largest * dim, largest + + +def get_segmented_attention_workspace(device, sizes): + """Keep scratch storage stable and separate for each runtime lane/ubatch.""" + + def allocate(): + return tuple( + torch.empty(size, device=device, dtype=torch.float32) for size in sizes + ) + + if is_workspace_manager_initialized(): + return current_workspace_manager().get_persistent_resource( + ("rocm_segmented_attention", device, sizes), allocate + ) + return allocate() + + +def can_use_segmented_attention( + q, k, v, out, kc, vc, table, starts, lengths, k_scale, v_scale +): + """Check tensor metadata; unsupported feature checks live in the caller.""" + if ( + k is None + or v is None + or q.ndim != 3 + or kc.ndim != 4 + or vc.ndim != 4 + or q.dtype not in (torch.bfloat16, torch.float16) + or out.dtype != q.dtype + or q.shape != out.shape + or q.shape[-1] not in (64, 128, 256) + or kc.dtype != vc.dtype + or k.dtype != q.dtype + or v.dtype != q.dtype + or kc.dtype not in (q.dtype, torch.float8_e4m3fn, torch.float8_e4m3fnuz) + or table.ndim != 2 + or starts.ndim != 1 + or lengths.ndim != 1 + or table.shape[0] != lengths.numel() + or starts.numel() != lengths.numel() + 1 + or table.dtype != torch.int32 + or starts.dtype != torch.int32 + or lengths.dtype != torch.int32 + or starts.stride(0) != 1 + or lengths.stride(0) != 1 + or not q.is_cuda + ): + return False + hkv = kc.shape[2] + dim = q.shape[-1] + page = kc.shape[1] + if hkv < 1 or q.shape[1] % hkv or not 1 <= q.shape[1] // hkv <= 16: + return False + if ( + k.shape != (q.shape[0], hkv, dim) + or v.shape != k.shape + or kc.shape != (kc.shape[0], page, hkv, dim) + or vc.shape != kc.shape + or any( + t.device != q.device for t in (k, v, out, kc, vc, table, starts, lengths) + ) + or any(t.stride(-1) != 1 for t in (q, k, v, out, kc, vc)) + ): + return False + if kc.element_size() == 1: + return all( + isinstance(s, torch.Tensor) + and s.numel() == 1 + and s.dtype == torch.float32 + and s.device == q.device + for s in (k_scale, v_scale) + ) + return True + + +def run_segmented_attention( + q, + out, + kc, + vc, + table, + starts, + lengths, + max_query_len, + max_seq_len, + k_scale, + v_scale, + scale, + *, + sliding_window=-1, + causal=True, + sinks=None, + config=None, + workspace=None, +): + """Write eligible unified-cache attention rows; leave other requests untouched.""" + batch, hq, dim = lengths.numel(), q.shape[1], q.shape[2] + if batch == 0 or max_query_len == 0 or q.numel() == 0: + return out + query_len = min(max_query_len, MAX_QUERY_LEN) + qcap = segmented_query_capacity(query_len) + fp8 = kc.element_size() == 1 + hk = kc.shape[2] + attention_span = max_seq_len + if sliding_window >= 0: + attention_span = min(max_seq_len, sliding_window + max_query_len) + cfg = config or select_segmented_config( + batch, query_len, attention_span, hq, hk, dim, fp8 + ) + splits = cfg["splits"] + shapes = segmented_workspace_shapes(batch, qcap, hq, hk, dim, splits) + if shapes is None: + partial = lse = out + elif workspace is not None: + partial, lse = ( + buffer.view(-1)[: shape.numel()].view(shape) + for buffer, shape in zip(workspace, map(torch.Size, shapes)) + ) + elif is_workspace_manager_initialized(): + partial, lse = current_workspace_manager().get_simultaneous( + (shapes[0], torch.float32), (shapes[1], torch.float32) + ) + else: + partial = torch.empty(shapes[0], device=q.device, dtype=torch.float32) + lse = torch.empty(shapes[1], device=q.device, dtype=torch.float32) + _launch_segmented_attention( + q, + out, + kc, + vc, + table, + starts, + lengths, + k_scale, + v_scale, + sinks, + scale, + cfg, + partial, + lse, + qcap, + sliding_window, + causal, + compile_only=False, + ) + return out + + +def compile_segmented_attention( + q, + out, + kc, + vc, + table, + starts, + lengths, + max_query_len, + k_scale, + v_scale, + scale, + config, + workspace, + *, + sliding_window=-1, + causal=True, + sinks=None, +): + """Compile one segmented attention configuration without launching it.""" + batch, hq, dim = lengths.numel(), q.shape[1], q.shape[2] + qcap = segmented_query_capacity(min(max_query_len, MAX_QUERY_LEN)) + hk = kc.shape[2] + shapes = segmented_workspace_shapes(batch, qcap, hq, hk, dim, config["splits"]) + if shapes is None: + partial = lse = out + else: + partial, lse = workspace + _launch_segmented_attention( + q, + out, + kc, + vc, + table, + starts, + lengths, + k_scale, + v_scale, + sinks, + scale, + config, + partial, + lse, + qcap, + sliding_window, + causal, + compile_only=True, + ) + + +def _launch_segmented_attention( + q, + out, + kc, + vc, + table, + starts, + lengths, + k_scale, + v_scale, + sinks, + scale, + cfg, + partial, + lse, + qcap, + sliding_window, + causal, + *, + compile_only, +): + """Launch or compile the exact stage and reduction specializations.""" + batch, hq, dim = lengths.numel(), q.shape[1], q.shape[2] + hk = kc.shape[2] + fp8 = kc.element_size() == 1 + splits = cfg["splits"] + stage_grid = ( + batch * hk, + triton.cdiv(qcap * (hq // hk), cfg["bm"]), + splits, + ) + stage_args = ( + q, + kc, + vc, + table, + starts, + lengths, + k_scale, + v_scale, + sinks, + partial, + lse, + out, + q.stride(0), + q.stride(1), + out.stride(0), + out.stride(1), + *kc.stride(), + *vc.stride(), + *table.stride(), + kc.shape[1], + hq, + hk, + dim, + qcap, + MAX_QUERY_LEN, + scale, + fp8, + cfg["bm"], + cfg["bn"], + cfg["bk"], + splits, + sliding_window, + causal, + sinks is not None, + cfg.get("prefix_fast", False), + cfg.get("pv_split", False), + cfg.get("qk_pipeline"), + ) + stage_options = dict( + num_warps=cfg["warps"], + num_stages=cfg["stages"], + waves_per_eu=cfg.get("waves_per_eu", 2), + ) + if compile_only: + _segmented_attention_stage.warmup(*stage_args, grid=stage_grid, **stage_options) + else: + _segmented_attention_stage[stage_grid](*stage_args, **stage_options) + if splits > 1: + reduce_d = cfg.get("reduce_d", 64 if batch * qcap * hq < 64 else dim) + reduce_grid = ( + qcap * (hq // hk), + batch * hk, + triton.cdiv(dim, reduce_d), + ) + reduce_args = ( + partial, + lse, + out, + starts, + out.stride(0), + out.stride(1), + hq, + hk, + dim, + qcap, + MAX_QUERY_LEN, + splits, + reduce_d, + ) + reduce_options = {"num_warps": cfg.get("reduce_warps", 4)} + if compile_only: + _segmented_attention_reduce.warmup( + *reduce_args, grid=reduce_grid, **reduce_options + ) + else: + _segmented_attention_reduce[reduce_grid](*reduce_args, **reduce_options) + + +def get_segmented_config(*args, **kwargs): + """Load the tuner on demand to avoid a kernel/tuner import cycle.""" + from vllm.model_executor.warmup.rocm_segmented_attn_autotune_warmup import ( + get_segmented_config as tuned_config, + ) + + return tuned_config(*args, **kwargs) + + +def segmented_attention( + query: torch.Tensor, + key: torch.Tensor, + value: torch.Tensor, + output: torch.Tensor, + kv_cache_dtype: str, + key_cache: torch.Tensor, + value_cache: torch.Tensor, + block_table: torch.Tensor, + query_start_loc: torch.Tensor, + seq_lens: torch.Tensor, + max_seq_len: int, + max_query_len: int, + k_scale: torch.Tensor, + v_scale: torch.Tensor, + sm_scale: float, + *, + sliding_window: int = -1, + output_scale: torch.Tensor | None = None, + sinks: torch.Tensor | None = None, + causal: bool | torch.Tensor = True, + softcap: float = 0.0, + workspace=None, + use_tuned_config: bool = True, +) -> None: + """Run segmented attention when eligible, otherwise use unified Triton.""" + if kv_cache_dtype in ("fp8", "fp8_e4m3"): + fp8_dtype = current_platform.fp8_dtype() + if key_cache.dtype == torch.uint8: + key_cache = key_cache.view(fp8_dtype) + value_cache = value_cache.view(fp8_dtype) + + from vllm.platforms.rocm import on_gfx1x, on_gfx12x + + segmented_pattern = ( + isinstance(causal, bool) + and (on_gfx12x() if key_cache.element_size() == 1 else on_gfx1x()) + and not softcap + and output_scale is None + and ( + sinks is None + or ( + sinks.ndim == 1 + and sinks.shape[0] == query.shape[1] + and sinks.device == query.device + and sinks.dtype in (torch.float16, torch.bfloat16, torch.float32) + ) + ) + ) + if ( + segmented_pattern + and 0 < max_query_len <= MAX_QUERY_LEN + and ( + can_use_segmented_attention( + query, + key, + value, + output, + key_cache, + value_cache, + block_table, + query_start_loc, + seq_lens, + k_scale, + v_scale, + ) + ) + ): + attention_span = max_seq_len + if sliding_window >= 0: + attention_span = min(max_seq_len, sliding_window + max_query_len) + config = None + if use_tuned_config: + config = get_segmented_config( + query.device, + query.dtype, + key_cache.dtype, + query.shape[1], + key_cache.shape[2], + query.shape[2], + key_cache.shape[1], + sm_scale, + len(seq_lens), + max_query_len, + attention_span, + sliding_window, + causal, + has_sinks=sinks is not None, + ) + if config is None: + config = select_segmented_config( + len(seq_lens), + max_query_len, + attention_span, + query.shape[1], + key_cache.shape[2], + query.shape[2], + key_cache.element_size() == 1, + ) + run_segmented_attention( + query, + output, + key_cache, + value_cache, + block_table, + query_start_loc, + seq_lens, + max_query_len, + max_seq_len, + k_scale, + v_scale, + sm_scale, + sliding_window=sliding_window, + causal=causal, + sinks=sinks, + config=config, + workspace=workspace, + ) + return + + unified_attention( + q=query, + k=key_cache, + v=value_cache, + out=output, + cu_seqlens_q=query_start_loc, + max_seqlen_q=max_query_len, + seqused_k=seq_lens, + max_seqlen_k=max_seq_len, + softmax_scale=sm_scale, + causal=causal, + window_size=(-1, -1) if sliding_window < 0 else (sliding_window, 0), + block_table=block_table, + softcap=softcap, + q_descale=None, + k_descale=k_scale, + v_descale=v_scale, + output_scale=output_scale, + sinks=sinks, + kv_quant_mode=get_kv_quant_mode(kv_cache_dtype), + ) diff --git a/vllm/v1/attention/ops/triton_prefill_attention.py b/vllm/v1/attention/ops/triton_prefill_attention.py index 1ed55c3e28..9e8482b5ec 100644 --- a/vllm/v1/attention/ops/triton_prefill_attention.py +++ b/vllm/v1/attention/ops/triton_prefill_attention.py @@ -243,9 +243,10 @@ def context_attention_fwd( b_seq_len: [b] out: [b * s, head, head_dim] """ - BLOCK = get_block_size(q.dtype) - Lq, Lk, _ = q.shape[-1], k.shape[-1], v.shape[-1] + BLOCK = get_block_size(q.dtype) + if Lk >= 512: + BLOCK = min(BLOCK, 32) sm_scale = 1.0 / (Lq**0.5) if softmax_scale is None else softmax_scale # rescale with 1/ln(2) for triton exp2 diff --git a/vllm/v1/attention/ops/triton_unified_attention.py b/vllm/v1/attention/ops/triton_unified_attention.py index 4d73e49e91..63a9cdc7f9 100644 --- a/vllm/v1/attention/ops/triton_unified_attention.py +++ b/vllm/v1/attention/ops/triton_unified_attention.py @@ -586,7 +586,15 @@ def kernel_unified_attention( P_v = (P * v_token_head_scales[None, :]).to(V.dtype) acc += tl.dot(P_v, V) else: - acc += tl.dot(P.to(V.dtype), V) + # Keep softmax weights representable when the P @ V dot uses FP8 + # tensor cores. Undo the power-of-two scale after the dot. + if V.dtype == tl.float8e4nv: + p_scale = 256.0 + elif V.dtype.is_fp8(): + p_scale = 128.0 + else: + p_scale = 1.0 + acc += tl.dot((P * p_scale).to(V.dtype), V) / p_scale # ---- Epilogue --------------------------------------------------------- if IS_3D: diff --git a/vllm/v1/attention/ops/triton_unified_attention_diffkv.py b/vllm/v1/attention/ops/triton_unified_attention_diffkv.py index eaf62b6bce..8480754256 100644 --- a/vllm/v1/attention/ops/triton_unified_attention_diffkv.py +++ b/vllm/v1/attention/ops/triton_unified_attention_diffkv.py @@ -60,6 +60,9 @@ def kernel_unified_attention_diffkv( query_ptr, key_cache_ptr, # view of packed cache: [..., :head_size_qk] value_cache_ptr, # view of packed cache: [..., head_size_qk:hqk+hv] + q_descale_ptr, + k_descale_ptr, + v_descale_ptr, sink_ptr, block_tables_ptr, seq_lens_ptr, @@ -83,6 +86,8 @@ def kernel_unified_attention_diffkv( USE_ALIBI_SQRT: tl.constexpr, USE_SOFTCAP: tl.constexpr, USE_SINKS: tl.constexpr, + USE_Q_SCALE: tl.constexpr, + USE_KV_SCALES: tl.constexpr, SLIDING_WINDOW: tl.constexpr, # Strides for both cache views (they share the same packed buffer, so # dims 0/1/2 strides match; only the per-head extent differs). @@ -154,6 +159,12 @@ def kernel_unified_attention_diffkv( other=0.0, ) + if USE_Q_SCALE: + scale *= tl.load(q_descale_ptr) + if USE_KV_SCALES: + scale *= tl.load(k_descale_ptr) + v_descale = tl.load(v_descale_ptr) + block_table_offset = seq_idx * block_table_stride M = init_softmax_M( @@ -262,6 +273,9 @@ def kernel_unified_attention_diffkv( acc += tl.dot(P.to(V.dtype), V) # ---- Epilogue -------------------------------------------------------- + if USE_KV_SCALES: + acc *= v_descale + if IS_3D: # Store per-segment partials; finalized by reduce_segments_diffkv. segm_output_offset = ( @@ -403,6 +417,9 @@ def unified_attention_diffkv( softmax_segm_output: torch.Tensor | None = None, softmax_segm_max: torch.Tensor | None = None, softmax_segm_expsum: torch.Tensor | None = None, + k_descale: torch.Tensor | None = None, + v_descale: torch.Tensor | None = None, + q_descale: torch.Tensor | None = None, ): assert causal, "Only causal attention is supported" @@ -470,6 +487,9 @@ def unified_attention_diffkv( query_ptr=q, key_cache_ptr=k, value_cache_ptr=v, + q_descale_ptr=q_descale, + k_descale_ptr=k_descale, + v_descale_ptr=v_descale, sink_ptr=sinks, block_tables_ptr=block_table, seq_lens_ptr=seqused_k, @@ -493,6 +513,8 @@ def unified_attention_diffkv( USE_ALIBI_SQRT=use_alibi_sqrt, USE_SOFTCAP=(softcap > 0), USE_SINKS=(sinks is not None), + USE_Q_SCALE=q_descale is not None, + USE_KV_SCALES=k_descale is not None, SLIDING_WINDOW=sliding_window_val, stride_k_cache_0=k.stride(0), stride_k_cache_1=k.stride(1), diff --git a/vllm/v1/attention/ops/ultraquant/__init__.py b/vllm/v1/attention/ops/ultraquant/__init__.py new file mode 100644 index 0000000000..dbe17bfdc8 --- /dev/null +++ b/vllm/v1/attention/ops/ultraquant/__init__.py @@ -0,0 +1,19 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""UltraQuant 4-bit KV cache: FP4 E2M1 codes + UE8M0 group-of-32 scales. + +Production decode uses the FlyDSL D=256 kernel on gfx950, with Triton +unified attention as the fallback. Slot size is ``slot_size(head_dim)`` +(272 B at D=256). Format helpers live in ``format``; import them from +there, not this package root. +""" + +__all__ = ["ultraquant_store"] + + +def __getattr__(name: str): + if name == "ultraquant_store": + from vllm.v1.attention.ops.ultraquant.triton_store import ultraquant_store + + return ultraquant_store + raise AttributeError(f"module {__name__!r} has no attribute {name!r}") diff --git a/vllm/v1/attention/ops/ultraquant/format.py b/vllm/v1/attention/ops/ultraquant/format.py new file mode 100644 index 0000000000..20977d528f --- /dev/null +++ b/vllm/v1/attention/ops/ultraquant/format.py @@ -0,0 +1,218 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""FP4 codepoint + UE8M0 scale constants for the UltraQuant KV cache. + +K/V codes are FP4 E2M1. Per-group scales are UE8M0 (one byte, a power of +two): ``s = 2^round(log2(c · absmax))`` with ``c = 0.156``. Decode +consumes Q as FP8 E4M3 so scaled F8F6F4 MFMA can run natively on CDNA4. + +AoS slot layout (group_size=32), per (token, head): + bytes [0 .. D/2) : K codes (FP4 nibbles, 2/byte) + bytes [D/2 .. D/2 + Gk) : K scales (UE8M0, 1 byte × Gk groups) + bytes [D/2 + Gk .. D + Gk) : V codes + bytes [D + Gk .. D + 2*Gk) : V scales + D=256 → Gk=8 → 272 B/slot. D=128 → Gk=4 → 136 B/slot. + +No per-token norm-fold. No V rotation. K is Hadamard-rotated at store. +""" + +from __future__ import annotations + +import math + +# ── 15 unique FP4 E2M1 levels, sorted ascending ──── +FP4_LEVELS_SORTED: tuple[float, ...] = ( + -6.0, + -4.0, + -3.0, + -2.0, + -1.5, + -1.0, + -0.5, + 0.0, + 0.5, + 1.0, + 1.5, + 2.0, + 3.0, + 4.0, + 6.0, +) +assert len(FP4_LEVELS_SORTED) == 15 + +# Midpoints between consecutive sorted levels (14 boundaries). +MIDPOINTS_SORTED: tuple[float, ...] = tuple( + (FP4_LEVELS_SORTED[i] + FP4_LEVELS_SORTED[i + 1]) / 2.0 + for i in range(len(FP4_LEVELS_SORTED) - 1) +) +assert len(MIDPOINTS_SORTED) == 14 + +# FP4 E2M1 bit-pattern decode (16 entries indexed by 4-bit code). +# Sign bit (MSB) | 2 exp bits | 1 mantissa bit (LSB). +FP4_BITS_TO_VALUE: tuple[float, ...] = ( + 0.0, + 0.5, + 1.0, + 1.5, + 2.0, + 3.0, + 4.0, + 6.0, + -0.0, + -0.5, + -1.0, + -1.5, + -2.0, + -3.0, + -4.0, + -6.0, +) +assert len(FP4_BITS_TO_VALUE) == 16 + +# Sorted index → FP4 E2M1 bit pattern (uint4). +SORTED_TO_BITS: tuple[int, ...] = ( + 0b1111, + 0b1110, + 0b1101, + 0b1100, + 0b1011, + 0b1010, + 0b1001, + 0b0000, + 0b0001, + 0b0010, + 0b0011, + 0b0100, + 0b0101, + 0b0110, + 0b0111, +) +assert len(SORTED_TO_BITS) == 15 +for _i, _bits in enumerate(SORTED_TO_BITS): + _v = FP4_BITS_TO_VALUE[_bits] + _v = 0.0 if _v == 0.0 else _v + assert _v == FP4_LEVELS_SORTED[_i] + +# ── Format constants ────────────────────────────────────────────────────── +GROUP_SIZE: int = 32 +"""Elements per group; one E8M0 scale per group. Fixed at 32 because the +AMD scaled F8F6F4 MFMA instruction also consumes one E8M0 scale per 32 +elements along K. Other group sizes would force a fallback to plain MFMA ++ accumulator-side scale multiply (no hardware fast-path).""" + +FP4_MAX: float = 6.0 +"""Largest representable FP4 magnitude.""" + +DEFAULT_CONSTANT_C: float = 0.156 +"""MSE-optimal scale constant; ``s_raw = c * absmax``, then UE8M0-snapped.""" + + +def get_constant_c() -> float: + """Return the fixed UltraQuant scale constant.""" + return DEFAULT_CONSTANT_C + + +def get_group_size() -> int: + """Return the fixed group size required by scaled MFMA.""" + return GROUP_SIZE + + +# ── Scale encoding ──────────────────────────────────────────────────────── +# Per-group scale `s = pow2_round(c · absmax)` stored as the E8M0 byte; +# codebook is the raw FP4 grid {0, ±0.5, ..., ±6}; dequant = code · s. + + +# ── UE8M0 helpers ────────────────────────────────────────────────────────── +# E8M0 = 8 exponent bits, 0 mantissa bits, no sign. The byte value `e` +# represents `2^(e - 127)` for e in [1, 254]. e=0 and e=255 are reserved +# (zero and NaN respectively in the OCP spec). We use e=0 as our zero +# sentinel for sink-zero groups. + +UE8M0_BIAS: int = 127 + +# fp32 minimum positive normal = 2^-126 ≈ 1.175e-38. Anything smaller +# clamps to the zero sentinel. +_UE8M0_MIN_EXP: int = -126 +_UE8M0_MAX_EXP: int = 127 + + +def ue8m0_encode(s: float) -> int: + """Snap a positive fp32 scale `s` to the nearest power of 2 and + encode as a UE8M0 byte. `s <= 0` encodes as 0 (zero sentinel).""" + if s <= 0.0 or not math.isfinite(s): + return 0 + exp = int(round(math.log2(s))) + if exp < _UE8M0_MIN_EXP: + return 0 + if exp > _UE8M0_MAX_EXP: + exp = _UE8M0_MAX_EXP + return exp + UE8M0_BIAS + + +def ue8m0_decode(byte: int) -> float: + """Decode a UE8M0 byte back to fp32. 0 → 0.0 (zero sentinel).""" + if byte == 0: + return 0.0 + return 2.0 ** (byte - UE8M0_BIAS) + + +# ── Per-slot AoS byte layout ────────────────────────────────────────────── +# Per slot, per head (group_size=GS=32, Gk = D/GS): +# bytes [0 .. D/2) : K codes (2 nibbles/byte) +# bytes [D/2 .. D/2 + Gk) : K scales (E8M0, 1 byte each) +# bytes [D/2 + Gk .. D + Gk) : V codes +# bytes [D + Gk .. D + 2*Gk) : V scales (E8M0, 1 byte each) +# D=256 → 272 B/slot. D=128 → 136 B/slot. + + +def k_codes_bytes(head_dim: int) -> int: + """Bytes for one head's packed K codes.""" + if head_dim % 2 != 0: + raise ValueError(f"head_dim must be even, got {head_dim}") + return head_dim // 2 + + +def n_groups(head_dim: int, group_size: int | None = None) -> int: + gs = group_size if group_size is not None else get_group_size() + if head_dim % gs != 0: + raise ValueError(f"head_dim={head_dim} must be a multiple of group_size={gs}") + return head_dim // gs + + +def k_scales_bytes(head_dim: int, group_size: int | None = None) -> int: + """Bytes for one head's K scales (one E8M0 byte per group).""" + return n_groups(head_dim, group_size) + + +def v_codes_bytes(head_dim: int) -> int: + return k_codes_bytes(head_dim) + + +def v_scales_bytes(head_dim: int, group_size: int | None = None) -> int: + return k_scales_bytes(head_dim, group_size) + + +def slot_size(head_dim: int, group_size: int | None = None) -> int: + """Bytes per (token, head) slot.""" + return ( + k_codes_bytes(head_dim) + + k_scales_bytes(head_dim, group_size) + + v_codes_bytes(head_dim) + + v_scales_bytes(head_dim, group_size) + ) + + +def k_codes_offset(head_dim: int, group_size: int | None = None) -> int: + return 0 + + +def k_scales_offset(head_dim: int, group_size: int | None = None) -> int: + return k_codes_bytes(head_dim) + + +def v_codes_offset(head_dim: int, group_size: int | None = None) -> int: + return k_codes_bytes(head_dim) + k_scales_bytes(head_dim, group_size) + + +def v_scales_offset(head_dim: int, group_size: int | None = None) -> int: + return v_codes_offset(head_dim, group_size) + v_codes_bytes(head_dim) diff --git a/vllm/v1/attention/ops/ultraquant/reference.py b/vllm/v1/attention/ops/ultraquant/reference.py new file mode 100644 index 0000000000..0d7dbe745b --- /dev/null +++ b/vllm/v1/attention/ops/ultraquant/reference.py @@ -0,0 +1,349 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""PyTorch reference for the UltraQuant KV cache format. + +Scale is UE8M0 (power of two): after ``s = c * absmax`` snap +``s = 2^round(log2(s))`` and store one byte. Q is Hadamard-rotated then +cast to FP8 E4M3 before the QK matmul, matching the kernel launcher. + +Same FP4 E2M1 grid. ``c = 0.156``. No per-token L2 norm. No V rotation. +K is rotated at encode; Q arrives pre-rotated at decode. + +This module is the bit-comparison ground truth for the kernels. +""" + +from __future__ import annotations + +from dataclasses import dataclass + +import torch + +from vllm.v1.attention.ops.ultraquant.format import ( + FP4_LEVELS_SORTED, + GROUP_SIZE, + MIDPOINTS_SORTED, + SORTED_TO_BITS, + UE8M0_BIAS, + get_constant_c, +) + +_LEVELS_CACHE: dict[tuple[torch.device, torch.dtype], torch.Tensor] = {} +_MIDPOINTS_CACHE: dict[tuple[torch.device, torch.dtype], torch.Tensor] = {} +_SORTED_TO_BITS_CACHE: dict[torch.device, torch.Tensor] = {} + + +def _get_levels(device: torch.device, dtype: torch.dtype) -> torch.Tensor: + key = (device, dtype) + t = _LEVELS_CACHE.get(key) + if t is None: + t = torch.tensor(FP4_LEVELS_SORTED, device=device, dtype=dtype) + _LEVELS_CACHE[key] = t + return t + + +def _get_midpoints(device: torch.device, dtype: torch.dtype) -> torch.Tensor: + key = (device, dtype) + t = _MIDPOINTS_CACHE.get(key) + if t is None: + t = torch.tensor(MIDPOINTS_SORTED, device=device, dtype=dtype) + _MIDPOINTS_CACHE[key] = t + return t + + +def _get_sorted_to_bits(device: torch.device) -> torch.Tensor: + t = _SORTED_TO_BITS_CACHE.get(device) + if t is None: + t = torch.tensor(SORTED_TO_BITS, device=device, dtype=torch.uint8) + _SORTED_TO_BITS_CACHE[device] = t + return t + + +_HADAMARD_CACHE: dict[tuple[int, torch.device, torch.dtype], torch.Tensor] = {} + + +def hadamard_matrix( + dim: int, device: torch.device, dtype: torch.dtype = torch.float32 +) -> torch.Tensor: + """Sylvester Hadamard, normalised so H @ H.T = I.""" + if dim <= 0 or (dim & (dim - 1)) != 0: + raise ValueError(f"hadamard_matrix requires power-of-two dim, got {dim}") + key = (dim, device, dtype) + cached = _HADAMARD_CACHE.get(key) + if cached is not None: + return cached + H = torch.tensor([[1.0]], dtype=torch.float64) + while H.shape[0] < dim: + H = torch.cat([torch.cat([H, H], dim=1), torch.cat([H, -H], dim=1)], dim=0) + H = (H / (dim**0.5)).to(device=device, dtype=dtype).contiguous() + _HADAMARD_CACHE[key] = H + return H + + +# ── Core quantization primitives ─────────────────────────────────────────── +def _snap_to_sorted_idx( + x_norm: torch.Tensor, *, dtype: torch.dtype = torch.float32 +) -> torch.Tensor: + """Snap normalised values to nearest FP4 level, returning sorted index + in [0, 14]. Same semantic as `torch.bucketize` against midpoints + (reproduced by 14 `tl.where` in the kernel).""" + boundaries = _get_midpoints(x_norm.device, dtype) + x = x_norm.to(dtype) + # ROCm's bucketize uses int32 element indexing and raises + # hipErrorInvalidConfiguration once numel >= 2**31 (e.g. a [B, Hk, N, D] + # K tensor at B=128, N=16384, Hk=8, D=128 is exactly 2**31). Chunk the + # flattened input so each call stays under the limit (bit-identical output). + _INT32_LIMIT = 2**31 + if x.numel() < _INT32_LIMIT: + return torch.bucketize(x, boundaries) + flat = x.reshape(-1) + out = torch.empty_like(flat, dtype=torch.long) + chunk = _INT32_LIMIT // 2 + for i in range(0, flat.numel(), chunk): + out[i : i + chunk] = torch.bucketize(flat[i : i + chunk], boundaries) + return out.reshape(x.shape) + + +def _pack_nibbles_last_dim(codes: torch.Tensor) -> torch.Tensor: + if codes.shape[-1] % 2 != 0: + raise ValueError(f"pack_nibbles last dim must be even, got {codes.shape[-1]}") + lo = codes[..., 0::2] & 0xF + hi = codes[..., 1::2] & 0xF + return (lo | (hi << 4)).to(torch.uint8) + + +def _unpack_nibbles_last_dim(packed: torch.Tensor, out_dim: int) -> torch.Tensor: + if out_dim != 2 * packed.shape[-1]: + packed_dim = packed.shape[-1] + raise ValueError( + f"unpack_nibbles: out_dim={out_dim} must be 2*packed.last={2 * packed_dim}" + ) + p = packed.to(torch.int32) + lo = (p & 0xF).to(torch.uint8) + hi = ((p >> 4) & 0xF).to(torch.uint8) + out_shape = packed.shape[:-1] + (out_dim,) + out = torch.empty(out_shape, dtype=torch.uint8, device=packed.device) + out[..., 0::2] = lo + out[..., 1::2] = hi + return out + + +def _ue8m0_snap_tensor( + s_raw: torch.Tensor, +) -> tuple[torch.Tensor, torch.Tensor]: + """Snap a positive-scale tensor `s_raw` to a power of two and return + ``(s_snapped_fp32, byte_uint8)``. + + Matches the kernel: zero / non-finite inputs map to byte=0 / value=0.0. + Round-half-up via ``floor(log2(s) + 0.5)``, matching Triton's + ``tl.floor(... + 0.5)`` idiom. + """ + is_zero = (s_raw <= 0) | ~torch.isfinite(s_raw) + s_safe = torch.where(is_zero, torch.ones_like(s_raw), s_raw) + log2_s = torch.log2(s_safe) + exp = torch.floor(log2_s + 0.5).to(torch.int32) + # Clamp to UE8M0's representable exponent range (signed pre-bias). + exp = torch.clamp(exp, min=-126, max=127) + byte = (exp + UE8M0_BIAS).to(torch.int32) + byte = torch.where(is_zero, torch.zeros_like(byte), byte).to(torch.uint8) + s_snapped = torch.where( + is_zero, torch.zeros_like(s_raw), torch.exp2(exp.to(torch.float32)) + ) + return s_snapped, byte + + +@dataclass +class UltraQuantEncoded: + """Encoded ultraquant representation of one tensor along its last axis. + + Shapes assume input shape `[..., D]`: + - codes_packed: `[..., D // 2]` uint8 — 2 FP4 nibbles per byte + - scale_bytes: `[..., D // GROUP_SIZE]` uint8 — E8M0 byte per group + Dequant is ``code · 2^(byte - 127)`` (`c` is folded into the byte). + """ + + codes_packed: torch.Tensor # uint8, [..., D//2] + scale_bytes: torch.Tensor # uint8, [..., D//GROUP_SIZE] + + @property + def scales_fp32(self) -> torch.Tensor: + """Decode the E8M0 bytes back to fp32 (for dequant / inspection).""" + b = self.scale_bytes.to(torch.int32) + exp = b - UE8M0_BIAS + is_zero = b == 0 + s = torch.where( + is_zero, + torch.zeros_like(b, dtype=torch.float32), + torch.exp2(exp.to(torch.float32)), + ) + return s + + +def ultraquant_encode( + x: torch.Tensor, + *, + rotate: bool = True, + constant_c: float | None = None, +) -> UltraQuantEncoded: + """Encode `x` (shape [..., D]) to FP4 codes + E8M0 per-group-of-32 scales. + + Steps: + 1. Optional Hadamard rotation: `x_rot = x @ H.T` + 2. Reshape to `[..., G, GROUP_SIZE]`. + 3. `absmax = max(|x_rot|)` per group. + 4. `s_raw = c · absmax`; `s_snapped = 2^round(log2(s_raw))`. + Zero-amax groups encode byte=0. + 5. `sorted_idx = bucketize(x_g / s_snapped, midpoints)` ∈ [0, 14]. + 6. Remap sorted_idx → FP4 E2M1 bit pattern via SORTED_TO_BITS. + 7. Pack pairs of 4-bit codes into bytes. + """ + if x.shape[-1] % GROUP_SIZE != 0: + raise ValueError( + f"ultraquant_encode: last dim {x.shape[-1]} must be a multiple of " + f"GROUP_SIZE={GROUP_SIZE}" + ) + D = x.shape[-1] + G = D // GROUP_SIZE + c = constant_c if constant_c is not None else get_constant_c() + + x_f32 = x.to(torch.float32) + if rotate: + H = hadamard_matrix(D, x.device, torch.float32) + x_rot = x_f32 @ H.T + else: + x_rot = x_f32 + + x_g = x_rot.reshape(*x.shape[:-1], G, GROUP_SIZE) + absmax = x_g.abs().amax(dim=-1, keepdim=True) # [..., G, 1] + s_raw = absmax * c + s_snapped, scale_bytes = _ue8m0_snap_tensor(s_raw.squeeze(-1)) + s_snapped = s_snapped.unsqueeze(-1) # [..., G, 1] + s_for_div = torch.where(s_snapped == 0, torch.ones_like(s_snapped), s_snapped) + + sorted_idx = _snap_to_sorted_idx(x_g / s_for_div) + sorted_idx = torch.where( + (s_snapped == 0).expand_as(sorted_idx), + torch.full_like(sorted_idx, 7), # sorted idx 7 → +0.0 + sorted_idx, + ) + + sorted_to_bits = _get_sorted_to_bits(x.device) + fp4_bits = sorted_to_bits[sorted_idx] # [..., G, GS] + fp4_bits_flat = fp4_bits.reshape(*x.shape[:-1], D) + codes_packed = _pack_nibbles_last_dim(fp4_bits_flat) # [..., D//2] + + return UltraQuantEncoded( + codes_packed=codes_packed, + scale_bytes=scale_bytes, + ) + + +def ultraquant_dequant( + encoded: UltraQuantEncoded, + head_dim: int, +) -> torch.Tensor: + """Decode `encoded` to fp32 (no inverse rotation — result in the + rotated basis if rotation was applied at encode). + + Dequant is ``code · 2^(byte - 127)`` (`c` is already folded into the byte). + """ + G = head_dim // GROUP_SIZE + codes = _unpack_nibbles_last_dim(encoded.codes_packed, head_dim) + bits_to_val = torch.tensor( + [ + 0.0, + 0.5, + 1.0, + 1.5, + 2.0, + 3.0, + 4.0, + 6.0, + -0.0, + -0.5, + -1.0, + -1.5, + -2.0, + -3.0, + -4.0, + -6.0, + ], + device=encoded.codes_packed.device, + dtype=torch.float32, + ) + decoded = bits_to_val[codes.to(torch.long)] # [..., D] fp32 + decoded_g = decoded.reshape(*decoded.shape[:-1], G, GROUP_SIZE) + scales = encoded.scales_fp32.unsqueeze(-1) # [..., G, 1] + return (decoded_g * scales).reshape(*decoded.shape[:-1], head_dim) + + +def ultraquant_encode_decode( + x: torch.Tensor, + *, + rotate: bool = True, + constant_c: float | None = None, +) -> torch.Tensor: + """Full encode→decode round-trip in the rotated basis. Returns fp32.""" + enc = ultraquant_encode( + x, + rotate=rotate, + constant_c=constant_c, + ) + return ultraquant_dequant(enc, x.shape[-1]) + + +# ── Reference attention against the ultraquant encoded cache ────────────────── +def reference_ultraquant_attention( + query: torch.Tensor, # [B, Hq, D] bf16 — RAW (will be rotated + cast to FP8 here) + key: torch.Tensor, # [B, N, Hk, D] bf16 + value: torch.Tensor, # [B, N, Hk, D] bf16 + *, + scale: float, + sinks: torch.Tensor | None = None, + constant_c: float | None = None, +) -> torch.Tensor: + """End-to-end reference attention with ultraquant K/V encode→decode in + the rotated basis. Q is Hadamard-rotated AND cast through FP8 E4M3 + (scale = 1) before the QK matmul — mirrors what the launcher does. + + Output shape: [B, Hq, D] in query.dtype. + """ + B, Hq, D = query.shape + B2, N, Hk, D2 = key.shape + assert B == B2 and D == D2, (B, B2, D, D2) + assert value.shape == key.shape + assert Hq % Hk == 0, f"Hq={Hq} must be a multiple of Hk={Hk}" + g = Hq // Hk + + H = hadamard_matrix(D, query.device, torch.float32) + q_rot = query.to(torch.float32) @ H.T + # Round-trip Q through FP8 E4M3 with scale=1 — same precision haircut + # as the launcher (`q_rot.to(torch.float8_e4m3fn)`). + q_fp8 = q_rot.to(torch.float8_e4m3fn) + q_for_matmul = q_fp8.to(torch.float32) + + k_recon_rot = ultraquant_encode_decode( + key.transpose(1, 2).contiguous(), + rotate=True, + constant_c=constant_c, + ) + v_recon = ultraquant_encode_decode( + value.transpose(1, 2).contiguous(), + rotate=False, + constant_c=constant_c, + ) + + k_recon_rot = k_recon_rot.unsqueeze(2).expand(B, Hk, g, N, D).reshape(B, Hq, N, D) + v_recon = v_recon.unsqueeze(2).expand(B, Hk, g, N, D).reshape(B, Hq, N, D) + + qk = torch.einsum("bhd,bhnd->bhn", q_for_matmul, k_recon_rot) * scale # [B, Hq, N] + + if sinks is not None: + sink_log = sinks.to(torch.float32).reshape(1, Hq, 1).expand(B, Hq, 1) + qk_full = torch.cat([qk, sink_log], dim=-1) + p_full = torch.softmax(qk_full, dim=-1) + p = p_full[..., :N] + else: + p = torch.softmax(qk, dim=-1) + + out = torch.einsum("bhn,bhnd->bhd", p, v_recon) + return out.to(query.dtype) diff --git a/vllm/v1/attention/ops/ultraquant/triton_dequant.py b/vllm/v1/attention/ops/ultraquant/triton_dequant.py new file mode 100644 index 0000000000..9ff8d01bf4 --- /dev/null +++ b/vllm/v1/attention/ops/ultraquant/triton_dequant.py @@ -0,0 +1,190 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Full KV dequant for the UltraQuant cache format. + +Used by continuation prefill: when a chunk brings many new query tokens on +top of a long cached prefix, dequanting the prefix once and running a dense +prefill kernel beats replaying the decode kernel per query token. + +Reads FP4 codes plus UE8M0 group scales and writes K (Hadamard-rotated, as +stored) and V (raw) into pre-allocated fp16/bf16 buffers. +""" + +from __future__ import annotations + +import torch + +from vllm.triton_utils import tl, triton +from vllm.v1.attention.ops.ultraquant.format import ( + FP4_BITS_TO_VALUE, + UE8M0_BIAS, + get_group_size, + k_scales_offset, + n_groups, + v_codes_offset, + v_scales_offset, +) +from vllm.v1.attention.ops.ultraquant.triton_store import _kv_cache_flat + +_FP4_DECODE_CACHE: dict[tuple[torch.device, torch.dtype], torch.Tensor] = {} + + +def _get_fp4_decode_table(device: torch.device, dtype: torch.dtype) -> torch.Tensor: + """16-entry FP4 E2M1 bit-pattern -> value table (cached per device).""" + key = (device, dtype) + t = _FP4_DECODE_CACHE.get(key) + if t is None: + t = torch.tensor(FP4_BITS_TO_VALUE, device=device, dtype=dtype).contiguous() + _FP4_DECODE_CACHE[key] = t + return t + + +@triton.jit +def _ultraquant_full_dequant_kv( + KV_cache_ptr, # uint8 view, flat + Block_table_ptr, # [B, max_num_blocks] int32 + Fp4_decode_ptr, # [16] FP4 bit pattern -> value + K_out_ptr, + V_out_ptr, + stride_ko_b: tl.int64, + stride_ko_h: tl.int64, + stride_ko_s: tl.int64, + stride_vo_b: tl.int64, + stride_vo_h: tl.int64, + stride_vo_s: tl.int64, + stride_cache_block: tl.int64, + stride_cache_pos: tl.int64, + stride_cache_head: tl.int64, + stride_bt_b: tl.int64, + HEAD_DIM: tl.constexpr, + BLOCK_SIZE: tl.constexpr, + NUM_KV_HEADS: tl.constexpr, + K_SCALES_OFFSET: tl.constexpr, + V_CODES_OFFSET: tl.constexpr, + V_SCALES_OFFSET: tl.constexpr, + GROUP_SIZE_C: tl.constexpr, + N_GROUPS_C: tl.constexpr, + BLOCK_D: tl.constexpr, + OUT_BF16: tl.constexpr, + UE8M0_BIAS_C: tl.constexpr, +): + pos = tl.program_id(0) + bh = tl.program_id(1) + bid = bh // NUM_KV_HEADS + hid = bh % NUM_KV_HEADS + + page_idx = pos // BLOCK_SIZE + page_off = pos % BLOCK_SIZE + block_num = tl.load(Block_table_ptr + bid * stride_bt_b + page_idx).to(tl.int64) + slot_base = ( + block_num * stride_cache_block + + page_off * stride_cache_pos + + tl.cast(hid, tl.int64) * stride_cache_head + ) + + d_offs = tl.arange(0, BLOCK_D) + d_mask = d_offs < HEAD_DIM + byte_idx = (d_offs // 2).to(tl.int32) + nibble_shift = ((d_offs % 2) * 4).to(tl.int32) + g_offs = tl.arange(0, N_GROUPS_C) + + out_dtype = tl.bfloat16 if OUT_BF16 else tl.float16 + + # K side: codes are packed nibbles, one UE8M0 byte per group of 32. + k_byte_raw = tl.load(KV_cache_ptr + slot_base + byte_idx, mask=d_mask, other=0).to( + tl.int32 + ) + k_codes = (k_byte_raw >> nibble_shift) & 0xF + k_dec = tl.load(Fp4_decode_ptr + k_codes).to(tl.float32) + k_dec = tl.where(d_mask, k_dec, 0.0) + + # UE8M0 scale: value = 2^(byte - bias); byte == 0 encodes exact zero. + k_scale_bytes = tl.load(KV_cache_ptr + slot_base + K_SCALES_OFFSET + g_offs).to( + tl.int32 + ) + k_scales = tl.where( + k_scale_bytes == 0, + 0.0, + tl.exp2(tl.cast(k_scale_bytes - UE8M0_BIAS_C, tl.float32)), + ) + + k_g = tl.reshape(k_dec, [N_GROUPS_C, GROUP_SIZE_C]) + k_recon = tl.reshape(k_g * k_scales[:, None], [BLOCK_D]) + k_recon = tl.where(d_mask, k_recon, 0.0) + + ko_base = bid * stride_ko_b + hid * stride_ko_h + pos * stride_ko_s + tl.store(K_out_ptr + ko_base + d_offs, k_recon.to(out_dtype), mask=d_mask) + + # V side mirrors K but is stored unrotated. + v_byte_raw = tl.load( + KV_cache_ptr + slot_base + V_CODES_OFFSET + byte_idx, + mask=d_mask, + other=0, + ).to(tl.int32) + v_codes = (v_byte_raw >> nibble_shift) & 0xF + v_dec = tl.load(Fp4_decode_ptr + v_codes).to(tl.float32) + v_dec = tl.where(d_mask, v_dec, 0.0) + + v_scale_bytes = tl.load(KV_cache_ptr + slot_base + V_SCALES_OFFSET + g_offs).to( + tl.int32 + ) + v_scales = tl.where( + v_scale_bytes == 0, + 0.0, + tl.exp2(tl.cast(v_scale_bytes - UE8M0_BIAS_C, tl.float32)), + ) + + v_g = tl.reshape(v_dec, [N_GROUPS_C, GROUP_SIZE_C]) + v_recon = tl.reshape(v_g * v_scales[:, None], [BLOCK_D]) + v_recon = tl.where(d_mask, v_recon, 0.0) + + vo_base = bid * stride_vo_b + hid * stride_vo_h + pos * stride_vo_s + tl.store(V_out_ptr + vo_base + d_offs, v_recon.to(out_dtype), mask=d_mask) + + +def ultraquant_full_dequant_kv( + kv_cache: torch.Tensor, + block_table: torch.Tensor, + k_out: torch.Tensor, + v_out: torch.Tensor, + alloc_len: int, +) -> None: + """Dequant ``alloc_len`` cached positions into ``k_out`` / ``v_out``. + + ``k_out`` / ``v_out`` are ``[B, Hk, alloc_len, D]`` in fp16 or bf16. + """ + B = block_table.shape[0] + Hk = kv_cache.shape[2] + D = k_out.shape[3] + block_size = kv_cache.shape[1] + + gs = get_group_size() + _ultraquant_full_dequant_kv[(alloc_len, B * Hk)]( + _kv_cache_flat(kv_cache), + block_table, + _get_fp4_decode_table(kv_cache.device, k_out.dtype), + k_out, + v_out, + k_out.stride(0), + k_out.stride(1), + k_out.stride(2), + v_out.stride(0), + v_out.stride(1), + v_out.stride(2), + kv_cache.stride(0), + kv_cache.stride(1), + kv_cache.stride(2), + block_table.stride(0), + HEAD_DIM=D, + BLOCK_SIZE=block_size, + NUM_KV_HEADS=Hk, + K_SCALES_OFFSET=k_scales_offset(D, gs), + V_CODES_OFFSET=v_codes_offset(D, gs), + V_SCALES_OFFSET=v_scales_offset(D, gs), + GROUP_SIZE_C=gs, + N_GROUPS_C=n_groups(D, gs), + BLOCK_D=triton.next_power_of_2(D), + OUT_BF16=1 if k_out.dtype == torch.bfloat16 else 0, + UE8M0_BIAS_C=UE8M0_BIAS, + num_warps=4, + ) diff --git a/vllm/v1/attention/ops/ultraquant/triton_store.py b/vllm/v1/attention/ops/ultraquant/triton_store.py new file mode 100644 index 0000000000..b43f4264e7 --- /dev/null +++ b/vllm/v1/attention/ops/ultraquant/triton_store.py @@ -0,0 +1,351 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Triton store kernel for the ultraquant KV cache format. + +Per-token, per-head, per-group-of-32 the kernel computes: + + s_raw = c * absmax(group) # c = 0.156 by default + s = 2^round(log2(s_raw)) # UE8M0 snap (power of 2) + code = quantize_to_fp4(group / s) # OCP E2M1, 4 bits/elt + +and stores ``code`` as packed nibbles + ``s`` as a single UE8M0 byte per +group. The output layout matches ``format.py`` (272 B/slot for D=256). + +K is Hadamard-rotated in-register; V is not. There is no per-token L2 +norm. The UE8M0 snap makes ``s`` a power of two so scaled F8F6F4 MFMA +can consume it on the read side. +""" + +from __future__ import annotations + +from typing import Any + +import torch + +from vllm.triton_utils import tl, triton +from vllm.v1.attention.ops.ultraquant.format import ( + MIDPOINTS_SORTED, + SORTED_TO_BITS, + UE8M0_BIAS, + get_constant_c, + get_group_size, + k_scales_offset, + n_groups, + slot_size, + v_codes_offset, + v_scales_offset, +) + + +def _kv_cache_flat(kv_cache: torch.Tensor) -> torch.Tensor: + """1-D uint8 view of the raw KV allocation (handles padded layouts).""" + if kv_cache.is_contiguous(): + return kv_cache.reshape(-1) + n = kv_cache.untyped_storage().nbytes() // kv_cache.element_size() + return torch.empty(0, dtype=kv_cache.dtype, device=kv_cache.device).set_( + kv_cache.untyped_storage(), 0, (n,) + ) + + +_SUPPORTED_DTYPES = {torch.float16, torch.bfloat16} + +# Built lazily on first launch so importing this module does not require triton. +_MIDPOINTS_C: Any = None + + +def _ensure_const_tables() -> None: + global _MIDPOINTS_C + if _MIDPOINTS_C is None: + _MIDPOINTS_C = tl.constexpr(tuple(float(m) for m in MIDPOINTS_SORTED)) + + +# ── Cached per-device constant tensors ───────────────────────────────────── +_MIDPOINTS_CACHE: dict[torch.device, torch.Tensor] = {} +_SORTED_TO_BITS_CACHE: dict[torch.device, torch.Tensor] = {} +_PIT_CACHE: dict[tuple[int, torch.device, torch.dtype], torch.Tensor] = {} + + +def _get_midpoints_tensor(device: torch.device) -> torch.Tensor: + t = _MIDPOINTS_CACHE.get(device) + if t is None: + t = torch.tensor(MIDPOINTS_SORTED, device=device, dtype=torch.float32) + _MIDPOINTS_CACHE[device] = t + return t + + +def _get_sorted_to_bits_tensor(device: torch.device) -> torch.Tensor: + t = _SORTED_TO_BITS_CACHE.get(device) + if t is None: + t = torch.tensor(SORTED_TO_BITS, device=device, dtype=torch.int32) + _SORTED_TO_BITS_CACHE[device] = t + return t + + +def _get_hadamard( + dim: int, device: torch.device, dtype: torch.dtype = torch.float32 +) -> torch.Tensor: + """Sylvester Hadamard in fp32 (cached, normalised).""" + key = (dim, device, dtype) + cached = _PIT_CACHE.get(key) + if cached is not None: + return cached + if dim <= 0 or (dim & (dim - 1)) != 0: + raise ValueError(f"ultraquant store requires power-of-two dim, got {dim}") + H = torch.tensor([[1.0]], dtype=torch.float64) + while H.shape[0] < dim: + H = torch.cat([torch.cat([H, H], dim=1), torch.cat([H, -H], dim=1)], dim=0) + H = (H / (dim**0.5)).to(device=device, dtype=dtype).contiguous() + _PIT_CACHE[key] = H + return H + + +# ═══════════════════════════════════════════════════════════════════════════ +# Triton kernel +# ═══════════════════════════════════════════════════════════════════════════ + + +@triton.jit +def _ultraquant_store_kernel( + Key_ptr, # [N*H, D] in raw dtype (bf16 or fp16) + Value_ptr, # [N*H, D] in raw dtype + KV_cache_ptr, # raw uint8 view, flat + Slot_mapping_ptr, # [N] int64 + Midpoints_ptr, # [14] fp32 + SortedToBits_ptr, # [15] int32 + # Cache strides (in bytes) + stride_cache_block: tl.constexpr, + stride_cache_pos: tl.constexpr, + stride_cache_head: tl.constexpr, + # Dimensions + HEAD_DIM: tl.constexpr, + H: tl.constexpr, + BLOCK_SIZE: tl.constexpr, + BLOCK_D: tl.constexpr, + LOG2_D: tl.constexpr, + # Layout + GROUP_SIZE_C: tl.constexpr, + N_GROUPS_C: tl.constexpr, + K_SCALES_OFFSET: tl.constexpr, + V_CODES_OFFSET: tl.constexpr, + V_SCALES_OFFSET: tl.constexpr, + FP4_C: tl.constexpr, + UE8M0_BIAS_C: tl.constexpr, + # When 1, the FP4 midpoints and the sorted->bits remap are materialised as + # immediates instead of being read from `Midpoints_ptr`/`SortedToBits_ptr`. + # That removes 28 scalar global loads and two 256-lane divergent gathers + # from the critical path of a kernel whose runtime is pure latency (flat at + # ~22 us from batch 1 to batch 64). Numerically identical: the immediates + # are the same fp32 values, and the remap formula is exact (asserted in + # `format`, verified for all 15 entries). + CONST_TABLES: tl.constexpr = 1, +): + pid = tl.program_id(0) + token_idx = pid // H + head_idx = pid % H + + slot = tl.load(Slot_mapping_ptr + token_idx) + if slot < 0: + return + blk = slot // BLOCK_SIZE + off = slot % BLOCK_SIZE + slot_base = ( + blk * stride_cache_block + off * stride_cache_pos + head_idx * stride_cache_head + ).to(tl.int64) + k_code_base = slot_base + k_scale_base = slot_base + K_SCALES_OFFSET + v_code_base = slot_base + V_CODES_OFFSET + v_scale_base = slot_base + V_SCALES_OFFSET + + base = pid * HEAD_DIM + d_offs = tl.arange(0, BLOCK_D) + d_mask = d_offs < HEAD_DIM + shifts_4 = tl.arange(0, 2) * 4 # 0, 4 for nibble packing + g_offs = tl.arange(0, N_GROUPS_C) + + # ── K side ───────────────────────────────────────────────────────── + # 1. Load + Hadamard-rotate K via WHT butterfly. + k_raw = tl.load(Key_ptr + base + d_offs, mask=d_mask, other=0.0).to(tl.float32) + k_rot = tl.where(d_mask, k_raw, 0.0) + + INV_SQRT2: tl.constexpr = 0.7071067811865476 + for _stage in tl.static_range(LOG2_D): + _bmask = 1 << _stage + _pidx = (d_offs ^ _bmask).to(tl.int32) + _pval = tl.gather(k_rot, _pidx, 0) + _sign = tl.where((d_offs & _bmask) == 0, 1.0, -1.0).to(tl.float32) + k_rot = (_sign * k_rot + _pval) * INV_SQRT2 + k_rot = tl.where(d_mask, k_rot, 0.0) + + # 2. Per-group absmax. + k_g = tl.reshape(k_rot, [N_GROUPS_C, GROUP_SIZE_C]) + k_absmax = tl.max(tl.abs(k_g), axis=1) # [N_GROUPS] + k_is_zero = k_absmax == 0.0 + + # 3. UE8M0 snap: s_raw = c * absmax; exp = round(log2(s_raw)); + # divisor = 2^exp (c folded in). Zero groups write byte=0. + k_s_raw = k_absmax * tl.cast(FP4_C, tl.float32) + k_s_safe = tl.where(k_is_zero, 1.0, k_s_raw) + k_log2 = tl.log2(k_s_safe) + k_exp = tl.cast(tl.floor(k_log2 + 0.5), tl.int32) # round-half-up + k_s_snapped = tl.exp2(tl.cast(k_exp, tl.float32)) + k_s_div = tl.where(k_is_zero, 1.0, k_s_snapped) + + # 4. Normalize + snap to sorted FP4 idx via 14 sequential tl.where (`>` for + # bucketize(right=False) parity with the reference). + k_norm = k_g / k_s_div[:, None] + k_sorted = tl.zeros([N_GROUPS_C, GROUP_SIZE_C], dtype=tl.int32) + for i in tl.static_range(14): + mid = _MIDPOINTS_C.value[i] if CONST_TABLES else tl.load(Midpoints_ptr + i) + k_sorted += tl.where(k_norm > mid, 1, 0) + k_sorted = tl.where(k_is_zero[:, None], 7, k_sorted) + + # 5. Remap sorted idx → FP4 E2M1 bit pattern. + if CONST_TABLES: + k_bits = tl.where(k_sorted < 7, 15 - k_sorted, k_sorted - 7) + else: + k_bits = tl.load(SortedToBits_ptr + k_sorted) + + # 6. Pack pairs of 4-bit codes into uint8 bytes. + k_bits_flat = tl.reshape(k_bits, [HEAD_DIM]) + k_pairs = tl.reshape(k_bits_flat, [HEAD_DIM // 2, 2]) + k_packed = tl.sum((k_pairs & 0xF) << shifts_4[None, :], axis=1).to(tl.uint8) + k_code_addrs = k_code_base + tl.arange(0, HEAD_DIM // 2) + tl.store(KV_cache_ptr + k_code_addrs, k_packed) + + # 7. Store K scales as E8M0 bytes (one per group). Zero-amax groups + # write byte=0 (zero sentinel). + k_byte = tl.where(k_is_zero, 0, k_exp + UE8M0_BIAS_C) + k_byte = tl.where((k_byte < 0) | (k_byte > 255), 0, k_byte).to(tl.uint8) + k_scale_addrs = k_scale_base + g_offs + tl.store(KV_cache_ptr + k_scale_addrs, k_byte) + + # ── V side (mirror K but skip rotation) ──────────────────────────── + v_raw = tl.load(Value_ptr + base + d_offs, mask=d_mask, other=0.0).to(tl.float32) + v_raw = tl.where(d_mask, v_raw, 0.0) + + v_g = tl.reshape(v_raw, [N_GROUPS_C, GROUP_SIZE_C]) + v_absmax = tl.max(tl.abs(v_g), axis=1) + v_is_zero = v_absmax == 0.0 + + v_s_raw = v_absmax * tl.cast(FP4_C, tl.float32) + v_s_safe = tl.where(v_is_zero, 1.0, v_s_raw) + v_log2 = tl.log2(v_s_safe) + v_exp = tl.cast(tl.floor(v_log2 + 0.5), tl.int32) + v_s_snapped = tl.exp2(tl.cast(v_exp, tl.float32)) + v_s_div = tl.where(v_is_zero, 1.0, v_s_snapped) + + v_norm = v_g / v_s_div[:, None] + v_sorted = tl.zeros([N_GROUPS_C, GROUP_SIZE_C], dtype=tl.int32) + for i in tl.static_range(14): + mid = _MIDPOINTS_C.value[i] if CONST_TABLES else tl.load(Midpoints_ptr + i) + v_sorted += tl.where(v_norm > mid, 1, 0) + v_sorted = tl.where(v_is_zero[:, None], 7, v_sorted) + + if CONST_TABLES: + v_bits = tl.where(v_sorted < 7, 15 - v_sorted, v_sorted - 7) + else: + v_bits = tl.load(SortedToBits_ptr + v_sorted) + v_bits_flat = tl.reshape(v_bits, [HEAD_DIM]) + v_pairs = tl.reshape(v_bits_flat, [HEAD_DIM // 2, 2]) + v_packed = tl.sum((v_pairs & 0xF) << shifts_4[None, :], axis=1).to(tl.uint8) + v_code_addrs = v_code_base + tl.arange(0, HEAD_DIM // 2) + tl.store(KV_cache_ptr + v_code_addrs, v_packed) + + v_byte = tl.where(v_is_zero, 0, v_exp + UE8M0_BIAS_C) + v_byte = tl.where((v_byte < 0) | (v_byte > 255), 0, v_byte).to(tl.uint8) + v_scale_addrs = v_scale_base + g_offs + tl.store(KV_cache_ptr + v_scale_addrs, v_byte) + + +# ═══════════════════════════════════════════════════════════════════════════ +# Launcher +# ═══════════════════════════════════════════════════════════════════════════ + + +def ultraquant_store( + key: torch.Tensor, # [N, H, D] bf16 or fp16 + value: torch.Tensor, # [N, H, D] bf16 or fp16 + kv_cache: torch.Tensor, # [num_blocks, block_size, Hk, slot_size_aligned] uint8 + slot_mapping: torch.Tensor, # [N] int64 + *, + PiT: torch.Tensor | None = None, # unused; kept for call-site compatibility + constant_c: float | None = None, +) -> None: + """Launch the ultraquant store kernel. Writes K/V into `kv_cache` + in-place at the slots specified by `slot_mapping`.""" + if key.dtype not in _SUPPORTED_DTYPES: + raise ValueError( + f"ultraquant_store: key.dtype must be one of " + f"{_SUPPORTED_DTYPES}, got {key.dtype}" + ) + if value.dtype != key.dtype: + raise ValueError( + f"ultraquant_store: key/value dtype mismatch ({key.dtype} vs {value.dtype})" + ) + if slot_mapping.dtype != torch.int64: + slot_mapping = slot_mapping.to(torch.int64) + + N, H, D = key.shape + NH = N * H + block_size = kv_cache.shape[1] + num_kv_heads = kv_cache.shape[2] + padded_slot = kv_cache.shape[3] + + expected_slot = slot_size(D) + if padded_slot < expected_slot: + raise ValueError( + f"ultraquant_store: kv_cache slot {padded_slot} < expected " + f"{expected_slot} for head_dim={D}" + ) + if num_kv_heads != H: + raise ValueError( + f"ultraquant_store: kv_cache num_kv_heads {num_kv_heads} != key heads {H}" + ) + + midpoints = _get_midpoints_tensor(key.device) + sorted_to_bits = _get_sorted_to_bits_tensor(key.device) + + k_flat = key.reshape(NH, D).contiguous() + v_flat = value.reshape(NH, D).contiguous() + + gs = get_group_size() + BLOCK_D = triton.next_power_of_2(D) + N_GROUPS_C = n_groups(D, gs) + K_SCALES_OFFSET = k_scales_offset(D, gs) + V_CODES_OFFSET = v_codes_offset(D, gs) + V_SCALES_OFFSET = v_scales_offset(D, gs) + + stride_block = kv_cache.stride(0) + stride_pos = kv_cache.stride(1) + stride_head = kv_cache.stride(2) + + c = constant_c if constant_c is not None else get_constant_c() + + _ensure_const_tables() + grid = (NH,) + _ultraquant_store_kernel[grid]( + k_flat, + v_flat, + _kv_cache_flat(kv_cache), + slot_mapping, + midpoints, + sorted_to_bits, + stride_cache_block=stride_block, + stride_cache_pos=stride_pos, + stride_cache_head=stride_head, + HEAD_DIM=D, + H=H, + BLOCK_SIZE=block_size, + BLOCK_D=BLOCK_D, + LOG2_D=int(D).bit_length() - 1, + GROUP_SIZE_C=gs, + N_GROUPS_C=N_GROUPS_C, + K_SCALES_OFFSET=K_SCALES_OFFSET, + V_CODES_OFFSET=V_CODES_OFFSET, + V_SCALES_OFFSET=V_SCALES_OFFSET, + FP4_C=c, + UE8M0_BIAS_C=UE8M0_BIAS, + CONST_TABLES=1, + num_warps=4, + num_stages=1, + ) diff --git a/vllm/v1/attention/ops/ultraquant/triton_unified_attention.py b/vllm/v1/attention/ops/ultraquant/triton_unified_attention.py new file mode 100644 index 0000000000..69ec744a07 --- /dev/null +++ b/vllm/v1/attention/ops/ultraquant/triton_unified_attention.py @@ -0,0 +1,1087 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Unified Triton fallback for the UltraQuant 4-bit KV-cache format. + +K and V use packed FP4 E2M1 values with UE8M0 group-of-32 scales. QK uses +native scaled FP4×E4M3 MFMA on CDNA4. Production choices are fixed in code; +there are no environment-variable tuning switches. +""" + +from __future__ import annotations + +import torch + +from vllm.platforms import current_platform +from vllm.triton_utils import tl, triton +from vllm.v1.attention.ops.triton_unified_attention import ( + reduce_segments, +) + +# Reuse the sequence-index search and vectorized partition reducer. +from vllm.v1.attention.ops.turboquant_soa.triton_turboquant_unified_attention import ( + _find_seq_idx, +) +from vllm.v1.attention.ops.ultraquant.format import ( + UE8M0_BIAS, + get_group_size, + k_scales_offset, + n_groups, + slot_size, + v_codes_offset, + v_scales_offset, +) +from vllm.v1.attention.ops.ultraquant.triton_store import _kv_cache_flat + +_is_hip = current_platform.is_rocm() + + +@triton.jit +def _ultraquant_fp4_decode_arith(codes): + """Arithmetic FP4 E2M1 decode, bit-exact to format.FP4_BITS_TO_VALUE.""" + mag = codes & 7 + e = mag >> 1 + m = mag & 1 + exp_field = 126 + e + mant_field = tl.where(e != 0, m, 0) << 22 + bits = (exp_field << 23) | mant_field + magval = bits.to(tl.float32, bitcast=True) + magval = tl.where(mag == 0, 0.0, magval) + signf = tl.where((codes & 8) != 0, -1.0, 1.0) + return magval * signf + + +@triton.jit +def _ultraquant_load_k_packed( + KV_cache_ptr, + data_bases, + k_scales_addrs, + d_half_offs, + half_mask, + tile_mask, + BLOCK_D: tl.constexpr, + HEAD_DIM: tl.constexpr, + GROUP_SIZE_C: tl.constexpr, + N_GROUPS_C: tl.constexpr, + TILE_SIZE: tl.constexpr, + UNMASKED: tl.constexpr, +): + """Load packed FP4 K codes and UE8M0 scales for scaled QK MFMA.""" + addrs = data_bases[:, None] + d_half_offs[None, :] + if UNMASKED: + K_codes = tl.load(KV_cache_ptr + addrs, mask=half_mask[None, :], other=0) + else: + K_codes = tl.load( + KV_cache_ptr + addrs, + mask=tile_mask[:, None] & half_mask[None, :], + other=0, + ) + K_T_packed = tl.trans(K_codes) + grp = tl.arange(0, N_GROUPS_C) + scale_addrs = k_scales_addrs[:, None] + grp[None, :] + if UNMASKED: + K_scales = tl.load(KV_cache_ptr + scale_addrs) + else: + K_scales = tl.load(KV_cache_ptr + scale_addrs, mask=tile_mask[:, None], other=0) + _ = HEAD_DIM + _ = GROUP_SIZE_C + _ = BLOCK_D + return K_T_packed, K_scales + + +@triton.jit +def _ultraquant_load_v_tile( + KV_cache_ptr, + val_bases, + v_scales_addrs, + d_offs, + d_mask, + tile_mask, + OUT_DTYPE: tl.constexpr, + HEAD_DIM: tl.constexpr, + BLOCK_D: tl.constexpr, + GROUP_SIZE_C: tl.constexpr, + N_GROUPS_C: tl.constexpr, + TILE_SIZE: tl.constexpr, + UNMASKED: tl.constexpr, + UE8M0_BIAS_C: tl.constexpr, +): + """Load and dequant a V tile with arithmetic FP4 decode.""" + half_idx = d_offs // 2 + nibble_shift = (d_offs % 2) * 4 + addrs = val_bases[:, None] + half_idx[None, :] + if UNMASKED: + byte_raw = tl.load(KV_cache_ptr + addrs, mask=d_mask[None, :], other=0).to( + tl.int32 + ) + else: + byte_raw = tl.load( + KV_cache_ptr + addrs, + mask=tile_mask[:, None] & d_mask[None, :], + other=0, + ).to(tl.int32) + codes = (byte_raw >> nibble_shift[None, :]) & 0xF + grp = tl.arange(0, N_GROUPS_C) + scale_addrs = v_scales_addrs[:, None] + grp[None, :] + if UNMASKED: + scale_bytes = tl.load(KV_cache_ptr + scale_addrs).to(tl.int32) + else: + scale_bytes = tl.load( + KV_cache_ptr + scale_addrs, mask=tile_mask[:, None], other=0 + ).to(tl.int32) + scale_exp = scale_bytes - UE8M0_BIAS_C + fp4_vals = _ultraquant_fp4_decode_arith(codes) + scales = tl.where(scale_bytes == 0, 0.0, tl.exp2(tl.cast(scale_exp, tl.float32))) + V_g = tl.reshape(fp4_vals, [TILE_SIZE, N_GROUPS_C, GROUP_SIZE_C]) + return tl.reshape(V_g * scales[:, :, None], [TILE_SIZE, BLOCK_D]).to(OUT_DTYPE) + + +@triton.jit +def _ultraquant_pv( + P, + KV_cache_ptr, + val_bases, + v_scales_addrs, + d_offs, + d_mask, + tile_mask, + HEAD_DIM: tl.constexpr, + BLOCK_D: tl.constexpr, + BLOCK_M: tl.constexpr, + GROUP_SIZE_C: tl.constexpr, + N_GROUPS_C: tl.constexpr, + TILE_SIZE: tl.constexpr, + UNMASKED: tl.constexpr, + UE8M0_BIAS_C: tl.constexpr, +): + V = _ultraquant_load_v_tile( + KV_cache_ptr, + val_bases, + v_scales_addrs, + d_offs, + d_mask, + tile_mask, + OUT_DTYPE=tl.bfloat16, + HEAD_DIM=HEAD_DIM, + BLOCK_D=BLOCK_D, + GROUP_SIZE_C=GROUP_SIZE_C, + N_GROUPS_C=N_GROUPS_C, + TILE_SIZE=TILE_SIZE, + UNMASKED=UNMASKED, + UE8M0_BIAS_C=UE8M0_BIAS_C, + ) + _ = BLOCK_M + return tl.dot(P.to(tl.bfloat16), V, out_dtype=tl.float32) + + +@triton.jit +def _ultraquant_qk( + Q, + KV_cache_ptr, + data_bases, + k_scales_addrs, + offs_d_half, + half_mask, + tile_mask, + BLOCK_D: tl.constexpr, + HEAD_DIM: tl.constexpr, + GROUP_SIZE_C: tl.constexpr, + N_GROUPS_C: tl.constexpr, + TILE_SIZE: tl.constexpr, + UNMASKED: tl.constexpr, +): + K_T_packed, K_scales = _ultraquant_load_k_packed( + KV_cache_ptr, + data_bases, + k_scales_addrs, + offs_d_half, + half_mask, + tile_mask, + BLOCK_D=BLOCK_D, + HEAD_DIM=HEAD_DIM, + GROUP_SIZE_C=GROUP_SIZE_C, + N_GROUPS_C=N_GROUPS_C, + TILE_SIZE=TILE_SIZE, + UNMASKED=UNMASKED, + ) + return tl.dot_scaled( + Q, + None, + "e4m3", + K_T_packed, + K_scales, + "e2m1", + out_dtype=tl.float32, + ) + + +@triton.jit +def _ultraquant_slot_addrs( + block_base, + slot_within_block, + kv_head_idx, + stride_cache_pos: tl.int64, + stride_cache_head: tl.int64, + K_SCALES_OFFSET: tl.constexpr, + V_CODES_OFFSET: tl.constexpr, + V_SCALES_OFFSET: tl.constexpr, +): + data_bases = ( + block_base + + slot_within_block * stride_cache_pos + + tl.cast(kv_head_idx, tl.int64) * stride_cache_head + ) + return ( + data_bases, + data_bases + K_SCALES_OFFSET, + data_bases + V_CODES_OFFSET, + data_bases + V_SCALES_OFFSET, + ) + + +# =========================================================================== +# Unified 2D attention kernel (prefill / short-context decode) +# =========================================================================== + + +@triton.jit +def kernel_ultraquant_unified_attention_2d( + output_ptr, + query_ptr, # FP8 E4M3, already Hadamard-rotated + KV_cache_ptr, # uint8 view + block_tables_ptr, + seq_lens_ptr, + query_start_len_ptr, + sinks_ptr, + scale, + num_query_heads: tl.constexpr, + num_queries_per_kv: tl.constexpr, + block_table_stride: tl.int64, + query_stride_0: tl.int64, + query_stride_1: tl.int64, + output_stride_0: tl.int64, + output_stride_1: tl.int64, + stride_cache_block: tl.int64, + stride_cache_pos: tl.int64, + stride_cache_head: tl.int64, + BLOCK_SIZE: tl.constexpr, + TILE_SIZE: tl.constexpr, + HEAD_SIZE: tl.constexpr, + HEAD_SIZE_PADDED: tl.constexpr, + BLOCK_Q: tl.constexpr, + BLOCK_M: tl.constexpr, + num_seqs: tl.int32, + K_SCALES_OFFSET: tl.constexpr, + V_CODES_OFFSET: tl.constexpr, + V_SCALES_OFFSET: tl.constexpr, + GROUP_SIZE_C: tl.constexpr, + N_GROUPS_C: tl.constexpr, + UE8M0_BIAS_C: tl.constexpr, + USE_SINKS: tl.constexpr = 0, + SLIDING_WINDOW: tl.constexpr = 0, +): + q_block_global_idx = tl.program_id(0) + kv_head_idx = tl.program_id(1) + + seq_idx = _find_seq_idx( + query_start_len_ptr, q_block_global_idx, num_seqs, BLOCK_Q, True + ) + q_block_start_idx = tl.load(query_start_len_ptr + seq_idx) // BLOCK_Q + seq_idx + q_block_local_idx = q_block_global_idx - q_block_start_idx + + cur_batch_in_all_start_index = tl.load(query_start_len_ptr + seq_idx) + cur_batch_in_all_stop_index = tl.load(query_start_len_ptr + seq_idx + 1) + cur_batch_query_len = cur_batch_in_all_stop_index - cur_batch_in_all_start_index + + if q_block_local_idx * BLOCK_Q >= cur_batch_query_len: + return + + offs_m = tl.arange(0, BLOCK_M) + offs_d = tl.arange(0, HEAD_SIZE_PADDED) + offs_d_half = tl.arange(0, HEAD_SIZE_PADDED // 2) + offs_t = tl.arange(0, TILE_SIZE) + query_pos = q_block_local_idx * BLOCK_Q + offs_m // num_queries_per_kv + + query_offset_0 = cur_batch_in_all_start_index + query_pos + query_offset_1 = kv_head_idx * num_queries_per_kv + offs_m % num_queries_per_kv + query_offset = ( + query_offset_0[:, None] * query_stride_0 + + query_offset_1[:, None] * query_stride_1 + + offs_d[None, :] + ) + + dim_mask = tl.where(offs_d < HEAD_SIZE, 1, 0).to(tl.int1) + half_mask = tl.where(offs_d_half * 2 < HEAD_SIZE, 1, 0).to(tl.int1) + query_mask_0 = tl.where(query_pos < cur_batch_query_len, 1, 0).to(tl.int1) + query_mask_1 = tl.where(query_offset_1 < num_query_heads, 1, 0).to(tl.int1) + + Q = tl.load( + query_ptr + query_offset, + mask=dim_mask[None, :] & query_mask_0[:, None] & query_mask_1[:, None], + other=tl.zeros([], tl.float8e4nv), + ) + + block_table_offset = seq_idx * block_table_stride + + if USE_SINKS: + M = tl.load( + sinks_ptr + query_offset_1, + mask=query_mask_1, + other=float("-inf"), + ).to(tl.float32) + else: + M = tl.full([BLOCK_M], float("-inf"), dtype=tl.float32) + L = tl.full([BLOCK_M], 1.0, dtype=tl.float32) + acc = tl.zeros([BLOCK_M, HEAD_SIZE_PADDED], dtype=tl.float32) + + seq_len = tl.load(seq_lens_ptr + seq_idx) + context_len = seq_len - cur_batch_query_len + + max_seq_prefix_len = ( + context_len + + q_block_local_idx * BLOCK_Q + + (BLOCK_M - 1) // num_queries_per_kv + + 1 + ) + max_seq_prefix_len = tl.minimum(max_seq_prefix_len, seq_len) + num_tiles = tl.cdiv(max_seq_prefix_len, TILE_SIZE) + + tile_start = 0 + tile_end = num_tiles + if SLIDING_WINDOW > 0: + qpos_lo = q_block_local_idx * BLOCK_Q + qpos_hi = tl.minimum( + qpos_lo + (BLOCK_M - 1) // num_queries_per_kv, + cur_batch_query_len - 1, + ) + first_allowed_key = context_len + qpos_lo - SLIDING_WINDOW + 1 + last_allowed_key = context_len + qpos_hi + tile_start = tl.maximum(0, first_allowed_key // TILE_SIZE) + tile_end = tl.minimum((last_allowed_key // TILE_SIZE) + 1, num_tiles) + + query_abs_pos = context_len + query_pos[:, None] + dummy_tile_mask = tl.full([TILE_SIZE], 1, tl.int1) + + for j in range(tile_start, tile_end - 1): + seq_offset = j * TILE_SIZE + offs_t + if TILE_SIZE == BLOCK_SIZE: + physical_block_idx = tl.load(block_tables_ptr + block_table_offset + j).to( + tl.int64 + ) + slot_within_block = offs_t.to(tl.int64) + else: + physical_block_idx = tl.load( + block_tables_ptr + block_table_offset + seq_offset // BLOCK_SIZE + ).to(tl.int64) + slot_within_block = (seq_offset % BLOCK_SIZE).to(tl.int64) + block_base = physical_block_idx * stride_cache_block + data_bases, k_scales_addrs, val_bases, v_scales_addrs = _ultraquant_slot_addrs( + block_base, + slot_within_block, + kv_head_idx, + stride_cache_pos, + stride_cache_head, + K_SCALES_OFFSET=K_SCALES_OFFSET, + V_CODES_OFFSET=V_CODES_OFFSET, + V_SCALES_OFFSET=V_SCALES_OFFSET, + ) + + S = scale * _ultraquant_qk( + Q, + KV_cache_ptr, + data_bases, + k_scales_addrs, + offs_d_half, + half_mask, + dummy_tile_mask, + BLOCK_D=HEAD_SIZE_PADDED, + HEAD_DIM=HEAD_SIZE, + GROUP_SIZE_C=GROUP_SIZE_C, + N_GROUPS_C=N_GROUPS_C, + TILE_SIZE=TILE_SIZE, + UNMASKED=True, + ) + seq_mask = seq_offset[None, :] <= query_abs_pos + if SLIDING_WINDOW > 0: + seq_mask = seq_mask & ( + (query_abs_pos - seq_offset[None, :]) < SLIDING_WINDOW + ) + S = tl.where( + query_mask_1[:, None] & query_mask_0[:, None] & seq_mask, + S, + float("-inf"), + ) + m_j = tl.maximum(M, tl.max(S, axis=1)) + m_j = tl.where(m_j > float("-inf"), m_j, 0.0) + P = tl.exp(S - m_j[:, None]) + l_j = tl.sum(P, axis=1) + alpha = tl.exp(M - m_j) + acc = acc * alpha[:, None] + L = L * alpha + l_j + M = m_j + acc += _ultraquant_pv( + P, + KV_cache_ptr, + val_bases, + v_scales_addrs, + offs_d, + dim_mask, + dummy_tile_mask, + HEAD_DIM=HEAD_SIZE, + BLOCK_D=HEAD_SIZE_PADDED, + BLOCK_M=BLOCK_M, + GROUP_SIZE_C=GROUP_SIZE_C, + N_GROUPS_C=N_GROUPS_C, + TILE_SIZE=TILE_SIZE, + UNMASKED=True, + UE8M0_BIAS_C=UE8M0_BIAS_C, + ) + + # Tail tile (masked) + if tile_end > tile_start: + j = tile_end - 1 + seq_offset = j * TILE_SIZE + offs_t + tile_mask = seq_offset < max_seq_prefix_len + if TILE_SIZE == BLOCK_SIZE: + physical_block_idx = tl.load(block_tables_ptr + block_table_offset + j).to( + tl.int64 + ) + slot_within_block = offs_t.to(tl.int64) + else: + physical_block_idx = tl.load( + block_tables_ptr + block_table_offset + seq_offset // BLOCK_SIZE + ).to(tl.int64) + slot_within_block = (seq_offset % BLOCK_SIZE).to(tl.int64) + block_base = physical_block_idx * stride_cache_block + data_bases, k_scales_addrs, val_bases, v_scales_addrs = _ultraquant_slot_addrs( + block_base, + slot_within_block, + kv_head_idx, + stride_cache_pos, + stride_cache_head, + K_SCALES_OFFSET=K_SCALES_OFFSET, + V_CODES_OFFSET=V_CODES_OFFSET, + V_SCALES_OFFSET=V_SCALES_OFFSET, + ) + + S = scale * _ultraquant_qk( + Q, + KV_cache_ptr, + data_bases, + k_scales_addrs, + offs_d_half, + half_mask, + tile_mask, + BLOCK_D=HEAD_SIZE_PADDED, + HEAD_DIM=HEAD_SIZE, + GROUP_SIZE_C=GROUP_SIZE_C, + N_GROUPS_C=N_GROUPS_C, + TILE_SIZE=TILE_SIZE, + UNMASKED=False, + ) + seq_mask = seq_offset[None, :] <= query_abs_pos + if SLIDING_WINDOW > 0: + seq_mask = seq_mask & ( + (query_abs_pos - seq_offset[None, :]) < SLIDING_WINDOW + ) + S = tl.where( + query_mask_1[:, None] & query_mask_0[:, None] & seq_mask, + S, + float("-inf"), + ) + m_j = tl.maximum(M, tl.max(S, axis=1)) + m_j = tl.where(m_j > float("-inf"), m_j, 0.0) + P = tl.exp(S - m_j[:, None]) + l_j = tl.sum(P, axis=1) + alpha = tl.exp(M - m_j) + acc = acc * alpha[:, None] + L = L * alpha + l_j + M = m_j + acc += _ultraquant_pv( + P, + KV_cache_ptr, + val_bases, + v_scales_addrs, + offs_d, + dim_mask, + tile_mask, + HEAD_DIM=HEAD_SIZE, + BLOCK_D=HEAD_SIZE_PADDED, + BLOCK_M=BLOCK_M, + GROUP_SIZE_C=GROUP_SIZE_C, + N_GROUPS_C=N_GROUPS_C, + TILE_SIZE=TILE_SIZE, + UNMASKED=False, + UE8M0_BIAS_C=UE8M0_BIAS_C, + ) + + acc = acc / L[:, None] + + output_offset = ( + query_offset_0[:, None] * output_stride_0 + + query_offset_1[:, None] * output_stride_1 + + offs_d[None, :] + ) + tl.store( + output_ptr + output_offset, + acc, + mask=dim_mask[None, :] & query_mask_0[:, None] & query_mask_1[:, None], + ) + + +# =========================================================================== +# Unified 3D (split-KV) attention kernel +# =========================================================================== + + +@triton.jit +def kernel_ultraquant_unified_attention_3d( + segm_output_ptr, + segm_max_ptr, + segm_expsum_ptr, + query_ptr, # FP8 E4M3, already Hadamard-rotated + KV_cache_ptr, + block_tables_ptr, + seq_lens_ptr, + query_start_len_ptr, + sinks_ptr, + scale, + num_query_heads: tl.constexpr, + num_queries_per_kv: tl.constexpr, + block_table_stride: tl.int64, + query_stride_0: tl.int64, + query_stride_1: tl.int64, + stride_cache_block: tl.int64, + stride_cache_pos: tl.int64, + stride_cache_head: tl.int64, + BLOCK_SIZE: tl.constexpr, + TILE_SIZE: tl.constexpr, + HEAD_SIZE: tl.constexpr, + HEAD_SIZE_PADDED: tl.constexpr, + BLOCK_Q: tl.constexpr, + BLOCK_M: tl.constexpr, + num_seqs: tl.int32, + NUM_SEGMENTS_PER_SEQ: tl.constexpr, + K_SCALES_OFFSET: tl.constexpr, + V_CODES_OFFSET: tl.constexpr, + V_SCALES_OFFSET: tl.constexpr, + GROUP_SIZE_C: tl.constexpr, + N_GROUPS_C: tl.constexpr, + UE8M0_BIAS_C: tl.constexpr, + USE_SINKS: tl.constexpr = 0, + SLIDING_WINDOW: tl.constexpr = 0, +): + q_block_global_idx = tl.program_id(0) + kv_head_idx = tl.program_id(1) + segm_idx = tl.program_id(2) + + seq_idx = _find_seq_idx( + query_start_len_ptr, q_block_global_idx, num_seqs, BLOCK_Q, True + ) + q_block_start_idx = tl.load(query_start_len_ptr + seq_idx) // BLOCK_Q + seq_idx + q_block_local_idx = q_block_global_idx - q_block_start_idx + + cur_batch_in_all_start_index = tl.load(query_start_len_ptr + seq_idx) + cur_batch_in_all_stop_index = tl.load(query_start_len_ptr + seq_idx + 1) + cur_batch_query_len = cur_batch_in_all_stop_index - cur_batch_in_all_start_index + + if q_block_local_idx * BLOCK_Q >= cur_batch_query_len: + return + + seq_len = tl.load(seq_lens_ptr + seq_idx) + tiles_per_segment = tl.cdiv(seq_len, NUM_SEGMENTS_PER_SEQ * TILE_SIZE) + if segm_idx * tiles_per_segment * TILE_SIZE >= seq_len: + return + + offs_m = tl.arange(0, BLOCK_M) + offs_d = tl.arange(0, HEAD_SIZE_PADDED) + offs_d_half = tl.arange(0, HEAD_SIZE_PADDED // 2) + offs_t = tl.arange(0, TILE_SIZE) + query_pos = q_block_local_idx * BLOCK_Q + offs_m // num_queries_per_kv + + query_offset_0 = cur_batch_in_all_start_index + query_pos + query_offset_1 = kv_head_idx * num_queries_per_kv + offs_m % num_queries_per_kv + query_offset = ( + query_offset_0[:, None] * query_stride_0 + + query_offset_1[:, None] * query_stride_1 + + offs_d[None, :] + ) + + dim_mask = tl.where(offs_d < HEAD_SIZE, 1, 0).to(tl.int1) + half_mask = tl.where(offs_d_half * 2 < HEAD_SIZE, 1, 0).to(tl.int1) + query_mask_0 = tl.where(query_pos < cur_batch_query_len, 1, 0).to(tl.int1) + query_mask_1 = tl.where(query_offset_1 < num_query_heads, 1, 0).to(tl.int1) + + Q = tl.load( + query_ptr + query_offset, + mask=dim_mask[None, :] & query_mask_0[:, None] & query_mask_1[:, None], + other=tl.zeros([], tl.float8e4nv), + ) + + block_table_offset = seq_idx * block_table_stride + + if USE_SINKS and segm_idx == 0: + M = tl.load( + sinks_ptr + query_offset_1, + mask=query_mask_1, + other=float("-inf"), + ).to(tl.float32) + else: + M = tl.full([BLOCK_M], float("-inf"), dtype=tl.float32) + L = tl.full([BLOCK_M], 1.0, dtype=tl.float32) + acc = tl.zeros([BLOCK_M, HEAD_SIZE_PADDED], dtype=tl.float32) + + context_len = seq_len - cur_batch_query_len + max_seq_prefix_len = ( + context_len + + q_block_local_idx * BLOCK_Q + + (BLOCK_M - 1) // num_queries_per_kv + + 1 + ) + max_seq_prefix_len = tl.minimum(max_seq_prefix_len, seq_len) + num_tiles = tl.cdiv(max_seq_prefix_len, TILE_SIZE) + + tile_lo = segm_idx * tiles_per_segment + tile_hi = tl.minimum((segm_idx + 1) * tiles_per_segment, num_tiles) + + if SLIDING_WINDOW > 0: + qpos_lo = q_block_local_idx * BLOCK_Q + qpos_hi = tl.minimum( + qpos_lo + (BLOCK_M - 1) // num_queries_per_kv, + cur_batch_query_len - 1, + ) + first_allowed_key = context_len + qpos_lo - SLIDING_WINDOW + 1 + last_allowed_key = context_len + qpos_hi + swa_tile_start = tl.maximum(0, first_allowed_key // TILE_SIZE) + swa_tile_end = tl.minimum((last_allowed_key // TILE_SIZE) + 1, num_tiles) + tile_lo = tl.maximum(tile_lo, swa_tile_start) + tile_hi = tl.minimum(tile_hi, swa_tile_end) + + query_abs_pos = context_len + query_pos[:, None] + dummy_tile_mask = tl.full([TILE_SIZE], 1, tl.int1) + tail = tile_hi - 1 + + for j in range(tile_lo, tile_hi - 1): + seq_offset = j * TILE_SIZE + offs_t + if TILE_SIZE == BLOCK_SIZE: + physical_block_idx = tl.load(block_tables_ptr + block_table_offset + j).to( + tl.int64 + ) + slot_within_block = offs_t.to(tl.int64) + else: + physical_block_idx = tl.load( + block_tables_ptr + block_table_offset + seq_offset // BLOCK_SIZE + ).to(tl.int64) + slot_within_block = (seq_offset % BLOCK_SIZE).to(tl.int64) + block_base = physical_block_idx * stride_cache_block + data_bases, k_scales_addrs, val_bases, v_scales_addrs = _ultraquant_slot_addrs( + block_base, + slot_within_block, + kv_head_idx, + stride_cache_pos, + stride_cache_head, + K_SCALES_OFFSET=K_SCALES_OFFSET, + V_CODES_OFFSET=V_CODES_OFFSET, + V_SCALES_OFFSET=V_SCALES_OFFSET, + ) + + S = scale * _ultraquant_qk( + Q, + KV_cache_ptr, + data_bases, + k_scales_addrs, + offs_d_half, + half_mask, + dummy_tile_mask, + BLOCK_D=HEAD_SIZE_PADDED, + HEAD_DIM=HEAD_SIZE, + GROUP_SIZE_C=GROUP_SIZE_C, + N_GROUPS_C=N_GROUPS_C, + TILE_SIZE=TILE_SIZE, + UNMASKED=True, + ) + seq_mask = seq_offset[None, :] <= query_abs_pos + if SLIDING_WINDOW > 0: + seq_mask = seq_mask & ( + (query_abs_pos - seq_offset[None, :]) < SLIDING_WINDOW + ) + S = tl.where( + query_mask_1[:, None] & query_mask_0[:, None] & seq_mask, + S, + float("-inf"), + ) + m_j = tl.maximum(M, tl.max(S, axis=1)) + m_j = tl.where(m_j > float("-inf"), m_j, 0.0) + P = tl.exp(S - m_j[:, None]) + l_j = tl.sum(P, axis=1) + alpha = tl.exp(M - m_j) + acc = acc * alpha[:, None] + L = L * alpha + l_j + M = m_j + acc += _ultraquant_pv( + P, + KV_cache_ptr, + val_bases, + v_scales_addrs, + offs_d, + dim_mask, + dummy_tile_mask, + HEAD_DIM=HEAD_SIZE, + BLOCK_D=HEAD_SIZE_PADDED, + BLOCK_M=BLOCK_M, + GROUP_SIZE_C=GROUP_SIZE_C, + N_GROUPS_C=N_GROUPS_C, + TILE_SIZE=TILE_SIZE, + UNMASKED=True, + UE8M0_BIAS_C=UE8M0_BIAS_C, + ) + + # Tail + if tile_lo < tile_hi: + j = tail + seq_offset = j * TILE_SIZE + offs_t + tile_mask = seq_offset < max_seq_prefix_len + if TILE_SIZE == BLOCK_SIZE: + physical_block_idx = tl.load(block_tables_ptr + block_table_offset + j).to( + tl.int64 + ) + slot_within_block = offs_t.to(tl.int64) + else: + physical_block_idx = tl.load( + block_tables_ptr + block_table_offset + seq_offset // BLOCK_SIZE + ).to(tl.int64) + slot_within_block = (seq_offset % BLOCK_SIZE).to(tl.int64) + block_base = physical_block_idx * stride_cache_block + data_bases, k_scales_addrs, val_bases, v_scales_addrs = _ultraquant_slot_addrs( + block_base, + slot_within_block, + kv_head_idx, + stride_cache_pos, + stride_cache_head, + K_SCALES_OFFSET=K_SCALES_OFFSET, + V_CODES_OFFSET=V_CODES_OFFSET, + V_SCALES_OFFSET=V_SCALES_OFFSET, + ) + + S = scale * _ultraquant_qk( + Q, + KV_cache_ptr, + data_bases, + k_scales_addrs, + offs_d_half, + half_mask, + tile_mask, + BLOCK_D=HEAD_SIZE_PADDED, + HEAD_DIM=HEAD_SIZE, + GROUP_SIZE_C=GROUP_SIZE_C, + N_GROUPS_C=N_GROUPS_C, + TILE_SIZE=TILE_SIZE, + UNMASKED=False, + ) + seq_mask = seq_offset[None, :] <= query_abs_pos + if SLIDING_WINDOW > 0: + seq_mask = seq_mask & ( + (query_abs_pos - seq_offset[None, :]) < SLIDING_WINDOW + ) + S = tl.where( + query_mask_1[:, None] & query_mask_0[:, None] & seq_mask, + S, + float("-inf"), + ) + m_j = tl.maximum(M, tl.max(S, axis=1)) + m_j = tl.where(m_j > float("-inf"), m_j, 0.0) + P = tl.exp(S - m_j[:, None]) + l_j = tl.sum(P, axis=1) + alpha = tl.exp(M - m_j) + acc = acc * alpha[:, None] + L = L * alpha + l_j + M = m_j + acc += _ultraquant_pv( + P, + KV_cache_ptr, + val_bases, + v_scales_addrs, + offs_d, + dim_mask, + tile_mask, + HEAD_DIM=HEAD_SIZE, + BLOCK_D=HEAD_SIZE_PADDED, + BLOCK_M=BLOCK_M, + GROUP_SIZE_C=GROUP_SIZE_C, + N_GROUPS_C=N_GROUPS_C, + TILE_SIZE=TILE_SIZE, + UNMASKED=False, + UE8M0_BIAS_C=UE8M0_BIAS_C, + ) + + # Write segment partials for stage-2 reduce. + segm_output_offset = ( + query_offset_0[:, None].to(tl.int64) + * (num_query_heads * NUM_SEGMENTS_PER_SEQ * HEAD_SIZE_PADDED) + + query_offset_1[:, None] * (NUM_SEGMENTS_PER_SEQ * HEAD_SIZE_PADDED) + + segm_idx * HEAD_SIZE_PADDED + + tl.arange(0, HEAD_SIZE_PADDED)[None, :] + ) + tl.store( + segm_output_ptr + segm_output_offset, + acc, + mask=dim_mask[None, :] & query_mask_0[:, None] & query_mask_1[:, None], + ) + segm_offset = ( + query_offset_0.to(tl.int64) * (num_query_heads * NUM_SEGMENTS_PER_SEQ) + + query_offset_1 * NUM_SEGMENTS_PER_SEQ + + segm_idx + ) + tl.store(segm_max_ptr + segm_offset, M, mask=query_mask_0 & query_mask_1) + tl.store(segm_expsum_ptr + segm_offset, L, mask=query_mask_0 & query_mask_1) + + +# =========================================================================== +# Launcher +# =========================================================================== + + +_HADAMARD_CACHE: dict[tuple[int, torch.device, torch.dtype], torch.Tensor] = {} + + +def _get_pit(dim: int, device: torch.device, dtype: torch.dtype) -> torch.Tensor: + """Sylvester Hadamard projection / sqrt(dim). PiT = PiT.T (symmetric).""" + key = (dim, device, dtype) + H = _HADAMARD_CACHE.get(key) + if H is None: + assert (dim & (dim - 1)) == 0, f"dim={dim} must be power of 2" + H = torch.tensor([[1.0]], dtype=torch.float32) + while H.shape[0] < dim: + H = torch.cat([torch.cat([H, H], dim=1), torch.cat([H, -H], dim=1)], dim=0) + H = (H / (dim**0.5)).to(device=device, dtype=dtype).contiguous() + _HADAMARD_CACHE[key] = H + return H + + +def ultraquant_unified_attention( + query: torch.Tensor, # [num_tokens, Hq, D] fp16/bf16 (raw) + kv_cache: torch.Tensor, # [num_blocks, block_size, Hk, padded_slot] uint8 + block_table: torch.Tensor, # [num_seqs, max_num_blocks] int32 + seq_lens: torch.Tensor, # [num_seqs] int32 + query_start_loc: torch.Tensor, # [num_seqs+1] int32 + scale: float, + PiT: torch.Tensor | None = None, + output: torch.Tensor | None = None, + tile_size: int | None = None, + max_query_len: int | None = None, + max_seq_len: int | None = None, + num_kv_splits: int | None = None, + force_2d: bool = False, + sinks: torch.Tensor | None = None, + sliding_window: int | None = None, +) -> torch.Tensor: + """Launch unified UltraQuant attention with scaled F8F6F4 MFMA.""" + assert query.dim() == 3, f"query must be [N, Hq, D], got {query.shape}" + num_tokens, Hq, D = query.shape + Hk = kv_cache.shape[2] + block_size = kv_cache.shape[1] + padded_slot = kv_cache.shape[3] + kv_group_size = Hq // Hk + num_seqs = int(query_start_loc.shape[0] - 1) + device = query.device + + gs = get_group_size() + if padded_slot < slot_size(D, gs): + raise ValueError( + f"ultraquant_unified: cache slot {padded_slot} < expected " + f"{slot_size(D, gs)} for D={D} group_size={gs}" + ) + if Hq % Hk != 0: + raise ValueError(f"Hq={Hq} must be a multiple of Hk={Hk}") + + if PiT is None: + PiT = _get_pit(D, device, torch.float32) + elif PiT.dtype != torch.float32 or not PiT.is_contiguous(): + PiT = PiT.to(torch.float32).contiguous() + + q_rot = (query.float() @ PiT).contiguous() + q_for_kernel = q_rot.to(torch.float8_e4m3fn).contiguous() + + if sinks is not None: + sinks_f32 = sinks if sinks.dtype == torch.float32 else sinks.to(torch.float32) + if not sinks_f32.is_contiguous(): + sinks_f32 = sinks_f32.contiguous() + assert sinks_f32.numel() == Hq, ( + f"sinks must have shape [Hq={Hq}], got numel={sinks_f32.numel()}" + ) + use_sinks = True + else: + sinks_f32 = q_for_kernel # harmless dummy; never dereferenced when USE_SINKS=0 + use_sinks = False + + if output is None: + output = torch.empty_like(query) + + # BLOCK_M heuristic shared with the compressed-KV fallback kernels. + if max_query_len is not None: + is_prefill_like = max_query_len > 1 + else: + is_prefill_like = num_tokens > num_seqs + + if is_prefill_like: + BLOCK_M = max(128, triton.next_power_of_2(kv_group_size)) + else: + BLOCK_M = 16 if kv_group_size <= 16 else triton.next_power_of_2(kv_group_size) + BLOCK_Q = BLOCK_M // kv_group_size + + total_num_q_blocks = num_tokens // BLOCK_Q + num_seqs + + if tile_size is None: + tile_size = 32 if is_prefill_like else 16 + + num_stages_2d = 1 if _is_hip else 2 + num_stages_3d = 3 if _is_hip else 2 + + BLOCK_D = triton.next_power_of_2(D) + N_GROUPS_C = n_groups(D, gs) + K_SCALES_OFFSET = k_scales_offset(D, gs) + V_CODES_OFFSET = v_codes_offset(D, gs) + V_SCALES_OFFSET = v_scales_offset(D, gs) + + scale_for_kernel = float(scale) + + kv_flat = _kv_cache_flat(kv_cache) + + # Dispatch: 2D for prefill / chunked; 3D for pure decode with long KV. + if max_seq_len is None: + max_seq_len_hint = int(block_table.shape[1]) * int(block_size) + else: + max_seq_len_hint = int(max_seq_len) + use_3d = (not force_2d) and (not is_prefill_like) and max_seq_len_hint >= 1024 + + if not use_3d: + kernel_ultraquant_unified_attention_2d[(total_num_q_blocks, Hk)]( + output_ptr=output, + query_ptr=q_for_kernel, + KV_cache_ptr=kv_flat, + block_tables_ptr=block_table, + seq_lens_ptr=seq_lens, + query_start_len_ptr=query_start_loc, + sinks_ptr=sinks_f32, + scale=scale_for_kernel, + num_query_heads=Hq, + num_queries_per_kv=kv_group_size, + block_table_stride=block_table.stride(0), + query_stride_0=q_for_kernel.stride(0), + query_stride_1=q_for_kernel.stride(1), + output_stride_0=output.stride(0), + output_stride_1=output.stride(1), + stride_cache_block=kv_cache.stride(0), + stride_cache_pos=kv_cache.stride(1), + stride_cache_head=kv_cache.stride(2), + BLOCK_SIZE=block_size, + TILE_SIZE=tile_size, + HEAD_SIZE=D, + HEAD_SIZE_PADDED=BLOCK_D, + BLOCK_Q=BLOCK_Q, + BLOCK_M=BLOCK_M, + num_seqs=num_seqs, + K_SCALES_OFFSET=K_SCALES_OFFSET, + V_CODES_OFFSET=V_CODES_OFFSET, + V_SCALES_OFFSET=V_SCALES_OFFSET, + GROUP_SIZE_C=gs, + N_GROUPS_C=N_GROUPS_C, + UE8M0_BIAS_C=UE8M0_BIAS, + USE_SINKS=1 if use_sinks else 0, + SLIDING_WINDOW=int(sliding_window) + if sliding_window and sliding_window > 0 + else 0, + num_warps=4, + num_stages=num_stages_2d, + ) + return output + + # 3D split-KV path + if num_kv_splits is None: + num_kv_splits = 16 + if num_kv_splits < 1: + num_kv_splits = 1 + if num_kv_splits & (num_kv_splits - 1) != 0: + num_kv_splits = 1 << (num_kv_splits.bit_length() - 1) + max_possible_splits = max(1, (max_seq_len_hint + tile_size - 1) // tile_size) + num_segments = max(1, min(num_kv_splits, max_possible_splits)) + + segm_output = torch.empty( + (num_tokens, Hq, num_segments, BLOCK_D), + dtype=torch.float32, + device=device, + ) + segm_max = torch.empty( + (num_tokens, Hq, num_segments), + dtype=torch.float32, + device=device, + ) + segm_expsum = torch.empty( + (num_tokens, Hq, num_segments), + dtype=torch.float32, + device=device, + ) + + kernel_ultraquant_unified_attention_3d[(total_num_q_blocks, Hk, num_segments)]( + segm_output_ptr=segm_output, + segm_max_ptr=segm_max, + segm_expsum_ptr=segm_expsum, + query_ptr=q_for_kernel, + KV_cache_ptr=kv_flat, + block_tables_ptr=block_table, + seq_lens_ptr=seq_lens, + query_start_len_ptr=query_start_loc, + sinks_ptr=sinks_f32, + scale=scale_for_kernel, + num_query_heads=Hq, + num_queries_per_kv=kv_group_size, + block_table_stride=block_table.stride(0), + query_stride_0=q_for_kernel.stride(0), + query_stride_1=q_for_kernel.stride(1), + stride_cache_block=kv_cache.stride(0), + stride_cache_pos=kv_cache.stride(1), + stride_cache_head=kv_cache.stride(2), + BLOCK_SIZE=block_size, + TILE_SIZE=tile_size, + HEAD_SIZE=D, + HEAD_SIZE_PADDED=BLOCK_D, + BLOCK_Q=BLOCK_Q, + BLOCK_M=BLOCK_M, + num_seqs=num_seqs, + NUM_SEGMENTS_PER_SEQ=num_segments, + K_SCALES_OFFSET=K_SCALES_OFFSET, + V_CODES_OFFSET=V_CODES_OFFSET, + V_SCALES_OFFSET=V_SCALES_OFFSET, + GROUP_SIZE_C=gs, + N_GROUPS_C=N_GROUPS_C, + UE8M0_BIAS_C=UE8M0_BIAS, + USE_SINKS=1 if use_sinks else 0, + SLIDING_WINDOW=int(sliding_window) + if sliding_window and sliding_window > 0 + else 0, + num_warps=2, + num_stages=num_stages_3d, + ) + + # Reduce split-KV partials with the shared vectorized reducer. + reduce_segments[(num_tokens, Hq)]( + output_ptr=output, + segm_output_ptr=segm_output, + segm_max_ptr=segm_max, + segm_expsum_ptr=segm_expsum, + seq_lens_ptr=seq_lens, + num_seqs=num_seqs, + num_query_heads=Hq, + out_scale_inv=1.0, + output_stride_0=output.stride(0), + output_stride_1=output.stride(1), + block_table_stride=block_table.stride(0), + TILE_SIZE=tile_size, + HEAD_SIZE=D, + HEAD_SIZE_PADDED=BLOCK_D, + query_start_len_ptr=query_start_loc, + BLOCK_Q=BLOCK_Q, + NUM_SEGMENTS_PER_SEQ=num_segments, + USE_FP8=False, + ) + + return output diff --git a/vllm/v1/core/block_pool.py b/vllm/v1/core/block_pool.py index 664804d50d..3b0102fe84 100644 --- a/vllm/v1/core/block_pool.py +++ b/vllm/v1/core/block_pool.py @@ -479,8 +479,8 @@ class BlockPool: kv_cache_group_id: KV cache group that owns the partial entry. block_size: Cache block size for the owning group. The partial entry hash itself is always the prefix-chain hash at - ``num_tokens``; ``block_size`` is used to assert that the - entry is partial within the owning cache block. + ``num_tokens``; ``block_size`` must be a multiple of the hash + block size. replace_existing_hashes: Whether the block contents were replaced and all existing cache entries must be removed before the new entry is registered. @@ -494,9 +494,6 @@ class BlockPool: return None assert block_size % self.hash_block_size == 0 - assert replace_existing_hashes or ( - block_size > self.hash_block_size and num_tokens % block_size != 0 - ) block_hash = self._get_partial_block_hash(request, num_tokens) num_hash_blocks = num_tokens // self.hash_block_size block_hash_with_group_id = make_block_hash_with_group_id( @@ -829,7 +826,7 @@ class BlockPool: def reset_prefix_cache(self) -> bool: """Reset prefix cache. This function may be used in RLHF - flows to invalid prefix caching after the weights are updated, + flows to invalidate prefix caching after the weights are updated, or used for resetting prefix caching status for benchmarking. Returns: diff --git a/vllm/v1/core/kv_cache_coordinator.py b/vllm/v1/core/kv_cache_coordinator.py index 9b91bb1920..c32173a635 100644 --- a/vllm/v1/core/kv_cache_coordinator.py +++ b/vllm/v1/core/kv_cache_coordinator.py @@ -4,13 +4,14 @@ from abc import ABC, abstractmethod from collections.abc import Sequence from typing import NamedTuple -from vllm.logger import init_logger from vllm.utils.math_utils import cdiv, round_down from vllm.v1.core.block_pool import BlockPool from vllm.v1.core.kv_cache_metrics import KVCacheMetricsCollector from vllm.v1.core.kv_cache_utils import ( BlockHash, KVCacheBlock, + eagle_proof_margin, + partial_hash_hits_enabled, ) from vllm.v1.core.single_type_kv_cache_manager import ( CrossAttentionManager, @@ -28,8 +29,6 @@ from vllm.v1.kv_cache_interface import ( ) from vllm.v1.request import Request -logger = init_logger(__name__) - def _validate_prefix_cache_retention_interval( retention_interval: int | None, @@ -341,6 +340,14 @@ class KVCacheCoordinator(ABC): extension = request.num_prompt_tokens // block * block return tuple(sorted({max(resend - block, 0), max(extension - block, 0)})) + def get_num_cacheable_tokens( + self, num_computed_tokens: int, kv_cache_group_id: int + ) -> int: + """Return the prefix eligible for hashing in this group.""" + # Only cache tokens with finalized KV. The last num_reprefillable_tokens + # tokens can be re-prefilled during multi-module MTP. + return max(0, num_computed_tokens - self.num_reprefillable_tokens) + def cache_blocks(self, request: Request, num_computed_tokens: int) -> None: """Cache the blocks for the request. @@ -352,17 +359,12 @@ class KVCacheCoordinator(ABC): """ boundaries = self.get_replay_boundaries(request) - for manager in self.single_type_managers: + for group_id, manager in enumerate(self.single_type_managers): if not manager.enable_caching: continue - # Only cache tokens with finalized KV. The last num_reprefillable_tokens - # tokens can be re-prefilled during multi-module MTP. - num_tokens_to_cache = max( - 0, num_computed_tokens - self.num_reprefillable_tokens - ) manager.cache_blocks( request, - num_tokens_to_cache, + self.get_num_cacheable_tokens(num_computed_tokens, group_id), retention_interval=self.retention_interval, replay_boundaries=boundaries, ) @@ -692,37 +694,15 @@ class HybridKVCacheCoordinator(KVCacheCoordinator): # managers in every group. TP needs hashing finer than the Mamba block; # DCP accepts equality because it scales the effective full-attention # block instead. - has_partial_mamba_group = any( - isinstance(g.kv_cache_spec, MambaSpec) - and g.kv_cache_spec.mamba_cache_mode == "align" - and ( - (dcp_world_size == 1 and g.kv_cache_spec.block_size > hash_block_size) - or ( - dcp_world_size > 1 and g.kv_cache_spec.block_size >= hash_block_size - ) - ) - for g in kv_cache_config.kv_cache_groups - ) self.enable_partial_hash_hits = ( - allow_partial_hash_hits and has_partial_mamba_group + allow_partial_hash_hits + and partial_hash_hits_enabled( + kv_cache_config.kv_cache_groups, + hash_block_size, + dcp_world_size, + [type(manager) for manager in self.single_type_managers], + ) ) - if self.enable_partial_hash_hits: - unsupported_partial_hit_managers = { - type(manager).__name__ - for manager, group in zip( - self.single_type_managers, kv_cache_config.kv_cache_groups - ) - if group.kv_cache_spec.prefix_cacheable - and not manager.supports_fine_grained_hash_lookup - and manager.block_size != hash_block_size - } - if unsupported_partial_hit_managers: - self.enable_partial_hash_hits = False - logger.warning_once( - "Disabling fine-grained prefix-cache hits because these KV " - "cache managers require block-aligned lookups: %s.", - ", ".join(sorted(unsupported_partial_hit_managers)), - ) cache_hit_alignment_tokens = self._cache_hit_alignment_tokens for manager in self.single_type_managers: manager.cache_hit_alignment_tokens = cache_hit_alignment_tokens @@ -809,39 +789,22 @@ class HybridKVCacheCoordinator(KVCacheCoordinator): return num_tokens return round_down(num_tokens, self.scheduler_block_size) - def cache_blocks(self, request: Request, num_computed_tokens: int) -> None: + def get_num_cacheable_tokens( + self, num_computed_tokens: int, kv_cache_group_id: int + ) -> int: cached_num_computed_tokens = self._align_cacheable(num_computed_tokens) - boundaries = self.get_replay_boundaries(request) - for manager in self.single_type_managers: - if not manager.enable_caching: - continue - num_tokens_to_cache = cached_num_computed_tokens - # EAGLE groups match one block past each aligned boundary and drop - # it, so make that lookahead block eligible to be cached. - if manager.use_eagle and cached_num_computed_tokens > 0: - # Only cache tokens with finalized KV. The last - # num_reprefillable_tokens tokens can be re-prefilled during - # multi-module MTP. - num_finalized_computed_tokens = max( - 0, num_computed_tokens - self.num_reprefillable_tokens - ) - cached_num_finalized_computed_tokens = self._align_cacheable( - num_finalized_computed_tokens - ) - num_tokens_to_cache = min( - num_finalized_computed_tokens, - cached_num_finalized_computed_tokens + manager.block_size, - ) - # The manager already knows the fine hit granularity - # (``scheduler_block_size``); retention is passed separately so it - # can keep both the coarse segment tails and the fine replay - # boundary (which needs the fine value). - manager.cache_blocks( - request, - num_tokens_to_cache, - retention_interval=self.retention_interval, - replay_boundaries=boundaries, + manager = self.single_type_managers[kv_cache_group_id] + # EAGLE matches one block past the aligned boundary, then drops it. + if manager.use_eagle and cached_num_computed_tokens > 0: + num_finalized_computed_tokens = max( + 0, num_computed_tokens - self.num_reprefillable_tokens ) + return min( + num_finalized_computed_tokens, + self._align_cacheable(num_finalized_computed_tokens) + + manager.block_size, + ) + return cached_num_computed_tokens def find_longest_cache_hit( self, @@ -913,15 +876,15 @@ class HybridKVCacheCoordinator(KVCacheCoordinator): # mamba: its finder never drops (draft models have no mamba # layers), so the hit would grow past the candidate. if drop_eagle_block and not isinstance(spec, MambaSpec): - eagle_margin = ( - self.hash_block_size - if self.enable_partial_hash_hits - and manager_cls.supports_fine_grained_hash_lookup - and group_block_size > self.hash_block_size - else group_block_size + eagle_margin = eagle_proof_margin( + group_block_size, + self.hash_block_size, + self.enable_partial_hash_hits + and manager_cls.supports_fine_grained_hash_lookup, ) _max_length = min( - curr_hit_length + eagle_margin, max_cache_hit_length + curr_hit_length + eagle_margin, + len(block_hashes) * self.hash_block_size, ) hit_blocks, _new_hit_length = manager_cls.find_longest_cache_hit( block_hashes=block_hashes, @@ -997,9 +960,21 @@ class HybridKVCacheCoordinator(KVCacheCoordinator): for spec, group_ids, manager_cls, use_eagle in self.attention_groups: manager = self.single_type_managers[group_ids[0]] + lookup_length = max_cache_hit_length + if use_eagle and not isinstance(spec, MambaSpec): + eagle_margin = eagle_proof_margin( + manager.block_size, + self.hash_block_size, + self.enable_partial_hash_hits + and manager_cls.supports_fine_grained_hash_lookup, + ) + lookup_length = min( + max_cache_hit_length + eagle_margin, + len(block_hashes) * self.hash_block_size, + ) blocks, group_hit = manager_cls.find_longest_cache_hit( block_hashes=block_hashes, - max_length=max_cache_hit_length, + max_length=lookup_length, kv_cache_group_ids=group_ids, block_pool=manager.block_pool, kv_cache_spec=spec, diff --git a/vllm/v1/core/kv_cache_utils.py b/vllm/v1/core/kv_cache_utils.py index 46934d530b..53a01a5cab 100644 --- a/vllm/v1/core/kv_cache_utils.py +++ b/vllm/v1/core/kv_cache_utils.py @@ -53,6 +53,7 @@ from vllm.v1.utils import tensor_data if TYPE_CHECKING: from vllm.v1.core.block_pool import BlockPool + from vllm.v1.core.single_type_kv_cache_manager import SingleTypeKVCacheManager # BlockHash represents the hash of a single KV-cache block used for @@ -303,19 +304,17 @@ class FreeKVCacheBlockQueue: The first free block. """ - if ( - self.fake_free_list_head.next_free_block is self.fake_free_list_tail - or self.fake_free_list_head.next_free_block is None - ): + head = self.fake_free_list_head + first_block = head.next_free_block + if first_block is self.fake_free_list_tail or first_block is None: assert self.num_free_blocks == 0, ( f"num_free_blocks ({self.num_free_blocks}) is out of sync " "with the free list." ) raise ValueError("No free blocks available") - first_block: KVCacheBlock = self.fake_free_list_head.next_free_block - - if first_block.next_free_block is None: + next_block = first_block.next_free_block + if next_block is None: # This should not happen if the block is from the free list. # It indicates a bug in the caller's logic. raise RuntimeError( @@ -325,8 +324,8 @@ class FreeKVCacheBlockQueue: # Connect fake_head and the next block of first_block (i.e. second block # or fake tail). - self.fake_free_list_head.next_free_block = first_block.next_free_block - first_block.next_free_block.prev_free_block = self.fake_free_list_head + head.next_free_block = next_block + next_block.prev_free_block = head # Remove the block from the linked list. first_block.prev_free_block = first_block.next_free_block = None @@ -350,6 +349,16 @@ class FreeKVCacheBlockQueue: self.num_free_blocks -= n curr_block = self.fake_free_list_head.next_free_block + if n == 1: + assert curr_block is not None + next_block = curr_block.next_free_block + curr_block.prev_free_block = None + curr_block.next_free_block = None + if next_block is not None: + self.fake_free_list_head.next_free_block = next_block + next_block.prev_free_block = self.fake_free_list_head + return [curr_block] + # Pop n blocks from the head of the list ret = [] for _ in range(n): @@ -375,15 +384,17 @@ class FreeKVCacheBlockQueue: block: The block to remove. """ - if block.prev_free_block is None or block.next_free_block is None: + prev_block = block.prev_free_block + next_block = block.next_free_block + if prev_block is None or next_block is None: # This should not happen if the block is from the free list. # It indicates a bug in the caller's logic. raise RuntimeError(f"remove() called on an invalid block: {block}") # Link the previous block to the next block. - block.prev_free_block.next_free_block = block.next_free_block + prev_block.next_free_block = next_block # Link the next block to the previous block. - block.next_free_block.prev_free_block = block.prev_free_block + next_block.prev_free_block = prev_block # Remove the block from the linked list. block.prev_free_block = block.next_free_block = None @@ -397,19 +408,19 @@ class FreeKVCacheBlockQueue: block: The block to append. """ - if self.fake_free_list_tail.prev_free_block is None: + tail = self.fake_free_list_tail + last_block = tail.prev_free_block + if last_block is None: raise RuntimeError( "prev_free_block of fake_free_list_tail should always exist" ) - last_block: KVCacheBlock = self.fake_free_list_tail.prev_free_block - # Connect the new block after the last block. last_block.next_free_block = block block.prev_free_block = last_block # Connect the fake tail after the new block. - block.next_free_block = self.fake_free_list_tail - self.fake_free_list_tail.prev_free_block = block + block.next_free_block = tail + tail.prev_free_block = block self.num_free_blocks += 1 @@ -441,10 +452,11 @@ class FreeKVCacheBlockQueue: blocks: The blocks to append. """ - if len(blocks) == 0: + if not blocks: return - last_block = self.fake_free_list_tail.prev_free_block + tail = self.fake_free_list_tail + last_block = tail.prev_free_block assert last_block is not None, ( "prev_free_block of fake_free_list_tail should always exist" ) @@ -455,8 +467,8 @@ class FreeKVCacheBlockQueue: last_block = block # Connect the last block of to the fake tail - last_block.next_free_block = self.fake_free_list_tail - self.fake_free_list_tail.prev_free_block = last_block + last_block.next_free_block = tail + tail.prev_free_block = last_block self.num_free_blocks += len(blocks) @@ -736,6 +748,60 @@ def resolve_dcp_kv_cache_spec(spec: KVCacheSpec, dcp_world_size: int) -> KVCache return replace(spec, block_size=block_size) +def partial_hash_hits_enabled( + kv_cache_groups: Sequence[KVCacheGroupSpec], + hash_block_size: int, + dcp_world_size: int = 1, + manager_classes: Sequence[type["SingleTypeKVCacheManager"]] | None = None, +) -> bool: + """Whether aligned Mamba states support sub-block prefix-cache hits. + + If ``manager_classes`` (one per group) is given, also require every other + prefix-cacheable group to support fine-grained lookups. + """ + if not any( + isinstance(spec, MambaSpec) + and spec.mamba_cache_mode == "align" + and ( + (dcp_world_size == 1 and spec.block_size > hash_block_size) + or (dcp_world_size > 1 and spec.block_size >= hash_block_size) + ) + for group in kv_cache_groups + for spec in iter_layer_specs(group.kv_cache_spec) + ): + return False + if manager_classes is None: + return True + unsupported_managers = { + manager_cls.__name__ + for group, manager_cls in zip(kv_cache_groups, manager_classes) + if group.kv_cache_spec.prefix_cacheable + and not manager_cls.supports_fine_grained_hash_lookup + and resolve_dcp_kv_block_size(group.kv_cache_spec, dcp_world_size) + != hash_block_size + } + if unsupported_managers: + logger.warning_once( + "Disabling fine-grained prefix-cache hits because these KV " + "cache managers require block-aligned lookups: %s.", + ", ".join(sorted(unsupported_managers)), + ) + return False + return True + + +def eagle_proof_margin( + block_size: int, hash_block_size: int, fine_grained_lookup: bool +) -> int: + """Tokens an EAGLE group matches past a cache hit before dropping them. + + Fine-grained lookups drop one hash unit; others drop one cache block. + """ + if fine_grained_lookup and block_size > hash_block_size: + return hash_block_size + return block_size + + def resolve_kv_cache_block_sizes( kv_cache_config: KVCacheConfig, vllm_config: VllmConfig, @@ -764,8 +830,7 @@ def resolve_kv_cache_block_sizes( if groups and not groups[0].kv_cache_spec.dcp_sharded: dcp = 1 bs = cache_config.block_size * dcp - # The Mamba prefill checkpoint builder reads prefix_match_unit directly, - # so a value dropped here puts its checkpoints off the scheduler's grid. + # Reject a prefix_match_unit that would otherwise be silently ignored. if ( cache_config.prefix_match_unit not in (None, bs) and len(groups) == 1 @@ -786,8 +851,8 @@ def resolve_kv_cache_block_sizes( ] scheduler_block_size = math.lcm(*group_block_sizes) - logger.info("kv cache group sizes %s", group_block_sizes) - logger.info("kv lcm block sizes %s", scheduler_block_size) + logger.info_once("kv cache group sizes %s", tuple(group_block_sizes)) + logger.info_once("kv lcm block sizes %s", scheduler_block_size) # Block hashes are only consumed by prefix caching and KV connectors # (P/D, offloading); when neither is active, keep hash_block_size equal @@ -828,18 +893,10 @@ def resolve_kv_cache_block_sizes( and isinstance(spec.tokens_per_state, int) and spec.tokens_per_state > 1 } - has_partial_mamba_group = any( - isinstance(spec, MambaSpec) - and spec.mamba_cache_mode == "align" - and ( - (dcp == 1 and block_size > hash_block_size) - or (dcp > 1 and block_size >= hash_block_size) - ) - for group, block_size in zip(groups, group_block_sizes) - for spec in iter_layer_specs(group.kv_cache_spec) - ) cache_hit_alignment = ( - hash_block_size if has_partial_mamba_group else scheduler_block_size + hash_block_size + if partial_hash_hits_enabled(groups, hash_block_size, dcp) + else scheduler_block_size ) if any(cache_hit_alignment % alignment for alignment in prefix_alignments): raise ValueError( @@ -850,6 +907,38 @@ def resolve_kv_cache_block_sizes( return scheduler_block_size, hash_block_size +def resolve_cache_hit_alignment_tokens( + kv_cache_config: KVCacheConfig, + vllm_config: VllmConfig, + scheduler_block_size: int, + hash_block_size: int, +) -> int: + """Token granularity at which prefix-cache hits land. + + Mirrors the alignment the hybrid KV cache coordinator pushes to its + managers: fine-grained hits land on ``hash_block_size``, all others on + ``scheduler_block_size``. + """ + groups = kv_cache_config.kv_cache_groups + if not vllm_config.cache_config.enable_prefix_caching or len(groups) <= 1: + return scheduler_block_size + manager_classes = [] + for group in groups: + manager_cls = KVCacheSpecRegistry.get_manager_class( + group.kv_cache_spec, group.role + ) + assert manager_cls is not None + manager_classes.append(manager_cls) + if partial_hash_hits_enabled( + groups, + hash_block_size, + vllm_config.parallel_config.decode_context_parallel_size, + manager_classes, + ): + return hash_block_size + return scheduler_block_size + + def get_request_block_hasher( hash_block_size: int, caching_hash_fn: Callable[[Any], bytes], diff --git a/vllm/v1/core/sched/interface.py b/vllm/v1/core/sched/interface.py index c868d249c8..dd88d99c1f 100644 --- a/vllm/v1/core/sched/interface.py +++ b/vllm/v1/core/sched/interface.py @@ -26,7 +26,7 @@ class PauseState(enum.IntEnum): - UNPAUSED: Normal operation - PAUSE_NEW: No new requests are scheduled, requests already in - running state are scheduled. + running state or holding KV blocks are scheduled. - PAUSE_ALL: No requests are scheduled """ diff --git a/vllm/v1/core/sched/scheduler.py b/vllm/v1/core/sched/scheduler.py index b390f1aadb..0758c61225 100644 --- a/vllm/v1/core/sched/scheduler.py +++ b/vllm/v1/core/sched/scheduler.py @@ -147,9 +147,15 @@ class Scheduler(SchedulerInterface): self.kv_events_config is not None and self.kv_events_config.enable_kv_cache_events ) - # Diffusion models may not sample any tokens for a denoising step. + # Pooling and diffusion models do not append a sampled token after + # prefill, so they should not reserve one context slot while scheduling. self.num_sampled_tokens_per_step = ( - 1 if not vllm_config.model_config.is_diffusion else 0 + 0 + if ( + vllm_config.model_config.runner_type == "pooling" + or vllm_config.model_config.is_diffusion + ) + else 1 ) # Create KVConnector for the Scheduler. Note that each Worker @@ -284,10 +290,14 @@ class Scheduler(SchedulerInterface): # DSV41 SWA bounded replay: groups that declare a replay window are rebuilt # after a prefix hit by recomputing its trailing tokens. One window # for all such groups, so the rewind matches every group's allocation. - replay_windows = { - group.kv_cache_spec.prefix_replay_tokens - for group in kv_cache_config.kv_cache_groups + self.prefix_replay_group_ids = tuple( + group_id + for group_id, group in enumerate(kv_cache_config.kv_cache_groups) if group.kv_cache_spec.prefix_replay_tokens > 0 + ) + replay_windows = { + kv_cache_config.kv_cache_groups[group_id].kv_cache_spec.prefix_replay_tokens + for group_id in self.prefix_replay_group_ids } assert len(replay_windows) <= 1, ( f"Prefix replay windows should agree: {sorted(replay_windows)}" @@ -371,9 +381,8 @@ class Scheduler(SchedulerInterface): or group.kv_cache_spec.num_prefill_checkpoint_blocks > 0 for group in kv_cache_config.kv_cache_groups ) - # A finer prefix_match_unit is configured: a mamba partial tail entry - # can only be registered by a step ending exactly at the prompt's last - # hash boundary, so the split adds that stop. + # Fine-grained hits reuse the prompt's hash-aligned Mamba checkpoint. + # Split there only if the backend cannot save it within the final chunk. self.mamba_partial_cache_hit = ( self.need_mamba_block_aligned_split and self.hash_block_size < self.block_size @@ -450,14 +459,19 @@ class Scheduler(SchedulerInterface): # The last block-aligned position whose state can be cached. With # Eagle, FullAttn prunes the last matching block, so back off one # block to avoid a Mamba cache miss. - last_cache_position = request.num_tokens - request.num_tokens % block_size - if self.use_eagle_block_drop: - last_cache_position = max(last_cache_position - block_size, 0) + ckpt_block_size = ( + self.hash_block_size if self.mamba_partial_cache_hit else self.block_size + ) + last_cache_position = get_mamba_prefill_checkpoint_position( + request.num_tokens, + block_size, + self.use_eagle_block_drop, + ) end = start + num_new_tokens checkpoint_position = get_mamba_prefill_checkpoint_position( prefill_end, - self.hash_block_size, + ckpt_block_size, drop_eagle_block=self.use_eagle_block_drop, ) use_internal_checkpoint = ( @@ -490,18 +504,14 @@ class Scheduler(SchedulerInterface): next_block_boundary = (start // block_size + 1) * block_size tail_boundary = ( - request.num_prompt_tokens // self.hash_block_size * self.hash_block_size + get_mamba_prefill_checkpoint_position( + request.num_prompt_tokens, + self.hash_block_size, + self.use_eagle_block_drop, + ) if self.mamba_partial_cache_hit and not use_internal_checkpoint else 0 ) - if tail_boundary and self.use_eagle_block_drop: - # Eagle matches one hash unit past the candidate and drops it, so - # nothing proves the prompt's own last hash boundary. Materialize - # the state one unit lower, where the hit can actually land. Keyed on - # the block-drop bit, not plain use_eagle: this shift exists only to - # compensate for the drop, and the Mamba manager's matching gate - # reads the same bit (the coordinator is handed use_eagle_block_drop). - tail_boundary = max(tail_boundary - self.hash_block_size, 0) junction = request.shared_prefix_boundary # Block-floored: a sub-block junction's state is not separately cacheable. block_floored = start + (junction - start) // block_size * block_size @@ -522,8 +532,7 @@ class Scheduler(SchedulerInterface): else 0, # Never run past the last cacheable block boundary mid-chunk. last_cache_position, - # Fine-grained hits: the prompt's partial-tail entry can only be - # registered by a chunk ending exactly at its last hash boundary. + # Stop at the prompt checkpoint unless the backend saves it internally. tail_boundary if last_cache_position < tail_boundary < request.num_prompt_tokens else 0, @@ -709,6 +718,7 @@ class Scheduler(SchedulerInterface): # Schedule encoder inputs. encoder_inputs_to_schedule = None external_load_encoder_input: list[int] = [] + duplicate_encoder_inputs: list[int] = [] new_encoder_compute_budget = encoder_compute_budget if request.has_encoder_inputs: ( @@ -716,6 +726,7 @@ class Scheduler(SchedulerInterface): num_new_tokens, new_encoder_compute_budget, external_load_encoder_input, + duplicate_encoder_inputs, ) = self._try_schedule_encoder_inputs( request, request.num_computed_tokens, @@ -856,17 +867,13 @@ class Scheduler(SchedulerInterface): # Encoder-related. if encoder_inputs_to_schedule: scheduled_encoder_inputs[request_id] = encoder_inputs_to_schedule - # Allocate the encoder cache. - for i in encoder_inputs_to_schedule: - self.encoder_cache_manager.allocate(request, i) - if self.ec_connector is not None: - self.ec_connector.update_state_after_alloc(request, i) encoder_compute_budget = new_encoder_compute_budget - if external_load_encoder_input: - for i in external_load_encoder_input: - self.encoder_cache_manager.allocate(request, i) - if self.ec_connector is not None: - self.ec_connector.update_state_after_alloc(request, i) + if encoder_inputs_to_schedule or external_load_encoder_input: + self._allocate_encoder_inputs( + request, + (encoder_inputs_to_schedule or []) + external_load_encoder_input, + duplicate_encoder_inputs, + ) # Record the LoRAs in scheduled_running_reqs scheduled_loras: set[int] = set() @@ -879,7 +886,7 @@ class Scheduler(SchedulerInterface): assert len(scheduled_loras) <= self.lora_config.max_loras # Next, schedule the WAITING requests. - if not preempted_reqs and self._pause_state == PauseState.UNPAUSED: + if not preempted_reqs and self._pause_state != PauseState.PAUSED_ALL: step_skipped_waiting: deque[Request] = deque() step_skipped_kv_holding: deque[Request] = deque() @@ -892,8 +899,11 @@ class Scheduler(SchedulerInterface): step_skipped_waiting.appendleft(request_to_skip) while token_budget > 0: - # Requests holding KV blocks are always drained first. - request_queue = self.kv_holding_waiting or self.waiting + # Requests holding KV blocks are always drained first, and are + # the only ones admitted under PAUSED_NEW (e.g. async KV loads). + request_queue = self.kv_holding_waiting + if not request_queue and self._pause_state == PauseState.UNPAUSED: + request_queue = self.waiting if not request_queue or input_budget <= draft_slots: break # Paused streaming sessions (WAITING_FOR_STREAMING_REQ) are not @@ -969,14 +979,17 @@ class Scheduler(SchedulerInterface): skip_request(request_queue) continue - if self.prefix_replay_tokens: - # SWA bounded replay recomputes the hit's last + if self.prefix_replay_tokens and not ( + ext_tokens and self._load_restores_replay_window(request) + ): + # DSV41 SWA bounded replay recomputes the hit's last # window, from the block holding its first # token; the sliding-window groups retire # whole blocks below the window of the hit's # next token. Hits end on a block boundary, or # a hit ending one token short of one would - # retire the block the replay starts in. + # retire the block the replay starts in. A load + # that carries the window replays nothing. ext_tokens -= ext_tokens % self.block_size load_kv_async = load_kv_async and ext_tokens > 0 @@ -1022,9 +1035,17 @@ class Scheduler(SchedulerInterface): num_new_local_computed_tokens + num_external_computed_tokens ) assert num_computed_tokens <= request.num_tokens - if 0 < num_computed_tokens <= self.prefix_replay_tokens: - # SWA bounded replay: a hit no longer than the replayed - # window would be recomputed in full and save nothing. + # DSV41 SWA bounded replay: a KV load that carries the + # window is taken as is; any other hit no longer than the + # replayed window would be recomputed in full and save + # nothing. + window_loaded = bool( + num_external_computed_tokens + ) and self._load_restores_replay_window(request) + if ( + not window_loaded + and 0 < num_computed_tokens <= self.prefix_replay_tokens + ): new_computed_blocks = ( self.kv_cache_manager.empty_kv_cache_blocks ) @@ -1065,6 +1086,7 @@ class Scheduler(SchedulerInterface): encoder_inputs_to_schedule = None external_load_encoder_input = [] + duplicate_encoder_inputs = [] new_encoder_compute_budget = encoder_compute_budget pad_spec_decode = False @@ -1086,7 +1108,7 @@ class Scheduler(SchedulerInterface): if did_prefix_cache_lookup: # Fresh admission: local and/or sync external hit. num_replay_tokens = self._mark_prefix_replay( - request, num_computed_tokens + request, num_computed_tokens, window_loaded ) num_computed_tokens -= num_replay_tokens # Number of tokens to be scheduled. @@ -1166,6 +1188,7 @@ class Scheduler(SchedulerInterface): num_new_tokens, new_encoder_compute_budget, external_load_encoder_input, + duplicate_encoder_inputs, ) = self._try_schedule_encoder_inputs( request, num_computed_tokens, @@ -1261,10 +1284,20 @@ class Scheduler(SchedulerInterface): self.connector_prefix_cache_stats is not None and connector_prefix_cache_queries != 0 ): + preempted = request.num_preemptions > 0 self.connector_prefix_cache_stats.record( num_tokens=connector_prefix_cache_queries, num_hits=connector_prefix_cache_hits, - preempted=request.num_preemptions > 0, + preempted=preempted, + # The connector knows the tier only after the load + # plan above exists. + hits_by_source=( + self.connector.get_external_cache_hit_sources( + request, connector_prefix_cache_hits + ) + if connector_prefix_cache_hits and not preempted + else None + ), ) # Record at admission so unscheduled lookups are not counted. @@ -1335,18 +1368,14 @@ class Scheduler(SchedulerInterface): # Encoder-related. if encoder_inputs_to_schedule: scheduled_encoder_inputs[request_id] = encoder_inputs_to_schedule - # Allocate the encoder cache. - for i in encoder_inputs_to_schedule: - self.encoder_cache_manager.allocate(request, i) - if self.ec_connector is not None: - self.ec_connector.update_state_after_alloc(request, i) encoder_compute_budget = new_encoder_compute_budget - # Allocate for external load encoder cache - if external_load_encoder_input: - for i in external_load_encoder_input: - self.encoder_cache_manager.allocate(request, i) - if self.ec_connector is not None: - self.ec_connector.update_state_after_alloc(request, i) + if encoder_inputs_to_schedule or external_load_encoder_input: + self._allocate_encoder_inputs( + request, + (encoder_inputs_to_schedule or []) + + external_load_encoder_input, + duplicate_encoder_inputs, + ) # re-queue requests skipped in this pass ahead of older skipped items. if step_skipped_kv_holding: @@ -1632,6 +1661,7 @@ class Scheduler(SchedulerInterface): session.num_prompt_tokens : num_computed_tokens ] del session._all_token_ids[num_computed_tokens:] + del session.block_hashes[num_computed_tokens // self.hash_block_size :] session._output_token_ids.clear() assert session.prompt_token_ids is not None # Extend prompt with kept output tokens. @@ -1742,6 +1772,20 @@ class Scheduler(SchedulerInterface): self.encoder_cache_mismatch_reqs.add(request.request_id) return True + def _allocate_encoder_inputs( + self, + request: Request, + input_ids: list[int], + duplicate_input_ids: list[int], + ) -> None: + for input_id in input_ids: + self.encoder_cache_manager.allocate(request, input_id) + if self.ec_connector is not None: + self.ec_connector.update_state_after_alloc(request, input_id) + + for input_id in duplicate_input_ids: + self.encoder_cache_manager.check_and_update_cache(request, input_id) + def _try_schedule_encoder_inputs( self, request: Request, @@ -1749,7 +1793,7 @@ class Scheduler(SchedulerInterface): num_new_tokens: int, encoder_compute_budget: int, shift_computed_tokens: int = 0, - ) -> tuple[list[int], int, int, list[int]]: + ) -> tuple[list[int], int, int, list[int], list[int]]: """Determine which encoder inputs need to be scheduled in the current step, and update `num_new_tokens` and encoder token budget accordingly. @@ -1770,12 +1814,13 @@ class Scheduler(SchedulerInterface): blocks and externally cached blocks (via KVConnector). """ if num_new_tokens == 0 or not request.has_encoder_inputs: - return [], num_new_tokens, encoder_compute_budget, [] + return [], num_new_tokens, encoder_compute_budget, [], [] encoder_inputs_to_schedule: list[int] = [] mm_features = request.mm_features assert mm_features is not None assert len(mm_features) > 0 external_load_encoder_input = [] + duplicate_encoder_inputs = [] # NOTE: since scheduler operates on the request level (possibly with # multiple encoder inputs per request), we need to create temporary @@ -1832,9 +1877,10 @@ class Scheduler(SchedulerInterface): scheduled_num_encoder_embeds, num_encoder_embeds, ): - return [], 0, encoder_compute_budget, [] + return [], 0, encoder_compute_budget, [], [] # The same encoder input has already been scheduled in the # current step. + duplicate_encoder_inputs.append(i) continue cached_num_encoder_embeds = ( @@ -1849,7 +1895,7 @@ class Scheduler(SchedulerInterface): num_encoder_embeds, ) ): - return [], 0, encoder_compute_budget, [] + return [], 0, encoder_compute_budget, [], [] if self.encoder_cache_manager.check_and_update_cache(request, i): # The encoder input is already computed and cached from a @@ -1925,6 +1971,7 @@ class Scheduler(SchedulerInterface): num_new_tokens, encoder_compute_budget, external_load_encoder_input, + duplicate_encoder_inputs, ) def _make_scheduled_encoder_input_stats( @@ -2757,7 +2804,8 @@ class Scheduler(SchedulerInterface): return 0 num_running, num_waiting = self.get_request_counts() if self._pause_state == PauseState.PAUSED_NEW: - return num_running + # Requests holding KV blocks still drain; queued ones stay queued. + num_waiting = len(self.kv_holding_waiting) num_waiting -= self.num_waiting_for_streaming_input return num_waiting + num_running @@ -3053,16 +3101,31 @@ class Scheduler(SchedulerInterface): self._request_remaining_blocks(req) for req in self._inflight_prefills ) - def _mark_prefix_replay(self, request: Request, num_hit_tokens: int) -> int: + def _load_restores_replay_window(self, request: Request) -> bool: + """Whether the KV connector's load for ``request`` restores every DSV41 + SWA bounded-replay group, as a P/D transfer of the request's own blocks + does; a prefix-cache store holds no window.""" + assert self.connector is not None + return bool(self.prefix_replay_group_ids) and set( + self.prefix_replay_group_ids + ).issubset(self.connector.get_loaded_kv_cache_group_ids(request)) + + def _mark_prefix_replay( + self, request: Request, num_hit_tokens: int, window_loaded: bool + ) -> int: """Record where a prefix hit's replay starts and return the number of hit tokens to recompute; the caller rewinds the computed count by it. The replayed tokens are the hit's last window (see ``Request.replay_start``): the worker rebuilds their sliding-window KV - and leaves their cached KV alone. + and leaves their cached KV alone. A hit whose KV load carries the + window replays nothing. """ if not self.prefix_replay_tokens: return 0 + if window_loaded: + request.replay_start = 0 + return 0 num_replay_tokens = min(self.prefix_replay_tokens, num_hit_tokens) request.replay_start = num_hit_tokens - num_replay_tokens return num_replay_tokens @@ -3076,7 +3139,8 @@ class Scheduler(SchedulerInterface): """ assert self.connector is not None - if request.request_id in self.failed_recving_kv_req_ids: + load_failed = request.request_id in self.failed_recving_kv_req_ids + if load_failed: # Request had KV load failures; num_computed_tokens was already # updated in _update_requests_with_invalid_blocks if request.num_computed_tokens: @@ -3102,11 +3166,15 @@ class Scheduler(SchedulerInterface): # This will cache the blocks iff caching is enabled. self.kv_cache_manager.cache_blocks(request, request.num_computed_tokens) - # SWA bounded replay recomputes the tail of the hit, which covers the - # last token; otherwise a full prompt hit re-computes that token so the - # next one can be sampled. + # DSV41 SWA bounded replay recomputes the tail of a hit loaded without + # its window, or of a failed load, which may have lost it; the replay + # covers the last token. Otherwise a full prompt hit re-computes that + # token so the next one can be sampled. num_replay_tokens = self._mark_prefix_replay( - request, request.num_computed_tokens + request, + request.num_computed_tokens, + window_loaded=not load_failed + and self._load_restores_replay_window(request), ) if num_replay_tokens > 0: request.num_computed_tokens -= num_replay_tokens diff --git a/vllm/v1/core/single_type_kv_cache_manager.py b/vllm/v1/core/single_type_kv_cache_manager.py index 7f23a886ed..6e53d3a94b 100644 --- a/vllm/v1/core/single_type_kv_cache_manager.py +++ b/vllm/v1/core/single_type_kv_cache_manager.py @@ -876,7 +876,14 @@ class FullAttentionManager(SingleTypeKVCacheManager): block are intentionally skipped. """ hash_block_size = self.block_pool.hash_block_size - boundary_tokens = request.num_prompt_tokens // hash_block_size * hash_block_size + # A resend matches at most `prompt_len - 1` tokens, but EAGLE lookups + # read the whole prompt before dropping a hash unit. + token_limit = ( + request.num_prompt_tokens + if self.use_eagle + else request.num_prompt_tokens - 1 + ) + boundary_tokens = token_limit // hash_block_size * hash_block_size if boundary_tokens == 0 or boundary_tokens > num_tokens: return if boundary_tokens % self.block_size == 0: @@ -1797,7 +1804,7 @@ class MambaManager(SingleTypeKVCacheManager): # is kept, so `prefill_end` skips reserving blocks for the others. checkpoint_position = get_mamba_prefill_checkpoint_position( num_tokens, - self.block_pool.hash_block_size, + self.cache_hit_alignment_tokens, self.drop_eagle_checkpoint_block, ) if not self._needs_internal_checkpoint( @@ -1821,7 +1828,7 @@ class MambaManager(SingleTypeKVCacheManager): if num_new_blocks > 0: blocks_allocated = request_id in self._allocated_block_reqs physical_block_cap = 1 + int(has_partial_hit) + checkpoint_block - if not blocks_allocated or checkpoint_block: + if not blocks_allocated: physical_block_cap += self.num_speculative_blocks num_new_blocks = min(num_new_blocks, physical_block_cap) @@ -1894,12 +1901,14 @@ class MambaManager(SingleTypeKVCacheManager): [self._null_block for _ in range(prev_block_len, null_end)] ) - if blocks_allocated and not checkpoint_block: - # Relocate exclusively owned speculative scratch blocks. + if blocks_allocated: + # Relocate exclusively owned scratch blocks this step leaves + # behind. In a checkpoint step, one at the checkpoint index + # stays: the worker exports the checkpoint into that column. for block_idx in range( prev_block_len - self.num_speculative_blocks, prev_block_len ): - if block_idx < num_skipped_blocks: + if block_idx < num_skipped_blocks - checkpoint_block: self._relocate_speculative_block(req_blocks, block_idx) else: break @@ -1907,7 +1916,7 @@ class MambaManager(SingleTypeKVCacheManager): if has_partial_hit: num_new_blocks = max(num_new_blocks, 0) + 1 max_new_blocks = 1 + int(has_partial_hit) + checkpoint_block - if not blocks_allocated or checkpoint_block: + if not blocks_allocated: max_new_blocks += self.num_speculative_blocks assert num_new_blocks <= max_new_blocks new_blocks = self.block_pool.get_new_blocks(num_new_blocks) @@ -2080,20 +2089,29 @@ class MambaManager(SingleTypeKVCacheManager): checkpoint_position == get_mamba_prefill_checkpoint_position( request.num_prompt_tokens, - hash_block_size, + self.cache_hit_alignment_tokens, self.drop_eagle_checkpoint_block, ) ) + is_retained_checkpoint = False + if checkpoint_position % self.block_size == 0: + block_idx = checkpoint_position // self.block_size - 1 + retained = self.reachable_block_mask( + block_idx, + block_idx + 1, + self.cache_hit_alignment_tokens, + self.kv_cache_spec, + self.drop_eagle_checkpoint_block, + retention_interval=retention_interval, + ) + is_retained_checkpoint = retained is None or retained[0] if ( - retention_interval == 0 - and num_tokens < request.num_prompt_tokens - and not is_prompt_checkpoint + not is_prompt_checkpoint + and not is_retained_checkpoint and checkpoint_position != request.shared_prefix_boundary ): - # retention_interval == 0 keeps this transient checkpoint - # request-local. The slot may carry a hash from this step's - # full-block pass; that must go too, since the checkpoint - # state is about to overwrite the block. + # The transient state overwrites this slot, so neither it nor + # the full-block hash registered earlier this step is reusable. self.block_pool._maybe_evict_cached_block(checkpoint_block) return None if checkpoint_block.block_hash_num_tokens == checkpoint_position: @@ -2106,21 +2124,13 @@ class MambaManager(SingleTypeKVCacheManager): block_size=self.block_size, replace_existing_hashes=True, ) - if self.block_size == hash_block_size: + if num_tokens <= 0 or num_tokens % hash_block_size != 0: return None - if num_tokens % self.block_size == 0: - return None - if num_tokens % hash_block_size != 0: - return None - latest_prompt_hash_boundary = ( - request.num_prompt_tokens // hash_block_size - ) * hash_block_size - if self.drop_eagle_checkpoint_block: - # Eagle groups match one hash unit past the candidate and drop it, - # so register the tail one unit lower. - latest_prompt_hash_boundary = max( - latest_prompt_hash_boundary - hash_block_size, 0 - ) + latest_prompt_hash_boundary = get_mamba_prefill_checkpoint_position( + request.num_prompt_tokens, + hash_block_size, + self.drop_eagle_checkpoint_block, + ) # The junction is the other position a sibling resumes at: where one was # observed to stop, and where the scheduler already ends a chunk. Bounded # to the prompt chunk being computed -- during decode the target is the @@ -2133,7 +2143,7 @@ class MambaManager(SingleTypeKVCacheManager): ): return None - block_idx = num_tokens // self.block_size + block_idx = cdiv(num_tokens, self.block_size) - 1 blocks = self.req_to_blocks[request.request_id] if block_idx >= len(blocks): return None @@ -2148,7 +2158,7 @@ class MambaManager(SingleTypeKVCacheManager): kv_cache_group_id=self.kv_cache_group_id, block_size=self.block_size, ) - if partial_hash is not None: + if partial_hash is not None and num_tokens % self.block_size: self._partial_hit_reqs[request.request_id] = (block_idx, source_block) self.num_cached_block[request.request_id] = block_idx # Producer of this partial tail: the boundary state currently lives diff --git a/vllm/v1/engine/async_llm.py b/vllm/v1/engine/async_llm.py index 4a7a91cb8e..7e50aa0451 100644 --- a/vllm/v1/engine/async_llm.py +++ b/vllm/v1/engine/async_llm.py @@ -13,6 +13,10 @@ import vllm.envs as envs from vllm import TokensPrompt from vllm.config import VllmConfig from vllm.config.kv_events import KVEventsConfig +from vllm.config.profiler import ( + validate_profile_iteration_bounds, + validate_profile_prefix, +) from vllm.distributed.weight_transfer.base import ( WeightTransferInitRequest, WeightTransferUpdateRequest, @@ -23,6 +27,7 @@ from vllm.entrypoints.serve.elastic_ep.middleware import set_scaling_elastic_ep from vllm.exceptions import ( GracefulHTTPError, MaxQueuedTokensError, + ProfilerAlreadyActiveError, QueueOverflowError, VLLMClientError, VLLMValidationError, @@ -33,7 +38,7 @@ from vllm.lora.request import LoRARequest from vllm.multimodal import MULTIMODAL_REGISTRY, MultiModalRegistry from vllm.outputs import STREAM_FINISHED, PoolingRequestOutput, RequestOutput from vllm.pooling_params import PoolingParams -from vllm.profiler.wrapper import TorchProfilerWrapper +from vllm.profiler.wrapper import TorchProfilerWrapper, create_frontend_profiler from vllm.renderers import renderer_from_config from vllm.renderers.inputs.preprocess import extract_prompt_components from vllm.sampling_params import RequestOutputKind, SamplingParams @@ -211,24 +216,27 @@ class AsyncLLM(EngineClient): pass self.profiler = profiler - configured_activities = vllm_config.profiler_config.torch_profiler_activities + self._frontend_profiler_injected = profiler is not None + self._profile_lock = asyncio.Lock() + self._profile_session_guard_enabled = client_count == 1 + self._profile_session_active = False if ( - vllm_config.profiler_config.profiler == "torch" - and not vllm_config.profiler_config.ignore_frontend - and (configured_activities is None or "CPU" in configured_activities) + not self._frontend_profiler_injected + and vllm_config.profiler_config.should_profile_frontend ): - profiler_dir = vllm_config.profiler_config.torch_profiler_dir - logger.info( - "Torch profiler enabled. AsyncLLM CPU traces will be collected under %s", # noqa: E501 - profiler_dir, - ) - worker_name = f"{socket.gethostname()}_{os.getpid()}.async_llm" - self.profiler = TorchProfilerWrapper( - vllm_config.profiler_config, - worker_name=worker_name, - local_rank=0, - activities=["CPU"], - ) + if self._profile_session_guard_enabled: + self._frontend_profiler_worker_name = ( + f"{socket.gethostname()}_{os.getpid()}.async_llm" + ) + self.profiler = create_frontend_profiler( + vllm_config.profiler_config, + worker_name=self._frontend_profiler_worker_name, + ) + else: + logger.warning( + "Frontend CPU profiling is disabled when multiple API server " + "processes share an engine." + ) @classmethod def from_vllm_config( @@ -443,6 +451,13 @@ class AsyncLLM(EngineClient): "does not match the EngineCoreRequest.request_id attribute. The " "latter will be used, and the former will be ignored." ) + request_params = request.params + if isinstance(request_params, SamplingParams): + # This request object is owned by the engine from here on. + self.input_processor.apply_watermarking( + request_params, + self.input_processor.resolve_watermarking(request_params), + ) else: if isinstance(prompt, dict) and "type" in prompt: # Rendered EngineInput; no blocking preprocessing needed. @@ -910,6 +925,7 @@ class AsyncLLM(EngineClient): sampling_params=SamplingParams( max_tokens=1, extra_args={"kv_transfer_params": dict(kv_transfer_params)}, + watermarking=False, ), pooling_params=None, arrival_time=time.time(), @@ -1073,17 +1089,48 @@ class AsyncLLM(EngineClient): if self.errored: raise self.dead_error - async def start_profile(self, profile_prefix: str | None = None) -> None: - coros = [self.engine_core.profile_async(True, profile_prefix)] - if self.profiler is not None: - coros.append(asyncio.to_thread(self.profiler.start)) - await asyncio.gather(*coros) + async def start_profile( + self, + profile_prefix: str | None = None, + *, + delay_iterations: int | None = None, + max_iterations: int | None = None, + ) -> None: + async with self._profile_lock: + if self._profile_session_guard_enabled and self._profile_session_active: + raise ProfilerAlreadyActiveError() + validate_profile_prefix(profile_prefix) + validate_profile_iteration_bounds(delay_iterations, max_iterations) + + if self._profile_session_guard_enabled: + self._profile_session_active = True + # A cancelled start may still take effect, so only a reported + # engine failure clears the session; otherwise /stop_profile is needed. + try: + await self.engine_core.profile_async( + True, + profile_prefix, + delay_iterations, + max_iterations, + ) + except Exception: + self._profile_session_active = False + raise + if self.profiler is not None: + if not self._frontend_profiler_injected: + worker_name = self._frontend_profiler_worker_name + if profile_prefix is not None: + worker_name = f"{profile_prefix}_{worker_name}" + self.profiler.set_output_name(worker_name) + await asyncio.to_thread(self.profiler.start) async def stop_profile(self) -> None: - coros = [self.engine_core.profile_async(False)] - if self.profiler is not None: - coros.append(asyncio.to_thread(self.profiler.stop)) - await asyncio.gather(*coros) + async with self._profile_lock: + coros = [self.engine_core.profile_async(False)] + if self.profiler is not None: + coros.append(asyncio.to_thread(self.profiler.stop)) + await asyncio.gather(*coros) + self._profile_session_active = False async def reset_mm_cache(self) -> None: # Join the background MM warmup first: the mm_processor_cache is not diff --git a/vllm/v1/engine/core.py b/vllm/v1/engine/core.py index cc0a8beddd..bfbaa38d94 100644 --- a/vllm/v1/engine/core.py +++ b/vllm/v1/engine/core.py @@ -53,6 +53,7 @@ from vllm.v1.core.kv_cache_utils import ( get_kv_cache_configs, get_request_block_hasher, init_none_hash, + resolve_cache_hit_alignment_tokens, resolve_kv_cache_block_sizes, update_kv_cache_capacity, ) @@ -352,6 +353,19 @@ class EngineCore: vllm_config.validate_block_size() + scheduler_block_size, hash_block_size = resolve_kv_cache_block_sizes( + scheduler_kv_cache_config, vllm_config + ) + cache_hit_alignment_tokens = resolve_cache_hit_alignment_tokens( + scheduler_kv_cache_config, + vllm_config, + scheduler_block_size, + hash_block_size, + ) + for kv_cache_config in kv_cache_configs: + kv_cache_config.hash_block_size = hash_block_size + kv_cache_config.cache_hit_alignment_tokens = cache_hit_alignment_tokens + self.model_executor.initialize_from_config(kv_cache_configs) if not envs.VLLM_ELASTIC_EP_SCALE_UP_LAUNCH: self.model_executor.compile_or_warm_up_model() @@ -810,8 +824,19 @@ class EngineCore: cleanup_dist_env_and_memory() logger.debug_once("[shutdown] EngineCore: local resource teardown complete") - def profile(self, is_start: bool = True, profile_prefix: str | None = None): - self.model_executor.profile(is_start, profile_prefix) + def profile( + self, + is_start: bool = True, + profile_prefix: str | None = None, + delay_iterations: int | None = None, + max_iterations: int | None = None, + ): + self.model_executor.profile( + is_start, + profile_prefix, + delay_iterations=delay_iterations, + max_iterations=max_iterations, + ) def reset_mm_cache(self): # NOTE: Since this is mainly for debugging, we don't attempt to @@ -1385,6 +1410,50 @@ class EngineCoreProc(EngineCore): vllm_config.kv_transfer_config.engine_id, ) + # declare engine core as None and register signal handler before + # engine_core initialization in case engine core process is terminated + # in the middle of initialization. For example, frontend process exits + # unexpectedly, if signal handler is not registered, engine core process + # will use default signal handler which will terminate the process + # silently. If engine core model executor has been initialized, it will + # not be terminated asap. Register signal handler early and only log a + # message will make engine core process alive. engine core process and + # its subprocesses(model executor processes) will be force killed in + # CoreEngineProcManager.shutdown. + engine_core = None + # A termination signal can arrive while engine_core is still being + # constructed. At that point the request cannot be recorded on + # engine_core, so keep it in its own flag and apply it later. + shutdown_requested = False + + def wakeup_engine(): + # Wakes up idle engine via input_queue when shutdown is requested + # Not safe in a signal handler - we may interrupt the main thread + # while it is holding the non-reentrant input_queue.mutex + assert engine_core is not None + engine_core.input_queue.put_nowait((EngineCoreRequestType.WAKEUP, None)) + + def signal_handler(signum, frame): + nonlocal shutdown_requested + signal_name = signal.Signals(signum).name + logger.info( + "[shutdown] EngineCore: trigger received signal=%s", + signal_name, + ) + # Record the request first so it survives the construction window. + shutdown_requested = True + if engine_core is not None: + engine_core.shutdown_state = EngineShutdownState.REQUESTED + # signal_callback is armed only after engine_core exists. Don't + # trigger it earlier: it is one-shot, and a wakeup cannot be + # delivered while there is no input_queue to wake. + # For more info: https://github.com/vllm-project/vllm/pull/52299#pullrequestreview-5370255271 + if signal_callback is not None: + signal_callback.trigger() + + signal.signal(signal.SIGTERM, signal_handler) + signal.signal(signal.SIGINT, signal_handler) + parallel_config.data_parallel_index = dp_rank if data_parallel and vllm_config.model_config.is_moe: # Set data parallel rank for this engine process. @@ -1399,25 +1468,15 @@ class EngineCoreProc(EngineCore): assert engine_core is not None - def wakeup_engine(): - # Wakes up idle engine via input_queue when shutdown is requested - # Not safe in a signal handler - we may interrupt the main thread - # while it is holding the non-reentrant input_queue.mutex - engine_core.input_queue.put_nowait((EngineCoreRequestType.WAKEUP, None)) - - signal_callback = SignalCallback(wakeup_engine) - - def signal_handler(signum, frame): - signal_name = signal.Signals(signum).name - logger.info( - "[shutdown] EngineCore: trigger received signal=%s", - signal_name, - ) + # Apply a signal received during construction before entering the + # loop. run_busy_loop calls _handle_shutdown first, so the loop exits + # right away instead of blocking on an idle input_queue. + if shutdown_requested: engine_core.shutdown_state = EngineShutdownState.REQUESTED - signal_callback.trigger() - signal.signal(signal.SIGTERM, signal_handler) - signal.signal(signal.SIGINT, signal_handler) + # Arm the wakeup callback only now that input_queue exists, so a + # later signal can still wake an idle busy loop. + signal_callback = SignalCallback(wakeup_engine) engine_core.run_busy_loop() diff --git a/vllm/v1/engine/core_client.py b/vllm/v1/engine/core_client.py index cb6705966a..dccc8cba53 100644 --- a/vllm/v1/engine/core_client.py +++ b/vllm/v1/engine/core_client.py @@ -11,7 +11,6 @@ from collections import Counter, defaultdict from collections.abc import Awaitable, Callable, Sequence from concurrent.futures import Future from dataclasses import dataclass -from multiprocessing.connection import Connection from multiprocessing.queues import Queue from threading import Thread from typing import Any, TypeAlias, TypeVar @@ -56,7 +55,8 @@ from vllm.v1.engine.tensor_ipc import TensorIpcSender from vllm.v1.engine.utils import ( CoreEngineActorManager, CoreEngineProcManager, - get_engine_zmq_addresses, + EngineZmqAddresses, + get_engine_zmq_bind_addresses, launch_core_engines, ) from vllm.v1.executor import Executor @@ -181,7 +181,13 @@ class EngineCoreClient(ABC): def add_request(self, request: EngineCoreRequest) -> None: raise NotImplementedError - def profile(self, is_start: bool = True, profile_prefix: str | None = None) -> None: + def profile( + self, + is_start: bool = True, + profile_prefix: str | None = None, + delay_iterations: int | None = None, + max_iterations: int | None = None, + ) -> None: raise NotImplementedError def reset_mm_cache(self) -> None: @@ -275,7 +281,11 @@ class EngineCoreClient(ABC): raise NotImplementedError async def profile_async( - self, is_start: bool = True, profile_prefix: str | None = None + self, + is_start: bool = True, + profile_prefix: str | None = None, + delay_iterations: int | None = None, + max_iterations: int | None = None, ) -> None: raise NotImplementedError @@ -392,8 +402,19 @@ class InprocClient(EngineCoreClient): if self._finalizer.detach() is not None: self.engine_core.shutdown() - def profile(self, is_start: bool = True, profile_prefix: str | None = None) -> None: - self.engine_core.profile(is_start, profile_prefix) + def profile( + self, + is_start: bool = True, + profile_prefix: str | None = None, + delay_iterations: int | None = None, + max_iterations: int | None = None, + ) -> None: + self.engine_core.profile( + is_start, + profile_prefix, + delay_iterations=delay_iterations, + max_iterations=max_iterations, + ) def reset_mm_cache(self) -> None: self.engine_core.reset_mm_cache() @@ -517,10 +538,10 @@ class BackgroundResources: with contextlib.suppress(Exception): task.cancel() - if loop is not None: + if loop is not None and not loop.is_closed(): if in_loop(loop): close_sockets_and_tasks() - elif not loop.is_closed(): + else: loop.call_soon_threadsafe(close_sockets_and_tasks) else: # Loop has been closed, try to clean up directly. @@ -613,63 +634,50 @@ class MPClient(EngineCoreClient): self.stats_update_address = client_addresses.get("stats_update_address") # Tensor queues passed via client_addresses for multi-API-server case tensor_queue = client_addresses.get("tensor_queue") + input_listener = client_addresses.get("input_listener") + output_listener = client_addresses.get("output_listener") self.input_socket = self.resources.input_socket = make_zmq_socket( self.ctx, input_address, zmq.ROUTER, bind=True, router_handover=enable_input_socket_handover, + listener=input_listener, ) self.resources.output_socket = make_zmq_socket( - self.ctx, output_address, zmq.PULL + self.ctx, + output_address, + zmq.PULL, + listener=output_listener, ) - # Report bound endpoints back so the parent can forward - # them to engines (mirrors the DPCoordinator pattern). - actual_address_pipe: Connection | None = client_addresses.get( - "actual_address_pipe" - ) - if actual_address_pipe is not None: - try: - actual_input = self.input_socket.getsockopt( - zmq.LAST_ENDPOINT - ).decode() - actual_output = self.resources.output_socket.getsockopt( - zmq.LAST_ENDPOINT - ).decode() - actual_address_pipe.send( - { - "input_address": actual_input, - "output_address": actual_output, - } - ) - finally: - actual_address_pipe.close() # Engines are managed externally: this process does not fork # them, so there is no fork race; start the MM warmup now. self._start_mm_warmup() else: # Engines are managed by this client. - addresses = get_engine_zmq_addresses(vllm_config) + bind_addresses = get_engine_zmq_bind_addresses(vllm_config) self.input_socket = self.resources.input_socket = make_zmq_socket( self.ctx, - addresses.inputs[0], + bind_addresses.inputs[0], zmq.ROUTER, bind=True, router_handover=enable_input_socket_handover, ) self.resources.output_socket = make_zmq_socket( - self.ctx, addresses.outputs[0], zmq.PULL + self.ctx, bind_addresses.outputs[0], zmq.PULL ) - # Resolve ``tcp://host:0`` placeholders to bound endpoints - # before engines DEALER-connect. No-op for IPC. - addresses.inputs[0] = self.input_socket.getsockopt( - zmq.LAST_ENDPOINT - ).decode() - addresses.outputs[0] = self.resources.output_socket.getsockopt( - zmq.LAST_ENDPOINT - ).decode() + # EngineZmqAddresses contains only endpoints resolved by the + # sockets that own the bind operation. + addresses = EngineZmqAddresses( + inputs=[self.input_socket.getsockopt(zmq.LAST_ENDPOINT).decode()], + outputs=[ + self.resources.output_socket.getsockopt( + zmq.LAST_ENDPOINT + ).decode() + ], + ) with launch_core_engines( vllm_config, executor_class, log_stats, addresses @@ -1019,8 +1027,20 @@ class SyncMPClient(MPClient): if request_ids and not self.resources.engine_dead: self._send_input(EngineCoreRequestType.ABORT, request_ids) - def profile(self, is_start: bool = True, profile_prefix: str | None = None) -> None: - self.call_utility("profile", is_start, profile_prefix) + def profile( + self, + is_start: bool = True, + profile_prefix: str | None = None, + delay_iterations: int | None = None, + max_iterations: int | None = None, + ) -> None: + self.call_utility( + "profile", + is_start, + profile_prefix, + delay_iterations, + max_iterations, + ) def reset_mm_cache(self) -> None: self.call_utility("reset_mm_cache") @@ -1294,9 +1314,19 @@ class AsyncMPClient(MPClient): return await self.call_utility_consensus_async("is_scheduler_paused") async def profile_async( - self, is_start: bool = True, profile_prefix: str | None = None + self, + is_start: bool = True, + profile_prefix: str | None = None, + delay_iterations: int | None = None, + max_iterations: int | None = None, ) -> None: - await self.call_utility_async("profile", is_start, profile_prefix) + await self.call_utility_async( + "profile", + is_start, + profile_prefix, + delay_iterations, + max_iterations, + ) async def reset_mm_cache_async(self) -> None: await self.call_utility_async("reset_mm_cache") diff --git a/vllm/v1/engine/input_processor.py b/vllm/v1/engine/input_processor.py index 8b5a69bcf4..17e8af5cae 100644 --- a/vllm/v1/engine/input_processor.py +++ b/vllm/v1/engine/input_processor.py @@ -25,7 +25,7 @@ from vllm.platforms import current_platform from vllm.pooling_params import PoolingParams from vllm.renderers import BaseRenderer, renderer_from_config from vllm.renderers.inputs.preprocess import parse_model_prompt -from vllm.sampling_params import SamplingParams +from vllm.sampling_params import BeamSearchParams, SamplingParams from vllm.tasks import GENERATION_TASKS, POOLING_TASKS, SupportedTask from vllm.tokenizers import TokenizerLike from vllm.utils import length_from_prompt_token_ids_or_embeds, random_uuid @@ -112,12 +112,41 @@ class InputProcessor: return partial(validate_logits_processors_parameters, custom_logitsprocs) + def resolve_watermarking(self, params: SamplingParams | BeamSearchParams) -> bool: + """Return whether watermarking will be applied to this request. + + ``None`` and ``True`` request watermarking, while ``False`` opts out. + If watermarking is unavailable or incompatible with the request, emit + a warning and return ``False``. + """ + return self.vllm_config._check_supports_watermarking(params) + + def apply_watermarking(self, params: SamplingParams, watermarking: bool) -> None: + """Store the resolved `watermarking` on request-local params. + + Also records whether watermarking was requested on a watermarking + engine but could not be applied, as opposed to an explicit opt-out. + """ + # Sticky, so resolving an already resolved request keeps the flag. + params._watermarking_skipped = params._watermarking_skipped or ( + self.vllm_config.watermark_config is not None + and params.watermarking is not False + and not watermarking + ) + params.watermarking = watermarking + def _validate_params( self, params: SamplingParams | PoolingParams, supported_tasks: tuple[SupportedTask, ...], - ) -> None: - """Raise `ValueError` if SamplingParams or PoolingParams is not valid.""" + ) -> bool | None: + """Raise `ValueError` if SamplingParams or PoolingParams is not valid. + + Returns the resolved `watermarking` value for `SamplingParams` (`None` + for `PoolingParams`). The caller applies it to its private copy of the + params so that a caller-owned params object is never mutated. + """ + watermarking: bool | None = None if isinstance(params, SamplingParams): supported_generation_tasks = [ task for task in supported_tasks if task in GENERATION_TASKS @@ -189,6 +218,9 @@ class InputProcessor: "enabled. Start the engine with --enable-trace-replay " "to use it." ) + # Resolved last so that feature gates above report the missing + # feature rather than its incompatibility with watermarking. + watermarking = self.resolve_watermarking(params) elif isinstance(params, PoolingParams): supported_pooling_tasks = [ task for task in supported_tasks if task in POOLING_TASKS @@ -217,6 +249,8 @@ class InputProcessor: f"but got {type(params).__name__}" ) + return watermarking + def _normalize_trace_replay_params( self, sampling_params: SamplingParams, prompt_len: int ) -> None: @@ -352,7 +386,7 @@ class InputProcessor: session_id: str | None = None, kv_hints: KvHintsEnvelope | None = None, ) -> EngineCoreRequest: - self._validate_params(params, supported_tasks) + watermarking = self._validate_params(params, supported_tasks) self._validate_lora(lora_request) parallel_config = self.vllm_config.parallel_config @@ -413,6 +447,10 @@ class InputProcessor: if isinstance(params, SamplingParams): # TODO: can we avoid cloning here in multiproc case? sampling_params = params.clone() + # Resolve on the request-local copy: `params` may be shared across + # prompts, requests and even engines by the caller. + assert watermarking is not None + self.apply_watermarking(sampling_params, watermarking) prompt_len = length_from_prompt_token_ids_or_embeds( prompt_token_ids, prompt_embeds ) diff --git a/vllm/v1/engine/llm_engine.py b/vllm/v1/engine/llm_engine.py index c3fb3efe7e..2dd70fc179 100644 --- a/vllm/v1/engine/llm_engine.py +++ b/vllm/v1/engine/llm_engine.py @@ -12,9 +12,14 @@ from typing_extensions import TypeVar import vllm.envs as envs from vllm.config import ParallelConfig, VllmConfig +from vllm.config.profiler import ( + validate_profile_iteration_bounds, + validate_profile_prefix, +) from vllm.distributed import stateless_destroy_torch_distributed_process_group from vllm.distributed.parallel_state import get_dp_group from vllm.engine.arg_utils import EngineArgs +from vllm.exceptions import ProfilerAlreadyActiveError from vllm.inputs import EngineInput, PromptType from vllm.logger import configure_logging_if_needed, init_logger from vllm.lora.request import LoRARequest @@ -116,6 +121,7 @@ class LLMEngine: log_stats=self.log_stats, renderer=renderer, ) + self._profile_session_active = False self.logger_manager: StatLoggerManager | None = None if self.log_stats: @@ -254,6 +260,13 @@ class LLMEngine: "does not match the EngineCoreRequest.request_id attribute. The " "latter will be used, and the former will be ignored." ) + request_params = request.params + if isinstance(request_params, SamplingParams): + # This request object is owned by the engine from here on. + self.input_processor.apply_watermarking( + request_params, + self.input_processor.resolve_watermarking(request_params), + ) else: request = self.input_processor.process_inputs( request_id, @@ -351,11 +364,33 @@ class LLMEngine: return processed_outputs.request_outputs - def start_profile(self, profile_prefix: str | None = None): - self.engine_core.profile(True, profile_prefix) + def start_profile( + self, + profile_prefix: str | None = None, + *, + delay_iterations: int | None = None, + max_iterations: int | None = None, + ): + if self._profile_session_active: + raise ProfilerAlreadyActiveError() + validate_profile_prefix(profile_prefix) + validate_profile_iteration_bounds(delay_iterations, max_iterations) + + self._profile_session_active = True + try: + self.engine_core.profile( + True, + profile_prefix, + delay_iterations, + max_iterations, + ) + except BaseException: + self._profile_session_active = False + raise def stop_profile(self): self.engine_core.profile(False) + self._profile_session_active = False def reset_mm_cache(self): # Join the background MM warmup first: the mm_processor_cache is not diff --git a/vllm/v1/engine/utils.py b/vllm/v1/engine/utils.py index 203d1b407c..4f1be3bacd 100644 --- a/vllm/v1/engine/utils.py +++ b/vllm/v1/engine/utils.py @@ -23,9 +23,10 @@ from vllm.platforms import current_platform from vllm.ray.ray_env import get_env_vars_to_copy from vllm.utils import numa_utils from vllm.utils.network_utils import ( - get_open_port, + ZmqListener, get_open_zmq_ipc_path, get_tcp_uri, + make_zmq_listener, zmq_socket_ctx, ) from vllm.utils.system_utils import get_mp_context, set_env_var @@ -82,8 +83,18 @@ class CoreEngine: self.state = CoreEngineState.NEW +@dataclass +class EngineZmqBindAddresses: + """Frontend endpoints before their owning sockets bind.""" + + inputs: list[str] + outputs: list[str] + + @dataclass class EngineZmqAddresses: + """Resolved frontend endpoints advertised to engine processes.""" + # ZMQ input socket addresses for each front-end client (requests) inputs: list[str] # ZMQ output socket addresses for each front-end client (responses) @@ -98,6 +109,27 @@ class EngineZmqAddresses: frontend_stats_publish_address: str | None = None +@dataclass +class EngineZmqListeners: + inputs: list[ZmqListener] + outputs: list[ZmqListener] + + @property + def addresses(self) -> EngineZmqAddresses: + return EngineZmqAddresses( + inputs=[listener.address for listener in self.inputs], + outputs=[listener.address for listener in self.outputs], + ) + + def cleanup(self) -> None: + for listener in (*self.inputs, *self.outputs): + listener.cleanup() + + def __del__(self) -> None: + with contextlib.suppress(Exception): + self.cleanup() + + @dataclass class EngineHandshakeMetadata: """Metadata sent to each engine process during startup handshake, @@ -1079,22 +1111,16 @@ class CoreEngineActorManager: ray.util.remove_placement_group(pg) -def get_engine_zmq_addresses( +def get_engine_zmq_bind_addresses( vllm_config: VllmConfig, num_api_servers: int = 1, - *, - defer_api_server_ports: bool = True, -) -> EngineZmqAddresses: - """Allocate ZMQ addresses for engine-client communication. +) -> EngineZmqBindAddresses: + """Create ZMQ endpoint templates for engine-client communication. - By default each TCP address is a ``tcp://host:0`` placeholder; the - consumer (API-server child or single-process ``MPClient``) binds, then - recovers the kernel-assigned port via ``getsockopt(zmq.LAST_ENDPOINT)`` - and writes it back into ``addresses`` before the engine handshake. - - Set ``defer_api_server_ports=False`` only when the consumer cannot - report a bound port back (e.g. the Rust front-end). IPC paths are - unaffected.""" + Each TCP address is a ``tcp://host:0`` placeholder. A single-process + ``MPClient`` binds its own ZMQ sockets. Cross-process frontends use + ``bind_engine_zmq_listeners`` so the supervisor binds raw listeners before + launching engines and transfers listener ownership to each frontend.""" parallel_config = vllm_config.parallel_config local_engine_count = parallel_config.data_parallel_size_local local_start_index = parallel_config.data_parallel_rank_local @@ -1119,14 +1145,39 @@ def get_engine_zmq_addresses( def _addr() -> str: if client_local_only: return get_open_zmq_ipc_path() - return get_tcp_uri(host, 0 if defer_api_server_ports else get_open_port()) + return get_tcp_uri(host, 0) - return EngineZmqAddresses( + return EngineZmqBindAddresses( inputs=[_addr() for _ in range(num_api_servers)], outputs=[_addr() for _ in range(num_api_servers)], ) +def bind_engine_zmq_listeners( + vllm_config: VllmConfig, + num_api_servers: int = 1, +) -> EngineZmqListeners: + """Bind frontend transport listeners before launching engines.""" + bind_addresses = get_engine_zmq_bind_addresses( + vllm_config, + num_api_servers, + ) + inputs: list[ZmqListener] = [] + outputs: list[ZmqListener] = [] + try: + inputs.extend( + make_zmq_listener(address, zmq.ROUTER) for address in bind_addresses.inputs + ) + outputs.extend( + make_zmq_listener(address, zmq.PULL) for address in bind_addresses.outputs + ) + return EngineZmqListeners(inputs=inputs, outputs=outputs) + except BaseException: + for listener in (*inputs, *outputs): + listener.cleanup() + raise + + FrontendProcess = BaseProcess | _SubprocessWrapper @@ -1195,6 +1246,23 @@ def launch_core_engines( else: coordinator = None + # Hold a coordination TCPStore (alive for this frame) so engines pick DP + # master ports at bind time; pre-allocated ports can be taken before use. + coord_store = None + if ( + dp_size > 1 + and not offline_mode + and dp_rank == 0 + and local_engine_count > 0 + and not parallel_config.enable_elastic_ep + ): + from vllm.distributed.utils import create_tcp_store + + coord_store = create_tcp_store( + host, 0, is_master=True, world_size=-1, wait_for_workers=False + ) + parallel_config._coord_store_port = coord_store.port + if parallel_config.data_parallel_backend == "ray": logger.info("Starting ray-based data parallel backend") @@ -1415,6 +1483,7 @@ def wait_for_engine_startup( "data_parallel_master_ip", "data_parallel_master_port", "_data_parallel_master_port_list", + "_coord_store_port", "data_parallel_size", ) } diff --git a/vllm/v1/executor/abstract.py b/vllm/v1/executor/abstract.py index f234e943b4..0af9508662 100644 --- a/vllm/v1/executor/abstract.py +++ b/vllm/v1/executor/abstract.py @@ -1,5 +1,6 @@ # SPDX-License-Identifier: Apache-2.0 # SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import inspect import time from abc import ABC, abstractmethod from collections.abc import Callable @@ -38,6 +39,18 @@ SLEEP_TAGS = frozenset(("weights", "kv_cache")) FailureCallback = Callable[[], None] +def _check_worker_profile_kwargs(worker_cls: str, kwargs: dict[str, int]) -> None: + """Raise if the worker's ``profile`` cannot accept the given overrides.""" + params = inspect.signature(resolve_obj_by_qualname(worker_cls).profile).parameters + if any(p.kind is inspect.Parameter.VAR_KEYWORD for p in params.values()): + return + if unsupported := [name for name in kwargs if name not in params]: + raise ValueError( + f"Worker class {worker_cls} does not support per-session profiling " + f"overrides: {', '.join(unsupported)}" + ) + + class Executor(ABC): """Abstract base class for vLLM executors. @@ -269,8 +282,30 @@ class Executor(ABC): output: list[DraftTokenIds] = self.collective_rpc("take_draft_token_ids") return output[0] - def profile(self, is_start: bool = True, profile_prefix: str | None = None): - self.collective_rpc("profile", args=(is_start, profile_prefix)) + def profile( + self, + is_start: bool = True, + profile_prefix: str | None = None, + *, + delay_iterations: int | None = None, + max_iterations: int | None = None, + ): + kwargs: dict[str, int] = {} + if delay_iterations is not None: + kwargs["delay_iterations"] = delay_iterations + if max_iterations is not None: + kwargs["max_iterations"] = max_iterations + if kwargs: + # Reject before dispatch: a failing collective_rpc may leave + # unread worker replies queued. + _check_worker_profile_kwargs( + self.vllm_config.parallel_config.worker_cls, kwargs + ) + self.collective_rpc( + "profile", + args=(is_start, profile_prefix), + kwargs=kwargs or None, + ) def save_sharded_state( self, diff --git a/vllm/v1/executor/multiproc_executor.py b/vllm/v1/executor/multiproc_executor.py index 78290d25ba..cad3436854 100644 --- a/vllm/v1/executor/multiproc_executor.py +++ b/vllm/v1/executor/multiproc_executor.py @@ -364,7 +364,11 @@ class MultiprocExecutor(Executor): ) def execute_dummy_batch(self) -> None: - self.collective_rpc("execute_dummy_batch", unique_reply_rank=self.output_rank) + self.collective_rpc( + "execute_dummy_batch", + unique_reply_rank=self.output_rank, + timeout=envs.VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS, + ) def take_draft_token_ids(self) -> DraftTokenIds | None: # OPTIMIZATION: Get output only from a single worker (output_rank) diff --git a/vllm/v1/hisparse/binding.py b/vllm/v1/hisparse/binding.py index d021d8b7be..b8c0d5dff2 100644 --- a/vllm/v1/hisparse/binding.py +++ b/vllm/v1/hisparse/binding.py @@ -147,7 +147,6 @@ def init_hisparse_kv_cache( initialize_hisparse_runtime_buffers( cache_handles, max_num_reqs=vllm_config.scheduler_config.max_num_seqs, - max_num_batched_tokens=vllm_config.scheduler_config.max_num_batched_tokens, ) return kv_caches except Exception: @@ -197,9 +196,6 @@ def release_hisparse_profiling_cache(forward_context: dict[str, Any]) -> None: ) ) release_pinned_state(list(runtimes.values()), registered_pools, shared_region) - for cache in cache_handles: - cache.mirror_staging_cache = None - cache.mirror_staging_slots = None def bind_hisparse_kv_caches( diff --git a/vllm/v1/hisparse/coordinator.py b/vllm/v1/hisparse/coordinator.py index bc9bff1a8b..390e0d27ce 100644 --- a/vllm/v1/hisparse/coordinator.py +++ b/vllm/v1/hisparse/coordinator.py @@ -19,6 +19,7 @@ from vllm.v1.core.single_type_kv_cache_manager import ( SingleTypeKVCacheManager, ) from vllm.v1.hisparse.types import ( + ACTIVE_TAIL_PAGES, SparseKVOffloadCommand, SparseKVPageTransfer, SparseKVRowMirror, @@ -32,10 +33,6 @@ from vllm.v1.request import Request if TYPE_CHECKING: from vllm.v1.core.kv_cache_manager import KVCacheManager -# Sealed pages this many positions behind the block-table tail stay pinned so -# a page written by an in-flight step is never handed out under it. -_ACTIVE_TAIL_PAGES = 2 - @dataclass class _PendingPublication: @@ -171,6 +168,10 @@ class HiSparseCoordinator: # request -> prefix length an external load is still filling in. self._pending_imports: dict[str, int] = {} self.block_table_updates: set[str] = set() + # Requests whose resident pages were all non-null when last scanned. + # Pages only become null at admission or import (before the first + # scan) and when the pool reuses an unpinned page (_lose_page). + self._resident_requests: set[str] = set() self.spills_to_send: list[SparseKVPageTransfer] = [] self.pending_spills: dict[int, _PendingSpill] = {} self.request_states: dict[str, _HiSparseRequestState] = {} @@ -191,6 +192,7 @@ class HiSparseCoordinator: def commit_computed_blocks(self, request_id: str, num_host_pages: int) -> None: """Account for a prefix hit on host pages the request now references.""" + self._resident_requests.discard(request_id) if not self.resident_managers or self.host_manager is None: return host_blocks = self.host_manager.req_to_blocks.get(request_id, ()) @@ -343,7 +345,7 @@ class HiSparseCoordinator: req_blocks = manager.req_to_blocks.get(request_id) if ( req_blocks is None - or page_idx >= len(req_blocks) - _ACTIVE_TAIL_PAGES + or page_idx >= len(req_blocks) - ACTIVE_TAIL_PAGES or req_blocks[page_idx].is_null ): return None @@ -385,6 +387,7 @@ class HiSparseCoordinator: self._lose_page(request_id, page_idx) def _lose_page(self, request_id: str, page_idx: int) -> None: + self._resident_requests.discard(request_id) state = self.request_states.get(request_id) if state is None: return @@ -533,6 +536,7 @@ class HiSparseCoordinator: def record_pending_host_import(self, request_id: str, num_tokens: int) -> None: """Note a prefix an external load is populating in host pages.""" + self._resident_requests.discard(request_id) self._pending_imports[request_id] = num_tokens def finish_host_import(self, request_id: str, *, failed: bool) -> None: @@ -677,15 +681,22 @@ class HiSparseCoordinator: """Return whether every scheduled request can read only resident KV.""" if not self.resident_managers: return False - block_size = self.resident_managers[0].block_size + first = self.resident_managers[0] for request_id, num_computed_tokens, num_scheduled_tokens in requests: - num_pages = cdiv(num_computed_tokens + num_scheduled_tokens, block_size) - for page_idx in range(num_pages): - if any( - manager.get_resident_page(request_id, page_idx) is None - for manager in self.resident_managers - ): - return False + num_pages = cdiv( + num_computed_tokens + num_scheduled_tokens, first.block_size + ) + if len(first.req_to_blocks.get(request_id, ())) < num_pages: + return False + if request_id in self._resident_requests: + continue + if any( + block.is_null + for manager in self.resident_managers + for block in manager.req_to_blocks.get(request_id, ()) + ): + return False + self._resident_requests.add(request_id) return True def take_block_table_updates(self) -> dict[str, tuple[list[int], ...]]: @@ -809,6 +820,7 @@ class HiSparseCoordinator: def free(self, request_id: str) -> None: """Detach the request; its clean pages stay readable copies in the pool.""" + self._resident_requests.discard(request_id) self._pending_imports.pop(request_id, None) state = self.request_states.pop(request_id, None) if state is None: diff --git a/vllm/v1/hisparse/layout.py b/vllm/v1/hisparse/layout.py index 8ebc63c0ed..98636866f8 100644 --- a/vllm/v1/hisparse/layout.py +++ b/vllm/v1/hisparse/layout.py @@ -14,6 +14,7 @@ from vllm.v1.hisparse.runtime import ( get_hisparse_host_block_stride, use_shared_hisparse_host_pool, ) +from vllm.v1.hisparse.types import ACTIVE_TAIL_PAGES from vllm.v1.kv_cache_interface import ( HiSparseHotSpec, HiSparseResidentSpec, @@ -373,3 +374,34 @@ def get_hisparse_kv_cache_config( vllm_config.cache_config.prefix_cache_retention_interval ), ) + + +def get_hisparse_steady_state_concurrency( + vllm_config: VllmConfig, kv_cache_config: KVCacheConfig +) -> float: + """Max concurrency at max_model_len once running requests read from host. + + A request reading from host pins only its active tail of each resident + group, but admitting one still takes its full in-flight window, so the + bound is all-but-one requests at steady state plus one being admitted. + """ + admission_blocks = 0 + steady_blocks = 0 + host_blocks = 0 + for group in kv_cache_config.kv_cache_groups: + spec = group.kv_cache_spec + required = cdiv(spec.max_memory_usage_bytes(vllm_config), spec.page_size_bytes) + if group.host_resident: + host_blocks += required + continue + admission_blocks += required + if isinstance(spec, HiSparseResidentSpec): + required = min(required, ACTIVE_TAIL_PAGES) + steady_blocks += required + assert kv_cache_config.hisparse_host_num_blocks is not None + num_blocks = kv_cache_config.num_blocks + if num_blocks < admission_blocks: + gpu_concurrency = num_blocks / admission_blocks + else: + gpu_concurrency = 1 + (num_blocks - admission_blocks) / steady_blocks + return min(gpu_concurrency, kv_cache_config.hisparse_host_num_blocks / host_blocks) diff --git a/vllm/v1/hisparse/runtime.py b/vllm/v1/hisparse/runtime.py index fc415fa2ec..1f7a5ad5ca 100644 --- a/vllm/v1/hisparse/runtime.py +++ b/vllm/v1/hisparse/runtime.py @@ -894,14 +894,17 @@ class HiSparseRuntime: def begin_forward(self) -> None: self._swap_step = 0 + @property + def max_swap_rows(self) -> int: + return self.index_group.shared_topk.physical_topk_indices.shape[0] + def _step_rows(self, num_tokens: int) -> slice: start = self._swap_step * num_tokens stop = start + num_tokens - if stop > self.index_group.shared_topk.physical_topk_indices.shape[0]: + if stop > self.max_swap_rows: raise ValueError( "HiSparse swap rows exceed the configured speculative decode " - f"capacity: stop={stop}, capacity=" - f"{self.index_group.shared_topk.physical_topk_indices.shape[0]}." + f"capacity: stop={stop}, capacity={self.max_swap_rows}." ) self._swap_step += 1 return slice(start, stop) @@ -1120,10 +1123,7 @@ class HiSparseCacheHandle: self.num_decode_tokens = 0 self.req_id_per_token: torch.Tensor | None = None self.host_mirror_required = False - self.mirror_from_resident = False self.mirror_slot_mapping: torch.Tensor | None = None - self.mirror_staging_cache: torch.Tensor | None = None - self.mirror_staging_slots: torch.Tensor | None = None self.submit_layer_mirror: Callable[[], None] | None = None # Speculator layers write their rows after the target forward. self.draft_layer = False @@ -1160,26 +1160,6 @@ class HiSparseCacheHandle: num_rows = min(num_rows, self.runtime.max_num_reqs) return self.view.cache, self.slot_mapping[:num_rows], num_rows - def mirror_write_target( - self, num_rows: int - ) -> tuple[torch.Tensor, torch.Tensor] | None: - if ( - not self.host_mirror_required - or self.decode_batch - or self.mirror_from_resident - ): - return None - cache = self.mirror_staging_cache - slots = self.mirror_staging_slots - if cache is None or slots is None: - raise RuntimeError("HiSparse prefill mirror staging is not bound.") - if num_rows > slots.numel(): - raise RuntimeError( - "HiSparse prefill mirror exceeds staging capacity: " - f"{num_rows} > {slots.numel()}." - ) - return cache, slots[:num_rows] - def finish_kv_update(self) -> None: if self.dummy_batch: return @@ -1242,9 +1222,8 @@ def initialize_hisparse_runtime_buffers( cache_handles: list[HiSparseCacheHandle], *, max_num_reqs: int, - max_num_batched_tokens: int, ) -> None: - """Allocate shared request state and per-layer prefill staging buffers.""" + """Allocate the request state shared by every HiSparse runtime.""" assert cache_handles resident = cache_handles[0].view assert resident is not None @@ -1252,26 +1231,8 @@ def initialize_hisparse_runtime_buffers( request_state_indices = torch.full( (max_num_reqs,), -1, dtype=torch.int32, device=device ) - staging_blocks = ( - max_num_batched_tokens + resident.block_size - 1 - ) // resident.block_size - mirror_staging_caches = torch.empty( - ( - len(cache_handles), - staging_blocks, - resident.block_size, - resident.cache.shape[-1], - ), - dtype=resident.cache.dtype, - device=device, - ) - mirror_staging_slots = torch.arange( - max_num_batched_tokens, dtype=torch.int64, device=device - ) - for layer_index, cache_handle in enumerate(cache_handles): + for cache_handle in cache_handles: cache_handle.runtime.request_state_indices = request_state_indices - cache_handle.mirror_staging_cache = mirror_staging_caches[layer_index] - cache_handle.mirror_staging_slots = mirror_staging_slots def create_hisparse_cache_handle( @@ -1325,11 +1286,4 @@ def create_hisparse_cache_handle( config.device_buffer_size, max_num_reqs, ) - handle = HiSparseCacheHandle(runtime) - speculative_config = vllm_config.speculative_config - handle.mirror_from_resident = bool( - vllm_config.scheduler_config.async_scheduling - and speculative_config is not None - and speculative_config.uses_draft_kv_cache() - ) - return handle + return HiSparseCacheHandle(runtime) diff --git a/vllm/v1/hisparse/types.py b/vllm/v1/hisparse/types.py index 269793ea85..db0ee06e93 100644 --- a/vllm/v1/hisparse/types.py +++ b/vllm/v1/hisparse/types.py @@ -3,6 +3,10 @@ from dataclasses import dataclass +# Sealed pages this many positions behind the block-table tail stay pinned so +# a page written by an in-flight step is never handed out under it. +ACTIVE_TAIL_PAGES = 2 + @dataclass(frozen=True) class SparseKVPageTransfer: diff --git a/vllm/v1/kv_cache_interface.py b/vllm/v1/kv_cache_interface.py index 846e6bcad0..ca3f485917 100644 --- a/vllm/v1/kv_cache_interface.py +++ b/vllm/v1/kv_cache_interface.py @@ -11,10 +11,9 @@ from enum import Enum, IntEnum from fractions import Fraction from functools import cached_property from math import prod -from typing import TYPE_CHECKING, TypeVar +from typing import TYPE_CHECKING, Self, TypeVar import torch -from typing_extensions import Self from vllm.logger import init_logger from vllm.utils.math_utils import cdiv, round_up @@ -55,6 +54,7 @@ class KVQuantMode(IntEnum): TURBOQUANT_K3V4_NC = 8 TURBOQUANT_3BIT_NC = 9 NVFP4_DS_MLA = 10 # opaque-bytes NVFP4 DS-MLA layouts (FlashMLA sparse) + ULTRAQUANT_4BIT = 11 # packed FP4 values + UE8M0 group scales @property def is_per_token_head(self) -> bool: @@ -80,6 +80,11 @@ class KVQuantMode(IntEnum): KVQuantMode.TURBOQUANT_3BIT_NC, ) + @property + def is_ultraquant(self) -> bool: + """True for the UltraQuant 4-bit KV-cache format.""" + return self == KVQuantMode.ULTRAQUANT_4BIT + def get_kv_quant_mode(kv_cache_dtype: str) -> KVQuantMode: """Map a ``kv_cache_dtype`` string to a :class:`KVQuantMode`.""" @@ -98,6 +103,8 @@ def get_kv_quant_mode(kv_cache_dtype: str) -> KVQuantMode: return KVQuantMode.NVFP4 if isinstance(kv_cache_dtype, str) and kv_cache_dtype.startswith("turboquant_"): return KVQuantMode[kv_cache_dtype.upper()] + if kv_cache_dtype == "ultraquant_4bit": + return KVQuantMode.ULTRAQUANT_4BIT if isinstance(kv_cache_dtype, str) and kv_cache_dtype.startswith("fp8"): return KVQuantMode.FP8_PER_TENSOR return KVQuantMode.NONE @@ -1120,7 +1127,10 @@ def get_mamba_prefill_checkpoint_position( drop_eagle_block: bool, ) -> int: """Return the reusable Mamba checkpoint boundary for a prefill.""" - checkpoint_position = (num_tokens - 1) // hash_block_size * hash_block_size + # Without EAGLE, leave one token to recompute on resend. + # EAGLE's block drop already leaves tokens to recompute. + proof_limit = num_tokens if drop_eagle_block else num_tokens - 1 + checkpoint_position = proof_limit // hash_block_size * hash_block_size if drop_eagle_block: checkpoint_position -= hash_block_size return max(checkpoint_position, 0) @@ -1480,6 +1490,10 @@ class KVCacheConfig: """Resolved retention policy for local prefix-cache checkpoints.""" kv_cache_layout: str | None = None """The KV cache layout resolved by the engine core, adopted by all workers.""" + hash_block_size: int | None = None + """Tokens per prefix-cache block hash, resolved by the engine core.""" + cache_hit_alignment_tokens: int | None = None + """Token granularity of prefix-cache hits, resolved by the engine core.""" hisparse_host_num_blocks: int | None = None """Capacity of the dedicated HiSparse host-block manager, when enabled.""" diff --git a/vllm/v1/kv_offload/base.py b/vllm/v1/kv_offload/base.py index f8e92abed6..398bc269e6 100644 --- a/vllm/v1/kv_offload/base.py +++ b/vllm/v1/kv_offload/base.py @@ -18,6 +18,7 @@ if TYPE_CHECKING: from vllm.v1.kv_hints import KvHintsEnvelope from vllm.v1.kv_offload.config import OffloadingConfig +from vllm.v1.metrics.cache_hit_source import CacheHitSource # `OffloadKey` identifies an offloaded block. It combines a block hash with # its KV cache group index, encoded as raw bytes to avoid tuple GC overhead. @@ -248,6 +249,18 @@ class OffloadingManager(ABC): """ pass + def get_load_source( + self, key: OffloadKey, req_context: ReqContext + ) -> CacheHitSource: + """Return the cache tier that supplied a successful lookup. + + This is queried only after ``lookup`` resolves to ``HIT``. Composing + managers can override it to retain the origin across asynchronous + promotion into a primary tier. The default keeps out-of-tree managers + compatible while making unknown provenance explicit. + """ + return CacheHitSource.EXTERNAL_UNSPECIFIED + @abstractmethod def prepare_load( self, diff --git a/vllm/v1/kv_offload/cpu/gpu_worker.py b/vllm/v1/kv_offload/cpu/gpu_worker.py index a56febf702..f6053480a7 100644 --- a/vllm/v1/kv_offload/cpu/gpu_worker.py +++ b/vllm/v1/kv_offload/cpu/gpu_worker.py @@ -11,7 +11,6 @@ import numpy as np import torch from vllm import _custom_ops as ops -from vllm.distributed.device_communicators.cuda_wrapper import CudaRTLibrary from vllm.logger import init_logger from vllm.platforms import current_platform from vllm.triton_utils import HAS_TRITON, triton @@ -27,6 +26,7 @@ from vllm.v1.kv_offload.base import ( OffloadingWorker, TransferResult, ) +from vllm.v1.kv_offload.cpu.host_register import host_register, host_unregister from vllm.v1.kv_offload.cpu.shared_offload_region import SharedOffloadRegion from vllm.v1.kv_offload.cpu.swap_blocks_triton import ( THRESHOLD_BYTES, @@ -200,26 +200,7 @@ MAX_HOST_REGISTER_CHUNK_BYTES = 64 * 1024**3 def pin_mmap_region(region: SharedOffloadRegion) -> None: """Register row-aligned chunks, rolling back on failure.""" - if not current_platform.is_cuda_alike(): - logger.info( - "Skipping mmap host registration on %s; cudaHostRegister is only " - "available on CUDA/ROCm.", - current_platform.device_name, - ) - return - rank = region.rank - try: - cudart = CudaRTLibrary() - except (AssertionError, AttributeError, OSError): - logger.warning( - "Could not load the CUDA runtime for host registration on rank=%d; " - "the offload region stays pageable", - rank, - exc_info=True, - ) - return - base_ptr = region._base.data_ptr() total_size = region.total_size_bytes # Chunks end on block-row boundaries, which are page aligned, so neither the @@ -228,42 +209,30 @@ def pin_mmap_region(region: SharedOffloadRegion) -> None: rows_per_chunk = max(MAX_HOST_REGISTER_CHUNK_BYTES // region._row_stride, 1) chunk_size = rows_per_chunk * region._row_stride - # Register, drain and roll back through the same runtime handle, so a - # failed chunk leaves neither a pending error nor a partly pinned region. + # Register, drain and roll back through the same helper, so a failed + # chunk leaves neither a pending error nor a partly pinned region. addresses: list[int] = [] for offset in range(0, total_size, chunk_size): address = base_ptr + offset size = min(chunk_size, total_size - offset) - result = cudart.cudaHostRegister(address, size) - if result == 0: + if host_register(address, size): addresses.append(address) continue - cudart.cudaGetLastError() logger.warning( - "cudaHostRegister failed for rank=%d at %.2f of %.2f GB (code=%d); " + "host_register failed for rank=%d at %.2f of %.2f GB; " "the offload region stays pageable", rank, offset / 1e9, total_size / 1e9, - result, ) for registered in reversed(addresses): - unregister_result = cudart.cudaHostUnregister(registered) - if unregister_result != 0: - cudart.cudaGetLastError() - logger.warning( - "cudaHostUnregister failed for rank=%d at %#x (code=%d); " - "that chunk stays registered until the process exits", - rank, - registered, - unregister_result, - ) + host_unregister(registered) return region.pinned_addresses.extend(addresses) region.is_pinned = True logger.debug( - "cudaHostRegister rank=%d %.2f GB in %d chunk(s)", + "Host-registered mmap region rank=%d %.2f GB in %d chunk(s)", rank, total_size / 1e9, len(addresses), diff --git a/vllm/v1/kv_offload/cpu/host_register.py b/vllm/v1/kv_offload/cpu/host_register.py new file mode 100644 index 0000000000..3744885740 --- /dev/null +++ b/vllm/v1/kv_offload/cpu/host_register.py @@ -0,0 +1,97 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Registration of existing host memory for pinned DMA transfers. + +Host allocations that the offloading workers share across processes (the +`SharedOffloadRegion` mmap) cannot be allocated through the accelerator's +pinned allocator, so they must be registered after the fact. Pinning is a +pure bandwidth optimization: when registration is unavailable the transfers +still run correctly, just as pageable (slower) copies. +""" + +import torch + +from vllm.distributed.device_communicators.cuda_wrapper import CudaRTLibrary +from vllm.logger import init_logger +from vllm.platforms import current_platform + +logger = init_logger(__name__) + + +def _load_cudart() -> CudaRTLibrary | None: + try: + return CudaRTLibrary() + except (AssertionError, AttributeError, OSError): + logger.warning( + "Could not load the CUDA runtime for host registration; " + "the offload region stays pageable", + exc_info=True, + ) + return None + + +def host_register(ptr: int, num_bytes: int) -> bool: + """Register a host range as device-accessible pinned memory. + + Args: + ptr: Base address of the host range. + num_bytes: Size of the host range in bytes. + + Returns: + True if the range was registered and later needs `host_unregister`. + + """ + if not ptr or num_bytes <= 0: + return False + + if current_platform.is_cuda_alike(): + cudart = _load_cudart() + if cudart is None: + return False + result = cudart.cudaHostRegister(ptr, num_bytes) + if result != 0: + cudart.cudaGetLastError() + logger.warning( + "cudaHostRegister failed (code=%d) — transfers will still work " + "but may be slower (unpinned DMA)", + result, + ) + return False + return True + + if current_platform.is_xpu(): + if not torch.ops._C.xpu_host_register(ptr, num_bytes): + logger.warning( + "xpu_host_register failed — transfers will still work " + "but may be slower (unpinned DMA)" + ) + return False + return True + + logger.info_once( + "Host registration is not implemented on %s; KV offload transfers will " + "use unpinned host memory.", + current_platform.device_name, + ) + return False + + +def host_unregister(ptr: int) -> None: + """Release a host range previously registered by `host_register`.""" + if not ptr: + return + + if current_platform.is_cuda_alike(): + cudart = _load_cudart() + if cudart is None: + return + result = cudart.cudaHostUnregister(ptr) + if result != 0: + cudart.cudaGetLastError() + logger.warning("cudaHostUnregister failed (code=%d)", result) + return + + if current_platform.is_xpu(): + if not torch.ops._C.xpu_host_unregister(ptr): + logger.warning("xpu_host_unregister failed") + return diff --git a/vllm/v1/kv_offload/cpu/manager.py b/vllm/v1/kv_offload/cpu/manager.py index e42cb47f25..2c08ac897c 100644 --- a/vllm/v1/kv_offload/cpu/manager.py +++ b/vllm/v1/kv_offload/cpu/manager.py @@ -27,6 +27,7 @@ from vllm.v1.kv_offload.cpu.common import ( ) from vllm.v1.kv_offload.cpu.policies.base import CachePolicy, ChunkStatus from vllm.v1.kv_offload.cpu.policies.factory import CachePolicyFactory +from vllm.v1.metrics.cache_hit_source import CacheHitSource @dataclass(slots=True) @@ -193,6 +194,12 @@ class CPUOffloadingManager(OffloadingManager): return LookupResult.HIT_PENDING return LookupResult.HIT + @override + def get_load_source( + self, key: OffloadKey, req_context: ReqContext + ) -> CacheHitSource: + return CacheHitSource.HOST + @override def prepare_load( self, diff --git a/vllm/v1/kv_offload/cpu/shared_offload_region.py b/vllm/v1/kv_offload/cpu/shared_offload_region.py index af183fbc4f..7e0e97aa12 100644 --- a/vllm/v1/kv_offload/cpu/shared_offload_region.py +++ b/vllm/v1/kv_offload/cpu/shared_offload_region.py @@ -14,7 +14,7 @@ from vllm.distributed.device_communicators.shm_broadcast import ( check_shm_free_space, ) from vllm.logger import init_logger -from vllm.platforms import current_platform +from vllm.v1.kv_offload.cpu.host_register import host_unregister logger = init_logger(__name__) @@ -343,20 +343,11 @@ class SharedOffloadRegion: def cleanup(self) -> None: if self.is_pinned and self._base is not None: - if current_platform.is_cuda_alike(): - base_ptr = self._base.data_ptr() - addresses = self.pinned_addresses or [base_ptr] - for address in reversed(addresses): - result = torch.cuda.cudart().cudaHostUnregister(address) - if result.value != 0: - logger.warning( - "cudaHostUnregister failed for rank=%d, " - "address=%#x (code=%d)", - self.rank, - address, - result.value, - ) - self.pinned_addresses.clear() + base_ptr = self._base.data_ptr() + addresses = self.pinned_addresses or [base_ptr] + for address in reversed(addresses): + host_unregister(address) + self.pinned_addresses.clear() self.is_pinned = False # Release views before _base: each view holds a _base reference and a # direct StorageImpl reference. Freeing views first lets both refcounts diff --git a/vllm/v1/kv_offload/cpu/spec.py b/vllm/v1/kv_offload/cpu/spec.py index fddf566085..bf289d37a6 100644 --- a/vllm/v1/kv_offload/cpu/spec.py +++ b/vllm/v1/kv_offload/cpu/spec.py @@ -162,7 +162,9 @@ class CPUOffloadingSpec(OffloadingSpec): def _uses_shared_region(self) -> bool: """Whether the worker CPU buffer is the shared mmap region (vs a private per-rank tensor); replicated-layout dedup is gated on this being True.""" - return current_platform.is_cuda_alike() and not current_platform.is_rocm() + return ( + current_platform.is_cuda_alike() and not current_platform.is_rocm() + ) or current_platform.is_xpu() def create_worker(self, kv_caches: CanonicalKVCaches) -> CPUOffloadingWorker: mmap_region: SharedOffloadRegion | None = None diff --git a/vllm/v1/kv_offload/tiering/base.py b/vllm/v1/kv_offload/tiering/base.py index a90d33eb11..b855b6b000 100644 --- a/vllm/v1/kv_offload/tiering/base.py +++ b/vllm/v1/kv_offload/tiering/base.py @@ -24,6 +24,7 @@ from vllm.v1.kv_offload.base import ( from vllm.v1.kv_offload.tiering.backpressure import ( BackpressureDetector, ) +from vllm.v1.metrics.cache_hit_source import CacheHitSource if TYPE_CHECKING: from vllm.distributed.kv_transfer.kv_connector.v1.offloading.metrics import ( @@ -141,6 +142,17 @@ class SecondaryTierManager(ABC): """ medium: ClassVar[Medium | None] = None + # Cache tier reported for hits served from this tier. Derived from + # ``medium`` unless a subclass sets it; ``None`` medium means unknown. + cache_hit_source: ClassVar[CacheHitSource] = CacheHitSource.EXTERNAL_UNSPECIFIED + + def __init_subclass__(cls, **kwargs: Any) -> None: + super().__init_subclass__(**kwargs) + if "cache_hit_source" not in cls.__dict__ and cls.medium is not None: + cls.cache_hit_source = { + Medium.CPU: CacheHitSource.HOST, + Medium.STORAGE: CacheHitSource.DISK, + }[cls.medium] def __init__( self, diff --git a/vllm/v1/kv_offload/tiering/kvcr/manager.py b/vllm/v1/kv_offload/tiering/kvcr/manager.py index 18da2a6c27..4804f193e3 100644 --- a/vllm/v1/kv_offload/tiering/kvcr/manager.py +++ b/vllm/v1/kv_offload/tiering/kvcr/manager.py @@ -22,7 +22,6 @@ from kvcr import ( KVCRBindings, ) from kvcr.config import ( - FrameworkDramInput, G3Options, KeyAdapter, KVCRBackendConfigs, @@ -43,11 +42,12 @@ from kvcr.types import ( BlockKey, CacheTier, InventoryEvent, - MemDescriptor, + MemoryRef, OpHandle, PinRequestId, PinResult, QueryStatus, + RegionDescriptor, ) from typing_extensions import override @@ -82,6 +82,7 @@ from vllm.v1.kv_offload.tiering.base import ( if TYPE_CHECKING: from vllm.v1.kv_offload.base import OffloadingSpec + from vllm.v1.kv_offload.tiering.backpressure import BackpressureDetector _REQUIRED_ROUTER_CAPABILITIES = ROUTER_HINT_CAPABILITIES @@ -177,7 +178,7 @@ class _FrameworkPinAdapter: def _resolve_pin( self, parent: ParentManager, keys: Collection[BlockKey] - ) -> tuple[str, dict[BlockKey, list[MemDescriptor] | None]] | None: + ) -> tuple[str, dict[BlockKey, list[MemoryRef] | None]] | None: offload_keys = tuple(OffloadKey(bytes(key)) for key in keys) request_id = f"kvcr-source:{self._next_request_id}" self._next_request_id += 1 @@ -202,7 +203,7 @@ class _FrameworkPinAdapter: if len(job_keys) != len(chunk_ids) or set(job_keys) != set(hit_keys): return None - descriptors: dict[BlockKey, list[MemDescriptor] | None] = { + descriptors: dict[BlockKey, list[MemoryRef] | None] = { BlockKey(bytes(key)): None for key in offload_keys } descriptors.update( @@ -295,8 +296,11 @@ class KVCRSecondaryTierManager(SecondaryTierManager): g3: dict[str, Any] | None = None, local_dram_backend: str = "UCX", remote_fw_dram_backend: str = "UCX", + backpressure_detector: "BackpressureDetector | None" = None, ) -> None: - super().__init__(offloading_spec, primary_kv_view, tier_type) + super().__init__( + offloading_spec, primary_kv_view, tier_type, backpressure_detector + ) selected_policy = _resolve_policy(policy) if (kvcr_service_socket_path is None) != (compatibility_digest is None): raise ValueError( @@ -350,12 +354,6 @@ class KVCRSecondaryTierManager(SecondaryTierManager): with socket.socket() as _s: _s.bind(("", 0)) _nixl_listen_port = _s.getsockname()[1] - self._primary_base_addr = ctypes.addressof( - ctypes.c_char.from_buffer(primary_kv_view) - ) - if primary_kv_view.strides is None: - raise ValueError("primary KV memoryview must expose strides") - self._primary_row_stride = int(primary_kv_view.strides[0]) if secondary_g2_slots < 0: raise ValueError("secondary_g2_slots must be non-negative") if ( @@ -375,7 +373,7 @@ class KVCRSecondaryTierManager(SecondaryTierManager): ) else: local_mapping = mmap.mmap( - -1, secondary_g2_slots * self._primary_row_stride + -1, secondary_g2_slots * self.block_size_bytes ) local_dram = LocalDramOptions( pools=[ @@ -408,7 +406,7 @@ class KVCRSecondaryTierManager(SecondaryTierManager): try: self._kvcr = KVCR( KVCRConfig( - pool_layouts=[("", self._primary_row_stride)], + pool_layouts=[("", self.block_size_bytes)], enable_telemetry=enable_telemetry, operation_timeout_ms=operation_timeout_ms, abandon_timeout_ms=abandon_timeout_ms, @@ -429,9 +427,15 @@ class KVCRSecondaryTierManager(SecondaryTierManager): policy=selected_policy, ), KVCRBackendConfigs( - framework_dram=FrameworkDramInput( - self._primary_base_addr, primary_kv_view.nbytes - ), + framework_regions=[ + RegionDescriptor( + addr=ctypes.addressof( + ctypes.c_char.from_buffer(primary_kv_view) + ), + size=self.block_size_bytes, + count=len(primary_kv_view), + ) + ], local_dram=local_dram, g3=g3_config, remote_fw_dram=RemoteFWDramOptions( @@ -621,12 +625,8 @@ class KVCRSecondaryTierManager(SecondaryTierManager): ) ) - def _make_descriptor(self, chunk_id: int) -> MemDescriptor: - return MemDescriptor( + def _make_descriptor(self, chunk_id: int) -> MemoryRef: + return MemoryRef( end_point_name=self._kvcr.config.nixl_agent_name, - mem_type="DRAM", - addr=self._primary_base_addr + chunk_id * self._primary_row_stride, - size=self._primary_row_stride, - device_Id=0, - info="", + element_index=chunk_id, ) diff --git a/vllm/v1/kv_offload/tiering/manager.py b/vllm/v1/kv_offload/tiering/manager.py index ce518042b6..83ff2e2c12 100644 --- a/vllm/v1/kv_offload/tiering/manager.py +++ b/vllm/v1/kv_offload/tiering/manager.py @@ -54,6 +54,7 @@ from vllm.v1.kv_offload.tiering.base import ( TransferJob, ) from vllm.v1.kv_offload.tiering.metrics import TieringMetricsTracker +from vllm.v1.metrics.cache_hit_source import CacheHitSource logger = init_logger(__name__) @@ -232,6 +233,11 @@ class TieringOffloadingManager(OffloadingManager): # complete_store(), since complete_store() can still submit cascades. self._req_state: dict[str, RequestState] = {} + # Preserve the original tier for this request's cache-hit metrics, + # even after its KV blocks are promoted into host memory + # (otherwise a disk or P2P hit would be mislabeled as a host hit). + self._request_load_sources: dict[str, dict[OffloadKey, CacheHitSource]] = {} + # Cached ParentManager wrappers for each secondary tier. self._tier_parents: dict[SecondaryTierManager, _SecondaryTierFacingParent] = { tier: _SecondaryTierFacingParent(self, tier_idx) @@ -293,6 +299,13 @@ class TieringOffloadingManager(OffloadingManager): successful_keys = () failed_keys = transfer_job.keys + load_sources = self._request_load_sources.get(transfer_job.req_context.req_id) + if load_sources is not None: + source = self.secondary_tiers[job_metadata.tier_idx].cache_hit_source + for key in failed_keys: + if load_sources.get(key) == source: + del load_sources[key] + if successful_keys: self.primary_tier.complete_write( successful_keys, @@ -464,6 +477,15 @@ class TieringOffloadingManager(OffloadingManager): return LookupResult.RETRY return LookupResult.MISS + @override + def get_load_source( + self, key: OffloadKey, req_context: ReqContext + ) -> CacheHitSource: + load_sources = self._request_load_sources.get(req_context.req_id) + if load_sources is not None and key in load_sources: + return load_sources[key] + return self.primary_tier.get_load_source(key, req_context) + def _initiate_promotion( self, tier_idx: int, @@ -501,6 +523,10 @@ class TieringOffloadingManager(OffloadingManager): store_spec = primary_write_result.store_spec assert isinstance(store_spec, CPULoadStoreSpec) + load_sources = self._request_load_sources.setdefault(req_context.req_id, {}) + source = self.secondary_tiers[tier_idx].cache_hit_source + for promoted_key in primary_write_result.keys_to_store: + load_sources[promoted_key] = source # Defer submit_load to on_schedule_end(). Group by (tier, request) so # each request's chunks are submitted as one batched job per tier. tier_pending = self._pending_load_submissions.setdefault(tier_idx, {}) @@ -848,6 +874,7 @@ class TieringOffloadingManager(OffloadingManager): continue tier.on_request_finished(state.req_context) self._metrics.on_request_finished(state.req_context) + self._request_load_sources.pop(req_id, None) del self._req_state[req_id] @override @@ -930,6 +957,7 @@ class TieringOffloadingManager(OffloadingManager): # reset below invalidates; their submit_load() has not yet been # called so no tier I/O is touching that memory. self._pending_load_submissions.clear() + self._request_load_sources.clear() self._metrics.assert_idle() finished_req_ids = [] diff --git a/vllm/v1/kv_offload/tiering/p2p/manager.py b/vllm/v1/kv_offload/tiering/p2p/manager.py index 59b9c32069..f759a4481e 100644 --- a/vllm/v1/kv_offload/tiering/p2p/manager.py +++ b/vllm/v1/kv_offload/tiering/p2p/manager.py @@ -11,7 +11,7 @@ import time import uuid from collections.abc import Iterable, Sequence from dataclasses import dataclass, field -from typing import TYPE_CHECKING, Any +from typing import TYPE_CHECKING, Any, ClassVar from typing_extensions import override @@ -35,6 +35,7 @@ from vllm.v1.kv_offload.tiering.base import ( from vllm.v1.kv_offload.tiering.p2p.control import ControlTransport, ZmqTransport from vllm.v1.kv_offload.tiering.p2p.data import DataTransport, NixlTransport from vllm.v1.kv_offload.tiering.p2p.session import P2PSession +from vllm.v1.metrics.cache_hit_source import CacheHitSource if TYPE_CHECKING: from vllm.v1.kv_offload.base import OffloadingSpec @@ -206,6 +207,8 @@ class P2PSecondaryTierManager(SecondaryTierManager): and existing sessions are polled even when no requests are scheduled. """ + cache_hit_source: ClassVar[CacheHitSource] = CacheHitSource.P2P + def __init__( self, offloading_spec: OffloadingSpec, diff --git a/vllm/v1/metrics/cache_hit_source.py b/vllm/v1/metrics/cache_hit_source.py new file mode 100644 index 0000000000..b6fc5270e7 --- /dev/null +++ b/vllm/v1/metrics/cache_hit_source.py @@ -0,0 +1,32 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +from __future__ import annotations + +from collections.abc import Iterable +from enum import Enum + + +class CacheHitSource(str, Enum): + """Where cached prompt tokens' KV came from. + + Members are listed in the order blocks are offloaded: accelerator, then + host memory, then secondary tiers. Per-source token counts are plain + ``dict[CacheHitSource, int]`` mappings holding only non-zero entries. + """ + + DEVICE = "device" + HOST = "host" + P2P = "p2p" + DISK = "disk" + # Fallback: uninstrumented connector or remote store. Last, so mixing + # with a known tier never reports the known one. + EXTERNAL_UNSPECIFIED = "external_unspecified" + + @classmethod + def outermost(cls, sources: Iterable[CacheHitSource]) -> CacheHitSource: + """The tier in ``sources`` farthest from the accelerator.""" + return max(sources, key=_SOURCE_ORDER.__getitem__) + + +_SOURCE_ORDER = {source: rank for rank, source in enumerate(CacheHitSource)} diff --git a/vllm/v1/metrics/loggers.py b/vllm/v1/metrics/loggers.py index 681f80ce29..7ff1d2993c 100644 --- a/vllm/v1/metrics/loggers.py +++ b/vllm/v1/metrics/loggers.py @@ -23,6 +23,7 @@ from vllm.logger import init_logger from vllm.plugins import STAT_LOGGER_PLUGINS_GROUP, load_plugins_by_group from vllm.v1.engine import FinishReason from vllm.v1.metrics.buckets import histogram_buckets +from vllm.v1.metrics.cache_hit_source import CacheHitSource from vllm.v1.metrics.perf import PerfMetricsLogging, PerfMetricsProm from vllm.v1.metrics.prometheus import unregister_vllm_metrics from vllm.v1.metrics.stats import ( @@ -752,6 +753,23 @@ class PrometheusStatLogger(AggregateStatLoggerBase): counter_prompt_tokens_cached, per_engine_labelvalues ) + self.counter_prompt_tokens_cached_by_source = self._counter_cls( + name="vllm:prompt_tokens_cached_by_source", + documentation=( + "Prefix-cache hit tokens by the cache tier that supplied them: " + "device (local HBM), host (offloaded to DRAM), disk, p2p " + "(transferred from another vLLM instance) or " + "external_unspecified. Counted at admission; sums to " + "vllm:prefix_cache_hits + vllm:external_prefix_cache_hits." + ), + labelnames=labelnames + ["source"], + ) + for source in CacheHitSource: + for labelvalues in per_engine_labelvalues.values(): + self.counter_prompt_tokens_cached_by_source.labels( + *labelvalues, source.value + ) + counter_generation_tokens = self._counter_cls( name="vllm:generation_tokens", documentation="Number of generation tokens processed.", @@ -1111,13 +1129,23 @@ class PrometheusStatLogger(AggregateStatLoggerBase): scheduler_stats.prefix_cache_stats.hits ) - if scheduler_stats.connector_prefix_cache_stats is not None: + labelvalues = self.per_engine_labelvalues[engine_idx] + self.counter_prompt_tokens_cached_by_source.labels( + *labelvalues, CacheHitSource.DEVICE.value + ).inc(scheduler_stats.prefix_cache_stats.hits) + + connector_stats = scheduler_stats.connector_prefix_cache_stats + if connector_stats is not None: self.counter_connector_prefix_cache_queries[engine_idx].inc( - scheduler_stats.connector_prefix_cache_stats.queries + connector_stats.queries ) self.counter_connector_prefix_cache_hits[engine_idx].inc( - scheduler_stats.connector_prefix_cache_stats.hits + connector_stats.hits ) + for tier, num_tokens in connector_stats.hits_by_source.items(): + self.counter_prompt_tokens_cached_by_source.labels( + *labelvalues, tier.value + ).inc(num_tokens) if scheduler_stats.spec_decoding_stats is not None: self.spec_decoding_prom.observe( diff --git a/vllm/v1/metrics/perf.py b/vllm/v1/metrics/perf.py index 1cd8fa79f1..a2b3f534d8 100644 --- a/vllm/v1/metrics/perf.py +++ b/vllm/v1/metrics/perf.py @@ -10,12 +10,11 @@ import time from abc import ABC, abstractmethod from collections.abc import Iterable from dataclasses import asdict, dataclass, field -from typing import Any, Protocol +from typing import Any, Protocol, Self import prometheus_client import torch from pydantic import BaseModel, Field, ValidationError, model_validator -from typing_extensions import Self import vllm.envs as envs from vllm.config import VllmConfig diff --git a/vllm/v1/metrics/stats.py b/vllm/v1/metrics/stats.py index 4db98884fe..0046dcefa9 100644 --- a/vllm/v1/metrics/stats.py +++ b/vllm/v1/metrics/stats.py @@ -8,9 +8,13 @@ from typing import TYPE_CHECKING, Any import vllm.envs as envs from vllm.compilation.cuda_graph import CUDAGraphStat +from vllm.logger import init_logger +from vllm.v1.metrics.cache_hit_source import CacheHitSource from vllm.v1.metrics.perf import PerfStats from vllm.v1.spec_decode.metrics import SpecDecodingStats +logger = init_logger(__name__) + if TYPE_CHECKING: from vllm.v1.engine import EngineCoreEvent, EngineCoreOutput, FinishReason @@ -128,19 +132,42 @@ class PrefixCacheStats(BaseCacheStats): preempted_hits: int = 0 """The `hits` number for preempted requests.""" + hits_by_source: dict[CacheHitSource, int] = field(default_factory=dict) + """`hits` split by the cache tier that supplied them (connector stats only).""" - def record(self, num_tokens: int, num_hits: int, preempted: bool) -> None: + def record( + self, + num_tokens: int, + num_hits: int, + preempted: bool, + hits_by_source: dict[CacheHitSource, int] | None = None, + ) -> None: """Aggregate request information into the stats.""" if preempted: # Previously preempted request self.preempted_requests += 1 self.preempted_queries += num_tokens self.preempted_hits += num_hits - else: - # New request - self.requests += 1 - self.queries += num_tokens - self.hits += num_hits + return + # New request + self.requests += 1 + self.queries += num_tokens + self.hits += num_hits + if hits_by_source is None: + return + if sum(hits_by_source.values()) != num_hits: + logger.warning_once( + "Connector attributed %s for %d cached tokens; reporting them as %s.", + str(hits_by_source), + num_hits, + CacheHitSource.EXTERNAL_UNSPECIFIED.value, + ) + hits_by_source = {CacheHitSource.EXTERNAL_UNSPECIFIED: num_hits} + for source, num_source_hits in hits_by_source.items(): + if num_source_hits: + self.hits_by_source[source] = ( + self.hits_by_source.get(source, 0) + num_source_hits + ) @dataclass diff --git a/vllm/v1/sample/ops/topk_topp_sampler.py b/vllm/v1/sample/ops/topk_topp_sampler.py index a3084a09c2..daa94ebc3a 100644 --- a/vllm/v1/sample/ops/topk_topp_sampler.py +++ b/vllm/v1/sample/ops/topk_topp_sampler.py @@ -319,11 +319,11 @@ class TopKTopPSampler(nn.Module): k: torch.Tensor | None, p: torch.Tensor | None, ) -> tuple[torch.Tensor, torch.Tensor | None]: - """Fused Gumbel-max sampling for CPU. + """CPU Gumbel-max sampling with per-token SplitMix64 noise. - Uses a precomputed Gumbel table + single-pass argmax over logits, - skipping softmax and intermediate allocations entirely. - Falls back to the native path when fp64 Gumbel noise is requested. + Independent (seed, token) streams avoid the wrapping-window bias + (#59786). Falls back to native sampling when fp64 Gumbel noise + is requested. """ logits = apply_top_k_top_p(logits, k, p) logits_to_return = None diff --git a/vllm/v1/simple_kv_offload/manager.py b/vllm/v1/simple_kv_offload/manager.py index 9d95940a11..dfc24e420a 100644 --- a/vllm/v1/simple_kv_offload/manager.py +++ b/vllm/v1/simple_kv_offload/manager.py @@ -216,6 +216,7 @@ class SimpleCPUOffloadScheduler: scheduler_block_size=self.block_size, hash_block_size=self.hash_block_size, allow_partial_hash_hits=not lazy_offload, + num_prefill_lookahead=vllm_config.num_prefill_lookahead_tokens, ) self.group_block_sizes = self.cpu_coordinator.group_block_sizes # FA group's own resolved block_size; divides scheduler_block_size (the @@ -837,19 +838,6 @@ class SimpleCPUOffloadScheduler: ) request = state.request confirmed_tokens = request.num_computed_tokens - request.num_output_placeholders - # Truncate to the granularity a lookup can actually land on. With - # fine-grained hits that is hash_block_size; otherwise hits only land on - # the scheduler block (the group LCM). Using the LCM unconditionally - # would drop whole blocks that sit between the last LCM boundary and a - # reachable fine-grained boundary, so the other groups would hold that - # boundary and this one would not, and the joint hybrid lookup would - # reconcile to zero. - store_alignment = ( - self.hash_block_size - if self.cpu_coordinator.enable_partial_hash_hits - else self.block_size - ) - aligned_tokens = confirmed_tokens // store_alignment * store_alignment num_free = self.cpu_block_pool.get_num_free_blocks() for g, group_gpu_ids in enumerate(block_ids_by_group): @@ -864,7 +852,11 @@ class SimpleCPUOffloadScheduler: # num_stored_blocks can be stale and omit evicted blocks in # the middle of the request. group_size = self.group_block_sizes[g] - ready = min(len(group_gpu_ids), aligned_tokens // group_size) + # Mimic the GPU KV cache manager's cacheable prefix. + cacheable_tokens = self.cpu_coordinator.get_num_cacheable_tokens( + confirmed_tokens, g + ) + ready = min(len(group_gpu_ids), cacheable_tokens // group_size) resolved_hashes = resolve_block_hashes( request.block_hashes, self.hash_block_size, group_size ) diff --git a/vllm/v1/structured_output/__init__.py b/vllm/v1/structured_output/__init__.py index 3b35d39d14..2163a800a5 100644 --- a/vllm/v1/structured_output/__init__.py +++ b/vllm/v1/structured_output/__init__.py @@ -3,7 +3,7 @@ import multiprocessing from collections.abc import Iterable, Sequence from concurrent.futures import Future, ThreadPoolExecutor -from typing import TYPE_CHECKING +from typing import TYPE_CHECKING, NamedTuple from vllm.config import VllmConfig from vllm.logger import init_logger @@ -33,6 +33,17 @@ else: logger = init_logger(__name__) +class ConstraintBounds(NamedTuple): + """Where tokens start being fed to the grammar and being masked.""" + + grammar_start: int + constraint_start: int + + +_CONSTRAINED_FROM_START = ConstraintBounds(0, 0) +_NO_SPEC_TOKENS = ConstraintBounds(1, 1) + + class StructuredOutputManager: """Engine-level manager for structured output requests.""" @@ -217,32 +228,34 @@ class StructuredOutputManager: ) -> Future: return self.executor_for_fillmask.submit(self._fill_bitmasks, batch) - def _get_constraint_start( + def _get_constraint_bounds( self, request: "Request", spec_tokens: Sequence[int], spec_tokens_committed: bool = False, - ) -> int: + ) -> ConstraintBounds: """Return the index into `spec_tokens` where tokens should start being constrained by the grammar. Assumes `spec_tokens` are stripped of -1 padding tokens. Returns `len(spec_tokens) + 1` if no token should be constrained after accepting all `spec_tokens`. + This is `constraint_start`; `grammar_start` is one less if reasoning ends + implicitly on a content token (e.g. a tool-call start). `spec_tokens_committed` is True if spec_tokens is already in `request.all_token_ids`. """ if self.enable_in_reasoning: - return 0 + return _CONSTRAINED_FROM_START structured_req = request.structured_output_request assert structured_req is not None if structured_req.reasoning_ended: - return 0 + return _CONSTRAINED_FROM_START reasoner = self._get_reasoner(request) if reasoner is None: - return 0 + return _CONSTRAINED_FROM_START if structured_req.reasoning_ended is None: # This should be removed here, but since `openai_gptoss` @@ -250,30 +263,33 @@ class StructuredOutputManager: # After unifying the `openai_gptoss` and non-`openai_gptoss` styles, # it can be removed. if reasoner.is_reasoning_end(request.prompt_token_ids or []): - return 0 + return _CONSTRAINED_FROM_START structured_req.reasoning_ended = False num_spec_tokens = len(spec_tokens) if num_spec_tokens <= 0: - return num_spec_tokens + 1 + return _NO_SPEC_TOKENS - # Use `find_reasoning_end_offset` to find constraint start if supported + # Engine-based parsers also report whether the end token is content. if ( isinstance(reasoner, ParserEngineReasoningAdapter) and reasoner.reasoning_end_token_ids ): - offset = reasoner.find_reasoning_end_offset(spec_tokens) - if offset is not None: - return offset + 1 + end = reasoner.find_reasoning_end(spec_tokens) + if end is not None: + constraint_start = end.offset + 1 + if end.implicit: + return ConstraintBounds(end.offset, constraint_start) + return ConstraintBounds(constraint_start, constraint_start) - # Fallback to `find_reasoning_end_offset` + # Fallback: the grammar starts after the end token. # TODO: Build a read-only Sequence view over all_token_ids # instead of copying the entire all_token_ids into input_ids. if spec_tokens_committed: if not spec_tokens or not reasoner.is_reasoning_end_streaming( request.all_token_ids, spec_tokens ): - return num_spec_tokens + 1 + return ConstraintBounds(num_spec_tokens + 1, num_spec_tokens + 1) input_ids = request.all_token_ids.copy() delta_ids = list(spec_tokens) @@ -281,15 +297,15 @@ class StructuredOutputManager: input_ids = request.all_token_ids.copy() input_ids.extend(spec_tokens) if not reasoner.is_reasoning_end_streaming(input_ids, spec_tokens): - return num_spec_tokens + 1 + return ConstraintBounds(num_spec_tokens + 1, num_spec_tokens + 1) delta_ids = list(spec_tokens) for i in range(num_spec_tokens - 1, 0, -1): input_ids.pop() delta_ids.pop() if not reasoner.is_reasoning_end_streaming(input_ids, delta_ids): - return i + 1 - return 1 + return ConstraintBounds(i + 1, i + 1) + return ConstraintBounds(1, 1) def validate_tokens(self, request: "Request", spec_tokens: list[int]) -> list[int]: """Return the longest unconstrained or grammar-valid prefix of `spec_tokens`.""" @@ -297,8 +313,8 @@ class StructuredOutputManager: return spec_tokens spec_tokens = strip_speculative_padding(spec_tokens) - constraint_start = self._get_constraint_start(request, spec_tokens) - if constraint_start >= len(spec_tokens): + grammar_start, _ = self._get_constraint_bounds(request, spec_tokens) + if grammar_start >= len(spec_tokens): return spec_tokens structured_req = request.structured_output_request @@ -307,8 +323,8 @@ class StructuredOutputManager: grammar = structured_req.grammar if TYPE_CHECKING: assert isinstance(grammar, StructuredOutputGrammar) - prefix = spec_tokens[:constraint_start] - validated = grammar.validate_tokens(spec_tokens[constraint_start:]) + prefix = spec_tokens[:grammar_start] + validated = grammar.validate_tokens(spec_tokens[grammar_start:]) return prefix + validated def grammar_bitmask( @@ -358,7 +374,9 @@ class StructuredOutputManager: if TYPE_CHECKING: assert isinstance(grammar, StructuredOutputGrammar) - apply_bitmask = self._get_constraint_start(request, ()) == 0 + apply_bitmask = ( + self._get_constraint_bounds(request, ()).constraint_start == 0 + ) batch.append((grammar, cumulative_index, apply_bitmask)) if len(batch) == self.fill_bitmask_parallel_batch_size: promises.append(self._async_submit_fill_bitmask(batch)) @@ -384,7 +402,7 @@ class StructuredOutputManager: assert isinstance(grammar, StructuredOutputGrammar) req_tokens = scheduled_spec_decode_tokens.get(req_id, list()) - constraint_start = self._get_constraint_start( + grammar_start, constraint_start = self._get_constraint_bounds( request, strip_speculative_padding(req_tokens) ) state_advancements = 0 @@ -402,7 +420,7 @@ class StructuredOutputManager: self._fill_bitmasks(((grammar, cumulative_index, apply_bitmask),)) if token == -1: seen_padding = True - elif apply_bitmask: + elif not seen_padding and i >= grammar_start: if not grammar.is_terminated() and grammar.accept_tokens( req_id, [token] ): @@ -453,17 +471,15 @@ class StructuredOutputManager: if TYPE_CHECKING: assert isinstance(grammar, StructuredOutputGrammar) - constraint_start = self._get_constraint_start( + grammar_start, _ = self._get_constraint_bounds( request, new_token_ids, spec_tokens_committed=True ) - # Early return only when the constraint hasn't started. + # Early return only when no reasoning-end token is present. # Otherwise, latch structured_req.reasoning_ended. - if constraint_start > len(new_token_ids): + if grammar_start > len(new_token_ids): return True structured_req.reasoning_ended = True - return grammar.accept_tokens( - request.request_id, new_token_ids[constraint_start:] - ) + return grammar.accept_tokens(request.request_id, new_token_ids[grammar_start:]) def clear_backend(self) -> None: if self.backend is not None: diff --git a/vllm/v1/structured_output/backend_guidance.py b/vllm/v1/structured_output/backend_guidance.py index 5f43c5a890..6dc2dc91ff 100644 --- a/vllm/v1/structured_output/backend_guidance.py +++ b/vllm/v1/structured_output/backend_guidance.py @@ -34,17 +34,58 @@ else: logger = init_logger(__name__) +_SCHEMA_MAP_KEYWORDS = ( + "properties", + "patternProperties", + "$defs", + "definitions", + "dependentSchemas", + "dependencies", +) + +_SUBSCHEMA_KEYWORDS = ( + "additionalProperties", + "unevaluatedProperties", + "propertyNames", + "contains", + "additionalItems", + "unevaluatedItems", + "not", + "if", + "then", + "else", + "contentSchema", + "items", + "prefixItems", + "allOf", + "anyOf", + "oneOf", +) + + def _walk_json_for_additional_properties(data: object): if isinstance(data, dict): - for value in data.values(): - _walk_json_for_additional_properties(value) + for key in _SCHEMA_MAP_KEYWORDS: + value = data.get(key) + if isinstance(value, dict): + for subschema in value.values(): + if isinstance(subschema, dict): + _walk_json_for_additional_properties(subschema) + + for key in _SUBSCHEMA_KEYWORDS: + value = data.get(key) + if isinstance(value, (dict, list)): + _walk_json_for_additional_properties(value) + if "additionalProperties" not in data and ( - "properties" in data or "patternProperties" in data + isinstance(data.get("properties"), dict) + or isinstance(data.get("patternProperties"), dict) ): data["additionalProperties"] = False elif isinstance(data, list): for item in data: - _walk_json_for_additional_properties(item) + if isinstance(item, dict): + _walk_json_for_additional_properties(item) def has_guidance_unsupported_json_features(schema: dict[str, Any]) -> bool: diff --git a/vllm/v1/structured_output/backend_outlines.py b/vllm/v1/structured_output/backend_outlines.py index dab4f2aa04..1cfc6c8f5d 100644 --- a/vllm/v1/structured_output/backend_outlines.py +++ b/vllm/v1/structured_output/backend_outlines.py @@ -4,10 +4,9 @@ from __future__ import annotations import ast -import importlib import json -import sys from dataclasses import dataclass, field +from re import _constants, _parser # type: ignore[attr-defined] from typing import TYPE_CHECKING import torch @@ -36,19 +35,6 @@ else: oc = LazyLoader("oc", globals(), "outlines_core") json_schema = LazyLoader("json_schema", globals(), "outlines_core.json_schema") -# Python 3.11+ sre_parse and sre_constants -# are deprecated, so we must import them from re -if sys.version_info >= (3, 11): - # Hack to get around pre-commit regex module rule - # because going through re is the only way to get sre_parse - # and sre_constants in Python 3.11+ - _re = importlib.import_module("re") - sre_parse = _re._parser - sre_constants = _re._constants -else: - import sre_constants - import sre_parse - @dataclass class OutlinesBackend(StructuredOutputBackend): @@ -261,20 +247,20 @@ def _prefix_needs_context(parsed) -> bool: tokens = parsed.data if hasattr(parsed, "data") else parsed for ttype, tval in tokens: # literal, character class, or dot always consumes - if ttype in (sre_parse.LITERAL, sre_parse.IN, sre_parse.ANY): + if ttype in (_parser.LITERAL, _parser.IN, _parser.ANY): return True # quantified subpattern: check inner pattern - elif ttype == sre_parse.MAX_REPEAT: + elif ttype == _parser.MAX_REPEAT: _, mx, sub = tval if mx != 0 and subpattern_consumes(sub): return True # alternation: if any branch consumes, the whole does - elif ttype == sre_parse.BRANCH: + elif ttype == _parser.BRANCH: _, branches = tval if any(subpattern_consumes(br) for br in branches): return True # grouped subpattern: recurse into its contents - elif ttype == sre_parse.SUBPATTERN and subpattern_consumes(tval[3]): + elif ttype == _parser.SUBPATTERN and subpattern_consumes(tval[3]): return True # No consumers, return False return False @@ -282,14 +268,11 @@ def _prefix_needs_context(parsed) -> bool: tokens = parsed.data if hasattr(parsed, "data") else parsed for ttype, tval in tokens: # Direct anchors or look-around - if ttype == sre_parse.AT or ttype in ( - sre_constants.ASSERT, - sre_constants.ASSERT_NOT, - ): + if ttype == _parser.AT or ttype in (_constants.ASSERT, _constants.ASSERT_NOT): return True # Nested subpattern: check - if ttype == sre_parse.SUBPATTERN: + if ttype == _parser.SUBPATTERN: # tval: (group, add_flags, del_flags, subpattern) if _prefix_needs_context(tval[3]): return True @@ -298,7 +281,7 @@ def _prefix_needs_context(parsed) -> bool: # if any branch has a prefix anchor => True, # else if at least one branch consumes => prefix ends => False - elif ttype == sre_parse.BRANCH: + elif ttype == _parser.BRANCH: saw_consumer = False for br in tval[1]: if _prefix_needs_context(br): @@ -309,11 +292,11 @@ def _prefix_needs_context(parsed) -> bool: return False # Immediate consumer tokens - elif ttype in (sre_parse.LITERAL, sre_parse.IN, sre_parse.ANY): + elif ttype in (_parser.LITERAL, _parser.IN, _parser.ANY): return False # if subpattern has anchor => True, if it can consume => stop - elif ttype == sre_parse.MAX_REPEAT: + elif ttype == _parser.MAX_REPEAT: if _prefix_needs_context(tval[2]): return True if subpattern_consumes(tval[2]): @@ -327,26 +310,28 @@ def _check_unsupported(parsed) -> None: tokens = parsed.data if hasattr(parsed, "data") else parsed for ttype, tval in tokens: # backreference - if ttype in (sre_parse.GROUPREF, sre_parse.GROUPREF_EXISTS): + if ttype in (_parser.GROUPREF, _parser.GROUPREF_EXISTS): raise ValueError("Backreferences are unsupported.") # look-around assertion - elif ttype in (sre_constants.ASSERT, sre_constants.ASSERT_NOT): + elif ttype in (_constants.ASSERT, _constants.ASSERT_NOT): raise ValueError("Look-Around assertion are unsupported.") # unicode word boundaries - elif ttype == sre_parse.AT: - if tval in (sre_constants.AT_BOUNDARY, sre_constants.AT_NON_BOUNDARY): + elif ttype == _parser.AT: + if tval in (_constants.AT_BOUNDARY, _constants.AT_NON_BOUNDARY): raise ValueError("Unicode word boundaries are unsupported.") - elif ttype == sre_parse.BRANCH: + elif ttype == _parser.BRANCH: # tval is (None, branches) for branch in tval[1]: _check_unsupported(branch) - # tval is (min, max, subpattern) - elif ttype == sre_parse.MAX_REPEAT: - _check_unsupported(tval[2]) + else: + nested = tval if isinstance(tval, tuple) else (tval,) + for sub in nested: + if isinstance(sub, _parser.SubPattern): + _check_unsupported(sub) def validate_regex_is_buildable(pattern: str) -> None: @@ -357,9 +342,9 @@ def validate_regex_is_buildable(pattern: str) -> None: https://docs.rs/regex-automata/latest/regex_automata/dfa/trait.Automaton.html#method.universal_start_state """ try: - parsed = sre_parse.parse(pattern) + parsed = _parser.parse(pattern) - except sre_constants.error as e: + except _constants.error as e: raise VLLMValidationError(f"Error parsing regex: {e}") from e try: diff --git a/vllm/v1/structured_output/backend_xgrammar.py b/vllm/v1/structured_output/backend_xgrammar.py index 337b2285f0..0199c5f75d 100644 --- a/vllm/v1/structured_output/backend_xgrammar.py +++ b/vllm/v1/structured_output/backend_xgrammar.py @@ -10,7 +10,7 @@ import torch import vllm.envs from vllm.exceptions import VLLMValidationError from vllm.logger import init_logger -from vllm.sampling_params import SamplingParams +from vllm.sampling_params import SamplingParams, check_json_nesting from vllm.utils.import_utils import LazyLoader from vllm.utils.mistral import is_mistral_tokenizer from vllm.v1.structured_output.backend_types import ( @@ -89,13 +89,36 @@ class XgrammarBackend(StructuredOutputBackend): grammar_spec: str, stop_token_ids: set[int] | None = None, ) -> StructuredOutputGrammar: + # Note(arpera): + # Our flag disable_any_whitespace does NOT map directly to + # xgrammar's flag any_whitespace + # To achieve desired behavior of disable_any_whitespace + # we have to set not only any_whitespace + # but also specify a list of separators after which + # xgrammar must not insert spaces. + # This is a requirement of xgrammar's API, so we must comply with it. + # + # FIXME(arpera): + # Currently xgrammar v0.2.8 DOES emit spaces after comma + # even if we specify it in separators list. + # The bug has been reported to xgrammar team: + # https://github.com/mlc-ai/xgrammar/issues/945 + # Please, track that issue, and once it is resolved remove this comment. + # Upd. this bug was fixed in xgrammar main branch on Oct 8, 2026 + # and will be available in next release. + # So, remove this comment once xgrammar updates to v0.2.9 + separators = (",", ":") if self.disable_any_whitespace else None if request_type == StructuredOutputOptions.JSON: ctx = self.compiler.compile_json_schema( - grammar_spec, any_whitespace=not self.disable_any_whitespace + grammar_spec, + any_whitespace=not self.disable_any_whitespace, + separators=separators, ) elif request_type == StructuredOutputOptions.JSON_OBJECT: ctx = self.compiler.compile_json_schema( - '{"type": "object"}', any_whitespace=not self.disable_any_whitespace + '{"type": "object"}', + any_whitespace=not self.disable_any_whitespace, + separators=separators, ) elif request_type == StructuredOutputOptions.GRAMMAR: if grammar_is_likely_lark(grammar_spec): @@ -273,18 +296,29 @@ def has_xgrammar_unsupported_json_features(schema: dict[str, Any]) -> bool: schema_types = _schema_types(obj) - # Check for numeric ranges + # integer/number + multipleOf is unsupported by xgrammar + # This is known behavior and xgrammar emits warning in logs: + # [21:18:08] /project/cpp/json_schema_converter.cc:1053: + # Warning: multipleOf is not supported for type:number; ignoring multipleOf + # This warning was added in PR + # https://github.com/mlc-ai/xgrammar/pull/670 + # So, no need to track progress on this if (schema_types & {"integer", "number"}) and ("multipleOf" in obj): return True - # Check for array unsupported keywords + # array + some constraints is unsupported by xgrammar + # Tracking issue: https://github.com/mlc-ai/xgrammar/issues/968 if "array" in schema_types and any( key in obj for key in ("uniqueItems", "contains", "minContains", "maxContains") ): return True - # Unsupported keywords for strings + # string + format with unsupported keywords + # is unsupported by xgrammar + # Tracking issue: https://github.com/mlc-ai/xgrammar/issues/967 + # See tests on this in test_backend_xgrammar.py + # unsupported_string_schemas if ( "string" in schema_types and "format" in obj @@ -292,47 +326,67 @@ def has_xgrammar_unsupported_json_features(schema: dict[str, Any]) -> bool: ): return True - # A string mixing a generative constraint (pattern or format) with - # explicit length bounds. xgrammar compiles the pattern/format side - # and silently drops minLength/maxLength from the grammar, so output - # can violate the bound without any error surfacing. Verified against - # the compiled EBNF: pattern/format grammars come out byte-identical - # with and without the length keywords, while maxLength alone lowers - # to {0, N} correctly. + # string + format/pattern + length constraint is unsupported by xgrammar + # Tracking issue: https://github.com/mlc-ai/xgrammar/issues/966 + # See tests on this in test_backend_xgrammar.py + # unsupported_string_schemas if "string" in schema_types and _has_pattern_and_length_bounds(obj): return True - # propertyNames validates names, so it is a string schema even when it - # omits "type", which is the form that escapes the check above. - if ( - "object" in schema_types - and isinstance(obj.get("propertyNames"), dict) - and _has_pattern_and_length_bounds(obj["propertyNames"]) - ): - return True + # propertyNames is not supported in pair with some constraints + # in xgrammar + # See tests on this in test_backend_xgrammar.py + # unsupported_propertyNames_combinations + if "object" in schema_types and "propertyNames" in obj: + # propertyNames + maxLength is unsupported by xgrammar + # Tracking issue: https://github.com/mlc-ai/xgrammar/issues/960 + property_names = obj.get("propertyNames") + if isinstance(property_names, dict) and _has_pattern_and_length_bounds( + property_names + ): + return True + # propertyNames + patternProperties is unsupported by xgrammar + # Tracking issue: https://github.com/mlc-ai/xgrammar/issues/959 + if "patternProperties" in obj: + return True + # propertyNames + properties is unsupported by xgrammar + # Tracking issue: https://github.com/mlc-ai/xgrammar/issues/958 + if "properties" in obj: + return True + # propertyNames + unevaluatedProperties is unsupported by xgrammar + # Tracking issue: https://github.com/mlc-ai/xgrammar/issues/961 + if obj.get("unevaluatedProperties", True) is not True: + return True - # FIXME: propertyNames conflicts with properties/patternProperties/ - # additionalProperties/unevaluatedProperties under xgrammar. - # https://github.com/mlc-ai/xgrammar/issues/826 - if ( - "object" in schema_types - and "propertyNames" in obj - and ( - "properties" in obj - or "patternProperties" in obj - or isinstance(obj.get("additionalProperties"), dict) - or obj.get("unevaluatedProperties", True) is not True - ) - ): - return True + # patternProperties is not supported in pair with some constraints + # in xgrammar + # See tests on this in test_backend_xgrammar.py + # unsupported_patternProperties_combinations + if "object" in schema_types and isinstance(obj.get("patternProperties"), dict): + # patternProperties + properties is unsupported by xgrammar + # Tracking issue: https://github.com/mlc-ai/xgrammar/issues/964 + if "properties" in obj: + return True + # patternProperties + patternProperties is unsupported by xgrammar + # Tracking issue: https://github.com/mlc-ai/xgrammar/issues/965 + if len(obj["patternProperties"]) > 1: + return True - # FIXME: multiple patternProperties, or patternProperties alongside - # properties, conflict under xgrammar. - if ( - "object" in schema_types - and isinstance(obj.get("patternProperties"), dict) - and ("properties" in obj or len(obj["patternProperties"]) > 1) - ): + # Note(arpera): + # Xgrammar lacks support of multi-branch allOf + # For instance, this schema: + # { + # "allOf": [ + # { "type": "string" }, + # { "enum": ["yes", "no"] } + # ] + # } + # would accept any kind of json, such as + # "maybe", "", 42, {}, [], {"a": 1}, etc. + # which is NOT what is expected. + # Tracking issue: https://github.com/mlc-ai/xgrammar/issues/937 + allof = obj.get("allOf") + if isinstance(allof, list) and len(allof) >= 2: return True # Recursively check all nested objects and arrays @@ -392,6 +446,7 @@ def validate_xgrammar_grammar(sampling_params: SamplingParams) -> None: return if so_params.json: + check_json_nesting(so_params.json) if isinstance(so_params.json, str): try: schema = json.loads(so_params.json) @@ -400,13 +455,15 @@ def validate_xgrammar_grammar(sampling_params: SamplingParams) -> None: else: schema = so_params.json - if has_xgrammar_unsupported_json_features(schema): - raise VLLMValidationError( - "The provided JSON schema contains features not supported by xgrammar." - ) - try: + if has_xgrammar_unsupported_json_features(schema): + raise VLLMValidationError( + "The provided JSON schema contains features not supported " + "by xgrammar." + ) xgr.Grammar.from_json_schema(schema) + except VLLMValidationError: + raise except Exception as err: raise VLLMValidationError( f"Failed to transform json schema into a grammar: {err}" @@ -426,6 +483,7 @@ def validate_xgrammar_grammar(sampling_params: SamplingParams) -> None: return if so_params.structural_tag: + check_json_nesting(so_params.structural_tag, structural_tag=True) try: s_tag = json.loads(so_params.structural_tag) diff --git a/vllm/v1/utils.py b/vllm/v1/utils.py index ac877263a5..b8d98ff1cf 100644 --- a/vllm/v1/utils.py +++ b/vllm/v1/utils.py @@ -29,7 +29,7 @@ from torch.autograd.profiler import record_function import vllm.envs as envs from vllm.logger import init_logger from vllm.usage.usage_lib import UsageContext, is_usage_stats_enabled, usage_message -from vllm.utils.network_utils import get_open_zmq_ipc_path, get_tcp_uri +from vllm.utils.network_utils import ZmqListener, get_open_zmq_ipc_path, get_tcp_uri from vllm.utils.system_utils import decorate_logs, kill_process_tree, set_process_title from vllm.utils.torch_utils import PIN_MEMORY from vllm.v1.core.sched.output import SchedulerOutput @@ -177,28 +177,22 @@ class APIServerProcessManager: sock: Any, args: argparse.Namespace, num_servers: int, - input_addresses: list[str], - output_addresses: list[str], + input_listeners: list[ZmqListener], + output_listeners: list[ZmqListener], target_server_fn: Callable | None = None, stats_update_address: str | None = None, tensor_queue: Queue | None = None, ): """Initialize and start API server worker processes. - ``input_addresses``/``output_addresses`` may contain - ``tcp://host:0`` placeholders; each child must report the actual - bound endpoint over its ``actual_address_pipe`` in ``client_config`` - and the parent collects them via - :py:meth:`gather_actual_addresses`. - Args: target_server_fn: Override function to call for each API server process listen_address: Address to listen for client connections sock: Socket for client connections args: Command line arguments num_servers: Number of API server processes to start - input_addresses: Input addresses for each API server - output_addresses: Output addresses for each API server + input_listeners: Input listeners for each API server + output_listeners: Output listeners for each API server stats_update_address: Optional stats update address tensor_queue: Optional tensor IPC queue for sharing MM tensors @@ -209,7 +203,6 @@ class APIServerProcessManager: spawn_context = multiprocessing.get_context("spawn") self.processes: list[BaseProcess] = [] - self._address_pipes: list[connection.Connection] = [] admission_counters = None if num_servers > 1 and getattr(args, "max_num_queued_reqs", None) is not None: @@ -220,126 +213,75 @@ class APIServerProcessManager: SharedAdmissionStats.num_counters(num_servers), ) - for i, in_addr, out_addr in zip( - range(num_servers), input_addresses, output_addresses - ): - client_config: dict[str, Any] = { - "input_address": in_addr, - "output_address": out_addr, - "client_count": num_servers, - "client_index": i, - } - if admission_counters is not None: - client_config["mp_admission_counters"] = admission_counters - if stats_update_address is not None: - client_config["stats_update_address"] = stats_update_address - if tensor_queue is not None: - client_config["tensor_queue"] = tensor_queue + # Open connections per API server, to accept on the least loaded one; + # a server that has not started yet is never the least loaded. + api_server_loads = ( + spawn_context.RawArray("i", [2**31 - 1] * num_servers) + if num_servers > 1 + else None + ) - parent_recv, child_send = spawn_context.Pipe(duplex=False) - self._address_pipes.append(parent_recv) - client_config["actual_address_pipe"] = child_send - - proc = spawn_context.Process( - target=target_server_fn or run_api_server_worker_proc, - name=f"ApiServer_{i}", - args=(listen_address, sock, args, client_config), + listeners = [*input_listeners, *output_listeners] + self._listeners = listeners + try: + listener_pairs = zip( + range(num_servers), input_listeners, output_listeners, strict=True ) - self.processes.append(proc) - proc.start() + for i, input_listener, output_listener in listener_pairs: + client_config: dict[str, Any] = { + "input_address": input_listener.address, + "output_address": output_listener.address, + "input_listener": input_listener.socket, + "output_listener": output_listener.socket, + "client_count": num_servers, + "client_index": i, + } + if api_server_loads is not None: + client_config["api_server_loads"] = api_server_loads + if admission_counters is not None: + client_config["mp_admission_counters"] = admission_counters + if stats_update_address is not None: + client_config["stats_update_address"] = stats_update_address + if tensor_queue is not None: + client_config["tensor_queue"] = tensor_queue - # Drop parent's write end so reader sees EOF on child death. - child_send.close() + proc = spawn_context.Process( + target=target_server_fn or run_api_server_worker_proc, + name=f"ApiServer_{i}", + args=(listen_address, sock, args, client_config), + ) + self.processes.append(proc) + proc.start() + input_listener.close() + output_listener.close() + except BaseException: + _shutdown_processes_and_cleanup_listeners(self.processes, listeners) + raise logger.info("Started %d API server processes", len(self.processes)) # Shutdown only the API server processes on garbage collection # The extra processes are managed by their owners - self._finalizer = weakref.finalize(self, shutdown, self.processes) - - def gather_actual_addresses( - self, - timeout: float = envs.VLLM_ENGINE_READY_TIMEOUT_S, - ) -> tuple[list[str], list[str]]: - """Return (inputs, outputs) reported by each child, indexed by - ``client_index``. Raises ``RuntimeError`` on timeout or premature - child exit.""" - n = len(self._address_pipes) - inputs: list[str | None] = [None] * n - outputs: list[str | None] = [None] * n - pending: dict[connection.Connection, int] = { - pipe: i for i, pipe in enumerate(self._address_pipes) - } - sentinel_to_idx: dict[Any, int] = { - proc.sentinel: i for i, proc in enumerate(self.processes) - } - - deadline = time.monotonic() + timeout - try: - while pending: - remaining = deadline - time.monotonic() - if remaining <= 0: - missing = [self.processes[i].name for i in pending.values()] - raise RuntimeError( - f"Timed out after {timeout:.1f}s waiting for " - f"API server(s) to report bound ZMQ addresses: " - f"{missing}" - ) - waitables: list[Any] = list(pending.keys()) + list( - sentinel_to_idx.keys() - ) - ready = connection.wait(waitables, timeout=remaining) - # Drain pipes before checking sentinels: a child that sent - # its message and then exited can surface both events in - # the same poll, and we must record the success first. - for item in ready: - if isinstance(item, connection.Connection) and item in pending: - idx = pending.pop(item) - try: - msg: dict[str, str] = item.recv() - except EOFError as e: - raise RuntimeError( - f"API server {self.processes[idx].name} " - f"closed its address pipe without " - f"reporting its bound ZMQ addresses" - ) from e - inputs[idx] = msg["input_address"] - outputs[idx] = msg["output_address"] - item.close() - for item in ready: - if item in sentinel_to_idx: - idx = sentinel_to_idx.pop(item) - pipe = self._address_pipes[idx] - if pipe in pending: - proc = self.processes[idx] - raise RuntimeError( - f"API server process {proc.name} exited " - f"(code={proc.exitcode}) before reporting " - f"its bound ZMQ addresses" - ) - finally: - for pipe in pending: - with contextlib.suppress(Exception): - pipe.close() - - return inputs, outputs # type: ignore[return-value] + self._finalizer = weakref.finalize( + self, + _shutdown_processes_and_cleanup_listeners, + self.processes, + listeners, + ) def shutdown(self, timeout: float | None = None) -> None: """Shutdown API server processes with configurable timeout""" - for pipe in self._address_pipes: - with contextlib.suppress(Exception): - pipe.close() - self._address_pipes = [] - if self._finalizer.detach() is not None: - shutdown(self.processes, timeout=timeout) + _shutdown_processes_and_cleanup_listeners( + self.processes, self._listeners, timeout=timeout + ) class RustFrontendProcessManager: """Manages a single Rust frontend subprocess. Launches the Rust vllm-rs binary in 'frontend' mode, passing the - listening socket fd and ZMQ transport addresses. Provides the same + HTTP and ZMQ listener file descriptors. Provides the same interface as APIServerProcessManager for process monitoring. """ @@ -348,8 +290,8 @@ class RustFrontendProcessManager: binary_path: str, sock: Any, args: argparse.Namespace, - input_address: str, - output_address: str, + input_listener: ZmqListener, + output_listener: ZmqListener, engine_start_index: int, engine_count: int, data_parallel_size: int, @@ -359,18 +301,22 @@ class RustFrontendProcessManager: import os import subprocess - fd = sock.fileno() - os.set_inheritable(fd, True) + listen_fd = sock.fileno() + input_fd = input_listener.socket.fileno() + output_fd = output_listener.socket.fileno() + inherited_fds = [listen_fd, input_fd, output_fd] + for fd in inherited_fds: + os.set_inheritable(fd, True) cmd = [ binary_path, "frontend", "--listen-fd", - str(fd), - "--input-address", - input_address, - "--output-address", - output_address, + str(listen_fd), + "--input-listener-fd", + str(input_fd), + "--output-listener-fd", + str(output_fd), "--engine-start-index", str(engine_start_index), "--engine-count", @@ -378,12 +324,11 @@ class RustFrontendProcessManager: "--data-parallel-size", str(data_parallel_size), ] - pass_fds = [fd] if grpc_sock is not None: grpc_fd = grpc_sock.fileno() os.set_inheritable(grpc_fd, True) cmd.extend(["--grpc-listen-fd", str(grpc_fd)]) - pass_fds.append(grpc_fd) + inherited_fds.append(grpc_fd) if stats_update_address is not None: cmd.extend(["--coordinator-address", stats_update_address]) from vllm.entrypoints.serve.utils.api_utils import jsonify_non_default_args @@ -425,18 +370,34 @@ class RustFrontendProcessManager: redacted_json = json.dumps(redact_sensitive_args(args_dict), sort_keys=True) logger.info("Launching Rust frontend: %s", " ".join(cmd[:-1] + [redacted_json])) - self._proc = subprocess.Popen(cmd, pass_fds=pass_fds) + self._listeners = [input_listener, output_listener] + try: + self._proc = subprocess.Popen(cmd, pass_fds=inherited_fds) + except BaseException: + for listener in self._listeners: + listener.cleanup() + raise + else: + input_listener.close() + output_listener.close() # Create a process wrapper with a sentinel fd for monitoring self.processes: list[_SubprocessWrapper] = [ _SubprocessWrapper(self._proc, "RustFrontend") ] - self._finalizer = weakref.finalize(self, _shutdown_subprocesses, self.processes) + self._finalizer = weakref.finalize( + self, + _shutdown_subprocesses_and_cleanup_listeners, + self.processes, + self._listeners, + ) def shutdown(self, timeout: float | None = None) -> None: if self._finalizer.detach() is not None: - _shutdown_subprocesses(self.processes, timeout=timeout) + _shutdown_subprocesses_and_cleanup_listeners( + self.processes, self._listeners, timeout=timeout + ) class _SubprocessWrapper: @@ -533,6 +494,18 @@ def _shutdown_subprocesses( logger.debug_once("[shutdown] Subprocess manager: complete") +def _shutdown_subprocesses_and_cleanup_listeners( + procs: list[_SubprocessWrapper], + listeners: list[ZmqListener], + timeout: float | None = None, +) -> None: + try: + _shutdown_subprocesses(procs, timeout=timeout) + finally: + for listener in listeners: + listener.cleanup() + + def run_api_server_worker_proc( listen_address, sock, args, client_config=None, **uvicorn_kwargs ) -> None: @@ -687,6 +660,18 @@ def shutdown(procs: list[BaseProcess], timeout: float | None = None) -> None: logger.debug_once("[shutdown] Process manager: complete") +def _shutdown_processes_and_cleanup_listeners( + procs: list[BaseProcess], + listeners: list[ZmqListener], + timeout: float | None = None, +) -> None: + try: + shutdown(procs, timeout=timeout) + finally: + for listener in listeners: + listener.cleanup() + + def copy_slice( from_tensor: torch.Tensor, to_tensor: torch.Tensor, length: int ) -> torch.Tensor: diff --git a/vllm/v1/watermarking/gumbel.py b/vllm/v1/watermarking/gumbel.py index 655267d1e4..2f953dd389 100644 --- a/vllm/v1/watermarking/gumbel.py +++ b/vllm/v1/watermarking/gumbel.py @@ -221,7 +221,7 @@ class DualKeyGumbelWatermarkDetector(GumbelWatermarkDetector): p_value_threshold: float = 0.01, prf: WatermarkPRFName = "philox", deduplicate_contexts: bool = True, - alpha: float = 0.2, + alpha: float = 0.1, ) -> None: if not 0 <= alpha <= 1: raise ValueError("alpha must be between 0 and 1") diff --git a/vllm/v1/worker/cpu_worker.py b/vllm/v1/worker/cpu_worker.py index e36b15300b..b103215d65 100644 --- a/vllm/v1/worker/cpu_worker.py +++ b/vllm/v1/worker/cpu_worker.py @@ -268,7 +268,9 @@ class CPUWorker(Worker): self.model_runner.warming_up_model() # Reset the seed to ensure that the random state is not affected by # the model initialization and profiling. - set_random_seed(self.model_config.seed) + set_random_seed( + self.model_config.seed, self.parallel_config.data_parallel_index + ) return CompilationTimes( language_model=self.compilation_config.compilation_time, encoder=self.compilation_config.encoder_compilation_time, diff --git a/vllm/v1/worker/encoder_cudagraph.py b/vllm/v1/worker/encoder_cudagraph.py index 3b587ad71f..51c832001c 100644 --- a/vllm/v1/worker/encoder_cudagraph.py +++ b/vllm/v1/worker/encoder_cudagraph.py @@ -23,6 +23,7 @@ from vllm.model_executor.models.interfaces import ( from vllm.model_executor.models.utils import scatter_output_slices from vllm.model_executor.models.vision import get_load_balance_assignment from vllm.utils.gpu_sync_debug import gpu_sync_allowed +from vllm.utils.math_utils import round_up_to from vllm.utils.torch_utils import current_stream from vllm.v1.worker.encoder_cudagraph_defs import ( ENCODER_CUDAGRAPH_AXIS_KEYS_KWARG, @@ -347,17 +348,17 @@ class EncoderCudaGraphManager: def _find_smallest_fitting_budget_given_tokens( self, total_tokens: int, budgets: list[int] | None = None ) -> int | None: - """Find smallest budget >= total_tokens. - - Returns: - Token budget if found, None if no fitting budget. - - """ + """Find smallest budget >= total_tokens, or None if none fits.""" budgets = budgets if budgets is not None else self.token_budgets - for budget in budgets: - if budget >= total_tokens: - return budget - return None + return round_up_to(budgets, total_tokens) + + def _path_fitting_budget(self, path: str, tokens: int) -> int | None: + """Smallest captured budget on a path; 0 when idle, None if oversized.""" + if tokens == 0: + return 0 + return self._find_smallest_fitting_budget_given_tokens( + tokens, self.path_token_budgets[path] + ) def _get_item_specs(self, mm_kwargs: dict[str, Any]) -> list[EncoderItemSpec]: """Get item specs from the model.""" @@ -479,24 +480,38 @@ class EncoderCudaGraphManager: if not current_batch: return path_budgets = { - path: ( - 0 - if current_tokens[path] == 0 - else self._find_smallest_fitting_budget_given_tokens( - current_tokens[path], self.path_token_budgets[path] - ) - ) + path: self._path_fitting_budget(path, current_tokens[path]) for path in paths } batches.append((list(current_batch), path_budgets)) + def should_defer_item(item_tokens: dict[str, int]) -> bool: + """Whether closing the batch strictly reduces summed budgets.""" + for path in paths: + item = item_tokens[path] + if item == 0: + continue + cur = current_tokens[path] + b_cur = self._path_fitting_budget(path, cur) + b_alone = self._path_fitting_budget(path, item) + b_merged = self._path_fitting_budget(path, cur + item) + # Only called when the item fits; lookups land on a budget. + assert ( + b_cur is not None and b_alone is not None and b_merged is not None + ) + # Only defer items that alone outweigh the whole current + # batch; splitting smaller ones fragments flood packing. + if b_cur + b_alone < b_merged and b_alone >= b_cur: + return True + return False + for orig_idx in sorted_indices: item_tokens = {path: per_item_path_tokens[path][orig_idx] for path in paths} fits = len(current_batch) < self.max_batch_size and all( current_tokens[path] + item_tokens[path] <= max_path_budgets[path] for path in paths ) - if current_batch and not fits: + if current_batch and (not fits or should_defer_item(item_tokens)): append_current_batch() current_batch = [] current_tokens = dict.fromkeys(paths, 0) diff --git a/vllm/v1/worker/gpu/async_utils.py b/vllm/v1/worker/gpu/async_utils.py index 647b0c297c..0d6481585e 100644 --- a/vllm/v1/worker/gpu/async_utils.py +++ b/vllm/v1/worker/gpu/async_utils.py @@ -262,7 +262,7 @@ class AsyncPoolingOutput(AsyncModelRunnerOutput): def async_copy_to_np(x: torch.Tensor) -> np.ndarray: - return x.to("cpu", non_blocking=True).numpy() + return x.to("cpu", non_blocking=True, copy=True).numpy() @contextlib.contextmanager diff --git a/vllm/v1/worker/gpu/attn_utils.py b/vllm/v1/worker/gpu/attn_utils.py index 4fa308bcf4..8aa9ba3cb8 100644 --- a/vllm/v1/worker/gpu/attn_utils.py +++ b/vllm/v1/worker/gpu/attn_utils.py @@ -8,6 +8,7 @@ from typing import TYPE_CHECKING, Any, cast import numpy as np import torch +import vllm.envs as envs from vllm.config import ( VllmConfig, get_layers_from_vllm_config, @@ -224,6 +225,14 @@ def init_attn_backend( for layer_name in layer_names: attn_backend = attn_layers[layer_name].get_attn_backend() + if ( + envs.VLLM_BATCH_INVARIANT + and not attn_backend.supports_batch_invariance() + ): + raise RuntimeError( + "VLLM batch_invariant mode is not supported for " + f"{attn_backend.get_name()}." + ) if layer_name in fast_prefill_eligible_layers: attn_backend = create_fast_prefill_custom_backend( "FastPrefill", attn_backend diff --git a/vllm/v1/worker/gpu/buffer_utils.py b/vllm/v1/worker/gpu/buffer_utils.py index 0731072e59..32d997358c 100644 --- a/vllm/v1/worker/gpu/buffer_utils.py +++ b/vllm/v1/worker/gpu/buffer_utils.py @@ -54,8 +54,8 @@ class NonUvaBuffer: def uva(self, n: int | None = None) -> torch.Tensor: if n is None: - return self._uva.copy_(self.cpu) - return self._uva[:n].copy_(self.cpu[:n]) + return self._uva.copy_(self.cpu, non_blocking=True) + return self._uva[:n].copy_(self.cpu[:n], non_blocking=True) class UvaBufferPool: diff --git a/vllm/v1/worker/gpu/cudagraph_utils.py b/vllm/v1/worker/gpu/cudagraph_utils.py index 6778fea729..dda717cc49 100644 --- a/vllm/v1/worker/gpu/cudagraph_utils.py +++ b/vllm/v1/worker/gpu/cudagraph_utils.py @@ -272,6 +272,7 @@ class CudaGraphManager: if ( speculative_config and speculative_config.uses_dynamic_speculative_decoding() + and self.decode_query_len > self.vllm_config.num_speculative_tokens ): # decode_query_len = num_speculative_steps + num_new_sampled_tokens # _per_step. Recover num_new_sampled_tokens_per_step diff --git a/vllm/v1/worker/gpu/mm/encoder_runner.py b/vllm/v1/worker/gpu/mm/encoder_runner.py index 16e42d140a..4d7ab92013 100644 --- a/vllm/v1/worker/gpu/mm/encoder_runner.py +++ b/vllm/v1/worker/gpu/mm/encoder_runner.py @@ -150,9 +150,12 @@ class EncoderRunner: def execute_mm_encoder( self, mm_kwargs: list[tuple[str, MultiModalKwargsItem]] ) -> list[torch.Tensor]: + # Stable-sort items by modality so each modality is encoded in as few + # batches as possible. The returned outputs are restored to input order. + order = sorted(range(len(mm_kwargs)), key=lambda i: mm_kwargs[i][0]) encoder_outputs: list[torch.Tensor] = [] for modality, num_items, mm_kwargs_batch in group_and_batch_mm_kwargs( - mm_kwargs, device=self.device, pin_memory=PIN_MEMORY + [mm_kwargs[i] for i in order], device=self.device, pin_memory=PIN_MEMORY ): cg_manager = self.cudagraph_manager cudagraph_output = ( @@ -169,7 +172,7 @@ class EncoderRunner: ) sanity_check_mm_encoder_outputs(batch_outputs, expected_num_items=num_items) encoder_outputs.extend(batch_outputs) - return encoder_outputs + return [encoder_outputs[i] for i in np.argsort(order)] @contextmanager def timed_encoder_operation(self, request_ids: Collection[str]): diff --git a/vllm/v1/worker/gpu/mm/lora.py b/vllm/v1/worker/gpu/mm/lora.py index 819f3b1cb3..e73e028f30 100644 --- a/vllm/v1/worker/gpu/mm/lora.py +++ b/vllm/v1/worker/gpu/mm/lora.py @@ -6,6 +6,7 @@ import numpy as np from vllm.lora.layers import LoRAMapping, LoRAMappingType from vllm.lora.worker_manager import WorkerLoRAManager +from vllm.multimodal.inputs import MultiModalFeatureSpec from vllm.v1.worker.gpu.lora_utils import LoraState from vllm.v1.worker.gpu.mm.encoder_cache import EncoderCache @@ -31,7 +32,7 @@ def set_active_mm_loras( encoder_token_counts: list[int] = [] connector_token_counts: list[int] = [] - # iterate through images + mm_items: list[tuple[int, MultiModalFeatureSpec]] = [] for req_id, encoder_input_ids in scheduled_encoder_inputs.items(): req_idx = req_id_to_index.get(req_id) if req_idx is None: @@ -39,27 +40,29 @@ def set_active_mm_loras( lora_id = int(lora_state.lora_ids[req_idx]) mm_features = encoder_cache.mm_features[req_id] - - # iterate through visual tokens - for mm_input_id in encoder_input_ids: - pos_info = mm_features[mm_input_id].mm_position - - tower_tokens, connector_tokens = model.get_mm_lora_token_counts( - modality=mm_features[mm_input_id].modality, - mm_kwargs=mm_features[mm_input_id].data, - num_mm_embeds=pos_info.get_num_embeds(), - ) - - prompt_lora_mapping.append(lora_id) - token_lora_mapping.extend([lora_id] * tower_tokens) - encoder_token_counts.append(tower_tokens) - connector_token_counts.append(connector_tokens) + mm_items.extend((lora_id, mm_features[i]) for i in encoder_input_ids) if lora_id > 0: lora_request = lora_state.lora_requests.get(req_id) if lora_request is not None: lora_requests.add(lora_request) + # Runs items sorted by modality. + mm_items.sort(key=lambda x: x[1].modality) + for lora_id, mm_feature in mm_items: + pos_info = mm_feature.mm_position + + tower_tokens, connector_tokens = model.get_mm_lora_token_counts( + modality=mm_feature.modality, + mm_kwargs=mm_feature.data, + num_mm_embeds=pos_info.get_num_embeds(), + ) + + prompt_lora_mapping.append(lora_id) + token_lora_mapping.extend([lora_id] * tower_tokens) + encoder_token_counts.append(tower_tokens) + connector_token_counts.append(connector_tokens) + if not prompt_lora_mapping: return diff --git a/vllm/v1/worker/gpu/model_runner.py b/vllm/v1/worker/gpu/model_runner.py index 1b9d0608e2..61d6197ceb 100644 --- a/vllm/v1/worker/gpu/model_runner.py +++ b/vllm/v1/worker/gpu/model_runner.py @@ -265,13 +265,32 @@ class GPUModelRunner(LoRAModelRunnerMixin): self.encoder_cache = EncoderCache() self.ec_connector = get_ec_connector(vllm_config, self.encoder_cache) + self.num_speculative_steps = vllm_config.num_speculative_tokens + num_prefill_lookahead = max(1, vllm_config.num_prefill_lookahead_tokens) + use_dense_all_token_ids = ( + self.speculative_config is not None and self.speculative_config.use_ngram() + ) + + # General request states. + self.req_states = RequestState( + max_num_reqs=self.max_num_reqs, + max_model_len=self.max_model_len, + max_num_batched_tokens=self.max_num_tokens, + num_speculative_steps=self.num_speculative_steps, + vocab_size=self.vocab_size, + device=self.device, + num_prefill_lookahead=num_prefill_lookahead, + use_dense_all_token_ids=use_dense_all_token_ids, + ) + # Speculative decoding. self.speculator = None self.use_aux_hidden_state_outputs = False - self.num_speculative_steps = vllm_config.num_speculative_tokens if self.speculative_config is not None: if self.is_last_pp_rank: - self.speculator = init_speculator(self.vllm_config, self.device) + self.speculator = init_speculator( + self.vllm_config, self.device, self.req_states + ) if self.speculative_config.method in ( "eagle3", @@ -291,20 +310,7 @@ class GPUModelRunner(LoRAModelRunnerMixin): self.is_pooling_model = self.model_config.runner_type == "pooling" self.pooling_runner: PoolingRunner | None = None - num_prefill_lookahead = max(1, vllm_config.num_prefill_lookahead_tokens) - self.step_timing = StepTimingCollector() - - # General request states. - self.req_states = RequestState( - max_num_reqs=self.max_num_reqs, - max_model_len=self.max_model_len, - max_num_batched_tokens=self.max_num_tokens, - num_speculative_steps=self.num_speculative_steps, - vocab_size=self.vocab_size, - device=self.device, - num_prefill_lookahead=num_prefill_lookahead, - ) self.adaptive_verification: AdaptiveVerificationManager | None = None self.input_buffers = InputBuffers( max_num_reqs=self.max_num_reqs, @@ -482,7 +488,7 @@ class GPUModelRunner(LoRAModelRunnerMixin): custom = self.model_state.custom_sampler(self.sampler) if custom: - self.vllm_config._check_watermarking_unsupported(custom_sampler=True) + self.vllm_config._check_supports_watermarking(custom_sampler=True) self.sampler, self.rejection_sampler = custom elif self.speculative_config is not None: self.rejection_sampler = RejectionSampler( @@ -724,7 +730,6 @@ class GPUModelRunner(LoRAModelRunnerMixin): attn_cg_support.min_cg_attn_backend, self.decode_query_len, use_v2_model_runner=True, - tensor_parallel_size=self.parallel_config.tensor_parallel_size, kv_cache_config=self.kv_cache_config, max_num_reqs=self.max_num_reqs, is_profiling=is_profiling, @@ -772,6 +777,7 @@ class GPUModelRunner(LoRAModelRunnerMixin): self.kv_caches = [ cache for cache in kv_caches_dict.values() if cache.device == self.device ] + self.model_state.initialize_kv_cache(self.kv_cache_config, self.block_tables) if is_profiling: self.kv_connector = NO_OP_KV_CONNECTOR else: @@ -813,7 +819,7 @@ class GPUModelRunner(LoRAModelRunnerMixin): "skip_attn must only be True for initial memory profiling." ) - # Create a dummy scheduler output. Plain draft-model speculation adds + # Create a dummy scheduler output. Standalone AR speculation adds # one correction slot per request during prefill. The scheduler # accounts for these slots, while dummy runs bypass the scheduler. # Adjust the dummy token count only when the expanded draft batch @@ -909,14 +915,7 @@ class GPUModelRunner(LoRAModelRunnerMixin): ) mm_inputs = [], all_false - # Let the target override the hidden state fed to the drafter - # (e.g. DeepSeek V4 MTP needs the pre-hc_head residual). The - # target returns a persistent buffer sized at max_num_batched_tokens; - # slice to the active token count that propose() expects. - spec_hidden_states = hidden_states - if hasattr(self.model, "get_mtp_target_hidden_states"): - pre_hc_hidden_states = self.model.get_mtp_target_hidden_states() - spec_hidden_states = pre_hc_hidden_states[: hidden_states.shape[0]] # type: ignore[union-attr] + spec_hidden_states = self._get_drafter_hidden_states(hidden_states) if isinstance(self.sampler, GPUWatermarkSampler): self.speculator.prepare_watermarking( self.sampler, input_batch.idx_mapping @@ -1077,6 +1076,12 @@ class GPUModelRunner(LoRAModelRunnerMixin): self.lora_config, self ), ) + self.model_state.capture_inner_cudagraphs( + input_buffers, + self.block_tables, + self.attn_groups, + self.kv_cache_config, + ) if self.speculator is not None: with use_workspace_lane(self._draft_workspace_lane): self.speculator.capture() @@ -2057,6 +2062,7 @@ class GPUModelRunner(LoRAModelRunnerMixin): finished_req_ids=finished_req_ids, ec_connector_output=ec_connector_output, cudagraph_stats=cudagraph_stats, + num_spec_tokens_to_schedule=scheduler_output.num_spec_tokens_to_schedule, ) if not self.is_last_pp_rank: @@ -2068,6 +2074,24 @@ class GPUModelRunner(LoRAModelRunnerMixin): ) return None + def _get_drafter_hidden_states(self, hidden_states: torch.Tensor) -> torch.Tensor: + """Hidden states fed to the drafter. + + Targets such as DeepSeek V4 expose the pre-hc_head residual through + get_mtp_target_hidden_states(). The buffer is sized at + max_num_batched_tokens and only allocated for drafters that consume + target hidden states, so None means "use the regular hidden states". + """ + get_target_hidden_states = getattr( + self.model, "get_mtp_target_hidden_states", None + ) + if get_target_hidden_states is None: + return hidden_states + target_hidden_states = get_target_hidden_states() + if target_hidden_states is None: + return hidden_states + return target_hidden_states[: hidden_states.shape[0]] + @torch.inference_mode() @step_eplb_after() def sample_tokens( @@ -2086,6 +2110,7 @@ class GPUModelRunner(LoRAModelRunnerMixin): finished_req_ids = self.execute_model_state.finished_req_ids ec_connector_output = self.execute_model_state.ec_connector_output cudagraph_stats = self.execute_model_state.cudagraph_stats + num_spec_tokens = self.execute_model_state.num_spec_tokens_to_schedule self.execute_model_state = None if not self.is_last_pp_rank: @@ -2212,14 +2237,7 @@ class GPUModelRunner(LoRAModelRunnerMixin): self.speculator.observe_verification( input_batch.idx_mapping, num_sampled, num_rejected ) - # Let the target override the hidden state fed to the drafter - # (e.g. DeepSeek V4 MTP needs the pre-hc_head residual). The - # target returns a persistent buffer sized at max_num_batched_tokens; - # slice to the active token count that propose() expects. - spec_hidden_states = draft_hidden_states - if hasattr(self.model, "get_mtp_target_hidden_states"): - pre_hc_hidden_states = self.model.get_mtp_target_hidden_states() - spec_hidden_states = pre_hc_hidden_states[: draft_hidden_states.size(0)] + spec_hidden_states = self._get_drafter_hidden_states(draft_hidden_states) if isinstance(self.sampler, GPUWatermarkSampler): self.speculator.prepare_watermarking( self.sampler, input_batch.idx_mapping @@ -2237,9 +2255,12 @@ class GPUModelRunner(LoRAModelRunnerMixin): self.req_states.next_prefill_tokens, self.sampler.sampling_states.temperature.gpu, self.sampler.sampling_states.seeds.gpu, + num_speculative_tokens=num_spec_tokens, dp_sync=dp_sync, mm_inputs=mm_inputs, ) + if num_spec_tokens < self.num_speculative_steps: + draft_tokens[:, num_spec_tokens:] = -1 self.req_states.draft_tokens[input_batch.idx_mapping] = draft_tokens if self.adaptive_verification is not None: self.adaptive_verification.record_confidences( @@ -2251,7 +2272,7 @@ class GPUModelRunner(LoRAModelRunnerMixin): # not have a speculator (i.e. self.speculator is None) self.draft_tokens_handler.set_draft_tokens( input_batch, - self.req_states.draft_tokens[input_batch.idx_mapping], + self.req_states.draft_tokens[input_batch.idx_mapping, :num_spec_tokens], ) if self.pp_handler is not None: self.pp_handler.broadcast_drafts( @@ -2396,6 +2417,7 @@ class ExecuteModelState(NamedTuple): finished_req_ids: set[str] ec_connector_output: ECConnectorOutput | None cudagraph_stats: CUDAGraphStat | None + num_spec_tokens_to_schedule: int class BatchReqState(NamedTuple): diff --git a/vllm/v1/worker/gpu/model_states/interface.py b/vllm/v1/worker/gpu/model_states/interface.py index ee3e519a6e..3db3fc9e7a 100644 --- a/vllm/v1/worker/gpu/model_states/interface.py +++ b/vllm/v1/worker/gpu/model_states/interface.py @@ -17,7 +17,8 @@ from vllm.v1.attention.backend import AttentionCGSupport from vllm.v1.core.sched.output import NewRequestData from vllm.v1.kv_cache_interface import KVCacheConfig from vllm.v1.worker.encoder_cudagraph import EncoderCudaGraphManager -from vllm.v1.worker.gpu.input_batch import InputBatch +from vllm.v1.worker.gpu.block_table import BlockTables +from vllm.v1.worker.gpu.input_batch import InputBatch, InputBuffers from vllm.v1.worker.gpu.mm.encoder_cache import EncoderCache from vllm.v1.worker.gpu.mm.encoder_runner import EncoderRunner from vllm.v1.worker.gpu.states import RequestState @@ -126,6 +127,22 @@ class ModelState(ABC): def apply_staged_writes(self) -> None: return None + def initialize_kv_cache( + self, kv_cache_config: KVCacheConfig, block_tables: BlockTables + ) -> None: + """Hook run after the KV cache tensors are allocated and bound.""" + return None + + def capture_inner_cudagraphs( + self, + input_buffers: InputBuffers, + block_tables: BlockTables, + attn_groups: list[list[AttentionGroup]], + kv_cache_config: KVCacheConfig, + ) -> None: + """Capture the CUDA graphs this state runs inside the model's forward.""" + return None + def get_additional_cg_support(self) -> tuple[AttentionCGSupport, str | None]: """Cudagraph support of attention groups this ModelState builds outside ``init_attn_backend`` (e.g. encoder-only layers). diff --git a/vllm/v1/worker/gpu/model_states/mamba_hybrid.py b/vllm/v1/worker/gpu/model_states/mamba_hybrid.py index b8c44b16a9..e5bcc92eb4 100644 --- a/vllm/v1/worker/gpu/model_states/mamba_hybrid.py +++ b/vllm/v1/worker/gpu/model_states/mamba_hybrid.py @@ -21,6 +21,7 @@ from vllm.v1.core.sched.output import NewRequestData from vllm.v1.kv_cache_interface import KVCacheConfig, MambaSpec from vllm.v1.utils import CpuGpuBuffer from vllm.v1.worker.gpu.attn_utils import build_attn_metadata +from vllm.v1.worker.gpu.block_table import BlockTables from vllm.v1.worker.gpu.input_batch import InputBatch from vllm.v1.worker.gpu.mm.encoder_cache import EncoderCache from vllm.v1.worker.gpu.model_states.default import DefaultModelState @@ -114,8 +115,13 @@ class MambaHybridModelState(DefaultModelState): self.num_accepted_tokens_gpu[req_index].fill_(1) if self._align_mode: # Seed the running state block from the resumed/prefilled position. + # The column is in mamba blocks. cache_config.block_size is the + # min over prefix-cacheable groups, so a drafter group with its own + # smaller block (e.g. DFlash) pulls it below mamba_block_size. + mamba_block_size = self.cache_config.mamba_block_size + assert mamba_block_size is not None self._mamba_state_idx_gpu[req_index].fill_( - (new_req_data.num_computed_tokens - 1) // self.cache_config.block_size + (new_req_data.num_computed_tokens - 1) // mamba_block_size ) def _get_mamba_group_info( @@ -134,47 +140,42 @@ class MambaHybridModelState(DefaultModelState): self._mamba_spec = mamba_spec return self._mamba_group_ids, self._mamba_spec - def _ensure_align_ctx( - self, - kv_cache_config: KVCacheConfig, - mamba_group_ids: list[int], - block_tables: tuple[torch.Tensor, ...], - ) -> MambaSpecDecodeGPUContext: + def initialize_kv_cache( + self, kv_cache_config: KVCacheConfig, block_tables: BlockTables + ) -> None: + if not self._align_mode: + return + mamba_group_ids, _ = self._get_mamba_group_info(kv_cache_config) if self._mamba_state_copy_funcs is None: mamba_groups = get_mamba_groups(kv_cache_config) mamba_types = {spec.mamba_type for spec in mamba_groups} copy_funcs = self.model.get_mamba_state_copy_funcs(mamba_types) validate_mamba_state_copy_funcs(mamba_groups, copy_funcs) self._mamba_state_copy_funcs = copy_funcs - copy_funcs = self._mamba_state_copy_funcs - if self._mamba_ctx is None: - # Both SD and DS conv layouts support a >0 spec-decode shift: the - # fused pre-copy kernel (``_copy_mamba_state_block``) applies the - # ``token_bias = num_accepted - 1`` window shift per conv layout - # (SD: contiguous slice; DS: per-dim-row strided slice), matching - # the V1 ``get_conv_copy_spec`` semantics. - self._mamba_ctx = MambaSpecDecodeGPUContext.create( - max_num_reqs=self.max_num_reqs, - kv_cache_config=kv_cache_config, - copy_funcs=copy_funcs, - device=self.device, - make_buffer=lambda n, dtype: CpuGpuBuffer( - n, dtype=dtype, device=self.device - ), - ) - ctx = self._mamba_ctx - if not ctx.is_initialized: - forward_context = self.vllm_config.compilation_config.static_forward_context - # block_tables are batch-order slices of the persistent - # input_block_tables (stable data_ptr), so the metadata is captured - # once here and reused across steps. - ctx.initialize_from_forward_context( - kv_cache_config, - forward_context, - copy_funcs, - [block_tables[gid] for gid in mamba_group_ids], - ) - return ctx + # Both SD and DS conv layouts support a >0 spec-decode shift: the + # fused pre-copy kernel (``_copy_mamba_state_block``) applies the + # ``token_bias = num_accepted - 1`` window shift per conv layout + # (SD: contiguous slice; DS: per-dim-row strided slice), matching + # the V1 ``get_conv_copy_spec`` semantics. + self._mamba_ctx = MambaSpecDecodeGPUContext.create( + max_num_reqs=self.max_num_reqs, + kv_cache_config=kv_cache_config, + copy_funcs=self._mamba_state_copy_funcs, + device=self.device, + make_buffer=lambda n, dtype: CpuGpuBuffer( + n, dtype=dtype, device=self.device + ), + ) + # Under PP a step's postprocess runs after later steps re-gathered the + # batch-ordered input tables, so state copies read the per-request-slot + # tables. Aligned state indices are for the current batch only. + self._mamba_ctx.initialize_from_forward_context( + kv_cache_config, + self.vllm_config.compilation_config.static_forward_context, + self._mamba_state_copy_funcs, + [block_tables.block_tables[gid].gpu for gid in mamba_group_ids], + [block_tables.input_block_tables[gid] for gid in mamba_group_ids], + ) def preprocess_state( self, @@ -194,8 +195,9 @@ class MambaHybridModelState(DefaultModelState): num_reqs = input_batch.num_reqs if num_reqs == 0: return - mamba_group_ids, mamba_spec = self._get_mamba_group_info(kv_cache_config) - ctx = self._ensure_align_ctx(kv_cache_config, mamba_group_ids, block_tables) + _, mamba_spec = self._get_mamba_group_info(kv_cache_config) + ctx = self._mamba_ctx + assert ctx is not None # The state-advance + pre-copy kernels run every step; they fast-exit per # request when src_col < 0 or src_col == dst_col, so no copy happens on @@ -277,18 +279,19 @@ class MambaHybridModelState(DefaultModelState): ] # GDN uses >= 0 to select spec-decode rows, so non-decode rows - # need the -1 sentinel rather than a raw zero draft count. + # need the -1 sentinel. Decode rows without drafts stay spec rows: + # only that path applies the previous step's accepted offset. num_decode_draft_tokens_np = np.full(num_reqs, -1, dtype=np.int32) num_draft_tokens_per_req = input_batch.num_draft_tokens_per_req - if num_draft_tokens_per_req is not None: - # Test request state, not num_scheduled_tokens == draft_count+1: - # adaptive rewrites num_scheduled_tokens to an even split, so that - # equality rarely holds and would demote every verify row to decode. - is_decode = ~is_prefilling_np & (input_batch.num_scheduled_tokens > 0) - spec_decode_mask = (num_draft_tokens_per_req > 0) & is_decode - num_decode_draft_tokens_np[: input_batch.num_reqs] = np.where( - spec_decode_mask, num_draft_tokens_per_req, -1 - ) + if num_draft_tokens_per_req is None: + num_draft_tokens_per_req = np.zeros(input_batch.num_reqs, np.int32) + # Test request state, not num_scheduled_tokens == draft_count+1: + # adaptive rewrites num_scheduled_tokens to an even split, so that + # equality rarely holds and would demote every verify row to decode. + is_decode = ~is_prefilling_np & (input_batch.num_scheduled_tokens > 0) + num_decode_draft_tokens_np[: input_batch.num_reqs] = np.where( + is_decode, num_draft_tokens_per_req, -1 + ) num_decode_draft_tokens_cpu = torch.from_numpy(num_decode_draft_tokens_np) if self._align_mode: @@ -300,10 +303,8 @@ class MambaHybridModelState(DefaultModelState): if hasattr(builder, "mamba_aligned_state_indices"): aligned_index_builders.append((group_idx, builder)) if aligned_index_builders: - ctx = self._ensure_align_ctx( - kv_cache_config, mamba_group_ids, block_tables - ) - all_group_indices = ctx.compute_aligned_state_indices( + assert self._mamba_ctx is not None + all_group_indices = self._mamba_ctx.compute_aligned_state_indices( input_batch.seq_lens, num_reqs ) for group_idx, builder in aligned_index_builders: diff --git a/vllm/v1/worker/gpu/spec_decode/__init__.py b/vllm/v1/worker/gpu/spec_decode/__init__.py index 0616f12dd5..fc028941cf 100644 --- a/vllm/v1/worker/gpu/spec_decode/__init__.py +++ b/vllm/v1/worker/gpu/spec_decode/__init__.py @@ -1,11 +1,21 @@ # SPDX-License-Identifier: Apache-2.0 # SPDX-FileCopyrightText: Copyright contributors to the vLLM project +from typing import TYPE_CHECKING + import torch from vllm.config import VllmConfig +if TYPE_CHECKING: + from vllm.v1.worker.gpu.states import RequestState -def init_speculator(vllm_config: VllmConfig, device: torch.device): + +def init_speculator( + vllm_config: VllmConfig, + device: torch.device, + req_states: "RequestState", +): + """Build the speculator for this config.""" speculative_config = vllm_config.speculative_config assert speculative_config is not None if speculative_config.method == "extract_hidden_states": @@ -61,10 +71,16 @@ def init_speculator(vllm_config: VllmConfig, device: torch.device): return EagleSpeculator(vllm_config, device) elif speculative_config.uses_draft_model(): - from vllm.v1.worker.gpu.spec_decode.draft_model.speculator import ( - PlainDraftModelSpeculator, + from vllm.v1.worker.gpu.spec_decode.standalone_ar.speculator import ( + StandaloneARSpeculator, ) - return PlainDraftModelSpeculator(vllm_config, device) + return StandaloneARSpeculator(vllm_config, device) + elif speculative_config.use_ngram(): + from vllm.v1.worker.gpu.spec_decode.ngram.speculator import ( + NgramGPUSpeculator, + ) + + return NgramGPUSpeculator(vllm_config, device, req_states) else: raise NotImplementedError(f"{speculative_config.method} is not supported yet.") diff --git a/vllm/v1/worker/gpu/spec_decode/dflash/speculator.py b/vllm/v1/worker/gpu/spec_decode/dflash/speculator.py index 57110e0b05..d59d87ae7e 100644 --- a/vllm/v1/worker/gpu/spec_decode/dflash/speculator.py +++ b/vllm/v1/worker/gpu/spec_decode/dflash/speculator.py @@ -360,6 +360,7 @@ class DFlashSpeculator(DraftModelSpeculator): skip_attn_for_dummy_run: bool = False, mm_inputs: tuple[list[torch.Tensor], torch.Tensor] | None = None, is_profile: bool = False, + num_speculative_tokens: int | None = None, ) -> torch.Tensor: num_reqs = input_batch.num_reqs num_target_tokens = input_batch.num_tokens diff --git a/vllm/v1/worker/gpu/spec_decode/eagle/speculator.py b/vllm/v1/worker/gpu/spec_decode/eagle/speculator.py index e878872e62..0611d42b1c 100644 --- a/vllm/v1/worker/gpu/spec_decode/eagle/speculator.py +++ b/vllm/v1/worker/gpu/spec_decode/eagle/speculator.py @@ -3,13 +3,13 @@ import torch.nn as nn -from vllm.v1.worker.gpu.spec_decode.autoregressive.speculator import ( - AutoRegressiveSpeculator, -) from vllm.v1.worker.gpu.spec_decode.eagle.utils import load_eagle_model +from vllm.v1.worker.gpu.spec_decode.target_dependent_ar.speculator import ( + TargetDependentARSpeculator, +) -class EagleSpeculator(AutoRegressiveSpeculator): +class EagleSpeculator(TargetDependentARSpeculator): def load_draft_model( self, target_model: nn.Module, diff --git a/vllm/v1/worker/gpu/spec_decode/extract_hidden_states.py b/vllm/v1/worker/gpu/spec_decode/extract_hidden_states.py index c411eeae55..5afbe640b2 100644 --- a/vllm/v1/worker/gpu/spec_decode/extract_hidden_states.py +++ b/vllm/v1/worker/gpu/spec_decode/extract_hidden_states.py @@ -103,6 +103,7 @@ class ExtractHiddenStatesSpeculator(DraftModelSpeculator): skip_attn_for_dummy_run: bool = False, mm_inputs: tuple[list[torch.Tensor], torch.Tensor] | None = None, is_profile: bool = False, + num_speculative_tokens: int | None = None, ) -> torch.Tensor: del ( last_hidden_states, diff --git a/vllm/v1/worker/gpu/spec_decode/gemma4/speculator.py b/vllm/v1/worker/gpu/spec_decode/gemma4/speculator.py index 6f76e5055a..9d553830ae 100644 --- a/vllm/v1/worker/gpu/spec_decode/gemma4/speculator.py +++ b/vllm/v1/worker/gpu/spec_decode/gemma4/speculator.py @@ -17,8 +17,8 @@ from vllm.distributed.parallel_state import get_pp_group from vllm.logger import init_logger from vllm.model_executor.model_loader import get_model from vllm.model_executor.model_loader.utils import get_draft_load_config -from vllm.v1.worker.gpu.spec_decode.autoregressive.speculator import ( - AutoRegressiveSpeculator, +from vllm.v1.worker.gpu.spec_decode.target_dependent_ar.speculator import ( + TargetDependentARSpeculator, ) logger = init_logger(__name__) @@ -45,7 +45,7 @@ def _copy_target_kv_scales(attn: nn.Module, target_attn: nn.Module) -> None: getattr(attn, scale_name).copy_(getattr(target_attn, scale_name)) -class Gemma4Speculator(AutoRegressiveSpeculator): +class Gemma4Speculator(TargetDependentARSpeculator): @property def advance_draft_positions(self) -> bool: # Gemma4 MTP is Q-only and reads K/V from the target's existing cache. diff --git a/vllm/v1/worker/gpu/spec_decode/mtp/speculator.py b/vllm/v1/worker/gpu/spec_decode/mtp/speculator.py index d94702612d..a0064197f7 100644 --- a/vllm/v1/worker/gpu/spec_decode/mtp/speculator.py +++ b/vllm/v1/worker/gpu/spec_decode/mtp/speculator.py @@ -3,13 +3,13 @@ import torch.nn as nn -from vllm.v1.worker.gpu.spec_decode.autoregressive.speculator import ( - AutoRegressiveSpeculator, -) from vllm.v1.worker.gpu.spec_decode.eagle.utils import load_eagle_model +from vllm.v1.worker.gpu.spec_decode.target_dependent_ar.speculator import ( + TargetDependentARSpeculator, +) -class MTPSpeculator(AutoRegressiveSpeculator): +class MTPSpeculator(TargetDependentARSpeculator): share_mtp_topk_indices: bool = False def load_draft_model( diff --git a/vllm/v1/worker/gpu/spec_decode/multi_module_mtp/speculator.py b/vllm/v1/worker/gpu/spec_decode/multi_module_mtp/speculator.py index c7026acdb8..919e939c03 100644 --- a/vllm/v1/worker/gpu/spec_decode/multi_module_mtp/speculator.py +++ b/vllm/v1/worker/gpu/spec_decode/multi_module_mtp/speculator.py @@ -14,11 +14,11 @@ from vllm.v1.attention.backends.utils import PAD_SLOT_ID from vllm.v1.worker.gpu.attn_utils import build_slot_mappings_by_layer from vllm.v1.worker.gpu.dp_utils import DPSyncState, dispatch_cg_and_sync_dp from vllm.v1.worker.gpu.input_batch import InputBatch, InputBuffers -from vllm.v1.worker.gpu.spec_decode.autoregressive.cudagraph_utils import ( - SpeculatorCudaGraphManager, -) from vllm.v1.worker.gpu.spec_decode.eagle.utils import load_eagle_model from vllm.v1.worker.gpu.spec_decode.speculator import DraftModelSpeculator +from vllm.v1.worker.gpu.spec_decode.target_dependent_ar.cudagraph_utils import ( + SpeculatorCudaGraphManager, +) from vllm.v1.worker.utils import get_uniform_decode_token_count logger = init_logger(__name__) @@ -155,6 +155,7 @@ class MultiModuleMTPSpeculator(DraftModelSpeculator): skip_attn_for_dummy_run: bool = False, mm_inputs: tuple[list[torch.Tensor], torch.Tensor] | None = None, is_profile: bool = False, + num_speculative_tokens: int | None = None, ) -> torch.Tensor: num_reqs = input_batch.num_reqs seq_lens_cpu_upper_bound = input_batch.seq_lens_cpu_upper_bound diff --git a/vllm/v1/worker/gpu/spec_decode/ngram/__init__.py b/vllm/v1/worker/gpu/spec_decode/ngram/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/vllm/v1/worker/gpu/spec_decode/ngram/speculator.py b/vllm/v1/worker/gpu/spec_decode/ngram/speculator.py new file mode 100644 index 0000000000..7a61e3581a --- /dev/null +++ b/vllm/v1/worker/gpu/spec_decode/ngram/speculator.py @@ -0,0 +1,279 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +from __future__ import annotations + +from typing import TYPE_CHECKING, Any + +import torch + +from vllm.config import VllmConfig +from vllm.triton_utils import HAS_TRITON, tl, triton +from vllm.v1.worker.gpu.input_batch import InputBatch +from vllm.v1.worker.gpu.spec_decode.speculator import BaseSpeculator + +if TYPE_CHECKING: + from vllm.config.compilation import CUDAGraphMode + from vllm.v1.worker.gpu.dp_utils import DPSyncState + from vllm.v1.worker.gpu.states import RequestState + + +@triton.jit +def _ngram_scan_kernel( + token_ids_ptr, # *int32 [max_num_reqs, token_ids_stride] + token_ids_stride, + idx_mapping_ptr, # *int64 [B] batch_idx -> req_state_idx + total_len_ptr, # *int32 [max_num_reqs] + num_sampled_ptr, # *int32 [B] + scratch_ptr, # *int64 [B, scratch_stride] (output) + scratch_stride, + L, # int64 scalar (= max_model_len) + MIN_N: tl.constexpr, + MAX_N: tl.constexpr, + MAX_N_PO2: tl.constexpr, + BLOCK_L: tl.constexpr, +): + b = tl.program_id(0).to(tl.int64) + blk = tl.program_id(1).to(tl.int64) + Lp1 = tl.cast(L, tl.int64) + 1 + + req_state_idx = tl.load(idx_mapping_ptr + b).to(tl.int64) + seq_len = tl.load(total_len_ptr + req_state_idx).to(tl.int64) + num_sampled = tl.load(num_sampled_ptr + b) + eligible_row = (num_sampled > 0) & (seq_len >= MIN_N) + + scratch_off = b * scratch_stride + blk + + # Ineligible rows, and blocks fully past the last candidate match + # position, write 0 and exit. + if not (eligible_row & (blk * BLOCK_L <= seq_len - MIN_N - 1)): + tl.store(scratch_ptr + scratch_off, tl.zeros((), tl.int64)) + return + + row_off = req_state_idx * token_ids_stride + + # Load the length-MAX_N suffix once into registers. + suf_iota = tl.arange(0, MAX_N_PO2).to(tl.int64) + suf_pos = seq_len - MAX_N + suf_iota + suf_in_range = (suf_iota < MAX_N) & (suf_pos >= 0) & (suf_pos < seq_len) + suffix = tl.load( + token_ids_ptr + row_off + suf_pos, + mask=suf_in_range, + other=-1, + ).to(tl.int32) + + pos_iota = tl.arange(0, BLOCK_L).to(tl.int64) + pos = blk * BLOCK_L + pos_iota # ascending + + best_score = tl.zeros([BLOCK_L], dtype=tl.int64) + + for n_iter in tl.static_range(MIN_N, MAX_N + 1): + max_pos_n = seq_len - n_iter - 1 + match = (pos >= 0) & (pos <= max_pos_n) + for j in tl.static_range(0, n_iter): + tok = tl.load( + token_ids_ptr + row_off + (pos + j), + mask=match, + other=0, + ).to(tl.int32) + suf_idx = (MAX_N - n_iter) + j + suf_val = tl.sum(tl.where(suf_iota == suf_idx, suffix, 0)) + match = match & (tok == suf_val) + + # Pack (n, pos) so a single max yields longest-n, rightmost-pos. + cand = n_iter * Lp1 + pos + 1 + best_score = tl.where(match, cand, best_score) + + block_best = tl.max(best_score, axis=0) + tl.store(scratch_ptr + scratch_off, block_best) + + +@triton.jit +def _ngram_finalize_kernel( + token_ids_ptr, # *int32 [max_num_reqs, token_ids_stride] + token_ids_stride, + idx_mapping_ptr, # *int64 [B] + total_len_ptr, # *int32 [max_num_reqs] + num_sampled_ptr, # *int32 [B] + last_sampled_ptr, # *int64 [max_num_reqs] + scratch_ptr, # *int64 [B, scratch_stride] + scratch_stride, + drafts_ptr, # *int64 [B, K] (output) + L, + N_BLOCKS, + K: tl.constexpr, + K_PO2: tl.constexpr, + N_BLOCKS_PO2: tl.constexpr, +): + b = tl.program_id(0).to(tl.int64) + Lp1 = tl.cast(L, tl.int64) + 1 + NB = tl.cast(N_BLOCKS, tl.int64) + + req_state_idx = tl.load(idx_mapping_ptr + b).to(tl.int64) + + nb_iota = tl.arange(0, N_BLOCKS_PO2).to(tl.int64) + nb_in_range = nb_iota < NB + block_scores = tl.load( + scratch_ptr + b * scratch_stride + nb_iota, + mask=nb_in_range, + other=0, + ) + score = tl.max(block_scores, axis=0) + + seq_len = tl.load(total_len_ptr + req_state_idx).to(tl.int64) + num_sampled = tl.load(num_sampled_ptr + b) + last_tok = tl.load(last_sampled_ptr + req_state_idx) + + has_match = score > 0 + s1 = score - 1 + best_n = tl.where(has_match, s1 // Lp1, tl.zeros_like(s1)) + best_pos = tl.where(has_match, s1 - best_n * Lp1, tl.zeros_like(s1)) + draft_start = tl.where(has_match, best_pos + best_n, tl.zeros_like(s1)) + + tokens_avail = tl.maximum(seq_len - draft_start, 0) + write_ok = (num_sampled > 0) & has_match + + row_off = req_state_idx * token_ids_stride + k_iota = tl.arange(0, K_PO2).to(tl.int64) + k_in_range = k_iota < K + gather_idx = tl.minimum(draft_start + k_iota, tl.cast(L, tl.int64) - 1) + slot_valid = (k_iota < tokens_avail) & write_ok & k_in_range + gathered = tl.load( + token_ids_ptr + row_off + gather_idx, + mask=slot_valid, + other=0, + ).to(tl.int64) + # Invalid slots fall back to the last sampled token; they are verified as + # ordinary (rejectable) drafts, so the fill value only affects efficiency. + out = tl.where(slot_valid, gathered, last_tok) + tl.store(drafts_ptr + b * K + k_iota, out, mask=k_in_range) + + +class NgramGPUSpeculator(BaseSpeculator): + """V2-compatible GPU n-gram speculator.""" + + supports_mm_inputs = False + draft_logits = None + + def __init__( + self, + vllm_config: VllmConfig, + device: torch.device, + req_states: RequestState, + ): + if not HAS_TRITON: + raise RuntimeError("ngram_gpu speculative decoding requires Triton.") + spec = vllm_config.speculative_config + assert spec is not None + assert spec.prompt_lookup_min is not None, ( + "prompt_lookup_min must be configured for ngram_gpu" + ) + assert spec.prompt_lookup_max is not None, ( + "prompt_lookup_max must be configured for ngram_gpu" + ) + assert 1 <= spec.prompt_lookup_min <= spec.prompt_lookup_max + + self.vllm_config = vllm_config + self.device = device + self.req_states = req_states + self.speculative_config = spec + self.num_speculative_steps: int = spec.num_speculative_tokens + + self.min_n: int = spec.prompt_lookup_min + self.max_n: int = spec.prompt_lookup_max + + self.max_num_reqs: int = vllm_config.scheduler_config.max_num_seqs + self.max_model_len: int = vllm_config.model_config.max_model_len + + L = self.max_model_len + if L >= 1024: + self.block_l = 256 + elif L >= 256: + self.block_l = 128 + elif L >= 64: + self.block_l = 64 + else: + self.block_l = max(16, triton.next_power_of_2(max(L, 1))) + self.n_blocks = triton.cdiv(L, self.block_l) + + self.scratch = torch.zeros( + (self.max_num_reqs, self.n_blocks), dtype=torch.int64, device=device + ) + # Batch-ordered draft output, scattered into RequestState.draft_tokens + # by the model runner (same contract as the model-based speculators). + self.drafts = torch.zeros( + (self.max_num_reqs, self.num_speculative_steps), + dtype=torch.int64, + device=device, + ) + + def init_cudagraph_manager(self, cudagraph_mode: CUDAGraphMode) -> None: + del cudagraph_mode + + def capture(self) -> None: + return None + + @torch.inference_mode() + def propose( + self, + input_batch: InputBatch, + attn_metadata: Any, + slot_mappings: Any, + last_hidden_states: torch.Tensor, + aux_hidden_states: list[torch.Tensor] | None, + num_sampled: torch.Tensor, + num_rejected: torch.Tensor, + last_sampled: torch.Tensor, + next_prefill_tokens: torch.Tensor, + temperature: torch.Tensor, + seeds: torch.Tensor, + dp_sync: DPSyncState | None = None, + dummy_run: bool = False, + skip_attn_for_dummy_run: bool = False, + mm_inputs: tuple[list[torch.Tensor], torch.Tensor] | None = None, + is_profile: bool = False, + num_speculative_tokens: int | None = None, + ) -> torch.Tensor: + num_reqs = input_batch.num_reqs + if dummy_run: + return self.drafts[:num_reqs] + + req_states = self.req_states + token_ids = req_states.all_token_ids.gpu + idx_mapping = input_batch.idx_mapping + + _ngram_scan_kernel[(num_reqs, self.n_blocks)]( + token_ids, + token_ids.stride(0), + idx_mapping, + req_states.total_len.gpu, + num_sampled, + self.scratch, + self.scratch.stride(0), + self.max_model_len, + self.min_n, + self.max_n, + max(1, triton.next_power_of_2(self.max_n)), + self.block_l, + num_warps=4, + num_stages=2, + ) + + _ngram_finalize_kernel[(num_reqs,)]( + token_ids, + token_ids.stride(0), + idx_mapping, + req_states.total_len.gpu, + num_sampled, + last_sampled.view(-1), + self.scratch, + self.scratch.stride(0), + self.drafts, + self.max_model_len, + self.n_blocks, + self.num_speculative_steps, + max(1, triton.next_power_of_2(self.num_speculative_steps)), + max(1, triton.next_power_of_2(self.n_blocks)), + num_warps=2, + num_stages=1, + ) + return self.drafts[:num_reqs] diff --git a/vllm/v1/worker/gpu/spec_decode/speculator.py b/vllm/v1/worker/gpu/spec_decode/speculator.py index 77a9ece360..d07c043de9 100644 --- a/vllm/v1/worker/gpu/spec_decode/speculator.py +++ b/vllm/v1/worker/gpu/spec_decode/speculator.py @@ -97,6 +97,7 @@ class BaseSpeculator(ABC): skip_attn_for_dummy_run: bool = False, mm_inputs: tuple[list[torch.Tensor], torch.Tensor] | None = None, is_profile: bool = False, + num_speculative_tokens: int | None = None, ) -> torch.Tensor: pass @@ -561,3 +562,19 @@ class DraftModelSpeculator(BaseSpeculator): causal=causal, dcp_local_seq_lens=dcp_local_seq_lens, ) + + def _update_draft_decode_metadata( + self, attn_metadata: dict[str, Any], num_reqs: int + ) -> None: + if self.block_tables.cp_size > 1: + prepare_dcp_local_seq_lens( + self.input_buffers.dcp_local_seq_lens, + self.input_buffers.seq_lens, + num_reqs, + self.block_tables.cp_size, + self.block_tables.cp_rank, + self.block_tables.cp_interleave, + ) + for groups in self.attn_groups: + for group in groups: + group.update_draft_decode_metadata(attn_metadata) diff --git a/vllm/v1/worker/gpu/spec_decode/draft_model/__init__.py b/vllm/v1/worker/gpu/spec_decode/standalone_ar/__init__.py similarity index 100% rename from vllm/v1/worker/gpu/spec_decode/draft_model/__init__.py rename to vllm/v1/worker/gpu/spec_decode/standalone_ar/__init__.py diff --git a/vllm/v1/worker/gpu/spec_decode/draft_model/speculator.py b/vllm/v1/worker/gpu/spec_decode/standalone_ar/speculator.py similarity index 97% rename from vllm/v1/worker/gpu/spec_decode/draft_model/speculator.py rename to vllm/v1/worker/gpu/spec_decode/standalone_ar/speculator.py index 05bc997f7a..8bdfb8530b 100644 --- a/vllm/v1/worker/gpu/spec_decode/draft_model/speculator.py +++ b/vllm/v1/worker/gpu/spec_decode/standalone_ar/speculator.py @@ -26,22 +26,22 @@ from vllm.v1.worker.utils import AttentionGroup logger = init_logger(__name__) -class PlainDraftModelSpeculator(DraftModelSpeculator): - """Speculative decoding using a separate smaller draft LM. +class StandaloneARSpeculator(DraftModelSpeculator): + """Standalone autoregressive (AR) drafting with a separate smaller LM. Unlike Eagle, the draft model runs fully independently of the target model. Step 0 builds an expanded buffer (accepted + correction token + rejected slots) via a Triton kernel; steps 1..k-1 are single-token decode steps. """ - # Plain draft model needs one extra slot per request for correction. + # Standalone AR drafting needs one extra slot per request for correction. num_extra_query_per_req = 1 def __init__(self, vllm_config: VllmConfig, device: torch.device): super().__init__(vllm_config, device) # draft_max_seq_len is read by the parent's attention metadata builder. - # Plain draft model doesn't do per-batch adjustment; cap at max. + # Standalone AR drafting doesn't do per-batch adjustment; cap at max. self.draft_max_seq_len = self.max_model_len self.last_token_indices = torch.zeros( @@ -65,10 +65,10 @@ class PlainDraftModelSpeculator(DraftModelSpeculator): self.supports_mm_inputs = False def init_cudagraph_manager(self, cudagraph_mode: CUDAGraphMode) -> None: - pass # CUDA graph not yet supported for plain draft model speculator. + pass # CUDA graph not yet supported for standalone AR speculator. def capture(self) -> None: - pass # CUDA graph not yet supported for plain draft model speculator. + pass # CUDA graph not yet supported for standalone AR speculator. def set_attn( self, @@ -355,6 +355,7 @@ class PlainDraftModelSpeculator(DraftModelSpeculator): skip_attn_for_dummy_run: bool = False, mm_inputs: tuple[list[torch.Tensor], torch.Tensor] | None = None, is_profile: bool = False, + num_speculative_tokens: int | None = None, ) -> torch.Tensor: assert self.model is not None @@ -414,7 +415,7 @@ def _prepare_prefill_inputs_kernel( max_model_len, BLOCK_SIZE: tl.constexpr, ): - """Per-request step-0 input preparation for the plain draft-model speculator. + """Per-request step-0 input preparation for the standalone AR speculator. Output layout for request i (out_start = query_start_loc[i] + i): [out_start, out_start + num_valid) accepted tokens diff --git a/vllm/v1/worker/gpu/spec_decode/target_dependent_ar/__init__.py b/vllm/v1/worker/gpu/spec_decode/target_dependent_ar/__init__.py new file mode 100644 index 0000000000..208f01a7cb --- /dev/null +++ b/vllm/v1/worker/gpu/spec_decode/target_dependent_ar/__init__.py @@ -0,0 +1,2 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project diff --git a/vllm/v1/worker/gpu/spec_decode/autoregressive/cudagraph_utils.py b/vllm/v1/worker/gpu/spec_decode/target_dependent_ar/cudagraph_utils.py similarity index 57% rename from vllm/v1/worker/gpu/spec_decode/autoregressive/cudagraph_utils.py rename to vllm/v1/worker/gpu/spec_decode/target_dependent_ar/cudagraph_utils.py index fe3bdef269..a7d3eb52b5 100644 --- a/vllm/v1/worker/gpu/spec_decode/autoregressive/cudagraph_utils.py +++ b/vllm/v1/worker/gpu/spec_decode/target_dependent_ar/cudagraph_utils.py @@ -1,11 +1,13 @@ # SPDX-License-Identifier: Apache-2.0 # SPDX-FileCopyrightText: Copyright contributors to the vLLM project from collections.abc import Callable +from dataclasses import dataclass, replace import torch from vllm.config.compilation import CUDAGraphMode from vllm.v1.kv_cache_interface import KVCacheConfig +from vllm.v1.spec_decode.dynamic.utils import build_dynamic_sd_schedule_lookup from vllm.v1.worker.gpu.block_table import BlockTables from vllm.v1.worker.gpu.cudagraph_utils import ( BatchExecutionDescriptor, @@ -17,6 +19,11 @@ from vllm.v1.worker.gpu.model_states.interface import ModelState from vllm.v1.worker.utils import AttentionGroup +@dataclass(frozen=True) +class SpeculatorBatchDescriptor(BatchExecutionDescriptor): + num_speculative_tokens: int = 0 + + class SpeculatorCudaGraphManager(CudaGraphManager): """CudaGraphManager for draft prefill and decode. @@ -27,6 +34,18 @@ class SpeculatorCudaGraphManager(CudaGraphManager): earlier capture would execute kernels with stale buffer contents. """ + def specialize_spec_tokens( + self, desc: BatchExecutionDescriptor, num_speculative_tokens: int + ) -> BatchExecutionDescriptor: + if desc.cg_mode != CUDAGraphMode.FULL or desc in self.graphs: + return desc + desc = SpeculatorBatchDescriptor( + **vars(desc), num_speculative_tokens=num_speculative_tokens + ) + return ( + desc if desc in self.graphs else replace(desc, cg_mode=CUDAGraphMode.NONE) + ) + def capture( self, forward_fn: Callable, @@ -36,7 +55,25 @@ class SpeculatorCudaGraphManager(CudaGraphManager): attn_groups: list[list[AttentionGroup]], kv_cache_config: KVCacheConfig, progress_bar_desc: str = "Capturing CUDA graphs", + specialize_spec_tokens: bool = False, ) -> None: + if specialize_spec_tokens: + speculative_config = self.vllm_config.speculative_config + assert speculative_config is not None + dense_schedule = build_dynamic_sd_schedule_lookup( + speculative_config.num_speculative_tokens_per_batch_size, + vllm_max_batch_size=self.max_num_reqs, + vllm_num_speculative_tokens=self.vllm_config.num_speculative_tokens, + ) + self._capture_descs = { + mode: [ + SpeculatorBatchDescriptor(**vars(desc), num_speculative_tokens=k) + for desc in descs + for k in sorted(set(dense_schedule) - {0, 1}, reverse=True) + ] + for mode, descs in self._capture_descs.items() + } + def create_forward_fn( desc: BatchExecutionDescriptor, warmup: bool, @@ -60,6 +97,13 @@ class SpeculatorCudaGraphManager(CudaGraphManager): max_query_len=desc.max_query_len or desc.uniform_token_count, ) + forward_kwargs = {} + if self.decode_query_len == 1: + forward_kwargs["num_speculative_steps"] = ( + desc.num_speculative_tokens + if isinstance(desc, SpeculatorBatchDescriptor) + else self.vllm_config.num_speculative_tokens + ) return lambda cg_mode: forward_fn( num_reqs, num_tokens, @@ -67,6 +111,7 @@ class SpeculatorCudaGraphManager(CudaGraphManager): slot_mappings, num_tokens_across_dp, cg_mode, + **forward_kwargs, ) super().capture(create_forward_fn, progress_bar_desc) diff --git a/vllm/v1/worker/gpu/spec_decode/autoregressive/speculator.py b/vllm/v1/worker/gpu/spec_decode/target_dependent_ar/speculator.py similarity index 94% rename from vllm/v1/worker/gpu/spec_decode/autoregressive/speculator.py rename to vllm/v1/worker/gpu/spec_decode/target_dependent_ar/speculator.py index 897e7af8de..9a8ed7b8b4 100644 --- a/vllm/v1/worker/gpu/spec_decode/autoregressive/speculator.py +++ b/vllm/v1/worker/gpu/spec_decode/target_dependent_ar/speculator.py @@ -17,16 +17,22 @@ from vllm.v1.worker.gpu.cudagraph_utils import BatchExecutionDescriptor from vllm.v1.worker.gpu.dp_utils import DPSyncState, dispatch_cg_and_sync_dp from vllm.v1.worker.gpu.input_batch import InputBatch, InputBuffers from vllm.v1.worker.gpu.model_states.interface import ModelState -from vllm.v1.worker.gpu.spec_decode.autoregressive.cudagraph_utils import ( +from vllm.v1.worker.gpu.spec_decode.speculator import DraftModelSpeculator +from vllm.v1.worker.gpu.spec_decode.target_dependent_ar.cudagraph_utils import ( SpeculatorCudaGraphManager, ) -from vllm.v1.worker.gpu.spec_decode.speculator import DraftModelSpeculator from vllm.v1.worker.utils import AttentionGroup, get_uniform_decode_token_count logger = init_logger(__name__) -class AutoRegressiveSpeculator(DraftModelSpeculator): +class TargetDependentARSpeculator(DraftModelSpeculator): + """Autoregressive (AR) drafting that depends on target model state. + + Base for Eagle, MTP, and Gemma4 drafters, which consume the target's + hidden states (and, for Gemma4, its KV cache) at each draft step. + """ + def __init__(self, vllm_config: VllmConfig, device: torch.device): super().__init__(vllm_config, device) @@ -99,10 +105,6 @@ class AutoRegressiveSpeculator(DraftModelSpeculator): self.use_fused_multi_step_decode = False return - if not self.advance_draft_positions: - self.use_fused_multi_step_decode = True - return - unsupported_backends = sorted( { attn_group.backend.get_name() @@ -190,6 +192,8 @@ class AutoRegressiveSpeculator(DraftModelSpeculator): self.attn_groups, self.kv_cache_config, progress_bar_desc="Capturing decode CUDA graphs", + specialize_spec_tokens=self.use_fused_multi_step_decode + and self.speculative_config.uses_dynamic_speculative_decoding(), ) self.on_multi_step_decode_end(self.max_num_reqs) @@ -220,14 +224,17 @@ class AutoRegressiveSpeculator(DraftModelSpeculator): skip_attn_for_dummy_run: bool = False, mm_inputs: tuple[list[torch.Tensor], torch.Tensor] | None = None, is_profile: bool = False, + num_speculative_tokens: int | None = None, ) -> torch.Tensor: + num_reqs = input_batch.num_reqs + if num_speculative_tokens is None: + num_speculative_tokens = self.num_speculative_steps num_tokens = input_batch.num_tokens num_tokens_padded = input_batch.num_tokens_after_padding - num_reqs = input_batch.num_reqs max_query_len = input_batch.num_scheduled_tokens.max() max_seq_len = input_batch.seq_lens_cpu_upper_bound[:num_reqs].max().item() self.draft_max_seq_len = min( - max_seq_len + self.num_speculative_steps, self.max_model_len + max_seq_len + num_speculative_tokens, self.max_model_len ) # NOTE(woosuk): To avoid CPU-GPU synchronization without CPU knowing the @@ -322,9 +329,10 @@ class AutoRegressiveSpeculator(DraftModelSpeculator): ) self.on_prefill_end(num_reqs) - if self.num_speculative_steps == 1: - # Early exit. - return self.draft_tokens[:num_reqs, :1] + if num_speculative_tokens <= 1: + if num_speculative_tokens == 0: + self.draft_tokens[:num_reqs].fill_(-1) + return self.draft_tokens[:num_reqs] if self.pcp_manager is not None and not dummy_run: self.block_tables.gather_block_tables( @@ -360,6 +368,10 @@ class AutoRegressiveSpeculator(DraftModelSpeculator): need_eager=is_profile, dp_sync=decode_batch_sync, ) + if self.decode_cudagraph_manager is not None: + decode_batch_desc = self.decode_cudagraph_manager.specialize_spec_tokens( + decode_batch_desc, num_speculative_tokens + ) num_tokens_across_dp = ( decode_batch_sync.num_tokens_across_dp if decode_batch_sync is not None @@ -379,6 +391,7 @@ class AutoRegressiveSpeculator(DraftModelSpeculator): decode_batch_desc, num_tokens_across_dp, input_batch.seq_lens_cpu_upper_bound, + num_speculative_tokens, ) self.on_multi_step_decode_end(num_reqs) @@ -505,6 +518,7 @@ class AutoRegressiveSpeculator(DraftModelSpeculator): batch_desc: BatchExecutionDescriptor, num_tokens_across_dp: torch.Tensor | None, seq_lens_cpu_upper_bound: torch.Tensor, + num_speculative_steps: int, ) -> None: positions = self.input_buffers.positions[:num_reqs] query_start_loc = self.input_buffers.query_start_loc[: num_reqs + 1] @@ -512,7 +526,7 @@ class AutoRegressiveSpeculator(DraftModelSpeculator): attn_metadata = None slot_mappings_by_layer = None - for step in range(1, self.num_speculative_steps): + for step in range(1, num_speculative_steps): # Rebuild every step when positions advance, or just once # on the first step when positions are constant (Gemma4 MTP). if not skip_attn and (self.advance_draft_positions or step == 1): @@ -546,6 +560,7 @@ class AutoRegressiveSpeculator(DraftModelSpeculator): slot_mappings_by_layer, num_tokens_across_dp=num_tokens_across_dp, cudagraph_runtime_mode=batch_desc.cg_mode, + num_speculative_steps=num_speculative_steps, ) def _fused_multi_step_decode( @@ -555,11 +570,17 @@ class AutoRegressiveSpeculator(DraftModelSpeculator): batch_desc: BatchExecutionDescriptor, num_tokens_across_dp: torch.Tensor | None, seq_lens_cpu_upper_bound: torch.Tensor, + num_speculative_steps: int, ) -> None: positions = self.input_buffers.positions[:num_reqs] query_start_loc = self.input_buffers.query_start_loc[: num_reqs + 1] idx_mapping = self.idx_mapping[:num_reqs] + if batch_desc.cg_mode == CUDAGraphMode.FULL: + assert self.decode_cudagraph_manager is not None + self.decode_cudagraph_manager.run_fullgraph(batch_desc) + return + attn_metadata = None slot_mappings_by_layer = None if not skip_attn: @@ -569,10 +590,9 @@ class AutoRegressiveSpeculator(DraftModelSpeculator): positions, batch_desc.num_tokens, ) - if batch_desc.cg_mode != CUDAGraphMode.FULL: - slot_mappings_by_layer = build_slot_mappings_by_layer( - slot_mappings, self.kv_cache_config - ) + slot_mappings_by_layer = build_slot_mappings_by_layer( + slot_mappings, self.kv_cache_config + ) attn_metadata = self._build_uniform_attn_metadata( num_reqs=num_reqs, batch_desc=batch_desc, @@ -581,11 +601,6 @@ class AutoRegressiveSpeculator(DraftModelSpeculator): step=1, ) - if batch_desc.cg_mode == CUDAGraphMode.FULL: - assert self.decode_cudagraph_manager is not None - self.decode_cudagraph_manager.run_fullgraph(batch_desc) - return - self._generate_fused_drafts( num_reqs, batch_desc.num_tokens, @@ -593,6 +608,7 @@ class AutoRegressiveSpeculator(DraftModelSpeculator): slot_mappings_by_layer, num_tokens_across_dp, batch_desc.cg_mode, + num_speculative_steps, ) def _generate_fused_drafts( @@ -602,18 +618,25 @@ class AutoRegressiveSpeculator(DraftModelSpeculator): attn_metadata: dict[str, Any] | None, slot_mappings: dict[str, torch.Tensor] | None, num_tokens_across_dp: torch.Tensor | None, - cudagraph_runtime_mode: CUDAGraphMode = CUDAGraphMode.NONE, + cudagraph_runtime_mode: CUDAGraphMode, + num_speculative_steps: int, ) -> None: idx_mapping = self.idx_mapping[:num_reqs] positions = self.input_buffers.positions[:num_reqs] query_start_loc = self.input_buffers.query_start_loc[: num_reqs + 1] - attn_groups = ( - [group for groups in self.attn_groups for group in groups] - if attn_metadata is not None - else [] - ) - for step in range(1, self.num_speculative_steps): + for step in range(1, num_speculative_steps): + if attn_metadata is not None and ( + self.advance_draft_positions or step == 1 + ): + self.block_tables.compute_slot_mappings( + idx_mapping, + query_start_loc, + positions, + num_tokens_padded, + ) + self._update_draft_decode_metadata(attn_metadata, num_reqs) + self.current_draft_step.fill_(step) self._generate_draft( num_reqs, @@ -622,20 +645,8 @@ class AutoRegressiveSpeculator(DraftModelSpeculator): slot_mappings, num_tokens_across_dp, cudagraph_runtime_mode, + num_speculative_steps, ) - if ( - step < self.num_speculative_steps - 1 - and attn_metadata is not None - and self.advance_draft_positions - ): - self.block_tables.compute_slot_mappings( - idx_mapping, - query_start_loc, - positions, - num_tokens_padded, - ) - for attn_group in attn_groups: - attn_group.update_draft_decode_metadata(attn_metadata) def _generate_draft( self, @@ -644,7 +655,8 @@ class AutoRegressiveSpeculator(DraftModelSpeculator): attn_metadata: dict[str, Any] | None, slot_mappings: dict[str, torch.Tensor] | None, num_tokens_across_dp: torch.Tensor | None, - cudagraph_runtime_mode: CUDAGraphMode = CUDAGraphMode.NONE, + cudagraph_runtime_mode: CUDAGraphMode, + num_speculative_steps: int, ) -> None: self._prepare_eplb_forward(num_reqs) @@ -682,7 +694,7 @@ class AutoRegressiveSpeculator(DraftModelSpeculator): self.sample_src_positions, num_reqs, self.max_model_len, - self.num_speculative_steps, + num_speculative_steps, advance_draft_positions=self.advance_draft_positions, ) diff --git a/vllm/v1/worker/gpu/states.py b/vllm/v1/worker/gpu/states.py index db27bf013f..16260b0ddb 100644 --- a/vllm/v1/worker/gpu/states.py +++ b/vllm/v1/worker/gpu/states.py @@ -16,6 +16,7 @@ class RequestState: vocab_size: int, device: torch.device, num_prefill_lookahead: int = 1, + use_dense_all_token_ids: bool = False, ): self.max_num_reqs = max_num_reqs self.max_model_len = max_model_len @@ -30,12 +31,14 @@ class RequestState: # NOTE(woosuk): This tensor can be extremely large (e.g., several GBs) # depending on the configured max_num_reqs and max_model_len. - # To save GPU memory, we use UVA instead of GPU for this tensor. + # To save GPU memory, we use UVA instead of GPU by default, but + # ngram_gpu benefits from dense device residency because it scans + # active rows repeatedly during proposal. self.all_token_ids = StagedWriteTensor( (self.max_num_reqs, self.max_model_len), dtype=torch.int32, device=device, - uva_instead_of_gpu=True, + uva_instead_of_gpu=not use_dense_all_token_ids, ) # NOTE(woosuk): Distinguish clearly between prompt_len and prefill_len: # - prompt_len: Number of tokens in the user-provided prompt. diff --git a/vllm/v1/worker/gpu/warmup.py b/vllm/v1/worker/gpu/warmup.py index 2af97207ce..894e2e5b6b 100644 --- a/vllm/v1/worker/gpu/warmup.py +++ b/vllm/v1/worker/gpu/warmup.py @@ -139,9 +139,16 @@ def run_mixed_prefill_decode_warmup( next_block_id += num_blocks return block_ids - sampling_params = SamplingParams(max_tokens=2, temperature=0.0) + sampling_params = SamplingParams( + max_tokens=2, + temperature=0.0, + watermarking=False, + ) decode_prefill_output = SchedulerOutput.make_empty() + decode_prefill_output.num_spec_tokens_to_schedule = ( + model_runner.num_speculative_steps + ) decode_prefill_output.scheduled_new_reqs = [ NewRequestData( req_id=decode_req_id, @@ -171,6 +178,7 @@ def run_mixed_prefill_decode_warmup( ] mixed_output = SchedulerOutput.make_empty() + mixed_output.num_spec_tokens_to_schedule = model_runner.num_speculative_steps mixed_output.scheduled_cached_reqs = cached_decode_req mixed_output.scheduled_new_reqs = [ NewRequestData( @@ -349,6 +357,7 @@ def _warmup_kernels( ] prefill_output = SchedulerOutput.make_empty() + prefill_output.num_spec_tokens_to_schedule = num_spec_steps prefill_output.scheduled_new_reqs = new_reqs prefill_output.num_scheduled_tokens = {rid: prompt_len for rid in req_ids} prefill_output.total_num_scheduled_tokens = prompt_len * num_reqs @@ -408,6 +417,7 @@ def _warmup_kernels( step_spec_tokens[req_ids[i]] = [0] * num_spec_steps decode_output = SchedulerOutput.make_empty() + decode_output.num_spec_tokens_to_schedule = num_spec_steps decode_output.scheduled_cached_reqs = cached_req_data decode_output.num_scheduled_tokens = step_num_scheduled_tokens decode_output.scheduled_spec_decode_tokens = step_spec_tokens diff --git a/vllm/v1/worker/gpu_model_runner.py b/vllm/v1/worker/gpu_model_runner.py index 047fdc5ff0..c02bdc74e2 100644 --- a/vllm/v1/worker/gpu_model_runner.py +++ b/vllm/v1/worker/gpu_model_runner.py @@ -116,7 +116,7 @@ from vllm.tracing import instrument from vllm.utils import length_from_prompt_token_ids_or_embeds from vllm.utils.gc_utils import freeze_gc_for_cudagraph_capture from vllm.utils.gpu_sync_debug import gpu_sync_allowed -from vllm.utils.math_utils import cdiv, round_up +from vllm.utils.math_utils import cdiv from vllm.utils.mem_utils import DeviceMemoryProfiler, format_gib from vllm.utils.nvtx_pytorch_hooks import PytHooks from vllm.utils.platform_utils import num_compute_units @@ -222,7 +222,6 @@ from vllm.v1.worker.ubatch_utils import ( ) from vllm.v1.worker.utils import ( EncoderTimingStats, - is_residual_scattered_for_sp, raise_if_nan_logits, ) from vllm.v1.worker.workspace import lock_workspace @@ -840,6 +839,12 @@ class GPUModelRunner( self.num_accepted_tokens = self._make_buffer( self.max_num_reqs, dtype=torch.int32 ) + # The count includes the bonus token, so one is the minimum, and the + # recurrent kernels pick their state slot as `accepted - 1`. Dummy runs + # skip `_prepare_inputs`, so the zeros from the allocation would reach + # those kernels as index -1 and fault on the out-of-bounds read. + self.num_accepted_tokens.np.fill(1) + self.num_accepted_tokens.copy_to_gpu() # Only relevant for models using M-RoPE (e.g, Qwen2-VL) if self.uses_mrope: @@ -2199,7 +2204,7 @@ class GPUModelRunner( ) self.mrope_positions.gpu[:, :total_num_scheduled_tokens] += drift - use_spec_decode = len(scheduler_output.scheduled_spec_decode_tokens) > 0 + use_spec_decode = self.speculative_config is not None if not use_spec_decode: # NOTE(woosuk): Due to chunked prefills, the batch may contain # partial requests. While we should not sample any token @@ -2226,6 +2231,18 @@ class GPUModelRunner( num_draft_tokens[req_idx] = draft_len if num_scheduled_tokens[req_idx] == draft_len + 1: num_decode_draft_tokens[req_idx] = draft_len + # Some recurrent backends may read the previous step's state at an + # offset that only the speculative path applies. So a decode step + # that got no drafts must still count as a speculative row with + # zero drafts, not as a plain decode. Prompt chunks are left out: + # their tokens are prompt tokens and carry no such offset. + zero_draft_decode_mask = ( + self.input_batch.num_computed_tokens_cpu[:num_reqs] + >= self.input_batch.num_prompt_tokens[:num_reqs] + ) & (num_scheduled_tokens[:num_reqs] == 1) + num_decode_draft_tokens[ + (num_decode_draft_tokens < 0) & zero_draft_decode_mask + ] = 0 spec_decode_metadata = self._calc_spec_decode_metadata( num_draft_tokens, cu_num_tokens ) @@ -2912,7 +2929,14 @@ class GPUModelRunner( mm_kwargs.append((mm_feature.modality, mm_feature.data)) mm_lora_refs.append((req_id, mm_feature.mm_position)) - return mm_hashes, mm_kwargs, mm_lora_refs + # Stable-sort by modality so each modality is encoded in as few batches + # as possible. Encoder outputs are cached by mm_hash, so order is free. + order = sorted(range(len(mm_kwargs)), key=lambda i: mm_kwargs[i][0]) + return ( + [mm_hashes[i] for i in order], + [mm_kwargs[i] for i in order], + [mm_lora_refs[i] for i in order], + ) def _cache_encoder_output( self, @@ -2973,13 +2997,6 @@ class GPUModelRunner( and scheduler_output.scheduled_encoder_inputs ) - # Batch mm inputs as much as we can: if a request in the batch has - # multiple modalities or a different modality than the previous one, - # we process it separately to preserve item order. - # FIXME(ywang96): This is a hacky way to deal with multiple modalities - # in the same batch while still being able to benefit from batching - # multimodal inputs. The proper solution should be reordering the - # encoder outputs. model = cast(SupportsMultiModal, self.model) if self.lora_config and self.lora_manager.supports_tower_connector_lora(): @@ -3022,10 +3039,8 @@ class GPUModelRunner( ) self.lora_manager.set_active_adapters(lora_requests, tower_mapping) - # Only set connector mapping if the model actually has a connector. - # Some multimodal models inherit a stub `get_num_mm_connector_tokens` - # from `SupportsMultiModal`, which returns None and should not be - # treated as a signal that connector LoRA is supported. + # Only set connector mapping if the model actually has a connector + # and reports connector token counts. mm_mapping = ( self.model.get_mm_mapping() # type: ignore[attr-defined] if hasattr(self.model, "get_mm_mapping") @@ -3324,20 +3339,9 @@ class GPUModelRunner( ) -> IntermediateTensors: assert self.intermediate_tensors is not None - tp = self.vllm_config.parallel_config.tensor_parallel_size - is_rs = is_residual_scattered_for_sp(self.vllm_config, num_tokens) - - # When sequence parallelism is enabled, the "residual" tensor is - # sharded across TP ranks. All-gather it here because downstream - # QKV + Attention needs the full residual before the SP split point. if sync_self: assert intermediate_tensors is not None for k, v in intermediate_tensors.items(): - is_scattered = k == "residual" and is_rs - if is_scattered: - local_len = num_tokens // tp - v = get_tp_group().all_gather(v[:local_len], dim=0) - self.intermediate_tensors[k][:num_tokens].copy_( v[:num_tokens], non_blocking=True ) @@ -3432,14 +3436,6 @@ class GPUModelRunner( async_output_copy_stream=self._get_or_create_async_output_copy_stream(), ) - def _pad_for_sequence_parallelism(self, num_scheduled_tokens: int) -> int: - # Pad tokens to multiple of tensor_parallel_size when - # enabled collective fusion for SP - tp_size = self.vllm_config.parallel_config.tensor_parallel_size - if self.compilation_config.pass_config.enable_sp and tp_size > 1: - return round_up(num_scheduled_tokens, tp_size) - return num_scheduled_tokens - def _prepare_padding_mask( self, num_tokens_unpadded: int, num_tokens_padded: int ) -> torch.Tensor: @@ -3928,8 +3924,6 @@ class GPUModelRunner( ) has_lora = num_active_loras > 0 if force_has_lora is None else force_has_lora - num_tokens_padded = self._pad_for_sequence_parallelism(num_tokens) - def dispatch_cudagraph(num_tokens, disable_full=False, valid_modes=None): return self.cudagraph_dispatcher.dispatch( num_tokens=num_tokens, @@ -3941,18 +3935,9 @@ class GPUModelRunner( ) cudagraph_mode, batch_descriptor = dispatch_cudagraph( - num_tokens_padded, disable_full=use_cascade_attn or has_encoder_output + num_tokens, disable_full=use_cascade_attn or has_encoder_output ) num_tokens_padded = batch_descriptor.num_tokens - if self.compilation_config.pass_config.enable_sp: - assert ( - batch_descriptor.num_tokens - % self.vllm_config.parallel_config.tensor_parallel_size - == 0 - ), ( - "Sequence parallelism requires num_tokens to be " - "a multiple of tensor parallel size" - ) # Extra coordination when running data-parallel since we need to coordinate # across ranks @@ -4314,7 +4299,7 @@ class GPUModelRunner( self.mamba_state_idx, ) - use_spec_decode = len(scheduler_output.scheduled_spec_decode_tokens) > 0 + use_spec_decode = self.speculative_config is not None ubatch_slices_attn = ubatch_slices_padded if pad_attn else ubatch_slices slot_mappings_by_group, slot_mappings = self._get_slot_mappings( @@ -4437,15 +4422,9 @@ class GPUModelRunner( sample_hidden_states = hidden_states[logits_indices] if not get_pp_group().is_last_rank: - all_gather_tensors = { - "residual": not is_residual_scattered_for_sp( - self.vllm_config, num_tokens_padded - ) - } get_pp_group().send_tensor_dict( hidden_states.tensors, all_gather_group=get_tp_group(), - all_gather_tensors=all_gather_tensors, ) logits = None else: @@ -5518,9 +5497,11 @@ class GPUModelRunner( ) if weights_path is not None: - # The revision belongs to the model we are reloading away from, - # so it must not be carried over to the new path. + # The revision and any object-storage `model_weights` source + # belong to the model we are reloading away from, so they must + # not be carried over to the new path. self.model_config.model = weights_path + self.model_config.model_weights = "" self.model_config.revision = None weights_iterator = model_loader.get_all_weights(self.model_config, model) weights_iterator = cast( @@ -7117,7 +7098,6 @@ class GPUModelRunner( min_cg_attn_backend, self.uniform_decode_query_len, use_v2_model_runner=False, - tensor_parallel_size=self.parallel_config.tensor_parallel_size, kv_cache_config=self.kv_cache_config, max_num_reqs=self.max_num_reqs, is_profiling=is_profiling, diff --git a/vllm/v1/worker/gpu_worker.py b/vllm/v1/worker/gpu_worker.py index 3f416b94c7..f2dfc8a689 100644 --- a/vllm/v1/worker/gpu_worker.py +++ b/vllm/v1/worker/gpu_worker.py @@ -12,13 +12,13 @@ from fnmatch import filter as fnmatch_filter from types import NoneType from typing import TYPE_CHECKING, Any, cast -import numpy as np import torch import torch.nn as nn import vllm.envs as envs from vllm.config import CUDAGraphMode, VllmConfig, set_current_vllm_config from vllm.config.compilation import CompilationMode +from vllm.config.profiler import validate_profile_prefix from vllm.device_allocator import get_mem_allocator_instance from vllm.distributed import ( ensure_model_parallel_initialized, @@ -97,7 +97,6 @@ from vllm.v1.worker.startup_plan import ( maybe_apply_startup_plan, maybe_save_startup_plan, ) -from vllm.v1.worker.utils import is_residual_scattered_for_sp from vllm.v1.worker.worker_base import CompilationTimes, WorkerBase from vllm.v1.worker.workspace import init_workspace_manager @@ -822,10 +821,15 @@ class Worker(WorkerBase): # so that it's available to the warmup stage. self.cache_config.num_gpu_blocks = kv_cache_config.num_blocks - # Adopt the engine core's layout; workers spawned after resolution - # (e.g. elastic EP scale-up) only see it through the config. + # Adopt the engine core's layout and prefix-cache granularity; workers + # spawned after resolution (e.g. elastic EP scale-up) only see them + # through the config. if kv_cache_config.kv_cache_layout is not None: record_kv_cache_layout(self.cache_config, kv_cache_config.kv_cache_layout) + self.cache_config.hash_block_size = kv_cache_config.hash_block_size + self.cache_config.cache_hit_alignment_tokens = ( + kv_cache_config.cache_hit_alignment_tokens + ) # Init kv cache connector here, because it requires # `kv_cache_config`. @@ -1020,7 +1024,9 @@ class Worker(WorkerBase): # Reset the seed to ensure that the random state is not affected by # the model initialization and profiling. - set_random_seed(self.model_config.seed) + set_random_seed( + self.model_config.seed, self.parallel_config.data_parallel_index + ) # Eagerly trigger inductor's once-per-process lazy inits during # warmup (rather than on a later compile cache-miss at runtime). @@ -1270,45 +1276,10 @@ class Worker(WorkerBase): intermediate_tensors = None forward_pass = scheduler_output.total_num_scheduled_tokens > 0 - num_scheduled_tokens = scheduler_output.total_num_scheduled_tokens - all_gather_tensors = {} - compilation_config = self.vllm_config.compilation_config - parallel_config = self.vllm_config.parallel_config - - if ( - parallel_config.pipeline_parallel_size > 1 - and compilation_config.pass_config.enable_sp - and forward_pass - ): - # currently only supported by V1 GPUModelRunner - assert not self.use_v2_model_runner - num_scheduled_tokens_np = np.array( - list(scheduler_output.num_scheduled_tokens.values()), - dtype=np.int32, - ) - # TODO(lucas): This is pretty gross; ideally we should only ever call - # `_determine_batch_execution_and_padding` once (will get called again - # in `execute_model`) but this requires a larger refactor of PP. - _, batch_desc, _, _, _ = ( - self.model_runner._determine_batch_execution_and_padding( - num_tokens=num_scheduled_tokens, - num_reqs=len(num_scheduled_tokens_np), - num_scheduled_tokens_np=num_scheduled_tokens_np, - max_num_scheduled_tokens=num_scheduled_tokens_np.max(), - use_cascade_attn=False, # TODO(lucas): Handle cascade attention - ) - ) - all_gather_tensors = { - "residual": not is_residual_scattered_for_sp( - self.vllm_config, batch_desc.num_tokens - ) - } - if forward_pass and not get_pp_group().is_first_rank: tensor_dict, comm_handles, comm_postprocess = ( get_pp_group().irecv_tensor_dict( all_gather_group=get_tp_group(), - all_gather_tensors=all_gather_tensors, ) ) assert tensor_dict is not None @@ -1346,7 +1317,6 @@ class Worker(WorkerBase): handles = get_pp_group().isend_tensor_dict( output.tensors, all_gather_group=get_tp_group(), - all_gather_tensors=all_gather_tensors, ) self._pp_send_work = handles[1:] @@ -1357,7 +1327,14 @@ class Worker(WorkerBase): def take_draft_token_ids(self) -> DraftTokenIds | None: return self.model_runner.take_draft_token_ids() - def profile(self, is_start: bool = True, profile_prefix: str | None = None): + def profile( + self, + is_start: bool = True, + profile_prefix: str | None = None, + *, + delay_iterations: int | None = None, + max_iterations: int | None = None, + ): # Check if profiling is enabled if self.profiler_config is None or self.profiler_config.profiler is None: raise RuntimeError( @@ -1368,6 +1345,8 @@ class Worker(WorkerBase): ) if is_start: + validate_profile_prefix(profile_prefix) + # Generate the trace name by combining prefix with comprehensive rank suffix from vllm.distributed.utils import get_worker_rank_suffix @@ -1379,9 +1358,6 @@ class Worker(WorkerBase): else: trace_name = rank_suffix - if self.profiler_config.profiler == "proton" and self.profiler is not None: - self.profiler.set_output_name(trace_name) - # Create the profiler wrapper only on the first start call if self.profiler is None: self.profiler = create_worker_profiler( @@ -1390,7 +1366,11 @@ class Worker(WorkerBase): local_rank=self.local_rank, ) - self.profiler.start() + self.profiler.set_output_name(trace_name) + self.profiler.start( + delay_iterations=delay_iterations, + max_iterations=max_iterations, + ) else: if self.profiler is None: logger.warning("Profiler was not started, nothing to stop.") diff --git a/vllm/v1/worker/mamba_utils.py b/vllm/v1/worker/mamba_utils.py index 33a2b8b480..a3b4eab340 100644 --- a/vllm/v1/worker/mamba_utils.py +++ b/vllm/v1/worker/mamba_utils.py @@ -392,8 +392,8 @@ def postprocess_mamba_fused_kernel( # Output: num_accepted_tokens update (for src==dst case) num_accepted_tokens_out_ptr, # Optional: batch_idx -> req_idx mapping (V2 model runner / PP). The - # per-request decision arrays are in req-state-slot order; the block table - # is in batch order, so HAS_IDX_MAPPING splits the two indexings. + # per-request decision arrays and the block tables are both in + # req-state-slot order. idx_mapping_ptr, # Runtime parameter (varies per batch - NOT constexpr to avoid recompilation) num_reqs, @@ -476,10 +476,9 @@ def postprocess_mamba_fused_kernel( if src_block_idx == dest_block_idx and accept_token_bias == 0: return - bt_row_idx = batch_idx if HAS_IDX_MAPPING else req_idx _copy_mamba_state_block( state_idx, - bt_row_idx, + req_idx, src_block_idx, dest_block_idx, accept_token_bias, @@ -610,7 +609,7 @@ def precopy_mamba_align_fused_kernel( token_bias = tl.load(token_bias_ptr + req_idx) _copy_mamba_state_block( state_idx, - batch_idx, + req_idx, src_col, dst_col, token_bias, @@ -806,8 +805,10 @@ class MambaSpecDecodeGPUContext: # Per-group block-table base addresses: int64[num_groups]. Populated in # initialize_from_forward_context from the persistent per-group block - # table tensors (whose data_ptr is stable across steps). + # table tensors (whose data_ptr is stable across steps). Rows are request + # state slots for the state copies and batch rows for the aligned indices. block_table_ptrs: torch.Tensor + aligned_index_block_table_ptrs: torch.Tensor block_table_stride_req: int = 0 # persistent output for the once-per-step, all-group aligned-index launch. @@ -900,6 +901,9 @@ class MambaSpecDecodeGPUContext: block_table_ptrs=torch.zeros( len(mamba_group_ids), dtype=torch.int64, device=device ), + aligned_index_block_table_ptrs=torch.zeros( + len(mamba_group_ids), dtype=torch.int64, device=device + ), aligned_state_indices=torch.empty( ( len(mamba_group_ids), @@ -924,6 +928,7 @@ class MambaSpecDecodeGPUContext: forward_context: dict[str, Any], mamba_state_copy_funcs: MambaStateCopyFuncsByType, block_tables: list[torch.Tensor], + aligned_index_block_tables: list[torch.Tensor] | None = None, ) -> None: """Extract and cache memory layout metadata from Mamba state tensors. @@ -955,8 +960,12 @@ class MambaSpecDecodeGPUContext: mamba_state_copy_funcs: Mapping from MambaAttentionBackendEnum to copy functions. block_tables: per-mamba-group persistent block-table tensors, in - the same order as `mamba_group_ids`. Their `data_ptr()` / - `stride(0)` are captured once for the kernel to index into. + the same order as `mamba_group_ids`, with one row per request + state slot. Their `data_ptr()` / `stride(0)` are captured once + for the state-copy kernels to index into. + aligned_index_block_tables: per-mamba-group persistent tables + read by `compute_aligned_state_indices`, with one row per + batch row. Defaults to `block_tables`. """ if self.is_initialized: @@ -968,6 +977,7 @@ class MambaSpecDecodeGPUContext: forward_context, mamba_state_copy_funcs, block_tables, + aligned_index_block_tables or block_tables, ) def _populate_metadata( @@ -976,6 +986,7 @@ class MambaSpecDecodeGPUContext: forward_context: dict[str, Any], mamba_state_copy_funcs: MambaStateCopyFuncsByType, block_tables: list[torch.Tensor], + aligned_index_block_tables: list[torch.Tensor], ) -> None: idx = 0 for group_local_idx, mamba_group_id in enumerate(self.mamba_group_ids): @@ -1076,16 +1087,19 @@ class MambaSpecDecodeGPUContext: # `block_tables[i]` is the persistent 2D int32 block-table tensor for # `mamba_group_ids[i]`; `data_ptr()` / `stride(0)` are stable for the # engine's lifetime, so we capture them once here. - assert len(block_tables) == self.num_groups, ( - f"expected {self.num_groups} block tables, got {len(block_tables)}" - ) - strides = {bt.stride(0) for bt in block_tables} + assert len(block_tables) == len(aligned_index_block_tables) == self.num_groups + strides = {bt.stride(0) for bt in (*block_tables, *aligned_index_block_tables)} assert len(strides) == 1, ( f"all mamba block tables must share stride(0), got {strides}" ) self.block_table_stride_req = int(next(iter(strides))) - for i, bt in enumerate(block_tables): + for i, (bt, aligned_bt) in enumerate( + zip(block_tables, aligned_index_block_tables) + ): self.block_table_ptrs[i] = _reinterpret_u64_as_i64(bt.data_ptr()) + self.aligned_index_block_table_ptrs[i] = _reinterpret_u64_as_i64( + aligned_bt.data_ptr() + ) self.is_initialized = True @@ -1107,7 +1121,7 @@ class MambaSpecDecodeGPUContext: block_rows = 32 grid = (triton.cdiv(num_reqs, block_rows),) get_aligned_state_indices_multi_group_kernel[grid]( - self.block_table_ptrs, + self.aligned_index_block_table_ptrs, seq_lens, self.aligned_state_indices, self.block_table_stride_req, diff --git a/vllm/v1/worker/utils.py b/vllm/v1/worker/utils.py index 422ba8885f..db0f81b81c 100644 --- a/vllm/v1/worker/utils.py +++ b/vllm/v1/worker/utils.py @@ -10,7 +10,7 @@ from typing import Any import numpy as np import torch -from vllm.config import CacheConfig, VllmConfig +from vllm.config import CacheConfig from vllm.logger import init_logger from vllm.model_executor.layers.attention import Attention from vllm.model_executor.layers.mamba.mamba_mixer2 import share_replayssm_ring_trackers @@ -327,6 +327,30 @@ class AttentionGroup: self.get_metadata_builder().update_draft_decode_metadata(metadata) +def _block_size_is_supported( + backends: list[type[AttentionBackend]], block_size: int +) -> bool: + """Check if the block size is supported by all backends. + + An exact ``int`` declaration must match exactly; a ``MultipleOf`` + declaration accepts any multiple of its base. + """ + for backend in backends: + is_supported = False + for supported_size in backend.get_supported_kernel_block_sizes(): + if isinstance(supported_size, int): + if block_size == supported_size: + is_supported = True + elif isinstance(supported_size, MultipleOf): + if block_size % supported_size.base == 0: + is_supported = True + else: + raise ValueError(f"Unknown supported size: {supported_size}") + if not is_supported: + return False + return True + + def select_common_block_size( kv_manager_block_size: int, backends: list[type[AttentionBackend]], @@ -348,27 +372,7 @@ def select_common_block_size( ValueError: If no valid block size found. """ - - def block_size_is_supported( - backends: list[type[AttentionBackend]], block_size: int - ) -> bool: - """Check if the block size is supported by all backends.""" - for backend in backends: - is_supported = False - for supported_size in backend.get_supported_kernel_block_sizes(): - if isinstance(supported_size, int): - if block_size == supported_size: - is_supported = True - elif isinstance(supported_size, MultipleOf): - if block_size % supported_size.base == 0: - is_supported = True - else: - raise ValueError(f"Unknown supported size: {supported_size}") - if not is_supported: - return False - return True - - if block_size_is_supported(backends, kv_manager_block_size): + if _block_size_is_supported(backends, kv_manager_block_size): return kv_manager_block_size # MultipleOf constraints also accept the manager size if they accept a divisor. @@ -381,7 +385,7 @@ def select_common_block_size( } for size in sorted(candidates, reverse=True): - if block_size_is_supported(backends, size): + if _block_size_is_supported(backends, size): return size raise ValueError( f"No common block size for {kv_manager_block_size} (" @@ -493,9 +497,24 @@ def prepare_kernel_block_sizes( # This is an attention backend that supports virtual block splitting. kv_manager_block_size = kv_cache_group.kv_cache_spec.block_size group_backends = [g.backend for g in attn_groups[kv_cache_gid]] - selected_kernel_size = select_common_block_size( - kv_manager_block_size, group_backends + storage_block_size = ( + kv_cache_spec.storage_block_size + if isinstance(kv_cache_spec, MLAAttentionSpec) + else None ) + if storage_block_size is not None and _block_size_is_supported( + group_backends, storage_block_size + ): + # Storage-block specs (e.g. the GLM-5.3-Flash kpool indexer + # cache) address the cache in pool pages, and every other + # consumer (cache views, metadata builders, hisparse) already + # uses storage_block_size as the kernel block. Fall back to the + # backend vote when the group's backends do not accept it. + selected_kernel_size = storage_block_size + else: + selected_kernel_size = select_common_block_size( + kv_manager_block_size, group_backends + ) kernel_block_sizes.append(selected_kernel_size) elif isinstance(kv_cache_spec, MambaSpec): # This is likely Mamba or other non-attention cache, no splitting. @@ -794,35 +813,6 @@ def get_uniform_decode_token_count( return None -def is_residual_scattered_for_sp( - vllm_config: VllmConfig, num_input_tokens: int -) -> bool: - """Check if the residual tensor is scattered for sequence parallelism. - - The residual tensor is scattered across tensor parallel ranks when sequence - parallelism and tensor parallelism is enabled. SP is only supported in - full-graph compilation mode. - """ - if not vllm_config.compilation_config.pass_config.enable_sp: - return False - - tp = vllm_config.parallel_config.tensor_parallel_size - - if tp == 1: - return False - - assert ( - vllm_config.compilation_config.use_inductor_graph_partition - or not vllm_config.compilation_config.splitting_ops - ), "Sequence parallelism requires full-graph compilation" - - # When sequence parallelism is enabled, we always pad num_input_tokens - # to be a multiple of tensor_parallel_size (tp) earlier. - assert num_input_tokens % tp == 0 - - return True - - @dataclass class EncoderTimingStats: """Per-request timing statistics for encoder forward pass.""" diff --git a/vllm/v1/worker/workspace.py b/vllm/v1/worker/workspace.py index 4bae1f52be..a05534281c 100644 --- a/vllm/v1/worker/workspace.py +++ b/vllm/v1/worker/workspace.py @@ -139,7 +139,8 @@ class WorkspaceManager: AssertionError. Keys must identify the resource type and configuration. Callers must not - resize or replace tensor storage held by a cached resource. Uses of the + resize or replace tensor storage held by a cached resource once the + manager is locked. Uses of the same resource must not overlap; include the CUDA stream in the key when streams can execute concurrently. @@ -229,12 +230,7 @@ class WorkspaceManager: # Walk up the stack skipping WorkspaceManager frames curr_frame = curr_frame.f_back while curr_frame is not None: - # TODO: This only catches instance methods (self), missing - # classmethods and staticmethods. Once Python 3.11+ is the - # minimum supported version, use co_qualname instead: - # qualname = curr_frame.f_code.co_qualname - # if qualname.startswith("WorkspaceManager."): - if isinstance(curr_frame.f_locals.get("self"), WorkspaceManager): + if curr_frame.f_code.co_qualname.startswith("WorkspaceManager."): curr_frame = curr_frame.f_back continue filename = os.path.basename(curr_frame.f_code.co_filename)