Compare commits

..
34 Commits
Author SHA1 Message Date
MOVZX ac305c44cb Merge branch 'main' into patches
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 17:41:13 +00:00
MOVZX 4a3ebc6b5b Merge branch 'main' into patches 2026-10-05 08:21:46 +00:00
MOVZX f83ecfac16 Merge remote-tracking branch 'upstream/main' into patches 2026-10-05 08:52:31 +07:00
MOVZX 85c31d47ad Merge remote-tracking branch 'upstream/main' into patches
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-03 22:06:10 +07:00
MOVZX 2877121b2d Merge remote-tracking branch 'upstream/main' into patches
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-03 21:59:03 +07:00
MOVZXandHinata ae0350c0de [Spec Decode][MRV2] RecoverSSM for Qwen GDN and PLE short conv (PR #58863)
Intake of open upstream PR vllm-project/vllm#58863. Active only with
VLLM_USE_V2_MODEL_RUNNER=1; inert otherwise.

Speculative decoding on Gated DeltaNet snapshotted the full recurrent
state after every draft position and reserved one Mamba block per draft
position. On a 24 GB card that is reserved VRAM we never use, plus
per-step copy traffic.

RecoverSSM, the protocol Kimi-K3 KDA already uses, replaces this:
verify runs from a read-only checkpoint and stores a small per-token
replay record [correction (V), normalized key (K), gate (1)] in fp32;
commit replays only the accepted tokens and writes the recurrent state
once, plus the block-boundary state in align mode. The PLE dilated
short conv compacts its accepted window with the same plan.

Conflict in vllm/models/qwen4_exp/nvidia/mtp.py: our get_top_tokens()
work put LocalArgmaxMixin on Qwen4ExpMTP, the PR adds SupportsReplaySSM
to the same base list. Kept both.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-09-29 13:39:09 +07:00
MOVZXandHinata a1d3c2cf1e [Spec Decode] Fused draft metadata updates for the QSA side cache (PR #58449)
Intake of open upstream PR vllm-project/vllm#58449.

With MTP num_speculative_tokens > 1, the V2 autoregressive speculator
reuses one attention-metadata build across draft steps only when every
group's builder sets supports_draft_decode_metadata_update. The QSA
side-cache builder (QWEN4_EXP_EXP_QSA_STATE) did not, so every draft
step rebuilt attention metadata for the whole model.

QSAMetadataBuilder now implements update_draft_decode_metadata: it
re-launches the kernel with in-place seq_lens updates, so one metadata
build serves every draft step.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-09-29 13:36:09 +07:00
MOVZXandHinata cb275fdf20 [Qwen4Exp] Fix MTP buffer placement, config parsing, Triton warmup (PR #56742)
Intake of open upstream PR vllm-project/vllm#56742. Every rig here runs
MTP on both Qwen3.8 models, so these three bugs sit on our hot path.

The MTP hidden buffer was allocated with torch.empty and no device, so
it followed the ambient default instead of device_config.device; on a
CPU-default process the draft buffer lands on the wrong device. Both
NVIDIA and AMD backends now call make_mtp_hidden_buffer() in
vllm/models/qwen4_exp/common/mtp.py, which pins it to the configured
device.

Config parsing now accepts configs carrying M-RoPE keys and the
Transformers spelling of layer_types; both previously failed at load.

Triton warmup: Qwen4Exp was missing from the Qwen warmup dispatch, so
its Triton kernels compiled during the first served requests instead of
at startup.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-09-29 13:36:05 +07:00
MOVZX a63dca4afb Merge remote-tracking branch 'upstream/main' into patches
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-09-29 12:54:18 +07:00
MOVZX 76af434852 Merge remote-tracking branch 'upstream/main' into patches 2026-09-28 12:06:52 +07:00
lesj0610 521235d6df Merge remote-tracking branch 'origin/main' into sync/lesj/qwen4exp-bugfixes
Signed-off-by: lesj0610 <lesj0610@users.noreply.github.com>
2026-09-27 18:25:21 +09:00
5567cc1b25 [Mamba] RecoverSSM align: keep a boundary-ending state in its block
A speculative window whose accepted tokens end exactly on a recurrent block
boundary selected the following block-table column for the committed state
(commit plan) and recorded it as the running column (postprocess). That
column can still be unallocated, so the next step reads an uninitialized
page. Select the block that holds the last committed token, matching the
(n - 1) // block convention used elsewhere in align mode. Applies to
Kimi-K3 KDA and to the Qwen GDN/PLE RecoverSSM paths that share the plan.

Fix and test from local-inference-lab/vllm a02f585d (Martin Vit, root cause
reported by astra); the GDN/PLE align tests now expect the same convention.

Signed-off-by: Jürgen Schmied <juergenschmied70@gmail.com>
Co-authored-by: Martin Vit <martin@voipmonitor.org>
Co-authored-by: Claude <noreply@anthropic.com>
2026-09-27 11:12:40 +02:00
Jürgen Schmied ed2a9665ea Merge branch 'main' into gdn-recoverssm 2026-09-27 08:01:32 +02:00
MOVZX 5da84df6b7 [BugFix] Honor drop_eagle_block in MambaManager
On a hybrid model (full-attention + Mamba/GDN layers, e.g. Qwen3.6-35B-A3B),
enabling MTP/EAGLE speculative decoding together with prefix caching corrupts
the recurrent state, and the server serves wrong answers. #43559 reports
tool-eval accuracy dropping from ~90% to ~50%. Nothing raises an exception
and nothing shows up in the logs, so an operator has no signal that anything
is wrong, and prefix caching then spreads it, because the corrupted state is
hash-reachable and every later request sharing that prefix restores from it.

The root cause is the find_longest_cache_hit method on MambaManager,
at vllm/v1/core/single_type_kv_cache_manager.py:1049 on main, which takes
drop_eagle_block in its signature and never acts on it. Its body searches
right-to-left for the last cached block, builds [null]*i + [cached], and
breaks, referencing neither drop_eagle_block nor .pop(.

Signed-off-by: MOVZX <movzx@yahoo.com>
2026-09-27 10:45:10 +07:00
MOVZX 5e6656e7a0 [Bugfix][Spec Decode] Add get_top_tokens() to Qwen4ExpMTP draft model
use_local_argmax_reduction hard-fails at worker startup when the draft
model lacks get_top_tokens(): the V2 speculator probes for the method
in _validate_local_argmax_reduction and raises, with no fallback to
the full-vocab logits path. Qwen4ExpMTP (nvidia and amd variants) has
compute_logits but no get_top_tokens, so any Qwen4Exp MTP drafter with
the flag enabled crashed both TP ranks during load_model.

Qwen4ExpMTP owns a flat self.lm_head and its compute_logits is exactly
self.logits_processor(self.lm_head, hidden_states), the same shape as
Qwen3_5MTP, so LocalArgmaxMixin applies unchanged: vocab-parallel
argmax with a (value, index) gather, O(2*tp_size) vs O(vocab_size)
communication per draft token. No D2T remapping needed since the
draft vocabulary equals the target vocabulary.

Signed-off-by: MOVZX <movzx@yahoo.com>
2026-09-27 10:31:47 +07:00
MOVZXandHinata 43fbd6510b [Qwen4Exp] Serve PLE rows from the loader's mapped checkpoint shards (PR #58815)
Adds an opt-in backend, --engram-config '{"host_file_gather": true}', that
serves Qwen4Exp PLE rows from the checkpoint files through the page cache
instead of allocating the 47.7 GiB table on GPU or in pinned host memory.

- Qwen4ExpPLEFileGatherEmbedding allocates no table rows; load_weights keeps
  the default loader's zero-copy safetensors views of each shard. At bind
  (post-load hook) every view must be file-backed per /proc/self/maps (HF
  cache symlinks work; heap/pinned/anonymous memory is rejected) and the
  shards must cover the whole table.
- Each step, Qwen4ExpModelState.prepare_inputs computes the PLE n-gram ids,
  copies them to pinned host memory, syncs the stream once, issues
  POSIX_FADV_WILLNEED over the coalesced page ranges of the step's rows,
  gathers the rows on the host, and copies them into a fixed device staging
  buffer. The forward reads only the staging buffer, so FULL CUDA graph
  capture is unaffected.
- Rows stay in the checkpoint dtype (FP8 table stays FP8, dequantized on the
  GPU); dummy loads serve zeros; weight reload is rejected; takes precedence
  over cpu_offload; rejected on ROCm and with elastic EP.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-09-27 09:54:42 +07:00
MOVZXandHinata 5af48b498c [Qwen3.5] Build MTP head unquantized when checkpoint ships it dense (PR #58813)
Fixes #58807. A compressed-tensors pack-quantized checkpoint that ships
its MTP head dense (plain mtp.*.weight names in a separate bf16 shard)
crashes at weight load: Qwen3_5MultiTokenPredictor builds its Linears
from the target quant config, so the loader expects packed names
(weight_packed) the checkpoint never had, and the weight-map walk dies
with "no module or parameter named fc.weight".

Detect the dense MTP head from the checkpoint itself: read the
safetensors index via the same repo-file channel used for recipe.yaml
(local cache first, Hub download as fallback). If any mtp.* tensor
carries a quantization artifact suffix the head is properly quantized
and nothing changes; if every mtp.* entry is a plain .weight the whole
predictor builds unquantized (fc and layers), mirroring the existing
modelopt_fp4 and GPTQ dynamic carve-outs. Single-file checkpoints have
no index and keep current behavior; NVFP4 checkpoints are unaffected
(their mtp.layers.* carry artifacts).

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-09-27 09:54:24 +07:00
MOVZXandHinata 835c44af15 [Qwen4Exp] Reinterpret FP8 E4M3 PLE as int8 for SM86 pinned-host lookup (PR #58856)
The pinned-host PLE lookup Triton kernel only gathers and stores bytes,
but passing an fp8e4nv tensor to it fails on architectures without
native FP8 pointer support (e.g. SM86) with an error saying type
fp8e4nv is not supported (supported fp8 dtypes are fp8e4b15, fp8e5).

Since FP8 E4M3 and int8 both occupy one byte per element, reinterpret
the PLE weight view and the output tensor as int8 when the weight dtype
is float8_e4m3fn; the non-FP8 path is unchanged.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-09-27 09:54:05 +07:00
MOVZXandHinata 758cc4f9e1 [Qwen3.5] Pass quant_config and prefix to embed_tokens in Qwen3_5Model
Apply the qwen3_5_quantized_embedding patch: construct the embedding
through the model's quant config so quantized (e.g. fp8/int8)
embeddings are supported, and give it a stable prefix for weight
loading.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-09-27 09:53:21 +07:00
Jürgen SchmiedandClaude 03cae7837f [Spec Decode][Qwen4Exp] RecoverSSM for Qwen GDN and the PLE short conv
With --use-replayssm and speculative decoding, Qwen4Exp's Gated DeltaNet layers verify the
draft window from a read-only checkpoint and keep a per-token replay record
[correction(V), normalized key(K), gate(1)] in fp32. After sampling, one commit replays the
accepted tokens and writes the recurrent state once (plus the block-boundary state in align
mode), instead of snapshotting the state after every draft position. The PLE dilated short
conv follows the same protocol (accepted-suffix compaction). num_speculative_blocks = 0,
so no per-draft Mamba blocks are reserved.

- GDN verify/commit kernels: model_executor/layers/mamba/gdn/recoverssm_gdn.py
- metadata/backends: v1/attention/backends/gdn_recoverssm.py, ple_recoverssm.py
- the verify reads strided q/k/v views of the conv output and writes in place
- builders declare AttentionCGSupport.NEVER: FULL_AND_PIECEWISE falls back to PIECEWISE;
  the fallback now also counts breakable CUDA graphs (empty splitting_ops, with or
  without torch.compile) as piecewise
  attention instead of dropping to NONE
- the generic Mamba metadata builder skips the Mamba2 ReplaySSM ring-buffer workspaces
  when --use-replayssm selects RecoverSSM (the PLE short conv uses that builder)
- tests: GDN verify/commit vs the native kernels (fp32 and bf16 state, align boundaries,
  mixed and ragged batches, extreme gates), PLE vs token-by-token decode, config selection

Signed-off-by: Jürgen Schmied <juergenschmied70@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-09-27 01:17:24 +02:00
Jürgen SchmiedandClaude a80e41d690 [Mamba] Share RecoverSSM commit kernels and helpers across models
RecoverSSM's commit-plan and conv-window compaction kernels are model independent.
Move them from models/kimi_k3 into model_executor/layers/mamba/ops/recoverssm.py
(Kimi-K3 keeps its private names as import aliases; kernel source unchanged), add
mamba/recoverssm_utils.py (activation predicate, argument validation) and the
generic spec-row/request mapping helpers to v1/attention/backends/recoverssm_metadata.py.

CacheConfig.use_kda_recoverssm becomes use_recoverssm: the switch now selects RecoverSSM
for any model whose recurrent layers implement it, not only KDA.

No functional change for Kimi-K3.

Signed-off-by: Jürgen Schmied <juergenschmied70@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-09-27 00:07:04 +02:00
vr8vr8 848dbf89ec Merge branch 'main' into qwen4exp-qsa-fused-draft-metadata 2026-09-23 23:37:13 +03:00
bilikasandClaude Opus 5.5 31d78a21b9 [Spec Decode][Qwen3.8] Enable fused draft metadata updates for the QSA side cache
QSAMetadataBuilder did not set supports_draft_decode_metadata_update, so
MTP with num_speculative_tokens > 1 fell back to rebuilding attention
metadata between draft steps for the whole model.

Opt the builder in. update_draft_decode_metadata re-launches the builder's
own _build_qsa_metadata_kernel in place on the metadata's persistent
tensors, so draft-step token_to_req, logical_positions, visible_blocks,
slot_mapping and k_work_metadata come from the same code path as build().
The kernel launch is factored into _launch_qsa_metadata_kernel, and
QSAForwardMetadata carries common_slot_mapping and num_mapped_tokens so
the update does not read query_start_loc_cpu. The PyTorch fallback builder
keeps the rebuild path.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: bilikas <valdas@vbtech.eu>
2026-09-23 20:58:12 +03:00
lesj0610 045467c102 Merge remote-tracking branch 'origin/main' into sync/lesj/qwen4exp-bugfixes
Signed-off-by: lesj0610 <lesj0610@users.noreply.github.com>
2026-09-23 00:40:48 +09:00
lesj0610 51d5b3ac8a Merge remote-tracking branch 'origin/main' into sync/lesj/qwen4exp-bugfixes
Signed-off-by: lesj0610 <lesj0610@users.noreply.github.com>
2026-09-20 22:55:38 +09:00
lesj0610 5780c0125a Merge remote-tracking branch 'origin/main' into sync/lesj/qwen4exp-bugfixes
Signed-off-by: lesj0610 <lesj0610@users.noreply.github.com>
2026-09-20 09:27:57 +09:00
lesj0610 1eacb41e0f Merge remote-tracking branch 'origin/main' into sync/lesj/qwen4exp-bugfixes
Signed-off-by: lesj0610 <lesj0610@users.noreply.github.com>
2026-09-17 15:01:29 +09:00
lesj0610 b8adc7710e Merge remote-tracking branch 'origin/main' into sync/lesj/qwen4exp-bugfixes
Signed-off-by: lesj0610 <lesj0610@users.noreply.github.com>
2026-09-17 12:32:30 +09:00
lesj0610 77b8783b25 Merge remote-tracking branch 'origin/main' into sync/lesj/qwen4exp-bugfixes
Signed-off-by: lesj0610 <lesj0610@users.noreply.github.com>
2026-09-17 11:29:18 +09:00
lesj0610 6c5552776b Merge remote-tracking branch 'origin/main' into lesj/qwen4exp-bugfixes
Both sides edited the same import block in the Qwen4Exp NVIDIA and AMD
models and nothing else collided. This branch adds
`from ..common.mtp import make_mtp_hidden_buffer`, the helper the MTP
buffer extraction calls; upstream ec4a3a5370 widened the `..config`
import to `ATTENTION_LAYER_TYPES, QSA_LAYER_TYPE, Qwen4ExpConfig` for the
layer-type constants it introduced. The two changes are independent, so
the resolution keeps both lines. Every symbol still has a use outside the
import in both files.

Signed-off-by: lesj0610 <lesj0610@users.noreply.github.com>
2026-09-17 08:16:59 +09:00
lesj0610 e582b9f4a2 Merge remote-tracking branch 'origin/main' into sync/lesj/qwen4exp-bugfixes
Signed-off-by: lesj0610 <lesj0610@users.noreply.github.com>
2026-09-16 20:33:18 +09:00
lesj0610 21d2dd56d8 Merge remote-tracking branch 'origin/main' into sync/lesj/qwen4exp-bugfixes
Signed-off-by: lesj0610 <lesj0610@users.noreply.github.com>
2026-09-16 13:41:12 +09:00
lesj0610 a66a3e1b4e Merge remote-tracking branch 'origin/main' into sync/lesj/qwen4exp-bugfixes
Signed-off-by: lesj0610 <lesj0610@users.noreply.github.com>
2026-09-14 14:36:05 +09:00
lesj0610andClaude Opus 5 3b1911c0ca [Bugfix][Model] Fix Qwen4Exp MTP buffer placement, config parsing, and Triton warmup
Three initialization and config defects found serving Qwen4Exp, each with a
regression test.

**The MTP hidden buffer lands on the ambient device.** torch.empty for the
spec-decode hidden buffer takes no device, so it follows whatever device is
current rather than vllm_config.device_config.device. The allocation was
duplicated verbatim in the NVIDIA and AMD backends; it moves to
common/mtp.py so the two cannot drift, and the caller passes is_last_rank
rather than reaching for the process group from shared code.

**M-RoPE keys are reported as unrecognized.** Qwen4Exp drives M-RoPE through
mrope_section and mrope_interleaved, which transformers' rope validator does
not know for rope_type="default", so every load warns about a configuration
the model genuinely uses. Declare them at rope validation on the text config
and on the outer config that mirrors it; every other key still warns.

**transformers' layer_types spelling is unread.** Its Qwen4Exp config
canonicalizes "full_attention" entries to "qwen_sparse_attention" in memory,
so checkpoints re-exported through it serialize that name. vLLM keys QSA off
"full_attention" plus indexer_n_heads, so map the spelling back.

**Qwen4Exp is missing from the Triton warmup set.** Its GDN prefill and QSA
run on other backends, but the kernels this warmup covers -- the gated RMS
norm, the causal conv, the post-conv preparation, the decode state update --
are Triton whichever prefill kernel is chosen, so they compile on the first
prompt instead. qwen4_exp_mtp stays out: kernel_warmup passes the worker's
own model_config, which is the target model's.

tests/models/qwen4_exp/test_config.py: 15 passed.
tests/model_executor/test_qwen_triton_warmup.py: 6 passed.
ruff check, ruff format --check and mypy (Python 3.10) report what they
already report on the base commit.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: lesj0610 <lesj0610@users.noreply.github.com>
2026-09-14 07:25:01 +09:00
2222 changed files with 45159 additions and 190547 deletions
@@ -1,115 +0,0 @@
---
name: model-microbenchmark
description: Time a vLLM model's forward pass on exact batch shapes (decode, prefill, chunked prefill, prefix hit, spec-decode verify, or mixes) with `vllm bench forward`, single-GPU or DP/TP/EP across nodes. Use for per-step model latency, layout/parallelism comparisons, and checking a model-level optimization. Not for single kernels (use kernel-microbenchmark) or end-to-end serving (use `vllm bench serve`).
---
# Model Microbenchmark (`vllm bench forward`)
Builds the batch without the scheduler, runs it through MRV2's real
`execute_model`, and times the forward as a one-shot CUDA graph (prefill
included). Sampling, the speculator, and CPU step overhead are excluded, so
expect results somewhat below a serving step.
## Usage
```bash
vllm bench forward --model deepseek-ai/DeepSeek-V4-Flash \
--case decode=64*1@8192 --case prefill=8192 --case mix=4096,32*1@8192
# DP/TP/EP: one process per GPU (multi-node: add --nnodes/--node-rank/--master-addr)
torchrun --nproc-per-node=4 -m vllm.entrypoints.cli.main bench forward \
--model ... -dp 4 --enable-expert-parallel --case decode=128*1@100000
```
Requests: `[N*]Q[@K][c][:D]`, comma-separated, per DP rank.
| Form | Meaning |
| --- | --- |
| `8192` | fresh prefill, 8192 tokens |
| `32*1@8192` | 32 decodes, KV length 8192 (KV length **includes** the query) |
| `4@4096:3` | spec-decode verify, 3 draft tokens |
| `2048@10240` | prefill after an 8192-token prefix-cache hit |
| `2048@10240c` | chunked prefill: the prefix came from earlier chunks |
Verify cases require `--speculative-config` with `num_speculative_tokens`
at least as large as the case's draft count; insufficient capacity is rejected.
Hit vs chunk only differs on models with SWA bounded replay (DSV4.1): a hit
also recomputes the last window (128 tokens) of the prefix.
Or `--shapes file.json|yaml`:
`{"cases": [{"name": ..., "requests": [{"query_len", "kv_len", "num_draft_tokens", "chunked", "count"}]}]}`.
Key flags (plus all `EngineArgs`):
- `--kv-len-jitter 0.5` (default): prefixes drawn within ±50%, same on every
DP rank, capped at `max_model_len`. Use `0` for exact lengths.
- `--routing auto|real|balanced`: `auto` = balanced with dummy weights.
- `--iters 20 --samples 5`, `--output-json out.json` (per-rank latency stats,
`samples_us`, graph-vs-eager check).
- Defaults: `--load-format dummy`, prefix caching off, no torch.compile.
Read `median_us` (each replay is already the max over ranks).
## Kernel breakdown
Add `--profile-dir DIR` to see where the time goes. After timing, each case is
replayed 3 more times under `torch.profiler`; every rank writes
`DIR/<case>.rank<r>.kernels.json` (GPU time per kernel per forward, sorted)
and `DIR/<case>.rank<r>.trace.json.gz` (open in Perfetto). Rank 0's top
kernels are printed and included in `--output-json`.
```bash
python -m vllm.benchmarks.forward.profile DIR/decode.rank0.kernels.json
# Before/after a change, same case: per-kernel deltas, largest first.
python -m vllm.benchmarks.forward.profile base/decode.rank0.kernels.json new/decode.rank0.kernels.json
```
- Kernels are grouped by name without template or function arguments; the
JSON keeps one full name per group in `example`.
- Only the timed forward is included: the in-graph KV restore and rank
rendezvous are excluded (`pre_forward_us`). `span_us` (first to last
forward kernel) should be close to `median_us`, a bit higher from profiler
overhead; `busy_us` can exceed it when streams overlap.
- Compare the same rank across runs: under EP, ranks differ (e.g. collective
wait time shows up inside communication kernels).
For Nsight Systems (NVTX, NCCL, CPU side, GPU metrics), skip `--profile-dir`:
the two can't share CUPTI. Each case's 3 profiled replays run between
`cudaProfilerStart`/`Stop` in an NVTX range `forward/<case>`, so a capture range
records only them, one report per case (and rank):
```bash
nsys profile --capture-range=cudaProfilerApi --capture-range-end=repeat \
--cuda-graph-trace=node -o prof vllm bench forward ... # prof.1, prof.2, ...
torchrun --nproc-per-node=4 --no-python nsys profile ... -o prof_rank%q{RANK} \
python -m vllm.entrypoints.cli.main bench forward ...
nsys stats --report cuda_gpu_kern_sum --format csv -o kern prof.1.nsys-rep
```
Keep `--capture-range-end=repeat`: nsys's default stops after the first case
and leaves the bench hanging. `nsys stats` also counts the KV restore copies
that the JSON summary excludes.
## Caveats
- **Dummy weights lie for data-dependent kernels.** Dummy gates route every
token to the same experts (hence balanced routing). Cross-check suspicious
kernels with real weights.
- **torch.compile models (e.g. Qwen3/3.5) look slow**: no compile here. Use
them for correctness only; trust perf for models under `vllm/models/`.
- `sample spread > 0.5%` warns of unstable timing. DSV4.1 DEP prefill is
inherently ~0.5–1% (MegaMoE cross-rank sync, host engram reads).
- `graph_vs_eager` may be nonzero for small batches where models branch on
`is_current_stream_capturing()`; that's expected, not a bug.
- KV cache is zero-initialized (`--kv-init zeros`); random KV was within 3%.
- Uniform DP only: every rank gets the same case; no idle ranks.
- CUDA only: it hangs on ROCm (untested cause: likely the in-graph timing
events).
- Large prefills/prefix hits need room: raise `--max-num-batched-tokens` /
`--max-model-len` / `--gpu-memory-utilization` if build fails on KV blocks.
- Startup cost is JIT/autotune, not the bench: install FlashInfer jit-cache
and cubin; DeepGEMM warmup is skipped (each case's eager pass compiles its
own shapes).
- To compare against serving decode: `vllm bench latency` with 65- vs
1-token outputs, divided by 64, with a FULL decode graph.
+5 -11
View File
@@ -5,12 +5,9 @@ description: Find vLLM deployment recipes and configs for LLMs and Hugging Face
# Model Recipes
Model serving recipes are maintained in this repository's
[recipes/models/ directory](../../../recipes/models). Omni (image, video,
audio) recipes stay in vllm-project/recipes for now (moving to vllm-omni),
and Ascend NPU content is inline in these recipes (moving to vllm-ascend
overlays). The published catalog, rendered by
[vllm-project/recipes](https://github.com/vllm-project/recipes), is at
Model serving recipes are maintained in the
[models/ directory of vllm-project/recipes](https://github.com/vllm-project/recipes/tree/main/models).
The published catalog is at
[recipes.vllm.ai](https://recipes.vllm.ai).
## Usage notes
@@ -149,9 +146,6 @@ uploads, untested hardware claims, or non-standard flags that only work for one
setup. Before opening a pull request, confirm that the user ran the
configuration and can share the results.
Edit the YAML under [recipes/models/](../../../recipes/models) in this
repository, following the recipes repository's
Follow the recipes repository's
[add-recipe skill](https://github.com/vllm-project/recipes/blob/main/.agents/skills/add-recipe/SKILL.md)
for the schema. Validate and preview it from a checkout of
`vllm-project/recipes` pointed at this directory, as
[recipes/README.md](../../../recipes/README.md) describes.
in a checkout of `vllm-project/recipes` or the user's intended fork.
+5 -18
View File
@@ -121,9 +121,6 @@ For each deleted config:
Deleted processors follow the same steps in
`vllm/transformers_utils/processors/__init__.py`.
Before merging, rebase and grep again for the old field names: PRs that land
in between still read them.
Done when every line of the inventory output is deleted, or kept with a
reason that meets the bar in step 2. The PR also carries the config diff
results and evals for any changed values.
@@ -131,15 +128,13 @@ results and evals for any changed values.
## Upstreaming hazards
Upstream configs are `@strict` dataclasses. Checkpoint keys upstream does not
declare survive as plain attributes, but only when the checkpoint sets them.
Defaults that only vLLM sets, and values vLLM derives, vanish.
declare survive as plain attributes. Defaults that only vLLM sets, and values
vLLM derives, vanish.
| Hazard | Signal | Fix |
| --- | --- | --- |
| Upstream consumes legacy keys | `__post_init__` pops keys the model reads (`deepseek_v4` `compress_ratios`) | Read the upstream replacement (`layer_types`, `mlp_layer_types`, per-layer `rope_parameters`) |
| Renamed or dropped fields | Diff shows `<MISSING>` for a field the model reads | Read the upstream name; derive dropped flags from what implies them (`enable_moe_block` from `num_experts`) |
| Undeclared field without a default | `config.x` raises `AttributeError` for configs that omit `x`, e.g. minimal test configs (A.X-K1 `scoring_func`). Dumps of real checkpoints miss it | Upstream ignoring `x` usually means vLLM can stop reading it; if vLLM truly needs it, read `getattr(config, "x", <old default>)` |
| Top-level config stops forwarding | The vendored top-level config forwarded reads to `text_config`; upstream does not. `getattr(config, "x", default)` silently takes the default, e.g. on MTP draft configs from `SpeculativeConfig.hf_config_override`, which the dumps do not build | Read from `get_text_config()`; grep `getattr` reads of every field, including speculative-decoding paths |
| Type check on the config | `ctx.get_hf_config(SomeConfig)` relies on the vendored base class | Accept the upstream class too, e.g. a `Cosmos3ProcessingInfo` overriding `get_hf_config` |
| Remote code takes over | Checkpoint `config.json` has `auto_map`; with `--trust-remote-code`, the hub class replaces the native one once vLLM stops registering its own | Check the remote class loads; if not, register the upstream class for that `model_type` |
| Read-only upstream fields | Properties or derived fields break subclasses and `hf_overrides` (`nemotron_h` `hybrid_override_pattern`) | Override the source field (`layers_block_type`) instead |
@@ -164,10 +159,10 @@ Version gates from the inventory need a decision:
- **Patches citing a Transformers issue or PR:**
- Delete the patch when its fix is in the floor release.
- For an issue, find the PR that closed it and check that PR. A closed
issue is not a fixed one: issues get closed while the bug is still in
the release.
issue is not a fixed one: #43329 was closed while the bug was still in
5.16.1.
- Re-run the patched behaviour before deleting. A fix in the release does
not always cover vLLM's path: Emu3's processor gained BOS upstream, but
not always cover vLLM's path: #47924 made Emu3's processor add BOS, but
vLLM tokenizes without the processor.
- Links that only explain a design choice (e.g. a float32 RoPE note) are
not patches; leave them.
@@ -181,11 +176,3 @@ Version gates from the inventory need a decision:
Add `tests/models/test_initialization.py -k "<archs>"` where a GPU is
available, and a model eval for any model whose resolved config changed.
Models in `vllm/models/` are high priority: whenever their config or model
code changes, run GSM8k (`tests/evals/gsm8k/`) and GPQA Diamond
(`tests/evals/gpt_oss/`) on `main` and the branch, even when the dumps show
no diff, and put both scores in the PR. FlashInfer autotuning picks kernel
tactics afresh on every server start, which moved DeepSeek-V4-Flash GSM8k by
about 1 point; pass `--kernel-config '{"enable_flashinfer_autotune": false}'`
to both servers so greedy outputs are comparable.
+1
View File
@@ -17,6 +17,7 @@ ALLOWED_UNSTABLE_LIBRARIES: tuple[str, ...] = (
# _C_stable_libtorch, but these two modules still bind libtorch directly.
"_C.abi3.so",
"_rocm_C.abi3.so",
"third_party/deep_gemm/_C*.so",
)
@@ -56,6 +56,7 @@ steps:
- "!vllm/distributed/device_communicators/shm_object_storage.py"
- "!vllm/distributed/device_communicators/symm_mem.py"
- "!vllm/distributed/device_communicators/mnnvl_compat.py"
- "!vllm/distributed/device_communicators/ray_communicator.py"
- "!vllm/third_party/flashmla/"
- "!vllm/third_party/pynvml.py"
- "!vllm/_custom_ops.py"
+1 -1
View File
@@ -277,7 +277,7 @@ steps:
# `uv pip install "vllm[zen]"`, which forces uv to re-resolve vLLM's full
# dependency closure and chokes on its transitive triton-cpu URL dependency.
# Keep this pin in sync with the `zen` extra in setup.py.
ZENTORCH_VERSION: "2.14.0.1"
ZENTORCH_VERSION: "2.13.0.1"
retry:
automatic:
- exit_status: 1 # Transient Docker/BuildKit failure
@@ -15,7 +15,7 @@ steps:
limit: 2
- exit_status: -10 # Agent was lost
limit: 2
- label: ":intel: (BMG) Example Test"
- label: "XPU example Test"
key: xpu-example-test
depends_on:
- image-build-xpu
@@ -37,7 +37,7 @@ steps:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'bash .buildkite/scripts/hardware_ci/run-intel-ci-test.sh example'
- label: ":intel: (BMG) V1 test"
- label: "XPU V1 test"
key: xpu-v1-test
depends_on:
- image-build-xpu
@@ -60,7 +60,7 @@ steps:
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'bash .buildkite/scripts/hardware_ci/run-intel-ci-test.sh v1'
parallelism: 2
- label: ":intel: (BMG) W8A8 FP8 Linear Examples"
- label: "XPU W8A8 FP8 Linear Examples"
key: xpu-w8a8-fp8-linear-examples
depends_on:
- image-build-xpu
@@ -83,7 +83,7 @@ steps:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'bash .buildkite/scripts/hardware_ci/run-intel-ci-test.sh w8a8-fp8-linear'
- label: ":intel: (BMG) Server test"
- label: "XPU server test"
key: xpu-server-test
depends_on:
- image-build-xpu
@@ -105,7 +105,7 @@ steps:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'bash .buildkite/scripts/hardware_ci/run-intel-ci-test.sh server'
- label: ":intel: (BMG) Quantization test"
- label: "XPU quantization test"
key: xpu-quantization-test
depends_on:
- image-build-xpu
@@ -127,7 +127,7 @@ steps:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'bash .buildkite/scripts/hardware_ci/run-intel-ci-test.sh quantization'
- label: ":intel: (BMG) Compressed tensors FP8 test"
- label: "XPU compressed tensors FP8 test"
key: xpu-compressed-tensors-fp8-test
depends_on:
- image-build-xpu
@@ -150,7 +150,7 @@ steps:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'bash .buildkite/scripts/hardware_ci/run-intel-ci-test.sh compressed-tensors-fp8'
- label: ":intel: (BMG) Graph"
- label: "XPU Graph"
key: xpu-graph
depends_on:
- image-build-xpu
@@ -1,8 +1,8 @@
group: Basic Correctness Intel
group: Basic Correctness
depends_on:
- image-build-xpu
steps:
- label: ":intel: (BMG) Sleep Mode"
- label: XPU Sleep Mode
key: xpu-sleep-mode
timeout_in_minutes: 45
device: intel_gpu
+2 -2
View File
@@ -1,8 +1,8 @@
group: Benchmarks Intel
group: Benchmarks
depends_on:
- image-build-xpu
steps:
- label: ":intel: (BMG) Benchmarks CLI"
- label: Benchmarks CLI Test
key: benchmarks-cli-test
timeout_in_minutes: 40
device: intel_gpu
@@ -1,33 +0,0 @@
group: Distributed Intel
depends_on:
- image-build-xpu
steps:
- label: ":intel: (BMG) Distributed Torchrun + Shutdown"
key: distributed-torchrun-shutdown-tests-2-gpus
timeout_in_minutes: 50
device: intel_gpu
agent_tags:
label: production
gpu: 2+
mem: 24+
no_plugin: true
working_dir: "."
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
VLLM_TEST_DEVICE: "xpu"
source_file_dependencies:
- vllm/distributed/
- "!vllm/distributed/kv_transfer/"
- vllm/engine/
- vllm/executor/
- vllm/worker/worker_base.py
- vllm/v1/engine/
- vllm/v1/worker/
- tests/distributed/
- tests/v1/shutdown
- tests/v1/worker/test_worker_memory_snapshot.py
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests && pytest -v -s v1/shutdown'
+3 -3
View File
@@ -2,7 +2,7 @@ group: Engine Intel
depends_on:
- image-build-xpu
steps:
- label: ":intel: (BMG) Engine"
- label: Engine
key: engine
timeout_in_minutes: 40
device: intel_gpu
@@ -40,7 +40,7 @@ steps:
'cd tests &&
pytest -v -s engine/test_arg_utils.py test_sequence.py test_logger.py test_vllm_port.py jit_monitor/test_hooks.py'
- label: ":intel: (BMG) Engine (1 GPU)"
- label: Engine (1 GPU)
key: engine-1-gpu
timeout_in_minutes: 30
device: intel_gpu
@@ -65,7 +65,7 @@ steps:
pytest -v -s v1/engine --ignore v1/engine/test_preprocess_error_handling.py --ignore v1/engine/test_core_engine_actor_manager.py &&
pytest -v -s test_config.py'
- label: ":intel: (BMG) V1 e2e (2 GPUs)"
- label: V1 e2e (2 GPUs)
key: v1-e2e-2-gpus
timeout_in_minutes: 30
device: intel_gpu
+1 -1
View File
@@ -1,4 +1,4 @@
group: Entrypoints Intel
group: Entrypoints
depends_on:
- image-build-xpu
steps:
@@ -1,8 +1,8 @@
group: Expert Parallelism Intel
group: Expert Parallelism
depends_on:
- image-build-xpu
steps:
- label: ":intel: (BMG) EPLB Algorithm"
- label: EPLB Algorithm
key: eplb-algorithm
timeout_in_minutes: 45
device: intel_gpu
+2 -2
View File
@@ -2,7 +2,7 @@ group: Kernels Intel
depends_on:
- image-build-xpu
steps:
- label: ":intel: (BMG) vLLM IR Tests"
- label: vLLM IR Tests
key: vllm-ir-tests
timeout_in_minutes: 30
device: intel_gpu
@@ -26,7 +26,7 @@ steps:
pytest -v -s ir &&
pytest -v -s kernels/ir'
- label: ":intel: (BMG) UVA Kernel Tests"
- label: UVA Kernel Tests
key: uva-kernel-tests
timeout_in_minutes: 20
device: intel_gpu
+3 -3
View File
@@ -1,8 +1,8 @@
group: LM Eval Intel
group: LM Eval
depends_on:
- image-build-xpu
steps:
- label: ":intel: (BMG) GPQA Eval (GPT-OSS)"
- label: "XPU GPQA Eval (GPT-OSS)"
key: xpu-gpqa-eval-gpt-oss
timeout_in_minutes: 60
device: intel_gpu
@@ -25,7 +25,7 @@ steps:
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests &&
pytest -s -v evals/gpt_oss/test_gpqa_correctness.py --config-list-file=configs/models-xpu.txt'
- label: ":intel: (BMG) LM Eval Large Models(2 GPUs)"
- label: LM Eval Large Models(2 GPUs)
key: lm-eval-large-models-2-gpus
timeout_in_minutes: 45
device: intel_gpu
+7 -7
View File
@@ -2,7 +2,7 @@ group: LoRA Intel
depends_on:
- image-build-xpu
steps:
- label: ":intel: (BMG) LoRA Runtime + Utils"
- label: LoRA Runtime + Utils
key: lora-runtime-utils
timeout_in_minutes: 45
device: intel_gpu
@@ -35,7 +35,7 @@ steps:
pytest -v -s lora/test_utils.py &&
pytest -v -s lora/test_worker.py'
- label: ":intel: (BMG) LoRA Fused/MoE Kernels"
- label: LoRA Fused/MoE Kernels
key: lora-fused-moe-kernels
timeout_in_minutes: 45
device: intel_gpu
@@ -60,7 +60,7 @@ steps:
pytest -v -s lora/test_fused_moe_lora_kernel.py &&
pytest -v -s lora/test_moe_lora_align_sum.py --deselect="tests/lora/test_moe_lora_align_sum.py::test_moe_lora_align_block_size_mixed_base_and_lora[1]"'
- label: ":intel: (BMG) LoRA Punica Kernels"
- label: LoRA Punica Kernels
key: lora-punica-kernels
timeout_in_minutes: 45
device: intel_gpu
@@ -87,7 +87,7 @@ steps:
pytest -v -s lora/test_punica_ops.py::test_kernels_hidden_size &&
pytest -v -s lora/test_punica_ops.py::test_add_lora_fused_moe_early_exit'
- label: ":intel: (BMG) LoRA Punica FP8/XPU Ops"
- label: LoRA Punica FP8/XPU Ops
key: lora-punica-fp8-xpu-ops
timeout_in_minutes: 60
device: intel_gpu
@@ -112,7 +112,7 @@ steps:
pytest -v -s lora/test_punica_ops_fp8.py &&
pytest -v -s lora/test_punica_xpu_ops.py'
- label: ":intel: (BMG) LoRA Models"
- label: LoRA Models
key: lora-models
timeout_in_minutes: 45
device: intel_gpu
@@ -139,7 +139,7 @@ steps:
pytest -v -s lora/test_chatglm3_tp.py &&
pytest -v -s lora/test_llama_tp.py::test_llama_lora'
- label: ":intel: (BMG) LoRA Multimodal"
- label: LoRA Multimodal
key: lora-multimodal
timeout_in_minutes: 45
device: intel_gpu
@@ -163,7 +163,7 @@ steps:
export VLLM_WORKER_MULTIPROC_METHOD=spawn &&
pytest -v -s lora/test_whisper.py'
- label: ":intel: (BMG) LoRA Add + Multimodal"
- label: LoRA Add + Multimodal
key: lora-add-multimodal
timeout_in_minutes: 45
device: intel_gpu
+12 -12
View File
@@ -2,7 +2,7 @@ group: Miscellaneous Intel
depends_on:
- image-build-xpu
steps:
- label: ":intel: (BMG) Batch Invariance"
- label: XPU Batch Invariance
key: xpu-batch-invariance
timeout_in_minutes: 60
num_devices: 2
@@ -30,7 +30,7 @@ steps:
pytest -v -s v1/determinism/test_xpu_batch_invariant_ut.py &&
bash ../.buildkite/scripts/hardware_ci/run-xpu-batch-invariance.sh'
- label: ":intel: (BMG) V1 Core + KV + Metrics"
- label: V1 Core + KV + Metrics
key: v1-core-kv-metrics-intel
timeout_in_minutes: 45
device: intel_gpu
@@ -64,7 +64,7 @@ steps:
VLLM_BATCH_INVARIANT=1 pytest -v -s -m "not cpu_test" v1/ec_connector/unit
--ignore=v1/ec_connector/unit/test_ec_cpu_connector.py'
- label: ":intel: (BMG) EC CPU Connector (XPU)"
- label: EC CPU Connector (XPU)
key: ec-cpu-connector-xpu
timeout_in_minutes: 45
device: intel_gpu
@@ -96,7 +96,7 @@ steps:
cd tests &&
pytest -v -s v1/ec_connector/unit/test_ec_cpu_connector.py'
- label: ":intel: (BMG) V1 Sample + Logits"
- label: V1 Sample + Logits
key: v1-sample-logits-intel
timeout_in_minutes: 90
device: intel_gpu
@@ -141,7 +141,7 @@ steps:
limit: 1
parallelism: 4
- label: ":intel: (BMG) Basic Models Tests (Initialization)"
- label: Basic Models Tests (Initialization)
key: basic-models-tests-initialization
timeout_in_minutes: 60
device: intel_gpu
@@ -166,7 +166,7 @@ steps:
cd tests &&
pytest -v -s models/test_initialization.py::test_can_initialize_large_subset[Eagle3MiniMaxM2ForCausalLM]'
- label: ":intel: (BMG) CPU Offload"
- label: XPU CPU Offload
key: xpu-cpu-offload
timeout_in_minutes: 60
device: intel_gpu
@@ -194,7 +194,7 @@ steps:
pytest -v -s v1/kv_offload &&
pytest -v -s v1/kv_connector/unit/test_offloading_connector.py'
- label: ":intel: (BMG) NixlConnector PD accuracy (4 GPUs)"
- label: NixlConnector PD accuracy (4 GPUs)
key: nixlconnector-pd-accuracy-4-gpus
timeout_in_minutes: 60
num_devices: 4
@@ -223,7 +223,7 @@ steps:
PREFILLER_TP_SIZE=1 DECODER_TP_SIZE=2 bash v1/kv_connector/nixl_integration/run_xpu_disagg_accuracy_test.sh &&
PREFILLER_TP_SIZE=2 DECODER_TP_SIZE=2 bash v1/kv_connector/nixl_integration/run_xpu_disagg_accuracy_test.sh'
- label: ":intel: (BMG) Regression"
- label: Regression
key: regression
timeout_in_minutes: 50
device: intel_gpu
@@ -256,7 +256,7 @@ steps:
'cd tests &&
pytest -v -s test_regression.py'
- label: ":intel: (BMG) Metrics, Tracing (2 GPUs)"
- label: Metrics, Tracing (2 GPUs)
key: metrics-tracing-2-gpus
timeout_in_minutes: 45
num_devices: 2
@@ -291,7 +291,7 @@ steps:
'cd tests &&
pytest -v -s v1/tracing'
- label: ":intel: (BMG) Async Engine, Inputs, Utils, Worker"
- label: Async Engine, Inputs, Utils, Worker
key: async-engine-inputs-utils-worker
timeout_in_minutes: 55
device: intel_gpu
@@ -330,7 +330,7 @@ steps:
pytest -v -s -m "not cpu_test" ./multimodal &&
pytest -v -s utils_ --ignore=utils_/test_mem_utils.py'
- label: ":intel: (BMG) Detokenizer Stop Strings (XPU)"
- label: Detokenizer Stop Strings (XPU)
key: detokenizer-stop-strings-xpu
timeout_in_minutes: 45
device: intel_gpu
@@ -367,7 +367,7 @@ steps:
'cd tests &&
pytest -v -s detokenizer/test_stop_strings.py'
- label: ":intel: (BMG) Fusion Unit Tests"
- label: Fusion Unit Tests
key: fusion-unit-tests
timeout_in_minutes: 30
device: intel_gpu
@@ -2,7 +2,7 @@ group: Model Executor Intel
depends_on:
- image-build-xpu
steps:
- label: ":intel: (BMG) Model Executor"
- label: Model Executor (Intel)
key: model-executor-intel
timeout_in_minutes: 45
device: intel_gpu
@@ -28,9 +28,9 @@ steps:
export VLLM_WORKER_MULTIPROC_METHOD=spawn &&
export PYTHONFAULTHANDLER=1 &&
cd tests &&
pytest -v -s model_executor -m "not slow_test" --ignore="model_executor/layers/test_rocm_unquantized_gemm.py" --ignore="model_executor/model_loader/tensorizer_loader/test_tensorizer.py" --ignore="model_executor/model_loader/test_reload.py" --deselect="tests/model_executor/test_enabled_custom_ops.py::test_topk_gating_enabled_by_cdna"'
pytest -v -s model_executor -m "not slow_test" --ignore="model_executor/layers/test_rocm_unquantized_gemm.py" --ignore="model_executor/model_loader/tensorizer_loader/test_tensorizer.py" --deselect="tests/model_executor/model_loader/test_reload.py::test_kv_scale_reload"'
- label: ":intel: (BMG) Tensorizer Model Loader Tests (XPU)"
- label: Tensorizer Model Loader Tests (XPU)
key: tensorizer-model-loader-tests-xpu
timeout_in_minutes: 45
device: intel_gpu
@@ -1,8 +1,8 @@
group: Models - Distributed Intel
group: Models - Distributed
depends_on:
- image-build-xpu
steps:
- label: ":intel: (BMG) Distributed Model Tests"
- label: Distributed Model Tests (2 GPUs)
key: distributed-model-tests-2-gpus
timeout_in_minutes: 65
device: intel_gpu
@@ -1,8 +1,8 @@
group: Models - Multimodal Intel
group: Models - Multimodal
depends_on:
- image-build-xpu
steps:
- label: ":intel: (BMG) Multi-Modal Models (Standard) 1: qwen2"
- label: "Multi-Modal Models (Standard) 1: qwen2"
key: multi-modal-models-standard-1-qwen2
timeout_in_minutes: 70
device: intel_gpu
@@ -26,7 +26,7 @@ steps:
pytest -v -s models/multimodal/generation/test_common.py -m core_model -k "qwen2" &&
pytest -v -s models/multimodal/generation/test_ultravox.py -m core_model'
- label: ":intel: (BMG) Multi-Modal Models (Standard) 2: qwen3 + gemma"
- label: "Multi-Modal Models (Standard) 2: qwen3 + gemma"
key: multi-modal-models-standard-2-qwen3-gemma
timeout_in_minutes: 90
device: intel_gpu
@@ -51,7 +51,7 @@ steps:
pytest -v -s models/multimodal/generation/test_mm_prefix_lm.py -m core_model &&
pytest -v -s models/multimodal/generation/test_qwen2_5_vl.py -m core_model'
- label: ":intel: (BMG) Multi-Modal Models (Standard) 3: llava + qwen2_vl"
- label: "Multi-Modal Models (Standard) 3: llava + qwen2_vl"
key: multi-modal-models-standard-3-llava-qwen2-vl
timeout_in_minutes: 65
device: intel_gpu
@@ -76,7 +76,7 @@ steps:
pytest -v -s models/multimodal/generation/test_gemma_ragged_audio.py -m core_model
pytest -v -s models/multimodal/generation/test_qwen2_vl.py -m core_model'
- label: ":intel: (BMG) Multi-Modal Models (Standard) 4: other + whisper"
- label: "Multi-Modal Models (Standard) 4: other + whisper"
key: multi-modal-models-standard-4-other-whisper
timeout_in_minutes: 45
device: intel_gpu
@@ -99,7 +99,7 @@ steps:
'cd tests &&
pytest -v -s models/multimodal -m core_model --ignore models/multimodal/generation/test_common.py --ignore models/multimodal/generation/test_ultravox.py --ignore models/multimodal/generation/test_qwen2_5_vl.py --ignore models/multimodal/generation/test_qwen2_vl.py --ignore models/multimodal/generation/test_whisper.py --ignore models/multimodal/generation/test_gemma_ragged_audio.py --ignore models/multimodal/generation/test_memory_leak.py --ignore models/multimodal/processing'
- label: ":intel: (BMG) Multi-Modal Processor" # 44min
- label: Multi-Modal Processor # 44min
key: multi-modal-processor
timeout_in_minutes: 60
device: intel_gpu
@@ -126,7 +126,7 @@ steps:
--num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT --shard-id=$$BUILDKITE_PARALLEL_JOB'
parallelism: 4
- label: ":intel: (BMG) Multi-Modal Models (Extended Generation 1)"
- label: Multi-Modal Models (Extended Generation 1)
key: multi-modal-models-extended-generation-1
timeout_in_minutes: 60
device: intel_gpu
@@ -1,8 +1,8 @@
group: Quantization Intel
group: Quantization
depends_on:
- image-build-xpu
steps:
- label: ":intel: (BMG) Quantization"
- label: Quantization
key: quantization
timeout_in_minutes: 30
env:
+2 -2
View File
@@ -2,7 +2,7 @@ group: Samplers Intel
depends_on:
- image-build-xpu
steps:
- label: ":intel: (BMG) Samplers Test (XPU Kernel)"
- label: Samplers Test (XPU Kernel)
key: samplers-test-xpu-kernel-intel
timeout_in_minutes: 40
device: intel_gpu
@@ -29,7 +29,7 @@ steps:
VLLM_XPU_USE_SAMPLER_KERNEL=1 pytest -v -s samplers
--ignore=samplers/test_beam_search.py'
- label: ":intel: (BMG) Samplers Beam Search Test (XPU Kernel)"
- label: Samplers Beam Search Test (XPU Kernel)
key: samplers-beam-search-test-xpu-kernel-intel
timeout_in_minutes: 40
device: intel_gpu
+13 -13
View File
@@ -14,7 +14,7 @@ steps:
limit: 2
- exit_status: -10 # Agent was lost
limit: 2
- label: ":intel: (BMG) Example Test"
- label: "XPU example Test"
key: xpu-example-test
depends_on:
- image-build-xpu
@@ -30,7 +30,7 @@ steps:
REPO: "vllm-ci-test-repo"
source_file_dependencies:
- vllm/
- .buildkite/intel_jobs/test-intel.yaml
- .buildkite/intel_jobs/test-intel.yaml
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
@@ -48,7 +48,7 @@ steps:
python3 examples/basic/offline_inference/generate.py --model superjob/Qwen3-4B-Instruct-2507-GPTQ-Int4 --max-model-len 8192 &&
python3 examples/basic/offline_inference/generate.py --model INCModel/Qwen3-30B-A3B-Instruct-2507-MXFP4-LLMC -tp 2 --max-model-len 8192
'
- label: ":intel: (BMG) W8A8 FP8 Linear Examples"
- label: "XPU W8A8 FP8 Linear Examples"
key: xpu-w8a8-fp8-linear-examples
depends_on:
- image-build-xpu
@@ -76,7 +76,7 @@ steps:
python3 examples/basic/offline_inference/generate.py --linear-backend xpu --model meta-llama/Llama-3.2-1B-Instruct --quantization fp8_per_tensor --enforce-eager --max-model-len 4096 &&
python3 examples/basic/offline_inference/generate.py --linear-backend torch --model meta-llama/Llama-3.2-1B-Instruct --quantization fp8_per_tensor --enforce-eager --max-model-len 4096
'
- label: ":intel: (BMG) V1 test"
- label: "XPU V1 test"
key: xpu-v1-test
depends_on:
- image-build-xpu
@@ -92,7 +92,7 @@ steps:
REPO: "vllm-ci-test-repo"
source_file_dependencies:
- vllm/
- .buildkite/intel_jobs/test-intel.yaml
- .buildkite/intel_jobs/test-intel.yaml
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
@@ -105,9 +105,9 @@ steps:
pytest -v -s v1/test_serial_utils.py &&
pytest -v -s v1/e2e/general/test_correctness_sliding_window.py &&
pytest -v -s v1/spec_decode --ignore=v1/spec_decode/test_max_len.py --ignore=v1/spec_decode/test_speculators_eagle3.py --ignore=v1/spec_decode/test_acceptance_length.py --ignore=v1/spec_decode/test_speculators_correctness.py --deselect=tests/v1/spec_decode/test_mtp.py::test_glm_mtp_defers_lm_head &&
pytest -v -s v1/kv_connector/unit --ignore=v1/kv_connector/unit/test_lmcache_integration.py --ignore=v1/kv_connector/unit/test_hf3fs_client.py --ignore=v1/kv_connector/unit/test_hf3fs_connector.py --ignore=v1/kv_connector/unit/test_hf3fs_metadata_server.py --ignore=v1/kv_connector/unit/test_offloading_connector.py --ignore=v1/kv_connector/unit/test_multi_connector.py --ignore=v1/kv_connector/unit/test_example_connector.py --deselect=tests/v1/kv_connector/unit/test_nixl_connector.py::test_abort_timeout_on_prefiller[ray]'
pytest -v -s v1/kv_connector/unit --ignore=v1/kv_connector/unit/test_lmcache_integration.py --ignore=v1/kv_connector/unit/test_hf3fs_client.py --ignore=v1/kv_connector/unit/test_hf3fs_connector.py --ignore=v1/kv_connector/unit/test_hf3fs_metadata_server.py --ignore=v1/kv_connector/unit/test_offloading_connector.py --ignore=v1/kv_connector/unit/test_nixl_connector_hma.py --ignore=v1/kv_connector/unit/test_multi_connector.py --ignore=v1/kv_connector/unit/test_example_connector.py --deselect=tests/v1/kv_connector/unit/test_nixl_connector.py::test_abort_timeout_on_prefiller[ray]'
parallelism: 2
- label: ":intel: (BMG) Qwen3-30B + V1 KV Connector"
- label: "XPU Qwen3-30B + V1 KV Connector"
key: xpu-qwen3-30b-v1-kv-connector
depends_on:
- image-build-xpu
@@ -133,7 +133,7 @@ steps:
pytest -v -s v1/kv_connector/unit/test_example_connector.py &&
cd .. &&
VLLM_XPU_FUSED_MOE_USE_REF=1 python3 examples/basic/offline_inference/generate.py --model Qwen/Qwen3-30B-A3B-Instruct-2507-FP8 --enforce-eager -tp 2 --max-model-len 8192'
- label: ":intel: (BMG) Server test"
- label: "XPU server test"
key: xpu-server-test
depends_on:
- image-build-xpu
@@ -149,14 +149,14 @@ steps:
REPO: "vllm-ci-test-repo"
source_file_dependencies:
- vllm/
- .buildkite/intel_jobs/test-intel.yaml
- .buildkite/intel_jobs/test-intel.yaml
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests &&
pytest -v -s entrypoints/multimodal/openai/chat_completion/test_audio_in_video.py &&
pytest -v -s benchmarks/test_serve_cli.py'
- label: ":intel: (BMG) Quantization test"
- label: "XPU quantization test"
key: xpu-quantization-test
depends_on:
- image-build-xpu
@@ -172,14 +172,14 @@ steps:
REPO: "vllm-ci-test-repo"
source_file_dependencies:
- vllm/
- .buildkite/intel_jobs/test-intel.yaml
- .buildkite/intel_jobs/test-intel.yaml
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests &&
pytest -v -s quantization/test_auto_round.py &&
pytest -v -s quantization/test_online.py --deselect=tests/quantization/test_online.py::test_online_quantization_loads_real_weights --deselect="tests/quantization/test_online.py::test_online_quantization[False-requantization_mxfp8_ptcp_fp8]"'
- label: ":intel: (BMG) Compressed tensors FP8 test"
- label: "XPU compressed tensors FP8 test"
key: xpu-compressed-tensors-fp8-test
depends_on:
- image-build-xpu
@@ -204,7 +204,7 @@ steps:
'cd tests &&
pytest -v -s quantization/test_compressed_tensors.py::test_compressed_tensors_fp8 &&
pytest -v -s quantization/test_compressed_tensors.py::test_compressed_tensors_fp8_block_enabled'
- label: ":intel: (BMG) Graph"
- label: "XPU Graph"
key: xpu-graph
depends_on:
- image-build-xpu
+479 -214
View File
@@ -703,6 +703,385 @@ steps:
DOCKER_BUILDKIT: "1"
DOCKERHUB_USERNAME: "vllmbot"
# =============================================================================
# ROCm Release Pipeline (x86_64 only)
# =============================================================================
#
# vLLM version is determined by the Buildkite checkout (like CUDA pipeline).
# To build a specific version, trigger the build from that branch/tag.
#
# Environment variables for ROCm builds (set via Buildkite UI or schedule):
#
# Note: ROCm version is determined by BASE_IMAGE in docker/Dockerfile.rocm_base
#
# =============================================================================
- group: "Build ROCm Wheel / Image "
key: "build-rocm-wheel-image"
depends_on: ~
steps:
# ROCm Job 1: Build ROCm Base Wheels (with S3 caching)
- label: ":rocm: Build ROCm Base Image & Wheels"
id: build-rocm-base-wheels
depends_on: ~
agents:
queue: cpu_queue_release
commands:
- |
set -euo pipefail
# Generate cache key
CACHE_KEY=$$(.buildkite/scripts/cache-rocm-base-wheels.sh key)
ECR_CACHE_TAG="public.ecr.aws/q9t5s3a7/vllm-release-repo:$${CACHE_KEY}-rocm-base"
echo "========================================"
echo "ROCm Base Build Configuration"
echo "========================================"
echo " CACHE_KEY: $${CACHE_KEY}"
echo " ECR_CACHE_TAG: $${ECR_CACHE_TAG}"
echo "========================================"
# Login to ECR
aws ecr-public get-login-password --region us-east-1 | \
docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7
IMAGE_EXISTS=false
WHEELS_EXIST=false
# Check ECR for Docker image
if docker manifest inspect "$${ECR_CACHE_TAG}" > /dev/null 2>&1; then
IMAGE_EXISTS=true
echo "ECR image cache HIT"
fi
# Check S3 for wheels
WHEEL_CACHE_STATUS=$(.buildkite/scripts/cache-rocm-base-wheels.sh check)
if [ "$${WHEEL_CACHE_STATUS}" = "hit" ]; then
WHEELS_EXIST=true
echo "S3 wheels cache HIT"
fi
# Scenario 1: Both cached (best case)
if [ "$${IMAGE_EXISTS}" = "true" ] && [ "$${WHEELS_EXIST}" = "true" ]; then
echo ""
echo "FULL CACHE HIT - Reusing both image and wheels"
echo ""
# Download wheels
.buildkite/scripts/cache-rocm-base-wheels.sh download
# Save ECR tag for downstream jobs
buildkite-agent meta-data set "rocm-base-image-tag" "$${ECR_CACHE_TAG}"
# Scenario 2: Full rebuild needed
else
echo ""
echo " CACHE MISS - Building from scratch..."
echo ""
# Build full base image and push to ECR
DOCKER_BUILDKIT=1 docker buildx build \
--file docker/Dockerfile.rocm_base \
--tag "$${ECR_CACHE_TAG}" \
--build-arg USE_SCCACHE=1 \
--build-arg SCCACHE_BUCKET_NAME=vllm-build-sccache \
--build-arg SCCACHE_REGION_NAME=us-west-2 \
--build-arg SCCACHE_S3_NO_CREDENTIALS=0 \
--push \
.
# Build wheel extraction stage
DOCKER_BUILDKIT=1 docker buildx build \
--file docker/Dockerfile.rocm_base \
--tag rocm-base-debs:$${BUILDKITE_BUILD_NUMBER} \
--target debs_wheel_release \
--build-arg USE_SCCACHE=1 \
--build-arg SCCACHE_BUCKET_NAME=vllm-build-sccache \
--build-arg SCCACHE_REGION_NAME=us-west-2 \
--build-arg SCCACHE_S3_NO_CREDENTIALS=0 \
--load \
.
# Extract and upload wheels
mkdir -p artifacts/rocm-base-wheels
cid=$(docker create rocm-base-debs:$${BUILDKITE_BUILD_NUMBER})
docker cp $${cid}:/app/debs/. artifacts/rocm-base-wheels/
docker rm $${cid}
.buildkite/scripts/cache-rocm-base-wheels.sh upload
# Cache base docker image to ECR
docker push "$${ECR_CACHE_TAG}"
buildkite-agent meta-data set "rocm-base-image-tag" "$${ECR_CACHE_TAG}"
echo ""
echo " Build complete - Image and wheels cached"
fi
artifact_paths:
- "artifacts/rocm-base-wheels/*.whl"
env:
DOCKER_BUILDKIT: "1"
S3_BUCKET: "vllm-wheels"
# ROCm Job 2: Build vLLM ROCm Wheel
- label: ":python: Build vLLM ROCm Wheel - x86_64"
id: build-rocm-vllm-wheel
depends_on:
- step: build-rocm-base-wheels
allow_failure: false
agents:
queue: cpu_queue_release
timeout_in_minutes: 180
commands:
# Download artifacts and prepare Docker image
- |
set -euo pipefail
# Ensure git tags are up-to-date (Buildkite's default fetch doesn't update tags)
# This fixes version detection when tags are moved/force-pushed
echo "Fetching latest tags from origin..."
git fetch --tags --force origin
# Log tag information for debugging version detection
echo "========================================"
echo "Git Tag Verification"
echo "========================================"
echo "Current HEAD: $(git rev-parse HEAD)"
echo "git describe --tags: $(git describe --tags 2>/dev/null || echo 'No tags found')"
echo ""
echo "Recent tags (pointing to commits near HEAD):"
git tag -l --sort=-creatordate | head -5
echo "setuptools_scm version detection:"
pip install -q setuptools_scm 2>/dev/null || true
python3 -c "import setuptools_scm; print(' Detected version:', setuptools_scm.get_version())" 2>/dev/null || echo " (setuptools_scm not available in this environment)"
echo "========================================"
# Download wheel artifacts from current build
echo "Downloading wheel artifacts from current build"
buildkite-agent artifact download "artifacts/rocm-base-wheels/*.whl" .
# Get ECR image tag from metadata (set by build-rocm-base-wheels)
ECR_IMAGE_TAG="$$(buildkite-agent meta-data get rocm-base-image-tag 2>/dev/null || echo '')"
if [ -z "$${ECR_IMAGE_TAG}" ]; then
echo "ERROR: rocm-base-image-tag metadata not found"
echo "This should have been set by the build-rocm-base-wheels job"
exit 1
fi
echo "Pulling base Docker image from ECR: $${ECR_IMAGE_TAG}"
# Login to ECR
aws ecr-public get-login-password --region us-east-1 | \
docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7
# Pull base Docker image from ECR
docker pull "$${ECR_IMAGE_TAG}"
echo "Loaded base image: $${ECR_IMAGE_TAG}"
# Prepare base wheels for Docker build context
mkdir -p docker/context/base-wheels
touch docker/context/base-wheels/.keep
cp artifacts/rocm-base-wheels/*.whl docker/context/base-wheels/
echo "Base wheels for vLLM build:"
ls -lh docker/context/base-wheels/
echo "========================================"
echo "Building vLLM wheel with:"
echo " BUILDKITE_COMMIT: $${BUILDKITE_COMMIT}"
echo " BUILDKITE_BRANCH: $${BUILDKITE_BRANCH}"
echo " BASE_IMAGE: $${ECR_IMAGE_TAG}"
echo "========================================"
# Build vLLM wheel using local checkout (REMOTE_VLLM=0)
DOCKER_BUILDKIT=1 docker build \
--file docker/Dockerfile.rocm \
--target export_vllm_wheel_release \
--output type=local,dest=rocm-dist \
--build-arg BASE_IMAGE="$${ECR_IMAGE_TAG}" \
--build-arg REMOTE_VLLM=0 \
--build-arg GIT_REPO_CHECK=1 \
--build-arg USE_SCCACHE=1 \
--build-arg SCCACHE_BUCKET_NAME=vllm-build-sccache \
--build-arg SCCACHE_REGION_NAME=us-west-2 \
--build-arg SCCACHE_S3_NO_CREDENTIALS=0 \
.
echo "Built vLLM wheel:"
ls -lh rocm-dist/*.whl
# Copy wheel to artifacts directory
mkdir -p artifacts/rocm-vllm-wheel
cp rocm-dist/*.whl artifacts/rocm-vllm-wheel/
echo "Final vLLM wheel:"
ls -lh artifacts/rocm-vllm-wheel/
artifact_paths:
- "artifacts/rocm-vllm-wheel/*.whl"
env:
DOCKER_BUILDKIT: "1"
S3_BUCKET: "vllm-wheels"
# ROCm Job 3: Upload Wheels to S3
- label: ":s3: Upload ROCm Wheels to S3"
id: upload-rocm-wheels
depends_on:
- step: build-rocm-vllm-wheel
allow_failure: false
agents:
queue: cpu_queue_release
timeout_in_minutes: 60
commands:
# Download all wheel artifacts and run upload
- |
set -euo pipefail
# Download artifacts from current build
echo "Downloading artifacts from current build"
buildkite-agent artifact download "artifacts/rocm-base-wheels/*.whl" .
buildkite-agent artifact download "artifacts/rocm-vllm-wheel/*.whl" .
# # Run upload script
bash .buildkite/scripts/upload-rocm-wheels.sh
env:
DOCKER_BUILDKIT: "1"
S3_BUCKET: "vllm-wheels"
# ROCm Job 4: Annotate ROCm Wheel Release
- label: ":memo: Annotate ROCm wheel release"
id: annotate-rocm-release
depends_on:
- upload-rocm-wheels
agents:
queue: cpu_queue_release
commands:
- "bash .buildkite/scripts/annotate-rocm-release.sh"
env:
S3_BUCKET: "vllm-wheels"
# ROCm Job 5: Generate Root Index for ROCm Wheels (for release only)
# This is the job to create https://wheels.vllm.ai/rocm/ index allowing
# users to install with `uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/`
- block: "Generate Root Index for ROCm Wheels for Release"
key: block-generate-root-index-rocm-wheels
depends_on: upload-rocm-wheels
- label: ":package: Generate Root Index for ROCm Wheels for Release"
depends_on: block-generate-root-index-rocm-wheels
id: generate-root-index-rocm-wheels
agents:
queue: cpu_queue_release
commands:
- "bash tools/vllm-rocm/generate-rocm-wheels-root-index.sh"
env:
S3_BUCKET: "vllm-wheels"
VARIANT: "rocm723"
# ROCm Job 6: Build ROCm Release Docker Image
- label: ":docker: Build release image - x86_64 - ROCm"
id: build-rocm-release-image
depends_on:
- step: block-build-release-images
allow_failure: true
- step: build-rocm-base-wheels
allow_failure: false
agents:
queue: cpu_queue_release
timeout_in_minutes: 60
commands:
- |
set -euo pipefail
# Login to ECR
aws ecr-public get-login-password --region us-east-1 | \
docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7
# Get ECR image tag from metadata (set by build-rocm-base-wheels)
ECR_IMAGE_TAG="$$(buildkite-agent meta-data get rocm-base-image-tag 2>/dev/null || echo '')"
if [ -z "$${ECR_IMAGE_TAG}" ]; then
echo "ERROR: rocm-base-image-tag metadata not found"
echo "This should have been set by the build-rocm-base-wheels job"
exit 1
fi
echo "Pulling base Docker image from ECR: $${ECR_IMAGE_TAG}"
# Pull base Docker image from ECR
docker pull "$${ECR_IMAGE_TAG}"
echo "Loaded base image: $${ECR_IMAGE_TAG}"
# Pass the base image ECR tag to downstream steps (nightly publish)
buildkite-agent meta-data set "rocm-base-ecr-tag" "$${ECR_IMAGE_TAG}"
echo "========================================"
echo "Building vLLM ROCm release image with:"
echo " BASE_IMAGE: $${ECR_IMAGE_TAG}"
echo " BUILDKITE_COMMIT: $${BUILDKITE_COMMIT}"
echo "========================================"
# Build vLLM ROCm release image using cached base
DOCKER_BUILDKIT=1 docker build \
--build-arg max_jobs=16 \
--build-arg BASE_IMAGE="$${ECR_IMAGE_TAG}" \
--build-arg USE_SCCACHE=1 \
--build-arg SCCACHE_BUCKET_NAME=vllm-build-sccache \
--build-arg SCCACHE_REGION_NAME=us-west-2 \
--build-arg SCCACHE_S3_NO_CREDENTIALS=0 \
--build-arg INSTALL_LMCACHE=true \
--tag public.ecr.aws/q9t5s3a7/vllm-release-repo:$${BUILDKITE_COMMIT}-rocm \
--target vllm-openai \
--progress plain \
-f docker/Dockerfile.rocm .
# Push to ECR
docker push public.ecr.aws/q9t5s3a7/vllm-release-repo:$${BUILDKITE_COMMIT}-rocm
echo ""
echo " Successfully built and pushed ROCm release image"
echo " Image: public.ecr.aws/q9t5s3a7/vllm-release-repo:$${BUILDKITE_COMMIT}-rocm"
echo ""
env:
DOCKER_BUILDKIT: "1"
S3_BUCKET: "vllm-wheels"
- label: "Publish nightly XPU image to DockerHub"
depends_on:
- create-manifest-xpu
if: build.env("NIGHTLY") == "1"
agents:
queue: small_cpu_queue_release
commands:
- "bash .buildkite/scripts/xpu/push-nightly-builds-xpu.sh"
- "bash .buildkite/scripts/cleanup-nightly-builds.sh nightly- vllm/vllm-openai-xpu"
plugins:
- docker-login#v3.0.0:
username: vllmbot
password-env: DOCKERHUB_TOKEN
env:
DOCKER_BUILDKIT: "1"
DOCKERHUB_USERNAME: "vllmbot"
- label: "Publish nightly ROCm image to DockerHub"
depends_on:
- build-rocm-release-image
if: build.env("NIGHTLY") == "1"
agents:
queue: small_cpu_queue_release
commands:
- "bash .buildkite/scripts/push-nightly-builds-rocm.sh"
# Clean up old nightly builds (keep only last 14)
- "bash .buildkite/scripts/cleanup-nightly-builds.sh nightly- vllm/vllm-openai-rocm"
- "bash .buildkite/scripts/cleanup-nightly-builds.sh base-nightly- vllm/vllm-openai-rocm"
plugins:
- docker-login#v3.0.0:
username: vllmbot
password-env: DOCKERHUB_TOKEN
env:
DOCKER_BUILDKIT: "1"
DOCKERHUB_USERNAME: "vllmbot"
- trigger: "perf-eval"
label: "Trigger perf-eval (CUDA)"
key: trigger-perf-eval-cuda
@@ -724,7 +1103,7 @@ steps:
key: trigger-perf-eval-rocm
if: build.env("NIGHTLY") == "1" || build.env("RUN_PERF_EVAL") == "1"
depends_on:
- build-rocm100-release-image
- build-rocm-release-image
async: true
soft_fail: true
build:
@@ -733,250 +1112,138 @@ steps:
env:
NIGHTLY: 1 # run all perf-eval workloads
VLLM_COMMIT: "$BUILDKITE_COMMIT"
VLLM_IMAGE_ROCM: "public.ecr.aws/q9t5s3a7/vllm-release-repo:$BUILDKITE_COMMIT-rocm100"
VLLM_IMAGE_ROCM: "public.ecr.aws/q9t5s3a7/vllm-release-repo:$BUILDKITE_COMMIT-rocm"
# =============================================================================
# ROCm Release Pipelines (x86_64 only)
# ROCk Release Pipeline (x86_64 only)
# =============================================================================
#
# One group per ROCm stack, identified by its base Dockerfile (ROCM_BASE_DOCKERFILE).
# The scripts derive the ROCm version, wheel variant, cache keys and image tags from
# it (.buildkite/scripts/rocm/stack.sh), so both groups share one set of step
# definitions, anchored on the ROCm 10.0 group.
# Same shape as the ROCm pipeline above (Jobs 1 and 6 only), but built from
# docker/Dockerfile.rock_base and docker/Dockerfile.rock, which install the
# ROCm SDK from TheRock wheels instead of the ROCm apt repos.
#
# - ROCm 10.0 (TheRock, docker/Dockerfile.rocm{,_base}) is the default: it owns the
# :nightly / :latest images and the root wheel index.
# - ROCm 7.2 (docker/Dockerfile.rocm_72{,_base}) is kept for a transition period as
# rocm72 images and wheels.
# Note: ROCm SDK version is determined by ROCM_SDK_VERSION in
# docker/Dockerfile.rock_base
#
# vLLM version is determined by the Buildkite checkout (like the CUDA pipeline).
# =============================================================================
- group: "Build ROCm 10.0 Wheel / Image"
key: "build-rocm100"
- group: "Build ROCk Image"
key: "build-rock-image"
depends_on: ~
steps:
- label: ":rocm: Build ROCm 10.0 Base Image & Wheels"
id: build-rocm100-base
# ROCk Job 1: Build ROCk Base Image (with ECR caching)
- label: ":rocm: Build ROCk Base Image"
id: build-rock-base-image
depends_on: ~
agents: &rocm_build_agents
agents:
queue: cpu_queue_release
timeout_in_minutes: 360
env: &rocm100_env
ROCM_BASE_DOCKERFILE: "docker/Dockerfile.rocm_base"
ROCM_DEFAULT_ARG: "--default"
DOCKER_BUILDKIT: "1"
S3_BUCKET: "vllm-wheels"
DOCKERHUB_USERNAME: "vllmbot"
commands: &rocm_base_commands
- |
set -euo pipefail
source .buildkite/scripts/rocm/stack.sh "$${ROCM_BASE_DOCKERFILE}"
SCCACHE_ARGS=(--build-arg USE_SCCACHE=1 --build-arg SCCACHE_BUCKET_NAME=vllm-build-sccache
--build-arg SCCACHE_REGION_NAME=us-west-2 --build-arg SCCACHE_S3_NO_CREDENTIALS=0)
aws ecr-public get-login-password --region us-east-1 | \
docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7
# Base image, cached in ECR by the base Dockerfile's hash
if docker manifest inspect "$${ROCM_STACK_ECR_BASE}" > /dev/null 2>&1; then
echo "ECR image cache HIT: $${ROCM_STACK_ECR_BASE}"
else
docker buildx build --file "$${ROCM_BASE_DOCKERFILE}" --tag "$${ROCM_STACK_ECR_BASE}" \
"$${SCCACHE_ARGS[@]}" --push .
fi
# Base wheels, cached in S3 (see cache-rocm-base-wheels.sh)
CACHE=.buildkite/scripts/cache-rocm-base-wheels.sh
if [ "$$($${CACHE} "$${ROCM_BASE_DOCKERFILE}" check)" = "hit" ]; then
$${CACHE} "$${ROCM_BASE_DOCKERFILE}" download
else
DEBS="rocm-base-debs:$${BUILDKITE_BUILD_NUMBER}-$${ROCM_STACK_VARIANT}"
docker buildx build --file "$${ROCM_BASE_DOCKERFILE}" --tag "$${DEBS}" \
--target debs_wheel_release "$${SCCACHE_ARGS[@]}" --load .
mkdir -p "$${ROCM_STACK_BASE_WHEELS_DIR}"
cid=$$(docker create "$${DEBS}")
docker cp "$${cid}":/app/debs/. "$${ROCM_STACK_BASE_WHEELS_DIR}/"
docker rm "$${cid}"
$${CACHE} "$${ROCM_BASE_DOCKERFILE}" upload
fi
buildkite-agent artifact upload "$${ROCM_STACK_BASE_WHEELS_DIR}/*.whl"
- label: ":python: Build vLLM ROCm 10.0 Wheel - x86_64"
id: build-rocm100-vllm-wheel
depends_on: build-rocm100-base
agents: *rocm_build_agents
timeout_in_minutes: 180
env: *rocm100_env
commands: &rocm_vllm_wheel_commands
- |
set -euo pipefail
source .buildkite/scripts/rocm/stack.sh "$${ROCM_BASE_DOCKERFILE}"
# Buildkite's default fetch doesn't update moved tags; version detection needs them
git fetch --tags --force origin
buildkite-agent artifact download "$${ROCM_STACK_BASE_WHEELS_DIR}/*.whl" .
aws ecr-public get-login-password --region us-east-1 | \
docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7
docker pull "$${ROCM_STACK_ECR_BASE}"
mkdir -p docker/context/base-wheels
touch docker/context/base-wheels/.keep
cp "$${ROCM_STACK_BASE_WHEELS_DIR}"/*.whl docker/context/base-wheels/
docker build --file "$${ROCM_STACK_DOCKERFILE}" --target export_vllm_wheel_release \
--output type=local,dest=rocm-dist \
--build-arg BASE_IMAGE="$${ROCM_STACK_ECR_BASE}" --build-arg REMOTE_VLLM=0 \
--build-arg GIT_REPO_CHECK=1 --build-arg USE_SCCACHE=1 \
--build-arg SCCACHE_BUCKET_NAME=vllm-build-sccache \
--build-arg SCCACHE_REGION_NAME=us-west-2 --build-arg SCCACHE_S3_NO_CREDENTIALS=0 .
mkdir -p "$${ROCM_STACK_VLLM_WHEEL_DIR}"
cp rocm-dist/*.whl "$${ROCM_STACK_VLLM_WHEEL_DIR}/"
buildkite-agent artifact upload "$${ROCM_STACK_VLLM_WHEEL_DIR}/*.whl"
- label: ":s3: Upload ROCm 10.0 Wheels to S3"
id: upload-rocm100-wheels
depends_on: build-rocm100-vllm-wheel
agents: *rocm_build_agents
timeout_in_minutes: 60
env: *rocm100_env
commands: &rocm_upload_commands
- |
set -euo pipefail
source .buildkite/scripts/rocm/stack.sh "$${ROCM_BASE_DOCKERFILE}"
buildkite-agent artifact download "$${ROCM_STACK_BASE_WHEELS_DIR}/*.whl" .
buildkite-agent artifact download "$${ROCM_STACK_VLLM_WHEEL_DIR}/*.whl" .
bash .buildkite/scripts/upload-rocm-wheels.sh "$${ROCM_BASE_DOCKERFILE}"
- label: ":memo: Annotate ROCm 10.0 wheel release"
id: annotate-rocm100-release
depends_on: upload-rocm100-wheels
agents: *rocm_build_agents
env: *rocm100_env
commands: &rocm_annotate_commands
- bash .buildkite/scripts/annotate-rocm-release.sh "$${ROCM_BASE_DOCKERFILE}" $${ROCM_DEFAULT_ARG}
# Generate https://wheels.vllm.ai/rocm/ (the default stack's variant), so users can
# `uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/` (release only)
- block: "Generate Root Index for ROCm Wheels for Release"
key: block-generate-root-index-rocm-wheels
depends_on:
- upload-rocm100-wheels
- upload-rocm72-wheels
- label: ":package: Generate Root Index for ROCm Wheels for Release"
depends_on: block-generate-root-index-rocm-wheels
id: generate-root-index-rocm-wheels
agents: *rocm_build_agents
env: *rocm100_env
commands:
- "bash tools/vllm-rocm/generate-rocm-wheels-root-index.sh"
- label: ":docker: Build release image - x86_64 - ROCm 10.0"
id: build-rocm100-release-image
depends_on:
- step: block-build-release-images
allow_failure: true
- step: build-rocm100-base
allow_failure: false
agents: *rocm_build_agents
timeout_in_minutes: 180
env: *rocm100_env
commands: &rocm_release_image_commands
- |
set -euo pipefail
source .buildkite/scripts/rocm/stack.sh "$${ROCM_BASE_DOCKERFILE}"
# The SDK version, torch wheels and component branches all live in
# the Dockerfile, so its hash is the cache key.
CACHE_KEY=$$(sha256sum docker/Dockerfile.rock_base | cut -c1-16)
ECR_CACHE_TAG="public.ecr.aws/q9t5s3a7/vllm-release-repo:$${CACHE_KEY}-rock-base"
echo "========================================"
echo "ROCk Base Build Configuration"
echo "========================================"
echo " CACHE_KEY: $${CACHE_KEY}"
echo " ECR_CACHE_TAG: $${ECR_CACHE_TAG}"
echo "========================================"
# Login to ECR
aws ecr-public get-login-password --region us-east-1 | \
docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7
docker pull "$${ROCM_STACK_ECR_BASE}"
docker build --file "$${ROCM_STACK_DOCKERFILE}" --target vllm-openai \
--tag "$${ROCM_STACK_ECR_IMAGE}" --progress plain \
--build-arg max_jobs=16 --build-arg BASE_IMAGE="$${ROCM_STACK_ECR_BASE}" \
--build-arg INSTALL_LMCACHE=true --build-arg USE_SCCACHE=1 \
--build-arg SCCACHE_BUCKET_NAME=vllm-build-sccache \
--build-arg SCCACHE_REGION_NAME=us-west-2 --build-arg SCCACHE_S3_NO_CREDENTIALS=0 .
docker push "$${ROCM_STACK_ECR_IMAGE}"
- label: "Publish nightly ROCm 10.0 image to DockerHub"
id: publish-rocm100-nightly
depends_on: build-rocm100-release-image
if: build.env("NIGHTLY") == "1"
agents: &rocm_publish_agents
queue: small_cpu_queue_release
env: *rocm100_env
commands: &rocm_publish_nightly_commands
- bash .buildkite/scripts/push-nightly-builds-rocm.sh "$${ROCM_BASE_DOCKERFILE}" $${ROCM_DEFAULT_ARG}
plugins: &rocm_dockerhub_login
- docker-login#v3.0.0:
username: vllmbot
password-env: DOCKERHUB_TOKEN
if docker manifest inspect "$${ECR_CACHE_TAG}" > /dev/null 2>&1; then
echo "ECR image cache HIT"
else
echo "CACHE MISS - Building from scratch..."
- group: "Build ROCm 7.2 Wheel / Image"
key: "build-rocm72"
depends_on: ~
steps:
- label: ":rocm: Build ROCm 7.2 Base Image & Wheels"
id: build-rocm72-base
depends_on: ~
agents: *rocm_build_agents
timeout_in_minutes: 360
env: &rocm72_env
ROCM_BASE_DOCKERFILE: "docker/Dockerfile.rocm_72_base"
ROCM_DEFAULT_ARG: ""
DOCKER_BUILDKIT=1 docker buildx build \
--file docker/Dockerfile.rock_base \
--tag "$${ECR_CACHE_TAG}" \
--build-arg USE_SCCACHE=1 \
--build-arg SCCACHE_BUCKET_NAME=vllm-build-sccache \
--build-arg SCCACHE_REGION_NAME=us-west-2 \
--build-arg SCCACHE_S3_NO_CREDENTIALS=0 \
--push \
.
fi
# Save ECR tag for downstream jobs
buildkite-agent meta-data set "rock-base-image-tag" "$${ECR_CACHE_TAG}"
env:
DOCKER_BUILDKIT: "1"
S3_BUCKET: "vllm-wheels"
DOCKERHUB_USERNAME: "vllmbot"
commands: *rocm_base_commands
- label: ":python: Build vLLM ROCm 7.2 Wheel - x86_64"
id: build-rocm72-vllm-wheel
depends_on: build-rocm72-base
agents: *rocm_build_agents
timeout_in_minutes: 180
env: *rocm72_env
commands: *rocm_vllm_wheel_commands
- label: ":s3: Upload ROCm 7.2 Wheels to S3"
id: upload-rocm72-wheels
depends_on: build-rocm72-vllm-wheel
agents: *rocm_build_agents
timeout_in_minutes: 60
env: *rocm72_env
commands: *rocm_upload_commands
- label: ":memo: Annotate ROCm 7.2 wheel release"
id: annotate-rocm72-release
depends_on: upload-rocm72-wheels
agents: *rocm_build_agents
env: *rocm72_env
commands: *rocm_annotate_commands
- label: ":docker: Build release image - x86_64 - ROCm 7.2"
id: build-rocm72-release-image
# ROCk Job 2: Build ROCk Release Docker Image
- label: ":docker: Build release image - x86_64 - ROCk"
id: build-rock-release-image
depends_on:
- step: block-build-release-images
allow_failure: true
- step: build-rocm72-base
- step: build-rock-base-image
allow_failure: false
agents: *rocm_build_agents
agents:
queue: cpu_queue_release
timeout_in_minutes: 180
env: *rocm72_env
commands: *rocm_release_image_commands
commands:
- |
set -euo pipefail
- label: "Publish nightly ROCm 7.2 image to DockerHub"
id: publish-rocm72-nightly
depends_on: build-rocm72-release-image
if: build.env("NIGHTLY") == "1"
agents: *rocm_publish_agents
env: *rocm72_env
commands: *rocm_publish_nightly_commands
plugins: *rocm_dockerhub_login
# Login to ECR
aws ecr-public get-login-password --region us-east-1 | \
docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7
- label: "Publish nightly XPU image to DockerHub"
ECR_IMAGE_TAG="$$(buildkite-agent meta-data get rock-base-image-tag 2>/dev/null || echo '')"
if [ -z "$${ECR_IMAGE_TAG}" ]; then
echo "ERROR: rock-base-image-tag metadata not found"
echo "This should have been set by the build-rock-base-image job"
exit 1
fi
docker pull "$${ECR_IMAGE_TAG}"
# Pass the base image ECR tag to downstream steps (nightly publish)
buildkite-agent meta-data set "rock-base-ecr-tag" "$${ECR_IMAGE_TAG}"
echo "========================================"
echo "Building vLLM ROCk release image with:"
echo " BASE_IMAGE: $${ECR_IMAGE_TAG}"
echo " BUILDKITE_COMMIT: $${BUILDKITE_COMMIT}"
echo "========================================"
DOCKER_BUILDKIT=1 docker build \
--build-arg max_jobs=16 \
--build-arg BASE_IMAGE="$${ECR_IMAGE_TAG}" \
--build-arg USE_SCCACHE=1 \
--build-arg SCCACHE_BUCKET_NAME=vllm-build-sccache \
--build-arg SCCACHE_REGION_NAME=us-west-2 \
--build-arg SCCACHE_S3_NO_CREDENTIALS=0 \
--build-arg INSTALL_LMCACHE=true \
--tag public.ecr.aws/q9t5s3a7/vllm-release-repo:$${BUILDKITE_COMMIT}-rock \
--target vllm-openai \
--progress plain \
-f docker/Dockerfile.rock .
docker push public.ecr.aws/q9t5s3a7/vllm-release-repo:$${BUILDKITE_COMMIT}-rock
env:
DOCKER_BUILDKIT: "1"
- label: "Publish nightly ROCk image to DockerHub"
depends_on:
- create-manifest-xpu
- build-rock-release-image
if: build.env("NIGHTLY") == "1"
agents:
queue: small_cpu_queue_release
commands:
- "bash .buildkite/scripts/xpu/push-nightly-builds-xpu.sh"
- "bash .buildkite/scripts/cleanup-nightly-builds.sh nightly- vllm/vllm-openai-xpu"
- "bash .buildkite/scripts/push-nightly-builds-rocm.sh rock rocm100"
# Clean up old nightly builds (keep only last 14)
- "bash .buildkite/scripts/cleanup-nightly-builds.sh nightly-rocm100- vllm/vllm-openai-rocm"
- "bash .buildkite/scripts/cleanup-nightly-builds.sh base-nightly-rocm100- vllm/vllm-openai-rocm"
plugins:
- docker-login#v3.0.0:
username: vllmbot
@@ -997,8 +1264,7 @@ steps:
- create-multi-arch-manifest-ubuntu2404
- create-multi-arch-manifest-cuda-12-9-ubuntu2404
- create-manifest-xpu
- build-rocm100-release-image
- build-rocm72-release-image
- build-rocm-release-image
- input-release-version
# Wait for CPU builds if their block steps were unblocked, so publish
# doesn't race the in-progress CPU build. allow_failure lets publish
@@ -1023,7 +1289,6 @@ steps:
- "cuda-13-0-ubuntu-24-04"
- "cuda-12-9-ubuntu-24-04"
- "rocm"
- "rocm72"
- "xpu"
- "cpu"
plugins:
+66 -40
View File
@@ -2,23 +2,15 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
#
# Generate Buildkite annotation for a ROCm stack's wheel release.
#
# Usage: annotate-rocm-release.sh <base-dockerfile> [--default]
# base-dockerfile e.g. docker/Dockerfile.rocm_base or docker/Dockerfile.rocm_72_base
# --default the stack also owns the plain :latest / :v<ver> Docker Hub tags
# Generate Buildkite annotation for ROCm wheel release
set -ex
# shellcheck source=.buildkite/scripts/rocm/stack.sh
source "$(dirname "${BASH_SOURCE[0]}")/rocm/stack.sh" "${1:?Usage: $0 <base-dockerfile> [--default]}"
TAG_VARIANTS=("-${ROCM_STACK_VARIANT}")
if [[ "${2:-}" == "--default" ]]; then
TAG_VARIANTS=("" "-${ROCM_STACK_VARIANT}")
fi
ROCM_VERSION="$ROCM_STACK_VERSION"
VARIANT="$ROCM_STACK_VARIANT"
PYTHON_VERSION=$(sed -nE 's/^ARG PYTHON_VERSION="?([^" ]+)"?.*/\1/p' "$ROCM_STACK_BASE_DOCKERFILE")
PYTORCH_ROCM_ARCH=$(sed -nE 's/^ARG PYTORCH_ROCM_ARCH="?([^"]+)"?.*/\1/p' "$ROCM_STACK_BASE_DOCKERFILE")
# Extract build configuration from Dockerfile.rocm_base (single source of truth)
# Extract ROCm version dynamically from Dockerfile.rocm_base
# BASE_IMAGE format: rocm/dev-ubuntu-22.04:7.0-complete -> extracts "7.0"
ROCM_VERSION=$(grep -E '^ARG BASE_IMAGE=' docker/Dockerfile.rocm_base | sed -E 's/.*:([0-9]+\.[0-9]+).*/\1/' || echo "unknown")
PYTHON_VERSION=$(grep '^ARG PYTHON_VERSION=' docker/Dockerfile.rocm_base | sed 's/^ARG PYTHON_VERSION=//')
PYTORCH_ROCM_ARCH=$(grep '^ARG PYTORCH_ROCM_ARCH=' docker/Dockerfile.rocm_base | sed 's/^ARG PYTORCH_ROCM_ARCH=//')
# Get release version, default to 1.0.0.dev for nightly/per-commit builds
RELEASE_VERSION=$(buildkite-agent meta-data get release-version 2>/dev/null || echo "")
@@ -26,32 +18,22 @@ if [ -z "${RELEASE_VERSION}" ]; then
RELEASE_VERSION="1.0.0.dev"
fi
ROCM_BASE_CACHE_KEY=$(.buildkite/scripts/cache-rocm-base-wheels.sh key)
# S3 URLs
S3_BUCKET="${S3_BUCKET:-vllm-wheels}"
S3_REGION="${AWS_DEFAULT_REGION:-us-west-2}"
S3_HOST="${S3_BUCKET}.s3-website-${S3_REGION}.amazonaws.com"
S3_URL="http://${S3_HOST}"
WHEEL_PATH="rocm/${BUILDKITE_COMMIT}/${VARIANT}-wheels/"
S3_URL="http://${S3_BUCKET}.s3-website-${S3_REGION}.amazonaws.com"
DOCKER_CMDS="docker pull ${ROCM_STACK_ECR_BASE}
docker pull ${ROCM_STACK_ECR_IMAGE}"
for suffix in "${TAG_VARIANTS[@]}"; do
for tag in "latest${suffix}" "v${RELEASE_VERSION}${suffix}"; do
DOCKER_CMDS+="
docker tag ${ROCM_STACK_ECR_BASE} vllm/vllm-openai-rocm:${tag}-base
docker tag ${ROCM_STACK_ECR_IMAGE} vllm/vllm-openai-rocm:${tag}
docker push vllm/vllm-openai-rocm:${tag}-base
docker push vllm/vllm-openai-rocm:${tag}"
done
done
buildkite-agent annotate --style 'success' --context "rocm-release-workflow-${VARIANT}" << EOF
## ROCm ${ROCM_VERSION} Wheel and Docker Image Releases
# Format ROCm version for path (e.g., "7.1" -> "rocm710")
ROCM_VERSION_PATH="rocm$(echo "${ROCM_VERSION}" | tr -d '.')"
ROCM_PATH="rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}"
buildkite-agent annotate --style 'success' --context 'rocm-release-workflow' << EOF
## ROCm Wheel and Docker Image Releases
### Build Configuration
| Setting | Value |
|---------|-------|
| **ROCm Version** | ${ROCM_VERSION} |
| **Index Variant** | ${VARIANT} |
| **Python Version** | ${PYTHON_VERSION} |
| **GPU Architectures** | ${PYTORCH_ROCM_ARCH} |
| **Branch** | \`${BUILDKITE_BRANCH}\` |
@@ -59,20 +41,49 @@ buildkite-agent annotate --style 'success' --context "rocm-release-workflow-${VA
### :package: Installation
**Install from this build (by commit):**
\`\`\`bash
# This build (by commit)
pip install vllm --extra-index-url ${S3_URL}/rocm/${BUILDKITE_COMMIT}/${VARIANT}/ --trusted-host ${S3_HOST}
# Nightly (if published)
pip install vllm --extra-index-url ${S3_URL}/rocm/nightly/${VARIANT}/ --trusted-host ${S3_HOST}
pip install vllm --extra-index-url ${S3_URL}/${ROCM_PATH}/ --trusted-host ${S3_BUCKET}.s3-website-${S3_REGION}.amazonaws.com
# Example for ROCm ${ROCM_VERSION}:
pip install vllm --extra-index-url ${S3_URL}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/ --trusted-host ${S3_BUCKET}.s3-website-${S3_REGION}.amazonaws.com
\`\`\`
**Install from nightly (if published):**
\`\`\`bash
pip install vllm --extra-index-url ${S3_URL}/rocm/nightly/ --trusted-host ${S3_BUCKET}.s3-website-${S3_REGION}.amazonaws.com
\`\`\`
### :floppy_disk: Download Wheels Directly
\`\`\`bash
aws s3 ls s3://${S3_BUCKET}/${WHEEL_PATH}
aws s3 cp --recursive --exclude "*" --include "*.whl" s3://${S3_BUCKET}/${WHEEL_PATH} .
# List all ROCm wheels
aws s3 ls s3://${S3_BUCKET}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/
# Download specific wheels
aws s3 cp s3://${S3_BUCKET}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/vllm-*.whl .
aws s3 cp s3://${S3_BUCKET}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/torch-*.whl .
aws s3 cp s3://${S3_BUCKET}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/triton-*.whl .
aws s3 cp s3://${S3_BUCKET}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/triton-kernels-*.whl .
aws s3 cp s3://${S3_BUCKET}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/torchvision-*.whl .
aws s3 cp s3://${S3_BUCKET}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/torchaudio-*.whl .
aws s3 cp s3://${S3_BUCKET}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/amdsmi-*.whl .
aws s3 cp s3://${S3_BUCKET}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/amd_aiter-*.whl .
aws s3 cp s3://${S3_BUCKET}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/flash-attn-*.whl .
\`\`\`
### :gear: Included Packages
- **vllm**: vLLM with ROCm support
- **torch**: PyTorch built for ROCm ${ROCM_VERSION}
- **triton**: Triton
- **triton-kernels**: Triton kernels
- **torchvision**: TorchVision for ROCm PyTorch
- **torchaudio**: Torchaudio for ROCm PyTorch
- **amdsmi**: AMD SMI Python bindings
- **amd_aiter**: Aiter for ROCm
- **flash-attn**: Flash Attention for ROCm
### :warning: Notes
- These wheels are built for **ROCm ${ROCM_VERSION}** and will NOT work with CUDA GPUs
- Supported GPU architectures: ${PYTORCH_ROCM_ARCH}
@@ -80,8 +91,23 @@ aws s3 cp --recursive --exclude "*" --include "*.whl" s3://${S3_BUCKET}/${WHEEL_
### :package: Docker Image Release
To download and upload the image:
\`\`\`
${DOCKER_CMDS}
docker pull public.ecr.aws/q9t5s3a7/vllm-release-repo:${BUILDKITE_COMMIT}-rocm-base
docker pull public.ecr.aws/q9t5s3a7/vllm-release-repo:${BUILDKITE_COMMIT}-rocm
docker tag public.ecr.aws/q9t5s3a7/vllm-release-repo:${ROCM_BASE_CACHE_KEY}-rocm-base vllm/vllm-openai-rocm:${BUILDKITE_COMMIT}-base
docker tag vllm/vllm-openai-rocm:${BUILDKITE_COMMIT}-base vllm/vllm-openai-rocm:latest-base
docker tag vllm/vllm-openai-rocm:${BUILDKITE_COMMIT}-base vllm/vllm-openai-rocm:v${RELEASE_VERSION}-base
docker push vllm/vllm-openai-rocm:latest-base
docker push vllm/vllm-openai-rocm:v${RELEASE_VERSION}-base
docker tag public.ecr.aws/q9t5s3a7/vllm-release-repo:${BUILDKITE_COMMIT}-rocm vllm/vllm-openai-rocm:${BUILDKITE_COMMIT}
docker tag vllm/vllm-openai-rocm:${BUILDKITE_COMMIT} vllm/vllm-openai-rocm:latest
docker tag vllm/vllm-openai-rocm:${BUILDKITE_COMMIT} vllm/vllm-openai-rocm:v${RELEASE_VERSION}
docker push vllm/vllm-openai-rocm:latest
docker push vllm/vllm-openai-rocm:v${RELEASE_VERSION}
\`\`\`
EOF
+34 -28
View File
@@ -5,33 +5,39 @@
# Cache helper for ROCm base wheels
#
# This script manages caching of pre-built ROCm base wheels (torch, triton, etc.)
# to avoid rebuilding them when the stack's base Dockerfile hasn't changed.
# to avoid rebuilding them when Dockerfile.rocm_base hasn't changed.
#
# Usage: cache-rocm-base-wheels.sh <base-dockerfile> <command>
# base-dockerfile e.g. docker/Dockerfile.rocm_base or docker/Dockerfile.rocm_72_base
# Commands:
# check - Check if cache exists, outputs "hit" or "miss"
# upload - Upload wheels to cache
# download - Download wheels from cache
# key - Output the cache key
# path - Output the full S3 cache path
# Usage:
# cache-rocm-base-wheels.sh check - Check if cache exists, outputs "hit" or "miss"
# cache-rocm-base-wheels.sh upload - Upload wheels to cache
# cache-rocm-base-wheels.sh download - Download wheels from cache
# cache-rocm-base-wheels.sh key - Output the cache key
#
# Environment variables:
# S3_BUCKET - S3 bucket name (default: vllm-wheels)
#
# The cache key hashes the base Dockerfile and the repo files it COPYs in, which
# capture the ROCm version and every component pin (see rocm/stack.sh).
# Note: ROCm version is determined by BASE_IMAGE in Dockerfile.rocm_base,
# so changes to ROCm version are captured by the Dockerfile hash.
set -euo pipefail
BUCKET="${S3_BUCKET:-vllm-wheels}"
DOCKERFILE="docker/Dockerfile.rocm_base"
CACHE_PREFIX="rocm/cache"
# shellcheck source=.buildkite/scripts/rocm/stack.sh
source "$(dirname "${BASH_SOURCE[0]}")/rocm/stack.sh" "${1:?Usage: $0 <base-dockerfile> <command>}"
shift
CACHE_KEY="$ROCM_STACK_WHEEL_KEY"
WHEELS_DIR="$ROCM_STACK_BASE_WHEELS_DIR"
# Generate hash from Dockerfile content + build args
generate_cache_key() {
# Include Dockerfile content
if [[ ! -f "$DOCKERFILE" ]]; then
echo "ERROR: Dockerfile not found: $DOCKERFILE" >&2
exit 1
fi
local dockerfile_hash=$(sha256sum "$DOCKERFILE" | cut -c1-16)
echo "${dockerfile_hash}"
}
CACHE_KEY=$(generate_cache_key)
CACHE_PATH="s3://${BUCKET}/${CACHE_PREFIX}/${CACHE_KEY}/"
case "${1:-}" in
@@ -61,19 +67,19 @@ case "${1:-}" in
echo "Cache path: ${CACHE_PATH}"
echo ""
if [[ ! -d "${WHEELS_DIR}" ]]; then
echo "ERROR: ${WHEELS_DIR} directory not found" >&2
if [[ ! -d "artifacts/rocm-base-wheels" ]]; then
echo "ERROR: artifacts/rocm-base-wheels directory not found" >&2
exit 1
fi
WHEEL_COUNT=$(find "${WHEELS_DIR}" -maxdepth 1 -name '*.whl' 2>/dev/null | wc -l)
WHEEL_COUNT=$(find artifacts/rocm-base-wheels -maxdepth 1 -name '*.whl' 2>/dev/null | wc -l)
if [[ "$WHEEL_COUNT" -eq 0 ]]; then
echo "ERROR: No wheels found in ${WHEELS_DIR}/" >&2
echo "ERROR: No wheels found in artifacts/rocm-base-wheels/" >&2
exit 1
fi
echo "Uploading $WHEEL_COUNT wheels..."
aws s3 cp --recursive "${WHEELS_DIR}/" "${CACHE_PATH}"
aws s3 cp --recursive artifacts/rocm-base-wheels/ "${CACHE_PATH}"
echo ""
echo "Cache upload complete!"
@@ -87,17 +93,17 @@ case "${1:-}" in
echo "Cache key: ${CACHE_KEY}"
echo "Cache path: ${CACHE_PATH}"
echo ""
mkdir -p "${WHEELS_DIR}"
mkdir -p artifacts/rocm-base-wheels
# Use sync with include/exclude to only download .whl files
aws s3 sync "${CACHE_PATH}" "${WHEELS_DIR}/" \
aws s3 sync "${CACHE_PATH}" artifacts/rocm-base-wheels/ \
--exclude "*" \
--include "*.whl"
echo ""
echo "Downloaded wheels:"
find "${WHEELS_DIR}" -maxdepth 1 -name '*.whl' -exec ls -lh {} \;
WHEEL_COUNT=$(find "${WHEELS_DIR}" -maxdepth 1 -name '*.whl' 2>/dev/null | wc -l)
find artifacts/rocm-base-wheels -maxdepth 1 -name '*.whl' -exec ls -lh {} \;
WHEEL_COUNT=$(find artifacts/rocm-base-wheels -maxdepth 1 -name '*.whl' 2>/dev/null | wc -l)
echo ""
echo "Total: $WHEEL_COUNT wheels"
echo "========================================"
@@ -112,12 +118,12 @@ case "${1:-}" in
;;
*)
echo "Usage: $0 <base-dockerfile> {check|upload|download|key|path}" >&2
echo "Usage: $0 {check|upload|download|key|path}" >&2
echo "" >&2
echo "Commands:" >&2
echo " check - Check if cache exists, outputs 'hit' or 'miss'" >&2
echo " upload - Upload wheels from ${WHEELS_DIR}/ to cache" >&2
echo " download - Download wheels from cache to ${WHEELS_DIR}/" >&2
echo " upload - Upload wheels from artifacts/rocm-base-wheels/ to cache" >&2
echo " download - Download wheels from cache to artifacts/rocm-base-wheels/" >&2
echo " key - Output the cache key" >&2
echo " path - Output the full S3 cache path" >&2
exit 1
+54 -7
View File
@@ -17,7 +17,7 @@ DEFAULT_REPO_SLUG="vllm-project/vllm"
DEFAULT_CI_HCL_SOURCE="docker/ci-rocm.hcl"
DEFAULT_CI_BASE_CONTENT_FILES=".dockerignore requirements/common.txt requirements/rocm.txt requirements/test/rocm.txt tools/install_torchcodec_rocm.sh rust-toolchain.toml tests/vllm_test_utils"
DEFAULT_CI_BASE_DOCKERFILE="docker/Dockerfile.rocm"
DEFAULT_CI_BASE_DOCKERFILE_STAGES="base rust_toolchain_input_0 rust-toolchain-input rust-toolchain build_nixl lmcache_source build_lmcache build_deepep mori_base ci_base"
DEFAULT_CI_BASE_DOCKERFILE_STAGES="base rust_toolchain_input_0 rust-toolchain-input rust-toolchain build_nixl lmcache_source build_lmcache build_rocshmem build_deepep mori_base ci_base"
DEFAULT_CI_BASE_METADATA_VERSION="3"
# ROCm CI forces REMOTE_VLLM=0, so content identity covers only the selected
# local-source stages rather than unreachable remote-fetch alternatives.
@@ -555,6 +555,9 @@ configure_custom_rocm_stages() {
CI_BASE_DOCKERFILE_STAGES="${CI_BASE_DOCKERFILE_STAGES:-${stages}}"
ROCM_CSRC_DOCKERFILE_STAGES="${ROCM_CSRC_DOCKERFILE_STAGES:-${stages}}"
ROCM_RUST_DOCKERFILE_STAGES="${ROCM_RUST_DOCKERFILE_STAGES:-${stages}}"
if [[ " ${stages} " != *" build_rocshmem "* ]]; then
unset ROCSHMEM_BRANCH ROCSHMEM_CACHE_KEY DEEPEP_CACHE_KEY
fi
}
compose_dependency_cache_key() {
@@ -1721,11 +1724,14 @@ ci_base_metadata_pairs() {
metadata_pair "vllm.rocm.nixl_commit" "${NIXL_BRANCH:-$(resolve_dockerfile_arg_value "${dockerfile}" "NIXL_BRANCH")}"
metadata_pair "vllm.rocm.ucx_repo" "$(resolve_dockerfile_arg_value "${dockerfile}" "UCX_REPO")"
metadata_pair "vllm.rocm.ucx_commit" "${UCX_BRANCH:-$(resolve_dockerfile_arg_value "${dockerfile}" "UCX_BRANCH")}"
metadata_pair "vllm.rocm.rocshmem_repo" "$(resolve_dockerfile_arg_value "${dockerfile}" "ROCSHMEM_REPO")"
metadata_pair "vllm.rocm.rocshmem_commit" "${ROCSHMEM_BRANCH:-$(resolve_dockerfile_arg_value "${dockerfile}" "ROCSHMEM_BRANCH")}"
metadata_pair "vllm.rocm.deepep_repo" "$(resolve_dockerfile_arg_value "${dockerfile}" "DEEPEP_REPO")"
metadata_pair "vllm.rocm.deepep_commit" "${DEEPEP_BRANCH:-$(resolve_dockerfile_arg_value "${dockerfile}" "DEEPEP_BRANCH")}"
metadata_pair "vllm.rocm.deepep_nic" "$(resolve_dockerfile_arg_value "${dockerfile}" "DEEPEP_NIC")"
metadata_pair "vllm.rocm.deepep_rocm_arch" "$(resolve_dockerfile_arg_value "${dockerfile}" "DEEPEP_ROCM_ARCH")"
metadata_pair "vllm.rocm.nixl_cache_key" "${NIXL_CACHE_KEY:-}"
metadata_pair "vllm.rocm.rocshmem_cache_key" "${ROCSHMEM_CACHE_KEY:-}"
metadata_pair "vllm.rocm.deepep_cache_key" "${DEEPEP_CACHE_KEY:-}"
}
@@ -2308,7 +2314,7 @@ extract_dependency_pins() {
return 0
fi
for var in NIXL_BRANCH UCX_BRANCH DEEPEP_BRANCH; do
for var in NIXL_BRANCH UCX_BRANCH ROCSHMEM_BRANCH DEEPEP_BRANCH; do
if [[ -n "${!var:-}" ]]; then
echo "Using provided ${var}: ${!var}"
continue
@@ -2330,14 +2336,17 @@ compute_dependency_cache_keys() {
local dockerfile_rocm=""
local nixl_branch=""
local ucx_branch=""
local rocshmem_branch=""
local deepep_branch=""
local nixl_material=""
local rocshmem_material=""
local deepep_material=""
bake_dir=$(dirname "${VLLM_BAKE_FILE}")
dockerfile_rocm="${CI_BASE_DOCKERFILE:-${bake_dir}/Dockerfile.rocm}"
nixl_branch=$(resolve_dockerfile_arg_value "${dockerfile_rocm}" "NIXL_BRANCH")
ucx_branch=$(resolve_dockerfile_arg_value "${dockerfile_rocm}" "UCX_BRANCH")
rocshmem_branch=$(resolve_dockerfile_arg_value "${dockerfile_rocm}" "ROCSHMEM_BRANCH")
deepep_branch=$(resolve_dockerfile_arg_value "${dockerfile_rocm}" "DEEPEP_BRANCH")
if [[ -n "${nixl_branch}" && -n "${ucx_branch}" ]]; then
@@ -2351,11 +2360,22 @@ compute_dependency_cache_keys() {
echo "NIXL dependency cache key: ${NIXL_CACHE_KEY}"
fi
if [[ -n "${deepep_branch}" ]]; then
deepep_material=$(compose_stage_cache_material "${dockerfile_rocm}" "base build_deepep")
if [[ -n "${rocshmem_branch}" ]]; then
rocshmem_material=$(compose_stage_cache_material "${dockerfile_rocm}" "base build_rocshmem")
ROCSHMEM_CACHE_KEY=$(
compose_dependency_cache_key \
"${rocshmem_branch}" \
"${rocshmem_material}"
)
export ROCSHMEM_CACHE_KEY
echo "ROCShmem dependency cache key: ${ROCSHMEM_CACHE_KEY}"
fi
if [[ -n "${deepep_branch}" && -n "${rocshmem_branch}" ]]; then
deepep_material=$(compose_stage_cache_material "${dockerfile_rocm}" "base build_rocshmem build_deepep")
DEEPEP_CACHE_KEY=$(
compose_dependency_cache_key \
"${deepep_branch}" \
"${deepep_branch}-rocshmem-${rocshmem_branch}" \
"${deepep_material}"
)
export DEEPEP_CACHE_KEY
@@ -2395,11 +2415,18 @@ dependency_cache_ref_for_target() {
printf '%s\n' "${cache_repo}:nixl-rocm-${NIXL_BRANCH}-ucx-${UCX_BRANCH:-}"
fi
;;
rocshmem-rocm-ci)
if [[ -n "${ROCSHMEM_CACHE_KEY:-}" ]]; then
printf '%s\n' "${cache_repo}:rocshmem-rocm-${ROCSHMEM_CACHE_KEY}"
elif [[ -n "${ROCSHMEM_BRANCH:-}" ]]; then
printf '%s\n' "${cache_repo}:rocshmem-rocm-${ROCSHMEM_BRANCH}"
fi
;;
deepep-rocm-ci)
if [[ -n "${DEEPEP_CACHE_KEY:-}" ]]; then
printf '%s\n' "${cache_repo}:deepep-rocm-${DEEPEP_CACHE_KEY}"
elif [[ -n "${DEEPEP_BRANCH:-}" ]]; then
printf '%s\n' "${cache_repo}:deepep-rocm-${DEEPEP_BRANCH}"
printf '%s\n' "${cache_repo}:deepep-rocm-${DEEPEP_BRANCH}-rocshmem-${ROCSHMEM_BRANCH:-}"
fi
;;
esac
@@ -2417,6 +2444,7 @@ add_dependency_cache_target() {
resolve_ci_base_dependency_targets() {
local mode="${ROCM_DEP_CACHE_EXPORT_MODE:-missing}"
local nixl_ref=""
local rocshmem_ref=""
local deepep_ref=""
[[ "${TARGET}" == "ci-base-rocm-ci-with-deps" ]] || return 0
@@ -2424,7 +2452,7 @@ resolve_ci_base_dependency_targets() {
case "${mode}" in
always)
echo "ROCM_DEP_CACHE_EXPORT_MODE=always; exporting all dependency caches serially"
for target in nixl-rocm-ci deepep-rocm-ci; do
for target in nixl-rocm-ci rocshmem-rocm-ci deepep-rocm-ci; do
if [[ -n "$(dependency_cache_ref_for_target "${target}")" ]]; then
add_dependency_cache_target "${target}"
fi
@@ -2454,6 +2482,16 @@ resolve_ci_base_dependency_targets() {
fi
fi
if [[ "${mode}" != "always" && -n "${ROCSHMEM_CACHE_KEY:-}" ]]; then
rocshmem_ref=$(dependency_cache_ref_for_target "rocshmem-rocm-ci")
if dependency_cache_ref_exists "${rocshmem_ref}"; then
echo "ROCShmem dependency cache exists: ${rocshmem_ref}"
else
echo "ROCShmem dependency cache missing; will seed: ${rocshmem_ref}"
add_dependency_cache_target "rocshmem-rocm-ci"
fi
fi
if [[ "${mode}" != "always" && -n "${DEEPEP_CACHE_KEY:-}" ]]; then
deepep_ref=$(dependency_cache_ref_for_target "deepep-rocm-ci")
if dependency_cache_ref_exists "${deepep_ref}"; then
@@ -2464,6 +2502,15 @@ resolve_ci_base_dependency_targets() {
fi
fi
# DeepEP inherits from ROCShmem. If ROCShmem is being seeded, seed DeepEP too
# so the pair stays consistent for future ci_base rebuilds.
if printf '%s\n' "${DEPENDENCY_CACHE_TARGETS[@]}" | grep -qx "rocshmem-rocm-ci" \
&& ! printf '%s\n' "${DEPENDENCY_CACHE_TARGETS[@]}" | grep -qx "deepep-rocm-ci" \
&& [[ -n "${DEEPEP_BRANCH:-}" ]]; then
echo "ROCShmem cache is missing; also seeding DeepEP cache"
add_dependency_cache_target "deepep-rocm-ci"
fi
BAKE_TARGETS=("ci-base-rocm-ci")
if [[ ${#DEPENDENCY_CACHE_TARGETS[@]} -eq 0 ]]; then
echo "All dependency caches exist; building ci_base without dependency cache exports"
+2 -5
View File
@@ -5,17 +5,14 @@ set -ex
# Clean up old nightly builds from DockerHub, keeping only the last 14 builds
# This script uses DockerHub API to list and delete old tags with specified prefix
# Tags starting with "nightly-dev" are always excluded from deletion
# Usage: cleanup-nightly-builds.sh [TAG_PREFIX] [REPO] [EXCLUDE_PREFIX]
# Usage: cleanup-nightly-builds.sh [TAG_PREFIX] [REPO]
# Example: cleanup-nightly-builds.sh "nightly-"
# Example: cleanup-nightly-builds.sh "cu130-nightly-"
# Example: cleanup-nightly-builds.sh "nightly-" "vllm/vllm-openai-rocm"
# Example: cleanup-nightly-builds.sh "nightly-" "vllm/vllm-openai-rocm" "nightly-rocm"
# (EXCLUDE_PREFIX skips tags owned by another variant sharing TAG_PREFIX)
# Get tag prefix and repo from arguments
TAG_PREFIX="${1:-nightly-}"
REPO="${2:-vllm/vllm-openai}"
EXCLUDE_PREFIX="${3:-}"
echo "Cleaning up tags with prefix: $TAG_PREFIX in repository: $REPO"
@@ -60,7 +57,7 @@ get_all_tags() {
# Get both last_updated timestamp and tag name, separated by |
# Exclude mutable architecture tags and nightly-dev tags from cleanup
local tags=$(echo "$response" | jq -r --arg prefix "$TAG_PREFIX" --arg exclude "$EXCLUDE_PREFIX" '.results[] | select(.name | startswith($prefix)) | select($exclude == "" or (.name | startswith($exclude) | not)) | select(.name != ($prefix + "x86_64") and .name != ($prefix + "aarch64")) | select(.name | startswith("nightly-dev") | not) | "\(.last_updated)|\(.name)"')
local tags=$(echo "$response" | jq -r --arg prefix "$TAG_PREFIX" '.results[] | select(.name | startswith($prefix)) | select(.name != ($prefix + "x86_64") and .name != ($prefix + "aarch64")) | select(.name | startswith("nightly-dev") | not) | "\(.last_updated)|\(.name)"')
if [ -z "$tags" ]; then
break
+1 -6
View File
@@ -227,12 +227,7 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# five-minute lifetime and the loop above can run to the build-age deadline.
# Build 91827 waited 6h48m and then took a 401 on all 360 callbacks from a
# token that had expired hours earlier.
# Posting is paced under the relay's rate limit, which takes several minutes
# for a full nightly, so ask for a longer lifetime than the default. Fall back
# to the default if this agent rejects --lifetime.
OIDC_TOKEN="$(buildkite-agent oidc request-token --skip-redaction --audience "${AUDIENCE}" \
--lifetime 1800 2>/dev/null \
|| buildkite-agent oidc request-token --skip-redaction --audience "${AUDIENCE}" 2>/dev/null)"
OIDC_TOKEN="$(buildkite-agent oidc request-token --skip-redaction --audience "${AUDIENCE}" 2>/dev/null)"
if [[ -z "${OIDC_TOKEN}" ]]; then
echo "could not mint a Buildkite OIDC token after the wait -- skipping report"
exit 0
-16
View File
@@ -24,9 +24,7 @@ from __future__ import annotations
import argparse
import json
import os
import sys
import time
import urllib.error
import urllib.request
@@ -45,14 +43,6 @@ _CONCLUSION = {
}
# The relay rate-limits callbacks per repo over a sliding 60 s window
# (RATE_LIMIT_PER_MIN in pytorch/test-infra), and a nightly has far more jobs
# than that. Pace below the limit, and back off on a 429 for about a minute in
# total so the last retry lands in a new window.
_POST_INTERVAL_S = float(os.environ.get("CRCR_POST_INTERVAL_S", "0.65"))
_RATE_LIMIT_BACKOFF_S = (2, 4, 8, 16, 32)
def post(url: str, token: str, body: dict, timeout: int = 30) -> tuple[int, str]:
req = urllib.request.Request(
url,
@@ -123,12 +113,6 @@ def main() -> int:
ok += 1
continue
status, detail = post(args.callback_url, args.oidc_token, body)
for delay in _RATE_LIMIT_BACKOFF_S:
if status != 429:
break
time.sleep(delay)
status, detail = post(args.callback_url, args.oidc_token, body)
time.sleep(_POST_INTERVAL_S)
if 200 <= status < 300:
ok += 1
else:
+9 -63
View File
@@ -12,7 +12,7 @@ from dataclasses import asdict, dataclass
from datetime import datetime
from pathlib import Path
from typing import Any
from urllib.parse import quote, unquote, urlparse
from urllib.parse import quote
import regex as re
@@ -51,8 +51,6 @@ class WheelFileInfo:
platform_tag: str
variant: str | None
filename: str
# Absolute URL for files hosted elsewhere (see --external-links)
url: str | None = None
def parse_from_filename(file: str) -> WheelFileInfo:
@@ -112,29 +110,6 @@ def parse_from_filename(file: str) -> WheelFileInfo:
)
def parse_external_url(url: str) -> WheelFileInfo:
"""Parse an absolute wheel or sdist URL hosted on another index."""
filename = unquote(urlparse(url).path.rsplit("/", 1)[-1])
if filename.endswith(".whl"):
info = parse_from_filename(filename)
else:
match = re.match(r"^(?P<name>.+)-(?P<version>[^-]+)\.(tar\.gz|zip)$", filename)
if not match:
raise ValueError(f"Unsupported external file: {filename}")
info = WheelFileInfo(
package_name=match.group("name"),
version=match.group("version"),
build_tag=None,
python_tag="source",
abi_tag="none",
platform_tag="any",
variant=None,
filename=filename,
)
info.url = url
return info
def generate_project_list(subdir_names: list[str], comment: str = "") -> str:
"""Generate project list HTML content linking to each project & variant subdirectory."""
href_tags = []
@@ -154,19 +129,14 @@ def generate_package_index_and_metadata(
href_tags = []
metadata = []
for file in sorted(wheel_files, key=lambda x: x.filename):
if file.url:
file_path_quoted = file.url
else:
relative_path = (
wheel_base_dir.relative_to(index_base_dir, walk_up=True) / file.filename
)
# handle with '+' in URL, and avoid double-encoding '/' and already-encoded '%2B'
# NOTE: this is AWS S3 specific behavior!
file_path_quoted = quote(relative_path.as_posix(), safe=":%/")
relative_path = (
wheel_base_dir.relative_to(index_base_dir, walk_up=True) / file.filename
)
# handle with '+' in URL, and avoid double-encoding '/' and already-encoded '%2B'
# NOTE: this is AWS S3 specific behavior!
file_path_quoted = quote(relative_path.as_posix(), safe=":%/")
href_tags.append(f' <a href="{file_path_quoted}">{file.filename}</a><br/>')
file_meta = asdict(file)
if file_meta["url"] is None:
del file_meta["url"]
file_meta["path"] = file_path_quoted
metadata.append(file_meta)
index_str = INDEX_HTML_TEMPLATE.format(items="\n".join(href_tags), comment=comment)
@@ -181,7 +151,6 @@ def generate_index_and_metadata(
default_variant: str | None = None,
alias_to_default: str | None = None,
comment: str = "",
external_urls: list[str] | None = None,
):
"""Generate index for all wheel files.
@@ -191,8 +160,6 @@ def generate_index_and_metadata(
index_base_dir (Path): Base directory to store index files.
default_variant (str | None): The default variant name, if any.
alias_to_default (str | None): Alias variant name for the default variant, if any.
external_urls (list[str] | None): Absolute URLs of files hosted on another index,
listed in the generated indices without being copied.
comment (str | None): Optional comment to include in the generated HTML files.
First, parse all wheel files to extract metadata.
@@ -247,7 +214,6 @@ def generate_index_and_metadata(
"""
parsed_files = [parse_from_filename(f) for f in whl_files]
parsed_files += [parse_external_url(u) for u in external_urls or []]
if not parsed_files:
print("No wheel files found, skipping index generation.")
@@ -266,14 +232,10 @@ def generate_index_and_metadata(
print(f"Detected ROCm variant from vllm: {rocm_variant}")
break
# Apply ROCm variant to all wheels without a variant. Dependencies with their
# own ROCm local version (e.g. TheRock's torch-2.12.0+rocm10.0.0) belong to
# the same variant rather than one of their own.
# Apply ROCm variant to all wheels without a variant
if rocm_variant:
for file in parsed_files:
if file.variant is None or (
file.variant.startswith("rocm") and file.variant != rocm_variant
):
if file.variant is None:
file.variant = rocm_variant
print(f"Inherited variant '{rocm_variant}' for {file.filename}")
@@ -404,13 +366,6 @@ if __name__ == "__main__":
default=None,
help="Alias variant name for the default variant",
)
parser.add_argument(
"--external-links",
type=str,
default=None,
help="Optional file with one absolute URL per line for packages hosted on "
"another index (listed alongside, but not copied into, this index)",
)
parser.add_argument(
"--comment",
type=str,
@@ -508,14 +463,5 @@ if __name__ == "__main__":
default_variant=None,
alias_to_default=args.alias_to_default,
comment=args.comment.strip(),
external_urls=(
[
line.strip()
for line in Path(args.external_links).read_text().splitlines()
if line.strip() and not line.startswith("#")
]
if args.external_links
else None
),
)
print(f"Successfully generated index and metadata in {output_dir}")
+16 -28
View File
@@ -11,9 +11,9 @@ set -euo pipefail
TARGET="${1:-all}"
case "${TARGET}" in
cuda-13-0 | cuda-12-9 | cuda-13-0-ubuntu-24-04 | \
cuda-12-9-ubuntu-24-04 | rocm | rocm72 | xpu | cpu | all) ;;
cuda-12-9-ubuntu-24-04 | rocm | xpu | cpu | all) ;;
*)
echo "Usage: $0 {cuda-13-0|cuda-12-9|cuda-13-0-ubuntu-24-04|cuda-12-9-ubuntu-24-04|rocm|rocm72|xpu|cpu|all}"
echo "Usage: $0 {cuda-13-0|cuda-12-9|cuda-13-0-ubuntu-24-04|cuda-12-9-ubuntu-24-04|rocm|xpu|cpu|all}"
exit 2
;;
esac
@@ -143,35 +143,23 @@ if target_enabled cuda-12-9-ubuntu-24-04; then
fi
# ---- ROCm ----
# Each ROCm stack is identified by its base Dockerfile (see rocm/stack.sh). The
# default stack also owns the plain :latest / :v<ver> tags.
publish_rocm_stack() {
local base_dockerfile="$1" default="$2" suffix
(
# shellcheck source=.buildkite/scripts/rocm/stack.sh
source .buildkite/scripts/rocm/stack.sh "$base_dockerfile"
local suffixes=("-${ROCM_STACK_VARIANT}")
[[ "$default" == "1" ]] && suffixes=("" "-${ROCM_STACK_VARIANT}")
docker pull "$ROCM_STACK_ECR_IMAGE"
docker pull "$ROCM_STACK_ECR_BASE"
for suffix in "${suffixes[@]}"; do
for tag in "latest${suffix}" "v${RELEASE_VERSION}${suffix}"; do
docker tag "$ROCM_STACK_ECR_IMAGE" "vllm/vllm-openai-rocm:${tag}"
docker tag "$ROCM_STACK_ECR_BASE" "vllm/vllm-openai-rocm:${tag}-base"
docker push "vllm/vllm-openai-rocm:${tag}"
docker push "vllm/vllm-openai-rocm:${tag}-base"
done
done
)
}
if target_enabled rocm; then
publish_rocm_stack docker/Dockerfile.rocm_base 1
fi
ROCM_BASE_CACHE_KEY=$(.buildkite/scripts/cache-rocm-base-wheels.sh key)
echo "ROCm base cache key: ${ROCM_BASE_CACHE_KEY}"
if target_enabled rocm72; then
publish_rocm_stack docker/Dockerfile.rocm_72_base 0
docker pull "public.ecr.aws/q9t5s3a7/vllm-release-repo:${COMMIT}-rocm"
docker pull "public.ecr.aws/q9t5s3a7/vllm-release-repo:${ROCM_BASE_CACHE_KEY}-rocm-base"
docker tag "public.ecr.aws/q9t5s3a7/vllm-release-repo:${COMMIT}-rocm" vllm/vllm-openai-rocm:latest
docker tag "public.ecr.aws/q9t5s3a7/vllm-release-repo:${COMMIT}-rocm" "vllm/vllm-openai-rocm:v${RELEASE_VERSION}"
docker push vllm/vllm-openai-rocm:latest
docker push "vllm/vllm-openai-rocm:v${RELEASE_VERSION}"
docker tag "public.ecr.aws/q9t5s3a7/vllm-release-repo:${ROCM_BASE_CACHE_KEY}-rocm-base" vllm/vllm-openai-rocm:latest-base
docker tag "public.ecr.aws/q9t5s3a7/vllm-release-repo:${ROCM_BASE_CACHE_KEY}-rocm-base" "vllm/vllm-openai-rocm:v${RELEASE_VERSION}-base"
docker push vllm/vllm-openai-rocm:latest-base
docker push "vllm/vllm-openai-rocm:v${RELEASE_VERSION}-base"
fi
# ---- XPU ----
+65 -42
View File
@@ -2,36 +2,42 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
#
# Push a ROCm stack's nightly base image and nightly image from ECR to Docker Hub
# under vllm/vllm-openai-rocm, as base-nightly-<variant>, nightly-<variant> and their
# -<commit> forms, then prune old nightly tags. With --default, also push the plain
# base-nightly and nightly tags for the same images.
# Run when NIGHTLY=1 after the stack's release image has been pushed to ECR.
# Push a ROCm-family nightly base image and nightly image from ECR to Docker Hub
# under vllm/vllm-openai-rocm, as base-nightly[-<variant>], nightly[-<variant>]
# and their -<commit> forms.
# Run when NIGHTLY=1 after the matching build-*-release-image step has pushed to ECR.
#
# Usage: push-nightly-builds-rocm.sh <base-dockerfile> [--default]
# e.g. "docker/Dockerfile.rocm_base --default" (ROCm 10.0, also :nightly)
# and "docker/Dockerfile.rocm_72_base" (ROCm 7.2, :nightly-rocm72).
# Usage: push-nightly-builds-rocm.sh [ECR_TAG_SUFFIX TAG_VARIANT]
# Pass both arguments or neither. With none, the default ROCm apt build is
# published as :nightly. The TheRock build passes "rock" (the source ECR tag
# suffix) and "rocm714" (the Docker Hub tag flavor), giving :nightly-rocm714.
#
# Local testing (no push to Docker Hub):
# BUILDKITE_COMMIT=<commit-with-image-in-ecr> DRY_RUN=1 \
# bash .buildkite/scripts/push-nightly-builds-rocm.sh docker/Dockerfile.rocm_base --default
# BASE_ECR_IMAGE=<full-base-image-reference> \
# BUILDKITE_COMMIT=<commit-with-image-in-ecr> DRY_RUN=1 \
# bash .buildkite/scripts/push-nightly-builds-rocm.sh
# Requires: AWS CLI configured (for ECR public login), Docker. For full run: Docker Hub login.
set -euxo pipefail
usage() {
echo "Usage: $0 <base-dockerfile> [--default]" >&2
echo "Usage: $0 [ECR_TAG_SUFFIX TAG_VARIANT]" >&2
}
PUSH_DEFAULT=0
DOCKERHUB_REPO="vllm/vllm-openai-rocm"
case "$#" in
1) ;;
0)
ECR_TAG_SUFFIX="rocm"
TAG_VARIANT=""
;;
2)
if [[ "$2" != "--default" ]]; then
if [[ -z "$1" || -z "$2" ]]; then
usage
exit 2
fi
PUSH_DEFAULT=1
ECR_TAG_SUFFIX="$1"
TAG_VARIANT="$2"
;;
*)
usage
@@ -43,44 +49,61 @@ esac
BUILDKITE_COMMIT="${BUILDKITE_COMMIT:?Set BUILDKITE_COMMIT to the commit SHA that has the image in ECR (e.g. from a previous release pipeline run)}"
DRY_RUN="${DRY_RUN:-0}"
# shellcheck source=.buildkite/scripts/rocm/stack.sh
source "$(dirname "${BASH_SOURCE[0]}")/rocm/stack.sh" "$1"
DOCKERHUB_REPO="vllm/vllm-openai-rocm"
VARIANT="$ROCM_STACK_VARIANT"
ECR_REPO="public.ecr.aws/q9t5s3a7/vllm-release-repo"
BASE_TAGS=("base-nightly-${VARIANT}" "base-nightly-${VARIANT}-${BUILDKITE_COMMIT}")
TAGS=("nightly-${VARIANT}" "nightly-${VARIANT}-${BUILDKITE_COMMIT}")
if [[ "$PUSH_DEFAULT" == "1" ]]; then
BASE_TAGS+=("base-nightly" "base-nightly-${BUILDKITE_COMMIT}")
TAGS+=("nightly" "nightly-${BUILDKITE_COMMIT}")
# Get the base image ECR tag (set by the build-*-release-image pipeline step)
BASE_METADATA_KEY="${ECR_TAG_SUFFIX}-base-ecr-tag"
if [[ -n "${BASE_ECR_IMAGE:-}" ]]; then
if [[ "$DRY_RUN" != "1" ]]; then
echo "ERROR: BASE_ECR_IMAGE may only be used with DRY_RUN=1" >&2
exit 1
fi
BASE_ORIG_TAG="$BASE_ECR_IMAGE"
elif ! BASE_ORIG_TAG="$(buildkite-agent meta-data get "$BASE_METADATA_KEY")"; then
echo "ERROR: Failed to read required Buildkite metadata '$BASE_METADATA_KEY'" >&2
exit 1
fi
if [[ -z "$BASE_ORIG_TAG" ]]; then
echo "ERROR: Required Buildkite metadata '$BASE_METADATA_KEY' is empty" >&2
echo "Set BASE_ECR_IMAGE to the full ECR base image reference for local testing" >&2
exit 1
fi
echo "Pushing base image $ROCM_STACK_ECR_BASE as: ${BASE_TAGS[*]}"
echo "Pushing release image $ROCM_STACK_ECR_IMAGE as: ${TAGS[*]}"
ORIG_TAG="${BUILDKITE_COMMIT}-${ECR_TAG_SUFFIX}"
VARIANT_SUFFIX=""
[[ -n "$TAG_VARIANT" ]] && VARIANT_SUFFIX="-$TAG_VARIANT"
BASE_TAG_NAME="base-nightly${VARIANT_SUFFIX}"
TAG_NAME="nightly${VARIANT_SUFFIX}"
BASE_TAG_NAME_COMMIT="base-nightly${VARIANT_SUFFIX}-${BUILDKITE_COMMIT}"
TAG_NAME_COMMIT="nightly${VARIANT_SUFFIX}-${BUILDKITE_COMMIT}"
# Login to ECR and pull the stack's images
echo "Pushing base image from ECR: $BASE_ORIG_TAG"
echo "Pushing release image from ECR tag: $ORIG_TAG to Docker Hub as $DOCKERHUB_REPO:$TAG_NAME and $DOCKERHUB_REPO:$TAG_NAME_COMMIT"
[[ "$DRY_RUN" == "1" ]] && echo "[DRY_RUN] Skipping push to Docker Hub"
# Login to ECR and pull the images built by build-*-release-image
aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7
docker pull "$ROCM_STACK_ECR_BASE"
docker pull "$ROCM_STACK_ECR_IMAGE"
docker pull "$BASE_ORIG_TAG"
docker pull "${ECR_REPO}:${ORIG_TAG}"
for tag in "${BASE_TAGS[@]}"; do docker tag "$ROCM_STACK_ECR_BASE" "$DOCKERHUB_REPO:$tag"; done
for tag in "${TAGS[@]}"; do docker tag "$ROCM_STACK_ECR_IMAGE" "$DOCKERHUB_REPO:$tag"; done
# Tag for Docker Hub (base-nightly and base-nightly-<commit>, nightly and nightly-<commit>)
docker tag "$BASE_ORIG_TAG" "$DOCKERHUB_REPO":"$BASE_TAG_NAME"
docker tag "$BASE_ORIG_TAG" "$DOCKERHUB_REPO":"$BASE_TAG_NAME_COMMIT"
docker tag "${ECR_REPO}:${ORIG_TAG}" "$DOCKERHUB_REPO":"$TAG_NAME"
docker tag "${ECR_REPO}:${ORIG_TAG}" "$DOCKERHUB_REPO":"$TAG_NAME_COMMIT"
if [[ "$DRY_RUN" == "1" ]]; then
echo "[DRY_RUN] Would push $DOCKERHUB_REPO:$BASE_TAG_NAME and $DOCKERHUB_REPO:$BASE_TAG_NAME_COMMIT"
echo "[DRY_RUN] Would push $DOCKERHUB_REPO:$TAG_NAME and $DOCKERHUB_REPO:$TAG_NAME_COMMIT"
echo "[DRY_RUN] Local tags created. Exiting without push."
exit 0
fi
# Push to Docker Hub (docker-login plugin runs before this step in CI)
for tag in "${BASE_TAGS[@]}" "${TAGS[@]}"; do docker push "$DOCKERHUB_REPO:$tag"; done
echo "Pushed $DOCKERHUB_REPO: ${BASE_TAGS[*]} ${TAGS[*]}"
docker push "$DOCKERHUB_REPO":"$BASE_TAG_NAME"
docker push "$DOCKERHUB_REPO":"$BASE_TAG_NAME_COMMIT"
docker push "$DOCKERHUB_REPO":"$TAG_NAME"
docker push "$DOCKERHUB_REPO":"$TAG_NAME_COMMIT"
# Keep only the last 14 builds of each tag family. The plain nightly- families
# exclude the variant ones sharing their prefix; commit SHAs never start with "rocm".
CLEANUP=.buildkite/scripts/cleanup-nightly-builds.sh
bash "$CLEANUP" "nightly-${VARIANT}-" "$DOCKERHUB_REPO"
bash "$CLEANUP" "base-nightly-${VARIANT}-" "$DOCKERHUB_REPO"
if [[ "$PUSH_DEFAULT" == "1" ]]; then
bash "$CLEANUP" "nightly-" "$DOCKERHUB_REPO" "nightly-rocm"
bash "$CLEANUP" "base-nightly-" "$DOCKERHUB_REPO" "base-nightly-rocm"
fi
echo "Pushed $DOCKERHUB_REPO:$BASE_TAG_NAME and $DOCKERHUB_REPO:$BASE_TAG_NAME_COMMIT"
echo "Pushed $DOCKERHUB_REPO:$TAG_NAME and $DOCKERHUB_REPO:$TAG_NAME_COMMIT"
@@ -1,112 +0,0 @@
#!/bin/bash
# Install the AITER nightly wheel over this build's ci_base.
#
# Runs in CI when AITER_NIGHTLY=1, once the stock ci_base it builds on is
# published. In order:
# 1. Select the newest nightly wheel that fits the stock ci_base
# (select_aiter_nightly_wheel.py, beside this script, run inside that image)
# 2. Build a new image: pip install that wheel over the stock ci_base
# 3. Push the new image as this build's ci_base-build tag, replacing the stock
# one, so later steps and native jobs use it; record its digest
#
# Buildkite meta-data, a key-value store shared by every job in this build:
# reads rocm-ci-base-image the stock ci_base
# r/w aiter-nightly-wheel-url the wheel step 1 chose; a retry reads it
# back and installs that same wheel
# writes aiter-nightly-ci-base the new image as tag@digest; the vLLM
# image build uses it as its ci_base
#
# Exit status:
# 10 no wheel fits this ci_base
# 11 the wheel does not install
# Anything else is unexpected and passed through as is.
# Buildkite retries this step once on exit status 1 or 2, which it treats as
# transient Docker/registry failures (amd_infra_retry in the template).
# 10 and 11 sit outside that range: they are real AITER failures, and a retry
# would only fail the same way.
set -euo pipefail
SELECT_WHEEL="$(dirname "${BASH_SOURCE[0]}")/select_aiter_nightly_wheel.py"
BUILD_TAG="rocm/vllm-dev:ci_base-build-${BUILDKITE_BUILD_ID:?BUILDKITE_BUILD_ID is required}"
# Scratch space for the build context and captured errors. The trap deletes it
# however the script exits: success, fail, or set -e.
scratch="$(mktemp -d)"
trap 'rm -rf "${scratch}"' EXIT
errors="${scratch}/errors"
fail() {
local status="$1" title="$2" detail="$3"
echo "+++ :x: AITER nightly: ${title}"
echo "${detail}" >&2
# shellcheck disable=SC2016 # the backticks are a literal markdown fence
printf ':x: **AITER nightly: %s**\n\n```\n%s\n```\n' "${title}" "${detail}" \
| buildkite-agent annotate --style error --context aiter-nightly
exit "${status}"
}
# Set by ensure-ci-base-amd; this step's depends_on guarantees it is there.
ci_base="$(buildkite-agent meta-data get rocm-ci-base-image)"
# Set below on the first attempt. Meta-data is per build, so a retry reuses
# that wheel, not a newer nightly.
pinned="$(buildkite-agent meta-data get aiter-nightly-wheel-url --default "")"
echo "--- :docker: Pulling ${ci_base}"
docker pull "${ci_base}"
echo "--- :mag: Selecting the AITER nightly for ${ci_base}"
set +e
out="$(docker run --rm -i --entrypoint python3 "${ci_base}" \
-P - ${pinned:+--wheel-url "${pinned}"} \
< "${SELECT_WHEEL}" 2> "${errors}")"
status=$?
set -e
cat "${errors}" >&2
case "${status}" in
0) ;;
10) fail 10 "no wheel for this image" "${out}" ;;
*) fail "${status}" "could not select a wheel" "$(tail -n 20 "${errors}")" ;;
esac
# The first line is the wheel's URL; the rest, the annotation naming it.
wheel_url="$(head -n 1 <<< "${out}")"
note="$(tail -n +2 <<< "${out}")"
buildkite-agent meta-data set aiter-nightly-wheel-url "${wheel_url}"
cat > "${scratch}/Dockerfile" <<'EOF'
ARG CI_BASE_IMAGE
FROM ${CI_BASE_IMAGE}
ARG AITER_WHEEL_URL
RUN python3 -m pip install --no-cache-dir "$AITER_WHEEL_URL"
EOF
echo "--- :docker: Installing ${wheel_url##*/}"
if ! docker buildx build \
--builder default \
--provenance=false \
--progress plain \
--build-arg "CI_BASE_IMAGE=${ci_base}" \
--build-arg "AITER_WHEEL_URL=${wheel_url}" \
-t "${BUILD_TAG}" \
"${scratch}"; then
fail 11 "the wheel does not install" \
"Installing ${wheel_url##*/} over ${ci_base} failed; see this step's log."
fi
docker push "${BUILD_TAG}"
# Handed off as tag@digest: build-test-image.sh rejects a ci_base that is not
# digest-pinned.
digest="$(docker buildx imagetools inspect "${BUILD_TAG}" | awk '$1 == "Digest:" { print $2; exit }')"
if [[ ! "${digest}" =~ ^sha256:[0-9a-f]{64}$ ]]; then
echo "Could not resolve the pushed digest of ${BUILD_TAG}" >&2
exit 1
fi
# The vLLM image build uses this as its ci_base.
buildkite-agent meta-data set aiter-nightly-ci-base "${BUILD_TAG}@${digest}"
# The annotation starts with :warning: when the wheel is stale.
style=info
if [[ "${note}" == :warning:* ]]; then
style=warning
fi
echo "${note}" | buildkite-agent annotate --style "${style}" --context aiter-nightly
echo "--- :white_check_mark: ${BUILD_TAG}@${digest}"
+2 -2
View File
@@ -1,8 +1,8 @@
#!/usr/bin/env bash
# Shared Dockerfile selection for the normal AMD CI build chain.
# CI_ROCM_DOCKERFILE_BASE and CI_ROCM_DOCKERFILE accept repository-relative paths.
# For the legacy ROCm 7.2 stack, set CI_ROCM_DOCKERFILE_BASE=docker/Dockerfile.rocm_72_base
# and CI_ROCM_DOCKERFILE=docker/Dockerfile.rocm_72.
# For Rock, set CI_ROCM_DOCKERFILE_BASE=docker/Dockerfile.rock_base and
# CI_ROCM_DOCKERFILE=docker/Dockerfile.rock.
# Custom recipes must preserve the CI targets and artifact layout. Additional
# COPY inputs also need the corresponding *_CONTENT_FILES overrides.
@@ -1,260 +0,0 @@
#!/usr/bin/env python3
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""Select the AITER nightly wheel for a vLLM ROCm image.
aiter-nightly-overlay.sh runs this inside the stock ci_base, so it reads that
image's ROCm, torch and Python. It prints the wheel's URL, then the Buildkite
annotation naming it.
"""
import argparse
import datetime
import importlib.metadata
import os
import sys
import urllib.request
from dataclasses import dataclass
from html.parser import HTMLParser
from pathlib import Path
from urllib.parse import unquote, urldefrag, urljoin
import regex as re
INDEX_URL = "https://rocm.frameworks-nightlies.amd.com/whl-multi-arch/amd-aiter/"
AITER_COMMIT_URL = "https://github.com/ROCm/aiter/commit/"
STALE_AFTER = datetime.timedelta(days=1)
# Example:
# amd_aiter-0.1.25+rocm7.2.3.torch2.12.ae984f5.d20261008-cp312-cp312-linux_x86_64.whl
# is AITER main at commit ae984f5, built 2026-10-08, for ROCm 7.2.3, torch 2.12
# and Python 3.12. 0.1.25 is a guess at the next release, not a real one.
WHEEL_NAME = re.compile(
r"^amd_aiter-(?P<base_version>[^+-]+)\+(?P<local>[^-]+)"
r"-(?P<python>cp\d+)-(?P=python)-linux_x86_64\.whl$"
)
ROCM_FIELD = re.compile(r"(?:^|\.)rocm(\d+(?:\.\d+)*)(?:a\d+)?(?=\.|$)")
TORCH_FIELD = re.compile(r"(?:^|\.)torch(\d+\.\d+)(?=\.|$)")
COMMIT_AND_DATE_FIELD = re.compile(r"(?:^|\.)([0-9a-f]{7,40})\.d(\d{8})(?=\.|$)")
@dataclass(frozen=True)
class Wheel:
"""An AITER nightly wheel, read from its filename (see WHEEL_NAME)."""
filename: str
url: str
version: str
base_version: str
rocm: str
torch: str
commit: str
build_date: datetime.date
python: str
class _LinkParser(HTMLParser):
"""Collects every link on the index page, which lists one per wheel."""
def __init__(self):
super().__init__()
self.hrefs: list[str] = []
def handle_starttag(self, tag, attrs):
if tag == "a":
self.hrefs.extend(
value for name, value in attrs if name == "href" and value
)
def wheel_from_url(url: str) -> Wheel | None:
"""The AITER nightly wheel an absolute URL points at, if it is one."""
filename = unquote(urldefrag(url).url.rsplit("/", 1)[-1])
match = WHEEL_NAME.match(filename)
if not match:
return None
local = match["local"]
rocm, torch, build = (
field.search(local)
for field in (ROCM_FIELD, TORCH_FIELD, COMMIT_AND_DATE_FIELD)
)
if not (rocm and torch and build):
return None
return Wheel(
filename=filename,
url=url,
version=f"{match['base_version']}+{local}",
base_version=match["base_version"],
rocm=rocm[1],
torch=torch[1],
commit=build[1],
build_date=datetime.datetime.strptime(build[2], "%Y%m%d").date(),
python=match["python"],
)
def parse_index(html: str, index_url: str) -> list[Wheel]:
"""Every AITER nightly wheel linked from the index page, a plain HTML list
of links, one per file (the format pip reads)."""
parser = _LinkParser()
parser.feed(html)
wheels = (wheel_from_url(urljoin(index_url, href)) for href in parser.hrefs)
return [wheel for wheel in wheels if wheel]
def rocm_version() -> str:
"""This image's ROCm version; ROCm 10+ ships as the `rocm` pip SDK."""
try:
return importlib.metadata.version("rocm")
except importlib.metadata.PackageNotFoundError:
# No pip SDK: fall back to older ROCm releases, which install to ROCM_PATH.
rocm_path = Path(os.environ.get("ROCM_PATH", "/opt/rocm"))
return (rocm_path / ".info" / "version").read_text()
def rocm_release(version: str) -> str:
"""`7.2.3-49` or `10.0.0a20261001` -> `7.2.3` or `10.0.0`, as wheel names
spell it."""
match = re.match(r"\d+(?:\.\d+)*", version.strip())
if not match:
raise ValueError(f"Unrecognised ROCm version: {version!r}")
return match.group(0)
def torch_release(version: str) -> str:
"""`2.12.0a0+git6bbd260` -> `2.12`, as wheel names spell it."""
match = re.match(r"\d+\.\d+", version)
if not match:
raise ValueError(f"Unrecognised torch version: {version!r}")
return match.group(0)
def python_tag(version_info: tuple[int, int]) -> str:
"""(3, 12) -> `cp312`, as wheel names spell it."""
return f"cp{version_info[0]}{version_info[1]}"
def _base_version_key(wheel: Wheel) -> tuple[int, ...]:
"""`0.1.25` -> (0, 1, 25), so versions sort as numbers: 0.1.10 > 0.1.9."""
return tuple(int(part) for part in re.findall(r"\d+", wheel.base_version))
def _label(rocm: str, torch: str, python: str) -> str:
"""`rocm7.2.3/torch2.12/cp312`, for messages."""
return f"rocm{rocm}/torch{torch}/{python}"
def select_wheel(wheels: list[Wheel], rocm: str, torch: str, python: str) -> Wheel:
"""Newest build for this ROCm, torch and Python, by date: version labels lag.
Two builds with the same date and version: whichever the index lists first.
"""
target = (rocm, torch, python)
matching = [w for w in wheels if (w.rocm, w.torch, w.python) == target]
if not matching:
available = sorted({_label(w.rocm, w.torch, w.python) for w in wheels})
raise LookupError(
f"No AITER nightly for {_label(*target)}. "
f"Available: {', '.join(available) or 'none'}"
)
return max(matching, key=lambda w: (w.build_date, _base_version_key(w)))
def fetch(url: str) -> str:
"""GET url as text. A failure exits 1, so Buildkite retries the step."""
with urllib.request.urlopen(url, timeout=60) as response:
return response.read().decode()
def annotation(wheel: Wheel, today: datetime.date) -> str:
"""The Buildkite annotation naming the wheel this build tests.
Starts with :warning: if the wheel is stale; the overlay script styles it so.
"""
age = today - wheel.build_date
stale = (
f":warning: **AITER nightly is {age.days} days old**: "
f"no newer build has been published for "
f"{_label(wheel.rocm, wheel.torch, wheel.python)}.\n\n"
if age > STALE_AFTER
else ""
)
return stale + (
f":crescent_moon: **AITER nightly**: main at "
f"[`{wheel.commit}`]({AITER_COMMIT_URL}{wheel.commit}), "
f"built {wheel.build_date.isoformat()} for torch {wheel.torch} "
f"(`{wheel.version}`)"
)
class NoWheel(Exception):
"""No wheel fits this image's ROCm, torch and Python. Exit 10."""
def _image_wheel(wheel_url: str) -> Wheel:
"""The wheel at wheel_url, if it fits this image's ROCm, torch and Python."""
wheel = wheel_from_url(wheel_url)
if not wheel:
raise NoWheel(f"Not an AITER nightly wheel: {wheel_url}")
return _for_image([wheel])
def _image() -> tuple[str, str, str]:
"""This image's ROCm, torch and Python, as wheel names spell them."""
try:
rocm = rocm_release(rocm_version())
except (OSError, ValueError) as error:
raise NoWheel(f"Cannot read this image's ROCm version: {error}") from error
try:
torch = torch_release(importlib.metadata.version("torch"))
except (importlib.metadata.PackageNotFoundError, ValueError) as error:
raise NoWheel(f"Cannot read this image's torch version: {error}") from error
return rocm, torch, python_tag(sys.version_info[:2])
def _for_image(wheels: list[Wheel]) -> Wheel:
"""The newest of wheels for the image this runs in."""
try:
return select_wheel(wheels, *_image())
except LookupError as error:
raise NoWheel(str(error)) from error
def select(index_url: str, wheel_url: str = "") -> Wheel:
"""The wheel to install: wheel_url on a retry, else the newest.
However old the newest is, it is tested; the annotation shows its build date.
"""
if wheel_url:
return _image_wheel(wheel_url)
return _for_image(parse_index(fetch(index_url), index_url))
def _today() -> datetime.date:
"""Today in UTC; a function so tests can set the date."""
return datetime.datetime.now(datetime.UTC).date()
# Not 1 or 2, which Buildkite retries; see aiter-nightly-overlay.sh.
EXIT_NO_WHEEL = 10
def main(argv=None) -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--index-url", default=INDEX_URL)
parser.add_argument(
"--wheel-url", default="", help="a retry's earlier choice; skips the index"
)
args = parser.parse_args(argv)
try:
wheel = select(args.index_url, args.wheel_url)
except NoWheel as error:
print(error)
return EXIT_NO_WHEEL
print(wheel.url)
print(annotation(wheel, _today()))
return 0
if __name__ == "__main__":
sys.exit(main())
-78
View File
@@ -1,78 +0,0 @@
#!/usr/bin/env bash
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
#
# Derive everything the release pipeline needs about a ROCm stack from its base
# Dockerfile, so the stack's image is the single input to every ROCm script.
#
# Usage: source .buildkite/scripts/rocm/stack.sh <base-dockerfile>
# e.g. docker/Dockerfile.rocm_base (ROCm 10.0) or docker/Dockerfile.rocm_72_base
#
# Exports:
# ROCM_STACK_BASE_DOCKERFILE the base Dockerfile
# ROCM_STACK_DOCKERFILE the matching vLLM Dockerfile (base name minus _base)
# ROCM_STACK_THEROCK 1 if the ROCm SDK comes from TheRock wheels
# ROCM_STACK_VERSION ROCm version, e.g. 10.0.0 or 7.2.3
# ROCM_STACK_VARIANT wheel variant and Docker Hub tag flavor from major.minor,
# same rule as setup.py (rocm100, rocm72)
# ROCM_STACK_IMAGE_KEY hash of the base Dockerfile (base image cache key)
# ROCM_STACK_WHEEL_KEY hash of the base Dockerfile plus files it COPYs from
# the repo (base wheel cache key)
# ROCM_STACK_ECR_BASE ECR tag of the cached base image
# ROCM_STACK_ECR_IMAGE ECR tag of this commit's release image
# ROCM_STACK_BASE_WHEELS_DIR local dir for base wheels (also the artifact path)
# ROCM_STACK_VLLM_WHEEL_DIR local dir for the vLLM wheel (also the artifact path)
rocm_stack_init() {
local base="$1"
if [[ ! -f "$base" ]]; then
echo "ROCm base Dockerfile not found: $base" >&2
return 2
fi
ROCM_STACK_BASE_DOCKERFILE="$base"
ROCM_STACK_DOCKERFILE="${base%_base}"
local sdk
sdk=$(sed -nE 's/^ARG ROCM_SDK_VERSION=([0-9.]+).*/\1/p' "$base" | head -1)
if [[ -n "$sdk" ]]; then
ROCM_STACK_THEROCK=1
ROCM_STACK_VERSION="$sdk"
else
ROCM_STACK_THEROCK=0
# BASE_IMAGE format: rocm/dev-ubuntu-22.04:7.2.3-complete -> 7.2.3
ROCM_STACK_VERSION=$(sed -nE 's/^ARG BASE_IMAGE=.*:([0-9]+(\.[0-9]+)+).*/\1/p' "$base" | head -1)
fi
if [[ -z "$ROCM_STACK_VERSION" ]]; then
echo "Could not determine the ROCm version from $base" >&2
return 2
fi
local major minor
IFS=. read -r major minor _ <<< "$ROCM_STACK_VERSION"
ROCM_STACK_VARIANT="rocm${major}${minor}"
ROCM_STACK_IMAGE_KEY=$(sha256sum "$base" | cut -c1-16)
local inputs
inputs=$(grep -E '^COPY ' "$base" | grep -v -- '--from' | awk '{print $2}' || true)
if [[ -z "$inputs" ]]; then
ROCM_STACK_WHEEL_KEY="$ROCM_STACK_IMAGE_KEY"
else
# shellcheck disable=SC2086
ROCM_STACK_WHEEL_KEY=$(sha256sum "$base" $inputs | sha256sum | cut -c1-16)
fi
local ecr="public.ecr.aws/q9t5s3a7/vllm-release-repo"
ROCM_STACK_ECR_BASE="${ecr}:${ROCM_STACK_IMAGE_KEY}-${ROCM_STACK_VARIANT}-base"
ROCM_STACK_ECR_IMAGE="${ecr}:${BUILDKITE_COMMIT:-unknown}-${ROCM_STACK_VARIANT}"
ROCM_STACK_BASE_WHEELS_DIR="artifacts/${ROCM_STACK_VARIANT}-base-wheels"
ROCM_STACK_VLLM_WHEEL_DIR="artifacts/${ROCM_STACK_VARIANT}-vllm-wheel"
export ROCM_STACK_BASE_DOCKERFILE ROCM_STACK_DOCKERFILE ROCM_STACK_THEROCK \
ROCM_STACK_VERSION ROCM_STACK_VARIANT ROCM_STACK_IMAGE_KEY \
ROCM_STACK_WHEEL_KEY ROCM_STACK_ECR_BASE ROCM_STACK_ECR_IMAGE \
ROCM_STACK_BASE_WHEELS_DIR ROCM_STACK_VLLM_WHEEL_DIR
}
if [[ $# -gt 0 ]]; then
rocm_stack_init "$1"
fi
+20 -66
View File
@@ -4,31 +4,21 @@
#
# Upload ROCm wheels to S3 with proper index generation
#
# Usage: upload-rocm-wheels.sh <base-dockerfile>
# base-dockerfile the ROCm stack, e.g. docker/Dockerfile.rocm_base (see rocm/stack.sh)
#
# Required environment variables:
# AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY (or IAM role)
# S3_BUCKET (default: vllm-wheels)
#
# S3 path structure:
# s3://vllm-wheels/rocm/{commit}/<variant>-wheels/ - This stack's wheels for this commit
# s3://vllm-wheels/rocm/{commit}/<variant>/ - Index per ROCm variant (e.g. rocm100)
# s3://vllm-wheels/rocm/nightly/<variant>/ - Index pointing to latest nightly
# s3://vllm-wheels/rocm/{version}/<variant>/ - Index for release versions
# The top-level index.html at each prefix lists every variant present, since
# several ROCm stacks publish into the same prefixes.
# s3://vllm-wheels/rocm/{commit}/ - All wheels for this commit
# s3://vllm-wheels/rocm/nightly/ - Index pointing to latest nightly
# s3://vllm-wheels/rocm/{version}/ - Index for release versions
set -ex
# ======== Configuration ========
BUCKET="${S3_BUCKET:-vllm-wheels}"
# shellcheck source=.buildkite/scripts/rocm/stack.sh
source "$(dirname "${BASH_SOURCE[0]}")/rocm/stack.sh" "${1:?Usage: $0 <base-dockerfile>}"
ROCM_SUBPATH="rocm/${BUILDKITE_COMMIT}"
WHEEL_DIRS="$ROCM_STACK_BASE_WHEELS_DIR $ROCM_STACK_VLLM_WHEEL_DIR"
WHEEL_SUBPATH="$ROCM_SUBPATH/${ROCM_STACK_VARIANT}-wheels"
S3_WHEEL_PREFIX="s3://$BUCKET/$WHEEL_SUBPATH/"
S3_COMMIT_PREFIX="s3://$BUCKET/$ROCM_SUBPATH/"
INDICES_OUTPUT_DIR="rocm-indices"
echo "========================================"
@@ -59,11 +49,9 @@ source .buildkite/scripts/lib/manylinux.sh
# ======== Part 1: Collect and prepare wheels ========
# Collect all wheels
rm -rf all-rocm-wheels "$INDICES_OUTPUT_DIR"
mkdir -p all-rocm-wheels
for dir in $WHEEL_DIRS; do
cp "$dir"/*.whl all-rocm-wheels/ 2>/dev/null || true
done
cp artifacts/rocm-base-wheels/*.whl all-rocm-wheels/ 2>/dev/null || true
cp artifacts/rocm-vllm-wheel/*.whl all-rocm-wheels/ 2>/dev/null || true
WHEEL_COUNT=$(find all-rocm-wheels -maxdepth 1 -name '*.whl' 2>/dev/null | wc -l)
echo "Total wheels to upload: $WHEEL_COUNT"
@@ -96,9 +84,9 @@ ls -lh all-rocm-wheels/
# ======== Part 2: Upload wheels to S3 ========
echo ""
echo "Uploading wheels to $S3_WHEEL_PREFIX"
echo "Uploading wheels to $S3_COMMIT_PREFIX"
for wheel in all-rocm-wheels/*.whl; do
aws s3 cp "$wheel" "$S3_WHEEL_PREFIX"
aws s3 cp "$wheel" "$S3_COMMIT_PREFIX"
done
# ======== Part 3: Generate and upload indices ========
@@ -107,63 +95,28 @@ done
echo ""
echo "Generating indices..."
obj_json="rocm-objects.json"
aws s3api list-objects-v2 --bucket "$BUCKET" --prefix "$WHEEL_SUBPATH/" --delimiter / --output json > "$obj_json"
aws s3api list-objects-v2 --bucket "$BUCKET" --prefix "$ROCM_SUBPATH/" --delimiter / --output json > "$obj_json"
mkdir -p "$INDICES_OUTPUT_DIR"
# Use the existing generate-nightly-index.py
# HACK: Replace regex module with stdlib re (same as CUDA script)
sed -i 's/import regex as re/import re/g' .buildkite/scripts/generate-nightly-index.py
sed -i 's/import regex as re/import re/g' tools/vllm-rocm/therock_wheels.py
sed -i 's/import pybase64 as base64/import base64/g' tools/vllm-rocm/therock_wheels.py
INDEX_ARGS=(--version "$ROCM_SUBPATH" --wheel-dir "${WHEEL_SUBPATH#rocm/}")
# TheRock's SDK, device kernels and torchvision/torchaudio are linked, not hosted
if [[ "$ROCM_STACK_THEROCK" == "1" ]]; then
$PYTHON tools/vllm-rocm/therock_wheels.py external-links \
--dockerfile "$ROCM_STACK_BASE_DOCKERFILE" \
--torch-wheel "$(ls "$ROCM_STACK_BASE_WHEELS_DIR"/torch-*.whl)" > therock-external-links.txt
cat therock-external-links.txt
INDEX_ARGS+=(--external-links therock-external-links.txt)
fi
$PYTHON .buildkite/scripts/generate-nightly-index.py \
"${INDEX_ARGS[@]}" \
--version "$ROCM_SUBPATH" \
--current-objects "$obj_json" \
--output-dir "$INDICES_OUTPUT_DIR" \
--comment "ROCm commit $BUILDKITE_COMMIT"
# Upload this build's variant indices under an S3 prefix (e.g. rocm/nightly), then
# rebuild that prefix's top-level project list from every variant now present.
publish_indices() {
local prefix="$1"
local listing variants merged
echo "Uploading indices to s3://$BUCKET/$prefix/"
aws s3 cp --recursive "$INDICES_OUTPUT_DIR/" "s3://$BUCKET/$prefix/" --exclude "index.html"
listing=$(aws s3api list-objects-v2 --bucket "$BUCKET" --prefix "$prefix/" \
--delimiter / --query 'CommonPrefixes[].Prefix' --output text)
variants=$(tr '\t' '\n' <<< "$listing" | sed -n "s|^$prefix/\(rocm[0-9]*\)/$|\1|p" | sort -u)
merged=$(mktemp)
{
echo '<!DOCTYPE html>'
echo '<html>'
echo ' <meta name="pypi:repository-version" content="1.0">'
echo ' <body>'
for variant in $variants; do
echo " <a href=\"$variant/\">$variant/</a><br/>"
done
echo ' </body>'
echo '</html>'
} > "$merged"
echo "Variants under $prefix/: $(echo "$variants" | tr '\n' ' ')"
aws s3 cp "$merged" "s3://$BUCKET/$prefix/index.html"
rm -f "$merged"
}
publish_indices "$ROCM_SUBPATH"
# Upload indices to commit directory
echo "Uploading indices to $S3_COMMIT_PREFIX"
aws s3 cp --recursive "$INDICES_OUTPUT_DIR/" "$S3_COMMIT_PREFIX"
# Only scheduled nightly builds should update the moving nightly index.
if [[ "${NIGHTLY:-0}" == "1" ]]; then
publish_indices "rocm/nightly"
echo "Updating rocm/nightly/ index..."
aws s3 cp --recursive "$INDICES_OUTPUT_DIR/" "s3://$BUCKET/rocm/nightly/"
fi
# Extract version from vLLM wheel and update version-specific index
@@ -176,7 +129,8 @@ if [ -n "$VLLM_WHEEL" ]; then
echo "Pure version: $PURE_VERSION"
if [[ "$VERSION" != *"dev"* ]]; then
publish_indices "rocm/$PURE_VERSION"
echo "Updating rocm/$PURE_VERSION/ index..."
aws s3 cp --recursive "$INDICES_OUTPUT_DIR/" "s3://$BUCKET/rocm/$PURE_VERSION/"
fi
fi
@@ -188,14 +142,14 @@ echo "ROCm Wheel Upload Complete!"
echo "========================================"
echo ""
echo "Wheels available at:"
echo " s3://$BUCKET/$WHEEL_SUBPATH/"
echo " s3://$BUCKET/$ROCM_SUBPATH/"
echo ""
echo "Install command (by commit):"
echo " pip install vllm --extra-index-url https://${BUCKET}.s3.amazonaws.com/$ROCM_SUBPATH/$ROCM_STACK_VARIANT/"
echo " pip install vllm --extra-index-url https://${BUCKET}.s3.amazonaws.com/$ROCM_SUBPATH/"
echo ""
if [[ "${NIGHTLY:-0}" == "1" ]]; then
echo "Install command (nightly):"
echo " pip install vllm --extra-index-url https://${BUCKET}.s3.amazonaws.com/rocm/nightly/$ROCM_STACK_VARIANT/"
echo " pip install vllm --extra-index-url https://${BUCKET}.s3.amazonaws.com/rocm/nightly/"
fi
echo ""
echo "Wheel count: $WHEEL_COUNT"
+263 -478
View File
File diff suppressed because it is too large Load Diff
+8 -35
View File
@@ -2,9 +2,8 @@ group: Attention
depends_on:
- image-build
steps:
- label: ":nvidia: (H100) V1 Attention"
- label: ":nvidia: (H100) V1 Attention Shard %N"
key: v1-attention-h100-mi300
automatic_shard: true
timeout_in_minutes: 85
device: h100
source_file_dependencies:
@@ -13,18 +12,14 @@ steps:
- vllm/v1/attention
- tests/v1/attention
commands:
- pytest -v -s v1/attention
- pytest -v -s v1/attention --shard-id=$$BUILDKITE_PARALLEL_JOB --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT
parallelism: 2
mirror:
amd:
label: ":amd: (MI300) V1 Attention Shard %N"
dind: false
device: mi300_1
timeout_in_minutes: 125
# automatic_shard covers NVIDIA steps only; the mirror keeps its
# fixed pytest shards.
parallelism: 2
commands:
- pytest -v -s v1/attention --shard-id=$$BUILDKITE_PARALLEL_JOB --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT
depends_on:
- image-build-amd
source_file_dependencies:
@@ -35,14 +30,9 @@ steps:
- vllm/_aiter_ops.py
- vllm/envs.py
- vllm/platforms/rocm.py
- vllm/models/deepseek_v32/amd/rocm.py
commands:
- pytest -v -s v1/attention --ignore=v1/attention/test_sparse_mla_backends.py --shard-id=$$BUILDKITE_PARALLEL_JOB --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT
- VLLM_ROCM_USE_AITER=1 pytest -v -s v1/attention/test_sparse_mla_backends.py --shard-id=$$BUILDKITE_PARALLEL_JOB --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT
- label: ":nvidia: (B200) V1 Attention"
- label: ":nvidia: (B200) V1 Attention Shard %N"
key: v1-attention-b200
automatic_shard: true
timeout_in_minutes: 80
device: b200-k8s
source_file_dependencies:
@@ -51,16 +41,14 @@ steps:
- vllm/v1/attention
- tests/v1/attention
commands:
- pytest -v -s v1/attention
- pytest -v -s v1/attention --shard-id=$$BUILDKITE_PARALLEL_JOB --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT
parallelism: 2
mirror:
amd:
label: ":amd: (MI355 DPX) V1 Attention Shard %N"
dind: false
device: mi355_dpx
timeout_in_minutes: 125
# automatic_shard covers NVIDIA steps only; the mirror keeps its
# fixed pytest shards.
parallelism: 2
depends_on:
- image-build-amd
source_file_dependencies:
@@ -71,21 +59,6 @@ steps:
- vllm/_aiter_ops.py
- vllm/envs.py
- vllm/platforms/rocm.py
- vllm/models/deepseek_v32/amd/rocm.py
commands:
- pytest -v -s v1/attention --ignore=v1/attention/test_mla_backends.py --ignore=v1/attention/test_sparse_mla_backends.py --shard-id=$$BUILDKITE_PARALLEL_JOB --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT
- VLLM_ROCM_USE_AITER=1 pytest -v -s v1/attention/test_mla_backends.py v1/attention/test_sparse_mla_backends.py --shard-id=$$BUILDKITE_PARALLEL_JOB --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT
- label: ":nvidia: (DGX) Spark FlashInfer NVFP4 XQA"
key: flashinfer-nvfp4-xqa-dgx-spark
timeout_in_minutes: 30
device: dgx-spark
optional: true
num_devices: 1
depends_on:
- arm64-image-build
source_file_dependencies:
- vllm/v1/attention/backends/flashinfer.py
- tests/v1/attention/test_attention_backends.py
commands:
- pytest -v -s v1/attention/test_attention_backends.py -k test_flashinfer_xqa_nvfp4_decode_correctness
- pytest -v -s v1/attention --ignore=v1/attention/test_mla_backends.py --shard-id=$$BUILDKITE_PARALLEL_JOB --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT
- VLLM_ROCM_USE_AITER=1 pytest -v -s v1/attention/test_mla_backends.py --shard-id=$$BUILDKITE_PARALLEL_JOB --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT
+1 -1
View File
@@ -426,7 +426,7 @@ steps:
# Keep TP=2 prefill and TP=1 decode above both hybrid page-size minima.
PREFILL_BLOCK_SIZE: "2048"
DECODE_BLOCK_SIZE: "2048"
VLLM_SERVE_EXTRA_ARGS: "--trust-remote-code,--max-model-len 32768"
VLLM_SERVE_EXTRA_ARGS: "--trust-remote-code,--mamba-cache-mode align,--max-model-len 32768"
commands:
- bash /vllm-workspace/.buildkite/scripts/install-kv-connectors.sh
- bash v1/kv_connector/nixl_integration/run_accuracy_test.sh
@@ -4,8 +4,6 @@ depends_on:
steps:
- label: ":nvidia: (B200) Distributed MooncakeConnector PD accuracy"
key: distributed-mooncakeconnector-pd-accuracy-4-gpus
# Nightly only: 4-GPU B200 pods wait hours behind smaller jobs on per-PR runs.
optional: true
timeout_in_minutes: 10
working_dir: "/vllm-workspace/tests"
# Runs on b200-k8s because it is the only NVIDIA pool that can carry mooncake
@@ -24,7 +22,6 @@ steps:
mirror:
amd:
label: ":amd: (MI355) Distributed MooncakeConnector HIP PD accuracy"
optional: true
dind: false
device: mi355_4
num_devices: 4
+6 -15
View File
@@ -14,7 +14,6 @@ steps:
- tests/distributed
commands:
- pytest -v -s distributed/test_comm_ops.py
- pytest -v -s distributed/test_cpu_all_reduce.py
- pytest -v -s distributed/test_shm_broadcast.py
- pytest -v -s distributed/test_shm_buffer.py
- pytest -v -s distributed/test_shm_storage.py
@@ -78,7 +77,6 @@ steps:
- label: ":nvidia: (L4) Distributed DP Basic"
key: distributed-dp-tests-2-gpus
automatic_shard: true
timeout_in_minutes: 45
working_dir: "/vllm-workspace/tests"
device: l4
@@ -110,7 +108,6 @@ steps:
amd:
label: ":amd: (MI355) Distributed DP Basic"
dind: false
automatic_shard: true
device: mi355_2
timeout_in_minutes: 45
depends_on:
@@ -303,7 +300,6 @@ steps:
- label: ":nvidia: (L4) Distributed DP Extended"
key: distributed-dp-tests-4-gpus
automatic_shard: true
timeout_in_minutes: 45
working_dir: "/vllm-workspace/tests"
device: l4
@@ -333,7 +329,6 @@ steps:
amd:
label: ':amd: (MI355) Distributed DP Extended'
dind: false
automatic_shard: true
device: mi355_4
timeout_in_minutes: 60
working_dir: /vllm-workspace/tests
@@ -513,7 +508,9 @@ steps:
- examples/features/data_parallel/data_parallel_offline.py
- vllm/_aiter_ops.py
- vllm/platforms/rocm.py
- csrc/rocm/quickreduce/
- csrc/custom_quickreduce.cu
- csrc/ops.h
- csrc/torch_bindings.cpp
- vllm/model_executor/layers/
- vllm/entrypoints/llm.py
- vllm/config/parallel.py
@@ -525,17 +522,13 @@ steps:
- tests/distributed/test_rocm_aiter_custom_ar.py
- tests/distributed/test_rocm_quick_reduce.py
- tests/distributed/test_quick_all_reduce.py
- tests/distributed/test_aiter_rms_fallback.py
- tests/v1/e2e/general/test_rocm_aiter_custom_ar.py
- tests/utils.py
- examples/rl/rlhf_async_new_apis.py
- tests/distributed/test_weight_transfer.py
- tests/distributed/test_packed_tensor.py
- tests/v1/attention/test_rocm_mla_dcp_context.py
- vllm/v1/attention/ops/rocm_aiter_mla_prefill.py
commands:
- pytest -v -s tests/distributed/test_context_parallel.py
- pytest -v -s tests/v1/attention/test_rocm_mla_dcp_context.py
- VLLM_ALLOW_INSECURE_SERIALIZATION=1 python3 examples/rl/rlhf_async_new_apis.py
- VLLM_LOGGING_LEVEL=DEBUG python3 examples/features/data_parallel/data_parallel_offline.py --model=Qwen/Qwen1.5-MoE-A2.7B -tp=1 -dp=2 --max-model-len=2048 --all2all-backend=deepep_high_throughput
- VLLM_LOGGING_LEVEL=DEBUG python3 examples/features/data_parallel/data_parallel_offline.py --model=Qwen/Qwen1.5-MoE-A2.7B -tp=1 -dp=2 --max-model-len=2048 --all2all-backend=allgather_reducescatter --disable-nccl-for-dp-synchronization
@@ -546,7 +539,6 @@ steps:
- pytest -v -s tests/v1/e2e/general/test_rocm_aiter_custom_ar.py
- pytest -v -s tests/distributed/test_rocm_quick_reduce.py
- pytest -v -s tests/distributed/test_quick_all_reduce.py
- pytest -v -s tests/distributed/test_aiter_rms_fallback.py
- label: ":nvidia: (B200) Distributed"
key: distributed-tests-2xb200
@@ -640,11 +632,10 @@ steps:
- tests/distributed/
- examples/features/data_parallel/data_parallel_offline.py
commands:
- ./.buildkite/scripts/run-multi-node-test.sh /vllm-workspace/tests 2 2 $IMAGE_TAG "VLLM_TEST_SAME_HOST=0 torchrun --nnodes 2 --nproc-per-node=2 --rdzv_backend=c10d --rdzv_endpoint=192.168.10.10 distributed/test_same_node.py | grep 'Same node test passed' && NUM_NODES=2 torchrun --nnodes 2 --nproc-per-node=2 --rdzv_backend=c10d --rdzv_endpoint=192.168.10.10 distributed/test_node_count.py | grep 'Node count test passed' && python3 ../examples/features/data_parallel/data_parallel_offline.py -dp=2 -tp=1 --dp-num-nodes=2 --dp-node-rank=0 --dp-master-addr=192.168.10.10 --dp-master-port=12345 --enforce-eager --trust-remote-code && VLLM_MULTI_NODE=1 pytest -v -s distributed/test_pipeline_parallel.py" "VLLM_TEST_SAME_HOST=0 torchrun --nnodes 2 --nproc-per-node=2 --rdzv_backend=c10d --rdzv_endpoint=192.168.10.10 distributed/test_same_node.py | grep 'Same node test passed' && NUM_NODES=2 torchrun --nnodes 2 --nproc-per-node=2 --rdzv_backend=c10d --rdzv_endpoint=192.168.10.10 distributed/test_node_count.py | grep 'Node count test passed' && python3 ../examples/features/data_parallel/data_parallel_offline.py -dp=2 -tp=1 --dp-num-nodes=2 --dp-node-rank=1 --dp-master-addr=192.168.10.10 --dp-master-port=12345 --enforce-eager --trust-remote-code"
- ./.buildkite/scripts/run-multi-node-test.sh /vllm-workspace/tests 2 2 $IMAGE_TAG "VLLM_TEST_SAME_HOST=0 torchrun --nnodes 2 --nproc-per-node=2 --rdzv_backend=c10d --rdzv_endpoint=192.168.10.10 distributed/test_same_node.py | grep 'Same node test passed' && NUM_NODES=2 torchrun --nnodes 2 --nproc-per-node=2 --rdzv_backend=c10d --rdzv_endpoint=192.168.10.10 distributed/test_node_count.py | grep 'Node count test passed' && python3 ../examples/features/data_parallel/data_parallel_offline.py -dp=2 -tp=1 --dp-num-nodes=2 --dp-node-rank=0 --dp-master-addr=192.168.10.10 --dp-master-port=12345 --enforce-eager --trust-remote-code && VLLM_MULTI_NODE=1 pytest -v -s distributed/test_multi_node_assignment.py && VLLM_MULTI_NODE=1 pytest -v -s distributed/test_pipeline_parallel.py" "VLLM_TEST_SAME_HOST=0 torchrun --nnodes 2 --nproc-per-node=2 --rdzv_backend=c10d --rdzv_endpoint=192.168.10.10 distributed/test_same_node.py | grep 'Same node test passed' && NUM_NODES=2 torchrun --nnodes 2 --nproc-per-node=2 --rdzv_backend=c10d --rdzv_endpoint=192.168.10.10 distributed/test_node_count.py | grep 'Node count test passed' && python3 ../examples/features/data_parallel/data_parallel_offline.py -dp=2 -tp=1 --dp-num-nodes=2 --dp-node-rank=1 --dp-master-addr=192.168.10.10 --dp-master-port=12345 --enforce-eager --trust-remote-code"
- label: ":nvidia: (L4) Pipeline + Context Parallelism"
key: pipeline-context-parallelism-4-gpus
automatic_shard: true
timeout_in_minutes: 55
working_dir: "/vllm-workspace/tests"
device: l4
@@ -686,7 +677,6 @@ steps:
- label: ":nvidia: (L4) RayExecutorV2"
key: rayexecutorv2-4-gpus
automatic_shard: true
timeout_in_minutes: 45
working_dir: "/vllm-workspace/tests"
device: l4
@@ -700,6 +690,7 @@ steps:
- tests/distributed/test_pipeline_parallel.py
- tests/basic_correctness/models/test_basic_correctness.py
commands:
- export VLLM_USE_RAY_V2_EXECUTOR_BACKEND=1
- export NCCL_CUMEM_HOST_ENABLE=0
- pytest -v -s distributed/test_ray_v2_executor.py
- pytest -v -s distributed/test_ray_v2_executor_e2e.py
@@ -709,7 +700,6 @@ steps:
amd:
label: ':amd: (MI355) RayExecutorV2'
dind: false
automatic_shard: true
device: mi355_4
timeout_in_minutes: 75
working_dir: /vllm-workspace/tests
@@ -725,6 +715,7 @@ steps:
- tests/basic_correctness/models/test_basic_correctness.py
- vllm/platforms/rocm.py
commands:
- export VLLM_USE_RAY_V2_EXECUTOR_BACKEND=1
- pytest -v -s distributed/test_ray_v2_executor.py
- pytest -v -s distributed/test_ray_v2_executor_e2e.py
- pytest -v -s distributed/test_pipeline_parallel.py -k "ray"
+3 -4
View File
@@ -141,7 +141,7 @@ steps:
- tests/v1/e2e/general/
commands:
- pytest -v -s v1/e2e/general/test_kv_sharing_fast_prefill.py
- pytest -v -s v1/e2e/general/test_mamba_prefix_cache.py::test_mamba_prefix_cache_mrv2 v1/e2e/general/test_mamba_prefix_cache.py::test_mamba_prefix_cache_mrv2_async v1/e2e/general/test_mamba_prefix_cache.py::test_mamba_prefix_cache_mrv2_async_flashinfer
- pytest -v -s v1/e2e/general/test_mamba_prefix_cache.py::test_mamba_prefix_cache_mrv2 v1/e2e/general/test_mamba_prefix_cache.py::test_mamba_prefix_cache_mrv2_async
mirror:
amd:
label: ":amd: (MI355 DPX) E2E Core Large Memory"
@@ -164,7 +164,6 @@ steps:
device: h100
num_devices: 2
key: replayssm-e2e
automatic_shard: true
timeout_in_minutes: 90
working_dir: "/vllm-workspace/tests"
source_file_dependencies:
@@ -238,9 +237,9 @@ steps:
- pytest -v -s v1/e2e/test_hybrid_chunked_prefill.py
mirror:
amd:
label: ":amd: (MI355 DPX) V1 E2E Hybrid Chunked Prefill"
label: ":amd: (MI300) V1 E2E Hybrid Chunked Prefill"
dind: false
device: mi355_dpx
device: mi300_1
timeout_in_minutes: 35
depends_on:
- image-build-amd
-7
View File
@@ -31,7 +31,6 @@ steps:
working_dir: "/vllm-workspace/tests"
source_file_dependencies:
- vllm/entrypoints
- vllm/utils/flashinfer.py
- tests/entrypoints/unit_tests
- tests/entrypoints/launchers
commands:
@@ -41,7 +40,6 @@ steps:
amd:
label: ":amd: (MI355 DPX) Entrypoints Unit"
dind: false
automatic_shard: true
device: mi355_dpx
timeout_in_minutes: 50
depends_on:
@@ -122,7 +120,6 @@ steps:
- label: ":nvidia: (H200 MIG 35GB) Entrypoints Integration (OpenAI API chat_completion)"
device: h200_35gb
key: entrypoints-integration-api-server-openai-chat_completion
automatic_shard: true
timeout_in_minutes: 83
working_dir: "/vllm-workspace/tests"
source_file_dependencies:
@@ -144,7 +141,6 @@ steps:
- label: ":nvidia: (H200 MIG 35GB) Entrypoints Integration (API Server Generate)"
device: h200_35gb
key: entrypoints-integration-api-server-generate
automatic_shard: true
timeout_in_minutes: 50
working_dir: "/vllm-workspace/tests"
source_file_dependencies:
@@ -208,7 +204,6 @@ steps:
amd:
label: ":amd: (MI355 DPX) Entrypoints Integration (Speech to Text)"
dind: false
automatic_shard: true
device: mi355_dpx
timeout_in_minutes: 60
working_dir: "/vllm-workspace/tests"
@@ -218,7 +213,6 @@ steps:
- label: ":nvidia: (H200 MIG 35GB) Entrypoints Integration (Multimodal)"
device: h200_35gb
key: entrypoints-integration-multimodal
automatic_shard: true
timeout_in_minutes: 45
working_dir: "/vllm-workspace/tests"
source_file_dependencies:
@@ -254,7 +248,6 @@ steps:
amd:
label: ":amd: (MI355 DPX) Entrypoints Integration (Pooling)"
dind: false
automatic_shard: true
device: mi355_dpx
timeout_in_minutes: 65
working_dir: "/vllm-workspace/tests"
@@ -58,8 +58,6 @@ steps:
- label: ":nvidia: (H100) Elastic EP Scaling"
key: elastic-ep-scaling-test
# Nightly only: 4-GPU H100 pods wait behind smaller jobs on per-PR runs.
optional: true
timeout_in_minutes: 40
device: h100
working_dir: "/vllm-workspace/tests"
+33 -79
View File
@@ -30,10 +30,9 @@ steps:
- vllm/_aiter_ops.py
- vllm/platforms/rocm.py
- label: ":nvidia: (H200 MIG 35GB) Core Operation Kernels"
- label: ":nvidia: (H200 MIG 35GB) Core Operation Kernels Shard %N"
device: h200_35gb
key: kernels-core-operation-test
automatic_shard: true
timeout_in_minutes: 120
source_file_dependencies:
- csrc/
@@ -41,10 +40,10 @@ steps:
- tests/kernels/test_concat_mla_q.py
- tests/kernels/test_fused_qk_norm_rope_gate.py
commands:
- pytest -v -s kernels/core --ignore=kernels/core/test_minimax_reduce_rms.py kernels/test_concat_mla_q.py kernels/test_fused_qk_norm_rope_gate.py
- pytest -v -s kernels/core --ignore=kernels/core/test_minimax_reduce_rms.py kernels/test_concat_mla_q.py kernels/test_fused_qk_norm_rope_gate.py --shard-id=$$BUILDKITE_PARALLEL_JOB --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT
parallelism: 3
mirror:
amd:
parallelism: 3
label: ":amd: (MI300) Core Operation Kernels Shard %N"
dind: false
device: mi300_1
@@ -73,10 +72,10 @@ steps:
num_devices: 2
device: h100
source_file_dependencies:
- csrc/minimax_reduce_rms_kernel.cu
- csrc/minimax_reduce_rms_kernel.h
- csrc/ops.h
- csrc/torch_bindings.cpp
- csrc/libtorch_stable/minimax_reduce_rms_kernel.cu
- csrc/libtorch_stable/minimax_reduce_rms_kernel.h
- csrc/libtorch_stable/ops.h
- csrc/libtorch_stable/torch_bindings.cpp
- vllm/model_executor/layers/minimax_rms_norm/
- vllm/model_executor/layers/mamba/linear/minimax_linear_attn.py
- tests/kernels/core/test_minimax_reduce_rms.py
@@ -123,7 +122,7 @@ steps:
depends_on:
- image-build-amd
source_file_dependencies:
- csrc/fused_deepseek_v4_qnorm_rope_kv_insert_kernel.cu
- csrc/libtorch_stable/fused_deepseek_v4_qnorm_rope_kv_insert_kernel.cu
- vllm/models/deepseek_v4/common/ops/
- tests/kernels/test_fused_deepseek_v4_qnorm_rope_kv_insert.py
- vllm/platforms/rocm.py
@@ -138,15 +137,11 @@ steps:
- csrc/fused_deepseek_v4_qnorm_rope_kv_insert_kernel.cu
- vllm/models/deepseek_v4/common/ops/
- vllm/models/deepseek_v4/nvidia/
- vllm/model_executor/layers/fused_moe/deep_gemm_mega_moe.py
- vllm/model_executor/layers/fused_moe/prepare_megamoe.py
- vllm/utils/deep_gemm.py
- tests/kernels/test_fused_deepseek_v4_qnorm_rope_kv_insert.py
- tests/kernels/moe/test_deep_gemm_mega_moe.py
- tests/models/test_deepseek_v4_mega_moe.py
commands:
- pytest -v -s kernels/test_fused_deepseek_v4_*.py
- pytest -v -s kernels/moe/test_deep_gemm_mega_moe.py
- pytest -v -s models/test_deepseek_v4_mega_moe.py
mirror:
amd:
@@ -235,8 +230,8 @@ steps:
# Catch-all for test files at the tests/kernels root. This job collects
# the whole root so new files are wired by default.
# Files with dedicated jobs elsewhere in this file are excluded via --ignore
# (test_bf16x3_router_gemm_cutedsl runs in Kernels (B200); test_ll_bf16_gemm and
# test_ll_fp32w_gemm run in Gate Linear CuTeDSL Kernels (B200)).
# (test_bf16x3_router_gemm_cutedsl and test_ll_bf16_gemm run in their own jobs /
# Kernels (B200)).
- label: ":nvidia: (B200) Miscellaneous Kernels"
key: kernels-root-misc-test-b200
timeout_in_minutes: 30
@@ -265,7 +260,6 @@ steps:
--ignore=kernels/test_bf16x3_router_gemm_cutedsl.py
--ignore=kernels/test_bf16x3_router_gemm_rocm.py
--ignore=kernels/test_ll_bf16_gemm.py
--ignore=kernels/test_ll_fp32w_gemm.py
--ignore=kernels/test_shuffle_rows.py
# BROKEN on main, pending kernel fixes (B200):
# test_shuffle_rows.py (1: test_shuffle_rows_edge_cases)
@@ -290,6 +284,7 @@ steps:
kernels/test_apply_repetition_penalties.py
kernels/test_awq_int4_to_int8.py
kernels/test_bf16_skinny_gemm.py
kernels/test_bf16x3_router_gemm_rocm.py
kernels/test_cache_kernels.py
kernels/test_compressor_kv_cache.py
kernels/test_cp_gather_fp8.py
@@ -304,12 +299,14 @@ steps:
kernels/test_fused_inv_rope_fp8_quant.py
kernels/test_fused_minimax_m3_qknorm_rope_kv_insert.py
kernels/test_fused_quant_activation.py
kernels/test_gate_linear_rocm_dispatch.py
kernels/test_gemm_rs_ar.py
kernels/test_kpool_decode_update_batched.py
kernels/test_minimax_m3_amd_ops.py
kernels/test_minimax_m3_sparse_attn_fp8_scale.py
kernels/test_onednn.py
kernels/test_relu2_fp8_quant.py
kernels/test_rocm_fp32_router_gemm.py
--deselect=tests/kernels/test_bf16_skinny_gemm.py::test_every_dsv3_routed_shape_is_instantiated
- label: ":nvidia: (L4) Attention Kernels Shard %N"
@@ -319,7 +316,6 @@ steps:
timeout_in_minutes: 30
source_file_dependencies:
- csrc/attention/
- csrc/core/vec_dtypes/
- vllm/v1/attention
# TODO: remove this dependency (https://github.com/vllm-project/vllm/issues/32267)
- vllm/model_executor/layers/attention
@@ -339,7 +335,6 @@ steps:
- image-build-amd
source_file_dependencies:
- csrc/attention/
- csrc/core/vec_dtypes/
- vllm/v1/attention
- vllm/model_executor/layers/attention
- tests/kernels/attention
@@ -412,14 +407,10 @@ steps:
depends_on:
- image-build-amd
source_file_dependencies:
- vllm/models/deepseek_v41/amd/rocm.py
- vllm/models/deepseek_v41/sparse_mla.py
- vllm/v1/attention/backends/mla/rocm_aiter_mla.py
- vllm/v1/attention/ops/rocm_aiter_mla_sparse.py
- vllm/v1/attention/selector.py
- vllm/platforms/rocm.py
- vllm/_aiter_ops.py
- tests/kernels/attention/test_rocm_triton_attn_dsv4.py
- tests/kernels/attention/test_rocm_aiter_mla_decode.py
- tests/kernels/attention/test_rocm_aiter_mla_decode_metadata.py
- tests/kernels/attention/test_rocm_aiter_mla_fp8_support.py
@@ -430,7 +421,6 @@ steps:
- pytest -v -s tests/kernels/attention/test_rocm_aiter_mla_decode.py
- pytest -v -s tests/kernels/attention/test_rocm_aiter_mla_fp8_support.py
- pytest -v -s tests/kernels/attention/test_rocm_aiter_mla_op_registration.py
- pytest -v -s tests/kernels/attention/test_rocm_triton_attn_dsv4.py::test_dsv41_adaptive_mla_swa_metadata_graph_replay
- label: ":nvidia: (L4) Quantization Kernels Shard %N"
key: kernels-quantization-test
@@ -477,7 +467,7 @@ steps:
num_devices: 1
timeout_in_minutes: 50
source_file_dependencies:
- csrc/quantization/w8a8/cutlass/moe/
- csrc/quantization/cutlass_w8a8/moe/
- csrc/moe/
- tests/kernels/moe
- vllm/model_executor/layers/fused_moe/
@@ -495,7 +485,7 @@ steps:
device: mi300_1
timeout_in_minutes: 55
source_file_dependencies:
- csrc/quantization/w8a8/cutlass/moe/
- csrc/quantization/cutlass_w8a8/moe/
- csrc/moe/
- tests/kernels/moe
- vllm/model_executor/layers/fused_moe/
@@ -519,17 +509,14 @@ steps:
source_file_dependencies:
- csrc/mamba/
- tests/kernels/mamba
- vllm/model_executor/layers/mamba/checkpoint.py
- vllm/model_executor/layers/mamba/gdn
- vllm/model_executor/layers/mamba/ops
- vllm/v1/attention/backends/gdn_attn.py
commands:
- pytest -v -s kernels/mamba
mirror:
amd:
label: ":amd: (MI250) Mamba Kernels"
dind: false
automatic_shard: true
device: mi250_1
timeout_in_minutes: 55
depends_on:
@@ -545,6 +532,7 @@ steps:
num_devices: 1
source_file_dependencies:
- cmake/external_projects/deepgemm.cmake
- tools/install_deepgemm.sh
- tools/build_deepgemm_C.py
- tools/setup_deepgemm_pythons.sh
- tools/check_wheel_deepgemm.py
@@ -588,10 +576,11 @@ steps:
# optional: true
source_file_dependencies:
- csrc/quantization/fp4/
- csrc/nvfp4_kv_cache_kernels.cu
- csrc/quantization/w8a8/cutlass/
- csrc/libtorch_stable/nvfp4_kv_cache_kernels.cu
- csrc/libtorch_stable/quantization/fp4/
- csrc/libtorch_stable/quantization/w8a8/cutlass/
- csrc/attention/mla/
- csrc/quantization/w8a8/cutlass/moe/
- csrc/quantization/cutlass_w8a8/moe/
- vllm/model_executor/layers/fusion/
- vllm/model_executor/layers/fused_moe/experts/cutlass_moe.py
- vllm/model_executor/layers/fused_moe/experts/flashinfer_cutlass_moe.py
@@ -605,6 +594,9 @@ steps:
- vllm/v1/attention/backends/mla/flashinfer_mla.py
- vllm/v1/attention/selector.py
- vllm/platforms/cuda.py
- vllm/model_executor/kernels/linear/cute_dsl/ll_bf16.py
- vllm/model_executor/kernels/linear/cute_dsl/_ll_bf16_dotprod.py
- vllm/model_executor/kernels/linear/cute_dsl/_ll_bf16_splitk.py
- vllm/cute_utils/
- vllm/model_executor/layers/mamba/ops/gdn_chunk_cutedsl/
- vllm/model_executor/layers/fused_moe/router/bf16x3_router_gemm_cutedsl.py
@@ -614,9 +606,9 @@ steps:
- vllm/config/attention.py
- vllm/v1/attention/backends/registry.py
- vllm/_custom_ops.py
- csrc/fused_minimax_m3_qknorm_rope_kv_insert_kernel.cu
- csrc/ops.h
- csrc/torch_bindings.cpp
- csrc/libtorch_stable/fused_minimax_m3_qknorm_rope_kv_insert_kernel.cu
- csrc/libtorch_stable/ops.h
- csrc/libtorch_stable/torch_bindings.cpp
- tests/kernels/attention/test_flashinfer_mla_decode.py
- tests/kernels/attention/test_minimax_m3_msa_cutlass_sparse_decode.py
- tests/kernels/attention/test_cache.py
@@ -624,6 +616,7 @@ steps:
- tests/kernels/test_fused_minimax_m3_qknorm_rope_kv_insert.py
- tests/kernels/mamba/test_gdn_prefill_cutedsl.py
- tests/kernels/test_bf16x3_router_gemm_cutedsl.py
- tests/kernels/test_ll_bf16_gemm.py
- tests/kernels/test_top_k_per_row.py
- tests/kernels/quantization/test_silu_mul_nvfp4_quant.py
- tests/kernels/quantization/test_block_fp8.py
@@ -644,6 +637,7 @@ steps:
pytest -v -s tests/kernels/moe/test_nvfp4_moe.py
pytest -v -s tests/kernels/moe/test_ocp_mx_moe.py
pytest -v -s tests/kernels/test_bf16x3_router_gemm_cutedsl.py
pytest -v -s tests/kernels/test_ll_bf16_gemm.py
;;
1)
python3 examples/basic/offline_inference/chat.py
@@ -700,46 +694,6 @@ steps:
- amd-smi
- pytest -v -s kernels/quantization/test_nvfp4_emulation.py -k test_nvfp4_dense_emulation --shard-id=$$BUILDKITE_PARALLEL_JOB --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT
- label: ":nvidia: (B200) Gate Linear CuTeDSL Kernels"
key: kernels-gate-linear-cutedsl-b200
timeout_in_minutes: 15
working_dir: "/vllm-workspace/"
device: b200-k8s
source_file_dependencies:
- vllm/cute_utils/
- vllm/platforms/cuda.py
- vllm/model_executor/layers/fused_moe/router/gate_linear.py
- vllm/model_executor/kernels/linear/cute_dsl/_ll_router_common.py
- vllm/model_executor/kernels/linear/cute_dsl/ll_bf16.py
- vllm/model_executor/kernels/linear/cute_dsl/_ll_bf16_dotprod.py
- vllm/model_executor/kernels/linear/cute_dsl/_ll_bf16_splitk.py
- tests/kernels/test_ll_bf16_gemm.py
- vllm/model_executor/kernels/linear/cute_dsl/ll_fp32w.py
- vllm/model_executor/kernels/linear/cute_dsl/_ll_fp32w_dotprod.py
- tests/kernels/test_ll_fp32w_gemm.py
commands:
- nvidia-smi
- pytest -v -s tests/kernels/test_ll_bf16_gemm.py
- pytest -v -s tests/kernels/test_ll_fp32w_gemm.py
mirror:
amd:
label: ":amd: (MI355) Gate Linear Router Kernels"
dind: false
device: mi355_1
timeout_in_minutes: 20
depends_on:
- image-build-amd
source_file_dependencies:
- vllm/model_executor/layers/fused_moe/router/gate_linear.py
- vllm/model_executor/layers/fused_moe/router/rocm_fp32_router_gemm.py
- vllm/model_executor/layers/fused_moe/router/bf16x3_router_gemm_rocm.py
- vllm/platforms/rocm.py
- tests/kernels/test_rocm_fp32_router_gemm.py
- tests/kernels/test_gate_linear_rocm_dispatch.py
- tests/kernels/test_bf16x3_router_gemm_rocm.py
commands:
- pytest -v -s tests/kernels/test_rocm_fp32_router_gemm.py tests/kernels/test_gate_linear_rocm_dispatch.py tests/kernels/test_bf16x3_router_gemm_rocm.py
- label: ":nvidia: (DGX) Spark b12x Linear Kernels Nightly"
key: b12x-linear-kernels-dgx-spark-nightly
timeout_in_minutes: 60
@@ -772,7 +726,7 @@ steps:
- tests/quantization/reference_mxfp4.py
- tests/quantization/test_auto_round.py
commands:
- uv pip install --system b12x==1.5.1
- uv pip install --system b12x==1.3.0
- pytest -v -s model_executor/kernels/test_b12x_linear.py
model_executor/test_b12x_warmup.py
kernels/quantization/test_block_fp8.py
@@ -899,7 +853,7 @@ steps:
device: h100
num_devices: 2
source_file_dependencies:
- csrc/quantization/w8a8/cutlass/moe/
- csrc/quantization/cutlass_w8a8/moe/
- csrc/moe/
- tests/kernels/moe
- vllm/model_executor/layers/fused_moe/
@@ -938,11 +892,11 @@ steps:
- label: ":nvidia: (B200) FusedMoE Layer Kernels"
key: kernels-fusedmoe-layer-test-2-b200s
timeout_in_minutes: 45
timeout_in_minutes: 90
device: b200-k8s
num_devices: 2
source_file_dependencies:
- csrc/quantization/w8a8/cutlass/moe/
- csrc/quantization/cutlass_w8a8/moe/
- csrc/moe/
- tests/kernels/moe
- vllm/model_executor/layers/fused_moe/
@@ -1055,7 +1009,7 @@ steps:
source_file_dependencies:
- csrc/quantization/fp4/
- csrc/attention/mla/
- csrc/quantization/w8a8/cutlass/moe/
- csrc/quantization/cutlass_w8a8/moe/
- vllm/model_executor/layers/fused_moe/cutlass_moe.py
- vllm/v1/attention/backends/triton_attn.py
- vllm/v1/attention/backends/rocm_attn.py
+16 -27
View File
@@ -15,9 +15,9 @@ steps:
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-small.txt
mirror:
amd:
label: ":amd: (MI355) LM Eval Small Models"
label: ":amd: (MI300) LM Eval Small Models"
dind: false
device: mi355_dpx
device: mi300_1
timeout_in_minutes: 45
depends_on:
- image-build-amd
@@ -105,7 +105,6 @@ steps:
- label: ":nvidia: (H100) LM Eval Large Models"
key: lm-eval-large-models-4xh100
automatic_shard: true
device: h100
optional: true
num_devices: 4
@@ -140,7 +139,6 @@ steps:
- label: ":nvidia: (B200) LM Eval Small Models"
key: lm-eval-small-models-1xb200
automatic_shard: true
timeout_in_minutes: 50
device: b200-k8s
optional: true
@@ -232,13 +230,12 @@ steps:
- vllm/v1/attention/backends/mla/
- vllm/model_executor/layers/attention/sparse_mla_attention.py
- vllm/v1/worker/gpu/
- csrc/hisparse_kernels.cu
- csrc/libtorch_stable/hisparse_kernels.cu
commands:
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-hisparse.txt
- label: ":nvidia: (B200) LM Eval PCP"
key: lm-eval-pcp-4xb200
automatic_shard: true
timeout_in_minutes: 360
device: b200-k8s
num_devices: 4
@@ -374,7 +371,6 @@ steps:
- vllm/tokenizers/
- vllm/v1/attention/
- vllm/v1/core/kv_cache_utils.py
- vllm/v1/core/kv_cache_planning.py
- vllm/v1/spec_decode/metrics.py
- vllm/v1/worker/
- vllm/_aiter_ops.py
@@ -387,7 +383,6 @@ steps:
- label: ":nvidia: (B200) LM Eval Large Models EP"
key: lm-eval-large-models-ep-2xb200
automatic_shard: true
timeout_in_minutes: 60
device: b200-k8s
optional: true
@@ -521,7 +516,6 @@ steps:
- vllm/v1/spec_decode/
- vllm/v1/worker/gpu/spec_decode/
- vllm/v1/core/kv_cache_utils.py
- vllm/v1/core/kv_cache_planning.py
- vllm/model_executor/layers/layernorm.py
- csrc/
- vllm/_aiter_ops.py
@@ -565,7 +559,6 @@ steps:
- vllm/v1/spec_decode/
- vllm/v1/worker/gpu/spec_decode/
- vllm/v1/core/kv_cache_utils.py
- vllm/v1/core/kv_cache_planning.py
- vllm/model_executor/layers/layernorm.py
- csrc/
- vllm/_aiter_ops.py
@@ -609,7 +602,6 @@ steps:
- vllm/v1/spec_decode/
- vllm/v1/worker/gpu/spec_decode/
- vllm/v1/core/kv_cache_utils.py
- vllm/v1/core/kv_cache_planning.py
- vllm/model_executor/layers/layernorm.py
- csrc/
- vllm/_aiter_ops.py
@@ -653,7 +645,6 @@ steps:
- vllm/v1/spec_decode/
- vllm/v1/worker/gpu/spec_decode/
- vllm/v1/core/kv_cache_utils.py
- vllm/v1/core/kv_cache_planning.py
- vllm/model_executor/layers/layernorm.py
- csrc/
- vllm/_aiter_ops.py
@@ -697,7 +688,6 @@ steps:
- vllm/v1/spec_decode/
- vllm/v1/worker/gpu/spec_decode/
- vllm/v1/core/kv_cache_utils.py
- vllm/v1/core/kv_cache_planning.py
- vllm/model_executor/layers/layernorm.py
- csrc/
- vllm/_aiter_ops.py
@@ -707,17 +697,18 @@ steps:
- export VLLM_WORKER_MULTIPROC_METHOD=spawn
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-mi355-nemotron-3-super.txt
- label: ":amd: (MI355) LM Eval Large Models"
- label: ":amd: (MI300) LM Eval Large Models"
key: amd-lm-eval-large-models
dind: false
device: mi355_8
device: mi300_8
optional: true
timeout_in_minutes: 65
depends_on:
- image-build-amd
commands:
- export VLLM_WORKER_MULTIPROC_METHOD=spawn
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-gfx950-large.txt
- export PYTORCH_ROCM_ARCH=gfx942 # Limit Quark compilation to save time
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-mi3xx.txt
source_file_dependencies:
- vllm/_aiter_ops.py
- vllm/platforms/rocm.py
@@ -1194,7 +1185,7 @@ steps:
optional: true
num_devices: 4
source_file_dependencies:
- csrc/gdn/
- csrc/libtorch_stable/gdn/
- tests/evals/qwen4_exp/
- vllm/model_executor/layers/mamba/
- vllm/models/qwen4_exp/
@@ -1212,7 +1203,7 @@ steps:
optional: true
num_devices: 4
source_file_dependencies:
- csrc/gdn/
- csrc/libtorch_stable/gdn/
- tests/evals/qwen4_exp/
- vllm/model_executor/layers/mamba/
- vllm/models/qwen4_exp/
@@ -1233,7 +1224,7 @@ steps:
depends_on:
- image-build-amd
source_file_dependencies:
- csrc/gdn/
- csrc/libtorch_stable/gdn/
- tests/evals/qwen4_exp/
- vllm/model_executor/layers/mamba/
- vllm/models/qwen4_exp/
@@ -1297,9 +1288,9 @@ steps:
- pytest -s -v evals/gsm8k/test_gsm8k_offloading.py -k "nemotron-h-8b or gemma-4-e4b-it"
mirror:
amd:
label: ":amd: (MI355 DPX) LM Eval KV-Offload"
label: ":amd: (MI300) LM Eval KV-Offload"
dind: false
device: mi355_dpx
device: mi300_1
timeout_in_minutes: 55
depends_on:
- image-build-amd
@@ -1344,8 +1335,6 @@ steps:
- label: ":nvidia: (H100) LM Eval KV-Offload Large"
key: kv-offload-large
# Nightly only: 4-GPU H100 pods wait behind smaller jobs on per-PR runs.
optional: true
timeout_in_minutes: 40
device: h100
num_devices: 4
@@ -1359,9 +1348,9 @@ steps:
- pytest -s -v evals/gsm8k/test_gsm8k_offloading.py -k "deepseek-v4-flash"
mirror:
amd:
label: ":amd: (MI355) LM Eval KV-Offload Large"
label: ":amd: (MI300) LM Eval KV-Offload Large"
dind: false
device: mi355_4
device: mi300_4
timeout_in_minutes: 65
depends_on:
- image-build-amd
@@ -1426,10 +1415,10 @@ steps:
# Qwen2-57B has 28 attention heads and cannot run at TP8.
- pytest -s -v test_lm_eval_correctness.py --config-list-file=configs/models-large.txt --tp-size=4
- label: ":amd: (MI355) LM Eval Large Models ROCm Harness"
- label: ":amd: (MI300) LM Eval Large Models ROCm Harness"
key: amd-lm-eval-large-models-rocm-harness
dind: false
device: mi355_8
device: mi300_8
optional: true
timeout_in_minutes: 40
depends_on:
+5 -180
View File
@@ -32,7 +32,6 @@ steps:
amd:
label: ":amd: (MI355 DPX) V1 Sample"
dind: false
automatic_shard: true
device: mi355_dpx
timeout_in_minutes: 70
depends_on:
@@ -164,59 +163,6 @@ steps:
depends_on:
- image-build-amd
# The shared KV Connectors steps above exclude vllm/distributed/kv_transfer/ from
# their triggers, and CUDA skips MoRIIO's mori/aiter-gated tests. Run the whole
# MoRIIO suite on ROCm whenever the connector or its tests change.
- label: ":amd: (MI355 DPX) MoRIIO Connector Unit Tests"
key: amd-moriio-connector-unit-mi355
dind: false
device: mi355_dpx
num_devices: 1
timeout_in_minutes: 30
working_dir: /vllm-workspace/tests
depends_on:
- image-build-amd
source_file_dependencies:
- vllm/distributed/kv_transfer/kv_connector/v1/moriio/
- vllm/distributed/kv_transfer/kv_connector/v1/base.py
- vllm/distributed/kv_transfer/kv_connector/v1/ssm_conv_transfer_utils.py
- examples/disaggregated/disaggregated_serving/moriio_toy_proxy_server.py
- tests/v1/kv_connector/unit/test_moriio_
- tests/v1/kv_connector/unit/utils.py
- requirements/kv_connectors_rocm.txt
commands:
- bash /vllm-workspace/.buildkite/scripts/install-kv-connectors.sh
- export VLLM_WORKER_MULTIPROC_METHOD=spawn
- pytest -v -s -rs v1/kv_connector/unit/test_moriio_*.py
# Single-node MoRIIO WRITE P/D on dummy weights: prefill and decode on separate
# GPUs over xGMI, compared exactly against a direct serve.
- label: ":amd: (MI355) MoRIIO WRITE P/D Dummy-Weight E2E Shard %N"
key: amd-moriio-write-dummy-e2e-mi355
dind: false
device: mi355_2
num_devices: 2
# ~3.5-5 min per case; pytest-shard hashes test ids, so with 20 cases the
# largest of 6 shards holds 5 (~22 min on MI355). Every case picks free ports
# for its servers and proxy and its own compile-cache dir.
parallelism: 6
timeout_in_minutes: 45
working_dir: /vllm-workspace/tests
depends_on:
- image-build-amd
source_file_dependencies:
- vllm/distributed/kv_transfer/kv_connector/v1/moriio/
- vllm/distributed/kv_transfer/kv_connector/v1/base.py
- vllm/distributed/kv_transfer/kv_connector/v1/ssm_conv_transfer_utils.py
- vllm/model_executor/model_loader/dummy_loader.py
- examples/disaggregated/disaggregated_serving/moriio_toy_proxy_server.py
- tests/v1/kv_connector/moriio_integration/
- requirements/kv_connectors_rocm.txt
commands:
- bash /vllm-workspace/.buildkite/scripts/install-kv-connectors.sh
- export VLLM_WORKER_MULTIPROC_METHOD=spawn
- pytest -v -s -rs v1/kv_connector/moriio_integration/test_write_dummy_e2e_rocm.py --shard-id=$$BUILDKITE_PARALLEL_JOB --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT
- label: ":nvidia: (H200 MIG 35GB) V1 Metrics + LM Eval"
device: h200_35gb
key: v1-metrics-lmeval
@@ -267,14 +213,12 @@ steps:
- vllm/v1/
- tests/v1
- tests/watermarking
- examples/disaggregated/mooncake_connector/mooncake_connector_proxy.py
device: cpu-small
commands:
# split the test to avoid interference
- pytest -v -s -m 'cpu_test' v1/core
- pytest -v -s v1/structured_output
- pytest -v -s v1/test_serial_utils.py
- pytest -v -s v1/worker/test_cpu_worker_kv_budget.py
- pytest -v -s v1/test_kv_cache_spec_registry.py
- pytest -v -s v1/cudagraph/test_cudagraph_manager.py
- pytest -v -s -m 'cpu_test' v1/kv_connector/unit
@@ -502,17 +446,6 @@ steps:
- setup.py
- vllm/platforms/rocm.py
- label: ":nvidia: (H200 MIG 18GB) Python 3.14 Smoke Nightly"
key: python-314-smoke-nightly
depends_on: ~
optional: true
timeout_in_minutes: 30
device: h200_18gb
source_file_dependencies:
- tests/standalone_tests/python_only_compile.sh
commands:
- PYTHON_ONLY_COMPILE_PYTHON=3.14 bash standalone_tests/python_only_compile.sh
- label: ":nvidia: (H200 MIG 35GB) Async Engine, Inputs, Utils, Worker"
device: h200_35gb
key: async-engine-inputs-utils-worker
@@ -711,45 +644,21 @@ steps:
source_file_dependencies:
- vllm/v1/attention
- vllm/model_executor/layers
- vllm/model_executor/kernels/linear
- vllm/model_executor/determinism
- tests/v1/determinism/
commands:
- export VLLM_WORKER_MULTIPROC_METHOD=spawn
- pip install pytest-timeout pytest-forked
- pytest -v -s v1/determinism/test_batch_invariance.py
- VLLM_TEST_MODEL=deepseek-ai/DeepSeek-V2-Lite-Chat pytest -v -s v1/determinism/test_batch_invariance.py::test_v1_generation_is_deterministic_across_batch_sizes_with_needle -k TRITON_MLA
mirror:
amd:
label: ":amd: (MI250) Batch Invariance"
dind: false
device: mi250_1
timeout_in_minutes: 60
env:
VLLM_GPU_MEMORY_UTILIZATION: "0.8"
VLLM_TEST_CLEAN_GPU_MEMORY: "1"
depends_on:
- image-build-amd
source_file_dependencies:
- vllm/v1/attention
- vllm/model_executor/layers
- vllm/model_executor/kernels/linear
- vllm/model_executor/determinism
- tests/v1/determinism/
- vllm/_aiter_ops.py
- vllm/platforms/rocm.py
- label: ":nvidia: (H100) Batch Invariance"
key: batch-invariance-h100
automatic_shard: true
timeout_in_minutes: 60
device: h100
source_file_dependencies:
- vllm/v1/attention
- vllm/model_executor/layers
- vllm/lora/ops/triton_ops
- vllm/model_executor/kernels/linear
- vllm/model_executor/determinism
- tests/v1/determinism/
commands:
- export VLLM_WORKER_MULTIPROC_METHOD=spawn
@@ -761,19 +670,15 @@ steps:
- VLLM_TEST_MODEL=Qwen/Qwen3-30B-A3B-Thinking-2507-FP8 pytest -v -s v1/determinism/test_batch_invariance.py::test_v1_generation_is_deterministic_across_batch_sizes_with_needle -k FLASH_ATTN
mirror:
amd:
label: ":amd: (MI300) Batch Invariance"
label: ":amd: (MI250) Batch Invariance"
dind: false
automatic_shard: true
device: mi300_1
timeout_in_minutes: 60
device: mi250_1
timeout_in_minutes: 35
depends_on:
- image-build-amd
source_file_dependencies:
- vllm/v1/attention
- vllm/model_executor/layers
- vllm/lora/ops/triton_ops
- vllm/model_executor/kernels/linear
- vllm/model_executor/determinism
- tests/v1/determinism/
- vllm/_aiter_ops.py
- vllm/platforms/rocm.py
@@ -782,23 +687,15 @@ steps:
- pip install pytest-timeout pytest-forked
- pytest -v -s v1/determinism/test_batch_invariance.py
- pytest -v -s v1/determinism/test_rms_norm_batch_invariant.py
- pytest -v -s v1/determinism/test_lora_shrink_batch_invariant.py
- pytest -v -s v1/determinism/test_cutlass_batch_invariance.py
- pytest -v -s v1/determinism/test_matmul_batch_invariant.py
- VLLM_TEST_MODEL=deepseek-ai/DeepSeek-V2-Lite-Chat pytest -v -s v1/determinism/test_batch_invariance.py::test_v1_generation_is_deterministic_across_batch_sizes_with_needle -k TRITON_MLA
- VLLM_TEST_MODEL=Qwen/Qwen3-30B-A3B-Thinking-2507-FP8 pytest -v -s v1/determinism/test_batch_invariance.py::test_v1_generation_is_deterministic_across_batch_sizes_with_needle -k TRITON_ATTN
- label: ":nvidia: (B200) Batch Invariance"
key: batch-invariance-b200
automatic_shard: true
timeout_in_minutes: 45
device: b200-k8s
source_file_dependencies:
- vllm/v1/attention
- vllm/model_executor/layers
- vllm/lora/ops/triton_ops
- vllm/model_executor/kernels/linear
- vllm/model_executor/determinism
- tests/v1/determinism/
commands:
- export VLLM_WORKER_MULTIPROC_METHOD=spawn
@@ -813,80 +710,6 @@ steps:
- pytest -v -s v1/determinism/test_matmul_batch_invariant.py
- pytest -v -s v1/determinism/test_cutlass_batch_invariance.py
- pytest -v -s v1/determinism/test_online_batch_invariance.py
mirror:
amd:
label: ":amd: (MI355) Batch Invariance"
dind: false
device: mi355_1
timeout_in_minutes: 60
depends_on:
- image-build-amd
source_file_dependencies:
- vllm/v1/attention
- vllm/model_executor/layers
- vllm/lora/ops/triton_ops
- vllm/model_executor/kernels/linear
- vllm/model_executor/determinism
- tests/v1/determinism/
- vllm/_aiter_ops.py
- vllm/platforms/rocm.py
commands:
- export VLLM_WORKER_MULTIPROC_METHOD=spawn
- pip install pytest-timeout pytest-forked
- pytest -v -s v1/determinism/test_batch_invariance.py
- pytest -v -s v1/determinism/test_rms_norm_batch_invariant.py
- pytest -v -s v1/determinism/test_lora_shrink_batch_invariant.py
- pytest -v -s v1/determinism/test_batch_invariance_vlm.py
- VLLM_TEST_MODEL=deepseek-ai/DeepSeek-V2-Lite-Chat pytest -v -s v1/determinism/test_batch_invariance.py::test_v1_generation_is_deterministic_across_batch_sizes_with_needle -k TRITON_MLA
- VLLM_TEST_MODEL=Qwen/Qwen3-30B-A3B-Thinking-2507-FP8 pytest -v -s v1/determinism/test_batch_invariance.py::test_v1_generation_is_deterministic_across_batch_sizes_with_needle -k TRITON_ATTN
- pytest -v -s v1/determinism/test_nvfp4_batch_invariant.py
- pytest -v -s v1/determinism/test_matmul_batch_invariant.py
- pytest -v -s v1/determinism/test_cutlass_batch_invariance.py
- pytest -v -s v1/determinism/test_online_batch_invariance.py
- pytest -v -s v1/determinism/test_mx_linear_batch_invariant.py
- label: ":nvidia: (H100) Batch Invariance (Distributed)"
key: batch-invariance-distributed-4xh100
automatic_shard: true
timeout_in_minutes: 45
device: h100
optional: true
num_devices: 4
source_file_dependencies:
- vllm/distributed/device_communicators/
- vllm/model_executor/determinism
- vllm/model_executor/layers
- vllm/v1/attention
- tests/v1/determinism/
commands:
- export VLLM_WORKER_MULTIPROC_METHOD=spawn
- export VLLM_TEST_CLEAN_GPU_MEMORY=1
- VLLM_TEST_TP_SIZE=4 pytest -v -s v1/determinism/test_batch_invariance.py -k "bs1_vs_bsN or without_batch_invariance_should_fail"
- VLLM_TEST_TP_SIZE=4 VLLM_TEST_DISABLE_CUSTOM_AR=1 pytest -v -s v1/determinism/test_batch_invariance.py -k bs1_vs_bsN
- VLLM_TP_SIZE=4 VLLM_NEEDLE_TRIALS=2 pytest -v -s v1/determinism/test_batch_invariance.py::test_v1_generation_is_deterministic_across_batch_sizes_with_needle -k FLASH_ATTN
- VLLM_TEST_MODEL=allenai/OLMoE-1B-7B-0924 VLLM_TEST_SERVER_ARGS="--data-parallel-size 4 --enable-expert-parallel --max-model-len 4096" pytest -v -s v1/determinism/test_online_batch_invariance.py
mirror:
amd:
label: ":amd: (MI300) Batch Invariance (Distributed)"
dind: false
device: mi300_4
timeout_in_minutes: 45
depends_on:
- image-build-amd
source_file_dependencies:
- vllm/distributed/device_communicators/
- vllm/model_executor/determinism
- vllm/model_executor/layers
- vllm/v1/attention
- tests/v1/determinism/
- vllm/platforms/rocm.py
commands:
- export VLLM_WORKER_MULTIPROC_METHOD=spawn
- export VLLM_TEST_CLEAN_GPU_MEMORY=1
- VLLM_TEST_TP_SIZE=4 pytest -v -s v1/determinism/test_batch_invariance.py -k "bs1_vs_bsN or without_batch_invariance_should_fail"
- VLLM_TEST_TP_SIZE=4 VLLM_TEST_DISABLE_CUSTOM_AR=1 pytest -v -s v1/determinism/test_batch_invariance.py -k bs1_vs_bsN
- VLLM_TP_SIZE=4 VLLM_NEEDLE_TRIALS=2 pytest -v -s v1/determinism/test_batch_invariance.py::test_v1_generation_is_deterministic_across_batch_sizes_with_needle -k TRITON_ATTN
- VLLM_TEST_MODEL=allenai/OLMoE-1B-7B-0924 VLLM_TEST_SERVER_ARGS="--data-parallel-size 4 --enable-expert-parallel --max-model-len 4096" pytest -v -s v1/determinism/test_online_batch_invariance.py
- label: ":nvidia: (H200 MIG 35GB) Acceptance Length (Large Models)"
device: h200_35gb
@@ -898,6 +721,7 @@ steps:
working_dir: "/vllm-workspace/tests"
source_file_dependencies:
- vllm/v1/spec_decode/
- vllm/model_executor/models/mlp_speculator.py
- tests/v1/spec_decode/test_acceptance_length.py
commands:
- export VLLM_ALLOW_INSECURE_SERIALIZATION=1
@@ -913,5 +737,6 @@ steps:
working_dir: /vllm-workspace/tests
source_file_dependencies:
- vllm/v1/spec_decode/
- vllm/model_executor/models/mlp_speculator.py
- tests/v1/spec_decode/test_acceptance_length.py
- vllm/platforms/rocm.py
@@ -5,7 +5,6 @@ steps:
- label: ":nvidia: (H200 MIG 35GB) Model Executor"
device: h200_35gb
key: model-executor
automatic_shard: true
timeout_in_minutes: 60
source_file_dependencies:
- vllm/engine/arg_utils.py
+9 -39
View File
@@ -58,7 +58,6 @@ steps:
- label: ":nvidia: (H200 MIG 35GB) Basic Models (Other)"
device: h200_35gb
key: basic-models-tests-other
automatic_shard: true
timeout_in_minutes: 38
source_file_dependencies:
- vllm/
@@ -69,12 +68,10 @@ steps:
- tests/models/test_deepseek_v4_vl_rocm.py
- tests/models/test_deepseek_v41_decoder_replay_layers.py
- tests/models/test_deepseek_v41_replay_batch.py
- tests/models/test_deepseek_v41_pcp.py
- tests/models/transformers/test_layer_registry_e2e.py
- tests/models/test_deepseek_v41_replay_start.py
- tests/models/test_deepseek_v4_vl_stream_weights.py
commands:
- pytest -v -s models/test_terratorch.py models/transformers/test_backend.py models/test_registry.py models/test_deepseek_v4_vl_rocm.py models/transformers/test_layer_registry_e2e.py models/test_deepseek_v41_replay_start.py models/test_deepseek_v41_decoder_replay_layers.py models/test_deepseek_v41_replay_batch.py models/test_deepseek_v41_pcp.py models/test_deepseek_v4_vl_stream_weights.py
- pytest -v -s models/test_terratorch.py models/transformers/test_backend.py models/test_registry.py models/test_deepseek_v4_vl_rocm.py models/transformers/test_layer_registry_e2e.py models/test_deepseek_v41_replay_start.py models/test_deepseek_v41_decoder_replay_layers.py models/test_deepseek_v41_replay_batch.py
mirror:
amd:
label: ":amd: (MI355) Basic Models (Other)"
@@ -85,20 +82,17 @@ steps:
- image-build-amd
source_file_dependencies:
- vllm/models/deepseek_v4/
- vllm/models/deepseek_v41/
- tests/models/test_deepseek_v4_vl_rocm.py
- tests/models/test_hyv4_rocm.py
- tests/models/transformers/test_layer_registry_e2e.py
- tests/models/test_deepseek_v41_replay_start.py
- tests/models/test_deepseek_v41_decoder_replay_layers.py
- tests/models/test_deepseek_v41_replay_batch.py
- tests/models/test_deepseek_v41_amd_replay_forward.py
- tests/models/test_deepseek_v4_dspark_rocm.py
- tests/models/test_deepseek_v4_rocm_wo_a.py
- tests/models/test_deepseek_v4_rocm_compressor_gemm_fusion.py
- tests/models/test_dsv41_mono_decode.py
commands:
- pytest -v -s models/test_terratorch.py models/transformers/test_backend.py models/test_registry.py models/test_deepseek_v4_vl_rocm.py models/transformers/test_layer_registry_e2e.py models/test_deepseek_v41_replay_start.py models/test_deepseek_v41_decoder_replay_layers.py models/test_deepseek_v41_replay_batch.py models/test_deepseek_v41_amd_replay_forward.py models/test_deepseek_v4_dspark_rocm.py models/test_deepseek_v4_rocm_wo_a.py models/test_deepseek_v4_rocm_compressor_gemm_fusion.py models/test_dsv41_mono_decode.py
- pytest -v -s models/test_terratorch.py models/transformers/test_backend.py models/test_registry.py models/test_deepseek_v4_vl_rocm.py models/transformers/test_layer_registry_e2e.py models/test_deepseek_v41_replay_start.py models/test_deepseek_v41_decoder_replay_layers.py models/test_deepseek_v41_replay_batch.py models/test_deepseek_v4_dspark_rocm.py models/test_deepseek_v4_rocm_wo_a.py models/test_deepseek_v4_rocm_compressor_gemm_fusion.py
- VLLM_ROCM_USE_AITER=1 pytest -v -s models/test_hyv4_rocm.py -m 'not distributed'
- label: ":nvidia: (B200) Inkling"
@@ -137,7 +131,7 @@ steps:
device: b200-k8s
source_file_dependencies:
- vllm/models/kimi_k3/
- csrc/kimi_k3/
- csrc/libtorch_stable/kimi_k3/
- tests/models/kimi_k3/
- tests/kernels/attention/test_kimi_k3_mla_fused_epilogue.py
commands:
@@ -161,11 +155,8 @@ steps:
# cuda.bindings.driver), so it is deliberately left out here.
- pytest -v -s models/kimi_k3
- label: ":nvidia: (4xB200) Kimi K3 Prefix Cache"
- label: ":nvidia: (4xB200) Kimi K3 Prefix Cache Shard %N"
key: kimi-k3-prefix-cache-4xb200
# Nightly only: 4-GPU B200 pods wait hours behind smaller jobs on per-PR runs.
optional: true
automatic_shard: true
timeout_in_minutes: 75
device: b200-k8s
num_devices: 4
@@ -174,8 +165,6 @@ steps:
- vllm/v1/core/
- vllm/distributed/kv_transfer/kv_connector/v1/nixl/
- vllm/distributed/kv_transfer/kv_connector/v1/simple_cpu_offload_connector.py
- vllm/distributed/kv_transfer/kv_connector/v1/mooncake/store/
- vllm/distributed/mooncake_store.py
- vllm/v1/simple_kv_offload/
- vllm/v1/spec_decode/
- vllm/model_executor/layers/mamba/
@@ -183,7 +172,8 @@ steps:
- .buildkite/scripts/install-kv-connectors.sh
commands:
- bash /vllm-workspace/.buildkite/scripts/install-kv-connectors.sh
- pytest -v -s models/kimi_k3/test_prefix_cache.py
- pytest -v -s models/kimi_k3/test_prefix_cache.py --shard-id=$$BUILDKITE_PARALLEL_JOB --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT
parallelism: 2
- label: ":nvidia: (H200) GLM5Next"
key: glm5next-unit-tests
@@ -212,12 +202,12 @@ steps:
device: h200_35gb
source_file_dependencies:
- vllm/models/qwen4_exp/
- vllm/transformers_utils/configs/qwen4_exp.py
- vllm/config/compilation.py
- vllm/v1/attention/backends/short_conv_attn.py
- tests/models/qwen4_exp/
commands:
- pytest -v -s models/qwen4_exp/test_hc_ops.py models/qwen4_exp/test_qsa_prepare.py models/qwen4_exp/test_qsa_reference.py
- pytest -v -s models/qwen4_exp/test_ple.py
mirror:
amd:
label: ":amd: (MI355 DPX) Qwen4 Exp"
@@ -238,6 +228,7 @@ steps:
device: cpu-small
source_file_dependencies:
- vllm/models/qwen4_exp/
- vllm/transformers_utils/configs/qwen4_exp.py
- vllm/config/compilation.py
- vllm/v1/attention/backends/short_conv_attn.py
- tests/models/qwen4_exp/
@@ -256,10 +247,7 @@ steps:
- tests/models/test_vision.py
- tests/models/test_adapters.py
- tests/models/test_qwen3_5_mtp_config.py
- tests/models/test_alice_ai.py
- tests/models/test_alice_ai_config.py
- tests/models/transformers/fusers/
- tests/models/transformers/test_dynamic_rope.py
- tests/model_executor/layers/test_activation.py
- tests/models/test_language_model_cache_is_weak.py
- tests/models/deepseek_v32/test_sequence_parallel.py
@@ -268,22 +256,4 @@ steps:
- tests/models/transformers/test_layer_registry.py
device: cpu-small
commands:
- pytest -v -s models/test_utils.py models/test_vision.py models/test_adapters.py models/test_qwen3_5_mtp_config.py models/transformers/fusers/ models/transformers/test_dynamic_rope.py model_executor/layers/test_activation.py models/test_language_model_cache_is_weak.py models/deepseek_v32/test_sequence_parallel.py models/test_deepseek_v4_fi_moe_ep.py models/test_dspark_mla.py models/transformers/test_layer_registry.py models/test_alice_ai.py models/test_alice_ai_config.py
- label: ':amd: (MI355) DeepSeek-V4.1 Mono Decode Numerics'
key: dsv41-mono-decode-numerics-amd
dind: false
device: mi355_4
num_devices: 4
timeout_in_minutes: 40
working_dir: /vllm-workspace/
depends_on:
- image-build-amd
source_file_dependencies:
- docker/Dockerfile.rocm_base
- vllm/models/deepseek_v41/amd/
- tests/models/test_dsv41_mono_numerics.py
- tests/models/test_dsv41_mono_layers.py
commands:
- pytest -v -s tests/models/test_dsv41_mono_numerics.py
- pytest -v -s tests/models/test_dsv41_mono_layers.py
- pytest -v -s models/test_utils.py models/test_vision.py models/test_adapters.py models/test_qwen3_5_mtp_config.py models/transformers/fusers/ model_executor/layers/test_activation.py models/test_language_model_cache_is_weak.py models/deepseek_v32/test_sequence_parallel.py models/test_deepseek_v4_fi_moe_ep.py models/test_dspark_mla.py models/transformers/test_layer_registry.py
@@ -2,39 +2,6 @@ group: Models - Distributed
depends_on:
- image-build
steps:
- label: ":nvidia: (L4) Qwen4 Exp SP"
key: qwen4-exp-sp-tests-4-gpus
timeout_in_minutes: 30
working_dir: "/vllm-workspace/tests"
device: l4
num_devices: 4
source_file_dependencies:
- vllm/config/parallel.py
- vllm/models/common/ops/sequence_parallel.py
- vllm/models/qwen4_exp/
- vllm/model_executor/layers/vocab_parallel_embedding.py
- vllm/model_executor/models/qwen3_next.py
- tests/models/qwen4_exp/test_ple.py
- tests/models/qwen4_exp/test_tpsp.py
commands:
- pytest -v -s models/qwen4_exp/test_tpsp.py
mirror:
amd:
label: ":amd: (MI355) Qwen4 Exp SP"
dind: false
device: mi355_4
depends_on:
- image-build-amd
source_file_dependencies:
- vllm/config/parallel.py
- vllm/models/common/ops/sequence_parallel.py
- vllm/models/qwen4_exp/
- vllm/model_executor/layers/vocab_parallel_embedding.py
- vllm/model_executor/models/qwen3_next.py
- vllm/platforms/rocm.py
- tests/models/qwen4_exp/test_ple.py
- tests/models/qwen4_exp/test_tpsp.py
- label: ":nvidia: (L4) Distributed Models Shard %N"
key: distributed-model-tests-2-gpus
timeout_in_minutes: 35
+5 -11
View File
@@ -199,29 +199,23 @@ steps:
- vllm/platforms/rocm.py
- tests/models/language/generation_ppl_test
- label: ":nvidia: (H200 MIG 35GB) Language Models (Extended Pooling)"
- label: ":nvidia: (H200 MIG 35GB) Language Models (Extended Pooling) Shard %N"
device: h200_35gb
key: language-models-test-extended-pooling
automatic_shard: true
timeout_in_minutes: 120
parallelism: 4
optional: true
source_file_dependencies:
- vllm/
- "!vllm/distributed/kv_transfer/"
- tests/models/language/pooling
commands:
# Per-test timeout: a hung test fails with a stack dump instead of holding
# the job until Buildkite cancels it, which also blocks the full run's
# coverage collection. A whole shard takes 12-21 minutes.
- pytest -v -s models/language/pooling -m 'not core_model' --timeout=900 --timeout-method=thread
- pytest -v -s models/language/pooling -m 'not core_model' --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT --shard-id=$$BUILDKITE_PARALLEL_JOB
mirror:
amd:
parallelism: 4
commands:
- "pytest -v -s models/language/pooling -m 'not core_model' --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT --shard-id=$$BUILDKITE_PARALLEL_JOB"
label: ":amd: (MI355 DPX) Language Models (Extended Pooling) Shard %N"
label: ":amd: (MI300) Language Models (Extended Pooling) Shard %N"
dind: false
device: mi355_dpx
device: mi300_1
timeout_in_minutes: 95
depends_on:
- image-build-amd
+6 -7
View File
@@ -24,7 +24,6 @@ steps:
- label: ":nvidia: (H200 MIG 18GB) Multimodal Models (Standard) 2: qwen3 + gemma"
key: multi-modal-models-standard-2-qwen3-gemma
automatic_shard: true
timeout_in_minutes: 75
device: h200_18gb
source_file_dependencies:
@@ -163,9 +162,9 @@ steps:
- pytest -v -s models/multimodal/test_mapping.py
mirror:
amd:
label: ":amd: (MI355 DPX) Multimodal Models (Extended Generation 1)"
label: ":amd: (MI300) Multimodal Models (Extended Generation 1)"
dind: false
device: mi355_dpx
device: mi300_1
timeout_in_minutes: 100
depends_on:
- image-build-amd
@@ -182,9 +181,9 @@ steps:
- pytest -v -s models/multimodal/generation_ppl_test/
mirror:
amd:
label: ":amd: (MI355 DPX) Multimodal Models (PPL)"
label: ":amd: (MI300) Multimodal Models (PPL)"
dind: false
device: mi355_dpx
device: mi300_1
timeout_in_minutes: 90
depends_on:
- image-build-amd
@@ -240,9 +239,9 @@ steps:
- pytest -v -s models/multimodal/pooling -m 'not core_model'
mirror:
amd:
label: ":amd: (MI355 DPX) Multimodal Models (Extended Pooling)"
label: ":amd: (MI300) Multimodal Models (Extended Pooling)"
dind: false
device: mi355_dpx
device: mi300_1
timeout_in_minutes: 75
depends_on:
- image-build-amd
-1
View File
@@ -160,7 +160,6 @@ steps:
amd:
label: ':amd: (MI300) PyTorch Compilation Passes'
dind: false
automatic_shard: true
device: mi300_1
timeout_in_minutes: 80
working_dir: /vllm-workspace/tests
+12 -7
View File
@@ -2,27 +2,27 @@ group: Quantization
depends_on:
- image-build
steps:
- label: ":nvidia: (H200 MIG 35GB) Quantization"
- label: ":nvidia: (H200 MIG 35GB) Quantization Shard %N"
device: h200_35gb
key: quantization
automatic_shard: true
timeout_in_minutes: 40
parallelism: 4
env:
VLLM_USE_V2_MODEL_RUNNER: "0"
source_file_dependencies:
- csrc/
- vllm/model_executor/layers/quantization
- vllm/models/deepseek_v41/quant_config.py
- vllm/utils/humming.py
- tests/quantization
commands:
# Pin torchao to the version compatible with the CI PyTorch and CUDA stack.
- uv pip install --system torchao==0.18.0 --index-url https://download.pytorch.org/whl/cu130
- uv pip install --system torchao==0.17.0 --index-url https://download.pytorch.org/whl/cu130
- uv pip install --system conch-triton-kernels
# The SM90-only checkpoint currently contains a removed weight_chan_scale
# parameter. It was not exercised by the previous L4 job.
- VLLM_TEST_FORCE_LOAD_FORMAT=auto pytest -v -s quantization/ --ignore quantization/test_blackwell_moe.py --ignore quantization/test_rocm_moe.py -k 'not test_compressed_tensors_w4a8_fp8'
- VLLM_TEST_FORCE_LOAD_FORMAT=auto pytest -v -s quantization/ --ignore quantization/test_blackwell_moe.py --ignore quantization/test_rocm_moe.py -k 'not test_compressed_tensors_w4a8_fp8' --shard-id=$$BUILDKITE_PARALLEL_JOB --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT
mirror:
amd:
parallelism: 4
label: ":amd: (MI300) Quantization Shard %N"
dind: false
device: mi300_1
@@ -41,6 +41,8 @@ steps:
- tests/rocm/test_moe_weight_replay.py
- tests/rocm/test_moe_padding.py
commands:
# Use the ROCm default instead of the NVIDIA runner override.
- unset VLLM_USE_V2_MODEL_RUNNER
- uv pip install --system torchao==0.17.0
- uv pip install --system conch-triton-kernels
- VLLM_TEST_FORCE_LOAD_FORMAT=auto pytest -v -s quantization/ --ignore quantization/test_blackwell_moe.py --ignore quantization/test_rocm_moe.py --shard-id=$$BUILDKITE_PARALLEL_JOB --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT
@@ -128,6 +130,8 @@ steps:
key: quantized-models-test
automatic_shard: true
timeout_in_minutes: 65
env:
VLLM_USE_V2_MODEL_RUNNER: "0"
source_file_dependencies:
- vllm/model_executor/layers/quantization
- tests/models/quantization
@@ -137,7 +141,6 @@ steps:
amd:
label: ":amd: (MI355 DPX) Quantized Models"
dind: false
automatic_shard: true
device: mi355_dpx
timeout_in_minutes: 80
working_dir: "/vllm-workspace/tests"
@@ -150,4 +153,6 @@ steps:
- vllm/platforms/rocm.py
- vllm/model_executor/model_loader/
commands:
# Use the ROCm default instead of the NVIDIA runner override.
- unset VLLM_USE_V2_MODEL_RUNNER
- pytest -v -s models/quantization
+15 -15
View File
@@ -28,7 +28,6 @@ steps:
amd:
label: ":amd: (MI300) V1 Spec Decode"
dind: false
automatic_shard: true
device: mi300_1
timeout_in_minutes: 50
depends_on:
@@ -137,9 +136,9 @@ steps:
- pytest -v -s v1/e2e/spec_decode/speculators/
mirror:
amd:
label: ":amd: (MI355 DPX) Spec Decode Speculators"
label: ":amd: (MI300) Spec Decode Speculators"
dind: false
device: mi355_dpx
device: mi300_1
timeout_in_minutes: 40
depends_on:
- image-build-amd
@@ -167,9 +166,9 @@ steps:
- pytest -v -s v1/e2e/spec_decode/mtp/gemma4/
mirror:
amd:
label: ":amd: (MI355 DPX) Spec Decode MTP Gemma4"
label: ":amd: (MI300) Spec Decode MTP Gemma4"
dind: false
device: mi355_dpx
device: mi300_1
timeout_in_minutes: 40
depends_on:
- image-build-amd
@@ -198,9 +197,9 @@ steps:
- pytest -v -s v1/e2e/spec_decode/mtp/mimo/
mirror:
amd:
label: ":amd: (MI355 DPX) Spec Decode MTP DeepSeek + MiMo"
label: ":amd: (MI300) Spec Decode MTP DeepSeek + MiMo"
dind: false
device: mi355_dpx
device: mi300_1
timeout_in_minutes: 40
depends_on:
- image-build-amd
@@ -228,9 +227,9 @@ steps:
- pytest -v -s v1/e2e/spec_decode/mtp/qwen3_5/
mirror:
amd:
label: ":amd: (MI355 DPX) Spec Decode MTP Qwen3.5"
label: ":amd: (MI300) Spec Decode MTP Qwen3.5"
dind: false
device: mi355_dpx
device: mi300_1
timeout_in_minutes: 40
depends_on:
- image-build-amd
@@ -387,9 +386,9 @@ steps:
- pytest -v -s v1/spec_decode/test_speculators_correctness.py -m slow_test
mirror:
amd:
label: ":amd: (MI355 DPX) Speculators Correctness Nightly"
label: ":amd: (MI300) Speculators Correctness Nightly"
dind: false
device: mi355_dpx
device: mi300_1
timeout_in_minutes: 40
depends_on:
- image-build-amd
@@ -452,9 +451,9 @@ steps:
- pytest -v -s v1/e2e/spec_decode/acceptance_rates/dflash/ -k "not dflash2"
mirror:
amd:
label: ":amd: (MI355 DPX) Spec Decode AL DFlash Nightly"
label: ":amd: (MI300) Spec Decode AL DFlash Nightly"
dind: false
device: mi355_dpx
device: mi300_1
timeout_in_minutes: 120
commands:
- pytest -v -s v1/e2e/spec_decode/acceptance_rates/dflash/ -k "not dflash2"
@@ -531,9 +530,9 @@ steps:
- pytest -v -s v1/e2e/spec_decode/acceptance_rates/dspark/
mirror:
amd:
label: ":amd: (MI355 DPX) Spec Decode AL DSpark Nightly"
label: ":amd: (MI300) Spec Decode AL DSpark Nightly"
dind: false
device: mi355_dpx
device: mi300_1
timeout_in_minutes: 90
depends_on:
- image-build-amd
@@ -595,6 +594,7 @@ steps:
- vllm/model_executor/models/gemma4_mtp.py
- vllm/model_executor/models/llama.py
- vllm/model_executor/models/llama_eagle3.py
- vllm/model_executor/models/medusa.py
- vllm/model_executor/models/registry.py
- tests/evals/gsm8k/
- tests/utils.py
-1
View File
@@ -1 +0,0 @@
../../.agents/skills/model-microbenchmark
+23 -53
View File
@@ -3,14 +3,13 @@
# This lists cover the "core" components of vLLM that require careful review
/vllm/compilation @zou3519 @youkaichao @ProExpertProg @BoyuanFeng
/vllm/compilation/breakable_cudagraph.py @ZJY0516
/vllm/distributed/kv_transfer @NickLucche @ApostaC @orozery @xuechendi @ivanium @wzhao18
/vllm/distributed/kv_transfer/kv_connector/v1/hisparse @MatthewBonanni @LucasWilkinson
/vllm/lora @jeejeelee
/vllm/model_executor/layers/attention @LucasWilkinson @MatthewBonanni
/vllm/model_executor/layers/fused_moe @mgoin @pavanimajety @zyongye @wzhao18 @bnellnm
/vllm/model_executor/layers/fused_moe @mgoin @pavanimajety @zyongye @wzhao18
/vllm/model_executor/layers/quantization @mgoin @robertgshaw2-redhat @tlrmchlsmth @yewentao256 @pavanimajety @zyongye
/vllm/model_executor/layers/mamba @tdoublep @tomeras91 @ZJY0516
/vllm/model_executor/layers/mamba @tdoublep @tomeras91
/vllm/model_executor/layers/mamba/gdn/qwen_gdn_linear_attn.py @tdoublep @ZJY0516 @vadiklyutiy
/vllm/model_executor/model_loader @22quinn
/vllm/model_executor/determinism/ @yewentao256
@@ -27,7 +26,7 @@
# Any change to the VllmConfig changes can have a large user-facing impact,
# so spam a lot of people
/vllm/config @WoosukKwon @youkaichao @robertgshaw2-redhat @mgoin @tlrmchlsmth @houseroad @yewentao256 @ProExpertProg
/vllm/config/cache.py @heheda12345 @ivanium @ZJY0516
/vllm/config/cache.py @heheda12345 @ivanium
# Config utils
/vllm/config/utils.py @hmellor
@@ -35,48 +34,23 @@
/vllm/utils/argparse_utils.py
# Entrypoints
/vllm/entrypoints/anthropic @chaunceyjiang @mgoin @DarkLight1337
/vllm/entrypoints/cohere @DarkLight1337
/vllm/entrypoints/anthropic @mgoin @DarkLight1337
/vllm/entrypoints/cli @hmellor @mgoin @DarkLight1337
/vllm/entrypoints/mcp @heheda12345
/vllm/entrypoints/openai/chat_completion @chaunceyjiang @DarkLight1337
/vllm/entrypoints/openai/completion @chaunceyjiang @DarkLight1337
/vllm/entrypoints/openai/models @chaunceyjiang @DarkLight1337
/vllm/entrypoints/openai/parser @aarnphm @chaunceyjiang @sfeng33
/vllm/entrypoints/openai/responses @aarnphm @chaunceyjiang @sfeng33
/vllm/entrypoints/pooling @noooop @DarkLight1337
/vllm/entrypoints/rl @aoshen02 @njhill
/vllm/entrypoints/scale_out @aoshen02 @noooop @DarkLight1337
/vllm/entrypoints/openai @aarnphm @chaunceyjiang @DarkLight1337
/vllm/entrypoints/speech_to_text/realtime @njhill
/vllm/entrypoints/speech_to_text @NickLucche
/vllm/entrypoints/speech_to_text/realtime @njhill @NickLucche
/vllm/entrypoints/serve @njhill
/vllm/entrypoints/pooling @noooop
/vllm/entrypoints/serve/sagemaker @DarkLight1337
/vllm/entrypoints/serve @njhill
/vllm/entrypoints/*.py @njhill
/vllm/entrypoints/chat_utils.py @DarkLight1337
/vllm/entrypoints/common @DarkLight1337
/vllm/entrypoints/llm.py @DarkLight1337
/tests/entrypoints @robertgshaw2-redhat @aarnphm @NickLucche @AndreasKaratzas
/tests/entrypoints/anthropic @mgoin @DarkLight1337
/tests/entrypoints/cohere @DarkLight1337
/tests/entrypoints/cli @hmellor @mgoin @DarkLight1337
/tests/entrypoints/openai/chat_completion @chaunceyjiang @DarkLight1337
/tests/entrypoints/openai/completion @chaunceyjiang @DarkLight1337
/tests/entrypoints/openai/models @chaunceyjiang @DarkLight1337
/tests/entrypoints/openai/parser @aarnphm @chaunceyjiang @sfeng33
/tests/entrypoints/openai/responses @aarnphm @chaunceyjiang @sfeng33
/tests/entrypoints/pooling @noooop @DarkLight1337
/tests/entrypoints/rl @aoshen02 @njhill
/tests/entrypoints/scale_out @noooop @DarkLight1337
/tests/entrypoints/speech_to_text @NickLucche
/tests/entrypoints/speech_to_text/realtime @njhill @NickLucche
/tests/entrypoints/serve @njhill
/tests/entrypoints/serve/sagemaker @DarkLight1337
/tests/entrypoints/llm @DarkLight1337
/tests/entrypoints/llm/test_struct_output_generate.py @mgoin @aarnphm @arpera
# Rust Frontend
/rust/ @BugenZhao @njhill
/rust/src/bench @esmeetu
/rust/src/server/src/routes/inference @aoshen02 @BugenZhao @njhill
/tools/build_rust.sh @BugenZhao @njhill
/rust-toolchain.toml @BugenZhao @njhill
/.buildkite/test_areas/rust* @BugenZhao @njhill
@@ -90,12 +64,6 @@
/vllm/tool_parsers @aarnphm @chaunceyjiang @sfeng33 @bbrowning
/vllm/parser @aarnphm @chaunceyjiang @sfeng33 @bbrowning
/tests/tokenizers_ @DarkLight1337 @njhill
/tests/renderers @DarkLight1337 @njhill
/tests/reasoning @aarnphm @chaunceyjiang @sfeng33 @bbrowning
/tests/tool_parsers @aarnphm @chaunceyjiang @sfeng33 @bbrowning
/tests/tool_use @aarnphm @chaunceyjiang @sfeng33 @bbrowning
# vLLM V1
/vllm/v1/attention @LucasWilkinson @MatthewBonanni
/vllm/v1/attention/backend.py @WoosukKwon @zhuohan123 @youkaichao @alexm-redhat @njhill
@@ -103,18 +71,17 @@
/vllm/v1/attention/backends/flashinfer.py @mgoin @pavanimajety @vadiklyutiy @wzhao18
/vllm/v1/attention/backends/triton_attn.py @tdoublep
/vllm/v1/attention/backends/gdn_attn.py @ZJY0516 @vadiklyutiy
/vllm/v1/core @WoosukKwon @robertgshaw2-redhat @njhill @ywang96 @alexm-redhat @heheda12345 @ApostaC @orozery @ivanium @ZJY0516
/vllm/v1/core @WoosukKwon @robertgshaw2-redhat @njhill @ywang96 @alexm-redhat @heheda12345 @ApostaC @orozery @ivanium
/vllm/v1/sample @22quinn @houseroad @njhill
/vllm/v1/spec_decode @benchislett @luccafong @MatthewBonanni
/vllm/v1/structured_output @mgoin @aarnphm @benchislett @arpera
/vllm/v1/kv_cache_interface.py @heheda12345 @ivanium @ZJY0516
/vllm/v1/kv_cache_interface.py @heheda12345 @ivanium
/vllm/v1/kv_offload @ApostaC @orozery
/vllm/v1/simple_kv_offload @ivanium
/vllm/v1/hisparse @MatthewBonanni @LucasWilkinson
/vllm/v1/engine @njhill
/vllm/v1/executor @njhill
/vllm/v1/worker @njhill
/vllm/v1/worker/mamba_utils.py @njhill @ZJY0516
/vllm/v1/worker/kv_connector_model_runner_mixin.py @orozery @NickLucche @ivanium
/vllm/v1/worker/encoder_cudagraph.py @shen-shanshan
/vllm/v1/worker/encoder_cudagraph_defs.py @shen-shanshan
@@ -122,7 +89,6 @@
# Model runner V2
/vllm/v1/worker/gpu @WoosukKwon @njhill @yewentao256
/vllm/v1/worker/gpu/kv_connector.py @orozery
/vllm/v1/worker/gpu/model_states/mamba_hybrid.py @WoosukKwon @njhill @yewentao256 @ZJY0516
# CI & building
/.buildkite @Harry-Chen @khluu
@@ -135,23 +101,27 @@
/tests/distributed/test_multi_node_assignment.py @youkaichao
/tests/distributed/test_pipeline_parallel.py @youkaichao
/tests/distributed/test_same_node.py @youkaichao
/tests/evals @mgoin @vadiklyutiy @AndreasKaratzas @ZJY0516
/tests/entrypoints @DarkLight1337 @robertgshaw2-redhat @aarnphm @NickLucche @AndreasKaratzas
/tests/evals @mgoin @vadiklyutiy @AndreasKaratzas
/tests/kernels @mgoin @tlrmchlsmth @WoosukKwon @yewentao256 @zyongye @AndreasKaratzas
/tests/kernels/ir @ProExpertProg @tjtanaa
/tests/kernels/moe @bnellnm
/tests/models @DarkLight1337 @ywang96 @AndreasKaratzas
/tests/multimodal @DarkLight1337 @ywang96 @NickLucche @shen-shanshan
/tests/multimodal/media @Isotr0py
/tests/quantization @mgoin @robertgshaw2-redhat @yewentao256 @pavanimajety @zyongye @AndreasKaratzas
/tests/entrypoints/llm/test_struct_output_generate.py @mgoin @aarnphm @arpera
/tests/v1/structured_output @mgoin @aarnphm @arpera
/tests/v1/core @WoosukKwon @robertgshaw2-redhat @njhill @ywang96 @alexm-redhat @heheda12345 @ApostaC @orozery @ivanium @ZJY0516
/tests/v1/core @WoosukKwon @robertgshaw2-redhat @njhill @ywang96 @alexm-redhat @heheda12345 @ApostaC @orozery @ivanium
/tests/lora @jeejeelee
/tests/models/language/generation/test_hybrid.py @tdoublep @tomeras91 @ZJY0516
/tests/models/language/generation/test_hybrid.py @tdoublep @tomeras91
/tests/v1/kv_connector/nixl_integration @NickLucche
/tests/v1/kv_connector @ApostaC @orozery @ivanium @wzhao18
/tests/v1/kv_offload @ApostaC @orozery
/tests/v1/simple_kv_offload @ivanium
/tests/v1/determinism @yewentao256
/tests/reasoning @aarnphm @chaunceyjiang @sfeng33 @bbrowning
/tests/tool_parsers @aarnphm @chaunceyjiang @sfeng33 @bbrowning
/tests/tool_use @aarnphm @chaunceyjiang @sfeng33 @bbrowning
# Transformers modeling backend
/vllm/model_executor/models/transformers @hmellor
@@ -165,9 +135,6 @@
.readthedocs.yaml @hmellor
mkdocs.yaml @hmellor
# Model recipes
/recipes @esmeetu
# Linting
/tools/pre_commit/.markdownlint.yaml @hmellor
.pre-commit-config.yaml @hmellor
@@ -256,12 +223,15 @@ mkdocs.yaml @hmellor
/examples/pooling @noooop
/docs/models/pooling_models @noooop
/tests/models/*/pooling* @noooop
/tests/entrypoints/pooling @noooop
/vllm/config/pooler.py @noooop
/vllm/model_executor/layers/pooler @noooop
# Reinforcement learning
/vllm/distributed/weight_transfer @aoshen02
/vllm/entrypoints/rl @njhill @aoshen02
/tests/distributed/test_weight_transfer.py @aoshen02
/tests/entrypoints/rl @DarkLight1337 @robertgshaw2-redhat @njhill @aarnphm @NickLucche @AndreasKaratzas @aoshen02
/tests/v1/worker/test_gpu_worker_weight_transfer.py @aoshen02
/examples/rl @aoshen02
/docs/training/rlhf.md @aoshen02
+2 -1
View File
@@ -494,7 +494,8 @@ pull_request_rules:
- files~=^tests/v1/e2e/spec_decode/
- files~=^examples/features/speculative_decoding/
- files~=^vllm/model_executor/models/.*eagle.*\.py
- files~=^vllm/transformers_utils/configs/eagle\.py
- files=vllm/model_executor/models/mlp_speculator.py
- files~=^vllm/transformers_utils/configs/(eagle|medusa|mlp_speculator)\.py
actions:
label:
add:
+1
View File
@@ -279,6 +279,7 @@ jobs:
"speculative-decoding": {
keywords: [
{ term: "eagle", searchIn: "title" },
{ term: "medusa", searchIn: "title" },
],
substrings: [
{ term: "speculative decoding", searchIn: "title" },
-28
View File
@@ -1,28 +0,0 @@
name: Recipes
# Validate recipe changes against the renderer that publishes them
# (recipes.vllm.ai). The check logic lives in vllm-project/recipes and tracks
# its main branch, so a PR is held to the same validation that runs when
# vllm-project/recipes picks up this directory after merge.
on:
pull_request:
paths:
- 'recipes/**'
- '.github/workflows/recipes.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
permissions:
contents: read
jobs:
validate:
name: Recipes
if: github.repository_owner == 'vllm-project'
uses: vllm-project/recipes/.github/workflows/validate-recipes.yml@main
with:
source: vllm
path: recipes
+16 -14
View File
@@ -18,8 +18,22 @@ jobs:
if: >-
github.repository_owner == 'vllm-project' &&
github.event.issue.pull_request &&
(startsWith(github.event.comment.body, '/ci') ||
startsWith(github.event.comment.body, '/amd-ci'))
(github.event.comment.body == '/ci run' ||
github.event.comment.body == '/ci run all' ||
github.event.comment.body == '/ci run nightly' ||
github.event.comment.body == '/ci run --allow-stale' ||
github.event.comment.body == '/ci run all --allow-stale' ||
github.event.comment.body == '/ci run nightly --allow-stale' ||
github.event.comment.body == '/ci retry' ||
github.event.comment.body == '/ci cancel' ||
github.event.comment.body == '/amd-ci run' ||
github.event.comment.body == '/amd-ci run all' ||
github.event.comment.body == '/amd-ci run nightly' ||
github.event.comment.body == '/amd-ci run --allow-stale' ||
github.event.comment.body == '/amd-ci run all --allow-stale' ||
github.event.comment.body == '/amd-ci run nightly --allow-stale' ||
github.event.comment.body == '/amd-ci retry' ||
github.event.comment.body == '/amd-ci cancel')
runs-on: [self-hosted, linux, x64, vllm-runners]
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
@@ -39,16 +53,4 @@ jobs:
BUILDKITE_ORGANIZATION: vllm
BUILDKITE_PIPELINE: ci
CI_TRUSTED_USERS: ${{ vars.CI_TRUSTED_USERS }}
GH_BRANCH_UPDATE_TOKEN: ${{ secrets.CI_BRANCH_UPDATE_TOKEN }}
GH_TOKEN: ${{ github.token }}
- name: Report failure on the PR
if: failure()
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
with:
script: |
const runUrl = `${context.serverUrl}/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId}`;
await github.rest.issues.createComment({
...context.repo,
issue_number: context.issue.number,
body: `❌ @${context.payload.comment.user.login}, your CI command failed with an unexpected error, so CI may not have started. See the [workflow run](${runUrl}) for details.`,
});
+8 -138
View File
@@ -58,12 +58,9 @@ AMD_CI_COMMANDS = frozenset(
RETRY_COMMANDS = frozenset({COMMAND_RETRY_FAILED, COMMAND_RETRY_AMD_FAILED})
CANCEL_COMMANDS = frozenset({COMMAND_CANCEL_CI, COMMAND_CANCEL_AMD_CI})
ALL_CI_COMMANDS = UPSTREAM_CI_COMMANDS | AMD_CI_COMMANDS
CI_COMMAND_PREFIXES = frozenset({"/ci", "/amd-ci"})
CI_AUTHORIZED_COMMENT_MARKER = "<!-- vllm-ci-authorized -->"
READY_LABELS = {"ready", "ready-run-all-tests"}
TRUSTED_PERMISSIONS = {"admin", "maintain", "write"}
BRANCH_UPDATE_POLL_SECONDS = 3
BRANCH_UPDATE_POLL_ATTEMPTS = 40
ACTIVE_BUILD_STATES = {
"blocked",
"creating",
@@ -261,13 +258,6 @@ class GitHubClient:
raise ApiError(None, "GitHub returned an invalid commits-behind count.")
return behind
def update_branch(self, number: int, expected_head_sha: str) -> None:
self._request(
self._repo_path(f"/pulls/{number}/update-branch"),
body={"expected_head_sha": expected_head_sha},
method="PUT",
)
def list_pulls_for_commit(self, commit: str) -> list[dict[str, Any]]:
commit = urllib.parse.quote(commit, safe="")
response = self._request(self._repo_path(f"/commits/{commit}/pulls"))
@@ -471,17 +461,11 @@ class BuildkiteClient:
def parse_command(body: str) -> str | None:
body = body.strip()
if body in ALL_CI_COMMANDS:
return body
return None
def is_ci_command_attempt(body: str) -> bool:
words = body.split(maxsplit=1)
return bool(words) and words[0].casefold() in CI_COMMAND_PREFIXES
def pipeline_for_command(
command: str,
*,
@@ -621,16 +605,6 @@ def blocks_new_run(command: str, build: Mapping[str, Any]) -> bool:
return is_active_build(build)
def list_commit_builds(
buildkite: BuildkiteClient, command: str, commit: str
) -> list[dict[str, Any]]:
builds = buildkite.list_builds(commit)
if command not in AMD_CI_COMMANDS:
return builds
# bootstrap-amd.sh exits before uploading any test steps for PR webhook builds.
return [build for build in builds if build.get("source") != "webhook"]
def select_latest_build(
builds: Sequence[dict[str, Any]],
pr_number: int,
@@ -797,11 +771,10 @@ def notify_authorized(
(
f"✅ @{author}, CI is now available for this PR.\n\n"
"- `/ci run` starts upstream CI; `/amd-ci run` starts AMD CI only.\n"
"- Run commands test your branch with every commit currently on its "
"upstream target branch. If it is behind, the bot merges the target "
"branch into it first (pull before pushing again). Append "
"`--allow-stale` to a run command to test an outdated branch as is, "
"at your own risk.\n"
"- Your branch must contain every commit currently on its upstream "
"target branch. Merge or rebase onto the latest target branch, then "
"rerun the command. Append `--allow-stale` to a run command to test "
"an outdated branch at your own risk.\n"
"- `/ci retry` retries failed jobs in the CI build for the current "
"PR head. If the current head has no CI build, it starts a new CI "
"build for the current head containing only jobs that failed in "
@@ -849,65 +822,10 @@ def resolve_workflow_run_pr(
return find_matching_pr(github.list_pulls_for_commit(head_sha))
def merge_base_into_pr(
github: GitHubClient,
branch_updater: GitHubClient,
pr: Mapping[str, Any],
command: str,
lag: str,
) -> tuple[dict[str, Any], str]:
base_ref = pr["base"]["ref"]
def merge_failed(reason: str) -> CiPreparationError:
return CiPreparationError(
f"{lag} The bot could not merge upstream `{base_ref}` into this "
f"branch ({reason}). No new CI build was started. Either merge or "
f"rebase onto the latest `{base_ref}` yourself and rerun `{command}`, "
f"or use `{command}{ALLOW_STALE_SUFFIX}` to test this branch as is."
)
try:
branch_updater.update_branch(pr["number"], pr["head"]["sha"])
for _ in range(BRANCH_UPDATE_POLL_ATTEMPTS):
time.sleep(BRANCH_UPDATE_POLL_SECONDS)
current_pr = github.get_pr(pr["number"])
if current_pr["head"]["sha"] == pr["head"]["sha"]:
continue
try:
behind = github.get_commits_behind_base(
base_ref, current_pr["head"]["sha"]
)
except ApiError as error:
# A fork's new head is briefly unknown to the base repository.
if error.status == 404:
continue
raise
break
else:
raise merge_failed("the merge commit did not appear in time")
if (
current_pr["state"] != "open"
or current_pr["base"]["ref"] != base_ref
or behind
):
raise merge_failed("the PR changed during the merge")
except ApiError as error:
raise merge_failed(str(error)) from error
note = (
f"ℹ️ {lag} Merged upstream `{base_ref}` into this branch as "
f"`{current_pr['head']['sha'][:12]}` before starting CI. "
"Pull the branch before pushing more changes."
)
print(note)
return current_pr, note
def prepare_pr_for_ci(
github: GitHubClient,
pr: Mapping[str, Any],
command: str,
branch_updater: GitHubClient | None = None,
) -> tuple[dict[str, Any], str]:
base_ref = pr["base"]["ref"]
no_action = "No new CI build was started."
@@ -943,8 +861,6 @@ def prepare_pr_for_ci(
print(warning)
return current_pr, warning
if behind and branch_updater is not None:
return merge_base_into_pr(github, branch_updater, current_pr, command, lag)
if behind:
raise CiPreparationError(
f"{lag} Your branch must contain every commit currently on upstream "
@@ -964,8 +880,6 @@ def handle_run_ci(
command: str,
github: GitHubClient,
pr: Mapping[str, Any],
branch_updater: GitHubClient | None = None,
allow_repeat_run: bool = False,
) -> str:
ci_name = ci_name_for_command(command)
duplicate_builds = buildkite.list_builds(
@@ -978,7 +892,7 @@ def handle_run_ci(
f"{ci_name} was already requested by this comment: {duplicate['web_url']}"
)
current_builds = list_commit_builds(buildkite, command, pr["head"]["sha"])
current_builds = buildkite.list_builds(pr["head"]["sha"])
active_build = next(
(
build
@@ -992,19 +906,7 @@ def handle_run_ci(
f"{ci_name} is already running for this commit: {active_build['web_url']}"
)
tested_build = select_latest_build(
[build for build in current_builds if build.get("state") != "canceled"],
pr["number"],
)
if tested_build and not allow_repeat_run:
raise CiPreparationError(
f"{ci_name} already ran on this commit: {tested_build['web_url']}. "
"No new CI build was started. Push new changes to run CI again, use "
f"`{retry_command_for_command(command)}` to rerun failed jobs, or ask "
"a reviewer with write access to rerun it."
)
current_pr, warning = prepare_pr_for_ci(github, pr, command, branch_updater)
current_pr, warning = prepare_pr_for_ci(github, pr, command)
build = buildkite.create_build(
create_build_payload(
@@ -1033,7 +935,7 @@ def handle_retry_failed(
ci_name = ci_name_for_command(command)
run_command = run_command_for_command(command)
retry_command = retry_command_for_command(command)
builds = list_commit_builds(buildkite, command, pr["head"]["sha"])
builds = buildkite.list_builds(pr["head"]["sha"])
build = select_latest_build(builds, pr["number"])
if build:
metadata = build.get("meta_data") or {}
@@ -1184,34 +1086,11 @@ def handle_cancel_ci(
)
def reply_unrecognized_command(
event: Mapping[str, Any],
github: GitHubClient,
) -> None:
if "pull_request" not in event["issue"]:
return
issue_number = event["issue"]["number"]
comment_id = event["comment"]["id"]
actor = event["comment"]["user"]["login"]
if is_already_handled(github, issue_number, comment_id):
print(f"Comment {comment_id} was already handled.")
return
add_reaction_safely(github, comment_id, "-1")
commands = ", ".join(f"`{command}`" for command in sorted(ALL_CI_COMMANDS))
github.add_comment(
issue_number,
f"❌ @{actor}, that is not a recognized CI command, so no CI was "
"started. Post a new comment containing only one of: "
f"{commands}.\n\n{command_comment_marker(comment_id)}",
)
def run(
event: Mapping[str, Any],
github: GitHubClient,
buildkite: BuildkiteClient,
trusted_users_value: str = "",
branch_updater: GitHubClient | None = None,
) -> None:
command = parse_command(event["comment"]["body"])
if not command or "pull_request" not in event["issue"]:
@@ -1274,8 +1153,6 @@ def run(
command=command,
github=github,
pr=pr,
branch_updater=branch_updater,
allow_repeat_run=is_trusted_permission(permission),
)
elif command in RETRY_COMMANDS:
message = handle_retry_failed(
@@ -1340,11 +1217,8 @@ def main() -> None:
)
return
body = event["comment"]["body"]
command = parse_command(body)
command = parse_command(event["comment"]["body"])
if not command:
if is_ci_command_attempt(body):
reply_unrecognized_command(event, github)
return
pipeline = pipeline_for_command(
@@ -1357,15 +1231,11 @@ def main() -> None:
os.environ.get("BUILDKITE_ORGANIZATION", "vllm"),
pipeline,
)
branch_update_token = os.environ.get("GH_BRANCH_UPDATE_TOKEN", "")
run(
event,
github,
buildkite,
os.environ.get("CI_TRUSTED_USERS", ""),
GitHubClient(branch_update_token, os.environ["GITHUB_REPOSITORY"])
if branch_update_token
else None,
)
@@ -32,12 +32,10 @@ from run_ci_command import (
has_trusted_approval,
is_active_build,
is_build_for_pr,
is_ci_command_attempt,
notify_authorized,
parse_command,
parse_trusted_users,
pipeline_for_command,
reply_unrecognized_command,
resolve_workflow_run_pr,
run,
select_latest_build,
@@ -144,23 +142,6 @@ class FakeGitHub:
self.comments.append(body)
class FakeBranchUpdater:
def __init__(self, github: FakeGitHub, error: ApiError | None = None) -> None:
self.calls: list[tuple[int, str]] = []
self.error = error
self.github = github
def update_branch(self, number: int, expected_head_sha: str) -> None:
self.calls.append((number, expected_head_sha))
if self.error:
raise self.error
self.github.prs[number] = {
**self.github.prs[number],
"head": {**self.github.prs[number]["head"], "sha": "merge-commit-sha"},
}
self.github.behind = 0
class FakeBuildkite:
def __init__(
self,
@@ -359,46 +340,14 @@ class RunCiCommandTest(unittest.TestCase):
self.assertIsNone(parse_command("/ci run please"))
self.assertIsNone(parse_command("/ci run all please"))
self.assertIsNone(parse_command("/ci cancel please"))
self.assertIsNone(parse_command(" /ci run"))
self.assertIsNone(parse_command("/amd-ci run please"))
self.assertIsNone(parse_command("/amd-ci retry "))
self.assertIsNone(parse_command("/AMD-CI run"))
self.assertIsNone(parse_command("/amdci run"))
self.assertIsNone(parse_command("/ci run --allow-stale all"))
self.assertIsNone(parse_command("/amd-ci run --allow-stale please"))
def test_surrounding_whitespace_is_ignored(self) -> None:
self.assertEqual(parse_command("/ci run\r\n"), COMMAND_RUN_CI)
self.assertEqual(parse_command(" /ci run "), COMMAND_RUN_CI)
self.assertEqual(parse_command("/amd-ci retry\n"), COMMAND_RETRY_AMD_FAILED)
def test_ci_command_attempts_are_detected(self) -> None:
for body in ("/ci", "/ci rn", "/CI run", "/amd-ci run please", " /ci\nrun"):
with self.subTest(body=body):
self.assertTrue(is_ci_command_attempt(body))
for body in ("", "LGTM", "/cifoo", "/amdci run", "please /ci run"):
with self.subTest(body=body):
self.assertFalse(is_ci_command_attempt(body))
def test_unrecognized_command_gets_feedback(self) -> None:
github = FakeGitHub()
reply_unrecognized_command(make_event("/ci rn", actor="author"), github)
self.assertEqual(github.reactions, ["-1"])
self.assertEqual(len(github.comments), 1)
self.assertIn(
"@author, that is not a recognized CI command", github.comments[0]
)
self.assertIn(f"`{COMMAND_RUN_CI}`", github.comments[0])
self.assertIn("<!-- vllm-ci-command:99 -->", github.comments[0])
def test_unrecognized_command_feedback_is_not_repeated(self) -> None:
github = FakeGitHub(comments=["❌ earlier\n\n<!-- vllm-ci-command:99 -->"])
reply_unrecognized_command(make_event("/ci rn"), github)
self.assertEqual(len(github.comments), 1)
self.assertEqual(github.reactions, [])
def test_commands_select_only_their_configured_pipeline(self) -> None:
cases = (
(COMMAND_RUN_CI, "ci"),
@@ -633,82 +582,6 @@ class RunCiCommandTest(unittest.TestCase):
)
self.assertIn(command, github.comments[0])
def test_run_merges_base_into_outdated_branch_before_building(self) -> None:
github = FakeGitHub(behind=5)
updater = FakeBranchUpdater(github)
buildkite = FakeBuildkite([[], []])
with patch("run_ci_command.time.sleep"):
run(make_event(COMMAND_RUN_CI), github, buildkite, "", updater)
self.assertEqual(updater.calls, [(42, "0123456789abcdef")])
self.assertEqual(buildkite.created_builds[0]["commit"], "merge-commit-sha")
self.assertTrue(github.comments[0].startswith("✅ "))
self.assertIn("5 commits behind", github.comments[0])
self.assertIn("Merged upstream `main`", github.comments[0])
def test_only_committers_can_rerun_a_commit_that_already_ran_ci(self) -> None:
finished_build = {
"created_at": "2026-07-28T01:00:00Z",
"pull_request": {"id": 42},
"state": "failed",
"web_url": "https://buildkite.example/builds/122",
}
for actor, permission, command, allowed in (
("author", "read", COMMAND_RUN_CI, False),
("author", "read", f"{COMMAND_RUN_CI} --allow-stale", False),
("reviewer", "write", COMMAND_RUN_CI, True),
):
with self.subTest(actor=actor, command=command):
github = FakeGitHub(
behind=5,
permission=permission,
pr=make_pr(labels=[{"name": "ready"}]),
)
updater = FakeBranchUpdater(github)
buildkite = FakeBuildkite([[], [finished_build]])
with patch("run_ci_command.time.sleep"):
run(make_event(command, actor), github, buildkite, "", updater)
self.assertEqual(bool(buildkite.created_builds), allowed)
if not allowed:
self.assertEqual(updater.calls, [])
self.assertIn("CI already ran on this commit", github.comments[0])
self.assertIn("`/ci retry`", github.comments[0])
def test_run_waits_for_fork_merge_commit_to_reach_base_repo(self) -> None:
github = FakeGitHub(behind=5)
updater = FakeBranchUpdater(github)
buildkite = FakeBuildkite([[], []])
with (
patch("run_ci_command.time.sleep"),
patch.object(
github,
"get_commits_behind_base",
side_effect=[5, ApiError(404, "Not Found"), 0],
),
):
run(make_event(COMMAND_RUN_CI), github, buildkite, "", updater)
self.assertEqual(buildkite.created_builds[0]["commit"], "merge-commit-sha")
self.assertTrue(github.comments[0].startswith("✅ "))
def test_run_reports_failed_merge_without_building(self) -> None:
github = FakeGitHub(behind=5)
updater = FakeBranchUpdater(github, error=ApiError(422, "merge conflict"))
buildkite = FakeBuildkite([[], []])
run(make_event(COMMAND_RUN_CI), github, buildkite, "", updater)
self.assertEqual(buildkite.created_builds, [])
self.assertTrue(github.comments[0].startswith("❌ "))
self.assertIn("could not merge upstream `main`", github.comments[0])
self.assertIn("merge conflict", github.comments[0])
self.assertIn("No new CI build was started.", github.comments[0])
self.assertIn("`/ci run --allow-stale`", github.comments[0])
def test_allow_stale_preserves_author_permissions_and_run_configuration(
self,
) -> None:
@@ -883,42 +756,6 @@ class RunCiCommandTest(unittest.TestCase):
self.assertEqual(buildkite.created_builds, [])
self.assertIn("AMD CI is already running", github.comments[0])
def test_amd_run_ignores_finished_webhook_build_for_author(self) -> None:
webhook_build = {
"created_at": "2026-08-18T01:00:00Z",
"meta_data": {},
"number": 122,
"pull_request": {"id": 42},
"source": "webhook",
"state": "passed",
"web_url": "https://buildkite.example/amd-ci/builds/122",
}
ui_build = {**webhook_build, "source": "ui"}
command_build = {
**webhook_build,
"meta_data": {"github-comment-id": "98"},
"source": "api",
}
for build, allowed in (
(webhook_build, True),
(ui_build, False),
(command_build, False),
):
with self.subTest(source=build["source"]):
github = FakeGitHub(
permission="read",
pr=make_pr(labels=[{"name": "ready"}]),
)
buildkite = FakeBuildkite([[], [build]])
run(make_event(COMMAND_RUN_AMD_CI, "author"), github, buildkite)
self.assertEqual(bool(buildkite.created_builds), allowed)
if not allowed:
self.assertIn(
"AMD CI already ran on this commit", github.comments[0]
)
def test_run_all_sets_buildkite_environment(self) -> None:
github = FakeGitHub()
buildkite = FakeBuildkite([[], []])
@@ -1265,29 +1102,6 @@ class RunCiCommandTest(unittest.TestCase):
)
self.assertIn("Use `/amd-ci run`", github.comments[0])
def test_amd_ci_retry_ignores_webhook_build_for_the_current_head(self) -> None:
github = FakeGitHub(
permission="read",
pr=make_pr(labels=[{"name": "ready"}]),
)
webhook_build = {
"created_at": "2026-08-18T01:00:00Z",
"meta_data": {},
"number": 122,
"pull_request": {"id": 42},
"source": "webhook",
"state": "passed",
"web_url": "https://buildkite.example/amd-ci/builds/122",
}
buildkite = FakeBuildkite([[webhook_build]])
run(make_event(COMMAND_RETRY_AMD_FAILED, "author"), github, buildkite)
self.assertEqual(buildkite.retry_calls, [])
self.assertIn(
"No AMD CI build exists for the current PR head", github.comments[0]
)
def test_ci_retry_creates_filtered_build_for_new_head(self) -> None:
github = FakeGitHub(
behind=100,
+3
View File
@@ -237,6 +237,9 @@ benchmarks/**/*.json
actionlint
shellcheck*/
# Ignore moe/marlin_moe gen code
csrc/libtorch_stable/moe/marlin_moe_wna16/kernel_*
# Ignore ep_kernels_workspace folder
ep_kernels_workspace/
+2 -8
View File
@@ -21,7 +21,7 @@ repos:
rev: v21.1.2
hooks:
- id: clang-format
exclude: 'csrc/moe/topk_softmax_kernels.cu|vllm/third_party/.*'
exclude: 'csrc/libtorch_stable/moe/topk_softmax_kernels.cu|vllm/third_party/.*'
types_or: [c++, cuda]
args: [--style=file, --verbose]
- repo: https://github.com/DavidAnson/markdownlint-cli2
@@ -205,7 +205,7 @@ repos:
language: python
types_or: [python, pyi]
require_serial: true
additional_dependencies: ["mypy==1.20.2", regex, types-cachetools, types-setuptools, types-PyYAML, types-requests, types-torch, "pydantic==2.13.5"]
additional_dependencies: ["mypy==1.20.2", regex, types-cachetools, types-setuptools, types-PyYAML, types-requests, types-torch, pydantic]
- id: png-lint
name: Lint PNG exports from excalidraw
entry: tools/pre_commit/png-lint.sh
@@ -277,12 +277,6 @@ repos:
files: ^docker/(Dockerfile|versions\.json)$
pass_filenames: false
additional_dependencies: [dockerfile-parse]
- id: check-dockerignore-tracked
name: Check .dockerignore does not exclude tracked files
entry: python tools/pre_commit/check_dockerignore_tracked.py
language: python
always_run: true
pass_filenames: false
- id: check-boolean-context-manager
name: Check for boolean ops in with-statements
entry: python tools/pre_commit/check_boolean_context_manager.py
+124 -128
View File
@@ -68,8 +68,8 @@ endif()
# requirements.txt files and should be kept consistent. The ROCm torch
# versions are derived from docker/Dockerfile.rocm
#
set(TORCH_SUPPORTED_VERSION_CUDA "2.14.0")
set(TORCH_SUPPORTED_VERSION_ROCM "2.14.0")
set(TORCH_SUPPORTED_VERSION_CUDA "2.13.0")
set(TORCH_SUPPORTED_VERSION_ROCM "2.13.0")
# TORCH_NIGHTLY=1 builds run against unpinned nightly wheels, so the supported-
# version check would always warn. Only treat it as a nightly build when the
# value is exactly "1" (the bootstrap exports TORCH_NIGHTLY=0 by default, which
@@ -363,8 +363,8 @@ endif()
if(VLLM_GPU_LANG STREQUAL "HIP")
set(VLLM_EXT_SRC
"csrc/rocm/quickreduce/torch_bindings.cpp"
"csrc/rocm/quickreduce/custom_quickreduce.cu")
"csrc/torch_bindings.cpp"
"csrc/custom_quickreduce.cu")
message(STATUS "Enabling C extension.")
define_extension_target(
@@ -374,8 +374,16 @@ if(VLLM_GPU_LANG STREQUAL "HIP")
SOURCES ${VLLM_EXT_SRC}
COMPILE_FLAGS ${VLLM_GPU_FLAGS}
ARCHITECTURES ${VLLM_GPU_ARCHES}
INCLUDE_DIRECTORIES ${CUTLASS_INCLUDE_DIR}
INCLUDE_DIRECTORIES ${CUTLASS_TOOLS_UTIL_INCLUDE_DIR}
USE_SABI 3
WITH_SOABI)
# If CUTLASS is compiled on NVCC >= 12.5, it by default uses
# cudaGetDriverEntryPointByVersion as a wrapper to avoid directly calling the
# driver API. This causes problems when linking with earlier versions of CUDA.
# Setting this variable sidesteps the issue by calling the driver directly.
target_compile_definitions(_C PRIVATE CUTLASS_ENABLE_DIRECT_CUDA_DRIVER_CALL=1)
endif() # _C HIP endif
if(VLLM_GPU_LANG STREQUAL "CUDA" OR VLLM_GPU_LANG STREQUAL "HIP")
@@ -411,42 +419,36 @@ if(VLLM_GPU_LANG STREQUAL "CUDA" OR VLLM_GPU_LANG STREQUAL "HIP")
endif()
set(VLLM_STABLE_EXT_SRC
"csrc/torch_bindings.cpp"
"csrc/cuda_view.cu"
"csrc/cuda_utils_kernels.cu"
"csrc/activation_kernels.cu"
"csrc/muse_spark_mul_tanh_soft_cap_kernel.cu"
"csrc/ngram_embedding_kernels.cu"
"csrc/vocab_parallel_embedding_kernels.cu"
"csrc/quantization/activation_kernels.cu"
"csrc/quantization/w8a8/int8/scaled_quant.cu"
"csrc/quantization/w8a8/fp8/common.cu"
"csrc/quantization/w8a8/fp8/per_token_group_quant.cu"
"csrc/quantization/gptq/q_gemm.cu"
"csrc/pos_encoding_kernels.cu"
"csrc/fused_muse_spark_qknorm_rope_kv_insert_kernel.cu"
"csrc/fused_qknorm_rope_kernel.cu"
"csrc/fused_minimax_m3_qknorm_rope_kv_insert_kernel.cu"
"csrc/layernorm_kernels.cu"
"csrc/layernorm_quant_kernels.cu"
"csrc/quantization/fused_kernels/fused_layernorm_dynamic_per_token_quant.cu"
"csrc/quantization/fused_kernels/fused_silu_mul_block_quant.cu"
"csrc/attention/merge_attn_states.cu"
"csrc/sampler.cu"
"csrc/topk.cu"
"csrc/mamba/selective_scan_fwd.cu"
"csrc/cache_kernels.cu"
"csrc/cache_kernels_fused.cu"
"csrc/custom_all_gather_reduce_scatter.cu"
"csrc/custom_all_gather_reduce_scatter_ops.cpp"
"csrc/custom_all_reduce.cu"
"csrc/fused_deepseek_v4_qnorm_rope_kv_insert_kernel.cu"
"csrc/fused_kimi_k3_mla_key_concat_kv_cache_kernel.cu")
if(VLLM_GPU_LANG STREQUAL "CUDA")
list(APPEND VLLM_STABLE_EXT_SRC
"csrc/fused_qknorm_rope_cache_kernel.cu")
endif()
"csrc/libtorch_stable/torch_bindings.cpp"
"csrc/libtorch_stable/cuda_view.cu"
"csrc/libtorch_stable/cuda_utils_kernels.cu"
"csrc/libtorch_stable/activation_kernels.cu"
"csrc/libtorch_stable/ngram_embedding_kernels.cu"
"csrc/libtorch_stable/vocab_parallel_embedding_kernels.cu"
"csrc/libtorch_stable/quantization/activation_kernels.cu"
"csrc/libtorch_stable/quantization/w8a8/int8/scaled_quant.cu"
"csrc/libtorch_stable/quantization/w8a8/fp8/common.cu"
"csrc/libtorch_stable/quantization/w8a8/fp8/per_token_group_quant.cu"
"csrc/libtorch_stable/quantization/w8a8/int8/per_token_group_quant.cu"
"csrc/libtorch_stable/quantization/gptq/q_gemm.cu"
"csrc/libtorch_stable/pos_encoding_kernels.cu"
"csrc/libtorch_stable/fused_qknorm_rope_kernel.cu"
"csrc/libtorch_stable/fused_minimax_m3_qknorm_rope_kv_insert_kernel.cu"
"csrc/libtorch_stable/layernorm_kernels.cu"
"csrc/libtorch_stable/layernorm_quant_kernels.cu"
"csrc/libtorch_stable/quantization/fused_kernels/fused_layernorm_dynamic_per_token_quant.cu"
"csrc/libtorch_stable/quantization/fused_kernels/fused_silu_mul_block_quant.cu"
"csrc/libtorch_stable/attention/merge_attn_states.cu"
"csrc/libtorch_stable/sampler.cu"
"csrc/libtorch_stable/topk.cu"
"csrc/libtorch_stable/mamba/selective_scan_fwd.cu"
"csrc/libtorch_stable/cache_kernels.cu"
"csrc/libtorch_stable/cache_kernels_fused.cu"
"csrc/libtorch_stable/custom_all_gather_reduce_scatter.cu"
"csrc/libtorch_stable/custom_all_gather_reduce_scatter_ops.cpp"
"csrc/libtorch_stable/custom_all_reduce.cu"
"csrc/libtorch_stable/fused_deepseek_v4_qnorm_rope_kv_insert_kernel.cu"
"csrc/libtorch_stable/fused_kimi_k3_mla_key_concat_kv_cache_kernel.cu")
if(VLLM_GPU_LANG STREQUAL "CUDA" AND
DEFINED CMAKE_CUDA_COMPILER_VERSION AND
@@ -468,10 +470,10 @@ if(VLLM_GPU_LANG STREQUAL "CUDA" OR VLLM_GPU_LANG STREQUAL "HIP")
if(VLLM_GPU_LANG STREQUAL "CUDA")
list(APPEND VLLM_STABLE_EXT_SRC
"csrc/hisparse_kernels.cu"
"csrc/attention/dcp_utils/dcp_direct_a2a_lse_reduce.cu"
"csrc/attention/dcp_utils/dcp_direct_kv_gather.cu"
"csrc/attention/dcp_utils/dcp_direct_q_gather.cu")
"csrc/libtorch_stable/hisparse_kernels.cu"
"csrc/libtorch_stable/attention/dcp_utils/dcp_direct_a2a_lse_reduce.cu"
"csrc/libtorch_stable/attention/dcp_utils/dcp_direct_kv_gather.cu"
"csrc/libtorch_stable/attention/dcp_utils/dcp_direct_q_gather.cu")
SET(CUTLASS_ENABLE_HEADERS_ONLY ON CACHE BOOL "Enable only the header library")
@@ -506,11 +508,12 @@ if(VLLM_GPU_LANG STREQUAL "CUDA" OR VLLM_GPU_LANG STREQUAL "HIP")
FetchContent_MakeAvailable(cutlass)
list(APPEND VLLM_STABLE_EXT_SRC
"csrc/quantization/w8a8/cutlass/scaled_mm_entry.cu"
"csrc/quantization/fp4/nvfp4_quant_entry.cu"
"csrc/quantization/fp4/nvfp4_scaled_mm_entry.cu"
"csrc/quantization/awq/gemm_kernels.cu"
"csrc/minimax_reduce_rms_kernel.cu")
"csrc/libtorch_stable/cutlass_extensions/common.cpp"
"csrc/libtorch_stable/quantization/w8a8/cutlass/scaled_mm_entry.cu"
"csrc/libtorch_stable/quantization/fp4/nvfp4_quant_entry.cu"
"csrc/libtorch_stable/quantization/fp4/nvfp4_scaled_mm_entry.cu"
"csrc/libtorch_stable/quantization/awq/gemm_kernels.cu"
"csrc/libtorch_stable/minimax_reduce_rms_kernel.cu")
#
# Machete kernels
@@ -524,7 +527,7 @@ if(VLLM_GPU_LANG STREQUAL "CUDA" OR VLLM_GPU_LANG STREQUAL "HIP")
# preselected input type pairs and schedules.
# Generate sources:
set(MACHETE_GEN_SCRIPT
${CMAKE_CURRENT_SOURCE_DIR}/csrc/quantization/machete/generate.py)
${CMAKE_CURRENT_SOURCE_DIR}/csrc/libtorch_stable/quantization/machete/generate.py)
file(MD5 ${MACHETE_GEN_SCRIPT} MACHETE_GEN_SCRIPT_HASH)
message(STATUS "Machete generation script hash: ${MACHETE_GEN_SCRIPT_HASH}")
@@ -557,7 +560,7 @@ if(VLLM_GPU_LANG STREQUAL "CUDA" OR VLLM_GPU_LANG STREQUAL "HIP")
endif()
# Add machete generated sources
file(GLOB MACHETE_GEN_SOURCES "csrc/quantization/machete/generated/*.cu")
file(GLOB MACHETE_GEN_SOURCES "csrc/libtorch_stable/quantization/machete/generated/*.cu")
list(APPEND VLLM_STABLE_EXT_SRC ${MACHETE_GEN_SOURCES})
# forward compatible
@@ -566,7 +569,7 @@ if(VLLM_GPU_LANG STREQUAL "CUDA" OR VLLM_GPU_LANG STREQUAL "HIP")
CUDA_ARCHS "${MACHETE_ARCHS}")
list(APPEND VLLM_STABLE_EXT_SRC
csrc/quantization/machete/machete_pytorch.cu)
csrc/libtorch_stable/quantization/machete/machete_pytorch.cu)
message(STATUS "Building Machete kernels for archs: ${MACHETE_ARCHS}")
else()
if (NOT ${CMAKE_CUDA_COMPILER_VERSION} VERSION_GREATER_EQUAL 12.0
@@ -584,7 +587,7 @@ if(VLLM_GPU_LANG STREQUAL "CUDA" OR VLLM_GPU_LANG STREQUAL "HIP")
# The fused MiniMax-M3 KV insert quantizes NVFP4 with SM100-family-only
# cvt instructions, so its SM10x/11x code is built for the family targets.
set(MINIMAX_M3_FUSED_SRC
"csrc/fused_minimax_m3_qknorm_rope_kv_insert_kernel.cu")
"csrc/libtorch_stable/fused_minimax_m3_qknorm_rope_kv_insert_kernel.cu")
list(REMOVE_ITEM VLLM_STABLE_EXT_SRC "${MINIMAX_M3_FUSED_SRC}")
set_gencode_flags_for_srcs(
SRCS "${VLLM_STABLE_EXT_SRC}"
@@ -608,9 +611,9 @@ if(VLLM_GPU_LANG STREQUAL "CUDA" OR VLLM_GPU_LANG STREQUAL "HIP")
if(COOPERATIVE_TOPK_ARCHS)
list(APPEND VLLM_STABLE_EXT_SRC
"csrc/cooperative_topk.cu")
"csrc/libtorch_stable/cooperative_topk.cu")
set_gencode_flags_for_srcs(
SRCS "csrc/cooperative_topk.cu"
SRCS "csrc/libtorch_stable/cooperative_topk.cu"
CUDA_ARCHS "${COOPERATIVE_TOPK_ARCHS}")
endif()
@@ -653,7 +656,7 @@ if(VLLM_GPU_LANG STREQUAL "CUDA" OR VLLM_GPU_LANG STREQUAL "HIP")
# preselected input type pairs and schedules.
# Generate sources:
set(MARLIN_GEN_SCRIPT
${CMAKE_CURRENT_SOURCE_DIR}/csrc/quantization/marlin/generate_kernels.py)
${CMAKE_CURRENT_SOURCE_DIR}/csrc/libtorch_stable/quantization/marlin/generate_kernels.py)
file(MD5 ${MARLIN_GEN_SCRIPT} MARLIN_GEN_SCRIPT_HASH)
list(JOIN CUDA_ARCHS "," CUDA_ARCHS_STR)
set(MARLIN_GEN_SCRIPT_HASH_AND_ARCH "${MARLIN_GEN_SCRIPT_HASH}(ARCH:${CUDA_ARCHS_STR})")
@@ -688,7 +691,7 @@ if(VLLM_GPU_LANG STREQUAL "CUDA" OR VLLM_GPU_LANG STREQUAL "HIP")
endif()
if (MARLIN_ARCHS)
file(GLOB MARLIN_TEMPLATE_KERNEL_SRC "csrc/quantization/marlin/sm80_kernel_*_float16.cu")
file(GLOB MARLIN_TEMPLATE_KERNEL_SRC "csrc/libtorch_stable/quantization/marlin/sm80_kernel_*_float16.cu")
set_gencode_flags_for_srcs(
SRCS "${MARLIN_TEMPLATE_KERNEL_SRC}"
CUDA_ARCHS "${MARLIN_ARCHS}")
@@ -698,7 +701,7 @@ if(VLLM_GPU_LANG STREQUAL "CUDA" OR VLLM_GPU_LANG STREQUAL "HIP")
endif()
list(APPEND VLLM_STABLE_EXT_SRC ${MARLIN_TEMPLATE_KERNEL_SRC})
file(GLOB MARLIN_TEMPLATE_BF16_KERNEL_SRC "csrc/quantization/marlin/sm80_kernel_*_bfloat16.cu")
file(GLOB MARLIN_TEMPLATE_BF16_KERNEL_SRC "csrc/libtorch_stable/quantization/marlin/sm80_kernel_*_bfloat16.cu")
set_gencode_flags_for_srcs(
SRCS "${MARLIN_TEMPLATE_BF16_KERNEL_SRC}"
CUDA_ARCHS "${MARLIN_BF16_ARCHS}")
@@ -710,7 +713,7 @@ if(VLLM_GPU_LANG STREQUAL "CUDA" OR VLLM_GPU_LANG STREQUAL "HIP")
endif()
if (MARLIN_SM75_ARCHS)
file(GLOB MARLIN_TEMPLATE_SM75_KERNEL_SRC "csrc/quantization/marlin/sm75_kernel_*.cu")
file(GLOB MARLIN_TEMPLATE_SM75_KERNEL_SRC "csrc/libtorch_stable/quantization/marlin/sm75_kernel_*.cu")
set_gencode_flags_for_srcs(
SRCS "${MARLIN_TEMPLATE_SM75_KERNEL_SRC}"
CUDA_ARCHS "${MARLIN_SM75_ARCHS}")
@@ -722,7 +725,7 @@ if(VLLM_GPU_LANG STREQUAL "CUDA" OR VLLM_GPU_LANG STREQUAL "HIP")
endif()
if (MARLIN_FP8_ARCHS)
file(GLOB MARLIN_TEMPLATE_FP8_KERNEL_SRC "csrc/quantization/marlin/sm89_kernel_*.cu")
file(GLOB MARLIN_TEMPLATE_FP8_KERNEL_SRC "csrc/libtorch_stable/quantization/marlin/sm89_kernel_*.cu")
set_gencode_flags_for_srcs(
SRCS "${MARLIN_TEMPLATE_FP8_KERNEL_SRC}"
CUDA_ARCHS "${MARLIN_FP8_ARCHS}")
@@ -734,10 +737,10 @@ if(VLLM_GPU_LANG STREQUAL "CUDA" OR VLLM_GPU_LANG STREQUAL "HIP")
endif()
set(MARLIN_SRCS
"csrc/quantization/marlin/marlin.cu"
"csrc/quantization/marlin/marlin_int4_fp8_preprocess.cu"
"csrc/quantization/marlin/gptq_marlin_repack.cu"
"csrc/quantization/marlin/awq_marlin_repack.cu")
"csrc/libtorch_stable/quantization/marlin/marlin.cu"
"csrc/libtorch_stable/quantization/marlin/marlin_int4_fp8_preprocess.cu"
"csrc/libtorch_stable/quantization/marlin/gptq_marlin_repack.cu"
"csrc/libtorch_stable/quantization/marlin/awq_marlin_repack.cu")
set_gencode_flags_for_srcs(
SRCS "${MARLIN_SRCS}"
CUDA_ARCHS "${MARLIN_OTHER_ARCHS}")
@@ -760,7 +763,7 @@ if(VLLM_GPU_LANG STREQUAL "CUDA" OR VLLM_GPU_LANG STREQUAL "HIP")
cuda_archs_loose_intersection(DSV3_FUSED_A_GEMM_ARCHS "9.0a;10.0a;10.1a;10.3a;12.0a;12.1a" "${CUDA_ARCHS}")
endif()
if(${CMAKE_CUDA_COMPILER_VERSION} VERSION_GREATER_EQUAL 12.0 AND DSV3_FUSED_A_GEMM_ARCHS)
set(DSV3_FUSED_A_GEMM_SRCS "csrc/dsv3_fused_a_gemm.cu")
set(DSV3_FUSED_A_GEMM_SRCS "csrc/libtorch_stable/dsv3_fused_a_gemm.cu")
set_gencode_flags_for_srcs(
SRCS "${DSV3_FUSED_A_GEMM_SRCS}"
CUDA_ARCHS "${DSV3_FUSED_A_GEMM_ARCHS}")
@@ -775,8 +778,8 @@ if(VLLM_GPU_LANG STREQUAL "CUDA" OR VLLM_GPU_LANG STREQUAL "HIP")
cuda_archs_sm90plus(FP32_ROUTER_GEMM_ARCHS "${CUDA_ARCHS}")
if(${CMAKE_CUDA_COMPILER_VERSION} VERSION_GREATER_EQUAL 12.0 AND FP32_ROUTER_GEMM_ARCHS)
set(FP32_ROUTER_GEMM_SRCS
"csrc/fp32_router_gemm_entry.cu"
"csrc/fp32_router_gemm.cu")
"csrc/libtorch_stable/fp32_router_gemm_entry.cu"
"csrc/libtorch_stable/fp32_router_gemm.cu")
set_gencode_flags_for_srcs(
SRCS "${FP32_ROUTER_GEMM_SRCS}"
CUDA_ARCHS "${FP32_ROUTER_GEMM_ARCHS}")
@@ -796,11 +799,11 @@ if(VLLM_GPU_LANG STREQUAL "CUDA" OR VLLM_GPU_LANG STREQUAL "HIP")
cuda_archs_loose_intersection(SCALED_MM_ARCHS "9.0a;" "${CUDA_ARCHS}")
if(${CMAKE_CUDA_COMPILER_VERSION} VERSION_GREATER_EQUAL 12.0 AND SCALED_MM_ARCHS)
set(SCALED_MM_SM90_SRCS
"csrc/quantization/w8a8/cutlass/scaled_mm_c3x_sm90.cu"
"csrc/quantization/w8a8/cutlass/c3x/scaled_mm_sm90_fp8.cu"
"csrc/quantization/w8a8/cutlass/c3x/scaled_mm_sm90_int8.cu"
"csrc/quantization/w8a8/cutlass/c3x/scaled_mm_azp_sm90_int8.cu"
"csrc/quantization/w8a8/cutlass/c3x/scaled_mm_blockwise_sm90_fp8.cu")
"csrc/libtorch_stable/quantization/w8a8/cutlass/scaled_mm_c3x_sm90.cu"
"csrc/libtorch_stable/quantization/w8a8/cutlass/c3x/scaled_mm_sm90_fp8.cu"
"csrc/libtorch_stable/quantization/w8a8/cutlass/c3x/scaled_mm_sm90_int8.cu"
"csrc/libtorch_stable/quantization/w8a8/cutlass/c3x/scaled_mm_azp_sm90_int8.cu"
"csrc/libtorch_stable/quantization/w8a8/cutlass/c3x/scaled_mm_blockwise_sm90_fp8.cu")
set_gencode_flags_for_srcs(
SRCS "${SCALED_MM_SM90_SRCS}"
CUDA_ARCHS "${SCALED_MM_ARCHS}")
@@ -831,9 +834,9 @@ if(VLLM_GPU_LANG STREQUAL "CUDA" OR VLLM_GPU_LANG STREQUAL "HIP")
endif()
if(${CMAKE_CUDA_COMPILER_VERSION} VERSION_GREATER_EQUAL 12.8 AND SCALED_MM_ARCHS)
set(SCALED_MM_SM120_SRCS
"csrc/quantization/w8a8/cutlass/scaled_mm_c3x_sm120.cu"
"csrc/quantization/w8a8/cutlass/c3x/scaled_mm_sm120_fp8.cu"
"csrc/quantization/w8a8/cutlass/c3x/scaled_mm_blockwise_sm120_fp8.cu"
"csrc/libtorch_stable/quantization/w8a8/cutlass/scaled_mm_c3x_sm120.cu"
"csrc/libtorch_stable/quantization/w8a8/cutlass/c3x/scaled_mm_sm120_fp8.cu"
"csrc/libtorch_stable/quantization/w8a8/cutlass/c3x/scaled_mm_blockwise_sm120_fp8.cu"
)
set_gencode_flags_for_srcs(
SRCS "${SCALED_MM_SM120_SRCS}"
@@ -865,9 +868,9 @@ if(VLLM_GPU_LANG STREQUAL "CUDA" OR VLLM_GPU_LANG STREQUAL "HIP")
endif()
if(${CMAKE_CUDA_COMPILER_VERSION} VERSION_GREATER_EQUAL 12.8 AND SCALED_MM_ARCHS)
set(SCALED_MM_SM100_SRCS
"csrc/quantization/w8a8/cutlass/scaled_mm_c3x_sm100.cu"
"csrc/quantization/w8a8/cutlass/c3x/scaled_mm_sm100_fp8.cu"
"csrc/quantization/w8a8/cutlass/c3x/scaled_mm_blockwise_sm100_fp8.cu"
"csrc/libtorch_stable/quantization/w8a8/cutlass/scaled_mm_c3x_sm100.cu"
"csrc/libtorch_stable/quantization/w8a8/cutlass/c3x/scaled_mm_sm100_fp8.cu"
"csrc/libtorch_stable/quantization/w8a8/cutlass/c3x/scaled_mm_blockwise_sm100_fp8.cu"
)
set_gencode_flags_for_srcs(
SRCS "${SCALED_MM_SM100_SRCS}"
@@ -898,7 +901,7 @@ if(VLLM_GPU_LANG STREQUAL "CUDA" OR VLLM_GPU_LANG STREQUAL "HIP")
# subtract out the archs that are already built for 3x
list(REMOVE_ITEM SCALED_MM_2X_ARCHS ${SCALED_MM_3X_ARCHS})
if (SCALED_MM_2X_ARCHS)
set(SCALED_MM_C2X_SRCS "csrc/quantization/w8a8/cutlass/scaled_mm_c2x.cu")
set(SCALED_MM_C2X_SRCS "csrc/libtorch_stable/quantization/w8a8/cutlass/scaled_mm_c2x.cu")
set_gencode_flags_for_srcs(
SRCS "${SCALED_MM_C2X_SRCS}"
CUDA_ARCHS "${SCALED_MM_2X_ARCHS}")
@@ -924,7 +927,7 @@ if(VLLM_GPU_LANG STREQUAL "CUDA" OR VLLM_GPU_LANG STREQUAL "HIP")
# if it's possible to compile MoE kernels that use its output.
cuda_archs_loose_intersection(SCALED_MM_ARCHS "9.0a" "${CUDA_ARCHS}")
if(${CMAKE_CUDA_COMPILER_VERSION} VERSION_GREATER_EQUAL 12.3 AND SCALED_MM_ARCHS)
set(CUTLASS_MOE_SM90_SRCS "csrc/quantization/w8a8/cutlass/moe/grouped_mm_c3x_sm90.cu")
set(CUTLASS_MOE_SM90_SRCS "csrc/libtorch_stable/quantization/w8a8/cutlass/moe/grouped_mm_c3x_sm90.cu")
set_gencode_flags_for_srcs(
SRCS "${CUTLASS_MOE_SM90_SRCS}"
CUDA_ARCHS "${SCALED_MM_ARCHS}")
@@ -948,7 +951,7 @@ if(VLLM_GPU_LANG STREQUAL "CUDA" OR VLLM_GPU_LANG STREQUAL "HIP")
cuda_archs_loose_intersection(SCALED_MM_ARCHS "10.0a;10.1a;10.3a" "${CUDA_ARCHS}")
endif()
if(${CMAKE_CUDA_COMPILER_VERSION} VERSION_GREATER_EQUAL 12.8 AND SCALED_MM_ARCHS)
set(CUTLASS_MOE_SM100_SRCS "csrc/quantization/w8a8/cutlass/moe/grouped_mm_c3x_sm100.cu")
set(CUTLASS_MOE_SM100_SRCS "csrc/libtorch_stable/quantization/w8a8/cutlass/moe/grouped_mm_c3x_sm100.cu")
set_gencode_flags_for_srcs(
SRCS "${CUTLASS_MOE_SM100_SRCS}"
CUDA_ARCHS "${SCALED_MM_ARCHS}")
@@ -973,7 +976,7 @@ if(VLLM_GPU_LANG STREQUAL "CUDA" OR VLLM_GPU_LANG STREQUAL "HIP")
cuda_archs_loose_intersection(CUTLASS_MOE_DATA_ARCHS "9.0a;10.0a;10.1a;10.3a;12.0a;12.1a" "${CUDA_ARCHS}")
endif()
if(${CMAKE_CUDA_COMPILER_VERSION} VERSION_GREATER_EQUAL 12.3 AND CUTLASS_MOE_DATA_ARCHS)
set(CUTLASS_MOE_DATA_SRCS "csrc/quantization/w8a8/cutlass/moe/moe_data.cu")
set(CUTLASS_MOE_DATA_SRCS "csrc/libtorch_stable/quantization/w8a8/cutlass/moe/moe_data.cu")
set_gencode_flags_for_srcs(
SRCS "${CUTLASS_MOE_DATA_SRCS}"
CUDA_ARCHS "${CUTLASS_MOE_DATA_ARCHS}")
@@ -1005,12 +1008,12 @@ if(VLLM_GPU_LANG STREQUAL "CUDA" OR VLLM_GPU_LANG STREQUAL "HIP")
endif()
if(${CMAKE_CUDA_COMPILER_VERSION} VERSION_GREATER_EQUAL 12.8 AND FP4_SM120_ARCHS)
set(FP4_SM120_SRCS
"csrc/quantization/fp4/nvfp4_quant_kernels.cu"
"csrc/quantization/fp4/activation_nvfp4_quant_fusion_kernels.cu"
"csrc/quantization/fp4/nvfp4_experts_quant.cu"
"csrc/quantization/fp4/nvfp4_scaled_mm_sm120_kernels.cu"
"csrc/quantization/fp4/nvfp4_blockwise_moe_kernel.cu"
"csrc/nvfp4_kv_cache_kernels.cu")
"csrc/libtorch_stable/quantization/fp4/nvfp4_quant_kernels.cu"
"csrc/libtorch_stable/quantization/fp4/activation_nvfp4_quant_fusion_kernels.cu"
"csrc/libtorch_stable/quantization/fp4/nvfp4_experts_quant.cu"
"csrc/libtorch_stable/quantization/fp4/nvfp4_scaled_mm_sm120_kernels.cu"
"csrc/libtorch_stable/quantization/fp4/nvfp4_blockwise_moe_kernel.cu"
"csrc/libtorch_stable/nvfp4_kv_cache_kernels.cu")
set_gencode_flags_for_srcs(
SRCS "${FP4_SM120_SRCS}"
CUDA_ARCHS "${FP4_SM120_ARCHS}")
@@ -1031,15 +1034,15 @@ if(VLLM_GPU_LANG STREQUAL "CUDA" OR VLLM_GPU_LANG STREQUAL "HIP")
endif()
if(${CMAKE_CUDA_COMPILER_VERSION} VERSION_GREATER_EQUAL 12.8 AND FP4_SM100_ARCHS)
set(FP4_SM100_SRCS
"csrc/quantization/fp4/nvfp4_quant_kernels.cu"
"csrc/quantization/fp4/activation_nvfp4_quant_fusion_kernels.cu"
"csrc/quantization/fp4/nvfp4_experts_quant.cu"
"csrc/quantization/fp4/nvfp4_scaled_mm_kernels.cu"
"csrc/quantization/fp4/nvfp4_blockwise_moe_kernel.cu"
"csrc/quantization/fp4/mxfp4_experts_quant.cu"
"csrc/quantization/fp4/mxfp4_blockwise_moe_kernel.cu"
"csrc/nvfp4_kv_cache_kernels.cu"
"csrc/nvfp4_ds_mla_cache_kernels.cu")
"csrc/libtorch_stable/quantization/fp4/nvfp4_quant_kernels.cu"
"csrc/libtorch_stable/quantization/fp4/activation_nvfp4_quant_fusion_kernels.cu"
"csrc/libtorch_stable/quantization/fp4/nvfp4_experts_quant.cu"
"csrc/libtorch_stable/quantization/fp4/nvfp4_scaled_mm_kernels.cu"
"csrc/libtorch_stable/quantization/fp4/nvfp4_blockwise_moe_kernel.cu"
"csrc/libtorch_stable/quantization/fp4/mxfp4_experts_quant.cu"
"csrc/libtorch_stable/quantization/fp4/mxfp4_blockwise_moe_kernel.cu"
"csrc/libtorch_stable/nvfp4_kv_cache_kernels.cu"
"csrc/libtorch_stable/nvfp4_ds_mla_cache_kernels.cu")
if(NOT ${CMAKE_CUDA_COMPILER_VERSION} VERSION_GREATER_EQUAL 12.9)
message(STATUS
"Building mxfp4_experts_quant unsupported stubs because CUDA compiler version is not >= 12.9 (found ${CMAKE_CUDA_COMPILER_VERSION}).")
@@ -1063,9 +1066,9 @@ if(VLLM_GPU_LANG STREQUAL "CUDA" OR VLLM_GPU_LANG STREQUAL "HIP")
cuda_archs_loose_intersection(W4A8_ARCHS "9.0a" "${CUDA_ARCHS}")
if(${CMAKE_CUDA_COMPILER_VERSION} VERSION_GREATER_EQUAL 12.0 AND W4A8_ARCHS)
set(W4A8_SRCS
"csrc/quantization/cutlass_w4a8/w4a8_mm_entry.cu"
"csrc/quantization/cutlass_w4a8/w4a8_grouped_mm_entry.cu"
"csrc/quantization/cutlass_w4a8/w4a8_utils.cu"
"csrc/libtorch_stable/quantization/cutlass_w4a8/w4a8_mm_entry.cu"
"csrc/libtorch_stable/quantization/cutlass_w4a8/w4a8_grouped_mm_entry.cu"
"csrc/libtorch_stable/quantization/cutlass_w4a8/w4a8_utils.cu"
)
set_gencode_flags_for_srcs(
@@ -1098,7 +1101,7 @@ if(VLLM_GPU_LANG STREQUAL "CUDA" OR VLLM_GPU_LANG STREQUAL "HIP")
endif()
if(${CMAKE_CUDA_COMPILER_VERSION} VERSION_GREATER_EQUAL 12.8 AND MLA_ARCHS)
set(CUTLASS_MLA_SRCS
"csrc/attention/mla/sm100_cutlass_mla_kernel.cu")
"csrc/libtorch_stable/attention/mla/sm100_cutlass_mla_kernel.cu")
set_gencode_flags_for_srcs(
SRCS "${CUTLASS_MLA_SRCS}"
CUDA_ARCHS "${MLA_ARCHS}")
@@ -1119,15 +1122,10 @@ if(VLLM_GPU_LANG STREQUAL "CUDA" OR VLLM_GPU_LANG STREQUAL "HIP")
"9.0a;10.0f;12.0f" "${CUDA_ARCHS}")
cuda_archs_loose_intersection(FUSED_GDN_DECODE_ARCHS
"8.0;8.6;8.9;9.0a;10.0f;11.0f;12.0f" "${CUDA_ARCHS}")
else()
# The fused GDN decode kernel needs only cp.async and BF16 math. Only the
# family-specific targets (10.0f, 11.0f, 12.0f) need CUDA >= 13.0.
cuda_archs_loose_intersection(FUSED_GDN_DECODE_ARCHS
"8.0;8.6;8.9;9.0a" "${CUDA_ARCHS}")
endif()
if(FUSED_KDA_DECODE_ARCHS)
set(FUSED_KDA_DECODE_SRC
"csrc/kimi_k3/fused_kda_decode_kernel.cu")
"csrc/libtorch_stable/kimi_k3/fused_kda_decode_kernel.cu")
set_gencode_flags_for_srcs(
SRCS "${FUSED_KDA_DECODE_SRC}"
CUDA_ARCHS "${FUSED_KDA_DECODE_ARCHS}")
@@ -1139,7 +1137,7 @@ if(VLLM_GPU_LANG STREQUAL "CUDA" OR VLLM_GPU_LANG STREQUAL "HIP")
endif()
if(FUSED_GDN_DECODE_ARCHS)
set(FUSED_GDN_DECODE_SRC
"csrc/gdn/fused_gdn_decode_kernel.cu")
"csrc/libtorch_stable/gdn/fused_gdn_decode_kernel.cu")
set_gencode_flags_for_srcs(
SRCS "${FUSED_GDN_DECODE_SRC}"
CUDA_ARCHS "${FUSED_GDN_DECODE_ARCHS}")
@@ -1156,7 +1154,7 @@ if(VLLM_GPU_LANG STREQUAL "CUDA" OR VLLM_GPU_LANG STREQUAL "HIP")
endif()
if(KIMI_K3_ATTN_RES_ARCHS)
set(KIMI_K3_ATTN_RES_SRC
"csrc/kimi_k3/attn_res_kernel.cu")
"csrc/libtorch_stable/kimi_k3/attn_res_kernel.cu")
set_gencode_flags_for_srcs(
SRCS "${KIMI_K3_ATTN_RES_SRC}"
CUDA_ARCHS "${KIMI_K3_ATTN_RES_ARCHS}")
@@ -1171,7 +1169,7 @@ if(VLLM_GPU_LANG STREQUAL "CUDA" OR VLLM_GPU_LANG STREQUAL "HIP")
# Hadacore kernels
cuda_archs_loose_intersection(HADACORE_ARCHS "8.0+PTX;9.0+PTX" "${CUDA_ARCHS}")
if(HADACORE_ARCHS)
set(HADACORE_SRCS "csrc/quantization/hadamard/hadacore/hadamard_transform_cuda.cu")
set(HADACORE_SRCS "csrc/libtorch_stable/quantization/hadamard/hadacore/hadamard_transform_cuda.cu")
set_gencode_flags_for_srcs(
SRCS "${HADACORE_SRCS}"
CUDA_ARCHS "${HADACORE_ARCHS}")
@@ -1195,7 +1193,7 @@ if(VLLM_GPU_LANG STREQUAL "CUDA" OR VLLM_GPU_LANG STREQUAL "HIP")
list(FILTER FUSED_KDA_DECODE_HIP_ARCHS INCLUDE REGEX "gfx942|gfx950")
if(FUSED_KDA_DECODE_HIP_ARCHS)
set(FUSED_KDA_DECODE_HIP_SRC
"csrc/kimi_k3/fused_kda_decode_kernel_rocm.cu")
"csrc/libtorch_stable/kimi_k3/fused_kda_decode_kernel_rocm.cu")
list(APPEND VLLM_STABLE_EXT_SRC "${FUSED_KDA_DECODE_HIP_SRC}")
message(STATUS
"Building fused KDA decode for HIP archs: ${FUSED_KDA_DECODE_HIP_ARCHS}")
@@ -1205,7 +1203,7 @@ if(VLLM_GPU_LANG STREQUAL "CUDA" OR VLLM_GPU_LANG STREQUAL "HIP")
list(FILTER FUSED_KDA_CHUNK_HIP_ARCHS INCLUDE REGEX "gfx950")
if(FUSED_KDA_CHUNK_HIP_ARCHS)
set(FUSED_KDA_CHUNK_HIP_SRC
"csrc/kimi_k3/fused_kda_chunk_kernel_rocm.cu")
"csrc/libtorch_stable/kimi_k3/fused_kda_chunk_kernel_rocm.cu")
list(APPEND VLLM_STABLE_EXT_SRC "${FUSED_KDA_CHUNK_HIP_SRC}")
message(STATUS
"Building fused KDA chunk for HIP archs: ${FUSED_KDA_CHUNK_HIP_ARCHS}")
@@ -1291,22 +1289,21 @@ endif()
#
set(VLLM_MOE_EXT_SRC
"csrc/moe/torch_bindings.cpp"
"csrc/moe/moe_align_sum_kernels.cu"
"csrc/moe/topk_softmax_kernels.cu"
"csrc/moe/topk_softplus_sqrt_kernels.cu")
"csrc/libtorch_stable/moe/torch_bindings.cpp"
"csrc/libtorch_stable/moe/moe_align_sum_kernels.cu"
"csrc/libtorch_stable/moe/topk_softmax_kernels.cu"
"csrc/libtorch_stable/moe/topk_softplus_sqrt_kernels.cu")
if(VLLM_GPU_LANG STREQUAL "CUDA")
list(APPEND VLLM_MOE_EXT_SRC
"csrc/moe/moe_wna16.cu"
"csrc/moe/grouped_topk_kernels.cu"
"csrc/moe/fused_globalize_align_block_size.cu")
"csrc/libtorch_stable/moe/moe_wna16.cu"
"csrc/libtorch_stable/moe/grouped_topk_kernels.cu")
endif()
if(VLLM_GPU_LANG STREQUAL "CUDA")
set(MOE_PERMUTE_SRC
"csrc/moe/permute_unpermute_kernels/moe_permute_unpermute_kernel.cu"
"csrc/moe/moe_permute_unpermute_op.cu")
"csrc/libtorch_stable/moe/permute_unpermute_kernels/moe_permute_unpermute_kernel.cu"
"csrc/libtorch_stable/moe/moe_permute_unpermute_op.cu")
list(APPEND VLLM_MOE_EXT_SRC "${MOE_PERMUTE_SRC}")
endif()
@@ -1317,7 +1314,7 @@ set_gencode_flags_for_srcs(
if(VLLM_GPU_LANG STREQUAL "CUDA")
set(VLLM_MOE_WNA16_SRC
"csrc/moe/moe_wna16.cu")
"csrc/libtorch_stable/moe/moe_wna16.cu")
set_gencode_flags_for_srcs(
SRCS "${VLLM_MOE_WNA16_SRC}"
@@ -1352,7 +1349,7 @@ if(VLLM_GPU_LANG STREQUAL "CUDA")
# preselected input type pairs and schedules.
# Generate sources:
set(MOE_MARLIN_GEN_SCRIPT
${CMAKE_CURRENT_SOURCE_DIR}/csrc/quantization/marlin/generate_kernels.py)
${CMAKE_CURRENT_SOURCE_DIR}/csrc/libtorch_stable/moe/marlin_moe_wna16/generate_kernels.py)
file(MD5 ${MOE_MARLIN_GEN_SCRIPT} MOE_MARLIN_GEN_SCRIPT_HASH)
list(JOIN CUDA_ARCHS "," CUDA_ARCHS_STR)
set(MOE_MARLIN_GEN_SCRIPT_HASH_AND_ARCH "${MOE_MARLIN_GEN_SCRIPT_HASH}(ARCH:${CUDA_ARCHS_STR})")
@@ -1366,7 +1363,6 @@ if(VLLM_GPU_LANG STREQUAL "CUDA")
COMMAND ${CMAKE_COMMAND} -E env
PYTHONPATH=$ENV{PYTHONPATH}
${Python_EXECUTABLE} ${MOE_MARLIN_GEN_SCRIPT} ${CUDA_ARCHS_STR}
${CMAKE_CURRENT_SOURCE_DIR}/csrc/moe/marlin_moe_wna16
RESULT_VARIABLE moe_marlin_generation_result
OUTPUT_VARIABLE moe_marlin_generation_output
OUTPUT_FILE ${CMAKE_CURRENT_BINARY_DIR}/moe_marlin_generation.log
@@ -1388,7 +1384,7 @@ if(VLLM_GPU_LANG STREQUAL "CUDA")
endif()
if (MARLIN_MOE_ARCHS)
file(GLOB MARLIN_MOE_SRC "csrc/moe/marlin_moe_wna16/sm80_kernel_*.cu")
file(GLOB MARLIN_MOE_SRC "csrc/libtorch_stable/moe/marlin_moe_wna16/sm80_kernel_*.cu")
set_gencode_flags_for_srcs(
SRCS "${MARLIN_MOE_SRC}"
CUDA_ARCHS "${MARLIN_MOE_ARCHS}")
@@ -1400,7 +1396,7 @@ if(VLLM_GPU_LANG STREQUAL "CUDA")
endif()
if (MARLIN_MOE_SM75_ARCHS)
file(GLOB MARLIN_MOE_SM75_SRC "csrc/moe/marlin_moe_wna16/sm75_kernel_*.cu")
file(GLOB MARLIN_MOE_SM75_SRC "csrc/libtorch_stable/moe/marlin_moe_wna16/sm75_kernel_*.cu")
set_gencode_flags_for_srcs(
SRCS "${MARLIN_MOE_SM75_SRC}"
CUDA_ARCHS "${MARLIN_MOE_SM75_ARCHS}")
@@ -1412,7 +1408,7 @@ if(VLLM_GPU_LANG STREQUAL "CUDA")
endif()
if (MARLIN_MOE_FP8_ARCHS)
file(GLOB MARLIN_MOE_FP8_SRC "csrc/moe/marlin_moe_wna16/sm89_kernel_*.cu")
file(GLOB MARLIN_MOE_FP8_SRC "csrc/libtorch_stable/moe/marlin_moe_wna16/sm89_kernel_*.cu")
set_gencode_flags_for_srcs(
SRCS "${MARLIN_MOE_FP8_SRC}"
CUDA_ARCHS "${MARLIN_MOE_FP8_ARCHS}")
@@ -1423,7 +1419,7 @@ if(VLLM_GPU_LANG STREQUAL "CUDA")
list(APPEND VLLM_MOE_EXT_SRC ${MARLIN_MOE_FP8_SRC})
endif()
set(MARLIN_MOE_OTHER_SRC "csrc/moe/marlin_moe_wna16/ops.cu")
set(MARLIN_MOE_OTHER_SRC "csrc/libtorch_stable/moe/marlin_moe_wna16/ops.cu")
set_gencode_flags_for_srcs(
SRCS "${MARLIN_MOE_OTHER_SRC}"
CUDA_ARCHS "${MARLIN_MOE_OTHER_ARCHS}")
-1
View File
@@ -67,7 +67,6 @@ Install vLLM with [`uv`](https://docs.astral.sh/uv/) (recommended) or `pip`:
```bash
uv pip install vllm
vllm download-kernels # precompiled kernels for faster startup
```
Or [build from source](https://docs.vllm.ai/en/latest/getting_started/installation/gpu/index.html#build-wheel-from-source) for development.
+159 -164
View File
@@ -1,29 +1,23 @@
#!/usr/bin/env python3
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""Benchmark top-k/top-p filtering with Triton, FlashInfer, and optional PyTorch.
"""Benchmark comparing Triton vs PyTorch sort-based top-k/top-p implementations.
All implementations return masked logits. Timings include a fresh-input copy.
FlashInfer timings include softmax and mask reconstruction when its APIs require them.
L2 is flushed before each measured iteration, outside the timed region.
Compares:
- apply_top_k_top_p_triton (Triton binary search)
- apply_top_k_top_p (PyTorch sort-based)
Scenarios:
- top_k only (whole batch, partial batch)
- top_p only (whole batch, partial batch)
- mix of top_k and top_p
- optional sparse inputs via --num-finite-logits
"""
import argparse
import gc
import json
from dataclasses import dataclass
from pathlib import Path
import flashinfer.sampling as fi_sampling
import pandas as pd
import torch
from flashinfer.testing import bench_gpu_time_with_cupti
from vllm.v1.sample.ops.topk_topp_sampler import apply_top_k_top_p_pytorch
from vllm.v1.sample.ops.topk_topp_triton import (
@@ -71,18 +65,14 @@ def calculate_ops_pct(
def create_logits(
batch_size: int,
vocab_size: int,
device: str = "cuda",
num_finite_logits: int | None = None,
batch_size: int, vocab_size: int, device: str = "cuda"
) -> torch.Tensor:
"""Create random logits mimicking a realistic LLM distribution.
Uses a Zipf-like probability distribution (rank^-1.1) converted to logits
via log, then randomly permuted per row. This produces a peaked distribution
where a small number of tokens capture most probability mass, similar to
real model outputs. When num_finite_logits is set, retain that many ranks
before permuting their positions independently in each row.
real model outputs.
"""
# Create Zipf-like probabilities: p(rank) ~ rank^(-alpha)
ranks = torch.arange(1, vocab_size + 1, dtype=torch.float32, device=device)
@@ -91,8 +81,6 @@ def create_logits(
# Convert to logits (log-probabilities, unnormalized is fine)
base_logits = probs.log()
if num_finite_logits is not None:
base_logits[num_finite_logits:] = -float("inf")
# Broadcast to batch and randomly permute each row
logits = base_logits.unsqueeze(0).expand(batch_size, -1).clone()
@@ -102,26 +90,6 @@ def create_logits(
return logits
def apply_top_k_top_p_flashinfer_mask(logits, k, p):
"""Keep top-k in logits; adapt probability-based top-p back to logits."""
if k is not None:
logits = fi_sampling.top_k_mask_logits(logits, k)
if p is not None:
# Top-p renormalization requires softmax, even after logits-based top-k.
probs = logits.softmax(dim=-1)
probs = fi_sampling.top_p_renorm_probs(probs, p, is_deterministic=True)
# Return the selected original logits, not normalized probabilities.
logits = logits.masked_fill(probs == 0, -float("inf"))
return logits
IMPLEMENTATIONS = {
"triton": apply_top_k_top_p_triton,
"flashinfer-mask": apply_top_k_top_p_flashinfer_mask,
"pytorch": apply_top_k_top_p_pytorch,
}
def measure_memory() -> tuple[int, int]:
"""Return (allocated, reserved) memory in bytes."""
torch.accelerator.synchronize()
@@ -139,6 +107,53 @@ def reset_memory_stats():
gc.collect()
def benchmark_function(
func,
logits: torch.Tensor,
k: torch.Tensor | None,
p: torch.Tensor | None,
warmup_iters: int = 5,
benchmark_iters: int = 20,
) -> tuple[float, int]:
"""Benchmark a function and return (avg_time_ms, peak_memory_bytes).
Returns average time in milliseconds and peak memory usage.
"""
# Warmup
for _ in range(warmup_iters):
logits_copy = logits.clone()
func(logits_copy, k, p)
torch.accelerator.synchronize()
# Reset memory stats before benchmark
reset_memory_stats()
# Benchmark
start_events = [
torch.cuda.Event(enable_timing=True) for _ in range(benchmark_iters)
]
end_events = [torch.cuda.Event(enable_timing=True) for _ in range(benchmark_iters)]
for i in range(benchmark_iters):
logits_copy = logits.clone()
start_events[i].record()
func(logits_copy, k, p)
end_events[i].record()
torch.accelerator.synchronize()
# Calculate timing
times = [
start_events[i].elapsed_time(end_events[i]) for i in range(benchmark_iters)
]
avg_time = sum(times) / len(times)
# Get peak memory
_, peak_memory = measure_memory()
return avg_time, peak_memory
def create_benchmark_configs(
batch_sizes: list[int],
vocab_sizes: list[int],
@@ -271,103 +286,118 @@ def format_memory(bytes_val: int) -> str:
def run_benchmark(
configs: list[BenchmarkConfig],
implementations: list[str],
warmup_iters: int = 5,
benchmark_iters: int = 20,
verbose: bool = True,
num_finite_logits: int | None = None,
check: bool = False,
output: Path | None = None,
):
"""Run selected implementations with identical inputs and masked-logit outputs."""
"""Run all benchmarks and print results."""
results = []
for index, config in enumerate(configs):
logits = create_logits(
config.batch_size,
config.vocab_size,
num_finite_logits=num_finite_logits,
)
checks = {}
if check:
checks = check_implementations(logits, config, implementations)
k, p = config.k_values, config.p_values
timings = {}
for name in implementations:
func = IMPLEMENTATIONS[name]
reset_memory_stats()
times = bench_gpu_time_with_cupti(
# note: measured runtime includes logits.clone()
lambda func=func, logits=logits, k=k, p=p: func(logits.clone(), k, p),
dry_run_iters=warmup_iters,
repeat_iters=benchmark_iters,
use_cuda_graph=False, # sampling is outside cuda graph
cold_l2_cache=True,
)
elapsed = sum(times) / len(times)
_, memory = measure_memory()
timings[name] = dict(time_ms=elapsed, peak_memory=memory)
row = create_result_row(config, timings, checks)
results.append(row)
if output is not None:
with output.open("a") as file:
file.write(json.dumps(row) + "\n")
print("=" * 100)
print("Top-k/Top-p Benchmark: Triton vs PyTorch Sort-based")
print("=" * 100)
print()
for config in configs:
if verbose:
print(f"Completed {index + 1}/{len(configs)}: {config.name}", flush=True)
print(f"Running: {config.description}")
# Create fresh logits for this config
logits = create_logits(config.batch_size, config.vocab_size)
# Benchmark Triton
reset_memory_stats()
triton_time, triton_mem = benchmark_function(
apply_top_k_top_p_triton,
logits,
config.k_values,
config.p_values,
warmup_iters,
benchmark_iters,
)
# Benchmark PyTorch
reset_memory_stats()
pytorch_time, pytorch_mem = benchmark_function(
apply_top_k_top_p_pytorch,
logits,
config.k_values,
config.p_values,
warmup_iters,
benchmark_iters,
)
speedup = pytorch_time / triton_time if triton_time > 0 else float("inf")
mem_ratio = pytorch_mem / triton_mem if triton_mem > 0 else float("inf")
result = {
"config": config,
"triton_time_ms": triton_time,
"pytorch_time_ms": pytorch_time,
"triton_mem": triton_mem,
"pytorch_mem": pytorch_mem,
"speedup": speedup,
"mem_ratio": mem_ratio,
}
results.append(result)
if verbose:
print(f" Triton: {triton_time:.3f} ms, {format_memory(triton_mem)}")
print(f" PyTorch: {pytorch_time:.3f} ms, {format_memory(pytorch_mem)}")
print(f" Speedup: {speedup:.2f}x, Memory ratio: {mem_ratio:.2f}x")
print()
# Clean up
del logits
reset_memory_stats()
return results
def check_implementations(
logits: torch.Tensor,
config: BenchmarkConfig,
implementations: list[str],
) -> dict:
"""Compare filtered distributions against PyTorch outside the timed region."""
k, p = config.k_values, config.p_values
reference = apply_top_k_top_p_pytorch(logits.clone(), k, p).softmax(dim=-1)
checks = {}
for name in implementations:
actual = IMPLEMENTATIONS[name](logits.clone(), k, p).softmax(dim=-1)
total_variation = (actual - reference).abs().sum(-1).max().item() / 2
checks[name] = {
"close": torch.allclose(actual, reference, rtol=1e-5, atol=1e-6),
"max_total_variation": total_variation,
}
if not checks[name]["close"]:
print(f"Distribution difference: {name} {config.name} {checks[name]}")
del actual
return checks
def print_summary_table(results: list[dict]):
"""Print the same result rows saved to the output file."""
"""Print a summary table of results."""
print()
print("=" * 130)
print("SUMMARY TABLE")
print("=" * 130)
print()
print(pd.DataFrame(results).to_string(index=False, float_format="{:.3g}".format))
# Header
header = (
f"{'Scenario':<40} {'Batch':>6} {'Vocab':>7} {'Ops%':>6} "
f"{'Triton (ms)':>12} {'PyTorch (ms)':>13} {'Speedup':>8} "
f"{'Tri Mem':>10} {'Pyt Mem':>10}"
)
print(header)
print("-" * 130)
def create_result_row(config, timings, checks) -> dict:
"""Build a shared row for the summary table and JSONL output."""
row = {
"Scenario": config.name.split("_b")[0],
"Batch": config.batch_size,
"Vocab": config.vocab_size,
"Ops%": f"{config.ops_pct:.0f}%",
}
row.update({f"{name} (ms)": data["time_ms"] for name, data in timings.items()})
for name, data in timings.items():
row[f"{name} mem"] = format_memory(data["peak_memory"])
for name, data in checks.items():
row[f"{name} check"] = data["close"]
row[f"{name} max TV"] = data["max_total_variation"]
return row
# Group by scenario type
current_vocab = None
for result in results:
config = result["config"]
# Add separator between vocab sizes
if current_vocab != config.vocab_size:
if current_vocab is not None:
print("-" * 130)
current_vocab = config.vocab_size
scenario = config.name.split("_b")[0] # Extract scenario name
print(
f"{scenario:<40} {config.batch_size:>6} {config.vocab_size:>7} "
f"{config.ops_pct:>5.0f}% "
f"{result['triton_time_ms']:>12.3f} {result['pytorch_time_ms']:>13.3f} "
f"{result['speedup']:>7.2f}x "
f"{format_memory(result['triton_mem']):>10} "
f"{format_memory(result['pytorch_mem']):>10}"
)
print("=" * 130)
def main():
parser = argparse.ArgumentParser(
description="Benchmark Triton and FlashInfer top-k/top-p filtering"
description="Benchmark Triton vs PyTorch sort-based top-k/top-p implementations"
)
parser.add_argument(
"--batch-sizes",
@@ -401,60 +431,29 @@ def main():
help="Only print summary table",
)
parser.add_argument(
"--num-finite-logits",
type=int,
help="Finite logits per row; remaining entries are -inf (default: dense)",
)
parser.add_argument(
"--implementations",
nargs="+",
choices=list(IMPLEMENTATIONS),
default=["triton", "flashinfer-mask"],
help="Implementations to compare (PyTorch is an optional reference)",
)
parser.add_argument(
"--check",
action="store_true",
help="Check filtered distributions against PyTorch outside the timed region",
)
parser.add_argument(
"--scenarios",
nargs="+",
choices=[
"topk_whole",
"topk_partial",
"topp_whole",
"topp_partial",
"topk_topp_whole",
"mixed_partial",
],
help="Run only the selected scenarios (default: all)",
)
parser.add_argument("--output", type=Path, help="Save incremental JSONL results")
args = parser.parse_args()
if args.output is not None:
args.output.parent.mkdir(parents=True, exist_ok=True)
args.output.open("x").close()
if args.num_finite_logits is not None and not (
0 <= args.num_finite_logits <= min(args.vocab_sizes)
):
parser.error("--num-finite-logits must be between 0 and each vocabulary size")
torch.manual_seed(0)
# Print configuration
print(f"Batch sizes: {args.batch_sizes}")
print(f"Vocab sizes: {args.vocab_sizes}")
print(f"Warmup iterations: {args.warmup_iters}")
print(f"Benchmark iterations: {args.benchmark_iters}")
print()
# Check CUDA
if not torch.cuda.is_available():
print("ERROR: CUDA is not available. This benchmark requires a GPU.")
return
device_name = torch.cuda.get_device_name(0)
print(f"GPU: {device_name}")
print()
# Create configs
configs = create_benchmark_configs(args.batch_sizes, args.vocab_sizes)
if args.scenarios is not None:
configs = [c for c in configs if c.name.split("_b")[0] in args.scenarios]
configs = create_benchmark_configs(
args.batch_sizes,
args.vocab_sizes,
)
# Run benchmarks
results = run_benchmark(
@@ -462,10 +461,6 @@ def main():
warmup_iters=args.warmup_iters,
benchmark_iters=args.benchmark_iters,
verbose=not args.quiet,
num_finite_logits=args.num_finite_logits,
implementations=args.implementations,
check=args.check,
output=args.output,
)
# Print summary
+4 -26
View File
@@ -11,7 +11,7 @@ from vllm.utils.argparse_utils import FlexibleArgumentParser
num_tokens_range = [2**i for i in range(0, 8, 2)]
num_experts_range = [16, 32, 64, 128, 256, 512]
topk_range = [3, 4, 8, 10]
topk_range = [3, 4]
configs = list(itertools.product(num_tokens_range, num_experts_range, topk_range))
@@ -39,9 +39,9 @@ def get_benchmark(scoring_func):
x_names=["num_tokens", "num_experts", "topk"],
x_vals=[list(_) for _ in configs],
line_arg="provider",
line_vals=["torch", "vllm", "aiter_gating"],
line_names=["Torch", "vLLM", "AITER topk_gating"],
styles=[("blue", "-"), ("red", "-"), ("green", "-")],
line_vals=["torch", "vllm"],
line_names=["Torch", "vLLM"],
styles=[("blue", "-"), ("red", "-")],
ylabel="us",
plot_name=f"fused-topk-perf-{scoring_func}",
args={},
@@ -70,28 +70,6 @@ def get_benchmark(scoring_func):
),
quantiles=quantiles,
)
elif provider == "aiter_gating":
from vllm._aiter_ops import rocm_aiter_ops
if scoring_func != "softmax" or not rocm_aiter_ops.is_topk_gating_enabled():
return float("nan"), float("nan"), float("nan")
topk_weights = torch.empty(
(num_tokens, topk), dtype=torch.float32, device="cuda"
)
topk_ids = torch.empty((num_tokens, topk), dtype=torch.int32, device="cuda")
token_expert_indices = torch.empty(
(num_tokens, topk), dtype=torch.int32, device="cuda"
)
ms, min_ms, max_ms = triton.testing.do_bench(
lambda: rocm_aiter_ops.topk_gating(
topk_weights,
topk_ids,
token_expert_indices,
gating_output,
renormalize,
),
quantiles=quantiles,
)
else:
ms, min_ms, max_ms = triton.testing.do_bench(
lambda: fused_topk(
@@ -1,326 +0,0 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""Benchmark the fused Muse-Spark QK-norm + RoPE + KV-insert kernel vs unfused.
``tuned_us`` forces the cp.async schedule ``fused_qk_norm_rope`` only auto-selects
on SM 9.0; ``unfused_us`` is what production runs.
"""
import argparse
import itertools
import torch
import vllm._custom_ops as ops
HEAD_DIM = 64
DEFAULT_TOKENS = [1, 8, 64, 256, 1024, 4096, 8192, 16384, 32768, 65536]
def make_cos_sin_cache(max_pos, rotary_dim, base, device):
inv_freq = 1.0 / (
base
** (
torch.arange(0, rotary_dim, 2, dtype=torch.float32, device=device)
/ rotary_dim
)
)
t = torch.arange(max_pos, dtype=torch.float32, device=device)
freqs = torch.einsum("i,j->ij", t, inv_freq)
return torch.cat((freqs.cos(), freqs.sin()), dim=-1)
class Case:
"""Everything a single (num_tokens, config) measurement needs."""
def __init__(self, num_tokens, nq, nkv, dtype, kv_cache_dtype, device):
self.num_tokens = num_tokens
self.nq, self.nkv = nq, nkv
self.dtype = dtype
self.kv_cache_dtype = kv_cache_dtype
self.eps = 1e-5
self.block_size = 128
self.rotary_dim = HEAD_DIM
self.qsz, self.kvsz = nq * HEAD_DIM, nkv * HEAD_DIM
self.row = self.qsz + 2 * self.kvsz + self.qsz # [q | k | v | o_gate]
self.qkv = torch.randn(num_tokens, self.row, dtype=dtype, device=device)
self.pristine = self.qkv.clone()
max_pos = 32768
self.cos_sin = make_cos_sin_cache(max_pos, self.rotary_dim, 500_000.0, device)
self.positions = torch.randint(
0, max_pos, (num_tokens,), dtype=torch.int64, device=device
)
num_blocks = (num_tokens + self.block_size - 1) // self.block_size + 1
storage = torch.uint8 if kv_cache_dtype != "auto" else dtype
self.kv_cache = torch.zeros(
num_blocks,
nkv,
self.block_size,
2 * HEAD_DIM,
dtype=storage,
device=device,
)
self.k_cache, self.v_cache = self.kv_cache.transpose(1, 2).split(
HEAD_DIM, dim=-1
)
self.slot_mapping = torch.randperm(
num_blocks * self.block_size, dtype=torch.int64, device=device
)[:num_tokens]
self.k_scale = torch.tensor(1.0, dtype=torch.float32, device=device)
self.v_scale = torch.tensor(1.0, dtype=torch.float32, device=device)
self.ones = torch.ones(HEAD_DIM, dtype=dtype, device=device)
self.q_fp8 = torch.empty(
num_tokens, self.qsz, dtype=torch.float8_e4m3fn, device=device
)
self.q_scale = torch.tensor(0.37, dtype=torch.float32, device=device)
def views(self):
q, k, v, _ = self.qkv.split([self.qsz, self.kvsz, self.kvsz, self.qsz], dim=-1)
return q, k, v
def fused(self, use_rope=True, quant_query=False):
ops.fused_muse_spark_qknorm_rope_kv_insert(
self.qkv,
self.positions,
self.nq,
self.nkv,
HEAD_DIM,
self.eps,
cos_sin_cache=self.cos_sin if use_rope else None,
rotary_dim=self.rotary_dim if use_rope else 0,
slot_mapping=self.slot_mapping,
kv_cache=self.kv_cache,
block_size=self.block_size,
kv_cache_dtype=self.kv_cache_dtype,
k_scale=self.k_scale if self.kv_cache_dtype != "auto" else None,
v_scale=self.v_scale if self.kv_cache_dtype != "auto" else None,
q_fp8_out=self.q_fp8 if quant_query else None,
q_scale=self.q_scale if quant_query else None,
)
def separate_query_quant(self, use_rope=True):
self.fused(use_rope)
ops.scaled_fp8_quant(self.qkv[:, : self.qsz], scale=self.q_scale)
def _insert(self):
_, k, v = self.views()
ops.reshape_and_cache_flash(
k.view(-1, self.nkv, HEAD_DIM),
v.view(-1, self.nkv, HEAD_DIM),
self.k_cache,
self.v_cache,
self.slot_mapping,
self.kv_cache_dtype,
self.k_scale,
self.v_scale,
)
def unfused_rope(self, token_heads_per_warp: int = -1):
ops.fused_qk_norm_rope(
self.qkv,
self.nq,
self.nkv,
self.nkv + self.nq, # v heads plus the o-gate
HEAD_DIM,
self.eps,
self.ones,
self.ones,
self.cos_sin,
False,
self.positions,
token_heads_per_warp,
)
self._insert()
def fused_bytes(self):
el = self.qkv.element_size()
cache_el = self.kv_cache.element_size()
read = (self.nq + 2 * self.nkv) * HEAD_DIM * el
write = (self.nq + self.nkv) * HEAD_DIM * el
cache = 2 * self.nkv * HEAD_DIM * cache_el
return self.num_tokens * (read + write + cache)
def _l2_bytes(device) -> int:
return torch.cuda.get_device_properties(device).L2_cache_size
def _peak_bandwidth(device) -> float:
"""Measured large-copy bandwidth, in GB/s -- the practical ceiling."""
n = 1 << 28 # 256 Mi elements = 512 MiB each way at 2 bytes
a = torch.empty(n, dtype=torch.bfloat16, device=device)
b = torch.empty_like(a)
for _ in range(3):
b.copy_(a)
torch.accelerator.synchronize()
start, end = torch.cuda.Event(True), torch.cuda.Event(True)
start.record()
iters = 20
for _ in range(iters):
b.copy_(a)
end.record()
torch.accelerator.synchronize()
ms = start.elapsed_time(end) / iters
return (2 * a.numel() * a.element_size()) / (ms * 1e-3) / 1e9
def time_eager(fn, warmup=20, iters=100) -> float:
for _ in range(warmup):
fn()
torch.accelerator.synchronize()
start, end = torch.cuda.Event(True), torch.cuda.Event(True)
start.record()
for _ in range(iters):
fn()
end.record()
torch.accelerator.synchronize()
return start.elapsed_time(end) / iters * 1e3 # us
def time_graph(fn, warmup=10, iters=50, inner=32) -> float:
"""GPU time per call; ``inner`` calls per graph amortize the ~2 us replay."""
side = torch.cuda.Stream()
side.wait_stream(torch.cuda.current_stream())
with torch.cuda.stream(side):
for _ in range(warmup):
fn()
torch.cuda.current_stream().wait_stream(side)
graph = torch.cuda.CUDAGraph()
with torch.cuda.graph(graph):
for _ in range(inner):
fn()
for _ in range(3):
graph.replay()
torch.accelerator.synchronize()
start, end = torch.cuda.Event(True), torch.cuda.Event(True)
start.record()
for _ in range(iters):
graph.replay()
end.record()
torch.accelerator.synchronize()
return start.elapsed_time(end) / (iters * inner) * 1e3 # us
def main():
p = argparse.ArgumentParser()
p.add_argument("--nq", type=int, default=32, help="query heads per rank")
p.add_argument("--nkv", type=int, default=4, help="kv heads per rank")
p.add_argument("--kv-cache-dtype", default="auto", choices=["auto", "fp8"])
p.add_argument("--dtype", default="bfloat16", choices=["bfloat16", "float16"])
p.add_argument("--tokens", type=int, nargs="*", default=DEFAULT_TOKENS)
p.add_argument("--csv", default=None)
p.add_argument("--no-graph", action="store_true")
p.add_argument(
"--query-fp8", action="store_true", help="compare fused Q quantization"
)
args = p.parse_args()
device = "cuda"
dtype = getattr(torch, args.dtype)
if args.query_fp8:
if args.kv_cache_dtype != "fp8":
p.error("--query-fp8 requires --kv-cache-dtype fp8")
timer = time_eager if args.no_graph else time_graph
print(f"device: {torch.cuda.get_device_name()}; nq={args.nq}, nkv={args.nkv}")
print("layer,tokens,separate_us,fused_us,speedup")
rows = []
for use_rope, nt in itertools.product([True, False], args.tokens):
case = Case(nt, args.nq, args.nkv, dtype, args.kv_cache_dtype, device)
separate_us = timer(lambda c=case, r=use_rope: c.separate_query_quant(r))
case.qkv.copy_(case.pristine)
fused_us = timer(lambda c=case, r=use_rope: c.fused(r, quant_query=True))
row = (
"rope" if use_rope else "nope",
nt,
separate_us,
fused_us,
separate_us / fused_us,
)
rows.append(row)
print(f"{row[0]},{nt},{separate_us:.3f},{fused_us:.3f},{row[4]:.3f}")
if args.csv:
import csv
with open(args.csv, "w", newline="") as f:
writer = csv.writer(f)
writer.writerow(
["layer", "tokens", "separate_us", "fused_us", "speedup"]
)
writer.writerows(rows)
return
peak = _peak_bandwidth(device)
print(
f"device: {torch.cuda.get_device_name()} measured copy peak: {peak:.0f} GB/s"
)
print(
f"config: nq={args.nq} nkv={args.nkv} head_dim={HEAD_DIM} "
f"dtype={args.dtype} kv_cache={args.kv_cache_dtype}\n"
)
modes = ["rope"]
l2 = _l2_bytes(device)
print(
f"rows marked * have a working set inside the {l2 / 2**20:.0f} MiB L2, "
"so their bandwidth is not an HBM number\n"
)
header = (
f"{'layer':>5} {'tokens':>7} {'fused_us':>9} {'unfused_us':>11} "
f"{'tuned_us':>9} {'speedup':>8} {'GB/s':>8} {'SOL%':>6}"
)
rows = []
for mode, use_graph in itertools.product(modes, [False, True]):
if use_graph and args.no_graph:
continue
label = "graph" if use_graph else "eager"
print(f"── {mode} layer, {label} ──")
print(header)
timer = time_graph if use_graph else time_eager
for nt in args.tokens:
case = Case(nt, args.nq, args.nkv, dtype, args.kv_cache_dtype, device)
use_rope = mode == "rope"
fused_us = timer(lambda c=case, r=use_rope: c.fused(r))
baseline = case.unfused_rope
unfused_us = timer(baseline)
tuned_us = (
min(timer(lambda c=case, t=t: c.unfused_rope(t)) for t in (4, 8))
if use_rope
else unfused_us
)
nbytes = case.fused_bytes()
gbps = nbytes / (fused_us * 1e-6) / 1e9
sol = 100.0 * gbps / peak
resident = "*" if nbytes < l2 else " "
print(
f"{mode:>5} {nt:>7} {fused_us:>9.2f} {unfused_us:>11.2f} "
f"{tuned_us:>9.2f} {tuned_us / fused_us:>7.2f}x {gbps:>8.0f} "
f"{sol:>5.1f}%{resident}"
)
rows.append((mode, label, nt, fused_us, unfused_us, tuned_us, gbps, sol))
print()
if args.csv:
import csv
with open(args.csv, "w", newline="") as f:
w = csv.writer(f)
w.writerow(
[
"layer",
"timing",
"tokens",
"fused_us",
"unfused_us",
"tuned_us",
"gbps",
"sol",
]
)
w.writerows(rows)
print(f"wrote {args.csv}")
if __name__ == "__main__":
main()
@@ -1,119 +0,0 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""Compare in-tree vs AITER sparse-indexer prefill top-k at GLM-5.3-Flash shapes.
Example:
.venv/bin/python benchmarks/kernels/benchmark_sparse_indexer_topk_prefill.py \
--num-rows 1024 --num-cols 131072 --topk 512
"""
import argparse
import os
import time
import torch
from vllm import _custom_ops # noqa: F401
from vllm._aiter_ops import rocm_aiter_ops
def _make_inputs(num_rows, num_cols, causal, device):
logits = torch.randn(num_rows, num_cols, dtype=torch.float32, device=device)
row_starts = torch.zeros(num_rows, dtype=torch.int32, device=device)
if causal:
ramp = torch.arange(1, num_rows + 1, device=device, dtype=torch.float32)
row_ends = (ramp * (num_cols / num_rows)).ceil().to(torch.int32)
else:
row_ends = torch.full((num_rows,), num_cols, dtype=torch.int32, device=device)
return logits, row_starts, row_ends
def _time(fn, iters, warmup):
for _ in range(warmup):
fn()
torch.accelerator.synchronize()
start = time.perf_counter()
for _ in range(iters):
fn()
torch.accelerator.synchronize()
return (time.perf_counter() - start) / iters * 1e3
def _selected_set(indices, row_starts):
valid = indices >= 0
return [set(r[m].tolist()) for r, m in zip(indices.cpu(), valid.cpu())]
def main():
parser = argparse.ArgumentParser()
parser.add_argument("--num-rows", type=int, default=1024)
parser.add_argument("--num-cols", type=int, default=131072)
parser.add_argument("--topk", type=int, default=512)
parser.add_argument("--causal", action="store_true")
parser.add_argument("--iters", type=int, default=20)
parser.add_argument("--warmup", type=int, default=5)
args = parser.parse_args()
device = torch.device("cuda")
logits, row_starts, row_ends = _make_inputs(
args.num_rows, args.num_cols, args.causal, device
)
print(
f"rows={args.num_rows} cols={args.num_cols} k={args.topk} "
f"causal={args.causal} logits={logits.numel() * 4 / 2**20:.0f} MiB "
f"path={os.environ.get('TOPK_FORCE_PATH', 'auto')}"
)
native = torch.empty(args.num_rows, args.topk, dtype=torch.int32, device=device)
def run_native():
torch.ops._C.top_k_per_row_prefill(
logits,
row_starts,
row_ends,
native,
args.num_rows,
logits.stride(0),
logits.stride(1),
args.topk,
)
native_ms = _time(run_native, args.iters, args.warmup)
print(f"in-tree topKPerRowPrefill : {native_ms:8.3f} ms")
if not rocm_aiter_ops.is_indexer_top_k_enabled():
print(
"AITER path unavailable: requires ROCm gfx950 with AITER installed "
"and VLLM_ROCM_USE_AITER=1"
)
return
from vllm.v1.attention.ops.rocm_aiter_mla_sparse import (
_launch_aiter_top_k_per_row_prefill,
)
aiter_out = torch.empty(args.num_rows, args.topk, dtype=torch.int32, device=device)
def run_aiter():
_launch_aiter_top_k_per_row_prefill(
logits, row_starts, row_ends, aiter_out, args.topk
)
try:
aiter_ms = _time(run_aiter, args.iters, args.warmup)
except Exception as exc:
print(f"AITER path failed: {type(exc).__name__}: {exc}")
return
print(f"AITER top_k_per_row_prefill: {aiter_ms:8.3f} ms")
print(f"speedup : {native_ms / aiter_ms:.2f}x")
ref = _selected_set(native, row_starts)
got = _selected_set(aiter_out, row_starts)
mismatched = sum(1 for a, b in zip(ref, got) if a != b)
worst = max((len(a ^ b) for a, b in zip(ref, got)), default=0)
print(f"rows differing: {mismatched}/{args.num_rows} (worst sym-diff {worst})")
if __name__ == "__main__":
main()
+3 -3
View File
@@ -497,7 +497,7 @@ set(VLLM_EXT_SRC
"csrc/cpu/mla_decode.cpp"
"csrc/cpu/pos_encoding.cpp"
"csrc/cpu/mamba_cpu.cpp"
"csrc/cpu/dynamic_4bit_int_moe_cpu.cpp"
"csrc/moe/dynamic_4bit_int_moe_cpu.cpp"
"csrc/cpu/cpu_fused_moe.cpp"
"csrc/cpu/cpu_attn.cpp"
"csrc/cpu/cpu_isa.cpp"
@@ -599,7 +599,7 @@ if (ENABLE_X86_ISA)
"csrc/cpu/layernorm.cpp"
"csrc/cpu/mla_decode.cpp"
"csrc/cpu/pos_encoding.cpp"
"csrc/cpu/dynamic_4bit_int_moe_cpu.cpp")
"csrc/moe/dynamic_4bit_int_moe_cpu.cpp")
set(VLLM_EXT_SRC_AVX2
"csrc/cpu/sgl-kernels/fla.cpp"
@@ -617,7 +617,7 @@ if (ENABLE_X86_ISA)
"csrc/cpu/layernorm.cpp"
"csrc/cpu/mla_decode.cpp"
"csrc/cpu/pos_encoding.cpp"
"csrc/cpu/dynamic_4bit_int_moe_cpu.cpp")
"csrc/moe/dynamic_4bit_int_moe_cpu.cpp")
message(STATUS "CPU extension (AVX512F + BF16 + VNNI + AMX) source files: ${VLLM_EXT_SRC_AVX512} ${VLLM_EXT_SRC_SGL}")
message(STATUS "CPU extension (AVX512F) source files: ${VLLM_EXT_SRC_AVX512}")
+7 -7
View File
@@ -27,13 +27,14 @@ if(DEEPGEMM_SRC_DIR)
set(deepgemm_SOURCE_DIR "${DEEPGEMM_SRC_DIR}")
message(STATUS "DeepGEMM using local DEEPGEMM_SRC_DIR: ${deepgemm_SOURCE_DIR}")
else()
# Keep in sync with tools/install_deepgemm.sh
set(_DEEPGEMM_UPSTREAM_REPO "https://github.com/vllm-project/DeepGEMM.git")
# Pinned to the fork's dev branch: upstream's 26/09/30 release (2.8.1)
# plus the SM120 port, the SM90 paged-MQA kv_block=32/next_n=4 port,
# configurable SwiGLU alpha/beta, SiTU for FP8/FP4 Mega MoE, and NVFP4
# Mega MoE, plus swiglustep for BF16/FP8/NVFP4 Mega MoE.
# Uses the PyTorch 2.11 stable ABI and CPython 3.10 limited API.
set(_DEEPGEMM_UPSTREAM_TAG "a7a927731b109fdfec770df7c2a5be5d7e2216b5")
# Pinned to the fork's dev branch: upstream 2.8.0 plus the SM120
# port, the SM90 paged-MQA kv_block=32/next_n=4 port, configurable SwiGLU
# alpha/beta, and SiTU for FP8/FP4 Mega MoE. Also includes the CUDA 12.x
# layout header fix from vllm-project/DeepGEMM#12.
# TORCH_LIBRARY build with a CPython 3.10 stable-ABI floor.
set(_DEEPGEMM_UPSTREAM_TAG "1e1842a833699298f7afc02eefb2ed168fff6938")
set(_deepgemm_fc_root "${FETCHCONTENT_BASE_DIR}")
if(NOT _deepgemm_fc_root)
@@ -145,7 +146,6 @@ if(DEEPGEMM_ARCHS)
#
install(FILES
"${deepgemm_SOURCE_DIR}/deep_gemm/__init__.py"
"${deepgemm_SOURCE_DIR}/deep_gemm/locality_domain.py"
DESTINATION vllm/third_party/deep_gemm
COMPONENT _deep_gemm_C)
+64 -81
View File
@@ -19,7 +19,7 @@ else()
FetchContent_Declare(
flashmla
GIT_REPOSITORY https://github.com/vllm-project/FlashMLA
GIT_TAG 5dceadbd9eabc9acfc547ef18ec8f7b14b980682
GIT_TAG 0eee43b12f034b657133cf2afca6a72ebb6efccf
GIT_PROGRESS TRUE
CONFIGURE_COMMAND ""
BUILD_COMMAND ""
@@ -81,83 +81,72 @@ if(FLASH_MLA_ARCHS)
${flashmla_SOURCE_DIR}/csrc/api/dense_decode.cpp
${flashmla_SOURCE_DIR}/csrc/api/fused_norm_rope_attn_rope_cast_fwd.cpp
# Current kernels (DeepSeek V4.1, token-interleaved KV cache scales)
${flashmla_SOURCE_DIR}/csrc/cuda_kernels/smxx/decode/get_decoding_sched_meta/get_decoding_sched_meta.cu
${flashmla_SOURCE_DIR}/csrc/cuda_kernels/smxx/decode/combine/combine.cu
# sm100 dense prefill (inference-only: dense backward stays out)
${flashmla_SOURCE_DIR}/csrc/cuda_kernels/sm100/prefill/dense/fmha_cutlass_fwd_sm100.cu
# sm100 sparse prefill
${flashmla_SOURCE_DIR}/csrc/cuda_kernels/sm100/prefill/sparse/fwd/head64/instantiations/phase1_h64_k512.cu
${flashmla_SOURCE_DIR}/csrc/cuda_kernels/sm100/prefill/sparse/fwd/head128/instantiations/phase1_k512.cu
${flashmla_SOURCE_DIR}/csrc/cuda_kernels/sm100/prefill/sparse/fwd_for_small_topk/head128/instantiations/phase1_k512.cu
# sm100 sparse decode
${flashmla_SOURCE_DIR}/csrc/cuda_kernels/sm100/decode/sparse/head64/instantiations/v41_h64.cu
${flashmla_SOURCE_DIR}/csrc/cuda_kernels/sm100/decode/sparse/head64/instantiations/v41_h64_no_split.cu
${flashmla_SOURCE_DIR}/csrc/cuda_kernels/sm100/decode/sparse/head64/instantiations/v41fp4_h64.cu
${flashmla_SOURCE_DIR}/csrc/cuda_kernels/sm100/decode/sparse/head64/instantiations/v41fp4_h64_no_split.cu
${flashmla_SOURCE_DIR}/csrc/cuda_kernels/sm100/prefill/sparse/fwd_for_small_topk/head128/instantiations/phase1_decode_k512_v41.cu
${flashmla_SOURCE_DIR}/csrc/cuda_kernels/sm100/prefill/sparse/fwd_for_small_topk/head128/instantiations/phase1_decode_k512_v41_splitkv.cu
${flashmla_SOURCE_DIR}/csrc/cuda_kernels/sm100/prefill/sparse/fwd_for_small_topk/head128/instantiations/phase1_decode_k512_v41fp4.cu
${flashmla_SOURCE_DIR}/csrc/cuda_kernels/sm100/prefill/sparse/fwd_for_small_topk/head128/instantiations/phase1_decode_k512_v41fp4_splitkv.cu
# sm100 fused norm + rope + attn + rope + cast
${flashmla_SOURCE_DIR}/csrc/cuda_kernels/sm100/prefill/sparse/fused_norm_rope_attn_rope_cast_fwd/core_attn/instantiations/v41_h64_prefill_norm.cu
${flashmla_SOURCE_DIR}/csrc/cuda_kernels/sm100/prefill/sparse/fused_norm_rope_attn_rope_cast_fwd/core_attn/instantiations/v41_h64_prefill_nonorm.cu
${flashmla_SOURCE_DIR}/csrc/cuda_kernels/sm100/prefill/sparse/fused_norm_rope_attn_rope_cast_fwd/core_attn/instantiations/v41_h64_decode_norm.cu
${flashmla_SOURCE_DIR}/csrc/cuda_kernels/sm100/prefill/sparse/fused_norm_rope_attn_rope_cast_fwd/core_attn/instantiations/v41_h64_decode_nonorm.cu
${flashmla_SOURCE_DIR}/csrc/cuda_kernels/sm100/prefill/sparse/fused_norm_rope_attn_rope_cast_fwd/core_attn/instantiations/v41fp4_h64_decode_norm.cu
${flashmla_SOURCE_DIR}/csrc/cuda_kernels/sm100/prefill/sparse/fused_norm_rope_attn_rope_cast_fwd/core_attn/instantiations/v41fp4_h64_decode_nonorm.cu
${flashmla_SOURCE_DIR}/csrc/cuda_kernels/sm100/prefill/sparse/fused_norm_rope_attn_rope_cast_fwd/core_attn/instantiations/v41_h128_prefill_norm.cu
${flashmla_SOURCE_DIR}/csrc/cuda_kernels/sm100/prefill/sparse/fused_norm_rope_attn_rope_cast_fwd/core_attn/instantiations/v41_h128_prefill_nonorm.cu
${flashmla_SOURCE_DIR}/csrc/cuda_kernels/sm100/prefill/sparse/fused_norm_rope_attn_rope_cast_fwd/core_attn/instantiations/v41_h128_decode_norm.cu
${flashmla_SOURCE_DIR}/csrc/cuda_kernels/sm100/prefill/sparse/fused_norm_rope_attn_rope_cast_fwd/core_attn/instantiations/v41_h128_decode_nonorm.cu
${flashmla_SOURCE_DIR}/csrc/cuda_kernels/sm100/prefill/sparse/fused_norm_rope_attn_rope_cast_fwd/core_attn/instantiations/v41fp4_h128_decode_norm.cu
${flashmla_SOURCE_DIR}/csrc/cuda_kernels/sm100/prefill/sparse/fused_norm_rope_attn_rope_cast_fwd/core_attn/instantiations/v41fp4_h128_decode_nonorm.cu
${flashmla_SOURCE_DIR}/csrc/cuda_kernels/sm100/prefill/sparse/fused_norm_rope_attn_rope_cast_fwd/permute_q_b_proj/kernel.cu
${flashmla_SOURCE_DIR}/csrc/cuda_kernels/sm100/prefill/sparse/fused_norm_rope_attn_rope_cast_fwd/permute_wv_proj/kernel.cu
# Legacy kernels (upstream keeps them under cuda_kernels/legacy/):
# V4, V3.2 and V3.2-NVFP4, and everything on Hopper
${flashmla_SOURCE_DIR}/csrc/cuda_kernels/legacy/smxx/decode/get_decoding_sched_meta/get_decoding_sched_meta.cu
${flashmla_SOURCE_DIR}/csrc/cuda_kernels/legacy/smxx/decode/combine/combine.cu
# Misc kernels for decoding
${flashmla_SOURCE_DIR}/csrc/kernels/smxx/decode/get_decoding_sched_meta/get_decoding_sched_meta.cu
${flashmla_SOURCE_DIR}/csrc/kernels/smxx/decode/combine/combine.cu
# sm90 dense decode
${flashmla_SOURCE_DIR}/csrc/cuda_kernels/legacy/sm90/decode/dense/instantiations/fp16.cu
${flashmla_SOURCE_DIR}/csrc/cuda_kernels/legacy/sm90/decode/dense/instantiations/bf16.cu
${flashmla_SOURCE_DIR}/csrc/kernels/sm90/decode/dense/instantiations/fp16.cu
${flashmla_SOURCE_DIR}/csrc/kernels/sm90/decode/dense/instantiations/bf16.cu
# sm90 sparse decode
${flashmla_SOURCE_DIR}/csrc/cuda_kernels/legacy/sm90/decode/sparse/instantiations/v4_persistent_h64.cu
${flashmla_SOURCE_DIR}/csrc/cuda_kernels/legacy/sm90/decode/sparse/instantiations/v4_persistent_h128.cu
${flashmla_SOURCE_DIR}/csrc/cuda_kernels/legacy/sm90/decode/sparse/instantiations/v32_persistent_h64.cu
${flashmla_SOURCE_DIR}/csrc/cuda_kernels/legacy/sm90/decode/sparse/instantiations/v32_persistent_h128.cu
${flashmla_SOURCE_DIR}/csrc/kernels/sm90/decode/sparse/instantiations/v4_persistent_h64.cu
${flashmla_SOURCE_DIR}/csrc/kernels/sm90/decode/sparse/instantiations/v4_persistent_h128.cu
${flashmla_SOURCE_DIR}/csrc/kernels/sm90/decode/sparse/instantiations/v32_persistent_h64.cu
${flashmla_SOURCE_DIR}/csrc/kernels/sm90/decode/sparse/instantiations/v32_persistent_h128.cu
# sm90 sparse prefill
${flashmla_SOURCE_DIR}/csrc/cuda_kernels/legacy/sm90/prefill/sparse/instantiations/phase1_k512.cu
${flashmla_SOURCE_DIR}/csrc/cuda_kernels/legacy/sm90/prefill/sparse/instantiations/phase1_k512_topklen.cu
${flashmla_SOURCE_DIR}/csrc/cuda_kernels/legacy/sm90/prefill/sparse/instantiations/phase1_k576.cu
${flashmla_SOURCE_DIR}/csrc/cuda_kernels/legacy/sm90/prefill/sparse/instantiations/phase1_k576_topklen.cu
${flashmla_SOURCE_DIR}/csrc/kernels/sm90/prefill/sparse/instantiations/phase1_k512.cu
${flashmla_SOURCE_DIR}/csrc/kernels/sm90/prefill/sparse/instantiations/phase1_k512_topklen.cu
${flashmla_SOURCE_DIR}/csrc/kernels/sm90/prefill/sparse/instantiations/phase1_k576.cu
${flashmla_SOURCE_DIR}/csrc/kernels/sm90/prefill/sparse/instantiations/phase1_k576_topklen.cu
# sm100 sparse prefill, d_qk = 576
${flashmla_SOURCE_DIR}/csrc/cuda_kernels/legacy/sm100/prefill/sparse/fwd/head64/instantiations/phase1_h64_k576.cu
${flashmla_SOURCE_DIR}/csrc/cuda_kernels/legacy/sm100/prefill/sparse/fwd/head128/instantiations/phase1_k576.cu
# sm100 dense prefill (inference-only: dense backward stays out)
${flashmla_SOURCE_DIR}/csrc/kernels/sm100/prefill/dense/fmha_cutlass_fwd_sm100.cu
# sm100 sparse prefill
${flashmla_SOURCE_DIR}/csrc/kernels/sm100/prefill/sparse/fwd/head64/instantiations/phase1_h64_k512.cu
${flashmla_SOURCE_DIR}/csrc/kernels/sm100/prefill/sparse/fwd/head64/instantiations/phase1_h64_k576.cu
${flashmla_SOURCE_DIR}/csrc/kernels/sm100/prefill/sparse/fwd/head128/instantiations/phase1_k512.cu
${flashmla_SOURCE_DIR}/csrc/kernels/sm100/prefill/sparse/fwd/head128/instantiations/phase1_k576.cu
${flashmla_SOURCE_DIR}/csrc/kernels/sm100/prefill/sparse/fwd_for_small_topk/head128/instantiations/phase1_k512.cu
# sm100 fused norm + rope + attn + rope + cast
${flashmla_SOURCE_DIR}/csrc/kernels/sm100/prefill/sparse/fused_norm_rope_attn_rope_cast_fwd/core_attn/instantiations/v4_h64_prefill_norm.cu
${flashmla_SOURCE_DIR}/csrc/kernels/sm100/prefill/sparse/fused_norm_rope_attn_rope_cast_fwd/core_attn/instantiations/v4_h64_prefill_nonorm.cu
${flashmla_SOURCE_DIR}/csrc/kernels/sm100/prefill/sparse/fused_norm_rope_attn_rope_cast_fwd/core_attn/instantiations/v4_h128_prefill_norm.cu
${flashmla_SOURCE_DIR}/csrc/kernels/sm100/prefill/sparse/fused_norm_rope_attn_rope_cast_fwd/core_attn/instantiations/v4_h128_prefill_nonorm.cu
${flashmla_SOURCE_DIR}/csrc/kernels/sm100/prefill/sparse/fused_norm_rope_attn_rope_cast_fwd/core_attn/instantiations/v4_h64_decode_norm.cu
${flashmla_SOURCE_DIR}/csrc/kernels/sm100/prefill/sparse/fused_norm_rope_attn_rope_cast_fwd/core_attn/instantiations/v4_h64_decode_nonorm.cu
${flashmla_SOURCE_DIR}/csrc/kernels/sm100/prefill/sparse/fused_norm_rope_attn_rope_cast_fwd/core_attn/instantiations/v4_h128_decode_norm.cu
${flashmla_SOURCE_DIR}/csrc/kernels/sm100/prefill/sparse/fused_norm_rope_attn_rope_cast_fwd/core_attn/instantiations/v4_h128_decode_nonorm.cu
${flashmla_SOURCE_DIR}/csrc/kernels/sm100/prefill/sparse/fused_norm_rope_attn_rope_cast_fwd/core_attn/instantiations/v41_h64_decode_norm.cu
${flashmla_SOURCE_DIR}/csrc/kernels/sm100/prefill/sparse/fused_norm_rope_attn_rope_cast_fwd/core_attn/instantiations/v41_h64_decode_nonorm.cu
${flashmla_SOURCE_DIR}/csrc/kernels/sm100/prefill/sparse/fused_norm_rope_attn_rope_cast_fwd/core_attn/instantiations/v41_h128_decode_norm.cu
${flashmla_SOURCE_DIR}/csrc/kernels/sm100/prefill/sparse/fused_norm_rope_attn_rope_cast_fwd/core_attn/instantiations/v41_h128_decode_nonorm.cu
${flashmla_SOURCE_DIR}/csrc/kernels/sm100/prefill/sparse/fused_norm_rope_attn_rope_cast_fwd/core_attn/instantiations/v41fp4_h64_decode_norm.cu
${flashmla_SOURCE_DIR}/csrc/kernels/sm100/prefill/sparse/fused_norm_rope_attn_rope_cast_fwd/core_attn/instantiations/v41fp4_h64_decode_nonorm.cu
${flashmla_SOURCE_DIR}/csrc/kernels/sm100/prefill/sparse/fused_norm_rope_attn_rope_cast_fwd/core_attn/instantiations/v41fp4_h128_decode_norm.cu
${flashmla_SOURCE_DIR}/csrc/kernels/sm100/prefill/sparse/fused_norm_rope_attn_rope_cast_fwd/core_attn/instantiations/v41fp4_h128_decode_nonorm.cu
${flashmla_SOURCE_DIR}/csrc/kernels/sm100/prefill/sparse/fused_norm_rope_attn_rope_cast_fwd/permute_q_b_proj/kernel.cu
${flashmla_SOURCE_DIR}/csrc/kernels/sm100/prefill/sparse/fused_norm_rope_attn_rope_cast_fwd/permute_wv_proj/kernel.cu
# sm100 sparse decode
${flashmla_SOURCE_DIR}/csrc/cuda_kernels/legacy/sm100/decode/sparse/head64/instantiations/v32_h64.cu
${flashmla_SOURCE_DIR}/csrc/cuda_kernels/legacy/sm100/decode/sparse/head64/instantiations/v32_h64_no_split.cu
${flashmla_SOURCE_DIR}/csrc/cuda_kernels/legacy/sm100/decode/sparse/head64/instantiations/v4_h64.cu
${flashmla_SOURCE_DIR}/csrc/cuda_kernels/legacy/sm100/decode/sparse/head64/instantiations/v4_h64_no_split.cu
${flashmla_SOURCE_DIR}/csrc/cuda_kernels/legacy/sm100/decode/sparse/nvfp4_head64/instantiations/v32_nvfp4_fp8rope.cu
${flashmla_SOURCE_DIR}/csrc/cuda_kernels/legacy/sm100/prefill/sparse/fwd_for_small_topk/head128/instantiations/phase1_decode_k512.cu
${flashmla_SOURCE_DIR}/csrc/cuda_kernels/legacy/sm100/prefill/sparse/fwd_for_small_topk/head128/instantiations/phase1_decode_k512_splitkv.cu
# sm100 fused norm + rope + attn + rope + cast, V4 decode
${flashmla_SOURCE_DIR}/csrc/cuda_kernels/legacy/sm100/prefill/sparse/fused_norm_rope_attn_rope_cast_fwd/core_attn/instantiations/v4_h64_decode_norm.cu
${flashmla_SOURCE_DIR}/csrc/cuda_kernels/legacy/sm100/prefill/sparse/fused_norm_rope_attn_rope_cast_fwd/core_attn/instantiations/v4_h64_decode_nonorm.cu
${flashmla_SOURCE_DIR}/csrc/cuda_kernels/legacy/sm100/prefill/sparse/fused_norm_rope_attn_rope_cast_fwd/core_attn/instantiations/v4_h128_decode_norm.cu
${flashmla_SOURCE_DIR}/csrc/cuda_kernels/legacy/sm100/prefill/sparse/fused_norm_rope_attn_rope_cast_fwd/core_attn/instantiations/v4_h128_decode_nonorm.cu
${flashmla_SOURCE_DIR}/csrc/kernels/sm100/decode/sparse/head64/instantiations/v32_h64.cu
${flashmla_SOURCE_DIR}/csrc/kernels/sm100/decode/sparse/head64/instantiations/v32_h64_no_split.cu
${flashmla_SOURCE_DIR}/csrc/kernels/sm100/decode/sparse/head64/instantiations/v4_h64.cu
${flashmla_SOURCE_DIR}/csrc/kernels/sm100/decode/sparse/head64/instantiations/v4_h64_no_split.cu
${flashmla_SOURCE_DIR}/csrc/kernels/sm100/decode/sparse/head64/instantiations/v41_h64.cu
${flashmla_SOURCE_DIR}/csrc/kernels/sm100/decode/sparse/head64/instantiations/v41_h64_no_split.cu
${flashmla_SOURCE_DIR}/csrc/kernels/sm100/decode/sparse/head64/instantiations/v41fp4_h64.cu
${flashmla_SOURCE_DIR}/csrc/kernels/sm100/decode/sparse/head64/instantiations/v41fp4_h64_no_split.cu
${flashmla_SOURCE_DIR}/csrc/kernels/sm100/decode/sparse/nvfp4_head64/instantiations/v32_nvfp4_fp8rope.cu
${flashmla_SOURCE_DIR}/csrc/kernels/sm100/prefill/sparse/fwd_for_small_topk/head128/instantiations/phase1_decode_k512.cu
${flashmla_SOURCE_DIR}/csrc/kernels/sm100/prefill/sparse/fwd_for_small_topk/head128/instantiations/phase1_decode_k512_splitkv.cu
${flashmla_SOURCE_DIR}/csrc/kernels/sm100/prefill/sparse/fwd_for_small_topk/head128/instantiations/phase1_decode_k512_v41.cu
${flashmla_SOURCE_DIR}/csrc/kernels/sm100/prefill/sparse/fwd_for_small_topk/head128/instantiations/phase1_decode_k512_v41_splitkv.cu
${flashmla_SOURCE_DIR}/csrc/kernels/sm100/prefill/sparse/fwd_for_small_topk/head128/instantiations/phase1_decode_k512_v41fp4.cu
${flashmla_SOURCE_DIR}/csrc/kernels/sm100/prefill/sparse/fwd_for_small_topk/head128/instantiations/phase1_decode_k512_v41fp4_splitkv.cu
)
set(FlashMLA_Extension_SOURCES
@@ -169,17 +158,17 @@ if(FLASH_MLA_ARCHS)
set(FlashMLA_INCLUDES
${flashmla_SOURCE_DIR}/csrc
${flashmla_SOURCE_DIR}/csrc/3rdparty/kerutils/include
${flashmla_SOURCE_DIR}/csrc/3rdparty/cutlass/include
${flashmla_SOURCE_DIR}/csrc/3rdparty/cutlass/tools/util/include
${flashmla_SOURCE_DIR}/csrc/kerutils/include
${flashmla_SOURCE_DIR}/csrc/cutlass/include
${flashmla_SOURCE_DIR}/csrc/cutlass/tools/util/include
)
set(FlashMLA_Extension_INCLUDES
${flashmla_SOURCE_DIR}/csrc
${flashmla_SOURCE_DIR}/csrc/3rdparty/kerutils/include
${flashmla_SOURCE_DIR}/csrc/kerutils/include
${flashmla_SOURCE_DIR}/csrc/extension/sm90/dense_fp8/
${flashmla_SOURCE_DIR}/csrc/3rdparty/cutlass/include
${flashmla_SOURCE_DIR}/csrc/3rdparty/cutlass/tools/util/include
${flashmla_SOURCE_DIR}/csrc/cutlass/include
${flashmla_SOURCE_DIR}/csrc/cutlass/tools/util/include
)
set_gencode_flags_for_srcs(
@@ -213,12 +202,6 @@ if(FLASH_MLA_ARCHS)
target_compile_definitions(_flashmla_C PRIVATE USE_CUDA)
endif()
# Select the CUDA branch of FlashMLA's params.h and kerutils; kerutils
# detects CUDA by itself in .cu files, so only host files need the flag.
target_compile_definitions(_flashmla_C PRIVATE
FLASH_MLA_IS_BUILD_ON_CUDA
$<$<COMPILE_LANGUAGE:CXX>:KERUTILS_IS_BUILD_ON_CUDA>)
define_extension_target(
_flashmla_extension_C
DESTINATION vllm
+1 -10
View File
@@ -17,7 +17,7 @@ else()
FetchContent_Declare(
fmha_sm100
GIT_REPOSITORY https://github.com/vllm-project/MSA.git
GIT_TAG 968061ab16cb14991a5c7d4c0e8eabf531be0dcf
GIT_TAG be4a52aea420b6bef73be9af1a649859964790cb
GIT_PROGRESS TRUE
CONFIGURE_COMMAND ""
BUILD_COMMAND ""
@@ -36,14 +36,11 @@ set(FMHA_SM100_PY_ROOT "${fmha_sm100_SOURCE_DIR}/python/fmha_sm100")
install(FILES
"${FMHA_SM100_PY_ROOT}/__init__.py"
"${FMHA_SM100_PY_ROOT}/_jit_cache.py"
"${FMHA_SM100_PY_ROOT}/api.py"
"${FMHA_SM100_PY_ROOT}/bench_utils.py"
"${FMHA_SM100_PY_ROOT}/jit.py"
"${FMHA_SM100_PY_ROOT}/msa_warmup.py"
"${FMHA_SM100_PY_ROOT}/nvfp4_kv.py"
"${FMHA_SM100_PY_ROOT}/q8kv4_decode_adapter.py"
"${FMHA_SM100_PY_ROOT}/q8kv4_prefill_adapter.py"
"${FMHA_SM100_PY_ROOT}/sparse.py"
"${FMHA_SM100_PY_ROOT}/sparse_fmha_adapter.py"
DESTINATION vllm/third_party/fmha_sm100
@@ -68,12 +65,6 @@ install(DIRECTORY "${FMHA_SM100_PY_ROOT}/kvouter/"
PATTERN "__pycache__" EXCLUDE
PATTERN "*.pyc" EXCLUDE)
install(DIRECTORY "${FMHA_SM100_PY_ROOT}/prefill_q8kv4/"
DESTINATION vllm/third_party/fmha_sm100/prefill_q8kv4
COMPONENT fmha_sm100
PATTERN "__pycache__" EXCLUDE
PATTERN "*.pyc" EXCLUDE)
install(DIRECTORY "${FMHA_SM100_PY_ROOT}/cute/"
DESTINATION vllm/third_party/fmha_sm100/cute
COMPONENT fmha_sm100
@@ -39,7 +39,7 @@ else()
FetchContent_Declare(
vllm-flash-attn
GIT_REPOSITORY https://github.com/vllm-project/flash-attention.git
GIT_TAG b273a02621d3c94bc747df67204f591b67f38fae
GIT_TAG 9cd61de38763d712bb6ce56e2a02cc2bf718c89f
GIT_PROGRESS TRUE
# Don't share the vllm-flash-attn build between build types
BINARY_DIR ${CMAKE_BINARY_DIR}/vllm-flash-attn
+1 -1
View File
@@ -65,7 +65,7 @@ if __name__ == "__main__":
project_directory=args.project_dir,
output_directory=args.output_dir,
# Hipify resolves quoted includes next to the including file first; vLLM
# uses paths relative to csrc/ (e.g. "torch_utils.h"
# uses paths relative to csrc/ (e.g. "libtorch_stable/torch_utils.h"
# from quantization/w8a8/fp8/*.cu). Without an include root here, those
# headers are never found and are not hipified or rewritten in dependents.
header_include_dirs=["."],
+1 -4
View File
@@ -425,7 +425,6 @@ function(cuda_archs_loose_intersection OUT_CUDA_ARCHS SRC_CUDA_ARCHS TGT_CUDA_AR
if("${_base}" IN_LIST _TGT_CUDA_ARCHS)
list(REMOVE_ITEM _SRC_CUDA_ARCHS "${_arch}")
list(REMOVE_ITEM _TGT_CUDA_ARCHS "${_base}")
list(REMOVE_ITEM _TGT_CUDA_ARCHS "${_base}a" "${_base}f")
list(APPEND _CUDA_ARCHS "${_arch}")
endif()
endif()
@@ -465,10 +464,8 @@ function(cuda_archs_loose_intersection OUT_CUDA_ARCHS SRC_CUDA_ARCHS TGT_CUDA_AR
string(REGEX REPLACE "[af]$" "" _base "${_arch}")
if ("${_base}" IN_LIST _SRC_CUDA_ARCHS)
list(REMOVE_ITEM _TGT_CUDA_ARCHS "${_arch}")
list(REMOVE_ITEM _SRC_CUDA_ARCHS "${_base}")
list(APPEND _CUDA_ARCHS "${_arch}")
if (NOT "${_base}" IN_LIST _TGT_CUDA_ARCHS)
list(REMOVE_ITEM _SRC_CUDA_ARCHS "${_base}")
endif()
endif()
endif()
endforeach()
@@ -1,6 +1,6 @@
#pragma once
#include "dtype_generic.cuh"
#include "attention_generic.cuh"
#include "dtype_float16.cuh"
#include "dtype_float32.cuh"
#include "dtype_bfloat16.cuh"
@@ -20,7 +20,7 @@
*/
#pragma once
#include "dtype_generic.cuh"
#include "attention_generic.cuh"
#include "dtype_float32.cuh"
#ifndef USE_ROCM
@@ -20,7 +20,7 @@
*/
#pragma once
#include "dtype_generic.cuh"
#include "attention_generic.cuh"
#include "dtype_float32.cuh"
#ifdef USE_ROCM
@@ -20,7 +20,7 @@
*/
#pragma once
#include "dtype_generic.cuh"
#include "attention_generic.cuh"
#include <stdint.h>
@@ -1,8 +1,7 @@
#pragma once
#include "dtype_generic.cuh"
#include <torch/headeronly/util/Exception.h>
#include "attention_generic.cuh"
#include "torch_utils.h"
#include <stdint.h>
#ifdef ENABLE_FP8
@@ -32,7 +31,7 @@ inline Fp8KVCacheDataType get_fp8_kv_cache_data_type(
} else if (dtype_str == "fp8_e5m2") {
return Fp8KVCacheDataType::kFp8E5M2;
}
STD_TORCH_CHECK(false, "Unsupported fp8 kv cache data type: ", dtype_str);
TORCH_UTILS_CHECK(false, "Unsupported fp8 kv cache data type: ", dtype_str);
}
// fp8 vector types for quantization of kv cache
+93
View File
@@ -0,0 +1,93 @@
#pragma once
#include <torch/all.h>
#include <c10/util/Optional.h>
#include <map>
#include <vector>
void swap_blocks(torch::Tensor& src, torch::Tensor& dst,
int64_t block_size_in_bytes,
const torch::Tensor& block_mapping);
void swap_blocks_batch(const torch::Tensor& src_ptrs,
const torch::Tensor& dst_ptrs,
const torch::Tensor& sizes,
bool is_src_access_order_any);
void reshape_and_cache(torch::Tensor& key, torch::Tensor& value,
torch::Tensor& key_cache, torch::Tensor& value_cache,
torch::Tensor& slot_mapping,
const std::string& kv_cache_dtype,
torch::Tensor& k_scale, torch::Tensor& v_scale);
void reshape_and_cache_flash(torch::Tensor& key, torch::Tensor& value,
torch::Tensor& key_cache,
torch::Tensor& value_cache,
torch::Tensor& slot_mapping,
const std::string& kv_cache_dtype,
torch::Tensor& k_scale, torch::Tensor& v_scale);
void concat_and_cache_mla(torch::Tensor& kv_c, torch::Tensor& k_pe,
torch::Tensor& kv_cache, torch::Tensor& slot_mapping,
const std::string& kv_cache_dtype,
torch::Tensor& scale);
// NOTE: k_pe and kv_c order is flipped compared to concat_and_cache_mla
void concat_and_cache_mla_rope_fused(
torch::Tensor& positions, torch::Tensor& q_pe, torch::Tensor& k_pe,
torch::Tensor& kv_c, torch::Tensor& rope_cos_sin_cache, bool rope_is_neox,
torch::Tensor& kv_cache_slot_mapping, torch::Tensor& kv_cache,
const std::string& kv_cache_dtype, torch::Tensor& kv_cache_quant_scale);
// Just for unittest
void convert_fp8(torch::Tensor& dst_cache, torch::Tensor& src_cache,
const double scale, const std::string& kv_cache_dtype);
void gather_and_maybe_dequant_cache(
torch::Tensor const& src_cache, // [NUM_BLOCKS, BLOCK_SIZE, ENTRIES...]
torch::Tensor const& dst, // [TOT_TOKENS, ENTRIES...]
torch::Tensor const& block_table, // [BATCH, BLOCK_INDICES]
torch::Tensor const& cu_seq_lens, // [BATCH+1]
torch::Tensor const& token_to_seq, // [MAX_TOKEN_ACROSS_CHUNKS]
int64_t num_tokens, const std::string& kv_cache_dtype,
torch::Tensor const& scale,
std::optional<torch::Tensor> seq_starts = std::nullopt);
// TODO(hc): cp_gather_cache need support scaled kvcahe in the future.
void cp_gather_cache(
torch::Tensor const& src_cache, // [NUM_BLOCKS, BLOCK_SIZE, ENTRIES...]
torch::Tensor const& dst, // [TOT_TOKENS, ENTRIES...]
torch::Tensor const& block_table, // [BATCH, BLOCK_INDICES]
torch::Tensor const& cu_seq_lens, // [BATCH+1]
int64_t batch_size, std::optional<torch::Tensor> seq_starts = std::nullopt);
// Gather and upconvert FP8 KV cache to BF16 workspace
void cp_gather_and_upconvert_fp8_kv_cache(
torch::Tensor const& src_cache, // [NUM_BLOCKS, BLOCK_SIZE, 656]
torch::Tensor const& dst, // [TOT_TOKENS, 576]
torch::Tensor const& block_table, // [BATCH, BLOCK_INDICES]
torch::Tensor const& workspace_starts, // [BATCH]
int64_t batch_size, std::optional<torch::Tensor> seq_starts = std::nullopt);
// Indexer K quantization and cache function
void indexer_k_quant_and_cache(
torch::Tensor& k, // [num_tokens, head_dim]
torch::Tensor& kv_cache, // [num_blocks, block_size, cache_stride]
torch::Tensor& slot_mapping, // [num_tokens]
int64_t quant_block_size, // quantization block size
const std::string& scale_fmt);
// Concatenate query nope and rope for MLA/DSA attention
void concat_mla_q(
torch::Tensor& ql_nope, // [num_tokens, num_heads, nope_dim]
torch::Tensor& q_pe, // [num_tokens, num_heads, rope_dim]
torch::Tensor& q_out); // [num_tokens, num_heads, nope_dim + rope_dim]
// Extract function to gather quantized K cache
void cp_gather_indexer_k_quant_cache(
const torch::Tensor& kv_cache, // [num_blocks, block_size, cache_stride]
torch::Tensor& dst_k, // [num_tokens, head_dim]
torch::Tensor& dst_scale, // [num_tokens, head_dim / quant_block_size * 4]
const torch::Tensor& block_table, // [batch_size, num_blocks]
const torch::Tensor& cu_seq_lens); // [batch_size + 1]
+18
View File
@@ -0,0 +1,18 @@
#pragma once
#include <cstdlib>
#include <string>
namespace vllm {
// vllm_is_batch_invariant(); returns true
// if env VLLM_BATCH_INVARIANT=1
inline bool vllm_is_batch_invariant() {
static bool cached = []() {
std::string env_key = "VLLM_BATCH_INVARIANT";
const char* val = std::getenv(env_key.c_str());
return (val && std::atoi(val) != 0) ? 1 : 0;
}();
return cached;
}
} // namespace vllm
+3
View File
@@ -0,0 +1,3 @@
#pragma once
#define VLLM_IMPLIES(p, q) (!(p) || (q))
+5
View File
@@ -12,6 +12,11 @@
// could be a macro instead of a literal token.
#define TORCH_LIBRARY_EXPAND(NAME, MODULE) TORCH_LIBRARY(NAME, MODULE)
// A version of the TORCH_LIBRARY_IMPL macro that expands the NAME, i.e. so NAME
// could be a macro instead of a literal token.
#define TORCH_LIBRARY_IMPL_EXPAND(NAME, DEVICE, MODULE) \
TORCH_LIBRARY_IMPL(NAME, DEVICE, MODULE)
// REGISTER_EXTENSION allows the shared library to be loaded and initialized
// via python's import statement.
#define REGISTER_EXTENSION(NAME) \
+1 -1
View File
@@ -14,7 +14,7 @@
namespace cpu_attention {
enum class ISA { AMX, VEC, VEC16, NEON, VXE, RVV, VSX, AMX_FP8 };
// Mirrors csrc/core/vec_dtypes/dtype_fp8.cuh Fp8KVCacheDataType exactly.
// Mirrors csrc/attention/dtype_fp8.cuh Fp8KVCacheDataType exactly.
enum class Fp8KVCacheDataType {
kAuto = 0,
kFp8E4M3 = 1,
-46
View File
@@ -1,46 +0,0 @@
#pragma once
#include <optional>
#include <string>
#include <torch/library.h>
#include <tuple>
#include <vector>
void rms_norm(torch::Tensor& out, torch::Tensor& input,
std::optional<torch::Tensor> weight, double epsilon);
void fused_add_rms_norm(torch::Tensor& input, torch::Tensor& residual,
std::optional<torch::Tensor> weight, double epsilon);
void rotary_embedding(torch::Tensor& positions, torch::Tensor& query,
std::optional<torch::Tensor> key, int64_t head_size,
torch::Tensor& cos_sin_cache, bool is_neox,
int64_t rope_dim_offset, bool inverse);
void silu_and_mul(torch::Tensor& out, torch::Tensor& input);
void gelu_and_mul(torch::Tensor& out, torch::Tensor& input);
void gelu_tanh_and_mul(torch::Tensor& out, torch::Tensor& input);
void gelu_tanh(torch::Tensor& out, torch::Tensor& input);
void gelu_new(torch::Tensor& out, torch::Tensor& input);
void gelu_fast(torch::Tensor& out, torch::Tensor& input);
void gelu_quick(torch::Tensor& out, torch::Tensor& input);
void static_scaled_int8_quant(torch::Tensor& out, torch::Tensor const& input,
torch::Tensor const& scale,
std::optional<torch::Tensor> const& azp);
void dynamic_scaled_int8_quant(torch::Tensor& out, torch::Tensor const& input,
torch::Tensor& scales,
std::optional<torch::Tensor> const& azp);
torch::Tensor dynamic_4bit_int_moe_cpu(
torch::Tensor x, torch::Tensor topk_ids, torch::Tensor topk_weights,
torch::Tensor w13_packed, torch::Tensor w2_packed, int64_t hidden_size,
int64_t intermediate_size, int64_t group_size,
bool apply_router_weight_on_input, int64_t activation_kind);
+1 -1
View File
@@ -1,4 +1,4 @@
#include <torch/all.h>
#include "cache.h"
#include "ops.h"
#include "core/registration.h"
@@ -54,6 +54,12 @@ struct Utils {
__shfl_xor(var, lane_mask, width)
#endif
#ifndef USE_ROCM
#define VLLM_SHFL_SYNC(var, src_lane) __shfl_sync(uint32_t(-1), var, src_lane)
#else
#define VLLM_SHFL_SYNC(var, src_lane) __shfl(var, src_lane)
#endif
#ifndef USE_ROCM
#define VLLM_SHFL_DOWN_SYNC(var, lane_delta) \
__shfl_down_sync(uint32_t(-1), var, lane_delta)
+24 -1
View File
@@ -1,8 +1,21 @@
#pragma once
#include <cstdint>
#include <stdio.h>
#if defined(__HIPCC__)
#define HOST_DEVICE_INLINE __host__ __device__
#define DEVICE_INLINE __device__
#define HOST_INLINE __host__
#elif defined(__CUDACC__) || defined(_NVHPC_CUDA)
#define HOST_DEVICE_INLINE __host__ __device__ __forceinline__
#define DEVICE_INLINE __device__ __forceinline__
#define HOST_INLINE __host__ __forceinline__
#else
#define HOST_DEVICE_INLINE inline
#define DEVICE_INLINE inline
#define HOST_INLINE inline
#endif
#define CUDA_CHECK(cmd) \
do { \
cudaError_t e = cmd; \
@@ -16,3 +29,13 @@
int64_t get_device_attribute(int64_t attribute, int64_t device_id);
int64_t get_max_shared_memory_per_block_device_attribute(int64_t device_id);
namespace cuda_utils {
template <typename T>
HOST_DEVICE_INLINE constexpr std::enable_if_t<std::is_integral_v<T>, T>
ceil_div(T a, T b) {
return (a + b - 1) / b;
}
}; // namespace cuda_utils
+31
View File
@@ -216,6 +216,21 @@ DINLINE void barrier_at_start(const RankSignals& sg, Signal* self_sg,
if (threadIdx.x == 0) self_sg->_flag[blockIdx.x] = flag;
}
template <int ngpus>
DINLINE void barrier_at_start_release(const RankSignals& sg, Signal* self_sg,
int rank) {
__syncthreads();
uint32_t flag = self_sg->_flag[blockIdx.x] + 1;
if (threadIdx.x < ngpus) {
auto peer_counter_ptr = &sg.signals[threadIdx.x]->start[blockIdx.x][rank];
auto self_counter_ptr = &self_sg->start[blockIdx.x][threadIdx.x];
st_flag_release(peer_counter_ptr, flag);
while (ld_flag_acquire(self_counter_ptr) != flag);
}
__syncthreads();
if (threadIdx.x == 0) self_sg->_flag[blockIdx.x] = flag;
}
// This function is meant to be used as the second or the final
// synchronization barrier in the all reduce kernel. If it's the final
// synchronization barrier, we don't need to make any visibility guarantees
@@ -264,6 +279,22 @@ DINLINE void barrier_at_start(const RankSignals& sg, Signal* self_sg,
if (threadIdx.x == 0) self_sg->_flag[blockIdx.x] = flag;
}
template <int ngpus>
DINLINE void barrier_at_start_release(const RankSignals& sg, Signal* self_sg,
int rank) {
__syncthreads();
uint32_t flag = self_sg->_flag[blockIdx.x] + 1;
if (threadIdx.x < ngpus) {
__scoped_atomic_store_n(&sg.signals[threadIdx.x]->start[blockIdx.x][rank],
flag, __ATOMIC_RELEASE, __MEMORY_SCOPE_SYSTEM);
while (__scoped_atomic_load_n(&self_sg->start[blockIdx.x][threadIdx.x],
__ATOMIC_ACQUIRE,
__MEMORY_SCOPE_DEVICE) < flag);
}
__syncthreads();
if (threadIdx.x == 0) self_sg->_flag[blockIdx.x] = flag;
}
template <int ngpus, bool final_sync = false>
DINLINE void barrier_at_end(const RankSignals& sg, Signal* self_sg, int rank) {
__syncthreads();
@@ -5,7 +5,7 @@
#ifdef USE_ROCM
#include "rocm/quickreduce/quick_reduce.h"
#include "quickreduce/quick_reduce.h"
quickreduce::fptr_t init_custom_qr(int64_t rank, int64_t world_size,
std::optional<int64_t> qr_max_size) {
+1 -1
View File
@@ -2,7 +2,7 @@
#include "cutlass/half.h"
#include "cuda_bf16.h"
#include "vllm_custom_types.cuh"
#include "cutlass_extensions/vllm_custom_types.cuh"
namespace cutlass {
+129 -159
View File
@@ -1,188 +1,158 @@
/*
* Stable ABI compatible dispatch utilities for vLLM.
* Adapted from dispatch_utils.h to use PyTorch's header-only (THO_*) macros
* instead of the ATen (AT_*) macros.
*
* These macros use:
* - THO_DISPATCH_SWITCH instead of AT_DISPATCH_SWITCH
* - THO_DISPATCH_CASE instead of AT_DISPATCH_CASE
* - torch::headeronly::ScalarType instead of at::ScalarType
*
* Add more macros here as needed when migrating additional kernels.
* Adapted from
* https://github.com/pytorch/pytorch/blob/v2.0.1/aten/src/ATen/Dispatch.h
*/
#pragma once
#include <torch/headeronly/core/Dispatch.h>
#include <torch/headeronly/core/ScalarType.h>
#include <torch/headeronly/util/Exception.h>
#include <torch/all.h>
// Need a special dispatch case macro since we will nest the FP8 dispatch.
// Instead of the usual 'scalar_t', this names the dispatched type 'fp8_t'.
#define VLLM_STABLE_DISPATCH_FP8_CASE(enum_type, ...) \
THO_PRIVATE_CASE_TYPE_USING_HINT(enum_type, fp8_t, __VA_ARGS__)
#define AT_DISPATCH_FP8_CASE(enum_type, ...) \
AT_PRIVATE_CASE_TYPE_USING_HINT(enum_type, fp8_t, __VA_ARGS__)
// Same idea, for dispatching on an int32/int64 index tensor (e.g. topk_ids)
// nested inside a value-type dispatch. Named 'idx_t' instead of 'scalar_t'.
#define VLLM_STABLE_DISPATCH_IDX_CASE(enum_type, ...) \
THO_PRIVATE_CASE_TYPE_USING_HINT(enum_type, idx_t, __VA_ARGS__)
#define VLLM_DISPATCH_CASE_FLOATING_TYPES(...) \
AT_DISPATCH_CASE(at::ScalarType::Float, __VA_ARGS__) \
AT_DISPATCH_CASE(at::ScalarType::Half, __VA_ARGS__) \
AT_DISPATCH_CASE(at::ScalarType::BFloat16, __VA_ARGS__)
#define VLLM_STABLE_DISPATCH_CASE_IDX_TYPES(...) \
VLLM_STABLE_DISPATCH_IDX_CASE(torch::headeronly::ScalarType::Int, \
__VA_ARGS__) \
VLLM_STABLE_DISPATCH_IDX_CASE(torch::headeronly::ScalarType::Long, \
__VA_ARGS__)
#define VLLM_DISPATCH_FLOATING_TYPES(TYPE, NAME, ...) \
AT_DISPATCH_SWITCH(TYPE, NAME, VLLM_DISPATCH_CASE_FLOATING_TYPES(__VA_ARGS__))
#define VLLM_STABLE_DISPATCH_IDX_TYPES(TYPE, NAME, ...) \
THO_DISPATCH_SWITCH(TYPE, NAME, \
VLLM_STABLE_DISPATCH_CASE_IDX_TYPES(__VA_ARGS__))
#define VLLM_DISPATCH_CASE_HALF_TYPES(...) \
AT_DISPATCH_CASE(at::ScalarType::Half, __VA_ARGS__) \
AT_DISPATCH_CASE(at::ScalarType::BFloat16, __VA_ARGS__)
#define VLLM_STABLE_DISPATCH_CASE_FLOATING_TYPES(...) \
THO_DISPATCH_CASE(torch::headeronly::ScalarType::Float, __VA_ARGS__) \
THO_DISPATCH_CASE(torch::headeronly::ScalarType::Half, __VA_ARGS__) \
THO_DISPATCH_CASE(torch::headeronly::ScalarType::BFloat16, __VA_ARGS__)
#define VLLM_DISPATCH_HALF_TYPES(TYPE, NAME, ...) \
AT_DISPATCH_SWITCH(TYPE, NAME, VLLM_DISPATCH_CASE_HALF_TYPES(__VA_ARGS__))
#define VLLM_STABLE_DISPATCH_FLOATING_TYPES(TYPE, NAME, ...) \
THO_DISPATCH_SWITCH(TYPE, NAME, \
VLLM_STABLE_DISPATCH_CASE_FLOATING_TYPES(__VA_ARGS__))
#define VLLM_STABLE_DISPATCH_CASE_INTEGRAL_TYPES(...) \
THO_DISPATCH_CASE(torch::headeronly::ScalarType::Byte, __VA_ARGS__) \
THO_DISPATCH_CASE(torch::headeronly::ScalarType::Char, __VA_ARGS__) \
THO_DISPATCH_CASE(torch::headeronly::ScalarType::Short, __VA_ARGS__) \
THO_DISPATCH_CASE(torch::headeronly::ScalarType::Int, __VA_ARGS__) \
THO_DISPATCH_CASE(torch::headeronly::ScalarType::Long, __VA_ARGS__)
#define VLLM_STABLE_DISPATCH_CASE_INTEGRAL_AND_UNSIGNED_TYPES(...) \
VLLM_STABLE_DISPATCH_CASE_INTEGRAL_TYPES(__VA_ARGS__) \
THO_DISPATCH_CASE(torch::headeronly::ScalarType::UInt16, __VA_ARGS__) \
THO_DISPATCH_CASE(torch::headeronly::ScalarType::UInt32, __VA_ARGS__) \
THO_DISPATCH_CASE(torch::headeronly::ScalarType::UInt64, __VA_ARGS__)
#define VLLM_STABLE_DISPATCH_INTEGRAL_TYPES(TYPE, NAME, ...) \
THO_DISPATCH_SWITCH(TYPE, NAME, \
VLLM_STABLE_DISPATCH_CASE_INTEGRAL_TYPES(__VA_ARGS__))
#define VLLM_STABLE_DISPATCH_INTEGRAL_AND_UNSIGNED_TYPES(TYPE, NAME, ...) \
THO_DISPATCH_SWITCH( \
TYPE, NAME, \
VLLM_STABLE_DISPATCH_CASE_INTEGRAL_AND_UNSIGNED_TYPES(__VA_ARGS__))
// FP8 type dispatch - ROCm uses FNUZ format, CUDA uses OCP format
// ROCm devices might use either fn or fnuz, so set up dispatch table for both.
// A host-based check at runtime will create a preferred FP8 type for ROCm
// such that the correct kernel is dispatched.
#ifdef USE_ROCM
#define VLLM_STABLE_DISPATCH_CASE_FP8_TYPES(...) \
VLLM_STABLE_DISPATCH_FP8_CASE( \
torch::headeronly::ScalarType::Float8_e4m3fn, __VA_ARGS__) \
VLLM_STABLE_DISPATCH_FP8_CASE( \
torch::headeronly::ScalarType::Float8_e4m3fnuz, __VA_ARGS__)
#define VLLM_DISPATCH_CASE_FP8_TYPES(...) \
AT_DISPATCH_FP8_CASE(at::ScalarType::Float8_e4m3fn, __VA_ARGS__) \
AT_DISPATCH_FP8_CASE(at::ScalarType::Float8_e4m3fnuz, __VA_ARGS__)
#define VLLM_DISPATCH_CASE_QUANT_TYPES(...) \
AT_DISPATCH_CASE(at::ScalarType::Float8_e4m3fn, __VA_ARGS__) \
AT_DISPATCH_CASE(at::ScalarType::Float8_e4m3fnuz, __VA_ARGS__) \
AT_DISPATCH_CASE(at::ScalarType::Char, __VA_ARGS__)
#else
#define VLLM_STABLE_DISPATCH_CASE_FP8_TYPES(...) \
VLLM_STABLE_DISPATCH_FP8_CASE( \
torch::headeronly::ScalarType::Float8_e4m3fn, __VA_ARGS__)
#define VLLM_DISPATCH_CASE_FP8_TYPES(...) \
AT_DISPATCH_FP8_CASE(at::ScalarType::Float8_e4m3fn, __VA_ARGS__)
#define VLLM_DISPATCH_CASE_QUANT_TYPES(...) \
AT_DISPATCH_CASE(at::ScalarType::Float8_e4m3fn, __VA_ARGS__) \
AT_DISPATCH_CASE(at::ScalarType::Char, __VA_ARGS__)
#endif
// When using this dispatch macro, the type is 'fp8_t' not 'scalar_t'.
// See VLLM_STABLE_DISPATCH_FP8_CASE above.
#define VLLM_STABLE_DISPATCH_FP8_TYPES(TYPE, NAME, ...) \
THO_DISPATCH_SWITCH(TYPE, NAME, \
VLLM_STABLE_DISPATCH_CASE_FP8_TYPES(__VA_ARGS__))
// See AT_DISPATCH_FP8_CASE above.
#define VLLM_DISPATCH_FP8_TYPES(TYPE, NAME, ...) \
AT_DISPATCH_SWITCH(TYPE, NAME, VLLM_DISPATCH_CASE_FP8_TYPES(__VA_ARGS__))
// Half types dispatch (Half + BFloat16)
#define VLLM_STABLE_DISPATCH_CASE_HALF_TYPES(...) \
THO_DISPATCH_CASE(torch::headeronly::ScalarType::Half, __VA_ARGS__) \
THO_DISPATCH_CASE(torch::headeronly::ScalarType::BFloat16, __VA_ARGS__)
#define VLLM_DISPATCH_QUANT_TYPES(TYPE, NAME, ...) \
AT_DISPATCH_SWITCH(TYPE, NAME, VLLM_DISPATCH_CASE_QUANT_TYPES(__VA_ARGS__))
#define VLLM_STABLE_DISPATCH_HALF_TYPES(TYPE, NAME, ...) \
THO_DISPATCH_SWITCH(TYPE, NAME, \
VLLM_STABLE_DISPATCH_CASE_HALF_TYPES(__VA_ARGS__))
#define VLLM_DISPATCH_CASE_FLOATING_AND_BYTE_TYPES(...) \
AT_DISPATCH_CASE(at::ScalarType::Float, __VA_ARGS__) \
AT_DISPATCH_CASE(at::ScalarType::Half, __VA_ARGS__) \
AT_DISPATCH_CASE(at::ScalarType::BFloat16, __VA_ARGS__) \
AT_DISPATCH_CASE(at::ScalarType::Byte, __VA_ARGS__)
// Quant type dispatch (FP8 + INT8)
#ifdef USE_ROCM
#define VLLM_STABLE_DISPATCH_CASE_QUANT_TYPES(...) \
THO_DISPATCH_CASE(torch::headeronly::ScalarType::Float8_e4m3fn, \
__VA_ARGS__) \
THO_DISPATCH_CASE(torch::headeronly::ScalarType::Float8_e4m3fnuz, \
__VA_ARGS__) \
THO_DISPATCH_CASE(torch::headeronly::ScalarType::Char, __VA_ARGS__)
#else
#define VLLM_STABLE_DISPATCH_CASE_QUANT_TYPES(...) \
THO_DISPATCH_CASE(torch::headeronly::ScalarType::Float8_e4m3fn, \
__VA_ARGS__) \
THO_DISPATCH_CASE(torch::headeronly::ScalarType::Char, __VA_ARGS__)
#endif
#define VLLM_DISPATCH_FLOATING_AND_BYTE_TYPES(TYPE, NAME, ...) \
AT_DISPATCH_SWITCH(TYPE, NAME, \
VLLM_DISPATCH_CASE_FLOATING_AND_BYTE_TYPES(__VA_ARGS__))
#define VLLM_STABLE_DISPATCH_QUANT_TYPES(TYPE, NAME, ...) \
THO_DISPATCH_SWITCH(TYPE, NAME, \
VLLM_STABLE_DISPATCH_CASE_QUANT_TYPES(__VA_ARGS__))
#define VLLM_DISPATCH_CASE_INTEGRAL_TYPES(...) \
AT_DISPATCH_CASE(at::ScalarType::Byte, __VA_ARGS__) \
AT_DISPATCH_CASE(at::ScalarType::Char, __VA_ARGS__) \
AT_DISPATCH_CASE(at::ScalarType::Short, __VA_ARGS__) \
AT_DISPATCH_CASE(at::ScalarType::Int, __VA_ARGS__) \
AT_DISPATCH_CASE(at::ScalarType::Long, __VA_ARGS__)
// Group size dispatch (pure C++ if/else, no ATen dependency)
#define VLLM_STABLE_DISPATCH_GROUP_SIZE(group_size, const_group_size, ...) \
if (group_size == 128) { \
constexpr int const_group_size = 128; \
__VA_ARGS__(); \
} else if (group_size == 64) { \
constexpr int const_group_size = 64; \
__VA_ARGS__(); \
#define VLLM_DISPATCH_CASE_INTEGRAL_AND_UNSIGNED_TYPES(...) \
AT_DISPATCH_CASE(at::ScalarType::Byte, __VA_ARGS__) \
AT_DISPATCH_CASE(at::ScalarType::Char, __VA_ARGS__) \
AT_DISPATCH_CASE(at::ScalarType::Short, __VA_ARGS__) \
AT_DISPATCH_CASE(at::ScalarType::Int, __VA_ARGS__) \
AT_DISPATCH_CASE(at::ScalarType::Long, __VA_ARGS__) \
AT_DISPATCH_CASE(at::ScalarType::UInt16, __VA_ARGS__) \
AT_DISPATCH_CASE(at::ScalarType::UInt32, __VA_ARGS__) \
AT_DISPATCH_CASE(at::ScalarType::UInt64, __VA_ARGS__)
#define VLLM_DISPATCH_INTEGRAL_TYPES(TYPE, NAME, ...) \
AT_DISPATCH_SWITCH(TYPE, NAME, VLLM_DISPATCH_CASE_INTEGRAL_TYPES(__VA_ARGS__))
#define VLLM_DISPATCH_INTEGRAL_AND_UNSIGNED_TYPES(TYPE, NAME, ...) \
AT_DISPATCH_SWITCH( \
TYPE, NAME, VLLM_DISPATCH_CASE_INTEGRAL_AND_UNSIGNED_TYPES(__VA_ARGS__))
#define VLLM_DISPATCH_VEC_SIZE(VEC_SIZE, ...) \
switch (VEC_SIZE) { \
case 16: { \
constexpr int vec_size = 16; \
__VA_ARGS__(); \
break; \
} \
case 8: { \
constexpr int vec_size = 8; \
__VA_ARGS__(); \
break; \
} \
case 4: { \
constexpr int vec_size = 4; \
__VA_ARGS__(); \
break; \
} \
case 2: { \
constexpr int vec_size = 2; \
__VA_ARGS__(); \
break; \
} \
default: { \
constexpr int vec_size = 1; \
__VA_ARGS__(); \
break; \
} \
}
// Boolean dispatch
#define VLLM_STABLE_DISPATCH_BOOL(expr, const_expr, ...) \
if (expr) { \
constexpr bool const_expr = true; \
__VA_ARGS__(); \
} else { \
constexpr bool const_expr = false; \
__VA_ARGS__(); \
#define VLLM_DISPATCH_BOOL(expr, const_expr, ...) \
if (expr) { \
constexpr bool const_expr = true; \
__VA_ARGS__(); \
} else { \
constexpr bool const_expr = false; \
__VA_ARGS__(); \
}
// Vec size dispatch (pure C++ switch, no ATen dependency)
#define VLLM_STABLE_DISPATCH_VEC_SIZE(VEC_SIZE, ...) \
switch (VEC_SIZE) { \
case 16: { \
constexpr int vec_size = 16; \
__VA_ARGS__(); \
break; \
} \
case 8: { \
constexpr int vec_size = 8; \
__VA_ARGS__(); \
break; \
} \
case 4: { \
constexpr int vec_size = 4; \
__VA_ARGS__(); \
break; \
} \
case 2: { \
constexpr int vec_size = 2; \
__VA_ARGS__(); \
break; \
} \
default: { \
constexpr int vec_size = 1; \
__VA_ARGS__(); \
break; \
} \
#define VLLM_DISPATCH_GROUP_SIZE(group_size, const_group_size, ...) \
if (group_size == 128) { \
constexpr int const_group_size = 128; \
__VA_ARGS__(); \
} else if (group_size == 64) { \
constexpr int const_group_size = 64; \
__VA_ARGS__(); \
}
// Tensor rank dispatch (2D, 3D, 4D)
#define VLLM_STABLE_DISPATCH_RANK234(NUM_DIMS, ...) \
switch (NUM_DIMS) { \
case 2: { \
constexpr int tensor_rank = 2; \
__VA_ARGS__(); \
break; \
} \
case 3: { \
constexpr int tensor_rank = 3; \
__VA_ARGS__(); \
break; \
} \
case 4: { \
constexpr int tensor_rank = 4; \
__VA_ARGS__(); \
break; \
} \
default: \
STD_TORCH_CHECK( \
false, "Expects rank 2, 3 or 4 tensors but got unsupported rank"); \
#define VLLM_DISPATCH_RANK234(NUM_DIMS, ...) \
switch (NUM_DIMS) { \
case 2: { \
constexpr int tensor_rank = 2; \
__VA_ARGS__(); \
break; \
} \
case 3: { \
constexpr int tensor_rank = 3; \
__VA_ARGS__(); \
break; \
} \
case 4: { \
constexpr int tensor_rank = 4; \
__VA_ARGS__(); \
break; \
} \
default: \
TORCH_CHECK(false, "Expects rank 2, 3 or 4 tensors but got ", NUM_DIMS); \
}
@@ -1,752 +0,0 @@
/*
* SPDX-License-Identifier: Apache-2.0
* SPDX-FileCopyrightText: Copyright contributors to the vLLM project
*
* Fused Muse Spark attention pre-processing: weightless QK RMSNorm, GPT-J
* interleaved RoPE (skipped on NoPE layers, ``cos_sin_cache == None``) and
* the K/V insert of ``reshape_and_cache_flash``, over ``[q | k | v | o_gate]``
* rows.
*
* A group of ``head_dim * sizeof(scalar_t) / 16`` lanes owns one head (8 lanes
* for bf16 head_dim=64), so each lane moves one 16-byte vector, the rotated
* pair (2i, 2i+1) is lane-local, and the norm reduction is 3 shuffles.
*
* Slot enumeration per token:
* [0, nq) Q heads -> norm (+RoPE), in place or to
* ``q_fp8_out``
* [nq, nq+nkv) K heads -> norm (+RoPE), in place and into
* the K cache
* [nq+nkv, nq+2*nkv) V heads -> copied into the V cache (no norm)
* V slots exist only when a KV cache is bound.
*
* Norm and rotation run in fp32 and round once at the store. Both fp8 paths
* quantize after rounding to the model dtype, the same boundary
* ``reshape_and_cache_flash`` and ``scaled_fp8_quant`` use.
*/
#include <algorithm>
#include <cmath>
#include <cuda_runtime.h>
#include <type_traits>
#include "torch_utils.h"
#include "core/cuda_compat.h"
#include "type_convert.cuh"
#include "core/vec_dtypes/dtype_fp8.cuh"
#include "dispatch_utils.h"
#ifndef USE_ROCM
#include <cuda_fp8.h>
#include "quantization/w8a8/fp8/nvidia/quant_utils.cuh"
#else
#include <hip/hip_fp8.h>
#include "quantization/w8a8/fp8/amd/quant_utils.cuh"
#endif
#ifndef FINAL_MASK
#ifdef USE_ROCM
#define FINAL_MASK 0xffffffffffffffffULL
#else
#define FINAL_MASK 0xffffffffu
#endif
#endif
namespace vllm {
namespace muse_spark_fused_ops {
namespace {
inline int getSMVersion() {
auto* props = get_device_prop();
return props->major * 10 + props->minor;
}
inline int getSMCount() {
auto* props = get_device_prop();
return props->multiProcessorCount;
}
} // namespace
// 16 bytes per lane: the widest vector access, and the reason a head is owned
// by a group of lanes rather than a whole warp.
constexpr int kBytesPerLane = 16;
#ifdef USE_ROCM
using WarpMask = uint64_t;
constexpr unsigned kWarpSize = 64;
#else
using WarpMask = unsigned;
constexpr unsigned kWarpSize = 32;
#endif
// Slot -> (token, head) needs a division by a value only known at launch. The
// kernel is issue-bound, and a 32-bit hardware division is ~20 instructions of
// it, so the reciprocal is computed once on the host instead.
struct FastDiv {
uint32_t mul;
uint32_t shr;
int div;
};
inline FastDiv makeFastDiv(int denom) {
// The reciprocal form needs denom >= 2; denom == 1 is carried as a flag the
// device side selects on. Dividends stay below 2^31, which covers every slot
// index this kernel can form.
if (denom <= 1) return FastDiv{0u, 0u, 1};
int lg = 31 - __builtin_clz(denom);
if ((1 << lg) < denom) lg++;
uint32_t const p = 31 + static_cast<uint32_t>(lg);
uint64_t const m = ((1ull << p) + static_cast<uint64_t>(denom) - 1) /
static_cast<uint64_t>(denom);
return FastDiv{static_cast<uint32_t>(m), p - 32, denom};
}
__device__ __forceinline__ void fastDivmod(int& quo, int& rem, int src,
FastDiv const fd) {
int const q =
static_cast<int>(__umulhi(static_cast<uint32_t>(src), fd.mul) >> fd.shr);
// A select, not a branch: fd.div is uniform across the launch.
quo = (fd.div == 1) ? src : q;
rem = src - quo * fd.div;
}
// Sum across the LANES contiguous lanes that share one head. The group is
// LANES-aligned inside the warp, so an XOR butterfly over masks < LANES never
// leaves it. `group_mask` names exactly the group: whole groups are uniformly
// active or inactive, so a warp whose other groups bailed out still syncs.
template <int LANES>
__device__ __forceinline__ float groupReduceSum(float val,
WarpMask const group_mask) {
#pragma unroll
for (int mask = LANES >> 1; mask > 0; mask >>= 1) {
val += __shfl_xor_sync(group_mask, val, mask, 32);
}
return val;
}
// 16 contiguous bytes -> kElems fp32 registers, keeping the raw vector so a
// pass-through slot (V) can be re-stored without a conversion round trip.
template <typename scalar_t, int kElems>
__device__ __forceinline__ uint4 loadElems(scalar_t const* __restrict__ src,
float (&elems)[kElems]) {
using Converter = vllm::_typeConvert<scalar_t>;
using T2 = typename Converter::packed_hip_type;
uint4 raw = *reinterpret_cast<uint4 const*>(src);
auto const* p = reinterpret_cast<T2 const*>(&raw);
#pragma unroll
for (int i = 0; i < kElems / 2; i++) {
float2 f2 = Converter::convert(p[i]);
elems[2 * i] = f2.x;
elems[2 * i + 1] = f2.y;
}
return raw;
}
// The lane's slice of the rope table: kPairs cos and kPairs sin values, each
// run contiguous, so each is one vector load (16 bytes for the fp32 table, 8
// for a model-dtype one) instead of kPairs scalar ones. rotary_dim being a
// multiple of 16 is what keeps both runs naturally aligned.
template <typename rope_t, int kPairs>
__device__ __forceinline__ void loadRopeVec(rope_t const* __restrict__ cos_ptr,
rope_t const* __restrict__ sin_ptr,
float (&c)[kPairs],
float (&s)[kPairs]) {
static_assert(kPairs == 4, "rope vector load assumes 4 pairs per lane");
if constexpr (std::is_same_v<rope_t, float>) {
float4 const cv = VLLM_LDG(reinterpret_cast<float4 const*>(cos_ptr));
float4 const sv = VLLM_LDG(reinterpret_cast<float4 const*>(sin_ptr));
c[0] = cv.x, c[1] = cv.y, c[2] = cv.z, c[3] = cv.w;
s[0] = sv.x, s[1] = sv.y, s[2] = sv.z, s[3] = sv.w;
} else {
using Converter = vllm::_typeConvert<rope_t>;
using T2 = typename Converter::packed_hip_type;
uint2 const cv = VLLM_LDG(reinterpret_cast<uint2 const*>(cos_ptr));
uint2 const sv = VLLM_LDG(reinterpret_cast<uint2 const*>(sin_ptr));
auto const* cp = reinterpret_cast<T2 const*>(&cv);
auto const* sp = reinterpret_cast<T2 const*>(&sv);
#pragma unroll
for (int i = 0; i < kPairs / 2; i++) {
float2 const cf = Converter::convert(cp[i]);
float2 const sf = Converter::convert(sp[i]);
c[2 * i] = cf.x, c[2 * i + 1] = cf.y;
s[2 * i] = sf.x, s[2 * i + 1] = sf.y;
}
}
}
// kElems fp32 registers -> the 16-byte scalar_t vector they round to. Returned
// rather than stored so the fp8 cache path can reuse the *rounded* values,
// which is what reshape_and_cache_flash quantizes from.
template <typename scalar_t, int kElems>
__device__ __forceinline__ uint4 packElems(float const (&elems)[kElems]) {
using Converter = vllm::_typeConvert<scalar_t>;
using T2 = typename Converter::packed_hip_type;
uint4 raw;
auto* p = reinterpret_cast<T2*>(&raw);
#pragma unroll
for (int i = 0; i < kElems / 2; i++) {
p[i] = Converter::convert(make_float2(elems[2 * i], elems[2 * i + 1]));
}
return raw;
}
// Scatter one lane's 16 bytes into the KV cache. kAuto stores the model dtype
// verbatim; the fp8 dtypes quantize the value *after* it has been rounded to
// scalar_t, which is the boundary reshape_and_cache_flash uses.
//
// `inv_scale` is taken once per thread: per-element fp32 divisions would cost
// more than the rest of the kernel. The extra rounding is 2^-24 relative.
template <typename scalar_t, typename cache_t, Fp8KVCacheDataType kv_dt,
int kElems>
__device__ __forceinline__ void storeCache(cache_t* __restrict__ dst,
uint4 const& packed,
float const inv_scale) {
if constexpr (kv_dt == Fp8KVCacheDataType::kAuto) {
*reinterpret_cast<uint4*>(dst) = packed;
} else {
using Converter = vllm::_typeConvert<scalar_t>;
using HipT = typename Converter::hip_type;
auto const* rounded = reinterpret_cast<HipT const*>(&packed);
#ifndef USE_ROCM
constexpr __nv_fp8_interpretation_t kInterp =
(kv_dt == Fp8KVCacheDataType::kFp8E4M3) ? __NV_E4M3 : __NV_E5M2;
__nv_fp8x2_storage_t out[kElems / 2];
#pragma unroll
for (int i = 0; i < kElems / 2; i++) {
float2 v =
make_float2(Converter::convert(rounded[2 * i]) * inv_scale,
Converter::convert(rounded[2 * i + 1]) * inv_scale);
out[i] = __nv_cvt_float2_to_fp8x2(v, __NV_SATFINITE, kInterp);
}
// kElems fp8 bytes: 8 for the 2-byte dtypes this kernel serves.
static_assert(kElems == 8, "fp8 store assumes 8 elements per lane");
*reinterpret_cast<uint2*>(dst) = *reinterpret_cast<uint2 const*>(out);
#else
#pragma unroll
for (int i = 0; i < kElems; i++) {
float const v = Converter::convert(rounded[i]) * inv_scale;
dst[i] = fp8::scaled_convert<cache_t, float, kv_dt>(v, 1.0f);
}
#endif
}
}
// Grid is 1D over (token, head-chunk) pairs. A chunk is HEADS_PER_GROUP
// consecutive head slots of one token, served by kLanes contiguous lanes; the
// whole chunk shares the token's position and its slice of the rope table, so
// the slot division, the position load and the cos/sin load are paid once per
// chunk. `slots_per_token % HEADS_PER_GROUP == 0` keeps a chunk inside one
// token. `rope_t` is the cos/sin cache dtype, independent of the qkv dtype.
template <typename scalar_t, typename cache_t, Fp8KVCacheDataType kv_dt,
typename rope_t, int HEAD_DIM, int HEADS_PER_GROUP, bool kDoRope,
bool kInsertKV>
__global__
__launch_bounds__(256, 6) void fusedMuseSparkQKNormRopeKVInsertKernel(
scalar_t* __restrict__ qkv, // [N, qkv_stride] in/out
rope_t const* __restrict__ cos_sin_cache, // [max_pos, rotary_dim]
int64_t const* __restrict__ positions, // [N] i64
int64_t const* __restrict__ slot_mapping, // [N] i64 or nullptr
cache_t* __restrict__ kv_cache, // [nb, nkv, bs, 2*HEAD_DIM]
float const* __restrict__ k_scale, // [1] or nullptr
float const* __restrict__ v_scale, // [1] or nullptr
uint8_t* __restrict__ q_fp8_out, // [N, nq * HEAD_DIM] or nullptr
float const* __restrict__ q_scale, // [1] or nullptr
float const eps, int const rotary_dim, int const num_tokens, int const nq,
int const nkv, FastDiv const chunks_div, int64_t const qkv_stride,
FastDiv const block_div, int64_t const kv_s_block, int64_t const kv_s_head,
int64_t const kv_s_token) {
#if (!defined(__CUDA_ARCH__) || __CUDA_ARCH__ < 800) && !defined(USE_ROCM)
// _typeConvert<BFloat16> is unavailable on pre-Ampere; this kernel only ever
// runs with bf16/fp16 inputs. Discard the bf16 body there.
if constexpr (std::is_same_v<scalar_t, c10::BFloat16>) {
return;
} else {
#endif
constexpr int kElems = kBytesPerLane / sizeof(scalar_t);
constexpr int kLanes = HEAD_DIM / kElems;
constexpr int kPairs = kElems / 2;
constexpr int kH = HEADS_PER_GROUP;
static_assert(kLanes > 0 && kLanes <= 32 && (32 % kLanes) == 0,
"head_dim must give a power-of-two lane group <= 32");
int const tid = blockIdx.x * blockDim.x + threadIdx.x;
int const groupIdx = tid / kLanes;
int const dimBase = (tid % kLanes) * kElems;
int tokenIdx, chunk;
fastDivmod(tokenIdx, chunk, groupIdx, chunks_div);
int const firstSlot = chunk * kH;
// Out-of-range groups must not exit before the reductions below: the XOR
// butterfly needs every lane the mask names, and the mask covers the whole
// group. They ride along and skip every memory access instead.
bool const active = tokenIdx < num_tokens;
constexpr WarpMask kGroupBits =
(kLanes == 32) ? static_cast<WarpMask>(0xffffffffu)
: ((WarpMask{1} << kLanes) - WarpMask{1});
unsigned const group_shift =
(threadIdx.x & (kWarpSize - 1)) & ~static_cast<unsigned>(kLanes - 1);
WarpMask const group_mask = kGroupBits << group_shift;
// PDL: wait for the qkv-projection GEMM that produces `qkv`. No-op when
// PDL is not enabled on the launch.
#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900)
cudaGridDependencySynchronize();
#endif
// Q, K and V are contiguous head slots at the head of the row, so the slot
// index doubles as the head offset. The trailing o-gate is skipped by the
// row stride alone.
scalar_t* const base = qkv + static_cast<int64_t>(tokenIdx) * qkv_stride +
static_cast<int64_t>(firstSlot) * HEAD_DIM + dimBase;
// Issue every load in the chunk before touching any of it, so the group has
// kH loads in flight instead of one. An inactive group leaves these unset;
// nothing downstream reads them, since every store is gated on `active`.
uint4 raw[kH];
if (active) {
#pragma unroll
for (int h = 0; h < kH; h++) {
raw[h] = *reinterpret_cast<uint4 const*>(base + h * HEAD_DIM);
}
}
float cosv[kPairs], sinv[kPairs];
bool const laneRotates = kDoRope && dimBase < rotary_dim;
if constexpr (kDoRope) {
if (active && laneRotates) {
int64_t const pos = positions[tokenIdx];
rope_t const* cache_ptr = cos_sin_cache + pos * rotary_dim;
loadRopeVec<rope_t, kPairs>(cache_ptr + dimBase / 2,
cache_ptr + rotary_dim / 2 + dimBase / 2,
cosv, sinv);
}
}
int64_t sm = -1;
if constexpr (kInsertKV) {
if (active) sm = slot_mapping[tokenIdx];
}
int cache_block = 0, cache_tok = 0;
float k_inv = 1.0f, v_inv = 1.0f;
float const q_inv = q_fp8_out ? 1.0f / *q_scale : 1.0f;
if constexpr (kInsertKV) {
if (sm >= 0) {
fastDivmod(cache_block, cache_tok, static_cast<int>(sm), block_div);
}
if constexpr (kv_dt != Fp8KVCacheDataType::kAuto) {
k_inv = 1.0f / *k_scale;
v_inv = 1.0f / *v_scale;
}
}
#pragma unroll
for (int h = 0; h < kH; h++) {
int const slot = firstSlot + h;
bool const isV = kInsertKV && slot >= nq + nkv;
bool const isK = slot >= nq && slot < nq + nkv;
bool const isQK = active && !isV;
float elems[kElems];
uint4 packed = raw[h];
if (isQK) {
using Converter = vllm::_typeConvert<scalar_t>;
using T2 = typename Converter::packed_hip_type;
auto const* p = reinterpret_cast<T2 const*>(&raw[h]);
#pragma unroll
for (int i = 0; i < kElems / 2; i++) {
float2 const f2 = Converter::convert(p[i]);
elems[2 * i] = f2.x;
elems[2 * i + 1] = f2.y;
}
float sumsq = 0.0f;
#pragma unroll
for (int i = 0; i < kElems; i++) sumsq += elems[i] * elems[i];
sumsq = groupReduceSum<kLanes>(sumsq, group_mask);
float const rms_rcp =
rsqrtf(sumsq / static_cast<float>(HEAD_DIM) + eps);
#pragma unroll
for (int i = 0; i < kElems; i++) elems[i] *= rms_rcp;
if constexpr (kDoRope) {
if (laneRotates) {
#pragma unroll
for (int i = 0; i < kPairs; i++) {
float const v0 = elems[2 * i];
float const v1 = elems[2 * i + 1];
elems[2 * i] = v0 * cosv[i] - v1 * sinv[i];
elems[2 * i + 1] = v0 * sinv[i] + v1 * cosv[i];
}
}
}
// Rounding to scalar_t happens exactly once, here, and both the
// Q output and the fp8 cache quantization consume the same
// rounded value. V passes through with the vector it was loaded with.
packed = packElems<scalar_t, kElems>(elems);
if (q_fp8_out && slot < nq) {
int64_t const q_off =
(static_cast<int64_t>(tokenIdx) * nq + slot) * HEAD_DIM + dimBase;
storeCache<scalar_t, uint8_t, Fp8KVCacheDataType::kFp8E4M3, kElems>(
q_fp8_out + q_off, packed, q_inv);
} else {
*reinterpret_cast<uint4*>(base + h * HEAD_DIM) = packed;
}
}
if constexpr (kInsertKV) {
if (sm >= 0 && (isK || isV)) { // sm < 0 marks a padded token
int const head = isK ? (slot - nq) : (slot - nq - nkv);
int64_t const off = static_cast<int64_t>(cache_block) * kv_s_block +
head * kv_s_head +
static_cast<int64_t>(cache_tok) * kv_s_token +
(isK ? 0 : HEAD_DIM) + dimBase;
storeCache<scalar_t, cache_t, kv_dt, kElems>(kv_cache + off, packed,
isK ? k_inv : v_inv);
}
}
}
// PDL: let a dependent successor launch early.
#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900)
cudaTriggerProgrammaticLaunchCompletion();
#endif
#if (!defined(__CUDA_ARCH__) || __CUDA_ARCH__ < 800) && !defined(USE_ROCM)
}
#endif
}
template <typename scalar_t, typename cache_t, Fp8KVCacheDataType kv_dt,
typename rope_t, int HEAD_DIM, int HEADS_PER_GROUP>
void launchFusedMuseSparkTyped(
scalar_t* qkv, rope_t const* cos_sin_cache, int64_t const* positions,
int64_t const* slot_mapping, cache_t* kv_cache, float const* k_scale,
float const* v_scale, uint8_t* q_fp8_out, float const* q_scale,
float const eps, int const rotary_dim, int const num_tokens, int const nq,
int const nkv, int64_t const qkv_stride, int const block_size,
int64_t const kv_s_block, int64_t const kv_s_head, int64_t const kv_s_token,
bool const do_rope, bool const insert_kv, cudaStream_t stream) {
FastDiv const block_div = makeFastDiv(block_size);
constexpr int kElems = kBytesPerLane / sizeof(scalar_t);
constexpr int kLanes = HEAD_DIM / kElems;
int const slots_per_token = nq + nkv + (insert_kv ? nkv : 0);
int const chunks_per_token = slots_per_token / HEADS_PER_GROUP;
FastDiv const chunks_div = makeFastDiv(chunks_per_token);
int64_t const total_lanes =
static_cast<int64_t>(num_tokens) * chunks_per_token * kLanes;
if (total_lanes == 0) return;
constexpr int kBlockSize = 256;
int const grid =
static_cast<int>((total_lanes + kBlockSize - 1) / kBlockSize);
#ifndef USE_ROCM
// PDL wherever the hardware supports it (SM90+); a plain launch elsewhere.
static int const sm_version = getSMVersion();
cudaLaunchConfig_t config;
config.gridDim = dim3(grid);
config.blockDim = dim3(kBlockSize);
config.dynamicSmemBytes = 0;
config.stream = stream;
cudaLaunchAttribute attrs[1];
attrs[0].id = cudaLaunchAttributeProgrammaticStreamSerialization;
attrs[0].val.programmaticStreamSerializationAllowed = 1;
config.attrs = attrs;
config.numAttrs = (sm_version >= 90) ? 1 : 0;
#define LAUNCH_MUSE(DO_ROPE, INSERT) \
cudaLaunchKernelEx(&config, \
fusedMuseSparkQKNormRopeKVInsertKernel< \
scalar_t, cache_t, kv_dt, rope_t, HEAD_DIM, \
HEADS_PER_GROUP, DO_ROPE, INSERT>, \
qkv, cos_sin_cache, positions, slot_mapping, kv_cache, \
k_scale, v_scale, q_fp8_out, q_scale, eps, rotary_dim, \
num_tokens, nq, nkv, chunks_div, qkv_stride, block_div, \
kv_s_block, kv_s_head, kv_s_token)
#else
// clang-format off
#define LAUNCH_MUSE(DO_ROPE, INSERT) \
fusedMuseSparkQKNormRopeKVInsertKernel< \
scalar_t, cache_t, kv_dt, rope_t, HEAD_DIM, HEADS_PER_GROUP, DO_ROPE, \
INSERT><<<grid, kBlockSize, 0, stream>>>( \
qkv, cos_sin_cache, positions, slot_mapping, kv_cache, k_scale, \
v_scale, q_fp8_out, q_scale, eps, rotary_dim, num_tokens, nq, nkv, \
chunks_div, qkv_stride, block_div, kv_s_block, kv_s_head, kv_s_token)
// clang-format on
#endif
if (do_rope) {
if (insert_kv) {
LAUNCH_MUSE(true, true);
} else {
LAUNCH_MUSE(true, false);
}
} else {
if (insert_kv) {
LAUNCH_MUSE(false, true);
} else {
LAUNCH_MUSE(false, false);
}
}
#undef LAUNCH_MUSE
}
template <typename scalar_t, typename cache_t, Fp8KVCacheDataType kv_dt>
void launchFusedMuseSpark(
scalar_t* qkv, void const* cos_sin_cache, bool const rope_is_fp32,
int64_t const* positions, int64_t const* slot_mapping, cache_t* kv_cache,
float const* k_scale, float const* v_scale, uint8_t* q_fp8_out,
float const* q_scale, float const eps, int const rotary_dim,
int const num_tokens, int const nq, int const nkv, int const head_dim,
int64_t const qkv_stride, int const block_size, int64_t const kv_s_block,
int64_t const kv_s_head, int64_t const kv_s_token, bool const do_rope,
bool const insert_kv, cudaStream_t stream) {
// The group width must divide the slot count. Widen only while the grid
// still covers a full wave; decode-sized batches keep the narrow schedule.
int const slots_per_token = nq + nkv + (insert_kv ? nkv : 0);
constexpr int kElemsSel = kBytesPerLane / sizeof(scalar_t);
constexpr int kBlockSel = 256;
int const lanes_per_head = head_dim / kElemsSel;
int64_t const full_wave =
static_cast<int64_t>(getSMCount()) * (2048 / kBlockSel);
int heads_per_group = 1;
for (int h : {4, 2}) {
if (slots_per_token % h != 0) continue;
int64_t const blocks = (static_cast<int64_t>(num_tokens) *
(slots_per_token / h) * lanes_per_head +
kBlockSel - 1) /
kBlockSel;
if (blocks >= full_wave) {
heads_per_group = h;
break;
}
}
// TP4/TP8 amortize head addressing before filling a full CTA wave. RoPE
// benefits earlier; FP8 conversion keeps wider groups expensive for longer.
if (insert_kv && head_dim == 64 &&
((nq == 16 && nkv == 2) || (nq == 32 && nkv == 4))) {
int64_t const query_heads = static_cast<int64_t>(num_tokens) * nq;
if (do_rope && query_heads >= 24576) {
heads_per_group = std::max(heads_per_group, 2);
if constexpr (kv_dt == Fp8KVCacheDataType::kAuto) {
if (query_heads >= 49152) heads_per_group = 4;
}
} else if (!do_rope && query_heads >= 32768) {
if constexpr (kv_dt == Fp8KVCacheDataType::kAuto) {
heads_per_group = std::max(heads_per_group, 2);
}
}
}
#define DISPATCH_ARGS(RT, HD, HPG) \
launchFusedMuseSparkTyped<scalar_t, cache_t, kv_dt, RT, HD, HPG>( \
qkv, reinterpret_cast<RT const*>(cos_sin_cache), positions, \
slot_mapping, kv_cache, k_scale, v_scale, q_fp8_out, q_scale, eps, \
rotary_dim, num_tokens, nq, nkv, qkv_stride, block_size, kv_s_block, \
kv_s_head, kv_s_token, do_rope, insert_kv, stream)
#define DISPATCH_HPG(RT, HD) \
switch (heads_per_group) { \
case 4: \
DISPATCH_ARGS(RT, HD, 4); \
break; \
case 2: \
DISPATCH_ARGS(RT, HD, 2); \
break; \
default: \
DISPATCH_ARGS(RT, HD, 1); \
break; \
}
// A NoPE launch never reads the table, so it can share the fp32 branch.
#define DISPATCH_ROPE_T(HD) \
if (!do_rope || rope_is_fp32) { \
DISPATCH_HPG(float, HD); \
} else { \
DISPATCH_HPG(scalar_t, HD); \
}
switch (head_dim) {
case 64:
DISPATCH_ROPE_T(64);
break;
case 128:
DISPATCH_ROPE_T(128);
break;
case 256:
DISPATCH_ROPE_T(256);
break;
default:
STD_TORCH_CHECK(false, "unsupported head_dim: ", head_dim);
}
#undef DISPATCH_ROPE_T
#undef DISPATCH_HPG
#undef DISPATCH_ARGS
}
} // namespace muse_spark_fused_ops
} // namespace vllm
// clang-format off
#define CALL_FUSED_MUSE_SPARK(_RAW_T, CACHE_T, KV_DTYPE) \
vllm::muse_spark_fused_ops::launchFusedMuseSpark<st, CACHE_T, KV_DTYPE>( \
reinterpret_cast<st*>(qkv.data_ptr()), \
do_rope ? cos_sin_cache->data_ptr() : nullptr, rope_is_fp32, \
reinterpret_cast<int64_t const*>(positions.data_ptr()), \
insert_kv ? reinterpret_cast<int64_t const*>(slot_mapping->data_ptr()) \
: nullptr, \
insert_kv ? reinterpret_cast<CACHE_T*>(kv_cache->data_ptr()) : nullptr, \
needs_scales ? reinterpret_cast<float const*>(k_scale->data_ptr()) \
: nullptr, \
needs_scales ? reinterpret_cast<float const*>(v_scale->data_ptr()) \
: nullptr, \
q_fp8_out ? reinterpret_cast<uint8_t*>(q_fp8_out->data_ptr()) : nullptr, \
q_scale ? reinterpret_cast<float const*>(q_scale->data_ptr()) : nullptr, \
static_cast<float>(eps), static_cast<int>(rotary_dim), num_tokens, nq, \
nkv, hd, qkv_stride, static_cast<int>(block_size), kv_s_block, \
kv_s_head, kv_s_token, do_rope, insert_kv, stream)
// clang-format on
void fused_muse_spark_qknorm_rope_kv_insert(
torch::stable::Tensor& qkv, // [N, (nq + 2*nkv + gate)*hd]
torch::stable::Tensor const& positions, // [N] i64
int64_t num_heads, int64_t num_kv_heads, int64_t head_dim, double eps,
std::optional<torch::stable::Tensor> cos_sin_cache, // None => NoPE layer
int64_t rotary_dim,
std::optional<torch::stable::Tensor> slot_mapping, // [N] i64
std::optional<torch::stable::Tensor> kv_cache, // [nb, nkv, bs, 2*hd]
int64_t block_size, const std::string& kv_cache_dtype,
std::optional<torch::stable::Tensor> k_scale, // [1] fp32, fp8 cache only
std::optional<torch::stable::Tensor> v_scale,
std::optional<torch::stable::Tensor> q_fp8_out,
std::optional<torch::stable::Tensor> q_scale) {
STD_TORCH_CHECK(qkv.is_cuda() && qkv.dim() == 2 && qkv.stride(1) == 1,
"qkv must be a 2-D CUDA tensor with a contiguous row");
STD_TORCH_CHECK(
qkv.scalar_type() == torch::headeronly::ScalarType::Half ||
qkv.scalar_type() == torch::headeronly::ScalarType::BFloat16,
"qkv must be float16 or bfloat16");
STD_TORCH_CHECK(
positions.is_cuda() &&
positions.scalar_type() == torch::headeronly::ScalarType::Long,
"positions must be int64 CUDA");
int const num_tokens = static_cast<int>(qkv.size(0));
int const nq = static_cast<int>(num_heads);
int const nkv = static_cast<int>(num_kv_heads);
int const hd = static_cast<int>(head_dim);
int64_t const qkv_stride = qkv.stride(0);
STD_TORCH_CHECK(q_fp8_out.has_value() == q_scale.has_value(),
"q_fp8_out and q_scale must be provided together");
if (q_fp8_out) {
STD_TORCH_CHECK(
q_fp8_out->is_cuda() &&
q_fp8_out->get_device_index() == qkv.get_device_index() &&
q_fp8_out->scalar_type() ==
torch::headeronly::ScalarType::Float8_e4m3fn &&
q_fp8_out->dim() == 2 && q_fp8_out->is_contiguous() &&
q_fp8_out->size(0) == num_tokens &&
q_fp8_out->size(1) == static_cast<int64_t>(nq) * hd,
"q_fp8_out must be contiguous E4M3 [num_tokens, num_heads * head_dim] "
"on the qkv device");
STD_TORCH_CHECK(
q_scale->is_cuda() &&
q_scale->get_device_index() == qkv.get_device_index() &&
q_scale->scalar_type() == torch::headeronly::ScalarType::Float &&
q_scale->numel() == 1,
"q_scale must be a single fp32 value on the qkv device");
}
STD_TORCH_CHECK(hd == 64 || hd == 128 || hd == 256,
"head_dim must be 64, 128 or 256");
STD_TORCH_CHECK(qkv.size(1) >= static_cast<int64_t>(nq + 2 * nkv) * hd,
"qkv row must hold at least [q|k|v]");
// A lane moves 16 bytes, so every row start and every head start must land
// on a 16-byte boundary. Every supported head_dim is a multiple of 8 at 2
// bytes, so heads are fine; the row stride is the one that can break it.
STD_TORCH_CHECK(qkv_stride % 8 == 0,
"qkv row stride must be a multiple of 8 elements, got ",
qkv_stride);
bool const do_rope = cos_sin_cache.has_value();
bool rope_is_fp32 = true;
if (do_rope) {
STD_TORCH_CHECK(cos_sin_cache->is_cuda() && cos_sin_cache->dim() == 2 &&
cos_sin_cache->is_contiguous(),
"cos_sin_cache must be a contiguous 2-D CUDA tensor");
STD_TORCH_CHECK(cos_sin_cache->size(1) == rotary_dim,
"cos_sin_cache shape must be [max_pos, rotary_dim]");
STD_TORCH_CHECK(
cos_sin_cache->scalar_type() == torch::headeronly::ScalarType::Float ||
cos_sin_cache->scalar_type() == qkv.scalar_type(),
"cos_sin_cache must be float32 or match the qkv dtype");
rope_is_fp32 =
cos_sin_cache->scalar_type() == torch::headeronly::ScalarType::Float;
// A lane owns 16 bytes = 8 elements of the head and must be wholly inside
// or wholly outside the rotated prefix.
STD_TORCH_CHECK(rotary_dim > 0 && rotary_dim <= hd && rotary_dim % 16 == 0,
"rotary_dim must be a positive multiple of 16 and <= "
"head_dim");
}
vllm::Fp8KVCacheDataType const kv_dt =
vllm::get_fp8_kv_cache_data_type(kv_cache_dtype);
bool const insert_kv = kv_cache.has_value() && kv_cache->numel() > 0;
bool const needs_scales =
insert_kv && kv_dt != vllm::Fp8KVCacheDataType::kAuto;
int64_t kv_s_block = 0, kv_s_head = 0, kv_s_token = 0;
if (insert_kv) {
STD_TORCH_CHECK(
slot_mapping.has_value() && slot_mapping->is_cuda() &&
slot_mapping->scalar_type() == torch::headeronly::ScalarType::Long,
"insert mode requires an int64 CUDA slot_mapping");
STD_TORCH_CHECK(slot_mapping->numel() >= num_tokens,
"slot_mapping must cover every token");
STD_TORCH_CHECK(kv_cache->dim() == 4 && kv_cache->stride(3) == 1,
"kv_cache must be [nb, nkv, bs, 2*head_dim] with a "
"contiguous content dim");
STD_TORCH_CHECK(kv_cache->size(3) == 2 * hd,
"kv_cache content dim must be 2*head_dim");
STD_TORCH_CHECK(kv_cache->size(1) == nkv,
"kv_cache head dim must equal num_kv_heads");
STD_TORCH_CHECK(kv_cache->size(2) == block_size,
"kv_cache token dim must equal block_size");
if (kv_dt == vllm::Fp8KVCacheDataType::kAuto) {
STD_TORCH_CHECK(kv_cache->scalar_type() == qkv.scalar_type(),
"auto kv_cache dtype must match qkv");
} else {
STD_TORCH_CHECK(
kv_cache->scalar_type() == torch::headeronly::ScalarType::Byte ||
kv_cache->scalar_type() ==
torch::headeronly::ScalarType::Float8_e4m3fn,
"fp8 kv_cache must use uint8 or float8_e4m3fn storage");
STD_TORCH_CHECK(k_scale.has_value() && v_scale.has_value(),
"fp8 kv cache requires k_scale and v_scale");
STD_TORCH_CHECK(
k_scale->numel() == 1 && v_scale->numel() == 1 &&
k_scale->scalar_type() == torch::headeronly::ScalarType::Float &&
v_scale->scalar_type() == torch::headeronly::ScalarType::Float,
"k_scale and v_scale must be single fp32 values");
}
kv_s_block = kv_cache->stride(0);
kv_s_head = kv_cache->stride(1);
kv_s_token = kv_cache->stride(2);
}
const torch::stable::accelerator::DeviceGuard device_guard(
qkv.get_device_index());
auto stream = get_current_cuda_stream(qkv.get_device_index());
VLLM_STABLE_DISPATCH_HALF_TYPES(
qkv.scalar_type(), "fused_muse_spark_qknorm_rope_kv_insert", [&] {
using st = scalar_t;
DISPATCH_BY_KV_CACHE_DTYPE(qkv.scalar_type(), kv_cache_dtype,
CALL_FUSED_MUSE_SPARK);
});
}
#undef CALL_FUSED_MUSE_SPARK

Some files were not shown because too many files have changed in this diff Show More