Compare commits

..
33 Commits
Author SHA1 Message Date
MOVZX 4a3ebc6b5b Merge branch 'main' into patches 2026-10-05 08:21:46 +00:00
MOVZX f83ecfac16 Merge remote-tracking branch 'upstream/main' into patches 2026-10-05 08:52:31 +07:00
MOVZX 85c31d47ad Merge remote-tracking branch 'upstream/main' into patches
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-03 22:06:10 +07:00
MOVZX 2877121b2d Merge remote-tracking branch 'upstream/main' into patches
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-03 21:59:03 +07:00
MOVZXandHinata ae0350c0de [Spec Decode][MRV2] RecoverSSM for Qwen GDN and PLE short conv (PR #58863)
Intake of open upstream PR vllm-project/vllm#58863. Active only with
VLLM_USE_V2_MODEL_RUNNER=1; inert otherwise.

Speculative decoding on Gated DeltaNet snapshotted the full recurrent
state after every draft position and reserved one Mamba block per draft
position. On a 24 GB card that is reserved VRAM we never use, plus
per-step copy traffic.

RecoverSSM, the protocol Kimi-K3 KDA already uses, replaces this:
verify runs from a read-only checkpoint and stores a small per-token
replay record [correction (V), normalized key (K), gate (1)] in fp32;
commit replays only the accepted tokens and writes the recurrent state
once, plus the block-boundary state in align mode. The PLE dilated
short conv compacts its accepted window with the same plan.

Conflict in vllm/models/qwen4_exp/nvidia/mtp.py: our get_top_tokens()
work put LocalArgmaxMixin on Qwen4ExpMTP, the PR adds SupportsReplaySSM
to the same base list. Kept both.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-09-29 13:39:09 +07:00
MOVZXandHinata a1d3c2cf1e [Spec Decode] Fused draft metadata updates for the QSA side cache (PR #58449)
Intake of open upstream PR vllm-project/vllm#58449.

With MTP num_speculative_tokens > 1, the V2 autoregressive speculator
reuses one attention-metadata build across draft steps only when every
group's builder sets supports_draft_decode_metadata_update. The QSA
side-cache builder (QWEN4_EXP_EXP_QSA_STATE) did not, so every draft
step rebuilt attention metadata for the whole model.

QSAMetadataBuilder now implements update_draft_decode_metadata: it
re-launches the kernel with in-place seq_lens updates, so one metadata
build serves every draft step.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-09-29 13:36:09 +07:00
MOVZXandHinata cb275fdf20 [Qwen4Exp] Fix MTP buffer placement, config parsing, Triton warmup (PR #56742)
Intake of open upstream PR vllm-project/vllm#56742. Every rig here runs
MTP on both Qwen3.8 models, so these three bugs sit on our hot path.

The MTP hidden buffer was allocated with torch.empty and no device, so
it followed the ambient default instead of device_config.device; on a
CPU-default process the draft buffer lands on the wrong device. Both
NVIDIA and AMD backends now call make_mtp_hidden_buffer() in
vllm/models/qwen4_exp/common/mtp.py, which pins it to the configured
device.

Config parsing now accepts configs carrying M-RoPE keys and the
Transformers spelling of layer_types; both previously failed at load.

Triton warmup: Qwen4Exp was missing from the Qwen warmup dispatch, so
its Triton kernels compiled during the first served requests instead of
at startup.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-09-29 13:36:05 +07:00
MOVZX a63dca4afb Merge remote-tracking branch 'upstream/main' into patches
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-09-29 12:54:18 +07:00
MOVZX 76af434852 Merge remote-tracking branch 'upstream/main' into patches 2026-09-28 12:06:52 +07:00
lesj0610 521235d6df Merge remote-tracking branch 'origin/main' into sync/lesj/qwen4exp-bugfixes
Signed-off-by: lesj0610 <lesj0610@users.noreply.github.com>
2026-09-27 18:25:21 +09:00
5567cc1b25 [Mamba] RecoverSSM align: keep a boundary-ending state in its block
A speculative window whose accepted tokens end exactly on a recurrent block
boundary selected the following block-table column for the committed state
(commit plan) and recorded it as the running column (postprocess). That
column can still be unallocated, so the next step reads an uninitialized
page. Select the block that holds the last committed token, matching the
(n - 1) // block convention used elsewhere in align mode. Applies to
Kimi-K3 KDA and to the Qwen GDN/PLE RecoverSSM paths that share the plan.

Fix and test from local-inference-lab/vllm a02f585d (Martin Vit, root cause
reported by astra); the GDN/PLE align tests now expect the same convention.

Signed-off-by: Jürgen Schmied <juergenschmied70@gmail.com>
Co-authored-by: Martin Vit <martin@voipmonitor.org>
Co-authored-by: Claude <noreply@anthropic.com>
2026-09-27 11:12:40 +02:00
Jürgen Schmied ed2a9665ea Merge branch 'main' into gdn-recoverssm 2026-09-27 08:01:32 +02:00
MOVZX 5da84df6b7 [BugFix] Honor drop_eagle_block in MambaManager
On a hybrid model (full-attention + Mamba/GDN layers, e.g. Qwen3.6-35B-A3B),
enabling MTP/EAGLE speculative decoding together with prefix caching corrupts
the recurrent state, and the server serves wrong answers. #43559 reports
tool-eval accuracy dropping from ~90% to ~50%. Nothing raises an exception
and nothing shows up in the logs, so an operator has no signal that anything
is wrong, and prefix caching then spreads it, because the corrupted state is
hash-reachable and every later request sharing that prefix restores from it.

The root cause is the find_longest_cache_hit method on MambaManager,
at vllm/v1/core/single_type_kv_cache_manager.py:1049 on main, which takes
drop_eagle_block in its signature and never acts on it. Its body searches
right-to-left for the last cached block, builds [null]*i + [cached], and
breaks, referencing neither drop_eagle_block nor .pop(.

Signed-off-by: MOVZX <movzx@yahoo.com>
2026-09-27 10:45:10 +07:00
MOVZX 5e6656e7a0 [Bugfix][Spec Decode] Add get_top_tokens() to Qwen4ExpMTP draft model
use_local_argmax_reduction hard-fails at worker startup when the draft
model lacks get_top_tokens(): the V2 speculator probes for the method
in _validate_local_argmax_reduction and raises, with no fallback to
the full-vocab logits path. Qwen4ExpMTP (nvidia and amd variants) has
compute_logits but no get_top_tokens, so any Qwen4Exp MTP drafter with
the flag enabled crashed both TP ranks during load_model.

Qwen4ExpMTP owns a flat self.lm_head and its compute_logits is exactly
self.logits_processor(self.lm_head, hidden_states), the same shape as
Qwen3_5MTP, so LocalArgmaxMixin applies unchanged: vocab-parallel
argmax with a (value, index) gather, O(2*tp_size) vs O(vocab_size)
communication per draft token. No D2T remapping needed since the
draft vocabulary equals the target vocabulary.

Signed-off-by: MOVZX <movzx@yahoo.com>
2026-09-27 10:31:47 +07:00
MOVZXandHinata 43fbd6510b [Qwen4Exp] Serve PLE rows from the loader's mapped checkpoint shards (PR #58815)
Adds an opt-in backend, --engram-config '{"host_file_gather": true}', that
serves Qwen4Exp PLE rows from the checkpoint files through the page cache
instead of allocating the 47.7 GiB table on GPU or in pinned host memory.

- Qwen4ExpPLEFileGatherEmbedding allocates no table rows; load_weights keeps
  the default loader's zero-copy safetensors views of each shard. At bind
  (post-load hook) every view must be file-backed per /proc/self/maps (HF
  cache symlinks work; heap/pinned/anonymous memory is rejected) and the
  shards must cover the whole table.
- Each step, Qwen4ExpModelState.prepare_inputs computes the PLE n-gram ids,
  copies them to pinned host memory, syncs the stream once, issues
  POSIX_FADV_WILLNEED over the coalesced page ranges of the step's rows,
  gathers the rows on the host, and copies them into a fixed device staging
  buffer. The forward reads only the staging buffer, so FULL CUDA graph
  capture is unaffected.
- Rows stay in the checkpoint dtype (FP8 table stays FP8, dequantized on the
  GPU); dummy loads serve zeros; weight reload is rejected; takes precedence
  over cpu_offload; rejected on ROCm and with elastic EP.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-09-27 09:54:42 +07:00
MOVZXandHinata 5af48b498c [Qwen3.5] Build MTP head unquantized when checkpoint ships it dense (PR #58813)
Fixes #58807. A compressed-tensors pack-quantized checkpoint that ships
its MTP head dense (plain mtp.*.weight names in a separate bf16 shard)
crashes at weight load: Qwen3_5MultiTokenPredictor builds its Linears
from the target quant config, so the loader expects packed names
(weight_packed) the checkpoint never had, and the weight-map walk dies
with "no module or parameter named fc.weight".

Detect the dense MTP head from the checkpoint itself: read the
safetensors index via the same repo-file channel used for recipe.yaml
(local cache first, Hub download as fallback). If any mtp.* tensor
carries a quantization artifact suffix the head is properly quantized
and nothing changes; if every mtp.* entry is a plain .weight the whole
predictor builds unquantized (fc and layers), mirroring the existing
modelopt_fp4 and GPTQ dynamic carve-outs. Single-file checkpoints have
no index and keep current behavior; NVFP4 checkpoints are unaffected
(their mtp.layers.* carry artifacts).

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-09-27 09:54:24 +07:00
MOVZXandHinata 835c44af15 [Qwen4Exp] Reinterpret FP8 E4M3 PLE as int8 for SM86 pinned-host lookup (PR #58856)
The pinned-host PLE lookup Triton kernel only gathers and stores bytes,
but passing an fp8e4nv tensor to it fails on architectures without
native FP8 pointer support (e.g. SM86) with an error saying type
fp8e4nv is not supported (supported fp8 dtypes are fp8e4b15, fp8e5).

Since FP8 E4M3 and int8 both occupy one byte per element, reinterpret
the PLE weight view and the output tensor as int8 when the weight dtype
is float8_e4m3fn; the non-FP8 path is unchanged.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-09-27 09:54:05 +07:00
MOVZXandHinata 758cc4f9e1 [Qwen3.5] Pass quant_config and prefix to embed_tokens in Qwen3_5Model
Apply the qwen3_5_quantized_embedding patch: construct the embedding
through the model's quant config so quantized (e.g. fp8/int8)
embeddings are supported, and give it a stable prefix for weight
loading.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-09-27 09:53:21 +07:00
Jürgen SchmiedandClaude 03cae7837f [Spec Decode][Qwen4Exp] RecoverSSM for Qwen GDN and the PLE short conv
With --use-replayssm and speculative decoding, Qwen4Exp's Gated DeltaNet layers verify the
draft window from a read-only checkpoint and keep a per-token replay record
[correction(V), normalized key(K), gate(1)] in fp32. After sampling, one commit replays the
accepted tokens and writes the recurrent state once (plus the block-boundary state in align
mode), instead of snapshotting the state after every draft position. The PLE dilated short
conv follows the same protocol (accepted-suffix compaction). num_speculative_blocks = 0,
so no per-draft Mamba blocks are reserved.

- GDN verify/commit kernels: model_executor/layers/mamba/gdn/recoverssm_gdn.py
- metadata/backends: v1/attention/backends/gdn_recoverssm.py, ple_recoverssm.py
- the verify reads strided q/k/v views of the conv output and writes in place
- builders declare AttentionCGSupport.NEVER: FULL_AND_PIECEWISE falls back to PIECEWISE;
  the fallback now also counts breakable CUDA graphs (empty splitting_ops, with or
  without torch.compile) as piecewise
  attention instead of dropping to NONE
- the generic Mamba metadata builder skips the Mamba2 ReplaySSM ring-buffer workspaces
  when --use-replayssm selects RecoverSSM (the PLE short conv uses that builder)
- tests: GDN verify/commit vs the native kernels (fp32 and bf16 state, align boundaries,
  mixed and ragged batches, extreme gates), PLE vs token-by-token decode, config selection

Signed-off-by: Jürgen Schmied <juergenschmied70@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-09-27 01:17:24 +02:00
Jürgen SchmiedandClaude a80e41d690 [Mamba] Share RecoverSSM commit kernels and helpers across models
RecoverSSM's commit-plan and conv-window compaction kernels are model independent.
Move them from models/kimi_k3 into model_executor/layers/mamba/ops/recoverssm.py
(Kimi-K3 keeps its private names as import aliases; kernel source unchanged), add
mamba/recoverssm_utils.py (activation predicate, argument validation) and the
generic spec-row/request mapping helpers to v1/attention/backends/recoverssm_metadata.py.

CacheConfig.use_kda_recoverssm becomes use_recoverssm: the switch now selects RecoverSSM
for any model whose recurrent layers implement it, not only KDA.

No functional change for Kimi-K3.

Signed-off-by: Jürgen Schmied <juergenschmied70@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-09-27 00:07:04 +02:00
vr8vr8 848dbf89ec Merge branch 'main' into qwen4exp-qsa-fused-draft-metadata 2026-09-23 23:37:13 +03:00
bilikasandClaude Opus 5.5 31d78a21b9 [Spec Decode][Qwen3.8] Enable fused draft metadata updates for the QSA side cache
QSAMetadataBuilder did not set supports_draft_decode_metadata_update, so
MTP with num_speculative_tokens > 1 fell back to rebuilding attention
metadata between draft steps for the whole model.

Opt the builder in. update_draft_decode_metadata re-launches the builder's
own _build_qsa_metadata_kernel in place on the metadata's persistent
tensors, so draft-step token_to_req, logical_positions, visible_blocks,
slot_mapping and k_work_metadata come from the same code path as build().
The kernel launch is factored into _launch_qsa_metadata_kernel, and
QSAForwardMetadata carries common_slot_mapping and num_mapped_tokens so
the update does not read query_start_loc_cpu. The PyTorch fallback builder
keeps the rebuild path.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: bilikas <valdas@vbtech.eu>
2026-09-23 20:58:12 +03:00
lesj0610 045467c102 Merge remote-tracking branch 'origin/main' into sync/lesj/qwen4exp-bugfixes
Signed-off-by: lesj0610 <lesj0610@users.noreply.github.com>
2026-09-23 00:40:48 +09:00
lesj0610 51d5b3ac8a Merge remote-tracking branch 'origin/main' into sync/lesj/qwen4exp-bugfixes
Signed-off-by: lesj0610 <lesj0610@users.noreply.github.com>
2026-09-20 22:55:38 +09:00
lesj0610 5780c0125a Merge remote-tracking branch 'origin/main' into sync/lesj/qwen4exp-bugfixes
Signed-off-by: lesj0610 <lesj0610@users.noreply.github.com>
2026-09-20 09:27:57 +09:00
lesj0610 1eacb41e0f Merge remote-tracking branch 'origin/main' into sync/lesj/qwen4exp-bugfixes
Signed-off-by: lesj0610 <lesj0610@users.noreply.github.com>
2026-09-17 15:01:29 +09:00
lesj0610 b8adc7710e Merge remote-tracking branch 'origin/main' into sync/lesj/qwen4exp-bugfixes
Signed-off-by: lesj0610 <lesj0610@users.noreply.github.com>
2026-09-17 12:32:30 +09:00
lesj0610 77b8783b25 Merge remote-tracking branch 'origin/main' into sync/lesj/qwen4exp-bugfixes
Signed-off-by: lesj0610 <lesj0610@users.noreply.github.com>
2026-09-17 11:29:18 +09:00
lesj0610 6c5552776b Merge remote-tracking branch 'origin/main' into lesj/qwen4exp-bugfixes
Both sides edited the same import block in the Qwen4Exp NVIDIA and AMD
models and nothing else collided. This branch adds
`from ..common.mtp import make_mtp_hidden_buffer`, the helper the MTP
buffer extraction calls; upstream ec4a3a5370 widened the `..config`
import to `ATTENTION_LAYER_TYPES, QSA_LAYER_TYPE, Qwen4ExpConfig` for the
layer-type constants it introduced. The two changes are independent, so
the resolution keeps both lines. Every symbol still has a use outside the
import in both files.

Signed-off-by: lesj0610 <lesj0610@users.noreply.github.com>
2026-09-17 08:16:59 +09:00
lesj0610 e582b9f4a2 Merge remote-tracking branch 'origin/main' into sync/lesj/qwen4exp-bugfixes
Signed-off-by: lesj0610 <lesj0610@users.noreply.github.com>
2026-09-16 20:33:18 +09:00
lesj0610 21d2dd56d8 Merge remote-tracking branch 'origin/main' into sync/lesj/qwen4exp-bugfixes
Signed-off-by: lesj0610 <lesj0610@users.noreply.github.com>
2026-09-16 13:41:12 +09:00
lesj0610 a66a3e1b4e Merge remote-tracking branch 'origin/main' into sync/lesj/qwen4exp-bugfixes
Signed-off-by: lesj0610 <lesj0610@users.noreply.github.com>
2026-09-14 14:36:05 +09:00
lesj0610andClaude Opus 5 3b1911c0ca [Bugfix][Model] Fix Qwen4Exp MTP buffer placement, config parsing, and Triton warmup
Three initialization and config defects found serving Qwen4Exp, each with a
regression test.

**The MTP hidden buffer lands on the ambient device.** torch.empty for the
spec-decode hidden buffer takes no device, so it follows whatever device is
current rather than vllm_config.device_config.device. The allocation was
duplicated verbatim in the NVIDIA and AMD backends; it moves to
common/mtp.py so the two cannot drift, and the caller passes is_last_rank
rather than reaching for the process group from shared code.

**M-RoPE keys are reported as unrecognized.** Qwen4Exp drives M-RoPE through
mrope_section and mrope_interleaved, which transformers' rope validator does
not know for rope_type="default", so every load warns about a configuration
the model genuinely uses. Declare them at rope validation on the text config
and on the outer config that mirrors it; every other key still warns.

**transformers' layer_types spelling is unread.** Its Qwen4Exp config
canonicalizes "full_attention" entries to "qwen_sparse_attention" in memory,
so checkpoints re-exported through it serialize that name. vLLM keys QSA off
"full_attention" plus indexer_n_heads, so map the spelling back.

**Qwen4Exp is missing from the Triton warmup set.** Its GDN prefill and QSA
run on other backends, but the kernels this warmup covers -- the gated RMS
norm, the causal conv, the post-conv preparation, the decode state update --
are Triton whichever prefill kernel is chosen, so they compile on the first
prompt instead. qwen4_exp_mtp stays out: kernel_warmup passes the worker's
own model_config, which is the target model's.

tests/models/qwen4_exp/test_config.py: 15 passed.
tests/model_executor/test_qwen_triton_warmup.py: 6 passed.
ruff check, ruff format --check and mypy (Python 3.10) report what they
already report on the base commit.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: lesj0610 <lesj0610@users.noreply.github.com>
2026-09-14 07:25:01 +09:00
2786 changed files with 64206 additions and 248752 deletions
@@ -1,115 +0,0 @@
---
name: model-microbenchmark
description: Time a vLLM model's forward pass on exact batch shapes (decode, prefill, chunked prefill, prefix hit, spec-decode verify, or mixes) with `vllm bench forward`, single-GPU or DP/TP/EP across nodes. Use for per-step model latency, layout/parallelism comparisons, and checking a model-level optimization. Not for single kernels (use kernel-microbenchmark) or end-to-end serving (use `vllm bench serve`).
---
# Model Microbenchmark (`vllm bench forward`)
Builds the batch without the scheduler, runs it through MRV2's real
`execute_model`, and times the forward as a one-shot CUDA graph (prefill
included). Sampling, the speculator, and CPU step overhead are excluded, so
expect results somewhat below a serving step.
## Usage
```bash
vllm bench forward --model deepseek-ai/DeepSeek-V4-Flash \
--case decode=64*1@8192 --case prefill=8192 --case mix=4096,32*1@8192
# DP/TP/EP: one process per GPU (multi-node: add --nnodes/--node-rank/--master-addr)
torchrun --nproc-per-node=4 -m vllm.entrypoints.cli.main bench forward \
--model ... -dp 4 --enable-expert-parallel --case decode=128*1@100000
```
Requests: `[N*]Q[@K][c][:D]`, comma-separated, per DP rank.
| Form | Meaning |
| --- | --- |
| `8192` | fresh prefill, 8192 tokens |
| `32*1@8192` | 32 decodes, KV length 8192 (KV length **includes** the query) |
| `4@4096:3` | spec-decode verify, 3 draft tokens |
| `2048@10240` | prefill after an 8192-token prefix-cache hit |
| `2048@10240c` | chunked prefill: the prefix came from earlier chunks |
Verify cases require `--speculative-config` with `num_speculative_tokens`
at least as large as the case's draft count; insufficient capacity is rejected.
Hit vs chunk only differs on models with SWA bounded replay (DSV4.1): a hit
also recomputes the last window (128 tokens) of the prefix.
Or `--shapes file.json|yaml`:
`{"cases": [{"name": ..., "requests": [{"query_len", "kv_len", "num_draft_tokens", "chunked", "count"}]}]}`.
Key flags (plus all `EngineArgs`):
- `--kv-len-jitter 0.5` (default): prefixes drawn within ±50%, same on every
DP rank, capped at `max_model_len`. Use `0` for exact lengths.
- `--routing auto|real|balanced`: `auto` = balanced with dummy weights.
- `--iters 20 --samples 5`, `--output-json out.json` (per-rank latency stats,
`samples_us`, graph-vs-eager check).
- Defaults: `--load-format dummy`, prefix caching off, no torch.compile.
Read `median_us` (each replay is already the max over ranks).
## Kernel breakdown
Add `--profile-dir DIR` to see where the time goes. After timing, each case is
replayed 3 more times under `torch.profiler`; every rank writes
`DIR/<case>.rank<r>.kernels.json` (GPU time per kernel per forward, sorted)
and `DIR/<case>.rank<r>.trace.json.gz` (open in Perfetto). Rank 0's top
kernels are printed and included in `--output-json`.
```bash
python -m vllm.benchmarks.forward.profile DIR/decode.rank0.kernels.json
# Before/after a change, same case: per-kernel deltas, largest first.
python -m vllm.benchmarks.forward.profile base/decode.rank0.kernels.json new/decode.rank0.kernels.json
```
- Kernels are grouped by name without template or function arguments; the
JSON keeps one full name per group in `example`.
- Only the timed forward is included: the in-graph KV restore and rank
rendezvous are excluded (`pre_forward_us`). `span_us` (first to last
forward kernel) should be close to `median_us`, a bit higher from profiler
overhead; `busy_us` can exceed it when streams overlap.
- Compare the same rank across runs: under EP, ranks differ (e.g. collective
wait time shows up inside communication kernels).
For Nsight Systems (NVTX, NCCL, CPU side, GPU metrics), skip `--profile-dir`:
the two can't share CUPTI. Each case's 3 profiled replays run between
`cudaProfilerStart`/`Stop` in an NVTX range `forward/<case>`, so a capture range
records only them, one report per case (and rank):
```bash
nsys profile --capture-range=cudaProfilerApi --capture-range-end=repeat \
--cuda-graph-trace=node -o prof vllm bench forward ... # prof.1, prof.2, ...
torchrun --nproc-per-node=4 --no-python nsys profile ... -o prof_rank%q{RANK} \
python -m vllm.entrypoints.cli.main bench forward ...
nsys stats --report cuda_gpu_kern_sum --format csv -o kern prof.1.nsys-rep
```
Keep `--capture-range-end=repeat`: nsys's default stops after the first case
and leaves the bench hanging. `nsys stats` also counts the KV restore copies
that the JSON summary excludes.
## Caveats
- **Dummy weights lie for data-dependent kernels.** Dummy gates route every
token to the same experts (hence balanced routing). Cross-check suspicious
kernels with real weights.
- **torch.compile models (e.g. Qwen3/3.5) look slow**: no compile here. Use
them for correctness only; trust perf for models under `vllm/models/`.
- `sample spread > 0.5%` warns of unstable timing. DSV4.1 DEP prefill is
inherently ~0.5–1% (MegaMoE cross-rank sync, host engram reads).
- `graph_vs_eager` may be nonzero for small batches where models branch on
`is_current_stream_capturing()`; that's expected, not a bug.
- KV cache is zero-initialized (`--kv-init zeros`); random KV was within 3%.
- Uniform DP only: every rank gets the same case; no idle ranks.
- CUDA only: it hangs on ROCm (untested cause: likely the in-graph timing
events).
- Large prefills/prefix hits need room: raise `--max-num-batched-tokens` /
`--max-model-len` / `--gpu-memory-utilization` if build fails on KV blocks.
- Startup cost is JIT/autotune, not the bench: install FlashInfer jit-cache
and cubin; DeepGEMM warmup is skipped (each case's eager pass compiles its
own shapes).
- To compare against serving decode: `vllm bench latency` with 65- vs
1-token outputs, divided by 64, with a FULL decode graph.
-157
View File
@@ -1,157 +0,0 @@
---
name: model-recipes
description: Find vLLM deployment recipes and configs for LLMs and Hugging Face models by model and hardware. Use for recipe lookup, creation, and updates.
---
# Model Recipes
Model serving recipes are maintained in this repository's
[recipes/models/ directory](../../../recipes/models). Omni (image, video,
audio) recipes stay in vllm-project/recipes for now (moving to vllm-omni),
and Ascend NPU content is inline in these recipes (moving to vllm-ascend
overlays). The published catalog, rendered by
[vllm-project/recipes](https://github.com/vllm-project/recipes), is at
[recipes.vllm.ai](https://recipes.vllm.ai).
## Usage notes
1. Recipe lookup returns commands and deployment guidance without launching
`vllm serve`.
2. Use published configurations or the recipes repository's command generator
for supported selections. Return commands as published, including their
flags and parallelism settings.
3. Recipes are starting points for deployment and may not be tuned for your
workload. Check each recipe for its verified configurations, and use
benchmarks of your workload to guide further tuning.
4. To create or update a recipe, follow the
[authoring workflow below](#create-or-update-a-recipe).
## Find a recipe
Read recipes from the published JSON on recipes.vllm.ai. It is generated from
the source YAML with variant, hardware, and strategy settings already
resolved; the YAML contains comments and conditional overrides, so do not use
it for lookup. When presenting results, link the website page. Read the source
YAML only when creating or updating a recipe.
Each recipe has a page at `https://recipes.vllm.ai/<org>/<repo>` and JSON at
the same path plus `.json`. Paths use the full Hugging Face ID, preserving
case; do not substitute `meta.slug` or a display title. For example,
`Qwen/Qwen3.8-27B` maps to
[recipes.vllm.ai/Qwen/Qwen3.8-27B](https://recipes.vllm.ai/Qwen/Qwen3.8-27B).
Recipe JSON is often over 20 KB, mostly the `guide`. Instead of printing it,
read only what the request needs: each step below returns the paths for the
next, so stop once the request is answered. Commands need `curl` and `jq`.
[scripts/read_recipe.sh](scripts/read_recipe.sh) is in this skill's directory;
the examples use its path from the vLLM repository root.
### Search the catalog
Skip this step if the exact HF ID is known. Otherwise search by model ID,
family, or provider using case-insensitive keywords. Each
whitespace-separated word must occur in the ID, title, or provider:
```bash
curl -fsSL https://recipes.vllm.ai/models.json |
jq --arg q 'qwen 27b' '
($q | ascii_downcase | [scan("\\S+")]) as $words | .[] |
([.hf_id, .title, .provider] | join(" ") | ascii_downcase) as $text |
select(all($words[]; . as $word | $text | contains($word))) |
{hf_id, json, url, derived_from}'
```
`json` is the recipe data path; `url` is its page path. Prefix both with
`https://recipes.vllm.ai`. A `derived_from` entry is a variant of a parent
recipe: fetch its own `json` to get that checkpoint's command. Its page may
belong to the parent, so link the returned `url` rather than assuming every
checkpoint has a separate page.
If there is no exact match, search the family and inspect `variants` before
concluding the recipe is missing.
### Summarize the recipe
Save the recipe JSON to a file and summarize it:
```bash
read_recipe=.agents/skills/model-recipes/scripts/read_recipe.sh
recipe=$(mktemp)
curl -fsSL https://recipes.vllm.ai/Qwen/Qwen3.8-27B.json -o "$recipe"
"$read_recipe" "$recipe"
```
`default` is the recommended hardware, strategy, and variant; `hardware` maps
each hardware ID to its command path; `recipe_verified_hardware` lists hardware
where the recipe as a whole was verified end to end. For capability
requirements, such as tool calling or text-to-video, check `features` and
`omni_tasks`, then confirm in the guide.
Some recipes, including Omni, lack `hardware`; skip hardware selection and
use the guide's launch and task-specific sections.
### Select hardware and strategy
Match a hardware ID to the user's GPU and fetch its path from `hardware`,
prefixed with `https://recipes.vllm.ai`, rather than constructing a URL. That
response is the command object itself: single-node objects carry `command`,
and multi-node objects carry `head_command` and `worker_commands` instead, so
list its keys before selecting fields.
Check `hardware_profile` to confirm that the selected configuration matches
the user's GPU model and specifications. Its `gpu_count` describes the profile,
not necessarily the number of GPUs used by the command. Check the command's
parallelism settings, `node_count`, and all `worker_commands` against the
user's available topology, and compare the variant's `vram_minimum_gb`, an
estimate, with the total memory of the GPUs the command uses; a configuration
below that minimum is not a working option unless the guide says it was
tested. Ask when the user's hardware is unclear. If no
published command or guide command fits, report the mismatch and return the
closest one unchanged; list any adjustments the user's setup would need
separately and mark them unverified instead of editing the command.
For another strategy, follow `alternatives` in the same response. These are
static files; query parameters do not change their configuration. For other
feature/topology selections, use the recipes repository's command synthesis.
### Read deployment guidance
Read the guide sections that apply to the selection, such as prerequisites,
launch notes for the selected hardware or variant, troubleshooting, and known
limitations. Pass entries from `guide_sections` or prefixes of their text.
The script prints every matching section with its subsections and fails when
a section matches nothing:
```bash
"$read_recipe" "$recipe" prerequisites troubleshooting
```
Query other fields with `jq` on the saved file, such as `features.<name>`,
`hardware_overrides`, or `omni.tasks` for request examples.
### Return the result
Return the website link, selected checkpoint and hardware, the requested
command, and relevant prerequisites and caveats from the guide. Pick one
command and copy it, including its Docker form, exactly as the command object
or guide section gives it; do not combine commands from different sections or
add, drop, or reorder flags. List optional changes separately.
For each command, state its source, either a generated hardware or strategy
configuration or a named guide section, and whether that source says it was
tested. A `recipe_verified_hardware` entry covers the recipe on that
hardware, not every variant, strategy, or guide command, so it does not make a
command verified. Generated configurations are not automatically verified.
## Create or update a recipe
Recipes cover official or widely used checkpoints, with configurations verified
end to end on real hardware. Do not add recipes for arbitrary community
uploads, untested hardware claims, or non-standard flags that only work for one
setup. Before opening a pull request, confirm that the user ran the
configuration and can share the results.
Edit the YAML under [recipes/models/](../../../recipes/models) in this
repository, following the recipes repository's
[add-recipe skill](https://github.com/vllm-project/recipes/blob/main/.agents/skills/add-recipe/SKILL.md)
for the schema. Validate and preview it from a checkout of
`vllm-project/recipes` pointed at this directory, as
[recipes/README.md](../../../recipes/README.md) describes.
@@ -1,4 +0,0 @@
interface:
display_name: "Model Recipes"
short_description: "Find vLLM deployment recipes and configs for LLMs and Hugging Face models by model and hardware."
default_prompt: "Use $model-recipes to find a recipe for my model and hardware, and return its commands and deployment caveats."
@@ -1,66 +0,0 @@
#!/bin/bash
# Usage: read_recipe.sh <recipe.json> [guide-section...]
# Without sections, print a compact summary of a recipes.vllm.ai recipe JSON.
# With sections, print each guide section whose heading starts with one of
# them (case-insensitive), including its subsections.
set -euo pipefail
if [ $# -lt 1 ]; then
echo "Usage: $0 <recipe.json> [guide-section...]" >&2
exit 1
fi
# Print "<level>\t<heading text>\t<line>" for guide lines, marking headings
# outside code fences with their level and other lines with 0.
guide_lines() {
jq -r '.guide // ""' "$1" | awk '
/^[[:space:]]*(```|~~~)/ { fence = !fence }
!fence && match($0, /^#+ /) {
print RLENGTH - 1 "\t" substr($0, RLENGTH + 1) "\t" $0
next
}
{ print "0\t\t" $0 }'
}
if [ $# -gt 1 ]; then
recipe=$1
shift
status=0
for section in "$@"; do
guide_lines "$recipe" | awk -F '\t' -v h="$section" '
BEGIN { h = tolower(h); sub(/^#+[[:space:]]*/, "", h) }
$1 > 0 && lvl && $1 <= lvl { lvl = 0 }
$1 > 1 && !lvl && index(tolower($2), h) == 1 { lvl = $1; found = 1 }
lvl { sub(/^[^\t]*\t[^\t]*\t/, ""); print }
END {
if (!found) {
print "No guide section starts with: " h > "/dev/stderr"
exit 1
}
}' || status=1
done
exit $status
fi
sections=$(guide_lines "$1" | awk -F '\t' '$1 == 2 || $1 == 3 { print $3 }' |
jq -R . | jq -s .)
jq --argjson sections "$sections" '{
hf_id,
title: .meta.title,
tasks: .meta.tasks,
recipe_verified_hardware: .meta.hardware,
omni_tasks: (.omni.tasks // null |
if . then map(objects |= {id, label, endpoint, model_id, description})
else null end),
variants: (.variants // {} | map_values(
{model_id, precision, vram_minimum_gb, supported_hardware} |
with_entries(select(.value != null)))),
features: (.features // {} | keys),
opt_in_features,
default: (.recommended_command // null |
if . then {hardware, strategy, variant} else null end),
hardware: .recommended_command.by_hardware,
guide_sections: $sections
} | with_entries(select(.value != null))' "$1"
+1 -3
View File
@@ -1,6 +1,6 @@
--- ---
name: pr-checklist name: pr-checklist
description: Prepare a vLLM change for human PR review or re-review, or analyze an open pull request. Use before opening a PR, drafts included (e.g. `gh pr create`), or pushing review fixes, to check design fit, behavioral coverage, performance evidence, diff quality, and closure of previous feedback. description: Prepare a vLLM change for human PR review or re-review, or analyze an open pull request. Use to check design fit, behavioral coverage, performance evidence, diff quality, and closure of previous feedback before requesting maintainer attention.
--- ---
# PR Checklist # PR Checklist
@@ -181,8 +181,6 @@ Summarize model evaluations and benchmarks in compact tables with the configurat
When fixing bugs or issues, provide a clear root-cause analysis. Explain what caused the problem, how it was identified, and why the chosen solution effectively addresses it. A patch which resolves the issue without a proper root-cause analysis may lead to recurring problems and increased maintenance overhead. When fixing bugs or issues, provide a clear root-cause analysis. Explain what caused the problem, how it was identified, and why the chosen solution effectively addresses it. A patch which resolves the issue without a proper root-cause analysis may lead to recurring problems and increased maintenance overhead.
For fixes that involve disaggregated prefill/decode (PD) logic, such as the NIXL connector (`vllm/distributed/kv_transfer/kv_connector/v1/nixl/`), make a best effort to include a minimal example of the flow being changed and/or the initial flow that caused the failure. A short numbered sequence (or diagram) of a request's path across the P and D instances, e.g. scheduler admission, prefill execution, KV transfer registration and write, decode-side read, token generation, lets reviewers trace the root cause and the fix without reconstructing the distributed flow from the diff. Keep the example minimal: only the steps and components involved in the change. See the [disaggregated prefill](../../../docs/features/disagg_prefill.md), [NIXL KV push](../../../docs/design/nixl_kv_push_connector.md), and [NIXL KV cache lease](../../../docs/design/nixl_kv_cache_lease.md) docs for the flows involved.
#### 4.4: Implementation Details #### 4.4: Implementation Details
Provide a detailed explanation of the implementation, including design decisions, trade-offs, and any relevant technical details. Highlight any non-obvious choices, tradeoffs, potential limitations, and areas where maintainer attention is required. Provide a detailed explanation of the implementation, including design decisions, trade-offs, and any relevant technical details. Highlight any non-obvious choices, tradeoffs, potential limitations, and areas where maintainer attention is required.
@@ -1,191 +0,0 @@
---
name: transformers-upgrade
description: "Playbook for changing vLLM's Transformers version bounds. Use when bumping the pinned or maximum Transformers version, raising the minimum Transformers version, or deleting vendored configs, processors, or compat shims that upstream Transformers now provides."
---
# Upgrading Transformers
The supported range lives in `requirements/common.txt`. CI pins one exact
version in `requirements/test/*.in` and the compiled `requirements/test/*.txt`.
A version change is one of two jobs, and they usually come as a pair:
- **Bump the pin**: a new Transformers release becomes the CI version and
the upper bound. vLLM grows compat code for it.
- **Raise the floor**: the minimum moves up. That compat code, and every
vendored config or processor upstream now ships, becomes dead weight.
Every floor raise ends with an **unvendoring pass**.
## Bump the pin
1. Edit the pin line in each `requirements/test/*.in` and `*.txt` and the
upper bound in `requirements/common.txt`. Past bumps edit only the pin
line in the compiled files; see `git log -- requirements/common.txt`.
2. Read the release notes for config validation changes, renamed auto-mapping
formats, and removed attributes; these are what break vLLM.
3. Search `tests/` for `min_transformers_version` and skip markers naming the
new version, and un-gate them.
4. Run the suites in [Testing](#testing) and fix what breaks. Mark every
shim with a `# Transformers >= X` comment so the next floor raise finds it.
Done when the [Testing](#testing) suites pass on the new pin and every test
waiting on this version is un-gated.
## Raise the floor (unvendoring pass)
### 1. Inventory
Inventory candidates with the floor version installed (no vLLM install needed):
```bash
uv run --no-project --with transformers==<floor> \
python .agents/skills/transformers-upgrade/scripts/find_upstreamed.py
```
It lists:
- vendored configs whose `model_type` upstream now owns;
- `_CONFIG_REGISTRY` overrides of upstream types;
- registered processors that collide with upstream class names or models;
- version-gated code at or below the floor;
- code that reads the installed Transformers version, whatever its threshold;
- patches citing a Transformers issue or PR (with whether a referenced PR is
in the floor release) or a future minimum version.
It finds candidates by name only. Every hit still needs a verdict.
### 2. Port each candidate to upstream
Every hit is deleted by default. A vendored copy is a liability: it drifts
from upstream, hides upstream fixes, and shadows the native class. Port the
vLLM model code to the upstream schema instead, renaming attribute reads and
deriving values from upstream fields.
Fan candidates out to parallel subagents. Each one compares the vendored and
upstream classes, then greps every attribute the vLLM model code reads.
[Upstreaming hazards](#upstreaming-hazards) lists the usual fixes.
Keep a vendored class only when upstream cannot load a checkpoint vLLM must
keep serving, and record why in the PR description. A different upstream
schema, a model-code rewrite, or a needed eval is work for the PR, not a
reason to keep.
### 3. Prove it on real checkpoints
For every checkpoint in `tests/models/registry.py` that uses a ported
config, dump what vLLM builds before and after the change, then diff the two.
Run both dumps; each catches what the other misses:
- `dump_configs.py`: the resolved config, attribute by attribute.
- `dump_model.py`: the model built on the meta device with no GPU or weights.
It records ModelConfig sizes, every parameter shape, and every module's
scalar attributes (`head_dim`, `num_kv_heads`, `scaling`, ...). Missing
fields that are read with defaults only show up here.
```bash
git worktree add --detach <scratch>/vllm-base upstream/main
S=.agents/skills/transformers-upgrade/scripts
for dump in dump_configs dump_model; do
PYTHONPATH=<scratch>/vllm-base .venv/bin/python $S/$dump.py before.jsonl <repos>
PYTHONPATH=$PWD .venv/bin/python $S/$dump.py after.jsonl <repos>
.venv/bin/python $S/diff_dumps.py before.jsonl after.jsonl
done
```
Add `--trust-remote-code` for checkpoints whose `config.json` has an
`auto_map`, and dump both ways.
How to read the diffs:
- **A `<MISSING>` config field the model reads:** this crashes or silently
falls back to a default. Port the model code to the upstream field.
- **New upstream-only defaults** (`attention_bias=False`,
`pad_token_id=None`): usually benign. A model diff of 0 confirms it.
- **Any model diff:** this is a numerics change, and the PR needs a model
eval. For example, A.X-K1 gained the checkpoint's YaRN rope scaling, which
the vendored class had been dropping; it shows up as
`layers.*.self_attn.scaling (x61)`.
### 4. Delete
For each deleted config:
- Delete the file.
- Remove its entries from `_CLASS_TO_MODULE` and `__all__` in
`vllm/transformers_utils/configs/__init__.py`.
- Remove its entries from `_CONFIG_REGISTRY` in
`vllm/transformers_utils/config.py`.
- Switch every importer, including tests, to `from transformers import ...`.
- Remove the `source_file_dependencies` lines naming the file from
`.buildkite/`.
Deleted processors follow the same steps in
`vllm/transformers_utils/processors/__init__.py`.
Before merging, rebase and grep again for the old field names: PRs that land
in between still read them.
Done when every line of the inventory output is deleted, or kept with a
reason that meets the bar in step 2. The PR also carries the config diff
results and evals for any changed values.
## Upstreaming hazards
Upstream configs are `@strict` dataclasses. Checkpoint keys upstream does not
declare survive as plain attributes, but only when the checkpoint sets them.
Defaults that only vLLM sets, and values vLLM derives, vanish.
| Hazard | Signal | Fix |
| --- | --- | --- |
| Upstream consumes legacy keys | `__post_init__` pops keys the model reads (`deepseek_v4` `compress_ratios`) | Read the upstream replacement (`layer_types`, `mlp_layer_types`, per-layer `rope_parameters`) |
| Renamed or dropped fields | Diff shows `<MISSING>` for a field the model reads | Read the upstream name; derive dropped flags from what implies them (`enable_moe_block` from `num_experts`) |
| Undeclared field without a default | `config.x` raises `AttributeError` for configs that omit `x`, e.g. minimal test configs (A.X-K1 `scoring_func`). Dumps of real checkpoints miss it | Upstream ignoring `x` usually means vLLM can stop reading it; if vLLM truly needs it, read `getattr(config, "x", <old default>)` |
| Top-level config stops forwarding | The vendored top-level config forwarded reads to `text_config`; upstream does not. `getattr(config, "x", default)` silently takes the default, e.g. on MTP draft configs from `SpeculativeConfig.hf_config_override`, which the dumps do not build | Read from `get_text_config()`; grep `getattr` reads of every field, including speculative-decoding paths |
| Type check on the config | `ctx.get_hf_config(SomeConfig)` relies on the vendored base class | Accept the upstream class too, e.g. a `Cosmos3ProcessingInfo` overriding `get_hf_config` |
| Remote code takes over | Checkpoint `config.json` has `auto_map`; with `--trust-remote-code`, the hub class replaces the native one once vLLM stops registering its own | Check the remote class loads; if not, register the upstream class for that `model_type` |
| Read-only upstream fields | Properties or derived fields break subclasses and `hf_overrides` (`nemotron_h` `hybrid_override_pattern`) | Override the source field (`layers_block_type`) instead |
| Behavioural rewrite | Upstream derives fields differently (`diffusion_gemma` `sliding_window`) | Upstream matches the reference implementation; verify with an eval |
| Original checkpoint format | Upstream targets `-hf` conversions; vLLM serves the originals (`qwen3_asr`, `qianfan_ocr`) | Keep only if upstream cannot load the original at all |
| Processor output contract | Model code consumes keys upstream does not emit | Port the model to upstream's keys |
| Name collision | A registered vendored processor shares an upstream class name and hijacks `get_processor` (`InternVLProcessor`) | Drop the registry entry |
Version gates from the inventory need a decision:
- **Code** only reachable below the floor is deleted. This covers:
- comparisons against `TRANSFORMERS_VERSION`;
- branches labelled `# Transformers < X`;
- fallbacks for fields that upstream configs now always set, e.g.
`mlp_layer_types`, `per_layer_config` or legacy `layer_types` names.
A `skipif` whose condition is now always true becomes an unconditional
`skip` with the same reason.
- **`min_transformers_version`** at or below the floor is removed.
- **`max_transformers_version`** below the floor means CI never tests that
model; report it rather than deleting it silently.
- **Patches citing a Transformers issue or PR:**
- Delete the patch when its fix is in the floor release.
- For an issue, find the PR that closed it and check that PR. A closed
issue is not a fixed one: issues get closed while the bug is still in
the release.
- Re-run the patched behaviour before deleting. A fix in the release does
not always cover vLLM's path: Emu3's processor gained BOS upstream, but
vLLM tokenizes without the processor.
- Links that only explain a design choice (e.g. a float32 RoPE note) are
not patches; leave them.
## Testing
```bash
.venv/bin/python -m pytest tests/transformers_utils tests/config tests/tokenizers_
.venv/bin/python -m pytest tests/models/multimodal/processing/test_common.py -k "<models>"
```
Add `tests/models/test_initialization.py -k "<archs>"` where a GPU is
available, and a model eval for any model whose resolved config changed.
Models in `vllm/models/` are high priority: whenever their config or model
code changes, run GSM8k (`tests/evals/gsm8k/`) and GPQA Diamond
(`tests/evals/gpt_oss/`) on `main` and the branch, even when the dumps show
no diff, and put both scores in the PR. FlashInfer autotuning picks kernel
tactics afresh on every server start, which moved DeepSeek-V4-Flash GSM8k by
about 1 point; pass `--kernel-config '{"enable_flashinfer_autotune": false}'`
to both servers so greedy outputs are comparable.
@@ -1,4 +0,0 @@
interface:
display_name: "Transformers Upgrade"
short_description: "Bump vLLM's Transformers version and delete vendored code upstream now provides."
default_prompt: "Use $transformers-upgrade to raise the minimum Transformers version and remove vendored configs, processors, and shims it makes redundant."
@@ -1,51 +0,0 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""Diff two dump_configs.py or dump_model.py outputs, per repo.
python diff_dumps.py before.jsonl after.jsonl
"""
import json
import sys
from collections import Counter
import regex as re
IGNORED = {"transformers_version", "_name_or_path", "architectures", "auto_map"}
def load(path: str) -> dict[str, dict]:
with open(path) as f:
return {record["repo"]: record for record in map(json.loads, f)}
def diff(before: dict, after: dict, prefix: str = ""):
for key in sorted(set(before) | set(after)):
if key in IGNORED:
continue
old, new = before.get(key, "<MISSING>"), after.get(key, "<MISSING>")
if isinstance(old, dict) and isinstance(new, dict) and key != "rope_parameters":
yield from diff(old, new, f"{prefix}{key}.")
elif old != new:
yield f"{prefix}{key}", json.dumps(old), json.dumps(new)
def collapse(diffs) -> Counter:
"""Merge diffs that differ only by layer index, e.g. `layers.*.scaling`."""
return Counter(
(re.sub(r"\.\d+\.", ".*.", path), old, new) for path, old, new in diffs
)
before, after = load(sys.argv[1]), load(sys.argv[2])
for repo, old in before.items():
new = after.get(repo, {"error": "missing from after"})
if "error" in old or "error" in new:
print(f"== {repo}\n before: {old.get('error', 'ok')}")
print(f" after: {new.get('error', 'ok')}")
continue
diffs = collapse(diff(old["cfg"], new["cfg"]))
print(f"== {repo}: {old['cls']} -> {new['cls']} ({sum(diffs.values())} diffs)")
for (path, old_value, new_value), count in diffs.items():
times = f" (x{count})" if count > 1 else ""
print(f" {path}{times}: before={old_value} after={new_value}")
@@ -1,30 +0,0 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""Dump the config vLLM resolves for each checkpoint, one JSON line per repo.
Run once per vLLM tree (before and after a change), then compare the two
outputs with diff_dumps.py:
PYTHONPATH=<vllm tree> .venv/bin/python dump_configs.py out.jsonl <repo>...
Pass --trust-remote-code to resolve checkpoints the way users who set it do.
"""
import json
import sys
from vllm.transformers_utils.config import get_config
out, *args = sys.argv[1:]
trust_remote_code = "--trust-remote-code" in args
with open(out, "w") as f:
for repo in (arg for arg in args if arg != "--trust-remote-code"):
try:
config = get_config(repo, trust_remote_code=trust_remote_code)
cls = f"{type(config).__module__}.{type(config).__name__}"
record = {"repo": repo, "cls": cls, "cfg": config.to_dict()}
except Exception as e:
error = f"{type(e).__name__}: {e}".splitlines()[0]
record = {"repo": repo, "error": error}
f.write(json.dumps(record, default=str, sort_keys=True) + "\n")
f.flush()
@@ -1,114 +0,0 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""Build each model on the meta device and dump what vLLM constructs.
Records ModelConfig-derived sizes, every parameter shape, and scalar
attributes of every module (head_dim, num_kv_heads, sliding_window, ...).
Needs no GPU and no weights. Run once per vLLM tree, then compare the two
outputs with diff_dumps.py:
PYTHONPATH=<vllm tree> .venv/bin/python dump_model.py out.jsonl <repo>...
Pass --trust-remote-code to resolve checkpoints the way users who set it do.
"""
import json
import socket
import sys
import torch
from vllm.config import set_current_vllm_config
from vllm.config.vllm import VllmConfig
from vllm.distributed import (
init_distributed_environment,
initialize_model_parallel,
model_parallel_is_initialized,
)
from vllm.engine.arg_utils import EngineArgs
from vllm.model_executor.model_loader.utils import initialize_model
from vllm.utils.torch_utils import set_default_torch_dtype
# Construction only: skip model runner checks that need a GPU or Triton
VllmConfig._get_v1_model_runner_unsupported_features = lambda self: [] # type: ignore[method-assign]
# Model code that moves submodules to a device must leave them on meta
_module_to = torch.nn.Module.to
def _meta_safe_to(self, *args, **kwargs):
kwargs.pop("device", None)
args = tuple(a for a in args if not isinstance(a, (torch.device, str, int)))
return _module_to(self, *args, **kwargs)
torch.nn.Module.to = _meta_safe_to # type: ignore[method-assign]
SCALARS = (bool, int, float, str)
def module_attrs(model: torch.nn.Module) -> dict[str, dict]:
attrs = {}
for name, module in model.named_modules():
scalars = {
key: value
for key, value in vars(module).items()
if not key.startswith("_") and isinstance(value, SCALARS)
}
if scalars:
attrs[name or "<root>"] = scalars
return attrs
def dump(repo: str, trust_remote_code: bool) -> dict:
args = EngineArgs(
model=repo,
load_format="dummy",
enforce_eager=True,
max_model_len=4096,
trust_remote_code=trust_remote_code,
)
vllm_config = args.create_engine_config()
model_config = vllm_config.model_config
with set_current_vllm_config(vllm_config):
if not model_parallel_is_initialized():
with socket.socket() as sock:
sock.bind(("", 0))
port = sock.getsockname()[1]
init_distributed_environment(1, 0, f"tcp://127.0.0.1:{port}", 0, "gloo")
initialize_model_parallel(1, 1)
with set_default_torch_dtype(model_config.dtype), torch.device("meta"):
model = initialize_model(vllm_config=vllm_config)
return {
"repo": repo,
"cls": type(model).__name__,
"cfg": {
"model_config": {
"head_size": model_config.get_head_size(),
"total_num_kv_heads": model_config.get_total_num_kv_heads(),
"sliding_window": model_config.get_sliding_window(),
"num_layers": model_config.get_num_layers(vllm_config.parallel_config),
},
"params": {n: list(p.shape) for n, p in model.named_parameters()},
"modules": module_attrs(model),
},
}
def main() -> None:
out, *args = sys.argv[1:]
trust_remote_code = "--trust-remote-code" in args
repos = [arg for arg in args if arg != "--trust-remote-code"]
with open(out, "w") as f:
for repo in repos:
try:
record = dump(repo, trust_remote_code)
except Exception as e:
error = f"{type(e).__name__}: {e}".splitlines()[0]
record = {"repo": repo, "error": error}
f.write(json.dumps(record, default=str, sort_keys=True) + "\n")
f.flush()
if __name__ == "__main__":
main()
@@ -1,147 +0,0 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""List vendored vLLM code that the installed Transformers may make redundant.
Run with the Transformers version under test, from the vLLM repo root:
uv run --no-project --with transformers==<version> \
python .agents/skills/transformers-upgrade/scripts/find_upstreamed.py
Imports nothing from vLLM, so no vLLM install is needed. Uses the `gh` CLI, if
available, to check whether referenced Transformers PRs are in this release.
"""
import json
import subprocess
import sys
from pathlib import Path
import regex as re
import transformers
from packaging.version import Version
from transformers.models.auto.configuration_auto import CONFIG_MAPPING_NAMES
from transformers.models.auto.processing_auto import PROCESSOR_MAPPING_NAMES
ROOT = Path(sys.argv[1] if len(sys.argv) > 1 else ".")
FLOOR = Version(transformers.__version__)
UPSTREAM_NAMES = set(dir(transformers))
MODEL_TYPE = re.compile(r'model_type\s*(?::\s*str\s*)?=\s*"([^"]+)"')
REGISTRY_KEY = re.compile(r'^\s+(?:\*\*\{)?"?([\w-]+)"?(?:=|": )"(\w+)"', re.M)
CLASS_KEY = re.compile(r'^\s+"(\w+)": ', re.M)
UPSTREAM_REF = re.compile(
r"huggingface/transformers(?:/(?:issues|pull)/|#)(\d+)"
r"|(?:once|until|when) [Tt]ransformers [\d.]+ is the minimum"
)
VERSIONED = re.compile(
r"(?:min_transformers_version|max_transformers_version|check_version|Version)"
r'\(?\s*=?\s*"([45]\.\d+(?:\.\d+)?)[^"]*"'
r"|[Tt]ransformers\s*(?:v|[<>]=?\s*)([45]\.\d+(?:\.\d+)?|4)\b"
)
READS_VERSION = re.compile(
r"transformers import __version__|transformers\.__version__"
r"|version\([\"']transformers[\"']\)|TRANSFORMERS_VERSION|check_version\("
)
def section(title: str) -> None:
print(f"\n## {title}")
def vendored_configs() -> None:
section(f"Vendored configs whose model_type is upstream in {FLOOR}")
files = sorted(ROOT.glob("vllm/transformers_utils/configs/*.py"))
files += sorted(ROOT.glob("vllm/models/*/config*.py"))
files += sorted(ROOT.glob("vllm/models/*/configs/*.py"))
for path in files:
types = sorted(set(MODEL_TYPE.findall(path.read_text())))
hits = [t for t in types if t in CONFIG_MAPPING_NAMES]
if hits:
upstream = ", ".join(f"{t} -> {CONFIG_MAPPING_NAMES[t]}" for t in hits)
print(f"- {path.relative_to(ROOT)}: {upstream}")
registry = (ROOT / "vllm/transformers_utils/config.py").read_text()
start = registry.index("_CONFIG_REGISTRY")
body = registry[start : registry.index("\n)\n", start)]
section("_CONFIG_REGISTRY entries that override an upstream model_type")
for model_type, cls in REGISTRY_KEY.findall(body):
if model_type in CONFIG_MAPPING_NAMES:
print(f"- {model_type}: vLLM {cls} over {CONFIG_MAPPING_NAMES[model_type]}")
def vendored_processors() -> None:
init = (ROOT / "vllm/transformers_utils/processors/__init__.py").read_text()
names = sorted(set(CLASS_KEY.findall(init)))
section("Registered processor names that collide with upstream classes")
for name in names:
if name in UPSTREAM_NAMES:
print(f"- {name}")
section("Processor files named after a model_type with an upstream processor")
for path in sorted(ROOT.glob("vllm/transformers_utils/processors/*.py")):
if path.stem in PROCESSOR_MAPPING_NAMES:
upstream = PROCESSOR_MAPPING_NAMES[path.stem]
print(f"- {path.relative_to(ROOT)} -> {upstream}")
def version_gates() -> None:
files = sorted([*ROOT.glob("vllm/**/*.py"), *ROOT.glob("tests/**/*.py")])
section(f"Version-gated code at or below {FLOOR} (review each)")
for path in files:
for lineno, line in enumerate(path.read_text().splitlines(), 1):
for match in VERSIONED.finditer(line):
version = Version(match.group(1) or match.group(2))
if version <= FLOOR:
print(f"- {path.relative_to(ROOT)}:{lineno}: {line.strip()}")
break
section("Code that reads the installed Transformers version (review each)")
for path in files:
for lineno, line in enumerate(path.read_text().splitlines(), 1):
if READS_VERSION.search(line):
print(f"- {path.relative_to(ROOT)}:{lineno}: {line.strip()}")
def upstream_status(number: str) -> str:
"""Whether Transformers issue/PR `number` is resolved in this release."""
def gh(path: str) -> dict | None:
try:
out = subprocess.run(["gh", "api", path], capture_output=True, text=True)
except FileNotFoundError:
return None
return json.loads(out.stdout) if out.returncode == 0 else None
repo = "repos/huggingface/transformers"
if (issue := gh(f"{repo}/issues/{number}")) is None:
return "status unknown"
if "pull_request" not in issue:
return f"issue {issue['state']}, check which PR fixed it"
pr = gh(f"{repo}/pulls/{number}")
if pr is None or not pr["merged_at"]:
return "PR not merged"
compare = gh(f"{repo}/compare/v{FLOOR}...{pr['merge_commit_sha']}")
included = compare and compare["status"] in ("behind", "identical")
return f"PR merged, {'in' if included else 'not in'} v{FLOOR}"
def upstream_refs() -> None:
section("Patches referencing Transformers issues/PRs or a future minimum")
files = sorted([*ROOT.glob("vllm/**/*.py"), *ROOT.glob("tests/**/*.py")])
statuses: dict[str, str] = {}
for path in files:
for lineno, line in enumerate(path.read_text().splitlines(), 1):
if (match := UPSTREAM_REF.search(line)) is None:
continue
note = ""
if number := match.group(1):
if number not in statuses:
statuses[number] = upstream_status(number)
note = f" [#{number}: {statuses[number]}]"
print(f"- {path.relative_to(ROOT)}:{lineno}:{note} {line.strip()}")
if __name__ == "__main__":
print(f"# Transformers {FLOOR}")
vendored_configs()
vendored_processors()
version_gates()
upstream_refs()
+1
View File
@@ -17,6 +17,7 @@ ALLOWED_UNSTABLE_LIBRARIES: tuple[str, ...] = (
# _C_stable_libtorch, but these two modules still bind libtorch directly. # _C_stable_libtorch, but these two modules still bind libtorch directly.
"_C.abi3.so", "_C.abi3.so",
"_rocm_C.abi3.so", "_rocm_C.abi3.so",
"third_party/deep_gemm/_C*.so",
) )
@@ -56,6 +56,7 @@ steps:
- "!vllm/distributed/device_communicators/shm_object_storage.py" - "!vllm/distributed/device_communicators/shm_object_storage.py"
- "!vllm/distributed/device_communicators/symm_mem.py" - "!vllm/distributed/device_communicators/symm_mem.py"
- "!vllm/distributed/device_communicators/mnnvl_compat.py" - "!vllm/distributed/device_communicators/mnnvl_compat.py"
- "!vllm/distributed/device_communicators/ray_communicator.py"
- "!vllm/third_party/flashmla/" - "!vllm/third_party/flashmla/"
- "!vllm/third_party/pynvml.py" - "!vllm/third_party/pynvml.py"
- "!vllm/_custom_ops.py" - "!vllm/_custom_ops.py"
+1 -59
View File
@@ -20,10 +20,8 @@ steps:
- tests/kernels/moe/test_cpu_fused_moe.py - tests/kernels/moe/test_cpu_fused_moe.py
- tests/kernels/moe/test_cpu_quant_fused_moe.py - tests/kernels/moe/test_cpu_quant_fused_moe.py
- tests/kernels/moe/test_zen_cpu_int8_moe.py - tests/kernels/moe/test_zen_cpu_int8_moe.py
- tests/test_zentorch_version_pin.py
- vllm/model_executor/layers/fused_moe/experts/cpu_moe.py - vllm/model_executor/layers/fused_moe/experts/cpu_moe.py
- tests/kernels/test_onednn.py - tests/kernels/test_onednn.py
- tests/kernels/test_cpu_fused_sampling.py
- tests/kernels/test_awq_int4_to_int8.py - tests/kernels/test_awq_int4_to_int8.py
- tests/kernels/quantization/test_cpu_fp8_scaled_mm.py - tests/kernels/quantization/test_cpu_fp8_scaled_mm.py
- tests/kernels/quantization/test_zen_da8w4.py - tests/kernels/quantization/test_zen_da8w4.py
@@ -47,10 +45,8 @@ steps:
tests/kernels/moe/test_cpu_fused_moe.py \ tests/kernels/moe/test_cpu_fused_moe.py \
tests/kernels/moe/test_cpu_quant_fused_moe.py \ tests/kernels/moe/test_cpu_quant_fused_moe.py \
tests/kernels/moe/test_zen_cpu_int8_moe.py \ tests/kernels/moe/test_zen_cpu_int8_moe.py \
tests/test_zentorch_version_pin.py \
tests/kernels/mamba/test_cpu_short_conv.py \ tests/kernels/mamba/test_cpu_short_conv.py \
tests/kernels/test_onednn.py \ tests/kernels/test_onednn.py \
tests/kernels/test_cpu_fused_sampling.py \
tests/kernels/test_awq_int4_to_int8.py \ tests/kernels/test_awq_int4_to_int8.py \
tests/kernels/quantization/test_cpu_fp8_scaled_mm.py \ tests/kernels/quantization/test_cpu_fp8_scaled_mm.py \
tests/kernels/quantization/test_zen_da8w4.py \ tests/kernels/quantization/test_zen_da8w4.py \
@@ -89,7 +85,7 @@ steps:
- | - |
bash .buildkite/scripts/hardware_ci/run-cpu-test.sh 30m " bash .buildkite/scripts/hardware_ci/run-cpu-test.sh 30m "
pytest -x -v -s \ pytest -x -v -s \
tests/v1/worker/test_gpu_target_dependent_ar_speculator.py \ tests/v1/worker/test_gpu_autoregressive_speculator.py \
tests/v1/worker/test_gpu_extract_hidden_states_speculator.py \ tests/v1/worker/test_gpu_extract_hidden_states_speculator.py \
tests/v1/worker/test_gpu_batch_ordering.py \ tests/v1/worker/test_gpu_batch_ordering.py \
tests/v1/worker/test_gpu_rejection_sampler_chunking.py \ tests/v1/worker/test_gpu_rejection_sampler_chunking.py \
@@ -232,57 +228,3 @@ steps:
"$$BUILDKITE_PARALLEL_JOB" "$$BUILDKITE_PARALLEL_JOB"
"$REGISTRY/$REPO:$BUILDKITE_COMMIT-arm64-cpu" "$REGISTRY/$REPO:$BUILDKITE_COMMIT-arm64-cpu"
parallelism: 3 parallelism: 3
- label: ":docker: Build Zen CPU image"
key: zen-cpu-image-build
# Reuse the shared `<repo>:<commit>-cpu` image that image-build-cpu publishes
# to public ECR, so Dockerfile.zen just layers on top instead of recompiling
# vLLM from source. If the pull fails, the build script still falls back to
# building the CPU base from source.
depends_on:
- image-build-cpu
soft_fail: true
agents:
queue: amd-cpu
source_file_dependencies:
- .buildkite/hardware_tests/cpu.yaml
- .buildkite/image_build/image_build_zen_cpu.sh
- docker/Dockerfile.zen
- docker/Dockerfile.cpu
- requirements/cpu.txt
- requirements/build/cpu.txt
- requirements/test/cpu.txt
- cmake/cpu_extension.cmake
- vllm/model_executor/layers/utils.py
- vllm/model_executor/kernels/linear/zentorch_utils.py
- vllm/model_executor/kernels/linear/scaled_mm/zentorch.py
- vllm/model_executor/kernels/linear/mixed_precision/zentorch.py
- vllm/platforms/cpu.py
- vllm/platforms/zen_cpu.py
- vllm/platforms/__init__.py
- tests/model_executor/test_cpu_unquantized_gemm_dispatch.py
- tests/test_zen_cpu_platform_detection.py
- tests/kernels/moe/test_zen_cpu_fused_moe.py
- tests/kernels/moe/test_zen_cpu_int8_moe.py
- vllm/v1/attention/backends/zentorch_sdpa.py
- tests/test_zentorch_version_pin.py
- vllm/v1/attention/backends/mla/prefill/zen_cpu_sdpa.py
- vllm/model_executor/layers/fused_moe/experts/zentorch_moe.py
- tests/kernels/quantization/test_zen_da8w4.py
commands:
- .buildkite/image_build/image_build_zen_cpu.sh "$REGISTRY" "$REPO" "$BUILDKITE_COMMIT"
env:
DOCKER_BUILDKIT: "1"
# Install zentorch directly (if-branch in Dockerfile.zen) instead of
# `uv pip install "vllm[zen]"`, which forces uv to re-resolve vLLM's full
# dependency closure and chokes on its transitive triton-cpu URL dependency.
# Keep this pin in sync with the `zen` extra in setup.py.
ZENTORCH_VERSION: "2.14.0.1"
retry:
automatic:
- exit_status: 1 # Transient Docker/BuildKit failure
limit: 1
- exit_status: -1 # Agent was lost
limit: 2
- exit_status: -10 # Agent was lost
limit: 2
+2 -7
View File
@@ -2,15 +2,10 @@ group: Hardware
steps: steps:
- label: "GH200 Test" - label: "GH200 Test"
key: gh200-test key: gh200-test
timeout_in_minutes: 30
soft_fail: true soft_fail: true
device: gh200 device: gh200
no_plugin: true
optional: true optional: true
working_dir: "/vllm-workspace"
depends_on:
- arm64-image-build
env:
VLLM_WORKER_MULTIPROC_METHOD: "spawn"
commands: commands:
- nvidia-smi - nvidia-smi
- python3 examples/basic/offline_inference/generate.py --model meta-llama/Llama-3.2-1B - bash .buildkite/scripts/hardware_ci/run-gh200-test.sh
@@ -15,7 +15,7 @@ steps:
limit: 2 limit: 2
- exit_status: -10 # Agent was lost - exit_status: -10 # Agent was lost
limit: 2 limit: 2
- label: ":intel: (BMG) Example Test" - label: "XPU example Test"
key: xpu-example-test key: xpu-example-test
depends_on: depends_on:
- image-build-xpu - image-build-xpu
@@ -37,7 +37,7 @@ steps:
- >- - >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'bash .buildkite/scripts/hardware_ci/run-intel-ci-test.sh example' 'bash .buildkite/scripts/hardware_ci/run-intel-ci-test.sh example'
- label: ":intel: (BMG) V1 test" - label: "XPU V1 test"
key: xpu-v1-test key: xpu-v1-test
depends_on: depends_on:
- image-build-xpu - image-build-xpu
@@ -60,7 +60,7 @@ steps:
bash .buildkite/scripts/hardware_ci/run-intel-test.sh bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'bash .buildkite/scripts/hardware_ci/run-intel-ci-test.sh v1' 'bash .buildkite/scripts/hardware_ci/run-intel-ci-test.sh v1'
parallelism: 2 parallelism: 2
- label: ":intel: (BMG) W8A8 FP8 Linear Examples" - label: "XPU W8A8 FP8 Linear Examples"
key: xpu-w8a8-fp8-linear-examples key: xpu-w8a8-fp8-linear-examples
depends_on: depends_on:
- image-build-xpu - image-build-xpu
@@ -83,7 +83,7 @@ steps:
- >- - >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'bash .buildkite/scripts/hardware_ci/run-intel-ci-test.sh w8a8-fp8-linear' 'bash .buildkite/scripts/hardware_ci/run-intel-ci-test.sh w8a8-fp8-linear'
- label: ":intel: (BMG) Server test" - label: "XPU server test"
key: xpu-server-test key: xpu-server-test
depends_on: depends_on:
- image-build-xpu - image-build-xpu
@@ -105,7 +105,7 @@ steps:
- >- - >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'bash .buildkite/scripts/hardware_ci/run-intel-ci-test.sh server' 'bash .buildkite/scripts/hardware_ci/run-intel-ci-test.sh server'
- label: ":intel: (BMG) Quantization test" - label: "XPU quantization test"
key: xpu-quantization-test key: xpu-quantization-test
depends_on: depends_on:
- image-build-xpu - image-build-xpu
@@ -127,7 +127,7 @@ steps:
- >- - >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'bash .buildkite/scripts/hardware_ci/run-intel-ci-test.sh quantization' 'bash .buildkite/scripts/hardware_ci/run-intel-ci-test.sh quantization'
- label: ":intel: (BMG) Compressed tensors FP8 test" - label: "XPU compressed tensors FP8 test"
key: xpu-compressed-tensors-fp8-test key: xpu-compressed-tensors-fp8-test
depends_on: depends_on:
- image-build-xpu - image-build-xpu
@@ -150,7 +150,7 @@ steps:
- >- - >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'bash .buildkite/scripts/hardware_ci/run-intel-ci-test.sh compressed-tensors-fp8' 'bash .buildkite/scripts/hardware_ci/run-intel-ci-test.sh compressed-tensors-fp8'
- label: ":intel: (BMG) Graph" - label: "XPU Graph"
key: xpu-graph key: xpu-graph
depends_on: depends_on:
- image-build-xpu - image-build-xpu
@@ -61,8 +61,6 @@ NIGHTLY_UBUNTU_VERSION="24.04"
NIGHTLY_FINAL_BASE_IMAGE="nvidia/cuda:${NIGHTLY_CUDA_VERSION}-base-ubuntu${NIGHTLY_UBUNTU_VERSION}" NIGHTLY_FINAL_BASE_IMAGE="nvidia/cuda:${NIGHTLY_CUDA_VERSION}-base-ubuntu${NIGHTLY_UBUNTU_VERSION}"
echo "--- :docker: Building torch nightly image (CUDA ${NIGHTLY_CUDA_VERSION})" echo "--- :docker: Building torch nightly image (CUDA ${NIGHTLY_CUDA_VERSION})"
# Provenance labels match the bake `_labels` target in docker/docker-bake.hcl;
# the initialized-snapshot E2E refuses images whose commit labels differ.
docker buildx build --file docker/Dockerfile \ docker buildx build --file docker/Dockerfile \
--build-arg max_jobs=16 \ --build-arg max_jobs=16 \
--build-arg buildkite_commit="$BUILDKITE_COMMIT" \ --build-arg buildkite_commit="$BUILDKITE_COMMIT" \
@@ -72,10 +70,6 @@ docker buildx build --file docker/Dockerfile \
--build-arg BUILD_BASE_IMAGE="${NIGHTLY_BUILD_BASE_IMAGE}" \ --build-arg BUILD_BASE_IMAGE="${NIGHTLY_BUILD_BASE_IMAGE}" \
--build-arg FINAL_BASE_IMAGE="${NIGHTLY_FINAL_BASE_IMAGE}" \ --build-arg FINAL_BASE_IMAGE="${NIGHTLY_FINAL_BASE_IMAGE}" \
--build-arg torch_cuda_arch_list="8.0 8.9 9.0 10.0 12.0" \ --build-arg torch_cuda_arch_list="8.0 8.9 9.0 10.0 12.0" \
--label "org.opencontainers.image.revision=${BUILDKITE_COMMIT}" \
--label "ai.vllm.build.commit=${BUILDKITE_COMMIT}" \
--label "ai.vllm.build.pipeline=${BUILDKITE_PIPELINE_SLUG:-local}" \
--label "ai.vllm.build.url=${BUILDKITE_BUILD_URL:-}" \
--tag "$IMAGE_TAG" \ --tag "$IMAGE_TAG" \
--push \ --push \
--target test \ --target test \
@@ -1,75 +0,0 @@
#!/bin/bash
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
#
# Build the AMD Zen CPU image (vLLM + zentorch) by layering docker/Dockerfile.zen
# on top of a CPU base image that already has vLLM installed.
#
# To avoid recompiling vLLM from source, prefer reusing the shared CPU image that
# the image-build-cpu step already builds and publishes for this commit
# (`<repo>:<commit>-cpu`). It lives in public ECR, which allows anonymous pulls,
# so no registry credentials are required. If the pull fails (e.g. the image
# isn't published yet), fall back to building the base from docker/Dockerfile.cpu.
#
# The zen image is not pushed to a registry.
#
# See docker/Dockerfile.zen for the build workflow this mirrors.
set -euo pipefail
if [[ $# -lt 3 ]]; then
echo "Usage: $0 <registry> <repo> <commit>"
exit 1
fi
REGISTRY=$1
REPO=$2
BUILDKITE_COMMIT=$3
image_repo() {
if [[ -n "${REGISTRY}" ]]; then
printf '%s/%s' "${REGISTRY}" "${REPO}"
else
printf '%s' "${REPO}"
fi
}
IMAGE_REPO="$(image_repo)"
# The shared CPU image published by image-build-cpu (public ECR, anonymous pull).
SHARED_CPU_IMAGE=""
if [[ -n "${REGISTRY}" ]]; then
SHARED_CPU_IMAGE="$REGISTRY/$REPO:$BUILDKITE_COMMIT-cpu"
fi
FALLBACK_BASE_IMAGE="zen-cpu-base:$BUILDKITE_COMMIT"
IMAGE="$IMAGE_REPO:$BUILDKITE_COMMIT-zen-cpu"
# ZENTORCH_VERSION is optional; when unset the Dockerfile falls back to
# installing zentorch via `vllm[zen]`.
ZENTORCH_VERSION=${ZENTORCH_VERSION:-}
# Step 1: obtain the CPU base image that Dockerfile.zen layers on. Prefer pulling
# the published `-cpu` image; fall back to building it from source if unavailable.
if [[ -n "${SHARED_CPU_IMAGE}" ]] && docker pull "$SHARED_CPU_IMAGE"; then
echo "--- :docker: Using published CPU image as base: $SHARED_CPU_IMAGE"
BASE_IMAGE="$SHARED_CPU_IMAGE"
else
echo "--- :docker: Published CPU image unavailable; building base from source"
docker build --file docker/Dockerfile.cpu \
--platform linux/amd64 \
--build-arg buildkite_commit="$BUILDKITE_COMMIT" \
--build-arg VLLM_CPU_X86=true \
--tag "$FALLBACK_BASE_IMAGE" \
--target vllm-openai \
--progress plain .
BASE_IMAGE="$FALLBACK_BASE_IMAGE"
fi
# Step 2: build the zen test image on top of the CPU base.
echo "--- :docker: Building Zen test image"
docker build --file docker/Dockerfile.zen \
--platform linux/amd64 \
--build-arg BASE_IMAGE="$BASE_IMAGE" \
${ZENTORCH_VERSION:+--build-arg ZENTORCH_VERSION="$ZENTORCH_VERSION"} \
--tag "$IMAGE" \
--target vllm-zen-test \
--progress plain .
@@ -1,8 +1,8 @@
group: Basic Correctness Intel group: Basic Correctness
depends_on: depends_on:
- image-build-xpu - image-build-xpu
steps: steps:
- label: ":intel: (BMG) Sleep Mode" - label: XPU Sleep Mode
key: xpu-sleep-mode key: xpu-sleep-mode
timeout_in_minutes: 45 timeout_in_minutes: 45
device: intel_gpu device: intel_gpu
+2 -2
View File
@@ -1,8 +1,8 @@
group: Benchmarks Intel group: Benchmarks
depends_on: depends_on:
- image-build-xpu - image-build-xpu
steps: steps:
- label: ":intel: (BMG) Benchmarks CLI" - label: Benchmarks CLI Test
key: benchmarks-cli-test key: benchmarks-cli-test
timeout_in_minutes: 40 timeout_in_minutes: 40
device: intel_gpu device: intel_gpu
@@ -1,33 +0,0 @@
group: Distributed Intel
depends_on:
- image-build-xpu
steps:
- label: ":intel: (BMG) Distributed Torchrun + Shutdown"
key: distributed-torchrun-shutdown-tests-2-gpus
timeout_in_minutes: 50
device: intel_gpu
agent_tags:
label: production
gpu: 2+
mem: 24+
no_plugin: true
working_dir: "."
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
VLLM_TEST_DEVICE: "xpu"
source_file_dependencies:
- vllm/distributed/
- "!vllm/distributed/kv_transfer/"
- vllm/engine/
- vllm/executor/
- vllm/worker/worker_base.py
- vllm/v1/engine/
- vllm/v1/worker/
- tests/distributed/
- tests/v1/shutdown
- tests/v1/worker/test_worker_memory_snapshot.py
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests && pytest -v -s v1/shutdown'
+3 -3
View File
@@ -2,7 +2,7 @@ group: Engine Intel
depends_on: depends_on:
- image-build-xpu - image-build-xpu
steps: steps:
- label: ":intel: (BMG) Engine" - label: Engine
key: engine key: engine
timeout_in_minutes: 40 timeout_in_minutes: 40
device: intel_gpu device: intel_gpu
@@ -40,7 +40,7 @@ steps:
'cd tests && 'cd tests &&
pytest -v -s engine/test_arg_utils.py test_sequence.py test_logger.py test_vllm_port.py jit_monitor/test_hooks.py' pytest -v -s engine/test_arg_utils.py test_sequence.py test_logger.py test_vllm_port.py jit_monitor/test_hooks.py'
- label: ":intel: (BMG) Engine (1 GPU)" - label: Engine (1 GPU)
key: engine-1-gpu key: engine-1-gpu
timeout_in_minutes: 30 timeout_in_minutes: 30
device: intel_gpu device: intel_gpu
@@ -65,7 +65,7 @@ steps:
pytest -v -s v1/engine --ignore v1/engine/test_preprocess_error_handling.py --ignore v1/engine/test_core_engine_actor_manager.py && pytest -v -s v1/engine --ignore v1/engine/test_preprocess_error_handling.py --ignore v1/engine/test_core_engine_actor_manager.py &&
pytest -v -s test_config.py' pytest -v -s test_config.py'
- label: ":intel: (BMG) V1 e2e (2 GPUs)" - label: V1 e2e (2 GPUs)
key: v1-e2e-2-gpus key: v1-e2e-2-gpus
timeout_in_minutes: 30 timeout_in_minutes: 30
device: intel_gpu device: intel_gpu
+6 -9
View File
@@ -1,4 +1,4 @@
group: Entrypoints Intel group: Entrypoints
depends_on: depends_on:
- image-build-xpu - image-build-xpu
steps: steps:
@@ -19,14 +19,14 @@ steps:
source_file_dependencies: source_file_dependencies:
- vllm/entrypoints - vllm/entrypoints
- tests/entrypoints/unit_tests - tests/entrypoints/unit_tests
- tests/entrypoints/rl/weight_transfer - tests/entrypoints/weight_transfer
- tests/entrypoints/launchers - tests/entrypoints/launchers
commands: commands:
- >- - >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests && 'cd tests &&
pytest -v -s entrypoints/unit_tests && pytest -v -s entrypoints/unit_tests &&
pytest -v -s entrypoints/rl/weight_transfer' pytest -v -s entrypoints/weight_transfer'
- label: ":intel: (BMG) Entrypoints Integration (LLM)" - label: ":intel: (BMG) Entrypoints Integration (LLM)"
key: entrypoints-integration-llm-intel key: entrypoints-integration-llm-intel
@@ -71,17 +71,14 @@ steps:
- vllm/ - vllm/
- "!vllm/distributed/kv_transfer/" - "!vllm/distributed/kv_transfer/"
- tests/entrypoints/serve - tests/entrypoints/serve
- tests/entrypoints/rl
commands: commands:
- >- - >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'export VLLM_WORKER_MULTIPROC_METHOD=spawn && 'export VLLM_WORKER_MULTIPROC_METHOD=spawn &&
cd tests && cd tests &&
pytest -v -s entrypoints/serve --ignore=entrypoints/serve/dev/rpc && pytest -v -s entrypoints/serve --ignore=entrypoints/serve/dev/rpc
pytest -v -s entrypoints/rl --ignore=entrypoints/rl/weight_transfer --ignore=entrypoints/serve/dev/rlhf/test_weight_operation_metrics.py
--ignore=entrypoints/rl/return_routed_experts/test_openai_api.py --deselect="tests/entrypoints/serve/dev/rlhf/state_transitions/test_pause_resume.py::TestPauseResume::test_clear_cache_preserves_output_and_controls_prefix_cache[MRV1]" &&
--ignore=entrypoints/rl/weight_transfer_metrics
--deselect="tests/entrypoints/rl/state_transitions/test_pause_resume.py::TestPauseResume::test_clear_cache_preserves_output_and_controls_prefix_cache[MRV1]" &&
PYTHONPATH=/workspace/vllm pytest -v -s entrypoints/serve/dev/rpc' PYTHONPATH=/workspace/vllm pytest -v -s entrypoints/serve/dev/rpc'
- label: ":intel: (BMG) Entrypoints Integration (API Server Scale Out)" - label: ":intel: (BMG) Entrypoints Integration (API Server Scale Out)"
@@ -1,8 +1,8 @@
group: Expert Parallelism Intel group: Expert Parallelism
depends_on: depends_on:
- image-build-xpu - image-build-xpu
steps: steps:
- label: ":intel: (BMG) EPLB Algorithm" - label: EPLB Algorithm
key: eplb-algorithm key: eplb-algorithm
timeout_in_minutes: 45 timeout_in_minutes: 45
device: intel_gpu device: intel_gpu
+2 -2
View File
@@ -2,7 +2,7 @@ group: Kernels Intel
depends_on: depends_on:
- image-build-xpu - image-build-xpu
steps: steps:
- label: ":intel: (BMG) vLLM IR Tests" - label: vLLM IR Tests
key: vllm-ir-tests key: vllm-ir-tests
timeout_in_minutes: 30 timeout_in_minutes: 30
device: intel_gpu device: intel_gpu
@@ -26,7 +26,7 @@ steps:
pytest -v -s ir && pytest -v -s ir &&
pytest -v -s kernels/ir' pytest -v -s kernels/ir'
- label: ":intel: (BMG) UVA Kernel Tests" - label: UVA Kernel Tests
key: uva-kernel-tests key: uva-kernel-tests
timeout_in_minutes: 20 timeout_in_minutes: 20
device: intel_gpu device: intel_gpu
+3 -3
View File
@@ -1,8 +1,8 @@
group: LM Eval Intel group: LM Eval
depends_on: depends_on:
- image-build-xpu - image-build-xpu
steps: steps:
- label: ":intel: (BMG) GPQA Eval (GPT-OSS)" - label: "XPU GPQA Eval (GPT-OSS)"
key: xpu-gpqa-eval-gpt-oss key: xpu-gpqa-eval-gpt-oss
timeout_in_minutes: 60 timeout_in_minutes: 60
device: intel_gpu device: intel_gpu
@@ -25,7 +25,7 @@ steps:
bash .buildkite/scripts/hardware_ci/run-intel-test.sh bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests && 'cd tests &&
pytest -s -v evals/gpt_oss/test_gpqa_correctness.py --config-list-file=configs/models-xpu.txt' pytest -s -v evals/gpt_oss/test_gpqa_correctness.py --config-list-file=configs/models-xpu.txt'
- label: ":intel: (BMG) LM Eval Large Models(2 GPUs)" - label: LM Eval Large Models(2 GPUs)
key: lm-eval-large-models-2-gpus key: lm-eval-large-models-2-gpus
timeout_in_minutes: 45 timeout_in_minutes: 45
device: intel_gpu device: intel_gpu
+36 -59
View File
@@ -2,14 +2,14 @@ group: LoRA Intel
depends_on: depends_on:
- image-build-xpu - image-build-xpu
steps: steps:
- label: ":intel: (BMG) LoRA Runtime + Utils" - label: LoRA Runtime + Utils
key: lora-runtime-utils key: lora-runtime-utils
timeout_in_minutes: 45 timeout_in_minutes: 45
device: intel_gpu device: intel_gpu
agent_tags: agent_tags:
label: production label: production
gpu: 1+ gpu: 1+
mem: 16+ mem: 24+
no_plugin: true no_plugin: true
working_dir: "." working_dir: "."
env: env:
@@ -33,9 +33,10 @@ steps:
pytest -v -s lora/test_peft_helper.py && pytest -v -s lora/test_peft_helper.py &&
pytest -v -s lora/test_resolver.py && pytest -v -s lora/test_resolver.py &&
pytest -v -s lora/test_utils.py && pytest -v -s lora/test_utils.py &&
pytest -v -s lora/test_add_lora.py &&
pytest -v -s lora/test_worker.py' pytest -v -s lora/test_worker.py'
- label: ":intel: (BMG) LoRA Fused/MoE Kernels" - label: LoRA Fused/MoE Kernels
key: lora-fused-moe-kernels key: lora-fused-moe-kernels
timeout_in_minutes: 45 timeout_in_minutes: 45
device: intel_gpu device: intel_gpu
@@ -60,7 +61,7 @@ steps:
pytest -v -s lora/test_fused_moe_lora_kernel.py && pytest -v -s lora/test_fused_moe_lora_kernel.py &&
pytest -v -s lora/test_moe_lora_align_sum.py --deselect="tests/lora/test_moe_lora_align_sum.py::test_moe_lora_align_block_size_mixed_base_and_lora[1]"' pytest -v -s lora/test_moe_lora_align_sum.py --deselect="tests/lora/test_moe_lora_align_sum.py::test_moe_lora_align_block_size_mixed_base_and_lora[1]"'
- label: ":intel: (BMG) LoRA Punica Kernels" - label: LoRA Punica Kernels
key: lora-punica-kernels key: lora-punica-kernels
timeout_in_minutes: 45 timeout_in_minutes: 45
device: intel_gpu device: intel_gpu
@@ -87,7 +88,7 @@ steps:
pytest -v -s lora/test_punica_ops.py::test_kernels_hidden_size && pytest -v -s lora/test_punica_ops.py::test_kernels_hidden_size &&
pytest -v -s lora/test_punica_ops.py::test_add_lora_fused_moe_early_exit' pytest -v -s lora/test_punica_ops.py::test_add_lora_fused_moe_early_exit'
- label: ":intel: (BMG) LoRA Punica FP8/XPU Ops" - label: LoRA Punica FP8/XPU Ops
key: lora-punica-fp8-xpu-ops key: lora-punica-fp8-xpu-ops
timeout_in_minutes: 60 timeout_in_minutes: 60
device: intel_gpu device: intel_gpu
@@ -112,61 +113,10 @@ steps:
pytest -v -s lora/test_punica_ops_fp8.py && pytest -v -s lora/test_punica_ops_fp8.py &&
pytest -v -s lora/test_punica_xpu_ops.py' pytest -v -s lora/test_punica_xpu_ops.py'
- label: ":intel: (BMG) LoRA Models" - label: LoRA Models
key: lora-models key: lora-models
timeout_in_minutes: 45 timeout_in_minutes: 45
device: intel_gpu device: intel_gpu
agent_tags:
label: production
gpu: 2+
mem: 16+
no_plugin: true
working_dir: "."
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
VLLM_TEST_DEVICE: "xpu"
source_file_dependencies:
- vllm/lora
- tests/lora
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests &&
export VLLM_WORKER_MULTIPROC_METHOD=spawn &&
pytest -v -s lora/test_quant_model.py --deselect="tests/lora/test_quant_model.py::test_quant_model_lora[model0]" --deselect="tests/lora/test_quant_model.py::test_quant_model_lora[model1]" --deselect="tests/lora/test_quant_model.py::test_quant_model_tp_equality[model0]" &&
pytest -v -s lora/test_transformers_model.py &&
pytest -v -s lora/test_chatglm3_tp.py &&
pytest -v -s lora/test_llama_tp.py::test_llama_lora'
- label: ":intel: (BMG) LoRA Multimodal"
key: lora-multimodal
timeout_in_minutes: 45
device: intel_gpu
agent_tags:
label: production
gpu: 1+
mem: 16+
no_plugin: true
working_dir: "."
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
VLLM_TEST_DEVICE: "xpu"
source_file_dependencies:
- vllm/lora
- tests/lora
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests &&
export VLLM_WORKER_MULTIPROC_METHOD=spawn &&
pytest -v -s lora/test_whisper.py'
- label: ":intel: (BMG) LoRA Add + Multimodal"
key: lora-add-multimodal
timeout_in_minutes: 45
device: intel_gpu
agent_tags: agent_tags:
label: production label: production
gpu: 2+ gpu: 2+
@@ -185,6 +135,33 @@ steps:
bash .buildkite/scripts/hardware_ci/run-intel-test.sh bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests && 'cd tests &&
export VLLM_WORKER_MULTIPROC_METHOD=spawn && export VLLM_WORKER_MULTIPROC_METHOD=spawn &&
pytest -v -s lora/test_add_lora.py && pytest -v -s lora/test_quant_model.py --deselect="tests/lora/test_quant_model.py::test_quant_model_lora[model0]" --deselect="tests/lora/test_quant_model.py::test_quant_model_lora[model1]" --deselect="tests/lora/test_quant_model.py::test_quant_model_tp_equality[model0]" &&
pytest -v -s lora/test_default_mm_loras.py && pytest -v -s lora/test_transformers_model.py &&
pytest -v -s lora/test_chatglm3_tp.py &&
pytest -v -s lora/test_llama_tp.py::test_llama_lora &&
pytest -s -v lora/test_minicpmv_tp.py' pytest -s -v lora/test_minicpmv_tp.py'
- label: LoRA Multimodal
key: lora-multimodal
timeout_in_minutes: 45
device: intel_gpu
agent_tags:
label: production
gpu: 1+
mem: 24+
no_plugin: true
working_dir: "."
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
VLLM_TEST_DEVICE: "xpu"
source_file_dependencies:
- vllm/lora
- tests/lora
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests &&
export VLLM_WORKER_MULTIPROC_METHOD=spawn &&
pytest -v -s lora/test_default_mm_loras.py &&
pytest -v -s lora/test_whisper.py'
+20 -90
View File
@@ -2,7 +2,7 @@ group: Miscellaneous Intel
depends_on: depends_on:
- image-build-xpu - image-build-xpu
steps: steps:
- label: ":intel: (BMG) Batch Invariance" - label: XPU Batch Invariance
key: xpu-batch-invariance key: xpu-batch-invariance
timeout_in_minutes: 60 timeout_in_minutes: 60
num_devices: 2 num_devices: 2
@@ -10,7 +10,7 @@ steps:
agent_tags: agent_tags:
label: production label: production
gpu: 2+ gpu: 2+
mem: 16+ mem: 24+
no_plugin: true no_plugin: true
working_dir: "." working_dir: "."
env: env:
@@ -30,44 +30,10 @@ steps:
pytest -v -s v1/determinism/test_xpu_batch_invariant_ut.py && pytest -v -s v1/determinism/test_xpu_batch_invariant_ut.py &&
bash ../.buildkite/scripts/hardware_ci/run-xpu-batch-invariance.sh' bash ../.buildkite/scripts/hardware_ci/run-xpu-batch-invariance.sh'
- label: ":intel: (BMG) V1 Core + KV + Metrics" - label: V1 Core + KV + Metrics
key: v1-core-kv-metrics-intel key: v1-core-kv-metrics-intel
timeout_in_minutes: 45 timeout_in_minutes: 45
device: intel_gpu device: intel_gpu
agent_tags:
label: production
gpu: 1+
mem: 16+
no_plugin: true
working_dir: "."
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
VLLM_TEST_DEVICE: "xpu"
source_file_dependencies:
- vllm/
- tests/v1/core
- tests/v1/executor
- tests/v1/kv_offload
- tests/v1/worker
- tests/v1/kv_connector/unit
- tests/v1/ec_connector/unit
- tests/v1/metrics
- tests/entrypoints/openai/correctness/test_lmeval.py
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'pip install -r requirements/kv_connectors.txt &&
export VLLM_WORKER_MULTIPROC_METHOD=spawn &&
cd tests &&
pytest -v -s v1/executor &&
VLLM_BATCH_INVARIANT=1 pytest -v -s -m "not cpu_test" v1/ec_connector/unit
--ignore=v1/ec_connector/unit/test_ec_cpu_connector.py'
- label: ":intel: (BMG) EC CPU Connector (XPU)"
key: ec-cpu-connector-xpu
timeout_in_minutes: 45
device: intel_gpu
agent_tags: agent_tags:
label: production label: production
gpu: 1+ gpu: 1+
@@ -94,16 +60,17 @@ steps:
'pip install -r requirements/kv_connectors.txt && 'pip install -r requirements/kv_connectors.txt &&
export VLLM_WORKER_MULTIPROC_METHOD=spawn && export VLLM_WORKER_MULTIPROC_METHOD=spawn &&
cd tests && cd tests &&
pytest -v -s v1/ec_connector/unit/test_ec_cpu_connector.py' pytest -v -s v1/executor &&
VLLM_BATCH_INVARIANT=1 pytest -v -s -m "not cpu_test" v1/ec_connector/unit'
- label: ":intel: (BMG) V1 Sample + Logits" - label: V1 Sample + Logits
key: v1-sample-logits-intel key: v1-sample-logits-intel
timeout_in_minutes: 90 timeout_in_minutes: 90
device: intel_gpu device: intel_gpu
agent_tags: agent_tags:
label: production label: production
gpu: 1+ gpu: 1+
mem: 16+ mem: 24+
no_plugin: true no_plugin: true
working_dir: "." working_dir: "."
env: env:
@@ -141,7 +108,7 @@ steps:
limit: 1 limit: 1
parallelism: 4 parallelism: 4
- label: ":intel: (BMG) Basic Models Tests (Initialization)" - label: Basic Models Tests (Initialization)
key: basic-models-tests-initialization key: basic-models-tests-initialization
timeout_in_minutes: 60 timeout_in_minutes: 60
device: intel_gpu device: intel_gpu
@@ -166,7 +133,7 @@ steps:
cd tests && cd tests &&
pytest -v -s models/test_initialization.py::test_can_initialize_large_subset[Eagle3MiniMaxM2ForCausalLM]' pytest -v -s models/test_initialization.py::test_can_initialize_large_subset[Eagle3MiniMaxM2ForCausalLM]'
- label: ":intel: (BMG) CPU Offload" - label: XPU CPU Offload
key: xpu-cpu-offload key: xpu-cpu-offload
timeout_in_minutes: 60 timeout_in_minutes: 60
device: intel_gpu device: intel_gpu
@@ -194,7 +161,7 @@ steps:
pytest -v -s v1/kv_offload && pytest -v -s v1/kv_offload &&
pytest -v -s v1/kv_connector/unit/test_offloading_connector.py' pytest -v -s v1/kv_connector/unit/test_offloading_connector.py'
- label: ":intel: (BMG) NixlConnector PD accuracy (4 GPUs)" - label: NixlConnector PD accuracy (4 GPUs)
key: nixlconnector-pd-accuracy-4-gpus key: nixlconnector-pd-accuracy-4-gpus
timeout_in_minutes: 60 timeout_in_minutes: 60
num_devices: 4 num_devices: 4
@@ -223,7 +190,7 @@ steps:
PREFILLER_TP_SIZE=1 DECODER_TP_SIZE=2 bash v1/kv_connector/nixl_integration/run_xpu_disagg_accuracy_test.sh && PREFILLER_TP_SIZE=1 DECODER_TP_SIZE=2 bash v1/kv_connector/nixl_integration/run_xpu_disagg_accuracy_test.sh &&
PREFILLER_TP_SIZE=2 DECODER_TP_SIZE=2 bash v1/kv_connector/nixl_integration/run_xpu_disagg_accuracy_test.sh' PREFILLER_TP_SIZE=2 DECODER_TP_SIZE=2 bash v1/kv_connector/nixl_integration/run_xpu_disagg_accuracy_test.sh'
- label: ":intel: (BMG) Regression" - label: Regression
key: regression key: regression
timeout_in_minutes: 50 timeout_in_minutes: 50
device: intel_gpu device: intel_gpu
@@ -256,7 +223,7 @@ steps:
'cd tests && 'cd tests &&
pytest -v -s test_regression.py' pytest -v -s test_regression.py'
- label: ":intel: (BMG) Metrics, Tracing (2 GPUs)" - label: Metrics, Tracing (2 GPUs)
key: metrics-tracing-2-gpus key: metrics-tracing-2-gpus
timeout_in_minutes: 45 timeout_in_minutes: 45
num_devices: 2 num_devices: 2
@@ -291,49 +258,10 @@ steps:
'cd tests && 'cd tests &&
pytest -v -s v1/tracing' pytest -v -s v1/tracing'
- label: ":intel: (BMG) Async Engine, Inputs, Utils, Worker" - label: Async Engine, Inputs, Utils, Worker
key: async-engine-inputs-utils-worker key: async-engine-inputs-utils-worker
timeout_in_minutes: 55 timeout_in_minutes: 55
device: intel_gpu device: intel_gpu
agent_tags:
label: production
gpu: 1+
mem: 16+
no_plugin: true
working_dir: "."
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
VLLM_TEST_DEVICE: "xpu"
source_file_dependencies:
- vllm/assets/
- vllm/config/
- vllm/distributed/
- vllm/engine/
- vllm/inputs/
- vllm/model_executor/
- vllm/multimodal/
- vllm/platforms/
- vllm/sampling_params.py
- vllm/tokenizers/
- vllm/transformers_utils/
- vllm/utils/
- vllm/v1/
- tests/detokenizer
- tests/multimodal
- tests/utils_
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests &&
pytest -v -s detokenizer --ignore=detokenizer/test_stop_strings.py &&
pytest -v -s -m "not cpu_test" ./multimodal &&
pytest -v -s utils_ --ignore=utils_/test_mem_utils.py'
- label: ":intel: (BMG) Detokenizer Stop Strings (XPU)"
key: detokenizer-stop-strings-xpu
timeout_in_minutes: 45
device: intel_gpu
agent_tags: agent_tags:
label: production label: production
gpu: 1+ gpu: 1+
@@ -362,12 +290,14 @@ steps:
- tests/multimodal - tests/multimodal
- tests/utils_ - tests/utils_
commands: commands:
- >- - >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests && 'cd tests &&
pytest -v -s detokenizer/test_stop_strings.py' pytest -v -s detokenizer &&
pytest -v -s -m "not cpu_test" ./multimodal &&
pytest -v -s utils_ --ignore=utils_/test_mem_utils.py'
- label: ":intel: (BMG) Fusion Unit Tests" - label: Fusion Unit Tests
key: fusion-unit-tests key: fusion-unit-tests
timeout_in_minutes: 30 timeout_in_minutes: 30
device: intel_gpu device: intel_gpu
+11 -30
View File
@@ -2,38 +2,10 @@ group: Model Executor Intel
depends_on: depends_on:
- image-build-xpu - image-build-xpu
steps: steps:
- label: ":intel: (BMG) Model Executor" - label: Model Executor (Intel)
key: model-executor-intel key: model-executor-intel
timeout_in_minutes: 45 timeout_in_minutes: 45
device: intel_gpu device: intel_gpu
agent_tags:
label: production
gpu: 1+
mem: 16+
no_plugin: true
working_dir: "."
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
VLLM_TEST_DEVICE: "xpu"
source_file_dependencies:
- vllm/engine/arg_utils.py
- vllm/config/model.py
- vllm/model_executor
- tests/model_executor
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'apt-get update && apt-get install -y curl libsodium23 &&
export VLLM_WORKER_MULTIPROC_METHOD=spawn &&
export PYTHONFAULTHANDLER=1 &&
cd tests &&
pytest -v -s model_executor -m "not slow_test" --ignore="model_executor/layers/test_rocm_unquantized_gemm.py" --ignore="model_executor/model_loader/tensorizer_loader/test_tensorizer.py" --ignore="model_executor/model_loader/test_reload.py" --deselect="tests/model_executor/test_enabled_custom_ops.py::test_topk_gating_enabled_by_cdna"'
- label: ":intel: (BMG) Tensorizer Model Loader Tests (XPU)"
key: tensorizer-model-loader-tests-xpu
timeout_in_minutes: 45
device: intel_gpu
agent_tags: agent_tags:
label: production label: production
gpu: 1+ gpu: 1+
@@ -56,4 +28,13 @@ steps:
export VLLM_WORKER_MULTIPROC_METHOD=spawn && export VLLM_WORKER_MULTIPROC_METHOD=spawn &&
export PYTHONFAULTHANDLER=1 && export PYTHONFAULTHANDLER=1 &&
cd tests && cd tests &&
pytest -v -s model_executor/model_loader/tensorizer_loader/test_tensorizer.py -m "not slow_test"' pytest -v -s model_executor -m "not slow_test" --ignore="model_executor/layers/test_rocm_unquantized_gemm.py" --deselect="tests/model_executor/model_loader/test_reload.py::test_kv_scale_reload"
--deselect="tests/model_executor/layers/test_mm_input_norm.py::TestFusedMMInputNormModule::test_matches_reference_above_cudnn_grid_limit"
--deselect="tests/model_executor/layers/test_mm_input_norm.py::TestFusedMMInputNormDtypes::test_dtype_combinations[in_dtype0-out_dtype0]"
--deselect="tests/model_executor/layers/test_mm_input_norm.py::TestFusedMMInputNormDtypes::test_dtype_combinations[in_dtype1-out_dtype1]"
--deselect="tests/model_executor/layers/test_mm_input_norm.py::TestFusedMMInputNormDtypes::test_dtype_combinations[in_dtype2-out_dtype2]"
--deselect="tests/model_executor/layers/test_mm_input_norm.py::TestFusedMMInputNormDtypes::test_dtype_combinations[in_dtype3-out_dtype3]"
--deselect="tests/model_executor/layers/test_mm_input_norm.py::TestFusedMMInputNormDtypes::test_dtype_combinations[in_dtype4-out_dtype4]"
--deselect="tests/model_executor/layers/test_mm_input_norm.py::TestFusedMMInputNormDtypes::test_dtype_combinations[in_dtype5-out_dtype5]"
--deselect="tests/model_executor/layers/test_mm_input_norm.py::TestFusedMMInputNormShapes"
--deselect="tests/model_executor/layers/test_mm_input_norm.py::TestFusedMMInputNormInputHandling"'
@@ -1,8 +1,8 @@
group: Models - Distributed Intel group: Models - Distributed
depends_on: depends_on:
- image-build-xpu - image-build-xpu
steps: steps:
- label: ":intel: (BMG) Distributed Model Tests" - label: Distributed Model Tests (2 GPUs)
key: distributed-model-tests-2-gpus key: distributed-model-tests-2-gpus
timeout_in_minutes: 65 timeout_in_minutes: 65
device: intel_gpu device: intel_gpu
@@ -1,15 +1,15 @@
group: Models - Multimodal Intel group: Models - Multimodal
depends_on: depends_on:
- image-build-xpu - image-build-xpu
steps: steps:
- label: ":intel: (BMG) Multi-Modal Models (Standard) 1: qwen2" - label: "Multi-Modal Models (Standard) 1: qwen2"
key: multi-modal-models-standard-1-qwen2 key: multi-modal-models-standard-1-qwen2
timeout_in_minutes: 70 timeout_in_minutes: 70
device: intel_gpu device: intel_gpu
agent_tags: agent_tags:
label: production label: production
gpu: 1+ gpu: 1+
mem: 16+ mem: 24+
no_plugin: true no_plugin: true
working_dir: "." working_dir: "."
env: env:
@@ -26,14 +26,14 @@ steps:
pytest -v -s models/multimodal/generation/test_common.py -m core_model -k "qwen2" && pytest -v -s models/multimodal/generation/test_common.py -m core_model -k "qwen2" &&
pytest -v -s models/multimodal/generation/test_ultravox.py -m core_model' pytest -v -s models/multimodal/generation/test_ultravox.py -m core_model'
- label: ":intel: (BMG) Multi-Modal Models (Standard) 2: qwen3 + gemma" - label: "Multi-Modal Models (Standard) 2: qwen3 + gemma"
key: multi-modal-models-standard-2-qwen3-gemma key: multi-modal-models-standard-2-qwen3-gemma
timeout_in_minutes: 90 timeout_in_minutes: 90
device: intel_gpu device: intel_gpu
agent_tags: agent_tags:
label: production label: production
gpu: 1+ gpu: 1+
mem: 16+ mem: 24+
no_plugin: true no_plugin: true
working_dir: "." working_dir: "."
env: env:
@@ -51,7 +51,7 @@ steps:
pytest -v -s models/multimodal/generation/test_mm_prefix_lm.py -m core_model && pytest -v -s models/multimodal/generation/test_mm_prefix_lm.py -m core_model &&
pytest -v -s models/multimodal/generation/test_qwen2_5_vl.py -m core_model' pytest -v -s models/multimodal/generation/test_qwen2_5_vl.py -m core_model'
- label: ":intel: (BMG) Multi-Modal Models (Standard) 3: llava + qwen2_vl" - label: "Multi-Modal Models (Standard) 3: llava + qwen2_vl"
key: multi-modal-models-standard-3-llava-qwen2-vl key: multi-modal-models-standard-3-llava-qwen2-vl
timeout_in_minutes: 65 timeout_in_minutes: 65
device: intel_gpu device: intel_gpu
@@ -72,18 +72,16 @@ steps:
- >- - >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests && 'cd tests &&
pytest -v -s models/multimodal/generation/test_common.py -m core_model -k "not qwen2 and not qwen3 and not gemma" && pytest -v -s models/multimodal/generation/test_common.py -m core_model -k "not qwen2 and not qwen3 and not gemma"'
pytest -v -s models/multimodal/generation/test_gemma_ragged_audio.py -m core_model
pytest -v -s models/multimodal/generation/test_qwen2_vl.py -m core_model'
- label: ":intel: (BMG) Multi-Modal Models (Standard) 4: other + whisper" - label: "Multi-Modal Models (Standard) 4: other + whisper"
key: multi-modal-models-standard-4-other-whisper key: multi-modal-models-standard-4-other-whisper
timeout_in_minutes: 45 timeout_in_minutes: 45
device: intel_gpu device: intel_gpu
agent_tags: agent_tags:
label: production label: production
gpu: 1+ gpu: 1+
mem: 16+ mem: 24+
no_plugin: true no_plugin: true
working_dir: "." working_dir: "."
env: env:
@@ -97,9 +95,9 @@ steps:
- >- - >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests && 'cd tests &&
pytest -v -s models/multimodal -m core_model --ignore models/multimodal/generation/test_common.py --ignore models/multimodal/generation/test_ultravox.py --ignore models/multimodal/generation/test_qwen2_5_vl.py --ignore models/multimodal/generation/test_qwen2_vl.py --ignore models/multimodal/generation/test_whisper.py --ignore models/multimodal/generation/test_gemma_ragged_audio.py --ignore models/multimodal/generation/test_memory_leak.py --ignore models/multimodal/processing' pytest -v -s models/multimodal -m core_model --ignore models/multimodal/generation/test_common.py --ignore models/multimodal/generation/test_ultravox.py --ignore models/multimodal/generation/test_qwen2_5_vl.py --ignore models/multimodal/generation/test_qwen2_vl.py --ignore models/multimodal/generation/test_whisper.py --ignore models/multimodal/generation/test_memory_leak.py --ignore models/multimodal/processing'
- label: ":intel: (BMG) Multi-Modal Processor" # 44min - label: Multi-Modal Processor # 44min
key: multi-modal-processor key: multi-modal-processor
timeout_in_minutes: 60 timeout_in_minutes: 60
device: intel_gpu device: intel_gpu
@@ -126,14 +124,14 @@ steps:
--num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT --shard-id=$$BUILDKITE_PARALLEL_JOB' --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT --shard-id=$$BUILDKITE_PARALLEL_JOB'
parallelism: 4 parallelism: 4
- label: ":intel: (BMG) Multi-Modal Models (Extended Generation 1)" - label: Multi-Modal Models (Extended Generation 1)
key: multi-modal-models-extended-generation-1 key: multi-modal-models-extended-generation-1
timeout_in_minutes: 60 timeout_in_minutes: 60
device: intel_gpu device: intel_gpu
agent_tags: agent_tags:
label: production label: production
gpu: 1+ gpu: 1+
mem: 16+ mem: 24+
no_plugin: true no_plugin: true
working_dir: "." working_dir: "."
env: env:
@@ -1,8 +1,8 @@
group: Quantization Intel group: Quantization
depends_on: depends_on:
- image-build-xpu - image-build-xpu
steps: steps:
- label: ":intel: (BMG) Quantization" - label: Quantization
key: quantization key: quantization
timeout_in_minutes: 30 timeout_in_minutes: 30
env: env:
@@ -25,7 +25,5 @@ steps:
- >- - >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'VLLM_TEST_FORCE_LOAD_FORMAT=auto pytest -v -s tests/quantization/test_per_token_kv_cache.py --deselect="tests/quantization/test_per_token_kv_cache.py::test_triton_unified_attention_per_token_head_scale[int4-16-128-num_heads0-seq_lens1]" 'VLLM_TEST_FORCE_LOAD_FORMAT=auto pytest -v -s tests/quantization/test_per_token_kv_cache.py --deselect="tests/quantization/test_per_token_kv_cache.py::test_triton_unified_attention_per_token_head_scale[int4-16-128-num_heads0-seq_lens1]"
&& VLLM_TEST_FORCE_LOAD_FORMAT=auto pytest -v -s tests/quantization/test_auto_awq.py && VLLM_TEST_FORCE_LOAD_FORMAT=auto pytest -v -s tests/quantization/test_auto_awq.py'
&& VLLM_TEST_FORCE_LOAD_FORMAT=auto pytest -v -s tests/quantization/test_fp8_per_channel.py
&& VLLM_TEST_FORCE_LOAD_FORMAT=auto pytest -v -s tests/models/quantization/test_fp8_per_channel.py --deselect="tests/models/quantization/test_fp8_per_channel.py::test_fp8_per_channel_logprobs[moe]"'
+2 -29
View File
@@ -2,37 +2,10 @@ group: Samplers Intel
depends_on: depends_on:
- image-build-xpu - image-build-xpu
steps: steps:
- label: ":intel: (BMG) Samplers Test (XPU Kernel)" - label: Samplers Test (XPU Kernel)
key: samplers-test-xpu-kernel-intel key: samplers-test-xpu-kernel-intel
timeout_in_minutes: 40 timeout_in_minutes: 40
device: intel_gpu device: intel_gpu
agent_tags:
label: production
gpu: 1+
mem: 16+
no_plugin: true
working_dir: "."
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
VLLM_TEST_DEVICE: "xpu"
source_file_dependencies:
- vllm/model_executor/layers
- vllm/sampling_metadata.py
- tests/samplers
- tests/conftest.py
- vllm/entrypoints/generate/beam_search
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests &&
VLLM_XPU_USE_SAMPLER_KERNEL=1 pytest -v -s samplers
--ignore=samplers/test_beam_search.py'
- label: ":intel: (BMG) Samplers Beam Search Test (XPU Kernel)"
key: samplers-beam-search-test-xpu-kernel-intel
timeout_in_minutes: 40
device: intel_gpu
agent_tags: agent_tags:
label: production label: production
gpu: 1+ gpu: 1+
@@ -53,4 +26,4 @@ steps:
- >- - >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests && 'cd tests &&
VLLM_XPU_USE_SAMPLER_KERNEL=1 pytest -v -s samplers/test_beam_search.py' VLLM_XPU_USE_SAMPLER_KERNEL=1 pytest -v -s samplers'
+13 -38
View File
@@ -14,7 +14,7 @@ steps:
limit: 2 limit: 2
- exit_status: -10 # Agent was lost - exit_status: -10 # Agent was lost
limit: 2 limit: 2
- label: ":intel: (BMG) Example Test" - label: "XPU example Test"
key: xpu-example-test key: xpu-example-test
depends_on: depends_on:
- image-build-xpu - image-build-xpu
@@ -23,7 +23,7 @@ steps:
agent_tags: agent_tags:
label: production label: production
gpu: 2+ gpu: 2+
mem: 16+ mem: 24+
no_plugin: true no_plugin: true
env: env:
REGISTRY: "public.ecr.aws/q9t5s3a7" REGISTRY: "public.ecr.aws/q9t5s3a7"
@@ -46,9 +46,10 @@ steps:
python3 examples/basic/offline_inference/generate.py --model ibm-research/PowerMoE-3b --block-size 64 --enforce-eager -tp 2 && python3 examples/basic/offline_inference/generate.py --model ibm-research/PowerMoE-3b --block-size 64 --enforce-eager -tp 2 &&
python3 examples/basic/offline_inference/generate.py --model ibm-research/PowerMoE-3b --block-size 64 --enforce-eager -tp 2 --enable-expert-parallel && python3 examples/basic/offline_inference/generate.py --model ibm-research/PowerMoE-3b --block-size 64 --enforce-eager -tp 2 --enable-expert-parallel &&
python3 examples/basic/offline_inference/generate.py --model superjob/Qwen3-4B-Instruct-2507-GPTQ-Int4 --max-model-len 8192 && python3 examples/basic/offline_inference/generate.py --model superjob/Qwen3-4B-Instruct-2507-GPTQ-Int4 --max-model-len 8192 &&
VLLM_XPU_FUSED_MOE_USE_REF=1 python3 examples/basic/offline_inference/generate.py --model Qwen/Qwen3-30B-A3B-Instruct-2507-FP8 --enforce-eager -tp 2 --max-model-len 8192 &&
python3 examples/basic/offline_inference/generate.py --model INCModel/Qwen3-30B-A3B-Instruct-2507-MXFP4-LLMC -tp 2 --max-model-len 8192 python3 examples/basic/offline_inference/generate.py --model INCModel/Qwen3-30B-A3B-Instruct-2507-MXFP4-LLMC -tp 2 --max-model-len 8192
' '
- label: ":intel: (BMG) W8A8 FP8 Linear Examples" - label: "XPU W8A8 FP8 Linear Examples"
key: xpu-w8a8-fp8-linear-examples key: xpu-w8a8-fp8-linear-examples
depends_on: depends_on:
- image-build-xpu - image-build-xpu
@@ -57,7 +58,7 @@ steps:
agent_tags: agent_tags:
label: production label: production
gpu: 1+ gpu: 1+
mem: 16+ mem: 24+
no_plugin: true no_plugin: true
env: env:
REGISTRY: "public.ecr.aws/q9t5s3a7" REGISTRY: "public.ecr.aws/q9t5s3a7"
@@ -76,7 +77,7 @@ steps:
python3 examples/basic/offline_inference/generate.py --linear-backend xpu --model meta-llama/Llama-3.2-1B-Instruct --quantization fp8_per_tensor --enforce-eager --max-model-len 4096 && python3 examples/basic/offline_inference/generate.py --linear-backend xpu --model meta-llama/Llama-3.2-1B-Instruct --quantization fp8_per_tensor --enforce-eager --max-model-len 4096 &&
python3 examples/basic/offline_inference/generate.py --linear-backend torch --model meta-llama/Llama-3.2-1B-Instruct --quantization fp8_per_tensor --enforce-eager --max-model-len 4096 python3 examples/basic/offline_inference/generate.py --linear-backend torch --model meta-llama/Llama-3.2-1B-Instruct --quantization fp8_per_tensor --enforce-eager --max-model-len 4096
' '
- label: ":intel: (BMG) V1 test" - label: "XPU V1 test"
key: xpu-v1-test key: xpu-v1-test
depends_on: depends_on:
- image-build-xpu - image-build-xpu
@@ -85,7 +86,7 @@ steps:
agent_tags: agent_tags:
label: production label: production
gpu: 1+ gpu: 1+
mem: 16+ mem: 24+
no_plugin: true no_plugin: true
env: env:
REGISTRY: "public.ecr.aws/q9t5s3a7" REGISTRY: "public.ecr.aws/q9t5s3a7"
@@ -105,35 +106,9 @@ steps:
pytest -v -s v1/test_serial_utils.py && pytest -v -s v1/test_serial_utils.py &&
pytest -v -s v1/e2e/general/test_correctness_sliding_window.py && pytest -v -s v1/e2e/general/test_correctness_sliding_window.py &&
pytest -v -s v1/spec_decode --ignore=v1/spec_decode/test_max_len.py --ignore=v1/spec_decode/test_speculators_eagle3.py --ignore=v1/spec_decode/test_acceptance_length.py --ignore=v1/spec_decode/test_speculators_correctness.py --deselect=tests/v1/spec_decode/test_mtp.py::test_glm_mtp_defers_lm_head && pytest -v -s v1/spec_decode --ignore=v1/spec_decode/test_max_len.py --ignore=v1/spec_decode/test_speculators_eagle3.py --ignore=v1/spec_decode/test_acceptance_length.py --ignore=v1/spec_decode/test_speculators_correctness.py --deselect=tests/v1/spec_decode/test_mtp.py::test_glm_mtp_defers_lm_head &&
pytest -v -s v1/kv_connector/unit --ignore=v1/kv_connector/unit/test_lmcache_integration.py --ignore=v1/kv_connector/unit/test_hf3fs_client.py --ignore=v1/kv_connector/unit/test_hf3fs_connector.py --ignore=v1/kv_connector/unit/test_hf3fs_metadata_server.py --ignore=v1/kv_connector/unit/test_offloading_connector.py --ignore=v1/kv_connector/unit/test_multi_connector.py --ignore=v1/kv_connector/unit/test_example_connector.py --deselect=tests/v1/kv_connector/unit/test_nixl_connector.py::test_abort_timeout_on_prefiller[ray]' pytest -v -s v1/kv_connector/unit --ignore=v1/kv_connector/unit/test_lmcache_integration.py --ignore=v1/kv_connector/unit/test_hf3fs_client.py --ignore=v1/kv_connector/unit/test_hf3fs_connector.py --ignore=v1/kv_connector/unit/test_hf3fs_metadata_server.py --ignore=v1/kv_connector/unit/test_offloading_connector.py --ignore=v1/kv_connector/unit/test_nixl_connector_hma.py --deselect=tests/v1/kv_connector/unit/test_nixl_connector.py::test_abort_timeout_on_prefiller[ray]'
parallelism: 2 parallelism: 2
- label: ":intel: (BMG) Qwen3-30B + V1 KV Connector" - label: "XPU server test"
key: xpu-qwen3-30b-v1-kv-connector
depends_on:
- image-build-xpu
timeout_in_minutes: 90
device: intel_gpu
agent_tags:
label: production
gpu: 2+
mem: 24+
no_plugin: true
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
source_file_dependencies:
- vllm/
- .buildkite/intel_jobs/test-intel.yaml
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests &&
pip install dpctl &&
pytest -v -s v1/kv_connector/unit/test_multi_connector.py &&
pytest -v -s v1/kv_connector/unit/test_example_connector.py &&
cd .. &&
VLLM_XPU_FUSED_MOE_USE_REF=1 python3 examples/basic/offline_inference/generate.py --model Qwen/Qwen3-30B-A3B-Instruct-2507-FP8 --enforce-eager -tp 2 --max-model-len 8192'
- label: ":intel: (BMG) Server test"
key: xpu-server-test key: xpu-server-test
depends_on: depends_on:
- image-build-xpu - image-build-xpu
@@ -156,7 +131,7 @@ steps:
'cd tests && 'cd tests &&
pytest -v -s entrypoints/multimodal/openai/chat_completion/test_audio_in_video.py && pytest -v -s entrypoints/multimodal/openai/chat_completion/test_audio_in_video.py &&
pytest -v -s benchmarks/test_serve_cli.py' pytest -v -s benchmarks/test_serve_cli.py'
- label: ":intel: (BMG) Quantization test" - label: "XPU quantization test"
key: xpu-quantization-test key: xpu-quantization-test
depends_on: depends_on:
- image-build-xpu - image-build-xpu
@@ -178,8 +153,8 @@ steps:
bash .buildkite/scripts/hardware_ci/run-intel-test.sh bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests && 'cd tests &&
pytest -v -s quantization/test_auto_round.py && pytest -v -s quantization/test_auto_round.py &&
pytest -v -s quantization/test_online.py --deselect=tests/quantization/test_online.py::test_online_quantization_loads_real_weights --deselect="tests/quantization/test_online.py::test_online_quantization[False-requantization_mxfp8_ptcp_fp8]"' pytest -v -s quantization/test_online.py --deselect=tests/quantization/test_online.py::test_online_quantization_loads_real_weights'
- label: ":intel: (BMG) Compressed tensors FP8 test" - label: "XPU compressed tensors FP8 test"
key: xpu-compressed-tensors-fp8-test key: xpu-compressed-tensors-fp8-test
depends_on: depends_on:
- image-build-xpu - image-build-xpu
@@ -204,7 +179,7 @@ steps:
'cd tests && 'cd tests &&
pytest -v -s quantization/test_compressed_tensors.py::test_compressed_tensors_fp8 && pytest -v -s quantization/test_compressed_tensors.py::test_compressed_tensors_fp8 &&
pytest -v -s quantization/test_compressed_tensors.py::test_compressed_tensors_fp8_block_enabled' pytest -v -s quantization/test_compressed_tensors.py::test_compressed_tensors_fp8_block_enabled'
- label: ":intel: (BMG) Graph" - label: "XPU Graph"
key: xpu-graph key: xpu-graph
depends_on: depends_on:
- image-build-xpu - image-build-xpu
@@ -8,28 +8,16 @@ pytest -s -v test_lm_eval_correctness.py \
--tp-size=1 --tp-size=1
""" """
import gc
import os import os
import time
from contextlib import contextmanager from contextlib import contextmanager
import lm_eval import lm_eval
import pytest import pytest
import torch
import yaml import yaml
from lm_eval.api.registry import get_model
from lm_eval.utils import simple_parse_args_string
from vllm import LLM
from vllm.distributed import cleanup_dist_env_and_memory
from vllm.logger import init_logger
from vllm.platforms import current_platform from vllm.platforms import current_platform
logger = init_logger(__name__)
DEFAULT_RTOL = 0.08 DEFAULT_RTOL = 0.08
ROCM_ENGINE_SHUTDOWN_TIMEOUT_S = 60.0
MEMORY_RELEASE_TIMEOUT_S = 240.0
@contextmanager @contextmanager
@@ -58,48 +46,6 @@ def scoped_env_vars(new_env: dict[str, str]):
os.environ.pop(key, None) os.environ.pop(key, None)
def _wait_for_memory_release(gpu_memory_utilization: float) -> None:
devices = range(torch.accelerator.device_count())
deadline = time.monotonic() + MEMORY_RELEASE_TIMEOUT_S
while True:
memory = [torch.accelerator.get_memory_info(d) for d in devices]
if all(free >= total * gpu_memory_utilization for free, total in memory):
return
if time.monotonic() >= deadline:
usage = ", ".join(
f"{d}: {free / 2**30:.2f}/{total / 2**30:.2f} GiB free"
for d, (free, total) in zip(devices, memory)
)
logger.warning(
"GPU memory not released after %ss (%s); the next model needs %.2f",
MEMORY_RELEASE_TIMEOUT_S,
usage,
gpu_memory_utilization,
)
return
time.sleep(1)
def _shutdown_lm(lm) -> None:
if not current_platform.is_rocm():
return
llm = getattr(lm, "model", None)
if not isinstance(llm, LLM):
return
gpu_memory_utilization = (
llm.llm_engine.vllm_config.cache_config.gpu_memory_utilization
)
try:
llm.llm_engine.engine_core.shutdown(timeout=ROCM_ENGINE_SHUTDOWN_TIMEOUT_S)
except Exception:
logger.exception("Engine core shutdown raised; GPU memory may leak")
del llm
lm.model = None
gc.collect()
cleanup_dist_env_and_memory()
_wait_for_memory_release(gpu_memory_utilization)
def launch_lm_eval(eval_config, tp_size): def launch_lm_eval(eval_config, tp_size):
trust_remote_code = eval_config.get("trust_remote_code", False) trust_remote_code = eval_config.get("trust_remote_code", False)
max_model_len = eval_config.get("max_model_len", 4096) max_model_len = eval_config.get("max_model_len", 4096)
@@ -136,17 +82,9 @@ def launch_lm_eval(eval_config, tp_size):
env_vars = eval_config.get("env_vars", None) env_vars = eval_config.get("env_vars", None)
with scoped_env_vars(env_vars): with scoped_env_vars(env_vars):
if current_platform.is_rocm():
model = get_model(backend).create_from_arg_string(
model_args,
{"batch_size": batch_size, "max_batch_size": None, "device": None},
)
model_kwargs = {"metadata": simple_parse_args_string(model_args)}
else:
model = backend
model_kwargs = {"model_args": model_args}
results = lm_eval.simple_evaluate( results = lm_eval.simple_evaluate(
model=model, model=backend,
model_args=model_args,
tasks=[task["name"] for task in eval_config["tasks"]], tasks=[task["name"] for task in eval_config["tasks"]],
num_fewshot=eval_config["num_fewshot"], num_fewshot=eval_config["num_fewshot"],
limit=eval_config["limit"], limit=eval_config["limit"],
@@ -160,9 +98,7 @@ def launch_lm_eval(eval_config, tp_size):
# Forward decoding and early-stop controls (e.g., max_gen_toks, until=...) # Forward decoding and early-stop controls (e.g., max_gen_toks, until=...)
gen_kwargs=eval_config.get("gen_kwargs"), gen_kwargs=eval_config.get("gen_kwargs"),
batch_size=batch_size, batch_size=batch_size,
**model_kwargs,
) )
_shutdown_lm(model)
return results return results
+450 -222
View File
@@ -355,10 +355,6 @@ steps:
queue: arm64_cpu_queue_release queue: arm64_cpu_queue_release
commands: commands:
- "aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7" - "aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7"
# TODO(arpera):
# Remove BUILD_NIXL=true once NIXL has a wheel built against torch 2.15
# For details: https://github.com/vllm-project/vllm/pull/60090
# Tracking issue report: https://github.com/ai-dynamo/nixl/issues/2297
- | - |
DOCKER_BUILDKIT=1 docker buildx build --load \ DOCKER_BUILDKIT=1 docker buildx build --load \
$(bash .buildkite/scripts/docker-build-metadata-args.sh cu134) \ $(bash .buildkite/scripts/docker-build-metadata-args.sh cu134) \
@@ -370,7 +366,6 @@ steps:
--build-arg torch_cuda_arch_list="${CUDA_ARCH_AARCH64}" \ --build-arg torch_cuda_arch_list="${CUDA_ARCH_AARCH64}" \
--build-arg INSTALL_KV_CONNECTORS=true \ --build-arg INSTALL_KV_CONNECTORS=true \
--build-arg INSTALL_RUBIN_PRERELEASE=true \ --build-arg INSTALL_RUBIN_PRERELEASE=true \
--build-arg BUILD_NIXL=true \
--build-arg TRITON_INSTALL_FROM_SOURCE_REPO=https://github.com/triton-lang/triton.git \ --build-arg TRITON_INSTALL_FROM_SOURCE_REPO=https://github.com/triton-lang/triton.git \
--build-arg TRITON_INSTALL_FROM_SOURCE_REVISION=3f6e41132b5edf639bfb872ad73d4688765e08b8 \ --build-arg TRITON_INSTALL_FROM_SOURCE_REVISION=3f6e41132b5edf639bfb872ad73d4688765e08b8 \
--build-arg RUN_WHEEL_CHECK=false \ --build-arg RUN_WHEEL_CHECK=false \
@@ -703,270 +698,348 @@ steps:
DOCKER_BUILDKIT: "1" DOCKER_BUILDKIT: "1"
DOCKERHUB_USERNAME: "vllmbot" DOCKERHUB_USERNAME: "vllmbot"
- trigger: "perf-eval"
label: "Trigger perf-eval (CUDA)"
key: trigger-perf-eval-cuda
if: build.env("NIGHTLY") == "1" || build.env("RUN_PERF_EVAL") == "1"
depends_on:
- build-release-image-x86
async: true
soft_fail: true
build:
branch: "main"
message: "(NIGHTLY=${NIGHTLY:-0}) Evaluate vLLM $BUILDKITE_COMMIT (CUDA)"
env:
NIGHTLY: 1 # run all perf-eval workloads
VLLM_COMMIT: "$BUILDKITE_COMMIT"
VLLM_IMAGE_CUDA: "public.ecr.aws/q9t5s3a7/vllm-release-repo:$BUILDKITE_COMMIT-x86_64"
- trigger: "perf-eval"
label: "Trigger perf-eval (ROCm)"
key: trigger-perf-eval-rocm
if: build.env("NIGHTLY") == "1" || build.env("RUN_PERF_EVAL") == "1"
depends_on:
- build-rocm100-release-image
async: true
soft_fail: true
build:
branch: "main"
message: "(NIGHTLY=${NIGHTLY:-0}) Evaluate vLLM $BUILDKITE_COMMIT (ROCm)"
env:
NIGHTLY: 1 # run all perf-eval workloads
VLLM_COMMIT: "$BUILDKITE_COMMIT"
VLLM_IMAGE_ROCM: "public.ecr.aws/q9t5s3a7/vllm-release-repo:$BUILDKITE_COMMIT-rocm100"
# ============================================================================= # =============================================================================
# ROCm Release Pipelines (x86_64 only) # ROCm Release Pipeline (x86_64 only)
# ============================================================================= # =============================================================================
# #
# One group per ROCm stack, identified by its base Dockerfile (ROCM_BASE_DOCKERFILE). # vLLM version is determined by the Buildkite checkout (like CUDA pipeline).
# The scripts derive the ROCm version, wheel variant, cache keys and image tags from # To build a specific version, trigger the build from that branch/tag.
# it (.buildkite/scripts/rocm/stack.sh), so both groups share one set of step
# definitions, anchored on the ROCm 10.0 group.
# #
# - ROCm 10.0 (TheRock, docker/Dockerfile.rocm{,_base}) is the default: it owns the # Environment variables for ROCm builds (set via Buildkite UI or schedule):
# :nightly / :latest images and the root wheel index. #
# - ROCm 7.2 (docker/Dockerfile.rocm_72{,_base}) is kept for a transition period as # Note: ROCm version is determined by BASE_IMAGE in docker/Dockerfile.rocm_base
# rocm72 images and wheels.
# #
# vLLM version is determined by the Buildkite checkout (like the CUDA pipeline).
# ============================================================================= # =============================================================================
- group: "Build ROCm 10.0 Wheel / Image" - group: "Build ROCm Wheel / Image "
key: "build-rocm100" key: "build-rocm-wheel-image"
depends_on: ~ depends_on: ~
steps: steps:
- label: ":rocm: Build ROCm 10.0 Base Image & Wheels" # ROCm Job 1: Build ROCm Base Wheels (with S3 caching)
id: build-rocm100-base - label: ":rocm: Build ROCm Base Image & Wheels"
id: build-rocm-base-wheels
depends_on: ~ depends_on: ~
agents: &rocm_build_agents agents:
queue: cpu_queue_release queue: cpu_queue_release
timeout_in_minutes: 360 commands:
env: &rocm100_env - |
ROCM_BASE_DOCKERFILE: "docker/Dockerfile.rocm_base" set -euo pipefail
ROCM_DEFAULT_ARG: "--default"
# Generate cache key
CACHE_KEY=$$(.buildkite/scripts/cache-rocm-base-wheels.sh key)
ECR_CACHE_TAG="public.ecr.aws/q9t5s3a7/vllm-release-repo:$${CACHE_KEY}-rocm-base"
echo "========================================"
echo "ROCm Base Build Configuration"
echo "========================================"
echo " CACHE_KEY: $${CACHE_KEY}"
echo " ECR_CACHE_TAG: $${ECR_CACHE_TAG}"
echo "========================================"
# Login to ECR
aws ecr-public get-login-password --region us-east-1 | \
docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7
IMAGE_EXISTS=false
WHEELS_EXIST=false
# Check ECR for Docker image
if docker manifest inspect "$${ECR_CACHE_TAG}" > /dev/null 2>&1; then
IMAGE_EXISTS=true
echo "ECR image cache HIT"
fi
# Check S3 for wheels
WHEEL_CACHE_STATUS=$(.buildkite/scripts/cache-rocm-base-wheels.sh check)
if [ "$${WHEEL_CACHE_STATUS}" = "hit" ]; then
WHEELS_EXIST=true
echo "S3 wheels cache HIT"
fi
# Scenario 1: Both cached (best case)
if [ "$${IMAGE_EXISTS}" = "true" ] && [ "$${WHEELS_EXIST}" = "true" ]; then
echo ""
echo "FULL CACHE HIT - Reusing both image and wheels"
echo ""
# Download wheels
.buildkite/scripts/cache-rocm-base-wheels.sh download
# Save ECR tag for downstream jobs
buildkite-agent meta-data set "rocm-base-image-tag" "$${ECR_CACHE_TAG}"
# Scenario 2: Full rebuild needed
else
echo ""
echo " CACHE MISS - Building from scratch..."
echo ""
# Build full base image and push to ECR
DOCKER_BUILDKIT=1 docker buildx build \
--file docker/Dockerfile.rocm_base \
--tag "$${ECR_CACHE_TAG}" \
--build-arg USE_SCCACHE=1 \
--build-arg SCCACHE_BUCKET_NAME=vllm-build-sccache \
--build-arg SCCACHE_REGION_NAME=us-west-2 \
--build-arg SCCACHE_S3_NO_CREDENTIALS=0 \
--push \
.
# Build wheel extraction stage
DOCKER_BUILDKIT=1 docker buildx build \
--file docker/Dockerfile.rocm_base \
--tag rocm-base-debs:$${BUILDKITE_BUILD_NUMBER} \
--target debs_wheel_release \
--build-arg USE_SCCACHE=1 \
--build-arg SCCACHE_BUCKET_NAME=vllm-build-sccache \
--build-arg SCCACHE_REGION_NAME=us-west-2 \
--build-arg SCCACHE_S3_NO_CREDENTIALS=0 \
--load \
.
# Extract and upload wheels
mkdir -p artifacts/rocm-base-wheels
cid=$(docker create rocm-base-debs:$${BUILDKITE_BUILD_NUMBER})
docker cp $${cid}:/app/debs/. artifacts/rocm-base-wheels/
docker rm $${cid}
.buildkite/scripts/cache-rocm-base-wheels.sh upload
# Cache base docker image to ECR
docker push "$${ECR_CACHE_TAG}"
buildkite-agent meta-data set "rocm-base-image-tag" "$${ECR_CACHE_TAG}"
echo ""
echo " Build complete - Image and wheels cached"
fi
artifact_paths:
- "artifacts/rocm-base-wheels/*.whl"
env:
DOCKER_BUILDKIT: "1" DOCKER_BUILDKIT: "1"
S3_BUCKET: "vllm-wheels" S3_BUCKET: "vllm-wheels"
DOCKERHUB_USERNAME: "vllmbot"
commands: &rocm_base_commands
- |
set -euo pipefail
source .buildkite/scripts/rocm/stack.sh "$${ROCM_BASE_DOCKERFILE}"
SCCACHE_ARGS=(--build-arg USE_SCCACHE=1 --build-arg SCCACHE_BUCKET_NAME=vllm-build-sccache
--build-arg SCCACHE_REGION_NAME=us-west-2 --build-arg SCCACHE_S3_NO_CREDENTIALS=0)
aws ecr-public get-login-password --region us-east-1 | \
docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7
# Base image, cached in ECR by the base Dockerfile's hash # ROCm Job 2: Build vLLM ROCm Wheel
if docker manifest inspect "$${ROCM_STACK_ECR_BASE}" > /dev/null 2>&1; then - label: ":python: Build vLLM ROCm Wheel - x86_64"
echo "ECR image cache HIT: $${ROCM_STACK_ECR_BASE}" id: build-rocm-vllm-wheel
else depends_on:
docker buildx build --file "$${ROCM_BASE_DOCKERFILE}" --tag "$${ROCM_STACK_ECR_BASE}" \ - step: build-rocm-base-wheels
"$${SCCACHE_ARGS[@]}" --push . allow_failure: false
fi agents:
queue: cpu_queue_release
# Base wheels, cached in S3 (see cache-rocm-base-wheels.sh)
CACHE=.buildkite/scripts/cache-rocm-base-wheels.sh
if [ "$$($${CACHE} "$${ROCM_BASE_DOCKERFILE}" check)" = "hit" ]; then
$${CACHE} "$${ROCM_BASE_DOCKERFILE}" download
else
DEBS="rocm-base-debs:$${BUILDKITE_BUILD_NUMBER}-$${ROCM_STACK_VARIANT}"
docker buildx build --file "$${ROCM_BASE_DOCKERFILE}" --tag "$${DEBS}" \
--target debs_wheel_release "$${SCCACHE_ARGS[@]}" --load .
mkdir -p "$${ROCM_STACK_BASE_WHEELS_DIR}"
cid=$$(docker create "$${DEBS}")
docker cp "$${cid}":/app/debs/. "$${ROCM_STACK_BASE_WHEELS_DIR}/"
docker rm "$${cid}"
$${CACHE} "$${ROCM_BASE_DOCKERFILE}" upload
fi
buildkite-agent artifact upload "$${ROCM_STACK_BASE_WHEELS_DIR}/*.whl"
- label: ":python: Build vLLM ROCm 10.0 Wheel - x86_64"
id: build-rocm100-vllm-wheel
depends_on: build-rocm100-base
agents: *rocm_build_agents
timeout_in_minutes: 180 timeout_in_minutes: 180
env: *rocm100_env commands:
commands: &rocm_vllm_wheel_commands # Download artifacts and prepare Docker image
- | - |
set -euo pipefail set -euo pipefail
source .buildkite/scripts/rocm/stack.sh "$${ROCM_BASE_DOCKERFILE}"
# Buildkite's default fetch doesn't update moved tags; version detection needs them # Ensure git tags are up-to-date (Buildkite's default fetch doesn't update tags)
# This fixes version detection when tags are moved/force-pushed
echo "Fetching latest tags from origin..."
git fetch --tags --force origin git fetch --tags --force origin
buildkite-agent artifact download "$${ROCM_STACK_BASE_WHEELS_DIR}/*.whl" .
# Log tag information for debugging version detection
echo "========================================"
echo "Git Tag Verification"
echo "========================================"
echo "Current HEAD: $(git rev-parse HEAD)"
echo "git describe --tags: $(git describe --tags 2>/dev/null || echo 'No tags found')"
echo ""
echo "Recent tags (pointing to commits near HEAD):"
git tag -l --sort=-creatordate | head -5
echo "setuptools_scm version detection:"
pip install -q setuptools_scm 2>/dev/null || true
python3 -c "import setuptools_scm; print(' Detected version:', setuptools_scm.get_version())" 2>/dev/null || echo " (setuptools_scm not available in this environment)"
echo "========================================"
# Download wheel artifacts from current build
echo "Downloading wheel artifacts from current build"
buildkite-agent artifact download "artifacts/rocm-base-wheels/*.whl" .
# Get ECR image tag from metadata (set by build-rocm-base-wheels)
ECR_IMAGE_TAG="$$(buildkite-agent meta-data get rocm-base-image-tag 2>/dev/null || echo '')"
if [ -z "$${ECR_IMAGE_TAG}" ]; then
echo "ERROR: rocm-base-image-tag metadata not found"
echo "This should have been set by the build-rocm-base-wheels job"
exit 1
fi
echo "Pulling base Docker image from ECR: $${ECR_IMAGE_TAG}"
# Login to ECR
aws ecr-public get-login-password --region us-east-1 | \ aws ecr-public get-login-password --region us-east-1 | \
docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7 docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7
docker pull "$${ROCM_STACK_ECR_BASE}"
# Pull base Docker image from ECR
docker pull "$${ECR_IMAGE_TAG}"
echo "Loaded base image: $${ECR_IMAGE_TAG}"
# Prepare base wheels for Docker build context
mkdir -p docker/context/base-wheels mkdir -p docker/context/base-wheels
touch docker/context/base-wheels/.keep touch docker/context/base-wheels/.keep
cp "$${ROCM_STACK_BASE_WHEELS_DIR}"/*.whl docker/context/base-wheels/ cp artifacts/rocm-base-wheels/*.whl docker/context/base-wheels/
docker build --file "$${ROCM_STACK_DOCKERFILE}" --target export_vllm_wheel_release \ echo "Base wheels for vLLM build:"
--output type=local,dest=rocm-dist \ ls -lh docker/context/base-wheels/
--build-arg BASE_IMAGE="$${ROCM_STACK_ECR_BASE}" --build-arg REMOTE_VLLM=0 \
--build-arg GIT_REPO_CHECK=1 --build-arg USE_SCCACHE=1 \
--build-arg SCCACHE_BUCKET_NAME=vllm-build-sccache \
--build-arg SCCACHE_REGION_NAME=us-west-2 --build-arg SCCACHE_S3_NO_CREDENTIALS=0 .
mkdir -p "$${ROCM_STACK_VLLM_WHEEL_DIR}"
cp rocm-dist/*.whl "$${ROCM_STACK_VLLM_WHEEL_DIR}/"
buildkite-agent artifact upload "$${ROCM_STACK_VLLM_WHEEL_DIR}/*.whl"
- label: ":s3: Upload ROCm 10.0 Wheels to S3" echo "========================================"
id: upload-rocm100-wheels echo "Building vLLM wheel with:"
depends_on: build-rocm100-vllm-wheel echo " BUILDKITE_COMMIT: $${BUILDKITE_COMMIT}"
agents: *rocm_build_agents echo " BUILDKITE_BRANCH: $${BUILDKITE_BRANCH}"
echo " BASE_IMAGE: $${ECR_IMAGE_TAG}"
echo "========================================"
# Build vLLM wheel using local checkout (REMOTE_VLLM=0)
DOCKER_BUILDKIT=1 docker build \
--file docker/Dockerfile.rocm \
--target export_vllm_wheel_release \
--output type=local,dest=rocm-dist \
--build-arg BASE_IMAGE="$${ECR_IMAGE_TAG}" \
--build-arg REMOTE_VLLM=0 \
--build-arg GIT_REPO_CHECK=1 \
--build-arg USE_SCCACHE=1 \
--build-arg SCCACHE_BUCKET_NAME=vllm-build-sccache \
--build-arg SCCACHE_REGION_NAME=us-west-2 \
--build-arg SCCACHE_S3_NO_CREDENTIALS=0 \
.
echo "Built vLLM wheel:"
ls -lh rocm-dist/*.whl
# Copy wheel to artifacts directory
mkdir -p artifacts/rocm-vllm-wheel
cp rocm-dist/*.whl artifacts/rocm-vllm-wheel/
echo "Final vLLM wheel:"
ls -lh artifacts/rocm-vllm-wheel/
artifact_paths:
- "artifacts/rocm-vllm-wheel/*.whl"
env:
DOCKER_BUILDKIT: "1"
S3_BUCKET: "vllm-wheels"
# ROCm Job 3: Upload Wheels to S3
- label: ":s3: Upload ROCm Wheels to S3"
id: upload-rocm-wheels
depends_on:
- step: build-rocm-vllm-wheel
allow_failure: false
agents:
queue: cpu_queue_release
timeout_in_minutes: 60 timeout_in_minutes: 60
env: *rocm100_env commands:
commands: &rocm_upload_commands # Download all wheel artifacts and run upload
- | - |
set -euo pipefail set -euo pipefail
source .buildkite/scripts/rocm/stack.sh "$${ROCM_BASE_DOCKERFILE}"
buildkite-agent artifact download "$${ROCM_STACK_BASE_WHEELS_DIR}/*.whl" .
buildkite-agent artifact download "$${ROCM_STACK_VLLM_WHEEL_DIR}/*.whl" .
bash .buildkite/scripts/upload-rocm-wheels.sh "$${ROCM_BASE_DOCKERFILE}"
- label: ":memo: Annotate ROCm 10.0 wheel release" # Download artifacts from current build
id: annotate-rocm100-release echo "Downloading artifacts from current build"
depends_on: upload-rocm100-wheels buildkite-agent artifact download "artifacts/rocm-base-wheels/*.whl" .
agents: *rocm_build_agents buildkite-agent artifact download "artifacts/rocm-vllm-wheel/*.whl" .
env: *rocm100_env
commands: &rocm_annotate_commands
- bash .buildkite/scripts/annotate-rocm-release.sh "$${ROCM_BASE_DOCKERFILE}" $${ROCM_DEFAULT_ARG}
# Generate https://wheels.vllm.ai/rocm/ (the default stack's variant), so users can # # Run upload script
# `uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/` (release only) bash .buildkite/scripts/upload-rocm-wheels.sh
env:
DOCKER_BUILDKIT: "1"
S3_BUCKET: "vllm-wheels"
# ROCm Job 4: Annotate ROCm Wheel Release
- label: ":memo: Annotate ROCm wheel release"
id: annotate-rocm-release
depends_on:
- upload-rocm-wheels
agents:
queue: cpu_queue_release
commands:
- "bash .buildkite/scripts/annotate-rocm-release.sh"
env:
S3_BUCKET: "vllm-wheels"
# ROCm Job 5: Generate Root Index for ROCm Wheels (for release only)
# This is the job to create https://wheels.vllm.ai/rocm/ index allowing
# users to install with `uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/`
- block: "Generate Root Index for ROCm Wheels for Release" - block: "Generate Root Index for ROCm Wheels for Release"
key: block-generate-root-index-rocm-wheels key: block-generate-root-index-rocm-wheels
depends_on: depends_on: upload-rocm-wheels
- upload-rocm100-wheels
- upload-rocm72-wheels
- label: ":package: Generate Root Index for ROCm Wheels for Release" - label: ":package: Generate Root Index for ROCm Wheels for Release"
depends_on: block-generate-root-index-rocm-wheels depends_on: block-generate-root-index-rocm-wheels
id: generate-root-index-rocm-wheels id: generate-root-index-rocm-wheels
agents: *rocm_build_agents agents:
env: *rocm100_env queue: cpu_queue_release
commands: commands:
- "bash tools/vllm-rocm/generate-rocm-wheels-root-index.sh" - "bash tools/vllm-rocm/generate-rocm-wheels-root-index.sh"
env:
S3_BUCKET: "vllm-wheels"
VARIANT: "rocm723"
- label: ":docker: Build release image - x86_64 - ROCm 10.0" # ROCm Job 6: Build ROCm Release Docker Image
id: build-rocm100-release-image - label: ":docker: Build release image - x86_64 - ROCm"
id: build-rocm-release-image
depends_on: depends_on:
- step: block-build-release-images - step: block-build-release-images
allow_failure: true allow_failure: true
- step: build-rocm100-base - step: build-rocm-base-wheels
allow_failure: false allow_failure: false
agents: *rocm_build_agents agents:
timeout_in_minutes: 180 queue: cpu_queue_release
env: *rocm100_env timeout_in_minutes: 60
commands: &rocm_release_image_commands commands:
- | - |
set -euo pipefail set -euo pipefail
source .buildkite/scripts/rocm/stack.sh "$${ROCM_BASE_DOCKERFILE}"
# Login to ECR
aws ecr-public get-login-password --region us-east-1 | \ aws ecr-public get-login-password --region us-east-1 | \
docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7 docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7
docker pull "$${ROCM_STACK_ECR_BASE}"
docker build --file "$${ROCM_STACK_DOCKERFILE}" --target vllm-openai \ # Get ECR image tag from metadata (set by build-rocm-base-wheels)
--tag "$${ROCM_STACK_ECR_IMAGE}" --progress plain \ ECR_IMAGE_TAG="$$(buildkite-agent meta-data get rocm-base-image-tag 2>/dev/null || echo '')"
--build-arg max_jobs=16 --build-arg BASE_IMAGE="$${ROCM_STACK_ECR_BASE}" \ if [ -z "$${ECR_IMAGE_TAG}" ]; then
--build-arg INSTALL_LMCACHE=true --build-arg USE_SCCACHE=1 \ echo "ERROR: rocm-base-image-tag metadata not found"
echo "This should have been set by the build-rocm-base-wheels job"
exit 1
fi
echo "Pulling base Docker image from ECR: $${ECR_IMAGE_TAG}"
# Pull base Docker image from ECR
docker pull "$${ECR_IMAGE_TAG}"
echo "Loaded base image: $${ECR_IMAGE_TAG}"
# Pass the base image ECR tag to downstream steps (nightly publish)
buildkite-agent meta-data set "rocm-base-ecr-tag" "$${ECR_IMAGE_TAG}"
echo "========================================"
echo "Building vLLM ROCm release image with:"
echo " BASE_IMAGE: $${ECR_IMAGE_TAG}"
echo " BUILDKITE_COMMIT: $${BUILDKITE_COMMIT}"
echo "========================================"
# Build vLLM ROCm release image using cached base
DOCKER_BUILDKIT=1 docker build \
--build-arg max_jobs=16 \
--build-arg BASE_IMAGE="$${ECR_IMAGE_TAG}" \
--build-arg USE_SCCACHE=1 \
--build-arg SCCACHE_BUCKET_NAME=vllm-build-sccache \ --build-arg SCCACHE_BUCKET_NAME=vllm-build-sccache \
--build-arg SCCACHE_REGION_NAME=us-west-2 --build-arg SCCACHE_S3_NO_CREDENTIALS=0 . --build-arg SCCACHE_REGION_NAME=us-west-2 \
docker push "$${ROCM_STACK_ECR_IMAGE}" --build-arg SCCACHE_S3_NO_CREDENTIALS=0 \
--build-arg INSTALL_LMCACHE=true \
--tag public.ecr.aws/q9t5s3a7/vllm-release-repo:$${BUILDKITE_COMMIT}-rocm \
--target vllm-openai \
--progress plain \
-f docker/Dockerfile.rocm .
- label: "Publish nightly ROCm 10.0 image to DockerHub" # Push to ECR
id: publish-rocm100-nightly docker push public.ecr.aws/q9t5s3a7/vllm-release-repo:$${BUILDKITE_COMMIT}-rocm
depends_on: build-rocm100-release-image
if: build.env("NIGHTLY") == "1"
agents: &rocm_publish_agents
queue: small_cpu_queue_release
env: *rocm100_env
commands: &rocm_publish_nightly_commands
- bash .buildkite/scripts/push-nightly-builds-rocm.sh "$${ROCM_BASE_DOCKERFILE}" $${ROCM_DEFAULT_ARG}
plugins: &rocm_dockerhub_login
- docker-login#v3.0.0:
username: vllmbot
password-env: DOCKERHUB_TOKEN
- group: "Build ROCm 7.2 Wheel / Image" echo ""
key: "build-rocm72" echo " Successfully built and pushed ROCm release image"
depends_on: ~ echo " Image: public.ecr.aws/q9t5s3a7/vllm-release-repo:$${BUILDKITE_COMMIT}-rocm"
steps: echo ""
- label: ":rocm: Build ROCm 7.2 Base Image & Wheels" env:
id: build-rocm72-base
depends_on: ~
agents: *rocm_build_agents
timeout_in_minutes: 360
env: &rocm72_env
ROCM_BASE_DOCKERFILE: "docker/Dockerfile.rocm_72_base"
ROCM_DEFAULT_ARG: ""
DOCKER_BUILDKIT: "1" DOCKER_BUILDKIT: "1"
S3_BUCKET: "vllm-wheels" S3_BUCKET: "vllm-wheels"
DOCKERHUB_USERNAME: "vllmbot"
commands: *rocm_base_commands
- label: ":python: Build vLLM ROCm 7.2 Wheel - x86_64"
id: build-rocm72-vllm-wheel
depends_on: build-rocm72-base
agents: *rocm_build_agents
timeout_in_minutes: 180
env: *rocm72_env
commands: *rocm_vllm_wheel_commands
- label: ":s3: Upload ROCm 7.2 Wheels to S3"
id: upload-rocm72-wheels
depends_on: build-rocm72-vllm-wheel
agents: *rocm_build_agents
timeout_in_minutes: 60
env: *rocm72_env
commands: *rocm_upload_commands
- label: ":memo: Annotate ROCm 7.2 wheel release"
id: annotate-rocm72-release
depends_on: upload-rocm72-wheels
agents: *rocm_build_agents
env: *rocm72_env
commands: *rocm_annotate_commands
- label: ":docker: Build release image - x86_64 - ROCm 7.2"
id: build-rocm72-release-image
depends_on:
- step: block-build-release-images
allow_failure: true
- step: build-rocm72-base
allow_failure: false
agents: *rocm_build_agents
timeout_in_minutes: 180
env: *rocm72_env
commands: *rocm_release_image_commands
- label: "Publish nightly ROCm 7.2 image to DockerHub"
id: publish-rocm72-nightly
depends_on: build-rocm72-release-image
if: build.env("NIGHTLY") == "1"
agents: *rocm_publish_agents
env: *rocm72_env
commands: *rocm_publish_nightly_commands
plugins: *rocm_dockerhub_login
- label: "Publish nightly XPU image to DockerHub" - label: "Publish nightly XPU image to DockerHub"
depends_on: depends_on:
@@ -985,6 +1058,163 @@ steps:
DOCKER_BUILDKIT: "1" DOCKER_BUILDKIT: "1"
DOCKERHUB_USERNAME: "vllmbot" DOCKERHUB_USERNAME: "vllmbot"
- label: "Publish nightly ROCm image to DockerHub"
depends_on:
- build-rocm-release-image
if: build.env("NIGHTLY") == "1"
agents:
queue: small_cpu_queue_release
commands:
- "bash .buildkite/scripts/push-nightly-builds-rocm.sh"
# Clean up old nightly builds (keep only last 14)
- "bash .buildkite/scripts/cleanup-nightly-builds.sh nightly- vllm/vllm-openai-rocm"
- "bash .buildkite/scripts/cleanup-nightly-builds.sh base-nightly- vllm/vllm-openai-rocm"
plugins:
- docker-login#v3.0.0:
username: vllmbot
password-env: DOCKERHUB_TOKEN
env:
DOCKER_BUILDKIT: "1"
DOCKERHUB_USERNAME: "vllmbot"
# =============================================================================
# ROCk Release Pipeline (x86_64 only)
# =============================================================================
#
# Same shape as the ROCm pipeline above (Jobs 1 and 6 only), but built from
# docker/Dockerfile.rock_base and docker/Dockerfile.rock, which install the
# ROCm SDK from TheRock wheels instead of the ROCm apt repos.
#
# Note: ROCm SDK version is determined by ROCM_SDK_VERSION in
# docker/Dockerfile.rock_base
#
# =============================================================================
- group: "Build ROCk Image"
key: "build-rock-image"
depends_on: ~
steps:
# ROCk Job 1: Build ROCk Base Image (with ECR caching)
- label: ":rocm: Build ROCk Base Image"
id: build-rock-base-image
depends_on: ~
agents:
queue: cpu_queue_release
timeout_in_minutes: 360
commands:
- |
set -euo pipefail
# The SDK version, torch wheels and component branches all live in
# the Dockerfile, so its hash is the cache key.
CACHE_KEY=$$(sha256sum docker/Dockerfile.rock_base | cut -c1-16)
ECR_CACHE_TAG="public.ecr.aws/q9t5s3a7/vllm-release-repo:$${CACHE_KEY}-rock-base"
echo "========================================"
echo "ROCk Base Build Configuration"
echo "========================================"
echo " CACHE_KEY: $${CACHE_KEY}"
echo " ECR_CACHE_TAG: $${ECR_CACHE_TAG}"
echo "========================================"
# Login to ECR
aws ecr-public get-login-password --region us-east-1 | \
docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7
if docker manifest inspect "$${ECR_CACHE_TAG}" > /dev/null 2>&1; then
echo "ECR image cache HIT"
else
echo "CACHE MISS - Building from scratch..."
DOCKER_BUILDKIT=1 docker buildx build \
--file docker/Dockerfile.rock_base \
--tag "$${ECR_CACHE_TAG}" \
--build-arg USE_SCCACHE=1 \
--build-arg SCCACHE_BUCKET_NAME=vllm-build-sccache \
--build-arg SCCACHE_REGION_NAME=us-west-2 \
--build-arg SCCACHE_S3_NO_CREDENTIALS=0 \
--push \
.
fi
# Save ECR tag for downstream jobs
buildkite-agent meta-data set "rock-base-image-tag" "$${ECR_CACHE_TAG}"
env:
DOCKER_BUILDKIT: "1"
# ROCk Job 2: Build ROCk Release Docker Image
- label: ":docker: Build release image - x86_64 - ROCk"
id: build-rock-release-image
depends_on:
- step: block-build-release-images
allow_failure: true
- step: build-rock-base-image
allow_failure: false
agents:
queue: cpu_queue_release
timeout_in_minutes: 180
commands:
- |
set -euo pipefail
# Login to ECR
aws ecr-public get-login-password --region us-east-1 | \
docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7
ECR_IMAGE_TAG="$$(buildkite-agent meta-data get rock-base-image-tag 2>/dev/null || echo '')"
if [ -z "$${ECR_IMAGE_TAG}" ]; then
echo "ERROR: rock-base-image-tag metadata not found"
echo "This should have been set by the build-rock-base-image job"
exit 1
fi
docker pull "$${ECR_IMAGE_TAG}"
# Pass the base image ECR tag to downstream steps (nightly publish)
buildkite-agent meta-data set "rock-base-ecr-tag" "$${ECR_IMAGE_TAG}"
echo "========================================"
echo "Building vLLM ROCk release image with:"
echo " BASE_IMAGE: $${ECR_IMAGE_TAG}"
echo " BUILDKITE_COMMIT: $${BUILDKITE_COMMIT}"
echo "========================================"
DOCKER_BUILDKIT=1 docker build \
--build-arg max_jobs=16 \
--build-arg BASE_IMAGE="$${ECR_IMAGE_TAG}" \
--build-arg USE_SCCACHE=1 \
--build-arg SCCACHE_BUCKET_NAME=vllm-build-sccache \
--build-arg SCCACHE_REGION_NAME=us-west-2 \
--build-arg SCCACHE_S3_NO_CREDENTIALS=0 \
--build-arg INSTALL_LMCACHE=true \
--tag public.ecr.aws/q9t5s3a7/vllm-release-repo:$${BUILDKITE_COMMIT}-rock \
--target vllm-openai \
--progress plain \
-f docker/Dockerfile.rock .
docker push public.ecr.aws/q9t5s3a7/vllm-release-repo:$${BUILDKITE_COMMIT}-rock
env:
DOCKER_BUILDKIT: "1"
- label: "Publish nightly ROCk image to DockerHub"
depends_on:
- build-rock-release-image
if: build.env("NIGHTLY") == "1"
agents:
queue: small_cpu_queue_release
commands:
- "bash .buildkite/scripts/push-nightly-builds-rocm.sh rock rocm100"
# Clean up old nightly builds (keep only last 14)
- "bash .buildkite/scripts/cleanup-nightly-builds.sh nightly-rocm100- vllm/vllm-openai-rocm"
- "bash .buildkite/scripts/cleanup-nightly-builds.sh base-nightly-rocm100- vllm/vllm-openai-rocm"
plugins:
- docker-login#v3.0.0:
username: vllmbot
password-env: DOCKERHUB_TOKEN
env:
DOCKER_BUILDKIT: "1"
DOCKERHUB_USERNAME: "vllmbot"
# ============================================================================= # =============================================================================
# Publish to DockerHub and PyPI (at the end so all builds complete first) # Publish to DockerHub and PyPI (at the end so all builds complete first)
# ============================================================================= # =============================================================================
@@ -997,8 +1227,7 @@ steps:
- create-multi-arch-manifest-ubuntu2404 - create-multi-arch-manifest-ubuntu2404
- create-multi-arch-manifest-cuda-12-9-ubuntu2404 - create-multi-arch-manifest-cuda-12-9-ubuntu2404
- create-manifest-xpu - create-manifest-xpu
- build-rocm100-release-image - build-rocm-release-image
- build-rocm72-release-image
- input-release-version - input-release-version
# Wait for CPU builds if their block steps were unblocked, so publish # Wait for CPU builds if their block steps were unblocked, so publish
# doesn't race the in-progress CPU build. allow_failure lets publish # doesn't race the in-progress CPU build. allow_failure lets publish
@@ -1023,7 +1252,6 @@ steps:
- "cuda-13-0-ubuntu-24-04" - "cuda-13-0-ubuntu-24-04"
- "cuda-12-9-ubuntu-24-04" - "cuda-12-9-ubuntu-24-04"
- "rocm" - "rocm"
- "rocm72"
- "xpu" - "xpu"
- "cpu" - "cpu"
plugins: plugins:
+66 -40
View File
@@ -2,23 +2,15 @@
# SPDX-License-Identifier: Apache-2.0 # SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
# #
# Generate Buildkite annotation for a ROCm stack's wheel release. # Generate Buildkite annotation for ROCm wheel release
#
# Usage: annotate-rocm-release.sh <base-dockerfile> [--default]
# base-dockerfile e.g. docker/Dockerfile.rocm_base or docker/Dockerfile.rocm_72_base
# --default the stack also owns the plain :latest / :v<ver> Docker Hub tags
set -ex set -ex
# shellcheck source=.buildkite/scripts/rocm/stack.sh # Extract build configuration from Dockerfile.rocm_base (single source of truth)
source "$(dirname "${BASH_SOURCE[0]}")/rocm/stack.sh" "${1:?Usage: $0 <base-dockerfile> [--default]}" # Extract ROCm version dynamically from Dockerfile.rocm_base
TAG_VARIANTS=("-${ROCM_STACK_VARIANT}") # BASE_IMAGE format: rocm/dev-ubuntu-22.04:7.0-complete -> extracts "7.0"
if [[ "${2:-}" == "--default" ]]; then ROCM_VERSION=$(grep -E '^ARG BASE_IMAGE=' docker/Dockerfile.rocm_base | sed -E 's/.*:([0-9]+\.[0-9]+).*/\1/' || echo "unknown")
TAG_VARIANTS=("" "-${ROCM_STACK_VARIANT}") PYTHON_VERSION=$(grep '^ARG PYTHON_VERSION=' docker/Dockerfile.rocm_base | sed 's/^ARG PYTHON_VERSION=//')
fi PYTORCH_ROCM_ARCH=$(grep '^ARG PYTORCH_ROCM_ARCH=' docker/Dockerfile.rocm_base | sed 's/^ARG PYTORCH_ROCM_ARCH=//')
ROCM_VERSION="$ROCM_STACK_VERSION"
VARIANT="$ROCM_STACK_VARIANT"
PYTHON_VERSION=$(sed -nE 's/^ARG PYTHON_VERSION="?([^" ]+)"?.*/\1/p' "$ROCM_STACK_BASE_DOCKERFILE")
PYTORCH_ROCM_ARCH=$(sed -nE 's/^ARG PYTORCH_ROCM_ARCH="?([^"]+)"?.*/\1/p' "$ROCM_STACK_BASE_DOCKERFILE")
# Get release version, default to 1.0.0.dev for nightly/per-commit builds # Get release version, default to 1.0.0.dev for nightly/per-commit builds
RELEASE_VERSION=$(buildkite-agent meta-data get release-version 2>/dev/null || echo "") RELEASE_VERSION=$(buildkite-agent meta-data get release-version 2>/dev/null || echo "")
@@ -26,32 +18,22 @@ if [ -z "${RELEASE_VERSION}" ]; then
RELEASE_VERSION="1.0.0.dev" RELEASE_VERSION="1.0.0.dev"
fi fi
ROCM_BASE_CACHE_KEY=$(.buildkite/scripts/cache-rocm-base-wheels.sh key)
# S3 URLs # S3 URLs
S3_BUCKET="${S3_BUCKET:-vllm-wheels}" S3_BUCKET="${S3_BUCKET:-vllm-wheels}"
S3_REGION="${AWS_DEFAULT_REGION:-us-west-2}" S3_REGION="${AWS_DEFAULT_REGION:-us-west-2}"
S3_HOST="${S3_BUCKET}.s3-website-${S3_REGION}.amazonaws.com" S3_URL="http://${S3_BUCKET}.s3-website-${S3_REGION}.amazonaws.com"
S3_URL="http://${S3_HOST}"
WHEEL_PATH="rocm/${BUILDKITE_COMMIT}/${VARIANT}-wheels/"
DOCKER_CMDS="docker pull ${ROCM_STACK_ECR_BASE} # Format ROCm version for path (e.g., "7.1" -> "rocm710")
docker pull ${ROCM_STACK_ECR_IMAGE}" ROCM_VERSION_PATH="rocm$(echo "${ROCM_VERSION}" | tr -d '.')"
for suffix in "${TAG_VARIANTS[@]}"; do ROCM_PATH="rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}"
for tag in "latest${suffix}" "v${RELEASE_VERSION}${suffix}"; do buildkite-agent annotate --style 'success' --context 'rocm-release-workflow' << EOF
DOCKER_CMDS+=" ## ROCm Wheel and Docker Image Releases
docker tag ${ROCM_STACK_ECR_BASE} vllm/vllm-openai-rocm:${tag}-base
docker tag ${ROCM_STACK_ECR_IMAGE} vllm/vllm-openai-rocm:${tag}
docker push vllm/vllm-openai-rocm:${tag}-base
docker push vllm/vllm-openai-rocm:${tag}"
done
done
buildkite-agent annotate --style 'success' --context "rocm-release-workflow-${VARIANT}" << EOF
## ROCm ${ROCM_VERSION} Wheel and Docker Image Releases
### Build Configuration ### Build Configuration
| Setting | Value | | Setting | Value |
|---------|-------| |---------|-------|
| **ROCm Version** | ${ROCM_VERSION} | | **ROCm Version** | ${ROCM_VERSION} |
| **Index Variant** | ${VARIANT} |
| **Python Version** | ${PYTHON_VERSION} | | **Python Version** | ${PYTHON_VERSION} |
| **GPU Architectures** | ${PYTORCH_ROCM_ARCH} | | **GPU Architectures** | ${PYTORCH_ROCM_ARCH} |
| **Branch** | \`${BUILDKITE_BRANCH}\` | | **Branch** | \`${BUILDKITE_BRANCH}\` |
@@ -59,20 +41,49 @@ buildkite-agent annotate --style 'success' --context "rocm-release-workflow-${VA
### :package: Installation ### :package: Installation
**Install from this build (by commit):**
\`\`\`bash \`\`\`bash
# This build (by commit) pip install vllm --extra-index-url ${S3_URL}/${ROCM_PATH}/ --trusted-host ${S3_BUCKET}.s3-website-${S3_REGION}.amazonaws.com
pip install vllm --extra-index-url ${S3_URL}/rocm/${BUILDKITE_COMMIT}/${VARIANT}/ --trusted-host ${S3_HOST}
# Nightly (if published) # Example for ROCm ${ROCM_VERSION}:
pip install vllm --extra-index-url ${S3_URL}/rocm/nightly/${VARIANT}/ --trusted-host ${S3_HOST} pip install vllm --extra-index-url ${S3_URL}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/ --trusted-host ${S3_BUCKET}.s3-website-${S3_REGION}.amazonaws.com
\`\`\`
**Install from nightly (if published):**
\`\`\`bash
pip install vllm --extra-index-url ${S3_URL}/rocm/nightly/ --trusted-host ${S3_BUCKET}.s3-website-${S3_REGION}.amazonaws.com
\`\`\` \`\`\`
### :floppy_disk: Download Wheels Directly ### :floppy_disk: Download Wheels Directly
\`\`\`bash \`\`\`bash
aws s3 ls s3://${S3_BUCKET}/${WHEEL_PATH} # List all ROCm wheels
aws s3 cp --recursive --exclude "*" --include "*.whl" s3://${S3_BUCKET}/${WHEEL_PATH} . aws s3 ls s3://${S3_BUCKET}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/
# Download specific wheels
aws s3 cp s3://${S3_BUCKET}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/vllm-*.whl .
aws s3 cp s3://${S3_BUCKET}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/torch-*.whl .
aws s3 cp s3://${S3_BUCKET}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/triton-*.whl .
aws s3 cp s3://${S3_BUCKET}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/triton-kernels-*.whl .
aws s3 cp s3://${S3_BUCKET}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/torchvision-*.whl .
aws s3 cp s3://${S3_BUCKET}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/torchaudio-*.whl .
aws s3 cp s3://${S3_BUCKET}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/amdsmi-*.whl .
aws s3 cp s3://${S3_BUCKET}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/amd_aiter-*.whl .
aws s3 cp s3://${S3_BUCKET}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/flash-attn-*.whl .
\`\`\` \`\`\`
### :gear: Included Packages
- **vllm**: vLLM with ROCm support
- **torch**: PyTorch built for ROCm ${ROCM_VERSION}
- **triton**: Triton
- **triton-kernels**: Triton kernels
- **torchvision**: TorchVision for ROCm PyTorch
- **torchaudio**: Torchaudio for ROCm PyTorch
- **amdsmi**: AMD SMI Python bindings
- **amd_aiter**: Aiter for ROCm
- **flash-attn**: Flash Attention for ROCm
### :warning: Notes ### :warning: Notes
- These wheels are built for **ROCm ${ROCM_VERSION}** and will NOT work with CUDA GPUs - These wheels are built for **ROCm ${ROCM_VERSION}** and will NOT work with CUDA GPUs
- Supported GPU architectures: ${PYTORCH_ROCM_ARCH} - Supported GPU architectures: ${PYTORCH_ROCM_ARCH}
@@ -80,8 +91,23 @@ aws s3 cp --recursive --exclude "*" --include "*.whl" s3://${S3_BUCKET}/${WHEEL_
### :package: Docker Image Release ### :package: Docker Image Release
To download and upload the image:
\`\`\` \`\`\`
${DOCKER_CMDS} docker pull public.ecr.aws/q9t5s3a7/vllm-release-repo:${BUILDKITE_COMMIT}-rocm-base
docker pull public.ecr.aws/q9t5s3a7/vllm-release-repo:${BUILDKITE_COMMIT}-rocm
docker tag public.ecr.aws/q9t5s3a7/vllm-release-repo:${ROCM_BASE_CACHE_KEY}-rocm-base vllm/vllm-openai-rocm:${BUILDKITE_COMMIT}-base
docker tag vllm/vllm-openai-rocm:${BUILDKITE_COMMIT}-base vllm/vllm-openai-rocm:latest-base
docker tag vllm/vllm-openai-rocm:${BUILDKITE_COMMIT}-base vllm/vllm-openai-rocm:v${RELEASE_VERSION}-base
docker push vllm/vllm-openai-rocm:latest-base
docker push vllm/vllm-openai-rocm:v${RELEASE_VERSION}-base
docker tag public.ecr.aws/q9t5s3a7/vllm-release-repo:${BUILDKITE_COMMIT}-rocm vllm/vllm-openai-rocm:${BUILDKITE_COMMIT}
docker tag vllm/vllm-openai-rocm:${BUILDKITE_COMMIT} vllm/vllm-openai-rocm:latest
docker tag vllm/vllm-openai-rocm:${BUILDKITE_COMMIT} vllm/vllm-openai-rocm:v${RELEASE_VERSION}
docker push vllm/vllm-openai-rocm:latest
docker push vllm/vllm-openai-rocm:v${RELEASE_VERSION}
\`\`\` \`\`\`
EOF EOF
+1 -2
View File
@@ -25,7 +25,6 @@ export CMAKE_BUILD_PARALLEL_LEVEL="${CMAKE_BUILD_PARALLEL_LEVEL:-4}"
uv venv --python 3.12 uv venv --python 3.12
uv pip install -r requirements/build/cpu.txt --index-strategy unsafe-best-match uv pip install -r requirements/build/cpu.txt --index-strategy unsafe-best-match
uv build --wheel --no-build-isolation -o artifacts/dist \ uv build --wheel --no-build-isolation -o artifacts/dist
--config-setting=--build-option=--py-limited-api=cp38
ls -l artifacts/dist/*.whl ls -l artifacts/dist/*.whl
+34 -28
View File
@@ -5,33 +5,39 @@
# Cache helper for ROCm base wheels # Cache helper for ROCm base wheels
# #
# This script manages caching of pre-built ROCm base wheels (torch, triton, etc.) # This script manages caching of pre-built ROCm base wheels (torch, triton, etc.)
# to avoid rebuilding them when the stack's base Dockerfile hasn't changed. # to avoid rebuilding them when Dockerfile.rocm_base hasn't changed.
# #
# Usage: cache-rocm-base-wheels.sh <base-dockerfile> <command> # Usage:
# base-dockerfile e.g. docker/Dockerfile.rocm_base or docker/Dockerfile.rocm_72_base # cache-rocm-base-wheels.sh check - Check if cache exists, outputs "hit" or "miss"
# Commands: # cache-rocm-base-wheels.sh upload - Upload wheels to cache
# check - Check if cache exists, outputs "hit" or "miss" # cache-rocm-base-wheels.sh download - Download wheels from cache
# upload - Upload wheels to cache # cache-rocm-base-wheels.sh key - Output the cache key
# download - Download wheels from cache
# key - Output the cache key
# path - Output the full S3 cache path
# #
# Environment variables: # Environment variables:
# S3_BUCKET - S3 bucket name (default: vllm-wheels) # S3_BUCKET - S3 bucket name (default: vllm-wheels)
# #
# The cache key hashes the base Dockerfile and the repo files it COPYs in, which # Note: ROCm version is determined by BASE_IMAGE in Dockerfile.rocm_base,
# capture the ROCm version and every component pin (see rocm/stack.sh). # so changes to ROCm version are captured by the Dockerfile hash.
set -euo pipefail set -euo pipefail
BUCKET="${S3_BUCKET:-vllm-wheels}" BUCKET="${S3_BUCKET:-vllm-wheels}"
DOCKERFILE="docker/Dockerfile.rocm_base"
CACHE_PREFIX="rocm/cache" CACHE_PREFIX="rocm/cache"
# shellcheck source=.buildkite/scripts/rocm/stack.sh # Generate hash from Dockerfile content + build args
source "$(dirname "${BASH_SOURCE[0]}")/rocm/stack.sh" "${1:?Usage: $0 <base-dockerfile> <command>}" generate_cache_key() {
shift # Include Dockerfile content
CACHE_KEY="$ROCM_STACK_WHEEL_KEY" if [[ ! -f "$DOCKERFILE" ]]; then
WHEELS_DIR="$ROCM_STACK_BASE_WHEELS_DIR" echo "ERROR: Dockerfile not found: $DOCKERFILE" >&2
exit 1
fi
local dockerfile_hash=$(sha256sum "$DOCKERFILE" | cut -c1-16)
echo "${dockerfile_hash}"
}
CACHE_KEY=$(generate_cache_key)
CACHE_PATH="s3://${BUCKET}/${CACHE_PREFIX}/${CACHE_KEY}/" CACHE_PATH="s3://${BUCKET}/${CACHE_PREFIX}/${CACHE_KEY}/"
case "${1:-}" in case "${1:-}" in
@@ -61,19 +67,19 @@ case "${1:-}" in
echo "Cache path: ${CACHE_PATH}" echo "Cache path: ${CACHE_PATH}"
echo "" echo ""
if [[ ! -d "${WHEELS_DIR}" ]]; then if [[ ! -d "artifacts/rocm-base-wheels" ]]; then
echo "ERROR: ${WHEELS_DIR} directory not found" >&2 echo "ERROR: artifacts/rocm-base-wheels directory not found" >&2
exit 1 exit 1
fi fi
WHEEL_COUNT=$(find "${WHEELS_DIR}" -maxdepth 1 -name '*.whl' 2>/dev/null | wc -l) WHEEL_COUNT=$(find artifacts/rocm-base-wheels -maxdepth 1 -name '*.whl' 2>/dev/null | wc -l)
if [[ "$WHEEL_COUNT" -eq 0 ]]; then if [[ "$WHEEL_COUNT" -eq 0 ]]; then
echo "ERROR: No wheels found in ${WHEELS_DIR}/" >&2 echo "ERROR: No wheels found in artifacts/rocm-base-wheels/" >&2
exit 1 exit 1
fi fi
echo "Uploading $WHEEL_COUNT wheels..." echo "Uploading $WHEEL_COUNT wheels..."
aws s3 cp --recursive "${WHEELS_DIR}/" "${CACHE_PATH}" aws s3 cp --recursive artifacts/rocm-base-wheels/ "${CACHE_PATH}"
echo "" echo ""
echo "Cache upload complete!" echo "Cache upload complete!"
@@ -87,17 +93,17 @@ case "${1:-}" in
echo "Cache key: ${CACHE_KEY}" echo "Cache key: ${CACHE_KEY}"
echo "Cache path: ${CACHE_PATH}" echo "Cache path: ${CACHE_PATH}"
echo "" echo ""
mkdir -p "${WHEELS_DIR}" mkdir -p artifacts/rocm-base-wheels
# Use sync with include/exclude to only download .whl files # Use sync with include/exclude to only download .whl files
aws s3 sync "${CACHE_PATH}" "${WHEELS_DIR}/" \ aws s3 sync "${CACHE_PATH}" artifacts/rocm-base-wheels/ \
--exclude "*" \ --exclude "*" \
--include "*.whl" --include "*.whl"
echo "" echo ""
echo "Downloaded wheels:" echo "Downloaded wheels:"
find "${WHEELS_DIR}" -maxdepth 1 -name '*.whl' -exec ls -lh {} \; find artifacts/rocm-base-wheels -maxdepth 1 -name '*.whl' -exec ls -lh {} \;
WHEEL_COUNT=$(find "${WHEELS_DIR}" -maxdepth 1 -name '*.whl' 2>/dev/null | wc -l) WHEEL_COUNT=$(find artifacts/rocm-base-wheels -maxdepth 1 -name '*.whl' 2>/dev/null | wc -l)
echo "" echo ""
echo "Total: $WHEEL_COUNT wheels" echo "Total: $WHEEL_COUNT wheels"
echo "========================================" echo "========================================"
@@ -112,12 +118,12 @@ case "${1:-}" in
;; ;;
*) *)
echo "Usage: $0 <base-dockerfile> {check|upload|download|key|path}" >&2 echo "Usage: $0 {check|upload|download|key|path}" >&2
echo "" >&2 echo "" >&2
echo "Commands:" >&2 echo "Commands:" >&2
echo " check - Check if cache exists, outputs 'hit' or 'miss'" >&2 echo " check - Check if cache exists, outputs 'hit' or 'miss'" >&2
echo " upload - Upload wheels from ${WHEELS_DIR}/ to cache" >&2 echo " upload - Upload wheels from artifacts/rocm-base-wheels/ to cache" >&2
echo " download - Download wheels from cache to ${WHEELS_DIR}/" >&2 echo " download - Download wheels from cache to artifacts/rocm-base-wheels/" >&2
echo " key - Output the cache key" >&2 echo " key - Output the cache key" >&2
echo " path - Output the full S3 cache path" >&2 echo " path - Output the full S3 cache path" >&2
exit 1 exit 1
+54 -7
View File
@@ -17,7 +17,7 @@ DEFAULT_REPO_SLUG="vllm-project/vllm"
DEFAULT_CI_HCL_SOURCE="docker/ci-rocm.hcl" DEFAULT_CI_HCL_SOURCE="docker/ci-rocm.hcl"
DEFAULT_CI_BASE_CONTENT_FILES=".dockerignore requirements/common.txt requirements/rocm.txt requirements/test/rocm.txt tools/install_torchcodec_rocm.sh rust-toolchain.toml tests/vllm_test_utils" DEFAULT_CI_BASE_CONTENT_FILES=".dockerignore requirements/common.txt requirements/rocm.txt requirements/test/rocm.txt tools/install_torchcodec_rocm.sh rust-toolchain.toml tests/vllm_test_utils"
DEFAULT_CI_BASE_DOCKERFILE="docker/Dockerfile.rocm" DEFAULT_CI_BASE_DOCKERFILE="docker/Dockerfile.rocm"
DEFAULT_CI_BASE_DOCKERFILE_STAGES="base rust_toolchain_input_0 rust-toolchain-input rust-toolchain build_nixl lmcache_source build_lmcache build_deepep mori_base ci_base" DEFAULT_CI_BASE_DOCKERFILE_STAGES="base rust_toolchain_input_0 rust-toolchain-input rust-toolchain build_nixl lmcache_source build_lmcache build_rocshmem build_deepep mori_base ci_base"
DEFAULT_CI_BASE_METADATA_VERSION="3" DEFAULT_CI_BASE_METADATA_VERSION="3"
# ROCm CI forces REMOTE_VLLM=0, so content identity covers only the selected # ROCm CI forces REMOTE_VLLM=0, so content identity covers only the selected
# local-source stages rather than unreachable remote-fetch alternatives. # local-source stages rather than unreachable remote-fetch alternatives.
@@ -555,6 +555,9 @@ configure_custom_rocm_stages() {
CI_BASE_DOCKERFILE_STAGES="${CI_BASE_DOCKERFILE_STAGES:-${stages}}" CI_BASE_DOCKERFILE_STAGES="${CI_BASE_DOCKERFILE_STAGES:-${stages}}"
ROCM_CSRC_DOCKERFILE_STAGES="${ROCM_CSRC_DOCKERFILE_STAGES:-${stages}}" ROCM_CSRC_DOCKERFILE_STAGES="${ROCM_CSRC_DOCKERFILE_STAGES:-${stages}}"
ROCM_RUST_DOCKERFILE_STAGES="${ROCM_RUST_DOCKERFILE_STAGES:-${stages}}" ROCM_RUST_DOCKERFILE_STAGES="${ROCM_RUST_DOCKERFILE_STAGES:-${stages}}"
if [[ " ${stages} " != *" build_rocshmem "* ]]; then
unset ROCSHMEM_BRANCH ROCSHMEM_CACHE_KEY DEEPEP_CACHE_KEY
fi
} }
compose_dependency_cache_key() { compose_dependency_cache_key() {
@@ -1721,11 +1724,14 @@ ci_base_metadata_pairs() {
metadata_pair "vllm.rocm.nixl_commit" "${NIXL_BRANCH:-$(resolve_dockerfile_arg_value "${dockerfile}" "NIXL_BRANCH")}" metadata_pair "vllm.rocm.nixl_commit" "${NIXL_BRANCH:-$(resolve_dockerfile_arg_value "${dockerfile}" "NIXL_BRANCH")}"
metadata_pair "vllm.rocm.ucx_repo" "$(resolve_dockerfile_arg_value "${dockerfile}" "UCX_REPO")" metadata_pair "vllm.rocm.ucx_repo" "$(resolve_dockerfile_arg_value "${dockerfile}" "UCX_REPO")"
metadata_pair "vllm.rocm.ucx_commit" "${UCX_BRANCH:-$(resolve_dockerfile_arg_value "${dockerfile}" "UCX_BRANCH")}" metadata_pair "vllm.rocm.ucx_commit" "${UCX_BRANCH:-$(resolve_dockerfile_arg_value "${dockerfile}" "UCX_BRANCH")}"
metadata_pair "vllm.rocm.rocshmem_repo" "$(resolve_dockerfile_arg_value "${dockerfile}" "ROCSHMEM_REPO")"
metadata_pair "vllm.rocm.rocshmem_commit" "${ROCSHMEM_BRANCH:-$(resolve_dockerfile_arg_value "${dockerfile}" "ROCSHMEM_BRANCH")}"
metadata_pair "vllm.rocm.deepep_repo" "$(resolve_dockerfile_arg_value "${dockerfile}" "DEEPEP_REPO")" metadata_pair "vllm.rocm.deepep_repo" "$(resolve_dockerfile_arg_value "${dockerfile}" "DEEPEP_REPO")"
metadata_pair "vllm.rocm.deepep_commit" "${DEEPEP_BRANCH:-$(resolve_dockerfile_arg_value "${dockerfile}" "DEEPEP_BRANCH")}" metadata_pair "vllm.rocm.deepep_commit" "${DEEPEP_BRANCH:-$(resolve_dockerfile_arg_value "${dockerfile}" "DEEPEP_BRANCH")}"
metadata_pair "vllm.rocm.deepep_nic" "$(resolve_dockerfile_arg_value "${dockerfile}" "DEEPEP_NIC")" metadata_pair "vllm.rocm.deepep_nic" "$(resolve_dockerfile_arg_value "${dockerfile}" "DEEPEP_NIC")"
metadata_pair "vllm.rocm.deepep_rocm_arch" "$(resolve_dockerfile_arg_value "${dockerfile}" "DEEPEP_ROCM_ARCH")" metadata_pair "vllm.rocm.deepep_rocm_arch" "$(resolve_dockerfile_arg_value "${dockerfile}" "DEEPEP_ROCM_ARCH")"
metadata_pair "vllm.rocm.nixl_cache_key" "${NIXL_CACHE_KEY:-}" metadata_pair "vllm.rocm.nixl_cache_key" "${NIXL_CACHE_KEY:-}"
metadata_pair "vllm.rocm.rocshmem_cache_key" "${ROCSHMEM_CACHE_KEY:-}"
metadata_pair "vllm.rocm.deepep_cache_key" "${DEEPEP_CACHE_KEY:-}" metadata_pair "vllm.rocm.deepep_cache_key" "${DEEPEP_CACHE_KEY:-}"
} }
@@ -2308,7 +2314,7 @@ extract_dependency_pins() {
return 0 return 0
fi fi
for var in NIXL_BRANCH UCX_BRANCH DEEPEP_BRANCH; do for var in NIXL_BRANCH UCX_BRANCH ROCSHMEM_BRANCH DEEPEP_BRANCH; do
if [[ -n "${!var:-}" ]]; then if [[ -n "${!var:-}" ]]; then
echo "Using provided ${var}: ${!var}" echo "Using provided ${var}: ${!var}"
continue continue
@@ -2330,14 +2336,17 @@ compute_dependency_cache_keys() {
local dockerfile_rocm="" local dockerfile_rocm=""
local nixl_branch="" local nixl_branch=""
local ucx_branch="" local ucx_branch=""
local rocshmem_branch=""
local deepep_branch="" local deepep_branch=""
local nixl_material="" local nixl_material=""
local rocshmem_material=""
local deepep_material="" local deepep_material=""
bake_dir=$(dirname "${VLLM_BAKE_FILE}") bake_dir=$(dirname "${VLLM_BAKE_FILE}")
dockerfile_rocm="${CI_BASE_DOCKERFILE:-${bake_dir}/Dockerfile.rocm}" dockerfile_rocm="${CI_BASE_DOCKERFILE:-${bake_dir}/Dockerfile.rocm}"
nixl_branch=$(resolve_dockerfile_arg_value "${dockerfile_rocm}" "NIXL_BRANCH") nixl_branch=$(resolve_dockerfile_arg_value "${dockerfile_rocm}" "NIXL_BRANCH")
ucx_branch=$(resolve_dockerfile_arg_value "${dockerfile_rocm}" "UCX_BRANCH") ucx_branch=$(resolve_dockerfile_arg_value "${dockerfile_rocm}" "UCX_BRANCH")
rocshmem_branch=$(resolve_dockerfile_arg_value "${dockerfile_rocm}" "ROCSHMEM_BRANCH")
deepep_branch=$(resolve_dockerfile_arg_value "${dockerfile_rocm}" "DEEPEP_BRANCH") deepep_branch=$(resolve_dockerfile_arg_value "${dockerfile_rocm}" "DEEPEP_BRANCH")
if [[ -n "${nixl_branch}" && -n "${ucx_branch}" ]]; then if [[ -n "${nixl_branch}" && -n "${ucx_branch}" ]]; then
@@ -2351,11 +2360,22 @@ compute_dependency_cache_keys() {
echo "NIXL dependency cache key: ${NIXL_CACHE_KEY}" echo "NIXL dependency cache key: ${NIXL_CACHE_KEY}"
fi fi
if [[ -n "${deepep_branch}" ]]; then if [[ -n "${rocshmem_branch}" ]]; then
deepep_material=$(compose_stage_cache_material "${dockerfile_rocm}" "base build_deepep") rocshmem_material=$(compose_stage_cache_material "${dockerfile_rocm}" "base build_rocshmem")
ROCSHMEM_CACHE_KEY=$(
compose_dependency_cache_key \
"${rocshmem_branch}" \
"${rocshmem_material}"
)
export ROCSHMEM_CACHE_KEY
echo "ROCShmem dependency cache key: ${ROCSHMEM_CACHE_KEY}"
fi
if [[ -n "${deepep_branch}" && -n "${rocshmem_branch}" ]]; then
deepep_material=$(compose_stage_cache_material "${dockerfile_rocm}" "base build_rocshmem build_deepep")
DEEPEP_CACHE_KEY=$( DEEPEP_CACHE_KEY=$(
compose_dependency_cache_key \ compose_dependency_cache_key \
"${deepep_branch}" \ "${deepep_branch}-rocshmem-${rocshmem_branch}" \
"${deepep_material}" "${deepep_material}"
) )
export DEEPEP_CACHE_KEY export DEEPEP_CACHE_KEY
@@ -2395,11 +2415,18 @@ dependency_cache_ref_for_target() {
printf '%s\n' "${cache_repo}:nixl-rocm-${NIXL_BRANCH}-ucx-${UCX_BRANCH:-}" printf '%s\n' "${cache_repo}:nixl-rocm-${NIXL_BRANCH}-ucx-${UCX_BRANCH:-}"
fi fi
;; ;;
rocshmem-rocm-ci)
if [[ -n "${ROCSHMEM_CACHE_KEY:-}" ]]; then
printf '%s\n' "${cache_repo}:rocshmem-rocm-${ROCSHMEM_CACHE_KEY}"
elif [[ -n "${ROCSHMEM_BRANCH:-}" ]]; then
printf '%s\n' "${cache_repo}:rocshmem-rocm-${ROCSHMEM_BRANCH}"
fi
;;
deepep-rocm-ci) deepep-rocm-ci)
if [[ -n "${DEEPEP_CACHE_KEY:-}" ]]; then if [[ -n "${DEEPEP_CACHE_KEY:-}" ]]; then
printf '%s\n' "${cache_repo}:deepep-rocm-${DEEPEP_CACHE_KEY}" printf '%s\n' "${cache_repo}:deepep-rocm-${DEEPEP_CACHE_KEY}"
elif [[ -n "${DEEPEP_BRANCH:-}" ]]; then elif [[ -n "${DEEPEP_BRANCH:-}" ]]; then
printf '%s\n' "${cache_repo}:deepep-rocm-${DEEPEP_BRANCH}" printf '%s\n' "${cache_repo}:deepep-rocm-${DEEPEP_BRANCH}-rocshmem-${ROCSHMEM_BRANCH:-}"
fi fi
;; ;;
esac esac
@@ -2417,6 +2444,7 @@ add_dependency_cache_target() {
resolve_ci_base_dependency_targets() { resolve_ci_base_dependency_targets() {
local mode="${ROCM_DEP_CACHE_EXPORT_MODE:-missing}" local mode="${ROCM_DEP_CACHE_EXPORT_MODE:-missing}"
local nixl_ref="" local nixl_ref=""
local rocshmem_ref=""
local deepep_ref="" local deepep_ref=""
[[ "${TARGET}" == "ci-base-rocm-ci-with-deps" ]] || return 0 [[ "${TARGET}" == "ci-base-rocm-ci-with-deps" ]] || return 0
@@ -2424,7 +2452,7 @@ resolve_ci_base_dependency_targets() {
case "${mode}" in case "${mode}" in
always) always)
echo "ROCM_DEP_CACHE_EXPORT_MODE=always; exporting all dependency caches serially" echo "ROCM_DEP_CACHE_EXPORT_MODE=always; exporting all dependency caches serially"
for target in nixl-rocm-ci deepep-rocm-ci; do for target in nixl-rocm-ci rocshmem-rocm-ci deepep-rocm-ci; do
if [[ -n "$(dependency_cache_ref_for_target "${target}")" ]]; then if [[ -n "$(dependency_cache_ref_for_target "${target}")" ]]; then
add_dependency_cache_target "${target}" add_dependency_cache_target "${target}"
fi fi
@@ -2454,6 +2482,16 @@ resolve_ci_base_dependency_targets() {
fi fi
fi fi
if [[ "${mode}" != "always" && -n "${ROCSHMEM_CACHE_KEY:-}" ]]; then
rocshmem_ref=$(dependency_cache_ref_for_target "rocshmem-rocm-ci")
if dependency_cache_ref_exists "${rocshmem_ref}"; then
echo "ROCShmem dependency cache exists: ${rocshmem_ref}"
else
echo "ROCShmem dependency cache missing; will seed: ${rocshmem_ref}"
add_dependency_cache_target "rocshmem-rocm-ci"
fi
fi
if [[ "${mode}" != "always" && -n "${DEEPEP_CACHE_KEY:-}" ]]; then if [[ "${mode}" != "always" && -n "${DEEPEP_CACHE_KEY:-}" ]]; then
deepep_ref=$(dependency_cache_ref_for_target "deepep-rocm-ci") deepep_ref=$(dependency_cache_ref_for_target "deepep-rocm-ci")
if dependency_cache_ref_exists "${deepep_ref}"; then if dependency_cache_ref_exists "${deepep_ref}"; then
@@ -2464,6 +2502,15 @@ resolve_ci_base_dependency_targets() {
fi fi
fi fi
# DeepEP inherits from ROCShmem. If ROCShmem is being seeded, seed DeepEP too
# so the pair stays consistent for future ci_base rebuilds.
if printf '%s\n' "${DEPENDENCY_CACHE_TARGETS[@]}" | grep -qx "rocshmem-rocm-ci" \
&& ! printf '%s\n' "${DEPENDENCY_CACHE_TARGETS[@]}" | grep -qx "deepep-rocm-ci" \
&& [[ -n "${DEEPEP_BRANCH:-}" ]]; then
echo "ROCShmem cache is missing; also seeding DeepEP cache"
add_dependency_cache_target "deepep-rocm-ci"
fi
BAKE_TARGETS=("ci-base-rocm-ci") BAKE_TARGETS=("ci-base-rocm-ci")
if [[ ${#DEPENDENCY_CACHE_TARGETS[@]} -eq 0 ]]; then if [[ ${#DEPENDENCY_CACHE_TARGETS[@]} -eq 0 ]]; then
echo "All dependency caches exist; building ci_base without dependency cache exports" echo "All dependency caches exist; building ci_base without dependency cache exports"
+2 -5
View File
@@ -5,17 +5,14 @@ set -ex
# Clean up old nightly builds from DockerHub, keeping only the last 14 builds # Clean up old nightly builds from DockerHub, keeping only the last 14 builds
# This script uses DockerHub API to list and delete old tags with specified prefix # This script uses DockerHub API to list and delete old tags with specified prefix
# Tags starting with "nightly-dev" are always excluded from deletion # Tags starting with "nightly-dev" are always excluded from deletion
# Usage: cleanup-nightly-builds.sh [TAG_PREFIX] [REPO] [EXCLUDE_PREFIX] # Usage: cleanup-nightly-builds.sh [TAG_PREFIX] [REPO]
# Example: cleanup-nightly-builds.sh "nightly-" # Example: cleanup-nightly-builds.sh "nightly-"
# Example: cleanup-nightly-builds.sh "cu130-nightly-" # Example: cleanup-nightly-builds.sh "cu130-nightly-"
# Example: cleanup-nightly-builds.sh "nightly-" "vllm/vllm-openai-rocm" # Example: cleanup-nightly-builds.sh "nightly-" "vllm/vllm-openai-rocm"
# Example: cleanup-nightly-builds.sh "nightly-" "vllm/vllm-openai-rocm" "nightly-rocm"
# (EXCLUDE_PREFIX skips tags owned by another variant sharing TAG_PREFIX)
# Get tag prefix and repo from arguments # Get tag prefix and repo from arguments
TAG_PREFIX="${1:-nightly-}" TAG_PREFIX="${1:-nightly-}"
REPO="${2:-vllm/vllm-openai}" REPO="${2:-vllm/vllm-openai}"
EXCLUDE_PREFIX="${3:-}"
echo "Cleaning up tags with prefix: $TAG_PREFIX in repository: $REPO" echo "Cleaning up tags with prefix: $TAG_PREFIX in repository: $REPO"
@@ -60,7 +57,7 @@ get_all_tags() {
# Get both last_updated timestamp and tag name, separated by | # Get both last_updated timestamp and tag name, separated by |
# Exclude mutable architecture tags and nightly-dev tags from cleanup # Exclude mutable architecture tags and nightly-dev tags from cleanup
local tags=$(echo "$response" | jq -r --arg prefix "$TAG_PREFIX" --arg exclude "$EXCLUDE_PREFIX" '.results[] | select(.name | startswith($prefix)) | select($exclude == "" or (.name | startswith($exclude) | not)) | select(.name != ($prefix + "x86_64") and .name != ($prefix + "aarch64")) | select(.name | startswith("nightly-dev") | not) | "\(.last_updated)|\(.name)"') local tags=$(echo "$response" | jq -r --arg prefix "$TAG_PREFIX" '.results[] | select(.name | startswith($prefix)) | select(.name != ($prefix + "x86_64") and .name != ($prefix + "aarch64")) | select(.name | startswith("nightly-dev") | not) | "\(.last_updated)|\(.name)"')
if [ -z "$tags" ]; then if [ -z "$tags" ]; then
break break
+1 -6
View File
@@ -227,12 +227,7 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# five-minute lifetime and the loop above can run to the build-age deadline. # five-minute lifetime and the loop above can run to the build-age deadline.
# Build 91827 waited 6h48m and then took a 401 on all 360 callbacks from a # Build 91827 waited 6h48m and then took a 401 on all 360 callbacks from a
# token that had expired hours earlier. # token that had expired hours earlier.
# Posting is paced under the relay's rate limit, which takes several minutes OIDC_TOKEN="$(buildkite-agent oidc request-token --skip-redaction --audience "${AUDIENCE}" 2>/dev/null)"
# for a full nightly, so ask for a longer lifetime than the default. Fall back
# to the default if this agent rejects --lifetime.
OIDC_TOKEN="$(buildkite-agent oidc request-token --skip-redaction --audience "${AUDIENCE}" \
--lifetime 1800 2>/dev/null \
|| buildkite-agent oidc request-token --skip-redaction --audience "${AUDIENCE}" 2>/dev/null)"
if [[ -z "${OIDC_TOKEN}" ]]; then if [[ -z "${OIDC_TOKEN}" ]]; then
echo "could not mint a Buildkite OIDC token after the wait -- skipping report" echo "could not mint a Buildkite OIDC token after the wait -- skipping report"
exit 0 exit 0
-16
View File
@@ -24,9 +24,7 @@ from __future__ import annotations
import argparse import argparse
import json import json
import os
import sys import sys
import time
import urllib.error import urllib.error
import urllib.request import urllib.request
@@ -45,14 +43,6 @@ _CONCLUSION = {
} }
# The relay rate-limits callbacks per repo over a sliding 60 s window
# (RATE_LIMIT_PER_MIN in pytorch/test-infra), and a nightly has far more jobs
# than that. Pace below the limit, and back off on a 429 for about a minute in
# total so the last retry lands in a new window.
_POST_INTERVAL_S = float(os.environ.get("CRCR_POST_INTERVAL_S", "0.65"))
_RATE_LIMIT_BACKOFF_S = (2, 4, 8, 16, 32)
def post(url: str, token: str, body: dict, timeout: int = 30) -> tuple[int, str]: def post(url: str, token: str, body: dict, timeout: int = 30) -> tuple[int, str]:
req = urllib.request.Request( req = urllib.request.Request(
url, url,
@@ -123,12 +113,6 @@ def main() -> int:
ok += 1 ok += 1
continue continue
status, detail = post(args.callback_url, args.oidc_token, body) status, detail = post(args.callback_url, args.oidc_token, body)
for delay in _RATE_LIMIT_BACKOFF_S:
if status != 429:
break
time.sleep(delay)
status, detail = post(args.callback_url, args.oidc_token, body)
time.sleep(_POST_INTERVAL_S)
if 200 <= status < 300: if 200 <= status < 300:
ok += 1 ok += 1
else: else:
+9 -63
View File
@@ -12,7 +12,7 @@ from dataclasses import asdict, dataclass
from datetime import datetime from datetime import datetime
from pathlib import Path from pathlib import Path
from typing import Any from typing import Any
from urllib.parse import quote, unquote, urlparse from urllib.parse import quote
import regex as re import regex as re
@@ -51,8 +51,6 @@ class WheelFileInfo:
platform_tag: str platform_tag: str
variant: str | None variant: str | None
filename: str filename: str
# Absolute URL for files hosted elsewhere (see --external-links)
url: str | None = None
def parse_from_filename(file: str) -> WheelFileInfo: def parse_from_filename(file: str) -> WheelFileInfo:
@@ -112,29 +110,6 @@ def parse_from_filename(file: str) -> WheelFileInfo:
) )
def parse_external_url(url: str) -> WheelFileInfo:
"""Parse an absolute wheel or sdist URL hosted on another index."""
filename = unquote(urlparse(url).path.rsplit("/", 1)[-1])
if filename.endswith(".whl"):
info = parse_from_filename(filename)
else:
match = re.match(r"^(?P<name>.+)-(?P<version>[^-]+)\.(tar\.gz|zip)$", filename)
if not match:
raise ValueError(f"Unsupported external file: {filename}")
info = WheelFileInfo(
package_name=match.group("name"),
version=match.group("version"),
build_tag=None,
python_tag="source",
abi_tag="none",
platform_tag="any",
variant=None,
filename=filename,
)
info.url = url
return info
def generate_project_list(subdir_names: list[str], comment: str = "") -> str: def generate_project_list(subdir_names: list[str], comment: str = "") -> str:
"""Generate project list HTML content linking to each project & variant subdirectory.""" """Generate project list HTML content linking to each project & variant subdirectory."""
href_tags = [] href_tags = []
@@ -154,19 +129,14 @@ def generate_package_index_and_metadata(
href_tags = [] href_tags = []
metadata = [] metadata = []
for file in sorted(wheel_files, key=lambda x: x.filename): for file in sorted(wheel_files, key=lambda x: x.filename):
if file.url: relative_path = (
file_path_quoted = file.url wheel_base_dir.relative_to(index_base_dir, walk_up=True) / file.filename
else: )
relative_path = ( # handle with '+' in URL, and avoid double-encoding '/' and already-encoded '%2B'
wheel_base_dir.relative_to(index_base_dir, walk_up=True) / file.filename # NOTE: this is AWS S3 specific behavior!
) file_path_quoted = quote(relative_path.as_posix(), safe=":%/")
# handle with '+' in URL, and avoid double-encoding '/' and already-encoded '%2B'
# NOTE: this is AWS S3 specific behavior!
file_path_quoted = quote(relative_path.as_posix(), safe=":%/")
href_tags.append(f' <a href="{file_path_quoted}">{file.filename}</a><br/>') href_tags.append(f' <a href="{file_path_quoted}">{file.filename}</a><br/>')
file_meta = asdict(file) file_meta = asdict(file)
if file_meta["url"] is None:
del file_meta["url"]
file_meta["path"] = file_path_quoted file_meta["path"] = file_path_quoted
metadata.append(file_meta) metadata.append(file_meta)
index_str = INDEX_HTML_TEMPLATE.format(items="\n".join(href_tags), comment=comment) index_str = INDEX_HTML_TEMPLATE.format(items="\n".join(href_tags), comment=comment)
@@ -181,7 +151,6 @@ def generate_index_and_metadata(
default_variant: str | None = None, default_variant: str | None = None,
alias_to_default: str | None = None, alias_to_default: str | None = None,
comment: str = "", comment: str = "",
external_urls: list[str] | None = None,
): ):
"""Generate index for all wheel files. """Generate index for all wheel files.
@@ -191,8 +160,6 @@ def generate_index_and_metadata(
index_base_dir (Path): Base directory to store index files. index_base_dir (Path): Base directory to store index files.
default_variant (str | None): The default variant name, if any. default_variant (str | None): The default variant name, if any.
alias_to_default (str | None): Alias variant name for the default variant, if any. alias_to_default (str | None): Alias variant name for the default variant, if any.
external_urls (list[str] | None): Absolute URLs of files hosted on another index,
listed in the generated indices without being copied.
comment (str | None): Optional comment to include in the generated HTML files. comment (str | None): Optional comment to include in the generated HTML files.
First, parse all wheel files to extract metadata. First, parse all wheel files to extract metadata.
@@ -247,7 +214,6 @@ def generate_index_and_metadata(
""" """
parsed_files = [parse_from_filename(f) for f in whl_files] parsed_files = [parse_from_filename(f) for f in whl_files]
parsed_files += [parse_external_url(u) for u in external_urls or []]
if not parsed_files: if not parsed_files:
print("No wheel files found, skipping index generation.") print("No wheel files found, skipping index generation.")
@@ -266,14 +232,10 @@ def generate_index_and_metadata(
print(f"Detected ROCm variant from vllm: {rocm_variant}") print(f"Detected ROCm variant from vllm: {rocm_variant}")
break break
# Apply ROCm variant to all wheels without a variant. Dependencies with their # Apply ROCm variant to all wheels without a variant
# own ROCm local version (e.g. TheRock's torch-2.12.0+rocm10.0.0) belong to
# the same variant rather than one of their own.
if rocm_variant: if rocm_variant:
for file in parsed_files: for file in parsed_files:
if file.variant is None or ( if file.variant is None:
file.variant.startswith("rocm") and file.variant != rocm_variant
):
file.variant = rocm_variant file.variant = rocm_variant
print(f"Inherited variant '{rocm_variant}' for {file.filename}") print(f"Inherited variant '{rocm_variant}' for {file.filename}")
@@ -404,13 +366,6 @@ if __name__ == "__main__":
default=None, default=None,
help="Alias variant name for the default variant", help="Alias variant name for the default variant",
) )
parser.add_argument(
"--external-links",
type=str,
default=None,
help="Optional file with one absolute URL per line for packages hosted on "
"another index (listed alongside, but not copied into, this index)",
)
parser.add_argument( parser.add_argument(
"--comment", "--comment",
type=str, type=str,
@@ -508,14 +463,5 @@ if __name__ == "__main__":
default_variant=None, default_variant=None,
alias_to_default=args.alias_to_default, alias_to_default=args.alias_to_default,
comment=args.comment.strip(), comment=args.comment.strip(),
external_urls=(
[
line.strip()
for line in Path(args.external_links).read_text().splitlines()
if line.strip() and not line.startswith("#")
]
if args.external_links
else None
),
) )
print(f"Successfully generated index and metadata in {output_dir}") print(f"Successfully generated index and metadata in {output_dir}")
@@ -43,7 +43,6 @@ kernel_tests() {
docker exec "$CONTAINER_NAME" bash -c " docker exec "$CONTAINER_NAME" bash -c "
set -e set -e
pytest -x -v -s tests/kernels/test_onednn.py pytest -x -v -s tests/kernels/test_onednn.py
pytest -x -v -s tests/kernels/test_cpu_fused_sampling.py
pytest -x -v -s tests/kernels/attention/test_cpu_attn.py pytest -x -v -s tests/kernels/attention/test_cpu_attn.py
pytest -x -v -s tests/kernels/core/test_cpu_activation.py pytest -x -v -s tests/kernels/core/test_cpu_activation.py
pytest -x -v -s tests/kernels/moe/test_cpu_fused_moe.py pytest -x -v -s tests/kernels/moe/test_cpu_fused_moe.py
@@ -19,9 +19,10 @@ DISK_USAGE_THRESHOLD=${DISK_USAGE_THRESHOLD:-80}
CACHE_MAX_AGE=${CACHE_MAX_AGE:-24h} CACHE_MAX_AGE=${CACHE_MAX_AGE:-24h}
# Reclaim disk only when the host is under pressure, aging out anything unused # Reclaim disk only when the host is under pressure, aging out anything unused
# for less than CACHE_MAX_AGE so hot layers survive (`--filter until=<N>h`). # for less than CACHE_MAX_AGE so hot layers survive -- same `--filter
# `docker buildx prune --max-used-space` is a no-op on this host's `docker` # until=<N>h` pattern the TPU CI scripts already rely on. `docker buildx
# driver (BuildKit embedded in dockerd never enforces the size cap), so we don't use it. # prune --max-used-space` is a no-op on this host's `docker` driver (BuildKit
# embedded in dockerd never enforces the size cap), so we don't use it.
prune_if_disk_pressure() { prune_if_disk_pressure() {
local docker_root disk_usage local docker_root disk_usage
docker_root=$(docker info -f '{{.DockerRootDir}}' 2>/dev/null || true) docker_root=$(docker info -f '{{.DockerRootDir}}' 2>/dev/null || true)
@@ -0,0 +1,29 @@
#!/bin/bash
# This script build the GH200 docker image and run the offline inference inside the container.
# It serves a sanity check for compilation and basic model usage.
set -ex
# Skip the new torch installation during build since we are using the specified version for arm64 in the Dockerfile
python3 tools/use_existing_torch.py
# Try building the docker image
DOCKER_BUILDKIT=1 docker build . \
--file docker/Dockerfile \
--target vllm-openai \
--platform "linux/arm64" \
-t gh200-test \
--build-arg max_jobs=66 \
--build-arg nvcc_threads=2 \
--build-arg BUILD_BASE_IMAGE=pytorch/manylinuxaarch64-builder:cuda13.0-b8b5f17a7d9ccfc25bbc5cf17b3fcea12964a042 \
--build-arg torch_cuda_arch_list="9.0+PTX"
# Setup cleanup
remove_docker_container() { docker rm -f gh200-test || true; }
trap remove_docker_container EXIT
remove_docker_container
# Run the image and test offline inference
docker run -e HF_TOKEN -e VLLM_WORKER_MULTIPROC_METHOD=spawn -v /root/.cache/huggingface:/root/.cache/huggingface --name gh200-test --gpus=all --entrypoint="" gh200-test bash -c '
python3 examples/basic/offline_inference/generate.py --model meta-llama/Llama-3.2-1B
'
@@ -57,7 +57,7 @@ case "${test_suite}" in
cd tests cd tests
pytest -v -s quantization/test_auto_round.py pytest -v -s quantization/test_auto_round.py
pytest -v -s quantization/test_online.py --deselect=tests/quantization/test_online.py::test_online_quantization_loads_real_weights --deselect="tests/quantization/test_online.py::test_online_quantization[False-requantization_mxfp8_ptcp_fp8]" pytest -v -s quantization/test_online.py --deselect=tests/quantization/test_online.py::test_online_quantization_loads_real_weights
;; ;;
compressed-tensors-fp8) compressed-tensors-fp8)
cd tests cd tests
+166
View File
@@ -0,0 +1,166 @@
#!/bin/bash
set -xu
remove_docker_container() {
docker rm -f tpu-test || true;
}
trap remove_docker_container EXIT
# Remove the container that might not be cleaned up in the previous run.
remove_docker_container
# Build the docker image.
docker build -f docker/Dockerfile.tpu -t vllm-tpu .
# Set up cleanup.
cleanup_docker() {
# Get Docker's root directory
docker_root=$(docker info -f '{{.DockerRootDir}}')
if [ -z "$docker_root" ]; then
echo "Failed to determine Docker root directory."
exit 1
fi
echo "Docker root directory: $docker_root"
# Check disk usage of the filesystem where Docker's root directory is located
disk_usage=$(df "$docker_root" | tail -1 | awk '{print $5}' | sed 's/%//')
# Define the threshold
threshold=70
if [ "$disk_usage" -gt "$threshold" ]; then
echo "Disk usage is above $threshold%. Cleaning up Docker images and volumes..."
# Remove dangling images (those that are not tagged and not used by any container)
docker image prune -f
# Remove unused volumes / force the system prune for old images as well.
docker volume prune -f && docker system prune --force --filter "until=72h" --all
echo "Docker images and volumes cleanup completed."
else
echo "Disk usage is below $threshold%. No cleanup needed."
fi
}
cleanup_docker
# For HF_TOKEN.
source /etc/environment
docker run --privileged --net host --shm-size=16G -it \
-e "HF_TOKEN=$HF_TOKEN" --name tpu-test \
vllm-tpu /bin/bash -c '
set -e # Exit immediately if a command exits with a non-zero status.
set -u # Treat unset variables as an error.
echo "--- Starting script inside Docker container ---"
# Create results directory
RESULTS_DIR=$(mktemp -d)
# If mktemp fails, set -e will cause the script to exit.
echo "Results will be stored in: $RESULTS_DIR"
# Install dependencies
echo "--- Installing Python dependencies ---"
python3 -m pip install --progress-bar off git+https://github.com/thuml/depyf.git \
&& python3 -m pip install --progress-bar off pytest pytest-asyncio tpu-info \
&& python3 -m pip install --progress-bar off "lm-eval[api]>=0.4.12" \
&& python3 -m pip install --progress-bar off hf-transfer tblib==3.1.0
echo "--- Python dependencies installed ---"
export VLLM_XLA_CHECK_RECOMPILATION=1
export VLLM_XLA_CACHE_PATH=
echo "--- Hardware Information ---"
# tpu-info
echo "--- Starting Tests ---"
set +e
overall_script_exit_code=0
# --- Test Definitions ---
# If a test fails, this function will print logs and will not cause the main script to exit.
run_test() {
local test_num=$1
local test_name=$2
local test_command=$3
local log_file="$RESULTS_DIR/test_${test_num}.log"
local actual_exit_code
echo "--- TEST_$test_num: Running $test_name ---"
# Execute the test command.
eval "$test_command" > >(tee -a "$log_file") 2> >(tee -a "$log_file" >&2)
actual_exit_code=$?
echo "TEST_${test_num}_COMMAND_EXIT_CODE: $actual_exit_code" # This goes to main log
echo "TEST_${test_num}_COMMAND_EXIT_CODE: $actual_exit_code" >> "$log_file" # Also to per-test log
if [ "$actual_exit_code" -ne 0 ]; then
echo "TEST_$test_num ($test_name) FAILED with exit code $actual_exit_code." >&2
echo "--- Log for failed TEST_$test_num ($test_name) ---" >&2
if [ -f "$log_file" ]; then
cat "$log_file" >&2
else
echo "Log file $log_file not found for TEST_$test_num ($test_name)." >&2
fi
echo "--- End of log for TEST_$test_num ($test_name) ---" >&2
return "$actual_exit_code" # Return the failure code
else
echo "TEST_$test_num ($test_name) PASSED."
return 0 # Return success
fi
}
# Helper function to call run_test and update the overall script exit code
run_and_track_test() {
local test_num_arg="$1"
local test_name_arg="$2"
local test_command_arg="$3"
# Run the test
run_test "$test_num_arg" "$test_name_arg" "$test_command_arg"
local test_specific_exit_code=$?
# If the test failed, set the overall script exit code to 1
if [ "$test_specific_exit_code" -ne 0 ]; then
# No need for extra echo here, run_test already logged the failure.
overall_script_exit_code=1
fi
}
# --- Actual Test Execution ---
run_and_track_test 1 "test_struct_output_generate.py" \
"python3 -m pytest -s -v /workspace/vllm/tests/entrypoints/llm/test_struct_output_generate.py -k \"not test_structured_output_with_reasoning_matrices\""
run_and_track_test 2 "test_moe_pallas.py" \
"python3 -m pytest -s -v /workspace/vllm/tests/tpu/test_moe_pallas.py"
run_and_track_test 3 "test_lora.py" \
"VLLM_XLA_CHECK_RECOMPILATION=0 python3 -m pytest -s -v /workspace/vllm/tests/tpu/lora/test_lora.py"
run_and_track_test 4 "test_tpu_qkv_linear.py" \
"python3 -m pytest -s -v /workspace/vllm/tests/v1/tpu/test_tpu_qkv_linear.py"
run_and_track_test 5 "test_spmd_model_weight_loading.py" \
"python3 -m pytest -s -v /workspace/vllm/tests/v1/tpu/test_spmd_model_weight_loading.py"
run_and_track_test 6 "test_kv_cache_update_kernel.py" \
"python3 -m pytest -s -v /workspace/vllm/tests/v1/tpu/test_kv_cache_update_kernel.py"
run_and_track_test 7 "test_tpu_int8.py" \
"python3 -m pytest -s -v /workspace/vllm/tests/v1/tpu/test_tpu_int8.py"
# After all tests have been attempted, exit with the overall status.
if [ "$overall_script_exit_code" -ne 0 ]; then
echo "--- One or more tests FAILED. Overall script exiting with failure code 1. ---"
else
echo "--- All tests have completed and PASSED. Overall script exiting with success code 0. ---"
fi
exit "$overall_script_exit_code"
' # IMPORTANT: This is the closing single quote for the bash -c "..." command. Ensure it is present and correct.
# Capture the exit code of the docker run command
DOCKER_RUN_EXIT_CODE=$?
# The trap will run for cleanup.
# Exit the main script with the Docker run command's exit code.
if [ "$DOCKER_RUN_EXIT_CODE" -ne 0 ]; then
echo "Docker run command failed with exit code $DOCKER_RUN_EXIT_CODE."
exit "$DOCKER_RUN_EXIT_CODE"
else
echo "Docker run command completed successfully."
exit 0
fi
# TODO: This test fails because it uses RANDOM_SEED sampling
# pytest -v -s /workspace/vllm/tests/tpu/test_custom_dispatcher.py \
+172
View File
@@ -0,0 +1,172 @@
#!/bin/bash
set -xu
remove_docker_container() {
docker rm -f tpu-test || true;
}
trap remove_docker_container EXIT
# Remove the container that might not be cleaned up in the previous run.
remove_docker_container
# Build the docker image.
docker build -f docker/Dockerfile.tpu -t vllm-tpu .
# Set up cleanup.
cleanup_docker() {
# Get Docker's root directory
docker_root=$(docker info -f '{{.DockerRootDir}}')
if [ -z "$docker_root" ]; then
echo "Failed to determine Docker root directory."
exit 1
fi
echo "Docker root directory: $docker_root"
# Check disk usage of the filesystem where Docker's root directory is located
disk_usage=$(df "$docker_root" | tail -1 | awk '{print $5}' | sed 's/%//')
# Define the threshold
threshold=70
if [ "$disk_usage" -gt "$threshold" ]; then
echo "Disk usage is above $threshold%. Cleaning up Docker images and volumes..."
# Remove dangling images (those that are not tagged and not used by any container)
docker image prune -f
# Remove unused volumes / force the system prune for old images as well.
docker volume prune -f && docker system prune --force --filter "until=72h" --all
echo "Docker images and volumes cleanup completed."
else
echo "Disk usage is below $threshold%. No cleanup needed."
fi
}
cleanup_docker
# For HF_TOKEN.
source /etc/environment
docker run --privileged --net host --shm-size=16G -it \
-e "HF_TOKEN=$HF_TOKEN" --name tpu-test \
vllm-tpu /bin/bash -c '
set -e # Exit immediately if a command exits with a non-zero status.
set -u # Treat unset variables as an error.
echo "--- Starting script inside Docker container ---"
# Create results directory
RESULTS_DIR=$(mktemp -d)
# If mktemp fails, set -e will cause the script to exit.
echo "Results will be stored in: $RESULTS_DIR"
# Install dependencies
echo "--- Installing Python dependencies ---"
python3 -m pip install --progress-bar off git+https://github.com/thuml/depyf.git \
&& python3 -m pip install --progress-bar off pytest pytest-asyncio tpu-info \
&& python3 -m pip install --progress-bar off "lm-eval[api]>=0.4.12" \
&& python3 -m pip install --progress-bar off hf-transfer tblib==3.1.0
echo "--- Python dependencies installed ---"
export VLLM_XLA_CHECK_RECOMPILATION=1
export VLLM_XLA_CACHE_PATH=
echo "--- Hardware Information ---"
# tpu-info
echo "--- Starting Tests ---"
set +e
overall_script_exit_code=0
# --- Test Definitions ---
# If a test fails, this function will print logs and will not cause the main script to exit.
run_test() {
local test_num=$1
local test_name=$2
local test_command=$3
local log_file="$RESULTS_DIR/test_${test_num}.log"
local actual_exit_code
echo "--- TEST_$test_num: Running $test_name ---"
# Execute the test command.
eval "$test_command" > >(tee -a "$log_file") 2> >(tee -a "$log_file" >&2)
actual_exit_code=$?
echo "TEST_${test_num}_COMMAND_EXIT_CODE: $actual_exit_code" # This goes to main log
echo "TEST_${test_num}_COMMAND_EXIT_CODE: $actual_exit_code" >> "$log_file" # Also to per-test log
if [ "$actual_exit_code" -ne 0 ]; then
echo "TEST_$test_num ($test_name) FAILED with exit code $actual_exit_code." >&2
echo "--- Log for failed TEST_$test_num ($test_name) ---" >&2
if [ -f "$log_file" ]; then
cat "$log_file" >&2
else
echo "Log file $log_file not found for TEST_$test_num ($test_name)." >&2
fi
echo "--- End of log for TEST_$test_num ($test_name) ---" >&2
return "$actual_exit_code" # Return the failure code
else
echo "TEST_$test_num ($test_name) PASSED."
return 0 # Return success
fi
}
# Helper function to call run_test and update the overall script exit code
run_and_track_test() {
local test_num_arg="$1"
local test_name_arg="$2"
local test_command_arg="$3"
# Run the test
run_test "$test_num_arg" "$test_name_arg" "$test_command_arg"
local test_specific_exit_code=$?
# If the test failed, set the overall script exit code to 1
if [ "$test_specific_exit_code" -ne 0 ]; then
# No need for extra echo here, run_test already logged the failure.
overall_script_exit_code=1
fi
}
# --- Actual Test Execution ---
run_and_track_test 0 "test_perf.py" \
"python3 -m pytest -s -v /workspace/vllm/tests/v1/tpu/test_perf.py"
run_and_track_test 1 "test_compilation.py" \
"python3 -m pytest -s -v /workspace/vllm/tests/tpu/test_compilation.py"
run_and_track_test 2 "test_basic.py" \
"python3 -m pytest -s -v /workspace/vllm/tests/v1/tpu/test_basic.py"
run_and_track_test 3 "test_accuracy.py::test_lm_eval_accuracy_v1_engine" \
"python3 -m pytest -s -v /workspace/vllm/tests/entrypoints/llm/test_accuracy.py::test_lm_eval_accuracy_v1_engine"
run_and_track_test 4 "test_quantization_accuracy.py" \
"python3 -m pytest -s -v /workspace/vllm/tests/tpu/test_quantization_accuracy.py"
run_and_track_test 6 "test_tpu_model_runner.py" \
"python3 -m pytest -s -v /workspace/vllm/tests/v1/tpu/worker/test_tpu_model_runner.py"
run_and_track_test 7 "test_sampler.py" \
"python3 -m pytest -s -v /workspace/vllm/tests/v1/tpu/test_sampler.py"
run_and_track_test 8 "test_topk_topp_sampler.py" \
"python3 -m pytest -s -v /workspace/vllm/tests/v1/tpu/test_topk_topp_sampler.py"
run_and_track_test 9 "test_multimodal.py" \
"python3 -m pytest -s -v /workspace/vllm/tests/v1/tpu/test_multimodal.py"
run_and_track_test 10 "test_pallas.py" \
"python3 -m pytest -s -v /workspace/vllm/tests/v1/tpu/test_pallas.py"
# After all tests have been attempted, exit with the overall status.
if [ "$overall_script_exit_code" -ne 0 ]; then
echo "--- One or more tests FAILED. Overall script exiting with failure code 1. ---"
else
echo "--- All tests have completed and PASSED. Overall script exiting with success code 0. ---"
fi
exit "$overall_script_exit_code"
' # IMPORTANT: This is the closing single quote for the bash -c "..." command. Ensure it is present and correct.
# Capture the exit code of the docker run command
DOCKER_RUN_EXIT_CODE=$?
# The trap will run for cleanup.
# Exit the main script with the Docker run command's exit code.
if [ "$DOCKER_RUN_EXIT_CODE" -ne 0 ]; then
echo "Docker run command failed with exit code $DOCKER_RUN_EXIT_CODE."
exit "$DOCKER_RUN_EXIT_CODE"
else
echo "Docker run command completed successfully."
exit 0
fi
# TODO: This test fails because it uses RANDOM_SEED sampling
# pytest -v -s /workspace/vllm/tests/tpu/test_custom_dispatcher.py \
+1 -1
View File
@@ -10,7 +10,7 @@ fi
# TODO: Use the PyPI wheel once it is published. # TODO: Use the PyPI wheel once it is published.
uv pip install --system \ uv pip install --system \
"kvcr @ git+https://github.com/ai-dynamo/kvcr.git@v0.1.1" "kvcr @ git+https://github.com/ai-dynamo/kvcr.git@v0.1.0"
# Keep only the NIXL wheel matching the CI image's CUDA runtime. # Keep only the NIXL wheel matching the CI image's CUDA runtime.
NIXL_VERSION=$(uv pip show --system nixl | sed -n 's/^Version: //p') NIXL_VERSION=$(uv pip show --system nixl | sed -n 's/^Version: //p')
+16 -28
View File
@@ -11,9 +11,9 @@ set -euo pipefail
TARGET="${1:-all}" TARGET="${1:-all}"
case "${TARGET}" in case "${TARGET}" in
cuda-13-0 | cuda-12-9 | cuda-13-0-ubuntu-24-04 | \ cuda-13-0 | cuda-12-9 | cuda-13-0-ubuntu-24-04 | \
cuda-12-9-ubuntu-24-04 | rocm | rocm72 | xpu | cpu | all) ;; cuda-12-9-ubuntu-24-04 | rocm | xpu | cpu | all) ;;
*) *)
echo "Usage: $0 {cuda-13-0|cuda-12-9|cuda-13-0-ubuntu-24-04|cuda-12-9-ubuntu-24-04|rocm|rocm72|xpu|cpu|all}" echo "Usage: $0 {cuda-13-0|cuda-12-9|cuda-13-0-ubuntu-24-04|cuda-12-9-ubuntu-24-04|rocm|xpu|cpu|all}"
exit 2 exit 2
;; ;;
esac esac
@@ -143,35 +143,23 @@ if target_enabled cuda-12-9-ubuntu-24-04; then
fi fi
# ---- ROCm ---- # ---- ROCm ----
# Each ROCm stack is identified by its base Dockerfile (see rocm/stack.sh). The
# default stack also owns the plain :latest / :v<ver> tags.
publish_rocm_stack() {
local base_dockerfile="$1" default="$2" suffix
(
# shellcheck source=.buildkite/scripts/rocm/stack.sh
source .buildkite/scripts/rocm/stack.sh "$base_dockerfile"
local suffixes=("-${ROCM_STACK_VARIANT}")
[[ "$default" == "1" ]] && suffixes=("" "-${ROCM_STACK_VARIANT}")
docker pull "$ROCM_STACK_ECR_IMAGE"
docker pull "$ROCM_STACK_ECR_BASE"
for suffix in "${suffixes[@]}"; do
for tag in "latest${suffix}" "v${RELEASE_VERSION}${suffix}"; do
docker tag "$ROCM_STACK_ECR_IMAGE" "vllm/vllm-openai-rocm:${tag}"
docker tag "$ROCM_STACK_ECR_BASE" "vllm/vllm-openai-rocm:${tag}-base"
docker push "vllm/vllm-openai-rocm:${tag}"
docker push "vllm/vllm-openai-rocm:${tag}-base"
done
done
)
}
if target_enabled rocm; then if target_enabled rocm; then
publish_rocm_stack docker/Dockerfile.rocm_base 1 ROCM_BASE_CACHE_KEY=$(.buildkite/scripts/cache-rocm-base-wheels.sh key)
fi echo "ROCm base cache key: ${ROCM_BASE_CACHE_KEY}"
if target_enabled rocm72; then docker pull "public.ecr.aws/q9t5s3a7/vllm-release-repo:${COMMIT}-rocm"
publish_rocm_stack docker/Dockerfile.rocm_72_base 0 docker pull "public.ecr.aws/q9t5s3a7/vllm-release-repo:${ROCM_BASE_CACHE_KEY}-rocm-base"
docker tag "public.ecr.aws/q9t5s3a7/vllm-release-repo:${COMMIT}-rocm" vllm/vllm-openai-rocm:latest
docker tag "public.ecr.aws/q9t5s3a7/vllm-release-repo:${COMMIT}-rocm" "vllm/vllm-openai-rocm:v${RELEASE_VERSION}"
docker push vllm/vllm-openai-rocm:latest
docker push "vllm/vllm-openai-rocm:v${RELEASE_VERSION}"
docker tag "public.ecr.aws/q9t5s3a7/vllm-release-repo:${ROCM_BASE_CACHE_KEY}-rocm-base" vllm/vllm-openai-rocm:latest-base
docker tag "public.ecr.aws/q9t5s3a7/vllm-release-repo:${ROCM_BASE_CACHE_KEY}-rocm-base" "vllm/vllm-openai-rocm:v${RELEASE_VERSION}-base"
docker push vllm/vllm-openai-rocm:latest-base
docker push "vllm/vllm-openai-rocm:v${RELEASE_VERSION}-base"
fi fi
# ---- XPU ---- # ---- XPU ----
+65 -42
View File
@@ -2,36 +2,42 @@
# SPDX-License-Identifier: Apache-2.0 # SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
# #
# Push a ROCm stack's nightly base image and nightly image from ECR to Docker Hub # Push a ROCm-family nightly base image and nightly image from ECR to Docker Hub
# under vllm/vllm-openai-rocm, as base-nightly-<variant>, nightly-<variant> and their # under vllm/vllm-openai-rocm, as base-nightly[-<variant>], nightly[-<variant>]
# -<commit> forms, then prune old nightly tags. With --default, also push the plain # and their -<commit> forms.
# base-nightly and nightly tags for the same images. # Run when NIGHTLY=1 after the matching build-*-release-image step has pushed to ECR.
# Run when NIGHTLY=1 after the stack's release image has been pushed to ECR.
# #
# Usage: push-nightly-builds-rocm.sh <base-dockerfile> [--default] # Usage: push-nightly-builds-rocm.sh [ECR_TAG_SUFFIX TAG_VARIANT]
# e.g. "docker/Dockerfile.rocm_base --default" (ROCm 10.0, also :nightly) # Pass both arguments or neither. With none, the default ROCm apt build is
# and "docker/Dockerfile.rocm_72_base" (ROCm 7.2, :nightly-rocm72). # published as :nightly. The TheRock build passes "rock" (the source ECR tag
# suffix) and "rocm714" (the Docker Hub tag flavor), giving :nightly-rocm714.
# #
# Local testing (no push to Docker Hub): # Local testing (no push to Docker Hub):
# BUILDKITE_COMMIT=<commit-with-image-in-ecr> DRY_RUN=1 \ # BASE_ECR_IMAGE=<full-base-image-reference> \
# bash .buildkite/scripts/push-nightly-builds-rocm.sh docker/Dockerfile.rocm_base --default # BUILDKITE_COMMIT=<commit-with-image-in-ecr> DRY_RUN=1 \
# bash .buildkite/scripts/push-nightly-builds-rocm.sh
# Requires: AWS CLI configured (for ECR public login), Docker. For full run: Docker Hub login. # Requires: AWS CLI configured (for ECR public login), Docker. For full run: Docker Hub login.
set -euxo pipefail set -euxo pipefail
usage() { usage() {
echo "Usage: $0 <base-dockerfile> [--default]" >&2 echo "Usage: $0 [ECR_TAG_SUFFIX TAG_VARIANT]" >&2
} }
PUSH_DEFAULT=0 DOCKERHUB_REPO="vllm/vllm-openai-rocm"
case "$#" in case "$#" in
1) ;; 0)
ECR_TAG_SUFFIX="rocm"
TAG_VARIANT=""
;;
2) 2)
if [[ "$2" != "--default" ]]; then if [[ -z "$1" || -z "$2" ]]; then
usage usage
exit 2 exit 2
fi fi
PUSH_DEFAULT=1 ECR_TAG_SUFFIX="$1"
TAG_VARIANT="$2"
;; ;;
*) *)
usage usage
@@ -43,44 +49,61 @@ esac
BUILDKITE_COMMIT="${BUILDKITE_COMMIT:?Set BUILDKITE_COMMIT to the commit SHA that has the image in ECR (e.g. from a previous release pipeline run)}" BUILDKITE_COMMIT="${BUILDKITE_COMMIT:?Set BUILDKITE_COMMIT to the commit SHA that has the image in ECR (e.g. from a previous release pipeline run)}"
DRY_RUN="${DRY_RUN:-0}" DRY_RUN="${DRY_RUN:-0}"
# shellcheck source=.buildkite/scripts/rocm/stack.sh ECR_REPO="public.ecr.aws/q9t5s3a7/vllm-release-repo"
source "$(dirname "${BASH_SOURCE[0]}")/rocm/stack.sh" "$1"
DOCKERHUB_REPO="vllm/vllm-openai-rocm"
VARIANT="$ROCM_STACK_VARIANT"
BASE_TAGS=("base-nightly-${VARIANT}" "base-nightly-${VARIANT}-${BUILDKITE_COMMIT}") # Get the base image ECR tag (set by the build-*-release-image pipeline step)
TAGS=("nightly-${VARIANT}" "nightly-${VARIANT}-${BUILDKITE_COMMIT}") BASE_METADATA_KEY="${ECR_TAG_SUFFIX}-base-ecr-tag"
if [[ "$PUSH_DEFAULT" == "1" ]]; then if [[ -n "${BASE_ECR_IMAGE:-}" ]]; then
BASE_TAGS+=("base-nightly" "base-nightly-${BUILDKITE_COMMIT}") if [[ "$DRY_RUN" != "1" ]]; then
TAGS+=("nightly" "nightly-${BUILDKITE_COMMIT}") echo "ERROR: BASE_ECR_IMAGE may only be used with DRY_RUN=1" >&2
exit 1
fi
BASE_ORIG_TAG="$BASE_ECR_IMAGE"
elif ! BASE_ORIG_TAG="$(buildkite-agent meta-data get "$BASE_METADATA_KEY")"; then
echo "ERROR: Failed to read required Buildkite metadata '$BASE_METADATA_KEY'" >&2
exit 1
fi
if [[ -z "$BASE_ORIG_TAG" ]]; then
echo "ERROR: Required Buildkite metadata '$BASE_METADATA_KEY' is empty" >&2
echo "Set BASE_ECR_IMAGE to the full ECR base image reference for local testing" >&2
exit 1
fi fi
echo "Pushing base image $ROCM_STACK_ECR_BASE as: ${BASE_TAGS[*]}" ORIG_TAG="${BUILDKITE_COMMIT}-${ECR_TAG_SUFFIX}"
echo "Pushing release image $ROCM_STACK_ECR_IMAGE as: ${TAGS[*]}" VARIANT_SUFFIX=""
[[ -n "$TAG_VARIANT" ]] && VARIANT_SUFFIX="-$TAG_VARIANT"
BASE_TAG_NAME="base-nightly${VARIANT_SUFFIX}"
TAG_NAME="nightly${VARIANT_SUFFIX}"
BASE_TAG_NAME_COMMIT="base-nightly${VARIANT_SUFFIX}-${BUILDKITE_COMMIT}"
TAG_NAME_COMMIT="nightly${VARIANT_SUFFIX}-${BUILDKITE_COMMIT}"
# Login to ECR and pull the stack's images echo "Pushing base image from ECR: $BASE_ORIG_TAG"
echo "Pushing release image from ECR tag: $ORIG_TAG to Docker Hub as $DOCKERHUB_REPO:$TAG_NAME and $DOCKERHUB_REPO:$TAG_NAME_COMMIT"
[[ "$DRY_RUN" == "1" ]] && echo "[DRY_RUN] Skipping push to Docker Hub"
# Login to ECR and pull the images built by build-*-release-image
aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7 aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7
docker pull "$ROCM_STACK_ECR_BASE" docker pull "$BASE_ORIG_TAG"
docker pull "$ROCM_STACK_ECR_IMAGE" docker pull "${ECR_REPO}:${ORIG_TAG}"
for tag in "${BASE_TAGS[@]}"; do docker tag "$ROCM_STACK_ECR_BASE" "$DOCKERHUB_REPO:$tag"; done # Tag for Docker Hub (base-nightly and base-nightly-<commit>, nightly and nightly-<commit>)
for tag in "${TAGS[@]}"; do docker tag "$ROCM_STACK_ECR_IMAGE" "$DOCKERHUB_REPO:$tag"; done docker tag "$BASE_ORIG_TAG" "$DOCKERHUB_REPO":"$BASE_TAG_NAME"
docker tag "$BASE_ORIG_TAG" "$DOCKERHUB_REPO":"$BASE_TAG_NAME_COMMIT"
docker tag "${ECR_REPO}:${ORIG_TAG}" "$DOCKERHUB_REPO":"$TAG_NAME"
docker tag "${ECR_REPO}:${ORIG_TAG}" "$DOCKERHUB_REPO":"$TAG_NAME_COMMIT"
if [[ "$DRY_RUN" == "1" ]]; then if [[ "$DRY_RUN" == "1" ]]; then
echo "[DRY_RUN] Would push $DOCKERHUB_REPO:$BASE_TAG_NAME and $DOCKERHUB_REPO:$BASE_TAG_NAME_COMMIT"
echo "[DRY_RUN] Would push $DOCKERHUB_REPO:$TAG_NAME and $DOCKERHUB_REPO:$TAG_NAME_COMMIT"
echo "[DRY_RUN] Local tags created. Exiting without push." echo "[DRY_RUN] Local tags created. Exiting without push."
exit 0 exit 0
fi fi
# Push to Docker Hub (docker-login plugin runs before this step in CI) # Push to Docker Hub (docker-login plugin runs before this step in CI)
for tag in "${BASE_TAGS[@]}" "${TAGS[@]}"; do docker push "$DOCKERHUB_REPO:$tag"; done docker push "$DOCKERHUB_REPO":"$BASE_TAG_NAME"
echo "Pushed $DOCKERHUB_REPO: ${BASE_TAGS[*]} ${TAGS[*]}" docker push "$DOCKERHUB_REPO":"$BASE_TAG_NAME_COMMIT"
docker push "$DOCKERHUB_REPO":"$TAG_NAME"
docker push "$DOCKERHUB_REPO":"$TAG_NAME_COMMIT"
# Keep only the last 14 builds of each tag family. The plain nightly- families echo "Pushed $DOCKERHUB_REPO:$BASE_TAG_NAME and $DOCKERHUB_REPO:$BASE_TAG_NAME_COMMIT"
# exclude the variant ones sharing their prefix; commit SHAs never start with "rocm". echo "Pushed $DOCKERHUB_REPO:$TAG_NAME and $DOCKERHUB_REPO:$TAG_NAME_COMMIT"
CLEANUP=.buildkite/scripts/cleanup-nightly-builds.sh
bash "$CLEANUP" "nightly-${VARIANT}-" "$DOCKERHUB_REPO"
bash "$CLEANUP" "base-nightly-${VARIANT}-" "$DOCKERHUB_REPO"
if [[ "$PUSH_DEFAULT" == "1" ]]; then
bash "$CLEANUP" "nightly-" "$DOCKERHUB_REPO" "nightly-rocm"
bash "$CLEANUP" "base-nightly-" "$DOCKERHUB_REPO" "base-nightly-rocm"
fi
@@ -1,112 +0,0 @@
#!/bin/bash
# Install the AITER nightly wheel over this build's ci_base.
#
# Runs in CI when AITER_NIGHTLY=1, once the stock ci_base it builds on is
# published. In order:
# 1. Select the newest nightly wheel that fits the stock ci_base
# (select_aiter_nightly_wheel.py, beside this script, run inside that image)
# 2. Build a new image: pip install that wheel over the stock ci_base
# 3. Push the new image as this build's ci_base-build tag, replacing the stock
# one, so later steps and native jobs use it; record its digest
#
# Buildkite meta-data, a key-value store shared by every job in this build:
# reads rocm-ci-base-image the stock ci_base
# r/w aiter-nightly-wheel-url the wheel step 1 chose; a retry reads it
# back and installs that same wheel
# writes aiter-nightly-ci-base the new image as tag@digest; the vLLM
# image build uses it as its ci_base
#
# Exit status:
# 10 no wheel fits this ci_base
# 11 the wheel does not install
# Anything else is unexpected and passed through as is.
# Buildkite retries this step once on exit status 1 or 2, which it treats as
# transient Docker/registry failures (amd_infra_retry in the template).
# 10 and 11 sit outside that range: they are real AITER failures, and a retry
# would only fail the same way.
set -euo pipefail
SELECT_WHEEL="$(dirname "${BASH_SOURCE[0]}")/select_aiter_nightly_wheel.py"
BUILD_TAG="rocm/vllm-dev:ci_base-build-${BUILDKITE_BUILD_ID:?BUILDKITE_BUILD_ID is required}"
# Scratch space for the build context and captured errors. The trap deletes it
# however the script exits: success, fail, or set -e.
scratch="$(mktemp -d)"
trap 'rm -rf "${scratch}"' EXIT
errors="${scratch}/errors"
fail() {
local status="$1" title="$2" detail="$3"
echo "+++ :x: AITER nightly: ${title}"
echo "${detail}" >&2
# shellcheck disable=SC2016 # the backticks are a literal markdown fence
printf ':x: **AITER nightly: %s**\n\n```\n%s\n```\n' "${title}" "${detail}" \
| buildkite-agent annotate --style error --context aiter-nightly
exit "${status}"
}
# Set by ensure-ci-base-amd; this step's depends_on guarantees it is there.
ci_base="$(buildkite-agent meta-data get rocm-ci-base-image)"
# Set below on the first attempt. Meta-data is per build, so a retry reuses
# that wheel, not a newer nightly.
pinned="$(buildkite-agent meta-data get aiter-nightly-wheel-url --default "")"
echo "--- :docker: Pulling ${ci_base}"
docker pull "${ci_base}"
echo "--- :mag: Selecting the AITER nightly for ${ci_base}"
set +e
out="$(docker run --rm -i --entrypoint python3 "${ci_base}" \
-P - ${pinned:+--wheel-url "${pinned}"} \
< "${SELECT_WHEEL}" 2> "${errors}")"
status=$?
set -e
cat "${errors}" >&2
case "${status}" in
0) ;;
10) fail 10 "no wheel for this image" "${out}" ;;
*) fail "${status}" "could not select a wheel" "$(tail -n 20 "${errors}")" ;;
esac
# The first line is the wheel's URL; the rest, the annotation naming it.
wheel_url="$(head -n 1 <<< "${out}")"
note="$(tail -n +2 <<< "${out}")"
buildkite-agent meta-data set aiter-nightly-wheel-url "${wheel_url}"
cat > "${scratch}/Dockerfile" <<'EOF'
ARG CI_BASE_IMAGE
FROM ${CI_BASE_IMAGE}
ARG AITER_WHEEL_URL
RUN python3 -m pip install --no-cache-dir "$AITER_WHEEL_URL"
EOF
echo "--- :docker: Installing ${wheel_url##*/}"
if ! docker buildx build \
--builder default \
--provenance=false \
--progress plain \
--build-arg "CI_BASE_IMAGE=${ci_base}" \
--build-arg "AITER_WHEEL_URL=${wheel_url}" \
-t "${BUILD_TAG}" \
"${scratch}"; then
fail 11 "the wheel does not install" \
"Installing ${wheel_url##*/} over ${ci_base} failed; see this step's log."
fi
docker push "${BUILD_TAG}"
# Handed off as tag@digest: build-test-image.sh rejects a ci_base that is not
# digest-pinned.
digest="$(docker buildx imagetools inspect "${BUILD_TAG}" | awk '$1 == "Digest:" { print $2; exit }')"
if [[ ! "${digest}" =~ ^sha256:[0-9a-f]{64}$ ]]; then
echo "Could not resolve the pushed digest of ${BUILD_TAG}" >&2
exit 1
fi
# The vLLM image build uses this as its ci_base.
buildkite-agent meta-data set aiter-nightly-ci-base "${BUILD_TAG}@${digest}"
# The annotation starts with :warning: when the wheel is stale.
style=info
if [[ "${note}" == :warning:* ]]; then
style=warning
fi
echo "${note}" | buildkite-agent annotate --style "${style}" --context aiter-nightly
echo "--- :white_check_mark: ${BUILD_TAG}@${digest}"
+2 -2
View File
@@ -1,8 +1,8 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# Shared Dockerfile selection for the normal AMD CI build chain. # Shared Dockerfile selection for the normal AMD CI build chain.
# CI_ROCM_DOCKERFILE_BASE and CI_ROCM_DOCKERFILE accept repository-relative paths. # CI_ROCM_DOCKERFILE_BASE and CI_ROCM_DOCKERFILE accept repository-relative paths.
# For the legacy ROCm 7.2 stack, set CI_ROCM_DOCKERFILE_BASE=docker/Dockerfile.rocm_72_base # For Rock, set CI_ROCM_DOCKERFILE_BASE=docker/Dockerfile.rock_base and
# and CI_ROCM_DOCKERFILE=docker/Dockerfile.rocm_72. # CI_ROCM_DOCKERFILE=docker/Dockerfile.rock.
# Custom recipes must preserve the CI targets and artifact layout. Additional # Custom recipes must preserve the CI targets and artifact layout. Additional
# COPY inputs also need the corresponding *_CONTENT_FILES overrides. # COPY inputs also need the corresponding *_CONTENT_FILES overrides.
@@ -1,260 +0,0 @@
#!/usr/bin/env python3
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""Select the AITER nightly wheel for a vLLM ROCm image.
aiter-nightly-overlay.sh runs this inside the stock ci_base, so it reads that
image's ROCm, torch and Python. It prints the wheel's URL, then the Buildkite
annotation naming it.
"""
import argparse
import datetime
import importlib.metadata
import os
import sys
import urllib.request
from dataclasses import dataclass
from html.parser import HTMLParser
from pathlib import Path
from urllib.parse import unquote, urldefrag, urljoin
import regex as re
INDEX_URL = "https://rocm.frameworks-nightlies.amd.com/whl-multi-arch/amd-aiter/"
AITER_COMMIT_URL = "https://github.com/ROCm/aiter/commit/"
STALE_AFTER = datetime.timedelta(days=1)
# Example:
# amd_aiter-0.1.25+rocm7.2.3.torch2.12.ae984f5.d20261008-cp312-cp312-linux_x86_64.whl
# is AITER main at commit ae984f5, built 2026-10-08, for ROCm 7.2.3, torch 2.12
# and Python 3.12. 0.1.25 is a guess at the next release, not a real one.
WHEEL_NAME = re.compile(
r"^amd_aiter-(?P<base_version>[^+-]+)\+(?P<local>[^-]+)"
r"-(?P<python>cp\d+)-(?P=python)-linux_x86_64\.whl$"
)
ROCM_FIELD = re.compile(r"(?:^|\.)rocm(\d+(?:\.\d+)*)(?:a\d+)?(?=\.|$)")
TORCH_FIELD = re.compile(r"(?:^|\.)torch(\d+\.\d+)(?=\.|$)")
COMMIT_AND_DATE_FIELD = re.compile(r"(?:^|\.)([0-9a-f]{7,40})\.d(\d{8})(?=\.|$)")
@dataclass(frozen=True)
class Wheel:
"""An AITER nightly wheel, read from its filename (see WHEEL_NAME)."""
filename: str
url: str
version: str
base_version: str
rocm: str
torch: str
commit: str
build_date: datetime.date
python: str
class _LinkParser(HTMLParser):
"""Collects every link on the index page, which lists one per wheel."""
def __init__(self):
super().__init__()
self.hrefs: list[str] = []
def handle_starttag(self, tag, attrs):
if tag == "a":
self.hrefs.extend(
value for name, value in attrs if name == "href" and value
)
def wheel_from_url(url: str) -> Wheel | None:
"""The AITER nightly wheel an absolute URL points at, if it is one."""
filename = unquote(urldefrag(url).url.rsplit("/", 1)[-1])
match = WHEEL_NAME.match(filename)
if not match:
return None
local = match["local"]
rocm, torch, build = (
field.search(local)
for field in (ROCM_FIELD, TORCH_FIELD, COMMIT_AND_DATE_FIELD)
)
if not (rocm and torch and build):
return None
return Wheel(
filename=filename,
url=url,
version=f"{match['base_version']}+{local}",
base_version=match["base_version"],
rocm=rocm[1],
torch=torch[1],
commit=build[1],
build_date=datetime.datetime.strptime(build[2], "%Y%m%d").date(),
python=match["python"],
)
def parse_index(html: str, index_url: str) -> list[Wheel]:
"""Every AITER nightly wheel linked from the index page, a plain HTML list
of links, one per file (the format pip reads)."""
parser = _LinkParser()
parser.feed(html)
wheels = (wheel_from_url(urljoin(index_url, href)) for href in parser.hrefs)
return [wheel for wheel in wheels if wheel]
def rocm_version() -> str:
"""This image's ROCm version; ROCm 10+ ships as the `rocm` pip SDK."""
try:
return importlib.metadata.version("rocm")
except importlib.metadata.PackageNotFoundError:
# No pip SDK: fall back to older ROCm releases, which install to ROCM_PATH.
rocm_path = Path(os.environ.get("ROCM_PATH", "/opt/rocm"))
return (rocm_path / ".info" / "version").read_text()
def rocm_release(version: str) -> str:
"""`7.2.3-49` or `10.0.0a20261001` -> `7.2.3` or `10.0.0`, as wheel names
spell it."""
match = re.match(r"\d+(?:\.\d+)*", version.strip())
if not match:
raise ValueError(f"Unrecognised ROCm version: {version!r}")
return match.group(0)
def torch_release(version: str) -> str:
"""`2.12.0a0+git6bbd260` -> `2.12`, as wheel names spell it."""
match = re.match(r"\d+\.\d+", version)
if not match:
raise ValueError(f"Unrecognised torch version: {version!r}")
return match.group(0)
def python_tag(version_info: tuple[int, int]) -> str:
"""(3, 12) -> `cp312`, as wheel names spell it."""
return f"cp{version_info[0]}{version_info[1]}"
def _base_version_key(wheel: Wheel) -> tuple[int, ...]:
"""`0.1.25` -> (0, 1, 25), so versions sort as numbers: 0.1.10 > 0.1.9."""
return tuple(int(part) for part in re.findall(r"\d+", wheel.base_version))
def _label(rocm: str, torch: str, python: str) -> str:
"""`rocm7.2.3/torch2.12/cp312`, for messages."""
return f"rocm{rocm}/torch{torch}/{python}"
def select_wheel(wheels: list[Wheel], rocm: str, torch: str, python: str) -> Wheel:
"""Newest build for this ROCm, torch and Python, by date: version labels lag.
Two builds with the same date and version: whichever the index lists first.
"""
target = (rocm, torch, python)
matching = [w for w in wheels if (w.rocm, w.torch, w.python) == target]
if not matching:
available = sorted({_label(w.rocm, w.torch, w.python) for w in wheels})
raise LookupError(
f"No AITER nightly for {_label(*target)}. "
f"Available: {', '.join(available) or 'none'}"
)
return max(matching, key=lambda w: (w.build_date, _base_version_key(w)))
def fetch(url: str) -> str:
"""GET url as text. A failure exits 1, so Buildkite retries the step."""
with urllib.request.urlopen(url, timeout=60) as response:
return response.read().decode()
def annotation(wheel: Wheel, today: datetime.date) -> str:
"""The Buildkite annotation naming the wheel this build tests.
Starts with :warning: if the wheel is stale; the overlay script styles it so.
"""
age = today - wheel.build_date
stale = (
f":warning: **AITER nightly is {age.days} days old**: "
f"no newer build has been published for "
f"{_label(wheel.rocm, wheel.torch, wheel.python)}.\n\n"
if age > STALE_AFTER
else ""
)
return stale + (
f":crescent_moon: **AITER nightly**: main at "
f"[`{wheel.commit}`]({AITER_COMMIT_URL}{wheel.commit}), "
f"built {wheel.build_date.isoformat()} for torch {wheel.torch} "
f"(`{wheel.version}`)"
)
class NoWheel(Exception):
"""No wheel fits this image's ROCm, torch and Python. Exit 10."""
def _image_wheel(wheel_url: str) -> Wheel:
"""The wheel at wheel_url, if it fits this image's ROCm, torch and Python."""
wheel = wheel_from_url(wheel_url)
if not wheel:
raise NoWheel(f"Not an AITER nightly wheel: {wheel_url}")
return _for_image([wheel])
def _image() -> tuple[str, str, str]:
"""This image's ROCm, torch and Python, as wheel names spell them."""
try:
rocm = rocm_release(rocm_version())
except (OSError, ValueError) as error:
raise NoWheel(f"Cannot read this image's ROCm version: {error}") from error
try:
torch = torch_release(importlib.metadata.version("torch"))
except (importlib.metadata.PackageNotFoundError, ValueError) as error:
raise NoWheel(f"Cannot read this image's torch version: {error}") from error
return rocm, torch, python_tag(sys.version_info[:2])
def _for_image(wheels: list[Wheel]) -> Wheel:
"""The newest of wheels for the image this runs in."""
try:
return select_wheel(wheels, *_image())
except LookupError as error:
raise NoWheel(str(error)) from error
def select(index_url: str, wheel_url: str = "") -> Wheel:
"""The wheel to install: wheel_url on a retry, else the newest.
However old the newest is, it is tested; the annotation shows its build date.
"""
if wheel_url:
return _image_wheel(wheel_url)
return _for_image(parse_index(fetch(index_url), index_url))
def _today() -> datetime.date:
"""Today in UTC; a function so tests can set the date."""
return datetime.datetime.now(datetime.UTC).date()
# Not 1 or 2, which Buildkite retries; see aiter-nightly-overlay.sh.
EXIT_NO_WHEEL = 10
def main(argv=None) -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--index-url", default=INDEX_URL)
parser.add_argument(
"--wheel-url", default="", help="a retry's earlier choice; skips the index"
)
args = parser.parse_args(argv)
try:
wheel = select(args.index_url, args.wheel_url)
except NoWheel as error:
print(error)
return EXIT_NO_WHEEL
print(wheel.url)
print(annotation(wheel, _today()))
return 0
if __name__ == "__main__":
sys.exit(main())
-78
View File
@@ -1,78 +0,0 @@
#!/usr/bin/env bash
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
#
# Derive everything the release pipeline needs about a ROCm stack from its base
# Dockerfile, so the stack's image is the single input to every ROCm script.
#
# Usage: source .buildkite/scripts/rocm/stack.sh <base-dockerfile>
# e.g. docker/Dockerfile.rocm_base (ROCm 10.0) or docker/Dockerfile.rocm_72_base
#
# Exports:
# ROCM_STACK_BASE_DOCKERFILE the base Dockerfile
# ROCM_STACK_DOCKERFILE the matching vLLM Dockerfile (base name minus _base)
# ROCM_STACK_THEROCK 1 if the ROCm SDK comes from TheRock wheels
# ROCM_STACK_VERSION ROCm version, e.g. 10.0.0 or 7.2.3
# ROCM_STACK_VARIANT wheel variant and Docker Hub tag flavor from major.minor,
# same rule as setup.py (rocm100, rocm72)
# ROCM_STACK_IMAGE_KEY hash of the base Dockerfile (base image cache key)
# ROCM_STACK_WHEEL_KEY hash of the base Dockerfile plus files it COPYs from
# the repo (base wheel cache key)
# ROCM_STACK_ECR_BASE ECR tag of the cached base image
# ROCM_STACK_ECR_IMAGE ECR tag of this commit's release image
# ROCM_STACK_BASE_WHEELS_DIR local dir for base wheels (also the artifact path)
# ROCM_STACK_VLLM_WHEEL_DIR local dir for the vLLM wheel (also the artifact path)
rocm_stack_init() {
local base="$1"
if [[ ! -f "$base" ]]; then
echo "ROCm base Dockerfile not found: $base" >&2
return 2
fi
ROCM_STACK_BASE_DOCKERFILE="$base"
ROCM_STACK_DOCKERFILE="${base%_base}"
local sdk
sdk=$(sed -nE 's/^ARG ROCM_SDK_VERSION=([0-9.]+).*/\1/p' "$base" | head -1)
if [[ -n "$sdk" ]]; then
ROCM_STACK_THEROCK=1
ROCM_STACK_VERSION="$sdk"
else
ROCM_STACK_THEROCK=0
# BASE_IMAGE format: rocm/dev-ubuntu-22.04:7.2.3-complete -> 7.2.3
ROCM_STACK_VERSION=$(sed -nE 's/^ARG BASE_IMAGE=.*:([0-9]+(\.[0-9]+)+).*/\1/p' "$base" | head -1)
fi
if [[ -z "$ROCM_STACK_VERSION" ]]; then
echo "Could not determine the ROCm version from $base" >&2
return 2
fi
local major minor
IFS=. read -r major minor _ <<< "$ROCM_STACK_VERSION"
ROCM_STACK_VARIANT="rocm${major}${minor}"
ROCM_STACK_IMAGE_KEY=$(sha256sum "$base" | cut -c1-16)
local inputs
inputs=$(grep -E '^COPY ' "$base" | grep -v -- '--from' | awk '{print $2}' || true)
if [[ -z "$inputs" ]]; then
ROCM_STACK_WHEEL_KEY="$ROCM_STACK_IMAGE_KEY"
else
# shellcheck disable=SC2086
ROCM_STACK_WHEEL_KEY=$(sha256sum "$base" $inputs | sha256sum | cut -c1-16)
fi
local ecr="public.ecr.aws/q9t5s3a7/vllm-release-repo"
ROCM_STACK_ECR_BASE="${ecr}:${ROCM_STACK_IMAGE_KEY}-${ROCM_STACK_VARIANT}-base"
ROCM_STACK_ECR_IMAGE="${ecr}:${BUILDKITE_COMMIT:-unknown}-${ROCM_STACK_VARIANT}"
ROCM_STACK_BASE_WHEELS_DIR="artifacts/${ROCM_STACK_VARIANT}-base-wheels"
ROCM_STACK_VLLM_WHEEL_DIR="artifacts/${ROCM_STACK_VARIANT}-vllm-wheel"
export ROCM_STACK_BASE_DOCKERFILE ROCM_STACK_DOCKERFILE ROCM_STACK_THEROCK \
ROCM_STACK_VERSION ROCM_STACK_VARIANT ROCM_STACK_IMAGE_KEY \
ROCM_STACK_WHEEL_KEY ROCM_STACK_ECR_BASE ROCM_STACK_ECR_IMAGE \
ROCM_STACK_BASE_WHEELS_DIR ROCM_STACK_VLLM_WHEEL_DIR
}
if [[ $# -gt 0 ]]; then
rocm_stack_init "$1"
fi
@@ -23,7 +23,7 @@ export PATH="$CARGO_HOME/bin:$PATH"
CARGO_BINSTALL_VERSION="${CARGO_BINSTALL_VERSION:-1.20.1}" CARGO_BINSTALL_VERSION="${CARGO_BINSTALL_VERSION:-1.20.1}"
UV_VERSION="${UV_VERSION:-0.11.28}" UV_VERSION="${UV_VERSION:-0.11.28}"
UV_SCRIPT_PYTHON_VERSION="${UV_SCRIPT_PYTHON_VERSION:-3.12}" PYO3_PYTHON_VERSION="${PYO3_PYTHON_VERSION:-3.12}"
CARGO_SORT_VERSION_REQ="${CARGO_SORT_VERSION_REQ:-2}" CARGO_SORT_VERSION_REQ="${CARGO_SORT_VERSION_REQ:-2}"
CARGO_DENY_VERSION_REQ="${CARGO_DENY_VERSION_REQ:-0.20}" CARGO_DENY_VERSION_REQ="${CARGO_DENY_VERSION_REQ:-0.20}"
@@ -60,12 +60,12 @@ install_cargo_binstall() {
install_cargo_sort() { install_cargo_sort() {
log_section "Installing cargo-sort ${CARGO_SORT_VERSION_REQ}" log_section "Installing cargo-sort ${CARGO_SORT_VERSION_REQ}"
cargo binstall --no-confirm --force --locked "cargo-sort@${CARGO_SORT_VERSION_REQ}" cargo binstall --no-confirm --force "cargo-sort@${CARGO_SORT_VERSION_REQ}"
} }
install_cargo_deny() { install_cargo_deny() {
log_section "Installing cargo-deny ${CARGO_DENY_VERSION_REQ}" log_section "Installing cargo-deny ${CARGO_DENY_VERSION_REQ}"
cargo binstall --no-confirm --force --locked "cargo-deny@${CARGO_DENY_VERSION_REQ}" cargo binstall --no-confirm --force "cargo-deny@${CARGO_DENY_VERSION_REQ}"
} }
install_cargo_nextest() { install_cargo_nextest() {
@@ -74,7 +74,6 @@ install_cargo_nextest() {
--no-confirm \ --no-confirm \
--force \ --force \
--secure \ --secure \
--locked \
"cargo-nextest@${CARGO_NEXTEST_VERSION_REQ}" "cargo-nextest@${CARGO_NEXTEST_VERSION_REQ}"
} }
@@ -84,7 +83,34 @@ install_uv() {
"https://github.com/astral-sh/uv/releases/download/${UV_VERSION}/uv-installer.sh" \ "https://github.com/astral-sh/uv/releases/download/${UV_VERSION}/uv-installer.sh" \
| env UV_INSTALL_DIR="$CARGO_HOME/bin" sh | env UV_INSTALL_DIR="$CARGO_HOME/bin" sh
uv --version uv --version
uv python install "$UV_SCRIPT_PYTHON_VERSION" }
setup_pyo3_python() {
log_section "Installing Python ${PYO3_PYTHON_VERSION} for PyO3 tests"
uv python install "$PYO3_PYTHON_VERSION"
PYO3_PYTHON="$(uv python find \
--managed-python \
--no-project \
--resolve-links \
"$PYO3_PYTHON_VERSION")"
export PYO3_PYTHON
local python_libdir
python_libdir="$("$PYO3_PYTHON" - <<'PY'
import pathlib
import sysconfig
libdir = pathlib.Path(sysconfig.get_config_var("LIBDIR"))
ldlibrary = sysconfig.get_config_var("LDLIBRARY")
assert sysconfig.get_config_var("Py_ENABLE_SHARED") == 1
assert ldlibrary
assert (libdir / ldlibrary).exists(), libdir / ldlibrary
print(libdir)
PY
)"
export LD_LIBRARY_PATH="${python_libdir}:${LD_LIBRARY_PATH:-}"
export LIBRARY_PATH="${python_libdir}:${LIBRARY_PATH:-}"
} }
run_style_clippy() { run_style_clippy() {
@@ -118,6 +144,7 @@ run_style_clippy() {
run_tests() { run_tests() {
install_uv install_uv
setup_pyo3_python
install_cargo_binstall install_cargo_binstall
install_cargo_nextest install_cargo_nextest
+24
View File
@@ -0,0 +1,24 @@
#!/bin/bash
set -euo pipefail
docker_root=$(docker info -f '{{.DockerRootDir}}')
if [ -z "$docker_root" ]; then
echo "Failed to determine Docker root directory."
exit 1
fi
echo "Docker root directory: $docker_root"
# Check disk usage of the filesystem where Docker's root directory is located
disk_usage=$(df "$docker_root" | tail -1 | awk '{print $5}' | sed 's/%//')
# Define the threshold
threshold=70
if [ "$disk_usage" -gt "$threshold" ]; then
echo "Disk usage is above $threshold%. Cleaning up Docker images and volumes..."
# Remove dangling images (those that are not tagged and not used by any container)
docker image prune -f
# Remove unused volumes / force the system prune for old images as well.
docker volume prune -f && docker system prune --force --filter "until=24h" --all
echo "Docker images and volumes cleanup completed."
else
echo "Disk usage is below $threshold%. No cleanup needed."
fi
+14
View File
@@ -0,0 +1,14 @@
# Environment config
TEST_NAME=llama8b
CONTAINER_NAME=tpu-test
# vllm config
MODEL=meta-llama/Llama-3.1-8B-Instruct
MAX_NUM_SEQS=256
MAX_NUM_BATCHED_TOKENS=1024
TENSOR_PARALLEL_SIZE=1
MAX_MODEL_LEN=2048
DOWNLOAD_DIR=/mnt/disks/persist
EXPECTED_THROUGHPUT=8.0
INPUT_LEN=1800
OUTPUT_LEN=128
+91
View File
@@ -0,0 +1,91 @@
#!/bin/bash
if [ ! -f "$1" ]; then
echo "Error: The env file '$1' does not exist."
exit 1 # Exit the script with a non-zero status to indicate an error
fi
ENV_FILE=$1
# For testing on local vm, use `set -a` to export all variables
source /etc/environment
# shellcheck source=/dev/null
source "$ENV_FILE"
remove_docker_container() {
docker rm -f "$CONTAINER_NAME" || true;
}
trap remove_docker_container EXIT
# Remove the container that might not be cleaned up in the previous run.
remove_docker_container
LOG_ROOT=$(mktemp -d)
# If mktemp fails, set -e will cause the script to exit.
echo "Results will be stored in: $LOG_ROOT"
if [ -z "$HF_TOKEN" ]; then
echo "Error: HF_TOKEN is not set or is empty."
exit 1
fi
# Make sure mounted disk or dir exists
if [ ! -d "$DOWNLOAD_DIR" ]; then
echo "Error: Folder $DOWNLOAD_DIR does not exist. This is useually a mounted drive. If no mounted drive, just create a folder."
exit 1
fi
echo "Run model $MODEL"
echo
echo "starting docker...$CONTAINER_NAME"
echo
docker run \
-v "$DOWNLOAD_DIR":"$DOWNLOAD_DIR" \
--env-file "$ENV_FILE" \
-e HF_TOKEN="$HF_TOKEN" \
-e TARGET_COMMIT="$BUILDKITE_COMMIT" \
-e MODEL="$MODEL" \
-e WORKSPACE=/workspace \
--name "$CONTAINER_NAME" \
-d \
--privileged \
--network host \
-v /dev/shm:/dev/shm \
vllm/vllm-tpu-bm tail -f /dev/null
echo "run script..."
echo
docker exec "$CONTAINER_NAME" /bin/bash -c ".buildkite/scripts/tpu/run_bm.sh"
echo "copy result back..."
VLLM_LOG="$LOG_ROOT/$TEST_NAME"_vllm_log.txt
BM_LOG="$LOG_ROOT/$TEST_NAME"_bm_log.txt
docker cp "$CONTAINER_NAME:/workspace/vllm_log.txt" "$VLLM_LOG"
docker cp "$CONTAINER_NAME:/workspace/bm_log.txt" "$BM_LOG"
throughput=$(grep "Request throughput (req/s):" "$BM_LOG" | sed 's/[^0-9.]//g')
echo "throughput for $TEST_NAME at $BUILDKITE_COMMIT: $throughput"
if [ "$BUILDKITE" = "true" ]; then
echo "Running inside Buildkite"
buildkite-agent artifact upload "$VLLM_LOG"
buildkite-agent artifact upload "$BM_LOG"
else
echo "Not running inside Buildkite"
fi
#
# compare the throughput with EXPECTED_THROUGHPUT
# and assert meeting the expectation
#
if [[ -z "$throughput" || ! "$throughput" =~ ^[0-9]+([.][0-9]+)?$ ]]; then
echo "Failed to get the throughput"
exit 1
fi
if (( $(echo "$throughput < $EXPECTED_THROUGHPUT" | bc -l) )); then
echo "Error: throughput($throughput) is less than expected($EXPECTED_THROUGHPUT)"
exit 1
fi
@@ -0,0 +1,14 @@
# Environment config
TEST_NAME=llama8bw8a8
CONTAINER_NAME=tpu-test
# vllm config
MODEL=RedHatAI/Meta-Llama-3.1-8B-Instruct-quantized.w8a8
MAX_NUM_SEQS=128
MAX_NUM_BATCHED_TOKENS=1024
TENSOR_PARALLEL_SIZE=1
MAX_MODEL_LEN=2048
DOWNLOAD_DIR=/mnt/disks/persist
EXPECTED_THROUGHPUT=8.7
INPUT_LEN=1800
OUTPUT_LEN=128
+93
View File
@@ -0,0 +1,93 @@
#!/bin/bash
set -euo pipefail
VLLM_LOG="$WORKSPACE/vllm_log.txt"
BM_LOG="$WORKSPACE/bm_log.txt"
if [ -n "$TARGET_COMMIT" ]; then
head_hash=$(git rev-parse HEAD)
if [ "$TARGET_COMMIT" != "$head_hash" ]; then
echo "Error: target commit $TARGET_COMMIT does not match HEAD: $head_hash"
exit 1
fi
fi
echo "model: $MODEL"
echo
#
# create a log folder
#
mkdir "$WORKSPACE/log"
# TODO: Move to image building.
pip install pandas
pip install datasets
#
# create sonnet_4x
#
echo "Create sonnet_4x.txt"
echo "" > benchmarks/sonnet_4x.txt
for _ in {1..4}
do
cat benchmarks/sonnet.txt >> benchmarks/sonnet_4x.txt
done
#
# start vllm service in backend
#
echo "lanching vllm..."
echo "logging to $VLLM_LOG"
echo
vllm serve "$MODEL" \
--seed 42 \
--max-num-seqs "$MAX_NUM_SEQS" \
--max-num-batched-tokens "$MAX_NUM_BATCHED_TOKENS" \
--tensor-parallel-size "$TENSOR_PARALLEL_SIZE" \
--no-enable-prefix-caching \
--download_dir "$DOWNLOAD_DIR" \
--max-model-len "$MAX_MODEL_LEN" > "$VLLM_LOG" 2>&1 &
echo "wait for 20 minutes.."
echo
# sleep 1200
# wait for 10 minutes...
for _ in {1..120}; do
# TODO: detect other type of errors.
if grep -Fq "raise RuntimeError" "$VLLM_LOG"; then
echo "Detected RuntimeError, exiting."
exit 1
elif grep -Fq "Application startup complete" "$VLLM_LOG"; then
echo "Application started"
break
else
echo "wait for 10 seconds..."
sleep 10
fi
done
#
# run test
#
echo "run benchmark test..."
echo "logging to $BM_LOG"
echo
vllm bench serve \
--backend vllm \
--model "$MODEL" \
--dataset-name sonnet \
--dataset-path benchmarks/sonnet_4x.txt \
--sonnet-input-len "$INPUT_LEN" \
--sonnet-output-len "$OUTPUT_LEN" \
--ignore-eos > "$BM_LOG"
echo "completed..."
echo
throughput=$(grep "Request throughput (req/s):" "$BM_LOG" | sed 's/[^0-9.]//g')
echo "throughput: $throughput"
echo
+20 -66
View File
@@ -4,31 +4,21 @@
# #
# Upload ROCm wheels to S3 with proper index generation # Upload ROCm wheels to S3 with proper index generation
# #
# Usage: upload-rocm-wheels.sh <base-dockerfile>
# base-dockerfile the ROCm stack, e.g. docker/Dockerfile.rocm_base (see rocm/stack.sh)
#
# Required environment variables: # Required environment variables:
# AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY (or IAM role) # AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY (or IAM role)
# S3_BUCKET (default: vllm-wheels) # S3_BUCKET (default: vllm-wheels)
# #
# S3 path structure: # S3 path structure:
# s3://vllm-wheels/rocm/{commit}/<variant>-wheels/ - This stack's wheels for this commit # s3://vllm-wheels/rocm/{commit}/ - All wheels for this commit
# s3://vllm-wheels/rocm/{commit}/<variant>/ - Index per ROCm variant (e.g. rocm100) # s3://vllm-wheels/rocm/nightly/ - Index pointing to latest nightly
# s3://vllm-wheels/rocm/nightly/<variant>/ - Index pointing to latest nightly # s3://vllm-wheels/rocm/{version}/ - Index for release versions
# s3://vllm-wheels/rocm/{version}/<variant>/ - Index for release versions
# The top-level index.html at each prefix lists every variant present, since
# several ROCm stacks publish into the same prefixes.
set -ex set -ex
# ======== Configuration ======== # ======== Configuration ========
BUCKET="${S3_BUCKET:-vllm-wheels}" BUCKET="${S3_BUCKET:-vllm-wheels}"
# shellcheck source=.buildkite/scripts/rocm/stack.sh
source "$(dirname "${BASH_SOURCE[0]}")/rocm/stack.sh" "${1:?Usage: $0 <base-dockerfile>}"
ROCM_SUBPATH="rocm/${BUILDKITE_COMMIT}" ROCM_SUBPATH="rocm/${BUILDKITE_COMMIT}"
WHEEL_DIRS="$ROCM_STACK_BASE_WHEELS_DIR $ROCM_STACK_VLLM_WHEEL_DIR" S3_COMMIT_PREFIX="s3://$BUCKET/$ROCM_SUBPATH/"
WHEEL_SUBPATH="$ROCM_SUBPATH/${ROCM_STACK_VARIANT}-wheels"
S3_WHEEL_PREFIX="s3://$BUCKET/$WHEEL_SUBPATH/"
INDICES_OUTPUT_DIR="rocm-indices" INDICES_OUTPUT_DIR="rocm-indices"
echo "========================================" echo "========================================"
@@ -59,11 +49,9 @@ source .buildkite/scripts/lib/manylinux.sh
# ======== Part 1: Collect and prepare wheels ======== # ======== Part 1: Collect and prepare wheels ========
# Collect all wheels # Collect all wheels
rm -rf all-rocm-wheels "$INDICES_OUTPUT_DIR"
mkdir -p all-rocm-wheels mkdir -p all-rocm-wheels
for dir in $WHEEL_DIRS; do cp artifacts/rocm-base-wheels/*.whl all-rocm-wheels/ 2>/dev/null || true
cp "$dir"/*.whl all-rocm-wheels/ 2>/dev/null || true cp artifacts/rocm-vllm-wheel/*.whl all-rocm-wheels/ 2>/dev/null || true
done
WHEEL_COUNT=$(find all-rocm-wheels -maxdepth 1 -name '*.whl' 2>/dev/null | wc -l) WHEEL_COUNT=$(find all-rocm-wheels -maxdepth 1 -name '*.whl' 2>/dev/null | wc -l)
echo "Total wheels to upload: $WHEEL_COUNT" echo "Total wheels to upload: $WHEEL_COUNT"
@@ -96,9 +84,9 @@ ls -lh all-rocm-wheels/
# ======== Part 2: Upload wheels to S3 ======== # ======== Part 2: Upload wheels to S3 ========
echo "" echo ""
echo "Uploading wheels to $S3_WHEEL_PREFIX" echo "Uploading wheels to $S3_COMMIT_PREFIX"
for wheel in all-rocm-wheels/*.whl; do for wheel in all-rocm-wheels/*.whl; do
aws s3 cp "$wheel" "$S3_WHEEL_PREFIX" aws s3 cp "$wheel" "$S3_COMMIT_PREFIX"
done done
# ======== Part 3: Generate and upload indices ======== # ======== Part 3: Generate and upload indices ========
@@ -107,63 +95,28 @@ done
echo "" echo ""
echo "Generating indices..." echo "Generating indices..."
obj_json="rocm-objects.json" obj_json="rocm-objects.json"
aws s3api list-objects-v2 --bucket "$BUCKET" --prefix "$WHEEL_SUBPATH/" --delimiter / --output json > "$obj_json" aws s3api list-objects-v2 --bucket "$BUCKET" --prefix "$ROCM_SUBPATH/" --delimiter / --output json > "$obj_json"
mkdir -p "$INDICES_OUTPUT_DIR" mkdir -p "$INDICES_OUTPUT_DIR"
# Use the existing generate-nightly-index.py # Use the existing generate-nightly-index.py
# HACK: Replace regex module with stdlib re (same as CUDA script) # HACK: Replace regex module with stdlib re (same as CUDA script)
sed -i 's/import regex as re/import re/g' .buildkite/scripts/generate-nightly-index.py sed -i 's/import regex as re/import re/g' .buildkite/scripts/generate-nightly-index.py
sed -i 's/import regex as re/import re/g' tools/vllm-rocm/therock_wheels.py
sed -i 's/import pybase64 as base64/import base64/g' tools/vllm-rocm/therock_wheels.py
INDEX_ARGS=(--version "$ROCM_SUBPATH" --wheel-dir "${WHEEL_SUBPATH#rocm/}")
# TheRock's SDK, device kernels and torchvision/torchaudio are linked, not hosted
if [[ "$ROCM_STACK_THEROCK" == "1" ]]; then
$PYTHON tools/vllm-rocm/therock_wheels.py external-links \
--dockerfile "$ROCM_STACK_BASE_DOCKERFILE" \
--torch-wheel "$(ls "$ROCM_STACK_BASE_WHEELS_DIR"/torch-*.whl)" > therock-external-links.txt
cat therock-external-links.txt
INDEX_ARGS+=(--external-links therock-external-links.txt)
fi
$PYTHON .buildkite/scripts/generate-nightly-index.py \ $PYTHON .buildkite/scripts/generate-nightly-index.py \
"${INDEX_ARGS[@]}" \ --version "$ROCM_SUBPATH" \
--current-objects "$obj_json" \ --current-objects "$obj_json" \
--output-dir "$INDICES_OUTPUT_DIR" \ --output-dir "$INDICES_OUTPUT_DIR" \
--comment "ROCm commit $BUILDKITE_COMMIT" --comment "ROCm commit $BUILDKITE_COMMIT"
# Upload this build's variant indices under an S3 prefix (e.g. rocm/nightly), then # Upload indices to commit directory
# rebuild that prefix's top-level project list from every variant now present. echo "Uploading indices to $S3_COMMIT_PREFIX"
publish_indices() { aws s3 cp --recursive "$INDICES_OUTPUT_DIR/" "$S3_COMMIT_PREFIX"
local prefix="$1"
local listing variants merged
echo "Uploading indices to s3://$BUCKET/$prefix/"
aws s3 cp --recursive "$INDICES_OUTPUT_DIR/" "s3://$BUCKET/$prefix/" --exclude "index.html"
listing=$(aws s3api list-objects-v2 --bucket "$BUCKET" --prefix "$prefix/" \
--delimiter / --query 'CommonPrefixes[].Prefix' --output text)
variants=$(tr '\t' '\n' <<< "$listing" | sed -n "s|^$prefix/\(rocm[0-9]*\)/$|\1|p" | sort -u)
merged=$(mktemp)
{
echo '<!DOCTYPE html>'
echo '<html>'
echo ' <meta name="pypi:repository-version" content="1.0">'
echo ' <body>'
for variant in $variants; do
echo " <a href=\"$variant/\">$variant/</a><br/>"
done
echo ' </body>'
echo '</html>'
} > "$merged"
echo "Variants under $prefix/: $(echo "$variants" | tr '\n' ' ')"
aws s3 cp "$merged" "s3://$BUCKET/$prefix/index.html"
rm -f "$merged"
}
publish_indices "$ROCM_SUBPATH"
# Only scheduled nightly builds should update the moving nightly index. # Only scheduled nightly builds should update the moving nightly index.
if [[ "${NIGHTLY:-0}" == "1" ]]; then if [[ "${NIGHTLY:-0}" == "1" ]]; then
publish_indices "rocm/nightly" echo "Updating rocm/nightly/ index..."
aws s3 cp --recursive "$INDICES_OUTPUT_DIR/" "s3://$BUCKET/rocm/nightly/"
fi fi
# Extract version from vLLM wheel and update version-specific index # Extract version from vLLM wheel and update version-specific index
@@ -176,7 +129,8 @@ if [ -n "$VLLM_WHEEL" ]; then
echo "Pure version: $PURE_VERSION" echo "Pure version: $PURE_VERSION"
if [[ "$VERSION" != *"dev"* ]]; then if [[ "$VERSION" != *"dev"* ]]; then
publish_indices "rocm/$PURE_VERSION" echo "Updating rocm/$PURE_VERSION/ index..."
aws s3 cp --recursive "$INDICES_OUTPUT_DIR/" "s3://$BUCKET/rocm/$PURE_VERSION/"
fi fi
fi fi
@@ -188,14 +142,14 @@ echo "ROCm Wheel Upload Complete!"
echo "========================================" echo "========================================"
echo "" echo ""
echo "Wheels available at:" echo "Wheels available at:"
echo " s3://$BUCKET/$WHEEL_SUBPATH/" echo " s3://$BUCKET/$ROCM_SUBPATH/"
echo "" echo ""
echo "Install command (by commit):" echo "Install command (by commit):"
echo " pip install vllm --extra-index-url https://${BUCKET}.s3.amazonaws.com/$ROCM_SUBPATH/$ROCM_STACK_VARIANT/" echo " pip install vllm --extra-index-url https://${BUCKET}.s3.amazonaws.com/$ROCM_SUBPATH/"
echo "" echo ""
if [[ "${NIGHTLY:-0}" == "1" ]]; then if [[ "${NIGHTLY:-0}" == "1" ]]; then
echo "Install command (nightly):" echo "Install command (nightly):"
echo " pip install vllm --extra-index-url https://${BUCKET}.s3.amazonaws.com/rocm/nightly/$ROCM_STACK_VARIANT/" echo " pip install vllm --extra-index-url https://${BUCKET}.s3.amazonaws.com/rocm/nightly/"
fi fi
echo "" echo ""
echo "Wheel count: $WHEEL_COUNT" echo "Wheel count: $WHEEL_COUNT"
+415 -484
View File
File diff suppressed because it is too large Load Diff
+8 -35
View File
@@ -2,9 +2,8 @@ group: Attention
depends_on: depends_on:
- image-build - image-build
steps: steps:
- label: ":nvidia: (H100) V1 Attention" - label: ":nvidia: (H100) V1 Attention Shard %N"
key: v1-attention-h100-mi300 key: v1-attention-h100-mi300
automatic_shard: true
timeout_in_minutes: 85 timeout_in_minutes: 85
device: h100 device: h100
source_file_dependencies: source_file_dependencies:
@@ -13,18 +12,14 @@ steps:
- vllm/v1/attention - vllm/v1/attention
- tests/v1/attention - tests/v1/attention
commands: commands:
- pytest -v -s v1/attention - pytest -v -s v1/attention --shard-id=$$BUILDKITE_PARALLEL_JOB --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT
parallelism: 2
mirror: mirror:
amd: amd:
label: ":amd: (MI300) V1 Attention Shard %N" label: ":amd: (MI300) V1 Attention Shard %N"
dind: false dind: false
device: mi300_1 device: mi300_1
timeout_in_minutes: 125 timeout_in_minutes: 125
# automatic_shard covers NVIDIA steps only; the mirror keeps its
# fixed pytest shards.
parallelism: 2
commands:
- pytest -v -s v1/attention --shard-id=$$BUILDKITE_PARALLEL_JOB --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT
depends_on: depends_on:
- image-build-amd - image-build-amd
source_file_dependencies: source_file_dependencies:
@@ -35,14 +30,9 @@ steps:
- vllm/_aiter_ops.py - vllm/_aiter_ops.py
- vllm/envs.py - vllm/envs.py
- vllm/platforms/rocm.py - vllm/platforms/rocm.py
- vllm/models/deepseek_v32/amd/rocm.py
commands:
- pytest -v -s v1/attention --ignore=v1/attention/test_sparse_mla_backends.py --shard-id=$$BUILDKITE_PARALLEL_JOB --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT
- VLLM_ROCM_USE_AITER=1 pytest -v -s v1/attention/test_sparse_mla_backends.py --shard-id=$$BUILDKITE_PARALLEL_JOB --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT
- label: ":nvidia: (B200) V1 Attention" - label: ":nvidia: (B200) V1 Attention Shard %N"
key: v1-attention-b200 key: v1-attention-b200
automatic_shard: true
timeout_in_minutes: 80 timeout_in_minutes: 80
device: b200-k8s device: b200-k8s
source_file_dependencies: source_file_dependencies:
@@ -51,16 +41,14 @@ steps:
- vllm/v1/attention - vllm/v1/attention
- tests/v1/attention - tests/v1/attention
commands: commands:
- pytest -v -s v1/attention - pytest -v -s v1/attention --shard-id=$$BUILDKITE_PARALLEL_JOB --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT
parallelism: 2
mirror: mirror:
amd: amd:
label: ":amd: (MI355 DPX) V1 Attention Shard %N" label: ":amd: (MI355 DPX) V1 Attention Shard %N"
dind: false dind: false
device: mi355_dpx device: mi355_dpx
timeout_in_minutes: 125 timeout_in_minutes: 125
# automatic_shard covers NVIDIA steps only; the mirror keeps its
# fixed pytest shards.
parallelism: 2
depends_on: depends_on:
- image-build-amd - image-build-amd
source_file_dependencies: source_file_dependencies:
@@ -71,21 +59,6 @@ steps:
- vllm/_aiter_ops.py - vllm/_aiter_ops.py
- vllm/envs.py - vllm/envs.py
- vllm/platforms/rocm.py - vllm/platforms/rocm.py
- vllm/models/deepseek_v32/amd/rocm.py
commands: commands:
- pytest -v -s v1/attention --ignore=v1/attention/test_mla_backends.py --ignore=v1/attention/test_sparse_mla_backends.py --shard-id=$$BUILDKITE_PARALLEL_JOB --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT - pytest -v -s v1/attention --ignore=v1/attention/test_mla_backends.py --shard-id=$$BUILDKITE_PARALLEL_JOB --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT
- VLLM_ROCM_USE_AITER=1 pytest -v -s v1/attention/test_mla_backends.py v1/attention/test_sparse_mla_backends.py --shard-id=$$BUILDKITE_PARALLEL_JOB --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT - VLLM_ROCM_USE_AITER=1 pytest -v -s v1/attention/test_mla_backends.py --shard-id=$$BUILDKITE_PARALLEL_JOB --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT
- label: ":nvidia: (DGX) Spark FlashInfer NVFP4 XQA"
key: flashinfer-nvfp4-xqa-dgx-spark
timeout_in_minutes: 30
device: dgx-spark
optional: true
num_devices: 1
depends_on:
- arm64-image-build
source_file_dependencies:
- vllm/v1/attention/backends/flashinfer.py
- tests/v1/attention/test_attention_backends.py
commands:
- pytest -v -s v1/attention/test_attention_backends.py -k test_flashinfer_xqa_nvfp4_decode_correctness
+1 -1
View File
@@ -4,7 +4,7 @@ depends_on:
steps: steps:
- label: ":nvidia: (H200 MIG 18GB) Benchmarks CLI" - label: ":nvidia: (H200 MIG 18GB) Benchmarks CLI"
key: benchmarks-cli-test key: benchmarks-cli-test
timeout_in_minutes: 55 timeout_in_minutes: 45
device: h200_18gb device: h200_18gb
source_file_dependencies: source_file_dependencies:
- vllm/ - vllm/
+128
View File
@@ -2,6 +2,84 @@ group: Compile
depends_on: depends_on:
- image-build - image-build
steps: steps:
- label: ":nvidia: (B200) Sequence Parallel Correctness"
key: sequence-parallel-correctness-tests-2xb200
timeout_in_minutes: 45
working_dir: "/vllm-workspace/"
device: b200-k8s
optional: true
num_devices: 2
commands:
- export VLLM_TEST_CLEAN_GPU_MEMORY=1
- pytest -v -s tests/compile/correctness_e2e/test_sequence_parallel.py
mirror:
amd:
label: ":amd: (MI355) Sequence Parallel Correctness"
dind: false
device: mi355_2
num_devices: 2
timeout_in_minutes: 180
depends_on:
- image-build-amd
source_file_dependencies:
- .buildkite/test_areas/compile.yaml
- vllm/compilation/
- vllm/config/
- vllm/distributed/
- vllm/model_executor/kernels/linear/
- vllm/model_executor/layers/
- vllm/model_executor/models/llama.py
- vllm/v1/attention/
- vllm/v1/worker/
- vllm/v1/cudagraph_dispatcher.py
- vllm/platforms/rocm.py
- tests/compile/correctness_e2e/test_sequence_parallel.py
- tests/models/registry.py
- tests/utils.py
- label: ":nvidia: (B200) AsyncTP Correctness"
key: asynctp-correctness-tests-b200
timeout_in_minutes: 30
working_dir: "/vllm-workspace/"
device: b200-k8s
optional: true
num_devices: 2
commands:
- export VLLM_TEST_CLEAN_GPU_MEMORY=1
- pytest -v -s tests/compile/correctness_e2e/test_async_tp.py
mirror:
amd:
label: ":amd: (MI355) AsyncTP Correctness"
dind: false
device: mi355_2
num_devices: 2
timeout_in_minutes: 60
working_dir: "/vllm-workspace/"
depends_on:
- image-build-amd
source_file_dependencies:
- .buildkite/test_areas/compile.yaml
- vllm/compilation/
- vllm/config/
- vllm/distributed/
- vllm/model_executor/layers/
- vllm/model_executor/kernels/linear/
- vllm/model_executor/models/llama.py
- vllm/model_executor/models/qwen3.py
- vllm/v1/attention/
- vllm/_aiter_ops.py
- vllm/platforms/rocm.py
- vllm/transformers_utils/repo_utils.py
- tests/compile/correctness_e2e/test_async_tp.py
- tests/compile/fusions_e2e/common.py
- tests/models/registry.py
- tests/utils.py
commands:
- export VLLM_TEST_CLEAN_GPU_MEMORY=1
- pytest -v -s tests/compile/correctness_e2e/test_async_tp.py::test_rocm_async_tp_bf16_output_correctness
- pytest -v -s tests/compile/correctness_e2e/test_async_tp.py::test_async_tp_pass_correctness -k RedHatAI
- pytest -v -s tests/compile/correctness_e2e/test_async_tp.py::test_async_tp_pass_nvfp4_correctness
- label: ":nvidia: (H100) Distributed Compile" - label: ":nvidia: (H100) Distributed Compile"
key: distributed-compile-unit-tests-2xh100 key: distributed-compile-unit-tests-2xh100
timeout_in_minutes: 45 timeout_in_minutes: 45
@@ -222,6 +300,7 @@ steps:
- nvidia-smi - nvidia-smi
# Run all models and attn backends but only Inductor partition and native custom ops # Run all models and attn backends but only Inductor partition and native custom ops
- pytest -v -s tests/compile/fusions_e2e/test_tp2_ar_rms.py -k "inductor_partition and not +rms_norm and (not +quant_fp8 or +quant_fp8 and (qwen3 or deepseek))" - pytest -v -s tests/compile/fusions_e2e/test_tp2_ar_rms.py -k "inductor_partition and not +rms_norm and (not +quant_fp8 or +quant_fp8 and (qwen3 or deepseek))"
- pytest -v -s tests/compile/fusions_e2e/test_tp2_async_tp.py -k "inductor_partition and not +rms_norm and (not +quant_fp8 or +quant_fp8 and (qwen3 or deepseek))"
mirror: mirror:
amd: amd:
label: ":amd: (MI355) Fusion E2E TP2 Quick" label: ":amd: (MI355) Fusion E2E TP2 Quick"
@@ -309,6 +388,53 @@ steps:
- amd-smi - amd-smi
- VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_CUSTOM_AR=1 pytest -v -s tests/compile/fusions_e2e/test_tp2_ar_rms.py -k 'test_tp2_ar_rms_fp8_fusions and Llama-3 and ROCM_AITER_UNIFIED_ATTN and (dynamo_partition or +quant_fp8 or +rms_norm)' - VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_CUSTOM_AR=1 pytest -v -s tests/compile/fusions_e2e/test_tp2_ar_rms.py -k 'test_tp2_ar_rms_fp8_fusions and Llama-3 and ROCM_AITER_UNIFIED_ATTN and (dynamo_partition or +quant_fp8 or +rms_norm)'
- label: ":nvidia: (H100) Fusion E2E TP2 AsyncTP Config Sweep"
key: fusion-e2e-tp2-asynctp-config-sweep-h100
timeout_in_minutes: 40
working_dir: "/vllm-workspace/"
device: h100
num_devices: 2
source_file_dependencies:
- csrc/quantization/
- vllm/compilation/
# can affect pattern matching
- vllm/model_executor/layers/layernorm.py
- vllm/model_executor/layers/activation.py
- vllm/model_executor/layers/attention/attention.py
- vllm/model_executor/layers/quantization/input_quant_fp8.py
- tests/compile/fusions_e2e/
commands:
- nvidia-smi
# Run just llama3 (fp8 & bf16) for all config combinations
- pytest -v -s tests/compile/fusions_e2e/test_tp2_async_tp.py -k "llama-3"
mirror:
amd:
label: ':amd: (MI355) Fusion E2E TP2 AsyncTP BF16 Config Sweep'
dind: false
device: mi355_2
num_devices: 2
working_dir: /vllm-workspace/
timeout_in_minutes: 30
depends_on:
- image-build-amd
source_file_dependencies:
- vllm/compilation/
- vllm/model_executor/layers/linear.py
- vllm/model_executor/layers/utils.py
- vllm/distributed/
- vllm/v1/attention/backends/triton_attn.py
- tests/compile/fusions_e2e/
- tests/compile/passes/distributed/test_async_tp.py
- tests/compile/rocm/test_async_tp.py
- tests/compile/correctness_e2e/test_async_tp.py
- vllm/model_executor/kernels/linear/
- vllm/model_executor/models/qwen3.py
commands:
- rocm-smi
- pytest -v -s tests/compile/rocm/test_async_tp.py
- pytest -v -s tests/compile/fusions_e2e/test_tp2_rocm_async_tp.py
- pytest -v -s tests/compile/correctness_e2e/test_async_tp.py::test_rocm_async_tp_bf16_output_correctness
- label: ":nvidia: (B200) Fusion E2E TP2" - label: ":nvidia: (B200) Fusion E2E TP2"
key: fusion-e2e-tp2-b200 key: fusion-e2e-tp2-b200
timeout_in_minutes: 45 timeout_in_minutes: 45
@@ -327,6 +453,7 @@ steps:
# include qwen/deepseek with +quant_fp8 as -quant_fp8 rms+quant fusion is not supported # include qwen/deepseek with +quant_fp8 as -quant_fp8 rms+quant fusion is not supported
# for ar-rms-quant-fp4, also sweep llama3 # for ar-rms-quant-fp4, also sweep llama3
- pytest -v -s tests/compile/fusions_e2e/test_tp2_ar_rms.py -k "(FLASHINFER and inductor_partition and not +rms_norm and (not +quant_fp8 or +quant_fp8 and (qwen3 or deepseek))) or Llama-3.1-8B-Instruct-FP4" - pytest -v -s tests/compile/fusions_e2e/test_tp2_ar_rms.py -k "(FLASHINFER and inductor_partition and not +rms_norm and (not +quant_fp8 or +quant_fp8 and (qwen3 or deepseek))) or Llama-3.1-8B-Instruct-FP4"
- pytest -v -s tests/compile/fusions_e2e/test_tp2_async_tp.py -k "FLASHINFER and inductor_partition and not +rms_norm and (not +quant_fp8 or +quant_fp8 and (qwen3 or deepseek))"
mirror: mirror:
amd: amd:
label: ":amd: (MI355) Fusion E2E TP2 FP8 BF16 rewrites" label: ":amd: (MI355) Fusion E2E TP2 FP8 BF16 rewrites"
@@ -353,6 +480,7 @@ steps:
# TODO: FP4 output-quant and trained MXFP4 AR/RMS fusion remain unsupported. # TODO: FP4 output-quant and trained MXFP4 AR/RMS fusion remain unsupported.
- amd-smi - amd-smi
- VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_CUSTOM_AR=1 pytest -v -s tests/compile/fusions_e2e/test_tp2_ar_rms.py -k 'test_tp2_ar_rms_fp8_fusions and qwen3 and ROCM_AITER_UNIFIED_ATTN and +quant_fp8 and (dynamo_partition or not +rms_norm)' - VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_CUSTOM_AR=1 pytest -v -s tests/compile/fusions_e2e/test_tp2_ar_rms.py -k 'test_tp2_ar_rms_fp8_fusions and qwen3 and ROCM_AITER_UNIFIED_ATTN and +quant_fp8 and (dynamo_partition or not +rms_norm)'
- pytest -v -s tests/compile/fusions_e2e/test_tp2_rocm_moe.py
- label: ':amd: (MI355) Fusion E2E TP2 AR-RMS (Inductor Partition)' - label: ':amd: (MI355) Fusion E2E TP2 AR-RMS (Inductor Partition)'
key: fusion-e2e-tp2-ar-rms-inductor-partition-amd key: fusion-e2e-tp2-ar-rms-inductor-partition-amd
+1 -7
View File
@@ -33,20 +33,14 @@ steps:
- label: ":nvidia: (H200 MIG 18GB) CUDAGraph" - label: ":nvidia: (H200 MIG 18GB) CUDAGraph"
device: h200_18gb device: h200_18gb
key: cudagraph key: cudagraph
automatic_shard: true timeout_in_minutes: 30
timeout_in_minutes: 45
source_file_dependencies: source_file_dependencies:
- tests/v1/cudagraph - tests/v1/cudagraph
- vllm/v1/cudagraph_dispatcher.py - vllm/v1/cudagraph_dispatcher.py
- vllm/config/compilation.py - vllm/config/compilation.py
- vllm/config/vllm.py
- vllm/compilation - vllm/compilation
- vllm/v1/worker/gpu/cudagraph_utils.py
- vllm/v1/worker/gpu/dp_utils.py
- vllm/v1/worker/encoder_cudagraph.py - vllm/v1/worker/encoder_cudagraph.py
- vllm/v1/worker/encoder_cudagraph_defs.py - vllm/v1/worker/encoder_cudagraph_defs.py
- vllm/v1/worker/gpu/model_runner.py
- vllm/v1/worker/gpu/ubatch_utils.py
commands: commands:
- pytest -v -s v1/cudagraph/test_cudagraph_dispatch.py - pytest -v -s v1/cudagraph/test_cudagraph_dispatch.py
- pytest -v -s v1/cudagraph/test_cudagraph_mode.py - pytest -v -s v1/cudagraph/test_cudagraph_mode.py
+1 -1
View File
@@ -426,7 +426,7 @@ steps:
# Keep TP=2 prefill and TP=1 decode above both hybrid page-size minima. # Keep TP=2 prefill and TP=1 decode above both hybrid page-size minima.
PREFILL_BLOCK_SIZE: "2048" PREFILL_BLOCK_SIZE: "2048"
DECODE_BLOCK_SIZE: "2048" DECODE_BLOCK_SIZE: "2048"
VLLM_SERVE_EXTRA_ARGS: "--trust-remote-code,--max-model-len 32768" VLLM_SERVE_EXTRA_ARGS: "--trust-remote-code,--mamba-cache-mode align,--max-model-len 32768"
commands: commands:
- bash /vllm-workspace/.buildkite/scripts/install-kv-connectors.sh - bash /vllm-workspace/.buildkite/scripts/install-kv-connectors.sh
- bash v1/kv_connector/nixl_integration/run_accuracy_test.sh - bash v1/kv_connector/nixl_integration/run_accuracy_test.sh
@@ -4,8 +4,6 @@ depends_on:
steps: steps:
- label: ":nvidia: (B200) Distributed MooncakeConnector PD accuracy" - label: ":nvidia: (B200) Distributed MooncakeConnector PD accuracy"
key: distributed-mooncakeconnector-pd-accuracy-4-gpus key: distributed-mooncakeconnector-pd-accuracy-4-gpus
# Nightly only: 4-GPU B200 pods wait hours behind smaller jobs on per-PR runs.
optional: true
timeout_in_minutes: 10 timeout_in_minutes: 10
working_dir: "/vllm-workspace/tests" working_dir: "/vllm-workspace/tests"
# Runs on b200-k8s because it is the only NVIDIA pool that can carry mooncake # Runs on b200-k8s because it is the only NVIDIA pool that can carry mooncake
@@ -24,7 +22,6 @@ steps:
mirror: mirror:
amd: amd:
label: ":amd: (MI355) Distributed MooncakeConnector HIP PD accuracy" label: ":amd: (MI355) Distributed MooncakeConnector HIP PD accuracy"
optional: true
dind: false dind: false
device: mi355_4 device: mi355_4
num_devices: 4 num_devices: 4
+6 -17
View File
@@ -14,7 +14,6 @@ steps:
- tests/distributed - tests/distributed
commands: commands:
- pytest -v -s distributed/test_comm_ops.py - pytest -v -s distributed/test_comm_ops.py
- pytest -v -s distributed/test_cpu_all_reduce.py
- pytest -v -s distributed/test_shm_broadcast.py - pytest -v -s distributed/test_shm_broadcast.py
- pytest -v -s distributed/test_shm_buffer.py - pytest -v -s distributed/test_shm_buffer.py
- pytest -v -s distributed/test_shm_storage.py - pytest -v -s distributed/test_shm_storage.py
@@ -78,7 +77,6 @@ steps:
- label: ":nvidia: (L4) Distributed DP Basic" - label: ":nvidia: (L4) Distributed DP Basic"
key: distributed-dp-tests-2-gpus key: distributed-dp-tests-2-gpus
automatic_shard: true
timeout_in_minutes: 45 timeout_in_minutes: 45
working_dir: "/vllm-workspace/tests" working_dir: "/vllm-workspace/tests"
device: l4 device: l4
@@ -110,7 +108,6 @@ steps:
amd: amd:
label: ":amd: (MI355) Distributed DP Basic" label: ":amd: (MI355) Distributed DP Basic"
dind: false dind: false
automatic_shard: true
device: mi355_2 device: mi355_2
timeout_in_minutes: 45 timeout_in_minutes: 45
depends_on: depends_on:
@@ -303,7 +300,6 @@ steps:
- label: ":nvidia: (L4) Distributed DP Extended" - label: ":nvidia: (L4) Distributed DP Extended"
key: distributed-dp-tests-4-gpus key: distributed-dp-tests-4-gpus
automatic_shard: true
timeout_in_minutes: 45 timeout_in_minutes: 45
working_dir: "/vllm-workspace/tests" working_dir: "/vllm-workspace/tests"
device: l4 device: l4
@@ -333,7 +329,6 @@ steps:
amd: amd:
label: ':amd: (MI355) Distributed DP Extended' label: ':amd: (MI355) Distributed DP Extended'
dind: false dind: false
automatic_shard: true
device: mi355_4 device: mi355_4
timeout_in_minutes: 60 timeout_in_minutes: 60
working_dir: /vllm-workspace/tests working_dir: /vllm-workspace/tests
@@ -402,7 +397,6 @@ steps:
- label: ":nvidia: (H100) Distributed DP + EP" - label: ":nvidia: (H100) Distributed DP + EP"
key: distributed-tests-8xh100 key: distributed-tests-8xh100
timeout_in_minutes: 20 timeout_in_minutes: 20
optional: true
device: h100 device: h100
num_devices: 8 num_devices: 8
working_dir: "/vllm-workspace/tests" working_dir: "/vllm-workspace/tests"
@@ -425,7 +419,6 @@ steps:
amd: amd:
label: ":amd: (MI355) Distributed DP + EP" label: ":amd: (MI355) Distributed DP + EP"
dind: false dind: false
optional: false
device: mi355_8 device: mi355_8
timeout_in_minutes: 50 timeout_in_minutes: 50
depends_on: depends_on:
@@ -513,7 +506,9 @@ steps:
- examples/features/data_parallel/data_parallel_offline.py - examples/features/data_parallel/data_parallel_offline.py
- vllm/_aiter_ops.py - vllm/_aiter_ops.py
- vllm/platforms/rocm.py - vllm/platforms/rocm.py
- csrc/rocm/quickreduce/ - csrc/custom_quickreduce.cu
- csrc/ops.h
- csrc/torch_bindings.cpp
- vllm/model_executor/layers/ - vllm/model_executor/layers/
- vllm/entrypoints/llm.py - vllm/entrypoints/llm.py
- vllm/config/parallel.py - vllm/config/parallel.py
@@ -525,17 +520,13 @@ steps:
- tests/distributed/test_rocm_aiter_custom_ar.py - tests/distributed/test_rocm_aiter_custom_ar.py
- tests/distributed/test_rocm_quick_reduce.py - tests/distributed/test_rocm_quick_reduce.py
- tests/distributed/test_quick_all_reduce.py - tests/distributed/test_quick_all_reduce.py
- tests/distributed/test_aiter_rms_fallback.py
- tests/v1/e2e/general/test_rocm_aiter_custom_ar.py - tests/v1/e2e/general/test_rocm_aiter_custom_ar.py
- tests/utils.py - tests/utils.py
- examples/rl/rlhf_async_new_apis.py - examples/rl/rlhf_async_new_apis.py
- tests/distributed/test_weight_transfer.py - tests/distributed/test_weight_transfer.py
- tests/distributed/test_packed_tensor.py - tests/distributed/test_packed_tensor.py
- tests/v1/attention/test_rocm_mla_dcp_context.py
- vllm/v1/attention/ops/rocm_aiter_mla_prefill.py
commands: commands:
- pytest -v -s tests/distributed/test_context_parallel.py - pytest -v -s tests/distributed/test_context_parallel.py
- pytest -v -s tests/v1/attention/test_rocm_mla_dcp_context.py
- VLLM_ALLOW_INSECURE_SERIALIZATION=1 python3 examples/rl/rlhf_async_new_apis.py - VLLM_ALLOW_INSECURE_SERIALIZATION=1 python3 examples/rl/rlhf_async_new_apis.py
- VLLM_LOGGING_LEVEL=DEBUG python3 examples/features/data_parallel/data_parallel_offline.py --model=Qwen/Qwen1.5-MoE-A2.7B -tp=1 -dp=2 --max-model-len=2048 --all2all-backend=deepep_high_throughput - VLLM_LOGGING_LEVEL=DEBUG python3 examples/features/data_parallel/data_parallel_offline.py --model=Qwen/Qwen1.5-MoE-A2.7B -tp=1 -dp=2 --max-model-len=2048 --all2all-backend=deepep_high_throughput
- VLLM_LOGGING_LEVEL=DEBUG python3 examples/features/data_parallel/data_parallel_offline.py --model=Qwen/Qwen1.5-MoE-A2.7B -tp=1 -dp=2 --max-model-len=2048 --all2all-backend=allgather_reducescatter --disable-nccl-for-dp-synchronization - VLLM_LOGGING_LEVEL=DEBUG python3 examples/features/data_parallel/data_parallel_offline.py --model=Qwen/Qwen1.5-MoE-A2.7B -tp=1 -dp=2 --max-model-len=2048 --all2all-backend=allgather_reducescatter --disable-nccl-for-dp-synchronization
@@ -546,7 +537,6 @@ steps:
- pytest -v -s tests/v1/e2e/general/test_rocm_aiter_custom_ar.py - pytest -v -s tests/v1/e2e/general/test_rocm_aiter_custom_ar.py
- pytest -v -s tests/distributed/test_rocm_quick_reduce.py - pytest -v -s tests/distributed/test_rocm_quick_reduce.py
- pytest -v -s tests/distributed/test_quick_all_reduce.py - pytest -v -s tests/distributed/test_quick_all_reduce.py
- pytest -v -s tests/distributed/test_aiter_rms_fallback.py
- label: ":nvidia: (B200) Distributed" - label: ":nvidia: (B200) Distributed"
key: distributed-tests-2xb200 key: distributed-tests-2xb200
@@ -640,11 +630,10 @@ steps:
- tests/distributed/ - tests/distributed/
- examples/features/data_parallel/data_parallel_offline.py - examples/features/data_parallel/data_parallel_offline.py
commands: commands:
- ./.buildkite/scripts/run-multi-node-test.sh /vllm-workspace/tests 2 2 $IMAGE_TAG "VLLM_TEST_SAME_HOST=0 torchrun --nnodes 2 --nproc-per-node=2 --rdzv_backend=c10d --rdzv_endpoint=192.168.10.10 distributed/test_same_node.py | grep 'Same node test passed' && NUM_NODES=2 torchrun --nnodes 2 --nproc-per-node=2 --rdzv_backend=c10d --rdzv_endpoint=192.168.10.10 distributed/test_node_count.py | grep 'Node count test passed' && python3 ../examples/features/data_parallel/data_parallel_offline.py -dp=2 -tp=1 --dp-num-nodes=2 --dp-node-rank=0 --dp-master-addr=192.168.10.10 --dp-master-port=12345 --enforce-eager --trust-remote-code && VLLM_MULTI_NODE=1 pytest -v -s distributed/test_pipeline_parallel.py" "VLLM_TEST_SAME_HOST=0 torchrun --nnodes 2 --nproc-per-node=2 --rdzv_backend=c10d --rdzv_endpoint=192.168.10.10 distributed/test_same_node.py | grep 'Same node test passed' && NUM_NODES=2 torchrun --nnodes 2 --nproc-per-node=2 --rdzv_backend=c10d --rdzv_endpoint=192.168.10.10 distributed/test_node_count.py | grep 'Node count test passed' && python3 ../examples/features/data_parallel/data_parallel_offline.py -dp=2 -tp=1 --dp-num-nodes=2 --dp-node-rank=1 --dp-master-addr=192.168.10.10 --dp-master-port=12345 --enforce-eager --trust-remote-code" - ./.buildkite/scripts/run-multi-node-test.sh /vllm-workspace/tests 2 2 $IMAGE_TAG "VLLM_TEST_SAME_HOST=0 torchrun --nnodes 2 --nproc-per-node=2 --rdzv_backend=c10d --rdzv_endpoint=192.168.10.10 distributed/test_same_node.py | grep 'Same node test passed' && NUM_NODES=2 torchrun --nnodes 2 --nproc-per-node=2 --rdzv_backend=c10d --rdzv_endpoint=192.168.10.10 distributed/test_node_count.py | grep 'Node count test passed' && python3 ../examples/features/data_parallel/data_parallel_offline.py -dp=2 -tp=1 --dp-num-nodes=2 --dp-node-rank=0 --dp-master-addr=192.168.10.10 --dp-master-port=12345 --enforce-eager --trust-remote-code && VLLM_MULTI_NODE=1 pytest -v -s distributed/test_multi_node_assignment.py && VLLM_MULTI_NODE=1 pytest -v -s distributed/test_pipeline_parallel.py" "VLLM_TEST_SAME_HOST=0 torchrun --nnodes 2 --nproc-per-node=2 --rdzv_backend=c10d --rdzv_endpoint=192.168.10.10 distributed/test_same_node.py | grep 'Same node test passed' && NUM_NODES=2 torchrun --nnodes 2 --nproc-per-node=2 --rdzv_backend=c10d --rdzv_endpoint=192.168.10.10 distributed/test_node_count.py | grep 'Node count test passed' && python3 ../examples/features/data_parallel/data_parallel_offline.py -dp=2 -tp=1 --dp-num-nodes=2 --dp-node-rank=1 --dp-master-addr=192.168.10.10 --dp-master-port=12345 --enforce-eager --trust-remote-code"
- label: ":nvidia: (L4) Pipeline + Context Parallelism" - label: ":nvidia: (L4) Pipeline + Context Parallelism"
key: pipeline-context-parallelism-4-gpus key: pipeline-context-parallelism-4-gpus
automatic_shard: true
timeout_in_minutes: 55 timeout_in_minutes: 55
working_dir: "/vllm-workspace/tests" working_dir: "/vllm-workspace/tests"
device: l4 device: l4
@@ -686,7 +675,6 @@ steps:
- label: ":nvidia: (L4) RayExecutorV2" - label: ":nvidia: (L4) RayExecutorV2"
key: rayexecutorv2-4-gpus key: rayexecutorv2-4-gpus
automatic_shard: true
timeout_in_minutes: 45 timeout_in_minutes: 45
working_dir: "/vllm-workspace/tests" working_dir: "/vllm-workspace/tests"
device: l4 device: l4
@@ -700,6 +688,7 @@ steps:
- tests/distributed/test_pipeline_parallel.py - tests/distributed/test_pipeline_parallel.py
- tests/basic_correctness/models/test_basic_correctness.py - tests/basic_correctness/models/test_basic_correctness.py
commands: commands:
- export VLLM_USE_RAY_V2_EXECUTOR_BACKEND=1
- export NCCL_CUMEM_HOST_ENABLE=0 - export NCCL_CUMEM_HOST_ENABLE=0
- pytest -v -s distributed/test_ray_v2_executor.py - pytest -v -s distributed/test_ray_v2_executor.py
- pytest -v -s distributed/test_ray_v2_executor_e2e.py - pytest -v -s distributed/test_ray_v2_executor_e2e.py
@@ -709,7 +698,6 @@ steps:
amd: amd:
label: ':amd: (MI355) RayExecutorV2' label: ':amd: (MI355) RayExecutorV2'
dind: false dind: false
automatic_shard: true
device: mi355_4 device: mi355_4
timeout_in_minutes: 75 timeout_in_minutes: 75
working_dir: /vllm-workspace/tests working_dir: /vllm-workspace/tests
@@ -725,6 +713,7 @@ steps:
- tests/basic_correctness/models/test_basic_correctness.py - tests/basic_correctness/models/test_basic_correctness.py
- vllm/platforms/rocm.py - vllm/platforms/rocm.py
commands: commands:
- export VLLM_USE_RAY_V2_EXECUTOR_BACKEND=1
- pytest -v -s distributed/test_ray_v2_executor.py - pytest -v -s distributed/test_ray_v2_executor.py
- pytest -v -s distributed/test_ray_v2_executor_e2e.py - pytest -v -s distributed/test_ray_v2_executor_e2e.py
- pytest -v -s distributed/test_pipeline_parallel.py -k "ray" - pytest -v -s distributed/test_pipeline_parallel.py -k "ray"
+3 -6
View File
@@ -38,7 +38,6 @@ steps:
- label: ":nvidia: (L4) V1 Engine" - label: ":nvidia: (L4) V1 Engine"
key: engine-1-gpu key: engine-1-gpu
automatic_shard: true
device: l4 device: l4
num_devices: 1 num_devices: 1
timeout_in_minutes: 45 timeout_in_minutes: 45
@@ -109,7 +108,6 @@ steps:
- label: ":nvidia: (H200 MIG 18GB) E2E Core" - label: ":nvidia: (H200 MIG 18GB) E2E Core"
device: h200_18gb device: h200_18gb
key: e2e-core-1-gpu key: e2e-core-1-gpu
automatic_shard: true
timeout_in_minutes: 65 timeout_in_minutes: 65
source_file_dependencies: source_file_dependencies:
- vllm/v1/ - vllm/v1/
@@ -141,7 +139,7 @@ steps:
- tests/v1/e2e/general/ - tests/v1/e2e/general/
commands: commands:
- pytest -v -s v1/e2e/general/test_kv_sharing_fast_prefill.py - pytest -v -s v1/e2e/general/test_kv_sharing_fast_prefill.py
- pytest -v -s v1/e2e/general/test_mamba_prefix_cache.py::test_mamba_prefix_cache_mrv2 v1/e2e/general/test_mamba_prefix_cache.py::test_mamba_prefix_cache_mrv2_async v1/e2e/general/test_mamba_prefix_cache.py::test_mamba_prefix_cache_mrv2_async_flashinfer - pytest -v -s v1/e2e/general/test_mamba_prefix_cache.py::test_mamba_prefix_cache_mrv2 v1/e2e/general/test_mamba_prefix_cache.py::test_mamba_prefix_cache_mrv2_async
mirror: mirror:
amd: amd:
label: ":amd: (MI355 DPX) E2E Core Large Memory" label: ":amd: (MI355 DPX) E2E Core Large Memory"
@@ -164,7 +162,6 @@ steps:
device: h100 device: h100
num_devices: 2 num_devices: 2
key: replayssm-e2e key: replayssm-e2e
automatic_shard: true
timeout_in_minutes: 90 timeout_in_minutes: 90
working_dir: "/vllm-workspace/tests" working_dir: "/vllm-workspace/tests"
source_file_dependencies: source_file_dependencies:
@@ -238,9 +235,9 @@ steps:
- pytest -v -s v1/e2e/test_hybrid_chunked_prefill.py - pytest -v -s v1/e2e/test_hybrid_chunked_prefill.py
mirror: mirror:
amd: amd:
label: ":amd: (MI355 DPX) V1 E2E Hybrid Chunked Prefill" label: ":amd: (MI300) V1 E2E Hybrid Chunked Prefill"
dind: false dind: false
device: mi355_dpx device: mi300_1
timeout_in_minutes: 35 timeout_in_minutes: 35
depends_on: depends_on:
- image-build-amd - image-build-amd
+2 -11
View File
@@ -26,22 +26,21 @@ steps:
- label: ":nvidia: (H200 MIG 35GB) Entrypoints Unit" - label: ":nvidia: (H200 MIG 35GB) Entrypoints Unit"
device: h200_35gb device: h200_35gb
key: entrypoints-unit-tests key: entrypoints-unit-tests
automatic_shard: true
timeout_in_minutes: 40 timeout_in_minutes: 40
working_dir: "/vllm-workspace/tests" working_dir: "/vllm-workspace/tests"
source_file_dependencies: source_file_dependencies:
- vllm/entrypoints - vllm/entrypoints
- vllm/utils/flashinfer.py
- tests/entrypoints/unit_tests - tests/entrypoints/unit_tests
- tests/entrypoints/weight_transfer
- tests/entrypoints/launchers - tests/entrypoints/launchers
commands: commands:
- pytest -v -s entrypoints/unit_tests - pytest -v -s entrypoints/unit_tests
- pytest -v -s entrypoints/weight_transfer
- pytest -v -s entrypoints/launchers - pytest -v -s entrypoints/launchers
mirror: mirror:
amd: amd:
label: ":amd: (MI355 DPX) Entrypoints Unit" label: ":amd: (MI355 DPX) Entrypoints Unit"
dind: false dind: false
automatic_shard: true
device: mi355_dpx device: mi355_dpx
timeout_in_minutes: 50 timeout_in_minutes: 50
depends_on: depends_on:
@@ -122,7 +121,6 @@ steps:
- label: ":nvidia: (H200 MIG 35GB) Entrypoints Integration (OpenAI API chat_completion)" - label: ":nvidia: (H200 MIG 35GB) Entrypoints Integration (OpenAI API chat_completion)"
device: h200_35gb device: h200_35gb
key: entrypoints-integration-api-server-openai-chat_completion key: entrypoints-integration-api-server-openai-chat_completion
automatic_shard: true
timeout_in_minutes: 83 timeout_in_minutes: 83
working_dir: "/vllm-workspace/tests" working_dir: "/vllm-workspace/tests"
source_file_dependencies: source_file_dependencies:
@@ -144,7 +142,6 @@ steps:
- label: ":nvidia: (H200 MIG 35GB) Entrypoints Integration (API Server Generate)" - label: ":nvidia: (H200 MIG 35GB) Entrypoints Integration (API Server Generate)"
device: h200_35gb device: h200_35gb
key: entrypoints-integration-api-server-generate key: entrypoints-integration-api-server-generate
automatic_shard: true
timeout_in_minutes: 50 timeout_in_minutes: 50
working_dir: "/vllm-workspace/tests" working_dir: "/vllm-workspace/tests"
source_file_dependencies: source_file_dependencies:
@@ -173,7 +170,6 @@ steps:
- label: ":nvidia: (H200 MIG 35GB) Entrypoints Integration (Responses API)" - label: ":nvidia: (H200 MIG 35GB) Entrypoints Integration (Responses API)"
device: h200_35gb device: h200_35gb
key: entrypoints-integration-responses-api key: entrypoints-integration-responses-api
automatic_shard: true
timeout_in_minutes: 50 timeout_in_minutes: 50
working_dir: "/vllm-workspace/tests" working_dir: "/vllm-workspace/tests"
source_file_dependencies: source_file_dependencies:
@@ -194,7 +190,6 @@ steps:
- label: ":nvidia: (H200 MIG 35GB) Entrypoints Integration (Speech to Text)" - label: ":nvidia: (H200 MIG 35GB) Entrypoints Integration (Speech to Text)"
device: h200_35gb device: h200_35gb
key: entrypoints-integration-speech_to_text key: entrypoints-integration-speech_to_text
automatic_shard: true
timeout_in_minutes: 45 timeout_in_minutes: 45
working_dir: "/vllm-workspace/tests" working_dir: "/vllm-workspace/tests"
source_file_dependencies: source_file_dependencies:
@@ -208,7 +203,6 @@ steps:
amd: amd:
label: ":amd: (MI355 DPX) Entrypoints Integration (Speech to Text)" label: ":amd: (MI355 DPX) Entrypoints Integration (Speech to Text)"
dind: false dind: false
automatic_shard: true
device: mi355_dpx device: mi355_dpx
timeout_in_minutes: 60 timeout_in_minutes: 60
working_dir: "/vllm-workspace/tests" working_dir: "/vllm-workspace/tests"
@@ -218,7 +212,6 @@ steps:
- label: ":nvidia: (H200 MIG 35GB) Entrypoints Integration (Multimodal)" - label: ":nvidia: (H200 MIG 35GB) Entrypoints Integration (Multimodal)"
device: h200_35gb device: h200_35gb
key: entrypoints-integration-multimodal key: entrypoints-integration-multimodal
automatic_shard: true
timeout_in_minutes: 45 timeout_in_minutes: 45
working_dir: "/vllm-workspace/tests" working_dir: "/vllm-workspace/tests"
source_file_dependencies: source_file_dependencies:
@@ -240,7 +233,6 @@ steps:
- label: ":nvidia: (H200 MIG 35GB) Entrypoints Integration (Pooling)" - label: ":nvidia: (H200 MIG 35GB) Entrypoints Integration (Pooling)"
device: h200_35gb device: h200_35gb
key: entrypoints-integration-pooling key: entrypoints-integration-pooling
automatic_shard: true
timeout_in_minutes: 75 timeout_in_minutes: 75
working_dir: "/vllm-workspace/tests" working_dir: "/vllm-workspace/tests"
source_file_dependencies: source_file_dependencies:
@@ -254,7 +246,6 @@ steps:
amd: amd:
label: ":amd: (MI355 DPX) Entrypoints Integration (Pooling)" label: ":amd: (MI355 DPX) Entrypoints Integration (Pooling)"
dind: false dind: false
automatic_shard: true
device: mi355_dpx device: mi355_dpx
timeout_in_minutes: 65 timeout_in_minutes: 65
working_dir: "/vllm-workspace/tests" working_dir: "/vllm-workspace/tests"
@@ -58,8 +58,6 @@ steps:
- label: ":nvidia: (H100) Elastic EP Scaling" - label: ":nvidia: (H100) Elastic EP Scaling"
key: elastic-ep-scaling-test key: elastic-ep-scaling-test
# Nightly only: 4-GPU H100 pods wait behind smaller jobs on per-PR runs.
optional: true
timeout_in_minutes: 40 timeout_in_minutes: 40
device: h100 device: h100
working_dir: "/vllm-workspace/tests" working_dir: "/vllm-workspace/tests"
+34 -99
View File
@@ -30,10 +30,9 @@ steps:
- vllm/_aiter_ops.py - vllm/_aiter_ops.py
- vllm/platforms/rocm.py - vllm/platforms/rocm.py
- label: ":nvidia: (H200 MIG 35GB) Core Operation Kernels" - label: ":nvidia: (H200 MIG 35GB) Core Operation Kernels Shard %N"
device: h200_35gb device: h200_35gb
key: kernels-core-operation-test key: kernels-core-operation-test
automatic_shard: true
timeout_in_minutes: 120 timeout_in_minutes: 120
source_file_dependencies: source_file_dependencies:
- csrc/ - csrc/
@@ -41,10 +40,10 @@ steps:
- tests/kernels/test_concat_mla_q.py - tests/kernels/test_concat_mla_q.py
- tests/kernels/test_fused_qk_norm_rope_gate.py - tests/kernels/test_fused_qk_norm_rope_gate.py
commands: commands:
- pytest -v -s kernels/core --ignore=kernels/core/test_minimax_reduce_rms.py kernels/test_concat_mla_q.py kernels/test_fused_qk_norm_rope_gate.py - pytest -v -s kernels/core --ignore=kernels/core/test_minimax_reduce_rms.py kernels/test_concat_mla_q.py kernels/test_fused_qk_norm_rope_gate.py --shard-id=$$BUILDKITE_PARALLEL_JOB --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT
parallelism: 3
mirror: mirror:
amd: amd:
parallelism: 3
label: ":amd: (MI300) Core Operation Kernels Shard %N" label: ":amd: (MI300) Core Operation Kernels Shard %N"
dind: false dind: false
device: mi300_1 device: mi300_1
@@ -73,10 +72,10 @@ steps:
num_devices: 2 num_devices: 2
device: h100 device: h100
source_file_dependencies: source_file_dependencies:
- csrc/minimax_reduce_rms_kernel.cu - csrc/libtorch_stable/minimax_reduce_rms_kernel.cu
- csrc/minimax_reduce_rms_kernel.h - csrc/libtorch_stable/minimax_reduce_rms_kernel.h
- csrc/ops.h - csrc/libtorch_stable/ops.h
- csrc/torch_bindings.cpp - csrc/libtorch_stable/torch_bindings.cpp
- vllm/model_executor/layers/minimax_rms_norm/ - vllm/model_executor/layers/minimax_rms_norm/
- vllm/model_executor/layers/mamba/linear/minimax_linear_attn.py - vllm/model_executor/layers/mamba/linear/minimax_linear_attn.py
- tests/kernels/core/test_minimax_reduce_rms.py - tests/kernels/core/test_minimax_reduce_rms.py
@@ -123,7 +122,7 @@ steps:
depends_on: depends_on:
- image-build-amd - image-build-amd
source_file_dependencies: source_file_dependencies:
- csrc/fused_deepseek_v4_qnorm_rope_kv_insert_kernel.cu - csrc/libtorch_stable/fused_deepseek_v4_qnorm_rope_kv_insert_kernel.cu
- vllm/models/deepseek_v4/common/ops/ - vllm/models/deepseek_v4/common/ops/
- tests/kernels/test_fused_deepseek_v4_qnorm_rope_kv_insert.py - tests/kernels/test_fused_deepseek_v4_qnorm_rope_kv_insert.py
- vllm/platforms/rocm.py - vllm/platforms/rocm.py
@@ -138,15 +137,11 @@ steps:
- csrc/fused_deepseek_v4_qnorm_rope_kv_insert_kernel.cu - csrc/fused_deepseek_v4_qnorm_rope_kv_insert_kernel.cu
- vllm/models/deepseek_v4/common/ops/ - vllm/models/deepseek_v4/common/ops/
- vllm/models/deepseek_v4/nvidia/ - vllm/models/deepseek_v4/nvidia/
- vllm/model_executor/layers/fused_moe/deep_gemm_mega_moe.py
- vllm/model_executor/layers/fused_moe/prepare_megamoe.py
- vllm/utils/deep_gemm.py - vllm/utils/deep_gemm.py
- tests/kernels/test_fused_deepseek_v4_qnorm_rope_kv_insert.py - tests/kernels/test_fused_deepseek_v4_qnorm_rope_kv_insert.py
- tests/kernels/moe/test_deep_gemm_mega_moe.py
- tests/models/test_deepseek_v4_mega_moe.py - tests/models/test_deepseek_v4_mega_moe.py
commands: commands:
- pytest -v -s kernels/test_fused_deepseek_v4_*.py - pytest -v -s kernels/test_fused_deepseek_v4_*.py
- pytest -v -s kernels/moe/test_deep_gemm_mega_moe.py
- pytest -v -s models/test_deepseek_v4_mega_moe.py - pytest -v -s models/test_deepseek_v4_mega_moe.py
mirror: mirror:
amd: amd:
@@ -160,7 +155,6 @@ steps:
- image-build-amd - image-build-amd
source_file_dependencies: source_file_dependencies:
- tests/models/test_deepseek_v4_vl_rocm.py - tests/models/test_deepseek_v4_vl_rocm.py
- tests/models/test_deepseek_v4_aiter_mega_moe.py
- vllm/models/deepseek_v4/amd/ - vllm/models/deepseek_v4/amd/
- vllm/models/deepseek_v4/common/ - vllm/models/deepseek_v4/common/
- vllm/model_executor/layers/fused_moe/ - vllm/model_executor/layers/fused_moe/
@@ -176,7 +170,6 @@ steps:
# Common qnorm/RoPE/cache kernels already run in the H100 AMD mirror. # Common qnorm/RoPE/cache kernels already run in the H100 AMD mirror.
# Pin the tested BF16 Triton routing and shared-expert path before imports. # Pin the tested BF16 Triton routing and shared-expert path before imports.
- VLLM_ROCM_USE_AITER_MOE=0 pytest -v -s models/test_deepseek_v4_vl_rocm.py - VLLM_ROCM_USE_AITER_MOE=0 pytest -v -s models/test_deepseek_v4_vl_rocm.py
- pytest -v -s models/test_deepseek_v4_aiter_mega_moe.py
# Own job so mHC never shares a process with FlashInfer's import-time # Own job so mHC never shares a process with FlashInfer's import-time
# ptxas override. # ptxas override.
@@ -235,8 +228,8 @@ steps:
# Catch-all for test files at the tests/kernels root. This job collects # Catch-all for test files at the tests/kernels root. This job collects
# the whole root so new files are wired by default. # the whole root so new files are wired by default.
# Files with dedicated jobs elsewhere in this file are excluded via --ignore # Files with dedicated jobs elsewhere in this file are excluded via --ignore
# (test_bf16x3_router_gemm_cutedsl runs in Kernels (B200); test_ll_bf16_gemm and # (test_bf16x3_router_gemm_cutedsl and test_ll_bf16_gemm run in their own jobs /
# test_ll_fp32w_gemm run in Gate Linear CuTeDSL Kernels (B200)). # Kernels (B200)).
- label: ":nvidia: (B200) Miscellaneous Kernels" - label: ":nvidia: (B200) Miscellaneous Kernels"
key: kernels-root-misc-test-b200 key: kernels-root-misc-test-b200
timeout_in_minutes: 30 timeout_in_minutes: 30
@@ -261,11 +254,8 @@ steps:
--ignore=kernels/test_fused_qk_norm_rope_gate.py --ignore=kernels/test_fused_qk_norm_rope_gate.py
--ignore=kernels/test_fused_deepseek_v4_qnorm_rope_kv_insert.py --ignore=kernels/test_fused_deepseek_v4_qnorm_rope_kv_insert.py
--ignore=kernels/test_top_k_per_row.py --ignore=kernels/test_top_k_per_row.py
--ignore=kernels/test_cpu_fused_sampling.py
--ignore=kernels/test_bf16x3_router_gemm_cutedsl.py --ignore=kernels/test_bf16x3_router_gemm_cutedsl.py
--ignore=kernels/test_bf16x3_router_gemm_rocm.py
--ignore=kernels/test_ll_bf16_gemm.py --ignore=kernels/test_ll_bf16_gemm.py
--ignore=kernels/test_ll_fp32w_gemm.py
--ignore=kernels/test_shuffle_rows.py --ignore=kernels/test_shuffle_rows.py
# BROKEN on main, pending kernel fixes (B200): # BROKEN on main, pending kernel fixes (B200):
# test_shuffle_rows.py (1: test_shuffle_rows_edge_cases) # test_shuffle_rows.py (1: test_shuffle_rows_edge_cases)
@@ -304,12 +294,14 @@ steps:
kernels/test_fused_inv_rope_fp8_quant.py kernels/test_fused_inv_rope_fp8_quant.py
kernels/test_fused_minimax_m3_qknorm_rope_kv_insert.py kernels/test_fused_minimax_m3_qknorm_rope_kv_insert.py
kernels/test_fused_quant_activation.py kernels/test_fused_quant_activation.py
kernels/test_gate_linear_rocm_dispatch.py
kernels/test_gemm_rs_ar.py kernels/test_gemm_rs_ar.py
kernels/test_kpool_decode_update_batched.py kernels/test_kpool_decode_update_batched.py
kernels/test_minimax_m3_amd_ops.py kernels/test_minimax_m3_amd_ops.py
kernels/test_minimax_m3_sparse_attn_fp8_scale.py kernels/test_minimax_m3_sparse_attn_fp8_scale.py
kernels/test_onednn.py kernels/test_onednn.py
kernels/test_relu2_fp8_quant.py kernels/test_relu2_fp8_quant.py
kernels/test_rocm_fp32_router_gemm.py
--deselect=tests/kernels/test_bf16_skinny_gemm.py::test_every_dsv3_routed_shape_is_instantiated --deselect=tests/kernels/test_bf16_skinny_gemm.py::test_every_dsv3_routed_shape_is_instantiated
- label: ":nvidia: (L4) Attention Kernels Shard %N" - label: ":nvidia: (L4) Attention Kernels Shard %N"
@@ -319,7 +311,6 @@ steps:
timeout_in_minutes: 30 timeout_in_minutes: 30
source_file_dependencies: source_file_dependencies:
- csrc/attention/ - csrc/attention/
- csrc/core/vec_dtypes/
- vllm/v1/attention - vllm/v1/attention
# TODO: remove this dependency (https://github.com/vllm-project/vllm/issues/32267) # TODO: remove this dependency (https://github.com/vllm-project/vllm/issues/32267)
- vllm/model_executor/layers/attention - vllm/model_executor/layers/attention
@@ -339,7 +330,6 @@ steps:
- image-build-amd - image-build-amd
source_file_dependencies: source_file_dependencies:
- csrc/attention/ - csrc/attention/
- csrc/core/vec_dtypes/
- vllm/v1/attention - vllm/v1/attention
- vllm/model_executor/layers/attention - vllm/model_executor/layers/attention
- tests/kernels/attention - tests/kernels/attention
@@ -412,14 +402,10 @@ steps:
depends_on: depends_on:
- image-build-amd - image-build-amd
source_file_dependencies: source_file_dependencies:
- vllm/models/deepseek_v41/amd/rocm.py
- vllm/models/deepseek_v41/sparse_mla.py
- vllm/v1/attention/backends/mla/rocm_aiter_mla.py - vllm/v1/attention/backends/mla/rocm_aiter_mla.py
- vllm/v1/attention/ops/rocm_aiter_mla_sparse.py
- vllm/v1/attention/selector.py - vllm/v1/attention/selector.py
- vllm/platforms/rocm.py - vllm/platforms/rocm.py
- vllm/_aiter_ops.py - vllm/_aiter_ops.py
- tests/kernels/attention/test_rocm_triton_attn_dsv4.py
- tests/kernels/attention/test_rocm_aiter_mla_decode.py - tests/kernels/attention/test_rocm_aiter_mla_decode.py
- tests/kernels/attention/test_rocm_aiter_mla_decode_metadata.py - tests/kernels/attention/test_rocm_aiter_mla_decode_metadata.py
- tests/kernels/attention/test_rocm_aiter_mla_fp8_support.py - tests/kernels/attention/test_rocm_aiter_mla_fp8_support.py
@@ -430,7 +416,6 @@ steps:
- pytest -v -s tests/kernels/attention/test_rocm_aiter_mla_decode.py - pytest -v -s tests/kernels/attention/test_rocm_aiter_mla_decode.py
- pytest -v -s tests/kernels/attention/test_rocm_aiter_mla_fp8_support.py - pytest -v -s tests/kernels/attention/test_rocm_aiter_mla_fp8_support.py
- pytest -v -s tests/kernels/attention/test_rocm_aiter_mla_op_registration.py - pytest -v -s tests/kernels/attention/test_rocm_aiter_mla_op_registration.py
- pytest -v -s tests/kernels/attention/test_rocm_triton_attn_dsv4.py::test_dsv41_adaptive_mla_swa_metadata_graph_replay
- label: ":nvidia: (L4) Quantization Kernels Shard %N" - label: ":nvidia: (L4) Quantization Kernels Shard %N"
key: kernels-quantization-test key: kernels-quantization-test
@@ -477,7 +462,7 @@ steps:
num_devices: 1 num_devices: 1
timeout_in_minutes: 50 timeout_in_minutes: 50
source_file_dependencies: source_file_dependencies:
- csrc/quantization/w8a8/cutlass/moe/ - csrc/quantization/cutlass_w8a8/moe/
- csrc/moe/ - csrc/moe/
- tests/kernels/moe - tests/kernels/moe
- vllm/model_executor/layers/fused_moe/ - vllm/model_executor/layers/fused_moe/
@@ -495,7 +480,7 @@ steps:
device: mi300_1 device: mi300_1
timeout_in_minutes: 55 timeout_in_minutes: 55
source_file_dependencies: source_file_dependencies:
- csrc/quantization/w8a8/cutlass/moe/ - csrc/quantization/cutlass_w8a8/moe/
- csrc/moe/ - csrc/moe/
- tests/kernels/moe - tests/kernels/moe
- vllm/model_executor/layers/fused_moe/ - vllm/model_executor/layers/fused_moe/
@@ -514,22 +499,18 @@ steps:
- label: ":nvidia: (H200 MIG 35GB) Mamba Kernels" - label: ":nvidia: (H200 MIG 35GB) Mamba Kernels"
device: h200_35gb device: h200_35gb
key: kernels-mamba-test key: kernels-mamba-test
automatic_shard: true
timeout_in_minutes: 60 timeout_in_minutes: 60
source_file_dependencies: source_file_dependencies:
- csrc/mamba/ - csrc/mamba/
- tests/kernels/mamba - tests/kernels/mamba
- vllm/model_executor/layers/mamba/checkpoint.py
- vllm/model_executor/layers/mamba/gdn - vllm/model_executor/layers/mamba/gdn
- vllm/model_executor/layers/mamba/ops - vllm/model_executor/layers/mamba/ops
- vllm/v1/attention/backends/gdn_attn.py
commands: commands:
- pytest -v -s kernels/mamba - pytest -v -s kernels/mamba
mirror: mirror:
amd: amd:
label: ":amd: (MI250) Mamba Kernels" label: ":amd: (MI250) Mamba Kernels"
dind: false dind: false
automatic_shard: true
device: mi250_1 device: mi250_1
timeout_in_minutes: 55 timeout_in_minutes: 55
depends_on: depends_on:
@@ -545,6 +526,7 @@ steps:
num_devices: 1 num_devices: 1
source_file_dependencies: source_file_dependencies:
- cmake/external_projects/deepgemm.cmake - cmake/external_projects/deepgemm.cmake
- tools/install_deepgemm.sh
- tools/build_deepgemm_C.py - tools/build_deepgemm_C.py
- tools/setup_deepgemm_pythons.sh - tools/setup_deepgemm_pythons.sh
- tools/check_wheel_deepgemm.py - tools/check_wheel_deepgemm.py
@@ -564,21 +546,6 @@ steps:
- pytest -v -s kernels/attention/test_deepgemm_attention.py - pytest -v -s kernels/attention/test_deepgemm_attention.py
- pytest -v -s quantization/test_cutlass_w4a16.py - pytest -v -s quantization/test_cutlass_w4a16.py
- label: ":nvidia: (B200) MiniMax-M3 Attention Kernels"
key: kernels-minimax-m3-attention-b200
timeout_in_minutes: 30
working_dir: "/vllm-workspace/"
device: b200-k8s
num_devices: 1
source_file_dependencies:
- vllm/models/minimax_m3/
- vllm/config/cache.py
- vllm/v1/kv_cache_interface.py
- tests/v1/attention/utils.py
- tests/kernels/attention/test_minimax_m3.py
commands:
- pytest -v -s tests/kernels/attention/test_minimax_m3.py
- label: ":nvidia: (B200) Kernels Shard %N" - label: ":nvidia: (B200) Kernels Shard %N"
key: kernels-b200 key: kernels-b200
timeout_in_minutes: 30 timeout_in_minutes: 30
@@ -588,10 +555,11 @@ steps:
# optional: true # optional: true
source_file_dependencies: source_file_dependencies:
- csrc/quantization/fp4/ - csrc/quantization/fp4/
- csrc/nvfp4_kv_cache_kernels.cu - csrc/libtorch_stable/nvfp4_kv_cache_kernels.cu
- csrc/quantization/w8a8/cutlass/ - csrc/libtorch_stable/quantization/fp4/
- csrc/libtorch_stable/quantization/w8a8/cutlass/
- csrc/attention/mla/ - csrc/attention/mla/
- csrc/quantization/w8a8/cutlass/moe/ - csrc/quantization/cutlass_w8a8/moe/
- vllm/model_executor/layers/fusion/ - vllm/model_executor/layers/fusion/
- vllm/model_executor/layers/fused_moe/experts/cutlass_moe.py - vllm/model_executor/layers/fused_moe/experts/cutlass_moe.py
- vllm/model_executor/layers/fused_moe/experts/flashinfer_cutlass_moe.py - vllm/model_executor/layers/fused_moe/experts/flashinfer_cutlass_moe.py
@@ -605,6 +573,9 @@ steps:
- vllm/v1/attention/backends/mla/flashinfer_mla.py - vllm/v1/attention/backends/mla/flashinfer_mla.py
- vllm/v1/attention/selector.py - vllm/v1/attention/selector.py
- vllm/platforms/cuda.py - vllm/platforms/cuda.py
- vllm/model_executor/kernels/linear/cute_dsl/ll_bf16.py
- vllm/model_executor/kernels/linear/cute_dsl/_ll_bf16_dotprod.py
- vllm/model_executor/kernels/linear/cute_dsl/_ll_bf16_splitk.py
- vllm/cute_utils/ - vllm/cute_utils/
- vllm/model_executor/layers/mamba/ops/gdn_chunk_cutedsl/ - vllm/model_executor/layers/mamba/ops/gdn_chunk_cutedsl/
- vllm/model_executor/layers/fused_moe/router/bf16x3_router_gemm_cutedsl.py - vllm/model_executor/layers/fused_moe/router/bf16x3_router_gemm_cutedsl.py
@@ -614,9 +585,9 @@ steps:
- vllm/config/attention.py - vllm/config/attention.py
- vllm/v1/attention/backends/registry.py - vllm/v1/attention/backends/registry.py
- vllm/_custom_ops.py - vllm/_custom_ops.py
- csrc/fused_minimax_m3_qknorm_rope_kv_insert_kernel.cu - csrc/libtorch_stable/fused_minimax_m3_qknorm_rope_kv_insert_kernel.cu
- csrc/ops.h - csrc/libtorch_stable/ops.h
- csrc/torch_bindings.cpp - csrc/libtorch_stable/torch_bindings.cpp
- tests/kernels/attention/test_flashinfer_mla_decode.py - tests/kernels/attention/test_flashinfer_mla_decode.py
- tests/kernels/attention/test_minimax_m3_msa_cutlass_sparse_decode.py - tests/kernels/attention/test_minimax_m3_msa_cutlass_sparse_decode.py
- tests/kernels/attention/test_cache.py - tests/kernels/attention/test_cache.py
@@ -624,6 +595,8 @@ steps:
- tests/kernels/test_fused_minimax_m3_qknorm_rope_kv_insert.py - tests/kernels/test_fused_minimax_m3_qknorm_rope_kv_insert.py
- tests/kernels/mamba/test_gdn_prefill_cutedsl.py - tests/kernels/mamba/test_gdn_prefill_cutedsl.py
- tests/kernels/test_bf16x3_router_gemm_cutedsl.py - tests/kernels/test_bf16x3_router_gemm_cutedsl.py
- tests/kernels/attention/test_minimax_m3.py
- tests/kernels/test_ll_bf16_gemm.py
- tests/kernels/test_top_k_per_row.py - tests/kernels/test_top_k_per_row.py
- tests/kernels/quantization/test_silu_mul_nvfp4_quant.py - tests/kernels/quantization/test_silu_mul_nvfp4_quant.py
- tests/kernels/quantization/test_block_fp8.py - tests/kernels/quantization/test_block_fp8.py
@@ -644,6 +617,7 @@ steps:
pytest -v -s tests/kernels/moe/test_nvfp4_moe.py pytest -v -s tests/kernels/moe/test_nvfp4_moe.py
pytest -v -s tests/kernels/moe/test_ocp_mx_moe.py pytest -v -s tests/kernels/moe/test_ocp_mx_moe.py
pytest -v -s tests/kernels/test_bf16x3_router_gemm_cutedsl.py pytest -v -s tests/kernels/test_bf16x3_router_gemm_cutedsl.py
pytest -v -s tests/kernels/test_ll_bf16_gemm.py
;; ;;
1) 1)
python3 examples/basic/offline_inference/chat.py python3 examples/basic/offline_inference/chat.py
@@ -667,6 +641,7 @@ steps:
pytest -v -s tests/kernels/moe/test_flashinfer.py pytest -v -s tests/kernels/moe/test_flashinfer.py
pytest -v -s tests/kernels/moe/test_trtllm_nvfp4_moe.py pytest -v -s tests/kernels/moe/test_trtllm_nvfp4_moe.py
pytest -v -s tests/kernels/mamba/test_gdn_prefill_cutedsl.py pytest -v -s tests/kernels/mamba/test_gdn_prefill_cutedsl.py
pytest -v -s tests/kernels/attention/test_minimax_m3.py
;; ;;
*) *)
echo "unexpected BUILDKITE_PARALLEL_JOB=$$BUILDKITE_PARALLEL_JOB" >&2 echo "unexpected BUILDKITE_PARALLEL_JOB=$$BUILDKITE_PARALLEL_JOB" >&2
@@ -700,46 +675,6 @@ steps:
- amd-smi - amd-smi
- pytest -v -s kernels/quantization/test_nvfp4_emulation.py -k test_nvfp4_dense_emulation --shard-id=$$BUILDKITE_PARALLEL_JOB --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT - pytest -v -s kernels/quantization/test_nvfp4_emulation.py -k test_nvfp4_dense_emulation --shard-id=$$BUILDKITE_PARALLEL_JOB --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT
- label: ":nvidia: (B200) Gate Linear CuTeDSL Kernels"
key: kernels-gate-linear-cutedsl-b200
timeout_in_minutes: 15
working_dir: "/vllm-workspace/"
device: b200-k8s
source_file_dependencies:
- vllm/cute_utils/
- vllm/platforms/cuda.py
- vllm/model_executor/layers/fused_moe/router/gate_linear.py
- vllm/model_executor/kernels/linear/cute_dsl/_ll_router_common.py
- vllm/model_executor/kernels/linear/cute_dsl/ll_bf16.py
- vllm/model_executor/kernels/linear/cute_dsl/_ll_bf16_dotprod.py
- vllm/model_executor/kernels/linear/cute_dsl/_ll_bf16_splitk.py
- tests/kernels/test_ll_bf16_gemm.py
- vllm/model_executor/kernels/linear/cute_dsl/ll_fp32w.py
- vllm/model_executor/kernels/linear/cute_dsl/_ll_fp32w_dotprod.py
- tests/kernels/test_ll_fp32w_gemm.py
commands:
- nvidia-smi
- pytest -v -s tests/kernels/test_ll_bf16_gemm.py
- pytest -v -s tests/kernels/test_ll_fp32w_gemm.py
mirror:
amd:
label: ":amd: (MI355) Gate Linear Router Kernels"
dind: false
device: mi355_1
timeout_in_minutes: 20
depends_on:
- image-build-amd
source_file_dependencies:
- vllm/model_executor/layers/fused_moe/router/gate_linear.py
- vllm/model_executor/layers/fused_moe/router/rocm_fp32_router_gemm.py
- vllm/model_executor/layers/fused_moe/router/bf16x3_router_gemm_rocm.py
- vllm/platforms/rocm.py
- tests/kernels/test_rocm_fp32_router_gemm.py
- tests/kernels/test_gate_linear_rocm_dispatch.py
- tests/kernels/test_bf16x3_router_gemm_rocm.py
commands:
- pytest -v -s tests/kernels/test_rocm_fp32_router_gemm.py tests/kernels/test_gate_linear_rocm_dispatch.py tests/kernels/test_bf16x3_router_gemm_rocm.py
- label: ":nvidia: (DGX) Spark b12x Linear Kernels Nightly" - label: ":nvidia: (DGX) Spark b12x Linear Kernels Nightly"
key: b12x-linear-kernels-dgx-spark-nightly key: b12x-linear-kernels-dgx-spark-nightly
timeout_in_minutes: 60 timeout_in_minutes: 60
@@ -772,7 +707,7 @@ steps:
- tests/quantization/reference_mxfp4.py - tests/quantization/reference_mxfp4.py
- tests/quantization/test_auto_round.py - tests/quantization/test_auto_round.py
commands: commands:
- uv pip install --system b12x==1.5.1 - uv pip install --system b12x==1.3.0
- pytest -v -s model_executor/kernels/test_b12x_linear.py - pytest -v -s model_executor/kernels/test_b12x_linear.py
model_executor/test_b12x_warmup.py model_executor/test_b12x_warmup.py
kernels/quantization/test_block_fp8.py kernels/quantization/test_block_fp8.py
@@ -899,7 +834,7 @@ steps:
device: h100 device: h100
num_devices: 2 num_devices: 2
source_file_dependencies: source_file_dependencies:
- csrc/quantization/w8a8/cutlass/moe/ - csrc/quantization/cutlass_w8a8/moe/
- csrc/moe/ - csrc/moe/
- tests/kernels/moe - tests/kernels/moe
- vllm/model_executor/layers/fused_moe/ - vllm/model_executor/layers/fused_moe/
@@ -938,11 +873,11 @@ steps:
- label: ":nvidia: (B200) FusedMoE Layer Kernels" - label: ":nvidia: (B200) FusedMoE Layer Kernels"
key: kernels-fusedmoe-layer-test-2-b200s key: kernels-fusedmoe-layer-test-2-b200s
timeout_in_minutes: 45 timeout_in_minutes: 90
device: b200-k8s device: b200-k8s
num_devices: 2 num_devices: 2
source_file_dependencies: source_file_dependencies:
- csrc/quantization/w8a8/cutlass/moe/ - csrc/quantization/cutlass_w8a8/moe/
- csrc/moe/ - csrc/moe/
- tests/kernels/moe - tests/kernels/moe
- vllm/model_executor/layers/fused_moe/ - vllm/model_executor/layers/fused_moe/
@@ -1055,7 +990,7 @@ steps:
source_file_dependencies: source_file_dependencies:
- csrc/quantization/fp4/ - csrc/quantization/fp4/
- csrc/attention/mla/ - csrc/attention/mla/
- csrc/quantization/w8a8/cutlass/moe/ - csrc/quantization/cutlass_w8a8/moe/
- vllm/model_executor/layers/fused_moe/cutlass_moe.py - vllm/model_executor/layers/fused_moe/cutlass_moe.py
- vllm/v1/attention/backends/triton_attn.py - vllm/v1/attention/backends/triton_attn.py
- vllm/v1/attention/backends/rocm_attn.py - vllm/v1/attention/backends/rocm_attn.py
+23 -34
View File
@@ -5,7 +5,6 @@ steps:
- label: ":nvidia: (H200 MIG 35GB) LM Eval Small Models" - label: ":nvidia: (H200 MIG 35GB) LM Eval Small Models"
device: h200_35gb device: h200_35gb
key: lm-eval-small-models key: lm-eval-small-models
automatic_shard: true
timeout_in_minutes: 45 timeout_in_minutes: 45
source_file_dependencies: source_file_dependencies:
- csrc/ - csrc/
@@ -15,9 +14,9 @@ steps:
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-small.txt - pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-small.txt
mirror: mirror:
amd: amd:
label: ":amd: (MI355) LM Eval Small Models" label: ":amd: (MI300) LM Eval Small Models"
dind: false dind: false
device: mi355_dpx device: mi300_1
timeout_in_minutes: 45 timeout_in_minutes: 45
depends_on: depends_on:
- image-build-amd - image-build-amd
@@ -105,7 +104,6 @@ steps:
- label: ":nvidia: (H100) LM Eval Large Models" - label: ":nvidia: (H100) LM Eval Large Models"
key: lm-eval-large-models-4xh100 key: lm-eval-large-models-4xh100
automatic_shard: true
device: h100 device: h100
optional: true optional: true
num_devices: 4 num_devices: 4
@@ -140,7 +138,6 @@ steps:
- label: ":nvidia: (B200) LM Eval Small Models" - label: ":nvidia: (B200) LM Eval Small Models"
key: lm-eval-small-models-1xb200 key: lm-eval-small-models-1xb200
automatic_shard: true
timeout_in_minutes: 50 timeout_in_minutes: 50
device: b200-k8s device: b200-k8s
optional: true optional: true
@@ -211,6 +208,7 @@ steps:
- tests/evals/gsm8k/ - tests/evals/gsm8k/
- tests/utils.py - tests/utils.py
- vllm/distributed/ - vllm/distributed/
- vllm/transformers_utils/configs/diffusion_gemma.py
- vllm/v1/attention/ops/ - vllm/v1/attention/ops/
commands: commands:
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-mi3xx-fp8-and-mixed.txt - pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-mi3xx-fp8-and-mixed.txt
@@ -232,13 +230,12 @@ steps:
- vllm/v1/attention/backends/mla/ - vllm/v1/attention/backends/mla/
- vllm/model_executor/layers/attention/sparse_mla_attention.py - vllm/model_executor/layers/attention/sparse_mla_attention.py
- vllm/v1/worker/gpu/ - vllm/v1/worker/gpu/
- csrc/hisparse_kernels.cu - csrc/libtorch_stable/hisparse_kernels.cu
commands: commands:
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-hisparse.txt - pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-hisparse.txt
- label: ":nvidia: (B200) LM Eval PCP" - label: ":nvidia: (B200) LM Eval PCP"
key: lm-eval-pcp-4xb200 key: lm-eval-pcp-4xb200
automatic_shard: true
timeout_in_minutes: 360 timeout_in_minutes: 360
device: b200-k8s device: b200-k8s
num_devices: 4 num_devices: 4
@@ -273,9 +270,6 @@ steps:
source_file_dependencies: source_file_dependencies:
- .buildkite/test_areas/lm_eval.yaml - .buildkite/test_areas/lm_eval.yaml
- tests/evals/gsm8k/ - tests/evals/gsm8k/
- tests/watermarking/test_compatibility.py
- tests/watermarking/test_engine_compatibility.py
- tests/watermarking/test_parallel_compatibility.py
- vllm/config/speculative.py - vllm/config/speculative.py
- vllm/config/vllm.py - vllm/config/vllm.py
- vllm/config/watermarking.py - vllm/config/watermarking.py
@@ -288,7 +282,6 @@ steps:
autorun_on_main: true autorun_on_main: true
commands: commands:
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-watermark-dspark-tp2.txt - pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-watermark-dspark-tp2.txt
- pytest -s -v watermarking/test_compatibility.py watermarking/test_engine_compatibility.py watermarking/test_parallel_compatibility.py
mirror: mirror:
amd: amd:
label: ":amd: (MI355) LM Eval Watermark Feature Combination" label: ":amd: (MI355) LM Eval Watermark Feature Combination"
@@ -325,11 +318,9 @@ steps:
optional: true optional: true
source_file_dependencies: source_file_dependencies:
- tests/evals/gsm8k/configs/DeepSeek-V4-Flash-DSpark-confidence-TEP4.yaml - tests/evals/gsm8k/configs/DeepSeek-V4-Flash-DSpark-confidence-TEP4.yaml
- tests/evals/gsm8k/configs/DeepSeek-V4.1-Flash-NVFP4-DSpark-TP4.yaml
- tests/evals/gsm8k/configs/Kimi-K3-pruned75-DSpark-TP4.yaml - tests/evals/gsm8k/configs/Kimi-K3-pruned75-DSpark-TP4.yaml
- tests/evals/gsm8k/configs/models-spec-decode.txt - tests/evals/gsm8k/configs/models-spec-decode.txt
- vllm/models/deepseek_v4/ - vllm/models/deepseek_v4/
- vllm/models/deepseek_v41/
- vllm/models/kimi_k3/ - vllm/models/kimi_k3/
- vllm/model_executor/models/qwen3_dspark.py - vllm/model_executor/models/qwen3_dspark.py
- vllm/v1/spec_decode/metrics.py - vllm/v1/spec_decode/metrics.py
@@ -374,7 +365,6 @@ steps:
- vllm/tokenizers/ - vllm/tokenizers/
- vllm/v1/attention/ - vllm/v1/attention/
- vllm/v1/core/kv_cache_utils.py - vllm/v1/core/kv_cache_utils.py
- vllm/v1/core/kv_cache_planning.py
- vllm/v1/spec_decode/metrics.py - vllm/v1/spec_decode/metrics.py
- vllm/v1/worker/ - vllm/v1/worker/
- vllm/_aiter_ops.py - vllm/_aiter_ops.py
@@ -387,7 +377,6 @@ steps:
- label: ":nvidia: (B200) LM Eval Large Models EP" - label: ":nvidia: (B200) LM Eval Large Models EP"
key: lm-eval-large-models-ep-2xb200 key: lm-eval-large-models-ep-2xb200
automatic_shard: true
timeout_in_minutes: 60 timeout_in_minutes: 60
device: b200-k8s device: b200-k8s
optional: true optional: true
@@ -422,6 +411,8 @@ steps:
- vllm/model_executor/models/qwen3_next_mtp.py - vllm/model_executor/models/qwen3_next_mtp.py
- vllm/model_executor/models/nemotron_h.py - vllm/model_executor/models/nemotron_h.py
- vllm/model_executor/models/nemotron_h_mtp.py - vllm/model_executor/models/nemotron_h_mtp.py
- vllm/transformers_utils/configs/qwen3_next.py
- vllm/transformers_utils/configs/nemotron_h.py
- vllm/model_executor/model_loader/ - vllm/model_executor/model_loader/
- vllm/model_executor/layers/quantization/ - vllm/model_executor/layers/quantization/
- vllm/model_executor/layers/fused_moe/ - vllm/model_executor/layers/fused_moe/
@@ -448,6 +439,8 @@ steps:
source_file_dependencies: source_file_dependencies:
- vllm/model_executor/models/qwen3_5.py - vllm/model_executor/models/qwen3_5.py
- vllm/model_executor/models/qwen3_5_mtp.py - vllm/model_executor/models/qwen3_5_mtp.py
- vllm/transformers_utils/configs/qwen3_5.py
- vllm/transformers_utils/configs/qwen3_5_moe.py
- vllm/model_executor/models/qwen3_next.py - vllm/model_executor/models/qwen3_next.py
- vllm/model_executor/models/qwen3_next_mtp.py - vllm/model_executor/models/qwen3_next_mtp.py
- vllm/third_party/flash_linear_attention/ops/ - vllm/third_party/flash_linear_attention/ops/
@@ -464,6 +457,8 @@ steps:
source_file_dependencies: source_file_dependencies:
- vllm/model_executor/models/qwen3_5.py - vllm/model_executor/models/qwen3_5.py
- vllm/model_executor/models/qwen3_5_mtp.py - vllm/model_executor/models/qwen3_5_mtp.py
- vllm/transformers_utils/configs/qwen3_5.py
- vllm/transformers_utils/configs/qwen3_5_moe.py
- vllm/model_executor/models/qwen2.py - vllm/model_executor/models/qwen2.py
- vllm/model_executor/models/qwen3.py - vllm/model_executor/models/qwen3.py
- vllm/model_executor/models/qwen3_next.py - vllm/model_executor/models/qwen3_next.py
@@ -521,7 +516,6 @@ steps:
- vllm/v1/spec_decode/ - vllm/v1/spec_decode/
- vllm/v1/worker/gpu/spec_decode/ - vllm/v1/worker/gpu/spec_decode/
- vllm/v1/core/kv_cache_utils.py - vllm/v1/core/kv_cache_utils.py
- vllm/v1/core/kv_cache_planning.py
- vllm/model_executor/layers/layernorm.py - vllm/model_executor/layers/layernorm.py
- csrc/ - csrc/
- vllm/_aiter_ops.py - vllm/_aiter_ops.py
@@ -565,7 +559,6 @@ steps:
- vllm/v1/spec_decode/ - vllm/v1/spec_decode/
- vllm/v1/worker/gpu/spec_decode/ - vllm/v1/worker/gpu/spec_decode/
- vllm/v1/core/kv_cache_utils.py - vllm/v1/core/kv_cache_utils.py
- vllm/v1/core/kv_cache_planning.py
- vllm/model_executor/layers/layernorm.py - vllm/model_executor/layers/layernorm.py
- csrc/ - csrc/
- vllm/_aiter_ops.py - vllm/_aiter_ops.py
@@ -609,7 +602,6 @@ steps:
- vllm/v1/spec_decode/ - vllm/v1/spec_decode/
- vllm/v1/worker/gpu/spec_decode/ - vllm/v1/worker/gpu/spec_decode/
- vllm/v1/core/kv_cache_utils.py - vllm/v1/core/kv_cache_utils.py
- vllm/v1/core/kv_cache_planning.py
- vllm/model_executor/layers/layernorm.py - vllm/model_executor/layers/layernorm.py
- csrc/ - csrc/
- vllm/_aiter_ops.py - vllm/_aiter_ops.py
@@ -653,7 +645,6 @@ steps:
- vllm/v1/spec_decode/ - vllm/v1/spec_decode/
- vllm/v1/worker/gpu/spec_decode/ - vllm/v1/worker/gpu/spec_decode/
- vllm/v1/core/kv_cache_utils.py - vllm/v1/core/kv_cache_utils.py
- vllm/v1/core/kv_cache_planning.py
- vllm/model_executor/layers/layernorm.py - vllm/model_executor/layers/layernorm.py
- csrc/ - csrc/
- vllm/_aiter_ops.py - vllm/_aiter_ops.py
@@ -697,7 +688,6 @@ steps:
- vllm/v1/spec_decode/ - vllm/v1/spec_decode/
- vllm/v1/worker/gpu/spec_decode/ - vllm/v1/worker/gpu/spec_decode/
- vllm/v1/core/kv_cache_utils.py - vllm/v1/core/kv_cache_utils.py
- vllm/v1/core/kv_cache_planning.py
- vllm/model_executor/layers/layernorm.py - vllm/model_executor/layers/layernorm.py
- csrc/ - csrc/
- vllm/_aiter_ops.py - vllm/_aiter_ops.py
@@ -707,17 +697,18 @@ steps:
- export VLLM_WORKER_MULTIPROC_METHOD=spawn - export VLLM_WORKER_MULTIPROC_METHOD=spawn
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-mi355-nemotron-3-super.txt - pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-mi355-nemotron-3-super.txt
- label: ":amd: (MI355) LM Eval Large Models" - label: ":amd: (MI300) LM Eval Large Models"
key: amd-lm-eval-large-models key: amd-lm-eval-large-models
dind: false dind: false
device: mi355_8 device: mi300_8
optional: true optional: true
timeout_in_minutes: 65 timeout_in_minutes: 65
depends_on: depends_on:
- image-build-amd - image-build-amd
commands: commands:
- export VLLM_WORKER_MULTIPROC_METHOD=spawn - export VLLM_WORKER_MULTIPROC_METHOD=spawn
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-gfx950-large.txt - export PYTORCH_ROCM_ARCH=gfx942 # Limit Quark compilation to save time
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-mi3xx.txt
source_file_dependencies: source_file_dependencies:
- vllm/_aiter_ops.py - vllm/_aiter_ops.py
- vllm/platforms/rocm.py - vllm/platforms/rocm.py
@@ -1194,7 +1185,7 @@ steps:
optional: true optional: true
num_devices: 4 num_devices: 4
source_file_dependencies: source_file_dependencies:
- csrc/gdn/ - csrc/libtorch_stable/gdn/
- tests/evals/qwen4_exp/ - tests/evals/qwen4_exp/
- vllm/model_executor/layers/mamba/ - vllm/model_executor/layers/mamba/
- vllm/models/qwen4_exp/ - vllm/models/qwen4_exp/
@@ -1212,7 +1203,7 @@ steps:
optional: true optional: true
num_devices: 4 num_devices: 4
source_file_dependencies: source_file_dependencies:
- csrc/gdn/ - csrc/libtorch_stable/gdn/
- tests/evals/qwen4_exp/ - tests/evals/qwen4_exp/
- vllm/model_executor/layers/mamba/ - vllm/model_executor/layers/mamba/
- vllm/models/qwen4_exp/ - vllm/models/qwen4_exp/
@@ -1233,7 +1224,7 @@ steps:
depends_on: depends_on:
- image-build-amd - image-build-amd
source_file_dependencies: source_file_dependencies:
- csrc/gdn/ - csrc/libtorch_stable/gdn/
- tests/evals/qwen4_exp/ - tests/evals/qwen4_exp/
- vllm/model_executor/layers/mamba/ - vllm/model_executor/layers/mamba/
- vllm/models/qwen4_exp/ - vllm/models/qwen4_exp/
@@ -1297,9 +1288,9 @@ steps:
- pytest -s -v evals/gsm8k/test_gsm8k_offloading.py -k "nemotron-h-8b or gemma-4-e4b-it" - pytest -s -v evals/gsm8k/test_gsm8k_offloading.py -k "nemotron-h-8b or gemma-4-e4b-it"
mirror: mirror:
amd: amd:
label: ":amd: (MI355 DPX) LM Eval KV-Offload" label: ":amd: (MI300) LM Eval KV-Offload"
dind: false dind: false
device: mi355_dpx device: mi300_1
timeout_in_minutes: 55 timeout_in_minutes: 55
depends_on: depends_on:
- image-build-amd - image-build-amd
@@ -1344,8 +1335,6 @@ steps:
- label: ":nvidia: (H100) LM Eval KV-Offload Large" - label: ":nvidia: (H100) LM Eval KV-Offload Large"
key: kv-offload-large key: kv-offload-large
# Nightly only: 4-GPU H100 pods wait behind smaller jobs on per-PR runs.
optional: true
timeout_in_minutes: 40 timeout_in_minutes: 40
device: h100 device: h100
num_devices: 4 num_devices: 4
@@ -1359,9 +1348,9 @@ steps:
- pytest -s -v evals/gsm8k/test_gsm8k_offloading.py -k "deepseek-v4-flash" - pytest -s -v evals/gsm8k/test_gsm8k_offloading.py -k "deepseek-v4-flash"
mirror: mirror:
amd: amd:
label: ":amd: (MI355) LM Eval KV-Offload Large" label: ":amd: (MI300) LM Eval KV-Offload Large"
dind: false dind: false
device: mi355_4 device: mi300_4
timeout_in_minutes: 65 timeout_in_minutes: 65
depends_on: depends_on:
- image-build-amd - image-build-amd
@@ -1426,10 +1415,10 @@ steps:
# Qwen2-57B has 28 attention heads and cannot run at TP8. # Qwen2-57B has 28 attention heads and cannot run at TP8.
- pytest -s -v test_lm_eval_correctness.py --config-list-file=configs/models-large.txt --tp-size=4 - pytest -s -v test_lm_eval_correctness.py --config-list-file=configs/models-large.txt --tp-size=4
- label: ":amd: (MI355) LM Eval Large Models ROCm Harness" - label: ":amd: (MI300) LM Eval Large Models ROCm Harness"
key: amd-lm-eval-large-models-rocm-harness key: amd-lm-eval-large-models-rocm-harness
dind: false dind: false
device: mi355_8 device: mi300_8
optional: true optional: true
timeout_in_minutes: 40 timeout_in_minutes: 40
depends_on: depends_on:
+8 -186
View File
@@ -4,8 +4,7 @@ depends_on:
steps: steps:
- label: ":nvidia: (H200 MIG 18GB) V1 Sample" - label: ":nvidia: (H200 MIG 18GB) V1 Sample"
key: v1-sample key: v1-sample
automatic_shard: true timeout_in_minutes: 35
timeout_in_minutes: 45
device: h200_18gb device: h200_18gb
source_file_dependencies: &v1-sample-logits-deps source_file_dependencies: &v1-sample-logits-deps
- vllm/config/ - vllm/config/
@@ -32,7 +31,6 @@ steps:
amd: amd:
label: ":amd: (MI355 DPX) V1 Sample" label: ":amd: (MI355 DPX) V1 Sample"
dind: false dind: false
automatic_shard: true
device: mi355_dpx device: mi355_dpx
timeout_in_minutes: 70 timeout_in_minutes: 70
depends_on: depends_on:
@@ -40,7 +38,7 @@ steps:
- label: ":nvidia: (H200 MIG 18GB) V1 Logits + Oracle" - label: ":nvidia: (H200 MIG 18GB) V1 Logits + Oracle"
key: v1-logits-oracle key: v1-logits-oracle
timeout_in_minutes: 45 timeout_in_minutes: 35
device: h200_18gb device: h200_18gb
source_file_dependencies: *v1-sample-logits-deps source_file_dependencies: *v1-sample-logits-deps
commands: commands:
@@ -164,59 +162,6 @@ steps:
depends_on: depends_on:
- image-build-amd - image-build-amd
# The shared KV Connectors steps above exclude vllm/distributed/kv_transfer/ from
# their triggers, and CUDA skips MoRIIO's mori/aiter-gated tests. Run the whole
# MoRIIO suite on ROCm whenever the connector or its tests change.
- label: ":amd: (MI355 DPX) MoRIIO Connector Unit Tests"
key: amd-moriio-connector-unit-mi355
dind: false
device: mi355_dpx
num_devices: 1
timeout_in_minutes: 30
working_dir: /vllm-workspace/tests
depends_on:
- image-build-amd
source_file_dependencies:
- vllm/distributed/kv_transfer/kv_connector/v1/moriio/
- vllm/distributed/kv_transfer/kv_connector/v1/base.py
- vllm/distributed/kv_transfer/kv_connector/v1/ssm_conv_transfer_utils.py
- examples/disaggregated/disaggregated_serving/moriio_toy_proxy_server.py
- tests/v1/kv_connector/unit/test_moriio_
- tests/v1/kv_connector/unit/utils.py
- requirements/kv_connectors_rocm.txt
commands:
- bash /vllm-workspace/.buildkite/scripts/install-kv-connectors.sh
- export VLLM_WORKER_MULTIPROC_METHOD=spawn
- pytest -v -s -rs v1/kv_connector/unit/test_moriio_*.py
# Single-node MoRIIO WRITE P/D on dummy weights: prefill and decode on separate
# GPUs over xGMI, compared exactly against a direct serve.
- label: ":amd: (MI355) MoRIIO WRITE P/D Dummy-Weight E2E Shard %N"
key: amd-moriio-write-dummy-e2e-mi355
dind: false
device: mi355_2
num_devices: 2
# ~3.5-5 min per case; pytest-shard hashes test ids, so with 20 cases the
# largest of 6 shards holds 5 (~22 min on MI355). Every case picks free ports
# for its servers and proxy and its own compile-cache dir.
parallelism: 6
timeout_in_minutes: 45
working_dir: /vllm-workspace/tests
depends_on:
- image-build-amd
source_file_dependencies:
- vllm/distributed/kv_transfer/kv_connector/v1/moriio/
- vllm/distributed/kv_transfer/kv_connector/v1/base.py
- vllm/distributed/kv_transfer/kv_connector/v1/ssm_conv_transfer_utils.py
- vllm/model_executor/model_loader/dummy_loader.py
- examples/disaggregated/disaggregated_serving/moriio_toy_proxy_server.py
- tests/v1/kv_connector/moriio_integration/
- requirements/kv_connectors_rocm.txt
commands:
- bash /vllm-workspace/.buildkite/scripts/install-kv-connectors.sh
- export VLLM_WORKER_MULTIPROC_METHOD=spawn
- pytest -v -s -rs v1/kv_connector/moriio_integration/test_write_dummy_e2e_rocm.py --shard-id=$$BUILDKITE_PARALLEL_JOB --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT
- label: ":nvidia: (H200 MIG 35GB) V1 Metrics + LM Eval" - label: ":nvidia: (H200 MIG 35GB) V1 Metrics + LM Eval"
device: h200_35gb device: h200_35gb
key: v1-metrics-lmeval key: v1-metrics-lmeval
@@ -267,14 +212,12 @@ steps:
- vllm/v1/ - vllm/v1/
- tests/v1 - tests/v1
- tests/watermarking - tests/watermarking
- examples/disaggregated/mooncake_connector/mooncake_connector_proxy.py
device: cpu-small device: cpu-small
commands: commands:
# split the test to avoid interference # split the test to avoid interference
- pytest -v -s -m 'cpu_test' v1/core - pytest -v -s -m 'cpu_test' v1/core
- pytest -v -s v1/structured_output - pytest -v -s v1/structured_output
- pytest -v -s v1/test_serial_utils.py - pytest -v -s v1/test_serial_utils.py
- pytest -v -s v1/worker/test_cpu_worker_kv_budget.py
- pytest -v -s v1/test_kv_cache_spec_registry.py - pytest -v -s v1/test_kv_cache_spec_registry.py
- pytest -v -s v1/cudagraph/test_cudagraph_manager.py - pytest -v -s v1/cudagraph/test_cudagraph_manager.py
- pytest -v -s -m 'cpu_test' v1/kv_connector/unit - pytest -v -s -m 'cpu_test' v1/kv_connector/unit
@@ -353,6 +296,7 @@ steps:
- vllm/model_executor/models/qwen3_next.py - vllm/model_executor/models/qwen3_next.py
- vllm/model_executor/models/registry.py - vllm/model_executor/models/registry.py
- vllm/transformers_utils/configs/extract_hidden_states.py - vllm/transformers_utils/configs/extract_hidden_states.py
- vllm/transformers_utils/configs/qwen3_5.py
- vllm/v1/attention/backends/ - vllm/v1/attention/backends/
- vllm/v1/attention/selector.py - vllm/v1/attention/selector.py
- vllm/v1/kv_cache_interface.py - vllm/v1/kv_cache_interface.py
@@ -502,17 +446,6 @@ steps:
- setup.py - setup.py
- vllm/platforms/rocm.py - vllm/platforms/rocm.py
- label: ":nvidia: (H200 MIG 18GB) Python 3.14 Smoke Nightly"
key: python-314-smoke-nightly
depends_on: ~
optional: true
timeout_in_minutes: 30
device: h200_18gb
source_file_dependencies:
- tests/standalone_tests/python_only_compile.sh
commands:
- PYTHON_ONLY_COMPILE_PYTHON=3.14 bash standalone_tests/python_only_compile.sh
- label: ":nvidia: (H200 MIG 35GB) Async Engine, Inputs, Utils, Worker" - label: ":nvidia: (H200 MIG 35GB) Async Engine, Inputs, Utils, Worker"
device: h200_35gb device: h200_35gb
key: async-engine-inputs-utils-worker key: async-engine-inputs-utils-worker
@@ -623,8 +556,6 @@ steps:
- tests/multimodal - tests/multimodal
- tests/renderers - tests/renderers
- tests/standalone_tests/lazy_imports.py - tests/standalone_tests/lazy_imports.py
- tests/standalone_tests/test_precompiled_wheel.py
- setup.py
- tests/reasoning - tests/reasoning
- tools/pre_commit/check_test_tethering.py - tools/pre_commit/check_test_tethering.py
- tools/pre_commit/test_tethering_allowlist.txt - tools/pre_commit/test_tethering_allowlist.txt
@@ -641,7 +572,6 @@ steps:
commands: commands:
- pip install cmake # test_cmake_utils.py shells out to cmake; the CPU image has none - pip install cmake # test_cmake_utils.py shells out to cmake; the CPU image has none
- python3 standalone_tests/lazy_imports.py - python3 standalone_tests/lazy_imports.py
- python3 standalone_tests/test_precompiled_wheel.py
- pytest -v -s device_allocator - pytest -v -s device_allocator
- pytest -v -s test_uvicorn_logging.py - pytest -v -s test_uvicorn_logging.py
- pytest -v -s test_attention_backend_registry.py - pytest -v -s test_attention_backend_registry.py
@@ -711,45 +641,21 @@ steps:
source_file_dependencies: source_file_dependencies:
- vllm/v1/attention - vllm/v1/attention
- vllm/model_executor/layers - vllm/model_executor/layers
- vllm/model_executor/kernels/linear
- vllm/model_executor/determinism
- tests/v1/determinism/ - tests/v1/determinism/
commands: commands:
- export VLLM_WORKER_MULTIPROC_METHOD=spawn - export VLLM_WORKER_MULTIPROC_METHOD=spawn
- pip install pytest-timeout pytest-forked - pip install pytest-timeout pytest-forked
- pytest -v -s v1/determinism/test_batch_invariance.py - pytest -v -s v1/determinism/test_batch_invariance.py
- VLLM_TEST_MODEL=deepseek-ai/DeepSeek-V2-Lite-Chat pytest -v -s v1/determinism/test_batch_invariance.py::test_v1_generation_is_deterministic_across_batch_sizes_with_needle -k TRITON_MLA - VLLM_TEST_MODEL=deepseek-ai/DeepSeek-V2-Lite-Chat pytest -v -s v1/determinism/test_batch_invariance.py::test_v1_generation_is_deterministic_across_batch_sizes_with_needle -k TRITON_MLA
mirror:
amd:
label: ":amd: (MI250) Batch Invariance"
dind: false
device: mi250_1
timeout_in_minutes: 60
env:
VLLM_GPU_MEMORY_UTILIZATION: "0.8"
VLLM_TEST_CLEAN_GPU_MEMORY: "1"
depends_on:
- image-build-amd
source_file_dependencies:
- vllm/v1/attention
- vllm/model_executor/layers
- vllm/model_executor/kernels/linear
- vllm/model_executor/determinism
- tests/v1/determinism/
- vllm/_aiter_ops.py
- vllm/platforms/rocm.py
- label: ":nvidia: (H100) Batch Invariance" - label: ":nvidia: (H100) Batch Invariance"
key: batch-invariance-h100 key: batch-invariance-h100
automatic_shard: true
timeout_in_minutes: 60 timeout_in_minutes: 60
device: h100 device: h100
source_file_dependencies: source_file_dependencies:
- vllm/v1/attention - vllm/v1/attention
- vllm/model_executor/layers - vllm/model_executor/layers
- vllm/lora/ops/triton_ops - vllm/lora/ops/triton_ops
- vllm/model_executor/kernels/linear
- vllm/model_executor/determinism
- tests/v1/determinism/ - tests/v1/determinism/
commands: commands:
- export VLLM_WORKER_MULTIPROC_METHOD=spawn - export VLLM_WORKER_MULTIPROC_METHOD=spawn
@@ -761,19 +667,15 @@ steps:
- VLLM_TEST_MODEL=Qwen/Qwen3-30B-A3B-Thinking-2507-FP8 pytest -v -s v1/determinism/test_batch_invariance.py::test_v1_generation_is_deterministic_across_batch_sizes_with_needle -k FLASH_ATTN - VLLM_TEST_MODEL=Qwen/Qwen3-30B-A3B-Thinking-2507-FP8 pytest -v -s v1/determinism/test_batch_invariance.py::test_v1_generation_is_deterministic_across_batch_sizes_with_needle -k FLASH_ATTN
mirror: mirror:
amd: amd:
label: ":amd: (MI300) Batch Invariance" label: ":amd: (MI250) Batch Invariance"
dind: false dind: false
automatic_shard: true device: mi250_1
device: mi300_1 timeout_in_minutes: 35
timeout_in_minutes: 60
depends_on: depends_on:
- image-build-amd - image-build-amd
source_file_dependencies: source_file_dependencies:
- vllm/v1/attention - vllm/v1/attention
- vllm/model_executor/layers - vllm/model_executor/layers
- vllm/lora/ops/triton_ops
- vllm/model_executor/kernels/linear
- vllm/model_executor/determinism
- tests/v1/determinism/ - tests/v1/determinism/
- vllm/_aiter_ops.py - vllm/_aiter_ops.py
- vllm/platforms/rocm.py - vllm/platforms/rocm.py
@@ -782,23 +684,15 @@ steps:
- pip install pytest-timeout pytest-forked - pip install pytest-timeout pytest-forked
- pytest -v -s v1/determinism/test_batch_invariance.py - pytest -v -s v1/determinism/test_batch_invariance.py
- pytest -v -s v1/determinism/test_rms_norm_batch_invariant.py - pytest -v -s v1/determinism/test_rms_norm_batch_invariant.py
- pytest -v -s v1/determinism/test_lora_shrink_batch_invariant.py
- pytest -v -s v1/determinism/test_cutlass_batch_invariance.py
- pytest -v -s v1/determinism/test_matmul_batch_invariant.py
- VLLM_TEST_MODEL=deepseek-ai/DeepSeek-V2-Lite-Chat pytest -v -s v1/determinism/test_batch_invariance.py::test_v1_generation_is_deterministic_across_batch_sizes_with_needle -k TRITON_MLA
- VLLM_TEST_MODEL=Qwen/Qwen3-30B-A3B-Thinking-2507-FP8 pytest -v -s v1/determinism/test_batch_invariance.py::test_v1_generation_is_deterministic_across_batch_sizes_with_needle -k TRITON_ATTN
- label: ":nvidia: (B200) Batch Invariance" - label: ":nvidia: (B200) Batch Invariance"
key: batch-invariance-b200 key: batch-invariance-b200
automatic_shard: true
timeout_in_minutes: 45 timeout_in_minutes: 45
device: b200-k8s device: b200-k8s
source_file_dependencies: source_file_dependencies:
- vllm/v1/attention - vllm/v1/attention
- vllm/model_executor/layers - vllm/model_executor/layers
- vllm/lora/ops/triton_ops - vllm/lora/ops/triton_ops
- vllm/model_executor/kernels/linear
- vllm/model_executor/determinism
- tests/v1/determinism/ - tests/v1/determinism/
commands: commands:
- export VLLM_WORKER_MULTIPROC_METHOD=spawn - export VLLM_WORKER_MULTIPROC_METHOD=spawn
@@ -813,80 +707,6 @@ steps:
- pytest -v -s v1/determinism/test_matmul_batch_invariant.py - pytest -v -s v1/determinism/test_matmul_batch_invariant.py
- pytest -v -s v1/determinism/test_cutlass_batch_invariance.py - pytest -v -s v1/determinism/test_cutlass_batch_invariance.py
- pytest -v -s v1/determinism/test_online_batch_invariance.py - pytest -v -s v1/determinism/test_online_batch_invariance.py
mirror:
amd:
label: ":amd: (MI355) Batch Invariance"
dind: false
device: mi355_1
timeout_in_minutes: 60
depends_on:
- image-build-amd
source_file_dependencies:
- vllm/v1/attention
- vllm/model_executor/layers
- vllm/lora/ops/triton_ops
- vllm/model_executor/kernels/linear
- vllm/model_executor/determinism
- tests/v1/determinism/
- vllm/_aiter_ops.py
- vllm/platforms/rocm.py
commands:
- export VLLM_WORKER_MULTIPROC_METHOD=spawn
- pip install pytest-timeout pytest-forked
- pytest -v -s v1/determinism/test_batch_invariance.py
- pytest -v -s v1/determinism/test_rms_norm_batch_invariant.py
- pytest -v -s v1/determinism/test_lora_shrink_batch_invariant.py
- pytest -v -s v1/determinism/test_batch_invariance_vlm.py
- VLLM_TEST_MODEL=deepseek-ai/DeepSeek-V2-Lite-Chat pytest -v -s v1/determinism/test_batch_invariance.py::test_v1_generation_is_deterministic_across_batch_sizes_with_needle -k TRITON_MLA
- VLLM_TEST_MODEL=Qwen/Qwen3-30B-A3B-Thinking-2507-FP8 pytest -v -s v1/determinism/test_batch_invariance.py::test_v1_generation_is_deterministic_across_batch_sizes_with_needle -k TRITON_ATTN
- pytest -v -s v1/determinism/test_nvfp4_batch_invariant.py
- pytest -v -s v1/determinism/test_matmul_batch_invariant.py
- pytest -v -s v1/determinism/test_cutlass_batch_invariance.py
- pytest -v -s v1/determinism/test_online_batch_invariance.py
- pytest -v -s v1/determinism/test_mx_linear_batch_invariant.py
- label: ":nvidia: (H100) Batch Invariance (Distributed)"
key: batch-invariance-distributed-4xh100
automatic_shard: true
timeout_in_minutes: 45
device: h100
optional: true
num_devices: 4
source_file_dependencies:
- vllm/distributed/device_communicators/
- vllm/model_executor/determinism
- vllm/model_executor/layers
- vllm/v1/attention
- tests/v1/determinism/
commands:
- export VLLM_WORKER_MULTIPROC_METHOD=spawn
- export VLLM_TEST_CLEAN_GPU_MEMORY=1
- VLLM_TEST_TP_SIZE=4 pytest -v -s v1/determinism/test_batch_invariance.py -k "bs1_vs_bsN or without_batch_invariance_should_fail"
- VLLM_TEST_TP_SIZE=4 VLLM_TEST_DISABLE_CUSTOM_AR=1 pytest -v -s v1/determinism/test_batch_invariance.py -k bs1_vs_bsN
- VLLM_TP_SIZE=4 VLLM_NEEDLE_TRIALS=2 pytest -v -s v1/determinism/test_batch_invariance.py::test_v1_generation_is_deterministic_across_batch_sizes_with_needle -k FLASH_ATTN
- VLLM_TEST_MODEL=allenai/OLMoE-1B-7B-0924 VLLM_TEST_SERVER_ARGS="--data-parallel-size 4 --enable-expert-parallel --max-model-len 4096" pytest -v -s v1/determinism/test_online_batch_invariance.py
mirror:
amd:
label: ":amd: (MI300) Batch Invariance (Distributed)"
dind: false
device: mi300_4
timeout_in_minutes: 45
depends_on:
- image-build-amd
source_file_dependencies:
- vllm/distributed/device_communicators/
- vllm/model_executor/determinism
- vllm/model_executor/layers
- vllm/v1/attention
- tests/v1/determinism/
- vllm/platforms/rocm.py
commands:
- export VLLM_WORKER_MULTIPROC_METHOD=spawn
- export VLLM_TEST_CLEAN_GPU_MEMORY=1
- VLLM_TEST_TP_SIZE=4 pytest -v -s v1/determinism/test_batch_invariance.py -k "bs1_vs_bsN or without_batch_invariance_should_fail"
- VLLM_TEST_TP_SIZE=4 VLLM_TEST_DISABLE_CUSTOM_AR=1 pytest -v -s v1/determinism/test_batch_invariance.py -k bs1_vs_bsN
- VLLM_TP_SIZE=4 VLLM_NEEDLE_TRIALS=2 pytest -v -s v1/determinism/test_batch_invariance.py::test_v1_generation_is_deterministic_across_batch_sizes_with_needle -k TRITON_ATTN
- VLLM_TEST_MODEL=allenai/OLMoE-1B-7B-0924 VLLM_TEST_SERVER_ARGS="--data-parallel-size 4 --enable-expert-parallel --max-model-len 4096" pytest -v -s v1/determinism/test_online_batch_invariance.py
- label: ":nvidia: (H200 MIG 35GB) Acceptance Length (Large Models)" - label: ":nvidia: (H200 MIG 35GB) Acceptance Length (Large Models)"
device: h200_35gb device: h200_35gb
@@ -898,6 +718,7 @@ steps:
working_dir: "/vllm-workspace/tests" working_dir: "/vllm-workspace/tests"
source_file_dependencies: source_file_dependencies:
- vllm/v1/spec_decode/ - vllm/v1/spec_decode/
- vllm/model_executor/models/mlp_speculator.py
- tests/v1/spec_decode/test_acceptance_length.py - tests/v1/spec_decode/test_acceptance_length.py
commands: commands:
- export VLLM_ALLOW_INSECURE_SERIALIZATION=1 - export VLLM_ALLOW_INSECURE_SERIALIZATION=1
@@ -913,5 +734,6 @@ steps:
working_dir: /vllm-workspace/tests working_dir: /vllm-workspace/tests
source_file_dependencies: source_file_dependencies:
- vllm/v1/spec_decode/ - vllm/v1/spec_decode/
- vllm/model_executor/models/mlp_speculator.py
- tests/v1/spec_decode/test_acceptance_length.py - tests/v1/spec_decode/test_acceptance_length.py
- vllm/platforms/rocm.py - vllm/platforms/rocm.py
@@ -5,7 +5,6 @@ steps:
- label: ":nvidia: (H200 MIG 35GB) Model Executor" - label: ":nvidia: (H200 MIG 35GB) Model Executor"
device: h200_35gb device: h200_35gb
key: model-executor key: model-executor
automatic_shard: true
timeout_in_minutes: 60 timeout_in_minutes: 60
source_file_dependencies: source_file_dependencies:
- vllm/engine/arg_utils.py - vllm/engine/arg_utils.py
+11 -56
View File
@@ -58,7 +58,6 @@ steps:
- label: ":nvidia: (H200 MIG 35GB) Basic Models (Other)" - label: ":nvidia: (H200 MIG 35GB) Basic Models (Other)"
device: h200_35gb device: h200_35gb
key: basic-models-tests-other key: basic-models-tests-other
automatic_shard: true
timeout_in_minutes: 38 timeout_in_minutes: 38
source_file_dependencies: source_file_dependencies:
- vllm/ - vllm/
@@ -69,12 +68,8 @@ steps:
- tests/models/test_deepseek_v4_vl_rocm.py - tests/models/test_deepseek_v4_vl_rocm.py
- tests/models/test_deepseek_v41_decoder_replay_layers.py - tests/models/test_deepseek_v41_decoder_replay_layers.py
- tests/models/test_deepseek_v41_replay_batch.py - tests/models/test_deepseek_v41_replay_batch.py
- tests/models/test_deepseek_v41_pcp.py
- tests/models/transformers/test_layer_registry_e2e.py
- tests/models/test_deepseek_v41_replay_start.py
- tests/models/test_deepseek_v4_vl_stream_weights.py
commands: commands:
- pytest -v -s models/test_terratorch.py models/transformers/test_backend.py models/test_registry.py models/test_deepseek_v4_vl_rocm.py models/transformers/test_layer_registry_e2e.py models/test_deepseek_v41_replay_start.py models/test_deepseek_v41_decoder_replay_layers.py models/test_deepseek_v41_replay_batch.py models/test_deepseek_v41_pcp.py models/test_deepseek_v4_vl_stream_weights.py - pytest -v -s models/test_terratorch.py models/transformers/test_backend.py models/test_registry.py models/test_deepseek_v4_vl_rocm.py models/test_deepseek_v41_decoder_replay_layers.py models/test_deepseek_v41_replay_batch.py
mirror: mirror:
amd: amd:
label: ":amd: (MI355) Basic Models (Other)" label: ":amd: (MI355) Basic Models (Other)"
@@ -84,21 +79,10 @@ steps:
depends_on: depends_on:
- image-build-amd - image-build-amd
source_file_dependencies: source_file_dependencies:
- vllm/models/deepseek_v4/
- vllm/models/deepseek_v41/
- tests/models/test_deepseek_v4_vl_rocm.py - tests/models/test_deepseek_v4_vl_rocm.py
- tests/models/test_hyv4_rocm.py - tests/models/test_hyv4_rocm.py
- tests/models/transformers/test_layer_registry_e2e.py
- tests/models/test_deepseek_v41_replay_start.py
- tests/models/test_deepseek_v41_decoder_replay_layers.py
- tests/models/test_deepseek_v41_replay_batch.py
- tests/models/test_deepseek_v41_amd_replay_forward.py
- tests/models/test_deepseek_v4_dspark_rocm.py
- tests/models/test_deepseek_v4_rocm_wo_a.py
- tests/models/test_deepseek_v4_rocm_compressor_gemm_fusion.py
- tests/models/test_dsv41_mono_decode.py
commands: commands:
- pytest -v -s models/test_terratorch.py models/transformers/test_backend.py models/test_registry.py models/test_deepseek_v4_vl_rocm.py models/transformers/test_layer_registry_e2e.py models/test_deepseek_v41_replay_start.py models/test_deepseek_v41_decoder_replay_layers.py models/test_deepseek_v41_replay_batch.py models/test_deepseek_v41_amd_replay_forward.py models/test_deepseek_v4_dspark_rocm.py models/test_deepseek_v4_rocm_wo_a.py models/test_deepseek_v4_rocm_compressor_gemm_fusion.py models/test_dsv41_mono_decode.py - pytest -v -s models/test_terratorch.py models/transformers/test_backend.py models/test_registry.py models/test_deepseek_v4_vl_rocm.py
- VLLM_ROCM_USE_AITER=1 pytest -v -s models/test_hyv4_rocm.py -m 'not distributed' - VLLM_ROCM_USE_AITER=1 pytest -v -s models/test_hyv4_rocm.py -m 'not distributed'
- label: ":nvidia: (B200) Inkling" - label: ":nvidia: (B200) Inkling"
@@ -137,7 +121,7 @@ steps:
device: b200-k8s device: b200-k8s
source_file_dependencies: source_file_dependencies:
- vllm/models/kimi_k3/ - vllm/models/kimi_k3/
- csrc/kimi_k3/ - csrc/libtorch_stable/kimi_k3/
- tests/models/kimi_k3/ - tests/models/kimi_k3/
- tests/kernels/attention/test_kimi_k3_mla_fused_epilogue.py - tests/kernels/attention/test_kimi_k3_mla_fused_epilogue.py
commands: commands:
@@ -161,11 +145,8 @@ steps:
# cuda.bindings.driver), so it is deliberately left out here. # cuda.bindings.driver), so it is deliberately left out here.
- pytest -v -s models/kimi_k3 - pytest -v -s models/kimi_k3
- label: ":nvidia: (4xB200) Kimi K3 Prefix Cache" - label: ":nvidia: (4xB200) Kimi K3 Prefix Cache Shard %N"
key: kimi-k3-prefix-cache-4xb200 key: kimi-k3-prefix-cache-4xb200
# Nightly only: 4-GPU B200 pods wait hours behind smaller jobs on per-PR runs.
optional: true
automatic_shard: true
timeout_in_minutes: 75 timeout_in_minutes: 75
device: b200-k8s device: b200-k8s
num_devices: 4 num_devices: 4
@@ -174,8 +155,6 @@ steps:
- vllm/v1/core/ - vllm/v1/core/
- vllm/distributed/kv_transfer/kv_connector/v1/nixl/ - vllm/distributed/kv_transfer/kv_connector/v1/nixl/
- vllm/distributed/kv_transfer/kv_connector/v1/simple_cpu_offload_connector.py - vllm/distributed/kv_transfer/kv_connector/v1/simple_cpu_offload_connector.py
- vllm/distributed/kv_transfer/kv_connector/v1/mooncake/store/
- vllm/distributed/mooncake_store.py
- vllm/v1/simple_kv_offload/ - vllm/v1/simple_kv_offload/
- vllm/v1/spec_decode/ - vllm/v1/spec_decode/
- vllm/model_executor/layers/mamba/ - vllm/model_executor/layers/mamba/
@@ -183,7 +162,8 @@ steps:
- .buildkite/scripts/install-kv-connectors.sh - .buildkite/scripts/install-kv-connectors.sh
commands: commands:
- bash /vllm-workspace/.buildkite/scripts/install-kv-connectors.sh - bash /vllm-workspace/.buildkite/scripts/install-kv-connectors.sh
- pytest -v -s models/kimi_k3/test_prefix_cache.py - pytest -v -s models/kimi_k3/test_prefix_cache.py --shard-id=$$BUILDKITE_PARALLEL_JOB --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT
parallelism: 2
- label: ":nvidia: (H200) GLM5Next" - label: ":nvidia: (H200) GLM5Next"
key: glm5next-unit-tests key: glm5next-unit-tests
@@ -212,12 +192,12 @@ steps:
device: h200_35gb device: h200_35gb
source_file_dependencies: source_file_dependencies:
- vllm/models/qwen4_exp/ - vllm/models/qwen4_exp/
- vllm/transformers_utils/configs/qwen4_exp.py
- vllm/config/compilation.py - vllm/config/compilation.py
- vllm/v1/attention/backends/short_conv_attn.py - vllm/v1/attention/backends/short_conv_attn.py
- tests/models/qwen4_exp/ - tests/models/qwen4_exp/
commands: commands:
- pytest -v -s models/qwen4_exp/test_hc_ops.py models/qwen4_exp/test_qsa_prepare.py models/qwen4_exp/test_qsa_reference.py - pytest -v -s models/qwen4_exp/test_hc_ops.py models/qwen4_exp/test_qsa_prepare.py models/qwen4_exp/test_qsa_reference.py
- pytest -v -s models/qwen4_exp/test_ple.py
mirror: mirror:
amd: amd:
label: ":amd: (MI355 DPX) Qwen4 Exp" label: ":amd: (MI355 DPX) Qwen4 Exp"
@@ -226,7 +206,7 @@ steps:
depends_on: depends_on:
- image-build-amd - image-build-amd
commands: commands:
- pytest -v -s models/qwen4_exp/test_qsa_amd.py models/qwen4_exp/test_qsa_prepare_amd.py - pytest -v -s models/qwen4_exp/test_qsa_amd.py
# PLE and QSA import conflicting HC registrations in a shared process. # PLE and QSA import conflicting HC registrations in a shared process.
- pytest -v -s models/qwen4_exp/test_ple.py - pytest -v -s models/qwen4_exp/test_ple.py
@@ -238,6 +218,7 @@ steps:
device: cpu-small device: cpu-small
source_file_dependencies: source_file_dependencies:
- vllm/models/qwen4_exp/ - vllm/models/qwen4_exp/
- vllm/transformers_utils/configs/qwen4_exp.py
- vllm/config/compilation.py - vllm/config/compilation.py
- vllm/v1/attention/backends/short_conv_attn.py - vllm/v1/attention/backends/short_conv_attn.py
- tests/models/qwen4_exp/ - tests/models/qwen4_exp/
@@ -248,7 +229,7 @@ steps:
key: basic-models-test-other-cpu key: basic-models-test-other-cpu
depends_on: depends_on:
- image-build-cpu - image-build-cpu
timeout_in_minutes: 30 timeout_in_minutes: 20
source_file_dependencies: source_file_dependencies:
- vllm/ - vllm/
- "!vllm/distributed/kv_transfer/" - "!vllm/distributed/kv_transfer/"
@@ -256,34 +237,8 @@ steps:
- tests/models/test_vision.py - tests/models/test_vision.py
- tests/models/test_adapters.py - tests/models/test_adapters.py
- tests/models/test_qwen3_5_mtp_config.py - tests/models/test_qwen3_5_mtp_config.py
- tests/models/test_alice_ai.py
- tests/models/test_alice_ai_config.py
- tests/models/transformers/fusers/ - tests/models/transformers/fusers/
- tests/models/transformers/test_dynamic_rope.py
- tests/model_executor/layers/test_activation.py - tests/model_executor/layers/test_activation.py
- tests/models/test_language_model_cache_is_weak.py
- tests/models/deepseek_v32/test_sequence_parallel.py
- tests/models/test_deepseek_v4_fi_moe_ep.py
- tests/models/test_dspark_mla.py
- tests/models/transformers/test_layer_registry.py
device: cpu-small device: cpu-small
commands: commands:
- pytest -v -s models/test_utils.py models/test_vision.py models/test_adapters.py models/test_qwen3_5_mtp_config.py models/transformers/fusers/ models/transformers/test_dynamic_rope.py model_executor/layers/test_activation.py models/test_language_model_cache_is_weak.py models/deepseek_v32/test_sequence_parallel.py models/test_deepseek_v4_fi_moe_ep.py models/test_dspark_mla.py models/transformers/test_layer_registry.py models/test_alice_ai.py models/test_alice_ai_config.py - pytest -v -s models/test_utils.py models/test_vision.py models/test_adapters.py models/test_qwen3_5_mtp_config.py models/transformers/fusers/ model_executor/layers/test_activation.py
- label: ':amd: (MI355) DeepSeek-V4.1 Mono Decode Numerics'
key: dsv41-mono-decode-numerics-amd
dind: false
device: mi355_4
num_devices: 4
timeout_in_minutes: 40
working_dir: /vllm-workspace/
depends_on:
- image-build-amd
source_file_dependencies:
- docker/Dockerfile.rocm_base
- vllm/models/deepseek_v41/amd/
- tests/models/test_dsv41_mono_numerics.py
- tests/models/test_dsv41_mono_layers.py
commands:
- pytest -v -s tests/models/test_dsv41_mono_numerics.py
- pytest -v -s tests/models/test_dsv41_mono_layers.py
@@ -2,39 +2,6 @@ group: Models - Distributed
depends_on: depends_on:
- image-build - image-build
steps: steps:
- label: ":nvidia: (L4) Qwen4 Exp SP"
key: qwen4-exp-sp-tests-4-gpus
timeout_in_minutes: 30
working_dir: "/vllm-workspace/tests"
device: l4
num_devices: 4
source_file_dependencies:
- vllm/config/parallel.py
- vllm/models/common/ops/sequence_parallel.py
- vllm/models/qwen4_exp/
- vllm/model_executor/layers/vocab_parallel_embedding.py
- vllm/model_executor/models/qwen3_next.py
- tests/models/qwen4_exp/test_ple.py
- tests/models/qwen4_exp/test_tpsp.py
commands:
- pytest -v -s models/qwen4_exp/test_tpsp.py
mirror:
amd:
label: ":amd: (MI355) Qwen4 Exp SP"
dind: false
device: mi355_4
depends_on:
- image-build-amd
source_file_dependencies:
- vllm/config/parallel.py
- vllm/models/common/ops/sequence_parallel.py
- vllm/models/qwen4_exp/
- vllm/model_executor/layers/vocab_parallel_embedding.py
- vllm/model_executor/models/qwen3_next.py
- vllm/platforms/rocm.py
- tests/models/qwen4_exp/test_ple.py
- tests/models/qwen4_exp/test_tpsp.py
- label: ":nvidia: (L4) Distributed Models Shard %N" - label: ":nvidia: (L4) Distributed Models Shard %N"
key: distributed-model-tests-2-gpus key: distributed-model-tests-2-gpus
timeout_in_minutes: 35 timeout_in_minutes: 35
+9 -13
View File
@@ -184,6 +184,8 @@ steps:
source_file_dependencies: source_file_dependencies:
- vllm/model_executor/models/qwen3_5.py - vllm/model_executor/models/qwen3_5.py
- vllm/model_executor/models/qwen3_5_mtp.py - vllm/model_executor/models/qwen3_5_mtp.py
- vllm/transformers_utils/configs/qwen3_5.py
- vllm/transformers_utils/configs/qwen3_5_moe.py
- vllm/model_executor/models/qwen2.py - vllm/model_executor/models/qwen2.py
- vllm/model_executor/models/qwen3.py - vllm/model_executor/models/qwen3.py
- vllm/model_executor/models/qwen3_next.py - vllm/model_executor/models/qwen3_next.py
@@ -199,37 +201,31 @@ steps:
- vllm/platforms/rocm.py - vllm/platforms/rocm.py
- tests/models/language/generation_ppl_test - tests/models/language/generation_ppl_test
- label: ":nvidia: (H200 MIG 35GB) Language Models (Extended Pooling)" - label: ":nvidia: (H200 MIG 35GB) Language Models (Extended Pooling) Shard %N"
device: h200_35gb device: h200_35gb
key: language-models-test-extended-pooling key: language-models-test-extended-pooling
automatic_shard: true
timeout_in_minutes: 120 timeout_in_minutes: 120
parallelism: 4
optional: true optional: true
source_file_dependencies: source_file_dependencies:
- vllm/ - vllm/
- "!vllm/distributed/kv_transfer/" - "!vllm/distributed/kv_transfer/"
- tests/models/language/pooling - tests/models/language/pooling
commands: commands:
# Per-test timeout: a hung test fails with a stack dump instead of holding - pytest -v -s models/language/pooling -m 'not core_model' --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT --shard-id=$$BUILDKITE_PARALLEL_JOB
# the job until Buildkite cancels it, which also blocks the full run's
# coverage collection. A whole shard takes 12-21 minutes.
- pytest -v -s models/language/pooling -m 'not core_model' --timeout=900 --timeout-method=thread
mirror: mirror:
amd: amd:
parallelism: 4 label: ":amd: (MI300) Language Models (Extended Pooling) Shard %N"
commands:
- "pytest -v -s models/language/pooling -m 'not core_model' --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT --shard-id=$$BUILDKITE_PARALLEL_JOB"
label: ":amd: (MI355 DPX) Language Models (Extended Pooling) Shard %N"
dind: false dind: false
device: mi355_dpx device: mi300_1
timeout_in_minutes: 95 timeout_in_minutes: 95
depends_on: depends_on:
- image-build-amd - image-build-amd
- label: ":nvidia: (H200 MIG 35GB) Language Models (MTEB)" - label: ":nvidia: (H200 MIG 18GB) Language Models (MTEB)"
key: language-models-test-mteb key: language-models-test-mteb
timeout_in_minutes: 68 timeout_in_minutes: 68
device: h200_35gb device: h200_18gb
optional: true optional: true
source_file_dependencies: source_file_dependencies:
- vllm/ - vllm/
+6 -7
View File
@@ -24,7 +24,6 @@ steps:
- label: ":nvidia: (H200 MIG 18GB) Multimodal Models (Standard) 2: qwen3 + gemma" - label: ":nvidia: (H200 MIG 18GB) Multimodal Models (Standard) 2: qwen3 + gemma"
key: multi-modal-models-standard-2-qwen3-gemma key: multi-modal-models-standard-2-qwen3-gemma
automatic_shard: true
timeout_in_minutes: 75 timeout_in_minutes: 75
device: h200_18gb device: h200_18gb
source_file_dependencies: source_file_dependencies:
@@ -163,9 +162,9 @@ steps:
- pytest -v -s models/multimodal/test_mapping.py - pytest -v -s models/multimodal/test_mapping.py
mirror: mirror:
amd: amd:
label: ":amd: (MI355 DPX) Multimodal Models (Extended Generation 1)" label: ":amd: (MI300) Multimodal Models (Extended Generation 1)"
dind: false dind: false
device: mi355_dpx device: mi300_1
timeout_in_minutes: 100 timeout_in_minutes: 100
depends_on: depends_on:
- image-build-amd - image-build-amd
@@ -182,9 +181,9 @@ steps:
- pytest -v -s models/multimodal/generation_ppl_test/ - pytest -v -s models/multimodal/generation_ppl_test/
mirror: mirror:
amd: amd:
label: ":amd: (MI355 DPX) Multimodal Models (PPL)" label: ":amd: (MI300) Multimodal Models (PPL)"
dind: false dind: false
device: mi355_dpx device: mi300_1
timeout_in_minutes: 90 timeout_in_minutes: 90
depends_on: depends_on:
- image-build-amd - image-build-amd
@@ -240,9 +239,9 @@ steps:
- pytest -v -s models/multimodal/pooling -m 'not core_model' - pytest -v -s models/multimodal/pooling -m 'not core_model'
mirror: mirror:
amd: amd:
label: ":amd: (MI355 DPX) Multimodal Models (Extended Pooling)" label: ":amd: (MI300) Multimodal Models (Extended Pooling)"
dind: false dind: false
device: mi355_dpx device: mi300_1
timeout_in_minutes: 75 timeout_in_minutes: 75
depends_on: depends_on:
- image-build-amd - image-build-amd
-2
View File
@@ -123,7 +123,6 @@ steps:
- label: ":nvidia: (L4) PyTorch Compilation Passes" - label: ":nvidia: (L4) PyTorch Compilation Passes"
key: pytorch-compilation-passes-unit-tests key: pytorch-compilation-passes-unit-tests
automatic_shard: true
device: l4 device: l4
num_devices: 1 num_devices: 1
timeout_in_minutes: 45 timeout_in_minutes: 45
@@ -160,7 +159,6 @@ steps:
amd: amd:
label: ':amd: (MI300) PyTorch Compilation Passes' label: ':amd: (MI300) PyTorch Compilation Passes'
dind: false dind: false
automatic_shard: true
device: mi300_1 device: mi300_1
timeout_in_minutes: 80 timeout_in_minutes: 80
working_dir: /vllm-workspace/tests working_dir: /vllm-workspace/tests
+12 -10
View File
@@ -2,27 +2,26 @@ group: Quantization
depends_on: depends_on:
- image-build - image-build
steps: steps:
- label: ":nvidia: (H200 MIG 35GB) Quantization" - label: ":nvidia: (H200 MIG 35GB) Quantization Shard %N"
device: h200_35gb device: h200_35gb
key: quantization key: quantization
automatic_shard: true
timeout_in_minutes: 40 timeout_in_minutes: 40
parallelism: 4
env:
VLLM_USE_V2_MODEL_RUNNER: "0"
source_file_dependencies: source_file_dependencies:
- csrc/ - csrc/
- vllm/model_executor/layers/quantization - vllm/model_executor/layers/quantization
- vllm/models/deepseek_v41/quant_config.py
- vllm/utils/humming.py
- tests/quantization - tests/quantization
commands: commands:
# Pin torchao to the version compatible with the CI PyTorch and CUDA stack. # Pin torchao to the version compatible with the CI PyTorch and CUDA stack.
- uv pip install --system torchao==0.18.0 --index-url https://download.pytorch.org/whl/cu130 - uv pip install --system torchao==0.17.0 --index-url https://download.pytorch.org/whl/cu130
- uv pip install --system conch-triton-kernels - uv pip install --system conch-triton-kernels
# The SM90-only checkpoint currently contains a removed weight_chan_scale # The SM90-only checkpoint currently contains a removed weight_chan_scale
# parameter. It was not exercised by the previous L4 job. # parameter. It was not exercised by the previous L4 job.
- VLLM_TEST_FORCE_LOAD_FORMAT=auto pytest -v -s quantization/ --ignore quantization/test_blackwell_moe.py --ignore quantization/test_rocm_moe.py -k 'not test_compressed_tensors_w4a8_fp8' - VLLM_TEST_FORCE_LOAD_FORMAT=auto pytest -v -s quantization/ --ignore quantization/test_blackwell_moe.py --ignore quantization/test_rocm_moe.py -k 'not test_compressed_tensors_w4a8_fp8' --shard-id=$$BUILDKITE_PARALLEL_JOB --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT
mirror: mirror:
amd: amd:
parallelism: 4
label: ":amd: (MI300) Quantization Shard %N" label: ":amd: (MI300) Quantization Shard %N"
dind: false dind: false
device: mi300_1 device: mi300_1
@@ -41,6 +40,8 @@ steps:
- tests/rocm/test_moe_weight_replay.py - tests/rocm/test_moe_weight_replay.py
- tests/rocm/test_moe_padding.py - tests/rocm/test_moe_padding.py
commands: commands:
# Use the ROCm default instead of the NVIDIA runner override.
- unset VLLM_USE_V2_MODEL_RUNNER
- uv pip install --system torchao==0.17.0 - uv pip install --system torchao==0.17.0
- uv pip install --system conch-triton-kernels - uv pip install --system conch-triton-kernels
- VLLM_TEST_FORCE_LOAD_FORMAT=auto pytest -v -s quantization/ --ignore quantization/test_blackwell_moe.py --ignore quantization/test_rocm_moe.py --shard-id=$$BUILDKITE_PARALLEL_JOB --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT - VLLM_TEST_FORCE_LOAD_FORMAT=auto pytest -v -s quantization/ --ignore quantization/test_blackwell_moe.py --ignore quantization/test_rocm_moe.py --shard-id=$$BUILDKITE_PARALLEL_JOB --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT
@@ -78,7 +79,6 @@ steps:
- label: ":nvidia: (B200) Quantized MoE" - label: ":nvidia: (B200) Quantized MoE"
key: quantized-moe-test-b200 key: quantized-moe-test-b200
automatic_shard: true
timeout_in_minutes: 120 timeout_in_minutes: 120
working_dir: "/vllm-workspace/" working_dir: "/vllm-workspace/"
device: b200-k8s device: b200-k8s
@@ -126,8 +126,9 @@ steps:
- label: ":nvidia: (H200 MIG 35GB) Quantized Models" - label: ":nvidia: (H200 MIG 35GB) Quantized Models"
device: h200_35gb device: h200_35gb
key: quantized-models-test key: quantized-models-test
automatic_shard: true
timeout_in_minutes: 65 timeout_in_minutes: 65
env:
VLLM_USE_V2_MODEL_RUNNER: "0"
source_file_dependencies: source_file_dependencies:
- vllm/model_executor/layers/quantization - vllm/model_executor/layers/quantization
- tests/models/quantization - tests/models/quantization
@@ -137,7 +138,6 @@ steps:
amd: amd:
label: ":amd: (MI355 DPX) Quantized Models" label: ":amd: (MI355 DPX) Quantized Models"
dind: false dind: false
automatic_shard: true
device: mi355_dpx device: mi355_dpx
timeout_in_minutes: 80 timeout_in_minutes: 80
working_dir: "/vllm-workspace/tests" working_dir: "/vllm-workspace/tests"
@@ -150,4 +150,6 @@ steps:
- vllm/platforms/rocm.py - vllm/platforms/rocm.py
- vllm/model_executor/model_loader/ - vllm/model_executor/model_loader/
commands: commands:
# Use the ROCm default instead of the NVIDIA runner override.
- unset VLLM_USE_V2_MODEL_RUNNER
- pytest -v -s models/quantization - pytest -v -s models/quantization
-25
View File
@@ -1,25 +0,0 @@
group: RL
depends_on:
- image-build
steps:
- label: ":nvidia: (H200 MIG 35GB) RL Entrypoints"
device: h200_35gb
key: rl-entrypoints-tests
timeout_in_minutes: 40
working_dir: "/vllm-workspace/tests"
source_file_dependencies:
- vllm/
- "!vllm/distributed/kv_transfer/"
- tests/entrypoints/rl
commands:
- pytest -v -s entrypoints/rl/weight_transfer
- export VLLM_WORKER_MULTIPROC_METHOD=spawn
- pytest -v -s entrypoints/rl --ignore=entrypoints/rl/weight_transfer
mirror:
amd:
label: ":amd: (MI355 DPX) RL Entrypoints"
dind: false
device: mi355_dpx
timeout_in_minutes: 50
depends_on:
- image-build-amd
+2
View File
@@ -82,6 +82,7 @@ steps:
- tests/entrypoints/scale_out/token_in_token_out/test_serving_tokens.py - tests/entrypoints/scale_out/token_in_token_out/test_serving_tokens.py
- tests/entrypoints/serve/instrumentator/test_basic.py - tests/entrypoints/serve/instrumentator/test_basic.py
- tests/entrypoints/serve/instrumentator/test_metrics.py - tests/entrypoints/serve/instrumentator/test_metrics.py
# - tests/entrypoints/serve/dev/test_sleep.py
- tests/entrypoints/serve/tokenize/test_tokenization.py - tests/entrypoints/serve/tokenize/test_tokenization.py
commands: commands:
- export VLLM_USE_RUST_FRONTEND=1 - export VLLM_USE_RUST_FRONTEND=1
@@ -92,6 +93,7 @@ steps:
# test_generate_logprobs expects Python-style top_logprobs truncation (dedup sampled + cap at max(k, 1)). # test_generate_logprobs expects Python-style top_logprobs truncation (dedup sampled + cap at max(k, 1)).
- pytest -v -s entrypoints/scale_out/token_in_token_out/test_serving_tokens.py -k "not stream and not lora and not test_generate_logprobs and not stop_string_workflow" - pytest -v -s entrypoints/scale_out/token_in_token_out/test_serving_tokens.py -k "not stream and not lora and not test_generate_logprobs and not stop_string_workflow"
- pytest -v -s entrypoints/serve/instrumentator/test_metrics.py -k "text and not show and not run_batch and not test_metrics_counts and not test_metrics_exist" - pytest -v -s entrypoints/serve/instrumentator/test_metrics.py -k "text and not show and not run_batch and not test_metrics_counts and not test_metrics_exist"
# - pytest -v -s entrypoints/serve/dev/test_sleep.py
# /tokenizer_info is not implemented in the Rust frontend (the CLI flag is accepted as a no-op). # /tokenizer_info is not implemented in the Rust frontend (the CLI flag is accepted as a no-op).
- pytest -v -s entrypoints/serve/tokenize/test_tokenization.py -k "not tokenizer_info" - pytest -v -s entrypoints/serve/tokenize/test_tokenization.py -k "not tokenizer_info"
mirror: mirror:
@@ -14,6 +14,16 @@ steps:
- .buildkite/scripts/run-rust-frontend-cargo-ci.sh - .buildkite/scripts/run-rust-frontend-cargo-ci.sh
commands: commands:
- .buildkite/scripts/run-rust-frontend-cargo-ci.sh style-clippy - .buildkite/scripts/run-rust-frontend-cargo-ci.sh style-clippy
mirror:
amd:
label: ":amd: (MI250) Rust Frontend Cargo Style + Clippy"
dind: false
device: mi250_1
no_gpu: true
working_dir: "/vllm-workspace"
timeout_in_minutes: 35
depends_on:
- image-build-amd
- label: ":computer: (CPU) Rust Frontend Cargo" - label: ":computer: (CPU) Rust Frontend Cargo"
key: rust-frontend-cargo-tests key: rust-frontend-cargo-tests
@@ -28,3 +38,13 @@ steps:
- .buildkite/scripts/run-rust-frontend-cargo-ci.sh - .buildkite/scripts/run-rust-frontend-cargo-ci.sh
commands: commands:
- .buildkite/scripts/run-rust-frontend-cargo-ci.sh test - .buildkite/scripts/run-rust-frontend-cargo-ci.sh test
mirror:
amd:
label: ":amd: (MI250) Rust Frontend Cargo"
dind: false
device: mi250_1
no_gpu: true
working_dir: "/vllm-workspace"
timeout_in_minutes: 35
depends_on:
- image-build-amd
+15 -16
View File
@@ -5,7 +5,6 @@ steps:
- label: ":nvidia: (H200 MIG 35GB) V1 Spec Decode" - label: ":nvidia: (H200 MIG 35GB) V1 Spec Decode"
device: h200_35gb device: h200_35gb
key: v1-spec-decode key: v1-spec-decode
automatic_shard: true
timeout_in_minutes: 40 timeout_in_minutes: 40
source_file_dependencies: source_file_dependencies:
- vllm/config/ - vllm/config/
@@ -28,7 +27,6 @@ steps:
amd: amd:
label: ":amd: (MI300) V1 Spec Decode" label: ":amd: (MI300) V1 Spec Decode"
dind: false dind: false
automatic_shard: true
device: mi300_1 device: mi300_1
timeout_in_minutes: 50 timeout_in_minutes: 50
depends_on: depends_on:
@@ -137,9 +135,9 @@ steps:
- pytest -v -s v1/e2e/spec_decode/speculators/ - pytest -v -s v1/e2e/spec_decode/speculators/
mirror: mirror:
amd: amd:
label: ":amd: (MI355 DPX) Spec Decode Speculators" label: ":amd: (MI300) Spec Decode Speculators"
dind: false dind: false
device: mi355_dpx device: mi300_1
timeout_in_minutes: 40 timeout_in_minutes: 40
depends_on: depends_on:
- image-build-amd - image-build-amd
@@ -167,9 +165,9 @@ steps:
- pytest -v -s v1/e2e/spec_decode/mtp/gemma4/ - pytest -v -s v1/e2e/spec_decode/mtp/gemma4/
mirror: mirror:
amd: amd:
label: ":amd: (MI355 DPX) Spec Decode MTP Gemma4" label: ":amd: (MI300) Spec Decode MTP Gemma4"
dind: false dind: false
device: mi355_dpx device: mi300_1
timeout_in_minutes: 40 timeout_in_minutes: 40
depends_on: depends_on:
- image-build-amd - image-build-amd
@@ -198,9 +196,9 @@ steps:
- pytest -v -s v1/e2e/spec_decode/mtp/mimo/ - pytest -v -s v1/e2e/spec_decode/mtp/mimo/
mirror: mirror:
amd: amd:
label: ":amd: (MI355 DPX) Spec Decode MTP DeepSeek + MiMo" label: ":amd: (MI300) Spec Decode MTP DeepSeek + MiMo"
dind: false dind: false
device: mi355_dpx device: mi300_1
timeout_in_minutes: 40 timeout_in_minutes: 40
depends_on: depends_on:
- image-build-amd - image-build-amd
@@ -228,9 +226,9 @@ steps:
- pytest -v -s v1/e2e/spec_decode/mtp/qwen3_5/ - pytest -v -s v1/e2e/spec_decode/mtp/qwen3_5/
mirror: mirror:
amd: amd:
label: ":amd: (MI355 DPX) Spec Decode MTP Qwen3.5" label: ":amd: (MI300) Spec Decode MTP Qwen3.5"
dind: false dind: false
device: mi355_dpx device: mi300_1
timeout_in_minutes: 40 timeout_in_minutes: 40
depends_on: depends_on:
- image-build-amd - image-build-amd
@@ -387,9 +385,9 @@ steps:
- pytest -v -s v1/spec_decode/test_speculators_correctness.py -m slow_test - pytest -v -s v1/spec_decode/test_speculators_correctness.py -m slow_test
mirror: mirror:
amd: amd:
label: ":amd: (MI355 DPX) Speculators Correctness Nightly" label: ":amd: (MI300) Speculators Correctness Nightly"
dind: false dind: false
device: mi355_dpx device: mi300_1
timeout_in_minutes: 40 timeout_in_minutes: 40
depends_on: depends_on:
- image-build-amd - image-build-amd
@@ -452,9 +450,9 @@ steps:
- pytest -v -s v1/e2e/spec_decode/acceptance_rates/dflash/ -k "not dflash2" - pytest -v -s v1/e2e/spec_decode/acceptance_rates/dflash/ -k "not dflash2"
mirror: mirror:
amd: amd:
label: ":amd: (MI355 DPX) Spec Decode AL DFlash Nightly" label: ":amd: (MI300) Spec Decode AL DFlash Nightly"
dind: false dind: false
device: mi355_dpx device: mi300_1
timeout_in_minutes: 120 timeout_in_minutes: 120
commands: commands:
- pytest -v -s v1/e2e/spec_decode/acceptance_rates/dflash/ -k "not dflash2" - pytest -v -s v1/e2e/spec_decode/acceptance_rates/dflash/ -k "not dflash2"
@@ -531,9 +529,9 @@ steps:
- pytest -v -s v1/e2e/spec_decode/acceptance_rates/dspark/ - pytest -v -s v1/e2e/spec_decode/acceptance_rates/dspark/
mirror: mirror:
amd: amd:
label: ":amd: (MI355 DPX) Spec Decode AL DSpark Nightly" label: ":amd: (MI300) Spec Decode AL DSpark Nightly"
dind: false dind: false
device: mi355_dpx device: mi300_1
timeout_in_minutes: 90 timeout_in_minutes: 90
depends_on: depends_on:
- image-build-amd - image-build-amd
@@ -595,6 +593,7 @@ steps:
- vllm/model_executor/models/gemma4_mtp.py - vllm/model_executor/models/gemma4_mtp.py
- vllm/model_executor/models/llama.py - vllm/model_executor/models/llama.py
- vllm/model_executor/models/llama_eagle3.py - vllm/model_executor/models/llama_eagle3.py
- vllm/model_executor/models/medusa.py
- vllm/model_executor/models/registry.py - vllm/model_executor/models/registry.py
- tests/evals/gsm8k/ - tests/evals/gsm8k/
- tests/utils.py - tests/utils.py
-1
View File
@@ -1 +0,0 @@
../../.agents/skills/model-microbenchmark
-1
View File
@@ -1 +0,0 @@
../../.agents/skills/model-recipes
-1
View File
@@ -1 +0,0 @@
../../.agents/skills/transformers-upgrade
+26 -56
View File
@@ -3,14 +3,12 @@
# This lists cover the "core" components of vLLM that require careful review # This lists cover the "core" components of vLLM that require careful review
/vllm/compilation @zou3519 @youkaichao @ProExpertProg @BoyuanFeng /vllm/compilation @zou3519 @youkaichao @ProExpertProg @BoyuanFeng
/vllm/compilation/breakable_cudagraph.py @ZJY0516
/vllm/distributed/kv_transfer @NickLucche @ApostaC @orozery @xuechendi @ivanium @wzhao18 /vllm/distributed/kv_transfer @NickLucche @ApostaC @orozery @xuechendi @ivanium @wzhao18
/vllm/distributed/kv_transfer/kv_connector/v1/hisparse @MatthewBonanni @LucasWilkinson
/vllm/lora @jeejeelee /vllm/lora @jeejeelee
/vllm/model_executor/layers/attention @LucasWilkinson @MatthewBonanni /vllm/model_executor/layers/attention @LucasWilkinson @MatthewBonanni
/vllm/model_executor/layers/fused_moe @mgoin @pavanimajety @zyongye @wzhao18 @bnellnm /vllm/model_executor/layers/fused_moe @mgoin @pavanimajety @zyongye @wzhao18
/vllm/model_executor/layers/quantization @mgoin @robertgshaw2-redhat @tlrmchlsmth @yewentao256 @pavanimajety @zyongye /vllm/model_executor/layers/quantization @mgoin @robertgshaw2-redhat @tlrmchlsmth @yewentao256 @pavanimajety @zyongye
/vllm/model_executor/layers/mamba @tdoublep @tomeras91 @ZJY0516 /vllm/model_executor/layers/mamba @tdoublep @tomeras91
/vllm/model_executor/layers/mamba/gdn/qwen_gdn_linear_attn.py @tdoublep @ZJY0516 @vadiklyutiy /vllm/model_executor/layers/mamba/gdn/qwen_gdn_linear_attn.py @tdoublep @ZJY0516 @vadiklyutiy
/vllm/model_executor/model_loader @22quinn /vllm/model_executor/model_loader @22quinn
/vllm/model_executor/determinism/ @yewentao256 /vllm/model_executor/determinism/ @yewentao256
@@ -27,7 +25,7 @@
# Any change to the VllmConfig changes can have a large user-facing impact, # Any change to the VllmConfig changes can have a large user-facing impact,
# so spam a lot of people # so spam a lot of people
/vllm/config @WoosukKwon @youkaichao @robertgshaw2-redhat @mgoin @tlrmchlsmth @houseroad @yewentao256 @ProExpertProg /vllm/config @WoosukKwon @youkaichao @robertgshaw2-redhat @mgoin @tlrmchlsmth @houseroad @yewentao256 @ProExpertProg
/vllm/config/cache.py @heheda12345 @ivanium @ZJY0516 /vllm/config/cache.py @heheda12345 @ivanium
# Config utils # Config utils
/vllm/config/utils.py @hmellor /vllm/config/utils.py @hmellor
@@ -35,48 +33,23 @@
/vllm/utils/argparse_utils.py /vllm/utils/argparse_utils.py
# Entrypoints # Entrypoints
/vllm/entrypoints/anthropic @chaunceyjiang @mgoin @DarkLight1337 /vllm/entrypoints/anthropic @mgoin @DarkLight1337
/vllm/entrypoints/cohere @DarkLight1337
/vllm/entrypoints/cli @hmellor @mgoin @DarkLight1337 /vllm/entrypoints/cli @hmellor @mgoin @DarkLight1337
/vllm/entrypoints/mcp @heheda12345 /vllm/entrypoints/mcp @heheda12345
/vllm/entrypoints/openai/chat_completion @chaunceyjiang @DarkLight1337 /vllm/entrypoints/openai @aarnphm @chaunceyjiang @DarkLight1337
/vllm/entrypoints/openai/completion @chaunceyjiang @DarkLight1337 /vllm/entrypoints/speech_to_text/realtime @njhill
/vllm/entrypoints/openai/models @chaunceyjiang @DarkLight1337
/vllm/entrypoints/openai/parser @aarnphm @chaunceyjiang @sfeng33
/vllm/entrypoints/openai/responses @aarnphm @chaunceyjiang @sfeng33
/vllm/entrypoints/pooling @noooop @DarkLight1337
/vllm/entrypoints/rl @aoshen02 @njhill
/vllm/entrypoints/scale_out @aoshen02 @noooop @DarkLight1337
/vllm/entrypoints/speech_to_text @NickLucche /vllm/entrypoints/speech_to_text @NickLucche
/vllm/entrypoints/speech_to_text/realtime @njhill @NickLucche /vllm/entrypoints/pooling @noooop
/vllm/entrypoints/serve @njhill
/vllm/entrypoints/serve/sagemaker @DarkLight1337 /vllm/entrypoints/serve/sagemaker @DarkLight1337
/vllm/entrypoints/common @DarkLight1337 /vllm/entrypoints/serve @njhill
/vllm/entrypoints/*.py @njhill
/vllm/entrypoints/chat_utils.py @DarkLight1337
/vllm/entrypoints/offline_utils.py @DarkLight1337
/vllm/entrypoints/llm.py @DarkLight1337 /vllm/entrypoints/llm.py @DarkLight1337
/tests/entrypoints @robertgshaw2-redhat @aarnphm @NickLucche @AndreasKaratzas
/tests/entrypoints/anthropic @mgoin @DarkLight1337
/tests/entrypoints/cohere @DarkLight1337
/tests/entrypoints/cli @hmellor @mgoin @DarkLight1337
/tests/entrypoints/openai/chat_completion @chaunceyjiang @DarkLight1337
/tests/entrypoints/openai/completion @chaunceyjiang @DarkLight1337
/tests/entrypoints/openai/models @chaunceyjiang @DarkLight1337
/tests/entrypoints/openai/parser @aarnphm @chaunceyjiang @sfeng33
/tests/entrypoints/openai/responses @aarnphm @chaunceyjiang @sfeng33
/tests/entrypoints/pooling @noooop @DarkLight1337
/tests/entrypoints/rl @aoshen02 @njhill
/tests/entrypoints/scale_out @noooop @DarkLight1337
/tests/entrypoints/speech_to_text @NickLucche
/tests/entrypoints/speech_to_text/realtime @njhill @NickLucche
/tests/entrypoints/serve @njhill
/tests/entrypoints/serve/sagemaker @DarkLight1337
/tests/entrypoints/llm @DarkLight1337
/tests/entrypoints/llm/test_struct_output_generate.py @mgoin @aarnphm @arpera
# Rust Frontend # Rust Frontend
/rust/ @BugenZhao @njhill /rust/ @BugenZhao @njhill
/rust/src/bench @esmeetu /rust/src/bench @esmeetu
/rust/src/server/src/routes/inference @aoshen02 @BugenZhao @njhill
/tools/build_rust.sh @BugenZhao @njhill /tools/build_rust.sh @BugenZhao @njhill
/rust-toolchain.toml @BugenZhao @njhill /rust-toolchain.toml @BugenZhao @njhill
/.buildkite/test_areas/rust* @BugenZhao @njhill /.buildkite/test_areas/rust* @BugenZhao @njhill
@@ -90,12 +63,6 @@
/vllm/tool_parsers @aarnphm @chaunceyjiang @sfeng33 @bbrowning /vllm/tool_parsers @aarnphm @chaunceyjiang @sfeng33 @bbrowning
/vllm/parser @aarnphm @chaunceyjiang @sfeng33 @bbrowning /vllm/parser @aarnphm @chaunceyjiang @sfeng33 @bbrowning
/tests/tokenizers_ @DarkLight1337 @njhill
/tests/renderers @DarkLight1337 @njhill
/tests/reasoning @aarnphm @chaunceyjiang @sfeng33 @bbrowning
/tests/tool_parsers @aarnphm @chaunceyjiang @sfeng33 @bbrowning
/tests/tool_use @aarnphm @chaunceyjiang @sfeng33 @bbrowning
# vLLM V1 # vLLM V1
/vllm/v1/attention @LucasWilkinson @MatthewBonanni /vllm/v1/attention @LucasWilkinson @MatthewBonanni
/vllm/v1/attention/backend.py @WoosukKwon @zhuohan123 @youkaichao @alexm-redhat @njhill /vllm/v1/attention/backend.py @WoosukKwon @zhuohan123 @youkaichao @alexm-redhat @njhill
@@ -103,18 +70,16 @@
/vllm/v1/attention/backends/flashinfer.py @mgoin @pavanimajety @vadiklyutiy @wzhao18 /vllm/v1/attention/backends/flashinfer.py @mgoin @pavanimajety @vadiklyutiy @wzhao18
/vllm/v1/attention/backends/triton_attn.py @tdoublep /vllm/v1/attention/backends/triton_attn.py @tdoublep
/vllm/v1/attention/backends/gdn_attn.py @ZJY0516 @vadiklyutiy /vllm/v1/attention/backends/gdn_attn.py @ZJY0516 @vadiklyutiy
/vllm/v1/core @WoosukKwon @robertgshaw2-redhat @njhill @ywang96 @alexm-redhat @heheda12345 @ApostaC @orozery @ivanium @ZJY0516 /vllm/v1/core @WoosukKwon @robertgshaw2-redhat @njhill @ywang96 @alexm-redhat @heheda12345 @ApostaC @orozery @ivanium
/vllm/v1/sample @22quinn @houseroad @njhill /vllm/v1/sample @22quinn @houseroad @njhill
/vllm/v1/spec_decode @benchislett @luccafong @MatthewBonanni /vllm/v1/spec_decode @benchislett @luccafong @MatthewBonanni
/vllm/v1/structured_output @mgoin @aarnphm @benchislett @arpera /vllm/v1/structured_output @mgoin @aarnphm @benchislett @arpera
/vllm/v1/kv_cache_interface.py @heheda12345 @ivanium @ZJY0516 /vllm/v1/kv_cache_interface.py @heheda12345 @ivanium
/vllm/v1/kv_offload @ApostaC @orozery /vllm/v1/kv_offload @ApostaC @orozery
/vllm/v1/simple_kv_offload @ivanium /vllm/v1/simple_kv_offload @ivanium
/vllm/v1/hisparse @MatthewBonanni @LucasWilkinson
/vllm/v1/engine @njhill /vllm/v1/engine @njhill
/vllm/v1/executor @njhill /vllm/v1/executor @njhill
/vllm/v1/worker @njhill /vllm/v1/worker @njhill
/vllm/v1/worker/mamba_utils.py @njhill @ZJY0516
/vllm/v1/worker/kv_connector_model_runner_mixin.py @orozery @NickLucche @ivanium /vllm/v1/worker/kv_connector_model_runner_mixin.py @orozery @NickLucche @ivanium
/vllm/v1/worker/encoder_cudagraph.py @shen-shanshan /vllm/v1/worker/encoder_cudagraph.py @shen-shanshan
/vllm/v1/worker/encoder_cudagraph_defs.py @shen-shanshan /vllm/v1/worker/encoder_cudagraph_defs.py @shen-shanshan
@@ -122,7 +87,6 @@
# Model runner V2 # Model runner V2
/vllm/v1/worker/gpu @WoosukKwon @njhill @yewentao256 /vllm/v1/worker/gpu @WoosukKwon @njhill @yewentao256
/vllm/v1/worker/gpu/kv_connector.py @orozery /vllm/v1/worker/gpu/kv_connector.py @orozery
/vllm/v1/worker/gpu/model_states/mamba_hybrid.py @WoosukKwon @njhill @yewentao256 @ZJY0516
# CI & building # CI & building
/.buildkite @Harry-Chen @khluu /.buildkite @Harry-Chen @khluu
@@ -135,23 +99,27 @@
/tests/distributed/test_multi_node_assignment.py @youkaichao /tests/distributed/test_multi_node_assignment.py @youkaichao
/tests/distributed/test_pipeline_parallel.py @youkaichao /tests/distributed/test_pipeline_parallel.py @youkaichao
/tests/distributed/test_same_node.py @youkaichao /tests/distributed/test_same_node.py @youkaichao
/tests/evals @mgoin @vadiklyutiy @AndreasKaratzas @ZJY0516 /tests/entrypoints @DarkLight1337 @robertgshaw2-redhat @aarnphm @NickLucche @AndreasKaratzas
/tests/evals @mgoin @vadiklyutiy @AndreasKaratzas
/tests/kernels @mgoin @tlrmchlsmth @WoosukKwon @yewentao256 @zyongye @AndreasKaratzas /tests/kernels @mgoin @tlrmchlsmth @WoosukKwon @yewentao256 @zyongye @AndreasKaratzas
/tests/kernels/ir @ProExpertProg @tjtanaa /tests/kernels/ir @ProExpertProg @tjtanaa
/tests/kernels/moe @bnellnm
/tests/models @DarkLight1337 @ywang96 @AndreasKaratzas /tests/models @DarkLight1337 @ywang96 @AndreasKaratzas
/tests/multimodal @DarkLight1337 @ywang96 @NickLucche @shen-shanshan /tests/multimodal @DarkLight1337 @ywang96 @NickLucche @shen-shanshan
/tests/multimodal/media @Isotr0py /tests/multimodal/media @Isotr0py
/tests/quantization @mgoin @robertgshaw2-redhat @yewentao256 @pavanimajety @zyongye @AndreasKaratzas /tests/quantization @mgoin @robertgshaw2-redhat @yewentao256 @pavanimajety @zyongye @AndreasKaratzas
/tests/entrypoints/llm/test_struct_output_generate.py @mgoin @aarnphm @arpera
/tests/v1/structured_output @mgoin @aarnphm @arpera /tests/v1/structured_output @mgoin @aarnphm @arpera
/tests/v1/core @WoosukKwon @robertgshaw2-redhat @njhill @ywang96 @alexm-redhat @heheda12345 @ApostaC @orozery @ivanium @ZJY0516 /tests/v1/core @WoosukKwon @robertgshaw2-redhat @njhill @ywang96 @alexm-redhat @heheda12345 @ApostaC @orozery @ivanium
/tests/lora @jeejeelee /tests/lora @jeejeelee
/tests/models/language/generation/test_hybrid.py @tdoublep @tomeras91 @ZJY0516 /tests/models/language/generation/test_hybrid.py @tdoublep @tomeras91
/tests/v1/kv_connector/nixl_integration @NickLucche /tests/v1/kv_connector/nixl_integration @NickLucche
/tests/v1/kv_connector @ApostaC @orozery @ivanium @wzhao18 /tests/v1/kv_connector @ApostaC @orozery @ivanium @wzhao18
/tests/v1/kv_offload @ApostaC @orozery /tests/v1/kv_offload @ApostaC @orozery
/tests/v1/simple_kv_offload @ivanium /tests/v1/simple_kv_offload @ivanium
/tests/v1/determinism @yewentao256 /tests/v1/determinism @yewentao256
/tests/reasoning @aarnphm @chaunceyjiang @sfeng33 @bbrowning
/tests/tool_parsers @aarnphm @chaunceyjiang @sfeng33 @bbrowning
/tests/tool_use @aarnphm @chaunceyjiang @sfeng33 @bbrowning
# Transformers modeling backend # Transformers modeling backend
/vllm/model_executor/models/transformers @hmellor /vllm/model_executor/models/transformers @hmellor
@@ -165,9 +133,6 @@
.readthedocs.yaml @hmellor .readthedocs.yaml @hmellor
mkdocs.yaml @hmellor mkdocs.yaml @hmellor
# Model recipes
/recipes @esmeetu
# Linting # Linting
/tools/pre_commit/.markdownlint.yaml @hmellor /tools/pre_commit/.markdownlint.yaml @hmellor
.pre-commit-config.yaml @hmellor .pre-commit-config.yaml @hmellor
@@ -256,14 +221,19 @@ mkdocs.yaml @hmellor
/examples/pooling @noooop /examples/pooling @noooop
/docs/models/pooling_models @noooop /docs/models/pooling_models @noooop
/tests/models/*/pooling* @noooop /tests/models/*/pooling* @noooop
/tests/entrypoints/pooling @noooop
/vllm/config/pooler.py @noooop /vllm/config/pooler.py @noooop
/vllm/model_executor/layers/pooler @noooop /vllm/model_executor/layers/pooler @noooop
# Reinforcement learning # Reinforcement learning
/vllm/distributed/weight_transfer @aoshen02 /vllm/distributed/weight_transfer @aoshen02
/vllm/entrypoints/serve/dev/rlhf @njhill @aoshen02
/tests/distributed/test_weight_transfer.py @aoshen02 /tests/distributed/test_weight_transfer.py @aoshen02
/tests/entrypoints/weight_transfer @DarkLight1337 @robertgshaw2-redhat @aarnphm @NickLucche @AndreasKaratzas @aoshen02
/tests/entrypoints/serve/dev/rlhf @DarkLight1337 @robertgshaw2-redhat @aarnphm @NickLucche @AndreasKaratzas @aoshen02
/tests/v1/worker/test_gpu_worker_weight_transfer.py @aoshen02 /tests/v1/worker/test_gpu_worker_weight_transfer.py @aoshen02
/examples/rl @aoshen02 /examples/rl @aoshen02
/examples/features/pause_resume @aoshen02
/docs/training/rlhf.md @aoshen02 /docs/training/rlhf.md @aoshen02
/docs/training/async_rl.md @aoshen02 /docs/training/async_rl.md @aoshen02
/docs/training/weight_transfer @aoshen02 /docs/training/weight_transfer @aoshen02
+6 -1
View File
@@ -494,7 +494,8 @@ pull_request_rules:
- files~=^tests/v1/e2e/spec_decode/ - files~=^tests/v1/e2e/spec_decode/
- files~=^examples/features/speculative_decoding/ - files~=^examples/features/speculative_decoding/
- files~=^vllm/model_executor/models/.*eagle.*\.py - files~=^vllm/model_executor/models/.*eagle.*\.py
- files~=^vllm/transformers_utils/configs/eagle\.py - files=vllm/model_executor/models/mlp_speculator.py
- files~=^vllm/transformers_utils/configs/(eagle|medusa|mlp_speculator)\.py
actions: actions:
label: label:
add: add:
@@ -548,6 +549,8 @@ pull_request_rules:
- label != stale - label != stale
- or: - or:
- files~=tpu.py - files~=tpu.py
- files~=_tpu
- files~=tpu_
- files~=/tpu/ - files~=/tpu/
actions: actions:
label: label:
@@ -561,6 +564,8 @@ pull_request_rules:
- label != stale - label != stale
- and: - and:
- -files~=tpu.py - -files~=tpu.py
- -files~=_tpu
- -files~=tpu_
- -files~=/tpu/ - -files~=/tpu/
actions: actions:
label: label:
+1
View File
@@ -279,6 +279,7 @@ jobs:
"speculative-decoding": { "speculative-decoding": {
keywords: [ keywords: [
{ term: "eagle", searchIn: "title" }, { term: "eagle", searchIn: "title" },
{ term: "medusa", searchIn: "title" },
], ],
substrings: [ substrings: [
{ term: "speculative decoding", searchIn: "title" }, { term: "speculative decoding", searchIn: "title" },
-28
View File
@@ -1,28 +0,0 @@
name: Recipes
# Validate recipe changes against the renderer that publishes them
# (recipes.vllm.ai). The check logic lives in vllm-project/recipes and tracks
# its main branch, so a PR is held to the same validation that runs when
# vllm-project/recipes picks up this directory after merge.
on:
pull_request:
paths:
- 'recipes/**'
- '.github/workflows/recipes.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
permissions:
contents: read
jobs:
validate:
name: Recipes
if: github.repository_owner == 'vllm-project'
uses: vllm-project/recipes/.github/workflows/validate-recipes.yml@main
with:
source: vllm
path: recipes
+16 -14
View File
@@ -18,8 +18,22 @@ jobs:
if: >- if: >-
github.repository_owner == 'vllm-project' && github.repository_owner == 'vllm-project' &&
github.event.issue.pull_request && github.event.issue.pull_request &&
(startsWith(github.event.comment.body, '/ci') || (github.event.comment.body == '/ci run' ||
startsWith(github.event.comment.body, '/amd-ci')) github.event.comment.body == '/ci run all' ||
github.event.comment.body == '/ci run nightly' ||
github.event.comment.body == '/ci run --allow-stale' ||
github.event.comment.body == '/ci run all --allow-stale' ||
github.event.comment.body == '/ci run nightly --allow-stale' ||
github.event.comment.body == '/ci retry' ||
github.event.comment.body == '/ci cancel' ||
github.event.comment.body == '/amd-ci run' ||
github.event.comment.body == '/amd-ci run all' ||
github.event.comment.body == '/amd-ci run nightly' ||
github.event.comment.body == '/amd-ci run --allow-stale' ||
github.event.comment.body == '/amd-ci run all --allow-stale' ||
github.event.comment.body == '/amd-ci run nightly --allow-stale' ||
github.event.comment.body == '/amd-ci retry' ||
github.event.comment.body == '/amd-ci cancel')
runs-on: [self-hosted, linux, x64, vllm-runners] runs-on: [self-hosted, linux, x64, vllm-runners]
steps: steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
@@ -39,16 +53,4 @@ jobs:
BUILDKITE_ORGANIZATION: vllm BUILDKITE_ORGANIZATION: vllm
BUILDKITE_PIPELINE: ci BUILDKITE_PIPELINE: ci
CI_TRUSTED_USERS: ${{ vars.CI_TRUSTED_USERS }} CI_TRUSTED_USERS: ${{ vars.CI_TRUSTED_USERS }}
GH_BRANCH_UPDATE_TOKEN: ${{ secrets.CI_BRANCH_UPDATE_TOKEN }}
GH_TOKEN: ${{ github.token }} GH_TOKEN: ${{ github.token }}
- name: Report failure on the PR
if: failure()
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
with:
script: |
const runUrl = `${context.serverUrl}/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId}`;
await github.rest.issues.createComment({
...context.repo,
issue_number: context.issue.number,
body: `❌ @${context.payload.comment.user.login}, your CI command failed with an unexpected error, so CI may not have started. See the [workflow run](${runUrl}) for details.`,
});
+8 -138
View File
@@ -58,12 +58,9 @@ AMD_CI_COMMANDS = frozenset(
RETRY_COMMANDS = frozenset({COMMAND_RETRY_FAILED, COMMAND_RETRY_AMD_FAILED}) RETRY_COMMANDS = frozenset({COMMAND_RETRY_FAILED, COMMAND_RETRY_AMD_FAILED})
CANCEL_COMMANDS = frozenset({COMMAND_CANCEL_CI, COMMAND_CANCEL_AMD_CI}) CANCEL_COMMANDS = frozenset({COMMAND_CANCEL_CI, COMMAND_CANCEL_AMD_CI})
ALL_CI_COMMANDS = UPSTREAM_CI_COMMANDS | AMD_CI_COMMANDS ALL_CI_COMMANDS = UPSTREAM_CI_COMMANDS | AMD_CI_COMMANDS
CI_COMMAND_PREFIXES = frozenset({"/ci", "/amd-ci"})
CI_AUTHORIZED_COMMENT_MARKER = "<!-- vllm-ci-authorized -->" CI_AUTHORIZED_COMMENT_MARKER = "<!-- vllm-ci-authorized -->"
READY_LABELS = {"ready", "ready-run-all-tests"} READY_LABELS = {"ready", "ready-run-all-tests"}
TRUSTED_PERMISSIONS = {"admin", "maintain", "write"} TRUSTED_PERMISSIONS = {"admin", "maintain", "write"}
BRANCH_UPDATE_POLL_SECONDS = 3
BRANCH_UPDATE_POLL_ATTEMPTS = 40
ACTIVE_BUILD_STATES = { ACTIVE_BUILD_STATES = {
"blocked", "blocked",
"creating", "creating",
@@ -261,13 +258,6 @@ class GitHubClient:
raise ApiError(None, "GitHub returned an invalid commits-behind count.") raise ApiError(None, "GitHub returned an invalid commits-behind count.")
return behind return behind
def update_branch(self, number: int, expected_head_sha: str) -> None:
self._request(
self._repo_path(f"/pulls/{number}/update-branch"),
body={"expected_head_sha": expected_head_sha},
method="PUT",
)
def list_pulls_for_commit(self, commit: str) -> list[dict[str, Any]]: def list_pulls_for_commit(self, commit: str) -> list[dict[str, Any]]:
commit = urllib.parse.quote(commit, safe="") commit = urllib.parse.quote(commit, safe="")
response = self._request(self._repo_path(f"/commits/{commit}/pulls")) response = self._request(self._repo_path(f"/commits/{commit}/pulls"))
@@ -471,17 +461,11 @@ class BuildkiteClient:
def parse_command(body: str) -> str | None: def parse_command(body: str) -> str | None:
body = body.strip()
if body in ALL_CI_COMMANDS: if body in ALL_CI_COMMANDS:
return body return body
return None return None
def is_ci_command_attempt(body: str) -> bool:
words = body.split(maxsplit=1)
return bool(words) and words[0].casefold() in CI_COMMAND_PREFIXES
def pipeline_for_command( def pipeline_for_command(
command: str, command: str,
*, *,
@@ -621,16 +605,6 @@ def blocks_new_run(command: str, build: Mapping[str, Any]) -> bool:
return is_active_build(build) return is_active_build(build)
def list_commit_builds(
buildkite: BuildkiteClient, command: str, commit: str
) -> list[dict[str, Any]]:
builds = buildkite.list_builds(commit)
if command not in AMD_CI_COMMANDS:
return builds
# bootstrap-amd.sh exits before uploading any test steps for PR webhook builds.
return [build for build in builds if build.get("source") != "webhook"]
def select_latest_build( def select_latest_build(
builds: Sequence[dict[str, Any]], builds: Sequence[dict[str, Any]],
pr_number: int, pr_number: int,
@@ -797,11 +771,10 @@ def notify_authorized(
( (
f"✅ @{author}, CI is now available for this PR.\n\n" f"✅ @{author}, CI is now available for this PR.\n\n"
"- `/ci run` starts upstream CI; `/amd-ci run` starts AMD CI only.\n" "- `/ci run` starts upstream CI; `/amd-ci run` starts AMD CI only.\n"
"- Run commands test your branch with every commit currently on its " "- Your branch must contain every commit currently on its upstream "
"upstream target branch. If it is behind, the bot merges the target " "target branch. Merge or rebase onto the latest target branch, then "
"branch into it first (pull before pushing again). Append " "rerun the command. Append `--allow-stale` to a run command to test "
"`--allow-stale` to a run command to test an outdated branch as is, " "an outdated branch at your own risk.\n"
"at your own risk.\n"
"- `/ci retry` retries failed jobs in the CI build for the current " "- `/ci retry` retries failed jobs in the CI build for the current "
"PR head. If the current head has no CI build, it starts a new CI " "PR head. If the current head has no CI build, it starts a new CI "
"build for the current head containing only jobs that failed in " "build for the current head containing only jobs that failed in "
@@ -849,65 +822,10 @@ def resolve_workflow_run_pr(
return find_matching_pr(github.list_pulls_for_commit(head_sha)) return find_matching_pr(github.list_pulls_for_commit(head_sha))
def merge_base_into_pr(
github: GitHubClient,
branch_updater: GitHubClient,
pr: Mapping[str, Any],
command: str,
lag: str,
) -> tuple[dict[str, Any], str]:
base_ref = pr["base"]["ref"]
def merge_failed(reason: str) -> CiPreparationError:
return CiPreparationError(
f"{lag} The bot could not merge upstream `{base_ref}` into this "
f"branch ({reason}). No new CI build was started. Either merge or "
f"rebase onto the latest `{base_ref}` yourself and rerun `{command}`, "
f"or use `{command}{ALLOW_STALE_SUFFIX}` to test this branch as is."
)
try:
branch_updater.update_branch(pr["number"], pr["head"]["sha"])
for _ in range(BRANCH_UPDATE_POLL_ATTEMPTS):
time.sleep(BRANCH_UPDATE_POLL_SECONDS)
current_pr = github.get_pr(pr["number"])
if current_pr["head"]["sha"] == pr["head"]["sha"]:
continue
try:
behind = github.get_commits_behind_base(
base_ref, current_pr["head"]["sha"]
)
except ApiError as error:
# A fork's new head is briefly unknown to the base repository.
if error.status == 404:
continue
raise
break
else:
raise merge_failed("the merge commit did not appear in time")
if (
current_pr["state"] != "open"
or current_pr["base"]["ref"] != base_ref
or behind
):
raise merge_failed("the PR changed during the merge")
except ApiError as error:
raise merge_failed(str(error)) from error
note = (
f"ℹ️ {lag} Merged upstream `{base_ref}` into this branch as "
f"`{current_pr['head']['sha'][:12]}` before starting CI. "
"Pull the branch before pushing more changes."
)
print(note)
return current_pr, note
def prepare_pr_for_ci( def prepare_pr_for_ci(
github: GitHubClient, github: GitHubClient,
pr: Mapping[str, Any], pr: Mapping[str, Any],
command: str, command: str,
branch_updater: GitHubClient | None = None,
) -> tuple[dict[str, Any], str]: ) -> tuple[dict[str, Any], str]:
base_ref = pr["base"]["ref"] base_ref = pr["base"]["ref"]
no_action = "No new CI build was started." no_action = "No new CI build was started."
@@ -943,8 +861,6 @@ def prepare_pr_for_ci(
print(warning) print(warning)
return current_pr, warning return current_pr, warning
if behind and branch_updater is not None:
return merge_base_into_pr(github, branch_updater, current_pr, command, lag)
if behind: if behind:
raise CiPreparationError( raise CiPreparationError(
f"{lag} Your branch must contain every commit currently on upstream " f"{lag} Your branch must contain every commit currently on upstream "
@@ -964,8 +880,6 @@ def handle_run_ci(
command: str, command: str,
github: GitHubClient, github: GitHubClient,
pr: Mapping[str, Any], pr: Mapping[str, Any],
branch_updater: GitHubClient | None = None,
allow_repeat_run: bool = False,
) -> str: ) -> str:
ci_name = ci_name_for_command(command) ci_name = ci_name_for_command(command)
duplicate_builds = buildkite.list_builds( duplicate_builds = buildkite.list_builds(
@@ -978,7 +892,7 @@ def handle_run_ci(
f"{ci_name} was already requested by this comment: {duplicate['web_url']}" f"{ci_name} was already requested by this comment: {duplicate['web_url']}"
) )
current_builds = list_commit_builds(buildkite, command, pr["head"]["sha"]) current_builds = buildkite.list_builds(pr["head"]["sha"])
active_build = next( active_build = next(
( (
build build
@@ -992,19 +906,7 @@ def handle_run_ci(
f"{ci_name} is already running for this commit: {active_build['web_url']}" f"{ci_name} is already running for this commit: {active_build['web_url']}"
) )
tested_build = select_latest_build( current_pr, warning = prepare_pr_for_ci(github, pr, command)
[build for build in current_builds if build.get("state") != "canceled"],
pr["number"],
)
if tested_build and not allow_repeat_run:
raise CiPreparationError(
f"{ci_name} already ran on this commit: {tested_build['web_url']}. "
"No new CI build was started. Push new changes to run CI again, use "
f"`{retry_command_for_command(command)}` to rerun failed jobs, or ask "
"a reviewer with write access to rerun it."
)
current_pr, warning = prepare_pr_for_ci(github, pr, command, branch_updater)
build = buildkite.create_build( build = buildkite.create_build(
create_build_payload( create_build_payload(
@@ -1033,7 +935,7 @@ def handle_retry_failed(
ci_name = ci_name_for_command(command) ci_name = ci_name_for_command(command)
run_command = run_command_for_command(command) run_command = run_command_for_command(command)
retry_command = retry_command_for_command(command) retry_command = retry_command_for_command(command)
builds = list_commit_builds(buildkite, command, pr["head"]["sha"]) builds = buildkite.list_builds(pr["head"]["sha"])
build = select_latest_build(builds, pr["number"]) build = select_latest_build(builds, pr["number"])
if build: if build:
metadata = build.get("meta_data") or {} metadata = build.get("meta_data") or {}
@@ -1184,34 +1086,11 @@ def handle_cancel_ci(
) )
def reply_unrecognized_command(
event: Mapping[str, Any],
github: GitHubClient,
) -> None:
if "pull_request" not in event["issue"]:
return
issue_number = event["issue"]["number"]
comment_id = event["comment"]["id"]
actor = event["comment"]["user"]["login"]
if is_already_handled(github, issue_number, comment_id):
print(f"Comment {comment_id} was already handled.")
return
add_reaction_safely(github, comment_id, "-1")
commands = ", ".join(f"`{command}`" for command in sorted(ALL_CI_COMMANDS))
github.add_comment(
issue_number,
f"❌ @{actor}, that is not a recognized CI command, so no CI was "
"started. Post a new comment containing only one of: "
f"{commands}.\n\n{command_comment_marker(comment_id)}",
)
def run( def run(
event: Mapping[str, Any], event: Mapping[str, Any],
github: GitHubClient, github: GitHubClient,
buildkite: BuildkiteClient, buildkite: BuildkiteClient,
trusted_users_value: str = "", trusted_users_value: str = "",
branch_updater: GitHubClient | None = None,
) -> None: ) -> None:
command = parse_command(event["comment"]["body"]) command = parse_command(event["comment"]["body"])
if not command or "pull_request" not in event["issue"]: if not command or "pull_request" not in event["issue"]:
@@ -1274,8 +1153,6 @@ def run(
command=command, command=command,
github=github, github=github,
pr=pr, pr=pr,
branch_updater=branch_updater,
allow_repeat_run=is_trusted_permission(permission),
) )
elif command in RETRY_COMMANDS: elif command in RETRY_COMMANDS:
message = handle_retry_failed( message = handle_retry_failed(
@@ -1340,11 +1217,8 @@ def main() -> None:
) )
return return
body = event["comment"]["body"] command = parse_command(event["comment"]["body"])
command = parse_command(body)
if not command: if not command:
if is_ci_command_attempt(body):
reply_unrecognized_command(event, github)
return return
pipeline = pipeline_for_command( pipeline = pipeline_for_command(
@@ -1357,15 +1231,11 @@ def main() -> None:
os.environ.get("BUILDKITE_ORGANIZATION", "vllm"), os.environ.get("BUILDKITE_ORGANIZATION", "vllm"),
pipeline, pipeline,
) )
branch_update_token = os.environ.get("GH_BRANCH_UPDATE_TOKEN", "")
run( run(
event, event,
github, github,
buildkite, buildkite,
os.environ.get("CI_TRUSTED_USERS", ""), os.environ.get("CI_TRUSTED_USERS", ""),
GitHubClient(branch_update_token, os.environ["GITHUB_REPOSITORY"])
if branch_update_token
else None,
) )
@@ -32,12 +32,10 @@ from run_ci_command import (
has_trusted_approval, has_trusted_approval,
is_active_build, is_active_build,
is_build_for_pr, is_build_for_pr,
is_ci_command_attempt,
notify_authorized, notify_authorized,
parse_command, parse_command,
parse_trusted_users, parse_trusted_users,
pipeline_for_command, pipeline_for_command,
reply_unrecognized_command,
resolve_workflow_run_pr, resolve_workflow_run_pr,
run, run,
select_latest_build, select_latest_build,
@@ -144,23 +142,6 @@ class FakeGitHub:
self.comments.append(body) self.comments.append(body)
class FakeBranchUpdater:
def __init__(self, github: FakeGitHub, error: ApiError | None = None) -> None:
self.calls: list[tuple[int, str]] = []
self.error = error
self.github = github
def update_branch(self, number: int, expected_head_sha: str) -> None:
self.calls.append((number, expected_head_sha))
if self.error:
raise self.error
self.github.prs[number] = {
**self.github.prs[number],
"head": {**self.github.prs[number]["head"], "sha": "merge-commit-sha"},
}
self.github.behind = 0
class FakeBuildkite: class FakeBuildkite:
def __init__( def __init__(
self, self,
@@ -359,46 +340,14 @@ class RunCiCommandTest(unittest.TestCase):
self.assertIsNone(parse_command("/ci run please")) self.assertIsNone(parse_command("/ci run please"))
self.assertIsNone(parse_command("/ci run all please")) self.assertIsNone(parse_command("/ci run all please"))
self.assertIsNone(parse_command("/ci cancel please")) self.assertIsNone(parse_command("/ci cancel please"))
self.assertIsNone(parse_command(" /ci run"))
self.assertIsNone(parse_command("/amd-ci run please")) self.assertIsNone(parse_command("/amd-ci run please"))
self.assertIsNone(parse_command("/amd-ci retry "))
self.assertIsNone(parse_command("/AMD-CI run")) self.assertIsNone(parse_command("/AMD-CI run"))
self.assertIsNone(parse_command("/amdci run")) self.assertIsNone(parse_command("/amdci run"))
self.assertIsNone(parse_command("/ci run --allow-stale all")) self.assertIsNone(parse_command("/ci run --allow-stale all"))
self.assertIsNone(parse_command("/amd-ci run --allow-stale please")) self.assertIsNone(parse_command("/amd-ci run --allow-stale please"))
def test_surrounding_whitespace_is_ignored(self) -> None:
self.assertEqual(parse_command("/ci run\r\n"), COMMAND_RUN_CI)
self.assertEqual(parse_command(" /ci run "), COMMAND_RUN_CI)
self.assertEqual(parse_command("/amd-ci retry\n"), COMMAND_RETRY_AMD_FAILED)
def test_ci_command_attempts_are_detected(self) -> None:
for body in ("/ci", "/ci rn", "/CI run", "/amd-ci run please", " /ci\nrun"):
with self.subTest(body=body):
self.assertTrue(is_ci_command_attempt(body))
for body in ("", "LGTM", "/cifoo", "/amdci run", "please /ci run"):
with self.subTest(body=body):
self.assertFalse(is_ci_command_attempt(body))
def test_unrecognized_command_gets_feedback(self) -> None:
github = FakeGitHub()
reply_unrecognized_command(make_event("/ci rn", actor="author"), github)
self.assertEqual(github.reactions, ["-1"])
self.assertEqual(len(github.comments), 1)
self.assertIn(
"@author, that is not a recognized CI command", github.comments[0]
)
self.assertIn(f"`{COMMAND_RUN_CI}`", github.comments[0])
self.assertIn("<!-- vllm-ci-command:99 -->", github.comments[0])
def test_unrecognized_command_feedback_is_not_repeated(self) -> None:
github = FakeGitHub(comments=["❌ earlier\n\n<!-- vllm-ci-command:99 -->"])
reply_unrecognized_command(make_event("/ci rn"), github)
self.assertEqual(len(github.comments), 1)
self.assertEqual(github.reactions, [])
def test_commands_select_only_their_configured_pipeline(self) -> None: def test_commands_select_only_their_configured_pipeline(self) -> None:
cases = ( cases = (
(COMMAND_RUN_CI, "ci"), (COMMAND_RUN_CI, "ci"),
@@ -633,82 +582,6 @@ class RunCiCommandTest(unittest.TestCase):
) )
self.assertIn(command, github.comments[0]) self.assertIn(command, github.comments[0])
def test_run_merges_base_into_outdated_branch_before_building(self) -> None:
github = FakeGitHub(behind=5)
updater = FakeBranchUpdater(github)
buildkite = FakeBuildkite([[], []])
with patch("run_ci_command.time.sleep"):
run(make_event(COMMAND_RUN_CI), github, buildkite, "", updater)
self.assertEqual(updater.calls, [(42, "0123456789abcdef")])
self.assertEqual(buildkite.created_builds[0]["commit"], "merge-commit-sha")
self.assertTrue(github.comments[0].startswith("✅ "))
self.assertIn("5 commits behind", github.comments[0])
self.assertIn("Merged upstream `main`", github.comments[0])
def test_only_committers_can_rerun_a_commit_that_already_ran_ci(self) -> None:
finished_build = {
"created_at": "2026-07-28T01:00:00Z",
"pull_request": {"id": 42},
"state": "failed",
"web_url": "https://buildkite.example/builds/122",
}
for actor, permission, command, allowed in (
("author", "read", COMMAND_RUN_CI, False),
("author", "read", f"{COMMAND_RUN_CI} --allow-stale", False),
("reviewer", "write", COMMAND_RUN_CI, True),
):
with self.subTest(actor=actor, command=command):
github = FakeGitHub(
behind=5,
permission=permission,
pr=make_pr(labels=[{"name": "ready"}]),
)
updater = FakeBranchUpdater(github)
buildkite = FakeBuildkite([[], [finished_build]])
with patch("run_ci_command.time.sleep"):
run(make_event(command, actor), github, buildkite, "", updater)
self.assertEqual(bool(buildkite.created_builds), allowed)
if not allowed:
self.assertEqual(updater.calls, [])
self.assertIn("CI already ran on this commit", github.comments[0])
self.assertIn("`/ci retry`", github.comments[0])
def test_run_waits_for_fork_merge_commit_to_reach_base_repo(self) -> None:
github = FakeGitHub(behind=5)
updater = FakeBranchUpdater(github)
buildkite = FakeBuildkite([[], []])
with (
patch("run_ci_command.time.sleep"),
patch.object(
github,
"get_commits_behind_base",
side_effect=[5, ApiError(404, "Not Found"), 0],
),
):
run(make_event(COMMAND_RUN_CI), github, buildkite, "", updater)
self.assertEqual(buildkite.created_builds[0]["commit"], "merge-commit-sha")
self.assertTrue(github.comments[0].startswith("✅ "))
def test_run_reports_failed_merge_without_building(self) -> None:
github = FakeGitHub(behind=5)
updater = FakeBranchUpdater(github, error=ApiError(422, "merge conflict"))
buildkite = FakeBuildkite([[], []])
run(make_event(COMMAND_RUN_CI), github, buildkite, "", updater)
self.assertEqual(buildkite.created_builds, [])
self.assertTrue(github.comments[0].startswith("❌ "))
self.assertIn("could not merge upstream `main`", github.comments[0])
self.assertIn("merge conflict", github.comments[0])
self.assertIn("No new CI build was started.", github.comments[0])
self.assertIn("`/ci run --allow-stale`", github.comments[0])
def test_allow_stale_preserves_author_permissions_and_run_configuration( def test_allow_stale_preserves_author_permissions_and_run_configuration(
self, self,
) -> None: ) -> None:
@@ -883,42 +756,6 @@ class RunCiCommandTest(unittest.TestCase):
self.assertEqual(buildkite.created_builds, []) self.assertEqual(buildkite.created_builds, [])
self.assertIn("AMD CI is already running", github.comments[0]) self.assertIn("AMD CI is already running", github.comments[0])
def test_amd_run_ignores_finished_webhook_build_for_author(self) -> None:
webhook_build = {
"created_at": "2026-08-18T01:00:00Z",
"meta_data": {},
"number": 122,
"pull_request": {"id": 42},
"source": "webhook",
"state": "passed",
"web_url": "https://buildkite.example/amd-ci/builds/122",
}
ui_build = {**webhook_build, "source": "ui"}
command_build = {
**webhook_build,
"meta_data": {"github-comment-id": "98"},
"source": "api",
}
for build, allowed in (
(webhook_build, True),
(ui_build, False),
(command_build, False),
):
with self.subTest(source=build["source"]):
github = FakeGitHub(
permission="read",
pr=make_pr(labels=[{"name": "ready"}]),
)
buildkite = FakeBuildkite([[], [build]])
run(make_event(COMMAND_RUN_AMD_CI, "author"), github, buildkite)
self.assertEqual(bool(buildkite.created_builds), allowed)
if not allowed:
self.assertIn(
"AMD CI already ran on this commit", github.comments[0]
)
def test_run_all_sets_buildkite_environment(self) -> None: def test_run_all_sets_buildkite_environment(self) -> None:
github = FakeGitHub() github = FakeGitHub()
buildkite = FakeBuildkite([[], []]) buildkite = FakeBuildkite([[], []])
@@ -1265,29 +1102,6 @@ class RunCiCommandTest(unittest.TestCase):
) )
self.assertIn("Use `/amd-ci run`", github.comments[0]) self.assertIn("Use `/amd-ci run`", github.comments[0])
def test_amd_ci_retry_ignores_webhook_build_for_the_current_head(self) -> None:
github = FakeGitHub(
permission="read",
pr=make_pr(labels=[{"name": "ready"}]),
)
webhook_build = {
"created_at": "2026-08-18T01:00:00Z",
"meta_data": {},
"number": 122,
"pull_request": {"id": 42},
"source": "webhook",
"state": "passed",
"web_url": "https://buildkite.example/amd-ci/builds/122",
}
buildkite = FakeBuildkite([[webhook_build]])
run(make_event(COMMAND_RETRY_AMD_FAILED, "author"), github, buildkite)
self.assertEqual(buildkite.retry_calls, [])
self.assertIn(
"No AMD CI build exists for the current PR head", github.comments[0]
)
def test_ci_retry_creates_filtered_build_for_new_head(self) -> None: def test_ci_retry_creates_filtered_build_for_new_head(self) -> None:
github = FakeGitHub( github = FakeGitHub(
behind=100, behind=100,
+3
View File
@@ -237,6 +237,9 @@ benchmarks/**/*.json
actionlint actionlint
shellcheck*/ shellcheck*/
# Ignore moe/marlin_moe gen code
csrc/libtorch_stable/moe/marlin_moe_wna16/kernel_*
# Ignore ep_kernels_workspace folder # Ignore ep_kernels_workspace folder
ep_kernels_workspace/ ep_kernels_workspace/

Some files were not shown because too many files have changed in this diff Show More