Compare commits

..
79 Commits
Author SHA1 Message Date
MOVZXandHinata 31bee2c2fd [Qwen4Exp] Repair: import Qwen4ExpTextConfig from transformers
Upstream e3cae8d2ac (#57387) deleted vllm/transformers_utils/configs/
qwen4_exp.py and moved the config family into the model package, so
that module path no longer resolves. transformers 5.18 exports the
config itself, which is how the rest of this fork imports it
(nvidia/model.py, amd/mtp.py, and nvidia/qsa.py at the base).

vllm/models/qwen4_exp/common/mtp.py carried the dead path at the base,
which made the Qwen4Exp model package fail to import at all; the PLE
pull transport and the FP8-KV reader added two more users of it.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 21:36:51 +07:00
MOVZXandHinata bf3e4af6b1 [KV Cache] Hold a finished turn's prefix until its next turn is admitted
With a prefix unit finer than the block, a hybrid follow-up resumes from
a single GDN state at the previous prompt's tail, and that key sits in
the top attention block. Losing any one block of that prefix is a total
miss. The waiting-request pins only protect a hit that is still whole
when the follow-up is first looked up. Tracing the remaining misses
under a long-context multi-turn agent load shows they lost their top
attention block before that, in the gap between the previous turn
finishing and the next turn arriving. In that gap nothing referenced
the prefix. LRU then consumed it ahead of dead blocks freed later
(other conversations' handed-over CoW sources and superseded states).

A finished turn (stop, length, repetition, or an abort after decoding)
now keeps one extra ref on each of its hashed blocks, including its GDN
holds, as a grace pin. The ref is taken on both free paths, the
immediate free and the deferred pop_blocks_for_free behind an in-flight
step, so the prefix never passes through the free queue. A grace is
released when:

- the next turn is admitted on it. That is the grace holding the
  deepest hit block of every group, with a hit that reaches its prompt
  to within two prefix units, so a new turn sharing only a system
  prompt releases nothing. The table takes the blocks over, with no
  double ref.
- anyone needs blocks. Graces yield before waiter pins, oldest first and
  one per failed allocation, so they never cause a preemption.
- the bound (16) is exceeded. The oldest grace goes.
- reset_prefix_cache runs.

A head waiter whose admission fails pins its own hit first. It then
yields graces and other pins one at a time, and only when everything
yieldable covers its shortfall. Otherwise yielding would drop other
conversations' prefixes and still not admit it. The shortfall is
measured after the pin: the pin takes the proof block past the hit (and
any hit block the lookup did not return) out of the free queue, so the
shortfall of the failed attempt can be short by those blocks and the
guard would yield another conversation's grace for nothing. Its own
pinned blocks outside the hit it is admitted on, the proof block past
it, go back first: once the hit is taken over they protect nothing, and
they cost nobody else anything. At an exact fit they are the blocks it
needs, and nothing else ever yields a head waiter's own pin, so without
this a request that needs the whole pool waited forever.

While any pin or grace is held the scheduler reports no common prefix
to cascade attention. Those refs can make a block's count equal the
number of running requests when only some of them share it, and cascade
attention would then read it for all of them.

Graces and waiter pins now follow a producer's GDN hash into its CoW
copy, as holds already do. Otherwise they would keep a second ref on a
running state block that its owner must own exclusively. Graces apply
only to fine-grained hybrid hits (coordinator.enable_partial_hash_hits)
and are off with a KV connector; a block-sized unit (the 4096 lane) is
unchanged. When graces are active the scheduler logs one startup line
saying so, with the bound, the hash unit and the block size.

Tests: the arrival-gap reproduction fails on the previous commit (hit 0
instead of 168, both free paths). It passes here. Further tests cover
the handover refcounts, yield order and bound, finish statuses, a
running request at an exactly full pool, yieldable accounting,
moved-hash following, the pin's extras contract, and six runs on the
real scheduler: among them a head waiter whose pin raises its shortfall
(it gives its proof block back and the one grace that covers the rest
yields), a request that needs the whole pool (admitted on its own proof
block; it waited forever before), and the common-prefix count under a
grace (none reported; the count returns once the pins are gone).

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 20:18:18 +07:00
MOVZXandHinata f5d9021745 [Scheduler] Release a waiter's pin on the deferred free path too
The scheduler takes pop_blocks_for_free when a step may still be in
flight, for example when a preempted request is aborted before its
fence drained. A waiting request's pin is never written by any step, so
it is now released there at once. Before, it stayed until the next
yield or reset, keeping its blocks pinned and a waiter slot taken.

pin_waiting_hit touches each block in the pool it belongs to; private
pool groups route like free_blocks. Its docstring states the known gap:
a pin is a snapshot of the hit at pin time. It is not refreshed if the
hit grows while the request waits, or if a producer moves a pinned live
block's hash into its copy. The admission lookup still sees whatever is
cached, so the cost is a smaller hit, never a wrong one.
MAX_PINNED_WAITERS goes from 8 to 16.

Test: a pinned waiter popped through the deferred path leaves no pin,
no waiter slot and the pool at its baseline.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 20:16:37 +07:00
MOVZXandHinata 633c758d4c [Scheduler] Pin a waiting request's prefix-cache hit until it is admitted
A follow-up turn that waits for admission holds no reference on its
cached prefix. With a prefix unit finer than the block, its whole hit
depends on every one of those blocks: the resume key sits in the top
attention block, and the only GDN state is the held tail. So the first
of them that another request's allocation popped from the free queue
turned the next turn into a cold prefill; that was usually its own top
block, which is freed first. Under a long-context multi-turn agent load
(8 conversations, 80-100K-token prompts) this gave 4-5 total misses per
300 s at unit 64/512 and none at unit 4096. All of them happened while
the pool was at 88-99 % with one or two conversations waiting, and the
conversation that missed was usually the next victim too.

Now the manager can pin a waiter's hit:

- pin_waiting_hit looks the request up and takes a ref on every
  non-null hit block in every group. It also pins the block holding the
  EAGLE/MTP proof unit: one unit past the key, dropped from the returned
  hit, and a different block when the key ends a block. The pinned
  blocks stay out of the free queue.
- release_pinned_hit drops a pin; yield_pinned_hits drops the most
  recently pinned waiters' first.
- allocate_slots releases the request's pin when it succeeds (its table
  holds the hit from then on), free releases it (abort, finish), and
  reset_prefix_cache yields every pin before resetting.

Pinned blocks are referenced, so admission and the watermark see them
as used.

The scheduler pins the front of the waiting queue after each scheduling
pass: arrival order, at most MAX_PINNED_WAITERS = 8, only requests that
have not computed anything yet, never with a KV connector. It does this
also while the running set is full, when a waiter is not even looked
up. A pin never costs anyone blocks. A running request that fails to
allocate yields one waiter's pin and retries before any preemption
victim is chosen. A head-of-line waiter whose admission fails yields the
pins of the waiters behind it and retries once. When pins are active
the scheduler logs one startup line saying so, with the bound, the hash
unit and the block size.

On a closed-loop CPU model of that load (6-8 conversations, 1-2
waiting, pool 88-100 %), total misses went from 3/1/1/5 to 2/0/0/0 with
every waiter pinned. The remaining two sit at 99.2 % use, where the
running set needed the pinned blocks themselves (two yields, 65
blocks). Unit 4096 is unaffected except where a waiter was losing its
boundary block.

Tests:
- A pinned waiter survives another follow-up's admission and a burst,
  and resumes at its tail (a total miss without the pin).
- Pool full and a running request needs a block: the pin yields, nobody
  is preempted, and the waiter is still admitted later.
- Pins are released on abort, admission, yield and reset without
  leaks; the bound is respected; yield order is newest first.
- The scheduler's pass pins in arrival order up to the bound and skips
  partly computed requests and connector setups.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 20:16:37 +07:00
MOVZXandHinata 95ba99ed29 [Mamba] Key the prompt tail when it lands on a block end
With a hash unit finer than the block, the prompt's tail key lands
exactly on a block end once every block/unit turns. The tail key is the
position the next turn resumes at: the last unit boundary, one unit
lower under the eagle drop. It is then a full-block key, so the
partial-tail registration does nothing, and under retention interval 0
the mask kept only the replay boundary one block below it. That state
was never hashed. The next turn therefore resumed one block lower after
a cold turn, or one turn lower (at the held source) after a partial-tail
resume, re-prefilling the previous turn's extension.

MambaManager.cache_blocks now adds that block-aligned tail to the
reachable boundaries, so the block ending there is hashed like any
boundary state, retired into the held checkpoint and freed last. At the
block unit the tail is the replay boundary itself and nothing changes.
When the hash unit is below the Mamba block, MambaManager logs one
startup line saying the block-aligned tail key is hashed.

Tests: a cold and a partial-tail-resume case whose tail lands on a block
end, under a concurrent burst, with the no-leak check. Both fail on the
previous commit (the tail state is not keyed and the next turn resumes
lower). A block-unit case asserts the hashed set is exactly the replay
boundary.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 20:16:37 +07:00
MOVZXandHinata 944007d9e5 [Mamba] Hold the partial-tail CoW checkpoint until the request is freed
With --prefix-match-unit below the block size, a follow-up turn that
resumes at a partial tail (mid-block) never hashes a block-boundary GDN
state. Under retention interval 0 the only boundary kept is the replay
boundary, which lies below the resume point. The request's own tail is
published on its running block and moved into a CoW block by the next
chunk, and the scheduler releases both copy endpoints once the copy has
run. That block then sat at ref 0 in the free queue for the rest of the
request, ahead of its attention blocks, and concurrent prefills popped
it. The hybrid hit is the shortest group's, so the next turn resumed at
0. Under a long-context multi-turn agent load, 7 of 61 follow-ups at
unit 64 (6 at 256/512/1024) came back with cached_tokens 0 on 80-100K
prompts whose whole attention prefix was still cached. Every one
followed a partial-tail resume, none followed a block-aligned one, and
unit 4096 never missed.

The existing retention (VLLM_E2_MAMBA_RETAIN_CHECKPOINT) only held
hashed blocks retired from the table, so it never covered that copy.
Now, under the same switch:

- The running-request CoW branch holds the moved tail block in
  _retained_checkpoints with a ref of its own; the copy retention pair
  is unchanged. The new-request branch holds the hit source after
  _apply_cow for the same reason: a request that stops before its own
  tail still leaves the state it resumed from.
- _retained_checkpoints is keyed by block_hash_num_tokens instead of
  block index, so a sub-block tail key orders correctly against full
  block states; _retire_state_blocks uses the same key. The newest
  position supersedes and releases the older one, so a request still
  holds at most one block per group.
- When a producer moves its tail hashes into its CoW block, another
  request holding that source (a sibling that resumed at the tail while
  it was the producer's live block) follows the hashes to the copy.
- Admission reserves one extra block per Mamba group for a partial-tail
  hit: the held source and the moved tail coexist at the tail step.

The hold is released on every free path (finish, abort, preemption,
deferred pop_blocks_for_free) by the coordinator, last, after the
request's attention blocks, as before. When the hold is active (switch
on, hash unit below the Mamba block) MambaManager logs one startup line
saying so, with the hash unit and block size.

Tests (CPU): a replay of the miss that fails on v2.5.0 with the
retention switch on (next-turn hit 0 instead of the tail) and passes
here; the structural variant; admission accounting; no-leak checks for
a request freed with its copy still queued, in both release orders; and
the moved-hash follow. The retention suite's one block-index assertion
now reads the token position.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 20:16:36 +07:00
MOVZXandHinata db2851e54a [GDN] Keep the fused packed forward under RecoverSSM
RecoverSSM's verify needs the Triton decode kernel for its speculative rows, and the previous code met that by turning
off the fused CUDA decode for the whole layer. That also turned off the fused packed forward, which carries every
prefill chunk, so prefill ran the unfused path. The decode kernel stays Triton and the packed recurrent decode stays
off; the fused packed forward stays on. A small helper makes the choice, with CPU tests for each combination.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 20:16:36 +07:00
MOVZXandHinata 69aed12cf8 config, envs: pass use_replayssm to the breakable-graph fallback; register VLLM_FAIL_ON_PROFILING_KV_LEAK
CompilationConfig.resolve_cudagraph_mode_and_sizes read self.cache_config.use_replayssm, but CompilationConfig has
no cache_config, so with VLLM_USE_BREAKABLE_CUDAGRAPH=1 the fallback branch raised AttributeError. Both model
runners now pass cache_config.use_replayssm as an argument. The shipped launchers set VLLM_USE_BREAKABLE_CUDAGRAPH=0
and never reach this branch.

VLLM_FAIL_ON_PROFILING_KV_LEAK was read with a raw os.getenv, so setting it printed the unknown-variable warning. It
is now registered in vllm.envs with the same spellings (unset fails the boot; 0, false or no warn instead), listed
as an ignored compile factor and read through vllm.envs. Tests cover both.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 20:15:12 +07:00
MOVZXandHinata 86d314050f [Config] The MERLIN_* kernel toggles default to the shipped value when unset
MERLIN_FULL_K, MERLIN_QSA_SORT, MERLIN_TIE_DET, MERLIN_TIE_RECENT and MERLIN_TOPK_SORTED_EMIT defaulted to off in
code while every shipped launcher set them to 1, so a launch without them ran a different kernel configuration.
vllm/env_override.py now sets each to 1 when unset; 0 still turns one off. tests/test_merlin_toggle_defaults.py
covers it.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 20:15:11 +07:00
MOVZXandHinata b5f9fdbbac [PLE] VLLM_E2_PLE_PULL_TRANSPORT defaults to on when unset
With the switch unset, the PLE took upstream's UVA backend, whose Triton lookup kernel needs fp8e4nv, which sm_86
does not have, so a launcher without the switch failed at profile_run. Unset now means on, the shipped value;
0 still selects upstream's path.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 20:15:11 +07:00
MOVZXandHinata 72485816d9 [KV] Side-cache layout -- fold the PLE state into a GDN group, pack QSA rings
The packed pool strides every block by the widest group, and every group draws whole blocks. The PLE short-conv
state now shares a GDN cache group, and the QSA raw-key rings are packed as equal slots inside shared blocks, so
neither holds a whole block per request. VLLM_E2_SIDE_CACHE_LAYOUT (default 1) selects the old layout with 0.
On its own the layout raises the Flash-Next pool from 806,792 to 853,994 tokens at the same budget.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 20:15:11 +07:00
MOVZXandHinata 21fb7de7bf [Scheduler] Don't double-count the checkpoint of a prefix-hit admission
The admission gate reserved one retained-checkpoint block per Mamba group for every request not holding one. A
follow-up turn with a prefix hit already brings that block, so it was counted twice; the gate now counts it once.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 20:13:06 +07:00
MOVZXandHinata 010521f10d [Scheduler] Keep the running request's Mamba prefix checkpoint referenced until it finishes
In Mamba align mode a state block leaves the block table once the request has computed one block past it. When
that block is hashed it is the request's prefix checkpoint, and releasing it let a long concurrent session evict
it, so the next turn re-prefilled the whole conversation. The manager now keeps the newest hashed checkpoint
referenced until the request finishes, and admission reserves that block. VLLM_E2_MAMBA_RETAIN_CHECKPOINT
(default 1) switches it off. Scheduler tests cover eviction, admission, preemption and freeing.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 20:13:06 +07:00
MOVZXandHinata 0f72060d66 [Scheduler] Preserve per-group eviction order in deferred block free (port of upstream #48532)
Repair: the port driver logged this patch as resolved but produced no
commit; the -3 merge dropped the coordinator code hunk. Reapplied to HEAD.

Co-Authored-By: Hinata <hinata@localhost>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 20:13:04 +07:00
MOVZXandHinata 7c0a916c5d recoverssm: --use-replayssm without MTP falls back to the stock GDN path
On Qwen4Exp the flag selects RecoverSSM, which needs speculative decoding. With num_speculative_tokens=0 the
validator refused to start; it now logs a warning, clears use_replayssm (the no-flag path) and continues, so one
launch line serves with and without MTP. Other architectures keep the error. Config tests cover both.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 20:09:17 +07:00
MOVZXandHinata 8c518d4f09 [Worker] Gate the graph fallback and fail on profiling leaks
The breakable-CUDA-graph fallback in the compilation config now applies only with --use-replayssm, and a
profiling KV cache that survives the teardown fails the boot instead of only being logged
(VLLM_FAIL_ON_PROFILING_KV_LEAK=0 turns it back into a warning).

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 20:08:57 +07:00
MOVZXandHinata 4cfd09ebda [Worker] Report a profiling KV cache that survives the teardown
_teardown_profiling_state measures the storage behind the profiling KV caches on entry and the allocated memory
before and after each release, and logs any profiling KV cache that is still alive afterwards.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 20:08:57 +07:00
MOVZXandHinata 812d662602 recoverssm: allow pipeline parallelism for Qwen4Exp on the V2 runner
Upstream RecoverSSM commits a step in the same step's postprocess, hence PP=1. Under pipeline parallelism a
non-last rank learns a step's accepted counts only when the PP broadcast arrives pp_size steps later, while later
builds overwrite the persistent commit inputs. The commit is now deferred with its own copy of those inputs, and
the PP=1 check is lifted for Qwen4Exp on Model Runner V2. Config and CPU tests cover the deferral.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 20:08:56 +07:00
MOVZXandHinata 8918d40b59 recoverssm: GDN: commit forward with the verify step's own decay, bit-identical to per-token (port of upstream #58863, 514102a1db)
Port of the upstream vLLM change #58863 (Apache-2.0). The verify kernel records the decay it applied and the commit replays the
accepted tokens forward with the same update, so the committed state equals the per-token state bit for bit.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 20:08:56 +07:00
MOVZXandHinata 7e9cd428a1 recoverssm: align commit: fold final states before boundary states (port of upstream #58863, 3388ba1a25)
Port of the upstream vLLM change #58863 (Apache-2.0). The final-state folds run before the boundary-state folds, because in align
mode a boundary state can be written into the slot a final fold still reads.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 20:08:56 +07:00
MOVZXandHinata e5e541446f recoverssm: FULL cudagraphs for uniform spec-decode batches (port of upstream #58863, 3aa3d5b586)
Port of the upstream vLLM change #58863 (Apache-2.0). The GDN and PLE RecoverSSM metadata builders declare UNIFORM_BATCH, so
--use-replayssm resolves FULL_AND_PIECEWISE; graph padding rows get a null state slot and write nothing.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 20:08:56 +07:00
MOVZXandHinata 2c6efaabe8 recoverssm: one fold per program in the GDN RecoverSSM state commit (port of upstream #58863, af0d2b45d7)
Port of the upstream vLLM change #58863 (Apache-2.0). Each program folds either the final or the block-boundary state, not both,
which halves the live accumulator tiles.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 20:08:55 +07:00
MOVZXandHinata eb68fb9105 recoverssm: read the GDN RecoverSSM flag defensively in _forward_core (port of upstream #58863, c1d2da8773)
Port of the upstream vLLM change #58863 (Apache-2.0). One check read use_gdn_recoverssm directly; it now uses getattr with a False
default like its neighbours, so layers built without __init__ in tests work.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 20:08:55 +07:00
MOVZXandHinata cb27582783 recoverssm: release the RecoverSSM step after CUDA graph memory profiling (port of upstream #58863, fdcc43cf3e)
Port of the upstream vLLM change #58863 (Apache-2.0). The step recorded during graph-capture dummy runs held the profiling KV cache
alive after profiling; it is now released at teardown.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 20:08:55 +07:00
MOVZXandHinata 24bcbc5087 recoverssm: RecoverSSM for Qwen GDN and the PLE short conv (port of upstream #58863, 03cae7837f)
Port of the upstream vLLM change #58863 (Apache-2.0). With --use-replayssm and speculative decoding, Gated DeltaNet layers verify the
draft window from a read-only checkpoint and keep a per-token replay record; after sampling one commit replays
the accepted tokens and writes the recurrent state once. The PLE short conv gets the same treatment.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 20:08:55 +07:00
MOVZXandHinata 5df639bb4e [QSA] Start the clip-counter reader after capture, not on the first forward
The FP8 KV clip counter's reader thread used to start on the first eager forward, before CUDA graph capture, so
its device-to-host copies could overlap a capture. _qsa_clip_count now only allocates the per-layer counter; the
reader is started by a hook that guard_telemetry runs after warmup and capture (or at once if capture is already
done). Counting is unchanged. Tests cover the order, idempotence and a failing control that restores the old start.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 20:05:28 +07:00
MOVZXandHinata ef55e095c3 [Guards] The bounds-guard telemetry code default is warn
VLLM_E2_GUARD_MODE now defaults to warn in code, matching what the shipped launchers set; v2 through v2.2.0
defaulted to off. The comment above it, the MODES docstring, the vllm/envs.py default and its annotation follow.
tests/v1/worker/gpu/test_guard_default_mode.py checks that code, comment and envs.py agree, from the sources, so
it runs without CUDA, and fails on the earlier source.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 20:04:39 +07:00
MOVZXandHinata 036608b012 [Engine] Detect a fail-closed fault or death on any rank
A fault on a non-zero TP rank could hang the engine: the worker's sentinel pipe
was inherited by a helper process and stayed open, so the monitor never saw
the exit. Earlier releases carry the same monitor and fail-closed exit path.

- Worker exit is detected by pidfd, with the sentinel as fallback, before and
  after READY and for every worker through the whole initialisation window.
- A per-worker fault pipe: ple_fail_closed writes one byte before os._exit and
  the monitor starts the failure ladder at once; the fd stays registered
  through worker teardown.
- worker_main marks inherited pipes close-on-exec.
- Any unexpected busy-loop failure exits 1; the termination ladder uses
  time.monotonic(); forced-exit paths no longer wait on logging.
- Tests on real fork/spawn workers for rank 0 and rank 1, with and without an
  fd-inheriting helper.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 20:04:37 +07:00
MOVZXandHinata 6c956ee3bd [Engine] Drain-order shutdown
- EngineCore survives a late SIGTERM during teardown (bounded deferral) so
  every worker drains its guard telemetry.
- A worker exits right after its orderly shutdown.
- A worker failure racing shutdown stays a failure (exit 1).
- Rank 0 waits for the PLE child's whole thread group, bounded, instead of its
  reap; drain-aware failure ladder with a bounded grace for survivors whose
  shutdown already started.
- tests: the sorted-emit graph arm takes max_len on the host before capture.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 20:04:37 +07:00
MOVZXandHinata eeea036c12 [TopK] In-kernel canonical emission (MERLIN_TOPK_SORTED_EMIT)
Emits the selected indices in canonical ascending order inside the kernel so
the Python QSA sort can be skipped. Output-equivalent to the Python sort;
reference model, property tests and GPU arms included. Default off in this
commit.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 20:04:37 +07:00
MOVZXandHinata a4aecf9ec7 [PLE] The dying-path traceback runs no exception code
ple_fail_closed formatted its traceback with traceback.format_exception, which
runs exception-defined __str__/__getattribute__; if one blocked, the rank never
reached os._exit(1) and kept its GPUs. A safe formatter reads frames and plain
string arguments only.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 20:03:19 +07:00
MOVZXandHinata 3575cdb126 [Guards] Draft identity and draft PLE guard report the drafter's own state
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 20:03:19 +07:00
MOVZXandHinata e3fcd7e3ae [Rebase] Row dump removed, lint, knob read-site census
The debug PLE row dump is deleted; ruff findings on fork lines cleared; a
read-site census test pins which fork knobs are read and where.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 20:01:43 +07:00
MOVZXandHinata 7f6055caa2 ple, tests: debug-gated pull-flag check; diagnostics report that they ran
The pull-transport flag check is wired in (rank-gated, off by default, warns when enabled). Both
diagnostic checks log that they ran. Test repairs for helpers v0.30.0 renamed.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 20:01:26 +07:00
MOVZXandHinata fb52250c11 [Scheduler] Drop the one-victim preemption guard that upstream #49675 made dead
The cascade test now expects zero needless victims.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 20:01:26 +07:00
MOVZXandHinata fba504d5bb [QSA] Tag every raw-key ring row and validate it before pooling
The fused pre-indexer could read a stale or uncommitted raw-key ring row and
pass its int64 tail to RoPE as a position (illegal address). The writer now
tags each row with logical_position + 1; the reader checks the tag of every
source row and, on a mismatch, masks the pooled load, the RoPE-tail read and
the compressed-slot store. New guard site 11 (always fatal under warn). The
KV pool size is unchanged.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 20:01:26 +07:00
MOVZXandHinata ad80cac06d [Guards] Repair: carry patch 0027's slot encode/decode helpers
Landed 0027 with mamba_utils.py and guard_telemetry.py under the
ours-newer rule; that rule saw the slot-ownership check and the
diagnostic gate, not the bias-encoding helpers. decode_slot(),
encode_slot(), GUARD_MAX_ENCODABLE and GUARD_MIN_ENCODABLE are new
names, present in v2.5.1 and absent here, and patch 0049's ring-row
report calls decode_slot(). Replays 0027's guard_telemetry.py section
(3 hunks, patch -F 8) onto the current tree.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 19:58:32 +07:00
MOVZXandHinata dec8be19de [PLE] The pull registry refuses unbound or clashing names; test fixes
A schema walk fails on any pointer-like int parameter of a custom op on the
PLE path. The FULL_K test child calls the Marlin MoE GEMM by keyword against
its v0.30.0 signature.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 19:56:01 +07:00
MOVZXandHinata 281491646d [Worker] E2-identity and rank-tagged e2-toggles startup lines
Each worker logs its launch identity and, after warm-up and capture, one
e2-toggles line with the effective toggle state read back from the C++
extensions through two new tensor-free query ops. Both extensions must be
rebuilt.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 19:56:00 +07:00
MOVZXandHinata bef464a69d [TopK] Diagnostic-only build defines and qualification test cells
Compile-time defines (default 0, device code unchanged by default) for the
delayed-publication stress, the pre-repair control and a forced-fault cell;
test cells for the OFF-arm controls, the QSA_SORT contract, FULL_K attribution
on the Marlin MoE GEMM and legacy-tail repeated-row membership.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 19:56:00 +07:00
MOVZXandHinata cff46af086 [TopK] Follow-up fixes to the persistent top-k kernels
- Counter enum declared above the decode template (two-phase lookup).
- Emission-fault flag set atomically.
- Close the legacy large tie loop with a block barrier (inherited).
- Large-path fault injectors registered as hashed passthroughs.
- The tensor-free counter-slots op registered for CompositeExplicitAutograd.
- Tie measurements recorded only on rows neither fault verdict failed.
- Tests scoped to the properties the build promises.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 19:56:00 +07:00
MOVZXandHinata 2918e3fda1 [TopK] Determinism toggles for greedy repeatability
- MERLIN_FULL_K: deterministic Marlin MoE split-K (after unmerged upstream #46639).
- MERLIN_QSA_SORT: canonical order for the QSA selected blocks.
- MERLIN_TIE_DET: deterministic tie-break at the persistent_topk cutoff.
- MERLIN_TIE_RECENT: cutoff ties keep the highest indices.
- Decode candidate truncation is detected; one tie direction across the
  medium path; the kernel switches are registered.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 19:56:00 +07:00
MOVZXandHinata 512ca2fc0b [TopK] Large-path hardening
- Inline-PTX acquire/release helpers declare their memory effects.
- The fault verdict is aggregated across the CTA group before emission, so a
  faulted row is failed closed by every CTA.
- 64-bit device counters and an int64 reader (counter protocol v2).
- Tests: adversarial distributions, slot-residue coverage, an sm_90 gate for
  tests that need it.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 19:55:59 +07:00
MOVZXandHinata 63e2d8be20 [TopK] Deterministic placement and tie membership for the large-path radix_topk
Above RADIX_THRESHOLD a row is split across a group of CTAs. The pivot was
already order-independent; cross-CTA placement, intra-CTA placement and tie
membership were arrival-order races. They are now deterministic behind
MERLIN_TOPK_DET_LARGE (default on).

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 19:55:59 +07:00
MOVZXandHinata daa56356c8 [TopK] Barrier the histogram round before the leader publishes arrival
In radix_topk's large-path round loop the leader published arrival with no
block barrier after the histogram merge, so a peer CTA could observe an
incomplete histogram. Inherited by earlier releases; fixed below the placement
work so both arms of any comparison carry it.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 19:55:59 +07:00
MOVZXandHinata 66c3d11016 [TopK] Python reader for the exact-restart counters
Reads persistent_topk_exact_counters into {slot: value}, logs one line on a
cadence, and degrades to a one-time warning when the op is absent.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 19:55:59 +07:00
MOVZXandHinata 7f4385a422 [PLE] The pull op resolves live pointers by layer name
The wait/copy op took host pointers as Python-int arguments, which were baked
into the AOT compile cache as the tracing process's addresses; the first
forward after a cache-load boot then failed. The op now carries only
process-stable values and resolves pointers from a per-process registry.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 19:55:23 +07:00
MOVZXandHinata 54d4ab1b9a [Guards] The draft guard reports the draft model's own architecture
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 19:55:23 +07:00
MOVZXandHinata 09ea3a66e9 [KV] Restore PP fix #54795 -- PP stages need only share a KV layout
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 19:55:23 +07:00
MOVZXandHinata e2eaefdec9 [KV] Restore PP fix #54793 -- no KV tensors for another stage's layers
A KV cache group with no layer on a pipeline stage was still given tensors
there, and allocate_kv_cache raised StopIteration on the second stage. The
fix was lost in the v0.30.0 rebase. Test drives the real get_kv_cache_configs
with the production layout.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 19:55:23 +07:00
MOVZXandHinata 6705330924 [PLE] The pull wait/copy op is a graph splitting op
With the wait/copy inside a compiled piece, Inductor's post-grad
auto-functionalization decomposition failed on the stage-0 ranks.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 19:55:22 +07:00
MOVZXandHinata 2e68aba2a7 [Qwen4Exp] Restore torch.compile on the NVIDIA path; exact breakable pin; runtime graph-mode assertion
- torch.compile restored on the NVIDIA Qwen4Exp target and MTP so the
  production profile (VLLM_COMPILE + FULL_AND_PIECEWISE, breakable off) runs.
- The production pin requires VLLM_USE_BREAKABLE_CUDAGRAPH to be exactly "0".
- assert_resolved_cudagraph_modes() runs at the top of capture_model.
- CPU tests for the combined profile, the exact pin and the resolved modes.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 19:55:22 +07:00
MOVZXandHinata 23349e43b2 [PLE] Integrate the pull transport on v0.30.0 with upstream UVA refused
- VLLM_E2_PLE_PULL_TRANSPORT replaces the fork's old switch name, which
  v0.30.0 reuses for its UVA backend with the opposite default; strict parser;
  compile/capture factor.
- The pull transport is reachable on v0.30.0 through a pull embedding class
  that inherits upstream's without editing it; the upstream UVA backend is
  provably off while the pull switch is on.
- resolve_ple_mode()/assert_ple_mode() log and assert the effective PLE mode
  for target and drafter, and refuse a mixed configuration.
- Producer log lines for the PLE backend and weight dtype.
- CPU tests for mode resolution and the mixed-config refusal.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 19:52:54 +07:00
MOVZXandHinata 3b5fc7c7be [Worker] Boundary allocator telemetry and an opt-in PLE row dump
Env-gated diagnostics, inert unless set. Allocator telemetry is read only at
step boundaries, armed after warm-up and capture.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 19:49:50 +07:00
MOVZXandHinata 025fd08603 [PLE] Carry the host-mapped pull transport onto v0.30.0
Restores the fork's PLE offload package (not upstream) and its pull
transport: the offload process holds no CUDA context; the captured op waits on
a device-mapped host flag and copies host->GPU on the model stream; publication
is fenced; releases are synchronous across dummy/capture work; failures close
the rank. Includes the test-only fault-injection cuts for the transport.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 19:49:16 +07:00
MOVZXandHinata bc0a1cee82 [Guards] Executable graph-profile and breakable-cudagraph guards
New vllm/v1/e2_guards.py, called from the target and MTP constructors on every
rank before profiling or capture: the breakable-cudagraph pin is asserted
(three readings of the switch), a graph profile that cannot execute on v0.30.0
is refused, and VLLM_E2_EXPECTED_CUDAGRAPH_MODE is checked by name. Tests
cover the drafter. An inert knob is no longer hashed into the cache key.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 19:48:07 +07:00
MOVZXandHinata 7351759ecf [Rebase] Remove v0.30.0 merge artefacts; import order
- Drop the PLE-offload half of ple_layer.py that upstream #54371 deleted and a
  3-way merge resurrected (it no longer imported on v0.30.0).
- Drop the structured-output helper duplicate that v0.30.0 already provides,
  and an unused import.
- Re-sort the import blocks the merges left unsorted (ruff I001).

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 19:47:45 +07:00
MOVZXandHinata f1e5437a3a [Config] Register the fork's knobs by cache identity; lint; gpu test marker
- Every fork knob is registered in envs.py and classified as a compile/capture
  factor or runtime-only; envs.py documents that capture identity and AOT
  cache identity are different things.
- ruff findings cleared on lines this fork authored; upstream lines untouched.
- A `gpu` pytest marker so the GPU-free tier can be selected, not just skipped.
- Duplicate knob-table entries from two branches reconciled onto one
  off|warn|raise normaliser.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 19:47:45 +07:00
MOVZXandHinata 31ad7e113f [Worker] Debug-gated diagnostics for Mamba slot ownership and guard encoding
- Host-side Mamba slot-ownership check (off|warn|raise, default off) with a
  failing control that proves it detects a wrong block-table row.
- Guard telemetry: int64 encoding limits and the extreme-value pair closed by
  test; a decoded-None counter field is now named in the warning; a
  captured-replay sentinel probe.
- Both knobs registered in envs.py.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 19:44:27 +07:00
MOVZXandHinata 754fb3e29d [TopK] Exact restart of the medium path on candidate-buffer overflow
When the persistent top-k medium-path candidate buffer overflows, the row is
restarted exactly instead of being truncated. Capture-safe counters with atomic
publication and a coherent snapshot record every restart.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 19:43:58 +07:00
MOVZXandHinata 17c1c70447 [Spec Decode] Structured output under concurrency on a PP shape
Ports upstream vLLM PRs #54442 and #56802 (issue #54437): never sample from an
unmasked row, and key draft-token hand-off snapshots by scheduler step so an
alternating PP microbatch cannot overwrite the slot.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 19:43:58 +07:00
MOVZXandHinata 03c8ebae34 [Scheduler] Allocation log for KV capacity probes
Env-gated per-step log of blocks allocated and freed and scheduled tokens per
request; prefix-cache hits are read behind a schema check so an upstream field
rename fails visibly.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 19:43:56 +07:00
MOVZXandHinata 4c6c7af076 [QSA] Count skipped ring commits
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 19:43:56 +07:00
MOVZXandHinata c841dce4a4 [Scheduler] Defer ring-block free under overlapping steps
A circular-buffer ring block could be freed while an earlier, still-executing
step was reading it. The free is deferred until the overlapping step has
committed.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 19:43:56 +07:00
MOVZXandHinata 73ab1259c1 [QSA] Bound the RoPE position before it becomes an address
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 19:43:56 +07:00
MOVZXandHinata 5de9f4a6ae [GDN] Bound the recurrent state index
Both callers of the guarded kernel (fused recurrent and KDA) are updated in the
same commit.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 19:43:56 +07:00
MOVZXandHinata 9bc82b7ceb [Scheduler] Optional host-side ownership check before a block is zeroed
Host-side diagnostic behind VLLM_E2_ZERO_OWNERSHIP_CHECK, off by default.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 19:43:56 +07:00
MOVZXandHinata 5ac158fbe6 [Worker] Pool-bound the KV block zeroer
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 19:43:55 +07:00
MOVZXandHinata c646a6c792 [Worker] Validate req_idx in the block-table gather and the slot mapping
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 19:43:55 +07:00
MOVZXandHinata 7118c2f999 [QSA] Bound the indexer page-table values
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 19:43:55 +07:00
MOVZXandHinata 8ca58f1714 [QSA] Bound the pre-indexer request row
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 19:43:55 +07:00
MOVZXandHinata b1fdcd6e27 [GDN] Bound the decode state index
Ports the check from upstream vLLM PR #50021.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 19:43:54 +07:00
MOVZXandHinata 62acb3dab8 [Worker] Index the persistent per-request-slot block table under PP
Under pipeline parallelism the Mamba/GDN state-copy path indexed a block-table
row gathered for a different step. The align post-process now indexes the
persistent per-request-slot table, and fails closed when it is absent.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 19:43:54 +07:00
MOVZXandHinata 41f6691c26 [Qwen4Exp] Host-resident token embedding tables (VLLM_HOST_EMBED_TABLE)
Keeps the token embedding table in pinned host memory behind a UVA view for the
stage-0 target and the last-stage MTP drafter, freeing GPU memory for KV. TP
sharding, masking and the all-reduce are retained. The knob is registered as a
compile-cache factor.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 19:43:53 +07:00
MOVZXandHinata 66b3d8c108 [Engine] Workers cannot outlive a killed EngineCore
Workers set PR_SET_PDEATHSIG(SIGKILL) against their EngineCore parent with a
getppid race check. Start methods other than fork/spawn are refused before any
worker is created. Subprocess tests cover fork, spawn, graceful SIGTERM and the
refusal.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 19:43:53 +07:00
MOVZXandHinata a902e402c6 [PLE] Standalone fault-injection singleton used by the offload transport
A stdlib-only, one-shot, rank-selective fault selector shared by the PLE
offload transport and its tests. Inert unless armed: disarmed, every site is a
single None check outside any compiled or captured region.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 19:43:53 +07:00
MOVZXandHinata 7e5f658d75 [Qwen4Exp] Pipeline-stage ownership for PLE, the MTP drafter and the vision tower
Each pipeline stage builds only the parts it owns: PLE when every PLE layer
lives on stage 0, MTP under PP (port of vLLM PR #46994 minus its ROCm MLA file
and tests) with the drafter on the last stage, and the vision tower on the
first stage only.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 19:43:53 +07:00
MOVZXandHinata 461a8a0f82 [Qwen4Exp] FP8 E4M3 KV reader for QSA with a static scale sidecar
FP8 E4M3 KV read path for the Qwen4Exp sparse attention (QSA), indexer and
pre-indexer kernels on sm_86, dequantised with a static per-layer scale loaded
from a JSON sidecar, validated against the model's QSA layer set and applied
per pipeline stage. Includes the offline absmax collector that produces the
sidecar and a CUDA-graph-safe clipping counter.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 19:43:52 +07:00
MOVZXandHinata 03f5313f0f [Worker] Capture-safe guard-telemetry infrastructure, default off
Shared machinery for device-side bounds guards: persistent int64 counters per
site, written by guarded kernels and read by a background thread. The reader
starts only after warm-up and CUDA-graph capture and never waits on the
compute stream; a read failure arms the stop rather than popping the buffer.
Shutdown does one synchronous final drain so a hit on the last step is
reported before teardown. Release default: off.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 19:43:52 +07:00
2841 changed files with 85137 additions and 257097 deletions
@@ -1,115 +0,0 @@
---
name: model-microbenchmark
description: Time a vLLM model's forward pass on exact batch shapes (decode, prefill, chunked prefill, prefix hit, spec-decode verify, or mixes) with `vllm bench forward`, single-GPU or DP/TP/EP across nodes. Use for per-step model latency, layout/parallelism comparisons, and checking a model-level optimization. Not for single kernels (use kernel-microbenchmark) or end-to-end serving (use `vllm bench serve`).
---
# Model Microbenchmark (`vllm bench forward`)
Builds the batch without the scheduler, runs it through MRV2's real
`execute_model`, and times the forward as a one-shot CUDA graph (prefill
included). Sampling, the speculator, and CPU step overhead are excluded, so
expect results somewhat below a serving step.
## Usage
```bash
vllm bench forward --model deepseek-ai/DeepSeek-V4-Flash \
--case decode=64*1@8192 --case prefill=8192 --case mix=4096,32*1@8192
# DP/TP/EP: one process per GPU (multi-node: add --nnodes/--node-rank/--master-addr)
torchrun --nproc-per-node=4 -m vllm.entrypoints.cli.main bench forward \
--model ... -dp 4 --enable-expert-parallel --case decode=128*1@100000
```
Requests: `[N*]Q[@K][c][:D]`, comma-separated, per DP rank.
| Form | Meaning |
| --- | --- |
| `8192` | fresh prefill, 8192 tokens |
| `32*1@8192` | 32 decodes, KV length 8192 (KV length **includes** the query) |
| `4@4096:3` | spec-decode verify, 3 draft tokens |
| `2048@10240` | prefill after an 8192-token prefix-cache hit |
| `2048@10240c` | chunked prefill: the prefix came from earlier chunks |
Verify cases require `--speculative-config` with `num_speculative_tokens`
at least as large as the case's draft count; insufficient capacity is rejected.
Hit vs chunk only differs on models with SWA bounded replay (DSV4.1): a hit
also recomputes the last window (128 tokens) of the prefix.
Or `--shapes file.json|yaml`:
`{"cases": [{"name": ..., "requests": [{"query_len", "kv_len", "num_draft_tokens", "chunked", "count"}]}]}`.
Key flags (plus all `EngineArgs`):
- `--kv-len-jitter 0.5` (default): prefixes drawn within ±50%, same on every
DP rank, capped at `max_model_len`. Use `0` for exact lengths.
- `--routing auto|real|balanced`: `auto` = balanced with dummy weights.
- `--iters 20 --samples 5`, `--output-json out.json` (per-rank latency stats,
`samples_us`, graph-vs-eager check).
- Defaults: `--load-format dummy`, prefix caching off, no torch.compile.
Read `median_us` (each replay is already the max over ranks).
## Kernel breakdown
Add `--profile-dir DIR` to see where the time goes. After timing, each case is
replayed 3 more times under `torch.profiler`; every rank writes
`DIR/<case>.rank<r>.kernels.json` (GPU time per kernel per forward, sorted)
and `DIR/<case>.rank<r>.trace.json.gz` (open in Perfetto). Rank 0's top
kernels are printed and included in `--output-json`.
```bash
python -m vllm.benchmarks.forward.profile DIR/decode.rank0.kernels.json
# Before/after a change, same case: per-kernel deltas, largest first.
python -m vllm.benchmarks.forward.profile base/decode.rank0.kernels.json new/decode.rank0.kernels.json
```
- Kernels are grouped by name without template or function arguments; the
JSON keeps one full name per group in `example`.
- Only the timed forward is included: the in-graph KV restore and rank
rendezvous are excluded (`pre_forward_us`). `span_us` (first to last
forward kernel) should be close to `median_us`, a bit higher from profiler
overhead; `busy_us` can exceed it when streams overlap.
- Compare the same rank across runs: under EP, ranks differ (e.g. collective
wait time shows up inside communication kernels).
For Nsight Systems (NVTX, NCCL, CPU side, GPU metrics), skip `--profile-dir`:
the two can't share CUPTI. Each case's 3 profiled replays run between
`cudaProfilerStart`/`Stop` in an NVTX range `forward/<case>`, so a capture range
records only them, one report per case (and rank):
```bash
nsys profile --capture-range=cudaProfilerApi --capture-range-end=repeat \
--cuda-graph-trace=node -o prof vllm bench forward ... # prof.1, prof.2, ...
torchrun --nproc-per-node=4 --no-python nsys profile ... -o prof_rank%q{RANK} \
python -m vllm.entrypoints.cli.main bench forward ...
nsys stats --report cuda_gpu_kern_sum --format csv -o kern prof.1.nsys-rep
```
Keep `--capture-range-end=repeat`: nsys's default stops after the first case
and leaves the bench hanging. `nsys stats` also counts the KV restore copies
that the JSON summary excludes.
## Caveats
- **Dummy weights lie for data-dependent kernels.** Dummy gates route every
token to the same experts (hence balanced routing). Cross-check suspicious
kernels with real weights.
- **torch.compile models (e.g. Qwen3/3.5) look slow**: no compile here. Use
them for correctness only; trust perf for models under `vllm/models/`.
- `sample spread > 0.5%` warns of unstable timing. DSV4.1 DEP prefill is
inherently ~0.5–1% (MegaMoE cross-rank sync, host engram reads).
- `graph_vs_eager` may be nonzero for small batches where models branch on
`is_current_stream_capturing()`; that's expected, not a bug.
- KV cache is zero-initialized (`--kv-init zeros`); random KV was within 3%.
- Uniform DP only: every rank gets the same case; no idle ranks.
- CUDA only: it hangs on ROCm (untested cause: likely the in-graph timing
events).
- Large prefills/prefix hits need room: raise `--max-num-batched-tokens` /
`--max-model-len` / `--gpu-memory-utilization` if build fails on KV blocks.
- Startup cost is JIT/autotune, not the bench: install FlashInfer jit-cache
and cubin; DeepGEMM warmup is skipped (each case's eager pass compiles its
own shapes).
- To compare against serving decode: `vllm bench latency` with 65- vs
1-token outputs, divided by 64, with a FULL decode graph.
-157
View File
@@ -1,157 +0,0 @@
---
name: model-recipes
description: Find vLLM deployment recipes and configs for LLMs and Hugging Face models by model and hardware. Use for recipe lookup, creation, and updates.
---
# Model Recipes
Model serving recipes are maintained in this repository's
[recipes/models/ directory](../../../recipes/models). Omni (image, video,
audio) recipes stay in vllm-project/recipes for now (moving to vllm-omni),
and Ascend NPU content is inline in these recipes (moving to vllm-ascend
overlays). The published catalog, rendered by
[vllm-project/recipes](https://github.com/vllm-project/recipes), is at
[recipes.vllm.ai](https://recipes.vllm.ai).
## Usage notes
1. Recipe lookup returns commands and deployment guidance without launching
`vllm serve`.
2. Use published configurations or the recipes repository's command generator
for supported selections. Return commands as published, including their
flags and parallelism settings.
3. Recipes are starting points for deployment and may not be tuned for your
workload. Check each recipe for its verified configurations, and use
benchmarks of your workload to guide further tuning.
4. To create or update a recipe, follow the
[authoring workflow below](#create-or-update-a-recipe).
## Find a recipe
Read recipes from the published JSON on recipes.vllm.ai. It is generated from
the source YAML with variant, hardware, and strategy settings already
resolved; the YAML contains comments and conditional overrides, so do not use
it for lookup. When presenting results, link the website page. Read the source
YAML only when creating or updating a recipe.
Each recipe has a page at `https://recipes.vllm.ai/<org>/<repo>` and JSON at
the same path plus `.json`. Paths use the full Hugging Face ID, preserving
case; do not substitute `meta.slug` or a display title. For example,
`Qwen/Qwen3.8-27B` maps to
[recipes.vllm.ai/Qwen/Qwen3.8-27B](https://recipes.vllm.ai/Qwen/Qwen3.8-27B).
Recipe JSON is often over 20 KB, mostly the `guide`. Instead of printing it,
read only what the request needs: each step below returns the paths for the
next, so stop once the request is answered. Commands need `curl` and `jq`.
[scripts/read_recipe.sh](scripts/read_recipe.sh) is in this skill's directory;
the examples use its path from the vLLM repository root.
### Search the catalog
Skip this step if the exact HF ID is known. Otherwise search by model ID,
family, or provider using case-insensitive keywords. Each
whitespace-separated word must occur in the ID, title, or provider:
```bash
curl -fsSL https://recipes.vllm.ai/models.json |
jq --arg q 'qwen 27b' '
($q | ascii_downcase | [scan("\\S+")]) as $words | .[] |
([.hf_id, .title, .provider] | join(" ") | ascii_downcase) as $text |
select(all($words[]; . as $word | $text | contains($word))) |
{hf_id, json, url, derived_from}'
```
`json` is the recipe data path; `url` is its page path. Prefix both with
`https://recipes.vllm.ai`. A `derived_from` entry is a variant of a parent
recipe: fetch its own `json` to get that checkpoint's command. Its page may
belong to the parent, so link the returned `url` rather than assuming every
checkpoint has a separate page.
If there is no exact match, search the family and inspect `variants` before
concluding the recipe is missing.
### Summarize the recipe
Save the recipe JSON to a file and summarize it:
```bash
read_recipe=.agents/skills/model-recipes/scripts/read_recipe.sh
recipe=$(mktemp)
curl -fsSL https://recipes.vllm.ai/Qwen/Qwen3.8-27B.json -o "$recipe"
"$read_recipe" "$recipe"
```
`default` is the recommended hardware, strategy, and variant; `hardware` maps
each hardware ID to its command path; `recipe_verified_hardware` lists hardware
where the recipe as a whole was verified end to end. For capability
requirements, such as tool calling or text-to-video, check `features` and
`omni_tasks`, then confirm in the guide.
Some recipes, including Omni, lack `hardware`; skip hardware selection and
use the guide's launch and task-specific sections.
### Select hardware and strategy
Match a hardware ID to the user's GPU and fetch its path from `hardware`,
prefixed with `https://recipes.vllm.ai`, rather than constructing a URL. That
response is the command object itself: single-node objects carry `command`,
and multi-node objects carry `head_command` and `worker_commands` instead, so
list its keys before selecting fields.
Check `hardware_profile` to confirm that the selected configuration matches
the user's GPU model and specifications. Its `gpu_count` describes the profile,
not necessarily the number of GPUs used by the command. Check the command's
parallelism settings, `node_count`, and all `worker_commands` against the
user's available topology, and compare the variant's `vram_minimum_gb`, an
estimate, with the total memory of the GPUs the command uses; a configuration
below that minimum is not a working option unless the guide says it was
tested. Ask when the user's hardware is unclear. If no
published command or guide command fits, report the mismatch and return the
closest one unchanged; list any adjustments the user's setup would need
separately and mark them unverified instead of editing the command.
For another strategy, follow `alternatives` in the same response. These are
static files; query parameters do not change their configuration. For other
feature/topology selections, use the recipes repository's command synthesis.
### Read deployment guidance
Read the guide sections that apply to the selection, such as prerequisites,
launch notes for the selected hardware or variant, troubleshooting, and known
limitations. Pass entries from `guide_sections` or prefixes of their text.
The script prints every matching section with its subsections and fails when
a section matches nothing:
```bash
"$read_recipe" "$recipe" prerequisites troubleshooting
```
Query other fields with `jq` on the saved file, such as `features.<name>`,
`hardware_overrides`, or `omni.tasks` for request examples.
### Return the result
Return the website link, selected checkpoint and hardware, the requested
command, and relevant prerequisites and caveats from the guide. Pick one
command and copy it, including its Docker form, exactly as the command object
or guide section gives it; do not combine commands from different sections or
add, drop, or reorder flags. List optional changes separately.
For each command, state its source, either a generated hardware or strategy
configuration or a named guide section, and whether that source says it was
tested. A `recipe_verified_hardware` entry covers the recipe on that
hardware, not every variant, strategy, or guide command, so it does not make a
command verified. Generated configurations are not automatically verified.
## Create or update a recipe
Recipes cover official or widely used checkpoints, with configurations verified
end to end on real hardware. Do not add recipes for arbitrary community
uploads, untested hardware claims, or non-standard flags that only work for one
setup. Before opening a pull request, confirm that the user ran the
configuration and can share the results.
Edit the YAML under [recipes/models/](../../../recipes/models) in this
repository, following the recipes repository's
[add-recipe skill](https://github.com/vllm-project/recipes/blob/main/.agents/skills/add-recipe/SKILL.md)
for the schema. Validate and preview it from a checkout of
`vllm-project/recipes` pointed at this directory, as
[recipes/README.md](../../../recipes/README.md) describes.
@@ -1,4 +0,0 @@
interface:
display_name: "Model Recipes"
short_description: "Find vLLM deployment recipes and configs for LLMs and Hugging Face models by model and hardware."
default_prompt: "Use $model-recipes to find a recipe for my model and hardware, and return its commands and deployment caveats."
@@ -1,66 +0,0 @@
#!/bin/bash
# Usage: read_recipe.sh <recipe.json> [guide-section...]
# Without sections, print a compact summary of a recipes.vllm.ai recipe JSON.
# With sections, print each guide section whose heading starts with one of
# them (case-insensitive), including its subsections.
set -euo pipefail
if [ $# -lt 1 ]; then
echo "Usage: $0 <recipe.json> [guide-section...]" >&2
exit 1
fi
# Print "<level>\t<heading text>\t<line>" for guide lines, marking headings
# outside code fences with their level and other lines with 0.
guide_lines() {
jq -r '.guide // ""' "$1" | awk '
/^[[:space:]]*(```|~~~)/ { fence = !fence }
!fence && match($0, /^#+ /) {
print RLENGTH - 1 "\t" substr($0, RLENGTH + 1) "\t" $0
next
}
{ print "0\t\t" $0 }'
}
if [ $# -gt 1 ]; then
recipe=$1
shift
status=0
for section in "$@"; do
guide_lines "$recipe" | awk -F '\t' -v h="$section" '
BEGIN { h = tolower(h); sub(/^#+[[:space:]]*/, "", h) }
$1 > 0 && lvl && $1 <= lvl { lvl = 0 }
$1 > 1 && !lvl && index(tolower($2), h) == 1 { lvl = $1; found = 1 }
lvl { sub(/^[^\t]*\t[^\t]*\t/, ""); print }
END {
if (!found) {
print "No guide section starts with: " h > "/dev/stderr"
exit 1
}
}' || status=1
done
exit $status
fi
sections=$(guide_lines "$1" | awk -F '\t' '$1 == 2 || $1 == 3 { print $3 }' |
jq -R . | jq -s .)
jq --argjson sections "$sections" '{
hf_id,
title: .meta.title,
tasks: .meta.tasks,
recipe_verified_hardware: .meta.hardware,
omni_tasks: (.omni.tasks // null |
if . then map(objects |= {id, label, endpoint, model_id, description})
else null end),
variants: (.variants // {} | map_values(
{model_id, precision, vram_minimum_gb, supported_hardware} |
with_entries(select(.value != null)))),
features: (.features // {} | keys),
opt_in_features,
default: (.recommended_command // null |
if . then {hardware, strategy, variant} else null end),
hardware: .recommended_command.by_hardware,
guide_sections: $sections
} | with_entries(select(.value != null))' "$1"
+1 -3
View File
@@ -1,6 +1,6 @@
---
name: pr-checklist
description: Prepare a vLLM change for human PR review or re-review, or analyze an open pull request. Use before opening a PR, drafts included (e.g. `gh pr create`), or pushing review fixes, to check design fit, behavioral coverage, performance evidence, diff quality, and closure of previous feedback.
description: Prepare a vLLM change for human PR review or re-review, or analyze an open pull request. Use to check design fit, behavioral coverage, performance evidence, diff quality, and closure of previous feedback before requesting maintainer attention.
---
# PR Checklist
@@ -181,8 +181,6 @@ Summarize model evaluations and benchmarks in compact tables with the configurat
When fixing bugs or issues, provide a clear root-cause analysis. Explain what caused the problem, how it was identified, and why the chosen solution effectively addresses it. A patch which resolves the issue without a proper root-cause analysis may lead to recurring problems and increased maintenance overhead.
For fixes that involve disaggregated prefill/decode (PD) logic, such as the NIXL connector (`vllm/distributed/kv_transfer/kv_connector/v1/nixl/`), make a best effort to include a minimal example of the flow being changed and/or the initial flow that caused the failure. A short numbered sequence (or diagram) of a request's path across the P and D instances, e.g. scheduler admission, prefill execution, KV transfer registration and write, decode-side read, token generation, lets reviewers trace the root cause and the fix without reconstructing the distributed flow from the diff. Keep the example minimal: only the steps and components involved in the change. See the [disaggregated prefill](../../../docs/features/disagg_prefill.md), [NIXL KV push](../../../docs/design/nixl_kv_push_connector.md), and [NIXL KV cache lease](../../../docs/design/nixl_kv_cache_lease.md) docs for the flows involved.
#### 4.4: Implementation Details
Provide a detailed explanation of the implementation, including design decisions, trade-offs, and any relevant technical details. Highlight any non-obvious choices, tradeoffs, potential limitations, and areas where maintainer attention is required.
@@ -1,191 +0,0 @@
---
name: transformers-upgrade
description: "Playbook for changing vLLM's Transformers version bounds. Use when bumping the pinned or maximum Transformers version, raising the minimum Transformers version, or deleting vendored configs, processors, or compat shims that upstream Transformers now provides."
---
# Upgrading Transformers
The supported range lives in `requirements/common.txt`. CI pins one exact
version in `requirements/test/*.in` and the compiled `requirements/test/*.txt`.
A version change is one of two jobs, and they usually come as a pair:
- **Bump the pin**: a new Transformers release becomes the CI version and
the upper bound. vLLM grows compat code for it.
- **Raise the floor**: the minimum moves up. That compat code, and every
vendored config or processor upstream now ships, becomes dead weight.
Every floor raise ends with an **unvendoring pass**.
## Bump the pin
1. Edit the pin line in each `requirements/test/*.in` and `*.txt` and the
upper bound in `requirements/common.txt`. Past bumps edit only the pin
line in the compiled files; see `git log -- requirements/common.txt`.
2. Read the release notes for config validation changes, renamed auto-mapping
formats, and removed attributes; these are what break vLLM.
3. Search `tests/` for `min_transformers_version` and skip markers naming the
new version, and un-gate them.
4. Run the suites in [Testing](#testing) and fix what breaks. Mark every
shim with a `# Transformers >= X` comment so the next floor raise finds it.
Done when the [Testing](#testing) suites pass on the new pin and every test
waiting on this version is un-gated.
## Raise the floor (unvendoring pass)
### 1. Inventory
Inventory candidates with the floor version installed (no vLLM install needed):
```bash
uv run --no-project --with transformers==<floor> \
python .agents/skills/transformers-upgrade/scripts/find_upstreamed.py
```
It lists:
- vendored configs whose `model_type` upstream now owns;
- `_CONFIG_REGISTRY` overrides of upstream types;
- registered processors that collide with upstream class names or models;
- version-gated code at or below the floor;
- code that reads the installed Transformers version, whatever its threshold;
- patches citing a Transformers issue or PR (with whether a referenced PR is
in the floor release) or a future minimum version.
It finds candidates by name only. Every hit still needs a verdict.
### 2. Port each candidate to upstream
Every hit is deleted by default. A vendored copy is a liability: it drifts
from upstream, hides upstream fixes, and shadows the native class. Port the
vLLM model code to the upstream schema instead, renaming attribute reads and
deriving values from upstream fields.
Fan candidates out to parallel subagents. Each one compares the vendored and
upstream classes, then greps every attribute the vLLM model code reads.
[Upstreaming hazards](#upstreaming-hazards) lists the usual fixes.
Keep a vendored class only when upstream cannot load a checkpoint vLLM must
keep serving, and record why in the PR description. A different upstream
schema, a model-code rewrite, or a needed eval is work for the PR, not a
reason to keep.
### 3. Prove it on real checkpoints
For every checkpoint in `tests/models/registry.py` that uses a ported
config, dump what vLLM builds before and after the change, then diff the two.
Run both dumps; each catches what the other misses:
- `dump_configs.py`: the resolved config, attribute by attribute.
- `dump_model.py`: the model built on the meta device with no GPU or weights.
It records ModelConfig sizes, every parameter shape, and every module's
scalar attributes (`head_dim`, `num_kv_heads`, `scaling`, ...). Missing
fields that are read with defaults only show up here.
```bash
git worktree add --detach <scratch>/vllm-base upstream/main
S=.agents/skills/transformers-upgrade/scripts
for dump in dump_configs dump_model; do
PYTHONPATH=<scratch>/vllm-base .venv/bin/python $S/$dump.py before.jsonl <repos>
PYTHONPATH=$PWD .venv/bin/python $S/$dump.py after.jsonl <repos>
.venv/bin/python $S/diff_dumps.py before.jsonl after.jsonl
done
```
Add `--trust-remote-code` for checkpoints whose `config.json` has an
`auto_map`, and dump both ways.
How to read the diffs:
- **A `<MISSING>` config field the model reads:** this crashes or silently
falls back to a default. Port the model code to the upstream field.
- **New upstream-only defaults** (`attention_bias=False`,
`pad_token_id=None`): usually benign. A model diff of 0 confirms it.
- **Any model diff:** this is a numerics change, and the PR needs a model
eval. For example, A.X-K1 gained the checkpoint's YaRN rope scaling, which
the vendored class had been dropping; it shows up as
`layers.*.self_attn.scaling (x61)`.
### 4. Delete
For each deleted config:
- Delete the file.
- Remove its entries from `_CLASS_TO_MODULE` and `__all__` in
`vllm/transformers_utils/configs/__init__.py`.
- Remove its entries from `_CONFIG_REGISTRY` in
`vllm/transformers_utils/config.py`.
- Switch every importer, including tests, to `from transformers import ...`.
- Remove the `source_file_dependencies` lines naming the file from
`.buildkite/`.
Deleted processors follow the same steps in
`vllm/transformers_utils/processors/__init__.py`.
Before merging, rebase and grep again for the old field names: PRs that land
in between still read them.
Done when every line of the inventory output is deleted, or kept with a
reason that meets the bar in step 2. The PR also carries the config diff
results and evals for any changed values.
## Upstreaming hazards
Upstream configs are `@strict` dataclasses. Checkpoint keys upstream does not
declare survive as plain attributes, but only when the checkpoint sets them.
Defaults that only vLLM sets, and values vLLM derives, vanish.
| Hazard | Signal | Fix |
| --- | --- | --- |
| Upstream consumes legacy keys | `__post_init__` pops keys the model reads (`deepseek_v4` `compress_ratios`) | Read the upstream replacement (`layer_types`, `mlp_layer_types`, per-layer `rope_parameters`) |
| Renamed or dropped fields | Diff shows `<MISSING>` for a field the model reads | Read the upstream name; derive dropped flags from what implies them (`enable_moe_block` from `num_experts`) |
| Undeclared field without a default | `config.x` raises `AttributeError` for configs that omit `x`, e.g. minimal test configs (A.X-K1 `scoring_func`). Dumps of real checkpoints miss it | Upstream ignoring `x` usually means vLLM can stop reading it; if vLLM truly needs it, read `getattr(config, "x", <old default>)` |
| Top-level config stops forwarding | The vendored top-level config forwarded reads to `text_config`; upstream does not. `getattr(config, "x", default)` silently takes the default, e.g. on MTP draft configs from `SpeculativeConfig.hf_config_override`, which the dumps do not build | Read from `get_text_config()`; grep `getattr` reads of every field, including speculative-decoding paths |
| Type check on the config | `ctx.get_hf_config(SomeConfig)` relies on the vendored base class | Accept the upstream class too, e.g. a `Cosmos3ProcessingInfo` overriding `get_hf_config` |
| Remote code takes over | Checkpoint `config.json` has `auto_map`; with `--trust-remote-code`, the hub class replaces the native one once vLLM stops registering its own | Check the remote class loads; if not, register the upstream class for that `model_type` |
| Read-only upstream fields | Properties or derived fields break subclasses and `hf_overrides` (`nemotron_h` `hybrid_override_pattern`) | Override the source field (`layers_block_type`) instead |
| Behavioural rewrite | Upstream derives fields differently (`diffusion_gemma` `sliding_window`) | Upstream matches the reference implementation; verify with an eval |
| Original checkpoint format | Upstream targets `-hf` conversions; vLLM serves the originals (`qwen3_asr`, `qianfan_ocr`) | Keep only if upstream cannot load the original at all |
| Processor output contract | Model code consumes keys upstream does not emit | Port the model to upstream's keys |
| Name collision | A registered vendored processor shares an upstream class name and hijacks `get_processor` (`InternVLProcessor`) | Drop the registry entry |
Version gates from the inventory need a decision:
- **Code** only reachable below the floor is deleted. This covers:
- comparisons against `TRANSFORMERS_VERSION`;
- branches labelled `# Transformers < X`;
- fallbacks for fields that upstream configs now always set, e.g.
`mlp_layer_types`, `per_layer_config` or legacy `layer_types` names.
A `skipif` whose condition is now always true becomes an unconditional
`skip` with the same reason.
- **`min_transformers_version`** at or below the floor is removed.
- **`max_transformers_version`** below the floor means CI never tests that
model; report it rather than deleting it silently.
- **Patches citing a Transformers issue or PR:**
- Delete the patch when its fix is in the floor release.
- For an issue, find the PR that closed it and check that PR. A closed
issue is not a fixed one: issues get closed while the bug is still in
the release.
- Re-run the patched behaviour before deleting. A fix in the release does
not always cover vLLM's path: Emu3's processor gained BOS upstream, but
vLLM tokenizes without the processor.
- Links that only explain a design choice (e.g. a float32 RoPE note) are
not patches; leave them.
## Testing
```bash
.venv/bin/python -m pytest tests/transformers_utils tests/config tests/tokenizers_
.venv/bin/python -m pytest tests/models/multimodal/processing/test_common.py -k "<models>"
```
Add `tests/models/test_initialization.py -k "<archs>"` where a GPU is
available, and a model eval for any model whose resolved config changed.
Models in `vllm/models/` are high priority: whenever their config or model
code changes, run GSM8k (`tests/evals/gsm8k/`) and GPQA Diamond
(`tests/evals/gpt_oss/`) on `main` and the branch, even when the dumps show
no diff, and put both scores in the PR. FlashInfer autotuning picks kernel
tactics afresh on every server start, which moved DeepSeek-V4-Flash GSM8k by
about 1 point; pass `--kernel-config '{"enable_flashinfer_autotune": false}'`
to both servers so greedy outputs are comparable.
@@ -1,4 +0,0 @@
interface:
display_name: "Transformers Upgrade"
short_description: "Bump vLLM's Transformers version and delete vendored code upstream now provides."
default_prompt: "Use $transformers-upgrade to raise the minimum Transformers version and remove vendored configs, processors, and shims it makes redundant."
@@ -1,51 +0,0 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""Diff two dump_configs.py or dump_model.py outputs, per repo.
python diff_dumps.py before.jsonl after.jsonl
"""
import json
import sys
from collections import Counter
import regex as re
IGNORED = {"transformers_version", "_name_or_path", "architectures", "auto_map"}
def load(path: str) -> dict[str, dict]:
with open(path) as f:
return {record["repo"]: record for record in map(json.loads, f)}
def diff(before: dict, after: dict, prefix: str = ""):
for key in sorted(set(before) | set(after)):
if key in IGNORED:
continue
old, new = before.get(key, "<MISSING>"), after.get(key, "<MISSING>")
if isinstance(old, dict) and isinstance(new, dict) and key != "rope_parameters":
yield from diff(old, new, f"{prefix}{key}.")
elif old != new:
yield f"{prefix}{key}", json.dumps(old), json.dumps(new)
def collapse(diffs) -> Counter:
"""Merge diffs that differ only by layer index, e.g. `layers.*.scaling`."""
return Counter(
(re.sub(r"\.\d+\.", ".*.", path), old, new) for path, old, new in diffs
)
before, after = load(sys.argv[1]), load(sys.argv[2])
for repo, old in before.items():
new = after.get(repo, {"error": "missing from after"})
if "error" in old or "error" in new:
print(f"== {repo}\n before: {old.get('error', 'ok')}")
print(f" after: {new.get('error', 'ok')}")
continue
diffs = collapse(diff(old["cfg"], new["cfg"]))
print(f"== {repo}: {old['cls']} -> {new['cls']} ({sum(diffs.values())} diffs)")
for (path, old_value, new_value), count in diffs.items():
times = f" (x{count})" if count > 1 else ""
print(f" {path}{times}: before={old_value} after={new_value}")
@@ -1,30 +0,0 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""Dump the config vLLM resolves for each checkpoint, one JSON line per repo.
Run once per vLLM tree (before and after a change), then compare the two
outputs with diff_dumps.py:
PYTHONPATH=<vllm tree> .venv/bin/python dump_configs.py out.jsonl <repo>...
Pass --trust-remote-code to resolve checkpoints the way users who set it do.
"""
import json
import sys
from vllm.transformers_utils.config import get_config
out, *args = sys.argv[1:]
trust_remote_code = "--trust-remote-code" in args
with open(out, "w") as f:
for repo in (arg for arg in args if arg != "--trust-remote-code"):
try:
config = get_config(repo, trust_remote_code=trust_remote_code)
cls = f"{type(config).__module__}.{type(config).__name__}"
record = {"repo": repo, "cls": cls, "cfg": config.to_dict()}
except Exception as e:
error = f"{type(e).__name__}: {e}".splitlines()[0]
record = {"repo": repo, "error": error}
f.write(json.dumps(record, default=str, sort_keys=True) + "\n")
f.flush()
@@ -1,114 +0,0 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""Build each model on the meta device and dump what vLLM constructs.
Records ModelConfig-derived sizes, every parameter shape, and scalar
attributes of every module (head_dim, num_kv_heads, sliding_window, ...).
Needs no GPU and no weights. Run once per vLLM tree, then compare the two
outputs with diff_dumps.py:
PYTHONPATH=<vllm tree> .venv/bin/python dump_model.py out.jsonl <repo>...
Pass --trust-remote-code to resolve checkpoints the way users who set it do.
"""
import json
import socket
import sys
import torch
from vllm.config import set_current_vllm_config
from vllm.config.vllm import VllmConfig
from vllm.distributed import (
init_distributed_environment,
initialize_model_parallel,
model_parallel_is_initialized,
)
from vllm.engine.arg_utils import EngineArgs
from vllm.model_executor.model_loader.utils import initialize_model
from vllm.utils.torch_utils import set_default_torch_dtype
# Construction only: skip model runner checks that need a GPU or Triton
VllmConfig._get_v1_model_runner_unsupported_features = lambda self: [] # type: ignore[method-assign]
# Model code that moves submodules to a device must leave them on meta
_module_to = torch.nn.Module.to
def _meta_safe_to(self, *args, **kwargs):
kwargs.pop("device", None)
args = tuple(a for a in args if not isinstance(a, (torch.device, str, int)))
return _module_to(self, *args, **kwargs)
torch.nn.Module.to = _meta_safe_to # type: ignore[method-assign]
SCALARS = (bool, int, float, str)
def module_attrs(model: torch.nn.Module) -> dict[str, dict]:
attrs = {}
for name, module in model.named_modules():
scalars = {
key: value
for key, value in vars(module).items()
if not key.startswith("_") and isinstance(value, SCALARS)
}
if scalars:
attrs[name or "<root>"] = scalars
return attrs
def dump(repo: str, trust_remote_code: bool) -> dict:
args = EngineArgs(
model=repo,
load_format="dummy",
enforce_eager=True,
max_model_len=4096,
trust_remote_code=trust_remote_code,
)
vllm_config = args.create_engine_config()
model_config = vllm_config.model_config
with set_current_vllm_config(vllm_config):
if not model_parallel_is_initialized():
with socket.socket() as sock:
sock.bind(("", 0))
port = sock.getsockname()[1]
init_distributed_environment(1, 0, f"tcp://127.0.0.1:{port}", 0, "gloo")
initialize_model_parallel(1, 1)
with set_default_torch_dtype(model_config.dtype), torch.device("meta"):
model = initialize_model(vllm_config=vllm_config)
return {
"repo": repo,
"cls": type(model).__name__,
"cfg": {
"model_config": {
"head_size": model_config.get_head_size(),
"total_num_kv_heads": model_config.get_total_num_kv_heads(),
"sliding_window": model_config.get_sliding_window(),
"num_layers": model_config.get_num_layers(vllm_config.parallel_config),
},
"params": {n: list(p.shape) for n, p in model.named_parameters()},
"modules": module_attrs(model),
},
}
def main() -> None:
out, *args = sys.argv[1:]
trust_remote_code = "--trust-remote-code" in args
repos = [arg for arg in args if arg != "--trust-remote-code"]
with open(out, "w") as f:
for repo in repos:
try:
record = dump(repo, trust_remote_code)
except Exception as e:
error = f"{type(e).__name__}: {e}".splitlines()[0]
record = {"repo": repo, "error": error}
f.write(json.dumps(record, default=str, sort_keys=True) + "\n")
f.flush()
if __name__ == "__main__":
main()
@@ -1,147 +0,0 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""List vendored vLLM code that the installed Transformers may make redundant.
Run with the Transformers version under test, from the vLLM repo root:
uv run --no-project --with transformers==<version> \
python .agents/skills/transformers-upgrade/scripts/find_upstreamed.py
Imports nothing from vLLM, so no vLLM install is needed. Uses the `gh` CLI, if
available, to check whether referenced Transformers PRs are in this release.
"""
import json
import subprocess
import sys
from pathlib import Path
import regex as re
import transformers
from packaging.version import Version
from transformers.models.auto.configuration_auto import CONFIG_MAPPING_NAMES
from transformers.models.auto.processing_auto import PROCESSOR_MAPPING_NAMES
ROOT = Path(sys.argv[1] if len(sys.argv) > 1 else ".")
FLOOR = Version(transformers.__version__)
UPSTREAM_NAMES = set(dir(transformers))
MODEL_TYPE = re.compile(r'model_type\s*(?::\s*str\s*)?=\s*"([^"]+)"')
REGISTRY_KEY = re.compile(r'^\s+(?:\*\*\{)?"?([\w-]+)"?(?:=|": )"(\w+)"', re.M)
CLASS_KEY = re.compile(r'^\s+"(\w+)": ', re.M)
UPSTREAM_REF = re.compile(
r"huggingface/transformers(?:/(?:issues|pull)/|#)(\d+)"
r"|(?:once|until|when) [Tt]ransformers [\d.]+ is the minimum"
)
VERSIONED = re.compile(
r"(?:min_transformers_version|max_transformers_version|check_version|Version)"
r'\(?\s*=?\s*"([45]\.\d+(?:\.\d+)?)[^"]*"'
r"|[Tt]ransformers\s*(?:v|[<>]=?\s*)([45]\.\d+(?:\.\d+)?|4)\b"
)
READS_VERSION = re.compile(
r"transformers import __version__|transformers\.__version__"
r"|version\([\"']transformers[\"']\)|TRANSFORMERS_VERSION|check_version\("
)
def section(title: str) -> None:
print(f"\n## {title}")
def vendored_configs() -> None:
section(f"Vendored configs whose model_type is upstream in {FLOOR}")
files = sorted(ROOT.glob("vllm/transformers_utils/configs/*.py"))
files += sorted(ROOT.glob("vllm/models/*/config*.py"))
files += sorted(ROOT.glob("vllm/models/*/configs/*.py"))
for path in files:
types = sorted(set(MODEL_TYPE.findall(path.read_text())))
hits = [t for t in types if t in CONFIG_MAPPING_NAMES]
if hits:
upstream = ", ".join(f"{t} -> {CONFIG_MAPPING_NAMES[t]}" for t in hits)
print(f"- {path.relative_to(ROOT)}: {upstream}")
registry = (ROOT / "vllm/transformers_utils/config.py").read_text()
start = registry.index("_CONFIG_REGISTRY")
body = registry[start : registry.index("\n)\n", start)]
section("_CONFIG_REGISTRY entries that override an upstream model_type")
for model_type, cls in REGISTRY_KEY.findall(body):
if model_type in CONFIG_MAPPING_NAMES:
print(f"- {model_type}: vLLM {cls} over {CONFIG_MAPPING_NAMES[model_type]}")
def vendored_processors() -> None:
init = (ROOT / "vllm/transformers_utils/processors/__init__.py").read_text()
names = sorted(set(CLASS_KEY.findall(init)))
section("Registered processor names that collide with upstream classes")
for name in names:
if name in UPSTREAM_NAMES:
print(f"- {name}")
section("Processor files named after a model_type with an upstream processor")
for path in sorted(ROOT.glob("vllm/transformers_utils/processors/*.py")):
if path.stem in PROCESSOR_MAPPING_NAMES:
upstream = PROCESSOR_MAPPING_NAMES[path.stem]
print(f"- {path.relative_to(ROOT)} -> {upstream}")
def version_gates() -> None:
files = sorted([*ROOT.glob("vllm/**/*.py"), *ROOT.glob("tests/**/*.py")])
section(f"Version-gated code at or below {FLOOR} (review each)")
for path in files:
for lineno, line in enumerate(path.read_text().splitlines(), 1):
for match in VERSIONED.finditer(line):
version = Version(match.group(1) or match.group(2))
if version <= FLOOR:
print(f"- {path.relative_to(ROOT)}:{lineno}: {line.strip()}")
break
section("Code that reads the installed Transformers version (review each)")
for path in files:
for lineno, line in enumerate(path.read_text().splitlines(), 1):
if READS_VERSION.search(line):
print(f"- {path.relative_to(ROOT)}:{lineno}: {line.strip()}")
def upstream_status(number: str) -> str:
"""Whether Transformers issue/PR `number` is resolved in this release."""
def gh(path: str) -> dict | None:
try:
out = subprocess.run(["gh", "api", path], capture_output=True, text=True)
except FileNotFoundError:
return None
return json.loads(out.stdout) if out.returncode == 0 else None
repo = "repos/huggingface/transformers"
if (issue := gh(f"{repo}/issues/{number}")) is None:
return "status unknown"
if "pull_request" not in issue:
return f"issue {issue['state']}, check which PR fixed it"
pr = gh(f"{repo}/pulls/{number}")
if pr is None or not pr["merged_at"]:
return "PR not merged"
compare = gh(f"{repo}/compare/v{FLOOR}...{pr['merge_commit_sha']}")
included = compare and compare["status"] in ("behind", "identical")
return f"PR merged, {'in' if included else 'not in'} v{FLOOR}"
def upstream_refs() -> None:
section("Patches referencing Transformers issues/PRs or a future minimum")
files = sorted([*ROOT.glob("vllm/**/*.py"), *ROOT.glob("tests/**/*.py")])
statuses: dict[str, str] = {}
for path in files:
for lineno, line in enumerate(path.read_text().splitlines(), 1):
if (match := UPSTREAM_REF.search(line)) is None:
continue
note = ""
if number := match.group(1):
if number not in statuses:
statuses[number] = upstream_status(number)
note = f" [#{number}: {statuses[number]}]"
print(f"- {path.relative_to(ROOT)}:{lineno}:{note} {line.strip()}")
if __name__ == "__main__":
print(f"# Transformers {FLOOR}")
vendored_configs()
vendored_processors()
version_gates()
upstream_refs()
+1
View File
@@ -17,6 +17,7 @@ ALLOWED_UNSTABLE_LIBRARIES: tuple[str, ...] = (
# _C_stable_libtorch, but these two modules still bind libtorch directly.
"_C.abi3.so",
"_rocm_C.abi3.so",
"third_party/deep_gemm/_C*.so",
)
@@ -56,6 +56,7 @@ steps:
- "!vllm/distributed/device_communicators/shm_object_storage.py"
- "!vllm/distributed/device_communicators/symm_mem.py"
- "!vllm/distributed/device_communicators/mnnvl_compat.py"
- "!vllm/distributed/device_communicators/ray_communicator.py"
- "!vllm/third_party/flashmla/"
- "!vllm/third_party/pynvml.py"
- "!vllm/_custom_ops.py"
+1 -59
View File
@@ -20,10 +20,8 @@ steps:
- tests/kernels/moe/test_cpu_fused_moe.py
- tests/kernels/moe/test_cpu_quant_fused_moe.py
- tests/kernels/moe/test_zen_cpu_int8_moe.py
- tests/test_zentorch_version_pin.py
- vllm/model_executor/layers/fused_moe/experts/cpu_moe.py
- tests/kernels/test_onednn.py
- tests/kernels/test_cpu_fused_sampling.py
- tests/kernels/test_awq_int4_to_int8.py
- tests/kernels/quantization/test_cpu_fp8_scaled_mm.py
- tests/kernels/quantization/test_zen_da8w4.py
@@ -47,10 +45,8 @@ steps:
tests/kernels/moe/test_cpu_fused_moe.py \
tests/kernels/moe/test_cpu_quant_fused_moe.py \
tests/kernels/moe/test_zen_cpu_int8_moe.py \
tests/test_zentorch_version_pin.py \
tests/kernels/mamba/test_cpu_short_conv.py \
tests/kernels/test_onednn.py \
tests/kernels/test_cpu_fused_sampling.py \
tests/kernels/test_awq_int4_to_int8.py \
tests/kernels/quantization/test_cpu_fp8_scaled_mm.py \
tests/kernels/quantization/test_zen_da8w4.py \
@@ -89,7 +85,7 @@ steps:
- |
bash .buildkite/scripts/hardware_ci/run-cpu-test.sh 30m "
pytest -x -v -s \
tests/v1/worker/test_gpu_target_dependent_ar_speculator.py \
tests/v1/worker/test_gpu_autoregressive_speculator.py \
tests/v1/worker/test_gpu_extract_hidden_states_speculator.py \
tests/v1/worker/test_gpu_batch_ordering.py \
tests/v1/worker/test_gpu_rejection_sampler_chunking.py \
@@ -232,57 +228,3 @@ steps:
"$$BUILDKITE_PARALLEL_JOB"
"$REGISTRY/$REPO:$BUILDKITE_COMMIT-arm64-cpu"
parallelism: 3
- label: ":docker: Build Zen CPU image"
key: zen-cpu-image-build
# Reuse the shared `<repo>:<commit>-cpu` image that image-build-cpu publishes
# to public ECR, so Dockerfile.zen just layers on top instead of recompiling
# vLLM from source. If the pull fails, the build script still falls back to
# building the CPU base from source.
depends_on:
- image-build-cpu
soft_fail: true
agents:
queue: amd-cpu
source_file_dependencies:
- .buildkite/hardware_tests/cpu.yaml
- .buildkite/image_build/image_build_zen_cpu.sh
- docker/Dockerfile.zen
- docker/Dockerfile.cpu
- requirements/cpu.txt
- requirements/build/cpu.txt
- requirements/test/cpu.txt
- cmake/cpu_extension.cmake
- vllm/model_executor/layers/utils.py
- vllm/model_executor/kernels/linear/zentorch_utils.py
- vllm/model_executor/kernels/linear/scaled_mm/zentorch.py
- vllm/model_executor/kernels/linear/mixed_precision/zentorch.py
- vllm/platforms/cpu.py
- vllm/platforms/zen_cpu.py
- vllm/platforms/__init__.py
- tests/model_executor/test_cpu_unquantized_gemm_dispatch.py
- tests/test_zen_cpu_platform_detection.py
- tests/kernels/moe/test_zen_cpu_fused_moe.py
- tests/kernels/moe/test_zen_cpu_int8_moe.py
- vllm/v1/attention/backends/zentorch_sdpa.py
- tests/test_zentorch_version_pin.py
- vllm/v1/attention/backends/mla/prefill/zen_cpu_sdpa.py
- vllm/model_executor/layers/fused_moe/experts/zentorch_moe.py
- tests/kernels/quantization/test_zen_da8w4.py
commands:
- .buildkite/image_build/image_build_zen_cpu.sh "$REGISTRY" "$REPO" "$BUILDKITE_COMMIT"
env:
DOCKER_BUILDKIT: "1"
# Install zentorch directly (if-branch in Dockerfile.zen) instead of
# `uv pip install "vllm[zen]"`, which forces uv to re-resolve vLLM's full
# dependency closure and chokes on its transitive triton-cpu URL dependency.
# Keep this pin in sync with the `zen` extra in setup.py.
ZENTORCH_VERSION: "2.14.0.1"
retry:
automatic:
- exit_status: 1 # Transient Docker/BuildKit failure
limit: 1
- exit_status: -1 # Agent was lost
limit: 2
- exit_status: -10 # Agent was lost
limit: 2
+4 -9
View File
@@ -2,15 +2,10 @@ group: Hardware
steps:
- label: "GH200 Test"
key: gh200-test
timeout_in_minutes: 30
soft_fail: true
device: gh200
no_plugin: true
optional: true
working_dir: "/vllm-workspace"
depends_on:
- arm64-image-build
env:
VLLM_WORKER_MULTIPROC_METHOD: "spawn"
commands:
- nvidia-smi
- python3 examples/basic/offline_inference/generate.py --model meta-llama/Llama-3.2-1B
commands:
- nvidia-smi
- bash .buildkite/scripts/hardware_ci/run-gh200-test.sh
@@ -15,7 +15,7 @@ steps:
limit: 2
- exit_status: -10 # Agent was lost
limit: 2
- label: ":intel: (BMG) Example Test"
- label: "XPU example Test"
key: xpu-example-test
depends_on:
- image-build-xpu
@@ -37,7 +37,7 @@ steps:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'bash .buildkite/scripts/hardware_ci/run-intel-ci-test.sh example'
- label: ":intel: (BMG) V1 test"
- label: "XPU V1 test"
key: xpu-v1-test
depends_on:
- image-build-xpu
@@ -60,7 +60,7 @@ steps:
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'bash .buildkite/scripts/hardware_ci/run-intel-ci-test.sh v1'
parallelism: 2
- label: ":intel: (BMG) W8A8 FP8 Linear Examples"
- label: "XPU W8A8 FP8 Linear Examples"
key: xpu-w8a8-fp8-linear-examples
depends_on:
- image-build-xpu
@@ -83,7 +83,7 @@ steps:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'bash .buildkite/scripts/hardware_ci/run-intel-ci-test.sh w8a8-fp8-linear'
- label: ":intel: (BMG) Server test"
- label: "XPU server test"
key: xpu-server-test
depends_on:
- image-build-xpu
@@ -105,7 +105,7 @@ steps:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'bash .buildkite/scripts/hardware_ci/run-intel-ci-test.sh server'
- label: ":intel: (BMG) Quantization test"
- label: "XPU quantization test"
key: xpu-quantization-test
depends_on:
- image-build-xpu
@@ -127,7 +127,7 @@ steps:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'bash .buildkite/scripts/hardware_ci/run-intel-ci-test.sh quantization'
- label: ":intel: (BMG) Compressed tensors FP8 test"
- label: "XPU compressed tensors FP8 test"
key: xpu-compressed-tensors-fp8-test
depends_on:
- image-build-xpu
@@ -150,7 +150,7 @@ steps:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'bash .buildkite/scripts/hardware_ci/run-intel-ci-test.sh compressed-tensors-fp8'
- label: ":intel: (BMG) Graph"
- label: "XPU Graph"
key: xpu-graph
depends_on:
- image-build-xpu
@@ -61,8 +61,6 @@ NIGHTLY_UBUNTU_VERSION="24.04"
NIGHTLY_FINAL_BASE_IMAGE="nvidia/cuda:${NIGHTLY_CUDA_VERSION}-base-ubuntu${NIGHTLY_UBUNTU_VERSION}"
echo "--- :docker: Building torch nightly image (CUDA ${NIGHTLY_CUDA_VERSION})"
# Provenance labels match the bake `_labels` target in docker/docker-bake.hcl;
# the initialized-snapshot E2E refuses images whose commit labels differ.
docker buildx build --file docker/Dockerfile \
--build-arg max_jobs=16 \
--build-arg buildkite_commit="$BUILDKITE_COMMIT" \
@@ -72,10 +70,6 @@ docker buildx build --file docker/Dockerfile \
--build-arg BUILD_BASE_IMAGE="${NIGHTLY_BUILD_BASE_IMAGE}" \
--build-arg FINAL_BASE_IMAGE="${NIGHTLY_FINAL_BASE_IMAGE}" \
--build-arg torch_cuda_arch_list="8.0 8.9 9.0 10.0 12.0" \
--label "org.opencontainers.image.revision=${BUILDKITE_COMMIT}" \
--label "ai.vllm.build.commit=${BUILDKITE_COMMIT}" \
--label "ai.vllm.build.pipeline=${BUILDKITE_PIPELINE_SLUG:-local}" \
--label "ai.vllm.build.url=${BUILDKITE_BUILD_URL:-}" \
--tag "$IMAGE_TAG" \
--push \
--target test \
@@ -1,75 +0,0 @@
#!/bin/bash
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
#
# Build the AMD Zen CPU image (vLLM + zentorch) by layering docker/Dockerfile.zen
# on top of a CPU base image that already has vLLM installed.
#
# To avoid recompiling vLLM from source, prefer reusing the shared CPU image that
# the image-build-cpu step already builds and publishes for this commit
# (`<repo>:<commit>-cpu`). It lives in public ECR, which allows anonymous pulls,
# so no registry credentials are required. If the pull fails (e.g. the image
# isn't published yet), fall back to building the base from docker/Dockerfile.cpu.
#
# The zen image is not pushed to a registry.
#
# See docker/Dockerfile.zen for the build workflow this mirrors.
set -euo pipefail
if [[ $# -lt 3 ]]; then
echo "Usage: $0 <registry> <repo> <commit>"
exit 1
fi
REGISTRY=$1
REPO=$2
BUILDKITE_COMMIT=$3
image_repo() {
if [[ -n "${REGISTRY}" ]]; then
printf '%s/%s' "${REGISTRY}" "${REPO}"
else
printf '%s' "${REPO}"
fi
}
IMAGE_REPO="$(image_repo)"
# The shared CPU image published by image-build-cpu (public ECR, anonymous pull).
SHARED_CPU_IMAGE=""
if [[ -n "${REGISTRY}" ]]; then
SHARED_CPU_IMAGE="$REGISTRY/$REPO:$BUILDKITE_COMMIT-cpu"
fi
FALLBACK_BASE_IMAGE="zen-cpu-base:$BUILDKITE_COMMIT"
IMAGE="$IMAGE_REPO:$BUILDKITE_COMMIT-zen-cpu"
# ZENTORCH_VERSION is optional; when unset the Dockerfile falls back to
# installing zentorch via `vllm[zen]`.
ZENTORCH_VERSION=${ZENTORCH_VERSION:-}
# Step 1: obtain the CPU base image that Dockerfile.zen layers on. Prefer pulling
# the published `-cpu` image; fall back to building it from source if unavailable.
if [[ -n "${SHARED_CPU_IMAGE}" ]] && docker pull "$SHARED_CPU_IMAGE"; then
echo "--- :docker: Using published CPU image as base: $SHARED_CPU_IMAGE"
BASE_IMAGE="$SHARED_CPU_IMAGE"
else
echo "--- :docker: Published CPU image unavailable; building base from source"
docker build --file docker/Dockerfile.cpu \
--platform linux/amd64 \
--build-arg buildkite_commit="$BUILDKITE_COMMIT" \
--build-arg VLLM_CPU_X86=true \
--tag "$FALLBACK_BASE_IMAGE" \
--target vllm-openai \
--progress plain .
BASE_IMAGE="$FALLBACK_BASE_IMAGE"
fi
# Step 2: build the zen test image on top of the CPU base.
echo "--- :docker: Building Zen test image"
docker build --file docker/Dockerfile.zen \
--platform linux/amd64 \
--build-arg BASE_IMAGE="$BASE_IMAGE" \
${ZENTORCH_VERSION:+--build-arg ZENTORCH_VERSION="$ZENTORCH_VERSION"} \
--tag "$IMAGE" \
--target vllm-zen-test \
--progress plain .
@@ -1,8 +1,8 @@
group: Basic Correctness Intel
group: Basic Correctness
depends_on:
- image-build-xpu
steps:
- label: ":intel: (BMG) Sleep Mode"
- label: XPU Sleep Mode
key: xpu-sleep-mode
timeout_in_minutes: 45
device: intel_gpu
+2 -2
View File
@@ -1,8 +1,8 @@
group: Benchmarks Intel
group: Benchmarks
depends_on:
- image-build-xpu
steps:
- label: ":intel: (BMG) Benchmarks CLI"
- label: Benchmarks CLI Test
key: benchmarks-cli-test
timeout_in_minutes: 40
device: intel_gpu
@@ -1,33 +0,0 @@
group: Distributed Intel
depends_on:
- image-build-xpu
steps:
- label: ":intel: (BMG) Distributed Torchrun + Shutdown"
key: distributed-torchrun-shutdown-tests-2-gpus
timeout_in_minutes: 50
device: intel_gpu
agent_tags:
label: production
gpu: 2+
mem: 24+
no_plugin: true
working_dir: "."
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
VLLM_TEST_DEVICE: "xpu"
source_file_dependencies:
- vllm/distributed/
- "!vllm/distributed/kv_transfer/"
- vllm/engine/
- vllm/executor/
- vllm/worker/worker_base.py
- vllm/v1/engine/
- vllm/v1/worker/
- tests/distributed/
- tests/v1/shutdown
- tests/v1/worker/test_worker_memory_snapshot.py
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests && pytest -v -s v1/shutdown'
+3 -3
View File
@@ -2,7 +2,7 @@ group: Engine Intel
depends_on:
- image-build-xpu
steps:
- label: ":intel: (BMG) Engine"
- label: Engine
key: engine
timeout_in_minutes: 40
device: intel_gpu
@@ -40,7 +40,7 @@ steps:
'cd tests &&
pytest -v -s engine/test_arg_utils.py test_sequence.py test_logger.py test_vllm_port.py jit_monitor/test_hooks.py'
- label: ":intel: (BMG) Engine (1 GPU)"
- label: Engine (1 GPU)
key: engine-1-gpu
timeout_in_minutes: 30
device: intel_gpu
@@ -65,7 +65,7 @@ steps:
pytest -v -s v1/engine --ignore v1/engine/test_preprocess_error_handling.py --ignore v1/engine/test_core_engine_actor_manager.py &&
pytest -v -s test_config.py'
- label: ":intel: (BMG) V1 e2e (2 GPUs)"
- label: V1 e2e (2 GPUs)
key: v1-e2e-2-gpus
timeout_in_minutes: 30
device: intel_gpu
+6 -9
View File
@@ -1,4 +1,4 @@
group: Entrypoints Intel
group: Entrypoints
depends_on:
- image-build-xpu
steps:
@@ -19,14 +19,14 @@ steps:
source_file_dependencies:
- vllm/entrypoints
- tests/entrypoints/unit_tests
- tests/entrypoints/rl/weight_transfer
- tests/entrypoints/weight_transfer
- tests/entrypoints/launchers
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests &&
pytest -v -s entrypoints/unit_tests &&
pytest -v -s entrypoints/rl/weight_transfer'
pytest -v -s entrypoints/weight_transfer'
- label: ":intel: (BMG) Entrypoints Integration (LLM)"
key: entrypoints-integration-llm-intel
@@ -71,17 +71,14 @@ steps:
- vllm/
- "!vllm/distributed/kv_transfer/"
- tests/entrypoints/serve
- tests/entrypoints/rl
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'export VLLM_WORKER_MULTIPROC_METHOD=spawn &&
cd tests &&
pytest -v -s entrypoints/serve --ignore=entrypoints/serve/dev/rpc &&
pytest -v -s entrypoints/rl --ignore=entrypoints/rl/weight_transfer
--ignore=entrypoints/rl/return_routed_experts/test_openai_api.py
--ignore=entrypoints/rl/weight_transfer_metrics
--deselect="tests/entrypoints/rl/state_transitions/test_pause_resume.py::TestPauseResume::test_clear_cache_preserves_output_and_controls_prefix_cache[MRV1]" &&
pytest -v -s entrypoints/serve --ignore=entrypoints/serve/dev/rpc
--ignore=entrypoints/serve/dev/rlhf/test_weight_operation_metrics.py
--deselect="tests/entrypoints/serve/dev/rlhf/state_transitions/test_pause_resume.py::TestPauseResume::test_clear_cache_preserves_output_and_controls_prefix_cache[MRV1]" &&
PYTHONPATH=/workspace/vllm pytest -v -s entrypoints/serve/dev/rpc'
- label: ":intel: (BMG) Entrypoints Integration (API Server Scale Out)"
@@ -1,8 +1,8 @@
group: Expert Parallelism Intel
group: Expert Parallelism
depends_on:
- image-build-xpu
steps:
- label: ":intel: (BMG) EPLB Algorithm"
- label: EPLB Algorithm
key: eplb-algorithm
timeout_in_minutes: 45
device: intel_gpu
+2 -2
View File
@@ -2,7 +2,7 @@ group: Kernels Intel
depends_on:
- image-build-xpu
steps:
- label: ":intel: (BMG) vLLM IR Tests"
- label: vLLM IR Tests
key: vllm-ir-tests
timeout_in_minutes: 30
device: intel_gpu
@@ -26,7 +26,7 @@ steps:
pytest -v -s ir &&
pytest -v -s kernels/ir'
- label: ":intel: (BMG) UVA Kernel Tests"
- label: UVA Kernel Tests
key: uva-kernel-tests
timeout_in_minutes: 20
device: intel_gpu
+3 -3
View File
@@ -1,8 +1,8 @@
group: LM Eval Intel
group: LM Eval
depends_on:
- image-build-xpu
steps:
- label: ":intel: (BMG) GPQA Eval (GPT-OSS)"
- label: "XPU GPQA Eval (GPT-OSS)"
key: xpu-gpqa-eval-gpt-oss
timeout_in_minutes: 60
device: intel_gpu
@@ -25,7 +25,7 @@ steps:
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests &&
pytest -s -v evals/gpt_oss/test_gpqa_correctness.py --config-list-file=configs/models-xpu.txt'
- label: ":intel: (BMG) LM Eval Large Models(2 GPUs)"
- label: LM Eval Large Models(2 GPUs)
key: lm-eval-large-models-2-gpus
timeout_in_minutes: 45
device: intel_gpu
+36 -59
View File
@@ -2,14 +2,14 @@ group: LoRA Intel
depends_on:
- image-build-xpu
steps:
- label: ":intel: (BMG) LoRA Runtime + Utils"
- label: LoRA Runtime + Utils
key: lora-runtime-utils
timeout_in_minutes: 45
device: intel_gpu
agent_tags:
label: production
gpu: 1+
mem: 16+
mem: 24+
no_plugin: true
working_dir: "."
env:
@@ -33,9 +33,10 @@ steps:
pytest -v -s lora/test_peft_helper.py &&
pytest -v -s lora/test_resolver.py &&
pytest -v -s lora/test_utils.py &&
pytest -v -s lora/test_add_lora.py &&
pytest -v -s lora/test_worker.py'
- label: ":intel: (BMG) LoRA Fused/MoE Kernels"
- label: LoRA Fused/MoE Kernels
key: lora-fused-moe-kernels
timeout_in_minutes: 45
device: intel_gpu
@@ -60,7 +61,7 @@ steps:
pytest -v -s lora/test_fused_moe_lora_kernel.py &&
pytest -v -s lora/test_moe_lora_align_sum.py --deselect="tests/lora/test_moe_lora_align_sum.py::test_moe_lora_align_block_size_mixed_base_and_lora[1]"'
- label: ":intel: (BMG) LoRA Punica Kernels"
- label: LoRA Punica Kernels
key: lora-punica-kernels
timeout_in_minutes: 45
device: intel_gpu
@@ -87,7 +88,7 @@ steps:
pytest -v -s lora/test_punica_ops.py::test_kernels_hidden_size &&
pytest -v -s lora/test_punica_ops.py::test_add_lora_fused_moe_early_exit'
- label: ":intel: (BMG) LoRA Punica FP8/XPU Ops"
- label: LoRA Punica FP8/XPU Ops
key: lora-punica-fp8-xpu-ops
timeout_in_minutes: 60
device: intel_gpu
@@ -112,61 +113,10 @@ steps:
pytest -v -s lora/test_punica_ops_fp8.py &&
pytest -v -s lora/test_punica_xpu_ops.py'
- label: ":intel: (BMG) LoRA Models"
- label: LoRA Models
key: lora-models
timeout_in_minutes: 45
device: intel_gpu
agent_tags:
label: production
gpu: 2+
mem: 16+
no_plugin: true
working_dir: "."
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
VLLM_TEST_DEVICE: "xpu"
source_file_dependencies:
- vllm/lora
- tests/lora
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests &&
export VLLM_WORKER_MULTIPROC_METHOD=spawn &&
pytest -v -s lora/test_quant_model.py --deselect="tests/lora/test_quant_model.py::test_quant_model_lora[model0]" --deselect="tests/lora/test_quant_model.py::test_quant_model_lora[model1]" --deselect="tests/lora/test_quant_model.py::test_quant_model_tp_equality[model0]" &&
pytest -v -s lora/test_transformers_model.py &&
pytest -v -s lora/test_chatglm3_tp.py &&
pytest -v -s lora/test_llama_tp.py::test_llama_lora'
- label: ":intel: (BMG) LoRA Multimodal"
key: lora-multimodal
timeout_in_minutes: 45
device: intel_gpu
agent_tags:
label: production
gpu: 1+
mem: 16+
no_plugin: true
working_dir: "."
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
VLLM_TEST_DEVICE: "xpu"
source_file_dependencies:
- vllm/lora
- tests/lora
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests &&
export VLLM_WORKER_MULTIPROC_METHOD=spawn &&
pytest -v -s lora/test_whisper.py'
- label: ":intel: (BMG) LoRA Add + Multimodal"
key: lora-add-multimodal
timeout_in_minutes: 45
device: intel_gpu
agent_tags:
label: production
gpu: 2+
@@ -185,6 +135,33 @@ steps:
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests &&
export VLLM_WORKER_MULTIPROC_METHOD=spawn &&
pytest -v -s lora/test_add_lora.py &&
pytest -v -s lora/test_default_mm_loras.py &&
pytest -v -s lora/test_quant_model.py --deselect="tests/lora/test_quant_model.py::test_quant_model_lora[model0]" --deselect="tests/lora/test_quant_model.py::test_quant_model_lora[model1]" --deselect="tests/lora/test_quant_model.py::test_quant_model_tp_equality[model0]" &&
pytest -v -s lora/test_transformers_model.py &&
pytest -v -s lora/test_chatglm3_tp.py &&
pytest -v -s lora/test_llama_tp.py::test_llama_lora &&
pytest -s -v lora/test_minicpmv_tp.py'
- label: LoRA Multimodal
key: lora-multimodal
timeout_in_minutes: 45
device: intel_gpu
agent_tags:
label: production
gpu: 1+
mem: 24+
no_plugin: true
working_dir: "."
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
VLLM_TEST_DEVICE: "xpu"
source_file_dependencies:
- vllm/lora
- tests/lora
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests &&
export VLLM_WORKER_MULTIPROC_METHOD=spawn &&
pytest -v -s lora/test_default_mm_loras.py &&
pytest -v -s lora/test_whisper.py'
+20 -90
View File
@@ -2,7 +2,7 @@ group: Miscellaneous Intel
depends_on:
- image-build-xpu
steps:
- label: ":intel: (BMG) Batch Invariance"
- label: XPU Batch Invariance
key: xpu-batch-invariance
timeout_in_minutes: 60
num_devices: 2
@@ -10,7 +10,7 @@ steps:
agent_tags:
label: production
gpu: 2+
mem: 16+
mem: 24+
no_plugin: true
working_dir: "."
env:
@@ -30,44 +30,10 @@ steps:
pytest -v -s v1/determinism/test_xpu_batch_invariant_ut.py &&
bash ../.buildkite/scripts/hardware_ci/run-xpu-batch-invariance.sh'
- label: ":intel: (BMG) V1 Core + KV + Metrics"
- label: V1 Core + KV + Metrics
key: v1-core-kv-metrics-intel
timeout_in_minutes: 45
device: intel_gpu
agent_tags:
label: production
gpu: 1+
mem: 16+
no_plugin: true
working_dir: "."
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
VLLM_TEST_DEVICE: "xpu"
source_file_dependencies:
- vllm/
- tests/v1/core
- tests/v1/executor
- tests/v1/kv_offload
- tests/v1/worker
- tests/v1/kv_connector/unit
- tests/v1/ec_connector/unit
- tests/v1/metrics
- tests/entrypoints/openai/correctness/test_lmeval.py
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'pip install -r requirements/kv_connectors.txt &&
export VLLM_WORKER_MULTIPROC_METHOD=spawn &&
cd tests &&
pytest -v -s v1/executor &&
VLLM_BATCH_INVARIANT=1 pytest -v -s -m "not cpu_test" v1/ec_connector/unit
--ignore=v1/ec_connector/unit/test_ec_cpu_connector.py'
- label: ":intel: (BMG) EC CPU Connector (XPU)"
key: ec-cpu-connector-xpu
timeout_in_minutes: 45
device: intel_gpu
agent_tags:
label: production
gpu: 1+
@@ -94,16 +60,17 @@ steps:
'pip install -r requirements/kv_connectors.txt &&
export VLLM_WORKER_MULTIPROC_METHOD=spawn &&
cd tests &&
pytest -v -s v1/ec_connector/unit/test_ec_cpu_connector.py'
pytest -v -s v1/executor &&
VLLM_BATCH_INVARIANT=1 pytest -v -s -m "not cpu_test" v1/ec_connector/unit'
- label: ":intel: (BMG) V1 Sample + Logits"
- label: V1 Sample + Logits
key: v1-sample-logits-intel
timeout_in_minutes: 90
device: intel_gpu
agent_tags:
label: production
gpu: 1+
mem: 16+
mem: 24+
no_plugin: true
working_dir: "."
env:
@@ -141,7 +108,7 @@ steps:
limit: 1
parallelism: 4
- label: ":intel: (BMG) Basic Models Tests (Initialization)"
- label: Basic Models Tests (Initialization)
key: basic-models-tests-initialization
timeout_in_minutes: 60
device: intel_gpu
@@ -166,7 +133,7 @@ steps:
cd tests &&
pytest -v -s models/test_initialization.py::test_can_initialize_large_subset[Eagle3MiniMaxM2ForCausalLM]'
- label: ":intel: (BMG) CPU Offload"
- label: XPU CPU Offload
key: xpu-cpu-offload
timeout_in_minutes: 60
device: intel_gpu
@@ -194,7 +161,7 @@ steps:
pytest -v -s v1/kv_offload &&
pytest -v -s v1/kv_connector/unit/test_offloading_connector.py'
- label: ":intel: (BMG) NixlConnector PD accuracy (4 GPUs)"
- label: NixlConnector PD accuracy (4 GPUs)
key: nixlconnector-pd-accuracy-4-gpus
timeout_in_minutes: 60
num_devices: 4
@@ -223,7 +190,7 @@ steps:
PREFILLER_TP_SIZE=1 DECODER_TP_SIZE=2 bash v1/kv_connector/nixl_integration/run_xpu_disagg_accuracy_test.sh &&
PREFILLER_TP_SIZE=2 DECODER_TP_SIZE=2 bash v1/kv_connector/nixl_integration/run_xpu_disagg_accuracy_test.sh'
- label: ":intel: (BMG) Regression"
- label: Regression
key: regression
timeout_in_minutes: 50
device: intel_gpu
@@ -256,7 +223,7 @@ steps:
'cd tests &&
pytest -v -s test_regression.py'
- label: ":intel: (BMG) Metrics, Tracing (2 GPUs)"
- label: Metrics, Tracing (2 GPUs)
key: metrics-tracing-2-gpus
timeout_in_minutes: 45
num_devices: 2
@@ -291,49 +258,10 @@ steps:
'cd tests &&
pytest -v -s v1/tracing'
- label: ":intel: (BMG) Async Engine, Inputs, Utils, Worker"
- label: Async Engine, Inputs, Utils, Worker
key: async-engine-inputs-utils-worker
timeout_in_minutes: 55
device: intel_gpu
agent_tags:
label: production
gpu: 1+
mem: 16+
no_plugin: true
working_dir: "."
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
VLLM_TEST_DEVICE: "xpu"
source_file_dependencies:
- vllm/assets/
- vllm/config/
- vllm/distributed/
- vllm/engine/
- vllm/inputs/
- vllm/model_executor/
- vllm/multimodal/
- vllm/platforms/
- vllm/sampling_params.py
- vllm/tokenizers/
- vllm/transformers_utils/
- vllm/utils/
- vllm/v1/
- tests/detokenizer
- tests/multimodal
- tests/utils_
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests &&
pytest -v -s detokenizer --ignore=detokenizer/test_stop_strings.py &&
pytest -v -s -m "not cpu_test" ./multimodal &&
pytest -v -s utils_ --ignore=utils_/test_mem_utils.py'
- label: ":intel: (BMG) Detokenizer Stop Strings (XPU)"
key: detokenizer-stop-strings-xpu
timeout_in_minutes: 45
device: intel_gpu
agent_tags:
label: production
gpu: 1+
@@ -362,12 +290,14 @@ steps:
- tests/multimodal
- tests/utils_
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests &&
pytest -v -s detokenizer/test_stop_strings.py'
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests &&
pytest -v -s detokenizer &&
pytest -v -s -m "not cpu_test" ./multimodal &&
pytest -v -s utils_ --ignore=utils_/test_mem_utils.py'
- label: ":intel: (BMG) Fusion Unit Tests"
- label: Fusion Unit Tests
key: fusion-unit-tests
timeout_in_minutes: 30
device: intel_gpu
+11 -30
View File
@@ -2,38 +2,10 @@ group: Model Executor Intel
depends_on:
- image-build-xpu
steps:
- label: ":intel: (BMG) Model Executor"
- label: Model Executor (Intel)
key: model-executor-intel
timeout_in_minutes: 45
device: intel_gpu
agent_tags:
label: production
gpu: 1+
mem: 16+
no_plugin: true
working_dir: "."
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
VLLM_TEST_DEVICE: "xpu"
source_file_dependencies:
- vllm/engine/arg_utils.py
- vllm/config/model.py
- vllm/model_executor
- tests/model_executor
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'apt-get update && apt-get install -y curl libsodium23 &&
export VLLM_WORKER_MULTIPROC_METHOD=spawn &&
export PYTHONFAULTHANDLER=1 &&
cd tests &&
pytest -v -s model_executor -m "not slow_test" --ignore="model_executor/layers/test_rocm_unquantized_gemm.py" --ignore="model_executor/model_loader/tensorizer_loader/test_tensorizer.py" --ignore="model_executor/model_loader/test_reload.py" --deselect="tests/model_executor/test_enabled_custom_ops.py::test_topk_gating_enabled_by_cdna"'
- label: ":intel: (BMG) Tensorizer Model Loader Tests (XPU)"
key: tensorizer-model-loader-tests-xpu
timeout_in_minutes: 45
device: intel_gpu
agent_tags:
label: production
gpu: 1+
@@ -56,4 +28,13 @@ steps:
export VLLM_WORKER_MULTIPROC_METHOD=spawn &&
export PYTHONFAULTHANDLER=1 &&
cd tests &&
pytest -v -s model_executor/model_loader/tensorizer_loader/test_tensorizer.py -m "not slow_test"'
pytest -v -s model_executor -m "not slow_test" --ignore="model_executor/layers/test_rocm_unquantized_gemm.py" --deselect="tests/model_executor/model_loader/test_reload.py::test_kv_scale_reload"
--deselect="tests/model_executor/layers/test_mm_input_norm.py::TestFusedMMInputNormModule::test_matches_reference_above_cudnn_grid_limit"
--deselect="tests/model_executor/layers/test_mm_input_norm.py::TestFusedMMInputNormDtypes::test_dtype_combinations[in_dtype0-out_dtype0]"
--deselect="tests/model_executor/layers/test_mm_input_norm.py::TestFusedMMInputNormDtypes::test_dtype_combinations[in_dtype1-out_dtype1]"
--deselect="tests/model_executor/layers/test_mm_input_norm.py::TestFusedMMInputNormDtypes::test_dtype_combinations[in_dtype2-out_dtype2]"
--deselect="tests/model_executor/layers/test_mm_input_norm.py::TestFusedMMInputNormDtypes::test_dtype_combinations[in_dtype3-out_dtype3]"
--deselect="tests/model_executor/layers/test_mm_input_norm.py::TestFusedMMInputNormDtypes::test_dtype_combinations[in_dtype4-out_dtype4]"
--deselect="tests/model_executor/layers/test_mm_input_norm.py::TestFusedMMInputNormDtypes::test_dtype_combinations[in_dtype5-out_dtype5]"
--deselect="tests/model_executor/layers/test_mm_input_norm.py::TestFusedMMInputNormShapes"
--deselect="tests/model_executor/layers/test_mm_input_norm.py::TestFusedMMInputNormInputHandling"'
@@ -1,8 +1,8 @@
group: Models - Distributed Intel
group: Models - Distributed
depends_on:
- image-build-xpu
steps:
- label: ":intel: (BMG) Distributed Model Tests"
- label: Distributed Model Tests (2 GPUs)
key: distributed-model-tests-2-gpus
timeout_in_minutes: 65
device: intel_gpu
@@ -1,15 +1,15 @@
group: Models - Multimodal Intel
group: Models - Multimodal
depends_on:
- image-build-xpu
steps:
- label: ":intel: (BMG) Multi-Modal Models (Standard) 1: qwen2"
- label: "Multi-Modal Models (Standard) 1: qwen2"
key: multi-modal-models-standard-1-qwen2
timeout_in_minutes: 70
device: intel_gpu
agent_tags:
label: production
gpu: 1+
mem: 16+
mem: 24+
no_plugin: true
working_dir: "."
env:
@@ -26,14 +26,14 @@ steps:
pytest -v -s models/multimodal/generation/test_common.py -m core_model -k "qwen2" &&
pytest -v -s models/multimodal/generation/test_ultravox.py -m core_model'
- label: ":intel: (BMG) Multi-Modal Models (Standard) 2: qwen3 + gemma"
- label: "Multi-Modal Models (Standard) 2: qwen3 + gemma"
key: multi-modal-models-standard-2-qwen3-gemma
timeout_in_minutes: 90
device: intel_gpu
agent_tags:
label: production
gpu: 1+
mem: 16+
mem: 24+
no_plugin: true
working_dir: "."
env:
@@ -51,7 +51,7 @@ steps:
pytest -v -s models/multimodal/generation/test_mm_prefix_lm.py -m core_model &&
pytest -v -s models/multimodal/generation/test_qwen2_5_vl.py -m core_model'
- label: ":intel: (BMG) Multi-Modal Models (Standard) 3: llava + qwen2_vl"
- label: "Multi-Modal Models (Standard) 3: llava + qwen2_vl"
key: multi-modal-models-standard-3-llava-qwen2-vl
timeout_in_minutes: 65
device: intel_gpu
@@ -72,18 +72,16 @@ steps:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests &&
pytest -v -s models/multimodal/generation/test_common.py -m core_model -k "not qwen2 and not qwen3 and not gemma" &&
pytest -v -s models/multimodal/generation/test_gemma_ragged_audio.py -m core_model
pytest -v -s models/multimodal/generation/test_qwen2_vl.py -m core_model'
pytest -v -s models/multimodal/generation/test_common.py -m core_model -k "not qwen2 and not qwen3 and not gemma"'
- label: ":intel: (BMG) Multi-Modal Models (Standard) 4: other + whisper"
- label: "Multi-Modal Models (Standard) 4: other + whisper"
key: multi-modal-models-standard-4-other-whisper
timeout_in_minutes: 45
device: intel_gpu
agent_tags:
label: production
gpu: 1+
mem: 16+
mem: 24+
no_plugin: true
working_dir: "."
env:
@@ -97,9 +95,9 @@ steps:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests &&
pytest -v -s models/multimodal -m core_model --ignore models/multimodal/generation/test_common.py --ignore models/multimodal/generation/test_ultravox.py --ignore models/multimodal/generation/test_qwen2_5_vl.py --ignore models/multimodal/generation/test_qwen2_vl.py --ignore models/multimodal/generation/test_whisper.py --ignore models/multimodal/generation/test_gemma_ragged_audio.py --ignore models/multimodal/generation/test_memory_leak.py --ignore models/multimodal/processing'
pytest -v -s models/multimodal -m core_model --ignore models/multimodal/generation/test_common.py --ignore models/multimodal/generation/test_ultravox.py --ignore models/multimodal/generation/test_qwen2_5_vl.py --ignore models/multimodal/generation/test_qwen2_vl.py --ignore models/multimodal/generation/test_whisper.py --ignore models/multimodal/generation/test_memory_leak.py --ignore models/multimodal/processing'
- label: ":intel: (BMG) Multi-Modal Processor" # 44min
- label: Multi-Modal Processor # 44min
key: multi-modal-processor
timeout_in_minutes: 60
device: intel_gpu
@@ -126,14 +124,14 @@ steps:
--num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT --shard-id=$$BUILDKITE_PARALLEL_JOB'
parallelism: 4
- label: ":intel: (BMG) Multi-Modal Models (Extended Generation 1)"
- label: Multi-Modal Models (Extended Generation 1)
key: multi-modal-models-extended-generation-1
timeout_in_minutes: 60
device: intel_gpu
agent_tags:
label: production
gpu: 1+
mem: 16+
mem: 24+
no_plugin: true
working_dir: "."
env:
@@ -1,8 +1,8 @@
group: Quantization Intel
group: Quantization
depends_on:
- image-build-xpu
steps:
- label: ":intel: (BMG) Quantization"
- label: Quantization
key: quantization
timeout_in_minutes: 30
env:
@@ -25,7 +25,5 @@ steps:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'VLLM_TEST_FORCE_LOAD_FORMAT=auto pytest -v -s tests/quantization/test_per_token_kv_cache.py --deselect="tests/quantization/test_per_token_kv_cache.py::test_triton_unified_attention_per_token_head_scale[int4-16-128-num_heads0-seq_lens1]"
&& VLLM_TEST_FORCE_LOAD_FORMAT=auto pytest -v -s tests/quantization/test_auto_awq.py
&& VLLM_TEST_FORCE_LOAD_FORMAT=auto pytest -v -s tests/quantization/test_fp8_per_channel.py
&& VLLM_TEST_FORCE_LOAD_FORMAT=auto pytest -v -s tests/models/quantization/test_fp8_per_channel.py --deselect="tests/models/quantization/test_fp8_per_channel.py::test_fp8_per_channel_logprobs[moe]"'
&& VLLM_TEST_FORCE_LOAD_FORMAT=auto pytest -v -s tests/quantization/test_auto_awq.py'
+2 -29
View File
@@ -2,37 +2,10 @@ group: Samplers Intel
depends_on:
- image-build-xpu
steps:
- label: ":intel: (BMG) Samplers Test (XPU Kernel)"
- label: Samplers Test (XPU Kernel)
key: samplers-test-xpu-kernel-intel
timeout_in_minutes: 40
device: intel_gpu
agent_tags:
label: production
gpu: 1+
mem: 16+
no_plugin: true
working_dir: "."
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
VLLM_TEST_DEVICE: "xpu"
source_file_dependencies:
- vllm/model_executor/layers
- vllm/sampling_metadata.py
- tests/samplers
- tests/conftest.py
- vllm/entrypoints/generate/beam_search
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests &&
VLLM_XPU_USE_SAMPLER_KERNEL=1 pytest -v -s samplers
--ignore=samplers/test_beam_search.py'
- label: ":intel: (BMG) Samplers Beam Search Test (XPU Kernel)"
key: samplers-beam-search-test-xpu-kernel-intel
timeout_in_minutes: 40
device: intel_gpu
agent_tags:
label: production
gpu: 1+
@@ -53,4 +26,4 @@ steps:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests &&
VLLM_XPU_USE_SAMPLER_KERNEL=1 pytest -v -s samplers/test_beam_search.py'
VLLM_XPU_USE_SAMPLER_KERNEL=1 pytest -v -s samplers'
+17 -42
View File
@@ -14,7 +14,7 @@ steps:
limit: 2
- exit_status: -10 # Agent was lost
limit: 2
- label: ":intel: (BMG) Example Test"
- label: "XPU example Test"
key: xpu-example-test
depends_on:
- image-build-xpu
@@ -23,14 +23,14 @@ steps:
agent_tags:
label: production
gpu: 2+
mem: 16+
mem: 24+
no_plugin: true
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
source_file_dependencies:
- vllm/
- .buildkite/intel_jobs/test-intel.yaml
- .buildkite/intel_jobs/test-intel.yaml
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
@@ -46,9 +46,10 @@ steps:
python3 examples/basic/offline_inference/generate.py --model ibm-research/PowerMoE-3b --block-size 64 --enforce-eager -tp 2 &&
python3 examples/basic/offline_inference/generate.py --model ibm-research/PowerMoE-3b --block-size 64 --enforce-eager -tp 2 --enable-expert-parallel &&
python3 examples/basic/offline_inference/generate.py --model superjob/Qwen3-4B-Instruct-2507-GPTQ-Int4 --max-model-len 8192 &&
VLLM_XPU_FUSED_MOE_USE_REF=1 python3 examples/basic/offline_inference/generate.py --model Qwen/Qwen3-30B-A3B-Instruct-2507-FP8 --enforce-eager -tp 2 --max-model-len 8192 &&
python3 examples/basic/offline_inference/generate.py --model INCModel/Qwen3-30B-A3B-Instruct-2507-MXFP4-LLMC -tp 2 --max-model-len 8192
'
- label: ":intel: (BMG) W8A8 FP8 Linear Examples"
- label: "XPU W8A8 FP8 Linear Examples"
key: xpu-w8a8-fp8-linear-examples
depends_on:
- image-build-xpu
@@ -57,7 +58,7 @@ steps:
agent_tags:
label: production
gpu: 1+
mem: 16+
mem: 24+
no_plugin: true
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
@@ -76,7 +77,7 @@ steps:
python3 examples/basic/offline_inference/generate.py --linear-backend xpu --model meta-llama/Llama-3.2-1B-Instruct --quantization fp8_per_tensor --enforce-eager --max-model-len 4096 &&
python3 examples/basic/offline_inference/generate.py --linear-backend torch --model meta-llama/Llama-3.2-1B-Instruct --quantization fp8_per_tensor --enforce-eager --max-model-len 4096
'
- label: ":intel: (BMG) V1 test"
- label: "XPU V1 test"
key: xpu-v1-test
depends_on:
- image-build-xpu
@@ -85,14 +86,14 @@ steps:
agent_tags:
label: production
gpu: 1+
mem: 16+
mem: 24+
no_plugin: true
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
source_file_dependencies:
- vllm/
- .buildkite/intel_jobs/test-intel.yaml
- .buildkite/intel_jobs/test-intel.yaml
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
@@ -105,35 +106,9 @@ steps:
pytest -v -s v1/test_serial_utils.py &&
pytest -v -s v1/e2e/general/test_correctness_sliding_window.py &&
pytest -v -s v1/spec_decode --ignore=v1/spec_decode/test_max_len.py --ignore=v1/spec_decode/test_speculators_eagle3.py --ignore=v1/spec_decode/test_acceptance_length.py --ignore=v1/spec_decode/test_speculators_correctness.py --deselect=tests/v1/spec_decode/test_mtp.py::test_glm_mtp_defers_lm_head &&
pytest -v -s v1/kv_connector/unit --ignore=v1/kv_connector/unit/test_lmcache_integration.py --ignore=v1/kv_connector/unit/test_hf3fs_client.py --ignore=v1/kv_connector/unit/test_hf3fs_connector.py --ignore=v1/kv_connector/unit/test_hf3fs_metadata_server.py --ignore=v1/kv_connector/unit/test_offloading_connector.py --ignore=v1/kv_connector/unit/test_multi_connector.py --ignore=v1/kv_connector/unit/test_example_connector.py --deselect=tests/v1/kv_connector/unit/test_nixl_connector.py::test_abort_timeout_on_prefiller[ray]'
pytest -v -s v1/kv_connector/unit --ignore=v1/kv_connector/unit/test_lmcache_integration.py --ignore=v1/kv_connector/unit/test_hf3fs_client.py --ignore=v1/kv_connector/unit/test_hf3fs_connector.py --ignore=v1/kv_connector/unit/test_hf3fs_metadata_server.py --ignore=v1/kv_connector/unit/test_offloading_connector.py --ignore=v1/kv_connector/unit/test_nixl_connector_hma.py --deselect=tests/v1/kv_connector/unit/test_nixl_connector.py::test_abort_timeout_on_prefiller[ray]'
parallelism: 2
- label: ":intel: (BMG) Qwen3-30B + V1 KV Connector"
key: xpu-qwen3-30b-v1-kv-connector
depends_on:
- image-build-xpu
timeout_in_minutes: 90
device: intel_gpu
agent_tags:
label: production
gpu: 2+
mem: 24+
no_plugin: true
env:
REGISTRY: "public.ecr.aws/q9t5s3a7"
REPO: "vllm-ci-test-repo"
source_file_dependencies:
- vllm/
- .buildkite/intel_jobs/test-intel.yaml
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests &&
pip install dpctl &&
pytest -v -s v1/kv_connector/unit/test_multi_connector.py &&
pytest -v -s v1/kv_connector/unit/test_example_connector.py &&
cd .. &&
VLLM_XPU_FUSED_MOE_USE_REF=1 python3 examples/basic/offline_inference/generate.py --model Qwen/Qwen3-30B-A3B-Instruct-2507-FP8 --enforce-eager -tp 2 --max-model-len 8192'
- label: ":intel: (BMG) Server test"
- label: "XPU server test"
key: xpu-server-test
depends_on:
- image-build-xpu
@@ -149,14 +124,14 @@ steps:
REPO: "vllm-ci-test-repo"
source_file_dependencies:
- vllm/
- .buildkite/intel_jobs/test-intel.yaml
- .buildkite/intel_jobs/test-intel.yaml
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests &&
pytest -v -s entrypoints/multimodal/openai/chat_completion/test_audio_in_video.py &&
pytest -v -s benchmarks/test_serve_cli.py'
- label: ":intel: (BMG) Quantization test"
- label: "XPU quantization test"
key: xpu-quantization-test
depends_on:
- image-build-xpu
@@ -172,14 +147,14 @@ steps:
REPO: "vllm-ci-test-repo"
source_file_dependencies:
- vllm/
- .buildkite/intel_jobs/test-intel.yaml
- .buildkite/intel_jobs/test-intel.yaml
commands:
- >-
bash .buildkite/scripts/hardware_ci/run-intel-test.sh
'cd tests &&
pytest -v -s quantization/test_auto_round.py &&
pytest -v -s quantization/test_online.py --deselect=tests/quantization/test_online.py::test_online_quantization_loads_real_weights --deselect="tests/quantization/test_online.py::test_online_quantization[False-requantization_mxfp8_ptcp_fp8]"'
- label: ":intel: (BMG) Compressed tensors FP8 test"
pytest -v -s quantization/test_online.py --deselect=tests/quantization/test_online.py::test_online_quantization_loads_real_weights'
- label: "XPU compressed tensors FP8 test"
key: xpu-compressed-tensors-fp8-test
depends_on:
- image-build-xpu
@@ -204,7 +179,7 @@ steps:
'cd tests &&
pytest -v -s quantization/test_compressed_tensors.py::test_compressed_tensors_fp8 &&
pytest -v -s quantization/test_compressed_tensors.py::test_compressed_tensors_fp8_block_enabled'
- label: ":intel: (BMG) Graph"
- label: "XPU Graph"
key: xpu-graph
depends_on:
- image-build-xpu
@@ -8,28 +8,16 @@ pytest -s -v test_lm_eval_correctness.py \
--tp-size=1
"""
import gc
import os
import time
from contextlib import contextmanager
import lm_eval
import pytest
import torch
import yaml
from lm_eval.api.registry import get_model
from lm_eval.utils import simple_parse_args_string
from vllm import LLM
from vllm.distributed import cleanup_dist_env_and_memory
from vllm.logger import init_logger
from vllm.platforms import current_platform
logger = init_logger(__name__)
DEFAULT_RTOL = 0.08
ROCM_ENGINE_SHUTDOWN_TIMEOUT_S = 60.0
MEMORY_RELEASE_TIMEOUT_S = 240.0
@contextmanager
@@ -58,48 +46,6 @@ def scoped_env_vars(new_env: dict[str, str]):
os.environ.pop(key, None)
def _wait_for_memory_release(gpu_memory_utilization: float) -> None:
devices = range(torch.accelerator.device_count())
deadline = time.monotonic() + MEMORY_RELEASE_TIMEOUT_S
while True:
memory = [torch.accelerator.get_memory_info(d) for d in devices]
if all(free >= total * gpu_memory_utilization for free, total in memory):
return
if time.monotonic() >= deadline:
usage = ", ".join(
f"{d}: {free / 2**30:.2f}/{total / 2**30:.2f} GiB free"
for d, (free, total) in zip(devices, memory)
)
logger.warning(
"GPU memory not released after %ss (%s); the next model needs %.2f",
MEMORY_RELEASE_TIMEOUT_S,
usage,
gpu_memory_utilization,
)
return
time.sleep(1)
def _shutdown_lm(lm) -> None:
if not current_platform.is_rocm():
return
llm = getattr(lm, "model", None)
if not isinstance(llm, LLM):
return
gpu_memory_utilization = (
llm.llm_engine.vllm_config.cache_config.gpu_memory_utilization
)
try:
llm.llm_engine.engine_core.shutdown(timeout=ROCM_ENGINE_SHUTDOWN_TIMEOUT_S)
except Exception:
logger.exception("Engine core shutdown raised; GPU memory may leak")
del llm
lm.model = None
gc.collect()
cleanup_dist_env_and_memory()
_wait_for_memory_release(gpu_memory_utilization)
def launch_lm_eval(eval_config, tp_size):
trust_remote_code = eval_config.get("trust_remote_code", False)
max_model_len = eval_config.get("max_model_len", 4096)
@@ -136,17 +82,9 @@ def launch_lm_eval(eval_config, tp_size):
env_vars = eval_config.get("env_vars", None)
with scoped_env_vars(env_vars):
if current_platform.is_rocm():
model = get_model(backend).create_from_arg_string(
model_args,
{"batch_size": batch_size, "max_batch_size": None, "device": None},
)
model_kwargs = {"metadata": simple_parse_args_string(model_args)}
else:
model = backend
model_kwargs = {"model_args": model_args}
results = lm_eval.simple_evaluate(
model=model,
model=backend,
model_args=model_args,
tasks=[task["name"] for task in eval_config["tasks"]],
num_fewshot=eval_config["num_fewshot"],
limit=eval_config["limit"],
@@ -160,9 +98,7 @@ def launch_lm_eval(eval_config, tp_size):
# Forward decoding and early-stop controls (e.g., max_gen_toks, until=...)
gen_kwargs=eval_config.get("gen_kwargs"),
batch_size=batch_size,
**model_kwargs,
)
_shutdown_lm(model)
return results
+451 -223
View File
@@ -355,10 +355,6 @@ steps:
queue: arm64_cpu_queue_release
commands:
- "aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7"
# TODO(arpera):
# Remove BUILD_NIXL=true once NIXL has a wheel built against torch 2.15
# For details: https://github.com/vllm-project/vllm/pull/60090
# Tracking issue report: https://github.com/ai-dynamo/nixl/issues/2297
- |
DOCKER_BUILDKIT=1 docker buildx build --load \
$(bash .buildkite/scripts/docker-build-metadata-args.sh cu134) \
@@ -370,7 +366,6 @@ steps:
--build-arg torch_cuda_arch_list="${CUDA_ARCH_AARCH64}" \
--build-arg INSTALL_KV_CONNECTORS=true \
--build-arg INSTALL_RUBIN_PRERELEASE=true \
--build-arg BUILD_NIXL=true \
--build-arg TRITON_INSTALL_FROM_SOURCE_REPO=https://github.com/triton-lang/triton.git \
--build-arg TRITON_INSTALL_FROM_SOURCE_REVISION=3f6e41132b5edf639bfb872ad73d4688765e08b8 \
--build-arg RUN_WHEEL_CHECK=false \
@@ -703,270 +698,348 @@ steps:
DOCKER_BUILDKIT: "1"
DOCKERHUB_USERNAME: "vllmbot"
- trigger: "perf-eval"
label: "Trigger perf-eval (CUDA)"
key: trigger-perf-eval-cuda
if: build.env("NIGHTLY") == "1" || build.env("RUN_PERF_EVAL") == "1"
depends_on:
- build-release-image-x86
async: true
soft_fail: true
build:
branch: "main"
message: "(NIGHTLY=${NIGHTLY:-0}) Evaluate vLLM $BUILDKITE_COMMIT (CUDA)"
env:
NIGHTLY: 1 # run all perf-eval workloads
VLLM_COMMIT: "$BUILDKITE_COMMIT"
VLLM_IMAGE_CUDA: "public.ecr.aws/q9t5s3a7/vllm-release-repo:$BUILDKITE_COMMIT-x86_64"
- trigger: "perf-eval"
label: "Trigger perf-eval (ROCm)"
key: trigger-perf-eval-rocm
if: build.env("NIGHTLY") == "1" || build.env("RUN_PERF_EVAL") == "1"
depends_on:
- build-rocm100-release-image
async: true
soft_fail: true
build:
branch: "main"
message: "(NIGHTLY=${NIGHTLY:-0}) Evaluate vLLM $BUILDKITE_COMMIT (ROCm)"
env:
NIGHTLY: 1 # run all perf-eval workloads
VLLM_COMMIT: "$BUILDKITE_COMMIT"
VLLM_IMAGE_ROCM: "public.ecr.aws/q9t5s3a7/vllm-release-repo:$BUILDKITE_COMMIT-rocm100"
# =============================================================================
# ROCm Release Pipelines (x86_64 only)
# ROCm Release Pipeline (x86_64 only)
# =============================================================================
#
# One group per ROCm stack, identified by its base Dockerfile (ROCM_BASE_DOCKERFILE).
# The scripts derive the ROCm version, wheel variant, cache keys and image tags from
# it (.buildkite/scripts/rocm/stack.sh), so both groups share one set of step
# definitions, anchored on the ROCm 10.0 group.
# vLLM version is determined by the Buildkite checkout (like CUDA pipeline).
# To build a specific version, trigger the build from that branch/tag.
#
# - ROCm 10.0 (TheRock, docker/Dockerfile.rocm{,_base}) is the default: it owns the
# :nightly / :latest images and the root wheel index.
# - ROCm 7.2 (docker/Dockerfile.rocm_72{,_base}) is kept for a transition period as
# rocm72 images and wheels.
# Environment variables for ROCm builds (set via Buildkite UI or schedule):
#
# Note: ROCm version is determined by BASE_IMAGE in docker/Dockerfile.rocm_base
#
# vLLM version is determined by the Buildkite checkout (like the CUDA pipeline).
# =============================================================================
- group: "Build ROCm 10.0 Wheel / Image"
key: "build-rocm100"
- group: "Build ROCm Wheel / Image "
key: "build-rocm-wheel-image"
depends_on: ~
steps:
- label: ":rocm: Build ROCm 10.0 Base Image & Wheels"
id: build-rocm100-base
# ROCm Job 1: Build ROCm Base Wheels (with S3 caching)
- label: ":rocm: Build ROCm Base Image & Wheels"
id: build-rocm-base-wheels
depends_on: ~
agents: &rocm_build_agents
agents:
queue: cpu_queue_release
timeout_in_minutes: 360
env: &rocm100_env
ROCM_BASE_DOCKERFILE: "docker/Dockerfile.rocm_base"
ROCM_DEFAULT_ARG: "--default"
commands:
- |
set -euo pipefail
# Generate cache key
CACHE_KEY=$$(.buildkite/scripts/cache-rocm-base-wheels.sh key)
ECR_CACHE_TAG="public.ecr.aws/q9t5s3a7/vllm-release-repo:$${CACHE_KEY}-rocm-base"
echo "========================================"
echo "ROCm Base Build Configuration"
echo "========================================"
echo " CACHE_KEY: $${CACHE_KEY}"
echo " ECR_CACHE_TAG: $${ECR_CACHE_TAG}"
echo "========================================"
# Login to ECR
aws ecr-public get-login-password --region us-east-1 | \
docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7
IMAGE_EXISTS=false
WHEELS_EXIST=false
# Check ECR for Docker image
if docker manifest inspect "$${ECR_CACHE_TAG}" > /dev/null 2>&1; then
IMAGE_EXISTS=true
echo "ECR image cache HIT"
fi
# Check S3 for wheels
WHEEL_CACHE_STATUS=$(.buildkite/scripts/cache-rocm-base-wheels.sh check)
if [ "$${WHEEL_CACHE_STATUS}" = "hit" ]; then
WHEELS_EXIST=true
echo "S3 wheels cache HIT"
fi
# Scenario 1: Both cached (best case)
if [ "$${IMAGE_EXISTS}" = "true" ] && [ "$${WHEELS_EXIST}" = "true" ]; then
echo ""
echo "FULL CACHE HIT - Reusing both image and wheels"
echo ""
# Download wheels
.buildkite/scripts/cache-rocm-base-wheels.sh download
# Save ECR tag for downstream jobs
buildkite-agent meta-data set "rocm-base-image-tag" "$${ECR_CACHE_TAG}"
# Scenario 2: Full rebuild needed
else
echo ""
echo " CACHE MISS - Building from scratch..."
echo ""
# Build full base image and push to ECR
DOCKER_BUILDKIT=1 docker buildx build \
--file docker/Dockerfile.rocm_base \
--tag "$${ECR_CACHE_TAG}" \
--build-arg USE_SCCACHE=1 \
--build-arg SCCACHE_BUCKET_NAME=vllm-build-sccache \
--build-arg SCCACHE_REGION_NAME=us-west-2 \
--build-arg SCCACHE_S3_NO_CREDENTIALS=0 \
--push \
.
# Build wheel extraction stage
DOCKER_BUILDKIT=1 docker buildx build \
--file docker/Dockerfile.rocm_base \
--tag rocm-base-debs:$${BUILDKITE_BUILD_NUMBER} \
--target debs_wheel_release \
--build-arg USE_SCCACHE=1 \
--build-arg SCCACHE_BUCKET_NAME=vllm-build-sccache \
--build-arg SCCACHE_REGION_NAME=us-west-2 \
--build-arg SCCACHE_S3_NO_CREDENTIALS=0 \
--load \
.
# Extract and upload wheels
mkdir -p artifacts/rocm-base-wheels
cid=$(docker create rocm-base-debs:$${BUILDKITE_BUILD_NUMBER})
docker cp $${cid}:/app/debs/. artifacts/rocm-base-wheels/
docker rm $${cid}
.buildkite/scripts/cache-rocm-base-wheels.sh upload
# Cache base docker image to ECR
docker push "$${ECR_CACHE_TAG}"
buildkite-agent meta-data set "rocm-base-image-tag" "$${ECR_CACHE_TAG}"
echo ""
echo " Build complete - Image and wheels cached"
fi
artifact_paths:
- "artifacts/rocm-base-wheels/*.whl"
env:
DOCKER_BUILDKIT: "1"
S3_BUCKET: "vllm-wheels"
DOCKERHUB_USERNAME: "vllmbot"
commands: &rocm_base_commands
- |
set -euo pipefail
source .buildkite/scripts/rocm/stack.sh "$${ROCM_BASE_DOCKERFILE}"
SCCACHE_ARGS=(--build-arg USE_SCCACHE=1 --build-arg SCCACHE_BUCKET_NAME=vllm-build-sccache
--build-arg SCCACHE_REGION_NAME=us-west-2 --build-arg SCCACHE_S3_NO_CREDENTIALS=0)
aws ecr-public get-login-password --region us-east-1 | \
docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7
# Base image, cached in ECR by the base Dockerfile's hash
if docker manifest inspect "$${ROCM_STACK_ECR_BASE}" > /dev/null 2>&1; then
echo "ECR image cache HIT: $${ROCM_STACK_ECR_BASE}"
else
docker buildx build --file "$${ROCM_BASE_DOCKERFILE}" --tag "$${ROCM_STACK_ECR_BASE}" \
"$${SCCACHE_ARGS[@]}" --push .
fi
# Base wheels, cached in S3 (see cache-rocm-base-wheels.sh)
CACHE=.buildkite/scripts/cache-rocm-base-wheels.sh
if [ "$$($${CACHE} "$${ROCM_BASE_DOCKERFILE}" check)" = "hit" ]; then
$${CACHE} "$${ROCM_BASE_DOCKERFILE}" download
else
DEBS="rocm-base-debs:$${BUILDKITE_BUILD_NUMBER}-$${ROCM_STACK_VARIANT}"
docker buildx build --file "$${ROCM_BASE_DOCKERFILE}" --tag "$${DEBS}" \
--target debs_wheel_release "$${SCCACHE_ARGS[@]}" --load .
mkdir -p "$${ROCM_STACK_BASE_WHEELS_DIR}"
cid=$$(docker create "$${DEBS}")
docker cp "$${cid}":/app/debs/. "$${ROCM_STACK_BASE_WHEELS_DIR}/"
docker rm "$${cid}"
$${CACHE} "$${ROCM_BASE_DOCKERFILE}" upload
fi
buildkite-agent artifact upload "$${ROCM_STACK_BASE_WHEELS_DIR}/*.whl"
- label: ":python: Build vLLM ROCm 10.0 Wheel - x86_64"
id: build-rocm100-vllm-wheel
depends_on: build-rocm100-base
agents: *rocm_build_agents
# ROCm Job 2: Build vLLM ROCm Wheel
- label: ":python: Build vLLM ROCm Wheel - x86_64"
id: build-rocm-vllm-wheel
depends_on:
- step: build-rocm-base-wheels
allow_failure: false
agents:
queue: cpu_queue_release
timeout_in_minutes: 180
env: *rocm100_env
commands: &rocm_vllm_wheel_commands
commands:
# Download artifacts and prepare Docker image
- |
set -euo pipefail
source .buildkite/scripts/rocm/stack.sh "$${ROCM_BASE_DOCKERFILE}"
# Buildkite's default fetch doesn't update moved tags; version detection needs them
# Ensure git tags are up-to-date (Buildkite's default fetch doesn't update tags)
# This fixes version detection when tags are moved/force-pushed
echo "Fetching latest tags from origin..."
git fetch --tags --force origin
buildkite-agent artifact download "$${ROCM_STACK_BASE_WHEELS_DIR}/*.whl" .
# Log tag information for debugging version detection
echo "========================================"
echo "Git Tag Verification"
echo "========================================"
echo "Current HEAD: $(git rev-parse HEAD)"
echo "git describe --tags: $(git describe --tags 2>/dev/null || echo 'No tags found')"
echo ""
echo "Recent tags (pointing to commits near HEAD):"
git tag -l --sort=-creatordate | head -5
echo "setuptools_scm version detection:"
pip install -q setuptools_scm 2>/dev/null || true
python3 -c "import setuptools_scm; print(' Detected version:', setuptools_scm.get_version())" 2>/dev/null || echo " (setuptools_scm not available in this environment)"
echo "========================================"
# Download wheel artifacts from current build
echo "Downloading wheel artifacts from current build"
buildkite-agent artifact download "artifacts/rocm-base-wheels/*.whl" .
# Get ECR image tag from metadata (set by build-rocm-base-wheels)
ECR_IMAGE_TAG="$$(buildkite-agent meta-data get rocm-base-image-tag 2>/dev/null || echo '')"
if [ -z "$${ECR_IMAGE_TAG}" ]; then
echo "ERROR: rocm-base-image-tag metadata not found"
echo "This should have been set by the build-rocm-base-wheels job"
exit 1
fi
echo "Pulling base Docker image from ECR: $${ECR_IMAGE_TAG}"
# Login to ECR
aws ecr-public get-login-password --region us-east-1 | \
docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7
docker pull "$${ROCM_STACK_ECR_BASE}"
# Pull base Docker image from ECR
docker pull "$${ECR_IMAGE_TAG}"
echo "Loaded base image: $${ECR_IMAGE_TAG}"
# Prepare base wheels for Docker build context
mkdir -p docker/context/base-wheels
touch docker/context/base-wheels/.keep
cp "$${ROCM_STACK_BASE_WHEELS_DIR}"/*.whl docker/context/base-wheels/
docker build --file "$${ROCM_STACK_DOCKERFILE}" --target export_vllm_wheel_release \
--output type=local,dest=rocm-dist \
--build-arg BASE_IMAGE="$${ROCM_STACK_ECR_BASE}" --build-arg REMOTE_VLLM=0 \
--build-arg GIT_REPO_CHECK=1 --build-arg USE_SCCACHE=1 \
--build-arg SCCACHE_BUCKET_NAME=vllm-build-sccache \
--build-arg SCCACHE_REGION_NAME=us-west-2 --build-arg SCCACHE_S3_NO_CREDENTIALS=0 .
mkdir -p "$${ROCM_STACK_VLLM_WHEEL_DIR}"
cp rocm-dist/*.whl "$${ROCM_STACK_VLLM_WHEEL_DIR}/"
buildkite-agent artifact upload "$${ROCM_STACK_VLLM_WHEEL_DIR}/*.whl"
cp artifacts/rocm-base-wheels/*.whl docker/context/base-wheels/
echo "Base wheels for vLLM build:"
ls -lh docker/context/base-wheels/
- label: ":s3: Upload ROCm 10.0 Wheels to S3"
id: upload-rocm100-wheels
depends_on: build-rocm100-vllm-wheel
agents: *rocm_build_agents
echo "========================================"
echo "Building vLLM wheel with:"
echo " BUILDKITE_COMMIT: $${BUILDKITE_COMMIT}"
echo " BUILDKITE_BRANCH: $${BUILDKITE_BRANCH}"
echo " BASE_IMAGE: $${ECR_IMAGE_TAG}"
echo "========================================"
# Build vLLM wheel using local checkout (REMOTE_VLLM=0)
DOCKER_BUILDKIT=1 docker build \
--file docker/Dockerfile.rocm \
--target export_vllm_wheel_release \
--output type=local,dest=rocm-dist \
--build-arg BASE_IMAGE="$${ECR_IMAGE_TAG}" \
--build-arg REMOTE_VLLM=0 \
--build-arg GIT_REPO_CHECK=1 \
--build-arg USE_SCCACHE=1 \
--build-arg SCCACHE_BUCKET_NAME=vllm-build-sccache \
--build-arg SCCACHE_REGION_NAME=us-west-2 \
--build-arg SCCACHE_S3_NO_CREDENTIALS=0 \
.
echo "Built vLLM wheel:"
ls -lh rocm-dist/*.whl
# Copy wheel to artifacts directory
mkdir -p artifacts/rocm-vllm-wheel
cp rocm-dist/*.whl artifacts/rocm-vllm-wheel/
echo "Final vLLM wheel:"
ls -lh artifacts/rocm-vllm-wheel/
artifact_paths:
- "artifacts/rocm-vllm-wheel/*.whl"
env:
DOCKER_BUILDKIT: "1"
S3_BUCKET: "vllm-wheels"
# ROCm Job 3: Upload Wheels to S3
- label: ":s3: Upload ROCm Wheels to S3"
id: upload-rocm-wheels
depends_on:
- step: build-rocm-vllm-wheel
allow_failure: false
agents:
queue: cpu_queue_release
timeout_in_minutes: 60
env: *rocm100_env
commands: &rocm_upload_commands
commands:
# Download all wheel artifacts and run upload
- |
set -euo pipefail
source .buildkite/scripts/rocm/stack.sh "$${ROCM_BASE_DOCKERFILE}"
buildkite-agent artifact download "$${ROCM_STACK_BASE_WHEELS_DIR}/*.whl" .
buildkite-agent artifact download "$${ROCM_STACK_VLLM_WHEEL_DIR}/*.whl" .
bash .buildkite/scripts/upload-rocm-wheels.sh "$${ROCM_BASE_DOCKERFILE}"
- label: ":memo: Annotate ROCm 10.0 wheel release"
id: annotate-rocm100-release
depends_on: upload-rocm100-wheels
agents: *rocm_build_agents
env: *rocm100_env
commands: &rocm_annotate_commands
- bash .buildkite/scripts/annotate-rocm-release.sh "$${ROCM_BASE_DOCKERFILE}" $${ROCM_DEFAULT_ARG}
# Download artifacts from current build
echo "Downloading artifacts from current build"
buildkite-agent artifact download "artifacts/rocm-base-wheels/*.whl" .
buildkite-agent artifact download "artifacts/rocm-vllm-wheel/*.whl" .
# Generate https://wheels.vllm.ai/rocm/ (the default stack's variant), so users can
# `uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/` (release only)
# # Run upload script
bash .buildkite/scripts/upload-rocm-wheels.sh
env:
DOCKER_BUILDKIT: "1"
S3_BUCKET: "vllm-wheels"
# ROCm Job 4: Annotate ROCm Wheel Release
- label: ":memo: Annotate ROCm wheel release"
id: annotate-rocm-release
depends_on:
- upload-rocm-wheels
agents:
queue: cpu_queue_release
commands:
- "bash .buildkite/scripts/annotate-rocm-release.sh"
env:
S3_BUCKET: "vllm-wheels"
# ROCm Job 5: Generate Root Index for ROCm Wheels (for release only)
# This is the job to create https://wheels.vllm.ai/rocm/ index allowing
# users to install with `uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/`
- block: "Generate Root Index for ROCm Wheels for Release"
key: block-generate-root-index-rocm-wheels
depends_on:
- upload-rocm100-wheels
- upload-rocm72-wheels
depends_on: upload-rocm-wheels
- label: ":package: Generate Root Index for ROCm Wheels for Release"
depends_on: block-generate-root-index-rocm-wheels
id: generate-root-index-rocm-wheels
agents: *rocm_build_agents
env: *rocm100_env
agents:
queue: cpu_queue_release
commands:
- "bash tools/vllm-rocm/generate-rocm-wheels-root-index.sh"
env:
S3_BUCKET: "vllm-wheels"
VARIANT: "rocm723"
- label: ":docker: Build release image - x86_64 - ROCm 10.0"
id: build-rocm100-release-image
# ROCm Job 6: Build ROCm Release Docker Image
- label: ":docker: Build release image - x86_64 - ROCm"
id: build-rocm-release-image
depends_on:
- step: block-build-release-images
allow_failure: true
- step: build-rocm100-base
- step: build-rocm-base-wheels
allow_failure: false
agents: *rocm_build_agents
timeout_in_minutes: 180
env: *rocm100_env
commands: &rocm_release_image_commands
agents:
queue: cpu_queue_release
timeout_in_minutes: 60
commands:
- |
set -euo pipefail
source .buildkite/scripts/rocm/stack.sh "$${ROCM_BASE_DOCKERFILE}"
# Login to ECR
aws ecr-public get-login-password --region us-east-1 | \
docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7
docker pull "$${ROCM_STACK_ECR_BASE}"
docker build --file "$${ROCM_STACK_DOCKERFILE}" --target vllm-openai \
--tag "$${ROCM_STACK_ECR_IMAGE}" --progress plain \
--build-arg max_jobs=16 --build-arg BASE_IMAGE="$${ROCM_STACK_ECR_BASE}" \
--build-arg INSTALL_LMCACHE=true --build-arg USE_SCCACHE=1 \
# Get ECR image tag from metadata (set by build-rocm-base-wheels)
ECR_IMAGE_TAG="$$(buildkite-agent meta-data get rocm-base-image-tag 2>/dev/null || echo '')"
if [ -z "$${ECR_IMAGE_TAG}" ]; then
echo "ERROR: rocm-base-image-tag metadata not found"
echo "This should have been set by the build-rocm-base-wheels job"
exit 1
fi
echo "Pulling base Docker image from ECR: $${ECR_IMAGE_TAG}"
# Pull base Docker image from ECR
docker pull "$${ECR_IMAGE_TAG}"
echo "Loaded base image: $${ECR_IMAGE_TAG}"
# Pass the base image ECR tag to downstream steps (nightly publish)
buildkite-agent meta-data set "rocm-base-ecr-tag" "$${ECR_IMAGE_TAG}"
echo "========================================"
echo "Building vLLM ROCm release image with:"
echo " BASE_IMAGE: $${ECR_IMAGE_TAG}"
echo " BUILDKITE_COMMIT: $${BUILDKITE_COMMIT}"
echo "========================================"
# Build vLLM ROCm release image using cached base
DOCKER_BUILDKIT=1 docker build \
--build-arg max_jobs=16 \
--build-arg BASE_IMAGE="$${ECR_IMAGE_TAG}" \
--build-arg USE_SCCACHE=1 \
--build-arg SCCACHE_BUCKET_NAME=vllm-build-sccache \
--build-arg SCCACHE_REGION_NAME=us-west-2 --build-arg SCCACHE_S3_NO_CREDENTIALS=0 .
docker push "$${ROCM_STACK_ECR_IMAGE}"
--build-arg SCCACHE_REGION_NAME=us-west-2 \
--build-arg SCCACHE_S3_NO_CREDENTIALS=0 \
--build-arg INSTALL_LMCACHE=true \
--tag public.ecr.aws/q9t5s3a7/vllm-release-repo:$${BUILDKITE_COMMIT}-rocm \
--target vllm-openai \
--progress plain \
-f docker/Dockerfile.rocm .
# Push to ECR
docker push public.ecr.aws/q9t5s3a7/vllm-release-repo:$${BUILDKITE_COMMIT}-rocm
- label: "Publish nightly ROCm 10.0 image to DockerHub"
id: publish-rocm100-nightly
depends_on: build-rocm100-release-image
if: build.env("NIGHTLY") == "1"
agents: &rocm_publish_agents
queue: small_cpu_queue_release
env: *rocm100_env
commands: &rocm_publish_nightly_commands
- bash .buildkite/scripts/push-nightly-builds-rocm.sh "$${ROCM_BASE_DOCKERFILE}" $${ROCM_DEFAULT_ARG}
plugins: &rocm_dockerhub_login
- docker-login#v3.0.0:
username: vllmbot
password-env: DOCKERHUB_TOKEN
- group: "Build ROCm 7.2 Wheel / Image"
key: "build-rocm72"
depends_on: ~
steps:
- label: ":rocm: Build ROCm 7.2 Base Image & Wheels"
id: build-rocm72-base
depends_on: ~
agents: *rocm_build_agents
timeout_in_minutes: 360
env: &rocm72_env
ROCM_BASE_DOCKERFILE: "docker/Dockerfile.rocm_72_base"
ROCM_DEFAULT_ARG: ""
echo ""
echo " Successfully built and pushed ROCm release image"
echo " Image: public.ecr.aws/q9t5s3a7/vllm-release-repo:$${BUILDKITE_COMMIT}-rocm"
echo ""
env:
DOCKER_BUILDKIT: "1"
S3_BUCKET: "vllm-wheels"
DOCKERHUB_USERNAME: "vllmbot"
commands: *rocm_base_commands
- label: ":python: Build vLLM ROCm 7.2 Wheel - x86_64"
id: build-rocm72-vllm-wheel
depends_on: build-rocm72-base
agents: *rocm_build_agents
timeout_in_minutes: 180
env: *rocm72_env
commands: *rocm_vllm_wheel_commands
- label: ":s3: Upload ROCm 7.2 Wheels to S3"
id: upload-rocm72-wheels
depends_on: build-rocm72-vllm-wheel
agents: *rocm_build_agents
timeout_in_minutes: 60
env: *rocm72_env
commands: *rocm_upload_commands
- label: ":memo: Annotate ROCm 7.2 wheel release"
id: annotate-rocm72-release
depends_on: upload-rocm72-wheels
agents: *rocm_build_agents
env: *rocm72_env
commands: *rocm_annotate_commands
- label: ":docker: Build release image - x86_64 - ROCm 7.2"
id: build-rocm72-release-image
depends_on:
- step: block-build-release-images
allow_failure: true
- step: build-rocm72-base
allow_failure: false
agents: *rocm_build_agents
timeout_in_minutes: 180
env: *rocm72_env
commands: *rocm_release_image_commands
- label: "Publish nightly ROCm 7.2 image to DockerHub"
id: publish-rocm72-nightly
depends_on: build-rocm72-release-image
if: build.env("NIGHTLY") == "1"
agents: *rocm_publish_agents
env: *rocm72_env
commands: *rocm_publish_nightly_commands
plugins: *rocm_dockerhub_login
- label: "Publish nightly XPU image to DockerHub"
depends_on:
@@ -985,6 +1058,163 @@ steps:
DOCKER_BUILDKIT: "1"
DOCKERHUB_USERNAME: "vllmbot"
- label: "Publish nightly ROCm image to DockerHub"
depends_on:
- build-rocm-release-image
if: build.env("NIGHTLY") == "1"
agents:
queue: small_cpu_queue_release
commands:
- "bash .buildkite/scripts/push-nightly-builds-rocm.sh"
# Clean up old nightly builds (keep only last 14)
- "bash .buildkite/scripts/cleanup-nightly-builds.sh nightly- vllm/vllm-openai-rocm"
- "bash .buildkite/scripts/cleanup-nightly-builds.sh base-nightly- vllm/vllm-openai-rocm"
plugins:
- docker-login#v3.0.0:
username: vllmbot
password-env: DOCKERHUB_TOKEN
env:
DOCKER_BUILDKIT: "1"
DOCKERHUB_USERNAME: "vllmbot"
# =============================================================================
# ROCk Release Pipeline (x86_64 only)
# =============================================================================
#
# Same shape as the ROCm pipeline above (Jobs 1 and 6 only), but built from
# docker/Dockerfile.rock_base and docker/Dockerfile.rock, which install the
# ROCm SDK from TheRock wheels instead of the ROCm apt repos.
#
# Note: ROCm SDK version is determined by ROCM_SDK_VERSION in
# docker/Dockerfile.rock_base
#
# =============================================================================
- group: "Build ROCk Image"
key: "build-rock-image"
depends_on: ~
steps:
# ROCk Job 1: Build ROCk Base Image (with ECR caching)
- label: ":rocm: Build ROCk Base Image"
id: build-rock-base-image
depends_on: ~
agents:
queue: cpu_queue_release
timeout_in_minutes: 360
commands:
- |
set -euo pipefail
# The SDK version, torch wheels and component branches all live in
# the Dockerfile, so its hash is the cache key.
CACHE_KEY=$$(sha256sum docker/Dockerfile.rock_base | cut -c1-16)
ECR_CACHE_TAG="public.ecr.aws/q9t5s3a7/vllm-release-repo:$${CACHE_KEY}-rock-base"
echo "========================================"
echo "ROCk Base Build Configuration"
echo "========================================"
echo " CACHE_KEY: $${CACHE_KEY}"
echo " ECR_CACHE_TAG: $${ECR_CACHE_TAG}"
echo "========================================"
# Login to ECR
aws ecr-public get-login-password --region us-east-1 | \
docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7
if docker manifest inspect "$${ECR_CACHE_TAG}" > /dev/null 2>&1; then
echo "ECR image cache HIT"
else
echo "CACHE MISS - Building from scratch..."
DOCKER_BUILDKIT=1 docker buildx build \
--file docker/Dockerfile.rock_base \
--tag "$${ECR_CACHE_TAG}" \
--build-arg USE_SCCACHE=1 \
--build-arg SCCACHE_BUCKET_NAME=vllm-build-sccache \
--build-arg SCCACHE_REGION_NAME=us-west-2 \
--build-arg SCCACHE_S3_NO_CREDENTIALS=0 \
--push \
.
fi
# Save ECR tag for downstream jobs
buildkite-agent meta-data set "rock-base-image-tag" "$${ECR_CACHE_TAG}"
env:
DOCKER_BUILDKIT: "1"
# ROCk Job 2: Build ROCk Release Docker Image
- label: ":docker: Build release image - x86_64 - ROCk"
id: build-rock-release-image
depends_on:
- step: block-build-release-images
allow_failure: true
- step: build-rock-base-image
allow_failure: false
agents:
queue: cpu_queue_release
timeout_in_minutes: 180
commands:
- |
set -euo pipefail
# Login to ECR
aws ecr-public get-login-password --region us-east-1 | \
docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7
ECR_IMAGE_TAG="$$(buildkite-agent meta-data get rock-base-image-tag 2>/dev/null || echo '')"
if [ -z "$${ECR_IMAGE_TAG}" ]; then
echo "ERROR: rock-base-image-tag metadata not found"
echo "This should have been set by the build-rock-base-image job"
exit 1
fi
docker pull "$${ECR_IMAGE_TAG}"
# Pass the base image ECR tag to downstream steps (nightly publish)
buildkite-agent meta-data set "rock-base-ecr-tag" "$${ECR_IMAGE_TAG}"
echo "========================================"
echo "Building vLLM ROCk release image with:"
echo " BASE_IMAGE: $${ECR_IMAGE_TAG}"
echo " BUILDKITE_COMMIT: $${BUILDKITE_COMMIT}"
echo "========================================"
DOCKER_BUILDKIT=1 docker build \
--build-arg max_jobs=16 \
--build-arg BASE_IMAGE="$${ECR_IMAGE_TAG}" \
--build-arg USE_SCCACHE=1 \
--build-arg SCCACHE_BUCKET_NAME=vllm-build-sccache \
--build-arg SCCACHE_REGION_NAME=us-west-2 \
--build-arg SCCACHE_S3_NO_CREDENTIALS=0 \
--build-arg INSTALL_LMCACHE=true \
--tag public.ecr.aws/q9t5s3a7/vllm-release-repo:$${BUILDKITE_COMMIT}-rock \
--target vllm-openai \
--progress plain \
-f docker/Dockerfile.rock .
docker push public.ecr.aws/q9t5s3a7/vllm-release-repo:$${BUILDKITE_COMMIT}-rock
env:
DOCKER_BUILDKIT: "1"
- label: "Publish nightly ROCk image to DockerHub"
depends_on:
- build-rock-release-image
if: build.env("NIGHTLY") == "1"
agents:
queue: small_cpu_queue_release
commands:
- "bash .buildkite/scripts/push-nightly-builds-rocm.sh rock rocm100"
# Clean up old nightly builds (keep only last 14)
- "bash .buildkite/scripts/cleanup-nightly-builds.sh nightly-rocm100- vllm/vllm-openai-rocm"
- "bash .buildkite/scripts/cleanup-nightly-builds.sh base-nightly-rocm100- vllm/vllm-openai-rocm"
plugins:
- docker-login#v3.0.0:
username: vllmbot
password-env: DOCKERHUB_TOKEN
env:
DOCKER_BUILDKIT: "1"
DOCKERHUB_USERNAME: "vllmbot"
# =============================================================================
# Publish to DockerHub and PyPI (at the end so all builds complete first)
# =============================================================================
@@ -997,8 +1227,7 @@ steps:
- create-multi-arch-manifest-ubuntu2404
- create-multi-arch-manifest-cuda-12-9-ubuntu2404
- create-manifest-xpu
- build-rocm100-release-image
- build-rocm72-release-image
- build-rocm-release-image
- input-release-version
# Wait for CPU builds if their block steps were unblocked, so publish
# doesn't race the in-progress CPU build. allow_failure lets publish
@@ -1023,7 +1252,6 @@ steps:
- "cuda-13-0-ubuntu-24-04"
- "cuda-12-9-ubuntu-24-04"
- "rocm"
- "rocm72"
- "xpu"
- "cpu"
plugins:
+66 -40
View File
@@ -2,23 +2,15 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
#
# Generate Buildkite annotation for a ROCm stack's wheel release.
#
# Usage: annotate-rocm-release.sh <base-dockerfile> [--default]
# base-dockerfile e.g. docker/Dockerfile.rocm_base or docker/Dockerfile.rocm_72_base
# --default the stack also owns the plain :latest / :v<ver> Docker Hub tags
# Generate Buildkite annotation for ROCm wheel release
set -ex
# shellcheck source=.buildkite/scripts/rocm/stack.sh
source "$(dirname "${BASH_SOURCE[0]}")/rocm/stack.sh" "${1:?Usage: $0 <base-dockerfile> [--default]}"
TAG_VARIANTS=("-${ROCM_STACK_VARIANT}")
if [[ "${2:-}" == "--default" ]]; then
TAG_VARIANTS=("" "-${ROCM_STACK_VARIANT}")
fi
ROCM_VERSION="$ROCM_STACK_VERSION"
VARIANT="$ROCM_STACK_VARIANT"
PYTHON_VERSION=$(sed -nE 's/^ARG PYTHON_VERSION="?([^" ]+)"?.*/\1/p' "$ROCM_STACK_BASE_DOCKERFILE")
PYTORCH_ROCM_ARCH=$(sed -nE 's/^ARG PYTORCH_ROCM_ARCH="?([^"]+)"?.*/\1/p' "$ROCM_STACK_BASE_DOCKERFILE")
# Extract build configuration from Dockerfile.rocm_base (single source of truth)
# Extract ROCm version dynamically from Dockerfile.rocm_base
# BASE_IMAGE format: rocm/dev-ubuntu-22.04:7.0-complete -> extracts "7.0"
ROCM_VERSION=$(grep -E '^ARG BASE_IMAGE=' docker/Dockerfile.rocm_base | sed -E 's/.*:([0-9]+\.[0-9]+).*/\1/' || echo "unknown")
PYTHON_VERSION=$(grep '^ARG PYTHON_VERSION=' docker/Dockerfile.rocm_base | sed 's/^ARG PYTHON_VERSION=//')
PYTORCH_ROCM_ARCH=$(grep '^ARG PYTORCH_ROCM_ARCH=' docker/Dockerfile.rocm_base | sed 's/^ARG PYTORCH_ROCM_ARCH=//')
# Get release version, default to 1.0.0.dev for nightly/per-commit builds
RELEASE_VERSION=$(buildkite-agent meta-data get release-version 2>/dev/null || echo "")
@@ -26,32 +18,22 @@ if [ -z "${RELEASE_VERSION}" ]; then
RELEASE_VERSION="1.0.0.dev"
fi
ROCM_BASE_CACHE_KEY=$(.buildkite/scripts/cache-rocm-base-wheels.sh key)
# S3 URLs
S3_BUCKET="${S3_BUCKET:-vllm-wheels}"
S3_REGION="${AWS_DEFAULT_REGION:-us-west-2}"
S3_HOST="${S3_BUCKET}.s3-website-${S3_REGION}.amazonaws.com"
S3_URL="http://${S3_HOST}"
WHEEL_PATH="rocm/${BUILDKITE_COMMIT}/${VARIANT}-wheels/"
S3_URL="http://${S3_BUCKET}.s3-website-${S3_REGION}.amazonaws.com"
DOCKER_CMDS="docker pull ${ROCM_STACK_ECR_BASE}
docker pull ${ROCM_STACK_ECR_IMAGE}"
for suffix in "${TAG_VARIANTS[@]}"; do
for tag in "latest${suffix}" "v${RELEASE_VERSION}${suffix}"; do
DOCKER_CMDS+="
docker tag ${ROCM_STACK_ECR_BASE} vllm/vllm-openai-rocm:${tag}-base
docker tag ${ROCM_STACK_ECR_IMAGE} vllm/vllm-openai-rocm:${tag}
docker push vllm/vllm-openai-rocm:${tag}-base
docker push vllm/vllm-openai-rocm:${tag}"
done
done
buildkite-agent annotate --style 'success' --context "rocm-release-workflow-${VARIANT}" << EOF
## ROCm ${ROCM_VERSION} Wheel and Docker Image Releases
# Format ROCm version for path (e.g., "7.1" -> "rocm710")
ROCM_VERSION_PATH="rocm$(echo "${ROCM_VERSION}" | tr -d '.')"
ROCM_PATH="rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}"
buildkite-agent annotate --style 'success' --context 'rocm-release-workflow' << EOF
## ROCm Wheel and Docker Image Releases
### Build Configuration
| Setting | Value |
|---------|-------|
| **ROCm Version** | ${ROCM_VERSION} |
| **Index Variant** | ${VARIANT} |
| **Python Version** | ${PYTHON_VERSION} |
| **GPU Architectures** | ${PYTORCH_ROCM_ARCH} |
| **Branch** | \`${BUILDKITE_BRANCH}\` |
@@ -59,20 +41,49 @@ buildkite-agent annotate --style 'success' --context "rocm-release-workflow-${VA
### :package: Installation
**Install from this build (by commit):**
\`\`\`bash
# This build (by commit)
pip install vllm --extra-index-url ${S3_URL}/rocm/${BUILDKITE_COMMIT}/${VARIANT}/ --trusted-host ${S3_HOST}
# Nightly (if published)
pip install vllm --extra-index-url ${S3_URL}/rocm/nightly/${VARIANT}/ --trusted-host ${S3_HOST}
pip install vllm --extra-index-url ${S3_URL}/${ROCM_PATH}/ --trusted-host ${S3_BUCKET}.s3-website-${S3_REGION}.amazonaws.com
# Example for ROCm ${ROCM_VERSION}:
pip install vllm --extra-index-url ${S3_URL}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/ --trusted-host ${S3_BUCKET}.s3-website-${S3_REGION}.amazonaws.com
\`\`\`
**Install from nightly (if published):**
\`\`\`bash
pip install vllm --extra-index-url ${S3_URL}/rocm/nightly/ --trusted-host ${S3_BUCKET}.s3-website-${S3_REGION}.amazonaws.com
\`\`\`
### :floppy_disk: Download Wheels Directly
\`\`\`bash
aws s3 ls s3://${S3_BUCKET}/${WHEEL_PATH}
aws s3 cp --recursive --exclude "*" --include "*.whl" s3://${S3_BUCKET}/${WHEEL_PATH} .
# List all ROCm wheels
aws s3 ls s3://${S3_BUCKET}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/
# Download specific wheels
aws s3 cp s3://${S3_BUCKET}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/vllm-*.whl .
aws s3 cp s3://${S3_BUCKET}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/torch-*.whl .
aws s3 cp s3://${S3_BUCKET}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/triton-*.whl .
aws s3 cp s3://${S3_BUCKET}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/triton-kernels-*.whl .
aws s3 cp s3://${S3_BUCKET}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/torchvision-*.whl .
aws s3 cp s3://${S3_BUCKET}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/torchaudio-*.whl .
aws s3 cp s3://${S3_BUCKET}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/amdsmi-*.whl .
aws s3 cp s3://${S3_BUCKET}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/amd_aiter-*.whl .
aws s3 cp s3://${S3_BUCKET}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/flash-attn-*.whl .
\`\`\`
### :gear: Included Packages
- **vllm**: vLLM with ROCm support
- **torch**: PyTorch built for ROCm ${ROCM_VERSION}
- **triton**: Triton
- **triton-kernels**: Triton kernels
- **torchvision**: TorchVision for ROCm PyTorch
- **torchaudio**: Torchaudio for ROCm PyTorch
- **amdsmi**: AMD SMI Python bindings
- **amd_aiter**: Aiter for ROCm
- **flash-attn**: Flash Attention for ROCm
### :warning: Notes
- These wheels are built for **ROCm ${ROCM_VERSION}** and will NOT work with CUDA GPUs
- Supported GPU architectures: ${PYTORCH_ROCM_ARCH}
@@ -80,8 +91,23 @@ aws s3 cp --recursive --exclude "*" --include "*.whl" s3://${S3_BUCKET}/${WHEEL_
### :package: Docker Image Release
To download and upload the image:
\`\`\`
${DOCKER_CMDS}
docker pull public.ecr.aws/q9t5s3a7/vllm-release-repo:${BUILDKITE_COMMIT}-rocm-base
docker pull public.ecr.aws/q9t5s3a7/vllm-release-repo:${BUILDKITE_COMMIT}-rocm
docker tag public.ecr.aws/q9t5s3a7/vllm-release-repo:${ROCM_BASE_CACHE_KEY}-rocm-base vllm/vllm-openai-rocm:${BUILDKITE_COMMIT}-base
docker tag vllm/vllm-openai-rocm:${BUILDKITE_COMMIT}-base vllm/vllm-openai-rocm:latest-base
docker tag vllm/vllm-openai-rocm:${BUILDKITE_COMMIT}-base vllm/vllm-openai-rocm:v${RELEASE_VERSION}-base
docker push vllm/vllm-openai-rocm:latest-base
docker push vllm/vllm-openai-rocm:v${RELEASE_VERSION}-base
docker tag public.ecr.aws/q9t5s3a7/vllm-release-repo:${BUILDKITE_COMMIT}-rocm vllm/vllm-openai-rocm:${BUILDKITE_COMMIT}
docker tag vllm/vllm-openai-rocm:${BUILDKITE_COMMIT} vllm/vllm-openai-rocm:latest
docker tag vllm/vllm-openai-rocm:${BUILDKITE_COMMIT} vllm/vllm-openai-rocm:v${RELEASE_VERSION}
docker push vllm/vllm-openai-rocm:latest
docker push vllm/vllm-openai-rocm:v${RELEASE_VERSION}
\`\`\`
EOF
+1 -2
View File
@@ -25,7 +25,6 @@ export CMAKE_BUILD_PARALLEL_LEVEL="${CMAKE_BUILD_PARALLEL_LEVEL:-4}"
uv venv --python 3.12
uv pip install -r requirements/build/cpu.txt --index-strategy unsafe-best-match
uv build --wheel --no-build-isolation -o artifacts/dist \
--config-setting=--build-option=--py-limited-api=cp38
uv build --wheel --no-build-isolation -o artifacts/dist
ls -l artifacts/dist/*.whl
+34 -28
View File
@@ -5,33 +5,39 @@
# Cache helper for ROCm base wheels
#
# This script manages caching of pre-built ROCm base wheels (torch, triton, etc.)
# to avoid rebuilding them when the stack's base Dockerfile hasn't changed.
# to avoid rebuilding them when Dockerfile.rocm_base hasn't changed.
#
# Usage: cache-rocm-base-wheels.sh <base-dockerfile> <command>
# base-dockerfile e.g. docker/Dockerfile.rocm_base or docker/Dockerfile.rocm_72_base
# Commands:
# check - Check if cache exists, outputs "hit" or "miss"
# upload - Upload wheels to cache
# download - Download wheels from cache
# key - Output the cache key
# path - Output the full S3 cache path
# Usage:
# cache-rocm-base-wheels.sh check - Check if cache exists, outputs "hit" or "miss"
# cache-rocm-base-wheels.sh upload - Upload wheels to cache
# cache-rocm-base-wheels.sh download - Download wheels from cache
# cache-rocm-base-wheels.sh key - Output the cache key
#
# Environment variables:
# S3_BUCKET - S3 bucket name (default: vllm-wheels)
#
# The cache key hashes the base Dockerfile and the repo files it COPYs in, which
# capture the ROCm version and every component pin (see rocm/stack.sh).
# Note: ROCm version is determined by BASE_IMAGE in Dockerfile.rocm_base,
# so changes to ROCm version are captured by the Dockerfile hash.
set -euo pipefail
BUCKET="${S3_BUCKET:-vllm-wheels}"
DOCKERFILE="docker/Dockerfile.rocm_base"
CACHE_PREFIX="rocm/cache"
# shellcheck source=.buildkite/scripts/rocm/stack.sh
source "$(dirname "${BASH_SOURCE[0]}")/rocm/stack.sh" "${1:?Usage: $0 <base-dockerfile> <command>}"
shift
CACHE_KEY="$ROCM_STACK_WHEEL_KEY"
WHEELS_DIR="$ROCM_STACK_BASE_WHEELS_DIR"
# Generate hash from Dockerfile content + build args
generate_cache_key() {
# Include Dockerfile content
if [[ ! -f "$DOCKERFILE" ]]; then
echo "ERROR: Dockerfile not found: $DOCKERFILE" >&2
exit 1
fi
local dockerfile_hash=$(sha256sum "$DOCKERFILE" | cut -c1-16)
echo "${dockerfile_hash}"
}
CACHE_KEY=$(generate_cache_key)
CACHE_PATH="s3://${BUCKET}/${CACHE_PREFIX}/${CACHE_KEY}/"
case "${1:-}" in
@@ -61,19 +67,19 @@ case "${1:-}" in
echo "Cache path: ${CACHE_PATH}"
echo ""
if [[ ! -d "${WHEELS_DIR}" ]]; then
echo "ERROR: ${WHEELS_DIR} directory not found" >&2
if [[ ! -d "artifacts/rocm-base-wheels" ]]; then
echo "ERROR: artifacts/rocm-base-wheels directory not found" >&2
exit 1
fi
WHEEL_COUNT=$(find "${WHEELS_DIR}" -maxdepth 1 -name '*.whl' 2>/dev/null | wc -l)
WHEEL_COUNT=$(find artifacts/rocm-base-wheels -maxdepth 1 -name '*.whl' 2>/dev/null | wc -l)
if [[ "$WHEEL_COUNT" -eq 0 ]]; then
echo "ERROR: No wheels found in ${WHEELS_DIR}/" >&2
echo "ERROR: No wheels found in artifacts/rocm-base-wheels/" >&2
exit 1
fi
echo "Uploading $WHEEL_COUNT wheels..."
aws s3 cp --recursive "${WHEELS_DIR}/" "${CACHE_PATH}"
aws s3 cp --recursive artifacts/rocm-base-wheels/ "${CACHE_PATH}"
echo ""
echo "Cache upload complete!"
@@ -87,17 +93,17 @@ case "${1:-}" in
echo "Cache key: ${CACHE_KEY}"
echo "Cache path: ${CACHE_PATH}"
echo ""
mkdir -p "${WHEELS_DIR}"
mkdir -p artifacts/rocm-base-wheels
# Use sync with include/exclude to only download .whl files
aws s3 sync "${CACHE_PATH}" "${WHEELS_DIR}/" \
aws s3 sync "${CACHE_PATH}" artifacts/rocm-base-wheels/ \
--exclude "*" \
--include "*.whl"
echo ""
echo "Downloaded wheels:"
find "${WHEELS_DIR}" -maxdepth 1 -name '*.whl' -exec ls -lh {} \;
WHEEL_COUNT=$(find "${WHEELS_DIR}" -maxdepth 1 -name '*.whl' 2>/dev/null | wc -l)
find artifacts/rocm-base-wheels -maxdepth 1 -name '*.whl' -exec ls -lh {} \;
WHEEL_COUNT=$(find artifacts/rocm-base-wheels -maxdepth 1 -name '*.whl' 2>/dev/null | wc -l)
echo ""
echo "Total: $WHEEL_COUNT wheels"
echo "========================================"
@@ -112,12 +118,12 @@ case "${1:-}" in
;;
*)
echo "Usage: $0 <base-dockerfile> {check|upload|download|key|path}" >&2
echo "Usage: $0 {check|upload|download|key|path}" >&2
echo "" >&2
echo "Commands:" >&2
echo " check - Check if cache exists, outputs 'hit' or 'miss'" >&2
echo " upload - Upload wheels from ${WHEELS_DIR}/ to cache" >&2
echo " download - Download wheels from cache to ${WHEELS_DIR}/" >&2
echo " upload - Upload wheels from artifacts/rocm-base-wheels/ to cache" >&2
echo " download - Download wheels from cache to artifacts/rocm-base-wheels/" >&2
echo " key - Output the cache key" >&2
echo " path - Output the full S3 cache path" >&2
exit 1
+54 -7
View File
@@ -17,7 +17,7 @@ DEFAULT_REPO_SLUG="vllm-project/vllm"
DEFAULT_CI_HCL_SOURCE="docker/ci-rocm.hcl"
DEFAULT_CI_BASE_CONTENT_FILES=".dockerignore requirements/common.txt requirements/rocm.txt requirements/test/rocm.txt tools/install_torchcodec_rocm.sh rust-toolchain.toml tests/vllm_test_utils"
DEFAULT_CI_BASE_DOCKERFILE="docker/Dockerfile.rocm"
DEFAULT_CI_BASE_DOCKERFILE_STAGES="base rust_toolchain_input_0 rust-toolchain-input rust-toolchain build_nixl lmcache_source build_lmcache build_deepep mori_base ci_base"
DEFAULT_CI_BASE_DOCKERFILE_STAGES="base rust_toolchain_input_0 rust-toolchain-input rust-toolchain build_nixl lmcache_source build_lmcache build_rocshmem build_deepep mori_base ci_base"
DEFAULT_CI_BASE_METADATA_VERSION="3"
# ROCm CI forces REMOTE_VLLM=0, so content identity covers only the selected
# local-source stages rather than unreachable remote-fetch alternatives.
@@ -555,6 +555,9 @@ configure_custom_rocm_stages() {
CI_BASE_DOCKERFILE_STAGES="${CI_BASE_DOCKERFILE_STAGES:-${stages}}"
ROCM_CSRC_DOCKERFILE_STAGES="${ROCM_CSRC_DOCKERFILE_STAGES:-${stages}}"
ROCM_RUST_DOCKERFILE_STAGES="${ROCM_RUST_DOCKERFILE_STAGES:-${stages}}"
if [[ " ${stages} " != *" build_rocshmem "* ]]; then
unset ROCSHMEM_BRANCH ROCSHMEM_CACHE_KEY DEEPEP_CACHE_KEY
fi
}
compose_dependency_cache_key() {
@@ -1721,11 +1724,14 @@ ci_base_metadata_pairs() {
metadata_pair "vllm.rocm.nixl_commit" "${NIXL_BRANCH:-$(resolve_dockerfile_arg_value "${dockerfile}" "NIXL_BRANCH")}"
metadata_pair "vllm.rocm.ucx_repo" "$(resolve_dockerfile_arg_value "${dockerfile}" "UCX_REPO")"
metadata_pair "vllm.rocm.ucx_commit" "${UCX_BRANCH:-$(resolve_dockerfile_arg_value "${dockerfile}" "UCX_BRANCH")}"
metadata_pair "vllm.rocm.rocshmem_repo" "$(resolve_dockerfile_arg_value "${dockerfile}" "ROCSHMEM_REPO")"
metadata_pair "vllm.rocm.rocshmem_commit" "${ROCSHMEM_BRANCH:-$(resolve_dockerfile_arg_value "${dockerfile}" "ROCSHMEM_BRANCH")}"
metadata_pair "vllm.rocm.deepep_repo" "$(resolve_dockerfile_arg_value "${dockerfile}" "DEEPEP_REPO")"
metadata_pair "vllm.rocm.deepep_commit" "${DEEPEP_BRANCH:-$(resolve_dockerfile_arg_value "${dockerfile}" "DEEPEP_BRANCH")}"
metadata_pair "vllm.rocm.deepep_nic" "$(resolve_dockerfile_arg_value "${dockerfile}" "DEEPEP_NIC")"
metadata_pair "vllm.rocm.deepep_rocm_arch" "$(resolve_dockerfile_arg_value "${dockerfile}" "DEEPEP_ROCM_ARCH")"
metadata_pair "vllm.rocm.nixl_cache_key" "${NIXL_CACHE_KEY:-}"
metadata_pair "vllm.rocm.rocshmem_cache_key" "${ROCSHMEM_CACHE_KEY:-}"
metadata_pair "vllm.rocm.deepep_cache_key" "${DEEPEP_CACHE_KEY:-}"
}
@@ -2308,7 +2314,7 @@ extract_dependency_pins() {
return 0
fi
for var in NIXL_BRANCH UCX_BRANCH DEEPEP_BRANCH; do
for var in NIXL_BRANCH UCX_BRANCH ROCSHMEM_BRANCH DEEPEP_BRANCH; do
if [[ -n "${!var:-}" ]]; then
echo "Using provided ${var}: ${!var}"
continue
@@ -2330,14 +2336,17 @@ compute_dependency_cache_keys() {
local dockerfile_rocm=""
local nixl_branch=""
local ucx_branch=""
local rocshmem_branch=""
local deepep_branch=""
local nixl_material=""
local rocshmem_material=""
local deepep_material=""
bake_dir=$(dirname "${VLLM_BAKE_FILE}")
dockerfile_rocm="${CI_BASE_DOCKERFILE:-${bake_dir}/Dockerfile.rocm}"
nixl_branch=$(resolve_dockerfile_arg_value "${dockerfile_rocm}" "NIXL_BRANCH")
ucx_branch=$(resolve_dockerfile_arg_value "${dockerfile_rocm}" "UCX_BRANCH")
rocshmem_branch=$(resolve_dockerfile_arg_value "${dockerfile_rocm}" "ROCSHMEM_BRANCH")
deepep_branch=$(resolve_dockerfile_arg_value "${dockerfile_rocm}" "DEEPEP_BRANCH")
if [[ -n "${nixl_branch}" && -n "${ucx_branch}" ]]; then
@@ -2351,11 +2360,22 @@ compute_dependency_cache_keys() {
echo "NIXL dependency cache key: ${NIXL_CACHE_KEY}"
fi
if [[ -n "${deepep_branch}" ]]; then
deepep_material=$(compose_stage_cache_material "${dockerfile_rocm}" "base build_deepep")
if [[ -n "${rocshmem_branch}" ]]; then
rocshmem_material=$(compose_stage_cache_material "${dockerfile_rocm}" "base build_rocshmem")
ROCSHMEM_CACHE_KEY=$(
compose_dependency_cache_key \
"${rocshmem_branch}" \
"${rocshmem_material}"
)
export ROCSHMEM_CACHE_KEY
echo "ROCShmem dependency cache key: ${ROCSHMEM_CACHE_KEY}"
fi
if [[ -n "${deepep_branch}" && -n "${rocshmem_branch}" ]]; then
deepep_material=$(compose_stage_cache_material "${dockerfile_rocm}" "base build_rocshmem build_deepep")
DEEPEP_CACHE_KEY=$(
compose_dependency_cache_key \
"${deepep_branch}" \
"${deepep_branch}-rocshmem-${rocshmem_branch}" \
"${deepep_material}"
)
export DEEPEP_CACHE_KEY
@@ -2395,11 +2415,18 @@ dependency_cache_ref_for_target() {
printf '%s\n' "${cache_repo}:nixl-rocm-${NIXL_BRANCH}-ucx-${UCX_BRANCH:-}"
fi
;;
rocshmem-rocm-ci)
if [[ -n "${ROCSHMEM_CACHE_KEY:-}" ]]; then
printf '%s\n' "${cache_repo}:rocshmem-rocm-${ROCSHMEM_CACHE_KEY}"
elif [[ -n "${ROCSHMEM_BRANCH:-}" ]]; then
printf '%s\n' "${cache_repo}:rocshmem-rocm-${ROCSHMEM_BRANCH}"
fi
;;
deepep-rocm-ci)
if [[ -n "${DEEPEP_CACHE_KEY:-}" ]]; then
printf '%s\n' "${cache_repo}:deepep-rocm-${DEEPEP_CACHE_KEY}"
elif [[ -n "${DEEPEP_BRANCH:-}" ]]; then
printf '%s\n' "${cache_repo}:deepep-rocm-${DEEPEP_BRANCH}"
printf '%s\n' "${cache_repo}:deepep-rocm-${DEEPEP_BRANCH}-rocshmem-${ROCSHMEM_BRANCH:-}"
fi
;;
esac
@@ -2417,6 +2444,7 @@ add_dependency_cache_target() {
resolve_ci_base_dependency_targets() {
local mode="${ROCM_DEP_CACHE_EXPORT_MODE:-missing}"
local nixl_ref=""
local rocshmem_ref=""
local deepep_ref=""
[[ "${TARGET}" == "ci-base-rocm-ci-with-deps" ]] || return 0
@@ -2424,7 +2452,7 @@ resolve_ci_base_dependency_targets() {
case "${mode}" in
always)
echo "ROCM_DEP_CACHE_EXPORT_MODE=always; exporting all dependency caches serially"
for target in nixl-rocm-ci deepep-rocm-ci; do
for target in nixl-rocm-ci rocshmem-rocm-ci deepep-rocm-ci; do
if [[ -n "$(dependency_cache_ref_for_target "${target}")" ]]; then
add_dependency_cache_target "${target}"
fi
@@ -2454,6 +2482,16 @@ resolve_ci_base_dependency_targets() {
fi
fi
if [[ "${mode}" != "always" && -n "${ROCSHMEM_CACHE_KEY:-}" ]]; then
rocshmem_ref=$(dependency_cache_ref_for_target "rocshmem-rocm-ci")
if dependency_cache_ref_exists "${rocshmem_ref}"; then
echo "ROCShmem dependency cache exists: ${rocshmem_ref}"
else
echo "ROCShmem dependency cache missing; will seed: ${rocshmem_ref}"
add_dependency_cache_target "rocshmem-rocm-ci"
fi
fi
if [[ "${mode}" != "always" && -n "${DEEPEP_CACHE_KEY:-}" ]]; then
deepep_ref=$(dependency_cache_ref_for_target "deepep-rocm-ci")
if dependency_cache_ref_exists "${deepep_ref}"; then
@@ -2464,6 +2502,15 @@ resolve_ci_base_dependency_targets() {
fi
fi
# DeepEP inherits from ROCShmem. If ROCShmem is being seeded, seed DeepEP too
# so the pair stays consistent for future ci_base rebuilds.
if printf '%s\n' "${DEPENDENCY_CACHE_TARGETS[@]}" | grep -qx "rocshmem-rocm-ci" \
&& ! printf '%s\n' "${DEPENDENCY_CACHE_TARGETS[@]}" | grep -qx "deepep-rocm-ci" \
&& [[ -n "${DEEPEP_BRANCH:-}" ]]; then
echo "ROCShmem cache is missing; also seeding DeepEP cache"
add_dependency_cache_target "deepep-rocm-ci"
fi
BAKE_TARGETS=("ci-base-rocm-ci")
if [[ ${#DEPENDENCY_CACHE_TARGETS[@]} -eq 0 ]]; then
echo "All dependency caches exist; building ci_base without dependency cache exports"
+2 -5
View File
@@ -5,17 +5,14 @@ set -ex
# Clean up old nightly builds from DockerHub, keeping only the last 14 builds
# This script uses DockerHub API to list and delete old tags with specified prefix
# Tags starting with "nightly-dev" are always excluded from deletion
# Usage: cleanup-nightly-builds.sh [TAG_PREFIX] [REPO] [EXCLUDE_PREFIX]
# Usage: cleanup-nightly-builds.sh [TAG_PREFIX] [REPO]
# Example: cleanup-nightly-builds.sh "nightly-"
# Example: cleanup-nightly-builds.sh "cu130-nightly-"
# Example: cleanup-nightly-builds.sh "nightly-" "vllm/vllm-openai-rocm"
# Example: cleanup-nightly-builds.sh "nightly-" "vllm/vllm-openai-rocm" "nightly-rocm"
# (EXCLUDE_PREFIX skips tags owned by another variant sharing TAG_PREFIX)
# Get tag prefix and repo from arguments
TAG_PREFIX="${1:-nightly-}"
REPO="${2:-vllm/vllm-openai}"
EXCLUDE_PREFIX="${3:-}"
echo "Cleaning up tags with prefix: $TAG_PREFIX in repository: $REPO"
@@ -60,7 +57,7 @@ get_all_tags() {
# Get both last_updated timestamp and tag name, separated by |
# Exclude mutable architecture tags and nightly-dev tags from cleanup
local tags=$(echo "$response" | jq -r --arg prefix "$TAG_PREFIX" --arg exclude "$EXCLUDE_PREFIX" '.results[] | select(.name | startswith($prefix)) | select($exclude == "" or (.name | startswith($exclude) | not)) | select(.name != ($prefix + "x86_64") and .name != ($prefix + "aarch64")) | select(.name | startswith("nightly-dev") | not) | "\(.last_updated)|\(.name)"')
local tags=$(echo "$response" | jq -r --arg prefix "$TAG_PREFIX" '.results[] | select(.name | startswith($prefix)) | select(.name != ($prefix + "x86_64") and .name != ($prefix + "aarch64")) | select(.name | startswith("nightly-dev") | not) | "\(.last_updated)|\(.name)"')
if [ -z "$tags" ]; then
break
+1 -6
View File
@@ -227,12 +227,7 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# five-minute lifetime and the loop above can run to the build-age deadline.
# Build 91827 waited 6h48m and then took a 401 on all 360 callbacks from a
# token that had expired hours earlier.
# Posting is paced under the relay's rate limit, which takes several minutes
# for a full nightly, so ask for a longer lifetime than the default. Fall back
# to the default if this agent rejects --lifetime.
OIDC_TOKEN="$(buildkite-agent oidc request-token --skip-redaction --audience "${AUDIENCE}" \
--lifetime 1800 2>/dev/null \
|| buildkite-agent oidc request-token --skip-redaction --audience "${AUDIENCE}" 2>/dev/null)"
OIDC_TOKEN="$(buildkite-agent oidc request-token --skip-redaction --audience "${AUDIENCE}" 2>/dev/null)"
if [[ -z "${OIDC_TOKEN}" ]]; then
echo "could not mint a Buildkite OIDC token after the wait -- skipping report"
exit 0
-16
View File
@@ -24,9 +24,7 @@ from __future__ import annotations
import argparse
import json
import os
import sys
import time
import urllib.error
import urllib.request
@@ -45,14 +43,6 @@ _CONCLUSION = {
}
# The relay rate-limits callbacks per repo over a sliding 60 s window
# (RATE_LIMIT_PER_MIN in pytorch/test-infra), and a nightly has far more jobs
# than that. Pace below the limit, and back off on a 429 for about a minute in
# total so the last retry lands in a new window.
_POST_INTERVAL_S = float(os.environ.get("CRCR_POST_INTERVAL_S", "0.65"))
_RATE_LIMIT_BACKOFF_S = (2, 4, 8, 16, 32)
def post(url: str, token: str, body: dict, timeout: int = 30) -> tuple[int, str]:
req = urllib.request.Request(
url,
@@ -123,12 +113,6 @@ def main() -> int:
ok += 1
continue
status, detail = post(args.callback_url, args.oidc_token, body)
for delay in _RATE_LIMIT_BACKOFF_S:
if status != 429:
break
time.sleep(delay)
status, detail = post(args.callback_url, args.oidc_token, body)
time.sleep(_POST_INTERVAL_S)
if 200 <= status < 300:
ok += 1
else:
+9 -63
View File
@@ -12,7 +12,7 @@ from dataclasses import asdict, dataclass
from datetime import datetime
from pathlib import Path
from typing import Any
from urllib.parse import quote, unquote, urlparse
from urllib.parse import quote
import regex as re
@@ -51,8 +51,6 @@ class WheelFileInfo:
platform_tag: str
variant: str | None
filename: str
# Absolute URL for files hosted elsewhere (see --external-links)
url: str | None = None
def parse_from_filename(file: str) -> WheelFileInfo:
@@ -112,29 +110,6 @@ def parse_from_filename(file: str) -> WheelFileInfo:
)
def parse_external_url(url: str) -> WheelFileInfo:
"""Parse an absolute wheel or sdist URL hosted on another index."""
filename = unquote(urlparse(url).path.rsplit("/", 1)[-1])
if filename.endswith(".whl"):
info = parse_from_filename(filename)
else:
match = re.match(r"^(?P<name>.+)-(?P<version>[^-]+)\.(tar\.gz|zip)$", filename)
if not match:
raise ValueError(f"Unsupported external file: {filename}")
info = WheelFileInfo(
package_name=match.group("name"),
version=match.group("version"),
build_tag=None,
python_tag="source",
abi_tag="none",
platform_tag="any",
variant=None,
filename=filename,
)
info.url = url
return info
def generate_project_list(subdir_names: list[str], comment: str = "") -> str:
"""Generate project list HTML content linking to each project & variant subdirectory."""
href_tags = []
@@ -154,19 +129,14 @@ def generate_package_index_and_metadata(
href_tags = []
metadata = []
for file in sorted(wheel_files, key=lambda x: x.filename):
if file.url:
file_path_quoted = file.url
else:
relative_path = (
wheel_base_dir.relative_to(index_base_dir, walk_up=True) / file.filename
)
# handle with '+' in URL, and avoid double-encoding '/' and already-encoded '%2B'
# NOTE: this is AWS S3 specific behavior!
file_path_quoted = quote(relative_path.as_posix(), safe=":%/")
relative_path = (
wheel_base_dir.relative_to(index_base_dir, walk_up=True) / file.filename
)
# handle with '+' in URL, and avoid double-encoding '/' and already-encoded '%2B'
# NOTE: this is AWS S3 specific behavior!
file_path_quoted = quote(relative_path.as_posix(), safe=":%/")
href_tags.append(f' <a href="{file_path_quoted}">{file.filename}</a><br/>')
file_meta = asdict(file)
if file_meta["url"] is None:
del file_meta["url"]
file_meta["path"] = file_path_quoted
metadata.append(file_meta)
index_str = INDEX_HTML_TEMPLATE.format(items="\n".join(href_tags), comment=comment)
@@ -181,7 +151,6 @@ def generate_index_and_metadata(
default_variant: str | None = None,
alias_to_default: str | None = None,
comment: str = "",
external_urls: list[str] | None = None,
):
"""Generate index for all wheel files.
@@ -191,8 +160,6 @@ def generate_index_and_metadata(
index_base_dir (Path): Base directory to store index files.
default_variant (str | None): The default variant name, if any.
alias_to_default (str | None): Alias variant name for the default variant, if any.
external_urls (list[str] | None): Absolute URLs of files hosted on another index,
listed in the generated indices without being copied.
comment (str | None): Optional comment to include in the generated HTML files.
First, parse all wheel files to extract metadata.
@@ -247,7 +214,6 @@ def generate_index_and_metadata(
"""
parsed_files = [parse_from_filename(f) for f in whl_files]
parsed_files += [parse_external_url(u) for u in external_urls or []]
if not parsed_files:
print("No wheel files found, skipping index generation.")
@@ -266,14 +232,10 @@ def generate_index_and_metadata(
print(f"Detected ROCm variant from vllm: {rocm_variant}")
break
# Apply ROCm variant to all wheels without a variant. Dependencies with their
# own ROCm local version (e.g. TheRock's torch-2.12.0+rocm10.0.0) belong to
# the same variant rather than one of their own.
# Apply ROCm variant to all wheels without a variant
if rocm_variant:
for file in parsed_files:
if file.variant is None or (
file.variant.startswith("rocm") and file.variant != rocm_variant
):
if file.variant is None:
file.variant = rocm_variant
print(f"Inherited variant '{rocm_variant}' for {file.filename}")
@@ -404,13 +366,6 @@ if __name__ == "__main__":
default=None,
help="Alias variant name for the default variant",
)
parser.add_argument(
"--external-links",
type=str,
default=None,
help="Optional file with one absolute URL per line for packages hosted on "
"another index (listed alongside, but not copied into, this index)",
)
parser.add_argument(
"--comment",
type=str,
@@ -508,14 +463,5 @@ if __name__ == "__main__":
default_variant=None,
alias_to_default=args.alias_to_default,
comment=args.comment.strip(),
external_urls=(
[
line.strip()
for line in Path(args.external_links).read_text().splitlines()
if line.strip() and not line.startswith("#")
]
if args.external_links
else None
),
)
print(f"Successfully generated index and metadata in {output_dir}")
@@ -43,7 +43,6 @@ kernel_tests() {
docker exec "$CONTAINER_NAME" bash -c "
set -e
pytest -x -v -s tests/kernels/test_onednn.py
pytest -x -v -s tests/kernels/test_cpu_fused_sampling.py
pytest -x -v -s tests/kernels/attention/test_cpu_attn.py
pytest -x -v -s tests/kernels/core/test_cpu_activation.py
pytest -x -v -s tests/kernels/moe/test_cpu_fused_moe.py
@@ -19,9 +19,10 @@ DISK_USAGE_THRESHOLD=${DISK_USAGE_THRESHOLD:-80}
CACHE_MAX_AGE=${CACHE_MAX_AGE:-24h}
# Reclaim disk only when the host is under pressure, aging out anything unused
# for less than CACHE_MAX_AGE so hot layers survive (`--filter until=<N>h`).
# `docker buildx prune --max-used-space` is a no-op on this host's `docker`
# driver (BuildKit embedded in dockerd never enforces the size cap), so we don't use it.
# for less than CACHE_MAX_AGE so hot layers survive -- same `--filter
# until=<N>h` pattern the TPU CI scripts already rely on. `docker buildx
# prune --max-used-space` is a no-op on this host's `docker` driver (BuildKit
# embedded in dockerd never enforces the size cap), so we don't use it.
prune_if_disk_pressure() {
local docker_root disk_usage
docker_root=$(docker info -f '{{.DockerRootDir}}' 2>/dev/null || true)
@@ -0,0 +1,29 @@
#!/bin/bash
# This script build the GH200 docker image and run the offline inference inside the container.
# It serves a sanity check for compilation and basic model usage.
set -ex
# Skip the new torch installation during build since we are using the specified version for arm64 in the Dockerfile
python3 tools/use_existing_torch.py
# Try building the docker image
DOCKER_BUILDKIT=1 docker build . \
--file docker/Dockerfile \
--target vllm-openai \
--platform "linux/arm64" \
-t gh200-test \
--build-arg max_jobs=66 \
--build-arg nvcc_threads=2 \
--build-arg BUILD_BASE_IMAGE=pytorch/manylinuxaarch64-builder:cuda13.0-b8b5f17a7d9ccfc25bbc5cf17b3fcea12964a042 \
--build-arg torch_cuda_arch_list="9.0+PTX"
# Setup cleanup
remove_docker_container() { docker rm -f gh200-test || true; }
trap remove_docker_container EXIT
remove_docker_container
# Run the image and test offline inference
docker run -e HF_TOKEN -e VLLM_WORKER_MULTIPROC_METHOD=spawn -v /root/.cache/huggingface:/root/.cache/huggingface --name gh200-test --gpus=all --entrypoint="" gh200-test bash -c '
python3 examples/basic/offline_inference/generate.py --model meta-llama/Llama-3.2-1B
'
@@ -57,7 +57,7 @@ case "${test_suite}" in
cd tests
pytest -v -s quantization/test_auto_round.py
pytest -v -s quantization/test_online.py --deselect=tests/quantization/test_online.py::test_online_quantization_loads_real_weights --deselect="tests/quantization/test_online.py::test_online_quantization[False-requantization_mxfp8_ptcp_fp8]"
pytest -v -s quantization/test_online.py --deselect=tests/quantization/test_online.py::test_online_quantization_loads_real_weights
;;
compressed-tensors-fp8)
cd tests
+166
View File
@@ -0,0 +1,166 @@
#!/bin/bash
set -xu
remove_docker_container() {
docker rm -f tpu-test || true;
}
trap remove_docker_container EXIT
# Remove the container that might not be cleaned up in the previous run.
remove_docker_container
# Build the docker image.
docker build -f docker/Dockerfile.tpu -t vllm-tpu .
# Set up cleanup.
cleanup_docker() {
# Get Docker's root directory
docker_root=$(docker info -f '{{.DockerRootDir}}')
if [ -z "$docker_root" ]; then
echo "Failed to determine Docker root directory."
exit 1
fi
echo "Docker root directory: $docker_root"
# Check disk usage of the filesystem where Docker's root directory is located
disk_usage=$(df "$docker_root" | tail -1 | awk '{print $5}' | sed 's/%//')
# Define the threshold
threshold=70
if [ "$disk_usage" -gt "$threshold" ]; then
echo "Disk usage is above $threshold%. Cleaning up Docker images and volumes..."
# Remove dangling images (those that are not tagged and not used by any container)
docker image prune -f
# Remove unused volumes / force the system prune for old images as well.
docker volume prune -f && docker system prune --force --filter "until=72h" --all
echo "Docker images and volumes cleanup completed."
else
echo "Disk usage is below $threshold%. No cleanup needed."
fi
}
cleanup_docker
# For HF_TOKEN.
source /etc/environment
docker run --privileged --net host --shm-size=16G -it \
-e "HF_TOKEN=$HF_TOKEN" --name tpu-test \
vllm-tpu /bin/bash -c '
set -e # Exit immediately if a command exits with a non-zero status.
set -u # Treat unset variables as an error.
echo "--- Starting script inside Docker container ---"
# Create results directory
RESULTS_DIR=$(mktemp -d)
# If mktemp fails, set -e will cause the script to exit.
echo "Results will be stored in: $RESULTS_DIR"
# Install dependencies
echo "--- Installing Python dependencies ---"
python3 -m pip install --progress-bar off git+https://github.com/thuml/depyf.git \
&& python3 -m pip install --progress-bar off pytest pytest-asyncio tpu-info \
&& python3 -m pip install --progress-bar off "lm-eval[api]>=0.4.12" \
&& python3 -m pip install --progress-bar off hf-transfer tblib==3.1.0
echo "--- Python dependencies installed ---"
export VLLM_XLA_CHECK_RECOMPILATION=1
export VLLM_XLA_CACHE_PATH=
echo "--- Hardware Information ---"
# tpu-info
echo "--- Starting Tests ---"
set +e
overall_script_exit_code=0
# --- Test Definitions ---
# If a test fails, this function will print logs and will not cause the main script to exit.
run_test() {
local test_num=$1
local test_name=$2
local test_command=$3
local log_file="$RESULTS_DIR/test_${test_num}.log"
local actual_exit_code
echo "--- TEST_$test_num: Running $test_name ---"
# Execute the test command.
eval "$test_command" > >(tee -a "$log_file") 2> >(tee -a "$log_file" >&2)
actual_exit_code=$?
echo "TEST_${test_num}_COMMAND_EXIT_CODE: $actual_exit_code" # This goes to main log
echo "TEST_${test_num}_COMMAND_EXIT_CODE: $actual_exit_code" >> "$log_file" # Also to per-test log
if [ "$actual_exit_code" -ne 0 ]; then
echo "TEST_$test_num ($test_name) FAILED with exit code $actual_exit_code." >&2
echo "--- Log for failed TEST_$test_num ($test_name) ---" >&2
if [ -f "$log_file" ]; then
cat "$log_file" >&2
else
echo "Log file $log_file not found for TEST_$test_num ($test_name)." >&2
fi
echo "--- End of log for TEST_$test_num ($test_name) ---" >&2
return "$actual_exit_code" # Return the failure code
else
echo "TEST_$test_num ($test_name) PASSED."
return 0 # Return success
fi
}
# Helper function to call run_test and update the overall script exit code
run_and_track_test() {
local test_num_arg="$1"
local test_name_arg="$2"
local test_command_arg="$3"
# Run the test
run_test "$test_num_arg" "$test_name_arg" "$test_command_arg"
local test_specific_exit_code=$?
# If the test failed, set the overall script exit code to 1
if [ "$test_specific_exit_code" -ne 0 ]; then
# No need for extra echo here, run_test already logged the failure.
overall_script_exit_code=1
fi
}
# --- Actual Test Execution ---
run_and_track_test 1 "test_struct_output_generate.py" \
"python3 -m pytest -s -v /workspace/vllm/tests/entrypoints/llm/test_struct_output_generate.py -k \"not test_structured_output_with_reasoning_matrices\""
run_and_track_test 2 "test_moe_pallas.py" \
"python3 -m pytest -s -v /workspace/vllm/tests/tpu/test_moe_pallas.py"
run_and_track_test 3 "test_lora.py" \
"VLLM_XLA_CHECK_RECOMPILATION=0 python3 -m pytest -s -v /workspace/vllm/tests/tpu/lora/test_lora.py"
run_and_track_test 4 "test_tpu_qkv_linear.py" \
"python3 -m pytest -s -v /workspace/vllm/tests/v1/tpu/test_tpu_qkv_linear.py"
run_and_track_test 5 "test_spmd_model_weight_loading.py" \
"python3 -m pytest -s -v /workspace/vllm/tests/v1/tpu/test_spmd_model_weight_loading.py"
run_and_track_test 6 "test_kv_cache_update_kernel.py" \
"python3 -m pytest -s -v /workspace/vllm/tests/v1/tpu/test_kv_cache_update_kernel.py"
run_and_track_test 7 "test_tpu_int8.py" \
"python3 -m pytest -s -v /workspace/vllm/tests/v1/tpu/test_tpu_int8.py"
# After all tests have been attempted, exit with the overall status.
if [ "$overall_script_exit_code" -ne 0 ]; then
echo "--- One or more tests FAILED. Overall script exiting with failure code 1. ---"
else
echo "--- All tests have completed and PASSED. Overall script exiting with success code 0. ---"
fi
exit "$overall_script_exit_code"
' # IMPORTANT: This is the closing single quote for the bash -c "..." command. Ensure it is present and correct.
# Capture the exit code of the docker run command
DOCKER_RUN_EXIT_CODE=$?
# The trap will run for cleanup.
# Exit the main script with the Docker run command's exit code.
if [ "$DOCKER_RUN_EXIT_CODE" -ne 0 ]; then
echo "Docker run command failed with exit code $DOCKER_RUN_EXIT_CODE."
exit "$DOCKER_RUN_EXIT_CODE"
else
echo "Docker run command completed successfully."
exit 0
fi
# TODO: This test fails because it uses RANDOM_SEED sampling
# pytest -v -s /workspace/vllm/tests/tpu/test_custom_dispatcher.py \
+172
View File
@@ -0,0 +1,172 @@
#!/bin/bash
set -xu
remove_docker_container() {
docker rm -f tpu-test || true;
}
trap remove_docker_container EXIT
# Remove the container that might not be cleaned up in the previous run.
remove_docker_container
# Build the docker image.
docker build -f docker/Dockerfile.tpu -t vllm-tpu .
# Set up cleanup.
cleanup_docker() {
# Get Docker's root directory
docker_root=$(docker info -f '{{.DockerRootDir}}')
if [ -z "$docker_root" ]; then
echo "Failed to determine Docker root directory."
exit 1
fi
echo "Docker root directory: $docker_root"
# Check disk usage of the filesystem where Docker's root directory is located
disk_usage=$(df "$docker_root" | tail -1 | awk '{print $5}' | sed 's/%//')
# Define the threshold
threshold=70
if [ "$disk_usage" -gt "$threshold" ]; then
echo "Disk usage is above $threshold%. Cleaning up Docker images and volumes..."
# Remove dangling images (those that are not tagged and not used by any container)
docker image prune -f
# Remove unused volumes / force the system prune for old images as well.
docker volume prune -f && docker system prune --force --filter "until=72h" --all
echo "Docker images and volumes cleanup completed."
else
echo "Disk usage is below $threshold%. No cleanup needed."
fi
}
cleanup_docker
# For HF_TOKEN.
source /etc/environment
docker run --privileged --net host --shm-size=16G -it \
-e "HF_TOKEN=$HF_TOKEN" --name tpu-test \
vllm-tpu /bin/bash -c '
set -e # Exit immediately if a command exits with a non-zero status.
set -u # Treat unset variables as an error.
echo "--- Starting script inside Docker container ---"
# Create results directory
RESULTS_DIR=$(mktemp -d)
# If mktemp fails, set -e will cause the script to exit.
echo "Results will be stored in: $RESULTS_DIR"
# Install dependencies
echo "--- Installing Python dependencies ---"
python3 -m pip install --progress-bar off git+https://github.com/thuml/depyf.git \
&& python3 -m pip install --progress-bar off pytest pytest-asyncio tpu-info \
&& python3 -m pip install --progress-bar off "lm-eval[api]>=0.4.12" \
&& python3 -m pip install --progress-bar off hf-transfer tblib==3.1.0
echo "--- Python dependencies installed ---"
export VLLM_XLA_CHECK_RECOMPILATION=1
export VLLM_XLA_CACHE_PATH=
echo "--- Hardware Information ---"
# tpu-info
echo "--- Starting Tests ---"
set +e
overall_script_exit_code=0
# --- Test Definitions ---
# If a test fails, this function will print logs and will not cause the main script to exit.
run_test() {
local test_num=$1
local test_name=$2
local test_command=$3
local log_file="$RESULTS_DIR/test_${test_num}.log"
local actual_exit_code
echo "--- TEST_$test_num: Running $test_name ---"
# Execute the test command.
eval "$test_command" > >(tee -a "$log_file") 2> >(tee -a "$log_file" >&2)
actual_exit_code=$?
echo "TEST_${test_num}_COMMAND_EXIT_CODE: $actual_exit_code" # This goes to main log
echo "TEST_${test_num}_COMMAND_EXIT_CODE: $actual_exit_code" >> "$log_file" # Also to per-test log
if [ "$actual_exit_code" -ne 0 ]; then
echo "TEST_$test_num ($test_name) FAILED with exit code $actual_exit_code." >&2
echo "--- Log for failed TEST_$test_num ($test_name) ---" >&2
if [ -f "$log_file" ]; then
cat "$log_file" >&2
else
echo "Log file $log_file not found for TEST_$test_num ($test_name)." >&2
fi
echo "--- End of log for TEST_$test_num ($test_name) ---" >&2
return "$actual_exit_code" # Return the failure code
else
echo "TEST_$test_num ($test_name) PASSED."
return 0 # Return success
fi
}
# Helper function to call run_test and update the overall script exit code
run_and_track_test() {
local test_num_arg="$1"
local test_name_arg="$2"
local test_command_arg="$3"
# Run the test
run_test "$test_num_arg" "$test_name_arg" "$test_command_arg"
local test_specific_exit_code=$?
# If the test failed, set the overall script exit code to 1
if [ "$test_specific_exit_code" -ne 0 ]; then
# No need for extra echo here, run_test already logged the failure.
overall_script_exit_code=1
fi
}
# --- Actual Test Execution ---
run_and_track_test 0 "test_perf.py" \
"python3 -m pytest -s -v /workspace/vllm/tests/v1/tpu/test_perf.py"
run_and_track_test 1 "test_compilation.py" \
"python3 -m pytest -s -v /workspace/vllm/tests/tpu/test_compilation.py"
run_and_track_test 2 "test_basic.py" \
"python3 -m pytest -s -v /workspace/vllm/tests/v1/tpu/test_basic.py"
run_and_track_test 3 "test_accuracy.py::test_lm_eval_accuracy_v1_engine" \
"python3 -m pytest -s -v /workspace/vllm/tests/entrypoints/llm/test_accuracy.py::test_lm_eval_accuracy_v1_engine"
run_and_track_test 4 "test_quantization_accuracy.py" \
"python3 -m pytest -s -v /workspace/vllm/tests/tpu/test_quantization_accuracy.py"
run_and_track_test 6 "test_tpu_model_runner.py" \
"python3 -m pytest -s -v /workspace/vllm/tests/v1/tpu/worker/test_tpu_model_runner.py"
run_and_track_test 7 "test_sampler.py" \
"python3 -m pytest -s -v /workspace/vllm/tests/v1/tpu/test_sampler.py"
run_and_track_test 8 "test_topk_topp_sampler.py" \
"python3 -m pytest -s -v /workspace/vllm/tests/v1/tpu/test_topk_topp_sampler.py"
run_and_track_test 9 "test_multimodal.py" \
"python3 -m pytest -s -v /workspace/vllm/tests/v1/tpu/test_multimodal.py"
run_and_track_test 10 "test_pallas.py" \
"python3 -m pytest -s -v /workspace/vllm/tests/v1/tpu/test_pallas.py"
# After all tests have been attempted, exit with the overall status.
if [ "$overall_script_exit_code" -ne 0 ]; then
echo "--- One or more tests FAILED. Overall script exiting with failure code 1. ---"
else
echo "--- All tests have completed and PASSED. Overall script exiting with success code 0. ---"
fi
exit "$overall_script_exit_code"
' # IMPORTANT: This is the closing single quote for the bash -c "..." command. Ensure it is present and correct.
# Capture the exit code of the docker run command
DOCKER_RUN_EXIT_CODE=$?
# The trap will run for cleanup.
# Exit the main script with the Docker run command's exit code.
if [ "$DOCKER_RUN_EXIT_CODE" -ne 0 ]; then
echo "Docker run command failed with exit code $DOCKER_RUN_EXIT_CODE."
exit "$DOCKER_RUN_EXIT_CODE"
else
echo "Docker run command completed successfully."
exit 0
fi
# TODO: This test fails because it uses RANDOM_SEED sampling
# pytest -v -s /workspace/vllm/tests/tpu/test_custom_dispatcher.py \
+1 -1
View File
@@ -10,7 +10,7 @@ fi
# TODO: Use the PyPI wheel once it is published.
uv pip install --system \
"kvcr @ git+https://github.com/ai-dynamo/kvcr.git@v0.1.1"
"kvcr @ git+https://github.com/ai-dynamo/kvcr.git@v0.1.0"
# Keep only the NIXL wheel matching the CI image's CUDA runtime.
NIXL_VERSION=$(uv pip show --system nixl | sed -n 's/^Version: //p')
+16 -28
View File
@@ -11,9 +11,9 @@ set -euo pipefail
TARGET="${1:-all}"
case "${TARGET}" in
cuda-13-0 | cuda-12-9 | cuda-13-0-ubuntu-24-04 | \
cuda-12-9-ubuntu-24-04 | rocm | rocm72 | xpu | cpu | all) ;;
cuda-12-9-ubuntu-24-04 | rocm | xpu | cpu | all) ;;
*)
echo "Usage: $0 {cuda-13-0|cuda-12-9|cuda-13-0-ubuntu-24-04|cuda-12-9-ubuntu-24-04|rocm|rocm72|xpu|cpu|all}"
echo "Usage: $0 {cuda-13-0|cuda-12-9|cuda-13-0-ubuntu-24-04|cuda-12-9-ubuntu-24-04|rocm|xpu|cpu|all}"
exit 2
;;
esac
@@ -143,35 +143,23 @@ if target_enabled cuda-12-9-ubuntu-24-04; then
fi
# ---- ROCm ----
# Each ROCm stack is identified by its base Dockerfile (see rocm/stack.sh). The
# default stack also owns the plain :latest / :v<ver> tags.
publish_rocm_stack() {
local base_dockerfile="$1" default="$2" suffix
(
# shellcheck source=.buildkite/scripts/rocm/stack.sh
source .buildkite/scripts/rocm/stack.sh "$base_dockerfile"
local suffixes=("-${ROCM_STACK_VARIANT}")
[[ "$default" == "1" ]] && suffixes=("" "-${ROCM_STACK_VARIANT}")
docker pull "$ROCM_STACK_ECR_IMAGE"
docker pull "$ROCM_STACK_ECR_BASE"
for suffix in "${suffixes[@]}"; do
for tag in "latest${suffix}" "v${RELEASE_VERSION}${suffix}"; do
docker tag "$ROCM_STACK_ECR_IMAGE" "vllm/vllm-openai-rocm:${tag}"
docker tag "$ROCM_STACK_ECR_BASE" "vllm/vllm-openai-rocm:${tag}-base"
docker push "vllm/vllm-openai-rocm:${tag}"
docker push "vllm/vllm-openai-rocm:${tag}-base"
done
done
)
}
if target_enabled rocm; then
publish_rocm_stack docker/Dockerfile.rocm_base 1
fi
ROCM_BASE_CACHE_KEY=$(.buildkite/scripts/cache-rocm-base-wheels.sh key)
echo "ROCm base cache key: ${ROCM_BASE_CACHE_KEY}"
if target_enabled rocm72; then
publish_rocm_stack docker/Dockerfile.rocm_72_base 0
docker pull "public.ecr.aws/q9t5s3a7/vllm-release-repo:${COMMIT}-rocm"
docker pull "public.ecr.aws/q9t5s3a7/vllm-release-repo:${ROCM_BASE_CACHE_KEY}-rocm-base"
docker tag "public.ecr.aws/q9t5s3a7/vllm-release-repo:${COMMIT}-rocm" vllm/vllm-openai-rocm:latest
docker tag "public.ecr.aws/q9t5s3a7/vllm-release-repo:${COMMIT}-rocm" "vllm/vllm-openai-rocm:v${RELEASE_VERSION}"
docker push vllm/vllm-openai-rocm:latest
docker push "vllm/vllm-openai-rocm:v${RELEASE_VERSION}"
docker tag "public.ecr.aws/q9t5s3a7/vllm-release-repo:${ROCM_BASE_CACHE_KEY}-rocm-base" vllm/vllm-openai-rocm:latest-base
docker tag "public.ecr.aws/q9t5s3a7/vllm-release-repo:${ROCM_BASE_CACHE_KEY}-rocm-base" "vllm/vllm-openai-rocm:v${RELEASE_VERSION}-base"
docker push vllm/vllm-openai-rocm:latest-base
docker push "vllm/vllm-openai-rocm:v${RELEASE_VERSION}-base"
fi
# ---- XPU ----
+65 -42
View File
@@ -2,36 +2,42 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
#
# Push a ROCm stack's nightly base image and nightly image from ECR to Docker Hub
# under vllm/vllm-openai-rocm, as base-nightly-<variant>, nightly-<variant> and their
# -<commit> forms, then prune old nightly tags. With --default, also push the plain
# base-nightly and nightly tags for the same images.
# Run when NIGHTLY=1 after the stack's release image has been pushed to ECR.
# Push a ROCm-family nightly base image and nightly image from ECR to Docker Hub
# under vllm/vllm-openai-rocm, as base-nightly[-<variant>], nightly[-<variant>]
# and their -<commit> forms.
# Run when NIGHTLY=1 after the matching build-*-release-image step has pushed to ECR.
#
# Usage: push-nightly-builds-rocm.sh <base-dockerfile> [--default]
# e.g. "docker/Dockerfile.rocm_base --default" (ROCm 10.0, also :nightly)
# and "docker/Dockerfile.rocm_72_base" (ROCm 7.2, :nightly-rocm72).
# Usage: push-nightly-builds-rocm.sh [ECR_TAG_SUFFIX TAG_VARIANT]
# Pass both arguments or neither. With none, the default ROCm apt build is
# published as :nightly. The TheRock build passes "rock" (the source ECR tag
# suffix) and "rocm714" (the Docker Hub tag flavor), giving :nightly-rocm714.
#
# Local testing (no push to Docker Hub):
# BUILDKITE_COMMIT=<commit-with-image-in-ecr> DRY_RUN=1 \
# bash .buildkite/scripts/push-nightly-builds-rocm.sh docker/Dockerfile.rocm_base --default
# BASE_ECR_IMAGE=<full-base-image-reference> \
# BUILDKITE_COMMIT=<commit-with-image-in-ecr> DRY_RUN=1 \
# bash .buildkite/scripts/push-nightly-builds-rocm.sh
# Requires: AWS CLI configured (for ECR public login), Docker. For full run: Docker Hub login.
set -euxo pipefail
usage() {
echo "Usage: $0 <base-dockerfile> [--default]" >&2
echo "Usage: $0 [ECR_TAG_SUFFIX TAG_VARIANT]" >&2
}
PUSH_DEFAULT=0
DOCKERHUB_REPO="vllm/vllm-openai-rocm"
case "$#" in
1) ;;
0)
ECR_TAG_SUFFIX="rocm"
TAG_VARIANT=""
;;
2)
if [[ "$2" != "--default" ]]; then
if [[ -z "$1" || -z "$2" ]]; then
usage
exit 2
fi
PUSH_DEFAULT=1
ECR_TAG_SUFFIX="$1"
TAG_VARIANT="$2"
;;
*)
usage
@@ -43,44 +49,61 @@ esac
BUILDKITE_COMMIT="${BUILDKITE_COMMIT:?Set BUILDKITE_COMMIT to the commit SHA that has the image in ECR (e.g. from a previous release pipeline run)}"
DRY_RUN="${DRY_RUN:-0}"
# shellcheck source=.buildkite/scripts/rocm/stack.sh
source "$(dirname "${BASH_SOURCE[0]}")/rocm/stack.sh" "$1"
DOCKERHUB_REPO="vllm/vllm-openai-rocm"
VARIANT="$ROCM_STACK_VARIANT"
ECR_REPO="public.ecr.aws/q9t5s3a7/vllm-release-repo"
BASE_TAGS=("base-nightly-${VARIANT}" "base-nightly-${VARIANT}-${BUILDKITE_COMMIT}")
TAGS=("nightly-${VARIANT}" "nightly-${VARIANT}-${BUILDKITE_COMMIT}")
if [[ "$PUSH_DEFAULT" == "1" ]]; then
BASE_TAGS+=("base-nightly" "base-nightly-${BUILDKITE_COMMIT}")
TAGS+=("nightly" "nightly-${BUILDKITE_COMMIT}")
# Get the base image ECR tag (set by the build-*-release-image pipeline step)
BASE_METADATA_KEY="${ECR_TAG_SUFFIX}-base-ecr-tag"
if [[ -n "${BASE_ECR_IMAGE:-}" ]]; then
if [[ "$DRY_RUN" != "1" ]]; then
echo "ERROR: BASE_ECR_IMAGE may only be used with DRY_RUN=1" >&2
exit 1
fi
BASE_ORIG_TAG="$BASE_ECR_IMAGE"
elif ! BASE_ORIG_TAG="$(buildkite-agent meta-data get "$BASE_METADATA_KEY")"; then
echo "ERROR: Failed to read required Buildkite metadata '$BASE_METADATA_KEY'" >&2
exit 1
fi
if [[ -z "$BASE_ORIG_TAG" ]]; then
echo "ERROR: Required Buildkite metadata '$BASE_METADATA_KEY' is empty" >&2
echo "Set BASE_ECR_IMAGE to the full ECR base image reference for local testing" >&2
exit 1
fi
echo "Pushing base image $ROCM_STACK_ECR_BASE as: ${BASE_TAGS[*]}"
echo "Pushing release image $ROCM_STACK_ECR_IMAGE as: ${TAGS[*]}"
ORIG_TAG="${BUILDKITE_COMMIT}-${ECR_TAG_SUFFIX}"
VARIANT_SUFFIX=""
[[ -n "$TAG_VARIANT" ]] && VARIANT_SUFFIX="-$TAG_VARIANT"
BASE_TAG_NAME="base-nightly${VARIANT_SUFFIX}"
TAG_NAME="nightly${VARIANT_SUFFIX}"
BASE_TAG_NAME_COMMIT="base-nightly${VARIANT_SUFFIX}-${BUILDKITE_COMMIT}"
TAG_NAME_COMMIT="nightly${VARIANT_SUFFIX}-${BUILDKITE_COMMIT}"
# Login to ECR and pull the stack's images
echo "Pushing base image from ECR: $BASE_ORIG_TAG"
echo "Pushing release image from ECR tag: $ORIG_TAG to Docker Hub as $DOCKERHUB_REPO:$TAG_NAME and $DOCKERHUB_REPO:$TAG_NAME_COMMIT"
[[ "$DRY_RUN" == "1" ]] && echo "[DRY_RUN] Skipping push to Docker Hub"
# Login to ECR and pull the images built by build-*-release-image
aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7
docker pull "$ROCM_STACK_ECR_BASE"
docker pull "$ROCM_STACK_ECR_IMAGE"
docker pull "$BASE_ORIG_TAG"
docker pull "${ECR_REPO}:${ORIG_TAG}"
for tag in "${BASE_TAGS[@]}"; do docker tag "$ROCM_STACK_ECR_BASE" "$DOCKERHUB_REPO:$tag"; done
for tag in "${TAGS[@]}"; do docker tag "$ROCM_STACK_ECR_IMAGE" "$DOCKERHUB_REPO:$tag"; done
# Tag for Docker Hub (base-nightly and base-nightly-<commit>, nightly and nightly-<commit>)
docker tag "$BASE_ORIG_TAG" "$DOCKERHUB_REPO":"$BASE_TAG_NAME"
docker tag "$BASE_ORIG_TAG" "$DOCKERHUB_REPO":"$BASE_TAG_NAME_COMMIT"
docker tag "${ECR_REPO}:${ORIG_TAG}" "$DOCKERHUB_REPO":"$TAG_NAME"
docker tag "${ECR_REPO}:${ORIG_TAG}" "$DOCKERHUB_REPO":"$TAG_NAME_COMMIT"
if [[ "$DRY_RUN" == "1" ]]; then
echo "[DRY_RUN] Would push $DOCKERHUB_REPO:$BASE_TAG_NAME and $DOCKERHUB_REPO:$BASE_TAG_NAME_COMMIT"
echo "[DRY_RUN] Would push $DOCKERHUB_REPO:$TAG_NAME and $DOCKERHUB_REPO:$TAG_NAME_COMMIT"
echo "[DRY_RUN] Local tags created. Exiting without push."
exit 0
fi
# Push to Docker Hub (docker-login plugin runs before this step in CI)
for tag in "${BASE_TAGS[@]}" "${TAGS[@]}"; do docker push "$DOCKERHUB_REPO:$tag"; done
echo "Pushed $DOCKERHUB_REPO: ${BASE_TAGS[*]} ${TAGS[*]}"
docker push "$DOCKERHUB_REPO":"$BASE_TAG_NAME"
docker push "$DOCKERHUB_REPO":"$BASE_TAG_NAME_COMMIT"
docker push "$DOCKERHUB_REPO":"$TAG_NAME"
docker push "$DOCKERHUB_REPO":"$TAG_NAME_COMMIT"
# Keep only the last 14 builds of each tag family. The plain nightly- families
# exclude the variant ones sharing their prefix; commit SHAs never start with "rocm".
CLEANUP=.buildkite/scripts/cleanup-nightly-builds.sh
bash "$CLEANUP" "nightly-${VARIANT}-" "$DOCKERHUB_REPO"
bash "$CLEANUP" "base-nightly-${VARIANT}-" "$DOCKERHUB_REPO"
if [[ "$PUSH_DEFAULT" == "1" ]]; then
bash "$CLEANUP" "nightly-" "$DOCKERHUB_REPO" "nightly-rocm"
bash "$CLEANUP" "base-nightly-" "$DOCKERHUB_REPO" "base-nightly-rocm"
fi
echo "Pushed $DOCKERHUB_REPO:$BASE_TAG_NAME and $DOCKERHUB_REPO:$BASE_TAG_NAME_COMMIT"
echo "Pushed $DOCKERHUB_REPO:$TAG_NAME and $DOCKERHUB_REPO:$TAG_NAME_COMMIT"
@@ -1,112 +0,0 @@
#!/bin/bash
# Install the AITER nightly wheel over this build's ci_base.
#
# Runs in CI when AITER_NIGHTLY=1, once the stock ci_base it builds on is
# published. In order:
# 1. Select the newest nightly wheel that fits the stock ci_base
# (select_aiter_nightly_wheel.py, beside this script, run inside that image)
# 2. Build a new image: pip install that wheel over the stock ci_base
# 3. Push the new image as this build's ci_base-build tag, replacing the stock
# one, so later steps and native jobs use it; record its digest
#
# Buildkite meta-data, a key-value store shared by every job in this build:
# reads rocm-ci-base-image the stock ci_base
# r/w aiter-nightly-wheel-url the wheel step 1 chose; a retry reads it
# back and installs that same wheel
# writes aiter-nightly-ci-base the new image as tag@digest; the vLLM
# image build uses it as its ci_base
#
# Exit status:
# 10 no wheel fits this ci_base
# 11 the wheel does not install
# Anything else is unexpected and passed through as is.
# Buildkite retries this step once on exit status 1 or 2, which it treats as
# transient Docker/registry failures (amd_infra_retry in the template).
# 10 and 11 sit outside that range: they are real AITER failures, and a retry
# would only fail the same way.
set -euo pipefail
SELECT_WHEEL="$(dirname "${BASH_SOURCE[0]}")/select_aiter_nightly_wheel.py"
BUILD_TAG="rocm/vllm-dev:ci_base-build-${BUILDKITE_BUILD_ID:?BUILDKITE_BUILD_ID is required}"
# Scratch space for the build context and captured errors. The trap deletes it
# however the script exits: success, fail, or set -e.
scratch="$(mktemp -d)"
trap 'rm -rf "${scratch}"' EXIT
errors="${scratch}/errors"
fail() {
local status="$1" title="$2" detail="$3"
echo "+++ :x: AITER nightly: ${title}"
echo "${detail}" >&2
# shellcheck disable=SC2016 # the backticks are a literal markdown fence
printf ':x: **AITER nightly: %s**\n\n```\n%s\n```\n' "${title}" "${detail}" \
| buildkite-agent annotate --style error --context aiter-nightly
exit "${status}"
}
# Set by ensure-ci-base-amd; this step's depends_on guarantees it is there.
ci_base="$(buildkite-agent meta-data get rocm-ci-base-image)"
# Set below on the first attempt. Meta-data is per build, so a retry reuses
# that wheel, not a newer nightly.
pinned="$(buildkite-agent meta-data get aiter-nightly-wheel-url --default "")"
echo "--- :docker: Pulling ${ci_base}"
docker pull "${ci_base}"
echo "--- :mag: Selecting the AITER nightly for ${ci_base}"
set +e
out="$(docker run --rm -i --entrypoint python3 "${ci_base}" \
-P - ${pinned:+--wheel-url "${pinned}"} \
< "${SELECT_WHEEL}" 2> "${errors}")"
status=$?
set -e
cat "${errors}" >&2
case "${status}" in
0) ;;
10) fail 10 "no wheel for this image" "${out}" ;;
*) fail "${status}" "could not select a wheel" "$(tail -n 20 "${errors}")" ;;
esac
# The first line is the wheel's URL; the rest, the annotation naming it.
wheel_url="$(head -n 1 <<< "${out}")"
note="$(tail -n +2 <<< "${out}")"
buildkite-agent meta-data set aiter-nightly-wheel-url "${wheel_url}"
cat > "${scratch}/Dockerfile" <<'EOF'
ARG CI_BASE_IMAGE
FROM ${CI_BASE_IMAGE}
ARG AITER_WHEEL_URL
RUN python3 -m pip install --no-cache-dir "$AITER_WHEEL_URL"
EOF
echo "--- :docker: Installing ${wheel_url##*/}"
if ! docker buildx build \
--builder default \
--provenance=false \
--progress plain \
--build-arg "CI_BASE_IMAGE=${ci_base}" \
--build-arg "AITER_WHEEL_URL=${wheel_url}" \
-t "${BUILD_TAG}" \
"${scratch}"; then
fail 11 "the wheel does not install" \
"Installing ${wheel_url##*/} over ${ci_base} failed; see this step's log."
fi
docker push "${BUILD_TAG}"
# Handed off as tag@digest: build-test-image.sh rejects a ci_base that is not
# digest-pinned.
digest="$(docker buildx imagetools inspect "${BUILD_TAG}" | awk '$1 == "Digest:" { print $2; exit }')"
if [[ ! "${digest}" =~ ^sha256:[0-9a-f]{64}$ ]]; then
echo "Could not resolve the pushed digest of ${BUILD_TAG}" >&2
exit 1
fi
# The vLLM image build uses this as its ci_base.
buildkite-agent meta-data set aiter-nightly-ci-base "${BUILD_TAG}@${digest}"
# The annotation starts with :warning: when the wheel is stale.
style=info
if [[ "${note}" == :warning:* ]]; then
style=warning
fi
echo "${note}" | buildkite-agent annotate --style "${style}" --context aiter-nightly
echo "--- :white_check_mark: ${BUILD_TAG}@${digest}"
+2 -2
View File
@@ -1,8 +1,8 @@
#!/usr/bin/env bash
# Shared Dockerfile selection for the normal AMD CI build chain.
# CI_ROCM_DOCKERFILE_BASE and CI_ROCM_DOCKERFILE accept repository-relative paths.
# For the legacy ROCm 7.2 stack, set CI_ROCM_DOCKERFILE_BASE=docker/Dockerfile.rocm_72_base
# and CI_ROCM_DOCKERFILE=docker/Dockerfile.rocm_72.
# For Rock, set CI_ROCM_DOCKERFILE_BASE=docker/Dockerfile.rock_base and
# CI_ROCM_DOCKERFILE=docker/Dockerfile.rock.
# Custom recipes must preserve the CI targets and artifact layout. Additional
# COPY inputs also need the corresponding *_CONTENT_FILES overrides.
@@ -1,260 +0,0 @@
#!/usr/bin/env python3
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""Select the AITER nightly wheel for a vLLM ROCm image.
aiter-nightly-overlay.sh runs this inside the stock ci_base, so it reads that
image's ROCm, torch and Python. It prints the wheel's URL, then the Buildkite
annotation naming it.
"""
import argparse
import datetime
import importlib.metadata
import os
import sys
import urllib.request
from dataclasses import dataclass
from html.parser import HTMLParser
from pathlib import Path
from urllib.parse import unquote, urldefrag, urljoin
import regex as re
INDEX_URL = "https://rocm.frameworks-nightlies.amd.com/whl-multi-arch/amd-aiter/"
AITER_COMMIT_URL = "https://github.com/ROCm/aiter/commit/"
STALE_AFTER = datetime.timedelta(days=1)
# Example:
# amd_aiter-0.1.25+rocm7.2.3.torch2.12.ae984f5.d20261008-cp312-cp312-linux_x86_64.whl
# is AITER main at commit ae984f5, built 2026-10-08, for ROCm 7.2.3, torch 2.12
# and Python 3.12. 0.1.25 is a guess at the next release, not a real one.
WHEEL_NAME = re.compile(
r"^amd_aiter-(?P<base_version>[^+-]+)\+(?P<local>[^-]+)"
r"-(?P<python>cp\d+)-(?P=python)-linux_x86_64\.whl$"
)
ROCM_FIELD = re.compile(r"(?:^|\.)rocm(\d+(?:\.\d+)*)(?:a\d+)?(?=\.|$)")
TORCH_FIELD = re.compile(r"(?:^|\.)torch(\d+\.\d+)(?=\.|$)")
COMMIT_AND_DATE_FIELD = re.compile(r"(?:^|\.)([0-9a-f]{7,40})\.d(\d{8})(?=\.|$)")
@dataclass(frozen=True)
class Wheel:
"""An AITER nightly wheel, read from its filename (see WHEEL_NAME)."""
filename: str
url: str
version: str
base_version: str
rocm: str
torch: str
commit: str
build_date: datetime.date
python: str
class _LinkParser(HTMLParser):
"""Collects every link on the index page, which lists one per wheel."""
def __init__(self):
super().__init__()
self.hrefs: list[str] = []
def handle_starttag(self, tag, attrs):
if tag == "a":
self.hrefs.extend(
value for name, value in attrs if name == "href" and value
)
def wheel_from_url(url: str) -> Wheel | None:
"""The AITER nightly wheel an absolute URL points at, if it is one."""
filename = unquote(urldefrag(url).url.rsplit("/", 1)[-1])
match = WHEEL_NAME.match(filename)
if not match:
return None
local = match["local"]
rocm, torch, build = (
field.search(local)
for field in (ROCM_FIELD, TORCH_FIELD, COMMIT_AND_DATE_FIELD)
)
if not (rocm and torch and build):
return None
return Wheel(
filename=filename,
url=url,
version=f"{match['base_version']}+{local}",
base_version=match["base_version"],
rocm=rocm[1],
torch=torch[1],
commit=build[1],
build_date=datetime.datetime.strptime(build[2], "%Y%m%d").date(),
python=match["python"],
)
def parse_index(html: str, index_url: str) -> list[Wheel]:
"""Every AITER nightly wheel linked from the index page, a plain HTML list
of links, one per file (the format pip reads)."""
parser = _LinkParser()
parser.feed(html)
wheels = (wheel_from_url(urljoin(index_url, href)) for href in parser.hrefs)
return [wheel for wheel in wheels if wheel]
def rocm_version() -> str:
"""This image's ROCm version; ROCm 10+ ships as the `rocm` pip SDK."""
try:
return importlib.metadata.version("rocm")
except importlib.metadata.PackageNotFoundError:
# No pip SDK: fall back to older ROCm releases, which install to ROCM_PATH.
rocm_path = Path(os.environ.get("ROCM_PATH", "/opt/rocm"))
return (rocm_path / ".info" / "version").read_text()
def rocm_release(version: str) -> str:
"""`7.2.3-49` or `10.0.0a20261001` -> `7.2.3` or `10.0.0`, as wheel names
spell it."""
match = re.match(r"\d+(?:\.\d+)*", version.strip())
if not match:
raise ValueError(f"Unrecognised ROCm version: {version!r}")
return match.group(0)
def torch_release(version: str) -> str:
"""`2.12.0a0+git6bbd260` -> `2.12`, as wheel names spell it."""
match = re.match(r"\d+\.\d+", version)
if not match:
raise ValueError(f"Unrecognised torch version: {version!r}")
return match.group(0)
def python_tag(version_info: tuple[int, int]) -> str:
"""(3, 12) -> `cp312`, as wheel names spell it."""
return f"cp{version_info[0]}{version_info[1]}"
def _base_version_key(wheel: Wheel) -> tuple[int, ...]:
"""`0.1.25` -> (0, 1, 25), so versions sort as numbers: 0.1.10 > 0.1.9."""
return tuple(int(part) for part in re.findall(r"\d+", wheel.base_version))
def _label(rocm: str, torch: str, python: str) -> str:
"""`rocm7.2.3/torch2.12/cp312`, for messages."""
return f"rocm{rocm}/torch{torch}/{python}"
def select_wheel(wheels: list[Wheel], rocm: str, torch: str, python: str) -> Wheel:
"""Newest build for this ROCm, torch and Python, by date: version labels lag.
Two builds with the same date and version: whichever the index lists first.
"""
target = (rocm, torch, python)
matching = [w for w in wheels if (w.rocm, w.torch, w.python) == target]
if not matching:
available = sorted({_label(w.rocm, w.torch, w.python) for w in wheels})
raise LookupError(
f"No AITER nightly for {_label(*target)}. "
f"Available: {', '.join(available) or 'none'}"
)
return max(matching, key=lambda w: (w.build_date, _base_version_key(w)))
def fetch(url: str) -> str:
"""GET url as text. A failure exits 1, so Buildkite retries the step."""
with urllib.request.urlopen(url, timeout=60) as response:
return response.read().decode()
def annotation(wheel: Wheel, today: datetime.date) -> str:
"""The Buildkite annotation naming the wheel this build tests.
Starts with :warning: if the wheel is stale; the overlay script styles it so.
"""
age = today - wheel.build_date
stale = (
f":warning: **AITER nightly is {age.days} days old**: "
f"no newer build has been published for "
f"{_label(wheel.rocm, wheel.torch, wheel.python)}.\n\n"
if age > STALE_AFTER
else ""
)
return stale + (
f":crescent_moon: **AITER nightly**: main at "
f"[`{wheel.commit}`]({AITER_COMMIT_URL}{wheel.commit}), "
f"built {wheel.build_date.isoformat()} for torch {wheel.torch} "
f"(`{wheel.version}`)"
)
class NoWheel(Exception):
"""No wheel fits this image's ROCm, torch and Python. Exit 10."""
def _image_wheel(wheel_url: str) -> Wheel:
"""The wheel at wheel_url, if it fits this image's ROCm, torch and Python."""
wheel = wheel_from_url(wheel_url)
if not wheel:
raise NoWheel(f"Not an AITER nightly wheel: {wheel_url}")
return _for_image([wheel])
def _image() -> tuple[str, str, str]:
"""This image's ROCm, torch and Python, as wheel names spell them."""
try:
rocm = rocm_release(rocm_version())
except (OSError, ValueError) as error:
raise NoWheel(f"Cannot read this image's ROCm version: {error}") from error
try:
torch = torch_release(importlib.metadata.version("torch"))
except (importlib.metadata.PackageNotFoundError, ValueError) as error:
raise NoWheel(f"Cannot read this image's torch version: {error}") from error
return rocm, torch, python_tag(sys.version_info[:2])
def _for_image(wheels: list[Wheel]) -> Wheel:
"""The newest of wheels for the image this runs in."""
try:
return select_wheel(wheels, *_image())
except LookupError as error:
raise NoWheel(str(error)) from error
def select(index_url: str, wheel_url: str = "") -> Wheel:
"""The wheel to install: wheel_url on a retry, else the newest.
However old the newest is, it is tested; the annotation shows its build date.
"""
if wheel_url:
return _image_wheel(wheel_url)
return _for_image(parse_index(fetch(index_url), index_url))
def _today() -> datetime.date:
"""Today in UTC; a function so tests can set the date."""
return datetime.datetime.now(datetime.UTC).date()
# Not 1 or 2, which Buildkite retries; see aiter-nightly-overlay.sh.
EXIT_NO_WHEEL = 10
def main(argv=None) -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--index-url", default=INDEX_URL)
parser.add_argument(
"--wheel-url", default="", help="a retry's earlier choice; skips the index"
)
args = parser.parse_args(argv)
try:
wheel = select(args.index_url, args.wheel_url)
except NoWheel as error:
print(error)
return EXIT_NO_WHEEL
print(wheel.url)
print(annotation(wheel, _today()))
return 0
if __name__ == "__main__":
sys.exit(main())
-78
View File
@@ -1,78 +0,0 @@
#!/usr/bin/env bash
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
#
# Derive everything the release pipeline needs about a ROCm stack from its base
# Dockerfile, so the stack's image is the single input to every ROCm script.
#
# Usage: source .buildkite/scripts/rocm/stack.sh <base-dockerfile>
# e.g. docker/Dockerfile.rocm_base (ROCm 10.0) or docker/Dockerfile.rocm_72_base
#
# Exports:
# ROCM_STACK_BASE_DOCKERFILE the base Dockerfile
# ROCM_STACK_DOCKERFILE the matching vLLM Dockerfile (base name minus _base)
# ROCM_STACK_THEROCK 1 if the ROCm SDK comes from TheRock wheels
# ROCM_STACK_VERSION ROCm version, e.g. 10.0.0 or 7.2.3
# ROCM_STACK_VARIANT wheel variant and Docker Hub tag flavor from major.minor,
# same rule as setup.py (rocm100, rocm72)
# ROCM_STACK_IMAGE_KEY hash of the base Dockerfile (base image cache key)
# ROCM_STACK_WHEEL_KEY hash of the base Dockerfile plus files it COPYs from
# the repo (base wheel cache key)
# ROCM_STACK_ECR_BASE ECR tag of the cached base image
# ROCM_STACK_ECR_IMAGE ECR tag of this commit's release image
# ROCM_STACK_BASE_WHEELS_DIR local dir for base wheels (also the artifact path)
# ROCM_STACK_VLLM_WHEEL_DIR local dir for the vLLM wheel (also the artifact path)
rocm_stack_init() {
local base="$1"
if [[ ! -f "$base" ]]; then
echo "ROCm base Dockerfile not found: $base" >&2
return 2
fi
ROCM_STACK_BASE_DOCKERFILE="$base"
ROCM_STACK_DOCKERFILE="${base%_base}"
local sdk
sdk=$(sed -nE 's/^ARG ROCM_SDK_VERSION=([0-9.]+).*/\1/p' "$base" | head -1)
if [[ -n "$sdk" ]]; then
ROCM_STACK_THEROCK=1
ROCM_STACK_VERSION="$sdk"
else
ROCM_STACK_THEROCK=0
# BASE_IMAGE format: rocm/dev-ubuntu-22.04:7.2.3-complete -> 7.2.3
ROCM_STACK_VERSION=$(sed -nE 's/^ARG BASE_IMAGE=.*:([0-9]+(\.[0-9]+)+).*/\1/p' "$base" | head -1)
fi
if [[ -z "$ROCM_STACK_VERSION" ]]; then
echo "Could not determine the ROCm version from $base" >&2
return 2
fi
local major minor
IFS=. read -r major minor _ <<< "$ROCM_STACK_VERSION"
ROCM_STACK_VARIANT="rocm${major}${minor}"
ROCM_STACK_IMAGE_KEY=$(sha256sum "$base" | cut -c1-16)
local inputs
inputs=$(grep -E '^COPY ' "$base" | grep -v -- '--from' | awk '{print $2}' || true)
if [[ -z "$inputs" ]]; then
ROCM_STACK_WHEEL_KEY="$ROCM_STACK_IMAGE_KEY"
else
# shellcheck disable=SC2086
ROCM_STACK_WHEEL_KEY=$(sha256sum "$base" $inputs | sha256sum | cut -c1-16)
fi
local ecr="public.ecr.aws/q9t5s3a7/vllm-release-repo"
ROCM_STACK_ECR_BASE="${ecr}:${ROCM_STACK_IMAGE_KEY}-${ROCM_STACK_VARIANT}-base"
ROCM_STACK_ECR_IMAGE="${ecr}:${BUILDKITE_COMMIT:-unknown}-${ROCM_STACK_VARIANT}"
ROCM_STACK_BASE_WHEELS_DIR="artifacts/${ROCM_STACK_VARIANT}-base-wheels"
ROCM_STACK_VLLM_WHEEL_DIR="artifacts/${ROCM_STACK_VARIANT}-vllm-wheel"
export ROCM_STACK_BASE_DOCKERFILE ROCM_STACK_DOCKERFILE ROCM_STACK_THEROCK \
ROCM_STACK_VERSION ROCM_STACK_VARIANT ROCM_STACK_IMAGE_KEY \
ROCM_STACK_WHEEL_KEY ROCM_STACK_ECR_BASE ROCM_STACK_ECR_IMAGE \
ROCM_STACK_BASE_WHEELS_DIR ROCM_STACK_VLLM_WHEEL_DIR
}
if [[ $# -gt 0 ]]; then
rocm_stack_init "$1"
fi
@@ -23,7 +23,7 @@ export PATH="$CARGO_HOME/bin:$PATH"
CARGO_BINSTALL_VERSION="${CARGO_BINSTALL_VERSION:-1.20.1}"
UV_VERSION="${UV_VERSION:-0.11.28}"
UV_SCRIPT_PYTHON_VERSION="${UV_SCRIPT_PYTHON_VERSION:-3.12}"
PYO3_PYTHON_VERSION="${PYO3_PYTHON_VERSION:-3.12}"
CARGO_SORT_VERSION_REQ="${CARGO_SORT_VERSION_REQ:-2}"
CARGO_DENY_VERSION_REQ="${CARGO_DENY_VERSION_REQ:-0.20}"
@@ -60,12 +60,12 @@ install_cargo_binstall() {
install_cargo_sort() {
log_section "Installing cargo-sort ${CARGO_SORT_VERSION_REQ}"
cargo binstall --no-confirm --force --locked "cargo-sort@${CARGO_SORT_VERSION_REQ}"
cargo binstall --no-confirm --force "cargo-sort@${CARGO_SORT_VERSION_REQ}"
}
install_cargo_deny() {
log_section "Installing cargo-deny ${CARGO_DENY_VERSION_REQ}"
cargo binstall --no-confirm --force --locked "cargo-deny@${CARGO_DENY_VERSION_REQ}"
cargo binstall --no-confirm --force "cargo-deny@${CARGO_DENY_VERSION_REQ}"
}
install_cargo_nextest() {
@@ -74,7 +74,6 @@ install_cargo_nextest() {
--no-confirm \
--force \
--secure \
--locked \
"cargo-nextest@${CARGO_NEXTEST_VERSION_REQ}"
}
@@ -84,7 +83,34 @@ install_uv() {
"https://github.com/astral-sh/uv/releases/download/${UV_VERSION}/uv-installer.sh" \
| env UV_INSTALL_DIR="$CARGO_HOME/bin" sh
uv --version
uv python install "$UV_SCRIPT_PYTHON_VERSION"
}
setup_pyo3_python() {
log_section "Installing Python ${PYO3_PYTHON_VERSION} for PyO3 tests"
uv python install "$PYO3_PYTHON_VERSION"
PYO3_PYTHON="$(uv python find \
--managed-python \
--no-project \
--resolve-links \
"$PYO3_PYTHON_VERSION")"
export PYO3_PYTHON
local python_libdir
python_libdir="$("$PYO3_PYTHON" - <<'PY'
import pathlib
import sysconfig
libdir = pathlib.Path(sysconfig.get_config_var("LIBDIR"))
ldlibrary = sysconfig.get_config_var("LDLIBRARY")
assert sysconfig.get_config_var("Py_ENABLE_SHARED") == 1
assert ldlibrary
assert (libdir / ldlibrary).exists(), libdir / ldlibrary
print(libdir)
PY
)"
export LD_LIBRARY_PATH="${python_libdir}:${LD_LIBRARY_PATH:-}"
export LIBRARY_PATH="${python_libdir}:${LIBRARY_PATH:-}"
}
run_style_clippy() {
@@ -118,6 +144,7 @@ run_style_clippy() {
run_tests() {
install_uv
setup_pyo3_python
install_cargo_binstall
install_cargo_nextest
+24
View File
@@ -0,0 +1,24 @@
#!/bin/bash
set -euo pipefail
docker_root=$(docker info -f '{{.DockerRootDir}}')
if [ -z "$docker_root" ]; then
echo "Failed to determine Docker root directory."
exit 1
fi
echo "Docker root directory: $docker_root"
# Check disk usage of the filesystem where Docker's root directory is located
disk_usage=$(df "$docker_root" | tail -1 | awk '{print $5}' | sed 's/%//')
# Define the threshold
threshold=70
if [ "$disk_usage" -gt "$threshold" ]; then
echo "Disk usage is above $threshold%. Cleaning up Docker images and volumes..."
# Remove dangling images (those that are not tagged and not used by any container)
docker image prune -f
# Remove unused volumes / force the system prune for old images as well.
docker volume prune -f && docker system prune --force --filter "until=24h" --all
echo "Docker images and volumes cleanup completed."
else
echo "Disk usage is below $threshold%. No cleanup needed."
fi
+14
View File
@@ -0,0 +1,14 @@
# Environment config
TEST_NAME=llama8b
CONTAINER_NAME=tpu-test
# vllm config
MODEL=meta-llama/Llama-3.1-8B-Instruct
MAX_NUM_SEQS=256
MAX_NUM_BATCHED_TOKENS=1024
TENSOR_PARALLEL_SIZE=1
MAX_MODEL_LEN=2048
DOWNLOAD_DIR=/mnt/disks/persist
EXPECTED_THROUGHPUT=8.0
INPUT_LEN=1800
OUTPUT_LEN=128
+91
View File
@@ -0,0 +1,91 @@
#!/bin/bash
if [ ! -f "$1" ]; then
echo "Error: The env file '$1' does not exist."
exit 1 # Exit the script with a non-zero status to indicate an error
fi
ENV_FILE=$1
# For testing on local vm, use `set -a` to export all variables
source /etc/environment
# shellcheck source=/dev/null
source "$ENV_FILE"
remove_docker_container() {
docker rm -f "$CONTAINER_NAME" || true;
}
trap remove_docker_container EXIT
# Remove the container that might not be cleaned up in the previous run.
remove_docker_container
LOG_ROOT=$(mktemp -d)
# If mktemp fails, set -e will cause the script to exit.
echo "Results will be stored in: $LOG_ROOT"
if [ -z "$HF_TOKEN" ]; then
echo "Error: HF_TOKEN is not set or is empty."
exit 1
fi
# Make sure mounted disk or dir exists
if [ ! -d "$DOWNLOAD_DIR" ]; then
echo "Error: Folder $DOWNLOAD_DIR does not exist. This is useually a mounted drive. If no mounted drive, just create a folder."
exit 1
fi
echo "Run model $MODEL"
echo
echo "starting docker...$CONTAINER_NAME"
echo
docker run \
-v "$DOWNLOAD_DIR":"$DOWNLOAD_DIR" \
--env-file "$ENV_FILE" \
-e HF_TOKEN="$HF_TOKEN" \
-e TARGET_COMMIT="$BUILDKITE_COMMIT" \
-e MODEL="$MODEL" \
-e WORKSPACE=/workspace \
--name "$CONTAINER_NAME" \
-d \
--privileged \
--network host \
-v /dev/shm:/dev/shm \
vllm/vllm-tpu-bm tail -f /dev/null
echo "run script..."
echo
docker exec "$CONTAINER_NAME" /bin/bash -c ".buildkite/scripts/tpu/run_bm.sh"
echo "copy result back..."
VLLM_LOG="$LOG_ROOT/$TEST_NAME"_vllm_log.txt
BM_LOG="$LOG_ROOT/$TEST_NAME"_bm_log.txt
docker cp "$CONTAINER_NAME:/workspace/vllm_log.txt" "$VLLM_LOG"
docker cp "$CONTAINER_NAME:/workspace/bm_log.txt" "$BM_LOG"
throughput=$(grep "Request throughput (req/s):" "$BM_LOG" | sed 's/[^0-9.]//g')
echo "throughput for $TEST_NAME at $BUILDKITE_COMMIT: $throughput"
if [ "$BUILDKITE" = "true" ]; then
echo "Running inside Buildkite"
buildkite-agent artifact upload "$VLLM_LOG"
buildkite-agent artifact upload "$BM_LOG"
else
echo "Not running inside Buildkite"
fi
#
# compare the throughput with EXPECTED_THROUGHPUT
# and assert meeting the expectation
#
if [[ -z "$throughput" || ! "$throughput" =~ ^[0-9]+([.][0-9]+)?$ ]]; then
echo "Failed to get the throughput"
exit 1
fi
if (( $(echo "$throughput < $EXPECTED_THROUGHPUT" | bc -l) )); then
echo "Error: throughput($throughput) is less than expected($EXPECTED_THROUGHPUT)"
exit 1
fi
@@ -0,0 +1,14 @@
# Environment config
TEST_NAME=llama8bw8a8
CONTAINER_NAME=tpu-test
# vllm config
MODEL=RedHatAI/Meta-Llama-3.1-8B-Instruct-quantized.w8a8
MAX_NUM_SEQS=128
MAX_NUM_BATCHED_TOKENS=1024
TENSOR_PARALLEL_SIZE=1
MAX_MODEL_LEN=2048
DOWNLOAD_DIR=/mnt/disks/persist
EXPECTED_THROUGHPUT=8.7
INPUT_LEN=1800
OUTPUT_LEN=128
+93
View File
@@ -0,0 +1,93 @@
#!/bin/bash
set -euo pipefail
VLLM_LOG="$WORKSPACE/vllm_log.txt"
BM_LOG="$WORKSPACE/bm_log.txt"
if [ -n "$TARGET_COMMIT" ]; then
head_hash=$(git rev-parse HEAD)
if [ "$TARGET_COMMIT" != "$head_hash" ]; then
echo "Error: target commit $TARGET_COMMIT does not match HEAD: $head_hash"
exit 1
fi
fi
echo "model: $MODEL"
echo
#
# create a log folder
#
mkdir "$WORKSPACE/log"
# TODO: Move to image building.
pip install pandas
pip install datasets
#
# create sonnet_4x
#
echo "Create sonnet_4x.txt"
echo "" > benchmarks/sonnet_4x.txt
for _ in {1..4}
do
cat benchmarks/sonnet.txt >> benchmarks/sonnet_4x.txt
done
#
# start vllm service in backend
#
echo "lanching vllm..."
echo "logging to $VLLM_LOG"
echo
vllm serve "$MODEL" \
--seed 42 \
--max-num-seqs "$MAX_NUM_SEQS" \
--max-num-batched-tokens "$MAX_NUM_BATCHED_TOKENS" \
--tensor-parallel-size "$TENSOR_PARALLEL_SIZE" \
--no-enable-prefix-caching \
--download_dir "$DOWNLOAD_DIR" \
--max-model-len "$MAX_MODEL_LEN" > "$VLLM_LOG" 2>&1 &
echo "wait for 20 minutes.."
echo
# sleep 1200
# wait for 10 minutes...
for _ in {1..120}; do
# TODO: detect other type of errors.
if grep -Fq "raise RuntimeError" "$VLLM_LOG"; then
echo "Detected RuntimeError, exiting."
exit 1
elif grep -Fq "Application startup complete" "$VLLM_LOG"; then
echo "Application started"
break
else
echo "wait for 10 seconds..."
sleep 10
fi
done
#
# run test
#
echo "run benchmark test..."
echo "logging to $BM_LOG"
echo
vllm bench serve \
--backend vllm \
--model "$MODEL" \
--dataset-name sonnet \
--dataset-path benchmarks/sonnet_4x.txt \
--sonnet-input-len "$INPUT_LEN" \
--sonnet-output-len "$OUTPUT_LEN" \
--ignore-eos > "$BM_LOG"
echo "completed..."
echo
throughput=$(grep "Request throughput (req/s):" "$BM_LOG" | sed 's/[^0-9.]//g')
echo "throughput: $throughput"
echo
+20 -66
View File
@@ -4,31 +4,21 @@
#
# Upload ROCm wheels to S3 with proper index generation
#
# Usage: upload-rocm-wheels.sh <base-dockerfile>
# base-dockerfile the ROCm stack, e.g. docker/Dockerfile.rocm_base (see rocm/stack.sh)
#
# Required environment variables:
# AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY (or IAM role)
# S3_BUCKET (default: vllm-wheels)
#
# S3 path structure:
# s3://vllm-wheels/rocm/{commit}/<variant>-wheels/ - This stack's wheels for this commit
# s3://vllm-wheels/rocm/{commit}/<variant>/ - Index per ROCm variant (e.g. rocm100)
# s3://vllm-wheels/rocm/nightly/<variant>/ - Index pointing to latest nightly
# s3://vllm-wheels/rocm/{version}/<variant>/ - Index for release versions
# The top-level index.html at each prefix lists every variant present, since
# several ROCm stacks publish into the same prefixes.
# s3://vllm-wheels/rocm/{commit}/ - All wheels for this commit
# s3://vllm-wheels/rocm/nightly/ - Index pointing to latest nightly
# s3://vllm-wheels/rocm/{version}/ - Index for release versions
set -ex
# ======== Configuration ========
BUCKET="${S3_BUCKET:-vllm-wheels}"
# shellcheck source=.buildkite/scripts/rocm/stack.sh
source "$(dirname "${BASH_SOURCE[0]}")/rocm/stack.sh" "${1:?Usage: $0 <base-dockerfile>}"
ROCM_SUBPATH="rocm/${BUILDKITE_COMMIT}"
WHEEL_DIRS="$ROCM_STACK_BASE_WHEELS_DIR $ROCM_STACK_VLLM_WHEEL_DIR"
WHEEL_SUBPATH="$ROCM_SUBPATH/${ROCM_STACK_VARIANT}-wheels"
S3_WHEEL_PREFIX="s3://$BUCKET/$WHEEL_SUBPATH/"
S3_COMMIT_PREFIX="s3://$BUCKET/$ROCM_SUBPATH/"
INDICES_OUTPUT_DIR="rocm-indices"
echo "========================================"
@@ -59,11 +49,9 @@ source .buildkite/scripts/lib/manylinux.sh
# ======== Part 1: Collect and prepare wheels ========
# Collect all wheels
rm -rf all-rocm-wheels "$INDICES_OUTPUT_DIR"
mkdir -p all-rocm-wheels
for dir in $WHEEL_DIRS; do
cp "$dir"/*.whl all-rocm-wheels/ 2>/dev/null || true
done
cp artifacts/rocm-base-wheels/*.whl all-rocm-wheels/ 2>/dev/null || true
cp artifacts/rocm-vllm-wheel/*.whl all-rocm-wheels/ 2>/dev/null || true
WHEEL_COUNT=$(find all-rocm-wheels -maxdepth 1 -name '*.whl' 2>/dev/null | wc -l)
echo "Total wheels to upload: $WHEEL_COUNT"
@@ -96,9 +84,9 @@ ls -lh all-rocm-wheels/
# ======== Part 2: Upload wheels to S3 ========
echo ""
echo "Uploading wheels to $S3_WHEEL_PREFIX"
echo "Uploading wheels to $S3_COMMIT_PREFIX"
for wheel in all-rocm-wheels/*.whl; do
aws s3 cp "$wheel" "$S3_WHEEL_PREFIX"
aws s3 cp "$wheel" "$S3_COMMIT_PREFIX"
done
# ======== Part 3: Generate and upload indices ========
@@ -107,63 +95,28 @@ done
echo ""
echo "Generating indices..."
obj_json="rocm-objects.json"
aws s3api list-objects-v2 --bucket "$BUCKET" --prefix "$WHEEL_SUBPATH/" --delimiter / --output json > "$obj_json"
aws s3api list-objects-v2 --bucket "$BUCKET" --prefix "$ROCM_SUBPATH/" --delimiter / --output json > "$obj_json"
mkdir -p "$INDICES_OUTPUT_DIR"
# Use the existing generate-nightly-index.py
# HACK: Replace regex module with stdlib re (same as CUDA script)
sed -i 's/import regex as re/import re/g' .buildkite/scripts/generate-nightly-index.py
sed -i 's/import regex as re/import re/g' tools/vllm-rocm/therock_wheels.py
sed -i 's/import pybase64 as base64/import base64/g' tools/vllm-rocm/therock_wheels.py
INDEX_ARGS=(--version "$ROCM_SUBPATH" --wheel-dir "${WHEEL_SUBPATH#rocm/}")
# TheRock's SDK, device kernels and torchvision/torchaudio are linked, not hosted
if [[ "$ROCM_STACK_THEROCK" == "1" ]]; then
$PYTHON tools/vllm-rocm/therock_wheels.py external-links \
--dockerfile "$ROCM_STACK_BASE_DOCKERFILE" \
--torch-wheel "$(ls "$ROCM_STACK_BASE_WHEELS_DIR"/torch-*.whl)" > therock-external-links.txt
cat therock-external-links.txt
INDEX_ARGS+=(--external-links therock-external-links.txt)
fi
$PYTHON .buildkite/scripts/generate-nightly-index.py \
"${INDEX_ARGS[@]}" \
--version "$ROCM_SUBPATH" \
--current-objects "$obj_json" \
--output-dir "$INDICES_OUTPUT_DIR" \
--comment "ROCm commit $BUILDKITE_COMMIT"
# Upload this build's variant indices under an S3 prefix (e.g. rocm/nightly), then
# rebuild that prefix's top-level project list from every variant now present.
publish_indices() {
local prefix="$1"
local listing variants merged
echo "Uploading indices to s3://$BUCKET/$prefix/"
aws s3 cp --recursive "$INDICES_OUTPUT_DIR/" "s3://$BUCKET/$prefix/" --exclude "index.html"
listing=$(aws s3api list-objects-v2 --bucket "$BUCKET" --prefix "$prefix/" \
--delimiter / --query 'CommonPrefixes[].Prefix' --output text)
variants=$(tr '\t' '\n' <<< "$listing" | sed -n "s|^$prefix/\(rocm[0-9]*\)/$|\1|p" | sort -u)
merged=$(mktemp)
{
echo '<!DOCTYPE html>'
echo '<html>'
echo ' <meta name="pypi:repository-version" content="1.0">'
echo ' <body>'
for variant in $variants; do
echo " <a href=\"$variant/\">$variant/</a><br/>"
done
echo ' </body>'
echo '</html>'
} > "$merged"
echo "Variants under $prefix/: $(echo "$variants" | tr '\n' ' ')"
aws s3 cp "$merged" "s3://$BUCKET/$prefix/index.html"
rm -f "$merged"
}
publish_indices "$ROCM_SUBPATH"
# Upload indices to commit directory
echo "Uploading indices to $S3_COMMIT_PREFIX"
aws s3 cp --recursive "$INDICES_OUTPUT_DIR/" "$S3_COMMIT_PREFIX"
# Only scheduled nightly builds should update the moving nightly index.
if [[ "${NIGHTLY:-0}" == "1" ]]; then
publish_indices "rocm/nightly"
echo "Updating rocm/nightly/ index..."
aws s3 cp --recursive "$INDICES_OUTPUT_DIR/" "s3://$BUCKET/rocm/nightly/"
fi
# Extract version from vLLM wheel and update version-specific index
@@ -176,7 +129,8 @@ if [ -n "$VLLM_WHEEL" ]; then
echo "Pure version: $PURE_VERSION"
if [[ "$VERSION" != *"dev"* ]]; then
publish_indices "rocm/$PURE_VERSION"
echo "Updating rocm/$PURE_VERSION/ index..."
aws s3 cp --recursive "$INDICES_OUTPUT_DIR/" "s3://$BUCKET/rocm/$PURE_VERSION/"
fi
fi
@@ -188,14 +142,14 @@ echo "ROCm Wheel Upload Complete!"
echo "========================================"
echo ""
echo "Wheels available at:"
echo " s3://$BUCKET/$WHEEL_SUBPATH/"
echo " s3://$BUCKET/$ROCM_SUBPATH/"
echo ""
echo "Install command (by commit):"
echo " pip install vllm --extra-index-url https://${BUCKET}.s3.amazonaws.com/$ROCM_SUBPATH/$ROCM_STACK_VARIANT/"
echo " pip install vllm --extra-index-url https://${BUCKET}.s3.amazonaws.com/$ROCM_SUBPATH/"
echo ""
if [[ "${NIGHTLY:-0}" == "1" ]]; then
echo "Install command (nightly):"
echo " pip install vllm --extra-index-url https://${BUCKET}.s3.amazonaws.com/rocm/nightly/$ROCM_STACK_VARIANT/"
echo " pip install vllm --extra-index-url https://${BUCKET}.s3.amazonaws.com/rocm/nightly/"
fi
echo ""
echo "Wheel count: $WHEEL_COUNT"
+415 -484
View File
File diff suppressed because it is too large Load Diff
+8 -35
View File
@@ -2,9 +2,8 @@ group: Attention
depends_on:
- image-build
steps:
- label: ":nvidia: (H100) V1 Attention"
- label: ":nvidia: (H100) V1 Attention Shard %N"
key: v1-attention-h100-mi300
automatic_shard: true
timeout_in_minutes: 85
device: h100
source_file_dependencies:
@@ -13,18 +12,14 @@ steps:
- vllm/v1/attention
- tests/v1/attention
commands:
- pytest -v -s v1/attention
- pytest -v -s v1/attention --shard-id=$$BUILDKITE_PARALLEL_JOB --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT
parallelism: 2
mirror:
amd:
label: ":amd: (MI300) V1 Attention Shard %N"
dind: false
device: mi300_1
timeout_in_minutes: 125
# automatic_shard covers NVIDIA steps only; the mirror keeps its
# fixed pytest shards.
parallelism: 2
commands:
- pytest -v -s v1/attention --shard-id=$$BUILDKITE_PARALLEL_JOB --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT
depends_on:
- image-build-amd
source_file_dependencies:
@@ -35,14 +30,9 @@ steps:
- vllm/_aiter_ops.py
- vllm/envs.py
- vllm/platforms/rocm.py
- vllm/models/deepseek_v32/amd/rocm.py
commands:
- pytest -v -s v1/attention --ignore=v1/attention/test_sparse_mla_backends.py --shard-id=$$BUILDKITE_PARALLEL_JOB --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT
- VLLM_ROCM_USE_AITER=1 pytest -v -s v1/attention/test_sparse_mla_backends.py --shard-id=$$BUILDKITE_PARALLEL_JOB --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT
- label: ":nvidia: (B200) V1 Attention"
- label: ":nvidia: (B200) V1 Attention Shard %N"
key: v1-attention-b200
automatic_shard: true
timeout_in_minutes: 80
device: b200-k8s
source_file_dependencies:
@@ -51,16 +41,14 @@ steps:
- vllm/v1/attention
- tests/v1/attention
commands:
- pytest -v -s v1/attention
- pytest -v -s v1/attention --shard-id=$$BUILDKITE_PARALLEL_JOB --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT
parallelism: 2
mirror:
amd:
label: ":amd: (MI355 DPX) V1 Attention Shard %N"
dind: false
device: mi355_dpx
timeout_in_minutes: 125
# automatic_shard covers NVIDIA steps only; the mirror keeps its
# fixed pytest shards.
parallelism: 2
depends_on:
- image-build-amd
source_file_dependencies:
@@ -71,21 +59,6 @@ steps:
- vllm/_aiter_ops.py
- vllm/envs.py
- vllm/platforms/rocm.py
- vllm/models/deepseek_v32/amd/rocm.py
commands:
- pytest -v -s v1/attention --ignore=v1/attention/test_mla_backends.py --ignore=v1/attention/test_sparse_mla_backends.py --shard-id=$$BUILDKITE_PARALLEL_JOB --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT
- VLLM_ROCM_USE_AITER=1 pytest -v -s v1/attention/test_mla_backends.py v1/attention/test_sparse_mla_backends.py --shard-id=$$BUILDKITE_PARALLEL_JOB --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT
- label: ":nvidia: (DGX) Spark FlashInfer NVFP4 XQA"
key: flashinfer-nvfp4-xqa-dgx-spark
timeout_in_minutes: 30
device: dgx-spark
optional: true
num_devices: 1
depends_on:
- arm64-image-build
source_file_dependencies:
- vllm/v1/attention/backends/flashinfer.py
- tests/v1/attention/test_attention_backends.py
commands:
- pytest -v -s v1/attention/test_attention_backends.py -k test_flashinfer_xqa_nvfp4_decode_correctness
- pytest -v -s v1/attention --ignore=v1/attention/test_mla_backends.py --shard-id=$$BUILDKITE_PARALLEL_JOB --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT
- VLLM_ROCM_USE_AITER=1 pytest -v -s v1/attention/test_mla_backends.py --shard-id=$$BUILDKITE_PARALLEL_JOB --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT
+1 -1
View File
@@ -4,7 +4,7 @@ depends_on:
steps:
- label: ":nvidia: (H200 MIG 18GB) Benchmarks CLI"
key: benchmarks-cli-test
timeout_in_minutes: 55
timeout_in_minutes: 45
device: h200_18gb
source_file_dependencies:
- vllm/
+128
View File
@@ -2,6 +2,84 @@ group: Compile
depends_on:
- image-build
steps:
- label: ":nvidia: (B200) Sequence Parallel Correctness"
key: sequence-parallel-correctness-tests-2xb200
timeout_in_minutes: 45
working_dir: "/vllm-workspace/"
device: b200-k8s
optional: true
num_devices: 2
commands:
- export VLLM_TEST_CLEAN_GPU_MEMORY=1
- pytest -v -s tests/compile/correctness_e2e/test_sequence_parallel.py
mirror:
amd:
label: ":amd: (MI355) Sequence Parallel Correctness"
dind: false
device: mi355_2
num_devices: 2
timeout_in_minutes: 180
depends_on:
- image-build-amd
source_file_dependencies:
- .buildkite/test_areas/compile.yaml
- vllm/compilation/
- vllm/config/
- vllm/distributed/
- vllm/model_executor/kernels/linear/
- vllm/model_executor/layers/
- vllm/model_executor/models/llama.py
- vllm/v1/attention/
- vllm/v1/worker/
- vllm/v1/cudagraph_dispatcher.py
- vllm/platforms/rocm.py
- tests/compile/correctness_e2e/test_sequence_parallel.py
- tests/models/registry.py
- tests/utils.py
- label: ":nvidia: (B200) AsyncTP Correctness"
key: asynctp-correctness-tests-b200
timeout_in_minutes: 30
working_dir: "/vllm-workspace/"
device: b200-k8s
optional: true
num_devices: 2
commands:
- export VLLM_TEST_CLEAN_GPU_MEMORY=1
- pytest -v -s tests/compile/correctness_e2e/test_async_tp.py
mirror:
amd:
label: ":amd: (MI355) AsyncTP Correctness"
dind: false
device: mi355_2
num_devices: 2
timeout_in_minutes: 60
working_dir: "/vllm-workspace/"
depends_on:
- image-build-amd
source_file_dependencies:
- .buildkite/test_areas/compile.yaml
- vllm/compilation/
- vllm/config/
- vllm/distributed/
- vllm/model_executor/layers/
- vllm/model_executor/kernels/linear/
- vllm/model_executor/models/llama.py
- vllm/model_executor/models/qwen3.py
- vllm/v1/attention/
- vllm/_aiter_ops.py
- vllm/platforms/rocm.py
- vllm/transformers_utils/repo_utils.py
- tests/compile/correctness_e2e/test_async_tp.py
- tests/compile/fusions_e2e/common.py
- tests/models/registry.py
- tests/utils.py
commands:
- export VLLM_TEST_CLEAN_GPU_MEMORY=1
- pytest -v -s tests/compile/correctness_e2e/test_async_tp.py::test_rocm_async_tp_bf16_output_correctness
- pytest -v -s tests/compile/correctness_e2e/test_async_tp.py::test_async_tp_pass_correctness -k RedHatAI
- pytest -v -s tests/compile/correctness_e2e/test_async_tp.py::test_async_tp_pass_nvfp4_correctness
- label: ":nvidia: (H100) Distributed Compile"
key: distributed-compile-unit-tests-2xh100
timeout_in_minutes: 45
@@ -222,6 +300,7 @@ steps:
- nvidia-smi
# Run all models and attn backends but only Inductor partition and native custom ops
- pytest -v -s tests/compile/fusions_e2e/test_tp2_ar_rms.py -k "inductor_partition and not +rms_norm and (not +quant_fp8 or +quant_fp8 and (qwen3 or deepseek))"
- pytest -v -s tests/compile/fusions_e2e/test_tp2_async_tp.py -k "inductor_partition and not +rms_norm and (not +quant_fp8 or +quant_fp8 and (qwen3 or deepseek))"
mirror:
amd:
label: ":amd: (MI355) Fusion E2E TP2 Quick"
@@ -309,6 +388,53 @@ steps:
- amd-smi
- VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_CUSTOM_AR=1 pytest -v -s tests/compile/fusions_e2e/test_tp2_ar_rms.py -k 'test_tp2_ar_rms_fp8_fusions and Llama-3 and ROCM_AITER_UNIFIED_ATTN and (dynamo_partition or +quant_fp8 or +rms_norm)'
- label: ":nvidia: (H100) Fusion E2E TP2 AsyncTP Config Sweep"
key: fusion-e2e-tp2-asynctp-config-sweep-h100
timeout_in_minutes: 40
working_dir: "/vllm-workspace/"
device: h100
num_devices: 2
source_file_dependencies:
- csrc/quantization/
- vllm/compilation/
# can affect pattern matching
- vllm/model_executor/layers/layernorm.py
- vllm/model_executor/layers/activation.py
- vllm/model_executor/layers/attention/attention.py
- vllm/model_executor/layers/quantization/input_quant_fp8.py
- tests/compile/fusions_e2e/
commands:
- nvidia-smi
# Run just llama3 (fp8 & bf16) for all config combinations
- pytest -v -s tests/compile/fusions_e2e/test_tp2_async_tp.py -k "llama-3"
mirror:
amd:
label: ':amd: (MI355) Fusion E2E TP2 AsyncTP BF16 Config Sweep'
dind: false
device: mi355_2
num_devices: 2
working_dir: /vllm-workspace/
timeout_in_minutes: 30
depends_on:
- image-build-amd
source_file_dependencies:
- vllm/compilation/
- vllm/model_executor/layers/linear.py
- vllm/model_executor/layers/utils.py
- vllm/distributed/
- vllm/v1/attention/backends/triton_attn.py
- tests/compile/fusions_e2e/
- tests/compile/passes/distributed/test_async_tp.py
- tests/compile/rocm/test_async_tp.py
- tests/compile/correctness_e2e/test_async_tp.py
- vllm/model_executor/kernels/linear/
- vllm/model_executor/models/qwen3.py
commands:
- rocm-smi
- pytest -v -s tests/compile/rocm/test_async_tp.py
- pytest -v -s tests/compile/fusions_e2e/test_tp2_rocm_async_tp.py
- pytest -v -s tests/compile/correctness_e2e/test_async_tp.py::test_rocm_async_tp_bf16_output_correctness
- label: ":nvidia: (B200) Fusion E2E TP2"
key: fusion-e2e-tp2-b200
timeout_in_minutes: 45
@@ -327,6 +453,7 @@ steps:
# include qwen/deepseek with +quant_fp8 as -quant_fp8 rms+quant fusion is not supported
# for ar-rms-quant-fp4, also sweep llama3
- pytest -v -s tests/compile/fusions_e2e/test_tp2_ar_rms.py -k "(FLASHINFER and inductor_partition and not +rms_norm and (not +quant_fp8 or +quant_fp8 and (qwen3 or deepseek))) or Llama-3.1-8B-Instruct-FP4"
- pytest -v -s tests/compile/fusions_e2e/test_tp2_async_tp.py -k "FLASHINFER and inductor_partition and not +rms_norm and (not +quant_fp8 or +quant_fp8 and (qwen3 or deepseek))"
mirror:
amd:
label: ":amd: (MI355) Fusion E2E TP2 FP8 BF16 rewrites"
@@ -353,6 +480,7 @@ steps:
# TODO: FP4 output-quant and trained MXFP4 AR/RMS fusion remain unsupported.
- amd-smi
- VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_CUSTOM_AR=1 pytest -v -s tests/compile/fusions_e2e/test_tp2_ar_rms.py -k 'test_tp2_ar_rms_fp8_fusions and qwen3 and ROCM_AITER_UNIFIED_ATTN and +quant_fp8 and (dynamo_partition or not +rms_norm)'
- pytest -v -s tests/compile/fusions_e2e/test_tp2_rocm_moe.py
- label: ':amd: (MI355) Fusion E2E TP2 AR-RMS (Inductor Partition)'
key: fusion-e2e-tp2-ar-rms-inductor-partition-amd
+1 -7
View File
@@ -33,20 +33,14 @@ steps:
- label: ":nvidia: (H200 MIG 18GB) CUDAGraph"
device: h200_18gb
key: cudagraph
automatic_shard: true
timeout_in_minutes: 45
timeout_in_minutes: 30
source_file_dependencies:
- tests/v1/cudagraph
- vllm/v1/cudagraph_dispatcher.py
- vllm/config/compilation.py
- vllm/config/vllm.py
- vllm/compilation
- vllm/v1/worker/gpu/cudagraph_utils.py
- vllm/v1/worker/gpu/dp_utils.py
- vllm/v1/worker/encoder_cudagraph.py
- vllm/v1/worker/encoder_cudagraph_defs.py
- vllm/v1/worker/gpu/model_runner.py
- vllm/v1/worker/gpu/ubatch_utils.py
commands:
- pytest -v -s v1/cudagraph/test_cudagraph_dispatch.py
- pytest -v -s v1/cudagraph/test_cudagraph_mode.py
+1 -1
View File
@@ -426,7 +426,7 @@ steps:
# Keep TP=2 prefill and TP=1 decode above both hybrid page-size minima.
PREFILL_BLOCK_SIZE: "2048"
DECODE_BLOCK_SIZE: "2048"
VLLM_SERVE_EXTRA_ARGS: "--trust-remote-code,--max-model-len 32768"
VLLM_SERVE_EXTRA_ARGS: "--trust-remote-code,--mamba-cache-mode align,--max-model-len 32768"
commands:
- bash /vllm-workspace/.buildkite/scripts/install-kv-connectors.sh
- bash v1/kv_connector/nixl_integration/run_accuracy_test.sh
@@ -4,8 +4,6 @@ depends_on:
steps:
- label: ":nvidia: (B200) Distributed MooncakeConnector PD accuracy"
key: distributed-mooncakeconnector-pd-accuracy-4-gpus
# Nightly only: 4-GPU B200 pods wait hours behind smaller jobs on per-PR runs.
optional: true
timeout_in_minutes: 10
working_dir: "/vllm-workspace/tests"
# Runs on b200-k8s because it is the only NVIDIA pool that can carry mooncake
@@ -24,7 +22,6 @@ steps:
mirror:
amd:
label: ":amd: (MI355) Distributed MooncakeConnector HIP PD accuracy"
optional: true
dind: false
device: mi355_4
num_devices: 4
+6 -17
View File
@@ -14,7 +14,6 @@ steps:
- tests/distributed
commands:
- pytest -v -s distributed/test_comm_ops.py
- pytest -v -s distributed/test_cpu_all_reduce.py
- pytest -v -s distributed/test_shm_broadcast.py
- pytest -v -s distributed/test_shm_buffer.py
- pytest -v -s distributed/test_shm_storage.py
@@ -78,7 +77,6 @@ steps:
- label: ":nvidia: (L4) Distributed DP Basic"
key: distributed-dp-tests-2-gpus
automatic_shard: true
timeout_in_minutes: 45
working_dir: "/vllm-workspace/tests"
device: l4
@@ -110,7 +108,6 @@ steps:
amd:
label: ":amd: (MI355) Distributed DP Basic"
dind: false
automatic_shard: true
device: mi355_2
timeout_in_minutes: 45
depends_on:
@@ -303,7 +300,6 @@ steps:
- label: ":nvidia: (L4) Distributed DP Extended"
key: distributed-dp-tests-4-gpus
automatic_shard: true
timeout_in_minutes: 45
working_dir: "/vllm-workspace/tests"
device: l4
@@ -333,7 +329,6 @@ steps:
amd:
label: ':amd: (MI355) Distributed DP Extended'
dind: false
automatic_shard: true
device: mi355_4
timeout_in_minutes: 60
working_dir: /vllm-workspace/tests
@@ -402,7 +397,6 @@ steps:
- label: ":nvidia: (H100) Distributed DP + EP"
key: distributed-tests-8xh100
timeout_in_minutes: 20
optional: true
device: h100
num_devices: 8
working_dir: "/vllm-workspace/tests"
@@ -425,7 +419,6 @@ steps:
amd:
label: ":amd: (MI355) Distributed DP + EP"
dind: false
optional: false
device: mi355_8
timeout_in_minutes: 50
depends_on:
@@ -513,7 +506,9 @@ steps:
- examples/features/data_parallel/data_parallel_offline.py
- vllm/_aiter_ops.py
- vllm/platforms/rocm.py
- csrc/rocm/quickreduce/
- csrc/custom_quickreduce.cu
- csrc/ops.h
- csrc/torch_bindings.cpp
- vllm/model_executor/layers/
- vllm/entrypoints/llm.py
- vllm/config/parallel.py
@@ -525,17 +520,13 @@ steps:
- tests/distributed/test_rocm_aiter_custom_ar.py
- tests/distributed/test_rocm_quick_reduce.py
- tests/distributed/test_quick_all_reduce.py
- tests/distributed/test_aiter_rms_fallback.py
- tests/v1/e2e/general/test_rocm_aiter_custom_ar.py
- tests/utils.py
- examples/rl/rlhf_async_new_apis.py
- tests/distributed/test_weight_transfer.py
- tests/distributed/test_packed_tensor.py
- tests/v1/attention/test_rocm_mla_dcp_context.py
- vllm/v1/attention/ops/rocm_aiter_mla_prefill.py
commands:
- pytest -v -s tests/distributed/test_context_parallel.py
- pytest -v -s tests/v1/attention/test_rocm_mla_dcp_context.py
- VLLM_ALLOW_INSECURE_SERIALIZATION=1 python3 examples/rl/rlhf_async_new_apis.py
- VLLM_LOGGING_LEVEL=DEBUG python3 examples/features/data_parallel/data_parallel_offline.py --model=Qwen/Qwen1.5-MoE-A2.7B -tp=1 -dp=2 --max-model-len=2048 --all2all-backend=deepep_high_throughput
- VLLM_LOGGING_LEVEL=DEBUG python3 examples/features/data_parallel/data_parallel_offline.py --model=Qwen/Qwen1.5-MoE-A2.7B -tp=1 -dp=2 --max-model-len=2048 --all2all-backend=allgather_reducescatter --disable-nccl-for-dp-synchronization
@@ -546,7 +537,6 @@ steps:
- pytest -v -s tests/v1/e2e/general/test_rocm_aiter_custom_ar.py
- pytest -v -s tests/distributed/test_rocm_quick_reduce.py
- pytest -v -s tests/distributed/test_quick_all_reduce.py
- pytest -v -s tests/distributed/test_aiter_rms_fallback.py
- label: ":nvidia: (B200) Distributed"
key: distributed-tests-2xb200
@@ -640,11 +630,10 @@ steps:
- tests/distributed/
- examples/features/data_parallel/data_parallel_offline.py
commands:
- ./.buildkite/scripts/run-multi-node-test.sh /vllm-workspace/tests 2 2 $IMAGE_TAG "VLLM_TEST_SAME_HOST=0 torchrun --nnodes 2 --nproc-per-node=2 --rdzv_backend=c10d --rdzv_endpoint=192.168.10.10 distributed/test_same_node.py | grep 'Same node test passed' && NUM_NODES=2 torchrun --nnodes 2 --nproc-per-node=2 --rdzv_backend=c10d --rdzv_endpoint=192.168.10.10 distributed/test_node_count.py | grep 'Node count test passed' && python3 ../examples/features/data_parallel/data_parallel_offline.py -dp=2 -tp=1 --dp-num-nodes=2 --dp-node-rank=0 --dp-master-addr=192.168.10.10 --dp-master-port=12345 --enforce-eager --trust-remote-code && VLLM_MULTI_NODE=1 pytest -v -s distributed/test_pipeline_parallel.py" "VLLM_TEST_SAME_HOST=0 torchrun --nnodes 2 --nproc-per-node=2 --rdzv_backend=c10d --rdzv_endpoint=192.168.10.10 distributed/test_same_node.py | grep 'Same node test passed' && NUM_NODES=2 torchrun --nnodes 2 --nproc-per-node=2 --rdzv_backend=c10d --rdzv_endpoint=192.168.10.10 distributed/test_node_count.py | grep 'Node count test passed' && python3 ../examples/features/data_parallel/data_parallel_offline.py -dp=2 -tp=1 --dp-num-nodes=2 --dp-node-rank=1 --dp-master-addr=192.168.10.10 --dp-master-port=12345 --enforce-eager --trust-remote-code"
- ./.buildkite/scripts/run-multi-node-test.sh /vllm-workspace/tests 2 2 $IMAGE_TAG "VLLM_TEST_SAME_HOST=0 torchrun --nnodes 2 --nproc-per-node=2 --rdzv_backend=c10d --rdzv_endpoint=192.168.10.10 distributed/test_same_node.py | grep 'Same node test passed' && NUM_NODES=2 torchrun --nnodes 2 --nproc-per-node=2 --rdzv_backend=c10d --rdzv_endpoint=192.168.10.10 distributed/test_node_count.py | grep 'Node count test passed' && python3 ../examples/features/data_parallel/data_parallel_offline.py -dp=2 -tp=1 --dp-num-nodes=2 --dp-node-rank=0 --dp-master-addr=192.168.10.10 --dp-master-port=12345 --enforce-eager --trust-remote-code && VLLM_MULTI_NODE=1 pytest -v -s distributed/test_multi_node_assignment.py && VLLM_MULTI_NODE=1 pytest -v -s distributed/test_pipeline_parallel.py" "VLLM_TEST_SAME_HOST=0 torchrun --nnodes 2 --nproc-per-node=2 --rdzv_backend=c10d --rdzv_endpoint=192.168.10.10 distributed/test_same_node.py | grep 'Same node test passed' && NUM_NODES=2 torchrun --nnodes 2 --nproc-per-node=2 --rdzv_backend=c10d --rdzv_endpoint=192.168.10.10 distributed/test_node_count.py | grep 'Node count test passed' && python3 ../examples/features/data_parallel/data_parallel_offline.py -dp=2 -tp=1 --dp-num-nodes=2 --dp-node-rank=1 --dp-master-addr=192.168.10.10 --dp-master-port=12345 --enforce-eager --trust-remote-code"
- label: ":nvidia: (L4) Pipeline + Context Parallelism"
key: pipeline-context-parallelism-4-gpus
automatic_shard: true
timeout_in_minutes: 55
working_dir: "/vllm-workspace/tests"
device: l4
@@ -686,7 +675,6 @@ steps:
- label: ":nvidia: (L4) RayExecutorV2"
key: rayexecutorv2-4-gpus
automatic_shard: true
timeout_in_minutes: 45
working_dir: "/vllm-workspace/tests"
device: l4
@@ -700,6 +688,7 @@ steps:
- tests/distributed/test_pipeline_parallel.py
- tests/basic_correctness/models/test_basic_correctness.py
commands:
- export VLLM_USE_RAY_V2_EXECUTOR_BACKEND=1
- export NCCL_CUMEM_HOST_ENABLE=0
- pytest -v -s distributed/test_ray_v2_executor.py
- pytest -v -s distributed/test_ray_v2_executor_e2e.py
@@ -709,7 +698,6 @@ steps:
amd:
label: ':amd: (MI355) RayExecutorV2'
dind: false
automatic_shard: true
device: mi355_4
timeout_in_minutes: 75
working_dir: /vllm-workspace/tests
@@ -725,6 +713,7 @@ steps:
- tests/basic_correctness/models/test_basic_correctness.py
- vllm/platforms/rocm.py
commands:
- export VLLM_USE_RAY_V2_EXECUTOR_BACKEND=1
- pytest -v -s distributed/test_ray_v2_executor.py
- pytest -v -s distributed/test_ray_v2_executor_e2e.py
- pytest -v -s distributed/test_pipeline_parallel.py -k "ray"
+3 -6
View File
@@ -38,7 +38,6 @@ steps:
- label: ":nvidia: (L4) V1 Engine"
key: engine-1-gpu
automatic_shard: true
device: l4
num_devices: 1
timeout_in_minutes: 45
@@ -109,7 +108,6 @@ steps:
- label: ":nvidia: (H200 MIG 18GB) E2E Core"
device: h200_18gb
key: e2e-core-1-gpu
automatic_shard: true
timeout_in_minutes: 65
source_file_dependencies:
- vllm/v1/
@@ -141,7 +139,7 @@ steps:
- tests/v1/e2e/general/
commands:
- pytest -v -s v1/e2e/general/test_kv_sharing_fast_prefill.py
- pytest -v -s v1/e2e/general/test_mamba_prefix_cache.py::test_mamba_prefix_cache_mrv2 v1/e2e/general/test_mamba_prefix_cache.py::test_mamba_prefix_cache_mrv2_async v1/e2e/general/test_mamba_prefix_cache.py::test_mamba_prefix_cache_mrv2_async_flashinfer
- pytest -v -s v1/e2e/general/test_mamba_prefix_cache.py::test_mamba_prefix_cache_mrv2 v1/e2e/general/test_mamba_prefix_cache.py::test_mamba_prefix_cache_mrv2_async
mirror:
amd:
label: ":amd: (MI355 DPX) E2E Core Large Memory"
@@ -164,7 +162,6 @@ steps:
device: h100
num_devices: 2
key: replayssm-e2e
automatic_shard: true
timeout_in_minutes: 90
working_dir: "/vllm-workspace/tests"
source_file_dependencies:
@@ -238,9 +235,9 @@ steps:
- pytest -v -s v1/e2e/test_hybrid_chunked_prefill.py
mirror:
amd:
label: ":amd: (MI355 DPX) V1 E2E Hybrid Chunked Prefill"
label: ":amd: (MI300) V1 E2E Hybrid Chunked Prefill"
dind: false
device: mi355_dpx
device: mi300_1
timeout_in_minutes: 35
depends_on:
- image-build-amd
+2 -11
View File
@@ -26,22 +26,21 @@ steps:
- label: ":nvidia: (H200 MIG 35GB) Entrypoints Unit"
device: h200_35gb
key: entrypoints-unit-tests
automatic_shard: true
timeout_in_minutes: 40
working_dir: "/vllm-workspace/tests"
source_file_dependencies:
- vllm/entrypoints
- vllm/utils/flashinfer.py
- tests/entrypoints/unit_tests
- tests/entrypoints/weight_transfer
- tests/entrypoints/launchers
commands:
- pytest -v -s entrypoints/unit_tests
- pytest -v -s entrypoints/weight_transfer
- pytest -v -s entrypoints/launchers
mirror:
amd:
label: ":amd: (MI355 DPX) Entrypoints Unit"
dind: false
automatic_shard: true
device: mi355_dpx
timeout_in_minutes: 50
depends_on:
@@ -122,7 +121,6 @@ steps:
- label: ":nvidia: (H200 MIG 35GB) Entrypoints Integration (OpenAI API chat_completion)"
device: h200_35gb
key: entrypoints-integration-api-server-openai-chat_completion
automatic_shard: true
timeout_in_minutes: 83
working_dir: "/vllm-workspace/tests"
source_file_dependencies:
@@ -144,7 +142,6 @@ steps:
- label: ":nvidia: (H200 MIG 35GB) Entrypoints Integration (API Server Generate)"
device: h200_35gb
key: entrypoints-integration-api-server-generate
automatic_shard: true
timeout_in_minutes: 50
working_dir: "/vllm-workspace/tests"
source_file_dependencies:
@@ -173,7 +170,6 @@ steps:
- label: ":nvidia: (H200 MIG 35GB) Entrypoints Integration (Responses API)"
device: h200_35gb
key: entrypoints-integration-responses-api
automatic_shard: true
timeout_in_minutes: 50
working_dir: "/vllm-workspace/tests"
source_file_dependencies:
@@ -194,7 +190,6 @@ steps:
- label: ":nvidia: (H200 MIG 35GB) Entrypoints Integration (Speech to Text)"
device: h200_35gb
key: entrypoints-integration-speech_to_text
automatic_shard: true
timeout_in_minutes: 45
working_dir: "/vllm-workspace/tests"
source_file_dependencies:
@@ -208,7 +203,6 @@ steps:
amd:
label: ":amd: (MI355 DPX) Entrypoints Integration (Speech to Text)"
dind: false
automatic_shard: true
device: mi355_dpx
timeout_in_minutes: 60
working_dir: "/vllm-workspace/tests"
@@ -218,7 +212,6 @@ steps:
- label: ":nvidia: (H200 MIG 35GB) Entrypoints Integration (Multimodal)"
device: h200_35gb
key: entrypoints-integration-multimodal
automatic_shard: true
timeout_in_minutes: 45
working_dir: "/vllm-workspace/tests"
source_file_dependencies:
@@ -240,7 +233,6 @@ steps:
- label: ":nvidia: (H200 MIG 35GB) Entrypoints Integration (Pooling)"
device: h200_35gb
key: entrypoints-integration-pooling
automatic_shard: true
timeout_in_minutes: 75
working_dir: "/vllm-workspace/tests"
source_file_dependencies:
@@ -254,7 +246,6 @@ steps:
amd:
label: ":amd: (MI355 DPX) Entrypoints Integration (Pooling)"
dind: false
automatic_shard: true
device: mi355_dpx
timeout_in_minutes: 65
working_dir: "/vllm-workspace/tests"
@@ -58,8 +58,6 @@ steps:
- label: ":nvidia: (H100) Elastic EP Scaling"
key: elastic-ep-scaling-test
# Nightly only: 4-GPU H100 pods wait behind smaller jobs on per-PR runs.
optional: true
timeout_in_minutes: 40
device: h100
working_dir: "/vllm-workspace/tests"
+34 -99
View File
@@ -30,10 +30,9 @@ steps:
- vllm/_aiter_ops.py
- vllm/platforms/rocm.py
- label: ":nvidia: (H200 MIG 35GB) Core Operation Kernels"
- label: ":nvidia: (H200 MIG 35GB) Core Operation Kernels Shard %N"
device: h200_35gb
key: kernels-core-operation-test
automatic_shard: true
timeout_in_minutes: 120
source_file_dependencies:
- csrc/
@@ -41,10 +40,10 @@ steps:
- tests/kernels/test_concat_mla_q.py
- tests/kernels/test_fused_qk_norm_rope_gate.py
commands:
- pytest -v -s kernels/core --ignore=kernels/core/test_minimax_reduce_rms.py kernels/test_concat_mla_q.py kernels/test_fused_qk_norm_rope_gate.py
- pytest -v -s kernels/core --ignore=kernels/core/test_minimax_reduce_rms.py kernels/test_concat_mla_q.py kernels/test_fused_qk_norm_rope_gate.py --shard-id=$$BUILDKITE_PARALLEL_JOB --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT
parallelism: 3
mirror:
amd:
parallelism: 3
label: ":amd: (MI300) Core Operation Kernels Shard %N"
dind: false
device: mi300_1
@@ -73,10 +72,10 @@ steps:
num_devices: 2
device: h100
source_file_dependencies:
- csrc/minimax_reduce_rms_kernel.cu
- csrc/minimax_reduce_rms_kernel.h
- csrc/ops.h
- csrc/torch_bindings.cpp
- csrc/libtorch_stable/minimax_reduce_rms_kernel.cu
- csrc/libtorch_stable/minimax_reduce_rms_kernel.h
- csrc/libtorch_stable/ops.h
- csrc/libtorch_stable/torch_bindings.cpp
- vllm/model_executor/layers/minimax_rms_norm/
- vllm/model_executor/layers/mamba/linear/minimax_linear_attn.py
- tests/kernels/core/test_minimax_reduce_rms.py
@@ -123,7 +122,7 @@ steps:
depends_on:
- image-build-amd
source_file_dependencies:
- csrc/fused_deepseek_v4_qnorm_rope_kv_insert_kernel.cu
- csrc/libtorch_stable/fused_deepseek_v4_qnorm_rope_kv_insert_kernel.cu
- vllm/models/deepseek_v4/common/ops/
- tests/kernels/test_fused_deepseek_v4_qnorm_rope_kv_insert.py
- vllm/platforms/rocm.py
@@ -138,15 +137,11 @@ steps:
- csrc/fused_deepseek_v4_qnorm_rope_kv_insert_kernel.cu
- vllm/models/deepseek_v4/common/ops/
- vllm/models/deepseek_v4/nvidia/
- vllm/model_executor/layers/fused_moe/deep_gemm_mega_moe.py
- vllm/model_executor/layers/fused_moe/prepare_megamoe.py
- vllm/utils/deep_gemm.py
- tests/kernels/test_fused_deepseek_v4_qnorm_rope_kv_insert.py
- tests/kernels/moe/test_deep_gemm_mega_moe.py
- tests/models/test_deepseek_v4_mega_moe.py
commands:
- pytest -v -s kernels/test_fused_deepseek_v4_*.py
- pytest -v -s kernels/moe/test_deep_gemm_mega_moe.py
- pytest -v -s models/test_deepseek_v4_mega_moe.py
mirror:
amd:
@@ -160,7 +155,6 @@ steps:
- image-build-amd
source_file_dependencies:
- tests/models/test_deepseek_v4_vl_rocm.py
- tests/models/test_deepseek_v4_aiter_mega_moe.py
- vllm/models/deepseek_v4/amd/
- vllm/models/deepseek_v4/common/
- vllm/model_executor/layers/fused_moe/
@@ -176,7 +170,6 @@ steps:
# Common qnorm/RoPE/cache kernels already run in the H100 AMD mirror.
# Pin the tested BF16 Triton routing and shared-expert path before imports.
- VLLM_ROCM_USE_AITER_MOE=0 pytest -v -s models/test_deepseek_v4_vl_rocm.py
- pytest -v -s models/test_deepseek_v4_aiter_mega_moe.py
# Own job so mHC never shares a process with FlashInfer's import-time
# ptxas override.
@@ -235,8 +228,8 @@ steps:
# Catch-all for test files at the tests/kernels root. This job collects
# the whole root so new files are wired by default.
# Files with dedicated jobs elsewhere in this file are excluded via --ignore
# (test_bf16x3_router_gemm_cutedsl runs in Kernels (B200); test_ll_bf16_gemm and
# test_ll_fp32w_gemm run in Gate Linear CuTeDSL Kernels (B200)).
# (test_bf16x3_router_gemm_cutedsl and test_ll_bf16_gemm run in their own jobs /
# Kernels (B200)).
- label: ":nvidia: (B200) Miscellaneous Kernels"
key: kernels-root-misc-test-b200
timeout_in_minutes: 30
@@ -261,11 +254,8 @@ steps:
--ignore=kernels/test_fused_qk_norm_rope_gate.py
--ignore=kernels/test_fused_deepseek_v4_qnorm_rope_kv_insert.py
--ignore=kernels/test_top_k_per_row.py
--ignore=kernels/test_cpu_fused_sampling.py
--ignore=kernels/test_bf16x3_router_gemm_cutedsl.py
--ignore=kernels/test_bf16x3_router_gemm_rocm.py
--ignore=kernels/test_ll_bf16_gemm.py
--ignore=kernels/test_ll_fp32w_gemm.py
--ignore=kernels/test_shuffle_rows.py
# BROKEN on main, pending kernel fixes (B200):
# test_shuffle_rows.py (1: test_shuffle_rows_edge_cases)
@@ -304,12 +294,14 @@ steps:
kernels/test_fused_inv_rope_fp8_quant.py
kernels/test_fused_minimax_m3_qknorm_rope_kv_insert.py
kernels/test_fused_quant_activation.py
kernels/test_gate_linear_rocm_dispatch.py
kernels/test_gemm_rs_ar.py
kernels/test_kpool_decode_update_batched.py
kernels/test_minimax_m3_amd_ops.py
kernels/test_minimax_m3_sparse_attn_fp8_scale.py
kernels/test_onednn.py
kernels/test_relu2_fp8_quant.py
kernels/test_rocm_fp32_router_gemm.py
--deselect=tests/kernels/test_bf16_skinny_gemm.py::test_every_dsv3_routed_shape_is_instantiated
- label: ":nvidia: (L4) Attention Kernels Shard %N"
@@ -319,7 +311,6 @@ steps:
timeout_in_minutes: 30
source_file_dependencies:
- csrc/attention/
- csrc/core/vec_dtypes/
- vllm/v1/attention
# TODO: remove this dependency (https://github.com/vllm-project/vllm/issues/32267)
- vllm/model_executor/layers/attention
@@ -339,7 +330,6 @@ steps:
- image-build-amd
source_file_dependencies:
- csrc/attention/
- csrc/core/vec_dtypes/
- vllm/v1/attention
- vllm/model_executor/layers/attention
- tests/kernels/attention
@@ -412,14 +402,10 @@ steps:
depends_on:
- image-build-amd
source_file_dependencies:
- vllm/models/deepseek_v41/amd/rocm.py
- vllm/models/deepseek_v41/sparse_mla.py
- vllm/v1/attention/backends/mla/rocm_aiter_mla.py
- vllm/v1/attention/ops/rocm_aiter_mla_sparse.py
- vllm/v1/attention/selector.py
- vllm/platforms/rocm.py
- vllm/_aiter_ops.py
- tests/kernels/attention/test_rocm_triton_attn_dsv4.py
- tests/kernels/attention/test_rocm_aiter_mla_decode.py
- tests/kernels/attention/test_rocm_aiter_mla_decode_metadata.py
- tests/kernels/attention/test_rocm_aiter_mla_fp8_support.py
@@ -430,7 +416,6 @@ steps:
- pytest -v -s tests/kernels/attention/test_rocm_aiter_mla_decode.py
- pytest -v -s tests/kernels/attention/test_rocm_aiter_mla_fp8_support.py
- pytest -v -s tests/kernels/attention/test_rocm_aiter_mla_op_registration.py
- pytest -v -s tests/kernels/attention/test_rocm_triton_attn_dsv4.py::test_dsv41_adaptive_mla_swa_metadata_graph_replay
- label: ":nvidia: (L4) Quantization Kernels Shard %N"
key: kernels-quantization-test
@@ -477,7 +462,7 @@ steps:
num_devices: 1
timeout_in_minutes: 50
source_file_dependencies:
- csrc/quantization/w8a8/cutlass/moe/
- csrc/quantization/cutlass_w8a8/moe/
- csrc/moe/
- tests/kernels/moe
- vllm/model_executor/layers/fused_moe/
@@ -495,7 +480,7 @@ steps:
device: mi300_1
timeout_in_minutes: 55
source_file_dependencies:
- csrc/quantization/w8a8/cutlass/moe/
- csrc/quantization/cutlass_w8a8/moe/
- csrc/moe/
- tests/kernels/moe
- vllm/model_executor/layers/fused_moe/
@@ -514,22 +499,18 @@ steps:
- label: ":nvidia: (H200 MIG 35GB) Mamba Kernels"
device: h200_35gb
key: kernels-mamba-test
automatic_shard: true
timeout_in_minutes: 60
source_file_dependencies:
- csrc/mamba/
- tests/kernels/mamba
- vllm/model_executor/layers/mamba/checkpoint.py
- vllm/model_executor/layers/mamba/gdn
- vllm/model_executor/layers/mamba/ops
- vllm/v1/attention/backends/gdn_attn.py
commands:
- pytest -v -s kernels/mamba
mirror:
amd:
label: ":amd: (MI250) Mamba Kernels"
dind: false
automatic_shard: true
device: mi250_1
timeout_in_minutes: 55
depends_on:
@@ -545,6 +526,7 @@ steps:
num_devices: 1
source_file_dependencies:
- cmake/external_projects/deepgemm.cmake
- tools/install_deepgemm.sh
- tools/build_deepgemm_C.py
- tools/setup_deepgemm_pythons.sh
- tools/check_wheel_deepgemm.py
@@ -564,21 +546,6 @@ steps:
- pytest -v -s kernels/attention/test_deepgemm_attention.py
- pytest -v -s quantization/test_cutlass_w4a16.py
- label: ":nvidia: (B200) MiniMax-M3 Attention Kernels"
key: kernels-minimax-m3-attention-b200
timeout_in_minutes: 30
working_dir: "/vllm-workspace/"
device: b200-k8s
num_devices: 1
source_file_dependencies:
- vllm/models/minimax_m3/
- vllm/config/cache.py
- vllm/v1/kv_cache_interface.py
- tests/v1/attention/utils.py
- tests/kernels/attention/test_minimax_m3.py
commands:
- pytest -v -s tests/kernels/attention/test_minimax_m3.py
- label: ":nvidia: (B200) Kernels Shard %N"
key: kernels-b200
timeout_in_minutes: 30
@@ -588,10 +555,11 @@ steps:
# optional: true
source_file_dependencies:
- csrc/quantization/fp4/
- csrc/nvfp4_kv_cache_kernels.cu
- csrc/quantization/w8a8/cutlass/
- csrc/libtorch_stable/nvfp4_kv_cache_kernels.cu
- csrc/libtorch_stable/quantization/fp4/
- csrc/libtorch_stable/quantization/w8a8/cutlass/
- csrc/attention/mla/
- csrc/quantization/w8a8/cutlass/moe/
- csrc/quantization/cutlass_w8a8/moe/
- vllm/model_executor/layers/fusion/
- vllm/model_executor/layers/fused_moe/experts/cutlass_moe.py
- vllm/model_executor/layers/fused_moe/experts/flashinfer_cutlass_moe.py
@@ -605,6 +573,9 @@ steps:
- vllm/v1/attention/backends/mla/flashinfer_mla.py
- vllm/v1/attention/selector.py
- vllm/platforms/cuda.py
- vllm/model_executor/kernels/linear/cute_dsl/ll_bf16.py
- vllm/model_executor/kernels/linear/cute_dsl/_ll_bf16_dotprod.py
- vllm/model_executor/kernels/linear/cute_dsl/_ll_bf16_splitk.py
- vllm/cute_utils/
- vllm/model_executor/layers/mamba/ops/gdn_chunk_cutedsl/
- vllm/model_executor/layers/fused_moe/router/bf16x3_router_gemm_cutedsl.py
@@ -614,9 +585,9 @@ steps:
- vllm/config/attention.py
- vllm/v1/attention/backends/registry.py
- vllm/_custom_ops.py
- csrc/fused_minimax_m3_qknorm_rope_kv_insert_kernel.cu
- csrc/ops.h
- csrc/torch_bindings.cpp
- csrc/libtorch_stable/fused_minimax_m3_qknorm_rope_kv_insert_kernel.cu
- csrc/libtorch_stable/ops.h
- csrc/libtorch_stable/torch_bindings.cpp
- tests/kernels/attention/test_flashinfer_mla_decode.py
- tests/kernels/attention/test_minimax_m3_msa_cutlass_sparse_decode.py
- tests/kernels/attention/test_cache.py
@@ -624,6 +595,8 @@ steps:
- tests/kernels/test_fused_minimax_m3_qknorm_rope_kv_insert.py
- tests/kernels/mamba/test_gdn_prefill_cutedsl.py
- tests/kernels/test_bf16x3_router_gemm_cutedsl.py
- tests/kernels/attention/test_minimax_m3.py
- tests/kernels/test_ll_bf16_gemm.py
- tests/kernels/test_top_k_per_row.py
- tests/kernels/quantization/test_silu_mul_nvfp4_quant.py
- tests/kernels/quantization/test_block_fp8.py
@@ -644,6 +617,7 @@ steps:
pytest -v -s tests/kernels/moe/test_nvfp4_moe.py
pytest -v -s tests/kernels/moe/test_ocp_mx_moe.py
pytest -v -s tests/kernels/test_bf16x3_router_gemm_cutedsl.py
pytest -v -s tests/kernels/test_ll_bf16_gemm.py
;;
1)
python3 examples/basic/offline_inference/chat.py
@@ -667,6 +641,7 @@ steps:
pytest -v -s tests/kernels/moe/test_flashinfer.py
pytest -v -s tests/kernels/moe/test_trtllm_nvfp4_moe.py
pytest -v -s tests/kernels/mamba/test_gdn_prefill_cutedsl.py
pytest -v -s tests/kernels/attention/test_minimax_m3.py
;;
*)
echo "unexpected BUILDKITE_PARALLEL_JOB=$$BUILDKITE_PARALLEL_JOB" >&2
@@ -700,46 +675,6 @@ steps:
- amd-smi
- pytest -v -s kernels/quantization/test_nvfp4_emulation.py -k test_nvfp4_dense_emulation --shard-id=$$BUILDKITE_PARALLEL_JOB --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT
- label: ":nvidia: (B200) Gate Linear CuTeDSL Kernels"
key: kernels-gate-linear-cutedsl-b200
timeout_in_minutes: 15
working_dir: "/vllm-workspace/"
device: b200-k8s
source_file_dependencies:
- vllm/cute_utils/
- vllm/platforms/cuda.py
- vllm/model_executor/layers/fused_moe/router/gate_linear.py
- vllm/model_executor/kernels/linear/cute_dsl/_ll_router_common.py
- vllm/model_executor/kernels/linear/cute_dsl/ll_bf16.py
- vllm/model_executor/kernels/linear/cute_dsl/_ll_bf16_dotprod.py
- vllm/model_executor/kernels/linear/cute_dsl/_ll_bf16_splitk.py
- tests/kernels/test_ll_bf16_gemm.py
- vllm/model_executor/kernels/linear/cute_dsl/ll_fp32w.py
- vllm/model_executor/kernels/linear/cute_dsl/_ll_fp32w_dotprod.py
- tests/kernels/test_ll_fp32w_gemm.py
commands:
- nvidia-smi
- pytest -v -s tests/kernels/test_ll_bf16_gemm.py
- pytest -v -s tests/kernels/test_ll_fp32w_gemm.py
mirror:
amd:
label: ":amd: (MI355) Gate Linear Router Kernels"
dind: false
device: mi355_1
timeout_in_minutes: 20
depends_on:
- image-build-amd
source_file_dependencies:
- vllm/model_executor/layers/fused_moe/router/gate_linear.py
- vllm/model_executor/layers/fused_moe/router/rocm_fp32_router_gemm.py
- vllm/model_executor/layers/fused_moe/router/bf16x3_router_gemm_rocm.py
- vllm/platforms/rocm.py
- tests/kernels/test_rocm_fp32_router_gemm.py
- tests/kernels/test_gate_linear_rocm_dispatch.py
- tests/kernels/test_bf16x3_router_gemm_rocm.py
commands:
- pytest -v -s tests/kernels/test_rocm_fp32_router_gemm.py tests/kernels/test_gate_linear_rocm_dispatch.py tests/kernels/test_bf16x3_router_gemm_rocm.py
- label: ":nvidia: (DGX) Spark b12x Linear Kernels Nightly"
key: b12x-linear-kernels-dgx-spark-nightly
timeout_in_minutes: 60
@@ -772,7 +707,7 @@ steps:
- tests/quantization/reference_mxfp4.py
- tests/quantization/test_auto_round.py
commands:
- uv pip install --system b12x==1.5.1
- uv pip install --system b12x==1.3.0
- pytest -v -s model_executor/kernels/test_b12x_linear.py
model_executor/test_b12x_warmup.py
kernels/quantization/test_block_fp8.py
@@ -899,7 +834,7 @@ steps:
device: h100
num_devices: 2
source_file_dependencies:
- csrc/quantization/w8a8/cutlass/moe/
- csrc/quantization/cutlass_w8a8/moe/
- csrc/moe/
- tests/kernels/moe
- vllm/model_executor/layers/fused_moe/
@@ -938,11 +873,11 @@ steps:
- label: ":nvidia: (B200) FusedMoE Layer Kernels"
key: kernels-fusedmoe-layer-test-2-b200s
timeout_in_minutes: 45
timeout_in_minutes: 90
device: b200-k8s
num_devices: 2
source_file_dependencies:
- csrc/quantization/w8a8/cutlass/moe/
- csrc/quantization/cutlass_w8a8/moe/
- csrc/moe/
- tests/kernels/moe
- vllm/model_executor/layers/fused_moe/
@@ -1055,7 +990,7 @@ steps:
source_file_dependencies:
- csrc/quantization/fp4/
- csrc/attention/mla/
- csrc/quantization/w8a8/cutlass/moe/
- csrc/quantization/cutlass_w8a8/moe/
- vllm/model_executor/layers/fused_moe/cutlass_moe.py
- vllm/v1/attention/backends/triton_attn.py
- vllm/v1/attention/backends/rocm_attn.py
+23 -34
View File
@@ -5,7 +5,6 @@ steps:
- label: ":nvidia: (H200 MIG 35GB) LM Eval Small Models"
device: h200_35gb
key: lm-eval-small-models
automatic_shard: true
timeout_in_minutes: 45
source_file_dependencies:
- csrc/
@@ -15,9 +14,9 @@ steps:
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-small.txt
mirror:
amd:
label: ":amd: (MI355) LM Eval Small Models"
label: ":amd: (MI300) LM Eval Small Models"
dind: false
device: mi355_dpx
device: mi300_1
timeout_in_minutes: 45
depends_on:
- image-build-amd
@@ -105,7 +104,6 @@ steps:
- label: ":nvidia: (H100) LM Eval Large Models"
key: lm-eval-large-models-4xh100
automatic_shard: true
device: h100
optional: true
num_devices: 4
@@ -140,7 +138,6 @@ steps:
- label: ":nvidia: (B200) LM Eval Small Models"
key: lm-eval-small-models-1xb200
automatic_shard: true
timeout_in_minutes: 50
device: b200-k8s
optional: true
@@ -211,6 +208,7 @@ steps:
- tests/evals/gsm8k/
- tests/utils.py
- vllm/distributed/
- vllm/transformers_utils/configs/diffusion_gemma.py
- vllm/v1/attention/ops/
commands:
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-mi3xx-fp8-and-mixed.txt
@@ -232,13 +230,12 @@ steps:
- vllm/v1/attention/backends/mla/
- vllm/model_executor/layers/attention/sparse_mla_attention.py
- vllm/v1/worker/gpu/
- csrc/hisparse_kernels.cu
- csrc/libtorch_stable/hisparse_kernels.cu
commands:
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-hisparse.txt
- label: ":nvidia: (B200) LM Eval PCP"
key: lm-eval-pcp-4xb200
automatic_shard: true
timeout_in_minutes: 360
device: b200-k8s
num_devices: 4
@@ -273,9 +270,6 @@ steps:
source_file_dependencies:
- .buildkite/test_areas/lm_eval.yaml
- tests/evals/gsm8k/
- tests/watermarking/test_compatibility.py
- tests/watermarking/test_engine_compatibility.py
- tests/watermarking/test_parallel_compatibility.py
- vllm/config/speculative.py
- vllm/config/vllm.py
- vllm/config/watermarking.py
@@ -288,7 +282,6 @@ steps:
autorun_on_main: true
commands:
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-watermark-dspark-tp2.txt
- pytest -s -v watermarking/test_compatibility.py watermarking/test_engine_compatibility.py watermarking/test_parallel_compatibility.py
mirror:
amd:
label: ":amd: (MI355) LM Eval Watermark Feature Combination"
@@ -325,11 +318,9 @@ steps:
optional: true
source_file_dependencies:
- tests/evals/gsm8k/configs/DeepSeek-V4-Flash-DSpark-confidence-TEP4.yaml
- tests/evals/gsm8k/configs/DeepSeek-V4.1-Flash-NVFP4-DSpark-TP4.yaml
- tests/evals/gsm8k/configs/Kimi-K3-pruned75-DSpark-TP4.yaml
- tests/evals/gsm8k/configs/models-spec-decode.txt
- vllm/models/deepseek_v4/
- vllm/models/deepseek_v41/
- vllm/models/kimi_k3/
- vllm/model_executor/models/qwen3_dspark.py
- vllm/v1/spec_decode/metrics.py
@@ -374,7 +365,6 @@ steps:
- vllm/tokenizers/
- vllm/v1/attention/
- vllm/v1/core/kv_cache_utils.py
- vllm/v1/core/kv_cache_planning.py
- vllm/v1/spec_decode/metrics.py
- vllm/v1/worker/
- vllm/_aiter_ops.py
@@ -387,7 +377,6 @@ steps:
- label: ":nvidia: (B200) LM Eval Large Models EP"
key: lm-eval-large-models-ep-2xb200
automatic_shard: true
timeout_in_minutes: 60
device: b200-k8s
optional: true
@@ -422,6 +411,8 @@ steps:
- vllm/model_executor/models/qwen3_next_mtp.py
- vllm/model_executor/models/nemotron_h.py
- vllm/model_executor/models/nemotron_h_mtp.py
- vllm/transformers_utils/configs/qwen3_next.py
- vllm/transformers_utils/configs/nemotron_h.py
- vllm/model_executor/model_loader/
- vllm/model_executor/layers/quantization/
- vllm/model_executor/layers/fused_moe/
@@ -448,6 +439,8 @@ steps:
source_file_dependencies:
- vllm/model_executor/models/qwen3_5.py
- vllm/model_executor/models/qwen3_5_mtp.py
- vllm/transformers_utils/configs/qwen3_5.py
- vllm/transformers_utils/configs/qwen3_5_moe.py
- vllm/model_executor/models/qwen3_next.py
- vllm/model_executor/models/qwen3_next_mtp.py
- vllm/third_party/flash_linear_attention/ops/
@@ -464,6 +457,8 @@ steps:
source_file_dependencies:
- vllm/model_executor/models/qwen3_5.py
- vllm/model_executor/models/qwen3_5_mtp.py
- vllm/transformers_utils/configs/qwen3_5.py
- vllm/transformers_utils/configs/qwen3_5_moe.py
- vllm/model_executor/models/qwen2.py
- vllm/model_executor/models/qwen3.py
- vllm/model_executor/models/qwen3_next.py
@@ -521,7 +516,6 @@ steps:
- vllm/v1/spec_decode/
- vllm/v1/worker/gpu/spec_decode/
- vllm/v1/core/kv_cache_utils.py
- vllm/v1/core/kv_cache_planning.py
- vllm/model_executor/layers/layernorm.py
- csrc/
- vllm/_aiter_ops.py
@@ -565,7 +559,6 @@ steps:
- vllm/v1/spec_decode/
- vllm/v1/worker/gpu/spec_decode/
- vllm/v1/core/kv_cache_utils.py
- vllm/v1/core/kv_cache_planning.py
- vllm/model_executor/layers/layernorm.py
- csrc/
- vllm/_aiter_ops.py
@@ -609,7 +602,6 @@ steps:
- vllm/v1/spec_decode/
- vllm/v1/worker/gpu/spec_decode/
- vllm/v1/core/kv_cache_utils.py
- vllm/v1/core/kv_cache_planning.py
- vllm/model_executor/layers/layernorm.py
- csrc/
- vllm/_aiter_ops.py
@@ -653,7 +645,6 @@ steps:
- vllm/v1/spec_decode/
- vllm/v1/worker/gpu/spec_decode/
- vllm/v1/core/kv_cache_utils.py
- vllm/v1/core/kv_cache_planning.py
- vllm/model_executor/layers/layernorm.py
- csrc/
- vllm/_aiter_ops.py
@@ -697,7 +688,6 @@ steps:
- vllm/v1/spec_decode/
- vllm/v1/worker/gpu/spec_decode/
- vllm/v1/core/kv_cache_utils.py
- vllm/v1/core/kv_cache_planning.py
- vllm/model_executor/layers/layernorm.py
- csrc/
- vllm/_aiter_ops.py
@@ -707,17 +697,18 @@ steps:
- export VLLM_WORKER_MULTIPROC_METHOD=spawn
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-mi355-nemotron-3-super.txt
- label: ":amd: (MI355) LM Eval Large Models"
- label: ":amd: (MI300) LM Eval Large Models"
key: amd-lm-eval-large-models
dind: false
device: mi355_8
device: mi300_8
optional: true
timeout_in_minutes: 65
depends_on:
- image-build-amd
commands:
- export VLLM_WORKER_MULTIPROC_METHOD=spawn
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-gfx950-large.txt
- export PYTORCH_ROCM_ARCH=gfx942 # Limit Quark compilation to save time
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-mi3xx.txt
source_file_dependencies:
- vllm/_aiter_ops.py
- vllm/platforms/rocm.py
@@ -1194,7 +1185,7 @@ steps:
optional: true
num_devices: 4
source_file_dependencies:
- csrc/gdn/
- csrc/libtorch_stable/gdn/
- tests/evals/qwen4_exp/
- vllm/model_executor/layers/mamba/
- vllm/models/qwen4_exp/
@@ -1212,7 +1203,7 @@ steps:
optional: true
num_devices: 4
source_file_dependencies:
- csrc/gdn/
- csrc/libtorch_stable/gdn/
- tests/evals/qwen4_exp/
- vllm/model_executor/layers/mamba/
- vllm/models/qwen4_exp/
@@ -1233,7 +1224,7 @@ steps:
depends_on:
- image-build-amd
source_file_dependencies:
- csrc/gdn/
- csrc/libtorch_stable/gdn/
- tests/evals/qwen4_exp/
- vllm/model_executor/layers/mamba/
- vllm/models/qwen4_exp/
@@ -1297,9 +1288,9 @@ steps:
- pytest -s -v evals/gsm8k/test_gsm8k_offloading.py -k "nemotron-h-8b or gemma-4-e4b-it"
mirror:
amd:
label: ":amd: (MI355 DPX) LM Eval KV-Offload"
label: ":amd: (MI300) LM Eval KV-Offload"
dind: false
device: mi355_dpx
device: mi300_1
timeout_in_minutes: 55
depends_on:
- image-build-amd
@@ -1344,8 +1335,6 @@ steps:
- label: ":nvidia: (H100) LM Eval KV-Offload Large"
key: kv-offload-large
# Nightly only: 4-GPU H100 pods wait behind smaller jobs on per-PR runs.
optional: true
timeout_in_minutes: 40
device: h100
num_devices: 4
@@ -1359,9 +1348,9 @@ steps:
- pytest -s -v evals/gsm8k/test_gsm8k_offloading.py -k "deepseek-v4-flash"
mirror:
amd:
label: ":amd: (MI355) LM Eval KV-Offload Large"
label: ":amd: (MI300) LM Eval KV-Offload Large"
dind: false
device: mi355_4
device: mi300_4
timeout_in_minutes: 65
depends_on:
- image-build-amd
@@ -1426,10 +1415,10 @@ steps:
# Qwen2-57B has 28 attention heads and cannot run at TP8.
- pytest -s -v test_lm_eval_correctness.py --config-list-file=configs/models-large.txt --tp-size=4
- label: ":amd: (MI355) LM Eval Large Models ROCm Harness"
- label: ":amd: (MI300) LM Eval Large Models ROCm Harness"
key: amd-lm-eval-large-models-rocm-harness
dind: false
device: mi355_8
device: mi300_8
optional: true
timeout_in_minutes: 40
depends_on:
+8 -186
View File
@@ -4,8 +4,7 @@ depends_on:
steps:
- label: ":nvidia: (H200 MIG 18GB) V1 Sample"
key: v1-sample
automatic_shard: true
timeout_in_minutes: 45
timeout_in_minutes: 35
device: h200_18gb
source_file_dependencies: &v1-sample-logits-deps
- vllm/config/
@@ -32,7 +31,6 @@ steps:
amd:
label: ":amd: (MI355 DPX) V1 Sample"
dind: false
automatic_shard: true
device: mi355_dpx
timeout_in_minutes: 70
depends_on:
@@ -40,7 +38,7 @@ steps:
- label: ":nvidia: (H200 MIG 18GB) V1 Logits + Oracle"
key: v1-logits-oracle
timeout_in_minutes: 45
timeout_in_minutes: 35
device: h200_18gb
source_file_dependencies: *v1-sample-logits-deps
commands:
@@ -164,59 +162,6 @@ steps:
depends_on:
- image-build-amd
# The shared KV Connectors steps above exclude vllm/distributed/kv_transfer/ from
# their triggers, and CUDA skips MoRIIO's mori/aiter-gated tests. Run the whole
# MoRIIO suite on ROCm whenever the connector or its tests change.
- label: ":amd: (MI355 DPX) MoRIIO Connector Unit Tests"
key: amd-moriio-connector-unit-mi355
dind: false
device: mi355_dpx
num_devices: 1
timeout_in_minutes: 30
working_dir: /vllm-workspace/tests
depends_on:
- image-build-amd
source_file_dependencies:
- vllm/distributed/kv_transfer/kv_connector/v1/moriio/
- vllm/distributed/kv_transfer/kv_connector/v1/base.py
- vllm/distributed/kv_transfer/kv_connector/v1/ssm_conv_transfer_utils.py
- examples/disaggregated/disaggregated_serving/moriio_toy_proxy_server.py
- tests/v1/kv_connector/unit/test_moriio_
- tests/v1/kv_connector/unit/utils.py
- requirements/kv_connectors_rocm.txt
commands:
- bash /vllm-workspace/.buildkite/scripts/install-kv-connectors.sh
- export VLLM_WORKER_MULTIPROC_METHOD=spawn
- pytest -v -s -rs v1/kv_connector/unit/test_moriio_*.py
# Single-node MoRIIO WRITE P/D on dummy weights: prefill and decode on separate
# GPUs over xGMI, compared exactly against a direct serve.
- label: ":amd: (MI355) MoRIIO WRITE P/D Dummy-Weight E2E Shard %N"
key: amd-moriio-write-dummy-e2e-mi355
dind: false
device: mi355_2
num_devices: 2
# ~3.5-5 min per case; pytest-shard hashes test ids, so with 20 cases the
# largest of 6 shards holds 5 (~22 min on MI355). Every case picks free ports
# for its servers and proxy and its own compile-cache dir.
parallelism: 6
timeout_in_minutes: 45
working_dir: /vllm-workspace/tests
depends_on:
- image-build-amd
source_file_dependencies:
- vllm/distributed/kv_transfer/kv_connector/v1/moriio/
- vllm/distributed/kv_transfer/kv_connector/v1/base.py
- vllm/distributed/kv_transfer/kv_connector/v1/ssm_conv_transfer_utils.py
- vllm/model_executor/model_loader/dummy_loader.py
- examples/disaggregated/disaggregated_serving/moriio_toy_proxy_server.py
- tests/v1/kv_connector/moriio_integration/
- requirements/kv_connectors_rocm.txt
commands:
- bash /vllm-workspace/.buildkite/scripts/install-kv-connectors.sh
- export VLLM_WORKER_MULTIPROC_METHOD=spawn
- pytest -v -s -rs v1/kv_connector/moriio_integration/test_write_dummy_e2e_rocm.py --shard-id=$$BUILDKITE_PARALLEL_JOB --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT
- label: ":nvidia: (H200 MIG 35GB) V1 Metrics + LM Eval"
device: h200_35gb
key: v1-metrics-lmeval
@@ -267,14 +212,12 @@ steps:
- vllm/v1/
- tests/v1
- tests/watermarking
- examples/disaggregated/mooncake_connector/mooncake_connector_proxy.py
device: cpu-small
commands:
# split the test to avoid interference
- pytest -v -s -m 'cpu_test' v1/core
- pytest -v -s v1/structured_output
- pytest -v -s v1/test_serial_utils.py
- pytest -v -s v1/worker/test_cpu_worker_kv_budget.py
- pytest -v -s v1/test_kv_cache_spec_registry.py
- pytest -v -s v1/cudagraph/test_cudagraph_manager.py
- pytest -v -s -m 'cpu_test' v1/kv_connector/unit
@@ -353,6 +296,7 @@ steps:
- vllm/model_executor/models/qwen3_next.py
- vllm/model_executor/models/registry.py
- vllm/transformers_utils/configs/extract_hidden_states.py
- vllm/transformers_utils/configs/qwen3_5.py
- vllm/v1/attention/backends/
- vllm/v1/attention/selector.py
- vllm/v1/kv_cache_interface.py
@@ -502,17 +446,6 @@ steps:
- setup.py
- vllm/platforms/rocm.py
- label: ":nvidia: (H200 MIG 18GB) Python 3.14 Smoke Nightly"
key: python-314-smoke-nightly
depends_on: ~
optional: true
timeout_in_minutes: 30
device: h200_18gb
source_file_dependencies:
- tests/standalone_tests/python_only_compile.sh
commands:
- PYTHON_ONLY_COMPILE_PYTHON=3.14 bash standalone_tests/python_only_compile.sh
- label: ":nvidia: (H200 MIG 35GB) Async Engine, Inputs, Utils, Worker"
device: h200_35gb
key: async-engine-inputs-utils-worker
@@ -623,8 +556,6 @@ steps:
- tests/multimodal
- tests/renderers
- tests/standalone_tests/lazy_imports.py
- tests/standalone_tests/test_precompiled_wheel.py
- setup.py
- tests/reasoning
- tools/pre_commit/check_test_tethering.py
- tools/pre_commit/test_tethering_allowlist.txt
@@ -641,7 +572,6 @@ steps:
commands:
- pip install cmake # test_cmake_utils.py shells out to cmake; the CPU image has none
- python3 standalone_tests/lazy_imports.py
- python3 standalone_tests/test_precompiled_wheel.py
- pytest -v -s device_allocator
- pytest -v -s test_uvicorn_logging.py
- pytest -v -s test_attention_backend_registry.py
@@ -711,45 +641,21 @@ steps:
source_file_dependencies:
- vllm/v1/attention
- vllm/model_executor/layers
- vllm/model_executor/kernels/linear
- vllm/model_executor/determinism
- tests/v1/determinism/
commands:
- export VLLM_WORKER_MULTIPROC_METHOD=spawn
- pip install pytest-timeout pytest-forked
- pytest -v -s v1/determinism/test_batch_invariance.py
- VLLM_TEST_MODEL=deepseek-ai/DeepSeek-V2-Lite-Chat pytest -v -s v1/determinism/test_batch_invariance.py::test_v1_generation_is_deterministic_across_batch_sizes_with_needle -k TRITON_MLA
mirror:
amd:
label: ":amd: (MI250) Batch Invariance"
dind: false
device: mi250_1
timeout_in_minutes: 60
env:
VLLM_GPU_MEMORY_UTILIZATION: "0.8"
VLLM_TEST_CLEAN_GPU_MEMORY: "1"
depends_on:
- image-build-amd
source_file_dependencies:
- vllm/v1/attention
- vllm/model_executor/layers
- vllm/model_executor/kernels/linear
- vllm/model_executor/determinism
- tests/v1/determinism/
- vllm/_aiter_ops.py
- vllm/platforms/rocm.py
- label: ":nvidia: (H100) Batch Invariance"
key: batch-invariance-h100
automatic_shard: true
timeout_in_minutes: 60
device: h100
source_file_dependencies:
- vllm/v1/attention
- vllm/model_executor/layers
- vllm/lora/ops/triton_ops
- vllm/model_executor/kernels/linear
- vllm/model_executor/determinism
- tests/v1/determinism/
commands:
- export VLLM_WORKER_MULTIPROC_METHOD=spawn
@@ -761,19 +667,15 @@ steps:
- VLLM_TEST_MODEL=Qwen/Qwen3-30B-A3B-Thinking-2507-FP8 pytest -v -s v1/determinism/test_batch_invariance.py::test_v1_generation_is_deterministic_across_batch_sizes_with_needle -k FLASH_ATTN
mirror:
amd:
label: ":amd: (MI300) Batch Invariance"
label: ":amd: (MI250) Batch Invariance"
dind: false
automatic_shard: true
device: mi300_1
timeout_in_minutes: 60
device: mi250_1
timeout_in_minutes: 35
depends_on:
- image-build-amd
source_file_dependencies:
- vllm/v1/attention
- vllm/model_executor/layers
- vllm/lora/ops/triton_ops
- vllm/model_executor/kernels/linear
- vllm/model_executor/determinism
- tests/v1/determinism/
- vllm/_aiter_ops.py
- vllm/platforms/rocm.py
@@ -782,23 +684,15 @@ steps:
- pip install pytest-timeout pytest-forked
- pytest -v -s v1/determinism/test_batch_invariance.py
- pytest -v -s v1/determinism/test_rms_norm_batch_invariant.py
- pytest -v -s v1/determinism/test_lora_shrink_batch_invariant.py
- pytest -v -s v1/determinism/test_cutlass_batch_invariance.py
- pytest -v -s v1/determinism/test_matmul_batch_invariant.py
- VLLM_TEST_MODEL=deepseek-ai/DeepSeek-V2-Lite-Chat pytest -v -s v1/determinism/test_batch_invariance.py::test_v1_generation_is_deterministic_across_batch_sizes_with_needle -k TRITON_MLA
- VLLM_TEST_MODEL=Qwen/Qwen3-30B-A3B-Thinking-2507-FP8 pytest -v -s v1/determinism/test_batch_invariance.py::test_v1_generation_is_deterministic_across_batch_sizes_with_needle -k TRITON_ATTN
- label: ":nvidia: (B200) Batch Invariance"
key: batch-invariance-b200
automatic_shard: true
timeout_in_minutes: 45
device: b200-k8s
source_file_dependencies:
- vllm/v1/attention
- vllm/model_executor/layers
- vllm/lora/ops/triton_ops
- vllm/model_executor/kernels/linear
- vllm/model_executor/determinism
- tests/v1/determinism/
commands:
- export VLLM_WORKER_MULTIPROC_METHOD=spawn
@@ -813,80 +707,6 @@ steps:
- pytest -v -s v1/determinism/test_matmul_batch_invariant.py
- pytest -v -s v1/determinism/test_cutlass_batch_invariance.py
- pytest -v -s v1/determinism/test_online_batch_invariance.py
mirror:
amd:
label: ":amd: (MI355) Batch Invariance"
dind: false
device: mi355_1
timeout_in_minutes: 60
depends_on:
- image-build-amd
source_file_dependencies:
- vllm/v1/attention
- vllm/model_executor/layers
- vllm/lora/ops/triton_ops
- vllm/model_executor/kernels/linear
- vllm/model_executor/determinism
- tests/v1/determinism/
- vllm/_aiter_ops.py
- vllm/platforms/rocm.py
commands:
- export VLLM_WORKER_MULTIPROC_METHOD=spawn
- pip install pytest-timeout pytest-forked
- pytest -v -s v1/determinism/test_batch_invariance.py
- pytest -v -s v1/determinism/test_rms_norm_batch_invariant.py
- pytest -v -s v1/determinism/test_lora_shrink_batch_invariant.py
- pytest -v -s v1/determinism/test_batch_invariance_vlm.py
- VLLM_TEST_MODEL=deepseek-ai/DeepSeek-V2-Lite-Chat pytest -v -s v1/determinism/test_batch_invariance.py::test_v1_generation_is_deterministic_across_batch_sizes_with_needle -k TRITON_MLA
- VLLM_TEST_MODEL=Qwen/Qwen3-30B-A3B-Thinking-2507-FP8 pytest -v -s v1/determinism/test_batch_invariance.py::test_v1_generation_is_deterministic_across_batch_sizes_with_needle -k TRITON_ATTN
- pytest -v -s v1/determinism/test_nvfp4_batch_invariant.py
- pytest -v -s v1/determinism/test_matmul_batch_invariant.py
- pytest -v -s v1/determinism/test_cutlass_batch_invariance.py
- pytest -v -s v1/determinism/test_online_batch_invariance.py
- pytest -v -s v1/determinism/test_mx_linear_batch_invariant.py
- label: ":nvidia: (H100) Batch Invariance (Distributed)"
key: batch-invariance-distributed-4xh100
automatic_shard: true
timeout_in_minutes: 45
device: h100
optional: true
num_devices: 4
source_file_dependencies:
- vllm/distributed/device_communicators/
- vllm/model_executor/determinism
- vllm/model_executor/layers
- vllm/v1/attention
- tests/v1/determinism/
commands:
- export VLLM_WORKER_MULTIPROC_METHOD=spawn
- export VLLM_TEST_CLEAN_GPU_MEMORY=1
- VLLM_TEST_TP_SIZE=4 pytest -v -s v1/determinism/test_batch_invariance.py -k "bs1_vs_bsN or without_batch_invariance_should_fail"
- VLLM_TEST_TP_SIZE=4 VLLM_TEST_DISABLE_CUSTOM_AR=1 pytest -v -s v1/determinism/test_batch_invariance.py -k bs1_vs_bsN
- VLLM_TP_SIZE=4 VLLM_NEEDLE_TRIALS=2 pytest -v -s v1/determinism/test_batch_invariance.py::test_v1_generation_is_deterministic_across_batch_sizes_with_needle -k FLASH_ATTN
- VLLM_TEST_MODEL=allenai/OLMoE-1B-7B-0924 VLLM_TEST_SERVER_ARGS="--data-parallel-size 4 --enable-expert-parallel --max-model-len 4096" pytest -v -s v1/determinism/test_online_batch_invariance.py
mirror:
amd:
label: ":amd: (MI300) Batch Invariance (Distributed)"
dind: false
device: mi300_4
timeout_in_minutes: 45
depends_on:
- image-build-amd
source_file_dependencies:
- vllm/distributed/device_communicators/
- vllm/model_executor/determinism
- vllm/model_executor/layers
- vllm/v1/attention
- tests/v1/determinism/
- vllm/platforms/rocm.py
commands:
- export VLLM_WORKER_MULTIPROC_METHOD=spawn
- export VLLM_TEST_CLEAN_GPU_MEMORY=1
- VLLM_TEST_TP_SIZE=4 pytest -v -s v1/determinism/test_batch_invariance.py -k "bs1_vs_bsN or without_batch_invariance_should_fail"
- VLLM_TEST_TP_SIZE=4 VLLM_TEST_DISABLE_CUSTOM_AR=1 pytest -v -s v1/determinism/test_batch_invariance.py -k bs1_vs_bsN
- VLLM_TP_SIZE=4 VLLM_NEEDLE_TRIALS=2 pytest -v -s v1/determinism/test_batch_invariance.py::test_v1_generation_is_deterministic_across_batch_sizes_with_needle -k TRITON_ATTN
- VLLM_TEST_MODEL=allenai/OLMoE-1B-7B-0924 VLLM_TEST_SERVER_ARGS="--data-parallel-size 4 --enable-expert-parallel --max-model-len 4096" pytest -v -s v1/determinism/test_online_batch_invariance.py
- label: ":nvidia: (H200 MIG 35GB) Acceptance Length (Large Models)"
device: h200_35gb
@@ -898,6 +718,7 @@ steps:
working_dir: "/vllm-workspace/tests"
source_file_dependencies:
- vllm/v1/spec_decode/
- vllm/model_executor/models/mlp_speculator.py
- tests/v1/spec_decode/test_acceptance_length.py
commands:
- export VLLM_ALLOW_INSECURE_SERIALIZATION=1
@@ -913,5 +734,6 @@ steps:
working_dir: /vllm-workspace/tests
source_file_dependencies:
- vllm/v1/spec_decode/
- vllm/model_executor/models/mlp_speculator.py
- tests/v1/spec_decode/test_acceptance_length.py
- vllm/platforms/rocm.py
@@ -5,7 +5,6 @@ steps:
- label: ":nvidia: (H200 MIG 35GB) Model Executor"
device: h200_35gb
key: model-executor
automatic_shard: true
timeout_in_minutes: 60
source_file_dependencies:
- vllm/engine/arg_utils.py
+11 -56
View File
@@ -58,7 +58,6 @@ steps:
- label: ":nvidia: (H200 MIG 35GB) Basic Models (Other)"
device: h200_35gb
key: basic-models-tests-other
automatic_shard: true
timeout_in_minutes: 38
source_file_dependencies:
- vllm/
@@ -69,12 +68,8 @@ steps:
- tests/models/test_deepseek_v4_vl_rocm.py
- tests/models/test_deepseek_v41_decoder_replay_layers.py
- tests/models/test_deepseek_v41_replay_batch.py
- tests/models/test_deepseek_v41_pcp.py
- tests/models/transformers/test_layer_registry_e2e.py
- tests/models/test_deepseek_v41_replay_start.py
- tests/models/test_deepseek_v4_vl_stream_weights.py
commands:
- pytest -v -s models/test_terratorch.py models/transformers/test_backend.py models/test_registry.py models/test_deepseek_v4_vl_rocm.py models/transformers/test_layer_registry_e2e.py models/test_deepseek_v41_replay_start.py models/test_deepseek_v41_decoder_replay_layers.py models/test_deepseek_v41_replay_batch.py models/test_deepseek_v41_pcp.py models/test_deepseek_v4_vl_stream_weights.py
- pytest -v -s models/test_terratorch.py models/transformers/test_backend.py models/test_registry.py models/test_deepseek_v4_vl_rocm.py models/test_deepseek_v41_decoder_replay_layers.py models/test_deepseek_v41_replay_batch.py
mirror:
amd:
label: ":amd: (MI355) Basic Models (Other)"
@@ -84,21 +79,10 @@ steps:
depends_on:
- image-build-amd
source_file_dependencies:
- vllm/models/deepseek_v4/
- vllm/models/deepseek_v41/
- tests/models/test_deepseek_v4_vl_rocm.py
- tests/models/test_hyv4_rocm.py
- tests/models/transformers/test_layer_registry_e2e.py
- tests/models/test_deepseek_v41_replay_start.py
- tests/models/test_deepseek_v41_decoder_replay_layers.py
- tests/models/test_deepseek_v41_replay_batch.py
- tests/models/test_deepseek_v41_amd_replay_forward.py
- tests/models/test_deepseek_v4_dspark_rocm.py
- tests/models/test_deepseek_v4_rocm_wo_a.py
- tests/models/test_deepseek_v4_rocm_compressor_gemm_fusion.py
- tests/models/test_dsv41_mono_decode.py
commands:
- pytest -v -s models/test_terratorch.py models/transformers/test_backend.py models/test_registry.py models/test_deepseek_v4_vl_rocm.py models/transformers/test_layer_registry_e2e.py models/test_deepseek_v41_replay_start.py models/test_deepseek_v41_decoder_replay_layers.py models/test_deepseek_v41_replay_batch.py models/test_deepseek_v41_amd_replay_forward.py models/test_deepseek_v4_dspark_rocm.py models/test_deepseek_v4_rocm_wo_a.py models/test_deepseek_v4_rocm_compressor_gemm_fusion.py models/test_dsv41_mono_decode.py
- pytest -v -s models/test_terratorch.py models/transformers/test_backend.py models/test_registry.py models/test_deepseek_v4_vl_rocm.py
- VLLM_ROCM_USE_AITER=1 pytest -v -s models/test_hyv4_rocm.py -m 'not distributed'
- label: ":nvidia: (B200) Inkling"
@@ -137,7 +121,7 @@ steps:
device: b200-k8s
source_file_dependencies:
- vllm/models/kimi_k3/
- csrc/kimi_k3/
- csrc/libtorch_stable/kimi_k3/
- tests/models/kimi_k3/
- tests/kernels/attention/test_kimi_k3_mla_fused_epilogue.py
commands:
@@ -161,11 +145,8 @@ steps:
# cuda.bindings.driver), so it is deliberately left out here.
- pytest -v -s models/kimi_k3
- label: ":nvidia: (4xB200) Kimi K3 Prefix Cache"
- label: ":nvidia: (4xB200) Kimi K3 Prefix Cache Shard %N"
key: kimi-k3-prefix-cache-4xb200
# Nightly only: 4-GPU B200 pods wait hours behind smaller jobs on per-PR runs.
optional: true
automatic_shard: true
timeout_in_minutes: 75
device: b200-k8s
num_devices: 4
@@ -174,8 +155,6 @@ steps:
- vllm/v1/core/
- vllm/distributed/kv_transfer/kv_connector/v1/nixl/
- vllm/distributed/kv_transfer/kv_connector/v1/simple_cpu_offload_connector.py
- vllm/distributed/kv_transfer/kv_connector/v1/mooncake/store/
- vllm/distributed/mooncake_store.py
- vllm/v1/simple_kv_offload/
- vllm/v1/spec_decode/
- vllm/model_executor/layers/mamba/
@@ -183,7 +162,8 @@ steps:
- .buildkite/scripts/install-kv-connectors.sh
commands:
- bash /vllm-workspace/.buildkite/scripts/install-kv-connectors.sh
- pytest -v -s models/kimi_k3/test_prefix_cache.py
- pytest -v -s models/kimi_k3/test_prefix_cache.py --shard-id=$$BUILDKITE_PARALLEL_JOB --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT
parallelism: 2
- label: ":nvidia: (H200) GLM5Next"
key: glm5next-unit-tests
@@ -212,12 +192,12 @@ steps:
device: h200_35gb
source_file_dependencies:
- vllm/models/qwen4_exp/
- vllm/transformers_utils/configs/qwen4_exp.py
- vllm/config/compilation.py
- vllm/v1/attention/backends/short_conv_attn.py
- tests/models/qwen4_exp/
commands:
- pytest -v -s models/qwen4_exp/test_hc_ops.py models/qwen4_exp/test_qsa_prepare.py models/qwen4_exp/test_qsa_reference.py
- pytest -v -s models/qwen4_exp/test_ple.py
mirror:
amd:
label: ":amd: (MI355 DPX) Qwen4 Exp"
@@ -226,7 +206,7 @@ steps:
depends_on:
- image-build-amd
commands:
- pytest -v -s models/qwen4_exp/test_qsa_amd.py models/qwen4_exp/test_qsa_prepare_amd.py
- pytest -v -s models/qwen4_exp/test_qsa_amd.py
# PLE and QSA import conflicting HC registrations in a shared process.
- pytest -v -s models/qwen4_exp/test_ple.py
@@ -238,6 +218,7 @@ steps:
device: cpu-small
source_file_dependencies:
- vllm/models/qwen4_exp/
- vllm/transformers_utils/configs/qwen4_exp.py
- vllm/config/compilation.py
- vllm/v1/attention/backends/short_conv_attn.py
- tests/models/qwen4_exp/
@@ -248,7 +229,7 @@ steps:
key: basic-models-test-other-cpu
depends_on:
- image-build-cpu
timeout_in_minutes: 30
timeout_in_minutes: 20
source_file_dependencies:
- vllm/
- "!vllm/distributed/kv_transfer/"
@@ -256,34 +237,8 @@ steps:
- tests/models/test_vision.py
- tests/models/test_adapters.py
- tests/models/test_qwen3_5_mtp_config.py
- tests/models/test_alice_ai.py
- tests/models/test_alice_ai_config.py
- tests/models/transformers/fusers/
- tests/models/transformers/test_dynamic_rope.py
- tests/model_executor/layers/test_activation.py
- tests/models/test_language_model_cache_is_weak.py
- tests/models/deepseek_v32/test_sequence_parallel.py
- tests/models/test_deepseek_v4_fi_moe_ep.py
- tests/models/test_dspark_mla.py
- tests/models/transformers/test_layer_registry.py
device: cpu-small
commands:
- pytest -v -s models/test_utils.py models/test_vision.py models/test_adapters.py models/test_qwen3_5_mtp_config.py models/transformers/fusers/ models/transformers/test_dynamic_rope.py model_executor/layers/test_activation.py models/test_language_model_cache_is_weak.py models/deepseek_v32/test_sequence_parallel.py models/test_deepseek_v4_fi_moe_ep.py models/test_dspark_mla.py models/transformers/test_layer_registry.py models/test_alice_ai.py models/test_alice_ai_config.py
- label: ':amd: (MI355) DeepSeek-V4.1 Mono Decode Numerics'
key: dsv41-mono-decode-numerics-amd
dind: false
device: mi355_4
num_devices: 4
timeout_in_minutes: 40
working_dir: /vllm-workspace/
depends_on:
- image-build-amd
source_file_dependencies:
- docker/Dockerfile.rocm_base
- vllm/models/deepseek_v41/amd/
- tests/models/test_dsv41_mono_numerics.py
- tests/models/test_dsv41_mono_layers.py
commands:
- pytest -v -s tests/models/test_dsv41_mono_numerics.py
- pytest -v -s tests/models/test_dsv41_mono_layers.py
- pytest -v -s models/test_utils.py models/test_vision.py models/test_adapters.py models/test_qwen3_5_mtp_config.py models/transformers/fusers/ model_executor/layers/test_activation.py
@@ -2,39 +2,6 @@ group: Models - Distributed
depends_on:
- image-build
steps:
- label: ":nvidia: (L4) Qwen4 Exp SP"
key: qwen4-exp-sp-tests-4-gpus
timeout_in_minutes: 30
working_dir: "/vllm-workspace/tests"
device: l4
num_devices: 4
source_file_dependencies:
- vllm/config/parallel.py
- vllm/models/common/ops/sequence_parallel.py
- vllm/models/qwen4_exp/
- vllm/model_executor/layers/vocab_parallel_embedding.py
- vllm/model_executor/models/qwen3_next.py
- tests/models/qwen4_exp/test_ple.py
- tests/models/qwen4_exp/test_tpsp.py
commands:
- pytest -v -s models/qwen4_exp/test_tpsp.py
mirror:
amd:
label: ":amd: (MI355) Qwen4 Exp SP"
dind: false
device: mi355_4
depends_on:
- image-build-amd
source_file_dependencies:
- vllm/config/parallel.py
- vllm/models/common/ops/sequence_parallel.py
- vllm/models/qwen4_exp/
- vllm/model_executor/layers/vocab_parallel_embedding.py
- vllm/model_executor/models/qwen3_next.py
- vllm/platforms/rocm.py
- tests/models/qwen4_exp/test_ple.py
- tests/models/qwen4_exp/test_tpsp.py
- label: ":nvidia: (L4) Distributed Models Shard %N"
key: distributed-model-tests-2-gpus
timeout_in_minutes: 35
+9 -13
View File
@@ -184,6 +184,8 @@ steps:
source_file_dependencies:
- vllm/model_executor/models/qwen3_5.py
- vllm/model_executor/models/qwen3_5_mtp.py
- vllm/transformers_utils/configs/qwen3_5.py
- vllm/transformers_utils/configs/qwen3_5_moe.py
- vllm/model_executor/models/qwen2.py
- vllm/model_executor/models/qwen3.py
- vllm/model_executor/models/qwen3_next.py
@@ -199,37 +201,31 @@ steps:
- vllm/platforms/rocm.py
- tests/models/language/generation_ppl_test
- label: ":nvidia: (H200 MIG 35GB) Language Models (Extended Pooling)"
- label: ":nvidia: (H200 MIG 35GB) Language Models (Extended Pooling) Shard %N"
device: h200_35gb
key: language-models-test-extended-pooling
automatic_shard: true
timeout_in_minutes: 120
parallelism: 4
optional: true
source_file_dependencies:
- vllm/
- "!vllm/distributed/kv_transfer/"
- tests/models/language/pooling
commands:
# Per-test timeout: a hung test fails with a stack dump instead of holding
# the job until Buildkite cancels it, which also blocks the full run's
# coverage collection. A whole shard takes 12-21 minutes.
- pytest -v -s models/language/pooling -m 'not core_model' --timeout=900 --timeout-method=thread
- pytest -v -s models/language/pooling -m 'not core_model' --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT --shard-id=$$BUILDKITE_PARALLEL_JOB
mirror:
amd:
parallelism: 4
commands:
- "pytest -v -s models/language/pooling -m 'not core_model' --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT --shard-id=$$BUILDKITE_PARALLEL_JOB"
label: ":amd: (MI355 DPX) Language Models (Extended Pooling) Shard %N"
label: ":amd: (MI300) Language Models (Extended Pooling) Shard %N"
dind: false
device: mi355_dpx
device: mi300_1
timeout_in_minutes: 95
depends_on:
- image-build-amd
- label: ":nvidia: (H200 MIG 35GB) Language Models (MTEB)"
- label: ":nvidia: (H200 MIG 18GB) Language Models (MTEB)"
key: language-models-test-mteb
timeout_in_minutes: 68
device: h200_35gb
device: h200_18gb
optional: true
source_file_dependencies:
- vllm/
+6 -7
View File
@@ -24,7 +24,6 @@ steps:
- label: ":nvidia: (H200 MIG 18GB) Multimodal Models (Standard) 2: qwen3 + gemma"
key: multi-modal-models-standard-2-qwen3-gemma
automatic_shard: true
timeout_in_minutes: 75
device: h200_18gb
source_file_dependencies:
@@ -163,9 +162,9 @@ steps:
- pytest -v -s models/multimodal/test_mapping.py
mirror:
amd:
label: ":amd: (MI355 DPX) Multimodal Models (Extended Generation 1)"
label: ":amd: (MI300) Multimodal Models (Extended Generation 1)"
dind: false
device: mi355_dpx
device: mi300_1
timeout_in_minutes: 100
depends_on:
- image-build-amd
@@ -182,9 +181,9 @@ steps:
- pytest -v -s models/multimodal/generation_ppl_test/
mirror:
amd:
label: ":amd: (MI355 DPX) Multimodal Models (PPL)"
label: ":amd: (MI300) Multimodal Models (PPL)"
dind: false
device: mi355_dpx
device: mi300_1
timeout_in_minutes: 90
depends_on:
- image-build-amd
@@ -240,9 +239,9 @@ steps:
- pytest -v -s models/multimodal/pooling -m 'not core_model'
mirror:
amd:
label: ":amd: (MI355 DPX) Multimodal Models (Extended Pooling)"
label: ":amd: (MI300) Multimodal Models (Extended Pooling)"
dind: false
device: mi355_dpx
device: mi300_1
timeout_in_minutes: 75
depends_on:
- image-build-amd
-2
View File
@@ -123,7 +123,6 @@ steps:
- label: ":nvidia: (L4) PyTorch Compilation Passes"
key: pytorch-compilation-passes-unit-tests
automatic_shard: true
device: l4
num_devices: 1
timeout_in_minutes: 45
@@ -160,7 +159,6 @@ steps:
amd:
label: ':amd: (MI300) PyTorch Compilation Passes'
dind: false
automatic_shard: true
device: mi300_1
timeout_in_minutes: 80
working_dir: /vllm-workspace/tests
+12 -10
View File
@@ -2,27 +2,26 @@ group: Quantization
depends_on:
- image-build
steps:
- label: ":nvidia: (H200 MIG 35GB) Quantization"
- label: ":nvidia: (H200 MIG 35GB) Quantization Shard %N"
device: h200_35gb
key: quantization
automatic_shard: true
timeout_in_minutes: 40
parallelism: 4
env:
VLLM_USE_V2_MODEL_RUNNER: "0"
source_file_dependencies:
- csrc/
- vllm/model_executor/layers/quantization
- vllm/models/deepseek_v41/quant_config.py
- vllm/utils/humming.py
- tests/quantization
commands:
# Pin torchao to the version compatible with the CI PyTorch and CUDA stack.
- uv pip install --system torchao==0.18.0 --index-url https://download.pytorch.org/whl/cu130
- uv pip install --system torchao==0.17.0 --index-url https://download.pytorch.org/whl/cu130
- uv pip install --system conch-triton-kernels
# The SM90-only checkpoint currently contains a removed weight_chan_scale
# parameter. It was not exercised by the previous L4 job.
- VLLM_TEST_FORCE_LOAD_FORMAT=auto pytest -v -s quantization/ --ignore quantization/test_blackwell_moe.py --ignore quantization/test_rocm_moe.py -k 'not test_compressed_tensors_w4a8_fp8'
- VLLM_TEST_FORCE_LOAD_FORMAT=auto pytest -v -s quantization/ --ignore quantization/test_blackwell_moe.py --ignore quantization/test_rocm_moe.py -k 'not test_compressed_tensors_w4a8_fp8' --shard-id=$$BUILDKITE_PARALLEL_JOB --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT
mirror:
amd:
parallelism: 4
label: ":amd: (MI300) Quantization Shard %N"
dind: false
device: mi300_1
@@ -41,6 +40,8 @@ steps:
- tests/rocm/test_moe_weight_replay.py
- tests/rocm/test_moe_padding.py
commands:
# Use the ROCm default instead of the NVIDIA runner override.
- unset VLLM_USE_V2_MODEL_RUNNER
- uv pip install --system torchao==0.17.0
- uv pip install --system conch-triton-kernels
- VLLM_TEST_FORCE_LOAD_FORMAT=auto pytest -v -s quantization/ --ignore quantization/test_blackwell_moe.py --ignore quantization/test_rocm_moe.py --shard-id=$$BUILDKITE_PARALLEL_JOB --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT
@@ -78,7 +79,6 @@ steps:
- label: ":nvidia: (B200) Quantized MoE"
key: quantized-moe-test-b200
automatic_shard: true
timeout_in_minutes: 120
working_dir: "/vllm-workspace/"
device: b200-k8s
@@ -126,8 +126,9 @@ steps:
- label: ":nvidia: (H200 MIG 35GB) Quantized Models"
device: h200_35gb
key: quantized-models-test
automatic_shard: true
timeout_in_minutes: 65
env:
VLLM_USE_V2_MODEL_RUNNER: "0"
source_file_dependencies:
- vllm/model_executor/layers/quantization
- tests/models/quantization
@@ -137,7 +138,6 @@ steps:
amd:
label: ":amd: (MI355 DPX) Quantized Models"
dind: false
automatic_shard: true
device: mi355_dpx
timeout_in_minutes: 80
working_dir: "/vllm-workspace/tests"
@@ -150,4 +150,6 @@ steps:
- vllm/platforms/rocm.py
- vllm/model_executor/model_loader/
commands:
# Use the ROCm default instead of the NVIDIA runner override.
- unset VLLM_USE_V2_MODEL_RUNNER
- pytest -v -s models/quantization
-25
View File
@@ -1,25 +0,0 @@
group: RL
depends_on:
- image-build
steps:
- label: ":nvidia: (H200 MIG 35GB) RL Entrypoints"
device: h200_35gb
key: rl-entrypoints-tests
timeout_in_minutes: 40
working_dir: "/vllm-workspace/tests"
source_file_dependencies:
- vllm/
- "!vllm/distributed/kv_transfer/"
- tests/entrypoints/rl
commands:
- pytest -v -s entrypoints/rl/weight_transfer
- export VLLM_WORKER_MULTIPROC_METHOD=spawn
- pytest -v -s entrypoints/rl --ignore=entrypoints/rl/weight_transfer
mirror:
amd:
label: ":amd: (MI355 DPX) RL Entrypoints"
dind: false
device: mi355_dpx
timeout_in_minutes: 50
depends_on:
- image-build-amd
+2
View File
@@ -82,6 +82,7 @@ steps:
- tests/entrypoints/scale_out/token_in_token_out/test_serving_tokens.py
- tests/entrypoints/serve/instrumentator/test_basic.py
- tests/entrypoints/serve/instrumentator/test_metrics.py
# - tests/entrypoints/serve/dev/test_sleep.py
- tests/entrypoints/serve/tokenize/test_tokenization.py
commands:
- export VLLM_USE_RUST_FRONTEND=1
@@ -92,6 +93,7 @@ steps:
# test_generate_logprobs expects Python-style top_logprobs truncation (dedup sampled + cap at max(k, 1)).
- pytest -v -s entrypoints/scale_out/token_in_token_out/test_serving_tokens.py -k "not stream and not lora and not test_generate_logprobs and not stop_string_workflow"
- pytest -v -s entrypoints/serve/instrumentator/test_metrics.py -k "text and not show and not run_batch and not test_metrics_counts and not test_metrics_exist"
# - pytest -v -s entrypoints/serve/dev/test_sleep.py
# /tokenizer_info is not implemented in the Rust frontend (the CLI flag is accepted as a no-op).
- pytest -v -s entrypoints/serve/tokenize/test_tokenization.py -k "not tokenizer_info"
mirror:
@@ -14,6 +14,16 @@ steps:
- .buildkite/scripts/run-rust-frontend-cargo-ci.sh
commands:
- .buildkite/scripts/run-rust-frontend-cargo-ci.sh style-clippy
mirror:
amd:
label: ":amd: (MI250) Rust Frontend Cargo Style + Clippy"
dind: false
device: mi250_1
no_gpu: true
working_dir: "/vllm-workspace"
timeout_in_minutes: 35
depends_on:
- image-build-amd
- label: ":computer: (CPU) Rust Frontend Cargo"
key: rust-frontend-cargo-tests
@@ -28,3 +38,13 @@ steps:
- .buildkite/scripts/run-rust-frontend-cargo-ci.sh
commands:
- .buildkite/scripts/run-rust-frontend-cargo-ci.sh test
mirror:
amd:
label: ":amd: (MI250) Rust Frontend Cargo"
dind: false
device: mi250_1
no_gpu: true
working_dir: "/vllm-workspace"
timeout_in_minutes: 35
depends_on:
- image-build-amd
+15 -16
View File
@@ -5,7 +5,6 @@ steps:
- label: ":nvidia: (H200 MIG 35GB) V1 Spec Decode"
device: h200_35gb
key: v1-spec-decode
automatic_shard: true
timeout_in_minutes: 40
source_file_dependencies:
- vllm/config/
@@ -28,7 +27,6 @@ steps:
amd:
label: ":amd: (MI300) V1 Spec Decode"
dind: false
automatic_shard: true
device: mi300_1
timeout_in_minutes: 50
depends_on:
@@ -137,9 +135,9 @@ steps:
- pytest -v -s v1/e2e/spec_decode/speculators/
mirror:
amd:
label: ":amd: (MI355 DPX) Spec Decode Speculators"
label: ":amd: (MI300) Spec Decode Speculators"
dind: false
device: mi355_dpx
device: mi300_1
timeout_in_minutes: 40
depends_on:
- image-build-amd
@@ -167,9 +165,9 @@ steps:
- pytest -v -s v1/e2e/spec_decode/mtp/gemma4/
mirror:
amd:
label: ":amd: (MI355 DPX) Spec Decode MTP Gemma4"
label: ":amd: (MI300) Spec Decode MTP Gemma4"
dind: false
device: mi355_dpx
device: mi300_1
timeout_in_minutes: 40
depends_on:
- image-build-amd
@@ -198,9 +196,9 @@ steps:
- pytest -v -s v1/e2e/spec_decode/mtp/mimo/
mirror:
amd:
label: ":amd: (MI355 DPX) Spec Decode MTP DeepSeek + MiMo"
label: ":amd: (MI300) Spec Decode MTP DeepSeek + MiMo"
dind: false
device: mi355_dpx
device: mi300_1
timeout_in_minutes: 40
depends_on:
- image-build-amd
@@ -228,9 +226,9 @@ steps:
- pytest -v -s v1/e2e/spec_decode/mtp/qwen3_5/
mirror:
amd:
label: ":amd: (MI355 DPX) Spec Decode MTP Qwen3.5"
label: ":amd: (MI300) Spec Decode MTP Qwen3.5"
dind: false
device: mi355_dpx
device: mi300_1
timeout_in_minutes: 40
depends_on:
- image-build-amd
@@ -387,9 +385,9 @@ steps:
- pytest -v -s v1/spec_decode/test_speculators_correctness.py -m slow_test
mirror:
amd:
label: ":amd: (MI355 DPX) Speculators Correctness Nightly"
label: ":amd: (MI300) Speculators Correctness Nightly"
dind: false
device: mi355_dpx
device: mi300_1
timeout_in_minutes: 40
depends_on:
- image-build-amd
@@ -452,9 +450,9 @@ steps:
- pytest -v -s v1/e2e/spec_decode/acceptance_rates/dflash/ -k "not dflash2"
mirror:
amd:
label: ":amd: (MI355 DPX) Spec Decode AL DFlash Nightly"
label: ":amd: (MI300) Spec Decode AL DFlash Nightly"
dind: false
device: mi355_dpx
device: mi300_1
timeout_in_minutes: 120
commands:
- pytest -v -s v1/e2e/spec_decode/acceptance_rates/dflash/ -k "not dflash2"
@@ -531,9 +529,9 @@ steps:
- pytest -v -s v1/e2e/spec_decode/acceptance_rates/dspark/
mirror:
amd:
label: ":amd: (MI355 DPX) Spec Decode AL DSpark Nightly"
label: ":amd: (MI300) Spec Decode AL DSpark Nightly"
dind: false
device: mi355_dpx
device: mi300_1
timeout_in_minutes: 90
depends_on:
- image-build-amd
@@ -595,6 +593,7 @@ steps:
- vllm/model_executor/models/gemma4_mtp.py
- vllm/model_executor/models/llama.py
- vllm/model_executor/models/llama_eagle3.py
- vllm/model_executor/models/medusa.py
- vllm/model_executor/models/registry.py
- tests/evals/gsm8k/
- tests/utils.py
-1
View File
@@ -1 +0,0 @@
../../.agents/skills/model-microbenchmark
-1
View File
@@ -1 +0,0 @@
../../.agents/skills/model-recipes
-1
View File
@@ -1 +0,0 @@
../../.agents/skills/transformers-upgrade
+26 -56
View File
@@ -3,14 +3,12 @@
# This lists cover the "core" components of vLLM that require careful review
/vllm/compilation @zou3519 @youkaichao @ProExpertProg @BoyuanFeng
/vllm/compilation/breakable_cudagraph.py @ZJY0516
/vllm/distributed/kv_transfer @NickLucche @ApostaC @orozery @xuechendi @ivanium @wzhao18
/vllm/distributed/kv_transfer/kv_connector/v1/hisparse @MatthewBonanni @LucasWilkinson
/vllm/lora @jeejeelee
/vllm/model_executor/layers/attention @LucasWilkinson @MatthewBonanni
/vllm/model_executor/layers/fused_moe @mgoin @pavanimajety @zyongye @wzhao18 @bnellnm
/vllm/model_executor/layers/fused_moe @mgoin @pavanimajety @zyongye @wzhao18
/vllm/model_executor/layers/quantization @mgoin @robertgshaw2-redhat @tlrmchlsmth @yewentao256 @pavanimajety @zyongye
/vllm/model_executor/layers/mamba @tdoublep @tomeras91 @ZJY0516
/vllm/model_executor/layers/mamba @tdoublep @tomeras91
/vllm/model_executor/layers/mamba/gdn/qwen_gdn_linear_attn.py @tdoublep @ZJY0516 @vadiklyutiy
/vllm/model_executor/model_loader @22quinn
/vllm/model_executor/determinism/ @yewentao256
@@ -27,7 +25,7 @@
# Any change to the VllmConfig changes can have a large user-facing impact,
# so spam a lot of people
/vllm/config @WoosukKwon @youkaichao @robertgshaw2-redhat @mgoin @tlrmchlsmth @houseroad @yewentao256 @ProExpertProg
/vllm/config/cache.py @heheda12345 @ivanium @ZJY0516
/vllm/config/cache.py @heheda12345 @ivanium
# Config utils
/vllm/config/utils.py @hmellor
@@ -35,48 +33,23 @@
/vllm/utils/argparse_utils.py
# Entrypoints
/vllm/entrypoints/anthropic @chaunceyjiang @mgoin @DarkLight1337
/vllm/entrypoints/cohere @DarkLight1337
/vllm/entrypoints/anthropic @mgoin @DarkLight1337
/vllm/entrypoints/cli @hmellor @mgoin @DarkLight1337
/vllm/entrypoints/mcp @heheda12345
/vllm/entrypoints/openai/chat_completion @chaunceyjiang @DarkLight1337
/vllm/entrypoints/openai/completion @chaunceyjiang @DarkLight1337
/vllm/entrypoints/openai/models @chaunceyjiang @DarkLight1337
/vllm/entrypoints/openai/parser @aarnphm @chaunceyjiang @sfeng33
/vllm/entrypoints/openai/responses @aarnphm @chaunceyjiang @sfeng33
/vllm/entrypoints/pooling @noooop @DarkLight1337
/vllm/entrypoints/rl @aoshen02 @njhill
/vllm/entrypoints/scale_out @aoshen02 @noooop @DarkLight1337
/vllm/entrypoints/openai @aarnphm @chaunceyjiang @DarkLight1337
/vllm/entrypoints/speech_to_text/realtime @njhill
/vllm/entrypoints/speech_to_text @NickLucche
/vllm/entrypoints/speech_to_text/realtime @njhill @NickLucche
/vllm/entrypoints/serve @njhill
/vllm/entrypoints/pooling @noooop
/vllm/entrypoints/serve/sagemaker @DarkLight1337
/vllm/entrypoints/common @DarkLight1337
/vllm/entrypoints/serve @njhill
/vllm/entrypoints/*.py @njhill
/vllm/entrypoints/chat_utils.py @DarkLight1337
/vllm/entrypoints/offline_utils.py @DarkLight1337
/vllm/entrypoints/llm.py @DarkLight1337
/tests/entrypoints @robertgshaw2-redhat @aarnphm @NickLucche @AndreasKaratzas
/tests/entrypoints/anthropic @mgoin @DarkLight1337
/tests/entrypoints/cohere @DarkLight1337
/tests/entrypoints/cli @hmellor @mgoin @DarkLight1337
/tests/entrypoints/openai/chat_completion @chaunceyjiang @DarkLight1337
/tests/entrypoints/openai/completion @chaunceyjiang @DarkLight1337
/tests/entrypoints/openai/models @chaunceyjiang @DarkLight1337
/tests/entrypoints/openai/parser @aarnphm @chaunceyjiang @sfeng33
/tests/entrypoints/openai/responses @aarnphm @chaunceyjiang @sfeng33
/tests/entrypoints/pooling @noooop @DarkLight1337
/tests/entrypoints/rl @aoshen02 @njhill
/tests/entrypoints/scale_out @noooop @DarkLight1337
/tests/entrypoints/speech_to_text @NickLucche
/tests/entrypoints/speech_to_text/realtime @njhill @NickLucche
/tests/entrypoints/serve @njhill
/tests/entrypoints/serve/sagemaker @DarkLight1337
/tests/entrypoints/llm @DarkLight1337
/tests/entrypoints/llm/test_struct_output_generate.py @mgoin @aarnphm @arpera
# Rust Frontend
/rust/ @BugenZhao @njhill
/rust/src/bench @esmeetu
/rust/src/server/src/routes/inference @aoshen02 @BugenZhao @njhill
/tools/build_rust.sh @BugenZhao @njhill
/rust-toolchain.toml @BugenZhao @njhill
/.buildkite/test_areas/rust* @BugenZhao @njhill
@@ -90,12 +63,6 @@
/vllm/tool_parsers @aarnphm @chaunceyjiang @sfeng33 @bbrowning
/vllm/parser @aarnphm @chaunceyjiang @sfeng33 @bbrowning
/tests/tokenizers_ @DarkLight1337 @njhill
/tests/renderers @DarkLight1337 @njhill
/tests/reasoning @aarnphm @chaunceyjiang @sfeng33 @bbrowning
/tests/tool_parsers @aarnphm @chaunceyjiang @sfeng33 @bbrowning
/tests/tool_use @aarnphm @chaunceyjiang @sfeng33 @bbrowning
# vLLM V1
/vllm/v1/attention @LucasWilkinson @MatthewBonanni
/vllm/v1/attention/backend.py @WoosukKwon @zhuohan123 @youkaichao @alexm-redhat @njhill
@@ -103,18 +70,16 @@
/vllm/v1/attention/backends/flashinfer.py @mgoin @pavanimajety @vadiklyutiy @wzhao18
/vllm/v1/attention/backends/triton_attn.py @tdoublep
/vllm/v1/attention/backends/gdn_attn.py @ZJY0516 @vadiklyutiy
/vllm/v1/core @WoosukKwon @robertgshaw2-redhat @njhill @ywang96 @alexm-redhat @heheda12345 @ApostaC @orozery @ivanium @ZJY0516
/vllm/v1/core @WoosukKwon @robertgshaw2-redhat @njhill @ywang96 @alexm-redhat @heheda12345 @ApostaC @orozery @ivanium
/vllm/v1/sample @22quinn @houseroad @njhill
/vllm/v1/spec_decode @benchislett @luccafong @MatthewBonanni
/vllm/v1/structured_output @mgoin @aarnphm @benchislett @arpera
/vllm/v1/kv_cache_interface.py @heheda12345 @ivanium @ZJY0516
/vllm/v1/kv_cache_interface.py @heheda12345 @ivanium
/vllm/v1/kv_offload @ApostaC @orozery
/vllm/v1/simple_kv_offload @ivanium
/vllm/v1/hisparse @MatthewBonanni @LucasWilkinson
/vllm/v1/engine @njhill
/vllm/v1/executor @njhill
/vllm/v1/worker @njhill
/vllm/v1/worker/mamba_utils.py @njhill @ZJY0516
/vllm/v1/worker/kv_connector_model_runner_mixin.py @orozery @NickLucche @ivanium
/vllm/v1/worker/encoder_cudagraph.py @shen-shanshan
/vllm/v1/worker/encoder_cudagraph_defs.py @shen-shanshan
@@ -122,7 +87,6 @@
# Model runner V2
/vllm/v1/worker/gpu @WoosukKwon @njhill @yewentao256
/vllm/v1/worker/gpu/kv_connector.py @orozery
/vllm/v1/worker/gpu/model_states/mamba_hybrid.py @WoosukKwon @njhill @yewentao256 @ZJY0516
# CI & building
/.buildkite @Harry-Chen @khluu
@@ -135,23 +99,27 @@
/tests/distributed/test_multi_node_assignment.py @youkaichao
/tests/distributed/test_pipeline_parallel.py @youkaichao
/tests/distributed/test_same_node.py @youkaichao
/tests/evals @mgoin @vadiklyutiy @AndreasKaratzas @ZJY0516
/tests/entrypoints @DarkLight1337 @robertgshaw2-redhat @aarnphm @NickLucche @AndreasKaratzas
/tests/evals @mgoin @vadiklyutiy @AndreasKaratzas
/tests/kernels @mgoin @tlrmchlsmth @WoosukKwon @yewentao256 @zyongye @AndreasKaratzas
/tests/kernels/ir @ProExpertProg @tjtanaa
/tests/kernels/moe @bnellnm
/tests/models @DarkLight1337 @ywang96 @AndreasKaratzas
/tests/multimodal @DarkLight1337 @ywang96 @NickLucche @shen-shanshan
/tests/multimodal/media @Isotr0py
/tests/quantization @mgoin @robertgshaw2-redhat @yewentao256 @pavanimajety @zyongye @AndreasKaratzas
/tests/entrypoints/llm/test_struct_output_generate.py @mgoin @aarnphm @arpera
/tests/v1/structured_output @mgoin @aarnphm @arpera
/tests/v1/core @WoosukKwon @robertgshaw2-redhat @njhill @ywang96 @alexm-redhat @heheda12345 @ApostaC @orozery @ivanium @ZJY0516
/tests/v1/core @WoosukKwon @robertgshaw2-redhat @njhill @ywang96 @alexm-redhat @heheda12345 @ApostaC @orozery @ivanium
/tests/lora @jeejeelee
/tests/models/language/generation/test_hybrid.py @tdoublep @tomeras91 @ZJY0516
/tests/models/language/generation/test_hybrid.py @tdoublep @tomeras91
/tests/v1/kv_connector/nixl_integration @NickLucche
/tests/v1/kv_connector @ApostaC @orozery @ivanium @wzhao18
/tests/v1/kv_offload @ApostaC @orozery
/tests/v1/simple_kv_offload @ivanium
/tests/v1/determinism @yewentao256
/tests/reasoning @aarnphm @chaunceyjiang @sfeng33 @bbrowning
/tests/tool_parsers @aarnphm @chaunceyjiang @sfeng33 @bbrowning
/tests/tool_use @aarnphm @chaunceyjiang @sfeng33 @bbrowning
# Transformers modeling backend
/vllm/model_executor/models/transformers @hmellor
@@ -165,9 +133,6 @@
.readthedocs.yaml @hmellor
mkdocs.yaml @hmellor
# Model recipes
/recipes @esmeetu
# Linting
/tools/pre_commit/.markdownlint.yaml @hmellor
.pre-commit-config.yaml @hmellor
@@ -256,14 +221,19 @@ mkdocs.yaml @hmellor
/examples/pooling @noooop
/docs/models/pooling_models @noooop
/tests/models/*/pooling* @noooop
/tests/entrypoints/pooling @noooop
/vllm/config/pooler.py @noooop
/vllm/model_executor/layers/pooler @noooop
# Reinforcement learning
/vllm/distributed/weight_transfer @aoshen02
/vllm/entrypoints/serve/dev/rlhf @njhill @aoshen02
/tests/distributed/test_weight_transfer.py @aoshen02
/tests/entrypoints/weight_transfer @DarkLight1337 @robertgshaw2-redhat @aarnphm @NickLucche @AndreasKaratzas @aoshen02
/tests/entrypoints/serve/dev/rlhf @DarkLight1337 @robertgshaw2-redhat @aarnphm @NickLucche @AndreasKaratzas @aoshen02
/tests/v1/worker/test_gpu_worker_weight_transfer.py @aoshen02
/examples/rl @aoshen02
/examples/features/pause_resume @aoshen02
/docs/training/rlhf.md @aoshen02
/docs/training/async_rl.md @aoshen02
/docs/training/weight_transfer @aoshen02
+6 -1
View File
@@ -494,7 +494,8 @@ pull_request_rules:
- files~=^tests/v1/e2e/spec_decode/
- files~=^examples/features/speculative_decoding/
- files~=^vllm/model_executor/models/.*eagle.*\.py
- files~=^vllm/transformers_utils/configs/eagle\.py
- files=vllm/model_executor/models/mlp_speculator.py
- files~=^vllm/transformers_utils/configs/(eagle|medusa|mlp_speculator)\.py
actions:
label:
add:
@@ -548,6 +549,8 @@ pull_request_rules:
- label != stale
- or:
- files~=tpu.py
- files~=_tpu
- files~=tpu_
- files~=/tpu/
actions:
label:
@@ -561,6 +564,8 @@ pull_request_rules:
- label != stale
- and:
- -files~=tpu.py
- -files~=_tpu
- -files~=tpu_
- -files~=/tpu/
actions:
label:
+1
View File
@@ -279,6 +279,7 @@ jobs:
"speculative-decoding": {
keywords: [
{ term: "eagle", searchIn: "title" },
{ term: "medusa", searchIn: "title" },
],
substrings: [
{ term: "speculative decoding", searchIn: "title" },
-28
View File
@@ -1,28 +0,0 @@
name: Recipes
# Validate recipe changes against the renderer that publishes them
# (recipes.vllm.ai). The check logic lives in vllm-project/recipes and tracks
# its main branch, so a PR is held to the same validation that runs when
# vllm-project/recipes picks up this directory after merge.
on:
pull_request:
paths:
- 'recipes/**'
- '.github/workflows/recipes.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
permissions:
contents: read
jobs:
validate:
name: Recipes
if: github.repository_owner == 'vllm-project'
uses: vllm-project/recipes/.github/workflows/validate-recipes.yml@main
with:
source: vllm
path: recipes
+16 -14
View File
@@ -18,8 +18,22 @@ jobs:
if: >-
github.repository_owner == 'vllm-project' &&
github.event.issue.pull_request &&
(startsWith(github.event.comment.body, '/ci') ||
startsWith(github.event.comment.body, '/amd-ci'))
(github.event.comment.body == '/ci run' ||
github.event.comment.body == '/ci run all' ||
github.event.comment.body == '/ci run nightly' ||
github.event.comment.body == '/ci run --allow-stale' ||
github.event.comment.body == '/ci run all --allow-stale' ||
github.event.comment.body == '/ci run nightly --allow-stale' ||
github.event.comment.body == '/ci retry' ||
github.event.comment.body == '/ci cancel' ||
github.event.comment.body == '/amd-ci run' ||
github.event.comment.body == '/amd-ci run all' ||
github.event.comment.body == '/amd-ci run nightly' ||
github.event.comment.body == '/amd-ci run --allow-stale' ||
github.event.comment.body == '/amd-ci run all --allow-stale' ||
github.event.comment.body == '/amd-ci run nightly --allow-stale' ||
github.event.comment.body == '/amd-ci retry' ||
github.event.comment.body == '/amd-ci cancel')
runs-on: [self-hosted, linux, x64, vllm-runners]
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
@@ -39,16 +53,4 @@ jobs:
BUILDKITE_ORGANIZATION: vllm
BUILDKITE_PIPELINE: ci
CI_TRUSTED_USERS: ${{ vars.CI_TRUSTED_USERS }}
GH_BRANCH_UPDATE_TOKEN: ${{ secrets.CI_BRANCH_UPDATE_TOKEN }}
GH_TOKEN: ${{ github.token }}
- name: Report failure on the PR
if: failure()
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
with:
script: |
const runUrl = `${context.serverUrl}/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId}`;
await github.rest.issues.createComment({
...context.repo,
issue_number: context.issue.number,
body: `❌ @${context.payload.comment.user.login}, your CI command failed with an unexpected error, so CI may not have started. See the [workflow run](${runUrl}) for details.`,
});
+8 -138
View File
@@ -58,12 +58,9 @@ AMD_CI_COMMANDS = frozenset(
RETRY_COMMANDS = frozenset({COMMAND_RETRY_FAILED, COMMAND_RETRY_AMD_FAILED})
CANCEL_COMMANDS = frozenset({COMMAND_CANCEL_CI, COMMAND_CANCEL_AMD_CI})
ALL_CI_COMMANDS = UPSTREAM_CI_COMMANDS | AMD_CI_COMMANDS
CI_COMMAND_PREFIXES = frozenset({"/ci", "/amd-ci"})
CI_AUTHORIZED_COMMENT_MARKER = "<!-- vllm-ci-authorized -->"
READY_LABELS = {"ready", "ready-run-all-tests"}
TRUSTED_PERMISSIONS = {"admin", "maintain", "write"}
BRANCH_UPDATE_POLL_SECONDS = 3
BRANCH_UPDATE_POLL_ATTEMPTS = 40
ACTIVE_BUILD_STATES = {
"blocked",
"creating",
@@ -261,13 +258,6 @@ class GitHubClient:
raise ApiError(None, "GitHub returned an invalid commits-behind count.")
return behind
def update_branch(self, number: int, expected_head_sha: str) -> None:
self._request(
self._repo_path(f"/pulls/{number}/update-branch"),
body={"expected_head_sha": expected_head_sha},
method="PUT",
)
def list_pulls_for_commit(self, commit: str) -> list[dict[str, Any]]:
commit = urllib.parse.quote(commit, safe="")
response = self._request(self._repo_path(f"/commits/{commit}/pulls"))
@@ -471,17 +461,11 @@ class BuildkiteClient:
def parse_command(body: str) -> str | None:
body = body.strip()
if body in ALL_CI_COMMANDS:
return body
return None
def is_ci_command_attempt(body: str) -> bool:
words = body.split(maxsplit=1)
return bool(words) and words[0].casefold() in CI_COMMAND_PREFIXES
def pipeline_for_command(
command: str,
*,
@@ -621,16 +605,6 @@ def blocks_new_run(command: str, build: Mapping[str, Any]) -> bool:
return is_active_build(build)
def list_commit_builds(
buildkite: BuildkiteClient, command: str, commit: str
) -> list[dict[str, Any]]:
builds = buildkite.list_builds(commit)
if command not in AMD_CI_COMMANDS:
return builds
# bootstrap-amd.sh exits before uploading any test steps for PR webhook builds.
return [build for build in builds if build.get("source") != "webhook"]
def select_latest_build(
builds: Sequence[dict[str, Any]],
pr_number: int,
@@ -797,11 +771,10 @@ def notify_authorized(
(
f"✅ @{author}, CI is now available for this PR.\n\n"
"- `/ci run` starts upstream CI; `/amd-ci run` starts AMD CI only.\n"
"- Run commands test your branch with every commit currently on its "
"upstream target branch. If it is behind, the bot merges the target "
"branch into it first (pull before pushing again). Append "
"`--allow-stale` to a run command to test an outdated branch as is, "
"at your own risk.\n"
"- Your branch must contain every commit currently on its upstream "
"target branch. Merge or rebase onto the latest target branch, then "
"rerun the command. Append `--allow-stale` to a run command to test "
"an outdated branch at your own risk.\n"
"- `/ci retry` retries failed jobs in the CI build for the current "
"PR head. If the current head has no CI build, it starts a new CI "
"build for the current head containing only jobs that failed in "
@@ -849,65 +822,10 @@ def resolve_workflow_run_pr(
return find_matching_pr(github.list_pulls_for_commit(head_sha))
def merge_base_into_pr(
github: GitHubClient,
branch_updater: GitHubClient,
pr: Mapping[str, Any],
command: str,
lag: str,
) -> tuple[dict[str, Any], str]:
base_ref = pr["base"]["ref"]
def merge_failed(reason: str) -> CiPreparationError:
return CiPreparationError(
f"{lag} The bot could not merge upstream `{base_ref}` into this "
f"branch ({reason}). No new CI build was started. Either merge or "
f"rebase onto the latest `{base_ref}` yourself and rerun `{command}`, "
f"or use `{command}{ALLOW_STALE_SUFFIX}` to test this branch as is."
)
try:
branch_updater.update_branch(pr["number"], pr["head"]["sha"])
for _ in range(BRANCH_UPDATE_POLL_ATTEMPTS):
time.sleep(BRANCH_UPDATE_POLL_SECONDS)
current_pr = github.get_pr(pr["number"])
if current_pr["head"]["sha"] == pr["head"]["sha"]:
continue
try:
behind = github.get_commits_behind_base(
base_ref, current_pr["head"]["sha"]
)
except ApiError as error:
# A fork's new head is briefly unknown to the base repository.
if error.status == 404:
continue
raise
break
else:
raise merge_failed("the merge commit did not appear in time")
if (
current_pr["state"] != "open"
or current_pr["base"]["ref"] != base_ref
or behind
):
raise merge_failed("the PR changed during the merge")
except ApiError as error:
raise merge_failed(str(error)) from error
note = (
f"ℹ️ {lag} Merged upstream `{base_ref}` into this branch as "
f"`{current_pr['head']['sha'][:12]}` before starting CI. "
"Pull the branch before pushing more changes."
)
print(note)
return current_pr, note
def prepare_pr_for_ci(
github: GitHubClient,
pr: Mapping[str, Any],
command: str,
branch_updater: GitHubClient | None = None,
) -> tuple[dict[str, Any], str]:
base_ref = pr["base"]["ref"]
no_action = "No new CI build was started."
@@ -943,8 +861,6 @@ def prepare_pr_for_ci(
print(warning)
return current_pr, warning
if behind and branch_updater is not None:
return merge_base_into_pr(github, branch_updater, current_pr, command, lag)
if behind:
raise CiPreparationError(
f"{lag} Your branch must contain every commit currently on upstream "
@@ -964,8 +880,6 @@ def handle_run_ci(
command: str,
github: GitHubClient,
pr: Mapping[str, Any],
branch_updater: GitHubClient | None = None,
allow_repeat_run: bool = False,
) -> str:
ci_name = ci_name_for_command(command)
duplicate_builds = buildkite.list_builds(
@@ -978,7 +892,7 @@ def handle_run_ci(
f"{ci_name} was already requested by this comment: {duplicate['web_url']}"
)
current_builds = list_commit_builds(buildkite, command, pr["head"]["sha"])
current_builds = buildkite.list_builds(pr["head"]["sha"])
active_build = next(
(
build
@@ -992,19 +906,7 @@ def handle_run_ci(
f"{ci_name} is already running for this commit: {active_build['web_url']}"
)
tested_build = select_latest_build(
[build for build in current_builds if build.get("state") != "canceled"],
pr["number"],
)
if tested_build and not allow_repeat_run:
raise CiPreparationError(
f"{ci_name} already ran on this commit: {tested_build['web_url']}. "
"No new CI build was started. Push new changes to run CI again, use "
f"`{retry_command_for_command(command)}` to rerun failed jobs, or ask "
"a reviewer with write access to rerun it."
)
current_pr, warning = prepare_pr_for_ci(github, pr, command, branch_updater)
current_pr, warning = prepare_pr_for_ci(github, pr, command)
build = buildkite.create_build(
create_build_payload(
@@ -1033,7 +935,7 @@ def handle_retry_failed(
ci_name = ci_name_for_command(command)
run_command = run_command_for_command(command)
retry_command = retry_command_for_command(command)
builds = list_commit_builds(buildkite, command, pr["head"]["sha"])
builds = buildkite.list_builds(pr["head"]["sha"])
build = select_latest_build(builds, pr["number"])
if build:
metadata = build.get("meta_data") or {}
@@ -1184,34 +1086,11 @@ def handle_cancel_ci(
)
def reply_unrecognized_command(
event: Mapping[str, Any],
github: GitHubClient,
) -> None:
if "pull_request" not in event["issue"]:
return
issue_number = event["issue"]["number"]
comment_id = event["comment"]["id"]
actor = event["comment"]["user"]["login"]
if is_already_handled(github, issue_number, comment_id):
print(f"Comment {comment_id} was already handled.")
return
add_reaction_safely(github, comment_id, "-1")
commands = ", ".join(f"`{command}`" for command in sorted(ALL_CI_COMMANDS))
github.add_comment(
issue_number,
f"❌ @{actor}, that is not a recognized CI command, so no CI was "
"started. Post a new comment containing only one of: "
f"{commands}.\n\n{command_comment_marker(comment_id)}",
)
def run(
event: Mapping[str, Any],
github: GitHubClient,
buildkite: BuildkiteClient,
trusted_users_value: str = "",
branch_updater: GitHubClient | None = None,
) -> None:
command = parse_command(event["comment"]["body"])
if not command or "pull_request" not in event["issue"]:
@@ -1274,8 +1153,6 @@ def run(
command=command,
github=github,
pr=pr,
branch_updater=branch_updater,
allow_repeat_run=is_trusted_permission(permission),
)
elif command in RETRY_COMMANDS:
message = handle_retry_failed(
@@ -1340,11 +1217,8 @@ def main() -> None:
)
return
body = event["comment"]["body"]
command = parse_command(body)
command = parse_command(event["comment"]["body"])
if not command:
if is_ci_command_attempt(body):
reply_unrecognized_command(event, github)
return
pipeline = pipeline_for_command(
@@ -1357,15 +1231,11 @@ def main() -> None:
os.environ.get("BUILDKITE_ORGANIZATION", "vllm"),
pipeline,
)
branch_update_token = os.environ.get("GH_BRANCH_UPDATE_TOKEN", "")
run(
event,
github,
buildkite,
os.environ.get("CI_TRUSTED_USERS", ""),
GitHubClient(branch_update_token, os.environ["GITHUB_REPOSITORY"])
if branch_update_token
else None,
)
@@ -32,12 +32,10 @@ from run_ci_command import (
has_trusted_approval,
is_active_build,
is_build_for_pr,
is_ci_command_attempt,
notify_authorized,
parse_command,
parse_trusted_users,
pipeline_for_command,
reply_unrecognized_command,
resolve_workflow_run_pr,
run,
select_latest_build,
@@ -144,23 +142,6 @@ class FakeGitHub:
self.comments.append(body)
class FakeBranchUpdater:
def __init__(self, github: FakeGitHub, error: ApiError | None = None) -> None:
self.calls: list[tuple[int, str]] = []
self.error = error
self.github = github
def update_branch(self, number: int, expected_head_sha: str) -> None:
self.calls.append((number, expected_head_sha))
if self.error:
raise self.error
self.github.prs[number] = {
**self.github.prs[number],
"head": {**self.github.prs[number]["head"], "sha": "merge-commit-sha"},
}
self.github.behind = 0
class FakeBuildkite:
def __init__(
self,
@@ -359,46 +340,14 @@ class RunCiCommandTest(unittest.TestCase):
self.assertIsNone(parse_command("/ci run please"))
self.assertIsNone(parse_command("/ci run all please"))
self.assertIsNone(parse_command("/ci cancel please"))
self.assertIsNone(parse_command(" /ci run"))
self.assertIsNone(parse_command("/amd-ci run please"))
self.assertIsNone(parse_command("/amd-ci retry "))
self.assertIsNone(parse_command("/AMD-CI run"))
self.assertIsNone(parse_command("/amdci run"))
self.assertIsNone(parse_command("/ci run --allow-stale all"))
self.assertIsNone(parse_command("/amd-ci run --allow-stale please"))
def test_surrounding_whitespace_is_ignored(self) -> None:
self.assertEqual(parse_command("/ci run\r\n"), COMMAND_RUN_CI)
self.assertEqual(parse_command(" /ci run "), COMMAND_RUN_CI)
self.assertEqual(parse_command("/amd-ci retry\n"), COMMAND_RETRY_AMD_FAILED)
def test_ci_command_attempts_are_detected(self) -> None:
for body in ("/ci", "/ci rn", "/CI run", "/amd-ci run please", " /ci\nrun"):
with self.subTest(body=body):
self.assertTrue(is_ci_command_attempt(body))
for body in ("", "LGTM", "/cifoo", "/amdci run", "please /ci run"):
with self.subTest(body=body):
self.assertFalse(is_ci_command_attempt(body))
def test_unrecognized_command_gets_feedback(self) -> None:
github = FakeGitHub()
reply_unrecognized_command(make_event("/ci rn", actor="author"), github)
self.assertEqual(github.reactions, ["-1"])
self.assertEqual(len(github.comments), 1)
self.assertIn(
"@author, that is not a recognized CI command", github.comments[0]
)
self.assertIn(f"`{COMMAND_RUN_CI}`", github.comments[0])
self.assertIn("<!-- vllm-ci-command:99 -->", github.comments[0])
def test_unrecognized_command_feedback_is_not_repeated(self) -> None:
github = FakeGitHub(comments=["❌ earlier\n\n<!-- vllm-ci-command:99 -->"])
reply_unrecognized_command(make_event("/ci rn"), github)
self.assertEqual(len(github.comments), 1)
self.assertEqual(github.reactions, [])
def test_commands_select_only_their_configured_pipeline(self) -> None:
cases = (
(COMMAND_RUN_CI, "ci"),
@@ -633,82 +582,6 @@ class RunCiCommandTest(unittest.TestCase):
)
self.assertIn(command, github.comments[0])
def test_run_merges_base_into_outdated_branch_before_building(self) -> None:
github = FakeGitHub(behind=5)
updater = FakeBranchUpdater(github)
buildkite = FakeBuildkite([[], []])
with patch("run_ci_command.time.sleep"):
run(make_event(COMMAND_RUN_CI), github, buildkite, "", updater)
self.assertEqual(updater.calls, [(42, "0123456789abcdef")])
self.assertEqual(buildkite.created_builds[0]["commit"], "merge-commit-sha")
self.assertTrue(github.comments[0].startswith("✅ "))
self.assertIn("5 commits behind", github.comments[0])
self.assertIn("Merged upstream `main`", github.comments[0])
def test_only_committers_can_rerun_a_commit_that_already_ran_ci(self) -> None:
finished_build = {
"created_at": "2026-07-28T01:00:00Z",
"pull_request": {"id": 42},
"state": "failed",
"web_url": "https://buildkite.example/builds/122",
}
for actor, permission, command, allowed in (
("author", "read", COMMAND_RUN_CI, False),
("author", "read", f"{COMMAND_RUN_CI} --allow-stale", False),
("reviewer", "write", COMMAND_RUN_CI, True),
):
with self.subTest(actor=actor, command=command):
github = FakeGitHub(
behind=5,
permission=permission,
pr=make_pr(labels=[{"name": "ready"}]),
)
updater = FakeBranchUpdater(github)
buildkite = FakeBuildkite([[], [finished_build]])
with patch("run_ci_command.time.sleep"):
run(make_event(command, actor), github, buildkite, "", updater)
self.assertEqual(bool(buildkite.created_builds), allowed)
if not allowed:
self.assertEqual(updater.calls, [])
self.assertIn("CI already ran on this commit", github.comments[0])
self.assertIn("`/ci retry`", github.comments[0])
def test_run_waits_for_fork_merge_commit_to_reach_base_repo(self) -> None:
github = FakeGitHub(behind=5)
updater = FakeBranchUpdater(github)
buildkite = FakeBuildkite([[], []])
with (
patch("run_ci_command.time.sleep"),
patch.object(
github,
"get_commits_behind_base",
side_effect=[5, ApiError(404, "Not Found"), 0],
),
):
run(make_event(COMMAND_RUN_CI), github, buildkite, "", updater)
self.assertEqual(buildkite.created_builds[0]["commit"], "merge-commit-sha")
self.assertTrue(github.comments[0].startswith("✅ "))
def test_run_reports_failed_merge_without_building(self) -> None:
github = FakeGitHub(behind=5)
updater = FakeBranchUpdater(github, error=ApiError(422, "merge conflict"))
buildkite = FakeBuildkite([[], []])
run(make_event(COMMAND_RUN_CI), github, buildkite, "", updater)
self.assertEqual(buildkite.created_builds, [])
self.assertTrue(github.comments[0].startswith("❌ "))
self.assertIn("could not merge upstream `main`", github.comments[0])
self.assertIn("merge conflict", github.comments[0])
self.assertIn("No new CI build was started.", github.comments[0])
self.assertIn("`/ci run --allow-stale`", github.comments[0])
def test_allow_stale_preserves_author_permissions_and_run_configuration(
self,
) -> None:
@@ -883,42 +756,6 @@ class RunCiCommandTest(unittest.TestCase):
self.assertEqual(buildkite.created_builds, [])
self.assertIn("AMD CI is already running", github.comments[0])
def test_amd_run_ignores_finished_webhook_build_for_author(self) -> None:
webhook_build = {
"created_at": "2026-08-18T01:00:00Z",
"meta_data": {},
"number": 122,
"pull_request": {"id": 42},
"source": "webhook",
"state": "passed",
"web_url": "https://buildkite.example/amd-ci/builds/122",
}
ui_build = {**webhook_build, "source": "ui"}
command_build = {
**webhook_build,
"meta_data": {"github-comment-id": "98"},
"source": "api",
}
for build, allowed in (
(webhook_build, True),
(ui_build, False),
(command_build, False),
):
with self.subTest(source=build["source"]):
github = FakeGitHub(
permission="read",
pr=make_pr(labels=[{"name": "ready"}]),
)
buildkite = FakeBuildkite([[], [build]])
run(make_event(COMMAND_RUN_AMD_CI, "author"), github, buildkite)
self.assertEqual(bool(buildkite.created_builds), allowed)
if not allowed:
self.assertIn(
"AMD CI already ran on this commit", github.comments[0]
)
def test_run_all_sets_buildkite_environment(self) -> None:
github = FakeGitHub()
buildkite = FakeBuildkite([[], []])
@@ -1265,29 +1102,6 @@ class RunCiCommandTest(unittest.TestCase):
)
self.assertIn("Use `/amd-ci run`", github.comments[0])
def test_amd_ci_retry_ignores_webhook_build_for_the_current_head(self) -> None:
github = FakeGitHub(
permission="read",
pr=make_pr(labels=[{"name": "ready"}]),
)
webhook_build = {
"created_at": "2026-08-18T01:00:00Z",
"meta_data": {},
"number": 122,
"pull_request": {"id": 42},
"source": "webhook",
"state": "passed",
"web_url": "https://buildkite.example/amd-ci/builds/122",
}
buildkite = FakeBuildkite([[webhook_build]])
run(make_event(COMMAND_RETRY_AMD_FAILED, "author"), github, buildkite)
self.assertEqual(buildkite.retry_calls, [])
self.assertIn(
"No AMD CI build exists for the current PR head", github.comments[0]
)
def test_ci_retry_creates_filtered_build_for_new_head(self) -> None:
github = FakeGitHub(
behind=100,
+3
View File
@@ -237,6 +237,9 @@ benchmarks/**/*.json
actionlint
shellcheck*/
# Ignore moe/marlin_moe gen code
csrc/libtorch_stable/moe/marlin_moe_wna16/kernel_*
# Ignore ep_kernels_workspace folder
ep_kernels_workspace/

Some files were not shown because too many files have changed in this diff Show More