Compare commits

..
40 Commits
Author SHA1 Message Date
MOVZX be95edb7f6 Merge branch 'main' into patches
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-16 17:35:42 +07:00
MOVZXandHinata 8dc54f35dd [Offloader] Stream every multimodal tower parameter to host RAM
Qwen3_VisionTransformer derived its compute device from
patch_embed.proj.weight and fed pos_embed.weight straight into the
position-embedding interpolation kernel, so the last two anchor weights
had to stay in VRAM for a tower to be streamable at all.

Record the device the tower was built on before the offloader moves
anything, and fall back to it when no weight is on the accelerator.
Move the interpolation table to that device per encode, which costs one
8.4 MiB host-to-device copy for a 456 MiB tower. Towers now leave no
parameters in VRAM between encodes; only small buffers such as the fused
input norm and rotary inv_freq remain.

Also let a tower restrict streaming through tower_offload_children, and
warn when a tower resolves its device property to another device type,
so a tower that reads an anchor weight reports the cause instead of a
Triton pointer error.

Co-authored-by: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-16 16:07:53 +07:00
MOVZXandHinata 18c06ed406 [Offloader] Stream multimodal towers from host RAM with bulk-copy
Add --offload-towers: place multimodal tower weights in pinned host
memory and bulk-copy each unit to the GPU only during its forward,
keeping towers out of VRAM between encodes. Unlike UVA zero-copy,
whole-tensor H2D transfers avoid re-reading operand tiles over PCIe
inside the GEMM inner loop. Tower streaming is independent of layer
group scheduling, so group_size may be 0 for tower-only offload.

Co-authored-by: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-16 14:32:14 +07:00
MOVZXandHinata 5d6754ff6b fix: keep the refusal-direction projection dynamo-safe
With VLLM_ABLITERATION set, boot crashed: dynamo traces
abliteration.register from Qwen3NextGatedDeltaNet.__init__, and
buf.copy_(r) fails on the meta device during the compile-time model
construction pass. Disable tracing for register and guard the copy so
an unmaterializable buffer turns projection off with a warning.

Log activation at registration instead of inside project_: project_
runs in the compiled region, and a logger call there aborts dynamo.
The width guard uses the same escape hatch.

Also project the embedding that Qwen3NextMultiTokenPredictor builds
for itself, which the first pass missed, and add tests for the
2560-value Flash-Next direction against a 5120 model and for a
half-width activation.

Co-authored-by: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-15 22:56:06 +07:00
MOVZXandHinata 0076adc78b [Qwen3.5] Project the refusal direction out of the residual stream at runtime
orcarouter/Qwen3.8-27B-Uncensored removes refusal by editing weights. The
Swift release we serve is AutoRound W8A16, so re-editing weights means
re-quantizing. Project one unit direction r out of every residual-stream
writer output instead: x <- x - r (r . x), which equals the weight edit
W' = W - r(r^T W) on every writer feeding that stream (Arditi et al. 2024).

Sites: attention output for both the full-attention and GDN branches, the
MLP output, and token embeddings, in Qwen3NextDecoderLayer.forward and
Qwen3NextModel.embed_input_ids, which Qwen3_5DecoderLayer and Qwen3_5Model
inherit but whose base constructors they skip, so both register there too.
The MTP drafter reuses Qwen3_5DecoderLayer and calls its own
embed_input_ids, so the draft path is covered at the same width, and
_maybe_share_embeddings at PP=1 hands the draft the target embedding
module. Sequence parallelism shards dim 0 only, so the last dim is
hidden_size at every site.

VLLM_ABLITERATION names a JSON file with key refusal_direction, a
safetensors tensor, or a bare name resolved under
/opt/qwen38/abliteration then /data/abliteration. Empty or 0 is off.
register() always creates the buffer _abliteration_r, zeros when off, so
one compiled graph serves ON and OFF and toggling the switch
(declared in envs.py, hence in compile_factors) invalidates the compile
cache. project_() is an in-place rank-1 update with no full-size
temporary, and skips tensors whose width is not hidden_size.

Co-authored-by: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-15 21:40:16 +07:00
MOVZX 76df77c04e Merge branch 'main' into patches
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-15 21:16:39 +07:00
MOVZX ac305c44cb Merge branch 'main' into patches
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 17:41:13 +00:00
MOVZX 4a3ebc6b5b Merge branch 'main' into patches 2026-10-05 08:21:46 +00:00
MOVZX f83ecfac16 Merge remote-tracking branch 'upstream/main' into patches 2026-10-05 08:52:31 +07:00
MOVZX 85c31d47ad Merge remote-tracking branch 'upstream/main' into patches
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-03 22:06:10 +07:00
MOVZX 2877121b2d Merge remote-tracking branch 'upstream/main' into patches
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-03 21:59:03 +07:00
MOVZXandHinata ae0350c0de [Spec Decode][MRV2] RecoverSSM for Qwen GDN and PLE short conv (PR #58863)
Intake of open upstream PR vllm-project/vllm#58863. Active only with
VLLM_USE_V2_MODEL_RUNNER=1; inert otherwise.

Speculative decoding on Gated DeltaNet snapshotted the full recurrent
state after every draft position and reserved one Mamba block per draft
position. On a 24 GB card that is reserved VRAM we never use, plus
per-step copy traffic.

RecoverSSM, the protocol Kimi-K3 KDA already uses, replaces this:
verify runs from a read-only checkpoint and stores a small per-token
replay record [correction (V), normalized key (K), gate (1)] in fp32;
commit replays only the accepted tokens and writes the recurrent state
once, plus the block-boundary state in align mode. The PLE dilated
short conv compacts its accepted window with the same plan.

Conflict in vllm/models/qwen4_exp/nvidia/mtp.py: our get_top_tokens()
work put LocalArgmaxMixin on Qwen4ExpMTP, the PR adds SupportsReplaySSM
to the same base list. Kept both.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-09-29 13:39:09 +07:00
MOVZXandHinata a1d3c2cf1e [Spec Decode] Fused draft metadata updates for the QSA side cache (PR #58449)
Intake of open upstream PR vllm-project/vllm#58449.

With MTP num_speculative_tokens > 1, the V2 autoregressive speculator
reuses one attention-metadata build across draft steps only when every
group's builder sets supports_draft_decode_metadata_update. The QSA
side-cache builder (QWEN4_EXP_EXP_QSA_STATE) did not, so every draft
step rebuilt attention metadata for the whole model.

QSAMetadataBuilder now implements update_draft_decode_metadata: it
re-launches the kernel with in-place seq_lens updates, so one metadata
build serves every draft step.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-09-29 13:36:09 +07:00
MOVZXandHinata cb275fdf20 [Qwen4Exp] Fix MTP buffer placement, config parsing, Triton warmup (PR #56742)
Intake of open upstream PR vllm-project/vllm#56742. Every rig here runs
MTP on both Qwen3.8 models, so these three bugs sit on our hot path.

The MTP hidden buffer was allocated with torch.empty and no device, so
it followed the ambient default instead of device_config.device; on a
CPU-default process the draft buffer lands on the wrong device. Both
NVIDIA and AMD backends now call make_mtp_hidden_buffer() in
vllm/models/qwen4_exp/common/mtp.py, which pins it to the configured
device.

Config parsing now accepts configs carrying M-RoPE keys and the
Transformers spelling of layer_types; both previously failed at load.

Triton warmup: Qwen4Exp was missing from the Qwen warmup dispatch, so
its Triton kernels compiled during the first served requests instead of
at startup.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-09-29 13:36:05 +07:00
MOVZX a63dca4afb Merge remote-tracking branch 'upstream/main' into patches
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-09-29 12:54:18 +07:00
MOVZX 76af434852 Merge remote-tracking branch 'upstream/main' into patches 2026-09-28 12:06:52 +07:00
lesj0610 521235d6df Merge remote-tracking branch 'origin/main' into sync/lesj/qwen4exp-bugfixes
Signed-off-by: lesj0610 <lesj0610@users.noreply.github.com>
2026-09-27 18:25:21 +09:00
5567cc1b25 [Mamba] RecoverSSM align: keep a boundary-ending state in its block
A speculative window whose accepted tokens end exactly on a recurrent block
boundary selected the following block-table column for the committed state
(commit plan) and recorded it as the running column (postprocess). That
column can still be unallocated, so the next step reads an uninitialized
page. Select the block that holds the last committed token, matching the
(n - 1) // block convention used elsewhere in align mode. Applies to
Kimi-K3 KDA and to the Qwen GDN/PLE RecoverSSM paths that share the plan.

Fix and test from local-inference-lab/vllm a02f585d (Martin Vit, root cause
reported by astra); the GDN/PLE align tests now expect the same convention.

Signed-off-by: Jürgen Schmied <juergenschmied70@gmail.com>
Co-authored-by: Martin Vit <martin@voipmonitor.org>
Co-authored-by: Claude <noreply@anthropic.com>
2026-09-27 11:12:40 +02:00
Jürgen Schmied ed2a9665ea Merge branch 'main' into gdn-recoverssm 2026-09-27 08:01:32 +02:00
MOVZX 5da84df6b7 [BugFix] Honor drop_eagle_block in MambaManager
On a hybrid model (full-attention + Mamba/GDN layers, e.g. Qwen3.6-35B-A3B),
enabling MTP/EAGLE speculative decoding together with prefix caching corrupts
the recurrent state, and the server serves wrong answers. #43559 reports
tool-eval accuracy dropping from ~90% to ~50%. Nothing raises an exception
and nothing shows up in the logs, so an operator has no signal that anything
is wrong, and prefix caching then spreads it, because the corrupted state is
hash-reachable and every later request sharing that prefix restores from it.

The root cause is the find_longest_cache_hit method on MambaManager,
at vllm/v1/core/single_type_kv_cache_manager.py:1049 on main, which takes
drop_eagle_block in its signature and never acts on it. Its body searches
right-to-left for the last cached block, builds [null]*i + [cached], and
breaks, referencing neither drop_eagle_block nor .pop(.

Signed-off-by: MOVZX <movzx@yahoo.com>
2026-09-27 10:45:10 +07:00
MOVZX 5e6656e7a0 [Bugfix][Spec Decode] Add get_top_tokens() to Qwen4ExpMTP draft model
use_local_argmax_reduction hard-fails at worker startup when the draft
model lacks get_top_tokens(): the V2 speculator probes for the method
in _validate_local_argmax_reduction and raises, with no fallback to
the full-vocab logits path. Qwen4ExpMTP (nvidia and amd variants) has
compute_logits but no get_top_tokens, so any Qwen4Exp MTP drafter with
the flag enabled crashed both TP ranks during load_model.

Qwen4ExpMTP owns a flat self.lm_head and its compute_logits is exactly
self.logits_processor(self.lm_head, hidden_states), the same shape as
Qwen3_5MTP, so LocalArgmaxMixin applies unchanged: vocab-parallel
argmax with a (value, index) gather, O(2*tp_size) vs O(vocab_size)
communication per draft token. No D2T remapping needed since the
draft vocabulary equals the target vocabulary.

Signed-off-by: MOVZX <movzx@yahoo.com>
2026-09-27 10:31:47 +07:00
MOVZXandHinata 43fbd6510b [Qwen4Exp] Serve PLE rows from the loader's mapped checkpoint shards (PR #58815)
Adds an opt-in backend, --engram-config '{"host_file_gather": true}', that
serves Qwen4Exp PLE rows from the checkpoint files through the page cache
instead of allocating the 47.7 GiB table on GPU or in pinned host memory.

- Qwen4ExpPLEFileGatherEmbedding allocates no table rows; load_weights keeps
  the default loader's zero-copy safetensors views of each shard. At bind
  (post-load hook) every view must be file-backed per /proc/self/maps (HF
  cache symlinks work; heap/pinned/anonymous memory is rejected) and the
  shards must cover the whole table.
- Each step, Qwen4ExpModelState.prepare_inputs computes the PLE n-gram ids,
  copies them to pinned host memory, syncs the stream once, issues
  POSIX_FADV_WILLNEED over the coalesced page ranges of the step's rows,
  gathers the rows on the host, and copies them into a fixed device staging
  buffer. The forward reads only the staging buffer, so FULL CUDA graph
  capture is unaffected.
- Rows stay in the checkpoint dtype (FP8 table stays FP8, dequantized on the
  GPU); dummy loads serve zeros; weight reload is rejected; takes precedence
  over cpu_offload; rejected on ROCm and with elastic EP.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-09-27 09:54:42 +07:00
MOVZXandHinata 5af48b498c [Qwen3.5] Build MTP head unquantized when checkpoint ships it dense (PR #58813)
Fixes #58807. A compressed-tensors pack-quantized checkpoint that ships
its MTP head dense (plain mtp.*.weight names in a separate bf16 shard)
crashes at weight load: Qwen3_5MultiTokenPredictor builds its Linears
from the target quant config, so the loader expects packed names
(weight_packed) the checkpoint never had, and the weight-map walk dies
with "no module or parameter named fc.weight".

Detect the dense MTP head from the checkpoint itself: read the
safetensors index via the same repo-file channel used for recipe.yaml
(local cache first, Hub download as fallback). If any mtp.* tensor
carries a quantization artifact suffix the head is properly quantized
and nothing changes; if every mtp.* entry is a plain .weight the whole
predictor builds unquantized (fc and layers), mirroring the existing
modelopt_fp4 and GPTQ dynamic carve-outs. Single-file checkpoints have
no index and keep current behavior; NVFP4 checkpoints are unaffected
(their mtp.layers.* carry artifacts).

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-09-27 09:54:24 +07:00
MOVZXandHinata 835c44af15 [Qwen4Exp] Reinterpret FP8 E4M3 PLE as int8 for SM86 pinned-host lookup (PR #58856)
The pinned-host PLE lookup Triton kernel only gathers and stores bytes,
but passing an fp8e4nv tensor to it fails on architectures without
native FP8 pointer support (e.g. SM86) with an error saying type
fp8e4nv is not supported (supported fp8 dtypes are fp8e4b15, fp8e5).

Since FP8 E4M3 and int8 both occupy one byte per element, reinterpret
the PLE weight view and the output tensor as int8 when the weight dtype
is float8_e4m3fn; the non-FP8 path is unchanged.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-09-27 09:54:05 +07:00
MOVZXandHinata 758cc4f9e1 [Qwen3.5] Pass quant_config and prefix to embed_tokens in Qwen3_5Model
Apply the qwen3_5_quantized_embedding patch: construct the embedding
through the model's quant config so quantized (e.g. fp8/int8)
embeddings are supported, and give it a stable prefix for weight
loading.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-09-27 09:53:21 +07:00
Jürgen SchmiedandClaude 03cae7837f [Spec Decode][Qwen4Exp] RecoverSSM for Qwen GDN and the PLE short conv
With --use-replayssm and speculative decoding, Qwen4Exp's Gated DeltaNet layers verify the
draft window from a read-only checkpoint and keep a per-token replay record
[correction(V), normalized key(K), gate(1)] in fp32. After sampling, one commit replays the
accepted tokens and writes the recurrent state once (plus the block-boundary state in align
mode), instead of snapshotting the state after every draft position. The PLE dilated short
conv follows the same protocol (accepted-suffix compaction). num_speculative_blocks = 0,
so no per-draft Mamba blocks are reserved.

- GDN verify/commit kernels: model_executor/layers/mamba/gdn/recoverssm_gdn.py
- metadata/backends: v1/attention/backends/gdn_recoverssm.py, ple_recoverssm.py
- the verify reads strided q/k/v views of the conv output and writes in place
- builders declare AttentionCGSupport.NEVER: FULL_AND_PIECEWISE falls back to PIECEWISE;
  the fallback now also counts breakable CUDA graphs (empty splitting_ops, with or
  without torch.compile) as piecewise
  attention instead of dropping to NONE
- the generic Mamba metadata builder skips the Mamba2 ReplaySSM ring-buffer workspaces
  when --use-replayssm selects RecoverSSM (the PLE short conv uses that builder)
- tests: GDN verify/commit vs the native kernels (fp32 and bf16 state, align boundaries,
  mixed and ragged batches, extreme gates), PLE vs token-by-token decode, config selection

Signed-off-by: Jürgen Schmied <juergenschmied70@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-09-27 01:17:24 +02:00
Jürgen SchmiedandClaude a80e41d690 [Mamba] Share RecoverSSM commit kernels and helpers across models
RecoverSSM's commit-plan and conv-window compaction kernels are model independent.
Move them from models/kimi_k3 into model_executor/layers/mamba/ops/recoverssm.py
(Kimi-K3 keeps its private names as import aliases; kernel source unchanged), add
mamba/recoverssm_utils.py (activation predicate, argument validation) and the
generic spec-row/request mapping helpers to v1/attention/backends/recoverssm_metadata.py.

CacheConfig.use_kda_recoverssm becomes use_recoverssm: the switch now selects RecoverSSM
for any model whose recurrent layers implement it, not only KDA.

No functional change for Kimi-K3.

Signed-off-by: Jürgen Schmied <juergenschmied70@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-09-27 00:07:04 +02:00
vr8vr8 848dbf89ec Merge branch 'main' into qwen4exp-qsa-fused-draft-metadata 2026-09-23 23:37:13 +03:00
bilikasandClaude Opus 5.5 31d78a21b9 [Spec Decode][Qwen3.8] Enable fused draft metadata updates for the QSA side cache
QSAMetadataBuilder did not set supports_draft_decode_metadata_update, so
MTP with num_speculative_tokens > 1 fell back to rebuilding attention
metadata between draft steps for the whole model.

Opt the builder in. update_draft_decode_metadata re-launches the builder's
own _build_qsa_metadata_kernel in place on the metadata's persistent
tensors, so draft-step token_to_req, logical_positions, visible_blocks,
slot_mapping and k_work_metadata come from the same code path as build().
The kernel launch is factored into _launch_qsa_metadata_kernel, and
QSAForwardMetadata carries common_slot_mapping and num_mapped_tokens so
the update does not read query_start_loc_cpu. The PyTorch fallback builder
keeps the rebuild path.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: bilikas <valdas@vbtech.eu>
2026-09-23 20:58:12 +03:00
lesj0610 045467c102 Merge remote-tracking branch 'origin/main' into sync/lesj/qwen4exp-bugfixes
Signed-off-by: lesj0610 <lesj0610@users.noreply.github.com>
2026-09-23 00:40:48 +09:00
lesj0610 51d5b3ac8a Merge remote-tracking branch 'origin/main' into sync/lesj/qwen4exp-bugfixes
Signed-off-by: lesj0610 <lesj0610@users.noreply.github.com>
2026-09-20 22:55:38 +09:00
lesj0610 5780c0125a Merge remote-tracking branch 'origin/main' into sync/lesj/qwen4exp-bugfixes
Signed-off-by: lesj0610 <lesj0610@users.noreply.github.com>
2026-09-20 09:27:57 +09:00
lesj0610 1eacb41e0f Merge remote-tracking branch 'origin/main' into sync/lesj/qwen4exp-bugfixes
Signed-off-by: lesj0610 <lesj0610@users.noreply.github.com>
2026-09-17 15:01:29 +09:00
lesj0610 b8adc7710e Merge remote-tracking branch 'origin/main' into sync/lesj/qwen4exp-bugfixes
Signed-off-by: lesj0610 <lesj0610@users.noreply.github.com>
2026-09-17 12:32:30 +09:00
lesj0610 77b8783b25 Merge remote-tracking branch 'origin/main' into sync/lesj/qwen4exp-bugfixes
Signed-off-by: lesj0610 <lesj0610@users.noreply.github.com>
2026-09-17 11:29:18 +09:00
lesj0610 6c5552776b Merge remote-tracking branch 'origin/main' into lesj/qwen4exp-bugfixes
Both sides edited the same import block in the Qwen4Exp NVIDIA and AMD
models and nothing else collided. This branch adds
`from ..common.mtp import make_mtp_hidden_buffer`, the helper the MTP
buffer extraction calls; upstream ec4a3a5370 widened the `..config`
import to `ATTENTION_LAYER_TYPES, QSA_LAYER_TYPE, Qwen4ExpConfig` for the
layer-type constants it introduced. The two changes are independent, so
the resolution keeps both lines. Every symbol still has a use outside the
import in both files.

Signed-off-by: lesj0610 <lesj0610@users.noreply.github.com>
2026-09-17 08:16:59 +09:00
lesj0610 e582b9f4a2 Merge remote-tracking branch 'origin/main' into sync/lesj/qwen4exp-bugfixes
Signed-off-by: lesj0610 <lesj0610@users.noreply.github.com>
2026-09-16 20:33:18 +09:00
lesj0610 21d2dd56d8 Merge remote-tracking branch 'origin/main' into sync/lesj/qwen4exp-bugfixes
Signed-off-by: lesj0610 <lesj0610@users.noreply.github.com>
2026-09-16 13:41:12 +09:00
lesj0610 a66a3e1b4e Merge remote-tracking branch 'origin/main' into sync/lesj/qwen4exp-bugfixes
Signed-off-by: lesj0610 <lesj0610@users.noreply.github.com>
2026-09-14 14:36:05 +09:00
lesj0610andClaude Opus 5 3b1911c0ca [Bugfix][Model] Fix Qwen4Exp MTP buffer placement, config parsing, and Triton warmup
Three initialization and config defects found serving Qwen4Exp, each with a
regression test.

**The MTP hidden buffer lands on the ambient device.** torch.empty for the
spec-decode hidden buffer takes no device, so it follows whatever device is
current rather than vllm_config.device_config.device. The allocation was
duplicated verbatim in the NVIDIA and AMD backends; it moves to
common/mtp.py so the two cannot drift, and the caller passes is_last_rank
rather than reaching for the process group from shared code.

**M-RoPE keys are reported as unrecognized.** Qwen4Exp drives M-RoPE through
mrope_section and mrope_interleaved, which transformers' rope validator does
not know for rope_type="default", so every load warns about a configuration
the model genuinely uses. Declare them at rope validation on the text config
and on the outer config that mirrors it; every other key still warns.

**transformers' layer_types spelling is unread.** Its Qwen4Exp config
canonicalizes "full_attention" entries to "qwen_sparse_attention" in memory,
so checkpoints re-exported through it serialize that name. vLLM keys QSA off
"full_attention" plus indexer_n_heads, so map the spelling back.

**Qwen4Exp is missing from the Triton warmup set.** Its GDN prefill and QSA
run on other backends, but the kernels this warmup covers -- the gated RMS
norm, the causal conv, the post-conv preparation, the decode state update --
are Triton whichever prefill kernel is chosen, so they compile on the first
prompt instead. qwen4_exp_mtp stays out: kernel_warmup passes the worker's
own model_config, which is the target model's.

tests/models/qwen4_exp/test_config.py: 15 passed.
tests/model_executor/test_qwen_triton_warmup.py: 6 passed.
ruff check, ruff format --check and mypy (Python 3.10) report what they
already report on the base commit.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: lesj0610 <lesj0610@users.noreply.github.com>
2026-09-14 07:25:01 +09:00
625 changed files with 16871 additions and 67898 deletions
@@ -31,9 +31,6 @@ Requests: `[N*]Q[@K][c][:D]`, comma-separated, per DP rank.
| `2048@10240` | prefill after an 8192-token prefix-cache hit | | `2048@10240` | prefill after an 8192-token prefix-cache hit |
| `2048@10240c` | chunked prefill: the prefix came from earlier chunks | | `2048@10240c` | chunked prefill: the prefix came from earlier chunks |
Verify cases require `--speculative-config` with `num_speculative_tokens`
at least as large as the case's draft count; insufficient capacity is rejected.
Hit vs chunk only differs on models with SWA bounded replay (DSV4.1): a hit Hit vs chunk only differs on models with SWA bounded replay (DSV4.1): a hit
also recomputes the last window (128 tokens) of the prefix. also recomputes the last window (128 tokens) of the prefix.
@@ -51,46 +48,6 @@ Key flags (plus all `EngineArgs`):
Read `median_us` (each replay is already the max over ranks). Read `median_us` (each replay is already the max over ranks).
## Kernel breakdown
Add `--profile-dir DIR` to see where the time goes. After timing, each case is
replayed 3 more times under `torch.profiler`; every rank writes
`DIR/<case>.rank<r>.kernels.json` (GPU time per kernel per forward, sorted)
and `DIR/<case>.rank<r>.trace.json.gz` (open in Perfetto). Rank 0's top
kernels are printed and included in `--output-json`.
```bash
python -m vllm.benchmarks.forward.profile DIR/decode.rank0.kernels.json
# Before/after a change, same case: per-kernel deltas, largest first.
python -m vllm.benchmarks.forward.profile base/decode.rank0.kernels.json new/decode.rank0.kernels.json
```
- Kernels are grouped by name without template or function arguments; the
JSON keeps one full name per group in `example`.
- Only the timed forward is included: the in-graph KV restore and rank
rendezvous are excluded (`pre_forward_us`). `span_us` (first to last
forward kernel) should be close to `median_us`, a bit higher from profiler
overhead; `busy_us` can exceed it when streams overlap.
- Compare the same rank across runs: under EP, ranks differ (e.g. collective
wait time shows up inside communication kernels).
For Nsight Systems (NVTX, NCCL, CPU side, GPU metrics), skip `--profile-dir`:
the two can't share CUPTI. Each case's 3 profiled replays run between
`cudaProfilerStart`/`Stop` in an NVTX range `forward/<case>`, so a capture range
records only them, one report per case (and rank):
```bash
nsys profile --capture-range=cudaProfilerApi --capture-range-end=repeat \
--cuda-graph-trace=node -o prof vllm bench forward ... # prof.1, prof.2, ...
torchrun --nproc-per-node=4 --no-python nsys profile ... -o prof_rank%q{RANK} \
python -m vllm.entrypoints.cli.main bench forward ...
nsys stats --report cuda_gpu_kern_sum --format csv -o kern prof.1.nsys-rep
```
Keep `--capture-range-end=repeat`: nsys's default stops after the first case
and leaves the bench hanging. `nsys stats` also counts the KV restore copies
that the JSON summary excludes.
## Caveats ## Caveats
- **Dummy weights lie for data-dependent kernels.** Dummy gates route every - **Dummy weights lie for data-dependent kernels.** Dummy gates route every
+5 -11
View File
@@ -5,12 +5,9 @@ description: Find vLLM deployment recipes and configs for LLMs and Hugging Face
# Model Recipes # Model Recipes
Model serving recipes are maintained in this repository's Model serving recipes are maintained in the
[recipes/models/ directory](../../../recipes/models). Omni (image, video, [models/ directory of vllm-project/recipes](https://github.com/vllm-project/recipes/tree/main/models).
audio) recipes stay in vllm-project/recipes for now (moving to vllm-omni), The published catalog is at
and Ascend NPU content is inline in these recipes (moving to vllm-ascend
overlays). The published catalog, rendered by
[vllm-project/recipes](https://github.com/vllm-project/recipes), is at
[recipes.vllm.ai](https://recipes.vllm.ai). [recipes.vllm.ai](https://recipes.vllm.ai).
## Usage notes ## Usage notes
@@ -149,9 +146,6 @@ uploads, untested hardware claims, or non-standard flags that only work for one
setup. Before opening a pull request, confirm that the user ran the setup. Before opening a pull request, confirm that the user ran the
configuration and can share the results. configuration and can share the results.
Edit the YAML under [recipes/models/](../../../recipes/models) in this Follow the recipes repository's
repository, following the recipes repository's
[add-recipe skill](https://github.com/vllm-project/recipes/blob/main/.agents/skills/add-recipe/SKILL.md) [add-recipe skill](https://github.com/vllm-project/recipes/blob/main/.agents/skills/add-recipe/SKILL.md)
for the schema. Validate and preview it from a checkout of in a checkout of `vllm-project/recipes` or the user's intended fork.
`vllm-project/recipes` pointed at this directory, as
[recipes/README.md](../../../recipes/README.md) describes.
+479 -214
View File
@@ -703,6 +703,385 @@ steps:
DOCKER_BUILDKIT: "1" DOCKER_BUILDKIT: "1"
DOCKERHUB_USERNAME: "vllmbot" DOCKERHUB_USERNAME: "vllmbot"
# =============================================================================
# ROCm Release Pipeline (x86_64 only)
# =============================================================================
#
# vLLM version is determined by the Buildkite checkout (like CUDA pipeline).
# To build a specific version, trigger the build from that branch/tag.
#
# Environment variables for ROCm builds (set via Buildkite UI or schedule):
#
# Note: ROCm version is determined by BASE_IMAGE in docker/Dockerfile.rocm_base
#
# =============================================================================
- group: "Build ROCm Wheel / Image "
key: "build-rocm-wheel-image"
depends_on: ~
steps:
# ROCm Job 1: Build ROCm Base Wheels (with S3 caching)
- label: ":rocm: Build ROCm Base Image & Wheels"
id: build-rocm-base-wheels
depends_on: ~
agents:
queue: cpu_queue_release
commands:
- |
set -euo pipefail
# Generate cache key
CACHE_KEY=$$(.buildkite/scripts/cache-rocm-base-wheels.sh key)
ECR_CACHE_TAG="public.ecr.aws/q9t5s3a7/vllm-release-repo:$${CACHE_KEY}-rocm-base"
echo "========================================"
echo "ROCm Base Build Configuration"
echo "========================================"
echo " CACHE_KEY: $${CACHE_KEY}"
echo " ECR_CACHE_TAG: $${ECR_CACHE_TAG}"
echo "========================================"
# Login to ECR
aws ecr-public get-login-password --region us-east-1 | \
docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7
IMAGE_EXISTS=false
WHEELS_EXIST=false
# Check ECR for Docker image
if docker manifest inspect "$${ECR_CACHE_TAG}" > /dev/null 2>&1; then
IMAGE_EXISTS=true
echo "ECR image cache HIT"
fi
# Check S3 for wheels
WHEEL_CACHE_STATUS=$(.buildkite/scripts/cache-rocm-base-wheels.sh check)
if [ "$${WHEEL_CACHE_STATUS}" = "hit" ]; then
WHEELS_EXIST=true
echo "S3 wheels cache HIT"
fi
# Scenario 1: Both cached (best case)
if [ "$${IMAGE_EXISTS}" = "true" ] && [ "$${WHEELS_EXIST}" = "true" ]; then
echo ""
echo "FULL CACHE HIT - Reusing both image and wheels"
echo ""
# Download wheels
.buildkite/scripts/cache-rocm-base-wheels.sh download
# Save ECR tag for downstream jobs
buildkite-agent meta-data set "rocm-base-image-tag" "$${ECR_CACHE_TAG}"
# Scenario 2: Full rebuild needed
else
echo ""
echo " CACHE MISS - Building from scratch..."
echo ""
# Build full base image and push to ECR
DOCKER_BUILDKIT=1 docker buildx build \
--file docker/Dockerfile.rocm_base \
--tag "$${ECR_CACHE_TAG}" \
--build-arg USE_SCCACHE=1 \
--build-arg SCCACHE_BUCKET_NAME=vllm-build-sccache \
--build-arg SCCACHE_REGION_NAME=us-west-2 \
--build-arg SCCACHE_S3_NO_CREDENTIALS=0 \
--push \
.
# Build wheel extraction stage
DOCKER_BUILDKIT=1 docker buildx build \
--file docker/Dockerfile.rocm_base \
--tag rocm-base-debs:$${BUILDKITE_BUILD_NUMBER} \
--target debs_wheel_release \
--build-arg USE_SCCACHE=1 \
--build-arg SCCACHE_BUCKET_NAME=vllm-build-sccache \
--build-arg SCCACHE_REGION_NAME=us-west-2 \
--build-arg SCCACHE_S3_NO_CREDENTIALS=0 \
--load \
.
# Extract and upload wheels
mkdir -p artifacts/rocm-base-wheels
cid=$(docker create rocm-base-debs:$${BUILDKITE_BUILD_NUMBER})
docker cp $${cid}:/app/debs/. artifacts/rocm-base-wheels/
docker rm $${cid}
.buildkite/scripts/cache-rocm-base-wheels.sh upload
# Cache base docker image to ECR
docker push "$${ECR_CACHE_TAG}"
buildkite-agent meta-data set "rocm-base-image-tag" "$${ECR_CACHE_TAG}"
echo ""
echo " Build complete - Image and wheels cached"
fi
artifact_paths:
- "artifacts/rocm-base-wheels/*.whl"
env:
DOCKER_BUILDKIT: "1"
S3_BUCKET: "vllm-wheels"
# ROCm Job 2: Build vLLM ROCm Wheel
- label: ":python: Build vLLM ROCm Wheel - x86_64"
id: build-rocm-vllm-wheel
depends_on:
- step: build-rocm-base-wheels
allow_failure: false
agents:
queue: cpu_queue_release
timeout_in_minutes: 180
commands:
# Download artifacts and prepare Docker image
- |
set -euo pipefail
# Ensure git tags are up-to-date (Buildkite's default fetch doesn't update tags)
# This fixes version detection when tags are moved/force-pushed
echo "Fetching latest tags from origin..."
git fetch --tags --force origin
# Log tag information for debugging version detection
echo "========================================"
echo "Git Tag Verification"
echo "========================================"
echo "Current HEAD: $(git rev-parse HEAD)"
echo "git describe --tags: $(git describe --tags 2>/dev/null || echo 'No tags found')"
echo ""
echo "Recent tags (pointing to commits near HEAD):"
git tag -l --sort=-creatordate | head -5
echo "setuptools_scm version detection:"
pip install -q setuptools_scm 2>/dev/null || true
python3 -c "import setuptools_scm; print(' Detected version:', setuptools_scm.get_version())" 2>/dev/null || echo " (setuptools_scm not available in this environment)"
echo "========================================"
# Download wheel artifacts from current build
echo "Downloading wheel artifacts from current build"
buildkite-agent artifact download "artifacts/rocm-base-wheels/*.whl" .
# Get ECR image tag from metadata (set by build-rocm-base-wheels)
ECR_IMAGE_TAG="$$(buildkite-agent meta-data get rocm-base-image-tag 2>/dev/null || echo '')"
if [ -z "$${ECR_IMAGE_TAG}" ]; then
echo "ERROR: rocm-base-image-tag metadata not found"
echo "This should have been set by the build-rocm-base-wheels job"
exit 1
fi
echo "Pulling base Docker image from ECR: $${ECR_IMAGE_TAG}"
# Login to ECR
aws ecr-public get-login-password --region us-east-1 | \
docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7
# Pull base Docker image from ECR
docker pull "$${ECR_IMAGE_TAG}"
echo "Loaded base image: $${ECR_IMAGE_TAG}"
# Prepare base wheels for Docker build context
mkdir -p docker/context/base-wheels
touch docker/context/base-wheels/.keep
cp artifacts/rocm-base-wheels/*.whl docker/context/base-wheels/
echo "Base wheels for vLLM build:"
ls -lh docker/context/base-wheels/
echo "========================================"
echo "Building vLLM wheel with:"
echo " BUILDKITE_COMMIT: $${BUILDKITE_COMMIT}"
echo " BUILDKITE_BRANCH: $${BUILDKITE_BRANCH}"
echo " BASE_IMAGE: $${ECR_IMAGE_TAG}"
echo "========================================"
# Build vLLM wheel using local checkout (REMOTE_VLLM=0)
DOCKER_BUILDKIT=1 docker build \
--file docker/Dockerfile.rocm \
--target export_vllm_wheel_release \
--output type=local,dest=rocm-dist \
--build-arg BASE_IMAGE="$${ECR_IMAGE_TAG}" \
--build-arg REMOTE_VLLM=0 \
--build-arg GIT_REPO_CHECK=1 \
--build-arg USE_SCCACHE=1 \
--build-arg SCCACHE_BUCKET_NAME=vllm-build-sccache \
--build-arg SCCACHE_REGION_NAME=us-west-2 \
--build-arg SCCACHE_S3_NO_CREDENTIALS=0 \
.
echo "Built vLLM wheel:"
ls -lh rocm-dist/*.whl
# Copy wheel to artifacts directory
mkdir -p artifacts/rocm-vllm-wheel
cp rocm-dist/*.whl artifacts/rocm-vllm-wheel/
echo "Final vLLM wheel:"
ls -lh artifacts/rocm-vllm-wheel/
artifact_paths:
- "artifacts/rocm-vllm-wheel/*.whl"
env:
DOCKER_BUILDKIT: "1"
S3_BUCKET: "vllm-wheels"
# ROCm Job 3: Upload Wheels to S3
- label: ":s3: Upload ROCm Wheels to S3"
id: upload-rocm-wheels
depends_on:
- step: build-rocm-vllm-wheel
allow_failure: false
agents:
queue: cpu_queue_release
timeout_in_minutes: 60
commands:
# Download all wheel artifacts and run upload
- |
set -euo pipefail
# Download artifacts from current build
echo "Downloading artifacts from current build"
buildkite-agent artifact download "artifacts/rocm-base-wheels/*.whl" .
buildkite-agent artifact download "artifacts/rocm-vllm-wheel/*.whl" .
# # Run upload script
bash .buildkite/scripts/upload-rocm-wheels.sh
env:
DOCKER_BUILDKIT: "1"
S3_BUCKET: "vllm-wheels"
# ROCm Job 4: Annotate ROCm Wheel Release
- label: ":memo: Annotate ROCm wheel release"
id: annotate-rocm-release
depends_on:
- upload-rocm-wheels
agents:
queue: cpu_queue_release
commands:
- "bash .buildkite/scripts/annotate-rocm-release.sh"
env:
S3_BUCKET: "vllm-wheels"
# ROCm Job 5: Generate Root Index for ROCm Wheels (for release only)
# This is the job to create https://wheels.vllm.ai/rocm/ index allowing
# users to install with `uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/`
- block: "Generate Root Index for ROCm Wheels for Release"
key: block-generate-root-index-rocm-wheels
depends_on: upload-rocm-wheels
- label: ":package: Generate Root Index for ROCm Wheels for Release"
depends_on: block-generate-root-index-rocm-wheels
id: generate-root-index-rocm-wheels
agents:
queue: cpu_queue_release
commands:
- "bash tools/vllm-rocm/generate-rocm-wheels-root-index.sh"
env:
S3_BUCKET: "vllm-wheels"
VARIANT: "rocm723"
# ROCm Job 6: Build ROCm Release Docker Image
- label: ":docker: Build release image - x86_64 - ROCm"
id: build-rocm-release-image
depends_on:
- step: block-build-release-images
allow_failure: true
- step: build-rocm-base-wheels
allow_failure: false
agents:
queue: cpu_queue_release
timeout_in_minutes: 60
commands:
- |
set -euo pipefail
# Login to ECR
aws ecr-public get-login-password --region us-east-1 | \
docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7
# Get ECR image tag from metadata (set by build-rocm-base-wheels)
ECR_IMAGE_TAG="$$(buildkite-agent meta-data get rocm-base-image-tag 2>/dev/null || echo '')"
if [ -z "$${ECR_IMAGE_TAG}" ]; then
echo "ERROR: rocm-base-image-tag metadata not found"
echo "This should have been set by the build-rocm-base-wheels job"
exit 1
fi
echo "Pulling base Docker image from ECR: $${ECR_IMAGE_TAG}"
# Pull base Docker image from ECR
docker pull "$${ECR_IMAGE_TAG}"
echo "Loaded base image: $${ECR_IMAGE_TAG}"
# Pass the base image ECR tag to downstream steps (nightly publish)
buildkite-agent meta-data set "rocm-base-ecr-tag" "$${ECR_IMAGE_TAG}"
echo "========================================"
echo "Building vLLM ROCm release image with:"
echo " BASE_IMAGE: $${ECR_IMAGE_TAG}"
echo " BUILDKITE_COMMIT: $${BUILDKITE_COMMIT}"
echo "========================================"
# Build vLLM ROCm release image using cached base
DOCKER_BUILDKIT=1 docker build \
--build-arg max_jobs=16 \
--build-arg BASE_IMAGE="$${ECR_IMAGE_TAG}" \
--build-arg USE_SCCACHE=1 \
--build-arg SCCACHE_BUCKET_NAME=vllm-build-sccache \
--build-arg SCCACHE_REGION_NAME=us-west-2 \
--build-arg SCCACHE_S3_NO_CREDENTIALS=0 \
--build-arg INSTALL_LMCACHE=true \
--tag public.ecr.aws/q9t5s3a7/vllm-release-repo:$${BUILDKITE_COMMIT}-rocm \
--target vllm-openai \
--progress plain \
-f docker/Dockerfile.rocm .
# Push to ECR
docker push public.ecr.aws/q9t5s3a7/vllm-release-repo:$${BUILDKITE_COMMIT}-rocm
echo ""
echo " Successfully built and pushed ROCm release image"
echo " Image: public.ecr.aws/q9t5s3a7/vllm-release-repo:$${BUILDKITE_COMMIT}-rocm"
echo ""
env:
DOCKER_BUILDKIT: "1"
S3_BUCKET: "vllm-wheels"
- label: "Publish nightly XPU image to DockerHub"
depends_on:
- create-manifest-xpu
if: build.env("NIGHTLY") == "1"
agents:
queue: small_cpu_queue_release
commands:
- "bash .buildkite/scripts/xpu/push-nightly-builds-xpu.sh"
- "bash .buildkite/scripts/cleanup-nightly-builds.sh nightly- vllm/vllm-openai-xpu"
plugins:
- docker-login#v3.0.0:
username: vllmbot
password-env: DOCKERHUB_TOKEN
env:
DOCKER_BUILDKIT: "1"
DOCKERHUB_USERNAME: "vllmbot"
- label: "Publish nightly ROCm image to DockerHub"
depends_on:
- build-rocm-release-image
if: build.env("NIGHTLY") == "1"
agents:
queue: small_cpu_queue_release
commands:
- "bash .buildkite/scripts/push-nightly-builds-rocm.sh"
# Clean up old nightly builds (keep only last 14)
- "bash .buildkite/scripts/cleanup-nightly-builds.sh nightly- vllm/vllm-openai-rocm"
- "bash .buildkite/scripts/cleanup-nightly-builds.sh base-nightly- vllm/vllm-openai-rocm"
plugins:
- docker-login#v3.0.0:
username: vllmbot
password-env: DOCKERHUB_TOKEN
env:
DOCKER_BUILDKIT: "1"
DOCKERHUB_USERNAME: "vllmbot"
- trigger: "perf-eval" - trigger: "perf-eval"
label: "Trigger perf-eval (CUDA)" label: "Trigger perf-eval (CUDA)"
key: trigger-perf-eval-cuda key: trigger-perf-eval-cuda
@@ -724,7 +1103,7 @@ steps:
key: trigger-perf-eval-rocm key: trigger-perf-eval-rocm
if: build.env("NIGHTLY") == "1" || build.env("RUN_PERF_EVAL") == "1" if: build.env("NIGHTLY") == "1" || build.env("RUN_PERF_EVAL") == "1"
depends_on: depends_on:
- build-rocm100-release-image - build-rocm-release-image
async: true async: true
soft_fail: true soft_fail: true
build: build:
@@ -733,250 +1112,138 @@ steps:
env: env:
NIGHTLY: 1 # run all perf-eval workloads NIGHTLY: 1 # run all perf-eval workloads
VLLM_COMMIT: "$BUILDKITE_COMMIT" VLLM_COMMIT: "$BUILDKITE_COMMIT"
VLLM_IMAGE_ROCM: "public.ecr.aws/q9t5s3a7/vllm-release-repo:$BUILDKITE_COMMIT-rocm100" VLLM_IMAGE_ROCM: "public.ecr.aws/q9t5s3a7/vllm-release-repo:$BUILDKITE_COMMIT-rocm"
# ============================================================================= # =============================================================================
# ROCm Release Pipelines (x86_64 only) # ROCk Release Pipeline (x86_64 only)
# ============================================================================= # =============================================================================
# #
# One group per ROCm stack, identified by its base Dockerfile (ROCM_BASE_DOCKERFILE). # Same shape as the ROCm pipeline above (Jobs 1 and 6 only), but built from
# The scripts derive the ROCm version, wheel variant, cache keys and image tags from # docker/Dockerfile.rock_base and docker/Dockerfile.rock, which install the
# it (.buildkite/scripts/rocm/stack.sh), so both groups share one set of step # ROCm SDK from TheRock wheels instead of the ROCm apt repos.
# definitions, anchored on the ROCm 10.0 group.
# #
# - ROCm 10.0 (TheRock, docker/Dockerfile.rocm{,_base}) is the default: it owns the # Note: ROCm SDK version is determined by ROCM_SDK_VERSION in
# :nightly / :latest images and the root wheel index. # docker/Dockerfile.rock_base
# - ROCm 7.2 (docker/Dockerfile.rocm_72{,_base}) is kept for a transition period as
# rocm72 images and wheels.
# #
# vLLM version is determined by the Buildkite checkout (like the CUDA pipeline).
# ============================================================================= # =============================================================================
- group: "Build ROCm 10.0 Wheel / Image" - group: "Build ROCk Image"
key: "build-rocm100" key: "build-rock-image"
depends_on: ~ depends_on: ~
steps: steps:
- label: ":rocm: Build ROCm 10.0 Base Image & Wheels" # ROCk Job 1: Build ROCk Base Image (with ECR caching)
id: build-rocm100-base - label: ":rocm: Build ROCk Base Image"
id: build-rock-base-image
depends_on: ~ depends_on: ~
agents: &rocm_build_agents agents:
queue: cpu_queue_release queue: cpu_queue_release
timeout_in_minutes: 360 timeout_in_minutes: 360
env: &rocm100_env
ROCM_BASE_DOCKERFILE: "docker/Dockerfile.rocm_base"
ROCM_DEFAULT_ARG: "--default"
DOCKER_BUILDKIT: "1"
S3_BUCKET: "vllm-wheels"
DOCKERHUB_USERNAME: "vllmbot"
commands: &rocm_base_commands
- |
set -euo pipefail
source .buildkite/scripts/rocm/stack.sh "$${ROCM_BASE_DOCKERFILE}"
SCCACHE_ARGS=(--build-arg USE_SCCACHE=1 --build-arg SCCACHE_BUCKET_NAME=vllm-build-sccache
--build-arg SCCACHE_REGION_NAME=us-west-2 --build-arg SCCACHE_S3_NO_CREDENTIALS=0)
aws ecr-public get-login-password --region us-east-1 | \
docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7
# Base image, cached in ECR by the base Dockerfile's hash
if docker manifest inspect "$${ROCM_STACK_ECR_BASE}" > /dev/null 2>&1; then
echo "ECR image cache HIT: $${ROCM_STACK_ECR_BASE}"
else
docker buildx build --file "$${ROCM_BASE_DOCKERFILE}" --tag "$${ROCM_STACK_ECR_BASE}" \
"$${SCCACHE_ARGS[@]}" --push .
fi
# Base wheels, cached in S3 (see cache-rocm-base-wheels.sh)
CACHE=.buildkite/scripts/cache-rocm-base-wheels.sh
if [ "$$($${CACHE} "$${ROCM_BASE_DOCKERFILE}" check)" = "hit" ]; then
$${CACHE} "$${ROCM_BASE_DOCKERFILE}" download
else
DEBS="rocm-base-debs:$${BUILDKITE_BUILD_NUMBER}-$${ROCM_STACK_VARIANT}"
docker buildx build --file "$${ROCM_BASE_DOCKERFILE}" --tag "$${DEBS}" \
--target debs_wheel_release "$${SCCACHE_ARGS[@]}" --load .
mkdir -p "$${ROCM_STACK_BASE_WHEELS_DIR}"
cid=$$(docker create "$${DEBS}")
docker cp "$${cid}":/app/debs/. "$${ROCM_STACK_BASE_WHEELS_DIR}/"
docker rm "$${cid}"
$${CACHE} "$${ROCM_BASE_DOCKERFILE}" upload
fi
buildkite-agent artifact upload "$${ROCM_STACK_BASE_WHEELS_DIR}/*.whl"
- label: ":python: Build vLLM ROCm 10.0 Wheel - x86_64"
id: build-rocm100-vllm-wheel
depends_on: build-rocm100-base
agents: *rocm_build_agents
timeout_in_minutes: 180
env: *rocm100_env
commands: &rocm_vllm_wheel_commands
- |
set -euo pipefail
source .buildkite/scripts/rocm/stack.sh "$${ROCM_BASE_DOCKERFILE}"
# Buildkite's default fetch doesn't update moved tags; version detection needs them
git fetch --tags --force origin
buildkite-agent artifact download "$${ROCM_STACK_BASE_WHEELS_DIR}/*.whl" .
aws ecr-public get-login-password --region us-east-1 | \
docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7
docker pull "$${ROCM_STACK_ECR_BASE}"
mkdir -p docker/context/base-wheels
touch docker/context/base-wheels/.keep
cp "$${ROCM_STACK_BASE_WHEELS_DIR}"/*.whl docker/context/base-wheels/
docker build --file "$${ROCM_STACK_DOCKERFILE}" --target export_vllm_wheel_release \
--output type=local,dest=rocm-dist \
--build-arg BASE_IMAGE="$${ROCM_STACK_ECR_BASE}" --build-arg REMOTE_VLLM=0 \
--build-arg GIT_REPO_CHECK=1 --build-arg USE_SCCACHE=1 \
--build-arg SCCACHE_BUCKET_NAME=vllm-build-sccache \
--build-arg SCCACHE_REGION_NAME=us-west-2 --build-arg SCCACHE_S3_NO_CREDENTIALS=0 .
mkdir -p "$${ROCM_STACK_VLLM_WHEEL_DIR}"
cp rocm-dist/*.whl "$${ROCM_STACK_VLLM_WHEEL_DIR}/"
buildkite-agent artifact upload "$${ROCM_STACK_VLLM_WHEEL_DIR}/*.whl"
- label: ":s3: Upload ROCm 10.0 Wheels to S3"
id: upload-rocm100-wheels
depends_on: build-rocm100-vllm-wheel
agents: *rocm_build_agents
timeout_in_minutes: 60
env: *rocm100_env
commands: &rocm_upload_commands
- |
set -euo pipefail
source .buildkite/scripts/rocm/stack.sh "$${ROCM_BASE_DOCKERFILE}"
buildkite-agent artifact download "$${ROCM_STACK_BASE_WHEELS_DIR}/*.whl" .
buildkite-agent artifact download "$${ROCM_STACK_VLLM_WHEEL_DIR}/*.whl" .
bash .buildkite/scripts/upload-rocm-wheels.sh "$${ROCM_BASE_DOCKERFILE}"
- label: ":memo: Annotate ROCm 10.0 wheel release"
id: annotate-rocm100-release
depends_on: upload-rocm100-wheels
agents: *rocm_build_agents
env: *rocm100_env
commands: &rocm_annotate_commands
- bash .buildkite/scripts/annotate-rocm-release.sh "$${ROCM_BASE_DOCKERFILE}" $${ROCM_DEFAULT_ARG}
# Generate https://wheels.vllm.ai/rocm/ (the default stack's variant), so users can
# `uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/` (release only)
- block: "Generate Root Index for ROCm Wheels for Release"
key: block-generate-root-index-rocm-wheels
depends_on:
- upload-rocm100-wheels
- upload-rocm72-wheels
- label: ":package: Generate Root Index for ROCm Wheels for Release"
depends_on: block-generate-root-index-rocm-wheels
id: generate-root-index-rocm-wheels
agents: *rocm_build_agents
env: *rocm100_env
commands: commands:
- "bash tools/vllm-rocm/generate-rocm-wheels-root-index.sh"
- label: ":docker: Build release image - x86_64 - ROCm 10.0"
id: build-rocm100-release-image
depends_on:
- step: block-build-release-images
allow_failure: true
- step: build-rocm100-base
allow_failure: false
agents: *rocm_build_agents
timeout_in_minutes: 180
env: *rocm100_env
commands: &rocm_release_image_commands
- | - |
set -euo pipefail set -euo pipefail
source .buildkite/scripts/rocm/stack.sh "$${ROCM_BASE_DOCKERFILE}"
# The SDK version, torch wheels and component branches all live in
# the Dockerfile, so its hash is the cache key.
CACHE_KEY=$$(sha256sum docker/Dockerfile.rock_base | cut -c1-16)
ECR_CACHE_TAG="public.ecr.aws/q9t5s3a7/vllm-release-repo:$${CACHE_KEY}-rock-base"
echo "========================================"
echo "ROCk Base Build Configuration"
echo "========================================"
echo " CACHE_KEY: $${CACHE_KEY}"
echo " ECR_CACHE_TAG: $${ECR_CACHE_TAG}"
echo "========================================"
# Login to ECR
aws ecr-public get-login-password --region us-east-1 | \ aws ecr-public get-login-password --region us-east-1 | \
docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7 docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7
docker pull "$${ROCM_STACK_ECR_BASE}"
docker build --file "$${ROCM_STACK_DOCKERFILE}" --target vllm-openai \
--tag "$${ROCM_STACK_ECR_IMAGE}" --progress plain \
--build-arg max_jobs=16 --build-arg BASE_IMAGE="$${ROCM_STACK_ECR_BASE}" \
--build-arg INSTALL_LMCACHE=true --build-arg USE_SCCACHE=1 \
--build-arg SCCACHE_BUCKET_NAME=vllm-build-sccache \
--build-arg SCCACHE_REGION_NAME=us-west-2 --build-arg SCCACHE_S3_NO_CREDENTIALS=0 .
docker push "$${ROCM_STACK_ECR_IMAGE}"
- label: "Publish nightly ROCm 10.0 image to DockerHub" if docker manifest inspect "$${ECR_CACHE_TAG}" > /dev/null 2>&1; then
id: publish-rocm100-nightly echo "ECR image cache HIT"
depends_on: build-rocm100-release-image else
if: build.env("NIGHTLY") == "1" echo "CACHE MISS - Building from scratch..."
agents: &rocm_publish_agents
queue: small_cpu_queue_release
env: *rocm100_env
commands: &rocm_publish_nightly_commands
- bash .buildkite/scripts/push-nightly-builds-rocm.sh "$${ROCM_BASE_DOCKERFILE}" $${ROCM_DEFAULT_ARG}
plugins: &rocm_dockerhub_login
- docker-login#v3.0.0:
username: vllmbot
password-env: DOCKERHUB_TOKEN
- group: "Build ROCm 7.2 Wheel / Image" DOCKER_BUILDKIT=1 docker buildx build \
key: "build-rocm72" --file docker/Dockerfile.rock_base \
depends_on: ~ --tag "$${ECR_CACHE_TAG}" \
steps: --build-arg USE_SCCACHE=1 \
- label: ":rocm: Build ROCm 7.2 Base Image & Wheels" --build-arg SCCACHE_BUCKET_NAME=vllm-build-sccache \
id: build-rocm72-base --build-arg SCCACHE_REGION_NAME=us-west-2 \
depends_on: ~ --build-arg SCCACHE_S3_NO_CREDENTIALS=0 \
agents: *rocm_build_agents --push \
timeout_in_minutes: 360 .
env: &rocm72_env fi
ROCM_BASE_DOCKERFILE: "docker/Dockerfile.rocm_72_base"
ROCM_DEFAULT_ARG: "" # Save ECR tag for downstream jobs
buildkite-agent meta-data set "rock-base-image-tag" "$${ECR_CACHE_TAG}"
env:
DOCKER_BUILDKIT: "1" DOCKER_BUILDKIT: "1"
S3_BUCKET: "vllm-wheels"
DOCKERHUB_USERNAME: "vllmbot"
commands: *rocm_base_commands
- label: ":python: Build vLLM ROCm 7.2 Wheel - x86_64" # ROCk Job 2: Build ROCk Release Docker Image
id: build-rocm72-vllm-wheel - label: ":docker: Build release image - x86_64 - ROCk"
depends_on: build-rocm72-base id: build-rock-release-image
agents: *rocm_build_agents
timeout_in_minutes: 180
env: *rocm72_env
commands: *rocm_vllm_wheel_commands
- label: ":s3: Upload ROCm 7.2 Wheels to S3"
id: upload-rocm72-wheels
depends_on: build-rocm72-vllm-wheel
agents: *rocm_build_agents
timeout_in_minutes: 60
env: *rocm72_env
commands: *rocm_upload_commands
- label: ":memo: Annotate ROCm 7.2 wheel release"
id: annotate-rocm72-release
depends_on: upload-rocm72-wheels
agents: *rocm_build_agents
env: *rocm72_env
commands: *rocm_annotate_commands
- label: ":docker: Build release image - x86_64 - ROCm 7.2"
id: build-rocm72-release-image
depends_on: depends_on:
- step: block-build-release-images - step: block-build-release-images
allow_failure: true allow_failure: true
- step: build-rocm72-base - step: build-rock-base-image
allow_failure: false allow_failure: false
agents: *rocm_build_agents agents:
queue: cpu_queue_release
timeout_in_minutes: 180 timeout_in_minutes: 180
env: *rocm72_env commands:
commands: *rocm_release_image_commands - |
set -euo pipefail
- label: "Publish nightly ROCm 7.2 image to DockerHub" # Login to ECR
id: publish-rocm72-nightly aws ecr-public get-login-password --region us-east-1 | \
depends_on: build-rocm72-release-image docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7
if: build.env("NIGHTLY") == "1"
agents: *rocm_publish_agents
env: *rocm72_env
commands: *rocm_publish_nightly_commands
plugins: *rocm_dockerhub_login
- label: "Publish nightly XPU image to DockerHub" ECR_IMAGE_TAG="$$(buildkite-agent meta-data get rock-base-image-tag 2>/dev/null || echo '')"
if [ -z "$${ECR_IMAGE_TAG}" ]; then
echo "ERROR: rock-base-image-tag metadata not found"
echo "This should have been set by the build-rock-base-image job"
exit 1
fi
docker pull "$${ECR_IMAGE_TAG}"
# Pass the base image ECR tag to downstream steps (nightly publish)
buildkite-agent meta-data set "rock-base-ecr-tag" "$${ECR_IMAGE_TAG}"
echo "========================================"
echo "Building vLLM ROCk release image with:"
echo " BASE_IMAGE: $${ECR_IMAGE_TAG}"
echo " BUILDKITE_COMMIT: $${BUILDKITE_COMMIT}"
echo "========================================"
DOCKER_BUILDKIT=1 docker build \
--build-arg max_jobs=16 \
--build-arg BASE_IMAGE="$${ECR_IMAGE_TAG}" \
--build-arg USE_SCCACHE=1 \
--build-arg SCCACHE_BUCKET_NAME=vllm-build-sccache \
--build-arg SCCACHE_REGION_NAME=us-west-2 \
--build-arg SCCACHE_S3_NO_CREDENTIALS=0 \
--build-arg INSTALL_LMCACHE=true \
--tag public.ecr.aws/q9t5s3a7/vllm-release-repo:$${BUILDKITE_COMMIT}-rock \
--target vllm-openai \
--progress plain \
-f docker/Dockerfile.rock .
docker push public.ecr.aws/q9t5s3a7/vllm-release-repo:$${BUILDKITE_COMMIT}-rock
env:
DOCKER_BUILDKIT: "1"
- label: "Publish nightly ROCk image to DockerHub"
depends_on: depends_on:
- create-manifest-xpu - build-rock-release-image
if: build.env("NIGHTLY") == "1" if: build.env("NIGHTLY") == "1"
agents: agents:
queue: small_cpu_queue_release queue: small_cpu_queue_release
commands: commands:
- "bash .buildkite/scripts/xpu/push-nightly-builds-xpu.sh" - "bash .buildkite/scripts/push-nightly-builds-rocm.sh rock rocm100"
- "bash .buildkite/scripts/cleanup-nightly-builds.sh nightly- vllm/vllm-openai-xpu" # Clean up old nightly builds (keep only last 14)
- "bash .buildkite/scripts/cleanup-nightly-builds.sh nightly-rocm100- vllm/vllm-openai-rocm"
- "bash .buildkite/scripts/cleanup-nightly-builds.sh base-nightly-rocm100- vllm/vllm-openai-rocm"
plugins: plugins:
- docker-login#v3.0.0: - docker-login#v3.0.0:
username: vllmbot username: vllmbot
@@ -997,8 +1264,7 @@ steps:
- create-multi-arch-manifest-ubuntu2404 - create-multi-arch-manifest-ubuntu2404
- create-multi-arch-manifest-cuda-12-9-ubuntu2404 - create-multi-arch-manifest-cuda-12-9-ubuntu2404
- create-manifest-xpu - create-manifest-xpu
- build-rocm100-release-image - build-rocm-release-image
- build-rocm72-release-image
- input-release-version - input-release-version
# Wait for CPU builds if their block steps were unblocked, so publish # Wait for CPU builds if their block steps were unblocked, so publish
# doesn't race the in-progress CPU build. allow_failure lets publish # doesn't race the in-progress CPU build. allow_failure lets publish
@@ -1023,7 +1289,6 @@ steps:
- "cuda-13-0-ubuntu-24-04" - "cuda-13-0-ubuntu-24-04"
- "cuda-12-9-ubuntu-24-04" - "cuda-12-9-ubuntu-24-04"
- "rocm" - "rocm"
- "rocm72"
- "xpu" - "xpu"
- "cpu" - "cpu"
plugins: plugins:
+66 -40
View File
@@ -2,23 +2,15 @@
# SPDX-License-Identifier: Apache-2.0 # SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
# #
# Generate Buildkite annotation for a ROCm stack's wheel release. # Generate Buildkite annotation for ROCm wheel release
#
# Usage: annotate-rocm-release.sh <base-dockerfile> [--default]
# base-dockerfile e.g. docker/Dockerfile.rocm_base or docker/Dockerfile.rocm_72_base
# --default the stack also owns the plain :latest / :v<ver> Docker Hub tags
set -ex set -ex
# shellcheck source=.buildkite/scripts/rocm/stack.sh # Extract build configuration from Dockerfile.rocm_base (single source of truth)
source "$(dirname "${BASH_SOURCE[0]}")/rocm/stack.sh" "${1:?Usage: $0 <base-dockerfile> [--default]}" # Extract ROCm version dynamically from Dockerfile.rocm_base
TAG_VARIANTS=("-${ROCM_STACK_VARIANT}") # BASE_IMAGE format: rocm/dev-ubuntu-22.04:7.0-complete -> extracts "7.0"
if [[ "${2:-}" == "--default" ]]; then ROCM_VERSION=$(grep -E '^ARG BASE_IMAGE=' docker/Dockerfile.rocm_base | sed -E 's/.*:([0-9]+\.[0-9]+).*/\1/' || echo "unknown")
TAG_VARIANTS=("" "-${ROCM_STACK_VARIANT}") PYTHON_VERSION=$(grep '^ARG PYTHON_VERSION=' docker/Dockerfile.rocm_base | sed 's/^ARG PYTHON_VERSION=//')
fi PYTORCH_ROCM_ARCH=$(grep '^ARG PYTORCH_ROCM_ARCH=' docker/Dockerfile.rocm_base | sed 's/^ARG PYTORCH_ROCM_ARCH=//')
ROCM_VERSION="$ROCM_STACK_VERSION"
VARIANT="$ROCM_STACK_VARIANT"
PYTHON_VERSION=$(sed -nE 's/^ARG PYTHON_VERSION="?([^" ]+)"?.*/\1/p' "$ROCM_STACK_BASE_DOCKERFILE")
PYTORCH_ROCM_ARCH=$(sed -nE 's/^ARG PYTORCH_ROCM_ARCH="?([^"]+)"?.*/\1/p' "$ROCM_STACK_BASE_DOCKERFILE")
# Get release version, default to 1.0.0.dev for nightly/per-commit builds # Get release version, default to 1.0.0.dev for nightly/per-commit builds
RELEASE_VERSION=$(buildkite-agent meta-data get release-version 2>/dev/null || echo "") RELEASE_VERSION=$(buildkite-agent meta-data get release-version 2>/dev/null || echo "")
@@ -26,32 +18,22 @@ if [ -z "${RELEASE_VERSION}" ]; then
RELEASE_VERSION="1.0.0.dev" RELEASE_VERSION="1.0.0.dev"
fi fi
ROCM_BASE_CACHE_KEY=$(.buildkite/scripts/cache-rocm-base-wheels.sh key)
# S3 URLs # S3 URLs
S3_BUCKET="${S3_BUCKET:-vllm-wheels}" S3_BUCKET="${S3_BUCKET:-vllm-wheels}"
S3_REGION="${AWS_DEFAULT_REGION:-us-west-2}" S3_REGION="${AWS_DEFAULT_REGION:-us-west-2}"
S3_HOST="${S3_BUCKET}.s3-website-${S3_REGION}.amazonaws.com" S3_URL="http://${S3_BUCKET}.s3-website-${S3_REGION}.amazonaws.com"
S3_URL="http://${S3_HOST}"
WHEEL_PATH="rocm/${BUILDKITE_COMMIT}/${VARIANT}-wheels/"
DOCKER_CMDS="docker pull ${ROCM_STACK_ECR_BASE} # Format ROCm version for path (e.g., "7.1" -> "rocm710")
docker pull ${ROCM_STACK_ECR_IMAGE}" ROCM_VERSION_PATH="rocm$(echo "${ROCM_VERSION}" | tr -d '.')"
for suffix in "${TAG_VARIANTS[@]}"; do ROCM_PATH="rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}"
for tag in "latest${suffix}" "v${RELEASE_VERSION}${suffix}"; do buildkite-agent annotate --style 'success' --context 'rocm-release-workflow' << EOF
DOCKER_CMDS+=" ## ROCm Wheel and Docker Image Releases
docker tag ${ROCM_STACK_ECR_BASE} vllm/vllm-openai-rocm:${tag}-base
docker tag ${ROCM_STACK_ECR_IMAGE} vllm/vllm-openai-rocm:${tag}
docker push vllm/vllm-openai-rocm:${tag}-base
docker push vllm/vllm-openai-rocm:${tag}"
done
done
buildkite-agent annotate --style 'success' --context "rocm-release-workflow-${VARIANT}" << EOF
## ROCm ${ROCM_VERSION} Wheel and Docker Image Releases
### Build Configuration ### Build Configuration
| Setting | Value | | Setting | Value |
|---------|-------| |---------|-------|
| **ROCm Version** | ${ROCM_VERSION} | | **ROCm Version** | ${ROCM_VERSION} |
| **Index Variant** | ${VARIANT} |
| **Python Version** | ${PYTHON_VERSION} | | **Python Version** | ${PYTHON_VERSION} |
| **GPU Architectures** | ${PYTORCH_ROCM_ARCH} | | **GPU Architectures** | ${PYTORCH_ROCM_ARCH} |
| **Branch** | \`${BUILDKITE_BRANCH}\` | | **Branch** | \`${BUILDKITE_BRANCH}\` |
@@ -59,20 +41,49 @@ buildkite-agent annotate --style 'success' --context "rocm-release-workflow-${VA
### :package: Installation ### :package: Installation
**Install from this build (by commit):**
\`\`\`bash \`\`\`bash
# This build (by commit) pip install vllm --extra-index-url ${S3_URL}/${ROCM_PATH}/ --trusted-host ${S3_BUCKET}.s3-website-${S3_REGION}.amazonaws.com
pip install vllm --extra-index-url ${S3_URL}/rocm/${BUILDKITE_COMMIT}/${VARIANT}/ --trusted-host ${S3_HOST}
# Nightly (if published) # Example for ROCm ${ROCM_VERSION}:
pip install vllm --extra-index-url ${S3_URL}/rocm/nightly/${VARIANT}/ --trusted-host ${S3_HOST} pip install vllm --extra-index-url ${S3_URL}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/ --trusted-host ${S3_BUCKET}.s3-website-${S3_REGION}.amazonaws.com
\`\`\`
**Install from nightly (if published):**
\`\`\`bash
pip install vllm --extra-index-url ${S3_URL}/rocm/nightly/ --trusted-host ${S3_BUCKET}.s3-website-${S3_REGION}.amazonaws.com
\`\`\` \`\`\`
### :floppy_disk: Download Wheels Directly ### :floppy_disk: Download Wheels Directly
\`\`\`bash \`\`\`bash
aws s3 ls s3://${S3_BUCKET}/${WHEEL_PATH} # List all ROCm wheels
aws s3 cp --recursive --exclude "*" --include "*.whl" s3://${S3_BUCKET}/${WHEEL_PATH} . aws s3 ls s3://${S3_BUCKET}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/
# Download specific wheels
aws s3 cp s3://${S3_BUCKET}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/vllm-*.whl .
aws s3 cp s3://${S3_BUCKET}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/torch-*.whl .
aws s3 cp s3://${S3_BUCKET}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/triton-*.whl .
aws s3 cp s3://${S3_BUCKET}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/triton-kernels-*.whl .
aws s3 cp s3://${S3_BUCKET}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/torchvision-*.whl .
aws s3 cp s3://${S3_BUCKET}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/torchaudio-*.whl .
aws s3 cp s3://${S3_BUCKET}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/amdsmi-*.whl .
aws s3 cp s3://${S3_BUCKET}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/amd_aiter-*.whl .
aws s3 cp s3://${S3_BUCKET}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/flash-attn-*.whl .
\`\`\` \`\`\`
### :gear: Included Packages
- **vllm**: vLLM with ROCm support
- **torch**: PyTorch built for ROCm ${ROCM_VERSION}
- **triton**: Triton
- **triton-kernels**: Triton kernels
- **torchvision**: TorchVision for ROCm PyTorch
- **torchaudio**: Torchaudio for ROCm PyTorch
- **amdsmi**: AMD SMI Python bindings
- **amd_aiter**: Aiter for ROCm
- **flash-attn**: Flash Attention for ROCm
### :warning: Notes ### :warning: Notes
- These wheels are built for **ROCm ${ROCM_VERSION}** and will NOT work with CUDA GPUs - These wheels are built for **ROCm ${ROCM_VERSION}** and will NOT work with CUDA GPUs
- Supported GPU architectures: ${PYTORCH_ROCM_ARCH} - Supported GPU architectures: ${PYTORCH_ROCM_ARCH}
@@ -80,8 +91,23 @@ aws s3 cp --recursive --exclude "*" --include "*.whl" s3://${S3_BUCKET}/${WHEEL_
### :package: Docker Image Release ### :package: Docker Image Release
To download and upload the image:
\`\`\` \`\`\`
${DOCKER_CMDS} docker pull public.ecr.aws/q9t5s3a7/vllm-release-repo:${BUILDKITE_COMMIT}-rocm-base
docker pull public.ecr.aws/q9t5s3a7/vllm-release-repo:${BUILDKITE_COMMIT}-rocm
docker tag public.ecr.aws/q9t5s3a7/vllm-release-repo:${ROCM_BASE_CACHE_KEY}-rocm-base vllm/vllm-openai-rocm:${BUILDKITE_COMMIT}-base
docker tag vllm/vllm-openai-rocm:${BUILDKITE_COMMIT}-base vllm/vllm-openai-rocm:latest-base
docker tag vllm/vllm-openai-rocm:${BUILDKITE_COMMIT}-base vllm/vllm-openai-rocm:v${RELEASE_VERSION}-base
docker push vllm/vllm-openai-rocm:latest-base
docker push vllm/vllm-openai-rocm:v${RELEASE_VERSION}-base
docker tag public.ecr.aws/q9t5s3a7/vllm-release-repo:${BUILDKITE_COMMIT}-rocm vllm/vllm-openai-rocm:${BUILDKITE_COMMIT}
docker tag vllm/vllm-openai-rocm:${BUILDKITE_COMMIT} vllm/vllm-openai-rocm:latest
docker tag vllm/vllm-openai-rocm:${BUILDKITE_COMMIT} vllm/vllm-openai-rocm:v${RELEASE_VERSION}
docker push vllm/vllm-openai-rocm:latest
docker push vllm/vllm-openai-rocm:v${RELEASE_VERSION}
\`\`\` \`\`\`
EOF EOF
+34 -28
View File
@@ -5,33 +5,39 @@
# Cache helper for ROCm base wheels # Cache helper for ROCm base wheels
# #
# This script manages caching of pre-built ROCm base wheels (torch, triton, etc.) # This script manages caching of pre-built ROCm base wheels (torch, triton, etc.)
# to avoid rebuilding them when the stack's base Dockerfile hasn't changed. # to avoid rebuilding them when Dockerfile.rocm_base hasn't changed.
# #
# Usage: cache-rocm-base-wheels.sh <base-dockerfile> <command> # Usage:
# base-dockerfile e.g. docker/Dockerfile.rocm_base or docker/Dockerfile.rocm_72_base # cache-rocm-base-wheels.sh check - Check if cache exists, outputs "hit" or "miss"
# Commands: # cache-rocm-base-wheels.sh upload - Upload wheels to cache
# check - Check if cache exists, outputs "hit" or "miss" # cache-rocm-base-wheels.sh download - Download wheels from cache
# upload - Upload wheels to cache # cache-rocm-base-wheels.sh key - Output the cache key
# download - Download wheels from cache
# key - Output the cache key
# path - Output the full S3 cache path
# #
# Environment variables: # Environment variables:
# S3_BUCKET - S3 bucket name (default: vllm-wheels) # S3_BUCKET - S3 bucket name (default: vllm-wheels)
# #
# The cache key hashes the base Dockerfile and the repo files it COPYs in, which # Note: ROCm version is determined by BASE_IMAGE in Dockerfile.rocm_base,
# capture the ROCm version and every component pin (see rocm/stack.sh). # so changes to ROCm version are captured by the Dockerfile hash.
set -euo pipefail set -euo pipefail
BUCKET="${S3_BUCKET:-vllm-wheels}" BUCKET="${S3_BUCKET:-vllm-wheels}"
DOCKERFILE="docker/Dockerfile.rocm_base"
CACHE_PREFIX="rocm/cache" CACHE_PREFIX="rocm/cache"
# shellcheck source=.buildkite/scripts/rocm/stack.sh # Generate hash from Dockerfile content + build args
source "$(dirname "${BASH_SOURCE[0]}")/rocm/stack.sh" "${1:?Usage: $0 <base-dockerfile> <command>}" generate_cache_key() {
shift # Include Dockerfile content
CACHE_KEY="$ROCM_STACK_WHEEL_KEY" if [[ ! -f "$DOCKERFILE" ]]; then
WHEELS_DIR="$ROCM_STACK_BASE_WHEELS_DIR" echo "ERROR: Dockerfile not found: $DOCKERFILE" >&2
exit 1
fi
local dockerfile_hash=$(sha256sum "$DOCKERFILE" | cut -c1-16)
echo "${dockerfile_hash}"
}
CACHE_KEY=$(generate_cache_key)
CACHE_PATH="s3://${BUCKET}/${CACHE_PREFIX}/${CACHE_KEY}/" CACHE_PATH="s3://${BUCKET}/${CACHE_PREFIX}/${CACHE_KEY}/"
case "${1:-}" in case "${1:-}" in
@@ -61,19 +67,19 @@ case "${1:-}" in
echo "Cache path: ${CACHE_PATH}" echo "Cache path: ${CACHE_PATH}"
echo "" echo ""
if [[ ! -d "${WHEELS_DIR}" ]]; then if [[ ! -d "artifacts/rocm-base-wheels" ]]; then
echo "ERROR: ${WHEELS_DIR} directory not found" >&2 echo "ERROR: artifacts/rocm-base-wheels directory not found" >&2
exit 1 exit 1
fi fi
WHEEL_COUNT=$(find "${WHEELS_DIR}" -maxdepth 1 -name '*.whl' 2>/dev/null | wc -l) WHEEL_COUNT=$(find artifacts/rocm-base-wheels -maxdepth 1 -name '*.whl' 2>/dev/null | wc -l)
if [[ "$WHEEL_COUNT" -eq 0 ]]; then if [[ "$WHEEL_COUNT" -eq 0 ]]; then
echo "ERROR: No wheels found in ${WHEELS_DIR}/" >&2 echo "ERROR: No wheels found in artifacts/rocm-base-wheels/" >&2
exit 1 exit 1
fi fi
echo "Uploading $WHEEL_COUNT wheels..." echo "Uploading $WHEEL_COUNT wheels..."
aws s3 cp --recursive "${WHEELS_DIR}/" "${CACHE_PATH}" aws s3 cp --recursive artifacts/rocm-base-wheels/ "${CACHE_PATH}"
echo "" echo ""
echo "Cache upload complete!" echo "Cache upload complete!"
@@ -87,17 +93,17 @@ case "${1:-}" in
echo "Cache key: ${CACHE_KEY}" echo "Cache key: ${CACHE_KEY}"
echo "Cache path: ${CACHE_PATH}" echo "Cache path: ${CACHE_PATH}"
echo "" echo ""
mkdir -p "${WHEELS_DIR}" mkdir -p artifacts/rocm-base-wheels
# Use sync with include/exclude to only download .whl files # Use sync with include/exclude to only download .whl files
aws s3 sync "${CACHE_PATH}" "${WHEELS_DIR}/" \ aws s3 sync "${CACHE_PATH}" artifacts/rocm-base-wheels/ \
--exclude "*" \ --exclude "*" \
--include "*.whl" --include "*.whl"
echo "" echo ""
echo "Downloaded wheels:" echo "Downloaded wheels:"
find "${WHEELS_DIR}" -maxdepth 1 -name '*.whl' -exec ls -lh {} \; find artifacts/rocm-base-wheels -maxdepth 1 -name '*.whl' -exec ls -lh {} \;
WHEEL_COUNT=$(find "${WHEELS_DIR}" -maxdepth 1 -name '*.whl' 2>/dev/null | wc -l) WHEEL_COUNT=$(find artifacts/rocm-base-wheels -maxdepth 1 -name '*.whl' 2>/dev/null | wc -l)
echo "" echo ""
echo "Total: $WHEEL_COUNT wheels" echo "Total: $WHEEL_COUNT wheels"
echo "========================================" echo "========================================"
@@ -112,12 +118,12 @@ case "${1:-}" in
;; ;;
*) *)
echo "Usage: $0 <base-dockerfile> {check|upload|download|key|path}" >&2 echo "Usage: $0 {check|upload|download|key|path}" >&2
echo "" >&2 echo "" >&2
echo "Commands:" >&2 echo "Commands:" >&2
echo " check - Check if cache exists, outputs 'hit' or 'miss'" >&2 echo " check - Check if cache exists, outputs 'hit' or 'miss'" >&2
echo " upload - Upload wheels from ${WHEELS_DIR}/ to cache" >&2 echo " upload - Upload wheels from artifacts/rocm-base-wheels/ to cache" >&2
echo " download - Download wheels from cache to ${WHEELS_DIR}/" >&2 echo " download - Download wheels from cache to artifacts/rocm-base-wheels/" >&2
echo " key - Output the cache key" >&2 echo " key - Output the cache key" >&2
echo " path - Output the full S3 cache path" >&2 echo " path - Output the full S3 cache path" >&2
exit 1 exit 1
+54 -7
View File
@@ -17,7 +17,7 @@ DEFAULT_REPO_SLUG="vllm-project/vllm"
DEFAULT_CI_HCL_SOURCE="docker/ci-rocm.hcl" DEFAULT_CI_HCL_SOURCE="docker/ci-rocm.hcl"
DEFAULT_CI_BASE_CONTENT_FILES=".dockerignore requirements/common.txt requirements/rocm.txt requirements/test/rocm.txt tools/install_torchcodec_rocm.sh rust-toolchain.toml tests/vllm_test_utils" DEFAULT_CI_BASE_CONTENT_FILES=".dockerignore requirements/common.txt requirements/rocm.txt requirements/test/rocm.txt tools/install_torchcodec_rocm.sh rust-toolchain.toml tests/vllm_test_utils"
DEFAULT_CI_BASE_DOCKERFILE="docker/Dockerfile.rocm" DEFAULT_CI_BASE_DOCKERFILE="docker/Dockerfile.rocm"
DEFAULT_CI_BASE_DOCKERFILE_STAGES="base rust_toolchain_input_0 rust-toolchain-input rust-toolchain build_nixl lmcache_source build_lmcache build_deepep mori_base ci_base" DEFAULT_CI_BASE_DOCKERFILE_STAGES="base rust_toolchain_input_0 rust-toolchain-input rust-toolchain build_nixl lmcache_source build_lmcache build_rocshmem build_deepep mori_base ci_base"
DEFAULT_CI_BASE_METADATA_VERSION="3" DEFAULT_CI_BASE_METADATA_VERSION="3"
# ROCm CI forces REMOTE_VLLM=0, so content identity covers only the selected # ROCm CI forces REMOTE_VLLM=0, so content identity covers only the selected
# local-source stages rather than unreachable remote-fetch alternatives. # local-source stages rather than unreachable remote-fetch alternatives.
@@ -555,6 +555,9 @@ configure_custom_rocm_stages() {
CI_BASE_DOCKERFILE_STAGES="${CI_BASE_DOCKERFILE_STAGES:-${stages}}" CI_BASE_DOCKERFILE_STAGES="${CI_BASE_DOCKERFILE_STAGES:-${stages}}"
ROCM_CSRC_DOCKERFILE_STAGES="${ROCM_CSRC_DOCKERFILE_STAGES:-${stages}}" ROCM_CSRC_DOCKERFILE_STAGES="${ROCM_CSRC_DOCKERFILE_STAGES:-${stages}}"
ROCM_RUST_DOCKERFILE_STAGES="${ROCM_RUST_DOCKERFILE_STAGES:-${stages}}" ROCM_RUST_DOCKERFILE_STAGES="${ROCM_RUST_DOCKERFILE_STAGES:-${stages}}"
if [[ " ${stages} " != *" build_rocshmem "* ]]; then
unset ROCSHMEM_BRANCH ROCSHMEM_CACHE_KEY DEEPEP_CACHE_KEY
fi
} }
compose_dependency_cache_key() { compose_dependency_cache_key() {
@@ -1721,11 +1724,14 @@ ci_base_metadata_pairs() {
metadata_pair "vllm.rocm.nixl_commit" "${NIXL_BRANCH:-$(resolve_dockerfile_arg_value "${dockerfile}" "NIXL_BRANCH")}" metadata_pair "vllm.rocm.nixl_commit" "${NIXL_BRANCH:-$(resolve_dockerfile_arg_value "${dockerfile}" "NIXL_BRANCH")}"
metadata_pair "vllm.rocm.ucx_repo" "$(resolve_dockerfile_arg_value "${dockerfile}" "UCX_REPO")" metadata_pair "vllm.rocm.ucx_repo" "$(resolve_dockerfile_arg_value "${dockerfile}" "UCX_REPO")"
metadata_pair "vllm.rocm.ucx_commit" "${UCX_BRANCH:-$(resolve_dockerfile_arg_value "${dockerfile}" "UCX_BRANCH")}" metadata_pair "vllm.rocm.ucx_commit" "${UCX_BRANCH:-$(resolve_dockerfile_arg_value "${dockerfile}" "UCX_BRANCH")}"
metadata_pair "vllm.rocm.rocshmem_repo" "$(resolve_dockerfile_arg_value "${dockerfile}" "ROCSHMEM_REPO")"
metadata_pair "vllm.rocm.rocshmem_commit" "${ROCSHMEM_BRANCH:-$(resolve_dockerfile_arg_value "${dockerfile}" "ROCSHMEM_BRANCH")}"
metadata_pair "vllm.rocm.deepep_repo" "$(resolve_dockerfile_arg_value "${dockerfile}" "DEEPEP_REPO")" metadata_pair "vllm.rocm.deepep_repo" "$(resolve_dockerfile_arg_value "${dockerfile}" "DEEPEP_REPO")"
metadata_pair "vllm.rocm.deepep_commit" "${DEEPEP_BRANCH:-$(resolve_dockerfile_arg_value "${dockerfile}" "DEEPEP_BRANCH")}" metadata_pair "vllm.rocm.deepep_commit" "${DEEPEP_BRANCH:-$(resolve_dockerfile_arg_value "${dockerfile}" "DEEPEP_BRANCH")}"
metadata_pair "vllm.rocm.deepep_nic" "$(resolve_dockerfile_arg_value "${dockerfile}" "DEEPEP_NIC")" metadata_pair "vllm.rocm.deepep_nic" "$(resolve_dockerfile_arg_value "${dockerfile}" "DEEPEP_NIC")"
metadata_pair "vllm.rocm.deepep_rocm_arch" "$(resolve_dockerfile_arg_value "${dockerfile}" "DEEPEP_ROCM_ARCH")" metadata_pair "vllm.rocm.deepep_rocm_arch" "$(resolve_dockerfile_arg_value "${dockerfile}" "DEEPEP_ROCM_ARCH")"
metadata_pair "vllm.rocm.nixl_cache_key" "${NIXL_CACHE_KEY:-}" metadata_pair "vllm.rocm.nixl_cache_key" "${NIXL_CACHE_KEY:-}"
metadata_pair "vllm.rocm.rocshmem_cache_key" "${ROCSHMEM_CACHE_KEY:-}"
metadata_pair "vllm.rocm.deepep_cache_key" "${DEEPEP_CACHE_KEY:-}" metadata_pair "vllm.rocm.deepep_cache_key" "${DEEPEP_CACHE_KEY:-}"
} }
@@ -2308,7 +2314,7 @@ extract_dependency_pins() {
return 0 return 0
fi fi
for var in NIXL_BRANCH UCX_BRANCH DEEPEP_BRANCH; do for var in NIXL_BRANCH UCX_BRANCH ROCSHMEM_BRANCH DEEPEP_BRANCH; do
if [[ -n "${!var:-}" ]]; then if [[ -n "${!var:-}" ]]; then
echo "Using provided ${var}: ${!var}" echo "Using provided ${var}: ${!var}"
continue continue
@@ -2330,14 +2336,17 @@ compute_dependency_cache_keys() {
local dockerfile_rocm="" local dockerfile_rocm=""
local nixl_branch="" local nixl_branch=""
local ucx_branch="" local ucx_branch=""
local rocshmem_branch=""
local deepep_branch="" local deepep_branch=""
local nixl_material="" local nixl_material=""
local rocshmem_material=""
local deepep_material="" local deepep_material=""
bake_dir=$(dirname "${VLLM_BAKE_FILE}") bake_dir=$(dirname "${VLLM_BAKE_FILE}")
dockerfile_rocm="${CI_BASE_DOCKERFILE:-${bake_dir}/Dockerfile.rocm}" dockerfile_rocm="${CI_BASE_DOCKERFILE:-${bake_dir}/Dockerfile.rocm}"
nixl_branch=$(resolve_dockerfile_arg_value "${dockerfile_rocm}" "NIXL_BRANCH") nixl_branch=$(resolve_dockerfile_arg_value "${dockerfile_rocm}" "NIXL_BRANCH")
ucx_branch=$(resolve_dockerfile_arg_value "${dockerfile_rocm}" "UCX_BRANCH") ucx_branch=$(resolve_dockerfile_arg_value "${dockerfile_rocm}" "UCX_BRANCH")
rocshmem_branch=$(resolve_dockerfile_arg_value "${dockerfile_rocm}" "ROCSHMEM_BRANCH")
deepep_branch=$(resolve_dockerfile_arg_value "${dockerfile_rocm}" "DEEPEP_BRANCH") deepep_branch=$(resolve_dockerfile_arg_value "${dockerfile_rocm}" "DEEPEP_BRANCH")
if [[ -n "${nixl_branch}" && -n "${ucx_branch}" ]]; then if [[ -n "${nixl_branch}" && -n "${ucx_branch}" ]]; then
@@ -2351,11 +2360,22 @@ compute_dependency_cache_keys() {
echo "NIXL dependency cache key: ${NIXL_CACHE_KEY}" echo "NIXL dependency cache key: ${NIXL_CACHE_KEY}"
fi fi
if [[ -n "${deepep_branch}" ]]; then if [[ -n "${rocshmem_branch}" ]]; then
deepep_material=$(compose_stage_cache_material "${dockerfile_rocm}" "base build_deepep") rocshmem_material=$(compose_stage_cache_material "${dockerfile_rocm}" "base build_rocshmem")
ROCSHMEM_CACHE_KEY=$(
compose_dependency_cache_key \
"${rocshmem_branch}" \
"${rocshmem_material}"
)
export ROCSHMEM_CACHE_KEY
echo "ROCShmem dependency cache key: ${ROCSHMEM_CACHE_KEY}"
fi
if [[ -n "${deepep_branch}" && -n "${rocshmem_branch}" ]]; then
deepep_material=$(compose_stage_cache_material "${dockerfile_rocm}" "base build_rocshmem build_deepep")
DEEPEP_CACHE_KEY=$( DEEPEP_CACHE_KEY=$(
compose_dependency_cache_key \ compose_dependency_cache_key \
"${deepep_branch}" \ "${deepep_branch}-rocshmem-${rocshmem_branch}" \
"${deepep_material}" "${deepep_material}"
) )
export DEEPEP_CACHE_KEY export DEEPEP_CACHE_KEY
@@ -2395,11 +2415,18 @@ dependency_cache_ref_for_target() {
printf '%s\n' "${cache_repo}:nixl-rocm-${NIXL_BRANCH}-ucx-${UCX_BRANCH:-}" printf '%s\n' "${cache_repo}:nixl-rocm-${NIXL_BRANCH}-ucx-${UCX_BRANCH:-}"
fi fi
;; ;;
rocshmem-rocm-ci)
if [[ -n "${ROCSHMEM_CACHE_KEY:-}" ]]; then
printf '%s\n' "${cache_repo}:rocshmem-rocm-${ROCSHMEM_CACHE_KEY}"
elif [[ -n "${ROCSHMEM_BRANCH:-}" ]]; then
printf '%s\n' "${cache_repo}:rocshmem-rocm-${ROCSHMEM_BRANCH}"
fi
;;
deepep-rocm-ci) deepep-rocm-ci)
if [[ -n "${DEEPEP_CACHE_KEY:-}" ]]; then if [[ -n "${DEEPEP_CACHE_KEY:-}" ]]; then
printf '%s\n' "${cache_repo}:deepep-rocm-${DEEPEP_CACHE_KEY}" printf '%s\n' "${cache_repo}:deepep-rocm-${DEEPEP_CACHE_KEY}"
elif [[ -n "${DEEPEP_BRANCH:-}" ]]; then elif [[ -n "${DEEPEP_BRANCH:-}" ]]; then
printf '%s\n' "${cache_repo}:deepep-rocm-${DEEPEP_BRANCH}" printf '%s\n' "${cache_repo}:deepep-rocm-${DEEPEP_BRANCH}-rocshmem-${ROCSHMEM_BRANCH:-}"
fi fi
;; ;;
esac esac
@@ -2417,6 +2444,7 @@ add_dependency_cache_target() {
resolve_ci_base_dependency_targets() { resolve_ci_base_dependency_targets() {
local mode="${ROCM_DEP_CACHE_EXPORT_MODE:-missing}" local mode="${ROCM_DEP_CACHE_EXPORT_MODE:-missing}"
local nixl_ref="" local nixl_ref=""
local rocshmem_ref=""
local deepep_ref="" local deepep_ref=""
[[ "${TARGET}" == "ci-base-rocm-ci-with-deps" ]] || return 0 [[ "${TARGET}" == "ci-base-rocm-ci-with-deps" ]] || return 0
@@ -2424,7 +2452,7 @@ resolve_ci_base_dependency_targets() {
case "${mode}" in case "${mode}" in
always) always)
echo "ROCM_DEP_CACHE_EXPORT_MODE=always; exporting all dependency caches serially" echo "ROCM_DEP_CACHE_EXPORT_MODE=always; exporting all dependency caches serially"
for target in nixl-rocm-ci deepep-rocm-ci; do for target in nixl-rocm-ci rocshmem-rocm-ci deepep-rocm-ci; do
if [[ -n "$(dependency_cache_ref_for_target "${target}")" ]]; then if [[ -n "$(dependency_cache_ref_for_target "${target}")" ]]; then
add_dependency_cache_target "${target}" add_dependency_cache_target "${target}"
fi fi
@@ -2454,6 +2482,16 @@ resolve_ci_base_dependency_targets() {
fi fi
fi fi
if [[ "${mode}" != "always" && -n "${ROCSHMEM_CACHE_KEY:-}" ]]; then
rocshmem_ref=$(dependency_cache_ref_for_target "rocshmem-rocm-ci")
if dependency_cache_ref_exists "${rocshmem_ref}"; then
echo "ROCShmem dependency cache exists: ${rocshmem_ref}"
else
echo "ROCShmem dependency cache missing; will seed: ${rocshmem_ref}"
add_dependency_cache_target "rocshmem-rocm-ci"
fi
fi
if [[ "${mode}" != "always" && -n "${DEEPEP_CACHE_KEY:-}" ]]; then if [[ "${mode}" != "always" && -n "${DEEPEP_CACHE_KEY:-}" ]]; then
deepep_ref=$(dependency_cache_ref_for_target "deepep-rocm-ci") deepep_ref=$(dependency_cache_ref_for_target "deepep-rocm-ci")
if dependency_cache_ref_exists "${deepep_ref}"; then if dependency_cache_ref_exists "${deepep_ref}"; then
@@ -2464,6 +2502,15 @@ resolve_ci_base_dependency_targets() {
fi fi
fi fi
# DeepEP inherits from ROCShmem. If ROCShmem is being seeded, seed DeepEP too
# so the pair stays consistent for future ci_base rebuilds.
if printf '%s\n' "${DEPENDENCY_CACHE_TARGETS[@]}" | grep -qx "rocshmem-rocm-ci" \
&& ! printf '%s\n' "${DEPENDENCY_CACHE_TARGETS[@]}" | grep -qx "deepep-rocm-ci" \
&& [[ -n "${DEEPEP_BRANCH:-}" ]]; then
echo "ROCShmem cache is missing; also seeding DeepEP cache"
add_dependency_cache_target "deepep-rocm-ci"
fi
BAKE_TARGETS=("ci-base-rocm-ci") BAKE_TARGETS=("ci-base-rocm-ci")
if [[ ${#DEPENDENCY_CACHE_TARGETS[@]} -eq 0 ]]; then if [[ ${#DEPENDENCY_CACHE_TARGETS[@]} -eq 0 ]]; then
echo "All dependency caches exist; building ci_base without dependency cache exports" echo "All dependency caches exist; building ci_base without dependency cache exports"
+2 -5
View File
@@ -5,17 +5,14 @@ set -ex
# Clean up old nightly builds from DockerHub, keeping only the last 14 builds # Clean up old nightly builds from DockerHub, keeping only the last 14 builds
# This script uses DockerHub API to list and delete old tags with specified prefix # This script uses DockerHub API to list and delete old tags with specified prefix
# Tags starting with "nightly-dev" are always excluded from deletion # Tags starting with "nightly-dev" are always excluded from deletion
# Usage: cleanup-nightly-builds.sh [TAG_PREFIX] [REPO] [EXCLUDE_PREFIX] # Usage: cleanup-nightly-builds.sh [TAG_PREFIX] [REPO]
# Example: cleanup-nightly-builds.sh "nightly-" # Example: cleanup-nightly-builds.sh "nightly-"
# Example: cleanup-nightly-builds.sh "cu130-nightly-" # Example: cleanup-nightly-builds.sh "cu130-nightly-"
# Example: cleanup-nightly-builds.sh "nightly-" "vllm/vllm-openai-rocm" # Example: cleanup-nightly-builds.sh "nightly-" "vllm/vllm-openai-rocm"
# Example: cleanup-nightly-builds.sh "nightly-" "vllm/vllm-openai-rocm" "nightly-rocm"
# (EXCLUDE_PREFIX skips tags owned by another variant sharing TAG_PREFIX)
# Get tag prefix and repo from arguments # Get tag prefix and repo from arguments
TAG_PREFIX="${1:-nightly-}" TAG_PREFIX="${1:-nightly-}"
REPO="${2:-vllm/vllm-openai}" REPO="${2:-vllm/vllm-openai}"
EXCLUDE_PREFIX="${3:-}"
echo "Cleaning up tags with prefix: $TAG_PREFIX in repository: $REPO" echo "Cleaning up tags with prefix: $TAG_PREFIX in repository: $REPO"
@@ -60,7 +57,7 @@ get_all_tags() {
# Get both last_updated timestamp and tag name, separated by | # Get both last_updated timestamp and tag name, separated by |
# Exclude mutable architecture tags and nightly-dev tags from cleanup # Exclude mutable architecture tags and nightly-dev tags from cleanup
local tags=$(echo "$response" | jq -r --arg prefix "$TAG_PREFIX" --arg exclude "$EXCLUDE_PREFIX" '.results[] | select(.name | startswith($prefix)) | select($exclude == "" or (.name | startswith($exclude) | not)) | select(.name != ($prefix + "x86_64") and .name != ($prefix + "aarch64")) | select(.name | startswith("nightly-dev") | not) | "\(.last_updated)|\(.name)"') local tags=$(echo "$response" | jq -r --arg prefix "$TAG_PREFIX" '.results[] | select(.name | startswith($prefix)) | select(.name != ($prefix + "x86_64") and .name != ($prefix + "aarch64")) | select(.name | startswith("nightly-dev") | not) | "\(.last_updated)|\(.name)"')
if [ -z "$tags" ]; then if [ -z "$tags" ]; then
break break
+9 -63
View File
@@ -12,7 +12,7 @@ from dataclasses import asdict, dataclass
from datetime import datetime from datetime import datetime
from pathlib import Path from pathlib import Path
from typing import Any from typing import Any
from urllib.parse import quote, unquote, urlparse from urllib.parse import quote
import regex as re import regex as re
@@ -51,8 +51,6 @@ class WheelFileInfo:
platform_tag: str platform_tag: str
variant: str | None variant: str | None
filename: str filename: str
# Absolute URL for files hosted elsewhere (see --external-links)
url: str | None = None
def parse_from_filename(file: str) -> WheelFileInfo: def parse_from_filename(file: str) -> WheelFileInfo:
@@ -112,29 +110,6 @@ def parse_from_filename(file: str) -> WheelFileInfo:
) )
def parse_external_url(url: str) -> WheelFileInfo:
"""Parse an absolute wheel or sdist URL hosted on another index."""
filename = unquote(urlparse(url).path.rsplit("/", 1)[-1])
if filename.endswith(".whl"):
info = parse_from_filename(filename)
else:
match = re.match(r"^(?P<name>.+)-(?P<version>[^-]+)\.(tar\.gz|zip)$", filename)
if not match:
raise ValueError(f"Unsupported external file: {filename}")
info = WheelFileInfo(
package_name=match.group("name"),
version=match.group("version"),
build_tag=None,
python_tag="source",
abi_tag="none",
platform_tag="any",
variant=None,
filename=filename,
)
info.url = url
return info
def generate_project_list(subdir_names: list[str], comment: str = "") -> str: def generate_project_list(subdir_names: list[str], comment: str = "") -> str:
"""Generate project list HTML content linking to each project & variant subdirectory.""" """Generate project list HTML content linking to each project & variant subdirectory."""
href_tags = [] href_tags = []
@@ -154,19 +129,14 @@ def generate_package_index_and_metadata(
href_tags = [] href_tags = []
metadata = [] metadata = []
for file in sorted(wheel_files, key=lambda x: x.filename): for file in sorted(wheel_files, key=lambda x: x.filename):
if file.url: relative_path = (
file_path_quoted = file.url wheel_base_dir.relative_to(index_base_dir, walk_up=True) / file.filename
else: )
relative_path = ( # handle with '+' in URL, and avoid double-encoding '/' and already-encoded '%2B'
wheel_base_dir.relative_to(index_base_dir, walk_up=True) / file.filename # NOTE: this is AWS S3 specific behavior!
) file_path_quoted = quote(relative_path.as_posix(), safe=":%/")
# handle with '+' in URL, and avoid double-encoding '/' and already-encoded '%2B'
# NOTE: this is AWS S3 specific behavior!
file_path_quoted = quote(relative_path.as_posix(), safe=":%/")
href_tags.append(f' <a href="{file_path_quoted}">{file.filename}</a><br/>') href_tags.append(f' <a href="{file_path_quoted}">{file.filename}</a><br/>')
file_meta = asdict(file) file_meta = asdict(file)
if file_meta["url"] is None:
del file_meta["url"]
file_meta["path"] = file_path_quoted file_meta["path"] = file_path_quoted
metadata.append(file_meta) metadata.append(file_meta)
index_str = INDEX_HTML_TEMPLATE.format(items="\n".join(href_tags), comment=comment) index_str = INDEX_HTML_TEMPLATE.format(items="\n".join(href_tags), comment=comment)
@@ -181,7 +151,6 @@ def generate_index_and_metadata(
default_variant: str | None = None, default_variant: str | None = None,
alias_to_default: str | None = None, alias_to_default: str | None = None,
comment: str = "", comment: str = "",
external_urls: list[str] | None = None,
): ):
"""Generate index for all wheel files. """Generate index for all wheel files.
@@ -191,8 +160,6 @@ def generate_index_and_metadata(
index_base_dir (Path): Base directory to store index files. index_base_dir (Path): Base directory to store index files.
default_variant (str | None): The default variant name, if any. default_variant (str | None): The default variant name, if any.
alias_to_default (str | None): Alias variant name for the default variant, if any. alias_to_default (str | None): Alias variant name for the default variant, if any.
external_urls (list[str] | None): Absolute URLs of files hosted on another index,
listed in the generated indices without being copied.
comment (str | None): Optional comment to include in the generated HTML files. comment (str | None): Optional comment to include in the generated HTML files.
First, parse all wheel files to extract metadata. First, parse all wheel files to extract metadata.
@@ -247,7 +214,6 @@ def generate_index_and_metadata(
""" """
parsed_files = [parse_from_filename(f) for f in whl_files] parsed_files = [parse_from_filename(f) for f in whl_files]
parsed_files += [parse_external_url(u) for u in external_urls or []]
if not parsed_files: if not parsed_files:
print("No wheel files found, skipping index generation.") print("No wheel files found, skipping index generation.")
@@ -266,14 +232,10 @@ def generate_index_and_metadata(
print(f"Detected ROCm variant from vllm: {rocm_variant}") print(f"Detected ROCm variant from vllm: {rocm_variant}")
break break
# Apply ROCm variant to all wheels without a variant. Dependencies with their # Apply ROCm variant to all wheels without a variant
# own ROCm local version (e.g. TheRock's torch-2.12.0+rocm10.0.0) belong to
# the same variant rather than one of their own.
if rocm_variant: if rocm_variant:
for file in parsed_files: for file in parsed_files:
if file.variant is None or ( if file.variant is None:
file.variant.startswith("rocm") and file.variant != rocm_variant
):
file.variant = rocm_variant file.variant = rocm_variant
print(f"Inherited variant '{rocm_variant}' for {file.filename}") print(f"Inherited variant '{rocm_variant}' for {file.filename}")
@@ -404,13 +366,6 @@ if __name__ == "__main__":
default=None, default=None,
help="Alias variant name for the default variant", help="Alias variant name for the default variant",
) )
parser.add_argument(
"--external-links",
type=str,
default=None,
help="Optional file with one absolute URL per line for packages hosted on "
"another index (listed alongside, but not copied into, this index)",
)
parser.add_argument( parser.add_argument(
"--comment", "--comment",
type=str, type=str,
@@ -508,14 +463,5 @@ if __name__ == "__main__":
default_variant=None, default_variant=None,
alias_to_default=args.alias_to_default, alias_to_default=args.alias_to_default,
comment=args.comment.strip(), comment=args.comment.strip(),
external_urls=(
[
line.strip()
for line in Path(args.external_links).read_text().splitlines()
if line.strip() and not line.startswith("#")
]
if args.external_links
else None
),
) )
print(f"Successfully generated index and metadata in {output_dir}") print(f"Successfully generated index and metadata in {output_dir}")
+16 -28
View File
@@ -11,9 +11,9 @@ set -euo pipefail
TARGET="${1:-all}" TARGET="${1:-all}"
case "${TARGET}" in case "${TARGET}" in
cuda-13-0 | cuda-12-9 | cuda-13-0-ubuntu-24-04 | \ cuda-13-0 | cuda-12-9 | cuda-13-0-ubuntu-24-04 | \
cuda-12-9-ubuntu-24-04 | rocm | rocm72 | xpu | cpu | all) ;; cuda-12-9-ubuntu-24-04 | rocm | xpu | cpu | all) ;;
*) *)
echo "Usage: $0 {cuda-13-0|cuda-12-9|cuda-13-0-ubuntu-24-04|cuda-12-9-ubuntu-24-04|rocm|rocm72|xpu|cpu|all}" echo "Usage: $0 {cuda-13-0|cuda-12-9|cuda-13-0-ubuntu-24-04|cuda-12-9-ubuntu-24-04|rocm|xpu|cpu|all}"
exit 2 exit 2
;; ;;
esac esac
@@ -143,35 +143,23 @@ if target_enabled cuda-12-9-ubuntu-24-04; then
fi fi
# ---- ROCm ---- # ---- ROCm ----
# Each ROCm stack is identified by its base Dockerfile (see rocm/stack.sh). The
# default stack also owns the plain :latest / :v<ver> tags.
publish_rocm_stack() {
local base_dockerfile="$1" default="$2" suffix
(
# shellcheck source=.buildkite/scripts/rocm/stack.sh
source .buildkite/scripts/rocm/stack.sh "$base_dockerfile"
local suffixes=("-${ROCM_STACK_VARIANT}")
[[ "$default" == "1" ]] && suffixes=("" "-${ROCM_STACK_VARIANT}")
docker pull "$ROCM_STACK_ECR_IMAGE"
docker pull "$ROCM_STACK_ECR_BASE"
for suffix in "${suffixes[@]}"; do
for tag in "latest${suffix}" "v${RELEASE_VERSION}${suffix}"; do
docker tag "$ROCM_STACK_ECR_IMAGE" "vllm/vllm-openai-rocm:${tag}"
docker tag "$ROCM_STACK_ECR_BASE" "vllm/vllm-openai-rocm:${tag}-base"
docker push "vllm/vllm-openai-rocm:${tag}"
docker push "vllm/vllm-openai-rocm:${tag}-base"
done
done
)
}
if target_enabled rocm; then if target_enabled rocm; then
publish_rocm_stack docker/Dockerfile.rocm_base 1 ROCM_BASE_CACHE_KEY=$(.buildkite/scripts/cache-rocm-base-wheels.sh key)
fi echo "ROCm base cache key: ${ROCM_BASE_CACHE_KEY}"
if target_enabled rocm72; then docker pull "public.ecr.aws/q9t5s3a7/vllm-release-repo:${COMMIT}-rocm"
publish_rocm_stack docker/Dockerfile.rocm_72_base 0 docker pull "public.ecr.aws/q9t5s3a7/vllm-release-repo:${ROCM_BASE_CACHE_KEY}-rocm-base"
docker tag "public.ecr.aws/q9t5s3a7/vllm-release-repo:${COMMIT}-rocm" vllm/vllm-openai-rocm:latest
docker tag "public.ecr.aws/q9t5s3a7/vllm-release-repo:${COMMIT}-rocm" "vllm/vllm-openai-rocm:v${RELEASE_VERSION}"
docker push vllm/vllm-openai-rocm:latest
docker push "vllm/vllm-openai-rocm:v${RELEASE_VERSION}"
docker tag "public.ecr.aws/q9t5s3a7/vllm-release-repo:${ROCM_BASE_CACHE_KEY}-rocm-base" vllm/vllm-openai-rocm:latest-base
docker tag "public.ecr.aws/q9t5s3a7/vllm-release-repo:${ROCM_BASE_CACHE_KEY}-rocm-base" "vllm/vllm-openai-rocm:v${RELEASE_VERSION}-base"
docker push vllm/vllm-openai-rocm:latest-base
docker push "vllm/vllm-openai-rocm:v${RELEASE_VERSION}-base"
fi fi
# ---- XPU ---- # ---- XPU ----
+65 -42
View File
@@ -2,36 +2,42 @@
# SPDX-License-Identifier: Apache-2.0 # SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
# #
# Push a ROCm stack's nightly base image and nightly image from ECR to Docker Hub # Push a ROCm-family nightly base image and nightly image from ECR to Docker Hub
# under vllm/vllm-openai-rocm, as base-nightly-<variant>, nightly-<variant> and their # under vllm/vllm-openai-rocm, as base-nightly[-<variant>], nightly[-<variant>]
# -<commit> forms, then prune old nightly tags. With --default, also push the plain # and their -<commit> forms.
# base-nightly and nightly tags for the same images. # Run when NIGHTLY=1 after the matching build-*-release-image step has pushed to ECR.
# Run when NIGHTLY=1 after the stack's release image has been pushed to ECR.
# #
# Usage: push-nightly-builds-rocm.sh <base-dockerfile> [--default] # Usage: push-nightly-builds-rocm.sh [ECR_TAG_SUFFIX TAG_VARIANT]
# e.g. "docker/Dockerfile.rocm_base --default" (ROCm 10.0, also :nightly) # Pass both arguments or neither. With none, the default ROCm apt build is
# and "docker/Dockerfile.rocm_72_base" (ROCm 7.2, :nightly-rocm72). # published as :nightly. The TheRock build passes "rock" (the source ECR tag
# suffix) and "rocm714" (the Docker Hub tag flavor), giving :nightly-rocm714.
# #
# Local testing (no push to Docker Hub): # Local testing (no push to Docker Hub):
# BUILDKITE_COMMIT=<commit-with-image-in-ecr> DRY_RUN=1 \ # BASE_ECR_IMAGE=<full-base-image-reference> \
# bash .buildkite/scripts/push-nightly-builds-rocm.sh docker/Dockerfile.rocm_base --default # BUILDKITE_COMMIT=<commit-with-image-in-ecr> DRY_RUN=1 \
# bash .buildkite/scripts/push-nightly-builds-rocm.sh
# Requires: AWS CLI configured (for ECR public login), Docker. For full run: Docker Hub login. # Requires: AWS CLI configured (for ECR public login), Docker. For full run: Docker Hub login.
set -euxo pipefail set -euxo pipefail
usage() { usage() {
echo "Usage: $0 <base-dockerfile> [--default]" >&2 echo "Usage: $0 [ECR_TAG_SUFFIX TAG_VARIANT]" >&2
} }
PUSH_DEFAULT=0 DOCKERHUB_REPO="vllm/vllm-openai-rocm"
case "$#" in case "$#" in
1) ;; 0)
ECR_TAG_SUFFIX="rocm"
TAG_VARIANT=""
;;
2) 2)
if [[ "$2" != "--default" ]]; then if [[ -z "$1" || -z "$2" ]]; then
usage usage
exit 2 exit 2
fi fi
PUSH_DEFAULT=1 ECR_TAG_SUFFIX="$1"
TAG_VARIANT="$2"
;; ;;
*) *)
usage usage
@@ -43,44 +49,61 @@ esac
BUILDKITE_COMMIT="${BUILDKITE_COMMIT:?Set BUILDKITE_COMMIT to the commit SHA that has the image in ECR (e.g. from a previous release pipeline run)}" BUILDKITE_COMMIT="${BUILDKITE_COMMIT:?Set BUILDKITE_COMMIT to the commit SHA that has the image in ECR (e.g. from a previous release pipeline run)}"
DRY_RUN="${DRY_RUN:-0}" DRY_RUN="${DRY_RUN:-0}"
# shellcheck source=.buildkite/scripts/rocm/stack.sh ECR_REPO="public.ecr.aws/q9t5s3a7/vllm-release-repo"
source "$(dirname "${BASH_SOURCE[0]}")/rocm/stack.sh" "$1"
DOCKERHUB_REPO="vllm/vllm-openai-rocm"
VARIANT="$ROCM_STACK_VARIANT"
BASE_TAGS=("base-nightly-${VARIANT}" "base-nightly-${VARIANT}-${BUILDKITE_COMMIT}") # Get the base image ECR tag (set by the build-*-release-image pipeline step)
TAGS=("nightly-${VARIANT}" "nightly-${VARIANT}-${BUILDKITE_COMMIT}") BASE_METADATA_KEY="${ECR_TAG_SUFFIX}-base-ecr-tag"
if [[ "$PUSH_DEFAULT" == "1" ]]; then if [[ -n "${BASE_ECR_IMAGE:-}" ]]; then
BASE_TAGS+=("base-nightly" "base-nightly-${BUILDKITE_COMMIT}") if [[ "$DRY_RUN" != "1" ]]; then
TAGS+=("nightly" "nightly-${BUILDKITE_COMMIT}") echo "ERROR: BASE_ECR_IMAGE may only be used with DRY_RUN=1" >&2
exit 1
fi
BASE_ORIG_TAG="$BASE_ECR_IMAGE"
elif ! BASE_ORIG_TAG="$(buildkite-agent meta-data get "$BASE_METADATA_KEY")"; then
echo "ERROR: Failed to read required Buildkite metadata '$BASE_METADATA_KEY'" >&2
exit 1
fi
if [[ -z "$BASE_ORIG_TAG" ]]; then
echo "ERROR: Required Buildkite metadata '$BASE_METADATA_KEY' is empty" >&2
echo "Set BASE_ECR_IMAGE to the full ECR base image reference for local testing" >&2
exit 1
fi fi
echo "Pushing base image $ROCM_STACK_ECR_BASE as: ${BASE_TAGS[*]}" ORIG_TAG="${BUILDKITE_COMMIT}-${ECR_TAG_SUFFIX}"
echo "Pushing release image $ROCM_STACK_ECR_IMAGE as: ${TAGS[*]}" VARIANT_SUFFIX=""
[[ -n "$TAG_VARIANT" ]] && VARIANT_SUFFIX="-$TAG_VARIANT"
BASE_TAG_NAME="base-nightly${VARIANT_SUFFIX}"
TAG_NAME="nightly${VARIANT_SUFFIX}"
BASE_TAG_NAME_COMMIT="base-nightly${VARIANT_SUFFIX}-${BUILDKITE_COMMIT}"
TAG_NAME_COMMIT="nightly${VARIANT_SUFFIX}-${BUILDKITE_COMMIT}"
# Login to ECR and pull the stack's images echo "Pushing base image from ECR: $BASE_ORIG_TAG"
echo "Pushing release image from ECR tag: $ORIG_TAG to Docker Hub as $DOCKERHUB_REPO:$TAG_NAME and $DOCKERHUB_REPO:$TAG_NAME_COMMIT"
[[ "$DRY_RUN" == "1" ]] && echo "[DRY_RUN] Skipping push to Docker Hub"
# Login to ECR and pull the images built by build-*-release-image
aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7 aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7
docker pull "$ROCM_STACK_ECR_BASE" docker pull "$BASE_ORIG_TAG"
docker pull "$ROCM_STACK_ECR_IMAGE" docker pull "${ECR_REPO}:${ORIG_TAG}"
for tag in "${BASE_TAGS[@]}"; do docker tag "$ROCM_STACK_ECR_BASE" "$DOCKERHUB_REPO:$tag"; done # Tag for Docker Hub (base-nightly and base-nightly-<commit>, nightly and nightly-<commit>)
for tag in "${TAGS[@]}"; do docker tag "$ROCM_STACK_ECR_IMAGE" "$DOCKERHUB_REPO:$tag"; done docker tag "$BASE_ORIG_TAG" "$DOCKERHUB_REPO":"$BASE_TAG_NAME"
docker tag "$BASE_ORIG_TAG" "$DOCKERHUB_REPO":"$BASE_TAG_NAME_COMMIT"
docker tag "${ECR_REPO}:${ORIG_TAG}" "$DOCKERHUB_REPO":"$TAG_NAME"
docker tag "${ECR_REPO}:${ORIG_TAG}" "$DOCKERHUB_REPO":"$TAG_NAME_COMMIT"
if [[ "$DRY_RUN" == "1" ]]; then if [[ "$DRY_RUN" == "1" ]]; then
echo "[DRY_RUN] Would push $DOCKERHUB_REPO:$BASE_TAG_NAME and $DOCKERHUB_REPO:$BASE_TAG_NAME_COMMIT"
echo "[DRY_RUN] Would push $DOCKERHUB_REPO:$TAG_NAME and $DOCKERHUB_REPO:$TAG_NAME_COMMIT"
echo "[DRY_RUN] Local tags created. Exiting without push." echo "[DRY_RUN] Local tags created. Exiting without push."
exit 0 exit 0
fi fi
# Push to Docker Hub (docker-login plugin runs before this step in CI) # Push to Docker Hub (docker-login plugin runs before this step in CI)
for tag in "${BASE_TAGS[@]}" "${TAGS[@]}"; do docker push "$DOCKERHUB_REPO:$tag"; done docker push "$DOCKERHUB_REPO":"$BASE_TAG_NAME"
echo "Pushed $DOCKERHUB_REPO: ${BASE_TAGS[*]} ${TAGS[*]}" docker push "$DOCKERHUB_REPO":"$BASE_TAG_NAME_COMMIT"
docker push "$DOCKERHUB_REPO":"$TAG_NAME"
docker push "$DOCKERHUB_REPO":"$TAG_NAME_COMMIT"
# Keep only the last 14 builds of each tag family. The plain nightly- families echo "Pushed $DOCKERHUB_REPO:$BASE_TAG_NAME and $DOCKERHUB_REPO:$BASE_TAG_NAME_COMMIT"
# exclude the variant ones sharing their prefix; commit SHAs never start with "rocm". echo "Pushed $DOCKERHUB_REPO:$TAG_NAME and $DOCKERHUB_REPO:$TAG_NAME_COMMIT"
CLEANUP=.buildkite/scripts/cleanup-nightly-builds.sh
bash "$CLEANUP" "nightly-${VARIANT}-" "$DOCKERHUB_REPO"
bash "$CLEANUP" "base-nightly-${VARIANT}-" "$DOCKERHUB_REPO"
if [[ "$PUSH_DEFAULT" == "1" ]]; then
bash "$CLEANUP" "nightly-" "$DOCKERHUB_REPO" "nightly-rocm"
bash "$CLEANUP" "base-nightly-" "$DOCKERHUB_REPO" "base-nightly-rocm"
fi
+2 -2
View File
@@ -1,8 +1,8 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# Shared Dockerfile selection for the normal AMD CI build chain. # Shared Dockerfile selection for the normal AMD CI build chain.
# CI_ROCM_DOCKERFILE_BASE and CI_ROCM_DOCKERFILE accept repository-relative paths. # CI_ROCM_DOCKERFILE_BASE and CI_ROCM_DOCKERFILE accept repository-relative paths.
# For the legacy ROCm 7.2 stack, set CI_ROCM_DOCKERFILE_BASE=docker/Dockerfile.rocm_72_base # For Rock, set CI_ROCM_DOCKERFILE_BASE=docker/Dockerfile.rock_base and
# and CI_ROCM_DOCKERFILE=docker/Dockerfile.rocm_72. # CI_ROCM_DOCKERFILE=docker/Dockerfile.rock.
# Custom recipes must preserve the CI targets and artifact layout. Additional # Custom recipes must preserve the CI targets and artifact layout. Additional
# COPY inputs also need the corresponding *_CONTENT_FILES overrides. # COPY inputs also need the corresponding *_CONTENT_FILES overrides.
-78
View File
@@ -1,78 +0,0 @@
#!/usr/bin/env bash
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
#
# Derive everything the release pipeline needs about a ROCm stack from its base
# Dockerfile, so the stack's image is the single input to every ROCm script.
#
# Usage: source .buildkite/scripts/rocm/stack.sh <base-dockerfile>
# e.g. docker/Dockerfile.rocm_base (ROCm 10.0) or docker/Dockerfile.rocm_72_base
#
# Exports:
# ROCM_STACK_BASE_DOCKERFILE the base Dockerfile
# ROCM_STACK_DOCKERFILE the matching vLLM Dockerfile (base name minus _base)
# ROCM_STACK_THEROCK 1 if the ROCm SDK comes from TheRock wheels
# ROCM_STACK_VERSION ROCm version, e.g. 10.0.0 or 7.2.3
# ROCM_STACK_VARIANT wheel variant and Docker Hub tag flavor from major.minor,
# same rule as setup.py (rocm100, rocm72)
# ROCM_STACK_IMAGE_KEY hash of the base Dockerfile (base image cache key)
# ROCM_STACK_WHEEL_KEY hash of the base Dockerfile plus files it COPYs from
# the repo (base wheel cache key)
# ROCM_STACK_ECR_BASE ECR tag of the cached base image
# ROCM_STACK_ECR_IMAGE ECR tag of this commit's release image
# ROCM_STACK_BASE_WHEELS_DIR local dir for base wheels (also the artifact path)
# ROCM_STACK_VLLM_WHEEL_DIR local dir for the vLLM wheel (also the artifact path)
rocm_stack_init() {
local base="$1"
if [[ ! -f "$base" ]]; then
echo "ROCm base Dockerfile not found: $base" >&2
return 2
fi
ROCM_STACK_BASE_DOCKERFILE="$base"
ROCM_STACK_DOCKERFILE="${base%_base}"
local sdk
sdk=$(sed -nE 's/^ARG ROCM_SDK_VERSION=([0-9.]+).*/\1/p' "$base" | head -1)
if [[ -n "$sdk" ]]; then
ROCM_STACK_THEROCK=1
ROCM_STACK_VERSION="$sdk"
else
ROCM_STACK_THEROCK=0
# BASE_IMAGE format: rocm/dev-ubuntu-22.04:7.2.3-complete -> 7.2.3
ROCM_STACK_VERSION=$(sed -nE 's/^ARG BASE_IMAGE=.*:([0-9]+(\.[0-9]+)+).*/\1/p' "$base" | head -1)
fi
if [[ -z "$ROCM_STACK_VERSION" ]]; then
echo "Could not determine the ROCm version from $base" >&2
return 2
fi
local major minor
IFS=. read -r major minor _ <<< "$ROCM_STACK_VERSION"
ROCM_STACK_VARIANT="rocm${major}${minor}"
ROCM_STACK_IMAGE_KEY=$(sha256sum "$base" | cut -c1-16)
local inputs
inputs=$(grep -E '^COPY ' "$base" | grep -v -- '--from' | awk '{print $2}' || true)
if [[ -z "$inputs" ]]; then
ROCM_STACK_WHEEL_KEY="$ROCM_STACK_IMAGE_KEY"
else
# shellcheck disable=SC2086
ROCM_STACK_WHEEL_KEY=$(sha256sum "$base" $inputs | sha256sum | cut -c1-16)
fi
local ecr="public.ecr.aws/q9t5s3a7/vllm-release-repo"
ROCM_STACK_ECR_BASE="${ecr}:${ROCM_STACK_IMAGE_KEY}-${ROCM_STACK_VARIANT}-base"
ROCM_STACK_ECR_IMAGE="${ecr}:${BUILDKITE_COMMIT:-unknown}-${ROCM_STACK_VARIANT}"
ROCM_STACK_BASE_WHEELS_DIR="artifacts/${ROCM_STACK_VARIANT}-base-wheels"
ROCM_STACK_VLLM_WHEEL_DIR="artifacts/${ROCM_STACK_VARIANT}-vllm-wheel"
export ROCM_STACK_BASE_DOCKERFILE ROCM_STACK_DOCKERFILE ROCM_STACK_THEROCK \
ROCM_STACK_VERSION ROCM_STACK_VARIANT ROCM_STACK_IMAGE_KEY \
ROCM_STACK_WHEEL_KEY ROCM_STACK_ECR_BASE ROCM_STACK_ECR_IMAGE \
ROCM_STACK_BASE_WHEELS_DIR ROCM_STACK_VLLM_WHEEL_DIR
}
if [[ $# -gt 0 ]]; then
rocm_stack_init "$1"
fi
+20 -66
View File
@@ -4,31 +4,21 @@
# #
# Upload ROCm wheels to S3 with proper index generation # Upload ROCm wheels to S3 with proper index generation
# #
# Usage: upload-rocm-wheels.sh <base-dockerfile>
# base-dockerfile the ROCm stack, e.g. docker/Dockerfile.rocm_base (see rocm/stack.sh)
#
# Required environment variables: # Required environment variables:
# AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY (or IAM role) # AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY (or IAM role)
# S3_BUCKET (default: vllm-wheels) # S3_BUCKET (default: vllm-wheels)
# #
# S3 path structure: # S3 path structure:
# s3://vllm-wheels/rocm/{commit}/<variant>-wheels/ - This stack's wheels for this commit # s3://vllm-wheels/rocm/{commit}/ - All wheels for this commit
# s3://vllm-wheels/rocm/{commit}/<variant>/ - Index per ROCm variant (e.g. rocm100) # s3://vllm-wheels/rocm/nightly/ - Index pointing to latest nightly
# s3://vllm-wheels/rocm/nightly/<variant>/ - Index pointing to latest nightly # s3://vllm-wheels/rocm/{version}/ - Index for release versions
# s3://vllm-wheels/rocm/{version}/<variant>/ - Index for release versions
# The top-level index.html at each prefix lists every variant present, since
# several ROCm stacks publish into the same prefixes.
set -ex set -ex
# ======== Configuration ======== # ======== Configuration ========
BUCKET="${S3_BUCKET:-vllm-wheels}" BUCKET="${S3_BUCKET:-vllm-wheels}"
# shellcheck source=.buildkite/scripts/rocm/stack.sh
source "$(dirname "${BASH_SOURCE[0]}")/rocm/stack.sh" "${1:?Usage: $0 <base-dockerfile>}"
ROCM_SUBPATH="rocm/${BUILDKITE_COMMIT}" ROCM_SUBPATH="rocm/${BUILDKITE_COMMIT}"
WHEEL_DIRS="$ROCM_STACK_BASE_WHEELS_DIR $ROCM_STACK_VLLM_WHEEL_DIR" S3_COMMIT_PREFIX="s3://$BUCKET/$ROCM_SUBPATH/"
WHEEL_SUBPATH="$ROCM_SUBPATH/${ROCM_STACK_VARIANT}-wheels"
S3_WHEEL_PREFIX="s3://$BUCKET/$WHEEL_SUBPATH/"
INDICES_OUTPUT_DIR="rocm-indices" INDICES_OUTPUT_DIR="rocm-indices"
echo "========================================" echo "========================================"
@@ -59,11 +49,9 @@ source .buildkite/scripts/lib/manylinux.sh
# ======== Part 1: Collect and prepare wheels ======== # ======== Part 1: Collect and prepare wheels ========
# Collect all wheels # Collect all wheels
rm -rf all-rocm-wheels "$INDICES_OUTPUT_DIR"
mkdir -p all-rocm-wheels mkdir -p all-rocm-wheels
for dir in $WHEEL_DIRS; do cp artifacts/rocm-base-wheels/*.whl all-rocm-wheels/ 2>/dev/null || true
cp "$dir"/*.whl all-rocm-wheels/ 2>/dev/null || true cp artifacts/rocm-vllm-wheel/*.whl all-rocm-wheels/ 2>/dev/null || true
done
WHEEL_COUNT=$(find all-rocm-wheels -maxdepth 1 -name '*.whl' 2>/dev/null | wc -l) WHEEL_COUNT=$(find all-rocm-wheels -maxdepth 1 -name '*.whl' 2>/dev/null | wc -l)
echo "Total wheels to upload: $WHEEL_COUNT" echo "Total wheels to upload: $WHEEL_COUNT"
@@ -96,9 +84,9 @@ ls -lh all-rocm-wheels/
# ======== Part 2: Upload wheels to S3 ======== # ======== Part 2: Upload wheels to S3 ========
echo "" echo ""
echo "Uploading wheels to $S3_WHEEL_PREFIX" echo "Uploading wheels to $S3_COMMIT_PREFIX"
for wheel in all-rocm-wheels/*.whl; do for wheel in all-rocm-wheels/*.whl; do
aws s3 cp "$wheel" "$S3_WHEEL_PREFIX" aws s3 cp "$wheel" "$S3_COMMIT_PREFIX"
done done
# ======== Part 3: Generate and upload indices ======== # ======== Part 3: Generate and upload indices ========
@@ -107,63 +95,28 @@ done
echo "" echo ""
echo "Generating indices..." echo "Generating indices..."
obj_json="rocm-objects.json" obj_json="rocm-objects.json"
aws s3api list-objects-v2 --bucket "$BUCKET" --prefix "$WHEEL_SUBPATH/" --delimiter / --output json > "$obj_json" aws s3api list-objects-v2 --bucket "$BUCKET" --prefix "$ROCM_SUBPATH/" --delimiter / --output json > "$obj_json"
mkdir -p "$INDICES_OUTPUT_DIR" mkdir -p "$INDICES_OUTPUT_DIR"
# Use the existing generate-nightly-index.py # Use the existing generate-nightly-index.py
# HACK: Replace regex module with stdlib re (same as CUDA script) # HACK: Replace regex module with stdlib re (same as CUDA script)
sed -i 's/import regex as re/import re/g' .buildkite/scripts/generate-nightly-index.py sed -i 's/import regex as re/import re/g' .buildkite/scripts/generate-nightly-index.py
sed -i 's/import regex as re/import re/g' tools/vllm-rocm/therock_wheels.py
sed -i 's/import pybase64 as base64/import base64/g' tools/vllm-rocm/therock_wheels.py
INDEX_ARGS=(--version "$ROCM_SUBPATH" --wheel-dir "${WHEEL_SUBPATH#rocm/}")
# TheRock's SDK, device kernels and torchvision/torchaudio are linked, not hosted
if [[ "$ROCM_STACK_THEROCK" == "1" ]]; then
$PYTHON tools/vllm-rocm/therock_wheels.py external-links \
--dockerfile "$ROCM_STACK_BASE_DOCKERFILE" \
--torch-wheel "$(ls "$ROCM_STACK_BASE_WHEELS_DIR"/torch-*.whl)" > therock-external-links.txt
cat therock-external-links.txt
INDEX_ARGS+=(--external-links therock-external-links.txt)
fi
$PYTHON .buildkite/scripts/generate-nightly-index.py \ $PYTHON .buildkite/scripts/generate-nightly-index.py \
"${INDEX_ARGS[@]}" \ --version "$ROCM_SUBPATH" \
--current-objects "$obj_json" \ --current-objects "$obj_json" \
--output-dir "$INDICES_OUTPUT_DIR" \ --output-dir "$INDICES_OUTPUT_DIR" \
--comment "ROCm commit $BUILDKITE_COMMIT" --comment "ROCm commit $BUILDKITE_COMMIT"
# Upload this build's variant indices under an S3 prefix (e.g. rocm/nightly), then # Upload indices to commit directory
# rebuild that prefix's top-level project list from every variant now present. echo "Uploading indices to $S3_COMMIT_PREFIX"
publish_indices() { aws s3 cp --recursive "$INDICES_OUTPUT_DIR/" "$S3_COMMIT_PREFIX"
local prefix="$1"
local listing variants merged
echo "Uploading indices to s3://$BUCKET/$prefix/"
aws s3 cp --recursive "$INDICES_OUTPUT_DIR/" "s3://$BUCKET/$prefix/" --exclude "index.html"
listing=$(aws s3api list-objects-v2 --bucket "$BUCKET" --prefix "$prefix/" \
--delimiter / --query 'CommonPrefixes[].Prefix' --output text)
variants=$(tr '\t' '\n' <<< "$listing" | sed -n "s|^$prefix/\(rocm[0-9]*\)/$|\1|p" | sort -u)
merged=$(mktemp)
{
echo '<!DOCTYPE html>'
echo '<html>'
echo ' <meta name="pypi:repository-version" content="1.0">'
echo ' <body>'
for variant in $variants; do
echo " <a href=\"$variant/\">$variant/</a><br/>"
done
echo ' </body>'
echo '</html>'
} > "$merged"
echo "Variants under $prefix/: $(echo "$variants" | tr '\n' ' ')"
aws s3 cp "$merged" "s3://$BUCKET/$prefix/index.html"
rm -f "$merged"
}
publish_indices "$ROCM_SUBPATH"
# Only scheduled nightly builds should update the moving nightly index. # Only scheduled nightly builds should update the moving nightly index.
if [[ "${NIGHTLY:-0}" == "1" ]]; then if [[ "${NIGHTLY:-0}" == "1" ]]; then
publish_indices "rocm/nightly" echo "Updating rocm/nightly/ index..."
aws s3 cp --recursive "$INDICES_OUTPUT_DIR/" "s3://$BUCKET/rocm/nightly/"
fi fi
# Extract version from vLLM wheel and update version-specific index # Extract version from vLLM wheel and update version-specific index
@@ -176,7 +129,8 @@ if [ -n "$VLLM_WHEEL" ]; then
echo "Pure version: $PURE_VERSION" echo "Pure version: $PURE_VERSION"
if [[ "$VERSION" != *"dev"* ]]; then if [[ "$VERSION" != *"dev"* ]]; then
publish_indices "rocm/$PURE_VERSION" echo "Updating rocm/$PURE_VERSION/ index..."
aws s3 cp --recursive "$INDICES_OUTPUT_DIR/" "s3://$BUCKET/rocm/$PURE_VERSION/"
fi fi
fi fi
@@ -188,14 +142,14 @@ echo "ROCm Wheel Upload Complete!"
echo "========================================" echo "========================================"
echo "" echo ""
echo "Wheels available at:" echo "Wheels available at:"
echo " s3://$BUCKET/$WHEEL_SUBPATH/" echo " s3://$BUCKET/$ROCM_SUBPATH/"
echo "" echo ""
echo "Install command (by commit):" echo "Install command (by commit):"
echo " pip install vllm --extra-index-url https://${BUCKET}.s3.amazonaws.com/$ROCM_SUBPATH/$ROCM_STACK_VARIANT/" echo " pip install vllm --extra-index-url https://${BUCKET}.s3.amazonaws.com/$ROCM_SUBPATH/"
echo "" echo ""
if [[ "${NIGHTLY:-0}" == "1" ]]; then if [[ "${NIGHTLY:-0}" == "1" ]]; then
echo "Install command (nightly):" echo "Install command (nightly):"
echo " pip install vllm --extra-index-url https://${BUCKET}.s3.amazonaws.com/rocm/nightly/$ROCM_STACK_VARIANT/" echo " pip install vllm --extra-index-url https://${BUCKET}.s3.amazonaws.com/rocm/nightly/"
fi fi
echo "" echo ""
echo "Wheel count: $WHEEL_COUNT" echo "Wheel count: $WHEEL_COUNT"
+300 -212
View File
@@ -399,6 +399,27 @@ steps:
commands: commands:
- pytest -v -s v1/e2e/spec_decode/draft_model/ - pytest -v -s v1/e2e/spec_decode/draft_model/
- label: ":amd: (MI250) Spec Decode Speculators + MTP" # TBD
timeout_in_minutes: 75
mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]
dind: false
agent_pool: mi250_1
optional: true
working_dir: "/vllm-workspace/tests"
source_file_dependencies:
- vllm/v1/spec_decode/
- vllm/v1/worker/gpu/spec_decode/
- vllm/model_executor/model_loader/
- vllm/v1/sample/
- vllm/model_executor/layers/
- vllm/transformers_utils/configs/speculators/
- tests/v1/e2e/spec_decode/
- vllm/platforms/rocm.py
- vllm/v1/attention/backends/
commands:
- pytest -v -s v1/e2e/spec_decode/speculators/
- pytest -v -s v1/e2e/spec_decode/mtp/
#------------------------------------------------------------- mi250 · misc ------------------------------------------------------------# #------------------------------------------------------------- mi250 · misc ------------------------------------------------------------#
- label: ":amd: (MI300) Python-only Installation" # TBD - label: ":amd: (MI300) Python-only Installation" # TBD
@@ -1073,6 +1094,25 @@ steps:
#----------------------------------------------------------- mi300 · evals -----------------------------------------------------------# #----------------------------------------------------------- mi300 · evals -----------------------------------------------------------#
- label: ":amd: (MI300) LM Eval Small Models" # TBD
timeout_in_minutes: 45
mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
dind: false
agent_pool: mi300_1
working_dir: "/vllm-workspace/tests"
source_file_dependencies:
- csrc/
- vllm/model_executor/layers/quantization
- vllm/model_executor/models/
- vllm/model_executor/model_loader/
- vllm/v1/attention/backends/
- vllm/v1/attention/selector.py
- vllm/_aiter_ops.py
- vllm/platforms/rocm.py
optional: true
commands:
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-small.txt
- label: ":amd: (MI300) DeepSeek V2-Lite Prefetch Offload Accuracy" # TBD - label: ":amd: (MI300) DeepSeek V2-Lite Prefetch Offload Accuracy" # TBD
timeout_in_minutes: 30 timeout_in_minutes: 30
mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300] mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
@@ -1113,6 +1153,23 @@ steps:
commands: commands:
- pytest -s -v evals/mrcr/test_mrcr_correctness.py --config-list-file=evals/mrcr/configs/models-small.txt - pytest -s -v evals/mrcr/test_mrcr_correctness.py --config-list-file=evals/mrcr/configs/models-small.txt
- label: ":amd: (MI300) LM Eval KV-Offload" # TBD
timeout_in_minutes: 55
mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
dind: false
agent_pool: mi300_1
optional: true
working_dir: "/vllm-workspace/tests"
source_file_dependencies:
- vllm/distributed/kv_transfer/kv_connector/v1/offloading/
- vllm/distributed/kv_transfer/kv_connector/v1/simple_cpu_offload_connector.py
- vllm/v1/kv_offload/
- vllm/v1/simple_kv_offload/
- tests/evals/gsm8k/test_gsm8k_offloading.py
- vllm/platforms/rocm.py
commands:
- pytest -s -v evals/gsm8k/test_gsm8k_offloading.py -k "nemotron-h-8b or gemma-4-e4b-it"
- label: ":amd: (MI300) LM Eval KV-Offload Medium" # TBD - label: ":amd: (MI300) LM Eval KV-Offload Medium" # TBD
timeout_in_minutes: 70 timeout_in_minutes: 70
mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300] mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
@@ -1131,6 +1188,24 @@ steps:
commands: commands:
- pytest -s -v evals/gsm8k/test_gsm8k_offloading.py -k "qwen3.5-35b or deepseek-v2-lite" - pytest -s -v evals/gsm8k/test_gsm8k_offloading.py -k "qwen3.5-35b or deepseek-v2-lite"
- label: ":amd: (MI300) LM Eval KV-Offload Large" # TBD
timeout_in_minutes: 65
mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
dind: false
agent_pool: mi300_4
num_gpus: 4
optional: true
working_dir: "/vllm-workspace/tests"
source_file_dependencies:
- vllm/distributed/kv_transfer/kv_connector/v1/offloading/
- vllm/distributed/kv_transfer/kv_connector/v1/simple_cpu_offload_connector.py
- vllm/v1/kv_offload/
- vllm/v1/simple_kv_offload/
- tests/evals/gsm8k/test_gsm8k_offloading.py
- vllm/platforms/rocm.py
commands:
- VLLM_ROCM_USE_AITER=1 pytest -s -v evals/gsm8k/test_gsm8k_offloading.py -k "deepseek-v4-flash"
- label: ":amd: (MI300) LM Eval Small Models Harness" # TBD - label: ":amd: (MI300) LM Eval Small Models Harness" # TBD
timeout_in_minutes: 35 timeout_in_minutes: 35
mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300] mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
@@ -1308,6 +1383,52 @@ steps:
commands: commands:
- VLLM_ENGINE_READY_TIMEOUT_S=1800 bash .buildkite/scripts/scheduled_integration_test/qwen3_next_mtp_async_eplb.sh 0.8 1319 8040 - VLLM_ENGINE_READY_TIMEOUT_S=1800 bash .buildkite/scripts/scheduled_integration_test/qwen3_next_mtp_async_eplb.sh 0.8 1319 8040
- label: ":amd: (MI300) LM Eval Large Models" # TBD
timeout_in_minutes: 65
mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
dind: false
agent_pool: mi300_8
optional: true
num_gpus: 8
working_dir: "/vllm-workspace/tests"
source_file_dependencies:
- vllm/model_executor/models/
- vllm/model_executor/model_loader/
- vllm/model_executor/layers/quantization/
- vllm/v1/attention/backends/
- vllm/v1/attention/selector.py
- vllm/model_executor/layers/layernorm.py
- csrc/
- vllm/_aiter_ops.py
- vllm/platforms/rocm.py
- tests/evals/
commands:
- export VLLM_WORKER_MULTIPROC_METHOD=spawn
- export PYTORCH_ROCM_ARCH=gfx942
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-mi3xx.txt
- label: ":amd: (MI300) LM Eval Large Models ROCm Harness" # TBD
timeout_in_minutes: 40
mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
dind: false
agent_pool: mi300_8
optional: true
num_gpus: 8
working_dir: "/vllm-workspace/.buildkite/lm-eval-harness"
source_file_dependencies:
- vllm/model_executor/models/
- vllm/model_executor/model_loader/
- vllm/model_executor/layers/quantization/
- vllm/v1/attention/backends/
- vllm/v1/attention/selector.py
- vllm/model_executor/layers/layernorm.py
- csrc/
- vllm/_aiter_ops.py
- vllm/platforms/rocm.py
commands:
- export VLLM_WORKER_MULTIPROC_METHOD=spawn
- pytest -s -v test_lm_eval_correctness.py --config-list-file=configs/models-large-rocm.txt --tp-size=8 --timeout=5400
- label: ":amd: (MI300) LM Eval Large Models FP8" # TBD - label: ":amd: (MI300) LM Eval Large Models FP8" # TBD
timeout_in_minutes: 40 timeout_in_minutes: 40
mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300] mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
@@ -1818,6 +1939,20 @@ steps:
commands: commands:
- pytest -v -s models/language/pooling_mteb_test - pytest -v -s models/language/pooling_mteb_test
- label: ":amd: (MI300) Language Models (Extended Pooling) Shard %N"
timeout_in_minutes: 95
parallelism: 4
mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
dind: false
agent_pool: mi300_1
optional: true
working_dir: "/vllm-workspace/tests"
source_file_dependencies:
- vllm/
- tests/models/language/pooling
commands:
- pytest -v -s models/language/pooling -m 'not core_model' --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT --shard-id=$$BUILDKITE_PARALLEL_JOB
- label: ":amd: (MI300) Language Models (Extended Generation)" # TBD - label: ":amd: (MI300) Language Models (Extended Generation)" # TBD
timeout_in_minutes: 105 timeout_in_minutes: 105
mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300] mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
@@ -1835,6 +1970,21 @@ steps:
#---------------------------------------------------- mi300 · models / multimodal ----------------------------------------------------# #---------------------------------------------------- mi300 · models / multimodal ----------------------------------------------------#
- label: ":amd: (MI300) Multimodal Models (Extended Generation 1)" # TBD
timeout_in_minutes: 100
mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
dind: false
agent_pool: mi300_1
optional: true
working_dir: "/vllm-workspace/tests"
source_file_dependencies:
- vllm/
- tests/models/multimodal/generation
- tests/models/multimodal/test_mapping.py
commands:
- pytest -v -s models/multimodal/generation -m 'not core_model' --ignore models/multimodal/generation/test_common.py
- pytest -v -s models/multimodal/test_mapping.py
- label: ":amd: (MI300) Multimodal Models (Extended Generation 2) Shard %N" - label: ":amd: (MI300) Multimodal Models (Extended Generation 2) Shard %N"
timeout_in_minutes: 100 timeout_in_minutes: 100
parallelism: 4 parallelism: 4
@@ -1863,6 +2013,19 @@ steps:
commands: commands:
- pytest -v -s models/multimodal/generation/test_common.py -m 'split(group=1) and not core_model' - pytest -v -s models/multimodal/generation/test_common.py -m 'split(group=1) and not core_model'
- label: ":amd: (MI300) Multimodal Models (PPL)"
timeout_in_minutes: 90
mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
dind: false
agent_pool: mi300_1
optional: true
working_dir: "/vllm-workspace/tests"
source_file_dependencies:
- vllm/
- tests/models/multimodal/generation_ppl_test/
commands:
- pytest -v -s models/multimodal/generation_ppl_test/
- label: ":amd: (MI300) Multimodal Models (Standard) 3: llava + qwen2_vl" # TBD - label: ":amd: (MI300) Multimodal Models (Standard) 3: llava + qwen2_vl" # TBD
timeout_in_minutes: 55 timeout_in_minutes: 55
mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300] mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
@@ -1894,6 +2057,19 @@ steps:
commands: commands:
- pytest -v -s models/multimodal/processing/test_tensor_schema.py --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT --shard-id=$$BUILDKITE_PARALLEL_JOB - pytest -v -s models/multimodal/processing/test_tensor_schema.py --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT --shard-id=$$BUILDKITE_PARALLEL_JOB
- label: ":amd: (MI300) Multimodal Models (Extended Pooling)" # TBD
timeout_in_minutes: 75
mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
dind: false
agent_pool: mi300_1
optional: true
working_dir: "/vllm-workspace/tests"
source_file_dependencies:
- vllm/
- tests/models/multimodal/pooling
commands:
- pytest -v -s models/multimodal/pooling -m 'not core_model'
#----------------------------------------------------- mi300 · models / quantized -----------------------------------------------------# #----------------------------------------------------- mi300 · models / quantized -----------------------------------------------------#
- label: ":amd: (MI300) Quantized Models" # TBD - label: ":amd: (MI300) Quantized Models" # TBD
@@ -2357,6 +2533,27 @@ steps:
- VLLM_TP_SIZE=4 VLLM_NEEDLE_TRIALS=2 pytest -v -s v1/determinism/test_batch_invariance.py::test_v1_generation_is_deterministic_across_batch_sizes_with_needle -k TRITON_ATTN - VLLM_TP_SIZE=4 VLLM_NEEDLE_TRIALS=2 pytest -v -s v1/determinism/test_batch_invariance.py::test_v1_generation_is_deterministic_across_batch_sizes_with_needle -k TRITON_ATTN
- VLLM_TEST_MODEL=allenai/OLMoE-1B-7B-0924 VLLM_TEST_SERVER_ARGS="--data-parallel-size 4 --enable-expert-parallel --max-model-len 4096" pytest -v -s v1/determinism/test_online_batch_invariance.py - VLLM_TEST_MODEL=allenai/OLMoE-1B-7B-0924 VLLM_TEST_SERVER_ARGS="--data-parallel-size 4 --enable-expert-parallel --max-model-len 4096" pytest -v -s v1/determinism/test_online_batch_invariance.py
- label: ":amd: (MI300) Spec Decode Speculators + MTP" # TBD
timeout_in_minutes: 75
mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
dind: false
agent_pool: mi300_1
optional: true
working_dir: "/vllm-workspace/tests"
source_file_dependencies:
- vllm/v1/spec_decode/
- vllm/v1/worker/gpu/spec_decode/
- vllm/model_executor/model_loader/
- vllm/v1/sample/
- vllm/model_executor/layers/
- vllm/transformers_utils/configs/speculators/
- tests/v1/e2e/spec_decode/
- vllm/platforms/rocm.py
- vllm/v1/attention/backends/
commands:
- pytest -v -s v1/e2e/spec_decode/speculators/
- pytest -v -s v1/e2e/spec_decode/mtp/
- label: ":amd: (MI300) Spec Decode DeepSeek MTP Parallel Load" - label: ":amd: (MI300) Spec Decode DeepSeek MTP Parallel Load"
timeout_in_minutes: 45 timeout_in_minutes: 45
mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300] mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
@@ -2384,6 +2581,60 @@ steps:
commands: commands:
- pytest -v -s v1/e2e/spec_decode/test_mtp_parallel_load.py - pytest -v -s v1/e2e/spec_decode/test_mtp_parallel_load.py
- label: ":amd: (MI300) Spec Decode AL DFlash Nightly"
timeout_in_minutes: 120
mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
dind: false
agent_pool: mi300_1
optional: true
working_dir: "/vllm-workspace/tests"
source_file_dependencies:
- vllm/v1/spec_decode/
- vllm/v1/worker/gpu/spec_decode/
- vllm/model_executor/models/qwen3_dflash.py
- vllm/model_executor/models/qwen3_dflash2.py
- vllm/model_executor/models/qwen3_5.py
- vllm/model_executor/models/laguna_dflash.py
- tests/v1/e2e/spec_decode/
- vllm/v1/attention/
- vllm/v1/sample/
- vllm/model_executor/model_loader/
- vllm/model_executor/layers/
- vllm/model_executor/models/qwen2.py
- vllm/model_executor/models/qwen3.py
- vllm/model_executor/models/laguna.py
- vllm/platforms/rocm.py
commands:
- pytest -v -s v1/e2e/spec_decode/acceptance_rates/dflash/ -k "not dflash2"
- label: ":amd: (MI300) Spec Decode AL DSpark Nightly"
timeout_in_minutes: 90
mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
dind: false
agent_pool: mi300_1
optional: true
working_dir: "/vllm-workspace/tests"
source_file_dependencies:
- vllm/v1/spec_decode/
- vllm/v1/worker/gpu/spec_decode/
- vllm/model_executor/models/qwen3_dspark.py
- vllm/model_executor/models/gemma4_dspark.py
- tests/v1/e2e/spec_decode/
- vllm/v1/attention/
- vllm/v1/sample/
- vllm/model_executor/model_loader/
- vllm/model_executor/layers/
- vllm/model_executor/models/qwen2.py
- vllm/model_executor/models/qwen3.py
- vllm/model_executor/models/qwen3_dflash.py
- vllm/model_executor/models/gemma4.py
- vllm/model_executor/models/gemma4_mm.py
- vllm/model_executor/models/gemma4_unified.py
- vllm/model_executor/models/gemma4_mtp.py
- vllm/platforms/rocm.py
commands:
- pytest -v -s v1/e2e/spec_decode/acceptance_rates/dspark/
- label: ":amd: (MI300) V1 Spec Decode" # TBD - label: ":amd: (MI300) V1 Spec Decode" # TBD
timeout_in_minutes: 50 timeout_in_minutes: 50
mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300] mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
@@ -2468,6 +2719,39 @@ steps:
- pytest -v -s v1/engine --ignore v1/engine/test_preprocess_error_handling.py - pytest -v -s v1/engine --ignore v1/engine/test_preprocess_error_handling.py
- pytest -v -s v1/test_tensor_ipc_queue.py - pytest -v -s v1/test_tensor_ipc_queue.py
- label: ":amd: (MI300) Speculators Correctness Nightly"
timeout_in_minutes: 40
mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
dind: false
agent_pool: mi300_1
optional: true
working_dir: "/vllm-workspace/tests"
source_file_dependencies:
- vllm/config/speculative.py
- vllm/engine/arg_utils.py
- vllm/transformers_utils/config.py
- vllm/transformers_utils/configs/speculators/
- vllm/v1/spec_decode/
- vllm/v1/worker/gpu/spec_decode/
- vllm/v1/worker/gpu_model_runner.py
- vllm/v1/sample/
- vllm/v1/attention/backends/
- vllm/v1/attention/selector.py
- vllm/model_executor/model_loader/
- vllm/model_executor/layers/
- vllm/model_executor/models/llama_eagle3.py
- vllm/model_executor/models/qwen3.py
- vllm/model_executor/models/qwen3_dflash.py
- vllm/model_executor/models/registry.py
- vllm/_aiter_ops.py
- tests/evals/gsm8k/
- tests/v1/spec_decode/test_speculators_correctness.py
- vllm/platforms/rocm.py
- vllm/v1/worker/gpu/spec_decode/dflash/
commands:
- export VLLM_ALLOW_INSECURE_SERIALIZATION=1
- pytest -v -s v1/spec_decode/test_speculators_correctness.py -m slow_test
- label: ":amd: (MI300) Extract Hidden States Integration" # TBD - label: ":amd: (MI300) Extract Hidden States Integration" # TBD
timeout_in_minutes: 30 timeout_in_minutes: 30
mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300] mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
@@ -2822,6 +3106,22 @@ steps:
- uv pip install --system -r /vllm-workspace/requirements/kv_connectors_rocm.txt - uv pip install --system -r /vllm-workspace/requirements/kv_connectors_rocm.txt
- ATTENTION_BACKEND=TRITON_ATTN bash v1/kv_connector/nixl_integration/run_multi_connector_edge_case_test.sh - ATTENTION_BACKEND=TRITON_ATTN bash v1/kv_connector/nixl_integration/run_multi_connector_edge_case_test.sh
- label: ":amd: (MI300) V1 E2E Hybrid Chunked Prefill" # TBD
timeout_in_minutes: 35
mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
dind: false
agent_pool: mi300_4
num_gpus: 4
optional: true
source_file_dependencies:
- vllm/v1/attention/backends/utils.py
- vllm/v1/worker/gpu_model_runner.py
- tests/v1/e2e/test_hybrid_chunked_prefill.py
- vllm/v1/core/
working_dir: /vllm-workspace/tests
commands:
- pytest -v -s v1/e2e/test_hybrid_chunked_prefill.py
######################################################################################################################################### #########################################################################################################################################
# # # #
# MI355 (gfx950) tests # # MI355 (gfx950) tests #
@@ -4304,82 +4604,6 @@ steps:
- export VLLM_WORKER_MULTIPROC_METHOD=spawn - export VLLM_WORKER_MULTIPROC_METHOD=spawn
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-mi355-nemotron-3-super.txt - pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-mi355-nemotron-3-super.txt
- label: ":amd: (MI355) LM Eval Small Models" # TBD
timeout_in_minutes: 45
mirror_hardwares: [amdexperimental, amdproduction, amdgfx950nightly, amdmi355]
dind: false
agent_pool: mi355_dpx
working_dir: "/vllm-workspace/tests"
source_file_dependencies:
- csrc/
- vllm/model_executor/layers/quantization
- vllm/model_executor/models/
- vllm/model_executor/model_loader/
- vllm/v1/attention/backends/
- vllm/v1/attention/selector.py
- vllm/_aiter_ops.py
- vllm/platforms/rocm.py
optional: true
commands:
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-small.txt
- label: ":amd: (MI355 DPX) LM Eval KV-Offload" # TBD
timeout_in_minutes: 55
mirror_hardwares: [amdexperimental, amdproduction, amdgfx950nightly, amdmi355]
dind: false
agent_pool: mi355_dpx
optional: true
working_dir: "/vllm-workspace/tests"
source_file_dependencies:
- vllm/distributed/kv_transfer/kv_connector/v1/offloading/
- vllm/distributed/kv_transfer/kv_connector/v1/simple_cpu_offload_connector.py
- vllm/v1/kv_offload/
- vllm/v1/simple_kv_offload/
- tests/evals/gsm8k/test_gsm8k_offloading.py
- vllm/platforms/rocm.py
commands:
- pytest -s -v evals/gsm8k/test_gsm8k_offloading.py -k "nemotron-h-8b or gemma-4-e4b-it"
- label: ":amd: (MI355) LM Eval KV-Offload Large" # TBD
timeout_in_minutes: 65
mirror_hardwares: [amdexperimental, amdproduction, amdgfx950nightly, amdmi355]
dind: false
agent_pool: mi355_4
num_gpus: 4
optional: true
working_dir: "/vllm-workspace/tests"
source_file_dependencies:
- vllm/distributed/kv_transfer/kv_connector/v1/offloading/
- vllm/distributed/kv_transfer/kv_connector/v1/simple_cpu_offload_connector.py
- vllm/v1/kv_offload/
- vllm/v1/simple_kv_offload/
- tests/evals/gsm8k/test_gsm8k_offloading.py
- vllm/platforms/rocm.py
commands:
- VLLM_ROCM_USE_AITER=1 pytest -s -v evals/gsm8k/test_gsm8k_offloading.py -k "deepseek-v4-flash"
- label: ":amd: (MI355) LM Eval Large Models ROCm Harness" # TBD
timeout_in_minutes: 40
mirror_hardwares: [amdexperimental, amdproduction, amdgfx950nightly, amdmi355]
dind: false
agent_pool: mi355_8
optional: true
num_gpus: 8
working_dir: "/vllm-workspace/.buildkite/lm-eval-harness"
source_file_dependencies:
- vllm/model_executor/models/
- vllm/model_executor/model_loader/
- vllm/model_executor/layers/quantization/
- vllm/v1/attention/backends/
- vllm/v1/attention/selector.py
- vllm/model_executor/layers/layernorm.py
- csrc/
- vllm/_aiter_ops.py
- vllm/platforms/rocm.py
commands:
- export VLLM_WORKER_MULTIPROC_METHOD=spawn
- pytest -s -v test_lm_eval_correctness.py --config-list-file=configs/models-large-rocm.txt --tp-size=8 --timeout=5400
#--------------------------------------------------------- mi355 · examples ----------------------------------------------------------# #--------------------------------------------------------- mi355 · examples ----------------------------------------------------------#
- label: ":amd: (MI355 DPX) Examples" # TBD - label: ":amd: (MI355 DPX) Examples" # TBD
@@ -5197,19 +5421,6 @@ steps:
- pytest models/multimodal/generation/test_memory_leak.py -m core_model - pytest models/multimodal/generation/test_memory_leak.py -m core_model
- cd .. && VLLM_WORKER_MULTIPROC_METHOD=spawn pytest -v -s tests/models/multimodal/generation/test_whisper.py -m core_model - cd .. && VLLM_WORKER_MULTIPROC_METHOD=spawn pytest -v -s tests/models/multimodal/generation/test_whisper.py -m core_model
- label: ":amd: (MI355 DPX) Multimodal Models (PPL)"
timeout_in_minutes: 90
mirror_hardwares: [amdexperimental, amdproduction, amdgfx950nightly, amdmi355]
dind: false
agent_pool: mi355_dpx
optional: true
working_dir: "/vllm-workspace/tests"
source_file_dependencies:
- vllm/
- tests/models/multimodal/generation_ppl_test/
commands:
- pytest -v -s models/multimodal/generation_ppl_test/
#----------------------------------------------------- mi355 · models / quantized -----------------------------------------------------# #----------------------------------------------------- mi355 · models / quantized -----------------------------------------------------#
- label: ":amd: (MI355 DPX) Quantized Models" # TBD - label: ":amd: (MI355 DPX) Quantized Models" # TBD
@@ -5354,27 +5565,6 @@ steps:
commands: commands:
- pytest -v -s v1/e2e/spec_decode/draft_model/ - pytest -v -s v1/e2e/spec_decode/draft_model/
- label: ":amd: (MI355 DPX) Spec Decode Speculators + MTP" # TBD
timeout_in_minutes: 75
mirror_hardwares: [amdexperimental, amdproduction, amdgfx950nightly, amdmi355]
dind: false
agent_pool: mi355_dpx
optional: true
working_dir: "/vllm-workspace/tests"
source_file_dependencies:
- vllm/v1/spec_decode/
- vllm/v1/worker/gpu/spec_decode/
- vllm/model_executor/model_loader/
- vllm/v1/sample/
- vllm/model_executor/layers/
- vllm/transformers_utils/configs/speculators/
- tests/v1/e2e/spec_decode/
- vllm/platforms/rocm.py
- vllm/v1/attention/backends/
commands:
- pytest -v -s v1/e2e/spec_decode/speculators/
- pytest -v -s v1/e2e/spec_decode/mtp/
- label: ":amd: (MI355 DPX) Spec Decode Eagle 1: DeepSeek + Qwen" - label: ":amd: (MI355 DPX) Spec Decode Eagle 1: DeepSeek + Qwen"
timeout_in_minutes: 60 timeout_in_minutes: 60
mirror_hardwares: [amdexperimental, amdproduction, amdgfx950nightly, amdmi355] mirror_hardwares: [amdexperimental, amdproduction, amdgfx950nightly, amdmi355]
@@ -5851,108 +6041,6 @@ steps:
- if [ "$$BUILDKITE_PARALLEL_JOB" = "1" ]; then pytest -v -s "v1/e2e/spec_decode/acceptance_rates/mtp_other/test_mtp.py::test_gemma4_mtp_acceptance_lengths[False]"; fi - if [ "$$BUILDKITE_PARALLEL_JOB" = "1" ]; then pytest -v -s "v1/e2e/spec_decode/acceptance_rates/mtp_other/test_mtp.py::test_gemma4_mtp_acceptance_lengths[False]"; fi
- if [ "$$BUILDKITE_PARALLEL_JOB" = "2" ]; then pytest -v -s "v1/e2e/spec_decode/acceptance_rates/mtp_other/test_mtp.py::test_gemma4_mtp_acceptance_lengths[True]"; fi - if [ "$$BUILDKITE_PARALLEL_JOB" = "2" ]; then pytest -v -s "v1/e2e/spec_decode/acceptance_rates/mtp_other/test_mtp.py::test_gemma4_mtp_acceptance_lengths[True]"; fi
- label: ":amd: (MI355 DPX) V1 E2E Hybrid Chunked Prefill" # TBD
timeout_in_minutes: 35
mirror_hardwares: [amdexperimental, amdproduction, amdgfx950nightly, amdmi355]
dind: false
agent_pool: mi355_dpx
optional: true
source_file_dependencies:
- vllm/v1/attention/backends/utils.py
- vllm/v1/worker/gpu_model_runner.py
- tests/v1/e2e/test_hybrid_chunked_prefill.py
- vllm/v1/core/
working_dir: /vllm-workspace/tests
commands:
- pytest -v -s v1/e2e/test_hybrid_chunked_prefill.py
- label: ":amd: (MI355 DPX) Speculators Correctness Nightly"
timeout_in_minutes: 40
mirror_hardwares: [amdexperimental, amdproduction, amdgfx950nightly, amdmi355]
dind: false
agent_pool: mi355_dpx
optional: true
working_dir: "/vllm-workspace/tests"
source_file_dependencies:
- vllm/config/speculative.py
- vllm/engine/arg_utils.py
- vllm/transformers_utils/config.py
- vllm/transformers_utils/configs/speculators/
- vllm/v1/spec_decode/
- vllm/v1/worker/gpu/spec_decode/
- vllm/v1/worker/gpu_model_runner.py
- vllm/v1/sample/
- vllm/v1/attention/backends/
- vllm/v1/attention/selector.py
- vllm/model_executor/model_loader/
- vllm/model_executor/layers/
- vllm/model_executor/models/llama_eagle3.py
- vllm/model_executor/models/qwen3.py
- vllm/model_executor/models/qwen3_dflash.py
- vllm/model_executor/models/registry.py
- vllm/_aiter_ops.py
- tests/evals/gsm8k/
- tests/v1/spec_decode/test_speculators_correctness.py
- vllm/platforms/rocm.py
- vllm/v1/worker/gpu/spec_decode/dflash/
commands:
- export VLLM_ALLOW_INSECURE_SERIALIZATION=1
- pytest -v -s v1/spec_decode/test_speculators_correctness.py -m slow_test
- label: ":amd: (MI355 DPX) Spec Decode AL DFlash Nightly"
timeout_in_minutes: 120
mirror_hardwares: [amdexperimental, amdproduction, amdgfx950nightly, amdmi355]
dind: false
agent_pool: mi355_dpx
optional: true
working_dir: "/vllm-workspace/tests"
source_file_dependencies:
- vllm/v1/spec_decode/
- vllm/v1/worker/gpu/spec_decode/
- vllm/model_executor/models/qwen3_dflash.py
- vllm/model_executor/models/qwen3_dflash2.py
- vllm/model_executor/models/qwen3_5.py
- vllm/model_executor/models/laguna_dflash.py
- tests/v1/e2e/spec_decode/
- vllm/v1/attention/
- vllm/v1/sample/
- vllm/model_executor/model_loader/
- vllm/model_executor/layers/
- vllm/model_executor/models/qwen2.py
- vllm/model_executor/models/qwen3.py
- vllm/model_executor/models/laguna.py
- vllm/platforms/rocm.py
commands:
- pytest -v -s v1/e2e/spec_decode/acceptance_rates/dflash/ -k "not dflash2"
- label: ":amd: (MI355 DPX) Spec Decode AL DSpark Nightly"
timeout_in_minutes: 90
mirror_hardwares: [amdexperimental, amdproduction, amdgfx950nightly, amdmi355]
dind: false
agent_pool: mi355_dpx
optional: true
working_dir: "/vllm-workspace/tests"
source_file_dependencies:
- vllm/v1/spec_decode/
- vllm/v1/worker/gpu/spec_decode/
- vllm/model_executor/models/qwen3_dspark.py
- vllm/model_executor/models/gemma4_dspark.py
- tests/v1/e2e/spec_decode/
- vllm/v1/attention/
- vllm/v1/sample/
- vllm/model_executor/model_loader/
- vllm/model_executor/layers/
- vllm/model_executor/models/qwen2.py
- vllm/model_executor/models/qwen3.py
- vllm/model_executor/models/qwen3_dflash.py
- vllm/model_executor/models/gemma4.py
- vllm/model_executor/models/gemma4_mm.py
- vllm/model_executor/models/gemma4_unified.py
- vllm/model_executor/models/gemma4_mtp.py
- vllm/platforms/rocm.py
commands:
- pytest -v -s v1/e2e/spec_decode/acceptance_rates/dspark/
#----------------------------------------------------------- mi355 · misc ------------------------------------------------------------# #----------------------------------------------------------- mi355 · misc ------------------------------------------------------------#
- label: ":amd: (MI355 DPX) Regression" # TBD - label: ":amd: (MI355 DPX) Regression" # TBD
+2 -2
View File
@@ -238,9 +238,9 @@ steps:
- pytest -v -s v1/e2e/test_hybrid_chunked_prefill.py - pytest -v -s v1/e2e/test_hybrid_chunked_prefill.py
mirror: mirror:
amd: amd:
label: ":amd: (MI355 DPX) V1 E2E Hybrid Chunked Prefill" label: ":amd: (MI300) V1 E2E Hybrid Chunked Prefill"
dind: false dind: false
device: mi355_dpx device: mi300_1
timeout_in_minutes: 35 timeout_in_minutes: 35
depends_on: depends_on:
- image-build-amd - image-build-amd
+2 -1
View File
@@ -545,6 +545,7 @@ steps:
num_devices: 1 num_devices: 1
source_file_dependencies: source_file_dependencies:
- cmake/external_projects/deepgemm.cmake - cmake/external_projects/deepgemm.cmake
- tools/install_deepgemm.sh
- tools/build_deepgemm_C.py - tools/build_deepgemm_C.py
- tools/setup_deepgemm_pythons.sh - tools/setup_deepgemm_pythons.sh
- tools/check_wheel_deepgemm.py - tools/check_wheel_deepgemm.py
@@ -938,7 +939,7 @@ steps:
- label: ":nvidia: (B200) FusedMoE Layer Kernels" - label: ":nvidia: (B200) FusedMoE Layer Kernels"
key: kernels-fusedmoe-layer-test-2-b200s key: kernels-fusedmoe-layer-test-2-b200s
timeout_in_minutes: 45 timeout_in_minutes: 90
device: b200-k8s device: b200-k8s
num_devices: 2 num_devices: 2
source_file_dependencies: source_file_dependencies:
+12 -11
View File
@@ -15,9 +15,9 @@ steps:
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-small.txt - pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-small.txt
mirror: mirror:
amd: amd:
label: ":amd: (MI355) LM Eval Small Models" label: ":amd: (MI300) LM Eval Small Models"
dind: false dind: false
device: mi355_dpx device: mi300_1
timeout_in_minutes: 45 timeout_in_minutes: 45
depends_on: depends_on:
- image-build-amd - image-build-amd
@@ -707,17 +707,18 @@ steps:
- export VLLM_WORKER_MULTIPROC_METHOD=spawn - export VLLM_WORKER_MULTIPROC_METHOD=spawn
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-mi355-nemotron-3-super.txt - pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-mi355-nemotron-3-super.txt
- label: ":amd: (MI355) LM Eval Large Models" - label: ":amd: (MI300) LM Eval Large Models"
key: amd-lm-eval-large-models key: amd-lm-eval-large-models
dind: false dind: false
device: mi355_8 device: mi300_8
optional: true optional: true
timeout_in_minutes: 65 timeout_in_minutes: 65
depends_on: depends_on:
- image-build-amd - image-build-amd
commands: commands:
- export VLLM_WORKER_MULTIPROC_METHOD=spawn - export VLLM_WORKER_MULTIPROC_METHOD=spawn
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-gfx950-large.txt - export PYTORCH_ROCM_ARCH=gfx942 # Limit Quark compilation to save time
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-mi3xx.txt
source_file_dependencies: source_file_dependencies:
- vllm/_aiter_ops.py - vllm/_aiter_ops.py
- vllm/platforms/rocm.py - vllm/platforms/rocm.py
@@ -1297,9 +1298,9 @@ steps:
- pytest -s -v evals/gsm8k/test_gsm8k_offloading.py -k "nemotron-h-8b or gemma-4-e4b-it" - pytest -s -v evals/gsm8k/test_gsm8k_offloading.py -k "nemotron-h-8b or gemma-4-e4b-it"
mirror: mirror:
amd: amd:
label: ":amd: (MI355 DPX) LM Eval KV-Offload" label: ":amd: (MI300) LM Eval KV-Offload"
dind: false dind: false
device: mi355_dpx device: mi300_1
timeout_in_minutes: 55 timeout_in_minutes: 55
depends_on: depends_on:
- image-build-amd - image-build-amd
@@ -1359,9 +1360,9 @@ steps:
- pytest -s -v evals/gsm8k/test_gsm8k_offloading.py -k "deepseek-v4-flash" - pytest -s -v evals/gsm8k/test_gsm8k_offloading.py -k "deepseek-v4-flash"
mirror: mirror:
amd: amd:
label: ":amd: (MI355) LM Eval KV-Offload Large" label: ":amd: (MI300) LM Eval KV-Offload Large"
dind: false dind: false
device: mi355_4 device: mi300_4
timeout_in_minutes: 65 timeout_in_minutes: 65
depends_on: depends_on:
- image-build-amd - image-build-amd
@@ -1426,10 +1427,10 @@ steps:
# Qwen2-57B has 28 attention heads and cannot run at TP8. # Qwen2-57B has 28 attention heads and cannot run at TP8.
- pytest -s -v test_lm_eval_correctness.py --config-list-file=configs/models-large.txt --tp-size=4 - pytest -s -v test_lm_eval_correctness.py --config-list-file=configs/models-large.txt --tp-size=4
- label: ":amd: (MI355) LM Eval Large Models ROCm Harness" - label: ":amd: (MI300) LM Eval Large Models ROCm Harness"
key: amd-lm-eval-large-models-rocm-harness key: amd-lm-eval-large-models-rocm-harness
dind: false dind: false
device: mi355_8 device: mi300_8
optional: true optional: true
timeout_in_minutes: 40 timeout_in_minutes: 40
depends_on: depends_on:
-54
View File
@@ -164,59 +164,6 @@ steps:
depends_on: depends_on:
- image-build-amd - image-build-amd
# The shared KV Connectors steps above exclude vllm/distributed/kv_transfer/ from
# their triggers, and CUDA skips MoRIIO's mori/aiter-gated tests. Run the whole
# MoRIIO suite on ROCm whenever the connector or its tests change.
- label: ":amd: (MI355 DPX) MoRIIO Connector Unit Tests"
key: amd-moriio-connector-unit-mi355
dind: false
device: mi355_dpx
num_devices: 1
timeout_in_minutes: 30
working_dir: /vllm-workspace/tests
depends_on:
- image-build-amd
source_file_dependencies:
- vllm/distributed/kv_transfer/kv_connector/v1/moriio/
- vllm/distributed/kv_transfer/kv_connector/v1/base.py
- vllm/distributed/kv_transfer/kv_connector/v1/ssm_conv_transfer_utils.py
- examples/disaggregated/disaggregated_serving/moriio_toy_proxy_server.py
- tests/v1/kv_connector/unit/test_moriio_
- tests/v1/kv_connector/unit/utils.py
- requirements/kv_connectors_rocm.txt
commands:
- bash /vllm-workspace/.buildkite/scripts/install-kv-connectors.sh
- export VLLM_WORKER_MULTIPROC_METHOD=spawn
- pytest -v -s -rs v1/kv_connector/unit/test_moriio_*.py
# Single-node MoRIIO WRITE P/D on dummy weights: prefill and decode on separate
# GPUs over xGMI, compared exactly against a direct serve.
- label: ":amd: (MI355) MoRIIO WRITE P/D Dummy-Weight E2E Shard %N"
key: amd-moriio-write-dummy-e2e-mi355
dind: false
device: mi355_2
num_devices: 2
# ~3.5-5 min per case; pytest-shard hashes test ids, so with 20 cases the
# largest of 6 shards holds 5 (~22 min on MI355). Every case picks free ports
# for its servers and proxy and its own compile-cache dir.
parallelism: 6
timeout_in_minutes: 45
working_dir: /vllm-workspace/tests
depends_on:
- image-build-amd
source_file_dependencies:
- vllm/distributed/kv_transfer/kv_connector/v1/moriio/
- vllm/distributed/kv_transfer/kv_connector/v1/base.py
- vllm/distributed/kv_transfer/kv_connector/v1/ssm_conv_transfer_utils.py
- vllm/model_executor/model_loader/dummy_loader.py
- examples/disaggregated/disaggregated_serving/moriio_toy_proxy_server.py
- tests/v1/kv_connector/moriio_integration/
- requirements/kv_connectors_rocm.txt
commands:
- bash /vllm-workspace/.buildkite/scripts/install-kv-connectors.sh
- export VLLM_WORKER_MULTIPROC_METHOD=spawn
- pytest -v -s -rs v1/kv_connector/moriio_integration/test_write_dummy_e2e_rocm.py --shard-id=$$BUILDKITE_PARALLEL_JOB --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT
- label: ":nvidia: (H200 MIG 35GB) V1 Metrics + LM Eval" - label: ":nvidia: (H200 MIG 35GB) V1 Metrics + LM Eval"
device: h200_35gb device: h200_35gb
key: v1-metrics-lmeval key: v1-metrics-lmeval
@@ -267,7 +214,6 @@ steps:
- vllm/v1/ - vllm/v1/
- tests/v1 - tests/v1
- tests/watermarking - tests/watermarking
- examples/disaggregated/mooncake_connector/mooncake_connector_proxy.py
device: cpu-small device: cpu-small
commands: commands:
# split the test to avoid interference # split the test to avoid interference
+1 -2
View File
@@ -72,9 +72,8 @@ steps:
- tests/models/test_deepseek_v41_pcp.py - tests/models/test_deepseek_v41_pcp.py
- tests/models/transformers/test_layer_registry_e2e.py - tests/models/transformers/test_layer_registry_e2e.py
- tests/models/test_deepseek_v41_replay_start.py - tests/models/test_deepseek_v41_replay_start.py
- tests/models/test_deepseek_v4_vl_stream_weights.py
commands: commands:
- pytest -v -s models/test_terratorch.py models/transformers/test_backend.py models/test_registry.py models/test_deepseek_v4_vl_rocm.py models/transformers/test_layer_registry_e2e.py models/test_deepseek_v41_replay_start.py models/test_deepseek_v41_decoder_replay_layers.py models/test_deepseek_v41_replay_batch.py models/test_deepseek_v41_pcp.py models/test_deepseek_v4_vl_stream_weights.py - pytest -v -s models/test_terratorch.py models/transformers/test_backend.py models/test_registry.py models/test_deepseek_v4_vl_rocm.py models/transformers/test_layer_registry_e2e.py models/test_deepseek_v41_replay_start.py models/test_deepseek_v41_decoder_replay_layers.py models/test_deepseek_v41_replay_batch.py models/test_deepseek_v41_pcp.py
mirror: mirror:
amd: amd:
label: ":amd: (MI355) Basic Models (Other)" label: ":amd: (MI355) Basic Models (Other)"
+3 -6
View File
@@ -210,18 +210,15 @@ steps:
- "!vllm/distributed/kv_transfer/" - "!vllm/distributed/kv_transfer/"
- tests/models/language/pooling - tests/models/language/pooling
commands: commands:
# Per-test timeout: a hung test fails with a stack dump instead of holding - pytest -v -s models/language/pooling -m 'not core_model'
# the job until Buildkite cancels it, which also blocks the full run's
# coverage collection. A whole shard takes 12-21 minutes.
- pytest -v -s models/language/pooling -m 'not core_model' --timeout=900 --timeout-method=thread
mirror: mirror:
amd: amd:
parallelism: 4 parallelism: 4
commands: commands:
- "pytest -v -s models/language/pooling -m 'not core_model' --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT --shard-id=$$BUILDKITE_PARALLEL_JOB" - "pytest -v -s models/language/pooling -m 'not core_model' --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT --shard-id=$$BUILDKITE_PARALLEL_JOB"
label: ":amd: (MI355 DPX) Language Models (Extended Pooling) Shard %N" label: ":amd: (MI300) Language Models (Extended Pooling) Shard %N"
dind: false dind: false
device: mi355_dpx device: mi300_1
timeout_in_minutes: 95 timeout_in_minutes: 95
depends_on: depends_on:
- image-build-amd - image-build-amd
+6 -6
View File
@@ -163,9 +163,9 @@ steps:
- pytest -v -s models/multimodal/test_mapping.py - pytest -v -s models/multimodal/test_mapping.py
mirror: mirror:
amd: amd:
label: ":amd: (MI355 DPX) Multimodal Models (Extended Generation 1)" label: ":amd: (MI300) Multimodal Models (Extended Generation 1)"
dind: false dind: false
device: mi355_dpx device: mi300_1
timeout_in_minutes: 100 timeout_in_minutes: 100
depends_on: depends_on:
- image-build-amd - image-build-amd
@@ -182,9 +182,9 @@ steps:
- pytest -v -s models/multimodal/generation_ppl_test/ - pytest -v -s models/multimodal/generation_ppl_test/
mirror: mirror:
amd: amd:
label: ":amd: (MI355 DPX) Multimodal Models (PPL)" label: ":amd: (MI300) Multimodal Models (PPL)"
dind: false dind: false
device: mi355_dpx device: mi300_1
timeout_in_minutes: 90 timeout_in_minutes: 90
depends_on: depends_on:
- image-build-amd - image-build-amd
@@ -240,9 +240,9 @@ steps:
- pytest -v -s models/multimodal/pooling -m 'not core_model' - pytest -v -s models/multimodal/pooling -m 'not core_model'
mirror: mirror:
amd: amd:
label: ":amd: (MI355 DPX) Multimodal Models (Extended Pooling)" label: ":amd: (MI300) Multimodal Models (Extended Pooling)"
dind: false dind: false
device: mi355_dpx device: mi300_1
timeout_in_minutes: 75 timeout_in_minutes: 75
depends_on: depends_on:
- image-build-amd - image-build-amd
+14 -14
View File
@@ -137,9 +137,9 @@ steps:
- pytest -v -s v1/e2e/spec_decode/speculators/ - pytest -v -s v1/e2e/spec_decode/speculators/
mirror: mirror:
amd: amd:
label: ":amd: (MI355 DPX) Spec Decode Speculators" label: ":amd: (MI300) Spec Decode Speculators"
dind: false dind: false
device: mi355_dpx device: mi300_1
timeout_in_minutes: 40 timeout_in_minutes: 40
depends_on: depends_on:
- image-build-amd - image-build-amd
@@ -167,9 +167,9 @@ steps:
- pytest -v -s v1/e2e/spec_decode/mtp/gemma4/ - pytest -v -s v1/e2e/spec_decode/mtp/gemma4/
mirror: mirror:
amd: amd:
label: ":amd: (MI355 DPX) Spec Decode MTP Gemma4" label: ":amd: (MI300) Spec Decode MTP Gemma4"
dind: false dind: false
device: mi355_dpx device: mi300_1
timeout_in_minutes: 40 timeout_in_minutes: 40
depends_on: depends_on:
- image-build-amd - image-build-amd
@@ -198,9 +198,9 @@ steps:
- pytest -v -s v1/e2e/spec_decode/mtp/mimo/ - pytest -v -s v1/e2e/spec_decode/mtp/mimo/
mirror: mirror:
amd: amd:
label: ":amd: (MI355 DPX) Spec Decode MTP DeepSeek + MiMo" label: ":amd: (MI300) Spec Decode MTP DeepSeek + MiMo"
dind: false dind: false
device: mi355_dpx device: mi300_1
timeout_in_minutes: 40 timeout_in_minutes: 40
depends_on: depends_on:
- image-build-amd - image-build-amd
@@ -228,9 +228,9 @@ steps:
- pytest -v -s v1/e2e/spec_decode/mtp/qwen3_5/ - pytest -v -s v1/e2e/spec_decode/mtp/qwen3_5/
mirror: mirror:
amd: amd:
label: ":amd: (MI355 DPX) Spec Decode MTP Qwen3.5" label: ":amd: (MI300) Spec Decode MTP Qwen3.5"
dind: false dind: false
device: mi355_dpx device: mi300_1
timeout_in_minutes: 40 timeout_in_minutes: 40
depends_on: depends_on:
- image-build-amd - image-build-amd
@@ -387,9 +387,9 @@ steps:
- pytest -v -s v1/spec_decode/test_speculators_correctness.py -m slow_test - pytest -v -s v1/spec_decode/test_speculators_correctness.py -m slow_test
mirror: mirror:
amd: amd:
label: ":amd: (MI355 DPX) Speculators Correctness Nightly" label: ":amd: (MI300) Speculators Correctness Nightly"
dind: false dind: false
device: mi355_dpx device: mi300_1
timeout_in_minutes: 40 timeout_in_minutes: 40
depends_on: depends_on:
- image-build-amd - image-build-amd
@@ -452,9 +452,9 @@ steps:
- pytest -v -s v1/e2e/spec_decode/acceptance_rates/dflash/ -k "not dflash2" - pytest -v -s v1/e2e/spec_decode/acceptance_rates/dflash/ -k "not dflash2"
mirror: mirror:
amd: amd:
label: ":amd: (MI355 DPX) Spec Decode AL DFlash Nightly" label: ":amd: (MI300) Spec Decode AL DFlash Nightly"
dind: false dind: false
device: mi355_dpx device: mi300_1
timeout_in_minutes: 120 timeout_in_minutes: 120
commands: commands:
- pytest -v -s v1/e2e/spec_decode/acceptance_rates/dflash/ -k "not dflash2" - pytest -v -s v1/e2e/spec_decode/acceptance_rates/dflash/ -k "not dflash2"
@@ -531,9 +531,9 @@ steps:
- pytest -v -s v1/e2e/spec_decode/acceptance_rates/dspark/ - pytest -v -s v1/e2e/spec_decode/acceptance_rates/dspark/
mirror: mirror:
amd: amd:
label: ":amd: (MI355 DPX) Spec Decode AL DSpark Nightly" label: ":amd: (MI300) Spec Decode AL DSpark Nightly"
dind: false dind: false
device: mi355_dpx device: mi300_1
timeout_in_minutes: 90 timeout_in_minutes: 90
depends_on: depends_on:
- image-build-amd - image-build-amd
+1 -5
View File
@@ -8,7 +8,7 @@
/vllm/distributed/kv_transfer/kv_connector/v1/hisparse @MatthewBonanni @LucasWilkinson /vllm/distributed/kv_transfer/kv_connector/v1/hisparse @MatthewBonanni @LucasWilkinson
/vllm/lora @jeejeelee /vllm/lora @jeejeelee
/vllm/model_executor/layers/attention @LucasWilkinson @MatthewBonanni /vllm/model_executor/layers/attention @LucasWilkinson @MatthewBonanni
/vllm/model_executor/layers/fused_moe @mgoin @pavanimajety @zyongye @wzhao18 @bnellnm /vllm/model_executor/layers/fused_moe @mgoin @pavanimajety @zyongye @wzhao18
/vllm/model_executor/layers/quantization @mgoin @robertgshaw2-redhat @tlrmchlsmth @yewentao256 @pavanimajety @zyongye /vllm/model_executor/layers/quantization @mgoin @robertgshaw2-redhat @tlrmchlsmth @yewentao256 @pavanimajety @zyongye
/vllm/model_executor/layers/mamba @tdoublep @tomeras91 @ZJY0516 /vllm/model_executor/layers/mamba @tdoublep @tomeras91 @ZJY0516
/vllm/model_executor/layers/mamba/gdn/qwen_gdn_linear_attn.py @tdoublep @ZJY0516 @vadiklyutiy /vllm/model_executor/layers/mamba/gdn/qwen_gdn_linear_attn.py @tdoublep @ZJY0516 @vadiklyutiy
@@ -138,7 +138,6 @@
/tests/evals @mgoin @vadiklyutiy @AndreasKaratzas @ZJY0516 /tests/evals @mgoin @vadiklyutiy @AndreasKaratzas @ZJY0516
/tests/kernels @mgoin @tlrmchlsmth @WoosukKwon @yewentao256 @zyongye @AndreasKaratzas /tests/kernels @mgoin @tlrmchlsmth @WoosukKwon @yewentao256 @zyongye @AndreasKaratzas
/tests/kernels/ir @ProExpertProg @tjtanaa /tests/kernels/ir @ProExpertProg @tjtanaa
/tests/kernels/moe @bnellnm
/tests/models @DarkLight1337 @ywang96 @AndreasKaratzas /tests/models @DarkLight1337 @ywang96 @AndreasKaratzas
/tests/multimodal @DarkLight1337 @ywang96 @NickLucche @shen-shanshan /tests/multimodal @DarkLight1337 @ywang96 @NickLucche @shen-shanshan
/tests/multimodal/media @Isotr0py /tests/multimodal/media @Isotr0py
@@ -165,9 +164,6 @@
.readthedocs.yaml @hmellor .readthedocs.yaml @hmellor
mkdocs.yaml @hmellor mkdocs.yaml @hmellor
# Model recipes
/recipes @esmeetu
# Linting # Linting
/tools/pre_commit/.markdownlint.yaml @hmellor /tools/pre_commit/.markdownlint.yaml @hmellor
.pre-commit-config.yaml @hmellor .pre-commit-config.yaml @hmellor
-28
View File
@@ -1,28 +0,0 @@
name: Recipes
# Validate recipe changes against the renderer that publishes them
# (recipes.vllm.ai). The check logic lives in vllm-project/recipes and tracks
# its main branch, so a PR is held to the same validation that runs when
# vllm-project/recipes picks up this directory after merge.
on:
pull_request:
paths:
- 'recipes/**'
- '.github/workflows/recipes.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
permissions:
contents: read
jobs:
validate:
name: Recipes
if: github.repository_owner == 'vllm-project'
uses: vllm-project/recipes/.github/workflows/validate-recipes.yml@main
with:
source: vllm
path: recipes
+2 -12
View File
@@ -621,16 +621,6 @@ def blocks_new_run(command: str, build: Mapping[str, Any]) -> bool:
return is_active_build(build) return is_active_build(build)
def list_commit_builds(
buildkite: BuildkiteClient, command: str, commit: str
) -> list[dict[str, Any]]:
builds = buildkite.list_builds(commit)
if command not in AMD_CI_COMMANDS:
return builds
# bootstrap-amd.sh exits before uploading any test steps for PR webhook builds.
return [build for build in builds if build.get("source") != "webhook"]
def select_latest_build( def select_latest_build(
builds: Sequence[dict[str, Any]], builds: Sequence[dict[str, Any]],
pr_number: int, pr_number: int,
@@ -978,7 +968,7 @@ def handle_run_ci(
f"{ci_name} was already requested by this comment: {duplicate['web_url']}" f"{ci_name} was already requested by this comment: {duplicate['web_url']}"
) )
current_builds = list_commit_builds(buildkite, command, pr["head"]["sha"]) current_builds = buildkite.list_builds(pr["head"]["sha"])
active_build = next( active_build = next(
( (
build build
@@ -1033,7 +1023,7 @@ def handle_retry_failed(
ci_name = ci_name_for_command(command) ci_name = ci_name_for_command(command)
run_command = run_command_for_command(command) run_command = run_command_for_command(command)
retry_command = retry_command_for_command(command) retry_command = retry_command_for_command(command)
builds = list_commit_builds(buildkite, command, pr["head"]["sha"]) builds = buildkite.list_builds(pr["head"]["sha"])
build = select_latest_build(builds, pr["number"]) build = select_latest_build(builds, pr["number"])
if build: if build:
metadata = build.get("meta_data") or {} metadata = build.get("meta_data") or {}
@@ -883,42 +883,6 @@ class RunCiCommandTest(unittest.TestCase):
self.assertEqual(buildkite.created_builds, []) self.assertEqual(buildkite.created_builds, [])
self.assertIn("AMD CI is already running", github.comments[0]) self.assertIn("AMD CI is already running", github.comments[0])
def test_amd_run_ignores_finished_webhook_build_for_author(self) -> None:
webhook_build = {
"created_at": "2026-08-18T01:00:00Z",
"meta_data": {},
"number": 122,
"pull_request": {"id": 42},
"source": "webhook",
"state": "passed",
"web_url": "https://buildkite.example/amd-ci/builds/122",
}
ui_build = {**webhook_build, "source": "ui"}
command_build = {
**webhook_build,
"meta_data": {"github-comment-id": "98"},
"source": "api",
}
for build, allowed in (
(webhook_build, True),
(ui_build, False),
(command_build, False),
):
with self.subTest(source=build["source"]):
github = FakeGitHub(
permission="read",
pr=make_pr(labels=[{"name": "ready"}]),
)
buildkite = FakeBuildkite([[], [build]])
run(make_event(COMMAND_RUN_AMD_CI, "author"), github, buildkite)
self.assertEqual(bool(buildkite.created_builds), allowed)
if not allowed:
self.assertIn(
"AMD CI already ran on this commit", github.comments[0]
)
def test_run_all_sets_buildkite_environment(self) -> None: def test_run_all_sets_buildkite_environment(self) -> None:
github = FakeGitHub() github = FakeGitHub()
buildkite = FakeBuildkite([[], []]) buildkite = FakeBuildkite([[], []])
@@ -1265,29 +1229,6 @@ class RunCiCommandTest(unittest.TestCase):
) )
self.assertIn("Use `/amd-ci run`", github.comments[0]) self.assertIn("Use `/amd-ci run`", github.comments[0])
def test_amd_ci_retry_ignores_webhook_build_for_the_current_head(self) -> None:
github = FakeGitHub(
permission="read",
pr=make_pr(labels=[{"name": "ready"}]),
)
webhook_build = {
"created_at": "2026-08-18T01:00:00Z",
"meta_data": {},
"number": 122,
"pull_request": {"id": 42},
"source": "webhook",
"state": "passed",
"web_url": "https://buildkite.example/amd-ci/builds/122",
}
buildkite = FakeBuildkite([[webhook_build]])
run(make_event(COMMAND_RETRY_AMD_FAILED, "author"), github, buildkite)
self.assertEqual(buildkite.retry_calls, [])
self.assertIn(
"No AMD CI build exists for the current PR head", github.comments[0]
)
def test_ci_retry_creates_filtered_build_for_new_head(self) -> None: def test_ci_retry_creates_filtered_build_for_new_head(self) -> None:
github = FakeGitHub( github = FakeGitHub(
behind=100, behind=100,
-6
View File
@@ -277,12 +277,6 @@ repos:
files: ^docker/(Dockerfile|versions\.json)$ files: ^docker/(Dockerfile|versions\.json)$
pass_filenames: false pass_filenames: false
additional_dependencies: [dockerfile-parse] additional_dependencies: [dockerfile-parse]
- id: check-dockerignore-tracked
name: Check .dockerignore does not exclude tracked files
entry: python tools/pre_commit/check_dockerignore_tracked.py
language: python
always_run: true
pass_filenames: false
- id: check-boolean-context-manager - id: check-boolean-context-manager
name: Check for boolean ops in with-statements name: Check for boolean ops in with-statements
entry: python tools/pre_commit/check_boolean_context_manager.py entry: python tools/pre_commit/check_boolean_context_manager.py
-12
View File
@@ -415,7 +415,6 @@ if(VLLM_GPU_LANG STREQUAL "CUDA" OR VLLM_GPU_LANG STREQUAL "HIP")
"csrc/cuda_view.cu" "csrc/cuda_view.cu"
"csrc/cuda_utils_kernels.cu" "csrc/cuda_utils_kernels.cu"
"csrc/activation_kernels.cu" "csrc/activation_kernels.cu"
"csrc/muse_spark_mul_tanh_soft_cap_kernel.cu"
"csrc/ngram_embedding_kernels.cu" "csrc/ngram_embedding_kernels.cu"
"csrc/vocab_parallel_embedding_kernels.cu" "csrc/vocab_parallel_embedding_kernels.cu"
"csrc/quantization/activation_kernels.cu" "csrc/quantization/activation_kernels.cu"
@@ -424,7 +423,6 @@ if(VLLM_GPU_LANG STREQUAL "CUDA" OR VLLM_GPU_LANG STREQUAL "HIP")
"csrc/quantization/w8a8/fp8/per_token_group_quant.cu" "csrc/quantization/w8a8/fp8/per_token_group_quant.cu"
"csrc/quantization/gptq/q_gemm.cu" "csrc/quantization/gptq/q_gemm.cu"
"csrc/pos_encoding_kernels.cu" "csrc/pos_encoding_kernels.cu"
"csrc/fused_muse_spark_qknorm_rope_kv_insert_kernel.cu"
"csrc/fused_qknorm_rope_kernel.cu" "csrc/fused_qknorm_rope_kernel.cu"
"csrc/fused_minimax_m3_qknorm_rope_kv_insert_kernel.cu" "csrc/fused_minimax_m3_qknorm_rope_kv_insert_kernel.cu"
"csrc/layernorm_kernels.cu" "csrc/layernorm_kernels.cu"
@@ -443,11 +441,6 @@ if(VLLM_GPU_LANG STREQUAL "CUDA" OR VLLM_GPU_LANG STREQUAL "HIP")
"csrc/fused_deepseek_v4_qnorm_rope_kv_insert_kernel.cu" "csrc/fused_deepseek_v4_qnorm_rope_kv_insert_kernel.cu"
"csrc/fused_kimi_k3_mla_key_concat_kv_cache_kernel.cu") "csrc/fused_kimi_k3_mla_key_concat_kv_cache_kernel.cu")
if(VLLM_GPU_LANG STREQUAL "CUDA")
list(APPEND VLLM_STABLE_EXT_SRC
"csrc/fused_qknorm_rope_cache_kernel.cu")
endif()
if(VLLM_GPU_LANG STREQUAL "CUDA" AND if(VLLM_GPU_LANG STREQUAL "CUDA" AND
DEFINED CMAKE_CUDA_COMPILER_VERSION AND DEFINED CMAKE_CUDA_COMPILER_VERSION AND
CMAKE_CUDA_COMPILER_VERSION VERSION_GREATER_EQUAL 12.0) CMAKE_CUDA_COMPILER_VERSION VERSION_GREATER_EQUAL 12.0)
@@ -1119,11 +1112,6 @@ if(VLLM_GPU_LANG STREQUAL "CUDA" OR VLLM_GPU_LANG STREQUAL "HIP")
"9.0a;10.0f;12.0f" "${CUDA_ARCHS}") "9.0a;10.0f;12.0f" "${CUDA_ARCHS}")
cuda_archs_loose_intersection(FUSED_GDN_DECODE_ARCHS cuda_archs_loose_intersection(FUSED_GDN_DECODE_ARCHS
"8.0;8.6;8.9;9.0a;10.0f;11.0f;12.0f" "${CUDA_ARCHS}") "8.0;8.6;8.9;9.0a;10.0f;11.0f;12.0f" "${CUDA_ARCHS}")
else()
# The fused GDN decode kernel needs only cp.async and BF16 math. Only the
# family-specific targets (10.0f, 11.0f, 12.0f) need CUDA >= 13.0.
cuda_archs_loose_intersection(FUSED_GDN_DECODE_ARCHS
"8.0;8.6;8.9;9.0a" "${CUDA_ARCHS}")
endif() endif()
if(FUSED_KDA_DECODE_ARCHS) if(FUSED_KDA_DECODE_ARCHS)
set(FUSED_KDA_DECODE_SRC set(FUSED_KDA_DECODE_SRC
@@ -1,326 +0,0 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""Benchmark the fused Muse-Spark QK-norm + RoPE + KV-insert kernel vs unfused.
``tuned_us`` forces the cp.async schedule ``fused_qk_norm_rope`` only auto-selects
on SM 9.0; ``unfused_us`` is what production runs.
"""
import argparse
import itertools
import torch
import vllm._custom_ops as ops
HEAD_DIM = 64
DEFAULT_TOKENS = [1, 8, 64, 256, 1024, 4096, 8192, 16384, 32768, 65536]
def make_cos_sin_cache(max_pos, rotary_dim, base, device):
inv_freq = 1.0 / (
base
** (
torch.arange(0, rotary_dim, 2, dtype=torch.float32, device=device)
/ rotary_dim
)
)
t = torch.arange(max_pos, dtype=torch.float32, device=device)
freqs = torch.einsum("i,j->ij", t, inv_freq)
return torch.cat((freqs.cos(), freqs.sin()), dim=-1)
class Case:
"""Everything a single (num_tokens, config) measurement needs."""
def __init__(self, num_tokens, nq, nkv, dtype, kv_cache_dtype, device):
self.num_tokens = num_tokens
self.nq, self.nkv = nq, nkv
self.dtype = dtype
self.kv_cache_dtype = kv_cache_dtype
self.eps = 1e-5
self.block_size = 128
self.rotary_dim = HEAD_DIM
self.qsz, self.kvsz = nq * HEAD_DIM, nkv * HEAD_DIM
self.row = self.qsz + 2 * self.kvsz + self.qsz # [q | k | v | o_gate]
self.qkv = torch.randn(num_tokens, self.row, dtype=dtype, device=device)
self.pristine = self.qkv.clone()
max_pos = 32768
self.cos_sin = make_cos_sin_cache(max_pos, self.rotary_dim, 500_000.0, device)
self.positions = torch.randint(
0, max_pos, (num_tokens,), dtype=torch.int64, device=device
)
num_blocks = (num_tokens + self.block_size - 1) // self.block_size + 1
storage = torch.uint8 if kv_cache_dtype != "auto" else dtype
self.kv_cache = torch.zeros(
num_blocks,
nkv,
self.block_size,
2 * HEAD_DIM,
dtype=storage,
device=device,
)
self.k_cache, self.v_cache = self.kv_cache.transpose(1, 2).split(
HEAD_DIM, dim=-1
)
self.slot_mapping = torch.randperm(
num_blocks * self.block_size, dtype=torch.int64, device=device
)[:num_tokens]
self.k_scale = torch.tensor(1.0, dtype=torch.float32, device=device)
self.v_scale = torch.tensor(1.0, dtype=torch.float32, device=device)
self.ones = torch.ones(HEAD_DIM, dtype=dtype, device=device)
self.q_fp8 = torch.empty(
num_tokens, self.qsz, dtype=torch.float8_e4m3fn, device=device
)
self.q_scale = torch.tensor(0.37, dtype=torch.float32, device=device)
def views(self):
q, k, v, _ = self.qkv.split([self.qsz, self.kvsz, self.kvsz, self.qsz], dim=-1)
return q, k, v
def fused(self, use_rope=True, quant_query=False):
ops.fused_muse_spark_qknorm_rope_kv_insert(
self.qkv,
self.positions,
self.nq,
self.nkv,
HEAD_DIM,
self.eps,
cos_sin_cache=self.cos_sin if use_rope else None,
rotary_dim=self.rotary_dim if use_rope else 0,
slot_mapping=self.slot_mapping,
kv_cache=self.kv_cache,
block_size=self.block_size,
kv_cache_dtype=self.kv_cache_dtype,
k_scale=self.k_scale if self.kv_cache_dtype != "auto" else None,
v_scale=self.v_scale if self.kv_cache_dtype != "auto" else None,
q_fp8_out=self.q_fp8 if quant_query else None,
q_scale=self.q_scale if quant_query else None,
)
def separate_query_quant(self, use_rope=True):
self.fused(use_rope)
ops.scaled_fp8_quant(self.qkv[:, : self.qsz], scale=self.q_scale)
def _insert(self):
_, k, v = self.views()
ops.reshape_and_cache_flash(
k.view(-1, self.nkv, HEAD_DIM),
v.view(-1, self.nkv, HEAD_DIM),
self.k_cache,
self.v_cache,
self.slot_mapping,
self.kv_cache_dtype,
self.k_scale,
self.v_scale,
)
def unfused_rope(self, token_heads_per_warp: int = -1):
ops.fused_qk_norm_rope(
self.qkv,
self.nq,
self.nkv,
self.nkv + self.nq, # v heads plus the o-gate
HEAD_DIM,
self.eps,
self.ones,
self.ones,
self.cos_sin,
False,
self.positions,
token_heads_per_warp,
)
self._insert()
def fused_bytes(self):
el = self.qkv.element_size()
cache_el = self.kv_cache.element_size()
read = (self.nq + 2 * self.nkv) * HEAD_DIM * el
write = (self.nq + self.nkv) * HEAD_DIM * el
cache = 2 * self.nkv * HEAD_DIM * cache_el
return self.num_tokens * (read + write + cache)
def _l2_bytes(device) -> int:
return torch.cuda.get_device_properties(device).L2_cache_size
def _peak_bandwidth(device) -> float:
"""Measured large-copy bandwidth, in GB/s -- the practical ceiling."""
n = 1 << 28 # 256 Mi elements = 512 MiB each way at 2 bytes
a = torch.empty(n, dtype=torch.bfloat16, device=device)
b = torch.empty_like(a)
for _ in range(3):
b.copy_(a)
torch.accelerator.synchronize()
start, end = torch.cuda.Event(True), torch.cuda.Event(True)
start.record()
iters = 20
for _ in range(iters):
b.copy_(a)
end.record()
torch.accelerator.synchronize()
ms = start.elapsed_time(end) / iters
return (2 * a.numel() * a.element_size()) / (ms * 1e-3) / 1e9
def time_eager(fn, warmup=20, iters=100) -> float:
for _ in range(warmup):
fn()
torch.accelerator.synchronize()
start, end = torch.cuda.Event(True), torch.cuda.Event(True)
start.record()
for _ in range(iters):
fn()
end.record()
torch.accelerator.synchronize()
return start.elapsed_time(end) / iters * 1e3 # us
def time_graph(fn, warmup=10, iters=50, inner=32) -> float:
"""GPU time per call; ``inner`` calls per graph amortize the ~2 us replay."""
side = torch.cuda.Stream()
side.wait_stream(torch.cuda.current_stream())
with torch.cuda.stream(side):
for _ in range(warmup):
fn()
torch.cuda.current_stream().wait_stream(side)
graph = torch.cuda.CUDAGraph()
with torch.cuda.graph(graph):
for _ in range(inner):
fn()
for _ in range(3):
graph.replay()
torch.accelerator.synchronize()
start, end = torch.cuda.Event(True), torch.cuda.Event(True)
start.record()
for _ in range(iters):
graph.replay()
end.record()
torch.accelerator.synchronize()
return start.elapsed_time(end) / (iters * inner) * 1e3 # us
def main():
p = argparse.ArgumentParser()
p.add_argument("--nq", type=int, default=32, help="query heads per rank")
p.add_argument("--nkv", type=int, default=4, help="kv heads per rank")
p.add_argument("--kv-cache-dtype", default="auto", choices=["auto", "fp8"])
p.add_argument("--dtype", default="bfloat16", choices=["bfloat16", "float16"])
p.add_argument("--tokens", type=int, nargs="*", default=DEFAULT_TOKENS)
p.add_argument("--csv", default=None)
p.add_argument("--no-graph", action="store_true")
p.add_argument(
"--query-fp8", action="store_true", help="compare fused Q quantization"
)
args = p.parse_args()
device = "cuda"
dtype = getattr(torch, args.dtype)
if args.query_fp8:
if args.kv_cache_dtype != "fp8":
p.error("--query-fp8 requires --kv-cache-dtype fp8")
timer = time_eager if args.no_graph else time_graph
print(f"device: {torch.cuda.get_device_name()}; nq={args.nq}, nkv={args.nkv}")
print("layer,tokens,separate_us,fused_us,speedup")
rows = []
for use_rope, nt in itertools.product([True, False], args.tokens):
case = Case(nt, args.nq, args.nkv, dtype, args.kv_cache_dtype, device)
separate_us = timer(lambda c=case, r=use_rope: c.separate_query_quant(r))
case.qkv.copy_(case.pristine)
fused_us = timer(lambda c=case, r=use_rope: c.fused(r, quant_query=True))
row = (
"rope" if use_rope else "nope",
nt,
separate_us,
fused_us,
separate_us / fused_us,
)
rows.append(row)
print(f"{row[0]},{nt},{separate_us:.3f},{fused_us:.3f},{row[4]:.3f}")
if args.csv:
import csv
with open(args.csv, "w", newline="") as f:
writer = csv.writer(f)
writer.writerow(
["layer", "tokens", "separate_us", "fused_us", "speedup"]
)
writer.writerows(rows)
return
peak = _peak_bandwidth(device)
print(
f"device: {torch.cuda.get_device_name()} measured copy peak: {peak:.0f} GB/s"
)
print(
f"config: nq={args.nq} nkv={args.nkv} head_dim={HEAD_DIM} "
f"dtype={args.dtype} kv_cache={args.kv_cache_dtype}\n"
)
modes = ["rope"]
l2 = _l2_bytes(device)
print(
f"rows marked * have a working set inside the {l2 / 2**20:.0f} MiB L2, "
"so their bandwidth is not an HBM number\n"
)
header = (
f"{'layer':>5} {'tokens':>7} {'fused_us':>9} {'unfused_us':>11} "
f"{'tuned_us':>9} {'speedup':>8} {'GB/s':>8} {'SOL%':>6}"
)
rows = []
for mode, use_graph in itertools.product(modes, [False, True]):
if use_graph and args.no_graph:
continue
label = "graph" if use_graph else "eager"
print(f"── {mode} layer, {label} ──")
print(header)
timer = time_graph if use_graph else time_eager
for nt in args.tokens:
case = Case(nt, args.nq, args.nkv, dtype, args.kv_cache_dtype, device)
use_rope = mode == "rope"
fused_us = timer(lambda c=case, r=use_rope: c.fused(r))
baseline = case.unfused_rope
unfused_us = timer(baseline)
tuned_us = (
min(timer(lambda c=case, t=t: c.unfused_rope(t)) for t in (4, 8))
if use_rope
else unfused_us
)
nbytes = case.fused_bytes()
gbps = nbytes / (fused_us * 1e-6) / 1e9
sol = 100.0 * gbps / peak
resident = "*" if nbytes < l2 else " "
print(
f"{mode:>5} {nt:>7} {fused_us:>9.2f} {unfused_us:>11.2f} "
f"{tuned_us:>9.2f} {tuned_us / fused_us:>7.2f}x {gbps:>8.0f} "
f"{sol:>5.1f}%{resident}"
)
rows.append((mode, label, nt, fused_us, unfused_us, tuned_us, gbps, sol))
print()
if args.csv:
import csv
with open(args.csv, "w", newline="") as f:
w = csv.writer(f)
w.writerow(
[
"layer",
"timing",
"tokens",
"fused_us",
"unfused_us",
"tuned_us",
"gbps",
"sol",
]
)
w.writerows(rows)
print(f"wrote {args.csv}")
if __name__ == "__main__":
main()
+1
View File
@@ -27,6 +27,7 @@ if(DEEPGEMM_SRC_DIR)
set(deepgemm_SOURCE_DIR "${DEEPGEMM_SRC_DIR}") set(deepgemm_SOURCE_DIR "${DEEPGEMM_SRC_DIR}")
message(STATUS "DeepGEMM using local DEEPGEMM_SRC_DIR: ${deepgemm_SOURCE_DIR}") message(STATUS "DeepGEMM using local DEEPGEMM_SRC_DIR: ${deepgemm_SOURCE_DIR}")
else() else()
# Keep in sync with tools/install_deepgemm.sh
set(_DEEPGEMM_UPSTREAM_REPO "https://github.com/vllm-project/DeepGEMM.git") set(_DEEPGEMM_UPSTREAM_REPO "https://github.com/vllm-project/DeepGEMM.git")
# Pinned to the fork's dev branch: upstream's 26/09/30 release (2.8.1) # Pinned to the fork's dev branch: upstream's 26/09/30 release (2.8.1)
# plus the SM120 port, the SM90 paged-MQA kv_block=32/next_n=4 port, # plus the SM120 port, the SM90 paged-MQA kv_block=32/next_n=4 port,
+1 -4
View File
@@ -425,7 +425,6 @@ function(cuda_archs_loose_intersection OUT_CUDA_ARCHS SRC_CUDA_ARCHS TGT_CUDA_AR
if("${_base}" IN_LIST _TGT_CUDA_ARCHS) if("${_base}" IN_LIST _TGT_CUDA_ARCHS)
list(REMOVE_ITEM _SRC_CUDA_ARCHS "${_arch}") list(REMOVE_ITEM _SRC_CUDA_ARCHS "${_arch}")
list(REMOVE_ITEM _TGT_CUDA_ARCHS "${_base}") list(REMOVE_ITEM _TGT_CUDA_ARCHS "${_base}")
list(REMOVE_ITEM _TGT_CUDA_ARCHS "${_base}a" "${_base}f")
list(APPEND _CUDA_ARCHS "${_arch}") list(APPEND _CUDA_ARCHS "${_arch}")
endif() endif()
endif() endif()
@@ -465,10 +464,8 @@ function(cuda_archs_loose_intersection OUT_CUDA_ARCHS SRC_CUDA_ARCHS TGT_CUDA_AR
string(REGEX REPLACE "[af]$" "" _base "${_arch}") string(REGEX REPLACE "[af]$" "" _base "${_arch}")
if ("${_base}" IN_LIST _SRC_CUDA_ARCHS) if ("${_base}" IN_LIST _SRC_CUDA_ARCHS)
list(REMOVE_ITEM _TGT_CUDA_ARCHS "${_arch}") list(REMOVE_ITEM _TGT_CUDA_ARCHS "${_arch}")
list(REMOVE_ITEM _SRC_CUDA_ARCHS "${_base}")
list(APPEND _CUDA_ARCHS "${_arch}") list(APPEND _CUDA_ARCHS "${_arch}")
if (NOT "${_base}" IN_LIST _TGT_CUDA_ARCHS)
list(REMOVE_ITEM _SRC_CUDA_ARCHS "${_base}")
endif()
endif() endif()
endif() endif()
endforeach() endforeach()
+27
View File
@@ -1059,6 +1059,33 @@ void situ_and_mul(torch::stable::Tensor& out, // [..., d]
}); });
} }
void masked_situ_and_mul(torch::stable::Tensor& out, // [E, T, d]
torch::stable::Tensor& input, // [E, T, 2 * d]
const torch::stable::Tensor& expert_num_tokens,
double beta, double linear_beta) {
int num_experts = input.size(0);
int max_num_tokens = input.size(1);
int d = input.size(2) / 2;
if (num_experts == 0 || max_num_tokens == 0 || d == 0) {
return;
}
constexpr int block_size = 256;
dim3 grid((d + block_size - 1) / block_size, num_experts);
dim3 block(block_size);
const torch::stable::accelerator::DeviceGuard device_guard(
input.get_device_index());
const cudaStream_t stream = get_current_cuda_stream();
VLLM_STABLE_DISPATCH_FLOATING_TYPES(
input.scalar_type(), "masked_situ_and_mul_kernel", [&] {
vllm::masked_situ_and_mul_kernel<scalar_t, true>
<<<grid, block, 0, stream>>>(
out.mutable_data_ptr<scalar_t>(),
input.const_data_ptr<scalar_t>(),
expert_num_tokens.const_data_ptr<int>(), max_num_tokens, d,
(float)beta, (float)linear_beta);
});
}
#define LAUNCH_MASKED_ACT_AND_MUL(KERNEL, ACT_FIRST, HAS_CLAMP, LIMIT, ALPHA, \ #define LAUNCH_MASKED_ACT_AND_MUL(KERNEL, ACT_FIRST, HAS_CLAMP, LIMIT, ALPHA, \
BETA) \ BETA) \
vllm::masked_act_and_mul_kernel<scalar_t, KERNEL<scalar_t>, ACT_FIRST, \ vllm::masked_act_and_mul_kernel<scalar_t, KERNEL<scalar_t>, ACT_FIRST, \
+5 -76
View File
@@ -1,7 +1,7 @@
// SPDX-License-Identifier: Apache-2.0 // SPDX-License-Identifier: Apache-2.0
// SPDX-FileCopyrightText: Copyright contributors to the vLLM project // SPDX-FileCopyrightText: Copyright contributors to the vLLM project
// //
// Router GEMM: activation(T) x weight(fp32) -> fp32, M<=48, for the // Router GEMM: activation(T) x weight(fp32) -> fp32, M<=32, for the
// supported (E, H) pairs listed at the bottom of this file. // supported (E, H) pairs listed at the bottom of this file.
// Supports bf16 or fp32 activation; weight is always fp32. // Supports bf16 or fp32 activation; weight is always fp32.
@@ -214,26 +214,8 @@ void invokeFp32RouterGemm(float* output, InputT const* mat_a,
// production-fidelity harness (CUDA-graph replay, per-layer cold weights). // production-fidelity harness (CUDA-graph replay, per-layer cold weights).
// Only enabled on the Blackwell family where it was validated; Hopper and // Only enabled on the Blackwell family where it was validated; Hopper and
// other shapes / fp32 activation keep the legacy geometry. // other shapes / fp32 activation keep the legacy geometry.
if constexpr (std::is_same_v<InputT, __nv_bfloat16> && kNumExperts == 256 && if constexpr (std::is_same_v<InputT, __nv_bfloat16> && kNumExperts == 128 &&
kHiddenDim == 5120) { kHiddenDim == 6144) {
if (!isBlackwellFamily()) {
launchFp32RouterGemm<InputT, 128, 1, kNumTokens, kNumExperts, kHiddenDim>(
output, mat_a, mat_b, stream);
return;
}
if constexpr (kNumTokens > 16 && kNumTokens % 8 == 0) {
constexpr int block_size = kNumTokens == 24 ? 160 : 128;
launchFp32RouterGemm<InputT, block_size, 2, kNumTokens, kNumExperts,
kHiddenDim, 4>(output, mat_a, mat_b, stream);
} else {
constexpr int block_size = kNumTokens == 1 ? 64 : 128;
constexpr int token_groups = kNumTokens % 2 == 0 ? 2 : 1;
launchFp32RouterGemm<InputT, block_size, 1, kNumTokens, kNumExperts,
kHiddenDim, token_groups>(output, mat_a, mat_b,
stream);
}
} else if constexpr (std::is_same_v<InputT, __nv_bfloat16> &&
kNumExperts == 128 && kHiddenDim == 6144) {
// MiniMax-M3. Legacy 128/1 only fills 128 blocks and pays the same // MiniMax-M3. Legacy 128/1 only fills 128 blocks and pays the same
// accumulator register cliffs; B300 sweep: // accumulator register cliffs; B300 sweep:
// even M in [6, 10] : BS=384, EPB=1, 2 token groups (1.26-1.43x) // even M in [6, 10] : BS=384, EPB=1, 2 token groups (1.26-1.43x)
@@ -279,27 +261,6 @@ void invokeFp32RouterGemm(float* output, InputT const* mat_a,
launchFp32RouterGemm<InputT, 128, 1, kNumTokens, kNumExperts, kHiddenDim>( launchFp32RouterGemm<InputT, 128, 1, kNumTokens, kNumExperts, kHiddenDim>(
output, mat_a, mat_b, stream); output, mat_a, mat_b, stream);
} }
} else if constexpr (std::is_same_v<InputT, __nv_bfloat16> &&
kNumExperts == 256 && kHiddenDim == 8192) {
// MuseSpark1x: the 8MB weight makes this shape FFMA-bound past M~10, so
// the wider block wins at every M and token groups clear the M >= 28
// accumulator cliff. Thresholds are from a B300 sweep. BS=192/384 are not
// instantiable: VPT=8 requires kBlockSize to divide kHiddenDim / 8 = 1024.
if (!isBlackwellFamily()) {
launchFp32RouterGemm<InputT, 128, 1, kNumTokens, kNumExperts, kHiddenDim>(
output, mat_a, mat_b, stream);
return;
}
if constexpr (kNumTokens >= 10 && kNumTokens % 2 == 0) {
launchFp32RouterGemm<InputT, 256, 2, kNumTokens, kNumExperts, kHiddenDim,
2>(output, mat_a, mat_b, stream);
} else if constexpr (kNumTokens >= 13) {
launchFp32RouterGemm<InputT, 256, 2, kNumTokens, kNumExperts, kHiddenDim>(
output, mat_a, mat_b, stream);
} else {
launchFp32RouterGemm<InputT, 256, 1, kNumTokens, kNumExperts, kHiddenDim>(
output, mat_a, mat_b, stream);
}
} else { } else {
launchFp32RouterGemm<InputT, 128, 1, kNumTokens, kNumExperts, kHiddenDim>( launchFp32RouterGemm<InputT, 128, 1, kNumTokens, kNumExperts, kHiddenDim>(
output, mat_a, mat_b, stream); output, mat_a, mat_b, stream);
@@ -307,9 +268,8 @@ void invokeFp32RouterGemm(float* output, InputT const* mat_a,
} }
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// Explicit instantiations for both input types. // Explicit instantiations: M=1..32, for both input types, for the supported
// Supported (E, H) pairs: (256, 3072), (128, 6144), (256, 5120), // (E, H) pairs: (256, 3072) [MiniMax-M2/M2.5] and (128, 6144) [MiniMax-M3].
// (352, 4096) [Step-5], and (256, 8192) [Muse Spark].
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
#define INSTANTIATE(T, M, E, H) \ #define INSTANTIATE(T, M, E, H) \
@@ -355,36 +315,5 @@ INSTANTIATE_ALL(__nv_bfloat16, 256, 3072)
INSTANTIATE_ALL(float, 128, 6144) INSTANTIATE_ALL(float, 128, 6144)
INSTANTIATE_ALL(__nv_bfloat16, 128, 6144) INSTANTIATE_ALL(__nv_bfloat16, 128, 6144)
INSTANTIATE_ALL(float, 256, 5120)
INSTANTIATE_ALL(__nv_bfloat16, 256, 5120)
INSTANTIATE_ALL(float, 352, 4096)
INSTANTIATE_ALL(__nv_bfloat16, 352, 4096)
INSTANTIATE_ALL(float, 256, 8192)
INSTANTIATE_ALL(__nv_bfloat16, 256, 8192)
#define INSTANTIATE_LARGE(T, E, H) \
INSTANTIATE(T, 33, E, H) \
INSTANTIATE(T, 34, E, H) \
INSTANTIATE(T, 35, E, H) \
INSTANTIATE(T, 36, E, H) \
INSTANTIATE(T, 37, E, H) \
INSTANTIATE(T, 38, E, H) \
INSTANTIATE(T, 39, E, H) \
INSTANTIATE(T, 40, E, H) \
INSTANTIATE(T, 41, E, H) \
INSTANTIATE(T, 42, E, H) \
INSTANTIATE(T, 43, E, H) \
INSTANTIATE(T, 44, E, H) \
INSTANTIATE(T, 45, E, H) \
INSTANTIATE(T, 46, E, H) \
INSTANTIATE(T, 47, E, H) \
INSTANTIATE(T, 48, E, H)
INSTANTIATE_LARGE(float, 256, 5120)
INSTANTIATE_LARGE(__nv_bfloat16, 256, 5120)
#undef INSTANTIATE_LARGE
#undef INSTANTIATE_ALL #undef INSTANTIATE_ALL
#undef INSTANTIATE #undef INSTANTIATE
+7 -20
View File
@@ -25,13 +25,10 @@ inline int getSMVersion() {
static constexpr int FP32_MAX_TOKENS = 32; static constexpr int FP32_MAX_TOKENS = 32;
// Supported (hidden_dim, num_experts) pairs (must match the instantiations in // Supported (hidden_dim, num_experts) pairs (must match the instantiations in
// fp32_router_gemm.cu). // fp32_router_gemm.cu): (3072, 256) for MiniMax-M2/M2.5, (6144, 128) for M3.
static inline bool fp32_router_gemm_supported(int hidden_dim, int num_experts) { static inline bool fp32_router_gemm_supported(int hidden_dim, int num_experts) {
return (hidden_dim == 3072 && num_experts == 256) || return (hidden_dim == 3072 && num_experts == 256) ||
(hidden_dim == 6144 && num_experts == 128) || (hidden_dim == 6144 && num_experts == 128);
(hidden_dim == 5120 && num_experts == 256) ||
(hidden_dim == 4096 && num_experts == 352) ||
(hidden_dim == 8192 && num_experts == 256);
} }
// Forward declarations — 4 template params must match fp32_router_gemm.cu // Forward declarations — 4 template params must match fp32_router_gemm.cu
@@ -63,7 +60,8 @@ struct Fp32LoopUnroller<InputT, kNumExperts, kHiddenDim, kEnd, kEnd> {
invokeFp32RouterGemm<InputT, kEnd, kNumExperts, kHiddenDim>( invokeFp32RouterGemm<InputT, kEnd, kNumExperts, kHiddenDim>(
output, mat_a, mat_b, stream); output, mat_a, mat_b, stream);
} else { } else {
throw std::invalid_argument("fp32_router_gemm: unsupported num_tokens"); throw std::invalid_argument(
"fp32_router_gemm: num_tokens must be in [1, 32]");
} }
} }
}; };
@@ -79,15 +77,6 @@ void dispatchFp32RouterGemm(int num_experts, int hidden_dim, int num_tokens,
} else if (num_experts == 128 && hidden_dim == 6144) { } else if (num_experts == 128 && hidden_dim == 6144) {
Fp32LoopUnroller<InputT, 128, 6144, 1, FP32_MAX_TOKENS>::unroll( Fp32LoopUnroller<InputT, 128, 6144, 1, FP32_MAX_TOKENS>::unroll(
num_tokens, output, mat_a, mat_b, stream); num_tokens, output, mat_a, mat_b, stream);
} else if (num_experts == 256 && hidden_dim == 5120) {
Fp32LoopUnroller<InputT, 256, 5120, 1, 48>::unroll(num_tokens, output,
mat_a, mat_b, stream);
} else if (num_experts == 352 && hidden_dim == 4096) {
Fp32LoopUnroller<InputT, 352, 4096, 1, FP32_MAX_TOKENS>::unroll(
num_tokens, output, mat_a, mat_b, stream);
} else if (num_experts == 256 && hidden_dim == 8192) {
Fp32LoopUnroller<InputT, 256, 8192, 1, FP32_MAX_TOKENS>::unroll(
num_tokens, output, mat_a, mat_b, stream);
} else { } else {
throw std::invalid_argument( throw std::invalid_argument(
"fp32_router_gemm: unsupported (hidden_dim, num_experts) pair"); "fp32_router_gemm: unsupported (hidden_dim, num_experts) pair");
@@ -122,11 +111,9 @@ void fp32_router_gemm(
STD_TORCH_CHECK( STD_TORCH_CHECK(
fp32_router_gemm_supported(hidden_dim, num_experts), fp32_router_gemm_supported(hidden_dim, num_experts),
"fp32_router_gemm: supported (hidden_dim, num_experts) pairs are " "fp32_router_gemm: supported (hidden_dim, num_experts) pairs are "
"(3072, 256), (6144, 128), (5120, 256), (4096, 352), and (8192, 256)"); "(3072, 256) and (6144, 128)");
const int max_tokens = STD_TORCH_CHECK(num_tokens <= FP32_MAX_TOKENS,
hidden_dim == 5120 && num_experts == 256 ? 48 : FP32_MAX_TOKENS; "fp32_router_gemm: num_tokens must be in [0, 32]");
STD_TORCH_CHECK(num_tokens <= max_tokens,
"fp32_router_gemm: num_tokens exceeds supported range");
STD_TORCH_CHECK( STD_TORCH_CHECK(
mat_a.scalar_type() == torch::headeronly::ScalarType::Float || mat_a.scalar_type() == torch::headeronly::ScalarType::Float ||
mat_a.scalar_type() == torch::headeronly::ScalarType::BFloat16, mat_a.scalar_type() == torch::headeronly::ScalarType::BFloat16,
@@ -1,752 +0,0 @@
/*
* SPDX-License-Identifier: Apache-2.0
* SPDX-FileCopyrightText: Copyright contributors to the vLLM project
*
* Fused Muse Spark attention pre-processing: weightless QK RMSNorm, GPT-J
* interleaved RoPE (skipped on NoPE layers, ``cos_sin_cache == None``) and
* the K/V insert of ``reshape_and_cache_flash``, over ``[q | k | v | o_gate]``
* rows.
*
* A group of ``head_dim * sizeof(scalar_t) / 16`` lanes owns one head (8 lanes
* for bf16 head_dim=64), so each lane moves one 16-byte vector, the rotated
* pair (2i, 2i+1) is lane-local, and the norm reduction is 3 shuffles.
*
* Slot enumeration per token:
* [0, nq) Q heads -> norm (+RoPE), in place or to
* ``q_fp8_out``
* [nq, nq+nkv) K heads -> norm (+RoPE), in place and into
* the K cache
* [nq+nkv, nq+2*nkv) V heads -> copied into the V cache (no norm)
* V slots exist only when a KV cache is bound.
*
* Norm and rotation run in fp32 and round once at the store. Both fp8 paths
* quantize after rounding to the model dtype, the same boundary
* ``reshape_and_cache_flash`` and ``scaled_fp8_quant`` use.
*/
#include <algorithm>
#include <cmath>
#include <cuda_runtime.h>
#include <type_traits>
#include "torch_utils.h"
#include "core/cuda_compat.h"
#include "type_convert.cuh"
#include "core/vec_dtypes/dtype_fp8.cuh"
#include "dispatch_utils.h"
#ifndef USE_ROCM
#include <cuda_fp8.h>
#include "quantization/w8a8/fp8/nvidia/quant_utils.cuh"
#else
#include <hip/hip_fp8.h>
#include "quantization/w8a8/fp8/amd/quant_utils.cuh"
#endif
#ifndef FINAL_MASK
#ifdef USE_ROCM
#define FINAL_MASK 0xffffffffffffffffULL
#else
#define FINAL_MASK 0xffffffffu
#endif
#endif
namespace vllm {
namespace muse_spark_fused_ops {
namespace {
inline int getSMVersion() {
auto* props = get_device_prop();
return props->major * 10 + props->minor;
}
inline int getSMCount() {
auto* props = get_device_prop();
return props->multiProcessorCount;
}
} // namespace
// 16 bytes per lane: the widest vector access, and the reason a head is owned
// by a group of lanes rather than a whole warp.
constexpr int kBytesPerLane = 16;
#ifdef USE_ROCM
using WarpMask = uint64_t;
constexpr unsigned kWarpSize = 64;
#else
using WarpMask = unsigned;
constexpr unsigned kWarpSize = 32;
#endif
// Slot -> (token, head) needs a division by a value only known at launch. The
// kernel is issue-bound, and a 32-bit hardware division is ~20 instructions of
// it, so the reciprocal is computed once on the host instead.
struct FastDiv {
uint32_t mul;
uint32_t shr;
int div;
};
inline FastDiv makeFastDiv(int denom) {
// The reciprocal form needs denom >= 2; denom == 1 is carried as a flag the
// device side selects on. Dividends stay below 2^31, which covers every slot
// index this kernel can form.
if (denom <= 1) return FastDiv{0u, 0u, 1};
int lg = 31 - __builtin_clz(denom);
if ((1 << lg) < denom) lg++;
uint32_t const p = 31 + static_cast<uint32_t>(lg);
uint64_t const m = ((1ull << p) + static_cast<uint64_t>(denom) - 1) /
static_cast<uint64_t>(denom);
return FastDiv{static_cast<uint32_t>(m), p - 32, denom};
}
__device__ __forceinline__ void fastDivmod(int& quo, int& rem, int src,
FastDiv const fd) {
int const q =
static_cast<int>(__umulhi(static_cast<uint32_t>(src), fd.mul) >> fd.shr);
// A select, not a branch: fd.div is uniform across the launch.
quo = (fd.div == 1) ? src : q;
rem = src - quo * fd.div;
}
// Sum across the LANES contiguous lanes that share one head. The group is
// LANES-aligned inside the warp, so an XOR butterfly over masks < LANES never
// leaves it. `group_mask` names exactly the group: whole groups are uniformly
// active or inactive, so a warp whose other groups bailed out still syncs.
template <int LANES>
__device__ __forceinline__ float groupReduceSum(float val,
WarpMask const group_mask) {
#pragma unroll
for (int mask = LANES >> 1; mask > 0; mask >>= 1) {
val += __shfl_xor_sync(group_mask, val, mask, 32);
}
return val;
}
// 16 contiguous bytes -> kElems fp32 registers, keeping the raw vector so a
// pass-through slot (V) can be re-stored without a conversion round trip.
template <typename scalar_t, int kElems>
__device__ __forceinline__ uint4 loadElems(scalar_t const* __restrict__ src,
float (&elems)[kElems]) {
using Converter = vllm::_typeConvert<scalar_t>;
using T2 = typename Converter::packed_hip_type;
uint4 raw = *reinterpret_cast<uint4 const*>(src);
auto const* p = reinterpret_cast<T2 const*>(&raw);
#pragma unroll
for (int i = 0; i < kElems / 2; i++) {
float2 f2 = Converter::convert(p[i]);
elems[2 * i] = f2.x;
elems[2 * i + 1] = f2.y;
}
return raw;
}
// The lane's slice of the rope table: kPairs cos and kPairs sin values, each
// run contiguous, so each is one vector load (16 bytes for the fp32 table, 8
// for a model-dtype one) instead of kPairs scalar ones. rotary_dim being a
// multiple of 16 is what keeps both runs naturally aligned.
template <typename rope_t, int kPairs>
__device__ __forceinline__ void loadRopeVec(rope_t const* __restrict__ cos_ptr,
rope_t const* __restrict__ sin_ptr,
float (&c)[kPairs],
float (&s)[kPairs]) {
static_assert(kPairs == 4, "rope vector load assumes 4 pairs per lane");
if constexpr (std::is_same_v<rope_t, float>) {
float4 const cv = VLLM_LDG(reinterpret_cast<float4 const*>(cos_ptr));
float4 const sv = VLLM_LDG(reinterpret_cast<float4 const*>(sin_ptr));
c[0] = cv.x, c[1] = cv.y, c[2] = cv.z, c[3] = cv.w;
s[0] = sv.x, s[1] = sv.y, s[2] = sv.z, s[3] = sv.w;
} else {
using Converter = vllm::_typeConvert<rope_t>;
using T2 = typename Converter::packed_hip_type;
uint2 const cv = VLLM_LDG(reinterpret_cast<uint2 const*>(cos_ptr));
uint2 const sv = VLLM_LDG(reinterpret_cast<uint2 const*>(sin_ptr));
auto const* cp = reinterpret_cast<T2 const*>(&cv);
auto const* sp = reinterpret_cast<T2 const*>(&sv);
#pragma unroll
for (int i = 0; i < kPairs / 2; i++) {
float2 const cf = Converter::convert(cp[i]);
float2 const sf = Converter::convert(sp[i]);
c[2 * i] = cf.x, c[2 * i + 1] = cf.y;
s[2 * i] = sf.x, s[2 * i + 1] = sf.y;
}
}
}
// kElems fp32 registers -> the 16-byte scalar_t vector they round to. Returned
// rather than stored so the fp8 cache path can reuse the *rounded* values,
// which is what reshape_and_cache_flash quantizes from.
template <typename scalar_t, int kElems>
__device__ __forceinline__ uint4 packElems(float const (&elems)[kElems]) {
using Converter = vllm::_typeConvert<scalar_t>;
using T2 = typename Converter::packed_hip_type;
uint4 raw;
auto* p = reinterpret_cast<T2*>(&raw);
#pragma unroll
for (int i = 0; i < kElems / 2; i++) {
p[i] = Converter::convert(make_float2(elems[2 * i], elems[2 * i + 1]));
}
return raw;
}
// Scatter one lane's 16 bytes into the KV cache. kAuto stores the model dtype
// verbatim; the fp8 dtypes quantize the value *after* it has been rounded to
// scalar_t, which is the boundary reshape_and_cache_flash uses.
//
// `inv_scale` is taken once per thread: per-element fp32 divisions would cost
// more than the rest of the kernel. The extra rounding is 2^-24 relative.
template <typename scalar_t, typename cache_t, Fp8KVCacheDataType kv_dt,
int kElems>
__device__ __forceinline__ void storeCache(cache_t* __restrict__ dst,
uint4 const& packed,
float const inv_scale) {
if constexpr (kv_dt == Fp8KVCacheDataType::kAuto) {
*reinterpret_cast<uint4*>(dst) = packed;
} else {
using Converter = vllm::_typeConvert<scalar_t>;
using HipT = typename Converter::hip_type;
auto const* rounded = reinterpret_cast<HipT const*>(&packed);
#ifndef USE_ROCM
constexpr __nv_fp8_interpretation_t kInterp =
(kv_dt == Fp8KVCacheDataType::kFp8E4M3) ? __NV_E4M3 : __NV_E5M2;
__nv_fp8x2_storage_t out[kElems / 2];
#pragma unroll
for (int i = 0; i < kElems / 2; i++) {
float2 v =
make_float2(Converter::convert(rounded[2 * i]) * inv_scale,
Converter::convert(rounded[2 * i + 1]) * inv_scale);
out[i] = __nv_cvt_float2_to_fp8x2(v, __NV_SATFINITE, kInterp);
}
// kElems fp8 bytes: 8 for the 2-byte dtypes this kernel serves.
static_assert(kElems == 8, "fp8 store assumes 8 elements per lane");
*reinterpret_cast<uint2*>(dst) = *reinterpret_cast<uint2 const*>(out);
#else
#pragma unroll
for (int i = 0; i < kElems; i++) {
float const v = Converter::convert(rounded[i]) * inv_scale;
dst[i] = fp8::scaled_convert<cache_t, float, kv_dt>(v, 1.0f);
}
#endif
}
}
// Grid is 1D over (token, head-chunk) pairs. A chunk is HEADS_PER_GROUP
// consecutive head slots of one token, served by kLanes contiguous lanes; the
// whole chunk shares the token's position and its slice of the rope table, so
// the slot division, the position load and the cos/sin load are paid once per
// chunk. `slots_per_token % HEADS_PER_GROUP == 0` keeps a chunk inside one
// token. `rope_t` is the cos/sin cache dtype, independent of the qkv dtype.
template <typename scalar_t, typename cache_t, Fp8KVCacheDataType kv_dt,
typename rope_t, int HEAD_DIM, int HEADS_PER_GROUP, bool kDoRope,
bool kInsertKV>
__global__
__launch_bounds__(256, 6) void fusedMuseSparkQKNormRopeKVInsertKernel(
scalar_t* __restrict__ qkv, // [N, qkv_stride] in/out
rope_t const* __restrict__ cos_sin_cache, // [max_pos, rotary_dim]
int64_t const* __restrict__ positions, // [N] i64
int64_t const* __restrict__ slot_mapping, // [N] i64 or nullptr
cache_t* __restrict__ kv_cache, // [nb, nkv, bs, 2*HEAD_DIM]
float const* __restrict__ k_scale, // [1] or nullptr
float const* __restrict__ v_scale, // [1] or nullptr
uint8_t* __restrict__ q_fp8_out, // [N, nq * HEAD_DIM] or nullptr
float const* __restrict__ q_scale, // [1] or nullptr
float const eps, int const rotary_dim, int const num_tokens, int const nq,
int const nkv, FastDiv const chunks_div, int64_t const qkv_stride,
FastDiv const block_div, int64_t const kv_s_block, int64_t const kv_s_head,
int64_t const kv_s_token) {
#if (!defined(__CUDA_ARCH__) || __CUDA_ARCH__ < 800) && !defined(USE_ROCM)
// _typeConvert<BFloat16> is unavailable on pre-Ampere; this kernel only ever
// runs with bf16/fp16 inputs. Discard the bf16 body there.
if constexpr (std::is_same_v<scalar_t, c10::BFloat16>) {
return;
} else {
#endif
constexpr int kElems = kBytesPerLane / sizeof(scalar_t);
constexpr int kLanes = HEAD_DIM / kElems;
constexpr int kPairs = kElems / 2;
constexpr int kH = HEADS_PER_GROUP;
static_assert(kLanes > 0 && kLanes <= 32 && (32 % kLanes) == 0,
"head_dim must give a power-of-two lane group <= 32");
int const tid = blockIdx.x * blockDim.x + threadIdx.x;
int const groupIdx = tid / kLanes;
int const dimBase = (tid % kLanes) * kElems;
int tokenIdx, chunk;
fastDivmod(tokenIdx, chunk, groupIdx, chunks_div);
int const firstSlot = chunk * kH;
// Out-of-range groups must not exit before the reductions below: the XOR
// butterfly needs every lane the mask names, and the mask covers the whole
// group. They ride along and skip every memory access instead.
bool const active = tokenIdx < num_tokens;
constexpr WarpMask kGroupBits =
(kLanes == 32) ? static_cast<WarpMask>(0xffffffffu)
: ((WarpMask{1} << kLanes) - WarpMask{1});
unsigned const group_shift =
(threadIdx.x & (kWarpSize - 1)) & ~static_cast<unsigned>(kLanes - 1);
WarpMask const group_mask = kGroupBits << group_shift;
// PDL: wait for the qkv-projection GEMM that produces `qkv`. No-op when
// PDL is not enabled on the launch.
#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900)
cudaGridDependencySynchronize();
#endif
// Q, K and V are contiguous head slots at the head of the row, so the slot
// index doubles as the head offset. The trailing o-gate is skipped by the
// row stride alone.
scalar_t* const base = qkv + static_cast<int64_t>(tokenIdx) * qkv_stride +
static_cast<int64_t>(firstSlot) * HEAD_DIM + dimBase;
// Issue every load in the chunk before touching any of it, so the group has
// kH loads in flight instead of one. An inactive group leaves these unset;
// nothing downstream reads them, since every store is gated on `active`.
uint4 raw[kH];
if (active) {
#pragma unroll
for (int h = 0; h < kH; h++) {
raw[h] = *reinterpret_cast<uint4 const*>(base + h * HEAD_DIM);
}
}
float cosv[kPairs], sinv[kPairs];
bool const laneRotates = kDoRope && dimBase < rotary_dim;
if constexpr (kDoRope) {
if (active && laneRotates) {
int64_t const pos = positions[tokenIdx];
rope_t const* cache_ptr = cos_sin_cache + pos * rotary_dim;
loadRopeVec<rope_t, kPairs>(cache_ptr + dimBase / 2,
cache_ptr + rotary_dim / 2 + dimBase / 2,
cosv, sinv);
}
}
int64_t sm = -1;
if constexpr (kInsertKV) {
if (active) sm = slot_mapping[tokenIdx];
}
int cache_block = 0, cache_tok = 0;
float k_inv = 1.0f, v_inv = 1.0f;
float const q_inv = q_fp8_out ? 1.0f / *q_scale : 1.0f;
if constexpr (kInsertKV) {
if (sm >= 0) {
fastDivmod(cache_block, cache_tok, static_cast<int>(sm), block_div);
}
if constexpr (kv_dt != Fp8KVCacheDataType::kAuto) {
k_inv = 1.0f / *k_scale;
v_inv = 1.0f / *v_scale;
}
}
#pragma unroll
for (int h = 0; h < kH; h++) {
int const slot = firstSlot + h;
bool const isV = kInsertKV && slot >= nq + nkv;
bool const isK = slot >= nq && slot < nq + nkv;
bool const isQK = active && !isV;
float elems[kElems];
uint4 packed = raw[h];
if (isQK) {
using Converter = vllm::_typeConvert<scalar_t>;
using T2 = typename Converter::packed_hip_type;
auto const* p = reinterpret_cast<T2 const*>(&raw[h]);
#pragma unroll
for (int i = 0; i < kElems / 2; i++) {
float2 const f2 = Converter::convert(p[i]);
elems[2 * i] = f2.x;
elems[2 * i + 1] = f2.y;
}
float sumsq = 0.0f;
#pragma unroll
for (int i = 0; i < kElems; i++) sumsq += elems[i] * elems[i];
sumsq = groupReduceSum<kLanes>(sumsq, group_mask);
float const rms_rcp =
rsqrtf(sumsq / static_cast<float>(HEAD_DIM) + eps);
#pragma unroll
for (int i = 0; i < kElems; i++) elems[i] *= rms_rcp;
if constexpr (kDoRope) {
if (laneRotates) {
#pragma unroll
for (int i = 0; i < kPairs; i++) {
float const v0 = elems[2 * i];
float const v1 = elems[2 * i + 1];
elems[2 * i] = v0 * cosv[i] - v1 * sinv[i];
elems[2 * i + 1] = v0 * sinv[i] + v1 * cosv[i];
}
}
}
// Rounding to scalar_t happens exactly once, here, and both the
// Q output and the fp8 cache quantization consume the same
// rounded value. V passes through with the vector it was loaded with.
packed = packElems<scalar_t, kElems>(elems);
if (q_fp8_out && slot < nq) {
int64_t const q_off =
(static_cast<int64_t>(tokenIdx) * nq + slot) * HEAD_DIM + dimBase;
storeCache<scalar_t, uint8_t, Fp8KVCacheDataType::kFp8E4M3, kElems>(
q_fp8_out + q_off, packed, q_inv);
} else {
*reinterpret_cast<uint4*>(base + h * HEAD_DIM) = packed;
}
}
if constexpr (kInsertKV) {
if (sm >= 0 && (isK || isV)) { // sm < 0 marks a padded token
int const head = isK ? (slot - nq) : (slot - nq - nkv);
int64_t const off = static_cast<int64_t>(cache_block) * kv_s_block +
head * kv_s_head +
static_cast<int64_t>(cache_tok) * kv_s_token +
(isK ? 0 : HEAD_DIM) + dimBase;
storeCache<scalar_t, cache_t, kv_dt, kElems>(kv_cache + off, packed,
isK ? k_inv : v_inv);
}
}
}
// PDL: let a dependent successor launch early.
#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900)
cudaTriggerProgrammaticLaunchCompletion();
#endif
#if (!defined(__CUDA_ARCH__) || __CUDA_ARCH__ < 800) && !defined(USE_ROCM)
}
#endif
}
template <typename scalar_t, typename cache_t, Fp8KVCacheDataType kv_dt,
typename rope_t, int HEAD_DIM, int HEADS_PER_GROUP>
void launchFusedMuseSparkTyped(
scalar_t* qkv, rope_t const* cos_sin_cache, int64_t const* positions,
int64_t const* slot_mapping, cache_t* kv_cache, float const* k_scale,
float const* v_scale, uint8_t* q_fp8_out, float const* q_scale,
float const eps, int const rotary_dim, int const num_tokens, int const nq,
int const nkv, int64_t const qkv_stride, int const block_size,
int64_t const kv_s_block, int64_t const kv_s_head, int64_t const kv_s_token,
bool const do_rope, bool const insert_kv, cudaStream_t stream) {
FastDiv const block_div = makeFastDiv(block_size);
constexpr int kElems = kBytesPerLane / sizeof(scalar_t);
constexpr int kLanes = HEAD_DIM / kElems;
int const slots_per_token = nq + nkv + (insert_kv ? nkv : 0);
int const chunks_per_token = slots_per_token / HEADS_PER_GROUP;
FastDiv const chunks_div = makeFastDiv(chunks_per_token);
int64_t const total_lanes =
static_cast<int64_t>(num_tokens) * chunks_per_token * kLanes;
if (total_lanes == 0) return;
constexpr int kBlockSize = 256;
int const grid =
static_cast<int>((total_lanes + kBlockSize - 1) / kBlockSize);
#ifndef USE_ROCM
// PDL wherever the hardware supports it (SM90+); a plain launch elsewhere.
static int const sm_version = getSMVersion();
cudaLaunchConfig_t config;
config.gridDim = dim3(grid);
config.blockDim = dim3(kBlockSize);
config.dynamicSmemBytes = 0;
config.stream = stream;
cudaLaunchAttribute attrs[1];
attrs[0].id = cudaLaunchAttributeProgrammaticStreamSerialization;
attrs[0].val.programmaticStreamSerializationAllowed = 1;
config.attrs = attrs;
config.numAttrs = (sm_version >= 90) ? 1 : 0;
#define LAUNCH_MUSE(DO_ROPE, INSERT) \
cudaLaunchKernelEx(&config, \
fusedMuseSparkQKNormRopeKVInsertKernel< \
scalar_t, cache_t, kv_dt, rope_t, HEAD_DIM, \
HEADS_PER_GROUP, DO_ROPE, INSERT>, \
qkv, cos_sin_cache, positions, slot_mapping, kv_cache, \
k_scale, v_scale, q_fp8_out, q_scale, eps, rotary_dim, \
num_tokens, nq, nkv, chunks_div, qkv_stride, block_div, \
kv_s_block, kv_s_head, kv_s_token)
#else
// clang-format off
#define LAUNCH_MUSE(DO_ROPE, INSERT) \
fusedMuseSparkQKNormRopeKVInsertKernel< \
scalar_t, cache_t, kv_dt, rope_t, HEAD_DIM, HEADS_PER_GROUP, DO_ROPE, \
INSERT><<<grid, kBlockSize, 0, stream>>>( \
qkv, cos_sin_cache, positions, slot_mapping, kv_cache, k_scale, \
v_scale, q_fp8_out, q_scale, eps, rotary_dim, num_tokens, nq, nkv, \
chunks_div, qkv_stride, block_div, kv_s_block, kv_s_head, kv_s_token)
// clang-format on
#endif
if (do_rope) {
if (insert_kv) {
LAUNCH_MUSE(true, true);
} else {
LAUNCH_MUSE(true, false);
}
} else {
if (insert_kv) {
LAUNCH_MUSE(false, true);
} else {
LAUNCH_MUSE(false, false);
}
}
#undef LAUNCH_MUSE
}
template <typename scalar_t, typename cache_t, Fp8KVCacheDataType kv_dt>
void launchFusedMuseSpark(
scalar_t* qkv, void const* cos_sin_cache, bool const rope_is_fp32,
int64_t const* positions, int64_t const* slot_mapping, cache_t* kv_cache,
float const* k_scale, float const* v_scale, uint8_t* q_fp8_out,
float const* q_scale, float const eps, int const rotary_dim,
int const num_tokens, int const nq, int const nkv, int const head_dim,
int64_t const qkv_stride, int const block_size, int64_t const kv_s_block,
int64_t const kv_s_head, int64_t const kv_s_token, bool const do_rope,
bool const insert_kv, cudaStream_t stream) {
// The group width must divide the slot count. Widen only while the grid
// still covers a full wave; decode-sized batches keep the narrow schedule.
int const slots_per_token = nq + nkv + (insert_kv ? nkv : 0);
constexpr int kElemsSel = kBytesPerLane / sizeof(scalar_t);
constexpr int kBlockSel = 256;
int const lanes_per_head = head_dim / kElemsSel;
int64_t const full_wave =
static_cast<int64_t>(getSMCount()) * (2048 / kBlockSel);
int heads_per_group = 1;
for (int h : {4, 2}) {
if (slots_per_token % h != 0) continue;
int64_t const blocks = (static_cast<int64_t>(num_tokens) *
(slots_per_token / h) * lanes_per_head +
kBlockSel - 1) /
kBlockSel;
if (blocks >= full_wave) {
heads_per_group = h;
break;
}
}
// TP4/TP8 amortize head addressing before filling a full CTA wave. RoPE
// benefits earlier; FP8 conversion keeps wider groups expensive for longer.
if (insert_kv && head_dim == 64 &&
((nq == 16 && nkv == 2) || (nq == 32 && nkv == 4))) {
int64_t const query_heads = static_cast<int64_t>(num_tokens) * nq;
if (do_rope && query_heads >= 24576) {
heads_per_group = std::max(heads_per_group, 2);
if constexpr (kv_dt == Fp8KVCacheDataType::kAuto) {
if (query_heads >= 49152) heads_per_group = 4;
}
} else if (!do_rope && query_heads >= 32768) {
if constexpr (kv_dt == Fp8KVCacheDataType::kAuto) {
heads_per_group = std::max(heads_per_group, 2);
}
}
}
#define DISPATCH_ARGS(RT, HD, HPG) \
launchFusedMuseSparkTyped<scalar_t, cache_t, kv_dt, RT, HD, HPG>( \
qkv, reinterpret_cast<RT const*>(cos_sin_cache), positions, \
slot_mapping, kv_cache, k_scale, v_scale, q_fp8_out, q_scale, eps, \
rotary_dim, num_tokens, nq, nkv, qkv_stride, block_size, kv_s_block, \
kv_s_head, kv_s_token, do_rope, insert_kv, stream)
#define DISPATCH_HPG(RT, HD) \
switch (heads_per_group) { \
case 4: \
DISPATCH_ARGS(RT, HD, 4); \
break; \
case 2: \
DISPATCH_ARGS(RT, HD, 2); \
break; \
default: \
DISPATCH_ARGS(RT, HD, 1); \
break; \
}
// A NoPE launch never reads the table, so it can share the fp32 branch.
#define DISPATCH_ROPE_T(HD) \
if (!do_rope || rope_is_fp32) { \
DISPATCH_HPG(float, HD); \
} else { \
DISPATCH_HPG(scalar_t, HD); \
}
switch (head_dim) {
case 64:
DISPATCH_ROPE_T(64);
break;
case 128:
DISPATCH_ROPE_T(128);
break;
case 256:
DISPATCH_ROPE_T(256);
break;
default:
STD_TORCH_CHECK(false, "unsupported head_dim: ", head_dim);
}
#undef DISPATCH_ROPE_T
#undef DISPATCH_HPG
#undef DISPATCH_ARGS
}
} // namespace muse_spark_fused_ops
} // namespace vllm
// clang-format off
#define CALL_FUSED_MUSE_SPARK(_RAW_T, CACHE_T, KV_DTYPE) \
vllm::muse_spark_fused_ops::launchFusedMuseSpark<st, CACHE_T, KV_DTYPE>( \
reinterpret_cast<st*>(qkv.data_ptr()), \
do_rope ? cos_sin_cache->data_ptr() : nullptr, rope_is_fp32, \
reinterpret_cast<int64_t const*>(positions.data_ptr()), \
insert_kv ? reinterpret_cast<int64_t const*>(slot_mapping->data_ptr()) \
: nullptr, \
insert_kv ? reinterpret_cast<CACHE_T*>(kv_cache->data_ptr()) : nullptr, \
needs_scales ? reinterpret_cast<float const*>(k_scale->data_ptr()) \
: nullptr, \
needs_scales ? reinterpret_cast<float const*>(v_scale->data_ptr()) \
: nullptr, \
q_fp8_out ? reinterpret_cast<uint8_t*>(q_fp8_out->data_ptr()) : nullptr, \
q_scale ? reinterpret_cast<float const*>(q_scale->data_ptr()) : nullptr, \
static_cast<float>(eps), static_cast<int>(rotary_dim), num_tokens, nq, \
nkv, hd, qkv_stride, static_cast<int>(block_size), kv_s_block, \
kv_s_head, kv_s_token, do_rope, insert_kv, stream)
// clang-format on
void fused_muse_spark_qknorm_rope_kv_insert(
torch::stable::Tensor& qkv, // [N, (nq + 2*nkv + gate)*hd]
torch::stable::Tensor const& positions, // [N] i64
int64_t num_heads, int64_t num_kv_heads, int64_t head_dim, double eps,
std::optional<torch::stable::Tensor> cos_sin_cache, // None => NoPE layer
int64_t rotary_dim,
std::optional<torch::stable::Tensor> slot_mapping, // [N] i64
std::optional<torch::stable::Tensor> kv_cache, // [nb, nkv, bs, 2*hd]
int64_t block_size, const std::string& kv_cache_dtype,
std::optional<torch::stable::Tensor> k_scale, // [1] fp32, fp8 cache only
std::optional<torch::stable::Tensor> v_scale,
std::optional<torch::stable::Tensor> q_fp8_out,
std::optional<torch::stable::Tensor> q_scale) {
STD_TORCH_CHECK(qkv.is_cuda() && qkv.dim() == 2 && qkv.stride(1) == 1,
"qkv must be a 2-D CUDA tensor with a contiguous row");
STD_TORCH_CHECK(
qkv.scalar_type() == torch::headeronly::ScalarType::Half ||
qkv.scalar_type() == torch::headeronly::ScalarType::BFloat16,
"qkv must be float16 or bfloat16");
STD_TORCH_CHECK(
positions.is_cuda() &&
positions.scalar_type() == torch::headeronly::ScalarType::Long,
"positions must be int64 CUDA");
int const num_tokens = static_cast<int>(qkv.size(0));
int const nq = static_cast<int>(num_heads);
int const nkv = static_cast<int>(num_kv_heads);
int const hd = static_cast<int>(head_dim);
int64_t const qkv_stride = qkv.stride(0);
STD_TORCH_CHECK(q_fp8_out.has_value() == q_scale.has_value(),
"q_fp8_out and q_scale must be provided together");
if (q_fp8_out) {
STD_TORCH_CHECK(
q_fp8_out->is_cuda() &&
q_fp8_out->get_device_index() == qkv.get_device_index() &&
q_fp8_out->scalar_type() ==
torch::headeronly::ScalarType::Float8_e4m3fn &&
q_fp8_out->dim() == 2 && q_fp8_out->is_contiguous() &&
q_fp8_out->size(0) == num_tokens &&
q_fp8_out->size(1) == static_cast<int64_t>(nq) * hd,
"q_fp8_out must be contiguous E4M3 [num_tokens, num_heads * head_dim] "
"on the qkv device");
STD_TORCH_CHECK(
q_scale->is_cuda() &&
q_scale->get_device_index() == qkv.get_device_index() &&
q_scale->scalar_type() == torch::headeronly::ScalarType::Float &&
q_scale->numel() == 1,
"q_scale must be a single fp32 value on the qkv device");
}
STD_TORCH_CHECK(hd == 64 || hd == 128 || hd == 256,
"head_dim must be 64, 128 or 256");
STD_TORCH_CHECK(qkv.size(1) >= static_cast<int64_t>(nq + 2 * nkv) * hd,
"qkv row must hold at least [q|k|v]");
// A lane moves 16 bytes, so every row start and every head start must land
// on a 16-byte boundary. Every supported head_dim is a multiple of 8 at 2
// bytes, so heads are fine; the row stride is the one that can break it.
STD_TORCH_CHECK(qkv_stride % 8 == 0,
"qkv row stride must be a multiple of 8 elements, got ",
qkv_stride);
bool const do_rope = cos_sin_cache.has_value();
bool rope_is_fp32 = true;
if (do_rope) {
STD_TORCH_CHECK(cos_sin_cache->is_cuda() && cos_sin_cache->dim() == 2 &&
cos_sin_cache->is_contiguous(),
"cos_sin_cache must be a contiguous 2-D CUDA tensor");
STD_TORCH_CHECK(cos_sin_cache->size(1) == rotary_dim,
"cos_sin_cache shape must be [max_pos, rotary_dim]");
STD_TORCH_CHECK(
cos_sin_cache->scalar_type() == torch::headeronly::ScalarType::Float ||
cos_sin_cache->scalar_type() == qkv.scalar_type(),
"cos_sin_cache must be float32 or match the qkv dtype");
rope_is_fp32 =
cos_sin_cache->scalar_type() == torch::headeronly::ScalarType::Float;
// A lane owns 16 bytes = 8 elements of the head and must be wholly inside
// or wholly outside the rotated prefix.
STD_TORCH_CHECK(rotary_dim > 0 && rotary_dim <= hd && rotary_dim % 16 == 0,
"rotary_dim must be a positive multiple of 16 and <= "
"head_dim");
}
vllm::Fp8KVCacheDataType const kv_dt =
vllm::get_fp8_kv_cache_data_type(kv_cache_dtype);
bool const insert_kv = kv_cache.has_value() && kv_cache->numel() > 0;
bool const needs_scales =
insert_kv && kv_dt != vllm::Fp8KVCacheDataType::kAuto;
int64_t kv_s_block = 0, kv_s_head = 0, kv_s_token = 0;
if (insert_kv) {
STD_TORCH_CHECK(
slot_mapping.has_value() && slot_mapping->is_cuda() &&
slot_mapping->scalar_type() == torch::headeronly::ScalarType::Long,
"insert mode requires an int64 CUDA slot_mapping");
STD_TORCH_CHECK(slot_mapping->numel() >= num_tokens,
"slot_mapping must cover every token");
STD_TORCH_CHECK(kv_cache->dim() == 4 && kv_cache->stride(3) == 1,
"kv_cache must be [nb, nkv, bs, 2*head_dim] with a "
"contiguous content dim");
STD_TORCH_CHECK(kv_cache->size(3) == 2 * hd,
"kv_cache content dim must be 2*head_dim");
STD_TORCH_CHECK(kv_cache->size(1) == nkv,
"kv_cache head dim must equal num_kv_heads");
STD_TORCH_CHECK(kv_cache->size(2) == block_size,
"kv_cache token dim must equal block_size");
if (kv_dt == vllm::Fp8KVCacheDataType::kAuto) {
STD_TORCH_CHECK(kv_cache->scalar_type() == qkv.scalar_type(),
"auto kv_cache dtype must match qkv");
} else {
STD_TORCH_CHECK(
kv_cache->scalar_type() == torch::headeronly::ScalarType::Byte ||
kv_cache->scalar_type() ==
torch::headeronly::ScalarType::Float8_e4m3fn,
"fp8 kv_cache must use uint8 or float8_e4m3fn storage");
STD_TORCH_CHECK(k_scale.has_value() && v_scale.has_value(),
"fp8 kv cache requires k_scale and v_scale");
STD_TORCH_CHECK(
k_scale->numel() == 1 && v_scale->numel() == 1 &&
k_scale->scalar_type() == torch::headeronly::ScalarType::Float &&
v_scale->scalar_type() == torch::headeronly::ScalarType::Float,
"k_scale and v_scale must be single fp32 values");
}
kv_s_block = kv_cache->stride(0);
kv_s_head = kv_cache->stride(1);
kv_s_token = kv_cache->stride(2);
}
const torch::stable::accelerator::DeviceGuard device_guard(
qkv.get_device_index());
auto stream = get_current_cuda_stream(qkv.get_device_index());
VLLM_STABLE_DISPATCH_HALF_TYPES(
qkv.scalar_type(), "fused_muse_spark_qknorm_rope_kv_insert", [&] {
using st = scalar_t;
DISPATCH_BY_KV_CACHE_DTYPE(qkv.scalar_type(), kv_cache_dtype,
CALL_FUSED_MUSE_SPARK);
});
}
#undef CALL_FUSED_MUSE_SPARK
-567
View File
@@ -1,567 +0,0 @@
// SPDX-License-Identifier: Apache-2.0
// SPDX-FileCopyrightText: Copyright contributors to the vLLM project
// Step5 fused QK norm, RoPE, and KV-cache update kernel.
// This kernel is built into `_C_stable_libtorch` and uses the libtorch stable
// ABI (torch::stable::Tensor / STD_TORCH_CHECK) instead of ATen and c10.
#include "torch_utils.h"
#include "dispatch_utils.h"
#include "type_convert.cuh"
#include "core/vec_dtypes/dtype_fp8.cuh"
#include "quantization/w8a8/fp8/nvidia/quant_utils.cuh"
#include <torch/csrc/stable/macros.h>
#include <cuda_runtime.h>
#include <algorithm>
#include <type_traits>
namespace vllm {
namespace {
constexpr int kWarpSize = 32;
constexpr int kThreadsPerBlock = 128;
template <typename scalar_t, typename weight_t, typename rope_t, int HEAD_DIM,
bool WIDE_192_ROTARY = false, bool FP8_CACHE = false>
__global__ void qknorm_rope_cache_kernel(
const void* __restrict__ qkv_void, void* __restrict__ q_out_void,
void* __restrict__ k_out_void, void* __restrict__ v_out_void,
const void* __restrict__ q_weight_void,
const void* __restrict__ k_weight_void, const void* __restrict__ cos_void,
const void* __restrict__ sin_void, const int64_t* __restrict__ positions,
const int64_t* __restrict__ slot_mapping, void* __restrict__ k_cache_void,
void* __restrict__ v_cache_void, int64_t qkv_token_stride,
int64_t q_token_stride, int64_t k_token_stride, int64_t v_token_stride,
int64_t cos_token_stride, int64_t sin_token_stride,
int64_t cache_block_stride, int64_t cache_page_stride,
int64_t cache_head_stride, int num_tokens, int num_slots, int num_q_heads,
int num_kv_heads, int rotary_pairs, int block_size, float epsilon,
int block_size_shift, float norm_weight_bias, const float* k_scale,
const float* v_scale) {
#if !defined(__CUDA_ARCH__) || __CUDA_ARCH__ < 800
if constexpr (std::is_same_v<scalar_t, torch::headeronly::BFloat16> ||
std::is_same_v<weight_t, torch::headeronly::BFloat16> ||
std::is_same_v<rope_t, torch::headeronly::BFloat16>) {
return;
} else {
#endif
using ScalarConverter = _typeConvert<scalar_t>;
using WeightConverter = _typeConvert<weight_t>;
using RopeConverter = _typeConvert<rope_t>;
using Scalar = typename ScalarConverter::hip_type;
using Weight = typename WeightConverter::hip_type;
using Rope = typename RopeConverter::hip_type;
// Preserve the original per-head reduction and activation rounding.
static_assert(!WIDE_192_ROTARY || HEAD_DIM == 192);
constexpr int kThreadsPerHead =
WIDE_192_ROTARY ? 32
: (HEAD_DIM <= 64 ? 8 : (HEAD_DIM <= 192 ? 16 : 32));
constexpr int kValuesPerVector =
WIDE_192_ROTARY ? 8 : (HEAD_DIM == 192 ? 4 : 8);
constexpr int kValuesPerTile = kThreadsPerHead * kValuesPerVector;
constexpr int kNumVectors =
(HEAD_DIM + kValuesPerTile - 1) / kValuesPerTile;
constexpr int kValuesPerThread = kValuesPerVector * kNumVectors;
constexpr int kTileSize = kValuesPerTile * kNumVectors;
constexpr int kHeadsPerBlock = kThreadsPerBlock / kThreadsPerHead;
const auto* qkv = static_cast<const Scalar*>(qkv_void);
auto* q_out = static_cast<Scalar*>(q_out_void);
auto* k_out = static_cast<Scalar*>(k_out_void);
auto* v_out = static_cast<Scalar*>(v_out_void);
const auto* q_weight = static_cast<const Weight*>(q_weight_void);
const auto* k_weight = static_cast<const Weight*>(k_weight_void);
const auto* cos = static_cast<const Rope*>(cos_void);
const auto* sin = static_cast<const Rope*>(sin_void);
auto* k_cache = static_cast<Scalar*>(k_cache_void);
auto* v_cache = static_cast<Scalar*>(v_cache_void);
const int head_in_block = threadIdx.x / kThreadsPerHead;
const int lane = threadIdx.x % kThreadsPerHead;
const int total_heads = num_q_heads + 2 * num_kv_heads;
int token;
int local_head;
const int64_t global_head =
static_cast<int64_t>(blockIdx.x) * kHeadsPerBlock + head_in_block;
token = global_head / total_heads;
// Reuse the quotient instead of spelling this as global_head %
// total_heads. nvcc otherwise emits a second runtime integer-division
// sequence for the flattened token/head mapping.
local_head = static_cast<int>(global_head -
static_cast<int64_t>(token) * total_heads);
if (token >= num_tokens) return;
const bool is_q = local_head < num_q_heads;
const bool is_k = !is_q && local_head < num_q_heads + num_kv_heads;
const int head = is_q ? local_head
: (is_k ? local_head - num_q_heads
: local_head - num_q_heads - num_kv_heads);
int64_t input_offset = token * qkv_token_stride;
Scalar* output;
int64_t output_offset;
if (is_q) {
input_offset += static_cast<int64_t>(head) * HEAD_DIM;
output = q_out;
output_offset =
token * q_token_stride + static_cast<int64_t>(head) * HEAD_DIM;
} else if (is_k) {
input_offset += static_cast<int64_t>(num_q_heads + head) * HEAD_DIM;
output = k_out;
output_offset =
token * k_token_stride + static_cast<int64_t>(head) * HEAD_DIM;
} else {
input_offset +=
static_cast<int64_t>(num_q_heads + num_kv_heads + head) * HEAD_DIM;
output = v_out;
output_offset =
token * v_token_stride + static_cast<int64_t>(head) * HEAD_DIM;
}
bool write_cache = false;
int64_t cache_offset = 0;
if (!is_q && token < num_slots) {
const int64_t slot = slot_mapping[token];
if (slot >= 0) {
int64_t block;
int64_t page;
if (block_size_shift >= 0) {
// vLLM production cache blocks are powers of two. Avoid the costly
// runtime int64 division on that path while retaining generic block
// sizes through the fallback below.
block = slot >> block_size_shift;
page = slot & (block_size - 1);
} else {
block = slot / block_size;
page = slot - block * block_size;
}
cache_offset = block * cache_block_stride + page * cache_page_stride +
static_cast<int64_t>(head) * cache_head_stride;
write_cache = true;
}
}
if (!is_q && !is_k) {
#pragma unroll
for (int block = 0; block < kNumVectors; ++block) {
#pragma unroll
for (int element = 0; element < kValuesPerVector; ++element) {
const int dim = block * kThreadsPerHead * kValuesPerVector +
lane * kValuesPerVector + element;
if constexpr (kTileSize > HEAD_DIM) {
if (dim >= HEAD_DIM) continue;
}
const Scalar value = qkv[input_offset + dim];
output[output_offset + dim] = value;
if (write_cache) {
if constexpr (FP8_CACHE) {
reinterpret_cast<uint8_t*>(v_cache_void)[cache_offset + dim] =
fp8::scaled_convert<uint8_t, float,
Fp8KVCacheDataType::kFp8E4M3>(
ScalarConverter::convert(value), *v_scale);
} else {
v_cache[cache_offset + dim] = value;
}
}
}
}
return;
}
float values[kValuesPerThread];
float sum_squares = 0.0f;
#pragma unroll
for (int block = 0; block < kNumVectors; ++block) {
#pragma unroll
for (int element = 0; element < kValuesPerVector; ++element) {
const int value_index = block * kValuesPerVector + element;
const int dim = block * kThreadsPerHead * kValuesPerVector +
lane * kValuesPerVector + element;
float value = 0.0f;
if constexpr (kTileSize > HEAD_DIM) {
if (dim < HEAD_DIM) {
value = ScalarConverter::convert(qkv[input_offset + dim]);
}
} else {
value = ScalarConverter::convert(qkv[input_offset + dim]);
}
values[value_index] = value;
// CuTeDSL lowers its fragment reduction to a sequential fma.rn
// chain. Keep this explicit: changing it changes RMSNorm bits.
sum_squares = __fmaf_rn(value, value, sum_squares);
}
}
const int warp_lane = threadIdx.x % kWarpSize;
const int subwarp_base = (warp_lane / kThreadsPerHead) * kThreadsPerHead;
constexpr uint32_t kSubwarpBits =
kThreadsPerHead == kWarpSize ? 0xffffffffu
: ((uint32_t{1} << kThreadsPerHead) - 1);
const uint32_t subwarp_mask = kSubwarpBits << subwarp_base;
#pragma unroll
for (int offset = 1; offset < kThreadsPerHead; offset <<= 1) {
// CuTeDSL's warp_reduce uses ascending XOR offsets (1, 2, 4, ...).
sum_squares +=
__shfl_xor_sync(subwarp_mask, sum_squares, offset, kWarpSize);
}
const float mean_square =
__fdiv_rn(sum_squares, static_cast<float>(HEAD_DIM));
const float inverse_rms = rsqrtf(mean_square + epsilon);
const Weight* weight = is_q ? q_weight : k_weight;
// CuTeDSL writes RMSNorm results to activation-typed shared memory and
// reloads them for RoPE. This intermediate FP16/BF16 rounding boundary
// is intentional and is necessary for bitwise output parity.
__shared__ Scalar normalized[kHeadsPerBlock * HEAD_DIM];
Scalar* normalized_head = normalized + head_in_block * HEAD_DIM;
#pragma unroll
for (int block = 0; block < kNumVectors; ++block) {
#pragma unroll
for (int element = 0; element < kValuesPerVector; ++element) {
const int value_index = block * kValuesPerVector + element;
const int dim = block * kThreadsPerHead * kValuesPerVector +
lane * kValuesPerVector + element;
if constexpr (kTileSize > HEAD_DIM) {
if (dim >= HEAD_DIM) continue;
}
const float x_hat = values[value_index] * inverse_rms;
const float norm_weight =
WeightConverter::convert(weight[dim]) + norm_weight_bias;
normalized_head[dim] = ScalarConverter::convert(x_hat * norm_weight);
}
}
__syncwarp(subwarp_mask);
// Apply NeoX-style RoPE. CuTeDSL assigns four rotary pairs to each
// participating lane and reloads activation-typed normalized values.
constexpr int kRotaryPairsPerThread = 4;
if (lane < rotary_pairs / kRotaryPairsPerThread) {
const int64_t position = positions[token];
const Rope* token_cos = cos + position * cos_token_stride;
const Rope* token_sin = sin + position * sin_token_stride;
#pragma unroll
for (int element = 0; element < kRotaryPairsPerThread; ++element) {
const int pair = lane * kRotaryPairsPerThread + element;
const int dim0 = pair;
const int dim1 = pair + rotary_pairs;
const float value0 = ScalarConverter::convert(normalized_head[dim0]);
const float value1 = ScalarConverter::convert(normalized_head[dim1]);
const float cos_value = RopeConverter::convert(token_cos[pair]);
const float sin_value = RopeConverter::convert(token_sin[pair]);
// Match CuTeDSL PTX exactly. The first half is mul + mul + sub; the
// second is one mul followed by one fused multiply-add.
const float rotated0 = __fsub_rn(__fmul_rn(value0, cos_value),
__fmul_rn(value1, sin_value));
const float rotated1 =
__fmaf_rn(value0, sin_value, __fmul_rn(value1, cos_value));
normalized_head[dim0] = ScalarConverter::convert(rotated0);
normalized_head[dim1] = ScalarConverter::convert(rotated1);
}
}
__syncwarp(subwarp_mask);
#pragma unroll
for (int block = 0; block < kNumVectors; ++block) {
#pragma unroll
for (int element = 0; element < kValuesPerVector; ++element) {
const int dim = block * kThreadsPerHead * kValuesPerVector +
lane * kValuesPerVector + element;
if constexpr (kTileSize > HEAD_DIM) {
if (dim >= HEAD_DIM) continue;
}
const Scalar value = normalized_head[dim];
output[output_offset + dim] = value;
if (write_cache) {
if constexpr (FP8_CACHE) {
reinterpret_cast<uint8_t*>(k_cache_void)[cache_offset + dim] =
fp8::scaled_convert<uint8_t, float,
Fp8KVCacheDataType::kFp8E4M3>(
ScalarConverter::convert(value), *k_scale);
} else {
k_cache[cache_offset + dim] = value;
}
}
}
}
#if !defined(__CUDA_ARCH__) || __CUDA_ARCH__ < 800
}
#endif
}
template <typename scalar_t, typename weight_t, typename rope_t, bool FP8_CACHE>
void launch_qknorm_rope_cache(
const torch::stable::Tensor& qkv, torch::stable::Tensor& q_out,
torch::stable::Tensor& k_out, torch::stable::Tensor& v_out,
const torch::stable::Tensor& q_weight,
const torch::stable::Tensor& k_weight, const torch::stable::Tensor& cos,
const torch::stable::Tensor& sin, const torch::stable::Tensor& positions,
const torch::stable::Tensor& slot_mapping, torch::stable::Tensor& k_cache,
torch::stable::Tensor& v_cache, int head_dim, int num_q_heads,
int num_kv_heads, int rotary_pairs, float epsilon, float norm_weight_bias,
cudaStream_t stream, const float* k_scale, const float* v_scale) {
const int total_heads = num_q_heads + 2 * num_kv_heads;
const int64_t total_work = qkv.size(0) * total_heads;
const int block_size = static_cast<int>(k_cache.size(1));
const int block_size_shift =
(block_size & (block_size - 1)) == 0
? __builtin_ctz(static_cast<unsigned int>(block_size))
: -1;
#define LAUNCH_QKNORM_ROPE_CACHE(HEAD_DIM, WIDE_192_ROTARY) \
do { \
constexpr int threads_per_head = \
WIDE_192_ROTARY ? 32 \
: (HEAD_DIM <= 64 ? 8 : (HEAD_DIM <= 192 ? 16 : 32)); \
constexpr int heads_per_block = kThreadsPerBlock / threads_per_head; \
const dim3 grid((total_work + heads_per_block - 1) / heads_per_block); \
const dim3 block(kThreadsPerBlock); \
qknorm_rope_cache_kernel<scalar_t, weight_t, rope_t, HEAD_DIM, \
WIDE_192_ROTARY, FP8_CACHE> \
<<<grid, block, 0, stream>>>( \
qkv.data_ptr(), q_out.data_ptr(), k_out.data_ptr(), \
v_out.data_ptr(), q_weight.data_ptr(), k_weight.data_ptr(), \
cos.data_ptr(), sin.data_ptr(), \
reinterpret_cast<const int64_t*>(positions.data_ptr()), \
reinterpret_cast<const int64_t*>(slot_mapping.data_ptr()), \
k_cache.data_ptr(), v_cache.data_ptr(), qkv.stride(0), \
q_out.stride(0), k_out.stride(0), v_out.stride(0), cos.stride(0), \
sin.stride(0), k_cache.stride(0), k_cache.stride(1), \
k_cache.stride(2), static_cast<int>(qkv.size(0)), \
static_cast<int>(slot_mapping.numel()), num_q_heads, num_kv_heads, \
rotary_pairs, block_size, epsilon, block_size_shift, \
norm_weight_bias, k_scale, v_scale); \
} while (false)
switch (head_dim) {
case 64:
LAUNCH_QKNORM_ROPE_CACHE(64, false);
break;
case 128:
LAUNCH_QKNORM_ROPE_CACHE(128, false);
break;
case 192:
if (rotary_pairs > 64) {
LAUNCH_QKNORM_ROPE_CACHE(192, true);
} else {
LAUNCH_QKNORM_ROPE_CACHE(192, false);
}
break;
case 256:
LAUNCH_QKNORM_ROPE_CACHE(256, false);
break;
default:
STD_TORCH_CHECK(false, "Unsupported head_dim: ", head_dim);
}
#undef LAUNCH_QKNORM_ROPE_CACHE
}
void check_cuda_tensor(const torch::stable::Tensor& tensor, const char* name) {
STD_TORCH_CHECK(tensor.is_cuda(), name, " must be a CUDA tensor");
}
void check_same_device(const torch::stable::Tensor& reference,
const torch::stable::Tensor& tensor, const char* name) {
STD_TORCH_CHECK(tensor.get_device_index() == reference.get_device_index(),
name, " must be on the same CUDA device as qkv");
}
} // namespace
void fused_qknorm_rope_cache(
torch::stable::Tensor& q_out, torch::stable::Tensor& k_out,
torch::stable::Tensor& v_out, torch::stable::Tensor& qkv,
torch::stable::Tensor& q_weight, torch::stable::Tensor& k_weight,
torch::stable::Tensor& cos, torch::stable::Tensor& sin,
torch::stable::Tensor& positions, torch::stable::Tensor& slot_mapping,
torch::stable::Tensor& k_cache, torch::stable::Tensor& v_cache,
int64_t head_dim, int64_t num_q_heads, int64_t num_kv_heads,
int64_t rotary_pairs, double epsilon, double norm_weight_bias,
const std::string& kv_cache_dtype,
std::optional<torch::stable::Tensor> k_scale,
std::optional<torch::stable::Tensor> v_scale) {
check_cuda_tensor(qkv, "qkv");
check_cuda_tensor(q_out, "q_out");
check_cuda_tensor(k_out, "k_out");
check_cuda_tensor(v_out, "v_out");
check_cuda_tensor(q_weight, "q_weight");
check_cuda_tensor(k_weight, "k_weight");
check_cuda_tensor(cos, "cos");
check_cuda_tensor(sin, "sin");
check_cuda_tensor(positions, "positions");
check_cuda_tensor(slot_mapping, "slot_mapping");
check_cuda_tensor(k_cache, "k_cache");
check_cuda_tensor(v_cache, "v_cache");
check_same_device(qkv, q_out, "q_out");
check_same_device(qkv, k_out, "k_out");
check_same_device(qkv, v_out, "v_out");
check_same_device(qkv, q_weight, "q_weight");
check_same_device(qkv, k_weight, "k_weight");
check_same_device(qkv, cos, "cos");
check_same_device(qkv, sin, "sin");
check_same_device(qkv, positions, "positions");
check_same_device(qkv, slot_mapping, "slot_mapping");
check_same_device(qkv, k_cache, "k_cache");
check_same_device(qkv, v_cache, "v_cache");
STD_TORCH_CHECK(qkv.dim() == 2 && qkv.stride(1) == 1,
"qkv must be a 2D tensor contiguous in its last dimension");
STD_TORCH_CHECK(q_out.dim() == 2 && q_out.is_contiguous(),
"q_out must be contiguous and 2D");
STD_TORCH_CHECK(k_out.dim() == 2 && k_out.is_contiguous(),
"k_out must be contiguous and 2D");
STD_TORCH_CHECK(v_out.dim() == 2 && v_out.is_contiguous(),
"v_out must be contiguous and 2D");
STD_TORCH_CHECK(q_weight.dim() == 1 && q_weight.is_contiguous(),
"q_weight must be contiguous and 1D");
STD_TORCH_CHECK(k_weight.dim() == 1 && k_weight.is_contiguous(),
"k_weight must be contiguous and 1D");
STD_TORCH_CHECK(cos.dim() == 2 && cos.stride(1) == 1,
"cos must be 2D and contiguous in its last dimension");
STD_TORCH_CHECK(sin.dim() == 2 && sin.stride(1) == 1,
"sin must be 2D and contiguous in its last dimension");
STD_TORCH_CHECK(
positions.scalar_type() == torch::headeronly::ScalarType::Long,
"positions must be an int64 tensor");
STD_TORCH_CHECK(positions.is_contiguous(), "positions must be contiguous");
STD_TORCH_CHECK(
slot_mapping.scalar_type() == torch::headeronly::ScalarType::Long,
"slot_mapping must be an int64 tensor");
STD_TORCH_CHECK(slot_mapping.is_contiguous(),
"slot_mapping must be contiguous");
STD_TORCH_CHECK(k_cache.dim() == 4 && v_cache.dim() == 4,
"K/V cache must have shape [blocks, block_size, heads, dim]");
for (int64_t d = 0; d < 4; ++d) {
STD_TORCH_CHECK(k_cache.size(d) == v_cache.size(d) &&
k_cache.stride(d) == v_cache.stride(d),
"K/V cache must have identical shapes and strides");
}
STD_TORCH_CHECK(k_cache.stride(3) == 1,
"K/V cache head dimension must be contiguous");
STD_TORCH_CHECK(num_q_heads > 0 && num_kv_heads > 0,
"Q/KV head counts must be positive");
STD_TORCH_CHECK(
head_dim == 64 || head_dim == 128 || head_dim == 192 || head_dim == 256,
"Unsupported head_dim");
STD_TORCH_CHECK(cos.size(0) == sin.size(0),
"cos/sin cache must have the same number of positions");
const int64_t num_tokens = qkv.size(0);
const int64_t packed_width = (num_q_heads + 2 * num_kv_heads) * head_dim;
STD_TORCH_CHECK(qkv.size(1) == packed_width,
"qkv width must equal (num_q_heads + 2 * num_kv_heads) * "
"head_dim");
STD_TORCH_CHECK(
q_out.size(0) == num_tokens && q_out.size(1) == num_q_heads * head_dim,
"q_out has an invalid shape");
STD_TORCH_CHECK(
k_out.size(0) == num_tokens && k_out.size(1) == num_kv_heads * head_dim,
"k_out has an invalid shape");
STD_TORCH_CHECK(
v_out.size(0) == k_out.size(0) && v_out.size(1) == k_out.size(1),
"v_out has an invalid shape");
STD_TORCH_CHECK(q_weight.numel() == head_dim && k_weight.numel() == head_dim,
"Q/K norm weights must have head_dim elements");
STD_TORCH_CHECK(positions.numel() == num_tokens,
"positions must have one entry per qkv token");
STD_TORCH_CHECK(rotary_pairs >= 0 && rotary_pairs * 2 <= head_dim &&
rotary_pairs % 4 == 0,
"rotary_pairs must be a multiple of 4 in [0, head_dim / 2]");
STD_TORCH_CHECK(cos.size(1) >= rotary_pairs && sin.size(1) >= rotary_pairs,
"cos/sin cache is smaller than rotary_pairs");
STD_TORCH_CHECK(
k_cache.size(2) == num_kv_heads && k_cache.size(3) == head_dim,
"K/V cache head shape does not match the kernel arguments");
STD_TORCH_CHECK(k_cache.size(1) > 0, "K/V cache block_size must be positive");
STD_TORCH_CHECK(slot_mapping.numel() <= num_tokens,
"slot_mapping cannot contain more entries than qkv tokens");
STD_TORCH_CHECK(qkv.scalar_type() == q_out.scalar_type() &&
qkv.scalar_type() == k_out.scalar_type() &&
qkv.scalar_type() == v_out.scalar_type(),
"qkv and Q/K/V outputs must have the same dtype");
const bool fp8_cache =
kv_cache_dtype == "fp8" || kv_cache_dtype == "fp8_e4m3";
STD_TORCH_CHECK(fp8_cache || kv_cache_dtype == "auto" ||
kv_cache_dtype == "bfloat16" ||
kv_cache_dtype == "float16",
"Unsupported KV cache dtype: ", kv_cache_dtype);
STD_TORCH_CHECK(k_cache.scalar_type() == v_cache.scalar_type(),
"K/V caches must have the same dtype");
if (fp8_cache) {
STD_TORCH_CHECK(
k_cache.scalar_type() == torch::headeronly::ScalarType::Byte ||
k_cache.scalar_type() ==
torch::headeronly::ScalarType::Float8_e4m3fn,
"FP8 cache requires uint8 or float8_e4m3fn storage");
STD_TORCH_CHECK(k_scale.has_value() && v_scale.has_value(),
"FP8 cache requires K/V scales");
for (const auto* scale : {&*k_scale, &*v_scale}) {
check_cuda_tensor(*scale, "KV scale");
check_same_device(qkv, *scale, "KV scale");
STD_TORCH_CHECK(
scale->numel() == 1 &&
scale->scalar_type() == torch::headeronly::ScalarType::Float,
"KV scales must be single FP32 values");
}
} else {
STD_TORCH_CHECK(qkv.scalar_type() == k_cache.scalar_type(),
"Unquantized cache must match QKV dtype");
}
STD_TORCH_CHECK(q_weight.scalar_type() == k_weight.scalar_type(),
"Q/K norm weights must have the same dtype");
STD_TORCH_CHECK(cos.scalar_type() == sin.scalar_type(),
"cos and sin must have the same dtype");
if (num_tokens == 0) return;
const torch::stable::accelerator::DeviceGuard device_guard(
qkv.get_device_index());
const cudaStream_t stream = get_current_cuda_stream(qkv.get_device_index());
VLLM_STABLE_DISPATCH_HALF_TYPES(
qkv.scalar_type(), "qknorm_rope_cache_input", [&] {
using input_t = scalar_t;
VLLM_STABLE_DISPATCH_FLOATING_TYPES(
q_weight.scalar_type(), "qknorm_rope_cache_weight", [&] {
using norm_weight_t = scalar_t;
VLLM_STABLE_DISPATCH_FLOATING_TYPES(
cos.scalar_type(), "qknorm_rope_cache_rope", [&] {
using rope_cache_t = scalar_t;
auto launch = [&](auto fp8_tag) {
launch_qknorm_rope_cache<input_t, norm_weight_t,
rope_cache_t,
decltype(fp8_tag)::value>(
qkv, q_out, k_out, v_out, q_weight, k_weight, cos,
sin, positions, slot_mapping, k_cache, v_cache,
static_cast<int>(head_dim),
static_cast<int>(num_q_heads),
static_cast<int>(num_kv_heads),
static_cast<int>(rotary_pairs),
static_cast<float>(epsilon),
static_cast<float>(norm_weight_bias), stream,
fp8_cache
? static_cast<const float*>(k_scale->data_ptr())
: nullptr,
fp8_cache
? static_cast<const float*>(v_scale->data_ptr())
: nullptr);
};
if (fp8_cache) {
launch(std::true_type{});
} else {
launch(std::false_type{});
}
});
});
});
STD_CUDA_KERNEL_LAUNCH_CHECK();
}
} // namespace vllm
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,70 @@
// auto generated by generate_kernels.py
// clang-format off
#include "kernel.h"
#include "marlin_template.h"
namespace MARLIN_NAMESPACE_NAME {
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), 256, 1, 8, 8, true, 4, 1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), 128, 1, 8, 4, true, 4, 1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), 128, 1, 4, 8, true, 4, 1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), 256, 1, 8, 8, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), 128, 1, 8, 4, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), 128, 1, 4, 8, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), 256, 2, 16, 4, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), 128, 2, 8, 4, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), 128, 2, 4, 8, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), 256, 3, 16, 4, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), 128, 3, 8, 4, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), 128, 3, 4, 8, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), 256, 4, 16, 4, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), 128, 4, 8, 4, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), 128, 4, 4, 8, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 256, 1, 8, 8, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 1, 8, 4, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 1, 4, 8, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 256, 1, 8, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 1, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 1, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 256, 2, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 2, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 2, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 256, 3, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 3, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 3, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 256, 4, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 4, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 4, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
}
@@ -0,0 +1,100 @@
// auto generated by generate_kernels.py
// clang-format off
#include "kernel.h"
#include "marlin_template.h"
namespace MARLIN_NAMESPACE_NAME {
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 256, 1, 8, 8, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 1, 8, 4, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 1, 4, 8, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 256, 1, 8, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 1, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 1, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 256, 2, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 2, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 2, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 256, 3, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 3, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 3, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 256, 4, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 4, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 4, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
}
@@ -0,0 +1,130 @@
// auto generated by generate_kernels.py
// clang-format off
#include "kernel.h"
#include "marlin_template.h"
namespace MARLIN_NAMESPACE_NAME {
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, true, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, true, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, true, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
}
@@ -0,0 +1,130 @@
// auto generated by generate_kernels.py
// clang-format off
#include "kernel.h"
#include "marlin_template.h"
namespace MARLIN_NAMESPACE_NAME {
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, true, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, true, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, true, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
}
@@ -0,0 +1,130 @@
// auto generated by generate_kernels.py
// clang-format off
#include "kernel.h"
#include "marlin_template.h"
namespace MARLIN_NAMESPACE_NAME {
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, true, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, true, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, true, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
}
@@ -0,0 +1,40 @@
// auto generated by generate_kernels.py
// clang-format off
#include "kernel.h"
#include "marlin_template.h"
namespace MARLIN_NAMESPACE_NAME {
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE2M1f.id(), vllm::kFloat16.id(), vllm::kFE4M3fn.id(), 256, 1, 8, 8, true, 4, 1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE2M1f.id(), vllm::kFloat16.id(), vllm::kFE4M3fn.id(), 128, 1, 8, 4, true, 4, 1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE2M1f.id(), vllm::kFloat16.id(), vllm::kFE4M3fn.id(), 128, 1, 4, 8, true, 4, 1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE2M1f.id(), vllm::kFloat16.id(), vllm::kFE4M3fn.id(), 256, 1, 8, 8, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE2M1f.id(), vllm::kFloat16.id(), vllm::kFE4M3fn.id(), 128, 1, 8, 4, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE2M1f.id(), vllm::kFloat16.id(), vllm::kFE4M3fn.id(), 128, 1, 4, 8, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE2M1f.id(), vllm::kFloat16.id(), vllm::kFE4M3fn.id(), 256, 2, 16, 4, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE2M1f.id(), vllm::kFloat16.id(), vllm::kFE4M3fn.id(), 128, 2, 8, 4, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE2M1f.id(), vllm::kFloat16.id(), vllm::kFE4M3fn.id(), 128, 2, 4, 8, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE2M1f.id(), vllm::kFloat16.id(), vllm::kFE4M3fn.id(), 256, 3, 16, 4, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE2M1f.id(), vllm::kFloat16.id(), vllm::kFE4M3fn.id(), 128, 3, 8, 4, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE2M1f.id(), vllm::kFloat16.id(), vllm::kFE4M3fn.id(), 128, 3, 4, 8, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE2M1f.id(), vllm::kFloat16.id(), vllm::kFE4M3fn.id(), 256, 4, 16, 4, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE2M1f.id(), vllm::kFloat16.id(), vllm::kFE4M3fn.id(), 128, 4, 8, 4, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE2M1f.id(), vllm::kFloat16.id(), vllm::kFE4M3fn.id(), 128, 4, 4, 8, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
}
@@ -0,0 +1,70 @@
// auto generated by generate_kernels.py
// clang-format off
#include "kernel.h"
#include "marlin_template.h"
namespace MARLIN_NAMESPACE_NAME {
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
}
@@ -0,0 +1,130 @@
// auto generated by generate_kernels.py
// clang-format off
#include "kernel.h"
#include "marlin_template.h"
namespace MARLIN_NAMESPACE_NAME {
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, true, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, true, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, true, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
}
@@ -0,0 +1,130 @@
// auto generated by generate_kernels.py
// clang-format off
#include "kernel.h"
#include "marlin_template.h"
namespace MARLIN_NAMESPACE_NAME {
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, true, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, true, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, true, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
}
@@ -0,0 +1,130 @@
// auto generated by generate_kernels.py
// clang-format off
#include "kernel.h"
#include "marlin_template.h"
namespace MARLIN_NAMESPACE_NAME {
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, true, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, true, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, true, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
}
@@ -0,0 +1,106 @@
// auto generated by generate_kernels.py
// clang-format off
#include "kernel.h"
#include "marlin_template.h"
namespace MARLIN_NAMESPACE_NAME {
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
}
@@ -0,0 +1,106 @@
// auto generated by generate_kernels.py
// clang-format off
#include "kernel.h"
#include "marlin_template.h"
namespace MARLIN_NAMESPACE_NAME {
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
}
@@ -0,0 +1,106 @@
// auto generated by generate_kernels.py
// clang-format off
#include "kernel.h"
#include "marlin_template.h"
namespace MARLIN_NAMESPACE_NAME {
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
}
@@ -0,0 +1,106 @@
// auto generated by generate_kernels.py
// clang-format off
#include "kernel.h"
#include "marlin_template.h"
namespace MARLIN_NAMESPACE_NAME {
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
}
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,70 @@
// auto generated by generate_kernels.py
// clang-format off
#include "kernel.h"
#include "marlin_template.h"
namespace MARLIN_NAMESPACE_NAME {
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), 256, 1, 8, 8, true, 4, 1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), 128, 1, 8, 4, true, 4, 1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), 128, 1, 4, 8, true, 4, 1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), 256, 1, 8, 8, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), 128, 1, 8, 4, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), 128, 1, 4, 8, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), 256, 2, 16, 4, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), 128, 2, 8, 4, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), 128, 2, 4, 8, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), 256, 3, 16, 4, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), 128, 3, 8, 4, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), 128, 3, 4, 8, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), 256, 4, 16, 4, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), 128, 4, 8, 4, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), 128, 4, 4, 8, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 256, 1, 8, 8, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 1, 8, 4, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 1, 4, 8, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 256, 1, 8, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 1, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 1, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 256, 2, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 2, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 2, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 256, 3, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 3, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 3, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 256, 4, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 4, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 4, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
}
@@ -0,0 +1,100 @@
// auto generated by generate_kernels.py
// clang-format off
#include "kernel.h"
#include "marlin_template.h"
namespace MARLIN_NAMESPACE_NAME {
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 256, 1, 8, 8, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 1, 8, 4, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 1, 4, 8, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 256, 1, 8, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 1, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 1, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 256, 2, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 2, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 2, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 256, 3, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 3, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 3, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 256, 4, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 4, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 4, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
}
@@ -0,0 +1,130 @@
// auto generated by generate_kernels.py
// clang-format off
#include "kernel.h"
#include "marlin_template.h"
namespace MARLIN_NAMESPACE_NAME {
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, true, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, true, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, true, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
}
@@ -0,0 +1,130 @@
// auto generated by generate_kernels.py
// clang-format off
#include "kernel.h"
#include "marlin_template.h"
namespace MARLIN_NAMESPACE_NAME {
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, true, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, true, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, true, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
}
@@ -0,0 +1,130 @@
// auto generated by generate_kernels.py
// clang-format off
#include "kernel.h"
#include "marlin_template.h"
namespace MARLIN_NAMESPACE_NAME {
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, true, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, true, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, true, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
}
@@ -0,0 +1,40 @@
// auto generated by generate_kernels.py
// clang-format off
#include "kernel.h"
#include "marlin_template.h"
namespace MARLIN_NAMESPACE_NAME {
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE2M1f.id(), vllm::kFloat16.id(), vllm::kFE4M3fn.id(), 256, 1, 8, 8, true, 4, 1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE2M1f.id(), vllm::kFloat16.id(), vllm::kFE4M3fn.id(), 128, 1, 8, 4, true, 4, 1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE2M1f.id(), vllm::kFloat16.id(), vllm::kFE4M3fn.id(), 128, 1, 4, 8, true, 4, 1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE2M1f.id(), vllm::kFloat16.id(), vllm::kFE4M3fn.id(), 256, 1, 8, 8, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE2M1f.id(), vllm::kFloat16.id(), vllm::kFE4M3fn.id(), 128, 1, 8, 4, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE2M1f.id(), vllm::kFloat16.id(), vllm::kFE4M3fn.id(), 128, 1, 4, 8, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE2M1f.id(), vllm::kFloat16.id(), vllm::kFE4M3fn.id(), 256, 2, 16, 4, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE2M1f.id(), vllm::kFloat16.id(), vllm::kFE4M3fn.id(), 128, 2, 8, 4, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE2M1f.id(), vllm::kFloat16.id(), vllm::kFE4M3fn.id(), 128, 2, 4, 8, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE2M1f.id(), vllm::kFloat16.id(), vllm::kFE4M3fn.id(), 256, 3, 16, 4, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE2M1f.id(), vllm::kFloat16.id(), vllm::kFE4M3fn.id(), 128, 3, 8, 4, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE2M1f.id(), vllm::kFloat16.id(), vllm::kFE4M3fn.id(), 128, 3, 4, 8, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE2M1f.id(), vllm::kFloat16.id(), vllm::kFE4M3fn.id(), 256, 4, 16, 4, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE2M1f.id(), vllm::kFloat16.id(), vllm::kFE4M3fn.id(), 128, 4, 8, 4, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE2M1f.id(), vllm::kFloat16.id(), vllm::kFE4M3fn.id(), 128, 4, 4, 8, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
}
@@ -0,0 +1,70 @@
// auto generated by generate_kernels.py
// clang-format off
#include "kernel.h"
#include "marlin_template.h"
namespace MARLIN_NAMESPACE_NAME {
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
}
@@ -0,0 +1,130 @@
// auto generated by generate_kernels.py
// clang-format off
#include "kernel.h"
#include "marlin_template.h"
namespace MARLIN_NAMESPACE_NAME {
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, true, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, true, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, true, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
}
@@ -0,0 +1,130 @@
// auto generated by generate_kernels.py
// clang-format off
#include "kernel.h"
#include "marlin_template.h"
namespace MARLIN_NAMESPACE_NAME {
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, true, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, true, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, true, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
}
@@ -0,0 +1,130 @@
// auto generated by generate_kernels.py
// clang-format off
#include "kernel.h"
#include "marlin_template.h"
namespace MARLIN_NAMESPACE_NAME {
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, true, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, true, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, true, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
}
@@ -0,0 +1,106 @@
// auto generated by generate_kernels.py
// clang-format off
#include "kernel.h"
#include "marlin_template.h"
namespace MARLIN_NAMESPACE_NAME {
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
}
@@ -0,0 +1,106 @@
// auto generated by generate_kernels.py
// clang-format off
#include "kernel.h"
#include "marlin_template.h"
namespace MARLIN_NAMESPACE_NAME {
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
}
@@ -0,0 +1,106 @@
// auto generated by generate_kernels.py
// clang-format off
#include "kernel.h"
#include "marlin_template.h"
namespace MARLIN_NAMESPACE_NAME {
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
}
@@ -0,0 +1,106 @@
// auto generated by generate_kernels.py
// clang-format off
#include "kernel.h"
#include "marlin_template.h"
namespace MARLIN_NAMESPACE_NAME {
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
}
-100
View File
@@ -1,100 +0,0 @@
// SPDX-License-Identifier: Apache-2.0
// SPDX-FileCopyrightText: Copyright contributors to the vLLM project
#include <cuda_runtime.h>
#include <torch/csrc/stable/accelerator.h>
#include <torch/csrc/stable/tensor.h>
#include <torch/headeronly/core/ScalarType.h>
#include <algorithm>
#include "dispatch_utils.h"
#include "quantization/vectorization_utils.cuh"
#include "torch_utils.h"
namespace vllm {
// 16 B of output per thread; low-precision inputs are widened in registers.
static constexpr int kMuseSoftCapVecSize = 4;
// blockIdx.x walks rows, blockIdx.y splits a row. Rows are the sampled tokens
// (often one, at decode) while a row is the whole vocab, so splitting the row
// is what keeps the GPU busy; a row-per-block grid would leave a single-token
// decode on one SM.
template <typename scalar_t>
__global__ void muse_spark_mul_tanh_soft_cap_kernel(
float* __restrict__ out, const scalar_t* __restrict__ input, int d,
float multiplier, float soft_cap_temp) {
const int64_t row_offset = static_cast<int64_t>(blockIdx.x) * d;
const scalar_t* __restrict__ in_row = input + row_offset;
float* __restrict__ out_row = out + row_offset;
const int tid = blockIdx.y * blockDim.x + threadIdx.x;
const int stride = gridDim.y * blockDim.x;
vectorize_with_alignment<kMuseSoftCapVecSize>(
in_row, out_row, d, tid, stride,
[=] __device__(float& dst, const scalar_t& src) {
// Multiply then divide, as MulTanhSoftCap.forward does, so fp32
// rounding matches the eager path.
dst = soft_cap_temp *
tanhf(static_cast<float>(src) * multiplier / soft_cap_temp);
});
}
} // namespace vllm
// out = soft_cap_temp * tanh(input * multiplier / soft_cap_temp), fp32.
void muse_spark_mul_tanh_soft_cap(torch::stable::Tensor& out,
torch::stable::Tensor& input,
double multiplier, double soft_cap_temp) {
STD_TORCH_CHECK(
input.scalar_type() == torch::headeronly::ScalarType::Float ||
input.scalar_type() == torch::headeronly::ScalarType::Half ||
input.scalar_type() == torch::headeronly::ScalarType::BFloat16,
"muse_spark_mul_tanh_soft_cap: input must be float32, float16 or "
"bfloat16");
STD_TORCH_CHECK(out.scalar_type() == torch::headeronly::ScalarType::Float,
"muse_spark_mul_tanh_soft_cap: out must be float32");
STD_TORCH_CHECK(input.is_contiguous(),
"muse_spark_mul_tanh_soft_cap: input must be contiguous");
STD_TORCH_CHECK(out.is_contiguous(),
"muse_spark_mul_tanh_soft_cap: out must be contiguous");
STD_TORCH_CHECK(out.numel() == input.numel(),
"muse_spark_mul_tanh_soft_cap: out and input must have the "
"same number of elements");
STD_TORCH_CHECK(
soft_cap_temp != 0.0,
"muse_spark_mul_tanh_soft_cap: soft_cap_temp must be nonzero");
if (input.numel() == 0) {
return;
}
const int d = static_cast<int>(input.size(input.dim() - 1));
const int64_t num_tokens = input.numel() / d;
constexpr int kThreads = 128;
const int64_t vecs_per_row =
(d + vllm::kMuseSoftCapVecSize - 1) / vllm::kMuseSoftCapVecSize;
const int64_t blocks_to_cover = (vecs_per_row + kThreads - 1) / kThreads;
constexpr int64_t kTargetBlocks = 32768;
const int64_t blocks_per_row = std::clamp<int64_t>(
kTargetBlocks / num_tokens, 1, std::max<int64_t>(blocks_to_cover, 1));
dim3 grid(static_cast<unsigned>(num_tokens),
static_cast<unsigned>(blocks_per_row));
dim3 block(kThreads);
const torch::stable::accelerator::DeviceGuard device_guard(
input.get_device_index());
const cudaStream_t stream = get_current_cuda_stream();
VLLM_STABLE_DISPATCH_FLOATING_TYPES(
input.scalar_type(), "muse_spark_mul_tanh_soft_cap", [&] {
vllm::muse_spark_mul_tanh_soft_cap_kernel<scalar_t>
<<<grid, block, 0, stream>>>(out.mutable_data_ptr<float>(),
input.const_data_ptr<scalar_t>(), d,
static_cast<float>(multiplier),
static_cast<float>(soft_cap_temp));
});
}
+4 -33
View File
@@ -363,35 +363,6 @@ minimax_allreduce_rms_qk(torch::stable::Tensor qkv,
int64_t const nranks, double const eps); int64_t const nranks, double const eps);
#endif #endif
#ifndef USE_ROCM
namespace vllm {
void fused_qknorm_rope_cache(
torch::stable::Tensor& q_out, torch::stable::Tensor& k_out,
torch::stable::Tensor& v_out, torch::stable::Tensor& qkv,
torch::stable::Tensor& q_weight, torch::stable::Tensor& k_weight,
torch::stable::Tensor& cos, torch::stable::Tensor& sin,
torch::stable::Tensor& positions, torch::stable::Tensor& slot_mapping,
torch::stable::Tensor& k_cache, torch::stable::Tensor& v_cache,
int64_t head_dim, int64_t num_q_heads, int64_t num_kv_heads,
int64_t rotary_pairs, double epsilon, double norm_weight_bias,
const std::string& kv_cache_dtype,
std::optional<torch::stable::Tensor> k_scale,
std::optional<torch::stable::Tensor> v_scale);
} // namespace vllm
#endif
void fused_muse_spark_qknorm_rope_kv_insert(
torch::stable::Tensor& qkv, torch::stable::Tensor const& positions,
int64_t num_heads, int64_t num_kv_heads, int64_t head_dim, double eps,
std::optional<torch::stable::Tensor> cos_sin_cache, int64_t rotary_dim,
std::optional<torch::stable::Tensor> slot_mapping,
std::optional<torch::stable::Tensor> kv_cache, int64_t block_size,
const std::string& kv_cache_dtype,
std::optional<torch::stable::Tensor> k_scale,
std::optional<torch::stable::Tensor> v_scale,
std::optional<torch::stable::Tensor> q_fp8_out,
std::optional<torch::stable::Tensor> q_scale);
// Horizontally-fused MiniMax-M3 QK-norm + partial NeoX RoPE (+ optional KV / // Horizontally-fused MiniMax-M3 QK-norm + partial NeoX RoPE (+ optional KV /
// index-cache insert). Dense layer: norm+RoPE only; sparse layer: also packs // index-cache insert). Dense layer: norm+RoPE only; sparse layer: also packs
// the index branch and scatters k/v/index_k into their paged caches. // the index branch and scatters k/v/index_k into their paged caches.
@@ -575,10 +546,6 @@ std::tuple<int64_t, torch::stable::Tensor> allocate_shared_buffer_and_handle(
int64_t open_mem_handle(torch::stable::Tensor& mem_handle); int64_t open_mem_handle(torch::stable::Tensor& mem_handle);
void free_shared_buffer(int64_t buffer); void free_shared_buffer(int64_t buffer);
void muse_spark_mul_tanh_soft_cap(torch::stable::Tensor& out,
torch::stable::Tensor& input,
double multiplier, double soft_cap_temp);
// Activation kernels (shared CUDA/ROCm) // Activation kernels (shared CUDA/ROCm)
void silu_and_mul(torch::stable::Tensor& out, torch::stable::Tensor& input); void silu_and_mul(torch::stable::Tensor& out, torch::stable::Tensor& input);
void silu_and_mul_clamp(torch::stable::Tensor& out, void silu_and_mul_clamp(torch::stable::Tensor& out,
@@ -612,6 +579,10 @@ void situ_and_mul_quant(
int64_t group_size = 0, int64_t group_size = 0,
std::optional<torch::stable::Tensor> num_valid_tokens = std::nullopt, std::optional<torch::stable::Tensor> num_valid_tokens = std::nullopt,
int64_t topk = 1); int64_t topk = 1);
void masked_situ_and_mul(torch::stable::Tensor& out,
torch::stable::Tensor& input,
const torch::stable::Tensor& expert_num_tokens,
double beta = 1.0, double linear_beta = -1.0);
void masked_moe_activation(torch::stable::Tensor& out, void masked_moe_activation(torch::stable::Tensor& out,
torch::stable::Tensor& input, torch::stable::Tensor& input,
const torch::stable::Tensor& valid_token_counts, const torch::stable::Tensor& valid_token_counts,
@@ -135,13 +135,6 @@ hadamard_transform_kernel(b16* a, b16* out, int total_num_chunks) {
#pragma unroll #pragma unroll
for (int64_t k = 0; k < num_chunks; k++) { for (int64_t k = 0; k < num_chunks; k++) {
// The last warp may own fewer chunks than num_chunks: loading the
// rest would read past the end of `a`. The cp.async.wait_group counts
// below already assume only real_num_chunks groups were committed.
if constexpr(enable_mask) {
if (k >= real_num_chunks)
break;
}
size_t shared_ptr = __cvta_generic_to_shared(b_frag_ptr); size_t shared_ptr = __cvta_generic_to_shared(b_frag_ptr);
#if (__CUDA_ARCH__ >= 900) // SM90 #if (__CUDA_ARCH__ >= 900) // SM90
asm volatile( asm volatile(
+4 -31
View File
@@ -501,27 +501,6 @@ STABLE_TORCH_LIBRARY_FRAGMENT(_C, ops) {
"float eps) -> (Tensor, Tensor)"); "float eps) -> (Tensor, Tensor)");
#endif #endif
#ifndef USE_ROCM
ops.def(
"fused_qknorm_rope_cache("
"Tensor! q_out, Tensor! k_out, Tensor! v_out, Tensor qkv, "
"Tensor q_weight, Tensor k_weight, Tensor cos, Tensor sin, "
"Tensor positions, Tensor slot_mapping, Tensor! k_cache, "
"Tensor! v_cache, int head_dim, int num_q_heads, int num_kv_heads, "
"int rotary_pairs, float epsilon, float norm_weight_bias, "
"str kv_cache_dtype=\"auto\", Tensor? k_scale=None, Tensor? "
"v_scale=None) -> ()");
#endif
ops.def(
"fused_muse_spark_qknorm_rope_kv_insert("
"Tensor! qkv, Tensor positions, int num_heads, int num_kv_heads, "
"int head_dim, float eps, Tensor? cos_sin_cache, int rotary_dim, "
"Tensor? slot_mapping, Tensor!? kv_cache, int block_size, "
"str kv_cache_dtype, Tensor? k_scale=None, Tensor? v_scale=None, "
"Tensor!? q_fp8_out=None, Tensor? q_scale=None) "
"-> ()");
// Horizontally-fused MiniMax-M3 QK-norm + partial NeoX RoPE + KV-insert. // Horizontally-fused MiniMax-M3 QK-norm + partial NeoX RoPE + KV-insert.
ops.def( ops.def(
"fused_minimax_m3_qknorm_rope_kv_insert(" "fused_minimax_m3_qknorm_rope_kv_insert("
@@ -619,10 +598,6 @@ STABLE_TORCH_LIBRARY_FRAGMENT(_C, ops) {
"y_q, Tensor! y_s, bool use_ue8m0) -> ()"); "y_q, Tensor! y_s, bool use_ue8m0) -> ()");
ops.def("weak_ref_tensor(Tensor input) -> Tensor"); ops.def("weak_ref_tensor(Tensor input) -> Tensor");
ops.def(
"muse_spark_mul_tanh_soft_cap(Tensor! out, Tensor input, "
"float multiplier, float soft_cap_temp) -> ()");
// Activation function used in SwiGLU. // Activation function used in SwiGLU.
ops.def("silu_and_mul(Tensor! result, Tensor input) -> ()"); ops.def("silu_and_mul(Tensor! result, Tensor input) -> ()");
@@ -665,6 +640,9 @@ STABLE_TORCH_LIBRARY_FRAGMENT(_C, ops) {
"situ_and_mul_quant(Tensor! out, Tensor! scale, Tensor input, " "situ_and_mul_quant(Tensor! out, Tensor! scale, Tensor input, "
"float beta=1.0, float linear_beta=-1.0, int group_size=0, " "float beta=1.0, float linear_beta=-1.0, int group_size=0, "
"Tensor? num_valid_tokens=None, int topk=1) -> ()"); "Tensor? num_valid_tokens=None, int topk=1) -> ()");
ops.def(
"masked_situ_and_mul(Tensor! out, Tensor input, Tensor "
"expert_num_tokens, float beta=1.0, float linear_beta=-1.0) -> ()");
ops.def( ops.def(
"masked_moe_activation(Tensor! out, Tensor input, Tensor " "masked_moe_activation(Tensor! out, Tensor input, Tensor "
"valid_token_counts, str activation, float clamp_limit=0.0, float " "valid_token_counts, str activation, float clamp_limit=0.0, float "
@@ -853,11 +831,7 @@ STABLE_TORCH_LIBRARY_IMPL(_C, CUDA, ops) {
TORCH_BOX(&fused_kimi_k3_mla_decode_q_concat_ds_mla_insert)); TORCH_BOX(&fused_kimi_k3_mla_decode_q_concat_ds_mla_insert));
#ifndef USE_ROCM #ifndef USE_ROCM
ops.impl("minimax_allreduce_rms_qk", TORCH_BOX(&minimax_allreduce_rms_qk)); ops.impl("minimax_allreduce_rms_qk", TORCH_BOX(&minimax_allreduce_rms_qk));
ops.impl("fused_qknorm_rope_cache",
TORCH_BOX(&vllm::fused_qknorm_rope_cache));
#endif #endif
ops.impl("fused_muse_spark_qknorm_rope_kv_insert",
TORCH_BOX(&fused_muse_spark_qknorm_rope_kv_insert));
ops.impl("fused_minimax_m3_qknorm_rope_kv_insert", ops.impl("fused_minimax_m3_qknorm_rope_kv_insert",
TORCH_BOX(&fused_minimax_m3_qknorm_rope_kv_insert)); TORCH_BOX(&fused_minimax_m3_qknorm_rope_kv_insert));
#ifdef VLLM_ENABLE_FUSED_KDA_DECODE #ifdef VLLM_ENABLE_FUSED_KDA_DECODE
@@ -893,8 +867,6 @@ STABLE_TORCH_LIBRARY_IMPL(_C, CUDA, ops) {
TORCH_BOX(&persistent_masked_m_silu_mul_quant)); TORCH_BOX(&persistent_masked_m_silu_mul_quant));
ops.impl("weak_ref_tensor", TORCH_BOX(&weak_ref_tensor)); ops.impl("weak_ref_tensor", TORCH_BOX(&weak_ref_tensor));
ops.impl("silu_and_mul_quant", TORCH_BOX(&silu_and_mul_quant)); ops.impl("silu_and_mul_quant", TORCH_BOX(&silu_and_mul_quant));
ops.impl("muse_spark_mul_tanh_soft_cap",
TORCH_BOX(&muse_spark_mul_tanh_soft_cap));
ops.impl("silu_and_mul", TORCH_BOX(&silu_and_mul)); ops.impl("silu_and_mul", TORCH_BOX(&silu_and_mul));
ops.impl("mul_and_silu", TORCH_BOX(&mul_and_silu)); ops.impl("mul_and_silu", TORCH_BOX(&mul_and_silu));
ops.impl("gelu_and_mul", TORCH_BOX(&gelu_and_mul)); ops.impl("gelu_and_mul", TORCH_BOX(&gelu_and_mul));
@@ -903,6 +875,7 @@ STABLE_TORCH_LIBRARY_IMPL(_C, CUDA, ops) {
ops.impl("swigluoai_and_mul", TORCH_BOX(&swigluoai_and_mul)); ops.impl("swigluoai_and_mul", TORCH_BOX(&swigluoai_and_mul));
ops.impl("situ_and_mul", TORCH_BOX(&situ_and_mul)); ops.impl("situ_and_mul", TORCH_BOX(&situ_and_mul));
ops.impl("situ_and_mul_quant", TORCH_BOX(&situ_and_mul_quant)); ops.impl("situ_and_mul_quant", TORCH_BOX(&situ_and_mul_quant));
ops.impl("masked_situ_and_mul", TORCH_BOX(&masked_situ_and_mul));
ops.impl("masked_moe_activation", TORCH_BOX(&masked_moe_activation)); ops.impl("masked_moe_activation", TORCH_BOX(&masked_moe_activation));
ops.impl("gelu_new", TORCH_BOX(&gelu_new)); ops.impl("gelu_new", TORCH_BOX(&gelu_new));
ops.impl("gelu_fast", TORCH_BOX(&gelu_fast)); ops.impl("gelu_fast", TORCH_BOX(&gelu_fast));
+1 -5
View File
@@ -502,11 +502,7 @@ RUN --mount=type=cache,target=/opt/uv/cache \
> /tmp/nccl-override.txt && \ > /tmp/nccl-override.txt && \
export UV_OVERRIDE=/tmp/nccl-override.txt; \ export UV_OVERRIDE=/tmp/nccl-override.txt; \
fi && \ fi && \
DEEPEP_ARCH_LIST='9.0a 10.0a' && \ export TORCH_CUDA_ARCH_LIST='9.0a 10.0a' && \
if [ "$(printf '%s\n' 12.9 "$CUDA_VERSION" | sort -V | head -n1)" = 12.9 ]; then \
DEEPEP_ARCH_LIST="$DEEPEP_ARCH_LIST 10.3a"; \
fi && \
export TORCH_CUDA_ARCH_LIST="$DEEPEP_ARCH_LIST" && \
/tmp/install_python_libraries.sh \ /tmp/install_python_libraries.sh \
--workspace /tmp/ep_kernels_workspace \ --workspace /tmp/ep_kernels_workspace \
--mode wheel \ --mode wheel \
@@ -3,7 +3,7 @@ ARG REMOTE_VLLM="0"
ARG VLLM_REPO="https://github.com/vllm-project/vllm.git" ARG VLLM_REPO="https://github.com/vllm-project/vllm.git"
ARG VLLM_BRANCH="main" ARG VLLM_BRANCH="main"
ARG COMMON_WORKDIR=/app ARG COMMON_WORKDIR=/app
ARG BASE_IMAGE=vllm/vllm-openai-rocm:base-nightly-rocm72 ARG BASE_IMAGE=rocm/vllm-dev:base
ARG CI_BASE_IMAGE=rocm/vllm-dev:ci_base ARG CI_BASE_IMAGE=rocm/vllm-dev:ci_base
ARG ROCM_TRITON_KERNELS_COMMIT=669b31acc1dd1b3fd93286afd5db67f65d9f7557 ARG ROCM_TRITON_KERNELS_COMMIT=669b31acc1dd1b3fd93286afd5db67f65d9f7557
ARG INSTALL_LMCACHE=false ARG INSTALL_LMCACHE=false
@@ -46,7 +46,7 @@ RUN --mount=type=cache,target=/root/.cache/pip \
# Note: mold is installed but not set as the system default linker because # Note: mold is installed but not set as the system default linker because
# some packages use JIT compilation at runtime with flags mold does not support. # some packages use JIT compilation at runtime with flags mold does not support.
# Build stages opt in via LDFLAGS="-fuse-ld=mold". # Build stages opt in via LDFLAGS="-fuse-ld=mold".
# Remove sccache only if not using sccache (it exists in base image from Dockerfile.rocm_72_base) # Remove sccache only if not using sccache (it exists in base image from Dockerfile.rocm_base)
ARG USE_SCCACHE ARG USE_SCCACHE
RUN if [ "$USE_SCCACHE" != "1" ]; then \ RUN if [ "$USE_SCCACHE" != "1" ]; then \
apt-get purge -y sccache || true; \ apt-get purge -y sccache || true; \
@@ -280,8 +280,9 @@ COPY --from=fetch_vllm \
${COMMON_WORKDIR}/vllm/requirements/rocm.txt \ ${COMMON_WORKDIR}/vllm/requirements/rocm.txt \
requirements/ requirements/
# Use constraints file to protect triton, pybind11, and torch dependencies.
RUN --mount=type=cache,id=vllm-rocm-uv,target=/root/.cache/uv \ RUN --mount=type=cache,id=vllm-rocm-uv,target=/root/.cache/uv \
uv pip install --system -r requirements/rocm.txt uv pip install --system -c /etc/rocm-constraints.txt -r requirements/rocm.txt
# Fetch only the pinned ROCm Triton kernels package. Keeping this before the # Fetch only the pinned ROCm Triton kernels package. Keeping this before the
# vLLM source COPYs makes the download reusable when native sources change. # vLLM source COPYs makes the download reusable when native sources change.
@@ -322,8 +323,9 @@ ENV SETUPTOOLS_SCM_PRETEND_VERSION="0.0.0+rocm.csrc.build"
RUN --mount=type=bind,source=pyproject.toml,target=${COMMON_WORKDIR}/vllm/pyproject.toml \ RUN --mount=type=bind,source=pyproject.toml,target=${COMMON_WORKDIR}/vllm/pyproject.toml \
--mount=type=cache,id=vllm-rocm-ccache,target=/root/.cache/ccache \ --mount=type=cache,id=vllm-rocm-ccache,target=/root/.cache/ccache \
grep -Fq "set(TRITON_KERNELS_TAG \"${ROCM_TRITON_KERNELS_COMMIT}\")" \ { grep -Fq "set(TRITON_KERNELS_TAG \"${ROCM_TRITON_KERNELS_COMMIT}\")" \
cmake/external_projects/triton_kernels.cmake \ cmake/external_projects/triton_kernels.cmake \
|| echo "WARNING: Dockerfile.rock ROCM_TRITON_KERNELS_COMMIT (${ROCM_TRITON_KERNELS_COMMIT}) does not match cmake TRITON_KERNELS_TAG; the Docker build uses the Dockerfile pin, cmake governs only non-Docker builds"; } \
&& export CCACHE_BASEDIR="$PWD" \ && export CCACHE_BASEDIR="$PWD" \
&& echo "=== ccache stats before ROCm native build ===" \ && echo "=== ccache stats before ROCm native build ===" \
&& (ccache --show-stats || true) \ && (ccache --show-stats || true) \
@@ -366,11 +368,12 @@ COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/requirements /requirements
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/benchmarks /benchmarks COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/benchmarks /benchmarks
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/tests /tests COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/tests /tests
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/examples /examples COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/examples /examples
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/tools /tools COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/tools/benchmark_helion_kernels.py /tools/benchmark_helion_kernels.py
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/tools/install_torchcodec_rocm.sh /tools/install_torchcodec_rocm.sh
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/docker/Dockerfile /docker/Dockerfile COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/docker/Dockerfile /docker/Dockerfile
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/docker/Dockerfile.cpu /docker/Dockerfile.cpu COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/docker/Dockerfile.cpu /docker/Dockerfile.cpu
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/docker/Dockerfile.rocm_72 /docker/ COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/docker/Dockerfile.rocm /docker/
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/docker/Dockerfile.rocm_72_base /docker/ COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/docker/Dockerfile.rocm_base /docker/Dockerfile.rocm_base
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/docker/ci-rocm.hcl /docker/ci-rocm.hcl COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/docker/ci-rocm.hcl /docker/ci-rocm.hcl
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/docker/docker-bake.hcl /docker/docker-bake.hcl COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/docker/docker-bake.hcl /docker/docker-bake.hcl
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/docker/docker-bake-rocm.hcl /docker/docker-bake-rocm.hcl COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/docker/docker-bake-rocm.hcl /docker/docker-bake-rocm.hcl
@@ -386,7 +389,6 @@ ARG NIXL_BRANCH="1683cf3b7f3d11674c03c5e861cea22339876c96"
ARG NIXL_REPO="https://github.com/ai-dynamo/nixl.git" ARG NIXL_REPO="https://github.com/ai-dynamo/nixl.git"
ARG UCX_BRANCH="96e58a16039f6d7d213bc967b8069238742c5194" ARG UCX_BRANCH="96e58a16039f6d7d213bc967b8069238742c5194"
ARG UCX_REPO="https://github.com/openucx/ucx.git" ARG UCX_REPO="https://github.com/openucx/ucx.git"
ENV ROCM_PATH=/opt/rocm
ENV UCX_HOME=/usr/local/ucx ENV UCX_HOME=/usr/local/ucx
ENV NIXL_HOME=/usr/local/nixl ENV NIXL_HOME=/usr/local/nixl
@@ -541,7 +543,7 @@ RUN --mount=type=tmpfs,target=/tmp/lmcache-wheel \
c_ops="$(find /tmp/lmcache-wheel -type f -name 'c_ops*.so')"; \ c_ops="$(find /tmp/lmcache-wheel -type f -name 'c_ops*.so')"; \
test "$(find /tmp/lmcache-wheel -type f -name 'c_ops*.so' | wc -l)" -eq 1; \ test "$(find /tmp/lmcache-wheel -type f -name 'c_ops*.so' | wc -l)" -eq 1; \
expected="$(printf '%s' "${LMCACHE_ROCM_ARCH}" | tr ';' '\n' | sort -u)"; \ expected="$(printf '%s' "${LMCACHE_ROCM_ARCH}" | tr ';' '\n' | sort -u)"; \
actual="$(/opt/rocm/llvm/bin/llvm-objdump --offloading "${c_ops}" \ actual="$(llvm-objdump --offloading "${c_ops}" \
| grep -oE 'gfx[0-9a-z]+' | sort -u)"; \ | grep -oE 'gfx[0-9a-z]+' | sort -u)"; \
if [ "${actual}" != "${expected}" ]; then \ if [ "${actual}" != "${expected}" ]; then \
printf 'LMCache code objects mismatch\nexpected:\n%s\nactual:\n%s\n' \ printf 'LMCache code objects mismatch\nexpected:\n%s\nactual:\n%s\n' \
@@ -549,32 +551,10 @@ RUN --mount=type=tmpfs,target=/tmp/lmcache-wheel \
exit 1; \ exit 1; \
fi fi
# ROCShmem build stage - split from DeepEP so changing DEEPEP_BRANCH does not # DeepEP build stage. ROCShmem is provided by TheRock SDK
# invalidate the slow ROCShmem build. # ROCSHMEM_DIR set to the SDK install path so DeepEP finds the headers, etc.
FROM base AS build_rocshmem FROM base AS build_deepep
ARG ROCSHMEM_BRANCH="f0acb0c6" ENV ROCSHMEM_DIR=${ROCM_PATH}
ARG ROCSHMEM_REPO="https://github.com/ROCm/rocm-systems.git"
# DeepEP only supports gfx942 and gfx950; build ROCShmem for the same set so
# it can be linked against DeepEP without arch mismatches.
ARG DEEPEP_ROCM_ARCH="gfx942;gfx950"
ENV ROCM_PATH=/opt/rocm
ENV ROCSHMEM_DIR=/opt/rocshmem
RUN --mount=type=cache,target=/root/.cache/ccache \
git clone --no-checkout --filter=blob:none ${ROCSHMEM_REPO} \
&& cd rocm-systems \
&& git sparse-checkout set --cone projects/rocshmem \
&& git checkout ${ROCSHMEM_BRANCH} \
&& mkdir -p projects/rocshmem/build \
&& cd projects/rocshmem/build \
&& CC="ccache gcc" CXX="ccache g++" INSTALL_PREFIX=${ROCSHMEM_DIR} \
bash ../scripts/build_configs/all_backends \
-DROCM_PATH=${ROCM_PATH} \
-DGPU_TARGETS="${DEEPEP_ROCM_ARCH}" \
-DUSE_EXTERNAL_MPI=OFF
# DeepEP build stage - depends on ROCShmem, builds the HIP kernel wheel.
FROM build_rocshmem AS build_deepep
ARG DEEPEP_BRANCH="0f63d3e9" ARG DEEPEP_BRANCH="0f63d3e9"
ARG DEEPEP_REPO="https://github.com/ROCm/DeepEP.git" ARG DEEPEP_REPO="https://github.com/ROCm/DeepEP.git"
ARG DEEPEP_NIC="cx7" ARG DEEPEP_NIC="cx7"
@@ -591,8 +571,8 @@ RUN --mount=type=cache,target=/root/.cache/ccache \
&& git checkout ${DEEPEP_BRANCH} \ && git checkout ${DEEPEP_BRANCH} \
&& LDFLAGS="-fuse-ld=mold" MAX_JOBS="${MAX_JOBS:-$(nproc)}" python3 setup.py --variant rocm --rocm-explicit-ctx --nic ${DEEPEP_NIC} --rocm-gfx942-fp8-fnuz-max 224 bdist_wheel --dist-dir=/app/deep_install && LDFLAGS="-fuse-ld=mold" MAX_JOBS="${MAX_JOBS:-$(nproc)}" python3 setup.py --variant rocm --rocm-explicit-ctx --nic ${DEEPEP_NIC} --rocm-gfx942-fp8-fnuz-max 224 bdist_wheel --dist-dir=/app/deep_install
# MoRI runtime dependencies live in Dockerfile.rocm_72 so NIC backend changes do # MoRI runtime dependencies live in Dockerfile.rocm so NIC backend changes do
# not force users to rebuild the long-lived Dockerfile.rocm_72_base image. # not force users to rebuild the long-lived Dockerfile.rocm_base image.
FROM base AS mori_base FROM base AS mori_base
ARG NIC_BACKEND ARG NIC_BACKEND
ARG AINIC_VERSION ARG AINIC_VERSION
@@ -693,8 +673,9 @@ RUN if [ "$GIT_REPO_CHECK" != "0" ]; then \
# Install the packaging helpers before modifying requirements/rocm.txt. # Install the packaging helpers before modifying requirements/rocm.txt.
RUN --mount=type=cache,target=/root/.cache/uv \ RUN --mount=type=cache,target=/root/.cache/uv \
grep -Fq "set(TRITON_KERNELS_TAG \"${ROCM_TRITON_KERNELS_COMMIT}\")" \ { grep -Fq "set(TRITON_KERNELS_TAG \"${ROCM_TRITON_KERNELS_COMMIT}\")" \
vllm/cmake/external_projects/triton_kernels.cmake \ vllm/cmake/external_projects/triton_kernels.cmake \
|| echo "WARNING: Dockerfile.rock ROCM_TRITON_KERNELS_COMMIT (${ROCM_TRITON_KERNELS_COMMIT}) does not match cmake TRITON_KERNELS_TAG; the Docker build uses the Dockerfile pin, cmake governs only non-Docker builds"; } \
&& cd vllm \ && cd vllm \
&& uv pip install --system setuptools_scm regex && uv pip install --system setuptools_scm regex
@@ -750,11 +731,12 @@ COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/requirements /requir
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/benchmarks /benchmarks COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/benchmarks /benchmarks
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/tests /tests COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/tests /tests
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/examples /examples COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/examples /examples
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/tools /tools COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/tools/benchmark_helion_kernels.py /tools/benchmark_helion_kernels.py
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/tools/install_torchcodec_rocm.sh /tools/install_torchcodec_rocm.sh
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/docker/Dockerfile /docker/Dockerfile COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/docker/Dockerfile /docker/Dockerfile
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/docker/Dockerfile.cpu /docker/Dockerfile.cpu COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/docker/Dockerfile.cpu /docker/Dockerfile.cpu
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/docker/Dockerfile.rocm_72 /docker/ COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/docker/Dockerfile.rocm /docker/
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/docker/Dockerfile.rocm_72_base /docker/ COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/docker/Dockerfile.rocm_base /docker/Dockerfile.rocm_base
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/docker/ci-rocm.hcl /docker/ci-rocm.hcl COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/docker/ci-rocm.hcl /docker/ci-rocm.hcl
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/docker/docker-bake.hcl /docker/docker-bake.hcl COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/docker/docker-bake.hcl /docker/docker-bake.hcl
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/docker/docker-bake-rocm.hcl /docker/docker-bake-rocm.hcl COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/docker/docker-bake-rocm.hcl /docker/docker-bake-rocm.hcl
@@ -795,9 +777,6 @@ RUN --mount=type=bind,from=build_nixl,src=/app/install,target=/nixl_install \
--mount=type=bind,from=build_deepep,src=/app/deep_install,target=/deep_install \ --mount=type=bind,from=build_deepep,src=/app/deep_install,target=/deep_install \
uv pip install --system /nixl_install/*.whl /deep_install/*.whl uv pip install --system /nixl_install/*.whl /deep_install/*.whl
# Copy ROCShmem runtime libraries.
COPY --from=build_rocshmem /opt/rocshmem /opt/rocshmem
# RDMA userspace libraries plus FFmpeg dev libs needed by torchcodec. # RDMA userspace libraries plus FFmpeg dev libs needed by torchcodec.
RUN timeout --kill-after=30s 5m apt-get update -q -y --error-on=any \ RUN timeout --kill-after=30s 5m apt-get update -q -y --error-on=any \
-o Acquire::Retries=3 \ -o Acquire::Retries=3 \
@@ -830,7 +809,8 @@ RUN --mount=type=cache,target=/root/.cache/uv \
# Pre-install shared ROCm runtime dependencies. # Pre-install shared ROCm runtime dependencies.
COPY requirements/common.txt requirements/rocm.txt /tmp/ci-base-requirements/ COPY requirements/common.txt requirements/rocm.txt /tmp/ci-base-requirements/
RUN --mount=type=cache,target=/root/.cache/uv \ RUN --mount=type=cache,target=/root/.cache/uv \
uv pip install --system -r /tmp/ci-base-requirements/rocm.txt \ uv pip install --system -c /etc/rocm-constraints.txt \
-r /tmp/ci-base-requirements/rocm.txt \
&& rm -rf /tmp/ci-base-requirements && rm -rf /tmp/ci-base-requirements
# Enable fast and less brittle model downloads in tests. # Enable fast and less brittle model downloads in tests.
@@ -908,8 +888,6 @@ RUN mkdir src && mv vllm src/vllm
# Catch GPU<->CPU syncs in execute_model/sample_tokens during tests. # Catch GPU<->CPU syncs in execute_model/sample_tokens during tests.
ENV VLLM_GPU_SYNC_CHECK=error ENV VLLM_GPU_SYNC_CHECK=error
# Run the structural smoke test inside BuildKit, where the test image layers
# are already available. Export only a build-scoped success marker.
FROM test AS test_smoke FROM test AS test_smoke
ARG ROCM_SMOKE_ID ARG ROCM_SMOKE_ID
RUN --network=none \ RUN --network=none \
@@ -956,15 +934,37 @@ RUN --mount=type=bind,from=export_vllm,src=/,target=/install \
--mount=type=cache,target=/root/.cache/uv \ --mount=type=cache,target=/root/.cache/uv \
cd /install \ cd /install \
&& echo "opentelemetry-api<=1.40.0" > /tmp/otel-constraints.txt \ && echo "opentelemetry-api<=1.40.0" > /tmp/otel-constraints.txt \
&& uv pip install --system -c /tmp/otel-constraints.txt -r requirements/rocm.txt \ && uv pip install --system -c /etc/rocm-constraints.txt -c /tmp/otel-constraints.txt -r requirements/rocm.txt \
&& pip uninstall -y vllm \ && pip uninstall -y vllm \
&& uv pip install --system -c /tmp/otel-constraints.txt *.whl \ && uv pip install --system -c /etc/rocm-constraints.txt -c /tmp/otel-constraints.txt *.whl \
&& rm /tmp/otel-constraints.txt && rm /tmp/otel-constraints.txt
# Install NIXL ROCm wheel # Install NIXL ROCm wheel
RUN --mount=type=bind,from=build_nixl,src=/app/install,target=/nixl_install \ RUN --mount=type=bind,from=build_nixl,src=/app/install,target=/nixl_install \
uv pip install --system /nixl_install/*.whl uv pip install --system /nixl_install/*.whl
# Recompute the base's triton fingerprint to detect resolver interference.
RUN EXPECTED=$(grep '^TRITON_SHA256: ' /app/versions.txt | cut -d' ' -f2) \
&& ACTUAL=$(find $(python3 -c 'import triton, os; print(os.path.dirname(triton.__file__))') \
-type f \( -name '*.py' -o -name '*.so' \) | sort | xargs sha256sum | sha256sum | cut -d' ' -f1) \
&& echo "triton sha256 expected=${EXPECTED} actual=${ACTUAL}" \
&& test -n "${EXPECTED}" \
&& test "${EXPECTED}" = "${ACTUAL}" \
&& python3 -c "import torch; assert torch.version.hip, f'non-ROCm torch: {torch.__version__}'; print(f'torch={torch.__version__}')"
# Recompute the base's pybind11 fingerprint to detect resolver interference.
RUN EXPECTED=$(grep '^PYBIND11_SHA256: ' /app/versions.txt | cut -d' ' -f2) \
&& EXPECTED_ABI=$(grep '^PYBIND11_INTERNALS_VERSION: ' /app/versions.txt | cut -d' ' -f2) \
&& INC=$(python3 -c 'import pybind11; print(pybind11.get_include())') \
&& ACTUAL=$(find "${INC}" -type f -name '*.h' | sort | xargs sha256sum | sha256sum | cut -d' ' -f1) \
&& ACTUAL_ABI=$(grep -rhE '^#[[:space:]]*define PYBIND11_INTERNALS_VERSION' "${INC}" | awk '{print $NF}') \
&& echo "pybind11 $(pip show pybind11 | awk '/^Version:/ {print $2}')" \
"sha256 expected=${EXPECTED} actual=${ACTUAL}" \
"internals expected=${EXPECTED_ABI} actual=${ACTUAL_ABI}" \
&& test -n "${EXPECTED}" \
&& test "${EXPECTED}" = "${ACTUAL}" \
&& test "${EXPECTED_ABI}" = "${ACTUAL_ABI}"
ARG COMMON_WORKDIR ARG COMMON_WORKDIR
ARG BASE_IMAGE ARG BASE_IMAGE
ARG NIC_BACKEND ARG NIC_BACKEND
+453
View File
@@ -0,0 +1,453 @@
ARG BASE_IMAGE=ubuntu:22.04
ARG PYTHON_VERSION=3.12
ARG TORCH_VERSION=2.12.0+rocm10.0.0
ARG TORCHVISION_VERSION=0.27.0+rocm10.0.0
ARG TORCHAUDIO_VERSION=2.11.0+rocm10.0.0
ARG TRITON_BRANCH="669b31ac" # release/internal/3.8.x as of 09/29/2026
ARG TRITON_VERSION=3.8.0+git669b31ac
ARG TRITON_REPO="https://github.com/ROCm/triton.git"
ARG FA_BRANCH="0e60e394"
ARG FA_REPO="https://github.com/Dao-AILab/flash-attention.git"
ARG AITER_BRANCH="v0.1.25"
ARG AITER_REPO="https://github.com/ROCm/aiter.git"
ARG MORI_BRANCH="v1.2.3.post1"
ARG MORI_REPO="https://github.com/ROCm/mori.git"
ARG ROCM_SYSTEMS_REPO="https://github.com/ROCm/rocm-systems.git"
ARG ROCM_RUNTIME_COMMIT="25e14349f2606a6b95c4ccd45eb92a1835646cf2"
# TheRock configuration
ARG SITE_PACKAGES=/usr/local/lib/python${PYTHON_VERSION}/dist-packages
ARG ROCM_RELEASE_WHEELS_MULTIARCH_URL="https://stable.repo.amd.com/rocm/whl-next/" # For pre-built GPU images, packages, etc.
ARG ROCM_NIGHTLY_WHEELS_MULTIARCH_URL="https://nightly.repo.amd.com/rocm/whl-next/" # For torch, triton, etc. wheels with latest fixes.
ARG ROCM_SDK_VERSION=10.0.0
# Build configuration
ARG PYTORCH_ROCM_ARCH="gfx90a;gfx942;gfx950;gfx1100;gfx1101;gfx1200;gfx1201;gfx1150;gfx1151"
ARG AITER_ROCM_ARCH="gfx942;gfx950"
# Sccache configuration (only used in release pipeline)
ARG USE_SCCACHE
ARG SCCACHE_DOWNLOAD_URL
ARG SCCACHE_ENDPOINT
ARG SCCACHE_BUCKET_NAME=vllm-build-sccache
ARG SCCACHE_REGION_NAME=us-west-2
ARG SCCACHE_S3_NO_CREDENTIALS=0
FROM ${BASE_IMAGE} AS base
ARG PYTORCH_ROCM_ARCH
ENV PYTORCH_ROCM_ARCH="${PYTORCH_ROCM_ARCH}"
ARG AITER_ROCM_ARCH
ENV AITER_ROCM_ARCH="${AITER_ROCM_ARCH}"
ARG SITE_PACKAGES
ENV SITE_PACKAGES=${SITE_PACKAGES}
ENV SDK_DEV="$SITE_PACKAGES/_rocm_sdk_devel" \
SDK_CORE="$SITE_PACKAGES/_rocm_sdk_core"
ENV ROCM_PATH="$SDK_DEV" \
ROCM_HOME="$SDK_DEV" \
HIP_PATH="$SDK_DEV" \
HIP_CLANG_PATH="$SDK_DEV/lib/llvm/bin" \
HIP_DEVICE_LIB_PATH="$SDK_CORE/lib/llvm/amdgcn/bitcode"
ENV LD_LIBRARY_PATH="$SDK_DEV/lib:$SDK_DEV/lib/rocm_sysdeps/lib:$SDK_CORE/lib:$SDK_CORE/lib/rocm_sysdeps/lib"
ENV PATH="$SDK_DEV/bin:$SDK_DEV/lib/llvm/bin:$SDK_CORE/bin:$PATH"
# Required for RCCL in ROCm7.1
ENV HSA_NO_SCRATCH_RECLAIM=1
ARG PYTHON_VERSION
ENV PYTHON_VERSION=${PYTHON_VERSION}
RUN mkdir -p /app
WORKDIR /app
ENV DEBIAN_FRONTEND=noninteractive
# Install Python and other dependencies
RUN apt-get update -y \
&& apt-get install -y software-properties-common build-essential git curl sudo vim less libgfortran5 libopenmpi-dev libpci-dev liblzma-dev pkg-config libdrm-dev libnuma-dev kmod \
&& for i in 1 2 3; do \
add-apt-repository -y ppa:deadsnakes/ppa && break || \
{ echo "Attempt $i failed, retrying in 5s..."; sleep 5; }; \
done \
&& apt-get update -y \
&& apt-get install -y python${PYTHON_VERSION} python${PYTHON_VERSION}-dev python${PYTHON_VERSION}-venv \
python${PYTHON_VERSION}-lib2to3 python-is-python3 \
&& update-alternatives --install /usr/bin/python3 python3 /usr/bin/python${PYTHON_VERSION} 1 \
&& update-alternatives --set python3 /usr/bin/python${PYTHON_VERSION} \
&& ln -sf /usr/bin/python${PYTHON_VERSION}-config /usr/bin/python3-config \
&& curl -sS https://bootstrap.pypa.io/get-pip.py | python${PYTHON_VERSION} \
&& python3 --version && python3 -m pip --version
RUN pip install -U packaging 'cmake<4' ninja wheel 'setuptools<80' pybind11==3.0.4 Cython
RUN apt-get update && apt-get install -y libjpeg-dev libsox-dev libsox-fmt-all sox && rm -rf /var/lib/apt/lists/*
# Install ROCm SDK via TheRock
ARG ROCM_RELEASE_WHEELS_MULTIARCH_URL
ARG ROCM_NIGHTLY_WHEELS_MULTIARCH_URL
ARG ROCM_SDK_VERSION
RUN pip install --index-url ${ROCM_RELEASE_WHEELS_MULTIARCH_URL} \
--extra-index-url ${ROCM_NIGHTLY_WHEELS_MULTIARCH_URL} \
"rocm[libraries,devel]${ROCM_SDK_VERSION:+==${ROCM_SDK_VERSION}}" \
&& rocm-sdk init
# Install PyTorch wheels (nightly for latest torch, release as fallback)
ARG TORCH_VERSION
ARG TORCHAUDIO_VERSION
ARG TORCHVISION_VERSION
RUN pip install \
--index-url ${ROCM_RELEASE_WHEELS_MULTIARCH_URL} \
--extra-index-url ${ROCM_NIGHTLY_WHEELS_MULTIARCH_URL} \
"torch==${TORCH_VERSION}" \
"torchaudio==${TORCHAUDIO_VERSION}" \
"torchvision==${TORCHVISION_VERSION}"
# Install the AMD-compiled device kernels corresponding to PYTORCH_ROCM_ARCH:
# Possible values that can be in PYTORCH_ROCM_ARCH:
#
# gfx90a (MI210/250), gfx942 (MI300), gfx950 (MI350), gfx1100 (7900XTX),
# gfx1101 (7800XT), gfx1200 (9070XT), gfx1201 (9070), gfx1150/1151 (Strix APU)
# For each arch in PYTORCH_ROCM_ARCH, install the precompiled GPU code that
# PyTorch needs to run on that specific chip.
SHELL ["/bin/bash", "-c"]
RUN TORCH_VER=$(python3 -c "import torch; print(torch.__version__)") \
&& for arch in $(echo "${PYTORCH_ROCM_ARCH}" | tr ';' ' '); do \
echo "Installing device packages for ${arch}" \
&& pip install --index-url ${ROCM_RELEASE_WHEELS_MULTIARCH_URL} \
--extra-index-url ${ROCM_NIGHTLY_WHEELS_MULTIARCH_URL} \
amd-torch-device-${arch}==${TORCH_VER} \
rocm-sdk-device-${arch}==${ROCM_SDK_VERSION}; \
done
# Remove gfx90a xnack kpacks (ASAN artifacts that cause hipErrorInvalidImage on MI250)
RUN rm -f "${SITE_PACKAGES}/torch/.kpack/torch_gfx90a:xnack+.kpack" \
"${SITE_PACKAGES}/torch/.kpack/torch_gfx90a:xnack-.kpack"
RUN cd $SITE_PACKAGES/_rocm_sdk_core/share/amd_smi && pip install .
ENV PYTHONPATH=$SITE_PACKAGES/_rocm_sdk_core/share/amd_smi
# TheRock runtime fixes
RUN if [ -f "${SITE_PACKAGES}/rocm_sdk/__init__.py" ]; then \
sed -i 's/rtld_global: bool = True/rtld_global: bool = False/g' "${SITE_PACKAGES}/rocm_sdk/__init__.py"; \
fi \
&& if [ -f "${SITE_PACKAGES}/_rocm_sdk_core/lib/librocm_smi64.so.1" ] \
&& [ ! -e "${ROCM_PATH}/lib/librocm_smi64.so.7" ]; then \
ln -sfn "${SITE_PACKAGES}/_rocm_sdk_core/lib/librocm_smi64.so.1" "${ROCM_PATH}/lib/librocm_smi64.so.7"; \
fi \
&& printf '%s\n%s\n' "${ROCM_PATH}/lib" "${SITE_PACKAGES}/_rocm_sdk_core/lib" > /etc/ld.so.conf.d/rocm-pip.conf \
&& for f in ${SITE_PACKAGES}/_rocm_sdk_core/lib/libamdhip64.so*; do \
bn=$(basename "$f"); \
if [ -f "${ROCM_PATH}/lib/${bn}" ] && [ -f "$f" ] && [ ! -L "$f" ]; then \
rm -f "$f"; \
ln -sfn "${ROCM_PATH}/lib/${bn}" "$f"; \
fi; \
done \
&& for f in librocprofiler-sdk.so.1 librocprofiler-register.so.0 \
librocprofiler-sdk-roctx.so.1; do \
if [ -e "${SITE_PACKAGES}/_rocm_sdk_core/lib/${f}" ] \
&& [ -e "${SITE_PACKAGES}/_rocm_sdk_devel/lib/${f}" ]; then \
rm -f "${SITE_PACKAGES}/_rocm_sdk_core/lib/${f}"; \
ln -sfn "${SITE_PACKAGES}/_rocm_sdk_devel/lib/${f}" \
"${SITE_PACKAGES}/_rocm_sdk_core/lib/${f}"; \
fi; \
done \
&& ldconfig
# Install sccache if USE_SCCACHE is enabled (for release builds)
ARG USE_SCCACHE
ARG SCCACHE_DOWNLOAD_URL
ARG SCCACHE_ENDPOINT
ARG SCCACHE_BUCKET_NAME
ARG SCCACHE_REGION_NAME
ARG SCCACHE_S3_NO_CREDENTIALS
RUN if [ "$USE_SCCACHE" = "1" ]; then \
echo "Installing sccache..." \
&& SCCACHE_ARCH="x86_64" \
&& SCCACHE_VERSION="v0.8.1" \
&& SCCACHE_DL_URL="${SCCACHE_DOWNLOAD_URL:-https://github.com/mozilla/sccache/releases/download/${SCCACHE_VERSION}/sccache-${SCCACHE_VERSION}-${SCCACHE_ARCH}-unknown-linux-musl.tar.gz}" \
&& curl -L -o /tmp/sccache.tar.gz ${SCCACHE_DL_URL} \
&& tar -xzf /tmp/sccache.tar.gz -C /tmp \
&& mv /tmp/sccache-${SCCACHE_VERSION}-${SCCACHE_ARCH}-unknown-linux-musl/sccache /usr/bin/sccache \
&& chmod +x /usr/bin/sccache \
&& rm -rf /tmp/sccache.tar.gz /tmp/sccache-${SCCACHE_VERSION}-${SCCACHE_ARCH}-unknown-linux-musl \
&& sccache --version; \
fi
# Setup sccache for HIP compilation via HIP_CLANG_PATH
# This creates wrapper scripts in a separate directory and points HIP to use them
# This avoids modifying the original ROCm binaries which can break detection
# NOTE: HIP_CLANG_PATH is NOT set as ENV to avoid affecting downstream images (Dockerfile.rocm)
# Instead, each build stage should export HIP_CLANG_PATH=/opt/sccache-wrappers if USE_SCCACHE=1
RUN if [ "$USE_SCCACHE" = "1" ]; then \
echo "Setting up sccache wrappers for HIP compilation..." \
&& mkdir -p /opt/sccache-wrappers \
&& printf '#!/bin/bash\nexec sccache %s/lib/llvm/bin/clang++ "$@"\n' "${ROCM_PATH}" > /opt/sccache-wrappers/clang++ \
&& chmod +x /opt/sccache-wrappers/clang++ \
&& printf '#!/bin/bash\nexec sccache %s/lib/llvm/bin/clang "$@"\n' "${ROCM_PATH}" > /opt/sccache-wrappers/clang \
&& chmod +x /opt/sccache-wrappers/clang \
&& echo "sccache wrappers created in /opt/sccache-wrappers"; \
fi
ENV SCCACHE_BUCKET=${USE_SCCACHE:+${SCCACHE_BUCKET_NAME}}
ENV SCCACHE_REGION=${USE_SCCACHE:+${SCCACHE_REGION_NAME}}
ENV SCCACHE_S3_NO_CREDENTIALS=${USE_SCCACHE:+${SCCACHE_S3_NO_CREDENTIALS}}
ENV SCCACHE_IDLE_TIMEOUT=${USE_SCCACHE:+0}
###
### MORI Build
###
FROM base AS build_mori
ARG MORI_BRANCH
ARG MORI_REPO
RUN git clone ${MORI_REPO}
RUN cd mori \
&& git checkout ${MORI_BRANCH} \
&& git submodule update --init --recursive \
&& ln -sfn ${ROCM_PATH} /opt/rocm \
&& pip install "setuptools_scm>=6.2" \
&& CMAKE_PREFIX_PATH="${ROCM_PATH}:${ROCM_PATH}/lib/rocm_sysdeps" \
python3 setup.py bdist_wheel --dist-dir=dist && ls /app/mori/dist/*.whl
RUN mkdir -p /app/install && cp /app/mori/dist/*.whl /app/install
###
### FlashAttention Build
###
FROM base AS build_fa
ARG FA_BRANCH
ARG FA_REPO
ARG USE_SCCACHE
RUN git clone ${FA_REPO}
RUN cd flash-attention \
&& git checkout ${FA_BRANCH} \
&& git submodule update --init \
&& if [ "$USE_SCCACHE" = "1" ]; then \
export HIP_CLANG_PATH=/opt/sccache-wrappers \
&& sccache --show-stats; \
fi \
&& GPU_ARCHS=$(echo ${PYTORCH_ROCM_ARCH} | sed -e 's/;gfx1[0-9]\{3\}//g') python3 setup.py bdist_wheel --dist-dir=dist \
&& if [ "$USE_SCCACHE" = "1" ]; then sccache --show-stats; fi
RUN mkdir -p /app/install && cp /app/flash-attention/dist/*.whl /app/install
###
### AITER Build
###
FROM base AS build_aiter
ARG AITER_BRANCH
ARG AITER_REPO
ARG USE_SCCACHE
RUN git clone --recursive --branch ${AITER_BRANCH} ${AITER_REPO}
RUN cd aiter \
&& git submodule update --init --recursive \
&& pip install -r requirements.txt
# AITER_USE_SYSTEM_TRITON keeps setup.py from resolving its own triton, which would
# displace the pinned one already installed in base.
RUN pip install pyyaml && cd aiter \
&& if [ "$USE_SCCACHE" = "1" ]; then \
export HIP_CLANG_PATH=/opt/sccache-wrappers \
&& sccache --show-stats; \
fi \
&& PREBUILD_KERNELS=1 AITER_USE_SYSTEM_TRITON=1 GPU_ARCHS=${AITER_ROCM_ARCH} python3 setup.py bdist_wheel --dist-dir=dist \
&& if [ "$USE_SCCACHE" = "1" ]; then sccache --show-stats; fi \
&& ls /app/aiter/dist/*.whl
RUN mkdir -p /app/install && cp /app/aiter/dist/*.whl /app/install
###
### Triton Build
###
FROM base AS build_triton
ARG TRITON_BRANCH
ARG TRITON_REPO
RUN git clone ${TRITON_REPO}
RUN cd triton \
&& git checkout ${TRITON_BRANCH} \
&& if [ ! -f setup.py ]; then cd python; fi \
&& python3 setup.py bdist_wheel --dist-dir=dist \
&& mkdir -p /app/install && cp dist/*.whl /app/install
###
### ROCr + CLR build
###
# Brings in multi-stream perf improvements; drop after these arrive in an official release
FROM base AS build_rocm_runtime
ARG ROCM_SYSTEMS_REPO
ARG ROCM_RUNTIME_COMMIT
# libgl-dev: rocclr does find_package(OpenGL REQUIRED) unconditionally and the wheel image has
# no GL headers. The link must not pick up a libGL DT_NEEDED, which is asserted after the build.
RUN apt-get update \
&& apt-get install -y --no-install-recommends g++ libdw-dev xxd libgl-dev libglx-dev \
&& rm -rf /var/lib/apt/lists/*
RUN pip install --no-cache-dir CppHeaderParser
RUN git init -q /src && cd /src \
&& git remote add origin ${ROCM_SYSTEMS_REPO} \
&& git config core.sparseCheckout true \
&& git config remote.origin.promisor true \
&& git config remote.origin.partialclonefilter blob:none \
&& git sparse-checkout init --cone \
&& git sparse-checkout set projects/rocr-runtime projects/clr projects/hip shared cmake \
&& git fetch --filter=blob:none --depth 1 origin ${ROCM_RUNTIME_COMMIT} \
&& git checkout -q FETCH_HEAD
# libelf is taken from the SDK's own vendored sysdeps rather than apt -- the wheel image has no
# /usr/include/libelf.h -- so ROCr links against the copy the wheel itself was built against.
ENV SYSDEPS="$SDK_DEV/lib/rocm_sysdeps"
# ROCr first, and installed into $SDK_DEV, because the clr build below needs its headers.
RUN cd /src/projects/rocr-runtime \
&& cmake -B build -G Ninja \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_PREFIX_PATH="$SDK_DEV;$SYSDEPS" \
-DCMAKE_INSTALL_PREFIX="$SDK_DEV" \
-DLIBELF_INCLUDE_DIRS="$SYSDEPS/include" \
-DLIBELF_LIBRARIES="$SYSDEPS/lib/libelf.so" \
&& cmake --build build --parallel 16 \
&& cmake --install build \
&& cmake --install build --prefix /rocr-install --strip
RUN cd /src/projects/clr \
&& cmake -B build -G Ninja \
-DCMAKE_BUILD_TYPE=Release \
-DCLR_BUILD_HIP=ON \
-DCLR_BUILD_OCL=OFF \
-DROCM_KPACK_ENABLED=ON \
-DHIP_COMMON_DIR=/src/projects/hip \
-DROCM_PATH="$SDK_DEV" \
-DCMAKE_PREFIX_PATH="$SDK_DEV;$SYSDEPS" \
-DCMAKE_INSTALL_PREFIX="$SDK_DEV" \
-DHIP_LLVM_ROOT="$SDK_DEV/lib/llvm" \
&& cmake --build build --parallel 16 \
&& cmake --install build --prefix /clr-install --strip
RUN mkdir -p /staging/lib \
&& cp -P /rocr-install/lib/libhsa-runtime64.so* /staging/lib/ \
&& cp -P /clr-install/lib/libamdhip64.so* /staging/lib/ \
&& ls -l /staging/lib
###
### Final Build
###
# Wheel release stage -
# only includes dependencies used by wheel release pipeline
FROM base AS debs_wheel_release
RUN mkdir /app/debs
RUN --mount=type=bind,from=build_triton,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_fa,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_aiter,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
FROM base AS debs
RUN mkdir /app/debs
RUN --mount=type=bind,from=build_triton,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_fa,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_aiter,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_mori,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
FROM base AS final
ARG TRITON_VERSION
# torch 2.12 implicitly attaches rocprofiler-sdk; its queue interposition hangs at HSA
# teardown, so disable it. TODO: drop once ROCm's stock rocprofiler carries the fix.
ENV ROCPROFILER_QUEUE_INTERPOSITION=0
COPY --from=build_rocm_runtime /staging/lib/ ${SDK_DEV}/lib/
# Hard link the patched library onto every stock file name, in both SDK trees, so that the
# SONAME lookup and the hard coded paths all resolve to it. Hard links rather than symlinks
# because ldconfig rewrites SONAME symlinks but cannot re-point an inode. $SDK_CORE keeps its
# own libhsa-runtime64 copy, which is on LD_LIBRARY_PATH and in ld.so.conf; relinking it is
# belt and braces rather than strictly required, but it leaves no stock ROCr in the image.
RUN set -eux; \
cd "$SDK_DEV/lib"; \
ours_hsa="$(readlink -f libhsa-runtime64.so.1)"; \
ours_hip="$(readlink -f libamdhip64.so.7)"; \
for f in "$SDK_DEV"/lib/libhsa-runtime64.so.1* "$SDK_CORE"/lib/libhsa-runtime64.so.1* \
"$SDK_DEV"/lib/libamdhip64.so.7* "$SDK_CORE"/lib/libamdhip64.so.7*; do \
[ -f "$f" ] && [ ! -L "$f" ] || continue; \
case "${f##*/}" in libhsa*) ours="$ours_hsa";; *) ours="$ours_hip";; esac; \
[ "$f" -ef "$ours" ] || ln -f "$ours" "$f"; \
done
RUN ldconfig
# This is to prevent resolvers from checking METADATA and RECORD and clobbering
# the Triton source build we installed. Constraints files are unable to help
# with this. However, these entries are optional and can be blanked which
# solves the problem.
RUN TORCH_DIST_INFO=$(ls -d ${SITE_PACKAGES}/torch-*.dist-info) \
&& grep -q '^Requires-Dist: triton' "$TORCH_DIST_INFO/METADATA" \
&& sed -i '/^Requires-Dist: triton/d' "$TORCH_DIST_INFO/METADATA" \
&& sed -i "s|^\(${TORCH_DIST_INFO##*/}/METADATA\),.*|\1,,|" "$TORCH_DIST_INFO/RECORD"
RUN --mount=type=bind,from=debs,src=/app/debs,target=/install \
pip install /install/*.whl \
&& ! pip check 2>&1 | grep -iE '^(torch|triton) ' \
&& test "$(pip show triton | awk '/^Version:/ {print $2}')" = "${TRITON_VERSION}" \
# Create a constraints file to protect the versions we need, especially
# triton, requirements will try to reinstall repeatedly.
&& pip list --format=freeze \
| grep -E '^(torch|triton|torchaudio|torchvision|pybind11)==' > /etc/rocm-constraints.txt \
&& echo "ROCm constraints:" && cat /etc/rocm-constraints.txt \
&& test "$(wc -l < /etc/rocm-constraints.txt)" -eq 5 \
&& grep -q '^pybind11==3\.0\.' /etc/rocm-constraints.txt \
&& python3 -c "import torch; assert torch.version.hip, f'non-ROCm torch: {torch.__version__}'; print(f'torch={torch.__version__}')" \
&& python3 -c "import triton; print(f'triton={triton.__version__}')" \
&& pip list 2>/dev/null | grep -iE '^(torch|triton) '
ARG BASE_IMAGE
ARG PYTHON_VERSION
ARG ROCM_SDK_VERSION
ARG TORCH_VERSION
ARG TORCHAUDIO_VERSION
ARG TORCHVISION_VERSION
ARG PYTORCH_ROCM_ARCH
ARG AITER_ROCM_ARCH
ARG TRITON_BRANCH
ARG TRITON_REPO
ARG FA_BRANCH
ARG FA_REPO
ARG AITER_BRANCH
ARG AITER_REPO
ARG MORI_BRANCH
ARG MORI_REPO
ARG ROCM_SYSTEMS_REPO
ARG ROCM_RUNTIME_COMMIT
ARG TRITON_VERSION
# Record hashes for Triton and pybind11 so we can verify later.
RUN echo "BASE_IMAGE: ${BASE_IMAGE}" > /app/versions.txt \
&& echo "PYTHON_VERSION: ${PYTHON_VERSION}" >> /app/versions.txt \
&& echo "ROCM_SDK_VERSION: ${ROCM_SDK_VERSION}" >> /app/versions.txt \
&& echo "TORCH_VERSION: ${TORCH_VERSION}" >> /app/versions.txt \
&& echo "TORCHAUDIO_VERSION: ${TORCHAUDIO_VERSION}" >> /app/versions.txt \
&& echo "TORCHVISION_VERSION: ${TORCHVISION_VERSION}" >> /app/versions.txt \
&& echo "PYTORCH_ROCM_ARCH: ${PYTORCH_ROCM_ARCH}" >> /app/versions.txt \
&& echo "AITER_ROCM_ARCH: ${AITER_ROCM_ARCH}" >> /app/versions.txt \
&& echo "TRITON_BRANCH: ${TRITON_BRANCH}" >> /app/versions.txt \
&& echo "TRITON_REPO: ${TRITON_REPO}" >> /app/versions.txt \
&& echo "FA_BRANCH: ${FA_BRANCH}" >> /app/versions.txt \
&& echo "FA_REPO: ${FA_REPO}" >> /app/versions.txt \
&& echo "AITER_BRANCH: ${AITER_BRANCH}" >> /app/versions.txt \
&& echo "AITER_REPO: ${AITER_REPO}" >> /app/versions.txt \
&& echo "MORI_BRANCH: ${MORI_BRANCH}" >> /app/versions.txt \
&& echo "MORI_REPO: ${MORI_REPO}" >> /app/versions.txt \
&& echo "ROCM_SYSTEMS_REPO: ${ROCM_SYSTEMS_REPO}" >> /app/versions.txt \
&& echo "ROCM_RUNTIME_COMMIT: ${ROCM_RUNTIME_COMMIT}" >> /app/versions.txt \
# Fingerprint our triton so Dockerfile.rocm can prove it survived the build.
&& echo "TRITON_VERSION: $(pip show triton | awk '/^Version:/ {print $2}')" >> /app/versions.txt \
&& echo "TRITON_SHA256: $(find $(python3 -c 'import triton, os; print(os.path.dirname(triton.__file__))') \
-type f \( -name '*.py' -o -name '*.so' \) | sort | xargs sha256sum | sha256sum | cut -d' ' -f1)" \
>> /app/versions.txt \
&& echo "PYBIND11_VERSION: $(pip show pybind11 | awk '/^Version:/ {print $2}')" >> /app/versions.txt \
&& echo "PYBIND11_INTERNALS_VERSION: $(grep -rhE '^#[[:space:]]*define PYBIND11_INTERNALS_VERSION' \
$(python3 -c 'import pybind11; print(pybind11.get_include())') | awk '{print $NF}')" \
>> /app/versions.txt \
&& echo "PYBIND11_SHA256: $(find $(python3 -c 'import pybind11; print(pybind11.get_include())') \
-type f -name '*.h' | sort | xargs sha256sum | sha256sum | cut -d' ' -f1)" \
>> /app/versions.txt
+41 -41
View File
@@ -280,9 +280,8 @@ COPY --from=fetch_vllm \
${COMMON_WORKDIR}/vllm/requirements/rocm.txt \ ${COMMON_WORKDIR}/vllm/requirements/rocm.txt \
requirements/ requirements/
# Use constraints file to protect triton, pybind11, and torch dependencies.
RUN --mount=type=cache,id=vllm-rocm-uv,target=/root/.cache/uv \ RUN --mount=type=cache,id=vllm-rocm-uv,target=/root/.cache/uv \
uv pip install --system -c /etc/rocm-constraints.txt -r requirements/rocm.txt uv pip install --system -r requirements/rocm.txt
# Fetch only the pinned ROCm Triton kernels package. Keeping this before the # Fetch only the pinned ROCm Triton kernels package. Keeping this before the
# vLLM source COPYs makes the download reusable when native sources change. # vLLM source COPYs makes the download reusable when native sources change.
@@ -323,9 +322,8 @@ ENV SETUPTOOLS_SCM_PRETEND_VERSION="0.0.0+rocm.csrc.build"
RUN --mount=type=bind,source=pyproject.toml,target=${COMMON_WORKDIR}/vllm/pyproject.toml \ RUN --mount=type=bind,source=pyproject.toml,target=${COMMON_WORKDIR}/vllm/pyproject.toml \
--mount=type=cache,id=vllm-rocm-ccache,target=/root/.cache/ccache \ --mount=type=cache,id=vllm-rocm-ccache,target=/root/.cache/ccache \
{ grep -Fq "set(TRITON_KERNELS_TAG \"${ROCM_TRITON_KERNELS_COMMIT}\")" \ grep -Fq "set(TRITON_KERNELS_TAG \"${ROCM_TRITON_KERNELS_COMMIT}\")" \
cmake/external_projects/triton_kernels.cmake \ cmake/external_projects/triton_kernels.cmake \
|| echo "WARNING: Dockerfile.rocm ROCM_TRITON_KERNELS_COMMIT (${ROCM_TRITON_KERNELS_COMMIT}) does not match cmake TRITON_KERNELS_TAG; the Docker build uses the Dockerfile pin, cmake governs only non-Docker builds"; } \
&& export CCACHE_BASEDIR="$PWD" \ && export CCACHE_BASEDIR="$PWD" \
&& echo "=== ccache stats before ROCm native build ===" \ && echo "=== ccache stats before ROCm native build ===" \
&& (ccache --show-stats || true) \ && (ccache --show-stats || true) \
@@ -368,8 +366,7 @@ COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/requirements /requirements
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/benchmarks /benchmarks COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/benchmarks /benchmarks
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/tests /tests COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/tests /tests
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/examples /examples COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/examples /examples
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/tools/benchmark_helion_kernels.py /tools/benchmark_helion_kernels.py COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/tools /tools
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/tools/install_torchcodec_rocm.sh /tools/install_torchcodec_rocm.sh
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/docker/Dockerfile /docker/Dockerfile COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/docker/Dockerfile /docker/Dockerfile
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/docker/Dockerfile.cpu /docker/Dockerfile.cpu COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/docker/Dockerfile.cpu /docker/Dockerfile.cpu
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/docker/Dockerfile.rocm /docker/ COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/docker/Dockerfile.rocm /docker/
@@ -389,6 +386,7 @@ ARG NIXL_BRANCH="1683cf3b7f3d11674c03c5e861cea22339876c96"
ARG NIXL_REPO="https://github.com/ai-dynamo/nixl.git" ARG NIXL_REPO="https://github.com/ai-dynamo/nixl.git"
ARG UCX_BRANCH="96e58a16039f6d7d213bc967b8069238742c5194" ARG UCX_BRANCH="96e58a16039f6d7d213bc967b8069238742c5194"
ARG UCX_REPO="https://github.com/openucx/ucx.git" ARG UCX_REPO="https://github.com/openucx/ucx.git"
ENV ROCM_PATH=/opt/rocm
ENV UCX_HOME=/usr/local/ucx ENV UCX_HOME=/usr/local/ucx
ENV NIXL_HOME=/usr/local/nixl ENV NIXL_HOME=/usr/local/nixl
@@ -543,7 +541,7 @@ RUN --mount=type=tmpfs,target=/tmp/lmcache-wheel \
c_ops="$(find /tmp/lmcache-wheel -type f -name 'c_ops*.so')"; \ c_ops="$(find /tmp/lmcache-wheel -type f -name 'c_ops*.so')"; \
test "$(find /tmp/lmcache-wheel -type f -name 'c_ops*.so' | wc -l)" -eq 1; \ test "$(find /tmp/lmcache-wheel -type f -name 'c_ops*.so' | wc -l)" -eq 1; \
expected="$(printf '%s' "${LMCACHE_ROCM_ARCH}" | tr ';' '\n' | sort -u)"; \ expected="$(printf '%s' "${LMCACHE_ROCM_ARCH}" | tr ';' '\n' | sort -u)"; \
actual="$(llvm-objdump --offloading "${c_ops}" \ actual="$(/opt/rocm/llvm/bin/llvm-objdump --offloading "${c_ops}" \
| grep -oE 'gfx[0-9a-z]+' | sort -u)"; \ | grep -oE 'gfx[0-9a-z]+' | sort -u)"; \
if [ "${actual}" != "${expected}" ]; then \ if [ "${actual}" != "${expected}" ]; then \
printf 'LMCache code objects mismatch\nexpected:\n%s\nactual:\n%s\n' \ printf 'LMCache code objects mismatch\nexpected:\n%s\nactual:\n%s\n' \
@@ -551,10 +549,32 @@ RUN --mount=type=tmpfs,target=/tmp/lmcache-wheel \
exit 1; \ exit 1; \
fi fi
# DeepEP build stage. ROCShmem is provided by TheRock SDK # ROCShmem build stage - split from DeepEP so changing DEEPEP_BRANCH does not
# ROCSHMEM_DIR set to the SDK install path so DeepEP finds the headers, etc. # invalidate the slow ROCShmem build.
FROM base AS build_deepep FROM base AS build_rocshmem
ENV ROCSHMEM_DIR=${ROCM_PATH} ARG ROCSHMEM_BRANCH="f0acb0c6"
ARG ROCSHMEM_REPO="https://github.com/ROCm/rocm-systems.git"
# DeepEP only supports gfx942 and gfx950; build ROCShmem for the same set so
# it can be linked against DeepEP without arch mismatches.
ARG DEEPEP_ROCM_ARCH="gfx942;gfx950"
ENV ROCM_PATH=/opt/rocm
ENV ROCSHMEM_DIR=/opt/rocshmem
RUN --mount=type=cache,target=/root/.cache/ccache \
git clone --no-checkout --filter=blob:none ${ROCSHMEM_REPO} \
&& cd rocm-systems \
&& git sparse-checkout set --cone projects/rocshmem \
&& git checkout ${ROCSHMEM_BRANCH} \
&& mkdir -p projects/rocshmem/build \
&& cd projects/rocshmem/build \
&& CC="ccache gcc" CXX="ccache g++" INSTALL_PREFIX=${ROCSHMEM_DIR} \
bash ../scripts/build_configs/all_backends \
-DROCM_PATH=${ROCM_PATH} \
-DGPU_TARGETS="${DEEPEP_ROCM_ARCH}" \
-DUSE_EXTERNAL_MPI=OFF
# DeepEP build stage - depends on ROCShmem, builds the HIP kernel wheel.
FROM build_rocshmem AS build_deepep
ARG DEEPEP_BRANCH="0f63d3e9" ARG DEEPEP_BRANCH="0f63d3e9"
ARG DEEPEP_REPO="https://github.com/ROCm/DeepEP.git" ARG DEEPEP_REPO="https://github.com/ROCm/DeepEP.git"
ARG DEEPEP_NIC="cx7" ARG DEEPEP_NIC="cx7"
@@ -673,9 +693,8 @@ RUN if [ "$GIT_REPO_CHECK" != "0" ]; then \
# Install the packaging helpers before modifying requirements/rocm.txt. # Install the packaging helpers before modifying requirements/rocm.txt.
RUN --mount=type=cache,target=/root/.cache/uv \ RUN --mount=type=cache,target=/root/.cache/uv \
{ grep -Fq "set(TRITON_KERNELS_TAG \"${ROCM_TRITON_KERNELS_COMMIT}\")" \ grep -Fq "set(TRITON_KERNELS_TAG \"${ROCM_TRITON_KERNELS_COMMIT}\")" \
vllm/cmake/external_projects/triton_kernels.cmake \ vllm/cmake/external_projects/triton_kernels.cmake \
|| echo "WARNING: Dockerfile.rocm ROCM_TRITON_KERNELS_COMMIT (${ROCM_TRITON_KERNELS_COMMIT}) does not match cmake TRITON_KERNELS_TAG; the Docker build uses the Dockerfile pin, cmake governs only non-Docker builds"; } \
&& cd vllm \ && cd vllm \
&& uv pip install --system setuptools_scm regex && uv pip install --system setuptools_scm regex
@@ -731,8 +750,7 @@ COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/requirements /requir
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/benchmarks /benchmarks COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/benchmarks /benchmarks
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/tests /tests COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/tests /tests
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/examples /examples COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/examples /examples
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/tools/benchmark_helion_kernels.py /tools/benchmark_helion_kernels.py COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/tools /tools
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/tools/install_torchcodec_rocm.sh /tools/install_torchcodec_rocm.sh
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/docker/Dockerfile /docker/Dockerfile COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/docker/Dockerfile /docker/Dockerfile
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/docker/Dockerfile.cpu /docker/Dockerfile.cpu COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/docker/Dockerfile.cpu /docker/Dockerfile.cpu
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/docker/Dockerfile.rocm /docker/ COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/docker/Dockerfile.rocm /docker/
@@ -777,6 +795,9 @@ RUN --mount=type=bind,from=build_nixl,src=/app/install,target=/nixl_install \
--mount=type=bind,from=build_deepep,src=/app/deep_install,target=/deep_install \ --mount=type=bind,from=build_deepep,src=/app/deep_install,target=/deep_install \
uv pip install --system /nixl_install/*.whl /deep_install/*.whl uv pip install --system /nixl_install/*.whl /deep_install/*.whl
# Copy ROCShmem runtime libraries.
COPY --from=build_rocshmem /opt/rocshmem /opt/rocshmem
# RDMA userspace libraries plus FFmpeg dev libs needed by torchcodec. # RDMA userspace libraries plus FFmpeg dev libs needed by torchcodec.
RUN timeout --kill-after=30s 5m apt-get update -q -y --error-on=any \ RUN timeout --kill-after=30s 5m apt-get update -q -y --error-on=any \
-o Acquire::Retries=3 \ -o Acquire::Retries=3 \
@@ -809,8 +830,7 @@ RUN --mount=type=cache,target=/root/.cache/uv \
# Pre-install shared ROCm runtime dependencies. # Pre-install shared ROCm runtime dependencies.
COPY requirements/common.txt requirements/rocm.txt /tmp/ci-base-requirements/ COPY requirements/common.txt requirements/rocm.txt /tmp/ci-base-requirements/
RUN --mount=type=cache,target=/root/.cache/uv \ RUN --mount=type=cache,target=/root/.cache/uv \
uv pip install --system -c /etc/rocm-constraints.txt \ uv pip install --system -r /tmp/ci-base-requirements/rocm.txt \
-r /tmp/ci-base-requirements/rocm.txt \
&& rm -rf /tmp/ci-base-requirements && rm -rf /tmp/ci-base-requirements
# Enable fast and less brittle model downloads in tests. # Enable fast and less brittle model downloads in tests.
@@ -888,6 +908,8 @@ RUN mkdir src && mv vllm src/vllm
# Catch GPU<->CPU syncs in execute_model/sample_tokens during tests. # Catch GPU<->CPU syncs in execute_model/sample_tokens during tests.
ENV VLLM_GPU_SYNC_CHECK=error ENV VLLM_GPU_SYNC_CHECK=error
# Run the structural smoke test inside BuildKit, where the test image layers
# are already available. Export only a build-scoped success marker.
FROM test AS test_smoke FROM test AS test_smoke
ARG ROCM_SMOKE_ID ARG ROCM_SMOKE_ID
RUN --network=none \ RUN --network=none \
@@ -934,37 +956,15 @@ RUN --mount=type=bind,from=export_vllm,src=/,target=/install \
--mount=type=cache,target=/root/.cache/uv \ --mount=type=cache,target=/root/.cache/uv \
cd /install \ cd /install \
&& echo "opentelemetry-api<=1.40.0" > /tmp/otel-constraints.txt \ && echo "opentelemetry-api<=1.40.0" > /tmp/otel-constraints.txt \
&& uv pip install --system -c /etc/rocm-constraints.txt -c /tmp/otel-constraints.txt -r requirements/rocm.txt \ && uv pip install --system -c /tmp/otel-constraints.txt -r requirements/rocm.txt \
&& pip uninstall -y vllm \ && pip uninstall -y vllm \
&& uv pip install --system -c /etc/rocm-constraints.txt -c /tmp/otel-constraints.txt *.whl \ && uv pip install --system -c /tmp/otel-constraints.txt *.whl \
&& rm /tmp/otel-constraints.txt && rm /tmp/otel-constraints.txt
# Install NIXL ROCm wheel # Install NIXL ROCm wheel
RUN --mount=type=bind,from=build_nixl,src=/app/install,target=/nixl_install \ RUN --mount=type=bind,from=build_nixl,src=/app/install,target=/nixl_install \
uv pip install --system /nixl_install/*.whl uv pip install --system /nixl_install/*.whl
# Recompute the base's triton fingerprint to detect resolver interference.
RUN EXPECTED=$(grep '^TRITON_SHA256: ' /app/versions.txt | cut -d' ' -f2) \
&& ACTUAL=$(find $(python3 -c 'import triton, os; print(os.path.dirname(triton.__file__))') \
-type f \( -name '*.py' -o -name '*.so' \) | sort | xargs sha256sum | sha256sum | cut -d' ' -f1) \
&& echo "triton sha256 expected=${EXPECTED} actual=${ACTUAL}" \
&& test -n "${EXPECTED}" \
&& test "${EXPECTED}" = "${ACTUAL}" \
&& python3 -c "import torch; assert torch.version.hip, f'non-ROCm torch: {torch.__version__}'; print(f'torch={torch.__version__}')"
# Recompute the base's pybind11 fingerprint to detect resolver interference.
RUN EXPECTED=$(grep '^PYBIND11_SHA256: ' /app/versions.txt | cut -d' ' -f2) \
&& EXPECTED_ABI=$(grep '^PYBIND11_INTERNALS_VERSION: ' /app/versions.txt | cut -d' ' -f2) \
&& INC=$(python3 -c 'import pybind11; print(pybind11.get_include())') \
&& ACTUAL=$(find "${INC}" -type f -name '*.h' | sort | xargs sha256sum | sha256sum | cut -d' ' -f1) \
&& ACTUAL_ABI=$(grep -rhE '^#[[:space:]]*define PYBIND11_INTERNALS_VERSION' "${INC}" | awk '{print $NF}') \
&& echo "pybind11 $(pip show pybind11 | awk '/^Version:/ {print $2}')" \
"sha256 expected=${EXPECTED} actual=${ACTUAL}" \
"internals expected=${EXPECTED_ABI} actual=${ACTUAL_ABI}" \
&& test -n "${EXPECTED}" \
&& test "${EXPECTED}" = "${ACTUAL}" \
&& test "${EXPECTED_ABI}" = "${ACTUAL_ABI}"
ARG COMMON_WORKDIR ARG COMMON_WORKDIR
ARG BASE_IMAGE ARG BASE_IMAGE
ARG NIC_BACKEND ARG NIC_BACKEND
-479
View File
@@ -1,479 +0,0 @@
ARG BASE_IMAGE=rocm/dev-ubuntu-22.04:7.2.3-complete
ARG TRITON_BRANCH="669b31a" # release/internal/3.8.x as of 09/17
ARG TRITON_REPO="https://github.com/ROCm/triton.git"
ARG PYTORCH_BRANCH="733fca1" # release/2.13 as of 09/17
ARG PYTORCH_REPO="https://github.com/ROCm/pytorch.git"
ARG PYTORCH_VISION_BRANCH="v0.28.0"
ARG PYTORCH_VISION_REPO="https://github.com/pytorch/vision.git"
ARG PYTORCH_AUDIO_BRANCH="v2.11.0"
ARG PYTORCH_AUDIO_REPO="https://github.com/pytorch/audio.git"
ARG FA_BRANCH="0e60e394"
ARG FA_REPO="https://github.com/Dao-AILab/flash-attention.git"
ARG AITER_BRANCH="v0.1.25"
ARG AITER_REPO="https://github.com/ROCm/aiter.git"
ARG MORI_BRANCH="v1.2.3.post1"
ARG MORI_REPO="https://github.com/ROCm/mori.git"
ARG ROCM_SYSTEMS_REPO="https://github.com/ROCm/rocm-systems.git"
ARG ROCPROFILER_SDK_COMMIT="2b22ab0195cc1461cd9abf3b969e9dd7c10af350"
ARG ROCM_RUNTIME_COMMIT="af70edc91f1dc5d228fd4c076df99e65e49c4521"
# Sccache configuration (only used in release pipeline)
ARG USE_SCCACHE
ARG SCCACHE_DOWNLOAD_URL
ARG SCCACHE_ENDPOINT
ARG SCCACHE_BUCKET_NAME=vllm-build-sccache
ARG SCCACHE_REGION_NAME=us-west-2
ARG SCCACHE_S3_NO_CREDENTIALS=0
FROM ${BASE_IMAGE} AS base
ENV PATH=/opt/rocm/llvm/bin:/opt/rocm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
ENV ROCM_PATH=/opt/rocm
ENV LD_LIBRARY_PATH=/opt/rocm/lib:/usr/local/lib:
ARG PYTORCH_ROCM_ARCH=gfx90a;gfx942;gfx950;gfx1100;gfx1101;gfx1200;gfx1201;gfx1150;gfx1151
ENV PYTORCH_ROCM_ARCH=${PYTORCH_ROCM_ARCH}
ENV AITER_ROCM_ARCH=gfx942;gfx950
# Note: Do not set MORI_GPU_ARCHS here, it is automatically inferred at runtime
# Required for RCCL in ROCm7.1
ENV HSA_NO_SCRATCH_RECLAIM=1
ARG PYTHON_VERSION=3.12
ENV PYTHON_VERSION=${PYTHON_VERSION}
RUN mkdir -p /app
WORKDIR /app
ENV DEBIAN_FRONTEND=noninteractive
# Install Python and other dependencies (plus gcc-13, selected later in build_pytorch).
RUN apt-get update -y \
&& apt-get install -y software-properties-common git curl sudo vim less libgfortran5 libopenmpi-dev libpci-dev liblzma-dev pkg-config \
&& for i in 1 2 3; do \
add-apt-repository -y ppa:deadsnakes/ppa && add-apt-repository -y ppa:ubuntu-toolchain-r/test && break || \
{ echo "Attempt $i failed, retrying in 5s..."; sleep 5; }; \
done \
&& apt-get update -y \
&& apt-get install -y python${PYTHON_VERSION} python${PYTHON_VERSION}-dev python${PYTHON_VERSION}-venv \
python${PYTHON_VERSION}-lib2to3 python-is-python3 gcc-13 g++-13 \
&& update-alternatives --install /usr/bin/python3 python3 /usr/bin/python${PYTHON_VERSION} 1 \
&& update-alternatives --set python3 /usr/bin/python${PYTHON_VERSION} \
&& ln -sf /usr/bin/python${PYTHON_VERSION}-config /usr/bin/python3-config \
&& curl -sS https://bootstrap.pypa.io/get-pip.py | python${PYTHON_VERSION} \
&& python3 --version && python3 -m pip --version
RUN pip install -U packaging 'cmake<4' ninja wheel 'setuptools<80' 'pybind11<3.1.0' Cython
RUN apt-get update && apt-get install -y libjpeg-dev libsox-dev libsox-fmt-all sox && rm -rf /var/lib/apt/lists/*
# Install sccache if USE_SCCACHE is enabled (for release builds)
ARG USE_SCCACHE
ARG SCCACHE_DOWNLOAD_URL
ARG SCCACHE_ENDPOINT
ARG SCCACHE_BUCKET_NAME
ARG SCCACHE_REGION_NAME
ARG SCCACHE_S3_NO_CREDENTIALS
RUN if [ "$USE_SCCACHE" = "1" ]; then \
echo "Installing sccache..." \
&& SCCACHE_ARCH="x86_64" \
&& SCCACHE_VERSION="v0.8.1" \
&& SCCACHE_DL_URL="${SCCACHE_DOWNLOAD_URL:-https://github.com/mozilla/sccache/releases/download/${SCCACHE_VERSION}/sccache-${SCCACHE_VERSION}-${SCCACHE_ARCH}-unknown-linux-musl.tar.gz}" \
&& curl -L -o /tmp/sccache.tar.gz ${SCCACHE_DL_URL} \
&& tar -xzf /tmp/sccache.tar.gz -C /tmp \
&& mv /tmp/sccache-${SCCACHE_VERSION}-${SCCACHE_ARCH}-unknown-linux-musl/sccache /usr/bin/sccache \
&& chmod +x /usr/bin/sccache \
&& rm -rf /tmp/sccache.tar.gz /tmp/sccache-${SCCACHE_VERSION}-${SCCACHE_ARCH}-unknown-linux-musl \
&& sccache --version; \
fi
# Setup sccache for HIP compilation via HIP_CLANG_PATH
# This creates wrapper scripts in a separate directory and points HIP to use them
# This avoids modifying the original ROCm binaries which can break detection
# NOTE: HIP_CLANG_PATH is NOT set as ENV to avoid affecting downstream images (Dockerfile.rocm_72)
# Instead, each build stage should export HIP_CLANG_PATH=/opt/sccache-wrappers if USE_SCCACHE=1
RUN if [ "$USE_SCCACHE" = "1" ]; then \
echo "Setting up sccache wrappers for HIP compilation..." \
&& mkdir -p /opt/sccache-wrappers \
&& printf '#!/bin/bash\nexec sccache /opt/rocm/lib/llvm/bin/clang++ "$@"\n' > /opt/sccache-wrappers/clang++ \
&& chmod +x /opt/sccache-wrappers/clang++ \
&& printf '#!/bin/bash\nexec sccache /opt/rocm/lib/llvm/bin/clang "$@"\n' > /opt/sccache-wrappers/clang \
&& chmod +x /opt/sccache-wrappers/clang \
&& echo "sccache wrappers created in /opt/sccache-wrappers"; \
fi
# Set sccache environment variables only when USE_SCCACHE=1
# This prevents S3 config from leaking into images when sccache is not used
ARG USE_SCCACHE
ENV SCCACHE_BUCKET=${USE_SCCACHE:+${SCCACHE_BUCKET_NAME}}
ENV SCCACHE_REGION=${USE_SCCACHE:+${SCCACHE_REGION_NAME}}
ENV SCCACHE_S3_NO_CREDENTIALS=${USE_SCCACHE:+${SCCACHE_S3_NO_CREDENTIALS}}
ENV SCCACHE_IDLE_TIMEOUT=${USE_SCCACHE:+0}
###
### Triton Build
###
FROM base AS build_triton
ARG TRITON_BRANCH
ARG TRITON_REPO
RUN git clone ${TRITON_REPO}
RUN cd triton \
&& git checkout ${TRITON_BRANCH} \
&& git config --global user.email "you@example.com" && git config --global user.name "Your Name" \
&& if [ ! -f setup.py ]; then cd python; fi \
&& MAX_JOBS=16 python3 setup.py bdist_wheel --dist-dir=dist \
&& mkdir -p /app/install && cp dist/*.whl /app/install
###
### AMD SMI Build
###
FROM base AS build_amdsmi
RUN cd /opt/rocm/share/amd_smi \
&& pip wheel . --wheel-dir=dist
RUN mkdir -p /app/install && cp /opt/rocm/share/amd_smi/dist/*.whl /app/install
###
### Pytorch build
###
FROM base AS build_pytorch
ARG PYTORCH_BRANCH
ARG PYTORCH_REPO
ARG USE_SCCACHE
RUN apt-get update && apt-get install -y pkg-config liblzma-dev
# torch 2.12 emits float->__bf16 libcalls (__truncsfbf2) absent from gcc-11 libgcc; build with gcc-13.
RUN update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-13 130 --slave /usr/bin/g++ g++ /usr/bin/g++-13
RUN git clone ${PYTORCH_REPO} pytorch
RUN cd pytorch && git checkout ${PYTORCH_BRANCH}
RUN cd pytorch \
&& pip install -r requirements.txt && git submodule update --init --recursive
RUN cd pytorch && python3 tools/amd_build/build_amd.py \
&& if [ "$USE_SCCACHE" = "1" ]; then \
export HIP_CLANG_PATH=/opt/sccache-wrappers \
&& export CMAKE_C_COMPILER_LAUNCHER=sccache \
&& export CMAKE_CXX_COMPILER_LAUNCHER=sccache \
&& sccache --show-stats; \
fi \
&& CMAKE_PREFIX_PATH=$(python3 -c 'import sys; print(sys.prefix)') python3 setup.py bdist_wheel --dist-dir=dist \
&& if [ "$USE_SCCACHE" = "1" ]; then sccache --show-stats; fi
RUN mkdir -p /app/install && cp /app/pytorch/dist/*.whl /app/install
###
### Shared PyTorch runtime for dependent builds
###
FROM base AS build_pytorch_runtime
RUN update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-13 130 --slave /usr/bin/g++ g++ /usr/bin/g++-13
RUN --mount=type=bind,from=build_pytorch,src=/app/install/,target=/install,ro \
pip install /install/*.whl
###
### TorchVision build
###
FROM build_pytorch_runtime AS build_torchvision
ARG PYTORCH_VISION_BRANCH
ARG PYTORCH_VISION_REPO
ARG USE_SCCACHE
RUN git clone ${PYTORCH_VISION_REPO} vision
RUN cd vision && git checkout ${PYTORCH_VISION_BRANCH} \
&& if [ "$USE_SCCACHE" = "1" ]; then \
export HIP_CLANG_PATH=/opt/sccache-wrappers \
&& export CMAKE_C_COMPILER_LAUNCHER=sccache \
&& export CMAKE_CXX_COMPILER_LAUNCHER=sccache; \
fi \
&& python3 setup.py bdist_wheel --dist-dir=dist \
&& if [ "$USE_SCCACHE" = "1" ]; then sccache --show-stats; fi
RUN mkdir -p /app/install && cp /app/vision/dist/*.whl /app/install
###
### TorchAudio build
###
FROM build_pytorch_runtime AS build_torchaudio
ARG PYTORCH_AUDIO_BRANCH
ARG PYTORCH_AUDIO_REPO
ARG USE_SCCACHE
RUN git clone ${PYTORCH_AUDIO_REPO} audio
RUN cd audio && git checkout ${PYTORCH_AUDIO_BRANCH} \
&& git submodule update --init --recursive
RUN cd audio && pip install -r requirements.txt
RUN cd audio \
&& if [ "$USE_SCCACHE" = "1" ]; then \
export HIP_CLANG_PATH=/opt/sccache-wrappers \
&& export CMAKE_C_COMPILER_LAUNCHER=sccache \
&& export CMAKE_CXX_COMPILER_LAUNCHER=sccache; \
fi \
&& python3 setup.py bdist_wheel --dist-dir=dist \
&& if [ "$USE_SCCACHE" = "1" ]; then sccache --show-stats; fi
RUN mkdir -p /app/install && cp /app/audio/dist/*.whl /app/install
###
### MORI Build
###
FROM base AS build_mori
ARG MORI_BRANCH
ARG MORI_REPO
RUN --mount=type=bind,from=build_pytorch,src=/app/install/,target=/install \
pip install /install/*.whl
RUN git clone ${MORI_REPO}
RUN cd mori \
&& git checkout ${MORI_BRANCH} \
&& git submodule update --init --recursive \
&& python3 setup.py bdist_wheel --dist-dir=dist && ls /app/mori/dist/*.whl
RUN mkdir -p /app/install && cp /app/mori/dist/*.whl /app/install
###
### FlashAttention Build
###
FROM base AS build_fa
ARG FA_BRANCH
ARG FA_REPO
ARG USE_SCCACHE
RUN --mount=type=bind,from=build_pytorch,src=/app/install/,target=/install \
pip install /install/*.whl
RUN git clone ${FA_REPO}
RUN cd flash-attention \
&& git checkout ${FA_BRANCH} \
&& git submodule update --init \
&& if [ "$USE_SCCACHE" = "1" ]; then \
export HIP_CLANG_PATH=/opt/sccache-wrappers \
&& sccache --show-stats; \
fi \
&& GPU_ARCHS=$(echo ${PYTORCH_ROCM_ARCH} | sed -e 's/;gfx1[0-9]\{3\}//g') python3 setup.py bdist_wheel --dist-dir=dist \
&& if [ "$USE_SCCACHE" = "1" ]; then sccache --show-stats; fi
RUN mkdir -p /app/install && cp /app/flash-attention/dist/*.whl /app/install
###
### AITER Build
###
FROM base AS build_aiter
ARG AITER_BRANCH
ARG AITER_REPO
ARG USE_SCCACHE
RUN --mount=type=bind,from=build_pytorch,src=/app/install/,target=/install \
pip install /install/*.whl
RUN git clone --recursive --branch ${AITER_BRANCH} ${AITER_REPO}
RUN cd aiter \
&& git submodule update --init --recursive \
&& pip install -r requirements.txt
RUN pip install pyyaml && cd aiter \
&& if [ "$USE_SCCACHE" = "1" ]; then \
export HIP_CLANG_PATH=/opt/sccache-wrappers \
&& sccache --show-stats; \
fi \
&& PREBUILD_KERNELS=1 AITER_USE_SYSTEM_TRITON=1 GPU_ARCHS=${AITER_ROCM_ARCH} python3 setup.py bdist_wheel --dist-dir=dist \
&& if [ "$USE_SCCACHE" = "1" ]; then sccache --show-stats; fi \
&& ls /app/aiter/dist/*.whl
RUN mkdir -p /app/install && cp /app/aiter/dist/*.whl /app/install
###
### Final Build
###
# Wheel release stage -
# only includes dependencies used by wheel release pipeline
FROM base AS debs_wheel_release
RUN mkdir /app/debs
RUN --mount=type=bind,from=build_triton,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_fa,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_amdsmi,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_pytorch,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_torchvision,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_torchaudio,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_aiter,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
# Full debs stage - includes Mori (used by Docker releases)
FROM base AS debs
RUN mkdir /app/debs
RUN --mount=type=bind,from=build_triton,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_fa,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_amdsmi,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_pytorch,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_torchvision,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_torchaudio,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_aiter,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_mori,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
###
### ROCr + CLR build
###
# Pull in updates to ROCr + clr from ROCm 7.2.4 and beyond.
FROM base AS build_rocm_runtime
ARG ROCM_SYSTEMS_REPO
ARG ROCM_RUNTIME_COMMIT
RUN apt-get update \
&& apt-get install -y --no-install-recommends g++ \
libelf-dev libdrm-dev libnuma-dev libdw-dev xxd \
&& rm -rf /var/lib/apt/lists/*
RUN pip install --no-cache-dir CppHeaderParser
RUN git init -q /src && cd /src \
&& git remote add origin ${ROCM_SYSTEMS_REPO} \
&& git config core.sparseCheckout true \
&& git config remote.origin.promisor true \
&& git config remote.origin.partialclonefilter blob:none \
&& git sparse-checkout init --cone \
&& git sparse-checkout set projects/rocr-runtime projects/clr projects/hip shared cmake \
&& git fetch --filter=blob:none --depth 1 origin ${ROCM_RUNTIME_COMMIT} \
&& git checkout -q FETCH_HEAD
# ROCr's trap_handler and blit_shaders call find_package(Clang/LLVM REQUIRED) only to get
# IMPORTED executables. The ROCm dev image ships the binaries but no CMake package config, so
# supply one pointing at them.
RUN mkdir -p /opt/rocm-cmake-shim \
&& printf 'if(NOT TARGET clang)\n add_executable(clang IMPORTED GLOBAL)\n set_target_properties(clang PROPERTIES IMPORTED_LOCATION "/opt/rocm/llvm/bin/clang")\nendif()\nset(Clang_PACKAGE_VERSION "rocm-image-shim")\n' > /opt/rocm-cmake-shim/ClangConfig.cmake \
&& printf 'if(NOT TARGET llvm-objcopy)\n add_executable(llvm-objcopy IMPORTED GLOBAL)\n set_target_properties(llvm-objcopy PROPERTIES IMPORTED_LOCATION "/opt/rocm/llvm/bin/llvm-objcopy")\nendif()\nset(LLVM_FOUND TRUE)\nset(LLVM_PACKAGE_VERSION "rocm-image-shim")\n' > /opt/rocm-cmake-shim/LLVMConfig.cmake
# ROCr first, and installed into /opt/rocm, because the clr build below needs its headers
RUN cd /src/projects/rocr-runtime \
&& cmake -B build -G Ninja \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_PREFIX_PATH=/opt/rocm \
-DCMAKE_INSTALL_PREFIX=/opt/rocm \
-DClang_DIR=/opt/rocm-cmake-shim \
-DLLVM_DIR=/opt/rocm-cmake-shim \
&& cmake --build build --parallel 16 \
&& cmake --install build \
&& cmake --install build --prefix /rocr-install --strip
RUN cd /src/projects/clr \
&& cmake -B build -G Ninja \
-DCMAKE_BUILD_TYPE=Release \
-DCLR_BUILD_HIP=ON \
-DCLR_BUILD_OCL=OFF \
-DHIP_COMMON_DIR=/src/projects/hip \
-DROCM_PATH=/opt/rocm \
-DCMAKE_PREFIX_PATH=/opt/rocm \
-DCMAKE_INSTALL_PREFIX=/opt/rocm \
-DLLVM_DIR=/opt/rocm-cmake-shim \
-DHIP_LLVM_ROOT=/opt/rocm/llvm \
&& cmake --build build --parallel 16 \
&& cmake --install build --prefix /clr-install --strip
RUN mkdir -p /staging/lib \
&& cp -P /rocr-install/lib/libhsa-runtime64.so* /staging/lib/ \
&& cp -P /clr-install/lib/libamdhip64.so* /staging/lib/ \
&& ls -l /staging/lib
# rocprofiler-sdk 1.3.2 fixes torch 2.12 profiler on ROCm 7.2.3
FROM base AS build_rocprofiler_sdk
ARG ROCM_SYSTEMS_REPO
ARG ROCPROFILER_SDK_COMMIT
ARG USE_SCCACHE
RUN apt-get update \
&& apt-get install -y --no-install-recommends libdw-dev libelf-dev libsqlite3-dev libzstd-dev \
&& rm -rf /var/lib/apt/lists/*
RUN git init -q /src && cd /src \
&& git remote add origin ${ROCM_SYSTEMS_REPO} \
&& git config core.sparseCheckout true \
&& git config remote.origin.promisor true \
&& git config remote.origin.partialclonefilter blob:none \
&& git sparse-checkout init --cone \
&& git sparse-checkout set projects/rocprofiler-sdk shared cmake \
&& git fetch --filter=blob:none --depth 1 origin ${ROCPROFILER_SDK_COMMIT} \
&& git checkout -q FETCH_HEAD
# Patch the ROCm 7.2.3 build of rocprofiler-sdk to include queue-interposition fixes that are
# absent from the pinned 7.14 build. This resolves hangs on torch.profiler on ROCm.
# TODO: Drop these once ${ROCPROFILER_SDK_COMMIT} is bumped to a build that already
# contains them, such as ROCm 10.0. See https://github.com/vllm-project/vllm/issues/54087.
ARG ROCPROFILER_SDK_PR_7796=fae7fd57ffb8a76e811d6410b8a0cee1e94d710a
ARG ROCPROFILER_SDK_PR_7924=f8db22adbe17686a9b05ae03367dd1ec73011653
RUN cd /src \
&& HSA_PATHS='projects/rocprofiler-sdk/source/lib/rocprofiler-sdk/hsa/*' \
&& curl -fsSL -o /tmp/qi-7796.patch "${ROCM_SYSTEMS_REPO%.git}/commit/${ROCPROFILER_SDK_PR_7796}.patch" \
&& curl -fsSL -o /tmp/qi-7924.patch "${ROCM_SYSTEMS_REPO%.git}/commit/${ROCPROFILER_SDK_PR_7924}.patch" \
&& git apply -p1 --include="$HSA_PATHS" --verbose /tmp/qi-7796.patch \
&& git apply -p1 --include="$HSA_PATHS" --verbose /tmp/qi-7924.patch
RUN cd /src/projects/rocprofiler-sdk \
&& if [ "$USE_SCCACHE" = "1" ]; then \
export CMAKE_C_COMPILER_LAUNCHER=sccache \
&& export CMAKE_CXX_COMPILER_LAUNCHER=sccache; \
fi \
&& cmake -B build -G Ninja \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_PREFIX_PATH=/opt/rocm \
-DCMAKE_INSTALL_PREFIX=/opt/rocm \
-DROCPROFILER_BUILD_TESTS=OFF \
-DROCPROFILER_BUILD_SAMPLES=OFF \
-DROCPROFILER_BUILD_DOCS=OFF \
&& cmake --build build --parallel "$(nproc)" \
&& cmake --install build --prefix /staging --strip
FROM base AS final
RUN --mount=type=bind,from=debs,src=/app/debs,target=/install \
pip install /install/*.whl
COPY --from=build_rocprofiler_sdk /staging/ /opt/rocm/
ENV ROCPROFILER_QUEUE_INTERPOSITION=0
COPY --from=build_rocm_runtime /staging/ /opt/rocm/
# Hard link the patched library onto every stock file name so that both the
# SONAME lookup and the hard coded paths resolve to it.
RUN set -eux; \
cd /opt/rocm/lib; \
ours_hsa="$(readlink -f libhsa-runtime64.so.1)"; \
ours_hip="$(readlink -f libamdhip64.so.7)"; \
for f in libhsa-runtime64.so.1.*; do \
[ "$f" = "${ours_hsa##*/}" ] || ln -f "$ours_hsa" "$f"; \
done; \
for f in libamdhip64.so.7.*; do \
[ "$f" = "${ours_hip##*/}" ] || ln -f "$ours_hip" "$f"; \
done
RUN ldconfig
ARG BASE_IMAGE
ARG TRITON_BRANCH
ARG TRITON_REPO
ARG PYTORCH_BRANCH
ARG PYTORCH_VISION_BRANCH
ARG PYTORCH_REPO
ARG PYTORCH_VISION_REPO
ARG PYTORCH_AUDIO_BRANCH
ARG PYTORCH_AUDIO_REPO
ARG FA_BRANCH
ARG FA_REPO
ARG AITER_BRANCH
ARG AITER_REPO
ARG MORI_BRANCH
ARG MORI_REPO
ARG ROCM_SYSTEMS_REPO
ARG ROCPROFILER_SDK_COMMIT
ARG ROCM_RUNTIME_COMMIT
RUN echo "BASE_IMAGE: ${BASE_IMAGE}" > /app/versions.txt \
&& echo "ROCM_SYSTEMS_REPO: ${ROCM_SYSTEMS_REPO}" >> /app/versions.txt \
&& echo "ROCPROFILER_SDK_COMMIT: ${ROCPROFILER_SDK_COMMIT}" >> /app/versions.txt \
&& echo "ROCM_RUNTIME_COMMIT: ${ROCM_RUNTIME_COMMIT}" >> /app/versions.txt \
&& echo "TRITON_BRANCH: ${TRITON_BRANCH}" >> /app/versions.txt \
&& echo "TRITON_REPO: ${TRITON_REPO}" >> /app/versions.txt \
&& echo "PYTORCH_BRANCH: ${PYTORCH_BRANCH}" >> /app/versions.txt \
&& echo "PYTORCH_VISION_BRANCH: ${PYTORCH_VISION_BRANCH}" >> /app/versions.txt \
&& echo "PYTORCH_REPO: ${PYTORCH_REPO}" >> /app/versions.txt \
&& echo "PYTORCH_VISION_REPO: ${PYTORCH_VISION_REPO}" >> /app/versions.txt \
&& echo "PYTORCH_AUDIO_BRANCH: ${PYTORCH_AUDIO_BRANCH}" >> /app/versions.txt \
&& echo "PYTORCH_AUDIO_REPO: ${PYTORCH_AUDIO_REPO}" >> /app/versions.txt \
&& echo "FA_BRANCH: ${FA_BRANCH}" >> /app/versions.txt \
&& echo "FA_REPO: ${FA_REPO}" >> /app/versions.txt \
&& echo "AITER_BRANCH: ${AITER_BRANCH}" >> /app/versions.txt \
&& echo "AITER_REPO: ${AITER_REPO}" >> /app/versions.txt \
&& echo "MORI_BRANCH: ${MORI_BRANCH}" >> /app/versions.txt \
&& echo "MORI_REPO: ${MORI_REPO}" >> /app/versions.txt
+270 -267
View File
@@ -1,11 +1,12 @@
ARG BASE_IMAGE=ubuntu:22.04 ARG BASE_IMAGE=rocm/dev-ubuntu-22.04:7.2.3-complete
ARG PYTHON_VERSION=3.12 ARG TRITON_BRANCH="669b31a" # release/internal/3.8.x as of 09/17
ARG TORCH_VERSION=2.12.0+rocm10.0.0
ARG TORCHVISION_VERSION=0.27.0+rocm10.0.0
ARG TORCHAUDIO_VERSION=2.11.0+rocm10.0.0
ARG TRITON_BRANCH="669b31ac" # release/internal/3.8.x as of 09/29/2026
ARG TRITON_VERSION=3.8.0+git669b31ac
ARG TRITON_REPO="https://github.com/ROCm/triton.git" ARG TRITON_REPO="https://github.com/ROCm/triton.git"
ARG PYTORCH_BRANCH="733fca1" # release/2.13 as of 09/17
ARG PYTORCH_REPO="https://github.com/ROCm/pytorch.git"
ARG PYTORCH_VISION_BRANCH="v0.28.0"
ARG PYTORCH_VISION_REPO="https://github.com/pytorch/vision.git"
ARG PYTORCH_AUDIO_BRANCH="v2.11.0"
ARG PYTORCH_AUDIO_REPO="https://github.com/pytorch/audio.git"
ARG FA_BRANCH="0e60e394" ARG FA_BRANCH="0e60e394"
ARG FA_REPO="https://github.com/Dao-AILab/flash-attention.git" ARG FA_REPO="https://github.com/Dao-AILab/flash-attention.git"
ARG AITER_BRANCH="v0.1.25" ARG AITER_BRANCH="v0.1.25"
@@ -13,18 +14,8 @@ ARG AITER_REPO="https://github.com/ROCm/aiter.git"
ARG MORI_BRANCH="v1.2.3.post1" ARG MORI_BRANCH="v1.2.3.post1"
ARG MORI_REPO="https://github.com/ROCm/mori.git" ARG MORI_REPO="https://github.com/ROCm/mori.git"
ARG ROCM_SYSTEMS_REPO="https://github.com/ROCm/rocm-systems.git" ARG ROCM_SYSTEMS_REPO="https://github.com/ROCm/rocm-systems.git"
ARG ROCM_RUNTIME_COMMIT="25e14349f2606a6b95c4ccd45eb92a1835646cf2" ARG ROCPROFILER_SDK_COMMIT="2b22ab0195cc1461cd9abf3b969e9dd7c10af350"
ARG ROCM_RUNTIME_COMMIT="af70edc91f1dc5d228fd4c076df99e65e49c4521"
# TheRock configuration
ARG SITE_PACKAGES=/usr/local/lib/python${PYTHON_VERSION}/dist-packages
ARG ROCM_RELEASE_WHEELS_MULTIARCH_URL="https://stable.repo.amd.com/rocm/whl-next/" # For pre-built GPU images, packages, etc.
ARG ROCM_NIGHTLY_WHEELS_MULTIARCH_URL="https://nightly.repo.amd.com/rocm/whl-next/" # For torch, triton, etc. wheels with latest fixes.
ARG ROCM_SDK_VERSION=10.0.0
# Build configuration
ARG PYTORCH_ROCM_ARCH="gfx90a;gfx942;gfx950;gfx1100;gfx1101;gfx1200;gfx1201;gfx1150;gfx1151"
ARG AITER_ROCM_ARCH="gfx942;gfx950"
# Sccache configuration (only used in release pipeline) # Sccache configuration (only used in release pipeline)
ARG USE_SCCACHE ARG USE_SCCACHE
@@ -34,128 +25,47 @@ ARG SCCACHE_BUCKET_NAME=vllm-build-sccache
ARG SCCACHE_REGION_NAME=us-west-2 ARG SCCACHE_REGION_NAME=us-west-2
ARG SCCACHE_S3_NO_CREDENTIALS=0 ARG SCCACHE_S3_NO_CREDENTIALS=0
FROM ${BASE_IMAGE} AS base FROM ${BASE_IMAGE} AS base
ARG PYTORCH_ROCM_ARCH ENV PATH=/opt/rocm/llvm/bin:/opt/rocm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
ENV PYTORCH_ROCM_ARCH="${PYTORCH_ROCM_ARCH}" ENV ROCM_PATH=/opt/rocm
ARG AITER_ROCM_ARCH ENV LD_LIBRARY_PATH=/opt/rocm/lib:/usr/local/lib:
ENV AITER_ROCM_ARCH="${AITER_ROCM_ARCH}"
ARG SITE_PACKAGES ARG PYTORCH_ROCM_ARCH=gfx90a;gfx942;gfx950;gfx1100;gfx1101;gfx1200;gfx1201;gfx1150;gfx1151
ENV SITE_PACKAGES=${SITE_PACKAGES} ENV PYTORCH_ROCM_ARCH=${PYTORCH_ROCM_ARCH}
ENV SDK_DEV="$SITE_PACKAGES/_rocm_sdk_devel" \ ENV AITER_ROCM_ARCH=gfx942;gfx950
SDK_CORE="$SITE_PACKAGES/_rocm_sdk_core" # Note: Do not set MORI_GPU_ARCHS here, it is automatically inferred at runtime
ENV ROCM_PATH="$SDK_DEV" \
ROCM_HOME="$SDK_DEV" \
HIP_PATH="$SDK_DEV" \
HIP_CLANG_PATH="$SDK_DEV/lib/llvm/bin" \
HIP_DEVICE_LIB_PATH="$SDK_CORE/lib/llvm/amdgcn/bitcode"
ENV LD_LIBRARY_PATH="$SDK_DEV/lib:$SDK_DEV/lib/rocm_sysdeps/lib:$SDK_CORE/lib:$SDK_CORE/lib/rocm_sysdeps/lib"
ENV PATH="$SDK_DEV/bin:$SDK_DEV/lib/llvm/bin:$SDK_CORE/bin:$PATH"
# Required for RCCL in ROCm7.1 # Required for RCCL in ROCm7.1
ENV HSA_NO_SCRATCH_RECLAIM=1 ENV HSA_NO_SCRATCH_RECLAIM=1
ARG PYTHON_VERSION ARG PYTHON_VERSION=3.12
ENV PYTHON_VERSION=${PYTHON_VERSION} ENV PYTHON_VERSION=${PYTHON_VERSION}
RUN mkdir -p /app RUN mkdir -p /app
WORKDIR /app WORKDIR /app
ENV DEBIAN_FRONTEND=noninteractive ENV DEBIAN_FRONTEND=noninteractive
# Install Python and other dependencies # Install Python and other dependencies (plus gcc-13, selected later in build_pytorch).
RUN apt-get update -y \ RUN apt-get update -y \
&& apt-get install -y software-properties-common build-essential git curl sudo vim less libgfortran5 libopenmpi-dev libpci-dev liblzma-dev pkg-config libdrm-dev libnuma-dev kmod \ && apt-get install -y software-properties-common git curl sudo vim less libgfortran5 libopenmpi-dev libpci-dev liblzma-dev pkg-config \
&& for i in 1 2 3; do \ && for i in 1 2 3; do \
add-apt-repository -y ppa:deadsnakes/ppa && break || \ add-apt-repository -y ppa:deadsnakes/ppa && add-apt-repository -y ppa:ubuntu-toolchain-r/test && break || \
{ echo "Attempt $i failed, retrying in 5s..."; sleep 5; }; \ { echo "Attempt $i failed, retrying in 5s..."; sleep 5; }; \
done \ done \
&& apt-get update -y \ && apt-get update -y \
&& apt-get install -y python${PYTHON_VERSION} python${PYTHON_VERSION}-dev python${PYTHON_VERSION}-venv \ && apt-get install -y python${PYTHON_VERSION} python${PYTHON_VERSION}-dev python${PYTHON_VERSION}-venv \
python${PYTHON_VERSION}-lib2to3 python-is-python3 \ python${PYTHON_VERSION}-lib2to3 python-is-python3 gcc-13 g++-13 \
&& update-alternatives --install /usr/bin/python3 python3 /usr/bin/python${PYTHON_VERSION} 1 \ && update-alternatives --install /usr/bin/python3 python3 /usr/bin/python${PYTHON_VERSION} 1 \
&& update-alternatives --set python3 /usr/bin/python${PYTHON_VERSION} \ && update-alternatives --set python3 /usr/bin/python${PYTHON_VERSION} \
&& ln -sf /usr/bin/python${PYTHON_VERSION}-config /usr/bin/python3-config \ && ln -sf /usr/bin/python${PYTHON_VERSION}-config /usr/bin/python3-config \
&& curl -sS https://bootstrap.pypa.io/get-pip.py | python${PYTHON_VERSION} \ && curl -sS https://bootstrap.pypa.io/get-pip.py | python${PYTHON_VERSION} \
&& python3 --version && python3 -m pip --version && python3 --version && python3 -m pip --version
RUN pip install -U packaging 'cmake<4' ninja wheel 'setuptools<80' pybind11==3.0.4 Cython RUN pip install -U packaging 'cmake<4' ninja wheel 'setuptools<80' 'pybind11<3.1.0' Cython
RUN apt-get update && apt-get install -y libjpeg-dev libsox-dev libsox-fmt-all sox && rm -rf /var/lib/apt/lists/* RUN apt-get update && apt-get install -y libjpeg-dev libsox-dev libsox-fmt-all sox && rm -rf /var/lib/apt/lists/*
# Install ROCm SDK via TheRock
ARG ROCM_RELEASE_WHEELS_MULTIARCH_URL
ARG ROCM_NIGHTLY_WHEELS_MULTIARCH_URL
ARG ROCM_SDK_VERSION
RUN pip install --index-url ${ROCM_RELEASE_WHEELS_MULTIARCH_URL} \
--extra-index-url ${ROCM_NIGHTLY_WHEELS_MULTIARCH_URL} \
"rocm[libraries,devel]${ROCM_SDK_VERSION:+==${ROCM_SDK_VERSION}}" \
&& rocm-sdk init
# Install PyTorch wheels (nightly for latest torch, release as fallback)
ARG TORCH_VERSION
ARG TORCHAUDIO_VERSION
ARG TORCHVISION_VERSION
RUN pip install \
--index-url ${ROCM_RELEASE_WHEELS_MULTIARCH_URL} \
--extra-index-url ${ROCM_NIGHTLY_WHEELS_MULTIARCH_URL} \
"torch==${TORCH_VERSION}" \
"torchaudio==${TORCHAUDIO_VERSION}" \
"torchvision==${TORCHVISION_VERSION}"
# Install the AMD-compiled device kernels corresponding to PYTORCH_ROCM_ARCH:
# Possible values that can be in PYTORCH_ROCM_ARCH:
#
# gfx90a (MI210/250), gfx942 (MI300), gfx950 (MI350), gfx1100 (7900XTX),
# gfx1101 (7800XT), gfx1200 (9070XT), gfx1201 (9070), gfx1150/1151 (Strix APU)
# For each arch in PYTORCH_ROCM_ARCH, install the precompiled GPU code that
# PyTorch needs to run on that specific chip.
SHELL ["/bin/bash", "-c"]
RUN TORCH_VER=$(python3 -c "import torch; print(torch.__version__)") \
&& for arch in $(echo "${PYTORCH_ROCM_ARCH}" | tr ';' ' '); do \
echo "Installing device packages for ${arch}" \
&& pip install --index-url ${ROCM_RELEASE_WHEELS_MULTIARCH_URL} \
--extra-index-url ${ROCM_NIGHTLY_WHEELS_MULTIARCH_URL} \
"torch[device-${arch}]==${TORCH_VER}" \
"torchvision[device-${arch}]==${TORCHVISION_VERSION}" \
"rocm[device-${arch}]==${ROCM_SDK_VERSION}"; \
done
# Remove gfx90a xnack kpacks (ASAN artifacts that cause hipErrorInvalidImage on MI250)
RUN rm -f "${SITE_PACKAGES}/torch/.kpack/torch_gfx90a:xnack+.kpack" \
"${SITE_PACKAGES}/torch/.kpack/torch_gfx90a:xnack-.kpack"
RUN cd $SITE_PACKAGES/_rocm_sdk_core/share/amd_smi && pip install .
ENV PYTHONPATH=$SITE_PACKAGES/_rocm_sdk_core/share/amd_smi
# TheRock runtime fixes
RUN if [ -f "${SITE_PACKAGES}/rocm_sdk/__init__.py" ]; then \
sed -i 's/rtld_global: bool = True/rtld_global: bool = False/g' "${SITE_PACKAGES}/rocm_sdk/__init__.py"; \
fi \
&& if [ -f "${SITE_PACKAGES}/_rocm_sdk_core/lib/librocm_smi64.so.1" ] \
&& [ ! -e "${ROCM_PATH}/lib/librocm_smi64.so.7" ]; then \
ln -sfn "${SITE_PACKAGES}/_rocm_sdk_core/lib/librocm_smi64.so.1" "${ROCM_PATH}/lib/librocm_smi64.so.7"; \
fi \
&& printf '%s\n%s\n' "${ROCM_PATH}/lib" "${SITE_PACKAGES}/_rocm_sdk_core/lib" > /etc/ld.so.conf.d/rocm-pip.conf \
&& for f in ${SITE_PACKAGES}/_rocm_sdk_core/lib/libamdhip64.so*; do \
bn=$(basename "$f"); \
if [ -f "${ROCM_PATH}/lib/${bn}" ] && [ -f "$f" ] && [ ! -L "$f" ]; then \
rm -f "$f"; \
ln -sfn "${ROCM_PATH}/lib/${bn}" "$f"; \
fi; \
done \
&& for f in librocprofiler-sdk.so.1 librocprofiler-register.so.0 \
librocprofiler-sdk-roctx.so.1; do \
if [ -e "${SITE_PACKAGES}/_rocm_sdk_core/lib/${f}" ] \
&& [ -e "${SITE_PACKAGES}/_rocm_sdk_devel/lib/${f}" ]; then \
rm -f "${SITE_PACKAGES}/_rocm_sdk_core/lib/${f}"; \
ln -sfn "${SITE_PACKAGES}/_rocm_sdk_devel/lib/${f}" \
"${SITE_PACKAGES}/_rocm_sdk_core/lib/${f}"; \
fi; \
done \
&& ldconfig
# Install sccache if USE_SCCACHE is enabled (for release builds) # Install sccache if USE_SCCACHE is enabled (for release builds)
ARG USE_SCCACHE ARG USE_SCCACHE
ARG SCCACHE_DOWNLOAD_URL ARG SCCACHE_DOWNLOAD_URL
@@ -184,35 +94,141 @@ RUN if [ "$USE_SCCACHE" = "1" ]; then \
RUN if [ "$USE_SCCACHE" = "1" ]; then \ RUN if [ "$USE_SCCACHE" = "1" ]; then \
echo "Setting up sccache wrappers for HIP compilation..." \ echo "Setting up sccache wrappers for HIP compilation..." \
&& mkdir -p /opt/sccache-wrappers \ && mkdir -p /opt/sccache-wrappers \
&& printf '#!/bin/bash\nexec sccache %s/lib/llvm/bin/clang++ "$@"\n' "${ROCM_PATH}" > /opt/sccache-wrappers/clang++ \ && printf '#!/bin/bash\nexec sccache /opt/rocm/lib/llvm/bin/clang++ "$@"\n' > /opt/sccache-wrappers/clang++ \
&& chmod +x /opt/sccache-wrappers/clang++ \ && chmod +x /opt/sccache-wrappers/clang++ \
&& printf '#!/bin/bash\nexec sccache %s/lib/llvm/bin/clang "$@"\n' "${ROCM_PATH}" > /opt/sccache-wrappers/clang \ && printf '#!/bin/bash\nexec sccache /opt/rocm/lib/llvm/bin/clang "$@"\n' > /opt/sccache-wrappers/clang \
&& chmod +x /opt/sccache-wrappers/clang \ && chmod +x /opt/sccache-wrappers/clang \
&& echo "sccache wrappers created in /opt/sccache-wrappers"; \ && echo "sccache wrappers created in /opt/sccache-wrappers"; \
fi fi
# Set sccache environment variables only when USE_SCCACHE=1
# This prevents S3 config from leaking into images when sccache is not used
ARG USE_SCCACHE
ENV SCCACHE_BUCKET=${USE_SCCACHE:+${SCCACHE_BUCKET_NAME}} ENV SCCACHE_BUCKET=${USE_SCCACHE:+${SCCACHE_BUCKET_NAME}}
ENV SCCACHE_REGION=${USE_SCCACHE:+${SCCACHE_REGION_NAME}} ENV SCCACHE_REGION=${USE_SCCACHE:+${SCCACHE_REGION_NAME}}
ENV SCCACHE_S3_NO_CREDENTIALS=${USE_SCCACHE:+${SCCACHE_S3_NO_CREDENTIALS}} ENV SCCACHE_S3_NO_CREDENTIALS=${USE_SCCACHE:+${SCCACHE_S3_NO_CREDENTIALS}}
ENV SCCACHE_IDLE_TIMEOUT=${USE_SCCACHE:+0} ENV SCCACHE_IDLE_TIMEOUT=${USE_SCCACHE:+0}
###
### Triton Build
###
FROM base AS build_triton
ARG TRITON_BRANCH
ARG TRITON_REPO
RUN git clone ${TRITON_REPO}
RUN cd triton \
&& git checkout ${TRITON_BRANCH} \
&& git config --global user.email "you@example.com" && git config --global user.name "Your Name" \
&& if [ ! -f setup.py ]; then cd python; fi \
&& MAX_JOBS=16 python3 setup.py bdist_wheel --dist-dir=dist \
&& mkdir -p /app/install && cp dist/*.whl /app/install
###
### AMD SMI Build
###
FROM base AS build_amdsmi
RUN cd /opt/rocm/share/amd_smi \
&& pip wheel . --wheel-dir=dist
RUN mkdir -p /app/install && cp /opt/rocm/share/amd_smi/dist/*.whl /app/install
###
### Pytorch build
###
FROM base AS build_pytorch
ARG PYTORCH_BRANCH
ARG PYTORCH_REPO
ARG USE_SCCACHE
RUN apt-get update && apt-get install -y pkg-config liblzma-dev
# torch 2.12 emits float->__bf16 libcalls (__truncsfbf2) absent from gcc-11 libgcc; build with gcc-13.
RUN update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-13 130 --slave /usr/bin/g++ g++ /usr/bin/g++-13
RUN git clone ${PYTORCH_REPO} pytorch
RUN cd pytorch && git checkout ${PYTORCH_BRANCH}
RUN cd pytorch \
&& pip install -r requirements.txt && git submodule update --init --recursive
RUN cd pytorch && python3 tools/amd_build/build_amd.py \
&& if [ "$USE_SCCACHE" = "1" ]; then \
export HIP_CLANG_PATH=/opt/sccache-wrappers \
&& export CMAKE_C_COMPILER_LAUNCHER=sccache \
&& export CMAKE_CXX_COMPILER_LAUNCHER=sccache \
&& sccache --show-stats; \
fi \
&& CMAKE_PREFIX_PATH=$(python3 -c 'import sys; print(sys.prefix)') python3 setup.py bdist_wheel --dist-dir=dist \
&& if [ "$USE_SCCACHE" = "1" ]; then sccache --show-stats; fi
RUN mkdir -p /app/install && cp /app/pytorch/dist/*.whl /app/install
###
### Shared PyTorch runtime for dependent builds
###
FROM base AS build_pytorch_runtime
RUN update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-13 130 --slave /usr/bin/g++ g++ /usr/bin/g++-13
RUN --mount=type=bind,from=build_pytorch,src=/app/install/,target=/install,ro \
pip install /install/*.whl
###
### TorchVision build
###
FROM build_pytorch_runtime AS build_torchvision
ARG PYTORCH_VISION_BRANCH
ARG PYTORCH_VISION_REPO
ARG USE_SCCACHE
RUN git clone ${PYTORCH_VISION_REPO} vision
RUN cd vision && git checkout ${PYTORCH_VISION_BRANCH} \
&& if [ "$USE_SCCACHE" = "1" ]; then \
export HIP_CLANG_PATH=/opt/sccache-wrappers \
&& export CMAKE_C_COMPILER_LAUNCHER=sccache \
&& export CMAKE_CXX_COMPILER_LAUNCHER=sccache; \
fi \
&& python3 setup.py bdist_wheel --dist-dir=dist \
&& if [ "$USE_SCCACHE" = "1" ]; then sccache --show-stats; fi
RUN mkdir -p /app/install && cp /app/vision/dist/*.whl /app/install
###
### TorchAudio build
###
FROM build_pytorch_runtime AS build_torchaudio
ARG PYTORCH_AUDIO_BRANCH
ARG PYTORCH_AUDIO_REPO
ARG USE_SCCACHE
RUN git clone ${PYTORCH_AUDIO_REPO} audio
RUN cd audio && git checkout ${PYTORCH_AUDIO_BRANCH} \
&& git submodule update --init --recursive
RUN cd audio && pip install -r requirements.txt
RUN cd audio \
&& if [ "$USE_SCCACHE" = "1" ]; then \
export HIP_CLANG_PATH=/opt/sccache-wrappers \
&& export CMAKE_C_COMPILER_LAUNCHER=sccache \
&& export CMAKE_CXX_COMPILER_LAUNCHER=sccache; \
fi \
&& python3 setup.py bdist_wheel --dist-dir=dist \
&& if [ "$USE_SCCACHE" = "1" ]; then sccache --show-stats; fi
RUN mkdir -p /app/install && cp /app/audio/dist/*.whl /app/install
### ###
### MORI Build ### MORI Build
### ###
FROM base AS build_mori FROM base AS build_mori
ARG MORI_BRANCH ARG MORI_BRANCH
ARG MORI_REPO ARG MORI_REPO
RUN --mount=type=bind,from=build_pytorch,src=/app/install/,target=/install \
pip install /install/*.whl
RUN git clone ${MORI_REPO} RUN git clone ${MORI_REPO}
RUN cd mori \ RUN cd mori \
&& git checkout ${MORI_BRANCH} \ && git checkout ${MORI_BRANCH} \
&& git submodule update --init --recursive \ && git submodule update --init --recursive \
&& ln -sfn ${ROCM_PATH} /opt/rocm \ && python3 setup.py bdist_wheel --dist-dir=dist && ls /app/mori/dist/*.whl
&& pip install "setuptools_scm>=6.2" \
&& CMAKE_PREFIX_PATH="${ROCM_PATH}:${ROCM_PATH}/lib/rocm_sysdeps" \
python3 setup.py bdist_wheel --dist-dir=dist && ls /app/mori/dist/*.whl
RUN mkdir -p /app/install && cp /app/mori/dist/*.whl /app/install RUN mkdir -p /app/install && cp /app/mori/dist/*.whl /app/install
### ###
### FlashAttention Build ### FlashAttention Build
### ###
@@ -220,6 +236,8 @@ FROM base AS build_fa
ARG FA_BRANCH ARG FA_BRANCH
ARG FA_REPO ARG FA_REPO
ARG USE_SCCACHE ARG USE_SCCACHE
RUN --mount=type=bind,from=build_pytorch,src=/app/install/,target=/install \
pip install /install/*.whl
RUN git clone ${FA_REPO} RUN git clone ${FA_REPO}
RUN cd flash-attention \ RUN cd flash-attention \
&& git checkout ${FA_BRANCH} \ && git checkout ${FA_BRANCH} \
@@ -240,12 +258,12 @@ FROM base AS build_aiter
ARG AITER_BRANCH ARG AITER_BRANCH
ARG AITER_REPO ARG AITER_REPO
ARG USE_SCCACHE ARG USE_SCCACHE
RUN --mount=type=bind,from=build_pytorch,src=/app/install/,target=/install \
pip install /install/*.whl
RUN git clone --recursive --branch ${AITER_BRANCH} ${AITER_REPO} RUN git clone --recursive --branch ${AITER_BRANCH} ${AITER_REPO}
RUN cd aiter \ RUN cd aiter \
&& git submodule update --init --recursive \ && git submodule update --init --recursive \
&& pip install -r requirements.txt && pip install -r requirements.txt
# AITER_USE_SYSTEM_TRITON keeps setup.py from resolving its own triton, which would
# displace the pinned one already installed in base.
RUN pip install pyyaml && cd aiter \ RUN pip install pyyaml && cd aiter \
&& if [ "$USE_SCCACHE" = "1" ]; then \ && if [ "$USE_SCCACHE" = "1" ]; then \
export HIP_CLANG_PATH=/opt/sccache-wrappers \ export HIP_CLANG_PATH=/opt/sccache-wrappers \
@@ -258,30 +276,58 @@ RUN mkdir -p /app/install && cp /app/aiter/dist/*.whl /app/install
### ###
### Triton Build ### Final Build
### ###
FROM base AS build_triton
ARG TRITON_BRANCH
ARG TRITON_REPO
RUN git clone ${TRITON_REPO}
RUN cd triton \
&& git checkout ${TRITON_BRANCH} \
&& if [ ! -f setup.py ]; then cd python; fi \
&& python3 setup.py bdist_wheel --dist-dir=dist \
&& mkdir -p /app/install && cp dist/*.whl /app/install
# Wheel release stage -
# only includes dependencies used by wheel release pipeline
FROM base AS debs_wheel_release
RUN mkdir /app/debs
RUN --mount=type=bind,from=build_triton,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_fa,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_amdsmi,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_pytorch,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_torchvision,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_torchaudio,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_aiter,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
# Full debs stage - includes Mori (used by Docker releases)
FROM base AS debs
RUN mkdir /app/debs
RUN --mount=type=bind,from=build_triton,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_fa,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_amdsmi,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_pytorch,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_torchvision,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_torchaudio,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_aiter,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_mori,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
### ###
### ROCr + CLR build ### ROCr + CLR build
### ###
# Brings in multi-stream perf improvements; drop after these arrive in an official release # Pull in updates to ROCr + clr from ROCm 7.2.4 and beyond.
FROM base AS build_rocm_runtime FROM base AS build_rocm_runtime
ARG ROCM_SYSTEMS_REPO ARG ROCM_SYSTEMS_REPO
ARG ROCM_RUNTIME_COMMIT ARG ROCM_RUNTIME_COMMIT
# libgl-dev: rocclr does find_package(OpenGL REQUIRED) unconditionally and the wheel image has
# no GL headers. The link must not pick up a libGL DT_NEEDED, which is asserted after the build.
RUN apt-get update \ RUN apt-get update \
&& apt-get install -y --no-install-recommends g++ libdw-dev xxd libgl-dev libglx-dev \ && apt-get install -y --no-install-recommends g++ \
libelf-dev libdrm-dev libnuma-dev libdw-dev xxd \
&& rm -rf /var/lib/apt/lists/* && rm -rf /var/lib/apt/lists/*
RUN pip install --no-cache-dir CppHeaderParser RUN pip install --no-cache-dir CppHeaderParser
RUN git init -q /src && cd /src \ RUN git init -q /src && cd /src \
@@ -293,17 +339,20 @@ RUN git init -q /src && cd /src \
&& git sparse-checkout set projects/rocr-runtime projects/clr projects/hip shared cmake \ && git sparse-checkout set projects/rocr-runtime projects/clr projects/hip shared cmake \
&& git fetch --filter=blob:none --depth 1 origin ${ROCM_RUNTIME_COMMIT} \ && git fetch --filter=blob:none --depth 1 origin ${ROCM_RUNTIME_COMMIT} \
&& git checkout -q FETCH_HEAD && git checkout -q FETCH_HEAD
# libelf is taken from the SDK's own vendored sysdeps rather than apt -- the wheel image has no # ROCr's trap_handler and blit_shaders call find_package(Clang/LLVM REQUIRED) only to get
# /usr/include/libelf.h -- so ROCr links against the copy the wheel itself was built against. # IMPORTED executables. The ROCm dev image ships the binaries but no CMake package config, so
ENV SYSDEPS="$SDK_DEV/lib/rocm_sysdeps" # supply one pointing at them.
# ROCr first, and installed into $SDK_DEV, because the clr build below needs its headers. RUN mkdir -p /opt/rocm-cmake-shim \
&& printf 'if(NOT TARGET clang)\n add_executable(clang IMPORTED GLOBAL)\n set_target_properties(clang PROPERTIES IMPORTED_LOCATION "/opt/rocm/llvm/bin/clang")\nendif()\nset(Clang_PACKAGE_VERSION "rocm-image-shim")\n' > /opt/rocm-cmake-shim/ClangConfig.cmake \
&& printf 'if(NOT TARGET llvm-objcopy)\n add_executable(llvm-objcopy IMPORTED GLOBAL)\n set_target_properties(llvm-objcopy PROPERTIES IMPORTED_LOCATION "/opt/rocm/llvm/bin/llvm-objcopy")\nendif()\nset(LLVM_FOUND TRUE)\nset(LLVM_PACKAGE_VERSION "rocm-image-shim")\n' > /opt/rocm-cmake-shim/LLVMConfig.cmake
# ROCr first, and installed into /opt/rocm, because the clr build below needs its headers
RUN cd /src/projects/rocr-runtime \ RUN cd /src/projects/rocr-runtime \
&& cmake -B build -G Ninja \ && cmake -B build -G Ninja \
-DCMAKE_BUILD_TYPE=Release \ -DCMAKE_BUILD_TYPE=Release \
-DCMAKE_PREFIX_PATH="$SDK_DEV;$SYSDEPS" \ -DCMAKE_PREFIX_PATH=/opt/rocm \
-DCMAKE_INSTALL_PREFIX="$SDK_DEV" \ -DCMAKE_INSTALL_PREFIX=/opt/rocm \
-DLIBELF_INCLUDE_DIRS="$SYSDEPS/include" \ -DClang_DIR=/opt/rocm-cmake-shim \
-DLIBELF_LIBRARIES="$SYSDEPS/lib/libelf.so" \ -DLLVM_DIR=/opt/rocm-cmake-shim \
&& cmake --build build --parallel 16 \ && cmake --build build --parallel 16 \
&& cmake --install build \ && cmake --install build \
&& cmake --install build --prefix /rocr-install --strip && cmake --install build --prefix /rocr-install --strip
@@ -312,12 +361,12 @@ RUN cd /src/projects/clr \
-DCMAKE_BUILD_TYPE=Release \ -DCMAKE_BUILD_TYPE=Release \
-DCLR_BUILD_HIP=ON \ -DCLR_BUILD_HIP=ON \
-DCLR_BUILD_OCL=OFF \ -DCLR_BUILD_OCL=OFF \
-DROCM_KPACK_ENABLED=ON \
-DHIP_COMMON_DIR=/src/projects/hip \ -DHIP_COMMON_DIR=/src/projects/hip \
-DROCM_PATH="$SDK_DEV" \ -DROCM_PATH=/opt/rocm \
-DCMAKE_PREFIX_PATH="$SDK_DEV;$SYSDEPS" \ -DCMAKE_PREFIX_PATH=/opt/rocm \
-DCMAKE_INSTALL_PREFIX="$SDK_DEV" \ -DCMAKE_INSTALL_PREFIX=/opt/rocm \
-DHIP_LLVM_ROOT="$SDK_DEV/lib/llvm" \ -DLLVM_DIR=/opt/rocm-cmake-shim \
-DHIP_LLVM_ROOT=/opt/rocm/llvm \
&& cmake --build build --parallel 16 \ && cmake --build build --parallel 16 \
&& cmake --install build --prefix /clr-install --strip && cmake --install build --prefix /clr-install --strip
RUN mkdir -p /staging/lib \ RUN mkdir -p /staging/lib \
@@ -325,115 +374,82 @@ RUN mkdir -p /staging/lib \
&& cp -P /clr-install/lib/libamdhip64.so* /staging/lib/ \ && cp -P /clr-install/lib/libamdhip64.so* /staging/lib/ \
&& ls -l /staging/lib && ls -l /staging/lib
# rocprofiler-sdk 1.3.2 fixes torch 2.12 profiler on ROCm 7.2.3
### FROM base AS build_rocprofiler_sdk
### Final Build ARG ROCM_SYSTEMS_REPO
### ARG ROCPROFILER_SDK_COMMIT
ARG USE_SCCACHE
# Wheel release stage - RUN apt-get update \
# only includes dependencies used by wheel release pipeline && apt-get install -y --no-install-recommends libdw-dev libelf-dev libsqlite3-dev libzstd-dev \
FROM base AS debs_wheel_release && rm -rf /var/lib/apt/lists/*
RUN mkdir /app/debs RUN git init -q /src && cd /src \
RUN --mount=type=bind,from=build_triton,src=/app/install/,target=/install \ && git remote add origin ${ROCM_SYSTEMS_REPO} \
cp /install/*.whl /app/debs && git config core.sparseCheckout true \
RUN --mount=type=bind,from=build_fa,src=/app/install/,target=/install \ && git config remote.origin.promisor true \
cp /install/*.whl /app/debs && git config remote.origin.partialclonefilter blob:none \
RUN --mount=type=bind,from=build_aiter,src=/app/install/,target=/install \ && git sparse-checkout init --cone \
cp /install/*.whl /app/debs && git sparse-checkout set projects/rocprofiler-sdk shared cmake \
# The wheel index also serves a torch repacked to require our triton instead of && git fetch --filter=blob:none --depth 1 origin ${ROCPROFILER_SDK_COMMIT} \
# the one TheRock built it with, and amdsmi, which TheRock ships only as sources && git checkout -q FETCH_HEAD
# inside rocm-sdk-core. # Patch the ROCm 7.2.3 build of rocprofiler-sdk to include queue-interposition fixes that are
ARG TORCH_VERSION # absent from the pinned 7.14 build. This resolves hangs on torch.profiler on ROCm.
ARG ROCM_RELEASE_WHEELS_MULTIARCH_URL # TODO: Drop these once ${ROCPROFILER_SDK_COMMIT} is bumped to a build that already
ARG ROCM_NIGHTLY_WHEELS_MULTIARCH_URL # contains them, such as ROCm 10.0. See https://github.com/vllm-project/vllm/issues/54087.
COPY tools/vllm-rocm/therock_wheels.py /tmp/therock_wheels.py ARG ROCPROFILER_SDK_PR_7796=fae7fd57ffb8a76e811d6410b8a0cee1e94d710a
# regex and pybase64 aren't installed here; fall back to the stdlib modules. ARG ROCPROFILER_SDK_PR_7924=f8db22adbe17686a9b05ae03367dd1ec73011653
RUN sed -i -e 's/import regex as re/import re/' \ RUN cd /src \
-e 's/import pybase64 as base64/import base64/' /tmp/therock_wheels.py \ && HSA_PATHS='projects/rocprofiler-sdk/source/lib/rocprofiler-sdk/hsa/*' \
&& pip download --no-deps --dest /tmp/torch-dl \ && curl -fsSL -o /tmp/qi-7796.patch "${ROCM_SYSTEMS_REPO%.git}/commit/${ROCPROFILER_SDK_PR_7796}.patch" \
--index-url ${ROCM_RELEASE_WHEELS_MULTIARCH_URL} \ && curl -fsSL -o /tmp/qi-7924.patch "${ROCM_SYSTEMS_REPO%.git}/commit/${ROCPROFILER_SDK_PR_7924}.patch" \
--extra-index-url ${ROCM_NIGHTLY_WHEELS_MULTIARCH_URL} \ && git apply -p1 --include="$HSA_PATHS" --verbose /tmp/qi-7796.patch \
"torch==${TORCH_VERSION}" \ && git apply -p1 --include="$HSA_PATHS" --verbose /tmp/qi-7924.patch
&& TRITON_VER=$(basename /app/debs/triton-*.whl | cut -d- -f2) \ RUN cd /src/projects/rocprofiler-sdk \
&& python3 /tmp/therock_wheels.py repack-torch /tmp/torch-dl/torch-*.whl \ && if [ "$USE_SCCACHE" = "1" ]; then \
--triton-version "${TRITON_VER}" --out-dir /app/debs \ export CMAKE_C_COMPILER_LAUNCHER=sccache \
&& python3 /tmp/therock_wheels.py patch-amdsmi "${SDK_CORE}/share/amd_smi" \ && export CMAKE_CXX_COMPILER_LAUNCHER=sccache; \
--out-dir /tmp/amd_smi \ fi \
&& pip wheel --no-deps --wheel-dir /app/debs /tmp/amd_smi \ && cmake -B build -G Ninja \
&& rm -rf /tmp/torch-dl /tmp/amd_smi /tmp/therock_wheels.py \ -DCMAKE_BUILD_TYPE=Release \
&& ls -l /app/debs -DCMAKE_PREFIX_PATH=/opt/rocm \
-DCMAKE_INSTALL_PREFIX=/opt/rocm \
FROM base AS debs -DROCPROFILER_BUILD_TESTS=OFF \
RUN mkdir /app/debs -DROCPROFILER_BUILD_SAMPLES=OFF \
RUN --mount=type=bind,from=build_triton,src=/app/install/,target=/install \ -DROCPROFILER_BUILD_DOCS=OFF \
cp /install/*.whl /app/debs && cmake --build build --parallel "$(nproc)" \
RUN --mount=type=bind,from=build_fa,src=/app/install/,target=/install \ && cmake --install build --prefix /staging --strip
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_aiter,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_mori,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
FROM base AS final FROM base AS final
ARG TRITON_VERSION RUN --mount=type=bind,from=debs,src=/app/debs,target=/install \
pip install /install/*.whl
# torch 2.12 implicitly attaches rocprofiler-sdk; its queue interposition hangs at HSA COPY --from=build_rocprofiler_sdk /staging/ /opt/rocm/
# teardown, so disable it. TODO: drop once ROCm's stock rocprofiler carries the fix.
ENV ROCPROFILER_QUEUE_INTERPOSITION=0 ENV ROCPROFILER_QUEUE_INTERPOSITION=0
COPY --from=build_rocm_runtime /staging/lib/ ${SDK_DEV}/lib/ COPY --from=build_rocm_runtime /staging/ /opt/rocm/
# Hard link the patched library onto every stock file name, in both SDK trees, so that the # Hard link the patched library onto every stock file name so that both the
# SONAME lookup and the hard coded paths all resolve to it. Hard links rather than symlinks # SONAME lookup and the hard coded paths resolve to it.
# because ldconfig rewrites SONAME symlinks but cannot re-point an inode. $SDK_CORE keeps its
# own libhsa-runtime64 copy, which is on LD_LIBRARY_PATH and in ld.so.conf; relinking it is
# belt and braces rather than strictly required, but it leaves no stock ROCr in the image.
RUN set -eux; \ RUN set -eux; \
cd "$SDK_DEV/lib"; \ cd /opt/rocm/lib; \
ours_hsa="$(readlink -f libhsa-runtime64.so.1)"; \ ours_hsa="$(readlink -f libhsa-runtime64.so.1)"; \
ours_hip="$(readlink -f libamdhip64.so.7)"; \ ours_hip="$(readlink -f libamdhip64.so.7)"; \
for f in "$SDK_DEV"/lib/libhsa-runtime64.so.1* "$SDK_CORE"/lib/libhsa-runtime64.so.1* \ for f in libhsa-runtime64.so.1.*; do \
"$SDK_DEV"/lib/libamdhip64.so.7* "$SDK_CORE"/lib/libamdhip64.so.7*; do \ [ "$f" = "${ours_hsa##*/}" ] || ln -f "$ours_hsa" "$f"; \
[ -f "$f" ] && [ ! -L "$f" ] || continue; \ done; \
case "${f##*/}" in libhsa*) ours="$ours_hsa";; *) ours="$ours_hip";; esac; \ for f in libamdhip64.so.7.*; do \
[ "$f" -ef "$ours" ] || ln -f "$ours" "$f"; \ [ "$f" = "${ours_hip##*/}" ] || ln -f "$ours_hip" "$f"; \
done done
RUN ldconfig RUN ldconfig
# This is to prevent resolvers from checking METADATA and RECORD and clobbering
# the Triton source build we installed. Constraints files are unable to help
# with this. However, these entries are optional and can be blanked which
# solves the problem.
RUN TORCH_DIST_INFO=$(ls -d ${SITE_PACKAGES}/torch-*.dist-info) \
&& grep -q '^Requires-Dist: triton' "$TORCH_DIST_INFO/METADATA" \
&& sed -i '/^Requires-Dist: triton/d' "$TORCH_DIST_INFO/METADATA" \
&& sed -i "s|^\(${TORCH_DIST_INFO##*/}/METADATA\),.*|\1,,|" "$TORCH_DIST_INFO/RECORD"
RUN --mount=type=bind,from=debs,src=/app/debs,target=/install \
pip install /install/*.whl \
&& ! pip check 2>&1 | grep -iE '^(torch|triton) ' \
&& test "$(pip show triton | awk '/^Version:/ {print $2}')" = "${TRITON_VERSION}" \
# Create a constraints file to protect the versions we need, especially
# triton, requirements will try to reinstall repeatedly.
&& pip list --format=freeze \
| grep -E '^(torch|triton|torchaudio|torchvision|pybind11)==' > /etc/rocm-constraints.txt \
&& echo "ROCm constraints:" && cat /etc/rocm-constraints.txt \
&& test "$(wc -l < /etc/rocm-constraints.txt)" -eq 5 \
&& grep -q '^pybind11==3\.0\.' /etc/rocm-constraints.txt \
&& python3 -c "import torch; assert torch.version.hip, f'non-ROCm torch: {torch.__version__}'; print(f'torch={torch.__version__}')" \
&& python3 -c "import triton; print(f'triton={triton.__version__}')" \
&& pip list 2>/dev/null | grep -iE '^(torch|triton) '
ARG BASE_IMAGE ARG BASE_IMAGE
ARG PYTHON_VERSION
ARG ROCM_SDK_VERSION
ARG TORCH_VERSION
ARG TORCHAUDIO_VERSION
ARG TORCHVISION_VERSION
ARG PYTORCH_ROCM_ARCH
ARG AITER_ROCM_ARCH
ARG TRITON_BRANCH ARG TRITON_BRANCH
ARG TRITON_REPO ARG TRITON_REPO
ARG PYTORCH_BRANCH
ARG PYTORCH_VISION_BRANCH
ARG PYTORCH_REPO
ARG PYTORCH_VISION_REPO
ARG PYTORCH_AUDIO_BRANCH
ARG PYTORCH_AUDIO_REPO
ARG FA_BRANCH ARG FA_BRANCH
ARG FA_REPO ARG FA_REPO
ARG AITER_BRANCH ARG AITER_BRANCH
@@ -441,36 +457,23 @@ ARG AITER_REPO
ARG MORI_BRANCH ARG MORI_BRANCH
ARG MORI_REPO ARG MORI_REPO
ARG ROCM_SYSTEMS_REPO ARG ROCM_SYSTEMS_REPO
ARG ROCPROFILER_SDK_COMMIT
ARG ROCM_RUNTIME_COMMIT ARG ROCM_RUNTIME_COMMIT
ARG TRITON_VERSION
# Record hashes for Triton and pybind11 so we can verify later.
RUN echo "BASE_IMAGE: ${BASE_IMAGE}" > /app/versions.txt \ RUN echo "BASE_IMAGE: ${BASE_IMAGE}" > /app/versions.txt \
&& echo "PYTHON_VERSION: ${PYTHON_VERSION}" >> /app/versions.txt \ && echo "ROCM_SYSTEMS_REPO: ${ROCM_SYSTEMS_REPO}" >> /app/versions.txt \
&& echo "ROCM_SDK_VERSION: ${ROCM_SDK_VERSION}" >> /app/versions.txt \ && echo "ROCPROFILER_SDK_COMMIT: ${ROCPROFILER_SDK_COMMIT}" >> /app/versions.txt \
&& echo "TORCH_VERSION: ${TORCH_VERSION}" >> /app/versions.txt \ && echo "ROCM_RUNTIME_COMMIT: ${ROCM_RUNTIME_COMMIT}" >> /app/versions.txt \
&& echo "TORCHAUDIO_VERSION: ${TORCHAUDIO_VERSION}" >> /app/versions.txt \
&& echo "TORCHVISION_VERSION: ${TORCHVISION_VERSION}" >> /app/versions.txt \
&& echo "PYTORCH_ROCM_ARCH: ${PYTORCH_ROCM_ARCH}" >> /app/versions.txt \
&& echo "AITER_ROCM_ARCH: ${AITER_ROCM_ARCH}" >> /app/versions.txt \
&& echo "TRITON_BRANCH: ${TRITON_BRANCH}" >> /app/versions.txt \ && echo "TRITON_BRANCH: ${TRITON_BRANCH}" >> /app/versions.txt \
&& echo "TRITON_REPO: ${TRITON_REPO}" >> /app/versions.txt \ && echo "TRITON_REPO: ${TRITON_REPO}" >> /app/versions.txt \
&& echo "PYTORCH_BRANCH: ${PYTORCH_BRANCH}" >> /app/versions.txt \
&& echo "PYTORCH_VISION_BRANCH: ${PYTORCH_VISION_BRANCH}" >> /app/versions.txt \
&& echo "PYTORCH_REPO: ${PYTORCH_REPO}" >> /app/versions.txt \
&& echo "PYTORCH_VISION_REPO: ${PYTORCH_VISION_REPO}" >> /app/versions.txt \
&& echo "PYTORCH_AUDIO_BRANCH: ${PYTORCH_AUDIO_BRANCH}" >> /app/versions.txt \
&& echo "PYTORCH_AUDIO_REPO: ${PYTORCH_AUDIO_REPO}" >> /app/versions.txt \
&& echo "FA_BRANCH: ${FA_BRANCH}" >> /app/versions.txt \ && echo "FA_BRANCH: ${FA_BRANCH}" >> /app/versions.txt \
&& echo "FA_REPO: ${FA_REPO}" >> /app/versions.txt \ && echo "FA_REPO: ${FA_REPO}" >> /app/versions.txt \
&& echo "AITER_BRANCH: ${AITER_BRANCH}" >> /app/versions.txt \ && echo "AITER_BRANCH: ${AITER_BRANCH}" >> /app/versions.txt \
&& echo "AITER_REPO: ${AITER_REPO}" >> /app/versions.txt \ && echo "AITER_REPO: ${AITER_REPO}" >> /app/versions.txt \
&& echo "MORI_BRANCH: ${MORI_BRANCH}" >> /app/versions.txt \ && echo "MORI_BRANCH: ${MORI_BRANCH}" >> /app/versions.txt \
&& echo "MORI_REPO: ${MORI_REPO}" >> /app/versions.txt \ && echo "MORI_REPO: ${MORI_REPO}" >> /app/versions.txt
&& echo "ROCM_SYSTEMS_REPO: ${ROCM_SYSTEMS_REPO}" >> /app/versions.txt \
&& echo "ROCM_RUNTIME_COMMIT: ${ROCM_RUNTIME_COMMIT}" >> /app/versions.txt \
# Fingerprint our triton so Dockerfile.rocm can prove it survived the build.
&& echo "TRITON_VERSION: $(pip show triton | awk '/^Version:/ {print $2}')" >> /app/versions.txt \
&& echo "TRITON_SHA256: $(find $(python3 -c 'import triton, os; print(os.path.dirname(triton.__file__))') \
-type f \( -name '*.py' -o -name '*.so' \) | sort | xargs sha256sum | sha256sum | cut -d' ' -f1)" \
>> /app/versions.txt \
&& echo "PYBIND11_VERSION: $(pip show pybind11 | awk '/^Version:/ {print $2}')" >> /app/versions.txt \
&& echo "PYBIND11_INTERNALS_VERSION: $(grep -rhE '^#[[:space:]]*define PYBIND11_INTERNALS_VERSION' \
$(python3 -c 'import pybind11; print(pybind11.get_include())') | awk '{print $NF}')" \
>> /app/versions.txt \
&& echo "PYBIND11_SHA256: $(find $(python3 -c 'import pybind11; print(pybind11.get_include())') \
-type f -name '*.h' | sort | xargs sha256sum | sha256sum | cut -d' ' -f1)" \
>> /app/versions.txt
+29 -5
View File
@@ -83,6 +83,10 @@ variable "UCX_BRANCH" {
default = "" default = ""
} }
variable "ROCSHMEM_BRANCH" {
default = ""
}
variable "DEEPEP_BRANCH" { variable "DEEPEP_BRANCH" {
default = "" default = ""
} }
@@ -91,6 +95,10 @@ variable "NIXL_CACHE_KEY" {
default = "" default = ""
} }
variable "ROCSHMEM_CACHE_KEY" {
default = ""
}
variable "DEEPEP_CACHE_KEY" { variable "DEEPEP_CACHE_KEY" {
default = "" default = ""
} }
@@ -228,7 +236,7 @@ function "get_cache_to_rocm_rust" {
]) ])
} }
# Cache functions for upstream dependency stages (NIXL/UCX, DeepEP). # Cache functions for upstream dependency stages (NIXL/UCX, ROCShmem, DeepEP).
# These stages are pinned to specific upstream commit hashes, so cache keys use # These stages are pinned to specific upstream commit hashes, so cache keys use
# those hashes rather than the Buildkite commit. This means the cache persists # those hashes rather than the Buildkite commit. This means the cache persists
# across all vLLM commits as long as the upstream dependency pins don't change. # across all vLLM commits as long as the upstream dependency pins don't change.
@@ -237,7 +245,8 @@ function "get_cache_from_rocm_deps" {
params = [] params = []
result = compact([ result = compact([
NIXL_CACHE_KEY != "" ? "type=registry,ref=${DOCKERHUB_CACHE_REPO}:nixl-rocm-${NIXL_CACHE_KEY}" : (NIXL_BRANCH != "" ? "type=registry,ref=${DOCKERHUB_CACHE_REPO}:nixl-rocm-${NIXL_BRANCH}-ucx-${UCX_BRANCH}" : ""), NIXL_CACHE_KEY != "" ? "type=registry,ref=${DOCKERHUB_CACHE_REPO}:nixl-rocm-${NIXL_CACHE_KEY}" : (NIXL_BRANCH != "" ? "type=registry,ref=${DOCKERHUB_CACHE_REPO}:nixl-rocm-${NIXL_BRANCH}-ucx-${UCX_BRANCH}" : ""),
DEEPEP_CACHE_KEY != "" ? "type=registry,ref=${DOCKERHUB_CACHE_REPO}:deepep-rocm-${DEEPEP_CACHE_KEY}" : (DEEPEP_BRANCH != "" ? "type=registry,ref=${DOCKERHUB_CACHE_REPO}:deepep-rocm-${DEEPEP_BRANCH}" : ""), ROCSHMEM_CACHE_KEY != "" ? "type=registry,ref=${DOCKERHUB_CACHE_REPO}:rocshmem-rocm-${ROCSHMEM_CACHE_KEY}" : (ROCSHMEM_BRANCH != "" ? "type=registry,ref=${DOCKERHUB_CACHE_REPO}:rocshmem-rocm-${ROCSHMEM_BRANCH}" : ""),
DEEPEP_CACHE_KEY != "" ? "type=registry,ref=${DOCKERHUB_CACHE_REPO}:deepep-rocm-${DEEPEP_CACHE_KEY}" : (DEEPEP_BRANCH != "" ? "type=registry,ref=${DOCKERHUB_CACHE_REPO}:deepep-rocm-${DEEPEP_BRANCH}-rocshmem-${ROCSHMEM_BRANCH}" : ""),
]) ])
} }
@@ -248,10 +257,17 @@ function "get_cache_to_rocm_nixl" {
]) ])
} }
function "get_cache_to_rocm_rocshmem" {
params = []
result = compact([
ROCSHMEM_CACHE_KEY != "" ? "type=registry,ref=${DOCKERHUB_CACHE_REPO}:rocshmem-rocm-${ROCSHMEM_CACHE_KEY},mode=min" : (ROCSHMEM_BRANCH != "" ? "type=registry,ref=${DOCKERHUB_CACHE_REPO}:rocshmem-rocm-${ROCSHMEM_BRANCH},mode=min" : ""),
])
}
function "get_cache_to_rocm_deepep" { function "get_cache_to_rocm_deepep" {
params = [] params = []
result = compact([ result = compact([
DEEPEP_CACHE_KEY != "" ? "type=registry,ref=${DOCKERHUB_CACHE_REPO}:deepep-rocm-${DEEPEP_CACHE_KEY},mode=min" : (DEEPEP_BRANCH != "" ? "type=registry,ref=${DOCKERHUB_CACHE_REPO}:deepep-rocm-${DEEPEP_BRANCH},mode=min" : ""), DEEPEP_CACHE_KEY != "" ? "type=registry,ref=${DOCKERHUB_CACHE_REPO}:deepep-rocm-${DEEPEP_CACHE_KEY},mode=min" : (DEEPEP_BRANCH != "" ? "type=registry,ref=${DOCKERHUB_CACHE_REPO}:deepep-rocm-${DEEPEP_BRANCH}-rocshmem-${ROCSHMEM_BRANCH},mode=min" : ""),
]) ])
} }
@@ -369,6 +385,14 @@ target "nixl-rocm-ci" {
output = ["type=cacheonly"] output = ["type=cacheonly"]
} }
target "rocshmem-rocm-ci" {
inherits = ["_common-rocm", "_ci-rocm"]
target = "build_rocshmem"
cache-from = get_cache_from_rocm_deps()
cache-to = get_cache_to_rocm_rocshmem()
output = ["type=cacheonly"]
}
target "deepep-rocm-ci" { target "deepep-rocm-ci" {
inherits = ["_common-rocm", "_ci-rocm"] inherits = ["_common-rocm", "_ci-rocm"]
target = "build_deepep" target = "build_deepep"
@@ -394,7 +418,7 @@ target "ci-base-rocm-ci" {
CI_BASE_IMAGE_TAG != "" ? "type=registry,ref=${CI_BASE_IMAGE_TAG}" : "", CI_BASE_IMAGE_TAG != "" ? "type=registry,ref=${CI_BASE_IMAGE_TAG}" : "",
CI_BASE_TRUSTED_CONTENT_REF != "" ? "type=registry,ref=${CI_BASE_TRUSTED_CONTENT_REF}" : "", CI_BASE_TRUSTED_CONTENT_REF != "" ? "type=registry,ref=${CI_BASE_TRUSTED_CONTENT_REF}" : "",
]), ]),
# Import upstream dependency caches so NIXL/DeepEP stages # Import upstream dependency caches so NIXL/ROCShmem/DeepEP stages
# are cache hits even when ci_base itself needs rebuilding. # are cache hits even when ci_base itself needs rebuilding.
get_cache_from_rocm_deps(), get_cache_from_rocm_deps(),
) )
@@ -407,5 +431,5 @@ target "ci-base-rocm-ci" {
# Group for ci_base builds -- exports dependency stage caches alongside the # Group for ci_base builds -- exports dependency stage caches alongside the
# ci_base image so future rebuilds can reuse them independently. # ci_base image so future rebuilds can reuse them independently.
group "ci-base-rocm-ci-with-deps" { group "ci-base-rocm-ci-with-deps" {
targets = ["nixl-rocm-ci", "deepep-rocm-ci", "ci-base-rocm-ci"] targets = ["nixl-rocm-ci", "rocshmem-rocm-ci", "deepep-rocm-ci", "ci-base-rocm-ci"]
} }
+4
View File
@@ -65,6 +65,10 @@ variable "UCX_BRANCH" {
default = "" default = ""
} }
variable "ROCSHMEM_BRANCH" {
default = ""
}
variable "DEEPEP_BRANCH" { variable "DEEPEP_BRANCH" {
default = "" default = ""
} }
+1 -1
View File
@@ -136,7 +136,7 @@ vllm bench {forward, latency, serve, throughput}
### forward ### forward
Benchmark the model forward of exact batches: each case lists its requests as `[N*]Q[@K][c][:D]` (query length Q, KV length K including the query, D draft tokens). A prefill's prefix is a prefix-cache hit unless `c` marks a chunked prefill; they differ only on models with SWA bounded replay, which recompute the last window of a hit. Each request's prefix length varies by ±50% by default (`--kv-len-jitter`), identically on every DP rank. Each forward is captured as a one-shot CUDA graph; sampling and CPU overhead aren't measured. Use `torchrun`, one process per GPU, for DP/TP/EP. CUDA only for now. Add `--profile-dir DIR` to write a Perfetto trace and a per-kernel GPU time summary of each case; `python -m vllm.benchmarks.forward.profile A.kernels.json [B.kernels.json]` prints a summary or compares two. Under Nsight Systems, `nsys profile --capture-range=cudaProfilerApi --capture-range-end=repeat --cuda-graph-trace=node` records just those replays, one report per case. Benchmark the model forward of exact batches: each case lists its requests as `[N*]Q[@K][c][:D]` (query length Q, KV length K including the query, D draft tokens). A prefill's prefix is a prefix-cache hit unless `c` marks a chunked prefill; they differ only on models with SWA bounded replay, which recompute the last window of a hit. Each request's prefix length varies by ±50% by default (`--kv-len-jitter`), identically on every DP rank. Each forward is captured as a one-shot CUDA graph; sampling and CPU overhead aren't measured. Use `torchrun`, one process per GPU, for DP/TP/EP. CUDA only for now.
```bash ```bash
vllm bench forward \ vllm bench forward \
+37 -4
View File
@@ -83,7 +83,7 @@ directly from the Triton kernel through unified virtual addressing (UVA),
prefetching on a side CUDA stream so transfers overlap with the compute of prefetching on a side CUDA stream so transfers overlap with the compute of
preceding layers. The freed GPU memory goes to the KV cache. preceding layers. The freed GPU memory goes to the KV cache.
`cpu_offload` defaults to `true` (on) `cpu_offload` defaults to `true` and is set only through `--engram-config`.
To keep the tables resident on GPU instead (e.g. when host memory is To keep the tables resident on GPU instead (e.g. when host memory is
scarce): scarce):
@@ -93,8 +93,40 @@ vllm serve deepseek-ai/DeepSeek-V4.1-Flash \
--engram-config.cpu_offload false --engram-config.cpu_offload false
``` ```
CPU offload requires a GPU with UVA support; CPU offload requires a GPU with UVA support; vLLM fails fast when it is
vLLM fails fast when it is unavailable. unavailable.
## Host file gather (Qwen4Exp)
`host_file_gather` serves Qwen4Exp PLE rows from the loader's own file
mapping of the safetensors shards instead of allocating the table. Set
`--engram-config '{"host_file_gather": true}'`. It applies only to Qwen4Exp
and takes precedence over `cpu_offload`.
Rows stay in the checkpoint dtype: an FP8 table stays FP8 and is dequantized
on the GPU, and an unquantized table must match `--dtype`. CPU offload
converts on load; this mode rejects the mismatch at load.
Faulted pages are reclaimable page cache, shared across co-located processes.
They appear in each process's RSS as file-backed (`RssFile`), and PSS is
where they are split across processes. A write to the mapping copies that
page into anonymous RSS. Each step pays a host sync, and every TP rank
gathers all rows. Co-located ranks share the cache; ranks on different
machines do not.
Serving depends on the checkpoint files staying unchanged while the server
runs: an in-place overwrite serves the new bytes, truncation kills the worker
with SIGBUS, and deleting or renaming the files is safe.
The first touch of a row costs a disk read, and a page evicted under memory
pressure is read again on the next miss. By default no prewarm is needed; for
a fast first request, read the PLE files once after the server has started
and allocated its KV cache.
Reload is unsupported and fails closed. ROCm and elastic EP reject the key.
Load paths that do not hand over file-backed safetensors tensors fail at bind
(`eager`, `.bin`, runai, `fastsafetensors`, `instanttensor`). The multithread
loader is file-backed and is admitted. `--load-format dummy` serves zeros.
## Data-parallel topologies (DeepSeek V4.1) ## Data-parallel topologies (DeepSeek V4.1)
@@ -115,7 +147,8 @@ vllm serve deepseek-ai/DeepSeek-V4.1-Flash \
``` ```
`embedding_across_dp` is not supported with elastic expert parallelism yet. `embedding_across_dp` is not supported with elastic expert parallelism yet.
Qwen4Exp PLE tables are ETP-sharded instead and honor `cpu_offload` only. Qwen4Exp PLE tables are ETP-sharded instead and honor only `cpu_offload` and
`host_file_gather`.
## Sharing host tables across DP replicas ## Sharing host tables across DP replicas
-29
View File
@@ -411,35 +411,6 @@ Example (OpenAI-compatible completions request):
} }
``` ```
## Explicit pool sizing for `SimpleCPUOffloadConnector`
`SimpleCPUOffloadConnector` accepts an optional `num_cpu_blocks` in
`kv_connector_extra_config`. The scheduler and every worker use this exact
number of offload blocks instead of automatic sizing from
`cpu_bytes_to_use_per_rank` or `cpu_bytes_to_use / world_size`.
Use the same positive integer on all ranks, chosen
to fit the smallest per-worker capacity. This supports pipeline stages with
different KV cache sizes without adding a collective during initialization.
```json
{
"kv_connector": "SimpleCPUOffloadConnector",
"kv_role": "kv_both",
"kv_connector_extra_config": {
"cpu_bytes_to_use_per_rank": 1073741824,
"num_cpu_blocks": 64,
"lazy_offload": false
}
}
```
The example count must be adjusted for the model, KV layout, and parallelism.
The configured byte budget remains an upper bound: startup fails if the
requested count exceeds a worker's capacity. For the `disk` backend the count
selects disk slots and is checked against `disk_capacity_bytes`. Omitting
`num_cpu_blocks` preserves automatic per-worker sizing. This option applies
to `SimpleCPUOffloadConnector`, separately from `OffloadingConnector` above.
## Further Reading ## Further Reading
- [vLLM blog: KV Offloading Connector](https://vllm.ai/blog/2026-01-08-kv-offloading-connector) — motivation, architecture (DMA-based async transfer), and benchmarks (TTFT and throughput). - [vLLM blog: KV Offloading Connector](https://vllm.ai/blog/2026-01-08-kv-offloading-connector) — motivation, architecture (DMA-based async transfer), and benchmarks (TTFT and throughput).
-8
View File
@@ -136,14 +136,6 @@ WRITE mode is used by default. READ mode can be configured by setting `--kv-tran
!!! note !!! note
`notify_port` is used as a *base* port: each (DP rank, TP rank) pair within an instance uses `notify_port + offset` where the offset is based on the rank. Make sure the range starting at `notify_port` is free on the host. `notify_port` is used as a *base* port: each (DP rank, TP rank) pair within an instance uses `notify_port + offset` where the offset is based on the rank. Make sure the range starting at `notify_port` is free on the host.
### Hybrid attention and recurrent state
WRITE mode supports hybrid attention plus Mamba/KDA models, including DSpark, with equal producer and consumer TP sizes. Use matching cache layouts on both instances, `--mamba-cache-mode align`, and `read_mode: false` in each instance's `kv_connector_extra_config`. Both peers must support hybrid WRITE. Sliding-window WRITE and hybrid speculative methods other than DSpark remain unsupported.
For DSpark, configure the same draft model on both instances through `--speculative-config`. Target and draft attention groups retain their own block IDs and token spans; only the running recurrent state is transferred, not speculative scratch slots. The producer transfers the state after the first N-1 prompt tokens, and the consumer recomputes the last prompt token. Convolution and SSM transfers must both finish before the layer is complete.
Hybrid READ keeps its existing single-transferable-attention-group requirement. Plain-attention WRITE retains its existing notification format.
### Transport configuration ### Transport configuration
MoRI has two transport backends: RDMA and xGMI. You can select backend using `--kv-transfer-config.kv_connector_extra_config.backend $BACKEND`, with `$BACKEND` being `rdma` or `xgmi`. RDMA is the default backend and should be used in multi-node deployments. MoRI has two transport backends: RDMA and xGMI. You can select backend using `--kv-transfer-config.kv_connector_extra_config.backend $BACKEND`, with `$BACKEND` being `rdma` or `xgmi`. RDMA is the default backend and should be used in multi-node deployments.
+3 -3
View File
@@ -187,7 +187,7 @@ Multi-image input can be extended to perform video captioning. We show this with
#### Custom RGBA Background Color #### Custom RGBA Background Color
When loading images with transparency (including RGBA images and PNGs with a tRNS chunk), vLLM converts them to RGB format. By default, transparent pixels are replaced with white background. You can customize this background color using the `rgba_background_color` parameter in `media_io_kwargs`. When loading RGBA images (images with transparency), vLLM converts them to RGB format. By default, transparent pixels are replaced with white background. You can customize this background color using the `rgba_background_color` parameter in `media_io_kwargs`.
??? code ??? code
@@ -212,7 +212,7 @@ When loading images with transparency (including RGBA images and PNGs with a tRN
!!! note !!! note
- The `rgba_background_color` accepts RGB values as a list `[R, G, B]` or tuple `(R, G, B)` where each value is 0-255 - The `rgba_background_color` accepts RGB values as a list `[R, G, B]` or tuple `(R, G, B)` where each value is 0-255
- This setting affects images with transparency, including RGB PNGs with a transparent color; opaque RGB images are unchanged - This setting only affects RGBA images with transparency; RGB images are unchanged
- If not specified, the default white background `(255, 255, 255)` is used for backward compatibility - If not specified, the default white background `(255, 255, 255)` is used for backward compatibility
#### Moondream3 Prompt Recipes { #moondream3-prompt-recipes } #### Moondream3 Prompt Recipes { #moondream3-prompt-recipes }
@@ -1089,7 +1089,7 @@ By passing this metadata, the model can better understand the temporal distribut
#### Custom RGBA Background Color #### Custom RGBA Background Color
To use a custom background color for images with transparency, pass the `rgba_background_color` parameter via `--media-io-kwargs`: To use a custom background color for RGBA images, pass the `rgba_background_color` parameter via `--media-io-kwargs`:
```bash ```bash
# Example: Black background for dark theme # Example: Black background for dark theme
-24
View File
@@ -201,30 +201,6 @@ checkpoint's output format.
!!! note !!! note
With `tool_choice="auto"`, structural-tag constraints require both `VLLM_ENFORCE_STRICT_TOOL_CALLING=true` (the default) and at least one tool with `strict: true`, or a server-side floor set via `--tool-strict-level`. When these conditions are met and the selected parser supports structural tags, vLLM constrains the tool-call envelope and pins the argument schema of each tool that sets `strict: true` (or of every tool under `--tool-strict-level parameter`). Otherwise, vLLM extracts tool calls from raw text, so arguments may occasionally be malformed or violate the function's parameter schema. With `tool_choice="auto"`, structural-tag constraints require both `VLLM_ENFORCE_STRICT_TOOL_CALLING=true` (the default) and at least one tool with `strict: true`, or a server-side floor set via `--tool-strict-level`. When these conditions are met and the selected parser supports structural tags, vLLM constrains the tool-call envelope and pins the argument schema of each tool that sets `strict: true` (or of every tool under `--tool-strict-level parameter`). Otherwise, vLLM extracts tool calls from raw text, so arguments may occasionally be malformed or violate the function's parameter schema.
### Beam Models (`beam`)
Beam checkpoints describe their output format with `response_template` metadata, so the `beam` parser reads it like the `hf` parser does:
```bash
vllm serve <model> \
--enable-auto-tool-choice \
--tool-call-parser beam \
--reasoning-parser beam
```
Unlike `hf`, the `beam` parser also constrains its output. For strict tools, required or named tool choice, and structured outputs, it derives a
structural-tag grammar from the template. The grammar follows the template's tool call format, pins the arguments of strict tools when their
encoding can be derived, allows a single call with `parallel_tool_calls=false`, and frames a structured output as the answer. Raw string arguments
keep the `format` or `pattern` of their schema, as XGrammar keeps them in JSON strings, unless a match could end the value early. With
`tool_choice="none"`, the grammar allows only reasoning and an answer, so a model that still sees the tools in its prompt cannot call them. When no
grammar can be derived, required and named tool choice are rejected, and strict tools under `tool_choice="auto"` are left unconstrained with a
warning.
It also constrains the output of requests without strict tools, forced tool choice, or structured outputs to the template's format: tool calls with
free arguments or an answer, or just the rest of an answer the prompt already opened. This always-on grammar needs the `xgrammar` structured
outputs backend, the default, and is disabled with `VLLM_BEAM_ALWAYS_CONSTRAIN_OUTPUT=0`. From the message start, every `beam` grammar opens with
reasoning that holds visible text, unless `enable_thinking` is false.
### Hermes Models (`hermes`) ### Hermes Models (`hermes`)
All Nous Research Hermes-series models newer than Hermes 2 Pro should be supported. All Nous Research Hermes-series models newer than Hermes 2 Pro should be supported.
@@ -1,7 +1,7 @@
<!-- markdownlint-disable MD041 MD051 --> <!-- markdownlint-disable MD041 MD051 -->
--8<-- [start:installation] --8<-- [start:installation]
vLLM supports AMD GPUs with ROCm 6.3 or above. Pre-built wheels are available for ROCm 10.0 (the default) and ROCm 7.2. vLLM supports AMD GPUs with ROCm 6.3 or above. Pre-built wheels are available for ROCm 7.0 and ROCm 7.2.1.
#### Prebuilt Wheels #### Prebuilt Wheels
@@ -9,8 +9,6 @@ vLLM supports AMD GPUs with ROCm 6.3 or above. Pre-built wheels are available fo
| ------------ | -------------- | ------------ | ----------------- | ------------------ | | ------------ | -------------- | ------------ | ----------------- | ------------------ |
| `rocm700` | 3.12 | 7.0 | >= 2.35 | `0.14.0` to `0.18.0` | | `rocm700` | 3.12 | 7.0 | >= 2.35 | `0.14.0` to `0.18.0` |
| `rocm721` | 3.12 | 7.2.1 | >= 2.35 | Nightly releases after commit `171775f306a333a9cf105bfd533bf3e113d401d9` | | `rocm721` | 3.12 | 7.2.1 | >= 2.35 | Nightly releases after commit `171775f306a333a9cf105bfd533bf3e113d401d9` |
| `rocm72` | 3.12 | 7.2 | >= 2.39 | Nightly releases on the legacy ROCm 7.2 stack |
| `rocm100` | 3.12 | 10.0 | >= 2.35 | Nightly releases (default) |
--8<-- [end:installation] --8<-- [end:installation]
--8<-- [start:requirements] --8<-- [start:requirements]
@@ -42,17 +40,12 @@ If you need a different ROCm version or want to use an existing PyTorch installa
source .venv/bin/activate source .venv/bin/activate
``` ```
To install the latest version of vLLM for Python 3.12, ROCm 10.0 and `glibc >= 2.35`: To install the latest version of vLLM for Python 3.12, ROCm 7.0 and `glibc >= 2.35`.
```bash ```bash
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/ --upgrade uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/ --upgrade
``` ```
!!! tip
The ROCm 10.0 wheels install the ROCm SDK from pip ([TheRock](https://github.com/ROCm/TheRock)), including the GPU kernels for every supported architecture.
Wheels for the legacy ROCm 7.2 stack, which use a system ROCm installation, remain available at `https://wheels.vllm.ai/rocm/${VLLM_VERSION}/rocm72`.
!!! tip !!! tip
You can find out about which ROCm version the latest vLLM supports by checking the `vllm` package in index in extra-index-url <https://wheels.vllm.ai/rocm/> at [https://wheels.vllm.ai/rocm/vllm](https://wheels.vllm.ai/rocm/vllm) . You can find out about which ROCm version the latest vLLM supports by checking the `vllm` package in index in extra-index-url <https://wheels.vllm.ai/rocm/> at [https://wheels.vllm.ai/rocm/vllm](https://wheels.vllm.ai/rocm/vllm) .
@@ -202,9 +195,9 @@ install it without re-running the full installation:
and place it in `vllm/vllm-rs`. and place it in `vllm/vllm-rs`.
If you see an error about a wheel not being found, the wheel for your base If you see an error about a wheel not being found, the wheel for your base
commit and ROCm version might not be available. Check the available commit and ROCm patch version might not be available. Check the available
variants under `https://wheels.vllm.ai/rocm/<commit>/`. Variants are named by ROCm variants under `https://wheels.vllm.ai/rocm/<commit>/`. For example, ROCm 7.2.1
major and minor version, e.g. ROCm 7.2.3 uses the `rocm72` variant. uses the `rocm721` variant.
There are more environment variables to control the behavior of Python-only There are more environment variables to control the behavior of Python-only
build: build:
@@ -215,7 +208,7 @@ build:
- `VLLM_PRECOMPILED_WHEEL_COMMIT`: override the full commit hash used to - `VLLM_PRECOMPILED_WHEEL_COMMIT`: override the full commit hash used to
download the pre-compiled wheel. download the pre-compiled wheel.
- `VLLM_PRECOMPILED_WHEEL_VARIANT`: specify the ROCm variant subdirectory, e.g., - `VLLM_PRECOMPILED_WHEEL_VARIANT`: specify the ROCm variant subdirectory, e.g.,
`rocm72` or `rocm100`. If not specified, the variant is auto-detected based `rocm700` or `rocm721`. If not specified, the variant is auto-detected based
on your system's ROCm version. An explicitly specified variant must match the on your system's ROCm version. An explicitly specified variant must match the
detected environment. detected environment.
@@ -233,34 +226,23 @@ You can find more information about vLLM's wheels in
#### Full build (with compilation) {#full-build} #### Full build (with compilation) {#full-build}
!!! tip !!! tip
- The steps below follow [docker/Dockerfile.rocm_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_base), which pins the validated version of every component. If a step does not work for you, refer to it; a Dockerfile is a form of installation steps. - If you found that the following installation step does not work for you, please refer to [docker/Dockerfile.rocm_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_base). Dockerfile is a form of installation steps.
- The legacy ROCm 7.2 stack uses a system ROCm installation instead; its steps are in [docker/Dockerfile.rocm_72_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_72_base).
0. Install the ROCm SDK and PyTorch (skip if you are already in an environment or docker image with them installed, e.g. `vllm/vllm-openai-rocm:base-nightly`). 0. Install prerequisites (skip if you are already in an environment/docker with the following installed):
ROCm 10.0 is installed from pip ([TheRock](https://github.com/ROCm/TheRock)), together with PyTorch and the GPU kernels for your GPU's architecture. In a Python 3.12 environment: - [ROCm](https://rocm.docs.amd.com/en/latest/deploy/linux/index.html)
- [PyTorch](https://pytorch.org/)
For installing PyTorch, you can start from a fresh docker image, e.g, `rocm/pytorch:rocm7.0_ubuntu22.04_py3.10_pytorch_release_2.8.0`, `rocm/pytorch-nightly`. If you are using docker image, you can skip to Step 3.
Alternatively, you can install PyTorch using PyTorch wheels. You can check PyTorch installation guide in PyTorch [Getting Started](https://pytorch.org/get-started/locally/). Example:
```bash ```bash
# Your GPU's architecture, e.g. gfx942 for MI300 or gfx950 for MI350 # Install PyTorch
export GPU_ARCH=$(uvx --from rocm-bootstrap rocm-bootstrap-detect --unique) pip uninstall torch -y
export ROCM_INDEX=https://stable.repo.amd.com/rocm/whl-next/ pip install --no-cache-dir torch torchvision --index-url https://download.pytorch.org/whl/nightly/rocm7.0
pip install --index-url ${ROCM_INDEX} \
"rocm[libraries,devel,device-${GPU_ARCH}]==10.0.0" \
"torch[device-${GPU_ARCH}]==2.12.0+rocm10.0.0" \
"torchvision[device-${GPU_ARCH}]==0.27.0+rocm10.0.0"
rocm-sdk init
# Point builds at the pip-installed ROCm SDK
export ROCM_PATH=$(rocm-sdk path --root)
export ROCM_HOME=${ROCM_PATH} HIP_PATH=${ROCM_PATH}
export PATH=$(rocm-sdk path --bin):${PATH}
``` ```
!!! note
- The validated `ROCM_SDK_VERSION`, `TORCH_VERSION` and `TORCHVISION_VERSION` can be found in [docker/Dockerfile.rocm_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_base).
- To support several GPU architectures, add a `device-<arch>` extra for each of them.
1. Install [Triton for ROCm](https://github.com/ROCm/triton.git) 1. Install [Triton for ROCm](https://github.com/ROCm/triton.git)
Install ROCm's Triton following the instructions from [ROCm/triton](https://github.com/ROCm/triton.git) Install ROCm's Triton following the instructions from [ROCm/triton](https://github.com/ROCm/triton.git)
@@ -270,7 +252,8 @@ You can find more information about vLLM's wheels in
pip uninstall -y triton pip uninstall -y triton
git clone https://github.com/ROCm/triton.git git clone https://github.com/ROCm/triton.git
cd triton cd triton
git checkout $TRITON_BRANCH # git checkout $TRITON_BRANCH
git checkout f9e5bf54
if [ ! -f setup.py ]; then cd python; fi if [ ! -f setup.py ]; then cd python; fi
python3 setup.py install python3 setup.py install
cd ../.. cd ../..
@@ -282,14 +265,17 @@ You can find more information about vLLM's wheels in
2. Optionally, if you choose to use CK flash attention, you can install [flash attention for ROCm](https://github.com/Dao-AILab/flash-attention.git) 2. Optionally, if you choose to use CK flash attention, you can install [flash attention for ROCm](https://github.com/Dao-AILab/flash-attention.git)
Install ROCm's flash attention following the instructions from [ROCm/flash-attention](https://github.com/Dao-AILab/flash-attention#amd-rocm-support) Install ROCm's flash attention (v2.8.0) following the instructions from [ROCm/flash-attention](https://github.com/Dao-AILab/flash-attention#amd-rocm-support)
For example, for ROCm 7.0, suppose your gfx arch is `gfx942`. To get your gfx architecture, run `rocminfo |grep gfx`.
```bash ```bash
git clone https://github.com/Dao-AILab/flash-attention.git git clone https://github.com/Dao-AILab/flash-attention.git
cd flash-attention cd flash-attention
git checkout $FA_BRANCH # git checkout $FA_BRANCH
git checkout 0e60e394
git submodule update --init git submodule update --init
GPU_ARCHS="${GPU_ARCH}" python3 setup.py install GPU_ARCHS="gfx942" python3 setup.py install
cd .. cd ..
``` ```
@@ -304,8 +290,7 @@ You can find more information about vLLM's wheels in
cd aiter cd aiter
git checkout $AITER_BRANCH_OR_COMMIT git checkout $AITER_BRANCH_OR_COMMIT
git submodule sync; git submodule update --init --recursive git submodule sync; git submodule update --init --recursive
# Keep the Triton installed in step 1 python3 setup.py develop
AITER_USE_SYSTEM_TRITON=1 python3 setup.py develop
``` ```
!!! note !!! note
@@ -315,34 +300,32 @@ You can find more information about vLLM's wheels in
4. Optionally, if you want to use MORI for EP or PD disaggregation, you can install [MORI](https://github.com/ROCm/mori) using the following steps: 4. Optionally, if you want to use MORI for EP or PD disaggregation, you can install [MORI](https://github.com/ROCm/mori) using the following steps:
```bash ```bash
pip install "setuptools_scm>=6.2"
git clone https://github.com/ROCm/mori.git git clone https://github.com/ROCm/mori.git
cd mori cd mori
git checkout $MORI_BRANCH_OR_COMMIT git checkout $MORI_BRANCH_OR_COMMIT
git submodule sync; git submodule update --init --recursive git submodule sync; git submodule update --init --recursive
# The ROCm SDK keeps its CMake configs for system dependencies (e.g. NUMA) under rocm_sysdeps MORI_GPU_ARCHS="gfx942;gfx950" python3 setup.py install
CMAKE_PREFIX_PATH="${ROCM_PATH}:${ROCM_PATH}/lib/rocm_sysdeps" \
MORI_GPU_ARCHS="gfx942;gfx950" python3 setup.py install
``` ```
!!! note !!! note
- You will need to config the `$MORI_BRANCH_OR_COMMIT` for your purpose. - You will need to config the `$MORI_BRANCH_OR_COMMIT` for your purpose.
- The validated `$MORI_BRANCH_OR_COMMIT` can be found in the [docker/Dockerfile.rocm_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_base). - The validated `$MORI_BRANCH_OR_COMMIT` can be found in the [docker/Dockerfile.rocm_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_base).
5. Build vLLM. For example, vLLM on ROCm 10.0 can be built with the following steps: 5. Build vLLM. For example, vLLM on ROCM 7.0 can be built with the following steps:
???+ console "Commands" ???+ console "Commands"
```bash ```bash
pip install --upgrade pip pip install --upgrade pip
# Install AMD SMI, which the ROCm SDK ships as sources # Build & install AMD SMI
SDK_CORE=$(python3 -c "import _rocm_sdk_core, os; print(os.path.dirname(_rocm_sdk_core.__file__))") pip install /opt/rocm/share/amd_smi
pip install ${SDK_CORE}/share/amd_smi
# It loads the SDK's libamd_smi relative to its own directory
export PYTHONPATH=${SDK_CORE}/share/amd_smi
# Install dependencies # Install dependencies
pip install --upgrade numba \
scipy \
huggingface-hub \
setuptools_scm
pip install -r requirements/rocm.txt pip install -r requirements/rocm.txt
# To build for a single architecture (e.g., MI300) for faster installation (recommended): # To build for a single architecture (e.g., MI300) for faster installation (recommended):
@@ -369,9 +352,8 @@ You can find more information about vLLM's wheels in
vLLM offers official Docker images for deployment. vLLM offers official Docker images for deployment.
The images can be used to run OpenAI compatible server and are available on Docker Hub as [vllm/vllm-openai-rocm](https://hub.docker.com/r/vllm/vllm-openai-rocm/tags). The images can be used to run OpenAI compatible server and are available on Docker Hub as [vllm/vllm-openai-rocm](https://hub.docker.com/r/vllm/vllm-openai-rocm/tags).
- `vllm/vllm-openai-rocm:latest` — stable release, built on ROCm 10.0 (also tagged `latest-rocm100`) - `vllm/vllm-openai-rocm:latest` — stable release
- `vllm/vllm-openai-rocm:nightly` — preview build from the latest development branch on ROCm 10.0 (also tagged `nightly-rocm100`), use this if you want the latest features and fixes - `vllm/vllm-openai-rocm:nightly` — preview build from the latest development branch, use this if you want the latest features and fixes
- `vllm/vllm-openai-rocm:latest-rocm72` / `nightly-rocm72` — the same builds on the legacy ROCm 7.2 stack, kept for a transition period
```bash ```bash
docker run --rm \ docker run --rm \
@@ -427,7 +409,7 @@ You can build and run vLLM from source via the provided [docker/Dockerfile.rocm]
??? info "(Optional) Build an image with ROCm software stack" ??? info "(Optional) Build an image with ROCm software stack"
Build a docker image from [docker/Dockerfile.rocm_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_base) which setup ROCm software stack needed by the vLLM. Build a docker image from [docker/Dockerfile.rocm_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_base) which setup ROCm software stack needed by the vLLM.
**This step is optional as this rocm_base image is usually prebuilt and store at [Docker Hub](https://hub.docker.com/r/rocm/vllm-dev) under tag `rocm/vllm-dev:base` to speed up user experience.** For the legacy ROCm 7.2 stack, the base image is built from [docker/Dockerfile.rocm_72_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_72_base) and published as `vllm/vllm-openai-rocm:base-nightly-rocm72`. **This step is optional as this rocm_base image is usually prebuilt and store at [Docker Hub](https://hub.docker.com/r/rocm/vllm-dev) under tag `rocm/vllm-dev:base` to speed up user experience.**
If you choose to build this rocm_base image yourself, the steps are as follows. If you choose to build this rocm_base image yourself, the steps are as follows.
It is important that the user kicks off the docker build using buildkit. Either the user put `DOCKER_BUILDKIT=1` as environment variable when calling docker build command, or the user needs to set up buildkit in the docker daemon configuration `/etc/docker/daemon.json` as follows and restart the daemon: It is important that the user kicks off the docker build using buildkit. Either the user put `DOCKER_BUILDKIT=1` as environment variable when calling docker build command, or the user needs to set up buildkit in the docker daemon configuration `/etc/docker/daemon.json` as follows and restart the daemon:
@@ -440,7 +422,7 @@ You can build and run vLLM from source via the provided [docker/Dockerfile.rocm]
} }
``` ```
To build vllm on ROCm 10.0 for all supported GPUs, you can use the default: To build vllm on ROCm 7.0 for MI200 and MI300 series, you can use the default:
```bash ```bash
DOCKER_BUILDKIT=1 docker build \ DOCKER_BUILDKIT=1 docker build \
@@ -459,15 +441,15 @@ It is important that the user kicks off the docker build using buildkit. Either
} }
``` ```
[docker/Dockerfile.rocm](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm) uses ROCm 10.0 (installed from TheRock wheels) by default. The legacy ROCm 7.2 stack is available as [docker/Dockerfile.rocm_72](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_72) with [docker/Dockerfile.rocm_72_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_72_base); older vLLM branches support ROCm 5.7 through 7.0. [docker/Dockerfile.rocm](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm) uses ROCm 7.0 by default, but also supports ROCm 5.7, 6.0, 6.1, 6.2, 6.3, and 6.4, in older vLLM branches.
It provides flexibility to customize the build of docker image using the following arguments: It provides flexibility to customize the build of docker image using the following arguments:
- `BASE_IMAGE`: specifies the base image used when running `docker build`. The default value `rocm/vllm-dev:base` is an image published and maintained by AMD. It is being built using [docker/Dockerfile.rocm_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_base). For [docker/Dockerfile.rocm_72](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_72), the default is `vllm/vllm-openai-rocm:base-nightly-rocm72` - `BASE_IMAGE`: specifies the base image used when running `docker build`. The default value `rocm/vllm-dev:base` is an image published and maintained by AMD. It is being built using [docker/Dockerfile.rocm_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_base)
- `ARG_PYTORCH_ROCM_ARCH`: Allows to override the gfx architecture values from the base docker image - `ARG_PYTORCH_ROCM_ARCH`: Allows to override the gfx architecture values from the base docker image
Their values can be passed in when running `docker build` with `--build-arg` options. Their values can be passed in when running `docker build` with `--build-arg` options.
To build vllm on ROCm 10.0 for all supported GPUs, you can use the default (which build a docker image with `vllm serve` as entrypoint): To build vllm on ROCm 7.0 for MI200 and MI300 series, you can use the default (which build a docker image with `vllm serve` as entrypoint):
```bash ```bash
DOCKER_BUILDKIT=1 docker build -f docker/Dockerfile.rocm -t vllm/vllm-openai-rocm . DOCKER_BUILDKIT=1 docker build -f docker/Dockerfile.rocm -t vllm/vllm-openai-rocm .
-27
View File
@@ -455,8 +455,6 @@ Score templates are supported for **cross-encoder** models only. If you are usin
Some scoring models require a specific prompt format to work correctly. You can specify a custom score template using the `--chat-template` parameter (see [Chat Template](../../serving/online_serving/README.md#chat-template)). Some scoring models require a specific prompt format to work correctly. You can specify a custom score template using the `--chat-template` parameter (see [Chat Template](../../serving/online_serving/README.md#chat-template)).
For offline scoring, pass `chat_template` to the `LLM` constructor to use it for every `LLM.score` call. A `chat_template` passed to `LLM.score` overrides the constructor's template for that call.
Like chat templates, the score template receives a `messages` list. For scoring, each message has a `role` attribute—either `"query"` or `"document"`. For the usual kind of point-wise cross-encoder, you can expect exactly two messages: one query and one document. To access the query and document content, use Jinja's `selectattr` filter: Like chat templates, the score template receives a `messages` list. For scoring, each message has a `role` attribute—either `"query"` or `"document"`. For the usual kind of point-wise cross-encoder, you can expect exactly two messages: one query and one document. To access the query and document content, use Jinja's `selectattr` filter:
- **Query**: `{{ (messages | selectattr("role", "eq", "query") | first).content }}` - **Query**: `{{ (messages | selectattr("role", "eq", "query") | first).content }}`
@@ -466,31 +464,6 @@ This approach is more robust than index-based access (`messages[0]`, `messages[1
Example template file: [examples/pooling/score/template/nemotron-rerank.jinja](../../../examples/pooling/score/template/nemotron-rerank.jinja) Example template file: [examples/pooling/score/template/nemotron-rerank.jinja](../../../examples/pooling/score/template/nemotron-rerank.jinja)
#### Templates declared by Sentence Transformers checkpoints
vLLM does not apply a checkpoint's chat template by default, since many rerankers inherit an unrelated template from their base LLM. The exception is a Sentence Transformers (v5.4+) `CrossEncoder` with a single `sequence-classification` Transformer module whose `sentence_bert_config.json` declares the `message` modality. Sentence Transformers formats the pairs of such checkpoints as `query`/`document` messages, so vLLM does the same with the saved template:
```json
{
"transformer_task": "sequence-classification",
"modality_config": {
"text": {"method": "forward", "method_output_name": "logits"},
"message": {"method": "forward", "method_output_name": "logits", "format": "flat"}
},
"module_output_name": "scores",
"processing_kwargs": {"chat_template": {"add_generation_prompt": true}}
}
```
- The default chat template is used, unless `processing_kwargs.chat_template.chat_template` names another template saved with the tokenizer (e.g. `additional_chat_templates/score.jinja` for `"score"`).
- The other `processing_kwargs.chat_template` settings, such as `add_generation_prompt`, are passed to the template; the request's `chat_template_kwargs` override them.
- Message contents are passed as strings, as with `--chat-template`.
- vLLM does not restore the template suffix after truncation (`restore_suffix`). Use `max_tokens_per_doc` rather than `truncate_prompt_tokens` to truncate long documents.
- `--chat-template` (or the `chat_template` argument of `LLM.score`) takes precedence over the saved template and its settings.
- Models with a built-in score template keep using it.
The template in use is logged at startup.
### Enable/disable activation ### Enable/disable activation
You can enable or disable activation via `use_activation` only works for cross-encoder models. You can enable or disable activation via `use_activation` only works for cross-encoder models.
@@ -313,12 +313,14 @@ Start the server:
```shell ```shell
vllm serve nvidia/llama-nemotron-rerank-vl-1b-v2 \ vllm serve nvidia/llama-nemotron-rerank-vl-1b-v2 \
--runner pooling \ --runner pooling \
--trust-remote-code --trust-remote-code \
--chat-template examples/pooling/score/template/nemotron-vl-rerank.jinja
``` ```
!!! note !!! note
The checkpoint declares its score template in its Sentence Transformers config, so vLLM applies it automatically (see [Templates declared by Sentence Transformers checkpoints](scoring.md#templates-declared-by-sentence-transformers-checkpoints)). The chat template bundled with this checkpoint's tokenizer is not suitable
Revisions older than October 2026 need `--chat-template examples/pooling/score/template/nemotron-vl-rerank.jinja`. for the Score/Rerank APIs. Use the provided override template when serving:
`examples/pooling/score/template/nemotron-vl-rerank.jinja`.
Score a text query against an image document: Score a text query against an image document:
+2 -2
View File
@@ -105,7 +105,7 @@ The server keeps no state between calls. Everything the next call needs is in `s
- **Send `prompt_token_ids` on the first chunk, even without a parser.** It seeds detokenization so the first token's leading space matches the coupled endpoint (see [Request format](#request-format)). Later chunks already carry that context in `stream_state`. - **Send `prompt_token_ids` on the first chunk, even without a parser.** It seeds detokenization so the first token's leading space matches the coupled endpoint (see [Request format](#request-format)). Later chunks already carry that context in `stream_state`.
- **Don't forward `[DONE]`.** It marks the end of the generate stream and isn't a chunk. - **Don't forward `[DONE]`.** It marks the end of the generate stream and isn't a chunk.
`/inference/v1/generate` returns `GenerateLogProbs` with integer `token_id`s (the generate server has no tokenizer), and derender turns those into `ChatCompletionLogProbs` / `CompletionLogProbs`, filling `token` and `bytes` from the tokenizer with the usual U+FFFD byte-fallback correction. Generate returns every top-k candidate the engine produced; derender keeps the ones the coupled endpoint would, with the same helpers: the first `top_logprobs` on chat (all of them for `top_logprobs=-1`) and the first `logprobs + 1` on completions (all of them for `logprobs=-1`). It reads those values from `chat_request` / `completion_request`, so without the original request, or with one that does not ask for logprobs, it returns every candidate. Requests that use `logprob_token_ids` get no output logprobs from generate (their `sampling_params.logprobs` is unset), so derender has none to return for them either. Streaming derender does the same per chunk on both the plain detokenization path and the parser path (see [Streaming state and logprobs](#streaming-state-and-logprobs)). As on `/v1/chat/completions`, a request with `include_reasoning: false` and a parser configured gets no logprobs at all, batch or streaming, since the decoded logprob tokens would reveal the hidden reasoning. `/inference/v1/generate` returns `GenerateLogProbs` with integer `token_id`s (the generate server has no tokenizer), and derender turns those into `ChatCompletionLogProbs` / `CompletionLogProbs`, filling `token` and `bytes` from the tokenizer with the usual U+FFFD byte-fallback correction. Generate returns every top-k candidate the engine produced; derender keeps the ones the coupled endpoint would, with the same helpers: the first `top_logprobs` on chat (all of them for `top_logprobs=-1`) and the first `logprobs + 1` on completions (all of them for `logprobs=-1`). It reads those values from `chat_request` / `completion_request`, so without the original request, or with one that does not ask for logprobs, it returns every candidate. Requests that use `logprob_token_ids` get no output logprobs from generate (their `sampling_params.logprobs` is unset), so derender has none to return for them either. Streaming derender does the same per chunk on the plain detokenization path (see [Streaming state and logprobs](#streaming-state-and-logprobs)). The parser path doesn't resolve them yet and drops them.
## Streaming cost ## Streaming cost
@@ -129,7 +129,7 @@ Streaming derender is stateless on the server side: all mutable state lives in t
- `logprob_context_token_ids`: the trailing sampled token IDs (at most 4) from previous chunks, used to seed byte-fallback (U+FFFD) correction so multi-byte characters whose tokens split across chunk boundaries still resolve to real strings - `logprob_context_token_ids`: the trailing sampled token IDs (at most 4) from previous chunks, used to seed byte-fallback (U+FFFD) correction so multi-byte characters whose tokens split across chunk boundaries still resolve to real strings
- `logprob_text_offset`: the cumulative emitted text length, so `text_offset` in completion streaming logprobs stays absolute across chunks instead of restarting at 0 - `logprob_text_offset`: the cumulative emitted text length, so `text_offset` in completion streaming logprobs stays absolute across chunks instead of restarting at 0
When a streamed `GenerateTokensStreamChoice` carries `logprobs`, the integer `token_id`s are decoded per chunk and the resolved logprobs are attached to the corresponding streamed choice. For chat it is as `ChatCompletionLogProbs` and for completions it is converted to the flat `CompletionLogProbs` lists. Chunks without `logprobs` produce choices with `logprobs: null`. On the parser path the chunk's logprobs are attached to whatever delta the parser emits for that chunk (reasoning, content or a tool call delta, or an empty delta while the parser is still buffering markup), which is what the coupled endpoint does; replay never touches them, since each chunk's `logprobs` cover exactly that chunk's tokens. When a streamed `GenerateTokensStreamChoice` carries `logprobs`, the integer `token_id`s are decoded per chunk and the resolved logprobs are attached to the corresponding streamed choice. For chat it is as `ChatCompletionLogProbs` and for completions it is converted to the flat `CompletionLogProbs` lists. Chunks without `logprobs` produce choices with `logprobs: null`.
## Example ## Example
@@ -1,50 +1,8 @@
# Tokens In <> Tokens Out API # Tokens In <> Tokens Out API
`POST /inference/v1/generate` is the token-in/token-out (TITO) HTTP endpoint: it takes prompt token IDs and returns generated token IDs, logprobs keyed by integer token ID, and `finish_reason`/`stop_reason`, with no detokenization. Use it when the caller does its own tokenization and chat templating, such as an RL rollout worker. It is also the generation step of the `render → generate → derender` pipeline used in disaggregated serving (see the [Renderer APIs](renderer.md) and [Derenderer APIs](derenderer.md)). `/inference/v1/generate` takes prompt token IDs and returns generated token IDs. It is the generation step of the `render → generate → derender` pipeline used in disaggregated serving (see the [Renderer APIs](renderer.md) and [Derenderer APIs](derenderer.md)).
## Enabling the endpoint (`--enable-scale-out`, `--tokens-only`) The endpoint is registered on `vllm serve` with `--enable-scale-out`, and always with `--tokens-only`. `--tokens-only` also skips loading the tokenizer, so the server can't turn token IDs back into text.
- `--enable-scale-out` registers it, with `/render`, `/derender` and `/inference/v1/abort_requests`.
- `--tokens-only` always registers it, skips loading the tokenizer and forces `detokenize=False`. Stop strings (`sampling_params.stop`) are then rejected with a 400; use `stop_token_ids`.
- With neither flag, it returns 403 `Scale-out endpoints are disabled. Restart vllm serve with --enable-scale-out to enable them.`
```bash
vllm serve Qwen/Qwen3-0.6B --enable-scale-out
curl -X POST http://localhost:8000/inference/v1/generate \
-H "Content-Type: application/json" \
-d '{"token_ids": [151644, 872, 198], "sampling_params": {"max_tokens": 32, "logprobs": 1}}'
```
A full client is in [examples/scale_out/token_generation_client.py](../../../examples/scale_out/token_generation_client.py).
## Request fields
| Field | Notes |
| --- | --- |
| `token_ids` | Required `list[int]`: the already-templated prompt. Non-empty, no negative IDs. |
| `sampling_params` | Required. Any [SamplingParams](../../api/README.md#inference-parameters) field, e.g. `max_tokens`, `temperature`, `logprobs`, `stop_token_ids`. Without `max_tokens`, the server applies its own default (as on the OpenAI endpoints), not 16. |
| `output_mode` | `"tokens"` (default) or `"text"`, see [Output modes](#output-modes). |
| `stream`, `stream_options` | SSE chunks ending with `data: [DONE]`. |
| `return_token_ids` | Echo the final prompt IDs as `prompt_token_ids`. |
| `features` / `content_parts` | Multimodal input, mutually exclusive. |
| `request_id`, `model`, `cache_salt`, `priority`, `kv_transfer_params`, `ec_transfer_params` | Optional. |
## Response fields
| `choices[i]` field | Notes |
| --- | --- |
| `token_ids` | Generated token IDs. |
| `finish_reason` | `stop`, `length` or `abort`. |
| `stop_reason` | Stop token ID or string; `null` on EOS, length and abort. |
| `logprobs` | `{"content": [{"token_id", "logprob", "rank", "top_logprobs": [...]}]}`, one entry per token, sampled token first. `token_id` is an integer, not the older `"token_id:<id>"` string. Needs `sampling_params.logprobs`. |
| `weight_versions`, `routed_experts`, `sampling_mask` | Optional, see [Weight versions](#weight-versions); the last two need `--enable-return-routed-experts` / `--return-sampling-mask`. |
The top level adds `request_id`, `usage`, `prompt_token_ids` and `prompt_logprobs`. Stream chunks carry the same per-choice fields for the new tokens.
## Targeting a data-parallel rank (`X-data-parallel-rank`)
Send the header `X-data-parallel-rank: <int>` to pin a request to one data-parallel (DP) engine rank; there is no body field. Without the header, or with a non-integer value, the request is load balanced. A rank outside `[0, data_parallel_size)` is rejected.
## Output modes ## Output modes
-4
View File
@@ -76,10 +76,6 @@ for span in completion.weight_versions:
The offsets cover the whole output, so with streamed delta outputs, collect the tokens before slicing. `/inference/v1/generate` returns the same list on each choice; see [Tokens In <> Tokens Out API](../serving/online_serving/token_in_token_out.md#weight-versions). The offsets cover the whole output, so with streamed delta outputs, collect the tokens before slicing. `/inference/v1/generate` returns the same list on each choice; see [Tokens In <> Tokens Out API](../serving/online_serving/token_in_token_out.md#weight-versions).
`/v1/completions` and `/v1/chat/completions` return it on each choice when the request sets `return_weight_versions: true`. The offsets count the choice's output tokens, which `return_token_ids` returns as `token_ids`, and a streamed choice gets the list on its last chunk.
An aborted request returns spans over its partial output. If you resubmit that output as part of a new prompt, the new request's spans start again at 0, so shift them by the number of tokens already generated before joining the two lists.
## Example ## Example
The [async RLHF example](../../examples/rl/rlhf_async_new_apis.py) demonstrates this pattern with `vllm.AsyncLLMEngine`, NCCL weight transfer, and mid-flight pause/resume with validation. The [async RLHF example](../../examples/rl/rlhf_async_new_apis.py) demonstrates this pattern with `vllm.AsyncLLMEngine`, NCCL weight transfer, and mid-flight pause/resume with validation.
-2
View File
@@ -18,8 +18,6 @@ The following open-source RL libraries use vLLM for fast rollouts (sorted alphab
For weight synchronization between training and inference, see the [Weight Transfer](weight_transfer/README.md) documentation, which covers the pluggable backend system with [NCCL](weight_transfer/nccl.md) (multi-GPU) and [IPC](weight_transfer/ipc.md) (same-GPU) engines. To verify that an update landed on every rank, see the [Weight Checker](weight_checker.md). For weight synchronization between training and inference, see the [Weight Transfer](weight_transfer/README.md) documentation, which covers the pluggable backend system with [NCCL](weight_transfer/nccl.md) (multi-GPU) and [IPC](weight_transfer/ipc.md) (same-GPU) engines. To verify that an update landed on every rank, see the [Weight Checker](weight_checker.md).
For MoE policies, see [Routed Experts Return](routed_experts.md) (`--enable-return-routed-experts`) to get the per-token expert routing of each rollout for routing replay (R3) in the trainer.
For pipelining generation and training to improve GPU utilization and throughput, see the [Async Reinforcement Learning](async_rl.md) guide, which covers the pause/resume API for safely updating weights mid-flight. For pipelining generation and training to improve GPU utilization and throughput, see the [Async Reinforcement Learning](async_rl.md) guide, which covers the pause/resume API for safely updating weights mid-flight.
See the following notebooks showing how to use vLLM for GRPO: See the following notebooks showing how to use vLLM for GRPO:
-54
View File
@@ -1,54 +0,0 @@
# Routed Experts Return (MoE Routing Replay / R3)
`--enable-return-routed-experts` makes vLLM return the routed experts (the expert IDs each MoE layer's router selected) for every token of a request. RL trainers use it for MoE routing replay (R3): the training forward pass reuses the rollout's routing.
```bash
vllm serve Qwen/Qwen3-30B-A3B --enable-return-routed-experts
# offline: LLM("Qwen/Qwen3-30B-A3B", enable_return_routed_experts=True)
```
The flag sets `AuxOutputConfig.enable_return_routed_experts` (full form: `--aux-output-config '{"enable_return_routed_experts": true, "max_bytes": N}'`). There is no per-request on/off switch.
## Requirements
Startup fails with a `ValueError` unless:
- **Model Runner V2** is used (`VLLM_USE_V2_MODEL_RUNNER=1` if vLLM falls back to V1);
- the runner is `generate` and the model is **MoE**;
- **prefix caching is enabled** (the default). `--no-enable-prefix-caching` fails; the old v0.21 advice to disable it is wrong.
Rejected at startup: adaptive speculative verification, PP > 1, DCP/PCP > 1, and the `NixlConnector`, `NixlPullConnector`, `NixlPushConnector`, `MoRIIOConnector`, `MooncakeConnector` KV connectors. Regular speculative decoding works.
### How it works: aux output connector
Routing is stored by the aux output connector (`vllm/distributed/aux_output_connector/`, configured via `--aux-output-config`), keyed by KV block hash and shared with prefix caching, which is why prefix caching is required. `max_bytes` is its host-memory LRU capacity (default: derived from KV cache capacity). If a prefix-cache hit needs evicted rows, the worker raises `BlockObjectStoreError`; increase `max_bytes`.
## Output format
| API | Field |
| --- | --- |
| Offline (`LLM`, `AsyncLLM`) | `CompletionOutput.routed_experts`: `np.ndarray` or `None` |
| `/v1/completions`, `/v1/chat/completions` | `choices[i].routed_experts`: base64 `.npy`, **non-streaming only** |
| `/inference/v1/generate` | `routed_experts` per choice (base64 `.npy`); with `stream: true`, on the final chunk |
- **Shape:** `(num_tokens - 1 - routed_experts_prompt_start, num_moe_layers, num_experts_per_tok)`, `num_tokens` = prompt + output. The last sampled token has no row.
- **Rows:** row `i` is position `routed_experts_prompt_start + i` of `prompt_token_ids + output_token_ids`.
- **dtype:** `uint8` for at most 256 experts, else `uint16`; logical expert IDs.
- Filled only when the request finishes. `None` if the flag is off or the request aborted before any forward pass.
```python
import base64, io
import numpy as np
experts = np.load(io.BytesIO(base64.b64decode(choice["routed_experts"])))
```
## `routed_experts_prompt_start`
Per-request, default `0`: drops the first N prompt-token rows (e.g. routing already received in earlier turns). Must be `0 <= value <= prompt_len`, else HTTP 400. Pass it top-level on `/v1/*completions`, inside `sampling_params` on `/inference/v1/generate`, or as `SamplingParams(routed_experts_prompt_start=...)`.
## Limitations
- The Rust HTTP frontend never returns `routed_experts` and always sends `routed_experts_prompt_start = 0`. Use the Python API server.
Example: [examples/rl/routed_experts_e2e.py](../../examples/rl/routed_experts_e2e.py).
+2 -3
View File
@@ -139,9 +139,8 @@ Each position then carries the sampled token's logprob and the top-128
drop them. The finite entries are the sampling mask when it has at most `k` drop them. The finite entries are the sampling mask when it has at most `k`
tokens. tokens.
- The sampled token is always the first entry and may repeat inside the top-k. - The sampled token is always the first entry and may repeat inside the top-k.
- The `token_id:<id>` strings come from the OpenAI endpoints with - `/inference/v1/generate` encodes IDs as `token_id:<id>`; the OpenAI
`return_tokens_as_token_ids`; `/inference/v1/generate` returns an integer endpoints need `return_tokens_as_token_ids`.
`token_id` instead.
## Limitations ## Limitations
+1 -8
View File
@@ -145,7 +145,7 @@ engine.send_weights() # called on every rank
## API Endpoints ## API Endpoints
When running vLLM as an HTTP server, the following endpoints are available for weight transfer. `HTTPVLLMWeightSyncClient` speaks the first five for you. A trainer may also implement the protocol directly without importing vLLM: see the wire contracts in [NCCL](nccl.md#trainer-without-vllm-over-http) and [IPC](ipc.md#trainer-without-vllm-over-http). This keeps the trainer decoupled from vLLM: it needs no vLLM install or version pin, only this documented HTTP and wire contract, so trainer and server can upgrade independently (non-PyTorch trainers such as JAX can use it too). When running vLLM as an HTTP server, the following endpoints are available for weight transfer. `HTTPVLLMWeightSyncClient` speaks the first four for you.
| Endpoint | Method | Description | | Endpoint | Method | Description |
| -------- | ------ | ----------- | | -------- | ------ | ----------- |
@@ -153,7 +153,6 @@ When running vLLM as an HTTP server, the following endpoints are available for w
| `/start_weight_update` | POST | Start a weight update | | `/start_weight_update` | POST | Start a weight update |
| `/update_weights` | POST | Transfer a batch of weights with backend-specific metadata | | `/update_weights` | POST | Transfer a batch of weights with backend-specific metadata |
| `/finish_weight_update` | POST | Finish the update and optionally commit its `weight_version` | | `/finish_weight_update` | POST | Finish the update and optionally commit its `weight_version` |
| `/shutdown_weight_transfer_engine` | POST | Release the transfer channel without stopping inference |
| `/update_weight_version` | POST | Update `weight_version` without changing model weights | | `/update_weight_version` | POST | Update `weight_version` without changing model weights |
| `/weight_info` | GET | Get the latest committed weight version | | `/weight_info` | GET | Get the latest committed weight version |
| `/pause` | POST | Pause generation before weight sync to handle inflight requests | | `/pause` | POST | Pause generation before weight sync to handle inflight requests |
@@ -163,12 +162,6 @@ When running vLLM as an HTTP server, the following endpoints are available for w
!!! note !!! note
The HTTP weight transfer endpoints require `VLLM_SERVER_DEV_MODE=1` to be set. The HTTP weight transfer endpoints require `VLLM_SERVER_DEV_MODE=1` to be set.
!!! warning
If `/update_weights` fails, the model is left mid-reload while `/health`
stays 200, and `/finish_weight_update` is rejected. Before generating again,
resend a full `/start_weight_update` → `/update_weights` →
`/finish_weight_update` (see [#57854](https://github.com/vllm-project/vllm/pull/57854)).
The Rust frontend's optional gRPC `Control` service exposes the same pause, sleep, weight-transfer, and weight-version lifecycle for trusted sidecars. The `ServerInfo.rl_capabilities` response reports whether weight transfer and sleep mode were configured. Backend-specific `init_info` and `update_info` remain JSON metadata; model tensors continue to move over the configured NCCL, IPC, sparse-NCCL, or sharded-RDT transport. The Rust frontend's optional gRPC `Control` service exposes the same pause, sleep, weight-transfer, and weight-version lifecycle for trusted sidecars. The `ServerInfo.rl_capabilities` response reports whether weight transfer and sleep mode were configured. Backend-specific `init_info` and `update_info` remain JSON metadata; model tensors continue to move over the configured NCCL, IPC, sparse-NCCL, or sharded-RDT transport.
## Extending the System ## Extending the System
-38
View File
@@ -110,43 +110,6 @@ mode carries a per-chunk barrier across ranks: without it, a rank could overwrit
its buffer while its colocated worker is still reading the current chunk. This is its buffer while its colocated worker is still reading the current chunk. This is
handled inside `send_weights()`. handled inside `send_weights()`.
## Trainer Without vLLM over HTTP
A colocated trainer can speak the [HTTP endpoints](README.md#api-endpoints)
directly, with only PyTorch and an HTTP client. The server needs
`VLLM_ALLOW_INSECURE_SERIALIZATION=1`; without it, init and start return 200 and
the first `/update_weights` returns 500. Reference:
[`rlhf_http_ipc_no_vllm.py`](../../../examples/rl/rlhf_http_ipc_no_vllm.py).
`POST /init_weight_transfer_engine` body: `{"init_info": {"packed": false}}`.
`POST /update_weights` body `{"update_info": {...}}`:
| Field | Meaning |
| ----- | ------- |
| `names`, `dtype_names`, `shapes` | Checkpoint names, `torch` dtype attribute names (`"bfloat16"`), full shapes |
| `ipc_handles_pickled` | `base64(pickle.dumps(handles))`. Unpacked: a list with one dict per name. Packed: one dict for the shared buffer |
| `tensor_sizes` | Packed only: byte size of each tensor in the chunk, in `names` order |
- A handle dict is `{gpu_uuid: reduce_tensor(t)[1]}` with one entry per GPU the
server uses (`torch.multiprocessing.reductions.reduce_tensor`).
`gpu_uuid = str(torch.cuda.get_device_properties(i).uuid)`, **without** the
`GPU-` prefix that `nvidia-smi` prints. The worker overwrites `args[6]`
(device index) with its own.
- `ipc_handles` (unpickled) is not JSON-serializable; over HTTP use
`ipc_handles_pickled`.
- Packed: export one `uint8` buffer once and reuse it for every chunk. Copy
`t.contiguous().view(-1).view(torch.uint8)` in `names` order and close a chunk
**before** the next tensor would overflow it (unlike NCCL, which closes after).
A tensor larger than the buffer is an error. Synchronize before each
`/update_weights`; do not overwrite the buffer until the call returns.
- Keep every exported tensor alive until `/finish_weight_update` returns, then
drop the references and call `torch.cuda.ipc_collect()`.
- Multi-rank trainers: each rank exports handles for its own GPU, the ranks
gather the `{uuid: args}` dicts to rank 0 over their own `torch.distributed`
group and merge them per index, and rank 0 sends all HTTP. In packed mode,
barrier before and after each `/update_weights`.
## Rank-Local Updates ## Rank-Local Updates
For different parameter subsets per worker, `update_info` may be a list indexed For different parameter subsets per worker, `update_info` may be a list indexed
@@ -156,5 +119,4 @@ worker to participate.
## Examples ## Examples
- [RLHF with IPC weight syncing (`vllm serve`, HTTP)](../../../examples/rl/rlhf_http_ipc.py) - **Start here.** Server and training model share a single GPU; HTTP control plane, CUDA IPC data plane. Launches and tears down its own server - [RLHF with IPC weight syncing (`vllm serve`, HTTP)](../../../examples/rl/rlhf_http_ipc.py) - **Start here.** Server and training model share a single GPU; HTTP control plane, CUDA IPC data plane. Launches and tears down its own server
- [IPC weight syncing from a trainer that does not import vLLM](../../../examples/rl/rlhf_http_ipc_no_vllm.py) - `torch` + `requests` only, `ipc_handles_pickled` over HTTP; see [Trainer Without vLLM over HTTP](#trainer-without-vllm-over-http)
- [RLHF with IPC + FSDP2 and expert parallelism](../../../examples/rl/rlhf_ipc_fsdp_ep.py) - Multi-rank trainer colocated with a `--data-parallel-size 4` server on the same 4 GPUs: every FSDP rank builds an engine and joins the handle all-gather, with packed chunking and sleep/wake around the transfer - [RLHF with IPC + FSDP2 and expert parallelism](../../../examples/rl/rlhf_ipc_fsdp_ep.py) - Multi-rank trainer colocated with a `--data-parallel-size 4` server on the same 4 GPUs: every FSDP rank builds an engine and joins the handle all-gather, with packed chunking and sleep/wake around the transfer
+13 -18
View File
@@ -10,14 +10,18 @@ The `nccl_m2n` weight transfer engine uses [NCCL M2N](https://github.com/NVIDIA/
## Requirements ## Requirements
This backend depends on the `nccl-extensions` Python package (and its `nccl4py` dependency), which vLLM does not install. Install both against a compatible NCCL 2.30.5 or newer runtime. The backend creates its data-plane communicator through `nccl4py` and passes that communicator directly to `nccl.m2n`. This backend depends on the `nccl-extensions` Python package (and its `nccl4py` dependency), which vLLM does not install. It needs NCCL 2.30.5 or newer, and `libnccl_m2n.so` is linked against a specific NCCL build, so vLLM must load the same library:
```bash
export VLLM_NCCL_SO_PATH=/path/to/the/same/libnccl.so
```
The backend imports the runtime lazily, so vLLM is unaffected unless you select it. The backend imports the runtime lazily, so vLLM is unaffected unless you select it.
## How It Works ## How It Works
1. A `StatelessProcessGroup` exchanges metadata and the NCCL unique ID. The trainer ranks and all inference workers then create **one** `nccl.core.Communicator`: trainer ranks occupy `[0, T)` and workers `[T, T + N)`. The communicator is passed directly to each M2N collective. 1. The trainer ranks and all inference workers join **one** NCCL communicator via `StatelessProcessGroup`. Trainer ranks occupy `[0, T)` and workers `[T, T + N)` — m2n meshes are contiguous rank intervals, which is exactly what vLLM's existing `rank_offset` convention produces.
2. At the init handshake the trainer ships the full transfer plan: the session-wide destination mesh, then each parameter's name, dtype, shape, source mesh, and source placements. The workers resolve the matching **destination placements** and publish them over the data-plane communicator; the meshes themselves are declared, not negotiated. 2. At the init handshake the trainer ships the full transfer plan: both meshes — one per side, shared by every parameter — then each parameter's name, dtype, shape and placement. The workers resolve the matching **destination placements** and publish them back over the shared group; the meshes themselves are declared, not negotiated.
3. Each round, both sides issue one `reshard` per parameter in the same order. Every rank in the communicator participates in every reshard, so — as with the NCCL engine — the worker must be inside `update_weights` while the trainer sends. The trainer engine handles that concurrency itself. 3. Each round, both sides issue one `reshard` per parameter in the same order. Every rank in the communicator participates in every reshard, so — as with the NCCL engine — the worker must be inside `update_weights` while the trainer sends. The trainer engine handles that concurrency itself.
### Destination resolution ### Destination resolution
@@ -79,28 +83,19 @@ engine.send_weights()
`send_weights()` drives the whole round — `start_weight_update`, `update_weights`, `finish_weight_update`. Every rank in the communicator must be inside the same reshard, so `update_weights` has to be in flight while the trainer sends; the engine runs it on a helper thread so the trainer loop does not have to open-code that. `send_weights()` drives the whole round — `start_weight_update`, `update_weights`, `finish_weight_update`. Every rank in the communicator must be inside the same reshard, so `update_weights` has to be in flight while the trainer sends; the engine runs it on a helper thread so the trainer loop does not have to open-code that.
Trainers with a custom producer (a Megatron export, MoE re-fusing) subclass Trainers with a custom producer (a Megatron export, MoE re-fusing) subclass `M2NWeightSource` and supply layouts explicitly.
`M2NWeightSource` and supply layouts explicitly. A source may yield any
nccl4py-compatible CUDA buffer: a torch tensor, a CUDA Array Interface object,
or a raw device pointer. vLLM keeps buffer owners alive through stream
completion, using allocator stream tracking when the buffer provides it.
Sources must provide buffers on the communicator's current device; nccl4py
resolves their pointers but does not verify device ownership. Raw-pointer
sources are additionally responsible for pointer lifetime and matching their
declared dtype and local shape.
See [`examples/rl/rlhf_m2n.py`](../../../examples/rl/rlhf_m2n.py) for a runnable FSDP → TP example. See [`examples/rl/rlhf_m2n.py`](../../../examples/rl/rlhf_m2n.py) for a runnable FSDP → TP example.
## Limitations ## Limitations
- vLLM validates tensor-rank and dtype support while agreeing on the complete - Tensor rank 1..3; 4-D parameters are rejected.
plan; the installed M2N runtime validates them again for each reshard. - dtypes: int8/uint8, fp8 e4m3/e5m2, fp16, bf16, int32/uint32, fp32, int64/uint64, fp64. No fp4.
- Each parameter's source mesh must cover the same contiguous trainer-rank interval starting at 0, and be 1-D or 2-D. - The trainer's device mesh must cover a contiguous rank interval starting at 0, and be 1-D or 2-D.
- `Partial` placements are not supported. - `Partial` placements are not supported.
- Models with known model-level checkpoint preprocessing use the full-tensor - Models with known model-level checkpoint preprocessing use the full-tensor
fallback for the affected module. fallback for the affected module.
- A reshard cannot be issued from inside a CUDA graph capture (weight updates run outside capture, so this only matters for custom callers). - A reshard cannot be issued from inside a CUDA graph capture (weight updates run outside capture, so this only matters for custom callers).
- vLLM checks every parameter's shard overlap against conservative 16-source - The reshard plan is capped per *shard*, not per mesh: at most 16 source shards may feed one destination shard, and one source shard may feed at most 64 destination shards (`MAX_SOURCES` / `MAX_TARGETS` in the m2n build; raising either needs a rebuild).
and 64-target capacities before committing the plan. The installed M2N - A directly-resharded parameter spreads over `tp_size` destination shards, so each one is fed by fewer source shards. A parameter on the fallback path is replicated, i.e. a single destination shard fed by every source shard, so a trainer that **sharded** its parameters is still limited to 16 ranks whenever any parameter falls back — which fused parameters always do. A replicated trainer is unaffected.
runtime remains authoritative and validates each reshard again.
- Each communicator holds a cached staging pool (~256 MiB by default: 4 channels x 64 MiB). - Each communicator holds a cached staging pool (~256 MiB by default: 4 channels x 64 MiB).
-46
View File
@@ -130,51 +130,6 @@ pass — this is the invariant to test first.
IPC does not read `metadata()` at all: it derives the update info from IPC does not read `metadata()` at all: it derives the update info from
iteration as it goes, so it cannot observe a divergence. iteration as it goes, so it cannot observe a divergence.
## Trainer Without vLLM over HTTP
A trainer can speak the [HTTP endpoints](README.md#api-endpoints) directly, with
only PyTorch and an HTTP client. It mints the `ncclUniqueId` itself and is NCCL
rank 0. Reference: [`rlhf_http_nccl_no_vllm.py`](../../../examples/rl/rlhf_http_nccl_no_vllm.py).
`POST /init_weight_transfer_engine` body `{"init_info": {...}}`:
| Field | Default | Meaning |
| ----- | ------- | ------- |
| `nccl_unique_id_b64` | — | Standard (not URL-safe) base64 of the 128-byte `ncclUniqueId`. Replaces `master_address` / `master_port` |
| `rank_offset` | — | First worker rank; `1` for one trainer rank |
| `world_size` | — | Trainer ranks + `GET /get_world_size` |
| `packed` | `false` | Packed broadcasting |
| `packed_buffer_size_bytes` | 1 GiB | Must match the trainer's chunking |
| `packed_num_buffers` | 2 | Worker-side receive buffers |
- Worker rank = `rank_offset + dp_index * (TP * PP) + rank`.
- `/init_weight_transfer_engine` returns only after all ranks join: send it in
the background while the trainer calls `ncclCommInitRank(world_size, uid, 0)`.
There is no timeout; if the trainer never joins, the workers wait forever.
- Right after init every worker issues a warm-up `all_reduce` (sum) of
`torch.zeros(1)` (one float32 element). The trainer must issue the same one
before any other collective.
Each sync: `POST /start_weight_update`, then `POST /update_weights` with
`{"update_info": {"names", "dtype_names", "shapes"}}` **in the background**
while rank 0 broadcasts in `names` order, then `POST /finish_weight_update`.
`dtype_names` are `torch` attribute names (`"bfloat16"`); `shapes` are full
checkpoint shapes.
- Unpacked: one broadcast per tensor.
- Packed: concatenate `t.contiguous().view(-1).view(torch.uint8)` in `names`
order; broadcast a chunk as soon as its size **exceeds**
`packed_buffer_size_bytes` (the crossing tensor stays in the chunk), then the
remainder.
Trainer-side communicators without vLLM:
| Communicator | Status |
| ------------ | ------ |
| `torch.distributed.ProcessGroupNCCL` on a private `TCPStore` + `eager_connect_single_device` | Works; the unique id is read from store key `"0"`, a torch implementation detail (used by the example) |
| `ctypes` against torch's bundled `libnccl.so.2` | Works |
| `torch.cuda.nccl.init_rank` | Fails on Python 3.10+ (`PY_SSIZE_T_CLEAN` `SystemError`) |
## Sparse NCCL ## Sparse NCCL
Sparse, flat-index weight patches use Sparse, flat-index weight patches use
@@ -241,7 +196,6 @@ Sparse NCCL sends only `O(nnz)` indices and values. Checkpoint application still
## Examples ## Examples
- [RLHF with NCCL weight syncing (`vllm serve`, HTTP)](../../../examples/rl/rlhf_http_nccl.py) - **Start here.** Trainer on one GPU, 2x tensor-parallel fp8 server on two others; HTTP control plane, NCCL data plane. Launches and tears down its own server - [RLHF with NCCL weight syncing (`vllm serve`, HTTP)](../../../examples/rl/rlhf_http_nccl.py) - **Start here.** Trainer on one GPU, 2x tensor-parallel fp8 server on two others; HTTP control plane, NCCL data plane. Launches and tears down its own server
- [NCCL weight syncing from a trainer that does not import vLLM](../../../examples/rl/rlhf_http_nccl_no_vllm.py) - `torch` + `requests` only, `nccl_unique_id_b64` rendezvous; see [Trainer Without vLLM over HTTP](#trainer-without-vllm-over-http)
- [RLHF with NCCL + FSDP2 and expert parallelism](../../../examples/rl/rlhf_nccl_fsdp_ep.py) - Multi-rank trainer: every FSDP rank builds an engine and joins the `full_tensor()` gather while only rank 0 touches the wire - [RLHF with NCCL + FSDP2 and expert parallelism](../../../examples/rl/rlhf_nccl_fsdp_ep.py) - Multi-rank trainer: every FSDP rank builds an engine and joins the `full_tensor()` gather while only rank 0 touches the wire
- [RLHF with sparse NCCL weight syncing (offline, Ray)](../../../examples/rl/rlhf_sparse_nccl.py) - Qwen3 MoE per-expert checkpoint updates with one trainer GPU and two TP2/EP2 inference GPUs - [RLHF with sparse NCCL weight syncing (offline, Ray)](../../../examples/rl/rlhf_sparse_nccl.py) - Qwen3 MoE per-expert checkpoint updates with one trainer GPU and two TP2/EP2 inference GPUs
- [RLHF with async weight syncing (offline, Ray)](../../../examples/rl/rlhf_async_new_apis.py) - Async generation with mid-flight pause, weight sync, resume, and validation against a fresh model; uses an in-process `AsyncLLMEngine` with `RayVLLMWeightSyncClient` - [RLHF with async weight syncing (offline, Ray)](../../../examples/rl/rlhf_async_new_apis.py) - Async generation with mid-flight pause, weight sync, resume, and validation against a fresh model; uses an in-process `AsyncLLMEngine` with `RayVLLMWeightSyncClient`
+2 -2
View File
@@ -423,9 +423,9 @@ Most cache paths default to subdirectories under a single root. Changing `VLLM_C
| Environment Variable | Default | Description | | Environment Variable | Default | Description |
| --- | --- | --- | | --- | --- | --- |
| `VLLM_CACHE_ROOT` | `~/.cache/vllm` | Base directory for vLLM-owned caches. Respects `XDG_CACHE_HOME` if set. | | `VLLM_CACHE_ROOT` | `~/.cache/vllm` | Base cache directory. Respects `XDG_CACHE_HOME` if set. All paths below inherit from this unless explicitly overridden. |
| *(torch.compile)* | `$VLLM_CACHE_ROOT/torch_compile_cache/` | Compilation cache for AOT-compiled models, Inductor graphs, and Triton kernels. Controlled by `VLLM_DISABLE_COMPILE_CACHE` (set to `1` to disable). | | *(torch.compile)* | `$VLLM_CACHE_ROOT/torch_compile_cache/` | Compilation cache for AOT-compiled models, Inductor graphs, and Triton kernels. Controlled by `VLLM_DISABLE_COMPILE_CACHE` (set to `1` to disable). |
| `VLLM_FLASHINFER_AUTOTUNE_CACHE_DIR` | Managed: `~/.cache/flashinfer/autotune/`; legacy: `$VLLM_CACHE_ROOT/flashinfer_autotune_cache/<flashinfer-version>/<arch>/<cache-hash>/` | Overrides cache placement for both FlashInfer autotune cache APIs. | | `VLLM_FLASHINFER_AUTOTUNE_CACHE_DIR` | `$VLLM_CACHE_ROOT/flashinfer_autotune_cache/<flashinfer-version>/<arch>/<cache-hash>/` | FlashInfer autotune config cache. |
| `VLLM_ASSETS_CACHE` | `$VLLM_CACHE_ROOT/assets/` | Downloaded assets (e.g., tokenizer files). | | `VLLM_ASSETS_CACHE` | `$VLLM_CACHE_ROOT/assets/` | Downloaded assets (e.g., tokenizer files). |
| `VLLM_XLA_CACHE_PATH` | `$VLLM_CACHE_ROOT/xla_cache/` | XLA/TPU compilation cache. | | `VLLM_XLA_CACHE_PATH` | `$VLLM_CACHE_ROOT/xla_cache/` | XLA/TPU compilation cache. |
| `VLLM_MEDIA_CACHE` | *(disabled)* | Optional cache for downloaded media (images, video, audio). Not enabled unless explicitly set. | | `VLLM_MEDIA_CACHE` | *(disabled)* | Optional cache for downloaded media (images, video, audio). Not enabled unless explicitly set. |
@@ -473,11 +473,9 @@ async def stop_profile():
if __name__ == "__main__": if __name__ == "__main__":
parser = argparse.ArgumentParser() parser = argparse.ArgumentParser()
parser.add_argument("--port", type=int, default=10001) parser.add_argument("--port", type=int, default=10001)
# Port the prefill/decode servers register on (their proxy_ping_port).
parser.add_argument("--service-discovery-port", type=int, default=36367)
args = parser.parse_args() args = parser.parse_args()
t = start_service_discovery("0.0.0.0", args.service_discovery_port) t = start_service_discovery("0.0.0.0", 36367)
# High-concurrency hardening. Quart's app.run() uses a shallow listen # High-concurrency hardening. Quart's app.run() uses a shallow listen
# backlog (100) and, with app.debug=True, adds per-request overhead that # backlog (100) and, with app.debug=True, adds per-request overhead that
# starves the single accept loop. Under a burst of ~512 simultaneous client # starves the single accept loop. Under a burst of ~512 simultaneous client
@@ -340,10 +340,7 @@ async def _handle_completions(api: str, request: Request):
): ):
yield chunk yield chunk
media_type = ( return StreamingResponse(generate_stream(), media_type="application/json")
"text/event-stream" if req_data.get("stream") else "application/json"
)
return StreamingResponse(generate_stream(), media_type=media_type)
except Exception as e: except Exception as e:
import sys import sys

Some files were not shown because too many files have changed in this diff Show More